Author Archives: ant.weiss@gmail.com

  • 0

Impressions from DevOpsDays Moscow 2017

Taking the Stage at DevOpsDays Moscow

The Russian Link

I’m on my way back from the first ever DevOpsDays event in Russia where I had the privilege to share the stage with shockingly gifted and knowledgeable speakers. This may sound like just another DevOpsDays to you, but for me it was a big deal. As some of you may know I was born in Russia and lived in St.Petersburg until the age of 15. I am a native russian speaker, and though I’ve spent almost two thirds of my life in Israel – the link to russian culture has never been broken. So when I saw the event planned on the official DevOpsDays community site – I applied to give a talk and was happy to get accepted.

 

During my 17 years in the industry I never had a chance to work with russian software engineers (not counting the Israeli Russians – these I’ve seen plenty). So I had no idea about how big and developed the Russian IT industry is. Well, we all recently heard that the mighty Russian hackers helped Trump to become president, so I was quite curious to experience all that brainpower firsthand.

I must say – I was not disappointed.

Applauding the Organisers!

First of all – the conference was very well organised. The venue was great, video and sound worked fine and the conference networking app supplied by http://meyou.ru was probably the best I’ve ever seen.

The event was conceived and brought to life by two local IT consulting companies – Logrocon and Express42. Special thanks go to Boris Zlobin, Evgeny Ognevoy, Alexander Titov and Mikhail Krivilev for conducting and keeping up the good vibes all through the day.

But the best thing about the conference are its participants. Both the speakers and the listeners. The Russian devops crowd struck me as very thoughtful, knowledgeable and passionate about their work. All that with a healthy dose of hackerly cynicism.

The Highlights

The keynote was delivered by Jan De Vries who told us about applying Nassim Taleb’s concept of anti-fragility to software architecture and delivery. The talk was good, but no eye-opener for me personally as this same idea was proposed by Asher Sterkin in his presentation at DevOpsDays TLV in 2013.

Interestingly some of the best talks came from big russian institutions and banks. The dev manager and the ops manager from Raiffeisen Bank gave a super-impressive pair talk describing their long and winding road to DevOps. Another great pair talk came from Alpha-Bank – full of humorous but very practical advice on how to implement ‘DevOps without bullshit’.

Leon Fayer came all the way from Baltimore, US to give a fiery ignite about what DevOps is not and also a powerful introduction to taking DevOps all the way to BizOps.

In general – it was good to see that we’re all playing the same game. People were eagerly discussing problems of cooperation and burnout along with how to run containers and if configuration management tools are any good.

Konstantin Nazarov from Tarantool presented a simple, no frills Docker orchestration solution based on Consul and a self-written Python wrapper.

Our gifted compatriot and my friend Ilya Sher expressed the growing frustration with existing CM tools and suggested using his New Generation Shell for building simple and manageable custom solutions.

For those among us who still feel CM tools can provide certain value and not only pain ( I do believe there are such scenarios)  – Ansible was definitely the star of the show. It got featured in a number of talks and there was a workshop on writing custom Ansible modules.

 

Marcin Wielgus  – the lead Kubernetes developer from Poland who we had the pleasure to host at DevConTLV X now brought his message of smart container orchestration to Russia. Kubernetes is certainly a hot topic all around the globe. So much so that it became the subject of one of the open space sessions at the end of the day.

 

Open Spaces

I personally chose to visit another open space that was dedicated to developing the DevOps community in Russia. The session touched me deeply because folks were talking about the need to support each other, to fight snobbism and ban flame wars. Everybody agreed that generation of quality content is the key to community-building. Human societies are centered around shared stories. Konstantin Nazarov outlined the fact that technical people are rarely good in writing and speaking as this is something that needs to be learned. He then offered mentorship to whoever wants to share their knowledge but isn’t sure where and how to start.

 

This was very inspiring. We can certainly use this type of mentorship back in Israel. Being the Jenkins Area Meetup organiser for the last year I’ve come to realise that it’s not easy to find community speakers. Consultants and evangelists (like myself) can also hold great, valuable talks, but it’s much harder to learn about the actual experiences from the trenches of corporate wars.

So if you’re reading this and thinking that you’d like to tell your own story – feel free to drop me a line and I’ll be happy to share some speaking/writing tips.

Conclusion

Moscow is beautiful, Russian hackers are great guys and we can all learn from each other. Containers and their orchestrators are still all the buzz, there’s some talk of the serverless, and Ansible is responsible for picking up what’s left for configuration management. And the largest challenge is the same as everywhere – how to get humans to work together effectively at scale. It certainly seems Russians can teach us a few lessons here – they know about scale. I hope we have some Russian speakers on our next DevOpsDays TLV.


  • 0

Thank you Intel Sports!

Category : Tools

intelinsports

Mission completed! We’ve done a full month of getting the #Intel Sports developers up to speed with git. It’s always fun to train bright folks – and the engineers at Intel are certainly among the brighthest we ‘ve had the privilege to preach git to.

While providing the training we’ve also developed a few ideas regarding git subtree and the plan is to share these ideas in a follow-up post to this one (which compares submodules to repo)

Have a great weekend!

 


  • 0

CI/CD for Microservices – Challenges and Best Practices

2625fb74-c674-49e3-b87d-08b6d807c273_749_499

Introduction:

Microservice Software Architecture is a software system architecture pattern whereas an application or a system is composed of a number of smaller interconnected services. This is in opposite to the previously popular monolith architectures in which, even if having a logically modular, component-based structure the application is packaged and deployed as a monolith.

The Microservice architectural pattern while having many benefits (which we’ll briefly outline in the following paragraph) also presents new challenges all along our software delivery pipeline. This whitepaper strives to map out these challenges and define the best practices for tackling them to ensure a streamlined and quality-oriented delivery process.

Microservice Architecture Benefits:

  • Smaller Application Footprint (per Service)

The ‘Micro’ notion of the concept has been getting some justified critic – as it’s not really about the size of the codebase, but more about correct logical separation of concerns. Still once we do split our existing or potential monolith into a number of services  – each separate service will inevitably have a smaller footprint which brings us the following benefits:

  • Comprehensibility

Smaller applications are generally easier to understand, debug and change than complex, large systems.

  • Shorter Development Time

Smaller applications start-up time is shorter, making developers more efficient in their dev-test cycles and allowing for agile development.

  • Improved Continuous Delivery/Deployment Support

Updating a large complex system takes a lot of time while the implication of each change is hard to identify. On the other hand – when working with microservices – each service is (or should be) deployable independently, which makes it much easier to update just a part of the system and rollback only the breaking change if anything goes wrong.

  • Scalability Improvement

When each service is scalable independently it becomes easier to provision the resources adapted for its special needs. Additionally we don’t need to scale everything. We can for example run a few instances of front-end logic but only one instance of business logic.

  • Easier to adopt new frameworks/technologies.

Unlike a monolith we don’t have to write everything in Java or Python (for example). As each service is installed and built independently – it can be written in a different language and based on a different framework. As long as it supports the well-defined API/message protocol to communicate with other services.

  • Allows for a more efficient organizational structure.

It’s been noted by a number of studies that individual performance is reduced when the overall team size grows beyond a specific size (namely 5-7 engineers). (See also the 2-Pizza-Team concept) This is caused by several reasons like coordination/communication complexity, the ‘social loafing’ and ‘free rider’ phenomena. Microservice architecture allows a small team to maintain the development of each service – thus allowing for more efficient development iterations and improved communication.

And now that we’ve outlined the benefits, let’s look at the challenges of this architectural pattern as manifested in CI/CD concerns.

The Challenges of Microservice Architecture (from CI/CD perspective)

All challenges of microservices are caused by the complexity which originates from the fact that we are dealing with a distributed system. Distributed systems are inherently more difficult to plan, build and reason about. Here’s a list of specific challenges we will encounter:

  • Dependency Management

One of the biggest enemies of distributed architectures are dependencies. In a microservice-based architecture, services are modeled as isolated units that manage a reduced set of problems. However, fully functional systems rely on the cooperation and integration of its parts, and microservice architectures are not an exception. The question then becomes: how do we manage dependencies between multiple fast-moving, independently evolving components?

  • Versioning and Backward Compatibility

Microservices are developed in isolation with each service having its own distinct lifecycle. This requires us to define very specific versioning rules and requirements. Each service has to make absolutely clear which versions of dependent services it relies upon.

  • Data partitioning and sharing

Best practices of microservice development propose having a separate database for each service. However this isn’t always feasible and surely is never easy when you have transactions spanning multiple services. In CI/CD context this may mean we have to deal with multiple inter-related schema updates.

  • Testing

While being able to operate in isolation – a microservice isn’t worth much without its counterparts. On the other hand – bringing up the full system topology for testing just one service cancels out the benefits of modularity and encapsulation that microservices are supposed to bring. The challenge here is to define the minimum viable system testing subset and/or provide good enough mockups/stubs for testing in absence of real services. Additional challenges lie in service communication patterns which mostly occur over network and therefore must take in account possible network hiccups and outages.

  • Resource Provisioning and Deployment

In a microservice architecture each service should be independently deployable. On the other hand we need a way to know where and how to find this service. We need a way to tell our services where the shared resources (like databases, data collectors and message queues) reside and how to access them. This brings about the need for service discovery, configuration separation from business functionality and failure resilience in case certain service is temporarily missing/unavailable.

  • Variety/Polyglossia

Microservices allow us to develop each service in a different language and using a different framework. This lets us use the right tool for the job in each individual case but it’s a mixed blessing from the delivery viewpoint. Because our delivery pipeline is most efficient when it defines a clear unified framework with distinct building blocks and a simple API. This may become challenging when having to support a variety of technologies, build tools and frameworks.

Tackling the Microservice Architecture Challenges in the CI/CD Pipeline (and beyond)

Now that we’ve outlined the challenges accompanying the delivery of microservice-based systems, let’s define the best practices for dealing with them when building a modern CI/CD pipeline.

Dependency Management:

Even before looking at build-time dependency management we need to look at the wider concepts of service inter-dependency. With microservices each service is meant to be able to operate on its own. Therefore in an ideal setting no direct build-time dependencies should be needed at all. At the maximum a dependency on a common communication protocol and API version can be in place, with version changes taken care of by backward compatibility and consumer-driven contracts.

In order to achieve this the architectural concepts of loose coupling and problem locality should be applied when splitting up our system into separate services.

  • Loose coupling: microservices should be able to be modified without requiring changes in other microservices.
  • Problem locality: related problems should be grouped together (in other words, if a change requires an update in another part of the system, those parts should be close to each other).

 

      • If two or more services are too tightly coupled – i.e. have too many correlated changes which require careful coordination – it should be considered to unify them into one service.
      • If we’re not in the ideal setting of loose coupling and concern separation and re-architecting the system is currently impossible (for lack of resources or business reasons) then strict semantic versioning should be applied to all interdependent services. This is to make sure we are building and deploying against correct versions of service counterparts.

Versioning:

As stated in the previous paragraph – semantic versioning is a good way of signifying when a breaking change occurs in the service semantics or data schema. In practice this means that any given service should be able to talk to another service as long as the contract between them is sealed. With the MAJOR field of semantic version being the guarantee of that seal. For experimental or feature branches – the name of the branch can be added as metadata to the version name as suggested here: http://semver.org/#spec-item-10

Data Partitioning:

  • If each service is based on its own database then database schema changes and deployment becomes the responsibility of that service installation scripts. For the CI/CD pipeline it means we need to be able to support multiple database deployment in our test and staging environment provisioning cycles.
  • If services share databases it means we need to identify all the data dependencies and retest all the dependent services whenever a shared data schema is changed.  

Testing

  • For a much deeper look at microservice testing patterns look here: http://martinfowler.com/articles/microservice-testing
  • Deployment of individual services should be a part of the end-to-end test to verify successful upgrade and rollback procedures as part of the full system test.
  • End-to-End Tests should be only executed after unit and integration tests have completed successfully and test coverage thresholds have been met. This is because the setup and execution of the e2e environment tends to be difficult and error-prone and we should introduce sufficient gating to ensure its stability.
  • In such a case it may be a good idea to separate integration tests in CI pipeline so that external outages don’t block development.
  • Due to interservice communication reliance on network and overall system complexity integration tests can be expected to fail with higher frequency due to non-related infrastructure or version dependency errors.
  • Integration tests: As stated earlier – the minimum viable subset of interdependent services should be identified wherever possible to simplify testing environment provisioning and deployment.
  • Automated Deployment becomes absolutely necessary with each service deployable by itself and a deployment orchestration solution (e.g Ansible playbook) describing the various topologies and event sequences.
  • Test doubles (Mocks, Stubs, etc) should be encouraged as a tool for testing service functionality in isolation.
  • Coverage thresholds are a good strategy for ensuring we’re writing tests for all the new functionality.
  • Unit tests become especially important in microservice environment as they allow for faster feedback without the need to instantiate the collaborating services. Test-Driven Development should be encouraged and unit test coverage should be measured.

Resource Provisioning and Deployment

  • Infrastructure-as-Code approach should be used for versioned and testable provisioning and deployment processes.
  • Microservices should enable horizontal scaling across a compute resource cluster. This calls for using:
    • A central configuration mechanism in a form of a distributed Key-Value store (such as Consul or etcd). Our CI/CD pipeline should support separate deployment of configuration to that store.
    • A cluster task scheduler (e.g Docker Swarm, Mesos, Kubernetes or Nomad). The CD process needs to interface with whichever system we choose  for implementing scratch rollouts, rolling updates and blue/green deployments.
    • Microservice architectures are often enabled by OS container technologies like Docker. Containers as a packaging an delivery mechanism should definitely be evaluated.

Variety/Polyglossia

It is very desirable to base the CI/CD flow for each service on the same template which includes the same building blocks and a well defined interface. I.e each service should provide similar ‘build’ , ‘test’, ‘deploy’ and ‘promote’ endpoints for integration into the CI system. Additionally the interface for querying service interdependency should be clearly defined. This will allow for almost instant CI/CD flow creation for each new service and will reduce the load on the CI/CD team. Ultimately this should allow developers to plug-n-play new services into the CI/CD system in a fully self-service mode.

Otomato Software ltd. 2016 All Rights Reserved.®


  • 0

DevOps Flow Metrics – http://devopsflowmetrics.org

Category : Tools

grpah

DevOps transformation goals can be defined as:

  • Heightened Release Agility
  • Improved Software Quality

Or simply:

Delivering Better Software Faster

Therefore measurable DevOps success criteria would be:

  • Being able to release versions faster and more often.
  • Having less defects and failures.

Measurement is one of the cornerstones of DevOps. But how do we measure flow?

In order to track the flow (the amount of change getting pushed through our pipeline in a given unit of time) we’ve developed the 12 DevOps Flow Metrics.

They are based on our industry experience and ideas from other DevOps practitioners and are a result of 10 years of implementing DevOps and CI/CD in large organisations.

The metrics were initially publicly presented by Anton Weiss at a DevOpsDays TLV 2016 ignite talk. The talk got a lot of people interested and that’s why we decided to share the metrics with the community.

We’ve created a github pages based minisite where everyone can learn about the metrics, download the presentation and submit comments and pull requests.

Looking forward to your feedback!

Get the metrics here : http://devopsflowmetrics.org

 


  • 0

  • 0

Jenkins and the Future of Software Delivery

Are you optimistic when you look into the future and try to see what it brings? Do you believe in robot apocalypse or the utopia of singularity? Do you think the world will change to the better or to the worse? Or are you just too busy fixing bugs in production and making sure all your systems are running smoothly?

Ant Weiss of Otomato describing the bright future of software delivery at Jenkins User Conference Israel 2016.


  • 0

How OpenStack is Built – the Video now Online

Watch Ant Weiss of Otomato provide an overview of the OpenStack CI – probably one of the most advanced Jenkins-based CI infrastructures in the world.


  • 0

The Number One Symptom Of not Actually ‘Doing Devops’

Category : Tools

apply-now-job-hiring-help-ss-1920So I recently talked to a release engineering team leader at a very well-known american software+hardware company located in California.
They contacted me looking for top-notch build infrastructure engineers and we spent a very interesting hour discussing their technological stack and people processes. On the surface – they are moving in the right direction – automating all the things, using Chef for config management, codifying the infrastructure, using Artifactory for binaries… But one thing struck me in our conversation. The team leader (clearly a very smart and talented guy) said : “I’ve been trying to hire for two years. Good professionals are so hard to find…”
I agreed at first – we all know the market is starving for technical talent.
But reflecting on our conversation afterwards I realized that he in fact showed me the number one symptom of how far their team is from ‘doing devops’.

Yes – hiring is challenging. I’ve discussed this before. It may take some time to find the person whose mindset, values and the overall vibe suite your expectations. Even more so – if you’re low on budget, as the market has become highly competitive. But – for this specific team leader the budget was not an issue. He was failing to hire because of his organisation’s unwillingness to invest in mentoring and development.

He was looking for the ready-made fullstack ninja with a very specific skillset. there is no other way to explain his 2 year long quest.

Can you see how this totally opposes the ever important Devops value of Sharing? So you’ve built a few things to be proud of, you’re technologically savvy, you’re on the bleeding edge. Now is the time to share your knowledge! Now is the time to go hire bright novices. They are out there, they are hungry to learn from your experience and by sharing with them you will build the real devops on your team.

So if you’re failing to hire for more than a couple of months – don’t blame the market. Don’t complain about lousy candidates. Go revise your hiring process and more importantly – the way your team works. You may have all the technology, but it looks like your culture is broken. And with broken culture – even if you eventually succeed in hiring , you’ll have hard time reaping the true benefits of the Devops way – agility, quality, motivation and trust.

And may the devops be with you!


  • 0

Jenkins 2.0: With Jenkins Creator – Kohsuke Kawaguchi (KK)

Category : Tools

Hi all!
We really hope you’ve already registered to the Jenkins User Conference Israel which was SOLD OUT yesterday.
But even if you haven’t – there’s still a chance to hang out with Kohsuke Kawaguchi – the father of Jenkins himself – and other Jenkins fans at a meetup the good people at JFrog are organising the day after the conference.
Here is the link: http://meetu.ps/e/BHkfn/jvVGM/d
Right now there are still 22 slots available.
Go grab yours.

Keep delivering!


  • 0

Discount tickets to Jenkins User Conference Tel Aviv

Category : Tools

jenkinsCIA
We are happy to announce that @antweiss will be speaking about the future of software delivery at JUC Israel 2016

We have a couple of 100 NIS discount tickets to the conference and ticket sale is closing tomorrow!

The first two folks to comment on this post will get the discount code.

Type them comments!!!