• 0

Impressions from DevOpsDays Moscow 2017

Taking the Stage at DevOpsDays Moscow

The Russian Link

I’m on my way back from the first ever DevOpsDays event in Russia where I had the privilege to share the stage with shockingly gifted and knowledgeable speakers. This may sound like just another DevOpsDays to you, but for me it was a big deal. As some of you may know I was born in Russia and lived in St.Petersburg until the age of 15. I am a native russian speaker, and though I’ve spent almost two thirds of my life in Israel – the link to russian culture has never been broken. So when I saw the event planned on the official DevOpsDays community site – I applied to give a talk and was happy to get accepted.

 

During my 17 years in the industry I never had a chance to work with russian software engineers (not counting the Israeli Russians – these I’ve seen plenty). So I had no idea about how big and developed the Russian IT industry is. Well, we all recently heard that the mighty Russian hackers helped Trump to become president, so I was quite curious to experience all that brainpower firsthand.

I must say – I was not disappointed.

Applauding the Organisers!

First of all – the conference was very well organised. The venue was great, video and sound worked fine and the conference networking app supplied by http://meyou.ru was probably the best I’ve ever seen.

The event was conceived and brought to life by two local IT consulting companies – Logrocon and Express42. Special thanks go to Boris Zlobin, Evgeny Ognevoy, Alexander Titov and Mikhail Krivilev for conducting and keeping up the good vibes all through the day.

But the best thing about the conference are its participants. Both the speakers and the listeners. The Russian devops crowd struck me as very thoughtful, knowledgeable and passionate about their work. All that with a healthy dose of hackerly cynicism.

The Highlights

The keynote was delivered by Jan De Vries who told us about applying Nassim Taleb’s concept of anti-fragility to software architecture and delivery. The talk was good, but no eye-opener for me personally as this same idea was proposed by Asher Sterkin in his presentation at DevOpsDays TLV in 2013.

Interestingly some of the best talks came from big russian institutions and banks. The dev manager and the ops manager from Raiffeisen Bank gave a super-impressive pair talk describing their long and winding road to DevOps. Another great pair talk came from Alpha-Bank – full of humorous but very practical advice on how to implement ‘DevOps without bullshit’.

Leon Fayer came all the way from Baltimore, US to give a fiery ignite about what DevOps is not and also a powerful introduction to taking DevOps all the way to BizOps.

In general – it was good to see that we’re all playing the same game. People were eagerly discussing problems of cooperation and burnout along with how to run containers and if configuration management tools are any good.

Konstantin Nazarov from Tarantool presented a simple, no frills Docker orchestration solution based on Consul and a self-written Python wrapper.

Our gifted compatriot and my friend Ilya Sher expressed the growing frustration with existing CM tools and suggested using his New Generation Shell for building simple and manageable custom solutions.

For those among us who still feel CM tools can provide certain value and not only pain ( I do believe there are such scenarios)  – Ansible was definitely the star of the show. It got featured in a number of talks and there was a workshop on writing custom Ansible modules.

 

Marcin Wielgus  – the lead Kubernetes developer from Poland who we had the pleasure to host at DevConTLV X now brought his message of smart container orchestration to Russia. Kubernetes is certainly a hot topic all around the globe. So much so that it became the subject of one of the open space sessions at the end of the day.

 

Open Spaces

I personally chose to visit another open space that was dedicated to developing the DevOps community in Russia. The session touched me deeply because folks were talking about the need to support each other, to fight snobbism and ban flame wars. Everybody agreed that generation of quality content is the key to community-building. Human societies are centered around shared stories. Konstantin Nazarov outlined the fact that technical people are rarely good in writing and speaking as this is something that needs to be learned. He then offered mentorship to whoever wants to share their knowledge but isn’t sure where and how to start.

 

This was very inspiring. We can certainly use this type of mentorship back in Israel. Being the Jenkins Area Meetup organiser for the last year I’ve come to realise that it’s not easy to find community speakers. Consultants and evangelists (like myself) can also hold great, valuable talks, but it’s much harder to learn about the actual experiences from the trenches of corporate wars.

So if you’re reading this and thinking that you’d like to tell your own story – feel free to drop me a line and I’ll be happy to share some speaking/writing tips.

Conclusion

Moscow is beautiful, Russian hackers are great guys and we can all learn from each other. Containers and their orchestrators are still all the buzz, there’s some talk of the serverless, and Ansible is responsible for picking up what’s left for configuration management. And the largest challenge is the same as everywhere – how to get humans to work together effectively at scale. It certainly seems Russians can teach us a few lessons here – they know about scale. I hope we have some Russian speakers on our next DevOpsDays TLV.


  • 0

Thank you Intel Sports!

Category : Tools

intelinsports

Mission completed! We’ve done a full month of getting the #Intel Sports developers up to speed with git. It’s always fun to train bright folks – and the engineers at Intel are certainly among the brighthest we ‘ve had the privilege to preach git to.

While providing the training we’ve also developed a few ideas regarding git subtree and the plan is to share these ideas in a follow-up post to this one (which compares submodules to repo)

Have a great weekend!

 


  • 0

CI/CD for Microservices – Challenges and Best Practices

2625fb74-c674-49e3-b87d-08b6d807c273_749_499

Introduction:

Microservice Software Architecture is a software system architecture pattern whereas an application or a system is composed of a number of smaller interconnected services. This is in opposite to the previously popular monolith architectures in which, even if having a logically modular, component-based structure the application is packaged and deployed as a monolith.

The Microservice architectural pattern while having many benefits (which we’ll briefly outline in the following paragraph) also presents new challenges all along our software delivery pipeline. This whitepaper strives to map out these challenges and define the best practices for tackling them to ensure a streamlined and quality-oriented delivery process.

Microservice Architecture Benefits:

  • Smaller Application Footprint (per Service)

The ‘Micro’ notion of the concept has been getting some justified critic – as it’s not really about the size of the codebase, but more about correct logical separation of concerns. Still once we do split our existing or potential monolith into a number of services  – each separate service will inevitably have a smaller footprint which brings us the following benefits:

  • Comprehensibility

Smaller applications are generally easier to understand, debug and change than complex, large systems.

  • Shorter Development Time

Smaller applications start-up time is shorter, making developers more efficient in their dev-test cycles and allowing for agile development.

  • Improved Continuous Delivery/Deployment Support

Updating a large complex system takes a lot of time while the implication of each change is hard to identify. On the other hand – when working with microservices – each service is (or should be) deployable independently, which makes it much easier to update just a part of the system and rollback only the breaking change if anything goes wrong.

  • Scalability Improvement

When each service is scalable independently it becomes easier to provision the resources adapted for its special needs. Additionally we don’t need to scale everything. We can for example run a few instances of front-end logic but only one instance of business logic.

  • Easier to adopt new frameworks/technologies.

Unlike a monolith we don’t have to write everything in Java or Python (for example). As each service is installed and built independently – it can be written in a different language and based on a different framework. As long as it supports the well-defined API/message protocol to communicate with other services.

  • Allows for a more efficient organizational structure.

It’s been noted by a number of studies that individual performance is reduced when the overall team size grows beyond a specific size (namely 5-7 engineers). (See also the 2-Pizza-Team concept) This is caused by several reasons like coordination/communication complexity, the ‘social loafing’ and ‘free rider’ phenomena. Microservice architecture allows a small team to maintain the development of each service – thus allowing for more efficient development iterations and improved communication.

And now that we’ve outlined the benefits, let’s look at the challenges of this architectural pattern as manifested in CI/CD concerns.

The Challenges of Microservice Architecture (from CI/CD perspective)

All challenges of microservices are caused by the complexity which originates from the fact that we are dealing with a distributed system. Distributed systems are inherently more difficult to plan, build and reason about. Here’s a list of specific challenges we will encounter:

  • Dependency Management

One of the biggest enemies of distributed architectures are dependencies. In a microservice-based architecture, services are modeled as isolated units that manage a reduced set of problems. However, fully functional systems rely on the cooperation and integration of its parts, and microservice architectures are not an exception. The question then becomes: how do we manage dependencies between multiple fast-moving, independently evolving components?

  • Versioning and Backward Compatibility

Microservices are developed in isolation with each service having its own distinct lifecycle. This requires us to define very specific versioning rules and requirements. Each service has to make absolutely clear which versions of dependent services it relies upon.

  • Data partitioning and sharing

Best practices of microservice development propose having a separate database for each service. However this isn’t always feasible and surely is never easy when you have transactions spanning multiple services. In CI/CD context this may mean we have to deal with multiple inter-related schema updates.

  • Testing

While being able to operate in isolation – a microservice isn’t worth much without its counterparts. On the other hand – bringing up the full system topology for testing just one service cancels out the benefits of modularity and encapsulation that microservices are supposed to bring. The challenge here is to define the minimum viable system testing subset and/or provide good enough mockups/stubs for testing in absence of real services. Additional challenges lie in service communication patterns which mostly occur over network and therefore must take in account possible network hiccups and outages.

  • Resource Provisioning and Deployment

In a microservice architecture each service should be independently deployable. On the other hand we need a way to know where and how to find this service. We need a way to tell our services where the shared resources (like databases, data collectors and message queues) reside and how to access them. This brings about the need for service discovery, configuration separation from business functionality and failure resilience in case certain service is temporarily missing/unavailable.

  • Variety/Polyglossia

Microservices allow us to develop each service in a different language and using a different framework. This lets us use the right tool for the job in each individual case but it’s a mixed blessing from the delivery viewpoint. Because our delivery pipeline is most efficient when it defines a clear unified framework with distinct building blocks and a simple API. This may become challenging when having to support a variety of technologies, build tools and frameworks.

Tackling the Microservice Architecture Challenges in the CI/CD Pipeline (and beyond)

Now that we’ve outlined the challenges accompanying the delivery of microservice-based systems, let’s define the best practices for dealing with them when building a modern CI/CD pipeline.

Dependency Management:

Even before looking at build-time dependency management we need to look at the wider concepts of service inter-dependency. With microservices each service is meant to be able to operate on its own. Therefore in an ideal setting no direct build-time dependencies should be needed at all. At the maximum a dependency on a common communication protocol and API version can be in place, with version changes taken care of by backward compatibility and consumer-driven contracts.

In order to achieve this the architectural concepts of loose coupling and problem locality should be applied when splitting up our system into separate services.

  • Loose coupling: microservices should be able to be modified without requiring changes in other microservices.
  • Problem locality: related problems should be grouped together (in other words, if a change requires an update in another part of the system, those parts should be close to each other).

 

      • If two or more services are too tightly coupled – i.e. have too many correlated changes which require careful coordination – it should be considered to unify them into one service.
      • If we’re not in the ideal setting of loose coupling and concern separation and re-architecting the system is currently impossible (for lack of resources or business reasons) then strict semantic versioning should be applied to all interdependent services. This is to make sure we are building and deploying against correct versions of service counterparts.

Versioning:

As stated in the previous paragraph – semantic versioning is a good way of signifying when a breaking change occurs in the service semantics or data schema. In practice this means that any given service should be able to talk to another service as long as the contract between them is sealed. With the MAJOR field of semantic version being the guarantee of that seal. For experimental or feature branches – the name of the branch can be added as metadata to the version name as suggested here: http://semver.org/#spec-item-10

Data Partitioning:

  • If each service is based on its own database then database schema changes and deployment becomes the responsibility of that service installation scripts. For the CI/CD pipeline it means we need to be able to support multiple database deployment in our test and staging environment provisioning cycles.
  • If services share databases it means we need to identify all the data dependencies and retest all the dependent services whenever a shared data schema is changed.  

Testing

  • For a much deeper look at microservice testing patterns look here: http://martinfowler.com/articles/microservice-testing
  • Deployment of individual services should be a part of the end-to-end test to verify successful upgrade and rollback procedures as part of the full system test.
  • End-to-End Tests should be only executed after unit and integration tests have completed successfully and test coverage thresholds have been met. This is because the setup and execution of the e2e environment tends to be difficult and error-prone and we should introduce sufficient gating to ensure its stability.
  • In such a case it may be a good idea to separate integration tests in CI pipeline so that external outages don’t block development.
  • Due to interservice communication reliance on network and overall system complexity integration tests can be expected to fail with higher frequency due to non-related infrastructure or version dependency errors.
  • Integration tests: As stated earlier – the minimum viable subset of interdependent services should be identified wherever possible to simplify testing environment provisioning and deployment.
  • Automated Deployment becomes absolutely necessary with each service deployable by itself and a deployment orchestration solution (e.g Ansible playbook) describing the various topologies and event sequences.
  • Test doubles (Mocks, Stubs, etc) should be encouraged as a tool for testing service functionality in isolation.
  • Coverage thresholds are a good strategy for ensuring we’re writing tests for all the new functionality.
  • Unit tests become especially important in microservice environment as they allow for faster feedback without the need to instantiate the collaborating services. Test-Driven Development should be encouraged and unit test coverage should be measured.

Resource Provisioning and Deployment

  • Infrastructure-as-Code approach should be used for versioned and testable provisioning and deployment processes.
  • Microservices should enable horizontal scaling across a compute resource cluster. This calls for using:
    • A central configuration mechanism in a form of a distributed Key-Value store (such as Consul or etcd). Our CI/CD pipeline should support separate deployment of configuration to that store.
    • A cluster task scheduler (e.g Docker Swarm, Mesos, Kubernetes or Nomad). The CD process needs to interface with whichever system we choose  for implementing scratch rollouts, rolling updates and blue/green deployments.
    • Microservice architectures are often enabled by OS container technologies like Docker. Containers as a packaging an delivery mechanism should definitely be evaluated.

Variety/Polyglossia

It is very desirable to base the CI/CD flow for each service on the same template which includes the same building blocks and a well defined interface. I.e each service should provide similar ‘build’ , ‘test’, ‘deploy’ and ‘promote’ endpoints for integration into the CI system. Additionally the interface for querying service interdependency should be clearly defined. This will allow for almost instant CI/CD flow creation for each new service and will reduce the load on the CI/CD team. Ultimately this should allow developers to plug-n-play new services into the CI/CD system in a fully self-service mode.

Otomato Software ltd. 2016 All Rights Reserved.®


  • 0

DevOps Flow Metrics – http://devopsflowmetrics.org

Category : Tools

grpah

DevOps transformation goals can be defined as:

  • Heightened Release Agility
  • Improved Software Quality

Or simply:

Delivering Better Software Faster

Therefore measurable DevOps success criteria would be:

  • Being able to release versions faster and more often.
  • Having less defects and failures.

Measurement is one of the cornerstones of DevOps. But how do we measure flow?

In order to track the flow (the amount of change getting pushed through our pipeline in a given unit of time) we’ve developed the 12 DevOps Flow Metrics.

They are based on our industry experience and ideas from other DevOps practitioners and are a result of 10 years of implementing DevOps and CI/CD in large organisations.

The metrics were initially publicly presented by Anton Weiss at a DevOpsDays TLV 2016 ignite talk. The talk got a lot of people interested and that’s why we decided to share the metrics with the community.

We’ve created a github pages based minisite where everyone can learn about the metrics, download the presentation and submit comments and pull requests.

Looking forward to your feedback!

Get the metrics here : http://devopsflowmetrics.org

 


  • 0

  • 0

Jenkins and the Future of Software Delivery

Are you optimistic when you look into the future and try to see what it brings? Do you believe in robot apocalypse or the utopia of singularity? Do you think the world will change to the better or to the worse? Or are you just too busy fixing bugs in production and making sure all your systems are running smoothly?

Ant Weiss of Otomato describing the bright future of software delivery at Jenkins User Conference Israel 2016.


  • 0

How OpenStack is Built – the Video now Online

Watch Ant Weiss of Otomato provide an overview of the OpenStack CI – probably one of the most advanced Jenkins-based CI infrastructures in the world.


  • 0

The Number One Symptom Of not Actually ‘Doing Devops’

Category : Tools

apply-now-job-hiring-help-ss-1920So I recently talked to a release engineering team leader at a very well-known american software+hardware company located in California.
They contacted me looking for top-notch build infrastructure engineers and we spent a very interesting hour discussing their technological stack and people processes. On the surface – they are moving in the right direction – automating all the things, using Chef for config management, codifying the infrastructure, using Artifactory for binaries… But one thing struck me in our conversation. The team leader (clearly a very smart and talented guy) said : “I’ve been trying to hire for two years. Good professionals are so hard to find…”
I agreed at first – we all know the market is starving for technical talent.
But reflecting on our conversation afterwards I realized that he in fact showed me the number one symptom of how far their team is from ‘doing devops’.

Yes – hiring is challenging. I’ve discussed this before. It may take some time to find the person whose mindset, values and the overall vibe suite your expectations. Even more so – if you’re low on budget, as the market has become highly competitive. But – for this specific team leader the budget was not an issue. He was failing to hire because of his organisation’s unwillingness to invest in mentoring and development.

He was looking for the ready-made fullstack ninja with a very specific skillset. there is no other way to explain his 2 year long quest.

Can you see how this totally opposes the ever important Devops value of Sharing? So you’ve built a few things to be proud of, you’re technologically savvy, you’re on the bleeding edge. Now is the time to share your knowledge! Now is the time to go hire bright novices. They are out there, they are hungry to learn from your experience and by sharing with them you will build the real devops on your team.

So if you’re failing to hire for more than a couple of months – don’t blame the market. Don’t complain about lousy candidates. Go revise your hiring process and more importantly – the way your team works. You may have all the technology, but it looks like your culture is broken. And with broken culture – even if you eventually succeed in hiring , you’ll have hard time reaping the true benefits of the Devops way – agility, quality, motivation and trust.

And may the devops be with you!


  • 0

Custom deploy process at Utab

Hi! I’m Ilya Sher.

This guest post will describe the deploy process at Utab, one of my clients.

Background – system architecture summary

Utab uses the services architecture. Some services are written in Java, others in NodeJS. Each server has either one Java application or one NodeJS application. The production environment uses (with the exception of load balancing configuration) the immutable server approach.

Requirements for the deploy process

The client specified the following requirements:

  1. Support for staging and production environments
  2. Manually triggered deploy for each of the environments
  3. Health check before adding a server to load balancing in production environment
  4. Option to easily and quickly rollback to previous version in production environment
  5. Simple custom tools are to be used ( no Chef/Puppet/… )

 

Solution background

The Utab’s deploy and other scripts were made specifically for the client. Such custom scripts are usually simpler than any ready-made solution. It means they break less and are easier to modify and operate. No workarounds are required in cases when a chosen framework is not flexible enough.

Also note that custom solution solves the whole problem, not just parts of the problem as it’s often the case with ready-made solutions. For example the scripts also handle the following aspects: firewall rules configuration, ec2 volumes attachment, ec2 instances and images tagging, running tests before adding to load balancer. When all of this is solved by a single set of custom scripts and not by using several ready-made solutions plus your own scripts the result looks much better and more consistent.

We estimate that such custom solution has lower TCO despite the development. Also note that in case of Utab, the development effort for the framework scripts was not big: about ten working days upfront and another few days over three years. I attribute this at least partly to sound and simple software and system architecture.

 

Note that the opinion in the summary above has many opponents. My experience shows that using ready-made solutions for the tasks described above lead to higher TCO. This mostly happens because of complexity of such solutions and other reasons that I’ll be discussing in another post.

The build

  1. A developer pushes a commit to Github
  2. Github hook triggers Jenkins
  3. Jenkins does the build and runs the tests: Mocha unit tests for NodeJS or JUnit tests for Java
  4. A small script embeds git branch name, git commit and Jenkins build number in pom.xml, in the description field.
  5. Jenkins places the resulting artifact in the Apache Archiva repository (only if #3 succeeds)

 

What you see above is common practice and you can find a lot about these steps on the internet except for number 4, which was our customization.

Note that while the repository is Java oriented we use it for both Java and NodeJS artifacts for uniformity. We use mvn deploy:deploy-file to upload the NodeJS artifacts.

For NodeJS artifacts, static files that go to a CDN are included in the artifact. Java services are not exposed directly to the browser so Java artifacts do not contain any static files for the CDN.

 

Custom scripts involved in the deployment process

Scripts that use the AWS API are in Python. Rest of them are in bash. I would be happy to use one language but bash is not convenient for using AWS API and Python is not as good as bash for systems tasks such as running programs or manipulating files.

I do hope to solve this situation by developing a new language (and a shell), NGS, which should be convenient for the system tasks, which today include talking to APIs, not just working with files and running processes.

create-image.py

  1. Creates EC2 instance (optional step)
  2. Runs upload-and-run.sh (see the description below).
  3. Runs configure-server.sh to configure the server, including the application (optional step)
  4. Runs tests (optional step)
  5. Creates an AMI

 

upload-and-run.sh

Gets destination IP or hostname and the role to deploy there.

  1. Pulls the required artifacts from the repository. Default is the latest build available in the repository. A git branch or a build number can be specified to pick another build. If a branch is given the latest build on that branch is used. The information embedded in pom.xml  .
  2. Runs all relevant “PRE” hook files (which for example upload static files to S3)
  3. Packages all required scripts and artifacts into a .tar.gz file
  4. Uploads the .tar.gz file
  5. Unpacks it to a temporary directory
  6. Runs setup.sh (a per role file: website/setup.sh, player/setup.sh, etc) which controls the whole installation.

 

deploy.py

  1. Starts EC2 instance from one of the AMIs created by the create-image.py script
  2. Runs applications tests
  3. Updates the relevant load balancers switching from old-version-servers to new-version-servers.

 

The Python scripts, when successful, notify the developers using dedicated Twitter account. This is as simple as:

   api = twitter.Api(c['consumer_key'], c['consumer_secret'], c['access_token_key'], c['access_token_secret'])
   return api.PostUpdate(s)

Pulling artifacts from the repository before deploying

I would like to elaborate about pulling the required artifacts from the repository. The artifacts are pulled from the repository to the machine where the deploy script runs – the management machine (yet another server in the cloud, near the repository and rest of the servers). I have not seen this implemented anywhere else.

Note that pulling artifacts to the machine where the script runs works fine when a management machine is used. You should not run the deployment script from your office as downloading and especially uploading artifacts would take some time. This limitation is a result of a trade off. The gained conceptual simplicity outweighs the minuses. When the setup scripts are uploaded to the destination server, the artifacts are uploaded [with them] so the destination application servers never need to talk to the repository and hence no security issues to be handled for example.

 

Deploying to staging environment

  1. Artifact is ready (see “The build” section above)
  2. One of the developers runs create-image.py with flags that tell create-image.py not to create a new instance and not to creat AMI from instance. This limits create-image.py to the depoy process only: run upload-and-run.sh  .

    Since “ENV” environment variable is also passed, the configuration step is also run ( configure-server.sh )

    Since there are switches and environment variables that create-image.py needs that none of us would like to remember there are several wrapper scripts named deploy-ROLE-to-staging.sh

 

Deploying to production environment

It’s a responsibility of developers to make sure that the build being packaged at this step is one of the builds that were tested in the staging environment.

  1. Artifact is ready (see “The build” section above)
  2. One of the developers runs create-image.py and the script creates an AMI
    The solution to (not) remembering all the right switches and environment variables is documentation in the markdown formatted readme file at the top level of the repository.
  3. One of the people that are authorized to deploy to production runs deploy.py

 

Rollback to previous version

This can be done in one of the following ways:

 

New deploy

Run the deploy.py script giving an older version as an argument

 

Manual quick fix

When old servers are removed from the load balancing, they are not immediately terminated. Their termination is manual, after the “no rollback” decision. The servers that rotated out of the load balancing are now tagged with “env” tag value “removed-from:prod”.

  1. Change the “env” tag of the new servers to “removed-from:prod” or anything else that is not “prod”
  2. Change “env” tag on the old servers to “prod”
  3. Run the load balancers configuration script. The arguments for this script are the environment name and the role of the server. The script updates all the relevant load balancers that point to servers with the given role.

Rollback script

We never finished it as rollbacks were very rare occasion and the other two methods work fine.

Instances tagging and naming

Instances naming: role/build number

For instances with several artifacts that have build number: role/B1-B2-B3-…, where B1 is primary artifacts build number and others follow in order of decreasing importance.

The status tag is updated during script runs so when deploying to production it can for example look like: “Configuring” or “Self test attempt 3/18” or “Self tests failed” (rare occasion!) or “Configuring load balancing”.

 

Summary

This post described one of the possible solutions. I think the solution described above is one of the best possible solutions for Utab. This does not mean it’s a good solution for any particular other company. Use your best judgement when adopting similar solution or any parts of it.

 

You should always assume that a solution you are looking at is not a good fit for your problem/situation and then go on and prove that it is, considering possible alternatives. Such approach helps avoiding the confirmation bias.


  • 2

git submodules vs. google’s repo tool

Category : Tools

git_logo

I was recently asked by a customer to outline the pros and cons of using git submodules vs. google repo tool to manage multi-repository integrations in git.

There are a lot of articles on the internet bashing each of the tools, but in our opinion – most of it comes from misunderstanding the tool’s design or trying to apply it in an unappropriate context.

This post summarizes the general rules of thumb we at Otomato follow when choosing a solution for this admittedly non-trivial situation.

First of all – whenever possible – we recommend integrating your components on binary package level rather than compiling everything from source each time. I.e. : packaging components to jars, npms, eggs, rpms or docker images, uploading to a binary repo and pulling in as versioned dependencies during the build. 

Still – sometimes this is not an optimal solution, especially if you do a lot of feature branch development (which in itself is an anti-pattern in classical Continuous Delivery approach – see here for example).

For these cases we stick to the following guidelines.

Git Submodules :

Pros:

  1. An integrated solution, part of git since v1.5
  2. Deterministic relationship definition (parent project always points to a specific commit in submodule)
  3. Integration points are recorded in parent repo.
  4. Easy to recreate historical configurations.
  5. Total separation of lifecycles between the parent and the submodules.
  6. Supported by jenkins git plugin.

Cons:

  1. Management overhead. (Need separate clones to introduce changes in submodules)
  2. Developers get confused if they don’t understand the inner mechanics.
  3. Need for additional commands (‘clone –recursive’ and ‘submodule update’)
  4. External tools support is not perfect (bitbucket, sourcetree, ide plugins)

—————————————————————-

Google repo:

Pros:

  1. Tracking synchronized development effort is easier.
  2. Gerrit integration (?)
  3. A separate jenkins plugin.

Cons:

  1. An external obscure mechanism
  2. Requires an extra repository for management.
  3. Non-deterministic relationship definition (each repo version can be defined as a floating head)
  4. Hard to reconstruct old versions.
  5. No support in major repo managers (bitbucket, gitlab) or gui tools.

====================================

 

In general : Whenever we want to integrate separate decoupled components with distinct lifecycles – we recommend submodules over repo, but their implementation must come with proper education regarding the special workflow they require. In the long run it pays off – as integration points can be managed in deterministic manner and with knowledge comes the certainty in the tool.

If you find your components are too tightly coupled or you you’re in need of continuous intensive development occurring concurrently in multiple repos you should probably use git subtrees or just spare yourself the headache and drop everything into one big monorepo. (This depends, of course, on how big your codebase is)

To read more about git subtree – see here.

The important thing to understand is that software integration is never totally painless and there is no perfect cure for the pain. Choose the solution that makes your life easier and assume the responsibilty of learning the accompanying workflow. As they say : “it’s not the tool – it’s how you use it.”

I’ll be happy to hear what you think, as this is a controversial issue with many different opinions flying around on the internet.

Keep delivering!

 

And  – if you’re looking for some git consulting – drop us a note and we’ll be happy to help.