Category Archives: Tools

  • 0

git submodules vs. google’s repo tool

Category : Tools


I was recently asked by a customer to outline the pros and cons of using git submodules vs. google repo tool to manage multi-repository integrations in git.

There are a lot of articles on the internet bashing each of the tools, but in our opinion – most of it comes from misunderstanding the tool’s design or trying to apply it in an unappropriate context.

This post summarizes the general rules of thumb we at Otomato follow when choosing a solution for this admittedly non-trivial situation.

First of all – whenever possible – we recommend integrating your components on binary package level rather than compiling everything from source each time. I.e. : packaging components to jars, npms, eggs, rpms or docker images, uploading to a binary repo and pulling in as versioned dependencies during the build. 

Still – sometimes this is not an optimal solution, especially if you do a lot of feature branch development (which in itself is an anti-pattern in classical Continuous Delivery approach – see here for example).

For these cases we stick to the following guidelines.

Git Submodules :


  1. An integrated solution, part of git since v1.5
  2. Deterministic relationship definition (parent project always points to a specific commit in submodule)
  3. Integration points are recorded in parent repo.
  4. Easy to recreate historical configurations.
  5. Total separation of lifecycles between the parent and the submodules.
  6. Supported by jenkins git plugin.


  1. Management overhead. (Need separate clones to introduce changes in submodules)
  2. Developers get confused if they don’t understand the inner mechanics.
  3. Need for additional commands (‘clone –recursive’ and ‘submodule update’)
  4. External tools support is not perfect (bitbucket, sourcetree, ide plugins)


Google repo:


  1. Tracking synchronized development effort is easier.
  2. Gerrit integration (?)
  3. A separate jenkins plugin.


  1. An external obscure mechanism
  2. Requires an extra repository for management.
  3. Non-deterministic relationship definition (each repo version can be defined as a floating head)
  4. Hard to reconstruct old versions.
  5. No support in major repo managers (bitbucket, gitlab) or gui tools.



In general : Whenever we want to integrate separate decoupled components with distinct lifecycles – we recommend submodules over repo, but their implementation must come with proper education regarding the special workflow they require. In the long run it pays off – as integration points can be managed in deterministic manner and with knowledge comes the certainty in the tool.

If you find your components are too tightly coupled or you you’re in need of continuous intensive development occurring concurrently in multiple repos you should probably use git subtrees or just spare yourself the headache and drop everything into one big monorepo. (This depends, of course, on how big your codebase is)

To read more about git subtree – see here.

The important thing to understand is that software integration is never totally painless and there is no perfect cure for the pain. Choose the solution that makes your life easier and assume the responsibilty of learning the accompanying workflow. As they say : “it’s not the tool – it’s how you use it.”

I’ll be happy to hear what you think, as this is a controversial issue with many different opinions flying around on the internet.

Keep delivering!


And  – if you’re looking for some git consulting – drop us a note and we’ll be happy to help.

  • 0

Jenkins 2.0: With Jenkins Creator – Kohsuke Kawaguchi (KK)

Category : Tools

Hi all!
We really hope you’ve already registered to the Jenkins User Conference Israel which was SOLD OUT yesterday.
But even if you haven’t – there’s still a chance to hang out with Kohsuke Kawaguchi – the father of Jenkins himself – and other Jenkins fans at a meetup the good people at JFrog are organising the day after the conference.
Here is the link:
Right now there are still 22 slots available.
Go grab yours.

Keep delivering!

  • 0

Discount tickets to Jenkins User Conference Tel Aviv

Category : Tools

We are happy to announce that @antweiss will be speaking about the future of software delivery at JUC Israel 2016

We have a couple of 100 NIS discount tickets to the conference and ticket sale is closing tomorrow!

The first two folks to comment on this post will get the discount code.

Type them comments!!!

  • -

Microservice Development Toolkits part 1 – Otto


Otomato is happy to announce we’re collaborating with Codefresh on their great continuous delivery for docker containers solution. (If you’re using docker in your development/production or only thinking of doing it – go check out ) Our collaboration is mostly focused on industry analysis and docker development ecosystem research with the goal of identifying pain points and providing effective solutions. With quite a bit of evangelism on top. The linked post is the first fruit of that. ottologo

Microservice Development Workflow with Otto


  • -

Have you heard of Wajig?

We love productivity tools of all kind and believe that software should be making our lives easier. For some reason many software products we have to work with don’t really feel like their creators agree. They may have great architecture, reliability and functionality but it looks like usability is snapped on as an afterthought.
One example of this is apt – the Advanced Packaging Tool used on Debian/Ubuntu Linux. While a great tool in itself, for some reason it provides a number of different user interfaces for different purposes.

To install/remove/upgrade/download a package there's the apt-get tool:

apt-get is a simple command line interface for downloading and
installing packages. The most frequently used commands are update
and install.

   update - Retrieve new lists of packages
   upgrade - Perform an upgrade
   install - Install new packages (pkg is libc6 not libc6.deb)
   remove - Remove packages
   autoremove - Remove automatically all unused packages
   purge - Remove packages and config files
   source - Download source archives
   build-dep - Configure build-dependencies for source packages
   dist-upgrade - Distribution upgrade, see apt-get(8)
   dselect-upgrade - Follow dselect selections
   clean - Erase downloaded archive files
   autoclean - Erase old downloaded archive files
   check - Verify that there are no broken dependencies
   changelog - Download and display the changelog for the given package
   download - Download the binary package into the current directory


But then if you want to search or query for packages you have to remember that a different utility is used – apt-cache:

apt-cache is a low-level tool used to query information
from APT's binary cache files

   gencaches - Build both the package and source cache
   showpkg - Show some general information for a single package
   showsrc - Show source records
   stats - Show some basic statistics
   dump - Show the entire file in a terse form
   dumpavail - Print an available file to stdout
   unmet - Show unmet dependencies
   search - Search the package list for a regex pattern
   show - Show a readable record for the package
   depends - Show raw dependency information for a package
   rdepends - Show reverse dependency information for a package
   pkgnames - List the names of all packages in the system
   dotty - Generate package graphs for GraphViz
   xvcg - Generate package graphs for xvcg
   policy - Show policy settings

And what if we want to list all the files installed by a specific package? Well we’re supposed to remember that there’s a special utility for that – apt-file. And it’s not installed by default…

Figures we weren’t the only ones puzzled by this multitude of interfaces. Wajig is targeted at exactly that – providing a unified interface to all package-related functionality on Debian/Ubuntu. Or as stated on the site:
“Wajig is a simplifed and more unified command-line interface for package management. It adds a more intuitive quality to the user interface.  Wajig commands are entered as the first argument to wajig. For example: “wajig install gnome”. Written in Python, Wajig uses traditional Debian administration and user tools including apt-get, dpkg, apt-cache, wget, and others. It is intended to unify and simplify common administrative tasks. … You will also love the fact that it logs what you do so you have a trail of bread crumbs to back track with if you install something that breaks things.”

Another nice thing – wajig knows when root privileges are needed and takes care of that, so we don’t need to rerun a command each time we forget to prepend it with sudo.

We still need to play with wajig a bit more to decide if it performs on all its promises but it definitely is going into a right direction.

Wishing you all user-friendly software and a good week.

  • -

Jenkins User Conference Israel 2016 Announced

Category : Tools

Reposted from the official event page:

Jenkins User Conference hits Israel fifth year in a row!

Come learn how to optimize Jenkins across the software delivery process!

With more than 100,000 active installations and more than 1,000 plugins Jenkins is no doubt a leader in the CI and CD domain. Our 2015 Jenkins User Conference in Israel drew more than 700 developers, and was so successful that this year we know it will be a blast!

JUC IL 2015

The Jenkins User Conference focuses on the use of Jenkins for continuous integration (CI) and continuous delivery (CD) as the fundamental best practice for enterprise software delivery. Our presenters are experienced Jenkins developers, build managers, QA, DevOps practitioners, IT managers/executives, architects and IT operations who are luminaries within the Jenkins community. They represent the many organizations around the world that are leveraging the use of Jenkins within the software delivery lifecycle.

We welcome you and other leading Jenkins developers, QA, DevOps and operations personnel to the Jenkins User Conference World Tour. As the organizing sponsor of the Jenkins User Conferences, CloudBees has helped the community grow the Jenkins User Conferences worldwide over the last four years. In 2015, the community saw a 70% increase in attendance over 2014.

In 2016, the World Tour will bring together the full strength of the Jenkins community—now over 100,000 installations strong—and the ever-expanding Jenkins partner ecosystem, allowing attendees to learn, explore, network face-to-face and to shape the next evolution of Jenkins development. Kohsuke Kawaguchi will kick off the event with a keynote address and lead us into the conference. Come get the knowledge you need to make your current and future Jenkins projects a success.

Call For Papers is Now Open>

  • -

DevConTlv update

Category : Tools

Just went to DevConTLV pre-conf speaker dinner. Had a lot of fun talking about tech, life and business with some of the finest minds in the industry.

Now – it figures I was wrong regarding the conference format. This one isn’t going to have beers and live music… But! It will have great speakers who are going to rock the stage with talks on cutting edge software development tools and metodologies. On my part I will do my best to keep the audience entertained. So if you want to be a ninja – make sure to attend.

  • -

Service-Oriented Collaboration

Category : Tools


This post is yet another take on how we should be creating software together. I’m now working on a book named “Coding Together” that will be reviewing all the challenges of collaborative software delivery and the ways of overcoming them for maximum creativity and efficiency. As I’m gathering materials – my understanding not only deepens but shifts – complex systems viewed holistically definitely cannot be analyzed as a sum of their parts.
Two years ago I gave a few talks and wrote about the importance of enabling self-service principle in a smart and secure way for smooth and effective value delivery in software engineering organisations. I still hold a strong belief in self-service builds, tests, deployments and infrastructure provisioning. All these can greatly enhance the rate of innovation and the quality of produced software.
If done correctly, that is…

But it can be quite disastrous when this self-service concept implementation isn’t designed and delivered with due attention. What’s more annoying and demotivating than being told that you can build, test and deploy with a click of a button but ending up with a failed process and a bunch of cryptic error messages?!
And then opening a ticket to the suporting team and waiting for hours because “they are busy developing the new self-service infrastructure”…

I’ve seen this happening too often in real life and that led me to understand that self-service is really worth nothing when the other most important ingredient is missing. And that is – human service consciousness. Not serving oneself but serving others should be the focus.

That’s easy to say, but how do you practically implement this? How should we structure our software development organization around service consciousness and how is this any different than what we used to have before continuious delivery and automation became common knowledge?

Microservice architectures are becoming the de-facto standard for building modern highly scalable and reliable software systems. In line with Conway’s law – small service teams based on consumer-driven contracts should become the foundation of a highly scalable, flexible and agile IT organisation. And it’s really less about the team size but more about being focused on serving the members of other teams – the consumers of your team’s api, the users of your self-service automation, the implementers of your feature requests or the developers you’re assigning bugs to. It’s about building a decentralized Team of Teams  where each unit takes pride in the level of service it provides and not in its special status in organisational hierarchy.

Advanced communication is what allows us humans to collaborate on massive scale. But communication can also ruin collaboration if what we transmit is a negative, elitist or egocentric attitude.

As Sabine Bendixen rightfully writes – a highly effective agile organization starts with effective communication.

Communication is always the first thing we at Otomato look at when performing a software delivery assessment for our clients. How are changes getting communicated? How is knowledge managed within the organization? How are decisions made? How do the members of different teams see themselves and their role in the value delivery stream?
Whenever there’s hostility, resentment and lack of motivation or transparency – we know we struck a bottleneck. And this is a great place to start. Start building trust and visibility, reviewing and streamlining the processes. But most important – start building collaboration around providing important services among teams. Around treating all the internal interactions as interactions with customers in which customers invariably come first and their experience is the utmost measure of our performance.

I like to call this ‘service-oriented collaboration’ and that’s the only type of collaboration that’s truly sustainable, adaptable and robust. This is the kind of collaboration that will allow your business to quickly change course when needed and keep engineers creative and motivated. And once you have this – self-service infrastructure will take you to the moon!

Now did you notice how this whole post didn’t mention DevOps even once?
Watch our follow-up posts to learn why.

Happy delivering!

  • -

Cloud-aware, provider-agnostic monitoring

Category : Tools

I’ve never had to deal so much with monitoring. I’ve established a few Nagios instances in earlier days, I’ve used Amazon CloudWatch, Pingdom and New Relic lately for cloud setups, but I don’t consider myself a monitoring expert. At Otomato we are currently mainly focused on software delivery processes, on how to get those bits from dev machines and into production in the most effective and agile manner without compromising quality. Monitoring, although an important part of running software and assuring successful value delivery (should I say an important part of DevOps?) , has been largely out of our scope.

But now we are working on a  new project where we had to deal with the full-full cycle – building, deploying and running. Moreover – the requirement was for cloud-provider-agnostic solutions as this is going to be a multicloud setup.

And with that came the most exciting part of any project – the research!

I didn’t want to go back to Nagios as my previous encounters with it left a bad aftertaste. Cumbersome setup, dated WebUI and no real support for ephemeral cloud servers. So I started reading up to understand  what are the newer tools on the market and if #monitoringsucks less now than it used to.

Here are the links to a few articles that really helped me in my research:

A series of posts on Florin’s blog:

All James Turnbull has to say about monitoring and his experience with Riemann:

This comparison of Nagios, Sensu and Icinga2 :

And this great post on Sensu

And there are more…

To sum things up – I really liked what Riemann has to offer, and I’d like to research it more for future project but currently we’ve decided to go with Sensu. Reasons: scalability, cloud-awareness and yes – a good-looking WebUI.

Needless to say – we’ve only touched the tip of the iceberg here – modern monitoring is a world of its own. Metrics collection, event handling, threshold definition, log analysis, anomaly detection – you name it. We promise to revisit all these in more depth but until then – please comment here with links and insights of  your own.

Happy delivering!




  • -

DevConTLV is only 2 days away!!!

Category : Tools

If you’ve never been to DevConTLV – you are seriously missing out!!! Running under the banners of “Server Side Development and Rock’n’Roll” this is the grooviest software conference in Middle East, (or maybe even in the whole world?!)

Great speakers, free beer and live music – what more can a geek ask for?

The one taking place on the 22nd of March is probably the largest, the most bombastic DevConTLV ever. Featuring speakers from all parts of the planet, a Terraform training day and no other than Otomato’s own Ant Weiss as the DevOps track MC.

There are only late bird ticket left now, they are a bit pricey, but you can use the ‘ANTWEISS‘ coupon to get a very nice discount.

Hoping to see you all!

P.S: and this is a sweet memory from the last year DevConTLV : Sputnik Hi-FI feat. Ant Weiss – Software is Eating the World!