Dynamically spinning up Jenkins slaves on Docker clusters

  • 0

Dynamically spinning up Jenkins slaves on Docker clusters


Being able to dynamically spin up slave containers is great. But if we want to support significant build volumes we need more than a few Docker hosts. Defining a separate Docker cloud instance for each new host is definitely not something we want to do – especially as we’d need to redefine the slave templates for each new host. A much nicer solution is combining our Docker hosts into a cluster managed by a so-called container orchestrator (or scheduler) and then define that whole cluster as one cloud instance in Jenkins.
This way we can easily expand the cluster by adding new nodes into it without needing to update anything in Jenkins configuration.

There are 4 leading container orchestration platforms today and they are:

Kubernetes (open-sourced and maintained by Google)

Docker Swarm (from Docker Inc. – the company behind Docker)

Marathon (a part of the Mesos project)

Nomad (from Hashicorp)

A container orchestrator (or scheduler) is a software tool for the deployment and management of OS containers across a cluster of computers (physical or virtual). Besides running and auditing the containers, orchestrators provide such features as software-defined network routing, service discovery and load-balancing, secret management and more.

There are dedicated Jenkins plugins for Kubernetes and Nomad using the Cloud extension point. Which means they both provide the same ability of spinning up slaves on demand. But instead of doing it on a single Docker host they talk to the Kubernetes or Nomad master API respectively in order to provision slave containers somewhere in the cluster.


Nomad plugin was originally developed by Ivo Verberk and further enhanced by yours truly while doing an exploratory project for Taboola. A detailed post describing our experience will be up on Taboola engineering blog sometime next month.
Describing Nomad usage is out of the scope of this book, but in general – exactly as the YAD plugin – it allows one to define a Nomad cloud and a number of slave templates. You can also define the resource requirements for each template so Nomad will only send your slaves to nodes that can provide the necessary amount of resources.
Currently there are no dedicated Pipeline support features in the Nomad plugin.
Here’s a screenshot of Nomad slave template configuration:Screen Shot 2017-07-11 at 12.24.48 AM


The Kubernetes Plugin was developed and is still being maintained by Carlos Sanchez. The special thing about Kubernetes is that its basic deployment unit is a kubernetes pod which could consist of one or more containers. So here you get to define pod templates. Each pod template can hold multiple container templates. This is definitely great when we want predefined testing resources to be provisioned in a Kubernetes cluster as a part of the build.

The Kubernetes plugin has strong support for Jenkins pipelines with things like this available:

podTemplate(label: 'mypod', containers: [
 containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat'),
 containerTemplate(name: 'golang', image: 'golang:1.8.0', ttyEnabled: true, command: 'cat')]) 
  node('mypod') {
    stage('Checkout JAVA') {
    git 'https://github.com/jenkinsci/kubernetes-plugin.git'
    container('maven') {
      stage('Build Maven') {
        sh 'mvn -B clean install'
 stage('Checkout Go') {
 git url: 'https://github.com/hashicorp/terraform.git'
 container('golang') {
 stage('Build Go) {
 sh """
 mkdir -p /go/src/github.com/hashicorp
 ln -s `pwd` /go/src/github.com/hashicorp/terraform
 cd /go/src/github.com/hashicorp/terraform && make core-dev

There’s detailed documentation in plugin’s README.md on github.


There is a Jenkins marathon plugin but instead of spinning up build slaves it simply provides support for deploying applications to a Marathon-managed cluster
It requires a Marathon .json file to be present in the project workspace.
There’s also support for Pipeline code. Here’s an example of its usage:

   url: 'http://otomato-marathon',
   id: 'otoid',
   docker: otomato/oto-trigger')


Docker Swarm

I used to think there was no dedicated plugin for Swarm but then I found this. As declared in the README.md this plugin doesn’t use the Jenkins cloud API. Even though it does connect into it for label-based slave startup. This non-standard approach is probably the reason why the plugin isn’t hosted on the official Jenkins plugin repository.
The last commit on the github repository dates back 9 months ago, so it may also be outdated – as Docker and Swarm are changing all the time and so does the API.

Learn More:

This post is a chapter from the ebook ‘Docker, Jenkins, Docker’ which we recently released at Jenkins User Conference TLV 2017.  Follow this link to download the full ebook: http://otomato.link/go/docker-jenkins-docker-download-the-ebook-2/

  • 0

Jenkins and the Future of Software Delivery

Are you optimistic when you look into the future and try to see what it brings? Do you believe in robot apocalypse or the utopia of singularity? Do you think the world will change to the better or to the worse? Or are you just too busy fixing bugs in production and making sure all your systems are running smoothly?

Ant Weiss of Otomato describing the bright future of software delivery at Jenkins User Conference Israel 2016.

  • 0

How OpenStack is Built – the Video now Online

Watch Ant Weiss of Otomato provide an overview of the OpenStack CI – probably one of the most advanced Jenkins-based CI infrastructures in the world.

  • -

Openstack CI infrastructure Overview

Openstack is one of the largest OSS projects today with hundreds of commits flowing in daily. This high rate of change requires an advanced CI infrastructure. The purpose of the talk is to provide an overview of this infrastructure, explaining the role of each tool and the pipelines along which changes have to travel before they find their way into the approved Openstack codebase.
Talk delivered by Anton Weiss at Openstack Day Israel 2016 : http://www.openstack-israel.org/#!agenda/cjg9

  • -

User-friendly monitoring for your Jenkins server

We all love Jenkins. It’s flexible, scalable, has unbelievable community support (more than 1010 plugins available) and is very easy to get started with. No wonder Jenkins is the CI/CD server used in at least 70% of IT and R&D organisations around the globe.

Once you start using Jenkins you quickly get hooked. It’s so easy to automate any development or system task, add a button and let your users push it whenever needed.

And your Jenkins instance begins to grow. Very soon you have views, nested views, pipeline views, dozens of plugins for every little thing, jobs for dev, jobs for QA, jobs for project managers, an ever growing bunch of slave nodes, you name it.Everyone in the organisation is using Jenkins, everyone falls in love with it as much as you originally did.

But then the day comes and your Jenkins that was so lively and fast when you installed it and ran your first job suddenly starts showing signs of fatigue. The UI takes time to load, jobs get stuck for unclear reasons, plugins conflict with one another, nodes get lost, disks fill up…

But users still expect the same level of service they grew accustomed to. You – the Jenkins admin – start getting emails and phone calls – sometimes in very uncomfortable hours.

And that’s when the need for monitoring becomes evident. You realise that your users would be happier and your own life easier if you could get alerts before things get actually broken. Alerts about disk space, memory consumption, lost build slaves and application errors.

But we all know that #monitoring sucks. Moreover – if you’re a Jenkins admin in an enterprise – chances are that you don’t have the access to existing monitoring infrastructure. Instead of being able to configure things the way you want them you’ll have to explain your needs to sysadmins and they configure it for you. And you don’t always know all your needs at once. So it can take a lot of time and pain to get that Jenkins instance monitoring right.

But what if I told you that there is a tool that is laughably easy to configure, can run on your commodity Windows machine and is already configured to check all the basic metrics of your Jenkins server. Doesn’t this sound like any enterprise Jenkins admin’s wet dream?

I think it does. And the great thing – now there is such a tool on the market!

A few days ago I had the pleasure to meet with Tamir Gefen and David Cohen of ALMToolBox. What they presented to me was the new version of ALM Vitality tool with added Jenkins monitoring support.

ALM Vitality was originally built for application-aware monitoring of ClearCase and ClearQuest and it seems to be doing a great job for those two as well. God knows I’ve spent a couple of years managing these monsters and their performance bottleneck analysis and resolution can be a major pain in the butt.

But for this session we were specifically looking at ALM Vitality for Jenkins and it really seems a nice little tool that hits a sweet spot for hassle-free Jenkins monitoring.

It is very easy to configure. Basically all it needs is an access to your Jenkins server and the credentials to the application itself. After that you’re up and running. Your Jenkins is monitored for disk space, application availability and stuck jobs. More checks (like lost slaves and memory consumption) are coming soon according to Tamir. Beside having access to your system state through a clean and simple web UI (how about putting a screen on the wall for all to see?)  one can also configure email notifications for when things go wrong.

All in all, I certainly believe ALM Vitality is a great addition to a Jenkins admin toolbox.

You can watch a webcast on ALMToolox website for more details.

And if you decide to try the tool out  – tell them I sent you 😉

  • 2

PMD formatter for TSLint

One of the development teams we work with is developing in TypeScript and using tslint for code analysis.

We’ve established a CD process for them which involves compiling typsecript to javascript, minifying the resulting code and deploying it to a staging server. Naturally they wanted to run tslint as a part of this flow in order to get notified on any new warnings a commit might have introduced.

When running static code analysis in CD flow one wants some statistics on analysis results and a nice graphic representation of the results. All these are already implemented in the Jenkins PMD plug-in.  (We’ve already used the PMD plugin successfully for representing the OCLint analysis results we run for an iOS project we support.)

So the only thing we needed was formatting the tslint output according to PMD format, which is in fact a very simple xml markup.

Writing a formatter was really easy following the guidelines on tslint README.

And it was already integrated into tslint trunk today (this commit) , so now you can use it for your own project – just use ‘-t pmd’ in oclint CLI.

Continue delivering!





  • 2

Use gitlist as git repository browser in Jenkins

Jenkins git-scm plugin provides support for various git repo browser applications, but the wonderful gitlist isn’t one of them…
Still I found you can fool Jenkins into using gitlist as your repository browser.
(We’ve been using gitlist on one of the ALM environments I manage and gitLab on another one. At some stage I noticed that gitLab‘s commit url is build exactly the same as a commit url in gitlist  : <gitlist_url>/<repo-name>.git/commit/<commit_hash> vs. <gitlab_url>/<repo_name(without .git extension)>/commit/<commit_hash>.
Actually what jenkins git plugin does when constructing the repository browser link for gitlab is add the ‘commit/<commit_hash>’ on top of gitlab project url. And it works perfectly fine for gitlist too!)

To use gitlist as your git repo browser in Jenkins:

in git plugin define the following:

Repository browser : gitlab

URL: <your gitlist url (repo name with .git)> (eg: http://gitserver/gitlist/myRepo.git)

Version: 5.4

Now for every commit in the changes list on build page you’ll get a ‘gitlab’ link which will lead you to the commit page on gitlist.

I relaize it is quite easy to add this support to the plugin itself and get the correctly named ‘gitlist’ link, but until I or someone else find time for that – this is a nice workaround.

and here’s a groovy script to set this for all your git jobs:

[sourcecode language=”groovy”]

import hudson.model.*;
import hudson.plugins.git.*
import hudson.plugins.git.browser.*

//replace with your own values
def view = Hudson.instance.getView("myViewName")
def myRepoUrl = "http://gitserver/gitlist/myRepo.git"

for(item in view.getItems())

def SCM = item.getScm()
if(SCM instanceof GitSCM ) {
println (item.getName());
def browser = SCM.getBrowser();
if (browser){
println browser.getUrl()
browser = new GitLab( myRepoUrl, "5.4");
SCM.browser = browser