Docker: a jenkins-slave0 strategy

Well, yes I love Docker.

Thanks to Máté Varga and others, working at PKB, I delved into this wonderful concept.

First, I wanted to create a small architecture for continuous builds, that would run Maven on blank environments, thus not relying on any cached artifacts in the local repository. I expect the following benefits:

• Ensure that each developer is able to checkout any Maven project and build it independently on a fresh new environment.
• Detect errors in dependencies.
• Make those builds disposable: They’re here for error detection, but they may take a long time. Therefore, the actual development process mustn’t rely on them at all. For instance, at this stage they don’t push their results to the company’s Maven repository.

Using a complete VM for that kind of task, even with snapshots, is a little tedious, and uses a lot of empty space. Even with Dynamically allocated storages, just think of the RAM reserved for the VM.

But, maintaining a Dockerfile in the SCM, letting this file be equivalent to a Docker image, and running this image as a container somewhere, disposing of it right after each build, well, that sounds a lot better.

Internally, I named this image “jenkins-slave0”, because it’s being used as a Jenkins slave, with the minimum installed on it.

Workflow

So, here’s the workflow I wanted to set up:

1. Maintain some “jenkins-slave0” Dockerfile in some “my-sysadmin” project, in git or Mercurial.
2. From this Dockerfile, have a “jenkins-slave0” image built at the beginning of the workflow. Optional: Store this image in a custom Docker registry.
3. For each job, have Jenkins connect to Docker, start a new container.
4. Have Jenkins run the job on it. At the end of each job, Docker automatically removes the container.

Steps 1 and 2 would occur less than once a month.

Step 3 and 4 would occur many times a day.

Components

The Docker image contains:

• A SSH server (it’s for Jenkins to connect to it, remember?)
• Jenkins’ public key
• Git and Mercurial
• A JDK 1.7

It doesn’t contain Maven, since Jenkins downloads it anyway.

Here’s the corresponding Dockerfile:

FROM debian:7.8
MAINTAINER david.andriana@gmail.com

# apt-get
RUN apt-get update && apt-get upgrade
RUN apt-get install -y openssh-server
RUN apt-get install -y git mercurial
RUN apt-get install -y openjdk-7-jdk
RUN apt-get clean all

# SSH Server
RUN mkdir -p /var/run/sshd; \
    sed -i "/PermitRootLogin/s/yes/no/" /etc/ssh/sshd_config; \
    sed -i "/^PasswordAuthentication/d" /etc/ssh/sshd_config && \
    echo "PasswordAuthentication no" >> /etc/ssh/sshd_config
EXPOSE 22

# User: jenkins
RUN useradd -m -s /bin/bash jenkins
ADD [ "jenkins.pub", "/home/jenkins/" ]
RUN mkdir /home/jenkins/.ssh/; \
    cat /home/jenkins/jenkins.pub >> /home/jenkins/.ssh/authorized_keys; \
    chown -R jenkins:jenkins /home/jenkins/.ssh/; \
    chmod 700 /home/jenkins/.ssh/; \
    chmod 600 /home/jenkins/.ssh/authorized_keys

# Start the SSH Server
CMD [ "/usr/sbin/sshd", "-D" ]

The “jenkins.pub” file solely contains the public key I will use to connect via SSH. It’s of the form:

ssh-rsa AAAAB3NzaC1yc2...KP2YEnMub jenkins@host1

Jenkins uses the Docker Plugin. It allows to connect to the Docker server, fetch the image and run the container.

The Docker image creation is launched manually. I didn’t even write any Jenkins job for it. Here’s my “create_image.sh” script:

#!/bin/sh
test -f Dockerfile  || (echo "*** Dockerfile not found"  >&2 && exit 1)
test -f jenkins.pub || (echo "*** jenkins.pub not found" >&2 && exit 1)
docker build --no-cache -t avantage-compris/jenkins-slave0 .

Then, the image is available from the Docker environment.

If I want to push it to the Docker registry:

$ docker tag avantage-compris/jenkins-slave0 localhost:5000/avantage-compris/jenkins-slave0
$ docker push localhost:5000/avantage-compris/jenkins-slave0

And if I want to start a container from the image:

$ docker run -d -p 22000:22 avantage-compris/jenkins-slave0

If I want to connect to the container via SSH:

$ ssh -i /path/to/jenkins.key -p 22000 jenkins@localhost

Architecture #1

For some reasons, I needed the Docker environment to be completely remote.

Also, Docker itself runs in a VM, because:

• Maybe it’s not the best idea to mix responsibilities between the host, which runs services such as iptables, and other components, especially with Docker, which is very low-level and acts under sudo.

• Not sure if the host OS would accept Docker — For instance, in case of CentOS, only CentOS 7 would support it.

I wanted a Docker registry so I can sometimes prepare Docker images on another environment and upload them there.

The Docker registry contains the image for the Jenkins slave.

Note that the image is stored in the Docker environment anyway, so, for this article, the Docker registry is optional.

The Docker registry runs in a container.

The data for the Docker registry is on a shared folder on the host, outside the VM.

The Jenkins slave runs in a container.

The complete architecture is as follows.

Architecture #1: Nodes

Unfortunately, this architecture doesn’t allow to use the Docker Plugin in Jenkins, because even though we manage, thanks to a nginx reverse proxy, to give Jenkins access to Docker (e.g. it can build images, start and stop containers remotely), it cannot reach the Docker containers via SSH because the Docker Plugin uses only random SSH ports, which we cannot predict in our iptables configuration for port forwarding.

Therefore in this architecture, we will only have one Docker container, managed outside of Jenkins.

Hosts:

• Host 1, running VirtualBox and two VMs.
• Host 2, running VirtualBox and one VM.
• Developer machine, aka “my laptop”.

VMs:

• VM-repo, which runs git and Mercurial (hg).
• VM-ci, which runs Jenkins, with the Docker Plugin installed.
• VM-with-docker, which runs Docker.

Docker containers:

• jenkins-slave0, where Jenkins executes its jobs.
• docker-registry, that contains the image for jenkins-slave0.

Shared folders:

• /var/docker-registry, on Host 2, accessed by the container “docker-registry”.

Versions:

• VirtualBox 4.3.30
• CentOS 7.1.1503 for “VM-with-docker”
• Docker 1.7.0
• nginx 1.6.2
• Jenkins 1.6.23

As for today, the Dockerfile builds an image with:

• Debian 7.8
• openssh-server 1:6.0
• Java 1.7.0_79

Architecture #1: Steps 1 and 2

I retrieve the Dockerfile from the SCM.

I log into “VM-with-docker” and execute the “create_image.sh” script (see above).

I start the Docker container myself (“jenkins-slave0”).

Architecture #1: Steps 3 and 4

Thanks to port forwarding or other techniques, Jenkins connects via SSH to the “jenkins-slave0” Docker container and runs jobs on it.

Architecture #2

We still give Docker and Jenkins two separate VMs, but this time on the same host, so they share the same host-only network.

Now there are several Jenkins slaves running as Docker containers.

The complete architecture is as follows.

Architecture #2: Nodes

This architecture allows the Docker Plugin in Jenkins to access the Docker containers on random ports via SSH.

The Docker Plugin makes it very easy to start, say, 10 containers, to build up to 10 jobs in parallel. Each container is specific to a build and is removed once the build has ended.

Hosts:

• Host 2, running VirtualBox and two VMs.
• Developer machine, aka “my laptop”.

VMs:

• VM-ci, which runs Jenkins, with the Docker Plugin installed.
• VM-with-docker, which runs Docker.

Docker containers:

• jenkins-slave<n>, where Jenkins executes its jobs.
• docker-registry, that contains the image for the jenkins-slave<n> containers.

Shared folders:

• /var/docker-registry, on Host 2, accessed by the container “docker-registry”.

Versions:

• VirtualBox 4.3.30
• CentOS 7.1.1503 for “VM-with-docker”
• Docker 1.7.0 — The Docker Plugin doesn’t support 1.7.1 yet.
• Jenkins 1.6.23
• Docker Plugin 0.10.2

As for today, the Dockerfile builds an image with:

• Debian 7.8
• openssh-server 1:6.0
• Java 1.7.0_79

Architecture #2: Steps 1 and 2

(Dockerfile retrieved from the SCM.)

I log into “VM-with-docker” and execute the “create_image.sh” script (see above).

Architecture #2: Steps 3 and 4

The Docker plugin in Jenkins communicates with Docker via its REST API and has it start/stop containers based on the “jenkins-slave0” image.

Connecting via SSH to each container, Jenkins runs one job on each one.

After a job is done, Jenkins discards the corresponding container.

Initial Setup

All about installing Docker and the Docker registry, port mappings, etc. here: Docker: jenkins-slave0 Initial Setup

BizCommands

Business command objects, or “BizCommands”, are at the core of our architecture.

There are several applications, or “AppCenters”, and one business core, or “BizMaster”.

States

• Each application may have its own data storage.
• The real data is the data managed at the business level.
• The application data may be seen as a view of the business state.
• States (that is, data storages) can be seen as caches.

Commands

• Each business command is a transition from a business state “n” to another state “n+1”.
• Ideally, the business data could be reconstructed from a savepoint, by performing all business commands again.
• Business commands hold the real information.

Default Architecture

The applications post business commands, and the business core is responsible for handling them. Then the business core calls the application level back, which, notified of a business change, performs the corresponding changes in the application data. This may be seen as caching.

This can be done synchronously or asynchronously. The default mode is asynchronous.

The default architecture follows the following pattern:

The two data models “app data1” and “app data2” may be completely different.

The “b_converter” parts make sure the BizMaster doesn’t access the application data directly.

The main idea is that it’s the BizMaster that answers business commands sent by the AppCenters, and not the AppCenters that call the BizMaster synchronously.

Let’s look in detail at the communication mechanism between an AppCenter and the BizMaster:

We can see that:

• The AppCenter never connects to the BizMaster.
  • Note: This can be nice even from a security perspective.
• The BizMaster connects to the AppCenter:
  
  1. Via a queue subscription, to get notified when a new business command is posted.
  2. Synchronously, to manage remote commands.
  3. Synchronously, to inject changes into the app data.

Generally, business commands are also stored within the “business data” resource on the BizMaster’s side (generally a database), but semantically, business commands and business data may be separated.

Workers are not event-driven: They are really asynchronous. Think of an infinite loop performing business tasks. Using a queue for notifications is a default implementation.

On the contrary, the business converter (“b_converter”) and the command manager (“cmd_mgr”) are event-driven.

Benefits

• All business command is logged, even when it didn’t succeed: Very easy to audit the app from a business perspective.
• Data migrations and app migrations can be run smoothly. If a delta exists between business states before and after a migration, it means that some new business commands have been performed in the old system prior to switching to the new one: This can be resorbed by running those same business commands on the new system (this should be idempotent with respect to the corresponding app data, though).
• Data replication (even at the business level) is non-blocking.

Following are a list of migration scenarios, taking benefits of the loose coupling between the AppCenter and the BizMaster.

• Migration Scenario #1: webapp
• Migration Scenario #2: b_converter
• Migration Scenario #3: webapp + app data + b_converter
• Migration Scenario #4: business workers
• Migration Scenario #5: b_converter + business workers
• Migration Scenario #6: business data + business workers

Migration Scenario #1: webapp

We switch from a v0 app to a v1 app, but no changes are required in the app data.

Deploy the new app:

Link the new app to the existing resources:

Switch the entry point (namely, update the nginx configuration and restart nginx):

Get rid of the old app:

Migration Scenario #2: b_converter

We switch from a v0 converter to a v1 converter, but no changes are required in the app data or in the web app.

Deploy the new converter:

Link the new converter to the existing resources:

Stop the existing workers. Incoming business commands will be held in the queue. Restart the workers, having them point to the new converter:

Get rid of the old converter:

Migration Scenario #3: webapp + app data + b_converter

Here we have a data migration.

The old data can take a while to copy and translate into the new schema, so it’s likely the system will receive some new business commands from the moment the copy starts and the moment it ends.

Say we start the copy+translation while the v0 app data is in state “1205001”. We capture the current pending business commands.

When we finish the copy+translation, we have a v1 app data in state “1205001”, and a v0 app data in state “1205489”.

The delta results from the business commands that have been performed (and added to app data v0) during the copy.

At this point, we can fill the delta with some ad hoc batch programs:

Once the batch programs are done reaching the “1205489” state in our v1 data, some more new business commands have been added to our v0 data state, which is now “1205506”.

We shutdown the workers, so the v0 data will not be updated any more, and keep running the batch programs.

See that the web app is still running as v0, recording business commands and holding them in the queue, but no business command is actually performed.

From a user perspective, we can also freeze the web app (message: “Sorry, we are in maintenance mode!”)

Once the “1205506” state has been reached for our v1 data, we can shutdown the batch programs, restart the workers and have them point to the new v1 converter. It is safe to switch the app to v1.

We get rid of old components.

Migration Scenario #4: business workers

Deploy the new workers:

Make them point to the local resources, but not yet subscribe to the queue and/or call the remote b_converter:

Stop the old workers. At this point, no business command is being performed at all.

Make the new workers point to the remote b_converter and subscribe to the queue:

Migration Scenario #5: b_converter + business workers

Deploy the new components:

Link them to the resources:

Stop the old workers:

Start the new workers and have them point to the new converter:

Migration Scenario #6: business data + business workers

Just as in Scenario #3, we have a data migration.

Say we start the copy+translation while the v0 business data is in state “9315001”. We capture the current pending business commands.

When we finish the copy+translation, we have a v1 business data in state “9315001”, and a v0 business data in state “9315489”.

The delta results from the business commands that have been performed during the copy.

At this point, we can fill the delta with some ad hoc batch programs:

Once the batch programs are done reaching the “9315489” state in our v1 data, some more new business commands have been added to our v0 data state, which is now “9315506”.

We shutdown the old workers, so the v0 data will not be updated any more, and keep running the batch programs.

See that the web app is still recording business commands and holding them in the queue, but no business command is actually performed.

From a user perspective, we can also freeze the web app (message: “Sorry, we are in maintenance mode!”)

Once the “9315506” state has been reached for our v1 data, we can shutdown the batch programs, and start the new workers.

We get rid of old components.