Docker: a jenkins-slave0 strategy

Well, yes I love Docker.

Thanks to Máté Varga and others, working at PKB, I delved into this wonderful concept.

First, I wanted to create a small architecture for continuous builds, that would run Maven on blank environments, thus not relying on any cached artifacts in the local repository. I expect the following benefits:

• Ensure that each developer is able to checkout any Maven project and build it independently on a fresh new environment.
• Detect errors in dependencies.
• Make those builds disposable: They’re here for error detection, but they may take a long time. Therefore, the actual development process mustn’t rely on them at all. For instance, at this stage they don’t push their results to the company’s Maven repository.

Using a complete VM for that kind of task, even with snapshots, is a little tedious, and uses a lot of empty space. Even with Dynamically allocated storages, just think of the RAM reserved for the VM.

But, maintaining a Dockerfile in the SCM, letting this file be equivalent to a Docker image, and running this image as a container somewhere, disposing of it right after each build, well, that sounds a lot better.

Internally, I named this image “jenkins-slave0”, because it’s being used as a Jenkins slave, with the minimum installed on it.

Workflow

So, here’s the workflow I wanted to set up:

1. Maintain some “jenkins-slave0” Dockerfile in some “my-sysadmin” project, in git or Mercurial.
2. From this Dockerfile, have a “jenkins-slave0” image built at the beginning of the workflow. Optional: Store this image in a custom Docker registry.
3. For each job, have Jenkins connect to Docker, start a new container.
4. Have Jenkins run the job on it. At the end of each job, Docker automatically removes the container.

Steps 1 and 2 would occur less than once a month.

Step 3 and 4 would occur many times a day.

Components

The Docker image contains:

• A SSH server (it’s for Jenkins to connect to it, remember?)
• Jenkins’ public key
• Git and Mercurial
• A JDK 1.7

It doesn’t contain Maven, since Jenkins downloads it anyway.

Here’s the corresponding Dockerfile:

FROM debian:7.8
MAINTAINER david.andriana@gmail.com

# apt-get
RUN apt-get update && apt-get upgrade
RUN apt-get install -y openssh-server
RUN apt-get install -y git mercurial
RUN apt-get install -y openjdk-7-jdk
RUN apt-get clean all

# SSH Server
RUN mkdir -p /var/run/sshd; \
    sed -i "/PermitRootLogin/s/yes/no/" /etc/ssh/sshd_config; \
    sed -i "/^PasswordAuthentication/d" /etc/ssh/sshd_config && \
    echo "PasswordAuthentication no" >> /etc/ssh/sshd_config
EXPOSE 22

# User: jenkins
RUN useradd -m -s /bin/bash jenkins
ADD [ "jenkins.pub", "/home/jenkins/" ]
RUN mkdir /home/jenkins/.ssh/; \
    cat /home/jenkins/jenkins.pub >> /home/jenkins/.ssh/authorized_keys; \
    chown -R jenkins:jenkins /home/jenkins/.ssh/; \
    chmod 700 /home/jenkins/.ssh/; \
    chmod 600 /home/jenkins/.ssh/authorized_keys

# Start the SSH Server
CMD [ "/usr/sbin/sshd", "-D" ]

The “jenkins.pub” file solely contains the public key I will use to connect via SSH. It’s of the form:

ssh-rsa AAAAB3NzaC1yc2...KP2YEnMub jenkins@host1

Jenkins uses the Docker Plugin. It allows to connect to the Docker server, fetch the image and run the container.

The Docker image creation is launched manually. I didn’t even write any Jenkins job for it. Here’s my “create_image.sh” script:

#!/bin/sh
test -f Dockerfile  || (echo "*** Dockerfile not found"  >&2 && exit 1)
test -f jenkins.pub || (echo "*** jenkins.pub not found" >&2 && exit 1)
docker build --no-cache -t avantage-compris/jenkins-slave0 .

Then, the image is available from the Docker environment.

If I want to push it to the Docker registry:

$ docker tag avantage-compris/jenkins-slave0 localhost:5000/avantage-compris/jenkins-slave0
$ docker push localhost:5000/avantage-compris/jenkins-slave0

And if I want to start a container from the image:

$ docker run -d -p 22000:22 avantage-compris/jenkins-slave0

If I want to connect to the container via SSH:

$ ssh -i /path/to/jenkins.key -p 22000 jenkins@localhost

Architecture #1

For some reasons, I needed the Docker environment to be completely remote.

Also, Docker itself runs in a VM, because:

• Maybe it’s not the best idea to mix responsibilities between the host, which runs services such as iptables, and other components, especially with Docker, which is very low-level and acts under sudo.

• Not sure if the host OS would accept Docker — For instance, in case of CentOS, only CentOS 7 would support it.

I wanted a Docker registry so I can sometimes prepare Docker images on another environment and upload them there.

The Docker registry contains the image for the Jenkins slave.

Note that the image is stored in the Docker environment anyway, so, for this article, the Docker registry is optional.

The Docker registry runs in a container.

The data for the Docker registry is on a shared folder on the host, outside the VM.

The Jenkins slave runs in a container.

The complete architecture is as follows.

Architecture #1: Nodes

Unfortunately, this architecture doesn’t allow to use the Docker Plugin in Jenkins, because even though we manage, thanks to a nginx reverse proxy, to give Jenkins access to Docker (e.g. it can build images, start and stop containers remotely), it cannot reach the Docker containers via SSH because the Docker Plugin uses only random SSH ports, which we cannot predict in our iptables configuration for port forwarding.

Therefore in this architecture, we will only have one Docker container, managed outside of Jenkins.

Hosts:

• Host 1, running VirtualBox and two VMs.
• Host 2, running VirtualBox and one VM.
• Developer machine, aka “my laptop”.

VMs:

• VM-repo, which runs git and Mercurial (hg).
• VM-ci, which runs Jenkins, with the Docker Plugin installed.
• VM-with-docker, which runs Docker.

Docker containers:

• jenkins-slave0, where Jenkins executes its jobs.
• docker-registry, that contains the image for jenkins-slave0.

Shared folders:

• /var/docker-registry, on Host 2, accessed by the container “docker-registry”.

Versions:

• VirtualBox 4.3.30
• CentOS 7.1.1503 for “VM-with-docker”
• Docker 1.7.0
• nginx 1.6.2
• Jenkins 1.6.23

As for today, the Dockerfile builds an image with:

• Debian 7.8
• openssh-server 1:6.0
• Java 1.7.0_79

Architecture #1: Steps 1 and 2

I retrieve the Dockerfile from the SCM.

I log into “VM-with-docker” and execute the “create_image.sh” script (see above).

I start the Docker container myself (“jenkins-slave0”).

Architecture #1: Steps 3 and 4

Thanks to port forwarding or other techniques, Jenkins connects via SSH to the “jenkins-slave0” Docker container and runs jobs on it.

Architecture #2

We still give Docker and Jenkins two separate VMs, but this time on the same host, so they share the same host-only network.

Now there are several Jenkins slaves running as Docker containers.

The complete architecture is as follows.

Architecture #2: Nodes

This architecture allows the Docker Plugin in Jenkins to access the Docker containers on random ports via SSH.

The Docker Plugin makes it very easy to start, say, 10 containers, to build up to 10 jobs in parallel. Each container is specific to a build and is removed once the build has ended.

Hosts:

• Host 2, running VirtualBox and two VMs.
• Developer machine, aka “my laptop”.

VMs:

• VM-ci, which runs Jenkins, with the Docker Plugin installed.
• VM-with-docker, which runs Docker.

Docker containers:

• jenkins-slave<n>, where Jenkins executes its jobs.
• docker-registry, that contains the image for the jenkins-slave<n> containers.

Shared folders:

• /var/docker-registry, on Host 2, accessed by the container “docker-registry”.

Versions:

• VirtualBox 4.3.30
• CentOS 7.1.1503 for “VM-with-docker”
• Docker 1.7.0 — The Docker Plugin doesn’t support 1.7.1 yet.
• Jenkins 1.6.23
• Docker Plugin 0.10.2

As for today, the Dockerfile builds an image with:

• Debian 7.8
• openssh-server 1:6.0
• Java 1.7.0_79

Architecture #2: Steps 1 and 2

(Dockerfile retrieved from the SCM.)

I log into “VM-with-docker” and execute the “create_image.sh” script (see above).

Architecture #2: Steps 3 and 4

The Docker plugin in Jenkins communicates with Docker via its REST API and has it start/stop containers based on the “jenkins-slave0” image.

Connecting via SSH to each container, Jenkins runs one job on each one.

After a job is done, Jenkins discards the corresponding container.

Initial Setup

All about installing Docker and the Docker registry, port mappings, etc. here: Docker: jenkins-slave0 Initial Setup