Jenkins setups for Kubernetes and Docker Workflow


During the summer I had the chance to play a little bit with Jenkins inside Kubernetes. More specifically I wanted to see what's the best way to get the Docker Workflow Plugin running.

So, the idea was to have a Pod running Jenkins and use it to run builds that are defined using Docker Workflow Plugin.  After a lot of reading and a lot more experimenting I found out that there are many ways of doing this, with different pros and different cons each. 

This post goes through all the available options. More specifically:
  1. Builds running directly on Master
  2. Using the Docker Plugin to start Slaves
  3. Using the Docker Plugin and Docker in Docker
  4. Using Swarm clients
  5. Swarm with Docker in Docker
Before I go through all the possible setups, I think that it might be helpful to describe what are all these plugins.

Docker Plugin 

Jenkins plugin that is using Docker in order to create and use slaves. It uses http in order to communicate with Docker and create new containers. These containers only need to be java ready and also run SSHD, so that the master can ssh into them and do its magic. There are a lot of images for slave containers over the internet, the most popular at the time of my reattach was the evarga jenkins slave.

The plugin is usable but feels a little bit flaky, as it creates the Docker container but sometimes it fails to connect to the slave and retries (it usually takes 2 to 3 attempts). Tried with many different slave images and many different authentication methods (password, key auth etc) with similar experiences. 


Having a plugin to create the slave is one approach. The other is "Bring your own slaves" and this is pretty much what swarm is all about. The idea is that the Jenkins master is running the Swarm plugin and the users are responsible for starting the swarm clients (its just a java process). 

The client connects to the master and let's it know that it is up and running. Then the master is able to start builds on the client.

Docker Workflow Plugin

This plugin allows you to use Docker images and containers in workflow scripts, or in other words execute workflow steps inside Docker containers & create Docker from workflow scripts.


To encapsulate all the requirements of your build in a Docker image and not worry on how to install and configure them.
Here's how an example Docker Workflow script looks like:

Note: You don't need to use the Docker Plugin to you the Docker Workflow Plugin.
Also: The Docker Workflow Plugin is using the Docker binary. This means that you need to have the docker client installed wherever you intend to use the  Docker Workflow Plugin.
Almost forgot: The "executor" of the build and the containers that participate in the workflow, need to share the project workspace. I won't go into details, right now. Just keep in mind that it usually requires access to specific paths on the docker host (or some short of shared filesystem). Failure to satisfy this requirements leads to "hard to detect" issues like builds hunging forever etc. 

Now we are ready to see what are the possible setups.

No slaves

This is the simplest approach. It doesn't involve Jenkins slaves, the builds run directly on the master by configuring a fixed pool of executors.

Since there are no slaves, the container that runs Jenkins itself will need to have the Docker binary installed and configured to point to the actual Docker host.

How to use the docker host inside Kubernetes?

There are two approaches:
  1. Using the Kubernetes API
  2. By mounting /var/run/docker.sock
You can do (1) by using a simple shell script like the one below.

You can (2) by specifying a hostDir volume mount on Jenkins POD.

An actual example of such setup can be found here.


  1. Simplest possible approach
  2. Minimal number of plugins


  1. Doesn't scale
  2. Direct access to the Docker daemon
  3. Requires access to specific paths on the host (see notes on Docker Workflow Plugin)

Docker Plugin managed Slaves

The previous approach doesn't scale for the obvious reasons. Since, Docker and Kubernetes are already in place, it sounds like a good idea to use them as a pool of resources.

So we can add Docker Plugin and have it create a slave container for each build we want to run. This means that we need a Docker container that will have access to the Docker binary (docker workflow requirement) and will also mount the workspace of the project from the master.

As mentioned above the master will need to connect via ssh into the slave. For this to succeed, either credentials need to get configured or the proper ssh keys. In both cases the xml configuration of the docker plugin needs to get updated in order to refer to the id of the Jenkins credentials configuration (for example see this config.xml).

So what exactly is this id?

Jenkins is using the Credentials Plugin to store and retrieve credentials. Each set of credentials has a unique id and other plugins can use this id in order to refer to a set of credentials. For security reasons the passwords, passphrase etc are not stored in plain text, but instead they are encrypted using SHA256. They key that is used for encryption is also encrypted so that things are more secure. You can find more details on the subject on this great post on "Credentials storage in Jenkins".

What I want you to note, is that due to the way credentials are stored in Jenkins  its not trivial to create a master and a slave image that talk to each other, without human interaction. One could try to use scripts like:

To generate the secret and the master key. And to use them for encrypting a password you can use a  script like:

To actually encrypt the passwords. I wouldn't recommend this to anyone, I am just showing the scripts to emphasise on how complex this is. Of course, scripts like that also make use of details internal to Credentials Plugin  and also feels a little  hacky. What I found a slightly more elegant approach to configure credentials by throwing the following groovy script inside Jenkins init.groovy.d:

The snippet above demonstrates how to create both username/password credentials and also SSH private key with an empty passphrase.

  1. Simple enough
  1. Docker Plugin is currently not there yet?
  2. Direct access to the Docker daemon
  3. Requires access to specific paths on the host (see notes on Docker Workflow Plugin)
Even if we put the issues with the Docker Plugin aside, I'd still like to go for an approach that wouldn't directly talk to the Docker daemon that is running behind Kubernetes.

Docker Plugin managed Slaves with D.I.N.D.

Why would one want to use Docker in Docker?

In our case in order to avoid going behind Kubernetes back.

The number of possibilities here grows. One could use DIND directly on the Kubernetes master, or one could combine it with the Docker Plugin so that each slave runs its own daemon and be 100% isolated.

Either way,  what happens during the build is completely isolated from the rest of the world. On the other hand it does require the use of privileged mode. This can be an issue as the mode may not be available in some environments (i.e. it wasn't available on Google Container Engine last time I checked).

Note: By hosting a docker daemon in the slave, frees us from the requirement of using volume mounts on the outer docker (remember, only the executor and the workflow steps need to share workspace).

  1. 100% Isolation
  2. Doesn't require access to specific paths on outer docker!
  1. Complexity
  2. Requires Privileged Mode
  3. Docker images are not "cached"

Using Swarm Clients

D.I.N.D. or not one still has to come up with a solution for scaling and Docker Plugin so far doesn't seem like an ideal solution. Also the equivalent of the Docker Plugin for Kubernetes (the Kubernetes Plugin) does seem that it needs a little more attention. So we are left with Swarm.

Using the Swarm does seem like a good fit, since we are using Kubernetes and its pretty trivial to start N number of containers running the Swarm client. We could use a replication controller with the appropriate image.

  1. Fast
  2. Scaleable
  3. Robust
  1. Slaves need to get managed externally.
  2. Requires access to specific paths on the host (see notes on Docker Workflow Plugin)

Using Swarm Clients with D.I.N.D.

The main issue with D.I.N.D. in the this use case, is the fact that the images in the "in Docker"  are not cached. One could try to experiment with sharing the Docker Registry but I am not sure if this is even possible.

On the other hand with most of the remaining options we need to use hostPath mounts, which may not work in some environments.

A solution that solves both of the issues above is to combine Swarm with D.I.N.D.

With Swarm the clients stay (rather than get wiped after each build). This solves the image caching issues.

Also, with D.I.N.D. we no longer need to use hostPath mounts via Kubernetes.

So we have a win - win.

  1. Fast
  2. Scaleable
  3. Robust
  4. 100% Isolation
  5. Images are cached
  1. Slaves need to get managed externally.

Closing thoughts

I tired all of the above setups as part of a poc I was doing: "Jenkins for Docker Workflow on Kubernetes" and I thought that I should share. There are still things I'd like to try like:

  • Use secrets for authentication to the slaves.
  • Remove clutter
  • etc

Feel free to add experiences, suggestions, correction in the comments.
I hope you found it useful.


Fabric8 Kubernetes and Openshift Java DSL


The first releases of the Fabric8 v2 have been using a JAX-RS based Kubernetes client that was using Apache CXF. The client was great, but we always wanted to provide something thinner, with less dependencies (so that its easier to adopt). We also wanted to give it a fecelift and build a DSL around it so that it becomes easier to use and read.

The new client currently lives at: https://github.com/fabric8io/kubernetes-client and it provides the following modules:
  1. Kubernetes client.
  2. An Openshift client.
  3. Mocking framework for all the above (based on EasyMock)
A first glance at the client

Let's have a quick look on how you can create, list and delete things using the client:

The snippet above is pretty much self explanatory (and that's the beauty of using a DSL), but I still have a blog post to fill, so I'll provide as many details as possible.

The client domain model

You could think of the client as a union of two things:
  1. The Kubernetes domain model.
  2. The DSL around the model.
The domain model is a set of objects that represents the data that are exchanged between the client and Kubernetes / Openshift. The raw format of the data is JSON. These JSON objects are quite complex and their structure is pretty strict, so hand crafting them is not a trivial task.

We needed to have a way of manipulating these JSON objects in Java (and being able to take advantage of code completion etc) but also stay as close as possible to the original format. Using a POJO representation of the JSON objects can be used for manipulation, but it doesn't quite feel like JSON and is also not really usable for JSON with deep nesting. So instead, we decided to generate fluent builders on top of those POJOs that used the exact same structure with the original JSON.

For example, here the JSON object of a Kubernetes Service:

The Java equivalent using Fluent Builders could be:

The domain model lives on its own project: Fabric8's Kubernetes Model.  The model is generated from Kubernetes and Openshift code after a long process:
  1. Go source conversion JSON schema
  2. JSON schema conversion POJO
  3. Generation of Fluent Builders
Fluent builders are generated by a tiny project called sundrio, which I'll cover in a future post.

Getting an instance of the client

Getting an instance of the default client instance is pretty trivial since an empty constructor is provided. When the empty constructor is used the client will use the default settings which are:

  • Kubernetes URL
    1. System property "kubernetes.master"
    2. Environment variable "KUBERNETES_MASTER"
    3. From ".kube/config" file inside user home.
    4. Using DNS: "https://kubernetes.default.svc"
  • Service account path "/var/run/secrets/kubernetes.io/serviceaccount/"

More fine grained configuration can be provided by passing an instance of the Config object.

Client extensions and adapters

To support Kubernetes extensions (e.g Openshift)  the client uses the notion of the Extension and the Adapter. The idea is pretty simple. An extension client extends the default client and implements the Extension. Each client instance can be adapted to the Extension as long as an Adapter can be found via Java's ServiceLoader (forgive me father). 

Here's an example of how to adapt any instance of the client to an instance of the OpenshiftClient:

The code above will work only if /oapi exists in the list of root paths returned by the Kubernetes Client (i.e. the client points to an open shift installation). If not it will throw an IllegalArugementException.

In case the user is writing code that is bound to Openshift he can always directly instantiate an Instance of the default openshift client.

Testing and Mocking

Mocking a client that is talking to an external system is a pretty common case. When the client is flat (doesn't support method chaining) mocking is trivial and there are tons of frameworks out there that can be used for the job. When using a DSL though, things get more complex and require a lot of boilerplate code to wire the pieces together. If the reason is not obvious, let's just say that with mocks you define the behaviour of the mock per method invocation.  DSLs tend to have way more methods (with fewer arguments) compared to the equivalent Flat objects. That alone increases the work needed to define the behaviour. Moreover, those methods are chained together by returning intermediate objects, which means that they need to be mocked too, which further increases both the workload and the complexity. 

To remove all the boilerplate and make mocking the client pretty trivial to use we combined the DSL of the client, with the DSL of a mocking framework: EasyMock. This means that the entry point to this DSL is the Kubernetes client DSL itself, but the terminal methods have been modified so that they return "Expectation Setters". An example should make this easier to comprehend. 

The mocking framework can be easily combined with other Fabric8 components, like the CDI extension. You just have to create @Produces method that returns the mock.



Using Camel, CDI inside Kubernetes with Fabric8


I recently blogged about Injecting Kubernetes Services with CDI. In this post I am going to take things one step further and bring Apache Camel into the picture. So, I am going to use Camel's CDI support to wire my components and routes, along with Fabric8's CDI extension to automatically inject Kubernetes services into my components.

I am going to reuse stuff from my previous post (so give it a read if u haven't already) to build an standalone camel cdi application that is going to expose the contents of a database via http (a simple http to jdbc and back again). Everything will run in Docker and orchestration will be done by Kubernetes.

So first thing first. How camel and cdi works....

The camel cdi registry

Apache Camel is using the notion of a registry. It uses the registry to lookup for objects, that are needed by the routes. Those lookups may by type or by name.

The most common use of the registry is when the endpoint uri is processed, camel will parse the scheme and will lookup the registry by name for the appropriate component. Other cases involve passing bean references to endpoints by name and so on...

 In other words Apache Camel may perform lookups on the bean registry on runtime.

Any extension that needs to play nicely with Apache Camel needs to provide beans with a predictable names.

The @Alias annotation

Fabric8's CDI extension, for any given service, may register more than one beans (one per service per type, per protocol ...). So, it's impossible to have service beans named after the service. Also the user shouldn't have to memorise the naming conventions that are used internally...

"So, how does Fabric8 play with frameworks that rely on 'by name' lookups?"

Fabric8 provides the @Alias annotation which allows the developer to explicitly specify the bean name of the injected service. Here's an example:

"What happens here?"
The Fabric8 cdi extension will receive an event that there is an injection point of type String, with 2 qualifiers:

  1. ServiceName with value "mysql".
  2. Alias with value "mysqldb".
So when it creates beans and producers for that service it will use the "mysqldb" as a name. This is what allows control over the Fabric8 managed beans and makes name lookups possible.

Using @Factory to create or configure Camel components or endpoints
In my previous post, I went through some examples on how you could use Fabric8's @Factory annotation in order to create jdbc connections. Now, I am going to create a factory for a jdbc datasource, which then is going to be added to the Apache Camel Cdi Bean Registry.

Now if we wanted to refer this datasource from an Apache Camel endpoint, we would have to specify the "name" of the datasource to the endpoint uri. For example "jdbc:custmersds", where customersds is the name of the datasource.

"But, how can I name the fabric8 managed datasource?"

This is how the @Alias saves the day:

This is a typical RouteBuilder for CDI based Camel application. What is special about it is that we inject a DataSource named "customersds".

 "Who provides the DataSource?"

Short answer: Fabric8.

Not so short answerThe @ServiceName("mysql") annotation tells Fabric8 that the DataSource refers to the "mysql" Kubernetes service.  Fabric8 will obtain the url to that service for us. Since the type of the field is neither String, nor URL but DataSource, Fabric8 will lookup for @Factory methods that are capable of converting a String to a DataSource. In our case it will find the DataSourceFactory class which does exactly that. As this was not awesome enough the DataSourceFactory also accepts @Configuration MysqlConfiguration, so that we can specify things like database name, credentials etc (see my previous post).

Configuring the DataSource
Before I start explaining how we can configure the DataSource, let me take one step back and recall MysqlConfiguration from my previous post:
As I mentioned in my previous post we can use environment variables in order to pass configuration to our app. Remember this app is intended to live inside a Docker container....

MysqlConfiguration contains 3 fields:

  1. Field username for environment variable USERNAME
  2. Field password for environment variable PASSWORD
  3. Field databseName for environmnet variable DATABASE_NAME 
So we need 3 environment variables one for each fields. Then our DataSourceFactory will be passed an instance of MysqlConfiguration with whatever values can be retrieved from the environment, so that it create the actual DataSource.

 "But how could I reuse MysqlConfiguration to configure multiple different services ?"

So, the idea is that a @Factory and a @Configuration can be reusable. After all no need to have factories and model classes bound to the underlying services, right?

Fabric8 helps by using the service name as a prefix for the environment variables. It does that on runtime and it works like this:

  1. The Fabric8 extension discovers an Injection Point annotated with @ServiceName
  2. It will check the target type and it will lookup for a @Factory if needed.
  3. The @Factory accepts the service URL and an instance MysqlConfiguration
  4. MysqlConfiguration will be instantiated using the value of @ServiceName as an environment variable prefix.
So for our example to work we would need to package our application as a Docker container and then use the following Kubernetes configuration:

Now if we need to create an additional DataSource (say for a jdbc to jdbc bridge) inside the same container, we would have to just specify additional environment variable for the additional Kubernetes. Now, if the name of the service was  "mysql-target", then our Kubernetes configuration  would need to look like:

... and we could use that by adding to our project an injection point with the qualifier @ServiceName("mysql-target").

You can find similar examples inside the Fabric8 quickstarts. And more specifically the camel-cdi-amq quick start.

Stay tuned
I hope you enjoyed it. There are going to be more related topics soon (including writing integration tests for Java application running on Kubernetes).


Injecting Kubernetes Services in CDI managed beans using Fabric8


The thing I love the most in Kubernetes is the way services are discovered. Why?

Mostly because the user code doesn't have to deal with registering, looking up services and also because there are no networking surprises (if you've ever tried a registry based approach, you'll know what I am talking about).

This post is going to cover how you can use Fabric8 in order to inject Kubernetes services in Java using CDI.

Kubernetes Services

Covering in-depth Kubernetes Services is beyond the scope of this post, but I'll try to give a very brief overview of them.

In Kubernetes,  applications are packaged as Docker containers.  Usually, it's a nice idea to split the application into individual pieces, so you will have multiple Docker containers that most probably need communicate with each other. Some containers may be collocated together by placing them in the same Pod, while others may be remote and need a way to talk to each other. This is where Services get in the picture.

A container may bind to one or more ports providing one or more "services" to other containers. For example:
  • A database server.
  • A message broker.
  • A rest service.

The question is "How other containers know how to access those services?"

So, Kubernetes allows you to "label" each Pod and use those labels to "select" Pods that provide a logical service. Those labels are simple key, value pairs.

Here's an example of how we can "label" a pod by specifying a label with key name and value mysql.
And here's an example of how we can define a Service that exposes the mysql port. The service selector is using the key/value pair we specified above in order to define which are the pod(s) that provide the service.

The Service information passed to each container as environment variables by Kubernetes. For each container that gets created Kubernetes will make sure that the appropriate environment variables will be passed for ALL services visible to the container.

For the mysql service of the example above, the environment variables will be:

Fabric8 provides a CDI extension which can be used in order to simplify development of Kubernetes apps, by providing injection of Kubernetes resources.

Getting started with the Fabric8 CDI extension
To use the cdi extension the first step is to add the dependency to the project. Next step is to decide which service you want to inject to what field and then add a @ServiceName annotation to it.
In the example above we have a class that needs a JDBC connection to a mysql database that is available via Kubernetes Services.

The injected serivceUrl will have the form: [tcp|udp]://[host]:[port]. Which is a perfectly fine url, but its not a proper jdbc url. So we need a utility to convert that. This is the purpose of the toJdbcUrl.

Even though its possible to specify the protocol when defining the service, one is only able to specify core transportation protocols such as TCP or UDP and not something like http, jdbc etc.

The @Protocol annotation

Having to find and replace the "tcp" or "udp" values with the application protocol, is smelly and it gets old really fast. To remove that boilerplate Fabric8 provides the @Protocol annotation. This annotation allows you to select that application protocol that you want in your injected service url. In the previous example that is "jdbc:mysql". So the code could look like:

Undoubtably, this is much cleaner. Still it doesn't include information about the actual database or any parameters that are usually passed as part of the JDBC Url, so there is room for improvement here.

One would expect that in the same spirit a @Path or a @Parameter annotations would be available, but both of these are things that belong to configuration data and are not a good fit for hardcoding into code. Moreover, the CDI extension of Fabric8 doesn't aspire to become a URL transformation framework. So, instead it takes things up a notch by allowing you to directly instantiate the client for accessing any given service and inject it into the source.

Creating clients for Services using the @Factory annotation

In the previous example we saw how we could obtain the url for a service and create a JDBC connection with it. Any project that wants a JDBC connection can copy that snippet and it will work great, as long as the user remembers that he needs to set the actual database name.

Wouldn't it be great, if instead of copying and pasting that snippet one could component-ise it and reuse it? Here's where the factory annotation kicks in. You can annotate with @Factory any method that accept as an argument a service url and returns an object created using the URL (e.g. a client to a service).  So for the previous example we could have a MysqlConnectionFactory:
Then instead of injecting the URL one could directly inject the connection, as shown below.
What happens here?

When the CDI application starts, the Fabric8 extension will receive events about all annotated methods. It will track all available factories, so for for any non-String injection point annotated with @ServiceName, it will create a Producer that under the hood uses the matching @Factory.

In the example above first the MysqlConnectionFactory will get registered, and when Connection instance with the @ServiceName qualifier gets detected a Producer that delegates to the MysqlConnectionFactory will be created (all qualifiers will be respected).

This is awesome, but it is also simplistic too. Why?
Because rarely a such a factory only requires a url to the service. In most cases other configuration parameters are required, like:

  • Authentication information
  • Connection timeouts
  • more ....

Using @Factory with @Configuration

In the next section we are going to see factories that use configuration data. I am going to use the mysql jdbc example and add support for specifying configurable credentials. But before that I am going to ask a rhetorical question?

"How, can you configure a containerised application?" 

The shortest possible answer is "Using Environment Variables".

So in this example I'll assume that the credentials are passed to the container that needs to access mysql using the following environment variables:
Now we need to see how our @Factory can use those.

I've you wanted to use environment variables inside CDI in the past, chances are that you've used Apache DeltaSpike. This project among other provides the @ConfigProperty annotation, which allows you to inject an environment variable into a CDI bean (it does more than that actually).

This bean could be combined with the @Factory method, so that we can pass configuration to the factory itself.

But what if we had multiple database servers, configured with a different set of credentials, or multiple databases? In this case we could use the service name as a prefix, and let Fabric8 figure out which environment variables it should look up for each @Configuration instance.

Now, we have a reusable component that can be used with any mysql database running inside kubernetes and is fully configurable.

There are additional features in the Fabric8 CDI extension, but since this post is already too long, they will be covered in future posts.

Stay tuned.