Wisen Tanasa | ceilfors

Ramblings and musings on software development

JUC 2015 - Europe

04 July 2015

This blog contains the report of my attendance for JUC 2015 - Europe in London. I only wrote the bits that I find fresh to my brain and a set of keywords for researchability. More information is captured by workingwithdevs.com liveblog: day 1, day 2 (do note that we attended different talks).

Deployment Pipeline

Deployment techniques

A lot of talks in JUC were talking about blue green deployment i.e. Deploy green with new URL, slowly route everything to green, slowly remove blue when green is stable. At his talk, @sufyaan_kazi shows that CloudFoundry can act as the blue green deployment’s router implementation. CloudFoundry sells the idea that PaaS is the way to go for cloud as it will abstract out the cloud implementation hence not restricting your application to a single cloud. 12factor.net is a recommended reading.

@markusrendall did a live demo to showcase their blue green deployment in AWS. This is the best live demo shown in JUC. He added a twitter button to his web application during the demo and deployed it live. The conference attendees can hit the URL live and see that 50% of the hit don’t see the twitter button in their web page. Accenture referred this technique as A/B deployment. Seems like we haven’t reached a consensus yet on how we should call this technique.

Another deployment technique is shown by @mariocruz by using AWS Elastic Beanstalk. Elastic Beanstalk has an out of the box feature to instantly swap URL in between boxes. In summary, you deploy the new application in a new box, gain some confidence, switch your application URL to your new application by a push of a button. As a CTO, he also shows the importance of how he can rollback this URL swap from his mobile, again with a push of a button.

Anti fragile design

Good mind shift from @martincroker, summary:

  • Assume failure is unavoidable
  • Focus on reducing failures impact, not reducing failures probability
  • Reduce time to recover from failures

Feature flags

Other than the Elastic Beanstalk showcase, @mariocruz also encourage us to adopt feature flags for a smoother and a more flexible app deployment. I think this is a synonym to feature toggle. He is using Netflix Archaius for his feature toggle implementation.

On the site note, he encourage the attendees repetitively that we should check out Netflix OSS as there are a lot of excellent freebies.


New currency: Docker

In the d1key and d1tiger, @kohsukekawa shows how he changed his deployment currency from wars to containers. This is an absolute game changer in continuous delivery paradigm, everyone in the deployment pipeline is interacting only with containers. Container is the new deployment package and Jenkins will be the one who is creating these containers.

During the transition, Cloudbees was faced with some challenges and has made some Jenkins Docker plugins to tackle them:

The sample application that @kohsukekawa used during the demo can be found here:

  • docker-jenkins-demo-base
    Contains the a server.xml that will be built onto a Docker container. This file is the file that he changed during the demo and being reflected after the deployment pipeline is completed.
  • docker-jenkins-demo-app
    The web application source code that is used for the demo. The change of the server.xml is reflected in the HTTP response header. Contains Jenkinsfile, which is a workflow-plugin script; interesting file naming convention.

Container cluster

@CamundaBPM talk talked about how they are running 6 Docker hosts as 1 swarm. The interesting bit is that they are using Docker Plugin to point to Docker swarm. With this setup, the number of slaves are transparent to Jenkins and we can dynamically add more hosts to the swarm to scale Jenkins builds.

In the keynote however, @kohsukekawa was announcing that they will support kubernetes. Kubernetes features seems to be an overlap or even a superset of Docker swarm.

Pets and cattle

This is a funny analogy thrown by @martincroker during his talk.

Pets Cattle
Pets are given names like pussinboots.cern.ch Cattle are given numbers like vm0042.cern.ch
They are unique, lovingly hand raised and cared for They are almost identical to other cattle
When they get ill, you nurse them back to health When they get ill, you get another one

Credits for the table above from it20.info

Of course, he encourages the attendees to use more cattle. The live demo by @markusrendall was showing that all of their demo servers were cattles. They showcased a demo application that is using AWS Cloud Formation. When their JSON file is deployed in the demo, all of the servers, nginx, node app (with A/B deployment), Jenkins, etc, 8 VM with 4 separated subnets goes live. They had nothing but that single json file for all of those to be up. When those servers are up, @markusrendall continued with the demo and ended it by tearing down all those servers.

Automation test

Real world automation test

There are lot of good stuffs from @viktorclerc talk, check out his slides!

Some of the best bits I grasped:

  • Test reports must support and consolidate multiple test tools reports. Don’t try to standardize testing tools, it’s impossible to achieve. Let the testers/developers use any tools they want, we should do the hardwork to create a custom reporting in Jenkins. It’s more important to have tests rather than spending effort standardizing everything. The product they are trying to sell to achieve this is XL Test View.
  • Realise that in real world we can still deploy with failing tests, hence each of the tests need to be labeled properly and visible to everyone. For example if Safari test is failing and there are only 2% of our customers are using Safari, we can deploy them with “known failures”. The Safari label here is important.
  • The world is changing: Testing = Automation. Full stop, it’s an equals symbol there. Testers are becoming developers.
  • Speed of feedback is very important, radical parallelization is the key. Also kill the nightlies.

Automated test tools

Chaos monkey is used by @mariocruz to test his AWS instances. All these monkeys has now formed an army and become Simian Army.

An alternative to JMeter was used by @markusrendall during his live demo: gatling. Not much information about it were provided in the talk.

Masoon Jan is using FitNesse as a tool to do an acceptance test. This test is integration in his deployment pipeline.

Jenkins optimization

Workflow plugin

This plugin was boasted last year and shockingly has been adopted by a lot of speakers in JUC. Workflow plugin is developed by @tyvole and @kohsukekawa in CloudBees. Jenkins core’s code has been modified a lot to support this plugin ever since the development of this plugin was started. Some of the plugin features are only available in Jenkins Enterprise e.g. checkpoints.

Jenkins Blueprint

Jonathan Zenou shared a blueprint for his entire Jenkins setup. The blueprint can be found here. Supposingly anyone can just use that blueprint, spin up Jenkins and browse around the configuration (I haven’t tried this yet). He shared a lot of plugins that he used to tackle common problems in Jenkins. Most of these can also be seen in his slides.

Failure management

Lots of times developers don’t bother to read the entire Jenkins output console and will ask you, the maintainer of Jenkins, to find out what the problem is. Most of the times the cause of the failure is obvious, but it is hard to find as they are in the middle of all other errors. Build Failure Analyzer provides a knowledge base by scanning your console output and provide a meaninful description of the error in the build page. Over time your knowledge base will grow and you will have less context switch!

Pushing for Continuous Integration requires dicipline from developers to fix the broken build as quick as possible. Showing a build failure with a big fat red panel will certainly help. Shared by Masoon Jan, he is using Build Monitor Plugin to achieve this.

Retaining Jenkins build history

In his talk, Robert McNulty was showing how he can retain Jenkins build history by using Groovy and Neo4j. He basically stores all of the build environment and any other specific information that he wants from a particular Jenkins build to Neo4j. After a certain period of time, this data can used for auditing or reviewing. He showed how Neo4j dashboard can display those data in an interesting graphs ouf of the box.

Software compliance

I didn’t manage to capture much information here, but if there is one speaker that talks about this topic it’s @IlkkaTurunen from Sonatype. He showed how putting a colour and code for compliance in Jenkins dashboard would help. The importance of compliance for large financial or technology firms were also shown by some statistics in Maven Central e.g. average downloads with known vulnerabilities, percentage with known defective parts, component downloads with known defects older than 2013, percentage with known defects older than 2013, etc. I vaguely remembered that he was showing “Application composition report”, most likely it’s a part of the software that they are selling called Sonatype CLM.

Others tools worth mentioning

  • Phabricrator: JIRA + Confluence + FishEye + Stash + HipChat all bundled together for free!
  • Terraform: Like CloudFormation but cloud agnostic. Can be used for AWS and OpenStack. I might be oversimplifying, Google it out yourself.
  • IBM Bluemix: PaaS from IBM.


Test code as a source code, Infrastructure as a code, everything as a code, the new common language in DevOps is code.

List of attended talks

Not all slides are made available by @jenkinsconf yet. Latest update.

Day 1

  1. Keynote 1 (d1key) - @singh_harpreet @kohsukekawa
  2. An Integrated Deployment Pipeline with Jenkins and Cloud Foundry (d1pipeline) - @sufyaan_kazi
  3. How to Optimize Automated Testing with Everyone’s Favorite Butler (d1butler) - @viktorclerc - Slides
  4. Making Strides towards Enterprise-Scale DevOps…with Jenkins in Your Tool Chain (d1enterprise) - @IlkkaTurunen
  5. Bringing Continuous Delivery to Cloud-Scale with Jenkins, Docker and “Tiger” (d1tiger) - @kohsukekawa @singh_harpreet
  6. Enabling Continuous Delivery for Major Retailers (d1retailer) - Masood Jan

Day 2

  1. Keynote 2: Why 50 deploys per day is essential (d2key) - @martincroker @markusrendall - Slides
  2. Optimizing Your CI: Lessons Learned from a Successful Jenkins Rebuild (d2rebuild) - Jonathann Zenou - Slides
  3. From DevOps to NoOps (d2noops) - @mariocruz
  4. Hey! What Did We Just Release? (d2release) - Robert McNulty
  5. From Virtual Machines to Containers: Achieving Continuous Integration, Build Reproducibility, Isolation and Scalability (d2container) - @CamundaBPM - Slides
  6. Lightning talks
  7. Multi-Node Environment as a Jenkins Slave (Compound-Slave) (d2compound) - @dchernilevskiy