Wednesday, January 30, 2019

Test Ansible Roles with Test-Kitchen and Vagrant


This writing aims to serve as notes and an introductory tutorial on the use of test-kitchen for the testing framework and exploring some of its strengths toward ansible playbooks integration testing. 

  The source of this article can be found here.

First things first, what is test-kitchen
Test Kitchen is an integration tool for developing and testing infrastructure code and software on isolated target platforms.

Test-kitchen Installation 

test-kitchen is a RubyGem and can be installed with:

$ gem install test-kitchen 
* If you use Bundler, you can add gem "test-kitchen" to your Gemfile and make sure to run bundle install.

The next step is to initialize a new project and add support to your library, Chef cookbook, or empty project :

$ kitchen init

Install the kitchen vagrant driver:

$ gem install kitchen-vagrant kitchen-ansible

     for docker support install 'kitchen-docker'

Lets tests everything works as expected. Start by getting a listing of your platform instances with:

$ kitchen list

Instance Driver Provisioner Verifier Transport Last Action Last Error
default-ubuntu-1604 Vagrant ChefSolo Busser Ssh <Not Created> <None> default-centos-7 Vagrant ChefSolo Busser Ssh <Not Created> <None>

Run Chef on an instance, in this case,default-ubuntu-1604 with:
$ kitchen converge default-ubuntu-1604

Destroy all instances with:
$ kitchen destroy

Run tests with: 
$ kitchen test

Basic Kitchen commands

  • kitchen create – Will create the VM or Docker Container to execute the tests on
  • kitchen converge – Will Execute the playbook in test (e.g. chef or ansible) 
  • kitchen verify – Will execute your verifier tests (e.g. ServerSpec)
  • kitchen destroy – Will destroy the Kitchen tests environment

Writing our first test

Before we move into creating our test lets review some basic terms of test kitchen i.e DriverPlatformsProvisioners and Test Suites.

  Drivers are the components that allow test kitchen to interface with various cloud providers and virtualization technologies in order to run your code.

   Another popular choice could be docker but can also be providers such as AWS, EC2, GCE etc

  Platforms are the operating systems you want to test on. This can be almost any operating system or even multiple. In this example, we use ubuntu trusty and centos 7.

 The provisioner is the tool that we are going to use to converge our machine and/or product.  This guide in focused on ansible configuration management tool which is to be used as out provisioner. Another popular provisioner includes chefsaltstack etc.

     In the provisioner section, we can define  require_ruby_for_busser as true. This instructs Test  Kitchen to install ruby in platform box.  In our case, this is required because we are about to unit test and ensure that we expect to see specific changes.

     Defining this property we can specify the path our ruby executable file exists.

Test suites
 Tests Suites or just suites corresponds the lists of tests to run against each platform.

Test an Ansible role

Let's start to create an Ansible role that we will test using test-kitchen.  

Now, let's initialize test-kitchen for testing our role using vagrant as a driver and ansible as a provisioner:

$ kitchen init --driver=vagrant --provisioner=ansible create kitchen.yml create chefignore create test/integration/default

From the init command we can see that kitchen.yml and test/integration/default folder to have been created. The kitchen.yml file aims to host our configuration while the test/integration/default folder to host our tests.

The contents of kitchen.yml should be :
---driver: name: vagrant
provisioner: name: ansible
- name: ubuntu-16.04 - name: centos-7
suites: - name: default run_list: attributes:
The platform names resolve to vagrant boxes and some modification to further adjust the provisioner section configuration changes should be made as seen below:
--- driver: name: vagrant provisioner: # please correct if ansible only provided name: ansible_playbook # Use the latest version on ansible for the test # (please specify the one you wish to test against) ansible_version: latest # It is a good practive to keep logging verbose for # while testing your playbooks, it provide valuable # info ansible_verbose: true # define verbosity level ansible_verbosity: 2 require_ansible_repo: true # install ruby on the testing platform require_ruby_for_busser: true # Playbook configuration name: ansible_playbook # define the host of the testing platform hosts: test-kitchen # set env to local extra_vars: env: local
busser: # define the ruby execiutable location ruby_bindir: /usr/bin platforms: - name: ubuntu-16.04 - name: centos-7 # testing suites to run against the deployment # and verify state suites: - name: default driver: network: # define a specific address for the vm - ["private_network", { ip: "" } ]
Let's verify our installation and configuration so far. In order to do a first sanity check use kitchen list command to list all kitchen instances.

$ kitchen list
Instance Driver Provisioner Verifier Transport Last Action
default-ubuntu-1604 Vagrant AnsiblePlaybook Busser Ssh <Not Created>
default-centos-7 Vagrant AnsiblePlaybook Busser Ssh <Not Created>

Create a sanity test role to test against the platforms, i.e copy-paste the following snippet in test/integration/default/default.yml
--- - hosts: test-kitchen tasks: - name: echo hello command: echo 'Hello World' - name: create a folder file: path: /tmp/test mode: 0755 state: directory
Now proceed with testing in both ubuntu and centos platforms.
$ kitchen create # to create the vms
$ kitchen converge # to run the playbooks in test
-----> Kitchen is finished. (0m2.01s)
Up until now, we have created the VMS and successfully run the playbooks against them. The next step is to verify our changes. For this part, we are going to use ServerSpec framework (i.e RSpec for infrastructure).

  The ansible role has just printed a "hello" message and created a "/tmp/test" folder.  In this example test, we are going to verify that the folder is created indeed. The following two files will allow us to proceed with this test.

 Firstly create a file test/integration/default/serverspec/spec_helper.rb with the following contents .
# -*- encoding: utf-8 -*- require 'serverspec' set :backend, :exec
And then define the serverspec test in test/integration/default/serverspec/sample_spec.rb with the following contents.

# -*- encoding: utf-8 -*- require 'spec_helper' describe 'Nothing' do describe file('/tmp/test') do it { should be_directory } end end
In order to run the tests and verify the changes, we need to run the verify command. Let's try to verify ubuntu VM i.e

$ kitchen verify ubuntu -----> Starting Kitchen (v1.23.3) -----> Verifying <default-ubuntu-1604>... Preparing files for transfer -----> Busser installation detected (busser) -----> Busser plugin detected: busser-serverspec Removing /tmp/verifier/suites/serverspec Transferring files to <default-ubuntu-1604> -----> Running serverspec test suite /usr/bin/ruby2.3 -I/tmp/verifier/suites/serverspec -I/tmp/verifier/gems/gems/rspec-support-3.8.0/lib:/tmp/verifier/gems/gems/rspec-core-3.8.0/lib /tmp/verifier/gems/bin/rspec --pattern /tmp/verifier/suites/serverspec/\*\*/\*_spec.rb --color --format documentation --default-path /tmp/verifier/suites/serverspec Nothing File "/tmp/test" should be directory Finished in 0.07129 seconds (files took 0.17919 seconds to load) 1 example, 0 failures Finished verifying <default-ubuntu-1604> (0m1.16s). -----> Kitchen is finished. (0m1.28s)
NOTE: Please note that centos testing would not work out of the box and some additional work will need, this will not part of the guide.
And the test shows us that the changes are successful! 


Test-kitchen with ansible and serverspec is a powerful combination which allows to fully test our deployments in multiple environments. The very first setup might seem a bit complex but is easy and fast to get the idea and create powerful feedback loops allowing a Test-Driven Development  (TDD) approach to infrastructure deployment toward more stable and safe deployments.



Dependencies issues

In case of issues risen installing the gem dependencies by hand please consider a Gemfile i.e
source '' 
gem 'test-kitchen', '~> 1.8.0' gem 'kitchen-vagrant' gem 'kitchen-ansible' gem 'net-ssh' gem 'serverspec' gem 'kitchen-verifier-serverspec'
And run bundler install in order to proceed with installing them.

Windows Support

Test-kitchen tests support Windows, assuming your test server contains software required to support winrm. The winrm module can be used and provide the connection configuration for the windows server.
For example in .kitchen.yml we need to define the following properties:
ansible_connection: winrm
require_windows_support: true
require_chef_for_busser: false

Parsing date error 

In some cases, we have encountered the following error from the safe_yaml package when try to parse date i.e:
$ kitchen/var/lib/gems/2.3.0/gems/safe_yaml-1.0.4/lib/safe_yaml/parse/date.rb:22:in `<class:Date>': uninitialized constant SafeYAML::Parse::Date::DateTime (NameError)Did you mean? SafeYAML::Parse::Date::DATE_MATCHER from /var/lib/gems/2.3.0/gems/safe_yaml-1.0.4/lib/safe_yaml/parse/date.rb:3:in `<class:Parse>' from /var/lib/gems/2.3.0/gems/safe_yaml-1.0.4/lib/safe_yaml/parse/date.rb:2:in `<module:SafeYAML>' from /var/lib/gems/2.3.0/gems/safe_yaml-1.0.4/lib/safe_yaml/parse/date.rb:1:in `<top (required)>' from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require' from /var/lib/gems/2.3.0/gems/safe_yaml-1.0.4/lib/safe_yaml/load.rb:14:in `<top (required)>' from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:127:in `require' from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:127:in `rescue in require' from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:40:in `require' from /var/lib/gems/2.3.0/gems/test-kitchen-1.19.2/lib/kitchen/loader/yaml.rb:21:in `<top (required)>' from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require' from /var/lib/gems/2.3.0/gems/test-kitchen-1.19.2/lib/kitchen.rb:42:in `<top (required)>' from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require' from /var/lib/gems/2.3.0/gems/test-kitchen-1.19.2/lib/kitchen/cli.rb:21:in `<top (required)>' from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require' from /var/lib/gems/2.3.0/gems/test-kitchen-1.19.2/bin/kitchen:10:in `<top (required)>' from /usr/local/bin/kitchen:22:in `load' from /usr/local/bin/kitchen:22:in `<main>'
For more information and a fix please refer to safe_yaml issue 80

Saturday, October 20, 2018

Snappy Packages

Long time no post ....

In this post, I would like to mention a couple of very early and basic packages I have created recently while exploring the snappy package manager.

First things first, i.e "what is snappy?". As mention in the Wikipedia entry on Snappy package manager "Snappy is a software deployment and package management system originally designed and built by Canonical for the Ubuntu phone operating system. The packages, called 'snaps' and the tool for using them 'snapd', work across a range of Linux distributions and allow therefore distro-agnostic upstream software deployment. The system is designed to work for internet of things, cloud and desktop computing.[2]"

Snappy gains more and more attention in the community and for very good reasons. In my attempt to further explore snappy I have created packages for two favorite CLI tools i.e. bosh and bosh-bootloader.

You can find the source and some basic usage instructions in the links below:

 Your feedback would be more than welcome

Wednesday, November 1, 2017

Use Terraform to serve Azure Container Service (ACS) orchestration platforms

Azure Container Service (ACS), provides a unique approach to managing containers in the cloud by offering a simple way to scale containers in production through proven open source container orchestration technology. Combining Azure Container Service (ACS) with Terraform allows for greater flexibility and control of the infrastructure plane.

This post is all about this, introducing the terraform-azurerm-acs project.

The terraform-azurerm-acs project is a Terraform module to deploy ACS orchestration technologies  (i.e. DCOSKubernetesSwarm) while provisioning the minimal infrastructure required for them in a controlled and flexible way.

For more information on how to use it along with examples if you please the terraform-azurerm-acs repository.

Sunday, October 1, 2017

Enriching Cloud Foundry's marketplace with managed services!

 In today's world of cloud computing "as-a-Service" services has been made possible to minimize operational and infrastructural costs involved in business processes. The cloud services have been able to turn IT operations into an operational expense, instead of a capital expenditure as it used to be considered. These managed services, along with its challenges, introduce new dynamics and possibilities in low cost for a business, often considerably lower than in-house solutions.

Following the hype and the trend of managed services I would like to introduce three new proof-of-concept contributions exposing some of 84codes managed services in Cloud Foundry's marketplace i.e.

For further details please follow the links to access each project's repository.  

  The projects above at present state aim serve as proof-of-concepts and not intended to be used in production.


Sunday, February 26, 2017

Ansible-spiff !

 I would like to present and share with the community another weekend-long pet project ansible-spiff!

 Ansible-spiff is a binary module and is written in GoLang and aims to provide integration between ansible and cloudfoundry's declarative BOSH deployment-manifest builder spiff. The first objective is to act as a mechanism to further enable and improve ansible as the cloudfoundry deployment tool. The second objective is to explore golang as the language of choice for module development and further more to serve as a preliminary framework for golang based modules.

Source code can be found at

P.S contributions are more than welcome

Sunday, October 30, 2016

NullDaddy-DDNS a poor-man's DDNS!

I would like to proudly present NullDaddy-DDNS

 NullDaddy-DDNS is a weekend's long pet project attempting to provide a poor man'sDynamic-DNS using ipify and GoDaddy's - Developer APIs. It is written in Go and by default can be used as a CLI or daemon in any operating system that is supported by Go. A docker image is also available here.

The source code and documentation are available on GitHub.

Thursday, February 27, 2014

Apache Camel meets Kafka!

 LinkedIn engineers in their work in [1] have introduced Kafka, a distributed messaging system that was initially developed for collecting and delivering high volumes of log data with low latency, which was later on donated to Apache Software Foundation (ASF).

 Kafkadue to its architecture, performance and scalability characteristics, has proven to be revolutionary in today’s messaging technologies and has been used with success in several domains and projects - most notably among them the LinkedIn’s real-time activity data pipeline [2].

 It is the Kafka design (more details and background are given in the relevant paper[1]) along with benchmarks, comparisons and discussion found in the community, the reason that I wanted to get involved and learn more about this technology. In the process to learn more about Kafkahow it works and create some toy projetcs , I started also working on an Apache Camel component.

 Apache Camel is another ASF project, one of my most favourite ones, which I consider as an absolute must in many cases, especially when it comes to distributed systems and more over to integration projects.

 This way, I decided to bring together Apache Kafka with Apache Camel, in a similar way like I was doing in my previous work on Camel-Gora component for NoSQL databases.

 All in all, this post is about announcing camel-kafka and providing links to the source code and wiki  pages.  Camel-kafka component gives the ability to utilize Apache Kafka through camel and therefore integrate it in your stack.

Note: documentation and examples to come soon!


[1] Kreps, Jay, Neha Narkhede, and Jun Rao. "Kafka: A distributed messaging system for log processing.Proceedings of the NetDB. 2011.

[2] Goodhope, Ken, et al. "Building LinkedIn's Real-time Activity Data Pipeline."IEEE Data Eng. Bull. 35.2 (2012): 33-45.In