/dev/random

Making a Status Page

2016-03-11T00:00:00+00:00

We recently had (another) a major outage from one of our payment providers. I decided to make a simple status page that we could direct customers to when something like this happened next time.

Vision

The status page should use existing monitoring infrastructure - we don’t want to duplicate the monitoring effort already in place. The status page itself should be statically hosted, to minimize the risk of the status page itself going down, and fetch service status dynamically - client side - so it does not need to be updated manually.

Status page is not a new thing, in fact there quite a few alternatives out there. pyupio/statuspage and CachetHQ/Cachet are two open source alternatives; while Status.io and StatusPage.io are two commercial proprietary options. As a hacker for a non-profit organization the proprietary ones were out of the question :wink:

I ended up borrowing layout, and concepts, from pyupio/statuspage - a shout out goes to @pyupio for sharing such a solid piece of work with the community :tada:

Service Status

As I mentioned, we already had extensive monitoring of applications, internal- and external services set up through Pingdom. All outages are posted directly to a Slack channel in real time. If left unresolved for enough time alerts are sent out through push notifications and SMS.

The Pingdom API requires user authentication - of course - in order to get out any status data. This would be a problem for the client side status page vision. In order to overcome this restriction I made a simple API proxy (written in Node.js :rocket:) dubbed Starefossen/status-api.

GET /api/v1/checks HTTP/1.1
Host: status.app.dnt.no

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json; charset=utf-8

{
    "checks": [ ... ]
}

We had more checks than we would like, or should, expose on the status page - they needed to be organized. Pingdom had the solution – organization by tags. We tagged all checks we would like to show up on the status page with public. From the API we could now filter using ?tags=public. We also created service type tags to group different checks together; app, service, and payment.

Incident Messages

The next thing we would like for our status page was the ability to post messages about various incidents. We wanted to reuse existing infrastructure as much as possible and we know GitHub had an awesome API, that could even be used without any user authentication.

We could simply create issues directly in the Turistforeningen/status issue tracker on GitHub and fetch them using the GitHub API. This was just what we needed! We added some labels to indicate outage status levels and others to indicate the various systems that could be affected.

Closing the issues would resolve the incident and return the status page to it’s normal green status. This way we would only need to log incidents to GitHub and not worry about some other tool or system to manage this.

Gluing it all together

Now that we have both the individual status checks and incident messages we can start writing some pseudocode for how the status page should work. As you can see from the code below, the whole page is fairly comprehensible.

# get checks and messages
checks = http.get("https://status.app.dnt.no/api/v1/checks")
messages = http.get("https://api.github.com/repos/Turistforeningen/status/issues")

alert = null

# add messages to page
for message in messages
  if message.status is not "resolved"
    alert = message.title

  addMessage(message.title, message.body, message.status)

# add status checks to page
for check in checks
  if check.status is not "up" and alert is null
    alert = check.name + "is down"

  addCheck(check.name, check.status, check.tag)

# show alert header message
showAlertMessage(alert)

The client side implementation for the status page can be found in script.js in the repository up on GitHub. Except from the asynchronous nature of JavaScript as well as the DOM manipulation necessary for adding new elements to a HTML page; you should be able to see the familiarity with the above pseudocode.

If you open up your dev tools on the status page you will see the two XHR request happening before the checks and messages are displayed on the page.

The Power of GitHub Pages

The last piece of the puzzle was a reliable place to host the static files necessary for the status page to function. An index.html, some JavaScript, and little bit of CSS. Since the source code was already hosted on GitHub, and we have had good experience with using GitHub Pages in the past it was the obvious decision. This very blog you are reading is hosted on GitHub Pages as well.

Also read: Jekyll 3 and GitHub Pages

Just push the code to a branch named gh-pages and let GitHub do the rest! We even could customize the domain name simply by adding a CNAME file with the desired domain name – status.dnt.no. The complete source code for our new status page is up at Turistforeningen/status and licensed under the MIT license.

Future Work

As any other project, the Status Page is not done done - there is always more features that could be implemented. Here are some of my thoughts for future improvements.

Response Time

Service response time is currently not considered. Service status could be based on a threshold with regards to the average response time over x amount of time in addition to the downtime.

Response time graphs would also be a nice addition to the operational and major outage labels.

Downtime

Currently the Status Page only checks how long since last reported downtime. This is because the Pingdom API only returns this when listing all the checks. A natural thing to do would be to fetch the duration from the summary API endpoint for services with downtime within some amount of time.

Spaghetti, Notebooks, and Modeling

2016-03-09T00:00:00+00:00

Microservices dependency spaghetti, the concept of an engineering notebook, and the importance of software modeling was three main topics discussed at this meeting with Lean Coffee Bergen.

Microservice Spaghetti

The microservice oriented approach is not a new concept in the world of software engineering, and is more traditionally known as service oriented architecture¹ - or SOA for short. Such a services should be unassociated, loosely coupled units of functionality that are self-contained.

In reality, however, many of the services had in fact hard dependencies such the ability for inter process communication; programming language and version. Even though they appeared unassociated there ware in fact tightly coupled with one another.

A more modern definition of microservices are web services. They implement the same functionality as their SOA counterpart over standard Internet protocols independent of platforms and programming languages. This allows services to be moved around, switched out, rebuilt, and scaled without having to worry about other their dependents (in theory at last).

The Engineering Notebook

The concept of an Engineering Notebook is the method of to precisely record all work, from concept to prototype - preferably in a notebook by hand. Should software engineers adopt more of this mindset too?

The notebook is always on and it comes with no distracting notifications. You will be able to have one canonical source of history - instead of sifting through multiple git repositories or issue trackers. The notebook also has an infinite amount of different drawings and support for complex notations without installing any special tool or ad-on.

On the other hand, notebooks are not easily searchable and depending on your handwriting may not be easily readable after some amount of time. Some attempts has been made to use digital notebooks (tablets) with handwriting recognition software with varied amount of success.

A notebook can be quite useful for personal planning. Project management tools like JIRA are often better for information sharing than personal task management. Simply making a list in your notebook and crossing them out as you go during the day may be a good solution for keeping track of your day to day activities.

Software Modeling

Many software engineers have a formal understanding of software modeling through UML diagrams² from their university education and the formal waterfall software development process³. But what role does software modeling play in today’s world of agile software development?

With Agile development there is less design up front, however, this is not an excuse for not making proper models and diagrams of the system(s) in question. This is a good practice that all engineers must restrain and remember to use.

Diagrams are useful for modeling entities and their relations or dependencies. Some diagram can be thrown away when the implementation is done, while others can be kept as reference or documentation for later. Be mindful if the diagram is one or the other.

If you are using JIRA it might be worth to take a look at Gliffy - a digital tool to create and manage flowcharts, wireframes, UML diagrams directly from issues and wiki pages.

Footnotes

Node.js Drama of 2016: Express.js

2016-02-13T00:00:00+00:00

Express was originally created by TJ Holowaychuk (@tj) in 2009, under the the slogan «Insanely fast (and small) server-side JavaScript web development framework». It has since grown to become one of the most widely installed package from the npm registry with over 4.5 million monthly downloads. Now, it may be facing a new beginning!

Transferred to StrongLoop

To understand the drama surrounding Express you need to know some of the history of the Express project. Lets rewind to the spring of 2014. Node.js is growing rapidly much thanked to modules like Express which makes it super easy for anyone to spin up websites and services using Node.js.

Since 2013 TJ had greatly reduced his involvement in Epxress development, and by May of 2014 Douglas Wilson (@dougwilson) had more or less taken over the role as lead maintainer of Express. Then, shortly thereafter, TJ announced he was leaving the Node.js community¹ and the Express project would find a new home at StrongLoop².

StrongLoop is the author of the LoopBack, a framework for creating APIs built on top of, you might have already guessed it, Express. The transfer of Express to StrongLoop came as a shock to the maintainers³. They had not been informed, and they had lost access to the main repository during the transfer.

[…] TJ has not been the one maintaining express day to day for a long time and I think this move came as a surprise to all of the current maintainers.

–@defunctzombie (source)

The maintainers were upset and they felt this was a move in the wrong direction. Rumors of StrongLoop purchasing Express from TJ did not exactly help. TJ even did a follow up post in an attempt to defuse the situation⁴ but to little or no avail. StrongLoop co-founder Bert Belder (@piscisaureus) promised Express would remain an open source project.

[…] express will remain a community open source project; that’s what makes it interesting in the first place.

-@piscisaureus (source)

Acquired by IBM

Fast forward to September 10Th, 2015. The dust over the transfer to StrongLoop had settled, and the development of Express had slowly declined. IBM announced they had acquired StrongLoop⁵ and a prophecy from a year earlier may be about to come true.

Strongloop may not be a thing in 1 year or 2 years […]

–@defunctzombie (source)

This leads to a very special situation where the new owner of Express, IBM, and the employer of the lead maintainer for Express, Sencha Labs, are competing companies.

[…] @dougwilson (who has maintained express for two years) can no longer contribute to the Express project because IBM is a direct competitor to his employer.

–@chipersoft (source)

Whats left of the maintainers, and other concerned user, makes another attempt to have Express moved to it’s own organization on GitHub - under the banner of «returning Express back to the community». Several attempts was made with IBM during the fall of 2015 but the situation looked more deadlocked than ever. This was apparently the last drop for Doug Wilson in a growing frustration with corporate overlords.

I did not voluntarily give up on Express, IBM forced me out of this repository.

-@dougwilson (source)

By January of 2016 IBM was still figuring out what to do with the sticky situation they had inherited from StrongLoop acquisition. While more and more disgruntled users piled on the discussion about what should happen with Express next, James M Snell (@jasnell) from IBM and Ritchie Martori (@ritch) from StrongLoop was working with various stakeholders within IBM to conclude a solution⁶.

The express project is applying to become an incubating top level project.

–@jasnell (source)

This stirred up the discussion again, and after a lengthy (and heated) discussion back and forth, the incubating application was merged. This means, if everything goes as planned, that Express will become an official Node.js project outside of what is known as Node.js core.

Epilogue

Express has now been transferred to it’s new home at expressjs/express, and Doug Wilson also transferred the ownership of his modules that Express depends on, either directly or indirectly, to the Node.js foundation.

I am donating all of my Node.js modules to the Node.js foundation, to go along with Express.

–@dougwilson (source)

The intellectual property of Express, however, is still owned by IBM and will probably remain so for a long time. The API documentation, a blog post worthy in and of itself, will be pulled into the main repository⁷.

[…] currently there is no IP transfer indicated. The question over the expressjs.com domain name will need to be looked at again a bit later but it will not be included in this initial proposal.

–@jasnell (source)

:sparkles: Stay tuned for more drama! :sparkles:

Footnotes

Jekyll 3 and GitHub Pages

2016-02-11T00:00:00+00:00

Jekyll, the static site generator used by GitHub Pages, recently released their third major version 💯 🌟 and GitHub Pages just announced their support. Here is a list of all the new cool stuff and what you need to change!

What is a static site generator?

Jekyll is one of many static site generators perfect for making for personal, project, or organization sites. Think of it like a file-based CMS, but without all the complexity. A static site generator takes your content, often written in some sort of Markdown flavor, and out comes a complete, static website ready to be served by Apache, Nginx or another web server.

Because the output it is just bunch of static files it does not require any database, it can be cached and served extremely efficient over HTTP. Jekyll is the engine behind GitHub Pages, which you can use to host sites right from your GitHub repositories for free.

What is new in Jekyll 3.0?

Many of the important changes in Jekyll 3.0 are under the hood. The two most prominent new features are related to performance and profiling.

The --incremental regeneration of changes makes local builds significantly faster. This is especially important for large sites, and you will be able to preview changes instantly.

The --profile option tells Jekyll to analyze your site’s build time. This enables you to find exactly the spots where you can speed things up if compile time is an issue.

> jekyll build --profile

Filename	Count	Bytes	Time
feed.xml	1	58.48K	0.200
_layouts/default.html	12	120.54K	0.088
sitemap.xml	1	3.06K	0.054
_includes/side.html	12	11.59K	0.007
_layouts/post.html	9	53.38K	0.006
_includes/head.html	12	18.30K	0.003
index.html	1	9.02K	0.003
_includes/header.html	12	1.05K	0.002
timeline.html	1	6.42K	0.001
_includes/footer.html	12	10.09K	0.001

The default highlighter has been changed from Pygments to Rouge - a pure-ruby code highlighter that is compatible with Pygments. The change should be seamless for most users of Jekyll and GitHub Pages.

The {{ site.collections }} is now array of collections and {{ site.posts }} is now a part of all the collections and not a special array. Read more about Jekyll Collections.

New Filters

Also new is Liquid 3.0. Liquid is the templating engine which makes all the {{ }} and {% %} magic work. As with Jekyll, many of the changes in Liquid 3.0 are under the hood, but this release also includes some new Liquid Filters.

Filter	Example	Output
uniq	`{{ [1,1,3,2,3,1,4,3,2,1] \| uniq }}`	[1,3,2,4]
url_encode	`{{ "Jackô & Jones" \| url_encode }}`	Jack%C3%B4%20%26%20Jones
strip	`{{ "\tab c \n \t" \| strip }}`	ab c
default	`{{ first_name \| default: 'User' }}`	User

What is new in GitHub Pages?

As you might have guessed, GitHub Pages now uses Jekyll 3¹ with all of the new stuff mentioned above. However, there are some other changes as well, and the most important is the change of Markdown parsing engine to Kramdown. If you are using Rediscount or Redcarpet for rendering your Markdown (like I did) you should read on!

When working with the new changes locally I immediately ran into problems. First of all there were some minor inconsistencies in how Jekyll was rendering Markdown. Fenced code blocks using three or more backticks did not evaluate to syntax highlighted code. This was solved by setting input: GFM².

Setting this suddenly all my articles had very strange line breaks, after some googling around I found a post on StackOverflow which suggested setting hard_wrap: false which solved the line break problem³.

Still, syntax highlighting was not working correctly and after even more googling I found a blog post which explained how to get Jekyll working with Rouge syntax highlighter by explicitly setting syntax_highlighter: rouge⁴.

Your final _config.yml should include the following:

markdown: kramdown
highlighter: rouge

kramdown:
    input: GFM
    hard_wrap: false
    syntax_highlighter: rouge

Running Locally

The easiest way to run your Jekyll site locally is to use the GitHub Pages Docker Image .

> docker pull starefossen/github-pages
> docker run -v "$PWD":/usr/src/app -p "4000:4000" starefossen/github-pages

Footnotes

Automatic Publishing of npm Packages

2015-09-20T00:00:00+00:00

If you are actively maintaining more than a couple of packages in the npm registry this will be a life saver for you; automatically publish new versions of your packages directly from your CI server 📡

As the only package maintainer at the Norwegian Trekking Association, and currently maintaining 16 packages in the public npm registry, time is of essence. Anything that can ease the burden of maintaining all the packages is a welcomed addition 😅

Also read: Tagging software releases with git and GitHub

The problem

So what exactly is the problem we are trying to solve? The npm publish command requires the user to be authenticated before they can publish. If you publish from multiple machines and/or have multiple maintainers you will need to make sure that all of those are authenticated with npm.

By “manually” having to publish to npm you introduce the risk of publishing versions that do not pass the tests (you do have tests right?!), and introduce the possibility of inconsistencies between published versions.

npm publish

The npm publish (and more recently npm adduser) commands requires some user interaction to type in user credentials. This makes it problematic for use on servers since there is no user there to type in the username and password.

Because of this limitation some CI servers offers plugins or addons to make the integration with npm easier for maintainers.

npm publish for Travis CI

Travis CI can automatically release your npm package to the npm registry after a successful build. All you need to do is add the following to your .travis.yml configuration:

deploy:
  provider: npm
  api_key: "YOUR API KEY"
  on:
    - tags: true

Also Read: npm Releasing - Travis CI

npm publish for Wercker CI

Wercker CI can automatically release your npm package to the npm registry after a successful build. All you need to do is add the following to your wercker.yml configuration and add the NPM_TOKEN environment variable:

deploy:
  steps:
    - turistforeningen/npm-publish

You also need to add your NPM_TOKEN as an Environment Variable to your Deploy Pipeline in the Wercker settings page for your project.

Also checkout: turistforeningen/wercker-npm-publish

Atom (RSS) feeds for GitHub Pages

2015-09-16T00:00:00+00:00

GitHub recently rolled out support for automatic ATOM (RSS) feed generation for their GitHub Pages platform. It literally only take seconds to set up and get going!

_config.yml

Add the jekyll-feed gem to the list of installed gems for your Jekyll blog. Make sure you have filled out name, description, and author information too.

name: "Blog Name"
description: "Blog Description"
author:
    name: "Your Name"
    email: "you@email.com"

...

gems:
    - jekyll-feed

<head>

Inside the <head> tag, add the {% feed_meta %} tag to automatically add a link to your ATOM feed. This makes you ATOM feed visible to users visiting your blog with a capable feed reader.

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <meta name="description" content="{{ site.description }}">

    <!-- jekyll-feed link -->
    {% feed_meta %}
  </head>

🐳 Testing Locally

You are running Docker right?! If so, just pull the starefossen/github-pages Docker Image and your blog is up and running locally without having to install a single thing!

$ docker pull starefossen/github-pages
$ docker run --rm -it -v "$PWD":/usr/src/app -p "4000:4000" starefossen/github-pages

ProTip™ Changes reloads automatically as you go when using the Docker Image!

💰 Profit

Commit and push you changes to GitHub and your ATOM feed is up and running in under a minute!

Recording H.264 Streams

2015-08-19T00:00:00+00:00

This is a project I did a while a go, but I never got a round to write a post about it 😩 So here it is. It was a ☔️ wet and cold spring in Norway (like most of them are). We were going on a 🚙 trip for the weekend, and I wanted to bring some 📺 TV shows my son could watch on the way.

Summer marked trails in Norway by Hans Kristian Flaatten. Licensed under CC BY-SA 4.0.

Here in Norway we have a public broadcasting company - the Norwegian Broadcasting Corporation (NRK). I have had the opportunity to work with them on UT.no - Norway’s Trip Planner - but more on that later. NRK makes available all of their shows through their online media player tv.nrk.no for a period of 6 to 12 months before they are taken offline.

All this is well and good, except for the disability to save shows for offline viewing. Since we have so spotty cellular connection because of all the high mountains and deep fjords, and streaming 200 episodes of Sesame Street is not an economically viable option anyways, I opted for making my own offline copies. ffmpeg 🎥 to the rescue!

m3u-to-mp4

Getting the m3u-steam was just a matter of right clicking on it in Safari and saving to disk. Once I had the m3u playlist file it was time to fire up ffmpeg, and by the way, there’s a Docker Image for that 😅

m3u is not the actual video stream itself. It is just a playlist composed of links to several small videos that are downloaded and played in rapid succession. ffmpeg has built in support for downloading and concatenate into one video file.

ffmpeg -i playlist.m3u -c copy -bsf:a aac_adtstoasc recording.mp4

Here is a rundown of what the different cli flags do:

-i playlist.m3u is location of the playlist file we want to record.
-c copy do not re-encoding the video stream since it is already the desired format of H.264.
-bsf:a aac_adtstoasc converts the audio such that it is compatible with the mp4 container.
recording.mp4 is the location of the outputted recording.

🐳 Docker All Things

Since I do not install program directly on my computer any more I of course had to run this too in a Docker container. You can check out the project on github.com/Starefossen/m3u-to-mp4 🚀

docker run --rm
  -v $(pwd)/playlists:/playlists
  -v $(pwd)/recordings:/recordings
  nachochip/ffmpeg
    -i /playlists/episode1.m3u
    -c copy
    -bsf:a aac_adtstoasc
    /recordings/episode1.mp4

Building Docker Images from remote repositories

2015-08-16T00:00:00+00:00

I recently wanted to build a Docker Image directly form a source repository without having to clone the repository locally before building it. This turned out to be such a neat little trick that I thought it was worth sharing.

Inspecting what the docker build command had to offer with regards to building repote repositories yield an interesting parameter named URL.

$ docker build --help
> Usage: docker build [OPTIONS] PATH | >>> URL <<< | -
> Build a new image from the source code at PATH

Consulting the online build documentation confirmed my suspicion that this was in fact an option to build a Docker Image directly form a remote repository.

The URL parameter can specify the location of a Git repository; the repository acts as the build context.

The URL parameter also has the optional possibilities to specify a which branch, tag, or commit sha to build, as well as the location (directory) of the Dockerfile you want to build.

$ docker build https://github.com/docker-library/hello-world.git#master:/
> Sending build context to Docker daemon 132.1 k
> Sending build context to Docker daemon
> Step 0 : FROM scratch
>  --->
> Step 1 : COPY hello /
>  ---> a1e6124c9115
> Removing intermediate container b27bf9c2271f
> Step 2 : CMD /hello
>  ---> Running in 80cd51769b8e
>  ---> 3ec9c8c47f66
> Removing intermediate container 80cd51769b8e
> Successfully built 3ec9c8c47f66

Epilogue

I later realized that the repository is cloned locally behind the scenes which makes that sense when you think about authentication, vpn etc to private repositories.

This command runs in a temporary directory on your local host. After the command succeeds, the directory is sent to the Docker daemon as the context. Local clones give you the ability to access private repositories using local user credentials, VPNs, and so forth.

A Node.JS Example

If you are interested in how to build a remote repository using the Docker Remote API with the dockerode Docker client library for Node.JS, this is how you do it:

```javascript const options = { t: ‘docker-library/hello-world’, remote: ‘https://github.com/docker-library/hello-world.git’, };

docker.buildImage(null, options, function(err, res) { if (err) { throw err; }

res.on(‘data’, function(data) { console.log(data.toString()); }); });

Django Performance: POSTGIS_VERSION

2015-05-21T00:00:00+00:00

This is the first post in a series of posts about boosting performance in the main Django application for the Norwegian Trekking Association (DNT). The application is scheduled to serve all of DNTs 200+ official web sites with a total of 1.5 million monthly page views by the end of 2015.

Sherpa is a fairly typical Django application. It uses Django 1.7 served through Gunicorn 1.8 via Nginx 1.9 using memcachd 1.4 for caching and Postgres 9.3 with Postgis 2.1 as its main database. I will try to get a another post out outlining the technical details of our production setup soon.

Step 1 - Gathering Data

We have been measuring the response time of some key pages through our Pingdom monitoring checks for years, and they all yielded the same average of 730ms (originating from Europe since our servers are located in the eu-west-1 aws region).

Other than that we had fairly little data to work with so we decided to log all Postgres queries ~~longer than 300 ms~~ over the weekend using the log_min_duration_statement method as suggested in this Postgres wiki page on logging difficult queries. Update: We later learned that our configuration was not entirely correct, and subsequently all queries were logged. 😁 Oops!

Step 2 - Analyze

When we returned to work next Monday we had 10 GB of query logs to sift through. 😥. A few days later the log file had doubled in size! Postgres System Impact report (pgsi) to the rescue!

perl pgsi.pl --file=postgresql-head.log > pg_analyze.html

Now we had a nicely formatted html page with all the different queries ordered by type (SELECT, INSERT, UPDATE, etc) and system impact. The first entry immediately stood out as a big surprise! 😮

System Impact:    3.43
Mean Duration:    9.69 ms
Median Duration:  5.34 ms
Total Count:      624439
Mean Interval:    282.11 ms
Std. Deviation:   208.81 ms

SELECT postgis_lib_version()

The query with the most system impact of all the logged queries was a simple call to retrieve the version of the installed PostGIS library. How could this be? The total count for this query indicated that it had to be executed for each request to the application.

Searching the django/django repository on GitHub returned one result for a test, and one in what appeared to be the PostGIS driver for Django / GeoDjango – no issues 😕.

def postgis_lib_version(self):
    "Returns the version number of the PostGIS library used with PostgreSQL."
    return self._get_postgis_func('postgis_lib_version')

Upon further inspection of the driver we reviled the following function call chain: postgis_lib_version() <= postgis_version_tuple() <= spatial_version(). In the spatial_version() function definition there where this comment:

Trying to get the PostGIS version because the function signatures will depend on the version used. The cost here is a database query to determine the version, which can be mitigated by setting POSTGIS_VERSION with a 3-tuple comprising user-supplied values for the major, minor, and subminor revision of PostGIS.

And there we had it! For each connection to the Postgres database Django’s PostGIS driver would try to determine what version of PostGIS was installed in order for it to communicate correctly. This is done by executing the postgis_lib_version function unless POSTGIS_VERSION setting is set.

Step 3 - Act

We quickly added POSTGIS_VERSION based on an environment variable to our Django settings, and deployed to production.

if 'POSTGIS_VERSION' in os.environ:
    POSTGIS_VERSION = os.environ['POSTGIS_VERSION']

Step 4 - Profit

Response time immediately dropped by 45%, from 730ms to 400ms. We would never have expected these two lines of code to impact the overall performance of the system in such a way!

Testing Docker apps with Wercker

2015-05-19T00:00:00+00:00

Wercker is a free and hosted continuous delivery platform with a lot of flexibility. Wercker offers everything you would expect from a modern CI service, and they recently announced full support for the Docker container runtime.

The Wercker platform has two modes (known as stacks) to run applications; Classic using custom made pre-built containers, and Ewok using the Docker container runtime. If you don’t know what Docker is, I suggest you read this article in order to get a basic understanding of Docker.

With the Ewok stack your application is run using Docker containers from Docker Hub - linked together just as you would have locally or in production. Yay for identical environments!!

Boxes

You application code is run in a primary container (known as a box). In Ewok the name this box is the name of a Docker image from Docker Hub + an optional image tag.

box: node:0.10

Services

Adhering to the Docker best practices you should only run a single process per container. If your application depends on another service (such as a database or a cache) you can link multiple containers together using services.

The services directive looks just like a box, but you can define as many of them as you like. They will automatically be linked together with your box before your application starts.

services:
    - mongo:2.6
    - redis:2.8

When Wercker starts, any service you have defined will be started and linked together with your box automatically. The IP-address of a given service will be available as a hostname corresponding to the name of the image. Any ports exposed by the container can be used as they are.

Wercker also has support for advanced service configurations, such as injecting custom environment variables and altering the startup command.

services:
    - id: mariadb
      env:
          MYSQL_ROOT_USERNAME: myusername
          MYSQL_ROOT_PASSWORD: mysecretpassword

Build Steps

Your containers are started and linked together, and it is time to run the build. This is done with one, or many, build steps. Wercker has an astonishingly amount of pre-defined build steps for every language in their step registry.

build:
    steps:
        - npm-install
        - npm-test

You can also submit your own build steps to the step registry, or script them in your build configuration using the script build step like this.

build:
    steps:
        - script:
            name: my custom script
            code: |
                # some command
                # more command

Putting it all togheter

And there you have it; box, services and build steps all go into one file in your project repository named wercker.yml. This is your Wercker build configuration file.

box: node:0.10

services:
    - mongo:2.6
    - redis:2.8

build:
    steps:
        - npm-install
        - npm-test

Getting Started

You can read more about the Wercker platform at devcenter.wercker.com, or check out their GitHub repo werkcer/docs where you can search through all of their documentation which is nicely formatted in Markdown.

Wercker also provides a lot of example projects to get you started. Check them out as well: