Continous Integration, Grunt, Javascript, Tests, Xvfb

Running UI tests on real browsers in continuous integration using X virtual framebuffer (trough a task-runner)

Many times when projects run UI tests as part of CI (continuous integration) they rely on PhantomJS, a great headless browser. Unfortunately, using PhantomJS has some drawbacks:

  1. You really don’t cover quirks happening from browser to browser (working on chrome, not on FF, working on FF, not working on IE), so you might actually ship a buggy screen to production
  2. Debugging is hard. Although PhantomJS runs on top of WebKit, it has it own quirks, so you might get failing tests while all of your browsers show that everything’s OK. Debugging something you can’t see is troublesome.

UI tests with real browsers

So you’re not completely satisfied with PhantomJS and you want to run your UI tests on real browsers, which is a good idea that have a little difficulty (or not): you want your UI tests to run as part of CI, but most of CI servers don’t have displays.

Enter Xvfb (X virtual framebuffer). From Wikipedia: Xvfb is a display server implementing the X11 display server protocol. In contrast to other display servers, Xvfb performs all graphical operations in memory without showing any screen output. From the point of view of the client, it acts exactly like any other X display server, serving requests and sending events and errors as appropriate. However, no output is shown. This virtual server does not require the computer it is running on to have a screen or any input device. Only a network layer is necessary.

So basically what you’ll need to do is:

  1. Start Xvfb
  2. Run your tests
  3. Kill Xvfb

Running Xvfb trough a task runner

In this example we’ll use Grunt to run Xvfb as one of the tasks, but this is also possible with Gulp (I’m not sure about the others).
First, we’ll install the required packages, make sure you’re not forgetting to –save-dev since we want to update our package.json with the new dependency for development.

npm install --save-dev grunt-env
npm install --save-dev grunt-shell-spawn

Next we’ll setup grunt tasks for Xvfb:

        shell: {
            xvfb: {
                command: 'Xvfb :99 -ac -screen 0 1600x1200x24',
                options: {
                    async: true
        env: {
            xvfb: {
                DISPLAY: ':99'

Now we’ll setup the UI test task, for the example we’ll use protractor but it can be any other library.

protractor: {
            options: {
                keepAlive: false,
                configFile: "protractor.conf.js"
            run: {}

In protractor.conf.js we should create configuration to run UI tests on the required browsers (I assume you have already the proper setup for this, since it’s not in the scope of this post).

Now we got:

  1. Grunt test task configuration
  2. Protractor configuration which will run the tests on all the required browsers
  3. Grunt Xvfb tasks configuration

Let’s create a task to combine everything together:

grunt.registerTask('CI-E2E', ['shell:xvfb', 'env:xvfb', 'protractor:run', 'shell:xvfb:kill']);

After this, a can simply run


and all your UI tests will run on top of Xvfb. That’s it.

4 thoughts on “Running UI tests on real browsers in continuous integration using X virtual framebuffer (trough a task-runner)

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s