Accessibility Testing with aXe-core and WebdriverJS

In this tutorial, we’ll set up automated accessibility testing for a JavaScript project using axe-webdriverjs, a Node.js module making axe-core easy to use with Selenium Webdriver. This tutorial uses the Jasmine testing framework–which could be swapped out for Mocha or another testing solution–but is otherwise framework-agnostic and can be integrated into any project.

Why automated accessibility testing?

Automated testing can free up 30% of your manual testing effort, making it a valuable tool for software development. By scripting a browser to programmatically open web pages and perform user tasks, you can validate features without having to open each page yourself. By integrating axe-core into your tests, you can add coverage for accessibility without having to become an expert. In addition, you can benchmark accessibility support and catch regressions in your builds, preventing broken code from going to production.

Requirements and Setup

To get started, check out the demo for this tutorial from Github. You will need Node.js and npm (Node package manager) installed. Selenium Webdriver can be used to test multiple browsers, including Firefox, Chrome, Safari, Edge and Opera. This demo uses Chrome, for which you’ll need ChromeDriver installed somewhere on your $PATH. [How to install Chromedriver on OSX]

Setting up a test project with axe-webdriverjs is very straightforward using npm. There are a few dependencies in our demo’s package.json file, including selenium-webdriver, jasmine and axe-webdriverjs, which we can install seamlessly using npm install. These modules, along with the axe-core module (installed automatically), give us everything we need to write browser integration tests for accessibility.

To recap, you will need:

• Git
• Node.js
• npm
• Chrome
• ChromeDriver

Github demo: https://github.com/marcysutton/axe-webdriverjs-demo

Install dependencies:

npm install

That’s it for setup!
[embedyt] https://www.youtube.com/watch?v=1QAvRJM-zR8[/embedyt]

Working with Tests

To programmatically test a web page, you can direct Selenium Webdriver to a URL, provide any necessary setup details, and assert that it has no accessibility errors in that state. If your environment already has integration tests, it’s easy to work in aXe since you’ve already got an infrastructure for loading URLs and scripting user scenarios.

Here’s an example of testing a page running on a local web server:

var selenium = require('selenium-webdriver'),
    AxeBuilder = require('axe-webdriverjs');

describe('Accessibility', function() {
  var driver;

  beforeEach(function(done) {
      driver = new selenium.Builder()
          .forBrowser('chrome')
          .build();

      driver.get('http://localhost:4000/')
          .then(function () {
              done();
          });
  });

  // Close website after each test is run (so it is opened fresh each time)
  afterEach(function(done) {
      driver.quit().then(function () {
          done();
      });
  });

  it('should change state with the keyboard', function() {
    var selector = 'span[role="radio"][aria-labelledby="radiogroup-0-label-0"]';

    driver.findElement(selenium.By.css(selector))
      .then(function (element) {
          element.sendKeys(Key.SPACE);
          return element;
      })
      .then(function (element) {
          return element.getAttribute('aria-checked')
      })
      .then(function (attr) {
          expect(attr).toEqual('true');
      });
  });

  it('should analyze the page with aXe', function (done) {
     AxeBuilder(driver)
       .analyze(function(results) {
            console.log('Accessibility Violations: ', results.violations.length);
            if (results.violations.length > 0) {
                console.log(results.violations);
            }
            expect(results.violations.length).toBe(0);
            done();
        })
  });
});

In the test file above, we pull in our dependencies of selenium-webdriver and axe-webdriverjs, allowing us to call functions built into those libraries. An existing integration test checks keyboard and ARIA support of a custom radio button. Using that test’s Webdriver setup along with axe-webdriverjs, a second test calls AxeBuilder to analyze the page with axe-core. Once AxeBuilder has returned its audit results through a JavaScript promise, we can output them to the console.

Harnessing the value of aXe

Integrating aXe into a larger test environment with many developers means you’ll have to do more with the results–simply printing out “expected 2 to equal 0” for accessibility violations won’t provide much insight to a developer who unknowingly introduced regressions. There are a few options for dealing with aXe results data beyond the basics: the WorldSpace Attest Reporting API, which can be easily integrated into your existing test infrastructure; or, building your own functionality to log and then report on those results across the entire test suite once completed.

A useful reporter would digest accessibility test results and display the information necessary to quickly diagnose failures, such as the test name, impact level and relevant markup. A utility for logging the results object during testing may come in handy as well, to provide developers with more information while avoiding repetitive boilerplate code in every test file.

Gotchas

Timeouts

If you are integrating aXe into existing functional tests, you will already be waiting for pages to load and checking for specific UI widgets to appear before testing for assertions. However, if you don’t already have those mechanisms in place, you may run into a few tricky spots. First, there are significant timing differences between local and remote URLs: without any kind of network throttling to simulate real world latency and download speeds, locally running URLs will return much faster than remote ones. If you try testing remote URLs (i.e. https://www.deque.com, versus http://localhost:4000, which would run on only your machine), you’ll see timeout errors left and right. Fortunately, this can be fixed by increasing Jasmine and Webdriver’s timeout intervals.

To increase Jasmine’s default timeout interval, increase it anywhere in your test file:

jasmine.DEFAULT_TIMEOUT_INTERVAL = 60000;

Once you have instantiated Selenium Webdriver, you can also increase its script timeout limit:

browser.manage().timeouts().setScriptTimeout(60000);

Promises

Another big gotcha when working with WebdriverJS is the issue of asynchronous commands in a synchronous environment. To deal with this, you must be comfortable using JavaScript Promises. Every command takes time to execute and you have to wait for confirmation before you can utilize the data returned. You’ll learn to love the return statement so you can keep your continuous stream of data flowing. For more on learning how to use Promises in WebdriverJS, check out my blog post on writing Protractor plugins.

Conclusion

It’s clear that automated testing provides value in software development. By using tools like axe-core, you can get a significant portion of your accessibility testing done without any manual testing. You can start testing for things like color contrast, form labels, or ARIA usage without having to become an expert on implementation algorithms. Further, if accessibility support regresses in your app, you’ll know right away because you’ve already benchmarked the current level of support.

Online course provider edX was able to cut down their manual accessibility testing significantly by integrating aXe into their testing framework. From Mark Sadecki:

“Because axe-core is designed to eliminate false positives, edX can confidently use the ruleset in our acceptance testing framework, keeping some of the most egregious accessibility failures out of our production environment.”

Let automated tests identify the low-hanging fruit with regards to accessible software quality in your project and take a big step towards digital equality!