Accessibility APIs and end-to-end testing are a great way to catch up to 50% of your organization’s accessibility defects. But what if your organization doesn’t have automated testing built out? How can you test for accessibility defects while you code?
Furthermore, end-to-end testing doesn’t test for specific user flows. How can you make sure you’re testing the user flows which are most important for accessibility compliance? In this blog post, we’ll cover these important questions and dive into the details of the benefits of end-to-end testing vs. scripted user flow testing for accessibility.
Benefits of End-to-End Testing for Accessibility
End-to-end tests are a type of integration test that test a “flow” of an application from start to finish. The purpose of carrying out end-to-end tests is to identify system dependencies and to ensure that the right information is passed between various system components and systems.
Note: The use of the word flow in this context is different from user flow, for end-to-end testing, it means from start to finish of an application. A Critical User Flow is a specific user feature, function, sequence, or key path you wish users to take on your website.
If your team is already doing automated testing in a CI/CD pipeline such as Selenium, Cucumber, or Jenkins, etc. the axe DevTools accessibility API seamlessly integrates into the environment of your choice. End-to-end testing for accessibility is great for preventing and fixing accessibility defects as you code.
Here is an example of axe DevTools in multiple test environments:
Benefits of Scripted User Flow Testing
By monitoring critical user flows, you will be able to remediate the most important parts of your site that are inaccessible. The most common example of a critical user flow is someone adding an item to a shopping cart and being able to successfully check out.
As previously mentioned, end-to-end testing monitors how an application works by static individual scans or by testing a flow from beginning to end. To test for critical user flows you need to implement unit tests for accessibility.
Now, not every organization has unit testing setup in its test infrastructure. At Deque, our solution for this is called axe DevTools CLI, a component of axe DevTools. Axe DevTools CLI is a scripting tool that allows for testing multiple pages (similar to end-to-end testing) and critical user flows for accessibility issues.
In more technical terms, it is a JavaScript API and a CLI for fetching and scanning web pages for accessibility issues using the axe DevTools rules engine. Axe DevTools CLI will also provide a detailed report of those issues, compared to some scanning tools which only point out what is broken in one large report.
Manual Testing vs. Scripted User Flow Testing
Automation will cover about 50% of total accessibility issues. The rest needs to be covered with manual testing. Manual testing involves testing apps or web content with assistive technologies such as NVDA, Voiceover, or by keyboard only, to name a few.
Another great feature of axe DevTools CLI testing is that it allows you to test for specific interactions that previously could only be done through manual testing. For example, you could record a script for a specific user interaction or pattern, such as a screen reader user interaction, so that you could automatically re-run it on the page to avoid multiple tests for manual testing.
How to Identify a Critical User Flow
As previously mentioned, a critical user flow is the most common use case pattern for users on your site. If you’re a retail site, what is the average path someone takes to add something to their cart? If you’re a banking site, what is the direct line to logging into their account?
There are different ways to identify a critical flow. To do this, organizations need to monitor how your users navigate and interact with your site.
To identify critical flows organizations should:
- Track the top 15-20 pages for traffic
- Identify any interactivity patterns that exist within those pages itself
- Watch how users navigate and interact with your site in user testing
- Gather customer evaluations and feedback
Axe DevTools CLI
Now we’ve talked about how to identify critical user flows, let’s look at some code snippets of axe DevTools CLI:
Example spec file that Axe DevTools CLI takes to scan multiple pages:
{
"projects": [
{
"name": "abcdTech Static Scanner",
"id": "abcTechStatic",
"pageList": [
{
"name": "abcdTech homepage",
"url": "http://abcdcomputech.dequecloud.com/"
},
{
"name": "abcdTech Search Results",
"url": "http://abcdcomputech.dequecloud.com/laptopsandnotebooks.php",
"actions": [
"type \"a11y\" into element \"input[name=q]\"",
"click element \"input[type=submit]\"",
"wait for element \"#container\" to be found",
"analyze with title \"search results\""
]
},
{
"name": "abcdTech Desktops",
"url": "http://abcdcomputech.dequecloud.com/desktops.php"
},
{
"name": "abcdTech Support to Cart",
"url": "http://abcdcomputech.dequecloud.com/support.php",
"actions": [
"wait for 5s",
"analyze with title \"Support Page\"",
"click element \"#topnav li:nth-child(4) a\"",
"wait for 5s",
"analyze with title \"Cart page\""
]
}
]
}
]
}
Here’s an example of how to execute a command to run spec command:
aget spec ./specs/abcTechStatic.json ./results/abcTechStatic
This command takes the JSON spec file and will scan each page with axe DevTools and then return the results from the scan to the specified folder. When this runs, a JSON file and an HTML report are created for each page.
In Summary
End-to-end testing with accessibility APIs is a great way to catch accessibility defects while you code. But if your organization doesn’t have an automated test environment set-up, axe DevTools CLI is a great solution for running scripted user flows for the most important features of functions on your website or application. Even if you do have end-to-end testing set-up, it’s still a good best practice to test these specific user flows using this methodology. This scripted use flow feature can also provide you more coverage as it can run scripts simulating assistive technologies.