Automated Testing as Part of Your 360-Degree Approach to Web Testing

360-Degree Web Testing: Post 2 of 4

How do you test for web accessibility?  Whether you are testing for WCAG 2.0, 508, or any of the other accessibility standards used internationally, you will benefit from using a mix of manual and automated testing techniques, applied regularly and consistently.

While manual testing can yield detailed and accurate results, the cost of this method is high, and the shelf life of the testing results can be extraordinarily short.

On average, experienced web accessibility experts conducting manual tests spend anywhere from 1 – 3 hours doing a thorough examination of a page for violations of WCAG 2.0.   Factors that affect the length of the review include complexity of the page, number of violations on that page, requests to identify every single violation on every page, and misleading browser and assistive technology bugs.  In addition to discovering each issue, the expert is often asked to clearly explain, in writing, exactly where the problem is located and how to resolve it.

Now imagine a project where the objective is to manually review 100 pages on a specific site.  The client has a modern, dynamic and interactive site that is updated regularly with new content.  The release cycle for infrastructure changes to the site is based on an agile timeline, so changes to the underlying template and code happen as frequently as every 4 weeks.

• Approximate hours for manual accessibility review for 100 pages = 100-300 hours
• Potential shelf life of that review – 4 weeks

Clearly this approach is not feasible for most organizations.

So, how do you balance practicality with accuracy?   The smart solution is to use automated accessibility scans to monitor the breadth and depth of your site(s), coupled with focused manual testing on key representative pages.  In other words, use the power of automation as a proactive warning system and use manual testing to confirm that key users paths are truly accessible.

What is the right mix of manual testing (high touch) and automated testing (high tech)?  Based on my experience, the most effective and efficient use of your resources is to employ the 80/20 rule.  80% of your testing effort should be automated and 20% should be manual.  I recommend monthly, automated scans for unambiguous or explicit accessibility issues.  These smart scans will alert you if accessibility testing was skipped during development, or if  obviously inaccessible third party content was added.

What are the top 5 explicit accessibility issues that can be identified with an automated scan?

Running a monthly scan across your site to identify explicit accessibility issues is one of the very best methods for detecting accessibility problems when you have limited resources.  These monthly scans will help you get your head turned in the right direction so you can focus your detailed manual testing efforts in the smartest way possible.  An approach that combines robust automated testing with select manual testing is an approach that is practical and realistic for organizations to implement.

Remember, what gets measured gets done.  If you don’t monitor the accessibility health of a web site on an ongoing basis, it will decay.

This post is the second in a series on what we call “360 Degree” web testing: the approach that Deque advocates for obtaining the most detailed and accurate sense of a site’s accessibility.

Future posts in this series will include:

• A piece on where manual testing is indispensable.
• A miniature case study that puts the entire 360 degree approach together in a story of how a customer used this approach to strengthen the level of accessibility their organization achieved.