Why you need to monitor and report on accessibility—all the time

Aren’t you “done” with accessibility when you fix all the issues in an audit? While an accessibility audit is an important first step, the truth is that you’re just getting started once you finish your audit. You now have to maintain the progress that you’ve made with your website and application.

We live in a digital world that is ever-changing and evolving, so the accessibility of websites and applications must keep pace. Experts in the industry like to say that accessibility is a journey and not a one-and-done project.

In this blog post, I’ll cover the basics of accessibility monitoring and reporting so you can maintain your digital properties efficiently and at scale. We’ll review:

• Why do you need accessibility monitoring and reporting?
• How to monitor: Automated crawlers vs. manual audits
• What should be monitored
• Intro to accessibility reporting

Why do you need to monitor and report on accessibility?

Legal and Compliance Requirements

We are at a regulatory inflection point where every organization can be the target of an accessibility complaint or lawsuit. In a report published by Seyfarth Shaw, digital accessibility lawsuits in 2021 increased by 14% from 2020.

Your C-Suite, legal and compliance team, and product owners will want to mitigate this risk. If you are in the middle of a settlement or have a consent decree already in place, you’ll need to know how accessible you are at all times.

Not being accessible can result in very public disclosures that harm your brand, but there are also significant financial consequences attached to lawsuits. The 2008 Target settlement resulted in a $6 million class damages fine, and that was not the only cost. Target had to pay upwards of $3 million in plaintiff legal fees on top of their own defense legal costs— which they have not disclosed.

Create an Accessibility Backlog

Monitoring and reporting helps your development teams. They won’t be able to fix anything without knowing the full scope of issues. If you’re unfamiliar, a backlog is a term used by development teams for a list of tasks that they need to complete.

Development team leaders need a summarized version of issues so they can prioritize teams’ work. Keep in mind that accessibility issues are not the only things dev teams are working on.  There are other issues to deal with and they’re under constant pressure to deliver new functionality. Dev team leaders should also be looking at the backlog for opportunities to prevent the same issues from reoccurring and plan appropriate training for their teams.

Developers themselves need the issues with all the details so they can address them quickly and accurately. Most importantly, they want to ensure they’re not fixing things that aren’t really broken. We call them “false positives” here at Deque and we’re committed to returning zero “nonissues” in all of our products.

One of our customers found that false positives actually increased the cost of development by 386%. So it’s very important, for budgeting purposes, to have the most up-to-date list of issues so your organization isn’t wasting valuable development resources. While this is a stunning number, it’s not unusual. Think about how much time it takes your developer to discover that the code is OK. Having the most accurate reporting possible is critical to your development velocity.

Transparency and Enablement

Your teams need accurate, up-to-date reporting to enable them to improve and learn. Monitoring and reporting enables team leaders to develop high-value training programs and the opportunity to give kudos to teams that are doing well.

For example, the image above shows an accessibility score in axe Monitor, Deque’s monitoring and reporting solution. You’d want to congratulate the team responsible for “Sports” for their significant improvement. The team responsible for “News” may need an action plan as their overall score is decreasing over time.

How to Monitor for Accessibility

Automated Accessibility Crawlers – Pros and Cons

Before we get into what should be monitored, let’s first review the different testing techniques for accessibility. The first technique is monitoring with automated scanners. In this case, we’ll focus on a crawl scan.

A crawl scan begins with a defined URL. The crawler navigates to the provided URL, performs a scan for accessibility, identifies all the links from that page to other web pages, and then traverses the entire web application by navigating to the linked URLs and performing additional scans for accessibility on each page. These scans are able to identify the majority of accessibility violations present on each page.

Using an accessibility crawler is a very quick and automated way to perform an extensive website scan and find the low-hanging fruit. Testing quickly and at scale is the primary benefit of using an automated crawl scan. Modern websites have thousands and thousands of pages. It’ll take thousands of hours to test these manually, whereas it takes minutes to test with a crawler, saving precious time for your monitoring and testing team.

Additionally, many pages on these websites are built from a content management system (CMS). In other words, they’re actually not coded in on the page, but rather the code on the page is a placeholder that is filled in with content from a CMS. But, the only way to test this content is in production, where all the content finally appears.

It’s important to note there are limitations to automated testing. There are specific issues in Web Content Accessibility Guidelines (WCAG) that cannot be tested for by an automated crawl scan. An example of this is alternate text for website images. It’s easy for an automated crawl scan to test whether the image contains alternate text or not, but how do you validate whether the alternate text is appropriate? That requires a manual check.

The inability to scan dynamic site content is another limitation of automated crawlers. This is content that presents itself after a specific action or set of actions from the user. Retail websites are good examples of this scenario. Sometimes the options available for a shoe change based on the size selection. These options will get exposed to the user only if that particular size is selected. An accessibility crawler usually cannot scan this content.

Lastly, automated crawlers are robots. Websites are very often coded for robots not to be able to access them for security. There are ways to remove that limitation, but it requires development work at either the website or firewall level to allow the robot to come in and scan.

Manual Accessibility Audits – Pros and Cons

The second type of testing is a manual accessibility audit. Everything that was not found by the automated crawler will be covered with a manual accessibility audit. This is the only way to get to one hundred percent compliance.

Manual testing covers dynamic content because someone is actually sitting and driving the page. It also overcomes multi-factor authentication hurdles because someone actually looks at the authenticator app and punches in the authenticator’s code.

However, manual accessibility audits are very high-effort. And, the effort is linear, requiring the tester to go page by page. If one page takes three hours and you want to test 10 pages, the effort will be close to 30 hours.

The other limitation is the knowledge and the skill required. People who can do manual testing are almost always experts in the WCAG. Manual accessibility testers must also know how to interpret the guidelines to apply them correctly across different types of content. How does a guideline apply to a website versus a mobile site versus a document? There are a lot of nuances to completing that “last mile” of testing so deep understanding and expertise is critical.

What should be monitored for accessibility?

1. Webpages: Scan all known breakpoints, all views, and all static and dynamic content
2. Mobile content: For example, Native iOS and Android apps
3. All content types: For example, PDF files, Microsoft Word files, and spreadsheets
4. Multimedia. For example, videos, audio, GIFs, etc.

The bottom line? If it’s hosted on your website, you’re responsible for it—even third-party widgets or content!

Accessibility Reporting

Now let’s look at what metrics should be included in the reports. The answer is tricky because different roles look at the results for different information. So let’s go back to our first section where we talked about the use cases for accessibility monitoring. Different use cases call for different reports.

Accessibility Reporting for Legal and Compliance Teams

Legal and Compliance teams want to monitor risk, meet consent decree requirements, and prevent legal action. This is not an exhaustive list, but they are the top metrics based on our customers’ experience.

The first metric this team cares about is an overall score. How you create this score depends on a lot of factors, but we’ll go into that in more detail later on.

Secondly, they care about the number and severity of the issues–whether it’s a critical issue, serious issue, moderate, or minor based on what standards you use in your organization. The most common accessibility compliance standards are WCAG 2.0 Level A/AA and WCAG 2.1 Level A/AA.

If their report’s purpose is to meet consent decree or settlement requirements, the settlement statement or the consent decree itself will determine what needs to be in the report.

We recommend tracking the following information on a regular basis. Usually monthly, but it depends on the organization and your release cadence. An example report to your compliance team could be the following:

Let’s talk about the accessibility score: This is a contentious topic because there’s no standard for how to create a scoring algorithm. For example, if you get zero defects from an automated crawler, does that mean your score should be 100%? Maybe, maybe not. We know that crawlers don’t account for manual issues. OK, then what should you use?

There’s no right or wrong answer here. Some approaches we have seen used successfully:

• Assign each property a score based on the percentage of WCAG Success Criteria passed/failed
• Use severity of defects on the property as a weight to derive a single number for all pages
• Use the popularity of the page as a weight to derive a single number for all pages

You can get as detailed as you want in determining the ‘right number’. At the end of the day, it’s what works for you. The benefit of having a single score is that it helps you establish a single baseline for your entire organization.

Accessibility Reporting for Development Teams

Development leadership, like product owners and managers, also need accessibility scores as they help to prioritize resources. Additionally, they need the total number of issues by category and by severity to focus their developers on the highest-impact issues first.

Furthermore, they need to know the issue details to create good tickets for the developers to address. It’s also important for leaders to know issue age, oftentimes the older the issue, the higher priority it gets. Lastly, it always helps to have an organizational target for what the score should be.

Developers will need very specific information in their reports. To perform remediation and fix issues, they need the following details:

• What page the issue is on
• What is the issue type
• The issue description
• The issue severity
• Source code where the issue was detected
• WCAG success criteria for issue
• Suggestions for how to fix the issue
• Screenshot of where the issue is

Clearly, even within a single development organization, different roles need different information.

Accessibility Reporting for Leadership

An accessibility report for leadership focuses on summarization. Here is a good list of information to include:

• Key metrics (KPIs)
• Scoring trend (how are we doing over a period of time)
• Total number of issues by category
• Total number of issues by severity
• Drill-down by accountability chart
• Issue age

This information will enable your leadership team to manage the overall accessibility initiative accordingly.

In Summary

Looking back at the different ways and reasons to monitor and report on accessibility, it’s clear that having a comprehensive monitoring system can save significant time and resources. But, to reach full compliance, manual testing is still required to complete that “last mile”.

And, a comprehensive monitoring and reporting solution also supplies accurate, detailed, and regular metrics for:

• Compliance teams: To limit any potential risk
• Development teams: To drive accessibility remediation and improvement
• Leadership: To help manage ongoing accessibility efforts.

Axe Monitor is the industry’s most advanced production accessibility monitoring and reporting solution. It automatically scans complex interactive business flows as well as traditionally crawlable content and documents, delivering comprehensive, enterprise-level accessibility scans of production websites, and providing extensive user-centric reporting options from the same core data. Ensure that your complex, dynamic web properties become—and remain accessible. Learn more about axe Monitor or contact us for a demo today.