Accessibility Program Fundamentals: Choosing the right accessibility metrics

Choosing the right digital accessibility analytics is critical to the long-term success of your accessibility testing efforts. To hold weight, these metrics need to be both; actionable and relevant. Let’s break down a collection of appropriate accessibility metrics for IT management, central accessibility function/teams (if you have one), senior management and execs.

Defining Actionable Metrics

Your accessibility metrics should help break accessibility into achievable, measurable milestones. Remember, quick and easy wins stack up over time, and can provide the justification needed for continued investment and interest in practicing accessibility. If you work outside the development team, it’s also important to remember that the devs will ultimately be responsible for fixing your accessibility issues. Don’t overburden them with unstructured actions.

1. Do: Ensure your code is clean of all automatically detected issues from a testing tool, such as Attest extension.
2. Don’t: Aim for 100% accessibility conformance, covering all WCAG 2.1 success criteria right out of the gate.

Defining Relevant Metrics

When choosing your organization’s accessibility metrics, you should consider the audience you are presenting the metrics to, and the accountability that comes with them. You might be presenting metrics to your IT management, central accessibility function/team, senior management team, or executives– and each group will have different concerns and key performance indicators.

1. Do: Ask Developers to measure the average time it takes to correct an accessibility defect from the time it was submitted as a ticket.
2. Don’t: Ask Developers about measuring WCAG success criteria coverage.

Here are a few more “Do’s” for each audience:

Central Accessibility Function/Team

• Accessibility issues by severity: With this data, your central accessibility team can assess the current state of your risk and provide strategic next steps.
• Accessibility issues by category: e.g. ARIA, color-contrast, forms. This can help your team identify opportunities to improve coding and/or design practices.
• Accessibility trends over time: Are problem or high-risk areas improving? Getting worse? Are you reducing your number of issues overall?
• WCAG success criteria passed during initial tests at all phases: This will help indicate when your dev team is creating accessible by default and understand how well they are applying accessibility practices.
• Accessibility training completion rates: Who is completing their training? Who is struggling to make time for training? Where are the gaps in training or lessons that need extra attention? When is new training required due to technology changes?

Product Owner/IT Management

• Accessibility issues by the development team: See which teams are doing well and which could use some extra help.
• Remaining Accessibility Work: For example, the number of issues waiting to be fixed. If you’re addressing certain groups of accessibility issues per sprint, the work remaining should measure the targeted groups of accessibility issues.
• Net daily number of accessibility defects: How many new issues are being introduced each day? Ideally, this number would go down.
• Accessibility defects by priority: Help your team focus on top priority issues.
• Amount of time it takes to correct a defect from JIRA: A.k.a.remediation response time. These numbers can help draw attention to the amount of attention accessibility issues are getting and how seriously they’re being taken.
• Mean Time Between Failures (MTBF): How often serious defects make it into production.
• Mean Time to Repair (MTTR) – how long it takes to correct serious defects in production

Senior Management

• High-level summary of critical issues: Give them a sense of major issues without going too deeply into detail.
• Accessibility Score: There are tools that will provide scores, and managers and legal teams sometimes find them helpful. This can be grouped by product.
• Trends in your accessibility compliance over time: What issues are decreasing? What problem areas are seeing improvement? How are your teams progressing in their training/understanding?
• Time until remediation is complete: Based on your other data, you should be able to give a reasonably good estimate.
• Level of Effort (LOE) and dollar estimates for top critical barriers: Help these stakeholders understand the anticipated impact.

Executives

• Risk assessment
  • No serious defects are knowingly put into production through the exception process
  • No serious defects are found in periodic production scanning
  • Targets are met for accessibility health in periodic production scanning
  • Costs of digital accessibility integration are reduced and contained
• Costs of digital accessibility integration are reduced and contained
  • Downward trajectory in the year to year investment to a stable level
  • Downward trajectory in SDLC time commitment to a stable level
  • Downward trajectory in reliance on external consulting to self-sufficiency
  • Downward trajectory in size of centralized accessibility function to a stable level

Results Your Organization Should See as Accessibility Improves

As your development teams start improving and fixing accessibility defects and your central accessibility team measures risk and improve training, accessibility at your organization should start to yield the following results:

• Costs of digital accessibility integration are reduced and contained
• Year-to-year investment for accessibility has a downward trend
• Development cycle time towards accessibility remediation has a downward trend
• Reliance on external consulting and outsourcing has a downward trend
• The size of the central accessibility function has a downward trend

All of these results, measure with the metrics, illustrate how your organization’s accessibility practices are becoming more and more efficient and sustainable. The more you invest in accessibility as a program and a culture shift, the more your organization will save on accessibility in the future.

In Summary

Defining the right accessibility metrics lay the foundation for an efficient culture of accessibility. The right metrics will provide the necessary insight to target and tackle a backlog of accessibility issues while prioritizing compliance and improving the accessibility skills of your development teams. With the right metrics and success criteria, accessibility can be both cost-effective and sustainable.