What to look for in an accessibility audit

If your organization is completely new to digital accessibility, an audit or assessment is a great place to start to see where your organization stands.

What is an accessibility audit?

An accessibility audit is a combination of automated and manual testing done by accessibility experts using assistive technologies in various testing states. A detailed audit is a thorough review of accessibility that provides full coverage, that automated testing alone cannot provide. There are other types of audits, such as validation or risk audits, which I will cover in more detail below.

In this post, I will cover what types of audits exist, what to include the scope of an audit, the logistics of an audit, good characteristics of an audit report, and how to determine if you have a good audit report.

Why Should Your Organization Care for Accessibility?

Before we dive into what to expect from an accessibility audit, let’s briefly discuss why your organization should be motivated to incorporate accessibility at all.

There are multiple drivers for accessibility, one includes compliance and legal risk. Not addressing accessibility can leave your organization at risk for accessibility complaints and lawsuits. In fact, ADA Web Accessibility-Related Lawsuits exploded 181% – from 814 lawsuits in 2017 to 2285 in 2018. They continue to rise from 2018-2019 as customers rely more and more on eCommerce in their daily lives. Real-world numbers from an accessibility-related lawsuit show that legal fees (not including settlement fees) alone could cost you more than $400,000.

Furthermore, having an inclusive website or application can significantly increase your organization’s potential market share. Approximately one in five people in the United States, or 64 million, have a disability. The total after-tax disposable income for working-age people with disabilities is approximately $490 billion. For comparison, African Americans’ disposable income is $501 billion and for people of Hispanic origin, it is $582 billion.

Now that you have an introduction as to why your organization should be proactive about accessibility, we’ll cover the key things to look for when shopping for an accessibility audit or assessment.

Types of Accessibility Audits/Assessments

There are many types of audits, and which audit is right for you depends on what you want to test and what the end goal of your audit is. For example, if your organization wishes to focus solely on compliance, a risk audit could be right for you. A risk audit tests for severe barriers that could potentially lead to litigation or should be covered under industry regulations.

Below are the definitions of Deque’s audit offerings:

• Level-of-effort (LOE) Audit: A report which estimates the magnitude and cost of an accessibility remediation project. The report includes the number of pages to fix and the number of defects on each page.
• Risk Audit: Identifies severe and critical blockers that users with disabilities would encounter. This report does not include remediation recommendations.
• Detailed Audit: Identifies improvement based on the client’s preferred standards (WCAG 2.0 A/AA, WCAG 2.1 A/AA, or Section 508) using automated and manual testing that covers the scope of the client’s choosing. This report provides remediation recommendations.
  • Detailed audits are also available for specific regulations or technologies – such as CVAA, Kiosks, and native mobile apps.
• Screen Reader Acceptance Testing: Experts will test a task/use case/user flow for assistive technology/browser/OS version combinations of the client’s choosing and provide a rating on a scale of difficulty or failure.
• Validation Audit: Performed on a web page, set of web pages, or applications that have previously undergone an audit by Deque. Validation audits may also be done on mobile apps or PDFs too.
• Usability Testing: Accessibility (conformation to WCAG or Section 508) does not always lead to usability. Usability testing reveals what’s usable to people with disabilities.
• Accessibility Conformance Statements: A Conformance Statement is a document from a trusted third party that details the level of accessibility for your organization’s website or application.
• Voluntary Product Accessibility Template (VPATs): A procurement report required for selling web-based software to the US Federal Government under Section 508.
• Design Audits: A Design Comp Accessibility Annotation (DCAA), is a markup of UX and UI wireframes and comprehensive designs (comps) with accessibility requirements for developers.

If you’re not sure which type of audit is right for you, don’t worry. Our experienced team at Deque will help determine what’s right for you based on your goals and budget.

What to Include in the Scope of an Accessibility Audit

Accessibility can apply to multiple digital assets and interfaces. While most focus on the web, such as a webpage or website, digital assets also include documents, video, audio, desktop software, and native mobile apps. Accessibility can also apply to hardware such as ATMs, kiosks, phones, and point of sale devices. Lastly, voice assistants such as Alexa, Siri, or Cortana should be accessible, too.

Ultimately, if it’s digital it can be tested for accessibility. A common starting place for accessibility is the web because it’s open, customer-facing, and public-facing. Oftentimes, websites include your organization’s core surfaces and functionalities. The scope of the audit should include the types of interfaces an accessibility partner will test.

Furthermore, the scope of an audit should include what browser(s) you wish to test in and with what assistive technologies. To get the most value for your money, you don’t need to test every possible assistive technology on every type of browser.

A common smart choice to get the most bang for your buck is to focus your test on a Windows PC running the NVDA screen reader and Chrome. A great accessibility partner should always be willing and able to test with any combination of OS, assistive technology, and browser you can imagine. But consider the cost and ROI that works best for your situation. What I’ve seen in my 20 years of experience is for web accessibility using Chrome/NVDA  as the primary test platform, you’ll catch more than 90% of your issues.

Lastly, you’ll need to determine which pages and screens to test. If this is your first audit, you may be inclined to test every page and screen to get the full picture of where your organization stands. However, the best ROI and wisest thing to test for in your first audit are your site’s key entry points, core paths, highest traffic pages, and most critical user flows. Knowing if you have critical accessibility issues in your core content/functionality, and fixing these issues will greatly increase the usability of your site for users with disabilities.

A common critical user path is adding something to your cart and checking out:

Which Accessibility Standards is Right for You?

In order to begin an accessibility audit, it’s very important to know what standard you need to test against. The primary driver of the accessibility audit is going to determine your testing standards.

If your primary business is selling into one particular vertical market, then that vertical market may ask you for a specific level of compliance to a standard that exists.

Furthermore, the standard you choose applies to what level of accessibility compliance your organization wishes to accomplish. What most organizations don’t know is that you don’t have to make this decision on your own. A good accessibility partner will take out the guesswork and assess what standard, and therefore, determine the type of audit that is the best fit for your organization based on their years of experience.

Lastly, if the audit is driven by a complaint or a lawsuit, the standard may be referenced in the briefing. Be sure to review the legal documents before deciding which standard to test against.

The most common and widely-accepted standard to test against is WCAG, a.k.a. Web Content Accessibility Guidelines. This standard created by the World Wide Web Consortium (W3C) defines technical guidelines for creating accessible web-based content. WCAG serves as the basis of accessibility regulations across the globe including the US, Canada, the UK, the EU, Australia, and Japan.

WCAG Success Criteria are broken down into different “levels of conformance”: A (basic conformance), AA (intermediate conformance), and AAA (advanced conformance). The current standard for compliance is both WCAG 2.1 Level A and AA.

Level A is the lowest level of conformance. It helps with many barriers for people who have blindness, deafness, or motor disabilities. Level AA picks up some very important barriers for low vision and has a tiny bit of help for people with cognitive disabilities while also being inclusive of Level A conformance requirements.

If you don’t have specific accessibility regulations that apply to your organization but want to avoid legal risk, WCAG 2.1 A and AA compliance is a reasonable standard to adopt.

The Logistics of an Accessibility Audit

If accessibility is completely new to your organization, having an accessibility expert or partner perform your audit is the best approach. There are proven methodologies for measuring accessibility that requires both manual and automated testing. Regardless of the standard, you are testing against, manual testing is always required to determine full compliance.

A great accessibility partner will test with a proven methodology that is accurate, consistent, and comprehensive.

When choosing an accessibility partner, it’s important to ask about the company’s staff certification and level of expertise. The accessibility world has grown, and with so many consultancies it’s important to know who the real experts are. Be sure to ask:

• What are your staff certifications?
• What are the credentials behind the team that is actually doing the testing?
• How many audits have you done?
• What is your QA process for audits?
• How long have you been in the industry?

Because of the legal drivers that were mentioned previously in this blog post, accessibility is a growing market. It’s really important to make sure that the professionals you’re using have the depth of experience, are staying up-to-date with innovations and are committed to helping you become self-sufficient in resolving accessibility issues, and can provide you the knowledge to actually work with you after the audit results have been presented.

Ideally, you want an accessibility expert that can help your organization develop a sustainable and proactive accessibility program so your digital assets are born accessible.

What are Good Characteristics of an Audit Report?

Delivery of an Audit

If an Accessibility Partner understands your goals, they will also know how to deliver the results of the audit to you and give you the proper guidance for remediation. If an audit is the result of an accessibility complaint or lawsuit, you’ll also need to receive your results quickly. At Deque, large accessibility audits take two to three weeks.

An accessibility audit can also be iterative, i.e. delivered in an agile approach for those who wish to integrate testing and validation into their software development lifecycle (SDLC) for accessibility.

Format of the Audit

An excellent audit report will include an executive summary dashboard, specific issues found, impact on people with disabilities, and how to fix the results. Your organization should be able to look at the report and see how many of the required standards of WCAG 2.1 Level A and AA you passed (some only show what issues you fail).

It’s also important that the audit report illustrates the issues from a user impact perspective because not all accessibility issues are not created equal. For example, the report could show the impact on a five-point scale, which is what we offer here at Deque: blocker, critical, serious, moderate, and minor. This description helps organizations prioritize issues when they’re performing remediation.

Deque’s audit reports provide links to learn more repositories that help your team learn while they remediate, so your audits can act as a library for learning after it is done.

Lastly, a good audit will also point out the top types of issues and which categories they are from. Having this dashboard view is great from a leadership prioritization standpoint. Additionally, your audit report should not only show you the specific issues found, but also which WCAG requirement specifically failed, a description of the issue, how to fix the issue with code examples, and the ability to see the issue in a browser (only valid for web audits).

In addition to providing a dashboard with high-level statistics and test results that allow you to drill down into details, Deque provides an executive summary report with each audit. This report highlights the following for your executive team:

• Testing methodology
• Statistics on total Issues, total blockers, percentage of issues found by automation, user impact by severity (including definitions for severity type)
• High-level details regarding highest user impact issues
• Next steps and where to go after the audit
• Number of issues (by severity) by page/component

All of these deliverables will make it much easier to act actionably on the results that are provided in order to reach your compliance goals.

How Do you Know if You Have a Good Audit Report?

First and foremost: not all accessibility professionals are equal. A good qualifier of this is how long the professional has been in the business. Be sure to also ask them where their knowledge comes from and what certifications they hold, and how they keep up-to-date with changes in the field. At Deque, our experts have years and years of experience in applied theory, answering difficult questions, participating in the W3C standard working groups which create the accessibility standards and required professional development to increase our knowledge

Ask your potential partner to comment on their years of experience to ensure you’re not dealing with a new accessibility vendor so you don’t get caught with a half-baked audit report. Deque has the largest, most experienced, most certified accessibility business in the world. Our experts even have regular competitions to make sure we’re doing quality work for our clients.

As Deque’s Director of Accessibility Services, I ensure that at Deque we have effective, accurate, and consistent results. To do this, we:

• Clearly define the exact accessibility standard we’re testing for each project.
• Understand the difference between normative and informative.
  • Normative, if you’re not familiar with the word is required to conform. When you’re new to the standards, especially to WCAG 2.0 or 2.1, it’s easy to misunderstand which pieces of WCAG are the requirements versus which pieces are just informative, i.e. not required. I see a lot of experts that think they understand standards, but they actually over call and state that best practices act like requirements. That is incorrect, as an expert should never want to tell you that a best practice is a requirement when you might have a very limited budget with a crazy timeline.
• Have a detailed testing methodology with step by step instructions on how our accessibility experts find the issues, so we have repeatable results.
• Have an issue description library that’s been vetted by certified experts – describing what is failing the requirements or what is a suggested best practice. Our library allows us to have consistent experts and therefore results, but it also acts as a tool for our clients.

In Summary

An audit is a great first step to any organization’s accessibility journey.  The quality of an audit will vary greatly from one accessibility vendor to another.  We want to make sure that the audit you receive is accurate and actionable.

• On a strategic level, your executive leadership should know the high-level state of accessibility (for the digital asset tested), the impact of issues on people with disabilities, and suggested next steps for resolving the highest priority issues.
• On a tactical level, you should know what accessibility issues you have, how to recreate the issues, guidance on how to resolve the issues, and how to validate that you’ve truly fixed an issue.

When choosing an accessibility partner, it’s important to have the right information and to ask the right questions. We hope this post has given you the foundation for what an audit is and what to expect. Feel free to contact us for more information on audits and remediation services that Deque provides.