Caveat #1: This blog post assumes you are familiar with the language and practices of usability testing. If you are not, here are some great resources:
- Observing the user experience – Mike Kuniavsky
- Handbook of usability testing – Jeffrey Rubin
- Just Ask: Integrating Accessibility Throughout Design
Caveat #2: While this blog post covers usability test considerations for accessibility as a whole, the majority of examples are related to assistive technology (AT) users.
Accessibility is a sub-discipline of Usability. If accessibility is a consideration for the application you are creating, then your usability test should reflect that. You’re probably familiar with running usability tests for sighted/mouse users, this post will discuss considerations for accessibility for non-sighted/AT users when conducting a usability test.
Considerations for accessibility when designing a usability test
1) Create a recruitment plan
Having the right users for your usability study is crucial to its success. Recruiting can be tedious, but it is foundational to conducting a usability study. When recruiting for an accessibility usability test, it is imperative to consider the following types of participants to holistically test for accessibility.
Here is an example of a usability test for accessibility that includes the following disability types:
- Visual – legally blind and low vision
- Auditory – legally deaf
- Motor – heavy usage of keyboard and/or other peripherals.
- Cognitive – focus, perception.
- Speech – unable to use voice
When you’re choosing the number of participants for your test, it is important to have an appropriately representative sample based on the medium or artifact you are testing (i.e. the number is a sliding scale). Below is a sample recruitment plan illustrating which categories of disability to include in your usability test depending on the medium or artifact you are testing:
Examples:
- Testing a media application (e.g. YouTube) – Visual, Low Vision, Cognitive (focus), Cognitive (perception), Motor & Auditory.
- Online Retail (e.g. Target) – Visual, Low Vision, Cognitive (focus), Cognitive (perception), Motor & Auditory.
- Gaming company (e.g. Nintendo) – Motor, Visual, Low Vision, Cognitive (focus), Cognitive (perception), Auditory. For example, if you are designing a/for a rumble pad and the game itself (i.e. you are testing both hardware and software).
| Category | Test Subject Parameters | Disability | Notes |
|---|---|---|---|
| Visual (Blind) | Screen reader only (does not use vision for tasks) |
|
By testing with a screen reader who does not use vision for these tasks, we efficiently and effectively cover all three of these use cases. |
| Low Vision | Screen Magnifier User (with visual acuity between 20/70 and 20/200) that uses 4x to 6x magnification. | Visual acuity between 20/70 and 20/200 in the better-seeing eye with best conventional correction. | |
| Auditory | User who relies on captions/transcripts (not audio) |
|
Only required if there is audio content |
| Motor | Sighted user who only uses a keyboard. User with a tremor frequency of multiple tremors per second in the body part that is used to interact with technology |
|
|
| Cognitive (Perception) | User has documented diagnosis for dyslexia that took regular courses in K12 or Higher Ed. | Dyslexia | A single user may have multiple cognitive disabilities. That user may be counted as filling multiple categories. |
| Cognitive (Focus) | User that has/had a documented 504 plan (or doctor’s diagnosis) for focus issues that took regular courses in K12 or Higher Ed. | Attention disorders | A single user may have multiple cognitive disabilities. That user may be counted as filling multiple categories. |
When conducting usability tests, it is not recommended to conduct tasks with the following disability types:
- Colorblind – use trusted scientific methods/tools to reliably ensure that the content meets the needs of the colorblind.
- Cognitive (seizure) – use trusted scientific methods/tools to reliably ensure that the content does not cause seizures. Instead, use trusted scientific methods/tools to reliably ensure that the content does not cause seizures.
- Speech – Only required if voice control is the only method of computer interaction.
Use Caution when including the following disability types:
- Cognitive (memory) – CAUTION: Some complex online tasks or complex interfaces may be overwhelming. If you conduct usability studies for for people with memory issues there is a risk that some complex tasks could cause cognitive overload problems.
- Cognitive (executive function) – CAUTION: Some complex online tasks or complex interfaces may be overwhelming. If you conduct usability studies for for people with executive function disorders there is a risk that some complex tasks could cause cognitive overload problems.
After your company screens, validates, and recruits for the following disability types and schedule the participants, note that the average no-show rate is around 11%. To minimize this rate, you may want to offer a monetary incentive to participants for their participation.
2) The script is the same
The goal-oriented script you will be writing to test whatever path, narrative, or functionality in the system is the same regardless of the user performing the test. Like all good usability test scripts, it should be devoid of leading language.
3) Considerations when conducting the usability test
When setting up your test with for a user who uses AT, consider a couple extra steps before getting started:
- Ask them what assistive technology they use.
- See if they are willing to turn the speed of the AT down (odds are it will be really fast!).
- See if they are willing to turn the volume of the AT up.
- It is a good idea to have a quick test page to make sure everything is coming through clearly on your end.
In general, we recommend sticking with moderated testing as the feedback you will get from the user’s AT and watching their keyboard pathing in real time yield critical data in understanding usage.
**TIP: When conducting usability tests for people with disabilities, follow the proper etiquette:
- Don’t make assumptions
- Ask before you help
- Talk directly to the person
- Speak as you usually would
- Use “people first” language (i.e. “people with disabilities”, “people with low vision,” or “people who are blind” NOT “blind man”, “disabled”, etc.)
- Avoid potentially offensive terms or euphemisms (i.e. “lame”)
- Be aware of personal space
Final Important Note: Many prototyping software and applications do not have a lot of affordance for accessibility, meaning the artifact you create and ultimately test with may not be able to support the level of accessibility that would merit a proper test. A shadow system or working software may be the only option for being able to test with users with some disabilities such as AT users.
4) How to analyze your test results when considering accessibility
It’s important to note the goal of the analysis is the same for accessibility as it is for all other usability tests: make your software usable. When looking at the data from users with disabilities, the most important question to answer during the analysis process is the same for any other user: did the user accomplish the goal of the test?
If no, like any other usability test, you will need to determine to what degree there was a failure, what caused it, and ultimately what is needed to remedy the issue for subsequent testing. The second question to consider is to what degree of friction your user encountered when trying to complete the task or test. Below are a few examples of uniques accessibility factors to consider when doing your analysis:
- How many tab stops did it take to get to the element the user needed to advance in the narrative? (Issue might be Reading order or Tab order)
- Did they end up cycling through? Did they use any shortcuts such as Skip Links. (Functionality may not be marked up for AT properly. Tooltips not available or clear. Might just be a Discoverability issue.)
- Did they get into a keyboard trap? (That’s not good)
- Were the live region readouts clear as to what the information presented was or with how it was supposed to be engaged? (Functionality should be accurately described for AT)
- General understanding as to how certain pieces of functionality work. (Basic workflow and usability)
Conclusion
Conducting usability tests while considering accessibility will not have a major impact or change on your usual process. The script will be the same, the way you conduct the test will be largely the same, the follow-up and post qualitative questions will be the same. Ultimately, your litmus for what is a successful usability test from a macro standpoint is the same, on a micro level there are unique accessibility measurements to consider.