Real-Time Remote Usability Testing with Screen Reader Users, Part 2: Tips & Tricks
The following post is the second in Caitlin Geier’s 2-part series, “Tips and Tricks for Real-Time Remote Usability Testing with Screen Readers. This post details how to implement usability testing for screen reader users with helpful advice and how-to tricks.The first post (Real-Time Remote Usability Testing with Screen Readers, Part 1: Practical Overview) serves as an introduction to remote usability testing and outlines the advantages and disadvantages, along with tips and important points for consideration.
Now that I’ve gone over the basics of remote usability testing for screen reader users, I’m going to cover the process in more detail.
Setting Up the Usability Test
In my mind, there are 4 major questions to answer when setting up a usability test:
- What tasks will you ask the user to perform?
- What information (if any) do they need ahead of time?
- How will you observe as they perform the tasks?
- How will you understand and record the way they’re using your application?
These points are relevant regardless of who you’re doing the usability testing with. You may need to make some minor adjustments to account for the different technologies your users may be using, but ultimately, you want the test to be as similar as possible for all of your users. The more alike the setup is for each user, the more likely your results will speak to the spectrum of users you’re testing with.
1. Tasks
Screen reader users (and for that matter, users with any types of disabilities) should be given the same tasks that you would give anyone else. The key to having an accessible website or application is making sure that any user – regardless of ability – can use all of the features in the application. You won’t be doing yourself any favors by giving easier tasks or customized tasks to screen reader users. If you’re concerned that a screen reader user will fail at the task you’ve given them because your application isn’t accessible enough, then let them fail. Doing this will help you understand why your application isn’t accessible, and will give you ammunition to push for accessibility fixes.
2. Communicating Ahead of Time
Most likely, screen reader users will be using their own computer and technology to access your application. If they need to have anything installed, need to be on a VPN, or need to log in to a system in order to perform the tasks in your test, give them that information ahead of time! In fact, if you’re interested to see whether operating systems, browsers, or other technology may make a difference in how people use your application, you may want to consider requesting that all of your users use their own computer to test your application. In that case, you’ll want to give the same instructions to everyone.
I always send instructions via email. And I make sure that the subject line is clear (for example, “Preparation for usability test session on Tuesday, April 8th”). That way, users can easily reference your instructions before the test. If you have any non-disclosure agreements that need to be signed by your users before the test, you can add them as attachments. If possible, make sure that agreement forms are accessible and can be digitally signed – otherwise, screen reader users may have to ask someone to help them sign the agreements.
Also, be prepared to talk your users through any setup they need to do when you meet for the test. When you send setup instructions, you’re asking your user to spend additional time on top of the actual session time. Some of your users may not be willing to take the extra time, or they may assume that all they need to do is show up for the session.
3. Observing the Tasks
If you’re testing remotely, ideally you will want to see the screen of the user you’re testing with, and perhaps also their webcam feed. There are a lot of applications out there that allow you to screenshare, often in conjunction with a conference call, but few of these are accessible. In my experience, Google Hangouts and WebEx work the best when testing with screen reader users, because the controls are accessible to screen readers. Both require new users to download and install browser plugins, and Google Hangouts requires that the user have a Google account. Whichever you choose, make sure your users are aware that some setup will be required if they haven’t used the service before. In my experience, WebEx is a bit easier to set up as a new user than Google Hangouts.
As with all guided usability tests, encourage your user to talk aloud about what they’re thinking as they go through the tasks. With screen reader users, it’s important to emphasize that they narrate what they’re doing so that you can follow along more easily. You won’t be able to follow the user’s mouse because they likely won’t be using one. You can, however, follow focus indicators. But for the most part, you will need to rely on the user to tell you where they are on the page. If they seem stuck or confused, ask them what their screen reader is telling them.
Finally, give your screen reader users extra time if they need it. Sometimes that means skipping a task or two, or skipping a few post-test questions. Sometimes it means being flexible in your scheduling and allowing for extra time at the end of a session. Or it could mean following up with your user via email or a subsequent call in order to finish the session or to ask follow-up questions.
4. Recording the Tasks
Unless I’m using meeting software with a built-in recording feature, I typically record video of sessions using Snagit, while also taking notes on paper. If you intend to record task success / failure rate and time spent, make sure you record everything consistently across users.
The Rewards are Worth the Effort!
Testing with screen reader users may sound a bit complicated at first, but once you know what to expect, it’s fairly straightforward. It can sometimes take a bit of extra planning, but in my experience, the rewards are well worth it. Most all screen reader users who volunteer to do testing with you will be pleased to explain how their screen reader works, or why they think something on your site isn’t working for them. By performing usability testing with screen reader users, you’ll have an opportunity to learn a ton about a whole subset of your users. And you’ll be taking a big step forward in helping your application become more accessible.
Caitlin Geier is a UX designer on Deque’s product team. As a UX designer, Caitlin’s work with accessible design flourished once she began working for Deque. She is passionate about understanding the users she’s designing for, and she continually strives to incorporate accessibility elements into her work in order to ensure that all users can benefit from inclusive design.
