6. Test Your Site with Users with Limited Literacy Skills
Introduction
There’s no surefire way to control who comes to your website—and why
would you want to? The web is for everyone.
As mentioned previously in this guide, as many as half of U.S. adults
have limited literacy skills—and nearly 9 in 10 have limited health
literacy skills.94,95 And while the
information in this guide is a good place to start when designing for
users with limited literacy skills, it can’t predict user behavior on a
specific tool or website.
That’s why it’s so important to test your websites and digital products
with individuals with limited literacy skills.
Usability testing can take many forms. In general, it involves “watching
people try to use what you’re creating/designing/building (or something
you have already created/designed/built), with the intention of making
it easier for people to use or proving that it is easy to
use.”96
The goal is to get feedback from users to truly understand their
motivations and behaviors online. Conducting usability testing will help
you answer important questions like:
- Can users find the information they’re looking for?
- Can our users understand and act on this information?
- Are user interface elements intuitive and easy to use?
Suggestion
Have users try out your website as you plan, design, and develop
it—early and often.
The specific approach you choose will depend on the goals of your
research, how much access you have to your users, and the details of
your project, timeline, and budget. Learn more about usability
testing.
Below we outline important strategies for you to consider when
conducting usability testing with limited-literacy users.
6.1 Recruit users with limited literacy skills—and limited health literacy skills.
Most screening tools designed to measure health literacy skills (like
the Test of Functional Health Literacy in
Adults and the Rapid
Estimate of Adult Literacy in
Medicine)
have to be used in person and are meant for patients in a clinical
setting.97
These options may not be practical or very useful for web and health
content developers, especially if you’re using a private company’s
recruitment database. Instead, you can use a proxy for health literacy
based on commonly collected demographic data.97 For example, the
criteria below could be used as a proxy for identifying web users with
limited health literacy skills:
- High school education or below
- Low income—defined as 200% of the federal poverty level or below (or
less than $48,500 for a family of 4 in 2015)98
- Have not searched for health information online in the past year
You can also try a simple 1-question health literacy measure. As part of
screening, ask your participants: “How confident are you filling out
medical forms by yourself?”99
Keep in mind that some kind of personal connection can make a big
difference during recruitment—when you can say that a mutual
acquaintance suggested the connection, people are more likely to
consider participating in your study.100
Use trusted community recruiters to help you recruit from community
contexts—like adult learning centers, federally qualified community
health centers, and senior centers. This can also help you get
participants from your target populations.
6.2 Identify and eliminate logistical barriers to participation.
Choose a testing location that is familiar and accessible to
participants, like a local library, community center, or even a health
clinic. Be mindful of factors that may affect participants’ ability to
get to your testing site—like public transportation options or parking
availability and cost.
Avoid remote and online testing if possible. If you have to do remote
testing, make sure someone is on site with the participant to provide
technical assistance.
It’s also important to consider accommodations that your target
population may need. For example, if you’re testing with older adults,
have reading glasses available for people who might need
them.3 Also make sure your testing location is
accessible to those with limited mobility. If your target audience’s
primary language is not English, you may need to use an interpreter.
Finally, use cash incentives instead of gift cards when possible.
Participants may want to use the incentive for basic living expenses
like rent or utility bills.
6.3 Create plain language testing materials.
Write your screeners, consent forms, and moderator guides in plain
language. Find sample testing documents on
Usability.gov.
Limit the number of tasks and questions when conducting usability
testing with users with limited literacy skills. Be realistic about what
you can accomplish in each session. Make sure the most critical tasks
are early in your protocol—it’s best to keep the session brief.
Users with limited literacy skills can have difficulty understanding
consent forms—you’ll often find that they’ll sign without reading the
form.101 In addition to providing easy-to-read consent
documents, consider using a consent process that doesn’t rely on
participants’ health literacy skills or English proficiency. You can:
- Review consent forms verbally3
- Ask participants to initial each thing they consent to—for example,
written notes, having the screen recorded, or being videotaped
See the Informed Consent and Authorization Toolkit for Minimal Risk
Research [PDF – 300
KB]
from the Agency for Healthcare Research and Quality for more
information.102
6.4 Test whether your content is understandable and actionable.
When designing your study, think about ways to find out if your content
is both understandable and actionable. To evaluate whether participants
understand the content, ask them to:
- Think out loud as they complete tasks
- Describe what they’ve read in their own words
- Describe what action they would take after reading the content
Learning whether users can understand your content is a very important
part of usability testing. But it doesn’t stop there—ultimately, your
goal is that users will act on the health messages you created. That’s
why you also need to find out if your content is actionable.
With this in mind, don’t limit what you test to measures of
comprehension and usability. Include a mix of quantitative and
qualitative measures—for example, you can ask users:
- What they would do after reading your content
- How confident they are that they could do what the content is asking
Qualitative measures like these also offer insight into self-efficacy,
an important predictor of a user’s ability to adopt healthy behaviors.
6.5 Use moderators who have experience with users with limited literacy skills.
Whenever possible, use moderators experienced in conducting usability
testing with people with limited literacy skills or with people who
don’t have a lot of experience using the internet.
Moderators familiar with limited-literacy participants in a usability
testing setting know what to expect—they’re sensitive to participants’
challenges and know how to effectively support participants through
tasks. For example, experienced moderators are more likely to be patient
with participants completing tasks very slowly—or they may immediately
recognize the need to repeat a task prompt when a participant is getting
frustrated.
Ultimately, choosing moderators who have experience working with
limited-literacy users can go a long way toward making your testing
sessions a success.
6.6 Pretest your moderator’s guide.
It’s very important to pretest your moderator’s guide with participants
with limited literacy skills so you can fine-tune tasks and timing.
These pretest—or “dry run”—sessions can help you:
- Clarify tasks that were confusing for users with limited literacy
skills
- Figure out how much time to leave between sessions so you can
accommodate users who may need extra time
If you’re new to testing with limited-literacy users, don’t be surprised
if testing sessions with users with limited literacy skills move more
slowly than testing sessions you’ve facilitated in the past. That’s
normal.
6.7 Use multiple strategies to make sure participants understand what you want them to do.
During introductions, make it clear to users that they can reread the
task at any time if they’re confused or if they’ve forgotten what to
do.
During sessions, have the moderator read tasks out loud and provide
them in writing (only 1 task per sheet). This will remind users of the
task you are asking them to accomplish. Also have the moderator repeat
the task during the session.80
Try this
When testing with limited-literacy users, consider
offering short breaks.3
Keep in mind that participants with limited literacy skills tend to
focus on the specific task—sometimes to a fault. During sessions, remind
users that you’re less interested in the answer and more interested in
where and how they would try to find the answer.
6.8 Test on mobile.
Users with limited literacy skills are more likely to access your
information on mobile, so always test your website on mobile devices.
Testing on mobile will also allow you to collect information about your
site’s responsive design. Remember, you want your website to work well
on a variety of devices.
Testing on mobile is different from testing on a desktop computer. Here
are some things to think about before you test on mobile with
limited-literacy users:
- Record movements using a mobile testing set up if you want to review
your testing session later or share it with other members of your team.
Make sure you think through your technical needs in advance.
Figure 6.1
The moderator sets up a small camera to record participants'
movements during mobile testing.
- Let participants use their own device. Users are accustomed to how
their smartphone or tablet is set up—you’ll get more accurate
information if they test your site on a device they’re familiar with.
- Be mindful of data consumption limits. Since you’re asking
participants to use their own device, you need to consider data use.
Always make sure participants can access free WiFi. And in general,
think through what you’ll be asking them to do in advance—for example,
will they need to download an app or stream multiple videos? Activities
like these may use more data than simpler tasks.
Summary
Conducting usability testing with users with limited literacy skills
requires special preparation. Strategies like working with an
experienced moderator, eliminating logistical barriers for your
participants, and creating plain language testing materials can increase
your likelihood of success.