Original User Research
For the 1st edition of Health Literacy Online in 2010, ODPHP commissioned 15 studies over a period of 4 years to inform the development of an online collection of prevention and wellness content and to redesign healthfinder.gov. More than 750 people aged 18 to 84 participated in these studies. Special care was taken to recruit and test adults with limited literacy skills.
Below is an overview of the iterative research process, followed by a brief description of each study.
- Literature Review
- Website (Content) Analysis
- Expert Interviews
- Structured Interviews
- Mental Models Study
- Card-Sorting Study
- Prototype Test (Paper and Clickable)
- Usability Studies 1–4
Post-Launch Quality Improvement Phase (Ongoing)
- Card-Sorting Study
- Widget Usability Test
- Intermediary Interviews
- Usability Study 5
Formative Research Phase
Literature Review
ODPHP conducted a literature review to answer the following questions:
- Which audience characteristics are the strongest determinants of homogeneity in terms of seeking prevention information?
- In what ways and to what extent is currently available prevention information responsive to these patterns?
- Is prevention information that is targeted in content design and delivery more effective than information that is not targeted?
- How can prevention information be presented effectively online?
The literature generally supported the finding that tailored and targeted materials that respond to individual characteristics, such as readiness to change, can be effective in engaging interest and leading to health behavior change. The review also found that interactive information is more engaging and motivating than information that is not interactive.
Website (Content) Analysis
ODPHP reviewed content on 9 prevention topics available from 4 leading health websites: American Cancer Society, Mayo Clinic, WebMD, and Family Doctor. Based on this review, the authors made several general recommendations for designing health information for websites, including:
- Keep content at a reasonable length.
- Use headings and subheadings.
- Use hyperlinks so users can explore relevant content.
- Provide interactive tools as appropriate.
- Convey information using graphics or visual displays.
Expert Interviews
Subject-matter experts in 9 health content areas identified common questions asked by members of the public. The goal was to identify audience segments based on motivations for seeking information. The audience segments developed from the expert interviews included:
- Segment 1: Those seeking information about a health problem for themselves or someone they know
- Segment 2: Those wanting to find out whether they or someone they know has a health problem or reason to be concerned about a health problem
- Segment 3: Those seeking information to help prevent the onset of health problems
Structured Interviews
ODPHP conducted structured interviews with 200 diverse participants between the ages of 18 and 65 to test the audience segmentation strategy mentioned above. Examples of key findings include:
- The audience segmentation strategy was validated (significant differences were found in motives/preferences between the 3 segments).
- Respondents did not remain in a particular audience segment over time.
- Content should reflect the needs/motives of each segment.
Mental Models Study
In-depth interviews were conducted with 35 English-speaking adults, aged 18 to 65, with limited health literacy. The primary intent of this research was to determine how people naturally group disease prevention topics (i.e., their “mental models”) to inform the development of a prevention prototype. Examples of key findings include:
- Participants grouped the topics of nutrition, obesity, and physical activity together.
- Other topics were commonly stand-alone concepts, such as preventing falls and getting flu shots.
- Participants associated the topics of talking to kids about smoking and talking to kids about substance abuse.
Card-Sorting Study
Eighty-one diverse participants completed the card-sorting study. As a result of the study, ODPHP created a prioritized list of the most useful types of content across the 3 audience segments. These content types emerged as generally “useful” and “important”:
- Basics I need to know (Understanding)
- I would like to learn more (Assessment)
- I can do this (Overcoming Barriers)
- How will this help me? (Motivators)
- Ways I can take action (Strategies)
- Where can I go for help? (Community Resources)
Prototyping Phase
Prototype Test (Paper and Clickable)
ODPHP developed and tested a prevention information prototype with a diverse, nationwide sample of 300 adults. Participants evaluated the prototype and prevention content on measures of engagement, self-efficacy, acceptability, and applicability. Examples of key findings include:
- Participants found the content acceptable (relevant and useful), but did not rate it as highly for inspiring self-efficacy.
- Qualitative data showed participants’ preference for the following content characteristics:
- Informal, clear, and concise writing
- “Tabbed” organization
- Actionable information with a small-steps approach
- Interactive tools
- Graphics that aid understanding
- An interface that allows users to “drill down” through related chunks of information
Usability Study 1
ODPHP performed an initial usability study with 40 adult women with limited health literacy skills recruited from federally qualified community health centers in Baltimore, MD. The study was designed to observe and record any problems encountered by users as they navigated the prototype, to learn whether changes to the content and interface design would positively affect self-efficacy measures, and to gauge levels of understanding and engagement. Examples of key findings include:
- Participants particularly liked checklists, lists of questions to ask a doctor, access to related resources, graphics that aid understanding, and options to print.
- The tabbed approach helped orient participants to where they were in the content.
- Participants found the interactive tools (e.g., calculators, quizzes, menu planners) useful because they were immediate, relevant, and in context.
- Providing small steps for taking action helped improve participants’ self-efficacy.
- Content organization of “What is the behavior?” (Basics), “Why is it important/relevant to me?” (Benefits), and “What do I do about it?” (Take Action) was logical to participants.
Usability Study 2
ODPHP conducted a 2nd usability test with 13 adult women with limited health literacy recruited from federally qualified community health centers in Baltimore, MD. This test was performed to validate changes made from the prior usability test (Usability Study 1) and to continue to test levels of engagement, self-efficacy, and understanding. Examples of key findings include:
- Participants had difficulty using the search function and often clicked on the “Search” button without entering a search term.
- Most users expected to see a list of topics in an A-to-Z format.
- When using the myhealthfinder tool, participants did not easily connect the information they entered into the tool with the search results they received.
- Tab navigation needed to be more clearly delineated.
- Participants preferred photographs of “real” people to icons.
- Participants noticed content at the center of the page more than top, bottom, and side content.
Usability Study 3
ODPHP performed a 3rd usability test with 7 diverse participants recruited in Knoxville, TN. This study provided an overall test of the new healthfinder.gov site and of the changes made based on previous testing. Examples of key findings include:
- Most participants used the A-to-Z section, rather than the search box, to browse.
- When prompted to use the search function, participants had little trouble and were able to scan the results page easily.
- Participants were able to navigate easily back to the home page through strong lefthand navigation.
- Participants particularly liked the photos of “real” people and the Take Action content.
Usability Study 4 (myhealthfinder)
ODPHP conducted a usability test on the 1st iteration of the myhealthfinder tool with 15 diverse participants. Examples of key findings include:
- Participants had difficulty signing in. Many attempted to sign in even though they had never created an account.
- Many participants had difficulty creating an account. For example, they did not distinguish between required and optional fields, had difficulty creating a unique username and password, and did not understand error messages.
- Participants did not want to enter personal information (e.g., height, weight, smoking status).
- Participants often skipped over context and welcome information.
- Participants needed more than just recommendations; they wanted corresponding “how-to” information.
Post-Launch Quality Improvement Phase (Ongoing)
Card-Sorting Study
ODPHP conducted a card-sorting activity with 30 adults with limited health literacy recruited through the WIC program (Special Supplemental Nutrition Program for Women, Infants, and Children) in Alexandria, VA, and from a federally qualified community health center in Baltimore, MD. During each of 4 focus group sessions, participants completed 2 card sorts to refine the organization of the Quick Guide to Healthy Living topic pages and to develop categories and labels for a new Quick Guide landing page. Examples of key findings include:
- Participants had difficulty understanding the Benefits tab and tended to organize all information in the Basics and Take Action categories.
- Participants were very interested in 1-page, printable, action-oriented topics and tools using bullet points instead of paragraphs.
- Participants preferred topics to be placed under multiple categories.
- Participants tended to like categories of information labeled for certain groups (e.g., women, older adults, parents).
Widget Usability Test (Be Active Your Way) (2 Rounds)
ODPHP held 2 usability tests with 18 participants recruited from adult literacy programs in Washington, DC. These studies specifically focused on the usability, usefulness, and understandability of a widget providing targeted physical activity tips. Examples of key findings include:
- Participants did not notice the widgets in the left and right margins.
- Many participants thought the sidebar graphic (widget icon) might be an ad.
- Participants didn’t respond well to the word “barriers.”
- Participants often glossed over difficult words, such as “moderate” and “vigorous.”
- Participants often ignored or skipped over very short words like “is” and “in.”
- Participants wanted to use the “Back” button rather than other navigational elements.
Intermediary Interviews
ODPHP conducted interviews with 10 health intermediaries recruited from WIC programs and federally qualified community health centers. The goal of the interviews was to obtain feedback on how healthfinder.gov content could better serve WIC and community health center clients. Examples of key findings include:
- Participants thought the organizational structure of the content (Basics, Benefits, Take Action) was logical and would help them explain prevention behaviors to their clients.
- Participants cautioned that content should reflect the cultural sensitivities and economic realities of priority audiences.
- Participants liked the use of positive, empowering language.
- Participants suggested an increase in the use of images and symbols to make points “stick.”
Usability Study 5
ODPHP conducted a 5th usability test with 9 participants with limited health literacy recruited through 2 community adult learning centers in Columbia, MD, to provide feedback on a new Quick Guide landing page and topic structure. Examples of key findings include:
- Participants often missed items on the right side.
- Participants struggled with scrolling or didn’t scroll at all.
- Participants had high success rates in finding topics through the Quick Guide organizational structure.
- Participants easily noticed and used the left-hand navigation.
- Participants needed everything that looked “clickable” to be “clickable” and wanted navigational elements to be big and easy to notice.
