Effectively moderating usability tests is critical to gaining insights and understanding about your users needs. User experience researchers use many techniques to moderate participant sessions. Some techniques introduce unintended consequences, while others are more pure. It is important to think about your goals for each study and plan your moderating technique accordingly.
Techniques at a Glance
|Concurrent Think Aloud (CTA)||
Understand participants’ thoughts as they occur and as they attempt to work through issues they encounter
Elicit real-time feedback and emotional responses
|Can interfere with usability metrics, such as accuracy and time on task|
|Retrospective Think Aloud (RTA)||Does not interfere with usability metrics||
Overall session length increases
Difficulty in remembering thoughts from up to an hour before = poor data
|Concurrent Probing (CP)||Understand participants’ thoughts as they attempt to work through a task||Interferes with natural thought process and progression that participants would make on their own, if uninterrupted|
|Retrospective Probing (RP)||Does not interfere with usability metrics||Difficulty in remembering = poor data|
Concurrent Think Aloud (CTA)
One technique commonly used to understand participants’ thoughts as they interact with a product is having them think aloud while they work. In CTA, the moderator only uses prompts such as “mm hmm,” and “keep talking.” The goal is to encourage participants to keep a running stream of consciousness as they work.
With CTA, we are able to understand participants’ thoughts as they work through issues. CTA is excellent at eliciting real-time feedback and emotional responses. However, the dual task of thinking aloud while working sometimes interferes with usability metrics, such as accuracy and time on task.
Retrospective Think Aloud (RTA)
In RTA, the moderator asks participants to retrace their steps when the session is complete. Often participants watch a video replay of their actions, which may or may not contain eye-gaze patterns. Because RTA allows participants to work in silence, it is an obvious way to control for the negative effects of CTA.
The process of thinking aloud does not interfere with usability metrics (such as time on task), although overall session length increases dramatically. RTA appears to elicit more insight words than CTA; however, memory is fallible, and participants often do not stay on task during RTA. (For a brief review of CTA vs. RTA, see Olmsted-Hawala & Romano Bergstrom, 2012).
Concurrent Probing (CP)
Another technique is probing participants as they work on tasks—when they say something interesting or do something unique, the researcher asks follow-up questions. Usability tests rarely include CP because it interferes with the natural thought process. It can also affect the progression participants would make on their own, without interruptions.
However, CP can be valuable if goals do not include strict user experience measurements. Survey researchers often use CP when pretesting terms and wording to assess how well people understand the questions and concepts.
Retrospective Probing (RP):
Another technique is waiting until the session is complete and then asking questions about the participant’s thoughts and actions. Researchers often use RP in conjunction with other methods—as the participant makes comments or actions, the researcher takes notes and follows up with additional questions at the end of the session.
Selecting a Moderating Technique
Choosing which moderating technique is best depends on your testing goals. Things to consider when you are deciding on which technique to employ include:
- Can the participant work completely alone?
- Will you need time on task and accuracy data?
- Are the tasks multi layered and require concentration on the part of the participant?
- Will you be conducting eye tracking (though not covered here, see Romano Bergstrom & Olmsted Hawala 2012, for the effects of think aloud on eye-tracking data)?
Whatever technique you choose, it is valuable to gain some level of insight into participants’ thoughts. Even if you are conducting unmoderated remote sessions, you can instruct participants to think aloud while they work, and they can respond to follow-up questions at the end. While watching users’ actions may be useful in determining usability issues, gaining insight into their thoughts provides a much richer understanding of the user experience.
Olmsted-Hawala, E. L. & Romano Bergstrom, J. C. (2012). Think-aloud protocols. Does age make a difference? Proceedings from the Society for Technical Communication Summit, May 2012, Chicago, IL.
Romano Bergstrom, J. C. & Olmsted-Hawala, E. L. (2012). Effects of Age and Think-Aloud Protocol on Eye-Tracking Data and Usability Measures. Paper presentation at EyeTrackUX, Las Vegas, NV, June 2012.