Minimize text size Reset text to normal size Increase text size

Running a Usability Test

Once you have planned your test and recruited your test participants, it’s time to get ready to conduct your test.  To do so, you’ll want to think about which moderating technique is right for your test, set up your space and equipment, and make sure that you do a pilot test prior to testing with actual participants.

Choosing a Moderating Technique

In her Moderating Usability Tests article, Jen Romano Bergstrom notes that choosing the best moderating technique for your test depends on your session goals.  Some common moderating techniques include:

  • Concurrent Think Aloud (CTA) is used to understand participants’ thoughts as they interact with a product by having them think aloud while they work. The goal is to encourage participants to keep a running stream of consciousness as they work.
  • In Retrospective Think Aloud (RTA), the moderator asks participants to retrace their steps when the session is complete. Often participants watch a video replay of their actions, which may or may not contain eye-gaze patterns.
  • Concurrent Probing (CP) requires that as participants work on tasks—when they say something interesting or do something unique, the researcher asks follow-up questions.
  • Retrospective Probing (RP) requires waiting until the session is complete and then asking questions about the participant’s thoughts and actions. Researchers often use RP in conjunction with other methods—as the participant makes comments or actions, the researcher takes notes and follows up with additional questions at the end of the session.

It’s important to weigh the pros and cons when you are trying to decide which technique to use:

Techniques

Pros

Cons

Concurrent Think Aloud (CTA)

  • Understand participants’ thoughts as they occur and as they attempt to work through issues they encounter
  • Elicit real-time feedback and emotional responses
  • Can interfere with usability metrics, such as accuracy and time on task

Retrospective Think Aloud (RTA)

  • Does not interfere with usability metrics
  • Overall session length increases
  • Difficulty in remembering thoughts from up to an hour before = poor data

Concurrent Probing (CP)

  • Understand participants’ thoughts as they attempt to work through a task
  • Interferes with natural thought process and progression that participants would make on their own, if uninterrupted

Retrospective Probing (RP)

  • Does not interfere with usability metrics
  • Difficulty in remembering = poor data

Pilot Testing  

Prior to conducting a usability test, make sure you have all of your materials, consents and documentation prepared and checked.  It is important to pilot test equipment and materials with a volunteer participant. Run the pilot test 1-2 days prior to the first test session so that you have time to deal with any technical issues, or change the scenarios or other materials if necessary.  The pilot test allows you to:

  • Test the equipment
  • Provides practice for the facilitator and note-takers
  • Get a good sense whether your questions and scenarios are clear to the participant
  • Make any last minute adjustments

Best Practices

  • Treat participants with respect and make them feel comfortable.
  • Remember that you are testing the site not the users. Help them understand that they are helping us test the prototype or Web site.
  • Remain neutral – you are there to listen and watch. If the participant asks a question, reply with “What do you think?” or “I am interested in what you would do.”
  • Do not jump in and help participants immediately and do not lead the participant. If the participant gives up and asks for help, you must decide whether to end the scenario, give a hint, or give more substantial help.
  • The team should decide how much of a hint you will give and how long you will allow the participants to work on a scenario when they are clearly going down an unproductive path.
  • Take good notes. Note-takers should capture what the participant did in as much detail as possible as well as what they say (in their words).  The better the notes are that are taken during the session, the easier the analysis will be.
  • Measure both performance and subjective (preference) metrics. People's performance and preference do not always match. Often users will perform poorly but their subjective ratings are very high. Conversely, they may perform well but subjective ratings are very low.
    • Performance measures include: success, time, errors, etc.
    • Subjective measures include: user's self reported satisfaction and comfort ratings. 

Example Usability Test Session

Here is an example test session.

  1. The facilitator will welcome the participant and explain the test session, ask the participant to sign the release form, and ask any pre-test or demographic questions.
  2. The facilitator explains thinking aloud and asks if the participant has any additional questions. The facilitator explains where to start.
  3. The participant reads the task scenario aloud and begins working on the scenario while they think aloud.
  4. The note-takers take notes of the participant’s behaviors, comments, errors and completion (success or failure) on each task.
  5. The session continues until all task scenarios are completed or time allotted has elapsed.
  6. The facilitator either asks the end-of session subjective questions or sends them to an online survey, thanks the participant, gives the participant the agreed-on incentive, and escorts them from the testing environment.

The facilitator then resets the materials and equipment, speaks briefly with the observers and waits for the next participant to arrive.