Minimize text size Reset text to normal size Increase text size

Remote Testing

Remote usability testing allows you to conduct user research with participants in their natural environment by employing screen-sharing software or online remote usability vendor services.  In general, tests should be about 15–30 minutes long made up of about 3-5 tasks. 

woman participating in a remote test

Remote usability tests can be “Moderated” in the same manner you would for an in-person lab test or, “Un-moderated” where participants complete the tasks independently.

When to Consider Remote Testing

There are certain conditions when it makes sense to consider remote usability testing.  Some of those conditions include:

  • Timelines might prevent in-person testing due to scheduling issues
  • The target audience/participants are geographically dispersed making travel for them or the experimenters difficult
  • The participants need to use a particular work machine due to software or security requirements
  • The participants have accessibility issues which require that they use their own software or equipment
  • You can potentially run multiple tests at once

Although this method is flexible, as with any test, you must consider the benefits and cost of this methodology.

Benefits Challenges
  • Eliminates both the need for a lab environment and the effect of a lab environment on participants
  • Accommodates diverse groups of participants
  • Generally is less expensive than a traditional in-person lab testing
  • In the case of un-moderated testing, allows you to extend your test day, possibly allowing you access to a larger pool of participants
  • It is an opportunity to administer the test  a larger group of people than you might be able to accommodate in a lab environment
  • Security could be compromised if testing sensitive, privileged, or intellectual property
  • Restricted  or no view of the user’s body language might inhibit some of the cues to their reactions to the material being tested
  • Technical difficulties are likely if users:
    • Are not comfortable with the technology involved
    • Have conflicting software or equipment on their machines
    • Are unable to share their screen over the Internet
    • Have unreliable or slow connection speeds, or
    • If your test requires special equipment or software downloads or plug-ins which the participants may be unwilling or unable to download


Developing a Remote Usability Test

The processes for developing a remote test are much the same as developing the test and materials for a test you are conducting in-person. Often you can leverage an in-person test you might have run for use in a remote setting.  It’s important to:

  • Remember that tests should be about 15–30 minutes long made up of about 3-5 tasks.
  • Develop straightforward tasks that have well-defined end states
  • If you are using a screener, be sure to include the minimum system requirements, both of the site or tool you are testing, but also for the screen sharing service that you propose to use
  • Make sure you have the correct contact information for your participants for reminders and follow-up if needs be
  • Prepare introductory and/or test materials so the participants will know what is expected of them as well as what they can expect from you
  • Prepare test consent forms
  • Prepare compensations and receipts for compensation should you choose to pay your participants

 Remote testing differs mainly is in the technology.  You will need to assure that: 

  • Whatever you are testing is accessible outside the firewall of your business, agency or dev environment
  • You will need to determine if any firewall issues might affect the ability of your participants to access the site or tool you are testing
  • Participants can download/access the screen-sharing software, or online remote usability vendor services

Moderated Remote Usability Testing

During moderated remote testing, participants are observed on interacted with while they complete the tasks for the test. Moderated testing is best for complex tasks that do not have a structured sequence of steps or where a more interaction and questioning will benefit testing.

To conduct a moderated remote usability test, in addition to the general guidance noted above, you will need to:

  • Define the test and decide what tasks you are going to ask participants to perform, the order of the tasks, and any follow-up questions
  • Schedule test sessions
  • Build in enough time in the test session to allow for resolving technology issues

During the moderated remote usability tests you will be able to:

  • Assign interview-based task performance
  • Ask follow up questions
  • Ensure test completion
  • Test and analyze as planned

Depending on the technology or methodology used, you may also be able to analyze such metrics as task-completion rate, time on task, time on page, clickstream paths, and satisfaction ratings or opinion rankings.

Un-moderated Remote Usability Testing

During un-moderated remote testing, participants independently complete testing without an interaction with the test moderator.  Un-moderated testing is most effective when you have very specific questions about how people use a user interface for relatively straightforward tasks.

To conduct an un-moderated remote usability test, in addition to the general guidance noted above, you will need to:

  • Define the test and decide what tasks you are going to ask participants to perform, including the order of the tasks
  • Determine who will participate. Many un-moderated testing tools allow you to either intercept users on your Web site or recruit them from the tool developers’ own list of test participants
  • Determine ahead of testing whether your screen sharing service either has recording capability or will allow for the use of your recording equipment or software. 
  • Be aware that picture-in-picture(PIP) or webcams may not be captured if your software can only accept one video input.
  • If choosing from a pool of recruited participants, make sure they are representative of your target audience
  • Include qualitative research questions when possible and analyze self-reported feedback
  • Pay attention to time-on-task and open-ended feedback to filter out participants who may be participating for the sole purpose of earning the incentive
  • If your participants have any vision challenges, consider asking if they are familiar with the screen share software you are using.  If not consider asking whether a colleague or family member will be available during testing to assure that they can share their screen

During an un-moderated remote usability test, you will need to:

  • Launch the test and send email invitations to participants to complete it.
  • After the sessions have been completed you can analyze your results. Many un-moderated testing tools automatically generate analysis for you.

Depending on the technology un-moderated remote usability tests may also allow you to analyze such metrics as task-completion rate, time on task, time on page, clickstream paths, satisfaction ratings or opinion rankings, web analytics data (such as browser, operating system, and screen resolution).

References