It is vital to recruit participants who are similar to your site users for your usability testing. Depending on the site or product, you may have multiple potential users groups. Try to include representatives of all these groups or optimally, perform testing with each group separately if you really want to concentrate on role-based information or functionality.
Remember that you are not your user. Use your internal staff as participants only if…
- They have had no involvement in the design or development of the site or product and
- They represent a target audience
Internal staff may be used for pilot testing since you are testing the technology and the flow of the test and the data is not factored into the final results.
Internal staff should never be used to supplement during testing. It is not just any participants you need; it’s participants from the right audience.
How many participants are enough?
Nielsen outlines the number of participants that you need based on a number of case studies:
- Usability Tests: test 5 users lets you find almost as many usability problems as you’d find using many more test participants.
- Quantitative studies (aiming at statistics, not insights): test at least 20 users to get statistically significant numbers; tight confidence intervals require more users.
- Card sorting: test at least 15 users.
- Eye tracking: test 39 users if you want stable heat maps.
If you are testing in the federal space, please review OMB guidelines related to the Paperwork Reduction Act for usability testing. For diagnostic usability testing, six to eight users of a given target audience are usually enough to uncover the major problems in a product.
Note: If you plan to do iterative (repeated) usability testing over the course of developing the site, you will need to recruit a new group for each test. You will need to factor that into your planning, recruitment, and budgeting.
Participant screeners are composed of questions that will help those recruiting for your test to rule individuals in or out of contention. They may be as simple as gender and age or as complex as your target audience dictates. For examples, please see our screener templates.
Costs of Recruitment
Recruiters generally charge a fee for each participant “successfully” recruited. A successful recruit is one that meets the criteria, appears for testing and is able to complete the test. A good recruiter will screen, schedule and remind the participants about their test appointment to assure all of their recruits are successful.
If needs be, you may also engage the recruiter to handle additional administrative duties such as administering incentives for participants (i.e. gifts or money), and in some cases travel/parking expenses. There will be a fee for additional services so it will be best to discuss any additional services needed by your team during your initial discussions with the recruiter.
Working with a Recruiter
If the team has access to representative users you can recruit from those individuals. If the team does not have access to representative users, you will have to hire a commercial recruiting company. Most recruiting companies require two to three weeks to find and schedule the necessary number and types of participants. Here is the basic information you should have available when you speak to a recruiter or request a quote for their services:
- How many participants you will need; total and within specified groups.
- The location, date and time for testing. It is helpful to provide a draft of your screeners (government-focused template and non-government template), a detailed schedule and a map or directions to your facility, if possible.
- How long each session will take.
- If you will be compensating for testing, how much [and in what format; cash, check or gift cards or certificates), and what if anything else you will be providing (i.e. travel or parking).
Compensating a Participant
When determining how and how much to compensate participants for their time it is important to determine what methods and amounts have been used previously. Your recruiter may be helpful in learning what is typical for the tests on which they have recruited. Your colleagues may also be helpful, if they have either conducted or participated in testing.
Some things to consider when compensating participants:
- If your participants are federal employees, you cannot pay them for their time.
- If your participants are non-federal employees, the mode of compensation should be in-demand by potential participants. Money in any form is generally acceptable but it may be more convenient to provide gift cards or certificates for online or local vendors. Keep in mind that purchases from an online vendor will generally charge your participants for shipping. You may want to adjust the compensation accordingly.
- If you are doing remote testing, you may want to consider an electronic mode of compensation such as an eCertificate to an online vendor.
Remember to provide a receipt for your participants (one for adults and one for minors are included in the templates section) to sign for the purposes of:
- Showing that they received the compensation and
- Providing documentation to your accounting department or personnel.