One of the first steps in each round of usability testing is to develop a plan for the test. The purpose of the plan is to document what you are going to do, how you are going to conduct the test, what metrics you are going to capture, number of participants you are going to test, and what scenarios you will use.
Typically, the usability specialist meets with the site or product owner and members of the development team to decide on the major elements of the plan. Often, the usability specialist then drafts the plan, which circulates to management and the rest of the team. Once everyone has commented and a final plan agreed upon, the usability specialist revises the written plan to reflect the final decisions.
Elements of a Test Plan
You will need to include these elements in the usability test plan.
- Scope: Indicate what you are testing: Give the name of the Web site, Web application, or other product. Specify how much of the product the test will cover (e.g. the prototype as of a specific date; the navigation; navigation and content).
- Purpose: Identify the concerns, questions, and goals for this test. These can be quite broad; for example, “Can users navigate to important information from the prototype’s home page?” They can be quite specific; for example, “Will users easily find the search box in its present location?” In each round of testing, you will probably have several general and several specific concerns to focus on. Your concerns should drive the scenarios you choose for the usability test.
- Schedule & Location: Indicate when and where you will do the test. If you have the schedule set, you may want to be specific about how many sessions you will hold in a day and exactly what times the sessions will be.
- Sessions: You will want to describe the sessions, the length of the sessions (typically one hour to 90 minutes). When scheduling participants, remember to leave time, usually 30 minutes, between session to reset the environment, to briefly review the session with observer(s) and to allow a cushion for sessions that might end a little late or participants who might arrive a little late
- Equipment: Indicate the type of equipment you will be using in the test; desktop, laptop, mobile/Smartphone. If pertinent, include information about the monitor size and resolution, operating system, browser etc. Also indicate if you are planning on recording or audio taping the test sessions or using any special usability testing and/or accessibility tools.
- Participants: Indicate the number and types of participants to be tested you will be recruiting. Describe how these participants were or will be recruited and consider including the screener as part of the appendix.
- Scenarios: Indicate the number and types of tasks included in testing. Typically, for a 60 min. test, you should end up with approximately 10 (+/-2) scenarios for desktop or laptop testing and 8 (+/- 2) scenarios for a mobile/smartphone test. You may want to include more in the test plan so the team can choose the appropriate tasks.
- Metrics: Subjective metrics: Include the questions you are going to ask the participants prior to the sessions (e.g., background questionnaire), after each task scenario is completed (ease and satisfaction questions about the task), and overall ease, satisfaction and likelihood to use/recommend questions when the sessions is completed.
- Quantitative metrics: Indicate the quantitative data you will be measuring in your test (e.g., successful completion rates, error rates, time on task).
- Roles: Include a list of the staff who will participate in the usability testing and what role each will play. The usability specialist should be the facilitator of the sessions. The usability team may also provide the primary note-taker. Other team members should be expected to participate as observers and, perhaps, as note-takers.
Identifying Test Metrics
There are several metrics that you may want to collect during the course of testing.
- Successful Task Completion: Each scenario requires the participant to obtain specific data that would be used in a typical task. The scenario is successfully completed when the participant indicates they have found the answer or completed the task goal. In some cases, you may want give participants multiple-choice questions. Remember to include the questions and answers in the test plan and provide them to note-takers and observers.
- Critical Errors: Critical errors are deviations at completion from the targets of the scenario. For example, reporting the wrong data value due to the participant’s workflow. Essentially the participant will not be able to finish the task. Participant may or may not be aware that the task goal is incorrect or incomplete.
- Non-Critical Errors: Non-critical errors are errors that are recovered by the participant and do not result in the participant’s ability to successfully complete the task. These errors result in the task being completed less efficiently. For example, exploratory behaviors such as opening the wrong navigation menu item or using a control incorrectly are non-critical errors.
- Error-Free Rate: Error-free rate is the percentage of test participants who complete the task without any errors (critical or non-critical errors).
- Time On Task: The amount of time it takes the participant to complete the task.
- Subjective Measures: These evaluations are self-reported participant ratings for satisfaction, ease of use, ease of finding information, etc where participants rate the measure on a 5 to 7-point Likert scale.
- Likes, Dislikes and Recommendations: Participants provide what they liked most about the site, what they liked least about the site, and recommendations for improving the site.