User testing sample size

Who you need as participants, and how many you need, depends on the type of testing or research you’re doing and the size of the project. Here are some general guidelines for usability testing:

• In 3 sessions, you’ll usually get a basic indication of the differences in behaviour. You’ll see whether the 3 participants react consistently or differently to the tasks.
• After about 5 sessions, clear themes should start to emerge. You’ll start to recognise patterns of behaviour and perception between the different participants.
• By 8 sessions you might feel like the most prominent findings are repeating, but you’ll still uncover new – and likely surprising – details and observations.

Usability

Rule: 5 to 8 participants for each user type is usually enough to represent the experience of 80% of that user group.

Diagram showing when to test usability and method to use.

Detailed description of graph

This line graph shows ‘Issues identified’ on the Y axis and ‘Number of users’ on the X axis. The line shows that the number of issues identified increases with the number of users but the curve flattens out after about 8 to 10 users. This highlights the optimal number of users is about 8 per user group.

Recruiting participants

There are several approaches to recruiting users and running usability tests. Choose the best fit for your project.

Guerrilla style

Guerilla style is approaching people on the street and asking them to participate in a quick usability test.

Features

• Useful in the early design stages to test ideas and clarify questions with a few users.
• You’ll need a portable prototype.
  Prototyping
• Extremely economical. Participants are often happy to receive a chocolate bar, a free coffee or similar.
• Usually unrepresentative of real end users. Tests are often quite short (usually 5 to 10 mins). It’s also easy for the facilitator to bias the participant.

Family and friends/snowball

For the snowball approach you use your own networks to find participants — asking friends of friends.

Features

• Often used when you have a solid idea of what you’re testing or what your product might look like.
• Recruitment is simple and fast.
• Can be biased. Facilitating a session with someone you know isn’t recommended, as it’s difficult for the facilitator and the participant to remain neutral.

Targeted

Recruitment is run by professional recruiters, or internal databases. You’ll need permission to contact people for research reasons.

Features

• Targeted recruitment is appropriate for larger projects that have a high impact on the way a service is provided to a wider audience.
• Less biased — participants’ feedback is likely to be more honest and direct.
• Generally requires either a recruiting budget or access to a database that holds contact details of people likely to be in your audience.

Community groups

For this approach, contact community groups about asking their members.

Features

• Useful when the service is for a particular type of user in a particular situation.
• Participants will likely be the real end users of a service.
• May take some network building.

Co-workers

For this approach, ask your co-workers and colleagues to do some early prototype testing. Make sure they’re not working on the project too.

Features

• Useful when ideas are relatively new or on the first prototype.
• Simple and fast. This will also help you catch factors early before taking it to external participants.
• Likely to be biased. Often your co-workers have more exposure to government process and will have a different view to your customer. They might also be overly respectful of your feelings.

Ethics

It’s important to tell your participants what they’re agreeing to when participating in a usability session. We do this verbally and with a consent form. If your agency already has a consent form, make sure it will work for a usability test. If not, you’ll need to make one.

If the session is recorded or has observers, the participants need to give their consent. To give them confidence, explain what the recording will be used for. Make sure participants clearly understand all of the consent form. Let them know that they can ask questions and that you won’t start the test until the consent forms are signed.

Have 2 consent forms ready for the participant to sign just before the test starts. The participant must sign both — you keep one and they keep the other.

Check what your agency’s policy is for compensation and thank you gifts. These should be signed for — this can be included in the consent form. You’ll need to decide when, in the recruitment phase, you inform participants of compensation and how you frame it. It’s important that the participant understands you’re not paying them to do this. You’re thanking them for their time and compensating time and travel costs.

Compensation

Since participants are giving us their time, it is common practice to thank them with a voucher. The amount depends on the type of participant, the length of the session, the level of involvement, and your agency’s policies around this.

A few examples for guidance:

• A 5-minute test on the street or with co-workers might be rewarded with chocolate.
• A simple web-based usability test for 60 minutes, where participants are ‘general users’ (no or little specific user requirements), might be rewarded with a voucher in the range of $30 to $50.
• A 60-minute web-based test with a very specific user (for example, you needed to find a person who was in the exact situation that related to your scope) might raise it to the range to $40 to $60.

When deciding how much to give a participant, consider:

• the wider situation a participant might be in
• how far they might have to travel
• they may need to take time off work
• they may have had to arrange child care etc.

Remember that your participants are expressing goodwill by agreeing to participate. Make the room comfortable for them and thank them for their time and feedback.