Moderated usability testing is a commonly used technique in user research. It involves watching users trying to complete relevant tasks on your product or service. It will show you how well your product or service works, and help you identify specific usability issues and insights.
In principle, moderated usability testing is a straightforward process. Typically, you get some participants, give them some tasks and often ask them to think aloud while they are solving the tasks. In practice, it is much more nuanced and complicated.
If done right, usability testing can give you high-value findings that will help you identify and address issues with your product or service. It will show you how it might work in the real world.
But, if not done right, research participants may not interact with your thing in a natural way. This could result in unreliable and invalid findings. It could also lead you to either miss important problems, or have false alarms and try to fix the wrong thing.
This blog post will give you some practical tips on how to carry out moderated think-aloud usability testing. It’s based on what I’ve learnt from academic research and from practical experience, both inside and outside government.
Put the participants at ease
This is something you need to think about before you start the testing. You’ll get the best results if participants feel at ease and are comfortable, so always make sure you never skip this step, even if under time pressure.
- Try getting to know participants before you start. You could ask them, ‘how was your journey here?’ or ‘how has your day been so far?’
- Ask an unusual question first to get people thinking. Try ‘you’ve won the lottery and are travelling the world – where would you like to go first?’
- Make sure participants aren’t worried about offending you. Say ‘we didn’t design this, so please don’t worry that what you say might upset us.’
- Make sure participants do not feel they are being tested, which can cause anxiety and change their behaviour. Say 'it's not a test of you – we just want to understand how the service works for you'. Avoid using phrases like ‘usability testing’ or ‘test’ – say ‘study’ or ‘research’ instead
Don’t steer participants
Remember that you need to keep the usability testing as objective as possible. You need to ensure that participants are telling you what is going through their minds spontaneously, not what they think you want to hear.
- Try using neutral instructions when introducing the purpose of the study – for example, say ‘we want you to say out loud everything that is going through your mind’, rather than presenting specific areas you are interested in
- Make sure your tasks do not contain task cues or specific actions that would either lead participants to success or bias their natural way of working. It’s useful to have tasks based on actual user goals. For example, if you want to understand how useful the ‘search’ function is, avoid using tasks like, ‘you want to search for survey software’, and use ‘you need to get something to conduct a survey’ instead
- Use neutral acknowledgements to show you’re listening, like ‘uh huh’ or ‘mmm’ (but don’t overdo it)
- Avoid using words like ‘yes’ and ‘right’, as this may make participants feel you agree with what they are doing or saying
- Make sure you don’t correct or challenge what people say or do
- Conceal your thoughts about participants’ approaches – do not say things like ‘that’s surprising’ or ‘I haven’t seen anyone do that before’
Be cautious about when and how to prompt
You should spend most of your time observing and listening, and be cautious about when and how to prompt. Each time you prompt, you introduce the risk of unintentionally biasing and interrupting participants’ from their natural workflow, so it’s important to know when you should prompt to get useful information.
- Try giving participants 15-20 seconds if they fall into silence, then say ‘what are you thinking now?’
- Avoid using the word ‘why’ to elicit reasons. Use ‘because…’ or ‘could you explain a bit more?’
- Do not complete participants’ sentences if their reply is incomplete. Instead, you can simply repeat the keyword with an intonation. This also works well as a way to get an explanation
- Avoid stopping participants from what they were doing in order to examine a specific feature or area that they missed
I find it very useful to prompt based on behaviour and/or comment triggers, and I’ll share this in detail in a follow-up blog post.
Be flexible
It’s important to follow a process for usability testing, but you also need to be flexible and respond to participants and scenarios you might find yourself in.
- If appropriate, try personalising the session according to each participant. For example, if a participant has failed several tasks in a row, try adding in an easy one, or if a participant accidentally succeeds in a task, try repeating it a bit later by changing the task slightly
- Change task details or customise questions to make them specific to each participant. For example, if the task is about visa requirements, you could tailor it to the country the participant is from or has recently visited to make the task more relatable for them
How do you do usability testing?
I hope this helps you when you next carry out usability testing. I’ll write several other blog posts to explore specific areas in more detail.
If you have any advice from your experiences or want to know more about usability testing please let us know in the comments.
9 comments
Comment by Rik Williams posted on
Nice primer. I liked your idea to 'ask an unusual question' in the preamble/warm-up — will give this a try.
Comment by Tingting Zhao posted on
Thanks Rik. Looking forward to hearing how it went : )
Comment by Aisling posted on
Good blog. I am thinking of doing a 'dummy' task before the real task so that the person can settle down before using the product we are actually interested in. For example, if you were doing research on one section of the website, would it be useful to start on a different section first?
Comment by Tingting Zhao posted on
Hello Aisling, thanks for reading the blog post. This is a good idea. In fact, according to psychologists Ericsson and Simon, who wrote Protocol Analysis: Verbal Reports as Data, and studied think-aloud methods extensively for years, they advised to use a simple practice to help participants get use to the concept of thinking aloud. For example, asking people to think-aloud while putting staples into stapler, or assembling a pen. Such a task helps people to settle down, warm up to the think-aloud as well as avoid any potential priming effect : )
Comment by Rik Williams posted on
I play a video clip of (non-related to research) of a think-aloud protocol or demonstrate a simple example. However, I really like this non-digital — stapler — example. It seems:
- neutral,
- familiar,
- might elicit access needs,
- product/service/process insights.
Have you used an analogue task in anger?
Comment by Tingting posted on
Hello Rik, yes, you summarised really well in terms of the benefits of . using a non-digital practice as a warm up. In terms of 'an analogue task in anger', please forgive my ignorance, I'm not sure what this is, and would be keen to find out from you : ) Thanks.
Comment by Tingting Zhao posted on
Hello Rik, you summarised really well in terms of the benefits of using a non-digital practice as a warm up. In terms of 'an analogue task in anger', please forgive my ignorance, I'm not sure what it is, and would be keen to find out more from you : ) Thanks.
Comment by Nikki posted on
How do you actually attract your users, Tingting? This is something I'm grappling with. So, for example, if I wanted to get a cross-Civil Service group of managers of disabled people? Or disabled people themselves? The key element is freeing them up from their normal workload: how do you get them released from work to participate (a big issue is that the workload backs up in their absence). Thanks
Comment by Tingting Zhao posted on
Hello Nikki. This is one of the challenges researchers are facing and that's why participants recruitment is one of the topics that research ops is looking at. You are certainly not alone 😉 Home office has done some great work on research ops, and if you haven't talked to them already, perhaps reach out to them? It's a bit difficult to give specific advise without understanding more about the project, but here are a few things I found helpful personally: 1. make it visible in terms of why this piece of research would be beneficial for the people you're recruiting for (not about incentive, but about why this matters and how the outcome can help people with their work or life). I often pair up with content designer to craft the message. 2. if possible, align with someone who know/work with the people you want to talk to well. With their help, they can schedule in research at times that minimise the impact on workload. 3. communicate research findings and changes made based on the research to the participants and people who helped you. This way, they can see the results of their input and felt their effort being impactful and appreciated. I'd love to hear what you have tried and how you approach this : )