How to Conduct a Usability Study
I wrote on the Kera Blog a few days ago about conducting usability studies. It's a very personal and valuable post that I want to keep. It's not good to have duplicate content for Google rankings but it's okay to have some. Furthermore, my personal blog is mostly for myself. Here's the re-post:
I've conducted several usability tests for different web applications but it's still time consuming to plan and I always feel like I'm forgetting something. Many people know the importance of doing usability studies but not so many are able to do it well.
Similar to me, many developers, designers, and product managers fell into this UX role and had to learn it themselves. It’s taken me a while to learn how to conduct effective, valuable user testing so I thought I’d write a guide and checklist for those who don’t do this as their full-time job or just starting out.
Similar to me, many developers, designers, and product managers fell into this UX role and had to learn it themselves. It's taken me a while to learn how to conduct effective, valuable user testing so I thought I'd write a guide and checklist for those who don't do this as their full-time job or just starting out.
Basic Idea of Usability Study
If something isn't performing well, you have to go be a detective and figure out why. That's the essence of doing user testing.
This is something I really enjoy doing because as a product owner, you always want to build the best product possible and there's nothing like watching real people use your product. For my maps business, I literally stood in the retail stores and watched how people responded to the product: if they noticed the product, how they responded to it, how they decided whether or not to buy, why they bought it, etc. It's like being a modern-day Sherlock Holmes but less dangerous.
Recently, I performed a usability test for our latest product, the Get Started Widget and will use examples from it.
Reasons to do a Usability Test
The main benefits of doing web usability testing are
- You get unfiltered and high bandwidth feedback
- You can see how users think and points of confusion
- You get to figure out the why
- You feel like Sherlock Holmes
Usability testing is a qualitative method of research which is much better suited for answering question about why or how to fix a problem. Quantitative methods such as A/B testing do a much better job answering how many and how much questions. Nielsen Norman Group has a great article on which user experience research methods to use.
Aside: If your company has a support team, the team lead should be involved with user experience in general because those people really understand the pain of the customer and can often provide ideas and suggestions to vastly improve the product.
For example, at Wimdu we were missing out on signups because of our Facebook Connect design. We didn't know this but we confused some users into thinking we required Facebook in order to sign up. People actually called in to try to figure out how to sign up without Facebook. We fixed this with by adding a horizontal rule with the word "or" between the sign up methods. That ceased those types of support calls.
Step 1. Planning (1-3 hours)
Have an objective(s). Why are you doing this study? What questions do you want answered?
Kera Example
Study Objective 1: Do people know what what they're signing up for?
Study Objective 2: Why are people not embedding?
Study Objective 3: How easy is it for users to use our application?
Decide on goals for the user to accomplish. What would success look like?
Kera Example
Goal: User creates a mission and embeds widget on his or her site.
Have hypotheses (optional). Although it is recommend to have one, I have often been really successful without one especially early on. Sometimes product challenges are so difficult, you need to do research just to get a hunch. In this case, keep an open mind and just watch users and take in what is happening without any bias to prove or disprove.
Alternatively, you may want to conduct a heuristic evaluation, a review by a usability expert of your UI against accepted heuristics, to generate some hypotheses. This can also help you create appropriate tests and prototypes for the next step. See also: Tips on doing it yourself and a primer.
Kera Example
Hypothesis 1: People don't understand how to use the application.
Hypothesis 2: It's not friendly to marketers.
Choose a location. This should be as close to the environment of the actual user as possible.
Kera Example
Location: Office environment, on users' own computers.
Make a list of questions you want to ask. These include interview questions you might want to ask, prompts for follow-up questions, as well as post-experiment questions that you want to ask each participant.
Screening Questions:
What's age? What do you do? Have you ever embedded a YouTube video?Experiment Questions:
Did you understand you'd be getting a widget for your website?What do you think we do?
How would you describe us to your friends?
You can also try to understand what is going through the user's head by asking them questions as they are using the application. See "asking question to get the why" in Step 3.
A usability test plan template. Here's a template that can help you plan your study.
Step 2. Setting Up the Study. (1-6 hours)
Get the prototype or test environment ready. Create and ready the materials you need for the study. For example, do you need build anything, create mockups, set up a test or local environment, disable A/B testing, etc.?
You may want to pilot test the study to help you identify gaps in your experiment or test setup.
Prepare a way to record the study. Recording the user is best because you may want to reference it again if you missed anything and it allows you to administer the test better since you're not focused on taking notes. (And I'm pretty sure all good spies use recording equipment.) If you're not recording, figure out how to take good notes. If you're collecting quantitive data, prepare a form or table to help you quickly take notes and stay organised.
Get the people. Ideally you'll have a pool of candidates to ask to come in for a usability study. I suggest putting up posters offering cash for participation around offices, incubators, and universities. And online on craigslist, etc.
If you don't have a pool to select from, you can try finding participants on the spot at cafes, offices, universities, meetup groups, and hallways.
You should pick a few types of people for your user groups because it's interesting to see how different groups respond but if you're testing for general usability, random people with normal intelligence are just as effective. I'd recommend 4-8 participants. Since our product is for web users, we wanted to get people who are internet savvy. We put people in 3 groups: business/marketing, engineering, and general.
A usability study checklist. Here's a checklist to help you prepare for conducting a web usability test by Steve Krug's Rocket Surgery Made Easy book.
Step 3. Executing the User Research Study. (1-4 hours)
Greeting the participant. You should brief the users in a standardised way that allows you to understand them and helps them give you the best feedback possible. A sample script:
Kera Example
Intro to Usability Study:
Have you done a usability study before? (yes/no) Great, just to give you an idea of how this works. We'll be asking you to take a look at our website and asking you some questions. Then we'll get you to try our application to perform some tasks. It should take about 25 minutes in total. We're going to be testing the web application to see if it is understandable and usable. We're not testing you. There are no right or wrong answers.
Intro to Marketing Site Analysis:
Say a friend has sent you a link to this site. I'm going to give you a few moments to look at the site and then ask you what it's about. Look around the site as you would normally. This is yours– go ahead. (Open the site/test environment and give them the laptop or mouse/keyboard.)
Intro to Task Completion:
Now we're going to ask you to use our application. (After finding out participant was a history major). I'm going to give you a scenario. Let's say you're a project manager for a museum's website that has users. You want your users to be more engaged and active so someone has recommended that you take a look at this site. You have developer resources but you're going to take a look first and evaluate it because you're responsible for this project.
I highly recommend watching this short video on how to run in-person experiments on Stanford's HCI course (slides) and this Zipcar usability test example. If the videos are no longer available you can still find them in my shared Usability Toolkit folder on Box below.
Lastly, collect information on the participant and get them to fill out any paperwork if necessary (i.e. contact info, demographic data, consent, NDA, etc.). Many templates are in the Usability Toolkit folder at the end.
Observing and taking notes. Observing the user is one of the main benefits of in-person experiments. Watch users as they attempt to perform tasks with the user interface. Collect quantitive data on task completion. Record interesting findings. Log points of confusion. Steps users took to complete a task or to explore/navigate your website. Note what users were looking at and what they ignored. What users say and ask.
Kera Example
Some notes I made:
- she clicked on 'get started' button because she wanted to get more info; not interested in pricing yet
- visuals were the main thing she looked at; headlines second; text copy was ignored
- visuals did not communicate to user what our product was. it was confusing
- welcome message– user ignored it
- get started icon– she was not interested in clicking it and didn't know what it was
- user did not click on it was because didn't know what it was
Asking questions to get the why. There are many questions you can ask during the study to understand why users are doing certain things, why they did something, what mental models they have, and what they were guessing or attempting to do. Some questions I ask are:
Kera Example
Can you describe what is happening?
How would you go about doing [something]? Take a guess.
What do you want to click on now?
What do you want to know now?
Why did you do that?
What are you thinking?
What do you think this is? (point to a UI element).
Is there anything you're confused about or that stands out to you?
People don't always tell Sherlock Holmes the truth. As NN Group explains self-reporting can be skewed because often users tell you what they think you want to hear, what they remember, and rationalise.You should use your judgement on what's really happening based on your observations and use questions as another source in your investigation.
Asking the user to talk aloud. The think aloud protocol is an effective method to understand what is going through a user's mind. It's similar to asking questions to get to why but here the user is talking aloud as he or she is performing the task uninterrupted. You'd need to teach the user how to think aloud first. I usually say something like this:
As you’re doing the task, I want you to say aloud whatever you are thinking. For example, if I am thinking aloud (take a pen), I’d say: I see this pen. It has a blue cap so I assume the ink will be blue. It looks like I can just pull it off so I’ll try to pull it. Great, it worked.
Or see 8:25 of the in-person experiment video to see a Stanford professor do it.
Often this is hard for a user so you'd need to revert back to prompting the user or asking questions to get the why. Remember: participants who think aloud will be slower at performing tasks and may behave differently.
Debrief the participant. This is the time when you can 1. inform them about the purpose of study if you couldn't before 2. get them fill out paper survey questionnaires and 3. get them to sign a receipt of remuneration.
Step 4. Summarising and Presenting Findings. (2-5 hours)
Refining notes. Right after each participant, take the time to write down all your observations and particularities while they are still fresh in your mind. Quickly check the notes you made during the study to see if they can be understood later on and convert to long-form if necessary.
Aggregate findings and performing calculations. You'll get an idea of the most critical discoveries which will drive your report. Compile all the qualitative findings into main takeaways, citing numbers or percentages to indicate importance. Do the same with quantitive data. Perform calculations as need. Here's a website with many useful calculators for usability testing.
Reporting findings. Summarise the goal, participant characteristics, results, and recommendations. The report depends on the scale of the project and your audience. If it's a small study for your startup, it can be relatively informal. If it's for a client, you might need to package your presentation accordingly. Here are some tips.
Presenting. It's great to give a presentation to the team in person so you can be sure you're communicating the findings effectively. You can also take this time to answer questions, discuss the implications, and brainstorm improvements or follow-on studies together.
Final Thoughts
You might want to outsource this. While I enjoy doing it in-house and being close to users, there are definitely companies that specialise in UX and conduct great usability tests. A couple relatively cheap and easy to use ones are YouEye and usertesting.com
To me, data without reason is just noise. For example, if there is correlation between flu outbreaks and iPhone releases, would you trade on it? Probably not because there is no underlying rationale. There are many dangers to basing decisions on just data such as randomness, data dredging, confounding variables, correlation vs. causation, Simpson's Paradox, false discoveries, etc.
I'd recommend using both A/B testing and usability studies to give you both strong data but also help you uncover the reasons for why things are happening. Usability studies and talking to your users is the best way to confirm or disprove your reasoning.
Books and other excellent sources that helped me
Checklists for your usability study. Get it here.
Template for planning usability study. Get it here. More templates on usability.gov
Example of an expert conducting a usability study for Zipcar. See it on YouTube.
Example of usability study process for Boxee. Read the article here.
UX Books, classes, people and more.
This is a prerequisite for starting out in user experience. The Design of Everyday Things by Donald A. Norman. Here's a nice review.
This is the most respected web usability guide. Don't Make Me Think: A Common Sense Approach to Web Usability by Steve Krug.
Usability Testing Demystified by Dana Chisnell. She also has several great articles on UX Magazine and her website.
Human-Computer Interaction Course (Stanford + Coursera.org)
Hack Design. An easy to follow design course for hackers who do amazing things.
Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests, 2nd Edition (Downloadable sample eBook or on Amazon) by Jeffrey Rubin and Dana Chisnell
Whitney Hess is a UX designer who has really gone out of her way to help aspiring UX designers and people interested in this field like me. Some good ones: So you want to be a user experience designer, You're not a user experience designer if, Whitney Hess's recommend UX resources, and UXers Twitter List.
This is my Usability Toolkit shared folder where I keep a lot of my resources for user experience design. Say hi and get in touch over Twitter if you want to be a collaborator.
---
Taige Zhang is a Product Marketer at Kera. You can follow him on Twitter @taigeair.