Usability testing – The quest for honest user feedback


Usability testing being conducted

“Staying on this page, can you look for section X?”

This seemingly straightforward question revealed interesting user behaviour during some recent usability testing. It wasn’t so much how difficult participants found the task to complete, but the response they made on completing the task itself. While a number of participants clearly found it difficult to finish finding section X to our satisfaction, some of those same participants suggested that they found it relatively easy. This seemed somewhat contradictory.

I observed one user scrolling up and down the page endlessly searching for the illusive section X. 20 seconds passed. 30 seconds passed. 40 seconds in total had passed, an absolute eternity on the web. I’m certain if anyone had to spend this amount of time looking for anything on an e-commerce website, they’d simply give up and move on to a competitor. One could suggest they spent this amount of time looking because they were explicitly told to during a test.

At the end of the searching for section X task, participants would be asked “How would you rate this task out of 5? 1 being difficult, 5 being easy.” To my surprise, a number of users who looked to struggle with the task consistently rated 4 out of 5, suggesting that they didn’t have much difficulty at all and were able to complete the task with ease. Could it be possible that users spending this length of time on a task was the norm? Research conducted elsewhere would suggest otherwise. This was by no means an isolated incident either; this was something that could be consistently observed through a number of different tests.

The Hawthorn Effect

From Wikipedia, “The Hawthorne effect (also referred to as the observer effect) is a type of reactivity in which individuals modify an aspect of their behavior in response to their awareness of being observed. The original research at the Hawthorne Works in Cicero, Illinois, on lighting changes and work structure changes such as working hours and break times was originally interpreted by Elton Mayo and others to mean that paying attention to overall worker needs would improve productivity.”

User interviews are a great way to glean information about behaviours, but it should always be stressed that the very act of interviewing someone, remotely or in-person, changes their behaviour itself. Dan Nessler wrote a great article titled, “How to conduct the best user interview? Don’t interview.”  It really emphasises a less formal, conversational approach with participants.

setting the scene & asking the right questions

As a UX Designer, I spend a lot of time considering questions to put forward for usability testing. Nick Babich from suggests that, “Interviewing users requires effort. You might spend several weeks preparing for the sessions, several days talking to your users, and several days analysing results. You want to make sure all that effort won’t be thrown away. That’s why it’s essential to spend time to properly plan your questions.”

Generally speaking, the majority of my usability testing is carried out online using a platform such as – effectively a way in which we can perform unmoderated, remote interviews. This still requires the careful thinking, planning and analysis that Nick suggests. However, it’s absolutely not the same as in person, moderated usability testing, though it is relatively inexpensive, convenient and usually has a quick turnaround. If I want to generate insights from 5-10 different users, I’m pretty confident that all 10 will have gotten back to me within 48 hours.

So planning is key to ensuring that questions not only make sense, but will generate answers that can drive design decisions going forward. When testing, the team and I are looking for pain points voiced by multiple users that form a pattern of concern. We then can analyse in greater depth for potential UX improvements. We can only find these pain points however if the users participating in the test feel free to be critical. The answers need to be reflective and authentic.

Knowing that the very act of interviewing someone can potentially change results, what can we do?

reassure the user

Having interviewed lots of users, I’ve noticed that while the majority are very forthright, some seem a little bit more reluctant to open up about what’s in front of them. This could be for a number of reasons, one of which could be that that’s as critical as they will get, and that’s fine. Everyone is different, and that’s the beauty of interviewing – we’re able to make less assumptions and become more empathic about who we’re designing for.

For others who are reluctant to be critical about what’s in front of them, a little nudge in that direction generally suffices. Consider the below example of copy that users could see before the testing begins.

This is an opportunity for you to provide us with critical feedback. This critical feedback will help us build a better product for our customers. If you love something, or hate something, speak out loud and tell us why you love or hate something! Remember, you’re not being tested – our product is.

Disclaimers like the above should offer some reassurance to the user that they can say what they want. When starting a user test, let the user know that it’s not them that’s being tested, but what’s in front of them. It’s perfectly natural for people participating in usability testing to be a bit nervous, even if they’re just sitting in front of their computer with a microphone. It therefore shouldn’t be surprising that some folks can even blame themselves for not being able to complete tasks, when in actual fact, they’re highlighting potentially poor user experience in our product.

An article by IxD suggests telling a little white lie at the start of the session. “You need to tell a small fib to your participants when you begin the testing process. Explain that you are testing something for a third party and that you have no relationship with that third party.” The reasoning behind this being that participants perhaps feel more at ease because you’re not directly implicated with whatever they say.

Give realistic scenarios

During the early days of my UX career, it would be fair to say that some scenarios for some people just didn’t work. During in-person, moderated testing, I’d often give them a scenario like “You’re purchasing a bike for your son, speaking out loud, can you show me what you’d do next?” The problem here is that some people simply didn’t have children. It might seem like such a small thing, but changing son to friend and changing the purchase item to something smaller like a t-shirt, made the scenario more realistic for the user.

They were instantly able to relate with that situation, which made the test go a lot smoother. Ultimately it’s about being aware of who you’re interviewing, and it’s a perfect opportunity to small talk before the test takes place.  This again is something that generally makes the participant feel at ease, as the environment suddenly becomes less formal.

You could take this one step further by sending out a small 5-10 question prescreen questionnaire to help you understand who they are before they sit down for actual testing. Sure, there’s the chance they won’t fill it in, but for those who do, you’re reducing the chance of making awkward assumptions.

Change the scenario slightly if you need to, based on what you know about who you’re talking to – otherwise just keep it a bit more generalised so participants don’t need to suspend disbelief too much.

Remember as well that if you’re user testing online you can usually ask a prescreen question before the user participates. So if you’re only interested in individuals, for example, that are really into extreme ironing, you can ensure all tests going forward match this criteria and completely negate folks who only iron sans any extremeness.

Usability testing can offer amazing insights…

As UX designers, we have to remember that while usability testing will hopefully make for a better product, it shouldn’t be a research method used in isolation. Results will always vary from method to method. Bear in mind limitations, and ensure that you use multiple research methods to retrieve valuable information, and your work will stand up to even the toughest scrutiny.

Most importantly, however, is that user interviews will always be better than not doing user interviews at all.

Note: Throughout this piece, usability testing and user interview are used interchangeably. User Interviews can be wide ranging, whilst usability testing is specifically testing a product or service by testing it on users.

About the author


Chris Kernaghan us a UX designer based in Newtownabbey, just outside Belfast. He's passionate about empathetic design and creating amazing user experiences.

Add comment