A friend of mine was recently telling me about his new job.
“They have the fanciest in-house usability testing lab I’ve ever seen – one-way mirrors and video cameras, eye-tracking technology… and, come to think of it, their products have the worst user experience I’ve ever seen.”
Unfortunately, the damage probably won’t be limited to this unnamed company’s consumer products. When “best practices” go bad, they create skeptics – product managers, engineers, and executives who conclude that those techniques are useless.
Persona analysis, usability testing, voice of the customer feedback, and surveys – these are great tools for learning about your customers. Or rather, they can be. If you don’t follow these examples…
Persona Analysis
By far the prettiest user personas I’ve ever seen were created by a major online portal. They were full of detail – each persona had a full-color headshot, a name, and extensive detail about where they lived, what job they held, number of kids, where they shopped.
The problem came when I tried to use those personas to help drive product decisions. I asked some questions:
- Four personas – what percentage of their user base did each one account for?
- Which one represented the majority of customers?
- How valuable are each of these user types? Is one more likely to engage in revenue-generating behaviors? More likely to be a loyal customer or recommend their product?
No one could answer these questions. This meant that there was no way to decide what should take precedence when the personas’ needs contradicted each other. We couldn’t make intelligent decisions about which features to emphasize and which could be hidden behind an extra click.
As it turned out, the personas were created in complete isolation from the business stakeholders. They looked great, they were detailed, and they were probably even accurate representations of online portal’s customers. But they couldn’t be used to prioritize or make decisions, so the product managers ignored them – that is, until they were asked if they did persona analysis. Then they answered “of course we do!”
The problem: Going through the motions without understanding the purpose of the exercise.
How to avoid this: Skip the pretty pictures and answer the “5 Hows” – how many, how profitable, how comfortable with technology, how concerned with convenience vs. quality, how they trade off between money and time. Going more bare-bones with persona analysis increases your chances at getting all the stakeholders to read them and give the feedback you need to make them useful.
Usability Testing
I was sitting in the observation room behind the one-way mirror with the bank product managers and designer. We watched the first, second, and then third user have trouble with a specific task, and hypothesized out loud that it was the inline text that was confusing them. “Let’s quickly mock up another variation with different text,” I suggested. “That way, we’ll learn whether it’s the text or something else that’s causing the problem.”
“We can’t,” said the product manager. “We need to run the test the exact same way for all eight users, or else the report we produce won’t be scientific enough.”
It’s common practice to encourage test participants to “think out loud” as they navigate through the website or application. This means sometimes they go off on tangents, offering opinions that weren’t directly asked for. On this particular day, four of the eight testers independently made very similar comments about a frustration they had with the current version of the application. I was scribbling notes madly – this was a great insight, and I could already envision a new feature that would solve this frustration and differentiate the product.
But those comments never made it into the usability testing report that was created for management. Why? Because they weren’t part of the “official” testing script. “People say all kinds of things,” said the usability test moderator dismissively. “I don’t put anecdotal stuff into the reports.”
The problem: Confusing qualitative testing with quantitative data. Second problem: Making testing so expensive and resource-intensive that you need a “scientific-looking” report to management in order to justify it.
How to avoid this: Set expectations appropriately that the point of usability testing is to learn as much as possible. If changing questions mid-stream allows you to test out twice as many hypotheses, that’s a good thing. And again, go bare-bones: when testing is cheaper and easier, you don’t need to spend as much effort justifying it.
“Voice of the Customer” Feedback
When is listening to your customer a bad thing? When you:
- take feedback at face value without attempting to understand the context
- assume that the X customers asking for a feature are representative of your entire user base
- build features that contradict your product strategy “because X customers asked for it”
- don’t validate that these are legitimate customers
A major credit card provider demanded that we make a change to our product. According to their voice of the customer data, it was a critical request and they felt we couldn’t be successful without it.
I asked for an export of their “voice of the customer” data so I could get a bit more detail. As I read through the verbatim customer comments, I noticed that most of the customers requesting this change had never actually used our product. They had read the splash page, and based on that, assumed (incorrectly, I might add) that the product worked a certain way.
Making the requested change wouldn’t have satisfied these customers at all! Instead, the company changed their splash page, and that resolved the issue.
The problem: Listening without understanding.
How to avoid this: Follow up and ask more questions. Why is this a problem? If they have a suggested solution, what do they think having this solution would help them to do? What is the context? Could this be a one-time issue or is it something systemic? What are the alternatives?
Surveys
“How concerned are you, on a scale from 1-10, about privacy?”
“I’d say 9 – very concerned.”
“OK, thank you for your time. In order for me to get you your check for participation, you’ll need to write down your name and Social Security number for me.”
“OK, no problem!”
99% of surveys will tell you that customers are very concerned about security, privacy, and getting the best deal. That they want features A, B, C, D, and the kitchen sink. That they floss daily and eat 5 servings of vegetables a day. Then you watch as their buying and usage behavior completely contradicts what they just told you.
Most surveys are close to useless because they ask questions without addressing the necessary tradeoffs. Do customers like the idea of more features? Of course! Do they want to spend an extra hour learning how to use your product, or spend an extra $100 on it? Of course not!
The problem: Asking questions in isolation without mentioning the accompanying benefits or tradeoffs.
How to avoid this: Choose freeform, not yes-no, questions when it comes to matters of security, privacy, health, finances. People feel like they should answer “yes”, so they do and you get crummy data. Balance features with tradeoffs (“would you spend $x more for [feature]?” or “would you want [feature] if it meant x more learning time?”) or give users a limited number of votes to “spend” on specific features so that they are forced to prioritize.
