Better Product Managers, and Product Management

“I really should do user testing, but…”

You know that early validation can save weeks of working down the wrong path, right?  You may have listened to a few tangential comments from users that illuminated a whole new path to differentiation.  You’ve probably seen an interface that was completely intuitive to everyone in your company … and completely baffling to everyone outside it.

But if you’re like most product managers and entrepreneurs, you’re not testing.

First of all, testing has the sense of a big, lofty thing.

We all remember creating science fair projects years ago – you needed a formal hypothesis, a control group and an experimental group, all the variables had to be controlled, you needed to take notes, and the whole thing culminated in a typed, double-spaced report with graphs and charts.  (If you’ve worked with User Research within a large enterprise company, you still see research presentations just like this – except in PowerPoint instead of a tri-fold posterboard.)

Axe it. Forget it. I officially absolve you of needing to be super-scientific and organized.  If anyone asks, you can say “Cindy said this was okay,” and send them to me. Some data is better than no data.

Let me repeat that:

SOME DATA IS BETTER THAN NO DATA.

SOME DATA IS BETTER THAN NO DATA.

SOME DATA IS BETTER THAN NO DATA.

Are we okay now? Good. Let’s keep going.

Second of all, you don’t know where to start.

There are tons of freely available web-based tools, and blog posts giving you advice and how-to and do’s-and-don’ts.  What should I use? What’s the right tool?  Is it really okay to skip face-to-face user testing?

What tool you use depends on what you are trying to find out.  Here are a few common questions and tools:

• Is this product even remotely interesting to people (or am I crazy?)
• Do people understand what my site/product does?
• Are users able to complete the tasks that are core to my site/product? (i.e. completing specific actions, purchasing)
• What are the areas of my site/product that are confusing or annoying users?
• What are the alternatives to my site/product?  (what are people using instead of me?)
• Which audience finds my site/product most appealing?
• What other features should my product offer? (how could I make this product more appealing/differentiated)
• Am I organizing my site content in a way that makes sense to users?

(If you don’t see your question type here, leave it in the comments and I will write a Part 2 to this post.)

Is this product even remotely interesting to people (or am I crazy?)

What You’re Looking For: This is the first question that follows a “Eureka!” moment.  You have an idea, it sounds good to you.  You ask your friend and it sounds good to her, too.  But you might both be crazy, so you need some validation – just enough to convince you that it’s worth putting in more time.

Tool: SEM on $5/day. Build a really simple landing page. Buy Google AdWords or targeted Facebook apps that describe your concept, and measure how many clicks you get.  Vary your ad the next day and see if clicks improve.  If no one clicks, no one cares.  (and you just saved yourself a lot of work implementing something no one wants.) If people click, you can move on to embedding a simple survey to ask 1-2 questions about what they would pay for.

Do people understand what my site/product does?

What You’re Looking For: Validation that your splash page or email marketing copy makes sense.  You want people to see it and think X and Y.  Are they thinking X and Y?

Tool: FiveSecondTest (or equivalent).  I’m actually not crazy about FiveSecondTest, just because five seconds is even too short for skimming.  But you can easily create your own online version.  Show your splash page for 10 or 15 or 30 seconds, then advance user to a single survey question that asks “What do you think [this product] would allow you to do?” I recommend Wufoo for creating a single-question form and embedding it directly in your page.  It’s free and their simple reports do the job.

For an extended version, you may have a third screen that asks “Did you think that this product did X?  Did you think this product did Y?”    But first you want their untainted impressions.

Are users able to complete the tasks that are core to my site/product? (i.e. completing specific actions, purchasing)

What You’re Looking For: Your product has certain things that ARE the product.  If you’re an e-commerce site, people need to buy things.  If you’re a bill payment product, people need to be able to pay a bill.  This is a quantitative question.

Tool: Google Analytics. Identify your core tasks and the pages that a user must go through if they successfully complete it.  Create a funnel to identify how many users make it through each stage of the funnel.  (Note: your product does not have more than 5 core tasks, even if it’s “really complex.”  A Boeing 747 has 3 core tasks – taking off, staying in the air, and landing.)

Tool: UserFly.  UserFly records mini-movies of your users as they navigate through your site.  You can watch (sometimes painfully) as users hover hesitantly over a button, then click the wrong one, or enter the wrong information into a form field.

What are the areas of my site/product that are confusing or annoying users?

What You’re Looking For: You know there’s a problem – you just don’t know what it is.  People aren’t completing the core tasks of your product, or your bounce rate is extremely high, and you don’t know why.  This is a qualitative question.

Tool: Task-based user testing. Get 5 users off Craig’s List and offer them $50 to come to your office for a 30-minute test.  Either videotape them or have one person to moderate and one to scribble notes furiously.  Ask them to complete core task #1 and encourage them to narrate out loud as they navigate.   Resist the urge to comment or correct them.  Do ask questions like “what do you think will happen if you click there?”  Do notice their body language – are they relaxed? tense? frowning?  Remember, these people were paid to be here – so their default mode should be “happy”.

If this sounds too hard, try getting 2 users off Craig’s List.  Or skip that and just beg your non-technical friend or neighbor who doesn’t know anything about your product to come be your test subject.  Even ONE person who is not you, can reveal a ton of insight.  One person testing takes less than an hour.  You must be willing to invest ONE hour that will save you days or weeks of development and frustration.

What are the alternatives to my site/product?  (what are people using instead of me?)

What You’re Looking For: What’s the bigger story of how you fit into your users’ lives?  What are their behaviors and the environment they conduct them in?  For example, before Netflix, users rented videos from Blockbuster, or didn’t rent movies at all because it was too much of a hassle compared to just watching whatever was on cable.

Tool: In-person (or phone) interview. Get 5 people off Craig’s List and compensate them to come to your office or agree to a phone interview.  Ask them to tell you about the last time they did [your behavior].  For example, “Tell me about the last time you watched a movie at home”.  Resist the urge to comment, but ask helper questions to keep the narrative going – “How did you decide to watch a movie?”, “Was there a specific type of movie you were interested in?”, “Why did you watch a movie instead of doing something else?”, “How did you decide which movie to watch?” , “Who else was part of this decision?”, “What do you wish had been simpler/less frustrating/faster about the process?”

Tool: SurveyMonkey. Ask similar questions but via online survey.  I find this less insightful because you are far less likely to get freeform, “tangent” answers, but it’s very fast and doesn’t require someone who is comfortable giving an interview verbally.

Which audience finds my site/product most appealing?

What You’re Looking For: Should I be targeting [these people] or [those people]?  What description will convince people to convert? Where should I focus my further research/feature development?

Tool: SEM on $5/day. You can buy very targeted ads for cheap on Facebook (by now, all the demographics are there.) – set up multiple audiences and send them to your landing page, use an embedded survey to ask a few questions.  Measure the clickthrough rates and the survey response rates to get your first pass at who is most interested in your product.

Tool: Google Website Optimizer. Create two splash pages with different messaging that targets two different audiences.  Each one needs a “Sign Up Now” button that leads to a confirmation page. That’s all you need – don’t even need to link to your product yet.  GWO will compare the percentage who converted (landed on the confirmation page).  Clicking the “Sign Up Now” button is a clear measure of interest.  Repeat with as many “markets” as make sense.

What other features should my product offer? (how could I make this product more appealing/differentiated)

What You’re Looking For: Solving the big pain point.  What is it?

Tool: Task-based user testing. Get 5 users off Craig’s List and offer them $50 to come to your office for a 30-minute test.  Either videotape them or have one person to moderate and one to scribble notes furiously.  Ask them to complete core task #1 and encourage them to narrate out loud as they navigate.  As they navigate your site, have them describe where they are and what they’re doing when using your product (or the general behavioral context).

For example, the idea for one-handed baby wipes came from mothers describing what they did — “and I’ve got one hand keeping the wiggling baby from rolling off the changing table, and now I need to reach for a wipe…”

Note: there are various customer tools like UserVoice and SuggestionBox that allow customers to suggest features.  I’m deliberately not recommending those, because I subscribe to the Henry Ford philosophy that his customers would’ve suggested a faster horse. Users, generally speaking, can always come up with things they want but are not as good at articulating true pain points.  That’s what the experienced product manager or entrepreneur has to glean from a wider context.

Am I organizing my site content/product features in a way that makes sense to users?

What You’re Looking For: Products that are described as “intuitive”, usually are because they align with how the user thinks about performing this task.   How do your users think about your task?  How would they logically group features or site sections?

Tool: Online card-sorting. Card sorting is an exercise where you list out all of the concepts involved in your product/site, and allow users to group them in any way that makes sense to them.  This you have to see to believe, because my first thought was “these will all be completely different! No two people think exactly alike!”  And yet, my experience has always been that strong patterns emerge with as few as 20 users.

List all of your features and the OptimalSort tool will ask users to group them however they see fit (you can do an “open” sort, where there are no constraints; or a “closed” sort, where you choose the categories and people sort concepts into them.)  It shows the grouping and patterns for you.  Because this is an online exercise, the compensation can be on the lower side.  You can also ask questions at the end – I like to ask “are there other features that should be here?”  Because you’ve set the context, you tend to get more constructive feedback in this context than if you normally asked “what else do you want?”

There.  Now you have no excuses!  Go forth and test.  If you have questions that I didn’t cover, list them in the comments.

Popularity: 1% [?]

Popularity: 1% [?] blog comments powered by Disqus