Better Product Managers, and Product Management

“The plural of anecdote is not data.” — Frank Kotsonis

That’s a good way to shut down someone who is trying to use sparse “I know someone who…” stories to second-guess a decision.  But in general, it’s not true.  The plural of anecdotes IS data – as long as you’ve collected them properly and challenged them rigorously.

A couple weeks ago I wrote:

It’s tempting to try and turn as many questions as possible into quantitative ones, because those are easier to measure through unobtrusive means – web analytics, looking for data patterns, surveys – and give clear numerical answers.

The lack of numbers is hard for a lot of companies to deal with.  How exactly do you turn qualitative data into something meaningful?  How do you avoid squeezing the data to fit what you wanted it to say?

“Everyone is concerned about security”

At my Lean Startups presentation at Web 2.0 Expo last week, I said “The customer is the expert; you’re just a notetaker.”  Which, naturally, was taken out of context by quite a few people.

What I mean, though, is that when you conduct customer interviews, you need to sit back and not say things that will influence the interviewee’s responses.  If you agree too enthusiastically, or go from sounding interested to not interested, people will pick up on those cues and (consciously or not), tailor what they say next to make you happy.

I was a psychology major in college, and we had to practice before we could run experiments with human subjects – using a neutral tone of voice, not varying our script, giving very non-committal responses.  This isn’t much different.

When I worked at Yodlee, many of our potential customers felt that their end-users would be reluctant to share their passwords with us.  One large financial institution cited this as proof: in a recent user survey, over 90% of customers had said they were “concerned” or “very concerned” about data security.

Well, of course.  Everyone is concerned about security.  There’s no cost to saying you’re concerned about security.

But if you asked those same people if they’d be willing to pay $ for more security, or give up features in exchange for security, suddenly they’re not so concerned.  In our case, we showed them the product and explained the features and asked “is there anything that would prevent you from wanting to use this?”  No.  “What about security concerns?”  Well… no, we trust you.

It’s all about how you frame the question.  Be as neutral as possible.  Ask a friend outside the company to help you spot potential biases in your wording.

“A friend-of-a-friend said…”

The reason anecdotes get a bad rap is because most of them are full of holes.  When you have an interesting story, make sure you get the background:

• Who said it?
• What is their role?
• Is there anything about the person or their company that is outside your expected target market? (i.e. they’re a nonprofit and you usually sell to commercial companies)
• Might they be a nutcase?  (i.e. that guy who says he never accepts cookies, belongs to no social networks, and only browses the web through an anonymizer… you can probably safely ignore his concerns about security.)

This is critical because otherwise it’s going to seriously frustrate you when you see seemingly contradictory statements in your notes — oh, then you notice all the marketing people have one set of concerns, the engineers have a different set.

“Let me be clear: are you saying…?”

It is really exciting when a customer says something that validates your hypothesis.  Unfortunately, sometimes you want that so much that you conveniently mishear part of what they’re saying.

When someone starts to agree with you, first, write down everything they say.  Don’t try to pick out the important parts during the interview.

Next, repeat back to them what you think they were expressing: “Let me be clear: are you saying that your deployment takes a week and that is too long?”

Don’t stop there – proceed with a modified 5 Whys approach.  (Modified because you don’t want to sound too annoying.)  Even once you know the “pain”, you need to know WHY it’s a pain.

For example:

“Surveying customers is a pain.”

“Why?”

“It takes so long to write a survey… it’s not worth the time.”  (Problem: too time-consuming?)

“Why isn’t it worth the time?”

“Because so few people respond, we don’t really get useful data.”  (Aha! TOTALLY different problem: not effective!)

Commit yourself to revisiting the data

Once you start getting a fair amount of customer data, you’ll have a picture in your head of what the consensus from these customers is.  That picture in your head is highly subject to wishful thinking.

Commit to re-read your raw, unedited notes, after every few interviews to make sure you and reality stay in touch.

Popularity: 2% [?]

Popularity: 2% [?] blog comments powered by Disqus