Better Product Managers, and Product Management

This is another post about “later-stage customer development”.

You’ve probably heard the “medicine vs. vitamin” analogy of products — that is, to maximize your chances at success, it’s easier to sell medicine (that fix a specific, painful ill) than vitamins (that offer a vague promise of ‘feeling better’).

But I’ve realized there’s another category of product – the “gym membership” product:

• Your customers have accepted that they have a problem they really want to deal with (even if it’s not ‘life-or-death’).
• Of course there’s a free alternative, and some people get along just fine using it, but most people need the accountability, the support, and the motivation of a paid solution.
• Some people will use it religiously every day, some will use it once a week, and some will use it gung-ho for a week and then peter out for a few months.
• Some people will actually never use it, even though they keep paying you.  They’ll kind of resent you for it, though.
• There are a lot of different ways to use it to meet your goals. Some people walk in full of purpose, and know exactly how they need to use it.
• Most people need suggestions on how to use it effectively.  Otherwise they use it poorly or just wander around aimlessly.  They’ll be unsatisfied and rate you poorly.
• But if you force people through a long “initiation” session, they’ll find that obnoxious.
• Once people feel comfortable using it, they’ll find it gives them a lot more energy.  They’ll wonder how they ever got along without it!

You could also call these aspirational products.  We sign up because we want to be that kind of person.

Unlike most products — where the first and biggest challenge is getting people to give a damn enough to give you five seconds of their attention  — it isn’t that big a challenge to get people to sign up.   They see the equivalent of “better body in 90 days” and that’s enough to get them to click.

The challenge is getting people to get started and come back.   To do that, you have to anticipate and answer customer questions.  And the more subtly you do this, the more your app will feel intuitive/”just works”/delightful.

These are the first two questions that I’m trying to answer right now for both KISSmetrics and KISSinsights.

Question 1: What do I do first?

You probably have a pretty good sense for, in an ideal world, what you wish your customers would do first in order to get the most value out of your product.  However, you may be wrong.  This is where, now that you’re in later-stage customer development, it can be tremendously useful to walk someone through using your product the way you think it should be used.

One of two things is likely to happen: they’ll say, “oh, that makes sense – why doesn’t the app tell me I should start by doing that?” or they’ll say “that doesn’t make sense – why can’t I do X first?”

For KISSmetrics, we heard a lot more of the former, so we built a first-user experience designed to tell you how you should start.  Of course, we’re continuing to realize that there’s a lot more the app could be “telling” people, so we continue to tweak.  For KISSinsights, I suspect the latter is more accurate, which means we’ll need to make bigger changes to the first-user experience.

Question 2: How are other people using it?

I think this question really encompasses three things: “I want proof that other people are actually getting value from this”, “I want to see how much trouble they had to go through to get results”, and just plain curiosity.

I posted some examples for KISSinsights, which seems to have been a big help already (judging by the number of emails I’ve gotten since then).

One thing I’m still pondering, though — how do you identify the “goes to the gym every day” customer vs. the “tries to make it once or twice a week” customer, when they’ve just started to use your product?

Dealing with these questions has meant a very different kind of customer interview.  I wrote before that I was wary of telling the customer “here’s our product and here’s how you should use it” – now I think the way to think about it might be:

“Traditional” product development: “Let me tell you about us”

Early-stage customer development: “Tell me about you”

Later-stage customer development: “Let me tell you about other people like you“

You know the phrase – to eat your own dogfood – meaning, to use your product thoroughly so you recognize when the workflows are awkward or the help text is unhelpful.

But what happens is that we’re eating dogfood with a side order of the Curse of Knowledge.  We know how our products ought to behave and we know what they’re capable of, and that affects our impression of how easy they are to use.

So what we really need to do is eat someone else’s dogfood – that is, experience our products as though we were someone else, a someone who lacks all the inside knowledge that we have.

Here’s what I did last week:

• I offered to set up customers’ KISSmetrics funnels for them (this offer still stands, if you’re interested)
• Customers who agreed, I walked through their sites and identified the key workflows that I believed they ought to be tracking
• I created their funnel events and reports, and checked back to troubleshoot them
• I followed up (via email or phone) to explain what I’d done with their accounts and why I recommended using the product in this way

And you know what?

1. It was a LOT of work.
2. It was totally and completely worthwhile.

Being completely unfamiliar with their sites and goals put me in a similarly handicapped position as the customer who is unfamiliar with my product.  It became painfully obvious that certain copy was being ignored and that some features were grouped poorly.  There were several configurations where I thought, this would be much easier if we’d provided a visual example instead of trying to write inline help.

Equally valuable was the followup.    I would say “I’d recommend you use the product in this way–”, and someone would respond with “–but we can’t, because of this detail of how our application works”.   A step-by-step walkthrough primes the customer for a lot of little “but…”s and “why?”s that might never make it into a bug report or feedback email.

–

As I’ve written this, I think “but doesn’t this sound like the ‘traditional’ product approach?  Telling the customer, “here’s our product and here’s how you should use it” instead of understanding their needs?

And I think the difference is that these conversations were preceded by months of customer development, and conducted (as best I could) as a conversation.   This is where tone makes a big difference.  “Here’s how you should use our product” is very different from “Based on [assumptions], I recommend you use our product in this way – let me know what you think as I explain the details.”

There has been a lot written about early stage customer development – finding your market, identifying pain points and constraints, creating a vision that reflects customer needs without being ‘what customers ask for’.  But I’m finding that later stage customer development is a bit of a different beast.

It’s the time to move beyond abstractions and into specifics.  You’ve proven your case in theory; that’s why customers are using your product.  But there are definitely gaps in practice between how you envision customers using the product and how they actually are.  I’ll continue to write more as I learn more.

“The plural of anecdote is not data.” — Frank Kotsonis

That’s a good way to shut down someone who is trying to use sparse “I know someone who…” stories to second-guess a decision.  But in general, it’s not true.  The plural of anecdotes IS data – as long as you’ve collected them properly and challenged them rigorously.

A couple weeks ago I wrote:

It’s tempting to try and turn as many questions as possible into quantitative ones, because those are easier to measure through unobtrusive means – web analytics, looking for data patterns, surveys – and give clear numerical answers.

The lack of numbers is hard for a lot of companies to deal with.  How exactly do you turn qualitative data into something meaningful?  How do you avoid squeezing the data to fit what you wanted it to say?

“Everyone is concerned about security”

At my Lean Startups presentation at Web 2.0 Expo last week, I said “The customer is the expert; you’re just a notetaker.”  Which, naturally, was taken out of context by quite a few people.

What I mean, though, is that when you conduct customer interviews, you need to sit back and not say things that will influence the interviewee’s responses.  If you agree too enthusiastically, or go from sounding interested to not interested, people will pick up on those cues and (consciously or not), tailor what they say next to make you happy.

I was a psychology major in college, and we had to practice before we could run experiments with human subjects – using a neutral tone of voice, not varying our script, giving very non-committal responses.  This isn’t much different.

When I worked at Yodlee, many of our potential customers felt that their end-users would be reluctant to share their passwords with us.  One large financial institution cited this as proof: in a recent user survey, over 90% of customers had said they were “concerned” or “very concerned” about data security.

Well, of course.  Everyone is concerned about security.  There’s no cost to saying you’re concerned about security.

But if you asked those same people if they’d be willing to pay $ for more security, or give up features in exchange for security, suddenly they’re not so concerned.  In our case, we showed them the product and explained the features and asked “is there anything that would prevent you from wanting to use this?”  No.  “What about security concerns?”  Well… no, we trust you.

It’s all about how you frame the question.  Be as neutral as possible.  Ask a friend outside the company to help you spot potential biases in your wording.

“A friend-of-a-friend said…”

The reason anecdotes get a bad rap is because most of them are full of holes.  When you have an interesting story, make sure you get the background:

• Who said it?
• What is their role?
• Is there anything about the person or their company that is outside your expected target market? (i.e. they’re a nonprofit and you usually sell to commercial companies)
• Might they be a nutcase?  (i.e. that guy who says he never accepts cookies, belongs to no social networks, and only browses the web through an anonymizer… you can probably safely ignore his concerns about security.)

This is critical because otherwise it’s going to seriously frustrate you when you see seemingly contradictory statements in your notes — oh, then you notice all the marketing people have one set of concerns, the engineers have a different set.

“Let me be clear: are you saying…?”

It is really exciting when a customer says something that validates your hypothesis.  Unfortunately, sometimes you want that so much that you conveniently mishear part of what they’re saying.

When someone starts to agree with you, first, write down everything they say.  Don’t try to pick out the important parts during the interview.

Next, repeat back to them what you think they were expressing: “Let me be clear: are you saying that your deployment takes a week and that is too long?”

Don’t stop there – proceed with a modified 5 Whys approach.  (Modified because you don’t want to sound too annoying.)  Even once you know the “pain”, you need to know WHY it’s a pain.

For example:

“Surveying customers is a pain.”

“Why?”

“It takes so long to write a survey… it’s not worth the time.”  (Problem: too time-consuming?)

“Why isn’t it worth the time?”

“Because so few people respond, we don’t really get useful data.”  (Aha! TOTALLY different problem: not effective!)

Commit yourself to revisiting the data

Once you start getting a fair amount of customer data, you’ll have a picture in your head of what the consensus from these customers is.  That picture in your head is highly subject to wishful thinking.

Commit to re-read your raw, unedited notes, after every few interviews to make sure you and reality stay in touch.

I’ve seen a lot of great examples of situations where a company built out a product or feature and then no one cared.  But what about the reverse?

“What if we don’t build the right (beautiful/efficient/fully-featured/scalable) thing,

and because of that, something horrible happens?”

We try pretty hard to be lean at KISSmetrics, but we still have a lot of internal debate sometimes around whether and when we should build things.  These are 4 arguments we’ve had over the last 6 or 7 months, and in all cases, Lean was right.

We need a First User Experience as part of the initial KISSmetrics beta.

The argument: I’ve written multiple times about the importance of a good guiding first user experience and I looked at what we were starting out with and said, “everyone is going to show up, get confused, and never come back.”  I mean, I totally thought I was right on this one.

What happened: Our initial beta customers were classic early adopter types who didn’t mind experimenting with our API and writing some Javascript, and were able to use the product and provide tons of valuable feedback to us, all without requiring a wizard to guide them through the process.

Now, 6+ months later, we’re starting to get beyond that initial “early adopter” population, and we’re definitely seeing that beta customers would benefit from a guided setup.  So we’re investing the time in building out a proper first user experience.  But did we need it back in October ‘09?  No.

Lean 1, Fat 0.

We need more granular controls around how much data we show in the initial KISSmetrics beta.

The argument: In the interest of releasing quickly, the initial version only showed your funnel for your last 10,000 events.  No date filtering, and no historical data.  For high-traffic sites, this might be less than a day’s worth of activity!  Come on, we really should let people show the last week or the last month, or give them date-pickers – right?

What happened: About half of our earliest beta customers were smaller startups, so the low amount of data wasn’t an issue.  The other half weren’t crazy about it, but they gave us useful feedback on how they wanted to dice up their data.

Now, 6+ months later, we just released custom date pickers this morning.  Yeah, people have been complaining about this for awhile – but they’ve stuck around and continued using our product in the meantime.

Lean 2, Fat 0.

We need a super cool relative date picker in Sharefeed.

The argument: Shouldn’t a product that’s built around scheduling have a really nice, dead-simple way of indicating dates?  Like, if I want a tweet to go out tomorrow, why can’t I just type in “tomorrow”?  (The classic “wouldn’t it be cool if…” problem.)

What happened: Other priorities intervened and no developers got past the initial experimenting state.  We’ve been using Sharefeed internally for over 4 months now, and the lack of a cool date picker is not a pain point.  Not a single person has emailed us complaining about our date picker.

Lean 3, Fat 0.

“KISSinsights – there’s not much there, is there?”

The argument: OK, I don’t think those were Hiten’s exact words, but they weren’t far off.   The very first version was missing most of the survey configuration options, an “about” page, password reset, and the admin pages (which only we could see) had typos and stray list element bullets.  It was not pretty.

What happened: No one (except me) noticed the lack of an “about” page.  One person emailed me so we could manually reset his password.   By getting that product out in the hands of real people quickly, we knew exactly what features to build/refine next.

2 months later, we have a still-minimal but definitely-viable product, and 700+ beta users.

Lean 4, Fat 0.

I’m speaking at Startup Lessons Learned tomorrow – hope to see you there!