Better Product Managers, and Product Management

You know the phrase – to eat your own dogfood – meaning, to use your product thoroughly so you recognize when the workflows are awkward or the help text is unhelpful.

But what happens is that we’re eating dogfood with a side order of the Curse of Knowledge.  We know how our products ought to behave and we know what they’re capable of, and that affects our impression of how easy they are to use.

So what we really need to do is eat someone else’s dogfood – that is, experience our products as though we were someone else, a someone who lacks all the inside knowledge that we have.

Here’s what I did last week:

• I offered to set up customers’ KISSmetrics funnels for them (this offer still stands, if you’re interested)
• Customers who agreed, I walked through their sites and identified the key workflows that I believed they ought to be tracking
• I created their funnel events and reports, and checked back to troubleshoot them
• I followed up (via email or phone) to explain what I’d done with their accounts and why I recommended using the product in this way

And you know what?

1. It was a LOT of work.
2. It was totally and completely worthwhile.

Being completely unfamiliar with their sites and goals put me in a similarly handicapped position as the customer who is unfamiliar with my product.  It became painfully obvious that certain copy was being ignored and that some features were grouped poorly.  There were several configurations where I thought, this would be much easier if we’d provided a visual example instead of trying to write inline help.

Equally valuable was the followup.    I would say “I’d recommend you use the product in this way–”, and someone would respond with “–but we can’t, because of this detail of how our application works”.   A step-by-step walkthrough primes the customer for a lot of little “but…”s and “why?”s that might never make it into a bug report or feedback email.

–

As I’ve written this, I think “but doesn’t this sound like the ‘traditional’ product approach?  Telling the customer, “here’s our product and here’s how you should use it” instead of understanding their needs?

And I think the difference is that these conversations were preceded by months of customer development, and conducted (as best I could) as a conversation.   This is where tone makes a big difference.  “Here’s how you should use our product” is very different from “Based on [assumptions], I recommend you use our product in this way – let me know what you think as I explain the details.”

There has been a lot written about early stage customer development – finding your market, identifying pain points and constraints, creating a vision that reflects customer needs without being ‘what customers ask for’.  But I’m finding that later stage customer development is a bit of a different beast.

It’s the time to move beyond abstractions and into specifics.  You’ve proven your case in theory; that’s why customers are using your product.  But there are definitely gaps in practice between how you envision customers using the product and how they actually are.  I’ll continue to write more as I learn more.

Popularity: 1% [?]

“The plural of anecdote is not data.” — Frank Kotsonis

That’s a good way to shut down someone who is trying to use sparse “I know someone who…” stories to second-guess a decision.  But in general, it’s not true.  The plural of anecdotes IS data – as long as you’ve collected them properly and challenged them rigorously.

A couple weeks ago I wrote:

It’s tempting to try and turn as many questions as possible into quantitative ones, because those are easier to measure through unobtrusive means – web analytics, looking for data patterns, surveys – and give clear numerical answers.

The lack of numbers is hard for a lot of companies to deal with.  How exactly do you turn qualitative data into something meaningful?  How do you avoid squeezing the data to fit what you wanted it to say?

“Everyone is concerned about security”

At my Lean Startups presentation at Web 2.0 Expo last week, I said “The customer is the expert; you’re just a notetaker.”  Which, naturally, was taken out of context by quite a few people.

What I mean, though, is that when you conduct customer interviews, you need to sit back and not say things that will influence the interviewee’s responses.  If you agree too enthusiastically, or go from sounding interested to not interested, people will pick up on those cues and (consciously or not), tailor what they say next to make you happy.

I was a psychology major in college, and we had to practice before we could run experiments with human subjects – using a neutral tone of voice, not varying our script, giving very non-committal responses.  This isn’t much different.

When I worked at Yodlee, many of our potential customers felt that their end-users would be reluctant to share their passwords with us.  One large financial institution cited this as proof: in a recent user survey, over 90% of customers had said they were “concerned” or “very concerned” about data security.

Well, of course.  Everyone is concerned about security.  There’s no cost to saying you’re concerned about security.

But if you asked those same people if they’d be willing to pay $ for more security, or give up features in exchange for security, suddenly they’re not so concerned.  In our case, we showed them the product and explained the features and asked “is there anything that would prevent you from wanting to use this?”  No.  “What about security concerns?”  Well… no, we trust you.

It’s all about how you frame the question.  Be as neutral as possible.  Ask a friend outside the company to help you spot potential biases in your wording.

“A friend-of-a-friend said…”

The reason anecdotes get a bad rap is because most of them are full of holes.  When you have an interesting story, make sure you get the background:

• Who said it?
• What is their role?
• Is there anything about the person or their company that is outside your expected target market? (i.e. they’re a nonprofit and you usually sell to commercial companies)
• Might they be a nutcase?  (i.e. that guy who says he never accepts cookies, belongs to no social networks, and only browses the web through an anonymizer… you can probably safely ignore his concerns about security.)

This is critical because otherwise it’s going to seriously frustrate you when you see seemingly contradictory statements in your notes — oh, then you notice all the marketing people have one set of concerns, the engineers have a different set.

“Let me be clear: are you saying…?”

It is really exciting when a customer says something that validates your hypothesis.  Unfortunately, sometimes you want that so much that you conveniently mishear part of what they’re saying.

When someone starts to agree with you, first, write down everything they say.  Don’t try to pick out the important parts during the interview.

Next, repeat back to them what you think they were expressing: “Let me be clear: are you saying that your deployment takes a week and that is too long?”

Don’t stop there – proceed with a modified 5 Whys approach.  (Modified because you don’t want to sound too annoying.)  Even once you know the “pain”, you need to know WHY it’s a pain.

For example:

“Surveying customers is a pain.”

“Why?”

“It takes so long to write a survey… it’s not worth the time.”  (Problem: too time-consuming?)

“Why isn’t it worth the time?”

“Because so few people respond, we don’t really get useful data.”  (Aha! TOTALLY different problem: not effective!)

Commit yourself to revisiting the data

Once you start getting a fair amount of customer data, you’ll have a picture in your head of what the consensus from these customers is.  That picture in your head is highly subject to wishful thinking.

Commit to re-read your raw, unedited notes, after every few interviews to make sure you and reality stay in touch.

Popularity: 2% [?]

I’ve seen a lot of great examples of situations where a company built out a product or feature and then no one cared.  But what about the reverse?

“What if we don’t build the right (beautiful/efficient/fully-featured/scalable) thing,

and because of that, something horrible happens?”

We try pretty hard to be lean at KISSmetrics, but we still have a lot of internal debate sometimes around whether and when we should build things.  These are 4 arguments we’ve had over the last 6 or 7 months, and in all cases, Lean was right.

We need a First User Experience as part of the initial KISSmetrics beta.

The argument: I’ve written multiple times about the importance of a good guiding first user experience and I looked at what we were starting out with and said, “everyone is going to show up, get confused, and never come back.”  I mean, I totally thought I was right on this one.

What happened: Our initial beta customers were classic early adopter types who didn’t mind experimenting with our API and writing some Javascript, and were able to use the product and provide tons of valuable feedback to us, all without requiring a wizard to guide them through the process.

Now, 6+ months later, we’re starting to get beyond that initial “early adopter” population, and we’re definitely seeing that beta customers would benefit from a guided setup.  So we’re investing the time in building out a proper first user experience.  But did we need it back in October ’09?  No.

Lean 1, Fat 0.

We need more granular controls around how much data we show in the initial KISSmetrics beta.

The argument: In the interest of releasing quickly, the initial version only showed your funnel for your last 10,000 events.  No date filtering, and no historical data.  For high-traffic sites, this might be less than a day’s worth of activity!  Come on, we really should let people show the last week or the last month, or give them date-pickers – right?

What happened: About half of our earliest beta customers were smaller startups, so the low amount of data wasn’t an issue.  The other half weren’t crazy about it, but they gave us useful feedback on how they wanted to dice up their data.

Now, 6+ months later, we just released custom date pickers this morning.  Yeah, people have been complaining about this for awhile – but they’ve stuck around and continued using our product in the meantime.

Lean 2, Fat 0.

We need a super cool relative date picker in Sharefeed.

The argument: Shouldn’t a product that’s built around scheduling have a really nice, dead-simple way of indicating dates?  Like, if I want a tweet to go out tomorrow, why can’t I just type in “tomorrow”?  (The classic “wouldn’t it be cool if…” problem.)

What happened: Other priorities intervened and no developers got past the initial experimenting state.  We’ve been using Sharefeed internally for over 4 months now, and the lack of a cool date picker is not a pain point.  Not a single person has emailed us complaining about our date picker.

Lean 3, Fat 0.

“KISSinsights – there’s not much there, is there?”

The argument: OK, I don’t think those were Hiten’s exact words, but they weren’t far off.   The very first version was missing most of the survey configuration options, an “about” page, password reset, and the admin pages (which only we could see) had typos and stray list element bullets.  It was not pretty.

What happened: No one (except me) noticed the lack of an “about” page.  One person emailed me so we could manually reset his password.   By getting that product out in the hands of real people quickly, we knew exactly what features to build/refine next.

2 months later, we have a still-minimal but definitely-viable product, and 700+ beta users.

Lean 4, Fat 0.

I’m speaking at Startup Lessons Learned tomorrow – hope to see you there!

Popularity: 4% [?]

What is customer development?

Customer development is the opposite of product development, or “if we build it, they will come” thinking.   We all know that’s not the case – often you build it, and no one cares.

The idea is to validate that you have a market, with problems, that they’re willing to pay to solve — and then build something that solves those problems.   The emphasis is on learning and discovery before you write a 50-page spec or spend months writing code. Read Eric Ries’ post  What Is Customer Development for a lot more detail.

“This doesn’t sound like anything new,” you may be thinking, and you’re pretty much right.   There are plenty of examples of successful companies who used interviews and cheap prototypes to validate ideas before committing to building them.

What has changed is the availability of free/cheap tools and direct access to customers through social media, making it more practical for an individual product manager to do this on their own, on a skunkworks basis even.

But asking the customer what they want never works –

I know.  Customers tend to ask for “me-too” features, or they ask for something and then after you build it, say “I know we asked for this, but it isn’t what we really wanted.”

Customer development isn’t asking customers what they want – it’s seeking to understand what they need, how they work, where their pain points and highest priorities are.  Customers may not be able to articulate what they want, but they can’t hide what they need.

What will I learn?

The short answer: how people are really getting a task done, who is doing what, and why it sucks.

The more detailed answer: What you should be learning from customer development interviews

How is this different from usability testing?

Usability testing teaches you whether or not people are capable of using your product; it doesn’t tell you how likely they are to actually buy it.

In usability testing, you learn about the product you’ve already built.   If your product hypothesis was wrong, it’s kind of too late to do much about it.  In customer development, you learn about your customer before you build the product.  That way, if your product hypothesis is proven wrong, you can change course quickly without having wasted resources and time.

How long will it take?

I don’t know.

Sorry, but that’s the truth.  I can’t guarantee that this methodology will get you useful answers in two days or two weeks.  I can guarantee that you’ll start learning useful new tidbits from your first customer interview, but it may not be something actionable.

At KISSmetrics, I quickly learned that the reason so many companies loved our Survey.io customer development survey is that “I don’t have to write anything – you tell what I should be asking and how to ask it”.

People don’t like to write – in fact, they’ll put off a task if it requires them to think and write something.  That was a good insight, but not enough (on its own) to drive any new product decisions.  We needed more of those insights before they added up to something we could base a product on.

For this reason, customer development needs to be a process — not a project.  Always be listening, always be asking why, and always be testing hypotheses, and it will pay off.  You just never know when.

How do I get started?

If you already have existing customers, reach out to one today and schedule some time to talk with them about how they get their jobs done.  Ask about how they use your products, but also take a step back and ask about the general task that your product is supposed to help them with (i.e. if your product is a bug tracker, ask them about QA testing in general).

You don’t have a hypothesis to test (yet), so this is just an open-ended conversation.  Let the customer talk.   The Evolution of the Customer Development Interview has good tips.

You may be at a company who has existing products and customers, but you don’t have access to them.  As insane as this is, it’s not uncommon to try and lock the product managers up in a little room to wrangle up requirements somehow.   The fastest way to get access is probably to bargain directly with sales.

Ask to join them on a sales visit or conference call.  Offer to help with demos.  If you have to promise to be a “fly on the wall” in order to make them feel comfortable, do it.    In my experience, customers love having a product manager in the meeting – they feel like you’re an ally who isn’t just trying to sell them — and as a result, they’re often MORE likely to buy.  (Tell your salesfolk that.)

How can I find customers before I’ve even built a product?

How were you planning on finding them after you’ve built a product?

Seriously, this is the million-dollar question, and the answer is not “we’ll build it and they’ll come” or even “we’ll advertise a LOT” — unless you’re Apple, Microsoft, Google, or Facebook, you don’t have the dollars or distribution to make those work.

So you have to get creative.   I’ve previously written about how to find prospective customers using methods that take less than $20 and a couple days.

How do I convince my boss that this is a good use of my time?

Unless your boss is already a devotee of Toyota Lean Manufacturing processes or Four Steps to the Epiphany, start out skunkworks style. Remember, bosses like solutions, not problems.  Don’t say “we need to be doing X”, figure out how it can be done and prove the benefits.

Carve out some time, do some customer interviews.  Work with a designer to throw together a fake splash marketing page for a new product and get feedback.  Once you’ve learned something useful, use that to get buy-in for making this a permanent part of your process.

How does this change when I have an established brand and customers?

You may need to create somewhat more polished splash pages/demos in order to not hurt your credibility.

You may also feel more comfortable testing concepts without connection to your company name.  Using another domain name is a good option and worth the time to set up if you’re going to be running experiments frequently.  But you can go even simpler and use tools like shared Google docs (create a new Google account that isn’t linked to your name) or Skitch to post screenshots privately.

Where can I learn how to do this?

Books:

• The Four Steps to the Epiphany
• Tuned In: Uncover the Extraordinary Opportunities That Lead to Business Breakthroughs
• Inspired: How To Create Products Customers Love

Blog posts:

• Why Your Customer Feedback is Useless by Laura Klein
• 7 More Ways People Suck at Customer Development Interviews by Laura Klein
• The Evolution of the Customer Development Interview by Sachin Agarwal
• Customer Development Interviews How-to: Finding People by Cindy Alvarez
• Customer Development Interviews How-to: What You Should be Learning by Cindy Alvarez
• How I Learned to Grok Customer Development by Ash Maurya
• How I Built My Minimum Viable Product by Ash Maurya
• Customer Development Checklist for my Web Startup by Ash Maurya
• What Is Customer Development? by Eric Ries
• When NOT to listen to your customers; when NOT to rely on split-tests by Eric Ries
• How to Develop Your Customers Like You Develop Your Product by Nivi at VentureHacks
• Customer Development is not a Focus Group by Steve Blank
• Customer Development – Past, Present, and Future by Steve Blank
• Customer Development Update (and why I’m sticking with 1-on-1 talks) by Giff Constable
• Seller Beware: Customers Have Their Own Agenda by Brant Cooper
• How to Find Early Adopters by Brant Cooper

People to follow on Twitter:

• @ashmaurya
• @brantcooper
• @cindyalvarez
• @ericries
• @giffconstable
• @hnshah
• @lauraklein
• @richcollins
• @sgblank
• the #leanstartup and #customerdev hashtags

Popularity: 30% [?]