Better Product Managers, and Product Management

“But all you have to do is option-click on that little “pi” symbol in the corner…”

I have a challenge for you.

“By the way, did you know that you could do X with our app?”

1. Write down a list of the coolest features in your application — the most useful, the ones that differentiate you from the competition, the ones with the highest ‘delight’ factor.
2. For the next week, each time you communicate with a customer, pick one of those features and ask that customer if they’re familiar with it.
3. Write down their response.

A lot of them are going to say “no”.

This may because they just started using your product, or because they’re not very tech-savvy, or because that particular feature is not relevant for them.   But more likely, it’s because those features are about as accessible as an Easter Egg:

• Your site doesn’t mention that those features are available
• There’s no inline help explaining why someone should try them
• They require a little workaround in order to use them… which is not explained on your site
• They’re only available to paid subscribers – but neither free nor paid subscribers know this
• They look ’scary’ (because it’s not clear what activating them will do.  Is there a chance it will lose data or undo previous tasks?)
• There are no demos, screenshots, or any other way to preview them

This is bad.

I mean, you built these features already.   You probably had a good hypothesis as to why they’d be useful.  You invested development and design time into implementing them.  But while they’re in “Easter Egg” state, you will not learn from them.

You can’t know if a feature is useful if no one knows it’s there.  And if you kill a feature while it’s in this state, you’re going to draw false conclusions.  Sure, no one complained.  You don’t complain about missing something if you didn’t know it existed.

So, take the challenge.  Figure out what customers don’t know about.  Anything “unknown” that is core to your business — that had better become the next focus of your attention: figuring how, as quickly as possible, to start “un-Easter Egg-ing” those features so both you and your customers benefit.

As last week’s commenters pointed out, there’s a challenge in offering multiple choices vs. asking for freeform responses: You might get more responses but still be missing the root cause of customers’ concerns/problems/ideas.

They’re right.  Freeform answers alone are flawed.  Multiple choice options alone are flawed.  You need to use them both together in order to generate unstoppable, reliable hybrid feedback!

There are two approaches to cultivating your hybrid feedback – pick the one that is most relevant to your situation.

Interview First, Then Survey

Use this method when: You have no idea what you’d even suggest as multiple choice options.

Example: People are really excited when they sign up for your book exchange website. But almost no one is completing a successful exchange, and honestly, you have no idea why. You can’t see a pattern to where people are dropping out of your workflow, and you’re not getting a lot of bug reports or complaints in your support inbox.

Where to start?: Talk to people.

You can call and ask questions about their book exchange needs (see earlier post on “what you should be learning?”), or ask a couple people to go through your website while you watch over their shoulder or using UserTesting.com, or some combination of the three.

Once you’ve talked to 5-10 people, you will usually see some patterns – concerns or obstacles that are affecting more than one person and that seem pretty plausible based on what you know of your own product.

You may find that the problems reported in these first 5-10 conversations/user tests are so fundamental that they’re actually preventing you from getting deeper feedback!  (For example, if people are having problems logging in, they’ll never get to use your core product enough to give you useful feedback on it.)   In that case, skip the survey and start fixing!

Or you can use this initial feedback to populate your multiple choice options, and run the numbers.

Personally (and yes, this is my opinion, not proven data), when I see a multiple choice question from a company and all of the options are really well-written and show a deep understanding of their product, I feel like they really care and am more likely to spend extra time writing out an “Other” response.

Survey First, Then Interview

Use this method when: You want to guide the conversation to specific, relevant options. OR You have a pretty good guess as to what the potential responses are.

Example: You have several large features on your product roadmap, all of which are aligned with your product vision, have supporting market research, and will provide different customer benefits.  But you’d like to validate that they really will be valuable to the customer, and get some subjective feedback on which one will “delight” your customers the most.

Where to start?: Survey.

Questions that may work for this context:

• Which of [list options] features are you most excited about?
• Which of [list tasks] do you use most frequently?
• Would one of these [list options] have convinced you to complete your purchase?
• Would you be willing to be a beta tester for one of these upcoming features [list]
• Have you experienced one of these issues [list]?  How did it affect you?

Once you’ve gotten 20-30 responses, you will usually see a clear winner (or two) emerge.

But now, you have to make sure you properly “decode” that feedback.  You need to understand the “what else” and the “why”.

What else? It may be true that the majority of your customers are most excited about [feature X], but they are assuming that it will magically “just work”.  It’s your job to understand and solve for things like Will this change my workflow?  Will this involve new people or exclude people who previously used it?  My boss is used to [competitor product] so she’ll want it to work like that does.

Why? It may be true that everyone is selecting “add customer reviews and commenting” as their preferred next feature.  But it may not be for the reason you assume.

Let me illustrate with a personal example:

CA: “I didn’t end up buying that [group buying site] Pilates deal, even though it was a good discount.”

Friend: “Why not?”

CA: “It wasn’t clear when you could use it — It would’ve been a waste of money if the classes were only at a time when I couldn’t go.  They should make the vendors give them more information or update their websites to be more clear.”

Friend: “Did you realize that [group buying site] offers a money-back guarantee?  You could’ve bought it and returned it if it wasn’t the right schedule.”

CA: “Ohhhh.  I didn’t know that!  I would’ve bought it if I’d known that!”

Which is easier: getting hundreds of vendors to submit information, and then wrangling that information into a CMS; or just highlighting a money-back guarantee icon?

So that’s why you then follow up with interviews.  Get details, try to disprove your assumptions, and then you’ll finally have the full understanding of your feedback.

It would be nice if we could just say “give us feedback” and our customers could just turn on the faucet and dump out all of their concerns and experiences and ideas.  But that’s not the way our brains work.

Let me illustrate.

Quick, without Googling, what were the venues for the last 4 Summer Olympics?

If you’re like me, a neat list of 4 cities did not immediately pop into your head.  In fact, even though I’m a pretty die-hard Olympics-watching fan, my brain is struggling to remember anything before Beijing.

But what about if I asked the question in this way?

From the list below, pick the venues for the last 4 Summer Olympics:

• Seoul
• Beijing
• Los Angeles
• Montreal
• Athens
• Rio de Janeiro
• Sydney
• Tokyo
• Johannesburg
• Atlanta

Easier this time, right?  Beijing, Athens, Sydney, Atlanta.

That’s an illustration of recall vs. recognition in human memory.

Recall memory involves delving into your brain, looking for specific pieces of information, finding them, confirming that they are the information you wanted, and then putting them to use.

Recognition skips the first three steps.  You are somehow prompted with the specific pieces of information, then all you have to do is confirm that they are correct and put them to use.

In both cases, the information is stored somewhere in your brain.  If I had never heard of the Olympics, I wouldn’t be able to pick the correct cities out of a list – all of the list items would’ve been similarly meaningless to me.

Why is this important?

As you might have guessed, recall memory is harder.  It’s legitimately harder for people to answer a question that doesn’t provide any contextual prompts.

If I used your product on Monday, and you email me on Wednesday saying “Any feedback?”, I’m likely to draw a blank.  Not because I don’t care about you, not because I’m a “stupid user”, but because I’ve got no context to use to dig into my recall memory.

Recall memory is like asking your brain to do push-ups.  Some people can’t do many or any pushups; others can, but we’d rather procrastinate and get around to it later.

So how do I ask for feedback in a “brain-friendly” way?

Use recognition.

Instead of asking “what should we build next?“, ask “which of these new features [list] would be most helpful to you, and why?” (Not only is this “brain-friendly”, it’s also “product manager-friendly”, allowing you to guide their feedback in alignment with your long-term product vision.  There’s no value in letting people ask for faster horses over and over again if you’re building Model Ts.)

If you suspect a specific page is confusing, don’t ask “what about the page was confusing?”, show them a picture of the page and let them point to the confusing parts.

Give customers a sentiment and allow them to agree (or disagree) and elaborate.  For example, “some customers felt this workflow was confusing – why do you think that is?”

Use immediacy.

If you ask a question while the customer is using your site, there’s nothing to remember – they just did, or are still doing, the thing you’re asking about.

In my experience, immediacy cancels out one of the biggest complaints I hear about customer feedback: “it’s useless because it’s all extremes, 1 star or 5 stars!”   This makes perfect “brain sense” – after you’ve completed an action or experience and time has passed, your memory becomes distorted towards the elements that left you with the biggest emotional impact – whether that’s joy at a gorgeous interface, or (more likely) fury at a confusing workflow.

But while your customer is experiencing something, they’re going through a series of impressions:  Ooh, nice refresh! Hmm, I don’t understand that sentence.  Oh good, this will do exactly what I wanted.  Wait, did that charge my credit card or not?  I wish this workflow was more straightforward.  This will make my life easier.  How do I undo what I just did?  Oh no!  Oh, phew, I guess it’s okay.

Wouldn’t you rather overhear that entire stream of impressions vs. a 3-star rating or “yeah, I kind of like your site”?

Use interactivity.

It’s not always possible (and sometimes downright intrusive) to get customer feedback while they’re in the middle of trying to get something done.

The next best method is to be interactive.  In-person, phone, or IM chat interviews allow you to provide context and prompts to make it easier for your customer to give you feedback. You can describe a feature “do you remember using the ’sort and select’ feature? Let me describe it.  OK, now tell me about how you used it…”  and then it’s far easier for the customer to provide valuable feedback.

(While this method is primarily qualitative, you can ask for 1-5 ratings on the phone just as easily as in a survey.  Even better, because you can confirm that the customer really understands what they’re rating instead of just picking “3″ because they don’t remember what that feature was.)

Use pictures.

There are times when you need more responses than you can scalably get through interviews.  In those situations, illustrate!  Before asking a question, include a screenshot or even a quick video, to remind customers what they’re responding to.

Unfortunately, neither SurveyMonkey nor Wufoo nor Google Forms support embedding images in between questions by default, so this is less easy than it should be.  UserTesting.com is one option, or you can hack something together with multiple Wufoo or Google Forms pages and stick your own images in between.

What’s your biggest obstacle to getting customer feedback?

It’s easy to improve on already-existing products.

I’m going to come right out and say that, because I think we give ourselves too much credit for simply improving upon the status quo. And it’s really not that hard.  If you use a product regularly, it’s not hard to notice the little flaws or omissions that would make it better.

So then we think, great, our product will be better.  We will do all the things that our competitor does well, AND we’ll fix those flaws and add in those missing pieces.

Congratulations, you now have a superior product.  What you do not have: customers beating down your door.

You’ve done the first step: make it better.  But you still have several more steps:  make it even better.  And then, make it even better than that.  And then, when you’re convinced you can’t improve it any more, make it just a tiny bit better still.

There are two reasons why you need to keep improving your product to a point that seems like fanaticism:

Everyone exaggerates.

Your website says “The best way to do X” or “The easiest way to do X”.  Guess what? So does everyone else’s.  Your competitor – the one whose shoddy product inspired you in the first place – makes the exact same claim on their website.

People are fundamentally wired to not change their behaviors.

(I hear the phrase “users hate change” a lot, and I dislike it – it sets up that us vs. them mentality, that “if only our customers were smarter, they’d get it” excuse.  It’s not that your dumb users hate change. It’s that everyone, even smart people who pride themselves on trying new things, default to retreading familiar patterns.    OK, rant over.  Back to the post.)

It takes work to try a new product.

It requires learning (even if the new product is simple).

It requires risk (what if I invest time and then the new product doesn’t work?)

It’s much easier to use the thing you know, even if you the thing you know kind of sucks.   So even if I know, objectively and absolutely, that your product is 20% better than the alternative, I won’t switch.  Even if I’d recoup the lost time within a week, it’s not worth changing my behavior.  If your product is 50% better?  Maybe.  If your product is 200% better?  Now we’re talking.

Now, if you’ll excuse me, I have two products to make better.  Much, much better.