Better Product Managers, and Product Management

If you read last week’s blog post and followed through, you now have:

• picked a fast and measurable improvement to try
• figured out what part of your audience was going to see it
• launched it

You might think the hard part is done now.

But this is exactly where most people fall apart — they fail to close the loop.

Closing the Loop

By “closing the loop”, I mean:

• Validating that your change worked
• Measuring how big the effect was
• Following up with your team so they know what happened and what you learned

The last step is what makes or breaks your ability to keep making your product better.  You can brute-force your team through a couple rounds of improvement without it, but it’s not sustainable.

When you have a developer who spent a full day reworking a feature, and they never hear that whether it made any difference on usage, it’s going to be harder to keep them motivated.  This is especially important when you’re doing quick iterations — workflow changes and copy tweaks are not, generally, the meaty problems that developers are intrinsically excited to work on.   But if those “boring” changes lead to more customers, that’s something to be excited about.

When you’re solving an unknown problem — and really, the vast majority of “making it better” problems are unknown — you benefit from using all the brains in your company.  But when you don’t follow up with what you learned, you’re handicapping those brains.  Suppose you learned that removing configuration options increased conversion, but you don’t share that with your team.  How are they going to know to stop thinking up power-user features?

If you are like me, you probably think you are being very clear in your followups, and you probably are not.

How to be really clear when closing the loop

We had the best results with KISSinsights when I gave a verbal followup in this approximate format:

• Remember, our goal was to improve [metric]
• We started at [number]
• We deployed [these changes] to try and move that number
• Now we’re at [number]
• We think these changes did work / might work but we want to watch for another week / didn’t work
• Here’s any anecdotal evidence from customer interviews, support emails, tweets, etc.

My goal for moving forward is to share this information in writing as well – discussing it in meetings is great, but lots of people just do better grokking the written word.   I also think, as we move from very short-term goals to longer-term goals, it will be more useful to have easily-searchable written records to look back on.

Popularity: 4% [?]

If you read last week’s blog post and followed through, you now have:

• identified a problem and written it down
• found a way to measure the problem
• explored some possible reasons why there’s a problem
• brainstormed some possible solutions

Now what?

Look at speed.

When you first start trying to improve, you really don’t know what is going to work.  So don’t try to evaluate the potential solutions on “merit” – you don’t have merits yet, just opinions.

Instead, look at what can be put in front of customers the fastest.   Copy changes are often faster than single-page UI changes, which are often faster than workflow changes, which are often faster than back-end changes.  (When I say ‘speed’, I really mean a combination of level of effort and risk — a single line of code change may take an engineer 2 minutes to write, but if it requires tweaking your test scripts or thinking about how it might impact existing customers in unknown ways, it’s not a “high-speed” change.)

Look at measurability.

If you made this change, how would you know if it worked?  If you can’t answer that, it’s not measurable enough for you to start with.  (Your instincts on how to effective measure success will get better over time; I’m just saying, if you don’t grok it now, start with something simpler.)

Pick one.

If your team is like my team, they’re smart and they came up with a bunch of creative solutions to try out.   So now you’re excited and thinking, how fast can we do all of these?

And the answer is: one at a time.

When you try to make multiple changes at once, it’s almost impossible to know the impact of each individual change.  Maybe Change A increased your conversion rate by 10%, but the Change B reduced it by 5%.  All you’ll see is a net 5% gain — which looks great on paper — but when you continue making more Change B-like changes you’ll just keep hurting yourself.

Don’t know which of your proposed fast and measurable changes to start with?  Flip a coin.  Just take action.

Feature/workflow change? Make it even smaller.

If you picked a feature change or workflow change to start with, challenge yourself to make it even smaller.  Can you build half of what you originally proposed and still learn something?   Setting a very short timeframe (“what can we do in one day?”) is a good way to enforce this.

Why make it smaller?  Because you don’t know if it will work yet.   You don’t lay a down big bet before you’ve seen your cards; you start small and double down when you see face cards (or minimize your losses when you fold).

Figure out who’s going to see it.

Everyone sees it. This is less scientific but a lot simpler.  And if you’re small or just getting started, it’s usually good enough.  Unless there’s a reason why your traffic/usage would significantly differ between 1-week or 2-week periods (i.e. over a holiday or after getting heavy press attention), you will know if “before” or “after” is better.

Random “B” group sees it. Reduces the risk that time periods are somehow different; tends to make companies feel more comfortable since it is more scientific.

Only new customers see it. Change is often initially hated, even if customers eventually realize it’s better.  If you think your customers will find ongoing changes upsetting, you can use new customers as guinea pigs and only roll out changes when you’re confident they’re a significant improvement.   Some risk that your “new customer” user base may be somehow different than your “existing customer” user base, though.

Communicate.

Make sure everyone on your team knows what “before” and “after” look like (taking screenshots of each and sharing them is one easy way to do this).  No one should be filing bugs on “B” because their screen is showing “A”, or sending inaccurate customer service email responses because they’re looking at “B” and the customer still has “A”.

Go!

Release it.  Be excited.  Make a note to check back on progress after 2 days (probably too short to notice a difference, but enough time to notice a potential catastrophe, like “B is generating a ton of customer support queries!”), after 1 week, after 2 weeks.

If you test one thing at a time but run a test every single week, you’ll get through 52 experiments in a year.  I guarantee that will move your numbers significantly.

Popularity: 6% [?]

There is something you want to improve about your site or product.  (If there wasn’t, you probably wouldn’t find this blog terribly interesting.)

Maybe you’re unhappy with the low percentage of visitors who continue past your homepage.  Maybe too many people are abandoning their full carts.  Maybe you’re getting a ton of customer support requests related to a specific feature.

But where do you start?  It can feel very daunting, which is why most people are not doing as much active testing and improvement as they should.

But don’t worry – it gets much easier when you have a plan.   We ran a ton of focused A/B tests on KISSinsights last year and more than doubled our homepage to signup conversion rate.  Here’s how we started:

Step 1:  Write it down.

No, you won’t “just remember”.  (And even if you do, the rest of your team certainly won’t).    Fire up a new doc and write down your problem:  Not enough visitors continuing past the KISSinsights homepage. And put the date at the top so you remember when you started.

Step 2: Quantify it.

How bad has the problem been over the past 7 days?  I like ‘past 7 days’ because it includes both weekdays and weekends — which tend to have very different usage patterns — but is still pretty recent, so it’s not affected by things you did months ago.

If you’re using a percentage (i.e. only 1.1% of customers continued beyond the homepage), be sure to include the raw numbers as well (11 visitors out of 1000).

If you’re using a delta (i.e. 14 customer submitted service tickets related to feature X), be sure to include a comparison (usually we don’t get more than 3 service tickets on any one specific feature).

As you see in the delta example, your “quantify” doesn’t have to be super-precise.  I’ve written down things like “support emails seem angrier this week – usually we only get one complainer and this week we’ve had 5 already” — that’s pretty subjective, but it’s still something that I can come back to later and compare to.

Step 3: Ask yourself why.

Quickly take a look at the problem page, workflow, or feature.  Do you see anything that stands out?

…but don’t spend a lot of time on this.  You’re probably not that useful.

Step 4: Ask the data why.

Here are some techniques for figuring out why:

• Check your browser/OS stats: By far the simplest explanation; if the problem is extremely pronounced among IE8 users, there’s almost certainly a bug.  These are the easiest – fix it and you’ll probably see an instant improvement bump.
• Ask an unfamiliar person: Grab someone outside your company — anyone; they don’t need to match your target customer profile — and ask them to use the page/workflow/feature in question and ‘think out loud’.   Things to look for – are they confused by the copy? Do they hesitate because they have a question that’s gone unanswered?  Do they scroll up and down looking for the button to click?
• Check your heatmap: Use a tool like Crazy Egg to check where people are clicking.  In our case, when we had a 5-field form, we could clearly see that most people only clicked on the first 2 fields – signaling that we were asking too much effort of them.  We reduced that form to 2 fields and conversion bumped up immediately.   Heatmaps can also make it obvious that people aren’t scrolling below the fold, or that they’re trying to click on a non-clickable element, or clicking frequently on a ‘distraction’ item instead of the next step.
• Ask a quick question [shameless plug]: With KISSinsights, you can configure a question that pops up after a time delay – we asked what other information customers needed to convince them to sign up, and their responses gave us a to-do list of copy to reword and information to add.
• Ask a successful customer: It’s often easier to find someone who succeeded despite the problem (than someone who dropped off).  Try contacting a customer who completed the workflow or purchase and ask, “Was there anything you found confusing about the process?  Was there information that you expected to see there that you couldn’t find?”  They may well remember and be able to tell you the things that made them hesitate, even though they went ahead and continued.
• Answer support questions with your own questions: When replying to your customer support emails, ask a question of your own (“May I ask you a question?  Have you tried feature X? Is there anything about it you found confusing or difficult?”)

Step 5: Brainstorm.

More heads are better than one.  And now you have the data to bring this problem to your team.   At KISSmetrics, we use a format like this:

Here’s the problem and the current performance data.

Here are some issues we’ve identified that are causing or contributing to the problem.

What can we quickly do to try to fix those issues?

Remember that the operative word here is “quickly” — you don’t want to spend the next two weeks brainstorming the “perfect” solution (which doesn’t exist anyways.)  If your team has that tendency, one tip can be to set time limits on the solution period, i.e.  It’s Monday morning.  Let’s think of possible solutions until our noon Tuesday meeting and then we’ll pick the best one or ones to start implementing.

This should give you a good kick-start on making it better.  Next week I’ll cover how you pick your solution, deploy, and measure.

Popularity: 7% [?]

As last week’s commenters pointed out, there’s a challenge in offering multiple choices vs. asking for freeform responses: You might get more responses but still be missing the root cause of customers’ concerns/problems/ideas.

They’re right.  Freeform answers alone are flawed.  Multiple choice options alone are flawed.  You need to use them both together in order to generate unstoppable, reliable hybrid feedback!

There are two approaches to cultivating your hybrid feedback – pick the one that is most relevant to your situation.

Interview First, Then Survey

Use this method when: You have no idea what you’d even suggest as multiple choice options.

Example: People are really excited when they sign up for your book exchange website. But almost no one is completing a successful exchange, and honestly, you have no idea why. You can’t see a pattern to where people are dropping out of your workflow, and you’re not getting a lot of bug reports or complaints in your support inbox.

Where to start?: Talk to people.

You can call and ask questions about their book exchange needs (see earlier post on “what you should be learning?”), or ask a couple people to go through your website while you watch over their shoulder or using UserTesting.com, or some combination of the three.

Once you’ve talked to 5-10 people, you will usually see some patterns – concerns or obstacles that are affecting more than one person and that seem pretty plausible based on what you know of your own product.

You may find that the problems reported in these first 5-10 conversations/user tests are so fundamental that they’re actually preventing you from getting deeper feedback!  (For example, if people are having problems logging in, they’ll never get to use your core product enough to give you useful feedback on it.)   In that case, skip the survey and start fixing!

Or you can use this initial feedback to populate your multiple choice options, and run the numbers.

Personally (and yes, this is my opinion, not proven data), when I see a multiple choice question from a company and all of the options are really well-written and show a deep understanding of their product, I feel like they really care and am more likely to spend extra time writing out an “Other” response.

Survey First, Then Interview

Use this method when: You want to guide the conversation to specific, relevant options. OR You have a pretty good guess as to what the potential responses are.

Example: You have several large features on your product roadmap, all of which are aligned with your product vision, have supporting market research, and will provide different customer benefits.  But you’d like to validate that they really will be valuable to the customer, and get some subjective feedback on which one will “delight” your customers the most.

Where to start?: Survey.

Questions that may work for this context:

• Which of [list options] features are you most excited about?
• Which of [list tasks] do you use most frequently?
• Would one of these [list options] have convinced you to complete your purchase?
• Would you be willing to be a beta tester for one of these upcoming features [list]
• Have you experienced one of these issues [list]?  How did it affect you?

Once you’ve gotten 20-30 responses, you will usually see a clear winner (or two) emerge.

But now, you have to make sure you properly “decode” that feedback.  You need to understand the “what else” and the “why”.

What else? It may be true that the majority of your customers are most excited about [feature X], but they are assuming that it will magically “just work”.  It’s your job to understand and solve for things like Will this change my workflow?  Will this involve new people or exclude people who previously used it?  My boss is used to [competitor product] so she’ll want it to work like that does.

Why? It may be true that everyone is selecting “add customer reviews and commenting” as their preferred next feature.  But it may not be for the reason you assume.

Let me illustrate with a personal example:

CA: “I didn’t end up buying that [group buying site] Pilates deal, even though it was a good discount.”

Friend: “Why not?”

CA: “It wasn’t clear when you could use it — It would’ve been a waste of money if the classes were only at a time when I couldn’t go.  They should make the vendors give them more information or update their websites to be more clear.”

Friend: “Did you realize that [group buying site] offers a money-back guarantee?  You could’ve bought it and returned it if it wasn’t the right schedule.”

CA: “Ohhhh.  I didn’t know that!  I would’ve bought it if I’d known that!”

Which is easier: getting hundreds of vendors to submit information, and then wrangling that information into a CMS; or just highlighting a money-back guarantee icon?

So that’s why you then follow up with interviews.  Get details, try to disprove your assumptions, and then you’ll finally have the full understanding of your feedback.

Popularity: 1% [?]