Better Product Managers, and Product Management

There is something you want to improve about your site or product.  (If there wasn’t, you probably wouldn’t find this blog terribly interesting.)

Maybe you’re unhappy with the low percentage of visitors who continue past your homepage.  Maybe too many people are abandoning their full carts.  Maybe you’re getting a ton of customer support requests related to a specific feature.

But where do you start?  It can feel very daunting, which is why most people are not doing as much active testing and improvement as they should.

But don’t worry – it gets much easier when you have a plan.   We ran a ton of focused A/B tests on KISSinsights last year and more than doubled our homepage to signup conversion rate.  Here’s how we started:

Step 1:  Write it down.

No, you won’t “just remember”.  (And even if you do, the rest of your team certainly won’t).    Fire up a new doc and write down your problem:  Not enough visitors continuing past the KISSinsights homepage. And put the date at the top so you remember when you started.

Step 2: Quantify it.

How bad has the problem been over the past 7 days?  I like ‘past 7 days’ because it includes both weekdays and weekends — which tend to have very different usage patterns — but is still pretty recent, so it’s not affected by things you did months ago.

If you’re using a percentage (i.e. only 1.1% of customers continued beyond the homepage), be sure to include the raw numbers as well (11 visitors out of 1000).

If you’re using a delta (i.e. 14 customer submitted service tickets related to feature X), be sure to include a comparison (usually we don’t get more than 3 service tickets on any one specific feature).

As you see in the delta example, your “quantify” doesn’t have to be super-precise.  I’ve written down things like “support emails seem angrier this week – usually we only get one complainer and this week we’ve had 5 already” — that’s pretty subjective, but it’s still something that I can come back to later and compare to.

Step 3: Ask yourself why.

Quickly take a look at the problem page, workflow, or feature.  Do you see anything that stands out?

…but don’t spend a lot of time on this.  You’re probably not that useful.

Step 4: Ask the data why.

Here are some techniques for figuring out why:

• Check your browser/OS stats: By far the simplest explanation; if the problem is extremely pronounced among IE8 users, there’s almost certainly a bug.  These are the easiest – fix it and you’ll probably see an instant improvement bump.
• Ask an unfamiliar person: Grab someone outside your company — anyone; they don’t need to match your target customer profile — and ask them to use the page/workflow/feature in question and ‘think out loud’.   Things to look for – are they confused by the copy? Do they hesitate because they have a question that’s gone unanswered?  Do they scroll up and down looking for the button to click?
• Check your heatmap: Use a tool like Crazy Egg to check where people are clicking.  In our case, when we had a 5-field form, we could clearly see that most people only clicked on the first 2 fields – signaling that we were asking too much effort of them.  We reduced that form to 2 fields and conversion bumped up immediately.   Heatmaps can also make it obvious that people aren’t scrolling below the fold, or that they’re trying to click on a non-clickable element, or clicking frequently on a ‘distraction’ item instead of the next step.
• Ask a quick question [shameless plug]: With KISSinsights, you can configure a question that pops up after a time delay – we asked what other information customers needed to convince them to sign up, and their responses gave us a to-do list of copy to reword and information to add.
• Ask a successful customer: It’s often easier to find someone who succeeded despite the problem (than someone who dropped off).  Try contacting a customer who completed the workflow or purchase and ask, “Was there anything you found confusing about the process?  Was there information that you expected to see there that you couldn’t find?”  They may well remember and be able to tell you the things that made them hesitate, even though they went ahead and continued.
• Answer support questions with your own questions: When replying to your customer support emails, ask a question of your own (“May I ask you a question?  Have you tried feature X? Is there anything about it you found confusing or difficult?”)

Step 5: Brainstorm.

More heads are better than one.  And now you have the data to bring this problem to your team.   At KISSmetrics, we use a format like this:

Here’s the problem and the current performance data.

Here are some issues we’ve identified that are causing or contributing to the problem.

What can we quickly do to try to fix those issues?

Remember that the operative word here is “quickly” — you don’t want to spend the next two weeks brainstorming the “perfect” solution (which doesn’t exist anyways.)  If your team has that tendency, one tip can be to set time limits on the solution period, i.e.  It’s Monday morning.  Let’s think of possible solutions until our noon Tuesday meeting and then we’ll pick the best one or ones to start implementing.

This should give you a good kick-start on making it better.  Next week I’ll cover how you pick your solution, deploy, and measure.

Popularity: 5% [?]

As last week’s commenters pointed out, there’s a challenge in offering multiple choices vs. asking for freeform responses: You might get more responses but still be missing the root cause of customers’ concerns/problems/ideas.

They’re right.  Freeform answers alone are flawed.  Multiple choice options alone are flawed.  You need to use them both together in order to generate unstoppable, reliable hybrid feedback!

There are two approaches to cultivating your hybrid feedback – pick the one that is most relevant to your situation.

Interview First, Then Survey

Use this method when: You have no idea what you’d even suggest as multiple choice options.

Example: People are really excited when they sign up for your book exchange website. But almost no one is completing a successful exchange, and honestly, you have no idea why. You can’t see a pattern to where people are dropping out of your workflow, and you’re not getting a lot of bug reports or complaints in your support inbox.

Where to start?: Talk to people.

You can call and ask questions about their book exchange needs (see earlier post on “what you should be learning?”), or ask a couple people to go through your website while you watch over their shoulder or using UserTesting.com, or some combination of the three.

Once you’ve talked to 5-10 people, you will usually see some patterns – concerns or obstacles that are affecting more than one person and that seem pretty plausible based on what you know of your own product.

You may find that the problems reported in these first 5-10 conversations/user tests are so fundamental that they’re actually preventing you from getting deeper feedback!  (For example, if people are having problems logging in, they’ll never get to use your core product enough to give you useful feedback on it.)   In that case, skip the survey and start fixing!

Or you can use this initial feedback to populate your multiple choice options, and run the numbers.

Personally (and yes, this is my opinion, not proven data), when I see a multiple choice question from a company and all of the options are really well-written and show a deep understanding of their product, I feel like they really care and am more likely to spend extra time writing out an “Other” response.

Survey First, Then Interview

Use this method when: You want to guide the conversation to specific, relevant options. OR You have a pretty good guess as to what the potential responses are.

Example: You have several large features on your product roadmap, all of which are aligned with your product vision, have supporting market research, and will provide different customer benefits.  But you’d like to validate that they really will be valuable to the customer, and get some subjective feedback on which one will “delight” your customers the most.

Where to start?: Survey.

Questions that may work for this context:

• Which of [list options] features are you most excited about?
• Which of [list tasks] do you use most frequently?
• Would one of these [list options] have convinced you to complete your purchase?
• Would you be willing to be a beta tester for one of these upcoming features [list]
• Have you experienced one of these issues [list]?  How did it affect you?

Once you’ve gotten 20-30 responses, you will usually see a clear winner (or two) emerge.

But now, you have to make sure you properly “decode” that feedback.  You need to understand the “what else” and the “why”.

What else? It may be true that the majority of your customers are most excited about [feature X], but they are assuming that it will magically “just work”.  It’s your job to understand and solve for things like Will this change my workflow?  Will this involve new people or exclude people who previously used it?  My boss is used to [competitor product] so she’ll want it to work like that does.

Why? It may be true that everyone is selecting “add customer reviews and commenting” as their preferred next feature.  But it may not be for the reason you assume.

Let me illustrate with a personal example:

CA: “I didn’t end up buying that [group buying site] Pilates deal, even though it was a good discount.”

Friend: “Why not?”

CA: “It wasn’t clear when you could use it — It would’ve been a waste of money if the classes were only at a time when I couldn’t go.  They should make the vendors give them more information or update their websites to be more clear.”

Friend: “Did you realize that [group buying site] offers a money-back guarantee?  You could’ve bought it and returned it if it wasn’t the right schedule.”

CA: “Ohhhh.  I didn’t know that!  I would’ve bought it if I’d known that!”

Which is easier: getting hundreds of vendors to submit information, and then wrangling that information into a CMS; or just highlighting a money-back guarantee icon?

So that’s why you then follow up with interviews.  Get details, try to disprove your assumptions, and then you’ll finally have the full understanding of your feedback.

Popularity: 1% [?]

“I really should do user testing, but…”

You know that early validation can save weeks of working down the wrong path, right?  You may have listened to a few tangential comments from users that illuminated a whole new path to differentiation.  You’ve probably seen an interface that was completely intuitive to everyone in your company … and completely baffling to everyone outside it.

But if you’re like most product managers and entrepreneurs, you’re not testing.

First of all, testing has the sense of a big, lofty thing.

We all remember creating science fair projects years ago – you needed a formal hypothesis, a control group and an experimental group, all the variables had to be controlled, you needed to take notes, and the whole thing culminated in a typed, double-spaced report with graphs and charts.  (If you’ve worked with User Research within a large enterprise company, you still see research presentations just like this – except in PowerPoint instead of a tri-fold posterboard.)

Axe it. Forget it. I officially absolve you of needing to be super-scientific and organized.  If anyone asks, you can say “Cindy said this was okay,” and send them to me. Some data is better than no data.

Let me repeat that:

SOME DATA IS BETTER THAN NO DATA.

SOME DATA IS BETTER THAN NO DATA.

SOME DATA IS BETTER THAN NO DATA.

Are we okay now? Good. Let’s keep going.

(more…)

Popularity: 1% [?]

A couple weeks ago, I wanted to make some visual changes to this blog to see if I improve upon some basic engagement metrics – time spent on site, number of entries read, repeat visits, etc.  And the counsel that I frequently give to others immediately intoned:

THOU SHALT NOT MAKE CHANGES WITHOUT A/B TESTING.

OK. But this isn’t as straightforward as it sounds.   Google Web Optimizer will not help you test your WordPress blog.

• GWO is focused on single-action conversions, such as completing a signup or purchase.  I wanted to compare a series of metrics.
• It’s also not ideal for a database-driven site that uses common headers and footers.
• Around 25% of my traffic hits my homepage directly, but WordPress doesn’t have a simple way to create two homepages.
• I tweet a lot of single-article links, which means that every page needed an alternate.  But I wanted to avoid being penalized by Google for having duplicate content.
• I realized there was a lot of value in permanently having an “A” and “B” site so I could always be testing something.

So I decided to roll my own solution. It’s not particularly elegant.  But I’ve laid it out step-by-step so that anyone else with a self-hosted WordPress blog can try this themselves.

(And maybe, someone with better programming chops than myself will write a more streamlined plug-in version and make it available to the blogging community.)

(more…)

Popularity: 4% [?]