But as you’ve probably figured out by now, my personal bias is towards starting with metrics where you can immediately translate them into a natural-language statement (“this means X customer type is more likely to pay us”).

So instead of talking about customer lifetime value or average revenue per user, I’m going to talk about revenue in terms of two simple questions you should be asking to start with:

• What percentage of Activated customers have given us money, even once?
• What types of customers were more likely to give us money?

These are both fairly short-term questions, but what I like about them is that they can immediately narrow where you should focus on improving next.

(Everyone ought to be improving their customer lifetime value, but that’s about as vague as your New Year’s Resolution to lose 10 pounds.  And about as likely to succeed, unless you break it down.)

So let’s look at some examples.

What percentage of Activated customers have given us money, even once?

Let’s look at fake-Amazon.com:

You’d have defined Activated customers as the people who were had reached the potential to get value — in this case, they found a product that was interesting enough for them to view in detail. This is your potential pool from which to extract Revenues.

How many of those people give you money?

In this example, 2.2% of your potential pool is giving you money.   So now the obvious question is, how can I get more of this potential pool of pre-qualified, full-of-intent, people to give us money?

I’d expect that it’s quite common for someone to view a product and not proceed with purchase — you could be just browsing, or perhaps you were just curious about that electronic toenail polisher.  But here I’d hone in on the big dropoff between “Added to Cart” and “Completed Purchase” — and try to figure out why via customer interviews, user testing, or a KISSinsights survey.

If I could win back just 5% of those “Added to Cart” people, that’s more than 150 additional purchases.  Assuming a $20 average sale, that’s more than $3,000 additional revenue per day, and the solution is probably just fixing something that you should’ve fixed anyways.

For a second example, let’s look at fake-Netflix.com:

This would probably signal to me that my focus belongs on getting more Acquired users, versus trying to improve upon either of these steps.

What types of customers were more likely to give us money?

Most of us are targeting more than one type of customer.  I strongly suggest collecting that information — either via the registration form, or some required post-signup configuration — and using it to confirm where your dollars are coming from.

In this example, suppose you were selling a website service, and asked your customers what type of site they maintained:

The overall percentage of upgraders — 2.8% is interesting — but a lot less interesting than the fact that people running e-commerce sites are dramatically more likely to pay you than other types of customers.

This tells me that, rather than trying to create a “generic” marketing homepage that appeals to all customers, you should try tailoring your marketing site to specifically appeal to e-commerce customers.

Once you’ve experimented and improved some of the more obvious areas, you’ll feel a lot more comfortable tackling the more complex “lifecycle” metrics (and these can only help those, as well).

Acquiring and Activating a customer is like getting a person inside your restaurant and sitting down with a menu in front of them.  But that’s no guarantee they’ll come back and eat there again.  In fact, there’s still the potential of you screwing up their experience so badly that they walk out without ordering.

Retention

Here’s the challenge with retention: most people think in terms of questions like “What percentage of our customers are logging in daily/weekly?” or “What percentage of our customers stick around for 3/6/12 months?”

Those are excellent questions, but they take too damn long to give you data.  If you’re early in your product lifecycle, you can’t afford to wait 3 or 6 months to get a sense of your Retention.  I recommend measuring in 2 ways: Retention threshold and ongoing Retention.

Your Retention threshold is analogous to the Activation threshold – it’s the point where your product has become a part of the customers’ behavior; something that they’re willing to invest time in.

How do you know what behaviors correspond to this threshold?  You don’t, at first.

This is where customer interviews can be really useful.  Don’t try to be too clever — just ask, “What’s the thing that convinced you to be a loyal customer? / What’s the thing that brings you back to our site regularly?” and listen to what people tell you.   A dozen conversations will get you to answers much, much faster than trying to tease this out with quantitative analytics alone.

So let’s guess that Facebook had done these interviews and identified a bunch of different behaviors, not all of them easily measurable:

• “when I started sharing all my photos via Facebook”
• “Facebook is the first thing I check in the morning and the last thing I check before I sleep”
• “I realized it was easier to invite people to my events using Facebook than Evite or SMSing”

Now, not all customers are going to do all things, but you could use this input to choose a few events as a reasonable proxy for crossing the Retention threshold — let’s say “Created a photo album”, “Created an event and invited at least 1 person”, “Set Facebook as browser homepage”.

You might see a funnel like this:

Note that all three behaviors were combined into one “retention action” and then broken out in the table beneath.

This gives you a few into how many people are crossing over the Retention threshold, and which behavior path they’re more likely to take.  You can then experiment with “nudging” people to complete these actions faster (i.e. via an email campaign encouraging customers to upload a photo album after a holiday weekend, or an interstitial prompting them to set your site as their homepage).

This is just a proxy; it’s not something I’d get obsessed with.  But it’s a good stopgap measure until someone has been a customer for long enough that a more standard “they keep logging in / they keep paying me” metric kicks in.

I’m going to talk less about ongoing Retention because it seems like it’s more widely understood.  My main peeve there is when companies blindly use “login frequency” as a measure of retention.  How often I log in is not equal to “how valuable I find your product”.

The real equation you’re trying to solve for is:  “Of the times when my customer has [these problems], what percentage of the time are they turning to me to solve them?”

Facebook’s goal is to be an everyday app and insinuate itself into your social world.  Since you’re part of your social world every day, their goal should be for you to log in daily.   Compare that to LinkedIn — it’s not every day that I need to check out someone’s professional credentials or research potential hires.  When I have that need, 100% of the time I turn to LinkedIn.  But I don’t have that need that often.

(That may be another problem, but I’ll save that for a different post.)

Referrals

There are 2 ways of looking at Referrals:

• What percentage of my customers will, if prompted, refer my product to others?
• What percentage of my customers have come from referrals?

A great example of #1 is Dropbox:

Dropbox offers an incentive for customers to refer the product to friends, which has several points of optimization:

• Do people care enough / trust you enough to start a referral process?
• Is the workflow of referring the product easy and fast enough for them to complete it?
• Do the referred / invited people care enough to come try it out?
• Is the signup process for invited people streamlined enough for them to complete it?

If explicit referrals are key to your distribution strategy, you’ll need to optimize the hell out of each of these stages.  That is a big if.  There are many companies who grow just fine without an explicit referral strategy, and there are many companies who may benefit from this in the future but aren’t ready yet.  If this doesn’t “fit” with you now, don’t stress about it.

Let’s say you’re not optimizing a referral workflow.   KISSmetrics is not.  KISSinsights attaches a signup call-to-action on our free surveys: we get customers that way, but it’s a very different workflow than “Your friend X endorses that you use Product Y”.   We could optimize that a lot more.  But in the meantime, some simple questions you can ask are:

• What percentage of signups originated from Twitter or Facebook referrers?
• What percentage of visitors who originated from Twitter/Facebook converted?
• What percentage of signups originated from a “click here to get your account” or “powered by” link?
• What percentage of visitors who originated from “click here to get your account” or “powered by” links converted?

These can serve as an indicator to let you know if you should start focusing more on one of these channels.  We get a lot of Twitter attention — i.e. lots of Acquired people — but the conversion rate isn’t super strong.  If it was, you can bet I’d be prompting customers to Tweet about us more.

Another indicator: if “passive referral” folks account for more than a few percent of your customers, that may indicate that your homepage isn’t adequately communicating your value proposition.  So only folks who’ve gotten the added credibility of a tweet from a trusted contact or seeing you live on another site are actually converting.  If that’s the case, time to start A/B testing new homepage marketing copy!

OK, I know I said I was covering all 3 R’s this week, but I think Revenue deserves its’ own entry.  So you’ll have to wait one more week…

So I’ve taken a site we’re all familiar with — Facebook — and walked through how they might apply these metrics.   (Note: Facebook is not a KISSmetrics customer nor associated with us, and I am in no way implying that their analytics are this rudimentary.  Also, all numbers below are totally, and hastily, made up.)

We’ll start this week with the A’s: Acquisition and Activation.

Acquisition

Acquiring a visitors means that they came to your site and did something to show interest.  People who bounce in and off your site in less than 5 seconds are not acquired.  They probably won’t even remember your name tomorrow.  Same goes for people who open your site in one forgotten browser tab and never look at it.

What should you use to measure “showed interest”?  It will vary based on what options your site offers to customers.  For an information-dense homepage, “remaining on the site for 20 seconds” might be a good proxy for interest.  For a shopping site, “clicking on navigation, conducting a product search, or clicking on a feature product” would show interest.

In the Facebook case, their homepage has the signup form embedded right there, with a very low barrier to entry.  I would consider “submitted signup form” to show interest:

So you might have a funnel that looked like this:
If your already-converted customers come to that homepage, you’re going to see skewed-low numbers.  (Obviously, if I’m already a customer, I am not going to continue to “Submitted Signup Form”, I’m going to click the sign-in link. )

One easy high-level way to “correct” for this skew is simply to create a complementary funnel for Viewed Homepage -> Signed In and subtract that percentage:

• 10,000 people view homepage and then 100 submit signup form (1% signup rate)
• 10,000 people view homepage and then 5,000 click the sign-in link
• those 5,000 were not actually candidates for signup (they’re already a customer) so assume more like a 2% signup rate

Not exact but good enough to get a feel for it.

There’s a ton you can dig into here around which channels, ads, search terms are leading to the highest acquisition.  That said, most companies I talk to are focusing too much on Acquisition too early. By far the biggest problem and highest risk is that no one will care about your product/site.  (Or be able to use it.)

I’d worry about getting more of the few people you do have, to give a damn, than trying to optimize/maximize how many more people you can get.  (Brant Cooper writes more about that, and the order in which you should tackle the AARRR.)

Activation

Like Acquisition, Activation is not immediate.  A person who has simply completed a signup is not necessarily activated unless they can immediately start using and benefitting from your product.

I define Activation as the point at which your potential customer is able to get value from your product.  In other words, they have completed those obstacles — entering information, configuring settings, installing code — which are required before they can do anything useful.

The Facebook example works particularly well here.  Until you’ve added at least one friend, Facebook is useless.  (“what is the sound of one friend friending?”)

So at minimum, you might track a report like this:

But how do you know if “added 1 friend” is enough of a proxy for Activation?  The short answer is, you don’t. You’ll have to guess, with some help from talking to customers and user-testing.

(After you have a fair amount of data, also, you’ll be able to do things like look at the people who log in regularly and identify their least common denominator.  But most people don’t have that much data to start — and for most sites who don’t enjoy the traffic levels of Facebook, quite honestly the patterns don’t just leap out like that.)

As it happens, I sat with my mom while she signed up for Facebook  (the perils of being related to me; I am probably surreptitiously conducting a user test on you).  Adding 1 friend wasn’t that exciting.  Sync’ing her Gmail contacts wasn’t that exciting (after all, the people she emails are the people she already sees all the time.)

What “tipped” her experience was searching for — and finding — long-lost relatives and getting friend requests from people she’d lost touch with.

These “tipping point” experiences are things you’d want to measure. You’d also want to measure the customer behaviors that facilitate these experiences — for example, if a user does not fill out their Facebook profile, they are unlikely to be ‘found’ by others.  Adding your high school and college affiliations don’t directly bring the customer value, but they indirectly lead to value by making that person findable.

So you might track a report like this:

Of course, there isn’t going to be a “perfect” proxy for Activation, so you may want to start with several different “hypotheses” — different sets of criteria that you look at and then triangulate by talking to customers and looking at longer-term data.

You’ll also want to use qualitative research to understand why some customers perform these actions (“I figured I’d get this benefit”) and others don’t (“I was afraid this would happen”) so that you can move more people to more quickly proceed from Acquired to Activated.

Next week: the RRRs!

Asking questions helps me make sense of metrics.  It’s easier for me to think in terms of questions first, then look for the ways to answer them (whether that involves quantitative analytics, qualitative research, user testing, or some other tool).

Not all of these questions will be relevant for your business, but seeing a checklist can help you identify the questions that do make sense for you.  (It’s also useful to revisit this list from time to time, because as your business evolves, the questions you focus on will change.)

Acquisition

• What sources are sending visitors to my site?
• What are visitors searching for that brings them to my site?
• What percentage of my visitors are coming from search / direct traffic / paid search / ads / referrals?
• Which ad campaigns are convincing people to visit my site?
• For the people coming to my site from direct traffic, how did they find out about us?
• If a particular (non-paid) source is sending a lot of traffic to my site, how are they describing us?

Activation

• Of all visitors to your site, what percentage of them convert to becoming a customer (signing up, subscribing, purchasing, starting a trial)?
• Of the visitors who fail to become customers, where are they dropping off?
• Which sources are most effectively sending me visitors who become customers?
• Which sources are sending me lots of traffic that is not converting?
• Which ad campaigns are most effectively sending me visitors who become customers?
• Are visitors who came via search more likely to become customers?
• How many times do visitors come to your site before they convert to customers?
• How much time (in hours/days/weeks) does it take for a visitor to your site to convert to being a customer?
• What is the first source that sent a customer to your site?
• What is the last source that sent a customer to your site immediately before they became customers?
• How does signup workflow (i.e. Facebook Connect vs. traditional signup, mobile vs. web signup, homepage vs. third-party partner) affect conversion rate?
• Were visitors who viewed a video or product tour more likely to convert to becoming a customer?
• How does collection of billing information (i.e. collected up-front, billed later, free trial to start, etc.) affect conversion rate?
• What percentage of customers experience an error during the signup process?
• Of the customers who experience an error during the signup process, how many persist and become customers?

Retention

• What percentage of customers come back and log in again within a week / month / 3 months / etc. ?
• What percentage of customers successfully complete configuration / setup such that they’re able to use your product?
• Are customers who signed up from a specific source more likely to complete setup / configuration?
• Of the customers who do not complete configuration / setup, where did they drop off?
• Of the customers who sign up for a free trial, how long does it take for them to start actively using their trial?
• How long does it take for the average customer to go from initial signup to successfully using the core features of your product?
• What percentage of your customers are successfully using some advanced features of your product?
• Which sources are most effectively sending me visitors who become customers who return?
• Are customers with a specific demographic profile more likely to successfully return and use your product?
• If you’ve offered discounts or coupon codes, are customers who used those codes more or less likely to successfully return and use your product?
• Are customers who sign up because of a referral more likely to return and use your product more actively than customers who did not come in via a referral?
• What percentage of your customers stop using / paying for your product each month?

Referral

• What percentage of your customers are actively recommending your product?
• What percentage of your incoming customers are coming in via referral?
• How does referral method (Facebook, Twitter, email invitations) affect number of referrals sent?
• How does referral method (Facebook, Twitter, email invitations) affect number of referrals that the recipient acts upon?
• How does incentive affect referral behavior (both sending and acting upon)?
• Who are your most active referrers / evangelists?
• Are customers with a specific demographic profile more likely to recommend your product?

Revenues

• How much in revenues are you making each month?
• Are you on track to increase revenues from last month?
• What percentage of your revenues are coming from new vs. returning customers?
• How do your revenues break down between various plans/service levels/products?
• Which of your customers are providing you with the most revenue?
• How long does your average customer remain a paying customer?
• How much time elapses between return purchases?
• What is your average customer lifetime value?
• Which sources are providing you with the most profitable customers?
• Are customers with a specific demographic profile more likely to be more (or less) profitable?
• What percentage of my customers are canceling due to billing errors as opposed to deliberate cancelation?
• What percentage of purchases are coming from returning customers?
• What percentage of your customers are upgrading from a free to paid plan?
• What percentage of your customers respond to a cross-sell or upsell purchase?
• If you offer multiple upsell calls to action, which ones are most successfully driving upgrades / additional purchases?
• If you offer multiple payment options, which one is driving the highest percentage of successfully completed transactions?
• What percentage of customers are experiencing an error during the billing / purchase process?
• Of the customers who experience an error, how many persist and complete the billing / purchase process?

• picked a fast and measurable improvement to try
• figured out what part of your audience was going to see it
• launched it

You might think the hard part is done now.

But this is exactly where most people fall apart — they fail to close the loop.

Closing the Loop

By “closing the loop”, I mean:

• Validating that your change worked
• Measuring how big the effect was
• Following up with your team so they know what happened and what you learned

The last step is what makes or breaks your ability to keep making your product better.  You can brute-force your team through a couple rounds of improvement without it, but it’s not sustainable.

When you have a developer who spent a full day reworking a feature, and they never hear that whether it made any difference on usage, it’s going to be harder to keep them motivated.  This is especially important when you’re doing quick iterations — workflow changes and copy tweaks are not, generally, the meaty problems that developers are intrinsically excited to work on.   But if those “boring” changes lead to more customers, that’s something to be excited about.

When you’re solving an unknown problem — and really, the vast majority of “making it better” problems are unknown — you benefit from using all the brains in your company.  But when you don’t follow up with what you learned, you’re handicapping those brains.  Suppose you learned that removing configuration options increased conversion, but you don’t share that with your team.  How are they going to know to stop thinking up power-user features?

If you are like me, you probably think you are being very clear in your followups, and you probably are not.

How to be really clear when closing the loop

We had the best results with KISSinsights when I gave a verbal followup in this approximate format:

• Remember, our goal was to improve [metric]
• We started at [number]
• We deployed [these changes] to try and move that number
• Now we’re at [number]
• We think these changes did work / might work but we want to watch for another week / didn’t work
• Here’s any anecdotal evidence from customer interviews, support emails, tweets, etc.

My goal for moving forward is to share this information in writing as well – discussing it in meetings is great, but lots of people just do better grokking the written word.   I also think, as we move from very short-term goals to longer-term goals, it will be more useful to have easily-searchable written records to look back on.

• identified a problem and written it down
• found a way to measure the problem
• explored some possible reasons why there’s a problem
• brainstormed some possible solutions

Now what?

Look at speed.

When you first start trying to improve, you really don’t know what is going to work.  So don’t try to evaluate the potential solutions on “merit” – you don’t have merits yet, just opinions.

Instead, look at what can be put in front of customers the fastest.   Copy changes are often faster than single-page UI changes, which are often faster than workflow changes, which are often faster than back-end changes.  (When I say ‘speed’, I really mean a combination of level of effort and risk — a single line of code change may take an engineer 2 minutes to write, but if it requires tweaking your test scripts or thinking about how it might impact existing customers in unknown ways, it’s not a “high-speed” change.)

Look at measurability.

If you made this change, how would you know if it worked?  If you can’t answer that, it’s not measurable enough for you to start with.  (Your instincts on how to effective measure success will get better over time; I’m just saying, if you don’t grok it now, start with something simpler.)

Pick one.

If your team is like my team, they’re smart and they came up with a bunch of creative solutions to try out.   So now you’re excited and thinking, how fast can we do all of these?

And the answer is: one at a time.

When you try to make multiple changes at once, it’s almost impossible to know the impact of each individual change.  Maybe Change A increased your conversion rate by 10%, but the Change B reduced it by 5%.  All you’ll see is a net 5% gain — which looks great on paper — but when you continue making more Change B-like changes you’ll just keep hurting yourself.

Don’t know which of your proposed fast and measurable changes to start with?  Flip a coin.  Just take action.

Feature/workflow change? Make it even smaller.

If you picked a feature change or workflow change to start with, challenge yourself to make it even smaller.  Can you build half of what you originally proposed and still learn something?   Setting a very short timeframe (“what can we do in one day?”) is a good way to enforce this.

Why make it smaller?  Because you don’t know if it will work yet.   You don’t lay a down big bet before you’ve seen your cards; you start small and double down when you see face cards (or minimize your losses when you fold).

Figure out who’s going to see it.

Everyone sees it. This is less scientific but a lot simpler.  And if you’re small or just getting started, it’s usually good enough.  Unless there’s a reason why your traffic/usage would significantly differ between 1-week or 2-week periods (i.e. over a holiday or after getting heavy press attention), you will know if “before” or “after” is better.

Random “B” group sees it. Reduces the risk that time periods are somehow different; tends to make companies feel more comfortable since it is more scientific.

Only new customers see it. Change is often initially hated, even if customers eventually realize it’s better.  If you think your customers will find ongoing changes upsetting, you can use new customers as guinea pigs and only roll out changes when you’re confident they’re a significant improvement.   Some risk that your “new customer” user base may be somehow different than your “existing customer” user base, though.

Communicate.

Make sure everyone on your team knows what “before” and “after” look like (taking screenshots of each and sharing them is one easy way to do this).  No one should be filing bugs on “B” because their screen is showing “A”, or sending inaccurate customer service email responses because they’re looking at “B” and the customer still has “A”.

Go!

Release it.  Be excited.  Make a note to check back on progress after 2 days (probably too short to notice a difference, but enough time to notice a potential catastrophe, like “B is generating a ton of customer support queries!”), after 1 week, after 2 weeks.

If you test one thing at a time but run a test every single week, you’ll get through 52 experiments in a year.  I guarantee that will move your numbers significantly.

Maybe you’re unhappy with the low percentage of visitors who continue past your homepage.  Maybe too many people are abandoning their full carts.  Maybe you’re getting a ton of customer support requests related to a specific feature.

But where do you start?  It can feel very daunting, which is why most people are not doing as much active testing and improvement as they should.

But don’t worry – it gets much easier when you have a plan.   We ran a ton of focused A/B tests on KISSinsights last year and more than doubled our homepage to signup conversion rate.  Here’s how we started:

Step 1:  Write it down.

No, you won’t “just remember”.  (And even if you do, the rest of your team certainly won’t).    Fire up a new doc and write down your problem:  Not enough visitors continuing past the KISSinsights homepage. And put the date at the top so you remember when you started.

Step 2: Quantify it.

How bad has the problem been over the past 7 days?  I like ‘past 7 days’ because it includes both weekdays and weekends — which tend to have very different usage patterns — but is still pretty recent, so it’s not affected by things you did months ago.

If you’re using a percentage (i.e. only 1.1% of customers continued beyond the homepage), be sure to include the raw numbers as well (11 visitors out of 1000).

If you’re using a delta (i.e. 14 customer submitted service tickets related to feature X), be sure to include a comparison (usually we don’t get more than 3 service tickets on any one specific feature).

As you see in the delta example, your “quantify” doesn’t have to be super-precise.  I’ve written down things like “support emails seem angrier this week – usually we only get one complainer and this week we’ve had 5 already” — that’s pretty subjective, but it’s still something that I can come back to later and compare to.

Step 3: Ask yourself why.

Quickly take a look at the problem page, workflow, or feature.  Do you see anything that stands out?

…but don’t spend a lot of time on this.  You’re probably not that useful.

Step 4: Ask the data why.

Here are some techniques for figuring out why:

• Check your browser/OS stats: By far the simplest explanation; if the problem is extremely pronounced among IE8 users, there’s almost certainly a bug.  These are the easiest – fix it and you’ll probably see an instant improvement bump.
• Ask an unfamiliar person: Grab someone outside your company — anyone; they don’t need to match your target customer profile — and ask them to use the page/workflow/feature in question and ‘think out loud’.   Things to look for – are they confused by the copy? Do they hesitate because they have a question that’s gone unanswered?  Do they scroll up and down looking for the button to click?
• Check your heatmap: Use a tool like Crazy Egg to check where people are clicking.  In our case, when we had a 5-field form, we could clearly see that most people only clicked on the first 2 fields – signaling that we were asking too much effort of them.  We reduced that form to 2 fields and conversion bumped up immediately.   Heatmaps can also make it obvious that people aren’t scrolling below the fold, or that they’re trying to click on a non-clickable element, or clicking frequently on a ‘distraction’ item instead of the next step.
• Ask a quick question [shameless plug]: With KISSinsights, you can configure a question that pops up after a time delay – we asked what other information customers needed to convince them to sign up, and their responses gave us a to-do list of copy to reword and information to add.
• Ask a successful customer: It’s often easier to find someone who succeeded despite the problem (than someone who dropped off).  Try contacting a customer who completed the workflow or purchase and ask, “Was there anything you found confusing about the process?  Was there information that you expected to see there that you couldn’t find?”  They may well remember and be able to tell you the things that made them hesitate, even though they went ahead and continued.
• Answer support questions with your own questions: When replying to your customer support emails, ask a question of your own (“May I ask you a question?  Have you tried feature X? Is there anything about it you found confusing or difficult?”)

Step 5: Brainstorm.

More heads are better than one.  And now you have the data to bring this problem to your team.   At KISSmetrics, we use a format like this:

Here’s the problem and the current performance data.

Here are some issues we’ve identified that are causing or contributing to the problem.

What can we quickly do to try to fix those issues?

Remember that the operative word here is “quickly” — you don’t want to spend the next two weeks brainstorming the “perfect” solution (which doesn’t exist anyways.)  If your team has that tendency, one tip can be to set time limits on the solution period, i.e.  It’s Monday morning.  Let’s think of possible solutions until our noon Tuesday meeting and then we’ll pick the best one or ones to start implementing.

This should give you a good kick-start on making it better.  Next week I’ll cover how you pick your solution, deploy, and measure.

They’re right.  Freeform answers alone are flawed.  Multiple choice options alone are flawed.  You need to use them both together in order to generate unstoppable, reliable hybrid feedback!

There are two approaches to cultivating your hybrid feedback – pick the one that is most relevant to your situation.

Interview First, Then Survey

Use this method when: You have no idea what you’d even suggest as multiple choice options.

Example: People are really excited when they sign up for your book exchange website. But almost no one is completing a successful exchange, and honestly, you have no idea why. You can’t see a pattern to where people are dropping out of your workflow, and you’re not getting a lot of bug reports or complaints in your support inbox.

Where to start?: Talk to people.

You can call and ask questions about their book exchange needs (see earlier post on “what you should be learning?”), or ask a couple people to go through your website while you watch over their shoulder or using UserTesting.com, or some combination of the three.

Once you’ve talked to 5-10 people, you will usually see some patterns – concerns or obstacles that are affecting more than one person and that seem pretty plausible based on what you know of your own product.

You may find that the problems reported in these first 5-10 conversations/user tests are so fundamental that they’re actually preventing you from getting deeper feedback!  (For example, if people are having problems logging in, they’ll never get to use your core product enough to give you useful feedback on it.)   In that case, skip the survey and start fixing!

Or you can use this initial feedback to populate your multiple choice options, and run the numbers.

Personally (and yes, this is my opinion, not proven data), when I see a multiple choice question from a company and all of the options are really well-written and show a deep understanding of their product, I feel like they really care and am more likely to spend extra time writing out an “Other” response.

Survey First, Then Interview

Use this method when: You want to guide the conversation to specific, relevant options. OR You have a pretty good guess as to what the potential responses are.

Example: You have several large features on your product roadmap, all of which are aligned with your product vision, have supporting market research, and will provide different customer benefits.  But you’d like to validate that they really will be valuable to the customer, and get some subjective feedback on which one will “delight” your customers the most.

Where to start?: Survey.

Questions that may work for this context:

• Which of [list options] features are you most excited about?
• Which of [list tasks] do you use most frequently?
• Would one of these [list options] have convinced you to complete your purchase?
• Would you be willing to be a beta tester for one of these upcoming features [list]
• Have you experienced one of these issues [list]?  How did it affect you?

Once you’ve gotten 20-30 responses, you will usually see a clear winner (or two) emerge.

But now, you have to make sure you properly “decode” that feedback.  You need to understand the “what else” and the “why”.

What else? It may be true that the majority of your customers are most excited about [feature X], but they are assuming that it will magically “just work”.  It’s your job to understand and solve for things like Will this change my workflow?  Will this involve new people or exclude people who previously used it?  My boss is used to [competitor product] so she’ll want it to work like that does.

Why? It may be true that everyone is selecting “add customer reviews and commenting” as their preferred next feature.  But it may not be for the reason you assume.

Let me illustrate with a personal example:

CA: “I didn’t end up buying that [group buying site] Pilates deal, even though it was a good discount.”

Friend: “Why not?”

CA: “It wasn’t clear when you could use it — It would’ve been a waste of money if the classes were only at a time when I couldn’t go.  They should make the vendors give them more information or update their websites to be more clear.”

Friend: “Did you realize that [group buying site] offers a money-back guarantee?  You could’ve bought it and returned it if it wasn’t the right schedule.”

CA: “Ohhhh.  I didn’t know that!  I would’ve bought it if I’d known that!”

Which is easier: getting hundreds of vendors to submit information, and then wrangling that information into a CMS; or just highlighting a money-back guarantee icon?

So that’s why you then follow up with interviews.  Get details, try to disprove your assumptions, and then you’ll finally have the full understanding of your feedback.

]]> http://www.cindyalvarez.com/data-driven/hybrid-feedback-is-stronger/feed/ 7 http://www.cindyalvarez.com/data-driven/youve-got-questions-ive-got-tools http://www.cindyalvarez.com/data-driven/youve-got-questions-ive-got-tools#comments Wed, 22 Jul 2009 20:43:57 +0000 Cindy http://www.cindyalvarez.com/?p=483 “I really should do user testing, but…”

You know that early validation can save weeks of working down the wrong path, right?  You may have listened to a few tangential comments from users that illuminated a whole new path to differentiation.  You’ve probably seen an interface that was completely intuitive to everyone in your company … and completely baffling to everyone outside it.

But if you’re like most product managers and entrepreneurs, you’re not testing.

First of all, testing has the sense of a big, lofty thing.

We all remember creating science fair projects years ago – you needed a formal hypothesis, a control group and an experimental group, all the variables had to be controlled, you needed to take notes, and the whole thing culminated in a typed, double-spaced report with graphs and charts.  (If you’ve worked with User Research within a large enterprise company, you still see research presentations just like this – except in PowerPoint instead of a tri-fold posterboard.)

Axe it. Forget it. I officially absolve you of needing to be super-scientific and organized.  If anyone asks, you can say “Cindy said this was okay,” and send them to me. Some data is better than no data.

Let me repeat that:

SOME DATA IS BETTER THAN NO DATA.

SOME DATA IS BETTER THAN NO DATA.

SOME DATA IS BETTER THAN NO DATA.

Are we okay now? Good. Let’s keep going.

Second of all, you don’t know where to start.

There are tons of freely available web-based tools, and blog posts giving you advice and how-to and do’s-and-don’ts.  What should I use? What’s the right tool?  Is it really okay to skip face-to-face user testing?

What tool you use depends on what you are trying to find out.  Here are a few common questions and tools:

• Is this product even remotely interesting to people (or am I crazy?)
• Do people understand what my site/product does?
• Are users able to complete the tasks that are core to my site/product? (i.e. completing specific actions, purchasing)
• What are the areas of my site/product that are confusing or annoying users?
• What are the alternatives to my site/product?  (what are people using instead of me?)
• Which audience finds my site/product most appealing?
• What other features should my product offer? (how could I make this product more appealing/differentiated)
• Am I organizing my site content in a way that makes sense to users?

(If you don’t see your question type here, leave it in the comments and I will write a Part 2 to this post.)

Is this product even remotely interesting to people (or am I crazy?)

What You’re Looking For: This is the first question that follows a “Eureka!” moment.  You have an idea, it sounds good to you.  You ask your friend and it sounds good to her, too.  But you might both be crazy, so you need some validation – just enough to convince you that it’s worth putting in more time.

Tool: SEM on $5/day. Build a really simple landing page. Buy Google AdWords or targeted Facebook apps that describe your concept, and measure how many clicks you get.  Vary your ad the next day and see if clicks improve.  If no one clicks, no one cares.  (and you just saved yourself a lot of work implementing something no one wants.) If people click, you can move on to embedding a simple survey to ask 1-2 questions about what they would pay for.

Do people understand what my site/product does?

What You’re Looking For: Validation that your splash page or email marketing copy makes sense.  You want people to see it and think X and Y.  Are they thinking X and Y?

Tool: FiveSecondTest (or equivalent).  I’m actually not crazy about FiveSecondTest, just because five seconds is even too short for skimming.  But you can easily create your own online version.  Show your splash page for 10 or 15 or 30 seconds, then advance user to a single survey question that asks “What do you think [this product] would allow you to do?” I recommend Wufoo for creating a single-question form and embedding it directly in your page.  It’s free and their simple reports do the job.

For an extended version, you may have a third screen that asks “Did you think that this product did X?  Did you think this product did Y?”    But first you want their untainted impressions.

Are users able to complete the tasks that are core to my site/product? (i.e. completing specific actions, purchasing)

What You’re Looking For: Your product has certain things that ARE the product.  If you’re an e-commerce site, people need to buy things.  If you’re a bill payment product, people need to be able to pay a bill.  This is a quantitative question.

Tool: Google Analytics. Identify your core tasks and the pages that a user must go through if they successfully complete it.  Create a funnel to identify how many users make it through each stage of the funnel.  (Note: your product does not have more than 5 core tasks, even if it’s “really complex.”  A Boeing 747 has 3 core tasks – taking off, staying in the air, and landing.)

Tool: UserFly.  UserFly records mini-movies of your users as they navigate through your site.  You can watch (sometimes painfully) as users hover hesitantly over a button, then click the wrong one, or enter the wrong information into a form field.

What are the areas of my site/product that are confusing or annoying users?

What You’re Looking For: You know there’s a problem – you just don’t know what it is.  People aren’t completing the core tasks of your product, or your bounce rate is extremely high, and you don’t know why.  This is a qualitative question.

Tool: Task-based user testing. Get 5 users off Craig’s List and offer them $50 to come to your office for a 30-minute test.  Either videotape them or have one person to moderate and one to scribble notes furiously.  Ask them to complete core task #1 and encourage them to narrate out loud as they navigate.   Resist the urge to comment or correct them.  Do ask questions like “what do you think will happen if you click there?”  Do notice their body language – are they relaxed? tense? frowning?  Remember, these people were paid to be here – so their default mode should be “happy”.

If this sounds too hard, try getting 2 users off Craig’s List.  Or skip that and just beg your non-technical friend or neighbor who doesn’t know anything about your product to come be your test subject.  Even ONE person who is not you, can reveal a ton of insight.  One person testing takes less than an hour.  You must be willing to invest ONE hour that will save you days or weeks of development and frustration.

What are the alternatives to my site/product?  (what are people using instead of me?)

What You’re Looking For: What’s the bigger story of how you fit into your users’ lives?  What are their behaviors and the environment they conduct them in?  For example, before Netflix, users rented videos from Blockbuster, or didn’t rent movies at all because it was too much of a hassle compared to just watching whatever was on cable.

Tool: In-person (or phone) interview. Get 5 people off Craig’s List and compensate them to come to your office or agree to a phone interview.  Ask them to tell you about the last time they did [your behavior].  For example, “Tell me about the last time you watched a movie at home”.  Resist the urge to comment, but ask helper questions to keep the narrative going – “How did you decide to watch a movie?”, “Was there a specific type of movie you were interested in?”, “Why did you watch a movie instead of doing something else?”, “How did you decide which movie to watch?” , “Who else was part of this decision?”, “What do you wish had been simpler/less frustrating/faster about the process?”

Tool: SurveyMonkey. Ask similar questions but via online survey.  I find this less insightful because you are far less likely to get freeform, “tangent” answers, but it’s very fast and doesn’t require someone who is comfortable giving an interview verbally.

Which audience finds my site/product most appealing?

What You’re Looking For: Should I be targeting [these people] or [those people]?  What description will convince people to convert? Where should I focus my further research/feature development?

Tool: SEM on $5/day. You can buy very targeted ads for cheap on Facebook (by now, all the demographics are there.) – set up multiple audiences and send them to your landing page, use an embedded survey to ask a few questions.  Measure the clickthrough rates and the survey response rates to get your first pass at who is most interested in your product.

Tool: Google Website Optimizer. Create two splash pages with different messaging that targets two different audiences.  Each one needs a “Sign Up Now” button that leads to a confirmation page. That’s all you need – don’t even need to link to your product yet.  GWO will compare the percentage who converted (landed on the confirmation page).  Clicking the “Sign Up Now” button is a clear measure of interest.  Repeat with as many “markets” as make sense.

What other features should my product offer? (how could I make this product more appealing/differentiated)

What You’re Looking For: Solving the big pain point.  What is it?

Tool: Task-based user testing. Get 5 users off Craig’s List and offer them $50 to come to your office for a 30-minute test.  Either videotape them or have one person to moderate and one to scribble notes furiously.  Ask them to complete core task #1 and encourage them to narrate out loud as they navigate.  As they navigate your site, have them describe where they are and what they’re doing when using your product (or the general behavioral context).

For example, the idea for one-handed baby wipes came from mothers describing what they did — “and I’ve got one hand keeping the wiggling baby from rolling off the changing table, and now I need to reach for a wipe…”

Note: there are various customer tools like UserVoice and SuggestionBox that allow customers to suggest features.  I’m deliberately not recommending those, because I subscribe to the Henry Ford philosophy that his customers would’ve suggested a faster horse. Users, generally speaking, can always come up with things they want but are not as good at articulating true pain points.  That’s what the experienced product manager or entrepreneur has to glean from a wider context.

Am I organizing my site content/product features in a way that makes sense to users?

What You’re Looking For: Products that are described as “intuitive”, usually are because they align with how the user thinks about performing this task.   How do your users think about your task?  How would they logically group features or site sections?

Tool: Online card-sorting. Card sorting is an exercise where you list out all of the concepts involved in your product/site, and allow users to group them in any way that makes sense to them.  This you have to see to believe, because my first thought was “these will all be completely different! No two people think exactly alike!”  And yet, my experience has always been that strong patterns emerge with as few as 20 users.

List all of your features and the OptimalSort tool will ask users to group them however they see fit (you can do an “open” sort, where there are no constraints; or a “closed” sort, where you choose the categories and people sort concepts into them.)  It shows the grouping and patterns for you.  Because this is an online exercise, the compensation can be on the lower side.  You can also ask questions at the end – I like to ask “are there other features that should be here?”  Because you’ve set the context, you tend to get more constructive feedback in this context than if you normally asked “what else do you want?”

There.  Now you have no excuses!  Go forth and test.  If you have questions that I didn’t cover, list them in the comments.

THOU SHALT NOT MAKE CHANGES WITHOUT A/B TESTING.

OK. But this isn’t as straightforward as it sounds.   Google Web Optimizer will not help you test your WordPress blog.

• GWO is focused on single-action conversions, such as completing a signup or purchase.  I wanted to compare a series of metrics.
• It’s also not ideal for a database-driven site that uses common headers and footers.
• Around 25% of my traffic hits my homepage directly, but WordPress doesn’t have a simple way to create two homepages.
• I tweet a lot of single-article links, which means that every page needed an alternate.  But I wanted to avoid being penalized by Google for having duplicate content.
• I realized there was a lot of value in permanently having an “A” and “B” site so I could always be testing something.

So I decided to roll my own solution. It’s not particularly elegant.  But I’ve laid it out step-by-step so that anyone else with a self-hosted WordPress blog can try this themselves.

(And maybe, someone with better programming chops than myself will write a more streamlined plug-in version and make it available to the blogging community.)

You will need:

• A self-hosted WordPress blog (I don’t think this will work for a *.wordpress.com blog unless you can directly edit your header.php and footer.php files)
• Ability to back up your WordPress database (if something I wrote kills your site, I want you to be able to retrieve it)
• Ability to create new directories and edit files and permissions on your server
• Two Google Analytics accounts (one for the “A” site and one for the “B” site)
• Willingness to do some messy WordPress hacking

Pick what you want to test

You want to test a single variable (if you make multiple changes, you won’t know which one is driving changes!)  A few suggestions:

• Two different WordPress themes
• Minor CSS/color scheme changes
• Front page settings (i.e. showing 10 posts on homepage vs. showing 5 posts on homepage)
• Two different sidebar designs (that’s what this blog is testing)

Creating the “B” blog:

Step 1: Back Up

First back up your WordPress database.

Step 2: Manually Install “B” blog:

Create a sub-directory called bblog where your “B” blog will live.

Copy your entire original wordpress/ directory into the bblog/ directory.   This is effectively doing a manual install of WordPress for the “B” blog.  This ensures that all of your settings, themes, and plugins get copied over.

Step 3: Set Up “B” blog database:

Now you need to set up the MySQL database that will power the “B” blog.

You can run both copies of your blog off the same MySQL database. To do so, open up bblog/wp-config.php and change the $table_prefix from wp_ to wp2_ :

Step 4: Complete installation of your “B” blog:

Access your new WordPress dashboard by going to http://www.yourblogname.com/bblog/wp-admin/.  You will see the WordPress setup screen:

Enter your blog title and email address (both should be the same as for your original blog).

You may wish to uncheck the “Allow my blog to appear in search engines like Google and Technorati” checkbox while you’re setting this up.  You can always check it again later.

Click “Install WordPress” and write down your new Admin password.  You’ve installed the “B” blog!

Configuring the “B” blog

Step 5: Import your original blog entries to your “B” blog:

If you’ve logged into your “B” blog, you might be thinking: Where are my posts?!

Don’t worry – importing your posts is quite simple.

Open your original blog’s WordPress dashboard in a new browser tab.  Go to the Tools menu and select Export and click Download Export File. Your blog entries should immediately download as an XML file.

Now open your “B” blog’s WordPress dashboard.  Go to the Tools menu and select Import.  Choose WordPress as the import type and you’ll be prompted to upload a file.  (Note: WordPress says “Upload your WordPress eXtended RSS (WXR) file”, but the file is an XML file.  Just upload the XML file you just exported and it will work.)

Upload all authors and import your attachments as well and click Submit.

Step 6: Prevent duplicate content problems by adding canonical links:

Download the Canonical-B plug-in (this is a hacky modified version of Joost de Valk’s excellent Canonical plug-in.  It tells Google that the “real” link for the pages of your site is http://www.yourblogname.com/ not http://www.yourblogname.com/bblog/)

Add the Canonical-B plug-in to the plug-ins directory for your “B” blog.

(Also: I recommend you subscribe to Joost de Valk‘s awesome WordPress newsletter. Full of useful tips.)

Step 7: Finish configuring your “B” blog:

Go to the Appearances menu to choose the same theme that you are using on your original blog.  (Your “B” blog defaults to using the Default WordPress theme. )

Go to the Plugins menu to activate plugins for this new blog.  (You should see the newly-added Canonical-B plug-in along with whatever other plug-ins you had previously installed on your original blog.)

Go to the Users menu to replicate whatever users you had on your original blog.  (WordPress automatically creates a user called “admin”, but you may have created another user.  For example, I post as the “Cindy” user.)

Step 8: Update Google Analytics for your “B” blog:

If you haven’t done so already, log in to your Google Analytics account and create a new account (from the “Analytics Settings” page, click on the + Add New Account link and create a new account for http://www.yourblogname.com/bblog/).  Copy the javascript code snippet.

I’m assuming you put Google Analytics code in the wp-content/themes/THEME_NAME/footer.php file of your original blog.

So you’ll want to open up your bblog/wp-content/themes/THEME_NAME/footer.php file and replace the original Google Analytics code with the new code you just copied.

Now Google Analytics will separately track metrics for http://www.yourblogname.com and http://www.yourblogname.com/bblog.

Step 9: Set up the test differences for your “B” blog

Your “B” blog installation is set to the WordPress Default theme.  You’ll want it to change it so that it matches your original blog, except for the changes you are making for your test conditions.

Click through several posts on your “B” blog to ensure that it displays the way you want.  Once it looks good, now you’re ready to direct part of your traffic there!

Setting up your A/B Traffic Split

Step 10:  Set up canonical links and get the “B” versions of your links:

Download the Canonical-A plug-in (this is a second hacky modified version of Joost de Valk’s excellent Canonical plug-in.  It puts the canonical link in the header of your original blog, and gives you a variable for the “B” blog version.

Add the Canonical-A plug-in to the plug-ins directory for your original blog.

Step 11: Add the redirect script to your header to send X% of your traffic to the “B” versions of your blog:

Open wp-content/themes/THEME_NAME/header.php and add the following lines to the beginning of the file:

<?php

$url = yoast_guess_url($wp_query);

$bblogurl = str_replace('.com/', '.com/bblog/', $url);

$number=rand(1,100);

if ($number>=50)

  echo ' ';

else

  echo header('Location:' . $bblogurl);
?>

These lines must be at the very beginning of the file – no blank spaces or lines or code can come first.

This code will do a 50-50 split, sending half of your traffic to http://www.yourblogname.com/ and half to http://www.yourblogname.com/bblog/.

It also works for deep links, sending half your traffic to http://www.blogname.com/category/this-is-your-post and half to http://www.blogname.com/bblog/category/this-is-your-post.

If you want to test with a smaller percentage of your audience, change the line if ($number>=50). For example, to send only 10% of traffic to your “B” blog, you would change it to if ($number>=10).

You should not edit your “B” blog header.php.

Launch it! (And Measure)

Step 12: Begin the experiment:

Once you’ve saved this header.php change, your experiment has begun!

From now on, whether visitors come to your homepage or follow a deep-link, 50% of them will be sent to the “B” version.

(If you want to immediately stop the A/B test, just delete the code you added in Step 11 from your header.php file.)

Step 13: Some Constraints

Note that this 50-50 split happens each time a user loads the header – in other words, the user who visits multiple pages may be flip-flopped from “A” to “B” to “A” and so on.

This is not optimal, but since many blog readers aren’t reading multiple pages, I decided to go ahead and launch like this.  (I’m working on a cookied version that will maintain the “A” or “B” setting for an entire session – stay tuned.)

Also: the two blogs will not stay in sync.  After the initial export/import, you will need to publish new posts to your original blog, and then manually copy them over to the “B” blog.  Since I only post once a week, this is not too much overhead.

Step 14: Measuring

Once you’ve had both versions running for a few days, you can log into Google Analytics and compare the two accounts (be sure to set them to the same time period).   I’ve done this by opening the cindyalvarez-a and cindyalvarez-b reports in two browser tabs and flipping back and forth – crude, but it works.

Hope you enjoyed this and are able to put it to use.  Not everyone has access to an e-commerce site or a website funnel, but many of us have personal blogs and this is a good opportunity to dive into the details of your blog and try out A/B testing.