They’re right.  Freeform answers alone are flawed.  Multiple choice options alone are flawed.  You need to use them both together in order to generate unstoppable, reliable hybrid feedback!

There are two approaches to cultivating your hybrid feedback – pick the one that is most relevant to your situation.

Interview First, Then Survey

Use this method when: You have no idea what you’d even suggest as multiple choice options.

Example: People are really excited when they sign up for your book exchange website. But almost no one is completing a successful exchange, and honestly, you have no idea why. You can’t see a pattern to where people are dropping out of your workflow, and you’re not getting a lot of bug reports or complaints in your support inbox.

Where to start?: Talk to people.

You can call and ask questions about their book exchange needs (see earlier post on “what you should be learning?”), or ask a couple people to go through your website while you watch over their shoulder or using UserTesting.com, or some combination of the three.

Once you’ve talked to 5-10 people, you will usually see some patterns – concerns or obstacles that are affecting more than one person and that seem pretty plausible based on what you know of your own product.

You may find that the problems reported in these first 5-10 conversations/user tests are so fundamental that they’re actually preventing you from getting deeper feedback!  (For example, if people are having problems logging in, they’ll never get to use your core product enough to give you useful feedback on it.)   In that case, skip the survey and start fixing!

Or you can use this initial feedback to populate your multiple choice options, and run the numbers.

Personally (and yes, this is my opinion, not proven data), when I see a multiple choice question from a company and all of the options are really well-written and show a deep understanding of their product, I feel like they really care and am more likely to spend extra time writing out an “Other” response.

Survey First, Then Interview

Use this method when: You want to guide the conversation to specific, relevant options. OR You have a pretty good guess as to what the potential responses are.

Example: You have several large features on your product roadmap, all of which are aligned with your product vision, have supporting market research, and will provide different customer benefits.  But you’d like to validate that they really will be valuable to the customer, and get some subjective feedback on which one will “delight” your customers the most.

Where to start?: Survey.

Questions that may work for this context:

• Which of [list options] features are you most excited about?
• Which of [list tasks] do you use most frequently?
• Would one of these [list options] have convinced you to complete your purchase?
• Would you be willing to be a beta tester for one of these upcoming features [list]
• Have you experienced one of these issues [list]?  How did it affect you?

Once you’ve gotten 20-30 responses, you will usually see a clear winner (or two) emerge.

But now, you have to make sure you properly “decode” that feedback.  You need to understand the “what else” and the “why”.

What else? It may be true that the majority of your customers are most excited about [feature X], but they are assuming that it will magically “just work”.  It’s your job to understand and solve for things like Will this change my workflow?  Will this involve new people or exclude people who previously used it?  My boss is used to [competitor product] so she’ll want it to work like that does.

Why? It may be true that everyone is selecting “add customer reviews and commenting” as their preferred next feature.  But it may not be for the reason you assume.

Let me illustrate with a personal example:

CA: “I didn’t end up buying that [group buying site] Pilates deal, even though it was a good discount.”

Friend: “Why not?”

CA: “It wasn’t clear when you could use it — It would’ve been a waste of money if the classes were only at a time when I couldn’t go.  They should make the vendors give them more information or update their websites to be more clear.”

Friend: “Did you realize that [group buying site] offers a money-back guarantee?  You could’ve bought it and returned it if it wasn’t the right schedule.”

CA: “Ohhhh.  I didn’t know that!  I would’ve bought it if I’d known that!”

Which is easier: getting hundreds of vendors to submit information, and then wrangling that information into a CMS; or just highlighting a money-back guarantee icon?

So that’s why you then follow up with interviews.  Get details, try to disprove your assumptions, and then you’ll finally have the full understanding of your feedback.

]]> http://www.cindyalvarez.com/data-driven/hybrid-feedback-is-stronger/feed/ 7 http://www.cindyalvarez.com/data-driven/youve-got-questions-ive-got-tools http://www.cindyalvarez.com/data-driven/youve-got-questions-ive-got-tools#comments Wed, 22 Jul 2009 20:43:57 +0000 Cindy http://www.cindyalvarez.com/?p=483 “I really should do user testing, but…”

You know that early validation can save weeks of working down the wrong path, right?  You may have listened to a few tangential comments from users that illuminated a whole new path to differentiation.  You’ve probably seen an interface that was completely intuitive to everyone in your company … and completely baffling to everyone outside it.

But if you’re like most product managers and entrepreneurs, you’re not testing.

First of all, testing has the sense of a big, lofty thing.

We all remember creating science fair projects years ago – you needed a formal hypothesis, a control group and an experimental group, all the variables had to be controlled, you needed to take notes, and the whole thing culminated in a typed, double-spaced report with graphs and charts.  (If you’ve worked with User Research within a large enterprise company, you still see research presentations just like this – except in PowerPoint instead of a tri-fold posterboard.)

Axe it. Forget it. I officially absolve you of needing to be super-scientific and organized.  If anyone asks, you can say “Cindy said this was okay,” and send them to me. Some data is better than no data.

Let me repeat that:

SOME DATA IS BETTER THAN NO DATA.

SOME DATA IS BETTER THAN NO DATA.

SOME DATA IS BETTER THAN NO DATA.

Are we okay now? Good. Let’s keep going.

Second of all, you don’t know where to start.

There are tons of freely available web-based tools, and blog posts giving you advice and how-to and do’s-and-don’ts.  What should I use? What’s the right tool?  Is it really okay to skip face-to-face user testing?

What tool you use depends on what you are trying to find out.  Here are a few common questions and tools:

• Is this product even remotely interesting to people (or am I crazy?)
• Do people understand what my site/product does?
• Are users able to complete the tasks that are core to my site/product? (i.e. completing specific actions, purchasing)
• What are the areas of my site/product that are confusing or annoying users?
• What are the alternatives to my site/product?  (what are people using instead of me?)
• Which audience finds my site/product most appealing?
• What other features should my product offer? (how could I make this product more appealing/differentiated)
• Am I organizing my site content in a way that makes sense to users?

(If you don’t see your question type here, leave it in the comments and I will write a Part 2 to this post.)

Is this product even remotely interesting to people (or am I crazy?)

What You’re Looking For: This is the first question that follows a “Eureka!” moment.  You have an idea, it sounds good to you.  You ask your friend and it sounds good to her, too.  But you might both be crazy, so you need some validation – just enough to convince you that it’s worth putting in more time.

Tool: SEM on $5/day. Build a really simple landing page. Buy Google AdWords or targeted Facebook apps that describe your concept, and measure how many clicks you get.  Vary your ad the next day and see if clicks improve.  If no one clicks, no one cares.  (and you just saved yourself a lot of work implementing something no one wants.) If people click, you can move on to embedding a simple survey to ask 1-2 questions about what they would pay for.

Do people understand what my site/product does?

What You’re Looking For: Validation that your splash page or email marketing copy makes sense.  You want people to see it and think X and Y.  Are they thinking X and Y?

Tool: FiveSecondTest (or equivalent).  I’m actually not crazy about FiveSecondTest, just because five seconds is even too short for skimming.  But you can easily create your own online version.  Show your splash page for 10 or 15 or 30 seconds, then advance user to a single survey question that asks “What do you think [this product] would allow you to do?” I recommend Wufoo for creating a single-question form and embedding it directly in your page.  It’s free and their simple reports do the job.

For an extended version, you may have a third screen that asks “Did you think that this product did X?  Did you think this product did Y?”    But first you want their untainted impressions.

Are users able to complete the tasks that are core to my site/product? (i.e. completing specific actions, purchasing)

What You’re Looking For: Your product has certain things that ARE the product.  If you’re an e-commerce site, people need to buy things.  If you’re a bill payment product, people need to be able to pay a bill.  This is a quantitative question.

Tool: Google Analytics. Identify your core tasks and the pages that a user must go through if they successfully complete it.  Create a funnel to identify how many users make it through each stage of the funnel.  (Note: your product does not have more than 5 core tasks, even if it’s “really complex.”  A Boeing 747 has 3 core tasks – taking off, staying in the air, and landing.)

Tool: UserFly.  UserFly records mini-movies of your users as they navigate through your site.  You can watch (sometimes painfully) as users hover hesitantly over a button, then click the wrong one, or enter the wrong information into a form field.

What are the areas of my site/product that are confusing or annoying users?

What You’re Looking For: You know there’s a problem – you just don’t know what it is.  People aren’t completing the core tasks of your product, or your bounce rate is extremely high, and you don’t know why.  This is a qualitative question.

Tool: Task-based user testing. Get 5 users off Craig’s List and offer them $50 to come to your office for a 30-minute test.  Either videotape them or have one person to moderate and one to scribble notes furiously.  Ask them to complete core task #1 and encourage them to narrate out loud as they navigate.   Resist the urge to comment or correct them.  Do ask questions like “what do you think will happen if you click there?”  Do notice their body language – are they relaxed? tense? frowning?  Remember, these people were paid to be here – so their default mode should be “happy”.

If this sounds too hard, try getting 2 users off Craig’s List.  Or skip that and just beg your non-technical friend or neighbor who doesn’t know anything about your product to come be your test subject.  Even ONE person who is not you, can reveal a ton of insight.  One person testing takes less than an hour.  You must be willing to invest ONE hour that will save you days or weeks of development and frustration.

What are the alternatives to my site/product?  (what are people using instead of me?)

What You’re Looking For: What’s the bigger story of how you fit into your users’ lives?  What are their behaviors and the environment they conduct them in?  For example, before Netflix, users rented videos from Blockbuster, or didn’t rent movies at all because it was too much of a hassle compared to just watching whatever was on cable.

Tool: In-person (or phone) interview. Get 5 people off Craig’s List and compensate them to come to your office or agree to a phone interview.  Ask them to tell you about the last time they did [your behavior].  For example, “Tell me about the last time you watched a movie at home”.  Resist the urge to comment, but ask helper questions to keep the narrative going – “How did you decide to watch a movie?”, “Was there a specific type of movie you were interested in?”, “Why did you watch a movie instead of doing something else?”, “How did you decide which movie to watch?” , “Who else was part of this decision?”, “What do you wish had been simpler/less frustrating/faster about the process?”

Tool: SurveyMonkey. Ask similar questions but via online survey.  I find this less insightful because you are far less likely to get freeform, “tangent” answers, but it’s very fast and doesn’t require someone who is comfortable giving an interview verbally.

Which audience finds my site/product most appealing?

What You’re Looking For: Should I be targeting [these people] or [those people]?  What description will convince people to convert? Where should I focus my further research/feature development?

Tool: SEM on $5/day. You can buy very targeted ads for cheap on Facebook (by now, all the demographics are there.) – set up multiple audiences and send them to your landing page, use an embedded survey to ask a few questions.  Measure the clickthrough rates and the survey response rates to get your first pass at who is most interested in your product.

Tool: Google Website Optimizer. Create two splash pages with different messaging that targets two different audiences.  Each one needs a “Sign Up Now” button that leads to a confirmation page. That’s all you need – don’t even need to link to your product yet.  GWO will compare the percentage who converted (landed on the confirmation page).  Clicking the “Sign Up Now” button is a clear measure of interest.  Repeat with as many “markets” as make sense.

What other features should my product offer? (how could I make this product more appealing/differentiated)

What You’re Looking For: Solving the big pain point.  What is it?

Tool: Task-based user testing. Get 5 users off Craig’s List and offer them $50 to come to your office for a 30-minute test.  Either videotape them or have one person to moderate and one to scribble notes furiously.  Ask them to complete core task #1 and encourage them to narrate out loud as they navigate.  As they navigate your site, have them describe where they are and what they’re doing when using your product (or the general behavioral context).

For example, the idea for one-handed baby wipes came from mothers describing what they did — “and I’ve got one hand keeping the wiggling baby from rolling off the changing table, and now I need to reach for a wipe…”

Note: there are various customer tools like UserVoice and SuggestionBox that allow customers to suggest features.  I’m deliberately not recommending those, because I subscribe to the Henry Ford philosophy that his customers would’ve suggested a faster horse. Users, generally speaking, can always come up with things they want but are not as good at articulating true pain points.  That’s what the experienced product manager or entrepreneur has to glean from a wider context.

Am I organizing my site content/product features in a way that makes sense to users?

What You’re Looking For: Products that are described as “intuitive”, usually are because they align with how the user thinks about performing this task.   How do your users think about your task?  How would they logically group features or site sections?

Tool: Online card-sorting. Card sorting is an exercise where you list out all of the concepts involved in your product/site, and allow users to group them in any way that makes sense to them.  This you have to see to believe, because my first thought was “these will all be completely different! No two people think exactly alike!”  And yet, my experience has always been that strong patterns emerge with as few as 20 users.

List all of your features and the OptimalSort tool will ask users to group them however they see fit (you can do an “open” sort, where there are no constraints; or a “closed” sort, where you choose the categories and people sort concepts into them.)  It shows the grouping and patterns for you.  Because this is an online exercise, the compensation can be on the lower side.  You can also ask questions at the end – I like to ask “are there other features that should be here?”  Because you’ve set the context, you tend to get more constructive feedback in this context than if you normally asked “what else do you want?”

There.  Now you have no excuses!  Go forth and test.  If you have questions that I didn’t cover, list them in the comments.

THOU SHALT NOT MAKE CHANGES WITHOUT A/B TESTING.

OK. But this isn’t as straightforward as it sounds.   Google Web Optimizer will not help you test your WordPress blog.

• GWO is focused on single-action conversions, such as completing a signup or purchase.  I wanted to compare a series of metrics.
• It’s also not ideal for a database-driven site that uses common headers and footers.
• Around 25% of my traffic hits my homepage directly, but WordPress doesn’t have a simple way to create two homepages.
• I tweet a lot of single-article links, which means that every page needed an alternate.  But I wanted to avoid being penalized by Google for having duplicate content.
• I realized there was a lot of value in permanently having an “A” and “B” site so I could always be testing something.

So I decided to roll my own solution. It’s not particularly elegant.  But I’ve laid it out step-by-step so that anyone else with a self-hosted WordPress blog can try this themselves.

(And maybe, someone with better programming chops than myself will write a more streamlined plug-in version and make it available to the blogging community.)

You will need:

• A self-hosted WordPress blog (I don’t think this will work for a *.wordpress.com blog unless you can directly edit your header.php and footer.php files)
• Ability to back up your WordPress database (if something I wrote kills your site, I want you to be able to retrieve it)
• Ability to create new directories and edit files and permissions on your server
• Two Google Analytics accounts (one for the “A” site and one for the “B” site)
• Willingness to do some messy WordPress hacking

Pick what you want to test

You want to test a single variable (if you make multiple changes, you won’t know which one is driving changes!)  A few suggestions:

• Two different WordPress themes
• Minor CSS/color scheme changes
• Front page settings (i.e. showing 10 posts on homepage vs. showing 5 posts on homepage)
• Two different sidebar designs (that’s what this blog is testing)

Creating the “B” blog:

Step 1: Back Up

First back up your WordPress database.

Step 2: Manually Install “B” blog:

Create a sub-directory called bblog where your “B” blog will live.

Copy your entire original wordpress/ directory into the bblog/ directory.   This is effectively doing a manual install of WordPress for the “B” blog.  This ensures that all of your settings, themes, and plugins get copied over.

Step 3: Set Up “B” blog database:

Now you need to set up the MySQL database that will power the “B” blog.

You can run both copies of your blog off the same MySQL database. To do so, open up bblog/wp-config.php and change the $table_prefix from wp_ to wp2_ :

Step 4: Complete installation of your “B” blog:

Access your new WordPress dashboard by going to http://www.yourblogname.com/bblog/wp-admin/.  You will see the WordPress setup screen:

Enter your blog title and email address (both should be the same as for your original blog).

You may wish to uncheck the “Allow my blog to appear in search engines like Google and Technorati” checkbox while you’re setting this up.  You can always check it again later.

Click “Install WordPress” and write down your new Admin password.  You’ve installed the “B” blog!

Configuring the “B” blog

Step 5: Import your original blog entries to your “B” blog:

If you’ve logged into your “B” blog, you might be thinking: Where are my posts?!

Don’t worry – importing your posts is quite simple.

Open your original blog’s WordPress dashboard in a new browser tab.  Go to the Tools menu and select Export and click Download Export File. Your blog entries should immediately download as an XML file.

Now open your “B” blog’s WordPress dashboard.  Go to the Tools menu and select Import.  Choose WordPress as the import type and you’ll be prompted to upload a file.  (Note: WordPress says “Upload your WordPress eXtended RSS (WXR) file”, but the file is an XML file.  Just upload the XML file you just exported and it will work.)

Upload all authors and import your attachments as well and click Submit.

Step 6: Prevent duplicate content problems by adding canonical links:

Download the Canonical-B plug-in (this is a hacky modified version of Joost de Valk’s excellent Canonical plug-in.  It tells Google that the “real” link for the pages of your site is http://www.yourblogname.com/ not http://www.yourblogname.com/bblog/)

Add the Canonical-B plug-in to the plug-ins directory for your “B” blog.

(Also: I recommend you subscribe to Joost de Valk’s awesome WordPress newsletter. Full of useful tips.)

Step 7: Finish configuring your “B” blog:

Go to the Appearances menu to choose the same theme that you are using on your original blog.  (Your “B” blog defaults to using the Default WordPress theme. )

Go to the Plugins menu to activate plugins for this new blog.  (You should see the newly-added Canonical-B plug-in along with whatever other plug-ins you had previously installed on your original blog.)

Go to the Users menu to replicate whatever users you had on your original blog.  (WordPress automatically creates a user called “admin”, but you may have created another user.  For example, I post as the “Cindy” user.)

Step 8: Update Google Analytics for your “B” blog:

If you haven’t done so already, log in to your Google Analytics account and create a new account (from the “Analytics Settings” page, click on the + Add New Account link and create a new account for http://www.yourblogname.com/bblog/).  Copy the javascript code snippet.

I’m assuming you put Google Analytics code in the wp-content/themes/THEME_NAME/footer.php file of your original blog.

So you’ll want to open up your bblog/wp-content/themes/THEME_NAME/footer.php file and replace the original Google Analytics code with the new code you just copied.

Now Google Analytics will separately track metrics for http://www.yourblogname.com and http://www.yourblogname.com/bblog.

Step 9: Set up the test differences for your “B” blog

Your “B” blog installation is set to the WordPress Default theme.  You’ll want it to change it so that it matches your original blog, except for the changes you are making for your test conditions.

Click through several posts on your “B” blog to ensure that it displays the way you want.  Once it looks good, now you’re ready to direct part of your traffic there!

Setting up your A/B Traffic Split

Step 10:  Set up canonical links and get the “B” versions of your links:

Download the Canonical-A plug-in (this is a second hacky modified version of Joost de Valk’s excellent Canonical plug-in.  It puts the canonical link in the header of your original blog, and gives you a variable for the “B” blog version.

Add the Canonical-A plug-in to the plug-ins directory for your original blog.

Step 11: Add the redirect script to your header to send X% of your traffic to the “B” versions of your blog:

Open wp-content/themes/THEME_NAME/header.php and add the following lines to the beginning of the file:

<?php

$url = yoast_guess_url($wp_query);

$bblogurl = str_replace('.com/', '.com/bblog/', $url);

$number=rand(1,100);

if ($number>=50)

  echo ' ';

else

  echo header('Location:' . $bblogurl);
?>

These lines must be at the very beginning of the file – no blank spaces or lines or code can come first.

This code will do a 50-50 split, sending half of your traffic to http://www.yourblogname.com/ and half to http://www.yourblogname.com/bblog/.

It also works for deep links, sending half your traffic to http://www.blogname.com/category/this-is-your-post and half to http://www.blogname.com/bblog/category/this-is-your-post.

If you want to test with a smaller percentage of your audience, change the line if ($number>=50). For example, to send only 10% of traffic to your “B” blog, you would change it to if ($number>=10).

You should not edit your “B” blog header.php.

Launch it! (And Measure)

Step 12: Begin the experiment:

Once you’ve saved this header.php change, your experiment has begun!

From now on, whether visitors come to your homepage or follow a deep-link, 50% of them will be sent to the “B” version.

(If you want to immediately stop the A/B test, just delete the code you added in Step 11 from your header.php file.)

Step 13: Some Constraints

Note that this 50-50 split happens each time a user loads the header – in other words, the user who visits multiple pages may be flip-flopped from “A” to “B” to “A” and so on.

This is not optimal, but since many blog readers aren’t reading multiple pages, I decided to go ahead and launch like this.  (I’m working on a cookied version that will maintain the “A” or “B” setting for an entire session – stay tuned.)

Also: the two blogs will not stay in sync.  After the initial export/import, you will need to publish new posts to your original blog, and then manually copy them over to the “B” blog.  Since I only post once a week, this is not too much overhead.

Step 14: Measuring

Once you’ve had both versions running for a few days, you can log into Google Analytics and compare the two accounts (be sure to set them to the same time period).   I’ve done this by opening the cindyalvarez-a and cindyalvarez-b reports in two browser tabs and flipping back and forth – crude, but it works.

Hope you enjoyed this and are able to put it to use.  Not everyone has access to an e-commerce site or a website funnel, but many of us have personal blogs and this is a good opportunity to dive into the details of your blog and try out A/B testing.

100% of your audience shows up.

Some % shows some sign of interest.

Some % commits an investment of time, personal information, and/or money into your product or service.

Looking at the numbers in between will prompt a lot of questions that you may not know the answer to, but should.   I can’t tell you what you’ll learn, but I can walk through some of the questions that your numbers should drive you to ask.

For ease of discussion, we’ll assume we’re looking at an online product, say, a sock-matching service.  (Same general rules apply for physical products, but data collection is more complicated.)

• Homepage: 20,000 views
• Register:  2,500 views
• Confirm registration: 600 views

20,000 views to the homepage

I hear a lot of product managers or marketers brag about page views. There are only two times when this is relevant:

1) It’s a really, really, really, really big number AND you’ve sold out ad inventory AND your overhead expenses are really low.  (Congratulations, you’re hotornot.com or icanhascheezburger.com, and you’re raking in the $.)

2) It’s not a very big number AND you spent tons of $ on ad buys or direct mailers hoping to get more people in the door.  (Sorry, you spent that marketing budget poorly.)

EVERY OTHER TIME, pageviews numbers is just a denominator.

Pageviews don’t translate into happy users,they don’t translate into revenues. They’re only relevant in comparison to the number of people who showed interest, and people who eventually convert to real customers.

For our sock matching service, we need happily socked users who will pay us – not flip-flop wearing visitors who come by and don’t see anything that serves their needs.

2,500 views to registration

I say “registration”, but generically you want to think in terms of “showed some sign of interest”.   What percentage of your audience clicked on a “learn more”, or watched a demo, or clicked into the “Solutions” section of your site?

2,500/20,000.  Some of those 20,000 might have been repeat visits from the same user, but that doesn’t necessarily matter.  If it takes repeat visits before a potential customer is willing to show some interest, something’s not resonating strongly enough with them.  So, 2,500/20,000 = 12.5% of your audience showed some interest.  87.5% did not.

87.5% is a big number.  Before we move on, let’s think about what that could mean.

Visitors could’ve been expecting something different (like people searching for “green living” and landing on a site selling green sofas rather than an eco-friendly site).

In Google Analytics, you can look for the search terms that sent visitors to you.  Do they make sense?  Are they terms that you feel are relevant to your business?  If not, look at the terms that do make sense and make sure your content contains more of them.

Look at the sites that referred visitors to you.  Did you get a huge wave of traffic because lifehacker.com or oprah.com linked to you?  Spikes like that might include lots of people that you aren’t targeting to be customers.

If you aren’t sure why so many people left without showing interest, quick surveys can help.   You could also use SurveyMonkey or 4Q to show very short surveys to people when they navigate away from your site.  I’d advocate non-open-ended questions (“Did you realize that Sock Sorters would prevent you from ever wearing navy and black mismatched socks again?”) – you get less information but a much higher response rate.

Visitors could’ve been completely confused by what your product was.

A good tool for this is the 5-second test.  It’s an online tool which will allow you to show a page to users for 5 seconds and then find out what they remembered about it.  You don’t need actual customers for this – any subjects you can find will reveal a lot about how people interpret the messaging you put out.

Find a few highly targeted customers and interview them.  You know that person X desperately needs sock sorting services, so ask them about their sock behavior.  What problems do they have, what steps do they go through, what things frustrate them.  Then walk through your site with them.  Do they “see” that you’re solving their problem?  Did they use words to describe their problem that are completely different from the language on the site?

Visitors could’ve been seen something intimidating/unprofessional/scary that made them distrust you.

The 5-second test would work well here.  So would a quick heuristic analysis by a neutral user experience person.  An experienced UX person can recognize certain things – lack of a secure site icon, specific phrases that don’t sound ‘right’, conventions that look spammy – that might not jump out at someone who has seen the page every day for months.

They just don’t see the button.

Visitors can’t respond to your call to action if they don’t notice it.   Your call to action may not pop enough.  Your language may be unclear.  CrazyEgg is a simple javascript software that shows where visitors are mousing and scrolling on your site.  If their activity is concentrated in one area and your call to action is somewhere else, a simple move could double your conversions.

Or you may not load in time.  With my current product, a module that loads in third-party sites, we found that many visitors were skimming the page so quickly that they were gone before we appeared on the page!    This was a critical discovery: it immediately shifted our product priorities.  There’s no sense building extra features to make 20% of the population happier when there’s an untapped market of 80% who never see the product!

600 views to confirm registration

Again, “confirmed registration” is shorthand for “successful conversion”.  You’ve got yourself a customer!  Now all you have to do is keep them.

But wait – not everyone who expressed interest turned into a successful conversion.  600/2,500 = 24% of people who expressed interest, who are definitely target customers, actually took the next step to become one.

Now, how bad (or good) is that?  It depends on how high the investment (or friction) is.  If you have a free product with one-click registration and only 24% of interested visitors convert, that’s pretty crappy.  If you have an expensive product, or one that requires the visitor to do several steps of setup, that’s not terrible.  (Most banks would be thrilled if 24% of people who started setting up bill pay actually finished setup and paid a bill, for example.)

Let’s assume we’re not happy with 24% conversion (76% non-conversion).  What can we look at to understand more about the 76% who didn’t make it through?

Did the people who dropped out come from a common origin?

In Google Analytics, you can look at Traffic Sources to find out more about your visitors based on who sent them to you.  One thing I noticed years ago when I worked for a gaming startup, is that hard-core, highly tech-savvy bloggers would write enthusiastic reviews of tools.  Their fans would race over… but not being as tech-savvy, were not as willing to invest in learning those tools.  The audience liked the idea of gaming tools, but needed clearer messaging and an easier user experience to benefit from them.

Did the people who dropped out visit during specific times?

In my experience doing consumer interviews for financial software, people commonly said, “Oh, I started setup while I was at work, then realized I needed information that I had in my desk at home” or “It sounded interesting, but I couldn’t spend that much time while at work so I meant to come home and do it later.”   Two ways to help with this: clearly set time and materials expectations upfront (you will need 5 minutes and your latest utility bill); send an email reminder to come back and finish later.   I implemented the expectation-setting with our products at Yodlee and it (with other best practices) contributed to a ~20% uptick in completed registrations for our credit card bill payment application.

Obviously, if you have a multi-step setup, it should be easy to stop and return later without having to re-enter data.  (Adaptive Path has an interesting case study on redesigning PayCycle’s setup flow to allow for this.)

Did the people who dropped out all leave on a specific step?

You can look at the Exit Pages in Google Analytics to see this.  Frequently, the page that asks for money is a dropoff page; if money’s not an issue, design often is.

How this knowledge should affect your roadmap

Customers need a voice, and often are the source of great product innovations.  But it’s easy to “innovate yourself into a corner” if you center your product roadmap decisions around your existing customers.

In the example above, we started with 20,000 people in our audience, got to 2,500 interested people, and ended with 600 customers.  If those 600 customers were paying you $1 each, you’d be making $600.  To double your revenues, you’d have to convince these customers to buy twice as much stuff, or you’d have to make them so ecstatically happy that you had 100% new customer referrals.  Both of these are possible, but really, really hard.

If you look at the 1,900 people who expressed interest but didn’t become customers, you only need to make about 30% of them happy in order to double your customer base and revenues.   Creating a feature that is neutral to your customers but a differentiator to your uncomfortable potential customers has a much higher potential return.

Logically, that audience – the population of people who are aware of you – is going to break down into three groups.

Current and future customers. If they’re not already using your product, it’s just a matter of time until they do.  It solves their problem and they’re ready and willing to take on the cost (whether measured in dollars, resources, or learning curve.)

Uncomfortable prospective customers. There’s something preventing this group from becoming customers.  Maybe it’s something very concrete like pricing or lack of a specific feature, or maybe it’s something they can’t quite put their finger on. Unless you remove the hurdle in their way, they’re unlikely to become customers.  If they do, it will be with some reluctance.

Non-customers. Your product doesn’t solve their problem.  Or it does, but they’re unyieldingly opposed to change, spending money, or trying new things.  You can’t add features or lower prices to tempt them; only time or a deux ex machina mandate to use your product will change their minds.

If you’ve read Crossing the Chasm, this may sound suspiciously like the technology adoption lifecycle curve: 

And it is.  Where I’m concerned is with the labeling of the groups: “innovators”, “early adopters”, “early majority”, “late majority”, “laggards”.

It suggests that certain populations just aren’t ready to use your product – as though that was a personal failing of theirs rather than a shortcoming of the product.   Most products organizations already identify more with the innovator/early adopter population than the mainstream.  Naturally!  We love technology, that’s why we’re working elbows-deep in it.   And that makes it very tempting to disregard feedback from people who aren’t “like us”, or to forget that they’re even there at all…

<– Most companies I’ve worked for or with think their audience breakdown looks something like this.

<– But what if it actually looks like this?

Coming up next: How we discovered our “pie” was out of whack, and how you move on from there.