I’ve seen a lot of bias towards looking for quantitative data as early as possible – and in my opinion, it’s often too early.

Here’s my rule of thumb:

• If you’re asking what, why, or how do you questions, you want qualitative data.
• If you’re asking which or how many/how often questions, you want quantitative data.

Here’s the the tricky part:  making sure your “which” or “how many/how often” questions aren’t actually “what/why/how” questions in disguise.  I see these two examples a lot:

• Which of these features do you find most valuable? Sounds like a “which” question – but if you’re just guessing at the choices, it’s probably a “what” question — What do you find most valuable about our product?
• How often are users logging in? Do you really need to know the frequency, or are you using this as a proxy measurement of user engagement?  If so, you should turn it into a “how” question — “How do you use our service?” or “How valuable do you find our service?”

It’s tempting to try and turn as many questions as possible into quantitative ones, because those are easier to measure through unobtrusive means – web analytics, looking for data patterns, surveys – and give clear numerical answers.

However, it’s the messy, hard-to-collect and harder-to-interpret data that will lead you to breakthrough insights – not the percentiles and bar graphs.  Talk to people!

Like many small startups, we don’t have a dedicated QA tester, and we don’t have fully automated testing practices in place.  Here’s some of what we did well (and not so well) to get ready for release beyond ourselves.

Browser Support

1. Pick which browsers you will support and write it down. This sounds so obvious that you probably won’t do it, and then you’ll find out later that someone spent four hours testing on Chrome even though you weren’t planning on supporting it yet.   For now, we’re supporting Win/IE8, Win/IE7, Win/Firefox, Mac/Firefox, Mac/Safari.
2. Do browser detection and show a message to people using other browsers. If you’re in early beta, you can get away with supporting fewer browsers, but you really do need to set expectations.
3. Divide up browser testing. If you don’t explicitly say “you test Win/IE7 and I’ll test Win/IE8, etc.” you will end up duplicating efforts and missing things.

Emails

1. Make sure you have a huge source of available email addresses to test signups. This time, I just set up my personal domain so that [any username] at [domain] would forward to me.  Another way is to pre-create a bunch of testing1, testing2, testing3 accounts at the company domain.
2. Have every kind of inbox available - GMail, Mail.app, Thunderbird, Outlook, iPhone, etc.  We didn’t do this and so I still haven’t previewed the emails we send in different environments.   Set this up in advance!
3. Have some third-party friends who can review. After the second or third time you check the emails your app sends, you will stop noticing things.

Bugs

Define bug reporting best practices.  Our developers did this about halfway through (after being annoyed by vague bugs).

For workflow bugs, we are now trying to follow the example below:

Flow:

1. User tries to activate a survey.
2. user sees error
3. checkbox is checked until page reload

Expected behavior
Checkbox becomes unchecked if the activation fails.

How to reproduce

1. have two or more surveys
2. set the url to be the same on both surveys (at least one must be deactivated)

For UI bugs, we are trying to follow:

1. Provide a screen shot.
2. Find out what browser/OS.
3. If it came from another user, try to replicate it yourself and tell us if you are able to.
4. Provide a link to where the issue was found
5. If at all possible: view source and look for anything weird

It would’ve saved a lot of time if we’d established these examples/guidelines up-front, but hopefully by writing this, you can and will.

…but somehow, people still don’t seem to be very excited.  What went wrong?

MVP without FUE = DOA.

(That’s “minimum viable product without first user experience = dead on arrival”.)

When you were working with early customers, it wasn’t that important to flesh out the first user experience.  After all, they were right there with you, listening to you articulate the value proposition and how your solution would help them with their problem.  Those customers had the benefit of your insights, explained through multiple interactions.  They share in the curse of knowledge.

But your new customers don’t.

After hundreds of customer development interviews, user testing sessions, and survey responses, I’m going to share with you the single most common barrier to adoption I’ve heard: “I don’t know where to start.”

Consumer vs. enterprise, techies vs. non-techies, it doesn’t matter. Whether it’s organizing their personal finances, engaging in customer research, eating healthier, starting a blog,  understanding their web analytics, or organizing their baby photos — this is what blocks people.

The Customer Excitement Lifecycle

People go through a pretty predictable cycle of emotions when they try to solve their pains.

1) In pain: “I really need to start eating healthier – I feel terrible.”

2) Glimmer of hope: If you’ve done a good job articulating your value proposition, the customer sees your product and feels hope: “Can this product can help me to eat healthier?  That would be so great…”

3) Leap of faith: Honestly, they’re not reading your ‘about’ page or watching your demo video – they’re just eager to get started fixing their problem, so they click ’sign up’: “This is the first step towards healthy eating!”

4) Expectations crash: There’s no immediate gratification – lots of links and features, but no guidance on where they should focus their attention.  Realization hits: “Hmm… is this really going to help me? This seems like a lot of work, not sure if I have the time and energy to get started…”

If you neglect the experience between steps 3 and 4, your product will wither and die. Yes, you should build a minimum viable product.  And your minimum viable product needs to include a compelling first user experience.

Most of us are only ’selling’ our customers once when we need to be selling them twice.  The first ’sell’ is convincing them to click “sign up” and give you that initial try – and most customer development practitioners are doing that pretty well.

The second ’sell’ is convincing the customer to continue investing time and thought into your product.   If they log in to their dashboard and all they see is “You have no new [whatever]” and a “View demo” link, they will leave and not come back.

Channel That Natural Enthusiasm

When your customers first sign up for your product, they are excited! They are motivated!  They have already decided to invest some time and energy in you – don’t waste it.

The best way to manage expectations crash is to channel the customer’s natural enthusiasm into an immediately productive activity.   Give them a task that they can complete quickly, that brings them closer to getting value from your solution.

A few sites that do this well:

Daily Burn asks you to set up fitness goals immediately, reinforcing the reason why you signed up.

Lil Grams gives you 3 things to do and 3 things to learn - enough to get you started without overwhelming.

Dropbox encourages you to try features you might not have noticed otherwise (and increases word of mouth behaviors at the same time).

Finding People

AdWords / Facebook Ads / Tweets.

Summarize your idea, invest some money in getting it in front of people who have expressed intent by searching for that term, clicking your ad, clicking a link.  (Read the original SEM on $5/day post for details.)

I haven’t used Twitter for this much yet, but my theory is that it may be a more effective way of reaching people (I am much more likely to click on links that appear in my hashtag saved searches than I am to click on an AdWords or Facebook ad.)

Twitter Search.

Look for people who have already discussed a similar product, problem, or solution and address a tweet directly to them:

@username Would love yr feedback on [product/problem/solution] – shd only take 2mins [URL] thanks!

Some people will ignore this, but many more will feel a bit flattered that they’re being asked.  Use this judiciously – more than one or two of these tweets per day and you’ll look like a spammer.

Google Alerts.

Set up Google Alerts for your product/problem/solution (you should have done this already anyways) – and when it finds relevant blog posts or comments, email those people and ask for their feedback:

I read your [post/comment] about [product/problem/solution].  I’m currently trying to validate a related idea and I think your opinion would be very valuable to me – could you take 2 minutes and check out [URL]?  Thank you – I’d be happy to return the favor any time.

Ask for introductions.

People are generally happy to make introductions for you, provided you do 3 things:

• Provide the exact text that they can copy and paste into a tweet or email (They’re doing you a favor! Make it as easy as possible for them.)
• Tell them exactly how you are going to communicate with their contacts (They’re risking a bit of social capital for you – if you are a jerk to their contacts, that will reflect badly on them.  Be very clear that you won’t spam or annoy people.)
• Tell them your goals (What do you think you’ll get/learn if they make this intro for you? People want to know that they’re contributing to a bigger picture!)

Email Request Template

I have a quick favor to ask.

I’ve got a product idea that I’m trying to validate with [type of customer]. My goal is for them to visit my splash page at [URL] and indicate their interest (or lack thereof).  I will only contact them if they explicitly give me permission to do so.

Could you send this message along to people you know who fit this target?  (Feel free to change it a bit if you like):

[Message - be sure to include the goal, the URL, and your contact information]

Twitter Request Template

I have a quick favor to ask.

I’ve got a product idea that I’m trying to validate with [type of customer]. My goal is for them to visit my splash page at [URL] and indicate their interest (or lack thereof).  I will only contact them if they explicitly give me permission to do so.

Since you have a number of followers who are the type of customer I’m trying to reach, could you tweet this for me? (Feel free to change a bit if you like):

[Message - include the URL, the topic, and keep it under 115 chars so it can be easily retweeted]

Asking for the Interview

You may be wondering, “so what is this URL I’m sending people to?  Can’t I just have people email me?”

You have three main goals with your splash page:

1. Communicate your idea in 10 seconds or less (seriously, that’s about how much time you have to grab someone’s attention)
2. Offer something interesting to the people who visit
3. Get contact information so you can ask for the interview

#1 is up to you (there’s a whole other blog post I could write about that…).

#2… when I say “offer something”, people generally think that means a tangible incentive.  You can, but you probably don’t need to – people like being asked for their expert opinion, they like the feeling that they’re contributing to something, and they like being part of a select group who gets a sneak preview at something.

You can cover #2 and #3 with a well-written survey template.  You can see the one we’ve used for the Survey.io beta, or here’s a partial screenshot:

FYI – I’ve used surveys with these 2 questions for multiple products, and so far, overall less than 20% of ALL respondents to this survey leave these blank – the vast majority choose at least one way in which they’d like to give feedback, AND give an email address.

Coming next week:  I’ve got email addresses – now what?

It’s OK to say you belong or reside within and have an innovation team within a corporate/business environment. Innovation is not a dirty word. - Carl Knibbs, Cup of Innovation, Anyone?

I agree!  I’ve seen two reasons why “innovation teams” are poorly regarded within larger organizations.  Actually, they apply to pretty much any type of “special task force/skunkworks” team:

1. Lack of internal credibility, i.e. the “what the heck does the INNOVATION team do? Yeah, I’d like to be an innovator instead of having real work to do” reaction
2. Silo syndrome, i.e. “the rest of us don’t need to be creative, we have an innovation team to handle that.”

Lack of Internal Credibility

This usually springs from the best of intentions.   Everyone else is saddled with meetings and process and existing projects – let’s give this team some breathing room so they can really focus on creating something great!

There’s often a lack of defined deliverables – after all, if you’re experimenting, you can’t guarantee what will work.  But when a team — any team — isn’t producing results of some kind, they’re going to be internally badmouthed.  When it’s time for them to advance a genuinely innovative and exciting project, they won’t have the credibility needed to work effectively with other internal teams.

A couple years ago, I was on a project with a large corporate organization who had an “innovation team”, and one of its leaders — we’ll call him “Clint” — was part of our cross-functional group.  Clint was tasked with exploring new technologies, and he did so enthusiastically.  In meetings where I could see his laptop screen, he was enthusiastically tweeting, blogging, and playing with various Web2.0 apps.

“We should really use a Flickr-style interface for that feature,” he’d say, breaking into a conversation that had been underway for 20 minutes.
Cool idea – but no business case or usability enhancement justification for it.  Unfortunately, that wasn’t part of his job.  He was tasked with “exploring”, not “delivering”, and as a result no one was really sure what he did.

The thing is, no one should be “innovating” 100% of the time.  It’s not even possible!

Solution #1: Define deliverables up-front.  Ideally, the leader of the team should reach out to other groups to find out what format would be most convenient for them.  A casual brown-bag presentation is a good bet – with a simple format of “here’s what we learned, here’s what happened, here’s what’s worth keeping” and some time for Q and A.  Sending out a “formatted for executives” PowerPoint deck or a long email is probably going to be less popular.

Solution #2: Ask other cross-functional teams what THEY need.   Does engineering wish they could get customer feedback on integrating a new technology? Does marketing wish they could do an ad hoc demographic survey?   Taking care of some other teams’ pet projects may only require a few hours – but it can be enormously helpful in getting their buy-in on future projects.

Silo Syndrome

Sometimes this is the fault of management: “If you have an idea, send it to the innovation team.” (subtext: No, you don’t get the fun of continuing to think about it, and you probably won’t get any credit for it if it turns out to be valuable.)

Sometimes this is the fault of the day-to-day folks: “We don’t have time to think about that idea, that’s what the innovation team is for.  Now let’s get back to doing things exactly the way we always do them!”

Either way, talk about demoralizing!  The point of introducing skunkworks teams is to try and spread new ideas and enthusiasm, not to tamp it down in everyone else.

The innovation team needs to make their energy and get-it-done agenda contagious.

This can be tricky if the silo-ization is coming from upper management, but it’s part of the job.

Solution #1: Find some “insiders” who will keep you informed if they hear about interesting ideas being quashed in other teams.  (Project managers are often the notetakers and have a really good sense of this.)  Actively seek out the people with ideas and ask them about it.  It may not be feasible, but you want to send the message that you’re listening.

Solution #2: Talk to the people who are fighting to maintain the status quo and understand what their concerns are.  Often, their performance is judged based on metrics from doing things “the usual way”.   There’s no incentive for them to try something new, only a pretty good chance of them looking bad because of it.   Look for ways that you can mitigate that risk to them – maybe by minimizing change, maybe by taking responsibility for outcomes.

The skunkworks team needs to see themselves as partners – it’s almost a good cop/bad cop relationship.  You do things your way, we’ll try this other way – but we’re both working together towards a common goal.

As a product manager in startups, I’ve usually approached larger customers with this partnership approach.  You have the big revenues and the big customer base; we have more freedom to operate quickly and outside of procedure.  What rules do you want us to help you bend?

In my experience, it has often been doing “unofficial” survey research, user testing or customer interviews, quick prototyping, or checking out competitive products – and using that to kick off new ideas or approaches.  It could be anything.  The point is, you need an objective, a connection to the core business objectives, and a commitment to sharing results.  Doesn’t sound “innovative” enough?  Give it a try!

It could have been avoided if I’d been able to successfully convince us that we should sacrifice a little bit of backwards-compatibility.  It would’ve meant customer complaints. Some customers would have delayed their upgrades; some may have even threatened to not renew their contracts (although the chance of them carrying out that threat was incredibly slim.)  In other words, it’s a decision that would’ve caused a lot of up-front pain.

Here’s the story, in short:

Product redesign.  Fundamental change to product functionality and positioning (and one that had been well-researched, thoroughly user-tested, and really well-received.)  Just one problem: we’d made the promise that the layout of the UI would be backwards-compatible (as a white-labeled software package, we were styled to match the branding and templates of the ‘parent’ website).  “You won’t have to change your templates or stylesheets or the order of the modules on your dashboard,” we’d said.

But it just wasn’t possible.  When you add information, you just can’t fit within the exact same dimensions.  After a lot of reworking, we had a lot of workarounds:  Customers could use flexible-width instead of fixed-width.  They could shift from a double-column to a single-column layout.  They could keep the double-column layout but make one column slightly wider than the other.

I was mortified, frankly.  It was a problem I should’ve caught earlier, and now that we were rapidly approaching release dates, I had no solution. I made the only recommendation I could.

“They’ll have to make a change,”  I said, “but we can help them make it as painless as possible.  We can do the stylesheet revision work for them to make it as close as possible.  Their end users may not even notice.”

I was overruled.  We’d told them backwards-compatible UI layout, that’s what we’ll give them.  Figure out a hack to make it work.  Postpone the pain.

And here’s the consequences:

So we hacked a solution for the first customer.  Reduced font-sizes, sliced off a few pixels of whitespace here and there.  It worked – kind of.   But hacks aren’t good at solving for edge cases.  So the bugs started rolling in – in certain circumstances, the layout would break again.  And we’d fix that issue, but then the next customer would find a different way of breaking the layout.

I can’t count the number of hours devoted to variations on this one bug.  Customer service had to deal with it, design, professional services, engineering, QA – every fix touched multiple departments, and never fully resolved the issue.

For all I know, the same bug is still being hacked on, 3+ years later.

It would’ve been painful to tell customers that we had to go back on our promise and break backwards-compatibility.   But that single decision probably cost tens of thousands of dollars in productivity loss.  Hours and hours in opportunity costs.  And the customer goodwill we’d hoped to preserve was at least somewhat eroded by the fact that we kept breaking their layout, anyways.

Pain fades with time. (Anyone who has lived through a website redesign knows this: you’ve seen a ton of vitriolic customer feedback, followed by little to no actual effect on traffic, followed by people forgetting that it was ever any other way.)  But trying to avoid pain at all costs takes constant, ongoing effort.

“They have the fanciest in-house usability testing lab I’ve ever seen – one-way mirrors and video cameras, eye-tracking technology… and, come to think of it, their products have the worst user experience I’ve ever seen.”

Unfortunately, the damage probably won’t be limited to this unnamed company’s consumer products.  When “best practices” go bad, they create skeptics – product managers, engineers, and executives who conclude that those techniques are useless.

Persona analysis, usability testing, voice of the customer feedback, and surveys – these are great tools for learning about your customers.  Or rather, they can be.  If you don’t follow these examples…

Persona Analysis

By far the prettiest user personas I’ve ever seen were created by a major online portal.  They were full of detail – each persona had a full-color headshot, a name, and extensive detail about where they lived, what job they held, number of kids, where they shopped.

The problem came when I tried to use those personas to help drive product decisions.  I asked some questions:

• Four personas – what percentage of their user base did each one account for?
• Which one represented the majority of customers?
• How valuable are each of these user types?  Is one more likely to engage in revenue-generating behaviors?  More likely to be a loyal customer or recommend their product?

No one could answer these questions. This meant that there was no way to decide what should take precedence when the personas’ needs contradicted each other.  We couldn’t make intelligent decisions about which features to emphasize and which could be hidden behind an extra click.

As it turned out, the personas were created in complete isolation from the business stakeholders.  They looked great, they were detailed, and they were probably even accurate representations of online portal’s customers.  But they couldn’t be used to prioritize or make decisions, so the product managers ignored them – that is, until they were asked if they did persona analysis.  Then they answered “of course we do!”

The problem: Going through the motions without understanding the purpose of the exercise.

How to avoid this: Skip the pretty pictures and answer the “5 Hows” – how many, how profitable, how comfortable with technology, how concerned with convenience vs. quality, how they trade off between money and time.  Going more bare-bones with persona analysis increases your chances at getting all the stakeholders to read them and give the feedback you need to make them useful.

Usability Testing

I was sitting in the observation room behind the one-way mirror with the bank product managers and designer.  We watched the first, second, and then third user have trouble with a specific task, and hypothesized out loud that it was the inline text that was confusing them.  “Let’s quickly mock up another variation with different text,” I suggested.  “That way, we’ll learn whether it’s the text or something else that’s causing the problem.”

“We can’t,” said the product manager.  “We need to run the test the exact same way for all eight users, or else the report we produce won’t be scientific enough.”

It’s common practice to encourage test participants to “think out loud” as they navigate through the website or application.  This means sometimes they go off on tangents, offering opinions that weren’t directly asked for.  On this particular day, four of the eight testers independently made very similar comments about a frustration they had with the current version of the application.  I was scribbling notes madly – this was a great insight, and I could already envision a new feature that would solve this frustration and differentiate the product.

But those comments never made it into the usability testing report that was created for management.  Why? Because they weren’t part of the “official” testing script.  “People say all kinds of things,” said the usability test moderator dismissively.  “I don’t put anecdotal stuff into the reports.”

The problem: Confusing qualitative testing with quantitative data.   Second problem: Making testing so expensive and resource-intensive that you need a “scientific-looking” report to management in order to justify it.

How to avoid this: Set expectations appropriately that the point of usability testing is to learn as much as possible.  If changing questions mid-stream allows you to test out twice as many hypotheses, that’s a good thing.  And again, go bare-bones: when testing is cheaper and easier, you don’t need to spend as much effort justifying it.

“Voice of the Customer” Feedback

When is listening to your customer a bad thing?  When you:

• take feedback at face value without attempting to understand the context
• assume that the X customers asking for a feature are representative of your entire user base
• build features that contradict your product strategy “because X customers asked for it”
• don’t validate that these are legitimate customers

A major credit card provider demanded that we make a change to our product.  According to their voice of the customer data, it was a critical request and they felt we couldn’t be successful without it.

I asked for an export of their “voice of the customer” data so I could get a bit more detail.  As I read through the verbatim customer comments, I noticed that most of the customers requesting this change had never actually used our product. They had read the splash page, and based on that, assumed (incorrectly, I might add) that the product worked a certain way.

Making the requested change wouldn’t have satisfied these customers at all!  Instead, the company changed their splash page, and that resolved the issue.

The problem: Listening without understanding.

How to avoid this: Follow up and ask more questions.  Why is this a problem?  If they have a suggested solution, what do they think having this solution would help them to do?  What is the context?  Could this be a one-time issue or is it something systemic?   What are the alternatives?

Surveys

“How concerned are you, on a scale from 1-10, about privacy?”

“I’d say 9 – very concerned.”

“OK, thank you for your time.  In order for me to get you your check for participation, you’ll need to write down your name and Social Security number for me.”

“OK, no problem!”

99% of surveys will tell you that customers are very concerned about security, privacy, and getting the best deal.  That they want features A, B, C, D, and the kitchen sink.  That they floss daily and eat 5 servings of vegetables a day.   Then you watch as their buying and usage behavior completely contradicts what they just told you.

Most surveys are close to useless because they ask questions without addressing the necessary tradeoffs.  Do customers like the idea of more features? Of course!  Do they want to spend an extra hour learning how to use your product, or spend an extra $100 on it?  Of course not!

The problem: Asking questions in isolation without mentioning the accompanying benefits or tradeoffs.

How to avoid this: Choose freeform, not yes-no, questions when it comes to matters of security, privacy, health, finances.   People feel like they should answer “yes”, so they do and you get crummy data.   Balance features with tradeoffs (“would you spend $x more for [feature]?” or “would you want [feature] if it meant x more learning time?”) or give users a limited number of votes to “spend” on specific features so that they are forced to prioritize.

You don’t hire a professional just because you don’t have the hours to do some hammering and electrical wiring; you’re also hiring their experience and the knowledge they’ve built up.

A good contractor will recognize that your ideas about kitchen design will leave you tearing your hair out later when you realize there are no outlets to plug in the blender, and propose alternate suggestions.

The same thing applies to your customers.  They aren’t buying software – they’re buying problem solved.

I hear some of you saying, “yeah, but I’m not a consultant!  It’s not scalable to look at every customer’s unique situation.”  To that I say:

• Is there more than one way to use your product?
• Is your product delivered as a white-labeled or configurable service?
• Do your customers make decisions on how your product is used, discovered, or integrated into their existing websites or processes?

If you answered “yes” to any of these questions, you need a Best Practices guide for your customers.  Without one you are not providing a whole product solution because you’re not providing your experience and subject matter knowledge.    How else can you maximize their chances at success and satisfaction?

What belongs in a Best Practices guide?  Well, it’s going to depend on your product, but you may want to think about:

Installation / Integration. In an ideal world, how would a customer install or integrate your product?  If you yourself are not the installer/engineer/IT person, find the person responsible for this internally and ask them.

More often that not, you’ll hear something like “well, of course they’ll want to create a new database table for that”… something that would NOT be obvious to your customer. Especially if you’re selling into large companies, IT resources are limited.  Their engineer shouldn’t have to read through dozens of pages of installation manual to know that they will need the DBA’s help installing.

Configuration. What are the impacts of certain configuration options?  Will enabling feature X decrease performance?  Does disabling feature Y make the product far less usable?

A few years ago, I worked with a customer who wanted to disable a feature.  Users of our web application could either pay a bill one time, or opt into having that bill automatically paid.  The latter was more profitable to the customer, but caused many users to exit the registration funnel.  Because we did not clearly communicate the usability and usage consequences of disabling this feature, we had a lose-lose situation.

User Interface / User Experience. Are there text labels or headlines that draw more users’ attention?  Should/shouldn’t your web application open a new window?  Have certain visual designs resulted in better performance?  Show screenshots of customers who are seeing good results.

Demos, Help, or Splash Pages. What works in getting new users up to speed with your product?  If Flash demos make it seem dead-simple, recommend that your customers create one (or better yet, offer them a bare-bones generic on.)  If you’ve noticed higher registration completion when the splash interstitial says “free”, that’s information your customers should know before they start creating their branded splash interstitial.

Customer Service Responses / FAQs.  What questions can your customer’s service people expect to get?  How should they respond to certain questions?  You may not think it’s your job to create canned answers for someone else’s customer service rep, but no one else has the same combination of a) invested in the product’s success and b) deep product knowledge.

Expectations. If the customer takes all of your recommendations, what benefits can they expect?  BE HONEST.  You need to make it clear that following your recommendations are worth their while; you also want to set expectations appropriately.

This list isn’t comprehensive (it’s probably too web-application-centric to be that) – but hopefully it gives you somewhere to start on your document.  If in doubt, brainstorm.  What do our best customers look like? What do our bad customers look like?  and use that to identify your best (and worst) practices.

Note: if you’re lucky, you have “good” customers to reference.  I haven’t always been that lucky.  With one of my products, the first customer, the one that was supposed to be the Reference Customer, had a terrible deployment.  Through web analytics and raw input from their customer service department, I was able to identify what they’d done wrong.

(But I didn’t – and would not recommend – writing the guidelines as “Don’ts” instead of “Do’s”.  Instead, I took the opposite of what they did and codified those as guidelines.  When other customers asked, verbally I would admit “here’s what another unnamed customer did and the bad results we saw” but I don’t generally recommend putting negative recommedations in writing.)

There were over 20,000 competitors in Sunday’s Nike Women’s Marathon in San Francisco. And 24-year-old Arien O’Connell, a fifth-grade teacher from New York City, ran the fastest time of any of the women.

But she didn’t win.  (At Women’s Marathon, Fastest time didn’t win, SF Chronicle)

For those unfamiliar with road races, the “elite start” is fairly common to major races.  Road races get a lot of notoriety from attracting big talent.  Two of the carrots that race organizers dangle to get that talent are prize money, and a clear start free from having to weave around hundreds or thousands of slower runners in your path.

The gun goes off and a couple dozen really fast, whippet-thin runners sprint off; five to twenty minutes later a second gun goes off for the rest of the field.  Most runners aren’t even aware that an “elite” start exists.

Arien O’Connor didn’t consider herself an “elite” runner.  With a personal best in the low 3 hour range, there’s no reason why she should have.  (Romania’s Constantina Tomescu won in Beijing in 2:26; Great Britain’s Paula Radcliffe holds the world record of 2:15.)

But she had a hell of a race that day! She finished in 2:55, the fastest she had ever run.  And as she discovered when Nike started to announce the “winners”, by far the best time of the race.

But when she pointed this out to Nike race officials, they were hardly congratulatory.

“They were just flabbergasted,” O’Connell said. “I don’t think it ever crossed their minds.”

No one seemed exactly sure what to do. The trophies had already been handed out and the official results announced. Now organizers seem to be hoping it will all go away.

“At this point,” Nike media relations manager Tanya Lopez said Monday, “we’ve declared our winner.” (At Women’s Marathon, Fastest time didn’t win, SF Chronicle)

Nike made two mistakes.

The first mistake: They failed to anticipate what was not a particularly unlikely scenario: what if someone in the non-elite pack has a faster time than the elites?

The appropriate answer would not have necessarily benefitted O’Connell.  In fact, the U.S. Association of Track and Field wouldn’t even consider that O’Connell and the elites were running in the same race.  Running is an incredibly mental sport, and it’s entirely possible that the elite winner could’ve run faster, even eleven minutes faster, if she had seen O’Connell out there in front of her.

Nike’s proactive response could’ve been to make it very clear that these were two separate races and that prize money was only open to the elite pack – to basically print in block letters “If you think you can win, apply to classify yourself as elite.”  (Applying entails submitting finishing times from other races.)

Or they could’ve offered two sets of prizes – again, reinforcing that these are two separate races, each with its’ own set of winners.

But wait, remember what I said before about O’Connell not having any reason to consider herself elite?

The second mistake: They failed to question their own assumption: does it make sense for us to offer an elite start?

Nike overlooked a few critical facts:

Fact one: The Nike Women’s Marathon is held two weeks prior to the world-famous, highly prestigious New York Marathon.  The best runners in the world spend months training for New York.  Two weeks prior, they are tapering their running distance and resting in anticipation – not racing!

Fact two: the Nike Women’s Marathon is heavily associated with the charitable organization Team In Training.

Team In Training raises lots of money for disease research, which is great.  They also take thousands of otherwise sedentary people and train and empower them to complete a marathon, which is also great.  What it does not do, however, is to turn out a field of zippy, highly competitive runners.

It didn’t make sense for Nike to offer an elite start at all.  And they could’ve anticipated this before they even opened up the race for registrations.

Even after the registration were opened, they could have noticed “Hmmm, our elites aren’t that elite.  Does this still make sense?” and decided to eliminate the separate start.  But no one asked, or no one followed through.

Several major newspaper articles, thousands of messages on Runner’s World and the USATF forums, and a lot of brand damage later, Nike recanted and declared O’Connell a winner.   And next year, no elite field.

So it all turned out okay.  But it all could’ve been avoided with a simple willingness to ask questions and take a pre-mortem approach.

Let’s look at the elements that lead to the good cohesive experience:

Try it now for FREE. The use of the bright orange color and bold type immediately draws my eye.  “Free” is powerful because it lowers my resistance.  I don’t yet know what this site offers, but knowing that it doesn’t cost anything lowers the friction of me reading further.

I’m often surprised to find the “oh yeah, and it’s free” treated almost like a disclaimer – in tiny print or hidden at the bottom of a page.   Is this a fear of sounding like a used car salesman?  (FREE FREE FREE at Crazy Eddie’s!)  One thing I’ve heard product managers say is that they think the word “free” suggests a lack of value of their product.  In the offline world, this might be true; online, every millisecond that consumers don’t see the word free increases the odds of them closing the browser window never to return.

Marketing materials in minutes.   Quickly answers the “what is this?” in the consumer’s mind.  It conveys the value of the service without constraints of too much detail.  (The alliteration with the repeated m’s is a particularly nice touch.)

Everyone has an idea of what “marketing materials” is, and even if my version doesn’t match yours, we both know that typically this is a task that takes longer than “minutes”.

Big gorgeous picture (“show me what I’m going to get”). I’m not sure that the average first-time user is going to produce something this polished-looking “in minutes”, but this image immediately suggests to me that I’m going to be able to put together something far superior to the standard Office Depot templated materials.

There actually is more detail on the page (in the dark gray callouts along the side) but they do not visually compete with the other elements.  Someone skimming the page won’t be distracted by them, but the type of consumer who reads a whole page first before making the decision to click will have at least a few of their questions answered.

If you still need convincing, go watch the demo. Many sites spend a lot of time making the demo into a band-aid for a weak splash page and for serious usability flaws in the product.  (Too bad only 10% of your customers will ever see your demo and get even the benefit of that band-aid.)

So what’s the best practice here?  Provide a demo, but don’t make it the primary call to action.

Consumers have a limited attention span: do you want to “spend” their attention on watching a demo, or jumping right into signing up?  Remember, once you have their email address, there’s always a chance to communicate with them later.  If they watch the demo and leave, they’re out of reach.

On the BrandDoozie splash page, the “See Demo” button is below the fold.  The primary calls to action – jumping right in and getting started, or logging in for existing users – are above the fold, endorsing them as the “preferred” actions.

Existing users – we’re not wasting your time. The log in call to action is right at the top of the page, and easy to see without being distracting to new users.   Smart, because most consumers will bookmark your homepage, not a more useful internal page.

Here’s another thing they do right: the email confirmation.  I sign up for a ton of web services, and I don’t always remember when I get back to my inbox which one does what.  This site recognizes that, and reminds me:

Obviously, you can always measure and tweak to continuously improve.  One minor issue is that this super-helpful description in their confirmation email is only conveyed via images.  I have images off by default, and so does a considerable percentage of the viewing population.  But this in general is a great example of where to start.