Better Product Managers, and Product Management

When should you seek quantitative vs. qualitative feedback?

I’ve seen a lot of bias towards looking for quantitative data as early as possible – and in my opinion, it’s often too early.

Here’s my rule of thumb:

• If you’re asking what, why, or how do you questions, you want qualitative data.
• If you’re asking which or how many/how often questions, you want quantitative data.

Here’s the the tricky part:  making sure your “which” or “how many/how often” questions aren’t actually “what/why/how” questions in disguise.  I see these two examples a lot:

• Which of these features do you find most valuable? Sounds like a “which” question – but if you’re just guessing at the choices, it’s probably a “what” question — What do you find most valuable about our product?
• How often are users logging in? Do you really need to know the frequency, or are you using this as a proxy measurement of user engagement?  If so, you should turn it into a “how” question — “How do you use our service?” or “How valuable do you find our service?”

It’s tempting to try and turn as many questions as possible into quantitative ones, because those are easier to measure through unobtrusive means – web analytics, looking for data patterns, surveys – and give clear numerical answers.

However, it’s the messy, hard-to-collect and harder-to-interpret data that will lead you to breakthrough insights – not the percentiles and bar graphs.  Talk to people!

We recently opened up beta for our new KISSinsights product.  Yay!

Like many small startups, we don’t have a dedicated QA tester, and we don’t have fully automated testing practices in place.  Here’s some of what we did well (and not so well) to get ready for release beyond ourselves.

Browser Support

1. Pick which browsers you will support and write it down. This sounds so obvious that you probably won’t do it, and then you’ll find out later that someone spent four hours testing on Chrome even though you weren’t planning on supporting it yet.   For now, we’re supporting Win/IE8, Win/IE7, Win/Firefox, Mac/Firefox, Mac/Safari.
2. Do browser detection and show a message to people using other browsers. If you’re in early beta, you can get away with supporting fewer browsers, but you really do need to set expectations.
3. Divide up browser testing. If you don’t explicitly say “you test Win/IE7 and I’ll test Win/IE8, etc.” you will end up duplicating efforts and missing things.

Emails

1. Make sure you have a huge source of available email addresses to test signups. This time, I just set up my personal domain so that [any username] at [domain] would forward to me.  Another way is to pre-create a bunch of testing1, testing2, testing3 accounts at the company domain.
2. Have every kind of inbox available - GMail, Mail.app, Thunderbird, Outlook, iPhone, etc.  We didn’t do this and so I still haven’t previewed the emails we send in different environments.   Set this up in advance!
3. Have some third-party friends who can review. After the second or third time you check the emails your app sends, you will stop noticing things.

Bugs

Define bug reporting best practices.  Our developers did this about halfway through (after being annoyed by vague bugs).

For workflow bugs, we are now trying to follow the example below:

Flow:

1. User tries to activate a survey.
2. user sees error
3. checkbox is checked until page reload

Expected behavior
Checkbox becomes unchecked if the activation fails.

How to reproduce

1. have two or more surveys
2. set the url to be the same on both surveys (at least one must be deactivated)

For UI bugs, we are trying to follow:

1. Provide a screen shot.
2. Find out what browser/OS.
3. If it came from another user, try to replicate it yourself and tell us if you are able to.
4. Provide a link to where the issue was found
5. If at all possible: view source and look for anything weird

It would’ve saved a lot of time if we’d established these examples/guidelines up-front, but hopefully by writing this, you can and will.

OK – you did your customer development homework.  You found a market who knew they had a problem and were willing to invest time and/or money in solving it.  You built a product that aligned with your customers’ pains and priorities –

…but somehow, people still don’t seem to be very excited.  What went wrong?

MVP without FUE = DOA.

(That’s “minimum viable product without first user experience = dead on arrival”.)

When you were working with early customers, it wasn’t that important to flesh out the first user experience.  After all, they were right there with you, listening to you articulate the value proposition and how your solution would help them with their problem.  Those customers had the benefit of your insights, explained through multiple interactions.  They share in the curse of knowledge.

But your new customers don’t.

After hundreds of customer development interviews, user testing sessions, and survey responses, I’m going to share with you the single most common barrier to adoption I’ve heard: “I don’t know where to start.”

Consumer vs. enterprise, techies vs. non-techies, it doesn’t matter. Whether it’s organizing their personal finances, engaging in customer research, eating healthier, starting a blog,  understanding their web analytics, or organizing their baby photos — this is what blocks people.

The Customer Excitement Lifecycle

People go through a pretty predictable cycle of emotions when they try to solve their pains.

1) In pain: “I really need to start eating healthier – I feel terrible.”

2) Glimmer of hope: If you’ve done a good job articulating your value proposition, the customer sees your product and feels hope: “Can this product can help me to eat healthier?  That would be so great…”

3) Leap of faith: Honestly, they’re not reading your ‘about’ page or watching your demo video – they’re just eager to get started fixing their problem, so they click ’sign up’: “This is the first step towards healthy eating!”

4) Expectations crash: There’s no immediate gratification – lots of links and features, but no guidance on where they should focus their attention.  Realization hits: “Hmm… is this really going to help me? This seems like a lot of work, not sure if I have the time and energy to get started…”

If you neglect the experience between steps 3 and 4, your product will wither and die. Yes, you should build a minimum viable product.  And your minimum viable product needs to include a compelling first user experience.

Most of us are only ’selling’ our customers once when we need to be selling them twice.  The first ’sell’ is convincing them to click “sign up” and give you that initial try – and most customer development practitioners are doing that pretty well.

The second ’sell’ is convincing the customer to continue investing time and thought into your product.   If they log in to their dashboard and all they see is “You have no new [whatever]” and a “View demo” link, they will leave and not come back.

Channel That Natural Enthusiasm

When your customers first sign up for your product, they are excited! They are motivated!  They have already decided to invest some time and energy in you – don’t waste it.

The best way to manage expectations crash is to channel the customer’s natural enthusiasm into an immediately productive activity.   Give them a task that they can complete quickly, that brings them closer to getting value from your solution.

A few sites that do this well:

Daily Burn asks you to set up fitness goals immediately, reinforcing the reason why you signed up.

Lil Grams gives you 3 things to do and 3 things to learn - enough to get you started without overwhelming.

Dropbox encourages you to try features you might not have noticed otherwise (and increases word of mouth behaviors at the same time).

“OK,” you say, “I’m convinced – I need to talk with potential customers to make sure my startup/product/service idea has potential.  But how do I find those people?”

Finding People

AdWords / Facebook Ads / Tweets.

Summarize your idea, invest some money in getting it in front of people who have expressed intent by searching for that term, clicking your ad, clicking a link.  (Read the original SEM on $5/day post for details.)

I haven’t used Twitter for this much yet, but my theory is that it may be a more effective way of reaching people (I am much more likely to click on links that appear in my hashtag saved searches than I am to click on an AdWords or Facebook ad.)

Twitter Search.

Look for people who have already discussed a similar product, problem, or solution and address a tweet directly to them:

@username Would love yr feedback on [product/problem/solution] – shd only take 2mins [URL] thanks!

Some people will ignore this, but many more will feel a bit flattered that they’re being asked.  Use this judiciously – more than one or two of these tweets per day and you’ll look like a spammer.

Google Alerts.

Set up Google Alerts for your product/problem/solution (you should have done this already anyways) – and when it finds relevant blog posts or comments, email those people and ask for their feedback:

I read your [post/comment] about [product/problem/solution].  I’m currently trying to validate a related idea and I think your opinion would be very valuable to me – could you take 2 minutes and check out [URL]?  Thank you – I’d be happy to return the favor any time.

Ask for introductions.

People are generally happy to make introductions for you, provided you do 3 things:

• Provide the exact text that they can copy and paste into a tweet or email (They’re doing you a favor! Make it as easy as possible for them.)
• Tell them exactly how you are going to communicate with their contacts (They’re risking a bit of social capital for you – if you are a jerk to their contacts, that will reflect badly on them.  Be very clear that you won’t spam or annoy people.)
• Tell them your goals (What do you think you’ll get/learn if they make this intro for you? People want to know that they’re contributing to a bigger picture!)

Email Request Template

I have a quick favor to ask.

I’ve got a product idea that I’m trying to validate with [type of customer]. My goal is for them to visit my splash page at [URL] and indicate their interest (or lack thereof).  I will only contact them if they explicitly give me permission to do so.

Could you send this message along to people you know who fit this target?  (Feel free to change it a bit if you like):

[Message - be sure to include the goal, the URL, and your contact information]

Twitter Request Template

I have a quick favor to ask.

I’ve got a product idea that I’m trying to validate with [type of customer]. My goal is for them to visit my splash page at [URL] and indicate their interest (or lack thereof).  I will only contact them if they explicitly give me permission to do so.

Since you have a number of followers who are the type of customer I’m trying to reach, could you tweet this for me? (Feel free to change a bit if you like):

[Message - include the URL, the topic, and keep it under 115 chars so it can be easily retweeted]

Asking for the Interview

You may be wondering, “so what is this URL I’m sending people to?  Can’t I just have people email me?”

You have three main goals with your splash page:

1. Communicate your idea in 10 seconds or less (seriously, that’s about how much time you have to grab someone’s attention)
2. Offer something interesting to the people who visit
3. Get contact information so you can ask for the interview

#1 is up to you (there’s a whole other blog post I could write about that…).

#2… when I say “offer something”, people generally think that means a tangible incentive.  You can, but you probably don’t need to – people like being asked for their expert opinion, they like the feeling that they’re contributing to something, and they like being part of a select group who gets a sneak preview at something.

You can cover #2 and #3 with a well-written survey template.  You can see the one we’ve used for the Survey.io beta, or here’s a partial screenshot:

FYI – I’ve used surveys with these 2 questions for multiple products, and so far, overall less than 20% of ALL respondents to this survey leave these blank – the vast majority choose at least one way in which they’d like to give feedback, AND give an email address.

Coming next week:  I’ve got email addresses – now what?