Better Product Managers, and Product Management

Let’s get a few things about A/B testing out of the way up front:

• Changing one thing will affect other (often unexpected) things
• You’re going to screw up
• It’ll be OK, because you’re nimble and can quickly course-correct

Changing one thing will affect other (often unexpected) things

Making our KISSinsights product more robust had left us with a fairly long snippet of Javascript that we were asking our customers to add to their sites.  (Longer than this paragraph.)

I’d seen in previous startups that “long chunks of code” performed badly.

(Setup:

• showed the same ‘install this widget on your site’ page – variation A was 2 lines long and variation B was 6 lines long.  Otherwise identical.
• People rated A as “seems easy, I could do this right now”)

So one of our developers tweaked our code until it was a slim and non-intimidating 2 lines.  Looked easy!  Installation rates went up!

…and I started getting a bunch of support emails from developers.   As it turns out, the “long” version had one advantage – if you were a person who cared about these things, you could scan the code and see for yourself that it was asynchronous-loading and supported both secure and non-secure web pages.   Shortening the code made it more obscure, so you could no longer “see” asynchronous and https support on your own.

But numbers went up!  Yes – sometimes information is hidden in averages. We were making life a lot easier for non-technical folks — the 90% of our customers who don’t care or even know what “asynchronous” means.   But our support inbox showed that we had, inadvertently, made life harder for that other 10%.

So we kept the shorter code.  But we added information to that page’s sidebar so that people scanning for the keywords “asynch” or “https” could easily get the information they needed.

You’re going to screw up

We had a bunch of feedback from less-technical users who were not sure they had installed their code correctly.  This made us suspect that there were a lot more silent users who would just decide not to risk installing us at all.  Fine, we thought, we’ll give them a button and that will give them the confidence and reassurance needed.

And, in true lean startup fashion, we built an MVP solution.  It didn’t work for every single case, but it worked most of the time.

Can you predict where this is going?  Yup.  Even when people were clearly using the product already — i.e. of course it was working, they would click on that “verify code” button.   So we created a big group of people who got a false negative and immediately panicked that they had done something wrong.  Never mind that they were able to use the product – the presence of that “verify failed” message was so alarming that they were worried that something was broken and there would be consequences later.

We solved (or started to solve) the problem of “customers can’t verify their installation”.  But that wasn’t the problem — the problem was “customers don’t feel confident enough in their technical abilities”, and we inadvertently made that worse.  Feature pulled.  Problem re-stated.  New solution attempt coming soon.

It’ll be OK, because you’re nimble and can quickly course-correct

6 months into the release of KISSinsights, I found myself answering a different breed of question.  Previously, our survey responses and support inbox had been centered around tinkering with the product and bug reports (pretty typical early-adopter communications) — but now, people were asking about premium features, wanting to see screenshots and customer testimonials, wanting to present to their bosses why they should subscribe to premium.  Welcome to the beginnings of mainstream (or mainerstream) adoption!

“We need to add this information to the website, now,” I said.

So we did.

And our conversion rate tanked.  Within days of adding the new pages, we saw our conversion rate drop by over 50%.

The homepage had a prominent “SIGN UP” call to action.  The new pages did not.  I’d been so eager to add that needed new content that I didn’t think about how to continue funneling those people to sign up.  So, basically, almost everyone who clicked off our homepage failed to convert. Oops.

This leads me a related point: statistical significance is highly overrated.  When you can tell an experiment is failing, pull the plug fast.  You do not need to subject your “B” audience to failure for an extra 3 days just so your Excel spreadsheet looks better.

We commented out the links.  Then we added call-to-action buttons to each new page, and only then did we add them back and start a new A/B test.  It took less than a week to recover to our previous conversion rate, and since then we’ve been moving in the right direction.

Popularity: 6% [?]

Talking with your target market is critical to finding product-market fit. (In case anyone thought my earlier post Anybody, As Long As It’s Not You implied otherwise).

But I’ve heard “target market” thrown around far too often around as an excuse for why other people didn’t understand your idea.  (“Well, of course he didn’t understand my online raccoon manicure product – he’s not a raccoon groomer!”)

But in general, ANYBODY SHOULD BE ABLE TO ‘GET’ YOUR IDEA.   If they don’t, it’s either a bad idea or you’re expressing it badly, and the sooner you realize that, the better.

Don’t ask “would you use it?”, ask “do you get it?”

As SOON as you have an idea or a very early mockup, show it to people.  Doesn’t matter who.  If they ‘get it’, THEN invest the time in building out more of an MVP, and at that point, definitely call in your target market.

Recently I was talking with some other entrepreneurs about a company that prints out physical wall calendars already-marked with all the birthdays from your Facebook account.  We all said, “I probably wouldn’t buy it” but we all UNDERSTOOD WHAT IT WAS, and that there was an audience (my teenage niece) who would probably find it a lot cooler than we did.

Start Fast and Cheap, then Target

Finding target market people takes time.  I can find a fellow product manager/startup person to give me feedback within hours.

It might take a day to find a targeted person – for a new entrepreneur who doesn’t have a big network yet, it might take a week.  That’s too long to wait for this very early feedback.

Asking target market people for their feedback burns some social capital.   I can easily ask a favor from a fellow product manager/startup person because I know they could probably use my help at some point in the near future.  I also know they’ll be both patient and merciless if I do a bad job explaining my idea.

If I have a target market person who is NOT my friend, and ask them for feedback, it’s a lot harder to ask again later.  If I do a bad job explaining my idea, they may incorrectly conclude that it doesn’t meet their needs.  Or it may just not trigger the feedback you need — most people are not able to articulate their needs straight-out.  It’s only when they hear an idea or see a sketch that they get in the right mindset to talk about their problems (you know, those things that you are trying to solve.)

So once your idea has gotten some rudimentary validation, target away!   Just don’t let targeting stand in the way of getting feedback EARLY EARLY EARLY.  Let dumb/impractical/incomprehensible/poorly articulated product ideas die a quick and painless death!

Popularity: 1% [?]

Question: Who should give you feedback on your early-stage product mockups/demo?

a) Your investors

b) Your most loyal, early-adopter existing customers

c) A demographically-balanced segment of people who have no familiarity with your product

d) Anybody, as long as it’s not you

OK, sure, there’s probably an “ideal” audience to show your product to.  But it probably doesn’t matter.

Earlier this week, I emailed a friend with a URL to ask for his feedback.   To be honest, he was far from my “ideal” user testing subject: an expert user, tech- and product-management-savvy, an existing user of previous versions of the product, super-familiar with our company and what we’re trying to do.  The kind of person who’d probably agree with me on what websites rock and what apps are lousy.

How useful could his feedback possibly be?

(I think you know where this is going.)

He was kind enough to send me a Screentoaster video of him navigating through the flows and “thinking out loud” — and from the first glance he started pointing out problems we hadn’t noticed.  Wording that was confusing.  A call-to-action button below the fold.  An interactive panel that we’d thought was awesome – that wasn’t nearly clear enough.  And on and on and on…

Even an “expert” noticed these problems – because he wasn’t me.

Even as someone who’s done tons of critiqueing UIs and workflows, I just hadn’t succeeding in stepping back and seeing our product concept through the eyes of a customer.

So — don’t wait until you have a full product, don’t wait until you’ve done another round of design, don’t wait until you’ve revised your copy.  Get someone to take a look at whatever you have NOW.  Doesn’t matter who – as long as it’s not you.

PS – hey readers, if you’ve got a few moments and can record a Screentoaster video of yourself walking through these flows, email me.  I’ll be happy to return the favor for you and your product!

Popularity: 2% [?]