Better Product Managers, and Product Management

“I really should do user testing, but…”

You know that early validation can save weeks of working down the wrong path, right?  You may have listened to a few tangential comments from users that illuminated a whole new path to differentiation.  You’ve probably seen an interface that was completely intuitive to everyone in your company … and completely baffling to everyone outside it.

But if you’re like most product managers and entrepreneurs, you’re not testing.

First of all, testing has the sense of a big, lofty thing.

We all remember creating science fair projects years ago – you needed a formal hypothesis, a control group and an experimental group, all the variables had to be controlled, you needed to take notes, and the whole thing culminated in a typed, double-spaced report with graphs and charts.  (If you’ve worked with User Research within a large enterprise company, you still see research presentations just like this – except in PowerPoint instead of a tri-fold posterboard.)

Axe it. Forget it. I officially absolve you of needing to be super-scientific and organized.  If anyone asks, you can say “Cindy said this was okay,” and send them to me. Some data is better than no data.

Let me repeat that:

SOME DATA IS BETTER THAN NO DATA.

SOME DATA IS BETTER THAN NO DATA.

SOME DATA IS BETTER THAN NO DATA.

Are we okay now? Good. Let’s keep going.

(more…)

A couple weeks ago, I wanted to make some visual changes to this blog to see if I improve upon some basic engagement metrics – time spent on site, number of entries read, repeat visits, etc.  And the counsel that I frequently give to others immediately intoned:

THOU SHALT NOT MAKE CHANGES WITHOUT A/B TESTING.

OK. But this isn’t as straightforward as it sounds.   Google Web Optimizer will not help you test your WordPress blog.

• GWO is focused on single-action conversions, such as completing a signup or purchase.  I wanted to compare a series of metrics.
• It’s also not ideal for a database-driven site that uses common headers and footers.
• Around 25% of my traffic hits my homepage directly, but WordPress doesn’t have a simple way to create two homepages.
• I tweet a lot of single-article links, which means that every page needed an alternate.  But I wanted to avoid being penalized by Google for having duplicate content.
• I realized there was a lot of value in permanently having an “A” and “B” site so I could always be testing something.

So I decided to roll my own solution. It’s not particularly elegant.  But I’ve laid it out step-by-step so that anyone else with a self-hosted WordPress blog can try this themselves.

(And maybe, someone with better programming chops than myself will write a more streamlined plug-in version and make it available to the blogging community.)

(more…)

Exactly how you measure your audience breakdown will depend on what type of software, physical product, web app or service you have.  But fundamentally, all audiences go through some sort of decision funnel:

100% of your audience shows up.

Some % shows some sign of interest.

Some % commits an investment of time, personal information, and/or money into your product or service.

Looking at the numbers in between will prompt a lot of questions that you may not know the answer to, but should.   I can’t tell you what you’ll learn, but I can walk through some of the questions that your numbers should drive you to ask.

For ease of discussion, we’ll assume we’re looking at an online product, say, a sock-matching service.  (Same general rules apply for physical products, but data collection is more complicated.)

• Homepage: 20,000 views
• Register:  2,500 views
• Confirm registration: 600 views

20,000 views to the homepage

I hear a lot of product managers or marketers brag about page views. There are only two times when this is relevant:

1) It’s a really, really, really, really big number AND you’ve sold out ad inventory AND your overhead expenses are really low.  (Congratulations, you’re hotornot.com or icanhascheezburger.com, and you’re raking in the $.)

2) It’s not a very big number AND you spent tons of $ on ad buys or direct mailers hoping to get more people in the door.  (Sorry, you spent that marketing budget poorly.)

EVERY OTHER TIME, pageviews numbers is just a denominator.

(more…)