Better Product Managers, and Product Management

Do you know who your customers are?

I mean, literally, can you call up a list of email addresses in less than 2 minutes, without having to bug another person?

If not, this should be the next thing you build.  I’m serious — postpone core product development, postpone customer-requested features, postpone non-major bug fixes — and build an internal tool, stat.

KISSmetrics is (sadly) the first company I’ve ever worked for where I’ve had a customer search tool.  Somehow I survived without it.  But this is like saying that people used to survive dentistry without anesthesia — why would you inflict that upon yourself once there’s another option?

These are some of the things I was easily able to do recently that would’ve been major efforts in the past:

• Find out who recently signed up for our product and send out some random check-in emails.
• Find out when the customer who just emailed signed up, to be sure and strike the right tone in my reply.
• Filter to find the appropriate customers (based on integration, signup date) for my survey.
• Search for that person who complained on Twitter – are they a long-time customer or a brand-new signup?
• Search for “good brand name” companies so I could provide a variety of examples of how customers you’ve heard of are using us.
• Look up customers who logged in within a couple days of releasing a new feature so I have a list of people to follow up with individually.
• Look up customers who “got stuck” at a certain point in onboarding and email them to offer assistance / find out why they did not continue.

These may sound like simple things, but they allow me to do a lot of very quick research and to run a lot of tiny experiments (send an email to a few people, realize you aren’t getting the type of responses you wanted, revise and send to a few more, etc.).

I’m sure there are more feature-rich solutions that you can buy, but I’m not sure those actually serve the same purpose.  As an internal tool, this can (and should) be quick-and-dirty.  I have basic filtering and search capabilities; for everything else I export via CSV and futz with the data in Excel.  I don’t have a built-in record of every communication with a user (but since I send emails from Gmail, I can quickly make sure I haven’t bothered that person too much recently by searching my sent-mail).

If you want to build one of these, here are some requirements you may wish to consider:

• Free-form search with autocomplete
• Filter by plan level
• Filter by integration level (i.e. signed up / started configuration / fully getting value from product)
• Filter by recent signup (i.e. people who signed up in last 2/last 7 days)
• Filter by login (i.e. people who logged in past 7 days, 8-30 days, 30+ days)
• Filter by activity (i.e. trial period / active / canceled)
• Export to CSV

Popularity: 8% [?]

Does paying for feedback taint your results?

Some people say “yes” for several reasons — it biases the person to like you/want to give you favorable responses, it attracts non-customers who just want the reward, it makes it more like a job so the person is motivated to reduce the amount of time they spend with you.

In my experience, it depends on a few factors:

• What method you’re using to collect feedback
• What type of compensation you’re offering
• How much they know about your product

I’ll go through these one by one.

Paying for Surveys is Risky

Surveys, especially when you ask people to rate the benefits of something, or ask how likely they would be to use it, are extremely subject to rewards tainting.   People, either consciously or subconsciously, want to feel like they’ve “earned” their reward, so they rate everything more positively.

The other problem with surveys is their anonymity.  You’d think that would encourage people to be more honest, but in my experience the lack of a human face to the survey makes people more likely to lie, just pick the first option regardless of how well it fits, or skip questions.

Not offering compensation for filling out a survey will lower your response rate. But the people who care enough to fill it out anyways are likely to be your best customers (or prospective customers) anyways.  They’re investing their time to answer your questions because they care about your product.  10 people who care are infinitely more valuable than 100 people who are ambivalent.

Your Money Is No Good Here

I only recommend offering cash money in one situation:  if you are conducting face to face usability testing sessions and make the participants come to your office.   In that case, compensation is less about paying for their feedback and more about recognizing that they already had to pull out their wallet (to pay for gas/transit/parking) and likely skipped their own lunch hour to come to your session.

Otherwise, money attracts people who need money.  Sometimes these are your customers; usually they are not.   Because they know that this is a one-time deal, they have zero motivation to “do a good job” — that is, to answer questions honestly and offer thoughtful feedback.

My preferred forms of compensation are:

• I’ll follow up with a summary of what I’ve learned from everyone (if your interviewees care about this subject, this is super-rewarding)
• Free use of my product for X period of time

These are rewarding to precisely the right people: people who care about your product/niche.

Of course, sometimes compensation is necessary because you have nothing to offer people (i.e. you don’t have a product).  I strongly believe that if you run a good interview or testing session and really listen to your interviewee, they will get value. They’ll get to be listened to and get to sound smart and have some fun.  But they don’t know that upfront (and it would arrogant for you to tell them that.)  In that case I recommend Amazon gift cards.   You can deliver them via email and add a thank you message; very simple.

Leading the Witness

Especially in early-stage customer development interviews, I see people inadvertently “leading the witness”.  That is, they say upfront that they’re building a social network for gardeners instead of asking about the interviewee’s gardening behavior.  They mention online bill pay before they ask how the interviewee goes through the process of paying their bills.

This is actually a problem with or without compensation. Because now, your interviewee will tailor their feedback to what they think would be useful for you know in the context of your product.  But they are not the product manager.  We all know that customers are not good at articulating what they want, but now you have put them in the position to attempt just that.

The less you can reveal about your product, the truer your feedback will be.  OK, so that’s easy for an interview, but what about for user testing a product or dry-testing a splash page?  Same thing.

Bias is actually less of a problem here.  Yes, the interviewee will try to please you.  It’s your job to set the tone that what would really please you is to know in great detail how they behave, what frustrates them, what their needs are, etc.

You will also need to watch out for signs of them tailoring their behavior to what they think your product is.   If you hear things like  “I could see myself doing…” or “That seems like it would be useful…” or “Maybe I could use…”  — beware.  Those are projections, not real behavior.  But I’ve generally had pretty good success in steering people away from those with a reminder that I’d really like to hear about what they’re doing now.

So the answer, like most answers in product development, is “it depends”.  But hopefully these guidelines will you evaluate your feedback plan and decide when and whether it makes sense to offer compensation.

Popularity: 6% [?]

“Oh, no, let me explain how that works–”:  Strike 1!

“Actually, that text isn’t very clear.  What it should say is–”: Strike 2!

“The way we intended you to use that was–”: Strike 3!

If you have already had to explain away some part of your UI three times, you need to stop right there and fix it.

I’m not talking about perfectionism or premature optimization or working on details because it’s easier than tackling your big hairy problems. I’m saying, you have already crossed the threshold where you are spending more time compensating for a bad UI element than it would take to just fix the damn thing, already.

Stopping to fix small things is frustrating.  You know you’re taking time away from your core features, your big hairy problems, your differentiators.  You’re pulling yourself out of “flow”.  You’re asking your engineers to spend time on some tedious task that isn’t even a blocker per se.

Or is it?

Yes, your confusing UI that caused 3 people to email you probably caused 15 more to just silently abandon you.  But I talk about customers all the time.  Forget the customer for a moment!  Think about the impact of bad UI on your internal team.

Three bad things happen when you let bad UI go untreated:

• You start subconsciously thinking that your customers are idiots
• You focus more on what I want to build than on solving a painful problem for customers
• You put your hopes on big, risky, “perfect” fixes instead of thinking “how can we make this better NOW?”

Here’s a little example

Sometimes we get these pesky 500 errors on our website.  At some point, we took the time to make them at least look pretty:

But I kept emailing people who were concerned about whether the error on our website affected the surveys we showed on their site.  (It did not.)

The ideal fix would be to never show a 500 error, of course.  But that isn’t a fix I could make right now.

It took less than 5 minutes to write better error text and have a developer push it out:

I have a list of about 10 more similarly-sized fixes that I need to make this week.   It’s likely that none of them will single-handedly increase revenues, or turn a hater into a fan.  But if getting them done makes us 5% more likely to stop and say, “how can we make this better now?” about the next crop of little issues, it’ll be worth it.

Popularity: 3% [?]

…It’s better than being a time waster.

I’m going to pull a number completely out of you-know-where and estimate that a full 20% of time in software companies is wasted on completing tasks to avoid the appearance of flakiness.

Product managers are pretty guilty of this, but they’re by no means the only offenders (and when product managers do this, it’s often because of a corporate culture that encourages it, such as bonuses being dependent on the % of your features that are completed and released.)

Here’s what happens:

• a need is identified – a new feature, a bug report, an inefficiency
• requirements are written and agreed upon
• work begins

– but then something happens.

Maybe further customer interviews reveal that the new feature isn’t really that valuable after all.  Maybe watching your usage metrics shows that the bug – as much as it may irk you – is not affecting your customers’ use of the product.  Maybe that improved database query really isn’t needed at current usage levels.

Whatever the reason, the conclusion should be clear: we should stop working on this.

But we don’t.

“We should just finish it – It’s bad for engineers’ morale to have a half-finished feature sitting around.”  No — it’s worse for morale to have a fully-finished feature released but still just sitting around (and probably cluttering up the UI and generating more customer support costs.)

“We may as well do it now even though we don’t need it yet.” No — when you DO need it, it’s almost certain that your needs will have changed and this thing you built months ago will need more hacking to shape it into a still barely workable solution.

“We shouldn’t interrupt development ‘flow’ to start on something totally different.”  No — ‘flow’ is not valuable by itself!  There’s no sense in building the wrong thing fast.

These all boil down to: I thought this was important -> If I admit that it isn’t, I will lose credibility and look like a flake.

And don’t do the weaselly version, either, where you silently lower the priority on tasks and hope people will forget about them.

Make an announcement: “We are no longer working on X.  Here’s why: we thought it was going to solve Problem Y, but [it actually won't/problem Y isn't actually much of a problem/it will, but problem Z came out of nowhere and we've got to tackle that.]  I’m sorry that we already invested time in this, but I’d rather change course than have us waste more time on something that is not needed.”

I say, flake proudly.  Don’t keep working on dumb stuff.

Popularity: 7% [?]