You Shouldn’t Use a Survey If…

Surveys are an incredibly useful market and customer research tool.   But you use them too often. (Not, you know, you personally.  But ‘you’ in a global companies and organizations kind of sense.)

You shouldn’t use a survey if:

You aren’t sure which type of people you should ask to take your survey.

There are almost zero contexts where you’ll get useful data out of having “anyone” take your survey.   (And thinking, “I’ll just get thousands of responses, and then filter by some criteria later” is both a copout and unlikely.)   Do you want to hear from existing customers?  People from a certain region? People who share a common activity?  People with a specific job title?

Your “type of people” can be subjective, too — in a recent survey I conducted, I wanted to hear from was “smart people that we’d want to hire if we had the chance”.  So I distributed the survey through coworkers’ personal networks, letting them make that subjective determination.

You know which type of people you want to take your survey but have no idea how to find them.

Surveys are not an “if you build it, they will come” exercise.  Don’t waste your time on a survey if you don’t have a ready bank of people to send it to or a distribution strategy.   It’s a waste of time and social capital to send out a survey and get only 4 or 5 responses.

You’re better off conducting freeform interviews first so you can increase the “learning density” you get from each person.   For the long term, you’ll need to invest time in figuring out how to find more people.  This usually means “find where these people already are, and put yourself there” – participate in forums, join clubs, build up your network.

All of the questions you want to ask require a freeform response.

This is a sign that you don’t really know what the questions are yet, that you aren’t really sure what the problem is yet.

If you have an existing product or customer base, look at usage patterns and past feedback first.  Then start with freeform interviews, so you can tease out information in a back-and-forth context.  The survey format is bad for this type of learning.   People don’t like to write a lot, and even if they do, their first response is usually not where all the ‘meat’ is.

You don’t have a clear plan of action for how you’re going to use the results.

If you’re thinking “we might learn something” or “it would be nice to know…”, then save yourself the time (and save the time of the people who would fill out your survey).   Data without action is meaningless.

Is this information going to help you make a better decision on a specific feature or project?  Will it help you choose better wording or smarter defaults in your application?  Will it validate a specific hypothesis so that you can continue or pivot?

You can’t prioritize your list of questions down to fewer than 10.

This is another sign that ‘you aren’t sure what the problem is yet’.   Of course you have more than 10 things that you’d like to know — you probably have hundreds of things you’d like to learn — but you can’t possibly act on more than 10 at a time anyways.

I often see long surveys where the first 3-4 questions are asking about age, gender, zip code, income level.  Those are lazy questions.  If you need to segment by demographic, that should be part of your distribution plan or you should “buy an audience” from a market research firm.

You aren’t willing to write a draft, have another set of eyes review it, and then revise.

The first survey draft you write will suffer from at least one of these flaws:

  • Unclear language (“wait, what are they asking?”)
  • Biased language (“how much do you like feature X?”)
  • Stilted language (technically makes sense but just sounds awkward)
  • Stupid question (“do you want X?”  of COURSE they will say ‘yes’ – that doesn’t reveal anything about actual intent)
  • Too many freeform questions (no more than 1 freeform for every 3 click-to-answer)
  • Mistakenly using single-choice instead of multiple-choice, or vice-versa
  • Doesn’t actually ask the question that you wanted answered
  • Inadvertent rudeness (use of words with negative connotations, or a phrasing that sounds brusque or judgemental)

It is very difficult for you to catch these on your own.  I’ve been writing surveys for years, and I still always have at least one other person read it and comment on anything that is weird or confusing, and I still always have to change at least one thing.

6 thoughts on “You Shouldn’t Use a Survey If…

  1. Great post, Cindy!  So true re: needing to have others look at a survey for issues that are hard to see on your own…and great tips for when to not survey…sometimes interviews are much more appropriate, especially in the ‘fuzzy front end’!

  2. Cindy, I would like to talk to you about your article about luck in Women’s Day magazine because I will be quoting from it in my blog. Could you please contact me?
    Marla Madison

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>