You Shouldn’t Use a Survey If…

Surveys are an incredibly useful market and customer research tool.   But you use them too often. (Not, you know, you personally.  But ‘you’ in a global companies and organizations kind of sense.)

You shouldn’t use a survey if:

You aren’t sure which type of people you should ask to take your survey.

There are almost zero contexts where you’ll get useful data out of having “anyone” take your survey.   (And thinking, “I’ll just get thousands of responses, and then filter by some criteria later” is both a copout and unlikely.)   Do you want to hear from existing customers?  People from a certain region? People who share a common activity?  People with a specific job title?

Your “type of people” can be subjective, too — in a recent survey I conducted, I wanted to hear from was “smart people that we’d want to hire if we had the chance”.  So I distributed the survey through coworkers’ personal networks, letting them make that subjective determination.

You know which type of people you want to take your survey but have no idea how to find them.

Surveys are not an “if you build it, they will come” exercise.  Don’t waste your time on a survey if you don’t have a ready bank of people to send it to or a distribution strategy.   It’s a waste of time and social capital to send out a survey and get only 4 or 5 responses.

You’re better off conducting freeform interviews first so you can increase the “learning density” you get from each person.   For the long term, you’ll need to invest time in figuring out how to find more people.  This usually means “find where these people already are, and put yourself there” – participate in forums, join clubs, build up your network.

All of the questions you want to ask require a freeform response.

This is a sign that you don’t really know what the questions are yet, that you aren’t really sure what the problem is yet.

If you have an existing product or customer base, look at usage patterns and past feedback first.  Then start with freeform interviews, so you can tease out information in a back-and-forth context.  The survey format is bad for this type of learning.   People don’t like to write a lot, and even if they do, their first response is usually not where all the ‘meat’ is.

You don’t have a clear plan of action for how you’re going to use the results.

If you’re thinking “we might learn something” or “it would be nice to know…”, then save yourself the time (and save the time of the people who would fill out your survey).   Data without action is meaningless.

Is this information going to help you make a better decision on a specific feature or project?  Will it help you choose better wording or smarter defaults in your application?  Will it validate a specific hypothesis so that you can continue or pivot?

You can’t prioritize your list of questions down to fewer than 10.

This is another sign that ‘you aren’t sure what the problem is yet’.   Of course you have more than 10 things that you’d like to know — you probably have hundreds of things you’d like to learn — but you can’t possibly act on more than 10 at a time anyways.

I often see long surveys where the first 3-4 questions are asking about age, gender, zip code, income level.  Those are lazy questions.  If you need to segment by demographic, that should be part of your distribution plan or you should “buy an audience” from a market research firm.

You aren’t willing to write a draft, have another set of eyes review it, and then revise.

The first survey draft you write will suffer from at least one of these flaws:

  • Unclear language (“wait, what are they asking?”)
  • Biased language (“how much do you like feature X?”)
  • Stilted language (technically makes sense but just sounds awkward)
  • Stupid question (“do you want X?”  of COURSE they will say ‘yes’ – that doesn’t reveal anything about actual intent)
  • Too many freeform questions (no more than 1 freeform for every 3 click-to-answer)
  • Mistakenly using single-choice instead of multiple-choice, or vice-versa
  • Doesn’t actually ask the question that you wanted answered
  • Inadvertent rudeness (use of words with negative connotations, or a phrasing that sounds brusque or judgemental)

It is very difficult for you to catch these on your own.  I’ve been writing surveys for years, and I still always have at least one other person read it and comment on anything that is weird or confusing, and I still always have to change at least one thing.

8 Responses to “You Shouldn’t Use a Survey If…”

Anonymous:

….you have not clearly highlighted the benefit(s) (of taking survey) to participants.

Trevor Owens:

Great post Cindy! I’m going to share this with the attendees of our Toronto workshop tonight :-)

Jen Berkley Jackson:

Great post, Cindy!  So true re: needing to have others look at a survey for issues that are hard to see on your own…and great tips for when to not survey…sometimes interviews are much more appropriate, especially in the ‘fuzzy front end’!

marla madison:

Cindy, I would like to talk to you about your article about luck in Women’s Day magazine because I will be quoting from it in my blog. Could you please contact me?
Marla Madison
mam887@gmail.com

Startup Strategies – Best of 2012 – Digital Cowboys:

[...] Cindy Alvarez on applying customer development techniques with different types of customers and on when to NOT use surveys. [...]

Idea Garage Sale: Inbetweenity Edition 2 | The [non]billable hour:

[...] You shouldn’t use a client survey if … [...]

Manuella:

This was absolutely deioclus! I made it for a group of 10 and everyone loved it. It satisfied everyones taste buds and food preferences. I added about 2 pounds more of ground turkey so it would feed more people. I also made it ahead of time before I baked it and it set up well in the fridge overnight. I didn’t have time to saute9 my onions so I grated them and added to the mix. It turned out well. Thank you, it’s one of my favorite recipes now!

Learn the Secrets if Simple Surveys:

I am just particularly satisfied with your ability as a copywriter and even while using the formatting on your web site. Is niagra a given style or maybe would you adjust it oneself? In any case remain in the great top quality publishing, it is extraordinary to seem a pleasant website such as this one currently.

Leave a Reply