Quantitative vs. Qualitative

When should you seek quantitative vs. qualitative feedback?

I’ve seen a lot of bias towards looking for quantitative data as early as possible – and in my opinion, it’s often too early.

Here’s my rule of thumb:

  • If you’re asking what, why, or how do you questions, you want qualitative data.
  • If you’re asking which or how many/how often questions, you want quantitative data.

Here’s the the tricky part:  making sure your “which” or “how many/how often” questions aren’t actually “what/why/how” questions in disguise.  I see these two examples a lot:

  • Which of these features do you find most valuable? Sounds like a “which” question – but if you’re just guessing at the choices, it’s probably a “what” question — What do you find most valuable about our product?
  • How often are users logging in? Do you really need to know the frequency, or are you using this as a proxy measurement of user engagement?  If so, you should turn it into a “how” question — “How do you use our service?” or “How valuable do you find our service?”

It’s tempting to try and turn as many questions as possible into quantitative ones, because those are easier to measure through unobtrusive means – web analytics, looking for data patterns, surveys – and give clear numerical answers.

However, it’s the messy, hard-to-collect and harder-to-interpret data that will lead you to breakthrough insights – not the percentiles and bar graphs.  Talk to people!

9 Responses to “Quantitative vs. Qualitative”

Tweets that mention Quantitative vs. Qualitative (are your quant questions really qual questions in disguise?) New blog post: #measure -- Topsy.com:

[...] This post was mentioned on Twitter by cindyalvarez, Dave Concannon. Dave Concannon said: RT @cindyalvarez: Quantitative vs. Qualitative (are your quant questions really qual questions?) New blog post: http://bit.ly/9Wb9bA // READ [...]

Mark Johnson:

Great post! I'm a big fan so please keep 'em coming!

I've found that it's not always a dichotomy: sometimes it's great to figure out what your problem is and use quantitative feedback to drive qualitative feedback or vice-versa. For example, if you notice that user engagement is going down, it's sometimes easier (and more valuable) to do some user testing to figure out the “whys” than to dive into the numbers further and still (possibly) come up with nothing. Likewise, information gleaned from user tests can often inform the metrics folks about where to look in the numbers for signals.

AndrewSkotzko:

Nice post! Wish I'd read this on Friday before LA Startup Weekend, but we learned a lot along these lines in our UX testing. Keep em coming!

Bilal Dar:

Indeed …pretty good advise, no doubt user feedback is the king..any advise on how to get them engaged in a fun interactive way, non conventional way…

ugg boots:

Thanks a landlord it! I acquired yet some insight. Life is so colorful, we should be able to live in, such as Korea and honor the planet. Human life is like rivers, slowly flowing, flowing rivers, flowing through the snow, flows through the prairie and ultimately into the sea, return to the embrace of nature, start a new reincarnation. Allow us to feel the meaning of life will come only to those you have those memories http://www.chaneloutletstores.com/Chanel-scarf.html

ugg boots:

“Well , the view of the passage is totally correct ,your louis vuitton handbags details is really reasonable and you guy give us valuable informative post, I totally agree the standpoint of upstairs. I often surfing on this forum when I m free and I find there are so much good information we can learn in this forum!

aptitude tests:

One can download practice aptitude tests containing different types of questions on verbal ability, numeric ability, abstract reasoning, spatial ability, mechanical reasoning, and fault diagnosis.

jay:

Smart talk. I like your ideals and your opportunity. Helped me consider some strategies myself. http://www.brewcityroyalties.org

Leave a Reply