Better Product Managers, and Product Management

Quantitative vs. Qualitative

April 29th, 2010

When should you seek quantitative vs. qualitative feedback?

I’ve seen a lot of bias towards looking for quantitative data as early as possible – and in my opinion, it’s often too early.

Here’s my rule of thumb:

  • If you’re asking what, why, or how do you questions, you want qualitative data.
  • If you’re asking which or how many/how often questions, you want quantitative data.

Here’s the the tricky part:  making sure your “which” or “how many/how often” questions aren’t actually “what/why/how” questions in disguise.  I see these two examples a lot:

  • Which of these features do you find most valuable? Sounds like a “which” question – but if you’re just guessing at the choices, it’s probably a “what” question — What do you find most valuable about our product?
  • How often are users logging in? Do you really need to know the frequency, or are you using this as a proxy measurement of user engagement?  If so, you should turn it into a “how” question — “How do you use our service?” or “How valuable do you find our service?”

It’s tempting to try and turn as many questions as possible into quantitative ones, because those are easier to measure through unobtrusive means – web analytics, looking for data patterns, surveys – and give clear numerical answers.

However, it’s the messy, hard-to-collect and harder-to-interpret data that will lead you to breakthrough insights – not the percentiles and bar graphs.  Talk to people!

Popularity: 16% [?]


Popularity: 16% [?]
  • http://topsy.com/trackback?utm_source=pingback&utm_campaign=L2&url=http://www.cindyalvarez.com/best-practices/quantitative-vs-qualitative Tweets that mention Quantitative vs. Qualitative (are your quant questions really qual questions in disguise?) New blog post: #measure — Topsy.com

    [...] This post was mentioned on Twitter by cindyalvarez, Dave Concannon. Dave Concannon said: RT @cindyalvarez: Quantitative vs. Qualitative (are your quant questions really qual questions?) New blog post: http://bit.ly/9Wb9bA // READ [...]

  • http://twitter.com/philosophygeek Mark Johnson

    Great post! I'm a big fan so please keep 'em coming!

    I've found that it's not always a dichotomy: sometimes it's great to figure out what your problem is and use quantitative feedback to drive qualitative feedback or vice-versa. For example, if you notice that user engagement is going down, it's sometimes easier (and more valuable) to do some user testing to figure out the “whys” than to dive into the numbers further and still (possibly) come up with nothing. Likewise, information gleaned from user tests can often inform the metrics folks about where to look in the numbers for signals.

  • http://giffconstable.com giffc

    Hear hear!

  • http://skotzko.com AndrewSkotzko

    Nice post! Wish I'd read this on Friday before LA Startup Weekend, but we learned a lot along these lines in our UX testing. Keep em coming!

  • Bilal Dar

    Indeed …pretty good advise, no doubt user feedback is the king..any advise on how to get them engaged in a fun interactive way, non conventional way…

  • http://twitter.com/cindyalvarez cindyalvarez

    Quantitative vs. Qualitative (are your quant questions really qual questions in disguise?) New blog post: http://bit.ly/9Wb9bA #measure

    This comment was originally posted on Twitter

  • http://twitter.com/daveconcannon daveconcannon

    RT @cindyalvarez: Quantitative vs. Qualitative (are your quant questions really qual questions?) New blog post: http://bit.ly/9Wb9bA // READ

    This comment was originally posted on Twitter

  • http://twitter.com/ericnsantos ericnsantos

    When should you seek quantitative vs. qualitative feedback? http://bit.ly/b8DOy2 Tip by @cindyalvarez

    This comment was originally posted on Twitter

  • http://twitter.com/venturehacks venturehacks

    How to turn quantitative metrics into qualitative questions for your users: http://j.mp/9R01iZ. /by @cindyalvarez

    This comment was originally posted on Twitter

  • http://twitter.com/honam honam

    RT @venturehacks: How to turn quantitative metrics into qualitative questions for your users: http://j.mp/9R01iZ. /by @cindyalvarez

    This comment was originally posted on Twitter

  • http://twitter.com/leanbot leanbot

    @venturehacks: How to turn quantitative metrics into qualitative questions for your users: http://j.mp/9R01iZ. /by @cindyalvarez

    This comment was originally posted on Twitter

  • http://twitter.com/MAJACKS0N MAJACKS0N

    Quantitative vs. Qualitative Feedback by @cindyalvarez http://bit.ly/cYt5dh – be careful of what you really want to know from your customers

    This comment was originally posted on Twitter

  • http://twitter.com/plumbinfo plumbinfo

    RT @tribeofangels: Qualitative versus quantitative product input http://bit.ly/bWLkVZ #startup # product #software

    This comment was originally posted on Twitter

  • http://twitter.com/idyeahc idyeahc

    experience IS the product: http://bit.ly/bWLkVZ

    This comment was originally posted on Twitter

  • http://twitter.com/fulcrum51 fulcrum51

    RT @cindyalvarez: Quantitative vs. Qualitative (are your quant questions really qual questions in disguise?) New blog post: http://bit.ly/9Wb9bA #measure

    This comment was originally posted on Twitter

  • http://twitter.com/AndrewSkotzko AndrewSkotzko

    @chrisgagne bro check out this cool article on UX testing http://bit.ly/bd3g9C

    This comment was originally posted on Twitter

  • http://twitter.com/danmartell danmartell

    Quantitative vs. Qualitative The Experience is the Product http://bit.ly/bvzLos /by @cindyalvarez #custdev

    This comment was originally posted on Twitter

  • http://twitter.com/mjmantey mjmantey

    RT @danmartell: Quantitative vs. Qualitative The Experience is the Product http://bit.ly/bvzLos /by @cindyalvarez #custdev

    This comment was originally posted on Twitter

  • http://twitter.com/SteveAkinsSEO SteveAkinsSEO

    Quantitative vs. Qualitative The Experience is the Product http://bit.ly/bvzLos /by @cindyalvarez #custdev RT @danmartell

    This comment was originally posted on Twitter

  • http://twitter.com/ToddrLevy ToddrLevy

    RT @danmartell: Quantitative vs. Qualitative The Experience is the Product http://bit.ly/bvzLos /by @cindyalvarez #custdev

    This comment was originally posted on Twitter

  • http://twitter.com/TCiiLondon TCiiLondon

    RT @danmartell: Quantitative vs. Qualitative The Experience is the Product http://bit.ly/bvzLos /by @cindyalvarez #custdev

    This comment was originally posted on Twitter

  • http://twitter.com/MaxStrategy MaxStrategy

    RT @ danmartell Quantitative vs. Qualitative The Experience is the Product http://bit.ly/bvzLos /by @cindyalvarez #custdev

    This comment was originally posted on Twitter

  • http://twitter.com/IamPramit IamPramit

    Quantitative vs. Qualitative The Experience is the Product http://bit.ly/bvzLos /by @cindyalvarez RT @danmartell | RT @SteveAkinsSEO

    This comment was originally posted on Twitter

  • http://twitter.com/Hadel Hadel

    RT @IamPramit Quantitativevs.QualitativeThe Experience is the Product http://bit.ly/bvzLos by @cindyalvarez RT @danmartell RT @SteveAkinsSEO

    This comment was originally posted on Twitter

  • http://twitter.com/SEOcopy SEOcopy

    RT @SteveAkinsSEO: Quantitative vs. Qualitative The Experience is the Product http://bit.ly/bvzLos /by @cindyalvarez #custdev RT @danmartell

    This comment was originally posted on Twitter

  • http://twitter.com/chrisgagne chrisgagne

    RT @AndrewSkotzko: @chrisgagne bro check out this cool article on UX testing http://bit.ly/bd3g9C

    This comment was originally posted on Twitter

  • http://twitter.com/see_jane_sell see_jane_sell

    RT @danmartell: Quantitative vs. Qualitative The Experience is the Product http://bit.ly/bvzLos /by @cindyalvarez #custdev

    This comment was originally posted on Twitter

blog comments powered by Disqus