When should you seek quantitative vs. qualitative feedback?
I’ve seen a lot of bias towards looking for quantitative data as early as possible – and in my opinion, it’s often too early.
Here’s my rule of thumb:
- If you’re asking what, why, or how do you questions, you want qualitative data.
- If you’re asking which or how many/how often questions, you want quantitative data.
Here’s the the tricky part: making sure your “which” or “how many/how often” questions aren’t actually “what/why/how” questions in disguise. I see these two examples a lot:
- Which of these features do you find most valuable? Sounds like a “which” question – but if you’re just guessing at the choices, it’s probably a “what” question — What do you find most valuable about our product?
- How often are users logging in? Do you really need to know the frequency, or are you using this as a proxy measurement of user engagement? If so, you should turn it into a “how” question — “How do you use our service?” or “How valuable do you find our service?”
It’s tempting to try and turn as many questions as possible into quantitative ones, because those are easier to measure through unobtrusive means – web analytics, looking for data patterns, surveys – and give clear numerical answers.
However, it’s the messy, hard-to-collect and harder-to-interpret data that will lead you to breakthrough insights – not the percentiles and bar graphs. Talk to people!