You shouldn’t use a survey if:

You aren’t sure which type of people you should ask to take your survey.

There are almost zero contexts where you’ll get useful data out of having “anyone” take your survey.   (And thinking, “I’ll just get thousands of responses, and then filter by some criteria later” is both a copout and unlikely.)   Do you want to hear from existing customers?  People from a certain region? People who share a common activity?  People with a specific job title?

Your “type of people” can be subjective, too — in a recent survey I conducted, I wanted to hear from was “smart people that we’d want to hire if we had the chance”.  So I distributed the survey through coworkers’ personal networks, letting them make that subjective determination.

You know which type of people you want to take your survey but have no idea how to find them.

Surveys are not an “if you build it, they will come” exercise.  Don’t waste your time on a survey if you don’t have a ready bank of people to send it to or a distribution strategy.   It’s a waste of time and social capital to send out a survey and get only 4 or 5 responses.

You’re better off conducting freeform interviews first so you can increase the “learning density” you get from each person.   For the long term, you’ll need to invest time in figuring out how to find more people.  This usually means “find where these people already are, and put yourself there” – participate in forums, join clubs, build up your network.

All of the questions you want to ask require a freeform response.

This is a sign that you don’t really know what the questions are yet, that you aren’t really sure what the problem is yet.

If you have an existing product or customer base, look at usage patterns and past feedback first.  Then start with freeform interviews, so you can tease out information in a back-and-forth context.  The survey format is bad for this type of learning.   People don’t like to write a lot, and even if they do, their first response is usually not where all the ‘meat’ is.

You don’t have a clear plan of action for how you’re going to use the results.

If you’re thinking “we might learn something” or “it would be nice to know…”, then save yourself the time (and save the time of the people who would fill out your survey).   Data without action is meaningless.

Is this information going to help you make a better decision on a specific feature or project?  Will it help you choose better wording or smarter defaults in your application?  Will it validate a specific hypothesis so that you can continue or pivot?

You can’t prioritize your list of questions down to fewer than 10.

This is another sign that ‘you aren’t sure what the problem is yet’.   Of course you have more than 10 things that you’d like to know — you probably have hundreds of things you’d like to learn — but you can’t possibly act on more than 10 at a time anyways.

I often see long surveys where the first 3-4 questions are asking about age, gender, zip code, income level.  Those are lazy questions.  If you need to segment by demographic, that should be part of your distribution plan or you should “buy an audience” from a market research firm.

You aren’t willing to write a draft, have another set of eyes review it, and then revise.

The first survey draft you write will suffer from at least one of these flaws:

• Unclear language (“wait, what are they asking?”)
• Biased language (“how much do you like feature X?”)
• Stilted language (technically makes sense but just sounds awkward)
• Stupid question (“do you want X?”  of COURSE they will say ‘yes’ – that doesn’t reveal anything about actual intent)
• Too many freeform questions (no more than 1 freeform for every 3 click-to-answer)
• Mistakenly using single-choice instead of multiple-choice, or vice-versa
• Doesn’t actually ask the question that you wanted answered
• Inadvertent rudeness (use of words with negative connotations, or a phrasing that sounds brusque or judgemental)

It is very difficult for you to catch these on your own.  I’ve been writing surveys for years, and I still always have at least one other person read it and comment on anything that is weird or confusing, and I still always have to change at least one thing.

• What if we hire this person and they aren’t able to execute?
• What if they can’t negotiate with our engineers?
• What if they can’t operate under conditions of uncertainty?
• What if they worry over every little detail until they’ve totally missed the big picture?
• What if they just say “yes” to everything so we lose our focus?
• What if they blithely accept our broken processes and bad habits so that we keep making the same mistakes?

In the past month, I’ve spent quite a bit of time being the interviewee, the interviewer, and the resume mentor.  And the biggest mistake that I’ve made myself and seen others make is in being too neutral.

You don’t want to make a bad first impression.  You don’t want to come across as someone argumentative and difficult to work with.  You want to make it clear that you’re flexible and adaptable.

And that leads you into exchanges like this:

Interviewer: “Which development methodology do you prefer working with?”
PM Candidate: “I’ve worked in companies where we did waterfall, as well as agile, as well as daily releases, and I’ve been able to be effective in all of them.”

The interviewer thinks: If you were working closely with engineers and customers, you’d have directly felt the pain of what didn’t work.  Either you were pretty hands-off (bad) or you don’t reflect on your work and process to see how you can drive improvement (also bad).

Interviewer: “Tell me about a time when you’ve dealt with conflict between product management and engineering/design/sales.”
PM Candidate: “I believe in working very closely with other teams and keeping the lines of communication open so we can avoid conflict.”

The interviewer thinks: Seriously? There are always differences of opinion and conflicting priorities.  Either you just say “yes” to everything (bad), or you are oblivious to the fact that sales hates you (also bad), or your upper management takes all decisions out of your hands (maybe not your fault, but doesn’t make me want to hire you).

Interviewer: “What type of difficulties do you foresee in working with our product/culture/customers/distribution model?”
PM Candidate: “I’ve worked with X before, so I’m very confident that I’ll be able to handle the job here.  Here’s an example of a related awesome thing I’ve done in the past.”

The interviewer thinks:  If you are a good candidate, this isn’t your only job option.  You should be critically thinking about the “cons” of this job and why you might not want it.  If you haven’t, that’s a warning sign.  If you have, but you’re not comfortable articulating them, how are you going to handle day-to-day communication and negotiation here?

It’s true that expressing your opinion may lose you a potential job.  If you’ve eloquently detailed how you advocate user-centered design and incorporating customer research into the development process, the company where engineers call all the shots, is not going to offer you a job.

This is not a bad thing.  You wouldn’t do great work at that job anyways — and how are you going to land your next job with a bank of mediocre work and no positive references?  Better to dodge that bullet now.

If you have work experience, you have opinions.   As a hiring manager, I want to hear them — as well as the context that you formed them in and how you arrived at them.  Maybe you dislike a certain process because you’ve been burned by it in the past but you can see how it could be beneficial… or maybe you’re just closed-minded and resistant to change.

No one worries that you’ll be flat-out incompetent — if you were, it’d be blatantly obvious after a couple weeks, your manager could document your flaws and have you out in no time.  But an employer is petrified that you’re going to be not-quite-right in more subtle, insidious ways that won’t be noticeable until it’s too late and you’ve botched a product release or customer relationship.

The tricky thing about product managers is that every candidate is different, and every past environment is different.   Just because the last product you managed was a runaway success doesn’t guarantee that you were the reason why — maybe it was “right place at the right time”, maybe your boss made all the decisions, maybe it succeeded in spite of you.  Same thing with product flops — maybe the market tanked, maybe your boss made all the (wrong) decisions despite your recommendations, maybe there was an engineering disaster.  So your resume doesn’t really set the hiring manager’s mind at ease.

It’s your job as a candidate to make the hiring manager feel better about those risks.  And how do you do that?  By being opinionated.  Give examples.  Explain the “why” and “what I learned”.   Write blog posts.  Answer questions on Quora.  The less of a “black box” you can be, the better candidate you’ll be.

• Invite people over for dinner
• Find a recipe, skim it, buy the ingredients
• Late afternoon, start cooking–
• …and realize that I’d missed the part where something needed to rise for 12 hours, or chill for 4 hours, or roast for 4 hours…

In other words, this dish was not going to be ready for dinner, no matter how badly I needed it.  I’d simply overlooked that there was a long cooking time, and now I had a choice of trying to make the best of a half-cooked entree or quickly whipping together an inadequate substitute.

Either way, my dinner guests were not going to be happy.

There are a lot of things in product management that have a long cooking time.  And somehow, we overlook that, until we really, really need whatever it is.

Some of these things are:

• Case Studies
• Educational content
• Reference customers
• Usage and retention metrics
• Revenue and customer lifetime value metrics

You probably don’t have all of these today (even I don’t have these to the degree I’d like to).  You may think you’re not ready (“I can’t think about case studies yet – I just released my beta last week”) — but you are ready to make a plan.

Ask yourself, what can I do today to make sure that I have this data 6 months from now?   Work backwards and write everything down, no matter how obvious it may seem:

To have case studies, you need successful customers who like you enough to collaborate with you.  To get there, you need:

• to have developed a good working relationship with customers
• to have identified customers who had the potential to be successes and nurtured them and worked with them to make sure they do succeed
• to know “what a successful customer looks like”

Even if you just launched your beta last week, you can start with that bottom rung: what do you think a successful customer might look like?  Write down your hypothesis.  That’s the first step.

For any of these “long-cooking” items, it may seem like a long, long ways off.  But 6 months from now, or 12 months from now, will come a lot faster than you think.  Start reading the recipe now.

Personas are, in my opinion, the often-neglected third leg of the stool in creating awesome products.  They live in between individual customer interviews – Bob X. says this — and a wider target market profile — we’re targeting small businesses making over $10MM revenues with these characteristics.

Persona Analysis is the process of identifying individual person representatives for your audience and fleshing out their traits, likes and dislikes, frustrations and limitations.  Personas are frequently used in the interaction design / user experience worlds, but much less so in product management.

For any product, there are different types of people who:

• need your product / have already bought your product
• are able to get value from your product
• have money to spend on your product

These different types of customers have different job titles, priorities, responsibilities, and technical abilities.   These differences affect how they decide to buy your product, what frustrates them or excites them about your product, and how you can help them start using your product quickly/effectively.

If you’re a startup, you are ready for personas as soon as you start hearing different types of answers from your interviews or you start witnessing different types of behaviors among your users.  (“Gita said she’d never use Wizard X because she’d rather configure it manually” / “Devon loves Wizard X because he finds manual configuration too difficult!”)

What does a persona look like?

Some people spend a lot of time giving their personas pictures, names, and back-stories.   To me, the only reason to spend time on that is if helps your team connect with your personas.  (In my experience, too much “marketing” of personas makes engineers distrust them.)

Here are the questions I asked to flesh out our personas.  They are geared towards a B2B software product, but many are actually fairly applicable to consumer and/or physical products as well:

• What is this person’s comfort level with technology?
• Why are they interested in solving the problem that your product solves? (i.e. In what way will it make their life better?)
• What challenges does this person face in their job? (in general – not specific to your product)
• What capabilities help this person do their job more effectively?
• What are this person’s primary limitations to doing their job well? (time/money/technical skill/access to information)?
• What are this person’s primary limitations to trying a new product/process/concept? (time/money/approval/technical skill/access to information/fear)
• How has this person been burned by products in the past? (didn’t do what they promised/too difficult to set up/limited flexibility/limited power)
• What claims would this person likely be skeptical of?
• How does this person tend to do due diligence / verify information?
• Which teams in their organization do they work closely with?
• What is this person afraid of? (i.e. what do they perceive as a risk that will damage their company or their ability to do their job)

When I looked at our actual customer interview notes, three distinct “personas” emerged.

(IMHO: 2-4 is probably ‘normal’; if you have only 1 persona you don’t have enough customers yet and if you have more than 4 you may be asking the wrong questions or have an overly generic product.)

For each one, I wrote up:

• a 2-sentence summary of their job role
• [Person] can do her job effectively when …
• [Person] is frustrated by / skeptical of …
• [Person] needs to be convinced …
• How do you market to / communicate with [Person]?

Laying them out side by side gave us a new appreciation of our customer types — and made it really clear how people trying to solve the same goal may approach it from almost completely opposite directions.

Personas need updating

If you are a startup, or are launching a new product, your personas are in flux.

You’ve read Crossing the Chasm – this is where it applies.  In the beginning, you will probably have 1 persona – the most technical, the most power-user, the most early-adopter.  These people are great: they give tons of feedback, they’re smart, they’re patient while you figure out what the heck you’re doing.  They’re also NOT what you want to build your product around (unless your aim is a niche market play).

So if you create personas and you only have 1 — try again in 3 months.  If you only have 2 — try again in 6 months.  Even if you think you have it covered, totally — revisit again in 6 months.  The internet keeps changing and customer expectations keep changing along with it, so these are not a “set it and forget it” exercise.

“Yeah, he seemed like a great candidate; had a lot of success with Similar Product, had exactly the kind of experience we really needed on the team, We really thought he would be the guy who could take us to the next level, but he just… hasn’t been able to deliver / isn’t the right cultural fit / just doesn’t seem to work well with the team.”

And I can always predict the reason: the candidate used to work at Some Big Company.

Big Company people are not universally bad for startups. In fact, you’ll never grow your startup without them.  But lots of them are not the right hire for your startup.

At KISSmetrics, I recently hired someone from a Big Company, and just acquired a co-worker from a Big Company.  Both of them are great additions to the team, and already bringing a needed infusion of energy, experience, and different perspectives.

But that hasn’t always been my experience.  I’ve worked with Big Company additions to startups that actually made me dread coming to work in the mornings.  I’ve even hired a Big(ger) Company person that I had to end up firing.

So how do you get it right?  What are the tips for finding the right Big Company people for your startup?

Here are the 3 traits  I’ve found that the great hires have in common:

Action

Lots of potential hires have ideas.  Surprisingly few have actions — that is, they’ve researched you, they’ve tried your products, they’ve identified areas where you need to improve and done something about it.

As an applicant, you have limited visibility into a company and their priorities. But life in a startup is always full of uncertainty.  You need people who are comfortable saying, “based on the information I have, this is what I will do / have done”, not people who wait to act until someone tells them it’s okay to do so.

Fusion

Having launched a really successful product tells me one thing: that you launched that successful product, at that company, at that point in time.   It does not mean that you can necessarily repeat that feat with this product, at this company, and this point in time.

Great candidates can take bits of past successful processes and blend them with new realities and constraints.  They have the insight to understand the context of past successes (team size, technical savvy, budget, 400lb gorilla bargaining power, tools, target audience) and adapt their experience to where they are now.

Another way of saying this is, there are 2 kinds of domain experts: the ones who have had 20 years of varied experiences, and the ones who have had the same 2 years of experience, repeated 10 times.

…and possibly the most important,

Impatience

Things move 10x faster at a startup than at a Big Company.

Your Big Company candidate ought to be squirming with impatience in their current environment.  They want their ideas to be implemented faster.  They want to stop with the extensive analysis and start experimenting.  They crave doing things vs. talking about them and planning them for next year.

If they aren’t frustrated and impatient and being scolded for trying to shortcut process now, they’re going to have a horrible culture shock when they show up at your office.    They’ll want to re-create the predictability of their old environment, which usually results in an employee who hides beyond process or cuts off new information that may spur a changed decision.

An impatient candidate will still probably have some culture shock.  But they’ll react in the opposite way: the energy will invigorate them to work harder and welcome new information and use it to look for the ways they can best contribute.

You went to the store with a $1 to spend, picked out something that cost $1, headed to the register — and found out it actually cost $1.05 with tax.  It probably totally derailed you.  (My mom didn’t give me the extra nickel; she just explained that tax meant I really only had 90-something cents to spend.  Probably a valuable economics lesson to learn, though I’m sure I didn’t see it that way at the time.)

Your customers (hopefully) are not being derailed by lack of a nickel. But they are being hit by the thinking tax.

The thinking tax is what happens when your customers approach your product, ready to perform a task, and then realize that there’s a decision they need to make.  Or a question they need to answer.  Or a choice they need to figure out.  And that extra bit of un-budgeted-for work is what makes them stop and say, “I’ll come back to this later.”

It can be incredibly frustrating as a product manager to watch this happen, because the “thinking tax” is usually very low.  It’s usually something like:

• looking up which settings you used last time
• finding an example of how other people did it
• remembering to download something
• thinking about what each option means so you can pick the right one
• making a list of the little related tasks you need to do

…but that’s enough to make your customer hesitate.

So often, they’ve got a short window of time – maybe 10 minutes before their next conference call starts, or 5 minutes while they’re waiting in the school pick-up parking lot – and they’re ready to act.   If they were able to think, “I should do this”, they would just do it. But when that thought subtly shifts to “Now, what should I do?”, they worry a bit.

Dan and Chip Heath, the “Made to Stick” guys, write about some of the academics behind this: Tase the Haze.

Who wants to make a mistake just because they’re in a hurry, or have a headache, or are in a noisy cafe?  Nope, better to wait until they have some peace and quiet and time to think.

Let me repeat that last bit:  “better to wait until they have some peace and quiet and time to think”.

Yeah.  I have a kid and I work in a startup.  That time might be 17 years from now.

But you can avoid the thinking tax on your product.  Here’s how:

• show examples
• make recommendations
• choose intelligent defaults
• reassure customers about any choices/decisions that can be changed later (i.e. no ‘penalty’ for choosing wrong)

Yesterday, I gave an Action Class talk for AppSumo, and my contact, @michaeldwp, did a great job with this.  He sent emails that clearly listed what he needed from me, provided links to examples of previous talks, outlined a suggested structure for my talk, provided links to the software I’d need to download, provided a short link for Tweeting, reminded me to add him on Skype, and recommended that I sign on early to confirm that audio and video were working.

Would I have been able to figure this out on my own?  Sure.  But it would’ve been much more stressful, and the odds would’ve been much higher that something would go wrong.  There was a pretty long list of logistics to take care of, but they didn’t really feel like work, because I didn’t have to stop and think about every single item.

I didn’t have to stress that I was forgetting something, or take time to break down the big task into little manageable chunks.  That was done for me.  I was free to focus on what I came to do — prepare for and give a talk.

BTW, for those who couldn’t make it:

• “Get More Useful Feedback” Presentation
• “Get More Useful Feedback” Action Class (if you haven’t already, you’ll need to sign up for Action Class emails to view)

IAQ stands for Infrequently Asked Questions.  An IAQ is a document where you know the specific questions that your customers should be asking and you  present them along with the answers.

(Neither of these are to be confused with an SSQ — the Self-Serving Questions document, where companies “ask” and answer questions like ‘Why is Company X a leader in the Y industry?’ or ‘How has Company X maintained a 90% customer satisfaction score for so long?’ that no customer has ever asked.)

So, what are the signs that you need an IAQ?

• Customers say things like “I don’t even know where to start”
• Customers ask, “How are other people using this product?”
• When you give a demo, the response is “Wow! I didn’t even realize I could do that!”
• “Power users” rate your product as off-the-charts awesome, but ‘everyone else’ rates it significantly lower
• Login frequency/usage is very high for the first week or two, then trails off significantly

All of these signs suggest that most of your customers aren’t getting the full value out of your product because they don’t realize what they can do with it.

They’re not looking for answers — they don’t even know what questions they should be asking.  This is the point of the IAQ — to bridge the gap between “how your customer is currently using your product” and “the product manager’s vision of how a customer should ideally be using the product.”   It’s the equivalent of a helpful friend leaning over your shoulder and saying, “did you realize you could do X?”

What should go into my IAQ?

• How to solve specific problems — emphasis on the problem, not the feature which solves it
• Supported use cases that the customer may not have even thought to ask about
• What problems other customers are solving with your product

With KISSinsights, I mixed the Infrequently Asked Questions in with the more traditional Frequently Asked Questions. Here’s a little detail about some of those questions:

“Can I control who sees the surveys?” NOT a question I’d been asked a lot.  But in talking to customers, I realized that they hadn’t really grasped that they could trigger surveys to only be shown to specific types of visitor.  I got the “I didn’t realize I could do that!” response.   Once I added the question, I got a lot of related questions.  Once that possibility was in customers’ minds, they were excited and had other ideas on how to use it.  But I had to plant that seed first, for most folks.

“Will I be able to tell who’s taking my survey?” was similar.  Most of our customers are familiar with traditional SurveyMonkey-type surveys, where you need to explicitly ask for an email address.  So I needed to plant the seed that they’d be able to pass that information behind-the-scenes.

“Can I show the survey in non-English languages?” Most web services from US companies are English-only, so potential customers don’t even ask.  By calling out that question, we’ve gotten a pretty large number of non-English-speaking customers — and that’s with zero advertising or marketing to those audiences.

“How are other people using KISSinsights?” This is the jackpot question.  It’s here on the page, and I send it out in emails at least once a week.  This list — and it’s just a simple list of ideas — is one of the best convincers I’ve worked with.  (By which I mean, anecdotally, of the people who email me to ask about KISSinsights, the percentage who upgrade after seeing this list is ridiculously high.)

If you had 5 full minutes of your customers’ time — an unlikely gift, I know — what would you want them to know about they could solve their problems using your product?

I’m not opposed to structure.  Repeatable processes are necessary to get from a tiny scrappy company to a successful one.  But they need to keep in mind that some of the smartest people you work with don’t think like you at all.

Not everyone is good at “real-time”

Some people can listen and think and summarize and talk at the same time.   (Given recent studies on multitasking, probably more of us think we’re good at this but actually aren’t.)

For others, it’s like being asked to listen to a Beatles song and write down the lyrics to a Rolling Stones song at the same time.   It’s stressful and you will almost certainly mix up bits of the two.   It’s a smart employee who recognizes they can’t do both and focuses on understanding first and offering solutions later.  Unfortunately, they’re often perceived as too shy, too passive, or “never having any good ideas.”

The stop, go away, come back method helps alleviate this.  So does sending out an agenda/information before a meeting, or making a point of following up via email afterwards to catch any asynchronous insights.

Everyone has a continuum of most-to-least effective communication methods (and they are different)

IM and face-to-face are my favorite mediums. Other people prefer phone calls, sketches, or writing up emails.  This is all fine as long as you alternate mediums.

When your corporate culture is strongly biased towards conference calls but half the team communicates better via writing, it’s bad for morale (people who aren’t good at phone calls know it, and know they’re going to have a hard time getting their ideas across) and it’s bad for efficiency.

I hated the focus on conference calls when I worked with an offshore team — interactions via IM were faster and there was more understanding going on.  Between bad Skype connections and both sides thinking the other team had heavy accents, I always felt like I spent more time saying “What?” than “Great idea!”

If all you have is a hammer…

Want to know why engineers are always offering technical solutions to a problem?  Because you hired them for being awesome with a hammer.

“Soft skill” methods are not innate.  No one is born knowing how to rapid brainstorm, ask the 5 Whys, interview a customer, do competitive analysis on competitor products, or formulate questions to ‘ask the data’.   How can you expect your engineers/designers to contribute using methods that they’ve never learned?

It’s not hard to take a few minutes to walk through various approaches (faced with this problem, here’s one method I’d use and why I’d choose it, and here’s how I’d start…) but apparently it’s easier to just mutter under your breath about how “they” “just don’t get it”.  Easier, but not recommended.

If you’re not sure how well your new process is working, you could wait to find out when you conduct a project postmortem.  Or you could try asking (using that person’s preferred communication method.) Were you able to understand the problem completely?  Did you feel like you were able to make others understand your concerns/ideas?  Is there a way that would make it easier for you to contribute?

Now that would be Thinking Different.

I’ve seen a lot of bias towards looking for quantitative data as early as possible – and in my opinion, it’s often too early.

Here’s my rule of thumb:

• If you’re asking what, why, or how do you questions, you want qualitative data.
• If you’re asking which or how many/how often questions, you want quantitative data.

Here’s the the tricky part:  making sure your “which” or “how many/how often” questions aren’t actually “what/why/how” questions in disguise.  I see these two examples a lot:

• Which of these features do you find most valuable? Sounds like a “which” question – but if you’re just guessing at the choices, it’s probably a “what” question — What do you find most valuable about our product?
• How often are users logging in? Do you really need to know the frequency, or are you using this as a proxy measurement of user engagement?  If so, you should turn it into a “how” question — “How do you use our service?” or “How valuable do you find our service?”

It’s tempting to try and turn as many questions as possible into quantitative ones, because those are easier to measure through unobtrusive means – web analytics, looking for data patterns, surveys – and give clear numerical answers.

However, it’s the messy, hard-to-collect and harder-to-interpret data that will lead you to breakthrough insights – not the percentiles and bar graphs.  Talk to people!

Like many small startups, we don’t have a dedicated QA tester, and we don’t have fully automated testing practices in place.  Here’s some of what we did well (and not so well) to get ready for release beyond ourselves.

Browser Support

1. Pick which browsers you will support and write it down. This sounds so obvious that you probably won’t do it, and then you’ll find out later that someone spent four hours testing on Chrome even though you weren’t planning on supporting it yet.   For now, we’re supporting Win/IE8, Win/IE7, Win/Firefox, Mac/Firefox, Mac/Safari.
2. Do browser detection and show a message to people using other browsers. If you’re in early beta, you can get away with supporting fewer browsers, but you really do need to set expectations.
3. Divide up browser testing. If you don’t explicitly say “you test Win/IE7 and I’ll test Win/IE8, etc.” you will end up duplicating efforts and missing things.

Emails

1. Make sure you have a huge source of available email addresses to test signups. This time, I just set up my personal domain so that [any username] at [domain] would forward to me.  Another way is to pre-create a bunch of testing1, testing2, testing3 accounts at the company domain.
2. Have every kind of inbox available - GMail, Mail.app, Thunderbird, Outlook, iPhone, etc.  We didn’t do this and so I still haven’t previewed the emails we send in different environments.   Set this up in advance!
3. Have some third-party friends who can review. After the second or third time you check the emails your app sends, you will stop noticing things.

Bugs

Define bug reporting best practices.  Our developers did this about halfway through (after being annoyed by vague bugs).

For workflow bugs, we are now trying to follow the example below:

Flow:

1. User tries to activate a survey.
2. user sees error
3. checkbox is checked until page reload

Expected behavior
Checkbox becomes unchecked if the activation fails.

How to reproduce

1. have two or more surveys
2. set the url to be the same on both surveys (at least one must be deactivated)

For UI bugs, we are trying to follow:

1. Provide a screen shot.
2. Find out what browser/OS.
3. If it came from another user, try to replicate it yourself and tell us if you are able to.
4. Provide a link to where the issue was found
5. If at all possible: view source and look for anything weird

It would’ve saved a lot of time if we’d established these examples/guidelines up-front, but hopefully by writing this, you can and will.