Better Product Managers, and Product Management

Some Startup QA Tactics – Browsers, Emails, and Bugs

April 8th, 2010

We recently opened up beta for our new KISSinsights product.  Yay!

Like many small startups, we don’t have a dedicated QA tester, and we don’t have fully automated testing practices in place.  Here’s some of what we did well (and not so well) to get ready for release beyond ourselves.

Browser Support

  1. Pick which browsers you will support and write it down. This sounds so obvious that you probably won’t do it, and then you’ll find out later that someone spent four hours testing on Chrome even though you weren’t planning on supporting it yet.   For now, we’re supporting Win/IE8, Win/IE7, Win/Firefox, Mac/Firefox, Mac/Safari.
  2. Do browser detection and show a message to people using other browsers. If you’re in early beta, you can get away with supporting fewer browsers, but you really do need to set expectations.
  3. Divide up browser testing. If you don’t explicitly say “you test Win/IE7 and I’ll test Win/IE8, etc.” you will end up duplicating efforts and missing things.

Emails

  1. Make sure you have a huge source of available email addresses to test signups. This time, I just set up my personal domain so that [any username] at [domain] would forward to me.  Another way is to pre-create a bunch of testing1, testing2, testing3 accounts at the company domain.
  2. Have every kind of inbox available - GMail, Mail.app, Thunderbird, Outlook, iPhone, etc.  We didn’t do this and so I still haven’t previewed the emails we send in different environments.   Set this up in advance!
  3. Have some third-party friends who can review. After the second or third time you check the emails your app sends, you will stop noticing things.

Bugs

Define bug reporting best practices.  Our developers did this about halfway through (after being annoyed by vague bugs).

For workflow bugs, we are now trying to follow the example below:

Flow:

  1. User tries to activate a survey.
  2. user sees error
  3. checkbox is checked until page reload

Expected behavior
Checkbox becomes unchecked if the activation fails.

How to reproduce

  1. have two or more surveys
  2. set the url to be the same on both surveys (at least one must be deactivated)

For UI bugs, we are trying to follow:

  1. Provide a screen shot.
  2. Find out what browser/OS.
  3. If it came from another user, try to replicate it yourself and tell us if you are able to.
  4. Provide a link to where the issue was found
  5. If at all possible: view source and look for anything weird

It would’ve saved a lot of time if we’d established these examples/guidelines up-front, but hopefully by writing this, you can and will.

Popularity: 2% [?]


Popularity: 2% [?]
  • http://www.twitter.com/kevnd Kevin

    Another option for the x-browser compatibility issue: build with browser independent front end tools such as flex or similar tools. One caution – you will need to solve the need to use the browser back button, as this is a very commonly used feature and creates a bad user experience when it doesn't work.

  • http://blog.zerosum.org zapnap

    It can also help to use an x-browser testing tool, that will show you any x-browser rendering issues that you currently have and allow you to gauge the level of effort required to be compliant. There are a few different services out there with different levels of functionality, depending on your needs. We're actually about to launch a service in this vein ourselves and would love your feedback if you're interested (signups at http://mogotest.com).

    Another thing I'd like to point out is reliance on a good analytics package. By using Google Analytics or a similar service, you can find out what the most commonly used browsers are that are visiting your site, and which pages are getting the most hits, which would allow you to prioritize fixes and browser support tasks.

  • http://www.cindyalvarez.com cindyalvarez

    @Kevin and @zapnap: good points. The goal is to move towards as much automation as possible – cross-browser tools as well as automated functional tests as your product matures.

    In the very early stages, though, I've found it's not a bad thing to do your testing manually – performing the core tasks of your product dozens of times is the best way to quickly identify where your workflow is smooth and easy and where it's slow and tedious. “Within a page” usability issues are pretty easy to spot and customers commonly report them. But workflow usability issues – when your customer just can't get into a natural state of “flow” – people are less likely to report them and more likely to just not use your product. So it's critical that you get a sense for them yourself, and manual testing is one way to make sure that happens quickly.

  • http://twitter.com/cindyalvarez cindyalvarez

    Some Startup QA Tactics: Emails, Browsers & Bugs – New blog post: http://bit.ly/9ZTY40 #startup #prodmgmt

    This comment was originally posted on Twitter

  • http://twitter.com/KISSmetrics KISSmetrics

    Some #Startup QA Tactics – Browsers, Emails, and Bugs http://klck.me/AhE /by @cindyalvarez

    This comment was originally posted on Twitter

  • http://twitter.com/omarjead omarjead

    RT @KISSmetrics: Some #Startup QA Tactics – Browsers, Emails, and Bugs http://klck.me/AhE /by @cindyalvarez

    This comment was originally posted on Twitter

  • http://twitter.com/hnshah hnshah

    Some #Startup QA Tactics We Use at @KISSmetrics – Browsers, Emails, and Bugs http://klck.me/AhE /by @cindyalvarez

    This comment was originally posted on Twitter

  • http://twitter.com/SteveAkinsSEO SteveAkinsSEO

    Some #Startup QA Tactics We Use at @KISSmetrics – Browsers, Emails, and Bugs http://klck.me/AhE /by @cindyalvarez via @hnshah

    This comment was originally posted on Twitter

  • http://twitter.com/jonathanross jonathanross

    RT @KISSmetrics: Some #Startup QA Tactics – Browsers, Emails, and Bugs http://klck.me/AhE /by @cindyalvarez

    This comment was originally posted on Twitter

  • http://twitter.com/NahumG NahumG

    RT @SteveAkinsSEO sum #Startup QA Tactics We Use at @KISSmetrics – Browsers, Emails, & Bugs http://klck.me/AhE /by @cindyalvarez via @hnshah

    This comment was originally posted on Twitter

  • http://twitter.com/sujanpatel sujanpatel

    #Startups QA Tactics Used at @KISSmetrics (Browsers, Emails, and Bugs): http://dld.bz/DvT

    This comment was originally posted on Twitter

  • http://twitter.com/singlegrain singlegrain

    #Startup QA Tactics Used at @KISSmetrics (Browsers, Emails, and Bugs): http://dld.bz/DvT

    This comment was originally posted on Twitter

blog comments powered by Disqus