Rainbow Banding : How to analyze results from Usability Testing

I have noticed that there is a wealth of literature online on how to conduct a Usability Study, but I have yet to find anything to demonstrate how to analyze the data collected from a study. Therefore, I thought I’d share a technique that works well for me.  I call it “Rainbow Banding”.

1. Prepare your testing protocol with spaces for participant feedback

Create a protocol template with numbered tasks and questions.  Ensure there is adequate space between each to record relevant answers and observations.

2. Record  the observations

Record the observations from each participant in a fresh copy of this document.  If changes to the protocol, ordering, or wording need to be made, ensure you make them as early in the testing process as possible.

3. Combine your observations

Once you have recorded observations from all of your participants, combine your results in a blank copy of your testing protocol.

Create a section above labelled “Participants” and assign each participant both a unique number and colour. Start by recording the observations from each participant under the appropriate numbered heading (shown below). Use the colour associated with that participant to record the observation. If a second  participant demonstrates the same behaviour or expresses the same sentiment, add a +1 to that observation in the second participant’s colour (this will make analysis easier later-on).

Once you have combined the results from 3 participants, review this list of combined observations and add subheadings to further combine and categorize the findings and observations.  Do this every few participants to make it easier to tally and analyze the results later-on.

In the example above both ‘Dave’ and ‘Holly’ demonstrated the same sentiment under 2) subheading Recent Posts.  +1 was added to the sentiment in each participant’s colour.  You will notice that this helps reduce visual clutter and makes analysis easier at the end.  You will also notice that the subheadings Readability  and  Navigation were added below 4).  Categorizing the feedback as you go along helps chunk the data and will make analysis easier at the end.

4) Tally the Results

Once you have combined the observations from all participants, you can now easily analyze the results adding up similar sentiments, and categorizing findings into a report.

 

Please write me and let me know if this technique worked for you, if you have a similar technique, or if you found it worked poorly.

Hurtwallet

Design First

Design first.  It is a simple principle that is too often disregarded.  I don’t just mean “Design” in terms of just aesthetics- also usability and user experience.  Design often takes a back seat to feature sets, marketing/branding requirements, legality, and as a result becomes compromised.  Good design is the difference between a stellar product, and a product that is lackluster.  The fastest car in the world will never be popular if it is an eyesore and if it is difficult to drive.  Design should not be an afterthought, but a defining requirement.

Until recently, design seemed ill-considered in the tech industry.  Early PCs were cream-coloured boxes with an overwhelming array of buttons, switches, and dials that were unintuitively placed.  Now PCs are sleek, sexy, and every button is purposefully placed.  What changed?  In 1998, Apple started to put design first in all of their products.  From then until now, Apple created some of the biggest lust items in their industry, overtaking tech giants of the time in relatively short period of time.  Arguably, their success inspired design changes in both the computing and mobile phone industries.  Engineering-minded companies who were once very profitable have been forced to change their way of operating and put design first in order to stay competitive; as a result, this shift towards design has created some fierce competitors who are now starting to gain back ground.  The end result will be a variety of better consumer products.

Working for one such ‘Engineering-minded’ company, I was told by a team of developers AFTER launching a lackluster product “we were told to ignore UX.  We now realize that was a mistake.”  This statement was both vindicating and depressing.  Together, we had the chance to create an amazing product, but in the end it fell short.  Functionally, the product was great: it met all the marketing requirements, and implemented new and useful features.  However, the visually stunning and intuitive design we created for them was disregarded.  As a result, the product looked terrible, and demonstrated a poor user experience. Users were unimpressed.

This is why I’m making a case for design.  Design should come first.  If you put design first,  you are showing the end user that they are what’s most important to you, and you are designing for them.  You make a personal connection with the user this way, and they reciprocate by investing in your product financially, mentally, and emotionally.  If you allow a 3rd party to dictate your requirements (a seller of some sorts- whether it be a carrier, a retailer, a stakeholder, etc.), you are putting design and the end user last.  They WILL notice, and reciprocate in the same way: by putting your product last.

Design is powerful.  Please use it responsibly