Making an impact: Measuring web design effectiveness

Everyone wants to make a great website, and as mentioned in my previous article, a good start is to ensure aesthetic design and user research are intimately related.

Why do this? Well, very few sites fall over under heavy traffic or contain broken links. They all do what they need to do, they all 'work', and with all these 'working' sites there's a need for you to differentiate your site from the others.

The best way to do this is to be fun and easy to use, pleasant on the eye and to encourage return visits. The aesthetic/emotional reaction to a site has a massive influence on this - studies have shown that people have made a judgment call on the quality of your site less than half a second after laying eyes on it! Before any form of interaction with your site most people will have already drawn a conclusion about whether it's worth their time. This means the initial impression needs to be overwhelmingly positive. How do you do this? Through good aesthetic design.

There are several recognised techniques to assess the usability of a site - expert evaluation, usability testing, cognitive walkthrough, etc. You can literally track how usable a site is (so there's no unpleasant surprises when a site goes live). This is a lot harder with aesthetics - attempting to assess the aesthetic or emotional reaction to a site is a much more difficult process. Most people use their own judgement, but there are more scientific methods available.

What can we measure?

Emotional reactions to websites can be measured in 2 ways: physically or through personal accounts. Physical reactions include (among other things) pupil dilation, facial expressions or leaning towards or away from the screen. Personal accounts need participants to say, or report in a questionnaire how they feel/felt when using the site.

Example methods

There are several methods available that attempt to measure peoples reactions to a site. None are entirely reliable, but all will give you a clear idea of how participants really find your site:

• Mood after use - A questionnaire can be used to measure the mood of the participant after using the site. This is more reliable if the change in mood is measured (i.e. measure the participants mood before and after use). You know your site's hitting the right spots if the average mood takes a jump after use.
  
  However, there's a weakness in this method - how can you be sure any change in mood is related to your site, and not to something the participant was thinking about? In order to ensure your results are reliable you need to do many more tests than financially feasible.
• Facial expressions - As people use sites, they express themselves. They smile, they frown, they look confused. Simply define which moods you're looking for (ensure both positive and negative are covered) and keep a tally as people use the site. The more positive expressions (or at least lack of negative expressions) the better. The problem with this method is the risk of missing or misinterpreting people's facial expressions.
• Estimated time of use - A simple method to measure how engaged a participant was after using a site is to ask them to report the length of time they felt they used your site for, and compare this to how long they actually used it for. If participants overestimate the time they aren't enjoying themselves, and if participants underestimate the time they are. This method is reliant upon participants being unaware of the time throughout using the site, and may be affected by what they are asked to do - someone asked to do something dull may overestimate the time taken no matter how good the site is.
• SEQAM - A method particularly popular in the car design industry is called SEQAM (Sensory Quality Assessment Method). Participants are not shown the design as a whole. They are shown a series of alternative design parts and asked their opinions of each. Using this method means the participant cannot see the whole design, and testing needs a large number of these alternative parts.
• Perceived target market - This method requires participants to define the target market of the site. This will give you great insight into how participants perceive your site. Participants may struggle to frame what they see and feel into a “target market”. You will need to define areas (age, interests, etc) for participants to fill out that will guide them through the process.
• Opposing scale questionnaires - This method requires participants to fill in scale questionnaires between 2 extremes of feelings, e.g. “How would you rate your experience with the site? Give 1 for pleasant, 5 for unpleasant”. This method requires heavy use of statistical analysis to ensure the results are reliable - but limited conclusions can be drawn from the results without this. This method often works best when 2 or more sites are compared, as participants give much more varied scores.
• Direct feedback - Simply ask participants how they found the site after using it.
  This method is surprisingly unreliable, people adjust what they say according to how they think you want them to answer. There have been times when a participant is literally unable to do anything on a site, yet somehow decided it was worthy of a 9/10!

Conclusion

So which is the best method to use? Each method has its strengths and weaknesses. Remember that they are all attempting to measure something that cannot really be measured - feelings. Use 2 (or even 3) of the methods, and ensure you listen to any comments participants make during the testing itself.

Attempting to take both usability and aesthetic reactions into account during a site design is tough. They often conflict but if a balance can be found, a great site is the result.