Measures of User Experience

I was thinking today about my experience of advising clients wanting to do user testing of their new software or digital media. Typically, clients new to user testing don't know what they want to measure and once introduced to what can be measured they want to record it all.

Having conducted user testing sessions and analyzed the resulting data, I can assure you that more data is not necessarily (and even rarely) better. There are many measures and metrics that are irrelevant to your organization goals and the needs of your users.

Software has not been invented yet that can adequately record everything and there never seems enough people that can be hired to observe and record all that can be recorded. Besides, the sophisticated user testing software is expensive and having more than a couple observers during a user testing session often just makes the participant feel too much like a guinea pig (although a two way mirror to another room can help). And making someone analyze and report on quantitative data that is not necessary and will likely never be used is just a cruel and unusual  punishment.

My advice is to start with determining your goals and priorities first. Then review my list below of some common digital media metrics and measures to figure out which ones will most work for what you want to achieve. (If you are not familiar with the term - just google them as they are quite standard.)

Designers and developers are increasingly interested in measures related to how an application makes users feel. It is important when doing user testing to not think about everything in terms of efficiency. If a new website feature can be used quickly and easily, but makes us angry and never want to return - it has grandiosely failed at a primary goal. The difficulty is in measuring subjective phenomena is being sure that the operational definition used accurately captures the phenomenon. For instance, a user smiling during a test can mean that they are happy, but there are also perplexed smiles and polite smiles that people give to strangers for social niceties.

Usability or User Experience Measures

• Task completion rate (also failure rate)
• Task Completion time
• Path analysis
• Number of clicks to desired content
• Number of times user clicked "Help" or "Search"
• Number of times user asked facilitator for help
• Error rate
• Time spent on X (as a measure for "engagement")
• Most used feature
• Least used feature
• Outcome based (e.g. if goal is to learn X)

Affective, Satisfaction, and Hedonic Measures

• User reported
• Task satisfaction rate
• Application level satisfaction rate
• Favourite feature
• Least liked feature
• Feelings observed or reported of 
• Happy or pleased
• Frustrated
• Nostalgic
• Angry or agitated
• Sad
• Confused
• Social behaviours exhibited (e.g. number of times "shares" feature)

These are just a few of the various many measures and metrics that can be done and each one has its uses and inherit problems.  So heed my works about planning well before testing and really consider if the measure will give you meaningful, useful, and accurate data.

Let me know if I missed a particularly common or useful measure.