Fatal Error: Neglecting Error Handling

For a long time, I've heard great things about Airbnb. Friends have recommended it for making travelling to places, such as Europe or New York City, finally affordable and as a good way to meet locals.

I have been eager to try Airbnb, but haven't had the opportunity until earlier this month. I was travelling to Vancouver for a conference and was put off by the B grade hotels in B grade locales charging on average about $300 a night.

Upon checking Airbnb, I was delighted to find a few great listings. I narrowed it down to one that sounded perfect. I was a bit nervous that the place didn't have any feedback from prior guests, but I decided to go ahead and book it. I was excited because not only was I getting a good rate, but it was in a great location.

But due to fatal errors with Airbnb and their partners, I won't get the chance to book any rooms with them any time soon or perhaps ever. Their site has an error that blocks the process of booking and they offer no workaround.

As a former web developer, the issue doesn't seem like a huge problem. Yet it is a necessary step that without completing stops a traveller worst than a wicked case of Montezuma's Revenge.

In brief, the problem is that some Airbnb's room renters require a guest to be authenticated. This process requires uploading a photo of government-issued identification. Suffice to say, my wife and spent over an hour trying every permeation offerred to do this several times and it never worked. Airbnb offers no error message to inform me of any problems on their or my end. I had no idea what was going wrong, instead I'd just be redirected back to where I started with no word from Airbnb on how to proceed. No alternatives are offerred nor is any way to contact customer support. As booking lodgings is often time sensitive, access to very fast, preferably live, customer support is essential - another one of Airbnb's problems.  So despite finding a great room, I could not book it.

The problem is so prominent and fatal that it should not have been overlooked by the company. The fact that it was missed speaks to a lack of adequate quality assurance that throws their whole service into disrepute.

I don't mean to pick on Airbnb - but this case is a great example of a company that has done very well developing a great online service, designing an attractive and generally usable website, and achieving a critical mass of users to make the service viable.

They obviously put a lot of very skilled work into all those crucial components of any digital media business. The business analysts, developers, designers and marketers should be proud.  But the the company obviously hasn't done an adequate job of error handling and user testing.

Having worked in this field for many years, this is not surprising as testing is often neglected. In the rush to get products live quality assurance testing - let alone user testing - is frequently sacrificed. Sometimes it is not done at all.

Testing is not the most fun part of launching any digital media product (it might actually be the most boring and certainly the least sexy). But neglect it at your peril. All the hard work of launching a great product comes to a grinding halt when a simple bug is overlooked.

****
P.S. I ended up only be able to get an overpriced room in Vancouver's "entertainment district". It turned out to be next to strip clubs and peep shows and be smaller than most hotel bathrooms (and didn't include a bathroom either). Staying at a gleaming new condo in a trendy neighbourhood for less money would have been MUCH PREFERABLE!!!

Measures of User Experience

I was thinking today about my experience of advising clients wanting to do user testing of their new software or digital media. Typically, clients new to user testing don't know what they want to measure and once introduced to what can be measured they want to record it all.

Having conducted user testing sessions and analyzed the resulting data, I can assure you that more data is not necessarily (and even rarely) better. There are many measures and metrics that are irrelevant to your organization goals and the needs of your users.

Software has not been invented yet that can adequately record everything and there never seems enough people that can be hired to observe and record all that can be recorded. Besides, the sophisticated user testing software is expensive and having more than a couple observers during a user testing session often just makes the participant feel too much like a guinea pig (although a two way mirror to another room can help). And making someone analyze and report on quantitative data that is not necessary and will likely never be used is just a cruel and unusual  punishment.

My advice is to start with determining your goals and priorities first. Then review my list below of some common digital media metrics and measures to figure out which ones will most work for what you want to achieve. (If you are not familiar with the term - just google them as they are quite standard.)

Designers and developers are increasingly interested in measures related to how an application makes users feel. It is important when doing user testing to not think about everything in terms of efficiency. If a new website feature can be used quickly and easily, but makes us angry and never want to return - it has grandiosely failed at a primary goal. The difficulty is in measuring subjective phenomena is being sure that the operational definition used accurately captures the phenomenon. For instance, a user smiling during a test can mean that they are happy, but there are also perplexed smiles and polite smiles that people give to strangers for social niceties.

Usability or User Experience Measures

• Task completion rate (also failure rate)
• Task Completion time
• Path analysis
• Number of clicks to desired content
• Number of times user clicked "Help" or "Search"
• Number of times user asked facilitator for help
• Error rate
• Time spent on X (as a measure for "engagement")
• Most used feature
• Least used feature
• Outcome based (e.g. if goal is to learn X)

Affective, Satisfaction, and Hedonic Measures

• User reported
• Task satisfaction rate
• Application level satisfaction rate
• Favourite feature
• Least liked feature
• Feelings observed or reported of 
• Happy or pleased
• Frustrated
• Nostalgic
• Angry or agitated
• Sad
• Confused
• Social behaviours exhibited (e.g. number of times "shares" feature)

These are just a few of the various many measures and metrics that can be done and each one has its uses and inherit problems.  So heed my works about planning well before testing and really consider if the measure will give you meaningful, useful, and accurate data.

Let me know if I missed a particularly common or useful measure.

Defeating the Apps First Approach to Tablet Interface

Despite a recent CBC study which found that 82% of Canadians mostly use tablets at home, I still consider tablets a mobile device and thus under my purview.  I also take an avid interest in Canadian innovations in digital media. So I was really excited to attend a talk last week by James Wu of Kobo on "Rethinking Tablet User Experience".

Kobo is a world-leader in e-readers. They are no longer owned by Canadians, but are still based in Toronto. The event was sponsored by ToRCHI, a leading organization for organizing events of interest to people working in or studying Canada's digital media.

It's been awhile since I have actively designed digital user interfaces, but when I did I found it a constant struggle (against many forces) to put the user's needs and their perspective first. James Wu, director of user experience at Kobo, opened his talk by addressing this point head on - "technology sucks for most people". Technology should not be the focus of design. Instead, Wu explained, technology should be out of the way and let people quickly and easily perform the task or access the content they want.

Yet, the standard interface of tablets (and I'd add smartphones) is the app. This dominant tablet user experience paradigm "is defined by facilitating a user’s navigation into, out of and between apps". When Wu asked tablet users, however, what they value most about their device it wasn't the apps. It was content. Thus tablet user interfaces "are always at least one step removed from what users want". Wu adds that is apps that define how and where users can access and organize their content rather than the users themselves.

"The focus on apps has taken us away from what apps do for us" Wu empathetically cautioned.

To redress this situation, Wu and Kobo embarked on a new project with the launch of the Kobo Arc e-reader and its Tapestries interface (here's a review).

To achieve this, Wu identified three main goals:

1. Focus on users' content
2. Support organic curation
3. Help users find more content

The Tapestries interface builds upon advances in visual bookmarking (such as Pinterest) and recommendation engines. Tapestries allows one to "pin" content (whether e-books, websites, songs, movies, etc.) to one's homepage. Users can organize their content in collections ("tapestries") in various ways, for example by media type or topic. The best way to get a sense of it, however, is to see a demo, as Wu does below via YouTube (the part about Tapestries starts at the 50 second mark).

Based on above-mentioned goals, here's what Tapestries achieves:

Focus on User's Content
Wu aptly noted that people (other than perhaps the attendees at the talk or readers of this blog) do not care about the movie player on their devices. Rather they care about watching their movies. Moreover, they don't want to have to care (or even think about) the movie player. So it's important to "put content at users' fingertips" metaphorically and literally. Wu noted that even in such content as email or Facebook which are incredibly personal, every user has the same icon on their device homepage - but it doesn't have to be this way.

Support Organic Curation
Tablet apps force organization by media type or date, rather than by activity (e.g. a vacation), topic, or other parameters. Wu asserts "let people organize content how they want to and maintain personal relevance".

Help Users Find More Content
The "Discover" feature of Tapestries engendered a lot of interest with the ToRCHI crowd.  To me, the Discover bar seemed much like recommendation engines on book websites that offer recommendations for content based on other people with similar interests histories or one's personal profile or viewing history. Discover, differentiates itself by allowing users some control in the content that is suggested to them.

User-centred Design
Wu also discussed the benefits and challenges of user-centred design. Before starting the Tapestries project, Wu conducted formal and informal research with tablet users - sometimes as informally as watching over strangers shoulders while they use their device (I do this too - it works great.)

They also conducted rounds of user testing. They found that the interface was so new for people that it took time for people to get used to it. But once they did, the feedback has been positive.

This comment reminded me of a great article by Bill Buxton and Saul Greenberg called "Usability Evaluation Considered Harmful (Some of the Time) ", which I mentioned.

In the article, Buxton and Greenberg caution that when innovations are really new users may need time to learn, adapt, or become comfortable with the changes. Rigidly following the outcomes of user testing can result in killing new interfaces and quashing innovation. Buxton and Greenberg offer this caveat against this:

How can we create what could become culturally significant systems if we demand that the system be validated before a culture is formed around it?

The rest of the article is well worth reading for any company or developer trying something new.

I was greatly impressed at how Tapestries is doing something new and different. I don't have an e-reader (when I get a device I need one that is optimized for data entry as much as consumption) but it made me want to get one. Nonetheless, much of what Wu pointed out applies not only to tablets but to anyone designing or developing for smartphones or even desktops.

I'm looking forward to more such innovations from Kobo and I truly hope that their Japanese owners keep such an innovative team in Canada.

People Should Still Not Have to Think!

I attended a talk recently by usability expert Steve Krug. His book written in 2000 Don't Make Me Think helped convince me in the early days of my Internet career of the importance of usability and the need to study it. So I totally geeked out when I had the opportunity after all these years later to hear him speak in person. I grabbed his book from my shelf, next to my other treasured classics such as by Nielsen and Holzschlag, and hoped to get an autograph.

Krug was sponsored by University of Toronto's Association for Information Systems, who very kindly squeezed me into their full event at UofT's Faculty of Information. His talk addressed the continued need for usability testing and recent developments that make it easier than ever to do so.

Despite the passage of time since publication of his landmark book, Krug still asserts that too much digital media design is not user-friendly and consequently "if you're not usability testing you must be nuts".

Krug noted that in the past usability testing was difficult and expensive, so there could be an excuse to not do it. Usability tests were conducted in speciality labs that had the ability to record testers and had a private room separated by one-way glass to allow developers/designers to observe unobtrusively. The labs and test experts were very expensive. Most often, labs were offsite. It was also difficult to recruit testers as they needed to physically be in lab.

But advances over recent years have made it easy and inexpensive to do usability testing. Now screensharing technology and software that records software usage is cheap and easy to use. So usability testing sessions can be set up pretty much anywhere and then broadcast to development teams located anywhere with an Internet connection. Remote testing is also an attractive option, Krug suggested, as it makes recruiting testers much easier and not much essential info is lost in the process.

To demonstrate the ease of doing such a test, Krug organized a testing session on the spot. He tested the mobile application Clear.

As usability testing should be done not for "statistical validity but actionable insight" the power of his impromptu test was immediately apparent as the tester reached roadblocks in her usage. The tester was requested to express her thought processes out loud (i.e. think aloud protocol) as she used Clean and was able to clearly articulate her problem.

Within moments, the tester provided evidence of problems and direction for changes. It wasn't complicated, expensive, or time-consuming but the input gained would dramatically improve the application (and likely make them more money).

Krug six maxims for usability testing
I condensed the maxims as follows:

1. Do usability testing (with 3 people) every month.
2. Start testing earlier in a project than you think (e.g. test prototypes or competitors products).
3. Recruit loosely and grade on a curve (i.e. don't get so hung up about finding the ideal target user that you don't test as frequently).
4. Make usability testing a spectator sport (i.e. invite as many people from the team to observe testing sessions together as "usability testing is the ultimate way to resolve debates around design issues").
5. Prioritize findings - you'll uncover a lot of problems so identify the top three problem by participant.
6. Tweaks are better than redesign.

Tips on testing mobile apps & sites
I asked Krug for some tips on testing mobile applications or sites. First, he noted that one can share their screen usage of a mobile device just as easily as a website (as witnessed by the Clear test session) so special video cameras to record mobile device usage are not necessary.

I also asked how one can overcome the difficulties of testing a mobile app or site in the context of use, particularly when the context is important - as with location based services. Krug offerred three points:

1. How important is the context? Is it essential functionality? If not, testing in context may not be that crucial.
2. How realistic does the context need to be, that is can it be simulated?
3. Even if context is crucial, there is still tremendous value testing anywhere as problems will still be uncovered.

So the message is clear - just test. And do it frequently.

At the end of Krug's talk, I hesitantly took out my book to ask for his autograph. While waiting in line to talk to Krug, I noticed another person doing the same thing  - for the same reason. I'm clearly not the only person who has found his advice tremendously useful and influential.

Cool website user testing

I was a guinea pig this evening for user testing for a prominent news and investor website.

They actually paid me to do the test - even though I would gladly have done it for free.

User testing is a great idea to really get an accurate idea of how your users actually use your site. People often say they use a site one way but do something else - or they say they want X feature but in reality they never use it.

What made this testing unique was that they recorded my eyeball movements and click-throughs as I scanned various news and investing pages. So it wasn't what I said I would do, but more like what I would actually do.

The only problem was that the pages I click-through on didn't actually load so in reality I would have ended up on different pages and may have then gone somewhere else. I probably wouldn't have spent as long on the homepages and landing pages as the study encouraged.

At the end of the study, they showed me my results. My web visit usage was way more erratic and frenetic than I imagined. While the standard pattern, the researcher and other studies I have read, for our culture is left to right and up and down. I was surprised by how much back-tracking I did. I did manage to deftly avoid all banners, fluff and navs. (Hope these studies don't eventually prove that there is actually no point in banner graphics and kill a lot of necessary advertising revenue.)

And yes, I sheepishly admit I "clicked" on the legitimate news item with the word porn in it. And the only ad I looked at all was the Telus adds with those irresistibly adorable monkeys.

I wasn't familiar with eye-tracking studies that were so conclusive - so I'm excited to see the results.

The research was conducted by Sensory Logic. I'm looking forward to seeing more from them - even if I don't get paid for it.