Friday, May 28, 2010

What to Consider Before Starting a Technology Project

I was recently advising someone how to begin a technology project. In my experience, there are four main considerations to examine before one determines an appropriate technology and implementation plan.

1) Determine practical constraints
I put this first as there are often practical factors that dictate what can and cannot be done. There's no point in wasting time with utopian planning when one has a shoestring budget.

So before beginning consider your:

  • budget
  • timeline and deadlines
  • available human resources (skill set of staff and their availability)
  • available technical resources (access to servers, database, hosting, etc.)
  • level of support needed post-launch
  • IT rules (there may be internal rules that require usage of certain software or against open-source or third-party code, etc.)
  • other business rules (e.g. corporate style guides, suitability of data being offshore, privacy, etc.)
After going through the other steps and getting some estimates you may want to or need to revise the budget or resources needed based on more solid details. Still initial limits here are necessary to refine the scope.

2) Formalize your applicable business goals
I'm constantly surprised by the number of websites that not only fail to meet the needs of the website (such as a functioning shopping cart) let alone meet the overall needs of the business. Before embarking on a project consider and prioritize the relevant business goals. All future choices and implementations (and budgets) flow from this. If something is not meeting the business goals then why do it? It really is okay to not do something - and this can be a better option than doing something poorly. Also no technology can meet all needs, so prioritizing your goals helps determine what is most needed and what can be sacrificed or postponed.

3) Know your users and would-be users
The profiles of your customers/clients/users can dictate your path as much as practical considerations. For example, if the bulk of your customers you wish to reach don't have highspeed access (it still happens) then streaming media probably doesn't make sense.

You should know the following about your intended audience:
  • platform (Windows or Mac operating system, version)
  • Browser used most
  • Plug-ins installed (most users will not install a special plug-in just for you, few organizations are important enough to motivate such action)
  • Internet access (highspeed or dial-up)
  • where they access the Internet from (work, home, laptop remotely, public library)
  • mobile devices (if have one and platform)
  • accessibility needs (e.g. visual or motor control issues)
  • education level (this can help determine literacy level which will can guide writing style)
  • technical literacy (how long have been using Internet, software, etc.)
  • age (not always relevant, but can relate to other factors, such as amount of leisure time, technical literacy, accessibility needs, etc.)

I didn't include gender as I have not seen any convincing studies that indicate women and men use technology differently. So I'm not sure it is relevant - other than possibly to guide visual design or writing style.

4) Plan the individual project goals
Goals can address high-level considerations such as if the plan is to test the waters or achieve parity with a competitor, to serve one market niche or another, to dovetail with other projects or create its own buzz, and so on. With theses goals in place, it then becomes easier to determine plans for launching, roll-out, promotion, timing, support, etc. For example, it can help determine whether a phased approach is desirable or a full launch. It also makes it easier to address the daily challenges and roadblocks that arise during implementation when you can refer back to the project's goals.

Once these four main considerations have been completely mapped out, it makes determining one's path much easier. It will help narrow down viable options and make implementation plans easier. In the end, it also helps deliver technology that needs of the business and users and will guide expectations.

Wednesday, May 19, 2010

Making Connections at Mesh Conference

Today is the fifth anniversary of Mesh, Canada’s web conference. After four years attending, I can spot perennial patterns. The location is the same – Mars, a centre for innovation in downtown Toronto. As I hoped, Red Bull is there again providing ample fuel for making it through the hours of listening. Rightsleeve had a new batch of fun and offbeat swag. Many of the attendees' faces are the same – as one of the conference speakers stated Mesh was integral to helping form the digital media and tech community in Toronto and clearly this community is loyal to Mesh. I have, however, also met more people this time from across Canada, which is great. The structure of sessions is the same with internationally-prominent keynote speakers in the mornings and panel discussions or workshops in the afternoon. The speakers represent a good cross-section of local and American or British people working in or studying the area.

The first session of the day was at the untenable hour of nine (insane for this type of conference). I did make it to the subsequent keynote by Joseph Menn. A great storyteller, Menn managed to shake me out of my complacency towards online security. According to Menn, over half of our computers are comprised by hackers (largely organized by the Russian mob). He advises not to ever bank online and to only shop at a few of the biggest online stores. He attributes this dire situation to foreign governments unwilling to crack down on cyber-crime, public apathy, and insufficient government and private-sector action. In his dystopian future, he predicts a return to the much-maligned walled garden approach of AOL as the only way to ensure security online. His prophecy particularly frightened me as my computer inexplicably started acting weird (really slow) a few weeks ago and I fear I have already joined the evil bot army.

The afternoon sessions presented a more hopeful vision of the positive role the Internet and mobile devices can play in media, culture, and society.

Well, maybe not that hopeful as the first panel session I attended on Government 2.0 revealed the potential that the Internet has and how other countries are achieving this already, but not Canada (see my prior post). There was a representative from the federal government who mentioned that the feds are using new media innovatively for internal projects. But when it comes to external projects they are more risk-averse and resistant to change. The developer of OpenParliament.ca, Michael Mulley, (a website highlighting and aggregating the statements and votes of Canadian members of parliament) acknowledged that in some ways it is easier for citizens such as himself to build these types of projects. Citizens don't have the same red-tape, policy handcuffs, or official scrutiny. However, in Mulley's experience the government was making it much harder than it needed to through their legendary bureaucracy and inertia. The session focused on open government, that is the sharing of data both to aid innovation and to increase transparency. I would have liked some discussion on how e-government can improve direct democracy, but they were enough hurdles to address to improve the sharing of information, let alone the co-creation.

The second panel session I attended was “The Effect of Real-Time on Content”. It focused on how Twitter and mobile devices are changing the role and our conceptions of journalism. With breaking news events, it is these media that are the first to publish and transmit the details. Now anyone in the midst of a news event – or frankly with an opinion – can become a prominent source for information, as happened with citizens covering events such as the Haiti earthquake or protests in Iran. A problem with this, identified by journalism professor Jay Rosen, is that we don’t have a reputation system that accompanies such citizen real-time coverage. So discerning the authenticity of the source and the credibility of the information becomes difficult. Jennifer Preston, a NY Times editor, noted that news organizations need to learn from technology companies and embrace the idea of iteration: post, add, revise…. A reporter, Joanna Smith, shared how she uses Twitter with and the real-time feedback from followers to guide her coverage of a story. In covering H1N1 a follower tweeted her an important consideration for a vulnerable group that Smith then fielded to a health official in her subsequent interview. This type of citizen-reporter collaboration represents to me the paragon of digital media - it uniquely combines the strengths of both the expanded perspective of citizens with the expertise of journalists to deliver better news.

My final session on “The Effect of Real-Time on Content” began as a commercial for an iPad (it worked, I’m dying to get one!) but then broadened to discuss the influence e-readers, tablets, and mobile devices will have upon news, books, and creative writing. Media consultant Jason Fry had insights into how user experience is still a limitation for online publishing. The problem with reading online compared to print magazines, he stated, is that magazines are not only more attractive and tactile, but they are finite - “With the Web you never actually finish, you just run out of time”. So he felt these new technologies might excel by offering finite writing forms. Fry also noted that the reason why iPhone applications were so much more popular than mobile-optimized webpages is due iPhone's dismal browser experience (which I can attest is also completely true of BlackBerry). This comment reminded me of a discussion I recently had on how some people are claiming these technologies are going to kill the print book and how if Dickens was alive he wouldn’t write a book, he’d build an app. Cynthia Good an academic with Humber’s Creative Book Publishing Program did suggest that these technologies might augur a return to old forms of storytelling such as the serial – or it might result in entirely new forms such as interactive, geospatially-responsive stories delivered via mobiles.

I keep going to Mesh as it is consistently delivers. My only disappointment of the day was with my BlackBerry. I was looking forward to conference backchannel conversations via Twitter. Last year tweeting during the conference was as much fun as passing notes in highschool was – and it offered extra insight and a way to actively participate. I couldn’t connect to either my cell network (Bell – no coverage in the basement rooms of Mars!) or Mars’ free Wi-Fi. I spent half an hour reading my BB manual trying to get it to even detect the Wi-Fi network to no avail. But the lack of tweeting did help me pay better attention, so I was better able to recall the many pearls of wisdom to recount here in such extended detail!

Monday, May 17, 2010

Participate in Shaping Canada's Digital Economy

Over a week ago Canada's federal government, led by Industry Ministry, released their Digital Economy Consultation to surprisingly little fanfare or resulting news coverage. Citizens have the next 52 days to offer their own ideas and/or vote on other ideas. Despite the opportunity to help guide our country's digital future, few Canadians are participating. The top items have no more than 55 votes.

Michael Geist's article Who's going to lead our digital strategy? provides more information and some of the key issues at stake. As well, read Hailey Eisen article for Backbone that covered the historical context leading to this effort.

I'm not sure why this isn't getting more coverage or participation. Is it cynicism? I am concerned that the government has not revealed how citizen contributions will be used. There is no word on any sort of binding nature of contributions (unlike the e-petition process in the U.K.) or how contributions are evaluated or even if they are heard at all Granted, some initial contributions are not all high calibre - some are repetitive, some are impractical, and some are beyond the scope of government. But I did read some good ideas and I'm sure there will be more to come.

I've jotted down below a few issues that I think are important to help improve our digital economy. Please disagree or let me know of others.

Ideas to Foster Canada's Digital Economy
  • help lower pricing for broadband Internet and mobile wireless access
  • improve access for remote, rural, and aboriginal areas
  • enforce net neutrality
  • aid the deployment of IPv6
  • help encourage Canadian companies to stay in Canada (as they inevitably seem to sell to American companies who immediately or gradually shut down Canadian operations)
  • foster innovation from unconventional sources - too many programs and events are directed at typical CEO types, despite that a lot, if not most, of the innovation in digital
  • media has not come from these types
  • mandate accessibility for disabled users
  • consider Canadian content restrictions online (see my online CanCon post)
  • funding for new content creation
  • continue to support CBC.ca - a digital media innovator
  • invest and help set-up post-secondary education in this area (i.e. no strong graduate Internet or mobile programs in Canada)
  • free e-learning on development and design topics (e.g. accessibility an example of how hard to do)
  • make this area a government department to ensure its continued importance and prominence
Although this focus is on Canada's economy, rather than government, I do think that Canada could spur innovation by modelling innovative use of e-government. The US, UK, Australia, New Zealand, and others are increasingly offering more forms of open, accessible, and participatory government through online means. Canada is falling behind. Showing Canadians and the world that we can be leaders in e-government would showcase and provide a roadmap for our digital future.

The project itself isn't a stellar example. The core functionality all works well, that is the ability to add an idea, comment, vote ideas up or down, and sort by most popular or recent. There is a lot of repetition that I'd like to see filtered out (acknowledging the technical and policy difficulties that would raise). There is the requisite video introduction by a government official but that's the extent of multimedia. A Twitter account and hashtag has been set-up (although both seem barely used), but that is the extent of social media integration or even promotion. I see zero signs of support for participating via mobile devices. A smartphone app would open up participation and be a great example of how participating in our digital future will look like.

Another problem is that there are not a lot of organizations in Canada that I'm aware of that are trumpeting these issues. OpenMedia is gearing up to lead the charge for net neutrality. From my past web accessibility research, I feel safe is saying no organization in Canada is effectively leading the charge for this issue.

I'm considering helping get an Internet Society chapter for Ontario going to make a formal submission. The government is also holding private consultations with the private sector and academia. So unless there are organizations representing the types of issues I raise above that I haven't heard of, if citizens don't participate to raise these issues and their profile, then who will?

Thursday, May 13, 2010

User Experience - More Than Just Usability

Lately, I’ve refined my research focus from online usability to user experience (UE). Yesterday, I told a colleague that I was studying user experience, to which she replied “You mean usability”. Her statement seems indicative of the prevailing thought on how we plan and evaluate online design.

Well, that’s not entirely true - as it still seems rather rare for many companies to adequately implement usability or to consider it at all. Hell, it seems many companies don’t even adequately consider their business goals for their websites or online applications.

Usability does seem to be considered more often nowadays. To begin with, however, there really isn’t a great, commonly-accepted definition of the term usability. I tend to use the International Standards Organization definition of usability as it appears to capture most of the agreed-upon elements and is frequently cited as a leading definition. ISO defines usability as “the extent to which the product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” (ISO 9241-11:1998.).

This definition is certainly not without contention. Olsen criticizes usability experts, such as guru Jakob Nielsen, who tend to focus on these factors while ignoring the larger human experience:

The problem is that it ignores the emotional subjective side of human beings, which as marketers and brand strategists have long known, is foolish to ignore. Why do we enjoy a good meal when nutritionally it's no different than hospital food? Unfortunately, Nielsen's pronouncements have all too often been like a restaurant critic insisting we should all eat only a McDonald's, since after all it's the most efficient restaurant around. (http://www.usabilitynews.com/news/article603.asp).

I believe it is fine to distinguish between usability’s focus on efficiency and the larger concept of user experience with its more holistic focus. Within user experience there are various factors that can apply at a macro level, for example an entire website or online application, or at the micro level, e.g. a specific online feature, piece of content, or tool. I will use the word product to encompass all such instances.

Various user experience factors include:
  • Affective – the emotions a product provokes, such as fun, anger, and frustration
  • Context – the physical and temporal aspects of the environment surrounding a given usage of a product
  • Hedonic – the ways in which a product results in pleasure
  • Social – how a product fits into a users social context, enables sharing and contributions from others
  • Value/usefulness – based on the costs (monetary, time, and other) of product usage, does it result in a sense of justifying the costs and does it achieve or surpass expectations
Accessibility – I often treat accessibility (ie. the ability for a product to address the needs of disabled users) as a usability issue. All users will at some time encounter difficulties using websites, however, so it does become important to distinguish between when a difficulty using an online product is at the individual or disability level. Shawn Henry describes this distinction: “When a person with a disability is at a disadvantage relative to a person without a disability, it is an accessibility issue” (http://www.uiaccess.com/accessucd/).

This is a rudimentary list and there is (growing) research that examines these issues in much more depth. However, this list is useful to begin considerations of a larger range of product issues beyond just usability. As without a sense of the various components of user experience, it is impossible to build applications that fully meet users needs or to even understand where an online application is delighting or failing its users.

Wednesday, May 05, 2010

Multipaper Dissertations - Considering an Alternative to Traditional Dissertations

I'm almost finished my first year of my PhD studies. I still haven't narrowed down my topic. I have concluded the format of my dissertation. I was considering the multipaper format that appears to be popular in computer science. I don't think it is the right option for me (or for my faculty).

In trying to decide if this alternative format would be appropriate and helpful, I found that there wasn't much written about this format. So here is background info I put together, which may help others in determining if this format is right for them.

What is a multipaper dissertation?
A traditional dissertation often relates the details of one large study (which may have various sub-components) and is normally comprised of an introduction, literature review, methodology, findings/results, discussion, and conclusion. An alternative is to publish a small number of papers in traditional journals and include them with a contextualization chapter or two to form a dissertation.

I encountered various names for this type of dissertation (some of which have shades of difference), such as multipaper, sandwich, alternative, stapled-papers, Scandinavian, paper-based, PhD by publication(s), thesis by publication, and PhD by published works. I prefer the term multipaper dissertation as it is suitably descriptive of its defining quality.

While not attempting to relate a definitive history of dissertations, a brief background is helpful to position the novel format. Our notion of the traditional doctoral degree and accompanying dissertation format was largely imported from Germany to Canada and the United States in the mid-nineteen century. This format has not changed significantly since that time. It appears the multipaper format was introduced into the United Kingdom and the United Statues in the 1960s and has grown slowly but progressively since then. I was unable to find any accounts of the format’s usage in Canada, but it is not uncommon to find it used in Scandinavia or Australia. In the UK and US, it is becoming increasingly used in the fields of chemistry, physics, biology, geology, computer science, finance, economics, marketing, and medicine

Components of a multipaper dissertation
Badley (2009) analyzed universities in the United Kingdom offering this format and the academic literature surrounding the topic; he found few commonalities and a lack of scholarship on the topic. Based on the literature I did find, there appears to be little consensus on a standard structure for a multipaper dissertation. According to Badley’s research, some UK universities do not require any documentation beyond the publications, while others may require a new document, a “critical appraisal”, be written to synthesize the collected works (p. 333, 2009).

Grant and Reed offer their take on what a multipaper dissertation should contain: an abstract, an introduction, an explanation or summary of the included papers, the published papers, a conclusion, and a literature review as an appendix (2006). Including the full contents of the published papers in the master document is not necessarily an obligatory component (although discussing them, no doubt, is). Multipaper dissertations do tend be shorter than the traditional format as it is felt that the published work alone is indicative of significant scholarship. Another variance is in the number of papers to include. Badley found no universities were explicit on the required number, but he offered that best practice indicates it should be not less than six and no more than ten.

Pros and cons of the multipaper format
The multipaper format paper has engendered criticism and disrepute arising largely from universities that allow students to submit any work published prior to enrollment. In the United Kingdom, the "PhD by publication" process was initially used to allow existing university faculty (for example, heads of laboratories, librarians, scientists) who had a distinguished record of publishing in academic journals to acquire a doctorate more easily. This practice remains despite criticisms of it being an insider perk. Further criticisms of this format arose upon some universities opening this option to any qualifying student (provoking derogatory remarks that anyone who pays tuition, qualifies).

Some universities also allow any pre-published or artistic work, including those neither peer-reviewed nor from an academic source. Starrs (2008) argues that evaluating the films of a director for a PhD in film studies makes sense, but allowing trade literature for any PhD program may not. Badley is further critical that accepting trade literature is more appropriate for professional doctorates than a PhD. It is this lack of traditional, advanced academic performance metrics that spurs criticisms that doctorates attained this way amount to degrees for purchase. Starrs equates degrees achieved in such as fashion as essentially honourary degrees, at times as dubious as the one the Long Island University awarded to Kermit the Frog. It is therefore important to distinguish between a dissertation comprised of peer-reviewed papers published in academic sources during one’s enrollment and a dissertation comprised of papers or work published before starting a PhD program.

The traditional dissertation format, however, would not have survived for such a long time if it did not offer strong advantages. The familiar format provides a standard template and shared experience for all doctoral students. This makes the doctoral process more consistent and understandable. It also provides training in delivering an in-depth piece of original research. This training is a singular opportunity in most academics’ careers that allows them to acquire the experience and evidence of mastering a topic and methods.

However, Duke and Beck believe traditional formats do not offer sufficient practice in writing the journal or trade articles that students will be expected to master upon graduation (1999). They posit that a traditional dissertation is a unique genre in both style and in readership. Dissertations are thus like a will; one only writes one once. Mastering this genre is consequently of minimal value. They also argue that the style and circumstances of a traditional dissertation are such that few are suitable for publishing or apt to be reworked into more than a couple publishable papers at best. Very few dissertations thus result in any publications. Duke and Beck therefore advocate for the multipaper format as it offers the opportunity for more real-life practice and increased readership.

The following additional benefits can be listed: it can accelerate a career by improving one’s curriculum vitae and connections with journal editors; it provokes comments from reviewers that can be invaluable and may come from scholars more knowledgeable on one’s specific domain than may otherwise be available; it encourages collaboration; it mitigates against poor results or arriving at a project failure too late in one’s dissertation process by focusing on smaller projects and getting progressive feedback through the publication process.

Multipaper dissertations have disadvantages: the process is slower due to publication timeframes, and if full publication is required this can amount to a substantial delay; one gets experience with several smaller projects but not the benefit of working on one big, in-depth project; peer review may be tougher and less open to discussion than one’s committee; and there may be a lack of experience at one’s university with this format that may provoke resistance or result in insufficient support.

To get around the time delays that academic publishing often entails, Duke and Beck suggest that a multipaper dissertation be allowed to include papers that are ready for consideration or in the process of consideration for publication (1999). Another way to combat publishing and feedback delays is to publish in conference proceedings. Further consideration should also be made to account for an individual student’s personality, circumstances, faculty, and topic that may – or may not – be conducive to such an approach.

Evaluation criteria
This nonstandard dissertation format introduces the problem of finding an evaluation criteria. Traditional quality metrics may not apply to such a format with profound differences in structure and goals.
The traditional format appears to have a general consensus on quality hallmarks. Garson is representative of the standard advice, stating that dissertations should start with framing an analytic question and offering a roadmap; have a robust, well-organized literature review; develop a model through operationalizing variables and a rigourous research design; present the case through clear writing and valid findings; and conclude by drawing inferences, making generalizations, and pointing ahead. More specialized advice for qualitative dissertations is offered by Piantanida and Garman (1999). They outline high-level qualities for optimal qualitative work, which are integrity (structural soundness), verité (rings true), rigour, utility, vitality, aesthetics, and ethics.

Unlike with the traditional format, there appears to be little guidance for multipaper formats; I was able to find only one source for any discussion of evaluation. Badley, in his overview of the status of the format in the UK does offer some quality benchmarks (2009). He believes such dissertations should strive for originality, rigour, significance, and coherence.

Badley acknowledges that these criteria are not necessarily unique, objective, or well-defined. He even questions whether a discussion of quality is necessary since the articles have already been peer-reviewed and accepted by journals. He notes that the multipaper format is evaluated more on process rather than the product.

Originality
Ascertaining a work’s originality within an entire subject is hard to assess for any work. With multipaper dissertations traditional ways to assess originality also applies, but a unique issue arises when the associated papers are jointly authored. This raises the question of individual contribution, which is rarely an issue for dissertations otherwise. In a traditional dissertation the work is assumed to be new work originating from the PhD student. With multipaper dissertations the master dissertation document would always be authored by an individual PhD student, but it is not uncommon for the associated publications to be comprised of jointly authored papers. This raises the question of how much work is done by the individual. Robins & Kanowski (2008) find this issue to be one of the main contentions with this format at many universities. They advise that universities should require a declaration from the PhD student regarding their level of contribution. They do not, however, advise how to determine sufficient individual effort. Universities accepting this format, it would seem, must be comfortable with collaborative rather than entirely individual effort.

Rigour
Again traditional evaluation criteria (e.g. appropriate sampling strategy, validity of findings, etc.) do also apply. Where the unique aspects of the multipaper format arises in regards to this criteria is in the length of reporting on methodology. Although methodology sections can be unsatisfactorily brief in traditional formats, in multipaper dissertations they are almost brief by definition. To further compound the problem are articles published in conferences proceedings as such articles are even shorter than conventional journals. This further reduces the opportunity to supply sufficient methodology details. Wilson encountered this situation when his university began accepting the format (2002). This posed a roadblock to committee members feeling they were able to adequately evaluate candidates' work. Wilson notes that this was rectified by requiring candidates to augment their dissertations with further methodology details.

Significance
As multipaper format dissertations are comprised of existing published works, evaluating significance can be done by examining the papers publishing records and feedback. This can be assessed based on impact factors, download totals, citation counts, prestige of publication source, reviews, and informal feedback. This data is useful to demonstrate that the work had importance and influence in the field – and also indicates that a dissertation was read by more than just a committee and a student’s spouse.

Coherence
Traditional dissertations can suffer from an inability to maintain a clear focus, but this problem is exacerbated when a dissertation is also comprised of multiple, possibly external, papers. To further compound the problem, the research could have been conducted through a series of separate projects conducted years apart. The goal of a dissertation is to demonstrate mastery of conducting research on a single topic, not to patch together a collection of assorted work that one has already conveniently published. A dissertation can explore various dimensions of a subject, but at some point trying to marry divergent works into one document becomes difficult. Cinderella’s wicked stepsister managed to get most of her foot in the glass slipper, but it clearly did not fit and it wasn’t pretty. Thus achieving coherence in a multipaper dissertation is an important, if difficult, undertaking.

PhD as process, not product
I heard a saying that the only good dissertation is a completed one. If dissertations are a journey then some see the destination as less important than how one got there. Badley notes that shares the journey analogy as it “should help candidates shift attention away from the traditional PhD emphasis on the research product – the thesis itself – towards the development of the autonomous scholar who is capable of undertaking further research journeys” (2009, p. 340).

Samples of multipaper format
The two dissertations below offer two different styles using this format. Harper uses the more conventional (of this unconventional format) approach of including his published papers in his dissertation with a framing introduction and conclusion. Roto, on the other hand, lists her published papers and then synthesizes from them to offer an entirely new paper in a conventional format.

Harper, F. (2009). The impact of social design on user contributions to online communities. University of Minnesota, Department of Computer Science and Engineering.

Roto, V. (2006). Web browsing on mobile phones – characteristics of user experience. Helsinki University of Technology, Department of Computer Science and Engineering.

Sources:
Badley, G. (2009). Publish and be doctor-rated: The PhD by published work. Quality Assurance in Education, 17(4), 331-342.

Duke, N. K., & Beck, S. W. (1999). Education should consider alternative formats for the dissertation. Educational Researcher, 28(3), 31-36.

Garson, G. D. (2001). Guide to writing empirical papers, theses, and dissertations. New York: M. Dekker.

Grant, D., & Reed, A. (2006, April 21). Multi-paper Dissertation. Retrieved April 23, 2010 from http://74.125.93.132/search?q=cache:hgjsmltiqo4j:saet.cs.depaul.edu/ multi- paper%2520dissertation%2520presentation%2520042106. ppt+multipaper+dissertation&cd=1&hl=en&ct=clnk&gl=ca

Piantanida, M., & Garman, D. N. (1999). The qualitative dissertation: A guide for students and faculty. Thousand Oaks, CA: Corwin Press.

Robins, L., & Kanowski, P. (2008). PhD by publication: A student's perspective. Journal of Research Practice, 4(3).

Starrs, B. (2008). Publish and graduate?: Earning a PhD by published papers in Australia. M/C Journal, 11(4).

Wilson, K. (2002). Quality assurance issues for a PhD by published work: A case study. Quality Assurance in Education, 10(2), 71-78.

Tuesday, May 04, 2010

Usability Evaluation Resources

Starting a usability initiative? There is great research on how to set up a useful and appropriate evaluation program. There are also websites with sample testing templates. Below is a list of resources I put together for a company beginning such a program. I think the resources listed below are helpful to anyone wanting to measure and improve the usability of their website, application, or online service.

Here are the resources in order of how to set up a usability program.

1) Prove the value of usability

Making the business case for usability:
* Bias, R. G., & Mayhew, D. J. (2005). Cost-Justifying Usability: An Update for the Internet Age, (2nd ed.). Morgan Kaufmann.

A sample report demonstrating a usability competitive analysis using individual and composite usability scores: Ecommerce Usability for High Street Retailers 2009

A study showing that lab usability testing can indeed uncover usability problems in the field.
* Marshall, C., McManus, B., & Prail, A. (1990). Usability of product X-lessons from a real product. Behaviour & Information Technology, 9(3), 243.

2) Measure the right things in the right way

Determine the appropriate usability metrics and how to best design a test to get at them. This book also has a few effective case studies on the value of usability testing:
* Tullis, T., & Albert, B. (2008). Measuring the user experience: Collecting, analyzing, and presenting usability metrics. Boston: Elsevier.

More info on how to determine appropriate metrics and the challenges of measuring them correctly:
* Hornbæk, K. (2006). Current practice in measuring usability: Challenges to usability studies and research. International Journal of Human-Computer Studies, 64(2), 79-102.
* Hughes, M. (1999). Rigor in usability testing. Technical Communication, 46 (4), 488–494.

Information on a common industry measure, the SUS score:
* Determining What Individual System Usability Scale Scores Mean: Adding an Adjective Rating Scale Journal of Usability Studies, Volume 4, Issue 3, May 2009, pp. 114-123

For a discussion on the problems of relying entirely on user self-reporting read usability guru Jakob Nielsen's First Rule of Usability? Don't Listen to Users

3) Set up your usability evaluation reporting

Templates for website usability evaluation.

Tailored for retail, but this sample demonstrates how a template can
be customized for a particular genre or website model.

User testing protocol with and applicable template.

Sample test script

4) Additional considerations

A useful article on the value of merely discovering usability problems or posing redesign consideration to developers. Also considers expert review vs. user testing:
* Hornbæk, K., & Frøkjær, E. (2005). Comparing usability problems and redesign proposals as input to practical systems development. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 391-400). Portland, Oregon, USA: ACM.

Can outside usability experts identify the intricacies of one's business domain or are internal specialists required?
* Følstad, A., Anda, B. C., & Sjøberg, D. I. (2010). The usability inspection performance of work-domain experts: An empirical study. Interacting with Computers, 22(2), 75-87.

Current limitations with usability studies and what is needed in the future:
* Lund, A. M. (2006). Post-modern usability. Journal of Usability Studies, 2(1), 1–6.

For insight on usability metrics and innovative measure (ie. social, hedonic, affective) see the chapters by Cockton and Lindgaard in:
* Law, E. L., Hvannberg, E. T., Cockton, G., Jeffries, R., & Wixon, D. (2007). Maturing Usability: Quality in Software, Interaction and Value. London: Springer.