I'm almost finished my first year of my PhD studies. I still haven't narrowed down my topic. I have concluded the format of my dissertation. I was considering the multipaper format that appears to be popular in computer science. I don't think it is the right option for me (or for my faculty).
In trying to decide if this alternative format would be appropriate and helpful, I found that there wasn't much written about this format. So here is background info I put together, which may help others in determining if this format is right for them.
What is a multipaper dissertation?
A traditional dissertation often relates the details of one large study (which may have various sub-components) and is normally comprised of an introduction, literature review, methodology, findings/results, discussion, and conclusion. An alternative is to publish a small number of papers in traditional journals and include them with a contextualization chapter or two to form a dissertation.
I encountered various names for this type of dissertation (some of which have shades of difference), such as multipaper, sandwich, alternative, stapled-papers, Scandinavian, paper-based, PhD by publication(s), thesis by publication, and PhD by published works. I prefer the term multipaper dissertation as it is suitably descriptive of its defining quality.
While not attempting to relate a definitive history of dissertations, a brief background is helpful to position the novel format. Our notion of the traditional doctoral degree and accompanying dissertation format was largely imported from Germany to Canada and the United States in the mid-nineteen century. This format has not changed significantly since that time. It appears the multipaper format was introduced into the United Kingdom and the United Statues in the 1960s and has grown slowly but progressively since then. I was unable to find any accounts of the format’s usage in Canada, but it is not uncommon to find it used in Scandinavia or Australia. In the UK and US, it is becoming increasingly used in the fields of chemistry, physics, biology, geology, computer science, finance, economics, marketing, and medicine
Components of a multipaper dissertation
Badley (2009) analyzed universities in the United Kingdom offering this format and the academic literature surrounding the topic; he found few commonalities and a lack of scholarship on the topic. Based on the literature I did find, there appears to be little consensus on a standard structure for a multipaper dissertation. According to Badley’s research, some UK universities do not require any documentation beyond the publications, while others may require a new document, a “critical appraisal”, be written to synthesize the collected works (p. 333, 2009).
Grant and Reed offer their take on what a multipaper dissertation should contain: an abstract, an introduction, an explanation or summary of the included papers, the published papers, a conclusion, and a literature review as an appendix (2006). Including the full contents of the published papers in the master document is not necessarily an obligatory component (although discussing them, no doubt, is). Multipaper dissertations do tend be shorter than the traditional format as it is felt that the published work alone is indicative of significant scholarship. Another variance is in the number of papers to include. Badley found no universities were explicit on the required number, but he offered that best practice indicates it should be not less than six and no more than ten.
Pros and cons of the multipaper format
The multipaper format paper has engendered criticism and disrepute arising largely from universities that allow students to submit any work published prior to enrollment. In the United Kingdom, the "PhD by publication" process was initially used to allow existing university faculty (for example, heads of laboratories, librarians, scientists) who had a distinguished record of publishing in academic journals to acquire a doctorate more easily. This practice remains despite criticisms of it being an insider perk. Further criticisms of this format arose upon some universities opening this option to any qualifying student (provoking derogatory remarks that anyone who pays tuition, qualifies).
Some universities also allow any pre-published or artistic work, including those neither peer-reviewed nor from an academic source. Starrs (2008) argues that evaluating the films of a director for a PhD in film studies makes sense, but allowing trade literature for any PhD program may not. Badley is further critical that accepting trade literature is more appropriate for professional doctorates than a PhD. It is this lack of traditional, advanced academic performance metrics that spurs criticisms that doctorates attained this way amount to degrees for purchase. Starrs equates degrees achieved in such as fashion as essentially honourary degrees, at times as dubious as the one the Long Island University awarded to Kermit the Frog. It is therefore important to distinguish between a dissertation comprised of peer-reviewed papers published in academic sources during one’s enrollment and a dissertation comprised of papers or work published before starting a PhD program.
The traditional dissertation format, however, would not have survived for such a long time if it did not offer strong advantages. The familiar format provides a standard template and shared experience for all doctoral students. This makes the doctoral process more consistent and understandable. It also provides training in delivering an in-depth piece of original research. This training is a singular opportunity in most academics’ careers that allows them to acquire the experience and evidence of mastering a topic and methods.
However, Duke and Beck believe traditional formats do not offer sufficient practice in writing the journal or trade articles that students will be expected to master upon graduation (1999). They posit that a traditional dissertation is a unique genre in both style and in readership. Dissertations are thus like a will; one only writes one once. Mastering this genre is consequently of minimal value. They also argue that the style and circumstances of a traditional dissertation are such that few are suitable for publishing or apt to be reworked into more than a couple publishable papers at best. Very few dissertations thus result in any publications. Duke and Beck therefore advocate for the multipaper format as it offers the opportunity for more real-life practice and increased readership.
The following additional benefits can be listed: it can accelerate a career by improving one’s curriculum vitae and connections with journal editors; it provokes comments from reviewers that can be invaluable and may come from scholars more knowledgeable on one’s specific domain than may otherwise be available; it encourages collaboration; it mitigates against poor results or arriving at a project failure too late in one’s dissertation process by focusing on smaller projects and getting progressive feedback through the publication process.
Multipaper dissertations have disadvantages: the process is slower due to publication timeframes, and if full publication is required this can amount to a substantial delay; one gets experience with several smaller projects but not the benefit of working on one big, in-depth project; peer review may be tougher and less open to discussion than one’s committee; and there may be a lack of experience at one’s university with this format that may provoke resistance or result in insufficient support.
To get around the time delays that academic publishing often entails, Duke and Beck suggest that a multipaper dissertation be allowed to include papers that are ready for consideration or in the process of consideration for publication (1999). Another way to combat publishing and feedback delays is to publish in conference proceedings. Further consideration should also be made to account for an individual student’s personality, circumstances, faculty, and topic that may – or may not – be conducive to such an approach.
Evaluation criteria
This nonstandard dissertation format introduces the problem of finding an evaluation criteria. Traditional quality metrics may not apply to such a format with profound differences in structure and goals.
The traditional format appears to have a general consensus on quality hallmarks. Garson is representative of the standard advice, stating that dissertations should start with framing an analytic question and offering a roadmap; have a robust, well-organized literature review; develop a model through operationalizing variables and a rigourous research design; present the case through clear writing and valid findings; and conclude by drawing inferences, making generalizations, and pointing ahead. More specialized advice for qualitative dissertations is offered by Piantanida and Garman (1999). They outline high-level qualities for optimal qualitative work, which are integrity (structural soundness), verité (rings true), rigour, utility, vitality, aesthetics, and ethics.
Unlike with the traditional format, there appears to be little guidance for multipaper formats; I was able to find only one source for any discussion of evaluation. Badley, in his overview of the status of the format in the UK does offer some quality benchmarks (2009). He believes such dissertations should strive for originality, rigour, significance, and coherence.
Badley acknowledges that these criteria are not necessarily unique, objective, or well-defined. He even questions whether a discussion of quality is necessary since the articles have already been peer-reviewed and accepted by journals. He notes that the multipaper format is evaluated more on process rather than the product.
Originality
Ascertaining a work’s originality within an entire subject is hard to assess for any work. With multipaper dissertations traditional ways to assess originality also applies, but a unique issue arises when the associated papers are jointly authored. This raises the question of individual contribution, which is rarely an issue for dissertations otherwise. In a traditional dissertation the work is assumed to be new work originating from the PhD student. With multipaper dissertations the master dissertation document would always be authored by an individual PhD student, but it is not uncommon for the associated publications to be comprised of jointly authored papers. This raises the question of how much work is done by the individual. Robins & Kanowski (2008) find this issue to be one of the main contentions with this format at many universities. They advise that universities should require a declaration from the PhD student regarding their level of contribution. They do not, however, advise how to determine sufficient individual effort. Universities accepting this format, it would seem, must be comfortable with collaborative rather than entirely individual effort.
Rigour
Again traditional evaluation criteria (e.g. appropriate sampling strategy, validity of findings, etc.) do also apply. Where the unique aspects of the multipaper format arises in regards to this criteria is in the length of reporting on methodology. Although methodology sections can be unsatisfactorily brief in traditional formats, in multipaper dissertations they are almost brief by definition. To further compound the problem are articles published in conferences proceedings as such articles are even shorter than conventional journals. This further reduces the opportunity to supply sufficient methodology details. Wilson encountered this situation when his university began accepting the format (2002). This posed a roadblock to committee members feeling they were able to adequately evaluate candidates' work. Wilson notes that this was rectified by requiring candidates to augment their dissertations with further methodology details.
Significance
As multipaper format dissertations are comprised of existing published works, evaluating significance can be done by examining the papers publishing records and feedback. This can be assessed based on impact factors, download totals, citation counts, prestige of publication source, reviews, and informal feedback. This data is useful to demonstrate that the work had importance and influence in the field – and also indicates that a dissertation was read by more than just a committee and a student’s spouse.
Coherence
Traditional dissertations can suffer from an inability to maintain a clear focus, but this problem is exacerbated when a dissertation is also comprised of multiple, possibly external, papers. To further compound the problem, the research could have been conducted through a series of separate projects conducted years apart. The goal of a dissertation is to demonstrate mastery of conducting research on a single topic, not to patch together a collection of assorted work that one has already conveniently published. A dissertation can explore various dimensions of a subject, but at some point trying to marry divergent works into one document becomes difficult. Cinderella’s wicked stepsister managed to get most of her foot in the glass slipper, but it clearly did not fit and it wasn’t pretty. Thus achieving coherence in a multipaper dissertation is an important, if difficult, undertaking.
PhD as process, not product
I heard a saying that the only good dissertation is a completed one. If dissertations are a journey then some see the destination as less important than how one got there. Badley notes that shares the journey analogy as it “should help candidates shift attention away from the traditional PhD emphasis on the research product – the thesis itself – towards the development of the autonomous scholar who is capable of undertaking further research journeys” (2009, p. 340).
Samples of multipaper format
The two dissertations below offer two different styles using this format. Harper uses the more conventional (of this unconventional format) approach of including his published papers in his dissertation with a framing introduction and conclusion. Roto, on the other hand, lists her published papers and then synthesizes from them to offer an entirely new paper in a conventional format.
Harper, F. (2009). The impact of social design on user contributions to online communities. University of Minnesota, Department of Computer Science and Engineering.
Roto, V. (2006). Web browsing on mobile phones – characteristics of user experience. Helsinki University of Technology, Department of Computer Science and Engineering.
Sources:
Badley, G. (2009). Publish and be doctor-rated: The PhD by published work. Quality Assurance in Education, 17(4), 331-342.
Duke, N. K., & Beck, S. W. (1999). Education should consider alternative formats for the dissertation. Educational Researcher, 28(3), 31-36.
Garson, G. D. (2001). Guide to writing empirical papers, theses, and dissertations. New York: M. Dekker.
Grant, D., & Reed, A. (2006, April 21). Multi-paper Dissertation. Retrieved April 23, 2010 from http://74.125.93.132/search?q=cache:hgjsmltiqo4j:saet.cs.depaul.edu/ multi- paper%2520dissertation%2520presentation%2520042106. ppt+multipaper+dissertation&cd=1&hl=en&ct=clnk&gl=ca
Piantanida, M., & Garman, D. N. (1999). The qualitative dissertation: A guide for students and faculty. Thousand Oaks, CA: Corwin Press.
Robins, L., & Kanowski, P. (2008). PhD by publication: A student's perspective. Journal of Research Practice, 4(3).
Starrs, B. (2008). Publish and graduate?: Earning a PhD by published papers in Australia. M/C Journal, 11(4).
Wilson, K. (2002). Quality assurance issues for a PhD by published work: A case study. Quality Assurance in Education, 10(2), 71-78.
Wednesday, May 05, 2010
Tuesday, May 04, 2010
Usability Evaluation Resources
Starting a usability initiative? There is great research on how to set up a useful and appropriate evaluation program. There are also websites with sample testing templates. Below is a list of resources I put together for a company beginning such a program. I think the resources listed below are helpful to anyone wanting to measure and improve the usability of their website, application, or online service.
Here are the resources in order of how to set up a usability program.
1) Prove the value of usability
Making the business case for usability:
* Bias, R. G., & Mayhew, D. J. (2005). Cost-Justifying Usability: An Update for the Internet Age, (2nd ed.). Morgan Kaufmann.
A sample report demonstrating a usability competitive analysis using individual and composite usability scores: Ecommerce Usability for High Street Retailers 2009
A study showing that lab usability testing can indeed uncover usability problems in the field.
* Marshall, C., McManus, B., & Prail, A. (1990). Usability of product X-lessons from a real product. Behaviour & Information Technology, 9(3), 243.
2) Measure the right things in the right way
Determine the appropriate usability metrics and how to best design a test to get at them. This book also has a few effective case studies on the value of usability testing:
* Tullis, T., & Albert, B. (2008). Measuring the user experience: Collecting, analyzing, and presenting usability metrics. Boston: Elsevier.
More info on how to determine appropriate metrics and the challenges of measuring them correctly:
* Hornbæk, K. (2006). Current practice in measuring usability: Challenges to usability studies and research. International Journal of Human-Computer Studies, 64(2), 79-102.
* Hughes, M. (1999). Rigor in usability testing. Technical Communication, 46 (4), 488–494.
Information on a common industry measure, the SUS score:
* Determining What Individual System Usability Scale Scores Mean: Adding an Adjective Rating Scale Journal of Usability Studies, Volume 4, Issue 3, May 2009, pp. 114-123
For a discussion on the problems of relying entirely on user self-reporting read usability guru Jakob Nielsen's First Rule of Usability? Don't Listen to Users
3) Set up your usability evaluation reporting
Templates for website usability evaluation.
Tailored for retail, but this sample demonstrates how a template can
be customized for a particular genre or website model.
User testing protocol with and applicable template.
Sample test script
4) Additional considerations
A useful article on the value of merely discovering usability problems or posing redesign consideration to developers. Also considers expert review vs. user testing:
* Hornbæk, K., & Frøkjær, E. (2005). Comparing usability problems and redesign proposals as input to practical systems development. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 391-400). Portland, Oregon, USA: ACM.
Can outside usability experts identify the intricacies of one's business domain or are internal specialists required?
* Følstad, A., Anda, B. C., & Sjøberg, D. I. (2010). The usability inspection performance of work-domain experts: An empirical study. Interacting with Computers, 22(2), 75-87.
Current limitations with usability studies and what is needed in the future:
* Lund, A. M. (2006). Post-modern usability. Journal of Usability Studies, 2(1), 1–6.
For insight on usability metrics and innovative measure (ie. social, hedonic, affective) see the chapters by Cockton and Lindgaard in:
* Law, E. L., Hvannberg, E. T., Cockton, G., Jeffries, R., & Wixon, D. (2007). Maturing Usability: Quality in Software, Interaction and Value. London: Springer.
Here are the resources in order of how to set up a usability program.
1) Prove the value of usability
Making the business case for usability:
* Bias, R. G., & Mayhew, D. J. (2005). Cost-Justifying Usability: An Update for the Internet Age, (2nd ed.). Morgan Kaufmann.
A sample report demonstrating a usability competitive analysis using individual and composite usability scores: Ecommerce Usability for High Street Retailers 2009
A study showing that lab usability testing can indeed uncover usability problems in the field.
* Marshall, C., McManus, B., & Prail, A. (1990). Usability of product X-lessons from a real product. Behaviour & Information Technology, 9(3), 243.
2) Measure the right things in the right way
Determine the appropriate usability metrics and how to best design a test to get at them. This book also has a few effective case studies on the value of usability testing:
* Tullis, T., & Albert, B. (2008). Measuring the user experience: Collecting, analyzing, and presenting usability metrics. Boston: Elsevier.
More info on how to determine appropriate metrics and the challenges of measuring them correctly:
* Hornbæk, K. (2006). Current practice in measuring usability: Challenges to usability studies and research. International Journal of Human-Computer Studies, 64(2), 79-102.
* Hughes, M. (1999). Rigor in usability testing. Technical Communication, 46 (4), 488–494.
Information on a common industry measure, the SUS score:
* Determining What Individual System Usability Scale Scores Mean: Adding an Adjective Rating Scale Journal of Usability Studies, Volume 4, Issue 3, May 2009, pp. 114-123
For a discussion on the problems of relying entirely on user self-reporting read usability guru Jakob Nielsen's First Rule of Usability? Don't Listen to Users
3) Set up your usability evaluation reporting
Templates for website usability evaluation.
Tailored for retail, but this sample demonstrates how a template can
be customized for a particular genre or website model.
User testing protocol with and applicable template.
Sample test script
4) Additional considerations
A useful article on the value of merely discovering usability problems or posing redesign consideration to developers. Also considers expert review vs. user testing:
* Hornbæk, K., & Frøkjær, E. (2005). Comparing usability problems and redesign proposals as input to practical systems development. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 391-400). Portland, Oregon, USA: ACM.
Can outside usability experts identify the intricacies of one's business domain or are internal specialists required?
* Følstad, A., Anda, B. C., & Sjøberg, D. I. (2010). The usability inspection performance of work-domain experts: An empirical study. Interacting with Computers, 22(2), 75-87.
Current limitations with usability studies and what is needed in the future:
* Lund, A. M. (2006). Post-modern usability. Journal of Usability Studies, 2(1), 1–6.
For insight on usability metrics and innovative measure (ie. social, hedonic, affective) see the chapters by Cockton and Lindgaard in:
* Law, E. L., Hvannberg, E. T., Cockton, G., Jeffries, R., & Wixon, D. (2007). Maturing Usability: Quality in Software, Interaction and Value. London: Springer.
Wednesday, April 28, 2010
Have e-Readers Arrived?
Last month, I tried my first e-Reader. I've read e-books on my PC for a few years, but hadn't tried a portable e-Reader until last month I took a Kindle for a test drive. Reading my comrade Bargainista's blog, she just got her hands and eyes on the latest e-Reader to hit Canada, the Kobo. Check out her blog post for an excellent review of Kobo.
I replied to her post about my first experience with portable e-Readers, which I included below. I'd love to know if anyone feels that my concerns have been resolved with other or newer tech, or any other thoughts about e-Readers.
I'd love to try a Kobo to see if the e-reader technology has improved. I tried a Kindle last month - granted it was a year old (so newer models might be better) - and I was really underwhelmed. I like to read with really high contrast, ie. black type on white is ideal. With the Kindle is was like black on grey. Also, the font size seemed to arbitrarily set a limit that was still not sufficient for people with visual disabilities. Trying to find a range of books (or any) in large print is next to impossible - so I thought these e-book reader companies would capitalize on that (literally - there's definitely money to made from people who need this feature).
The user experience was otherwise okay. I like the ability to add notes as I have never been able to bring myself to vandalize my own books by scribbling notes in the margins or highlighting.
I'm certainly running out of room in our house to store print books. As it is all pulp fiction we buy are given to Goodwill after reading as we have no room to store them. Now with my daughter starting to read, we're buying her lots of children's books. I can imagine it will be a long time (who knows, it'll probably be just five years) before e-Readers can offer full colour as the artwork in young kid's books is probably the most important criteria. Not to mention with comic books. Still, I'd love to have all the books we love without them taking up so much space.
But price is the big stumbling block for me (once readability improves, that is). E-books are roughly the same price or maybe $1 or $2 cheaper. But with print I have something I can share with friends and don't have to worry about the format becoming obsolete (like all our VHS movies).
Most importantly, considering my propensity to spill coffee, drop and lose thing, my accidents are not such a big deal when it's a $10 book vs. $150 e-reader.
I don't want to sound like a luddite curmudgeon or print fetishist, more like a poor-sighted cheapskate.
I replied to her post about my first experience with portable e-Readers, which I included below. I'd love to know if anyone feels that my concerns have been resolved with other or newer tech, or any other thoughts about e-Readers.
I'd love to try a Kobo to see if the e-reader technology has improved. I tried a Kindle last month - granted it was a year old (so newer models might be better) - and I was really underwhelmed. I like to read with really high contrast, ie. black type on white is ideal. With the Kindle is was like black on grey. Also, the font size seemed to arbitrarily set a limit that was still not sufficient for people with visual disabilities. Trying to find a range of books (or any) in large print is next to impossible - so I thought these e-book reader companies would capitalize on that (literally - there's definitely money to made from people who need this feature).
The user experience was otherwise okay. I like the ability to add notes as I have never been able to bring myself to vandalize my own books by scribbling notes in the margins or highlighting.
I'm certainly running out of room in our house to store print books. As it is all pulp fiction we buy are given to Goodwill after reading as we have no room to store them. Now with my daughter starting to read, we're buying her lots of children's books. I can imagine it will be a long time (who knows, it'll probably be just five years) before e-Readers can offer full colour as the artwork in young kid's books is probably the most important criteria. Not to mention with comic books. Still, I'd love to have all the books we love without them taking up so much space.
But price is the big stumbling block for me (once readability improves, that is). E-books are roughly the same price or maybe $1 or $2 cheaper. But with print I have something I can share with friends and don't have to worry about the format becoming obsolete (like all our VHS movies).
Most importantly, considering my propensity to spill coffee, drop and lose thing, my accidents are not such a big deal when it's a $10 book vs. $150 e-reader.
I don't want to sound like a luddite curmudgeon or print fetishist, more like a poor-sighted cheapskate.
Monday, April 12, 2010
Facebook and the Problem of Collapsed Identity
Privacy controls in social networking sites seems to be endlessly discussed. Facebook seems to regularly tinker with its user privacy controls and many users routinely ignore them.
One thing that I feel is missed in privacy discussions is that it is not a binary conception - either I’ll keep things just to me, my friends or network or I’ll share it. Such decisions are a foundational privacy issue. However, the information I want to share is often more complicated. It’s not a question of whether a user is an online recluse or virtual exhibitionist.
There are some types of information that is suitable to share in some contexts and with some types of people and then not with others. The Internet collapses barriers that otherwise kept various aspects of our identity separate. There used to be a clear distinction between work, home, political or religious spaces, etc. These barriers are being further collapsed online as Facebook increasingly becomes the defacto platform for all online social networking. Almost everyone I know is on Facebook - from my young and old relatives, work colleagues, old school friends, casual acquaintances, etc.
The benefits of sites like Facebook are that everyone is using it (critical mass) but the problem is that everyone is using it. Problems have been found with this particularly when work and social life collide online (e.g. cases of employees fired for criticizing employers, prospective employers not hiring due to seeing drunken photos, parents learning TMI about their kids’ leisure activities).
I love the term participatory surveillance as it encompasses the desire many of us have to share our personal details, stories, images, and mundane status updates online. The accompanying term should be induced voyeurism, as it is hard to not take note as this parade of info passes by on our social networks sites. But just because someone wants to share information, doesn’t mean they want to share it with everyone they know (and don’t know) online and have friended.
Facebook is gradually improving its privacy controls, but they haven’t made much progress on allowing users to segment the various dimensions of their identity. I have made groups in Facebooks for my friends based on whether I know them from work, school, family, professional associations, or are essentially strangers (friended as they share similar interests or friends). Other than restricting the photographs I upload based on these groups, there is little else I can do to recontextualize the various elements of my social life.
Could website features or user norms mitigate collapsed identities? Could any such features be sufficiently usable so users actually use them? Already privacy controls are often seldom used, would this just be further distractions?
The rich context available offline and spatial barriers preserved distinct identities. This will be difficult to replicate online, so from a longitude perspective will user behaviour adapt. Will collapsed identities become the norm?
One thing that I feel is missed in privacy discussions is that it is not a binary conception - either I’ll keep things just to me, my friends or network or I’ll share it. Such decisions are a foundational privacy issue. However, the information I want to share is often more complicated. It’s not a question of whether a user is an online recluse or virtual exhibitionist.
There are some types of information that is suitable to share in some contexts and with some types of people and then not with others. The Internet collapses barriers that otherwise kept various aspects of our identity separate. There used to be a clear distinction between work, home, political or religious spaces, etc. These barriers are being further collapsed online as Facebook increasingly becomes the defacto platform for all online social networking. Almost everyone I know is on Facebook - from my young and old relatives, work colleagues, old school friends, casual acquaintances, etc.
The benefits of sites like Facebook are that everyone is using it (critical mass) but the problem is that everyone is using it. Problems have been found with this particularly when work and social life collide online (e.g. cases of employees fired for criticizing employers, prospective employers not hiring due to seeing drunken photos, parents learning TMI about their kids’ leisure activities).
I love the term participatory surveillance as it encompasses the desire many of us have to share our personal details, stories, images, and mundane status updates online. The accompanying term should be induced voyeurism, as it is hard to not take note as this parade of info passes by on our social networks sites. But just because someone wants to share information, doesn’t mean they want to share it with everyone they know (and don’t know) online and have friended.
Facebook is gradually improving its privacy controls, but they haven’t made much progress on allowing users to segment the various dimensions of their identity. I have made groups in Facebooks for my friends based on whether I know them from work, school, family, professional associations, or are essentially strangers (friended as they share similar interests or friends). Other than restricting the photographs I upload based on these groups, there is little else I can do to recontextualize the various elements of my social life.
Could website features or user norms mitigate collapsed identities? Could any such features be sufficiently usable so users actually use them? Already privacy controls are often seldom used, would this just be further distractions?
The rich context available offline and spatial barriers preserved distinct identities. This will be difficult to replicate online, so from a longitude perspective will user behaviour adapt. Will collapsed identities become the norm?
Friday, March 26, 2010
Exemplary Antagonism
There is no established formula for determining a great work of academic research. However, Wheeler, Richter, & Sahadevan (2004) attempted to uncover the ingredients of great research by interviewing leading researchers in their field of organizational behavior and human resource management. They selected nine journal articles they personally felt were great works. No other details about their sampling strategy were provided, although the total number of citations was a key determinant. The sample of their great works begins with an article that received 124 citations (as provided by Thomson Reuters’ Social Sciences Citations Index). A study contemporaneous to that one is by Latham, Erez, & Locke (1988) which received a similar number of citations in SSCI at 110. Does this then also qualify it as a great work?
Wheeler, Richter, & Sahadevan asked the ten authors of their "great" journal articles why they felt their work were so well received. Analyzing the compiled interview data, they found researchers credit four central factors to their success: "1) timing of the article or addressing a need in the literature; 2) defining or clarifying constructs in the article; 3) focusing on multiple levels of the organization; and 4) becoming a foundation article" (p. 1328). Wheeler, Richter, & Sahadevan’s criteria allows one to move beyond the citation counts of articles and impact factors of journals to examine the qualities of exemplary research, particularly within the domain of organizational behaviour. Applying this framework to the Latham, Erez, & Locke article assists in illuminating the work’s key strengths that make it an exemplary work.
Background
Latham, Erez, & Locke’s article, entitled "Resolving scientific disputes by the joint design of crucial experiments by the antagonists: Application to the Erez-Latham disputes regarding participation in goal setting", was originally published in the prestigious Journal of Applied Psychology in 1988 and then included in the edited volume Doing Exemplary Research by Frost & Stablein in 1992. The two lead authors, Gary Latham and Miriam Erez, had in their prior work on participation in decision making (PDM) received contradictory findings. Erez’ and Latham’s mutual acquaintanceship with Locke lead to the three meeting and brainstorming possible explanations for their research discrepancies. Rather than follow the seeming tradition of slinging scholastic mud at each other, Erez and Latham decided to collaborate on a joint design of experiments to uncover their prior research differences. Locke acted as mediator to ensure a smooth process, although he states that he rarely had to intervene and never in a significant manner (Latham et al, 1988). They conducted a series of four iterative experiments, using their learnings from early experiments to guide subsequent experiment design. Their process of painstakingly analyzing their research design enabled them to uncover procedural differences and conflating variables that accounted for their prior oppositional findings. Their results constitute a crucial experiment that proved that PDM does not increase performance compared to supportively-stated goals, but that it does increase goal commitment. Through this research they made a strong addition to the knowledge on PDM and pioneered a method for collaborative problem solving.
Non-exemplary qualities
No research is ever entirely great. The authors interviewed by Wheeler, Richter, & Sahadevan all commented that they believed their work could be improved. Omissions, compromises, or errors are made for the sake of expediency or arising from a lack of omniscient knowledge. Before extolling a work’s virtues, I feel it is important to acknowledge where it falls short. Not wishing to fault research for the assumptions and traditions of its epistemological paradigm, I do believe Latham, Erez, & Locke’s article lacked clarity and evocativeness in the writing style. A complaint that experimental laboratory tests lack ecological validity is not original or entirely fair as findings from experiments have been shown to transfer outside of the lab. Nonetheless, decisions in the test task chosen, performance measurement rubrics, and test duration limit the study’s ability to be generalized to larger, field scenarios. The study is further limited in its generalizability by the authors’ sampling strategy. Despite the ubiquity in academia of drawing upon a convenient supply of university students for research (even giving them course credit, as was the case with this study), sampling from university students, and specifically business students, does limit a study’s population validity. University students, and in particular business majors, are not representative of the general population as they may possess non-standard levels of intelligence, ambition, and literacy. Finally, I believe the study failed to control for, or at least acknowledge, possible conflating variables. The study makes definitive claims about the success of participatory decision making, yet failed to examine or control variables such as personality types, gender, age, power structures, stress level, and worker morale. In acknowledgement of Erez' Israeli background, they do posit that cultural factors may play a role, but this is the extent of their controls for the possible role of social, psychological, or demographic factors. Accounting for all possible mitigating factors is infeasible in any research; yet failing to concede their possible role weakens the authors' claims. These problems, however, are not unique to this study and do not detract from the strengths of the work.
1) Timing of the article or addressing a need in the literature
The first criterion for quality research is the degree of its timeliness or filling a void in existing literature. It is the latter point, that I personally believe touches upon the work’s most exemplary quality: its innovative approach. Cummings and Earley believe this approach is novel and effective; they recommend it be adopted by other researchers as it leads to a full understanding of phenomena (1992). In the history of intellectual thought and research, it is likely that scholars with opposing views have collaborated before. Within academic literature "antagonist" collaboration appears rare enough that Latham, Erez, & Locke believe they have pioneered it (1988). Although this claim is not substantiated, it appears no one has refuted their claim. Latham, Erez, & Locke may not entirely have invented collaborative and mediated research, but they are clearly pioneers in its refinement and promotion.
I admire their approach as it offers a constructive alternative to academics with divergent views from arguing and belittling each other in protracted arguments in literature and at conferences. It appears Greenberg is similarly impressed by this innovative approach noting, "Instead of merely adding to derision, as so often occurs, the authors agreed to some crucial experiments designed to answer a key question to their mutual satisfaction. We need more of this kind of work, in my opinion" (as cited in Wheeler, Richter, & Sahadevan, 2004, p. 1337). Greenberg and Cummings & Earley, and Greenberg all comment on the rarity of any such collaboration, which may be due to the risks involved. Either researcher could have been exposed as incorrect or having conducted prior faulty research. I find this an exemplary quality of the researchers that to me reveals their commitment to quality scholarship over protecting their own ego or reputations. I thus commend Erez and Latham both for pioneering this innovative approach and for committing to it.
2) Defining or clarifying constructs in the article
A key problem with Erez and Latham’s earlier work, and a problem not unique to them, is that there were unstated assumptions, unknown conflating variables, and undocumented procedures. Another additional exemplary trait of Latham, Erez, & Locke’s article is their how their collaboration enabled them to uncover these mysteries to deliver a solid, rich understanding of the phenomenon. This quality is remarkable as Wheeler, Richter, & Sahadevan state, "articles that become ‘great works’ define or clarify constructs in a way that future researchers rely on" (2004, p. 1329). Frost and Stablein in their commentary of their exemplary works believe that researchers' attention to detail (p. 249) leads to quality research. Precision in research design is crucial not only to ensure a study’s reliability and internal validity, but more importantly to deliver construct validity. It seems evident from Erez and Latham’s writings and commentary on the article that their previous studies lacked construct validity – they were not studying exactly what they thought they were studying. Their studies had conflating variables that meant that prior control groups were exposed to differing treatments. In addition, their operational definitions of a key construct, what constitutes a group, differed fundamentally. Erez defined it as a group of five people with equivalent power, whereas Latham defined it as a dyad with a superior and subordinate. Such differences not only limit the ability to compare findings across studies, but undermine the validity of the study. Latham appears to have been studying a completely different phenomenon than what he states as his definition is at odds with common definitions of a group and fails to account at all for the overriding power dynamics of his dyads. Instead of studying PDM, Latham was studying boss-employee dynamics. Erez describes how such differences and omissions can arise:
3) Focusing on multiple levels of the organization
Wheeler, Richter, & Sahadevan’s third quality criterion is less well articulated than their other criteria. They offer little explanation beyond that quality work should "interest both organizational scholars and practitioners" (p. 1329). One can infer from their cited examples that research should appeal beyond narrow, cloistered corners. It is not surprising that scholars writing on business and management topics would value a study’s application beyond academia, considering the subject’s direct relationship with industry. The subject of Latham, Erez, and Locke's article is organizational behaviour, which is generally more readily applicable to field contexts than other academic subjects. More specifically, Latham, Erez, & Locke do present a phenomenon with implications for both scholars and practitioners. Their work centers on a common organizational activity, goal setting, and its outcomes, performance and commitment. Practitioners can apply the findings to their own work, while scholars are offered an examination that addresses both praxis and ideology. By focusing on a research problem with direct application to business settings, as well as addressing the larger issue of power and equality, the study is exemplary in its appeal to both academia and industry.
4) Becoming a foundation article
Regarding Wheeler, Richter, & Sahadevan’s final criterion, "becoming a foundation article", the authors believe that the degree of success in the preceding criteria will beget this one. Although Latham, Erez, and Locke have distinguished academic careers, it is arguable whether or not the article’s 110 citations in SSCI would designate this as a foundational piece. Examining the article in Google, however, does reveal that the article’s importance extends beyond refereed journals as it is referenced 262 times. The influence this article has for inspiring or enlightening scholars is less hard to pinpoint when it does not result in a citation. Initially, the article’s quality was affirmed by peers when the Academy of Management awarded it the outstanding publication in organizational behavior in 1988. More recently, in 2004 Professor Greenberg, one of Wheeler, Richter, & Sahadevan canonical authors, credits Latham, Erez, and Locke’s article as the work he most admires. The inclusion of the work in Frost and Stablein’s collection further adds to this work’s influence beyond its citation counts, as the book is reported to be required reading in many graduate classes (Rafaeli, 2009). Its continued relevance is also demonstrated by its citation history, provided by Scopus. Scopus credits the article with 76 citations since 1996, the year Scopus’ database begins. Since that year, however, the article has been cited every year - often up to eight or nine times. Being cited for over 20 years and the article’s more intangible influence as demonstrated by its textbook status does make a strong case that this work can be considered foundational.
Conclusion
The collaborative research by Latham, Erez, & Locke provides a model of a productive way to "add to our knowledge more efficiently and more rapidly than would otherwise be the case" (Latham, Erez, & Locke, 1988, p. 771). Their innovative approach resulted in a study which generated more insight into the phenomenon of PDM than their prior, individual work had uncovered. I believe they were able to generate this insight as their joint, even antagonistic, efforts lead them, even forced them, to conduct research with an exemplary degree of validity and reliability. Over twenty years after being published, Latham, Erez, & Locke’s article is still read and cited by scholars and practitioners thus maintaining its influence and claim as an exemplary work.
References
Campbell, J. P. (1992). Experiments as reforms. In P. J Frost & R. E. Stablein (Eds.), Doing Exemplary Research (pp. 173-176). Newbury Park, CA: Sage Publications
Cummings, L.L., & Earley, P. C. (1992). Comments on the Latham/ Erez/ Locke study. In P. J Frost & R. E. Stablein (Eds.), Doing Exemplary Research (pp. 167-172). Newbury Park, CA: Sage Publications.
Erez, M. (1992). Reflections on the Latham/ Erez/ Locke study. In P. J Frost & R. E. Stablein (Eds.), Doing Exemplary Research (pp. 155-166). Newbury Park, CA: Sage Publications.
Frost, P. J., & Stablein, R. E. (1992). Doing exemplary research. Newbury Park, CA: Sage Publications.
Latham, G. P., Erez, M., & Locke, E. (1988). Resolving scientific disputes by the joint design of crucial experiments by the antagonists: Application to the Erez–Latham dispute regarding participation in goal setting. Journal of Applied Psychology, 73(4), 753-772.
Rafaeli, A. (2009, March 11). Professor Anat Rafaeli. Technion - Israel Institute of Technology Retrieved February 10, 2010, from http://iew3.technion.ac.il/Home/Users/anatr.html
Wheeler, A. R., Richter, E., & Sahadevan, S. (2004). Looking back on their “great works”: Insights from the authors of great works in organizational behavior and human resource management. Management Decision, 42(10), 1326-1342.
Wheeler, Richter, & Sahadevan asked the ten authors of their "great" journal articles why they felt their work were so well received. Analyzing the compiled interview data, they found researchers credit four central factors to their success: "1) timing of the article or addressing a need in the literature; 2) defining or clarifying constructs in the article; 3) focusing on multiple levels of the organization; and 4) becoming a foundation article" (p. 1328). Wheeler, Richter, & Sahadevan’s criteria allows one to move beyond the citation counts of articles and impact factors of journals to examine the qualities of exemplary research, particularly within the domain of organizational behaviour. Applying this framework to the Latham, Erez, & Locke article assists in illuminating the work’s key strengths that make it an exemplary work.
Background
Latham, Erez, & Locke’s article, entitled "Resolving scientific disputes by the joint design of crucial experiments by the antagonists: Application to the Erez-Latham disputes regarding participation in goal setting", was originally published in the prestigious Journal of Applied Psychology in 1988 and then included in the edited volume Doing Exemplary Research by Frost & Stablein in 1992. The two lead authors, Gary Latham and Miriam Erez, had in their prior work on participation in decision making (PDM) received contradictory findings. Erez’ and Latham’s mutual acquaintanceship with Locke lead to the three meeting and brainstorming possible explanations for their research discrepancies. Rather than follow the seeming tradition of slinging scholastic mud at each other, Erez and Latham decided to collaborate on a joint design of experiments to uncover their prior research differences. Locke acted as mediator to ensure a smooth process, although he states that he rarely had to intervene and never in a significant manner (Latham et al, 1988). They conducted a series of four iterative experiments, using their learnings from early experiments to guide subsequent experiment design. Their process of painstakingly analyzing their research design enabled them to uncover procedural differences and conflating variables that accounted for their prior oppositional findings. Their results constitute a crucial experiment that proved that PDM does not increase performance compared to supportively-stated goals, but that it does increase goal commitment. Through this research they made a strong addition to the knowledge on PDM and pioneered a method for collaborative problem solving.
Non-exemplary qualities
No research is ever entirely great. The authors interviewed by Wheeler, Richter, & Sahadevan all commented that they believed their work could be improved. Omissions, compromises, or errors are made for the sake of expediency or arising from a lack of omniscient knowledge. Before extolling a work’s virtues, I feel it is important to acknowledge where it falls short. Not wishing to fault research for the assumptions and traditions of its epistemological paradigm, I do believe Latham, Erez, & Locke’s article lacked clarity and evocativeness in the writing style. A complaint that experimental laboratory tests lack ecological validity is not original or entirely fair as findings from experiments have been shown to transfer outside of the lab. Nonetheless, decisions in the test task chosen, performance measurement rubrics, and test duration limit the study’s ability to be generalized to larger, field scenarios. The study is further limited in its generalizability by the authors’ sampling strategy. Despite the ubiquity in academia of drawing upon a convenient supply of university students for research (even giving them course credit, as was the case with this study), sampling from university students, and specifically business students, does limit a study’s population validity. University students, and in particular business majors, are not representative of the general population as they may possess non-standard levels of intelligence, ambition, and literacy. Finally, I believe the study failed to control for, or at least acknowledge, possible conflating variables. The study makes definitive claims about the success of participatory decision making, yet failed to examine or control variables such as personality types, gender, age, power structures, stress level, and worker morale. In acknowledgement of Erez' Israeli background, they do posit that cultural factors may play a role, but this is the extent of their controls for the possible role of social, psychological, or demographic factors. Accounting for all possible mitigating factors is infeasible in any research; yet failing to concede their possible role weakens the authors' claims. These problems, however, are not unique to this study and do not detract from the strengths of the work.
1) Timing of the article or addressing a need in the literature
The first criterion for quality research is the degree of its timeliness or filling a void in existing literature. It is the latter point, that I personally believe touches upon the work’s most exemplary quality: its innovative approach. Cummings and Earley believe this approach is novel and effective; they recommend it be adopted by other researchers as it leads to a full understanding of phenomena (1992). In the history of intellectual thought and research, it is likely that scholars with opposing views have collaborated before. Within academic literature "antagonist" collaboration appears rare enough that Latham, Erez, & Locke believe they have pioneered it (1988). Although this claim is not substantiated, it appears no one has refuted their claim. Latham, Erez, & Locke may not entirely have invented collaborative and mediated research, but they are clearly pioneers in its refinement and promotion.
I admire their approach as it offers a constructive alternative to academics with divergent views from arguing and belittling each other in protracted arguments in literature and at conferences. It appears Greenberg is similarly impressed by this innovative approach noting, "Instead of merely adding to derision, as so often occurs, the authors agreed to some crucial experiments designed to answer a key question to their mutual satisfaction. We need more of this kind of work, in my opinion" (as cited in Wheeler, Richter, & Sahadevan, 2004, p. 1337). Greenberg and Cummings & Earley, and Greenberg all comment on the rarity of any such collaboration, which may be due to the risks involved. Either researcher could have been exposed as incorrect or having conducted prior faulty research. I find this an exemplary quality of the researchers that to me reveals their commitment to quality scholarship over protecting their own ego or reputations. I thus commend Erez and Latham both for pioneering this innovative approach and for committing to it.
2) Defining or clarifying constructs in the article
A key problem with Erez and Latham’s earlier work, and a problem not unique to them, is that there were unstated assumptions, unknown conflating variables, and undocumented procedures. Another additional exemplary trait of Latham, Erez, & Locke’s article is their how their collaboration enabled them to uncover these mysteries to deliver a solid, rich understanding of the phenomenon. This quality is remarkable as Wheeler, Richter, & Sahadevan state, "articles that become ‘great works’ define or clarify constructs in a way that future researchers rely on" (2004, p. 1329). Frost and Stablein in their commentary of their exemplary works believe that researchers' attention to detail (p. 249) leads to quality research. Precision in research design is crucial not only to ensure a study’s reliability and internal validity, but more importantly to deliver construct validity. It seems evident from Erez and Latham’s writings and commentary on the article that their previous studies lacked construct validity – they were not studying exactly what they thought they were studying. Their studies had conflating variables that meant that prior control groups were exposed to differing treatments. In addition, their operational definitions of a key construct, what constitutes a group, differed fundamentally. Erez defined it as a group of five people with equivalent power, whereas Latham defined it as a dyad with a superior and subordinate. Such differences not only limit the ability to compare findings across studies, but undermine the validity of the study. Latham appears to have been studying a completely different phenomenon than what he states as his definition is at odds with common definitions of a group and fails to account at all for the overriding power dynamics of his dyads. Instead of studying PDM, Latham was studying boss-employee dynamics. Erez describes how such differences and omissions can arise:
Very often, researchers are even unaware of contextual differences because they are part of the context and have no external reference point. By working together, the two researchers provided each other with the reference points needed to define the unique characteristics of each other’s procedures…. The most striking part of this process for Locke was the number of differences in procedures and design that can occur when two people are allegedly studying the same phenomenon. The number of little differences between studies can add up and have a significant effect on the differences in empirical findings and hence on conclusions (1992, p. 162).It is crucial to clearly operationalize the constructs being studied and control for (or account for) as many variables as possible, to be able to make definitive claims about a phenomenon. Campbell, in commenting on the article, commends the authors for their clear, unambiguous operationalizing of key constructs. He observes it is more common for researchers to "underspecify the nature of the variables we use" which creates "surplus meaning" (1992, p. 174). Citing the Hawthorne studies, Campbell believes a lack of clarity on what exact variables were manipulated leads to speculation over the results. Latham, Erez, & Locke may not have accounted for all possible variables, but their efforts to uncover their differences resulted in a study with a high degree of construct validity that allowed causal relationships to be determined about a precise phenomenon.
3) Focusing on multiple levels of the organization
Wheeler, Richter, & Sahadevan’s third quality criterion is less well articulated than their other criteria. They offer little explanation beyond that quality work should "interest both organizational scholars and practitioners" (p. 1329). One can infer from their cited examples that research should appeal beyond narrow, cloistered corners. It is not surprising that scholars writing on business and management topics would value a study’s application beyond academia, considering the subject’s direct relationship with industry. The subject of Latham, Erez, and Locke's article is organizational behaviour, which is generally more readily applicable to field contexts than other academic subjects. More specifically, Latham, Erez, & Locke do present a phenomenon with implications for both scholars and practitioners. Their work centers on a common organizational activity, goal setting, and its outcomes, performance and commitment. Practitioners can apply the findings to their own work, while scholars are offered an examination that addresses both praxis and ideology. By focusing on a research problem with direct application to business settings, as well as addressing the larger issue of power and equality, the study is exemplary in its appeal to both academia and industry.
4) Becoming a foundation article
Regarding Wheeler, Richter, & Sahadevan’s final criterion, "becoming a foundation article", the authors believe that the degree of success in the preceding criteria will beget this one. Although Latham, Erez, and Locke have distinguished academic careers, it is arguable whether or not the article’s 110 citations in SSCI would designate this as a foundational piece. Examining the article in Google, however, does reveal that the article’s importance extends beyond refereed journals as it is referenced 262 times. The influence this article has for inspiring or enlightening scholars is less hard to pinpoint when it does not result in a citation. Initially, the article’s quality was affirmed by peers when the Academy of Management awarded it the outstanding publication in organizational behavior in 1988. More recently, in 2004 Professor Greenberg, one of Wheeler, Richter, & Sahadevan canonical authors, credits Latham, Erez, and Locke’s article as the work he most admires. The inclusion of the work in Frost and Stablein’s collection further adds to this work’s influence beyond its citation counts, as the book is reported to be required reading in many graduate classes (Rafaeli, 2009). Its continued relevance is also demonstrated by its citation history, provided by Scopus. Scopus credits the article with 76 citations since 1996, the year Scopus’ database begins. Since that year, however, the article has been cited every year - often up to eight or nine times. Being cited for over 20 years and the article’s more intangible influence as demonstrated by its textbook status does make a strong case that this work can be considered foundational.
Conclusion
The collaborative research by Latham, Erez, & Locke provides a model of a productive way to "add to our knowledge more efficiently and more rapidly than would otherwise be the case" (Latham, Erez, & Locke, 1988, p. 771). Their innovative approach resulted in a study which generated more insight into the phenomenon of PDM than their prior, individual work had uncovered. I believe they were able to generate this insight as their joint, even antagonistic, efforts lead them, even forced them, to conduct research with an exemplary degree of validity and reliability. Over twenty years after being published, Latham, Erez, & Locke’s article is still read and cited by scholars and practitioners thus maintaining its influence and claim as an exemplary work.
References
Campbell, J. P. (1992). Experiments as reforms. In P. J Frost & R. E. Stablein (Eds.), Doing Exemplary Research (pp. 173-176). Newbury Park, CA: Sage Publications
Cummings, L.L., & Earley, P. C. (1992). Comments on the Latham/ Erez/ Locke study. In P. J Frost & R. E. Stablein (Eds.), Doing Exemplary Research (pp. 167-172). Newbury Park, CA: Sage Publications.
Erez, M. (1992). Reflections on the Latham/ Erez/ Locke study. In P. J Frost & R. E. Stablein (Eds.), Doing Exemplary Research (pp. 155-166). Newbury Park, CA: Sage Publications.
Frost, P. J., & Stablein, R. E. (1992). Doing exemplary research. Newbury Park, CA: Sage Publications.
Latham, G. P., Erez, M., & Locke, E. (1988). Resolving scientific disputes by the joint design of crucial experiments by the antagonists: Application to the Erez–Latham dispute regarding participation in goal setting. Journal of Applied Psychology, 73(4), 753-772.
Rafaeli, A. (2009, March 11). Professor Anat Rafaeli. Technion - Israel Institute of Technology Retrieved February 10, 2010, from http://iew3.technion.ac.il/Home/Users/anatr.html
Wheeler, A. R., Richter, E., & Sahadevan, S. (2004). Looking back on their “great works”: Insights from the authors of great works in organizational behavior and human resource management. Management Decision, 42(10), 1326-1342.
Thursday, March 25, 2010
Conference Presentation on Web Accessibility Challenges
I recently presented for my first time at an academic conference. The conference, Information Access: Commons, Control & Controversy, was held at my home faculty, iSchool at University of Toronto.
I presented on my research findings from my masters thesis. I thought it distilled my findings in a manageable way, so I thought I'd share it here.
Barriers to Adoption and Implementation of Web Accessibility
I'm Glen Farrelly, a first-year PhD student here at iSchool. My background and interests are in designing and understanding online user experience. My utopian dreams of the democratizing potential of the Internet have not been completely quelled by the realities of the ongoing digital divide. From this perspective, I'm concerned with how to create online experiences that strives to enable everyone to participate.
Web accessibility refers to users being able to perceive and operate publically available websites regardless of their abilities. Users can be blocked by design or code barriers based on: vision, hearing, cognitive ability, and mobility or motor control.
In line with the social model of disability, I believe that it is artificial barriers that create disability that may not otherwise exist. These barriers can apply not only to permanently disabled people but for those temporarily or situationally disabled as well.
Internet enables and disables
Many people use the Internet and find it helpful for various facets of their life from work to entertainment. The Internet has the potential to aid disabled people to overcome some barriers. For example, online shopping for people with mobility impairments helped reduced their self-reported rates of disability (Spillman, 2004). Also deaf people have been avid adopters of text and instant messaging as a preferred method of communication. It is for reasons such as these that lead to 54% of respondents to a UK survey (2002) listing Internet access as essential, compared to only 6% of the general population.
Despite this potential, accessibility rates in Canada and globally remain very low. The UN commissioned a review of popular websites in 2006 of 20 countries and only 7% passed the most basic accessibility measures. Inaccessibility of online content was attributed to result in less than half as many disabled people using the Internet compared to the nondisabled, in an US Study (Dobransky & Hargittai, 2006). This issue also affects more Canadians than most people realize, as 14 per cent of, or 4.4 million, identified themselves as disabled (Statistics Canada, 2007).
Web inaccessibility
Here’s an example U.of T. students and faculty may have encountered. The low contrast of the text on the navigational menus is hard to read probably for most users, but for those users with vision impairment this can be a complete block. Another problem with this webpage is the fan out menu at times requires very precise mouse movement or it closes. This is a block for those with motor impairments, such as those with Parkinson's disease. Personally, I can attest it's extremely difficult to use on a laptop.
Considering the diversity of human ability, it is unlikely that anyone can build a website that everyone can use. The goal then is to make content as accessible as possible.
The solution?
There are three approaches to making online content more accessible:
To aid practitioners in the nebulous goal of making their content accessible, the W3C, in collaboration with academic, industrial, and disability organizations worldwide published the Web Content Authoring Guidelines (WCAG) in 1999. WCAG is the world’s predominant guidelines and informs other guidelines such as Canada’s CLF and the US’ Section 508. There are 3 degrees of voluntary compliance. The W3C has focused their efforts on further refining the guidelines, I'd say more so than on education and outreach.
In Canada, at least, no organization or government office is leading the charge to deliver educational resources, training, or awareness campaigns. As accessibility is not expressly required in any jurisdiction I am aware of and as disability issues can be ignored by corporate power, web accessibility has tended to fall on individual practitioners.
The way to make more accessible websites is there, but practitioners have to find out about it themselves, figure out how to do it themselves, implement it themselves, and often make a convincing business case to do so. Numerous studies have shown however that most websites in a variety of sectors are not even following the most basic guidelines. To help make the web more accessible, I felt it is important to understand what are the stumbling blocks to adoption. To do this, one needs to understand those our society has charged with implementing this issue: web practitioners.
Web accessibility adoption challenges have largely been ignored both in academic and trade literature. In addition, research actually consulting web practitioners is even more rare as it appears that there are only three such studies.
My research
I conducted 21 semi-structured interviews with a variety of web practitioners across Canada. I asked practitioners not only how they encountered the issue of accessibility and disability, but also how they implemented in on their day-to-day job. They offered both their obstacles and solutions.
Web Accessibility Diffusion & Implementation Model
Out of the interviews various diffusion and implementation factors were identified, which lead me to develop this model.
Diffusion refers to how a practitioner hears about an issue, learns the specifics, and formulates attitudes. The diffusion of accessibility is also affected by knowledge and perceptions about disability in general. Implementation refers to the factors that impact a practitioner's ability to produce accessible online content.
Web practitioners
The first challenge in this issue is that practitioners are not a homogenous group, but represent various responsibilities, backgrounds, and skills.Web practitioners include:
Diffusion
Innovations are less apt to diffuse if they are difficult to understand or use, require significant resources, or are incompatible with existing norms and values. Web accessibility meets all these criteria.
The guidelines are highly technical and yet like much of disability issues, education and media were found to not adequately cover the topic. Most participants recalled that they had not encountered this issue suitably - or at all - in their career training, in the media or trade events.
While many accessibility measures can be implemented with no additional costs, some measures do require more effort and scale and competitive environment can be a factor. One participant who managed a large site with thousands of technical reports had to remove the bulk of them as it was taking days to convert a single report. There appears to be no specific tax breaks or financial incentives to help mitigate this cost.
As far as I'm aware, there are no laws expressly requiring websites to be accessible – although there are prohibitions on discrimination. Federal and provincial governments have passed policies requiring their own websites to be accessible. But Ontario is the only jurisdiction in Canada pursuing requiring web accessibility for all business and organizations that operate in the province. Even legally requiring accessibility is not guaranteed to increase compliance as a Brazilian study found. It also introduces a new problem of policing.
Implementation
In interviewing practitioners who had tried to or did implement accessibility, they were frustrated with the inadequate support. Most criticism was directed at W3C and WCAG. Two participants described using it: "If I can’t navigate it, who the hell can?" and "It just gets into tech-babble. It must be completely overwhelming for those less experienced." The guidelines are not particularly accessible – they are a quagmire and are written at a level that even veteran professionals found too difficult to interpret. There are reasons for this, but W3C has not done a sufficient job at making their guidelines usable, offering help, code libraries, and tools. Instructional material is out there but not in a prominent, central location. It can be hard to find, dated, time-consuming, and dubious.
Popular commercial authoring tools such as Dreamweaver are improving their support for accessibility. With content management systems and web-based software becoming increasingly popular, however, there is a continuous need to improve functionality and transparency of features.
Similarly, testing software to automatically check for accessibility is insufficient and often unknown by participants. Although testing software is improving, certain items must be tested with disabled users. In general there is a lack of any user testing, so the need for human testing is problematic.
Considering how difficult this is some practitioners have turned to hiring experts. As there are no organizations in Canada offering free guidance – even CNIB charges for accessibility services - web accessibility is being commercialized. Alarmingly, three participants hired so-called accessibility experts that amounted to charlatans and failed to deliver on some basic accessibility measures.
Conclusion
No matter the societal changes, software sophistication, or increased support, the onus will still fall on practitioners to make the line by line and graphic by graphic changes on a daily basis to make web accessibility a reality. To do this, practitioners needs to be aware of their important role in the disability divide. As one participant put it:
Finally, research is needed to study how web practitioners work, how they use the guidelines, support tools, and software. This research can then be used to make the implementation tools and resources more usable and transparent.
I presented on my research findings from my masters thesis. I thought it distilled my findings in a manageable way, so I thought I'd share it here.
Barriers to Adoption and Implementation of Web Accessibility
I'm Glen Farrelly, a first-year PhD student here at iSchool. My background and interests are in designing and understanding online user experience. My utopian dreams of the democratizing potential of the Internet have not been completely quelled by the realities of the ongoing digital divide. From this perspective, I'm concerned with how to create online experiences that strives to enable everyone to participate.
Web accessibility refers to users being able to perceive and operate publically available websites regardless of their abilities. Users can be blocked by design or code barriers based on: vision, hearing, cognitive ability, and mobility or motor control.
In line with the social model of disability, I believe that it is artificial barriers that create disability that may not otherwise exist. These barriers can apply not only to permanently disabled people but for those temporarily or situationally disabled as well.
Internet enables and disables
Many people use the Internet and find it helpful for various facets of their life from work to entertainment. The Internet has the potential to aid disabled people to overcome some barriers. For example, online shopping for people with mobility impairments helped reduced their self-reported rates of disability (Spillman, 2004). Also deaf people have been avid adopters of text and instant messaging as a preferred method of communication. It is for reasons such as these that lead to 54% of respondents to a UK survey (2002) listing Internet access as essential, compared to only 6% of the general population.
Despite this potential, accessibility rates in Canada and globally remain very low. The UN commissioned a review of popular websites in 2006 of 20 countries and only 7% passed the most basic accessibility measures. Inaccessibility of online content was attributed to result in less than half as many disabled people using the Internet compared to the nondisabled, in an US Study (Dobransky & Hargittai, 2006). This issue also affects more Canadians than most people realize, as 14 per cent of, or 4.4 million, identified themselves as disabled (Statistics Canada, 2007).
Web inaccessibility
Here’s an example U.of T. students and faculty may have encountered. The low contrast of the text on the navigational menus is hard to read probably for most users, but for those users with vision impairment this can be a complete block. Another problem with this webpage is the fan out menu at times requires very precise mouse movement or it closes. This is a block for those with motor impairments, such as those with Parkinson's disease. Personally, I can attest it's extremely difficult to use on a laptop.
Considering the diversity of human ability, it is unlikely that anyone can build a website that everyone can use. The goal then is to make content as accessible as possible.
The solution?
There are three approaches to making online content more accessible:
- Adaptive technology can allow for multimodal outputs or alternative inputs. For example, screen reader software can read the contents of a webpage to a blind person. But even though tech continues to get more sophisticated there are still problems, for example screen readers cannot compensate for images that don't have alternative text to describe it.
- Automated conversion can be done various ways. This month YouTube opened up its free software to automatically caption user videos. Solutions like this may address the challenge of making the extreme long tail of online content more accessible but they still present barriers. For example, I'm dubious that autocaptioning software will be able to compensate for poor audio levels, overlapping speech, slang, etc.
- Human development is probably the best way to make the content more accessible. That is, the content creators build the content in ways that extend its functionality and flexibility. For example, describing the meaning of an image is best done by its creator.
To aid practitioners in the nebulous goal of making their content accessible, the W3C, in collaboration with academic, industrial, and disability organizations worldwide published the Web Content Authoring Guidelines (WCAG) in 1999. WCAG is the world’s predominant guidelines and informs other guidelines such as Canada’s CLF and the US’ Section 508. There are 3 degrees of voluntary compliance. The W3C has focused their efforts on further refining the guidelines, I'd say more so than on education and outreach.
In Canada, at least, no organization or government office is leading the charge to deliver educational resources, training, or awareness campaigns. As accessibility is not expressly required in any jurisdiction I am aware of and as disability issues can be ignored by corporate power, web accessibility has tended to fall on individual practitioners.
The way to make more accessible websites is there, but practitioners have to find out about it themselves, figure out how to do it themselves, implement it themselves, and often make a convincing business case to do so. Numerous studies have shown however that most websites in a variety of sectors are not even following the most basic guidelines. To help make the web more accessible, I felt it is important to understand what are the stumbling blocks to adoption. To do this, one needs to understand those our society has charged with implementing this issue: web practitioners.
Web accessibility adoption challenges have largely been ignored both in academic and trade literature. In addition, research actually consulting web practitioners is even more rare as it appears that there are only three such studies.
My research
I conducted 21 semi-structured interviews with a variety of web practitioners across Canada. I asked practitioners not only how they encountered the issue of accessibility and disability, but also how they implemented in on their day-to-day job. They offered both their obstacles and solutions.
Web Accessibility Diffusion & Implementation Model
Out of the interviews various diffusion and implementation factors were identified, which lead me to develop this model.
Diffusion refers to how a practitioner hears about an issue, learns the specifics, and formulates attitudes. The diffusion of accessibility is also affected by knowledge and perceptions about disability in general. Implementation refers to the factors that impact a practitioner's ability to produce accessible online content.
Web practitioners
The first challenge in this issue is that practitioners are not a homogenous group, but represent various responsibilities, backgrounds, and skills.Web practitioners include:
- programmers
- designers
- developers
- webmasters
- producers
- writers and editors
- animators
- assistants
- managers
- bloggers
- podcasters
Diffusion
Innovations are less apt to diffuse if they are difficult to understand or use, require significant resources, or are incompatible with existing norms and values. Web accessibility meets all these criteria.
The guidelines are highly technical and yet like much of disability issues, education and media were found to not adequately cover the topic. Most participants recalled that they had not encountered this issue suitably - or at all - in their career training, in the media or trade events.
While many accessibility measures can be implemented with no additional costs, some measures do require more effort and scale and competitive environment can be a factor. One participant who managed a large site with thousands of technical reports had to remove the bulk of them as it was taking days to convert a single report. There appears to be no specific tax breaks or financial incentives to help mitigate this cost.
As far as I'm aware, there are no laws expressly requiring websites to be accessible – although there are prohibitions on discrimination. Federal and provincial governments have passed policies requiring their own websites to be accessible. But Ontario is the only jurisdiction in Canada pursuing requiring web accessibility for all business and organizations that operate in the province. Even legally requiring accessibility is not guaranteed to increase compliance as a Brazilian study found. It also introduces a new problem of policing.
Implementation
In interviewing practitioners who had tried to or did implement accessibility, they were frustrated with the inadequate support. Most criticism was directed at W3C and WCAG. Two participants described using it: "If I can’t navigate it, who the hell can?" and "It just gets into tech-babble. It must be completely overwhelming for those less experienced." The guidelines are not particularly accessible – they are a quagmire and are written at a level that even veteran professionals found too difficult to interpret. There are reasons for this, but W3C has not done a sufficient job at making their guidelines usable, offering help, code libraries, and tools. Instructional material is out there but not in a prominent, central location. It can be hard to find, dated, time-consuming, and dubious.
Popular commercial authoring tools such as Dreamweaver are improving their support for accessibility. With content management systems and web-based software becoming increasingly popular, however, there is a continuous need to improve functionality and transparency of features.
Similarly, testing software to automatically check for accessibility is insufficient and often unknown by participants. Although testing software is improving, certain items must be tested with disabled users. In general there is a lack of any user testing, so the need for human testing is problematic.
Considering how difficult this is some practitioners have turned to hiring experts. As there are no organizations in Canada offering free guidance – even CNIB charges for accessibility services - web accessibility is being commercialized. Alarmingly, three participants hired so-called accessibility experts that amounted to charlatans and failed to deliver on some basic accessibility measures.
Conclusion
No matter the societal changes, software sophistication, or increased support, the onus will still fall on practitioners to make the line by line and graphic by graphic changes on a daily basis to make web accessibility a reality. To do this, practitioners needs to be aware of their important role in the disability divide. As one participant put it:
Web developers can be proactive, so they need to let companies know it’s an issue and that including it is just part of our standard services. No companies would say no to this. Or you can just do it.
Finally, research is needed to study how web practitioners work, how they use the guidelines, support tools, and software. This research can then be used to make the implementation tools and resources more usable and transparent.
Monday, January 25, 2010
Political Participation Online in Canada
On the weekend, my wife, daughter and I joined thousands of Canadians protesting Stephen Harper proroguing parliament. The protests were particularly remarkable as they grew from a Facebook group "Canadians Against Proroguing Parliament". The group organized only a few weeks ago has over 200,000 Canadians - the group and the protests helped show that despite Harper's assertions Canadians do care about working democracy in this country!
Harper's dismissiveness of open, participatory government and the efficacy of Internet media for facilitating involvement are timely & relevant to my recent research. Governments are increasingly engaging people online and committing to open and collaborative efforts. Canada, however, under Harper's leadership, is missing the boat.
December 2009 saw the United Kingdom, United States, and Australian governments committing to more open and collaborative government, with the Internet central to achieving this. On December 8, the Obama administration in the US released their Open Government Initiative with three core goals of transparency, participation and collaboration. This effort builds on Obama's earlier commitment to more open, accessible government and the use of online technologies in delivering these goals. Brown, prime minister of the UK, released a plan addressing government reform on December 7. This report, Putting the Frontline First: Smarter Government, included online means to make government more responsive and engaging to citizens. Australia followed on December 22 with their report, entitled Engage: Getting on with Government 2.0. New Zealand was ahead of the pack, releasing a similar report, When Government Engages: Online participation - An introduction, in 2007.
Prior to these announcements, these countries were already making innovative forays into online participation. Earlier this year, the United States offerred citizens the opportunity to submit questions to Obama and he responded via online videos to those with the most citizen votes. The prime minister’s office of the UK instituted an e-petition process wherein the government guarantees a reply to any petition that obtains over 500 e-signatures. New Zealand’s Family Commission launched “The Coach,” a self-selected panel of citizens that answer polls and questionnaires to guide policy. Various municipalities in Australia have used online technologies to facilitate citizen discussion, information gathering, and polling on particular issues.
Given the many initiatives of these other countries, one might ask where is Canada on this issue?
For five years, from 2001 to 2005, Canada’s national government earned the distinction from Accenture of having the world’s best e-government. The survey has been discontinued, but Canada has continued to earn praise for its efforts to transfer administrative functions online. Administratively, Canada has made use of Internet technologies to launch innovations such as the world’s first online census and one of the first online tax filing systems.
In looking at Canada’s current state of e-government offerings promoting transparency, participation or collaboration, however, there are few examples and none particularly inspiring. Some examples are rather disheartening, as per an invitation to “join the conversation” on the website of Canada’s governing party, the Conservatives. The multimedia “conversation” begins with Mike Duffy greeting website visitors by name, but then launches into a monologue concluding in an appeal to donate to the party. Even the Green Party of Canada, a proponent of greater direct democracy and an early adopter of digital media, is largely using new media for old, traditional broadcasts.
As with other countries, Canada has enabled citizens to comment on policy electronically. Two examples drawn from last month, December 2009, include British Columbia’s proposed changes to their fresh water act or a national campaign for copyright reform.
Last November, the City of Toronto facilitated citizens to launch a website, Open Toronto, which allowed city data to be openly displayed online and used freely. Toronto appears an anomaly as the other Canadian cases of online participation I found were entirely consultative or communicative.
I was also unable to find any online Canadian government programs that entailed binding or formalized outcomes of citizen participation. Online political discussions, whether organized by the government or citizen-lead, may help influence policy, but their efficacy is severely doubted as there is no commitment from the government to listen or respond to these discussions.
The Internet is certainly being used in Canada to organize and mobilize protest (per the prorogue protest), to produce and disseminate information, as well as hacktivist and DIY projects.
Other countries are using the Internet to directly involve citizens and opening up governments - just not Canada it seems. In looking at the current government, it seems unlikely we'll see any improvements. Some researchers speculate that this is due to the Conservative's minority government status. I, however, wonder if there is not a more overt disdain on behalf of the Tories for the digital revolution and citizen involvement.
Harper's dismissiveness of open, participatory government and the efficacy of Internet media for facilitating involvement are timely & relevant to my recent research. Governments are increasingly engaging people online and committing to open and collaborative efforts. Canada, however, under Harper's leadership, is missing the boat.
December 2009 saw the United Kingdom, United States, and Australian governments committing to more open and collaborative government, with the Internet central to achieving this. On December 8, the Obama administration in the US released their Open Government Initiative with three core goals of transparency, participation and collaboration. This effort builds on Obama's earlier commitment to more open, accessible government and the use of online technologies in delivering these goals. Brown, prime minister of the UK, released a plan addressing government reform on December 7. This report, Putting the Frontline First: Smarter Government, included online means to make government more responsive and engaging to citizens. Australia followed on December 22 with their report, entitled Engage: Getting on with Government 2.0. New Zealand was ahead of the pack, releasing a similar report, When Government Engages: Online participation - An introduction, in 2007.
Prior to these announcements, these countries were already making innovative forays into online participation. Earlier this year, the United States offerred citizens the opportunity to submit questions to Obama and he responded via online videos to those with the most citizen votes. The prime minister’s office of the UK instituted an e-petition process wherein the government guarantees a reply to any petition that obtains over 500 e-signatures. New Zealand’s Family Commission launched “The Coach,” a self-selected panel of citizens that answer polls and questionnaires to guide policy. Various municipalities in Australia have used online technologies to facilitate citizen discussion, information gathering, and polling on particular issues.
Given the many initiatives of these other countries, one might ask where is Canada on this issue?
For five years, from 2001 to 2005, Canada’s national government earned the distinction from Accenture of having the world’s best e-government. The survey has been discontinued, but Canada has continued to earn praise for its efforts to transfer administrative functions online. Administratively, Canada has made use of Internet technologies to launch innovations such as the world’s first online census and one of the first online tax filing systems.
In looking at Canada’s current state of e-government offerings promoting transparency, participation or collaboration, however, there are few examples and none particularly inspiring. Some examples are rather disheartening, as per an invitation to “join the conversation” on the website of Canada’s governing party, the Conservatives. The multimedia “conversation” begins with Mike Duffy greeting website visitors by name, but then launches into a monologue concluding in an appeal to donate to the party. Even the Green Party of Canada, a proponent of greater direct democracy and an early adopter of digital media, is largely using new media for old, traditional broadcasts.
As with other countries, Canada has enabled citizens to comment on policy electronically. Two examples drawn from last month, December 2009, include British Columbia’s proposed changes to their fresh water act or a national campaign for copyright reform.
Last November, the City of Toronto facilitated citizens to launch a website, Open Toronto, which allowed city data to be openly displayed online and used freely. Toronto appears an anomaly as the other Canadian cases of online participation I found were entirely consultative or communicative.
I was also unable to find any online Canadian government programs that entailed binding or formalized outcomes of citizen participation. Online political discussions, whether organized by the government or citizen-lead, may help influence policy, but their efficacy is severely doubted as there is no commitment from the government to listen or respond to these discussions.
The Internet is certainly being used in Canada to organize and mobilize protest (per the prorogue protest), to produce and disseminate information, as well as hacktivist and DIY projects.
Other countries are using the Internet to directly involve citizens and opening up governments - just not Canada it seems. In looking at the current government, it seems unlikely we'll see any improvements. Some researchers speculate that this is due to the Conservative's minority government status. I, however, wonder if there is not a more overt disdain on behalf of the Tories for the digital revolution and citizen involvement.
Subscribe to:
Posts (Atom)