Friday, March 26, 2010

Exemplary Antagonism

There is no established formula for determining a great work of academic research. However, Wheeler, Richter, & Sahadevan (2004) attempted to uncover the ingredients of great research by interviewing leading researchers in their field of organizational behavior and human resource management. They selected nine journal articles they personally felt were great works. No other details about their sampling strategy were provided, although the total number of citations was a key determinant. The sample of their great works begins with an article that received 124 citations (as provided by Thomson Reuters’ Social Sciences Citations Index). A study contemporaneous to that one is by Latham, Erez, & Locke (1988) which received a similar number of citations in SSCI at 110. Does this then also qualify it as a great work?

Wheeler, Richter, & Sahadevan asked the ten authors of their "great" journal articles why they felt their work were so well received. Analyzing the compiled interview data, they found researchers credit four central factors to their success: "1) timing of the article or addressing a need in the literature; 2) defining or clarifying constructs in the article; 3) focusing on multiple levels of the organization; and 4) becoming a foundation article" (p. 1328). Wheeler, Richter, & Sahadevan’s criteria allows one to move beyond the citation counts of articles and impact factors of journals to examine the qualities of exemplary research, particularly within the domain of organizational behaviour. Applying this framework to the Latham, Erez, & Locke article assists in illuminating the work’s key strengths that make it an exemplary work.

Latham, Erez, & Locke’s article, entitled "Resolving scientific disputes by the joint design of crucial experiments by the antagonists: Application to the Erez-Latham disputes regarding participation in goal setting", was originally published in the prestigious Journal of Applied Psychology in 1988 and then included in the edited volume Doing Exemplary Research by Frost & Stablein in 1992. The two lead authors, Gary Latham and Miriam Erez, had in their prior work on participation in decision making (PDM) received contradictory findings. Erez’ and Latham’s mutual acquaintanceship with Locke lead to the three meeting and brainstorming possible explanations for their research discrepancies. Rather than follow the seeming tradition of slinging scholastic mud at each other, Erez and Latham decided to collaborate on a joint design of experiments to uncover their prior research differences. Locke acted as mediator to ensure a smooth process, although he states that he rarely had to intervene and never in a significant manner (Latham et al, 1988). They conducted a series of four iterative experiments, using their learnings from early experiments to guide subsequent experiment design. Their process of painstakingly analyzing their research design enabled them to uncover procedural differences and conflating variables that accounted for their prior oppositional findings. Their results constitute a crucial experiment that proved that PDM does not increase performance compared to supportively-stated goals, but that it does increase goal commitment. Through this research they made a strong addition to the knowledge on PDM and pioneered a method for collaborative problem solving.

Non-exemplary qualities
No research is ever entirely great. The authors interviewed by Wheeler, Richter, & Sahadevan all commented that they believed their work could be improved. Omissions, compromises, or errors are made for the sake of expediency or arising from a lack of omniscient knowledge. Before extolling a work’s virtues, I feel it is important to acknowledge where it falls short. Not wishing to fault research for the assumptions and traditions of its epistemological paradigm, I do believe Latham, Erez, & Locke’s article lacked clarity and evocativeness in the writing style. A complaint that experimental laboratory tests lack ecological validity is not original or entirely fair as findings from experiments have been shown to transfer outside of the lab. Nonetheless, decisions in the test task chosen, performance measurement rubrics, and test duration limit the study’s ability to be generalized to larger, field scenarios. The study is further limited in its generalizability by the authors’ sampling strategy. Despite the ubiquity in academia of drawing upon a convenient supply of university students for research (even giving them course credit, as was the case with this study), sampling from university students, and specifically business students, does limit a study’s population validity. University students, and in particular business majors, are not representative of the general population as they may possess non-standard levels of intelligence, ambition, and literacy. Finally, I believe the study failed to control for, or at least acknowledge, possible conflating variables. The study makes definitive claims about the success of participatory decision making, yet failed to examine or control variables such as personality types, gender, age, power structures, stress level, and worker morale. In acknowledgement of Erez' Israeli background, they do posit that cultural factors may play a role, but this is the extent of their controls for the possible role of social, psychological, or demographic factors. Accounting for all possible mitigating factors is infeasible in any research; yet failing to concede their possible role weakens the authors' claims. These problems, however, are not unique to this study and do not detract from the strengths of the work.

1) Timing of the article or addressing a need in the literature
The first criterion for quality research is the degree of its timeliness or filling a void in existing literature. It is the latter point, that I personally believe touches upon the work’s most exemplary quality: its innovative approach. Cummings and Earley believe this approach is novel and effective; they recommend it be adopted by other researchers as it leads to a full understanding of phenomena (1992). In the history of intellectual thought and research, it is likely that scholars with opposing views have collaborated before. Within academic literature "antagonist" collaboration appears rare enough that Latham, Erez, & Locke believe they have pioneered it (1988). Although this claim is not substantiated, it appears no one has refuted their claim. Latham, Erez, & Locke may not entirely have invented collaborative and mediated research, but they are clearly pioneers in its refinement and promotion.
I admire their approach as it offers a constructive alternative to academics with divergent views from arguing and belittling each other in protracted arguments in literature and at conferences. It appears Greenberg is similarly impressed by this innovative approach noting, "Instead of merely adding to derision, as so often occurs, the authors agreed to some crucial experiments designed to answer a key question to their mutual satisfaction. We need more of this kind of work, in my opinion" (as cited in Wheeler, Richter, & Sahadevan, 2004, p. 1337). Greenberg and Cummings & Earley, and Greenberg all comment on the rarity of any such collaboration, which may be due to the risks involved. Either researcher could have been exposed as incorrect or having conducted prior faulty research. I find this an exemplary quality of the researchers that to me reveals their commitment to quality scholarship over protecting their own ego or reputations. I thus commend Erez and Latham both for pioneering this innovative approach and for committing to it.

2) Defining or clarifying constructs in the article
A key problem with Erez and Latham’s earlier work, and a problem not unique to them, is that there were unstated assumptions, unknown conflating variables, and undocumented procedures. Another additional exemplary trait of Latham, Erez, & Locke’s article is their how their collaboration enabled them to uncover these mysteries to deliver a solid, rich understanding of the phenomenon. This quality is remarkable as Wheeler, Richter, & Sahadevan state, "articles that become ‘great works’ define or clarify constructs in a way that future researchers rely on" (2004, p. 1329). Frost and Stablein in their commentary of their exemplary works believe that researchers' attention to detail (p. 249) leads to quality research. Precision in research design is crucial not only to ensure a study’s reliability and internal validity, but more importantly to deliver construct validity. It seems evident from Erez and Latham’s writings and commentary on the article that their previous studies lacked construct validity – they were not studying exactly what they thought they were studying. Their studies had conflating variables that meant that prior control groups were exposed to differing treatments. In addition, their operational definitions of a key construct, what constitutes a group, differed fundamentally. Erez defined it as a group of five people with equivalent power, whereas Latham defined it as a dyad with a superior and subordinate. Such differences not only limit the ability to compare findings across studies, but undermine the validity of the study. Latham appears to have been studying a completely different phenomenon than what he states as his definition is at odds with common definitions of a group and fails to account at all for the overriding power dynamics of his dyads. Instead of studying PDM, Latham was studying boss-employee dynamics. Erez describes how such differences and omissions can arise:

Very often, researchers are even unaware of contextual differences because they are part of the context and have no external reference point. By working together, the two researchers provided each other with the reference points needed to define the unique characteristics of each other’s procedures…. The most striking part of this process for Locke was the number of differences in procedures and design that can occur when two people are allegedly studying the same phenomenon. The number of little differences between studies can add up and have a significant effect on the differences in empirical findings and hence on conclusions (1992, p. 162).
It is crucial to clearly operationalize the constructs being studied and control for (or account for) as many variables as possible, to be able to make definitive claims about a phenomenon. Campbell, in commenting on the article, commends the authors for their clear, unambiguous operationalizing of key constructs. He observes it is more common for researchers to "underspecify the nature of the variables we use" which creates "surplus meaning" (1992, p. 174). Citing the Hawthorne studies, Campbell believes a lack of clarity on what exact variables were manipulated leads to speculation over the results. Latham, Erez, & Locke may not have accounted for all possible variables, but their efforts to uncover their differences resulted in a study with a high degree of construct validity that allowed causal relationships to be determined about a precise phenomenon.

3) Focusing on multiple levels of the organization
Wheeler, Richter, & Sahadevan’s third quality criterion is less well articulated than their other criteria. They offer little explanation beyond that quality work should "interest both organizational scholars and practitioners" (p. 1329). One can infer from their cited examples that research should appeal beyond narrow, cloistered corners. It is not surprising that scholars writing on business and management topics would value a study’s application beyond academia, considering the subject’s direct relationship with industry. The subject of Latham, Erez, and Locke's article is organizational behaviour, which is generally more readily applicable to field contexts than other academic subjects. More specifically, Latham, Erez, & Locke do present a phenomenon with implications for both scholars and practitioners. Their work centers on a common organizational activity, goal setting, and its outcomes, performance and commitment. Practitioners can apply the findings to their own work, while scholars are offered an examination that addresses both praxis and ideology. By focusing on a research problem with direct application to business settings, as well as addressing the larger issue of power and equality, the study is exemplary in its appeal to both academia and industry.

4) Becoming a foundation article
Regarding Wheeler, Richter, & Sahadevan’s final criterion, "becoming a foundation article", the authors believe that the degree of success in the preceding criteria will beget this one. Although Latham, Erez, and Locke have distinguished academic careers, it is arguable whether or not the article’s 110 citations in SSCI would designate this as a foundational piece. Examining the article in Google, however, does reveal that the article’s importance extends beyond refereed journals as it is referenced 262 times. The influence this article has for inspiring or enlightening scholars is less hard to pinpoint when it does not result in a citation. Initially, the article’s quality was affirmed by peers when the Academy of Management awarded it the outstanding publication in organizational behavior in 1988. More recently, in 2004 Professor Greenberg, one of Wheeler, Richter, & Sahadevan canonical authors, credits Latham, Erez, and Locke’s article as the work he most admires. The inclusion of the work in Frost and Stablein’s collection further adds to this work’s influence beyond its citation counts, as the book is reported to be required reading in many graduate classes (Rafaeli, 2009). Its continued relevance is also demonstrated by its citation history, provided by Scopus. Scopus credits the article with 76 citations since 1996, the year Scopus’ database begins. Since that year, however, the article has been cited every year - often up to eight or nine times. Being cited for over 20 years and the article’s more intangible influence as demonstrated by its textbook status does make a strong case that this work can be considered foundational.

The collaborative research by Latham, Erez, & Locke provides a model of a productive way to "add to our knowledge more efficiently and more rapidly than would otherwise be the case" (Latham, Erez, & Locke, 1988, p. 771). Their innovative approach resulted in a study which generated more insight into the phenomenon of PDM than their prior, individual work had uncovered. I believe they were able to generate this insight as their joint, even antagonistic, efforts lead them, even forced them, to conduct research with an exemplary degree of validity and reliability. Over twenty years after being published, Latham, Erez, & Locke’s article is still read and cited by scholars and practitioners thus maintaining its influence and claim as an exemplary work.

Campbell, J. P. (1992). Experiments as reforms. In P. J Frost & R. E. Stablein (Eds.), Doing Exemplary Research (pp. 173-176). Newbury Park, CA: Sage Publications

Cummings, L.L., & Earley, P. C. (1992). Comments on the Latham/ Erez/ Locke study. In P. J Frost & R. E. Stablein (Eds.), Doing Exemplary Research (pp. 167-172). Newbury Park, CA: Sage Publications.

Erez, M. (1992). Reflections on the Latham/ Erez/ Locke study. In P. J Frost & R. E. Stablein (Eds.), Doing Exemplary Research (pp. 155-166). Newbury Park, CA: Sage Publications.

Frost, P. J., & Stablein, R. E. (1992). Doing exemplary research. Newbury Park, CA: Sage Publications.

Latham, G. P., Erez, M., & Locke, E. (1988). Resolving scientific disputes by the joint design of crucial experiments by the antagonists: Application to the Erez–Latham dispute regarding participation in goal setting. Journal of Applied Psychology, 73(4), 753-772.

Rafaeli, A. (2009, March 11). Professor Anat Rafaeli. Technion - Israel Institute of Technology Retrieved February 10, 2010, from

Wheeler, A. R., Richter, E., & Sahadevan, S. (2004). Looking back on their “great works”: Insights from the authors of great works in organizational behavior and human resource management. Management Decision, 42(10), 1326-1342.

Thursday, March 25, 2010

Conference Presentation on Web Accessibility Challenges

I recently presented for my first time at an academic conference. The conference, Information Access: Commons, Control & Controversy, was held at my home faculty, iSchool at University of Toronto.

I presented on my research findings from my masters thesis. I thought it distilled my findings in a manageable way, so I thought I'd share it here.

Barriers to Adoption and Implementation of Web Accessibility
I'm Glen Farrelly, a first-year PhD student here at iSchool. My background and interests are in designing and understanding online user experience. My utopian dreams of the democratizing potential of the Internet have not been completely quelled by the realities of the ongoing digital divide. From this perspective, I'm concerned with how to create online experiences that strives to enable everyone to participate.

Web accessibility refers to users being able to perceive and operate publically available websites regardless of their abilities. Users can be blocked by design or code barriers based on: vision, hearing, cognitive ability, and mobility or motor control.

In line with the social model of disability, I believe that it is artificial barriers that create disability that may not otherwise exist. These barriers can apply not only to permanently disabled people but for those temporarily or situationally disabled as well.

Internet enables and disables
Many people use the Internet and find it helpful for various facets of their life from work to entertainment. The Internet has the potential to aid disabled people to overcome some barriers. For example, online shopping for people with mobility impairments helped reduced their self-reported rates of disability (Spillman, 2004). Also deaf people have been avid adopters of text and instant messaging as a preferred method of communication. It is for reasons such as these that lead to 54% of respondents to a UK survey (2002) listing Internet access as essential, compared to only 6% of the general population.

Despite this potential, accessibility rates in Canada and globally remain very low. The UN commissioned a review of popular websites in 2006 of 20 countries and only 7% passed the most basic accessibility measures. Inaccessibility of online content was attributed to result in less than half as many disabled people using the Internet compared to the nondisabled, in an US Study (Dobransky & Hargittai, 2006). This issue also affects more Canadians than most people realize, as 14 per cent of, or 4.4 million, identified themselves as disabled (Statistics Canada, 2007).

Web inaccessibility
Here’s an example U.of T. students and faculty may have encountered. The low contrast of the text on the navigational menus is hard to read probably for most users, but for those users with vision impairment this can be a complete block. Another problem with this webpage is the fan out menu at times requires very precise mouse movement or it closes. This is a block for those with motor impairments, such as those with Parkinson's disease. Personally, I can attest it's extremely difficult to use on a laptop.

Considering the diversity of human ability, it is unlikely that anyone can build a website that everyone can use. The goal then is to make content as accessible as possible.

The solution?
There are three approaches to making online content more accessible:
  1. Adaptive technology can allow for multimodal outputs or alternative inputs. For example, screen reader software can read the contents of a webpage to a blind person. But even though tech continues to get more sophisticated there are still problems, for example screen readers cannot compensate for images that don't have alternative text to describe it.
  2. Automated conversion can be done various ways. This month YouTube opened up its free software to automatically caption user videos. Solutions like this may address the challenge of making the extreme long tail of online content more accessible but they still present barriers. For example, I'm dubious that autocaptioning software will be able to compensate for poor audio levels, overlapping speech, slang, etc.
  3. Human development is probably the best way to make the content more accessible. That is, the content creators build the content in ways that extend its functionality and flexibility. For example, describing the meaning of an image is best done by its creator.
To aid practitioners in the nebulous goal of making their content accessible, the W3C, in collaboration with academic, industrial, and disability organizations worldwide published the Web Content Authoring Guidelines (WCAG) in 1999. WCAG is the world’s predominant guidelines and informs other guidelines such as Canada’s CLF and the US’ Section 508. There are 3 degrees of voluntary compliance. The W3C has focused their efforts on further refining the guidelines, I'd say more so than on education and outreach.

In Canada, at least, no organization or government office is leading the charge to deliver educational resources, training, or awareness campaigns. As accessibility is not expressly required in any jurisdiction I am aware of and as disability issues can be ignored by corporate power, web accessibility has tended to fall on individual practitioners.

The way to make more accessible websites is there, but practitioners have to find out about it themselves, figure out how to do it themselves, implement it themselves, and often make a convincing business case to do so. Numerous studies have shown however that most websites in a variety of sectors are not even following the most basic guidelines. To help make the web more accessible, I felt it is important to understand what are the stumbling blocks to adoption. To do this, one needs to understand those our society has charged with implementing this issue: web practitioners.

Web accessibility adoption challenges have largely been ignored both in academic and trade literature. In addition, research actually consulting web practitioners is even more rare as it appears that there are only three such studies.

My research
I conducted 21 semi-structured interviews with a variety of web practitioners across Canada. I asked practitioners not only how they encountered the issue of accessibility and disability, but also how they implemented in on their day-to-day job. They offered both their obstacles and solutions.

Web Accessibility Diffusion & Implementation Model
Out of the interviews various diffusion and implementation factors were identified, which lead me to develop this model.

Flowchart showing the web accessibility's diffusion challenges, which are education, media, societal attitudes, market forces, and policy and the implementation challenges which are the guidelines, instructional material, authoring tools, testing support, hired help

Diffusion refers to how a practitioner hears about an issue, learns the specifics, and formulates attitudes. The diffusion of accessibility is also affected by knowledge and perceptions about disability in general. Implementation refers to the factors that impact a practitioner's ability to produce accessible online content.

Web practitioners
The first challenge in this issue is that practitioners are not a homogenous group, but represent various responsibilities, backgrounds, and skills.Web practitioners include:
  • programmers
  • designers
  • developers
  • webmasters
  • producers
  • writers and editors
  • animators
  • assistants
  • managers
  • bloggers
  • podcasters
These roles can be performed by professionals, volunteers, hobbyists, and those with multiple non-web responsibilities. With web 2.0, now almost anyone can author web content. There are challenges with web accessibility with professionals, let alone the additional challenges with amateurs due to the differing levels of skill and commitment. My study focused on professionals and avid amateurs, but the larger scope is clearly a considerable obstacle.

Innovations are less apt to diffuse if they are difficult to understand or use, require significant resources, or are incompatible with existing norms and values. Web accessibility meets all these criteria.

The guidelines are highly technical and yet like much of disability issues, education and media were found to not adequately cover the topic. Most participants recalled that they had not encountered this issue suitably - or at all - in their career training, in the media or trade events.

While many accessibility measures can be implemented with no additional costs, some measures do require more effort and scale and competitive environment can be a factor. One participant who managed a large site with thousands of technical reports had to remove the bulk of them as it was taking days to convert a single report. There appears to be no specific tax breaks or financial incentives to help mitigate this cost.

As far as I'm aware, there are no laws expressly requiring websites to be accessible – although there are prohibitions on discrimination. Federal and provincial governments have passed policies requiring their own websites to be accessible. But Ontario is the only jurisdiction in Canada pursuing requiring web accessibility for all business and organizations that operate in the province. Even legally requiring accessibility is not guaranteed to increase compliance as a Brazilian study found. It also introduces a new problem of policing.

In interviewing practitioners who had tried to or did implement accessibility, they were frustrated with the inadequate support. Most criticism was directed at W3C and WCAG. Two participants described using it: "If I can’t navigate it, who the hell can?" and "It just gets into tech-babble. It must be completely overwhelming for those less experienced." The guidelines are not particularly accessible – they are a quagmire and are written at a level that even veteran professionals found too difficult to interpret. There are reasons for this, but W3C has not done a sufficient job at making their guidelines usable, offering help, code libraries, and tools. Instructional material is out there but not in a prominent, central location. It can be hard to find, dated, time-consuming, and dubious.

Popular commercial authoring tools such as Dreamweaver are improving their support for accessibility. With content management systems and web-based software becoming increasingly popular, however, there is a continuous need to improve functionality and transparency of features.

Similarly, testing software to automatically check for accessibility is insufficient and often unknown by participants. Although testing software is improving, certain items must be tested with disabled users. In general there is a lack of any user testing, so the need for human testing is problematic.

Considering how difficult this is some practitioners have turned to hiring experts. As there are no organizations in Canada offering free guidance – even CNIB charges for accessibility services - web accessibility is being commercialized. Alarmingly, three participants hired so-called accessibility experts that amounted to charlatans and failed to deliver on some basic accessibility measures.

No matter the societal changes, software sophistication, or increased support, the onus will still fall on practitioners to make the line by line and graphic by graphic changes on a daily basis to make web accessibility a reality. To do this, practitioners needs to be aware of their important role in the disability divide. As one participant put it:

Web developers can be proactive, so they need to let companies know it’s an issue and that including it is just part of our standard services. No companies would say no to this. Or you can just do it.

Finally, research is needed to study how web practitioners work, how they use the guidelines, support tools, and software. This research can then be used to make the implementation tools and resources more usable and transparent.