Online Research: New Challenges and Opportunities

I delivered a guest lecture this week on online research - both online methodology and researching online phenomena. Below are the slides from my presentation and a coding exercise of online discourse I had the students do.


 

Online Research: New Challenges and Opportunities from Glen Farrelly

Methods to Study Digital Media

I'm teaching a class next week on "Online research: New challenges & opportunities". In preparation, I was thinking of all the ways to study online (and by this I mean both Internet and mobile) phenomena and to use online methods to study other things.

As I love typologies, I thought I'd prepare one on this.  Although such a list doesn't express the complexity of online research, I thought it would demonstrate the diverse ways to investigate digital media.

Online methods to examine online phenomena:

• computer-captured and compiled data (e.g., web metrics)
• email, virtual reality, VoIP telephone or video call, or instant messaging interviews
• diarying or user logs
• remote observation (participant or nonparticipant)
• online focus groups
• web-based or email surveys
• audience response systems
• remote user testing or experiments
• autoethnography (can be conducted through blogging)

Methods to analyze online phenomena:

• social network analysis
• semiotics or visual analysis
• content analysis
• discourse analysis 
• hermeneutics
• ethnography

Online methods to study offline phenomena:

• web-based or email surveys
• email, virtual reality, VoIP, or instant messaging interviews
• diarying (e.g. through special software or blogs)
• photo documentation via mobile device

Offline methods to study online phenomena (which may or may not involve having users interact with digital media while capturing data):

• face-to-face interviews or focus groups
• nonparticipant observation
• contextual inquiry
• verbal protocol analysis (talk-aloud method)
• eye-tracking studies
• user testing or experiments

As a colleague pointed out there's also design science and participatory design that involves creating digital media as research tool - this type of research could be place in every category.

Please let me know if I missed a major method, as I'd like this typology to evolve iteratively.

Research Proposals Methodology Checklist

Earlier this month, I finished work as a teaching assistants for my department's research methods class. I've TAed this for the past 2 years and I love it. There are so many (almost countless) research methods and planning a research project is probably the most fun part of the whole thing.

It's also great to read about what the masters students plan to study and the various dimensions of research possible in the field of Information. I've graded a large number of research proposals now and have been impressed by students' approaches and their research interests. However, even in the best proposals they are often missing key elements.

I put together a checklist of items to include in a research proposal for a guest lecture I did for the class. Some students found it useful, so I thought I'd share it here. Surprisingly, I haven't found a concise checklist like this in research methods texts.

I'm covering the major items that apply to common methods, such as interviews, observation, focus groups, discourse analysis, and surveys.

Sampling

• Population - describe the characteristics of the group you will be studying, mention any pertinent demographic (e.g. age, gender, location, occupation) and pyschographics (habits, attitudes, hobbies, opinions) -  if studying texts or artifacts, describe the unifying characteristics
• Sampling strategy - e.g. census, random, stratified random, or nonprobability sampling such as  convenience, purposive diversity/dissimilar, snowballing, key informant, etc.
• Sample size - number and if this number is appropriate (or the stoping point, e.g. saturation)
• Access - how you will get access to this group and permission to study them
• Recruitment techniques to be used - e.g. posters, web posts, emails, etc.
• Incentives - if using why is it necessary, how much, and how to be distributed
• Inclusion and exclusion criteria - conditions to be met to qualify or exempt people or texts from study

Method considerations

• Format - online, print, email, instant messaging - which one(s) and why
• Interview guide or questionnaire - how will questions be determined (if appropriate)
• Questioning style (e.g. structured, semi-structured, conversational for interviews) and question format (i.e. open or closed ended)
• Probing or follow-ups if used and when (e.g. during or after)
• Discussion aids - will think-aloud protocol be used, artificts (e.g. diaries, photos) or cultural probes
• Rapport - how to establish trust, confidence, and ease amongst participants

Session preparation

• Meeting location details - discuss any pertinent details such as the type of place to be held and why (e.g. convenient for participant, comfortable setting), logistics, arrangement of furniture, noise levels, position of camera, presence of others, etc.
• If communicating or observing online discuss the technology to be used and any applicable norms or constraints

Data collection

• Recording of session - audio or video taping, photodocumenting, log files
• Note-taking - during or after session?
• Third-party observers or facilitators - whether or not used, relationships to researcher, training provided, and any issues that may result
• Role of researcher - researcher biases, how will the researcher be involved in and shape data collection and any steps to mitigate or place the researcher (e.g. passive observer or participant observer)

Data analysis

• Transcription - style to be used (e.g. naturalistic, selective) and whom will do it
• Approach to data analysis (e.g. grounded theory, statistical tests, etc.)
• Software - name and describe how used
• Coding technique used and how codes determined
• Reliability measures - particularly if more than one person is coding results

Data presentation & dissemination

• Anticipated findings
• Presentation formats - e.g. case study, charts, narrative, performance, etc.
• Outcomes - e.g.recommendations, program evaluation, etc.
• Dissemination - planned conference presentations, sharing among an applicable association, publication plans, etc.
• How might the findings be shared other than a traditional paper (if applicable)?
• Sharing results with participants - e.g. send them the final paper and - will they get permission to edit?, post summary on blog,or participant community (e.g. trade association)

Ethics

• Considerations when studying a given group
• Informed consent - how it will be obtained and special situations (e.g. minors or other who may not be able to provide it)
• Deception - if used, explain why necessary
• Harm - plans to avoid any emotional or physical harm to participans, e.g. if sensitive topics raised how will this be handled, referals to specialists, etc.
• Debriefing plans (if applicable)
• Anonymity or confidentiality - steps taken in data collection,storage, and presentation to protect participants' privacy

Obviously an entire book, or several, could be written on this topic. This is meant to touch upon the major and common areas. But please let me know if I missed something crucial.

Sense of Place & Location-Based Services - Survey Summary

This past December, I designed a survey as part of my coursework for a PhD in Information at University at Toronto. As promised, I'm sharing my early results of the survey.

The survey sought to uncover foundational knowledge about how people form a sense of place (defined as the feelings and meanings attributed to place), the role of information in this process, and the potential of location-based services (LBS - defined as mobile applications or sites that deliver content based on a user's location). As location-based services are still an emerging technology, this subject has not been sufficiently addressed by academia or industry.

I'm still in the process of compiling and writing up the results, but here is a high-level summary of the findings grouped by research questions.

78 people completed the survey, with participants representing a cross-section of ages, education levels, and residences. 86% have a mobile device, with smartphones the most popular (68%) followed by GPS navigation device (31%), tablet (22%), and netbook (18%). Participants can be described as frequent users as over 80% (53 participants) indicated they used their device at least once a day.

1) What is the nature of relationships people have to places they encounter?

• 96% of participants indicating there was a place in which they associated strong feelings or meaning.
• When asked what place characteristics are meaningful, past personal experiences, physical qualities of place, and social dimensions received the strongest support with all over 75%.
• These findings confirmed phenomenologist place theorists, who believe that a relationship to place is a foundational experience in human existence and individual experience is the paramount way we process place.
• The factual elements of a place, its history and role in culture, were selected by at least half of respondents, the lack of stronger support for the these qualities was surprising as LBS can offer unique functionality to deliver this information (perhaps leading LBS have yet to effectively tap this).
• When describing a place where participants felt a "strong sense of place" in an open-ended question, 58 participants recounted experiences revealing personal experiences tied to place - a good example of this is: "Another place with a strong sense to it is an ordinary intersection... where I was told 'I love you' for the first time, in the romantic sense...the spot is still very dear!".
• Often accounts such as this reveal facets of a place hidden to others but are nonetheless significant - through LBS these types of accounts can now be made more visible.
• Descriptions of meaningful places often (67% of respondents to this question) mention emotions tied a place. Top mentioned emotions are nostalgia (fond remembrance), contentment, and excitement.

2) How are people currently using information in forming a sense of place?

• All participants agreed that access to information while at a given location is valuable.
• When asked to describe how they come to know of a specific place, experience was echoed here, with 50 participants indicating this was a primary information source.
• Personal experience was often described as an initial source of learning about a place, but was then followed by consulting various other information sources to learn more about place.
• Approximately half of respondents using two or more sources of information when learning about the place.
• When asked specifically the information sources participants use when visiting a new place for either a short (less than 1 day) or extended visit, the top five sources are (in descending order): Short visit - social network/word of mouth, website, pamphlets, mobile application, plaques. Extended visit: website, social network/word of mouth, print book, pamphlets, video or tv program.
• People are more apt to use an information source for extended visits than for short visits with the exception of mobile applications and plaques.
• In general, the more a visit demands of a person the more apt participants are to invest the time to consult a greater number of sources and more in-depth sources.

3) How, if at all, have mobile devices affected sense of place?

• The majority of participants with a mobile device are using it, in relation to place, to find location, directions or contact information.
• There is strong indication that participants are currently using mobiles to seek out facts about place beyond directions. This behaviour can be seen in the following comment: "I found out recently not to use a public car park ... after dark. The Foursquare entry had a tip that car thieves were targeting the car park".
• The ability to use mobile devices to record or share place was described by some participants as changing their experience of the place, as illustrated by this comment: "The process of inscribing your thoughts on a place while at that place is very useful in concretising your thoughts on that place. I think the practice of reflecting on the place to compose an entry requires a thinking about that place, and an ordering of thoughts about that place that gives meaning to the place (whether good or bad) and which in turn improves the memory one has of that place."
• Participants also expressed comments that they are experiencing the physical and digital worlds simultaneously, along the lines of "hybrid space" as discussed in academic litearature, this can be seen by this comment: "I pulled up a mobile app the other day when I took my children to a museum, so that I could supplement the information at the museum with information from the mobile app."
• The larger the device screen the more apt participants are to find something useful.
• The type of mobile device used does not have a strong association with usage of geotargetted information seeking via mobile device, the creation of place-based user generated content, or the likelihood to use a LBS.

4) What is the potential of location-based services to improve sense of place?

• Participants were not big users of LBS applications - of the people with a mobile device less than half have installed one of the most popular LBSs.
• The most installed used mobile applications with LBS functionality used by participants were Foursquare at 23%, Google 23%, and Facebook at 22%.
• All three of these, with the exception of Google’s Places, predominantly revolve around geo-social networking. Although place-based information can be found in the form of "tips" left by friends or made public, the core functionality of these leading apps are tracking the whereabouts of one’s social network.
• Of the participants using LBS, they expressed appreciation for the ability for the applications to enhance their sense of place by learning more about a place, as typified by these comments: "I think they have affected how much I can find out about a place and provides access to types of information about a place that I wouldn't otherwise have access to - provides richer dimensions to knowing about a place" and "Location based services have, by bringing other peoples social gazetteers into consideration, made a sense of place easier to achieve."
• LBSs were also mentioned by participants as improving their attachment to place.

5) What could be done with mobile technology to improve sense of place?

• To meet the needs of users, however, there improvements to LBS or mobile technology in general are need. Over 60% of participants offered suggestions.
• The top five suggestions were:  better content (32% support) , new functionality (26%), personalization features (21%), user experience improved (21%), and content available in more locations (17%).

Final thoughts

As one considers the possible implications for location-based services accessed via a mobile device to affect sense of place, the current iterations of technology may not reflect the qualities of future versions. The LBS market is rapidly evolving and new features and innovations as well as improvements to existing features will undoubtedly continue. Ideally, I would like my research as it continues to be able to inform future developments in this area.

Thanks to everyone who participated in this survey!

I held a random draw for an iTunes giftcard and a participant from Vancouver was chosen.

I'll revise and enhance these findings as I continue my data analysis. But I'd love to hear any thoughts below on these early results.

The Role of Geotargetted Information Via Mobile Devices in Shaping Sense of Place

Moving to the downtown core of a large city from my semi-suburban hometown, I felt impressed by - yet detached from - my urban surroundings. I enjoyed the heterogeneity of the architecture and historical roots of my new city and also appreciated the diversity of civic and citizen cultural locations – so unlike my hometown. Although I had visited the city before moving there and was familiar with the prominent buildings and main streets, I lacked a sense of the place. As a newcomer, I did not have the personal contacts to receive the insider knowledge from my neighbours.

The layers of meaning implicit in a given place include the social, historical, political, and personal. An example of these dimensions can be seen by looking at Toronto Street in downtown Toronto. It is easy to miss Toronto Street as it runs only one block, but it used to be a main thoroughfare. Today, Toronto Street is populated by a handful of generic, low and medium rise office towers. What is not immediately apparent is that this street is where Conrad Black destroyed evidence, that it has a restaurant with amazing paninis, that there is a hidden parkette providing an oasis of rose bushes and fountains, that falcons have been reintroduced nearby and often swoop and prey overhead, or that it is the former site of public executions, including leaders of the Upper Canada Rebellion.

Until recently one would not have been easily able to tap into that diversity of information instantly and on the spot. To learn the above information about Toronto Street (also the site of my former workplace) and the other new places I regularly traversed, I consulted various information sources, such as periodicals, books, and walking tours. Eventually, I got to know my new city well. Not only did I develop a strong sense of the civic and social history of the place, I also developed personal relationships to the places I frequented as my life experiences grew.

Sense of place is foundational, as it not only aids individuals in developing attachments to their surroundings, but it also contributes to the formation of personal identity. It is helpful at this point to note the geographic distinction between space, the physical terrain and features of Earth - and place, the meaning humans ascribe to space. Differing from spatial cognition with a psychological focus on our perceptions of space and way-finding, scholars of sense of place often consider phenomenological aspects. Drawing upon the philosophers Husserl and Heidegger, sense of place can be defined as how an individual conceives of space and ascribes meaning to it. Having a sense of place is seen as a fundamental component of human identity from fostering community to shaping our mindful existence.

Geographic technologies such as global positioning devices and geographic information systems have introduced powerful abilities to analyze and visualize space and place. Recent market and technological developments have now given citizens access to powerful geographic tools. The convergence of distributed network access offered by the Internet, growing ubiquity of mobile devices, and open geographic information systems (such as Google Maps), have propelled increasing user functionality for location-based applications. Place is no longer a back-drop for information seeking, creation, and sharing as current mobile applications can customize information based on a user's geographic position.

Information objects from fiction to non-fiction are rich with geographic references whether as subject, setting, or - in all cases - the location of the publication or production. Due to a lack of widespread georeferencing of information sources in libraries, on the Internet, or elsewhere, my search strategies were restricted to what turned up with a key word and category search. The novels of Margaret Atwood, for example, that refer to my neighborhood never turned up in a search. Ontario's public libraries are increasingly offering local historical information in digital format, such as the Ontario Time Machine and Knowledge Ontario's OurOntario.ca projects, yet rarely are such projects georeferenced, let alone optimized for viewing on a mobile device. Although I now feel a deep sense of place, it was a delayed and haphazard process that resulted in years where I felt detached from place.

With the growing ubiquity of mobile location-based applications, I was struck by the question of whether this technology could help foster a sense of place. Although numerous human geographers have grappled with the relational, cultural, and perceptual aspects of place, few scholars appear to have examined the role of information in the development of an individual's sense of a place. Moreover, the specific nature and impact of on-the-spot geographically relevant information appears to have not been adequately examined. Although my research is still in the formation stage, I hope in my doctoral research to explore how mobile location-based information systems affect one's sense of place. Over the months as I explore this research topic, I plan to blog about my findings and the nature of sense of place and geotargetted information.

Update on My Research Plans

Yesterday, I presented for my faculty an update on my research status. It includes my intended research topic and plans.

I got some great feedback and even more things to read and even more angles to consider. I was originally very keen to examine how a given mobile application/site's interface and interactions affects usage and could possibly be improved. I've been encouraged to focus on advancing theory more so than any possible applied findings. But two of my panelists, both professors at iSchool encouraged me to continue with an applied focus too.

The presentation here as a stronger focus on the background theory and possible contributions therein.

To see the speaker notes, click through to the actual full presentation on Google and click to open the speaker notes pop-up.

Any suggestions thoughts would be great.

Pondering Effects of foursquare

Over the last year, I have been researching and contemplating usage of the location-based mobile application foursquare. A one-time avid user, my own usage has lessened significantly over the last few months. This is due to the loss of novelty for me, a lack of critical mass of my friends using it, almost non-existent financial incentives, and foursquare's interface limitations.

My interest has perked up recently, however, when I got my first real-world reward for using it (a free, yumlicious gelato from Toronto's Hotel Gelato). I believe in the potential of the location-based services are great and will continue to watch them.

So I was intrigued when I was recently contacted by a fellow PhD student, Leighton Evans, who is also studying location-based services.

Leighton describes his research:

I'm interested in the effects of using location-based services and mobile phones as navigational devices. Traditional maps imposed one kind of spatial and cognitive orientation with regards to physical space, my research asks questions of whether mobile devices are offering a new type of reasoning of this kind, and what the implications of any change might be for the future.

I was similarly drawn to foursquare as I believe it has the power (if yet, often unrealized) to help citizens define and annotate their space for themselves, so I find his work to augur at the profound changes that will results from location-based changes.

Leighton is looking for foursquare users to be interviewed via email. If you are interested in participating or want more information please contact him.

Here are his questions with my corresponding answers.

Question #1: How has using foursquare (or any other location based service) made you aware of, or more aware of, the places around you?

Answer #1: foursquare was the first location-based service I used or had experience with, so initially I found it very useful to discover new places around me particularly restaurants. foursquare's interface is not well suited to specific searches by type of place or even for nearby venues (seems to get proximity wrong a lot and miss tons of key places), so when I heard about the YellowPages app, I switched to that for finding businesses and associated contact info.

I was hoping that foursquare would provide richer understanding of places I am in. Other than finding a hidden washroom at a subway station I frequent, this generally has not happened. Most of the comments on place are quite superficial, well-known, or narcissistic.

Question #2. Do you feel that in using your location services through your mobile device, that device has in effect become a tool for navigating your way through the world? If so, would being denied that tool affect your ability or desire to explore new places?

Answer #2: I would like to say that LBS has significantly affected my wayfinding or relationship to place - and I think it has the potential to. Other than finding businesses in a geotargetted fashion, however, LBS has not done this. That said, when I was recently on a personal vacation to Chicago (see my blog post), I was impressed at how much the city was embracing foursquare and using it as a tourism tool. Due to the exorbitant roaming charges by my carrier, I wasn't able to make use of this. But I constantly felt that there was so much that my mobile could do to greatly enhance my visit there.

Question #3: Do you have any experiences of using the tips left by other users and them actively being involved in the choices you make, either visiting a place or checking-in to a place?

Answer #3: Yes, I tried a restaurant that I had not previously heard of based on seeing various check-ins for it on foursquare. Also, at a restaurant I have chosen a menu item based on a prior customer's tip. I really appreciate this feature but have found that as foursquare grows the amount of noise and the lack of ability to sift through this has made finding valuable info like this very difficult and not worth the effort as other sites do this better (e.g. OurFaves).

Question #4: Has using foursquare (and using GPS) changed the importance of your mobile device compared to other phones or PDAs you used in the past? Does the device feel more integral to your everyday life than previously?

Answer #4: Location based services have been a nice-to-have functionality of my mobile and I think will eventually become integral functionality for me. Currently, however, it has not provided completely new, crucial functionality for me. I can get the same functionality through other, often easier, means.

Question #5: How aware are you of your check-ins and activity on foursquare being a means of providing information and feedback to the application and software itself? Does any such awareness influence your usage and choices with the software?

Answer #5: I have ambivalent feelings towards user-generated services that make their money on the free labour of their users (I like how an author referred to this as "loser generated content"). But services such as foursquare and Facebook do provide the infrastructure that offer me significant value such that I don't mind this potential exploitation. Regarding privacy, I have no concerns about the application knowing my whereabouts - I am not overly concerned with privacy and generally don't check-in anywhere that I would not be willing for anyone in the entire world to know.

Question #6: Given that foursquare is a social application, and social networks involve a measure of impression management, are you someone that checks-in to places with an awareness of how that contributes to an image your online friends have of you or that you are trying to create?

Answer #6: I have been acutely aware of how foursquare helps project identity. I noticed that many people only check in at hip, high-status locations or their work. I have never seen a check-in with anyone I know at a big box store or fast-food chain. Because of this I found it fun to deliberately check into Burger King and Walmart, for example, to make a point. Everyone goes to these types of places but why the reluctance to check in there? This trendiness that foursquare seems to provoke appears superficial and contrived to me. I am guilty of this as well as I am definitely more apt to check into a place that I - and my friends - perceive as interesting or cool.

Question #7: How has using location services changed your perception of the world?

Answer #7: There were a couple moments when foursquare strongly affected my relationship to space and others in it. The first one was when Toronto had its worst earthquake in over a hundred years. It wasn't a big one, but it was an odd, bewildering experience. I turned on my mobile and noticed that others had been checking into the earthquake (the location was the entire city). I felt much more connected to my fellow city inhabitants somehow through that. I also watched virtually the locations and people of Toronto's G20 protests. Even though these were extensively covered by the media, using foursquare somehow made it feel more real and nearby opposed to TV that feels distant and foreign even when it is local coverage. In general, foursquare does allow one (i.e. me) to leave my imprint on a place that does connect one more tangibly to a place. This feeling is magnified when one gets a "mayor" title, which I admit at first was quite rewarding to get. I feel these feelings less so now that I've been using foursquare for almost a year, so perhaps novelty was a key factor in this.

Please share your feelings towards foursquare here.

Foursquare - Need Help Researching

I'm conducting a study that I hope to publish and that will inform my dissertation research on foursquare.

Why foursquare? Although in its infancy, I am curious if foursquare will emerge as a leading platform for citizens to annotate their city and connect with like-mined individuals. Such online participation has larger implications, as it can foster similar participation in political, cultural, and economic realms. It is possible, however, that foursquare is popular for gaming and novelty, and it is not a new source of citizen civic participation.

Launched March 2009, foursquare is as a combination of social networking application, user-generated city guide, and entertainment/diversion. Users post their physical location, write reviews, view other users’ postings, and compete for honours (e.g. badges).

I therefore wish to explore: 1) what motivates users to participate, 2) how they use the application 3) what types of information are users seeking.

I propose to study users first by online observation of their publicly-available behaviour and then conduct email interviews. I'm also conducting autoethnography of my own usage, which I'll be posting here.

But I need help?
Please share below any things you love or hate about foursquare, problems you encountered, what you think its strengths or future potential is, any thoughts or experiences.

I also need people who would consent to be interviewed via email on their usage (5-7 questions). Finally, if there is a Toronto-area person who I could meet at a cafe and observe for about 30 minutes as they do a few check-ins that would be great. All research would be confidential.

But even if you just know of a good article or have some insight to share here, that would be greatly appreciated!

Here's my profile on my Faculty website.
http://www.ischool.utoronto.ca/students/glen-farrelly

Exemplary Antagonism

There is no established formula for determining a great work of academic research. However, Wheeler, Richter, & Sahadevan (2004) attempted to uncover the ingredients of great research by interviewing leading researchers in their field of organizational behavior and human resource management. They selected nine journal articles they personally felt were great works. No other details about their sampling strategy were provided, although the total number of citations was a key determinant. The sample of their great works begins with an article that received 124 citations (as provided by Thomson Reuters’ Social Sciences Citations Index). A study contemporaneous to that one is by Latham, Erez, & Locke (1988) which received a similar number of citations in SSCI at 110. Does this then also qualify it as a great work?

Wheeler, Richter, & Sahadevan asked the ten authors of their "great" journal articles why they felt their work were so well received. Analyzing the compiled interview data, they found researchers credit four central factors to their success: "1) timing of the article or addressing a need in the literature; 2) defining or clarifying constructs in the article; 3) focusing on multiple levels of the organization; and 4) becoming a foundation article" (p. 1328). Wheeler, Richter, & Sahadevan’s criteria allows one to move beyond the citation counts of articles and impact factors of journals to examine the qualities of exemplary research, particularly within the domain of organizational behaviour. Applying this framework to the Latham, Erez, & Locke article assists in illuminating the work’s key strengths that make it an exemplary work.

Background
Latham, Erez, & Locke’s article, entitled "Resolving scientific disputes by the joint design of crucial experiments by the antagonists: Application to the Erez-Latham disputes regarding participation in goal setting", was originally published in the prestigious Journal of Applied Psychology in 1988 and then included in the edited volume Doing Exemplary Research by Frost & Stablein in 1992. The two lead authors, Gary Latham and Miriam Erez, had in their prior work on participation in decision making (PDM) received contradictory findings. Erez’ and Latham’s mutual acquaintanceship with Locke lead to the three meeting and brainstorming possible explanations for their research discrepancies. Rather than follow the seeming tradition of slinging scholastic mud at each other, Erez and Latham decided to collaborate on a joint design of experiments to uncover their prior research differences. Locke acted as mediator to ensure a smooth process, although he states that he rarely had to intervene and never in a significant manner (Latham et al, 1988). They conducted a series of four iterative experiments, using their learnings from early experiments to guide subsequent experiment design. Their process of painstakingly analyzing their research design enabled them to uncover procedural differences and conflating variables that accounted for their prior oppositional findings. Their results constitute a crucial experiment that proved that PDM does not increase performance compared to supportively-stated goals, but that it does increase goal commitment. Through this research they made a strong addition to the knowledge on PDM and pioneered a method for collaborative problem solving.

Non-exemplary qualities
No research is ever entirely great. The authors interviewed by Wheeler, Richter, & Sahadevan all commented that they believed their work could be improved. Omissions, compromises, or errors are made for the sake of expediency or arising from a lack of omniscient knowledge. Before extolling a work’s virtues, I feel it is important to acknowledge where it falls short. Not wishing to fault research for the assumptions and traditions of its epistemological paradigm, I do believe Latham, Erez, & Locke’s article lacked clarity and evocativeness in the writing style. A complaint that experimental laboratory tests lack ecological validity is not original or entirely fair as findings from experiments have been shown to transfer outside of the lab. Nonetheless, decisions in the test task chosen, performance measurement rubrics, and test duration limit the study’s ability to be generalized to larger, field scenarios. The study is further limited in its generalizability by the authors’ sampling strategy. Despite the ubiquity in academia of drawing upon a convenient supply of university students for research (even giving them course credit, as was the case with this study), sampling from university students, and specifically business students, does limit a study’s population validity. University students, and in particular business majors, are not representative of the general population as they may possess non-standard levels of intelligence, ambition, and literacy. Finally, I believe the study failed to control for, or at least acknowledge, possible conflating variables. The study makes definitive claims about the success of participatory decision making, yet failed to examine or control variables such as personality types, gender, age, power structures, stress level, and worker morale. In acknowledgement of Erez' Israeli background, they do posit that cultural factors may play a role, but this is the extent of their controls for the possible role of social, psychological, or demographic factors. Accounting for all possible mitigating factors is infeasible in any research; yet failing to concede their possible role weakens the authors' claims. These problems, however, are not unique to this study and do not detract from the strengths of the work.

1) Timing of the article or addressing a need in the literature
The first criterion for quality research is the degree of its timeliness or filling a void in existing literature. It is the latter point, that I personally believe touches upon the work’s most exemplary quality: its innovative approach. Cummings and Earley believe this approach is novel and effective; they recommend it be adopted by other researchers as it leads to a full understanding of phenomena (1992). In the history of intellectual thought and research, it is likely that scholars with opposing views have collaborated before. Within academic literature "antagonist" collaboration appears rare enough that Latham, Erez, & Locke believe they have pioneered it (1988). Although this claim is not substantiated, it appears no one has refuted their claim. Latham, Erez, & Locke may not entirely have invented collaborative and mediated research, but they are clearly pioneers in its refinement and promotion.
I admire their approach as it offers a constructive alternative to academics with divergent views from arguing and belittling each other in protracted arguments in literature and at conferences. It appears Greenberg is similarly impressed by this innovative approach noting, "Instead of merely adding to derision, as so often occurs, the authors agreed to some crucial experiments designed to answer a key question to their mutual satisfaction. We need more of this kind of work, in my opinion" (as cited in Wheeler, Richter, & Sahadevan, 2004, p. 1337). Greenberg and Cummings & Earley, and Greenberg all comment on the rarity of any such collaboration, which may be due to the risks involved. Either researcher could have been exposed as incorrect or having conducted prior faulty research. I find this an exemplary quality of the researchers that to me reveals their commitment to quality scholarship over protecting their own ego or reputations. I thus commend Erez and Latham both for pioneering this innovative approach and for committing to it.

2) Defining or clarifying constructs in the article
A key problem with Erez and Latham’s earlier work, and a problem not unique to them, is that there were unstated assumptions, unknown conflating variables, and undocumented procedures. Another additional exemplary trait of Latham, Erez, & Locke’s article is their how their collaboration enabled them to uncover these mysteries to deliver a solid, rich understanding of the phenomenon. This quality is remarkable as Wheeler, Richter, & Sahadevan state, "articles that become ‘great works’ define or clarify constructs in a way that future researchers rely on" (2004, p. 1329). Frost and Stablein in their commentary of their exemplary works believe that researchers' attention to detail (p. 249) leads to quality research. Precision in research design is crucial not only to ensure a study’s reliability and internal validity, but more importantly to deliver construct validity. It seems evident from Erez and Latham’s writings and commentary on the article that their previous studies lacked construct validity – they were not studying exactly what they thought they were studying. Their studies had conflating variables that meant that prior control groups were exposed to differing treatments. In addition, their operational definitions of a key construct, what constitutes a group, differed fundamentally. Erez defined it as a group of five people with equivalent power, whereas Latham defined it as a dyad with a superior and subordinate. Such differences not only limit the ability to compare findings across studies, but undermine the validity of the study. Latham appears to have been studying a completely different phenomenon than what he states as his definition is at odds with common definitions of a group and fails to account at all for the overriding power dynamics of his dyads. Instead of studying PDM, Latham was studying boss-employee dynamics. Erez describes how such differences and omissions can arise:

Very often, researchers are even unaware of contextual differences because they are part of the context and have no external reference point. By working together, the two researchers provided each other with the reference points needed to define the unique characteristics of each other’s procedures…. The most striking part of this process for Locke was the number of differences in procedures and design that can occur when two people are allegedly studying the same phenomenon. The number of little differences between studies can add up and have a significant effect on the differences in empirical findings and hence on conclusions (1992, p. 162).

It is crucial to clearly operationalize the constructs being studied and control for (or account for) as many variables as possible, to be able to make definitive claims about a phenomenon. Campbell, in commenting on the article, commends the authors for their clear, unambiguous operationalizing of key constructs. He observes it is more common for researchers to "underspecify the nature of the variables we use" which creates "surplus meaning" (1992, p. 174). Citing the Hawthorne studies, Campbell believes a lack of clarity on what exact variables were manipulated leads to speculation over the results. Latham, Erez, & Locke may not have accounted for all possible variables, but their efforts to uncover their differences resulted in a study with a high degree of construct validity that allowed causal relationships to be determined about a precise phenomenon.

3) Focusing on multiple levels of the organization
Wheeler, Richter, & Sahadevan’s third quality criterion is less well articulated than their other criteria. They offer little explanation beyond that quality work should "interest both organizational scholars and practitioners" (p. 1329). One can infer from their cited examples that research should appeal beyond narrow, cloistered corners. It is not surprising that scholars writing on business and management topics would value a study’s application beyond academia, considering the subject’s direct relationship with industry. The subject of Latham, Erez, and Locke's article is organizational behaviour, which is generally more readily applicable to field contexts than other academic subjects. More specifically, Latham, Erez, & Locke do present a phenomenon with implications for both scholars and practitioners. Their work centers on a common organizational activity, goal setting, and its outcomes, performance and commitment. Practitioners can apply the findings to their own work, while scholars are offered an examination that addresses both praxis and ideology. By focusing on a research problem with direct application to business settings, as well as addressing the larger issue of power and equality, the study is exemplary in its appeal to both academia and industry.

4) Becoming a foundation article
Regarding Wheeler, Richter, & Sahadevan’s final criterion, "becoming a foundation article", the authors believe that the degree of success in the preceding criteria will beget this one. Although Latham, Erez, and Locke have distinguished academic careers, it is arguable whether or not the article’s 110 citations in SSCI would designate this as a foundational piece. Examining the article in Google, however, does reveal that the article’s importance extends beyond refereed journals as it is referenced 262 times. The influence this article has for inspiring or enlightening scholars is less hard to pinpoint when it does not result in a citation. Initially, the article’s quality was affirmed by peers when the Academy of Management awarded it the outstanding publication in organizational behavior in 1988. More recently, in 2004 Professor Greenberg, one of Wheeler, Richter, & Sahadevan canonical authors, credits Latham, Erez, and Locke’s article as the work he most admires. The inclusion of the work in Frost and Stablein’s collection further adds to this work’s influence beyond its citation counts, as the book is reported to be required reading in many graduate classes (Rafaeli, 2009). Its continued relevance is also demonstrated by its citation history, provided by Scopus. Scopus credits the article with 76 citations since 1996, the year Scopus’ database begins. Since that year, however, the article has been cited every year - often up to eight or nine times. Being cited for over 20 years and the article’s more intangible influence as demonstrated by its textbook status does make a strong case that this work can be considered foundational.

Conclusion
The collaborative research by Latham, Erez, & Locke provides a model of a productive way to "add to our knowledge more efficiently and more rapidly than would otherwise be the case" (Latham, Erez, & Locke, 1988, p. 771). Their innovative approach resulted in a study which generated more insight into the phenomenon of PDM than their prior, individual work had uncovered. I believe they were able to generate this insight as their joint, even antagonistic, efforts lead them, even forced them, to conduct research with an exemplary degree of validity and reliability. Over twenty years after being published, Latham, Erez, & Locke’s article is still read and cited by scholars and practitioners thus maintaining its influence and claim as an exemplary work.

References
Campbell, J. P. (1992). Experiments as reforms. In P. J Frost & R. E. Stablein (Eds.), Doing Exemplary Research (pp. 173-176). Newbury Park, CA: Sage Publications

Cummings, L.L., & Earley, P. C. (1992). Comments on the Latham/ Erez/ Locke study. In P. J Frost & R. E. Stablein (Eds.), Doing Exemplary Research (pp. 167-172). Newbury Park, CA: Sage Publications.

Erez, M. (1992). Reflections on the Latham/ Erez/ Locke study. In P. J Frost & R. E. Stablein (Eds.), Doing Exemplary Research (pp. 155-166). Newbury Park, CA: Sage Publications.

Frost, P. J., & Stablein, R. E. (1992). Doing exemplary research. Newbury Park, CA: Sage Publications.

Latham, G. P., Erez, M., & Locke, E. (1988). Resolving scientific disputes by the joint design of crucial experiments by the antagonists: Application to the Erez–Latham dispute regarding participation in goal setting. Journal of Applied Psychology, 73(4), 753-772.

Rafaeli, A. (2009, March 11). Professor Anat Rafaeli. Technion - Israel Institute of Technology Retrieved February 10, 2010, from http://iew3.technion.ac.il/Home/Users/anatr.html

Wheeler, A. R., Richter, E., & Sahadevan, S. (2004). Looking back on their “great works”: Insights from the authors of great works in organizational behavior and human resource management. Management Decision, 42(10), 1326-1342.

Academic Research Online Is a Walled Garden

I've been consumed for the past three weeks researching and writing papers for my Master’s program. I managed to write one paper an Internet topic (a semiotic analysis of folksonomies) and added as many Net references in the others as I could.

In the process of researching online on online topics, I noted two critiques of academia:
1) vehicles for searching online resources are inadequate
2) academic research remains too cloistered

The last few weeks were the first time ever for me that I was exposed to the wealth of electronic academic research. I thought there was lots of information on the Web before, but I was stunned by the quantity and quality of academic information available to students (the databases require log-in and an individual subscription is prohibitively expensive).

Royal Roads University has one of the largest electronic libraries in Canada, which is fitting as it is primarily an online university. Electronic information there takes the forms of:

1. e-books
2. online journal databases
   
3. electronic theses
   

I haven’t made the most of e-books, due to my dislike of reading for a long time onscreen and that e-books can’t accompany me to many of my regular reading places. The theses seem promising, although due to Royal Roads being a comparatively young university they don't have a lot of theses available.

Online academic resources a treasure, albeit hidden & sans map
I did extensively use online academic journals and this is where I was overjoyed and overwhelmed. I had no idea how many journals there were, some of which, believe it or not, aren’t completely esoteric.

There are essentially two problems that I discovered with online searching of these journal databases. Problem one is that there is a bewildering array of journal databases. Second, the search engines for pretty much all these services are, well, crappy. Granted, graduate students do require more advanced search skills than a normal online surfer would need, but still the search tools are unnecessarily complicated, buggy at times, and just plain miss things. I found a lot of instances where I was searching the entire body of articles and certain results would not appear, but later, having found these articles via other means, I would find the terms appearing prominently.

The journal database search engines were so generally poor that I had to use other means, serendipity being the most painful method for time-pressed procrastinators such as myself.

Google Scholar helps save my day
Fortunately, someone turned me onto Google Scholar. I found it retrieved items from academic databases better than the databases own search, plus Google Scholar pulls up other applicable information as well. Truly a very handy tool - thank you Google!

Ivory towers cloister useful research
My final complaint is that while I was also surprised by the quantity and quality of academic research on Internet topics, I was miffed that I never saw any of it before. I’ve worked in the Internet for years, have read books and articles, and been to conferences and was never exposed to this research before.

Granted, it is possible that this research could have come to me via other authors and speakers who digested and regurgitated it. Also, it's not like the research is fit for wider application as can be exceedingly and, I might add, unnecessarily obtuse and elitist (another complaint, sorry). But some research is fine for everyone working in the field as is, and in other cases the findings could be repurposed for wider distribution.

Frankly, I think that too much of academia is infatuated with itself and doesn’t make enough effort to share their research to the outside world. With this attitude one ends up with research for research’s sake. And those, like me, who can benefit from the information don’t get it.

Cool website user testing

I was a guinea pig this evening for user testing for a prominent news and investor website.

They actually paid me to do the test - even though I would gladly have done it for free.

User testing is a great idea to really get an accurate idea of how your users actually use your site. People often say they use a site one way but do something else - or they say they want X feature but in reality they never use it.

What made this testing unique was that they recorded my eyeball movements and click-throughs as I scanned various news and investing pages. So it wasn't what I said I would do, but more like what I would actually do.

The only problem was that the pages I click-through on didn't actually load so in reality I would have ended up on different pages and may have then gone somewhere else. I probably wouldn't have spent as long on the homepages and landing pages as the study encouraged.

At the end of the study, they showed me my results. My web visit usage was way more erratic and frenetic than I imagined. While the standard pattern, the researcher and other studies I have read, for our culture is left to right and up and down. I was surprised by how much back-tracking I did. I did manage to deftly avoid all banners, fluff and navs. (Hope these studies don't eventually prove that there is actually no point in banner graphics and kill a lot of necessary advertising revenue.)

And yes, I sheepishly admit I "clicked" on the legitimate news item with the word porn in it. And the only ad I looked at all was the Telus adds with those irresistibly adorable monkeys.

I wasn't familiar with eye-tracking studies that were so conclusive - so I'm excited to see the results.

The research was conducted by Sensory Logic. I'm looking forward to seeing more from them - even if I don't get paid for it.