The Journal of Extension - www.joe.org

December 2016 // Volume 54 // Number 6 // Tools of the Trade // 6TOT5

MTurk 101: An Introduction to Amazon Mechanical Turk for Extension Professionals

Abstract
Amazon Mechanical Turk (MTurk) is an online marketplace for labor recruitment that has become a popular platform for data collection. In particular, MTurk can be a valuable tool for Extension professionals. As an example, MTurk workers can provide feedback, write reviews, or give input on a website design. In this article we discuss the many uses of MTurk for Extension professionals and provide best practices for its use.


Jodi Dworkin
Associate Professor
Department of Family Social Science
jdworkin@umn.edu

Pooja Brar
Doctoral Student
Department of Family Social Science
brarx011@umn.edu

Heather Hessel
Doctoral Student
Department of Family Social Science
hhessel@umn.edu

Jessie Rudi
Research Associate
Institute for Translational Research in Children's Mental Health and Department of Family Social Science
conne262@umn.edu

University of Minnesota
Saint Paul, Minnesota

Internet-based surveys are a useful and low-cost method Extension professionals can use to recruit participants for research. Archer (2003) lists advantages and disadvantages of conducting surveys on the web and recommends the use of this environment for Extension surveys. Amazon Mechanical Turk (MTurk) (www.mturk.com) is an online marketplace for labor recruitment that has become a popular platform for collecting data online for social science research. Individuals or organizations looking for "workers" can register as "requesters" and post jobs or human intelligence tasks (HITs) for registered workers to complete. Common tasks include completing online surveys, categorizing photographs or other data, and providing feedback on websites or documents. Workers are paid upon successful task completion, with compensation ranging from pennies to several dollars. Research has shown MTurk to be cost effective for obtaining large, diverse samples for questionnaire research (Buhrmester, Kwang, & Gosling, 2011; Dworkin, Hessel, Rudi, & Gliske, 2015).

Studies have found that MTurk samples are more diverse in age, geographic location, and race/ethnicity than general Internet samples (Buhrmester et al., 2011; Goodman, Cryder, & Cheema, 2013). Amazon.com (2016) reports employing more than 500,000 MTurk workers in more than 190 countries. Tasks can be completed in 13 languages quickly and accurately (Pavlick, Post, Irvine, Kachaev, & Callison-Burch, 2014). Amazon typically pays international workers with gift cards but in 2009 began offering workers in India the option of being paid in Indian rupees, greatly increasing the percentage of Indian workers (Ipeirotis, 2010).

Data Quality

Some researchers are concerned about the quality of data from MTurk workers. However, a number of researchers have replicated studies to compare online and offline respondents and found that data quality was not significantly different between MTurk and offline and laboratory experiments (Horton, Rand, & Zeckhauser, 2011; Paolacci, Chandler, & Ipeirotis, 2010; Suri & Watts, 2011). MTurk workers provided high-quality data that met or exceeded psychometric standards associated with research conducted in face-to-face settings (Buhrmester et al., 2011).

Uses of MTurk by Extension Professionals

Possible applications of MTurk within Extension are wide ranging. Specifically, MTurk can be used for research in the following ways:

  • Researchers can collect data from national and international samples about individual behaviors and perceptions. A HIT can link workers to a website hosting an online survey (e.g., SurveyMonkey). MTurk also provides its own online survey tool.
  • The platform allows researchers to collect unique data in ways that previously could be done only in face-to-face settings, such as by simulating group interactions or synchronous tasks.
  • Researchers can create virtual "waiting rooms" to hold workers while others accept a HIT and then proceed with the experiment by having all workers act simultaneously (Mason & Suri, 2012).
  • Researchers can conduct in-depth qualitative interviews with geographically diverse participants, homebound participants, participants without college degrees, and minority groups (Williamson, 2014).
  • Participants can be recruited online for in-person data collection.

MTurk workers can provide a variety of services, including

  • writing reviews, descriptions, and blog entries for websites;
  • editing, transcribing, or rating the accuracy of search engine results;
  • giving feedback about videos and photos, advertisements, recruitment flyers, and program descriptions;
  • providing feedback about whether people or materials in photos are relatable and culturally appropriate; and
  • tagging photos or videos with keywords that can be used by practitioners to recruit a particular audience.

MTurk has many benefits over other methods of data collection:

  • It provides access to a worldwide workforce of people who can perform tasks in multiple languages.
  • The platform has workers who are available on-demand 24 hours a day, 7 days a week. The cost of developing an online recruitment space and recruiting participants is much lower than hiring staff or working with a consulting firm.
  • It provides templates for designing tasks. Using the MTurk HTML editor, requesters can incorporate images, figures, and videos into tasks.
  • It provides access to a more diverse sample than may be available in person or through other online recruitment strategies.
  • It can be used to obtain feedback from a particular group (e.g., parents of young children) about a website's format, design, or content. For example, workers could complete a usability task by exploring a website and answering a short survey that provides feedback about their experience.

Suggestions and Recommendations

Below are some guidelines to consider when using MTurk. See Table 1 for suggested resources and the appendix for step-by-step guidance for using MTurk.

  • Prior to setting up a task, consider the type of participant you want to recruit (e.g., single parents, non-college-going youth).
  • Become an MTurk worker yourself to become familiar with the worker experience, including with how to get paid and how to peruse the various HITs that are available.
  • Test a HIT in the MTurk Sandbox before publishing it to workers; it is a good idea to ensure that surveys work well with a target population and are thoroughly pilot tested before recruiting begins (Monroe & Adams, 2012). The MTurk Sandbox simulates the MTurk environment without any payment involved; it is solely for testing HITs. You can enter your task into the sandbox (just as you would in the actual MTurk website) and choose parameters for how many workers can pilot test the survey to check for errors and identify difficulties with the survey experience (Johnson & Borden, 2012).
  • Take advantage of MTurk's tools for sample selection by specifying worker requirements and inclusion and exclusion criteria for who can see HITs. For example, you can choose to show HITs to only experienced MTurk workers (those who have completed a certain number of HITs successfully), MTurk workers with high reputations (those with high completion ratings), or MTurk workers in particular countries. You can also exclude workers who have already completed the HIT, thereby avoiding duplicate responses.
  • Incorporate attention checks into your survey. Attention checks are questions that are not related to your study but allow you to determine whether respondents are paying attention (e.g., "Select 'Strongly Disagree' for this question").
Table 1.
Resources Supporting MTurk Use
URL Description
https://requester.mturk.com/ Official requester site of MTurk—used to create an account and a HIT
https://blog.mturk.com/ Official MTurk blog
http://MTurk.mit.edu/tutorial/ Site created in 2014 by Morris Alper, an undergraduate student at Massachusetts Institute of Technology, to show the step-by-step process for setting up a HIT on MTurk
https://www.youtube.com/watch?v=Eq_8ETBWkfk YouTube video that demonstrates the process of uploading a survey to MTurk
http://www.turkernation.com/ Online forum for MTurk workers to use to discuss various aspects of working on MTurk; used by some researchers as a forum for talking about their research and connecting with workers
http://turkopticon.ucsd.edu/ Website created by workers to connect with and support each other by reviewing requesters
http://top.sagepub.com/content/39/4/245.short Article written in 2012 by Dan R. Johnson and Lauren A. Borden that includes a brief user's guide for MTurk (getting started, designing a HIT/study, publishing a HIT/study, and managing a HIT/study)
http://rsw.sagepub.com/content/26/4/441.abstract Article written in 2015 by Chitat Chan and Michael J. Holosko that provides a brief overview of MTurk and discusses possible uses by social work professionals
https://workersandbox.mturk.com/mturk/welcome Developer sandbox that allows for practice in creating and completing HITs
Note. MTurk = Amazon Mechanical Turk. HIT = human intelligence task.

MTurk is useful to Extension professionals who want to learn more about the communities they work with. By providing access to individuals from diverse backgrounds who are able to provide quality data, MTurk supports both research (e.g., data collection) and outreach (e.g., receiving feedback on documents).

References

Amazon.com. (2016). Service highlights. Retrieved from https://requester.mturk.com/tour

Archer, T. M. (2003). Web-based surveys. Journal of Extension, 41(2) Article 4TOT6. Available at: http://www.joe.org/joe/2003august/tt6.php

Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3–5.

Dworkin, J., Hessel, H., Rudi, J., & Gliske, K. (2015). A comparison of three online recruitment strategies for engaging parents. Manuscript submitted for publication.

Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26(3), 213–224.

Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14(3), 399–425.

Ipeirotis, P. G. (2010, March). Demographics of Mechanical Turk. NYU Working Paper No. CEDER-10-01. Retrieved from http://ssrn.com/abstract=1585030

Johnson, D. R., & Borden, L. A. (2012). Participants at your fingertips: Using Amazon's Mechanical Turk to increase study-faculty collaborative research. Teaching of Psychology, 39(4), 245–251.

Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods, 44(1), 1–23.

Monroe, M. C., & Adams, D. C. (2012). Increasing response rates to web-based surveys. Journal of Extension, 50(6) Article 6TOT7. Available at: http://www.joe.org/joe/2012december/tt7.php

Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411–419.

Pavlick, E., Post, M., Irvine, A., Kachaev, D., & Callison-Burch, C. (2014). The language demographics of Amazon Mechanical Turk. Transactions of the Association for Computational Linguistics, 2, 79–92.

Suri, S., & Watts, D. J. (2011). Cooperation and contagion in web-based, networked public goods experiments. PLoS One, 6(3), e16836.

Williamson, V. (2014, August 10). On the ethics of crowd-sourced research. Retrieved from http://scholar.harvard.edu/files/williamson/files/mturk_ps_081014.pdf

Appendix

Guidance for Using MTurk

A. How to create an account with Amazon MTurk
  • Create a requester account with Amazon MTurk by going to https://www.mturk.com/mturk/welcome
  • Add money to your Amazon Payments account to purchase prepaid HITs and to calculate the amount you want to pay per worker (plus the fee to Amazon MTurk).
    • Keep in mind that Amazon charges a higher fee for more experienced workers (i.e., workers identified as Master workers within MTurk).
B. How to create a high-quality HIT
  • Create tasks that are simple and concise:
    • It is recommended that you give clear and easy-to-read instructions.
    • It is recommended that you break larger tasks into smaller parts.
  • Provide an example of acceptable work, if possible.
  • Either create a survey on your organization's or institution's platform (e.g., Qualtrics or SurveyMonkey) and link it to MTurk for recruitment, or use MTurk's online survey tool.
C. How to test the HIT in the sandbox
  • Log on as a requester.
  • Click the Developer tab and find the Sandbox.
  • Enter the requester Sandbox to create practice HITs:
    • First click Create.
    • Then click Survey Link to connect your survey to the MTurk platform.
  • Describe the HIT.
    • Provide a concise description of the task.
    • Provide an appropriate list of keywords to attract desired workers.
  • Enter the following information about payment, number of tasks, and timing:
    • Payment amount.
    • Number of completed tasks desired.
    • Amount of time workers have to complete tasks.
    • Length of time you want your HIT to be advertised.
    • General time line for approving the task and paying workers.
  • Click Advanced Options to specify worker requirements.
    • Click Customize Worker Requirements.
      • You must consider whether you need Master workers.
      • You can specify the location (country) of workers.
      • You must consider what kind of task approval rating is acceptable.
    • Show your HIT to workers who are qualified to complete it.
  • Create the design layout.
    • Click the Design Layout tab.
    • Type instructions for the survey.
    • Activate your survey by clicking Distribute Survey.
    • Copy and paste the survey link on the MTurk platform.
    • Provide a survey completion code for workers to enter in MTurk to confirm survey completion.
  • Preview your HIT.
    • Click the Preview tab.
    • Examine the HIT.
    • Click the Finish tab.
  • Test your HIT by using the worker account in the Sandbox.
    • Have people complete the practice task.
    • Get a sense of the time it takes to complete the HIT.
  • Make any necessary corrections or adjustments.
D. Reasons to visit online worker communities (optional; see Table 1)
  • Gain insight into workers by visiting online worker communities.
  • Introduce yourself and your HIT to recruit motivated participants.
E. Publish the HIT on MTurk
  • Put money in the Prepaid HIT Balance account on Amazon.
  • Create and publish HITs.
  • Watch and monitor as HITs are completed.
  • Approve payments within the specified time.