You are here

Conducting Online Research on Amazon Mechanical Turk and Beyond
Share

Conducting Online Research on Amazon Mechanical Turk and Beyond



May 2020 | 296 pages | SAGE Publications, Inc

Conducting Online Research on Amazon Mechanical Turk® and Beyond, written by Leib Litman and Jonathan Robinson, provides both students and experienced researchers with essential information about the online platforms most often used for social science research. This insightful and accessible text answers common questions like, “How do I maintain data quality in online studies?,” “What is the best way to recruit hard-to-reach samples?” and “How can researchers navigate the ethical issues that are unique to online research?” Drawing on their experiences as the founders of CloudResearch (formerly TurkPrime), the authors provide information that guides new users planning their first online studies and engages even the most experienced researchers with detailed discussions about the challenges of online research. The book begins with an overview of Amazon’s Mechanical Turk and its rapid rise within academic research. Then, the authors describe how to set up an MTurk study with screenshots that walk readers through the steps of creating an account, designing a study, collecting data, and using third-party applications to enhance MTurk’s functionality. Later chapters provide readers with a detailed understanding of the MTurk environment and use data from hundreds of thousands of participants and tens of millions of completed tasks to dive into issues like participant demographics, sources of sampling bias, and the generalizability of findings from MTurk. Finally, the book explores the benefits of using other online platforms as a complement to MTurk and the ethical issues that are unique to conducting research with online participant platforms. Throughout the book, the authors share hands-on advice and best practices, such as those for conducting longitudinal studies or carrying out complex studies. Altogether the mix of data, insight, and advice make this book an essential resource for researchers who want to understand the online environment and the most effective ways to conduct research online. 

 

 

 
Chapter 1: Introduction
A Scientific Revolution in the Making

 
A Brief History of Online Research in the Social and Behavioral Sciences: From HTML 2.0 to Mechanical Turk

 
The Use of Online Samples in Applied Behavioral Research

 
Amazon Mechanical Turk

 
Leib Litman, Cheskie Rosenzweig, Jonathan Robinson
Chapter 2: The Mechanical Turk Ecosystem
How Quality is Maintained

 
Reputation Mechanism

 
Selectively Recruiting Specific Workers

 
Protections for Workers

 
Communicating with Workers

 
A Worker’s Perspective

 
Worker Communities

 
 
Chapter 3: Conducting a Study on Mechanical Turk
Sample Project

 
Setting up a Requester Account on Mechanical Turk

 
Creating a HIT

 
The ‘Design Layout’ tab

 
Monitoring Progress on the Requester’s Dashboard

 
When a Worker Runs Out of Time

 
Sample Study Results

 
Conducting Follow-up Studies Using Requester-Issued Qualifications

 
Appendix A: Checklist for best practices of setting up a Mechanical Turk HIT

 
 
Chapter 4: API and Third Party Apps
Third Party API-based Platforms

 
Common Uses for API Scripts and Third Party API-based Apps

 
TurkPrime

 
Jesse Chandler, Gabriele Paolacci, David Hauser
Chapter 5: Data Quality Issues on MTurk
Defining and Measuring Data Quality

 
Measuring Individual Participant Data Quality

 
Causes of and Cures for Poor Data Quality

 
Concluding Thoughts

 
 
Chapter 6: Who are the Mechanical Turk Workers?
Sources of Data

 
Location of Workers in the US

 
Demographics of Mechanical Turk

 
 
Chapter 7: Sampling Mechanical Turk Workers: Problems and Solutions
Sampling on Mechanical Turk

 
Sources of Sampling Bias

 
The Problem of Superworkers

 
Time-of-day Effects

 
Pay Rate

 
Dropout

 
Sampling Best Practices

 
 
Chapter 8: Data Representativeness of Mechanical Turk Samples
Representativeness, surveys, and survey sampling

 
The methodology of survey sampling

 
Mechanical Turk as a sampling frame

 
The fit-for-purpose framework

 
Chapter Overview:

 
 
Chapter 9: Conducting Longitudinal Research on Amazon Mechanical Turk
Why Longitudinal Research?

 
Retention, Longitudinal Research, and MTurk

 
Case Studies

 
Best practices for longitudinal research

 
 
Chapter 10: Beyond Mechanical Turk: Using Online Market Research Platforms
Limitations of MTurk

 
Online probability-based panels

 
Online market research platforms

 
Overall comparisons between Mechanical Turk and market research platforms

 
 
Chapter 11: Conducting Ethical Online Research: A Data-Driven Approach
Historical background

 
Risk of harm in online research

 
Research on sensitive topics

 
A deeper dive into controversial and complex issues

 
Economics of Mechanical Turk: Considerations for setting wages

 
Setting wages: Considerations of ethics and methodology

 
Considerations for rejecting, blocking and disqualifying workers.

 
Practical advice for requester/worker interactions

 
Anonymity

 
 
Appendix A

“This is the best practical guide I have seen written in any methodologically written book!” 

Michaela Porubanova
Farmingdale College

“Great overview of literature developing in several fields; it's hugely valuable to have all of this in one place.” 

Kevin Munger
New York University

“They cover the whole gamut of history/background, practical concerns and scholarly concerns.” 

See-yeon Hwang
Sam Houston State University

"I am looking forward to being able to hand this to my students and know they'll be ready to get started on MTurk and really understand the whole system (i.e., not just the mechanics of getting data, but really "getting" the culture and so on). I would probably also find myself citing this book as it discusses many of the issues I personally research.” 

Alice M. Brawley
Gettysburg College

Leonid Litman

Leib Litman received a PhD in experimental psychology from the City University of New York. His early research was in the area of implicit learning, focusing on the relationship between conscious and unconscious information processing. After receiving his PhD, he was a postdoctoral fellow and research scientist in cognitive neuroscience at New York University, where he studied episodic memory using fMRI. Leib’s current methodological research interests are in the area of online data collection, especially in the application of online data collection tools to complex research designs, such as dyadic studies, experience sampling, intensive... More About Author

Jonathan Robinson

Jonathan Robinson has been creating software since he was 12 and started his first software company with a few high school friends at age 15. He continued to develop his software skills and apply his expertise to more complex problems as he completed his BA at Queens College, and his MS and PhD with a dissertation in the field of robotic vision and digital curve fitting while at the City University of New York. Jonathan conceived of using Amazon’s Mechanical Turk for participants before there was public academic discussion of its application to research. He designed and implemented the core CloudResearch.com (previously TurkPrime) system... More About Author

Purchasing options

Please select a format:

ISBN: 9781506391137
$68.00

SAGE Research Methods is a research methods tool created to help researchers, faculty and students with their research projects. SAGE Research Methods links over 175,000 pages of SAGE’s renowned book, journal and reference content with truly advanced search and discovery tools. Researchers can explore methods concepts to help them design research projects, understand particular methods or identify a new method, conduct their research, and write up their findings. Since SAGE Research Methods focuses on methodology rather than disciplines, it can be used across the social sciences, health sciences, and more.