Science factory

How does scientific research suffer if data comes from online survey forums? 

by Nicole Oran

What used to be the job of university students, being surveyed for scientific and academic research, is now often being done by people online, which could arguably lead to flawed and unreliable data. 

Mechanical Turk is an online job forum – an informal work force of people who participate in studies – basically a pool of professional survey-takers. Researchers are now taking data from such sources because it’s inexpensive and quick. But there could be discrepancies on the reliability of answers given. What kinds of surveys are we talking about?

“You have a lot of psychology, social science research, but also political scientists are using it. You even see it in medical research,” said Jenny Marder, who reported the story for PBS Newshour. “And we looked at a lot of the studies that are using Mechanical Turk, and a lot of them are asking really big questions. There’s a lot of research on human behavior, but also on teen alcohol abuse, a lot of research on decision-making, how people perceive scientists and climate scientists. So these aren’t obscure studies. And they’re asking some pretty big questions.” 

This issue with surveys like those from Mechanical Turk is that many people do them from home, even thousands of them, and there’s no way to know if the person was distracted, actually paying attention, or if they are on automatic because they have seen some of the same questions repeated over and over. The people who take these surveys get paid (albeit, very little) even if they answer questions in an automatic, not entirely thoughtful way. So what is the incentive to really sit down, focus, and answer truthfully? 

Data isn’t valuable if it isn’t an accurate reflection of the subject’s perspective and the topic, but it continues to be used.


Watch How pro survey-takers are shaping scientific research on PBS. See more from PBS NEWSHOUR