6 Ways Scientific Studies Can Trick You
Photo Credit: Alexander Raths/ Shutterstock.com
Stay up to date with the latest headlines via email.
Ever see a headline boasting of an outrageous conclusion that some new scientific study found? These headlines pop up regularly, and they are a boon for publications that get lots of eyeballs reading their articles about the shocking new findings. Factory farming is actually good for the environment! Labeling genetically engineered foods will raise your grocery bills! Wow, really? No way!
These too-good-to-be-true – or too-bad-to-be-true – headlines are accurate, in that there was a study and it did come to those conclusions. But how accurate was the study? As it turns out, scientific studies with phony findings are not as uncommon as they should be. And far too often, bad journalism results in the uncritical reporting of these phony findings.
If you ever read about a study finding that all-cupcake diets are the key to longevity and good health, read the study to see whether the cupcakes tested were made from spinach and wheat germ. Here are some favorite tactics used to design a study to get the findings you’re looking for.
1. Start With a Wrong Assumption
If you live in California, you may recall hearing how labeling genetically engineered (GE) foods would increase your grocery bill. When a ballot initiative to label GE foods was first announced, voters overwhelmingly approved it. But by the 2012 election, it narrowly lost. A study “proving” that GE food labeling would make food costs in California skyrocket may be why voters had such a sudden change of heart.
How was that conclusion reached? The study authors – partially funded by the “No on 37” campaign – began with a wrong assumption. American consumers are just like European consumers, they figured. And, just like in Europe, when GE foods must be labeled, most food manufacturers will instantly remove all GE ingredients from their products. Because GE ingredients like corn and soybeans are present in almost all processed foods, reformulating every food sold in California to remove them would be massively disruptive to the food industry. In fact, it would raise food prices!
But their assumption is wrong. Europeans are willing to pay more for non-GE food, but most Americans aren't. So why would food manufacturers reformulate their products, resulting in higher prices, if they know most Americans wouldn't pay for it? They wouldn’t.
The average voter in California never heard these details. They just heard that their food prices were going up unless they voted no on Prop 37. So they did.
2. Throw Out the Data You Don’t Like
In 2012, researchers in the Netherlands published a study finding that organic agriculture yielded only 80 percent as much as conventional agriculture. Wow, is that true? Well, if it is, this study certainly doesn’t prove it. The study doesn’t actually prove much of anything…because the researchers disregarded any data they did not like.
Michael Hansen, a senior scientist at Consumers Union and a formidable agriculture expert, read the study and reflected that, “When you actually look at the paper, you'll see it's incredibly biased in favor of conventional ag, but in a very technical way.”
The study authors essentially picked and chose which data to include, excluding any “organic” method that did not meet the very strictest definition of the term organic, and throwing out any data from conventional systems that included “unrepresentative yield levels.” Details provided on that say: “Yield data for industrialized countries were considered unrepresentative if conventional yields appeared to be far below the regional average, unless this was caused by factors that can also occur in real farming situations, such as pests, diseases or droughts. For developing countries ‘unrepresentative’ implied conventional yield levels that seemed to be far below yields achieved under best farmers’ management.”