Traditional and social media play an important role in disseminating scientific breakthroughs to the public. However, we as an audience, must be cautious in how we consume information from these publicly available sources.
From claims of harmful effects of vaccines to studies on the extent of climate change, we have learned that behind some news headlines or articles lie either questionable, oversold, or misinterpreted research findings.
So what should readers be aware of when reading news that contain scientific claims?
A lot of studies don’t hold up to replication
The first thing that readers should understand before coming to a conclusion when reading research findings in the news, is acknowledging that there is a well-known ‘replication crisis’ in academic research.
This means that a lot of studies that you read in the news fail to produce similar outcomes when other scientists try to confirm them.
For instance, Nature revealed that more than 70% of researchers have failed to reproduce another scientist’s findings, and more than 40% have even failed to reproduce their own findings.
Similarly, a 2012 study reported that only 11% of the 53 new cancer treatments they identified in the previous decade could be replicated, while another that examined 159 empirical economics studies showed that 80% of these papers had exaggerated their findings.
Factors that may lead to these non-reproducible results include honest human-error mistakes, poor sampling, “cherrypicking” scientific findings, and in rare cases data manipulation.
A survey from the University of Melbourne, Australia, that involved 800 ecologists and biologists, found that 64% of them had at least once failed to report results from their study because they were not “statistically significant” - meaning they did not show results that the scientists hoped for.
The media often feeds on our need for hope
Although the vast majority of scientific research are reputable and reliable, there is the potential for error, fraud, or overstatement of findings.
However, at times, the media can overlooks these flaws - intentionally or otherwise - particularly when it comes to medical research that offer hopes of curing diseases and illnesses.
Let’s recall a breaking news story in 2009 about an Italian researcher, Paolo Zamboni, who claimed to cure his wife’s Multiple Sclerosis (MS) by “unblocking” the veins in her neck. He challenged the mainstream belief about MS as a disorder of the immune system, and instead, theorised it as a vascular disease - one that could be cured by clearing blood vessels.
For the media, however, the most appealing part of this research may have been a man’s quest to save his beloved wife. This romance-fuelled medical triumph - which is a popular story for health reports - appeared to restore the hope of many patients around the world.
Sadly, however, Zamboni’s research had a very small sample size and the design of the experiment had some defects. What attracted much attention was the hype of his romantic story rather than what was supposed to be a medical breakthrough.
Since then, other researchers’ attempt to replicate his findings were not successful and many incidents of patients’ complications and relapses of the disorder were reported.
Zamboni’s case, however, was just a small story in the bigger picture of how the media can misinterpret or overstate research. It is common for promising health interventions, initially promoted in the media, to not be replicated and failing to result in actual clinical practice.
A 2003 study published in the American Journal of Medicine looked at 101 articles published in six major science journals that offered novel therapeutic promises. However, among them only five were licensed for clinical use 20 years later and only one had been proven to have a significant health impact.
There are potential incentives to misreport findings
Around the world, researchers’ job targets, income, bonus, and promotion can be tied to their publications.
On the other hand, many high-impact scientific journals - and consequently the media - can seem more attracted to ‘significant’ or positive results, even though non-‘significant’ results and unsuccessful replications can make substantial contributions to scientific knowledge.
Researchers from the University of California Davis in the US reviewed 359 studies published in leading medical journals in the 1990s, and stated that most of the studies were “reported in a potentially misleading way, with statistics designed to make the results more positive than if other statistical tests were used”.
Many faculty staff have also heard anecdotal accounts of researchers and PhD students re-framing their data or findings to support their initial hypotheses or vice versa. They may even delete, add alter their data to make their work more publishable and appealing for media coverage.
Every now and then the scientific community catches manipulated studies and journals would then retract them from publication.
We should read the news with a critical eye
Every research study has the potential to improve our understanding of the world we live in.
However, we should be careful of overstated findings, studies that have yet to be replicated, or research that has not been published in credible peer-reviewed sources.
It will take more effort, but readers should be cautious of single studies, and instead seek to look at what the broader scientific community says about the topic.
The COVID-19 pandemic highlighted the dangers of misinformation and how it can spread faster than any natural airborne virus. If the findings we read seem too good to be true, they probably are!