I once told a journalist that “the very fact that a piece of health research appears in the newspapers indicates that it is nonsense”. That may be too harsh, so I would like to suggest a slightly more refined version for dealing with science stories in the news, particularly medical ones.
Ask yourself: if the study had come up with a negative result, would I be hearing about it? If the answer is “no”, then don’t bother to read or listen to the story.
My initial comment was the result of a spate of studies claiming associations between ordinary daily habits and future bad outcomes: eating a lot of white bread with becoming obese, being cynical with getting dementia or light bedrooms with obesity (again). All these stories associate mundane exposures with later developing dread outcomes, that is the classic “cats cause cancer” genre. My argument is that, since we would not be reading about a study in which these associations had not been found, we should take no notice of these claims.
Why my cynicism? There has been a lot of public discussion of potential biases in the published scientific literature. For example, see commentaries in The Economist and Forbes. Also, a recent study argues that newspapers preferentially cover medical research with weaker methodology.
The general idea is that by the time research has been selected to be submitted, and then selected for publication, there is a good chance the results are false positives – results that appear true but are not.
Such selection bias is nothing compared to the hurdles overcome by stories that are not only published, but publicised via the mainstream, popular press. For a study to be publicised, it must have:
- Been considered worthwhile to write and submit to a journal
- Been accepted for publication by the referees and editors
- Been considered “newsworthy” enough to deserve a press release
- Been “sexy” enough to attract a journalist’s interest
- Got past a news editor
Anything that gets through all these hurdles stands a huge chance of being a freak finding. In fact, if the coverage is on the radio, I recommend sticking your fingers in your ears and loudly saying “la-la-la” to yourself.
The crucial idea is that since there is an unknown amount of evidence that I am not hearing about and that would contradict this story, there is no point in paying attention to whatever it is claiming. It is like watching a video of a football team scoring goals, and then suddenly realising that you are only being shown the “successes” and not the ones they let in: the evidence just shows that they are capable of scoring, but not whether they score more than they concede. So, if you are interested in assessing the quality of the team, stop watching the video.
This may seem rather cynical, so I should have make clear that I am only talking about single studies: proper reviews of the totality of evidence should be listened to. So this is not an excuse to ignore evidence connecting smoking and lung cancer.
This mental shortcut is even more appropriate when you hear or read of any survey by any organisation, particularly charities. Use this rule-of-thumb and save time. Maybe use that time to exercise, which has been definitively shown to improve your health.