We work at the Australasian Cochrane Centre and we dread being asked what we do for a living. This isn’t because we don’t like what we do, in fact we love it. It’s because when we explain that our job is about helping clinicians and policy makers to use research about what works to inform their treatment or policy decisions, we often receive blank stares or a slow, confused “You mean they don’t do that already?”.
Ideally they do but, for lots of reasons, this is not always the case. One of the challenges is knowing how to find out exactly what works, and one of the most powerful ways of doing this is by looking at systematic reviews.
Put simply, a systematic review is a reliable summary of all the research that’s available on a specific question, such as do antibiotics reduce the duration of symptoms of sore throat?
The systematic nature of these reviews is key. A good systematic review will describe the lengths the authors went to identify relevant studies; the criteria for including studies; how the quality of the studies was assessed; and how the results of the studies were combined.
The principle is that the methods followed should be transparent and replicable, minimising the extent to which authors can influence or bias the review, for example, by leaving out studies whose results they disagree with. In theory at least, this means that another team, following the same methods, would end up with very similar results.
Finding the entire body of research on a particular question is important because in any given area the results of some studies will agree and others won’t. You might search the internet and find a study that shows promising results for an eczema cream, but without reviewing all the studies, you have no way of knowing with any certainty if the results of this one study are consistent with the results of other studies of the cream.
Knowing when, how and why individual studies agree or disagree helps interpret and apply the evidence in practice.
Systematic reviews perfectly encapsulate the adage that the whole is greater than the sum of its parts. Small studies with inconclusive results are common but when looked at together, and combined using a neat statistical technique called meta-analysis, we can often find a conclusive overall answer.
The logo of The Cochrane Collaboration is an example of a meta-analysis.
One criticism of systematic reviews is their over-reliance on randomised trials, the gold standard approach to assessing the effects of interventions. This is changing rapidly and there’s growing awareness of the importance of other study designs in providing a more complete picture of the evidence.
Systematic reviews undertaken by The Cochrane Collaboration, the world’s largest producer of such reviews, now include a range of study designs appropriate to the question being asked:
qualitative research to explain, for example, why a particular treatment might work in some populations but not others;
economic analyses to address issues of cost and cost-benefit; and
non-randomised studies where it would be unethical or impractical to randomise.
Methodological advances have led to systematic reviews being much more sophisticated and useful summaries of evidence but there are still plenty of challenges. Finding all the research on a subject is a case in point. We have a good handle on published research, helped in recent times by the advent of prospective trials registers, but are only now beginning to appreciate the potential implications of the mountains of unpublished data.
Even when we know studies have been done it can be hard to get access to unpublished results. The authors of the recent Cochrane review on Tamiflu, for instance, had to comb through reams of unpublished data submitted to regulatory authorities to unearth possible under-reporting of side effects.
We know too that studies with unexciting results (showing that a treatment doesn’t work) are less likely to be published than studies with exciting results. This can lead to published literature painting a rosier picture than would otherwise be the case, so it’s important that authors assess and report the likelihood of publication bias in their review.
Other challenges include making systematic reviews more readable and relevant to the individual consumer, practitioner or policy maker. While the technical nature of these reviews are a strength, we know they can be challenging to the reader.
The Cochrane Collaboration summarises all its reviews in plain language and is continually considering how to present information from reviews in friendly, easily interpretable formats.
Systematic reviews are essential for making sense of research and helping consumers, practitioners and policy makers identify what works or doesn’t work. They also have a vital role in identifying uncertainties and priorities for future research. If only we could explain this in a sentence or two over a glass of wine.
Dale Bloom
Analyst
“Systematic reviews are essential for making sense of research and helping consumers, practitioners and policy makers identify what works or doesn’t work.”
Excellent.
I would hope similar could be applied to environmental research, and an area that needs major reform, restructuring, improvement and careful monitoring, social science research.
Michael McCarthy
ARC Future Fellow at University of Melbourne
There are attempts to generate systematic reviews in environmental science. For example:
http://www.environmentalevidence.org/Reviews.html
Cheers,
Mick
Dale Bloom
Analyst
Hi Michael
"Objectivity, reliability, transparency and repeatability of systematic reviews are at the heart of the operational and quality assurance processes of the CEE."
http://www.environmentalevidence.org/Peer-reviewers.html
That's good.
Michael McCarthy
ARC Future Fellow at University of Melbourne
One of the difficulties is that systematic reviews require an enormous effort, and there is not much base funding available to support this in environmental science. As a consequence, systematic reviews are not as common as they could be. It should also be noted that there are hopes to combine systematic reviews with consideration of cost-effectiveness, as mentioned for medicine in the above article. For example, there is a piece published in one of the journals that I help edit:
http://50q.sciencebasedconservation.org/uploaded/Segan%20et%20al.%20Evidence%20based%20conservation%20Cons%20Biol%202011.pdf
Cheers,
Mick
Colin MacGillivray
Retired architect
Really good essay; liked the background to the logo too. Aren't many logos that mean anything.
Next time you are asked what you do for a living bring out the logo and say "This says it all!"
Ashraf Fawzy Nabhan
logged in via Facebook
Beautiful Minds
Nearly 2 months ago when the Scientific Committee at the Egyptian General Medical Syndicate asked me to share in the 2 day conference for graduates, I decided to talk about "How do we know what works". Today at 1 PM, I had the privilege of giving this 1 hour lecture to hundreds of keen minds. I came home and as I started reading a new article by 2 of my fellows in The Cochrane Collaboration, Steve McDonald and Tari Turner, I was not surprised that, exactly at the same day, we were telling the same story of beautiful minds (of the collaboration) working to help clinicians and policy makers to use research about what works to inform their treatment or policy decisions.
Cris Kerr
Volunteer Community Health Researcher, Advocate for the value of Patient Testimony
Patients have been testifying that a single low dose (3-4.5mg) of naltrexone (LDN) at bedtime benefits immune system disorders (MS, HIV, ALS, Fibromyalgia, Crohn's disease, Cancer, etc, etc) and minimizes their disease progression.
Multiple Sclerosis and other patented drugs cost thousands of dollars.
Naltrexone is a very old, very cheap out-of-patent drug with a good safety profile, hence; the potential risk/benefit ratio here is clearly very attractive.
This is the reason I wrote to the…
Read moreWendy Oakes
Program Manager
As a program manager I would have to say that I have found Cochrane reviews not very helpful in the area in which I work - lifestyle risk factors for chronic disease. They commonly conclude that the evidence is limited and of poor quality which is not much help in deciding how to proceed. My view is that asking "what works?" or "did X work?" provides less useful information for complex health issues than asking "What did we learn from doing this?" or Pawson and Tilley's realist evaluation question of "What works, for whom and in what context?"
Tari Turner
Senior Research Fellow, Australasian Cochrane Centre at Monash University
Hi Wendy, thanks for your thoughts. I agree that Cochrane reviews do generally address questions of effectiveness or "what works", though they are starting to explore the contextual aspects you mention (and also diagnostic questions).
While many reviews do conclude that there isn't enough research available to guide practice, I'm not sure we can blame the reviews for that; and that conclusion does at least provide a good case in support of more research to answer the question.
I also really like the realist evaluation approach, and we are using it in some of our work on use of research by policymakers. Fascinating stuff!
laughingmind
logged in via Twitter
Hi Tari and Wendy. I completely understand where Wendy is coming from, as a health practitioner trying to distill 'best practice for my practice' learnings out of research data. I do find the Cochrane work valuable, but the point you've made Tari where you state "does at least provide a good case in support of more research" gets me thinking that more of the SAME style of research won't necessarily yield any greater value, and we need to question what research methods might yield better answers to…
Read moreWendy Oakes
Program Manager
Hi Tari. It's good to have the opportunity to discuss this difficult question. I really appreciate the work Cochrane does but the point I am making is that there is a problem with the current protocols for answering many questions related to chronic disease prevention and management. And I don't really think that more of the same research is the answer. I think we need to devise better ways to use the research we already have. As an example, one Cochrane review on obesity interventions in health settings reduced 12,097 articles to 399 which were worth reading of which 6 met the protocol criteria and then concluded they needed further investigation before they could be recommended as effective strategies. Apart from my concern apart the usefulness of trying to aggregate such disparate data, is that really all we learned from all that work?
Tari Turner
Senior Research Fellow, Australasian Cochrane Centre at Monash University
Hi Wendy. You make a really good point. We do need better ways to make sense of and get the most benefit from the huge amount of research that is out there. Cochrane reviews make an important contribution to this, but I totally agree that they are only one part of the solution. They are also already (as you note) a huge undertaking, even just to address relatively narrow questions. Do you have any suggestions about how to best provide evidence to support the kinds of decisions you mention, without compromising the reliability of the methods or requiring (too much) more time or resources?
Gill Westhorp
Director, Community Matters
Hi Wendy and Tari. I'm not directly involved in health research but I use and teach realist synthesis - currently, to investigate complex questions in international development. There are a number of reasons why I like the approach. One is that you can, legitimately and rigorously, include a wide variety of research. Another is that you can use it to answer different kinds of questions. Another (and Ray Pawson might squawk about this, but I think) you can apply a realist lens within a less-than-completely-systematic review - so (for example) within a 'brief review' or to a set that didn't follow complete systematic review processes (so long as you then acknowledge the potential short-fall in your search process). But most of all, I like it because it acknowledges and incorporates complexity, and because it doesn't just ask 'whether', but 'how'.
Tari Turner
Senior Research Fellow, Australasian Cochrane Centre at Monash University
Hi Gill, I have read about realist reviews but haven't (yet) attempted to do one. I find the idea fascinating and potentially really powerful, but don't have a sense of how they work in practice, other than from reading them. I'd like to have a go at some stage, maybe when I'm ready I should contact you for help!
Gill Westhorp
Director, Community Matters
Hi Tari. Happy to help if you get the chance to give it a go. There's interesting discussion available on the Jiscmail discussion list for the RAMESES project, which is currently developing publication standards, quality standards and training materials for realist synthesis and meta-narrative review (another alternative method). https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=RAMESES is the link.