Menu Close
The Conversation

Critically important: the need for self-criticism in science

Michael J. I. Brown, Monash University

Author

Disclosure statement

Michael J. I. Brown receives funding from the ARC to undertake research into galaxy evolution and active galactic nuclei.

Partners

Monash University provides funding as a founding partner of The Conversation AU.

View all partners

Want to be a scientist? Take a long, hard look at yourself. edenpictures

THE STATE OF SCIENCE: To produce good science, researchers have to be prepared to doubt themselves. Michael Brown investigates the importance of holding science up to scrutiny.

Scientists and the public can have very different views of scientific debates.

Are vaccines responsible for increased rates of autism? Does increasing carbon dioxide lead to a rise in global temperatures? For most scientists, these debates have been resolved, but they remain very visible in the public arena.

In part, these debates remain alive due to the contributions of researchers (I use the term broadly) who promote contrary views via the media.

The presence of such researchers in the public sphere is not new. In the 1970s, claims of impending earthquakes due to alignments of the planets were propagated via the media and popular books.

Charles Richter (of the Richter Scale) was particularly dismissive of these researchers:

“What ails them is exaggerated ego plus imperfect or ineffective education, so that they have not absorbed one of the fundamental rules of science – self-criticism. Their wish for attention distorts their perception of facts, and sometimes leads them on into actual lying.”

How is self-criticism fundamental to science? Why don’t we learn this in school? How is self-criticism relevant in the current debate?

It isn’t easy

Only rarely do scientists have such remarkable insight that science becomes easy. Usually it is hard work. Very hard work.

In press releases, scientists obtain their results and then proclaim their insights. In the real world, scientists must critically examine their own results before saying anything.

They have to ask themselves a number of questions:

  • How could the results be wrong?
  • Are the results consistent with established theory?
  • Are the results consistent with the available data?
  • Have the best possible methods been used?
  • What are the potential sources of error?
  • Do assumptions affect the results?
  • Is the preferred hypothesis the only and most plausible explanation?
  • Are the results statistically significant?

Good scientists can spend more time answering these questions than obtaining their original results. To answer these questions, scientists must examine their own work critically.

Imagine you’re a scientist

Imagine you’re a scientist. Think of yourself in a white coat, wearing elbow patches, in hiking boots or clothed in scuba gear.

You get a great preliminary result. Then you review the literature again. Your result disagrees with those of many leading scientists. What do you do?

Unfortunately, the first step is to assume you are in error. Remember, science is hard and mistakes are easy, no matter how smart you are. For this reason, scientists treat individual results with caution, preferring results that have been confirmed multiple times.

Some errors are hard to track down. A simple typo in computer code can have major consequences. Sometimes all the numbers are correct, but the interpretation of them is wrong.

Repeating each step of the process again and again will mitigate errors, but is time consuming.

What if your result stands? Are the leading researchers all wrong? Perhaps, but unless you have an outstanding insight, it probably comes down to a subtle error.

This has been my own experience.

In 2007, my colleagues and I found that the largest galaxies have grown slowly over billions of years, while two other studies found they grew quite rapidly. The other studies had used good data, but had interpreted their results with a method that (in hindsight) was only applicable to smaller galaxies. The error was subtle, and not at all obvious, even to experts.

Before publication, scientists present their work at science conferences and seminars. In part, scientists do this to get feedback from experts who disagree with the conclusions being made. In this context, it can be a relief to be asked difficult questions.

Scientists finally present their results in articles submitted for peer review. Errors can result in rejection from publication, so there is a strong incentive to critically examine your work prior to submission.

Peer reviewers should catch big mistakes and glaring omissions, but won’t always catch more subtle errors. Your reputation suffers when your publications contain errors, so it is in your interest to critically examine your work.

A lost lesson

The importance of self-criticism is sometimes lost in science education.

A student may have only hours to undertake and report the results of an experiment. There is no time for critical examination of the results. Often there is only one plausible explanation rather than many. Students may even know “the answer” before starting the experiment.

For example, a student studying the time it takes for a heavy ball to fall a short distance can only interpret the results in the context of gravity. Unfortunately, this may teach students that experiments only verify rather than confront scientific theories. This experience may also skew the public perception of science.

Science educators are aware of these problems, but they are difficult to overcome when time and resources are limited.

The current climate debate

How is self-criticism relevant to current debates?

Many climate “sceptics” self-publish research online, write for the general media, or submit their work to journals with ineffective peer review. They are thus sidestepping a key incentive to be self-critical – the risk that the article will be rejected and not published.

“Sceptics” often falsely accuse climate scientists of startling errors. However, the frequent mistakes of vocal Australian “sceptics” suggest it is they who are not self-critical. The many errors in Ian Plimer’s “Heaven and Earth” are just the start.

John McLean has predicted that 2011 will be the coolest year since 1956. However, the flaws in his model strongly linking the El Niño cycle and global climate had already been flagged in the peer-reviewed literature. Rather than being cool, 2011 is now vying to be in the top ten of hot years.

In an online essay, John Nicol (chairman of the Australian Climate Science Coalition) claims scientists wrongly model how the atmosphere absorbs light. However, years after his essay appeared online, Nicol has yet to compare his predictions to readily available data.

David Archibald claims solar cycles and climate are so strongly linked that “a severe cool period is now inevitable”. However, Archibald uses temperatures from just a few locations to back his claim, and overlooks studies that find a far weaker link between solar cycles and climate.

Such glaring errors would be evident to those who critically evaluate their own work. The absence of self-criticism leads to error, or even pseudo-science. While it is foolish to assume every scientific paper is 100% correct, it is surely more foolish to believe critics of science who are not demonstrably self-critical of their own ideas.

This is the ninth part of The State of Science. To read the other instalments, follow the links below.

Want to write?

Write an article and join a growing community of more than 181,800 academics and researchers from 4,938 institutions.

Register now