John Cleese, the British comedian, once summed up the idea of the Dunning–Kruger effect as, “If you are really, really stupid, then it’s impossible for you to know you are really, really stupid.” A quick search of the news brings up dozens of headlines connecting the Dunning–Kruger effect to everything from work to empathy and even to why Donald Trump was elected president.
As a math professor who teaches students to use data to make informed decisions, I am familiar with common mistakes people make when dealing with numbers. The Dunning-Kruger effect is the idea that the least skilled people overestimate their abilities more than anyone else. This sounds convincing on the surface and makes for excellent comedy. But my colleagues and I suggest that the mathematical approach used to show this effect may be incorrect.
What Dunning and Kruger showed
In the 1990s, David Dunning and Justin Kruger were professors of psychology at Cornell University and wanted to test whether incompetent people were unaware of their incompetence.
To test this, they gave 45 undergraduate students a 20-question logic test and then asked them to rate their own performance in two different ways.
First, Dunning and Kruger asked the students to estimate how many questions they got correct – a fairly straightforward assessment. Then, Dunning and Kruger asked the students to estimate how they did compared with the other students who took the test. This type of self-assessment requires students to make guesses about how others performed and is subject to a common cognitive mistake – most people consider themselves better than average.
Research shows that 93% of Americans think they are better drivers than average, 90% of teachers think they are more skilled than their peers, and this overestimation is pervasive across many skills – including logic tests. But it is mathematically impossible for most people to be better than average at a certain task.
After giving students the logic test, Dunning and Kruger divided them into four groups based on their scores. The lowest-scoring quarter of the students got, on average, 10 of the 20 questions correct. In comparison, the top-scoring quarter of students got an average of 17 questions correct. Both groups estimated they got about 14 correct. This is not terrible self-assessment by either group. The least skilled overestimated their scores by around 20 percentage points, while the top performers underestimated their scores by roughly 15 points.
The results appear more striking when looking at how students rated themselves against their peers, and here is where the better-than-average effect is on full display. The lowest-scoring students estimated that they did better than 62% of the test-takers, while the highest-scoring students thought they scored better than 68%.
By definition, being in the bottom 25% means that, at best, you will score better than 25% of people and, on average, better than just 12.5%. Estimating you did better than 62% of your peers, while only scoring better than 12.5% of them, gives a whopping 49.5 percentage-point overestimation.
The measure of how students compared themselves to others, rather than to their actual scores, is where the Dunning–Kruger effect arose. It grossly exaggerates the overestimation of the bottom 25% and seems to show, as Dunning and Kruger titled their paper, that the least skilled students were “unskilled and unaware.”
Using the protocol laid out by Dunning and Kruger, many researchers since have “confirmed” this effect in their own fields of study, leading to the sense that the Dunning–Kruger effect is intrinsic to how human brains work. For everyday people, the Dunning-Kruger effect seems true because the overly arrogant fool is a familiar and annoying stereotype.
Debunking the Dunning-Kruger effect
There are three reasons Dunning and Kruger’s analysis is misleading.
The worst test-takers would also overestimate their performance the most because they are simply the furthest from getting a perfect score. Additionally, the least skilled people, like most people, assume they are better than average. Finally, the lowest scorers aren’t markedly worse at estimating their objective performance.
To establish the Dunning-Kruger effect is an artifact of research design, not human thinking, my colleagues and I showed it can be produced using randomly generated data.
First, we created 1,154 fictional people and randomly assigned them both a test score and a self-assessment ranking compared with their peers.
Then, just as Dunning and Kruger did, we divided these fake people into quarters based on their test scores. Because the self-assessment rankings were also randomly assigned a score from 1 to 100, each quarter will revert to the mean of 50. By definition, the bottom quarter will outperform only 12.5% of participants on average, but from the random assignment of self-assessment scores they will consider themselves better than 50% of test-takers. This gives an overestimation of 37.5 percentage points without any humans involved.
To prove the last point – that the least skilled can adequately judge their own skill – required a different approach.
My colleague Ed Nuhfer and his team gave students a 25-question scientific literacy test. After answering each question, the students would rate their own performance on each question as either “nailed it,” “not sure” or “no idea.”
Working with Nuhfer, we found that unskilled students are pretty good at estimating their own competence. In this study of unskilled students who scored in the bottom quarter, only 16.5% significantly overestimated their abilities. And, it turns out, 3.9% significantly underestimated their score. That means nearly 80% of unskilled students were fairly good at estimating their real ability – a far cry from the idea put forth by Dunning and Kruger that the unskilled consistently overestimate their skills.
Dunning–Kruger today
The original paper by Dunning and Kruger starts with the quote: “It is one of the essential features of incompetence that the person so inflicted is incapable of knowing that they are incompetent.” This idea has spread far and wide through both scientific literature and pop culture alike. But according to the work of my colleagues and me, the reality is that very few people are truly unskilled and unaware.
The Dunning and Kruger experiment did find a real effect – most people think they are better than average. But according to my team’s work, that is all Dunning and Kruger showed. The reality is that people have an innate ability to gauge their competence and knowledge. To claim otherwise suggests, incorrectly, that much of the population is hopelessly ignorant.