21Unskilled and Unaware of It
Justin Kruger and David Dunning, Journal of Personality and Social Psychology 77(6), pp. 1121–1134, 1999
Linked to 30 chapters: supports 12 · contradicts 6 · mechanism 17 · same claim 4 · pairing 5
In 1995 McArthur Wheeler robbed two Pittsburgh banks in daylight with nothing on his face but lemon juice, which he believed made him invisible to video cameras. Kruger and Dunning open with him to make a point that is not obvious: when a person's strategies are bad, that person is in no position to notice. The skills that produce a correct answer, they argue, are largely the same skills that recognise one: whatever lets a person write a grammatical sentence is what lets them see that a sentence is ungrammatical. Someone who lacks the first therefore lacks the second, and is left not merely wrong but confident. Darwin had said as much in 1871, in a line the paper quotes: "ignorance more frequently begets confidence than does knowledge". Four studies test the claim on humour, logical reasoning and English grammar. In each, students in the bottom quarter placed themselves near or above the middle; pooled, their scores put them in the 12th percentile and their estimates in the 62nd. Two further results follow from the same account. The unskilled could not read competence in other people's work either, so watching better performers taught them nothing. And when the authors made incompetent participants competent, their self-estimates fell.
Imperfect self-assessments
The tendency the paper sets out to explain was already documented. High school students rate themselves above average in leadership, getting on with others and written expression; business managers rate themselves more able than the typical manager; football players think themselves shrewder about the game than their teammates. The above-average effect, as the literature calls it, defies descriptive statistics, and the usual explanations for it are motivational: people want to think well of themselves. Kruger and Dunning add a cognitive source. If a person cannot see that a performance went badly, and people generally assume they have taken the most reasonable option available, then the absence of a signal of failure is read as success.
The materials for the account were also in place, in the research on expertise. Novices in physics are worse than experts at judging how hard a problem is, novices in chess at judging how often they must see a board position before they can reproduce it, novices in tennis at telling whether a shot came off. Weak readers misjudge their comprehension, and drivers who have failed a test predict their reaction times least accurately. None of this work had asked whether the metacognitive deficit produced inflated estimates, or tied it to the above-average effect. That is what the four studies do, through four predictions: that the incompetent will overestimate themselves dramatically, that they will be less able to recognise competence anywhere, that they will therefore learn nothing from social comparison, and that making them competent will correct all three.
Humour, logic and grammar
Study 1 chose a domain where competence means knowing what other people will find funny. Sixty-five Cornell undergraduates rated thirty jokes on an eleven-point scale, and their ratings were compared with those of eight professional comedians. One comedian's ratings failed to correlate with the rest and were dropped, leaving a panel reliability of .76; the jokes ranged from one the panel rated 1.3 to one it rated 9.6. Participants then placed their ability to recognise what is funny against the average Cornell student on a percentile scale. The average student put himself in the 66th percentile. Perceived and actual ability were correlated, at .39, so the students were not guessing blindly. The bottom sixteen scorers, whose ratings actually placed them in the 12th percentile, put themselves in the 58th — above average, in a domain in which they were among the worst in the room.
The authors raised the obvious objection against themselves at once. Perceptions correlate imperfectly with reality, people at the bottom have nowhere to go but up, and regression to the mean guarantees a result of this shape. Their first answer is the asymmetry: regression alone predicts equal and opposite miscalibration at the top, and the top quarter's error was far smaller. The real answer is deferred to the two later studies.
Study 2 changed domains to remove a different objection. Humour has no uncontroversial criterion, and people define ambiguous abilities in whatever way flatters them. Forty-five students took twenty logical reasoning items from a Law School Admission Test preparation guide, where the answers are not matters of taste, and estimated their reasoning ability, their test score in percentile terms, and the number of items they had got right. The class placed its ability in the 66th percentile and its scores in the 61st, while being roughly accurate about raw scores, 13.3 claimed against 12.9 actual. The bottom eleven, who scored in the 12th percentile, put their ability in the 68th and their test score in the 62nd, and claimed 14.2 correct answers when they had 9.6, an overstatement of nearly fifty per cent. That last number is not comparative: the error is about the self, not about a misjudged peer group.
Study 3 repeated the design on American Standard Written English, twenty items from a teacher examination guide, with eighty-four students. The bottom seventeen scored in the 10th percentile and placed their ability in the 67th and their performance in the 61st, claiming 12.9 correct where they had 9.2.
It takes one to know one
The second half of the thesis is that the deficit also blinds a person to competence in others, and the third study tested it four to six weeks later. Its bottom seventeen and top nineteen performers came back and were each handed five completed grammar tests spanning the range of scores their classmates had produced, a fact they were told. They graded each one, then re-rated themselves.
The top performers graded well: the correlation between the marks they gave and the marks the tests deserved averaged .66, against .37 for the bottom performers, and their summed absolute error was 9.2 against 17.4. The self-ratings that followed divided as sharply. Having just read five tests, several of them clearly better than their own, the bottom group did not move: their ability estimate drifted from the 66.8th percentile to the 63.2nd, their test estimate rose from the 60.5th to the 65.4th, and neither change approached significance. The top group raised its ability estimate to the 77.2nd percentile and its test estimate to the 79.7th, toward an actual standing of 88.7.
That pattern explains something that had shown up in every study so far without being accounted for: the best performers consistently underrated themselves. Kruger and Dunning read it as a false-consensus effect. The top performers were not wrong about their own work; they were wrong about everyone else's, assuming that answers as obvious as theirs would be obvious to their peers. The prediction that distinguishes this from simple regression is precise. Learning that your peers did badly tells you nothing about your own raw score, so a false-consensus error should be corrected for comparative estimates and leave absolute ones alone. The top group's estimate of its own raw score went from 16.9 to 16.6, against an actual 16.4, and did not budge.
Competence begets calibration
The fourth study manipulated the variable rather than measuring it, which is what separates the metacognitive account from the statistical one. A hundred and forty students took ten problems built on the Wason selection task, in which four cards and a rule are given and the participant must say which cards would have to be turned over to test the rule. They rated themselves as before. Then seventy of them, chosen at random, worked through a ten-minute training packet on testing logical rules, modelled on Cheng and colleagues' work from 1986, while the rest did an unrelated filler task of the same length. Everyone then marked which problems they thought they had got right, and rated themselves a second time.
Before training, the bottom thirty-seven scored in the 13th percentile, placed their ability in the 55th and their performance in the 53rd, and believed they had solved 5.5 of the ten problems. They had solved 0.3.
Training changed what they could see. Asked to mark their own answers right or wrong, trained participants classified 9.3 of ten correctly against 6.3 for the untrained, and among the bottom quarter the gap was 9.3 against 3.5, which left them no worse at monitoring themselves than the top scorers at 9.9. Their self-estimates followed: ability from the 55th percentile to the 44th, test performance from the 51st to the 32nd, and claimed raw score from 5.3 correct to 1.0, against an actual 0.4. They still overestimated their standing, but they were no longer miscalibrated about their score. The untrained bottom quarter did not improve on any measure, and its estimate of its own raw score moved the wrong way, from 5.8 to 6.3.
Two mediation analyses tie the parts together. Among the untrained, test performance predicted metacognitive skill at .75, and metacognitive deficits predicted inflated self-assessment on all three measures with objective performance held constant, betas of −.40 to −.49. Among the bottom quarter, once metacognitive skill was controlled, training itself no longer predicted anything: the improvement in self-knowledge travelled entirely through the improvement in the ability to tell a right answer from a wrong one. Regression to the mean predicts none of this, since it says nothing about what happens to self-estimates when a person's skill is altered after the test is scored. The paradox the authors leave standing is that the cure removes the patient. The way to make people see their incompetence is to make them competent, at which point there is nothing left to see.
The burden of expertise, and what the account does not cover
The general discussion offers anosognosia as the analogy: damage that both paralyses the left side and removes the capacity to notice the paralysis, so that patients asked why they will not lift a cup say they are tired. Across the four studies the bottom quarter did not merely overestimate itself but believed itself above average, and was close to four times more miscalibrated than the top quarter.
The paper is careful about how little this explains on its own. Bottom-quarter participants accounted for most of the above-average effect, overestimating by about fifty percentile points, but the other three quarters overestimated too, by just over six points, and the metacognitive account says nothing about that residue. Motivational bias, self-serving definitions of ability and selective recall are all still in play. Nor is the deficit universal: few readers, the authors note, would bet on themselves against Michael Jordan. The prediction holds where knowing the subject is what competence consists of, as in grammar and logic, and fails where competence is physical. Coaches understand basketball and cannot dunk; art appraisers judge calligraphy they could not produce. A second limit is a floor: to overestimate yourself you need enough theory to believe you can produce an answer at all. Nobody overrates their ability to translate Slovenian proverbs, and in domains that blank people rate themselves below their peers.
One puzzle the paper poses and does not solve is why ordinary life fails to correct the error. It offers four reasons. People rarely tell each other bad news. Many tasks return no self-correcting information. Failure is ambiguous in a way success is not, since success needs skill and effort and luck together while failure needs only one of them missing, so a person told they have failed can always blame something else. And the remaining channel, watching other people, is the one the third study closed.
The article ends on its own exposure. Having argued that incompetence hides itself, the authors concede that their paper may contain faulty logic or methodological error, and that to whatever extent it does, "it is not a sin we have committed knowingly".
Since 1999
The finding left the literature almost immediately. Kruger and Dunning were awarded an Ig Nobel Prize in psychology in 2000, and the phrase "Dunning–Kruger effect", which appears nowhere in the paper, was attached to it by others and spread into ordinary speech within a decade. Most of that usage is a claim the paper never makes: that stupid people think they are clever. The studies are about specific domains, and a footnote sets aside the generally incompetent person as no part of the research. They are also not about a reversal of rank. Bottom-quarter students placed themselves lower than top-quarter students did in every study. What the studies show is a compression, everyone bunched near the upper middle of a scale on which they are in fact spread out.
The statistical critique arrived in 2002, when Krueger and Mueller argued in the same journal that the pattern falls out of two things already known: regression to the mean, since test scores are measured with error and self-estimates correlate only weakly with them, and the better-than-average effect, which moves everyone's estimate up. Between them these produce crossing lines of the paper's shape without any metacognitive deficit. Burson, Larrick and Klayman pressed a related point in 2006: whether the poor performers or the good ones are the worse judges depends on how hard the task is, which makes difficulty rather than skill the moving part. Later work by Nuhfer and colleagues and by Gignac and Zajenkowski showed that random numbers can be plotted into a figure much like Figure 1, and that when actual and self-assessed ability are modelled directly the relation is mostly linear.
The critique is largely about the graph, and the graph was never the whole argument. The measures regression cannot touch are the other ones: how well people sorted their own answers into right and wrong, how well they graded a stranger's test, and what happened when ten minutes of instruction changed their skill after their score was fixed. Ehrlinger, Johnson, Banner, Dunning and Kruger returned to these in 2008 with incentives for accuracy and with participants found at a debate tournament and a gun club rather than in a psychology course, and the overestimation by low performers survived. What has held up is smaller than the popular version and harder to dismiss than the artefact version: self-assessments of ability track performance weakly at best, the weakest performers are the worst placed to know it, and what most reliably improves a person's judgment of their own work is getting better at the work.