20Does the Whole Exceed its Parts?

Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro and Daniel S. Weld, CHI Conference on Human Factors in Computing Systems, Yokohama, 2021

Linked to 25 chapters: supports 18 · contradicts 2 · mechanism 13 · same claim 5 · pairing 7

A decision-support system is worth building only if the pair it creates is better than either of its parts. Bansal, Wu and their colleagues make that the standard an explanation has to meet — complementary performance, team accuracy above the person working alone and above the model working alone — and then show that the studies offered as evidence for explaining machine recommendations had never met it. In every one of them the model, left to run by itself, was more accurate than the assisted team, so the rational move would have been to remove the person. They rebuild the experiment so that the standard can be tested: tasks crowd workers can do without training, and items chosen so that the model's accuracy matches theirs, which opens a band above both partners for a team to occupy. With 1,626 workers on three tasks, the teams did occupy it. But no explanation beat the plainest baseline available, which is printing the model's confidence beside its answer. Highlighted words, arguments written by hand, and a scheme that changes the explanation when the model is unsure all performed the same as a number. Splitting the results by whether the recommendation was right shows why: explanations raised accuracy when the model was correct and lowered it when the model was wrong, and the two movements cancelled.

The test nobody had run

The opening argument is arithmetic. A developer deploys a team only if the team adds utility, which means beating both partners; anything less is an argument for automation. Table 1 of the paper assembles eight such results, each reporting that explanations improved human accuracy, and adds the column those papers had not compared. In deceptive-review detection, workers alone scored 51.1 per cent, the model 87, the best team 74.6. In income-category prediction the three figures were 65, 75 and 73. Predicting house prices, workers were out by $331,000 on average, the model by $200,000, the team by $232,000. In each case the explanation did lift the person above where they had been, and in each case the machine would have done better without them.

Two things could account for that. The first is the design of the task. When the model is much the stronger partner, the band above both of them is narrow, and the person's errors are likely to be a superset of the machine's, which leaves little for a human overseer to catch. The second is the mechanism of collaboration itself: incentives, the format of the explanation and whether uncertainty is displayed may all push a person towards a crude policy — always trust the machine, or never trust it — rather than towards judging each case. The authors set out to remove the first problem by construction and then to test the second.

A task with room above both

They chose two kinds of task. Sentiment classification of reviews needs no expertise, and they used two datasets so that the result would not belong to one of them: beer reviews and Amazon book reviews. For something harder they took logic questions from a Law School Admission Test preparation book, each with four options and one right answer. A question of that kind took about a minute to answer; a review took about seventeen seconds.

The models were fine-tuned RoBERTa classifiers, the reasoning model trained on ReClor, a dataset of logic questions from standardised exams. Accuracy was matched to the person rather than maximised. Pilots established that unassisted workers scored 87 per cent on beer reviews and 85 on book reviews, so the authors selected fifty unambiguous items per dataset on which the model scored 84, with equal false-positive and false-negative rates. For the logic task they used twenty questions, verified as not searchable online and absent from the training data, on which the model scored 65 per cent against the 67 per cent workers had managed unaided.

Explanations for the review tasks came from LIME and were shown as inline highlights, colour-matched to the answer buttons. Highlighting cannot show a logical contradiction, so the logic questions were explained in prose: one author condensed the preparation book's explanation of the right answer to two sentences and wrote supporting arguments for the other three options in the same voice. Workers were recruited on Mechanical Turk, screened with a qualifying test, given feedback after every item, and paid a bonus per correct answer with an extra bonus at 90 and 95 per cent accuracy — set above the model's 84 so that copying the machine could not earn it. About a hundred workers took each condition.

Three ways to explain

A pilot on the beer data, three conditions of fifty workers each with a logistic-regression classifier, established the shape of the problem before the main study. Explaining only the predicted class, by highlighting the words that pushed towards it, produced better accuracy than also highlighting the words that pushed the other way. Neither improved on showing confidence alone. And the split that would become the paper's central result was already visible: explaining the predicted class helped a little when the recommendation was right and hurt when it was wrong.

Out of that came the authors' own proposal. If explaining the alternative makes people sceptical, and mistakes correlate with low confidence, then the assistant should explain the alternative only when it is unsure. Their adaptive explanation switches between the two strategies at a confidence threshold, set at each dataset's median so that half the items were shown each way: 89.2 per cent for beer, 88.9 for books. The split was fortunate. Of the twenty-five items shown with the alternative explained, seven were items the model had got wrong; only one of its errors fell on the other side. They also added explanations written by one of the authors, as an upper bound on what a better explainer could produce, since pilot workers had complained that the machine's highlights did not match its behaviour.

What happened

Every assisted condition reached the band. On beer reviews, showing the recommendation and its confidence brought workers to 0.89 accuracy against 0.84 for the model and 0.82 unassisted. Book reviews and logic questions produced the same pattern, with the confidence-only team gaining 2.2 and 20.1 per cent respectively over unassisted workers. That much answers the paper's title question affirmatively: the whole did exceed its parts.

None of the explanations improved on that. Explaining one class and explaining two did not differ from each other on any dataset, and neither differed from the confidence baseline: on beer reviews the confidence team scored 0.89 and the explained team 0.88, a difference the authors could not distinguish from noise. The adaptive strategy did not beat confidence on any of the three tasks, and switching its explanations from the model's to the author's made no difference. Three of the four hypotheses went unsupported, and the fourth was the prediction that explanations would not help.

The reason appears when the items are split by whether the machine was right. Explaining the predicted class improved decisions when the recommendation was correct and worsened them when it was not, so the averages came out flat. This is the paper's finding about explanations as such: they increase the chance that a person accepts the recommendation, and they do it whether the recommendation deserves acceptance or not. Psychology had shown that a human explanation makes listeners agree even when the explanation is wrong, and earlier work had found that explanations led data scientists to over-trust models; what this study adds is the effect measured on the accuracy of real decisions at scale rather than on reported trust.

The adaptive strategy moved the right variable without moving the outcome. Measuring agreement between worker and machine, the authors found it did what it was designed to do: agreement fell on the low-confidence items, where the model was more likely to be wrong, and held on the high-confidence ones. It was not enough, for two reasons they name. On the confident items the worker still has to spot the mistakes the model makes with high confidence, which are the hardest kind to catch. On the unconfident items the worker has to disagree and then solve the problem correctly alone, and doubting the machine does not supply the answer.

The three tasks also behaved differently, and the difference was in the overlap of errors. Plotting each study item by how many unassisted workers got it right, the authors found that beer reviews and logic questions contained a substantial set of items the model handled and people did not, which is where assistance pays. Book reviews contained few; the model's mistakes there were mostly on items people found easy. Accordingly, book-review workers learned to discount the assistant, and it was the one dataset where explaining the predicted class improved matters when the model was wrong, by five per cent over confidence alone. Comparable accuracy, in other words, is not the same as complementary skill.

Subjective ratings tracked the quality of the explanation rather than the quality of the decisions. Seventy per cent of beer workers called the assistance useful under the author-written adaptive design against fifty-seven per cent under confidence alone, and the hand-written explanations were rated useful ten to fifteen points more often than the machine's. None of that preference showed up in accuracy.

How the workers said they used it

A coded analysis of 409 free-text responses describes what the interface produced. Most workers put themselves between deference and refusal: 47 per cent used the recommendation as a starting reference before reading, 25 per cent made their own decision and then checked it, 22 per cent mostly ignored the assistant, and 6 per cent mostly followed it. The share who ignored it rose as the assistant's errors grew familiar — 11 per cent on the logic task, 23 on beer, 30 on book reviews.

Confidence was the feature people built a model of. Of the ninety workers who mentioned using it, 27 per cent reported an explicit cut-off below which they stopped believing the machine — around 83 for beer, 89 for books, 90 for logic, and as low as 65 for some. One put it plainly: "If Marvin was above 85% confidence, I took his word for it." The same share mentioned using confidence whether or not they had been given explanations. Thirteen per cent handed the decision over when they could not decide themselves, which is the truth-default behaviour psychology describes. Of those who used the explanations, most used them to read faster; five per cent used them to check whether the machine's reasoning made sense, and on the logic task almost nobody did, because verifying an argument costs more than verifying a highlight.

Informative rather than convincing

What the authors take from this is that explanations as designed produce compliance rather than judgment, which they call ethically fraught in exactly the domains where a person is kept in the loop for moral or legal reasons. If the explanation merely soothes the doctor, the requirement that a doctor sign off has been satisfied in form and emptied in substance.

Their design imperative follows from the fact that hand-written explanations, the best available, helped no more than the model's: "Explanations should be informative, instead of just convincing." Building an explainer to produce the most persuasive justification of an answer, which is what many methods optimise for, is then the wrong objective. The example they hold up is a model of pneumonia risk whose structure let clinicians see that it had learned that asthma lowers the risk of dying — a spurious correlation the explanation made visible. An explanation that only justifies cannot do that.

Two directions follow. Since confidence cannot reveal the confident mistakes, an assistant might adapt to agreement instead, arguing against its own recommendation when the person concurs too readily, or presenting evidence for several options and letting the person ask for more. And timing matters as much as content: every condition here showed the machine's answer before the person had formed their own, which is close to a designed anchoring effect. Delaying the recommendation, showing it after a first answer, or offering it only on disagreement would each protect independence, and each would cost the speed that makes assistance attractive.

The limits are stated plainly. Accuracy was the only dimension measured, and a team might be complementary in speed or effort instead. The tasks were crowdsourceable, so the results need not carry to expert domains, and expertise was not tested. Only two formats were tried. The conclusion is not that explanations are useless, but that on this measure they added nothing to a number the model was producing anyway.

Since 2021

This paper and Buçinca's appeared within months of each other and together changed the question the field was asking. From then on, the baseline an explanation had to beat was the model's confidence score, and the measure was the accuracy of the decision rather than a person's ability to predict what the model would say. The vocabulary settled around appropriate reliance, and by 2022 there were metrics for it that separate correct overrides from correct deference.

The diagnosis has held. Explanations that justify an answer raise acceptance without improving discrimination, and later work has mostly established when the rule breaks rather than overturned it. Vasconcelos and colleagues showed in 2023 that explanations reduce over-reliance where checking the explanation is cheaper than doing the task, which recasts the problem as one of cost. Fok and Weld, both authors here, argued the same year that explanations rarely produce complementary performance because they justify an answer instead of letting the person verify it, which is this paper's design imperative restated as a design principle. The evidence on complementary performance itself has been less kind than this study's own result. Vaccaro, Almaatouq and Malone's meta-analysis of 106 experiments, published in 2024, found that on average human-machine combinations performed worse than the better of the two alone, with losses concentrated in decision tasks of just this kind and gains in tasks of creation, and with synergy appearing mainly where the person was already the stronger partner. Read against that, what this paper achieved by matching the model's accuracy to the worker's looks less like a typical result than like a demonstration of the narrow conditions under which teaming pays at all.