18To Trust or to Think
Zana Buçinca, Maja Barbara Malaya and Krzysztof Z. Gajos, Proceedings of the ACM on Human-Computer Interaction 5, CSCW1, article 188, 2021
Linked to 33 chapters: supports 24 · contradicts 5 · mechanism 11 · same claim 8 · pairing 10
People who are given a machine's recommendation tend to follow it even when it is wrong and they would have done better alone. Adding an explanation of how the machine reached its answer was meant to fix this, by letting the person see when the reasoning was faulty. It has not. Buçinca, Malaya and Gajos start from that failure and offer a reason for it. Reading an explanation carefully is effortful, and people do not do effortful things for every decision; they form a general impression of whether the machine is competent and act on that. An explanation, on this account, works as a signal of competence rather than as something examined, and can raise reliance rather than lower it. The remedy they borrow from medicine is the cognitive forcing function, an intervention at the moment of decision that interrupts the fast route and makes the person think. Three such designs, in a nutrition task with a simulated assistant that was right three times in four, cut the rate at which people followed wrong suggestions compared with two ordinary explanation designs, though they did not remove it. The cost was that people liked the designs that worked least, and the benefit went mostly to the people who already enjoy thinking.
The problem
The expectation behind decision support was that a person and a model together would beat either alone. What studies have found instead is that the pair usually beats the person and usually loses to the model, in medical treatment selection, deceptive review detection, loan default prediction and other tasks. The authors' reading is that the shortfall comes from overreliance: when the model is wrong, the person follows it anyway. Explanations were supposed to make wrong suggestions detectable, and the evidence to 2021 was that they did not; teams with explanations still did worse than the model alone, and one study found that the mere presence of an explanation raised trust and reliance whatever its content.
The paper's account of why uses the two-system vocabulary. Most decisions run on fast heuristic thinking, which serves well and is prone to predictable errors; analytical thinking is slow and costly and is invoked rarely. The authors are careful not to call the fast route bad, since pattern recognition built on experience is what expertise is, but experts too fall prey to bias, and the design of explainable systems assumes a person who engages analytically with every explanation. Since people are averse to that much effort, the assumption is likely false. So the task, they argue, is not only to make better explanations but to raise the person's motivation to engage with them at all.
Cognitive forcing functions
The idea comes from clinical medicine, where diagnostic errors are more often failures of information synthesis than of knowledge. Two kinds of remedy have been tried: educational strategies, which teach clinicians about pitfalls in advance, and cognitive forcing functions, which act at the moment of decision. Checklists, diagnostic time-outs and deliberate slowing are the standard examples, and the evidence is that the interventions applied at decision time work better than the ones taught beforehand. The authors list three analogues already tried in machine-assisted settings: making the person decide before seeing the recommendation, which counters anchoring; delaying the recommendation, since a slower algorithm has been shown to improve people's judgment of its accuracy; and letting the person choose whether and when to see it, since unsolicited advice that contradicts a first idea provokes resistance. They predict a price. Across visualisation and education research, people prefer easier designs and learn and perform better with harder ones, so an intervention that forces thought will probably be liked less.
The experiment
Experts are hard to recruit, so the task was designed for laypeople. Participants saw a photograph of a meal and had to make it low in carbohydrate by picking the ingredient highest in carbohydrate from a list of about ten and choosing a replacement, from another list of ten, that cut carbohydrate while keeping the flavour. Each meal had one ingredient with far more carbohydrate than the rest. The assistant was simulated rather than trained, so that its errors could be controlled: it recognised the right ingredient three times in four, and when it failed it recommended replacing the second highest ingredient, which on these plates was a low-carbohydrate one. Its explanation showed the top four substitutes with an estimated carbohydrate reduction and a flavour similarity computed from a database of flavour molecules, in the style of feature-based explanations.
Six designs were compared. With no assistant, participants saw the photograph and the two menus. Two simple explainable designs showed the suggestion and explanation at once, one of them adding a confidence statement when the assistant was unsure. Three forcing designs withheld it in different ways. On demand hid the suggestion behind a button. Update required an unassisted decision first, then showed the suggestion and allowed a revision, on the hypothesis that a disagreement would make the person curious about the explanation. Wait showed a processing message for thirty seconds before the suggestion appeared, on the hypothesis that the person would form their own answer in the meantime and then check the explanation against it. Each of 199 participants recruited online, after exclusions, saw two designs in two blocks of thirteen questions, with three wrong suggestions in each block, and rated preference, trust, mental demand and complexity after each. After the first block they answered four items from the Need for Cognition scale.
What happened
On all questions taken together, any assistance beat none, and the forcing designs did no better than the simple ones. The difference appeared where the assistant was wrong. On those questions people with no assistant did best of all, people with simple explanations did worst, and people under cognitive forcing were in between: they identified the carbohydrate source correctly 27 per cent of the time against 8 per cent with simple explanations and 49 per cent unassisted. For the full decision, source and replacement together, the figures were 9, 3 and 18 per cent. Where people under simple explanation went wrong they overwhelmingly followed the assistant; forcing reduced that following, from 64 to 48 per cent on source detection, and raised correct answers, without changing the rate of the person's own unassisted errors. The three forcing designs did not differ from one another, and neither did the two simple ones.
| On wrong suggestions | Simple explanation | Cognitive forcing |
|---|---|---|
| Carbohydrate source found | 8% | 27% |
| Followed the wrong suggestion | 64% | 48% |
| Own error | 27% | 26% |
| Whole decision correct | 3% | 9% |
The subjective ratings ran the other way. Participants trusted the simple designs a little more, preferred any assistance to none, found the unassisted task most demanding, and rated the forcing designs as more complex than the simple ones. Trust and preference correlated negatively with performance on wrong suggestions; mental demand correlated positively with it. People did best in the conditions they preferred least, trusted least and found hardest, which is the trade-off the authors had predicted. They note one further implication. Studies that evaluate explanations with proxy tasks, asking people to predict what the model will say, force attention on the explanation and so behave like a strong forcing function, which is why such studies report better results than studies of real decisions.
Who it helped
The authors then audit their own intervention for what they call intervention-generated inequality, a benefit that flows mostly to a group already advantaged. The relevant trait here, they argue, is need for cognition, a stable disposition to enjoy effortful thinking, which earlier work had linked to the uptake of complex interface features. Splitting participants at the median, they found that people high in the trait did better on the task overall and that the gains from forcing fell mostly to them: for the full decision, forcing improved high scorers significantly and low scorers not at all, while both groups improved on source detection. High scorers also trusted and preferred the forcing designs less than the simple ones; low scorers rated the two the same. The result cuts both ways. The interventions widened a gap, and since low scorers did not mind them, stricter interventions might be aimed at the people who need them most without being rejected.
The paper closes on its limits. One non-critical task, one population, a simulated assistant; the authors expect the results to travel because experts are known to fall into heuristic thinking too, but they have not shown it. And an intervention people dislike may not be adopted, which leads them to propose adaptive designs that deploy the strongest forcing only on the instances where a wrong suggestion is likely and a person is likely to follow it.
Since 2021
The paper named a category, and cognitive forcing functions became standard vocabulary in the study of assisted decisions. Its central negative finding, that explanations raise reliance without improving its calibration, was confirmed in the same year by a larger study from Bansal and colleagues and in later meta-analyses, which find that explanations lift accuracy slightly and appropriate reliance hardly at all. Its positive finding was refined rather than overturned. Vasconcelos and colleagues showed in 2023 that explanations can reduce overreliance when the task is hard enough and the explanation easy enough that checking is cheaper than deferring, which turns the paper's motivation story into a cost-benefit one. Gajos and Mamykina found in 2022 that people who receive a recommendation first also learn less from the task, adding a second cost to the fast route. The adaptive designs proposed in the closing pages were built by the same group, which trained a policy to decide when to force and when to leave the person alone. The trade-off has not gone away. Every intervention that reduces overreliance so far makes the interface slower or harder, people still rate the easy designs higher, and the systems in wide use have kept the easy designs.