47When Combinations of Humans and AI Are Useful

Michelle Vaccaro, Abdullah Almaatouq and Thomas Malone, Nature Human Behaviour 8, pp. 2293–2303, 2024

Linked to 22 chapters: supports 17 · contradicts 4 · mechanism 7 · same claim 1 · pairing 2

The promise attached to machine assistance is that a person and a system together will do better than either apart. Vaccaro, Almaatouq and Malone put the promise to the experimental record. They searched three databases for studies published between January 2020 and June 2023 that reported all three of the quantities the question needs — the person working alone, the machine working alone, and the two working together — found 74 papers reporting 106 experiments, and extracted 370 effect sizes. Pooled against the better of the two partners, the combinations came out below it, by a small margin that was statistically reliable. Pooled against the person alone, the same combinations came out well above. Both numbers describe the same experiments. The systems helped the people in them and would still have been beaten, on the measures the researchers themselves chose, by discarding the person or discarding the machine, whichever was stronger. The rest of the paper is an account of when that is not so: the combinations gained where the person was the better partner, lost where the machine was, and gained on tasks that asked for something to be made rather than something to be chosen.

Synergy and augmentation

The paper begins by separating two outcomes that the literature had been running together. Human augmentation means the combination performs better than the person alone. Synergy — strong synergy, in the vocabulary the authors borrow from research on small groups — means the combination performs better than the person alone and better than the machine alone. Augmentation is the weaker test, and it is the one most studies apply. The authors accept that it is often the right one: where full automation is barred by law, by ethics or by safety, or where the system does not share the values of the people it serves, whether the person is better off with the tool is exactly the question that matters.

But when people speak about the potential of these systems, the authors argue, they mean the stronger claim, because otherwise "they would just use the best of the two". Which baseline you pick decides what you find. Reported against the person alone, a system is a success; reported against the stronger of its two parts, the same system is an expensive way to do worse.

The search

The review was preregistered. On the advice of a library specialist, the authors searched the Association for Computing Machinery Digital Library, the Web of Science Core Collection and the Association for Information Systems eLibrary, the three places where work on assisted decisions is scattered across computer science, information systems and the social sciences. The search string joined four facets with AND, each a list of synonyms: a human term, a machine-learning term, a collaboration term and an experiment term, matched in the abstract. The window ran from 1 January 2020 to 30 June 2023, chosen to keep the sample to current systems. Every study that qualified was then searched forward and backward through its citations.

To be included, a paper had to report an original experiment with human participants and to give quantitative performance for all three conditions. Papers that reported the person and the combination but not the machine alone were excluded, as were the reverse, and so were reviews, theory, qualitative studies, commentary and simulations. The paper also had to supply the design, the sample sizes and the standard deviations, or enough to compute them. That last requirement did most of the filtering: 5,126 papers came back from the search and 74 survived. Where the numbers existed only inside a plot, the authors wrote to the corresponding author and, failing a reply, read the values off the figure with a digitiser.

Every comparison was converted to Hedges' g, a standardised mean difference that carries no units, so that accuracy, error rate and quality score can be pooled, and that corrects the upward bias of Cohen's d in small samples. For synergy the baseline is whichever partner performed better alone; for augmentation it is the person alone. Because a single experiment often contributes several effect sizes — several treatment arms, several outcome measures — the authors fit a three-level random-effects model with effect sizes nested inside experiments, computed variance estimates clustered at the level of the experiment, and applied the Knapp–Hartung adjustment when testing.

Worse than the better of the two

Against the stronger baseline the pooled effect was negative: g = −0.23, 95% confidence interval −0.39 to −0.07, P = 0.005. By convention that is a small effect, and it is reliably below zero. Of the 370 effect sizes, 213, or 58 per cent, fell below the better partner and 157 rose above it. Against the person alone the pooled effect was g = 0.64, interval 0.53 to 0.74, medium to large, with 314 of the 370 effect sizes positive.

Both averages sit on top of enormous spread: I² was 97.7 per cent for synergy and 93.8 per cent for augmentation, which is to say that almost none of the variation between studies is sampling noise. The authors put the headline through four checks. Clustering by paper rather than by experiment gave −0.22 and 0.65. Dropping the eleven outliers gave −0.25. Leaving out one effect size, experiment or paper at a time moved the synergy estimate only between −0.28 and −0.19, significant in every case. Dropping the values that had been read off plots gave −0.21.

Funnel plots and Egger's regression found no asymmetry for synergy, and the authors add an argument that does not depend on the test: a publication bias in this field would favour papers showing gains, and the pooled result shows a loss. For augmentation the picture is different. Egger's regression did indicate bias (β = 1.96, P = 0.002), as did the rank correlation test, which the authors read as a sign that journals and researchers implicitly expect the comparison to the person alone and print it when it comes out well. They left the estimate uncorrected rather than apply adjustment methods that can overcorrect.

Decisions and creations

Task type moderated synergy (F(1,104) = 7.84, P = 0.006). Among decision tasks — those where the participant chose among a fixed set of options — the pooled effect was −0.27, interval −0.44 to −0.10. Among creation tasks — those producing open-ended content — it was +0.19, interval −0.09 to 0.48. The creation estimate rests on 34 effect sizes and cannot be told apart from zero, but it is reliably different from the decision estimate. The imbalance is the point: about 85 per cent of the effect sizes in the sample came from decision tasks and about 10 per cent from creation tasks. The type of data the task involved — images in 172 effect sizes, text in 96, several kinds in 78, numbers in 24, video in 8 — moderated both outcomes.

The explanation the authors offer for the creation advantage is that a creation task carries its own division of labour. Producing an image takes an idea of what the image should be, and then a quantity of routine detail. Producing a document takes knowledge the machine does not have, and then boilerplate it can supply. The task splits itself into the part where the person is better and the part where the machine is, and the split happens without anyone designing it. In the decision tasks that fill the sample, both parties performed the whole task and the person chose at the end, so there was nothing to split.

That points at the moderator with almost no evidence behind it. Only 3 of the 106 experiments assigned separate subtasks to the partner better suited to them in advance. Their four effect sizes pooled to g = 0.22 with an interval from −0.42 to 0.87. The authors present this not as a finding but as the design the field has barely tried.

Who was better alone

The strongest moderator by far was which partner performed better on its own (F(1,104) = 81.79). Where the person beat the machine alone, the combination beat both: g = 0.46, interval 0.28 to 0.66. Where the machine beat the person, the combination fell below the machine: g = −0.54, interval −0.71 to −0.37. Augmentation moves the other way. Where the machine was the better partner, the gain over the person alone was at its largest, g = 0.74. The two measures are most divergent in exactly the case where the tool looks most impressive.

The pattern also disposes of the simplest model of what a combination does. If joint performance were some average of the two performances, it would always lie between them, and, as the authors put it, "human–AI synergy could never exist". It exists; so the combination is doing something other than averaging.

Their account of what it is doing turns on deferral. In more than 95 per cent of the systems in the sample, the person made the final call after seeing the machine's output, so the person's real task is to judge, case by case, whose answer to take. Skill at that judgment is not independent of skill at the task. Someone who cannot tell a hard instance from an easy one also cannot tell when the machine has gone wrong, and their intervention subtracts.

The illustration comes from Cabrera, Perer and Hong, who in 2023 ran the same interface, the same participant pool and the same machine accuracy across three separate tasks. On detecting fake hotel reviews the machine scored 73 per cent, the people 55, and the pair 69 — below the machine. On classifying bird images the machine scored 73, the people 81, and the pair 90 — above both. Nothing changed but which partner was more accurate.

Several other moderators reached significance: the type of machine used (F(2,103) = 3.77, P = 0.026, splitting deep models, shallow models and human confederates standing in for a system), the year of publication, the kind of output the task demanded, and, for augmentation only, whether the design used dependent or independent samples. The ones that did not are the more interesting list.

Explanations and confidence displays are the two interventions the field had concentrated on for five years, and across more than 300 effect sizes neither shifted the pooled result. Nor did it matter whether the participants were domain experts or paid online workers. The authors' recommendation is to move attention off those questions and onto the three that did move the estimate: which partner is better to begin with, what kind of task it is, and how the work is divided.

What the analysis cannot settle

The inclusion rule that makes the comparison possible also narrows it. A study qualifies only if all three performances were measured, which excludes every task that neither partner can do alone — and those are the cases where the argument for combining them is strongest.

Pooling Hedges' g across accuracy, error and quality, and across doctors, students and paid online workers, buys comparability at some cost to it. Measurement error varies too: one study in the sample estimated accuracy from 500 images, another from 15, and the three-level model weights by variance across participants without accounting for that difference. Study quality varies in ways the peer-review filter does not catch, since attention checks and incentive schemes differed from experiment to experiment. Most of the heterogeneity remains unexplained, and the authors suspect interactions among the coded variables that this many studies cannot detect.

The limitation they press hardest is about what researchers chose to study. The results apply to the tasks, processes and participant pools that appear in the literature, and those may not resemble how such systems are actually configured outside a laboratory. Even without publication bias, there may be a topic-selection bias.

They end by arguing that overall accuracy is the wrong single measure. Near the ceiling, equal increments are not equally hard to win. In radiology or bail prediction, rare errors carry costs out of all proportion to their frequency, so a person who removes rare catastrophic errors can be worth keeping even where the machine is more accurate on average — and for those settings augmentation is the appropriate test after all. The paper asks for composite measures that carry completion time, financial cost and the cost of each kind of error, for a set of commensurability criteria covering benchmark tasks, quality thresholds, incentive schemes and a taxonomy of interaction processes, and for an open repository of raw data and interaction logs so that results can be synthesised rather than merely counted.

Since 2024

The search window closed in June 2023, which puts almost the whole of the generative era outside the sample. What was pooled is mostly classifiers on images and text, judged by paid online workers making bounded choices, and the paper's own roadmap says so: study creation tasks, and report the machine alone.

The pattern has held where it has been tested. Goh and colleagues randomised physicians in October 2024 to diagnostic reasoning with a language model or with conventional resources; the assisted physicians did not reliably beat the unassisted ones, and the model working alone scored above both groups — the losing case the paper predicts whenever the machine is the better partner. In 2025 a trial of sixteen experienced open-source developers found that issues on which AI assistance was allowed took 19 per cent longer than issues on which it was forbidden, which is a creation task where the combination lost, and a caution against reading the creation-task result as a general licence. Dell'Acqua and colleagues, running a field experiment at Procter & Gamble the same year, found that individuals working with a language model matched two-person teams working without one, but reported no condition in which the model worked alone, so the synergy question cannot be asked of it.

That omission is the finding that has travelled least. Reporting the person and the pair while leaving out the machine remains the default in studies of generative tools, and it is precisely the missing number. The distinction the paper drew has been adopted in name; the measurement discipline it asked for has not.