The Evaluating Self

34Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

Joel Becker, Nate Rush, Beth Barnes and David Rein, METR, arXiv 2507.09089, 2025

Linked to 30 chapters: supports 20 · contradicts 5 · mechanism 13 · same claim 5 · pairing 4

A field trial of AI coding tools — the argument in five moves The gap The trial The result Where thetime went Twenty-onefactors Lab studies use faketasks; field studiescount lines andcommits. 16 developers, 246real issues, a coinflip, and screenrecordings. Forecast 24% faster.Measured 19%slower. Believed20% faster after. Coding and readingfall; prompting,waiting, reviewingand idling rise. Five contribute, tenunresolved, sixruled out. The developers who had just been slowed down estimated that they had been sped up.
Figure 34.0 — The paper in five moves: a measurement problem, a trial, a result nobody predicted, a breakdown of the working hours, and an audit of twenty-one ways the result could be an artefact.

Sixteen experienced open-source developers handed the scheduling of their own work to a coin. Each supplied a list of real issues on a repository they had contributed to for an average of five years. Each issue was forecast, then randomly assigned to allow or forbid AI assistance, then implemented, submitted as a pull request, reviewed by another maintainer and merged. Before the coin was tossed, the developers expected AI to cut their completion time by 24 per cent. Economists asked to forecast the study's outcome said 39 per cent faster; machine learning researchers said 38. The issues on which AI was allowed took 19 per cent longer than the issues on which it was forbidden. Asked at the end, after months inside the experiment, the same developers estimated that AI had made them 20 per cent faster. Becker, Rush, Barnes and Rein spend the rest of the paper trying to break their own result. They list twenty-one properties of their setting and their procedure that could have produced it and report the evidence for and against each. Five survive as likely contributors, ten are left unresolved, six are ruled out. The distance between what the developers felt and what the clock recorded survives all of it.

What the earlier studies measured

Two bodies of work had already answered the question and answered it the other way. The first ran controlled experiments on synthetic tasks: Peng and colleagues measured a 56 per cent speedup, Weber and colleagues a 65 per cent rise in requirements satisfied, Paradis and colleagues 21 per cent faster. The authors' objection is that the tasks are not the work. Peng's participants wrote a basic HTTP server in JavaScript against test cases shown to them in advance, a task both unrepresentative of most development and likely to resemble a great deal of training data. The second body of work used real repositories and natural or randomised variation, and reported increases of 14 to 51 per cent — but measured lines of code, commits and pull requests. Every one of those can move without any work getting done faster. Code can be more verbose and functionally identical; a unit of work can be split into three pull requests instead of one.

So the paper fixes the outcome. The unit is one issue, defined and forecast before the coin is tossed, and the measure is how long it takes to finish it. Nothing the developer or the model does afterwards can change what counts as done. That single decision is what separates this study from its predecessors, and it costs the authors nearly everything else: no scale, no automation, sixteen people and 246 issues.

The design

Developers came from the authors' professional networks and from outreach to active contributors on large GitHub projects. They are experienced engineers, typically with more than a decade behind them, and they are insiders on the repositories they worked in: five years of contributions on average, 1,500 commits, a span covering 59 per cent of the project's life. They were paid $150 an hour. The repositories are stdlib, jsdom, scikit-learn, GHC, Cabal, Hypothesis, flair, transformers, gpt-neox and others — on average about ten years old, 23,000 stars, 4,900 forks, 710 committers and over 1,100,000 lines of code, with contribution standards to match.

One issue, from definition to a recorded completion time Developer lists real issues Forecasts it with AI and without Simulated coin flip Implemented, screen recorded Pull request reviewed and merged 246 issues, each under two hours of work The forecast doubles as the difficulty control Randomisation comes after the task is fixed 136 allowed: any tool, mostly Cursor Pro with Claude 3.5 or 3.7 Sonnet. 110 disallowed. Time self-reported before and after review
Figure 34.1 — The protocol. The issue and its two forecasts are fixed before randomisation, so the treatment cannot change what counts as finishing.

Each developer listed real issues, breaking anything longer than two hours into smaller pieces, and forecast for each how long it would take with AI and without. Those forecasts do double duty: they measure what the developer expects from AI, and they serve as a difficulty control in the regression, valid precisely because they were collected before assignment. A simulated coin flip then sent each issue to one condition or the other, producing 136 AI-allowed and 110 AI-disallowed issues, averaging two hours each. In the allowed condition a developer could use any tool or none; the authors supplied Cursor Pro subscriptions and basic training, and developers mostly used Cursor with Claude 3.5 or 3.7 Sonnet. In the disallowed condition no generative tooling was permitted, though search engines remained. Developers recorded their screens and self-reported implementation time before and after review. The effect is estimated by regressing the logarithm of completion time on the treatment and on the logarithm of the developer's own no-AI forecast.

The forecasts and the result

Raw averages make the shape plain. Across all 246 issues the developers forecast that AI-allowed work would take 0.76 times as long as AI-disallowed work. The issues that were actually allowed AI took 1.34 times as long as those that were not, with a confidence interval from 1.05 to 1.70. The regression pulls the estimate down to 19 per cent, because the coin happened to send slightly harder issues to the AI-allowed arm, and the forecast control corrects for that. Nearly every quantile of the completion-time distribution shows the allowed issues taking longer, and three developers in four were slowed.

Predicted and measured change in completion time 0% faster slower -39% -38% -24% -20% +19% Economicsexperts (34) Machinelearning (54) Developers,before Developers,after Measured
Figure 34.2 — Predicted and measured change in completion time when AI is allowed, from the paper's Figure 1. Negative is faster.

The developers were not bad at estimating work. Their forecasts correlate with actual times at 0.64 for AI-allowed issues and 0.59 for AI-disallowed ones: they know which issues are hard. What is inverted is only the sign of the AI term. They are even calibrated in a relative way — on the third of issues where they predicted the largest speedup, they were not slowed down at all, and the slowdown falls steadily as the predicted speedup rises. The error is not noise but a constant offset applied to every estimate, and it did not shrink with experience. Thirty-four economists and fifty-four machine learning researchers, given the design, the repositories and the participants' AI background in advance, missed in the same direction by twice as much.

Where the time went

The authors labelled 128 screen recordings by hand at a resolution of about ten seconds, 143 hours of video covering 29 per cent of all the time developers spent, then filtered out broken recordings, the handful of detected violations and any issue whose recording disagreed with the self-report by more than a fifth. Seventy-four recordings and 84 hours survived, labelled into eight activities and twenty-seven finer categories.

Share of working time, by activity (per cent) AI disallowed (30 recordings) AI allowed (44 recordings) only when AI is allowed 09 08 04 3625 2517 1814 1312 610 ReviewingAI output PromptingAI Waitingon AI Activecoding Reading &searching Testing &debugging Git &setup Idle /other
Figure 34.3 — Share of working time by activity, from the paper's Figure 6. The three activities on the left exist only in the AI-allowed condition.

Three activities appear only when AI is allowed and together take about a fifth of the working time: prompting the model, waiting for it to generate, and reviewing what it produced. The time is not drawn from one place. Active coding falls from about 36 to about 24 per cent of the session, reading and searching from about 24 to about 16, testing and debugging from about 17 to about 14. Git and environment work is unchanged. Idle time rises from about 6 per cent to about 10. The developer stops writing and stops reading, and the recovered hours go into operating the model and into nothing at all.

Twenty-one ways to be wrong

The audit is the longest part of the paper and the reason it is worth reading twice. The twenty-one candidate explanations are sorted into four types: mechanisms by which AI directly costs time, artefacts of the experiment, features of the setting that raise the human's performance, and features that lower the model's.

Twenty-one candidate explanations for the slowdown Evidence for — 5 Unresolved — 10 Ruled out — 6 Over-optimism about AI Familiarity with the repo Large, complex repos Low AI reliability Implicit repo context Overuse from the study Task distribution AI enlarging scope Completion order Recruitment bias Speed traded for ease Weak first pull requests Below-average tool use Generation latency Suboptimal elicitation Unfamiliar editor Cheating or non-use Issue dropout Outcome measure Estimator choice Older models Two of the five that survive — deep familiarity and very large codebases — are absent from most work.
Figure 34.4 — The audit, from the paper's Table 1: what the authors' own evidence supports, leaves open, and rules out.

Five factors have evidence behind them. Developers over-estimate AI's usefulness both before and after each issue, which is why they keep using a tool that is costing them time. Their familiarity with the repository works against the model: slowdown is larger on issues where they rate their prior exposure to the task type as high and their need for external documentation as low, and one maintainer of a specialised area put it flatly — "there is no way agent mode can do better than me". The repositories are large and old, and developers describe edits leaking into unrelated parts of big files. Reliability is low: fewer than 44 per cent of Cursor's generations are accepted, three developers in four read every line of what is accepted, more than half often rewrite it substantially, all of them modify it, and 9 per cent of AI-allowed working time goes to reviewing and cleaning output. And the repositories carry knowledge that is nowhere written down — which special case must be preserved for backward compatibility, where a change belongs — that the developer holds and the model cannot be given.

Ten factors remain unresolved, and the authors say so rather than picking a side. Developers who report overusing AI because of the experiment appear more slowed than those who say they used it normally, which the authors decline to read as a clear effect. The issues were real but small: one developer called them "the same set of tasks, but sampled from the small end". Developers wrote 47 per cent more code per forecast hour when AI was allowed, which looks like scope creep until you notice that the developers who report scope creep are slowed less than those who do not. Latency accounts for only 4 per cent of the time. Cursor samples few tokens, so better elicitation might help — but the studies reporting speedup sampled few tokens too.

Six are ruled out. Restricting to developers who used comparable editors in both arms gives a slowdown of 24 per cent, so the unfamiliar editor is not the cause. AI went unused in only 16 per cent of allowed recordings, and there were three violations in fifty-four disallowed ones, one of them apparently accidental. Excluding developers with any dropped issues gives 21 per cent. Using recording length instead of self-reported time gives 25 per cent; deliberately extreme imputations of the missing post-review times give 14 and 23; four alternative estimators all land in the same place. And the models were current: Claude 3.7 Sonnet with and without thinking in 59 per cent of issues, Claude 3.5 Sonnet in 23.

Since 2025

The authors fence their result more carefully than most of its readers did. They state in a table what they have not shown: that AI does not speed up most developers, that it fails in other domains, that future models will not speed up this exact setting, that better prompting or scaffolding or repository-specific tuning could not turn the sign around, that developers with far more tool experience would see no gain. Some of their own participants were sped up. Two of the five factors they do credit — deep familiarity and enormous mature codebases — are absent from most software work, and they say outright that small greenfield projects and unfamiliar codebases are consistent with substantial speedup. They also report preliminary evidence that autonomous agents on Claude 3.7 Sonnet could implement the core of issues in these same repositories while failing the documentation, the linting and the tests.

The one hint that experience reverses the effect did not hold. The published figure showed positive speedup in the bucket above fifty hours of Cursor use, on the strength of a single developer; after publication a second developer reported having had more than a hundred hours of prior experience, and the authors added a footnote to the July revision saying that including him moves that bucket from roughly 25 per cent speedup to roughly zero. They left the figure unchanged and flagged the correction instead.

What is contested is the size and reach of the slowdown: sixteen developers, 246 issues, a confidence interval whose lower end sits near no effect, and a sample selected for exactly the conditions the authors name as causes. Most of the work was done in February and March 2025, before Claude 4 and before the command-line agents that arrived later that year, so the tooling measured was superseded within months of being measured. What has held up is the other half, and it is the half that does not depend on the size of the effect or on which model was used: the developers' estimate of their own speed was wrong in sign, wrong by the same amount before and after, and wrong for people who had just spent months being measured. That is a claim about self-knowledge rather than about any particular tool, and the method the paper used to expose it — fix the task before the coin, measure the time, and do not ask the person afterwards how it went — outlasts the generation of models it was built to test.

Claims drawn from this chapter · 7

evidenceTwenty-one ways to be wrong

Developers reject most AI suggestions and spend real time fixing the rest

In METR's trial, developers accepted fewer than 44 per cent of what the model wrote. Reviewing and cleaning up the code they did accept took 9 per cent of their working time when AI was allowed.

Becausethe acceptance rate comes from Cursor's own logs and the review share from the hand-labelled recordings; three developers in four report reading every accepted line and more than half often rewrite it substantially.
LimitsAcceptance rates are tool-specific and move with model quality; the 9 per cent is an average over a small sample of recordings. Nothing here says review time is wasted — only that it is not free.
PracticeFailure-mode review — budget review and repair as a line item rather than assuming generated code arrives finished, and record what you had to change.
After AIAcceptance rates have risen since 2025, which moves the number and not the structure: whatever is accepted still has to be read by someone who could have written it, and that person's ability to read it is what the next claim in this chain is about.
evidenceWhere the time went

Developers using AI spend a fifth of their time prompting and reviewing

METR found that about a fifth of working time in AI-allowed sessions went to prompting, waiting and reviewing. That time came out of writing code (36 to 24 per cent) and reading (24 to 16), not out of debugging.

Because74 hand-labelled screen recordings covering 84 hours show active coding falling from about 36 to 24 per cent of the session and reading and searching from 24 to 16, while testing and debugging barely moves and idle time rises from 6 to 10.
LimitsThe recordings cover 29 per cent of the study's time before filtering and 84 hours after it, at a labelling resolution of about ten seconds. The activity split is descriptive and cannot say which displacement causes the slowdown.
PracticeReliance audit — account for where an assisted hour actually goes before claiming the assistant saved one; the recovered time comes out of the activities that build the knowledge you will need to check its output.
After AIThe three activities that exist only in the assisted condition — prompting, waiting, reviewing — are the ones that grow as tools become agentic. The displacement of reading is the part to watch, because reading the system is how the reviewer stays able to review.
evidenceWhat the earlier studies measured, and Twenty-one ways to be wrong

Developers wrote more code in the condition where the work took longer

How much work appears to be produced can move in the opposite direction from how long the work takes. In METR's trial developers wrote 47 per cent more code per estimated hour while the same issues took them 19 per cent longer.

Becausethe two quantities are measured on the same issues in the same trial and move in opposite directions, which is what the field studies reporting 14 to 51 per cent gains could not have detected, because lines, commits and pull requests were all they measured.
LimitsThe 47 per cent figure sits among the ten factors the authors leave unresolved: it looks like scope creep, except that developers who report scope creep were slowed less than those who do not. The claim is about what output measures cannot show, not about what they always show.
PracticeReliance audit — fix the unit of work and its definition of done before the tool is introduced, so that nothing the tool does downstream can change what counts as finished.
After AIEvery dashboard built to demonstrate an assistant's value counts the things that moved here in the wrong direction. The only measure that survived the trial is elapsed time on a unit of work fixed in advance.
evidenceThe forecasts and the result

Developers' estimates of AI's effect on their speed get the direction wrong

The same developers predicted AI would make them 24 per cent faster, and said afterwards it had made them 20 per cent faster. Measured, they were 19 per cent slower. Months of using the tool did not shrink the error.

Becausethe same people forecast 24 per cent faster before the trial and reported 20 per cent faster after months inside it, against 19 per cent slower measured; the error is a constant offset rather than noise, and it did not shrink with Cursor hours.
LimitsThe offset is measured on one kind of work by sixteen people. The developers were not bad estimators in general: their forecasts correlate with actual times at about 0.6, and only the AI term is inverted. Whether the same offset holds for greenfield work is untested.
PracticeCalibration log — record the estimate before the work and the clock afterwards, because the estimate does not repair itself through use.
After AIThe paper's method — fix the task before the coin, measure the time, and do not ask the person afterwards how it went — outlasts the model generation it was built to test. Any claim about an assistant's value that rests on how the work felt is now the weakest kind of evidence available.
evidenceThe forecasts and the result

Experienced developers took 19 per cent longer using AI on familiar code

In METR's randomised trial, experienced developers fixing issues in large codebases they knew well took about 19 per cent longer when they used AI assistance than when they did not.

Becausea randomised trial on 246 real issues from 16 developers, with a raw ratio of 1.34 (CI 1.05–1.70) and a regression estimate of 19 per cent using the pre-assignment forecast as the difficulty control.
LimitsSixteen developers and a confidence interval whose lower end sits near no effect. The sample was selected for the exact conditions the authors name as causes — deep familiarity and enormous mature codebases — and both are absent from most software work. Some participants were sped up. Most of the work was done in February and March 2025, before the command-line agents that arrived later that year.
PracticeManual flying hours — keep a protected block of work done without the assistant on the systems you know best, and time it, so the comparison exists as a measurement rather than an impression.
After AIThe tooling this measured was superseded within months, so the number is dated on arrival. What survives is the shape: on ground the worker already owns, the delegate's contribution is negative and invisible to the worker.
evidenceThe forecasts and the result

Experts predicting the trial were more wrong than the developers in it

Experts who predicted the METR trial's result from outside were wrong in the same direction as the developers taking part, and by a wider margin: economists predicted 39 per cent faster and machine learning researchers 38, against 19 per cent slower measured.

Becausethirty-four economists and fifty-four machine learning researchers were given the design, the repositories and the participants' AI background in advance, and still missed by roughly twice the participants' margin.
LimitsTwo convenience samples of forecasters answering a single question. Nothing here separates a bias about AI from a general optimism about any intervention described in advance.
PracticeCalibration log — write down what you expect a tool to do for a team before it is deployed, with a date and a number, and score it; expertise in the field does not substitute for the record.
After AIThe forecasters were not lay observers but the professions that advise on AI deployment. Their miss is the reason an organisation's expectations of an assistant should be treated as a hypothesis to be scored rather than a premise to plan against.
evidenceTwenty-one ways to be wrong

In METR's study developers lost most time on code they knew best

The AI assistance slowed developers down most on the parts of the code where their own knowledge was greatest and least written down. Where they had worked longest, they lost the most time.

Becauseslowdown is larger on issues the developers rate high for prior exposure and low for need of external documentation, in repositories averaging ten years, 1.1 million lines and five years of contribution per developer.
LimitsThe factor is one of five the authors credit and one of twenty-one they consider; ten of the twenty-one are left unresolved. Self-rated prior exposure is a coarse proxy for tacit knowledge.
PracticeDomain audit — mark the parts of your system where what you know is not written down anywhere, and treat those as the places where delegation costs rather than saves.
After AIThe boundary the model cannot cross is what nobody wrote down. Retrieval over a repository has narrowed it since 2025 for anything that exists in text, and left it exactly where it was for the conventions that live only in the maintainer's head.

All claims · Source map