35How AI Impacts Skill Formation

Judy Hanwen Shen and Alex Tamkin, Anthropic, arXiv 2601.20245, 2026

Linked to 23 chapters: supports 18 · contradicts 1 · mechanism 8 · same claim 4 · pairing 6

Studies of AI at work measure the product: issues resolved per hour, tasks completed, pull requests merged. Shen and Tamkin measure what is left in the worker afterwards. They gave fifty-two professional Python programmers a library none of them had used, Trio, and thirty-five minutes to write two small programs with it. Half had a chat assistant that could produce the finished code for both tasks on request; half had the documentation and a search engine. Everyone then sat the same quiz on the library, with no assistant. The assisted group scored 4.15 points lower out of 27 — two grade points, a seventeen per cent drop, an effect size of 0.738 — and finished no faster. The assistance cost understanding and bought nothing. The average hides a split. Shen and Tamkin watched the screen recording of every participant and sorted the assisted group into six ways of using the assistant; the three that asked for reasons rather than only for code produced quiz scores of 65 per cent and above, and the three that handed over the writing or the debugging produced 24 to 39 per cent. Their conclusion is that productivity gained this way is "not a shortcut to competence", and that the skill it erodes first, debugging, is the one a person needs in order to supervise code a machine wrote.

The question the productivity studies leave open

The gains are real and documented. Peng and colleagues found crowd-sourced developers finished a task 55.5 per cent faster with Copilot; Cui and colleagues, inside large software companies, measured a 26.8 per cent rise in pull requests, commits and builds; Brynjolfsson, Li and Raymond found call-centre workers resolving 15 per cent more issues, and Dell'Acqua and colleagues found consultants completing 12.2 per cent more tasks. The same pattern runs through them: the less experienced the worker, the larger the gain. The exception is Otis and colleagues' trial with Kenyan small-business owners, where AI advice helped the strongest performers and hurt the weakest.

Every one of those studies measures output. None measures what the worker knows at the end. A second literature suggests the question is worth asking. Gerlich's survey associates frequent AI use with weaker critical thinking and more cognitive offloading, and Lee and colleagues find knowledge workers reporting less cognitive effort and less confidence. Wu and colleagues found a gain from generative AI that did not persist once people worked alone, and Wiles and colleagues called the effect an exoskeleton: technical ability that stands up only while it is worn. But the surveys are observational, and the retention studies ask whether a gain persists, not whether any learning happened at all.

Shen and Tamkin's hypothesis is plain. Accounts of learning by doing, from Kolb's experiential cycle to problem-based learning, treat completing a real task as the occasion on which the concepts are acquired; asking a model to generate the code, they argue, is a shortcut to completion with no learning stage in it. Software engineering makes a good test because the stakes are stated: if people are to verify machine-written code before it ships, they must be able to read and debug it.

Learning Trio in thirty-five minutes

Trio is a Python library for structured concurrency and input-output. Shen and Tamkin chose it for three reasons: it is less well known than asyncio, measured by the number of questions about it on Stack Overflow; it carries concepts a fluent Python programmer will not already have; and it is designed to be easy to use, so failure to learn it cannot be laid at the library's door. They built five tasks around it and kept the first two. The first asks for a timer that prints each passing second while other functions run, which introduces nurseries and running functions concurrently. The second asks for a record-retrieval function that handles missing records, which introduces error propagation and memory channels. Each task carries a problem description, starter code and a short account of the concepts it needs — the shape of an onboarding exercise for an internal library.

Every participant began with a warm-up on the same coding platform and no assistant: add a border around a list of strings, about four minutes' work, nothing asynchronous in it. Then thirty-five minutes on the two Trio tasks, with the treatment group prompted to use a chat assistant built on GPT-4o that could see their current code and, if asked, write the complete correct solution to either task. Then the quiz, which nobody took with help.

The quiz is the instrument the paper rests on. Following Cheng and colleagues' meta-analysis of assessment in introductory computer science, the authors distinguish four abilities: debugging, code reading, code writing and conceptual understanding. They drop code writing, on the ground that forgetting a function's syntax is a thirty-second problem for anyone with a search box, and keep the three a supervisor of machine-written code actually needs. Fourteen questions, twenty-seven points, seven Trio concepts. Five versions were screened by item response theory, so that each question correlated with the total, sat at a sensible average, and gave away no other answer. The grading rubric was filed with the pre-registration before the experiment ran.

Getting to that design took four pilots, and their failures are worth recording. On the first crowd-working platform 35 per cent of the control group used AI anyway, caught by logs showing them copy the instructions out and paste answers back; sharper instructions brought that down only to 25 per cent, and the authors moved to a second platform where they could verify compliance by watching the recordings. A third showed participants inferring answers from the code snippets in other questions, so the quiz was split across pages. In the fourth, only 60 per cent of the control group finished both tasks inside the limit against 90 per cent of the treatment group, and the recordings showed why: they were stuck on try/except blocks and string formatting, plain Python with nothing of Trio in it. Syntax hints were added for the main study. That pilot, before the hints, produced both effects at once — the AI group faster (d = 1.11, p = 0.03) and worse on the quiz (d = 1.7, p = 0.003).

What the quiz found

Fifty-two people completed the main study, twenty-six in each arm, balanced on years of coding experience, frequency of Python use, prior use of asyncio and a five-question asynchronous-programming check taken at recruitment. On time, the result was nothing: the difference in total task time between the arms was not significant (p = 0.391), even though the assistant would write both solutions on demand. On the quiz, a clear gap: 4.15 points of 27, Cohen's d = 0.738, p = 0.010, and d = 0.725, p = 0.016 with warm-up time held as a covariate. The gap holds at every level of experience — the control average is higher in the one-to-three, four-to-six and seven-plus year bands alike — which tells the authors the task presented something new to everyone rather than only to novices. The one thing the assistant reliably bought was completion: four of the twenty-six control participants ran out of time on the second task and none of the assisted group did.

Breaking the score down by question type — exploratory work, not pre-registered — puts the largest gap on the debugging questions and the smallest on code reading. Shen and Tamkin read that as a consequence of the process: both groups read a comparable amount of code, so the reading scores converge, but only the unassisted group spent the session meeting and resolving errors, so only they practised diagnosis. The asymmetry lands on the skill the paper began by singling out.

The participants felt the difference and misjudged it. The control group reported higher learning on a seven-point scale, and both groups reported enjoying the task. The assisted group rated the task itself easier, then rated the quiz exactly as hard as the control group did. One of them wrote afterwards, "I feel like I got lazy", and said he had not read the library's introduction as closely as he otherwise would.

Six ways of using the assistant

Shen and Tamkin annotated the screen recordings of fifty-one of the fifty-two participants by hand, marking every query typed, answer received, web search, paste, manual copy, error and completion. From those annotations they build a typology of six ways of working with the assistant, separated by what was asked, how often, and how much of the session went on coding rather than prompting. The groups are small, two to seven people each, but they divide cleanly.

The three low-scoring patterns all handed something over. Four participants practised what the authors call AI delegation: they asked only for code and pasted it in, finished fastest of anyone at 19.5 minutes against the control group's 23, and scored 39 per cent. Four more began with a question or two and then delegated the rest — progressive reliance — finishing in 22 minutes with 35 per cent, their losses concentrated in the second task's concepts, which they never engaged with. The last four used the assistant as a debugger, pasting errors and asking whether the code looked right, five to fifteen times; they were the slowest in the study at 31 minutes and scored 24 per cent, the lowest of all.

The three high-scoring patterns all asked for reasons. Seven participants, the largest group, asked the assistant only conceptual questions and resolved their own errors; they took 22 minutes, the fastest of the high scorers and second fastest overall, and scored 65 per cent. Three composed hybrid queries asking for code and an explanation of it together, spent 24 minutes reading what came back, and scored 68. Two generated the code first and then interrogated it: 24 minutes, 86 per cent, the best result in the study. That last pattern is the one to notice, because from the outside it is almost indistinguishable from delegation. The same request, the same paste; the difference is the questions asked afterwards.

The query counts show the same shape in aggregate. Explanations were the commonest request, seventy-nine of them, and twenty-one of the twenty-five assisted participants made at least one. Generation came second at fifty-one, but only sixteen asked for code at all and four asked for nothing else; three of the eight lowest scorers came from that group of four. Requests to debug were rare, nine in total, and they are the ones that predict: the higher a participant's fraction of debugging queries, the slower the completion (r = 0.43, p = 0.033) and the lower the quiz score (r = −0.41, p = 0.043). The sheer number of queries predicts neither.

Why did the assisted group not go faster? Because the time went to the assistant. Some spent up to eleven minutes of a thirty-five-minute session interacting with it, and one spent six minutes composing a single query against a median completion time of nineteen minutes; the recordings show people rereading the instructions and rewriting the question before sending it. Shen and Tamkin make the obvious point about the cost and then the less obvious one about the benefit: composing a question well requires understanding the task, so the activity that ate the speed gain is part of what preserved the learning. An interface that removes the need to compose — an agentic tool, or autocomplete — should therefore show a larger speed gain and a larger loss.

Typing the code out is not the remedy. Nine participants pasted the assistant's output directly and were the fastest; nine typed it by hand and four mixed the two, and both finished at roughly the pace of the control group — with no better quiz score for the typing. What separates the groups is not how the code entered the file but whether anyone asked what it did.

The mechanism the authors propose for the control group's advantage is error. Median errors per participant across both tasks: three without the assistant, one with it. Of the twelve people who got through both tasks without meeting a single error, ten were in the assisted group. The kind of error matters as much as the count. NameError and AttributeError, the commonest overall, are misspelled variables and function names, fixed in seconds and teaching nothing. RuntimeWarning, raised when a coroutine is never awaited, and TypeError, raised when a Trio function receives a coroutine object where an async function belongs, cannot be cleared without grasping how the library treats await — which is what the quiz asks about. The control group met many more of those.

What the experiment cannot settle

The authors list the limits, and several of them cut deep. The study is one hour, on one library, through one chat interface, while skill formation on a job happens over months. The participants were professional programmers but had no reason to want Trio, which removes the motivation a real onboarding would supply. Prompting skill was self-reported and never measured, so how well people asked cannot be separated from what they asked. Learning was measured by a quiz rather than by a second unaided task, a weaker test of transfer. There is no human-assistance arm, so the comparison is help from a model against no help at all, not against a colleague or a code review. The interaction clusters that carry the paper's most interesting result rest on two, three, four and seven people. And the pilots showed how fragile the control condition is: a quarter to a third of unwatched participants used AI regardless.

The authors are careful about the direction of the remaining uncertainty. A chat assistant that has to be prompted in sentences is the least automated case, so they treat their result as a lower bound, reasoning that an agentic tool asks still less of the person. That is an argument, not a measurement.

Since 2026

The paper appeared as a preprint in February 2026 with its pre-registration, its annotated recordings and its quiz released, so the design can be repeated on other libraries and other tools. The lower-bound claim about agentic assistants is the obvious next experiment and has not yet been run. On productivity, the null result sits with Becker and colleagues' 2025 finding that experienced open-source developers were slower with AI while believing themselves faster, and against the large field results from Copilot deployments; the difference probably lies in whether the task demands knowledge the worker does not yet have, the variable this experiment isolates. The three patterns that preserved learning here — asking for concepts, asking for explanation with the code, asking about the code once it arrives — are recognisably what the study and explanatory modes now shipped by the major assistants are built to encourage. Whether switching one on reproduces the effect is untested.