The Evaluating Self

81Language Models in a Programming Course

Gregor Jošt, Viktor Taneski and Sašo Karakatič, Applied Sciences 14(10), 4115, 2024

Linked to 13 chapters: supports 6 · contradicts 1 · mechanism 7 · same claim 1 · pairing 2

A React course, ten weeks, 2024 — the argument in five moves The gapin the studies The coursenine weeks The roomno model The splitin the uses What follows Reviews weigh thetool. None measureswhat the studentcan do after it. Four React tasks,any model, twohuman assistants,no encouragement. Week ten: two hours,nothing new, nomodel. That markis the outcome. Generating code anddebugging: negative.Asking for reasons:nothing detected. Teach them later,after writing andfixing code arein place. Thirty-two second-year students, four assignments, one two-hour test with no help.
Figure 81.0 — The paper's shape. A literature that weighs the tool, a course that measures the student, and a division inside the use that the average hides.

Jošt, Taneski and Karakatič ran a second-year React course at the University of Maribor and turned the last week of it into a measurement. For nine weeks thirty-two students built four applications in React and TypeScript, free to use any language model they wanted and never told to; two assistants with more than ten years of programming behind them were on hand as well. In week ten the students sat for two hours with Google and the React documentation and no model, watched to make sure of it, and wrote an application that asked nothing of them they had not already done. The mark on that application is what the paper measures against. On a five-point scale filled in after the marks were out, each student said how often they had used a model for three things: to generate code, to debug, and to ask for an explanation. Generating code correlates with the two-hour mark at Spearman's rho of −0.305, debugging at −0.360, and both bootstrap intervals sit entirely below zero. Asking for explanations, which students did nearly as often as they debugged, gives −0.201 and an interval straddling zero. The line does not fall between using a model and not using one. It falls between having it do the work and having it explain the work.

What the reviews could not settle

By early 2024 the literature on language models in education was already large and almost entirely appraisal. Fuchs weighed personalised learning and on-demand support against reduced human contact and bias. Lo read fifty articles on GPT-3.5 and came back with incorrect information and plagiarism on one side, course materials and active learning on the other. Grassini, Halaweh and Montenegro-Rueda and colleagues each arrived at the same shape of conclusion: promise, risk, and a recommendation that instructors be trained and assessments redesigned. On the tools themselves the work was sharper. Dakhel and colleagues found Copilot's output correct and efficient on basic algorithmic problems, better than a junior developer's, and dependent on how clearly the prompt was written — and noted that telling a flawed solution from a sound one still required an experienced developer. Vaithilingam, Zhang and Glassman found that Copilot did not reliably shorten task time or raise the success rate, though people preferred working with it.

None of that measures a student. The reviews ask whether the tool is good and whether the institution can cope; the Copilot studies ask whether the generated code is correct. What nobody had asked, on the authors' reading, is what a student who has used these tools through a course can do when the course is over and the tools are gone. Jošt and colleagues restrict their survey to models the public can reach without special hardware or machine-learning knowledge, which dates the whole question to the eighteen months since ChatGPT opened, and set out four predictions in advance: that heavier average use goes with lower final marks; that using a model to generate code goes with lower marks; that using one to debug goes with lower marks; and that using one for explanations does not. Each was tested one-sided, in the direction stated.

Nine weeks with the tools, one without

The participants were chosen for what they did and did not know. All thirty-two had a first year of HTML, CSS and plain JavaScript behind them, and none had been taught React, so the course would be their first encounter with component hierarchies, props and state. The nine weeks carried four assignments that built on each other: a basic component hierarchy first, then routing, then state management, then lifting state up, each set in a scenario of the kind a working developer would meet. The reference solutions, written by the course's own staff, run to 137, 461, 427 and 921 lines.

Through all four, use of a model was, in the authors' phrase, "unrestricted but not specifically encouraged". Students could use ChatGPT, Copilot, Bing Chat, Gemini or anything else, for whatever they found it useful for, and they also had two experienced assistants to ask. That combination is the point of the design: it reproduces the ordinary condition of a person learning to program in 2024, with a model in one window and a human being down the corridor, rather than an artificial one in which help is rationed.

How the two numbers are made Weeks one to nine — any model allowed, none required Week ten — supervised, docs only Task 1137 lines Task 2461 lines Task 3427 lines Task 4921 lines Track TV shows, no model 915 lines in two hours Questionnaire, after marks issued generation · explanation · debugging The mark on that application, 0 to 10 Spearman correlation with bootstrap intervals
Figure 81.1 — The design. Nine weeks of ordinary coursework produce one self-report; the tenth week produces one mark; the paper correlates them.

Week ten removed the model and kept everything else. Two hours, Google and the official React documentation allowed, the two assistants in the room watching that nobody opened a chat window. The task was an application for tracking television shows: a list on the home page with each show's name and whether it had been watched, rendered conditionally; details opening below the list when a show is clicked; a form for adding a show with a name and a description, defaulting to unwatched; props and lifted state to move the data between components. Nothing in it was new. The staff solution runs to 915 lines, about the size of the fourth assignment, and the mark on it is the paper's only outcome variable.

The measure of use came afterwards. Students filled in a questionnaire in Slovenian, after their marks had been issued so that nothing they admitted could cost them anything, rating three statements on a five-point scale: that they had used models for code generation, for additional explanation of coding problems, and for debugging. Participation was voluntary, and anyone who declined was left out of the analysis.

Three uses, three correlations

The reported quantities are worth reading before the correlations. Code generation was the least common use, at a mean of 2.59 with a median of 2.50 — students were not, on the whole, having the model write their assignments. Explanation stood at 3.75 and debugging at 3.78, both with a median of 4, and the average across the three uses at 3.38. The marks were spread wide: a mean of 6.72 against a median of 8.00, with a standard deviation of 3.10 and a range from 0.50 to 10. A mean more than a point below the median means a tail of students who could not build the application at all in two hours.

What students said they used the model for (one to five) Generating code Additional explanations Debugging Average of the three 2.59 · median 2.50 3.75 · median 4.00 3.78 · median 4.00 3.38 · median 3.67 1 2 3 4 5 reported frequency of use Rust: the two uses that go with a lower mark. Teal: the one that does not.
Figure 81.2 — Reported use on the five-point scale, from Table 1. The least-used function and the most-used one are the two that correlate with lower marks.

Shapiro–Wilk rejected normality for every variable, so the analysis is Spearman's rank correlation, one-sided, with bootstrap confidence intervals to show how far the estimate might move under resampling from a set of thirty-two.

Use of the modelSpearman's rho95% bootstrap intervalp
Generating code−0.305−0.595 to −0.0580.045
Additional explanations−0.201−0.523 to 0.2200.135
Debugging−0.360−0.628 to −0.0110.021
Average across the three−0.347−0.626 to −0.0440.026

Three of the four intervals lie below zero. Debugging gives the strongest negative correlation in the study and is also the use students reported most; code generation, reported least, still correlates negatively; and the average of the three carries a negative correlation of its own, so that a student's overall reliance predicts the tenth-week mark as well as any single use of it does.

Each use against the mark on the unaided task Generating code · p = 0.045 Additional explanations · p = 0.135 Debugging · p = 0.021 Average of the three · p = 0.026 −0.305 −0.201 −0.360 −0.347 −0.6 −0.4 −0.2 0 0.2 Spearman's rho, with the 95 per cent bootstrap interval Only the interval for explanations reaches across zero, and it reaches a long way.
Figure 81.3 — Spearman's rho with 95 per cent bootstrap intervals, from Table 2. Only the interval for explanations crosses zero.

The explanation result should be read for what it is. The authors predicted no significant effect and found none, and they are right that the finding separates explanation from the other two uses. But a p of 0.135 on thirty-two students is a failure to detect rather than a demonstration of safety, and the interval runs from −0.523 to +0.220 — wide enough to contain a harm larger than the one measured for code generation. What the data establish is a difference in order between the three uses, not a clean bill for one of them.

Generation and debugging against explanation

The authors read the split by what each use takes away from the student. Generating code and finding the fault in code are the two activities in a programming assignment that demand a model of how the system works; hand either to a machine and the assignment can be completed without ever forming that model. Asking what a hook does, or why an effect runs twice, leaves the student to write and fix the code afterwards.

Three uses, and what each leaves the student to do Ask for the code Ask what is wrong Ask why it works The model writes it. The code runs. The model names the fault. It runs. The model explains. The code stays yours. No model of the code is ever built. The search for the fault is skipped. Writing and fixing are still done by hand. Week ten asks for the two things the first two columns removed.
Figure 81.4 — The mechanism the authors propose. Two of the three uses remove the step that the tenth week puts back.

Debugging is where the paper puts its weight, and it borrows the case from elsewhere. Michaeli and Romeike treat systematic search for a fault as a core competence of professional developers, hard for novices and a common source of frustration, which is exactly why it has to be practised rather than avoided. Carver and Risinger, writing in 1987, found that teaching debugging transfers to problems outside programming altogether. Pudari and Ernst note that models help with syntax and error messages while code smells, idiom and design remain out of reach, so the part of the work a model covers is the part that teaches least. Haque and Li, on ChatGPT as a debugger, reach the same practical conclusion: useful as a starting point, not a substitute for human review and testing.

Explanation is the exception, and the authors are careful about how much they claim for it. It may serve as a supplement rather than, in their words, "a crutch that impedes skill development" — a sentence with two hedges in it, and they add that further work is needed before the role of explanation is settled. Their recommendation to instructors follows from the ordering rather than from any one correlation: bring the tools in later, after the basic principles of writing and fixing code are in place, so that students get the productivity of the model without losing the competence that lets them judge its output. They do not argue for a ban. They argue about sequence.

Since 2024

The limits are the ones a ten-week course imposes. Thirty-two students in one place, one language and one library; ten weeks against a skill that forms over years; and a measure of use that is what a student remembers and is willing to write down weeks later on a five-point scale. There is no control group and no random assignment, so the correlations cannot carry a direction on their own — and the authors name the alternative reading themselves, in the form of a question they did not answer: whether the students who used models most were the ones already struggling, reaching for help because they were behind rather than falling behind because they reached. Until someone checks prior marks against model use, that account fits the data as well as theirs does. The raw data are available from the authors on request.

Two years later Shen and Tamkin put the same question to an experiment rather than a course: fifty-two professional programmers, a library none of them knew, random assignment to an assistant or to documentation, and a quiz at the end taken by everyone without help. The assisted group scored lower, and when the screen recordings were sorted by what people had asked for, the ranking matched Maribor's — the participants who asked the model to write the code or to find the fault scored under forty per cent, and those who asked it why scored sixty-five and above. Neither study can stand alone. The course has ten real weeks and no randomisation; the experiment has randomisation and one hour. They fail in different directions and agree about where the line falls.

What has changed underneath both is the interface. The models in this study were chat windows a student had to type a question into, and typing the question well requires understanding the problem well enough to ask. The editors that have since become ordinary complete the line before the question is formed. On the authors' own account of why generation and debugging cost what they cost, that change should move the correlation, not soften it.

Claims drawn from this chapter · 6

evidenceThree uses, three correlations

A student's overall model use predicted the unassisted exam mark

Averaging a student's reported use of a language model across generating code, debugging and asking for explanations predicts the two-hour unassisted mark about as well as any single use does — a rank correlation of −0.347, interval −0.626 to −0.044, p 0.026.

Becausethe composite is the fourth row of the paper's correlation table, and it sits between the two significant single uses rather than above or below them, so overall reliance carries the signal on its own.
LimitsThe average is built from three items on a five-point scale, two of which correlate negatively on their own, so it is not independent evidence. Thirty-two students, self-report, no randomisation.
PracticeReliance audit — a crude count of how much you lean on the tool is already diagnostic, so audit the overall level before arguing about which kind of request is acceptable.
After AIThe paper offers it as a summary statistic; read against the split it reports, it is the weaker finding of the two. Reliance predicts, but what you ask for predicts better, and only the second is a thing you can change on Monday.
opinionNine weeks with the tools, one without

Skill left by a course shows only in a task done unassisted

Nine weeks of assisted coursework tell you nothing about what a student can do, so Jošt and colleagues made their only outcome a two-hour invigilated application, on material already covered, with the models removed and the documentation left in.

Becauseit is a choice of design rather than a result, but the marks it produced were spread from 0.50 to 10 with a mean of 6.72 against a median of 8.00, a tail no assignment grade had shown.
LimitsOne two-hour task is a narrow measure of a skill that forms over years, and taking the tool away measures work under a condition nobody will meet again. A mark earned without a model may understate what a person can do with one.
PracticeManual flying hours — put one piece of real work a week through your own hands with the tools off, and treat that, not the shipped output, as the reading of what you still have.
After AIThe productivity literature measures what was shipped while the tool was in hand; this measures what is left in the person when it is gone. For a team, the equivalent reading is not a metric anyone currently collects.
evidenceThree uses, three correlations

Students who had a model write or fix their code scored lower

In a ten-week React course at Maribor, how often a student used a language model to generate code and to debug both went with a lower mark on an application written two hours later without one — rank correlations of −0.305 and −0.360, with both 95 per cent intervals below zero.

Becausethirty-two second-year students, nine weeks of unrestricted model use across four assignments, then one invigilated two-hour task with only Google and the React documentation; p 0.045 for generation and 0.021 for debugging.
LimitsThirty-two students, one library, ten weeks, and use measured by what each student remembered weeks later on a five-point scale. No control group and no randomisation, so the correlation carries no direction on its own.
PracticeDreyfus mapping and tutor-only rule — on material you are still learning, let the model explain and write and debug the code yourself; the two things it is most useful for are the two that cost most.
After AIWritten about students, it describes any adult meeting an unfamiliar stack. The two uses that cost most are the two an agentic editor now performs unasked.
opinionGeneration and debugging against explanation

Teach students to write and fix code before giving them a model

Jošt and colleagues recommend introducing language models later in a curriculum, once a student can write and debug code unaided, so that the productivity arrives without costing the competence that lets the student judge the output. They argue about sequence and do not propose a ban.

Becauseit is the instructional conclusion the authors draw from the ordering across the three uses rather than from any single correlation, and the timing they propose was never itself tested.
LimitsNo arm of the study delayed the tools, so the recommendation is an inference from a correlation on thirty-two students. It also assumes an instructor can hold a line nobody can see being crossed at home.
PracticeDreyfus mapping and tutor-only rule — set the allowed kind of request by the person's stage in that domain rather than by policy for everyone, and move the line as they pass competence.
After AIIn a workplace nobody schedules the stages, so the same rule has to be set by the individual against work they have chosen. The person deciding when they are ready is the one who benefits from saying they already are.
contestedThree uses, three correlations

The explanation result was a failure to detect, not a clean bill

The one use of a model that showed no significant cost in Jošt's course — asking it to explain a coding problem — returned a rank correlation of −0.201 on thirty-two students, with p at 0.135 and an interval running from −0.523 to +0.220. That interval contains a harm larger than the one measured for generating code.

Becausethe authors predicted no effect and found none, which on a small sample is not evidence of safety; they say themselves that further work is needed before the role of explanation is settled.
LimitsWhat the data establish is a difference in order between the three uses, not a verdict on any one of them. A larger study could put the explanation correlation anywhere in that interval, including below the generation result.
PracticeReliance audit — treat "I only ask it to explain" as an untested exemption rather than a safe one, and check what you can still do unaided in the areas where you ask most.
After AIThe tutor mode is the safe harbour everyone now claims, in courses and in the assistants' own study modes. Its evidence is one underpowered null and one small experiment, which is thinner than the confidence placed on it.
contestedSince 2024

The students who used models most may have been struggling already

Jošt's correlations have no control group and no random assignment, so the reverse reading — that the students who reached for a model most were the ones already behind — fits the data as well as the authors' reading does. Nobody checked prior marks against model use.

Becausethe authors raise the alternative themselves and leave it unanswered, and the data they collected cannot separate the two accounts; the first-year marks that would settle it were never brought in.
LimitsThe randomised study that does control for prior ability finds the same split, which makes the reverse reading less likely without removing it here. A correlation surviving replication under randomisation is not thereby causal in this sample.
PracticeDomain audit — before concluding that a tool is damaging a team, check what the heaviest users were doing before it arrived; help-seeking and falling behind look identical in a usage log.
After AIEvery internal dashboard now correlates model use against some output measure, and almost all of them have this problem. The people who use the tool most are rarely a random sample of the people who work there.

All claims · Source map