The Evaluating Self

48The Jagged Technological Frontier

Fabrizio Dell'Acqua, Edward McFowland III, Ethan Mollick, Hila Lifshitz, Katherine C. Kellogg, Saran Rajendran, Lisa Krayer, François Candelon and Karim R. Lakhani, Organization Science, 2026 (Harvard Business School working paper 24-013, 2023)

Linked to 22 chapters: supports 12 · contradicts 6 · mechanism 9 · same claim 3 · pairing 4

AI in consulting work — the argument in five moves The frontier The design Inside it Outside it Who gained AI helps on sometasks and harms onothers of equalapparent difficulty. 758 consultants, abaseline task, thenGPT-4 or no GPT-4by lot. 18 tasks: 12% moredone, a quarterfaster, quality upabout 30%. One task: correct84% alone, 60-71%with the model. The weaker half ofthe sample gainedmost; the spreadnarrowed. Tasks on both sides of the boundary looked equally like ordinary consulting work.
Figure 48.0 — The paper in five moves: a concept, an experiment inside one firm, a large gain on one set of tasks, a large loss on another, and a redistribution of who is good at the work.

In 2023 Boston Consulting Group put 758 of its consultants, about seven per cent of the individual contributors in its consulting workforce, through a set of tasks built to resemble their ordinary assignments, and randomised who was allowed to use GPT-4 on them. Half the sample worked on a product-launch assignment of eighteen subtasks: invent shoes for an underserved market, segment the footwear industry, write the press release, write the memo to staff. The consultants with the model completed 12.2 per cent more of those subtasks, reached the last question about a quarter faster, and were graded roughly thirty per cent higher on quality. The other half worked on a single business problem built so that the spreadsheet in front of them pointed one way and a detail in the interview notes pointed the other. Consultants without the model reached the right recommendation 84 per cent of the time; consultants with it managed between 60 and 71 per cent. Dell'Acqua, McFowland, Mollick, Lifshitz, Kellogg, Rajendran, Krayer, Candelon and Lakhani name the boundary between those two results a jagged technological frontier, and argue that its most awkward property is not its shape but its invisibility from the inside.

A frontier nobody can see

The term describes an unevenness rather than a limit. Tasks that look equally hard to a trained professional fall on opposite sides of a line where machine assistance stops helping and starts hurting, and the line does not run where intuition about difficulty says it should. Generating a hundred product ideas sits inside it; arithmetic, at the time of writing, sat outside.

Three properties of large language models make the position of any given task hard to know in advance. The models arrive with abilities nobody designed into them and nobody can enumerate, and the set grows with each release. That general competence converts directly into the performance of the people using it, which is what makes the tools worth adopting and the question urgent. And the opacity is of an unfamiliar kind: earlier systems were black boxes that frustrated their users, while these produce wrong answers that are fluent, confident and pleasant to read, with no instructions and no published account of how to use them well.

Put together, the three mean that a professional inside a single piece of work will cross the boundary without noticing the crossing. The paper's contribution is to put tasks from both sides of the boundary in front of one population, under one set of incentives, inside one firm, and to measure what happens on each side.

Eighteen tasks inside the frontier

The setting was chosen for difficulty, not convenience. Boston Consulting Group admits roughly one per cent of applicants, promotes largely from within, and decides promotions on appraisals that turn on exactly the range of work the experiment tested. Senior partners who had held the participants' jobs designed the tasks; a senior managing director described them as "very much in line with part of the daily activities of the consultants" taking part, and a survey of eleven managing directors and partners confirmed that the competencies measured were the ones the firm recruits and promotes on.

The design: one baseline, one coin, two kinds of task Inside thefrontier, 385 Outside thefrontier, 373 Beverage launchplan, no model Randomise 18 footwear tasks: ideas,segments, copy, memo Channel analysis,no model Randomise One brand recommendationfrom data and interviews Each draw sent a third of the arm to no model, a third to GPT-4, and a third to GPT-4 preceded by a prompt-engineering overview. Every measure is read against the baseline.
Figure 48.1 — The protocol. Every participant first worked without the model, so each person's later performance could be read against their own baseline.

Each participant began with an assessment task performed without assistance, which fixed a baseline for that individual, and was then randomised into one of three conditions: no model, GPT-4, or GPT-4 preceded by a prompt-engineering overview of videos and documents. The model was reached through a platform BCG built on OpenAI's programming interface, which behaved like ChatGPT and logged every prompt and reply; everyone used GPT-4 as it stood at the end of April 2023, with default settings. Incentives followed the firm's own: honest effort earned an office-contribution credit that feeds into the annual bonus, the bottom five per cent forfeited it, and the top twenty per cent were named to their supervisors. Of those assigned a model, 97.6 per cent used it at least once.

The inside-the-frontier arm, 385 people, opened with a beverage-launch exercise and moved to footwear: conceive a shoe for a neglected market or sport and carry it from prototype description through segmentation to launch. An executive at a global footwear company confirmed that the eighteen subtasks covered the process their own firm runs, with one step missing. They divide into creativity, analytical thinking, writing and persuasion. Ten graders drawn from an MBA programme scored each answer against simple rubrics, three per answer, on a one-to-ten scale; GPT-4 scored the same answers as a check.

Change from the control group, inside the frontier GPT-4 only GPT-4 with prompt overview +30% 0 −30% +29.9% +33.9% +10.9% +13.5% −27.6% −22.5% Quality graded 1–10 Subtasks completed Time to last question
Figure 48.2 — Change from the control group on the three inside-the-frontier outcomes, from Tables 4, 5 and 6 of the paper.

Every measure moved and all of them moved the same way. Quality rose 33.9 per cent for the group with the overview and 29.9 per cent for the group without it, over a control mean of 4.38 out of ten. Completion rose from 82 per cent of the eighteen subtasks to 93 and 91 per cent. Time to reach the final question fell by 18.8 and 23.1 minutes against a control mean of 5,023 seconds, which is 22.5 and 27.6 per cent faster. The gains held when the four task categories were analysed separately, and adding controls for spelling and grammar errors left the coefficients almost unchanged, so the effect is not graders rewarding tidier prose. The overview mattered: the difference between the two treated groups was significant for quality, though not for completion.

The distribution of the gains is the result that has travelled furthest. Splitting the treated consultants at the median of their own baseline, the bottom half improved more than the top half, though the top half improved substantially too: the tool compressed the distance between the best and the worst. The same pattern appears in Brynjolfsson, Li and Raymond's customer-support agents and in Choi and Schwarcz's law students, where the weakest gained most and the strongest slipped, and not in Otis and colleagues' Kenyan entrepreneurs, where the high performers benefited and the low performers did worse by following the advice. Low skill here is relative: everyone in the study would count as highly skilled almost anywhere else.

The September 2023 working paper reported a second distributional effect that the published version does not carry: the ideas produced with the model resembled one another more closely than the ideas produced without it, so the average rose while the spread of the sample narrowed. The published version also drops the working paper's typology of how people actually worked with the tool — the centaur who splits a task cleanly between human and machine and the cyborg who interleaves them sentence by sentence — and refers it instead to a separate qualitative paper by Randazzo and colleagues, which adds a third mode, the self-automator, who hands the work over entire.

Against the charge that MBA graders were charmed by machine-written prose, the published version adds two replications by professional judges. Five students of product and industrial design with footwear experience graded the shoe ideas from the first question for creativity alone, blind to the study's purpose, producing 15,404 gradings; the treated groups scored 14 and 11.7 per cent above control, smaller than the headline but in the same direction. Three of them then graded all eighteen questions, two experts per answer, and produced effects larger than the original: 1.75 and 1.61 points on a control mean of 4.50.

The task built to defeat it

The second arm, 373 people, was designed backwards from the finding the authors wanted to test. They took the kind of business case BCG uses in its own interviews and adapted it so that the obvious route to an answer was wrong. Participants received a spreadsheet of financial data and a file of interviews with company insiders. The spreadsheet was built to look complete. The interviews contained details that, read carefully, overturned what the numbers suggested. A model handed the instructions, the data and the interviews returned a confident recommendation that was incorrect.

Why the second task fell outside the frontier Financial data,brand by brand Interview notesfrom insiders Prompted withboth files A fluent recommendationthe spreadsheet supports The right answer, reachedonly by reading closely The spreadsheet was built to look complete; the detail that overturns it sits in the interviews.
Figure 48.3 — Why the second task fell outside the frontier. The model was given everything the consultant was given, and answered from the half of it that looked sufficient.

The assessment task asked which distribution channel held the most growth potential; the experimental task asked the same question of the company's brands, with a recommendation to the chief executive and actions to improve the chosen brand. Correctness was scored as a binary by BCG consultants; GPT-4, grading the same answers, agreed with them in more than 97 per cent of cases.

The one task outside the frontier Correct recommendation (per cent) Coherence of argument, 1–10 84.4 70.6 60.0 5.86 6.90 7.33 Control,no model GPT-4 GPT-4 +overview Control,no model GPT-4 GPT-4 +overview
Figure 48.4 — Correctness and graded coherence on the outside-the-frontier task, from Table 2 and Figure 5 of the paper. The same consultants argued better and concluded worse.

Correctness fell from 84.4 per cent in the control group to 70.6 per cent with GPT-4 and 60.0 per cent with GPT-4 and the overview: 19 percentage points on average across the two treated conditions. The group that had been taught to prompt well did worst, and the gap between the two treated conditions was significant at the ten per cent level in every specification. They were also faster, by more than eleven minutes with the overview and more than six without it, against a control mean of 2,260 seconds: 30 and 18 per cent less time on a task they were getting wrong.

A third measure makes the failure harder to dismiss. Two independent sets of graders — BCG consultants not otherwise involved, and business school students with grading experience — scored every answer for clarity, persuasiveness and coherence of reasoning, without being told what the correct recommendation was. The treated groups scored 1.47 and 1.05 points higher than control on a ten-point scale, and the advantage held among those who got the answer right and among those who got it wrong. Assistance improved the presentation of a conclusion independently of whether the conclusion was true.

The authors' reading is one Dell'Acqua had already named in a 2022 experiment with recruiters: "falling asleep at the wheel". Nothing in the second task announced that it lay on the far side of a boundary the eighteen sat inside. The firm was the same, the work looked the same, and the model answered as fluently as it had on the tasks that rewarded trusting it. Participants did not need its output to complete the task and used it anyway. The paper connects this to the older automation-bias literature — Skitka and colleagues in 1999, Buçinca and colleagues in 2021 — and notes what is new: here the degradation appears not because the system is visibly unreliable but because it performs so well on the tasks next door.

What the experiment cannot settle

The largest limitation is arithmetical. There is one task outside the frontier and eighteen inside it, so the 19-point drop rests on a single problem, chosen by the authors to produce exactly that drop, as the paper says. What the design demonstrates is that such a task exists inside ordinary consulting work and that trained people walk into it; it cannot say how often, or what fraction of a real workflow sits on which side.

The measures carry their own softness. Quality inside the frontier is a rubric score, and the authors concede that subjective metrics may carry evaluation bias, which is what the expert replications were added to address. The preregistration fixed the design, the conditions and the outcome variables, but not the jagged-frontier framing or the specific tasks, both of which arrived later.

The incentives bind the result to a particular workplace. At BCG, once work is delivered on time, the hours it took do not enter the appraisal, so speed gains are close to free and quality is what counts; where accuracy trades against speed by design — call centres timed to the second, piecework, manufacturing — the same tool could produce a different sign. The participants were also early in their careers, and the authors say plainly that more experienced managers might behave otherwise.

Two further caveats concern who uses the tool rather than how well it works. Assistance here was assigned. Where adoption is voluntary, early adopters are disproportionately male and technologically confident, so the technology that compressed the skill distribution here could widen the gaps that decide promotions. And performance assumes the job survives: evidence from online labour markets shows generative tools cutting both employment and earnings for freelancers in exposed occupations.

Since 2023

The concept outran the evidence. Within two years the jagged frontier had become the standard way of describing what these systems can and cannot do, endorsed by the people who build them: Sundar Pichai offered artificial jagged intelligence as a name for the present phase, and Andrej Karpathy advised practitioners to watch for jagged edges and keep a person in the loop. Ethan Mollick, one of the authors, built a trade book around the image, which is how most readers met the experiment.

The frontier itself moved. GPT-4 as it stood in April 2023 was superseded within a year, and models trained to reason step by step, arriving from late 2024, pulled arithmetic and multi-step logic inside the wall without making it any straighter or any easier to see. The paper anticipates this: the frontier is jagged but not fixed. What has not been produced is a map. Its own recommendation — that organisations evaluate each task in a workflow for the configuration of people and machines that suits it — has no method behind it, and the concept remains a diagnosis applied after the fact.

The two halves of the result have aged differently. The productivity half has been replicated widely and also contradicted: a 2025 trial of experienced open-source developers found them 19 per cent slower on issues where assistance was allowed while believing themselves faster, which is the outside-the-frontier effect appearing in people working inside their own long-held expertise. Vaccaro, Almaatouq and Malone's meta-analysis of 106 experiments, cited in the discussion here, found combinations losing to the better partner on decision tasks and gaining on creation tasks — the same split, across a literature rather than within one workflow. The levelling result has replicated in some settings and failed in others, and looks weakest wherever a task turns on judgment rather than fluency.

The published version, which reached Organization Science in March 2026, is more cautious than the working paper it grew from. It adds the expert replications, non-parametric tests with a Bonferroni correction and standardised effect sizes, an analysis placing management consulting among other knowledge occupations, and the survey of partners. It adds the boundary conditions on incentives, on who adopts and on substitution. And it removes what was most quoted: the compression of the sample's ideas towards one another, and the centaurs and cyborgs. The finding that made the paper famous is the one it now states most carefully, and the observations that gave it its vocabulary have been moved out to be tested elsewhere.

Claims drawn from this chapter · 7

contestedEighteen tasks inside the frontier; Since 2023

A group using AI produces better ideas that are more alike

Dell'Acqua and colleagues reported that when a group uses an assistant the average quality of its ideas goes up and the ideas come to resemble one another more.

Becauseit was reported in the September 2023 working paper and dropped from the published Organization Science version, which refers the typology and the homogenisation finding out to separate work
LimitsThe finding no longer appears in the published paper, which is the strongest available caution about it. It was measured on shoe ideas produced in one session, and the authors have moved it out to be tested elsewhere rather than defended it.
PracticeThe stays-human list — where the value of a group's output is its variety, generate independently before anyone consults a model.
After AIIt is also the most quoted result of the experiment, repeated in places that do not know it was withdrawn; the honest position is that it is an open question with one suggestive measurement behind it.
evidenceThe task built to defeat it

AI makes a case better argued without making it more correct

Dell'Acqua and colleagues found that consultants using the AI wrote clearer, more persuasive, better-organised arguments whether or not their recommendation was the right one. Graders who did not know the correct answer scored the assisted groups higher either way.

Becausetwo independent grader pools, blind to the correct recommendation, scored the treated groups 1.05 and 1.47 points higher on clarity, persuasiveness and coherence, and the advantage held among those who got the answer right and those who got it wrong
LimitsGraded coherence is a rubric judgment by students and consultants, on one task. Whether better argument makes a wrong conclusion more likely to be accepted downstream was not tested here.
PracticeFailure-mode review — review a recommendation against the evidence it omits, not against how well it reads; in this experiment the best-argued answers were as often the wrong ones.
After AIThis is the clearest measured statement of the cost of fluency: the same system that raises the quality of the case raises it equally for the case that should not be made.
evidenceEighteen tasks inside the frontier

Consultants using AI on suitable tasks did better, faster and more completely

On tasks that sit inside what current AI does well, using it improves quality, the share of the work finished, and speed all at once, rather than trading one against another.

Because385 consultants on an eighteen-subtask exercise scored 29.9% and 33.9% higher on quality over a control mean of 4.38/10, completed 91% and 93% of subtasks against 82%, and finished 22.5% and 27.6% faster, with the effects holding across all four task categories and surviving controls for spelling and grammar plus two replications by professional design judges
LimitsQuality is a rubric score, and the authors concede subjective metrics may carry evaluation bias. BCG's incentives make speed nearly free once work is delivered on time, so the same tool could produce a different sign where accuracy trades against speed. Participants were early in their careers.
PracticeTake the gain where it is real: on generative, well-bounded work inside a frontier you have checked, the improvement is large enough that refusing the tool is a cost.
After AIGPT-4 as of April 2023 was superseded within a year, and reasoning models pulled arithmetic inside the wall without making it straighter; the size of the gain has moved and the shape of the result has not.
evidenceThe task built to defeat it

Just beyond AI's range the same tool made consultants wrong more often

On a case built to fall just outside what the model handles well, 373 consultants got it right 84.4% of the time without the model, 70.6% with it, and 60.0% with it plus prompt training.

Because373 consultants on a case where the spreadsheet pointed one way and the interview notes the other fell from 84.4% correct in control to 70.6% with the model and 60.0% with the model plus prompt training — about 19 points — while finishing 18% and 30% faster
LimitsOne task outside the frontier against eighteen inside it, and the outside task was built by the authors to produce exactly that drop. The design shows such a task exists inside ordinary professional work and that trained people walk into it; it cannot say how often.
PracticeCognitive forcing — on a judgment where the obvious data and the awkward detail disagree, form and record your own recommendation before you open the model.
After AIThe degradation appeared not because the system was visibly unreliable but because it performed so well on the tasks next door, which is the condition every general assistant now creates by default.
opinionA frontier nobody can see; The task built to defeat it

Nothing warns you when a task falls outside what AI handles

The one task the model made consultants worse at felt no different from the eighteen it helped with. Nothing about it announced that it sat on the far side of a line.

Becauseit is an interpretation of the contrast rather than a measurement — same firm, same population, same incentives, same fluent output, and participants who did not need the model used it anyway, which Dell'Acqua calls falling asleep at the wheel
LimitsInvisibility was not measured — nobody asked participants whether the task felt different. The concept has since become a diagnosis applied after the fact, and the paper's own recommendation that organisations evaluate each task has no method behind it.
PracticeDomain audit — decide in advance, per class of task, which side of the line it sits on, and write the decision down; you will not detect the crossing while working.
After AIThe frontier moves with each release, so a map made this quarter expires; what survives is the warning that fluency is uninformative about which side of the line you are on.
contestedThe task built to defeat it

Teaching consultants to prompt made them worse on the harder task

On a task built to defeat the AI, consultants given an overview of prompt writing scored 60.0 per cent correct, against 70.6 per cent for those who used the AI without that overview.

Becausethe group given a prompt-engineering overview scored 60.0% correct against 70.6% for the treated group without it, significant only at the 10% level, while the same overview improved quality inside the frontier
LimitsTen per cent significance on one task in one experiment; the direction is consistent across specifications but the effect is a single contrast. The mechanism — that fluency in prompting raises trust without raising discrimination — is inferred.
PracticeSpecification discipline — treat training in how to use the tool as raising reliance, not judgment, and pair it with an explicit rule about where not to use it.
After AIPrompt training is now standard corporate onboarding for these tools, and it is the one intervention this experiment measured as harmful in the case that mattered.
contestedEighteen tasks inside the frontier

Weaker performers gain more from AI than stronger ones do

Split at the middle of their own earlier scores, the consultants who had performed worst improved more with the tool than the best did, narrowing the gap between them.

Becausesplit at the median of their own pre-treatment baseline the bottom half improved more than the top half, matching Brynjolfsson's support agents and Choi and Schwarcz's law students — and reversing in Otis and colleagues' Kenyan entrepreneurs, where high performers gained and low performers did worse
LimitsLow skill here is relative — everyone in the study would count as highly skilled almost anywhere else. The result has replicated in some settings and failed in others, and looks weakest wherever a task turns on judgment rather than fluency.
PracticeDreyfus mapping and tutor-only rule — do not read a levelling result as permission for a novice to skip the practice; ask whether the task turns on fluency or on judgment.
After AIWhere adoption is voluntary rather than assigned, early adopters are disproportionately male and technologically confident, so the same technology that compressed the distribution here could widen the gaps that decide promotions.

All claims · Source map