33The Mythical Man-Month

Frederick P. Brooks, Jr., The Mythical Man-Month: Essays on Software Engineering, Addison-Wesley, 1975; anniversary edition with four new chapters, 1995

Linked to 22 chapters: supports 16 · contradicts 5 · mechanism 5 · same claim 4 · pairing 7

Brooks was an architect of the IBM System/360 computer family and then, from 1964, the manager of the Operating System/360 that ran on it. OS/360 shipped late, cost several times its estimate, took more memory than planned and did not perform well until several releases in; the flaws in its control program dated from the design period of 1964 and 1965, and Brooks laid them to his own charge. This book is his account of why. Its argument, carried by chapters two through seven, is that a large programming project suffers management problems different in kind from a small one, and that the difference comes from the division of labour, from the sheer number of minds involved. What division of labour destroys is conceptual integrity, the property of a system that presents its user one coherent set of ideas, and conceptual integrity, Brooks says, is the most important consideration in system design. The rest follows. Because men and months are not interchangeable, adding people to a late project delays it further. Because a product must be conceived by one mind and built by many hands, architecture has to be separated from implementation, and implementation organised so that few minds design while many hands raise their effectiveness. The remaining essays are the working parts: documents, milestones, estimates, tools, testing. Twenty years later Brooks added four chapters in which he defended one prediction, withdrew two recommendations and conceded a long argument to David Parnas.

The tar pit

The book opens on a mural of the La Brea tar pits, with mammoths and sabre-toothed tigers struggling in the asphalt: the fiercer the struggle, the more entangling the tar, and no beast is strong enough not to sink. Large-system programming, Brooks writes, has been such a pit. No single thing causes the difficulty, and any particular paw can be pulled free; it is the accumulation of simultaneous and interacting factors that brings slower and slower motion.

The first thing to get straight is what is being produced. A program, complete in itself and run by its author on the machine it was written on, is what comes out of the garage, and it is what a programmer uses when estimating his own productivity. Two boundaries can be crossed from there. A program becomes a programming product when its inputs are generalised, when it has been tested against a bank of cases that probe the boundaries of that range, and when it is documented so that anyone can use, fix and extend it; that costs three times as much as the debugged program. A program becomes a component in a programming system when its every input and output conforms to precisely defined interfaces, when it lives within a budget of memory, devices and time, and when it has been tested with the other components in all expected combinations; that also costs three times as much. The programming systems product in the far corner costs nine times the program, and it is what most large efforts are trying to build. The factor of nine is the gap between what a programmer believes he can do and what his employer needs.

Against the tar he sets the joys of the craft, among them the delight of working in a medium so tractable that one builds castles in the air, from air. The woes are as particular. One must perform perfectly, and adjusting to that requirement is the hardest part of learning to program. Others set one's objectives and supply one's resources, so authority never equals responsibility; actual authority comes instead from the momentum of accomplishment. Debugging converges linearly or worse when one expects it to converge faster near the end. And the product looks obsolete on completion, though the real tiger is never a match for the paper one.

The mythical man-month

More software projects have gone awry for lack of calendar time, Brooks writes, than for all other causes combined, and he gives five reasons. Estimating techniques are poorly developed, carry the unvoiced assumption that all will go well, and confuse effort with progress. Managers, uncertain of their estimates, lack the courteous stubbornness of the chef at Antoine's, whose menu explains that good cooking takes time. Schedule progress is poorly monitored. And when slippage is at last recognised, the traditional response is to add people, which is like dousing a fire with gasoline.

Optimism comes first because it is structural rather than temperamental. In most crafts the medium is intractable — lumber splits, paint smears, circuits ring — and the difficulty of implementation gets blamed on the medium. Programming's medium is pure thought-stuff, so implementation is expected to be easy; but the ideas themselves are faulty, so there are bugs, and the optimism is unjustified. For one task there is a finite probability of no delay at all; for a large effort made of many tasks chained end to end, the probability that each goes well becomes vanishingly small.

The unit of effort carries the second error inside it. Cost does vary as the product of men and months; progress does not. Men and months trade evenly only when a task can be partitioned with no communication among the workers, as in reaping wheat. When sequential constraints forbid partition, more effort has no effect at all: the bearing of a child takes nine months no matter how many women are assigned, and much of debugging has that character. When a task can be partitioned but its parts must be coordinated, the communication effort has to be added to the work. Training each new worker cannot be partitioned, so it grows linearly with the number of workers; intercommunication grows as n(n−1)/2, so that three workers need three times as much pairwise communication as two, and four need six times as much. Software construction is an exercise in complex interrelationships, so communication effort is large and soon swallows the gain from partitioning.

Testing is where the arithmetic collects. Brooks's rule of thumb allots one third of the schedule to planning, one sixth to coding, one quarter to component test and early system test, and one quarter to system test with all components in hand. Coding, the one part that is easy to estimate, gets a sixth; the half given to debugging completed code is much larger than usual. Examining conventionally scheduled projects, he found that few allowed half the schedule for testing but most spent half the actual schedule on it, running on time until system test. Because the delay falls at the end, nobody knows there is trouble until nearly the delivery date, when the project is fully staffed and cost per day is at its maximum.

The remedy usually applied makes things worse, and Brooks works the example. A task estimated at twelve man-months goes to three people for four months, with milestones at each month's end; the first is reached only after two months. Nine man-months and two months remain, so four and a half people are needed and two are added. Those two must be trained by one of the three, which costs perhaps three man-months that were never in the estimate, and the task, partitioned three ways, must be repartitioned into five, so some finished work is lost and system testing lengthens. At the end of the third month more than seven man-months remain, five trained people are available and one month is left. The product is just as late as if nobody had been added, and the temptation is to repeat the cycle. Brooks states the conclusion as a law, deliberately simplified:

Adding manpower to a late software project makes it later.

What he endorses instead is rescheduling, taking no small slips so that the new date will not have to be revised again, or trimming the task formally rather than watching it be trimmed silently by hasty design and incomplete testing. The months a project needs are set by its sequential constraints; the maximum number of people is set by the number of independent subtasks. From those two one can build schedules with fewer people and more months, but never a workable schedule with more people and fewer months.

The surgical team

Everyone at a computing conference says they would rather have a small sharp team than hundreds of mediocre programmers, and the preference evades the question of how large systems get built at all. The size of the individual differences is not in doubt: Sackman, Erikson and Grant, measuring a group of experienced programmers, found ratios between best and worst performance averaging about ten to one on productivity and five to one on the speed and space of the resulting programs, with no correlation whatever between experience and performance.

The arithmetic then defeats the preference. At its peak more than a thousand people worked on OS/360, and from 1963 through 1966 some five thousand man-years went into it. A team of two hundred would have needed twenty-five years if men and months traded evenly. Suppose instead a team of ten, granted the most generous bounds: seven times the productivity of ordinary programmers, and a further factor of seven from the communication they avoid. Five thousand divided by ten times seven times seven is ten years. The product would be obsolete before it shipped.

Harlan Mills's proposal resolves the dilemma by changing the shape of the team rather than its size. A segment of a large job goes to a team organised like a surgical team rather than a hog-butchering team: one person does the cutting and the others give him every support that raises his effectiveness. The surgeon, whom Mills calls the chief programmer, personally defines the specifications, designs the program, codes it, tests it and writes its documentation. The copilot is his alter ego, able to do any part of the job and knowing all the code, but responsible for none of it. The others are specialists: an administrator for money, people, space and machines; an editor who reworks the surgeon's draft; a program clerk who keeps every technical record in a library and makes all runs visible to the whole team, which is what Mills meant by turning programming from a private art into a public practice; a toolsmith who builds what this surgeon wants regardless of what other teams want; a tester; a language lawyer who can serve two or three surgeons. Ten people, seven of them professionals, work the problem, and the system is the product of one mind, or at most two acting as one.

The difference from a conventional pair lies in two places. Conventional partners divide the work and each owns part of the design, and being equals they must talk out every difference of judgment, compounded by differences of interest over whose part gets the buffer space. The surgeon and copilot are cognisant of all the design and all the code, there is nothing to divide, and the surgeon settles judgments unilaterally.

Aristocracy, democracy, and system design

The chapter opens on Reims cathedral. Most European cathedrals show changes of plan and style between the parts built by different generations, until the peaceful Norman transept contradicts the soaring Gothic nave. Reims is unified because eight generations of builders each gave up some of their own ideas so that the whole might be of one design. Programming systems, Brooks says, show conceptual disunity worse than the cathedrals do, and not from a succession of masters but from the separation of design into many tasks done by many men at once.

Conceptual integrity is the most important consideration in system design: better a system that omits certain anomalous features but reflects one set of ideas than one that contains many good but uncoordinated ones. The measure that follows is the ratio of function to conceptual complexity, which Brooks calls the ultimate test of a design, because ease of use is the purpose and the ratio is what expresses it. Neither half will do alone. OS/360 was hailed by its builders as the finest ever built because it had the most function; the Time-Sharing System for the PDP-10 was hailed by its builders for the spareness of its concepts, and its function was not in the same class. Each reached for half the goal.

Conceptual integrity requires that the design proceed from one mind or a very few agreeing minds, and schedules require many hands. Brooks's resolutions are the surgical team and, on large projects, a division of labour between architecture and implementation. By the architecture of a system he means the complete and detailed specification of the user interface: for a computer, the programming manual; for a compiler, the language manual; for a whole system, the union of the manuals a user must consult. Blaauw's formulation is that architecture tells what happens and implementation tells how it is made to happen, and his example is a clock, whose architecture is the face, the hands and the winding knob. A child who has learned that architecture reads a wristwatch and a church tower alike, and what goes on inside the case is another matter. A single System/360 architecture was implemented differently in nine models. The architect is the user's agent, bringing technical knowledge to bear in the user's interest as against the salesman's or the fabricator's.

To the charge of elitism Brooks answers yes and no. Yes, in that there must be few architects, that their product outlives the implementer's, and that if a system is to have conceptual integrity someone must control the concepts; that is an aristocracy which needs no apology. No, in that setting external specifications is not more creative than designing implementations, merely different: cost and performance depend most heavily on the implementer just as ease of use depends most heavily on the architect. Discipline, he argues, is good for art: Bach's output does not seem to have been squelched by having to produce a cantata each week, and in an unconstrained implementing group most thought goes into architectural argument while implementation gets short shrift.

The counter-argument he records against himself is the one he paid for. Writing the external specifications for OS/360, the architecture manager had ten good people and said they would take ten months, three more than the schedule allowed. The control program manager had a hundred and fifty and said his people could prepare the specifications on schedule with the architects coordinating, and that otherwise his hundred and fifty would sit idle. The architecture manager replied that the result would be three months late anyway and of much lower quality. Brooks gave the job to the implementers; it was late and it was worse, and he estimates that the missing conceptual integrity added a year to debugging time. The objection that the many must sit idle while specifications come through a narrow funnel he calls an illusion, since architecture, implementation and realisation can begin in parallel and proceed together.

Why did the tower of Babel fail?

Brooks audits the Babel project as an engineering manager would. It had a clear mission, plenty of manpower, abundant clay and bitumen, no time constraint, and adequate technology, since a conical masonry structure is stable and spreads its load well. It failed on communication and its consequent, organization. So it is today: schedule disaster, functional misfit and system bugs all arise because the left hand does not know what the right hand is doing, and teams drift apart in their assumptions. The purpose of organization, Brooks writes, is to reduce the amount of communication necessary; the authority structure is a tree, the communication structure is a network, and the dotted lines exist to make up the difference.

The essays between here and the new chapters are the working parts of that argument. The second system a person designs is the most dangerous, because the caution of the first has worn off and the ideas it had no room for now demand entry; OS/360 is Brooks's own example, and assigning values in bytes and microseconds to functions is his discipline against the tendency. An architecture needs both a formal definition for precision and a prose one for comprehension, and the cleanest way to enforce it is direct incorporation, letting others build one's own code into their products rather than asking them to reimplement a specification. Estimating cannot be done by timing the coding and multiplying, because effort rises as a power of size, with an exponent the studies of the day put near 1.5.

The documentary hypothesis holds that amid a wash of paper a small number of documents become the pivots around which a project's management turns: the objectives, the manual, the schedule, the budget, the organization chart. Preparing each one forces hundreds of small decisions and so separates an exact policy from a fuzzy one; maintaining each one is a warning mechanism. The eleventh essay tells builders to plan to throw one away, on the analogy of the pilot plant that chemical engineers interpose between the laboratory and the factory: the first system built is barely usable, the discard and redesign will happen whether or not it was planned, and the only question is whether the throwaway is delivered to customers. The chapter on catastrophe answers its own question — a project gets to be a year late one day at a time — and prescribes milestones defined with knife-edge sharpness, because a programmer will rarely lie about a milestone he cannot deceive himself about, whereas coding is ninety per cent finished for half of the coding time.

No silver bullet

The 1986 paper that Brooks added as the book's sixteenth chapter begins with werewolves, which terrify because they transform unexpectedly from the familiar, and with the silver bullets sought for them. Brooks predicts that no single development in technology or management technique will by itself give an order-of-magnitude improvement in productivity, reliability or simplicity within a decade. He notes first that the anomaly is not that software progress is slow but that hardware progress has been unmatched in human history, six orders of magnitude of price-performance in thirty years, which came from turning computer manufacture from an assembly industry into a process industry.

Then, following Aristotle, he divides the difficulties into essence, inherent in the nature of software, and accident, which attends its production today without being inherent. The essence of a software entity is a construct of interlocking concepts — data sets, relations among data items, algorithms, invocations of functions — abstract in that the same construct survives many representations, and the hard part of the work is specifying, designing and testing that construct rather than representing it. Four properties make the essence hard. Complexity: software entities are more complex for their size than any other human construct, because no two parts are alike above the statement level, since parts that are alike get made into one; scaling up is not repetition but an increase in the number of different elements, interacting nonlinearly. From this come the communication difficulties, the unenumerable states and hence the unreliability. Conformity: the physicist can believe that unifying principles exist, because the complexity he faces was not designed by committees; much of the software engineer's complexity is arbitrary, forced by the human institutions his product must match, and no redesign of the software alone can simplify it away. Changeability: the known high cost of changing a building dampens the whims of its owner, whereas software embodies the function, which is what most feels the pressure to change, and it is malleable enough to invite it. Invisibility: software has no natural geometric representation, because its structure is several superimposed directed graphs — control flow, data flow, dependency, time sequence, name space — usually not even planar, so the mind is deprived of one of its strongest tools.

The three breakthroughs that did work each attacked an accident, and each has a visible ceiling. High-level languages freed programs from the bits, registers and branches that were never part of the abstract program, and most observers credit them with a factor of five; the most a language can do is supply every construct the programmer imagines. Time-sharing preserved the programmer's grasp of a complex system across the interruption of a batch turnaround, and its limit arrives when response time drops below the threshold of noticeability, about a hundred milliseconds. Unified environments such as Unix attacked the accidents of using programs together.

From the division comes the arithmetic that carries the argument. If the accidental part of the work is less than nine tenths of the total, then reducing it to zero, which would take magic, cannot give a tenfold improvement. Brooks therefore takes each proposed bullet and asks which part it attacks. Ada is another high-level language, and the large payoff came from the first transition; its real contribution will be that adopting it trains programmers in modern design. Object-oriented programming removes a higher order of accidental difficulty and is a real advance, but the complexity of the design itself is essential and untouched. Artificial intelligence, on Parnas's reading that Brooks adopts, names no transferable body of technique, and the hard thing about building software is deciding what to say, not saying it; expert systems could put the experience of the best programmers at the service of the inexperienced, but the prerequisite for building one is having an expert. Automatic programming has always been a euphemism for programming in a higher-level language than the one at hand. Graphical programming rests on the flow chart, which programmers draw after writing the code, and on an analogy with chip design, which misleads because a chip is a layered two-dimensional object whose geometry is its essence. Program verification at best establishes that a program meets its specification, whereas much of the essence of building a program is debugging the specification. Better environments and faster workstations return only marginal gains, since think-time already dominates the programmer's day.

The attacks he does endorse address the essence. The most radical is not to build the software at all: a purchased package costs about what one programmer-year costs and is delivered immediately, and Brooks calls the mass market the deepest long-run trend in the discipline, noting that what made packages acceptable was not the packages but the hardware-to-software cost ratio, since the buyer of a fifty-thousand-dollar office machine adapts his payroll procedures rather than pay for a custom program. Rapid prototyping attacks the hardest single part of building a software system, which is deciding precisely what to build; clients do not know what they want, and cannot in Brooks's view specify a modern product correctly before having built and tried some version of it. Incremental development replaces the metaphor of building with the metaphor of growing: Mills's proposal was that a system first be made to run while doing nothing but calling the proper set of dummy subprograms, then be fleshed out function by function, so that one always has a working system. And the last attack is on people: great designs come from great designers, the difference between the great and the average approaches an order of magnitude, and the systems that have excited passionate followings are the work of one or a few designing minds. Organisations spend real effort finding and cultivating management prospects and almost none finding designers, and Brooks proposes that the two be made equivalent in salary, recognition, office, equipment and support.

The chapter he added beside it in 1995 answers nine years of rebuttals. By accidental, Brooks clarifies, he meant incidental in Aristotle's sense, not unfortunate or random. He restates the argument as a question of fact — what fraction of software effort goes to representing the conceptual construct rather than crafting it — puts the representational part at about half or less, and notes that no correspondent has claimed it is as large as nine tenths, which is what the tenfold claim requires. Capers Jones supplies the emendation he accepts most readily, that the focus on productivity was misplaced, since late and costly projects spend their extra time finding and repairing errors of specification and design, so quality should be pursued and productivity will follow. Reuse, he concludes, is far easier to say than to do: a reusable component costs about what productising costs, which returns the argument to the factor of three in the first essay.

Since 1975

The twentieth-anniversary chapter separates what Brooks still held, what he thought wrong, and what had surprised him. He was more convinced than ever of the central argument. Conceptual integrity is what makes a product usable, a system architect is the single largest step toward it, and the role is a full-time job separable from the manager's the way a film's director is separable from its producer. After teaching a software engineering laboratory more than twenty times, he came to require that student teams of four choose a manager and a separate architect, and found that it worked. His example of conceptual integrity achieved is the windows-and-menus interface, which took a familiar mental model, the desktop, and extended it consistently enough that dragging a disk to the trash to eject it is jarring; and whose uniformity across applications was enforced not by decree but by putting the interface in read-only memory, so that using it was cheaper than building one's own.

What he withdrew, he withdrew flatly. Plan to throw one away is wrong, he writes, not because it is too radical but because it is too simplistic: it assumes the sequential waterfall model, which runs through the whole book and shows itself first in the scheduling rule of the second essay. The waterfall assumes a project passes through the stages once and that the mistakes will be in the realisation, so that repair can be interleaved with testing; it puts system test, and therefore user test, at the end, after full construction has been paid for. Experience from each downstream stage must instead leap upstream, and a project may iterate through two or more architecture-and-implementation cycles before any code is written. The replacement is the incremental build: a running skeleton, then function added module by module with a working system at every stage, which allows user testing early and a build-to-budget strategy that cannot overrun the schedule, at the cost of possible shortfall in function. Microsoft, he records, rebuilt its developing system every night and stopped the whole process when a build broke; Bell Northern Research rebuilt twelve million lines every week.

The second withdrawal is the one he names in a chapter heading: Parnas was right, and I was wrong. The seventh essay had every OS/360 programmer holding a copy of a project workbook that grew past ten thousand pages, on Mills's argument that programming should be a public process, and had dismissed Parnas's encapsulated modules with hidden interiors as a recipe for disaster. Brooks now holds that information hiding, since embodied in object-oriented programming, is the only way to raise the level of software design, because it is the first step toward composing programs out of chunks much larger than a statement: Parnas's module, with its own data model and its own operations, became the abstract data type, which became the class. The documentary hypothesis survives as a principle and not as a mechanism. The small set of pivotal documents still governs a project, but the five-foot looseleaf binder and the microfiche that replaced it are gone, and the proposition that every team member should see all the material Brooks softened to the claim that every member should be able to see it.

The surprise he shares with every other practitioner of his generation is the microcomputer and the shrink-wrapped industry it created. The economics invert: development cost is divided across hundreds of thousands of units, and schedule and function dominate it, where in the classical in-house industry schedule and function were negotiable and cost was not. The consequence Brooks finds most interesting for his own argument is metaprogramming: building on a shrink-wrapped database or spreadsheet means acquiring a large module of function whose internal conceptual structure does not have to be designed at all, which attacks the essence rather than the accident.

Later data has mostly held the book up. Barry Boehm's study of sixty-three projects found a cost-optimum schedule that goes as the cube root of the expected effort in man-months, and hardly any project succeeding in less than three quarters of that optimum regardless of how many people were applied; his cost model also found the quality of the team to be the largest factor in success, about four times more potent than the next. The law itself has been examined directly. Abdel-Hamid and Madnick, modelling project dynamics with a learning curve for new people, concluded that adding people to a late project always makes it more costly but does not always make it later, and that adding early is much safer than adding late, since new people have an immediate negative effect that takes weeks to repay. Neither their model nor Stutzke's includes the repartitioning that Brooks says he has repeatedly found nontrivial. He accepts the qualifications and keeps the bald statement as a rule of thumb against the instinctive fix.

What has dated is mostly apparatus. The surgical team as literally staffed — two secretaries, a program clerk filing batch output in a chronological archive — dissolved once version control and interactive terminals did the clerk's work, though the shape it was built to produce, a design owned by one or two minds with everyone else raising their effectiveness, is recognisable in how small product teams are still assembled. The scheduling rule, the size budgets and the productivity figures in lines of assembler belong to their decade. The silver-bullet prediction, however, has now run four decades rather than one without a single development producing the tenfold gain by itself, and the distinction that carries it, between accidental difficulty that tools can remove and the essential difficulty of deciding what to build, has outlived every candidate the paper examined. Incremental delivery, continuous integration and user testing before full construction, which Brooks arrived at by repudiating his own advice, became the default practice of the industry. The part still least acted on is the one he put last: that organisations look hard for managers and hardly at all for designers.