85The Mythos of Model Interpretability

Zachary C. Lipton, ICML Workshop on Human Interpretability in Machine Learning, New York, 2016

Linked to 12 chapters: supports 6 · contradicts 1 · mechanism 6 · same claim 2 · pairing 3

By 2016 a machine learning paper could claim interpretability the way it claimed accuracy, without saying what had been claimed. Lipton reads that literature and offers two readings: either the definition is agreed and nobody has managed to write it down, or the term is ill-defined and claims made with it have what he calls a quasi-scientific character. He argues for the second, and separates the two questions inside the word. What an interpretation is wanted for, where he finds five answers that do not reduce to one another and occasionally conflict: trust, causality, transferability, informativeness, and fair and ethical decision-making. And what property of a model supplies it, where he finds two families that compete rather than combine: transparency, which asks how the model works, and post-hoc explanation, which asks what else the model can tell you, and which an opaque system can offer as readily as a clear one. One condition produces all five desiderata: the demand appears when the formal objective of supervised learning, error on a test set drawn from the training distribution, comes apart from what deployment will cost. And one consequence gave the paper its audience: the claim that a linear model is more interpretable than a deep network is true under one of his three notions of transparency and false or unavailable under the other two, and the human decision-maker, held up throughout that literature as the interpretable case, satisfies none of the three.

Where the demand comes from

The common evaluation metrics of supervised learning need two things and no more: the model's predictions and the ground truth. Any model can be scored that way, so the score cannot be what is missing when someone asks for an interpretation. Lipton takes the request as evidence of an objective that resists being written as a real-valued function. A model for hiring should optimise productivity, ethics and legality together, and the last two cannot be optimised directly; a model trained on longitudinal medical data is scored on error, which pure correlation can drive down, while the reason for building it was to find associations that might be causal; a model deployed into a moving environment can change that environment and invalidate its own predictions.

What an interpretation is wanted for

Trust comes first, and Lipton declines to let it stand as one thing. If it means confidence that the model will perform well, a sufficiently accurate model is already trustworthy and an interpretation adds nothing. It may instead mean confidence with respect to the real objective rather than the trained one: a model forecasting crime rates to allocate police officers may be trusted to predict accurately and not trusted to account for racial bias in its training data, or for its own part in a cycle of over-policing. Or it may mean comfort in relinquishing control, which turns not on how often the model is right but on which cases it is right for — if its mistakes fall where human mistakes fall, handing over costs nothing in expectation. That last sense became an experiment for Bansal and colleagues, who found that explanations did not raise a person and a model above either working alone.

Causality is the second. Supervised learning is optimised to make associations, and researchers read those associations as hypotheses about the world, a regression revealing thalidomide and birth defects or smoking and lung cancer. An association need not be causal, since an unobserved cause may produce both variables, so the most an interpretation can do is propose something for an experimenter to test.

Transferability is where the examples are sharpest. The usual test of generalisation is a random split of one distribution, while people transfer skills to situations drawn from nothing they have seen. Lipton takes from Caruana and colleagues in 2015 a model trained to predict death from pneumonia that assigned lower risk to asthmatic patients. The association was real: asthmatics did die less often, because they were admitted straight to intensive care. Deployed in triage, the model would have sent those patients to the less aggressive treatment that made the correlation false, and none of this is overfitting in the classical sense. The environment can also be hostile on purpose. FICO's technical report names interpretability as the reason it builds credit models with logistic regression, on features including debt ratio and the number of accounts in good standing — both of which an applicant can move at will, without changing anyone's ability to repay.

Informativeness covers the case in which a model's output is handed to a person who decides rather than fed to a decision rule. Lipton's analogy is a graduate student asking an advisor which venue suits a paper: naming one conference gives her nothing to combine with what she already knows. An interpretation can inform without saying anything about a model's workings, as a diagnostic system does when it points at similar past cases.

Fair and ethical decision-making is the last. Programs approve credit lines, filter job applicants, and predict recidivism for courts deciding who is released and who is detained, and accuracy offers little assurance that the resulting actions are acceptable. Lipton notes the European regulations then being drafted, which propose that people affected by an algorithmic decision have a right to explanation and that such decisions be contestable, and sets a condition that follows from the second word: an explanation is useful only if it reasons from falsifiable propositions and offers a way of contesting them and changing the decision when they fail. Almost nothing in the published work, he observes at the end, addresses contestability at all.

How does the model work?

Transparency is his name for the first family, and he takes it at three levels.

Simulatability is the strictest: a person contemplates the entire model at once, taking the input and the parameters and stepping through every calculation behind a prediction in reasonable time. It has two subtypes, since for a decision tree the number of nodes can grow much faster than the path from root to leaf, so model size and inference cost come apart. What counts as reasonable is subjective, but human capacity bounds the ambiguity to a few orders of magnitude, and within those bounds Lipton draws the conclusion the rest of the paper needs: neither linear models nor rule-based systems nor decision trees are intrinsically interpretable, and a high-dimensional linear model, an unwieldy rule list or a deep tree can each be less transparent than a compact neural network.

Decomposability asks that every part — each input, each parameter, each calculation — admit an intuitive explanation: a tree node that reads as all patients with diastolic blood pressure over 150, a linear weight that reads as a strength of association. It requires the inputs themselves to be interpretable, which disqualifies any model built on heavily engineered or anonymous features, and Lipton declines to take even the surviving case on faith. Linear weights are fragile with respect to feature selection: the association between flu risk and vaccination can come out positive or negative depending on whether the feature set includes indicators of old age, infancy or immunodeficiency.

Algorithmic transparency applies to the learning procedure rather than the learned object. For linear models the shape of the error surface is understood and training provably converges to a unique solution even on data not yet seen. The optimisation heuristics of deep learning work without anyone understanding why, and nothing can be guaranteed in advance about a new problem. Here the linear model wins outright, and Lipton immediately spends the win: humans exhibit none of these forms of transparency.

What else can the model tell me?

Post-hoc interpretation extracts information from a model already trained, without saying how it works, and costs no predictive performance. It is also the only sense in which people are interpretable at all: the process by which a person decides and the process by which they explain may be entirely distinct. Lipton sorts the techniques into four kinds.

Text explanations train a second model to speak for the first. Krening and colleagues pair a reinforcement learner choosing actions with a model that maps its state onto verbal explanations of strategy, fitted to the explanations of human players — so the account is trained towards what a person would have said, and need not describe the agent's decisions however plausible it looks. McAuley and Leskovec explain a recommendation by the top words of a topic model tied to the rating model's factors, a way of reading topic models that has itself been found wanting.

Visualisation renders the learned representation and hopes a person will recognise something. Mordvintsev and colleagues run gradient ascent on the input to enhance selected hidden units, and dog faces appear all over the image, presumably because the training corpus was full of animals; Mahendran and Vedaldi recover an image from a representation six layers into an AlexNet.

Local explanations describe a neighbourhood instead of the whole mapping. A saliency map takes the gradient of the correct class output with respect to the input and applies it as a mask, showing which regions would most change the answer; Ribeiro and colleagues instead fit a sparse linear model near the point of interest. Lipton's objection is exact: the map holds only locally, and moving a single pixel can produce a very different one, where a linear model at least states a global relation.

Explanation by example reports which other cases the model treats as nearest in its learned space — the technique behind reading word2vec by enumerating neighbours, and the thing doctors do when they cite a case study.

The warning attached to all four is that an explanation optimised to satisfy a person is optimised for plausibility, and people already do this: acceptance decisions credited to leadership or originality have repeatedly been shown to disguise racial or gender discrimination. In the rush to make machine learning acceptable, he writes, "we should be careful not to reproduce pathological behavior at scale". A transparency claim can be demonstrated directly; a post-hoc claim must fix an objective and produce evidence that the interpretation achieves it, which is what On the Biology of a Large Language Model does nine years later, validating its graphs by intervention and reporting the fraction of prompts on which they show nothing.

Linear models are not strictly more interpretable

The claim that gives the discussion its heading is true or false depending on which notion is meant, and Lipton lays out the accounting.

Read down the columns and the usual choice becomes a trade between algorithmic transparency and decomposability. Deep networks operate on raw or lightly processed features, so their inputs mean something, while a linear model often reaches comparable performance only on hand-engineered features and pays for it in the property it was chosen for; Lipton cites his own work on clinical time series as such a case. On the desiderata he concedes what the evidence supports and no more: linear models have the better record for studying the natural world, and he knows no theoretical reason why that must be so.

Transparency can also conflict with the broader objective, since arguments that rule out black boxes rule out any model that would match or exceed human ability on a hard task, and building trust with doctors by offering them transparent models can work against the longer goal of better medicine. The problem, in the end, is one of formulation rather than of method. The term, he writes in his summing-up, "holds no agreed upon meaning", while conferences publish papers that wield it in a quasi-mathematical way. When a method is flawed a better method can be proposed; when the problem statement is flawed, no algorithm and no experiment reaches it, which is why he ends by arguing that critical writing belongs at machine learning conferences.

Since 2016

The paper did what it asked for. Its vocabulary — transparency against post-hoc explanation, simulatability, decomposability, algorithmic transparency — became the standard way of asking which sense of the word a paper means, and it was reprinted for practitioners in ACM Queue in 2018. Doshi-Velez and Kim pressed the same complaint the following year and added what Lipton left out: a scheme for evaluating an interpretation against a real task, against a proxy task with lay subjects, or against a formal definition alone.

The warning about post-hoc methods aged best. Adebayo and colleagues showed in 2018 that several widely used saliency methods produce much the same map after a model's weights have been randomised, which is to say they were partly showing properties of the image rather than of the model. Rudin went further than Lipton in 2019, arguing that for decisions with high stakes the answer is not to explain a black box but to stop using one; his reply, that transparency may be at odds with the broader objective, still stands where he left it. The legal thread went the other way: the regulations he cited came into force in 2018, and Wachter, Mittelstadt and Floridi had already argued that they grant no right to an explanation of a particular decision, which left contestability where he found it.

What has changed most is the kind of model in question. Work on deep networks split into a post-hoc industry of local attributions and a smaller effort to take transparency literally by reading circuits out of trained weights. And the first of his four techniques, a second process trained to produce a plausible account of the first, stopped being a research proposal and became the ordinary way of putting a question to a model, which makes the faithfulness question he raised about it the one now most often asked.