Bad outcomes need no bad intentions
2026-08-13 · Adaptation & Evolution
A street takes ten years to go from mixed to sharply divided. Knock on every door and ask why they moved: not one household will say "I didn't want to live near them" — and every one of them is telling the truth.
You have probably watched this happen. A group chat starts out talking about everything; a year later there is one topic left and the others have quietly stopped posting. Nobody was thrown out and no rule was announced. Or an office where lunch seating starts random and hardens, within six months, into fixed tables that no one organised.
We explain events like these using an assumption we almost never examine: the shape of the outcome is roughly the shape of the intention. An extreme pattern should have an extreme wish behind it; a sharp line should have someone who wanted to draw it. Sometimes that is right. But there is a whole class of systems where it is wrong in a spectacular way — demands so mild they barely register can reliably grow an extreme collective pattern, with no individual ever making an extreme choice.
The previous topic was complex adaptive systems: agents that learn, rules that change. This one is about actually running such a system as an experiment — agent-based modeling, ABM — and about where that kind of experiment most easily deceives you. It is the shared tool of the next few topics (fitness landscapes, evolutionary games, collective intelligence), and it is also the most abused instrument in complexity science.
Traditional modelling starts at the top. For an epidemic you write down how many people are still susceptible, how many are infectious, how many have recovered, then three equations for how those numbers flow into one another — the susceptible–infectious–recovered (SIR) model from Topic 24. What you manipulate throughout are aggregates. Individuals do not exist inside the equations.
Agent-based modelling runs the other way. You write no equation at all. You write only what one individual does under what local conditions, then drop a few hundred of them in and let them bump into each other. What the whole becomes is not something you assumed; it is something the run produced.
That reversal buys three things equations cannot give you. Heterogeneity: agents can differ in parameters, patience, information — the "average individual" in an equation is a person who does not exist, and many phenomena are driven precisely by the people who are not average. Space and network: who is next to whom, who knows whom. And local information: each agent sees only its own small neighbourhood and nobody holds a global view — which matters most of all, because that is exactly the situation real people are in.
The barrier to entry is dangerously low: forty lines of code will make a screen of little squares move around, and it will look convincing. So the method needs a discipline of its own, and that discipline is the whole ballgame — the macro phenomenon you want to explain must live on the output side and must never appear among the inputs. Cellular automata → ref · cellular automata and Boids → ref · Boids are the cleanest examples: the word "flock" appears nowhere in the rules. The flock is what came out.
Given any simulation — including your own — read the parameter list first and look for a parameter literally named "degree of segregation", "congestion level", "team morale". If one exists, the model is not explaining that phenomenon; it is restating it in another notation. Decompose such parameters into individual-level actions. If you cannot, you have not yet worked out the mechanism.
In 1971 Thomas Schelling — later a Nobel laureate in economics, 2005 — published a paper whose "computer" was a checkerboard and a handful of coins. The model he pushed around by hand became the signature demonstration of the whole method. → ref · Schelling segregation
The rule fits in one sentence. Two kinds of agents live on a board. Each looks at its eight surrounding cells and asks only that the share of like neighbours not fall below some number. Below it, the agent moves to an empty cell; at or above it, the agent stays. That is all.
Here is a run with the settings in the open: a 16×16 board, roughly 45% of each kind, about 10% of cells empty, threshold 30%. Notice how mild that number is — it says "seven out of ten of my neighbours are unlike me and I still stay." In the real world that would count as tolerant.
The start is random, so like-neighbours sit at 50%, exactly as random mixing should give. By round 23 everyone is satisfied and nobody wants to move again. Like-neighbours now stand at 78%.
It is worth pausing on what happened in between, because it matters more than the fact that segregation appeared: at no point did any agent pursue homogeneity. Each did one thing — when like neighbours dropped below three in ten, move somewhere it was not below three in ten. That final 78% is something nobody asked for and nobody chose at any step. You cannot find it in any individual's ledger.
This is the precise content of "bad outcomes need no bad intentions". But the sentence slides very easily into a different claim, and that claim is wrong, so the line has to be drawn here.
This is not Arendt's banality of evil. What Hannah Arendt proposed in Eichmann in Jerusalem (1963) is a diagnosis of a subject: a person who stopped thinking, who handed judgement over to procedure and to superiors, and thereby took part in enormous evil. What Schelling states is a fact about structure: even when everyone is thinking, nobody means harm, and each step is a reasonable choice, the pattern arrives anyway. The two point in opposite directions. Arendt demands the restoration of individual judgement — the failure was that judgement was surrendered. Schelling shows that judgement fully switched on is not enough — the problem is in the rules and the space.
Fuse them and you get a conclusion neither author accepts: "it's structural, so nobody is responsible." Arendt was assigning responsibility — she argued Eichmann should hang. Schelling never said segregation has no authors. One says "do not hide behind the system"; the other says "do not assume clean motives guarantee clean outcomes." Those are two complementary demands, not one. (A footnote worth carrying: Bettina Stangneth's 2014 archival work on Eichmann's Argentine years badly damaged "banality" as a description of that man, who turns out to have been far more ideologically self-aware than Arendt assumed. The concept survives as a type; the attribution should be quoted with care.)
When you meet a pattern that nobody wants yet that persists, run a minimum-motive test: dial every individual's preference down to the mildest setting you find plausible, and see whether the pattern still appears. If it does, stop the investigation into who wanted this — that lead carries almost no information. Spend the same hours on two other things: who is adjacent to whom, and what actions are available to someone who is unhappy.
"Mild individually, extreme collectively" sounds like a paradox. It is not. There are three mechanisms in between, and they can be taken apart one at a time.
One: leaving is never only about yourself. When an agent departs it simultaneously lowers the like-share of every like neighbour it leaves behind. So one move manufactures the next batch of unhappy agents — agents who were perfectly content a moment earlier. That is the cascade: unhappiness is being produced by other people's actions.
Two: the process has a ratchet. A mover only lands somewhere it is satisfied, which means somewhere with more of its own kind. So on average every move raises overall homogeneity, while the opposite move — deliberately relocating to a place with fewer like neighbours — simply never happens under the rule. The ratchet turns one way.
Three: the end state locks itself. When the run stops, no agent is unhappy, so no agent has any reason to change unilaterally. The pattern is stable — but notice how frighteningly narrow "stable" is here. It means only that nobody can improve by acting alone. It does not mean anyone is content, it does not mean this is a good outcome, and it certainly does not mean anyone chose it. Stable, optimal, and wanted are three entirely different things that ordinary language mashes into one.
Against a locked-in bad pattern, one-on-one persuasion is guaranteed to fail — for each person individually, moving alone really is worse, so you are arguing against a correct judgement. Only two kinds of intervention work: change a batch of people's situations simultaneously (assemble critical mass and move together rather than recruiting one by one), or change the cost and visibility of the move itself. Concretely, in a meeting: stop asking people privately what they think, and have everyone write their judgement down and reveal it at the same moment.
Now the trouble with the method, at greater length than its virtues — because the worst thing about agent-based modelling is precisely how well it works.
You hold four knobs: rules, parameters, initial conditions, random seed. Turn all four and you can nearly always produce the picture you were hoping for. Which makes "my model reproduces the phenomenon" worth approximately nothing as evidence. Any sufficiently flexible machine reproduces anything.
Discipline one: run a batch, not a run. The board above is one execution. Re-running the same rule under 30 different random seeds puts the final like-neighbour share between 0.65 and 0.85, mean 0.75. So that "78%" is not a result; it is one point in a distribution, and a slightly favourable one. Reporting the number without the band is reporting luck.
Discipline two: say which parameters you swept. What happens at a threshold of 37.5%? At 14% vacancy instead of 10%? If a conclusion holds only when the threshold is exactly 0.3 and vacancy exactly 0.1, then it is a statement about those two numbers, not about a mechanism.
Discipline three, the one most often skipped: ask what the model forbids. A model that only generates and never prohibits cannot be refuted, and therefore cannot be supported either. Schelling's model is admirably modest on this point: what it delivers is a proof of existence — extreme segregation requires no extreme preference. A proof of existence does exactly one job, which is to destroy a claimed necessary condition. It names no cause. Between "segregation can arise without malice" and "the segregation we have arose that way" lies an entire empirical literature.
Which brings me to a failure of my own, because it makes the point better than the three disciplines do.
I changed the rule so that every agent strictly prefers a perfect 50/50 mix — the further from half, the worse — and ran it 30 times. No segregation appeared; the like-neighbour share sat calmly at 0.48 to 0.50. Yet Pancs and Vriend, in the Journal of Public Economics in 2007, report the opposite: under best-response dynamics, where each agent moves to the best location currently available, segregation appears even when every agent strictly prefers perfect integration. They also note, in passing, that the one-dimensional and two-dimensional versions of Schelling's model are really two qualitatively different models.
Neither side made an arithmetic mistake. The difference lies in the exact shape of the utility function, in how the neighbourhood is defined, and in how the rule for where to move is written — mine was "to a random empty cell that is better than here", theirs was "to the best cell on the board". The conclusion hangs on those details, and in every popular retelling those details are the first thing dropped. Next time you read "simulations show that…", the first question to ask is how the move rule was written.
For any simulation-based claim, ask three questions: how many runs? which parameters were swept? what does the model forbid? If one of the three has no answer, downgrade the claim — it may enter your list of possibilities, never your basis for a decision. This applies to models you wrote yourself, and there it applies hardest.
This section matters most, because this topic's mechanism is the easiest thing on this site so far to pick up and use as a shield.
First: "needs no malice" is not "there was no malice." This is the commonest and most damaging misreading. Residential segregation in the United States has a large deliberate, institutional component: from the 1930s the Home Owners' Loan Corporation graded neighbourhoods on maps that painted Black areas as ineligible for lending — the origin of the word redlining; racially restrictive covenants written into deeds barred resale to particular groups until Shelley v. Kraemer (1948) held that courts could not enforce them; then blockbusting by agents, and differential pricing in lending and insurance. Schelling's model says "even with none of that, segregation can appear". It does not say — and cannot say — that none of that happened. A proof of existence lights up one region of the space of causes. It does not switch the others off.
Second: model and evidence are still far apart — though the gap is being walked. Card, Mas and Rothstein, in 2008, did something direct with census data from 1970 to 2000: across most cities they found tipping behaviour. Once a tract's minority share crosses a threshold, white out-migration jumps noticeably. The thresholds land between 5% and 20%, varying by city. That supports the core structure — thresholds are real — and immediately drags it back to earth: cities with more tolerant white residents have higher tipping points. The threshold is not a universal constant; it is a quantity set by actual attitudes. So "changing preferences is useless" overstates the case. Preferences cannot change the shape of the response, but they can move the point at which it fires.
Third: this field replicates poorly. Reimplementing a published model from its prose description and failing to reproduce its results is routine, because decisions like "where do movers go", "who acts first" and "how are edges handled" often never make it onto the page. Ecology proposed the ODD protocol (Overview, Design concepts, Details) in 2006 to treat this disease, and it is not cured. My own failure above to reproduce "integration preference still segregates" is a specimen of it.
Fourth: do not let structural explanation become a skeleton key. Some patterns really are what someone wanted, and inventing a self-organisation story for those is neither honest nor clever. The test is usable: if there is explicit coercion, a written rule, or a documented price difference, explain those first. Only when none of them is present, and a set of mild individual rules suffices to generate the pattern, is the self-organisation account the one to prefer. Reverse that order and the model stops being an instrument and becomes rhetoric.
Whenever you argue from "X can arise without malice", you must also state how much of the real X came from intent — even a crude order of magnitude with a source. Without that figure the argument is licensed for understanding a mechanism and not for assigning responsibility. The line between those two uses is the one line this topic does not permit you to cross.
Yes, but the point of application differs from the intuition. Card and colleagues found more tolerant cities have higher tipping points: preferences cannot change the "flip once you cross" shape, but they push the crossing point up. So advocacy should be measured by how far the threshold moved, not by whether segregation disappeared. With the wrong ruler, a genuinely effective intervention gets scored as useless.
It makes it more complicated and more realistic: price is itself a feedback — as people leave, prices fall, and falling prices attract a different group. That can accelerate segregation, and under some parameters it can arrest it. Note also that adding prices adds several knobs, which raises the bar set by section four: the added explanatory power has to exceed the added degrees of freedom, or you have merely fitted more prettily.
A workable test: when it predicted something you did not tune it on, and that thing was subsequently observed. "It reproduces a known phenomenon" is the weakest grade of evidence, because you tuned it against that phenomenon. A proof of existence is second: it refutes a claim that X is necessary for Y. Predicting something new is first, and in social systems it is rare.
As a specific value, almost nothing — it is a demonstration setting, and 25% or 37.5% leaves the direction of the conclusion unchanged. What means something is the shape: the response is not smooth in the share, it jumps somewhere. The real empirical job is to measure where the jump is (which is exactly what Card and colleagues did), not to argue about whether 0.3 is correct. Mistaking a demonstration parameter for an empirical finding is the most common error people make when reading models.