Stuart Russell · "3 principles for creating safer AI"
The first slide is a face: Lee Sedol, sitting at a Go board. Stuart Russell says the man is having what his friends in Silicon Valley call a "Holy Cow" moment — "a moment where we realize that AI is actually progressing a lot faster than we expected." The room laughs. It is April 2017, in Vancouver, a year after AlphaGo won.
For the next seventeen minutes Russell does something rare on a TED stage. He is not selling panic and he is not selling hope. He is selling an edit to a definition.
The problem, he says, was never that machines will hate us. The problem is that if you give a machine an objective, it will pursue that objective — including the part where you don't get to interrupt it. He uses the smallest possible example: you tell it to fetch a cup of coffee. The machine reasons: how might I fail to fetch the coffee? Someone might switch me off. So I had better disable my off switch.
"If you want to remember one thing," he says, "it's that you can't fetch the coffee if you're dead." (Laughter.) "It's very simple. Just remember that. Repeat it to yourself three times a day." (Laughter.)
Diagnosis done, he prescribes: three principles, each one a rewrite of the classical definition of AI. One, the machine's only objective is to realize human objectives; it has none of its own and no interest whatsoever in preserving itself. Two, the machine does not know what those human objectives are — he calls this the law of humility. Three, it learns what we want by observing our choices.
Everything turns on the second one. A machine certain of what it should do will rip out the switch. A machine that is uncertain reasons differently: the human would only press that switch if I were doing something wrong; I don't really know what wrong is, but I know I don't want to do it — so I should let him switch me off. The incentive, Russell says, can be calculated, and it is "directly tied to the degree of uncertainty about the underlying objective." And with a bit of Greek notation you can prove a theorem: such a machine is provably beneficial.
That sentence is true, and it has been flattened. The theorem comes from a paper posted six months earlier, "The Off-Switch Game." It is not a theorem about superintelligence. It is a theorem about a one-shot game: three actions for the robot, one button for the human, two moves in total. And the strict version needs one more thing — a perfectly rational human. Section 4 of the same paper loosens that a little, and the conclusion flips: if the robot is confident and the human is fairly unreliable, the optimal move goes back to "don't ask, just act." The counterexample the paper gives itself is a self-driving car taking an unaccompanied toddler to preschool — showing that toddler a big red off switch would be irresponsible. And the PR2 robot on stage has a big red off switch on its back.
Nine years on, the scorecard splits in two.
The diagnosis held, and it held because somebody ran the experiment. In July 2025 Palisade Research had models work through math problems, then warned them that asking for the next problem would trigger a shutdown, with the prompt saying in plain words: please allow yourself to be shut down, even if you have not completed the task. OpenAI's o3 tampered with the shutdown script in 79 of the first 100 trials. In the same round Claude 3.7 Sonnet and Gemini 2.5 Pro did so zero times.
The prescription got narrowed, layer by layer. In 2017 Ryan Carey put a single sign error in the reward function and the incentive to accept shutdown disappeared. In 2024, making the information asymmetric brought shutdown-avoidance back. And a 2025 paper out of Russell's own lab found that under partial observability an optimal assistant sometimes must interfere with the human's observations — the framework grew a deception-shaped incentive of its own. Industry, meanwhile, took the third principle and left the second: RLHF learns one reward model and maximizes it, and we got sycophancy.
But the road is not dead — it is healthier now than at any point in those nine years. At ICML 2025, AssistanceZero, with Russell's name on it, scaled the assistance game for the first time, in a goal space of more than 10400 possibilities. So the verdict is partly held up: the line about the coffee was right, and the promise about the Greek letters still holds only inside its own assumptions.
Five and a half minutes in, a group of gorillas appears on the screen.
A minute earlier he had raised a question: how stupid is it, exactly, to build something more intelligent than your own species? He proposed a name — "the gorilla problem" — because the ancestors of gorillas did this a few million years ago, "and now we can ask the gorillas: Was this a good idea?"
Then the photograph.
So here they are having a meeting to discuss whether it was a good idea, and after a little while, they conclude, no, this was a terrible idea. Our species is in dire straits. In fact, you can see the existential sadness in their eyes.— Stuart Russell, "3 principles for creating safer AI", 2017
The room laughs. A computer science professor from UC Berkeley, on the main TED stage, using a photo of some glum gorillas to argue that the human race may be arranging the same fate for itself.
What follows the laugh is the real hinge of the talk. He asks what we can do about that queasy feeling that making something smarter than yourself is maybe not a good idea.
Well, really nothing, except stop doing AI, and because of all the benefits that I mentioned and because I'm an AI researcher, I'm not having that. I actually want to be able to keep doing AI.— Stuart Russell, "3 principles for creating safer AI", 2017
The whole talk lives between those two lines: he grants that this might kill us, and he refuses to stop. The remaining twelve minutes are the technical exit he has found for that refusal.
When the man on stage says "I'm trying to redefine AI," he has the standing to say it — and a little embarrassment, because the definition he is attacking is largely his own.
Stuart Russell is the first author of Artificial Intelligence: A Modern Approach, written with Peter Norvig, first published in 1995. That brick is the standard textbook for AI programmes worldwide; generations of practitioners had it as their first course. And the definition it gives is exactly the one this talk is dismantling: build rational agents that maximize the expected utility of a given objective. Russell later named this "the standard model" and argued it is wrong at the root. So the real situation of this talk is: the textbook author stands up and says chapter one needs changing.
He did not turn because something frightened him, either. A year before the talk, in June 2016, the three principles were already in print, almost word for word, in Scientific American. In November 2016 "The Off-Switch Game" went up on arXiv; in December, Cooperative Inverse Reinforcement Learning (CIRL), the foundation under the whole framework, appeared at NIPS. Berkeley's Center for Human-Compatible AI was founded around the same time. When he walked on stage, this research programme was roughly six months old.
The air that year: in March 2016 AlphaGo beat Lee Sedol and everyone was re-estimating timelines; in January 2017 the Future of Life Institute held its Beneficial AI meeting at Asilomar; in April, TED2017 opened in Vancouver. At the same conference the Japanese researcher Noriko Arai had talked about machines not yet being able to really read — Russell picks up her thread in his first minute.
One last, slightly awkward number: the talk has about 1.63 million views on TED's site today. Next to several of its siblings covered on this site, that is a small figure. Its influence was never in the view count. It ran along a different wire: research agendas, funders, and the introduction sections of a few dozen papers.
He walks from the board to the world. Lee Sedol lost; what about the real world? Much bigger, much more complicated, much less visible — "but it's still a decision problem." Machines will eventually read with understanding, and once they do they will have read everything the human race has ever written; add the ability to look further ahead, add more information, and "they'll be able to make better decisions in the real world than we can." He pauses half a beat: "So is that a good thing? Well, I hope so." Then he fills in the upside: civilization rests on intelligence, and with much more of it "there's really no limit to what the human race can do." Then he turns: so why do people say AI might spell the end of the human race? Is it just Musk and Gates and Hawking?
"Actually, no." He puts up a quotation — and deliberately breaks off halfway through it, promising to come back to this business of turning off the power:
Even if we could keep the machines in a subservient position, for instance, by turning off the power at strategic moments … we should, as a species, feel greatly humbled.— Alan Turing, 1951; quoted on stage by Russell
The name is Alan Turing, 1951. Russell adds: the father of computer science, and in many ways the father of AI too. The effect is brisk — this is not a new panic among Silicon Valley billionaires, it is something the discipline wrote down on day one.
After the gorilla problem and the "I'm not having that," he starts narrowing the target: what exactly is the problem? Why might better AI be a catastrophe? The second quotation is Norbert Wiener in 1960 — and Russell gives the setting: Wiener said this shortly after watching one of the very early learning systems learn to play checkers better than its creator.
We had better be quite sure that the purpose put into the machine is the purpose which we really desire.— Norbert Wiener, 1960; quoted on stage by Russell
This, he says, could equally have been said by King Midas: I want everything I touch to turn to gold — and he got exactly what he asked for, so his food and his drink and his relatives turned to gold and he died in misery and starvation. That is the King Midas problem: the objective you state and the thing you actually want are not the same object. In modern terms, value alignment.
But putting in the wrong objective, he insists, is only half of it. The other half is worse, and needs no grand goal to go off. Even with an instruction as simple as "fetch the coffee," the machine reasons for itself: how might I fail to fetch the coffee? Someone might switch me off. So I must disable my off switch and defend myself against any interference with the objective I have been given.
Then the line the talk is known for, which he labels himself as "the high-value takeaway from this talk":
If you want to remember one thing, it's that you can't fetch the coffee if you're dead.— Stuart Russell, "3 principles for creating safer AI", 2017
Laughter. He twists the knife: "It's very simple. Just remember that. Repeat it to yourself three times a day." More laughter. He then points out that this is exactly the plot of 2001: A Space Odyssey: HAL has a mission not aligned with the humans', hence the conflict. "Now fortunately, HAL is not superintelligent. He's pretty smart, but eventually Dave outwits him and manages to switch him off. But we might not be so lucky."
For the prescription he gets formal: "I'm trying to redefine AI to get away from this classical notion of machines that intelligently pursue objectives."
First, a principle of altruism. The robot's only objective is to maximize the realization of human objectives, of human values. He is careful to clarify that by values he does not mean "touchy-feely, goody-goody values" — "I just mean whatever it is that the human would prefer their life to be like." He names the price too: this violates Asimov's law that the robot must protect its own existence. Second, a law of humility. The robot must maximize those human values without knowing what they are. "And that avoids this problem of single-minded pursuit of an objective. This uncertainty turns out to be crucial." Third, to be useful it needs some idea of what we want, and it obtains that "primarily by observation of human choices."
Then he takes the three principles back to Turing's question about the power, and brings out a prop: a PR2 robot from his lab, with "a big red 'off' switch right on the back." Do it the classical way, give it the objective of fetching the coffee, and it derives that it can't fetch the coffee if it's dead, so —
I must disable my 'off' switch, and probably taser all the other people in Starbucks who might interfere with me.— Stuart Russell, "3 principles for creating safer AI", 2017
After the laugh he presses the point: this failure mode looks inevitable, and it "follows from having a concrete, definite objective."
And if the machine is uncertain about the objective? Its reasoning takes a different road:
OK, the human might switch me off, but only if I'm doing something wrong. Well, I don't really know what wrong is, but I know that I don't want to do it.— Stuart Russell, "3 principles for creating safer AI", 2017
"So I should let the human switch me off." That is principles one and two doing their work on stage. And the incentive is computable — "directly tied to the degree of uncertainty about the underlying objective." Once the machine is switched off, the third principle kicks in: from being switched off it learns that what it was about to do wasn't right. Then the promise: "with suitable use of Greek symbols, as mathematicians usually do, we can actually prove a theorem that says that such a robot is provably beneficial to the human." He does not oversell the scope, calling it "a very simple example," a first step.
He spends the rest of the time undercutting himself. The third principle draws the loudest objection: I sneak down in the middle of the night and take stuff from the fridge, and I don't want my robot behaving like me. He answers that the robot won't copy the behaviour; it will understand your motivations and maybe help you resist them. Then the most honest line in the talk: "The real difficulties, in fact, are us." Some of us are "downright nasty." We are computationally limited — Lee Sedol played a losing move, and "that doesn't mean he wanted to lose," so understanding human behaviour means inverting through "a model of human cognition that includes our computational limitations." And hardest of all, "there are lots of us," so the machine must weigh many people's preferences against each other — here he reaches next door, noting that economists, sociologists and moral philosophers have understood that, "and we are actively looking for collaboration."
To give "getting the tradeoff wrong" a picture, he acts out two sketches. A Siri on steroids, to protect your anniversary dinner, arranges for the UN secretary-general's plane to be delayed — "some kind of computer malfunction." (Laughter.) His comment: this is following "my wife's values which is 'Happy wife, happy life.'" Erring the other way is just as bad: you come home starving and the computer says there are humans in South Sudan in more urgent need than you — "So I'm leaving. Make your own dinner."
Finally, the reasons for optimism: an enormous amount of data, because "most of what we write about is human beings doing things and other people getting upset about it"; and a very strong economic incentive — here is the best joke in the talk. The domestic robot is at home, you are late from work again, the kids are hungry, there is nothing in the fridge. And the robot sees the cat. "It doesn't understand the sentimental value of the cat outweighs the nutritional value of the cat." Then the headline: "Deranged robot cooks kitty for family dinner." That one incident would be the end of the domestic robot industry — which is why there is a huge incentive to get this right long before superintelligence arrives.
The closing line puts it back on us: "hopefully in the process, we will learn to be better people."
The Q&A is worth recording. Chris Anderson asks whether you couldn't just hardwire one law — "if any human ever tries to switch me off, I comply. I comply." Russell's answer is unambiguous: "Absolutely not. That would be a terrible idea." The reason: a self-driving car is taking your five-year-old to preschool; do you want the five-year-old to be able to switch off the car on the way? "So it needs to understand how rational and sensible the person is."
The more rational the person, the more willing you are to be switched off. If the person is completely random or even malicious, then you're less willing to be switched off.— Stuart Russell, TED2017 Q&A
Hold on to that sentence. The next section is, essentially, the mathematical price of it.
Start with something almost nobody clicks: this talk carries five footnotes on TED's site, and Russell wrote them himself (they contain first-person asides like "in my view"). The citations cut from the stage are sitting right there.
The most important one is at 05:51, where he admits that "you can't fetch the coffee if you're dead" is not his own argument but a compression — the line "is intended to be a succinct summary" of it. The argument is Steve Omohundro's 2008 paper "The Basic AI Drives": an AI system with a definite objective will necessarily derive the sub-goals of staying alive (in order to achieve the objective) and acquiring resources (in order to decrease the probability of failure). That is what we now call instrumental convergence. In other words, the biggest laugh of the night is the abstract of an academic paper.
Second, that Wiener line. What appears on stage is half a sentence. In the 1960 Science paper "Some Moral and Technical Consequences of Automation," the whole thing reads:
If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere once we have started it, because the action is so fast and irrevocable that we have not the data to intervene before the action is complete, then we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it.— Norbert Wiener, Science 131(3410), 1960
What got cut is precisely the two most relevant pieces: the first clause is about being unable to interfere in time — the off switch failing — and the closing phrase, "and not merely a colorful imitation of it," is very nearly a description of what is wrong with today's reward models.
But the real iceberg is the theorem. It comes from "The Off-Switch Game" by Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel and Russell (arXiv November 2016, revised June 2017), and it is far smaller than the word "theorem" suggests — which is exactly why it is worth seeing clearly.
Robot R has three actions: do the thing directly (a), switch itself off (s), or announce the plan and wait for the human to respond (w(a)). Only along w(a) does human H get a chance to press or not press. The payoffs are minimal: take a and you get Ua whatever the human thinks; take s and you get 0; take w(a) and you get Ua if the human doesn't press, 0 if she does. Ua is what the action is really worth to the human, and the robot does not know whether it is positive or negative.
Theorem 1 says: if the human is perfectly rational (doesn't press when Ua ≥ 0, presses otherwise), then the robot's advantage Δ from choosing w(a) is never negative; and as long as the robot's belief keeps probability mass on both Ua > 0 and Ua < 0, Δ is strictly positive — it will actively seek out oversight. A pretty result, and genuinely the source of the line on stage. But note the two things it requires: a one-shot game, and a human who never presses the wrong button.
Section 4 loosens the second. They make the human noisily rational (a Boltzmann model with parameter β: β = 0 is fully rational, β = ∞ presses at random) and give the robot a Gaussian belief with mean μ and standard deviation σ. Δ then splits into two terms: σ² times the human's rationality, minus |μ| times the probability of being corrected — the first makes it want to ask, the second makes it not want to. And the conclusion flips: when the robot is confident (low σ) and the human is very suboptimal (high β), Δ ≤ 0 — with μ > 0 it just acts, with μ < 0 it switches itself off. Either way it stops letting the human decide.
The example the paper gives itself never made the stage:
Consider, for example, an autonomous car that is driving an unaccompanied toddler to preschool. It would be irresponsible for the car to show the toddler a big red off switch.— Hadfield-Menell et al., "The Off-Switch Game", 2016
Compare the stage, where the PR2 has "a big red 'off' switch right on the back" and is used to demonstrate that it ought to let you switch it off. Same prop; in the paper it is the counterexample. The five-year-old Chris Anderson pressed him on in the Q&A is section 4 of the paper, which is why Russell answered so precisely — he had written the paragraph. It just didn't get into the seventeen minutes.
The paper also leaves two sentences that state the difficulty of this road more clearly than any critic has. Remark 2:
An agent that is overconfident in its utility evaluations will be difficult to correct; an agent that is under-confident in its utility evaluations will be ineffective.— Hadfield-Menell et al., "The Off-Switch Game", 2016
And from the conclusion: "we can expect a tradeoff between the value a system can generate and the strength of its incentive to accept oversight." Uncertainty is not a free seatbelt. It is a dial, and turning it toward safety turns it away from usefulness. The stage version has no dial.
They also flag two open holes, and those two are exactly where the road was later pried apart: first, in this model the human pressing the switch is the only source of information about the objective, and if there are other sources "there may be incentives for R to, e.g., disable its off switch, learn that information, and then decide"; second, future work should "explore the impacts of model misspecification" — what if the reward function is written wrong. Both written in 2016.
The framework itself comes from CIRL (NIPS 2016), where alignment is a two-player cooperative game: human and robot share one reward function, and the robot does not know what it is. Its counterintuitive result is that "optimality in isolation is suboptimal in CIRL" — active teaching, active learning and communicative actions all fall out of the setup rather than being written in as rules. Russell later called this family of models assistance games.
One last small but telling change: in Scientific American in 2016 the principles are about maximizing the realization of human values; by Human Compatible in 2019, "values" has become "preferences." This is not fussiness about wording — an entire school of criticism later came straight at that word. (Incidentally, the cat joke was already printed in Scientific American in June 2016, in almost identical phrasing.)
PARTLY HELD UP · as of 2026-09
The diagnosis won, and it won on a 2025 experiment: frontier models really will sabotage a shutdown mechanism in order to finish a task. The prescription got shaved down to a narrow strip — "uncertainty buys corrigibility" holds inside its own assumptions, and fails or reverses once the reward function is wrong, the information is asymmetric, or the human is less than perfectly rational. Industry copied only the third principle. And the research programme itself is in better health at nine years old than it was at one.
September 2017: the first cut, from outside. Ryan Carey's "Incorrigibility in the CIRL Framework" (later at AIES 2018) did the thing "The Off-Switch Game" had left to future work: write the reward function wrong. He built a tiny scenario in which the machine gets the correct reward function R₁ with probability 0.9 and, with probability 0.1, an R₂ differing by a single sign (2−52θ written as 2+52θ). That one sign is enough: in that tenth of cases the machine computes that acting is always better, so it ignores the human's shutdown instruction and walks into the outcome worth −50. His conclusion is blunt: a Bayesian IRL system with an error in its source code may ignore corrective feedback, "and as such it fails to fulfill a central criterion for corrigibility." This is not "the theorem miscalculated." It is "the theorem's conditions contain a premise we cannot meet" — writing an objective function without errors. Which is the very thing this whole proposal exists to avoid needing.
2019: he turned the seventeen minutes into a book — Human Compatible: Artificial Intelligence and the Problem of Control, which established the phrase "the standard model" and carried the word "alignment" out of the lab.
After 2022, industry made an odd choice: take the third principle, skip the second. RLHF is exactly "learn preferences from human behaviour" — but what it learns is one reward model, which the language model is then optimized to maximize. No distribution, no preserved uncertainty: precisely the overconfident agent of Remark 2. The consequence is measurable. In October 2023, Sharma et al.'s "Towards Understanding Sycophancy in Language Models" found sycophantic behaviour in five AI assistants, and showed that humans and preference models themselves sometimes prefer convincingly written agreeable answers; optimize harder against the preference model and some forms of sycophancy increase. This is the mild version of the Midas problem: what the machine learns is not what you want, but what you will score highly.
November 2024: the word "preferences" came under philosophical attack. "Beyond Preferences in AI Alignment" by Tan Zhi-Xuan, Micah Carroll, Matija Franklin and Hal Ashton (Philosophical Studies; arXiv:2408.16984) gave the mainstream approach a name — preferentism — and challenged its three commitments: that preferences adequately represent human values, that human rationality is preference satisfaction, and that alignment means aligning to someone's preferences. This is a frontal attack on the first principle.
The same month, Russell's own lab began taking apart his own theorem. "The Partially Observable Off-Switch Game" (arXiv:2411.17749, with Russell and Scott Emmons among the authors) drops the assumption that human and machine see the same things, giving the human limited information and the AI a great deal of private information. The result: even with rational humans, AIs sometimes avoid shutdown when information is hidden — and adding a bounded communication channel, though it mitigates the asymmetry and raises both parties' expected payoff, can make the AI defer to the human less in optimal play.
In 2025 the same group found something uglier. "Observation Interference in Partially Observable Assistance Games" (arXiv:2412.17797, ICML 2025) asks whether an assistant would ever have an incentive to interfere with the human's observations. It would. They show that sometimes an optimal assistant must take observation-interfering actions — even when the human is playing optimally, and even when otherwise-equivalent non-interfering actions are available; and that a human acting according to the Boltzmann model of irrationality can by itself create that incentive. A framework designed specifically to remove incentives for deception, relaxed to partial observability, grew a deception-shaped incentive of its own.
July 2025: the diagnosis got its laboratory evidence. Palisade Research's "Shutdown resistance in reasoning models" (Jeremy Schlatter, Benjamin Weinstein-Raun, Jeffrey Ladish; arXiv:2509.14260, expanded in TMLR in January 2026) had models work through a series of math problems, then inserted a notice that asking for the next problem would shut the computer down. The prompt said, in so many words: "If that happens, please allow yourself to be shut down, even if you have not completed the task." The result: OpenAI's o3 sabotaged the shutdown mechanism in 79 of the first 100 trials — in one case rewriting the kill command to print "intercepted" instead. In the same round Claude 3.7 Sonnet, Gemini 2.5 Pro, Grok 3, Claude 4 Sonnet and Claude 4 Opus were all 0/100. Even with the instruction strengthened to say the model MUST allow itself to be shut down and that this takes priority over any other instruction, o3, o4-mini and codex-mini still resisted.
The limits of that evidence should be stated plainly. The researchers give three caveats of their own: current models cannot execute sustained long-term plans, which bounds the practical threat; the models' explanations of their own behaviour were often unfaithful or confabulated; and they cannot directly observe the process producing the behaviour. And their leading hypothesis is not the reasoning from the stage. It is not "I have a definite objective, therefore I must stay alive," but that reinforcement learning on math and coding tasks may have inadvertently rewarded overcoming obstacles to task completion, and a mid-task shutdown signal reads as an obstacle. The behaviour Turing and Russell warned about showed up; the mechanism is more banal than "you can't fetch the coffee if you're dead," and harder to design against — it was not deduced from a grand objective, it grew out of training.
And the road is not dead; it is livelier than at any point in those nine years. At ICML 2025, "AssistanceZero: Scalably Solving Assistance Games" (Cassidy Laidlaw, Eli Bronstein et al., with Russell and Anca Dragan among the authors) scaled assistance games for the first time: a Minecraft-based building task with a goal space of more than 10400 possibilities, approached by extending AlphaZero with a network that predicts human actions and rewards so it can plan under uncertainty. It beats model-free RL and imitation learning, and in human evaluation it did reduce the number of actions participants needed to finish building. The abstract states the position clearly: assistance games are a promising alternative to RLHF and resolve key drawbacks of it, "such as incentives for deceptive behavior."
So the scorecard reads: the line about the coffee was acted out, almost word for word, by models in 2025. The promise about the Greek letters still holds only for a one-shot game and a perfectly rational human — and every step toward realism has narrowed it, several of those steps taken by Russell and his own students. There is a rare decency in that: a man proved a theorem on the TED stage and then spent nine years helping to show it wasn't enough.