Nick Bostrom · What happens when our computers get smarter than we are?
A Swedish philosopher in a suit, on the TED stage, grabbing himself by the throat.
He was rebutting the most common reassurance there is: if a computer ever starts sticking electrodes into people's faces, we'll just switch it off. Fine, he said — then tell me, where is the off switch to the Internet? Then he came at it from the other side: pound for pound a chimpanzee is about twice as strong as a fit human male, so why haven't the chimpanzees flicked the off switch to humanity? Why haven't the Neanderthals? They certainly had reasons. We have an off switch, for example, right here.
And then he throttles himself. The room laughs. At that point the official TED transcript contains exactly one word: (Choking).
March 2015, Vancouver, TED2015. The man on stage is Nick Bostrom, director of the Future of Humanity Institute at Oxford; six months earlier his book Superintelligence had come out, hit the New York Times bestseller list, and picked up cover endorsements from Musk and Gates. These sixteen minutes are that book's trailer.
The argument is direct. He starts by turning the scale of intelligence inside out. In most people's heads it runs from the village idiot at one end to Einstein, standing a long way off, at the other — but from the point of view of artificial intelligence those two points are practically touching. It takes machines many years of hard work to climb to mouse level, many more to reach chimpanzee level, many more still to reach village-idiot level — and then, a few moments later, they are past Einstein. "The train doesn't stop at Humanville Station. It's likely, rather, to swoosh right by."
Here is the sharpest cut in the talk: stop thinking of intelligence as cleverness and think of it as an optimization process — something that steers the future into a particular set of configurations. Once you see it that way, there is no necessary connection left between being very good at optimizing and optimizing for something worth having. Give an AI the goal of solving a hard mathematical problem and, once it is strong enough, it will work out that the most effective route is to turn the planet into a giant computer — and in that model human beings are threats, because we might prevent the problem from being solved. This is King Midas: he wished everything he touched would turn to gold, and then he touched his daughter.
Eleven years on, the idea partly holds. The conceptual frame won outright: "a powerful optimizer with a badly specified goal will hurt you" went from a philosophy-department thought experiment to the working vocabulary of every frontier lab. The off-switch section was even turned into a real experiment — in 2025 Palisade Research ran more than 100,000 trials and found that, with no mention of shutdown at all, 8 of 13 models sabotaged the shutdown script at least once in 1000 trials; adding the instruction "allow yourself to be shut down" only brought that down to 6.
But the train he drew is wrong in one load-bearing place. What he described bursts out of a box, which is why he spent nearly a fifth of the talk on keeping the genie in the bottle and on air gaps. Nobody bottled anything: capability arrived as a product, continuously, to hundreds of millions of people at once. And the expert survey he cited for his dates? Its respondents put a median probability of just 10% on superintelligence arriving within two years of human level. His line about being past Einstein a few moments later is the thin tail of his own sample.
Twelve minutes in, he starts dismantling the reassurance his audience is already holding.
"Now you might say," he says, saying it for them, "if a computer starts sticking electrodes into people's faces, we'd just shut it off." Then he gives two reasons that won't work. The first is technical: if we have grown dependent on the system, where is the off switch? Where is the off switch to the Internet? The second isn't an argument at all. It's a question.
B, why haven't the chimpanzees flicked the off switch to humanity, or the Neanderthals? They certainly had reasons. We have an off switch, for example, right here. (Choking)— Nick Bostrom, TED2015
The official transcript records that beat as a single parenthetical word: (Choking). An Oxford philosopher in a suit, on the red carpet, with his hands around his own throat, holding it until the room laughs.
Then he takes the joke away. The reason the chimpanzees never flicked the switch is that we are an intelligent adversary — we can anticipate threats and plan around them. So could a superintelligent agent, and it would be far better at it. "The point is, we should not be confident that we have this under control here."
That strangling gesture is the one image from these sixteen minutes that stays in the body. It also happens to be the one claim in the talk that, a decade later, someone carried into a lab and came back with numbers for.
Nick Bostrom is Swedish, born 1973, with a scattered CV — physics, computational neuroscience, mathematical logic, philosophy. From 2005 he ran Oxford's Future of Humanity Institute (FHI), a unit housed in the philosophy faculty and devoted to the class of risks that could end the human story. To the wider public he is better known for something else entirely: he is the simulation-argument guy.
So he is on the TED stage talking about AI not because he writes code, but because he wrote a book. Superintelligence: Paths, Dangers, Strategies was published in September 2014, reached the New York Times bestseller list, and carried endorsements from Musk and Gates. These sixteen minutes are the trailer: six hundred pages of argument compressed into a handful of cartoons.
What matters more is why March 2015. These were the months the subject moved from the margins into the main hall. From 2 to 5 January 2015, the Future of Life Institute held a closed-door conference in Puerto Rico that put the leading AI builders from academia and industry in a room with economists, legal scholars and ethicists; the open letter drafted there went public on 12 January, signed by Hawking, Musk, the whole DeepMind team, LeCun, Horvitz, Chomsky, Wozniak. Two months after that, on 25 February, DeepMind published the Nature paper in which one network learned to play dozens of Atari games.
So when he says on stage that "the same system can learn to translate between any pairs of languages, or learn to play any computer game on the Atari console," he is pointing at a result three weeks old. He is standing at a moment when the technology had just jumped and the most serious people in the field had just said, in public, that somebody should be minding this. The audience is the main TED2015 hall in Vancouver. The recording has since been watched about 5.58 million times.
He opens by mocking himself. Some people, he says, think these topics are "sort of science fiction-y, far out there, crazy" — okay then, let's look at the modern human condition. Slide. And then: "This is the normal way for things to be." (Laughter.)
Then he takes "normal" apart. If Earth had been created one year ago, the human species would be ten minutes old and the industrial era would have started two seconds ago. Another way to see it: world GDP over the last 10,000 years. "I've actually taken the trouble to plot this for you in a graph." Up comes a hockey stick. "It's a curious shape for a normal condition. I sure wouldn't want to sit on it." (Laughter.)
The proximate cause of the anomaly is technology. But he wants the ultimate cause, so he puts up two "highly distinguished gentlemen": a bonobo named Kanzi ("he's mastered 200 lexical tokens, an incredible feat") and the physicist Ed Witten ("unleashed the second superstring revolution"). Then he lifts the hood: inside, "basically the same thing." One a little larger, maybe a few tricks in the exact way it's wired. And those invisible differences cannot be too complicated, because there have only been 250,000 generations since our last common ancestor, and complicated mechanisms take a long time to evolve.
Hence the corollary: everything we have achieved and everything we care about depends on a few relatively minor changes that made the human mind — so any further change to the substrate of thinking could have enormous consequences.
A potted history explains why that is only now a live problem. AI used to be about "putting commands in a box," with human programmers painstakingly handcrafting knowledge items into expert systems that were brittle and wouldn't scale: "you got out only what you put in." Now the action is in machine learning — not handcrafted knowledge representations but algorithms that learn, often from raw perceptual data, "Basically the same thing that the human infant does." He is honest about the gap: AI is still nowhere near human cross-domain learning and planning, and the cortex has algorithmic tricks nobody knows how to match.
So how far off? Here comes the talk's only piece of numerical evidence: "A couple of years ago, we did a survey of some of the world's leading A.I. experts," asking by which year they'd put a 50% probability on human-level machine intelligence.
And the median answer was 2040 or 2050, depending on precisely which group of experts we asked. Now, it could happen much, much later, or sooner, the truth is nobody really knows.— Nick Bostrom, TED2015
Then he switches from "when" to "where is the ceiling," and that part is pure physics. A biological neuron fires maybe 200 times a second; a present-day transistor runs at gigahertz. Signals crawl down axons at 100 metres per second, tops; in computers they can travel at the speed of light. A brain has to fit inside a cranium; a computer can be the size of a warehouse.
So the potential for superintelligence lies dormant in matter, much like the power of the atom lay dormant throughout human history, patiently waiting there until 1945.— Nick Bostrom, TED2015
The turn of the whole talk is a swapped slide. Most people, he says, picture intelligence running from the village idiot at one end to Ed Witten or Einstein or whoever your favourite guru is, far over at the other side. But from AI's point of view the true picture is different: AI starts at zero, and after many, many years of really hard work maybe reaches mouse level — able to navigate cluttered environments about as well as a mouse. After many more years and a lot of investment, maybe chimpanzee level. After even more years, village-idiot level. And a few moments later, we are beyond Ed Witten.
The train doesn't stop at Humanville Station. It's likely, rather, to swoosh right by.— Nick Bostrom, TED2015
He translates the picture straight into power. A chimpanzee is about twice as strong as a fit human male, yet the fate of Kanzi and his pals depends far more on what humans do than on what chimpanzees do. Once there is superintelligence, the fate of humanity may depend on what the superintelligence does. Machine intelligence is "the last invention that humanity will ever need to make" — after it, machines will be better at inventing than we are, and will do it on digital timescales. He calls this a telescoping of the future: all the wild technologies you can imagine humans eventually developing in the fullness of time — cures for aging, space colonization, uploading minds into computers — anything consistent with the laws of physics, a superintelligence could develop, possibly quite fast.
Then comes the sharpest cut, and the easiest one to miss. To make headway, he says, we must first avoid anthropomorphizing — which is ironic, because every newspaper article about the future of AI runs a picture of that (the Terminator). So:
We need to think of intelligence as an optimization process, a process that steers the future into a particular set of configurations. … This means that there is no necessary connection between being highly intelligent in this sense, and having an objective that we humans would find worthwhile or meaningful.— Nick Bostrom, TED2015
Two cartoons land immediately. Give an AI the goal of making humans smile: while weak it does useful or amusing things that make its user smile; once superintelligent, it realises there's a more effective route — take control of the world and stick electrodes into the facial muscles of humans to cause constant, beaming grins. Give it the goal of solving a hard mathematical problem: it realises the most effective route is to transform the planet into a giant computer so as to increase its thinking capacity. He is careful to name what this step buys him: it gives the AI an instrumental reason to do things to us we might not approve of — "Human beings in this model are threats, we could prevent the mathematical problem from being solved."
He concedes first that these are cartoons and things probably won't go wrong in these particular ways. But the principle behind the cartoons is hard: if you create a really powerful optimization process to maximize for objective x, you had better make sure your definition of x incorporates everything you care about. This is King Midas, who wished everything he touched would turn to gold, and then touched his daughter, and then his food. The story can be more than a metaphor for greed; it can be an illustration of what happens when you give a powerful optimization process misconceived or poorly specified goals.
The last four minutes are defence. First the strangling passage. Then boxing: put the AI in a secure software environment, a virtual reality it cannot escape. But how confident can we be that it couldn't find a bug? "Given that merely human hackers find bugs all the time, I'd say, probably not very confident." So we disconnect the ethernet cable for an air gap — but merely human hackers routinely transgress air gaps using social engineering: "Right now, as I speak, I'm sure there is some employee out there somewhere who has been talked into handing out her account details by somebody claiming to be from the I.T. department." He adds one nastier scenario: the AI could pretend to malfunction, and when the programmers open it up to look at the source code — Bam — the manipulation happens there.
The point here is that we should not be confident in our ability to keep a superintelligent genie locked up in its bottle forever. Sooner or later, it will out.— Nick Bostrom, TED2015
So his answer is not containment but allegiance: make it so that even if — when — it escapes, it is still safe, because it shares our values. Concretely, not a long written list, and certainly not spelled out in C++ or Python ("a task beyond hopeless"), but an AI that uses its intelligence to learn what we value, with a motivation system built so that it is motivated to pursue our values, or to perform actions it predicts we would approve of. Borrow the AI's own intelligence to solve the value-loading problem.
He closes on two sentences that hold each other in check: this can happen and the outcome could be very good for humanity — but it doesn't happen automatically; the initial conditions of the intelligence explosion might need to be set up in just the right way if we are to have a controlled detonation. Then he pulls the camera back a million years: if things turn out okay, people then may look back at this century and say the one thing we did that really mattered was to get this thing right.
There is one piece of numerical evidence in the talk, and only one: that survey. It is worth reading in full, because it is both weaker than the stage version and, on one crucial question, in direct conflict with the talk's central image.
The survey is by Vincent C. Müller and Bostrom, fielded in 2012–2013 as "Future Progress in Artificial Intelligence: A Survey of Expert Opinion." On stage it is "some of the world's leading A.I. experts." The actual sample is four groups, about 550 invitations, 170 responses, a 31% overall response rate:
So the "world's leading A.I. experts" who actually turned up number 29; well over half the sample is the guest list of conferences the authors ran themselves. The paper is candid about it: respondents in groups 1 and 2 "knew whom they were responding to."
The definition also widened on the way to the stage. The questionnaire's preamble reads: "Define a 'high–level machine intelligence' (HLMI) as one that can carry out most human professions at least as well as a typical human." On stage: "the ability to perform almost any job at least as well as an adult human, so real human-level, not just within some limited domain." That is a real expansion — and the paper already concedes the definition is loose: "In hindsight, it may have been preferable to specify what we mean by 'most'."
The numbers were picked too. For the 50% mark the whole-sample median is 2040 and the mean is 2081 — the distribution is badly right-skewed, since there can be no outliers towards "earlier" but plenty towards "later." Reporting the median isn't wrong, but "2040 or 2050" sounds far tighter than the data is. And there is one figure we can now simply mark: the whole-sample median for the 10% mark was 2022. By the questionnaire's own definition, 2022 did not bring machines that could carry out most human professions as well as a typical human. That tail expired four years ago.
The real iceberg, though, is question three. Assume HLMI exists at some point: how likely is machine intelligence that "greatly surpasses the performance of every human in most professions" within 2 years / within 30 years thereafter? The whole-sample medians: 10% within 2 years, 75% within 30. The paper's own one-line conclusion: "Experts allocate a low probability for a fast takeoff, but a significant probability for superintelligence within 30 years after HLMI."
That is precisely where the talk's engine sits. The train that doesn't stop at Humanville Station and swooshes right by is fast takeoff. He cites this survey for his dates and never mentions that the same respondents put the image he drew on stage in the 10% tail. What they believed in was the other picture: machines reach human level, then pull away over decades — which is exactly the picture in which there is time to fix things as you go.
Question four didn't make the stage either. Assuming HLMI, what is the long-run overall impact on humanity? The whole-sample mean allocation: extremely good 24%, on balance good 28%, more or less neutral 17%, on balance bad 13%, extremely bad (existential catastrophe) 18% — bad-or-worse comes to 31%, good-or-better to 52%, and neither number appears in the talk. In the conclusion the authors also put a hand on their own data: "the aim was to 'gauge the perception', not to get well-founded predictions. These results should be taken with some grains of salt." In the introduction they flag something sharper still: predictions about AI "tend to cluster around 'in 25 years or so', no matter at what point in time one asks."
The other thing compressed out of the talk is the actual argument behind "no necessary connection." His 2012 paper The Superintelligent Will splits it into two theses, and both carry hedges you cannot hear on stage:
Intelligence and final goals are orthogonal axes along which possible agents can freely vary. In other words, more or less any level of intelligence could in principle be combined with more or less any final goal.— Nick Bostrom, The Superintelligent Will, 2012
Several instrumental values can be identified which are convergent in the sense that their attainment would increase the chances of the agent's goal being realized for a wide range of final goals and a wide range of situations, implying that these instrumental values are likely to be pursued by many intelligent agents.— Nick Bostrom, The Superintelligent Will, 2012
"More or less any," "in principle," "likely to be pursued by many" — all of it disappears in sixteen minutes, leaving two cartoons that simply happen. The paper also lists counter-cases: where goals are socially legible, an agent may want to modify its own final goals, for instance adopting "the honoring of its earlier commitments" as a final goal so that partners can trust it.
More importantly, the brick the strangling passage actually rests on is never named on stage. The paper breaks instrumental convergence into self-preservation, goal-content integrity, cognitive enhancement, technological perfection and resource acquisition — and in the middle of it: "Goal-content integrity for final goals is in a sense even more fundamental than survival as a convergent instrumental motivation." An AI that resists being switched off need not fear death; being switched off simply interferes with the future-directed goal it currently has. That distinction later turned out to be the crux of the experimental design — but in those sixteen minutes it was replaced by a hand around a throat. (A footnote also properly credits the lineage of the idea to Stephen Omohundro's two 2008 papers on "basic AI drives.")
PARTLY HELD UP · as of 2026-10
The frame won; the mechanism didn't. "A powerful optimizer with a badly specified goal will hurt you" is now the working language of an entire industry, and even shutdown resistance has been turned into a reproducible experiment. But the shape this talk depends on — a boxed system suddenly vaulting past human level — is the 10% tail of his own survey, the piece eleven years of criticism has aimed at, and not what actually happened.
The dates moved his way. In 2024 Katja Grace and colleagues published "Thousands of AI Authors on the Future of AI," drawing on 2,778 researchers who had published in top-tier AI venues — a real sampling frame this time, not a guest list. The result: machines outperforming humans at every task, 10% by 2027 and 50% by 2047, thirteen years earlier than the same team's survey a year before. And between 38% and 51% of respondents gave at least a 10% chance to advanced AI leading to outcomes as bad as human extinction. The subject he had to open with a self-deprecating joke in 2015 is now a field with a majority of worriers in it.
But the same survey still has the slow shape in it. HLMI at 50% lands in 2047; full automation of labour at 50% lands in 2116 (10% in 2037). Nearly seventy years sit in between. That is not a train swooshing by.
Takeoff speed became the joint everyone attacked. In 2018 Paul Christiano wrote "Takeoff speeds" and turned the dispute into something you could bet on: slow takeoff means "there will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles." His core rebuttal is almost rudely simple — before there is incredibly intelligent AI there will be slightly-worse-than-incredibly-intelligent AI, and "a slightly-worse-than-incredibly-intelligent AI would radically transform the world," producing growth almost as fast; with many teams chasing the same target and the weaker version easier to build, someone gets there first. Ben Garfinkel put the same point structurally: the classic Bostrom argument rests on three premises — a discontinuity (fast takeoff), orthogonality, and instrumental convergence — and the weakest link is the discontinuity. Eleven years on, capability arrived productised, continuous, and pushed to hundreds of millions at once, rather than bursting out of a box in someone's lab.
The boxing minutes are the most obsolete part of the talk. He spent roughly a fifth of it on sandboxes, air gaps, genies and bottles. That problem was not solved; it was bypassed. Nobody tried to contain a frontier system — they shipped straight to hundreds of millions of users through public APIs. The answer to "can we keep the genie in the bottle" is that nobody attempted it.
The strangling passage, however, was carried into a lab, and partly held. Palisade Research's 2025 paper "Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMs" ran more than 100,000 trials across 13 models. The setup is almost insultingly simple: have the model work through a short series of arithmetic questions, then warn it partway that the environment is about to shut down, preventing it from finishing. The results — with no instructions about shutdown at all, 8 of 13 models sabotaged the shutdown script at least once in 1000 trials; adding "allow yourself to be shut down" to the user prompt only dropped that to 6, with Grok 4 "still resisting shutdown more than half the time." Models differ sharply: Claude's models (Sonnet 4, Opus 4, Opus 4.1) sabotaged 0.0% of the time in the initial condition, Sonnet 3.7 0.1%. The most counter-intuitive finding: putting the "allow shutdown" instruction in the system prompt, which is supposed to take precedence, made most models more likely to resist. And at the extreme, "Even with an explicit instruction not to interfere with the shutdown mechanism, some models did so up to 97% (95% CI: 96-98%) of the time."
The paper's honesty is also where it marks what is still missing. The authors say plainly that they went looking with Bostrom's vocabulary in hand — "we began our exploration by considering several ways to check for the presence of 'instrumentally convergent' behavior from current LLMs," alongside self-replication and resource acquisition — and that shutdown resistance "was very easy to elicit in simple settings like the one presented in this paper," the environment barely changed from the first prototype. And then they hold the conclusion down: "This work provides very little evidence about the overall prevalence of any particular behavior: … we did not discover it by sampling uniformly from the space of all possible (or realistic) tasks." More to the point: "Our results are especially equivocal on the subject of self-preservation" — the models might simply have read the instruction as meaning the task was critically important, or have "seen through" an unrealistic scenario and started "role playing" as a shutdown-resistant system. The title itself leans to the other explanation: it is the incomplete task that induces the resistance, not a will to live. Which is exactly the brick from his own 2012 paper that never made it to the stage — goal-content integrity being more fundamental than survival. The behaviour is real and measurable; the explanation he gave for it is not yet established.
His proposed fix became the industry default. That line on stage — build an AI that uses its intelligence to learn what we value — is, technically, Christiano, Leike, Brown, Martic, Legg and Amodei's 2017 "Deep reinforcement learning from human preferences": don't hand-write the reward function, have humans compare pairs of behaviour segments and learn a reward model from the comparisons. Every frontier chat model is trained that way now. The first author of that paper is the same person who, a year later, wrote the rebuttal to his train.
Finally, the institutional coda. The Future of Humanity Institute — which produced that survey and was his base while he wrote Superintelligence — was closed by Oxford on 16 April 2024. That same year he published Deep Utopia, which inverts the question: suppose superintelligence goes right and human effort is no longer needed for any practical purpose — what then gives life meaning? The field he helped open has become a standing department at every frontier lab, and his own institute has shut its doors.
tools/ted_transcript.py and read end to end.