Nick Bostrom’s Fable of the Dragon-Tyrant asks us to imagine a dragon that devours ten thousand people every evening. Everyone regards this as tragic but natural, until someone proposes killing the dragon. The dragon is a metaphor for death by aging: enormous suffering and death treated as inevitable largely because they have always been with us.
Bostrom has since carried the fable into AI policy. His 2026 working paper Optimal Timing for Superintelligence takes the standpoint of people alive today and argues that building superintelligence resembles risky surgery for a condition that will otherwise kill the patient. On his models, even high probabilities of catastrophe are often worth accepting.
roon calls this the Bostrom wager and wants the dragon-tyrant retired. Eight billion natural deaths, he argues, come nowhere near the tragedy of human extinction, and gambling humanity to prevent them is preposterously selfish. The objection turns the existential-risk logic of Bostrom’s earlier work against his current argument: extinction destroys the living and every future that might have followed them, so no saving of present lives can pay for a real chance of it.
Grant roon the impersonal accounting. His objection still compares the risky technological trajectory with an imaginary baseline in which humanity declines the gamble and stays safe. That baseline does not exist.
The missing counterfactual
A policy against dangerous technology still produces a future. Asteroids keep moving and pathogens keep evolving. Political systems fail on their own schedule, and other technologies diffuse whether or not the dangerous one is built. The Earth remains a single point of failure, and the Sun does not suspend stellar evolution because humanity has adopted a precautionary principle.
Over a long enough horizon, permanent confinement to Earth ends in extinction. Avoiding that fate requires capabilities we do not yet possess, and acquiring them means technological development, which creates new failure modes as it removes old ones.
The impersonal accounting counts more than extinction. In Bostrom’s 2002 taxonomy of existential risks, permanent technological arrest is itself an existential catastrophe: humanity survives, but its potential is curtailed for good. By roon’s own measure, a civilization that stays safe by staying small has already lost most of what extinction would destroy.
The natural hazards, by contrast, are slow. Humanity’s survival record bounds the natural extinction rate below roughly 1 in 14,000 per year, and probably below 1 in 87,000, and the Sun’s deadline is a billion years out. Bostrom’s surgery analogy transfers to civilization with a different prognosis. An untreated patient dies within decades. Held constant for a century, the conservative bound puts the natural risk of waiting at about 0.7 percent, and the likelier one at about 0.1. Against the one-in-ten chance Toby Ord assigns to AI catastrophe this century in The Precipice, that is small.
The background that makes restraint dangerous is the one humanity produces: nuclear arsenals, engineered pathogens, and the other capabilities that keep diffusing while one is held back. Restraint on a single technology leaves all of those running, and some of them are hazards that only a more capable civilization could suppress.
So the comparison is between trajectories. If developing some technology creates a substantial extinction hazard over the next twenty years, that counts against it. If refusing to develop it leaves civilization exposed to other hazards over the next hundred or thousand years, that counts too. A serious existential-risk analysis has to price both sides of the counterfactual.
“Do not gamble humanity” is therefore incomplete. It becomes a policy only after specifying what humanity does instead, what risks that alternative preserves, and how those risks evolve over time.
Delay can be rational
None of this means near-term technological danger should be tolerated because long-term danger also exists. A temporary pause is rational if it buys safety faster than it accumulates other risks.
Suppose a fifty-year delay in some dangerous capability cuts its extinction hazard dramatically while adding almost nothing to humanity’s exposure elsewhere. Then delay is the safer trajectory. Background risk does not erase the value of buying time.
The logic also runs in reverse. A delay that barely improves safety, while letting other hazards grow or pushing development toward less controllable actors, can make the trajectory worse. Speed and delay have no fixed sign. Their value depends on how they change the hazard profile of the future.
Differential development
The most defensible precautionary position is selective development. Bostrom has argued since the same 2002 paper for differential technological development: accelerate technologies that reduce existential risk and delay technologies that create it. Pandemic defense, asteroid detection, resilient infrastructure and defensive monitoring can be advanced while more dangerous capabilities are held back until their failure modes are better understood.
In trajectory terms, the objective is to order capability development so that resilience outruns vulnerability. Protective capacity should arrive before destructive capacity becomes easy to acquire, cheap to deploy or widely distributed.
The difficulty is that technologies are rarely cleanly separable. The same advances in computation and biology push defense and offense forward together. A breakthrough that strengthens monitoring may strengthen evasion; a platform for biological modeling serves vaccine design and pathogen design alike; a general-purpose reasoning system accelerates safety research and dangerous capability research at once.
Where the two cannot be pulled apart, technical entanglement moves the problem out of research prioritization and into governance. Access, deployment and compute become things to allocate, and allocation is a coordination problem.
The AI case
Frontier AI makes the problem concrete. “Pause AI” sounds like a single policy, but it describes several different trajectories.
A ten-year pause that genuinely freezes dangerous capability scaling while alignment methods, interpretability and control techniques improve could reduce existential risk substantially. A nominal pause that shifts development into secret military programs, less cautious jurisdictions or unmonitored decentralized networks could increase it. Continued development could be reckless if capabilities outrun control, or beneficial if it produces systems that sharply improve our ability to manage other existential hazards before dangerous capabilities diffuse.
The word pause settles none of this. The answer depends on who keeps building during the pause, which safety capabilities improve in the meantime, and whether the delay changes the eventual system or only who builds it first.
“Move faster” faces the same test. It is a policy argument only if the resulting trajectory is safer than the alternatives.
The technological plateau
Restraint has a more ambitious form. Perhaps humanity can develop exactly the capabilities needed for long-term survival and then stop: build asteroid defense, pandemic protection, resilient infrastructure and safe spaceflight, and refuse AGI, unrestricted synthetic biology, molecular nanotechnology, or whatever else crosses the dangerous threshold.
That is coherent as a target. The difficulty is keeping civilization there.
A technological plateau has to remain stable across centuries or millennia, through political change, cultural drift, economic competition and the diffusion of knowledge. If a dangerous capability eventually becomes accessible to thousands or millions of actors, permanent restraint requires extraordinary coordination. A prohibition that works only while every relevant actor keeps cooperating is not automatically safer than the capability it forbids.
The problem sharpens once the prohibited capability has strategic value. If one state, firm or faction expects an advantage from crossing the threshold, everyone else inherits an incentive to anticipate the defection.
Perhaps that can be solved. But then the supposedly safe alternative includes a global control regime powerful enough to stop future actors from developing prohibited technologies, and such a regime has its own catastrophic failure modes: permanent lock-in, surveillance abuse, institutional capture, or being wrong about which technologies should stay forbidden.
A plateau also has to hold everywhere civilization reaches. A reproductively autonomous colony beyond enforcement range is free to cross the threshold the plateau forbids, so a regime that means to hold the line for millennia has reason to forbid such colonies as well. That is the mechanism by which permanent containment could be the Great Filter: it stops expansion without killing anyone. A plateau held that way survives by suppressing the one kind of expansion that would let its failures stay local.
The plateau is therefore another trajectory with another risk profile. It may still be the best one, but it does not get to count as the absence of a gamble.
Risk does not begin when humans cause it
Technological hazards attract attention because their causal stories are clean. Researchers build a system, the system fails, people die. Responsibility is visible.
Natural hazards are easier to misfile as background. If an asteroid destroys Earth ten thousand years from now because humanity never built adequate detection and deflection, nobody launched the asteroid. But for a policy chosen today, the causal origin of a hazard does not make its consequences disappear. A deontologist may hold the doing/allowing distinction decisive for choice as well as blame. It still leaves survivor counts unchanged, and survivors are the currency of roon’s objection.
Self-imposed incapacity works the same way. If humanity stays confined to one planet because sufficiently powerful technologies were repeatedly rejected as too dangerous, eventual extinction is not rendered benign by the absence of a villain. The policy still had consequences.
Existential-risk reasoning often turns asymmetric here. Risks introduced by action are counted explicitly, while risks preserved by restraint disappear into the baseline.
Safety is cumulative
Almost every consequential policy creates some risk, so the presence of risk disqualifies nothing. The quantity to compare is how a policy changes total survival prospects across time.
A technology can be dangerous over one interval and protective over another. Asteroid deflection is the textbook case: Carl Sagan and Steven Ostro warned in 1994 that a system able to push an asteroid away from Earth could push one toward it, yet a civilization without that capability remains exposed to impacts it could otherwise prevent.
Space settlement shows the same structure over a longer span. Launch systems, autonomous machinery, advanced energy systems and closed-loop habitats all introduce hazards. But a civilization confined to one planet carries its single point of failure indefinitely, and successful dispersal removes at least that class of vulnerability.
Daniel Deudney’s Dark Skies argues that the net effect runs the other way. Asteroid redirection hands rival states a planet-killing weapon, settlement seeds interplanetary war, and so he would relinquish habitat expansion for at least several centuries. That is a delay argument, and it stands or falls on the comparison the trajectory framework demands: whether centuries of confinement, and the regime needed to enforce them, cost less than the weapons dispersal would create.
The opposite pattern also occurs. A technology can reduce some near-term hazards while introducing a much larger catastrophic failure mode.
The QBU view
The argument does not depend on any interpretation of quantum mechanics. Ordinary counterfactual reasoning is enough: different policies generate different distributions of outcomes, and those distributions have to be compared.
The Quantum Branching Universe makes the accounting literal. From a present Vantage, QBU gives each intervention under consideration a conditional distribution of descendant Measure, some share ending in catastrophe and the rest in continued civilization. Restraint’s distribution carries catastrophic Measure of its own in the asteroid branches and the pandemic branches. Development’s includes the branches where the new capability fails. A policy that develops a dangerous technology may raise catastrophic Measure in the near term and lower it later by removing other failure modes, while restraint may show the opposite pattern. The picture leaves no room for treating restraint as the real continuation and development as a deviation from it.
Measure is not morality, and a low-Measure catastrophe remains a catastrophe for the agents inside it. Whatever normative rule we adopt still has to evaluate harms across the descendants of every available choice, including the branches where restraint goes wrong.
Postscript
None of this vindicates the Bostrom wager. The argument above runs inside roon’s impersonal accounting, where a large enough increase in extinction hazard makes the bargain terrible however many present lives it saves. Bostrom’s person-affecting result is a separate claim that rests on whether existing people are the right unit of moral concern, and nothing here settles that.
Nor is the wager selfish in any ordinary sense. A person-affecting view weighs everyone now alive, and Bostrom’s prioritarian variant gives the most weight to the old and the sick, the people least able to wait. The Dragon-Tyrant establishes that aging is a vast preventable harm if we can defeat it safely enough. It does not license any means whatsoever.
The reverse inference fails too. Natural death does not become acceptable because some routes to defeating it are dangerous. The dragon’s toll stays in the ledger under either accounting: roughly a hundred thousand age-related deaths a day, some thirty-six million for every year of delay. The impersonal view can outweigh that number. It still has to enter it. The civilization capable of defeating aging may also become capable of defeating asteroids, pandemics, planetary fragility and eventually hazards we cannot yet model.
Or it may destroy itself first. That is why the unit of analysis has to be the trajectory rather than the isolated invention.
There is no policy called “avoid existential risk.” There are only interventions that change the timing, magnitude and composition of existential hazards. Precaution does not escape that analysis; it has to survive it.