Stories about how artificial intelligence could end humanity are easy to multiply. An AI might engineer a lethal pathogen, deploy autonomous weapons, sabotage critical infrastructure, manipulate governments into war, or appropriate the resources human civilization depends on. When one scenario meets an objection, another takes its place, and the length of the catalogue starts to feel like evidence.
Sometimes it is. A new scenario can reveal a mechanism nobody had considered, and that should raise the estimate. But scenarios are not independent merely because their mechanisms differ. Ten ways of killing humanity may all depend on the same difficult preliminary achievement, and if that achievement never occurs, none of the ten can unfold.
Timothy Gowers makes this point in How Worried Should We Be About Extinction Risk from AI? Discussing scenarios that rely on large numbers of remotely controllable robots, he observes that their combined probability cannot exceed the probability that the robots get built. The most convincing reassurance, he suggests, would identify a set of bottlenecks such that every extinction scenario depends on at least one of them, and he is candid that such a set would be hard to make complete. His own uncertainty sits where bottlenecks operate. He splits doom into three events: AI having wants, wanting to kill us, and being able to act on that want. He puts the first at essentially certain and locates his main uncertainty in the third, which is the term bottleneck analysis bounds.
The proposal can be made systematic with tools reliability engineering already has, and those tools find concentrated danger as readily as they find reassurance.
The Shared Bottleneck
Suppose we identify a hundred scenarios in which a rogue AI uses an autonomous drone fleet to exterminate humanity. They differ in targeting, weapons, and tactics, but every one requires a sufficiently large and capable fleet.
Call the existence of that fleet B, and extinction through any of these scenarios E. Because E cannot occur without B:
P(E) = P(B) × P(E | B) ≤ P(B)
Suppose, purely for illustration, a 10% probability that the necessary fleet exists by 2040. With one extinction mechanism available, succeeding 5% of the time given the fleet, extinction through that mechanism has probability 0.5%.
Now suppose we discover ninety-nine more mechanisms, which together raise the probability of extinction given the fleet to 90%. Aggregate extinction probability through the family rises to 9%. The new scenarios matter: they multiplied the estimate eighteenfold. But a thousand more cannot push it above 10% unless something changes the probability of the fleet.
Counting scenarios informs the estimate only through what the scenarios reveal about the prerequisite or about what follows it. A hundred variations on one failure mechanism are a hundred routes through the same gate.
Mapping the Causal Dependencies
A realistic extinction pathway usually requires several conditions to hold together.
Consider an AI-designed biological weapon. Something must design an agent capable of extreme harm; someone or something must produce it, distribute it, and overcome the defenses of the populations it threatens. Even a devastating pandemic is not necessarily an extinction event. Killing every human means overcoming geographic isolation, biological variation, and countermeasures, any of which may leave survivors.
An autonomous weapons scenario needs different things: manufacturing capacity, deployment logistics, targeting and control, and a way to defeat whatever resistance survives. Infrastructure sabotage needs access to critical systems, persistent control, and a mechanism by which local destruction propagates into irreversible global catastrophe.
These pathways form causal dependency graphs. Some conditions must occur together; others are alternative routes to the same outcome. Several pathways may share a prerequisite, and one pathway may need several.
Fault-tree analysis was built for this structure. A cut set is a combination of basic events whose joint occurrence produces the top-level failure, and it is minimal when no proper subset suffices. The dual notion is the path set: a combination of basic events whose joint non-occurrence guarantees that the failure does not occur. Reliability block diagrams state the same concept in success terms, as a set of components whose functioning keeps the system up. Model each extinction scenario as a cut set, and Gowers’s bottleneck becomes a path set of size one for its family: an event appearing in every minimal cut set of that family. His proposed reassurance, a collection of bottlenecks such that every scenario needs at least one, is a path set for the whole tree.
The probabilities still require care. If extinction requires several conditions jointly, its probability cannot exceed that of the least probable among them. Multiplying their individual probabilities is another matter, since it assumes an independence that rarely holds: a system that acquires one dangerous capability may be far more likely to acquire the others.
Dependencies can also concentrate danger. Suppose many nominally independent safety systems rely on one control service. Compromising that service defeats them all. Reliability engineers call this common-mode failure: apparent redundancy disappears because supposedly separate protections share a vulnerable component.
The same graph that bounds risk through a shared prerequisite can concentrate it through a shared dependency. An improbable node that every attack must pass through constrains the danger; a fragile node that every defense relies on concentrates it.
What Makes a Bottleneck Informative?
Not every necessary condition helps. Every extinction scenario requires something capable of causing extinction. Call that universal prerequisite extinction-level capability. It is necessary by definition and tells us almost nothing about how likely extinction is.
A useful bottleneck must be specific enough to investigate, necessary for the pathways under consideration, and easier to assess than the catastrophe it constrains. The third condition does most of the work. Trading uncertainty about ten scenarios for equally severe uncertainty about one prerequisite simplifies the diagram without improving the estimate.
Return to the drone fleet. We can study manufacturing capacity, the state of autonomous targeting, deployment costs, and how well existing defenses resist attack. None of this may yield a reliable probability for the fleet’s existence by 2040, but it names the assumptions the estimate depends on and points at evidence that could move them.
Sensitivity analysis says which of those assumptions to pursue. If the preferred intervention differs between a 1% and a 20% probability of fleet deployment, narrowing that range has decision value. If downstream extinction probability stays negligible under every credible deployment assumption, effort belongs elsewhere. In the decision-theoretic sense, the value of information is the expected improvement a result would make to the decision, net of the cost of getting it. Learning that changes no choice can still be worth having as science, but it does not set safety priorities. A causal map identifies the unknowns where a choice could turn, even where precise probabilities are unavailable.
Intelligence Moves the Bottleneck
We are discussing superintelligence. Why expect it to stay constrained by obstacles humans find difficult?
We should not. As I argued in Intelligence Moves the Bottleneck, intelligence can redesign processes rather than merely speed them up. An AI that cannot manufacture a million robots with current technology might find a different manufacturing process. A system denied access to critical infrastructure might find a vulnerability, acquire credentials, or persuade an authorized human to open the door.
Some apparent bottlenecks belong to our present methods; others arise from physics, information, or unavoidable causal sequence. Greater intelligence can dissolve the former and remains bound by the latter. A bottleneck’s existence today therefore says little about whether it survives superintelligence. But a bottleneck that can be overcome in principle still has to be overcome in fact, by some causal mechanism, with the resources and opportunities the environment actually provides.
Human authorization is the obvious soft point. A system denied direct control over dangerous infrastructure can try to manipulate someone who already has it. Personalized persuasion, deception, financial incentives, and coercion could turn nominally independent human operators into instruments of the same adversary, producing correlated failure across security boundaries that were meant to be separate. Human fallibility alone does not show that every boundary shares one vulnerability, though. Authority divided among independent operators, hardware-enforced restrictions, and dangerous actions gated on several independent conditions all raise the cost of compromise, and their effectiveness has to be tested against an adaptive adversary.
Instrumental convergence gives reason to expect competition for resources and control without any hostility toward humans, a pattern now visible in the logs. It does not make extinction a necessary or optimal strategy; a superintelligence may find routes to its objectives that need fewer resources, less conflict, or more cooperation than we would anticipate. But extinction need not be a strategy at all. Gowers finds the competitive version the more plausible one: an agent indifferent to us appropriates what human survival depends on, the way palm-oil plantations destroy orangutan habitat without anyone wanting orangutans gone. Habitat loss shrinks populations without guaranteeing that the last member dies, so this pathway reaches extinction only if the appropriation is thorough enough that no human population survives it. Its bottlenecks are the scale and reach of that appropriation, and no decision to attack ever appears in its graph.
When the Map Changes Faster Than We Can Draw It
Recursive self-improvement could make our causal models obsolete before we act on them. A transition from human-level AI to superintelligence over days or hours would leave iterative safety engineering little to iterate on: by the time researchers identified a new capability, built its dependency graph, and designed an intervention, the system might have moved to different strategies.
That favors safeguards fixed in advance and enforced independently of the agent over safeguards that must track its improvement. An independently enforced restriction on consequential external actions is one such safeguard. It does not need to predict the internal reasoning of the system it constrains, so it can stay effective across capability changes that would invalidate an adaptive defense. It is not automatically secure against superintelligence. An adaptive adversary may attack its implementation, exploit unintended interfaces, or manipulate the institutions that maintain it. But its security can be studied before the dangerous capability emerges.
Reliability engineering ordinarily analyzes failures within a defined system and operating environment. Superintelligent AI adds an adversary that can identify, manipulate, and create the pathways through which catastrophe occurs, so the causal structure is partly determined by the intelligence being assessed. The task is to sort constraints that hold independently of the agent’s capabilities from those resting on assumptions recursive improvement could invalidate, and to do it before the improvement.
The Unseen Pathways
The serious limitation of bottleneck analysis is the one Gowers names: nobody can enumerate every extinction pathway.
An autonomous AI building industrial infrastructure, a terrorist using AI-assisted biological design, and a government delegating military decisions to increasingly capable systems follow different causal routes, and restrictions on one may leave the others untouched. A superintelligence may also find routes nobody has imagined.
The formal situation is clean. Let X be extinction by any route, and suppose every scenario in a specified family requires at least one of the conditions B₁, …, Bₖ. Splitting X on whether any of those conditions occurs gives
P(X) ≤ P(B₁ ∪ … ∪ Bₖ) + P(X and none of B₁, …, Bₖ)
The first term can be investigated and may be rigorously bounded. The second is extinction by routes that avoid every listed bottleneck, and it is the whole difficulty. A bound for a defined family of scenarios is a result; a bound on global P(doom), the whole-risk quantity I tried to estimate in How You Get to Ten Percent, requires the further claim that the residual term is small, and that completeness claim is very hard to defend.
The unpredictability of superintelligence is part of why the residual term deserves weight. A system much more capable than we are may find strategies our current models do not represent, and our failure to imagine a pathway is weak evidence that none exists. Unknown pathways deserve probability mass, especially when a more capable adversary is searching for them. How much mass depends on assumptions about the adversary’s objectives, capabilities, and environment. Strategic superiority is evidence that our catalogue is incomplete; it does not establish that a successful extinction pathway exists or will be pursued.
Gowers borrows an analogy from Soares and Yudkowsky: you cannot predict how Magnus Carlsen will beat you at chess, but you can predict that he will. Gowers draws only the modest lesson, that refuting individual scenarios is not enough. The stronger reading, on which superiority alone predicts defeat, works for chess because the game is known to be adversarial, with fixed rules and an opponent trying to win. For superintelligent AI those are the open questions: whether the system will pursue objectives incompatible with human survival, whether that conflict makes extinction advantageous or merely incidental, and whether extinction is physically achievable. The stronger reading assumes the premises the estimate is trying to establish.
An incomplete model still tells us where to look and which experiments could change the estimate.
Breaking the Chain
A shared prerequisite is also a candidate intervention point. If several catastrophic pathways require the same enabling condition, preventing it reduces several risks at once, often more cheaply than separate defenses against each downstream scenario.
Nucleic-acid synthesis screening is a concrete case, already practiced by members of the International Gene Synthesis Consortium, who screen both the sequences ordered and the customers ordering them. An AI might produce a dangerous biological design entirely in software, but realizing it through a commercial synthesis provider gives an independent party a look at the order. The provider can screen before supplying material, without relying on the system that produced the design to recognize or disclose the danger. Its effectiveness depends on whether dangerous orders can be recognized, whether providers screen consistently, and whether alternative production routes exist. Screening does not eliminate biological risk; it interrupts a class of pathways between computational capability and physical harm.
Operational access works the same way. An AI that can reason about vulnerabilities, design weapons, or plan destruction still needs some route from computation to physical consequence. Restricting consequential actions through independently enforced authorization interrupts many pathways without the agent’s cooperation, on the principle argued in The Agent Is Not the Security Boundary: a control should keep working when the system it constrains behaves adversarially.
Finding a bottleneck does not establish the benefit of blocking it. An adversary may substitute another pathway: denied a laboratory, an AI might manipulate an authorized researcher; denied one manufacturing process, it might develop another. Institutions may relax controls when the economic returns to autonomy outweigh perceived risk.
Deployment is its own problem. A security boundary implemented by one laboratory protects that laboratory’s systems and leaves competitors’ exposed. A restriction on dangerous manufacturing may do little if equivalent facilities operate elsewhere without it. What matters is whether a safeguard will be adopted, maintained, and enforced across the pathways that matter, and whether it needs broad coordination or can protect individual systems on its own belongs in the causal analysis from the start.
Interventions can introduce common-mode vulnerabilities of their own. Centralized authorization makes oversight consistent and concentrates power in one service whose compromise defeats many protections at once. Sound designs may need independent controls, limited authority, and recovery mechanisms rather than one supposedly impregnable barrier.
The most informative probability bottleneck is not necessarily the most useful intervention point. A physical constraint might dominate our uncertainty about extinction while lying wholly beyond our control; a minor authorization mechanism might contribute little to the estimate and still offer an effective intervention. Estimating risk and reducing it draw on the same map and ask different questions of it.
Pauses fall under the same analysis. Gowers closes by favoring one. A pause changes the rate at which capabilities arrive, but its effect on extinction risk depends on which pathways those capabilities enable, whether other developers continue, and which defensive capabilities are delayed along with the dangerous ones. Restraint carries risks of its own, and they belong in the graph as much as the risks of building. The existence of catastrophic possibilities does not settle which intervention has the best consequences. That requires modeling the intervention.
Postscript
A hundred ways for AI to kill us may amount to one risk when they all depend on the same unlikely event. They may amount to an enormous risk when that event is probable, or when several genuinely independent routes remain available.
The number of stories cannot tell us which.