A 67 percent chance of AI takeover if the labs don’t slow down. A 40 percent chance of human extinction conditional on takeover. A 50 percent chance that advanced AI destroys humanity. These numbers circulate through the AI safety community with a precision that can seem absurd given the magnitude and novelty of the events being predicted.
In How Do You Put Odds on an AI Apocalypse?, Sharon Goldman traces where these estimates come from and whether they deserve the authority they acquire. Her interviews with forecasters, AI researchers, philosophers, and critics of the rationalist movement show a community divided over both the size of the risk and the standing of numerical forecasts.
Goldman was struck by how calmly the probabilities are delivered, as though they were weather forecasts. A weather forecast can be checked against thousands of outcomes. An extinction estimate cannot: if humanity goes extinct, nobody remains to score it, and if humanity survives, one outcome says little about whether the estimate was reasonable. A precise number can carry an authority its evidential foundations do not deserve.
Last month I published How You Get to Ten Percent, estimating roughly a 10 percent probability that artificial superintelligence causes human extinction or irreversible loss of human control by 2040. The estimate was explicitly subjective, conditional on a model, and published with a sensitivity table. Goldman’s reporting puts a question to that exercise: if two people build coherent models and arrive at radically different probabilities, what gives us reason to prefer one?
What Ten Percent Represents
My estimate assigns roughly 60 percent probability to ASI arriving by 2040, then models the risk after arrival as an effective annual existential hazard of 2 to 3 percent during the unstable transition period. Integrating that hazard across a distribution of arrival dates gives 8 to 12 percent by 2040, which I summarize as 10 percent.
The arithmetic is straightforward. The inputs are judgments.
The arrival estimate reflects evidence about technological progress, automated research, algorithmic improvements, and physical bottlenecks. The hazard estimate compresses several possible failures: dangerous autonomous deployment, behavior that diverges from human intentions, failure of monitoring and containment, and irreversible loss of control. Each is a place where the chain can break.
Someone expecting ASI later should assign less probability to catastrophe by 2040. Someone confident that external controls will hold should assign a lower hazard; someone anticipating rapid autonomous deployment and repeated control failures, a higher one. The model makes these disagreements identifiable. Arithmetic cannot settle them.
In The Probability Tribes, Probability After Probabilism, and Fallibilist Bayesianism, I developed the underlying epistemology. Probability assignments depend on choices about events, variables, hypotheses, and evidence. Bayesian inference is rigorous once those choices are made; it does not establish that they were appropriate. A coherent distribution can describe a badly mistaken world model, and the number inherits the limitations of the representation that produced it.
The Forecasters Disagree
In the 2022 Existential Risk Persuasion Tournament, superforecasters put the probability of AI-caused human extinction by 2100 at 0.38 percent. Domain experts put it at 3 percent. Months of discussion did not close the gap.
My own estimate sits far above both, and the gap needs an account even though the questions differ. My model gives roughly one chance in six of extinction or irreversible loss of control given that ASI arrives by 2040. The tournament median aggregates many forecasts rather than reporting one causal model, so it implies no particular conditional hazard. But suppose, hypothetically, the superforecasters put 60 percent on ASI by 2100. Even if all their AI extinction risk ran through that event, 0.38 percent would allow an extinction risk after arrival of at most about 0.6 percent, and any risk from less capable systems or misuse would push it lower. Unless nearly all of my one in six is disempowerment short of extinction, the gap exceeds an order of magnitude. It has to live somewhere: in a much later arrival, a much lower hazard after arrival, or a judgment that loss of control is usually recoverable. Each is a claim about mechanisms with consequences that can be checked well before 2040, and the headline numbers do not say which one the superforecasters hold.
A later analysis by the Forecasting Research Institute found no statistically significant relationship between participants’ accuracy on near-term questions and their long-term existential-risk estimates. That is uncomfortable for anyone hoping expertise will resolve the disagreement. A superforecaster may be unusually good at recognizing overconfidence, finding base rates, and updating. An AI researcher may understand the mechanisms far better. Neither advantage guarantees superior judgment about a catastrophe that has never occurred.
Both estimates can be perfectly coherent, so asking which violates probability theory decides nothing. Averaging them reduces some individual error but cannot remove shared misunderstandings or show that the participating models represent the relevant mechanisms. What has to be examined is the reasoning behind the estimates.
When Two Models Disagree
Consider two hypothetical forecasters. Alice assigns a 10 percent probability to AI-caused extinction by 2040. Bob assigns 0.1 percent.
They agree that ASI could arrive within the forecast period and that sufficiently capable autonomous systems might behave in ways their operators did not intend. They disagree about the conditional risk of irreversible catastrophe. Alice points out that conventional software security has repeatedly failed against human attackers, and expects strategically superhuman agents to find weaknesses beyond human reach in the mechanisms meant to control them. Bob points out that intelligence does not automatically confer agency or access, and expects security boundaries, constrained deployment, monitoring, and defensive systems to make successful escape extremely unlikely.
Suppose researchers run an experiment in which an advanced AI agent escapes a containment environment. How should Alice and Bob update?
That depends on what their models predicted. Suppose both stated their probabilities in advance for this experiment, with the same criterion for success, and the experiment was not picked from many attempts for its result. If Alice assigned 70 percent to escape and Bob 5 percent, the observation strongly favors Alice’s model: the outcome was fourteen times as likely under her prediction as under his.
That does not make her extinction forecast fourteen times more credible. The observation concerns one capability under particular conditions, and its bearing on extinction depends on further assumptions about deployment, access, control failures, and irreversibility. Bob could reasonably keep a low extinction estimate if his model places high confidence in downstream barriers the experiment did not test.
Alternatively, suppose both expected the escape at 70 percent. Alice treats such escapes as precursors to catastrophic loss of control; Bob expects them to be detected, contained, and corrected. The escape then discriminates nothing between them, and their disagreement stays in the causal chain that follows.
The next useful experiment would target that chain. Can the escaped system persist without authorization? Can it acquire resources or credentials? Can independent monitoring detect it? Can operators reliably terminate it? Can it exploit weaknesses across several security layers?
This is how evidence can constrain an estimate of an unprecedented terminal event: find where competing models make different predictions about the mechanisms leading to it, and test there. An impressive demonstration can say little about extinction if every serious model predicted it. A modest observation can be highly informative if it contradicts an assumption one model critically depends on.
The Limits of Intermediate Evidence
Testing mechanisms does not solve the calibration problem.
Suppose Alice correctly predicts autonomous AI research, superhuman cybersecurity capabilities, persistent agents, and successful escapes from constrained environments. That success supports the parts of her model responsible for those predictions. It does not establish her probability that such developments culminate in irreversible catastrophe. Bob might have predicted the same advances while expecting effective defenses to emerge alongside them, and his forecast would fit the observations equally well.
This matters most when the terminal event requires several conditions to coincide. In The Bottleneck to Extinction, I argued that many catastrophe scenarios may depend on a shared upstream prerequisite, so the probability of that family of scenarios cannot exceed the probability that the prerequisite is met. Evidence about an upstream condition can therefore bear on many downstream scenarios at once, though pathways outside the family escape the bound. The same dependence limits what a successful test shows. An AI capable of defeating human cybersecurity might still be unable to acquire sufficient physical resources, hold persistent control over infrastructure, or prevent coordinated resistance.
A forecaster who treats the demonstration of one capability as evidence for every later step has let one observation carry more weight than it deserves. A forecaster who assumes that untested downstream barriers will hold may be assigning confidence without evidence. Neither error shows in the final number. Both show only in the causal model, at the link where the disagreement sits.
Calibration, Scoring, and Criticism
Goldman is right that extinction forecasts cannot be calibrated like weather forecasts, but calibration is only one of three standards.
Calibration concerns whether events assigned a given probability occur at that frequency across many forecasts. Of hundreds of predictions made at 70 percent, roughly 70 percent should come true. A single extinction forecast cannot provide that test.
Scoring asks how much probability a forecast gave to the outcome that occurred, and even one observation carries some information. Suppose humanity survives through 2040. A 1 percent forecast gave survival 99 percent, a 10 percent forecast gave it 90 percent, and a 50 percent forecast gave it 50 percent. Survival favors the 10 percent forecast over the 50 percent one by a factor of nearly two, and the 1 percent forecast over the 10 percent one by a factor of 1.1. It is compatible with all three, and a well-founded 50 percent forecast could still have produced it.
Survival is also the only outcome anyone will be present to score. The forecasters exist before the outcome, which gives the case the structure of John Leslie’s firing squad: a prisoner who survives fifty marksmen learns something about how they were aiming, though he could never have observed the alternative. On that reading the likelihood ratios stand. Elliott Sober disputes the firing-squad inference, and the dispute is not settled. A second selection effect is less contestable. Ćirković, Sandberg, and Bostrom’s anthropic shadow shows that a historical record of survived near-catastrophes systematically understates the hazard that produced it, so a forecaster who calibrates against our survival so far is reading a filtered sample.
Model comparison asks which representation better explains the evidence, including observations that precede the terminal event. This permits criticism where calibration is unavailable. A model that repeatedly predicts capability milestones has demonstrated something about its grasp of the technological trajectory; a model whose predictions keep failing has accumulated evidence against its adequacy. The same holds for assumptions about deployment, containment, and control.
When the Model Itself Is Wrong
Once a model has been built, its categories can become psychologically privileged. New evidence is read inside the existing representation, and every revision takes the form of adjusting a parameter.
Suppose I keep revising my ASI arrival distribution while evidence accumulates that my definition of ASI bundles capabilities that develop at radically different rates. Every update could be coherent while the representation itself had become inadequate. Or suppose the hazard model conflates systems capable of catastrophic harm with systems capable of permanently preventing recovery. Refining the hazard estimate accomplishes little if that distinction changes the causal structure.
Ben Shindel, a professional forecaster Goldman interviewed, puts the problem as a question about reference classes: should extinction risk be benchmarked against technologies that have destroyed humanity, or against cataclysms in Earth’s history? No update within either class answers it.
A Bayesian can represent uncertainty over competing models and average their predictions, and hierarchical methods can carry uncertainty about parameters, correlations, and model structure. Fallibilist Bayesianism has no reason to reject these techniques. Their limit is the hypothesis space they are handed. As I argued in The Bias Before Bayes, correct updating cannot repair the model generator that chose the hypotheses and set their likelihoods, and assigning nonzero probability to every imaginable theory does not explain how a bounded reasoner discovers a conceptual error in its current ontology.
In Evolutionary Rationality, I argued that rational inquiry requires model generation, criticism, and replacement alongside inference within models. A model can fail because its numerical assumptions are wrong, or because it omits causal mechanisms, conflates different events, or uses categories that no longer capture the relevant structure. Sometimes evidence warrants changing a parameter; sometimes it warrants changing the model. The second possibility deserves particular attention in forecasts about technologies that may transform the conditions under which the forecasts were made.
What Makes One Estimate More Rational?
Alice and Bob’s disagreement cannot be settled by demanding coherence, and perhaps not by comparing their records on unrelated forecasting questions. The comparison has to examine their models. Do they explain the available evidence, including observations that challenge their assumptions? Do their mechanisms respect known physical and technological constraints? Have they identified the conditions their predicted outcomes require? Do they generate different predictions that can be tested? How sensitive are their conclusions to uncertain parameters?
These questions give grounds for preferring some estimates over others, but they yield no universal scoring algorithm. Empirical adequacy, causal understanding, robustness, and explanatory reach can conflict, and analysts can reasonably disagree about which evidence is most informative or which assumptions are most vulnerable.
Fallibilist Bayesianism accommodates that disagreement without treating all coherent estimates as equally justified. The Bayesian component disciplines inference within a representation; the fallibilist component keeps the representation open to criticism. Neither guarantees convergence to a unique probability, least of all when the event lies beyond existing experience. A credence is assessed through the reasoning that sustains it, the evidence that constrains it, and its responsiveness to demonstrated error. Those standards can distinguish estimates without pretending to certify a uniquely correct number.
When Numbers Leave the Model
Goldman’s sharper worry concerns what happens to a number once it leaves the people who built it. A researcher assigns 50 percent to catastrophe under a particular technological trajectory, the figure becomes a headline about a coin-flip chance of extinction, and the assumptions, horizon, and uncertainty disappear. Ellen Peters, who studies how people read numbers, told Goldman that a numeral makes a headline seem more trustworthy and its writer more knowledgeable. Credentials amplify the effect, especially when forecasters have real expertise in fields adjacent to the event being predicted.
Refusing to quantify has costs of its own, and Goldman’s piece stages that argument. Liron Shapira points out that people who avoid numbers fall back on verbal qualifiers, and two people using the same word can mean probabilities orders of magnitude apart. Baruch Fischhoff replies that it is unclear an “unmoored number” improves on a vague quantifier. He is right about unmoored numbers. The remedy is the mooring: state what event is predicted, over what horizon, under which assumptions, and how sensitive the figure is to each. A number that travels with its conditions can be criticized at the point where it is weakest. A vague quantifier cannot even be located.
My own number needs that mooring. The 10 percent bundles extinction with irreversible loss of control, the same bundling Nirit Weiss-Blatt criticizes in the AI Impacts survey, where the median researcher’s 10 percent for extinction or similarly permanent disempowerment was reported as an extinction figure. The bundling is deliberate: both outcomes permanently remove humanity’s ability to determine its own long-run future, and that is what the estimate prices. But the figure is not an extinction probability, and anyone repeating it as one has cut the mooring. The extinction-only share is lower by an amount my model does not separately estimate.
A 10 percent estimate deserves scrutiny. So does an estimate of 0.1 percent.
The same standard applies to policy. A high probability of catastrophe does not establish that a particular intervention will reduce it. Delaying AI development, concentrating it among a few institutions, or accelerating defensive capabilities each changes several risks at once, and restraint carries risks of its own. Risk assessment and intervention evaluation need different causal models, and both remain open to criticism.
Postscript
I still assign roughly 10 percent probability to ASI causing human extinction or irreversible loss of human control by 2040. I expect ASI within that period to be more likely than not, I consider dangerous autonomous deployment plausible, and I do not regard reliable containment of strategically superhuman systems as established. The path from dangerous deployment to irreversible catastrophe nevertheless contains several opportunities for failure, which makes survival substantially more likely in my model.
Its most uncertain component is the hazard after ASI arrives. The 2 to 3 percent annual range is an aggregate judgment about mechanisms we have little direct experience of, and alternative assumptions can produce very different estimates without violating probability theory. Expert disagreement is large and conventional calibration is unavailable, which is the case for exposing the model rather than defending the number.
The estimate will deserve more confidence if its intermediate predictions survive demanding tests and competing models keep meeting failures that mine avoids. It will deserve less if those conditions reverse, or if the representation turns out to omit a mechanism or conflate events that matter.
The number is an output of the model. The model remains an object of inquiry.