Making Sense of P(doom)
The Hidden Structure Behind AI Existential Risk Estimates
Introduction
Discussions about existential risks from Artificial Superintelligence (ASI) frequently invoke the concept of "P(doom)"—the probability of catastrophic outcomes such as human extinction. But what does it mean when experts assign probabilities like 10%, 50%, or even 90% to such events? This post clarifies the distinction between objective and subjective probabilities and why it's crucial to specify temporal bounds when discussing P(doom).
Defining Objective Probability in a Quantum Branching Universe (QBU)
Within the Quantum Branching Universe (QBU) framework—based on the Many-Worlds Interpretation (MWI)—objective probability is clearly defined:
"Objective probability within the QBU is defined by the measure of quantum timelines or branches associated with a particular event."
In other words, objective probabilities represent the proportion of quantum timelines branching from a specific moment (e.g., "now") in which a defined event occurs.
Applying the QBU Framework to ASI Doom
Given this framework, there must exist an objective P(doom)—a measure determined entirely by quantum mechanical branching from the current moment. Even though this measure objectively exists, we currently have no practical means of accessing it directly due to fundamental limitations in measurement and computation.
Thus, the objective probability of doom due to ASI is ontologically real but remains practically inaccessible to us.
Clarifying the Temporal Dimension of P(doom)
To meaningfully interpret P(doom), we must explicitly state the temporal boundaries:
Lower bound: Implicitly "now" (t=0).
Upper bound: Must be explicitly stated (e.g., within 100 years).
Without a clearly defined upper bound, the probability estimate is ambiguous and potentially meaningless.
Crucially, when plotting P(doom, t) over time:
The probability function is monotonically increasing—it never decreases as we move further into the future.
It approaches a clearly defined horizontal asymptote, representing the total measure of timelines that will eventually experience doom.
Resolving a Key Misunderstanding
A common misunderstanding is that extending the time horizon indefinitely would cause P(doom) to approach certainty (probability of 1). However, this intuition, borrowed from classical frequentist interpretations, doesn't apply in the QBU framework.
In QBU, the objective probability is fixed by the quantum branching structure at the initial moment ("now") and remains constant. Extending the horizon infinitely doesn't inherently increase this probability—it simply allows us to capture the full, fixed measure of eventual doom timelines.
Why Estimate Subjective P(doom) if Objective P(doom) is Inaccessible?
Even though the objective probability is inaccessible, subjective probability (credence) remains a valuable rational approximation. Subjective probability represents our best-informed estimate based on available evidence, theories, and expert reasoning, and it significantly influences practical decisions and policies.
Consider a practical analogy:
Objective Probability: Your actual probability of dying from cancer exists objectively—determined by complex biological factors right now—but remains inaccessible.
Subjective Probability: Your rational estimate of your cancer risk, based on genetics, lifestyle, medical advice, and evidence, directly influences real-life choices (health screenings, diet, insurance).
Applying this analogy to ASI doom:
Objective P(doom) is real but currently unknowable.
Subjective P(doom) is pragmatically crucial, informing existential risk strategies and AI alignment research priorities.
Implications and Practical Value
Clarifying these distinctions is essential because it sharpens our approach to existential risks:
Reinforces epistemic humility in acknowledging inaccessible realities.
Encourages rigorous Bayesian reasoning and continuous updating of subjective probabilities.
Helps prioritize and rationalize funding, research, and policy-making efforts around existential risk mitigation.
Conclusion
The distinction between objective and subjective probabilities, clearly defined temporal bounds, and careful interpretation of P(doom) enhances our clarity, precision, and effectiveness in confronting existential uncertainty. Even though objective probabilities remain inaccessible, subjective estimates remain indispensable, informing meaningful actions today to safeguard humanity's future.