Probability After Probabilism
The arithmetic begins after the world has been carved
David Chapman’s critique of probabilism lands because it attacks a real pathology without attacking probability theory itself. Probability theory is innocent. It is a formal calculus, and a magnificent one. The error begins when the calculus is mistaken for rationality.
Probabilism is the habit of treating every uncertainty as though it secretly contains a correct number. The number may be hard to find. It may require heroic estimation, priors, likelihoods, reference classes, or expert aggregation. But on the probabilist picture, the number is there in principle, waiting for a sufficiently disciplined rational agent to extract it.
That picture is false.
This is not an attack on Bayesian inference properly understood. Sophisticated Bayesians already know that probabilities are assigned relative to information, models, priors, and hypotheses. The problem begins when that discipline hardens into a cultural reflex: every uncertainty must be forced into a number, even when the event-space has not been earned. Bayesian updating is valid inside a frame. Probabilism is the habit of pretending the frame has already been given.
Probability theory can tell us how numbers transform once the event-space, evidence model, prior, and likelihoods have been supplied. It cannot tell us which distinctions reality makes relevant, which possibilities belong inside the model, or whether the model has earned numerical precision at all.
Probability Is Late
A probability statement looks simple because the hard work has already been hidden.
“There is a 70% chance of rain tomorrow.”
“The coin has a 50% chance of landing heads.”
“This treatment has a 12% risk of serious side effects.”
“There is a 20% chance AI kills everyone.”
The grammar treats these as the same kind of statement. They are not. Each one depends on a different relation between model and world. Weather forecasting compresses chaotic atmospheric dynamics through measurement and simulation. Coin-flipping relies on symmetry, physical scrambling, and stable experimental practice. Medical risk compresses population data, clinical definitions, and reference-class judgment. A civilizational p(doom) estimate compresses an enormous speculative world-model that may not actually exist.
The notation is the same but the epistemic work is not.
Probability theory arrives late in the process. Before it can operate, something has already decided what counts as an outcome, what counts as evidence, what counts as repetition, and what counts as irrelevant. These decisions are not delivered by probability theory. They are acts of semantic filtering.
Probability is a disciplined compression of uncertainty. It projects a dense field of causal, perceptual, and conceptual possibility into a tractable event-space. That projection may be excellent. It may be crude but useful. It may be fake precision. The probability calculus itself cannot tell the difference.
Events Are Carved, Not Given
The world does not arrive pre-divided into “events.”
A coin toss seems like the easy case. Heads or tails. Two outcomes. One probability distribution. But even here, the event-space is an idealization. The coin can land on its edge. It can be caught, fall through a grate, be biased, or be flipped in a physical regime where deterministic prediction is possible. We ignore these possibilities because, for ordinary purposes, the two-outcome compression is legitimate.
Legitimate does not mean metaphysically given. It means the compression preserves the distinctions needed for the agent’s purpose.
This becomes more obvious as soon as we leave engineered games of chance. What is the event-space for “AI doom”? What counts as doom? Human extinction? Permanent loss of human agency? Totalitarian lock-in? Value drift? Replacement by descendants we would recognize as ours? Replacement by descendants we would not?
Those are not details to be filled in after assigning the probability. They define the event. Change the event boundary, the causal model, or the reference class, and the probability changes with them.
Most probability errors are ontology errors hiding behind a number.
Conditionalism Before Calculation
Bayesianism is valid inside a frame. Probabilism forgets the frame.
A Bayesian update has the form: given this hypothesis space, given this prior, given this evidence representation, given this likelihood model, the posterior follows. That is an extraordinarily useful structure. But the power sits inside the word “given.”
Given by whom? For what purpose? Under which compression? From which position?
The Bayesian calculus does not select the hypothesis space. It does not guarantee that the evidence has been represented at the right level of abstraction. It does not certify the prior as meaningful or prove that the likelihood model captures the causal structure of the world. It performs valid transformations after those commitments have already been made.
Conditionalism makes the hidden dependency explicit. A probability is never free-floating. It is conditional on an event-space, a reference class, a model boundary, a position, and a purpose.
A probability is about the world-under-a-description.
This does not weaken probability. It disciplines it. It prevents the number from pretending to have more authority than the frame that produced it.
Probability as Semantic Compression
The world contains more distinction than any agent can use. The agent survives by filtering. It must carve relevance from an overfull reality and ignore almost everything else.
Probability is one way of making that ignorance coherent.
It says: for this purpose, under this description, these are the relevant possibilities; these are their weights; this is how evidence should shift them.
That is an astonishing achievement, one of the great constructors of formal thought. But it is still a constructor. It builds a usable compression. It does not abolish the distance between compression and world.
A probability estimate therefore has two different failure modes.
The first is internal failure. The arithmetic is wrong. The update violates the axioms. The model is incoherent on its own terms.
The second is external failure. The arithmetic is correct, but the carving is wrong. The event-space is malformed, the reference class is bad, the causal variables are missing, or the model boundary excludes the dominant structure. The number is formally disciplined and epistemically useless.
Modern rational culture is good at detecting the first kind of failure. It is much worse at detecting the second, because the second requires judgment before calculation.
That is the work before formalization: deciding which distinctions matter, which can be ignored, and which compression will survive contact with action. Chapman calls this circumrationality. The ordinary name is judgment.
How to Carve Better
If probability depends on semantic filtering, the next question is how to judge the filter. The answer cannot be another probability assigned from outside all frames. The judgment has to be practical, causal, and iterative.
A good carving has predictive contact. It improves anticipation of what happens next, or explains what already happened without merely redescribing it.
It has causal grip. It tracks mechanisms rather than superficial resemblance. “Other technologies were dangerous” is a weak reference class. “Technologies that created autonomous replication, strategic opacity, and rapid capability gain under competitive deployment” is stronger, if those mechanisms are actually present.
It is action-relevant. It preserves distinctions that would change what an agent should do. A model that cannot alter attention, design, policy, or preparation is usually decorative.
It is robust: its conclusion does not collapse when the event boundary is adjusted, the reference class is changed, or one plausible pathway is removed.
It exposes its errors. It says what evidence would weaken it, which assumptions carry the load, and where the compression is most likely to fail.
And it compresses without deleting the dominant structure. Simplicity is valuable only when the omitted details are genuinely secondary. If the omitted variables dominate the result, the model has not simplified the world. It has hidden it.
These standards do not eliminate judgment. They discipline it. They turn semantic filtering from private intuition into an inspectable practice.
Probability From a Position
Probability is also position-relative.
An agent never faces the future from nowhere. It faces possible continuations with limited information, particular goals, embodied constraints, and finite discriminatory power. Probability is always assigned from somewhere.
A die roll, a weather forecast, an election model, a poker hand, and a quantum measurement all involve uncertainty, but the uncertainty is not the same kind of thing.
Credence is an agent’s epistemic compression.
Frequency is a count over repeated outcomes.
Symmetry probability is an inference from invariance.
Statistical-mechanical probability compresses inaccessible microstructure under thermodynamic constraints.
Quantum probability is tied to the formal structure of physical theory.
Decision probability is a policy input under uncertainty.
The word “probability” unifies these for calculation. It does not erase their differences in interpretation. Treating them all as one metaphysical substance is a category error concealed by shared notation.
Where Probability Regains Authority
None of this implies that probability is weak, arbitrary, or merely subjective. That would be the opposite error.
Probability becomes powerful when the world supplies structure that licenses the compression.
A fair coin does not contain a metaphysical substance called 0.5. But symmetry, physical scrambling, and stable experimental practice make the assignment robust. The number survives contact with reality because the event-space is simple, the outcome boundary is clear, and the relevant physical invariances are strong.
Statistical mechanics is not merely personal ignorance. It is anchored in phase-space structure, thermodynamic constraint, and the overwhelming dominance of macrostates. We cannot track every molecule, but the compression is not arbitrary. The world gives us lawful aggregation.
Quantum probability is stranger, and its interpretation remains contested. But even there, probability is not merely a verbal habit. The formal measure is tied to the structure of physical theory, not invented by preference.
So the right lesson is not that probability has no contact with reality. The lesson is that probability does not interpret itself.
When the world supplies stable frequencies, symmetries, invariances, or physical measures, probability gains authority. When those supports are absent, probability may remain useful as a decision aid, but its numbers should not be mistaken for calibrated contact with reality.
p(doom) and Compression-Boundary Failure
This is why p(doom) is such a revealing case.
The problem is not that existential risk is unreal. It is not irrational to worry about advanced AI, or to ask whether technological acceleration could destroy human agency or create irreversible lock-in.
The problem is that a precise p(doom) number often pretends to summarize a world-model that no one has actually built.
A small-world model says: these are the relevant variables, these are the live pathways, these are the dependencies, and everything outside the boundary can be safely ignored. For dice, cards, insurance tables, and many engineered systems, that compression can be legitimate. For civilizational-scale AI risk, the omitted structure may dominate the result.
What is the relevant reference class? Nuclear weapons? Biological weapons? Market competition? Speciation? Financial contagion? Evolutionary replacement? None of these is obviously right, and each imports a different causal grammar.
What are the live pathways? Rogue model, corporate race, military escalation, recursive self-improvement, infrastructure dependency, gradual agency displacement. Again, the event-space is built before the probability is assigned.
What does the number do? It may express personal alarm. It may summarize a private model, function as a policy signal, or mark group membership. It may discipline thinking, or dramatize uncertainty, or obscure it.
Those are different uses. Only some deserve to be called probability in a strong sense.
A p(doom) estimate can be meaningful when embedded in a specific model with explicit assumptions, explicit pathways, and explicit sensitivity analysis. Detached from that machinery, it often becomes compression failure expressed with mathematical confidence.
Numbers as Probes
Rough numbers can still be useful as probes. Forcing someone to say “17%” rather than “substantial” may expose hidden assumptions. It invites the next questions: why not 2%? Why not 80%? Which pathway carries the weight? Which reference class is doing the work?
That use is legitimate. The number functions as scaffolding for inquiry.
The abuse begins when the scaffolding is mistaken for a load-bearing structure. A rough number that provokes model-building is useful. A rough number that substitutes for model-building is rationality theater.
This distinction matters in practice. A number used provisionally can discipline conversation. A number used performatively can anchor debate before the model has earned authority. Once numerical confidence enters the room, qualitative objections are often treated as less rational, even when they are pointing at the malformed event-space underneath the number.
If the event-space is wrong, refusing the number is the rational act.
Postscript
After probabilism, probability theory remains intact. Bayesian updating is still valid inside specified frames. Frequentist methods are still powerful where repetition and sampling are real. Statistical mechanics is still profound, quantum measure still physically serious, forecasting still useful.
Agents need to act under uncertainty. They need to price risk, allocate attention, update beliefs, make forecasts, and coordinate with other agents. Probability gives us a disciplined way to do this when the frame is legitimate enough for the purpose at hand.
What changes is the metaphysical posture.
We stop treating probability as the hidden essence of uncertainty. We stop treating every unknown as though it contains a correct number. We stop pretending that calculation can rescue a bad carving of the world.
Rationality begins with relevance. The agent first has to carve the world into distinctions that can survive contact with action. Only then can probability operate as formal compression.
Used inside a legitimate frame, probability is among the highest achievements of formal thought. Used outside one, it becomes a ritual of precision.
Probability theory does not interpret the world.
Agents do.