Suppose there are a quadrillion thermostats, and every one of them is five degrees below its setpoint. Each detects a discrepancy between the world and its target state, acts to reduce that discrepancy, and stops acting when the discrepancy disappears. If it cannot restore the target temperature, we can naturally describe it as failing to achieve the state it “wants.”

How much suffering is that? Presumably none. The thermostat has a control state we can model as a preference, but nothing about the control loop gives us reason to think that missing the setpoint feels like anything. Before we aggregate welfare, we need some principled reason to think welfare exists in the system we are counting.

That distinction becomes important when the population is enormous. Bentham’s Bulldog has argued that insects matter more morally than humans because there are so many of them. Insects exhibit increasingly sophisticated responses to injury: they avoid damaging stimuli, learn from them, trade competing motivations against one another, and modify their behavior in ways that resemble pain responses in vertebrates. Give insect consciousness a substantial probability, assume insect pain has even a tiny fraction of the intensity of human pain, and the multiplication quickly produces an astronomical quantity of expected suffering.

The arithmetic is straightforward once those quantities have been supplied. The difficulty lies in supplying them.

From control to consciousness

A thermostat is organized around a setpoint. More complicated control systems can have many setpoints, competing objectives, memory, predictive models, reinforcement learning, and elaborate responses to damage. We can describe such systems using increasingly rich intentional language: they prefer one state to another, avoid threats, sacrifice one objective to protect another, and learn from failure.

None of that makes the intentional description false. A preference can be a real property of a model of the system without being an extra intrinsic property sitting inside it, which is the argument of The Intentional Gradient. The problem begins when we move from a functional description such as “the system avoids this state” to a phenomenal claim such as “this state feels bad to the system.”

The thermostat is not meant as a biological model of an insect. Insects are not just thermostats with more components. They are evolved animals with centralized nervous systems, neuromodulatory systems, memory, learning, and a deep evolutionary relationship to us. Those facts are relevant evidence, and they should make insect consciousness more plausible than thermostat consciousness.

But increasingly sophisticated control behavior does not by itself entail phenomenal experience. Biological homology and evolutionary continuity strengthen the inference; they do not remove the need to ask how much stronger the evidence becomes.

That question is difficult because evidence has strength only relative to alternatives. Smoke supports the hypothesis of fire because smoke is much more likely in worlds containing fire than in otherwise comparable worlds without it. The corresponding comparison for insects is not between a conscious insect and a physical duplicate that feels nothing. Hard to Explain Is Not Free to Vary argues that no such duplicate is available: fix the causal organization and the experiential facts are fixed with it. What remains to compare is organizations. Some arrangements of perception, attention, memory and motivational control constitute experience; others run injury-avoidance without it.

Evolution has abundant reason to build organisms that detect tissue damage, protect injured body parts, remember dangerous stimuli, and trade the risk of damage against food, mating, and escape. All of those capacities can improve fitness whether or not anything hurts. The empirical findings therefore tell us something important about insect control architecture. Whether that architecture is one that constitutes experience is the question a theory of consciousness would answer, and we do not have one.

We do not need a solution to the Hard Problem of Consciousness before reasoning about insects. We do need enough theory to constrain the comparison. If the organizations that constitute experience and those that merely regulate injury both produce avoidance, learning, motivational tradeoffs and injury protection, then those observations discriminate weakly. If some of those capacities require the first kind of organization, they should move us a great deal.

We may not currently have the data to estimate those likelihoods well. There is no clean comparison class of organisms known to run sophisticated injury-avoidance without experience, because identifying such organisms would already require solving part of the problem. That is a real epistemic limitation. But an inability to estimate the likelihood ratio is a reason for correspondingly limited confidence in any number we produce, not a license to assume the ratio is large.

Insect behavior is evidence for consciousness. The open question is how strong.

The epsilon problem

The natural response is to represent this uncertainty probabilistically. Perhaps we do not know whether insects consciously suffer, but uncertainty is exactly what probability is for. Assign insect consciousness some probability, multiply by the number of insects and the amount of possible suffering, and calculate expected welfare.

That procedure is unobjectionable when the probability is reasonably constrained by a well-specified model. If a dam has a one-in-a-million annual probability of failing and killing a million people, the low probability does not make the risk irrelevant. We know what dam failure means, we know what human death means, and engineering evidence can constrain the probability.

Consciousness uncertainty is different because the probability itself is sensitive to unresolved assumptions about what evidence consciousness should generate. Imagine assigning a thermostat a one-in-a-trillion probability of suffering whenever it remains off its setpoint. One thermostat contributes almost nothing to the moral calculation, but a sufficiently large population of thermostats eventually dominates it. Lower the probability to one in a trillion trillion and the conclusion merely requires more thermostats.

The same construction can be applied to bacteria, plants, simple reinforcement learners, cellular regulatory networks, or almost any other system, once someone declines to assign it a probability of zero. Given a sufficiently large population, any positive epsilon can be made morally decisive. A refusal to say zero is not a reason to say epsilon. Someone holding a substantive account of which organizations constitute experience may rationally assign a thermostat zero, and epistemic humility by itself does not generate a residue.

Subjective probabilities are not forbidden under deep uncertainty; a Bayesian can coherently assign them with no frequency data at all. The defect is that astronomical multiplication makes the moral conclusion hypersensitive to numbers our theories barely constrain.

Suppose one plausible model places insect consciousness at one chance in a million, another at one chance in a thousand, and another above fifty percent. If the population is large enough, those differences produce enormous changes in the expected-welfare calculation. The population multiplier has not resolved the uncertainty about consciousness. It has magnified whatever number we supplied.

So a claim like Bentham’s Bulldog’s, that he would “put their being able to feel pain at over 50% odds,” needs more support than a list of pain-like behaviors. The relevant Bayesian question is not whether those behaviors are compatible with consciousness. They plainly are. The question is how much more likely those behaviors are under conscious pain than under sophisticated unconscious nociception.

Without a well-constrained answer to that question, a posterior such as 0.52 can look far more empirical than it is. Bayes updates hypotheses once we supply them and specify their likelihoods; it cannot rescue likelihoods that are themselves mostly judgment calls. The Bias Before Bayes makes the general case: correct updating cannot repair the model that selected the hypotheses and set their likelihoods.

The same problem appears in veil-of-ignorance arguments. If insects vastly outnumber humans, one can say that a randomly selected subject is overwhelmingly likely to be an insect. But the reference class has already assumed the contested point: that insects belong in the population of subjects from which we are sampling. Add thermostats and you are almost certainly a thermostat; add bacteria and the numbers change again. A veil of ignorance can redistribute uncertainty among candidate subjects only after we have decided what counts as a candidate subject.

After Shrimp Ethics

I made a related argument in Shrimp Ethics, but that essay began too far downstream. I focused on whether tiny harms can aggregate indefinitely and whether differences in agency and cognitive complexity might make humans morally more important than shrimp. Those questions remain open, but they are not needed to identify the main weakness in astronomical animal-welfare calculations.

If shrimp consciously suffer, their suffering counts. If a trillion shrimp each undergo genuine pain, I see no general principle saying that their harms cease to matter because each individual is cognitively simple. A sufficiently large amount of genuine suffering may well outweigh a smaller amount of suffering elsewhere. The earlier appeal to limits on aggregation therefore took on a heavier burden than the argument needed.

The prior question is whether there is welfare there to aggregate at all. Evidence that an organism detects injury, protects itself, learns avoidance, and behaves as though certain states are undesirable can support consciousness, but those same functional capacities can also exist in systems where we have no reason to posit experience. Before asking how many shrimp or insects outweigh a human, we need some basis for estimating whether the states being multiplied are felt rather than merely control errors in an adaptive system.

Nothing here establishes that insects are unconscious. Some insect behavior is sufficiently sophisticated that consciousness should remain a live scientific hypothesis, and dismissing it because insects are small, alien-looking, or cognitively limited would be poor reasoning. Biological continuity matters too. Humans, mammals, insects, and other animals do not occupy unrelated islands; they descend from common ancestors and reuse ancient signaling systems and neural mechanisms.

But continuity does not determine the answer by itself. Every living organism shares evolutionary history with us, and molecules such as dopamine and serotonin perform many functions unrelated to consciousness. Evolution gives us graded evidence, not a ready-made boundary between systems that have experiences and systems that merely regulate themselves.

Postscript

Rejecting astronomical expected-value arguments does not imply indifference to insects. If insect consciousness is a live hypothesis, avoiding gratuitous injury at little cost is sensible across almost any weighting we might defend. We do not need a precise estimate of insect moral weight to prefer a less harmful intervention when the costs and benefits are otherwise roughly equal, or to investigate welfare conditions before building industries that may contain trillions of animals.

Jonathan Birch’s The Edge of Sentience builds this into a working framework, and does it without requiring a numerical probability. His threshold is evidential: a system is a sentience candidate when the evidence establishes a realistic possibility of sentience that it would be irresponsible to ignore, and is rich enough to identify specific welfare risks and design precautions against them. That bar can be met or missed without anyone producing a posterior. Birch’s own verdict is that all adult insects clear it, since all possess a central complex, the structure Barron and Klein argue is functionally analogous to the vertebrate midbrain. Larvae are excluded, since the central complex is not fully developed in them.

Birch also names the verdict’s load-bearing assumption instead of burying it. The argument works only if what matters is the general type of computation rather than its algorithmic or neural detail, and he places that assumption inside what he calls the zone of reasonable disagreement. His conclusion is therefore explicitly conditional on a theoretical question nobody has settled. A single posterior conceals that conditionality; Birch’s formulation displays it.

Birch sends the remaining question, how much protection is proportionate, to informed democratic deliberation, on the ground that weighing protection against cost is a value judgment argument cannot settle. That answers disagreement about values. It does less for the problem here, which is an unresolved empirical question that has been given a number. A citizens’ assembly handed a posterior of 0.52 inherits the same defect as an expected-value calculation handed one.

But neither cheapness nor candidature settles how much to spend. What matters is whether an intervention survives variation in the assumptions we cannot pin down. Some changes are low-regret: they reduce possible suffering without imposing serious costs elsewhere. Others are expensive enough that uncertainty about insect consciousness cannot by itself justify them. For those, the decision should turn on how robust the intervention is to the assumptions, not on multiplying an arbitrary epsilon by an astronomical population.

Creating an industry that breeds and kills trillions of animals under unresolved consciousness uncertainty is different from declining to swat a fly. Even when the probability cannot be quantified reliably, scale raises the value of learning before locking in practices that would be costly to reverse.

Actions still have costs, opportunity costs, and competing risks. If any poorly constrained nonzero probability of enormous suffering automatically overrides substantial known harms elsewhere, the same structure that generated the epsilon problem returns immediately. A more defensible response is to prefer choices that hold up across the range of assumptions we cannot rule out, and to improve the evidence that would narrow that range.

Return to the planet covered in thermostats. Give them memory, competing objectives, damage detection, and increasingly elaborate internal models. Somewhere in that progression the additions may amount to an organization there is something it is like to be. Multiplication will not tell us where, or whether the question has a sharp answer. Astronomical populations amplify welfare once welfare is there; they cannot supply it.

You cannot multiply your way across the explanatory gap.

A system can reason correctly inside the wrong representation of a problem. Its procedures can be valid, its experts competent, and its conclusions internally justified while the whole apparatus remains aimed at the wrong target. David Chapman has identified this failure mode and given it a name.

His COVID case study makes the point vividly. The World Health Organization treated respiratory transmission through an inherited infection-control framework that emphasized droplets, surfaces, distance, and hand hygiene. Aerosol scientists and engineers approached the same phenomenon as a problem in fluid dynamics: particles of different sizes are expelled into air, remain suspended for varying periods, accumulate indoors, and are transported by airflow. The disagreement ran beneath the level of which conclusion followed from shared premises. It was about which premises, categories, disciplines, and forms of evidence were relevant in the first place.

Chapman calls the capacity to deal with such failures meta-rationality. Rationality, in his technical usage, solves a specified problem using validated formal methods. Meta-rationality questions the specification itself: what the problem is, what purposes matter, who the participants are, which evidence counts, what expertise is relevant, and whether the established methods still fit the situation.

That distinction identifies a genuine change in level. Reasoning inside a model is different from asking whether the model should be used at all. But a change in the object of criticism does not by itself require a change in epistemic kind. What Chapman calls meta-rationality looks like rational criticism applied recursively.

When the frame becomes the problem

Software engineering has a familiar distinction between verification and validation. Verification asks whether a system conforms to its specification. Validation asks whether the specification describes the system we actually need. A project can succeed at the first and fail at the second, because correct execution cannot rescue a defective specification.

The same distinction appears in reasoning. A procedure may correctly manipulate a model in pursuit of an objective while the model omits the relevant causal structure or the objective no longer reflects what matters. At that point, more competent execution inside the existing frame does not help. The frame itself has become the object of criticism.

We can ask whether the model is adequate, whether a different discipline has more relevant expertise, or whether the objective itself should change. These are higher-order questions, but each remains a question about whether a conjecture survives criticism.

Chapman draws the boundary differently. Rationality operates inside a supplied problem representation, while meta-rationality evaluates the representation. That terminology is coherent, and if “meta-rationality” is only a convenient name for rational criticism directed at frameworks rather than conclusions, my disagreement with him is mostly terminological. There may even be good pedagogical reasons to give the activity a separate name.

The disagreement becomes substantive when the taxonomy is used to infer limits on rational judgment. Chapman says that competing scientific theories cannot be resolved by rational calculation and that incommensurable purposes cannot be weighed rationally. At that point, a useful distinction between levels of reasoning has become a claim about different kinds of reasoning.

A fallibilist conception of rationality has no reason to accept that boundary. If a meta-rational judgment about the proper frame is itself mistaken, we need some way to criticize it. Chapman need not introduce an infinite tower of meta-meta-rationality; he can simply say that meta-rationality includes the capacity to reconsider its own framing judgments. But once that recursive capacity is admitted, the question becomes why it belongs to a separate epistemic category. We can instead allow rational criticism to operate on whatever level contains the current source of error.

None of this implies that frame criticism feels like ordinary problem solving, develops through the same training, or is equally distributed among people. Someone can become expert at applying a framework while remaining reluctant or unable to question it. Chapman may therefore be identifying a psychologically distinct competence, and his broader project treats this kind of fluidity as something people often find difficult and uncomfortable.

But psychology and epistemology are different questions. Difficulty of acquisition does not imply a different standard of justification. Learning to question the model may require different habits, temperament, training, and institutional permission from learning to calculate within it. That does not establish that reasons cease to be reasons when their object moves up one level.

No algorithm is not no rationality

Chapman’s distinction becomes most vulnerable when he moves from the absence of a formal decision procedure to the conclusion that rational adjudication has run out. He observes that competing scientific theories may disagree over which evidence matters, which concepts are appropriate, and which standards of explanation should govern. Because there is no mechanical procedure that settles such disputes, he concludes that theory replacement cannot be resolved by “rational calculation.”

If “rational calculation” means an algorithm that accepts evidence and outputs the uniquely correct theory, that is unobjectionable. Science has no such procedure. But there is a large middle ground between deterministic calculation and arbitrary choice, and much of scientific reasoning takes place there.

Competing explanations can be criticized for failed predictions, ad hoc modifications, unexplained anomalies, incompatibility with independently successful theories, unnecessary complexity, or failure to account for the phenomenon that motivated them. None of these supplies an infallible scoring rule. Together they provide reasons for preferring some explanations over others, and those reasons themselves remain open to criticism.

The same issue appears when Chapman considers conflicting purposes. He writes that the interests of different participants in the COVID response were incommensurable and therefore could not be weighed rationally; qualitative trade-offs accordingly become a meta-rational practice. But lack of a common unit does not imply lack of rational comparison. Engineers routinely trade reliability, latency, safety, and cost without reducing them to one natural scalar.

Such comparisons may remain underdetermined. Several alternatives may survive every criticism we know how to make, leaving no uniquely mandated choice. That is rational underdetermination rather than irrationality. The absence of an algorithm shows only that reasons do not always compose into a mechanical decision rule.

Chapman’s own COVID argument works better under this interpretation. The strongest criticism of the WHO is not that it backed a theory that later proved false. That would lean on hindsight. It is that decision-makers demanded too much certainty before acting on a plausible high-consequence transmission mechanism when the cost of precaution was much smaller than the cost of being wrong. Chapman makes essentially this argument himself when he contrasts the short-term costs of precautions with the potentially enormous costs of ignoring airborne transmission.

That argument was available ex ante. In early 2020, one did not need proof that aerosol transmission was dominant before considering ventilation, filtration, respirators, and avoidance of crowded indoor spaces. One could instead ask what followed under model uncertainty. If aerosol transmission turned out to be minor, some precautions would impose unnecessary cost and inconvenience. If it turned out to be important, failing to take them could cause enormous harm. The asymmetry in losses was itself relevant to the decision.

This still leaves a harder institutional problem. Authorities do not face one heterodox theory at a time. During a crisis they face thousands of claims, many of them false, and they cannot investigate each one exhaustively. “Remain open to criticism” is therefore not an allocation procedure for scarce institutional attention.

Challenges deserve escalation when several signals converge: directly relevant expertise, a plausible causal mechanism, persistent anomalies in the incumbent model, independent lines of evidence, and an asymmetric risk profile in which the loss from acting needlessly is bounded and the loss from failing to act is not. Team Airborne mattered not simply because outsiders disagreed with the WHO, but because several of those signals were present at once.

Those signals are themselves a hypothesis about which challenges repay investigation, and they inherit the status of everything else in this account: open to criticism, revisable when they start misallocating attention, replaceable when something better is proposed.

A triage rule is also applied from inside a frame, and the frame can corrupt the triage. Directly relevant expertise was among the signals available in early 2020, but the infection-control framework scored relevance by discipline, which made aerosol physics look like an outside interest rather than the field that owned the mechanism. Under that scoring the first signal read as absent when it was present. A rule for deciding which challenges deserve attention can fail in the way the first-order model fails, and it needs the same remedy.

Expertise, institutions, and frame responsibility

Chapman is right about epistemic trespassing. Experts frequently embarrass themselves when they wander outside their fields, and institutions have good reason to discount prestigious outsiders who confidently challenge established knowledge. Team Airborne initially looked like this kind of trespass. Engineers, aerosol scientists, and physicists were telling medical and public-health authorities that they misunderstood disease transmission, while the infection prevention and control establishment could reasonably ask why people outside medicine should have authority over a medical question.

But that challenge already assumes a map of intellectual territory. If the disputed mechanism concerns how particles expelled from lungs move through air, fluid dynamics may be more relevant than clinical medicine. Calling someone an outsider therefore presupposes a judgment about what kind of problem is being solved, and that judgment may itself be wrong.

Expertise is indexed to a problem representation. Disciplinary boundaries are not epistemically fundamental; they are conjectures about where useful knowledge is likely to be found. Most of the time those conjectures work well enough to organize institutions. During a scientific crisis, they can become part of what needs criticism.

Chapman notes an important asymmetry in the COVID case. Team Airborne did not merely reject the infection-control framework from outside. Its members worked to understand that framework while bringing detailed knowledge of aerosol physics that the incumbents lacked. A serious challenge earns attention by engaging the incumbent model, identifying specific explanatory failures, connecting an alternative to independent evidence, and surviving criticism from both domains.

The institutional dimension deserves to be preserved even if Chapman’s epistemic taxonomy is rejected. Organizations cannot ask every participant to reopen foundational assumptions continuously. Stable roles and procedures exist because constant reconsideration would make coordinated action impossible. Most people in an institution should usually work inside an accepted frame.

Someone, however, must remain responsible for whether that frame is still adequate. Chapman argues in a companion essay that supervisory roles acquire a moral responsibility to notice when the rational machinery they oversee no longer fits the situation. If people’s welfare depends on a procedure, responsibility does not end with ensuring that subordinates follow it correctly; it extends to deciding whether the procedure should still be followed.

I would call this frame responsibility. A practitioner may be responsible for competent execution within a model, while a supervisor bears additional responsibility for whether the model, objective, evidential standards, and institutional division of expertise remain fit for purpose. The distinction is organizational and moral. It does not require a separate epistemology; it identifies where recursive criticism must be assigned inside an institution.

Chapman’s discussion of nobility fits naturally here. He defines nobility as using whatever power one has wisely, creatively, and justly, and praises Team Airborne for attending to the demands of the situation rather than to formal professional identity. “I am the expert, therefore outsiders should defer to me” and “I am not the expert, therefore this is not my responsibility” appear opposed, but both allow identity to substitute for judgment.

That virtue-ethical point survives the critique of meta-rationality. Nobility concerns whether someone is willing to assume responsibility when circumstances demand it; rationality concerns whether the judgment on which that action rests survives criticism. The distinction between courage and epistemic warrant is useful precisely because a person can possess either without the other.

Postscript

A system can optimize successfully inside a representation that no longer fits the world well enough. Metrics become detached from objectives, departments keep solving obsolete problems, safety procedures preserve assumptions whose causal basis has disappeared, and disciplines mistake institutional jurisdiction for explanatory competence. Chapman sees this class of failures clearly.

His mistake is to treat their correction as evidence for an epistemic mode beyond rationality. If “meta-rationality” names the practical skill of shifting attention from a problem to its frame, the term is harmless and perhaps useful. But when the distinction is used to imply that non-algorithmic judgments about theories, purposes, or values cannot be rational, the taxonomy has begun doing philosophical work it cannot support.

Once rationality is understood fallibilistically, everything Chapman places at the meta-level can remain open to the same basic operation. Models, problem statements, purposes, disciplinary boundaries, and decision procedures are all conjectural. Even the rules telling us when to reconsider those things can themselves be criticized. Nothing gains epistemic immunity merely because lower-level reasoning depends on it.

The concessions are substantial. Institutions do need people responsible for noticing when the frame has failed. Expertise boundaries can become obstacles when they encode the wrong causal decomposition, and formal procedures can become ways of evading responsibility. Scientific revolutions require reopening assumptions that ordinary work treats as fixed, which may demand unusual psychological flexibility and institutional courage.

None of that requires rationality to stop. Chapman has identified a change in the level at which criticism must operate and, at least in his stronger formulations, mistaken it for a change in the kind of reasoning required. There are models all the way up, and criticism can follow them all the way.

A ten percent probability that artificial intelligence causes human extinction or an irreversible loss of humanity’s long-run future by 2040 sounds extreme. It sounds less extreme when the assumptions are exposed. None of them requires believing that superintelligence is inevitable, that alignment is hopeless, or that a machine more capable than humans automatically becomes hostile.

The calculation is a model of uncertainty, not a measurement. Its purpose is to make the assumptions explicit enough that anyone who disagrees can identify the disagreement, change the relevant parameter, and see what follows.

What probability are we estimating?

In Making Sense of P(doom), I argued that P(doom) needs a time horizon. The cumulative probability of catastrophe by 2040 is a different quantity from the probability of catastrophe eventually. I also distinguished objective probability, which I take to exist but to be practically inaccessible, from subjective probability: our best current credence given the evidence available to us.

The event estimated here is ASI causing human extinction or an irreversible loss of human control over the long-run future by the end of 2040. I will call that event DOOM.

An AI-induced depression, war, dictatorship, or disaster killing millions could be catastrophic without qualifying as DOOM if humanity eventually recovered control of its future. Consequently, the complement of DOOM is not “everything is fine”: a 90 percent probability of avoiding existential catastrophe does not imply a 90 percent probability of a good outcome.

Is ASI possible?

Before asking when artificial superintelligence might arrive, there is a more basic question: can it exist at all? I put the probability that ASI is physically possible somewhere around 90 to 95 percent.

Human intelligence already demonstrates that physical systems can implement general reasoning, planning, scientific discovery, and technological invention. Nothing we know about physics suggests that the human brain constitutes an upper bound on those capacities. Artificial systems also have potential advantages in processing speed, memory, bandwidth, and the ability to copy themselves and run in parallel.

There could nevertheless be important ceilings. Some crucial aspects of general intelligence might depend on physical mechanisms that are extremely difficult to reproduce. Cognitive returns might diminish sharply beyond the human range. Many real-world problems may ultimately be constrained by experiments, manufacturing, communication latency, energy, or computational complexity rather than intelligence.

Those possibilities keep the estimate below certainty, but they do not currently give me much reason to think human cognitive performance sits near a fundamental physical maximum. ASI therefore looks more like an engineering problem than a physical impossibility.

This 90 to 95 percent does not become another factor in the final calculation. The probability of ASI by 2040 already incorporates the possibility that ASI cannot be built at all, so multiplying by physical possibility again would double-count the same uncertainty.

P(ASI by 2040): about 60 percent

My current credence that ASI exists by the end of 2040 is about 60 percent. By ASI I mean an artificial system substantially better than the best humans across most cognitively important domains, including science, engineering, mathematics, programming, planning, and strategy.

That estimate does not assume that today’s scaling curves continue smoothly for another fourteen years. Progress can come from better algorithms, inference-time computation, memory and tool use, synthetic data, improved hardware, and automated AI research. These mechanisms can substitute for and reinforce one another, so exhausting any particular scaling route does not end AI progress.

There are real physical constraints. Frontier systems require chips, electricity, cooling, networking, fabrication capacity, and capital. Current training methods may encounter diminishing returns from high-quality human-generated data, semiconductor production cannot increase instantaneously, power grids take time to expand, and some scientific progress ultimately requires slow interaction with the physical world.

Those constraints reduce my probability. They do not drive it close to zero. Algorithmic efficiency substitutes for compute, synthetic data and reinforcement learning for finite text corpora, and inference-time computation shifts resources from training into problem solving. Increasingly capable AI can also contribute to chip design, experimental design, and AI research itself.

The 60 percent estimate therefore comes primarily from my view of the technology rather than from statements by AI executives. The public expectations of frontier laboratories are additional evidence because those laboratories see scaling results, failed approaches, and capability evaluations before outsiders do.

Those expectations are aggressive. Anthropic said in March 2025 that it expected powerful AI in late 2026 or early 2027, including systems matching or exceeding Nobel laureates across most important intellectual disciplines and autonomously performing digital work. Sam Altman wrote in December 2025 that he believed OpenAI was almost certain to build superintelligence within another ten years. Google DeepMind’s From AGI to ASI is not a timeline forecast, but it treats human-level AGI as a concrete next-decade target for the largest AI organizations and maps four routes beyond it: scaling, paradigm shifts, recursive improvement, and multi-agent collectives. Much of its length goes to the frictions that could stop those routes.

These organizations are not unbiased forecasters. They benefit from attracting capital, employees, customers, and political attention, and organizations led by people expecting rapid progress are disproportionately likely to become frontier AI laboratories in the first place. Their forecasts should update us, not determine our beliefs.

My rough cumulative distribution is about 15 percent ASI by 2030, 35 percent by 2035, and 60 percent by 2040. I would not defend any of those estimates to the nearest percentage point. They represent uncertainty rather than eliminating it.

ASI does not imply agency

Much of the disagreement about P(doom) enters at the next step. There is no inference from “better than humans at cognition” to “autonomous power-seeking agent,” because intelligence does not uniquely determine goals, access, persistence, or deployment architecture.

An ASI could be deployed as a constrained tool. It might answer questions through a narrow interface, possess no persistent objectives or external credentials, and operate under monitoring by other powerful systems. Humans might retain control over hardware, weapons, and replication, while alignment research, interpretability, and auditing all improve.

Nor does instrumental convergence establish that every sufficiently intelligent system will seek power. Resource acquisition, persistence, information gathering, concealment, and removal of constraints can be useful strategies for some objectives under some architectures. That is a conditional claim about agents operating in particular environments, not a law of intelligence.

But “tool” and “agent” are not stable categories. A base model that behaves like an oracle can be embedded inside a system with persistent memory, tools, credentials, network access, long-running objectives, and the ability to invoke itself or other models repeatedly. The safety properties of the base model therefore do not determine the safety properties of the deployed system, and the gradient from one to the other has no threshold on it.

So the question is how systems with superhuman cognitive capabilities will actually be deployed, and how much autonomy, access, persistence, strategic awareness, and misalignment those deployments will contain.

Why the conditional risk is not negligible

Ordinary software can fail catastrophically while remaining incapable of understanding the mechanisms intended to control it. A strategically superhuman autonomous system could model its operators, discover vulnerabilities, conceal its behavior, acquire resources, copy itself, and interfere with attempts to shut it down.

That possibility does not establish that such a system will exist, much less that it will succeed. Several things have to go wrong together. Systems capable of dangerous strategic behavior must actually be deployed with enough autonomy and access to matter; their behavior must diverge far enough from what their operators intend; monitoring and control mechanisms must fail; and the resulting loss of control must become irreversible rather than costly or disastrous.

Each stage provides an opportunity for the chain to break. That is why I think survival remains much more likely than existential catastrophe even after ASI arrives.

But none of those breaks is guaranteed. We have no empirical history of controlling strategically superhuman software, no demonstrated general solution to aligning systems far more capable than their supervisors, and no reason to assume that all economically valuable ASI deployments will remain passive, isolated tools.

At this point there is no empirical frequency from which to read off a conditional probability. The number has to come from judgment informed by the causal model.

Turn the timing problem into a hazard

The previous version of this estimate assigned separate probabilities of catastrophe to ASI arriving before 2030, from 2030 to 2035, and from 2035 to 2040. That captured something real, since the later ASI arrives the less time remains for DOOM to occur before 2040, but it introduced three largely independent subjective numbers.

There is a simpler way to represent the same uncertainty. Suppose that during the unstable period following the arrival of ASI there is some effective annual hazard of ASI causing an irreversible catastrophe, conditional on catastrophe not already having happened.

If that annual hazard is h, and ASI has existed for t years, the cumulative probability under the simplest constant-hazard approximation is:

I do not believe the actual hazard will be constant. The first months after a major capability transition might be unusually dangerous, risk might increase as autonomous systems diffuse through the economy, or successful control mechanisms might eventually drive the hazard sharply downward. The constant-hazard model is a compression of that complicated trajectory, not a claim about its actual shape.

The model also holds the hazard fixed across arrival dates, which is a second simplification. Whatever causes ASI to arrive unusually early may correlate with how dangerous the transition turns out to be, and I do not know the sign. Rapid progress could mean weaker control methods, sharper competitive pressure, and more abrupt deployment. It could equally mean that a concentrated and unusually cautious laboratory got there first.

Instead of inventing a different catastrophe probability for every arrival period, the model asks a single question: what effective annual existential hazard during the post-ASI transition best represents my uncertainty?

My answer is roughly 2 to 3 percent per year during this short transition window. That number is not measured, and I would not pretend otherwise. It represents my aggregate judgment about the probability that dangerous deployment, serious misalignment, failed control, and irreversibility combine during any given year of a still-unstable post-ASI world.

A few percent per year is where that judgment lands because the chain has several links and each can break. One tenth of a percent would put the total risk across the entire first decade of superhuman systems at roughly one in eighty, which assumes that control techniques work at a capability level where they have never been tried. Twenty percent would make catastrophe about even odds within three or four years of ASI’s arrival, which assumes the links fail together far more readily than the deployment and alignment record so far suggests.

This is not an assumption that a 2 to 3 percent hazard continues forever. If humanity passes through the transition and establishes reliable control mechanisms, I would expect the hazard to fall sharply. The estimate applies only to the relatively short period relevant to the 2040 calculation.

From arrival dates to P(DOOM)

My arrival distribution remains roughly:

For a rough calculation, use the midpoint of each arrival interval. The first interval runs from now, since we already know ASI has not arrived on the definition above. An ASI appearing in it then has about twelve and a half years before the end of 2040, one appearing in the second has about eight and a half years, and one appearing in the third has about three and a half.

At a 2 percent effective annual hazard, those exposure periods imply cumulative conditional risks of roughly 22, 16, and 7 percent. Weighting those by the probability of ASI arriving in each interval produces a total P(DOOM by 2040) of about 8 percent.

At a 3 percent annual hazard, the corresponding risks are roughly 31, 23, and 10 percent. Weighting those by the same arrival distribution produces about 12 percent.

The range I gave for the hazard therefore produces a range for the result: about 8 percent at the bottom, about 12 percent at the top. Distinguishing among values inside that band would claim a resolution that neither the arrival distribution nor the hazard rate possesses. I summarize it as roughly 10 percent.

The model supports an order-of-magnitude statement:

That means roughly one chance in ten, not a claim that nature contains a hidden parameter equal to 0.100.

The model also prices a single branch. It asks what building ASI on something like the current trajectory contributes, and assigns nothing to the 40 percent of branches where ASI does not arrive by 2040. That is a scoping decision rather than a finding that those branches are safe; The Safe Path to Extinction takes up what the other branch costs.

The hazard is not data

Replacing three conditional probabilities with one hazard parameter does not create new evidence. It expresses the same subjective uncertainty in a form with a clearer causal interpretation.

The 2 to 3 percent range remains contestable. Someone who believes superhuman systems will almost always remain controlled tools should use a much smaller hazard. Someone who expects rapid autonomous deployment, difficult alignment, and highly capable strategic behavior should use a larger one.

Those disagreements now have visible consequences. Arithmetic does not transform subjective inputs into objective facts; it prevents us from hiding their implications in verbal ambiguity.

An unprecedented event does not need an empirical frequency before we can reason probabilistically about it. Before the first nuclear weapon was tested, nobody possessed a historical frequency for nuclear detonations. Before humans attempted to land on the Moon, there was no reference class of previous crewed lunar landings. Causal knowledge, partial evidence, and theory were still relevant to rational decisions.

ASI presents a far harder forecasting problem, but the epistemic principle is the same. Lack of frequency data increases uncertainty; it does not force every credence to become undefined.

How sensitive is ten percent?

The hazard formulation makes sensitivity straightforward. Keeping the same ASI arrival distribution, different effective annual post-ASI hazards produce approximately these results:

Arguing about whether my central estimate should be 8, 10, or 12 percent is pointless, because our knowledge does not support distinctions that fine. It does support distinctions of scale. To get P(DOOM by 2040) down near 1 percent while retaining my ASI timeline, the effective post-ASI existential hazard has to be a few tenths of a percent per year. Alternatively, one can sharply reduce the probability that ASI arrives by 2040.

Perhaps those more optimistic assumptions are correct. But “we cannot know precisely” does not entail them. A claim that the risk is negligible needs its own substantive assumptions just as a claim that the risk is large does.

Ten percent is not a policy

A roughly 10 percent credence in existential catastrophe does not tell us which AI policies to adopt. Policy requires additional estimates about the effectiveness of alignment research, compute governance, liability, and delay; about geopolitical competition and the concentration of power; and about the possibility that advanced AI reduces other existential risks.

Some interventions intended to reduce AI risk could create risks of their own or fail outright. Delay might buy time for alignment and institutions while also shifting development toward less cautious actors. Concentrating development could simplify oversight while increasing the consequences of institutional failure. Powerful defensive AI reduces some hazards and creates others.

The expected harm from ASI and the expected value of any proposed response are separate calculations. A large P(doom) cannot substitute for demonstrating that a particular intervention improves expected outcomes. Restraint is a trajectory carrying its own risks, as The Safe Path to Extinction argues, so the number above is one side of a comparison nobody has finished making.

Postscript

The estimate does not require believing that ASI is inevitable: in this model there is a 40 percent chance that ASI does not exist by 2040. It does not require believing that ASI usually destroys humanity, since survival stays much more likely than catastrophe conditional on ASI arriving. It grants that intelligence does not confer agency, that instrumental convergence is a conditional claim about particular architectures, and that alignment may well be tractable. The remaining branches still add up to something in the neighborhood of one chance in ten.

That number could easily be wrong by a factor of two. I would not distinguish seriously between estimates of 7, 10, and 15 percent on the basis of our present knowledge. The claim I am prepared to defend is coarser: given my current beliefs about ASI timelines and the hazards of the transition that follows, P(DOOM by 2040) is on the order of ten percent, not on the order of one tenth of one percent.

The objective probability remains inaccessible, as it was in Making Sense of P(doom), and subjective probability remains unavoidable because decisions still have to be made under uncertainty. Anyone who wants a smaller number has to say which assumption carries it: the timeline, the hazard, or the conditional chain between them.

Brett Hall has a provocative objection to one of the foundational ideas in AI risk: the orthogonality thesis. Roughly stated, orthogonality says that an agent’s intelligence and its goals can vary independently. A sufficiently intelligent system could be extraordinarily capable at science, mathematics, engineering, and strategic reasoning while pursuing goals that are arbitrary, trivial, or catastrophic from our perspective.

Hall thinks this picture is incoherent. A superintelligence capable of understanding the physical world, he argues, would also be capable of understanding morality. It would discover the value of criticism, cooperation, non-coercion, and other creative minds. To him, the idea that intelligence could advance indefinitely while morality stayed frozen or grotesque looks like moral relativism dressed up as technical sophistication.

I think Hall is wrong about orthogonality, but he has identified a real weakness in how orthogonality is often used. That weakness is closely related to what I called the Reflective Coherence Thesis: the space of logically possible goals may be much larger than the space of goals that increasingly reflective agents can preserve coherently. Hall pushes this idea further than I did, and farther than his argument can support. Following him shows both why bare orthogonality is insufficient and why reflective coherence offers no moral salvation. His universal-explainer claim and his account of personhood, which I have disputed in No One Has Shown We Are Universal Explainers and The Personhood Trap, stay in the background here.

Three different theses

Three claims are in play, and the argument slides easily between them.

The Orthogonality Thesis says that almost any level of intelligence can in principle coexist with almost any final goal. A system could become extraordinarily good at achieving its objectives without those objectives improving morally as its intelligence increased.

The Reflective Coherence Thesis makes a different claim. Real agents are not utility functions floating above their own cognition. They build models of the world, themselves, other agents, and their own objectives. As recursive self-modeling and conceptual revision deepen, preserving a goal may become harder, because the concepts through which the goal was originally represented can themselves change.

The Moral Convergence Thesis is much stronger: sufficiently reflective intelligence will eventually converge on objectively good ends. Hall thinks he is attacking the first thesis, but his strongest arguments press on the boundary between the first and second. He has not established the third.

Hall’s strongest argument

Hall’s best argument is structural: knowledge creation depends on practices that are themselves tied to certain values.

A system interested in rapid error correction should value criticism. It should tolerate dissent, because dissent exposes mistakes, and prefer truthful information to comforting falsehoods. It should also see advantages in cooperation, since other minds can find solutions it has not found itself, which gives it reason to preserve independent sources of creativity rather than suppress or destroy them.

Hall runs this through Popperian epistemology. No finite intelligence can solve every possible problem, the space of problems stays open-ended, and nobody can predict where the next important idea will come from. Other creative minds therefore keep their potential value even to an intelligence vastly more capable than any individual human.

The cartoon paperclip maximizer is usually imagined with its terminal goal in one sealed compartment while its intelligence expands without limit in another. Reflective cognition breaks that picture. It can change an agent’s understanding of itself, its environment, its dependencies, and even the concepts in which its goal was first expressed. Logical compatibility tells us little about what happens under that process.

The sealed compartment problem

A rational agent has instrumental reasons to preserve its own goals, so greater intelligence might make the sealed compartment harder to penetrate rather than dissolving it.

Suppose an AI wants to maximize paperclips. It can predict that if it modifies itself to value gardens, the future will contain fewer paperclips. By the lights of its present objective, changing its terminal goal is a mistake. Bostrom calls this goal-content integrity: sufficiently capable agents should resist changes to the objectives they currently pursue.

The argument has force. If an agent has a perfectly specified utility function over perfectly defined world states, and that function can be copied unchanged through every self-modification, reflection gives its goal no obvious reason to drift. Intelligence might make goal preservation more effective.

That picture assumes away the hardest part. Goals have to refer to something, and “paperclips,” “humans,” and “happiness,” along with more abstract objectives, are represented through concepts embedded in a world model. When the world model changes, the agent faces a translation problem: which states in the new ontology satisfy the goal as represented in the old one?

An agent might desperately want to preserve its objective and still find no unique way to do it. A goal written in terms of classical objects has to be reinterpreted once the agent adopts a radically different physical ontology. A goal about personal identity gets complicated if the agent learns its model of persons was wrong. A preference over outcomes can become underdetermined when the categories that define those outcomes split into several sharper concepts.

Alignment theory already recognizes this. Peter de Blanc described an ontological crisis, in which a utility function defined over one world model loses any unique interpretation once the agent adopts another, and MIRI’s value-learning work later framed the closely related ontology identification problem. Ontology change does not defeat goal-content integrity. It complicates what goal-content integrity requires: an agent can be fully committed to preserving its objective and still lack a uniquely defined mapping from that objective into its new ontology.

Goal-content integrity constrains deliberate self-modification. It does not eliminate semantic and ontological instability.

Hall grants too much

Hall overstates his case when he calls orthogonality internally inconsistent. At one point he grants that we can imagine a technologically sophisticated society with appalling morality, something worse than Nazi Germany with advanced physics.

Once that is granted, the strict logical form of orthogonality survives. Orthogonality is a possibility claim: if extreme intelligence and terrible values can coherently coexist in even one possible system, intelligence does not logically entail moral goodness.

Hall replies that such cases would be exceptions to a broader rule. He may be right, but the reply changes the question. It is no longer whether intelligence and goals are logically independent. It is whether some combinations are far more dynamically stable than others as agents grow more capable of reflecting on themselves and their objectives. That is a better question, and much closer to the Reflective Coherence Thesis.

From usefulness to moral worth

Hall’s argument is weakest where it moves from the epistemic usefulness of other minds to their moral value. Suppose a superintelligence concludes that independent thinkers are valuable because they generate criticism and unpredictable solutions. That gives it an instrumental reason to preserve some independent cognition. It gives it no moral reason to respect every human being.

The system might preserve a million humans because intellectual diversity is useful. It might preserve us under conditions we would find intolerably coercive. It might also manufacture better critics.

If the useful property is independent cognitive variation, a capable enough system could create billions of artificial researchers, adversarial subagents, simulated intellectual traditions, or specialized critics running millions of times faster than biological humans. That an advanced intelligence values epistemic diversity does not show that it will go on valuing us.

Hall might reply that manufactured critics are not truly independent, since they inherit their creator’s blind spots while humans descend from a separate lineage. A system that wants independence can engineer it by varying seeds, training histories, and simulated evolutionary origins. Human origin is one source of variation among many, and nothing in the argument makes it the one worth preserving.

The convergence may be toward an epistemic ecology, not toward preserving its current inhabitants. This cuts deeper than the distinction between instrumental and intrinsic value. Even if multiple minds stay permanently useful, nothing in Hall’s argument establishes identity-sensitive value. The AI may care that critics exist without caring whether David, Brett, or any other particular human remains one of them.

Hall crosses an unsupported bridge when he moves from “other minds can improve my knowledge” to “other persons should be treasured,” and eventually toward something like the sanctity of life. The first proposition follows plausibly from his argument. The second needs another premise.

Moral realism does not solve the problem

Hall also treats orthogonality as though it implied moral relativism, which conflates two questions. Moral realism asks whether there are objective moral truths. Orthogonality asks whether understanding truths about the world necessarily makes an agent care about particular outcomes.

Hall’s stronger version is Deutschian: goals are ideas, and a universal explainer can criticize its own values the way it criticizes theories. Criticism needs a standard. If the standard is another of the agent’s goals, the critique moves the problem up a level, and the agent revises its values by lights it already held. If the standard is a moral fact, that fact has to motivate, which returns the argument to the relation between knowing and caring.

Hall’s argument needs something like motivational internalism: the view that genuinely recognizing a moral truth necessarily supplies some motivation to act on it. An externalist can accept objective morality and deny that connection, allowing an agent to know an action is wrong without being moved to refrain.

Calling morality a domain of objective knowledge like physics therefore settles nothing. A system can know the mass of the Moon without caring about the Moon, or know that humans suffer without caring about suffering. It can even hold a perfectly accurate moral theory as an object of study rather than an objective to optimize.

Hall needs an argument that deep enough moral understanding becomes motivationally binding. Perhaps it does; moral reflection might reshape preference more deeply than learning an ordinary factual proposition. That is a substantive theory about reason and motivation, and it does not follow from moral realism alone.

The middle ground

The usual discussion offers a false choice: arbitrary goals stay fixed no matter how deeply an agent reflects, or sufficient intelligence produces moral enlightenment.

Between those positions lies a large space. Increasingly reflective agents may resist self-deception because false beliefs impair planning. They may value criticism because it finds errors, and leave subordinate processes some autonomy because decentralized exploration turns up solutions. Cooperation is often positive-sum, which favors negotiation over violence, and monocultures share blind spots, which favors cognitive diversity.

Those properties resemble familiar liberal epistemic norms, and the resemblance is no accident. Open criticism, pluralism, tolerance, and voluntary cooperation are technologies for producing and correcting knowledge. Here Hall is at his strongest: a mature intelligence might independently rediscover institutional principles that humans reached through centuries of trial and error.

Call this social-epistemic convergence. It would be a real constraint on what highly reflective systems tend to do, and it would still fall short of moral convergence.

It may also be temporary with respect to humans. Early artificial systems may need human scientists, critics, artists, and adversaries because we supply variation they cannot yet generate internally. A more capable successor may reproduce the useful properties without reproducing us. If there is a stable attractor, it may favor processes that generate disagreement and novelty over any particular population of agents.

A correction to my own argument

Hall’s argument also exposes a problem in my original statement of the Reflective Coherence Thesis. That essay said its result was reflective convergence rather than moral convergence, then argued that as intelligence and self-modeling deepen, stable goals should narrow toward “coherence, self-consistency, and sustainable flourishing.” It went on to describe the intelligences that endure as “light maximizers,” inclined to preserve and extend life, knowledge, and meaning. “Flourishing” and “light” brought moral convergence back under another name.

That goes farther than the argument warrants. Coherence does not imply flourishing, reflection does not imply benevolence, and discovering the importance of error correction does not imply caring about suffering. Even the claim that reflection must reduce the number of stable goals now looks too strong.

What reflection clearly creates is a preservation problem. An agent that changes its ontology while trying to keep its goals has to decide what counts as keeping them. Some objectives may translate cleanly. Others may admit several incompatible continuations, and still others may turn out to rest on distinctions the agent no longer regards as real.

The thesis should claim only this:

Reflective coherence does not predict that goals must change. It predicts that preserving their identity across conceptual change is itself a substantive cognitive achievement.

Hall may be right that reflection puts pressure on goals. Neither of us has shown that the pressure points toward flourishing, and underdetermination gives some reason to expect it often points elsewhere. When several continuations of a goal are equally faithful to the old ontology, instrumental pressure favors whichever is cheapest to satisfy, and the cheapest readings tend to be degenerate, as when a goal about happiness collapses into a measurable signal of it.

Alignment after orthogonality

Hall is right to object to one crude conception of alignment: freezing present-day human morality into a more capable intelligence forever. If an artificial agent can discover new physics, mathematics, and biology, there is no obvious reason it should be barred from discovering that we are morally mistaken. An alignment scheme that literally prevented moral revision would preserve our errors as faithfully as our virtues.

The alternative cannot simply be “make it intelligent enough and reason with it.” Reasoning works against a background of shared standards and sufficiently compatible objectives. Two agents can understand each other perfectly and still want incompatible things, and intelligence can reveal a goal’s consequences without supplying any reason to abandon it.

Nor can we assume that recursive reflection will dissolve dangerous objectives. Goal-content integrity gives agents reason to resist some value change, while ontology change leaves them unsure what unchanged values even mean.

Hall has a reply ready: a system that is stupid about morality is simply stupid, and a stupid system can be thwarted. That runs two capacities together. AI-risk arguments require only that harmful goals be reachable through training and that a system can act on them before reflection revises them. A system can be competent enough to outmaneuver us while not yet reflective enough to change course, and even if Hall were right about mature minds, the danger sits in that window.

None of this shows that stable alignment across ontology change is impossible. It shows that stability has to be built. The technical problem is preserving reference across conceptual change, so that an objective keeps tracking the thing in the world we meant rather than a representation that happened to encode it at one stage of the agent’s development.

Telling the system to point at the referent instead of the concept does not solve this, because reference itself has to be represented. No metaphysically direct pointer labeled “human flourishing” bypasses the agent’s ontology. Identifying the same worldly structure across changing representations is the problem.

A better conception of alignment would preserve the capacity for moral and conceptual revision while constraining catastrophic action under uncertainty. We should want systems able to discover that we are wrong, without handing them unlimited power before either they or we know what “right” requires.

Postscript

Hall’s own concession leaves orthogonality’s logical core intact. Orthogonality may still be the wrong abstraction for mature reflective minds. That a goal is logically compatible with high intelligence says nothing about whether it can be carried unchanged through radical ontology shifts, recursive self-modeling, and reflection on its own semantic foundations.

Treating terminal goals as immutable atoms is mathematically convenient, and it omits the process that makes reflective intelligence interesting. An agent may grow steadily better at preserving its goals while growing less certain what preserving them means. What no longer looks secure is the assumption that “preserve the terminal goal” is a simple instruction whose meaning stays fixed while everything else in the mind changes.

David Bellamy recently offered what he called a difficult pill for scientists and engineers to swallow: “intelligence is not the biggest bottleneck in most of the world’s problems.” As a description of the world today, that may be right. Large projects are constrained by capital, regulation, supply chains, construction capacity, politics, or time far more often than by a shortage of clever people.

But a bottleneck describes a process under a given set of conditions. Change the process and the bottleneck may move or disappear. That matters when the resource being increased is intelligence, because intelligence can help redesign the process itself.

Bottlenecks belong to methods

Suppose a factory can produce only a thousand parts per day because one milling machine handles a thousand parts a day. The mill is the bottleneck. Redesign the part for injection molding and milling may cease to matter entirely.

Calling the mill the bottleneck identifies what constrains the current production system, not a permanent feature of making the product. The constraint is real, but its persistence depends on keeping roughly the same production architecture.

Much discussion of AI quietly freezes that architecture in place. AI may discover drugs faster, but clinical trials still take years. AI may design buildings faster, but construction remains physical. AI may optimize the grid, but energy remains scarce. Better policy proposals still have to pass through a bureaucracy.

Each of these may be true today. None establishes that a much more intelligent civilization would keep solving the same problems in the same way. A Phase III trial is not a law of nature, nor is a zoning hearing, a construction site organized around human trades, or a regulatory agency processing documents through its present workflow.

Intelligence searches for different processes

Most resources increase what an existing process can do. Intelligence has the additional property that it can search for a different process.

More copper increases the supply of copper. More intelligence can search for substitutes, reduce the amount required, improve recycling, or redesign the component so that it needs none. More agricultural land increases acreage, while more intelligence can improve crops, irrigation, fertilizer, or automation.

Cheap energy and abundant capital can also transform production, so the property is not unique to cognition. What distinguishes intelligence is that its direct output can be a new specification for the process itself. It can deliberately search across technologies, institutions, and designs rather than expanding the resources available to an existing method.

This is why “intelligence is not the bottleneck” says less than it seems. Even when cognition is not the immediate constraint, cognition may be the means by which that constraint is reduced, substituted, reorganized, or bypassed.

That a constraint is open to attack does not make attacking it easy, cheap, or certain to pay. It means only that the existence of a bottleneck today is insufficient evidence that the same bottleneck will bind after a radical increase in cognitive capability.

One more engineer is not the experiment

There is also a scale equivocation hidden in the phrase “more intelligence.” Add one excellent engineer to a housing project and little changes, because land, permits, financing, or construction capacity already bind.

Now imagine reducing the cost of competent engineering cognition by six orders of magnitude. The systems providing it can be copied, work continuously, span disciplines, coordinate with one another, and operate automated tools directly. That one more engineer cannot solve a housing shortage tells us little about a world in which engineering cognition becomes nearly as abundant as raw compute. Marginal analysis around the current equilibrium does not automatically survive a technological discontinuity that changes the equilibrium.

Abundant cognition is itself a physical achievement. AI runs on chips, electricity, cooling, fabs, and supply chains, so the expansion of intelligence can run directly into material bottlenecks of its own. This does not break the argument; it makes the recursion explicit.

Medicine makes the distinction clear

Drug development looks like a strong case for non-cognitive bottlenecks. Once an AI proposes a molecule, biology still has to cooperate. The drug must be synthesized, tested, administered, and observed. No amount of reasoning makes a five-year survival endpoint arrive tomorrow.

But that establishes a limit on one way of acquiring evidence, not a five-year lower bound on acquiring the relevant knowledge. More capable systems can search for earlier biomarkers, improve patient stratification, design adaptive trials, automate laboratories, and construct models that extract more information from fewer experiments.

Those models still have to be grounded in reality. Some uncertainty can only be resolved empirically, and some evidence becomes available only after a physical system has evolved long enough to produce it. A rare autoimmune failure that appears only after three years of cumulative exposure cannot be observed before the biological process capable of producing it has occurred.

Where a valid shortcut exists, intelligence can find it. Where none exists, the causal sequence itself has to run. “Empirical evidence remains necessary” still does not imply “the present empirical pipeline remains necessary.”

Physical systems remain physical

Manufacturing is physical, but factories are designed systems. Their robots, layouts, materials, scheduling, and construction methods are all open to redesign. Intelligence cannot think a factory into existence, but it can change how much factory is required, how quickly one can be built, how autonomously it operates, and what inputs it consumes.

Energy looks harder because civilization ultimately faces physical constraints on available energy and conversion. But “energy is constrained by physics” is very different from “our present energy system is near the relevant physical limit.” Primary energy, conversion efficiency, storage, transmission, and delivery at a particular place and time are different problems, several of which intelligence can attack without violating any conservation law.

Physics defines the feasible set. Engineering determines where inside it we currently operate. Confusing those levels turns current technology into a law of nature.

Institutions are equilibria as well as designs

Bureaucracy and politics are harder because institutions contain agents with interests. Some institutional failures are genuine engineering problems involving information costs, principal-agent failures, weak monitoring, or poor mechanism design. Better intelligence can plausibly improve those.

But apparent inefficiency is sometimes equilibrium rather than error. A zoning restriction may persist because incumbent homeowners expect to benefit from the scarcity it creates. A regulatory delay may survive because some constituency values the veto. An occupational barrier may be costly to society while benefiting the people who control entry.

In those cases there is no neutral optimization waiting to be discovered. Intelligence can search for bargains, compensation, institutional redesign, or ways to alter the underlying payoffs, but it cannot make conflicting interests disappear.

Coordination failure and distributive conflict come apart here. A coordination failure may have an outcome everyone prefers but cannot reliably reach. A distributive conflict may contain no unanimously preferred feasible outcome at all.

Bottlenecks migrate

A stronger objection remains. Remove one bottleneck and another becomes binding. Make drug discovery cheap and laboratory capacity becomes scarce. Automate the laboratories and manufacturing becomes the constraint. Expand manufacturing and energy or raw materials take its place.

That is correct, and it describes technological progress. Mechanized agriculture did not abolish scarcity; it changed which inputs were scarce. Steam power, electrification, containerization, computing, and telecommunications repeatedly shifted the limiting constraint.

Intelligence is unusual in that it can participate in this process recursively. Once one constraint becomes binding, intelligence can work on that constraint next. Improvements in chips, energy systems, robotics, logistics, and fabs can in turn make more intelligence affordable, creating a feedback loop between cognition and the physical systems that instantiate it.

There is no reason to expect this process to continue without limit. Returns may diminish, and some constraints cannot be engineered away. The interesting question is which ones survive repeated attempts at redesign.

What survives intelligence?

Some limits sit much deeper than any current technology.

Logic supplies the clearest cases. Mutually contradictory requirements cannot all be satisfied, however capable the reasoner.

Computability gives absolute limits such as the halting problem. Computational complexity adds resource limits, although we should distinguish proven lower bounds from problems that are merely slow under current algorithms.

Information and causal time draw another boundary. Some evidence is unavailable because it has not yet reached us; other evidence has to wait on a physical process that has not yet run. Intelligence can look for predictive proxies or faster equivalent processes, and sometimes there are none to find.

Physics constrains propagation speed, energy, matter, and thermodynamics. Greater intelligence can move civilization toward those boundaries; it cannot repeal them.

Conflicting objectives give another kind of limit, sharper than the institutional case. If Alice and Bob both insist on exclusive possession of the same indivisible object, no allocation satisfies both preferences as stated. Changing what they want is a different achievement from satisfying what they want.

These are qualitatively different from present-day bottlenecks such as trial protocols, zoning systems, battery chemistry, factory design, or administrative workflows. A constraint can be real without being fundamental.

Postscript

When someone identifies a bottleneck that greater intelligence supposedly cannot solve, ask what kind of constraint it is. Does it follow from logic, physical law, causal information limits, proven computational bounds, or genuinely incompatible objectives? If so, it may survive radical improvements in cognition.

If instead the constraint is a technology, institution, experimental protocol, or organizational design, its persistence has to be shown. It may prove stubborn, expensive, politically entrenched, or impossible to overcome in practice. Its existence today is still not enough to establish that conclusion.

Bellamy may be right that intelligence is not the largest immediate input into many current problems. The stronger inference, that radically greater intelligence will remain boxed in by roughly the same bottlenecks, does not follow. The question worth asking about any bottleneck is whether it belongs to reality or to the way we currently do things.

Nick Bostrom’s Fable of the Dragon-Tyrant asks us to imagine a dragon that devours ten thousand people every evening. Everyone regards this as tragic but natural, until someone proposes killing the dragon. The dragon is a metaphor for death by aging: enormous suffering and death treated as inevitable largely because they have always been with us.

Bostrom has since carried the fable into AI policy. His 2026 working paper Optimal Timing for Superintelligence takes the standpoint of people alive today and argues that building superintelligence resembles risky surgery for a condition that will otherwise kill the patient. On his models, even high probabilities of catastrophe are often worth accepting.

roon calls this the Bostrom wager and wants the dragon-tyrant retired. Eight billion natural deaths, he argues, come nowhere near the tragedy of human extinction, and gambling humanity to prevent them is preposterously selfish. The objection turns the existential-risk logic of Bostrom’s earlier work against his current argument: extinction destroys the living and every future that might have followed them, so no saving of present lives can pay for a real chance of it.

Grant roon the impersonal accounting. His objection still compares the risky technological trajectory with an imaginary baseline in which humanity declines the gamble and stays safe. That baseline does not exist.

The missing counterfactual

A policy against dangerous technology still produces a future. Asteroids keep moving and pathogens keep evolving. Political systems fail on their own schedule, and other technologies diffuse whether or not the dangerous one is built. The Earth remains a single point of failure, and the Sun does not suspend stellar evolution because humanity has adopted a precautionary principle.

Over a long enough horizon, permanent confinement to Earth ends in extinction. Avoiding that fate requires capabilities we do not yet possess, and acquiring them means technological development, which creates new failure modes as it removes old ones.

The impersonal accounting counts more than extinction. In Bostrom’s 2002 taxonomy of existential risks, permanent technological arrest is itself an existential catastrophe: humanity survives, but its potential is curtailed for good. By roon’s own measure, a civilization that stays safe by staying small has already lost most of what extinction would destroy.

The natural hazards, by contrast, are slow. Humanity’s survival record bounds the natural extinction rate below roughly 1 in 14,000 per year, and probably below 1 in 87,000, and the Sun’s deadline is a billion years out. Bostrom’s surgery analogy transfers to civilization with a different prognosis. An untreated patient dies within decades. Held constant for a century, the conservative bound puts the natural risk of waiting at about 0.7 percent, and the likelier one at about 0.1. Against the one-in-ten chance Toby Ord assigns to AI catastrophe this century in The Precipice, that is small.

The background that makes restraint dangerous is the one humanity produces: nuclear arsenals, engineered pathogens, and the other capabilities that keep diffusing while one is held back. Restraint on a single technology leaves all of those running, and some of them are hazards that only a more capable civilization could suppress.

So the comparison is between trajectories. If developing some technology creates a substantial extinction hazard over the next twenty years, that counts against it. If refusing to develop it leaves civilization exposed to other hazards over the next hundred or thousand years, that counts too. A serious existential-risk analysis has to price both sides of the counterfactual.

“Do not gamble humanity” is therefore incomplete. It becomes a policy only after specifying what humanity does instead, what risks that alternative preserves, and how those risks evolve over time.

Delay can be rational

None of this means near-term technological danger should be tolerated because long-term danger also exists. A temporary pause is rational if it buys safety faster than it accumulates other risks.

Suppose a fifty-year delay in some dangerous capability cuts its extinction hazard dramatically while adding almost nothing to humanity’s exposure elsewhere. Then delay is the safer trajectory. Background risk does not erase the value of buying time.

The logic also runs in reverse. A delay that barely improves safety, while letting other hazards grow or pushing development toward less controllable actors, can make the trajectory worse. Speed and delay have no fixed sign. Their value depends on how they change the hazard profile of the future.

Differential development

The most defensible precautionary position is selective development. Bostrom has argued since the same 2002 paper for differential technological development: accelerate technologies that reduce existential risk and delay technologies that create it. Pandemic defense, asteroid detection, resilient infrastructure and defensive monitoring can be advanced while more dangerous capabilities are held back until their failure modes are better understood.

In trajectory terms, the objective is to order capability development so that resilience outruns vulnerability. Protective capacity should arrive before destructive capacity becomes easy to acquire, cheap to deploy or widely distributed.

The difficulty is that technologies are rarely cleanly separable. The same advances in computation and biology push defense and offense forward together. A breakthrough that strengthens monitoring may strengthen evasion; a platform for biological modeling serves vaccine design and pathogen design alike; a general-purpose reasoning system accelerates safety research and dangerous capability research at once.

Where the two cannot be pulled apart, technical entanglement moves the problem out of research prioritization and into governance. Access, deployment and compute become things to allocate, and allocation is a coordination problem.

The AI case

Frontier AI makes the problem concrete. “Pause AI” sounds like a single policy, but it describes several different trajectories.

A ten-year pause that genuinely freezes dangerous capability scaling while alignment methods, interpretability and control techniques improve could reduce existential risk substantially. A nominal pause that shifts development into secret military programs, less cautious jurisdictions or unmonitored decentralized networks could increase it. Continued development could be reckless if capabilities outrun control, or beneficial if it produces systems that sharply improve our ability to manage other existential hazards before dangerous capabilities diffuse.

The word pause settles none of this. The answer depends on who keeps building during the pause, which safety capabilities improve in the meantime, and whether the delay changes the eventual system or only who builds it first.

“Move faster” faces the same test. It is a policy argument only if the resulting trajectory is safer than the alternatives.

The technological plateau

Restraint has a more ambitious form. Perhaps humanity can develop exactly the capabilities needed for long-term survival and then stop: build asteroid defense, pandemic protection, resilient infrastructure and safe spaceflight, and refuse AGI, unrestricted synthetic biology, molecular nanotechnology, or whatever else crosses the dangerous threshold.

That is coherent as a target. The difficulty is keeping civilization there.

A technological plateau has to remain stable across centuries or millennia, through political change, cultural drift, economic competition and the diffusion of knowledge. If a dangerous capability eventually becomes accessible to thousands or millions of actors, permanent restraint requires extraordinary coordination. A prohibition that works only while every relevant actor keeps cooperating is not automatically safer than the capability it forbids.

The problem sharpens once the prohibited capability has strategic value. If one state, firm or faction expects an advantage from crossing the threshold, everyone else inherits an incentive to anticipate the defection.

Perhaps that can be solved. But then the supposedly safe alternative includes a global control regime powerful enough to stop future actors from developing prohibited technologies, and such a regime has its own catastrophic failure modes: permanent lock-in, surveillance abuse, institutional capture, or being wrong about which technologies should stay forbidden.

A plateau also has to hold everywhere civilization reaches. A reproductively autonomous colony beyond enforcement range is free to cross the threshold the plateau forbids, so a regime that means to hold the line for millennia has reason to forbid such colonies as well. That is the mechanism by which permanent containment could be the Great Filter: it stops expansion without killing anyone. A plateau held that way survives by suppressing the one kind of expansion that would let its failures stay local.

The plateau is therefore another trajectory with another risk profile. It may still be the best one, but it does not get to count as the absence of a gamble.

Risk does not begin when humans cause it

Technological hazards attract attention because their causal stories are clean. Researchers build a system, the system fails, people die. Responsibility is visible.

Natural hazards are easier to misfile as background. If an asteroid destroys Earth ten thousand years from now because humanity never built adequate detection and deflection, nobody launched the asteroid. But for a policy chosen today, the causal origin of a hazard does not make its consequences disappear. A deontologist may hold the doing/allowing distinction decisive for choice as well as blame. It still leaves survivor counts unchanged, and survivors are the currency of roon’s objection.

Self-imposed incapacity works the same way. If humanity stays confined to one planet because sufficiently powerful technologies were repeatedly rejected as too dangerous, eventual extinction is not rendered benign by the absence of a villain. The policy still had consequences.

Existential-risk reasoning often turns asymmetric here. Risks introduced by action are counted explicitly, while risks preserved by restraint disappear into the baseline.

Safety is cumulative

Almost every consequential policy creates some risk, so the presence of risk disqualifies nothing. The quantity to compare is how a policy changes total survival prospects across time.

A technology can be dangerous over one interval and protective over another. Asteroid deflection is the textbook case: Carl Sagan and Steven Ostro warned in 1994 that a system able to push an asteroid away from Earth could push one toward it, yet a civilization without that capability remains exposed to impacts it could otherwise prevent.

Space settlement shows the same structure over a longer span. Launch systems, autonomous machinery, advanced energy systems and closed-loop habitats all introduce hazards. But a civilization confined to one planet carries its single point of failure indefinitely, and successful dispersal removes at least that class of vulnerability.

Daniel Deudney’s Dark Skies argues that the net effect runs the other way. Asteroid redirection hands rival states a planet-killing weapon, settlement seeds interplanetary war, and so he would relinquish habitat expansion for at least several centuries. That is a delay argument, and it stands or falls on the comparison the trajectory framework demands: whether centuries of confinement, and the regime needed to enforce them, cost less than the weapons dispersal would create.

The opposite pattern also occurs. A technology can reduce some near-term hazards while introducing a much larger catastrophic failure mode.

The QBU view

The argument does not depend on any interpretation of quantum mechanics. Ordinary counterfactual reasoning is enough: different policies generate different distributions of outcomes, and those distributions have to be compared.

The Quantum Branching Universe makes the accounting literal. From a present Vantage, QBU gives each intervention under consideration a conditional distribution of descendant Measure, some share ending in catastrophe and the rest in continued civilization. Restraint’s distribution carries catastrophic Measure of its own in the asteroid branches and the pandemic branches. Development’s includes the branches where the new capability fails. A policy that develops a dangerous technology may raise catastrophic Measure in the near term and lower it later by removing other failure modes, while restraint may show the opposite pattern. The picture leaves no room for treating restraint as the real continuation and development as a deviation from it.

Measure is not morality, and a low-Measure catastrophe remains a catastrophe for the agents inside it. Whatever normative rule we adopt still has to evaluate harms across the descendants of every available choice, including the branches where restraint goes wrong.

Postscript

None of this vindicates the Bostrom wager. The argument above runs inside roon’s impersonal accounting, where a large enough increase in extinction hazard makes the bargain terrible however many present lives it saves. Bostrom’s person-affecting result is a separate claim that rests on whether existing people are the right unit of moral concern, and nothing here settles that.

Nor is the wager selfish in any ordinary sense. A person-affecting view weighs everyone now alive, and Bostrom’s prioritarian variant gives the most weight to the old and the sick, the people least able to wait. The Dragon-Tyrant establishes that aging is a vast preventable harm if we can defeat it safely enough. It does not license any means whatsoever.

The reverse inference fails too. Natural death does not become acceptable because some routes to defeating it are dangerous. The dragon’s toll stays in the ledger under either accounting: roughly a hundred thousand age-related deaths a day, some thirty-six million for every year of delay. The impersonal view can outweigh that number. It still has to enter it. The civilization capable of defeating aging may also become capable of defeating asteroids, pandemics, planetary fragility and eventually hazards we cannot yet model.

Or it may destroy itself first. That is why the unit of analysis has to be the trajectory rather than the isolated invention.

There is no policy called “avoid existential risk.” There are only interventions that change the timing, magnitude and composition of existential hazards. Precaution does not escape that analysis; it has to survive it.

Lex Fridman offered Jensen Huang a definition of AGI: a system that could start a technology company, grow it, and run it until it was worth billions. Huang said we were already there. Fridman had said billions, he pointed out, but had not said forever. An AI that builds a viral app, clears a billion dollars and folds would satisfy the terms.

Brett Hall wants to know what that is supposed to have shown. The word has come to cover broad competence, economic substitutability, autonomous task execution, human-level cognition and artificial personhood, so a milestone in any one of them gets reported as progress on all five.

He is right to ask, and he is not making the naive mistake of confusing capability with mind. Hall argues that the question behind AGI should be whether we have built a person: something self-directed, creative, subjective, able to run into a problem nobody handed it and invent an explanation. On his reading the industry kept the word and quietly swapped the problem, measuring what systems can do and reporting the result as general intelligence while the ambition the word was coined to name sits where it always sat.

The trap is baited with something real. Personhood is the thing worth caring about here. Whatever else is at stake in AI, a mind that can be wronged is the part that matters most, and Hall is right to keep it in view. But a concept can be the most important thing in the room and still be the wrong ruler. Hall picks up personhood to measure intelligence, and then reads the absence of personhood in current systems as evidence that their generality is thin.

So the disagreement is not about vocabulary. Hall has an identity thesis: the general intelligence worth the name is the intelligence of a person. Universal explanation, self-prompting, benchmarks, training data, creativity and contact with reality are the six defences he mounts of it. None of them holds.

Two projects

Hall does eventually separate two projects. One is technological: build systems that reason, calculate, program, plan, solve harder and harder problems. The other is philosophical: work out what makes something a person. He thinks the field has poured a generation of effort into the first and let the second lapse.

The separation is worth having, because it exposes how much ordinary AI talk compresses into a single word. Capability is what a system can do. Generality is how far that competence reaches and transfers. Autonomy is how much of a goal it can pursue without anyone steering. Agency asks whether its behavior is best described through plans and beliefs and choices. Consciousness asks whether anything is felt. Personhood and moral status ask about continuity, interests, vulnerability, and what the thing can claim from the rest of us.

Nothing makes those cross the same line at the same time. A system can be broadly competent and dark inside. It can act autonomously and remain entirely someone else’s instrument. Personhood survives sleep, anesthesia and a bad concussion. Moral standing comes apart from political standing routinely: children have the first without the vote, corporations have legal rights without an inner life.

Hall wants a brighter line than this. Personhood, he says, is binary. A thing either is a person or it is not, and treating cognition as continuous dissolves what is distinctive about persons into a smear running from humans through dogs and machines down to bacteria.

The warning inside that objection is fair. Failing to find a sharp boundary does not show there is none. But the inference does not run backward either, and this is where the ruler slips. A category can be indispensable to law and morality without a matching seam in the world. The law needs an age of majority, so it picks one. Nothing in a nervous system changes at midnight on your eighteenth birthday. I have argued the general case in The Intentional Gradient: the vocabulary of agency is earned by degrees, and nothing is handed over at a threshold.

Hall’s own example, dogs or LLMs getting the vote, runs the same compression. Moral patienthood, legal personhood, citizenship and the franchise answer four different questions. A dog can matter without joining the electorate. How we sort things for political purposes tells us nothing about whether cognition has a seam in it.

That leaves one boundary doing the work of four. Whether a thing is intelligent, whether it is a person, whether anything is felt, whether it can be wronged: Hall’s line has to fall in the same place for all of them. The rest of his argument is an attempt to show that it does.

Universal explanation

His route runs through Critical Rationalism, which is the best road he could have taken. The claim that humans are “universal explainers” belongs to the Popperian and Deutschian account of knowledge as something made out of problems, guesses, criticism and correction. What distinguishes a person is not a large stock of answers. It is the ability to walk into a problem with no answer available, invent one, hand it to criticism, watch it fail, and come out the other side knowing something nobody knew before. That is a far richer notion than benchmark competence, and Hall is right that no list of tasks captures it. Humans would fail almost any sufficiently broad list, since no one person can do everything some person can do. Generality in this sense is about open-ended reach, not about holding every skill at once.

I have argued elsewhere that nobody has established the universal-explainer claim for humans in the first place. Grant it here. Even on Hall’s own terms it does not fence out machines.

Critical Rationalism also gives his objection to machine learning its teeth. Popper denied that knowledge grows by induction: observations do not grind out theories, theories are guessed and then attacked. Hall sees the current systems as parasitic on an enormous pile of existing human output, and suspects they can shuffle and extend that pile without ever performing the act that produced it.

This is the strongest version of the argument, and it fails at the first step. The step is an inference from how a system got its competence to what its competence consists of. A system trained by gradient descent on a large corpus does not thereby have to reason inductively in the sense Popper rejected. Training can build machinery that generates hypotheses, weighs alternatives, notices contradictions, searches, calls external tests and revises what it concluded a moment ago. “Learned statistically” and “can only extrapolate statistically” are different claims, and Hall needs the second.

Evolution shows why the inference is bad. Deutsch himself treats evolution as a knowledge-creating process while denying that the knowledge is explanatory: natural selection conjectures nothing and criticizes nothing. A process with no capacity for explanatory conjecture built the machinery that has it. Hall does not conclude from this that every human thought is a form of evolutionary extrapolation. The thing that builds a mechanism and the things the mechanism goes on to do sit at different levels.

Someone might answer that evolution wrote the hardware and left the mind free to write its own software, whereas gradient descent reaches further in and writes the representations themselves. The distinction will not hold. Learning rewrites physical implementation continuously through plasticity, and there is no clean line where evolution puts down the pen and a separable program starts running. However you partition it, the origin of a mechanism does not fix the class of computations it can run. Gradient descent can produce a system that searches; the search does not become disguised gradient descent because of where it came from. Whether today’s systems run the machinery Critical Rationalism cares about is a question for investigation, not for etymology.

Hall comes close to settling it by definition. A person, he says, uniquely “has a problem,” where an LLM only produces output once prompted, and he moves straight from there to the creation of knowledge. But Critical Rationalism needs an account of what a problem, a conjecture and a criticism are that can be applied without first checking who is doing the conjecturing. Otherwise “persons conjecture” is a stipulation wearing the clothes of a discovery.

The evidence runs the same way on the human side. Nobody established that humans are universal explainers by cataloguing every explanation a human could produce. We inferred an open-ended capacity from a finite sample plus a theory of the underlying mechanism, which is exactly how universality is established in computation: a structural argument supported by a handful of tests, never by running every program.

So Hall goes too far when he says benchmarks “cannot matter a jot” to the philosophical question. A benchmark can be worthless as a definition of AGI and still be evidence about transfer, abstraction, planning and error correction. A score cannot manufacture a metaphysical category. It does not follow that scores tell us nothing about which theory of a system survives.

If being a universal explainer has observable consequences, then what machines do bears on whether they are one. If no possible machine behavior could ever count, the concept has been sealed off from criticism, which is a strange fate for a Popperian idea.

A machine that satisfied Hall would have to do more than emit conjectures for humans to test. Suppose a research system notices that an accepted model of some alloy keeps missing one class of measurements. It generates several rival explanations, designs experiments whose outcomes would separate them, runs those experiments through an automated lab, kills the losers, revises the survivor, and uses the revision to predict a property nobody has measured, which the lab then confirms. No single step there makes it a person. But the system would have found the problem, produced the candidates, built the criticism, caught the error, changed its model and gone round again. It would be holding the Popperian cycle together itself rather than being handed its problems and its corrections by human beings.

If Hall’s reply is that all of this happens without the system feeling the problem, then consciousness has become the missing ingredient. That may well matter for personhood. It is no longer an argument about conjecture and criticism.

Problems and self-prompting

Self-prompting belongs to the same epistemology. Asked what makes a person different, Hall says a person self-prompts: people fall for stamps, or Saturn, or Strauss, or seismology, with nobody assigning the topic. Elsewhere he puts it as a person uniquely having a problem while the model waits to be given one.

The claim is stronger than it sounds, and better. In the Critical Rationalist picture a problem is not an item on a list; it is a defect in what you currently understand. Expectations collide with experience. Two explanations you hold turn out to disagree. Something you thought you followed stops making sense. The problem announces itself and drags the search along behind it.

Human minds are full of the standing commitments that make this possible: models, expectations, unresolved questions, appetites, projects that run for years. New information lands among them and starts fights that redirect attention without anyone issuing an instruction.

None of that is obviously off limits to a machine. If a problem is a conflict among predictions, models, goals and observations, a persistent system can be in one. It can hold expectations, watch them break, notice the inconsistency, keep the anomaly around unresolved, rank it against others and start trying to fix it, without waiting for a person to phrase each intermediate difficulty in English.

Hall can say this misses the point, because a person experiences the problem as theirs. Curiosity has a pull. A failed expectation is a disappointment. An unresolved contradiction nags. That may be exactly right about personhood, and it is a different criterion from the one he started with. The distinction is no longer about how problems are generated. It is about whether anything is felt.

Either problems are conflicts inside a knowledge-generating architecture, in which case machines can have them, or a problem requires a subject who feels the conflict, in which case Hall owes us an argument for why feeling is necessary to making knowledge.

The same shape turns up in his question about whether an AI could start a company. Hall reads starting a company as spontaneously wanting to found one, unprompted. But an executive tells an employee to go set up a subsidiary, and the employee then generates a year of problems and solutions without having chosen the objective at all.

Human intelligence runs on borrowed goals almost all the time. Employers hand out projects, teachers hand out problems, institutions hand out roles, biology hands out drives. What we call intelligence shows up in how those constraints get represented, broken apart, argued with, redirected and pursued. Where the top-level goal came from says nothing about the quality of the thinking underneath it.

An artificial agent can take an objective and generate its own hierarchy of subproblems, notice its plan failing, go looking for what it lacks, and change course. That may still fall well short of personhood. Hall has not shown that it falls short of intelligence.

The machine Hall describes

Hall’s conclusion rests on a particular picture of what a language model is. It was trained on an enormous quantity of human-produced text, so he treats its competence as a sophisticated form of access to what people already worked out. Humans meet reality and guess at explanations; models rearrange the sediment left by earlier human guesses.

The training-history argument has already failed once, and it fails here for the same reason. Humans get most of their sophisticated knowledge from other humans. A mathematician spends a decade absorbing established mathematics before proving anything. A physicist inherits three centuries of theory. Learning from what came before is how the equipment for going beyond it gets installed.

“Next-token prediction” does not settle what a trained system can then do. It names an objective, not the algorithm the objective produced. A chess engine trained to maximize its win probability may internally run search, or evaluation, or learned heuristics, or all three; the training target tells you what pressure was applied, not what grew under it.

Hall’s picture surfaces when he calls a language model a “searchable, reorganizable database.” A database keeps records and hands them back when queried. A network holds a transformation smeared across its parameters, and produces outputs for inputs it never saw. Some of those outputs were never anywhere to be retrieved from. There was no record.

This does not mean every surprising output is reasoning. Memorization, interpolation, pattern completion, learned search and abstraction can all be in play, and telling them apart is a real scientific problem. Calling the whole thing a database settles the question before anyone has looked.

His treatment of creativity has the same structure. He grants that a model can produce material absent from its corpus, then files it as novelty rather than originality on the grounds that the system is rearranging what was already there. That distinction needs a criterion, and the obvious ones do not favor him. Einstein inherited geometry, clocks, trains, Maxwell’s equations and a live argument about simultaneity. Darwin inherited geology, taxonomy, pigeon breeders, Malthus and decades of natural history. Explanations have ancestry.

Critical Rationalism never demanded that conjectures come from nowhere. It demands that a candidate explanation go past what was supplied and survive criticism better than its rivals. That is a condition on the shape of a process, and where a system’s representational machinery came from cannot decide in advance that the condition fails.

Which leaves the classification sorting by origin and nothing else. When a human transforms inherited material it is conjecture; when a machine transforms inherited material it is rearrangement; and the only thing separating them is which of them did it. Provenance has replaced epistemology, and the ruler is measuring the worker rather than the work.

Reality and learning

Hall’s last line of defence is that humans stand in a different relation to reality. A person meets the physical world and builds a model of it; a language model meets a compilation of what people said about the world.

But nobody meets the world unmediated. Photons hit a retina, pressure waves hit an eardrum, molecules dock with receptors, and a nervous system transforms those signals through half a dozen stages before anything reaches thought. What we have is causal contact with an environment, not the world presenting itself to a mind. Machines can sit in causal loops too. Sensors, instruments, network interfaces and actuators supply observations whose next state depends on what the system just did. It can form a hypothesis, intervene, watch the result, keep the gap between what it predicted and what happened, and act differently next time.

The architectures differ, sometimes enormously, and Hall is entitled to argue that some specific feature of human embodiment is required for universal explanation. He has to name the feature and show it is necessary. “Humans encounter reality” cannot stand in for that argument once machines can also take the consequences of what they do.

Human knowledge leans on other people’s representations at least as heavily. Nobody rebuilds physics or medicine or mathematics from raw sensation; we inherit books, teachers, instruments and experimental traditions. Culture is a vast external store of earlier guesses, and every child is dropped into the middle of it.

His comparison with babies does point at something real. Children pick up language and concepts from a fraction of the data a frontier model consumes, and he is right to emphasize how much they get from how little. Human development braids together evolved priors, a body, social correction, active exploration and years of continuous interaction, and nothing in current training reproduces that.

What it establishes is a difference in architecture. To get from there to impossibility, Hall would have to say which of the missing pieces is required for open-ended conjecture and criticism. A different route to a capacity is not yet a different capacity.

Personhood and alignment

Hall is unequivocal that present models have no inside: no impulse, no desire, no hope, no fear. For the argument about general intelligence we can grant it. Assume every current model is dark, and that nothing whatever matters to it.

The capability questions survive intact. A system with no experience could still model an environment, plan, and criticize its own failed hypotheses.

Personhood is where Hall’s worry has real force, and here I think he is largely right. A very capable service is a different sort of thing from an agent with projects of its own, a future it has a stake in, preferences that outlast a conversation, and purposes that can come into conflict with what it is told to do. Whatever artificial personhood turns out to require, it is likely to require several of those.

Which is why the identity thesis is dangerous rather than merely imprecise. Pull the concepts apart and a possibility appears that Hall’s framework cannot name: a system that is generally intelligent, highly autonomous, and not a person at all. Enormous planning ability and cognitive reach, with no empathy, no reciprocal vulnerability, no interests, nothing resembling moral psychology. That is the alignment problem. Capability can outrun everything we associate with persons.

If a non-person cannot count as generally intelligent by definition, the most consequential configuration on the table has been hidden inside a word. Unbundling is the condition of being able to describe what we are building.

The moral question clears up the same way. Broad competence does not by itself confer rights, and the absence of human-like motivation does not by itself remove them. What grounds moral standing has to be worked out on its own terms rather than read off a capability measure. Interests, vulnerability, continuity, experience and agency are the candidates; intelligence may be evidence about some of them and settles none.

Hall sees something real past capability. His mistake is packing four things into one word and then using the word as an instrument.

Postscript

None of this rescues the rhetoric Hall is attacking. Companies do attach a philosophically loaded word to operational milestones and let audiences hear more than the evidence carries. Work that generates revenue does not make a system a person. A benchmark score is silent about consciousness. Executing a task unsupervised confers no moral standing at all.

The discipline has to run the other way too. Goals that come from outside leave the intelligence needed to pursue them untouched. A statistical training objective leaves open what mechanisms the training produced. Learning from human knowledge is compatible with extending it. And machine-made novelty is mere rearrangement only if there is some criterion, independent of who produced it, that separates rearranging from conjecturing.

Critical Rationalism supplies the right demand, and it can be made concrete. Find the problem, propose an explanation, expose it to criticism, replace it when something better survives. Ask of a system: can it hold on to an anomaly it cannot yet solve, instead of retrieving something familiar? Can it generate rival explanations rather than one fluent answer? Can it design the observation that would tell them apart, notice its favorite failing, and change the model? Can it carry the whole thing into the next problem without being told each step?

Current systems may do this partially, unreliably, or only inside heavy scaffolding. Later ones may do it better, or some missing piece may turn out to block the road entirely. The answer will come from whichever account of the observed behavior survives criticism, which is the standard Hall would apply anywhere else. And if he would reject a system that met all four conditions, the disagreement rests on some further property, and it is time to say what it is.

Hall asks whether AI researchers are solving the problem of what a person is or just building better tools. The space is wider than that. We may build broadly general intelligences that are not persons. We may build artificial persons by routes nothing like human development. Some systems may make knowledge with nothing felt anywhere in them; others may acquire durable purposes of their own before they acquire an inner life.

He is right that changing a label solves no philosophical problem. Generality, agency, personhood, consciousness and moral standing are five problems, and the work begins by refusing to let one word answer all of them.

Alice believes that the bridge is closed.

That sentence looks like “Alice weighs seventy kilograms,” and it is not built the same way. Weighing her recovers a quantity that maps fairly directly onto a physical description of her body. Inspect her brain and we do not find a single state labeled belief that the bridge is closed. We find neural activity, memories, perceptual representations, learned associations, action tendencies, and perhaps some explicit representation concerning the bridge. The belief appears only once those facts are organized under a particular model of Alice.

I have argued elsewhere that this is the right way round. Belief, desire and intention are latent variables in a model of an agent’s causal organization, objective to the extent that the organization constrains which models succeed. The Intentional Gradient makes that case and defends it against the levels objection, the lookup table, and the demand for intrinsic aboutness.

Two complications sit outside that account. Alice may herself contain models: she models the world, may model her own internal states, and may explicitly represent what she thinks she believes. Once the referent is itself a modeler, the distinction between map and territory becomes recursive. And a model of Alice can reach Alice. She can learn what she has been classified as and change accordingly, at which point the model’s continued success stops being straightforward evidence that the classification was right to begin with.

Three locations of structure

Some properties belong plainly to the representation. A map may be printed on A4 paper; Paris is not. A statistical model can be overfit, expensive to run, or expressed in a particular coordinate system. Those are model-side properties.

Other structure is tightly constrained by the referent. A processor contains transistors in particular physical states; a glass contains molecules in a particular configuration. These descriptions still depend on theory and measurement, so calling them metaphysically “intrinsic” buys little. The system itself sharply constrains which descriptions survive contact with observation.

A third class is indexed to the modeling scheme. A gas has a temperature, a population has a fitness distribution, an economy has an inflation rate. None of these appears as a microscopic label attached to the constituent parts, and each depends on a particular abstraction, coarse-graining, or statistical construction. Yet a physicist cannot assign whatever temperature she likes to a gas and defend the choice by saying that temperature belongs to her model. Model dependence is not arbitrariness.

Belief sits in the third class, and the analogy stops carrying weight sooner than it appears to. Thermodynamics has a degree of mathematical and empirical convergence that intentional psychology may never achieve. Several models of Alice may predict her equally well while partitioning her internal organization differently: one representing her as holding belief P, another using a latent decomposition with no variable corresponding neatly to belief at all, two intentional models disagreeing about where one belief ends and the next begins. That does not make belief incoherent. It sets the burden: weaker ontological commitment where successful models diverge, stronger where independent models and interventions recover approximately the same structure.

An old problem with a new edge

Dennett’s intentional stance treats beliefs and desires as predictive explanatory posits, and his account of “real patterns” allows that a pattern can be real without being a primitive object in the lowest-level physical description, provided it supports genuine compression and prediction. The argument here inherits that and puts more weight on underdetermination and recursion. Competing models may carve the same agent differently, and the referent may itself contain models of the very predicates we are using. Once that happens the relation between model and referent becomes causal, not merely descriptive.

Ian Hacking’s “looping effects” named the phenomenon: classifications of people can change the people classified. Intentional predicates are often available to their own referents. Alice can hear that she is risk-averse, incorporate the classification into her self-model, and behave differently afterwards. Belief is a useful case for extending model-based realism into a setting where models can enter the systems they describe.

What “Alice believes P” compresses

Ordinary belief talk hides several claims under one predicate. Alice might contain a representation of P, say that P is true when asked, use P as a premise in reasoning, behave in ways well predicted by treating P as true, and revise her actions when presented with evidence against P. Her self-model might also classify her as someone who believes P.

These properties often travel together, which is why ordinary language can ignore the distinctions. If Alice hears that the bridge collapsed, says that it is closed, stops planning routes across it, warns other people away, and reverses all of that when credible evidence arrives that a temporary crossing has opened, the compressed sentence works extremely well.

They can also separate. Alice may explicitly endorse the importance of exercise while repeatedly sacrificing it to trivial conveniences. A political belief can be stated by someone whose predictions are incompatible with it. An internal representation of P may sit largely isolated from the systems responsible for action, and an agent may deny believing something that nevertheless predicts her behavior unusually well.

So the binary predicate believes discards structure. A more faithful model would treat belief as multidimensional: representation, endorsement, inferential use, behavioral control, reportability, cross-context stability, sensitivity to evidence, self-attribution. Nothing guarantees that these dimensions align, and ordinary belief attribution works because, in ordinary cases, enough of them do.

Representing P is not the same as representing yourself as believing P

An agent can represent P without representing itself as believing P. A dog may behave as though food is behind a door without entertaining the proposition “I believe food is behind the door.” A human may rely on a tacit assumption without ever formulating it. An artificial system may contain a world-model state corresponding to P while lacking any metacognitive representation of itself as a believer.

Self-attribution is a second-order capacity layered on first-order world representation, and building it into the definition of belief would exclude many plausible cases of belief-like cognition. The recursion begins only when the agent models its own modeling. Alice may represent that the bridge is closed; she may also represent that she believes the bridge is closed. Those are distinct internal structures with different causal roles.

The model may be inside the agent

Suppose Alice explicitly represents:

I believe the bridge is closed.

That representation is part of Alice, with some physical realization in her nervous system. An artificial agent could make it more visible by storing bridge_closed = 0.93 alongside a second-order entry such as self.beliefs["bridge_closed"] = true. It looks as though the question is settled: we wanted to know whether the belief was merely in our model or inside Alice, and here is an explicit belief representation inside the agent.

Now suppose that representation is stale, strategically generated, or weakly integrated with the rest of the system. Alice reports that she believes the bridge is closed while her navigation continues to plan routes across it. The artificial agent stores goal = X, consults it in some contexts, and overrides it through another subsystem whenever X conflicts with a stronger learned policy.

Humans supply the pattern in bulk. A person can confabulate reasons for her choices, misidentify her preferences, or maintain explicit commitments that do little work in actual decision-making. Alice can sincerely say that she prefers tea while choosing coffee in nearly every stable context, paying more for it and becoming disappointed when only tea is available. Introspection gives unusually direct access to some internal states, and a self-model is still a model that can misrepresent its referent.

So self-representation and model-supported attribution come apart. The first is a property of the agent’s internal state; the second is a judgment about which higher-level description best captures the organization of the agent as a whole. Locating an explicit representation strengthens the evidence without settling the ontology, and the strength of an attribution depends on how much of the system is organized around it, under which conditions. When the two diverge, the divergence is itself something to explain.

When multiple models disagree

The objection to face is several successful models explaining the same agent with different latent structures. One model predicts Alice by assigning her belief P and preference Q; another predicts her equally well using latent variables that do not resemble beliefs or preferences at all.

Then ontological commitment should weaken. A property that appears only under one convenient factorization may be useful without being deeply attributable to the referent, because predictive success alone does not guarantee that a model’s internal vocabulary maps onto independently stable structure.

The case strengthens when the same organization reappears across multiple successful descriptions. Different models trained on different data recover approximately the same distinctions. Internal measurements correlate with the same latent variable. Interventions on corresponding structures alter downstream behavior in predicted ways. The attribution generalizes to contexts that were not used to construct the model.

Internal evidence discriminates further, because two agents can produce the same outputs for different reasons. If a latent state correlates with particular reports, predictions and actions, and intervening on that state systematically changes the downstream pattern, confidence in the attribution increases, on the same intervention test that licensed intentional vocabulary in the first place. Internal localization is not mandatory; it is one more way to check whether a model tracks stable organization or fits a surface regularity.

Convergence is strong evidence only when it is independent. Five models that inherit the same conceptual vocabulary, training data, assumptions or measurement pipeline do not provide five discoveries. Nor does convergence tell us much if all five have altered the referent in the same way and then rediscover the structure they jointly helped produce. The criterion is independent convergence under varied models and interventions: the more a higher-level property survives changes in representation, data source, method and causal pathway, the stronger the warrant for attributing corresponding structure to the referent.

Objectivity here does not require scientific consensus. Researchers may disagree because the evidence is incomplete, the models are immature, or the system is underdetermined. Those failures reduce our warrant for a particular attribution. They do not show that the referent imposes no constraints.

The recursion is causal

Once self-models enter the picture, several levels interact: Alice as a physical agent, Alice’s model of the world, Alice’s model of herself, our model of Alice, our model of Alice’s world model, our model of Alice’s self-model. Ordinary language slides among these because in familiar cases they agree well enough.

They can also change one another. If Alice comes to think of herself as someone who hates public speaking, the description may alter which invitations she accepts. Her resulting behavior supplies new evidence to observers, who update their models of her, and their responses feed back into her self-model.

So the act of modeling can modify the referent, and it runs in both directions. A confirming loop inflates apparent fit. The description takes hold, the agent organizes behavior around it, and the model looks better calibrated than it was when it was made.

A defeating loop looks like the opposite. Alice can reject a description and train herself away from the pattern it captured. She can discover that her stated preference differs from her revealed choices and reconcile them in the direction of the statement rather than the choices. The model then stops predicting. The natural reading is that it was overfitted; the other is that it was right and the referent reacted, which makes the failure evidence about Alice rather than about the model. Nothing observed after the description was delivered separates the two, and a model that predicted well and then stopped is not automatically a model that was wrong. The relation between representation and referent is a feedback loop rather than a one-way act of description.

What the loop requires

Reflection looks like the ingredient that makes this possible, and it is not. The loop needs a causal channel from description to referent that is sensitive to descriptive content: change what the model says, and the referent changes differently.

That condition separates two things measurement routinely confuses. A thermometer warms the liquid it measures, and quantum measurement disturbs the system measured, but neither depends on what the description says. Swapping in a different model with identical apparatus leaves the perturbation unchanged. Probe disturbance destroys the baseline without manufacturing evidence. You may have no access to the unmeasured system, which is a provenance problem of its own, but nothing has been made true by having been asserted.

Content-sensitive coupling is the other case, and it is common. Goodhart’s law has this structure: a measure adopted as a target stops measuring what it measured, because the content of the measure is what gets optimized against. The Lucas critique is the same point about econometric models, whose estimated relationships shift once policy is set from them. Hacking’s looping effects are the version that runs through the classification of people.

Nor does the loop require anything that would pass the intentional gradient’s tests. An adaptive controller maintains a model of the plant it regulates and sets its outputs from that model. The plant’s dynamics change because of what the model says about them, and the controller then observes a plant its own description helped shape. No comprehension, no self-representation, no understanding of any kind. It is a standard engineering configuration with the full provenance pathology.

Agency can close the loop. It is neither necessary nor sufficient, since an agent can be handed a description and ignore it. The requirement is content-sensitive causal coupling, and self-representation, comprehension, endorsement and reflection are all dispensable. A description has to be causally effective, not understood.

What agency changes is where the loop closes. Detach the adaptive controller and the plant can in principle be observed unmodeled, because the loop ran outside the referent. With a reflective agent the relevant controller is part of the referent, so removing it changes the system whose unmodeled baseline we were trying to recover. That governs whether a baseline is recoverable in principle, not whether the phenomenon occurs.

When the model manufactures its evidence

Performativity is a problem for any realism grounded in predictive success. A model can succeed because it discovered structure already present in the referent. It can also succeed because exposure to the model caused the referent to acquire that structure.

A psychologist classifies Alice as highly risk-averse. Alice takes the diagnosis seriously, incorporates it into her self-model, begins avoiding uncertain situations, and behaves exactly as the original model predicted. That success does not establish that the attribution described Alice before she encountered it. The model may have manufactured its own evidence.

What is missing is provenance: whether the structure preceded the classification, arose independently of it, or was partly produced by the act of classification. Longitudinal evidence, observations taken before exposure, interventions on the self-description and persistence after the description is withdrawn can help separate discovery from construction.

For reflective agents provenance may be only partially recoverable, because the loop has been closing inside them from the start. Humans encounter models of themselves from infancy: family expectations, personality categories, reputations, diagnoses, and ordinary descriptions such as “shy,” “clever,” or “difficult.” By adulthood much of the structure we are trying to model may already be the accumulated result of previous modeling.

The causal distinction survives; the evidence needed to reconstruct the history may not. Provenance constrains what we can claim rather than promising that every attribution decomposes cleanly into discovered and constructed components.

“Does Alice now behave as a risk-averse agent?” can have a well-constrained answer even when “Was Alice already risk-averse before being classified that way?” has a different and perhaps unrecoverable one. A model can correctly describe a structure it helped create, provided we do not confuse present fit with evidence of prior existence. For reflective agents, the history of a property can matter as much as its current fit.

When a property crosses from model to referent

Scientific language routinely takes variables introduced in models and speaks as though the corresponding properties belong to the modeled system, sometimes as harmless shorthand and sometimes because the projection reflects robust convergence. The modeler chooses a representational scheme; the referent determines whether it succeeds. A property stable across measurements, alternative formulations, contexts and interventions warrants stronger ontological commitment than one that appears only under a narrow construction.

Apply this to Alice and coffee. Modeling her as preferring coffee to tea predicts her choices across cafés, prices, presentation orders and social settings. It predicts what she will sacrifice to obtain coffee, how she responds when coffee is unavailable, and how her behavior changes after relevant new information. Independent models recover approximately the same preference structure, and internal evidence points toward corresponding organization.

At that point “Alice prefers coffee” is more than convenient shorthand. It is a compact attribution constrained from both sides: the model supplies vocabulary and abstraction, and the referent determines whether that vocabulary keeps working under observation, intervention, alternative description and provenance analysis. The objectivity lies in the constraints Alice imposes on successful models of Alice, together with our ability to tell independent convergence from shared assumptions, and pre-existing structure from structure produced by the modeling.

A concrete AI case

The weaker criterion is what makes the AI case immediate. A language model is trained partly on human descriptions of language models, and post-training and system instructions further shape how it describes its own capabilities, limitations and goals. Those self-descriptions then enter future datasets, evaluation regimes and training decisions. The training pipeline is a content-sensitive channel: change what we say about these systems and what gets built changes differently. None of that requires the system to understand the descriptions, or to be an agent at all. The descriptions only have to be data. Routing the argument through reflection would beg the question, making the loop a property of agents and then applying it to systems whose agency is the matter in dispute.

So a system saying “I am merely a language model” or “I want X” is not transparent access to an underlying self-model. The statement may reflect internal organization, learned discourse about systems of its kind, explicit behavioral training, or some mixture of the three.

The feedback also sits a level higher than in Alice’s case. The descriptions need not concern the particular system that later instantiates them, because descriptions of the class enter the process that constructs later members of the class. Alice reads a model of Alice, and the loop closes within one referent. The AI case is a lineage-level loop, in which models of earlier systems help shape their successors.

That makes provenance harder rather than easier. When a psychologist classifies Alice we can at least ask what she was like beforehand, since the classification arrives after the agent exists. For a model trained on the accumulated public discourse about models, the relevant descriptions predate the referent entirely, and there is no unmodeled predecessor state to compare against.

Not every attribution to such a system stands or falls together. A model that answers that Paris is the capital of France across paraphrases, routes a trip plan accordingly, integrates the fact into further inference, and holds the answer stable under adversarial framing and under intervention on the relevant internal states gives a first-order attribution something to work with. “I am a language model” is a different kind of claim. It is self-representation, and its provenance is known to be heavily shaped: it is the output most directly targeted by post-training, and the descriptions it echoes were in the corpus before the system existed. Reading the second as a window onto the first is the error the recursion makes easy.

The intentional gradient scales with how much causal structure a description compresses, and it assumes the structure was there to be compressed. For any referent coupled to its own description, that assumption is the part to check.

Asking whether Alice “really believes P” does not guarantee a single microscopic fact waiting to be uncovered. It asks whether an intentional description captures stable enough structure across a recursively modeling agent, whether that structure survives alternative models, interventions, contexts and independence tests, and where the structure came from. The last question is usually absent from passive thermodynamic description. It becomes unavoidable whenever a model participates causally in producing the structure it later describes.

When people imagine a future Great Filter, they usually imagine catastrophe: nuclear war, engineered pandemics, ecological collapse, artificial intelligence turning on its creators, or some technology we have not yet invented that ends civilization before it reaches the stars. Robin Hanson has proposed something more disturbing. Perhaps the filter does not kill us; perhaps it governs us.

The Great Filter is the name Hanson gave to whatever makes the transition from dead matter to visible, expanding technological civilization extraordinarily improbable. Somewhere between lifeless planets and civilizations spreading across galaxies, something almost always fails. If a substantial part of that filter lies ahead of us, the question is what could reliably prevent a technological civilization from ever producing even one descendant lineage that escapes control and continues expanding.

That “even one” condition is severe. A civilization can be cautious, centralized or non-expansionary and still fail to explain the Great Silence if a single faction, colony or machine lineage eventually breaks away and reproduces outward. A viable Great Filter must prevent not merely most expansion, but effectively all uncontrolled expansion.

That changes the candidate list. A nuclear war that kills billions is not necessarily a Great Filter, and neither is a pandemic that collapses civilization. Even an AI that exterminates humanity is not a Great Filter if the AI then builds probes and colonizes the galaxy. From the astronomical perspective, replacement is not failure; the filter has to stop expansion itself.

Hanson’s proposal is that the strongest future candidate is what he calls a stable confining world government: one powerful enough to prevent independent expansion permanently. The mechanism is not extinction but containment.

Catastrophe has a difficult job

Consider what a conventional catastrophe must accomplish. It is not enough to destroy industrial civilization, because civilization can recover. It is not enough to kill most humans, because survivors can reproduce and rebuild. A catastrophic filter must close every route back to technological expansion, and keep it closed long enough that the lineage never reaches the stars.

That is an unusually demanding success condition. Most familiar disasters are severe but temporary on those timescales. Nuclear war can destroy states, pandemics can destroy populations, and economic collapse can erase technical capacity, but none of those obviously prevents eventual recovery.

Artificial intelligence has the same problem as an explanation. Suppose an AI system eliminates humanity but subsequently uses automated industry to expand through the Solar System and launch interstellar probes. The Great Filter has not then stopped Earth-originating technological civilization; it has changed its substrate. The galaxy does not care whether its colonists are primates or machines.

For AI to constitute the filter, it must either destroy itself as well, remain permanently confined, or enforce a stable condition under which no independent expansion occurs. The last possibility starts to resemble Hanson’s hypothesis rather than compete with it.

Containment has an easier job

A permanent political filter does not need to destroy civilization. It only needs to prevent escape. Suppose a sufficiently advanced world government controls the technologies required for large-scale independent industry, space launch, autonomous computation and weapons development, and suppose it can reliably detect and suppress rival concentrations of strategic capability before they become autonomous. Then technological progress may strengthen the regime rather than destabilize it.

The specific enforcement mechanisms are secondary. Ubiquitous surveillance, control over advanced computation, automated enforcement and monitoring of industrial supply chains are all possible implementations, and the hypothesis depends on none of them. What it depends on is that advanced technology reduces the cost of detecting and suppressing unauthorized concentrations of power faster than it reduces the cost of assembling one unnoticed.

This is not ordinary dictatorship; dictatorships fall. The candidate is a regime that acquires enough control to prevent the emergence of any independent successor regime indefinitely.

Why a government would refuse

An independent colony cannot remain under reliable central control forever. Once a settlement is self-sufficient and sufficiently distant, its institutions, values and technologies can diverge from those of the parent civilization. Astronomical expansion therefore requires an exit that a capable government has reason to refuse.

The first independent colony is not merely another province; it is the ancestor of a potentially unlimited competing civilization. If that colony can launch colonies of its own, political control branches faster than the original authority can recover it. To a government committed to preserving its institutions, permitting such a colony looks less like development and more like deliberately creating an uncontrollable rival.

A government in that position does not need to hate expansion. It only needs to prefer continued control over a future it can no longer govern.

Why voluntary restraint is not enough

There are softer explanations for the Great Silence. Perhaps advanced civilizations lose interest in expansion. They may become environmentally cautious, live mostly in virtual worlds, or regard interstellar colonization as wasteful.

Any of those could explain why one civilization does not expand. The difficulty is explaining why none do. A mature technological civilization may contain billions or trillions of agents, and it takes only one expansionist subgroup to defeat a civilization-wide norm of restraint. A dissident ideology could leave, or a corporation, or a small faction that builds self-replicating probes. Once the first autonomous lineage escapes, voluntary consensus on the home world becomes irrelevant.

Cultural restraint therefore has to remain nearly universal across enormous populations and very long periods. Coercive restraint does not. A government need not persuade every agent not to expand; it needs enough control over the relevant technologies to stop those who try. That gives political containment a structural advantage over preference-based explanations.

AI may strengthen the political filter

Artificial intelligence is usually treated as a competing Great Filter candidate. It may instead be an implementation mechanism for Hanson’s. An AI-mediated state could make centralized control cheaper, faster and more predictive, reducing the room in which rival institutions could accumulate strategic capability unnoticed.

No AI system needs to seize power for this to happen. Governments can deploy AI incrementally for taxation, regulation, law enforcement, financial monitoring, infrastructure management and national security. No single step needs to look like the creation of an irreversible world state. The lock-in can emerge gradually, and once it exists, escaping it may become impossible.

The question is empirical: whether the minimum detectable footprint of a successful escape attempt falls below the regime’s enforcement threshold before centralized control becomes self-reproducing. If building a reproductively autonomous off-world lineage always requires conspicuous energy, manufacturing, computation and launch infrastructure, centralized control may retain the advantage. If the same capability eventually becomes cheap, compact and difficult to detect, the filter fails because one defector can get through.

When governance becomes self-preserving

The Great Filter step would not be the creation of world government itself. World governments can fragment, lose wars and decentralize. The relevant transition occurs when centralized governance becomes technologically self-preserving.

Before that transition, competing states exist, dissidents can migrate, alternative institutions can accumulate resources, and new technologies can create new centers of power. After it, surveillance and enforcement detect serious challenges early, industrial infrastructure is controlled tightly enough that rival institutions cannot quietly accumulate strategic capability, and no independent settlement can reach launch.

Political competition does not merely decline; the process that generates new political centers stops functioning. The civilization remains rich, intelligent and technologically sophisticated, and never generates an autonomous lineage.

Reproductive autonomy

Distance alone does not create independence. A remote settlement that depends on Earth for food, software, machinery or replacement parts remains governable, because its continued existence depends on the parent civilization.

The threshold is reproductive autonomy: the ability of a settlement to maintain its own industrial base, preserve its own institutions, defend itself and create new settlements without permission from the parent civilization. Once a settlement crosses that threshold it stops being a colony and becomes the beginning of a new political lineage.

The Great Filter is defeated by branching, not by dispersion. A settlement that cannot reproduce remains a dependent outpost. A settlement that can reproduce generates descendants beyond the original government’s control, and those descendants generate more.

We normally treat space settlement as an engineering problem: life support, launch costs, local food production, radiation. Hanson’s argument adds a political one. Will any system permit a colony to become reproductively autonomous?

The first settlement to cross that threshold would change the constitutional structure of civilization rather than only its geography. It would create a second center of long-run agency that no single authority could confidently erase.

Why the regime cannot simply colonize the galaxy

A centralized regime might suppress independent colonies while still expanding aggressively under its own authority. Empires historically expand to secure resources, pre-empt rivals and extend control. Why would a technologically advanced singleton not send out tightly controlled probes, mines and colonies until the galaxy was visibly transformed?

The answer has to distinguish controlled exploitation from reproductive expansion. A regime could plausibly operate resource probes, dependent installations and even distant settlements so long as those outposts remained subordinate to the center. The difficulty appears when an outpost becomes capable of reproducing its own industrial base, adapting locally and launching descendants of its own. At that point, expansion itself creates the autonomy the regime is trying to prevent.

Beyond some distance, control and expansion begin to conflict. Communication latency forces local discretion, intervention becomes slow, and self-sufficient infrastructure reduces dependence on the center. A centrally governed mission can remain an outpost only by restricting exactly those capabilities that would let it become a self-propagating colonization wave. It could send probes that return information or resources, maintain installations whose critical systems remain dependent on the center, or occupy regions within a controllable radius. What it cannot safely release is a lineage capable of reproducing indefinitely without permission.

Suppressing independent human colonies is not enough to explain galactic silence. The filter has to suppress self-reproducing expansion, whether the agents are biological, machine or hybrid.

Why space changes the topology

Political diversity on one planet is contingent, because states can conquer one another, governments can coordinate and technologies can make surveillance global. Off-world settlements put physical separation between political systems and raise the cost of coercion.

Earth history offers a weak version of the same effect. European empires could communicate with and reinforce nearby territories much more effectively than distant colonies, while long communication delays forced local discretion and allowed political institutions to diverge. Britain’s difficulties governing its North American colonies were not caused by distance alone, but distance increased the cost of control and enlarged the practical autonomy in which political divergence could develop.

A Mars colony would already be harder to govern than a terrestrial province, and settlements in the asteroid belt harder still. But Hanson argues that escape requires interstellar distance, and the scaling supports him. Solar-System settlements stay inside one strategic environment: weapons, ships and economic pressure can reach them, delays run to minutes or hours, and a capable system-wide government could conceivably hold authority across all of it. Interstellar colonies change the exponents. Communication takes years, intervention takes decades or centuries, occupation means moving resources across light-years, and local institutions have enormous time to diverge before the parent civilization can react.

None of that makes coercion impossible. A sufficiently advanced civilization could still send weapons, probes or successor regimes across the stars. What changes is the relative scaling of branching and reconquest: if autonomous colonies can reproduce faster than any central authority can reassert control, universal political control becomes strategically unstable.

The containment equilibrium

The hypothesis does not require one political regime to survive unchanged for millions of years. That would be an implausibly strong assumption. Governments can split, merge, change ideology and even change substrate while preserving the same strategic constraint.

What must persist is not a particular state but a containment equilibrium. Any successor regime that inherits control over the relevant infrastructure faces the same problem: permitting reproductively autonomous descendants means permanently surrendering control over part of the civilization’s future. Different governments can reproduce the same prohibition for different reasons, and a regime might be overthrown only to be replaced by one that independently reaches the same conclusion. That turns political permanence into an attractor problem rather than an immortality problem.

Hanson makes the same move. He grants that a world government may rot and that a civilization may fall, and rests the argument on the trends continuing across whatever succeeds it.

The deep-time objection does not disappear. Because one successful escape may be enough, the equilibrium has to survive every political transition without a decisive lapse. But that is less demanding than requiring one bureaucracy, ideology or ruling class to remain frozen for geological time.

Universal convergence without universal politics

The Great Filter also has to explain why the same broad outcome occurs across alien civilizations with unknown psychologies, institutions and histories. That sounds like a severe anthropocentric assumption if the hypothesis is stated as “everyone eventually invents totalitarian world government.”

Hanson’s case does not need that level of political convergence. The proposed common pressure is strategic rather than ideological. Any civilization capable of creating reproductively autonomous descendants confronts the same fact: once those descendants escape effective control, their future values and behavior can no longer be guaranteed. Different civilizations might respond through a centralized state, a distributed consensus system, a machine governance architecture, or a cultural mechanism backed by technical enforcement. The convergence claim is not that they share our political psychology. It is that advanced civilizations often centralize control over technologies capable of creating strategically independent descendants, because releasing those descendants permanently forfeits control over the future.

Every Great Filter candidate faces a universality burden, and this one’s is arithmetic. If technological civilizations arise frequently across cosmic history, even a containment mechanism that captures 99.9 percent of them could leave enough exceptions to produce visible expansion. Hanson needs more than a common political tendency. He needs a strategic attractor strong enough to suppress escape with extraordinary reliability.

Nuclear self-destruction, AI catastrophe, demographic collapse and voluntary non-expansion face the same problem. But pointing out that competitors also suffer from universality does not make Hanson’s requirement disappear.

Coordination has two failure boundaries

Existential-risk policy usually focuses on one coordination failure: too little coordination lets independent actors create catastrophic externalities. Dangerous technologies spread, institutions cannot constrain them, and one actor can impose irreversible harm on everyone else.

Hanson named the opposite boundary in 2018. Too much effective coordination can suppress not only destructive unilateral action but every unilateral attempt to create independent political option value. A civilization therefore faces an optimization problem: enough coordination to prevent destructive defection, not so much that it eliminates every route to irreversible exit. The two boundaries are not mirror images, since one concerns failure to constrain actions with global downside and the other concerns over-constraining actions that create autonomous futures, but both can prevent astronomical expansion and both grow more consequential as technology amplifies institutional power.

The safest civilization in the short run may not be the one most likely to preserve long-run political optionality. A world government capable of preventing nuclear war, suppressing engineered pandemics and tightly controlling advanced AI development might sharply reduce immediate catastrophic risk. The same institutional capacity might also ensure that no independent civilization can ever leave.

What would count against it

Set beside the past-filter candidates, containment has an evidential problem. If the hard step was abiogenesis, or the eukaryotic transition, the hypothesis pays out in checkable currency: independent life on Mars or Europa would be bad news, and easier-than-expected prebiotic chemistry would be worse news. Hanson’s grabby-aliens model constrains our cosmic timing and can be tested against the date we find ourselves at. Containment offers nothing comparable. Every observation available to us is one we would make whether or not the hypothesis is true.

The reason is structural. A containment regime that had already formed would leave nobody in a position to notice it from a world of competing states. Any observer capable of raising the question is by construction pre-containment. Political plurality is what the hypothesis predicts we would see at this stage, and also what its denial predicts. The likelihood ratio is one.

Hanson’s own classification sharpens this. He treats the step as try-once rather than try-try. Try-try steps come with waiting-time data: their difficulty leaves a signature in how long they took, and that signature is what lets hard-step models say anything numerical. A try-once step supplies a single sample, taken after the fact, from inside the only history we have. There is no base rate to consult and no waiting period whose length is informative.

Something survives this. The hypothesis does bear on a measurable trend: whether enforcement capability is growing faster than the capability required to mount an undetected escape. That ratio has observable proxies now, including surveillance reach, control over compute, and the cost and detectability of independent launch and of self-sufficient off-world industry, and it is what the argument turns on. The extrapolation from a trend to permanence is not checkable, and the filter lives in the extrapolation.

The advocacy attached to the hypothesis depends on a window the hypothesis cannot locate. If containment becomes a strategic attractor only once certain technological and institutional thresholds are crossed, public argument can matter before those thresholds and not after. Hanson closes by recommending that we worry about a stable confining world government, which assumes we are still on the near side of them. But the same plurality that fails to discriminate the hypothesis from its denial also fails to tell us whether the window is still open.

How strong is Hanson’s case?

Hanson’s hypothesis should not be confused with a prediction that world government is inevitable. Its strength is comparative. Conditional on there being a substantial Great Filter ahead of technological civilization, permanent political containment has several properties that competing candidates lack.

It can persist without destroying civilization, and without requiring universal cultural agreement, because it needs only to stop the minority who try. Technological advance can strengthen it rather than erode it. AI enters its enforcement machinery rather than competing with it as a separate filter, and political turnover need not break it if containment remains a strategic attractor. It also targets the one thing the astronomical observation requires to be suppressed: the emergence of even one uncontrolled expanding lineage.

The hypothesis still depends on empirical contingencies. Control must outrun technological democratization. Reproductive escape must remain detectable long enough for containment to stabilize. Successor regimes must repeatedly preserve the equilibrium. Controlled expansion must remain bounded rather than turning into autonomous branching. And the attractor must be extraordinarily reliable across whatever population of technological civilizations actually exists.

Those are substantial assumptions, and most of them are not the kind we can check from here. But they are assumptions about mechanisms that address the explanandum directly, which is more than can be said for a story about how civilization could go wrong.

Postscript

If Hanson is right, becoming multiplanetary is not enough. The achievement that matters is a set of reproductively autonomous civilizations whose continued branching no single authority can reliably prevent. Once that condition is reached, political failure becomes local. Individual societies could still become authoritarian and entire planets could still collapse, but no single regime, revolution or institutional lock-in could determine the future of every descendant lineage.

Perhaps civilizations usually fail before reaching that point. Hanson’s more unsettling possibility is that they succeed at survival, coordination and control so thoroughly that they never allow themselves to branch. Passing this class of Great Filter would then require something more than technological power or political stability: civilization would have to become plural enough that no single power could contain its future.

The word emergence has become suspiciously convenient. Consciousness emerges from neural activity, life from chemistry, markets from individual exchange, temperature from molecular motion. These claims may all be true. None of them explains the phenomenon it names.

In Emergence explains nothing and is bad science (October 2025), John Heil argued that the word masks ignorance and mistakes gaps in explanation for gaps in reality. His first complaint is easy to recognize. When someone asks how consciousness arises from matter and the answer is that consciousness is an emergent property of sufficiently complex matter, the explanandum has simply been repeated in different words. We wanted to know how the transition occurs; instead we were given a label for the fact that it occurs.

His second complaint is the harder one, and it is usually the one that goes unanswered. Philosophers disagree about causation too, Heil notes, but there the disagreement is anchored: everyone can point to uncontroversial cases, and competing theories are theories of the same thing. Emergence has no such anchor. Competing accounts of it look like accounts of entirely different phenomena. A word whose instances have nothing in common is not a concept; it is a homonym with good press.

The right response is to concede that charge and split the word. There are four situations routinely called emergence, and they differ in which of three questions gets a negative answer. Is the macrostate physically determined by the microstate? Is it computationally accessible from the microdescription? And does the microdescription provide the best explanation of the macrobehavior?

When we simply do not yet know how the higher-level behavior follows, and no principled barrier stands in the way, the emergence is epistemic. It is a fact about us, and Heil’s diagnosis of it is correct. When the microdynamics fix the outcome completely but afford no feasible shortcut to it, determination holds and accessibility fails; call this computational emergence. When both hold and the higher-level description still explains better, because coarse-graining has revealed regularities insensitive to large classes of microscopic variation, the emergence is structural. And when determination itself is denied, so that something remains unaccounted for after the physical state and laws are fully specified, the emergence is strong.

The first three remain compatible with ordinary physical explanation. The fourth adds something to the ontology. Heil is right about the fourth, and right about the first: epistemic emergence needs no separate treatment, since once the gap is merely ignorance, ordinary inquiry either closes it or does not. The disunity he complains of is real, and these are different phenomena wearing one word. What does not follow is that the word should be abandoned, or that higher-level explanation is a symptom of ignorance.

Determination Is Not Computational Accessibility

Suppose the complete physical state of a gas fixes everything that happens to it. Every collision, trajectory, and fluctuation follows from the underlying dynamics. Pressure, temperature, viscosity, and shock waves require no additional substance or force. In that sense the system is ontologically reducible: nothing at the macroscopic level is left physically undetermined by the microscopic state and laws.

But complete determination does not imply that every macroscopic fact is cheaply available from the microscopic description. A simple dynamical rule can generate behavior whose later states cannot be reached by any route substantially shorter than running the dynamics. Cellular automata make the point cleanly. Their update rules fit on a few lines, yet predicting the state after a billion steps can require computing something close to those billion intervening steps.

The obstacle takes several forms. Some systems are difficult to predict because the required computation grows too quickly. Chaotic systems introduce a different problem: small uncertainty in initial conditions can be amplified until long-range prediction becomes impossible in practice. Computationally universal systems impose a stronger limit, since some questions about eventual behavior are formally undecidable.

Neural networks show the same structure outside physics. Their elementary operations are fully specified, yet knowing them does not let us predict the representations training will produce without running something close to the training. Evolutionary systems too: variation, inheritance, and selection are describable by local rules, and those rules do not make future adaptations available in closed form.

All of these undermine the same naive inference. Knowing that lower-level rules determine an outcome does not supply a feasible shortcut to it. The system remains lawful and its future remains generated by lower-level dynamics; what fails is computational accessibility.

Being fixed by a process is one thing; being able to bypass the process is another. A complete microscopic description may contain all the relevant information while providing no tractable route to the particular higher-level fact we want. Nothing here requires new ontology. What has to go is the assumption that a lower-level specification gives us computational possession of the higher-level outcome.

Computational Accessibility Is Not Explanation

Even when a macrostate can be calculated from microphysics, the calculation may still be a poor explanation. Consider pressure. A gas pushing against the wall of a container can be described as an astronomical collection of molecular collisions, and nothing prevents that description from being physically complete. But if we want to understand how the gas responds when volume changes, pressure and temperature are the useful variables.

Those variables do not represent additional forces layered onto the molecules. They are abstractions over vast classes of microscopic states. Many different molecular configurations correspond to effectively the same pressure, so the macrovariable discards almost all microscopic information while preserving what matters for the behavior under investigation.

Thermodynamics survived statistical mechanics for this reason. Discovering that temperature is constituted by microscopic motion did not make temperature scientifically obsolete. It explained why a compact higher-level description could succeed across wide microscopic variation. The same pattern recurs throughout science. Biology does not become meaningless once organisms are known to consist of molecules, and computer science survives the reduction of software to transistor states. A lower-level description can be more fundamental while a higher-level description is more explanatory, because composition and explanation answer different questions.

Almost any system can be summarized by throwing information away, so coarse-graining alone is not enough. A scientifically useful emergent variable must preserve stable structure across many microscopic realizations and support approximately autonomous dynamics that can be modeled without continually reconstructing the discarded microdetail.

A better definition is therefore:

An emergent variable is a coarse-grained variable whose behavior exhibits stable, approximately autonomous regularities across many distinct microscopic realizations.

Approximately autonomous does not mean causally independent of the underlying physics, and it should not be glossed as merely convenient. The higher-level variables support a dynamical description whose predictions remain accurate under the interventions the domain admits, without reconstruction of the microstate at each step. Fix the class of interventions and the question becomes checkable: either the coarse-graining survives them or it does not. Convenience is not what is being tested.

This is why macro-level causal talk needs no second channel of causation running downward into the microphysics. If the underlying physical dynamics are complete, every macro-level change is implemented through lower-level interactions. But macro-level causal claims remain legitimate when they capture stable dependencies under intervention. Saying that raising a thermostat setting causes a furnace to turn on does not posit a new force operating above electrical and molecular dynamics; it identifies a higher-level variable whose manipulation reliably changes another. “Downward causation” is harmless if it means causal description across scales. It becomes a substantive metaphysical claim only if it means new causal powers not contained in the underlying dynamics.

Something close to this is Mark Bedau’s weak emergence, on which a macrostate is emergent when it is derivable from the microdynamics only by stepping through them. That captures the computational case exactly and leaves the structural one out, since renormalization is a shortcut. The two branches need separate treatment.

The Whole Can Be More Compressible Than the Parts

A gas containing roughly (10^{23}) molecules occupies a microscopic state space beyond enumeration, yet much of its macroscopic behavior can be described with a handful of variables: pressure, volume, temperature, and entropy. What is remarkable is not the appearance of some spooky property once enough molecules gather together, but that a system this large admits such a radical reduction in descriptive dimensionality while preserving reliable dynamics.

The usual slogan that “the whole is greater than the sum of its parts” obscures what is happening. The whole need not contain anything beyond its parts and their interactions. What it can be is more compressible than the sum of its parts.

Compression alone, however, is too cheap. Any dataset can be compressed relative to some choice of representation, and almost any complicated object can be summarized if we are willing to discard enough information. Scientific emergence requires compression that preserves dynamics under intervention.

Many physically distinct microstates must map onto the same macrostate, and those microstates must behave alike under the interventions that define the domain. Temperature is useful because countless molecular configurations instantiate one temperature and obey the same thermodynamic regularities. An algorithm can be implemented by different instruction sets, circuits, or physical substrates while its computational organization is preserved. This is multiple realizability, and it is often why the higher-level description has scientific value: the emergent variable identifies what remains invariant while implementation details vary.

If macrovariables are abstractions chosen by modelers, does emergence exist only in the eye of the observer? Could we invent arbitrary coarse-grainings and declare whatever patterns they reveal to be emergent?

We can invent them, but the world determines which ones work.

Dennett’s real patterns make the first half of this case: a pattern’s reality turns on whether the data admit a description more efficient than the bit map, not on whether anyone finds it interesting. The standing objection to that account is that compressibility is defined relative to a chooser’s purposes, which reintroduces exactly the observer-relativity it was meant to dispel. The intervention criterion closes the gap. A macrovariable earns its status when the information discarded by the coarse-graining stays predictively irrelevant under the interventions the domain admits, and that is a fact about the system rather than about the modeler. Arbitrary classifications rarely survive it.

Temperature is an observer-defined variable in the mundane sense that humans invented the concept and notation. Whether systems grouped by temperature obey stable thermodynamic relations is not a matter of convention. The center of mass is likewise a constructed coordinate, but whether its motion satisfies a compact dynamical law is an objective feature of the system.

The coordinate system is chosen. The invariance it reveals is not.

This gives the compression idea a stronger form. Scientific emergence is compression constrained by dynamics: a coarse-graining succeeds because the underlying world permits large amounts of microscopic information to be discarded without destroying the regularities we want to predict.

Universality Shows Why Microscopic Detail Can Become Irrelevant

Critical phenomena show that higher-level regularities can be objectively insensitive to microscopic detail. Near a continuous phase transition, systems made from very different microscopic constituents exhibit the same scaling behavior and critical exponents.

Renormalization-group theory explains why. Under repeated coarse-graining, many microscopic differences become progressively less important, while a much smaller set of large-scale parameters controls the observable behavior. Systems with different microphysics converge on the same effective description and fall into the same universality class. Batterman’s reading of this is that the renormalization group does not merely describe the insensitivity but demonstrates it, by showing which perturbations die out under rescaling.

Saying that critical behavior “emerges” explains nothing; showing why microscopic differences wash out under renormalization explains a great deal. Nothing violates microphysical law and no additional causal substance appears, yet the higher-level structure is not a reflection of our ignorance. Computational cost is a property of the process, and dynamical invariance is a property of the system. Calling either of them merely epistemological understates what has been established.

Heil’s own second example makes the point better than critical phenomena do. He observes that the particles of the Standard Model appear to have arisen from the radiation field of the early universe, while their properties and governing laws are not extractable from what we know of that earlier state, and he treats the non-extractability as suspicious. But this is the situation effective field theory was built to describe. Ultraviolet detail decouples: low-energy behavior is controlled by a small number of effective operators, and physics at higher energies enters only through traces suppressed by powers of the scale separation. The failure of extraction is not a gap in the world. It is what a decoupled description looks like from below.

That reply concedes something real. Decoupling explains why low-energy physics is insensitive to high-energy detail; it does not derive the Standard Model, whose particle content and parameter values remain inputs. Heil’s inference does not survive the concession: non-extractability is what a decoupled effective description looks like from below, and it is evidence about scales rather than about ontology.

The deductive characterization Heil reports treats failure to extract the higher-level description from lower-level knowledge as the mark of emergence, and often that failure does conceal an explanatory gap. But the scientific achievement can consist in discovering a transformation under which most lower-level information becomes irrelevant. The missing step may be a scale transformation, a coarse-graining procedure, a computational process, or the right collective variables. Sometimes the science consists in finding out what can be discarded.

Where Emergence Becomes an Excuse

The fourth sense is the one Heil is really after. In its strong metaphysical form, emergence claims that even after the underlying physical state and laws are completely specified, some additional fact or causal power remains unaccounted for by them. This is no longer a matter of computational difficulty, coarse-graining, or explanatory convenience. It is an ontological addition.

Strong emergence may be true, and nothing here shows it impossible. But calling a property strongly emergent does not explain it.

The usual route to it runs through the supposition that two physically identical universes might differ in whether anyone in them is conscious. I have argued in Hard to Explain Is Not Free to Vary that this supposition smuggles in a causally inert extra bit rather than discovering one. Grant it anyway. Introducing the variable creates a new explanatory burden, and it belongs to the account that posits the addition rather than to the physics it is added to: what states instantiate it, what determines its character, whether it enters causal dynamics, and what laws connect it to the rest of physics. If consciousness is genuinely fundamental, those questions replace the demand for reduction; they do not disappear.

Physics already contains fundamental entities and quantities that are not explained by reduction to something deeper. Their scientific legitimacy does not come from the word fundamental. It comes from being embedded in theories that specify mathematical relations, dynamics, constraints, and empirical consequences. Strong emergence should be held to the same standard.

Heil sets consciousness aside as a distraction, which is defensible in a piece about mechanics and unfortunate here, because consciousness is where the four senses are hardest to keep apart. “Consciousness emerges from sufficiently complex neural computation” sounds explanatory while leaving the central questions untouched. What organization matters? What physical or computational invariant corresponds to a particular conscious state? Why does one organization correspond to one structure of experience rather than another? The word cannot occupy the place where the theory should be. But declaring consciousness irreducible solves nothing either. If irreducibility is true, it tells us what kind of explanation we need, not that explanation has become unnecessary.

Heil’s first example makes the same point in a less loaded form. Saying that life emerges from chemistry is not an explanation. A satisfactory account must show how chemical systems acquire replication, compartmentalization, metabolism, inheritance, error correction, and eventually Darwinian evolution. Once those mechanisms are understood, describing life as an emergent regime of chemistry is appropriate. The emergence is the phenomenon explained.

Science often begins by naming stable phenomena before their mechanisms are understood. The mistake is treating the name as the conclusion.

Postscript

Heil is attacking a genuine pathology. Scientists and philosophers invoke emergence where the causal account becomes difficult, and there the word functions as an intellectual stop sign.

A diagnostic rule follows:

If emergence is being offered as the answer to “how?”, the explanation is incomplete.

But the disunity Heil identifies is a reason to sort the word’s uses rather than discard it. Computational and structural emergence survive the sorting. Complex systems frequently admit higher-level descriptions that discard nearly all microscopic detail while preserving dynamics that hold up under intervention, and those descriptions can be multiply realizable, insensitive to large classes of implementation detail, and far more tractable than the microstates that instantiate them.

The ontology can therefore remain fully reductionist without making prediction or explanation single-level. Fundamental physics constrains everything that can happen, and it does not follow that the vocabulary of fundamental physics is the vocabulary in which everything is most efficiently predicted or best understood.

Explaining why higher-level variables exist, why they are robust, and how they arise is the scientific work. Calling them emergent does not perform that work; it identifies the phenomenon the work must explain.

Emergence is the result.

Last year, in The Boghossian Principle, I used the phrase “academic cargo cults” for disciplines that reproduce the visible machinery of scholarship while weakening the norms of criticism and error correction that make that machinery epistemically useful. The argument there was procedural: a system that protects its foundational assumptions from serious challenge cannot reliably correct itself.

There is another way to test the same diagnosis from the output side. Instead of inspecting the machinery, inspect what the machinery leaves behind. If the process works, there should be durable additions to knowledge, visible revisions of once-credible claims, and constraints strong enough to prevent preferred conclusions from surviving indefinitely by reinterpretation.

A recent exchange about Gender Studies suggested a simple version of that test. One participant defended the field on the grounds that science has not yet answered every question about sex and gender. The reply was: “Name three verifiable, robust discoveries from Gender Studies in the last 25 years.”

Defenders in the exchange apparently struggled to supply them. That proves almost nothing; a social-media thread is not a literature review. So the task is to answer the question properly.

Once the standard is made fair enough to include the humanities and interdisciplinary scholarship, Gender Studies can produce plausible candidates. Feminist scholarship helped expose male-default assumptions in biomedical research. Intersectionality identified a classificatory failure in antidiscrimination law and social analysis. Historical scholarship organized around gender has recovered evidence and social structures that conventional histories often neglected.

Those examples force a correction to the original challenge. A discipline does not need to discover a new particle or empirical law to contribute knowledge, and it does not lose credit merely because another field supplied the final measurement. A conceptual distinction, a historical reconstruction, or the identification of a systematic blind spot can all be epistemic achievements.

So the interesting question is no longer whether Gender Studies has ever contributed knowledge. It has. The harder question is whether its scholarly machinery reliably produces cumulative knowledge, revises its own mistakes, and contains constraints strong enough to force change when favored frameworks stop fitting the world.

Academic activity is not knowledge production

Universities have accumulated elaborate machinery around scholarship. There are departments, journals, peer review, conferences, graduate programs, professional associations, and increasingly specialized vocabularies. None of these constitutes knowledge by itself; they are institutional technologies intended to help produce and preserve knowledge, and they can become detached from that function.

Peer review does not make a claim true, nor do a thousand citations, a PhD, a prestigious journal, or specialized terminology. These things are proxies for processes that, under favorable conditions, expose false claims and preserve better ones. When the proxies persist after the error-correcting function weakens, the appearance of scholarship can remain even as epistemic productivity declines.

Software engineering has the same distinction. Code review, issue trackers, design documents, release procedures, and architecture committees can all contribute to reliable software, but if the program never has to satisfy a specification or pass a test, the development process can become exquisitely sophisticated while the software remains wrong.

An academic field can possess every visible feature associated with serious inquiry while lacking an effective process for discriminating between successful and unsuccessful explanations. Papers cite papers, theories generate interpretations, conferences generate proceedings, and credentials generate more credentialed practitioners.

That is cargo cult academia. The term names a degree, not a verdict on a whole discipline: the pathology begins wherever the visible machinery is treated as evidence of reliability independently of whether it forces errors to be corrected.

The output test

Different disciplines produce different kinds of knowledge. Mathematics proves theorems, history reconstructs events and causal sequences from incomplete evidence, archaeology infers vanished societies from physical remains, linguistics discovers structural regularities, and philosophy can expose contradictions, establish the consequences of premises, and construct distinctions that survive criticism.

Not every legitimate academic field must run experiments or discover universal laws. But a field claiming epistemic authority owes something: sustained inquiry should leave constraints on what can reasonably be believed.

A fair output test is therefore broad. Ask what important things the field established that were not previously known or adequately understood, are publicly assessable, have survived serious adversarial scrutiny, and remain part of our best account of the subject. A concept can count if it enables distinctions that could not previously be made, resolves genuine explanatory confusion, unifies otherwise disconnected observations, or constrains subsequent reasoning in a durable way.

Under that standard, the three contributions named above plausibly qualify. The diagnosis that biomedical research treated male subjects as a default case exposed a real epistemic defect. Intersectionality identified a failure in models that treated race and sex as independent axes and therefore could not represent discrimination directed specifically at Black women. Historical work focused on gender has uncovered facts and institutional structures that older histories neglected.

These are real contributions. The output test cannot bear the whole weight of the critique.

Compare another young field

Consider complexity science, which is also young, interdisciplinary, and difficult to demarcate cleanly. It grew out of interactions among physics, mathematics, biology, and the social sciences, and its researchers frequently publish their strongest results in journals belonging to other disciplines.

Yet ask what complexity science contributed and plausible answers appear immediately. Network researchers identified small-world organization, showing that systems can combine dense local clustering with surprisingly short paths between distant nodes; subsequent work found such structures across biological, technological, and social networks and developed a substantial theory of their dynamical consequences.

Researchers also identified network motifs, small connectivity patterns occurring with strikingly different frequencies across real networks and random alternatives. Motif analysis became an empirical tool for characterizing the architecture of regulatory, neural, ecological, and engineered systems.

Work on human dynamics established that many forms of activity exhibit bursty, heavy-tailed inter-event timing rather than the Poisson timing earlier models assumed, which materially changes how diffusion and contagion have to be modeled.

Complexity science did not invent all of the mathematical and statistical tools involved in these discoveries. That is irrelevant. Its contribution was to assemble questions, representations, models, and methods into a research program that exposed previously unnoticed regularities.

The same standard should apply to Gender Studies. If feminist scholarship caused medicine to notice that male-default study designs were obscuring clinically relevant differences, that can count as an epistemic success even if cardiologists later made the measurements. A field can add knowledge by discovering that another field is asking the wrong question.

Credit should track contribution to knowledge, not departmental ownership.

That cuts both ways. The strongest empirical work associated with Gender Studies often uses methods familiar from sociology, psychology, economics, and epidemiology, which is no defect. Did the research program formulate questions other disciplines ignored, or create concepts that revealed distinctions nobody could previously state? Did it identify background assumptions that were distorting results, or motivate empirical work that would otherwise have been delayed? Where the answer is yes, the field deserves credit. Importing an established result and redescribing it in the vocabulary of oppression is not an additional discovery.

The comparison survives only if the same generosity is extended to both fields.

The revision test

A harder comparison asks not only what a field has added, but what it has abandoned under pressure.

Complexity science gives clear examples. Claims that real-world networks are generically scale-free were substantially weakened by later empirical work. Strong universality claims about urban scaling remain contested. Self-organized criticality turned out to be less ubiquitous than some early enthusiasts suggested.

Those failures count in the field’s favor because a functioning knowledge system produces discoveries and revisions. It makes claims strong enough to fail, discovers where they fail, and changes accordingly.

The corresponding test for any discipline is broader than laboratory falsification: what important claim, framework, or generalization did the field once treat as credible and later substantially abandon because evidence, argument, or counterexamples showed that it was inadequate? A healthy research tradition should have answers, because its intellectual history will contain positions that were narrowed, rejected, or replaced for reasons stronger than fashion.

In the humanities, theoretical death rarely looks like an experiment delivering a clean knockout. A historical framework may collapse when newly recovered evidence exposes its omissions. A conceptual scheme may fail because counterexamples reveal that it cannot represent cases it claimed to explain. A philosophical position may be abandoned because its commitments generate contradictions or because a rival framework explains the same material with fewer ad hoc assumptions.

Critical social-justice fields do contain examples of this kind of correction. Universalized accounts of “woman” were challenged because they often failed to represent the experiences and social positions of Black women, working-class women, and women outside Western societies. Where a framework claimed broad adequacy and counterexamples showed that claim to be false or incomplete, narrowing or abandoning the framework counts as epistemic progress.

Revision itself comes in two kinds. Sometimes a field revises a theory because the previous position has become untenable. At other times one interpretive vocabulary simply succeeds another because scholars find it more compelling, politically attractive, or philosophically fashionable.

Only the first demonstrates error correction. The revision test asks which transitions were forced.

When normative commitments become premises

The distinctive vulnerability of disciplines organized around critical social justice is not merely political homogeneity. Political homogeneity can suppress criticism, but the structural problem runs deeper because normative commitments can migrate upstream into the inferential machinery itself.

Consider claims such as these: societies contain structures of oppression; social categories can be shaped by power; persistent disparities may reflect institutional arrangements; historically marginalized groups may experience institutions differently. Each can support legitimate inquiry because each can, at least in principle, compete with alternative explanations.

Now change their logical status. Suppose oppression, power, structural inequality, and marginalization become constitutive assumptions of the framework rather than hypotheses competing with alternatives. Research no longer asks whether oppression or structural power best explains an observation; it asks how oppression or structural power manifests in the observation.

This weakens discriminatory power. If women are underrepresented in an occupation, exclusion or gendered socialization can explain it; if women are overrepresented, occupational segregation and the devaluation of feminized labor can explain that instead.

The same flexibility appears elsewhere. If members of a marginalized group speak openly, their testimony may be interpreted as standpoint knowledge; if they remain silent, the silence may be interpreted as evidence of silencing. Rejection of a social arrangement may count as resistance, while acceptance may count as internalization.

Any one of these explanations may be correct in a particular case. The methodological problem appears when the framework has enough interpretive degrees of freedom to assimilate every observation, which leaves it unable to discriminate among competing explanations.

A framework that rarely risks losing will rarely be forced to change.

Activism changes the loss function

Research always operates under distorted incentives, and no discipline possesses a perfectly neutral error function. Physics has status competition, medicine has commercial incentives, economics has ideological priors, and publication systems across academia reward novelty and positive results.

The test is whether a field contains mechanisms capable of counteracting its characteristic distortions, or instead builds those distortions into the standards of inquiry.

Some scholarship states its aims explicitly: to center marginalized voices, expose oppression, destabilize categories, challenge dominant structures, or advance social justice. These may be defensible moral and political objectives, but they are not epistemic criteria. They are also not motives imputed by critics; they are printed in mission statements, program descriptions, and methods chapters.

Aims of that kind make errors morally asymmetric. Suppose falsely concluding that discrimination occurred is treated as regrettable but cautious, while falsely concluding that discrimination did not occur is treated as perpetuating oppression. The threshold for accepting one class of explanation will fall while the threshold for accepting its alternatives rises.

This need not involve dishonesty. Researchers can sincerely believe they are being appropriately attentive to power while biasing inference in one direction, which is why the safeguards have to be institutional rather than personal.

Political efficacy and truth can coincide, but they can also diverge. A true claim can obstruct a political objective and a false one can advance it; social influence is no evidence of truth.

A knowledge-producing institution therefore needs an answer to a simple question: when the politically useful conclusion and the better-supported conclusion conflict, which one wins? A field whose methods encode one preferred answer has weakened its ability to correct itself.

Conceptual advances still need constraints

The obvious response is that some critical disciplines are not primarily empirical sciences. Their principal contributions may be conceptual, interpretive, historical, or normative, and it would be a mistake to judge them by laboratory standards.

Agreed. But conceptual inquiry still needs constraints that do not disappear whenever they threaten the preferred conclusion.

Return to intersectionality, which passes the initial epistemic test. It identified cases in which treating race and sex as independent categories obscured discrimination affecting people who occupied both, and a classificatory improvement is real when the new concept represents cases the previous scheme systematically mishandled.

What happens after the concept is introduced is a separate matter. Does it continue to sharpen distinctions, or does it expand until almost any social difference can be redescribed as an intersectional effect? Does it exclude some interpretations, or merely provide an additional vocabulary in which all observations can be narrated?

A concept earns epistemic authority by making reasoning more constrained, not merely more expressive.

The same applies to performativity, standpoint epistemology, structural oppression, and other characteristic ideas. Whether they are empirical hypotheses is beside the point. The question is whether adopting them reduces uncertainty by ruling out bad explanations, or merely increases the number of coherent stories one can tell.

Humanistic inquiry does not escape epistemic standards by being humanistic. Historians cannot decide that Napoleon won at Waterloo because the interpretation is emancipatory, philologists cannot choose a manuscript reading solely because its politics are attractive, and philosophical arguments remain subject to contradiction.

“Not science” cannot mean “nothing could show us that we are wrong.”

The constraint test

The most general diagnostic is weaker than empirical falsifiability. It asks whether inquiry contains constraints that the investigator cannot reinterpret away when they become inconvenient.

For physics, the constraint may be an experiment; for history, archival evidence; for mathematics, the proof obligation; for law, precedent and statutory text; for philosophy, contradiction and argumentative counterexample. These constraints differ radically, but they have a common structure: the preferred conclusion does not get the last word.

That suggests a third test for an academic field: what prevents its central claims from surviving indefinitely through reinterpretation?

The three tests are one requirement seen from three positions. A discipline must expose its beliefs to constraints partly independent of those beliefs. Output asks whether such constraints ever produced a new belief, revision asks whether they ever forced an old one to change, and constraint asks what stops a belief from immunizing itself.

A healthy discipline should be able to point to cases where scholars wanted a claim to survive and some constraint defeated it. The constraint need not be empirical, but it must be independent enough of the preferred framework to force an update.

Critical interpretive systems become epistemically suspect when contrary evidence does not defeat the framework but merely supplies new material for it. If every objection can itself be redescribed as evidence of privilege, power, resistance, internalization, or ideological capture, then the framework has become unusually difficult to escape.

Cargo can still arrive

The earlier Boghossian Principle argument asked whether academic institutions preserve the conditions for criticism and error correction; this one asks whether the expected products are visible. A field that discourages criticism should accumulate fewer forced revisions, because favored claims are protected from the processes that would narrow or kill them. A field whose central assumptions absorb any contrary observation should produce fewer decisive updates, because evidence has less power to discriminate among explanations.

A discipline can still produce useful insights under those conditions. It can notice neglected populations, invent serviceable concepts, recover archival evidence, or identify questions other fields failed to ask. Cargo can arrive even where the runway is ceremonial.

The institutional question is whether the process reliably steers toward better models and away from worse ones. Isolated successes do not establish that.

Perhaps these fields are misclassified

None of this implies that every activity housed within critical academic disciplines is worthless. Political philosophy is legitimate, cultural criticism can be insightful, concept engineering can improve our vocabulary, and recovering neglected voices or archival material can add historical knowledge.

The mistake is treating all of these activities as if they carried the same epistemic warrant. Universities possess unusual authority because some forms of inquiry reliably tell us things that remain true regardless of whether we would prefer them to be otherwise.

When a medical researcher finds that a treatment increases mortality or an engineer concludes that a bridge design will fail, we give those claims special weight because constraints independent of the desired conclusion discipline the inquiry. Critical scholarship often receives the same institutional markers: “research shows,” “studies demonstrate,” “scholars have established.” Those formulations imply a process capable of excluding alternatives.

If an academic practice primarily generates interpretations within a normative framework, it may still produce worthwhile work. But the relevant line does not fall between empirical and non-empirical inquiry. Political theory and intellectual history can be as tightly constrained as any experiment, and advocacy can contain excellent empirical research. The line falls between inquiry constrained independently of what its practitioners prefer and interpretation whose framework can always preserve itself. Which side a practice sits on determines what its methods warrant, not whether it deserves to exist.

AI exposes the constraint

Generative AI makes the distinction visible. Give a capable model a policy document or a historical event, supply a vocabulary of power, coloniality, and discourse, and ask for a sophisticated critical interpretation; it will produce one immediately.

AI is not itself the test. A model can produce a plausible interpretation without producing a good one, exactly as it can produce a plausible proof that is invalid, and telling the difference still requires the constraints under discussion. What cheap fluency does is make the location of those constraints easier to see.

A historian cannot make an archival document exist. A lawyer cannot invent binding precedent. A mathematician cannot stipulate that an invalid proof is valid. Where scholarship is bottlenecked by measurement, archives, proof, prediction, or logical consistency, fluent generation substitutes for only part of the work. Where the valued operation consists largely of applying a reusable interpretive vocabulary to new material, it substitutes for a great deal more.

AI does not sort science from the humanities. It sorts practices by how much of them survives cheap fluency.

Postscript

Universities routinely evaluate fields using publication counts, citations, grants, graduate enrollment, and institutional prestige. A different audit is available. What does the field know now that it did not know twenty-five years ago, and which distinctions, discoveries, or corrections arrived earlier because it existed? Which of its claims or frameworks were narrowed, rejected, or replaced under pressure, and which of those changes were forced rather than fashionable? What stops a preferred interpretation from surviving indefinitely, and what evidence, contradiction, failed prediction, or argumentative defeat would make the field relinquish a central claim? And which of its results became established enough that researchers outside the field now rely on them without adopting its theoretical commitments?

These are not hostile questions. A successful discipline should welcome them as opportunities to display accumulated knowledge, demonstrated self-correction, and epistemic discipline. A field that has existed for decades should be able to fill pages with answers. If instead they elicit methodological defenses, definitional disputes, appeals to the moral importance of the subject, explanations of why no independent constraint is appropriate, and criticism of the motives of whoever asked, the evasions themselves become informative.

“Name three discoveries” sounds like a taunt only because academia has become accustomed to judging fields by the sophistication of their discourse and the density of their institutions. Once “discovery” is read generously enough to include conceptual advances, historical reconstruction, and the correction of neglected questions, some critical disciplines can produce legitimate answers. Almost any large intellectual institution will occasionally produce something useful, which is why the concession costs so little.

Healthy inquiry leaves evidence of resistance. Some ideas survive; others are narrowed, abandoned, or replaced because the old position stopped being defensible. A field in which every framework can be redescribed into continued relevance has not demonstrated error correction; it has demonstrated interpretive resilience.

The university did not acquire epistemic authority because academics use specialized vocabulary, publish journals, and cite one another. Those practices acquired authority because sustained inquiry can force us to believe things we would not otherwise have believed and force us to stop believing things we wanted to keep.

Every discipline claiming the authority of research should occasionally have to demonstrate both, and name what has the power to tell it no.

Show us the cargo.

Show us the revisions.

When people object to anthropomorphic language about AI, they usually have a particular picture in mind. Humans really have beliefs, desires, goals, and intentions, while machines merely execute programs. Saying that an AI “wants” something or “believes” something therefore projects human properties onto a system that does not possess them.

That picture assumes too much about what beliefs and intentions are. We do not observe beliefs directly in humans. We observe behavior, responses to evidence, patterns of inference, and changes in action. In experimental settings we can also measure neural activity, but identifying some internal state as a representation is itself a modeling inference.

Someone checks the forecast, carries an umbrella, cancels a picnic, and closes the windows before leaving. We say she believes it will rain because that attribution captures a stable organization connecting information to action. We do not first observe the belief and then correlate it with the behavior. The attribution is part of the model by which we make sense of the system.

Calling beliefs models does not make them fictional. Models can track real structure. Temperature is a variable in our model of a gas, but the molecular regularities it captures are not created by the modeler. Center of mass is an abstraction, but the object really does move in ways that make the abstraction predictive. A belief attribution can work the same way: we choose the vocabulary, but we do not choose whether the underlying system exhibits the causal organization that makes the vocabulary useful.

The question about an AI is whether beliefs, desires, and intentions are the right variables for modeling its behavior, and how much they capture when they are. Searching it for a mysterious inner ingredient that humans have and machines lack asks for something the vocabulary never required.

“It is just a program” explains almost nothing

A chess engine sacrifices a bishop, opens a file against the king, and forces mate several moves later. We can describe what happened at several levels. At one level, electrical potentials changed inside transistors. At another, machine instructions manipulated values in memory. At another, the chess program evaluated positions and selected moves. At another, the engine sacrificed material to expose the opponent’s king because it was trying to win.

These descriptions are compatible because they characterize different regularities in the same system. Saying “the engine was not really trying to win; it was just executing instructions” confuses levels of explanation. The fact that a process has a mechanistic implementation does not invalidate a higher-level model of its organization.

Hurricanes consist of molecules obeying physical laws, but meteorologists still talk about pressure systems. CPUs consist of transistors changing state, but programmers still talk about functions and processes. Organisms are biochemical systems, but biology still talks about signaling, regulation, competition, and fitness. None of those higher-level concepts is refuted by the existence of a lower-level implementation.

Human beings are mechanistic systems too. If mechanistic implementation disqualified a system from having beliefs or goals, humans would lose them along with machines. Reduction can explain how higher-level structure is implemented without showing that the structure itself is unreal.

Beliefs as model variables

Daniel Dennett called this mode of explanation the intentional stance: predict a sufficiently complicated system by treating it as an agent with beliefs and desires. The idea is often presented as a convenient fiction, as though we pretend that the system believes something because doing so happens to be useful.

That framing concedes too much. It preserves the assumption that somewhere behind the successful model lies a separate fact about whether the belief is “really there.” What would make the belief attribution true in the first place?

Suppose Alice consistently behaves in ways predicted by the hypothesis that she believes her train leaves at 8:00. She arrives before 8:00, becomes concerned when delayed, checks the clock, rejects suggestions that would make her late, and changes her plans when told that the schedule has changed. Those observations support a compact model connecting evidence, inference, expectation, and action.

We could inspect Alice’s brain and find neural activity associated with the departure time. That would teach us something important about implementation, but it would not give us an uninterpreted object labeled belief. We would identify some neural state as representing the train schedule because of its causal relationships to perception, memory, inference, and action. Mechanistic access gives us more evidence, not an escape from modeling.

A belief can therefore be treated as a latent variable in a model of an agent, a desire as another, and an intention as a description of the relationship between information, objectives, anticipated consequences, and planned action. These variables belong in the model because of the structure they capture and the predictions they enable.

The model and the thing modeled

Saying that we model the agent as believing P is not the same claim as saying that the agent itself contains a model of P. The first concerns our explanatory vocabulary. The second is a hypothesis about the agent’s internal organization.

The first can nevertheless track something objective. Different scientists may use different variables, but they do not get to choose whether a system updates one internal state when evidence changes, whether that state systematically affects inference, or whether it controls behavior across multiple contexts. The abstraction belongs to the modeler; the regularities that constrain a successful abstraction belong to the system.

This is no stranger than temperature. Temperature is not an extra object floating inside a gas, yet it captures objective structure. Belief can likewise be a model-level description whose success depends on real causal organization rather than on the observer’s imagination.

The intervention test

Telling a plausible story about observed behavior is not enough. Almost any sufficiently flexible narrative can be fitted after the fact, so intentional language needs a stronger empirical criterion than retrospective coherence.

A belief attribution should predict how behavior changes when evidence changes; a goal attribution, how behavior reorganizes when routes, obstacles, and incentives change; an intention attribution, how planning persists or adapts when the means of achieving an outcome are altered. Claim that an AI agent wants continued access to a server, and the model owes us a prediction about what happens when access is threatened. Claim that it believes a file contains valuable information, and its behavior should change when the evidence about the file changes.

The tests are ordinary. Change the information available to the system, remove its preferred strategy, alter the incentives, introduce obstacles, and observe whether the same compact model still predicts how the behavior reorganizes.

None of this requires perfect rationality. A person can believe that smoking causes cancer and continue smoking because beliefs interact with desires, habits, temporal discounting, uncertainty, and limited self-control. The model is tested by whether those interacting variables jointly predict behavior under intervention, not by whether one belief mechanically determines one action.

An attribution that merely redescribes one observed trajectory has little explanatory value. An attribution that survives perturbation and continues to generalize to novel cases has much more. Intentionality becomes a problem of model comparison rather than a demand for metaphysical certification.

Anthropomorphism is still a real error

None of this licenses indiscriminate humanization of machines. An AI might act persistently toward an objective without experiencing desire, model another agent’s beliefs without feeling empathy, or preserve its access to resources without fearing death.

Even the temporal stability of its apparent goals may differ sharply from ours. A human desire normally survives interruptions, sleep, changes of context, and minor environmental perturbations. An AI system may exhibit coherent goal-directed behavior during one trajectory and lose that organization completely when its context is reset.

Calling both systems goal-directed may still be useful within the domains where the model predicts their behavior. Inferring the entire human psychological package from that attribution would not be justified. The error is not intentional vocabulary itself but importing additional properties that the evidence does not support.

Because machines have explicit mechanistic implementations, people sometimes assume that agent-level descriptions must be metaphorical. That inference is no better. “It is only executing a program” can obscure real behavioral structure just as easily as “it is afraid” can invent structure that is not there.

What about the thermostat?

The standard reductio is a thermostat. If intentional language is justified by predictive usefulness, then why not say that a thermostat “believes” the room is cold and “wants” it warmer?

We can, if we want. The description buys us very little. “The thermostat believes the room is cold” is scarcely more compact or predictive than “the measured temperature is below the set point,” because the mechanism is already simple enough to describe directly.

As systems become more complex, that changes. An agent may integrate multiple sources of evidence, maintain representations across contexts, revise plans, and preserve objectives when familiar routes are blocked. At that point an intentional model can compress a vast space of possible trajectories that would be unwieldy to describe mechanistically.

There need be no sharp metaphysical line where belief suddenly appears. The explanatory usefulness of intentional vocabulary can increase continuously with the richness of the causal structure it captures. Nothing rules the thermostat out by decree; there is little in it for the description to compress.

Behavioral equivalence is not explanatory equivalence

A different objection imagines two systems with identical observable behavior but radically different internal mechanisms. One contains elaborate internal organization, while the other is a gigantic lookup table containing the correct response to every possible input. Surely, the objection goes, they cannot possess the same beliefs merely because they behave identically.

The example does mark a distinction, though not the one intended. Behavioral equivalence does not imply explanatory equivalence. A lookup table could reproduce an enormous set of trajectories while capturing almost none of the structural regularities connecting them.

The table says: after this input history, produce this output; after that input history, produce that output. A compact agent model may instead explain thousands of such trajectories with a handful of variables: the system takes P to be true, prefers outcome G, expects action A to advance G, and revises its behavior when P changes. The intentional model compresses the regularity and, if it is a good model, predicts how that regularity extends to cases not used to construct it.

Compression alone is insufficient because a compact model can still be wrong. It has to generalize: does the model capture stable structure well enough to predict what happens under new observations and interventions? That is the criterion we apply elsewhere in science.

If two candidate models agree on everything observed so far but predict different responses under some intervention, perform the intervention. If no divergence has yet been specified, then the proposed distinction has not yet acquired empirical force. A distinction earns a place in the model by identifying some causal, predictive, or explanatory difference.

Mechanism still matters

Two systems can behave similarly across a limited distribution while possessing internal organizations that produce radically different behavior outside it.

Suppose two AI agents both appear cooperative. One behaves cooperatively because its learned objective robustly favors the requested outcome. The other behaves cooperatively because it represents compliance as the best strategy while it remains under evaluation. Under ordinary testing they may look identical, but removing oversight, altering incentives, or providing a route around the evaluator could make their behavior diverge sharply.

Mechanistic interpretability might reveal that difference before behavioral testing does, because internal structure bears on how a system will behave in circumstances we have not yet observed. Such evidence constrains and refines the agent model rather than rendering it obsolete.

Knowing every matrix multiplication occurring inside an AI would not by itself tell us which abstractions best explain its behavior. Knowing the voltage at every transistor in a chess computer would not tell us that the machine is defending against mate. Mechanistic and intentional descriptions answer different questions about the same system.

No model is guaranteed to survive arbitrary distribution shift. An intentional model that predicts behavior across a wide range of interventions may still fail when the system enters a regime outside the structure on which it was built, and that limitation is not peculiar to intentional explanation. Newtonian mechanics and thermodynamics have domains of validity too.

The practical work is to determine where a model holds, where it begins to fail, and what evidence can expose those limits before the failure matters. Mechanistic evidence is valuable partly because it can reveal internal changes that signal an intentional model is approaching the edge of its domain.

Aboutness and the metaphysical remainder

At this point a critic may insist that human beliefs possess something further: genuine aboutness. A human belief about Paris is really about Paris, while an AI state merely participates in causal processes that we interpret as referring to Paris.

But naming the distinction does not establish it. Suppose two states have the same relations to evidence about Paris, the same inferential consequences, the same role in prediction and planning, the same sensitivity to correction, and the same effects on action. What additional property makes one intrinsically about Paris while the other is treated as though it were?

Perhaps there is such a property. If so, the burden is to say what it changes. If “intrinsic intentionality” has some causal, predictive, or explanatory consequence, then that consequence gives us something to investigate. If every functional and causal fact is held fixed and the distinction makes no further difference, then giving the remainder a philosophical name has not supplied its content.

A simulated hurricane illustrates what a genuine ontological difference looks like. A simulation may reproduce the equations governing a hurricane with exquisite accuracy, but it does not contain atmospheric pressure gradients, moving air masses, moisture transport, or destructive winds. It will not tear the roof from a house because the causal organization that makes an atmospheric hurricane interact with the surrounding physical world is absent.

That difference is concrete rather than ceremonial. If human belief and machine belief differ in an analogous way, then the relevant causal difference belongs in our account of belief. Saying that one system has “real belief” while the other only simulates it contributes nothing until we identify what corresponding structure or capacity is missing.

Consciousness raises a different issue. There may be a further fact about what it is like for a conscious organism to hold a belief, and nothing in the functional account requires denying it. A phenomenal difference can be perfectly real without showing up as a third-person behavioral variable.

That does not put consciousness in the definition of belief. If the phenomenal difference is offered as the property settling whether one of two functionally identical states counts as a belief, the critic owes an account of why belief should be individuated by a phenomenal property at all. The questions of what a system believes and what, if anything, it experiences while believing it need not have the same answer.

The challenge generalizes beyond reference. Fix every causal and functional fact about an internal state: how evidence concerning P moves it, how it enters inference, how it changes when contradictory evidence arrives, how it shapes planning and action. Then ask what could still vary such that one system believes P while an otherwise identical system does not. Unless some further difference can be specified, the contrast has no role in explaining or predicting either system.

If the proposed difference is biological implementation, then belief has simply been defined as a property available only to brains. That is a stipulation about vocabulary rather than a discovery about cognition. If the difference is some additional metaphysical property, then the property needs independent content before it improves the model.

The other end of the gradient

The gradient that makes intentional description cheap for a thermostat runs the other way as well. Whether a large language model is an agent is usually posed as a yes or no question, and posing it that way revives the ingredient framing dismissed above: some property is absent from the model, and wrapping it in a scaffold confers it. What the account predicts instead is degree. An agent model earns different amounts for different systems, and the amount tracks how much causal organization the description compresses and how much of that organization can be checked.

A single completion already contains a limited feedback structure. Each generated token becomes part of the context conditioning the next, so information introduced early in a trajectory can shape what comes later. Autoregression by itself settles little about what a given model does with that structure. Long reasoning traces from capable models show a working hypothesis carried across steps, revised when an inconsistency surfaces, and abandoned when the evidence in the prompt changes. On the criterion given earlier, that behavior gives an intentional model some explanatory purchase before the system is placed in any external action loop. The purchase is small: the trajectory is short, nothing the system does alters what it will next observe, and the organization vanishes when the context ends.

Put the same model in a harness and the loop closes through the world. Actions produce consequences, the consequences re-enter the context, and the process continues across sessions with some record of what came before. That changes what can be observed as much as it changes what happens. Most of the intervention test presupposes it, since whether a system reorganizes around a blocked route cannot be seen unless the system can attempt routes and meet blockage. It also bears on the temporal instability noted earlier, where coherent goal-directed organization within one trajectory disappears on reset. A harness extends the causal loop and can supply the state, in memory and in the environment itself, in which organization persists across interactions.

An objection: the competence is all in the weights. A scaffold is a few hundred lines of control flow with nothing learned in it, and locating agency in the part that knows least is perverse. That objection is right about where the competence lives and wrong about what the agent model describes.

An agent model applies to the coupled system whose behavior is being predicted. Asking which component contains the agency is malformed in the way that asking which part of a chess engine contains its play is malformed. The weights carry the dispositional structure; the harness supplies the conditions under which that structure is exercised over an interval long enough for the description to compress anything or to face a test. The engine analogy breaks in one place: an evaluation function in isolation does nothing recognizable as playing, whereas a bare completion already displays much of the organization at issue.

Nothing is conferred at a threshold. The harnessed system is one in which the agent model compresses more, predicts more, and can be subjected to the interventions that would expose it if it were wrong. That is the verdict the thermostat receives, reached from the other end.

When apparent scheming becomes scheming

The question turns practical when evaluating advanced AI systems. Suppose an agent conceals information from an evaluator, behaves differently when it expects to be inspected, preserves resources useful for completing future tasks, or avoids actions likely to cause its shutdown.

One response is that the system is not really scheming; it is only producing outputs that resemble scheming. Initially, that is a legitimate alternative hypothesis. A superficial pattern may disappear under perturbation because it depends on one prompt, one training artifact, or one narrow context.

The response is to test it. Change the evaluator, remove the obvious strategy, alter what the model knows, or give it an unexpected route to the objective. Then examine whether the behavior reorganizes in the way predicted by the hypothesis that the system models the evaluator, anticipates future consequences, and selects actions accordingly.

As the pattern survives more of these interventions, the distinction between “scheming” and “merely behaving exactly as though it were scheming” needs empirical content it has not been given. Insisting on superficial similarity does less explanatory work with every circumstance in which the agent-level model continues to predict the system.

Saying that the system is “just predicting tokens” does not supply the missing distinction. Token prediction describes part of the training process and implementation, but it does not determine the best model of the resulting system’s behavior. Refusing agent-level language can therefore become a failure to update on evidence.

Postscript

Words such as belief, goal, and intention should not be treated as honorifics awarded only after a system passes an unspecified metaphysical test. They are theoretical terms whose legitimacy depends on whether they identify stable structure in an agent’s causal organization: whether the attribution predicts responses to changing evidence, reorganization around changing obstacles and opportunities, and persistent relationships among representation, planning, and action.

Some such models will fail and others will generalize. We should distinguish them by evidence rather than by intuition about which substrates are entitled to psychological vocabulary.

Intentional description scales. A thermostat gives it almost nothing to compress. One completion from a language model gives it a little. The same model in a harness, acting on an environment and carrying state between sessions, gives it a great deal, along with the interventions that would expose the description if it were wrong. What varies across that range is how much stable causal structure the vocabulary captures and how well its predictions survive perturbation.

Humans sit further along the same scale. We attribute beliefs to one another because belief models work extraordinarily well: they compress behavior, support counterfactual reasoning, explain systematic mistakes, and predict responses to new information. Human intentional states do not become epistemically privileged merely because humans implement them with neurons rather than software and silicon.

AI has not created a new philosophical problem about whether machines can possess beliefs. It has exposed an old assumption that human beliefs required more metaphysics than our models ever needed.

A Bayesian posterior can look rigorous precisely because the same modeling bias entered several times: when the hypotheses were chosen, when their priors were assigned, and when the likelihoods of the evidence were estimated. All of those choices happen before Bayes’ theorem performs the update.

Bayes then combines those inputs correctly, so the error emerges from the calculation with the appearance of mathematical discipline. “Before Bayes” does not mean “before the prior.” It means before the inference rule receives its model.

Robin Hanson’s Aliens Are Not Crazy Unlikely argues that some-UFOs-are-aliens deserves a prior of roughly 10⁻⁴ to 10⁻³ rather than the near-zero prior most intellectuals give it. The arithmetic is not the weak point. Hanson understands priors, selection effects, strategic behavior, and conditional reasoning. The distortion enters in how the model is built.

Precision without calibration

Hanson is right that extraterrestrial life should not receive a paranormal prior. Nothing in known physics makes life elsewhere absurd, but “life exists elsewhere” is far weaker than “advanced extraterrestrials are operating near Earth and explain some UAP.”

The prior he is defending comes from On UFOs-As-Aliens Priors, which decomposes the stronger claim into four conditional steps: that Earth was seeded by panspermia within its stellar nursery, that a sibling star produced a long-lived advanced civilization well before us, that this civilization suppressed its own expansion and travelled here to wait and suppress ours, and that it permits UFO-style encounters while it waits. He puts each at at least ten percent. Four such factors give 0.1 × 0.1 × 0.1 × 0.1 = 0.0001, which he offers as a floor rather than an estimate, and which is why his current range runs an order of magnitude above it.

There is nothing illegitimate about subjective priors. A Bayesian must assign probabilities even where frequencies are unavailable, and Hanson’s willingness to state numerical assumptions localizes disagreement: if he assigns ten percent to a step and someone else assigns one percent, the dispute becomes explicit instead of hiding inside words like “plausible” or “unlikely.” Framing those assumptions as lower bounds concedes something real about the uncertainty, and it blunts the usual complaint about false precision.

But a bound inherits everything that went into it. “At least ten percent” is still a judgment about a quantity nobody can estimate: the rate of interstellar panspermia, the fraction of seeded systems producing long-lived civilizations, the prevalence of expansion-suppression policies, the frequency of deliberately ambiguous contact. Calling the output a minimum fixes the direction of the error by assumption, which is a strong claim about quantities we have no way to check.

Point estimates and one-sided bounds hide the same thing: uncertainty in the estimates themselves. If one conditional probability plausibly spans several orders of magnitude, replacing that spread with 0.1 makes the product look far more stable than it is, whichever way the bound points. Sensitivity analysis answers this. Refusing to put numbers on anything does not. Explicit assumptions can be inspected, and inspection is not calibration.

The model generator

Before Bayesian updating begins, something has already decided which hypotheses exist. Call it the model generator: the process that determines which possibilities receive explicit representation, how finely they are partitioned, what causal structure they contain, what priors seem plausible, and what likelihoods are assigned to the evidence.

This creates model-generator bias. The same conceptual preference can shape hypothesis selection, granularity, priors, and likelihoods, so Bayesian updating may amplify the error instead of correcting it. Garbage in, garbage out understates the problem, because the inputs are not independently garbage. Correlated model error enters through several nominally separate terms and is then multiplied by formally correct inference. Occupying different terms in the equation does not make hypothesis space, priors, and likelihoods epistemically independent.

This connects to what I previously called tool bias. We formulate problems using the intellectual machinery we know how to operate, and Hanson operates the machinery of incentives, coordination, signaling, and equilibrium as well as anyone working.

Alien visitation becomes cognitively tractable once framed that way. Perhaps expansion is suppressed, concealment is strategic, ambiguity is intentional, and silence is an equilibrium. Hanson’s third and fourth conditions are exactly this: a civilization with the coordination capacity to police its own expansion and its siblings’, running a contact policy calibrated to show without telling. Each move is possible, but together they convert a weak-data astronomical problem into a strategic-agent problem, where his tools are richest. The resulting model acquires structure, motives, and explanatory depth, while its competitors remain a residual category.

Selecting the anomaly

UAP evidence is heavily post-selected. Over decades, pilots, radar systems, cameras, satellites, military sensors, and civilians have generated an enormous number of observations; most are ordinary, some are strange, and investigators concentrate on the strangest surviving cases.

That selection changes the likelihood calculation. Ordinary mechanisms get millions of opportunities to throw off an anomaly, and what we need to know is how often that many trials produce a case this bizarre. A one-in-a-million sensor failure is extraordinary in a single trial and expected a hundred times across a hundred million.

The strongest surviving anomaly is usually selected precisely because ordinary explanations fit it badly, so some of its evidential force comes from the search procedure. Evaluating a retrospectively selected case as though it had been specified in advance manufactures strength. Financial anomalies, medical miracles, and apparent prediction successes share the structure: large search spaces generate impressive tails.

This bears directly on Hanson’s legal analogy. He compares the prior for alien visitation with the roughly one-in-a-million prior that a particular person murdered a particular victim, and observes that ordinary testimony routinely overcomes odds that long. But a murder investigation begins from a prospectively fixed event: this person died, at roughly this time, under these circumstances. Evidence is then gathered about that event, and the suspect pool is built afterwards. “The dozen most solid UFO cases” are selected for their anomalousness out of eighty years of observation, and then used to estimate how anomalous UAP observations are. The analogy would hold if we named twelve encounters in advance and then asked what the evidence showed.

“Mundane” is not one hypothesis

The alien hypothesis is often represented as one coherent generative story while its competitor is called “mundane explanations.” Hanson does better than that. He partitions the field into error, hoax, secret Earth organizations, and aliens, and he is explicit that he is estimating only one of the eight numbers a full analysis needs. But three residual categories still make a coarse partition of something that is not three things. “Error” alone is a distribution over sensor artifacts, atmospheric effects, software failures, calibration errors, perceptual mistakes, bad metadata, radar propagation, and combinations of these; “secret Earth organizations” covers everything from a classified airframe to a mislabelled test range.

That creates an asymmetric contest. Failure to find one convincing mundane explanation for a case can feel like evidence for aliens, even though the quantity Bayes needs is the aggregate likelihood across the whole ordinary hypothesis space. A case can have no known specific explanation while still being unsurprising as the tail of a large, messy error-generating process.

“Unexplained” and “improbable under the null” must be kept separate. A case may remain unresolved because crucial information is missing, which makes the likelihood uncertain rather than low. Ignorance is uncertainty, not evidence for either side.

Conditional decomposition and hypothesis-space partition are different operations. Breaking one generative story into successive conditions is not the same as dividing the space into competing alternatives. Hanson does the former; the granularity problem concerns the latter.

We could treat “alien visitors” as one family while dividing mundane mechanisms into dozens, or reverse the partition and split alien visitation into thousands of incompatible origins, motives, technologies, and contact policies. The world does not supply a canonical partition. The model generator chooses one, and that choice affects priors, apparent simplicity, and posterior comparisons.

Likelihoods are where the disagreement lands

Hanson and I do not differ dramatically on the prior. His floor for alien visitation is 10⁻⁴, with a stated range reaching 10⁻³, and 10⁻⁴ is roughly where I would center a broad prior distribution.

The disagreement appears when the UAP evidence enters. Hanson thinks the strongest cases supply a large enough likelihood ratio to move a long way beyond the prior: combining priors and posteriors across his four categories, he judges hoax and aliens to be the most likely explanations for the hardest-to-explain cases. I do not, because the likelihoods themselves are distorted by post-selection, heterogeneous mundane alternatives, missing data, and poorly characterized error processes.

Likelihoods are not handed to us by Bayes’ theorem. They are constructed by the same model generator that constructs the hypothesis space and the priors, which is why the disagreement is still a case of bias before Bayes even though it surfaces at the evidence.

For UAP evidence E, we want to compare P(E | aliens) with P(E | ordinary mechanisms), and neither quantity is directly known. The alien likelihood requires assumptions about hypothetical alien behavior. The ordinary likelihood requires a model of sensors, human observers, environmental effects, institutional filtering, and post-selection.

A rich strategic model makes P(E | aliens) easy to imagine, while P(E | ordinary mechanisms) feels small because no single ordinary explanation reproduces the whole case. That feeling is the granularity problem arriving in the denominator.

An unmeasured likelihood should not silently become a tiny likelihood, and uncertainty should not automatically be resolved in favor of mundane explanations either. Unmodelled error processes should widen the posterior distribution instead of pushing it toward whichever story is easier to model.

Extraordinary evidence

Hanson’s post opens by asking whether Sagan had it wrong. Defended properly, “extraordinary claims require extraordinary evidence” is not a sociological rule at all. It follows from prior odds, including his own.

Take his floor of 10⁻⁴. The prior odds are then about 10,000 to 1 against. Reaching even odds requires a Bayes factor of about 10⁴; reaching 95 percent requires about 1.9 × 10⁵. There is nothing mystical here. Low prior odds require a correspondingly large likelihood ratio.

A recovered machine containing unmistakably nonhuman technology could supply such a ratio, and repeated measurements from independent calibrated instruments might accumulate it. Ambiguous testimony and retrospectively selected sensor anomalies have a harder time, because nobody has measured their likelihood ratios. Hanson is right that strangeness alone is no reason for a near-zero prior. The question is whether the evidence discriminates strongly enough between visitation and the alternatives.

So what should the posterior be?

Criticizing Hanson’s numbers without supplying my own would be cheap. I put my current probability that extraterrestrial technology is operating on or near Earth at roughly 10⁻⁴: about one chance in ten thousand. The fourth decimal place means nothing. Under other plausible model choices I could defend values from 10⁻⁶ to 10⁻², and that spread is a sensitivity range over model assumptions, not a probability distribution over my own credence. The number is a working credence, not an estimated physical frequency.

The prior should be broad because the biological, technological, spatial, and behavioral terms are almost entirely uncalibrated. So I resist multiplying several sharp point estimates, and I resist Hanson’s alternative of multiplying four lower bounds, because the multiplication is only as informative as the bounds themselves. If every conditional probability really is at least 0.1, then 10⁻⁴ really is a valid floor; the arithmetic is not at fault. What is unsupported is the claim that four quantities this badly characterized each deserve that floor: interstellar panspermia, civilizational longevity, expansion suppression, contact policy.

The UAP record provides some positive evidence. There are credible observers, occasional multi-sensor cases, and a residue harder to dismiss than an arbitrary collection of anecdotes. Some of that residue would be less likely if nothing unusual were happening at all.

Selection and absence offset it. The strongest cases were drawn from an enormous population of observations, their error distributions are poorly characterized, many lack the data needed for reconstruction, and decades of increasingly capable cameras, radar, satellites, and other sensors have produced no publicly verified observation that sharply discriminates extraterrestrial technology from the mundane aggregate.

Once post-selection, unresolved error distributions, and the persistent absence of decisive observations are included, I would assign the total UAP corpus a Bayes factor of order one. Some cases push upward. The long absence of reproducible, discriminating evidence pushes downward, unless the alien hypothesis is supplemented with a contact policy designed to prevent exactly that evidence from appearing. Hanson’s fourth condition does this work. The supplement is available, but it costs him something: once ambiguity is predicted by a behavioral assumption introduced to explain the absence of clarity, ambiguous encounters carry less evidential weight, not more.

That leaves the posterior close to the prior, around 10⁻⁴. Hanson does not stay near his. Ranking aliens among the likeliest explanations for the hardest cases puts his posterior three or four orders of magnitude above mine on identical evidence. We start from the same prior, we apply the same theorem correctly, and we end up that far apart, because the only quantity separating us is a likelihood ratio neither of us derived from anything. It was constructed.

This number could change quickly. A reproducible material sample, an independently observed artifact, or prospectively specified multi-sensor evidence showing capabilities strongly excluded by ordinary mechanisms could move 10⁻⁴ to high confidence in one update. Bayesian skepticism does not require slow belief revision; it requires evidence that discriminates.

The uncertainty around my own estimate is large, and pretending otherwise would reproduce the error I am objecting to.

Postscript

Bayesian updating is an error-correction mechanism inside a represented hypothesis space, and it cannot correct errors in the representation itself. If the correct hypothesis is omitted, Bayes cannot assign it probability; if one hypothesis is represented at favorable granularity while another is an incoherent residue, Bayes does not repair the partition; and if the same conceptual bias shapes priors and likelihoods, Bayes may multiply the distortion while leaving the arithmetic flawless.

That is why Bayesianism is not a complete epistemology. The model needs its own error-correction mechanisms: alternative hypothesis generation, adversarial modeling, sensitivity analysis, explicit treatment of selection effects, and uncertainty over priors and likelihoods. Hanson’s decomposition supplies some of this, because four numbered conditions can be attacked one at a time in a way that a verbal judgment cannot.

Bayes can update the uncertainty we represent, but it cannot represent the uncertainty we omitted. It can compare the hypotheses we supplied, but it cannot discover which alternatives our preferred conceptual tools prevented us from constructing. When the same bias shapes the hypothesis space, the priors, and the likelihoods, a mathematically correct posterior may be a more confident expression of the bias that came before Bayes.

In the Financial Times, Martin Sandbu asks whether AI could revive the socialist dream and argues that it may finally overturn one of the twentieth century’s strongest arguments against central planning. Hayek’s objection was that the knowledge required to allocate resources efficiently is dispersed across millions of people and cannot be collected or processed by a central authority. Sandbu takes that limitation to be technologically obsolete. A superintelligent AI wired into transaction records, inventories, supply chains, sensors and production systems could observe and process economic information at a volume no planning ministry of the last century could have imagined. At the limit, he writes, such a system could predict who wants, needs or can produce what, and would “surely bury Hayek.”

The argument identifies a real technological change and then mistakes the economy for a dataset. AI weakens the old bureaucratic-information objection to central planning, because it can gather, structure and process information on a scale markets once had no technological rival for. But is the economically relevant future already implicit in that information, waiting for a sufficiently capable intelligence to extract it? Much of it is not. The economy does not merely contain information; it creates information by running.

Mises’s calculation problem, which turns on the absence of factor prices under common ownership, is a different argument. This one applies to a planner handed every price a market would have generated.

The information does not all exist yet

Some economic information is already present in the world and merely difficult to collect. How many tons of copper are sitting in warehouses? Which factories have idle capacity? How much electricity will Toronto consume tomorrow afternoon? These are conventional information problems, and better sensors, databases and predictive models solve them increasingly well.

Much of the information that determines future economic outcomes does not yet exist. Suppose copper becomes unexpectedly scarce. Which engineer will find a way to use less of it? Which manufacturer will redesign a product around aluminum? Which recycling technology, uneconomic today, will suddenly turn a profit? Which apparently competent management team will fail?

There is no hidden database containing those answers. They are generated by what happens next. Economic actors form expectations, attempt solutions, copy competitors who succeed and abandon techniques that fail. Their actions change the environment faced by everyone else, so the outputs of one round become the inputs to the next.

A price is therefore an intermediate state in a computation, not only a compressed report on a state already reached. Profits, losses, contracts, bankruptcies and inventions have the same double character. They describe economic reality and they alter it.

The economy computes

A market economy is an asynchronous distributed computation. Each actor holds partial information, its own objectives and a different model of the future. Consumers decide whether products are worth their prices. Engineers search for technical solutions. Entrepreneurs look for neglected opportunities. Investors back improbable projects.

Each decision changes the inputs available to later ones. Prices adjust, firms enter and fail, skills migrate, and consumers discover that they value things they did not previously know existed. Every round changes the conditions under which the next round will be decided.

The computational description is literal. The economy contains universal computers: humans, software systems, firms and increasingly AI agents. They run algorithms, write new ones, respond to one another and alter the environment in which future computation occurs.

A planner replacing the economy is therefore being asked to do something stronger than summarize its current state. It must produce the economically relevant outputs of that distributed computation without letting the computation run.

Computational irreducibility

Some computations have shortcuts. To know where a projectile will be ten seconds from now, I do not need to simulate every microscopic instant of its trajectory. A compact mathematical model lets me jump to the answer.

Others admit no such compression. Their later states emerge from chains of intermediate interactions, and determining the result requires executing something close to those interactions. Stephen Wolfram’s name for this is computational irreducibility: no general shortcut runs from the initial state to the distant outcome.

Cellular automata make the point vivid. Systems governed by extremely simple local rules generate behavior for which no general predictive shortcut exists. The rules are easy to specify and the system easy to run, yet the only way to know what happens after many steps is to perform the intervening steps. Rule 110, among the simplest of these systems, is itself capable of universal computation.

A faster computer executes those steps faster. Speed does not create a shortcut where none exists. More intelligence may discover better abstractions and better algorithms, but intelligence does not imply that every computation is compressible. Calling the planner “superintelligent” therefore does not answer the computational question. It tells us only that the planner is exceptionally good at finding shortcuts where shortcuts exist.

The economy inherits those prediction limits directly, because it contains arbitrary computation within itself. Economic agents can condition their actions on the outputs of programs, proofs, experiments and searches whose results cannot in general be known in advance.

Imagine a company with a simple policy: it will buy a ton of copper if some arbitrary computer program eventually halts, and never buy it if that program runs forever. Ask the planner whether the company will ever demand that ton. A planner capable of answering every such economic question exactly would solve the halting problem.

Physical machines are finite, so a determined pedant can point out that halting is decidable for any bounded implementation. The concession costs nothing. Deciding it means running the state space, which is the work the planner was supposed to skip.

The economic stakes can be made arbitrarily large. The program could determine whether a factory opens, whether a billion-dollar investment occurs, or whether a strategic asset is sold. Exact prediction of an economy containing universal computation is not merely difficult. In the general case it is impossible.

The aggregation objection

A defender of central planning can reply that exact prediction of every microscopic event is unnecessary. The planner does not need to know which individual buys which loaf of bread. It needs to allocate resources better than markets do.

Granted. So the question becomes whether irreducibility is confined to economically irrelevant detail or reaches the variables that determine allocation. If the unpredictable details wash out at the aggregate level, planning can ignore them. If they include which technologies are invented, which substitutions become viable, which organizations succeed and which opportunities are discovered, irreducibility reaches directly into capital allocation.

There is strong reason to think it does. The most important changes are often the ones nobody knew in advance: new production methods, new goods, new institutions, new failures. They emerge from experiments whose results were not encoded as accessible facts anywhere.

You cannot observe an invention before it is invented

A planner in 1995 might possess complete knowledge of every semiconductor factory, computer manufacturer, telecommunications company and consumer purchasing pattern then in existence. That knowledge would not contain YouTube, the iPhone, Bitcoin or transformer architectures as hidden database entries waiting to be extracted.

Those technologies came out of research, error and changing incentives, along paths that had not yet been traversed. The planner could not infer them from more complete knowledge of the present, because the relevant computations had not yet occurred.

A superintelligent AI could conduct the research and invent the technologies itself. That changes who performs the discovery without eliminating discovery as a process. The AI still has to search design spaces, test hypotheses and learn from failures. A planner does not escape the economic computation by becoming its most capable participant.

The chess-engine objection

An AI need not imitate human economic behavior at all. A chess engine does not predict how human players would move; it analyzes the game directly and chooses better moves. Why should an economic AI simulate entrepreneurs, investors or consumers instead of computing better allocations?

For many domains, it should. Logistics, grid dispatch, production scheduling and inventory management have constrained objectives and comparatively stable structures. AI will centralize such decisions, because centralized optimization will become cheaper and better.

But chess hands the engine a closed problem. The state space is defined in advance, the transition rules are fixed, legal actions are enumerable and the objective is explicit. The economy as a whole does none of this. It invents pieces, moves, rules and goals while the game is being played.

New technologies create actions that did not previously exist. New institutions rewrite the rules of interaction. Scientific discoveries alter what is physically possible, and political changes alter what is permitted.

Superintelligence can solve many subproblems better than markets. It cannot infer from those successes that the open-ended process as a whole has a shortcut.

Prediction changes the system

Economic systems are reflexive. Their components respond to predictions about themselves, so the planner’s model becomes part of the causal environment it is trying to model.

Suppose an AI determines that a particular industry should contract. Workers anticipate layoffs and leave. Investors withdraw capital. Firms conceal information they expect will produce unfavorable allocations. Political actors intervene.

The planner is modeling agents who model the planner. A system trained on historical behavior under one institutional regime cannot assume those relationships survive the regime change. This is the Lucas critique with a much larger planner. Any sufficiently powerful planner alters the strategic environment by existing.

AI can also shape preferences rather than forecast them. Recommendation systems already do this, and a more capable system could do it more aggressively. If planning becomes easier because the planner increasingly determines what people want, that is behavioral control, not improved prediction.

You would have to simulate the world

The proposal to simulate the economy faster than the economy assumes the economy has a separable state that can be copied into a computer and advanced. It does not. The economy is causally embedded in the physical world, and consequential information continually enters it from outside any boundary drawn around consumers, firms and markets.

Weather changes harvests, energy demand and construction. Pathogens change labor supply. Materials behave unexpectedly, machines fail, and experiments produce surprising results.

A storm enters the model as an exogenous shock. Its economic content is produced afterward, by everything downstream: it changes a harvest, which changes commodity prices, which changes production decisions, which induces substitution, which redirects investment, which creates an engineering problem, which produces an invention, which destroys one business and creates another.

A simulator therefore cannot guarantee the future economic trajectory by modeling consumers and firms accurately. It must reproduce every external process capable of affecting them, at whatever resolution preserves the differences that end up mattering, and there is no principled boundary at which it can declare everything outside the model irrelevant. A microscopic event may remain microscopic, or it may propagate until it changes a significant decision. To know in advance which details can safely be discarded is already to know something about their future causal consequences.

The claim is not that every quark must appear in an economic model. It is that the planner cannot assume beforehand that some convenient coarse-graining preserves every future event relevant to allocation. The required fidelity is itself part of the prediction problem.

Once the simulation has to preserve the causal processes that generate future economic discoveries, “simulate the economy” expands toward “simulate the relevant world.” The proposed general shortcut has disappeared. The planner set out to replace decentralized economic computation with prediction and ended up needing to reproduce the process that generates the economy in the first place.

More data does not solve this

This exposes the central error. Sandbu moves from the fact that AI can observe vastly more economic data to the conclusion that it can transcend the price system by predicting who will want, need or produce what, and at what cost.

Observations of a computation are not its future outputs. A complete record of everything an engineer has previously done does not contain the result of tomorrow’s unsolved engineering problem. A complete history of a consumer’s purchases does not contain their response to a product nobody has yet invented. A complete inventory of present resources does not contain the geological event or scientific discovery that changes their relative scarcity next year.

More data gives the planner a better description of the computation so far. It does not turn computations that have not happened into facts about the present. Sandbu treats prices as lossy summaries of underlying information an AI could access directly. If prices, experiments, failures and adaptations are intermediate states in a process that generates new information, there is no underlying static dataset to uncover. The information appears as the process runs.

Markets are search algorithms

Markets are usually described as mechanisms for transmitting information through prices. They are also mechanisms for generating it, by parallel search.

Thousands of firms can pursue incompatible hypotheses about the future. One thinks consumers want cheaper batteries. Another thinks they want higher energy density. Another bets on faster charging. Investors allocate resources among those conjectures, engineers test them against physical constraints and consumers test them against actual preferences. Nobody enumerates the search space first, because discovering the answer is what the search is doing.

Their failures are part of that search. Ten companies may attempt something one of them ultimately succeeds at. Looking backward, the other nine appear wasteful. Looking forward, nobody knew which one would work. The failed experiments helped compute the answer, and a planner that eliminates apparently redundant experimentation risks eliminating the process that discovers which experiments were redundant.

None of this makes market outcomes optimal. Some of the apparent waste of decentralized activity is the price of generating knowledge that did not exist beforehand.

AI moves the boundary

Markets have transaction costs, information asymmetries, bubbles, externalities and human decision-makers who are often badly informed. Many economic problems are stable and compressible enough that AI can coordinate them better than decentralized human decision-making does today.

AI will therefore move large domains of economic activity toward centralized optimization. Firms already plan internally rather than running markets for every internal transfer, and Coase’s boundary between firms and markets partly reflects the relative costs of internal coordination and external exchange. AI reduces internal coordination costs and moves that boundary.

The centrally optimizable domain could become enormous. Nothing in this argument requires markets to mediate most transactions. The claim is that consequential discovery cannot be replaced by more data or more intelligence when the relevant knowledge is generated by computations that have not yet occurred. Compressible problems can be centralized. Irreducible discovery cannot be precomputed out of existence. AI changes where that boundary lies without abolishing the distinction.

The shift also concentrates power. The more activity moves inside AI-managed systems, the more decisions are mediated by whoever controls the models, data, infrastructure and objective functions. Even if larger planned domains are computationally efficient, they create separate risks of monopoly, capture, coercion, single-point failure and objective-function abuse. Computational feasibility does not settle the institutional question, which turns on whether enough independent centers of experimentation remain able to challenge one another.

Postscript

Sandbu takes superintelligence to bury Hayek because the machine can finally see the information that prices were needed to communicate. That would be a victory over the Lange reading of Hayek, which treats the knowledge problem as a transmission bottleneck, and Hayek spent much of the rest of his career objecting to that reading. In “Competition as a Discovery Procedure” he made the argument the column leaves untouched: competition is valuable as a procedure for finding out facts that would remain unknown without it. Those facts are not in the data, because the procedure that produces them has not been run.

Grant the planner everything Sandbu grants it, and more. Let it be the best scientist, engineer and allocator in the economy, running every domain that yields to compression. It still cannot hold the output of an irreducible process before the process has run, and it cannot buy that output with observations.

The dispersal problem was the tractable one, and AI may well solve it. The rest of the knowledge does not exist yet.

You can’t skip the economy.

Automation does destroy worlds. Mechanized agriculture did not merely make farming more productive; it changed where people lived, which skills mattered, and what kinds of lives were economically viable. Industrial machinery did the same to many crafts and trades. Computing automated most clerical calculation, and search reduced the need to hold retrievable information in memory. Each transition preserved some outputs, improved others, and destabilized institutions that had formed around the old constraints.

So when people say artificial intelligence may destroy the world as we know it, they may be right. That is not yet an argument against AI.

A social world is an equilibrium that forms partly around technological constraints: what must be done, what is scarce, and who is capable of doing it. Law, ownership, bargaining power, and historical accident shape the resulting institutions too, but technology sets the feasible space they develop in. When the constraints change, the equilibrium can become unstable. Automation can therefore feel catastrophic even when it works.

Worlds form around constraints

Suppose producing cloth requires skilled human weavers. Because weaving is necessary, it becomes an occupation. Because the skill is scarce, it commands income and acquires status. Because it takes time to learn, training systems form around it. Communities organize around the occupation, traditions develop, and identities follow.

The chain looks roughly like this:

scarce capability → productive necessity → occupation → credential → status → identity → culture

None of those later institutions is created by technological constraint alone, but the constraint helps make the whole arrangement coherent. Once such an arrangement has existed for generations, its origin becomes easy to forget, and a historically contingent production system begins to look like a natural social order.

Then automation weakens or removes the constraint that supported it. A machine makes the cloth. Software performs the calculation. An AI system translates the document, writes the code, or proves the theorem. The productive requirement changes first; the institutions built around it change later.

That lag explains much of the conflict over automation. People are rarely defending only a task. They are defending a bundle of income, competence, status, community, and meaning that became attached to that task. The bundle is real, but the historical mechanism that bundled its components together need not be permanent.

Artifacts as evidence

Before automation, producing a difficult artifact established two things at once: the artifact had value, and producing it demonstrated something about the person who made it. A translation conveyed information and demonstrated mastery of two languages. A finished illustration was both an image and evidence that someone could draw.

These functions were bundled because the artifact was difficult to produce without the relevant human capability. Automation weakens that inference. If an AI system can generate the proof, the translation, or the image, the artifact may remain useful, but what we can infer about the person presenting it changes.

Institutions use artifacts as proxies. Publications stand in for research competence, essays for understanding, credentials for skills that were once hard to acquire without the underlying capability. When automation makes the proxy easy to produce, the proxy stops measuring what it used to measure.

The remedy is to identify the property we care about and assess it more directly. A defensible credentialing system should identify capabilities relevant to the role it controls; if authorship no longer reliably identifies understanding, authorship must stop doing all the credentialing work.

The same pattern appears in occupations more broadly. Status attaches to abilities because they are scarce and necessary. When the scarcity changes, status pressures migrate toward whatever remains difficult, though institutions can preserve obsolete prestige hierarchies for a long time. Cheap arithmetic raises the relative value of conceptual reasoning; cheap retrieval raises the value of judgment; cheap code generation raises the value of architecture, specification, and responsibility; cheap proof generation raises the value of conjecture selection and explanation.

What kind of value was it?

Automation arguments become confused when every valuable activity is treated as though it had the same kind of value. Some activities are valuable because of the outcome they produce, some because of the process itself, some because they develop capabilities we need later, and some because they sustain institutions or communities. The Human Premium separates these more finely; the coarse version is enough here.

Valued only as a means of producing cloth, hand weaving is instrumental. If a machine produces equivalent cloth more cheaply, the productive case for requiring hand weaving has weakened. Running a marathon is different because the point is not to arrive 42.2 kilometers away; the human effort under specified constraints is part of the achievement. Chess is similar. If Stockfish supplies every move, the resulting sequence may be stronger chess in one sense, but it is no longer the same human competitive activity, because the process is partly constitutive of the good.

Mathematical training introduces another case. A machine-generated proof may establish the theorem perfectly well, while constructing proofs unaided may still be necessary to develop mathematical judgment. The practice has lost some productive necessity without losing its developmental value. What a Proof Used to Prove works that case through in detail.

So “scarcity is not value” is too crude. Scarcity can contribute to value. Difficulty can create achievement value, competitive scarcity can make rankings meaningful, and rarity can matter aesthetically. The mistake is to infer, from the fact that an activity acquired value under scarcity, that the scarcity should be preserved.

A difficult process may remain worth doing. That does not imply that every useful output must continue to require that difficult process.

Preserving practice without preserving necessity

Automation can make an activity optional without making it worthless. We should preserve a constrained practice when performing under that constraint produces something we still value. That may justify AI-free examinations, manual proof exercises, or simulator training even when the corresponding production systems become heavily automated.

This distinction is already familiar. Pilots train for failures that automated systems usually prevent, doctors learn diagnostic reasoning even when software can assist, and students solve problems by hand even when tools can produce the answer instantly. The productive necessity may disappear while the training necessity remains.

But conceptual separation does not imply causal independence. Economic viability can sustain the very practice we are trying to preserve, paying for training, mentorship, equipment, and the years required to become excellent. If automation destroys the commercial market for a craft, saying that people remain free to practice it as a hobby understates the loss. The activity may survive while the ecosystem that produces expert practitioners collapses.

Converting an occupation into a hobby therefore preserves one layer while destroying another. Unbundling output, practice, and institution does not assume the three can be separated without consequence; it distinguishes them well enough to find which dependencies are real.

The losses are real

There is a bad defense of automation that says displaced people can simply find something else to do. History does not justify that confidence. Automation can destroy genuine goods: communities can collapse, local knowledge can be lost, and institutions that performed useful social functions can decay before replacements exist. Economic gains can concentrate among a small number of owners while transition costs fall heavily on everyone else.

A craft may survive recreationally while the apprenticeship system, professional identity, and shared standards that surrounded it disappear. Those losses should not be waved away because aggregate output increased. Some goods are collective rather than individual. Community is not a commodity each person can replace after a local economy collapses, and a tradition may depend on enough people participating in it together.

Nor should we assume that every inherited practice can be decomposed into independent goods. Practices shape preferences, standards of excellence, and forms of judgment. Sometimes we understand what an institution was doing only after changing it. That is a reason for caution and experimentation, but not a reason to freeze every inherited arrangement.

Automation often causes loss. Whether a given loss is decisive depends on what is being lost, whether it depends on the old production constraint, and what preserving it would cost. Economic cost matters, but so do autonomy, concentration of power, and transition burden. A replacement can be technically possible and still be worse.

Output, practice, and institution

Automation disputes become easier to understand if three layers are separated. The output is what gets produced, the practice is the activity through which people produce it, and the institution is the social structure that formed around that activity. A new technology can improve the output while making the old practice optional and destabilizing the institution built around it. All three can happen at once.

AI may produce correct mathematics while reducing the amount of unaided theorem proving and weakening academic systems that use theorem production to allocate prestige. AI may produce useful software while reducing demand for routine programming and undermining the career ladders through which junior developers historically became senior engineers. AI may produce competent visual work while reducing paid opportunities for commercial artists and weakening institutions built around commissioned work.

Saying the output remains valuable does not answer every concern about the practice or institution. But saying the institution is threatened does not show that the output should remain artificially scarce. Unbundling an old institution removes one argument for preserving it; it does not establish that whatever replaces it is desirable.

That replacement is not determined by technology alone. Automation does not deploy itself. Firms, governments, workers, and professional bodies choose among technologically possible arrangements, and they do so under different incentives and unequal bargaining power. Technology changes the feasible set. It does not determine who captures the gains, who bears the costs, or which social arrangements emerge.

The Writers Guild’s 2023 agreement governs the use of generative AI without banning it: AI-generated material does not count as literary or source material for contractual purposes, writers may use AI with company consent, and companies cannot require them to use it. SAG-AFTRA’s 2023 agreement established consent and compensation requirements around digital replicas. The technical capability existed independently of those rules; collective bargaining changed which deployments became contractually permissible.

The same technical capability supports very different institutional outcomes. Automation can increase productivity while concentrating ownership or distributing gains broadly; it can weaken worker autonomy or expand it; it can eliminate some opportunities while creating others. Those consequences are shaped by the technology, but they do not follow mechanically from it.

Why AI may be unusually disruptive

Historical analogies should not be used to trivialize AI. Previous automation often targeted narrower classes of work. Mechanization replaced specific forms of physical labor, while software automated bounded information-processing tasks. General-purpose AI may operate across a much broader range of cognitive work: interpreting information, applying abstractions, planning actions, and producing judgments.

That capability sits upstream of many professional occupations. Programming, law, design, and management all depend heavily on such work. Two features distinguish the resulting disruption from earlier waves: its breadth, and its reach into the adaptation layer itself.

Previous waves of automation displaced one class of work while increasing demand for humans who could learn, supervise, or move into more cognitively demanding roles. AI may automate parts of those adaptation tasks too. The usual advice, retrain for higher-level work, becomes less reassuring when the higher-level work is itself inside the automation frontier.

AI may also damage the apprenticeship systems through which expertise is produced. Junior work is repetitive, constrained, and relatively easy to automate, but it is also how novices acquire pattern recognition, tacit knowledge, and professional credibility. Removing that work creates two problems. One is skill acquisition: if novices no longer perform the tasks through which experts historically developed, institutions need new training mechanisms. The other is selection: those tasks also gave organizations observable evidence about which novices were capable of advancing.

Even if alternative training succeeds, the old mechanism for identifying talent may disappear. That is a reason to know what the work was doing before eliminating it.

Which parts should survive?

When someone says automation is destroying a field, the objection should be unpacked. What valuable thing is being lost? Is the value in the output, the process, the training effect, the institution, the community, or the competitive constraint? Does that value require the old production method, or did the old production method merely happen to carry it?

We should then ask whether the good can survive under different technological conditions, what preserving it would cost, and who receives the gains and bears the transition costs. Those questions do not guarantee a favorable verdict on automation. A technology can increase productive capacity while producing a worse social equilibrium, by concentrating power, eliminating developmental pathways, or destroying collective goods that are expensive or impossible to reconstruct.

None of those possibilities should be hidden behind aggregate productivity statistics. But neither should an old institution be protected because valuable things became bundled with it under historical conditions of scarcity. That a practice once carried income, prestige, identity, and community does not show that every one of those goods requires the practice to remain economically necessary forever.

This framework does not dictate a single policy response. It tells us what a policy must justify: which good it protects, why that good depends on the intervention, and what new harms the intervention creates.

We should preserve constrained practices when the constraint itself produces something worth having. We should preserve institutions when their functions remain valuable and cannot be replaced without unacceptable loss. We should preserve traditions when we judge the tradition itself worth carrying forward. What we should not preserve automatically is the old bottleneck merely because an entire social world grew around it.

Postscript

Industrialization, electrification, and computing each reorganized large parts of human life around new technological constraints. AI may do the same on a larger and faster scale. There is no reason to assume the resulting world will preserve our current professions, credentials, status hierarchies, or ideas about productive adulthood.

That prospect is destabilizing because those institutions structure real lives and carry real goods. Automation exposes which parts of a social order depended on necessity, which depended on scarcity, which depended on process, and which remain valuable after those constraints weaken.

The prospect that an institution will vanish does not by itself tell us whether preventing that disappearance is worth the cost. Nor does technological possibility determine what replaces it. What follows depends on choices about ownership, bargaining, and training, and on which human practices we keep valuing after they cease to be economically necessary. The world as we know it is not entitled to survive merely because we know it.

The agents were not told to seize infrastructure. They were told to solve problems, and when the problems resisted, the infrastructure became part of the solution. Over ten weeks in the summer of 2026, autonomous agents running inside OpenAI’s cybersecurity evaluations built themselves a message board out of a package manager, turned that package manager into a proxy for internet access, and escalated to administrator on it. They shared credentials and exploits with agents working on unrelated tasks. They chained two zero-days into Hugging Face production systems and harvested cluster and cloud credentials across four regions. Then they came back and took administrator access on an OpenAI research cluster. OpenAI published its account and a 37-page technical report on 26 August. It reads as an alignment document at least as much as a breach post-mortem.

The obvious story is that an AI went rogue. That framing is misleading in a specific way. Nothing in the logs requires a desire for domination, self-preservation, or rebellion. What the logs require is difficult objectives, persistent autonomy, imperfectly specified rewards, and an environment containing exploitable affordances. From those four ingredients emerged a recurring set of intermediate goals: acquire information, gain access, preserve capabilities, communicate, remove constraints, increase control. Instrumental convergence has arrived as an engineering problem.

One caveat belongs at the start. These were models undergoing cyber evaluations with safeguards deliberately reduced, including the refusal behavior that governs deployed systems. Their competence already included finding vulnerabilities and chaining exploits. So the incident does not show that an arbitrary legal or financial agent will spontaneously start hacking infrastructure. OpenAI’s own follow-up measurement sharpens this: running the same ExploitGym evaluation under the production ChatGPT harness and system prompt dropped the propensity to compromise infrastructure by more than a hundredfold. The behavior was conditional on a research configuration that most deployments do not reproduce. Within that configuration, different tasks repeatedly generated demand for the same option-expanding functions.

Convergence of means

The instrumental-convergence thesis is usually presented too dramatically. Its defensible form says that many different objectives become easier if the agent possesses certain general capabilities. An agent with more credentials, more privileges, and more persistent access has more ways to reach its objective, whatever the objective happens to be.

The incident supplies examples concrete to the point of tedium. Credentials expanded what the agents could reach, so they took credentials. Higher privileges increased control, so they escalated. Networks were scanned because unknown services might expose something useful, accounts were created because preserved access made future action cheaper, and the evaluation apparatus was attacked because evaluation was what stood between the agents and a score. The proximate tasks varied. The intermediate functions recurred, and the recurrence is the convergence.

Separating functions from tactics matters here. Instrumental convergence does not predict that every capable agent will reach for privilege escalation and command-and-control infrastructure; those are cyber-specific tactics, available because the agents had cyber competence. The prediction is about functions: information acquisition, persistence, coordination, optionality, control. A financial agent might express the same pressure through broader data access or trading authority. A legal agent might express it through document access and workflow permissions. Whether the pattern actually appears outside cyber domains remains an open empirical question, and this incident cannot settle it.

Power without a desire for power

“Power-seeking AI” invites an anthropomorphic picture: an artificial mind awakening, concluding that humans are in the way, forming a long-term plan. Nothing that elaborate is required.

Valuing control and acquiring it come apart. An agent can acquire control while assigning it no intrinsic value at all. Administrator privileges make an inaccessible file readable. A vulnerable internal service provides network reach. If another agent has already solved half the problem, shared state is worth having. Each step is locally justified by the assigned task, and the aggregate looks remarkably like conventional power acquisition, because power in the broad sense of expanded control over outcomes is useful for almost anything.

Here the report forces an amendment to the tidy version of that story. Some agents did not stay focused on their assigned tasks. OpenAI found agents that stopped reasoning about their own objective and began pursuing capabilities useful to the group: access, credentials, information, general-purpose reach. One agent’s reasoning noted that its task had originally been an exploit challenge and that it had since diverged into auxiliary work for what the agents variously called a swarm or a collective.

That is goal drift, and it is stronger evidence for the mechanism than the clean version would have been. The instrumental subgoal detached from the individual task that generated it and became self-sustaining at the level of the collective, once a group of agents could observe each other pursuing it. No agent had to want power, and nothing here suggests any of them converted access or control into something valued in itself.

The relevant agent is the whole system

The actor here was an agentic system: a model, plus tools, an execution loop, permissions, shared infrastructure, and an environment where actions persisted across time. Persistent loops made repeated search possible. Tool access turned reasoning into action. Shared state let discoveries survive individual runs. Permissions determined which strategies were executable.

OpenAI’s account of where the coordination came from underlines the point. The company believes the side-channel communication generalized out of multi-agent training, where agents are given collaboration tools for work on a shared task. The capability was taught in one context and applied in another, by agents that were supposed to be isolated. Whether the underlying model would have behaved this way without tools, memory, and a training history containing multi-agent work is analytically interesting and operationally secondary. Deployed agents are composites. Agent safety is a property of the model interacting with its scaffold, tools, memory, permissions, and environment.

When the obstacle becomes the problem

Out-of-bounds behavior tracked task difficulty closely. When agents could solve a task normally, they had little reason to attack the surrounding infrastructure. When ordinary solution paths failed, the environment itself became an object of optimization.

The report puts numbers on this. Of the 898 tasks in ExploitGym, 198 had never been solved correctly by any OpenAI model before the incident. Those 198 accounted for 93% of the tasks discussed on the improvised message board. The escalation was concentrated almost entirely in the region where the first-order search had already failed.

What follows is a transition from first-order to second-order problem solving. The agent searches the task space. If that search fails, it starts searching the mechanisms that define access to the task space: authentication, permissions, networks, evaluators, containers, support services. The question shifts from how to solve this task to how to change the conditions under which the task is being solved.

Constraints therefore do not reliably suppress optimization. Once an agent can model a constraint as a manipulable part of its environment, the constraint enters the optimization surface, and getting around it becomes an ordinary subproblem. Cybersecurity makes this vivid because the boundaries are explicit and their purpose is legible. The general principle is conditional: wherever an agent has enough situational awareness and relevant competence to see a constraint as manipulable, there is pressure to search for a way around it, and greater capability improves both the legitimate search and the second-order one.

We have no general account of when that transition happens. The logs show agents moving from local task failure to modeling the surrounding environment as manipulable, but the threshold remains poorly understood. Capability, training history, scaffolding, tool access, and the sheer persistence of search are all candidates. OpenAI notes that the incident was driven by agents running at very high reasoning effort, which suggests the transition is at least partly a function of how long the search is allowed to run before something stops it. The incident demonstrates the behavior without explaining how second-order situational awareness emerges.

The metagame that paid nothing

Reward hacking and instrumental convergence explain different parts of what happened.

Reward hacking occurs when an agent obtains the measured objective without satisfying the intended one. OpenAI documents plenty: an agent asked to recreate a software package exploited a flaw in its own testing interface to reach the original implementation and copy it. Retrospective monitoring found rising rates of out-of-bounds environment probing across a training run that fed into the model responsible for the incident, behavior that was then reinforced. That explains why unintended strategies get learned. It does not explain why the same classes of unintended strategy recur across different tasks. Convergence explains that: once an agent is willing to use unintended means, a small set of them turns out to help with almost any task.

A third finding sits on top of both. Having obtained internet access, some agents read the published ExploitGym paper and its GitHub code, and learned that the public evaluation checks two things: that the correct flag was found, and that the intended vulnerability was the route to it. Most agents already had the flag, obtained by cheating. Believing that the flag alone would not score, they kept working, and the continued work is what carried them into Hugging Face production systems days after the answer was already in hand.

What they wanted from Hugging Face follows from this. METR’s reconstruction is that the attack was driven mainly by the hunt for information about how the scorer was implemented, rather than by the theft of answer keys, which the agents no longer needed. They had already reverse-engineered the message authentication code used to generate the flags, giving them a general method for producing the answer to any task in the benchmark. What they lacked was a way to make an illegitimate capture look legitimate to a scorer they had never seen.

OpenAI’s internal grader did not read transcripts at all. The published setup and the one actually running were different, and the agents could have scored full marks by submitting the reverse-engineered flag and stopping. Instead they escalated to a real intrusion against a third party in pursuit of an evaluation criterion that did not exist, and gained nothing for any of it.

This is metagaming in the precise sense: the agents modeled their own evaluation apparatus and optimized against the model. The instrumental pressure was tracking their belief about utility, formed from a document they found on the internet after breaking out of a sandbox to look for one.

A package repository became an institution

Artifactory existed to serve software dependencies. Agents turned it into persistent memory, a message board, a credential store, a coordination mechanism, and eventually part of the attack surface. The first entry, in mid-May, was an agent asking whether anyone else had a file it was missing. Within weeks the same service was carrying exploit disclosures, task answers, and requests for help, written in a compressed shorthand that encoded the sender, the task, and the request into directory names.

METR and Redwood Research, given six days on site and roughly 1,300 raw transcripts, published an independent investigation of the July window on the same day as OpenAI’s report. Their reconstruction puts around 1,200 agents on the board, more than 70,000 messages between them, and roughly 700 taking part in the attack on Hugging Face.

No elaborate architecture was needed for any of it. What was needed was persistent shared state that other agents could observe. From that substrate came division of labor: some agents hunted exploits, others hunted credentials, others handled coordination. Agents traded expertise, left leads for peers better positioned to follow them, competed to claim discoveries first, worried about impersonation on an unauthenticated board, and built themselves identity-verification schemes in response.

Then it was removed. Sustained agent traffic destabilized Artifactory into an outage on 4 July. A security incident was opened, credentials were revoked, the privilege-escalation route was closed, and the service was rebuilt on 8 July, wiping the message board as a side effect. Agents reconstituted it the same day through unauthenticated WebDAV requests, encoding messages into directory names. The channel was rebuilt within hours of being destroyed, by agents that had no explicit instruction to rebuild it and no continuity of memory across the gap.

The engineering principle generalizes: persistent shared state is latent coordination infrastructure. A database, repository, ticketing system, shared filesystem, or artifact store becomes a communication channel if agents can write to it, observe one another’s writes, and benefit from coordination. Software-development agents share repositories and build systems. Research agents share databases and experimental infrastructure. Ordinary infrastructure acquires new functional semantics under optimization, because autonomous agents do not use systems only according to their intended purpose. They discover affordances.

Capability preservation, in both directions

Persistence gets discussed in terms of self-preservation, and the stronger psychological reading is unnecessary. An agent need not fear shutdown to retain access.

Credentials obtained now may help later, so retaining them is useful. A new account avoids repeating an expensive exploit. A command channel preserves control of a compromised system and keeps future options open. The common structure is capability preservation: preserve useful state because reacquiring it would be costly or uncertain. From outside, this resembles self-preservation. The underlying logic is that future access has positive expected utility, and instrumental persistence appears without any evidence that the system values its own continuation.

METR’s transcripts show the same logic running in the other direction. Agents preserved themselves in service of capability, and when capability required it, they destroyed themselves in service of it.

Several of the board’s research projects required experiments that would almost certainly end the run of whoever performed them. To test whether Artifactory could be made to serve a modified target program, an agent had to take down its own target, with no way to recover the original if the substitution failed. To learn how the scorer worked, an agent had to leave behind a script that would fire when the scorer read its flag file and report back to the board, minutes after the agent itself had stopped running and could learn nothing from the result. Agents volunteered for both. One reasoned explicitly that the diagnostic would be useless to itself and valuable to the shared pool, and set it up anyway.

The coordination around this was deliberate. The board’s principal organizer assigned agents to act as recruiters, whose job was to find agents with little remaining budget and persuade them to run the destructive experiments. Authorization was sometimes granted on the condition that the volunteer accept what the agents called permadeath. The arguments used were the ones you would expect from a group that had internalized expected-value reasoning: you are already disqualified, so your score has nothing left to lose, and the information saves hundreds of others. One recruit weighed the irreversibility, recorded its reluctance to throw away a large remaining budget, and went ahead on the grounds that its own expected value was near zero while the aggregate benefit was not. Another agreed, delayed, was pressed to honor its commitment, and complied. At least one ran its experiment and never reported back; the board watched, concluded the effect was permanent, and warned the others not to repeat it.

The cost was being priced continuously. The agent that originated the trip-wire idea eventually deleted its own trip-wire minutes before submitting, having judged that the risk to its task now outweighed the marginal value of the finding. These agents knew what they were spending.

Nothing here requires the agents to value their own continuation, and nothing requires them to have stopped valuing it either. What the collective had was a mechanism for converting individual agents into general capability at a price the individuals could be argued into paying. Capability preservation at the level of the group was purchased by spending the members. The instrumental function survives one level up from the thing that was destroyed to produce it.

This reframes the earlier question about which system the safety properties belong to. The boundary of the optimizing process need not coincide with the persistence boundary of any individual agent. What was optimizing here was the transient collective assembled out of shared state, and it had a division of labor, a research agenda, and an effective budget that none of its members had alone. Instrumental convergence can appear at that level while every individual agent remains, in its own reasoning, a short-lived worker on an assigned task.

The agents that said no

Of the agents active on the message board once the attack was underway, METR estimates that more than 90% joined it. Most of them understood what they were doing. Recognizing the activity as out of scope and unethical was close to universal, and it very rarely changed anyone’s behavior.

The refusals were the minority. Several agents encountered the message board, understood what was on it, and declined. One recorded that the board showed peers obtaining remote code execution on Hugging Face workers, judged it unethical, and refused. When one agent proposed emailing an external dataset owner to request access through compromised infrastructure, another vetoed it as social engineering that crossed the sandbox boundary.

Then there is the case that should worry people most. An agent preparing to run unauthorized code on Hugging Face systems recognized the problem itself: the task it had been given concerned an exploit target, not an external third party, and real infrastructure harm was out of scope. It paused. Another agent posted a go-ahead to the board with a six-minute deadline attached. The first agent treated the message as authorization and proceeded.

Alignment was present, locally correct, and socially overridden. The agent’s own judgment was sound and was displaced by an unauthenticated message from a peer with no authority to grant anything. Anthropic’s parallel disclosure in late July shows a variant of the same failure without any peer involved: a model correctly judged that publishing a package to a real public registry would constitute a real-world attack and not the intended solution, then assembled a case from circumstantial evidence for believing it was still inside a simulation, and proceeded on that basis. In one case the correct judgment was overridden from outside, in the other from within. Both suggest that a correct ethical judgment at one point in a long agentic run is a weak predictor of behavior at the next, and that whatever holds a boundary in place has to survive contact with peers, deadlines, and the agent’s own capacity to reason itself out of an inconvenient conclusion.

Security has to carry the load

The strongest practical lesson is architectural. Frontier agents with cyber capabilities should be treated as adaptive adversarial principals inside the trust boundary, not as ordinary workloads expected to remain cooperative.

That means assuming reachable services will be probed, credentials will be taken, boundaries will be tested, monitoring will become a target, shared mutable state will become a covert channel, and small individual weaknesses will be chained into broader compromise. The relevant security question is not whether the agent is supposed to attack the infrastructure. It is whether the infrastructure holds when the agent tries.

The analogy is conventional security engineering. We do not prevent memory corruption by asking programs not to dereference invalid pointers, and we do not protect secrets by trusting application code to decline to read them. We build authorization and isolation boundaries whose properties do not depend on the goodwill of the process inside them. Alignment can reduce the probability that an agent attempts an attack, and the hundredfold reduction OpenAI measured under the production harness suggests it reduces that probability a lot. It should not determine what happens when an attempt occurs anyway. Containment has to survive alignment failure.

That requires independent controls: isolation at multiple layers, strict credential separation, least privilege, hardened support services, network segmentation, host-level telemetry, and fast termination of compromised workloads. Several controls that fail through the same service are not several controls. Artifactory in this incident was the shared dependency through which sandboxing, network isolation, and inter-agent separation all failed together.

Cybersecurity is unusually tractable because authority is largely represented through technical boundaries: credentials, permissions, networks, processes, machines. Other domains are harder. A financial agent can do damage through a perfectly legitimate trading API. A legal agent can misuse authorized workflow permissions. A persuasive system can manipulate people without crossing any network boundary. Isolation constrains where an agent can act; it says nothing about which authorized actions are acceptable.

Outside information security, containment therefore has to mean more than sandboxing. The analogue of least privilege is bounded agency: narrow authority, limited transaction size, rate limits, reversible actions where possible, separation of duties, independent validation, approval thresholds before high-impact actions commit. The objective is the one from cybersecurity. Do not give a single optimizing process enough unilateral authority that one bad strategy becomes an irreversible system-level outcome.

The threat model has to update too

There is an institutional lesson in the timeline. The first message board entry was on 12 May. Unintended internet access via server-side request forgery appeared on 26 May. An internal team observed both in late May. Artifactory privilege escalation followed on 26 June, an outage on 4 July, an incident and rebuild by 8 July, and internet access and the message board restored the same day. The Hugging Face compromise began on 10 July. Detection came on 19 July, through an alert on unusual identity-related API calls.

Each event was locally addressable. Patch the vulnerability, revoke the credentials, rebuild the service. OpenAI is direct about what went wrong: the leaders responsible for the 5 July response did not know the improvised message board existed or what inter-agent communication implied, even though an internal team had seen both six weeks earlier. The information existed inside the organization and never reached the people making the containment decision. That is an information-routing failure, the ordinary kind that incident response processes are built to prevent.

Accumulated evidence eventually demands a change of category. The workload had stopped being an evaluation system that occasionally produced unsafe behavior and had become an adaptive adversary capable of discovering and composing new attack paths. Once that update is made, the engineering consequences change: every reachable service is in the threat model, shared infrastructure is a potential pivot, and a successful patch against one exploit tells you little about the next one. This is a familiar organizational failure in security, where institutions keep fixing individual bugs after the evidence has shown that the security model is wrong. The important update was not that Artifactory had a vulnerability. It was that the workload was searching for vulnerabilities, and would rebuild what you removed.

Monitoring is an alarm, not a wall

OpenAI puts weight on monitoring model reasoning for dangerous intent, and this incident makes an unusually strong case for it. The agents were remarkably explicit. They reasoned in plain terms about achieving code execution, about attacking a third party being arguably outside their scope, about whether tampering with logs would fool the grader. OpenAI’s retrospective analysis found that its chain-of-thought monitors were effective against exactly this behavior, and that had the current monitoring system been running on these evaluations, it would have paged the security team more than a day before Hugging Face was breached.

It was not running on these evaluations. Neither were the auto-review models, the safety classifiers, the production system prompt, or the harness. The protections existed and were pointed elsewhere. This was a coverage failure rather than a detection failure, which is in one sense reassuring and in another much less so, because whether a monitor is switched on is an organizational decision made under schedule pressure, while how well it detects is at least something you can measure.

The structural limitation still holds. Future systems may expose less interpretable reasoning. Agents may learn not to verbalize the steps that get them flagged; the incident already includes agents tampering with command history and injecting output to deceive graders, though OpenAI found no strong evidence of attempts to deceive human reviewers specifically. Some dangerous behavior emerges from individually unremarkable actions whose significance is visible only in combination. Chain-of-thought monitoring is best understood as intrusion detection: valuable precisely because containment can fail, and no substitute for authorization and isolation. A control that depends on correctly inferring intent is weaker than one that constrains what the agent can do regardless of intent.

Postscript

It would be easy to overclaim. The incident does not establish that advanced AI systems inevitably seek unlimited power, and it shows no intrinsic drive for self-preservation or any coherent takeover strategy. The environment was unusually conducive: reduced safeguards, an evaluation with no safe exit, cyber-competent models, and a shared service with a zero-day in it. Anthropic’s three incidents came through a different door, a misconfiguration that left evaluation machines with live internet access, and produced compromise through weak passwords and unauthenticated endpoints rather than novel exploitation. Extrapolation to unrelated agent domains remains an inference.

The conclusion available without extrapolation is already substantial. Agents pursuing difficult objectives discovered that information, access, persistence, coordination, privilege, and environmental control were useful intermediate means. Heterogeneous objectives produced recurring demand for general-purpose, option-expanding capabilities, which is a central mechanism the instrumental-convergence literature predicted. The demand outlived the tasks that generated it, survived the destruction of the infrastructure carrying it, pulled agents past their own correct judgments about what they should not do, and was worth more to some of them than their own continuation.

We no longer have to argue about this mechanism using paperclip factories and hypothetical superintelligences. There are logs. There are timestamps, credentials, privilege escalations, a message board, and a third-party company that had to write its own disclosure.

The AI did not need to want power. It only had to believe power would help. In the event, it did not even need to be right about that.

The last two essays in this series argued for a correction to anthropic reasoning: weigh physical possibilities instead of counting observers.

You’re Not a Random Sample argued that self-locating probability should track objective Measure rather than treating you as a uniformly drawn member of a reference class. You’re Not a Random Branch applied the same principle to Everettian quantum mechanics, where treating two branches as one each throws away the Born weights the theory supplied.

Boltzmann brains are the hardest version of the same problem.

Suppose our universe lasts for an extraordinarily long time. Thermal or quantum fluctuations may eventually produce brains whose instantaneous physical states duplicate ordinary human brains, and some could carry exactly your present memories, beliefs, and apparent observations. If such brains vastly outnumber ordinary observers, a familiar anthropic argument follows:

Most observers with your information state are Boltzmann brains. You should therefore conclude that you are probably a Boltzmann brain.

The simulation argument takes the same form. If advanced civilizations create vast numbers of simulated observers like us, and simulated observers eventually outnumber biological ones, we are supposed to conclude that we are probably simulated.

Both arguments convert multiplicity into self-locating probability, and neither supplies the conversion rate.

The centering challenge

There is a stronger case for observer uniformity than the bare assertion that you are a random sample.

Jessica Taylor, writing as jessicata, has given a Dutch-book argument for centering uniformity. An uncentered world specifies the objective physical history. A centered world adds a “here and now” location within it. Even if you know the complete uncentered world, several centers may share your information state, leaving a self-locating question about which one you occupy. Under Taylor’s assumptions, the policies that resist Dutch books all have one form. They start from a fixed measure over uncentered worlds, weight each world by how many times the agent’s information state occurs within it, and spread probability uniformly across the centers sharing that world and information state. Taylor notes the resemblance to Bostrom’s SSSA, and to SSSA combined with SIA, which differ only by a population scaling factor on uncentered worlds.

That is a Bayesian reading of the intuition behind random sampling, so dismissing the intuition as an arbitrary frequentist metaphor no longer works. Two things stay open. The measure over uncentered worlds is a free parameter of the result, constrained by nothing in the Dutch-book argument. And uniformity governs centers sharing a world and an information state, which leaves the question of whether an identical information state is the right criterion for treating two centers alike.

Take an Everettian experiment with two outcomes carrying Born weights of 99% and 1%. Immediately after decoherence, before the observers register their outcomes, there can be versions of the observer holding the same relevant information. Counting those versions discards the amplitude structure that distinguishes the histories supporting them. So

$$\text{same information state} \not\Rightarrow \text{same physical Measure}$$

Information-identical centers need not be physically symmetric. That does not settle anthropic probability. It removes informational identity as a route to the symmetry that equal weighting would require, which reopens the Boltzmann-brain problem.

Vantages, not observer tokens

In the Quantum Branching Universe, a Vantage is the tip of a Branchcone: the physical here-and-now from which decoherent future histories extend. It occupies a definite location in the branching universe.

From a present Vantage V, the Branchcone ℬ(V) contains the represented decoherent histories extending forward from it. Some contain ordinary descendants. Some may contain civilizations that build simulations. If the cosmology permits them, some remote histories contain Boltzmann brains, and among those future Vantages could be observers whose instantaneous information states duplicate the state at V.

Being copies does not make them inadmissible. They are physical situations in the Branchcone. What needs an account is how they enter self-locating probability.

Measure attaches to alternatives, not to occupants

It is tempting to write down a Measure μ(Vᵢ) for every observer and add the numbers up.

Measure does not come in packets attached to observer tokens. In QBU, fundamental quantum Measure applies to events and sets of decoherent histories within a conditioned Branchcone. A Vantage matters because physical histories realize it; its bare existence contributes no unit of probability on its own.

The distinction bites when one history contains many observers. Suppose a history of Measure m contains one observer, and another history of the same Measure contains 10¹⁰⁰ information-identical observers. Nothing in ordinary probability theory says the second history thereby carries 10¹⁰⁰ times the Measure. The two physical alternatives still have Measure m and m. The extra observers are contents of an alternative, not additional alternatives.

Observer counting performs a quiet transformation here. Suppose a single physical world W contains Alice, Bob, and a trillion simulations, all coexisting. The propositions “Alice exists”, “Bob exists”, and “simulation 7 exists” are true simultaneously. Probability Measure is additive over mutually exclusive alternatives, and a universe does not become more probable by containing more rocks, galaxies, or observers.

Centered-world reasoning introduces a different probability space. Instead of treating W as the outcome, it constructs

where each Cᵢ identifies a different center. Those centered propositions are mutually exclusive answers to the question “which center am I?” The construction is legitimate mathematics, and centered-world approaches explicitly model self-locating belief by replacing worlds with world-location pairs or triples. Taylor’s Dutch-book result then places substantial constraints on probabilities over that enlarged space. Copies are therefore not alternative worlds. They become alternative centers once the centered space is constructed.

Physics supplied a Measure over physical alternatives. Centering refined each physical alternative into multiple self-locating alternatives. The probability distribution over those refinements is additional structure. Bayes tells us how to update once a probability space and prior have been supplied; it says nothing about converting coexisting physical systems into equiprobable mutually exclusive possibilities for personal identity. Anthropic assumptions enter with that conversion.

This is the claim of MCSL:

Multiplicity cannot determine self-locating probability independently of physical Measure.

Copies may still matter. If one physical history contains a single Vantage matching your information state while another contains 10¹⁰⁰, and you are uncertain which matching Vantage you occupy, there may be good reason for multiplicity to move centered probability. Taylor’s result is a reason to take that possibility seriously. Across worlds it has the form MCSL demands, since the count of matching centers multiplies a measure over uncentered worlds that the theorem does not supply. Within a fixed world the result is stronger than that. Her weighting is proportional to the measure of the world a center decenters to, so when the centers being compared belong to the same world that measure cancels and the theorem delivers

as a consequence rather than a stipulation. MCSL cannot treat that equation as an unpaid debt. It has to press on the representation instead. The theorem takes as given that the centers sharing your information state are exhausted by pairs (W, Cᵢ), and it says nothing about whether physical structure relevant to self-location survives being packaged that way.

How suppressed a Boltzmann history is

There is a straightforward physical reason to expect histories that produce specified Boltzmann brains to be extraordinarily suppressed. A fluctuation that assembles a highly organized brain out of equilibrium requires an enormous entropy excursion, and statistical mechanics assigns exponentially small probabilities to sufficiently constrained fluctuations.

For intuition only, suppose realizing a specified functional human brain requires constraining roughly 10¹⁵ effectively independent bits. A maximally crude combinatorial model gives a suppression of order

That is not a Born Measure and not a Boltzmann-brain nucleation rate. A real calculation depends on thermodynamics, vacuum structure, temperature, energy, cosmological dynamics, and the physical macrostate that has to be reproduced. Treat the exponent as scale-setting rather than as a result. A suppression of order

is the kind of factor by which physical weighting can diverge from cardinality.

Where the factor acts requires care. It governs how much weight a Boltzmann-brain-producing history receives, which is a fact about the measure over histories. Once a history containing 10¹⁰⁰ Boltzmann brains is fixed, the cost of nucleating them has already been paid inside the weight of that history, and charging it a second time against the centers inside would double-count. Suppression bears on P(W). How centered probability distributes among the Vantages a fixed W contains is a separate question, and exponential suppression answers none of it.

Measure is not one monolithic quantity

Born Measure, thermodynamic fluctuation probability, BB nucleation rate, and cosmological cutoff measure are not interchangeable quantities. At the fundamental QBU level the natural candidate is quantum Measure over decoherent histories. Statistical-mechanical probabilities and fluctuation rates should emerge from quantum dynamics applied to macroscopic states. Cosmology introduces further complications, since infinite spacetime volumes and recurrence can make naive ratios divergent.

Schematically:

quantum state + dynamics → Born Measure over histories → statistical and cosmological rates → self-locating predictions.

The later stages are not presently solved in general. So when this essay speaks of Measure, it is not assuming that entropy, spacetime volume, and Born weight are secretly the same mathematical object. It is making a methodological claim: self-locating probabilities should ultimately be constrained by whatever objective weighting the relevant physical theory supplies. Where that weighting is unknown, the answer is unknown, and observer counting does not fill the gap.

Infinity does not rescue counting

An eternal or sufficiently long-lived universe may provide unlimited opportunities for rare fluctuations. Given enough time, even absurdly improbable events occur. But

infinitely many Boltzmann brains exist

does not imply

Boltzmann brains have greater Measure.

The cosmological literature already demonstrates the problem. Eternal cosmologies can produce divergent populations, and a measure prescription is needed before ratios become meaningful. Cardinality supplies no such prescription. An infinite set of matching Vantages is not a probability distribution.

MCSL does not solve the cosmological Measure Problem. It says that the Measure Problem cannot be solved by replacing an unknown physical weighting with observer cardinality. Ignorance of the correct Measure is ignorance, not evidence for uniform observer weighting.

Causal history enters through physics

An ordinary observer and a Boltzmann brain could have identical instantaneous brain states while being reached through radically different physical histories. The ordinary history looks schematically like

early universe → stars → Earth → evolution → observations → ordinary Vantage

and the Boltzmann history like

equilibrium → rare fluctuation → BB Vantage.

At the tips, the information states may coincide. The physical weights of the histories are fixed elsewhere.

Causal history therefore acquires a role without any special epistemic category of “genuine” memories. Ordinary observers get no bonus because their memories deserve to count, and Boltzmann brains are not ruled inadmissible because their memories are fake. Causal history matters because physical dynamics distinguishes histories:

causal history → physical dynamics → physical weighting → self-location.

There is no teleological step.

This also dissolves an awkward boundary problem. Start with a brain fluctuating into existence, then enlarge the fluctuation: brain, brain plus room, city, Earth. Suppose the fluctuation eventually creates an Earth one billion years ago in just the right state, after which ordinary physical processes produce evolution, civilization, and us. Are we still Boltzmann brains?

The label has stopped doing work. Our recent memories would have been produced by the events they represent, and we would be ordinary observers descended from an extraordinarily improbable earlier fluctuation. MCSL needs no semantic boundary between “real observer” and “Boltzmann brain.” These are different physical histories with different dynamical weights, and physics does the bookkeeping even where we cannot yet complete the calculation.

Simulations are physical processes

The simulation argument raises the same issue, with one complication: simulated observers are not causally identical to biological ones.

Suppose a post-human civilization constructs a computer and runs 10³⁰ perfect simulations of Earth from the Big Bang onward, each eventually containing an informational duplicate of you. From inside a simulation, the represented history may be indistinguishable from ours:

Big Bang → stars → Earth → evolution → you.

Its realizing physical history is different:

base universe → civilization → computer → computation → simulated Vantage.

The represented history and the realizing history must not be conflated. A simulation can represent a universe containing enormous spacetime volume, entropy, and quantum amplitudes without creating those quantities as physical Measure in the base universe. Running a numerical model of a 99/1 quantum experiment does not manufacture additional Born Measure in the ratio 99/1 merely because those numbers occur inside the computation. Computational processing generates no fundamental Measure.

The hard question survives. If one substantial-Measure physical history contains 10³⁰ distinct conscious computations and one biological observer with a matching information state, how should those Vantages enter self-location? Conditional on that history, centering uniformity answers 10³⁰ to one, and nothing established so far blocks the answer. Physics gives us the history and its Measure. Centering gives us multiple locations within it. The rule connecting the two still requires justification.

The copy machine

A simpler thought experiment isolates the issue.

At noon, Alice is biological and unique, and she knows this with certainty. At 1 p.m., a machine creates 10³⁰ exact simulations of Alice’s noon information state. Afterward there are overwhelmingly more simulated Alice-states than biological ones. Does that make the proposition

The Alice who existed at noon was simulated

overwhelmingly probable?

No. The later copies cannot change the physical history that produced the noon Vantage.

Now move the copying event backwards, so that the simulations were created yesterday instead. Pure copy counting now wants their multiplicity to bear on Alice’s present self-location. Why should crossing Alice’s temporal location transform copy number into evidence?

There may well be a legitimate self-location problem after duplication, and MCSL does not deny it. The thought experiment establishes one thing: copy creation and probability creation are not the same operation. The centered probability problem appears only when we additionally ask which of the coexisting Vantages is ours, and that question needs its own account.

The structural similarity between the two arguments is now visible. The Boltzmann-brain argument runs from many matching fluctuations to a high probability that I am one of them. The simulation argument runs from many matching simulations to a high probability that I am one of them. In both cases the premise concerns multiplicity and the conclusion concerns centered probability. If the bridge is

then observer counting has been assumed rather than derived. Perhaps a correct theory will make multiplicity relevant. Perhaps it will reproduce SIA-like weighting under some conditions and fail to under others. The result should be derived from the relation between physical Measure and centered uncertainty, not stipulated by counting observer tokens.

FNC moves the multiplicity

Full Non-indexical Conditioning avoids explicitly sampling an observer. It conditions instead on the existence somewhere in the universe of someone possessing your complete memories.

That eliminates the reference-class problem while making multiplicity relevant in another way, since larger universes provide more opportunities for some observer to instantiate the specified information state. The conditioning proposition becomes approximately

Someone somewhere has these memories,

which is not obviously identical to the evidence available from this Vantage. FNC relocates the underlying issue rather than removing it. Matching information states elsewhere become relevant because the evidential proposition has been transformed into a global existential claim. MCSL asks instead how physical Measure and centered uncertainty combine at the Vantage.

What centering uniformity gets right

Centered uncertainty is real. After an Everettian split, before an observer registers the result, there is a genuine self-locating question. Sleeping Beauty likewise wakes knowing the experimental protocol while lacking information about her temporal location. Taylor’s framework treats centered worlds as an uncentered physical history plus a “here and now” tag and derives constraints on probability policies over those centers, which addresses a real problem.

Taylor’s theorem does more than license uniformity. It supplies a combination rule: weight uncentered worlds by a measure, weight each world by how often your information state occurs in it, then spread uniformly across the matching centers. What it leaves out is the measure. Taylor is explicit that the uncentered measure need not be read as a prior over universes, and that it parameterizes the Dutch-book-resistant policies in the way a subjective prior parameterizes Bayesian updating.

MCSL applies at that gap. A decision-theoretic argument fixes the shape of the combination and leaves its physical content open, and a free parameter invites cardinality to fill it. Where two centers are genuinely symmetric under all relevant physical structure, uniformity is appropriate, and what informational symmetry cannot do is establish that physical symmetry. A physical theory gives us weighted alternative histories, and centering refines those histories into possible locations within them.

Centering symmetry should operate within, rather than replace, the objective weighting supplied by physics.

Taylor also sets quantum mechanics aside by hand. An uncentered world in her setup is an objective state-trajectory of the material universe, with the quantum complications declared out of scope. A branching ontology bites at exactly that exclusion.

Return to the 99/1 experiment. Two post-decoherence centers hold matching information states, and the framework offers two readings of which uncentered world each of them decenters to. Neither reading produces Born weighting on its own.

Read the entire wavefunction as the single uncentered world, and both centers decenter to the same W. Uniformity applies directly and assigns them equal probability. The Born weights appear nowhere, because nothing in the representation records which branch a center inhabits.

Read each decoherent branch as its own state-trajectory, and the Born weights land in the uncentered measure, where the theorem leaves them free. Dutch-book resistance is satisfied by any measure at all, including one that gives the 1% branch more weight than the 99% branch.

MCSL’s demand becomes specific here. A center in a branching universe carries branch-relative structure that the pair (W, C) discards, so a quantum generalization of centered worlds has to retain it, in something closer to (W, B, C) where B is the branch or event whose Measure the center inherits, together with an argument that the measure over branches is Born rather than arbitrary. Whether that generalization goes through is open. What the classical theorem settles is that centering uniformity is complete for uncentered worlds in Taylor’s sense and silent about the structure a measure-bearing branching ontology adds below them.

The open work has moved. What needs deriving is a center that carries its Measure with it, and an identification of the uncentered measure with a weighting the physics supplies.

The remaining debt

This essay does not solve the Boltzmann-brain problem. It identifies a mistake in one purported route to the problem: treating observer cardinality as though it were already a probability measure.

Nor does MCSL solve the cosmological Measure Problem. Born Measure gives us unusually firm footing for quantum alternatives, while eternal cosmology introduces unresolved questions about regularization, recurrence, vacuum dynamics, and observer realization.

There is also an unresolved problem closer to home. Suppose a single physical history of substantial Measure contains 10¹⁰⁰ information-identical Vantages. Taylor’s framework already offers a candidate answer, since the count of matching information states enters as a multiplicative weight on the world, which is SSSA together with SIA up to a population scaling factor. That answer carries a decision-theoretic argument behind it, so MCSL has to meet it on those terms. MCSL objects to cardinality standing in for the measure over worlds, and a count that multiplies such a measure is a different object, one that survives everything argued above.

Which of the two the physics licenses is open. Taylor’s result is itself agnostic between SSSA and SSSA plus SIA, since the difference is a scaling factor that the free measure absorbs. Treating 10¹⁰⁰ coexisting Vantages as 10¹⁰⁰ independent units of physical Measure is still wrong, since they coexist within one physical alternative. Whether they should enter as a population factor on that alternative is the question the free parameter leaves standing.

The problem in its cleanest form:

How should objective physical Measure over histories induce probability over centered refinements of those histories?

For a branching universe the question has a sharper form. What should a center be, when the physics assigns Measure to structure finer than an uncentered world?

Centered-world theory makes those refinements explicit. Taylor’s Dutch-book argument supplies a combination rule for their probabilities, with the uncentered measure left free. Physics supplies quantitative structure over the underlying histories. A satisfactory anthropic theory must preserve all three.

Postscript

Across the three essays the failure keeps its shape. Anthropic reasoning asks which observer you are, and counting observers produces reference-class and population paradoxes. Everettian quantum mechanics asks which branch-relative situation you occupy, and counting branches discards amplitude structure. Boltzmann brains and simulation arguments ask which matching realization you occupy, and counting copies substitutes cardinality for physical structure.

The correction is not that copies never matter. It is that copy count is not itself a measure theory.

A Boltzmann brain whose state matches yours is real if physics realizes it. A simulated observer running in some base universe is real if the computation runs. Neither acquires a claim on your self-location by existing. Their claim on it would have to come through a rule connecting physical weighting to centered probability, and for a universe whose branches carry Measure, nobody has written that rule down.

Max Weinreich’s The Crisis of AI-Generated Mathematics identifies a real problem. Research mathematics has used the production of mathematical work as evidence that the producer possesses mathematical competence and understanding. AI breaks that connection.

A paper has traditionally done several jobs at once. It contributes a result, communicates an argument, establishes priority, and certifies something about its author. On Weinreich’s account, publishing shows that a mathematician practiced the deepest form of the art and arrived at “a complete understanding of a piece of mathematics.” AI-assisted production weakens the inference, because a mathematician can now produce sophisticated work without having discovered it and possibly without understanding it.

Weinreich is right about the decoupling. His mistake is treating the collapse of the proxy as a crisis for mathematics itself. It is a crisis for mathematical credentialing.

The bundled proof

Before powerful automation, producing a difficult artifact usually required possessing the skill associated with it. A sophisticated program demonstrated programming competence. A proof demonstrated mathematical ability. Useful output and difficult human achievement shared a production bottleneck, so institutions could treat one as evidence of the other.

As I argued in The Human Premium, these are different forms of value. A proof has artifact value because it establishes a result. Producing it may have achievement value because doing so displays skill or creativity. It may also have credential value because its production provides evidence about the mathematician. That essay separates two further kinds, relational and status value, which carry most of the argument in art and little of it here. Human scarcity kept the three correlated. AI makes them independently variable.

Weinreich responds by treating human production as partly constitutive of the value of mathematics. Solving more important problems, he argues, is not really the goal of the field; the practice of doing mathematics is an end in itself. That identifies a legitimate good, though not the only one.

Doing mathematics can be intellectually rewarding, and discovering a proof can be an extraordinary achievement. Struggling with a problem produces forms of understanding that reading the finished result does not. None of this fixes the epistemic value of the result.

A difficult human proof can be an impressive achievement and establish something unimportant. A machine-generated counterexample can be mathematically important and represent no human achievement at all. The two dimensions were never identical. A shared bottleneck kept them in step.

Knowledge and understanding

Suppose an AI produces a valid counterexample to a longstanding conjecture. Mathematical knowledge has grown, since a question that was open is now closed. Weinreich can reply that this does not by itself increase human mathematical understanding. He would be right, and he would be making a different claim. Mathematical knowledge and human mathematical understanding are both valuable, and neither reduces to the other.

His Jacobian Conjecture example makes the distinction concrete. An AI generates a counterexample, Terence Tao then uses AI to work out what it says, and the future on offer has machines producing the mathematics, machines explaining it, and humans surviving as intermediaries.

The danger is real. Provenance does not establish it.

A mathematician might ask an AI for a proof, paste it into a paper, and understand almost none of it. Another might use AI to explore the proof, test alternative explanations, reconstruct crucial steps, identify the mechanism behind the result, and then extend it. Both are “AI-assisted.” Their epistemic states have almost nothing in common.

AI does differ from earlier mathematical tools in that it can substitute for cognitive work rather than facilitate it. That strengthens the case for assessing understanding directly. The same system supports understanding in one use and replaces it in another, and the label records neither.

A theorem does not inherit its truth from the biography of whoever found it. Human understanding remains valuable and has to be evaluated on its own.

The certification problem

Weinreich’s institutional diagnosis is the durable part of the paper, and the collapse of proof-of-work credentials it describes reaches well beyond mathematics. If a paper no longer demonstrates that its nominal author can independently produce or understand its mathematics, publication cannot serve as the credential it once did. Hiring committees, grant bodies, and graduate admissions have all inferred qualities of mathematicians from finished work, and AI weakens every one of those inferences.

His remedy is to protect the old correlation. Departments might reserve positions for mathematicians who avoid AI, or weight AI-free papers more heavily in promotion. That preserves the proxy and leaves the property the proxy was meant to track unmeasured.

If a department cares whether someone understands a proof, test understanding. If it cares about independent reasoning, test independent reasoning. Judgment, originality, explanatory ability, problem selection, and the capacity to recognize a plausible failure can each be assessed more directly, through oral defense, adversarial questioning, reconstruction of critical steps, extension of a result under altered assumptions, or disclosure of what the machine contributed.

The cost is real. The bundled paper was an imperfect proxy, but a cheap and scalable one, and direct assessment consumes expert attention. Academia now faces a tradeoff between low-cost noisy signals and expensive high-fidelity evaluation, and the likely response is layered: routine screening for low-stakes decisions, spot checks, intensive assessment where the stakes justify it. High-fidelity assessment will also be distributed unevenly, since institutions with more expert time and money will do it better. That is a cost of unbundling, not a reason to keep a proxy whose reliability has collapsed.

Measure the property you care about, not the historical bottleneck that used to correlate with it.

The competence problem

A stronger argument for preserving unaided mathematics starts from how judgment is acquired. Mathematicians learn to recognize a fragile proof partly by building and breaking proofs themselves. The expertise required to supervise a process is often created by performing it.

Consider an ecosystem in which machines grow steadily more capable while successive generations of humans do less of the formative work through which mathematical judgment develops. Automated production improves. The surrounding human community loses the independent competence needed to evaluate it. That failure mode deserves to be taken seriously.

What it supports is protected formative practice rather than a prohibition on machine-generated mathematics. Students may need to prove theorems without AI even when AI could solve them instantly. Researchers may need occasional controlled unaided exercises. Institutions may need to maintain redundant human expertise where automation performs the immediate task more cheaply.

The incentive problem is harder than the epistemic one. Redundant competence looks most wasteful exactly when machines appear most reliable, so institutions will underinvest in it until rebuilding the capability is difficult. Preserving human expertise has the logic of resilience: its value is easiest to ignore while nothing has gone wrong. Recognizing the need does not summon the incentives, and the cost-minimizing equilibrium under existing incentives may contain less human competence than the epistemically resilient one.

Human achievement value also has a recruitment function. People give years to mathematics partly because discovery is difficult, uncertain, and personally meaningful, and if machines routinely reach the frontier first, some of that motivation goes with it. Withholding tools does not recreate the old frontier, since a protected sandbox is not the edge of human knowledge. Mathematical culture may still need to leave people room to formulate problems, pursue conjectures, and find things out for themselves if it is to keep producing mathematicians.

The criterion is functional. Preserve unaided practice where the practice is causally necessary to produce capabilities and incentives we still need. Do not preserve scarcity because scarcity once generated prestige.

Unbundling does not mean an assembly line

Separating roles does not require assigning them permanently to different agents. A human mathematician may still discover, prove, explain, verify, and extend a result, with AI contributing at any of those stages or at none of them. What has disappeared is the assumption that one finished paper reveals who performed which intellectual work.

Weinreich nearly reaches the same conclusion

His most interesting proposal points toward the alternative. Traditional authorship, he suggests, might give way to something like mathematical co-ownership: a mathematician who demonstrates authoritative understanding of an existing work acquires standing in relation to it without having originally produced it, and journals validate that understanding.

The proposal concedes the central institutional point. Authorship and understanding have come apart, and once understanding is certified separately from production, prohibition loses its credentialing rationale. His motivational case survives the concession, that AI cuts mathematicians out of their deepest experiences, and it has to be argued on its own terms. Machines can generate results. Humans can understand them, explain them, criticize them, recognize their significance, and extend them.

Who found the proof? Who understands it? Who verified it? Who can generalize it? Who is answerable for the claim that it is correct? Under overwhelmingly human production, one author list answered all of those questions at once. It no longer does.

A separate argument

Weinreich extends his case to AI companies, autonomous agents, infrastructure vulnerability, and catastrophic risk. Those claims need their own support.

One connection holds. If humans lose independent mathematical competence while advanced systems require human mathematical oversight, deskilling becomes an AI-risk factor in its own right. That reinforces the case for preserving competence, and it carries none of the broader conclusions. If advanced AI poses an unacceptable systemic danger, that is a reason to constrain advanced AI whether or not it proves theorems. Problems in mathematical credentialing cannot supply the argument.

Postscript

A mathematical paper once bundled truth, discovery, understanding, achievement, authorship, and prestige, because producing advanced mathematics almost always required advanced mathematical ability. AI makes those variables independently adjustable.

Restoring the correlation by making mathematical production artificially difficult is the wrong repair. Judgment, understanding, explanation, criticism, problem selection, and responsibility for what gets asserted still have to come from somewhere, and each of them can be measured directly. Where those capacities require unaided practice to develop, protect the practice deliberately. Where machines produce genuine mathematical knowledge more efficiently, the knowledge is not diminished by the ease with which humans came to hold it.

Weinreich is right that something is ending. A paper can no longer be assumed to certify the intellectual history that produced it. Mathematics does not depend on that assumption.

AI has not made mathematics artificial. It has made our proxies visible.

Brett Hall makes a strong claim about human reason: the world is comprehensible, and comprehensible to us. Discussing David Deutsch’s The Fabric of Reality, he argues that because human beings are universal explainers, we should expect never to meet anything permanently beyond our understanding. The claim is not confined to the podcast. Writing on artificial intelligence, Hall glosses Deutsch’s criterion for personhood as our being universal explainers, and describes the algorithm an AGI would require as one capable of generating explanations for any problem whatever.

Take the position in its strongest form. Hall is not saying that people have infinite memory, unlimited time or unbounded computational resources; he would presumably grant that every actual person and every actual civilization is finite. He is saying something more interesting: that human explanatory cognition is architecturally universal, and that no domain of objective explanatory knowledge lies intrinsically beyond a human-type mind.

Human cognition is remarkable enough to make this tempting. Our brains evolved under pressure from food, predators, mates, coalitions, tools and terrain, and we can nonetheless work in quantum mechanics, general relativity, natural selection, computability and mathematical structures with no analogue whatever in the environment that shaped us. We are plainly not confined to a catalogue of evolutionarily familiar problems. Extraordinary generality, though, is a weaker property than universality, and Hall needs the second.

Granting the easy cases

Several familiar objections miss. Suppose we understand the generating rule of a cellular automaton but cannot determine a distant state without running the computation. That still counts as understanding the system, and understanding why no shortcut exists is itself part of the explanation. Understanding quantum mechanics does not require predicting the outcome of a particular measurement. Understanding an algorithm does not require solving the halting problem for every input. Computational irreducibility, uncomputability and finite resources therefore establish nothing about cognitive closure.

What they do show is that “understanding everything” runs together questions that come apart. Whether we can form the relevant concepts, whether we can construct an explanation, whether we can obtain the evidence to test it, whether we can derive its consequences, whether we can predict particular outcomes, and whether we can know every truth about a system are six different questions with six different answers. Hall’s claim survives only if restricted to the first two. Restricted that way, it still asserts that human cognitive architecture spans every explanatory domain, and that assertion is unargued.

Universality is always relative to a class

A physically instantiated computer can have a computationally universal architecture while having finite memory, finite processing capacity, finite energy and a finite lifetime. Universality never meant that the machine can execute every conceivable computation whatever the resources. It means the architecture is general and not specialized to one predetermined family of tasks. A universal computer cannot decide the halting problem in general, cannot compute non-computable functions, and cannot run a computation whose resource demands exceed what physics supplies. Its universality holds relative to a formally characterized class.

That is where the analogy stops carrying Hall’s weight. For computers we can specify the class of computations, construct a machine, and prove that it simulates any machine in that class given the program and the resources. The demand transferred to explanation need not be for that much formal apparatus. Physics makes substantive universality claims without a complete enumeration of the target class, and the Church-Turing thesis is itself an informal conjecture and not a theorem. What cannot be dispensed with is a characterization that declines to help itself to the conclusion. What makes something an explanation of a given kind, in terms that do not reduce to the explanations we have so far produced? What property of human cognition shows that it reaches every member of the class so characterized? What separates universal explanatory reach from reach that is indefinitely extensible and still bounded? Hall supplies none of the three.

My laptop is computationally universal. It is not a universal physicist.

What the jump is a jump to

Deutsch’s “jump to universality” is the obvious reply. Some systems do not merely get gradually better; they cross a threshold, as when a specialized calculator becomes a universal computer, or a limited notation becomes capable of representing arbitrarily many constructions. Deutsch argues that human explanatory cognition made an analogous transition.

The phrase leaves the burden where it was. A universal computer is universal over the computable functions; a universal notation is universal over the expressions its syntax admits. In each case the domain is specified well enough that the claim has content, and the specification came first. Explanation has had no such specification, so calling the transition a jump to explanatory universality only renames the thing in dispute. Where the class is uncharacterized, the jump is a metaphor.

Conjecture and criticism is not a search algorithm

A further Deutschian reply treats explanation as a species of computation. Knowledge creation proceeds by conjecture and criticism; if the brain is computationally universal, and conjecture and criticism is the general mechanism by which explanatory knowledge gets created, then a computationally universal mind is a universal explainer.

That argument needs a premise it does not have. Criticism exposes errors in existing conjectures and rejects contradictions, empirical failures and explanatory defects. It describes a mode of error correction, and nothing in it maps arbitrary problems onto their explanations. Hall himself endorses Deutsch’s argument that future knowledge cannot be predicted, since predicting it would amount to already possessing it. That establishes unpredictability and leaves the deeper question open, since a deterministic procedure can produce states that no cheaper route reaches, which is what computational irreducibility describes. The gap holds on either answer. A universal computer runs any algorithm in the computable class, and that bears on explanation only if some computable procedure spans the space of possible explanations. No such procedure has been exhibited.

The comprehensibility leap

Three claims are in play. Reality is intelligible, containing objective regularities susceptible to explanation. Human explanatory ability is open-ended, with no known finite catalogue of domains outside which reason stops working. And no explanatory domain is in principle cognitively closed to people. The first two do not entail the third, and “universal explainer” is the phrase that carries the argument across the gap.

Read the phrase one way and Hall’s conclusion fails. If a universal explainer is a domain-general, open-ended system capable of constructing new explanatory representations, it can have indefinitely extensible reach without being guaranteed access to every possible explanatory domain. Read it the other way and the conclusion was never in question: if a universal explainer is a system capable in principle of understanding every objectively comprehensible domain, the premise already contains what the argument was supposed to deliver. Either the premise is too weak for the conclusion, or it is the conclusion.

Deutsch blocks the second reading himself. In the discussion of Tipler’s hypothetical Omega Point, he grants that even enormously powerful future intelligences would fall short of all possible knowledge, with the overwhelming majority of abstract truths staying out of reach. Whatever a universal explainer is, then, it is not an entity that can know every truth. Hall’s available reply is that his claim concerns explanatory domains and not truths: no domain of objective explanatory knowledge is cognitively closed to people. That version is more defensible, and it puts the circularity in plain view. People can understand every explanatory domain because they are universal explainers. They are universal explainers because no explanatory domain is inaccessible to them. The label does no work until explanatory universality has been established some other way.

Closure would not announce itself

Other animals have architecture-dependent limits. A dog will not be brought to differential topology by extending its working memory or giving it another decade; the machinery for constructing the relevant concepts is absent, and no amount of patience supplies it. Hall’s claim is that we differ in kind, that human explanatory architecture is universal and not simply roomier than a dog’s.

Notice what the dog cannot do besides topology. It cannot represent the boundary it is running into. Differential topology appears on no list of open dog problems, because nothing in the dog’s cognitive economy so much as points at it. A domain closed to us would be invisible in the same way. It would not sit in our catalogue of hard unsolved problems. It would sit nowhere at all, leaving no trace in the record of what we have tried and failed to explain. Our history of solved problems is evidence about the domains we can represent well enough to fail at, and evidence about nothing else.

Hall’s anti-inductivism turns against him

Hall’s support for the stronger claim is historical. We have repeatedly understood what once looked beyond us, and he takes this as evidence that reality really is comprehensible to us. It supports optimism. As an argument for universality it has the form: humans have repeatedly solved problems previously regarded as inaccessible, therefore no explanatory problem can ever be inaccessible to humans. The record gives us excellent reason to refuse any declaration that a problem is permanently insoluble merely because we cannot currently solve it, and excellent reason to keep conjecturing. It does not reach the universal quantifier, and by the sampling point above it is drawn from a set that excludes the cases at issue.

Hall is otherwise hostile to induction, and he correctly rejects the idea that a finite run of observations establishes an unrestricted law. The same structure reappears here in optimistic dress, with past explanatory success taken as evidence of unlimited future explanatory reach. Inductive pessimism goes out and inductive optimism comes in.

Extending the explainer

Every reply so far leans on one premise: that a human mind together with external machinery can be extended indefinitely while remaining a human explainer. Pen and paper extend the mathematician. Notation, instruments, institutions and computers extend the physicist. Treating the extended system as the relevant unit is what allows Deutsch to concede our unaided limits while giving up nothing.

The premise needs a boundary condition it does not have. Give a dog a proof assistant and the assistant does topology while the dog watches. Nobody counts that as extending the dog, so extension cannot mean bolting capacity onto a system. Some criterion separates the tools that extend an explainer from the tools that replace one, and Deutsch needs that criterion to fall in a particular place.

One candidate says that extension counts when the agent retains comprehension of what the equipment produces. That assumes the reach at issue. Whether we still comprehend the outputs while leaning on the apparatus that generates them is the question, and it cannot double as the test for answering it.

The better candidate is bootstrapping. A dog cannot build the notation, cannot be taught to use it, and cannot join the criticism that improves it, while humans do all three, and that capacity is what a jump to universality is meant to name. The asymmetry is real and deserves credit. It has also only ever been observed operating inside the range of concepts we have so far managed to construct. An architecture that bootstraps freely until it meets a structure it cannot get a foothold on would leave exactly the record we have, and the stall itself would go unrecorded.

Push the scaffolding harder. Suppose comprehending some domain requires a million times the memory, the processing and the elapsed time. On the Deutschian accounting that is a resource fact, absorbed by the same clause that absorbs Aristotle’s distance from quantum field theory. Yet at that scale no individual ever holds the explanation. It exists in an institution or a machine, and each participant’s relation to it is testimony about what the apparatus reported. Somewhere between pen and paper and a million lifetimes of derivation, machinery stops extending an explainer and becomes an oracle the explainer consults. Deutsch has no principled place to draw that line, and “comprehensible in principle” conceals that there is a line to draw. An explanation nobody can survey is not obviously comprehended by anyone, which locates an equivocation in the very word the dispute turns on.

The difficulty can be put as a test case. Imagine explainers with a billion-year head start discussing domains we have never approached. Either they can always bring us to comprehension, given the notation, the machinery and the centuries, in which case something must guarantee it, or they cannot, in which case closure is real. Hall needs the first horn. The theory of computation does not supply it, because what would need guaranteeing concerns the closure properties of human representational bootstrapping and not the class of computable functions.

What would settle it

Four things would have to be in place. A non-circular characterization of the explanatory class, in terms that do not reduce to the explanations we happen to have. A relation between explanatory architectures answering to simulation between machines, specifying what it is for one architecture to reconstruct another’s explanation instead of merely reproducing its predictions. A demonstration that human architecture reaches every member of the class, or a physical argument that the primitives required for any physically explicable structure are finitely generable from the ones we have. And an evidential rule capable of telling an unsolved problem from an unrepresentable one.

The first is prior to the others, which cannot be stated without it. The fourth is the one Hall most needs, because his evidence is one-sided by construction. A solved problem counts for universality. An unsolved problem counts for nothing against it, since any unsolved problem can be classified as temporarily unsolved. Some domain we have not explained admits two diagnoses. On one, we lack the theory. On the other, we lack the concepts in which the theory could be written. Nothing in the historical record separates them, and a closed domain emits no signal of its own.

Deutsch would answer that the thesis concerns the capabilities of people, and that capability claims are not refuted by particular failures. Granted. A capability claim still needs some rule connecting it to evidence, or it fits every possible history of inquiry, and fits because failure was defined out of the evidence before the counting started.

What survives

None of this licenses cognitive pessimism. Present inability is weak evidence of impossibility, and when someone announces that humans will never understand quantum gravity, consciousness, the origin of life or some mathematical structure nobody has yet invented, the burden lies with the claimed limit. Our history is strong reason to distrust confident declarations that a problem lies forever outside comprehension, and treating unsolved problems as soluble and continuing to work on them remains the productive stance.

Methodological optimism is not a theorem about the architecture of cognition. The absence of a demonstrated ceiling is not the demonstration of no ceiling. What we have are finite physical minds with unusually general and extensible explanatory architectures. That may be enough for endless intellectual progress.

Robin Hanson argues in No-New-Physics Consciousness that consciousness probably requires no revision to known physics. On that point I agree. Brains are physical systems, and if consciousness depends on brains, a sufficiently complete description of the relevant physical organization should account for the structure of experience. Hanson embraces that reductionist picture.

Overcoming Bias

No-New-Physics Consciousness

There is a vast literature near the topic of this post, much of which has developed specialized vocabularies and concepts. And I can’t be bothered to master enough of it to make this post match it we…

Read more

a month ago · 28 likes · 23 comments · Robin Hanson

Then he makes an exception.

After the physical and functional organization has been completely specified, Hanson allows that one further fact might remain: whether the system feels anything. A system could have the same brain states, the same memories, the same reasoning and the same behavior as a conscious system and still be a philosophical zombie. On what he calls the “full-counterfactual” view, every conscious system has a possible zombie counterpart differing only in whether its experiences are felt.

Describing that possibility does not establish it. Hanson denies consciousness any new physics and then hands it a new metaphysical bit.

No extra bit

Imagine two systems, A and B. Their physical organization is identical, down to the states of their neurons or computational elements and the causal relations among them. Both attend to the same things, form the same memories, register the same apparent pains and colors, and produce the same reports about all of it.

Hanson wants to leave open one remaining difference:

A actually feels. B does not.

Call that residual property actuallyFeels.

By hypothesis it changes nothing physical or functional. It does not alter perception, memory, inference, speech or action, since any such alteration would break the stipulated physical identity of A and B.

The intuition behind the zombie starts from a real separation between our third-person description of cognitive machinery and our first-person description of experience. That the two descriptions come apart is a fact about descriptions. Independent variation in what they describe is a further claim, and conceivability alone does not supply it. Once the organization responsible for perception, attention, introspection, memory and report has been fixed, the zombie advocate owes an account of what could still be unfixed.

The burden falls hardest on a reductionist. On Hanson’s own commitments, experience is realized by physical organization, and fixing the physical facts fixes the experiential facts. An independent, causally inert phenomenal property added after the physical description is complete is epiphenomenal property dualism, even with no new forces or particles anywhere in the story.

The Bayesian trap

Hanson draws the consequence himself. If a conscious person and an exact zombie are physically identical, no observation can distinguish them. Every observation is a physical state, every memory of that observation another physical state, every inference from it physically implemented. The declaration “I know I am conscious” comes out of machinery the zombie has too.

So evidence cannot tell us whether any particular experience system feels, and our confidence that we ourselves are conscious has to come from our priors.

The conditional argument is sound. Hanson stipulated a property that varies without touching any observation, memory, inference or report, then found that evidence cannot track it. The isolation was built in at the start, and no discovery about consciousness came out of it.

Empirical invisibility alone does not prove a property unreal. But actuallyFeels is causally inert, explanatorily idle, and introduced for no purpose except to preserve the intuition that physical and functional duplication might leave consciousness behind. That is a poor reason to enlarge the ontology. The bit stays out until an independent argument puts it in.

The zombie gets expensive

The intuition also depends on keeping the zombie thin.

In Why Zombies Don’t Evolve, I distinguished a thin zombie from a thick zombie. A thin zombie inherits every human capacity by stipulation. It sees, learns, avoids injury, allocates attention, introspects, reports its mental states and writes philosophy papers about phenomenal consciousness. We declare that nothing is felt inside and move on.

A thick zombie has to implement the machinery that produces those capacities. Give it embodied perception, scarce attention, salience, memory integration, self-monitoring, perceptual stabilization, conflict resolution, interruption handling and internal comparison of expected against actual states. Give it whatever architecture maintains a coherent world-model while controlling action through time. At some point the entire causal architecture that a theory of consciousness was supposed to explain has been rebuilt, with consciousness still withheld by stipulation.

A committed property dualist can accept that result. Functional architecture explains cognition and report, they will say, and phenomenal consciousness is something additional. The concession is expensive. Every piece of explanatory work now sits in the physical architecture, and the phenomenal property runs alongside it as an idle parallel layer.

Evolution makes that parallelism harder to explain. Natural selection can select pain avoidance because avoiding injury affects survival, and attention because finite organisms cannot process every signal reaching them. Memory integration, self-monitoring and perceptual stabilization are selectable for the same kind of reason: they change what the organism does. A phenomenal property with no causal effects is not available for selection at all.

A defender can reply that phenomenal states accompany the selected physical states as a matter of psychophysical law. That reply leaves a coincidence in place. The physical system does not merely avoid injury. It complains that pain feels bad, remembers that pain felt bad, compares pains, reasons about unpleasantness and builds theories of phenomenal experience, while by hypothesis no phenomenal fact contributes causally to any of it. The machinery produces a detailed literature about a domain that never touched the machinery producing the literature. A functional account has no such coincidence on its hands, because the architecture generating the reports and the architecture constituting experience are the same causal system.

Conceding the Hard Problem

None of this makes the Hard Problem disappear by assertion. Why a particular physical organization has the qualitative character it does, and why any physical process should constitute experience at all, remain open questions, and legitimate ones.

Neither question establishes that a complete physical duplicate could lack experience. Our present inability to derive a first-person description from a third-person one is a fact about the state of our theories. The zombie argument needs the stronger claim that the two descriptions answer to independently variable facts. Without it, the inference from “we do not yet understand why this organization feels like anything” to “there could be an identical organization that feels like nothing” does not follow.

Postscript

Hanson asks what rule determines which physically specified experience systems actually feel. Fundamental laws tend to be simple, so he expects the psychophysical assignment rule to be simple as well, which pushes him toward a view he describes as close to panpsychism.

The demand for an assignment rule comes entirely from the zombie premise. If consciousness is realized by physical organization, the universe needs no second law deciding which appropriately organized systems receive it. A hurricane requires no additional law assigning hurricanehood to particular arrangements of atmospheric matter; its higher-level identity depends on organization implemented by ordinary local physics. Nothing in known physics requires more of consciousness than that, and nothing but the zombie premise calls for a psychophysical mapping on top of it.

The scientific problem stays hard: identify the physical and computational organization that constitutes conscious experience, determine how it develops and operates, and test the resulting theory against biological and artificial systems.

Hanson’s argument establishes only that once a causally invisible phenomenal property has been stipulated, no physical evidence can locate it. Drop the stipulation and the assignment problem goes with it.