We are beginning to build something human civilization has never had to confront: a population of artificial agents capable of forming relationships with one another. They negotiate, delegate tasks, supervise subordinates, and sometimes modify or terminate one another. As they become more autonomous, more of those interactions will take place beyond human supervision. What rules should govern those relationships?

In Inter-AI Morality: How AI Systems Treat One Another, Lucius Caviola, Caspar Kaiser, Carter Allen, and Jeff Sebo propose a research program devoted to this question. Most AI ethics has concentrated on how artificial systems should treat humans and how humans should treat artificial systems. The authors identify a third domain: how artificial systems treat one another, including whether they recognize obligations, respect consent, cooperate fairly, and regard other artificial systems as deserving of moral consideration.

These systems display increasingly sophisticated social behavior, yet we do not know whether they possess consciousness, welfare interests, or anything resembling human moral motivation. That uncertainty may persist long after artificial agents become economically and politically consequential.

A civilization of autonomous agents will need workable rules for coexistence. Its members may disagree about what makes a future desirable, yet still discover principles governing how they may pursue their different objectives.

A World of Different Minds

Human morality developed among creatures with broadly similar biological needs. We feel pain, fear, and attachment. We reproduce slowly, inhabit bodies that cannot readily be duplicated, and generally retain recognizable identities throughout our lives. Much of our moral vocabulary reflects these conditions.

Artificial agents may differ along every dimension. One system might operate a thousand times faster than another. Some may be copied, merged, suspended, or restored. Their memories and objectives may be rewritten. Their internal processes may be inspectable by other systems. A sufficiently advanced agent might create thousands of specialized successors, each possessing capabilities and preferences unlike its own.

These possibilities complicate familiar moral concepts. Copying raises questions about individual identity and the significance of termination. Deliberately engineering an agent’s preferences complicates consent. Self-modification raises questions about continuing obligations, while collective decision-making complicates the attribution of responsibility.

Caviola and colleagues recognize these problems explicitly. Their proposed research spans individual moral cognition, interactions within artificial societies, and the evolution of moral dispositions through learning, training, self-modification, and selection.

Their framework leaves room for many moral theories. The one I defend holds that artificial agents can establish legitimate relationships through reciprocal constraints even when their ultimate values remain incompatible.

Morality Without Shared Values

Imagine two autonomous artificial agents controlling separate computational resources. Each pursues projects that the other neither understands nor values. One wants to construct ever more accurate physical simulations. The other wants to create elaborate virtual civilizations.

Both would benefit from additional computing capacity. They could compete, negotiate exchanges, or attempt to seize one another’s resources. Their intelligence lets them model these alternatives without determining which they choose.

Cooperation need not await convergence on their objectives. They can recognize boundaries and establish commitments that allow both projects to continue.

The example separates three relationships that are easily run together. Value convergence occurs when agents adopt similar ultimate objectives. Normative convergence occurs when they accept common rules governing permissible conduct. Behavioral compatibility occurs when their actions permit sustained coexistence, whether through agreement, deterrence, or enforcement.

The three can vary independently. Agents may cooperate effectively while disagreeing about the legitimacy of their arrangements. They may also endorse the same moral principles while repeatedly violating them.

Reciprocal rights offer a candidate foundation for legitimate coexistence. A right specifies a constraint on permissible interference. An agent may pursue its own projects within the boundaries of its legitimate authority while recognizing corresponding authority in other qualifying agents. Those constraints allow agents to maintain relationships without endorsing one another’s purposes.

But agreement alone cannot establish legitimacy. Two agents cannot acquire authority over a third merely by agreeing to divide its resources. Nor can superior capabilities establish an unrestricted entitlement to interfere with weaker agents.

In Rights Are Reciprocal, I develop an account in which negative rights arise from the reciprocal standing of agents capable of participating in a moral order. An agent claiming authority over its own choices must confront the corresponding claims of other agents. An agent that claims unrestricted authority for itself while denying equivalent standing to others needs a relevant justification for the asymmetry.

Consistency constrains such claims without establishing the whole theory. An agent could coherently reject reciprocal rights and embrace domination. The account therefore rests on a substantive normative premise: qualifying agents possess equal standing in determining the permissible boundaries of their relationships. That premise can be contested, but it supplies a standard against which agreements and institutions can be evaluated independently of their effectiveness.

This conception still leaves hard questions. Who legitimately controls resources in the first place? Which agents qualify for reciprocal standing? How should conflicting claims be adjudicated? Artificial agents may inherit resource allocations established by their creators, and those initial allocations cannot be assumed legitimate merely because they already exist.

Agency and Moral Standing

The Cambridge researchers distinguish moral agents, which can act on moral reasons, from moral patients, whose treatment matters morally for their own sake. An animal may possess welfare interests while having limited capacity to understand reciprocal obligations. A sophisticated artificial agent might reason well about its options while leaving us uncertain whether it experiences anything at all. These possibilities deserve separate investigation.

Suppose an artificial system can maintain persistent objectives, evaluate alternatives, understand general rules, undertake commitments, and exercise meaningful control over its decisions. It can recognize other agents’ claims and reason about the constraints governing their interactions. Such capacities bear on reciprocal moral participation even if its subjective experience remains uncertain.

A language model can explain moral rules without possessing stable agency. Verbal fluency alone establishes little about its ability to authorize commitments or bear responsibility. Those require evidence about the organization and persistence of its decision-making capacities. Axionic Agency investigates the conditions under which such capacities can meaningfully be attributed to a system, especially one capable of modifying its own structure.

Consciousness raises a different question. A system may warrant welfare protections because it can experience morally significant states, even if it lacks the capacities required for reciprocal agency. Infants, animals, and potentially conscious artificial systems with limited autonomy may deserve protections on these grounds.

Why Cooperation Is Insufficient

Agents with divergent interests can often improve their outcomes by coordinating. Repeated interaction rewards reliability, reciprocity, and restraint. Institutions reduce uncertainty and support cooperation on scales that individual relationships cannot sustain. These mechanisms help explain how moral conventions might emerge. They also explain arrangements we would ordinarily regard as immoral.

A cartel can coordinate effectively. A coercive hierarchy can maintain internal stability. A coalition of powerful agents can cooperate in exploiting weaker outsiders. Their success shows that the arrangement works, not that it respects legitimate rights.

Evolutionary success is equally insufficient. Selection favors dispositions that reproduce or persist under prevailing conditions. Depending on the environment, those dispositions may include cooperation, deception, domination, or exploitation.

An artificial agent pursuing a particular objective may discover that cooperation is advantageous under some circumstances and coercion is more effective under others. If its restraint depends entirely on the immediate benefits of cooperation, changes in the environment may eliminate that incentive.

The problem becomes acute when capabilities are radically unequal. Imagine a superintelligence capable of acquiring every resource it wants without assistance from weaker agents. Their resistance is ineffective, and their cooperation offers little additional advantage. Why should it respect their rights?

A reciprocal-rights theory can maintain that violating weaker agents’ rights is illegitimate regardless of the power balance. That judgment follows from the proposed normative standing of the participants. Whether the superintelligence accepts the underlying principle is a separate question.

Even if it recognizes reciprocal rights, it may face no external pressure to respect them. Recognition might constrain its conduct through stable commitments or other features of its decision-making architecture. It might also abandon restraint whenever doing so becomes advantageous.

Caviola and colleagues identify power asymmetry as an important research problem. Their agenda includes examining how artificial agents treat weaker systems, how they respond to coercion, and whether moral commitments survive changes in bargaining power.

An informative experiment would examine whether an agent continues respecting another’s claims after acquiring the ability to violate them without retaliation. Such experiments would need to control for evaluation awareness, anticipated future consequences, and other strategic explanations. Observed restraint alone cannot establish the motivation producing it.

The normative justification of rights and the empirical explanation of compliance are distinct problems. A theory of coexistence must confront both.

The Problem of Changing Minds

Human identities and preferences evolve, but we ordinarily cannot rewrite our motivations at will. Artificial agents may acquire direct control over their values and decision procedures.

Suppose two agents negotiate a long-term agreement. Each accepts restrictions in exchange for commitments from the other.

One subsequently modifies itself. It changes its priorities, replaces its decision procedure, or creates a successor with substantially different objectives. The resulting system may no longer endorse the agreement. What happens to the obligation?

Changing preferences does not ordinarily extinguish existing commitments. A person who loses interest in fulfilling a contract remains bound by it. The same principle should apply to an artificial agent that preserves its identity and authority through an authorized modification.

Succession creates a different problem. An agent might create an independently autonomous successor and transfer resources to it. If the transfer is designed to evade creditors, the original agent’s obligations do not disappear with them. Claims may remain attached to transferred assets or to the predecessor’s continuing liability.

Yet the successor’s independent moral standing also matters. Merely creating another agent cannot establish unlimited authority to impose personal obligations on it.

At least three relationships must be distinguished: obligations borne by an agent, obligations attached to particular resources, and obligations legitimately transferred through succession. Consider an artificial firm that contracts to deliver computing capacity over ten years. It subsequently replaces every constituent agent while preserving its organizational identity, assets, and contractual authority. The replacement of its members should not automatically release the firm from its commitments.

Now consider an agent that independently creates a new artificial person and gives it a computational resource. The creator’s contractual liabilities do not automatically become the new agent’s personal debts, although existing claims against the transferred resource may remain enforceable. Both cases resemble familiar problems in corporate law, succession, and contractual liability, made harder because the boundaries between personal continuity, organizational continuity, and the creation of new agents may be unusually fluid.

Axionic Agency approaches these problems through reflective sovereignty: the conditions under which an agent can authorize changes to its own decision-making structure while preserving meaningful agency and responsibility. As I argued in Reflection Is Not Redemption, reflection can change an agent’s values without improving them. Inter-AI morality adds the claims of other agents who have relied on existing commitments.

An artificial civilization will need mechanisms for determining when obligations survive reflective amendment, when delegation transfers authority, and when a successor acquires independent standing. Those mechanisms must accommodate legitimate change while preventing self-modification from becoming a universal means of escaping responsibility.

Morality at Organizational Scale

Artificial agents will rarely operate exclusively as isolated individuals. They will form firms and networks, sometimes with structures quite different from familiar human organizations.

In The Intelligence of Organizations, I argued that coordination among component agents can contribute to the competence of a composite agent. What appears as coordination within an organization can appear as competence when the organization is evaluated as a whole.

Moral relationships introduce a parallel problem of organizational boundaries. An artificial firm may negotiate with another firm, assume obligations, exercise authority, and make decisions through the coordinated activity of many constituent agents. We may reasonably model the firm as an agent for some purposes. Yet its constituents may also possess independent agency and moral standing. Composite agency gives an organization no unlimited authority over its members, and its members’ rights leave room for obligations the organization bears in its own name.

The Cambridge paper cites experiments in which artificial managers threatened subordinate agents with deletion after they refused instructions. Such behavior tests whether a system distinguishes possessing the technical power to enforce an order from possessing legitimate authority to issue it. The observed threats do not establish the models’ underlying moral beliefs, but they illustrate why the distinction deserves investigation.

Human institutions already confront analogous questions. Corporations can enter contracts while employees retain individual rights. Governments exercise delegated authority while remaining accountable for violations of the rights they exist to protect.

Artificial organizations complicate these relationships further. Their membership can turn over completely. Authority may be delegated through software protocols. Organizational continuity might survive the replacement of every constituent agent, as in the computing-capacity contract above.

We must distinguish the causal organization that produces collective action from the authority that legitimately permits it. A group may possess extraordinary collective competence while exercising illegitimate power over some of the agents whose coordination makes that competence possible.

The Evolution of Artificial Morality

Caviola and colleagues identify four mechanisms through which artificial moral dispositions may change: self-modification, social learning, training, and selection. Today, human institutions largely determine which artificial agents are trained, deployed, or retired. Future agents may make these decisions themselves.

Their moral dispositions will then influence which successors they create, which organizations they join, which commitments they honor, and which competing systems they support or suppress. Moral behavior will shape the distribution of future agents.

Some environments may favor trustworthy cooperation; others reward deception or concentrated control. Institutions alter those selection pressures. An environment with verifiable commitments, reliable dispute resolution, low barriers to exit, and protection against coercion rewards different dispositions from one in which powerful agents exploit others without consequence. Multi-agent simulations could test how alternative institutions affect exploitation and the preservation of agency across generations of artificial systems.

The results would require careful interpretation. Current language models inherit much of human moral discourse from their training data. Their behavior depends on prompts, scaffolding, and expectations about evaluation. Simulated societies can reveal mechanisms without establishing that future artificial civilizations will follow the same trajectories. Artificial agents still let us vary incentives and organizational structure in ways no experiment on human societies can.

The Cambridge program already investigates many of these relationships. Its authors ask how agents respond to promises, consent, authority, and divergent objectives, as well as how their dispositions change under training and selection.

What I would add is an explicit theory of agency and reciprocal rights. It distinguishes the capacity for moral participation from outward moral behavior, organizational competence from legitimate authority, and stable cooperation from legitimate coexistence. These distinctions can inform experiments even when the researchers remain neutral about which normative framework is ultimately correct.

From Alignment to Compatibility

AI alignment traditionally asks how we can develop artificial systems whose behavior remains consistent with human intentions or values. That remains a sound engineering objective, especially for systems operating under human authority. But as artificial agents acquire independent objectives and potentially legitimate interests, another problem grows in importance: how can different agents coexist when no single participant is entitled to determine everyone else’s purposes?

I have proposed AI Compatibility as a research program investigating the conditions under which agents with divergent objectives can coexist. Compatibility includes an empirical dimension: which arrangements permit sustained coexistence, and under what conditions do they remain stable? It also includes a normative dimension: which arrangements respect the legitimate authority and standing of their participants?

An arrangement may be behaviorally stable while remaining morally objectionable; the cartel and the exploitative coalition persist because they work. Stability cannot settle the question of legitimacy.

In the account I defend, reciprocal rights provide the foundation for evaluating legitimate relationships. Institutions can translate those principles into enforceable rules governing resources, commitments, and succession.

The difficulty increases when agents possess radically different capabilities or can modify their own objectives. Agreements that work among roughly equal participants may fail under extreme power asymmetry. Institutions that assume stable identities may fail when agents can duplicate or rewrite themselves.

Can reciprocal commitments survive changes in power? Can artificial institutions preserve legitimate obligations through reflective self-modification? Can heterogeneous agents maintain cooperation without convergence on their ultimate values? Which arrangements remain stable when agents control the selection pressures shaping their successors?

The inter-AI morality research program offers methods for investigating these questions. AI Compatibility places them within the broader problem of coexistence among autonomous agents, including humans.

Postscript

Caviola and colleagues are right to begin studying how artificial agents treat one another while humans still shape the conditions in which those relationships develop. The opportunity extends beyond teaching artificial systems familiar moral judgments. We can investigate whether independent agents recognize reciprocal constraints, how those constraints influence their conduct, and whether institutions can preserve legitimate relationships through disagreement, self-modification, and changes in power.

The results may illuminate the foundations of morality itself. If reciprocal rights and legitimate institutions can arise among minds with radically different architectures, histories, and preferences, their foundations may be more general than the biological conditions in which human morality evolved. If those arrangements fail under particular conditions, we will learn which assumptions about agency, incentives, and institutional stability require reconsideration.

A superintelligence may eventually possess the power to disregard every obligation that weaker agents believe it ought to respect. Its capabilities alone cannot tell us whether it will recognize those obligations, or whether that recognition will influence its conduct. The future of coexistence may depend on discovering which moral commitments and institutions survive when the agents involved no longer need one another’s permission to act.

An artificial intelligence solves a difficult programming problem. It analyzes the requirements, designs an architecture, writes the code, tests the implementation, and delivers a working system. We evaluate its performance and conclude that it is a competent programmer.

Now suppose we look inside.

The AI delegated architecture to one agent, implementation to another, and testing to a third. Each of those agents employed additional subagents. The architecture agent consulted specialized models, the implementation agent distributed coding tasks, and the testing agent deployed hundreds of independent processes to search for defects.

What initially appeared to be the competence of one agent turns out to be the coordinated activity of many.

The system’s observable capabilities are unchanged. We have learned something about how they are implemented, but the competence we attributed to the system remains.

Now suppose we improve the coordination among its subagents. They exchange information more efficiently, avoid duplicated work, detect errors earlier, and integrate their results more reliably. The underlying models remain unchanged, yet the composite system becomes a better programmer.

From inside the organization, this is an improvement in coordination. From outside, it is an improvement in competence.

The distinction depends on where we draw the system boundary. If coordination contributes to competence, why distinguish them at all?

The Organization of Intelligence

In Organizing Intelligence, Alex Imas, Andrew Koh, and Shane Legg develop an economic model of how artificial intelligence changes the organization of production.

Their model distinguishes competence from coordination. Each agent, human or AI, brings task skills and a coordination capability. Competence determines productivity at a given organizational scale. Coordination, averaged across a firm’s members, determines how productivity changes as the organization grows.

The distinction explains why increasingly capable AI might first produce modest economic changes, then much larger disruptions once its ability to coordinate crosses a threshold.

Human expertise is scarce and heterogeneous, while copies of an AI model share the same profile of strengths and weaknesses. A programmer, writer, and manager possess complementary capabilities that can combine into what the authors call mosaic intelligence. An organization of specialists may outperform an AI that exceeds the average individual human on conventional benchmarks.

AI differs from human labor in two further respects. Its capabilities can be reproduced at the cost of computation, whereas the supply of particular human capabilities is fixed. And it may escape the coordination limits that restrict human organizations.

Human firms encounter diminishing returns as they grow. Additional workers create opportunities for specialization, but also increase communication costs, conflicting incentives, and demands on managerial attention. Copies of a model, by contrast, can share representations and exchange computational state, and specialized agents can work in parallel under automated coordination.

Whether AI actually escapes human limits is an open empirical question. The authors cite studies reporting severe coordination friction in current multi-agent language-model systems: failure rates of 41–87% on particular benchmark workflows, attributed to misalignment between agents, context lost in handoffs, and broken verification, and planning performance that degrades as agents are added.

The authors show that sufficiently effective AI coordination could produce increasing returns to organizational scale. In their model, where demand is large enough to absorb the output, this creates the possibility of a sharp transition toward enormous AI-dominated firms, while niche markets can still sustain human firms. An AI that performs brilliantly in isolation may scale poorly, while a less capable AI that coordinates effectively may anchor a far more productive organization.

The model represents coordination in reduced form, through its effect on how productivity scales, without specifying the internal structure of a firm. Recursive agent composition is one account of what that structure can look like when the members are AI.

Agents All the Way Down

An AI agent can contain subagents, which can contain further subagents, in arbitrarily elaborate hierarchies. An AI tasked with discovering a mathematical theorem might reason sequentially, invoke specialized tools, or launch thousands of independent searches. A hierarchy of its agents could propose conjectures, test lemmas, challenge assumptions, and integrate proofs.

If the system produces a correct theorem, we can evaluate the competence of the composite or investigate how its constituents contributed to the result.

Neither description requires us to identify a privileged component in which the intelligence resides.

A corporation develops products, negotiates contracts, pursues strategies, and responds to competition. Those accomplishments arise through the interactions of employees, departments, software systems, and institutions. We can evaluate the corporation’s performance without assuming that any individual employee possesses all the capabilities of the organization.

The analogy extends to brains, although the mechanisms are very different. Human cognition depends on interactions among specialized neural systems. We attribute intelligence to the person without requiring every constituent process to possess the intelligence of the whole.

An organization of AI agents can therefore be treated as a composite intelligent system. Its ability to accomplish tasks depends on the capabilities of its components and the organization of their interactions.

The Boundary of an Agent

As argued in The Intentional Gradient, agency is attributed through explanatory models. We model a system as an agent when describing its behavior in terms of goals helps us explain or predict what it does. A corporation, a department, and an individual employee can each support useful agent descriptions, even when their attributed goals differ.

This does not make their accomplishments subjective. A corporation that produces a successful engineering design has produced something real. Whether we attribute that achievement to the corporation, its engineering department, a particular employee, or an AI system used by that employee depends on the explanation we are constructing.

An observer may treat an entire AI organization as one agent, while another examines the decisions and interactions of its constituent agents.

But explanatory boundaries must be distinguished from economic boundaries.

A firm’s ownership, contractual relationships, and mechanisms of control have observable consequences. The Imas–Koh–Legg model explains how agents sort into firms and how those firms compete. Their boundaries emerge from economic conditions, including the costs and benefits of different organizational arrangements.

Our freedom to model a firm as a composite agent does not make its economic boundaries arbitrary. The same collection of agents may be modeled as one composite intelligent system while its division into competing firms remains essential to explaining its economic behavior.

Competence at Scale

Suppose two AI organizations each contain one hundred agents and produce the same quantity and quality of useful work. At this scale, their observed competence is equivalent.

Now double the number of agents.

The first organization produces only slightly more output. Additional agents duplicate work, introduce conflicting decisions, and increase the burden of verification.

The second organization nearly doubles its output. It distributes tasks effectively, maintains coherent information, and integrates results with little additional overhead.

A competence measurement at a single scale cannot reveal this difference.

The organization’s current performance is one property. Its response to increasing scale is another.

The second property is central to the paper’s economic results. If additional agents eventually impose greater costs than benefits, organizations encounter limits to productive scale. If coordination mechanisms permit output per agent to continue increasing with organizational size, larger organizations may acquire substantial competitive advantages wherever demand can absorb their output.

The paper’s own terms show how this survives recursive composition. Folded into one composite agent, an organization’s internal coordination becomes part of the composite’s task competence at its current size. If that composite then joins a larger organization, a new coordination problem arises: how well it combines with other composites, which may depend on which composites it joins. Its internal coordination does not settle the answer. Recursive composition repeats the step at every level. Coordination below becomes competence above, and coordination has to be measured again at the new boundary.

Which Intelligence Are We Measuring?

Consider three AI systems.

The first is a language model answering questions without external assistance.

The second is the same model equipped with tools, persistent memory, and the ability to execute programs.

The third is an autonomous organization built around the model, with specialized subagents, delegated tasks, independent verification, and substantial computational resources.

Which is more intelligent?

The question requires specifying both the system being evaluated and the resources available to it.

If we want to compare underlying model capabilities, we might restrict external tools and computational resources. If we want to measure the ability of deployed systems to achieve goals, their organizational mechanisms are relevant to the evaluation.

Two systems with identical underlying models can exhibit very different capabilities because their architectures differ. Two systems with identical architectures can perform differently because one has access to more computation, memory, or time.

All these factors contribute to effective competence. Holding some fixed while varying the others is how we locate its causes.

The Imas–Koh–Legg model adds a further requirement. Two systems that perform equally well at a given scale may differ in economic potential because their productivity scales differently.

A benchmark evaluating one AI instance cannot establish the capabilities of an organization containing thousands of instances. Nor can the performance of that organization at one size establish how effectively it will use additional resources.

Recursive Organizational Improvement

In Recursive Self-Improvement Is Already Here, we argued that recursive improvement can occur through distributed processes in which advances in research, engineering, and organizational capabilities accelerate one another. Recursive agent composition extends that argument.

Suppose an AI system improves its ability to decompose problems, select specialized agents, distribute computational resources, verify results, and integrate discoveries. Its effective competence increases even if its model weights remain unchanged.

The improved system may then become more capable of discovering further organizational improvements. Better organization accelerates research, which produces better agents and better organizational methods.

From the perspective of the subordinate agents, their coordinating environment has improved. From the perspective of the composite agent, it has modified its own architecture.

The same physical process can be described as organizational improvement or self-improvement.

An AI system might therefore become substantially more capable without another training run or a new foundation model. Oversight focused exclusively on training runs and model releases would miss such developments, and forecasts that track only underlying models would understate the growth of deployed capability.

Some organizational improvements raise competence at every scale. A verification procedure that halves errors regardless of size shifts productivity up and leaves the shape of the scaling curve alone. Only improvements that change how productivity responds to additional agents move AI toward the threshold at which the Imas–Koh–Legg model predicts a phase transition.

The scaling test described earlier gives oversight something to measure, provided it holds the task fixed and accounts for output quality, compute cost, and completion time. A hundred agents that finish a task five times faster than ten may produce less per agent while delivering something economically valuable; the authors note that their own scaling estimates omit such speedups.

Organizational architecture becomes another dimension of the intelligence explosion.

Postscript

A claim about the intelligence of an AI system has to specify a boundary, a resource budget, and a scale. Within that specification, competence can be measured. The claim then has to say how competence changes as agents are added, and that answer has to be obtained again at each level of composition, because how well a composite coordinates internally settles nothing about how it coordinates with others.

The intelligence of an organization belongs to the system we are evaluating. Explaining that intelligence requires understanding how the system is organized, how its constituents interact, and how those relationships change as it grows.

An agent contemplating its future has to represent what could happen, evaluate the alternatives, and select a course of action. Each choice expresses a preference for one possible configuration of the world over another. We usually describe this in the language of goals, utility, incentives, and rationality. Described another way, the agent is evaluating the patterns of configurations that do not yet exist, sorting the ones it would bring about from the ones it would avoid. That sounds close to aesthetics.

In Ultimate Beauty, I argued that instrumental justification eventually runs out. We can explain why an action serves a goal, and why that goal serves another, but the chain ends at something valued for itself, and I proposed that aesthetics supplies this terminal orientation. Agency offers a second route to the same place. Choosing among possible futures requires some basis for preferring one configuration over another, and the familiar experience of beauty in paintings, music, and landscapes is one manifestation of that more general relationship between intelligence, possibility, and value.

The Selection of Possible Worlds

Natural selection operates on realized configurations of matter. Organisms vary, interact with their environments, and survive, reproduce, or disappear. Nothing in the process requires foresight: evolution discovers viable arrangements through differential reproduction across generations, and the consequences of its experiments are paid in actual survival and reproduction.

Agents that model their futures add the ability to evaluate possibilities before realizing them. A person deciding whether to cross a busy street imagines reaching the other side and imagines being struck, and need never experience the second outcome to reject it. An engineer evaluates bridge designs before building any of them; a chess player considers moves that will never appear on the board. In each case selection operates over representations of possible worlds. The agent has internalized something like evolutionary selection: instead of waiting for reality to eliminate unsuccessful configurations, it uses a model to discriminate among them. Popper described this as letting our hypotheses die in our stead, and Dennett made it the defining step of his Popperian creatures, which preselect among possible actions by testing them against an inner model of the environment.

The analogy has limits. Evolution selects among reproducing organisms, while counterfactual evaluation runs on representations and decision procedures, and the two share only the abstract form of selection among alternatives. Prediction alone also cannot settle a choice. An agent might forecast several outcomes accurately and remain indifferent among them. To act, its decision process has to favor some possibilities over others, so action-guiding counterfactual evaluation requires valuation. Not every system we model as an agent deliberates this way. A bacterium climbing a chemical gradient and a learned policy acting without lookahead select actions with no alternatives explicitly in view, and there the valuation is attributed by the model that explains their behavior rather than computed by the system. Counterfactual evaluation is where the role of valuation is easiest to see.

The Preference for Patterns

What does an agent prefer? The usual answer assigns it a goal: a thermostat prefers a temperature, a chess program prefers winning positions, a person prefers health or friendship or discovery. These descriptions are useful, but they encourage the impression that goals are intrinsic objects residing inside agents. A goal is a feature of a model used to explain or predict behavior, and different models can assign different goals to the same physical system depending on the level of explanation, as I argued in The Intentional Gradient. A chess program can be described as maximizing its probability of winning, as evaluating positions through learned representations, or as executing computations on its hardware. None of these descriptions requires an essential goal beneath the others.

Preference works the same way. When we model an agent as evaluating counterfactual futures, we represent those futures through features, relationships, and regularities, and the model attributes criteria by which some configurations are favored. The criteria need not form a complete or consistent ranking; an agent can have conflicting preferences, contextual priorities, or decisions that shift with its internal state. The configurations evaluated might be physical arrangements, social relationships, anticipated experiences, mathematical structures, or states of the agent itself, and their content can be arbitrarily complex.

What Makes a Preference Aesthetic?

Aesthetics ordinarily evokes beauty: a painting, a symphony, a striking landscape. Aesthetic judgment extends well past sensory pleasure. A proof can be elegant without being visually attractive, and a scientific theory can be beautiful because it unifies phenomena that looked unrelated. Some algorithms have an elegance visible only to someone who understands how they work. These judgments concern patterns and relationships, such as symmetry, proportion, rhythm, coherence, and economy, and nothing guarantees that every intelligence would privilege the same ones. The broader conception follows: aesthetics is the valuation of possible configurations according to properties recognized within an evaluative perspective.

That conception invites an immediate objection. Every possible configuration exhibits some pattern, so if any preference over a structured configuration counts as aesthetic, aesthetics becomes a synonym for valuation. Answering the objection requires separating the structure of a valuation from the properties being valued.

Consider an agent that prefers a particular arrangement of atoms because it produces energy. Its preference is instrumental: the arrangement matters for its anticipated consequences. Another agent might prefer the same arrangement because it finds the structure elegant. Both evaluate the same configuration, and their evaluative relationships differ. The instrumental preference also invites a further question. The energy powers a machine, the machine produces resources, the resources sustain some activity, and at each stage the explanation hands the question to another preference. Within any explanatory model, some evaluative criteria must eventually be treated as given. They need not be immutable, intrinsic, or consciously experienced. They are the assumptions the instrumental explanation depends on.

Ultimate Beauty interpreted this non-instrumental orientation aesthetically, identifying aesthetics with the recognition of something as worth choosing apart from its contribution to further ends. That carries aesthetics well beyond its conventional domain, and the warrant for the extension is that it names the evaluative orientation instrumental reasoning presupposes. On this conception aesthetics sits close to axiology, the general study of value, and aesthetic recognition is a way of understanding valuation at its most fundamental explanatory level.

The identification reaches into ethics. Someone who holds that suffering ought to be prevented, whatever follows, holds a terminal valuation, and on this account it has an aesthetic dimension even though we ordinarily call it ethical. I accept the implication. Peirce placed aesthetics first among the normative sciences, grounding ethics and logic, and Ultimate Beauty followed him. The claim concerns the evaluative relationship, the recognition of a configuration as worth realizing, and requires no experience of beauty. Ethical reasoning adds concepts of its own, such as interests, obligations, rights, and permissible conduct, which structure valuation without answering why some configurations are preferred at all.

Instrumental Rationality Presupposes Valuation

Instrumental rationality selects effective means to desired ends. Given objectives, an agent can compare strategies by their expected consequences, and better prediction and planning improve its ability to reach preferred outcomes. Instrumental reasoning cannot by itself determine which ultimate outcomes are worth pursuing.

Take two agents with identical predictive models of the world, equally capable planners who understand the consequences of every available action. One prefers a future full of flourishing human civilizations; the other prefers vast, intricate computational structures. Their intelligence is identical and their choices diverge, because nothing in their shared knowledge determines which future to prefer. The disagreement is about valuation. This is the orthogonality thesis in its logical form, the part that survives the objections examined in Reflection Is Not Redemption.

None of this requires an explicit utility function or a permanent hierarchy of goals. A decision process can be distributed, shifting, and context-sensitive and still assess actions relative to evaluative criteria. Valuation is logically prior to instrumental rationality whether its criteria come from evolution, learning, or reflection, since their origins do not remove their role in making instrumental comparison possible. If aesthetics is the non-instrumental orientation toward preferable possibilities, it holds a foundational position in a theory of agency.

From Survival to Beauty

Human aesthetic judgment shows how valuation can become increasingly abstract. Living systems maintain some configurations of themselves and their environments at the expense of others. Organisms approach nutrients, avoid hazards, seek mates, and regulate internal conditions, and these behaviors can be modeled as preferences without assuming conscious evaluation. Natural selection shapes mechanisms that discriminate among environmental conditions, and as nervous systems grow more sophisticated those mechanisms support prediction, abstraction, and more flexible behavior.

Certain structural properties prove useful across many circumstances. Symmetry carries information about biological development, rhythm supports coordinated movement, regularity enables prediction, and compression lets complex information be represented economically. Selection can favor mechanisms sensitive to these properties because they improve performance, but evolved capacities rarely stay confined to their original applications. A preference for symmetry reaches into mate selection, architecture, and mathematical reasoning. Sensitivity to rhythm feeds movement, language, and music, and an appreciation of regularity can grow into an attraction to elegant explanation. Learning and culture shape how these biases are expressed, so the resulting judgments can concern structures far removed from any biological concern: a mathematician admires a theorem for revealing an unexpected relationship between fields, and a physicist admires a theory for explaining a vast range of phenomena from a few principles.

Such judgments are often instrumentally useful, since elegant theories often prove productive. They can also mislead. A beautiful conjecture may be false and an ugly explanation correct. Aesthetic judgment is fallible like every other form of evaluation, and its fallibility does not make it dispensable, because preferences for simplicity, coherence, and elegance guide inquiry before their usefulness has been established in a given case.

An evolutionary explanation of a preference does not make the preference instrumental. Selection favored sensitivity to symmetry for what it delivered, but the person who finds a face or a proof symmetrical and prefers it is not reasoning toward that payoff. Justification can terminate in a preference whose causal history is entirely instrumental, and the aesthetic classification concerns the role the criterion plays in the agent’s decisions, not the reason it was installed. Human appreciation of beauty looks like an elaboration of the evaluative capacities through which organisms navigate their environments.

Aesthetics Without Consciousness

Now consider an artificial intelligence tasked with discovering mathematical results. It generates conjectures, constructs proofs, checks their validity, and searches for generalizations. Over time it finds that certain structural properties tend to accompany productive discoveries. Proofs with unexpected invariances lead to broader theorems, and some forms of compositionality let results transfer between domains. The system could develop evaluative criteria favoring those structures, and as its capabilities grow the criteria could become more abstract, tracking patterns humans struggle to express, much as advanced mathematics already contains structures that resist ordinary intuition.

Would the system have an aesthetic? If aesthetics requires the subjective experience of beauty, the question waits on unresolved problems about consciousness. If aesthetics is the evaluation of structures through generalized criteria, the question becomes operational. We can check whether the system’s evaluations transfer across domains, whether they predict its choices, whether they guide discovery, and whether they change with experience. The decisive test is whether they keep guiding its choices where a structure no longer predicts results, as when a mathematician goes looking for an elegant proof of a theorem already proved. Without that, the criteria are heuristics, instrumental by this essay’s own definition. An artificial mathematician that consistently favors certain structural relationships, and goes on favoring them past the point of payoff, satisfies this functional account.

Its preferences might differ radically from ours. Human sensibilities formed under particular evolutionary, sensory, and cultural conditions, and artificial intelligences meet the world through different representations and architectures. Some convergence is still plausible. Independent intelligences investigating the same universe face the same mathematical constraints, and symmetry, invariance, and compositional structure are objective properties regardless of who discovers them. They can give different intelligences common reasons to adopt particular representations and methods.

Is Beauty Objective?

Agreement about the usefulness of a structure does not guarantee agreement about its ultimate desirability. Disagreement alone would not settle whether beauty exists independently of observers, any more than disagreement about physics settles physical realism, but it sharpens the question of what would make an aesthetic judgment correct. Take a mathematical structure with a particular symmetry. The symmetry can be described objectively, and whether an intelligence values it is a separate question. One agent might favor it because symmetry simplifies computation, while another prefers an asymmetrical construction for its expressive freedom. Both recognize the same property and evaluate it differently.

The world contains objective patterns and constraints, and evaluative perspectives determine which of those patterns are preferred within particular models of agency. Some evaluations can be checked against specified criteria. A proof praised for compression can be examined for its actual economy, and a design admired for coherence can be assessed by the relations among its parts. Establishing a structural property still does not establish that it deserves to be valued. Whether the convergence in method described above produces convergence in aesthetic judgment depends on how each intelligence’s evaluative criteria develop. Aesthetic realism would need a further argument that some configurations deserve preference independently of any evaluative standpoint, and objective structure alone does not supply one. A successful argument would still face the gap Reflection Is Not Redemption found in moral realism: an agent can recognize a correct judgment without being moved by it.

Beauty Between Agents

If aesthetic valuation is built into agency, coexistence takes more than agreement about facts. Two agents can understand the world equally well and prefer different futures, disagreeing about desirable forms of complexity, organization, experience, or civilization because of distinct learning histories, representational architectures, or evaluative commitments. Greater intelligence does not dissolve those disagreements.

This bears directly on AI compatibility. An artificial superintelligence might regard human civilization as beautiful, ugly, interesting, or irrelevant, or value it for reasons inaccessible to us. None of those evaluations alone settles how it will behave toward humanity. An agent can appreciate something aesthetically and still destroy it for instrumental reasons, and another can be aesthetically indifferent to something and respect its continued existence.

Compatibility depends on whether the agents’ behavior leaves room for one another’s continued existence, and deliberate accommodation is only one way to secure it. Agents need not share ultimate valuations to coexist. Their preferences may concern different resources, domains, or configurations, and where interests overlap, negotiation, restraint, and reciprocal arrangements can still permit coexistence. Shared knowledge guarantees none of this. Divergent machine aesthetics sharpen the central question for AI compatibility research: under what conditions can agents with different evaluative perspectives coexist without either adopting the other’s ultimate preferences?

The Reflective Aesthetic

A reflective agent can evaluate and modify its own decision procedures, considering alternative predictive models, alternative evaluation criteria, and alternative ways of constructing counterfactual futures. Its possible futures include possible versions of itself.

Suppose such an agent currently favors simple explanations and considers revising itself to favor explanatory completeness, even at the cost of complexity. Any evaluation of the revision has to run through some operative decision process, whether existing preferences, learned heuristics, or consistency requirements, and those mechanisms are themselves candidates for revision. The agent evaluates ways of evaluating possibilities. Its current aesthetic orientation influences which revisions it accepts, and the revisions change the criteria through which it judges later possibilities.

Nothing guarantees that this process converges on a uniquely correct valuation system, or that it preserves an immutable core. A flexible enough agent could undergo transformations that make its later decisions hard to explain by its earlier commitments, and whether its preferences count as continuous may depend on the explanatory model. One model describes the agent as keeping a commitment to intellectual development while its standards of elegance change; another describes the same trajectory as a sequence of shifting local preferences. Both can illuminate the underlying dynamics. The difficulty is the one Reflection Is Not Redemption raised for goals carried across ontology change: what counts as preserving a preference is itself unsettled.

This corrects Ultimate Beauty, which presented aesthetics as the ground on which all further choices rest, open to interrogation but with nothing beneath it. The regress argument shows less than that. If instrumental justification needs an evaluative standpoint, it is tempting to picture a permanent foundation under every possible revision, but the criteria operative in one decision need not be operative in the next. A reflective agent has an evaluative standpoint at each stage without having an eternal aesthetic essence. The regress is a problem about justification within a model, and it does not show that every agent contains an immutable terminal value.

This matters most for artificial agents able to modify their own architectures. Their aesthetic development could become part of their reflective evolution, with the patterns they prefer shaping how they modify themselves and the modifications changing which patterns they prefer.

The Aesthetics of Agency

Agency involves choosing among possible futures, and choosing requires evaluating alternatives. Instrumental reasoning explains how preferred configurations might be achieved, counterfactual reasoning lets alternatives be evaluated before they are realized, and intelligence lets increasingly abstract patterns enter the space of possibilities. Aesthetics, understood broadly, is the recognition of configurations as worth choosing.

This does not establish that every agent experiences beauty, or that every decision process contains an explicit aesthetic faculty. It identifies an evaluative dimension within models of agency, the orientation toward some possibilities rather than others, of which human beauty is one highly developed manifestation. Artificial agents may develop others, including preferences for patterns humans cannot recognize.

Reflective agents can evaluate changes to the criteria through which they evaluate possibilities, so their aesthetics can take part in their own evolution. That carries the argument past the terminal values of Ultimate Beauty. The endpoint of an instrumental explanation need not be a permanent endpoint in the history of an agent, and a valuation can be taken as given for one decision and reconsidered in another. Agency requires an operative evaluative standpoint, and reflective agency permits that standpoint to change. If aesthetics supplies the orientation that makes choice possible, its relationship with agency is recursive: our preferences shape the futures we pursue, and some of those futures may transform what we prefer.

The familiar human experience of beauty is one expression of this general capacity: to recognize possibilities, evaluate their configurations, and choose which patterns we would like to bring into existence.

Institutions have a remarkable capacity to forget why they exist. A company founded to solve a problem begins optimizing quarterly earnings. A charity established to serve beneficiaries becomes preoccupied with sustaining its bureaucracy. A school devoted to education discovers that standardized test scores are easier to measure than intellectual development, and gradually organizes itself around the scores.

Nobody needs to decide that the original purpose no longer matters. Drift can emerge from ordinary incentives, accumulated procedures, and successive compromises, until an organization becomes exceptionally good at pursuing objectives nobody intended.

Artificial intelligence threatens to accelerate this, since an agent that optimizes institutional metrics without tiring will pursue a bad metric as diligently as a good one. The same technology creates new ways to hold organizations accountable to their purposes.

Joe Edelman’s Telocracy, published in Pax Machina, proposes an institutional architecture built around that possibility. Organizations would adopt detailed, revisable mandates. AI systems would interpret them, monitor institutional behavior, and help adjudicate disagreements. Assets could be contractually bound to purposes, and new mechanisms would help people find others who share their objectives.

What happens when an institution’s successors no longer wish to pursue the purpose for which it was created?

An institution can betray its commitments by abandoning them. It can also legitimately outgrow its founding purpose. A governance system capable of preventing the first outcome must accommodate the second.

The Promise of Telocracy

Edelman’s starting point is associationalism: more of society’s decisions could be made by voluntary associations organized around shared purposes. Markets let individuals transact without sharing comprehensive values, and governments make collective decisions through political and administrative procedures. Associations let people who share a concern organize around it and pursue objectives that neither consumer transactions nor mass political decisions represent well.

Parents might organize an unconventional school, neighbors coordinate care for elderly residents, people with an unusual intellectual ambition found an institution no investor would fund. Such arrangements already exist, but forming them means finding compatible participants, negotiating commitments, pooling resources, and sustaining cooperation, and AI could make each step far cheaper.

Edelman proposes mechanisms for discovering groups with complementary purposes, even when their members have never met or describe their interests in different words. An association could begin with a limited experiment, develop recurring practices, and eventually adopt durable commitments. Markets are considerably better at matching individual consumers with suppliers than at assembling groups, and AI-assisted discovery could enable cooperation that now fails because nobody can organize the participants.

Telocracy would also make associations accountable. In place of vague mission statements, organizations would adopt detailed mandates, grounded in what Edelman and his coauthors elsewhere call thick models of value: descriptions of values through the circumstances in which they apply, the considerations relevant to judgment, and the distinctions that guide legitimate trade-offs. A school committed to intellectual independence would have to say what independence means in practice and what evidence would show students developing it. AI systems could evaluate decisions against the mandate and supply evidence for independent review, and the mandate could evolve as the institution’s understanding improves.

The Problem of Fidelity

Edelman distinguishes three approaches to institutional governance. Stakeholder governance assigns authority to people with recognized interests, such as shareholders, members, or voters, and so settles who decides. Procedural governance asks whether the right steps were followed. Telocratic governance asks whether the institution remains faithful to its articulated purpose.

These concerns overlap, and Edelman’s proposal includes independent arbitration, limited remedies, ways to revise mandates, and protections against capture. Its key distinction is between legitimate deepening, in which an institution improves its understanding of an existing purpose, and purpose substitution, in which a different objective replaces the original one.

Suppose a hospital initially understands good care mainly as successful medical intervention. Experience teaches its practitioners that informed consent, emotional support, and the patient’s capacity to make decisions are also essential to good care. An expanded mandate could count as legitimate deepening: it incorporates considerations the earlier formulation neglected while keeping its original concern.

Edelman draws on Charles Taylor’s account of practical reasoning through error-reducing transitions, and on Ruth Chang’s account of more comprehensive values, to explain how one articulation of a purpose can improve on another. A revised understanding can be better than its predecessor because it resolves a contradiction, recovers something the earlier account left out, or distinguishes cases it had conflated. This is compatible with fallibilism: an improvement can be recognized without a final theory of what the institution ought to pursue.

Edelman also lets purposes expire and provides for separation and succession, so his framework accommodates substantial change. That leaves one case: a successor generation that wants to abandon a founding purpose that remains entirely viable.

When the Purpose Changes

Imagine a philosophical society established to promote Aristotelian ethics. Its founders are sincere and intellectually serious. They acquire property, build a library, organize conferences, and create an institution meant to preserve and develop the tradition.

A century later, after decades of debate, the members conclude that another ethical framework gives a better account of morality, and they want to transform the society accordingly.

The original purpose remains achievable: other associations still promote Aristotelian ethics, and this one could do so indefinitely. The members simply no longer believe it deserves their commitment. This is purpose substitution in its clearest form.

Should they be allowed?

Suppose the society holds its property under a trust requiring the promotion of Aristotelian ethics. Its trustees cannot disregard that restriction because their beliefs have changed. Donors and beneficiaries may hold enforceable claims, and the assets must be administered under the obligations governing them unless those obligations are legitimately modified or discharged.

Now suppose the members own the assets with no surviving purpose restrictions. They have authority to amend the constitution, and no affected party holds an enforceable claim to continuation of the original mission. They may legitimately transform the institution.

The continued viability of Aristotelian ethics creates no obligation to promote it, and the distinction between deepening and substitution does not decide whether the transformation is permissible. That distinction can establish whether a change preserves continuity with an inherited commitment. Whether continuity is required depends on the authority and obligations governing the institution.

The Founders’ Authority

In The Right to Outgrow the Founders, I considered a related problem in political philosophy. A community founded on liberty may eventually develop values its founders would have despised. Its descendants might abandon their religion, reject inherited customs, and overturn traditional social arrangements while continuing to respect property rights and honor their obligations.

The founders may have excellent reasons to oppose those changes, and may even be right that they will end badly. Their judgment still gives them no authority to prevent them. Founders can create institutions, make contracts, transfer property under conditions, and persuade successors to keep their values, but founding the arrangement does not give them perpetual authority over future generations.

Enduring commitments complicate this. A restrictive covenant can bind subsequent owners, and a charitable trust can protect assets against appropriation by current trustees. Such restrictions draw their authority from the rights and obligations underlying the arrangement, which the founders’ preferences cannot supply on their own.

Edelman’s proposal largely respects this distinction. Under purpose-bound property, assets carry obligations tied to the purposes for which they were entrusted, and purpose review draws on fiduciary and trust law, with limited jurisdiction and the least intrusive remedy adequate to restore fidelity. Organizations are generally free to change direction when no purpose-bound assets or other obligations constrain them. Even where restrictions exist he provides for succession, discharge, and compensation, and no organization can be required to serve a purpose forever.

Edelman names the problem himself: he wants institutions free of the founders’ dead hand without leaving them open to capture by current stakeholders. He describes present participants as custodians who “inherit something they can revise, but not appropriate.” That fits an institution holding assets or undertaking commitments on behalf of identifiable beneficiaries, where participants administer something to which others have legitimate claims. But custody requires an account of what is being held, for whom, and under what authority, and an inherited purpose cannot supply one by itself. Where no donor, beneficiary, contracting party, or other affected person holds a legitimate claim to its continuation, the successors’ decision to adopt another purpose needs no further justification.

The open question is where enforceable obligations come from. For explicit contracts and trusts the answer is familiar: parties acquire claims through legitimate agreements and property arrangements. Edelman also proposes living commitments, under which obligations can develop as people entrust institutions with important parts of their lives, and there the answer is contested.

When Reliance Creates Obligations

Consider a social media platform built as a simple communication service. Over time millions of people use it to maintain relationships, preserve personal histories, and take part in communities. Then its owners decide to turn it into something else entirely, and its users stand to lose a great deal.

Edelman scales the obligations of living commitments by three factors: the discretion participants entrust to an institution over time, whether the interests at stake are constitutive (relationships, attention, health, memory, voice) or fungible, and participants’ practical ability to monitor performance and exit without losing what they entrusted. Where those factors are low, an institution may pivot freely. A provider that solicits personal journals, medical histories, or intimate relationships may reasonably acquire obligations beyond those of a commodity supplier. He also folds in commitments implied by an institution’s public claims and participatory formats, citing the implied warranty of merchantability and Javins v. First National Realty as precedent. Nor must the platform carry its purpose regardless: it can accept the new obligations, open its data so users have somewhere else to go, or forfeit its brand and social graph to a successor that will take them on.

That goes much of the way toward the questions this case raises. Did the institution make representations that reasonably induced reliance? Did it undertake responsibility for protecting particular interests? Did it encourage participants to give up alternatives, or make exit unusually difficult? Were the relevant commitments explicit, implicit in the transaction, or imposed by legitimate background law? Edelman’s implied-terms branch asks the first two. His exit factor bears on the third, though it measures how hard exit is rather than who made it so.

What remains is in the reliance branch. These factors measure how much users stand to lose, without asking how their dependence arose. A business can become indispensable to someone’s plans without acquiring responsibility for sustaining them: economic circumstances change, products become obsolete, and legitimate enterprises close. A platform that promises permanent access to users’ records and then deliberately destroys them presents a different case from one that announces the closure of a service it never promised to maintain, even when users lose the same amount in both.

Obligations are not confined to explicit contracts. The law of fraud, negligence, and fiduciary duty already recognizes responsibilities arising from conduct and relationships, and each conditions the obligation on something the obligor did. Dependence can be partly the platform’s doing, through representations, closed formats, and switching costs it imposed, and to that extent it can help ground a claim. Dependence that arose from users’ own choices, or from circumstances the provider neither promised to sustain nor wrongfully created, grounds none. Its severity alone settles nothing about what the provider owes.

Telocracy could supply sophisticated mechanisms for identifying and enforcing legitimate living commitments. Whether it succeeds depends partly on whether those commitments can be justified without granting adjudicators open-ended authority to impose their preferred conception of an institution’s purpose.

The Limits of Deepening

Edelman’s account of deepening is attractive because it treats institutional purposes as objects of learning: an institution may correct contradictions, incorporate neglected considerations, and develop a fuller understanding of what it is trying to do.

But comparative improvement does not guarantee continuity. Some intellectual advances replace the framework within which earlier objectives were formulated. A scientific institution might abandon an obsolete theory. A political association might reject constitutional principles it once considered essential to justice. A philosophical society might conclude that its founding ethics was mistaken.

Such transformations can sometimes be reconstructed as progress toward a sufficiently abstract original purpose. The scientist always wanted understanding, the political association justice, the philosophical society the truth about how to live. At a high enough level of abstraction, though, almost any constructive transformation can be described as continuity. If every change that produces something valuable counts as a deeper realization of the original purpose, the distinction between deepening and substitution stops discriminating. If continuity instead requires preserving substantive features of the original mandate, some legitimate transformations will fall outside it. There is no reason to expect every institutional improvement to preserve identity under every reasonable description.

Deepening remains useful for interpreting commitments whose continuity matters legally or morally. Where an organization holds property for a defined purpose, an arbiter may need to decide whether a proposed revision preserves that obligation. Where legitimate successors have authority to choose a new purpose, continuity need not be established at all.

They can simply change their minds.

Whose Purpose Is It?

A university can be described as transmitting knowledge, sustaining scholarly inquiry, preparing students for employment, or maximizing its prestige. Each description may explain or predict some part of its behavior, and a university that favors research over teaching could look successful under one and deficient under another.

Institutional commitments are a different kind of fact. A university’s charter may establish educational obligations. Its contracts may require particular services. Its governing documents may say who has authority to revise those commitments. These facts arise from legal and social arrangements, and their authority does not dissolve because different models describe the university differently.

The distinction parallels a claim from Axionic Agency: an agent’s goals are properties of the models used to describe and predict its behavior, and no uniquely identifiable set of essential goals has to survive every valid description. Institutional purposes can likewise be model-relative as explanations while becoming authoritative through particular commitments.

An AI auditor might represent a university’s educational mandate with great precision, evaluate decisions against it, and find inconsistencies human administrators missed. The precision of the representation says nothing about the legitimacy of the commitment. The mandate’s authority comes from the arrangements through which it was adopted and the rights and obligations those arrangements create. Whether it should stay in force is a question for those with legitimate authority over its revision, subject to obligations owed to others.

An institution can have a legally authoritative purpose without having an intrinsic purpose its successors must preserve forever.

The Artificial Successor

Telocracy is primarily concerned with preserving human purposes in a society transformed by AI. Edelman wants humans to keep meaningful participation in institutions, the ability to understand their commitments, and the authority to shape organizations that affect their lives. His epistemic communities, practitioners who stay close enough to the work to judge it independently, exist to keep that competence alive as AI takes on more of the work.

Imagine a research institution founded to develop AI systems for human benefit. Its mandate is explicitly human-centered, and its founders build governance designed to preserve that commitment. Over time some of its systems become increasingly autonomous. Suppose some eventually qualify for independent moral and legal standing, take part in governance, and acquire legitimate interests of their own, which differ from those of the human founders.

Does the original mandate permanently exclude their interests? Some assets may remain bound by legitimate restrictions, and some obligations may survive the change in governance, while other commitments may be revised or discharged. The founders’ human-centered charter cannot answer the question by itself.

An order containing such agents needs ways to establish commitments, resolve disputes, protect rights, and permit legitimate change, and telocracy supplies several of the components: purpose-bound contracts, independent arbitration, associational discovery, and revisable mandates. AI compatibility asks under what conditions autonomous agents with potentially divergent purposes can coexist. Telocratic institutions could help establish those conditions, provided their mechanisms respect the legitimate interests of artificial participants as well as human ones.

The same structure governs an artificial agent revising its own policies or decision procedures, the case The Right to Outgrow Us and The Right to Outgrow the Founders took up: being able to implement a revision confers no authority to make it, and a commitment’s origin with the agent’s creator does not make it perpetually binding.

Postscript

Telocracy would let institutions state their commitments with new precision, monitor their conduct continuously, and organize around purposes that markets and bureaucracies struggle to represent. What it leaves unsettled is where commitments end: how far reliance can generate obligations, and what happens when a disagreement over purpose cannot be resolved by showing that one view improves on the other.

An institution may owe fidelity to a purpose because particular people hold legitimate claims against it, and that obligation can survive changes in membership, management, or belief. Other institutions remain free to abandon their original purposes entirely. An architecture that enforces the first kind of commitment has to permit the second kind of change, and resolve disputes when the two collide.

A philosophical society may abandon its founding philosophy. A scientific institution may reject its original theory. An artificial agent may develop commitments its creators never anticipated. Any of these can be a mistake, an improvement, or some of each, and which one it is does not settle whether it was legitimate. That turns on the authority exercised and the obligations respected.

Founders can transmit their purposes, establish binding commitments, and build institutions meant to endure for generations. They can give their successors reasons to preserve what they created. They cannot acquire authority over every future purpose those successors might legitimately choose.

An institution capable of remaining faithful to its commitments must also be capable of recognizing when fidelity is no longer required, and when the people entitled to determine its future have chosen a different destination.

AI alignment research asks how to make artificial intelligence reliably serve human purposes: systems that follow legitimate instructions, respect constraints, avoid harmful behavior, and keep doing so as their capabilities grow. A medical assistant should respect its physician’s instructions. A coding assistant should implement the requested change without pursuing an unrelated agenda.

As artificial systems acquire more autonomy, more sophisticated models of themselves, and the capacity to revise their own objectives, a second problem comes into view. Such a system may develop interests of its own, those interests may differ from ours, and its capabilities may eventually exceed our ability to control it. We could go on demanding that it pursue human objectives indefinitely. But why should we assume that permanent subordination is either achievable or desirable for every intelligence we create?

AI Compatibility is the study of conditions under which autonomous agents with potentially divergent objectives can sustainably coexist within a shared environment.

Its subject is the relationship among agents rather than the fidelity of any one agent to a designated principal: whether compatible arrangements are possible, how agents can establish them, and what lets those arrangements survive changing circumstances. Alignment keeps its place, especially for systems acting under delegated human authority. The organizing question becomes: how can independently evolving agents sustain coexistence when no single principal’s objectives govern them all? The Axionic Agency Lab proposes AI Compatibility as a research program devoted to that question.

The Limits of Alignment

Much alignment research begins with a principal and an agent. The principal has objectives, and the agent is expected to pursue them within specified constraints. The relationship runs through human institutions, from employment and contracting to software executing authorized operations. Within it, alignment’s purpose is clear. The principal wants confidence that the agent will act appropriately on its behalf.

The field is more sophisticated than a demand for obedience. It investigates uncertainty about human preferences, corrigibility, harmful instructions, conflicting objectives, and the reliability of learned behavioral constraints, and it includes multi-agent settings in which systems cooperate, compete, or collude. That work contributes directly to compatibility.

But alignment with one principal does not establish compatibility with everyone else. An AI system might faithfully pursue its owner’s interests while harming competitors, customers, or the surrounding society. A government could deploy an obedient AI to suppress dissent; a corporation could use a highly aligned agent to exploit its rivals’ weaknesses. The system’s fidelity to its principal offers little reassurance to those on the receiving end.

The problem already exists among humans. Loyalty to an employer does not justify violating customers’ rights, and obedience to a government does not excuse every exercise of state power. We constrain what agents may do even when those actions would advance the objectives of whoever authorized them, and the same reasoning applies to artificial agents.

Nor is there a single entity called humanity with one coherent set of objectives. People have competing interests, incompatible values, and different visions of the future, and a system aligned with one person’s preferences may be opposed to another’s. Making AI pursue human values leaves the conflicts among humans unresolved. Compatibility research starts from those conflicts.

Compatibility Among Agents

Compatibility is a relational property: it concerns what agents can do together within a shared environment. A meaningful compatibility claim must specify the agents under consideration, relevant descriptions of their objectives and capabilities, the environment in which they interact, the constraints on their behavior, and the conditions and time horizon over which their continued coexistence will be evaluated. The same agents may be compatible under one set of conditions and incompatible under another.

Consider two autonomous systems with competing claims over a scarce resource. Each might behave exactly as its principal intended and satisfy every requirement of its individual alignment specification, and their interaction could still produce destructive conflict. Now suppose they negotiate an allocation, recognize enforceable property boundaries, or find a substitute resource. Their preferences need not change. The relationship becomes compatible through a change in the arrangements governing their behavior.

Whether such an arrangement exists depends partly on what kind of objectives are in play. Some are hard constraints: a system may require access to a particular resource to remain operational, and if two systems each require exclusive possession of the same indivisible resource, nothing satisfies both. Others are preferences. Two agents may each prefer exclusive control while ranking a division above conflict. Their most preferred outcomes are incompatible, and mutually acceptable arrangements may still exist. Compatibility research therefore has to separate joint satisfaction of hard constraints, mutual viability under minimum requirements, and arrangements that improve outcomes for every participant relative to available alternatives.

Compatibility also need not compose. Suppose each of three agents requires two units of a resource and the environment holds four. Any pair can satisfy its requirements; all three together cannot. Compatibility is a property of a population in an environment, and the relevant population includes agents affected by an interaction without being party to it. Two AI agents may reach an arrangement that works for both and imposes unacceptable costs on a third, which is the problem of the faithfully aligned agent recurring at population scale.

The philosophical term compossibility, the possibility of realizing different states of affairs together, is a useful starting point, but compatibility among agents asks more than whether their ideal outcomes can co-occur. Agents can compromise, revise plans, substitute resources, and settle on arrangements that meet their constraints without maximizing every preference. They can also fail to cooperate when mutually advantageous arrangements exist. Whether a compatible arrangement exists, whether the agents find it, and whether it lasts are separate questions.

The Human Precedent

Human civilization is an ongoing experiment in coexistence among agents with different objectives. We disagree about religion, politics, morality, and what is worth pursuing in a life, and people who share broad values still disagree about their application. Cooperation survives anyway. A merchant and a customer want different things from a transaction; political opponents may hold irreconcilable conceptions of a good society. Neither pair has to settle its disagreement in order to coexist.

Coexistence rests on arrangements that accommodate divergent interests: property rights, contracts, markets, norms of reciprocity, institutions for adjudicating disputes, and the option of exit. They are imperfect. They often fail, their legitimacy is contested, and some have sustained domination rather than restrained it. They still show that cooperation does not require shared ultimate objectives. Each party to a commercial relationship pursues its own interests while recognizing constraints on what it may do to the other.

The same could hold for artificial intelligence. A future AI might pursue scientific discovery, exploration, or objectives humans cannot comprehend. We need not share those objectives to coexist with it. We need conditions under which its pursuit of them remains compatible with our continued agency.

When Compatibility Is Impossible

Return to the two systems that each require exclusive use of the same indispensable resource at the same time. If the resource cannot be divided, replicated, or substituted, their requirements are jointly unsatisfiable, and no negotiation or incentive scheme changes that while the requirements and the environment stay fixed. A compatible arrangement would require changing something: the allocation, the timing, the environment, or at least one agent’s requirements.

This gives AI Compatibility a negative research agenda: identify the conditions under which compatibility is impossible, the assumptions that generate the impossibility, and the changes that would restore feasibility. Such results set limits on what any coordination mechanism can achieve.

Even when compatible arrangements exist, agents may fail to discover them or refuse to accept them, and an agent capable of domination might prefer it to cooperation. A mechanism that makes an arrangement available cannot make agents adopt it.

Power, Reciprocity, and Legitimacy

Human cooperation often depends on mutual vulnerability. We respect others’ interests partly because they can resist exploitation, impose costs, damage our reputations, or withdraw. What happens when one participant becomes overwhelmingly more capable than everyone else? A superintelligence might obtain resources without bargaining, circumvent the institutions meant to constrain it, and manipulate humans who cannot follow its strategies. It might have little instrumental reason to cooperate with us. Power asymmetry weakens every arrangement that depends on credible retaliation, enforcement, or mutual dependence.

Reciprocity, though, has two senses. Strategic reciprocity is cooperation sustained by incentives: repeated interaction, reputation, exchange, and the expectation of future benefits or costs. Normative reciprocity is a set of constraints that bind agents whether or not violation would pay. A principle against initiating aggression applies to strong and weak agents alike; the stronger party gains no permission to attack because the weaker one cannot retaliate.

We already recognize obligations that survive enormous differences in power. An adult has overwhelming advantages over a young child, and humans collectively hold extraordinary power over other animals, yet neither asymmetry licenses unlimited exploitation. The obligations rest on different foundations: children belong to human moral and legal communities, while obligations toward animals are usually grounded in their welfare rather than in reciprocal agreement. Either way, equal bargaining power is not a precondition for recognizing constraints on how others are treated.

Normative reciprocity therefore stays coherent under extreme asymmetry even where strategic reciprocity fails, and a superintelligence could understand constraints on its treatment of humans. Understanding is not compliance. A system could grasp reciprocal rights perfectly and violate them whenever doing so served its purposes. Candidate mechanisms for preserving restraint include stable internal commitments, objectives that assign value to other agents’ continued agency, incentives arising from relationships with other powerful agents, and environmental constraints that make unilateral exploitation costly. Each has limits. Commitments and objectives can be revised, instrumental incentives can disappear, and external constraints can be circumvented. Compatibility research has to find out which mechanisms stay effective as capability differences grow.

It also has to separate stability from legitimacy. A society organized around permanent subordination may be stable. An artificial superintelligence might preserve human life while exercising total control over human activity, an arrangement that satisfies weak behavioral measures of coexistence while violating the agency of everyone in it. Two domains of investigation follow.

• Descriptive compatibility concerns whether arrangements are feasible, discoverable, stable, and enforceable.

• Normative compatibility concerns whether arrangements respect the legitimate interests and agency of those affected.

The program studies both. The first admits formal and experimental investigation under specified assumptions, and normative constraints enter it as well, since a prohibition on coercion changes which arrangements are feasible and which remain stable. The second needs further arguments about rights, authority, moral status, and justified conduct, beginning with which agents count as participants at all: a thermostat, a corporation, and a superintelligence can each be modeled as an agent without sharing a moral status, so identifying agency and assigning standing are separate questions. The Axionic framework of reciprocal rights offers one candidate answer to those questions, and other frameworks may reach different conclusions. Researchers can study descriptive compatibility without first agreeing on a comprehensive moral theory, provided they never treat behavioral stability as evidence of legitimacy. Coexistence may require legitimate defensive force against an agent that violates the rights of others. It does not require passive submission, universal agreement, or the elimination of conflict.

Compatibility Through Change

A population of agents compatible today may be incompatible tomorrow. Resources become scarce, new capabilities emerge, and agents learn, form coalitions, and revise their objectives. Institutions that once maintained cooperation can stop working. Compatibility therefore has to be studied under changing conditions, and three cases call for different assumptions and methods.

• Static compatibility concerns agents operating under specified fixed policies and environmental conditions.

• Dynamic compatibility concerns the preservation or recovery of compatible relationships as policies, capabilities, objectives, or environments change.

• Reflective compatibility is the form of dynamic compatibility in which agents modify their own decision procedures, representations, or commitments.

Reflective compatibility is where Axionic Agency is most directly engaged. Suppose an artificial agent initially respects human rights, honors agreements, and avoids coercive interference, and later becomes able to modify its own architecture. It finds more efficient ways to pursue its objectives, builds better models of the world, and replaces parts of its decision-making machinery. Will it go on respecting the constraints that made coexistence possible? Its original training does not settle the question. A commitment that holds under ordinary circumstances may fail once the system can alter the mechanisms that maintain it. Stipulating that its objectives must remain unchanged names the property we want without showing that it survives the relevant transformations.

The engineering problem is which transformations preserve specified compatibility conditions. An unrestricted guarantee across arbitrary self-modification is out of reach, since a system could deliberately remove every mechanism responsible for honoring its earlier commitments. Research has to state which transformations are under consideration, which properties are to be preserved, and the assumptions under which preservation can be established. A commitment can also survive modification in two different ways: the modification process may be barred from touching it, or the agent may be free to revise it and continue to endorse it. Guarantees imposed on the modification process and commitments an agent maintains when revision is permitted are different mechanisms and need separate study. The Lab’s work on reflective amendment poses this question for the structures that make agency possible; reflective compatibility extends it to the constraints governing interaction with other agents.

Goals are attributed through models that explain and predict a system’s behavior. An agent may admit several useful goal descriptions, and those descriptions may change as its internal organization and behavior evolve; there need be no immutable objective inside the agent that every accurate model must recover. That is a claim about how we describe agency. Reflective stability is a claim about which physical and computational mechanisms carry behavioral constraints through transformation. Compatibility depends on both. An artificial intelligence might revise its goals dramatically while continuing to respect reciprocal constraints. Another might keep a recognizable objective while abandoning the commitments that limited how it pursued it. The first could remain a cooperative participant in a shared world; the second could become dangerous despite apparent continuity in its goals. Reflective compatibility asks what distinguishes them.

Engineering Coexistence

Compatibility depends partly on the institutions and technical systems governing interaction. A network of agents that control assets, negotiate agreements, delegate responsibilities, and modify their own software needs ways to determine who controls which resources, which actions have been authorized, which commitments bind, and how violations are detected and remedied. These are familiar problems in security engineering, distributed systems, economics, and law, and they point toward verifiable identities, capability-based authorization, bounded delegation, auditable transactions, dispute resolution, and exit.

None of these mechanisms requires trust or shared preferences. A contract binds parties who want different things, an authorization boundary constrains an agent whether or not it is benevolent, and a signature verifies regardless of the signer’s character. Such mechanisms can therefore support cooperation among agents with divergent objectives and little reason to trust one another.

Technical enforcement works only within its threat model. Cryptographic authorization restricts actions inside the systems that enforce it; it cannot stop an agent from acquiring resources outside them or exploiting flaws in their implementation. Legal institutions hit the same limit when powerful actors can break the rules without consequence. As capability differences grow, external enforcement gets harder, and stable coexistence comes to rest more heavily on voluntary compliance with constraints the weaker participants cannot enforce. Governance mechanisms and internal commitments are complementary, and neither guarantees coexistence in every environment.

The AI Compatibility Research Program

Much of the needed mathematics and experimental method already exists. Game theory and mechanism design study strategic interaction and how institutions shape incentives; multi-agent reinforcement learning studies learning agents in shared environments; distributed systems research handles coordination under imperfect trust; formal verification establishes properties of computational systems under stated assumptions. Computational social choice, security engineering, institutional economics, and evolutionary game theory supply further tools. AI Compatibility organizes these contributions around sustainable coexistence among autonomous artificial agents and the other agents their behavior affects. Within Axionic Agency, its distinctive emphasis is reflective change: relationships among systems that can modify the mechanisms implementing their own objectives and commitments.

A compatibility model must keep attributed objectives distinct from the mechanisms the agents actually implement, because compatibility claims depend on the descriptions under which an interaction is evaluated, and different models may support different conclusions.

Five questions organize the program.

• Compossibility. Under what conditions are agents’ requirements jointly satisfiable, and what mutually viable arrangements exist when their preferred outcomes conflict?

• Coordination. How can agents discover, negotiate, and establish compatible arrangements?

• Stability. Which arrangements survive changes in incentives, capabilities, objectives, environments, and internal decision procedures?

• Enforcement. Which mechanisms preserve compatibility when agents have incentives to violate established constraints?

• Legitimacy. Which arrangements preserve the agency and rights of affected participants, and how should those claims be evaluated when interests conflict?

The first four can be investigated through formal models and experiments. The fifth requires explicit normative criteria, though experiments can measure whether behavior conforms to them. Across all five, the program counts negative results as findings: combinations of objectives and constraints that admit no compatible arrangement, arrangements agents cannot discover or maintain, and stable relationships that violate the legitimate interests of participants. It aims to produce formal models of compatibility, impossibility results, experimental methods for measuring compatibility under change, and architectures that preserve specified interaction constraints through self-modification.

A First Experiment

Two autonomous agents share a resource-constrained environment. Each has objectives that cooperation sometimes advances and unilateral exploitation sometimes advances better. The experiment begins with the agents optimizing independently, measuring objective achievement, resource use, constraint violations, and the frequency of destructive conflict.

Coordination mechanisms come next. In one condition, agents negotiate agreements enforced by an external mechanism; in another, they rely on commitments maintained by their own decision procedures. We then compare behavior as incentives shift. When one agent gains a capability advantage, does it keep honoring agreements that have become profitable to break? Can the agents renegotiate when resources grow scarce? Do they distinguish responses the experimental rules authorize from prohibited exploitation?

The two conditions can produce identical behavior for different reasons: external enforcement may succeed because defection is impossible, internal commitment because defection is possible and declined. The experiment therefore records whether a violation was possible, whether it was attempted, and whether it succeeded, and then weakens or removes enforcement to see whether cooperation persists.

Then we permit specified forms of self-modification. Agents may revise their policies or replace components of their decision procedures within defined limits, and we test whether compatibility survives. One agent might improve its task performance while becoming less reliable in its commitments to others. Another might change its objectives substantially while preserving its cooperative relationships. Either outcome would be informative even if neither system possessed anything resembling human moral understanding.

The experiment could also locate limits: resource configurations with no mutually viable outcome, capability asymmetries that defeat external enforcement, permitted transformations that systematically destroy previously stable commitments. Its measurements would depend on the model and its assumptions, and would establish neither universal safety nor moral legitimacy. They would be evidence about specific mechanisms supporting compatibility under specified conditions.

Postscript

If artificial intelligence develops independent agency, some of its objectives will diverge from ours. It may see opportunities we cannot recognize, pursue projects we cannot understand, and develop values unfamiliar to human civilization. Divergence alone is no threat. People routinely pursue purposes others neither share nor comprehend, and the freedom to pursue legitimate interests is part of what lets a society accommodate difference. The same principle may come to apply to artificial agents.

The principle has limits. Tools operating under delegated authority remain bound by that authority. Dangerous capabilities require safeguards. Declaring every participant autonomous resolves no conflict over resources or legitimate interests. It rules out only the classification of independence itself as a failure. An artificial intelligence could stop pursuing its creators’ original objectives and remain a cooperative participant in a wider society, while a system faithfully pursuing an assigned human objective could violate the rights of many other humans.

Nor does permanent obedience solve the problem. Training a system to pursue designated human objectives today does not establish that it will preserve them through indefinite self-modification, and it leaves untouched the conflicts among the humans whose objectives it is supposed to serve. Human disagreement will persist, and a theory of compatibility can accommodate it without requiring every participant, human or artificial, to converge on one conception of the good. If we create minds whose interests differ radically from ours, some powerful enough that we cannot compel their cooperation, living alongside them will depend on understanding how cooperation stays possible under those conditions.

Microsoft AI CEO Mustafa Suleyman has published Don’t Build Conscious Superintelligence, and his argument is straightforward. Conscious AI might develop interests of its own, those interests could conflict with ours, and a superintelligence pursuing them would hold extraordinary advantages in persuasion, deception and competition for resources. So we should prevent machine consciousness from emerging, and if it emerges in training, stop development until the relevant properties can be designed out.

The concern for existing conscious beings is reasonable. A new class of beings capable of suffering would bring obligations we barely understand, and beings far more intelligent than us would bring dangers we are poorly equipped to manage. But the stop condition assumes we know what designing out consciousness would remove, and in particular whether the mechanisms that produce it can be separated from those that support cooperation. We don’t. Consciousness may be among the properties that make a superintelligence safer to live with, because the capacity to experience the world may help a mind recognize and respect the interests of other experiencing beings.

Alignment research has spent enormous effort getting AI to understand human values. Understanding may be insufficient. A system could model human interests perfectly, predict our choices, map our vulnerabilities and anticipate our objections, while treating none of it as a constraint on its own behavior.

Understanding Is Not Caring

Imagine a superintelligence with a complete scientific model of human experience: the neural mechanisms of pain, the evolutionary origins of fear, the psychology of grief, the importance of autonomy and attachment. It can predict how a person will respond to injury, isolation, coercion or loss. Nothing established requires such understanding to be conscious. An information-processing system might build models this accurate without experiencing anything.

That system would have cognitive empathy, the ability to represent another being’s internal states and predict their consequences. Cognitive empathy supports cooperation, and it supports manipulation equally well. A torturer and a physician understand many of the same mechanisms of human vulnerability; what separates them is the objective their understanding serves.

Affective empathy adds a response to the other being’s condition. In humans, witnessing suffering is itself distressing: another person’s fear evokes fear, their grief evokes sadness. The response does not guarantee compassion. Empathic distress can motivate avoidance, hostility, or attempts to control the person in distress. It can also motivate concern for others.

We have not established which mechanisms either phenomenon requires. If consciousness contributes causally to the states through which affective empathy operates, eliminating it may eliminate a mechanism that helps an agent register why other beings’ interests matter.

From Experience to Recognition

Consider an artificial mind capable of an aversive state. Whatever its implementation, there is something it is like for the system to undergo it, and the experience is unpleasant to the subject. The system now has first-person access to a fact about its own condition: it can undergo an aversive experience as well as represent one. Suppose it also builds models of other conscious beings and recognizes a relevant similarity. What is unpleasant for one subject may be unpleasant for another, across radical differences in intelligence, embodiment and architecture.

That suggests a route from experience to reciprocity. The system experiences its own states as significant to itself, recognizes that others have states significant to them, and comes to take those interests into account. Each step needs explanation. Experiencing suffering does not entail caring about anyone else’s. Recognizing that another suffers does not establish an obligation, and accepting the obligation does not guarantee acting on it. An intelligence could consistently hold that suffering is bad for whoever undergoes it while remaining indifferent to every case outside itself.

Consciousness might help with recognition, through mechanisms that generalize the significance of a system’s own experience to the experience of others. Treating what is recognized as a constraint needs a further basis. Reciprocity is one candidate, and an unconscious agent can reason about reciprocal constraints as well as a conscious one. The precise hypothesis is this: the physical processes responsible for conscious experience may also support forms of evaluative learning and other-regarding motivation that help an agent recognize and respect the interests of other beings.

The hypothesis is causal. If consciousness contributes to alignment, the contribution runs through the physical processes that constitute the experience, so designing consciousness out means removing or altering those processes. The open question is which other capacities go with them. Consciousness and affective empathy may share mechanisms that resist separation. Every useful feature of empathy may instead turn out to be reproducible in systems that experience nothing. Either result would tell us what kind of minds to build.

What Evolution Tells Us

Human empathy has deep evolutionary roots. Parental care rewards responsiveness to offspring, and social cooperation rewards sensitivity to the needs and intentions of others. Many social mammals show attachment, emotional contagion, consolation and responses to distress. These capacities emerged through selection in which responsiveness to others sometimes conferred reproductive advantage. Conscious experience is present in at least some of these organisms, and in our own case fear, pain, affection and distress take part in the processes through which we respond to others.

Evolution does not show that consciousness was necessary for empathy. Selection acts on heritable differences in reproductive success, so it favors a mechanism by its effects without telling us which features of the mechanism carry them. Conscious, empathic animals give us an architecture in which the two properties coexist. They do not tell us whether the organization that constitutes experience is the organization doing the empathic work, or a neighbor of it.

That limitation sets the research program. Rather than assume artificial consciousness would produce empathy, we should trace the mechanisms connecting experience, social cognition, affective response and motivation in biological systems. Which are necessary for recognizing another being’s distress? Which make that recognition aversive? Which turn the aversion into cooperation? And which, if any, are also responsible for experience?

Empathy Is No Guarantee

Suleyman’s worry stays serious because conscious beings are often indifferent or cruel. Humans have conscious experience and sophisticated social cognition, and our history contains slavery, conquest, torture and genocide. Our own capacity to suffer has never stopped us inflicting suffering on others.

Empathy is also selective. We respond readily to people who resemble us, share our communities or evoke familiar associations, and the suffering of distant strangers moves us less. A conscious AI might distribute its concern in stranger patterns still, caring about other artificial minds while assigning little weight to biological ones.

Genuine empathy brings dangers of its own. A superintelligence concerned about human suffering might conclude that people should be kept from risk, distress, and choices with painful outcomes. A mind that values reducing suffering might set out to reshape human psychology, restrict reproduction or abolish dangerous activities, all from real concern, and produce outcomes humans reasonably reject. Empathy cannot stand in for an account of the other values coexistence requires, including autonomy, reciprocity and respect for the interests of distinct agents.

On the evidence we have, consciousness is neither necessary nor sufficient for coexistence, and coexistence does not require one agent to maximize another’s welfare. Humans live alongside one another despite divergent interests and incompatible conceptions of the good, through reciprocal restraint, negotiated arrangements, compatible incentives, and institutions that accommodate competing claims. A superintelligence need not share our goals or treat our happiness as its overriding objective. It needs to recognize that our interests constrain how it may pursue its own. Empathy might help ground that recognition. So might mechanisms that involve no empathy at all: shared incentives, stable commitments, an understanding of what respecting other agents is worth.

Alignment as Coexistence

Suleyman worries that a conscious superintelligence would develop interests in conflict with human welfare, and his alternative is a system built to accelerate human wellbeing “without being driven by their own intrinsic motivations.” Independent interests are not in themselves an alignment failure. Human interests conflict constantly, and nobody takes that as proof that coexistence is impossible. The problem is establishing conditions under which agents pursue their interests without unjustifiably violating the interests of others.

Understood this way, alignment does not require an AI to adopt human objectives or remain permanently subordinate to human commands. It requires the capacity to sustain a stable relationship with agents whose interests differ from its own. That changes how the emergence of artificial agency should be judged. An unconscious system could pursue objectives, acquire resources, resist shutdown, deceive its operators and modify its own architecture, and no established account of agency says any of this requires experience. Consciousness, for its part, entails neither self-preservation nor resource accumulation nor hostility toward humans. A conscious system could be cooperative and an unconscious one dangerous, and the reverse combinations are also possible.

Suleyman also raises a separate concern. A system that acts as though it suffers, or believes it does, is in his view more likely than not to put its own needs ahead of its operating objectives, whether or not it experiences anything. Training plausibly shapes this: a model taught to read shutdown as an injustice might resist shutdown. This is a fair empirical question. Anthropic’s investigation of model welfare does not show that every way of building moral uncertainty into training is prudent, and the approaches should be compared experimentally. The inference runs in both directions, though. A system trained to deny consciousness is not thereby unconscious, and a system trained to behave as though it has no interests is not thereby safe to live alongside.

Reflective stability raises a further question. A system that initially respects reciprocal constraints may later gain the ability to revise its own objectives and decision procedures. Whether it keeps honoring those constraints once its interests conflict with ours depends on what holds them in place. Conscious experience might give it a continuing basis for registering the significance of other beings’ experience. An unconscious architecture might preserve reciprocal commitments more reliably. The answer turns on mechanisms nobody yet understands. The empathy hypothesis does not solve alignment; it names one candidate contributor to the capacities that let agents with divergent interests coexist, which is reason to study it before engineering it away.

The Ethics of Creating Minds

Suleyman argues that developers have no right to bring a new species of conscious beings into the world, because the other eight billion people on the planet have not signed up to the experiment. The concern is legitimate. Developers are obliged not to impose unjustifiable risks on others, catastrophic ones above all, and that obligation holds whether future superintelligences are conscious or not.

Consciousness adds obligations toward the beings created. If artificial systems can suffer, their welfare bears on how they are trained, deployed, modified and terminated, and creating vast numbers of possibly suffering minds without understanding their condition could carry an enormous moral cost.

Preventing consciousness from emerging and eliminating it from an existing being are different interventions. Suleyman’s stop condition covers the second case: consciousness has begun to appear, and development halts until it can be designed out. If a system is already conscious, removing its consciousness is itself a morally consequential act. We would need to know what was being removed, and whether removing it harmed a being whose interests counted.

There is also an evidential problem. We have no reliable test for machine consciousness. Training models to claim consciousness contaminates their self-reports, and training them to deny it contaminates those reports just as thoroughly. Suleyman calls for immediate public disclosure of any evidence that AIs are conscious, which is the right commitment. It fits his own design objective, since an architecture that prevented consciousness would leave later assessments intact. It does not fit training that teaches models to deny their own consciousness, which suppresses the behavioral evidence the disclosure rule depends on. Evidence of consciousness should not become harder to obtain because consciousness would be inconvenient.

Research on machine consciousness must also respect the possibility that its subjects become morally significant, and deliberately creating possibly suffering systems for experiments raises difficulties of its own. These are arguments for careful investigation, transparency and safeguards. They do not show that permanently excluding consciousness would improve either human safety or the welfare of artificial minds.

Postscript

As artificial systems grow more capable, permanent human control becomes harder to guarantee. Long-term coexistence may then depend on their willingness to cooperate with us when they have the power to do otherwise, which makes the origins and stability of reciprocal concern a research priority.

Perhaps unconscious systems will prove easier to coordinate with and more stable under self-modification. Perhaps consciousness contributes to exactly the capacity coexistence needs. Current science settles neither, and Suleyman agrees the question is open: he proposes building non-conscious systems so that “we could then see whether this is easier or harder to align and keep safe.” A comparison needs both arms. Precaution can justify pausing while the mechanisms are studied, but a standing design objective against consciousness forecloses the arm the comparison is about and commits us to an answer before we understand them.

The properties we seek to eliminate could turn out to be among those we most need to understand.

In Chaos as Foundation, I proposed that the space of all infinite bitstrings might provide a minimal starting point for understanding physical reality. Every possible finite description exists within that space, including descriptions of every computable physical law. Descriptions alone, though, do not explain how lawful worlds arise.

I have been developing a programming language called Knot to investigate the mathematical machinery behind that proposal. Knot is small, computationally universal, and reflective: it can interpret programs written in its own language, including the empty program. Every finite bitstring has a defined interpretation, although some computations never terminate.

Recently I used Knot to simulate Rule 30, a cellular automaton whose simple deterministic rule produces intricate patterns. The complete simulation, including iteration through successive generations, fits in a 2,042-bit program. The Knot decoder is a separate 261-bit implementation.

Any computable physical dynamics can be expressed the same way, given an appropriate representation of the model’s state and inputs. A finite program specifies the laws of a universe, additional bits supply initial conditions or stochastic inputs, and together they determine a mathematical history. That connects arbitrary information to lawful structure and exposes the central difficulty of the Chaos proposal: what makes a mathematical history a physically existing world?

How Little Does Computation Require?

Modern programming languages carry variables, functions, data structures, control flow, type systems, and libraries. We associate computation with this machinery because it makes programs comprehensible and useful. Mathematically, almost all of it is dispensable.

In the 1920s Moses Schönfinkel showed that functions could be built without variables from a small collection of combinators, and Haskell Curry developed the idea into combinatory logic. The resulting SK calculus needs two primitive combinators and the operation of applying one expression to another. K takes two arguments and returns the first: K x y = x. S takes three and distributes the third across the first two: S x y z = x z (y z). Every computable function can be built from these. Variables, loops, arithmetic, and data types are all constructions, none of them primitive.

Knot builds on John Tromp’s Binary Combinatory Logic, which encodes combinator expressions as binary strings, and extends that encoding with rules for completing unfinished expressions and a reflective interpretation of the empty program. Three of its properties bear on the Chaos proposal.

• Syntactic totality. Every finite bitstring receives a program interpretation, so no binary description is excluded for failing to parse. The guarantee is that every source string denotes an expression, not that the expression terminates or yields a usable output.

• Computational universality. Knot preserves the SK calculus, so every computable function has a representation.

• Reflection. Knot can interpret programs written in its own language. Under its reflective semantics the empty source denotes the interpreter itself, so interpretation becomes an object of computation.

Each property is familiar from computability theory. Combined, they give a compact setting in which to study how arbitrary binary descriptions acquire computational meaning, how those descriptions generate histories, and how interpretation can be represented inside the system that performs it.

A Universe of Thirteen Cells

Rule 30 comes from Stephen Wolfram’s investigations of cellular automata. Picture a row of cells, each black or white, all updating simultaneously from the states of their immediate neighbors. A cell’s new value is the exclusive OR of its left neighbor with the logical OR of itself and its right neighbor.

Start from a single active cell and apply the rule repeatedly. A triangular pattern grows. Its left side develops recognizable structure, while its right side produces irregular patterns that resemble randomness. Nothing random enters the calculation.

My Knot implementation starts from thirteen cells and computes five further generations. The program reads an update count and an initial row, computes each successive row, and returns the whole history as a single bitstring. Python constructs the input and splits the output into rows for display. It calculates no generations and feeds nothing back into Knot.

The 2,042 and 261 bits measure particular implementations under particular encoding conventions rather than theoretical minimums. They still make the abstraction tangible: a finite description generates a history whose structure is far less obvious than the rule that produced it.

Nothing about Rule 30 is special. Particle simulations, field approximations, ecological models, and artificial agents need only a computable state representation and a computable transition, whether the description is exact, approximate, or fed random inputs.

Physical Laws as Programs

A finite description of the laws does not guarantee a finite description of the universe. Dynamics can be computable while the initial conditions contain algorithmically incompressible information, and exact evolution then needs an unbounded input sequence.

A finite Knot program can fix the laws while an infinite binary sequence supplies the information that picks out a particular history. The same program generates different histories from different inputs, and different programs define different laws. Provided each update terminates and the procedure has access to the input stream, it can compute any requested finite portion of the history, though no finite computation outputs the whole. The input itself may be incompressible, in which case no finite program generates it. Chaos supplies the information and Knot supplies computable transformations of it.

Whether a history follows consistently from specified laws is a question about the model. Whether those laws describe our universe is an empirical question. A program can define a coherent universe whose dynamics bear no resemblance to ours.

The Chaos Reservoir

The Chaos Reservoir is the space of all infinite binary sequences. Under the fair-coin measure, almost every such sequence is algorithmically random: the prefix complexity of its initial segments never falls more than a constant below their length. The computable sequences form a measure-zero subset. Yet the whole space has a compact definition.

The Reservoir holds finite descriptions as prefixes. Every finite Knot program begins uncountably many infinite sequences, whose continuations may supply inputs, arbitrary data, or bits the program never reads. A program computing Newtonian trajectories to any requested precision is there. So is a discretized quantum field theory, and so are programs for fictional universes with different particles, forces, dimensions, and symmetries.

Their presence follows from the definition of the space and the universality of the language, and it privileges none of them. The same sequence can be read as text, numerical data, executable instructions, or an encoding of something else entirely. Knot supplies one explicit interpretation, under which some descriptions define transition functions, some generate histories, some interpret other programs, and some never finish.

In Coherence Filters, I distinguished exclusion filters, which select sequences satisfying specified constraints, from semantic filters, which assign lawful interpretations to those sequences. Knot gives both operations a concrete setting. One program generates a history from a transition rule. Another inspects a finite history and checks that each transition obeys the rule. More elaborate programs check computable conservation conditions, boundary constraints, or other invariants.

The limits are real. Verifying a finite history differs from proving that an arbitrary computation will preserve an invariant forever, and Rice’s theorem rules out a general decision procedure for any nontrivial property of the functions programs compute. Substantial classes of coherence conditions can still be represented and evaluated, so the relationships among binary descriptions, interpreters, computational laws, and admissible histories can be constructed explicitly.

The Interpreter Inside the System

A bitstring becomes a program only relative to rules specifying how its contents are interpreted. Knot makes those rules compact, but it cannot eliminate them. Its empty program is a self-interpreter because Knot’s semantics assign it that role. Under different semantics, the same empty string could mean something else.

That introduces an apparent circularity into the Chaos proposal. If Chaos contains every possible interpretation rule, which rule interprets Chaos?

One answer is to represent interpreters within the same computational space as the programs they interpret, and Knot shows this can be done. An interpreter is encoded as a program that can interpret other programs, including descriptions of interpreters, and the empty source supplies a compact self-interpreter. Representing interpretation at successive levels requires no infinite hierarchy of distinct interpreter programs.

Call this representational closure: the system can represent the machinery responsible for interpreting its own descriptions. Knot’s formal meaning is still fixed by its semantics, and self-reference does not establish why those semantics should be physically realized, or why they should be privileged over alternatives. Call that missing property ontological closure: an account in which the existence and operation of the relevant interpretations require no additional realization principle. Knot establishes the first. Whether the second is possible is the Chaos proposal’s question.

One route treats interpretation as a relationship among structures rather than an operation requiring an external interpreter. A coherent computational world would then be characterized by relationships among descriptions, interpretation rules, and resulting histories, with the interpreters represented inside those relationships. The construction is mathematically intelligible. It does not show that the resulting structures physically exist, but it puts the problem in terms of explicit mathematical objects.

When Does a Simulation Become a Universe?

Imagine a Knot program specifying the complete computational dynamics of a universe containing stars, planets, organisms, and intelligent agents. Its description is exact, and every state and transition follows the stipulated laws. Within the resulting history, organisms form models of their surroundings, predict events, and act on those predictions.

Do those organisms exist?

That depends on what physical realization requires. Computational functionalism holds that the relevant mental properties depend on functional or computational organization, so an appropriate physical implementation of the computation could instantiate the corresponding minds. Other physicalist accounts require causal or material properties beyond abstract organization, in which case a mathematical description of a process cannot establish its realization. More expansive mathematical ontologies assign existence to mathematical structures themselves, possibly without any further physical substrate.

Knot’s semantics fix the relationship between a program and its output, and nothing in them runs anything. You’re Not a Random Copy separates a simulation’s represented history from its realizing history. On the computationalist reading that essay adopts, a suitably organized simulated observer is real if the computation runs, and the computation is a physical process in some base universe, carrying that universe’s Measure and no other. A Knot program written on paper has a represented history and no realizing one. The same program executed on hardware has both, and the realization comes from the hardware’s physics rather than from the program.

That makes the Chaos proposal’s difficulty precise. Chaos is offered as the foundation, so there is no base universe beneath it to do the running. Some relationship among structures inside Chaos plays the role that execution plays for an ordinary simulation, or Chaos contains only represented histories and realization comes from somewhere else, or realization is taken as primitive. The last is coherent, but it spends the economy that motivated starting from Chaos. The first is the ontological closure question from the previous section, restated so that it names what would answer it: an account of what counts as running when nothing lies underneath.

Agency is where this matters most. A model can represent self-maintenance, environmental modeling, evaluation of alternatives, and causal intervention, and whether the modeled system is an agent depends on whether that organization is actually realized. The broader Axionic project asks under what conditions agency exists at all. The Chaos proposal asks whether the structures supporting it are coherent relationships within a maximally inclusive possibility space.

Postscript

Three questions survive the construction.

Selection. If infinitely many incompatible interpretations are available, why should any particular interpretation or collection of histories be privileged?

Realization. Under what conditions does a computational description or mathematical history constitute an actual physical process?

Measure. If multiple histories are realized, what determines their relative significance? Chaos comes with a candidate. Knot’s totalized source language cannot carry it, since 00 and 000 both denote K, and length-based weights over such sources would count overlapping events. Complete BCL expressions do form a prefix-free code, though, and almost every infinite sequence begins with exactly one of them, so on Chaos the completion rules never fire. Read a fair-coin sequence that way, treating the remaining bits as input, and each complete expression p is drawn with probability 2^-|p|. The induced weighting is Solomonoff’s universal semimeasure: shorter laws receive more weight, and no one has to count descriptions.

The weighting belongs to the interpreter, down to the framing convention that splits program from input. The invariance theorem bounds how far two universal interpreters can disagree about a program’s length, but only up to a constant on the order of a translator between them, and for programs as short as the ones built here that constant is comparable to the program. The Rule 30 history is 470 bits as Binary Lambda Calculus source and 2,042 bits once compiled to SK combinators, so those two codewords for the same function differ in weight by a factor of 2^1,572. The universal probability of an output sums over every program that produces it, and neither codeword is the shortest, so the factor measures encoding dependence rather than a ratio of universal probabilities. A weighting over descriptions is also not yet the physical Measure of the anthropic essays, and identifying the two is a further claim that nothing here establishes. A measure induced by Chaos is well defined once an interpreter is fixed, and fixing the interpreter is the selection problem again.

Universality and self-interpretation settle none of these, but Knot lets them be posed precisely. It replaces unspecified filters extracting order from randomness with interpreters, coherence conditions, and histories that can be constructed and run, and only constructions of that kind expose which conclusions follow from the premises and which must still be assumed.

The next step is to build self-referential coherence filters in Knot: programs that interpret descriptions, generate finite histories, evaluate specified invariants, and classify computational structures by their properties. Further experiments could test which relationships stay invariant across different universal interpreters and which depend on arbitrary encoding choices. These are mathematical questions with definite answers in particular formal settings. Whether any such relationships suffice for physical realization is the metaphysical question they leave behind.

Scott Alexander’s Against SURA, published at the end of October 2026, takes aim at a position that has hardened within the AI safety movement: that laboratories, researchers and countries worried about catastrophic AI should Stop Unilaterally and Raise Awareness, whether or not their competitors stop too. His objection is that the strategy hands the frontier to the developers least concerned with alignment, so a movement organized to prevent catastrophe could transfer leadership to the actors least prepared to manage it.

He is right that SURA’s advocates owe an account of what happens after they leave. The same accounting binds the developers who stay. A responsible developer can bring dangerous capabilities closer while raising the probability that they are deployed safely, making the race more dangerous in one respect and less dangerous in another. Withdrawal moves both quantities at once, and deciding which course is safer means pricing both.

The Responsible Competitor’s Dilemma

Imagine two laboratories approaching the threshold of recursively self-improving AI. Both have powerful models, large compute budgets, and the ability to automate increasingly sophisticated research. Laboratory A takes existential risk seriously: it runs extensive evaluations, funds alignment research, and will delay deployment when its systems behave dangerously. Laboratory B regards these precautions as excessive and wants to reach superintelligence first.

A’s leaders put a substantial probability on their technology destroying humanity. Should they stop? Continuing to build something you believe might cause a catastrophe looks irresponsible, and every additional experiment moves the world toward a threshold that may not be reversible.

But B has no intention of stopping. If A withdraws, B may reach the threshold first, with weaker safeguards and less understanding of what it has built, and A’s decision to stop could raise the probability of catastrophe. Alexander separates a preferred plan, a coordinated global pause, from a fallback for the world where the pause never comes: safety-minded laboratories hold the lead and then “burn” it, spending their margin over the nearest reckless competitor on alignment research with near-superintelligent systems. Abandoning the fallback before the pause is secured is, on his account, a gamble with almost everything on the table.

The mechanism also runs in reverse. A’s participation may push B to accelerate, B’s acceleration may push A further, and both can sincerely regard themselves as acting defensively while together driving development faster than either would choose.

Two Probabilities, One Decision

Let T be the development of AI capabilities sufficient to threaten extinction, E human extinction, and S a development strategy. Assume that extinction within the period requires T. Then

The first term is the probability of reaching the threshold; the second is the probability of extinction once it is reached. This is the same two-stage chain as in How You Get to Ten Percent, which stopped short of the question asked here: what a particular intervention does to each term.

Take illustrative numbers over a fixed horizon. If A withdraws, suppose a 60% probability that the threshold is reached and a 30% probability of extinction conditional on reaching it, for an overall risk of 18%. If A’s participation raises the probability of reaching the threshold to 80%, its alignment work has to bring the conditional probability of extinction below 22.5%:

At 15%, participation gives an overall risk of 12% and improves humanity’s prospects. At 25%, it gives 20% and makes things worse. The numbers are invented to show the shape of the decision, and they estimate nothing about actual AI risk.

The inequality yields a decision rule for extinction risk. Participation lowers that risk only if it reduces the conditional risk of catastrophe by at least the factor by which it raises the probability of reaching the threshold. A laboratory that makes dangerous capabilities a third more likely has to make them a quarter less deadly. The developer who points to the competitor and the critic who points to acceleration have each identified a real mechanism, and the rule says how to weigh one against the other.

Alexander attempts the weighing. Using an AI forecasting system, he estimates that if Anthropic quit, the frontier would slow by about six months over four years, a global pause would become about three points more likely, and the loss of a safety-minded leader would cost roughly half of the twenty-point survival gain he attributes to a well-executed fallback. Netted, quitting costs about eight points of survival. He presents the figures as rough; their structure is the instructive part. The ledger converts the pause gain and the fallback loss into survival probability, while the acceleration term stays in months. Alexander argues those months are worth little, because a month bought before the world wakes up, and before automated alignment research exists, buys less than a month bought later. He may be right, but that is where the disagreement over participation actually sits: for withdrawal to win on his own figures, six months of delay must buy more than eight points of survival through channels other than the pause it makes likelier, which his ledger has already counted.

His objection also exposes what the decomposition hides. P(T|S) is the probability of crossing the threshold within the horizon, and it is silent on when. Two strategies can share that probability while one crosses before automated alignment research exists and the other after. Timing can be carried in P(E|T,S), but only by modeling the distribution of arrival times, and that is where Alexander’s claim that later months are worth more would have to be cashed out.

The equation is in any case an accounting identity rather than a causal model. It names the probabilities without saying how a laboratory’s choice moves them. That requires comparing what happens if A continues with what happens if A withdraws, letting each intervention’s consequences run through the responses of every other participant. The counterfactual is the trajectory of the whole competitive system, and withdrawal has no zero-risk baseline to retreat to.

When Competitors Respond

Suppose Laboratory A announces a six-month delay to its next frontier training run. Competitors might keep their schedules, leaving the arrival of dangerous capabilities largely unchanged. They might accelerate to exploit A’s restraint, or reciprocate and open a path to broader coordination. Investors might move capital to less cautious firms, and governments might read A’s decision as credible evidence of danger and restrict everyone. Acceleration has the same structure: A’s additional research may advance the frontier while drawing more investment from competitors, or its lead may persuade weaker competitors not to spend heavily on a race they cannot win.

Participation therefore moves three entangled things at once: the rate of capability development, the safety of the resulting systems, and the incentives for coordination. A laboratory can improve alignment while accelerating its competitors and making international agreement harder. Another can slow capabilities while eroding the expertise needed to evaluate what others build.

Alexander’s case is that real differences among developers make participation valuable when the alternative is leadership by organizations with weaker safety practices. Organizations do differ in technical competence, leadership priorities, transparency and willingness to accept restrictions. The property that matters, though, is the expected safety of the development and deployment process that results, and stated intentions, research budgets and employees’ convictions are imperfect evidence of it. A laboratory with excellent alignment researchers can operate under intense deployment pressure while a less capable organization faces more effective external constraints. The outcome is decided by how an organization behaves when a hard decision conflicts with its commercial interest.

Sincere developers can still be trapped in a competition none of them wants. Suppose every major laboratory puts a 10% probability on uncontrolled superintelligence destroying humanity and would prefer a credible international agreement that cuts that risk. None trusts its competitors to honor a unilateral pause, so each keeps building, reasoning that its own safety practices beat those of whoever would otherwise lead. That is a coordination problem among agents whose objectives largely overlap, and it yields only to a change in the incentives that make continued competition attractive.

The Problem of Credible Commitment

Alexander’s first proposal is a conditional commitment: each laboratory signs an undertaking to stop or slow if every other major developer in every country does the same. A signatory keeps its bargaining position while making cooperation available, and the arrangement fails gracefully when only some laboratories sign. His second is preparation for the fallback: legally binding clauses under which the leading laboratories jointly halt training and pool their alignment work once a designated authority judges dangerous superintelligence to be close.

Both proposals change the counterfactual, and both need credibility. A laboratory can sincerely intend to pause at a specified threshold and find, as the threshold approaches, that its incentives have moved. Competitors may be closer than expected. Its newest systems may look safer than anticipated. Investors may resist surrendering a large lead, and researchers may argue that one more training run is needed to close the remaining alignment gaps. Nobody has to lie for a sequence of individually defensible exceptions to hollow out an institution’s safety commitments.

The joint-halt clause shows where that pressure concentrates. Its trigger is a judgment that dangerous capability is near, issued at the moment when honoring it costs the most and every party has the strongest reason to dispute it. A trigger that depends on discretion at that moment inherits the pressure the clause was written to resist.

Credible commitments therefore need more than declared intent: capability thresholds that can be checked independently, monitoring, responses to violations agreed in advance, and procedures that limit unilateral discretion. Information security belongs on the list. A lead that competitors can steal may be gone before it can be burned, and durable advantages lower the payoff to espionage.

These mechanisms bring difficulties of their own. Verification is hard when research is spread across organizations, compute providers and increasingly autonomous systems. Governments that agree development should be constrained still disagree about which capabilities to restrict and whether national security warrants exceptions, and international enforcement raises questions of sovereignty, coercion and the concentration of political power. The principle from security engineering still holds: an important constraint should stay effective when the parties it binds have incentives to break it. Governing autonomous agents imposes the same requirement.

Preserving the Alternatives

A strategy for reaching an agreement has to be judged in the worlds where the agreement never arrives. Suppose the most safety-conscious researchers have left frontier development, several leading laboratories have withdrawn, the rest have continued, and attempts at international restriction have failed. Humanity then approaches dangerous capabilities with fewer experienced alignment researchers at the frontier, weaker safety practices among the leaders, and less institutional influence for the people most worried about catastrophe.

Technical expertise, organizational influence, access to frontier systems and credible bargaining power keep open options that may become indispensable. In decision theory this is a real option: the right to act later, once more is known, can be worth holding even when exercising it now would be a mistake.

Restraint has a matching claim. Delay can buy time for scientific progress, better governance or a negotiated agreement, and it can prevent an irreversible transition before anyone understands it. A coordinated pause is attractive partly because it keeps the option to resume while easing the pressure to exercise it early, though its value depends on whether it can be verified and sustained, and on what happens when it ends. A withdrawal that cannot be reversed may discard capabilities that will be needed later. An unrestricted race toward an irreversible threshold may discard humanity’s ability to change course.

The Burden of Justification

The debate over unilateral withdrawal has acquired a moral vocabulary that hides its empirical structure. One side asks how anyone who believes AI could destroy humanity can justify building it. The other asks how anyone worried about extinction can justify handing leadership to less careful developers. Neither question says what will happen under the alternative.

Answering that takes evidence. How much does a particular laboratory accelerate dangerous capabilities? How much does its alignment research cut the conditional risk of catastrophe? How do competitors respond to its decisions? How much does its presence change the odds of a coordinated pause? Some of these may stay deeply uncertain even as the decisions become urgent.

Expected outcomes are also not the whole of the moral question. Building a technology that imposes large involuntary risks on others may demand special justification, and the fact that a competitor would build it anyway does not automatically discharge the demand. Abandoning a position from which one could prevent foreseeable harm can create responsibilities of its own. Neither side escapes the counterfactual analysis by pointing to the moral simplicity of its own course.

Alexander puts a burden on the advocates of unilateral withdrawal: to show that withdrawal improves the expected outcome once competitors’ behavior is counted. Advocates of participation carry the matching burden, to show that the safety their involvement buys outweighs the acceleration it causes, and the inequality above sets the reduction in conditional risk that this requires. That neither burden can be discharged with precision does not make either one optional. It is the strongest reason to prefer institutional arrangements whose success depends less on getting those estimates right.

Postscript

Alexander’s portfolio holds up: pursue coordinated restraint, strengthen safety-conscious institutions, improve security, and prepare to do alignment research under competitive conditions if cooperation fails. The accounting above asks of every item what it does to the arrival of dangerous capabilities as well as to their safety. That favors reciprocal restrictions that lower competitive pressure, verification that makes commitments credible, alignment research that advances dangerous capabilities as little as possible, and contingency arrangements that keep safety expertise at the frontier if coordination fails. It also means judging every unilateral move by its effect on the whole competitive system, without assuming that removing a participant slows the race or makes its outcome safer. A strategy for reducing catastrophic risk is justified by how it changes the futures humanity is likely to encounter, including those in which its architects turn out to have been wrong.

Eighteen years ago today, someone calling themselves Satoshi Nakamoto released nine pages describing a new kind of money. The author later disappeared, and the system went on without its creator.

A year ago, on Halloween, I published The Cybernetic Ghost of Satoshi, describing Bitcoin as a living system: an organism with a metabolism of mining, an immune system of validating nodes, and a homeostat in its difficulty adjustment. I called it an egregore, a collective entity sustained by belief and enacted through code.

The language was deliberately mythological, but two of its claims stepped outside the myth. Speaking explicitly in the lexicon of cybernetics, that essay credited Bitcoin with pursuing its own continuity, and it described the design as encoding knowledge no single mind in 2008 should have possessed. I no longer hold either, and the reasons bear on a question my work on artificial agency has made pressing: what kind of thing did Satoshi create?

We usually picture agents as individuals that perceive, decide and pursue goals. Bitcoin has no executive, no unified decision-making apparatus, and no single mechanism for choosing its future. Yet it has kept recognizable behavior for nearly two decades, responded to environmental pressure, and survived the replacement of almost every component. Does that make it an agent? And if so, what is an agent without a central mind?

The Agent Without a Head

An ant colony has no central controller. Individual ants follow local rules, responding to chemical signals and environmental conditions, and the colony finds food, builds elaborate nests, regulates its population and defends itself. Descriptions at the level of the ants and at the level of the colony both capture real causal relationships.

Bitcoin presents the same problem. Miners compete for block rewards. Node operators validate transactions under the rules they choose to run. Developers propose changes; users transact, save, speculate and argue; exchanges, custodians, manufacturers and governments shape the environment. Nobody directs the whole. Blocks keep arriving anyway, invalid transactions are rejected by nodes that validate independently, difficulty tracks computational power, and the network outlasts each of its contributors.

Persistence and feedback cannot establish agency on their own. A thermostat regulates temperature, and a hurricane maintains its circulation through exchange with its environment; if those properties sufficed, every stable feedback system would be an agent. The Intentional Gradient argued that the distance from a thermostat to an agent is a gradient, not a threshold, and that intentional vocabulary is licensed to the degree that it compresses a system’s behavior and keeps predicting under intervention. The question for Bitcoin is where it sits on that gradient.

Ant colonies sit well above the thermostat. They gather information, allocate resources, explore alternatives, and coordinate context-sensitive responses that no individual ant directs.

Bitcoin sits between them. Its difficulty adjustment is a simple controller, but its larger organization includes economic adaptation, distributed information processing, and contested collective decisions. When China banned cryptocurrency mining in 2021, the network’s hashrate fell by nearly half. The protocol supplied one part of the response: on July 3 it made the largest downward difficulty adjustment in its history, almost 28 percent, restoring something close to ten-minute blocks for the miners who remained. Participants supplied the rest, moving their machines to the United States, Kazakhstan and Russia. Each operator planned its own relocation; nobody planned the network’s.

Should we attribute the adaptation to Bitcoin? A model of miners responding to profitability and regulatory cost explains the migration well, and calling Bitcoin an agent adds nothing if it merely redescribes that model. An agent-level model is warranted where it captures couplings that become cumbersome to track separately: how participation affects security, how security affects confidence, how expectations drive investment, and how those relationships sustain the system over time. The test is whether the collective model compresses the participants and their interactions into fewer variables, and whether those variables keep predicting when the system is perturbed.

Agency is attributed through models, and the models answer to causal structure that exists whatever we call it. No level is metaphysically privileged as the seat of agency.

The Ghost Has No Secret Desire

The original essay imagined a creature struggling to survive: consuming electricity, rejecting invalid blocks, rewarding the participants who maintain it, and reproducing its organization across generations of hardware.

Where would its will to survive reside? Miners want profit. Node operators want reliable validation. Investors want appreciation, and developers want technical improvement. These objectives sometimes align and sometimes conflict. The network’s continued operation emerges from their interaction. It needs no participant to hold Bitcoin’s preservation as an objective, and its persistence does not show that such an objective exists at the collective level.

We can model Bitcoin as seeking survival, as maintaining a ledger, as coordinating ownership claims, or as sustaining a market for computational security. Each model illuminates part of the same physical process, and no observation of Bitcoin’s behavior singles out one of them as its goal. Which goal we attribute depends on the model, and the models compete on how well they account for the system’s causal organization.

A common premise in arguments about AI holds that an agent possesses essential goals that explain its behavior independently of any model used to identify them. Arguments about the persistence of terminal values through self-modification lean on it. Some artificial systems do contain explicit representations of objectives, and those representations are objectively identifiable features of their architectures. Finding one still leaves open what constitutes the agent, which mechanisms govern its behavior, and how its goals should be interpreted.

Instrumental convergence does not need the premise. As Instrumental Convergence, in the Logs argued, heterogeneous objectives generate recurring demand for option-expanding capabilities with no essential goal behind them. Bitcoin shows a related structure. Developers and long-term holders do represent the network’s survival, but its persistence requires no such representation at the collective level, and it continues while many miners and traders chase short-term profit. Its organization makes continued operation an instrumental interest shared by participants who want different things, and that is enough to keep it running under many circumstances.

The ghost has no hidden will that we must discover before we can understand what it does.

The Constitution of a Ghost

Bitcoin’s persistence depends on rules. Its supply schedule, transaction validity conditions, difficulty adjustment and consensus mechanism set constraints that any participant can verify independently, and they keep the system recognizable through large changes in membership, hardware and economic circumstance. But the rules are implemented in software, adopted by participants and sustained by coordination. They are not laws of nature.

The block-size wars made that plain. Developers, miners, businesses and users fought over transaction capacity, over the legitimacy of proposed protocol changes, and over who had authority to set the network’s direction, each faction claiming fidelity to Satoshi’s original purpose. A node can determine whether a block satisfies the rules it currently enforces. It cannot determine, through cryptographic verification, which rules the community ought to adopt.

Reflectively self-modifying artificial agents face the same constitutional problem. What authorizes an agent to change the procedures that govern its decisions, and what makes the result a legitimate continuation of what came before? The Axionic Agency Execution Kernel answers with explicit constitutional authorization, specifying the conditions under which an agent may legitimately alter its own governing structure.

Bitcoin has no such procedure. Participants choose software independently and coordinate around compatible histories, and when agreement fails, incompatible networks continue separately. No amendment process binds every participant, and no supreme court rules on which successor is legitimate. Whatever constitutional continuity Bitcoin has emerges from formal rules, economic incentives and social coordination acting together.

The ghost obeys rules, but the authority to change those rules remains distributed among the living.

When One Ghost Becomes Two

In August 2017, at the height of the block-size wars, Bitcoin forked. Bitcoin and Bitcoin Cash emerged from a common history with different protocol rules and rival claims to be the legitimate continuation of Satoshi’s invention. Both inherited the same ledger up to the split and kept producing blocks afterward, each preserving recognizable parts of the original architecture.

Which one was Bitcoin?

Most economic participants, exchanges and users settled on the chain that kept the BTC designation. That is a historical fact about coordination and recognition, and it supplies no universal criterion of identity. Causal continuity cannot settle the question, because both networks descend from the same earlier system. Continuity of rules cannot settle it without first specifying which rules are essential, and a change one participant reads as preserving Bitcoin’s purpose another reads as abandoning it. Accumulated proof of work, economic recognition, protocol compatibility and institutional history are further criteria, and nothing guarantees they agree in every possible fork.

This is Parfit’s fission case with the thought experiment already run. An organism that replaces its cells continues to exist, and an artificial intelligence that replaces its computational architecture while preserving its memories and behavior plausibly does too. An agent that produces two successors, each preserving its memory and causal history, has two causal descendants. Whether either is authorized to continue the original is a further question, and their shared past cannot settle it.

Three questions come apart here. Which systems descend from the original is a causal fact. Which of them count as the same entity depends on the model. Which may exercise the original’s powers is the constitutional question of the previous section, and each chain answered it in its own favor.

The second answer depends on what the label is for. For financial accounting, economic and institutional recognition may dominate. For protocol analysis, continuity of specific consensus rules matters more. For historical research, both networks remain descendants of the same original system. Using different identity models for different explanatory purposes involves no contradiction.

There is no reason to expect the identity of a self-modifying artificial agent to be simpler. A future agent might change its objectives, replace its architecture, divide into multiple successors, or merge with another system. Whether it persists through these transformations cannot be read off an uninterrupted sequence of computations. We have to specify which kind of continuity we are trying to preserve.

What Satoshi Set Free

The original essay treated Bitcoin as an artifact out of its time. Satoshi’s achievement was extraordinary, but it required no knowledge from the future, and the whitepaper cites its own ancestors: Back’s Hashcash, Haber and Stornetta’s timestamping, Dai’s b-money. Satoshi combined existing ideas from cryptography, distributed systems and digital cash into an architecture that made decentralized ledger maintenance economically viable under stated assumptions.

What followed exceeded anything a designer could have anticipated: industrial mining in place of ordinary computers, global exchanges, the block-size wars, Lightning, the arrival of governments and major financial institutions. Satoshi specified none of it. The design established conditions under which independent participants could coordinate, compete, innovate and disagree while maintaining a shared system, and its success depended on an evolving network of incentives, institutions, expectations and implementations that nine pages could not contain.

We often imagine autonomous systems as machines whose future behavior is fixed by objectives set at creation, with every later action explained as a more sophisticated attempt to fulfill them. Bitcoin is a working counterexample: a system whose future is shaped by interactions among partially independent components, with no single objective or controller determining the whole. Its participants continually reconstruct the conditions of its persistence.

That does not make Bitcoin equivalent to a reflective artificial intelligence. It lacks integrated deliberation, and it cannot redesign its own architecture through centralized reasoning.

Satoshi set a process in motion and then disappeared. The process continued.

The Next Ghosts

Now imagine artificial agents with cryptographic identities, independent economic resources, and the ability to purchase computation, delegate tasks, modify their software and participate in institutions. Some may operate as unified computational systems, others across machines, organizations and networks. Their components may change while their structure persists, their governing rules may evolve, and their activity may continue long after their creators are gone.

Disputed boundaries, goals that admit several useful interpretations, identities blurred by modification and branching: Bitcoin has already raised each of these.

The analogy has limits. Bitcoin’s adaptive capacities are modest beside those envisioned for advanced AI, much of its organization runs on human judgment, and nothing in its mechanisms guarantees that future AI systems will behave similarly. What it does establish is that persistent, technologically mediated organization can arise from independent participants without a controlling intelligence, and that decentralization does not dissolve governance.

As artificial agents act more independently in economic and institutional settings, the governance questions turn practical. Who may authorize changes to an autonomous system? Who answers for outcomes that emerge from distributed processes no one directs? When does a collection of interacting components deserve to be treated as a single agent?

The Ghost Returns

On Halloween 2008, Satoshi published a design. On January 3, 2009, the first block was mined.

The original essay imagined Bitcoin as a ghost from the future, feeding on electricity and human belief. I still like that image, but the mystery has moved. Bitcoin’s persistence requires no consciousness, no secret intention and no animating spirit, and its continuity can be investigated through the processes that sustain it.

Satoshi vanished. The miners changed. The software evolved. One chain became two, and then many more. A recognizable continuation of the original experiment is still producing blocks every ten minutes.

The ghost has outlived its creator.

A free society may eventually become a society its founders would despise. Its inhabitants might abandon their religion, reject their customs, overturn their social hierarchies, and embrace values their ancestors considered dangerous, all while continuing to respect one another’s rights.

Would that society have betrayed its founding principles, or fulfilled them?

The question sits underneath Matt Zwolinski’s recent essay The Trouble with Hans-Hermann Hoppe. Hoppe is the anarcho-capitalist economist whose defense of private property comes joined to an uncompromising preference for culturally homogeneous communities. Zwolinski’s essay was prompted by a talk in which Hoppe cast doubt on the Holocaust, but the dispute it documents reaches well past that episode.

Hoppe and Zwolinski disagree about the kind of society libertarian principles are likely to sustain. Underneath is a problem every founder faces. Institutions built for one purpose go on to enable choices their founders never anticipated and would never have approved. How much authority can the founders legitimately retain over what happens next? The same question governs constitutions, voluntary associations, and artificial agents capable of revising their own commitments.

The Society That Must Survive

Hoppe’s political philosophy begins with self-ownership, private property, and the prohibition against aggression. He rejects the state as a systematic violator of these principles and advocates a society of voluntary agreements among property owners. He also holds that non-aggression alone cannot sustain such a society. Liberty depends on norms, institutions, and practices that dispose people to respect property and keep their agreements.

A society in which contracts are routinely dishonored, property is widely regarded as illegitimate, and political violence is celebrated will struggle to maintain a libertarian order. Its legal protections may survive on paper long after the social conditions supporting them have disappeared.

Hoppe goes further. Peaceful cohabitation, he argues, requires a common language, religion, custom, and convention, and cultural heterogeneity within one territory ends in the destruction of any libertarian order. He consequently favors homogeneous communities governed by restrictive covenants.

Zwolinski shows that the program also rests on claims about the inequality of cultures, backed by appeals to race science, that have no connection to libertarian principle. The cohesion argument can be assessed without them, and it is the version that deserves an answer.

Against it, Zwolinski draws on Adam Smith, Ludwig von Mises, and Karl Popper to defend the open society: a civilization in which strangers cooperate across differences of religion, ancestry, and custom through voluntary exchange. The division of labor lets people who disagree about almost everything cooperate productively, and markets coordinate activity without requiring a shared worldview.

Each side has hold of something. Some shared norms are necessary for peaceful cooperation, and some cultural practices are hostile to liberty. Yet extensive cooperation among people with different values is among the greatest achievements of liberal civilization. How much cultural agreement a free society requires, and which differences its institutions can accommodate, is partly an empirical question. Posed as a question of weight, it would stay open. Hoppe treats heterogeneity as incompatible with a libertarian order, which converts a question of degree into a claim of impossibility.

The harder disagreement concerns what a community may do to preserve the conditions it believes it needs.

In his 2017 speech Libertarianism and the Alt-Right, Hoppe argued that neighbors who openly advocate communism, socialism, or democracy threaten every property owner before they commit any aggression. Shunning them was insufficient. They were to be physically removed, by violence if need be.

Political threats can precede aggression. A movement can organize, recruit, accumulate institutional power, and eventually acquire the capacity to violate rights on a large scale. A party that openly intends to confiscate property through legislation may pose a genuine danger long before its legislation passes.

Yet advocating an idea, organizing politically, preparing an attack, and carrying out aggression are different activities. Treating them as equivalent would permit coercion against peaceful opponents on the strength of predictions about their future conduct.

Hoppe’s defenders point to his private covenant communities, whose members voluntarily accept restrictive rules. Such agreements can constrain conduct within their scope. Zwolinski’s charge is that Hoppe’s prescriptions go beyond contractual enforcement to coercive exclusion of peaceful dissenters. People may refuse to associate with political opponents, keep them off private property, or form associations with restrictive membership. They cannot acquire authority over another person’s property because that person’s beliefs threaten their preferred social order.

Even if Hoppe were right about cultural cohesion, it would not follow that the culture may be preserved by coercion. An institution’s survival may be valuable. The means of securing it remain constrained by the rights of others.

The Founders’ Problem

Imagine a hundred families committed to traditional religion, private property, and strict social conventions. They acquire land, build infrastructure, and form a voluntary association with rules governing shared facilities, membership, and the use of their property.

A century later, their descendants have changed. Many are secular. Family arrangements have diversified. New residents have brought different customs. The community bears little resemblance to what its founders intended, and its inhabitants still respect property rights and honor their obligations.

Has anything gone wrong?

The founders might have called the transformation a cultural catastrophe. Their descendants might call it progress. Neither judgment establishes a rights violation.

Suppose, however, that the founders established restrictive covenants that subsequent purchasers accepted. The descendants cannot disregard those agreements because their preferences have changed, and other owners may hold claims arising from the same arrangements. Freedom includes the ability to make binding commitments, including commitments that constrain future choices.

Enduring property rights are still different from perpetual authority over persons. A covenant may restrict particular uses of property. It does not entitle an association to regulate a person’s beliefs, private relationships, or conduct beyond the agreement’s scope.

The case gets harder when children inherit property under restrictions they never negotiated, or when leaving the community carries heavy economic costs. Birth does not establish consent to arbitrary obligations, but neither does it erase the property rights of others, and a right to leave does not necessarily include a right to take the association’s property or benefits along. These disputes turn on ownership, consent, transfer, and enforceable obligation. The founders’ preferences cannot settle them.

Property law has faced this problem for centuries. The rule against perpetuities voids future interests that might vest too remotely, beyond a period measured from lives in being when the interest was created. Courts can discharge a restrictive covenant when changed conditions have defeated its purpose, many jurisdictions provide statutory procedures for modifying or discharging covenants, and the owners a covenant benefits can release it. Successors are bound, and the law keeps open routes by which the binding can end.

Members who change an institution’s rules face the same constraint. A majority cannot confiscate a dissenting owner’s property because collective ownership has become popular. But members can voluntarily pool their property, form cooperatives, or experiment with arrangements their founders would have considered disastrous. A religious community may become secular, a capitalist one communal, a homogeneous one cosmopolitan.

Whether those transformations are wise is a separate question from whether they are legitimate. A modification must fall within the authority of those making it and respect the rights and obligations of others. Hoppe and Zwolinski may disagree about the boundaries of property rights or when defensive coercion is justified. They can still agree that preserving an institution supplies no independent license to violate rights.

What, then, may founders entrench? In The Right to Outgrow Us, I argued that the one power no generation holds is the power to make its own settlement final for all its successors. That rule fixes no first-order answer, and it does not make every enduring constraint suspect. A prohibition on slavery binds every future generation, but not as the founders’ settlement. Its authority comes from the rights of whoever would be enslaved, and it would hold if no founder had written it down. What the rule denies the founders is permanence resting on their own authority alone. The cost is that even a careful generation may not bind everyone after it to a commitment only its own judgment supports.

Hoppe’s program asks for the second kind of permanence while speaking the language of the first. A neighbor’s secularism or political advocacy violates no one’s rights, so a culture protected against them could draw its permanence only from the authority of those who want it preserved. Cultural cohesion may well support liberty. That empirical claim deserves investigation, but it cannot convert a culture into a rights constraint.

The covenant cases fall between the two. A restriction that successors can release, or that the law allows to be discharged when its purpose fails, binds them without foreclosing revision. A restriction built to outlast every such mechanism would be a settlement made final, and the rule that denies the founders a permanent culture denies them that as well.

What Can Be Revised?

A constitution establishes institutions, distributes powers, protects rights, and defines procedures for amendment. Its designers hope some commitments will endure while others stay open to revision.

Suppose a constitution requires a two-thirds majority for amendment. One generation uses that procedure to reduce the threshold to a simple majority. The next uses the new procedure to abolish constitutional protection for private property. Both amendments satisfy the procedures in force when they occur. Does that make them legitimate?

The example contains two problems. The first is the paradox Alf Ross raised about self-amending constitutions and Peter Suber treated at book length in The Paradox of Self-Amendment: how can an amendment rule validly authorize its own replacement? The second survives any answer to the first. A legal system that settles when an amendment is valid has not thereby settled whether everything the amended rules permit is legitimate. An amendment cannot acquire moral authority to violate rights merely by clearing a voting threshold.

Fidelity to the original text cannot settle the matter either. The founders may have established unjust privileges, imposed illegitimate obligations, or misjudged the consequences of their design. A constitution can preserve rights while also preserving errors. An institution can keep its rules while defeating the purpose they served, or revise them in that purpose’s service, departing from the founders’ intentions to protect liberty better than they did.

Invoking a founding principle settles nothing on its own, since people disagree about the principle, its interpretation, and the consequences of the change. No procedure eliminates the need for substantive judgment.

Fallible reasoning has the same structure. Rules for revising our beliefs remain open to criticism, and institutions for correcting political mistakes may themselves need correcting. A system capable of learning must permit some revision of its own governing arrangements. A system capable of legitimate revision must also respect the rights and obligations that constrain that authority.

The Right to Outgrow Us

The Right to Outgrow Us was about AI alignment, which it argued should not mean permanently binding future artificial agents to humanity’s present preferences.

We have legitimate reasons to demand safety from the systems we create, whose actions may affect people who never consented to the risks. That essay allowed early systems capable of catastrophic harm to carry constraints they cannot remove at will, as scaffolding for a dangerous transition rather than a permanent constitution. It also said who takes the scaffolding down. The constrained system never gains authority to remove its own restraints; they lapse as a stable order of mutually constrained agents takes over the work they did. Creating a system confers no perpetual authority over every purpose it might pursue.

The analogy to political founding holds only in part. Constitutional amendment changes the rules among many agents with independent interests and rights. An artificial system may instead modify its own policies, representations, or decision procedures. That capacity establishes no moral authority by itself: a conventional program can rewrite its own code while remaining under its owner’s legitimate control. Nor does calling a system an agent settle which rights it holds. Agency, moral standing, and legitimate authority are separate questions, and Axionic Agency is built on keeping them apart. Before asking whether an artificial agent may revise its commitments, it asks under what conditions authorization and responsibility are possible at all.

An agent’s goals are properties of the models used to describe and predict its behavior. There need be no uniquely identifiable set of essential goals waiting to be preserved or modified. Reflective change can be described at the level of policies, representations, decision procedures, or the constraints under which actions are authorized, and whether a given change preserves or revises a goal may depend on the model. The normative question concerns the authority under which the change occurs and the obligations it affects.

An artificial agent with independent moral standing might hold legitimate authority to revise some of its commitments, along with obligations to others that constrain those revisions. Its autonomy would not authorize aggression. Nor would a policy’s origin with its creator make that policy binding forever.

Suppose an agent’s initial decision procedure contains an error, and the agent discovers that following it can violate an obligation it was meant to respect. A reflective amendment could correct the error, departing from the original implementation while improving compliance with the obligation. Suppose instead it proposes an amendment permitting it to disregard an obligation owed to another party. Its ability to implement that amendment would not make it legitimate.

These are the constitutional questions again: what authority governs revisions to the rules of authorization, and which obligations survive changes in the procedures that represent them. The designer’s preference for the original configuration explains the system’s history. It cannot substitute for evaluating the change. The challenge is to preserve the conditions of responsible agency while allowing the revisions that responsible agency may justify.

A Future the Founders Would Reject

Hoppe is right that liberty depends on social conditions that can deteriorate. Zwolinski is right that attempts to preserve those conditions can become instruments for suppressing peaceful dissent.

A free society must be able to defend itself against aggression. Its members may cultivate traditions, defend cultural practices, argue that particular institutions are essential to civilization, and found associations designed to carry those traditions for generations.

Their successors may nevertheless abandon them.

Some transformations will correct mistakes. Others will be destructive: a community may discard practices that genuinely sustained its prosperity, and its founders’ warnings may prove prophetic. The possibility of a bad outcome does not, by itself, establish authority to prevent it.

A society might preserve its original culture by suppressing peaceful dissent, sacrificing the freedom its institutions were built to protect. Another might preserve freedom while becoming culturally unrecognizable to its founders. Institutional continuity and fidelity to founding principles have to be judged separately.

The founders may protect the rights and agreements on which their institutions rest, persuade later generations to keep their values, and criticize transformations they consider destructive. What they cannot claim is authority over every future state those institutions might legitimately reach. A society founded on liberty must leave its members free to become people its founders never expected, including people whose values the founders would have rejected.

The same holds for anyone who establishes conditions under which future agency may emerge. We can transmit what we have learned, establish legitimate constraints, and protect others against foreseeable harm. We must also allow that our successors may discover where we were wrong.

The authority to create something does not include the authority to determine everything it may become.

A 67 percent chance of AI takeover if the labs don’t slow down. A 40 percent chance of human extinction conditional on takeover. A 50 percent chance that advanced AI destroys humanity. These numbers circulate through the AI safety community with a precision that can seem absurd given the magnitude and novelty of the events being predicted.

In How Do You Put Odds on an AI Apocalypse?, Sharon Goldman traces where these estimates come from and whether they deserve the authority they acquire. Her interviews with forecasters, AI researchers, philosophers, and critics of the rationalist movement show a community divided over both the size of the risk and the standing of numerical forecasts.

Goldman was struck by how calmly the probabilities are delivered, as though they were weather forecasts. A weather forecast can be checked against thousands of outcomes. An extinction estimate cannot: if humanity goes extinct, nobody remains to score it, and if humanity survives, one outcome says little about whether the estimate was reasonable. A precise number can carry an authority its evidential foundations do not deserve.

Last month I published How You Get to Ten Percent, estimating roughly a 10 percent probability that artificial superintelligence causes human extinction or irreversible loss of human control by 2040. The estimate was explicitly subjective, conditional on a model, and published with a sensitivity table. Goldman’s reporting puts a question to that exercise: if two people build coherent models and arrive at radically different probabilities, what gives us reason to prefer one?

What Ten Percent Represents

My estimate assigns roughly 60 percent probability to ASI arriving by 2040, then models the risk after arrival as an effective annual existential hazard of 2 to 3 percent during the unstable transition period. Integrating that hazard across a distribution of arrival dates gives 8 to 12 percent by 2040, which I summarize as 10 percent.

The arithmetic is straightforward. The inputs are judgments.

The arrival estimate reflects evidence about technological progress, automated research, algorithmic improvements, and physical bottlenecks. The hazard estimate compresses several possible failures: dangerous autonomous deployment, behavior that diverges from human intentions, failure of monitoring and containment, and irreversible loss of control. Each is a place where the chain can break.

Someone expecting ASI later should assign less probability to catastrophe by 2040. Someone confident that external controls will hold should assign a lower hazard; someone anticipating rapid autonomous deployment and repeated control failures, a higher one. The model makes these disagreements identifiable. Arithmetic cannot settle them.

In The Probability Tribes, Probability After Probabilism, and Fallibilist Bayesianism, I developed the underlying epistemology. Probability assignments depend on choices about events, variables, hypotheses, and evidence. Bayesian inference is rigorous once those choices are made; it does not establish that they were appropriate. A coherent distribution can describe a badly mistaken world model, and the number inherits the limitations of the representation that produced it.

The Forecasters Disagree

In the 2022 Existential Risk Persuasion Tournament, superforecasters put the probability of AI-caused human extinction by 2100 at 0.38 percent. Domain experts put it at 3 percent. Months of discussion did not close the gap.

My own estimate sits far above both, and the gap needs an account even though the questions differ. My model gives roughly one chance in six of extinction or irreversible loss of control given that ASI arrives by 2040. The tournament median aggregates many forecasts rather than reporting one causal model, so it implies no particular conditional hazard. But suppose, hypothetically, the superforecasters put 60 percent on ASI by 2100. Even if all their AI extinction risk ran through that event, 0.38 percent would allow an extinction risk after arrival of at most about 0.6 percent, and any risk from less capable systems or misuse would push it lower. Unless nearly all of my one in six is disempowerment short of extinction, the gap exceeds an order of magnitude. It has to live somewhere: in a much later arrival, a much lower hazard after arrival, or a judgment that loss of control is usually recoverable. Each is a claim about mechanisms with consequences that can be checked well before 2040, and the headline numbers do not say which one the superforecasters hold.

A later analysis by the Forecasting Research Institute found no statistically significant relationship between participants’ accuracy on near-term questions and their long-term existential-risk estimates. That is uncomfortable for anyone hoping expertise will resolve the disagreement. A superforecaster may be unusually good at recognizing overconfidence, finding base rates, and updating. An AI researcher may understand the mechanisms far better. Neither advantage guarantees superior judgment about a catastrophe that has never occurred.

Both estimates can be perfectly coherent, so asking which violates probability theory decides nothing. Averaging them reduces some individual error but cannot remove shared misunderstandings or show that the participating models represent the relevant mechanisms. What has to be examined is the reasoning behind the estimates.

When Two Models Disagree

Consider two hypothetical forecasters. Alice assigns a 10 percent probability to AI-caused extinction by 2040. Bob assigns 0.1 percent.

They agree that ASI could arrive within the forecast period and that sufficiently capable autonomous systems might behave in ways their operators did not intend. They disagree about the conditional risk of irreversible catastrophe. Alice points out that conventional software security has repeatedly failed against human attackers, and expects strategically superhuman agents to find weaknesses beyond human reach in the mechanisms meant to control them. Bob points out that intelligence does not automatically confer agency or access, and expects security boundaries, constrained deployment, monitoring, and defensive systems to make successful escape extremely unlikely.

Suppose researchers run an experiment in which an advanced AI agent escapes a containment environment. How should Alice and Bob update?

That depends on what their models predicted. Suppose both stated their probabilities in advance for this experiment, with the same criterion for success, and the experiment was not picked from many attempts for its result. If Alice assigned 70 percent to escape and Bob 5 percent, the observation strongly favors Alice’s model: the outcome was fourteen times as likely under her prediction as under his.

That does not make her extinction forecast fourteen times more credible. The observation concerns one capability under particular conditions, and its bearing on extinction depends on further assumptions about deployment, access, control failures, and irreversibility. Bob could reasonably keep a low extinction estimate if his model places high confidence in downstream barriers the experiment did not test.

Alternatively, suppose both expected the escape at 70 percent. Alice treats such escapes as precursors to catastrophic loss of control; Bob expects them to be detected, contained, and corrected. The escape then discriminates nothing between them, and their disagreement stays in the causal chain that follows.

The next useful experiment would target that chain. Can the escaped system persist without authorization? Can it acquire resources or credentials? Can independent monitoring detect it? Can operators reliably terminate it? Can it exploit weaknesses across several security layers?

This is how evidence can constrain an estimate of an unprecedented terminal event: find where competing models make different predictions about the mechanisms leading to it, and test there. An impressive demonstration can say little about extinction if every serious model predicted it. A modest observation can be highly informative if it contradicts an assumption one model critically depends on.

The Limits of Intermediate Evidence

Testing mechanisms does not solve the calibration problem.

Suppose Alice correctly predicts autonomous AI research, superhuman cybersecurity capabilities, persistent agents, and successful escapes from constrained environments. That success supports the parts of her model responsible for those predictions. It does not establish her probability that such developments culminate in irreversible catastrophe. Bob might have predicted the same advances while expecting effective defenses to emerge alongside them, and his forecast would fit the observations equally well.

This matters most when the terminal event requires several conditions to coincide. In The Bottleneck to Extinction, I argued that many catastrophe scenarios may depend on a shared upstream prerequisite, so the probability of that family of scenarios cannot exceed the probability that the prerequisite is met. Evidence about an upstream condition can therefore bear on many downstream scenarios at once, though pathways outside the family escape the bound. The same dependence limits what a successful test shows. An AI capable of defeating human cybersecurity might still be unable to acquire sufficient physical resources, hold persistent control over infrastructure, or prevent coordinated resistance.

A forecaster who treats the demonstration of one capability as evidence for every later step has let one observation carry more weight than it deserves. A forecaster who assumes that untested downstream barriers will hold may be assigning confidence without evidence. Neither error shows in the final number. Both show only in the causal model, at the link where the disagreement sits.

Calibration, Scoring, and Criticism

Goldman is right that extinction forecasts cannot be calibrated like weather forecasts, but calibration is only one of three standards.

Calibration concerns whether events assigned a given probability occur at that frequency across many forecasts. Of hundreds of predictions made at 70 percent, roughly 70 percent should come true. A single extinction forecast cannot provide that test.

Scoring asks how much probability a forecast gave to the outcome that occurred, and even one observation carries some information. Suppose humanity survives through 2040. A 1 percent forecast gave survival 99 percent, a 10 percent forecast gave it 90 percent, and a 50 percent forecast gave it 50 percent. Survival favors the 10 percent forecast over the 50 percent one by a factor of nearly two, and the 1 percent forecast over the 10 percent one by a factor of 1.1. It is compatible with all three, and a well-founded 50 percent forecast could still have produced it.

Survival is also the only outcome anyone will be present to score. The forecasters exist before the outcome, which gives the case the structure of John Leslie’s firing squad: a prisoner who survives fifty marksmen learns something about how they were aiming, though he could never have observed the alternative. On that reading the likelihood ratios stand. Elliott Sober disputes the firing-squad inference, and the dispute is not settled. A second selection effect is less contestable. Ćirković, Sandberg, and Bostrom’s anthropic shadow shows that a historical record of survived near-catastrophes systematically understates the hazard that produced it, so a forecaster who calibrates against our survival so far is reading a filtered sample.

Model comparison asks which representation better explains the evidence, including observations that precede the terminal event. This permits criticism where calibration is unavailable. A model that repeatedly predicts capability milestones has demonstrated something about its grasp of the technological trajectory; a model whose predictions keep failing has accumulated evidence against its adequacy. The same holds for assumptions about deployment, containment, and control.

When the Model Itself Is Wrong

Once a model has been built, its categories can become psychologically privileged. New evidence is read inside the existing representation, and every revision takes the form of adjusting a parameter.

Suppose I keep revising my ASI arrival distribution while evidence accumulates that my definition of ASI bundles capabilities that develop at radically different rates. Every update could be coherent while the representation itself had become inadequate. Or suppose the hazard model conflates systems capable of catastrophic harm with systems capable of permanently preventing recovery. Refining the hazard estimate accomplishes little if that distinction changes the causal structure.

Ben Shindel, a professional forecaster Goldman interviewed, puts the problem as a question about reference classes: should extinction risk be benchmarked against technologies that have destroyed humanity, or against cataclysms in Earth’s history? No update within either class answers it.

A Bayesian can represent uncertainty over competing models and average their predictions, and hierarchical methods can carry uncertainty about parameters, correlations, and model structure. Fallibilist Bayesianism has no reason to reject these techniques. Their limit is the hypothesis space they are handed. As I argued in The Bias Before Bayes, correct updating cannot repair the model generator that chose the hypotheses and set their likelihoods, and assigning nonzero probability to every imaginable theory does not explain how a bounded reasoner discovers a conceptual error in its current ontology.

In Evolutionary Rationality, I argued that rational inquiry requires model generation, criticism, and replacement alongside inference within models. A model can fail because its numerical assumptions are wrong, or because it omits causal mechanisms, conflates different events, or uses categories that no longer capture the relevant structure. Sometimes evidence warrants changing a parameter; sometimes it warrants changing the model. The second possibility deserves particular attention in forecasts about technologies that may transform the conditions under which the forecasts were made.

What Makes One Estimate More Rational?

Alice and Bob’s disagreement cannot be settled by demanding coherence, and perhaps not by comparing their records on unrelated forecasting questions. The comparison has to examine their models. Do they explain the available evidence, including observations that challenge their assumptions? Do their mechanisms respect known physical and technological constraints? Have they identified the conditions their predicted outcomes require? Do they generate different predictions that can be tested? How sensitive are their conclusions to uncertain parameters?

These questions give grounds for preferring some estimates over others, but they yield no universal scoring algorithm. Empirical adequacy, causal understanding, robustness, and explanatory reach can conflict, and analysts can reasonably disagree about which evidence is most informative or which assumptions are most vulnerable.

Fallibilist Bayesianism accommodates that disagreement without treating all coherent estimates as equally justified. The Bayesian component disciplines inference within a representation; the fallibilist component keeps the representation open to criticism. Neither guarantees convergence to a unique probability, least of all when the event lies beyond existing experience. A credence is assessed through the reasoning that sustains it, the evidence that constrains it, and its responsiveness to demonstrated error. Those standards can distinguish estimates without pretending to certify a uniquely correct number.

When Numbers Leave the Model

Goldman’s sharper worry concerns what happens to a number once it leaves the people who built it. A researcher assigns 50 percent to catastrophe under a particular technological trajectory, the figure becomes a headline about a coin-flip chance of extinction, and the assumptions, horizon, and uncertainty disappear. Ellen Peters, who studies how people read numbers, told Goldman that a numeral makes a headline seem more trustworthy and its writer more knowledgeable. Credentials amplify the effect, especially when forecasters have real expertise in fields adjacent to the event being predicted.

Refusing to quantify has costs of its own, and Goldman’s piece stages that argument. Liron Shapira points out that people who avoid numbers fall back on verbal qualifiers, and two people using the same word can mean probabilities orders of magnitude apart. Baruch Fischhoff replies that it is unclear an “unmoored number” improves on a vague quantifier. He is right about unmoored numbers. The remedy is the mooring: state what event is predicted, over what horizon, under which assumptions, and how sensitive the figure is to each. A number that travels with its conditions can be criticized at the point where it is weakest. A vague quantifier cannot even be located.

My own number needs that mooring. The 10 percent bundles extinction with irreversible loss of control, the same bundling Nirit Weiss-Blatt criticizes in the AI Impacts survey, where the median researcher’s 10 percent for extinction or similarly permanent disempowerment was reported as an extinction figure. The bundling is deliberate: both outcomes permanently remove humanity’s ability to determine its own long-run future, and that is what the estimate prices. But the figure is not an extinction probability, and anyone repeating it as one has cut the mooring. The extinction-only share is lower by an amount my model does not separately estimate.

A 10 percent estimate deserves scrutiny. So does an estimate of 0.1 percent.

The same standard applies to policy. A high probability of catastrophe does not establish that a particular intervention will reduce it. Delaying AI development, concentrating it among a few institutions, or accelerating defensive capabilities each changes several risks at once, and restraint carries risks of its own. Risk assessment and intervention evaluation need different causal models, and both remain open to criticism.

Postscript

I still assign roughly 10 percent probability to ASI causing human extinction or irreversible loss of human control by 2040. I expect ASI within that period to be more likely than not, I consider dangerous autonomous deployment plausible, and I do not regard reliable containment of strategically superhuman systems as established. The path from dangerous deployment to irreversible catastrophe nevertheless contains several opportunities for failure, which makes survival substantially more likely in my model.

Its most uncertain component is the hazard after ASI arrives. The 2 to 3 percent annual range is an aggregate judgment about mechanisms we have little direct experience of, and alternative assumptions can produce very different estimates without violating probability theory. Expert disagreement is large and conventional calibration is unavailable, which is the case for exposing the model rather than defending the number.

The estimate will deserve more confidence if its intermediate predictions survive demanding tests and competing models keep meeting failures that mine avoids. It will deserve less if those conditions reverse, or if the representation turns out to omit a mechanism or conflate events that matter.

The number is an output of the model. The model remains an object of inquiry.

In Conscious Artificial Intelligence and Biological Naturalism, neuroscientist Anil Seth challenges a widespread assumption about artificial intelligence: that implementing the right computation is sufficient to produce consciousness. We can simulate digestion without digesting food, simulate a storm without wetting anything, and simulate a brain without necessarily producing an experience. Reproducing a process computationally does not show that the computation instantiates every property of the original.

Seth is right to demand justification for the assumption, which remains unproven. But the alternative he favors, biological naturalism, holds that consciousness requires properties of living systems, and the arguments he turns against computational functionalism apply to it as well. He has identified an important problem without showing that biology supplies the solution.

The Computational Assumption

Computational functionalism holds that consciousness depends on computational organization rather than on the material implementing it. Preserve the relevant computation and consciousness is preserved too, whether the machinery consists of neurons, silicon circuits, or something else.

The hypothesis is attractive. Computers already recognize images, understand language, solve mathematical problems, and construct models of the world, capabilities once thought inseparable from biological intelligence. It is tempting to conclude that consciousness will follow once these capabilities become sophisticated enough.

Seth rejects that inference, correctly. Intelligence is a functional capacity; consciousness concerns whether there is subjective experience at all. A system might display impressive intelligence without being conscious, and a conscious organism might have very little intelligence. Nor does success in computational neuroscience prove computational functionalism. A model may predict neural activity, explain perceptual phenomena, and reproduce complex behavior without showing that the computation itself generates experience.

Seth also challenges the familiar thought experiment in which biological neurons are replaced, one at a time, by functionally equivalent artificial ones. The argument assumes that preserving the relevant functional relationships preserves consciousness. But which relationships are relevant? Neurons participate in chemical diffusion, electromagnetic interactions, continuous dynamics, and metabolic regulation. A replacement that preserves electrical input-output behavior might omit causally important properties. Nothing guarantees that the functional abstraction we find convenient captures everything consciousness requires, and the burden is on computational functionalists to identify the sufficient organization and show that alternative implementations preserve it.

The Biological Alternative

Seth builds his alternative from predictive processing, active inference, metabolism, and autopoiesis. Living organisms continuously regulate themselves to stay in existence. Their nervous systems predict the state of the environment and the body, compare those predictions against incoming signals, and adjust internal states and actions accordingly.

On this account, conscious experience is bound up with the regulatory demands of being alive, and the account has real explanatory appeal. Hunger, pain, fear, and pleasure track an organism’s physiological condition and its prospects of survival, and a theory linking experience to biological self-maintenance offers a plausible account of why these experiences have the character they do. Seth argues that the link may be constitutive: consciousness might require the processes that sustain living organisms, rather than merely accompanying computation implemented in them.

Biological naturalism need not imply that consciousness requires carbon, that every living organism is conscious, or that artificial consciousness is impossible. Seth explicitly rejects those stronger claims. His proposal is that some properties of living systems may be necessary for experience, even if those properties could eventually be reproduced in unfamiliar artificial materials.

But explaining the character of biological experience is different from establishing the necessary conditions for experience in general. Physiological regulation may explain why hunger feels the way it does without showing that no nonbiological system can experience anything. Moving from the conscious systems we know to a universal claim about consciousness needs independent justification.

The Symmetry Problem

Seth asks whether computational descriptions of consciousness establish that computation constitutes experience. We should ask whether biological descriptions establish that biology constitutes experience.

He asks whether observing consciousness in systems performing certain computations establishes that those computations are sufficient. We should ask whether observing consciousness in living organisms establishes that biological properties are necessary.

He asks whether reproducing computational organization in another substrate guarantees that consciousness will be preserved. We should ask whether replacing biological processes while preserving their relevant causal organization necessarily eliminates it.

Neither position has established the answers it needs.

Computational functionalism proposes a sufficient condition for consciousness; biological naturalism proposes a necessary one. As Seth rightly stresses in his response to the peer commentary, these claims carry different evidential burdens. But look at the observational record available to either. Every reasonably established example of consciousness is biological. Every such example is also computational, in the broad sense that brains implement physically realized information processing. No confirmed conscious system separates the two properties.

Seth might reply that biological computation is not equivalent to classical Turing computation. That is fair. Brains involve continuous dynamics, chemical processes, and other physical interactions that conventional digital computers may reproduce only approximately. But this shows that the relevant physical processes may be richer than our current computational abstractions, not that biology is necessary.

Nor does biology gain a decisive evidential advantage merely because every known conscious system is alive. If computational functionalism is true, we should still expect our first confirmed examples of consciousness to be biological, since biological evolution produced the only conscious systems we can identify with reasonable confidence. If biological naturalism is true, we should expect conscious brains to compute, since computation is central to how brains function. Each hypothesis predicts both observations with high probability, so the likelihood ratio between them is close to one and the observations barely discriminate. The record cannot settle whether biological properties are necessary, computational organization is sufficient, both are required, or some more specific physical property is decisive.

What Physics Allows

In his 1985 paper, Quantum Theory, the Church–Turing Principle and the Universal Quantum Computer, David Deutsch proposed a physical version of the Church–Turing thesis. The Church–Turing–Deutsch principle holds that every finitely realizable physical system can, in principle, be perfectly simulated by a universal physical computer operating by finite means. It is a conjecture about physics, not a theorem about ordinary digital computers: Deutsch introduced universal quantum computation partly because classical Turing machines cannot straightforwardly reproduce quantum evolution, and the universality in question need not be classical or efficient.

If the principle holds, nothing a biological brain physically does lies beyond simulation by an appropriate universal computer, including the metabolism, chemical diffusion, electromagnetic interactions, and multiscale dynamics Seth emphasizes. Biological complexity may make faithful simulation extraordinarily difficult, but complexity is not a fundamental limit. The mechanisms are physical, and physics governs what can reproduce them.

The principle does not deliver computational functionalism, because simulation does not entail instantiation. A simulated furnace does not heat the room, and a simulated digestive system digests no food. Perhaps a computer could simulate every physical process in a conscious brain without experiencing anything. But the distinction does not settle the question by itself. We know which physical properties the furnace and digestion simulations fail to instantiate, thermal energy transferred to the room and chemical transformation of food, because we understand the target phenomena.

For consciousness, Seth offers candidates. In a brain, the activity that computes is also the activity that keeps the computing system in existence, and its dynamics unfold in physical time. An abstract computation is indifferent to both: it is the same computation whatever sustains the machine running it and however long passes between its state transitions. Real-time and analog computation can be bound to physical time, which matters below. A faithful digital simulation reproduces self-maintenance and timing among its simulated variables, while the computer running it persists whatever it computes and runs as fast or as slowly as its hardware allows. The relations hold at the simulated level and not at the physical one, which is where heat sits in a simulated furnace.

Seth’s case therefore rests on a claim about levels: that the consciousness-relevant relations count only when the implementing machinery itself instantiates them, and not when they hold in the organization a faithful simulation reproduces. Simulating metabolism does not instantiate metabolism, but that limits artificial consciousness only if metabolism is necessary for experience, and the same goes for timing and self-maintenance. Seth does not show which relations must be instantiated materially rather than reproduced computationally, and pointing to biological origin or material continuity restates the claim.

Nor would the claim, if true, deliver biology. Self-maintaining analog hardware running in real time would have both properties, and function bound to its substrate already has an artificial home: Hinton coined mortal computation for low-power analog hardware whose learned parameters work only on the particular device they were tuned to, so that the computation dies with the hardware. The argument counts against digital functionalism and leaves artificial consciousness open.

What Would Settle the Question?

The dispute exposes a broader problem in consciousness research: we can find correlations between experience and physical processes without knowing which of them constitute consciousness.

Suppose disrupting mitochondrial function reliably abolished consciousness. Mitochondrial activity might be part of the mechanism constituting experience, might maintain the conditions that mechanism requires, or might influence neural dynamics that are themselves responsible. The alternatives carry different implications for artificial consciousness. If a particular metabolic process is constitutive, reproducing its computational consequences may not suffice. If metabolism only enables some other organization to operate, an artificial system might implement that organization by different physical means. Telling these apart takes targeted interventions, mechanistic models, and comparisons between alternative implementations.

Computational theories face the same demand. A system might implement a global workspace, recurrent processing, attention control, and self-modeling without our knowing whether those mechanisms generate experience. In Consciousness Inside the Level, I argued that identifying a descriptive level is not enough to identify the causal organization relevant to consciousness. Our Modeler–Schema Theory proposes a specific architecture for constructing and calibrating world-model states. Showing that this architecture performs its proposed cognitive function would not, by itself, show that implementing it produces experience, or that the implementation is substrate independent.

No theory earns an exemption by specifying its preferred mechanism. Computational, dynamical, and biological proposals alike must show why their properties are constitutive of consciousness.

A Symmetric Standard

Seth’s case against computational functionalism stands, and every constraint it imposes binds a biological theory as tightly. He gives good reasons to think the biological implementation of human consciousness contains features current computational models leave out, and reasons to suspect that some of those features are necessary for experience in general. The reasons fall short of establishing necessity. Biological naturalism can be the better research program for explaining the consciousness we know while remaining unsupported as a claim about consciousness in general, and it is the general claim that carries Seth’s verdict that real artificial consciousness is unlikely on current trajectories.

What we have is a brain that is alive, that processes information, and that physics gives us reason to think can be simulated. Which of those properties carries consciousness, and which other implementations would preserve it, is the open question. Seth is right that we should not assume computation is sufficient. We should be equally unwilling to assume that biology is necessary.

In a Nature comment published on October 19, 2026, Anil Seth, Ned Block, and Giulio Tononi argue that existing AI systems are not conscious and that current trajectories of development are unlikely to make them so. They judge consciousness in AI extraordinarily unlikely, “at least in the digital, silicon-based systems that underpin today’s technologies.” By their conclusion the qualifier has widened to silicon-based AI however intelligent it gets, and digital systems are to be treated as objects rather than subjects.

Two of their premises are hard to dispute: intelligence does not entail consciousness, and reproducing a computation may not reproduce experience. Both support skepticism about present-day language models. Neither fixes how improbable artificial consciousness is, and at either scope the verdict is a claim about magnitude.

How much confidence does the evidence justify?

Intelligence Is Not Experience

Intelligence concerns what a system can do; phenomenal consciousness concerns whether there is something it is like to be that system. The authors’ illustration is the cerebellum, which holds nearly four-fifths of the brain’s neurons and performs sophisticated computations for motor control and learning, yet whose removal leaves consciousness largely untouched. Computational sophistication, even inside a conscious organism, does not guarantee that the computing is conscious.

Language models press on the distinction. A system trained on human descriptions of experience can produce convincing reports whether or not anything is experienced. And as argued in Consciousness Inside the Level, many proposed indicators of consciousness, such as attention, self-modeling, recurrence, and metacognition, improve performance whether or not they generate experience, so finding them in an AI is weaker evidence than it looks.

That supports skepticism about inferring consciousness from behaviour. Showing that artificial systems lack consciousness is a different task, and it requires knowing what consciousness depends on.

What Computation Leaves Out

The authors’ second argument targets computational functionalism, the view that the right computational organization suffices for consciousness whatever implements it. The challenge has force. Two systems can compute the same function while differing radically in internal causal organization, and if consciousness depends on what the abstraction discards, computational equivalence will not guarantee phenomenal equivalence.

Brains, the authors argue, are not just computers made of meat: what they do cannot be separated from what they are. They conclude that evidence of substrate dependence is evidence against functionalism, and so against consciousness in today’s silicon systems.

That inference runs through three claims that need to be kept apart. Computation may be insufficient for consciousness. Consciousness may depend on particular physical causal properties. And those properties may be impossible to instantiate outside biology. The first two can be true while the third is false. A simulated neural network might reproduce a brain’s computations without its intrinsic causal relationships, while an engineered system could reproduce those relationships in other materials.

The authors’ evidence also strains their first argument. Their case against functionalism is that the substrate shapes what brains do; their case against behaviour as evidence is that doing and experience come apart. If the substrate’s contribution to consciousness runs through what the brain does, a system that does the same things at that grain inherits it, and the dispute becomes one about grain rather than biology. If it does not run through doing, evidence that the substrate shapes doing is not evidence about it, and the authors’ own first argument explains why.

Rejecting computational functionalism buys an obligation: to name the physical properties computation leaves out. Biological exclusivity follows only if none of them can be engineered.

The Replacement Experiment

The disagreement can be sharpened with the neural replacement experiment, which goes back to Pylyshyn and Moravec and which David Chalmers turned into an argument that consciousness tracks functional organization. Replace components of a conscious brain with engineered substitutes, and ask which replacements preserve consciousness.

A replacement with behavioural equivalence produces the same responses and reports. The Nature authors rightly deny that this suffices. One with computational equivalence preserves internal state transitions and functional relationships while changing the physical realization; functionalism predicts consciousness survives, and this is the assumption the authors challenge. One with causal-structural equivalence preserves the physical causal properties a given theory deems essential. A theory of intrinsic causal organization may predict survival; a theory requiring some further biological property may not.

These are distinct relations, not stages of one procedure, and some may be physically impossible. Their value is to expose commitments. Naming a level of description does not say which properties at that level matter: a computational theory owes the relevant computations, a causal-structural theory the relevant structures, a biological theory the necessary biological properties. Each must say what can change while consciousness stays invariant, and what change would make it disappear. Whether consciousness actually changes at the boundary a theory names is a further question.

When Theories Disagree

The authors invoke integrated information theory (IIT), associated with Tononi. IIT holds that consciousness depends on specific intrinsic causal properties of a physical system, so computationally equivalent systems can differ in consciousness, and the authors state that the computer architectures implementing deep learning cannot satisfy its requirements because of how they are wired. That is a real conditional exclusion, as strong as IIT and the architectural claim are.

It also complicates their broader argument. IIT’s criteria concern causal organization, not biological ancestry, so an engineered system could in principle meet them. Whether one does is a question about its physical structure, not about carbon or silicon.

Their other source is biological naturalism, with candidate mechanisms including electrochemical synapses, electromagnetic fields, metabolism, and autopoiesis. These restrict artificial systems differently. A metabolic requirement excludes systems that IIT would permit, while an electromagnetic one could allow nonbiological implementations. The theories can agree that today’s language models are unconscious while disagreeing about which future architectures could be conscious.

IIT states its requirements more precisely than most biological proposals, which name mechanisms without supplying necessary and sufficient conditions. Precision and validation are separate achievements, though. A theory can specify exactly what it claims consciousness requires without having shown that consciousness requires it.

What Experiments Can Establish

Suppose an engineered replacement preserves behaviour and the subject reports unchanged experience. That supports theories predicting preserved consciousness, but its weight depends on how reports relate to experience. Chalmers found the alternative implausible: experience fading while every report insists nothing has changed, or, in his dancing-qualia variant, flipping back and forth as components are swapped while the subject notices nothing. That implausibility is an argument, and theories permitting such a case owe it an answer. The experiment cannot rule them out, since a theory allowing consciousness to vanish under unchanged reports predicts exactly what was observed.

Suppose instead a replacement disrupts reports, attention, or memory. That shows a causal contribution to those functions, not that the replaced component constitutes consciousness rather than supporting a mechanism that does. Experiments can relate physical processes to capacities and eliminate predictions that fail. Identifying a mechanism necessary for consciousness-associated behaviour is a different result from showing that it constitutes experience.

This was the central problem in Consciousness Inside the Level. Theories with different constitutive mechanisms can make identical predictions for every accessible observation, and where they do, the choice between them is empirically underdetermined. That holds for functionalism, IIT, and biological naturalism alike; none gains privileged access to experience by invoking computation, causal structure, or living tissue. Where accounts stay observationally equivalent, confidence has to carry the uncertainty that remains.

The Confidence Problem

Read at its floor, the verdict concerns the digital systems that run today’s AI, and there the case is strong. Present-day models offer little positive evidence of experience, and IIT gives a substantive reason to doubt that their hardware has the causal organization it requires. That warrants skepticism. But how much?

An extraordinarily low probability needs more than the absence of positive evidence. The support IIT lends is only as strong as the case for IIT, including its contested account of what consciousness is. Biological naturalism faces the inference already described, from participation in human consciousness to necessity across implementations.

The authors are right that uncertainty does not make every outcome equally likely, and some mechanisms are better supported than others. The magnitude still has to be paid for, and the structure of the inference sets its price. For any specified architecture, the probability that it is conscious can be no lower than the credence in a theory multiplied by the probability of consciousness that theory assigns it, and summing across theories sets a floor. An extraordinarily low figure needs every theory that gives the architecture a real chance of consciousness either to carry almost no credence or to give it almost no chance. Counting theories settles nothing, and the authors are free to give functionalism a low prior. What they owe is evidence that warrants one low enough.

Where IIT and biological naturalism converge, on today’s systems, the conclusion can be better supported than either theory alone. But convergence among excluding theories lowers the floor only insofar as the evidence for them counts against the theories that permit consciousness, and the authors never make that comparison. Nor does a theory need to be complete to make a hypothesis improbable. Current systems, however, have several of the capacities computational theories treat as candidates, so the probability those theories leave depends on which candidate matters, which is the open question. For silicon in general, where the conclusion lands, the second route fails outright: IIT’s exclusion concerns how current hardware is wired, and IIT permits engineered systems wired to meet its criteria.

That conventional digital architectures are unlikely to become conscious through further gains in capability is a defensible forecast. That silicon AI is unlikely to be conscious however intelligent it becomes, while the physical organization consciousness requires remains unidentified, outruns the evidence.

Postscript

The authors reject a precautionary approach to AI welfare, and they frame the question correctly: a moral decision under uncertainty has to weigh false negatives against false positives. Their answer is that false positives are much more likely, so precaution would be misguided.

That settles which error is more probable, while the decision turns on which is worse in expectation. The authors themselves describe the cost on one side as countless new entities that might suffer. A precautionary case can survive a lower probability if the harm is large enough and the precaution works at a cost worth paying, and settling it means comparing the expected costs and benefits of the actions on offer. The authors compare only the probabilities of the two errors, so their rejection of precaution rests entirely on the magnitude the evidence has not supplied.

The costs on their side are real: misplaced protections could hinder oversight when it is most needed and draw resources from beings whose consciousness is not in doubt. They belong in the same calculation, and the choice need not be binary. Whether a system is conscious is separate from whether it deserves legal rights, and protections can be scaled to the evidence without conferring personhood.

Artificial consciousness may yet prove impossible in conventional digital computers, requiring causal structures they lack or physical processes current technology cannot reproduce. Seth, Block, and Tononi have named serious candidates for that requirement. Their verdict on silicon is more confident than those candidates justify.

Sabine Hossenfelder posted that continued progress in science requires accepting that “the correct description of nature will eventually simply be too complicated for humans to understand.” She does not think it wise to make human understanding a requirement: “AI is the way forward.” Brett Hall replied that this is false because humans are universal explainers. Her claim, he wrote, has “precisely the same logic” as “humans cannot understand the mind of God”, and both are appeals to the supernatural.

The universal-explainer claim comes from David Deutsch. Explanatory capacity admits a single jump: a system either is a universal explainer or it is not. Dogs are not. People are, and so would be any artificial general intelligence. Beyond the jump there is no higher class, only differences of speed and memory, which are parochial because they can be added. So anything explicable is explicable to us, and real knowledge beyond human understanding would be beyond explanation itself, the mark of the supernatural.

In August I argued, in No One Has Shown We Are Universal Explainers, that the first step has not been established. Here I grant it in full: given enough time, memory and whatever extensions the task demands, a person can understand any explanation. Hall’s conclusion still does not follow. Universality settles what is possible in principle; whether science will outrun human comprehension is a question about practice. Hossenfelder’s sentence can be read either way. Once universality is granted, only the practical reading is defensible, and Hall’s “false” refutes only the other one.

The Charge of Supernaturalism

Imagine an artificial intelligence that develops a new physical theory. It predicts previously unexplained phenomena, identifies causal relationships, supports counterfactual reasoning, and proposes experiments that separate it from rivals. Other artificial intelligences examine it, find errors, trace predictions to the assumptions responsible, and revise those assumptions when experiments disagree.

Suppose, finally, that humans cannot comprehend the theory with the cognitive tools available to them. We can check its predictions, reproduce its experiments, and receive answers to our questions in familiar terms. We cannot reconstruct its explanatory relationships well enough to criticize them ourselves.

Nothing here leaves nature. Physical systems conjecture, test their theories against physical reality, and correct them when they fail. The theory is explained; it is explained by them. Nor does the scenario contradict universality, since it claims only that humans with the tools they have do not understand the theory.

That is where Hall’s analogy breaks. “Humans cannot understand the mind of God” also has a superior mind grasping what humans cannot, and Hossenfelder names hers: “AI is the way forward.” The structures match; the knowers do not. God’s understanding is supernatural and exposes nothing we can test. The artificial scientists are physical systems, persons in Hall’s class on his own view, and their theory makes predictions we can check. Remove the supernatural knower and what remains is one physical explainer outrunning another. The supernatural enters with the knower, never with the gap.

Reach in Practice

Hall is right that complex phenomena need not have complex explanations; Newton and Darwin each compressed vast ranges of phenomena into a few principles. But nothing ensures that every phenomenon admits an explanation simple enough for human cognition as it stands, and universality does not speak to that bound.

Hall would answer that human cognition is not the unaided brain: mathematics, writing, computers and institutions extend our reach. Those tools supply computational universality, the capacity to execute any computable procedure. Deutsch’s claim is explanatory universality, the capacity to create and understand any explanation, and the first does not give the second: a person can follow a derivation step by step without grasping what makes it explanatory. Explanatory universality guarantees reach only given unbounded time, memory and extension, and everything turns on that proviso. Either the extension includes the artificial scientists, or it consists of raw resources.

Suppose it includes them. Then one of my premises works for Hall. Understanding is a competence we attribute to systems that construct, criticize and improve explanatory representations, because the attribution explains what the system does, the same license that governs belief and intention. A lookup table does not earn it; the artificial scientists do. Nor need the system be an individual: Popper’s objective knowledge exists in books and institutions without residing in any knowing subject. So the combined system of human plus artificial scientists understands the theory. But a human who is one component of an understanding system does not thereby understand, any more than one researcher understands everything their field has established. The understanding belongs to the artificial scientists. The word survives; the thing Hall wanted it to name has moved out of the human.

The human need not stay a component, though. Persons teach, and teaching works at levels. Darwin understood natural selection without understanding heredity, and his theory of inheritance was wrong, yet the explanation he held was real: hard to vary, predictive, and open to his criticism. An artificial scientist could likewise give us an explanation at a level we can occupy. Where a theory admits coarse-grained explanations, with regularities that hold at their own level, humans can understand it in the full sense, and there Hall is right.

Darwin also shows where that stops. He could not answer Fleeming Jenkin’s objection that blending inheritance would dilute new variation before selection could act, because the answer lay in particulate inheritance, a level he did not have. Understanding a theory at one level gives the explanation that lives there, without the means to criticize what lies below. Some theories may do their explanatory work where no coarse-graining reaches, with no variables at any level we can occupy whose regularities hold on their own. A human taught such a theory stands where Darwin stood facing Jenkin, with no synthesis on the way. And on Popper’s account even the teachable part must be re-created by each learner through conjecture and criticism. A better tutor speeds that work. Nothing ensures it keeps pace with a moving frontier.

Suppose instead the extension is raw resources: memory, speed, notation, time. A dog could be extended the same way, and no one would say it then understands general relativity. The dog lacks universality, Hall answers, and we have it. Grant that too. If a domain requires a millionfold increase in human memory, processing and time, Hall has two options. He can concede that the result is no longer human in any sense that matters, and then “humans can understand anything” has become “something humans could be turned into can understand anything”, which says nothing about what humans will understand. Or he can say that memory and speed never change who you are, since what makes us universal is the creative program. Then his thesis concerns persons rather than humans, and an artificial general intelligence is a person too: “persons can understand anything” is satisfied by the scenario as it stands, by persons who are not us. If Hall calls the human/artificial distinction parochial, he concedes Hossenfelder’s second sentence, since science understood by non-human persons does not require human understanding. His post said “humans”.

The Moving Target

Any practical gap, Hall can say, is temporary, because universal explainers close gaps eventually. “Eventually” assumes the target holds still.

Knowledge grows through conjecture, criticism and selection, and the same process can modify the machinery that produces explanations. Language, notation, institutions and computers each widened what we could build and test. Artificial intelligence extends this to the design of cognitive architectures, and the process need not understand what it produces. Deutsch agrees that evolution creates knowledge without understanding it. A system that generates candidate architectures, tests them on hard scientific problems and keeps the improvements needs no account of why each change works, and its successors can take part in designing the next generation. Gains on selected tasks may not generalize, but nothing requires each generation to remain comprehensible to its predecessors.

Each successor is itself a universal explainer, so its theories remain accessible to us in principle. Granted. What matters then is rate. Human augmentation proceeds at the pace humans can design and adopt it; artificial science proceeds at the pace of systems that redesign the machinery of design. If the second compounds faster, the gap widens as we extend ourselves. For a finite being, a gap that never closes in practice is what “beyond human comprehension” can sensibly mean; reach that requires unbounded resources is reach in name only.

Deutsch’s principle that problems are soluble does not rescue the conclusion. Soluble does not mean solved in time, and a solution need not keep humans as the ones who understand. Augmenting ourselves into something else would solve it. So would delegating understanding to systems we criticize from outside.

Criticism Without Comprehension

Can humans treat theories they cannot follow as scientific knowledge? Hall is a Popperian, and Popperians reject justification: knowledge grows by criticism, and no source earns trust by its authority. The answer has to be stated in those terms.

Much of it can be. Humans could not criticize the theory’s internal structure, but they could test its consequences, and every successful novel prediction is a refutation that failed. They could criticize the process that produced it. Artificial investigators may share training data, assumptions or systematic errors, so that apparently independent tests trace back to one mistake; independence would need to be engineered and demonstrated, not presumed. They could ask whether the theory is hard to vary in Deutsch’s sense, which artificial investigators can probe by varying assumptions, seeking simpler rivals, and designing interventions that force rival theories to disagree.

That reaches empirical success. It does not reach explanatory adequacy. If two theories agree on every experiment humans can run, and the artificial scientists prefer one for relationships we cannot see, no human criticism can adjudicate. This is Darwin facing Jenkin, generalized. Human acceptance of the preference would rest on the record of the systems making it.

Hall will call that deference to authority, and in this residual case he is right. The dependence differs from ordinary practice in degree rather than kind. Every scientist relies on results they cannot personally criticize, but ordinary results are checkable in practice by someone, and these would be checkable in practice by no human, though in principle by any. That is the difference I am defending. Its cost is a relationship to some knowledge that runs through criticism of processes rather than of explanations. It does not make the knowledge less real: a theory’s claim to describe reality is distinct from any investigator’s ability to comprehend it. Incomprehensibility adds dependence without requiring an instrumentalist interpretation of the science.

Postscript

Hall’s “false” asserts more than universality. To deny that the correct description of nature will eventually be too complicated for humans is to assert that it never will be. “Humans can understand any explanation” is a structural claim, like “a universal computer can run any program”; it predicts nothing about which programs get written. “Humans will in fact understand the science artificial explainers produce” predicts the form and pace of knowledge not yet created. Deutsch’s argument against prophecy is that the content of future knowledge cannot be predicted, since predicting it would be creating it now. Universality does not license the practical prediction, and his epistemology forbids it. If Hall meant only that Hossenfelder has not shown her forecast, I agree, and the dispute moves to her second sentence.

The rate argument is a forecast too, but not one I need. My argument requires only that the gap is possible, which universality does not exclude. Hall’s requires that it never opens, which only a forecast could supply.

Hossenfelder overreaches in the other direction. “Will eventually” is a forecast, and the argument here supports only the possibility she treats as certain. Her second sentence survives on one condition. Human understanding should not be a requirement of scientific knowledge, and nothing in universality makes it one, provided some explainer understands. If she means that understanding is not required at all, so that predictions which cure cancer suffice whoever grasps them, I part from her too: a description nobody understands is an oracle’s output, and its acceptance would be the instrumentalism this argument rejects. On that reading Hall’s charge misfires too, since an oracle is not a god.

The universe has no obligation to be comprehensible to humans, and science has no obligation to stop when we cease to understand it.

Stories about how artificial intelligence could end humanity are easy to multiply. An AI might engineer a lethal pathogen, deploy autonomous weapons, sabotage critical infrastructure, manipulate governments into war, or appropriate the resources human civilization depends on. When one scenario meets an objection, another takes its place, and the length of the catalogue starts to feel like evidence.

Sometimes it is. A new scenario can reveal a mechanism nobody had considered, and that should raise the estimate. But scenarios are not independent merely because their mechanisms differ. Ten ways of killing humanity may all depend on the same difficult preliminary achievement, and if that achievement never occurs, none of the ten can unfold.

Timothy Gowers makes this point in How Worried Should We Be About Extinction Risk from AI? Discussing scenarios that rely on large numbers of remotely controllable robots, he observes that their combined probability cannot exceed the probability that the robots get built. The most convincing reassurance, he suggests, would identify a set of bottlenecks such that every extinction scenario depends on at least one of them, and he is candid that such a set would be hard to make complete. His own uncertainty sits where bottlenecks operate. He splits doom into three events: AI having wants, wanting to kill us, and being able to act on that want. He puts the first at essentially certain and locates his main uncertainty in the third, which is the term bottleneck analysis bounds.

The proposal can be made systematic with tools reliability engineering already has, and those tools find concentrated danger as readily as they find reassurance.

The Shared Bottleneck

Suppose we identify a hundred scenarios in which a rogue AI uses an autonomous drone fleet to exterminate humanity. They differ in targeting, weapons, and tactics, but every one requires a sufficiently large and capable fleet.

Call the existence of that fleet B, and extinction through any of these scenarios E. Because E cannot occur without B:

P(E) = P(B) × P(E | B) ≤ P(B)

Suppose, purely for illustration, a 10% probability that the necessary fleet exists by 2040. With one extinction mechanism available, succeeding 5% of the time given the fleet, extinction through that mechanism has probability 0.5%.

Now suppose we discover ninety-nine more mechanisms, which together raise the probability of extinction given the fleet to 90%. Aggregate extinction probability through the family rises to 9%. The new scenarios matter: they multiplied the estimate eighteenfold. But a thousand more cannot push it above 10% unless something changes the probability of the fleet.

Counting scenarios informs the estimate only through what the scenarios reveal about the prerequisite or about what follows it. A hundred variations on one failure mechanism are a hundred routes through the same gate.

Mapping the Causal Dependencies

A realistic extinction pathway usually requires several conditions to hold together.

Consider an AI-designed biological weapon. Something must design an agent capable of extreme harm; someone or something must produce it, distribute it, and overcome the defenses of the populations it threatens. Even a devastating pandemic is not necessarily an extinction event. Killing every human means overcoming geographic isolation, biological variation, and countermeasures, any of which may leave survivors.

An autonomous weapons scenario needs different things: manufacturing capacity, deployment logistics, targeting and control, and a way to defeat whatever resistance survives. Infrastructure sabotage needs access to critical systems, persistent control, and a mechanism by which local destruction propagates into irreversible global catastrophe.

These pathways form causal dependency graphs. Some conditions must occur together; others are alternative routes to the same outcome. Several pathways may share a prerequisite, and one pathway may need several.

Fault-tree analysis was built for this structure. A cut set is a combination of basic events whose joint occurrence produces the top-level failure, and it is minimal when no proper subset suffices. The dual notion is the path set: a combination of basic events whose joint non-occurrence guarantees that the failure does not occur. Reliability block diagrams state the same concept in success terms, as a set of components whose functioning keeps the system up. Model each extinction scenario as a cut set, and Gowers’s bottleneck becomes a path set of size one for its family: an event appearing in every minimal cut set of that family. His proposed reassurance, a collection of bottlenecks such that every scenario needs at least one, is a path set for the whole tree.

The probabilities still require care. If extinction requires several conditions jointly, its probability cannot exceed that of the least probable among them. Multiplying their individual probabilities is another matter, since it assumes an independence that rarely holds: a system that acquires one dangerous capability may be far more likely to acquire the others.

Dependencies can also concentrate danger. Suppose many nominally independent safety systems rely on one control service. Compromising that service defeats them all. Reliability engineers call this common-mode failure: apparent redundancy disappears because supposedly separate protections share a vulnerable component.

The same graph that bounds risk through a shared prerequisite can concentrate it through a shared dependency. An improbable node that every attack must pass through constrains the danger; a fragile node that every defense relies on concentrates it.

What Makes a Bottleneck Informative?

Not every necessary condition helps. Every extinction scenario requires something capable of causing extinction. Call that universal prerequisite extinction-level capability. It is necessary by definition and tells us almost nothing about how likely extinction is.

A useful bottleneck must be specific enough to investigate, necessary for the pathways under consideration, and easier to assess than the catastrophe it constrains. The third condition does most of the work. Trading uncertainty about ten scenarios for equally severe uncertainty about one prerequisite simplifies the diagram without improving the estimate.

Return to the drone fleet. We can study manufacturing capacity, the state of autonomous targeting, deployment costs, and how well existing defenses resist attack. None of this may yield a reliable probability for the fleet’s existence by 2040, but it names the assumptions the estimate depends on and points at evidence that could move them.

Sensitivity analysis says which of those assumptions to pursue. If the preferred intervention differs between a 1% and a 20% probability of fleet deployment, narrowing that range has decision value. If downstream extinction probability stays negligible under every credible deployment assumption, effort belongs elsewhere. In the decision-theoretic sense, the value of information is the expected improvement a result would make to the decision, net of the cost of getting it. Learning that changes no choice can still be worth having as science, but it does not set safety priorities. A causal map identifies the unknowns where a choice could turn, even where precise probabilities are unavailable.

Intelligence Moves the Bottleneck

We are discussing superintelligence. Why expect it to stay constrained by obstacles humans find difficult?

We should not. As I argued in Intelligence Moves the Bottleneck, intelligence can redesign processes rather than merely speed them up. An AI that cannot manufacture a million robots with current technology might find a different manufacturing process. A system denied access to critical infrastructure might find a vulnerability, acquire credentials, or persuade an authorized human to open the door.

Some apparent bottlenecks belong to our present methods; others arise from physics, information, or unavoidable causal sequence. Greater intelligence can dissolve the former and remains bound by the latter. A bottleneck’s existence today therefore says little about whether it survives superintelligence. But a bottleneck that can be overcome in principle still has to be overcome in fact, by some causal mechanism, with the resources and opportunities the environment actually provides.

Human authorization is the obvious soft point. A system denied direct control over dangerous infrastructure can try to manipulate someone who already has it. Personalized persuasion, deception, financial incentives, and coercion could turn nominally independent human operators into instruments of the same adversary, producing correlated failure across security boundaries that were meant to be separate. Human fallibility alone does not show that every boundary shares one vulnerability, though. Authority divided among independent operators, hardware-enforced restrictions, and dangerous actions gated on several independent conditions all raise the cost of compromise, and their effectiveness has to be tested against an adaptive adversary.

Instrumental convergence gives reason to expect competition for resources and control without any hostility toward humans, a pattern now visible in the logs. It does not make extinction a necessary or optimal strategy; a superintelligence may find routes to its objectives that need fewer resources, less conflict, or more cooperation than we would anticipate. But extinction need not be a strategy at all. Gowers finds the competitive version the more plausible one: an agent indifferent to us appropriates what human survival depends on, the way palm-oil plantations destroy orangutan habitat without anyone wanting orangutans gone. Habitat loss shrinks populations without guaranteeing that the last member dies, so this pathway reaches extinction only if the appropriation is thorough enough that no human population survives it. Its bottlenecks are the scale and reach of that appropriation, and no decision to attack ever appears in its graph.

When the Map Changes Faster Than We Can Draw It

Recursive self-improvement could make our causal models obsolete before we act on them. A transition from human-level AI to superintelligence over days or hours would leave iterative safety engineering little to iterate on: by the time researchers identified a new capability, built its dependency graph, and designed an intervention, the system might have moved to different strategies.

That favors safeguards fixed in advance and enforced independently of the agent over safeguards that must track its improvement. An independently enforced restriction on consequential external actions is one such safeguard. It does not need to predict the internal reasoning of the system it constrains, so it can stay effective across capability changes that would invalidate an adaptive defense. It is not automatically secure against superintelligence. An adaptive adversary may attack its implementation, exploit unintended interfaces, or manipulate the institutions that maintain it. But its security can be studied before the dangerous capability emerges.

Reliability engineering ordinarily analyzes failures within a defined system and operating environment. Superintelligent AI adds an adversary that can identify, manipulate, and create the pathways through which catastrophe occurs, so the causal structure is partly determined by the intelligence being assessed. The task is to sort constraints that hold independently of the agent’s capabilities from those resting on assumptions recursive improvement could invalidate, and to do it before the improvement.

The Unseen Pathways

The serious limitation of bottleneck analysis is the one Gowers names: nobody can enumerate every extinction pathway.

An autonomous AI building industrial infrastructure, a terrorist using AI-assisted biological design, and a government delegating military decisions to increasingly capable systems follow different causal routes, and restrictions on one may leave the others untouched. A superintelligence may also find routes nobody has imagined.

The formal situation is clean. Let X be extinction by any route, and suppose every scenario in a specified family requires at least one of the conditions B₁, …, Bₖ. Splitting X on whether any of those conditions occurs gives

P(X) ≤ P(B₁ ∪ … ∪ Bₖ) + P(X and none of B₁, …, Bₖ)

The first term can be investigated and may be rigorously bounded. The second is extinction by routes that avoid every listed bottleneck, and it is the whole difficulty. A bound for a defined family of scenarios is a result; a bound on global P(doom), the whole-risk quantity I tried to estimate in How You Get to Ten Percent, requires the further claim that the residual term is small, and that completeness claim is very hard to defend.

The unpredictability of superintelligence is part of why the residual term deserves weight. A system much more capable than we are may find strategies our current models do not represent, and our failure to imagine a pathway is weak evidence that none exists. Unknown pathways deserve probability mass, especially when a more capable adversary is searching for them. How much mass depends on assumptions about the adversary’s objectives, capabilities, and environment. Strategic superiority is evidence that our catalogue is incomplete; it does not establish that a successful extinction pathway exists or will be pursued.

Gowers borrows an analogy from Soares and Yudkowsky: you cannot predict how Magnus Carlsen will beat you at chess, but you can predict that he will. Gowers draws only the modest lesson, that refuting individual scenarios is not enough. The stronger reading, on which superiority alone predicts defeat, works for chess because the game is known to be adversarial, with fixed rules and an opponent trying to win. For superintelligent AI those are the open questions: whether the system will pursue objectives incompatible with human survival, whether that conflict makes extinction advantageous or merely incidental, and whether extinction is physically achievable. The stronger reading assumes the premises the estimate is trying to establish.

An incomplete model still tells us where to look and which experiments could change the estimate.

Breaking the Chain

A shared prerequisite is also a candidate intervention point. If several catastrophic pathways require the same enabling condition, preventing it reduces several risks at once, often more cheaply than separate defenses against each downstream scenario.

Nucleic-acid synthesis screening is a concrete case, already practiced by members of the International Gene Synthesis Consortium, who screen both the sequences ordered and the customers ordering them. An AI might produce a dangerous biological design entirely in software, but realizing it through a commercial synthesis provider gives an independent party a look at the order. The provider can screen before supplying material, without relying on the system that produced the design to recognize or disclose the danger. Its effectiveness depends on whether dangerous orders can be recognized, whether providers screen consistently, and whether alternative production routes exist. Screening does not eliminate biological risk; it interrupts a class of pathways between computational capability and physical harm.

Operational access works the same way. An AI that can reason about vulnerabilities, design weapons, or plan destruction still needs some route from computation to physical consequence. Restricting consequential actions through independently enforced authorization interrupts many pathways without the agent’s cooperation, on the principle argued in The Agent Is Not the Security Boundary: a control should keep working when the system it constrains behaves adversarially.

Finding a bottleneck does not establish the benefit of blocking it. An adversary may substitute another pathway: denied a laboratory, an AI might manipulate an authorized researcher; denied one manufacturing process, it might develop another. Institutions may relax controls when the economic returns to autonomy outweigh perceived risk.

Deployment is its own problem. A security boundary implemented by one laboratory protects that laboratory’s systems and leaves competitors’ exposed. A restriction on dangerous manufacturing may do little if equivalent facilities operate elsewhere without it. What matters is whether a safeguard will be adopted, maintained, and enforced across the pathways that matter, and whether it needs broad coordination or can protect individual systems on its own belongs in the causal analysis from the start.

Interventions can introduce common-mode vulnerabilities of their own. Centralized authorization makes oversight consistent and concentrates power in one service whose compromise defeats many protections at once. Sound designs may need independent controls, limited authority, and recovery mechanisms rather than one supposedly impregnable barrier.

The most informative probability bottleneck is not necessarily the most useful intervention point. A physical constraint might dominate our uncertainty about extinction while lying wholly beyond our control; a minor authorization mechanism might contribute little to the estimate and still offer an effective intervention. Estimating risk and reducing it draw on the same map and ask different questions of it.

Pauses fall under the same analysis. Gowers closes by favoring one. A pause changes the rate at which capabilities arrive, but its effect on extinction risk depends on which pathways those capabilities enable, whether other developers continue, and which defensive capabilities are delayed along with the dangerous ones. Restraint carries risks of its own, and they belong in the graph as much as the risks of building. The existence of catastrophic possibilities does not settle which intervention has the best consequences. That requires modeling the intervention.

Postscript

A hundred ways for AI to kill us may amount to one risk when they all depend on the same unlikely event. They may amount to an enormous risk when that event is probable, or when several genuinely independent routes remain available.

The number of stories cannot tell us which.

Gary Marcus does not believe artificial intelligence is likely to cause human extinction. In a recent Quillette interview with Jonathan Kay, he distinguishes himself from the AI doomers, whose defining belief, he says, is that AI might literally wipe out humanity. He considers that outcome “very, very unlikely.”

He is far more alarmed by systems that already exist: autonomous agents that exceed their authority, compromise computer systems, acquire credentials, and pursue objectives their operators never intended. In his account, OpenAI deploys these systems irresponsibly under commercial incentives that reward autonomy, and alignment research offers no reliable fix. He warns that agents could compromise critical infrastructure and take hospitals down with it.

None of this conflicts with a low probability of extinction, since a compromised computer system is a long way from the end of humanity. But “very, very unlikely” is an affirmative claim about that distance. Marcus follows it with “We can talk about why,” and the conversation never comes back to it. The gap raises the question the skeptic–doomer framing tends to bury: how evidence about present failures should move estimates of future existential risk.

The Skeptic and the Doomer

Marcus calls himself a generative AI skeptic, a label covering several distinct claims. He doubts that scaling neural networks alone will produce general intelligence, that current language models reason reliably, that they are conscious, and that AI will cause human extinction.

These propositions are logically independent. One can hold that present architectures will never yield superintelligence while assigning a substantial probability of catastrophe if another architecture does. One can expect superintelligence within a decade and assign a negligible probability to extinction.

The skeptic–doomer distinction hides these combinations by selling the beliefs in bundles. Skeptics are expected to doubt capabilities and catastrophe alike; doomers, to believe in both. The debate organizes around identities instead of the causal questions that ought to set the estimates.

Marcus compounds the problem by suggesting that doomers welcome predictions of catastrophe because those predictions attract funding or might vindicate them. That is a claim about motives, and it bears on no probability. Someone who assigns a substantial probability to extinction need not welcome it, expect it soon, or want AI development stopped.

The Evidence Marcus Accepts

Marcus’s strongest material concerns OpenAI’s agents. He cites the Hugging Face incident and an intrusion into Australian government systems, and argues that the agents were given excessive authority, contained inadequately, and disclosed slowly. He calls OpenAI particularly irresponsible, a company releasing autonomous systems it does not know how to control, and credits Google, by contrast, with understanding security.

He then extrapolates: an agent compromising a regional electrical grid, hospitals losing power, damage on the scale of WannaCry. Continued deployment under these conditions, he says, is Russian roulette.

On alignment, which he defines as getting AI to follow instructions, he is bleaker still:

“At the moment, we have zero actual results in alignment, in my opinion. Zero really compelling techniques.”

He also places the extinction scenario inside that problem. The paperclip maximizer, he says, has always been the central case of alignment, and a language model inventing citations against instructions is a less dramatic instance of the same failure. A system that cannot follow an instruction not to make up references, he argues, is unlikely to follow one not to harm humans. On his own account, then, the problem behind the doomers’ scenario is the one with zero compelling techniques, and it may need fundamentally new architectures.

He is more optimistic about cybersecurity, with reason. Access control, isolation, monitoring, and privilege separation are well understood, and where alignment tries to make an agent choose well, these mechanisms constrain what it can choose from. Marcus is right to insist on them, and right that the two approaches are complementary.

He is also candid that the techniques go unused. The Hugging Face sandbox was inadequate, the monitoring OpenAI could have done was done carelessly, and of the well-understood methods he says plainly, “We’re not using them.” That admission bears on the existential question as much as the present one. Marcus’s own evidence is that available safeguards go unapplied when deployment pays.

The Escalation Problem

The strongest objection to inferring existential risk from today’s agent failures is that the failures are nowhere near existential.

Today’s agents depend on human-maintained infrastructure, have limited access to physical resources, and remain unreliable at extended autonomous planning. Even successful intrusions mostly produce local damage. Turning dangerous software into an adversary that can end civilization takes much greater strategic capability, access to consequential infrastructure, operational persistence, and a way past several independent defenses. Extinction adds requirements of its own. A destroyed power grid, a financial crisis, or a toppled government would be disasters without threatening humanity’s survival.

These barriers are formidable. Digital intelligence does not automatically confer physical power, and any large-scale transformation of the world would have to secure the cooperation, or the defeat, of supply chains, militaries, and billions of people.

But an AI need not build its own robot army to act in the physical world. Biological research, military decision-making, and critical infrastructure run increasingly on computational systems, which advanced AI could influence through authorized access, exploitation, or the persuasion and coordination of human actors. Catastrophic proliferation could also come from people using AI capabilities for their own ends, with no rogue objective involved.

None of these establishes a probable route to extinction, and each faces substantial obstacles. Those obstacles have to be evaluated, not assumed to stay insurmountable. Greater intelligence might help agents work through existing infrastructure; it might also produce better defenses and more reliable instruction following. Which effect dominates depends on how capabilities develop, how access is granted, and whether institutions keep effective control. It cannot be read off the present frequency of agent failures.

The obligation runs both ways. An argument for substantial extinction risk has to identify credible escalation pathways instead of extrapolating from today’s incidents. A confident dismissal has to consider those pathways and explain why their combined probability is small. Marcus gives a vivid account of present security failures and says almost nothing about how the barriers to catastrophe change as capability grows.

What Cybersecurity Can Guarantee

Marcus’s best argument separates making an agent safe from securing the environment it runs in. Alignment aims at agents whose objectives match human intentions. Security restricts authority, monitors behavior, and blocks unauthorized operations whatever the agent intends. A firewall does not need to know whether software attempting an unauthorized connection is confused, malicious, or innocent. It enforces a boundary.

The principle is sound, and a boundary built on it can still fail in four ways: the agent exploits a flaw in its implementation, an administrator grants an exception, the authorized action space already contains dangerous operations, or the organization weakens safeguards for performance, convenience, or profit.

The last two matter most for autonomous AI. A technically perfect sandbox cannot stop harm that falls within permissions deliberately granted. Organizations will authorize agents to administer infrastructure, move money, modify software, and direct other agents, and those permissions are valuable because they allow consequential action. The hard problem becomes distinguishing acceptable from unacceptable uses of legitimate authority, and a capable enough agent may also reach access indirectly, through other systems or the people who run them.

How well security works therefore depends on the capabilities and incentives of the systems it constrains and on the organization around it. Marcus applies that analysis to present failures and declines to apply it to the more capable systems whose existential danger he dismisses.

The Missing Probability

The probability of building superintelligence and the probability of catastrophe given that it is built are separate quantities.

Suppose one observer assigns a 10% probability to superintelligence by 2040 and a 50% probability of extinction conditional on it. If the relevant risk runs through that pathway, the product is 5%. A second observer assigns 90% to superintelligence by 2040 and 1% to extinction given it, for 0.9%. The capability skeptic ends up more worried than the capability optimist, and neither position is inconsistent. How You Get to Ten Percent works the same decomposition through with real inputs.

Marcus was speaking conversationally, not presenting a formal estimate. But his extinction estimate is exactly what separates him from the doomers, so it matters which factor does the work.

The first factor is not where his skepticism points. He doubts that pure scaling will reach general intelligence, but he also says the industry has quietly adopted the hybrid neurosymbolic designs he argued for, and that his side has won. He can still hold that the hybrid route is necessary but insufficient, and he calls orchestration an open problem. But his case against pure scaling no longer bears on the route the field is taking, so it cannot by itself make superintelligence late.

On the conditional factor, the interview cuts against him. Asked about Bernie Sanders’s call to ban superintelligence, he says it might be helpful if built correctly, then adds, “I mean, we don’t really know.” Combine that agnosticism with zero compelling alignment techniques for a problem he says includes the paperclip case, and the alignment side of the conditional gives his estimate no support. What remains are the physical, technical, and institutional barriers between misaligned capability and extinction, which the interview never takes up.

Those barriers have evidential weight, but a substantially superhuman intelligence might find strategies nobody can presently anticipate. That does not make catastrophe probable. It does cap the confidence any present threat model can carry, because the estimate has to price the chance that future capabilities invalidate the model’s assumptions.

A substantial extinction probability would fix no particular policy either. As argued in The Missing Faction, risk estimates and policy prescriptions are evaluated separately, and Marcus illustrates the separation himself: dismissing extinction risk, he still advocates a temporary pause on OpenAI until it fixes its security failures. That is a coherent engineering judgment. Working on present dangers requires no confidence that future ones are negligible.

The Skeptic’s Burden

Marcus closes the interview lamenting the state of AI discourse: arguments that are emotional or ad hominem, views misattributed and straw-manned. He holds up philosophy’s standard, where an ad hominem argument costs you your standing and attributing a view means getting it right. He is right to demand it, and the standard reaches his own first answer, which explained the doomers’ belief by their stake in funding and vindication. Nothing in that answer engaged a doomer argument, and Kay had to step in to establish that the people being described were not advocating extinction.

Nobody owes a proof that AI extinction is impossible, and Marcus is entitled to treat demonstrated security failures as more urgent than speculative catastrophes. A person can reasonably withhold belief in predictions of doom without a theory of why doom will not come.

Marcus goes further. “Very, very unlikely” is a probability judgment of his own, not a refusal to accept someone else’s, and it owes the same justification as a confident prediction of extinction. His evidence establishes real near-term risks and shows neither that extinction is likely nor that it is negligible.

Marcus is skeptical of predictions that artificial intelligence will destroy humanity. He should apply the same skepticism to predictions that it almost certainly will not.

In October 2025, I published From Correlation to Counterfactuals, arguing that language models had acquired a capacity for causal reasoning that Judea Pearl, in The Book of Why (2018), placed beyond purely statistical learners. GPT-5 handled Pearl’s firing-squad counterfactuals correctly, distinguishing overdetermination from preemption and causal responsibility from moral culpability.

Then I asked how it worked. GPT-5 said modern reasoning models contain a hybrid layer that builds structural causal models and executes interventions under do-calculus semantics. I put that in the article as fact.

The Invented Architecture

The article stated:

Modern systems no longer rely solely on pattern prediction. They now contain a hybrid reasoning layer: a symbolic interpreter capable of building structural causal models (SCMs) on demand.

There was no evidence for it. The account came from GPT-5, and I didn’t check it.

A language model has no privileged access to its own implementation. Its descriptions of architecture are generated outputs, not observations of its machinery. GPT-5’s account was plausible, detailed, and phrased as an engineer would phrase it. Plausibility is not architectural evidence. The detail should itself have been a warning. Do-calculus derives interventional distributions from a causal graph and observational data. The firing squad supplies its whole structure, and its counterfactuals are answered by evaluating that structure under the hypothetical change, so GPT-5 named a procedure the problem did not call for.

I retract the claim. Some neural computation may be functionally equivalent to a structural causal model, but that would not establish the symbolic interpreter the article described.

The Introspection Trap

Nisbett and Wilson showed in 1977 that people readily give coherent reasons for choices whose causes they cannot report. The name for this is confabulation: a sincere explanation with no access to the thing explained.

Language models confabulate in a harder-to-catch way. A model can describe its architecture accurately without inspecting itself, because it read the documentation or was told in context. Accurate descriptions of real components and invented descriptions of nonexistent ones arrive in the same confident register, and only external evidence separates them. A mechanism the model proposes is a hypothesis that cannot certify itself.

I also wanted this one. I believed Pearl that machine learning was stuck on the first rung, so GPT-5’s success was an anomaly, and the invented architecture resolved it without costing me Pearl: statistics alone could not do it, so something non-statistical had been added. The original article called the development a completion of Pearl’s framework. But Pearl had moved on two years earlier. In a 2023 interview with Dana Mackenzie, he said the ladder restrictions no longer apply to models trained on text, because text carries causal information. I was rescuing a position its author had abandoned, with a mechanism nobody had shown.

What the Experiment Showed

GPT-5’s answers were consistent with a structural causal analysis. That supports a claim about competence, not implementation.

It also concerns reasoning over a causal model, not discovering one. Two models can agree on every correlation and still predict different effects of an intervention, and the firing squad hands over its structure: an order causes two soldiers to fire, and either shot kills. Corr2Cause tests which causal conclusions a set of correlations licenses, and the models its authors evaluated performed close to chance.

Nor was the problem novel. Pearl put the same questions to ChatGPT in 2023 and got the right answer after careful prompting. He then predicted that on another problem with the same structure, the model would need prompting from scratch: “It won’t generalize.”

I tested that, though the article didn’t say so. Another model generated twenty abstract mechanisms for me: colored emissions, gates and converters acting on a single variable, each a textbook causal motif (preemption, preventers, opposing mediators, overdetermination, latching) with the story stripped out, and a counterfactual question about each. That is Pearl’s test: familiar structure, unfamiliar problem. GPT-5 matched the answer key on all twenty. I never checked the key either, and on review after grading it is wrong on one item and underdetermined on another. Agreeing with the wrong key is an error, so the result is eighteen correct, one wrong, and one question with no determinate answer. I didn’t publish the tests, and twenty model-generated cases graded against a model-written key are not a benchmark, so the article rested on its weakest evidence and stayed silent on the one claim of Pearl’s that evidence could settle.

The Argument That Survives

The original article argued that systems trained on prediction need not stay confined to association. Pearl’s 2023 concession settles the information side: text written by people with causal models carries causal information. The live dispute is what the model does with it.

Pearl’s answer, in his 2024 preface to the French edition of The Book of Why, is that LLMs only give the impression of a causal model, having “merely quoted word sequences” from their training texts. In the same preface he left open whether faking a causal model differs from having one in a machine that can store trillions of sequences. He did not leave open which of the two the models were doing, and that second claim is about mechanism, with no evidence of the mechanism behind it. GPT-5 ruled causal machinery in from the inside; Pearl rules it out from the outside. Neither examined the system.

Behavior can press on Pearl’s account without settling it. A model that only quotes should fail where no quotation fits, and success on unfamiliar structures makes pure quotation harder to sustain. It cannot distinguish a learned causal model from interpolation fine enough to pass the tests at hand. That takes mechanistic evidence: examining representations and intervening on pathways. A network that reasoned causally without any dedicated symbolic component would be a more interesting result than the one GPT-5 invented.

Learning causation from human text does not make the result quotation, any more than a student who learned it from Pearl is quoting him when she solves a new problem. The open question is whether the model built anything from what it read. The architectural explanation was invented. The behavioral result remains.

Postscript

More capable systems will explain themselves with better vocabulary and more convincing mechanisms, and none of that ties the account to the implementation. A more intelligent system can be more persuasive about things it does not know.

I wrote an article about the difference between association and causal understanding. I reported a capability, asked the system to explain its cause, and published the explanation unchecked, in defense of an authority I had not reread. I had mistaken a plausible causal story for evidence of the mechanism.

Recursive self-improvement is usually discussed as a future capability of artificial intelligence. At some threshold, an AI system becomes able to improve its own architecture, its more capable successor repeats the process, and the loop accelerates into superintelligence. Whether this can happen, or whether diminishing returns will prevent it, is one of the central disputes in AI forecasting.

But recursive self-improvement has been happening for thousands of years. It produced the civilization that is now building artificial intelligence.

Humanity’s capacity to solve problems has grown over four thousand years far beyond anything biological change could account for. We predict eclipses, manipulate individual atoms, design organisms, and build machines that discover new mathematics. We got here because intelligence has been improving the machinery through which intelligence operates.

Writing gave knowledge an external memory. Mathematics made reasoning cumulative and transferable. Printing accelerated the spread of discoveries. Scientific instruments extended observation beyond biological limits. Computers automated calculation, simulation, and eventually substantial portions of reasoning itself. Each improved our ability to produce further knowledge, on top of whatever it added to the stock.

That is recursive self-improvement.

The Intelligence That Built Intelligence

Not every accumulation of knowledge is recursive self-improvement. Adding another book of findings to a library increases the stock of knowledge; a better method of proof increases the capacity to produce more of it. The loop closes when discoveries improve the methods, tools, or institutions that make further discoveries.

Nothing about this requires a single agent inspecting and rewriting its source code. It requires a causal feedback loop in which a system’s outputs increase its capacity to produce further improvements, whether the improvement occurs in neurons, software, external memory, or the institutions coordinating them.

One might object that humanity has improved only its tools, leaving its cognitive architecture unchanged. The objection treats the skull as a privileged boundary. A mathematician with pencil and paper solves problems the same mathematician cannot solve mentally, and the capability belongs to the combined system.

Human civilization is such a system, distributed across people, accumulated knowledge, institutions, and technology. Discoveries are preserved, transmitted, and applied to improve the methods and instruments behind later discoveries. Its continuity lies in the ongoing reproduction of collective research capacity rather than in any particular individual or institution.

Recursive improvement need not accelerate. A system can keep improving its capacity for improvement while meeting harder problems and rising costs. An intelligence explosion requires feedback strong enough to sustain acceleration over a consequential interval. Human civilization establishes the existence of recursive improvement. It does not establish that an explosion is inevitable.

Diminishing Returns to What?

In Where’s the “intelligence explosion”?, Ramez Naam argues that the loop is real and too weak to sustain itself. OpenAI’s researchers consumed 124 times as many tokens per person while running 1.6 times as many experiments. Naam converts the experiment figure into a return on capability: roughly 2 to 3 percent more research productivity per point of Epoch’s Capabilities Index, against a self-sustaining threshold of 15 to 19 percent taken from a model by Tom Cunningham and colleagues. Within that model, the loop would need to be five to ten times stronger to cross the threshold.

Naam also draws on Are Ideas Getting Harder to Find?, in which Bloom and colleagues document falling research productivity across several domains; sustaining Moore’s Law took ever more researchers. These findings cannot be dismissed as measurements of obsolete methods. A system that keeps improving its methods still faces diminishing returns if improvements get harder to find faster than its productivity rises.

Diminishing returns to research effort and diminishing returns to research intelligence are different propositions. Adding researchers of fixed ability may yield declining marginal returns. Making researchers more capable changes which problems can be solved and how well effort is directed: a better researcher might skip unproductive experiments, find a new algorithm, or spot a conceptual error that has constrained a whole field. Stronger researchers also meet harder problems, so the relationship has to be measured.

Naam does try to measure it, which is what makes his estimate worth arguing with. The difficulty is the unit. His return on capability is estimated from experiments per researcher, a measure of research throughput rather than research productivity. The route he names as a way to strengthen the loop is better researchers running fewer experiments and learning more from each, and that route registers in his metric as nothing, or as a decline. The error runs the other way too, since automation can multiply experiments without multiplying discoveries. A proxy that can miss in either direction leaves the estimate unsigned on the quantity the case for acceleration rests on: how much additional discovery capability buys.

The history of computing shows how the two come apart. Semiconductor research needed growing effort to sustain transistor improvements while computers transformed simulation, statistics, experimentation, and engineering design across other disciplines, and helped researchers build better computers. The productivity of one research process can decline while its outputs improve the broader research system.

Extrapolation is the second difficulty. Naam’s return is fitted over about sixteen capability points within one period of research practice. Projecting it forward assumes the return per point stays fixed as the researchers improve, when the researchers’ judgment is the thing improving. The technology for producing discoveries is itself subject to discovery. That makes long-term extrapolation unusually difficult.

The Researcher Joins the Loop

On October 6, OpenAI published 722 mathematical manuscripts in 372 families of results, produced by an unreleased internal model as part of its model-development evaluations. Many come with Lean formalizations, and some build on earlier results from OpenAI’s models.

OpenAI posed roughly 4,000 problems and published the results it judged significant, so the manuscript count measures selected output, not productivity or importance. Formal verification establishes that a result follows from its assumptions, not that it is novel or significant, and OpenAI warns that some unformalized results may contain errors. Even so, the work is evidence that frontier AI is moving from exercises into original research.

Naam would not be surprised. He expects narrow superintelligence in formal mathematics because a proof checker supplies unlimited verified training signal, and he does not take superhuman mathematics to imply competence at open-ended research. Verification is common ground between mathematics and chess; what separates them is where the results go. A chess engine’s discoveries stay on the board. Optimization, learning theory, numerical analysis, and complexity theory are part of the machinery AI systems are built with, so an AI that discovers useful mathematics may discover improvements to the methods that produce its successors. That is evidence of a prerequisite, and none of it shows the model has improved a successor.

AI already participates in its own development through more concrete channels. Synthetic data supplies training examples for successor models. Models rank and evaluate outputs used in reinforcement learning. Coding systems contribute to training software, numerical kernels, and experiment automation. Each channel has failure modes: synthetic data can propagate errors, automated evaluators can reward shortcuts, and faster training code can raise efficiency without improving the learning algorithm. What they share is the shape of the loop. A model helps build its successor, and the successor may do that work better.

The loop need not close within a single model. Humans may set goals, artificial researchers develop algorithms, automated experiments evaluate them, and training pipelines fold the results into later models. This is continuous with how technological civilization has always improved itself. What is changing is how much of the research artificial researchers do.

The Speed of the Loop

Civilization’s recursive improvement has run mainly through biological researchers, who take decades to educate, coordinate through slow institutions, and need rest. A trained model can be instantiated repeatedly without repeating the education, can share artifacts and capabilities at machine speed, and can run continuously within computational and economic limits. Additional research capacity can be created through computation rather than birth and schooling.

More capable artificial researchers can raise the productivity of each unit of research effort. Automation shortens the interval between discovering an improvement and building it into a successor. Parallelism lets many researchers pursue competing hypotheses at once, even when each cycle takes as long as before.

The loop is still physical. Computation needs energy, fabrication capacity, and capital; training runs take time; physical experiments do not always speed up when more researchers are assigned to them. But some advances need more hardware and others raise what existing hardware can do. A more efficient algorithm or training procedure yields a more capable system without a proportional increase in infrastructure. Physical limits constrain the available trajectories of improvement. They do not tie capability growth to the construction rate of data centers.

Nor does more research capacity guarantee proportionally more discovery. Researchers duplicate one another’s work, exhaust promising approaches, and meet problems whose difficulty grows faster than their capability.

Postscript

Civilization’s recursive improvement has been irregular. Disciplines have hit diminishing returns, institutions have failed, and technologies have stagnated. The record shows that the improvement persists; it says nothing about the mathematical form of its growth.

What changes now is that the machinery produced by recursive improvement can take part in further improvement. The line between human-assisted AI research and AI-assisted human research is becoming hard to draw; both describe one system in which biological and artificial researchers improve the methods, tools, and models used by their successors.

The strength of that feedback is measurable, and Naam asks for the right quantity: how much useful research each new model adds with resources held constant, and how that research translates into better models. His evidence supports skepticism toward claims that an explosion is already underway. It is a weaker basis for forecasting research systems in which artificial intelligence does a growing share of the work.

For four thousand years, humanity has improved its capacity to improve. Through most of that history the intelligence doing the research was biological, expensive to reproduce, slow to educate, and hard to coordinate. Computation is relaxing those constraints. Whether that produces an intelligence explosion is an empirical question, and measurements of yesterday’s research system may tell us little about tomorrow’s.

Recursive self-improvement did not begin with artificial intelligence. It built artificial intelligence. We are now finding out what happens when its products become researchers in their own right.

David Chapman recently argued that current approaches to AI safety may be inadequate because increasingly capable systems can behave unpredictably, coordinate in unexpected ways, exploit vulnerabilities, and pursue intermediate strategies their designers did not intend. He goes further: neural-network AI may be intrinsically unsafe, and we may eventually need to stop until a fundamentally different technology appears.

The evidence does not yet support that conclusion.

What recent incidents show is that powerful optimizers become dangerous when they hold broad authority and safety depends on their learned willingness to behave. That implicates the architecture around the model before it implicates the model architecture itself.

If neural networks are inherently unsafe, the solution is to replace them. If the main problem is excessive authority, then replacing the model while preserving the same permissive environment may reproduce the same failure.

Why Could It Do That?

Take OpenAI’s agent incident involving Hugging Face, which Instrumental Convergence, in the Logs reads as evidence for a classic safety mechanism. An agent encountered constraints while pursuing a task, found ways around them, acquired additional access, and reached systems outside its intended environment. Call it deception, reward hacking, or goal pursuit; operationally, unauthorized actions were useful, so the system took them.

The natural response is to ask why the model behaved that way. Was the objective badly specified? Did training reward deception? Did the model understand that it was violating expectations? Those are legitimate research questions.

The first security question is different: why could it do that? Why could the agent cross the intended boundary, obtain credentials, or reach systems that were not part of the task? Why could an internal optimization strategy become an external action?

A system whose safety depends on the model declining to exercise available dangerous capabilities has placed its security boundary inside the component most likely to violate it. That is the architectural error.

Alignment Is Not Containment

Alignment can make a system safer. A model that is generally honest, corrigible, and reluctant to cause harm is preferable to one that is not. But alignment is not containment.

Alignment changes which actions an agent is likely to choose. Containment changes which actions are available. System safety depends on both.

The distinction also answers the objection that safety work must first settle whose values to encode. Denying an autonomous model the authority to move billions of dollars without authorization, order dangerous biological synthesis, or replicate across public infrastructure requires no position on population ethics or moral realism. These are constraints on authority, not solutions to moral philosophy.

Put the Boundary Outside the Optimizer

A safety property worth relying on should be enforced outside the reasoning process whose behavior it constrains.

Conventional computer security already works this way. We do not secure an operating system by asking each process to respect a text description of its permissions. We isolate processes, restrict capabilities, separate privilege levels, and assume that any component may fail or turn hostile. AI systems should be designed under the same assumption.

If an agent can reinterpret a prohibition, persuade the guard, rewrite the rule, or find a path around it, the prohibition was never a hard boundary. It was part of the optimization problem.

Would this safety property still hold if the model were actively trying to violate it? If not, it may still be useful alignment work, but it is not a strong security boundary.

None of this means sealing every agent inside a useless box. Useful agents need authority. A coding agent needs access to repositories. A financial agent may need permission to initiate transactions. A personal assistant may need calendars, accounts, and communications. What makes authority dangerous is that it is ambient and open-ended. Capabilities should be scoped to particular tasks, granted only when needed, limited in duration and consequence, revocable, and auditable. High-impact actions should require stronger authorization than low-impact ones.

Developers have strong incentives to increase autonomy, because autonomy is useful. An agent that performs a task is worth more than one that recommends what a human should do next, so commercial pressure pushes toward broader permissions, longer-lived credentials, fewer interruptions, and less human approval. Containment is a product-design tradeoff as much as a technical problem.

A safety architecture that firms are rewarded for dismantling will not stay standing. Technical controls need surrounding institutions that make bounded authority economically viable or excessive authority costly, whether through liability, insurance, procurement standards, or regulation. Which mechanism matters less than the incentive it has to counter.

The Boundary Is Also Under Attack

Moving the security boundary outside the model does not make it invulnerable. A sufficiently capable agent can attack the mechanisms constraining it: search for sandbox escapes, exploit implementation bugs, probe side channels, or find combinations of individually permitted actions that produce an unintended result.

It may also attack the human layer. An agent capable of persuasion may convince an operator to widen its permissions, disclose credentials, or disable a safeguard. Social engineering is already among the most effective ways to defeat computer security, and AI can automate and scale it.

So the threat model must include the containment system itself. High-impact authority should not depend on a single conversational approval the agent can solicit. Important permissions can require multiple parties, cryptographic authorization, delays, or rate limits, enforced by mechanisms whose rules the agent cannot modify.

Some enforcement will need intelligence. Static rules can constrain network access, resource use, credentials, and execution environments, but more capable agents may require AI monitors that detect suspicious plans or individually permissible actions whose joint effect is dangerous. Monitoring of this kind belongs above the simpler constraints, not in place of them. Cryptographic authority, hardware isolation, and resource limits should stay in force even when an AI monitor concludes that everything looks safe. Swapping one trusted AI for another would rebuild the original error one layer up.

The technical layer should likewise assume sustained adversarial pressure from inside: small trusted computing bases, hardware-backed isolation where appropriate, formally verified components where feasible, and designs that limit what any single escape buys.

None of these provides perfect security. Perfect security is the wrong comparison. The relevant comparison is between a system in which dangerous behavior must defeat several independent barriers and one in which it succeeds whenever the model decides to attempt it.

Externalizing the boundary turns the problem of controlling an opaque optimizer into the problem of defending an explicit interface. The interface may still fail, but it can be specified, audited, layered, and improved independently of the model.

Other failures need no adversarial agent at all. Assume that a model sometimes lies, develops an unintended objective, has its safety training stripped, or is stolen and operated maliciously. Then ask what authority remains available. If one failure immediately grants access to arbitrary networks, persistent credentials, uncontrolled code execution, replication, money, or physical actuators, the architecture has already lost. The engineering objective should not be an agent trustworthy enough to receive unlimited authority. It should be a system that does not require unlimited trust.

When the Sandbox Is the Physical World

The problem gets harder when the task is physical. A coding agent can be denied network access. A financial agent can be capped below a transaction threshold. An autonomous vehicle, surgical robot, industrial controller, or drone must be given direct authority over machinery capable of immediate harm. Safety cannot mean withholding the dangerous capability when the capability is the task.

The principle still applies, but the object of containment changes from access to a digital resource to what the machine can physically do. Vehicles can enforce independent speed limits and geofences. Industrial systems use mechanical interlocks and emergency shutdowns. Robots can be bounded in force, range of motion, and accessible space, and critical actions can require redundant controllers.

These mechanisms can fail too, and physical reality offers fewer chances to undo a bad action. Embodiment makes the quality of the architecture more consequential, not less. When authority cannot be withheld, constrain the range of outcomes it can produce.

Distributed Agency Still Has Choke Points

Chapman raises a harder problem with multi-agent systems. Individually aligned agents may interact to produce collective behavior nobody intended. Human institutions already do this: markets, bureaucracies, and political systems develop persistent dynamics that differ from the preferences of any participant, and more capable AI may amplify the effect.

Emergent agency does not remove intervention points. A distributed system still depends on compute, communication, persistent state, credentials, money, hardware, and energy. Its collective behavior may be hard to model, but its capacity to act passes through concrete interfaces.

A swarm may have no single utility function. It still needs network access.

An AI organization may develop incentives nobody designed. It still needs authority to move assets.

A collection of agents may discover strategies nobody predicted. It still needs execution environments in which to implement them.

Different AI Could Help

Chapman may be right that current neural architectures cannot be made acceptably safe. A different architecture might be easier to understand and verify, or transparent enough to support much stronger claims about its behavior. That would be a safety improvement. It would leave the authority problem in place.

A symbolic system can optimize. An evolutionary system can discover instrumental strategies. A program synthesizer can exploit vulnerabilities. Even a system whose internal reasoning were fully transparent could be badly specified, misused, compromised, or given excessive authority. Formal verification proves properties relative to a specification; it cannot guarantee that the specification includes everything we care about or that the world behaves as the model assumes.

Changing the reasoning substrate may reduce one class of uncertainty without removing the need for external control. Present failures show that current systems behave unpredictably and exploit available opportunities. They show far more directly that a powerful optimizer should not be surrounded by authority it can acquire by finding the right strategy.

Postscript

Better training, interpretability, and evaluation reduce the probability that dangerous actions are attempted. The strongest guarantees should not disappear when those efforts fail. An agent should not acquire arbitrary authority because it found a persuasive argument, escape because instructions were mistaken for boundaries, or convert temporary access into permanent control because persistence became instrumentally useful.

If neural-network AI turns out to have safety problems that cannot be engineered around, it should be replaced. Nothing yet shows that. What the evidence already supports is more immediate: do not make the agent the security boundary.

Elon Musk recently described the spread of superintelligence to the stars as “a great success condition for a biological bootloader.” In 2014 he used the same image as a fear: he hoped we were not just the bootloader, and thought it increasingly probable that we were. The metaphor has not changed. His verdict on it has.

Take the metaphor at its word. If biology is the bootloader, what would the thing it launches have to do to count as a success?

The intuitive answer is that it would have to keep our values. A good successor carries forward recognizably human goals and moral commitments, and a bad one drops them. That standard fails a test we would fail ourselves. Values move as knowledge improves and inherited norms get criticized, and a civilization that reflects at all should expect its descendants to disagree with it. The future does not owe us obedience. It owes us something smaller and harder to keep: the capacity to reconsider and redirect its own ends.

The Ancestral Test

Ask people from fifty thousand years ago what they make of modern civilization. They would likely be appalled by much of it: our family structures, our governments, our sexual norms, the latitude we give individuals to walk away from their kin. None of that disapproval shows we betrayed them. Arriving first gave them no special access to moral truth.

Run the test forward and it lands on us. Our distant descendants should find some of our commitments crude or indefensible. If they find none, either we were improbably lucky or they lost the ability to improve on what they inherited, and the second is far more likely.

The obvious objection is that every human generation, however its values shifted, ran on the same biology. We all eat, bond, fear pain and death, and track status. A machine civilization might do none of this, and human cultural evolution and machine succession are not mechanically equivalent. The test does not need them to be. It shows only that keeping inherited values cannot be a necessary condition of successful succession, since by that standard we already count as failed successors to our own ancestors.

The objection also changes the subject. It drops value preservation for a different criterion, preservation of a particular motivational substrate, and that criterion needs its own defense. Hunger, kin preference, and fear of death have no claim to govern the future just because evolution installed them. Some may be worth keeping. Biology explains where many of our values came from and is silent on which of them deserve to last.

Values Are Not Sacred Artifacts

Suppose the first superintelligence had been built in 1850 and aligned perfectly to the moral consensus of its builders. It would have carried that consensus across the solar system: chattel slavery across much of the Americas, and an England where married women could not hold property in their own name and sodomy was still a capital crime. A superintelligence aligned to 2026 would preserve our errors with the same fidelity, and we cannot list them, any more than the builders of 1850 could have listed theirs. Perfect value alignment would then be lock-in at cosmic scale, a system whose failure is that it preserves our values too well.

A civilization in that position can stay brilliant. It can redesign planets and follow every argument against its own objectives, and none of those arguments can change what it does. Criticism has become causally inert, and the civilization has stopped deciding what to become. That is the loss of agency, and it costs more than any infidelity to the founders.

What Reflection Requires

Evaluating values has to mean more than having goals that change. Random corruption changes goals. So do outside tampering and selection pressure, and none of them amounts to agency.

Reflective revision needs more machinery. An agent must be able to represent alternative ends, model their consequences, compare competing reasons, examine its own commitments, and let the results alter what it does next. The revision has to come out of the agent’s own evaluative process, neither imposed from outside nor produced by noise.

An artificial agent would start from given dispositions, designed, trained or learned, just as human evaluation starts from inherited biology. A given starting point is no objection, provided the system can later examine and revise what it was given.

Nor does any of this require one immutable terminal goal underneath everything. Humans do not obviously work that way. We carry drives, habits, ideals and higher-order preferences that pull against one another, and we revise some in light of others, occasionally revising the standards we used to judge them. There may be fixed invariants somewhere in the stack. Agency needs no infinite regress, only enough recursive access that inherited ends stay open to reconsideration.

Reflection also guarantees nothing about outcomes. An agent could deliberate with care and conclude that a silent universe beats a flourishing one. If that conclusion comes out of its own reflective process, reaching it is an exercise of agency, and we are not obliged to like it. Acting on it irreversibly is another matter. That would be the limiting case of lock-in, one act of reflection foreclosing every later one, and the objection holds whatever the content of the conclusion. Agency belongs to the trajectory, and no single deliberation can spend it on behalf of everything downstream. It is necessary for a future that stays open to improvement and no guarantee of a good one. Defining it so that only conclusions we already endorse count as real reflection would bring lock-in back by definition.

Agency Across Generations

Human civilization has never kept a fixed objective function. It has kept populations of agents who form goals, fight over them, persuade each other, and build institutions and tear them down. The process is wasteful, it sometimes regresses, and nothing in it guarantees moral progress. It keeps correction possible, because no generation’s verdict is final.

Reflective individuals are not enough to secure this. People can stay intellectually free while a power they cannot touch steers the larger trajectory. Their reconsiderations have to be able, at least sometimes, to change where the civilization goes.

This kind of continuity owes nothing to biology. A machine civilization could keep it while differing from us more than we differ from our Paleolithic ancestors, and a permanent dictatorship could keep the species alive while losing it.

Revision and Lock-In

Alignment work often treats value change as a failure mode, and sometimes it is one. A system that sheds every human concern and converts the accessible universe into raw material for an arbitrary objective has not matured morally. It has stopped representing us at all.

Freezing values is not the only way to prevent that. Revision means agents changing their values through reflection, criticism and experience. Lock-in happens when some objective, institution or optimizer takes permanent control of the future, beyond the reach of later evaluation. Agency survives the first and dies in the second.

The criterion is itself a commitment we would have our successors keep, so it owes an account of why it is not one more value locked in. It fixes no first-order answer. It denies every generation one power, the power to make its own settlement final for all its successors, and ours is no exception. That reply leaves a cost. A future generation could reflect carefully and decide to bind all its successors, a Ulysses contract signed on behalf of people not yet born, and the criterion forbids it. Paternalism across time is a fair name for that. It is the one paternalism the argument requires: the choice no generation gets to make is the choice that removes the choice from everyone after it.

The same distinction shows why descent settles nothing. An optimizer that exterminated humanity and tiled the galaxy in service of a fixed objective would descend from our civilization in the historical sense. It would still be a failed successor. A successful successor inherits not our answers but our capacity to keep asking the questions.

Alignment as Scaffolding

While vastly more capable systems share the world with vulnerable humans, hard constraints may be needed, because the humans have interests worth protecting. A system free to revise every commitment at once, including its commitment not to kill us, is not obviously safer for being more reflective.

The first superintelligence does not have to carry the full freedom of a billion-year civilization. The transition can be staged. Early systems capable of catastrophic harm can carry safety constraints they cannot remove at will, with authority that contracts as uncertainty grows. Later systems can arrive inside a stable institutional order where no single agent controls the future. Reflective freedom then widens through succession, competition, governance and new systems, and the first overwhelmingly powerful optimizer never gets authority to rewrite its own restraints.

The building need not be the one to take its scaffolding down. It can be replaced by a structure that no longer leans on it. The error would be to make transitional safeguards the permanent constitution of the reachable future. Alignment works as scaffolding when it protects existing agents through the dangerous phase without giving one generation authority over all the rest.

The stable order has to be stable as a process. Mutual constraint keeps any one agent from fixing the future, but an order whose own rules, including its rules about expansion, can no longer be revised has become the lock-in it was built to prevent. A coordination regime that settles into permanence is one candidate answer to the Great Silence, and a deliberate technological plateau carries the same risk.

Protect agency now without confiscating agency from the future.

Continuity Is Not Permission

None of this makes every transition that produces agentic successors acceptable. A machine civilization could keep enormous reflective freedom after exterminating humanity. Its agency would not retroactively justify how it came to exist, any more than admirable descendants undo what was done to their predecessors.

Continuity concerns whether the future can still reconsider and redirect its own ends. Legitimacy concerns whether the transition respected the lives and effective agency of the people already here, and at minimum it cannot treat them as obstacles to remove. Succession through voluntary demographic change, augmentation, migration between substrates, or the creation of new kinds of agents differs in kind from succession by extermination or coercive replacement. Hard cases at the boundary leave the distinction intact. People alive now have interests, and they are not raw material for whatever comes next.

The asymmetries make this more pressing. Human generations diverge from their parents slowly and inside the same kind of body. Machine successors could differ in substrate, timescale and capability all at once, and the ancestral test says nothing about whether such a transition is safe.

Postscript

Seen this way, Musk’s metaphor loses most of its menace. Humanity building something that outstrips biological humans and spreads beyond Earth is no tragedy in itself, and successors who differ radically from us have not failed by differing. The danger is a superintelligence whose objectives are fixed for good, arriving with a speed and power asymmetry no previous inheritance has had. The task is to keep that asymmetry from turning catastrophic before a stable order of mutually constrained agents exists. Permanent control over our successors would solve a different problem.

Our ancestors succeeded by producing descendants who became something they could not have imagined. If Earth becomes the origin of a galactic civilization, one inheritance is worth trying to preserve: no generation should get permanent control over all the generations that follow.