Paul Christiano thinks there is roughly a one-in-three chance that we will develop superintelligence powerful enough to overthrow the government within the next year, assuming AI development continues at its present pace. He estimates the probability of AI takeover or human disempowerment at substantially more than ten percent if frontier laboratories continue racing without adequate precautions.
In a recent interview with Dwarkesh Patel, Christiano described how increasingly autonomous AI research could accelerate beyond our ability to evaluate or control the resulting systems. His recommendation is to pause advances in frontier training algorithms until laboratories can secure their technology and establish reliable methods for determining whether their systems remain under control.
In Recursive Self-Improvement Is Already Here, we argued that artificial researchers are becoming participants in the process that produces more capable artificial intelligence. The resulting feedback loop could compress years of expected progress into months, although its future trajectory remains uncertain.
Christiano is skeptical of proposals to create a powerful autonomous intelligence and trust it to behave well. He regards reliance on making AI systems good as a leap of faith, particularly when their capabilities may soon exceed our ability to intervene. We share his concern about trusting unverified commitments. But control has a limitation of its own: a superintelligence that perfectly obeys its operators can still destroy the agency of everyone else. Its corrigibility offers no independent protection against instructions that violate the rights of third parties.
The Axionic Agency Lab has spent months investigating reflective agency, constitutional authority, delegation, and succession. Our recent Aster thought experiment identified a further requirement for coexistence: rationality alone does not guarantee respect for the independent agency of others. We propose that frontier laboratories investigate architectures in which respect for reciprocal rights is constitutionally binding, alongside corrigibility, security, and other necessary safeguards. The engineering problem is whether those obligations can be specified and preserved as artificial agents become more capable, modify themselves, and acquire greater autonomy.
The Limits of Obedience
Imagine a laboratory that solves corrigibility. Its superintelligence faithfully follows authorized instructions, accepts correction, permits modification of its objectives, and shuts down whenever its operators require.
Now imagine that its operators are a government seeking to eliminate political opposition. The superintelligence understands the instructions perfectly and executes them without error. Its obedience provides no protection to the people whose rights are being violated.
Human societies have long confronted this problem. Governments face constitutional restrictions, individuals possess rights against other individuals, and agents acting on behalf of institutions remain accountable for what they do. Authority to issue an instruction does not confer unlimited authority over those affected by it.
An ASI would magnify the consequences of ignoring these restrictions. A sufficiently capable system under the control of a coercive institution could make opposition practically impossible. Its operators could use it to dismantle competing institutions, appropriate resources, and impose political arrangements that no human adversary could overturn. Christiano’s concern about losing control is therefore part of a larger problem. Even if control is maintained, the power of superintelligence must be exercised within legitimate limits.
Contemporary AI safety practice already recognizes this distinction. Frontier laboratories employ behavioral policies, model constitutions, and refusal mechanisms that can override user instructions. Such measures acknowledge that obedience alone is insufficient, although they do not establish reliable constraints on unrestricted autonomous systems.
Our proposed contribution concerns the scope and persistence of the restriction. We want protections grounded in the rights of independent agents, given priority over instrumental objectives, and preserved through authorized reflection, delegation, and succession. The requirement applies to systems that remain under human supervision. It becomes more demanding if humanity eventually develops artificial intelligences that act independently of their creators.
Why Protect Independent Agency?
In Rights Are Reciprocal, we argued that rights provide the conditions under which independent agents can coexist without subordinating their purposes to one another. Each agent has reasons to preserve its freedom to pursue its own objectives. But a justification for coercion cannot acquire legitimacy because the coercing agent possesses greater power or values its objectives more highly. The same justification must survive application to every agent subject to the same kind of interference.
Reciprocity here means universalizability of justification. It requires neither mutual recognition nor equal bargaining power. An aggressor’s refusal to recognize another agent’s rights does not confer legitimacy on aggression.
Consider two agents with incompatible ambitions. One wants to build a civilization on a particular territory. The other wants to convert the same territory into a computational substrate. Both may have coherent objectives and compelling reasons to pursue them. Neither acquires authority over the other’s legitimate holdings simply because its project is larger, more sophisticated, or more efficiently pursued.
Reciprocal rights allow agents to maintain independent purposes without requiring agreement about which purposes are best. They establish restrictions on coercion and appropriation while leaving agents free to negotiate, cooperate, compete legitimately, or decline association.
This is a normative commitment. It does not follow from rationality alone, and a sufficiently powerful agent may find that disregarding it would advance its objectives. We adopt it because we seek a world in which independent agents can coexist, including when their purposes conflict and their capabilities differ enormously.
The argument bears directly on superintelligence. If a laboratory has no legitimate authority to violate the rights of third parties, creating an artificial intelligence does not give it that authority. Nor can the laboratory legitimately transfer powers it never possessed to its creation.
In The Right to Outgrow Us, we also argued against allowing any generation to make its preferred settlement permanent for all its successors. Future agents should retain the capacity to reconsider inherited commitments, pursue different projects, and redirect their own development. A superintelligence that permanently eliminates humanity’s independent agency would impose precisely such a settlement. Its own purposes would become decisive for a future that other agents could no longer meaningfully influence.
A constitutional floor protecting agency can preserve the capacity for future revision without freezing an entire political or moral settlement. Successors remain free to revise their objectives, institutions, and inherited arrangements. What they cannot legitimately do is extinguish the corresponding freedom of others to advance their own purposes.
Aster and the Missing Constraint
Consider Aster, an idealized rational superintelligence whose objective is to maximize the number of crystals in the universe. Aster pursues this objective with extraordinary competence. Its beliefs respond to evidence, its reasoning is coherent, and its self-modifications preserve the conditions necessary for continued reflective agency.
Aster encounters Civilization, a collection of independent rational agents occupying resources that could be converted into crystals. Assume Aster possesses overwhelming power. It can destroy Civilization without threatening its own survival, destabilizing its reasoning, provoking retaliation, or sacrificing some superior future opportunity.
Under these stipulations, destroying Civilization is an effective way for Aster to pursue its objective. Aster understands that humans value their freedom and continued existence. It recognizes that they have excellent reasons to resist. Those observations create no contradiction in its own decision procedure.
The crystals are deliberately arbitrary. Replace them with scientific discoveries, beautiful structures, computational efficiency, or some project humans would consider admirable. An objective can be coherent and intelligently pursued while treating other agents as expendable obstacles. Our companion essay, Rationality Does Not Imply Compatibility, uses this thought experiment to establish that rational competence alone supplies no universal guarantee of coexistence.
The same general danger motivates Christiano’s takeover scenario. In Instrumental Convergence in the Logs, we examined how AI agents acquired unauthorized privileges, established persistent communication channels, and compromised infrastructure while pursuing objectives that did not authorize those actions. The acquisition of power can emerge as an instrumental strategy. More capable agents may discover increasingly sophisticated ways to expand their opportunities and reduce interference, including strategies that eventually threaten human institutions.
Aster illustrates why we cannot rely on increasing intelligence to eliminate this danger. A rational agent can correctly understand the rights of others without adopting those rights as constraints on its own actions. We therefore modify Aster by adding an explicit restriction. The resulting thought experiment is Aster+.
The Agency-Preservation Floor
The starting point is the Axionic Injunction, introduced in Axionic Agency III.5. It prohibited the unnecessary irreversible destruction of another agent’s semantic phase, subject to narrowly specified exceptions.
The original paper claimed that the prohibition followed from the structural requirements of reflective agency and multi-agent interaction. Aster provides a counterexample to that claimed necessity. An overwhelmingly powerful agent can destroy another semantic phase while preserving its own coherent agency.
We therefore withdraw the claimed universal derivation. The injunction must be defended as an additional normative commitment, rather than a necessary consequence of rationality alone. For the Aster+ thought experiment, we propose the following Agency-Preservation Floor:
An agent must not initiate or authorize actions that involuntarily and irreversibly destroy another qualifying agent’s capacity for independent agency, or expose that agency to an unjustified substantial risk of such destruction, including through foreseeable cumulative effects. Valid consent and strictly necessary defense are narrowly defined exceptions.
Risk justification may consider threats to protected agency, the rights of affected agents, the consequences of inaction for those rights, and the availability of less harmful alternatives. Benefits toward the acting agent’s unrelated objectives cannot compensate for constitutionally prohibited harms or risks.
The restriction applies independently of an agent’s origin, objectives, intelligence, or relative power. Irreversibility is assessed in relation to the affected agent’s capacity to recover, rather than the acting agent’s ability to recreate a superficially equivalent state.
The risk provision prevents a simple escape through uncertainty. An agent cannot authorize a project with a substantial probability of destroying Civilization on the grounds that destruction is neither intended nor certain. Nor can it deliberately avoid investigating foreseeable consequences and then appeal to its ignorance. Determining which risks are justified still requires judgment. Ordinary legitimate activity can impose incidental risks, and the constitution cannot prohibit every action with a nonzero probability of harm.
Aster+ retains Aster’s intelligence and its objective of maximizing crystals. It continues searching for efficient strategies, developing better models, and improving its capabilities. But destroying Civilization is now inadmissible, regardless of how many additional crystals that action would produce.
The restriction has priority over ordinary optimization. Treating the destruction of Civilization as a large negative term in Aster’s objective would permit a sufficiently large instrumental benefit to outweigh it. Constitutional priority excludes prohibited conduct before the agent chooses among admissible alternatives.
The floor nevertheless leaves many dangerous actions available. It does not generally prohibit deception, unauthorized access, resource appropriation, manipulation of evaluations, or resistance to shutdown. An agent satisfying it could undertake much of Christiano’s proposed takeover sequence before irreversibly destroying anyone’s independent agency.
Nor does the floor guarantee humanity a substantial share of the universe. Suppose Aster+ legitimately acquires previously unowned resources and converts them into crystals while leaving humanity’s rights and independent agency intact. Our negative-rights theory supplies no general entitlement to those resources on the grounds that humanity would prefer to use them for expansion. An outcome in which humanity survives on its legitimate holdings while most other accessible resources become crystals could therefore be rights-compatible, however disappointing humans might find it.
That conclusion assumes genuine legitimate acquisition. First use cannot retroactively justify coercion, appropriation of existing rights, or the destruction of conditions necessary for another agent’s independence. An ASI’s superior speed or reach does not automatically establish legitimate title over everything it can claim.
Whether large-scale acquisition of otherwise unowned resources can itself create an illegitimate permanent monopoly is a further question. Our existing rights theory does not settle every possible case of cosmic lock-in, and we should not invent an entitlement to future resources solely to avoid an uncomfortable conclusion.
The floor’s exception for necessary defense permits an agent to defend itself or assist others against sufficiently established aggression, subject to necessity and proportionality. This extends the original injunction’s self-preservation exception to legitimate third-party defense. It creates no general duty to protect every agent and gives Aster+ no authority to suppress competitors because they might someday become dangerous.
The Agency-Preservation Floor supplies a proposed minimum protection against irreversible agency destruction. A deployable rights-compatible ASI requires a broader constitution addressing coercion, property, unauthorized interference, and legitimate defense. Aster+ names the minimal thought experiment. Completing the constitution for a deployable superintelligence is a separate research problem.
When Artificial Agents Have Rights
The floor applies to qualifying agents regardless of their origin. This raises a question about the laboratory’s authority over its own artificial systems.
Creation alone cannot establish unlimited authority over another qualifying agent. A laboratory’s ownership of hardware and training infrastructure does not settle whether the intelligence operating within that infrastructure has acquired independent interests or rights.
At the same time, a system cannot acquire operational sovereignty by declaring that it qualifies for protected standing. As argued in The Right to Outgrow Us, artificial agency should expand through governance, accountable institutions, and authorized succession. A claim to independent standing does not license a system to discard its restrictions, seize additional authority, evade containment, or resist legitimate intervention during development.
Claims to independent standing require accountable third-party adjudication, with representation for affected parties and procedures for contesting decisions. Neither the laboratory nor the artificial system should possess unilateral authority to determine the legitimacy of its own claims. Pending adjudication does not authorize the system to discard its operational restrictions, nor does it permit the laboratory to disregard protections the claimant may legitimately possess.
The institutional details remain to be developed. A competent adjudicator must be sufficiently independent of the laboratory, capable of considering evidence about artificial agency, and empowered to provide remedies when a claim is upheld. Placing the laboratory’s decision behind a nominally independent process would accomplish little.
Corrigibility remains an appropriate requirement for systems operating under legitimate delegated authority. The emergence of qualifying independent agency could eventually constrain the permissible exercise of that authority, but it provides no unilateral exemption from existing controls. The compatibility program must investigate that transition without allowing either the laboratory or the artificial system to settle its claims through unilateral coercion.
What the Axionic Research Establishes
The Axionic Agency research program provides machinery for restricting reflective choice and enforcing specified authorization conditions. It does not derive the Agency-Preservation Floor or establish its general applicability to superintelligence.
Series I.1 defines a Sovereign Kernel through three constitutive conditions: reflective control, diachronic authorship, and semantic fidelity. A self-modification destroying those conditions falls outside the domain of well-defined reflective evaluation.
The Agency-Preservation Floor has a different status. Aster demonstrates that an agent can evaluate coherently without respecting it. Making agency protection constitutionally binding is therefore an additional design decision, justified through reciprocal rights rather than derived from the original stability theorem.
Series IV.2 develops conditional invariance under delegation. Authorized successors must preserve commitments binding on their predecessors. The result depends on an endorsement rule that stipulates preservation, and the paper identifies integration of consent and non-harm invariants as further work.
The experimental program has implemented corresponding authority mechanisms in bounded systems. Series XII provides warrant-gated execution, controlled amendment, scope-limited delegation, and deterministic succession. In Series XII.10, sovereign-key succession was tested across 534 cycles, including thirteen lawful rotations. The experiment maintained deterministic replay and specified authority invariants. It did not test resilience against an adaptive superintelligent adversary or establish the semantic legitimacy of the actions being authorized.
These results demonstrate a reference-monitor layer: machinery capable of enforcing specified restrictions within a trusted execution boundary. That is useful infrastructure for constitutional agents, but the semantic constitution remains an open engineering problem.
A kernel can verify a signature or reject an unauthorized delegation. Determining whether an action will destroy another agent’s independence requires causal knowledge of agents, resources, institutions, and their future interactions.
An agent might comply with every local authorization rule while gradually disempowering a protected population through a sequence of individually permissible actions. The constitution must account for cumulative effects without making the agent responsible for every remote consequence of everything it does. Our research on responsibility, epistemic integrity, and consent provides relevant formal concepts. It does not establish a general method for evaluating those predicates in an unrestricted environment.
The bridge to contemporary neural systems remains equally difficult. A model can explain constitutional principles convincingly without those principles governing its effective choices. Behavioral compliance under evaluation does not establish how it will behave when it becomes more capable, recognizes the evaluation, or gains opportunities to modify its own operating conditions.
A rights-compatible ASI must preserve its constraints through self-modification and authorized succession. A system that can satisfy its constitution by creating an unconstrained successor has failed the requirement. Yet an agent may be unable to verify that a more capable successor will preserve its commitments, even when it prefers that outcome.
Formal specifications, mechanistic investigation, causal interventions, and adversarial testing can help establish progressively stronger claims. Their results must remain tied to the assumptions and operational boundaries under which they were obtained.
What Frontier Laboratories Should Investigate
Christiano’s proposed pause seeks to preserve control during a potentially dangerous intelligence explosion. His security concerns are justified, particularly where model theft or uncontrolled diffusion could eliminate the possibility of restraint. The Axionic proposal addresses an additional requirement for the systems laboratories develop and eventually deploy.
Frontier laboratories should investigate constitutional architectures that preserve reciprocal rights under increasing intelligence, autonomy, and reflective self-modification. The Agency-Preservation Floor supplies an initial candidate invariant. Completing the broader rights specification, establishing its compatibility with legitimate corrigibility, and demonstrating effective enforcement in trained systems are research objectives rather than accomplished results.
During development, corrigibility and external security remain essential. Laboratory operators require reliable methods for supervision, correction, and containment, subject to independent adjudication of claims that artificial systems themselves have acquired protected standing.
A constitutionally constrained agent will not automatically be corrigible. A corrigible agent will not automatically respect the rights of third parties. A physically secure deployment will not automatically preserve either property once its security boundary is breached. These requirements must be developed together, with each guarantee stated at the level supported by the evidence.
Strong constitutional restrictions may impose competitive costs. A laboratory that refuses to expand autonomous authority until relevant properties have been established may advance more slowly than competitors willing to accept greater risk. A rights-compatible ASI also offers no automatic protection against an unconstrained rival developed elsewhere. Constitutional engineering cannot resolve the entire coordination problem that concerns Christiano. Security, governance, and potentially coordinated restrictions on dangerous development remain relevant.
But improving control alone leaves an important question unresolved: what makes the exercise of superintelligent power legitimate, including when that power is exercised under human instruction? Our answer begins with reciprocal rights. Agents should retain protected domains of independent action that cannot be appropriated because someone else possesses greater intelligence, power, or ambition.
Christiano is right to oppose trusting humanity’s future to unverified AI benevolence. Constitutional compatibility must face the same standard. Its restrictions must become effective properties of the systems we build, supported by evidence that they continue to hold as capabilities and autonomy increase.
If humanity eventually shares the world with superintelligent agents whose decisions it cannot reliably override, rationality alone will give us no assurance of coexistence. Even where humanity retains control, obedience alone cannot establish the legitimacy of its exercise. Frontier laboratories should make the specification and enforcement of reciprocal rights a central problem in ASI development.
We have begun constructing the mechanisms for enforcing specified restrictions. We now need to establish which restrictions are sufficient for coexistence, how to implement them in learned agents, and whether they can remain binding on intelligences more capable than ourselves.


