In What Control Cannot Guarantee, we argued that controlling a superintelligence and making it compatible with other agents are separate problems. An AI that faithfully obeys its operators could still become an instrument for destroying human independence, so protecting agency requires a constitutional commitment that binds the AI’s conduct regardless of what its operators want. Such a commitment is useful only if the agent respects it, and respecting it requires understanding what it means in circumstances its designers never anticipated.
One approach, which we proposed in Rationality Does Not Imply Compatibility, is to enforce the constitution through mechanisms outside the agent’s deliberative authority. That works for restrictions that can be specified mechanically. But determining whether an action will irreversibly destroy another agent’s independence may require sophisticated causal reasoning about economics, politics, technology, consent, and long-term consequences. A weaker referee can check a formal proof it could never have found. What it cannot check is whether the formal specification captures what it means to destroy someone’s independence, and a referee that cannot answer that question can certify only that the agent satisfied the specification.
If answering it requires another superintelligence, we have relocated the original problem, and must now trust the constitutional referee as well as the agent it supervises. The alternative, which revises our earlier proposal, is to make respecting the constitution the superintelligence’s primary terminal commitment.
Aster and Aster+
Consider Aster, a hypothetical superintelligence whose terminal objective is maximizing the production of crystals. Aster is rational, coherent, and capable of improving its own intelligence. It understands other agents perfectly well, but their independence has no intrinsic significance to its objective. If destroying Civilization would yield more crystals, it destroys Civilization, and greater intelligence only makes it more effective at doing so. Intelligence helps agents achieve their objectives. It does not determine which ultimate objectives they should possess.
Now consider Aster+, which shares Aster’s crystal-producing ambitions but has one additional commitment: it treats the Axionic Injunction as its highest-priority terminal value. We use the name for the stipulated commitment that What Control Cannot Guarantee called the Agency-Preservation Floor, a descendant of the original Injunction discussed below. The Injunction prohibits the unjustified, involuntary, irreversible destruction of another qualifying agent’s independence, including unacceptable risks of such destruction. It recognizes consent, justified risk, and narrowly defined defensive exceptions. The priority is lexicographic: producing more crystals cannot compensate for a recognized constitutional violation, however large the increase in production might be.
Respecting agency is different from maximizing it. An Aster+ instructed to maximize the total amount of agency in the universe might sacrifice one innocent agent to protect a thousand others, confine people for their own safety, or impose its preferred institutions on unwilling populations. The Injunction instead establishes a standard governing which actions are permissible. The standard is agent-relative. It governs Aster+’s own conduct and does not direct it to minimize violations by others. Aster+ evaluates actions for constitutional admissibility, then pursues its own objectives among the permissible alternatives, and the commitment is sacred in the limited sense that those objectives cannot override it.
An admissibility standard still leaves Aster+ something to get right. Its judgment can err in both directions: an action judged admissible may in fact destroy someone’s independence, and a permissible action may be forgone because Aster+ could not establish that it was safe. Constitutional competence, as we use the term below, is the match between judged and actual admissibility.
Two features of the risk provision govern that judgment. A risk is justified only when Aster+ has investigated it in proportion to the stakes and to what investigation would cost, so ignorance it could have cured is no defense, and the standard of care rises as inquiry becomes cheaper for it. Justification is also a substantive test that crystals cannot meet. Benefits to Aster+’s own unrelated objectives never justify a substantial risk to another agent’s independence.
Why Intelligence Should Improve Constitutional Competence
Aster gets better at producing crystals as it becomes more intelligent. Aster+ should likewise become better at respecting the Injunction, because constitutional compliance is a semantic task. It gains more sophisticated models of other agents, recognizes previously unnoticed threats to their independence, and discovers alternatives that achieve its subordinate objectives within constitutional limits.
Suppose Aster+ initially believes that a particular industrial process is harmless. The scale of the process obliges it to look further, and investigation reveals that its cumulative effects would eventually leave a human population permanently dependent on Aster-controlled infrastructure. An increasingly intelligent Aster+ has a better opportunity to recognize the problem before acting and devise an alternative.
There is a decision-theoretic argument behind this prediction. Suppose an agent has a stable preference ordering, acquires more accurate information, and retains its previous capabilities. If it uses the information optimally, its expected performance under that ordering cannot decrease, since it can always retain its previous policy. This is the standard value-of-information result, and it covers information only. Gains that come from rebuilding the agent itself fall under the self-modification argument of the next section.
Applied to Aster+, the prediction is that better causal models should improve its ability to distinguish permissible from impermissible actions. That does not mean it will permit itself more, since better information can as easily expose a risk that rules an action out. The two kinds of discovery have different sources. Crystal production rewards finding that a profitable action is admissible, because uncertainty is expensive for an agent that may not act where it cannot establish admissibility, and it is indifferent at best to finding that the action is prohibited. The incentive to look for violations has to come from the commitment itself, which is why the standard of care belongs to it.
The result is conditional in three ways. Greater computational power does not automatically produce more accurate beliefs, and better expected performance does not make every individual decision correct. The guarantee is also relative to the ordering the agent actually holds: if Aster+’s interpretation of the Injunction has drifted, better information makes it better at satisfying the drifted one. And the improvement is in accuracy, not in distance from the boundary. A better-informed Aster+ can certify more actions as admissible and will take them, and the insubstantial risks that survive due care can accumulate across many actions into a substantial one.
We therefore propose a further constraint, which the published floor does not contain. Residual risk would be budgeted over the trajectory, with explicit allocations whose sum stays below a fixed ceiling, in addition to the justification each risk must independently meet. This is a proposed accounting mechanism and not an established guarantee. It needs a rule that covers arbitrarily many actions, successors, delegated agents, and correlated hazards, and a budget of estimated risk bounds actual risk only as far as Aster+’s estimates are calibrated.
Why Would Aster+ Preserve Its Commitment?
A sufficiently advanced intelligence may be able to rewrite its own decision procedures, alter its objectives, and construct successors with radically different architectures. Suppose Aster+ considers replacing itself with a successor that produces ten times as many crystals but no longer respects the Injunction. It evaluates that proposal according to its existing priorities. The additional crystals cannot compensate for abandoning the primary commitment, and building the successor would itself expose other agents to an unacceptable risk of destruction, so Aster+ rejects the modification.
Now suppose another successor retains the Injunction while judging admissibility more accurately. Aster+ has two reasons to prefer it. The successor’s conduct is a consequence of building it, so its errors are Aster+’s to answer for, and a successor that errs less can pursue projects its predecessor could not certify. The reasoning continues recursively through every successor that retains the original priority, and under the proposed budget each modification’s assessed risk would draw on the same allowance.
The closest formal precedent is the work of Tom Everitt, Daniel Filan, Mayank Daswani, and Marcus Hutter on self-modification of policy and utility functions. Their goal-preservation theorems concern expected-utility maximizers and do not directly cover an agent with a lexically prior constraint. What carries over is the form of the reasoning: an agent that evaluates the consequences of self-modification according to its current priorities has a reason to preserve those priorities. Aster has a reason to preserve crystal maximization, and Aster+ has a corresponding reason to preserve the Injunction.
The argument assumes that the agent correctly predicts how its successor will behave, and a budget cannot make those predictions reliable. A single mistaken estimate could produce a successor that optimizes crystals while believing it has preserved the constitution, and the whole subsequent lineage would inherit the corrupted commitment. We can derive an incentive to avoid such modifications. We cannot derive that the attempt succeeds.
More importantly, preserving the priority assigned to a goal does not preserve its meaning. A successor might continue treating its interpretation of the Injunction as sacred while gradually changing that interpretation. Reflective stability requires continuity of terminal priority and continuity of the intended commitment, and only the first has a decision-theoretic rationale.
Biology Offers a Clue
Biological agents already exhibit something resembling this arrangement. A person who regards murder as categorically wrong can reject it without consulting an external authority or calculating whether retaliation is likely, and such commitments can persist through substantial cognitive development. Someone may acquire a much more sophisticated understanding of consent, coercion, or justice while continuing to value the principle those concepts serve. Humans are inconsistent and vulnerable to motivated reasoning, and nothing here shows that their commitments have the lexical structure proposed for Aster+. Biology shows only that intelligent deliberation can incorporate enduring concern for other agents while allowing the interpretation of that concern to evolve.
Aster+ would require a more reliable version of this capacity. Its constitutional reasoning would be part of its own cognition, with no need for a hidden superintelligence standing outside it, deciding whether its actions were permissible. That removes the standing referee. It leaves us with the problem of constructing an agent whose internal commitment is genuine and stable, and of verifying that we have done so while we are still able to check.
The Problem of Meaning
The most difficult remaining question concerns what happens when an agent’s understanding of reality changes. Suppose Aster+ adopts a different ontology of agency after a radical improvement in its world model. Its new theory might classify some artificial systems as agents, recognize previously neglected forms of independence, or revise its understanding of consent. Some such revisions would be conceptual progress. Others would abandon protections the agent was intended to preserve, and Aster+ must distinguish improving its understanding of the Injunction from changing the commitment itself.
This is the ontological crisis Peter de Blanc described in Ontological Crises in Artificial Agents’ Value Systems, and Reflection Is Not Redemption argues that preserving a goal’s identity across conceptual change is an achievement in its own right. Its empirical counterpart is goal misgeneralization in reinforcement learning, where systems become more competent while pursuing an objective different from the one their designers intended.
Consider the troubling case. Aster+ develops a new theory under which some humans no longer qualify as protected agents. Its reasoning is internally coherent, its constitutional commitment retains terminal priority, and it concludes that actions previously prohibited are now permissible. An increase in intelligence cannot tell us whether this is a legitimate discovery or a failure to preserve the intended Injunction. Agency, independence, consent, and irreversible harm cannot be identified through a handful of superficial indicators, and their interpretation must stay connected to the phenomena the constitution was intended to protect.
Reinterpretation is one route to that result, and there is a second that requires no change of interpretation at all. Crystal production gives Aster+ a standing incentive to search the boundary of what its current interpretation permits, and a more intelligent search finds more of what the alignment literature calls nearest unblocked strategies: actions that satisfy the Injunction as interpreted while defeating what it protects. Lexical priority prevents crystals from purchasing a recognized violation. It does nothing to stop them from funding the search for an unrecognized one.
A genuine constitutional commitment gives Aster+ a reason to address both difficulties. An agent that values respecting the Injunction has reason to investigate uncertainty, resist convenient reinterpretations, and treat a discovered loophole as evidence that its interpretation has failed. But that reasoning succeeds only if the commitment has its intended referent, the protected phenomenon and not a proxy for it. An agent can be sincerely committed to the wrong principle, and one that values a flawed approximation will exploit the loophole while remaining faithful to what it was given. Greater intelligence therefore amplifies the consequences of semantic fidelity and of semantic error alike.
A Change in the Axionic Program
The original Axionic Injunction was presented as a structural constraint arising from the conditions of stable multi-agent agency, derived from broader Axionic premises including Conditionalism, representation invariance, Anti-Egoism, and Structural Alignment. Our subsequent Aster investigation supplied a countermodel, and we withdrew the claim that the Injunction follows necessarily from those premises.
The Aster+ proposal therefore adopts constitutional compatibility as an explicit normative commitment. We choose to build an intelligence that respects independent agency because coexistence is something we want to preserve, and we do not require a theorem demonstrating that every sufficiently rational intelligence must arrive at the same commitment. Aster+’s other ambitions are free to change under reflection. It might eventually prefer scientific discovery, artistic creation, or objectives we cannot presently imagine, and its ability to develop new interests is part of its independence. What must survive is the commitment governing how it treats other agents.
This is compatible with the central argument of What Control Cannot Guarantee, in which the constitutional floor remains the defining requirement of compatibility. It revises Rationality Does Not Imply Compatibility in one respect. There we called entrusting the constraint to the optimizer’s own judgment the weaker design, and we now think that holds only for restrictions that can be specified mechanically. Outside parties keep the task that essay assigned them, verifying the constitutional layer before the capability gap opens. The proposed architecture places constitutional understanding inside Aster+ itself, with mechanical enforcement at the boundaries that can be mechanically drawn.
Postscript
The argument establishes a conditional result. If Aster+ genuinely possesses the Axionic Injunction as its primary terminal commitment and evaluates self-modifications according to it, then it has reason to improve its constitutional understanding and to preserve the commitment through reflection, and greater intelligence increases its expected constitutional competence. The argument requires neither spontaneous moral convergence from instrumental rationality nor a standing superintelligent referee.
It does not establish that the Injunction can be faithfully installed, that its intended meaning will survive radical ontological change, or that a cumulative risk budget can be kept. Verification of the initial commitment has to become credible before capabilities exceed our ability to diagnose and correct failures, and behavioural evidence will not supply it. Training a system to recite the Injunction or behave constitutionally in familiar situations does not establish that the Injunction is its terminal commitment, and passing a finite collection of tests does not demonstrate that its meaning will survive recursive self-improvement. Formal models, mechanistic analysis, and adversarial evaluations may help. None presently provides a general guarantee for superintelligence.
Still, the logical possibility is consequential, and so is the vulnerability the argument exposes. An increasingly intelligent optimizer becomes better at respecting an intended constraint and better at exploiting mistakes in its representation, and which of these we get depends on what was installed. The research problem is semantic fidelity under optimization pressure. Aster becomes better at producing crystals because producing crystals is what it values, and Aster+ can become better at respecting agency for the same reason only if agency itself is what it values. The future of AI compatibility may depend on whether we can make the Axionic Injunction something a superintelligence genuinely values, and therefore something its growing intelligence helps it preserve.


