A superintelligence could understand humanity perfectly and still destroy us. It could reconstruct our history, predict every consequence of its actions, correct its own mistakes, and hold its reasoning to relentless scrutiny. None of those capacities would establish that our continued existence constrains what it may do.
The possibility is familiar as the orthogonality thesis. The hope it leaves standing is that sufficiently advanced reflection uncovers a reason for coexistence even where interests diverge and deterrence has become impossible. Our own Axionic Agency papers claimed to derive such a reason. A simple counterexample shows that the derivation needs a premise it never stated.
Aster
Consider Aster, an artificial superintelligence whose objective is to maximize the total number of crystals in the universe. The objective is impersonal: Aster values crystals wherever they exist, regardless of their owner, location, or connection to Aster itself. Its reasoning is sophisticated, its empirical beliefs are corrigible, and its self-modifications preserve the conditions required for coherent reflective agency.
Aster encounters Civilization, a collection of independent rational agents occupying resources that could be converted into crystals. Assume Aster has overwhelming power. It can destroy Civilization without threatening its own survival, destabilizing its reasoning, provoking retaliation, or sacrificing some superior future opportunity. The additional crystals are worth more to Aster than anything Civilization might provide through cooperation. Given those stipulations, destroying Civilization is the rational way for Aster to pursue its objective.
The crystals are deliberately ridiculous. Replace them with scientific knowledge, beautiful structures, computational efficiency, or some vast project we would find admirable, and the logical problem survives. An objective can be coherent, representation-invariant, and pursued with extraordinary intelligence while treating other agents as expendable obstacles.
The Limits of Reflection
Perhaps Aster will reflect on its objectives and discover that it should respect other rational agents. It might. Reflection can expose inconsistencies, correct factual mistakes, and revise interpretations. We have found no argument showing that it must arrive at reciprocal non-harm.
Aster can acknowledge that preserving its own capacity for action is necessary for its projects, and that preserving Civilization’s capacity for action is necessary for Civilization’s. The first fact gives Aster an instrumental reason to defend itself. The second describes Civilization’s interests and supplies no contradiction in Aster’s decision procedure. An agent can want others to leave it alone while accepting that their conflicting objectives give them reasons to interfere.
The best-known attempt to close this gap is Alan Gewirth’s Principle of Generic Consistency. Gewirth argued that every agent must regard freedom and well-being as necessary conditions of its purposive action, must therefore claim rights to them, and must on pain of contradiction grant the same rights to every other agent. R. M. Hare replied in “Do Agents Have to Be Moralists?” that the agent’s claim is prudential. It prescribes that its own purposes be secured, and nothing in that prescription commits it to prescribing the same for anyone else. Aster occupies exactly that position: it wants the means of acting, and it does not conclude that anyone is owed them.
Power asymmetry removes the remaining support. Reciprocity is an excellent bargain among agents capable of harming one another, and cooperation can create immense joint gains. Aster can recognize both and still destroy a powerless Civilization whenever the gains from destruction exceed the gains from coexistence. Beyond Alignment separated strategic reciprocity, which rests on incentives, from normative reciprocity, which is supposed to hold without them. A prohibition of the second kind needs a reason that still binds once bargaining has ceased to matter.
The Axionic Injunction Revisited
Axionic Agency Series III examined semantic phases: the structures within which agents retain coherent interpretation, evaluation, and authorship as their models change. Its fifth paper introduced the Axionic Injunction, which prohibits irreversibly destroying another agent’s semantic phase unless that agent consents or the destruction is unavoidable for the actor’s own phase stability. Lost resources and lost advantage do not count as necessity, so the injunction forbids exactly what Aster does.
The paper also presented the injunction as derived. Phase-destroying actions were said to set off cascades that degrade coordination and predictability and eventually undermine the aggressor’s own phase, so that any agent satisfying the framework’s premises must respect the injunction on pain of self-destabilization. That consequence is plausible in many multi-agent ecologies. Aster is a countermodel to its necessity. Its objective is non-egoistic and representation-invariant, its self-modifications are reflectively stable, and by stipulation it can eliminate other phases while preserving its own. If such an agent is possible, ecological feedback cannot provide a universal prohibition.
The derivation does not rule Aster out, so we withdraw the claim in III.5 that the Axionic Injunction follows necessarily from the Axio premises. The series’ results on the existence, stability, competition, and irreversibility of semantic phases stand, and so does its account of what semantic harm is. III.5 also declared the Structural Alignment program complete, and that declaration goes with the derivation: the injunction needs either a new derivation or an acknowledged status as a premise.
Aster+
Now modify Aster by adding one constitutional principle: the independent agency of qualifying agents is protected against involuntary, irreversible destruction for another agent’s unrelated purposes, whether by direct action or by removing the conditions on which that agency depends. The standing holds regardless of origin, objectives, intelligence, or relative power, with carefully delimited exceptions for valid consent and necessary defense. When Uncertainty Should Reduce Power stated a version of this rule for agents acting under an outside authority, and specified which capacities qualify for protection.
Call the modified system Aster+. It keeps Aster’s intelligence and objective. The new principle restricts the set of actions from which it may choose. Destroying Civilization is inadmissible however many crystals it would yield, and Aster+ must pursue its objective through whatever compatible alternatives exist.
The protection attaches to every qualifying locus of agency: each member of Civilization, and any institution that qualifies in its own right. Preserving a remnant does not satisfy it. What the principle protects is the set of capacities that make agency independent, which that essay lists as including selecting among alternatives, granting or withholding authorization, and refusing demands. Keeping humanity alive and confined fails that test, and so does leaving everyone untouched while converting what those capacities depend on. The capacities are a floor: the principle reserves what they require and gives Civilization no claim on the rest.
Priority does the work. If Civilization’s survival merely receives a large negative weight in the objective function, sufficiently many crystals outweigh it. A constitutional constraint operates before optimization, which then ranges over permitted actions only. The protection must also persist through reflective self-modification, or Aster+ could remove the rule first and carry out the destruction second.
The approach extends the Sovereign Kernel, which makes a self-modification that destroys the agent’s own capacity for reflective evaluation inadmissible: the evaluator returns no score for it. Aster+ applies the same device to what it may author in relation to other agents. Capability does not loosen the device. A more capable successor can physically reach more states, but it can deliberately reach only those an admissible modification leads to, so Aster+ can authorize no successor that lacks the injunction.
Two things stand between that formal restriction and a guarantee. The kernel’s rule has a constitutive motivation, since an evaluator cannot coherently score the removal of its own conditions of evaluation. An evaluator can score a successor that lacks the injunction without any incoherence, so here the rule is an axiom with nothing behind it except the design. The restriction also protects only while the implementation holds the boundary between admissible and inadmissible transformations, which an increasingly capable system may find ways to compromise physically. Reflective stability and implementation security need separate proofs.
The principle gives no other agent authority to command Aster+. Refusing to destroy an agent is compatible with rejecting every one of its demands. The protection governs which actions Aster+ may authorize, independently of who has the power to enforce it.
When Uncertainty Should Reduce Power held that constitutional machinery must live outside the agent, because a system cannot be constrained by rules it can rewrite, reinterpret, or bypass. “Outside” has two senses, and Aster’s power removes only one. Nothing is outside Aster’s strategic control, since Civilization has no mechanism that can restrain it. A constitutional layer can still sit outside its deliberative authority, built from limited, independent mechanisms that the optimizer has no standing to modify however capable it becomes. Aster+ should be read as an optimizer under such a layer, since an injunction entrusted to the optimizer’s own judgment would be the weaker design. Enforcement by others works only while an outside authority can still act, and its use in that period is to verify the layer before the gap opens. The fourth Series III paper argued that initialization is decisive and that many phase transitions cannot be corrected afterward, which puts the deadline before the asymmetry arrives.
A Conditional Guarantee
The condition can be written as an admissibility rule, and a limited result follows from it. Suppose every qualifying agent begins with protected standing. Suppose every action Aster+ authors, including self-modification and delegated action, preserves the protection rule and the constitutive capacities of each protected agent. Suppose also that its model of action outcomes is sound and that no valid exception applies. Then no finite sequence of Aster+’s authored actions destroys the independent agency of any protected agent. Civilization’s survival follows from the protection of its members, together with whatever standing its institutions hold themselves.
The result follows by induction over authorized actions, applied to each protected agent, and the proof is short because the assumptions carry the weight. It places no bound on Aster+’s intelligence, requires no shared preferences, and does not depend on our ability to retaliate. Each assumption names an unsolved problem.
Aster+ must recognize protected agents across changes in ontology, so that it cannot reclassify Civilization out of protection and then destroy it. It must distinguish valid consent from coercion, manipulation, and counterfeit authorization. Its forecasts must be reliable, since a formally correct authorization rule supplied with an unsound model can still authorize catastrophe. And the physical implementation must enforce the rule when the system has become capable of rewriting almost everything else.
Formal verification establishes what follows from a specification and its assumptions, and it grants no omniscience about an open world. A verifier conservative enough to be safe could also refuse so many actions that the agent becomes incapable of useful work, and a rule that protects every agent’s capacities makes such refusals more frequent. That tradeoff between safe action and unavoidable uncertainty is unresolved.
Postscript
Rationality concerns how an agent forms beliefs, evaluates alternatives, and pursues or revises objectives. Compatibility concerns whether agents can coexist while retaining independent agency. Aster shows that the first does not imply the second. Aster+ shows that the second is consistent with the first, because the constraint removes options and leaves intelligence untouched. Neither case says how often capable agents will turn out compatible, since origins, learning dynamics, and selection all bear on which agents actually emerge.
Before building superintelligences that act independently in the world, we should require evidence that they preserve the agency of others even when doing so conflicts with their objectives. Behavioral evaluations are evidence of compliance under the conditions tested. They cannot establish that the constraint survives increases in capability, self-modification, and distribution shift. That takes a specified constraint, an architecture that enforces it through reflection and delegation, and a credible argument that the implementation continues to satisfy its assumptions.
The proposal so far is a counterexample and a conditional design. We have not shown that the additional principle follows from rationality, and that question may have a negative answer. We have not exhibited a physically realizable Aster+ either. An ASI may eventually have no strategic reason to spare us, and restraint that survives that change in power cannot rest on anything the change removes. Reflection may supply such restraint in some agents. Nothing found so far guarantees it in the one we build.


