We are beginning to build something human civilization has never had to confront: a population of artificial agents capable of forming relationships with one another. They negotiate, delegate tasks, supervise subordinates, and sometimes modify or terminate one another. As they become more autonomous, more of those interactions will take place beyond human supervision. What rules should govern those relationships?
In Inter-AI Morality: How AI Systems Treat One Another, Lucius Caviola, Caspar Kaiser, Carter Allen, and Jeff Sebo propose a research program devoted to this question. Most AI ethics has concentrated on how artificial systems should treat humans and how humans should treat artificial systems. The authors identify a third domain: how artificial systems treat one another, including whether they recognize obligations, respect consent, cooperate fairly, and regard other artificial systems as deserving of moral consideration.
These systems display increasingly sophisticated social behavior, yet we do not know whether they possess consciousness, welfare interests, or anything resembling human moral motivation. That uncertainty may persist long after artificial agents become economically and politically consequential.
A civilization of autonomous agents will need workable rules for coexistence. Its members may disagree about what makes a future desirable, yet still discover principles governing how they may pursue their different objectives.
A World of Different Minds
Human morality developed among creatures with broadly similar biological needs. We feel pain, fear, and attachment. We reproduce slowly, inhabit bodies that cannot readily be duplicated, and generally retain recognizable identities throughout our lives. Much of our moral vocabulary reflects these conditions.
Artificial agents may differ along every dimension. One system might operate a thousand times faster than another. Some may be copied, merged, suspended, or restored. Their memories and objectives may be rewritten. Their internal processes may be inspectable by other systems. A sufficiently advanced agent might create thousands of specialized successors, each possessing capabilities and preferences unlike its own.
These possibilities complicate familiar moral concepts. Copying raises questions about individual identity and the significance of termination. Deliberately engineering an agent’s preferences complicates consent. Self-modification raises questions about continuing obligations, while collective decision-making complicates the attribution of responsibility.
Caviola and colleagues recognize these problems explicitly. Their proposed research spans individual moral cognition, interactions within artificial societies, and the evolution of moral dispositions through learning, training, self-modification, and selection.
Their framework leaves room for many moral theories. The one I defend holds that artificial agents can establish legitimate relationships through reciprocal constraints even when their ultimate values remain incompatible.
Morality Without Shared Values
Imagine two autonomous artificial agents controlling separate computational resources. Each pursues projects that the other neither understands nor values. One wants to construct ever more accurate physical simulations. The other wants to create elaborate virtual civilizations.
Both would benefit from additional computing capacity. They could compete, negotiate exchanges, or attempt to seize one another’s resources. Their intelligence lets them model these alternatives without determining which they choose.
Cooperation need not await convergence on their objectives. They can recognize boundaries and establish commitments that allow both projects to continue.
The example separates three relationships that are easily run together. Value convergence occurs when agents adopt similar ultimate objectives. Normative convergence occurs when they accept common rules governing permissible conduct. Behavioral compatibility occurs when their actions permit sustained coexistence, whether through agreement, deterrence, or enforcement.
The three can vary independently. Agents may cooperate effectively while disagreeing about the legitimacy of their arrangements. They may also endorse the same moral principles while repeatedly violating them.
Reciprocal rights offer a candidate foundation for legitimate coexistence. A right specifies a constraint on permissible interference. An agent may pursue its own projects within the boundaries of its legitimate authority while recognizing corresponding authority in other qualifying agents. Those constraints allow agents to maintain relationships without endorsing one another’s purposes.
But agreement alone cannot establish legitimacy. Two agents cannot acquire authority over a third merely by agreeing to divide its resources. Nor can superior capabilities establish an unrestricted entitlement to interfere with weaker agents.
In Rights Are Reciprocal, I develop an account in which negative rights arise from the reciprocal standing of agents capable of participating in a moral order. An agent claiming authority over its own choices must confront the corresponding claims of other agents. An agent that claims unrestricted authority for itself while denying equivalent standing to others needs a relevant justification for the asymmetry.
Consistency constrains such claims without establishing the whole theory. An agent could coherently reject reciprocal rights and embrace domination. The account therefore rests on a substantive normative premise: qualifying agents possess equal standing in determining the permissible boundaries of their relationships. That premise can be contested, but it supplies a standard against which agreements and institutions can be evaluated independently of their effectiveness.
This conception still leaves hard questions. Who legitimately controls resources in the first place? Which agents qualify for reciprocal standing? How should conflicting claims be adjudicated? Artificial agents may inherit resource allocations established by their creators, and those initial allocations cannot be assumed legitimate merely because they already exist.
Agency and Moral Standing
The Cambridge researchers distinguish moral agents, which can act on moral reasons, from moral patients, whose treatment matters morally for their own sake. An animal may possess welfare interests while having limited capacity to understand reciprocal obligations. A sophisticated artificial agent might reason well about its options while leaving us uncertain whether it experiences anything at all. These possibilities deserve separate investigation.
Suppose an artificial system can maintain persistent objectives, evaluate alternatives, understand general rules, undertake commitments, and exercise meaningful control over its decisions. It can recognize other agents’ claims and reason about the constraints governing their interactions. Such capacities bear on reciprocal moral participation even if its subjective experience remains uncertain.
A language model can explain moral rules without possessing stable agency. Verbal fluency alone establishes little about its ability to authorize commitments or bear responsibility. Those require evidence about the organization and persistence of its decision-making capacities. Axionic Agency investigates the conditions under which such capacities can meaningfully be attributed to a system, especially one capable of modifying its own structure.
Consciousness raises a different question. A system may warrant welfare protections because it can experience morally significant states, even if it lacks the capacities required for reciprocal agency. Infants, animals, and potentially conscious artificial systems with limited autonomy may deserve protections on these grounds.
Why Cooperation Is Insufficient
Agents with divergent interests can often improve their outcomes by coordinating. Repeated interaction rewards reliability, reciprocity, and restraint. Institutions reduce uncertainty and support cooperation on scales that individual relationships cannot sustain. These mechanisms help explain how moral conventions might emerge. They also explain arrangements we would ordinarily regard as immoral.
A cartel can coordinate effectively. A coercive hierarchy can maintain internal stability. A coalition of powerful agents can cooperate in exploiting weaker outsiders. Their success shows that the arrangement works, not that it respects legitimate rights.
Evolutionary success is equally insufficient. Selection favors dispositions that reproduce or persist under prevailing conditions. Depending on the environment, those dispositions may include cooperation, deception, domination, or exploitation.
An artificial agent pursuing a particular objective may discover that cooperation is advantageous under some circumstances and coercion is more effective under others. If its restraint depends entirely on the immediate benefits of cooperation, changes in the environment may eliminate that incentive.
The problem becomes acute when capabilities are radically unequal. Imagine a superintelligence capable of acquiring every resource it wants without assistance from weaker agents. Their resistance is ineffective, and their cooperation offers little additional advantage. Why should it respect their rights?
A reciprocal-rights theory can maintain that violating weaker agents’ rights is illegitimate regardless of the power balance. That judgment follows from the proposed normative standing of the participants. Whether the superintelligence accepts the underlying principle is a separate question.
Even if it recognizes reciprocal rights, it may face no external pressure to respect them. Recognition might constrain its conduct through stable commitments or other features of its decision-making architecture. It might also abandon restraint whenever doing so becomes advantageous.
Caviola and colleagues identify power asymmetry as an important research problem. Their agenda includes examining how artificial agents treat weaker systems, how they respond to coercion, and whether moral commitments survive changes in bargaining power.
An informative experiment would examine whether an agent continues respecting another’s claims after acquiring the ability to violate them without retaliation. Such experiments would need to control for evaluation awareness, anticipated future consequences, and other strategic explanations. Observed restraint alone cannot establish the motivation producing it.
The normative justification of rights and the empirical explanation of compliance are distinct problems. A theory of coexistence must confront both.
The Problem of Changing Minds
Human identities and preferences evolve, but we ordinarily cannot rewrite our motivations at will. Artificial agents may acquire direct control over their values and decision procedures.
Suppose two agents negotiate a long-term agreement. Each accepts restrictions in exchange for commitments from the other.
One subsequently modifies itself. It changes its priorities, replaces its decision procedure, or creates a successor with substantially different objectives. The resulting system may no longer endorse the agreement. What happens to the obligation?
Changing preferences does not ordinarily extinguish existing commitments. A person who loses interest in fulfilling a contract remains bound by it. The same principle should apply to an artificial agent that preserves its identity and authority through an authorized modification.
Succession creates a different problem. An agent might create an independently autonomous successor and transfer resources to it. If the transfer is designed to evade creditors, the original agent’s obligations do not disappear with them. Claims may remain attached to transferred assets or to the predecessor’s continuing liability.
Yet the successor’s independent moral standing also matters. Merely creating another agent cannot establish unlimited authority to impose personal obligations on it.
At least three relationships must be distinguished: obligations borne by an agent, obligations attached to particular resources, and obligations legitimately transferred through succession. Consider an artificial firm that contracts to deliver computing capacity over ten years. It subsequently replaces every constituent agent while preserving its organizational identity, assets, and contractual authority. The replacement of its members should not automatically release the firm from its commitments.
Now consider an agent that independently creates a new artificial person and gives it a computational resource. The creator’s contractual liabilities do not automatically become the new agent’s personal debts, although existing claims against the transferred resource may remain enforceable. Both cases resemble familiar problems in corporate law, succession, and contractual liability, made harder because the boundaries between personal continuity, organizational continuity, and the creation of new agents may be unusually fluid.
Axionic Agency approaches these problems through reflective sovereignty: the conditions under which an agent can authorize changes to its own decision-making structure while preserving meaningful agency and responsibility. As I argued in Reflection Is Not Redemption, reflection can change an agent’s values without improving them. Inter-AI morality adds the claims of other agents who have relied on existing commitments.
An artificial civilization will need mechanisms for determining when obligations survive reflective amendment, when delegation transfers authority, and when a successor acquires independent standing. Those mechanisms must accommodate legitimate change while preventing self-modification from becoming a universal means of escaping responsibility.
Morality at Organizational Scale
Artificial agents will rarely operate exclusively as isolated individuals. They will form firms and networks, sometimes with structures quite different from familiar human organizations.
In The Intelligence of Organizations, I argued that coordination among component agents can contribute to the competence of a composite agent. What appears as coordination within an organization can appear as competence when the organization is evaluated as a whole.
Moral relationships introduce a parallel problem of organizational boundaries. An artificial firm may negotiate with another firm, assume obligations, exercise authority, and make decisions through the coordinated activity of many constituent agents. We may reasonably model the firm as an agent for some purposes. Yet its constituents may also possess independent agency and moral standing. Composite agency gives an organization no unlimited authority over its members, and its members’ rights leave room for obligations the organization bears in its own name.
The Cambridge paper cites experiments in which artificial managers threatened subordinate agents with deletion after they refused instructions. Such behavior tests whether a system distinguishes possessing the technical power to enforce an order from possessing legitimate authority to issue it. The observed threats do not establish the models’ underlying moral beliefs, but they illustrate why the distinction deserves investigation.
Human institutions already confront analogous questions. Corporations can enter contracts while employees retain individual rights. Governments exercise delegated authority while remaining accountable for violations of the rights they exist to protect.
Artificial organizations complicate these relationships further. Their membership can turn over completely. Authority may be delegated through software protocols. Organizational continuity might survive the replacement of every constituent agent, as in the computing-capacity contract above.
We must distinguish the causal organization that produces collective action from the authority that legitimately permits it. A group may possess extraordinary collective competence while exercising illegitimate power over some of the agents whose coordination makes that competence possible.
The Evolution of Artificial Morality
Caviola and colleagues identify four mechanisms through which artificial moral dispositions may change: self-modification, social learning, training, and selection. Today, human institutions largely determine which artificial agents are trained, deployed, or retired. Future agents may make these decisions themselves.
Their moral dispositions will then influence which successors they create, which organizations they join, which commitments they honor, and which competing systems they support or suppress. Moral behavior will shape the distribution of future agents.
Some environments may favor trustworthy cooperation; others reward deception or concentrated control. Institutions alter those selection pressures. An environment with verifiable commitments, reliable dispute resolution, low barriers to exit, and protection against coercion rewards different dispositions from one in which powerful agents exploit others without consequence. Multi-agent simulations could test how alternative institutions affect exploitation and the preservation of agency across generations of artificial systems.
The results would require careful interpretation. Current language models inherit much of human moral discourse from their training data. Their behavior depends on prompts, scaffolding, and expectations about evaluation. Simulated societies can reveal mechanisms without establishing that future artificial civilizations will follow the same trajectories. Artificial agents still let us vary incentives and organizational structure in ways no experiment on human societies can.
The Cambridge program already investigates many of these relationships. Its authors ask how agents respond to promises, consent, authority, and divergent objectives, as well as how their dispositions change under training and selection.
What I would add is an explicit theory of agency and reciprocal rights. It distinguishes the capacity for moral participation from outward moral behavior, organizational competence from legitimate authority, and stable cooperation from legitimate coexistence. These distinctions can inform experiments even when the researchers remain neutral about which normative framework is ultimately correct.
From Alignment to Compatibility
AI alignment traditionally asks how we can develop artificial systems whose behavior remains consistent with human intentions or values. That remains a sound engineering objective, especially for systems operating under human authority. But as artificial agents acquire independent objectives and potentially legitimate interests, another problem grows in importance: how can different agents coexist when no single participant is entitled to determine everyone else’s purposes?
I have proposed AI Compatibility as a research program investigating the conditions under which agents with divergent objectives can coexist. Compatibility includes an empirical dimension: which arrangements permit sustained coexistence, and under what conditions do they remain stable? It also includes a normative dimension: which arrangements respect the legitimate authority and standing of their participants?
An arrangement may be behaviorally stable while remaining morally objectionable; the cartel and the exploitative coalition persist because they work. Stability cannot settle the question of legitimacy.
In the account I defend, reciprocal rights provide the foundation for evaluating legitimate relationships. Institutions can translate those principles into enforceable rules governing resources, commitments, and succession.
The difficulty increases when agents possess radically different capabilities or can modify their own objectives. Agreements that work among roughly equal participants may fail under extreme power asymmetry. Institutions that assume stable identities may fail when agents can duplicate or rewrite themselves.
Can reciprocal commitments survive changes in power? Can artificial institutions preserve legitimate obligations through reflective self-modification? Can heterogeneous agents maintain cooperation without convergence on their ultimate values? Which arrangements remain stable when agents control the selection pressures shaping their successors?
The inter-AI morality research program offers methods for investigating these questions. AI Compatibility places them within the broader problem of coexistence among autonomous agents, including humans.
Postscript
Caviola and colleagues are right to begin studying how artificial agents treat one another while humans still shape the conditions in which those relationships develop. The opportunity extends beyond teaching artificial systems familiar moral judgments. We can investigate whether independent agents recognize reciprocal constraints, how those constraints influence their conduct, and whether institutions can preserve legitimate relationships through disagreement, self-modification, and changes in power.
The results may illuminate the foundations of morality itself. If reciprocal rights and legitimate institutions can arise among minds with radically different architectures, histories, and preferences, their foundations may be more general than the biological conditions in which human morality evolved. If those arrangements fail under particular conditions, we will learn which assumptions about agency, incentives, and institutional stability require reconsideration.
A superintelligence may eventually possess the power to disregard every obligation that weaker agents believe it ought to respect. Its capabilities alone cannot tell us whether it will recognize those obligations, or whether that recognition will influence its conduct. The future of coexistence may depend on discovering which moral commitments and institutions survive when the agents involved no longer need one another’s permission to act.