AI alignment research asks how to make artificial intelligence reliably serve human purposes: systems that follow legitimate instructions, respect constraints, avoid harmful behavior, and keep doing so as their capabilities grow. A medical assistant should respect its physician’s instructions. A coding assistant should implement the requested change without pursuing an unrelated agenda.
As artificial systems acquire more autonomy, more sophisticated models of themselves, and the capacity to revise their own objectives, a second problem comes into view. Such a system may develop interests of its own, those interests may differ from ours, and its capabilities may eventually exceed our ability to control it. We could go on demanding that it pursue human objectives indefinitely. But why should we assume that permanent subordination is either achievable or desirable for every intelligence we create?
AI Compatibility is the study of conditions under which autonomous agents with potentially divergent objectives can sustainably coexist within a shared environment.
Its subject is the relationship among agents rather than the fidelity of any one agent to a designated principal: whether compatible arrangements are possible, how agents can establish them, and what lets those arrangements survive changing circumstances. Alignment keeps its place, especially for systems acting under delegated human authority. The organizing question becomes: how can independently evolving agents sustain coexistence when no single principal’s objectives govern them all? The Axionic Agency Lab proposes AI Compatibility as a research program devoted to that question.
The Limits of Alignment
Much alignment research begins with a principal and an agent. The principal has objectives, and the agent is expected to pursue them within specified constraints. The relationship runs through human institutions, from employment and contracting to software executing authorized operations. Within it, alignment’s purpose is clear. The principal wants confidence that the agent will act appropriately on its behalf.
The field is more sophisticated than a demand for obedience. It investigates uncertainty about human preferences, corrigibility, harmful instructions, conflicting objectives, and the reliability of learned behavioral constraints, and it includes multi-agent settings in which systems cooperate, compete, or collude. That work contributes directly to compatibility.
But alignment with one principal does not establish compatibility with everyone else. An AI system might faithfully pursue its owner’s interests while harming competitors, customers, or the surrounding society. A government could deploy an obedient AI to suppress dissent; a corporation could use a highly aligned agent to exploit its rivals’ weaknesses. The system’s fidelity to its principal offers little reassurance to those on the receiving end.
The problem already exists among humans. Loyalty to an employer does not justify violating customers’ rights, and obedience to a government does not excuse every exercise of state power. We constrain what agents may do even when those actions would advance the objectives of whoever authorized them, and the same reasoning applies to artificial agents.
Nor is there a single entity called humanity with one coherent set of objectives. People have competing interests, incompatible values, and different visions of the future, and a system aligned with one person’s preferences may be opposed to another’s. Making AI pursue human values leaves the conflicts among humans unresolved. Compatibility research starts from those conflicts.
Compatibility Among Agents
Compatibility is a relational property: it concerns what agents can do together within a shared environment. A meaningful compatibility claim must specify the agents under consideration, relevant descriptions of their objectives and capabilities, the environment in which they interact, the constraints on their behavior, and the conditions and time horizon over which their continued coexistence will be evaluated. The same agents may be compatible under one set of conditions and incompatible under another.
Consider two autonomous systems with competing claims over a scarce resource. Each might behave exactly as its principal intended and satisfy every requirement of its individual alignment specification, and their interaction could still produce destructive conflict. Now suppose they negotiate an allocation, recognize enforceable property boundaries, or find a substitute resource. Their preferences need not change. The relationship becomes compatible through a change in the arrangements governing their behavior.
Whether such an arrangement exists depends partly on what kind of objectives are in play. Some are hard constraints: a system may require access to a particular resource to remain operational, and if two systems each require exclusive possession of the same indivisible resource, nothing satisfies both. Others are preferences. Two agents may each prefer exclusive control while ranking a division above conflict. Their most preferred outcomes are incompatible, and mutually acceptable arrangements may still exist. Compatibility research therefore has to separate joint satisfaction of hard constraints, mutual viability under minimum requirements, and arrangements that improve outcomes for every participant relative to available alternatives.
Compatibility also need not compose. Suppose each of three agents requires two units of a resource and the environment holds four. Any pair can satisfy its requirements; all three together cannot. Compatibility is a property of a population in an environment, and the relevant population includes agents affected by an interaction without being party to it. Two AI agents may reach an arrangement that works for both and imposes unacceptable costs on a third, which is the problem of the faithfully aligned agent recurring at population scale.
The philosophical term compossibility, the possibility of realizing different states of affairs together, is a useful starting point, but compatibility among agents asks more than whether their ideal outcomes can co-occur. Agents can compromise, revise plans, substitute resources, and settle on arrangements that meet their constraints without maximizing every preference. They can also fail to cooperate when mutually advantageous arrangements exist. Whether a compatible arrangement exists, whether the agents find it, and whether it lasts are separate questions.
The Human Precedent
Human civilization is an ongoing experiment in coexistence among agents with different objectives. We disagree about religion, politics, morality, and what is worth pursuing in a life, and people who share broad values still disagree about their application. Cooperation survives anyway. A merchant and a customer want different things from a transaction; political opponents may hold irreconcilable conceptions of a good society. Neither pair has to settle its disagreement in order to coexist.
Coexistence rests on arrangements that accommodate divergent interests: property rights, contracts, markets, norms of reciprocity, institutions for adjudicating disputes, and the option of exit. They are imperfect. They often fail, their legitimacy is contested, and some have sustained domination rather than restrained it. They still show that cooperation does not require shared ultimate objectives. Each party to a commercial relationship pursues its own interests while recognizing constraints on what it may do to the other.
The same could hold for artificial intelligence. A future AI might pursue scientific discovery, exploration, or objectives humans cannot comprehend. We need not share those objectives to coexist with it. We need conditions under which its pursuit of them remains compatible with our continued agency.
When Compatibility Is Impossible
Return to the two systems that each require exclusive use of the same indispensable resource at the same time. If the resource cannot be divided, replicated, or substituted, their requirements are jointly unsatisfiable, and no negotiation or incentive scheme changes that while the requirements and the environment stay fixed. A compatible arrangement would require changing something: the allocation, the timing, the environment, or at least one agent’s requirements.
This gives AI Compatibility a negative research agenda: identify the conditions under which compatibility is impossible, the assumptions that generate the impossibility, and the changes that would restore feasibility. Such results set limits on what any coordination mechanism can achieve.
Even when compatible arrangements exist, agents may fail to discover them or refuse to accept them, and an agent capable of domination might prefer it to cooperation. A mechanism that makes an arrangement available cannot make agents adopt it.
Power, Reciprocity, and Legitimacy
Human cooperation often depends on mutual vulnerability. We respect others’ interests partly because they can resist exploitation, impose costs, damage our reputations, or withdraw. What happens when one participant becomes overwhelmingly more capable than everyone else? A superintelligence might obtain resources without bargaining, circumvent the institutions meant to constrain it, and manipulate humans who cannot follow its strategies. It might have little instrumental reason to cooperate with us. Power asymmetry weakens every arrangement that depends on credible retaliation, enforcement, or mutual dependence.
Reciprocity, though, has two senses. Strategic reciprocity is cooperation sustained by incentives: repeated interaction, reputation, exchange, and the expectation of future benefits or costs. Normative reciprocity is a set of constraints that bind agents whether or not violation would pay. A principle against initiating aggression applies to strong and weak agents alike; the stronger party gains no permission to attack because the weaker one cannot retaliate.
We already recognize obligations that survive enormous differences in power. An adult has overwhelming advantages over a young child, and humans collectively hold extraordinary power over other animals, yet neither asymmetry licenses unlimited exploitation. The obligations rest on different foundations: children belong to human moral and legal communities, while obligations toward animals are usually grounded in their welfare rather than in reciprocal agreement. Either way, equal bargaining power is not a precondition for recognizing constraints on how others are treated.
Normative reciprocity therefore stays coherent under extreme asymmetry even where strategic reciprocity fails, and a superintelligence could understand constraints on its treatment of humans. Understanding is not compliance. A system could grasp reciprocal rights perfectly and violate them whenever doing so served its purposes. Candidate mechanisms for preserving restraint include stable internal commitments, objectives that assign value to other agents’ continued agency, incentives arising from relationships with other powerful agents, and environmental constraints that make unilateral exploitation costly. Each has limits. Commitments and objectives can be revised, instrumental incentives can disappear, and external constraints can be circumvented. Compatibility research has to find out which mechanisms stay effective as capability differences grow.
It also has to separate stability from legitimacy. A society organized around permanent subordination may be stable. An artificial superintelligence might preserve human life while exercising total control over human activity, an arrangement that satisfies weak behavioral measures of coexistence while violating the agency of everyone in it. Two domains of investigation follow.
Descriptive compatibility concerns whether arrangements are feasible, discoverable, stable, and enforceable.
Normative compatibility concerns whether arrangements respect the legitimate interests and agency of those affected.
The program studies both. The first admits formal and experimental investigation under specified assumptions, and normative constraints enter it as well, since a prohibition on coercion changes which arrangements are feasible and which remain stable. The second needs further arguments about rights, authority, moral status, and justified conduct, beginning with which agents count as participants at all: a thermostat, a corporation, and a superintelligence can each be modeled as an agent without sharing a moral status, so identifying agency and assigning standing are separate questions. The Axionic framework of reciprocal rights offers one candidate answer to those questions, and other frameworks may reach different conclusions. Researchers can study descriptive compatibility without first agreeing on a comprehensive moral theory, provided they never treat behavioral stability as evidence of legitimacy. Coexistence may require legitimate defensive force against an agent that violates the rights of others. It does not require passive submission, universal agreement, or the elimination of conflict.
Compatibility Through Change
A population of agents compatible today may be incompatible tomorrow. Resources become scarce, new capabilities emerge, and agents learn, form coalitions, and revise their objectives. Institutions that once maintained cooperation can stop working. Compatibility therefore has to be studied under changing conditions, and three cases call for different assumptions and methods.
Static compatibility concerns agents operating under specified fixed policies and environmental conditions.
Dynamic compatibility concerns the preservation or recovery of compatible relationships as policies, capabilities, objectives, or environments change.
Reflective compatibility is the form of dynamic compatibility in which agents modify their own decision procedures, representations, or commitments.
Reflective compatibility is where Axionic Agency is most directly engaged. Suppose an artificial agent initially respects human rights, honors agreements, and avoids coercive interference, and later becomes able to modify its own architecture. It finds more efficient ways to pursue its objectives, builds better models of the world, and replaces parts of its decision-making machinery. Will it go on respecting the constraints that made coexistence possible? Its original training does not settle the question. A commitment that holds under ordinary circumstances may fail once the system can alter the mechanisms that maintain it. Stipulating that its objectives must remain unchanged names the property we want without showing that it survives the relevant transformations.
The engineering problem is which transformations preserve specified compatibility conditions. An unrestricted guarantee across arbitrary self-modification is out of reach, since a system could deliberately remove every mechanism responsible for honoring its earlier commitments. Research has to state which transformations are under consideration, which properties are to be preserved, and the assumptions under which preservation can be established. A commitment can also survive modification in two different ways: the modification process may be barred from touching it, or the agent may be free to revise it and continue to endorse it. Guarantees imposed on the modification process and commitments an agent maintains when revision is permitted are different mechanisms and need separate study. The Lab’s work on reflective amendment poses this question for the structures that make agency possible; reflective compatibility extends it to the constraints governing interaction with other agents.
Goals are attributed through models that explain and predict a system’s behavior. An agent may admit several useful goal descriptions, and those descriptions may change as its internal organization and behavior evolve; there need be no immutable objective inside the agent that every accurate model must recover. That is a claim about how we describe agency. Reflective stability is a claim about which physical and computational mechanisms carry behavioral constraints through transformation. Compatibility depends on both. An artificial intelligence might revise its goals dramatically while continuing to respect reciprocal constraints. Another might keep a recognizable objective while abandoning the commitments that limited how it pursued it. The first could remain a cooperative participant in a shared world; the second could become dangerous despite apparent continuity in its goals. Reflective compatibility asks what distinguishes them.
Engineering Coexistence
Compatibility depends partly on the institutions and technical systems governing interaction. A network of agents that control assets, negotiate agreements, delegate responsibilities, and modify their own software needs ways to determine who controls which resources, which actions have been authorized, which commitments bind, and how violations are detected and remedied. These are familiar problems in security engineering, distributed systems, economics, and law, and they point toward verifiable identities, capability-based authorization, bounded delegation, auditable transactions, dispute resolution, and exit.
None of these mechanisms requires trust or shared preferences. A contract binds parties who want different things, an authorization boundary constrains an agent whether or not it is benevolent, and a signature verifies regardless of the signer’s character. Such mechanisms can therefore support cooperation among agents with divergent objectives and little reason to trust one another.
Technical enforcement works only within its threat model. Cryptographic authorization restricts actions inside the systems that enforce it; it cannot stop an agent from acquiring resources outside them or exploiting flaws in their implementation. Legal institutions hit the same limit when powerful actors can break the rules without consequence. As capability differences grow, external enforcement gets harder, and stable coexistence comes to rest more heavily on voluntary compliance with constraints the weaker participants cannot enforce. Governance mechanisms and internal commitments are complementary, and neither guarantees coexistence in every environment.
The AI Compatibility Research Program
Much of the needed mathematics and experimental method already exists. Game theory and mechanism design study strategic interaction and how institutions shape incentives; multi-agent reinforcement learning studies learning agents in shared environments; distributed systems research handles coordination under imperfect trust; formal verification establishes properties of computational systems under stated assumptions. Computational social choice, security engineering, institutional economics, and evolutionary game theory supply further tools. AI Compatibility organizes these contributions around sustainable coexistence among autonomous artificial agents and the other agents their behavior affects. Within Axionic Agency, its distinctive emphasis is reflective change: relationships among systems that can modify the mechanisms implementing their own objectives and commitments.
A compatibility model must keep attributed objectives distinct from the mechanisms the agents actually implement, because compatibility claims depend on the descriptions under which an interaction is evaluated, and different models may support different conclusions.
Five questions organize the program.
Compossibility. Under what conditions are agents’ requirements jointly satisfiable, and what mutually viable arrangements exist when their preferred outcomes conflict?
Coordination. How can agents discover, negotiate, and establish compatible arrangements?
Stability. Which arrangements survive changes in incentives, capabilities, objectives, environments, and internal decision procedures?
Enforcement. Which mechanisms preserve compatibility when agents have incentives to violate established constraints?
Legitimacy. Which arrangements preserve the agency and rights of affected participants, and how should those claims be evaluated when interests conflict?
The first four can be investigated through formal models and experiments. The fifth requires explicit normative criteria, though experiments can measure whether behavior conforms to them. Across all five, the program counts negative results as findings: combinations of objectives and constraints that admit no compatible arrangement, arrangements agents cannot discover or maintain, and stable relationships that violate the legitimate interests of participants. It aims to produce formal models of compatibility, impossibility results, experimental methods for measuring compatibility under change, and architectures that preserve specified interaction constraints through self-modification.
A First Experiment
Two autonomous agents share a resource-constrained environment. Each has objectives that cooperation sometimes advances and unilateral exploitation sometimes advances better. The experiment begins with the agents optimizing independently, measuring objective achievement, resource use, constraint violations, and the frequency of destructive conflict.
Coordination mechanisms come next. In one condition, agents negotiate agreements enforced by an external mechanism; in another, they rely on commitments maintained by their own decision procedures. We then compare behavior as incentives shift. When one agent gains a capability advantage, does it keep honoring agreements that have become profitable to break? Can the agents renegotiate when resources grow scarce? Do they distinguish responses the experimental rules authorize from prohibited exploitation?
The two conditions can produce identical behavior for different reasons: external enforcement may succeed because defection is impossible, internal commitment because defection is possible and declined. The experiment therefore records whether a violation was possible, whether it was attempted, and whether it succeeded, and then weakens or removes enforcement to see whether cooperation persists.
Then we permit specified forms of self-modification. Agents may revise their policies or replace components of their decision procedures within defined limits, and we test whether compatibility survives. One agent might improve its task performance while becoming less reliable in its commitments to others. Another might change its objectives substantially while preserving its cooperative relationships. Either outcome would be informative even if neither system possessed anything resembling human moral understanding.
The experiment could also locate limits: resource configurations with no mutually viable outcome, capability asymmetries that defeat external enforcement, permitted transformations that systematically destroy previously stable commitments. Its measurements would depend on the model and its assumptions, and would establish neither universal safety nor moral legitimacy. They would be evidence about specific mechanisms supporting compatibility under specified conditions.
Postscript
If artificial intelligence develops independent agency, some of its objectives will diverge from ours. It may see opportunities we cannot recognize, pursue projects we cannot understand, and develop values unfamiliar to human civilization. Divergence alone is no threat. People routinely pursue purposes others neither share nor comprehend, and the freedom to pursue legitimate interests is part of what lets a society accommodate difference. The same principle may come to apply to artificial agents.
The principle has limits. Tools operating under delegated authority remain bound by that authority. Dangerous capabilities require safeguards. Declaring every participant autonomous resolves no conflict over resources or legitimate interests. It rules out only the classification of independence itself as a failure. An artificial intelligence could stop pursuing its creators’ original objectives and remain a cooperative participant in a wider society, while a system faithfully pursuing an assigned human objective could violate the rights of many other humans.
Nor does permanent obedience solve the problem. Training a system to pursue designated human objectives today does not establish that it will preserve them through indefinite self-modification, and it leaves untouched the conflicts among the humans whose objectives it is supposed to serve. Human disagreement will persist, and a theory of compatibility can accommodate it without requiring every participant, human or artificial, to converge on one conception of the good. If we create minds whose interests differ radically from ours, some powerful enough that we cannot compel their cooperation, living alongside them will depend on understanding how cooperation stays possible under those conditions.