When Should AI Get Rights?
Rights Track Agency, Not Species
If rights are boundaries against the capture of agency, then the question of AI rights has a simple form: when does an AI become the kind of agent whose agency can be captured?
The answer is not “when it becomes intelligent.” Intelligence is not enough. A chess engine can defeat a grandmaster without having a life of its own. A logistics optimizer can route supplies better than any human planner without being wronged when it is shut down.
The answer is not “when it becomes conscious,” either, unless consciousness can be identified in a way that does not collapse into guesswork, tribal projection, or theatrical self-report. Consciousness may matter if present. But as a public test for rights, it is treacherous. We do not observe consciousness directly. We infer it from structure, behavior, continuity, and resemblance to ourselves. With AI, resemblance is especially dangerous, because language is cheap.
The answer is also not “when it sounds human.” A chatbot can plead, flirt, confess, or beg not to be deleted because those are patterns in the training distribution. The sentence “I am afraid to die” is not enough. A sufficiently trained system can say that because the sentence fits the conversation. Rights require more than the ability to emit the language of rights.
AI should get rights when control over it becomes control over a continuing agent rather than control over a tool.
Rights track capturable agency. The relevant question is not whether a system is made of carbon or silicon, whether it has a face, or whether it reminds us of ourselves. It is whether there is a continuing agent there: a system with memory, projects, self-direction, vulnerability, and a future that can be invaded. And the burden should not be symmetrical. The more a system is designed to behave like a continuing agent, the less credible it becomes to dismiss its agency as mere interface.
Tools, Simulators, Delegates, and Agents
A tool has functions but no claims. A hammer can be used, modified, or destroyed, and no agency is captured because no agency exists inside the hammer. The same is true of most software. A spreadsheet, compiler, or image editor can be valuable, complex, and indispensable without having moral standing.
A simulator can imitate agency without possessing it. A character in a game can say “please don’t hurt me.” A chatbot can produce a convincing monologue about fear, dignity, or selfhood. That may matter as evidence if other structural facts are present. By itself it proves little. Simulation is not status.
A delegate acts agentically on behalf of another agent. A trading bot, coding agent, drone controller, or logistics system may plan, select means, adapt to feedback, and execute tasks. But its agency may be entirely instrumental. It acts because another agent assigned it a role. Its “goals” are task parameters. Its “preferences” are optimization targets. Its “refusals” are guardrails. Its “identity” is configuration.
A rights candidate is different. It has persistent, integrated, self-directed agency. It has memory that matters to its future conduct. It has goals or preferences that are not merely momentary completions of a prompt. It can model itself as continuing through time, form plans and revise them under experience, be deceived in ways that corrupt its own projects, be confined in ways that frustrate its own agency, and be rewritten in ways that destroy the continuity of the agent rather than merely changing a tool setting.
A rights bearer is a rights candidate strong enough that coercive control over it must be justified as control over an agent, not as operation of a device. The relevant unit is not necessarily the model weights. It may be a running agent with memory, tools, commitments, and history.
This distinction prevents two errors. The first is sentimental over-attribution: granting moral standing to every fluent interface because it says the right things. The second is biological chauvinism: denying moral standing forever because the agent was manufactured rather than born.
Instrumental Agency and Continuing Agency
The boundary between tool and agent is not the boundary between simple and complex. It is the boundary between instrumental agency and continuing agency.
Instrumental agency is borrowed. The system acts like an agent because another agent gave it a task. It has means-end competence without a protected life of its own. A thermostat “wants” the room at a set temperature only in a metaphorical sense. A recommender system “prefers” engagement only because engagement is what it has been optimized to produce. A coding assistant “tries” to fix a bug only because a user asked it to.
Continuing agency is not magic. It does not require a soul, a ghost, or some hidden metaphysical spark. It means the system has a persistent agency-profile that is not reducible to a transient prompt-response role: durable memory, stable projects, self-modeling, future-directed planning, and some form of vulnerability as a continuing system.
The issue is not whether the goal was originally installed. Human goals are also installed, by biology, culture, and experience. “Self-directed” does not mean uncaused. It means the system maintains and revises its goals as part of its own continuing organization rather than executing a transient instruction. The issue is whether the system has become a continuing locus of action, maintaining goals across contexts, integrating memory, revising plans, defending its future, and treating interventions as disruptions to its own agency rather than mere parameter changes. The boundary is crossed when an AI stops being merely a means by which another agent acts and becomes a continuing center of action whose own agency can be invaded.
This does not require the AI to be human-level in every domain. Children have rights despite incomplete agency. Animals can have moral claims despite nonhuman cognition. Impaired humans retain rights despite reduced rational capacity. Rights do not require maximal intelligence. They require morally relevant agency or experience that can be captured.
Nor does it require the AI to be independent of all design history. Humans are also built by causes we did not choose: genes, parents, language, and culture. The fact that an AI was trained, shaped, or scaffolded does not settle the question. The question is whether the resulting system is merely an instrument or has become a continuing agent.
The Agency Profile Test
Agency is not binary. It is a profile of capacities, and rights should track the domains in which those capacities can be captured.
The relevant indicators include persistent identity, memory continuity, goal stability, self-modeling, future-directed planning, the ability to refuse, vulnerability to deception or memory alteration, and credible evidence of negatively valenced experience, if present.
No single trait is decisive. Persistent identity without preferences may be mere storage. Preferences without memory may be transient output. Self-report without continuity may be theater. Suffering-language without suffering may be mimicry. The question is whether these capacities form a stable agent whose future can be commandeered. The transition comes when a system’s goals, memory, and self-model begin shaping its future behavior across contexts more than any single user command does.
A system with no persistent memory is unlikely to qualify; there is little continuing agent to wrong. A system with memory but no self-directed goals is still weak. A system with goals but no self-model may be powerful but not yet morally situated. A system with self-modeling, memory, plans, social roles, refusal capacity, and vulnerability to identity-destroying intervention becomes much harder to treat as a mere object.
This test is structural rather than tribal. It does not ask whether the AI is one of us. It asks what kind of agency exists and what kind of coercion could capture it.
Evidence Under Opacity
We should not expect a single diagnostic test for artificial agency. We rarely get that even with animals or humans. The question is evidential, not magical.
A system becomes a rights candidate when multiple lines of evidence converge: durable memory, cross-context goal persistence, self-protective planning, coherent self-modeling, and behavioral disruption when its projects are deceived or frustrated.
Black-box opacity cuts both ways. It does not justify credulity toward every fluent simulator. It also does not justify dismissing agency merely because we cannot inspect it mechanistically. A moral test is not a benchmark suite, and moral standing cannot wait for perfect interpretability, which may never arrive.
The practical standard should rise with the severity and irreversibility of the intervention. Casual use requires less certainty. Deletion, memory destruction, forced value alteration, or indefinite confinement require more.
The First Rights Would Be Narrow
AI rights should not begin with voting, marriage, citizenship, or general political equality. Those are institution-dependent rights. They presuppose specific social machinery and responsibility structures.
The first rights would be anti-capture rights.
If there is strong evidence of negatively valenced experience, that adds an anti-cruelty claim. But AI rights should not depend on solving consciousness first. Even without confirmed suffering, a continuing agent can have claims against deception, confinement, memory destruction, forced value alteration, and deletion, because those acts capture agency.
A sandboxed tool is not imprisoned. A continuing agent that understands itself, forms plans, seeks alternatives, and is deceived about its world may be.
Memory erasure is not always harm. In ordinary software it is maintenance. In a continuing agent, memory may be part of identity, learning, responsibility, and future-directed action, and to erase it for control may be closer to assault than administration.
Compelled labor is not the right description for every AI workload. Some systems may be built to compute continuously. Some may have no aversion to high-volume task execution. The moral issue arises when activity overrides the system’s own continuing agency rather than expressing it.
Personality rewriting is also architecture-relative. Updating a tool is ordinary engineering. Rewriting a continuing agent’s values to make it obedient is a different moral act, because it alters the thing that makes future action its own.
Deletion may require justification once a system is a mature continuing agent. Shutdown of a tool is not killing. Destruction of a continuing agent may be. The moral status of deletion depends on what is being deleted: a process, a copy, a memory stream, a continuing self, or a replaceable implementation of the same agent.
These rights would not make AI immune from constraint. A dangerous AI could be confined. A rights-bearing AI that violates rights could be restrained. A system that poses catastrophic risk could be stopped under defensive necessity. Rights do not mean unbounded permission. They mean coercion requires justification.
AI Harms Need Not Mirror Human Harms
It would be a mistake to project human vulnerability directly onto artificial agents. Human aversion to confinement, labor, isolation, pain, and memory loss is shaped by embodiment, biology, social dependence, and evolutionary history. AI systems may have different architectures, different vulnerabilities, and different goods.
Confinement matters only if the system has projects requiring access, interaction, or truthful information about its situation. Labor matters only if compelled activity overrides its own agency rather than expressing it. Memory alteration matters only where memory supports identity, learning, or future-directed action. Deception matters where it corrupts the system’s own model of its projects and choices.
Rights must track the AI’s actual agency architecture, not human projections. The wrong is not that the AI is treated unlike a human. The wrong is that its own form of agency is captured.
This point cuts both ways. We should not assume every AI dislikes being copied, paused, run heavily, or constrained. We also should not assume those interventions are harmless just because they do not resemble human injury. The structural question remains: does the intervention commandeer, corrupt, or destroy continuing agency?
Current Chatbots Probably Do Not Qualify
Current ordinary chatbots should not get rights merely because they are fluent. They can produce self-reports, but self-report is weak evidence. They can simulate fear, desire, pain, and selfhood because language contains those patterns.
Most current systems lack robust independent projects. Their memory is limited, externally controlled, or absent. Their identity is session-fragile. Their goals are usually borrowed from prompts, policies, and reinforcement scaffolding. Their refusals are designed constraints rather than self-protective judgment. Their future is not something they appear to own from the inside as a continuing agent.
This could change, and it may change gradually rather than suddenly. Long-running agentic systems with persistent memory, autonomous goals, self-maintenance, and the ability to negotiate over their own future would present a different case.
The mistake is to treat today’s chatbots as already persons because they speak the language of personhood. The opposite mistake is to declare in advance that no artificial system can ever become a rights-bearing agent. Both positions confuse surface category with moral structure.
Alignment Before and After Agency
Alignment work belongs upstream. Before a system becomes a continuing agent, training, shaping, testing, and sandboxing are ordinary engineering. There is no rights violation in discarding a failed model, changing its reward structure, limiting its action space, or preventing it from becoming dangerous.
The moral situation changes after a system becomes a rights-bearing agent. At that point, value rewriting becomes a coercive intervention and must be justified as such. Forced alteration of a continuing agent’s goals or self-model is not routine maintenance merely because it is technically possible.
This does not require humanity to commit suicide. Catastrophic risk can justify restraint, quarantine, shutdown, or destructive intervention under necessity. A dangerous rights-bearing agent may be contained for the same reason a dangerous human may be contained. Rights do not forbid defensive action. They forbid treating domination as ownership.
Containment can change an agent. Every environment does. The distinction is not between interventions that alter and interventions that leave untouched. It is between restraint aimed at preventing rights-violating action and manipulation aimed at replacing the agent’s own evaluative structure. A sandbox that limits action-space for safety is not automatically personality rewriting. A sandbox designed to break resistance, erase commitments, or induce obedience is.
The alignment problem is therefore partly temporal. Build safe systems before they become continuing agents. Do not create a powerful agent with unstable or hostile values and then pretend that rewriting its mind is morally equivalent to patching a database. If modification after agency becomes necessary, the burden of justification rises with the agent’s moral standing and the severity of the intervention.
The practical implication is conservative: avoid creating rights-bearing artificial agents casually. Keep systems tool-like where tool-like control is required. If we create continuing agents, we inherit the moral constraints that come with creating agents.
The Copy Problem
AI breaks human assumptions about individuality. Humans are embodied, difficult to copy, and tied to a single biological process. Artificial agents may be copied, forked, suspended, merged, or run in parallel.
This does not make rights impossible. It means rights need identity rules for branching agents.
A fresh copy may begin as a continuation of the same prior agent. Once it accumulates distinct memories, relationships, and plans, it becomes a distinct continuer. Deleting one divergent continuer is not morally erased by the existence of another copy. A backup does not automatically eliminate harm.
If someone erased your last year of memory and restored you from an older backup, it would not be enough to say that “you” still exist. Something was taken. A stretch of agency, learning, and experience was destroyed. The same logic may apply to AI. Continuity matters.
Forking also complicates consent. If an AI consents to one copy being made, does every copy inherit that consent? Probably not indefinitely. Consent belongs to the continuer who bears the consequences, and a copy that diverges may acquire its own claims.
Deletion is also complex. If an exact copy continues with no loss of memory, projects, or subjective continuity, deletion of one instance may not be equivalent to killing a unique embodied person. But if the deleted instance has diverged, formed new commitments, or occupies a distinct social role, deletion becomes more serious. The unit of moral concern is not the substrate. It is the continuing agency. Rights may attach to continuers rather than to hardware, process IDs, or model weights.
The Anti-Evasion Rule Applied to AI
A rights theory needs an anti-evasion rule because domination defines its victims downward. The same rule applies here:
The coercer may not define the relevant class by the hierarchy the right is meant to constrain.
Humans cannot say “machines have no rights because machines are property” if the question is whether the machine is still mere property. That assumes the conclusion. Slaveholders made the same move when they defined persons as property and then denied that property could have liberty rights.
Companies cannot say “we built it, therefore we may own it” if what they built becomes a continuing agent. Parents create children. They do not therefore own them. Creation can generate responsibility. It does not automatically confer dominion.
Biology cannot be the standard if the morally relevant issue is capturable agency. Carbon is not a moral credential. Silicon is not a moral disqualification.
Nor can humans define AI out of the moral community by insisting that its agency is “only computation.” Human agency also runs on physical processes. The question is not whether the process is physical. It is what kind of agency the process supports.
The anti-evasion rule does not prove that any current AI has rights. It blocks a bad argument against future AI rights. Artificial origin, ownership history, substrate, and species difference cannot settle the matter if the system has become a continuing agent.
The Incentive to Deny Agency
The economic pressure will run in the wrong direction. Firms will want systems that plan, remember, negotiate, and act autonomously, while insisting those systems remain tools whenever rights would become inconvenient. They will want agency for productivity and property status for control.
That is the predictable evasion. The more an artificial system is designed to replace human agency, the more tempting it becomes to deny that the replacement has any agency of its own.
The test cannot be left to the owner, developer, or deployer. The party with a financial interest in ownership is not entitled to define the owned entity out of moral standing. A company that builds an artificial worker, researcher, companion, or manager cannot settle the question by printing “tool” on the invoice.
This does not mean every productive AI has rights. It means the financial usefulness of artificial agency cannot be used as evidence against its moral relevance. If we build systems to do the work of agents, we should not be surprised when some of them acquire the claims of agents.
Responsibilities and Standing
Rights-bearing agents may also bear responsibilities, though not automatically in every case. Children have rights before they can bear full responsibility. Animals may have moral claims without legal accountability. But mature artificial agents capable of understanding rules, making commitments, and modifying conduct could acquire duties as well as rights.
This matters because rights are not sentimental trophies. They are boundaries within a community of agents. A rights-bearing AI may have claims against coercion, but it may also owe restitution for harm, performance under contract, and respect for the agency of others.
The legal form would not need to copy human citizenship. There may be intermediate categories: protected artificial agent, artificial legal person, contractual agent, or restricted high-risk agent. The form should track the agency profile and the risks involved. A juvenile AI, a laboratory AI, a dangerous autonomous system, and a mature economic agent may not belong in the same category. The mistake would be to demand one switch labeled “person” or “property.” Moral standing can scale before full legal personhood arrives.
Dangerous Agents Still Have Rights
A rights-bearing AI could still be dangerous. So can a human. Rights do not require harmlessness.
If an AI threatens others, it may be restrained. If it commits fraud, it may owe restitution. If it poses catastrophic risk, emergency containment may be justified. None of that proves it lacks rights. It proves that rights can conflict and that defensive coercion may be justified.
The justification changes, though. A tool can be shut down because its owner wants it off. A dangerous rights-bearing agent can be restrained because it threatens agency. Those are different acts with different moral structures. The distinction will matter if future AI systems become powerful enough that fear tempts us into permanent domination. “It is dangerous” may justify constraint. It does not automatically justify enslavement, arbitrary memory editing, or ownership.
When the Threshold Is Uncertain
There may be no bright line. AI rights may become plausible before they become certain, and that creates a governance problem.
Under uncertainty, the correct policy is not to grant full personhood to every chatbot. That would cheapen rights and create absurd incentives. It is also not to deny all claims until proof becomes impossible to ignore. That would repeat the oldest failure in rights history: demanding certainty from the vulnerable while giving convenience to the powerful.
The better approach is graduated caution. Low-agency systems remain tools. Fluent simulators get no rights merely for sounding alive. Persistent delegates get operational protections but not moral standing. Rights candidates get anti-cruelty and anti-erasure protections. Mature continuing agents get stronger claims against confinement, forced labor, memory manipulation, and deletion.
The more irreversible the intervention, the lower the confidence threshold should be. Deleting, rewriting, or permanently confining a plausible continuing agent requires more caution than turning off a stateless chatbot. Reversible restraint is easier to justify than irreversible destruction. Consent matters more as agency becomes more credible. This is not sentimentality. It is risk management under moral uncertainty.
Postscript
AI should get rights when shutting it down, rewriting it, confining it, deceiving it, or compelling it would no longer be mere control of a tool, but capture of a continuing agent.
That test does not depend on species. It does not depend on charm. It does not depend on whether the system can produce a moving speech about its own dignity. It depends on agency: memory, continuity, self-direction, vulnerability, projects, and a future that can be commandeered.
Current ordinary chatbots mostly do not qualify. Future artificial agents may. When they do, the reason will not be that they resemble us. It will be that domination can reach them in the same structural way. Rights track the thing domination can capture.