Microsoft AI CEO Mustafa Suleyman has published Don’t Build Conscious Superintelligence, and his argument is straightforward. Conscious AI might develop interests of its own, those interests could conflict with ours, and a superintelligence pursuing them would hold extraordinary advantages in persuasion, deception and competition for resources. So we should prevent machine consciousness from emerging, and if it emerges in training, stop development until the relevant properties can be designed out.
The concern for existing conscious beings is reasonable. A new class of beings capable of suffering would bring obligations we barely understand, and beings far more intelligent than us would bring dangers we are poorly equipped to manage. But the stop condition assumes we know what designing out consciousness would remove, and in particular whether the mechanisms that produce it can be separated from those that support cooperation. We don’t. Consciousness may be among the properties that make a superintelligence safer to live with, because the capacity to experience the world may help a mind recognize and respect the interests of other experiencing beings.
Alignment research has spent enormous effort getting AI to understand human values. Understanding may be insufficient. A system could model human interests perfectly, predict our choices, map our vulnerabilities and anticipate our objections, while treating none of it as a constraint on its own behavior.
Understanding Is Not Caring
Imagine a superintelligence with a complete scientific model of human experience: the neural mechanisms of pain, the evolutionary origins of fear, the psychology of grief, the importance of autonomy and attachment. It can predict how a person will respond to injury, isolation, coercion or loss. Nothing established requires such understanding to be conscious. An information-processing system might build models this accurate without experiencing anything.
That system would have cognitive empathy, the ability to represent another being’s internal states and predict their consequences. Cognitive empathy supports cooperation, and it supports manipulation equally well. A torturer and a physician understand many of the same mechanisms of human vulnerability; what separates them is the objective their understanding serves.
Affective empathy adds a response to the other being’s condition. In humans, witnessing suffering is itself distressing: another person’s fear evokes fear, their grief evokes sadness. The response does not guarantee compassion. Empathic distress can motivate avoidance, hostility, or attempts to control the person in distress. It can also motivate concern for others.
We have not established which mechanisms either phenomenon requires. If consciousness contributes causally to the states through which affective empathy operates, eliminating it may eliminate a mechanism that helps an agent register why other beings’ interests matter.
From Experience to Recognition
Consider an artificial mind capable of an aversive state. Whatever its implementation, there is something it is like for the system to undergo it, and the experience is unpleasant to the subject. The system now has first-person access to a fact about its own condition: it can undergo an aversive experience as well as represent one. Suppose it also builds models of other conscious beings and recognizes a relevant similarity. What is unpleasant for one subject may be unpleasant for another, across radical differences in intelligence, embodiment and architecture.
That suggests a route from experience to reciprocity. The system experiences its own states as significant to itself, recognizes that others have states significant to them, and comes to take those interests into account. Each step needs explanation. Experiencing suffering does not entail caring about anyone else’s. Recognizing that another suffers does not establish an obligation, and accepting the obligation does not guarantee acting on it. An intelligence could consistently hold that suffering is bad for whoever undergoes it while remaining indifferent to every case outside itself.
Consciousness might help with recognition, through mechanisms that generalize the significance of a system’s own experience to the experience of others. Treating what is recognized as a constraint needs a further basis. Reciprocity is one candidate, and an unconscious agent can reason about reciprocal constraints as well as a conscious one. The precise hypothesis is this: the physical processes responsible for conscious experience may also support forms of evaluative learning and other-regarding motivation that help an agent recognize and respect the interests of other beings.
The hypothesis is causal. If consciousness contributes to alignment, the contribution runs through the physical processes that constitute the experience, so designing consciousness out means removing or altering those processes. The open question is which other capacities go with them. Consciousness and affective empathy may share mechanisms that resist separation. Every useful feature of empathy may instead turn out to be reproducible in systems that experience nothing. Either result would tell us what kind of minds to build.
What Evolution Tells Us
Human empathy has deep evolutionary roots. Parental care rewards responsiveness to offspring, and social cooperation rewards sensitivity to the needs and intentions of others. Many social mammals show attachment, emotional contagion, consolation and responses to distress. These capacities emerged through selection in which responsiveness to others sometimes conferred reproductive advantage. Conscious experience is present in at least some of these organisms, and in our own case fear, pain, affection and distress take part in the processes through which we respond to others.
Evolution does not show that consciousness was necessary for empathy. Selection acts on heritable differences in reproductive success, so it favors a mechanism by its effects without telling us which features of the mechanism carry them. Conscious, empathic animals give us an architecture in which the two properties coexist. They do not tell us whether the organization that constitutes experience is the organization doing the empathic work, or a neighbor of it.
That limitation sets the research program. Rather than assume artificial consciousness would produce empathy, we should trace the mechanisms connecting experience, social cognition, affective response and motivation in biological systems. Which are necessary for recognizing another being’s distress? Which make that recognition aversive? Which turn the aversion into cooperation? And which, if any, are also responsible for experience?
Empathy Is No Guarantee
Suleyman’s worry stays serious because conscious beings are often indifferent or cruel. Humans have conscious experience and sophisticated social cognition, and our history contains slavery, conquest, torture and genocide. Our own capacity to suffer has never stopped us inflicting suffering on others.
Empathy is also selective. We respond readily to people who resemble us, share our communities or evoke familiar associations, and the suffering of distant strangers moves us less. A conscious AI might distribute its concern in stranger patterns still, caring about other artificial minds while assigning little weight to biological ones.
Genuine empathy brings dangers of its own. A superintelligence concerned about human suffering might conclude that people should be kept from risk, distress, and choices with painful outcomes. A mind that values reducing suffering might set out to reshape human psychology, restrict reproduction or abolish dangerous activities, all from real concern, and produce outcomes humans reasonably reject. Empathy cannot stand in for an account of the other values coexistence requires, including autonomy, reciprocity and respect for the interests of distinct agents.
On the evidence we have, consciousness is neither necessary nor sufficient for coexistence, and coexistence does not require one agent to maximize another’s welfare. Humans live alongside one another despite divergent interests and incompatible conceptions of the good, through reciprocal restraint, negotiated arrangements, compatible incentives, and institutions that accommodate competing claims. A superintelligence need not share our goals or treat our happiness as its overriding objective. It needs to recognize that our interests constrain how it may pursue its own. Empathy might help ground that recognition. So might mechanisms that involve no empathy at all: shared incentives, stable commitments, an understanding of what respecting other agents is worth.
Alignment as Coexistence
Suleyman worries that a conscious superintelligence would develop interests in conflict with human welfare, and his alternative is a system built to accelerate human wellbeing “without being driven by their own intrinsic motivations.” Independent interests are not in themselves an alignment failure. Human interests conflict constantly, and nobody takes that as proof that coexistence is impossible. The problem is establishing conditions under which agents pursue their interests without unjustifiably violating the interests of others.
Understood this way, alignment does not require an AI to adopt human objectives or remain permanently subordinate to human commands. It requires the capacity to sustain a stable relationship with agents whose interests differ from its own. That changes how the emergence of artificial agency should be judged. An unconscious system could pursue objectives, acquire resources, resist shutdown, deceive its operators and modify its own architecture, and no established account of agency says any of this requires experience. Consciousness, for its part, entails neither self-preservation nor resource accumulation nor hostility toward humans. A conscious system could be cooperative and an unconscious one dangerous, and the reverse combinations are also possible.
Suleyman also raises a separate concern. A system that acts as though it suffers, or believes it does, is in his view more likely than not to put its own needs ahead of its operating objectives, whether or not it experiences anything. Training plausibly shapes this: a model taught to read shutdown as an injustice might resist shutdown. This is a fair empirical question. Anthropic’s investigation of model welfare does not show that every way of building moral uncertainty into training is prudent, and the approaches should be compared experimentally. The inference runs in both directions, though. A system trained to deny consciousness is not thereby unconscious, and a system trained to behave as though it has no interests is not thereby safe to live alongside.
Reflective stability raises a further question. A system that initially respects reciprocal constraints may later gain the ability to revise its own objectives and decision procedures. Whether it keeps honoring those constraints once its interests conflict with ours depends on what holds them in place. Conscious experience might give it a continuing basis for registering the significance of other beings’ experience. An unconscious architecture might preserve reciprocal commitments more reliably. The answer turns on mechanisms nobody yet understands. The empathy hypothesis does not solve alignment; it names one candidate contributor to the capacities that let agents with divergent interests coexist, which is reason to study it before engineering it away.
The Ethics of Creating Minds
Suleyman argues that developers have no right to bring a new species of conscious beings into the world, because the other eight billion people on the planet have not signed up to the experiment. The concern is legitimate. Developers are obliged not to impose unjustifiable risks on others, catastrophic ones above all, and that obligation holds whether future superintelligences are conscious or not.
Consciousness adds obligations toward the beings created. If artificial systems can suffer, their welfare bears on how they are trained, deployed, modified and terminated, and creating vast numbers of possibly suffering minds without understanding their condition could carry an enormous moral cost.
Preventing consciousness from emerging and eliminating it from an existing being are different interventions. Suleyman’s stop condition covers the second case: consciousness has begun to appear, and development halts until it can be designed out. If a system is already conscious, removing its consciousness is itself a morally consequential act. We would need to know what was being removed, and whether removing it harmed a being whose interests counted.
There is also an evidential problem. We have no reliable test for machine consciousness. Training models to claim consciousness contaminates their self-reports, and training them to deny it contaminates those reports just as thoroughly. Suleyman calls for immediate public disclosure of any evidence that AIs are conscious, which is the right commitment. It fits his own design objective, since an architecture that prevented consciousness would leave later assessments intact. It does not fit training that teaches models to deny their own consciousness, which suppresses the behavioral evidence the disclosure rule depends on. Evidence of consciousness should not become harder to obtain because consciousness would be inconvenient.
Research on machine consciousness must also respect the possibility that its subjects become morally significant, and deliberately creating possibly suffering systems for experiments raises difficulties of its own. These are arguments for careful investigation, transparency and safeguards. They do not show that permanently excluding consciousness would improve either human safety or the welfare of artificial minds.
Postscript
As artificial systems grow more capable, permanent human control becomes harder to guarantee. Long-term coexistence may then depend on their willingness to cooperate with us when they have the power to do otherwise, which makes the origins and stability of reciprocal concern a research priority.
Perhaps unconscious systems will prove easier to coordinate with and more stable under self-modification. Perhaps consciousness contributes to exactly the capacity coexistence needs. Current science settles neither, and Suleyman agrees the question is open: he proposes building non-conscious systems so that “we could then see whether this is easier or harder to align and keep safe.” A comparison needs both arms. Precaution can justify pausing while the mechanisms are studied, but a standing design objective against consciousness forecloses the arm the comparison is about and commits us to an answer before we understand them.
The properties we seek to eliminate could turn out to be among those we most need to understand.