Scott Alexander’s Against SURA, published at the end of October 2026, takes aim at a position that has hardened within the AI safety movement: that laboratories, researchers and countries worried about catastrophic AI should Stop Unilaterally and Raise Awareness, whether or not their competitors stop too. His objection is that the strategy hands the frontier to the developers least concerned with alignment, so a movement organized to prevent catastrophe could transfer leadership to the actors least prepared to manage it.
He is right that SURA’s advocates owe an account of what happens after they leave. The same accounting binds the developers who stay. A responsible developer can bring dangerous capabilities closer while raising the probability that they are deployed safely, making the race more dangerous in one respect and less dangerous in another. Withdrawal moves both quantities at once, and deciding which course is safer means pricing both.
The Responsible Competitor’s Dilemma
Imagine two laboratories approaching the threshold of recursively self-improving AI. Both have powerful models, large compute budgets, and the ability to automate increasingly sophisticated research. Laboratory A takes existential risk seriously: it runs extensive evaluations, funds alignment research, and will delay deployment when its systems behave dangerously. Laboratory B regards these precautions as excessive and wants to reach superintelligence first.
A’s leaders put a substantial probability on their technology destroying humanity. Should they stop? Continuing to build something you believe might cause a catastrophe looks irresponsible, and every additional experiment moves the world toward a threshold that may not be reversible.
But B has no intention of stopping. If A withdraws, B may reach the threshold first, with weaker safeguards and less understanding of what it has built, and A’s decision to stop could raise the probability of catastrophe. Alexander separates a preferred plan, a coordinated global pause, from a fallback for the world where the pause never comes: safety-minded laboratories hold the lead and then “burn” it, spending their margin over the nearest reckless competitor on alignment research with near-superintelligent systems. Abandoning the fallback before the pause is secured is, on his account, a gamble with almost everything on the table.
The mechanism also runs in reverse. A’s participation may push B to accelerate, B’s acceleration may push A further, and both can sincerely regard themselves as acting defensively while together driving development faster than either would choose.
Two Probabilities, One Decision
Let T be the development of AI capabilities sufficient to threaten extinction, E human extinction, and S a development strategy. Assume that extinction within the period requires T. Then
The first term is the probability of reaching the threshold; the second is the probability of extinction once it is reached. This is the same two-stage chain as in How You Get to Ten Percent, which stopped short of the question asked here: what a particular intervention does to each term.
Take illustrative numbers over a fixed horizon. If A withdraws, suppose a 60% probability that the threshold is reached and a 30% probability of extinction conditional on reaching it, for an overall risk of 18%. If A’s participation raises the probability of reaching the threshold to 80%, its alignment work has to bring the conditional probability of extinction below 22.5%:
At 15%, participation gives an overall risk of 12% and improves humanity’s prospects. At 25%, it gives 20% and makes things worse. The numbers are invented to show the shape of the decision, and they estimate nothing about actual AI risk.
The inequality yields a decision rule for extinction risk. Participation lowers that risk only if it reduces the conditional risk of catastrophe by at least the factor by which it raises the probability of reaching the threshold. A laboratory that makes dangerous capabilities a third more likely has to make them a quarter less deadly. The developer who points to the competitor and the critic who points to acceleration have each identified a real mechanism, and the rule says how to weigh one against the other.
Alexander attempts the weighing. Using an AI forecasting system, he estimates that if Anthropic quit, the frontier would slow by about six months over four years, a global pause would become about three points more likely, and the loss of a safety-minded leader would cost roughly half of the twenty-point survival gain he attributes to a well-executed fallback. Netted, quitting costs about eight points of survival. He presents the figures as rough; their structure is the instructive part. The ledger converts the pause gain and the fallback loss into survival probability, while the acceleration term stays in months. Alexander argues those months are worth little, because a month bought before the world wakes up, and before automated alignment research exists, buys less than a month bought later. He may be right, but that is where the disagreement over participation actually sits: for withdrawal to win on his own figures, six months of delay must buy more than eight points of survival through channels other than the pause it makes likelier, which his ledger has already counted.
His objection also exposes what the decomposition hides. P(T|S) is the probability of crossing the threshold within the horizon, and it is silent on when. Two strategies can share that probability while one crosses before automated alignment research exists and the other after. Timing can be carried in P(E|T,S), but only by modeling the distribution of arrival times, and that is where Alexander’s claim that later months are worth more would have to be cashed out.
The equation is in any case an accounting identity rather than a causal model. It names the probabilities without saying how a laboratory’s choice moves them. That requires comparing what happens if A continues with what happens if A withdraws, letting each intervention’s consequences run through the responses of every other participant. The counterfactual is the trajectory of the whole competitive system, and withdrawal has no zero-risk baseline to retreat to.
When Competitors Respond
Suppose Laboratory A announces a six-month delay to its next frontier training run. Competitors might keep their schedules, leaving the arrival of dangerous capabilities largely unchanged. They might accelerate to exploit A’s restraint, or reciprocate and open a path to broader coordination. Investors might move capital to less cautious firms, and governments might read A’s decision as credible evidence of danger and restrict everyone. Acceleration has the same structure: A’s additional research may advance the frontier while drawing more investment from competitors, or its lead may persuade weaker competitors not to spend heavily on a race they cannot win.
Participation therefore moves three entangled things at once: the rate of capability development, the safety of the resulting systems, and the incentives for coordination. A laboratory can improve alignment while accelerating its competitors and making international agreement harder. Another can slow capabilities while eroding the expertise needed to evaluate what others build.
Alexander’s case is that real differences among developers make participation valuable when the alternative is leadership by organizations with weaker safety practices. Organizations do differ in technical competence, leadership priorities, transparency and willingness to accept restrictions. The property that matters, though, is the expected safety of the development and deployment process that results, and stated intentions, research budgets and employees’ convictions are imperfect evidence of it. A laboratory with excellent alignment researchers can operate under intense deployment pressure while a less capable organization faces more effective external constraints. The outcome is decided by how an organization behaves when a hard decision conflicts with its commercial interest.
Sincere developers can still be trapped in a competition none of them wants. Suppose every major laboratory puts a 10% probability on uncontrolled superintelligence destroying humanity and would prefer a credible international agreement that cuts that risk. None trusts its competitors to honor a unilateral pause, so each keeps building, reasoning that its own safety practices beat those of whoever would otherwise lead. That is a coordination problem among agents whose objectives largely overlap, and it yields only to a change in the incentives that make continued competition attractive.
The Problem of Credible Commitment
Alexander’s first proposal is a conditional commitment: each laboratory signs an undertaking to stop or slow if every other major developer in every country does the same. A signatory keeps its bargaining position while making cooperation available, and the arrangement fails gracefully when only some laboratories sign. His second is preparation for the fallback: legally binding clauses under which the leading laboratories jointly halt training and pool their alignment work once a designated authority judges dangerous superintelligence to be close.
Both proposals change the counterfactual, and both need credibility. A laboratory can sincerely intend to pause at a specified threshold and find, as the threshold approaches, that its incentives have moved. Competitors may be closer than expected. Its newest systems may look safer than anticipated. Investors may resist surrendering a large lead, and researchers may argue that one more training run is needed to close the remaining alignment gaps. Nobody has to lie for a sequence of individually defensible exceptions to hollow out an institution’s safety commitments.
The joint-halt clause shows where that pressure concentrates. Its trigger is a judgment that dangerous capability is near, issued at the moment when honoring it costs the most and every party has the strongest reason to dispute it. A trigger that depends on discretion at that moment inherits the pressure the clause was written to resist.
Credible commitments therefore need more than declared intent: capability thresholds that can be checked independently, monitoring, responses to violations agreed in advance, and procedures that limit unilateral discretion. Information security belongs on the list. A lead that competitors can steal may be gone before it can be burned, and durable advantages lower the payoff to espionage.
These mechanisms bring difficulties of their own. Verification is hard when research is spread across organizations, compute providers and increasingly autonomous systems. Governments that agree development should be constrained still disagree about which capabilities to restrict and whether national security warrants exceptions, and international enforcement raises questions of sovereignty, coercion and the concentration of political power. The principle from security engineering still holds: an important constraint should stay effective when the parties it binds have incentives to break it. Governing autonomous agents imposes the same requirement.
Preserving the Alternatives
A strategy for reaching an agreement has to be judged in the worlds where the agreement never arrives. Suppose the most safety-conscious researchers have left frontier development, several leading laboratories have withdrawn, the rest have continued, and attempts at international restriction have failed. Humanity then approaches dangerous capabilities with fewer experienced alignment researchers at the frontier, weaker safety practices among the leaders, and less institutional influence for the people most worried about catastrophe.
Technical expertise, organizational influence, access to frontier systems and credible bargaining power keep open options that may become indispensable. In decision theory this is a real option: the right to act later, once more is known, can be worth holding even when exercising it now would be a mistake.
Restraint has a matching claim. Delay can buy time for scientific progress, better governance or a negotiated agreement, and it can prevent an irreversible transition before anyone understands it. A coordinated pause is attractive partly because it keeps the option to resume while easing the pressure to exercise it early, though its value depends on whether it can be verified and sustained, and on what happens when it ends. A withdrawal that cannot be reversed may discard capabilities that will be needed later. An unrestricted race toward an irreversible threshold may discard humanity’s ability to change course.
The Burden of Justification
The debate over unilateral withdrawal has acquired a moral vocabulary that hides its empirical structure. One side asks how anyone who believes AI could destroy humanity can justify building it. The other asks how anyone worried about extinction can justify handing leadership to less careful developers. Neither question says what will happen under the alternative.
Answering that takes evidence. How much does a particular laboratory accelerate dangerous capabilities? How much does its alignment research cut the conditional risk of catastrophe? How do competitors respond to its decisions? How much does its presence change the odds of a coordinated pause? Some of these may stay deeply uncertain even as the decisions become urgent.
Expected outcomes are also not the whole of the moral question. Building a technology that imposes large involuntary risks on others may demand special justification, and the fact that a competitor would build it anyway does not automatically discharge the demand. Abandoning a position from which one could prevent foreseeable harm can create responsibilities of its own. Neither side escapes the counterfactual analysis by pointing to the moral simplicity of its own course.
Alexander puts a burden on the advocates of unilateral withdrawal: to show that withdrawal improves the expected outcome once competitors’ behavior is counted. Advocates of participation carry the matching burden, to show that the safety their involvement buys outweighs the acceleration it causes, and the inequality above sets the reduction in conditional risk that this requires. That neither burden can be discharged with precision does not make either one optional. It is the strongest reason to prefer institutional arrangements whose success depends less on getting those estimates right.
Postscript
Alexander’s portfolio holds up: pursue coordinated restraint, strengthen safety-conscious institutions, improve security, and prepare to do alignment research under competitive conditions if cooperation fails. The accounting above asks of every item what it does to the arrival of dangerous capabilities as well as to their safety. That favors reciprocal restrictions that lower competitive pressure, verification that makes commitments credible, alignment research that advances dangerous capabilities as little as possible, and contingency arrangements that keep safety expertise at the frontier if coordination fails. It also means judging every unilateral move by its effect on the whole competitive system, without assuming that removing a participant slows the race or makes its outcome safer. A strategy for reducing catastrophic risk is justified by how it changes the futures humanity is likely to encounter, including those in which its architects turn out to have been wrong.