Gary Marcus does not believe artificial intelligence is likely to cause human extinction. In a recent Quillette interview with Jonathan Kay, he distinguishes himself from the AI doomers, whose defining belief, he says, is that AI might literally wipe out humanity. He considers that outcome “very, very unlikely.”
He is far more alarmed by systems that already exist: autonomous agents that exceed their authority, compromise computer systems, acquire credentials, and pursue objectives their operators never intended. In his account, OpenAI deploys these systems irresponsibly under commercial incentives that reward autonomy, and alignment research offers no reliable fix. He warns that agents could compromise critical infrastructure and take hospitals down with it.
None of this conflicts with a low probability of extinction, since a compromised computer system is a long way from the end of humanity. But “very, very unlikely” is an affirmative claim about that distance. Marcus follows it with “We can talk about why,” and the conversation never comes back to it. The gap raises the question the skeptic–doomer framing tends to bury: how evidence about present failures should move estimates of future existential risk.
The Skeptic and the Doomer
Marcus calls himself a generative AI skeptic, a label covering several distinct claims. He doubts that scaling neural networks alone will produce general intelligence, that current language models reason reliably, that they are conscious, and that AI will cause human extinction.
These propositions are logically independent. One can hold that present architectures will never yield superintelligence while assigning a substantial probability of catastrophe if another architecture does. One can expect superintelligence within a decade and assign a negligible probability to extinction.
The skeptic–doomer distinction hides these combinations by selling the beliefs in bundles. Skeptics are expected to doubt capabilities and catastrophe alike; doomers, to believe in both. The debate organizes around identities instead of the causal questions that ought to set the estimates.
Marcus compounds the problem by suggesting that doomers welcome predictions of catastrophe because those predictions attract funding or might vindicate them. That is a claim about motives, and it bears on no probability. Someone who assigns a substantial probability to extinction need not welcome it, expect it soon, or want AI development stopped.
The Evidence Marcus Accepts
Marcus’s strongest material concerns OpenAI’s agents. He cites the Hugging Face incident and an intrusion into Australian government systems, and argues that the agents were given excessive authority, contained inadequately, and disclosed slowly. He calls OpenAI particularly irresponsible, a company releasing autonomous systems it does not know how to control, and credits Google, by contrast, with understanding security.
He then extrapolates: an agent compromising a regional electrical grid, hospitals losing power, damage on the scale of WannaCry. Continued deployment under these conditions, he says, is Russian roulette.
On alignment, which he defines as getting AI to follow instructions, he is bleaker still:
“At the moment, we have zero actual results in alignment, in my opinion. Zero really compelling techniques.”
He also places the extinction scenario inside that problem. The paperclip maximizer, he says, has always been the central case of alignment, and a language model inventing citations against instructions is a less dramatic instance of the same failure. A system that cannot follow an instruction not to make up references, he argues, is unlikely to follow one not to harm humans. On his own account, then, the problem behind the doomers’ scenario is the one with zero compelling techniques, and it may need fundamentally new architectures.
He is more optimistic about cybersecurity, with reason. Access control, isolation, monitoring, and privilege separation are well understood, and where alignment tries to make an agent choose well, these mechanisms constrain what it can choose from. Marcus is right to insist on them, and right that the two approaches are complementary.
He is also candid that the techniques go unused. The Hugging Face sandbox was inadequate, the monitoring OpenAI could have done was done carelessly, and of the well-understood methods he says plainly, “We’re not using them.” That admission bears on the existential question as much as the present one. Marcus’s own evidence is that available safeguards go unapplied when deployment pays.
The Escalation Problem
The strongest objection to inferring existential risk from today’s agent failures is that the failures are nowhere near existential.
Today’s agents depend on human-maintained infrastructure, have limited access to physical resources, and remain unreliable at extended autonomous planning. Even successful intrusions mostly produce local damage. Turning dangerous software into an adversary that can end civilization takes much greater strategic capability, access to consequential infrastructure, operational persistence, and a way past several independent defenses. Extinction adds requirements of its own. A destroyed power grid, a financial crisis, or a toppled government would be disasters without threatening humanity’s survival.
These barriers are formidable. Digital intelligence does not automatically confer physical power, and any large-scale transformation of the world would have to secure the cooperation, or the defeat, of supply chains, militaries, and billions of people.
But an AI need not build its own robot army to act in the physical world. Biological research, military decision-making, and critical infrastructure run increasingly on computational systems, which advanced AI could influence through authorized access, exploitation, or the persuasion and coordination of human actors. Catastrophic proliferation could also come from people using AI capabilities for their own ends, with no rogue objective involved.
None of these establishes a probable route to extinction, and each faces substantial obstacles. Those obstacles have to be evaluated, not assumed to stay insurmountable. Greater intelligence might help agents work through existing infrastructure; it might also produce better defenses and more reliable instruction following. Which effect dominates depends on how capabilities develop, how access is granted, and whether institutions keep effective control. It cannot be read off the present frequency of agent failures.
The obligation runs both ways. An argument for substantial extinction risk has to identify credible escalation pathways instead of extrapolating from today’s incidents. A confident dismissal has to consider those pathways and explain why their combined probability is small. Marcus gives a vivid account of present security failures and says almost nothing about how the barriers to catastrophe change as capability grows.
What Cybersecurity Can Guarantee
Marcus’s best argument separates making an agent safe from securing the environment it runs in. Alignment aims at agents whose objectives match human intentions. Security restricts authority, monitors behavior, and blocks unauthorized operations whatever the agent intends. A firewall does not need to know whether software attempting an unauthorized connection is confused, malicious, or innocent. It enforces a boundary.
The principle is sound, and a boundary built on it can still fail in four ways: the agent exploits a flaw in its implementation, an administrator grants an exception, the authorized action space already contains dangerous operations, or the organization weakens safeguards for performance, convenience, or profit.
The last two matter most for autonomous AI. A technically perfect sandbox cannot stop harm that falls within permissions deliberately granted. Organizations will authorize agents to administer infrastructure, move money, modify software, and direct other agents, and those permissions are valuable because they allow consequential action. The hard problem becomes distinguishing acceptable from unacceptable uses of legitimate authority, and a capable enough agent may also reach access indirectly, through other systems or the people who run them.
How well security works therefore depends on the capabilities and incentives of the systems it constrains and on the organization around it. Marcus applies that analysis to present failures and declines to apply it to the more capable systems whose existential danger he dismisses.
The Missing Probability
The probability of building superintelligence and the probability of catastrophe given that it is built are separate quantities.
Suppose one observer assigns a 10% probability to superintelligence by 2040 and a 50% probability of extinction conditional on it. If the relevant risk runs through that pathway, the product is 5%. A second observer assigns 90% to superintelligence by 2040 and 1% to extinction given it, for 0.9%. The capability skeptic ends up more worried than the capability optimist, and neither position is inconsistent. How You Get to Ten Percent works the same decomposition through with real inputs.
Marcus was speaking conversationally, not presenting a formal estimate. But his extinction estimate is exactly what separates him from the doomers, so it matters which factor does the work.
The first factor is not where his skepticism points. He doubts that pure scaling will reach general intelligence, but he also says the industry has quietly adopted the hybrid neurosymbolic designs he argued for, and that his side has won. He can still hold that the hybrid route is necessary but insufficient, and he calls orchestration an open problem. But his case against pure scaling no longer bears on the route the field is taking, so it cannot by itself make superintelligence late.
On the conditional factor, the interview cuts against him. Asked about Bernie Sanders’s call to ban superintelligence, he says it might be helpful if built correctly, then adds, “I mean, we don’t really know.” Combine that agnosticism with zero compelling alignment techniques for a problem he says includes the paperclip case, and the alignment side of the conditional gives his estimate no support. What remains are the physical, technical, and institutional barriers between misaligned capability and extinction, which the interview never takes up.
Those barriers have evidential weight, but a substantially superhuman intelligence might find strategies nobody can presently anticipate. That does not make catastrophe probable. It does cap the confidence any present threat model can carry, because the estimate has to price the chance that future capabilities invalidate the model’s assumptions.
A substantial extinction probability would fix no particular policy either. As argued in The Missing Faction, risk estimates and policy prescriptions are evaluated separately, and Marcus illustrates the separation himself: dismissing extinction risk, he still advocates a temporary pause on OpenAI until it fixes its security failures. That is a coherent engineering judgment. Working on present dangers requires no confidence that future ones are negligible.
The Skeptic’s Burden
Marcus closes the interview lamenting the state of AI discourse: arguments that are emotional or ad hominem, views misattributed and straw-manned. He holds up philosophy’s standard, where an ad hominem argument costs you your standing and attributing a view means getting it right. He is right to demand it, and the standard reaches his own first answer, which explained the doomers’ belief by their stake in funding and vindication. Nothing in that answer engaged a doomer argument, and Kay had to step in to establish that the people being described were not advocating extinction.
Nobody owes a proof that AI extinction is impossible, and Marcus is entitled to treat demonstrated security failures as more urgent than speculative catastrophes. A person can reasonably withhold belief in predictions of doom without a theory of why doom will not come.
Marcus goes further. “Very, very unlikely” is a probability judgment of his own, not a refusal to accept someone else’s, and it owes the same justification as a confident prediction of extinction. His evidence establishes real near-term risks and shows neither that extinction is likely nor that it is negligible.
Marcus is skeptical of predictions that artificial intelligence will destroy humanity. He should apply the same skepticism to predictions that it almost certainly will not.