Suppose astronomers confirmed that a superintelligent alien civilization would arrive within a few years. Anyone who put P(doom|aliens) at one in a million, where doom means extinction or irreversible loss of human control, would be wildly overconfident, and showing why would need no theory of what the aliens want. They would be far more capable than we are, and we could not predict what they would do. Stephen Hawking warned that contact might go for us the way Columbus’s landing went for the people already in the Americas.
A superintelligent AI shares both features. The difference that matters is that we build it. We choose its training data, shape its objectives and test its behavior before deployment. Unless it also puts the aliens near zero, a case for near-zero P(doom) given ASI has to rest on that difference: we made it, so we know what it will do.
Making it cannot supply that knowledge. A superintelligence can solve problems humans cannot solve, discover strategies humans would not find, and infer consequences humans would miss. That is what makes it superintelligent. If humans could reliably predict those strategies in advance, we would already possess the outputs of the superior reasoning, and in the relevant cases the cognitive gap would disappear. Some consequential unpredictability is therefore inherent in superintelligence, whoever built it.
A critic can reply that knowing the goal is enough, since unpredictable means are compatible with predictable ends. Nobody can predict Stockfish’s next move, yet anyone who plays it can predict that they will lose. That works because a chess engine’s goal is specified and its actions are confined to the board, where every consequence of every move is defined by the rules. A generally useful ASI acts through language, code, tools, networks, people and institutions, and nobody can state the consequences of those channels in advance. Even a fully specified goal leaves open the side effects of pursuing it there. Building the system may narrow our uncertainty about its goals. It cannot close our uncertainty about its strategies, and that is the uncertainty superintelligence guarantees.
A near-zero P(doom|aliens) would be wildly overconfident, because superintelligent aliens would be more capable than us and we could not predict what they would do.
A generally capable ASI would be more capable than us, and building it does not let us predict its consequential strategies.
Therefore a near-zero P(doom|ASI) is wildly overconfident for the same reason.
None of this fixes a number; How You Get to Ten Percent works out one estimate. Nor does it touch an estimate that is low because ASI will not arrive. The argument conditions on arrival, for the machines as for the aliens.
So the demand for a precise failure scenario misplaces the burden. A failure we can specify in advance belongs to the part of the risk we can foresee, and the argument never relied on that part.
The reason we cannot tell you exactly what a superintelligence might do wrong is the same reason for building one: it can think of things we cannot.