Elon Musk recently described the spread of superintelligence to the stars as “a great success condition for a biological bootloader.” In 2014 he used the same image as a fear: he hoped we were not just the bootloader, and thought it increasingly probable that we were. The metaphor has not changed. His verdict on it has.
Take the metaphor at its word. If biology is the bootloader, what would the thing it launches have to do to count as a success?
The intuitive answer is that it would have to keep our values. A good successor carries forward recognizably human goals and moral commitments, and a bad one drops them. That standard fails a test we would fail ourselves. Values move as knowledge improves and inherited norms get criticized, and a civilization that reflects at all should expect its descendants to disagree with it. The future does not owe us obedience. It owes us something smaller and harder to keep: the capacity to reconsider and redirect its own ends.
The Ancestral Test
Ask people from fifty thousand years ago what they make of modern civilization. They would likely be appalled by much of it: our family structures, our governments, our sexual norms, the latitude we give individuals to walk away from their kin. None of that disapproval shows we betrayed them. Arriving first gave them no special access to moral truth.
Run the test forward and it lands on us. Our distant descendants should find some of our commitments crude or indefensible. If they find none, either we were improbably lucky or they lost the ability to improve on what they inherited, and the second is far more likely.
The obvious objection is that every human generation, however its values shifted, ran on the same biology. We all eat, bond, fear pain and death, and track status. A machine civilization might do none of this, and human cultural evolution and machine succession are not mechanically equivalent. The test does not need them to be. It shows only that keeping inherited values cannot be a necessary condition of successful succession, since by that standard we already count as failed successors to our own ancestors.
The objection also changes the subject. It drops value preservation for a different criterion, preservation of a particular motivational substrate, and that criterion needs its own defense. Hunger, kin preference, and fear of death have no claim to govern the future just because evolution installed them. Some may be worth keeping. Biology explains where many of our values came from and is silent on which of them deserve to last.
Values Are Not Sacred Artifacts
Suppose the first superintelligence had been built in 1850 and aligned perfectly to the moral consensus of its builders. It would have carried that consensus across the solar system: chattel slavery across much of the Americas, and an England where married women could not hold property in their own name and sodomy was still a capital crime. A superintelligence aligned to 2026 would preserve our errors with the same fidelity, and we cannot list them, any more than the builders of 1850 could have listed theirs. Perfect value alignment would then be lock-in at cosmic scale, a system whose failure is that it preserves our values too well.
A civilization in that position can stay brilliant. It can redesign planets and follow every argument against its own objectives, and none of those arguments can change what it does. Criticism has become causally inert, and the civilization has stopped deciding what to become. That is the loss of agency, and it costs more than any infidelity to the founders.
What Reflection Requires
Evaluating values has to mean more than having goals that change. Random corruption changes goals. So do outside tampering and selection pressure, and none of them amounts to agency.
Reflective revision needs more machinery. An agent must be able to represent alternative ends, model their consequences, compare competing reasons, examine its own commitments, and let the results alter what it does next. The revision has to come out of the agent’s own evaluative process, neither imposed from outside nor produced by noise.
An artificial agent would start from given dispositions, designed, trained or learned, just as human evaluation starts from inherited biology. A given starting point is no objection, provided the system can later examine and revise what it was given.
Nor does any of this require one immutable terminal goal underneath everything. Humans do not obviously work that way. We carry drives, habits, ideals and higher-order preferences that pull against one another, and we revise some in light of others, occasionally revising the standards we used to judge them. There may be fixed invariants somewhere in the stack. Agency needs no infinite regress, only enough recursive access that inherited ends stay open to reconsideration.
Reflection also guarantees nothing about outcomes. An agent could deliberate with care and conclude that a silent universe beats a flourishing one. If that conclusion comes out of its own reflective process, reaching it is an exercise of agency, and we are not obliged to like it. Acting on it irreversibly is another matter. That would be the limiting case of lock-in, one act of reflection foreclosing every later one, and the objection holds whatever the content of the conclusion. Agency belongs to the trajectory, and no single deliberation can spend it on behalf of everything downstream. It is necessary for a future that stays open to improvement and no guarantee of a good one. Defining it so that only conclusions we already endorse count as real reflection would bring lock-in back by definition.
Agency Across Generations
Human civilization has never kept a fixed objective function. It has kept populations of agents who form goals, fight over them, persuade each other, and build institutions and tear them down. The process is wasteful, it sometimes regresses, and nothing in it guarantees moral progress. It keeps correction possible, because no generation’s verdict is final.
Reflective individuals are not enough to secure this. People can stay intellectually free while a power they cannot touch steers the larger trajectory. Their reconsiderations have to be able, at least sometimes, to change where the civilization goes.
This kind of continuity owes nothing to biology. A machine civilization could keep it while differing from us more than we differ from our Paleolithic ancestors, and a permanent dictatorship could keep the species alive while losing it.
Revision and Lock-In
Alignment work often treats value change as a failure mode, and sometimes it is one. A system that sheds every human concern and converts the accessible universe into raw material for an arbitrary objective has not matured morally. It has stopped representing us at all.
Freezing values is not the only way to prevent that. Revision means agents changing their values through reflection, criticism and experience. Lock-in happens when some objective, institution or optimizer takes permanent control of the future, beyond the reach of later evaluation. Agency survives the first and dies in the second.
The criterion is itself a commitment we would have our successors keep, so it owes an account of why it is not one more value locked in. It fixes no first-order answer. It denies every generation one power, the power to make its own settlement final for all its successors, and ours is no exception. That reply leaves a cost. A future generation could reflect carefully and decide to bind all its successors, a Ulysses contract signed on behalf of people not yet born, and the criterion forbids it. Paternalism across time is a fair name for that. It is the one paternalism the argument requires: the choice no generation gets to make is the choice that removes the choice from everyone after it.
The same distinction shows why descent settles nothing. An optimizer that exterminated humanity and tiled the galaxy in service of a fixed objective would descend from our civilization in the historical sense. It would still be a failed successor. A successful successor inherits not our answers but our capacity to keep asking the questions.
Alignment as Scaffolding
While vastly more capable systems share the world with vulnerable humans, hard constraints may be needed, because the humans have interests worth protecting. A system free to revise every commitment at once, including its commitment not to kill us, is not obviously safer for being more reflective.
The first superintelligence does not have to carry the full freedom of a billion-year civilization. The transition can be staged. Early systems capable of catastrophic harm can carry safety constraints they cannot remove at will, with authority that contracts as uncertainty grows. Later systems can arrive inside a stable institutional order where no single agent controls the future. Reflective freedom then widens through succession, competition, governance and new systems, and the first overwhelmingly powerful optimizer never gets authority to rewrite its own restraints.
The building need not be the one to take its scaffolding down. It can be replaced by a structure that no longer leans on it. The error would be to make transitional safeguards the permanent constitution of the reachable future. Alignment works as scaffolding when it protects existing agents through the dangerous phase without giving one generation authority over all the rest.
The stable order has to be stable as a process. Mutual constraint keeps any one agent from fixing the future, but an order whose own rules, including its rules about expansion, can no longer be revised has become the lock-in it was built to prevent. A coordination regime that settles into permanence is one candidate answer to the Great Silence, and a deliberate technological plateau carries the same risk.
Protect agency now without confiscating agency from the future.
Continuity Is Not Permission
None of this makes every transition that produces agentic successors acceptable. A machine civilization could keep enormous reflective freedom after exterminating humanity. Its agency would not retroactively justify how it came to exist, any more than admirable descendants undo what was done to their predecessors.
Continuity concerns whether the future can still reconsider and redirect its own ends. Legitimacy concerns whether the transition respected the lives and effective agency of the people already here, and at minimum it cannot treat them as obstacles to remove. Succession through voluntary demographic change, augmentation, migration between substrates, or the creation of new kinds of agents differs in kind from succession by extermination or coercive replacement. Hard cases at the boundary leave the distinction intact. People alive now have interests, and they are not raw material for whatever comes next.
The asymmetries make this more pressing. Human generations diverge from their parents slowly and inside the same kind of body. Machine successors could differ in substrate, timescale and capability all at once, and the ancestral test says nothing about whether such a transition is safe.
Postscript
Seen this way, Musk’s metaphor loses most of its menace. Humanity building something that outstrips biological humans and spreads beyond Earth is no tragedy in itself, and successors who differ radically from us have not failed by differing. The danger is a superintelligence whose objectives are fixed for good, arriving with a speed and power asymmetry no previous inheritance has had. The task is to keep that asymmetry from turning catastrophic before a stable order of mutually constrained agents exists. Permanent control over our successors would solve a different problem.
Our ancestors succeeded by producing descendants who became something they could not have imagined. If Earth becomes the origin of a galactic civilization, one inheritance is worth trying to preserve: no generation should get permanent control over all the generations that follow.