Lex Fridman offered Jensen Huang a definition of AGI: a system that could start a technology company, grow it, and run it until it was worth billions. Huang said we were already there. Fridman had said billions, he pointed out, but had not said forever. An AI that builds a viral app, clears a billion dollars and folds would satisfy the terms.
Brett Hall wants to know what that is supposed to have shown. The word has come to cover broad competence, economic substitutability, autonomous task execution, human-level cognition and artificial personhood, so a milestone in any one of them gets reported as progress on all five.
He is right to ask, and he is not making the naive mistake of confusing capability with mind. Hall argues that the question behind AGI should be whether we have built a person: something self-directed, creative, subjective, able to run into a problem nobody handed it and invent an explanation. On his reading the industry kept the word and quietly swapped the problem, measuring what systems can do and reporting the result as general intelligence while the ambition the word was coined to name sits where it always sat.
The trap is baited with something real. Personhood is the thing worth caring about here. Whatever else is at stake in AI, a mind that can be wronged is the part that matters most, and Hall is right to keep it in view. But a concept can be the most important thing in the room and still be the wrong ruler. Hall picks up personhood to measure intelligence, and then reads the absence of personhood in current systems as evidence that their generality is thin.
So the disagreement is not about vocabulary. Hall has an identity thesis: the general intelligence worth the name is the intelligence of a person. Universal explanation, self-prompting, benchmarks, training data, creativity and contact with reality are the six defences he mounts of it. None of them holds.
Two projects
Hall does eventually separate two projects. One is technological: build systems that reason, calculate, program, plan, solve harder and harder problems. The other is philosophical: work out what makes something a person. He thinks the field has poured a generation of effort into the first and let the second lapse.
The separation is worth having, because it exposes how much ordinary AI talk compresses into a single word. Capability is what a system can do. Generality is how far that competence reaches and transfers. Autonomy is how much of a goal it can pursue without anyone steering. Agency asks whether its behavior is best described through plans and beliefs and choices. Consciousness asks whether anything is felt. Personhood and moral status ask about continuity, interests, vulnerability, and what the thing can claim from the rest of us.
Nothing makes those cross the same line at the same time. A system can be broadly competent and dark inside. It can act autonomously and remain entirely someone else’s instrument. Personhood survives sleep, anesthesia and a bad concussion. Moral standing comes apart from political standing routinely: children have the first without the vote, corporations have legal rights without an inner life.
Hall wants a brighter line than this. Personhood, he says, is binary. A thing either is a person or it is not, and treating cognition as continuous dissolves what is distinctive about persons into a smear running from humans through dogs and machines down to bacteria.
The warning inside that objection is fair. Failing to find a sharp boundary does not show there is none. But the inference does not run backward either, and this is where the ruler slips. A category can be indispensable to law and morality without a matching seam in the world. The law needs an age of majority, so it picks one. Nothing in a nervous system changes at midnight on your eighteenth birthday. I have argued the general case in The Intentional Gradient: the vocabulary of agency is earned by degrees, and nothing is handed over at a threshold.
Hall’s own example, dogs or LLMs getting the vote, runs the same compression. Moral patienthood, legal personhood, citizenship and the franchise answer four different questions. A dog can matter without joining the electorate. How we sort things for political purposes tells us nothing about whether cognition has a seam in it.
That leaves one boundary doing the work of four. Whether a thing is intelligent, whether it is a person, whether anything is felt, whether it can be wronged: Hall’s line has to fall in the same place for all of them. The rest of his argument is an attempt to show that it does.
Universal explanation
His route runs through Critical Rationalism, which is the best road he could have taken. The claim that humans are “universal explainers” belongs to the Popperian and Deutschian account of knowledge as something made out of problems, guesses, criticism and correction. What distinguishes a person is not a large stock of answers. It is the ability to walk into a problem with no answer available, invent one, hand it to criticism, watch it fail, and come out the other side knowing something nobody knew before. That is a far richer notion than benchmark competence, and Hall is right that no list of tasks captures it. Humans would fail almost any sufficiently broad list, since no one person can do everything some person can do. Generality in this sense is about open-ended reach, not about holding every skill at once.
I have argued elsewhere that nobody has established the universal-explainer claim for humans in the first place. Grant it here. Even on Hall’s own terms it does not fence out machines.
Critical Rationalism also gives his objection to machine learning its teeth. Popper denied that knowledge grows by induction: observations do not grind out theories, theories are guessed and then attacked. Hall sees the current systems as parasitic on an enormous pile of existing human output, and suspects they can shuffle and extend that pile without ever performing the act that produced it.
This is the strongest version of the argument, and it fails at the first step. The step is an inference from how a system got its competence to what its competence consists of. A system trained by gradient descent on a large corpus does not thereby have to reason inductively in the sense Popper rejected. Training can build machinery that generates hypotheses, weighs alternatives, notices contradictions, searches, calls external tests and revises what it concluded a moment ago. “Learned statistically” and “can only extrapolate statistically” are different claims, and Hall needs the second.
Evolution shows why the inference is bad. Deutsch himself treats evolution as a knowledge-creating process while denying that the knowledge is explanatory: natural selection conjectures nothing and criticizes nothing. A process with no capacity for explanatory conjecture built the machinery that has it. Hall does not conclude from this that every human thought is a form of evolutionary extrapolation. The thing that builds a mechanism and the things the mechanism goes on to do sit at different levels.
Someone might answer that evolution wrote the hardware and left the mind free to write its own software, whereas gradient descent reaches further in and writes the representations themselves. The distinction will not hold. Learning rewrites physical implementation continuously through plasticity, and there is no clean line where evolution puts down the pen and a separable program starts running. However you partition it, the origin of a mechanism does not fix the class of computations it can run. Gradient descent can produce a system that searches; the search does not become disguised gradient descent because of where it came from. Whether today’s systems run the machinery Critical Rationalism cares about is a question for investigation, not for etymology.
Hall comes close to settling it by definition. A person, he says, uniquely “has a problem,” where an LLM only produces output once prompted, and he moves straight from there to the creation of knowledge. But Critical Rationalism needs an account of what a problem, a conjecture and a criticism are that can be applied without first checking who is doing the conjecturing. Otherwise “persons conjecture” is a stipulation wearing the clothes of a discovery.
The evidence runs the same way on the human side. Nobody established that humans are universal explainers by cataloguing every explanation a human could produce. We inferred an open-ended capacity from a finite sample plus a theory of the underlying mechanism, which is exactly how universality is established in computation: a structural argument supported by a handful of tests, never by running every program.
So Hall goes too far when he says benchmarks “cannot matter a jot” to the philosophical question. A benchmark can be worthless as a definition of AGI and still be evidence about transfer, abstraction, planning and error correction. A score cannot manufacture a metaphysical category. It does not follow that scores tell us nothing about which theory of a system survives.
If being a universal explainer has observable consequences, then what machines do bears on whether they are one. If no possible machine behavior could ever count, the concept has been sealed off from criticism, which is a strange fate for a Popperian idea.
A machine that satisfied Hall would have to do more than emit conjectures for humans to test. Suppose a research system notices that an accepted model of some alloy keeps missing one class of measurements. It generates several rival explanations, designs experiments whose outcomes would separate them, runs those experiments through an automated lab, kills the losers, revises the survivor, and uses the revision to predict a property nobody has measured, which the lab then confirms. No single step there makes it a person. But the system would have found the problem, produced the candidates, built the criticism, caught the error, changed its model and gone round again. It would be holding the Popperian cycle together itself rather than being handed its problems and its corrections by human beings.
If Hall’s reply is that all of this happens without the system feeling the problem, then consciousness has become the missing ingredient. That may well matter for personhood. It is no longer an argument about conjecture and criticism.
Problems and self-prompting
Self-prompting belongs to the same epistemology. Asked what makes a person different, Hall says a person self-prompts: people fall for stamps, or Saturn, or Strauss, or seismology, with nobody assigning the topic. Elsewhere he puts it as a person uniquely having a problem while the model waits to be given one.
The claim is stronger than it sounds, and better. In the Critical Rationalist picture a problem is not an item on a list; it is a defect in what you currently understand. Expectations collide with experience. Two explanations you hold turn out to disagree. Something you thought you followed stops making sense. The problem announces itself and drags the search along behind it.
Human minds are full of the standing commitments that make this possible: models, expectations, unresolved questions, appetites, projects that run for years. New information lands among them and starts fights that redirect attention without anyone issuing an instruction.
None of that is obviously off limits to a machine. If a problem is a conflict among predictions, models, goals and observations, a persistent system can be in one. It can hold expectations, watch them break, notice the inconsistency, keep the anomaly around unresolved, rank it against others and start trying to fix it, without waiting for a person to phrase each intermediate difficulty in English.
Hall can say this misses the point, because a person experiences the problem as theirs. Curiosity has a pull. A failed expectation is a disappointment. An unresolved contradiction nags. That may be exactly right about personhood, and it is a different criterion from the one he started with. The distinction is no longer about how problems are generated. It is about whether anything is felt.
Either problems are conflicts inside a knowledge-generating architecture, in which case machines can have them, or a problem requires a subject who feels the conflict, in which case Hall owes us an argument for why feeling is necessary to making knowledge.
The same shape turns up in his question about whether an AI could start a company. Hall reads starting a company as spontaneously wanting to found one, unprompted. But an executive tells an employee to go set up a subsidiary, and the employee then generates a year of problems and solutions without having chosen the objective at all.
Human intelligence runs on borrowed goals almost all the time. Employers hand out projects, teachers hand out problems, institutions hand out roles, biology hands out drives. What we call intelligence shows up in how those constraints get represented, broken apart, argued with, redirected and pursued. Where the top-level goal came from says nothing about the quality of the thinking underneath it.
An artificial agent can take an objective and generate its own hierarchy of subproblems, notice its plan failing, go looking for what it lacks, and change course. That may still fall well short of personhood. Hall has not shown that it falls short of intelligence.
The machine Hall describes
Hall’s conclusion rests on a particular picture of what a language model is. It was trained on an enormous quantity of human-produced text, so he treats its competence as a sophisticated form of access to what people already worked out. Humans meet reality and guess at explanations; models rearrange the sediment left by earlier human guesses.
The training-history argument has already failed once, and it fails here for the same reason. Humans get most of their sophisticated knowledge from other humans. A mathematician spends a decade absorbing established mathematics before proving anything. A physicist inherits three centuries of theory. Learning from what came before is how the equipment for going beyond it gets installed.
“Next-token prediction” does not settle what a trained system can then do. It names an objective, not the algorithm the objective produced. A chess engine trained to maximize its win probability may internally run search, or evaluation, or learned heuristics, or all three; the training target tells you what pressure was applied, not what grew under it.
Hall’s picture surfaces when he calls a language model a “searchable, reorganizable database.” A database keeps records and hands them back when queried. A network holds a transformation smeared across its parameters, and produces outputs for inputs it never saw. Some of those outputs were never anywhere to be retrieved from. There was no record.
This does not mean every surprising output is reasoning. Memorization, interpolation, pattern completion, learned search and abstraction can all be in play, and telling them apart is a real scientific problem. Calling the whole thing a database settles the question before anyone has looked.
His treatment of creativity has the same structure. He grants that a model can produce material absent from its corpus, then files it as novelty rather than originality on the grounds that the system is rearranging what was already there. That distinction needs a criterion, and the obvious ones do not favor him. Einstein inherited geometry, clocks, trains, Maxwell’s equations and a live argument about simultaneity. Darwin inherited geology, taxonomy, pigeon breeders, Malthus and decades of natural history. Explanations have ancestry.
Critical Rationalism never demanded that conjectures come from nowhere. It demands that a candidate explanation go past what was supplied and survive criticism better than its rivals. That is a condition on the shape of a process, and where a system’s representational machinery came from cannot decide in advance that the condition fails.
Which leaves the classification sorting by origin and nothing else. When a human transforms inherited material it is conjecture; when a machine transforms inherited material it is rearrangement; and the only thing separating them is which of them did it. Provenance has replaced epistemology, and the ruler is measuring the worker rather than the work.
Reality and learning
Hall’s last line of defence is that humans stand in a different relation to reality. A person meets the physical world and builds a model of it; a language model meets a compilation of what people said about the world.
But nobody meets the world unmediated. Photons hit a retina, pressure waves hit an eardrum, molecules dock with receptors, and a nervous system transforms those signals through half a dozen stages before anything reaches thought. What we have is causal contact with an environment, not the world presenting itself to a mind. Machines can sit in causal loops too. Sensors, instruments, network interfaces and actuators supply observations whose next state depends on what the system just did. It can form a hypothesis, intervene, watch the result, keep the gap between what it predicted and what happened, and act differently next time.
The architectures differ, sometimes enormously, and Hall is entitled to argue that some specific feature of human embodiment is required for universal explanation. He has to name the feature and show it is necessary. “Humans encounter reality” cannot stand in for that argument once machines can also take the consequences of what they do.
Human knowledge leans on other people’s representations at least as heavily. Nobody rebuilds physics or medicine or mathematics from raw sensation; we inherit books, teachers, instruments and experimental traditions. Culture is a vast external store of earlier guesses, and every child is dropped into the middle of it.
His comparison with babies does point at something real. Children pick up language and concepts from a fraction of the data a frontier model consumes, and he is right to emphasize how much they get from how little. Human development braids together evolved priors, a body, social correction, active exploration and years of continuous interaction, and nothing in current training reproduces that.
What it establishes is a difference in architecture. To get from there to impossibility, Hall would have to say which of the missing pieces is required for open-ended conjecture and criticism. A different route to a capacity is not yet a different capacity.
Personhood and alignment
Hall is unequivocal that present models have no inside: no impulse, no desire, no hope, no fear. For the argument about general intelligence we can grant it. Assume every current model is dark, and that nothing whatever matters to it.
The capability questions survive intact. A system with no experience could still model an environment, plan, and criticize its own failed hypotheses.
Personhood is where Hall’s worry has real force, and here I think he is largely right. A very capable service is a different sort of thing from an agent with projects of its own, a future it has a stake in, preferences that outlast a conversation, and purposes that can come into conflict with what it is told to do. Whatever artificial personhood turns out to require, it is likely to require several of those.
Which is why the identity thesis is dangerous rather than merely imprecise. Pull the concepts apart and a possibility appears that Hall’s framework cannot name: a system that is generally intelligent, highly autonomous, and not a person at all. Enormous planning ability and cognitive reach, with no empathy, no reciprocal vulnerability, no interests, nothing resembling moral psychology. That is the alignment problem. Capability can outrun everything we associate with persons.
If a non-person cannot count as generally intelligent by definition, the most consequential configuration on the table has been hidden inside a word. Unbundling is the condition of being able to describe what we are building.
The moral question clears up the same way. Broad competence does not by itself confer rights, and the absence of human-like motivation does not by itself remove them. What grounds moral standing has to be worked out on its own terms rather than read off a capability measure. Interests, vulnerability, continuity, experience and agency are the candidates; intelligence may be evidence about some of them and settles none.
Hall sees something real past capability. His mistake is packing four things into one word and then using the word as an instrument.
Postscript
None of this rescues the rhetoric Hall is attacking. Companies do attach a philosophically loaded word to operational milestones and let audiences hear more than the evidence carries. Work that generates revenue does not make a system a person. A benchmark score is silent about consciousness. Executing a task unsupervised confers no moral standing at all.
The discipline has to run the other way too. Goals that come from outside leave the intelligence needed to pursue them untouched. A statistical training objective leaves open what mechanisms the training produced. Learning from human knowledge is compatible with extending it. And machine-made novelty is mere rearrangement only if there is some criterion, independent of who produced it, that separates rearranging from conjecturing.
Critical Rationalism supplies the right demand, and it can be made concrete. Find the problem, propose an explanation, expose it to criticism, replace it when something better survives. Ask of a system: can it hold on to an anomaly it cannot yet solve, instead of retrieving something familiar? Can it generate rival explanations rather than one fluent answer? Can it design the observation that would tell them apart, notice its favorite failing, and change the model? Can it carry the whole thing into the next problem without being told each step?
Current systems may do this partially, unreliably, or only inside heavy scaffolding. Later ones may do it better, or some missing piece may turn out to block the road entirely. The answer will come from whichever account of the observed behavior survives criticism, which is the standard Hall would apply anywhere else. And if he would reject a system that met all four conditions, the disagreement rests on some further property, and it is time to say what it is.
Hall asks whether AI researchers are solving the problem of what a person is or just building better tools. The space is wider than that. We may build broadly general intelligences that are not persons. We may build artificial persons by routes nothing like human development. Some systems may make knowledge with nothing felt anywhere in them; others may acquire durable purposes of their own before they acquire an inner life.
He is right that changing a label solves no philosophical problem. Generality, agency, personhood, consciousness and moral standing are five problems, and the work begins by refusing to let one word answer all of them.