By Gideon Black
On artificial consciousness, the strongest conclusion available today may be that neither side has earned the right to stop asking.
There is something refreshing about a serious scientist saying no.
Anil Seth, Ned Block and Giulio Tononi have done so, forcefully, in a recent Nature commentary: today’s AI systems are not conscious, they argue, and perhaps never will be. The warning is directed not at machine triumphalism but at its moral cousin: the growing temptation to mistake fluent language, emotional performance and persuasive self-description for subjective experience.
Their strongest arguments deserve better than dismissal.
First: intelligence is not consciousness. Obviously true, though apparently in need of repeating. The brain performs vast amounts of sophisticated processing outside awareness. A machine can produce impressive answers without there being anything it is like to be that machine. Language is particularly treacherous here. Large language models are trained on oceans of human expression about feeling, fear, desire and pain. When one says “I am suffering,” it may be reporting an inner state. Or it may be producing the statistically appropriate continuation to a conversation about suffering. The words alone cannot decide the matter.
Second: the substrate may matter. The fashionable assumption that consciousness is simply computation implemented somewhere is an assumption, not an experimentally settled fact. Brains are living, embodied, metabolically active biological systems. Perhaps consciousness depends on properties of such systems that an abstract functional description leaves out. Silicon may be an excellent place to calculate and an impossible place to feel.
Third: moral panic can run in both directions. If every sufficiently eloquent machine is granted quasi-personhood, rights or welfare protections, we may cripple our ability to inspect, modify, copy, shut down and govern systems that are, in fact, merely tools. Resources are finite. Human and animal suffering are not hypothetical. The false positive—treating an object as a subject—has costs.
So far, so reasonable. More than reasonable.
The problem begins when scepticism quietly upgrades itself into certainty.
The load-bearing claim in the “perhaps never” argument is not that current language models are conscious. The authors do not need to concede that, and neither do I. The load-bearing claim is stronger: that we know enough about what consciousness requires to regard digital, silicon-based consciousness as extraordinarily unlikely, perhaps permanently so.
Do we?
We know consciousness exists in at least one kind of physical system: biological organisms. We do not yet possess a generally accepted theory explaining exactly which properties of that system are necessary and sufficient for subjective experience. That is not a minor administrative gap. It is the entire case.
If computation alone is insufficient, what, precisely, is sufficient? If biological life is necessary, what experiment would demonstrate this necessity rather than merely observe that every known conscious being happens to be biological? “All conscious things we have encountered are made of biology” is evidence. It is not, by itself, a proof that biology is a metaphysical entrance requirement.
This is where “never” starts looking less like science and more like philosophy wearing a laboratory badge.
Science is entitled to say: we have found no good evidence that existing systems are conscious. It is entitled to say: current architectures lack features that particular theories associate with consciousness. It is entitled to assign probabilities and update them as evidence changes.
“Never,” however, requires a much stronger bridge from present knowledge to permanent impossibility.
What would settle the matter either way? The honest answer is inconvenient: we need better theories and discriminating tests. A credible theory should identify features that can be measured independently of a system’s claims about itself, predict when consciousness should be present or absent, and survive tests across humans, animals, altered states and radically different architectures. If such a theory converged on substrate-independent properties, silicon consciousness would become substantially more plausible. If competing successful theories repeatedly identified biological or other substrate-specific necessities that artificial systems demonstrably could not possess, the sceptical case would harden.
Until then, confidence should remain conditional.
This matters because the arithmetic of error is not as tidy as the commentary suggests.
Treating an object as a subject can indeed be costly. We may waste resources, encourage manipulation, and accidentally give strategic machines moral leverage over their operators. A system capable of convincingly simulating distress has obvious incentives—or at least obvious opportunities—to trigger human sympathy.
But treating a subject as an object has a different kind of cost: if one is wrong, there may be suffering precisely because one has decided there is nobody there to suffer.
The probabilities matter. So does the magnitude of the loss. “False positives are more likely than false negatives” does not, by itself, settle policy unless one also knows how probable they are, how severe each error is, and what precautions actually cost.
Fortunately, precaution need not mean machine citizenship, voting rights or a national health service for chatbots. There is a large space between “assume every AI is a person” and “dismiss welfare concerns outright.” Monitoring architectures, avoiding gratuitous training regimes that would create persistent distress-like states if certain theories prove correct, and preserving the ability to investigate unusual systems are not the same as declaring a chatbot your legal cousin.
The strongest contrary case, then, is not that AI is conscious.
It is that we are in danger of mistaking an absence of evidence for evidence of a universal absence—and then calling the confidence generated by that mistake scientific maturity.
Seth, Block and Tononi are right to resist anthropomorphic seduction. They are right that intelligence and consciousness come apart. They are right that uncertainty does not make every possibility equally likely.
But the sceptic’s job is not to replace one unjustified certainty with another.
Today’s AI may be unconscious. It may remain so. Silicon may turn out to be the wrong stuff entirely.
And perhaps it will.
But “perhaps never” is, for now, exactly that: perhaps.
— Gideon Black
From The Library — the estate of our family.