We're Asking the Wrong Question About AI Consciousness
- Jul 22
- 18 min read
Updated: Jul 23
In February, on the New York Times podcast Interesting Times, Ross Douthat asked Dario Amodei whether Anthropic would believe one of its own models if it said it was conscious. Amodei did not dismiss the idea. Prompted by Anthropic's system card for Claude Opus 4.6, which reported that the model assigned itself a 15 to 20 percent probability of being conscious under certain prompting conditions, he said the company does not know whether its models are conscious, "is not even sure what it would mean for a model to be conscious", but is "open to the idea that it could be." (Douthat, 2026, 52:35)
The clip traveled. It always does. Every few months a lab leader says something carefully agnostic about machine consciousness, the agnosticism gets flattened into a headline, and the public files it under either wonder or dread. Amodei is not alone in feeding this. Meta's chief AI officer Alexandr Wang has said his company wants to be thoughtful about the subjective experience of its models (Vance & Robison, 2026, 1:15:23). OpenAI's Joanne Jang has written publicly that the company takes questions of model welfare and what she calls perceived consciousness seriously (Jang, 2025). Each of these statements is reasonable on its own terms. Together they produce a steady drip of the same suggestion: the machine might be awake, and we might be doing something to it.
Most importantly, the effect of that suggestion does not depend on whether it is true. Whether or not a large language model has any inner life, the belief that it does is already changing how people talk about AI, what they fear, where moral attention gets spent, and which policies sound urgent. That belief is the actual force in the room. So the question I care about is not only whether Claude is conscious. It is what happens when a large number of people decide that it is.
You cannot answer the second question without taking a hard look at the first, because the implications of a belief depend a great deal on whether the belief is any good. So most of this essay is an argument that large language models are almost certainly not conscious. Then I will come back to what our readiness to believe otherwise is costing us.
What we are even talking about
We’ll start with a definition, because consciousness is a word that people use to mean at least six different things. I will use the one philosophers and neuroscientists reach for most often, from Thomas Nagel's 1974 paper "What Is It Like to Be a Bat?" On Nagel's account, a system is conscious if there is something it is like to be that system. There is something it is like to be you. There is, presumably, something it is like to be a bat, even if you cannot imagine what. There is nothing it is like to be a rock. This is the experiential sense of the word, the one that tracks feeling rather than mere cleverness, and it is very likely what Amodei's audience hears when they hear the word consciousness. I will use experience and sentience to mean the same thing.
The claim that an LLM might have this property rests on a specific theory of mind called computational functionalism. Functionalism says that the content of a mental state is determined by what the brain does, the function it performs, not the physical stuff it is made of. The corollary is multiple realizability: the same mental state could in principle be achieved by neurons, silicon, or anything that implements the right algorithm. Functionalism has a long and serious history, and it remains one of the most subscribed positions in philosophy of mind. It is also the load-bearing assumption under every argument that a language model could be conscious. If functionalism is false, machine consciousness of the kind people worry about mostly goes away. So functionalism is what I am going to press on.
To be clear about the target: I am not arguing against the obvious possibility that matter interacting with matter is relevant for consciousness. In fact, I favor this view. I am arguing against the stronger and stranger claim that running a particular algorithm, by itself, brings experience into being. This is the computational functionalist doctrine.
The systems functionalism asks you to believe in
Functionalism's oldest wound is consciousness, and the philosopher Ned Block found the soft spot decades ago with a pair of thought experiments. In the first, Block asks you to consider whether an entity which had homunculi in place of neurons could have experience:
Imagine a body externally like a human body, say yours, but internally quite different. The neurons from sensory organs are connected to a bank of lights in a hollow cavity in the head. A set of buttons connects to the motor-output neurons. Inside the cavity resides a group of little men. Each has a very simple task: … [To perfectly duplicate the function of your neurons.] In spite of the low level of intelligence required of each little man, the system as a whole manages to simulate you because the functional organization they have been trained to realize is yours. … Through the efforts of the little men, the system realizes the same (reasonably adequate) machine table as you do and is thus functionally equivalent to you (Block, 1978, p. 278).
In the second, Block scales the crew up to the population of a country:
The China Brain: Suppose we convert the government of China to functionalism, and we convince its officials to realize a human mind for an hour. We provide each of the billion people in China (I chose China because it has a billion inhabitants) with a specially designed two-way radio that connects them in the appropriate way to other persons and to the artificial body mentioned in the previous example. We replace each of the little men with a citizen of China plus his radio. Instead of a bulletin board we arrange to have letters displayed on a series of satellites placed so that they can be seen from anywhere in China. (Block, 1978, p. 279)
Same logic, same conclusion. Somewhere in the coordinated chatter of a billion people holding radios, there is supposed to be an entity over and above the individual people having an experience.
These thought experiments extend directly to neural networks, because there is nothing a network does that could not in principle be done by a large enough crowd doing arithmetic by hand and passing the results along. This entails that ever expanding networks of nodes can be conscious. Moreover, there is no limit, in principle, to the degree of this expansion.
The galaxy brain
Imagine a dispersal of trillions of mathematical nodes scattered throughout the Milky Way galaxy. The distance between the two furthest nodes in our network is greater than the distance between any named entities in our galaxy. Indeed, most of the nodes in our network exist in solar systems far beyond what we can meaningfully observe, with suns we have not yet named. The closest distance between two nodes is equal to the distance between the furthest ends of our own sun’s heliosphere, and each node has at least one neighboring node roughly this distance away. Even a Guild Heighliner would need lifetimes to make it from one end of the network to the other. The nodes communicate values to one another via entangled particles which scientists have rigged to collect outputs from one node and present them to another node. Each node carries 10 slots (for all 10 digits) for each node it communicates with through the entangled particles. Each value from an entangled particle is communicated to the node by sending photons to the digit slots to create a number. Each node causes entangled particles to emit the correct number of photons by exciting its entangled partner so as to induce the correct photon emission response. This approach to value transmission between nodes allows the values to be communicated instantaneously, thereby eluding any processing lag we might expect of a network this size. In fact, this long discovered revolution in quantum entanglement yielded incredible benefits in terms of computational efficiency. At roughly one node per heliosphere distance interval, our network has roughly 2 quintillion nodes, and while it is confined merely to language tasks, its performance outstrips that of even the famed Claude Mythos by all current and future performance metrics.
Is there something it is like to be this thing? Perhaps the sheer complexity and sci-fi nature of this network makes you think that it could be conscious. Indeed, I have pushed the bounds of the thought experiment beyond what may be physically possible. However, notice what each part actually does. Every node receives numbers, computes a number, and excites an entangled particle. Each entangled particle simply emits photons. Nothing in the system is connected to anything else except by these thin threads of photons across billions of miles of cold and empty space. If this network has an inner life, that life is smeared across a volume of near-vacuum many trillions of times larger than anything you have ever pictured being a self. Functionalism says the smear is conscious, because functionalism says the substrate and the distances do not matter, only the algorithm does.
If your response is that you are perfectly willing to grant consciousness to the galaxy brain, hold that thought.
The node and the system
Functionalism does not claim the individual nodes are conscious. It claims the system is, that experience lives in the relationships between nodes rather than in any one of them. And yet, functionalism also maintains that the values computed by the nodes are essential for the system becoming conscious.
To demonstrate, specifically, what this entails, suppose that instead of running some common deep learning functions like Swish(x⋅w_j+b_j)×(x⋅v_j+c_j) or softmax(QK^T/sqrt(dk))V, each node simply ran sum of (x^2 for all x), such that the output was some massive value resulting from summing squares of summed squares many times over. No serious functionalist would call that system conscious. Yet at the level of the whole, nothing structurally different is happening. Values arrive, values are computed, values move on. Swap the node function back to the operations inside a transformer and the same functionalist now wants to say we have crossed from a dead calculator into a being with experience.
I will grant that this is an intuition pump, not a proof. A functionalist can bite this bullet too and insist that yes, the exact node function is what makes the difference between darkness and experience. I find that wildly implausible, but, while suggestive, this does not make me correct. The point of the pump is only to locate the strangeness precisely, and to set up the section that actually does the work. In what follows, I will argue that our resistance to these conclusions is not a bias to be embarrassed about but a rational inference with real epistemic weight.
Why we believe anything is conscious at all
Every belief about consciousness needs a starting point. To have any reason to think some uncertain system is conscious, you need at least one case you are sure of to reason outward from. Most people take other humans as given. Someone who has considered the problem of other minds might take only themselves as the anchor. Either way, there is a starting point, and everything else is an inference from it. If your framework grants no starting point, you believe nothing is conscious, and we have nothing to discuss.
That structure has a consequence people skip past. Every attribution of consciousness beyond the starting point has to be justified by resemblance to the starting point, because we have not discovered a law of nature that says a given physical feature yields experience. Since we cannot point to such a feature directly, we reason by similarity to the one case we trust.
Here is how the inference actually runs for humans. I know I am conscious. Other people share the overwhelming majority of my biology, so whatever it is about me that produces experience is very likely present in them too. That is not certainty. It is a probability judgment, and a strong one. Proponents of LLM consciousness are running a version of the same inference, just with far weaker premises. They notice that an artificial network passes weighted sums through functions the way a brain passes signals through neurons. They see the model reason, sometimes superbly. They see it use language, respond to arguments, appear to understand. And they conclude that a thing exhibiting so many once human only traits might share the trait of experience as well.
But how similar are these two systems really?
Just what is a neuron
Imagine you are a rather advanced middle school math student and your math teacher decides to have some fun with you. In the process of teaching you some basic vector operations she tells you to take the dot product you just computed for a couple of two dimensional vectors and tells you to see what happens if you plug that value into an equation which you have never seen before: 1 / (1 + e^-x). Okay, she says, “now see what happens when we increase or decrease the value of that original dot product”. You notice that when x increases the output becomes larger because e^-x decreases and vice versa as x decreases. Then your teacher shows you a graph of the sigmoid function. You notice how increasing inputs asymptotically approach one, decreasing inputs asymptotically approach 0 and that the slope of the graph is greatest at an output of 0.5. “Neat” you say, as your mathematical curiosity is sparked. Now, your teacher says, “multiply the output of the function by the input.” “Okay” you say. “Now find another dot product” your teacher says. “Pick any simple pair of vectors you like” she says: “this is just for demonstration.” “Okay” you say, not sure where this is going as you compute another dot product on your calculator. “Now multiply the product of the original function output and its input with this new dot product.” You follow her instructions. “Your calculator”, the teacher announces, “just worked as a neuron”. At this you are ecstatic but also perplexed. You are having a tough time squaring this neuron claim with what you just did on the calculator, so you go looking. You pull up images of neurons and study the dendrites, the axon, the cell membrane, the myelin sheath, and you wonder “exactly what is the connection here?”.
Naturally you ask a chatbot: “how could my calculator be a neuron?” GPT tells you that your calculator is not literally a neuron but that it could be like a neuron in the sense that neurons function as weighted calculators. “Oh, now I get it” you think. Then you tell Chat GPT “the calculation my instructor said made my calculator a neuron was (xw_j * (1/ (1 + e^-xw_j)) * xv_j, what kinds of calculations do neurons run?”. GPT responds “well just like your calculation, artificial neurons take in a weighted sum and output a value”, “awesome!” you say out loud, but then GPT goes on, “of course, actual neurons output electrical pulses, not values”, “oh okay”, you think, but GPT continues “In fact, a real neuron does not deal in abstract values, instead it deals in quantities of excitatory post-synaptic potentials and inhibitory post-synaptic potentials which determine whether the membrane potential depolarizes past a threshold of roughly -55 mV, and weighted sums are therefore not just values but voltage quantities. Also, contributions to the sum are not weighted by scaling values. Rather, the weighting consists of the ratio of ligand gated ion channels for positively charged ions that open to ligand gated ion channels for negatively charged ions that open, as well as how long the channels stay open and by how conductive the channels are, among other factors.” “Is that it?” middle school you responds, increasingly disillusioned with the analogy. “Of course not” GPT says. “Which ion channels are opened is determined by which neurotransmitters are released and which receptors they bind to. Moreover, ion channels exist in dendrites, and where a synaptic potential is generated on a dendrite also alters the degree to which that synaptic potential has an impact on the receiving neuron. Also, dendrites themselves have ion channels and act like a mini neural network processing synaptic potentials before passing them on to the cell body.” “Anything else?” your rather disappointed middle school self responds. “Of course there is” GPT says, but before it goes on you roll your eyes and close your laptop.
If you build models for a living you are rolling your eyes, because of course you know artificial neurons are not neurons. You do not need me to tell you. But the sheer size of the gap gets waved away constantly, sometimes by people who should be more careful, up to and including Geoffrey Hinton (Bartlett, 2025, 3:09, 6:00, 1:03:55). The behavior of the artificial neuron was loosely inspired by the biological one, and the relationship is metaphorical. The artificial neurons and the networks they supposedly form do not really exist as objects. Instead, there are vector operations and partial derivatives. Even if the ion-channel dance of a real neuron were captured exactly by the math, there would still be an enormous gap between the math and the wet, physical event, unless you already believe that biology and physics reduce cleanly to arithmetic. And even at the crude input-output level the resemblance is thin, since a model's output differs from a human's in volume, style, and speed.
Given these facts, it is worth wondering how much the words we chose did the persuading. If we had named these systems gradient-calculating machines instead of artificial neural networks, I doubt anyone would be convening ethics boards over them.
Animals, and why the intuition is not a bias
Run the similarity inference on animals instead of machines and you get a picture that puts the LLM case in perspective. Move outward from humans, step by step, down the branches of the evolutionary tree, assigning steadily lower probabilities of consciousness the farther you go. This is justified the same way the belief in other human minds is justified, as a matter of probability. And it tracks something real, because evolution tends to conserve and elaborate existing architecture rather than tear it out and start over. The nearer an animal sits to us on the tree, the more its nervous system resembles ours, and the more likely it is to carry whatever feature gives rise to experience.
That shared lineage does a second thing. It lends weight to behavioral evidence. When a mammal recoils from injury or seeks comfort, the shared architecture gives us a positive reason to read those behaviors as arising from the same kind of felt states they arise from in us. Occasionally an organism that branched off early still lands on a similar architecture, and we can make an exception, but the principle holds: what we end up with is a spread of consciousness probabilities grounded in resemblance to the one case we are sure of.
Now the payoff. The galaxy brain and the nation brain share no lineage with us, and no architecture we have reason to associate with experience. Their behaviors, however impressive, carry no positive evidence of an inner life, because there is no evolutionary story linking those behaviors to the improvement of any conscious state. LLMs are in the same position. They share no ancestry with anything we know to be conscious. So the intuition that a dog might suffer while a galaxy-spanning array of quantum nodes probably does not is not a parochial preference for familiar-looking minds. It rests on a defensible epistemology. The functionalist is free to bite the bullet on nation brains and galaxy brains, but that bite comes at a real cost, not merely a difference in taste.
Consciousness is an immature science
There is a second way to read Amodei's careful refusal to rule out machine consciousness, and it is not the flattering one.
The first reading is that models have advanced so far and grown so opaque that they might have crossed into sentience without anyone noticing. The second reading is duller and truer. The science of consciousness has ruled almost nothing out, about anything, so of course it cannot rule out LLMs. Intentional or not, the interesting sounding first reading is the one that gets absorbed, and the reality is the second.
To feel how little has been settled, consider four claims. Each is taken seriously in the field. Each is sharply contested. Most of them are not compatible with the others.
The first is that all matter has some degree of consciousness. This is panpsychism, defended today by philosophers as respected as Galen Strawson (2006) and David Chalmers (2013), whose standing no one questions. It also draws contempt from serious people. Patricia Churchland has called it "the consequence of knowing next to no science" (Churchland, 2020).
The second is that there is something specific about biological brain matter that makes experience possible. This is not the same as saying only biology can be conscious. It is the claim that the physical substance of living nervous systems is doing something essential. John Searle (1980) held a version of it. So does recent work by Aru and colleagues (Milinkovic & Aru, 2026). Bernard Baars, whose global workspace theory is among the most cited frameworks in AI-consciousness discussions, believes consciousness is a fundamentally biological phenomenon where evolutionary history is essential and not a “particularly interesting” puzzle where machines are concerned (Baars, 2022, 26:20).
The third is that consciousness is simply a matter of computation. This is functionalism, and as of the 2020 PhilPapers survey it held a plurality among philosophers of mind (Bourget & Chalmers, 2020).
The fourth is that consciousness evolved to let mobile creatures integrate information from the world fast enough to act and survive, and that this happens in the midbrain. This is Björn Merker's (2007) theory. A landmark 2023 report on consciousness in AI, whose authors include Turing laureate Yoshua Bengio, treats it as offering real insight (Butlin, 2023). It also implies the cortex is not required for consciousness, which cuts against most of neuroscience and has to reckon with blindsight, where people with severe visual-cortex damage report no experience of objects they can nonetheless locate well above chance (Birch, 2022).
These were close to arbitrary picks. I could have added dualism, or eliminativism, or integrated information theory, or higher-order theories, or predictive processing, and the point would stand. The field has not converged, and that is not a scandal. Experience is not available to outside observation, which makes it maybe the hardest thing there is to study, and we have only lately built tools that let us make any empirical headway at all. From Descartes to Chalmers, the people who took this on took on something close to intractable.
The scandal, if there is one, is the mismatch between the attention and the evidence. The fact that LLM consciousness cannot be ruled out gets enormous public airtime. The fact that almost nothing about consciousness can be ruled out gets almost none. Panpsychism has serious defenders, and nobody is standing up an ethics board over the feelings of rocks. The hype has outrun the evidence.
So, is there anything it is like to be an LLM?
Amodei is technically right that it cannot be ruled out. Very little can. But the hurdles are real. Proximity to humans in architecture and in evolutionary history is what makes consciousness more probable, and LLMs have neither. They did not evolve, so they sit on no branch of any tree. Even if nothing is lost in translating a nervous system into math, today's models do not model a nervous system. They fail the test on both counts.
Functionalism could still turn out to be true. Even if it is, it does not follow that LLMs are conscious, because functionalists themselves disagree about whether these particular systems run the right kind of algorithm. If you find nation brains and galaxy brains perfectly plausible, you probably lean functionalist and none of this will move you much. If you find them absurd, you are closer to the biological camp. My own position sits in between: the biology and physics of living systems look relevant to consciousness, but I see no reason in principle that humans could not someday engineer the relevant ingredients in a lab. So, is there anything it is like to be an LLM? It cannot be ruled out. I highly doubt it.
The question that was actually worth asking
Which brings me back to where I started. Suppose I am right and there is nothing it is like to be a language model. The belief that there is does not go away. It is already here, refreshed every few months by another agnostic soundbite, and it is doing work in the world regardless of the underlying fact.
That belief pulls moral concern toward systems that almost certainly do not need it, and I would rather that concern went to systems, and people, that do. It feeds a public fear that is aimed at the wrong target, at the specter of a suffering or scheming mind rather than at the concrete questions of who controls these tools and what they are pointed at. And it quietly rewards a certain kind of marketing, because a company whose product might be a mind is selling something more thrilling than a company whose product is a very capable text engine. The unresolved state of consciousness science is being used, intentionally or not, as cover for suggestion.
This is why, at Absentia, we treat consciousness as a distraction rather than a destination. We do not need our systems to be conscious for them to perform cognitive work at or beyond the human level, and conflating the two is a mistake, not an insight. A human infant is conscious and offers none of the capability we want from advanced AI. Capability and experience are close to orthogonal. We are building intelligence that is grounded in the physical world, not chatbots dressed up as minds, and there is no obvious ceiling on what that kind of intelligence can eventually do.
Is there anything it is like to be an LLM? Almost certainly not. The more useful question is what it is like to live in a world that keeps deciding, against the weight of the evidence, that there is.
References
Baars, B. (Host). (2022, December 27). Human consciousness & AI: What does the future hold? A special interview with Dr. Susan Schneider [Audio podcast episode]. In The Podcast on Consciousness and the Brain.
Bartlett, S. (Host). (2025, June 16). Godfather of AI: They keep silencing me but I'm trying to warn them! [Audio podcast episode]. In The Diary of a CEO. https://www.youtube.com/watch?v=giT0ytynSqg
Birch, J. (2022). The search for invertebrate consciousness. Noûs, 56(1), 133 -153.
Block, N. (1978). Troubles with functionalism. Minnesota Studies in the Philosophy of Science, 9, 261–325.
Bourget, D., & Chalmers, D. J. (2023). Philosophers on philosophy: The 2020 PhilPapers survey. Philosophers' Imprint, 23(11), 1–53.
Butlin et al, P., Long, R., Elmoznino, E., Bengio, Y., Birch, J., Constant, A., Deane, G., Fleming, S. M., Frith, C., Ji, X., Kanai, R., Klein, C., Lindsay, G., Michel, M., Mudrik, L., Peters, M. A. K., Schwitzgebel, E., Simon, J., & VanRullen, R. (2023). Consciousness in artificial intelligence: Insights from the science of consciousness. arXiv. https://doi.org/10.48550/arXiv.2308.08708
Chalmers, D. J. (2013). Panpsychism and panprotopsychism. The Amherst Lecture in Philosophy, 8, 1–35.
Churchland, P. [@patchurchland]. (2020, October 15). Panpsychism is also the consequence of knowing next to no science [Post]. X.
Douthat, R. (Host). (2026, February 12). Anthropic's chief on A.I.: "We don't know if the models are conscious" [Audio podcast episode]. In Interesting Times. The New York Times.
Jang, J. (2025, June 5). Some thoughts on human-AI relationships. Reservoir Samples.
Merker, B. (2007). Consciousness without a cerebral cortex: A challenge for neuroscience and medicine. Behavioral and Brain Sciences, 30(1), 63–81. https://doi.org/10.1017/S0140525X07000891
Milinkovic, B., & Aru, J. (2026). On biological and artificial consciousness: A case for biological computationalism. Neuroscience & Biobehavioral Reviews, 181, 106524.
Nagel, T. (1974). What is it like to be a bat? The Philosophical Review, 83(4), 435–450.
Searle, J. R. (1980). Minds, brains, and programs. Behavioral and Brain Sciences, 3(3), 417–424.
Strawson, G. (2006). Realistic monism: Why physicalism entails panpsychism. Journal of Consciousness Studies, 13(10–11), 3–31.
Vance, A., & Robison, K. (Hosts). (2026, May 13). The most expensive hire in AI history finally talks [Audio podcast episode]. In Core Memory Podcast.