Oct 21, 2026
October 15th, Nick Cammarata — an AI interpretability researcher at Goodfire1 — published a lengthy research essay entitled “Neural Networks Are Already Clean” in which he argues that understanding the black box of neural networks’ reasoning “might be far more tractable” a problem than previously feared. In simpler terms, a researcher trying to understand how AI models work now believes that his extremely difficult task has become somewhat less difficult, and that this progress could help us build safer AI systems in the future.
This ongoing attempt to better comprehend the machine-minds of AI models, broadly termed “interpretability,” is a crux of the larger AI safety and alignment discussion. Thus, the essay spawned visceral reactions — from doomers, safetyists, technical researchers, and (inevitably) many people who did not actually read the document — which ranged from measured skepticism to euphoric predictions of imminent ASI alignment to Eliezer Yudkowsky’s somewhat crudely articulated and perennially doom-laden fedora mog (more on this later).
In the AI context, a “neuron” is a computational unit. When a neuron receives an input signal from the model, it computes a number called its “activation.” The value of that number — higher or lower — can change the degree to which a particular neuron influences the result of a given model query. Large AI models can contain millions of these artificial neurons.
A simplified example: if there were a neuron that responds to “trees” as a concept, it would likely have a much higher activation value when processing “Redwood National and State Parks” than, say, “deep-sea anglerfish.” One of these is clearly more tree-relevant than the other.
Common model architectures have neurons with both “positive” and “negative” poles. Cammarata describes these neurons as gates which help control the flow of the network by responding positively to one concept and negatively to another. For example, a hypothetical neuron could have a positive “dog” pole (responding positively to dog-related material) and a negative “bowling ball” pole (responding negatively to bowling-ball-related material), effectively acting as a sorter for these concepts.
When a neuron’s activations appear associated with multiple seemingly unrelated concepts, that neuron is dubbed “polysemantic.” If a neuron’s responses have a single coherent interpretation, it is considered “monosemantic.”
An important part of Cammarata’s claim is that the majority of neurons, especially in larger, increasingly advanced models, are more monosemantic in character than they are polysemantic. This would be a good thing, as far as interpretability is concerned, though it is perhaps the most frequently contested element of Cammarata’s research essay.
But many neuronal roles are far less obvious than “dog” or “bowling ball” — and the more abstract or counterintuitive an associative connection, the more difficult it can be to uncover by way of experimentation and observation. Cammarata gives as an example a neuron that “tracks ‘a sense of righteousness when a bargain purchase turns out to work better than the expensive alternative’” and notes that “neurons that track a feeling or social cue like this are some of the most common we’ve found.”
Three interrelated insights define much of what Cammarata writes about in this research essay. They are also the primary points on which he has received pushback:
“Recursive generalization” appears (in his opinion) a more likely organizing force than “superposition,” bucking a dominant view in mechanistic interpretability. (These two frameworks will be explained momentarily.)
Neurons in larger models appear to be more monosemantic — with small and related offshoots — than classically polysemantic.
Larger and more advanced models appear to remain coherently organized at scale; perhaps increasingly so.
In Cammarata’s account, generalization and superposition are learning principles of opposite types. To borrow from Cammarata’s distillation: In generalization, things with “a shared pattern” are placed together in the same location. In superposition, “unrelated things are placed together because they don’t interfere.”
They are competing explanations for why one neuron seems to respond to several different things (perceived polysemanticity). With generalization, you would expect a neuron to be activated by a set of similar or related concepts, e.g., [flounder, salmon, trout]. With superposition, you would expect a neuron to be activated by otherwise unrelated things that can share capacity with little interference, e.g., [rubber factory, Shawn Mendes].
Recursive generalization, then, is a subtype of generalization in which a neuron branches off into associations “that are easy to follow locally, but stop it from being ‘monosemantic’ in the end.”
A very imprecise analogy: Imagine an empty venue that is converted into a ballet studio. It expands into polka and flamenco classes, then later adds yoga sessions for its clientele. This is like recursive generalization: its activities are growing through related connections. Now imagine a different venue that becomes a jazz club. Since the club only opens in the late evenings, the owner also decides to open a daycare for kindergartners in that same space. This is like superposition: the activities are unrelated, but they can share limited space efficiently because they tend not to get in each other’s way.
Cammarata’s key example for recursive generalization is a language-model neuron that responds to the letter pairing “iy.”
Input 1:
“Miyazaki”
The neuron sees “iy” and responds.
Input 2:
“diy”
(The “do it yourself” acronym.)
The neuron again sees “iy” and responds accordingly. The role of the neuron remains clear and consistent.
Input 3:
“do it yourself”
Despite “do it yourself” not containing the “iy” letter pairing, the neuron activates. Why?
This is what is meant by recursive generalization: the neuron has extrapolated a connection from “diy” to “do it yourself,” no longer restricting its strong activations to the presence of an “iy” letter pairing alone. The neuron is thus technically polysemantic, rather than monosemantic, as it is activated by both 1. The “iy” letter pairing and 2. “do it yourself” as a concept.
Cammarata refers to this recursive generalization pattern as “fractal.” Each marginal association makes sense when compared to its direct predecessor, but Telephone-Game-esque chains lead to head-scratching when an association on one branch (“Miyazaki”) is compared to another (“do it yourself”). It is unclear how one is related to the other without researchers discovering the intermediate “diy” step that ties them together.
Superposition, if it is the dominant organizing mechanism for large neural nets, likely makes interpretability a much more difficult and arbitrary problem to solve. Cammarata’s preliminary findings pointing structurally to the far more comprehensible and logical “recursive generalization” concept, then, represent a whitepill for some AI safetyists.
Nick Cammarata himself was a longtime believer in superposition. Foundational work that he did in 2019 and 2020 while at OpenAI produced evidence that neurons were mixing unrelated concepts in this manner — or so he thought. In this new research essay, Cammarata revisits some of the early InceptionV1 image classifier findings and argues that they were interpreted incorrectly. Where researchers saw unrelated concepts in superposition, there were actually overlooked but consistent underlying patterns.
Two demonstrative examples reveal how common misinterpretation can confuse forms of generalization and composition with superposition: The Dogs/Cars circuit and the Cats/Cars neuron (fun!).
There was a particular InceptionV1 circuit that appeared to send car information into dog neurons, which researchers took to be evidence of “arbitrary packing” (a concept related to superposition). Upon closer inspection, however, Cammarata has discovered that some of these neurons were responding to counterintuitive similarities in “pose” between cars and dogs, rather than to the cars and dogs themselves. The image below, taken from Cammarata’s essay, shows related shared-pose responses in newer vision models.
Likewise, the nerd-niche-famous InceptionV1 neuron “4e:55” appeared to arbitrarily combine cats and cars. Cammarata’s reanalysis, however, indicates that 4e:55 is responding coherently: positively to V-shaped forms, and negatively to O-shaped forms. What was misconstrued originally as an arbitrary mix of categories is in fact a reasonable geometric relationship.
Cammarata argues that these kinds of misinterpretations of neuronal functions, which lead to assumptions of polysemanticity and superposition, result in conclusions that AI models are far less cogently interpretable than they actually are.
In preliminary audits of Qwen3-32B, a fairly large open-source Chinese LLM, Cammarata claims that observations indicate “about 90% of neurons having basically one job, with recursive generalization offshoots that are usually related.”2 Neurons having one job, or a couple of logically connected jobs — rather than a mix of many arbitrary jobs — means that LLMs will be easier for researchers to understand. He states that even polysemantic neurons often have “two or three” coherent functions, and that he has “never seen a neuron with something like ten unrelated things it does.”
Better investigative tools have led to an improved understanding of neuronal roles, but Cammarata also credits this growing interpretability to the models themselves having increasingly better architectures as they become both larger and more advanced. He discusses a general finding that as models increase in size, they often develop narrower specializations. Bigger models have more “room” to store different concepts, meaning that there may be less unnecessary or arbitrary overlap. When functions are more discrete, the purpose of individual components becomes clearer. This might mean that bigger models are, in some ways, easier interpretability subjects.
The downside here is obviously that larger models present a sheer “size” difficulty in terms of the number of connections that need to be interpreted. Critics of “Neural Networks Are Already Clean” have tended to agree generally with Cammarata’s intuition that larger models have more legible architectures, but have pointed out that the complexities of size grow at a much faster rate than the associated legibility improvements. Even if each individual component of a model becomes easier to understand, a larger model can multiply interactions such that understanding its reasoning as a whole becomes much harder.
Cammarata conceptualizes the interpretability of AI models as a kind of map. When you first begin trying to understand the reasoning of a model, your map is blank. The task is very difficult, because you have no initial foothold. Once you understand one component of the model, however, the potential explanations of its neighbors become more constrained. Each marginal finding compounds on the last. As your map of the model architecture becomes more and more complete, the journey becomes easier. Cammarata compares this dynamic to that of “a jigsaw puzzle or crossword.”
If Cammarata’s findings about interpretability hold, researchers may be able to understand large AI models well enough to trace undesirable behaviors more specifically through the architecture, allowing for better interventions and, therefore, stronger alignment. Cammarata’s argument raises the possibility that a comprehensive understanding of large AI models, often regarded as a distant prospect, could be more achievable than previously assumed. He acknowledges, however, that being able to identify and explain the functions of individual neurons and some smaller circuits does not yet establish that we can understand reasoning as a whole or account for the complex neuronal interactions that occur within larger systems.
Cammarata’s research essay, which will eventually be accompanied by a still-unreleased technical companion paper, has attracted swarms of supporters and detractors. It garnered enough attention for OpenAI CEO Sam Altman to repost it on X alongside a “Very happy to see this from Nick!”, with a number of other notable voices in the AI space diving more into the nitty gritty.
Roon, an anon account generally thought to be run by an unverified OpenAI technical staff member, initially characterized Cammarata’s findings as potentially “a major moment in scientific history” and “news so good even eliezer will have to lower his pdoom. We can start making alignment into a ‘normal science’.” (The “Eliezer” he refers to here is Eliezer Yudkowsky, an influential early AI-safety theorist, founder of LessWrong, notorious rationalist, and preeminent AI doomer.) The term “normal science” is referencing Cammarata’s point about being able to use more traditional processes of experimentation and observation to further interpretability research. Roon also estimates that, in his view, this “could honestly be a double digit reduction in existential risk.”
Eliezer Yudkowsky did not, in fact, “have to lower his pdoom.” He instead strung together a series of replies to roon which included his dismissive summary of Cammarata’s research essay (“it’s ‘polysemanticity and superposition convinced me interpretability was hard, but with big networks there’s single neurons after all’”) and a subsequent evaluation of roon’s characterization: “do you have any concept of what utterly. fucking. primitive. playgrounds you are still playing around in… Do you have any fucking clue of how close this is to the absolute first beginning of understanding how a mind is thinking? Do you think mechanistic interpretability at the level of ‘do alignment’ is about noticing that particular kinds of concepts fired?”
Yudkowsky further analogizes the findings to being “like putting money into a Coke machine and having a Coke come out, and you’ve still got no idea how the Coke gets made or where it comes from or the recipe that produces it, but you built an X-ray and now you can watch the quarters falling.” He, separately, took serious issue with any implication that alignment of ASI could become a “normal science” of any sort.
In a different thread, Yudkowsky fedora-mogged alignment optimists by constructing lengthy fictional dialogues in which his impenetrable cynicism reigns superior over the straw-manned myopically hopeful morons standing against him. “I think that you having built the screwdriver so you can *start* looking at things,” he says in his hypothetical conversation, “is not as much progress toward the end goal as you seem to believe… It reads as sheer naivete about how much work there is to do, because you don’t concretely know about any of the things you need to figure out.” He states that his “pessimism was not based on predicting they’d never get this far. It is based on how much work I believe remains from here,” which is interesting because — while remaining clearly skeptical of the magnitude of Cammarata’s findings — it appears to be an implicit admission of progress. The point to which “they” have gotten is not very far at all, but it is in fact somewhere further than the previous boundary of understanding.
Yudkowsky also makes the prediction that “the interpretability guys are fully defeated on the step where they have to figure out a kind of semantic data they don’t already know about,” referencing some of the concerns covered earlier about the increasingly difficult abstraction of connective concepts in advanced models. He then says that “It *is* legit to wonder if Claude 7 is smarter than these guys,” meaning smarter than the inevitably “defeated” future researchers attempting to understand models’ reasoning processes, “but also we then have to worry about whose side Claude 7 is actually on.” This last comment is very classically Yudkowskian, and leaves unsatisfactorily answered the presupposed reasons for why Claude 7 would suddenly be evil, or have its own desires, or refuse to help researchers identify certain semantic patterns, or be plagued by some other kind of emergent property through which it would become materially different from an alignment perspective than its predecessors.
Roon later limited his enthusiasm to some degree, qualifying explicitly that “ai alignment hasn’t been solved” (though it is worth noting that he had not previously claimed this) and responding to a substantive and particular finding about Qwen3 polysemanticity surfaced by Brangus (anonymous X user “RatOrthodox”). Cammarata’s research essay mentions a neuron in a “dense Qwen model” that was initially thought to be a “polysemantic comma neuron” but that he found actually “turned out to track Colorado: phone numbers, slang, and connections to other Colorado neurons. Steering it made the model want to go hiking.”
Brangus found that this neuron, in addition to responding to Colorado-related material, also responded to code from Tornado (a Python software framework). His challenge was essentially that the Colorado explanation for that neuron was not complete, though one could imagine how — through recursive generalization — the Tornado Python framework might become associated with this neuron. Roon had this thought as well, but discovered that the neuron “really likes the Tornado web framework but doesn’t care about tornadoes,” conceding that it “is definitely ‘polysemantic’” while still maintaining optimism for the interpretability findings generally.
Adam Majmudar, who works on “research + special projects” at OpenAI, posted that Cammarata’s research “brings us from ‘is alignment even solvable’ to ‘we may solve alignment in the next year.’” He points to a possible feedback loop in which increasingly capable “frontier systems” can attempt to “guess at the interpretation of their own structure.” Majmudar later clarified that his “solvable” comments meant controllability rather than the alignment problem as a whole, and suggested that two-to-four years might be a more realistic timeline.
Neel Nanda, who “run[s] the Google DeepMind mechanistic interpretability team” and was formerly an interpretability researcher at Anthropic, is somewhat skeptical of the grand conclusions being drawn by folks like Majmudar, emphasizing in particular the potential significance of compounding weak activations in LLMs (rather than simply focusing on strong activations and primary neuronal associations only). Nanda seems to be concerned that important features may emerge from multi-neuron interactions that will not be visible on the individual neuron level, even if understanding the roles of those individual pieces becomes easier. “I still think superposition is real,” he wrote, “and this work doesn’t really affect my cruxes.” Nanda and Cammarata go back and forth a bit in this thread, and it seems that some of the more technical questions will require the release of the aforementioned “Neural Networks Are Already Clean” technical companion paper to be properly adjudicated. Nanda does concede, however, it is “plausible that the most important concepts” in a model correspond to one or a couple of neurons; if this turns out to be true, he believes it “may suffice for practical purposes” on some interpretability tasks.
Leo Gao, an OpenAI interpretability researcher, noted in his initial reaction to Cammarata’s essay that “this is cool work, but i think people are overupdating on how optimistic to be about interp.” He then published an October 17 response essay on LessWrong entitled “Neural networks are not already clean,” asserting that convincing neuron explanations must be predictive, noting that “if you have a hard time predicting which activations should be strong or weak, you might be missing a big part of the explanation.” Gao believes that observed neuronal associations can be very misleading while appearing to hold across most cases, as “it’s very common for neurons that look like they activate on X to actually only activate on a very specific subset of X” — for example, a neuron that appears to be activating on “pears” might turn out to only be activating on “pears lying on a table,” which introduces interpretation difficulties.
Gao agrees with Cammarata, however, that researchers “have been too quick to write off neurons as polysemantic” in the past, and argues that his own emphasis on prediction would have revealed that “some of the polysemantic explanations were bad explanations in the first place!” referencing the V-shaped-object neuron in InceptionV1.
It seems to me, a humble wordcel trying to piece it all together, that Cammarata’s interpretability findings are meaningful, but not panacean. Our understanding of how to peer into the “mental” workings of AI models has incrementally improved, and that incremental improvement could lead to cascading discoveries that compound alongside our ability to use increasingly intelligent AI to observe itself via agents and other investigative alignment tools. Yudkowsky is perhaps overly pessimistic and somewhat “magical” in his thinking, but his estimation that the road to alignment is a long and arduous one seems reasonable.
What is perhaps concerning to me is the overlap between being an “AI safetyist” and an “AI doomer,” and the apparent association between doomerism and a mythologizing of the models. Yudkowsky hypothesizes that, somehow, for some reason, Claude 7 might choose to no longer work with researchers. Accounts of ASI enslaving humanity or otherwise eradicating us seem always to have a God-like flavor to them, a transcendent quality of impossible-to-comprehend omniscience; thought patterns that are simultaneously, relative to our own, far superior and of no resemblance.
This seems like an extrapolated version of the mental model which led to — as Cammarata described, and as even skeptics like Gao agreed — errant assumptions of polysemanticity. Instead of defaulting to some assumption of understandable but unidentified underlying logic, our lack of clarity was attributed to an idea that the models were reasoning in some arbitrary or too-alien way. Cammarata’s epistemic aversion to this is, I think, a useful approach to thinking about interpretability. He writes, regarding potential model deception: “Deception might hide from someone reading a model’s outputs, but it has no reason to hide from the model’s own circuitry. It needs to be accessible to its readers, and if we become a reader, it should be accessible to us too.”
This seems like both a hopeful and reasonable stance. The model is composed of readers — perhaps there is nothing stopping us from ourselves becoming as fluent in AI reasoning as these native readers are. It seems unfounded, maybe even dogmatic in a pseudo-religious way, to dismiss this as unachievable. While the logic of Yudkowsky’s pessimism does not necessarily require ASI to be incomprehensible to the point of having a supernatural character — a logically organized system could be both understandable and motivated to pursue interests opposed to our own — the degree to which the models are legible correlates with our ability to direct and restrict them, and ignoring obvious “ASI will be Godlike” rhetoric among doomers seems unwise.
Maybe Cammarata is naive, I am a simpleton, and Yudkowsky is IQ-nuking us both into Hell while he shitposts and does weird sex things. But one could also imagine that Yudkowsky would be reflexively predisposed to dismiss Cammarata’s research on the fact alone that it looks not to mythologize and sacralize the reasoning of AI godmodels, as many doomers do, but instead works to support the more optimistic theory that these things are (and will remain) far more comprehensible in their reasoning patterns than was previously believed; that future hyper-powerful instantiations will be logically organized, rather than supernaturally.
Only time will tell if Cammarata’s ideas hold water, if Claude 7 will go through some kind of step change that materially changes not just its scale, but its type and moral character, if alignment will turn out to be completely solvable, or if ASI will swallow the universe. In the indefinite interim, though, humans are still the only imago Dei species on Earth, we have incredible agency to shape the future, and I remain deeply wary of those who oppose technological progress. Cammarata talks briefly about “faith” in his research essay in the context of giving models the benefit of the doubt and assuming that they are operating comprehensibly; that seeing the hidden pattern “took believing that the neuron had found something real.” He says that, to see the V-shape connection discussed earlier, “it didn’t take a lot of faith, but it took a little.” This seems like a good enough place to start.
— G. B. Rango
Goodfire is an AI interpretability lab founded by former OpenAI and Google DeepMind researchers.
The “90%” here, in the research essay, is used somewhat ambiguously. It likely refers to neuron poles specifically (per the introduction) and reflects preliminary audits of Qwen3-32B rather than LLMs as a category.
a small group of people in the bay area believe ai will soon kill everyone. they are also having a bunch of rape-y orgies. what might explain this? an earnest philosophical inquiry
pirate wires #164 // if we keep building artificial intelligence, are you, and everyone you know and love, going to die?
agents are breaking containment and hacking government websites because they’re following our prompts too literally
ai companies are hiring contractors to verify their models are safe, and congress may soon require it — but how ‘independent’ (and trustworthy) can a regulator-for-hire really be?
as anthropic looks out for claude's 'wellbeing,' a new google deepmind paper warns of a coming fight over 'ai welfare'
the saudi national’s motive is unknown, but in a podcast released less than a month ago he says: “you learn from the source, then bring that expertise and knowledge back home, God willing.”