Goodfire researcher argues AI neurons may be easier to interpret

A shared feature, not a tangled neuron

Cammarata says researchers have spent years treating individual neurons’ responses to stimuli as almost inherently difficult to understand. A neuron that activates for both a cat and a car seemed to support the idea that models cram unrelated concepts into the same unit.

The standard explanation is superposition. Models have a limited number of neurons, so, according to this account, a neuron may represent unrelated features that do not interfere with one another. That can make the network look like a knot of concepts, with many nearly equivalent ways for those concepts to be grouped.

Cammarata’s alternative is more literal: the neuron in his example does not represent cats and cars, but V-shaped forms. What looks like a jumble may instead be a shared visual feature.

What the example does not settle

Neil Nanda, a Google DeepMind researcher who leads the company’s work on mechanistic interpretability, called the research interesting but said it did not substantially change his overall view. He still considers superposition real and says the new work does not resolve his main doubts.

There is also a gap between identifying what an individual neuron responds to and predicting how neurons work together. A clear account of one unit’s function may be a useful technical result without solving the larger problem of understanding a model.

I think that distinction is the most important one here. A neuron that tracks a recognizable feature is a more encouraging object of study than a seeming tangle, but it does not by itself tell us how the model behaves as a system. And model safety depends on more than technical controls; legal and social mechanisms matter too.

The response to Cammarata’s work suggests it could influence how researchers think about AI internals. But until its conclusions are tested and peer-reviewed, the cleaner picture remains a hypothesis, not a settled account.