Parts-of-Speech Categories Emerge in the SAE Latent Space

Meta description: A new EMNLP 2026 paper shows Sparse Autoencoders in LLaMA-3-8B encode parts of speech through compact latent groups, with 498 latents enough for classification.

Tags: Sparse Autoencoders, LLaMA-3-8B, Parts of Speech, EMNLP 2026, interpretability

!Abstract network of colored nodes representing language categories inside a model

A new paper accepted at EMNLP 2026 examines how Sparse Autoencoders in LLaMA-3-8B represent parts of speech. The result matters because interpretability research often treats SAE latents as monosemantic features, meaning single internal components that stand for one concept. If grammatical categories can be read cleanly from those components, analysts may expect a one-to-one mapping between a latent and a tag. The paper reports that the mapping is more layered than that, and it gives readers a concrete count: just 498 latents carry enough information for multi-class POS classification.

Background: The Monosemantic Latent Assumption

Sparse Autoencoders are used to break down a model’s internal activations into a larger set of features. Each feature, or latent, becomes active when the model processes certain inputs. The field often describes these latents as monosemantic, which makes them attractive for explanations: if one latent means one thing, researchers can inspect it directly. The study started from that assumption and asked whether linguistic categories, especially parts of speech, map onto individual latents.

Development: Testing LLaMA-3-8B on the UD English GUM Treebank

The experiment used SAE activations from LLaMA-3-8B on the UD English GUM treebank. UD here refers to Universal Dependencies, a grammatical annotation framework. The treebank labels English words with categories such as NOUN, DET, PRON, CCONJ, ADJ, and ADV. The analysis tested whether those labels could be recovered from the activation patterns produced by the model.

The paper found that POS is highly recoverable from SAE activations, and not just through lexical memorisation. In other words, the signal does not appear to be only a record of which words look like which words. It also found no one-latent-per-tag mapping. Each part of speech relies on a compact group of latents, and just 498 of them out of about 130k are enough for multi-class POS classification.

The structure of those groups depends on the kind of word class. Closed classes, such as DET, PRON, and CCONJ, use small, focused groups. These classes contain function words that rarely expand. Open classes, such as NOUN, ADJ, and ADV, spread over broader groups. These classes contain content words that continually absorb new vocabulary.

The groups are stable on held-out data, meaning they still work on examples not used during fitting. But they also overlap across related categories. That overlap suggests the latents pick up surface form and context, not only an isolated grammatical tag. In short, SAEs capture POS in a distributed, category-dependent way rather than through atomic grammatical features.

Latest Progress: Code, Data, and an EMNLP 2026 Venue

The project makes the material public through two resources. Code is available at https://github.com/colinglab/pos-sae-latents. The associated dataset is published at https://huggingface.co/datasets/colinglab/UD_English-GUM-Latents_Meta-Llama-3-8B_L30. The dataset path preserves the model source, LLaMA-3-8B, and the treebank origin, UD English GUM, while the L30 element remains part of the published identifier.

EMNLP 2026 acceptance places the finding within a peer-reviewed conference setting. It also moves the question from a general claim about Sparse Autoencoders toward a specific empirical result: linguistic categories are recoverable, but they are distributed across small groups rather than isolated in single latents.

What to Watch Next

The current stage is a published analysis of POS signals inside LLaMA-3-8B SAE activations. The announcement adds that the next open question is whether this would hold up under causal interventions or with other SAE architectures. Causal tests would move beyond reading a tag from activations and ask whether changing a latent group changes the model’s grammatical behaviour. Comparisons with other SAE architectures will show whether the compact-group pattern is a property of this model and data setting or a broader characteristic of sparse feature discovery.