LLMs Can Represent Two Ideas Simultaneously: Evidence for Linear Superposition in Transformers

Meta description: LLM superposition lets one forward pass decode two text streams after averaging embeddings, while fine-tuning restores a trait eroded by pretraining.

Tags: LLM, embeddings, superposition, inference, token generation

Featured image: Abstract technology style, two luminous embedding streams crossing into a single decoder path, no real people, no company logos, no text.

The described result gives a clear example of LLM superposition: an LLM can carry two meanings at once. If the embeddings of two unrelated texts are averaged, the next-token prediction can track both streams together. That matters because a normally single-output inference step becomes a shared carrier for two continuations, then those continuations can be separated again.

Background: Averaging Embeddings Creates Two Streams

Large language models represent text through embeddings, vectors that encode word meaning and context for each token. In ordinary use, an LLM takes one prompt, converts it into a sequence of embeddings, and predicts one next token at a time. The described result starts from a different setup: two unrelated texts, rather than one coherent prompt.

The key move is averaging the embeddings of those two texts. Averaging is not a new text with a new meaning; it is a mixed vector that still carries information from both sources. The observation is that the model’s prediction mechanism can read that mixed representation as two streams simultaneously. In other words, the next-token computation is not forced to choose one text’s trajectory. It can support two trajectories at once.

Development: Architecture, Pretraining, and Light Fine-Tuning

The result attributes the behavior to the architecture itself. It is not presented as a special feature added by a prompt, a tool, or an external decoder. The architecture can hold superposed content: several possible meanings or continuations coexisting inside the same internal representation. Superposition in neural networks refers to the ability to represent more concepts than the number of dimensions would seem to allow, by overlapping signals inside shared spaces.

The timeline of that ability has a clear order. First, the architecture supplies the capacity. Then pretraining erodes it. Pretraining teaches the model to produce coherent next-token predictions from ordinary language data, and that training pressure can reduce the usefulness of mixed, multi-stream states. The behavior becomes harder to read after this stage.

A light fine-tune restores it. The term “light” suggests a relatively small intervention: the described adjustment does not rely on rebuilding the model from scratch or applying a major new training procedure. It points to a behavior that the architecture already made possible.

Latest Progress: One Forward Pass Splits Into Two Continuations

The practical contribution is separation. If two streams can enter a single next-token process, the remaining question is whether they can be pulled apart before they blur into one output. The described method splits the mixed state back apart and generates two continuations in one forward pass.

A forward pass is one complete run of the model over an input to produce predictions. Normally, that pass yields one continuation path. Here, the same pass contains two possible paths. That gives the result an operational meaning: the model can be used as a shared computation step for two outputs, then read out as two distinct continuations.

The important number is still the one already present in the setup: two unrelated texts, two streams, and two continuations, but only one forward pass. The value is not that the model invents two prompts; it is that the architecture’s superposed state can be decoded back into two coherent paths.

Current Stage: A Demonstrated Separation, Not a Standard Workflow

The current stage is a demonstrated separation, not a standard workflow. The result ties the effect to the model architecture itself, notes that pretraining erodes the behavior, and shows that a light fine-tune restores it. It also demonstrates a way to separate the mixed representation again and decode two continuations within one forward pass.

What to watch next is whether the separation holds across different model families, longer generations, and harder text pairs. Because the effect is described as architectural, readers should track how much of it remains robust when the inputs are less unrelated, the continuations become longer, or the model is optimized for a single-output task. For now, the timeline stops at a clear observation: an averaged embedding state can carry two streams, and one pass can be made to split them again.