Paragraph Boundaries Are More Than Whitespace: Compression Depth Reveals Hierarchical Structure

Meta description: Hierarchical RoPE tests whether transformers encode paragraph structure beyond token distance, showing deeper attention compression across real sections.

Tags: hierarchical RoPE, transformer interpretability, positional embeddings, paragraph structure, LLM attention

Featured image: Abstract curved grid representing hierarchical attention boundaries, no people, no logos, no text.

An experiment built around hierarchical RoPE probes whether transformer models track document structure, or whether their attention is mainly shaped by token distance. The work is summarized in a blog post titled Your LLM Has a Curved Space of Paragraphs and released as code at https://github.com/ShuyangenFrance/hrope. Its result is not a simple yes or no: cross-paragraph attention compresses in every corpus examined, but the depth of that compression appears to distinguish meaningful paragraph structure from random labels.

Background: From token distance to document structure

Transformers need positional information because the model itself does not inherently know which token comes first. Rotary position embeddings, often abbreviated RoPE, are a common way to encode this order by rotating query and key vectors according to position. This makes relative distance a natural part of attention calculations.

For short sequences, token distance is often enough. A paragraph is a higher-level unit: two sentences may be far apart in raw tokens but still belong to the same idea, while adjacent tokens can cross a section boundary. The question behind the project is whether a model can reflect that hierarchy, or whether it only sees a flat line of tokens.

Development process: How hierarchical RoPE separates the axes

The project uses hierarchical RoPE with three separate channels: paragraph, sentence, and token. Instead of treating position as a single one-dimensional coordinate, it gives the model multiple axes for structure. The experiment then intervenes on the paragraph coordinate while holding the token sequence fixed.

This design isolates the effect of paragraph-level placement. If attention patterns change when the paragraph coordinate changes, but the tokens and their local order remain the same, the model is showing sensitivity to more than raw distance. The setup also compares genuine paragraph labels with a control channel using random labels.

Latest progress: Compression depth as the signal

The key finding is that cross-paragraph attention is compressed across all corpora examined. That compression alone is not enough to prove that a transformer understands paragraphs, because a control channel with random labels also produces compression. In other words, the model's attention can shrink across boundaries even when those boundaries are not semantically meaningful.

What separates real structure from noise is the depth of the compression. Genuine paragraphs show a deeper compression effect, and the strength varies by corpus. This suggests that paragraph structure can leave a measurable trace in attention, but the trace is not uniform and may depend on the text being analyzed.

What to watch next: Reproducibility and corpus dependence

The released code gives readers a way to inspect the method directly, and the project invites discussion. Future checks can test more corpora, longer documents, and different model setups to see whether the compression-depth signal remains stable.

For readers, the value is a sharper interpretability question: not only how far tokens are apart, but whether a model's geometry bends around the units humans use to organize text. The result is a measured middle ground, where transformers show structure-sensitive attention patterns without proving full semantic understanding.