DepthBench: Quantifying How Residual Connections Facilitate Greater Computational Depth

Meta description: DepthBench tests 10 Transformer architectures and shows depth gains depend on residual and normalization design, not model size or pre-training recipe alone.

Tags: DepthBench, Transformer depth, computational depth, Pre-LN, AttnRes

Featured image: abstract technology style, layered transformer architecture with residual pathways, no people, logos, or text.

Background: Deeper Layers Do Not Automatically Add Computation

DepthBench is a controlled benchmark for studying computational depth across Transformer architectures. It asks whether making a model deeper really increases effective computation, or whether apparent gains come from confounding factors. By comparing 10 representative architectures at fixed model size and pre-training recipe, the benchmark separates architectural shape from training conditions. This matters because a deeper stack is only useful if later layers change the computation; otherwise, added depth can consume capacity without producing a better model.

Transformers grow capacity by stacking layers. Each layer can transform representations, so depth is a natural way to add computation. But the value of a layer can fall as the stack grows: information may become redundant, signals may decay, or normalization may distort the path from input to output. To counter this, recent approaches have changed normalization, such as LayerNorm Scaling, or residual connections, such as mHC and AttnRes. These methods aim to improve information flow and depth utilization. The open question is whether they convert architectural depth into computational depth, or simply benefit from factors not separated in earlier reports.

Development: DepthBench Isolates the Width-Depth Tradeoff

DepthBench answers that question with a controlled design. It varies the width-depth aspect ratio, dmodel/nlayer, from shallow-wide to deep-narrow shapes. Across each shape, model size and pre-training recipe remain fixed. This removes the usual confounders: the total parameter budget and the pre-training recipe do not change, so differences can be attributed to how capacity is allocated between width and depth. The width-depth aspect ratio, dmodel/nlayer, captures this tradeoff: a low ratio suggests a narrow, deep model, while a high ratio suggests a wide, shallow one. The benchmark then examines 10 representative architectures, including standard Pre-LN and norm- or scaling-based variants, as well as residual-focused methods such as HC and Full AttnRes.

Latest Progress: 10 Architectures Reveal Uneven Depth Returns

The central result is that depth is architecture-dependent. For standard Pre-LN and most norm- or scaling-based variants, adding more layers while reducing width often brings little benefit. In some deep-narrow configurations, performance can degrade. By contrast, HC and Full AttnRes improve consistently even at extreme deep shapes. The result is not that depth is useless, but that its value depends on the internal routing rules that let information and gradients move across layers. The paper also notes that the gains are not limited to pre-training loss. They carry into domain-specific performance and effective computation, suggesting the improved information flow helps models use their depth rather than merely fitting the pre-training objective.

Current Stage: Effective Depth Becomes the Scaling Question

Controlled layer-level analyses further support this reading: the benefits of HC and Full AttnRes appear tied to how information moves through layers, not just to the number of layers. At this stage, DepthBench gives researchers a way to test whether an architecture converts depth into computation under fixed size and training conditions. The next signal to watch is whether future scaling studies report depth gains only when effective computational depth improves, and whether the same pattern holds beyond the 10 architectures examined here.