Continuous Depth Batching for Depth-Adaptive Inference in Looped Language Models

Meta description: CDB enables first efficient depth adaptive batching for looped LLMs, letting tokens with different loop counts share compute between loop steps.

Tags: Looped LLMs, Inference Batching, CDB, vLLM, Dynamic Depth

Featured image: Abstract technology image: flowing neural network layers with adaptive loop paths, no people, logos, or text.

Background: Fixed Compute Shapes LLM Serving Economics

Large language model serving faces a cost problem: the same forward pass and the same compute are used for every single token, even though tokens differ in difficulty. A paper introduces CDB as the first efficient method for depth-adaptive looped LLMs, and the core idea is depth-adaptive batching between loop steps.

A forward pass is the single execution of the model that turns current context into the next token distribution. Batching groups many tokens or requests into one execution, which improves throughput on modern accelerators. The cost is that the serving system treats all tokens as if they need the same amount of work.

That assumption becomes awkward when model design starts to vary the depth of computation. Looped LMs are models that can repeat internal processing steps instead of using one fixed path for every token. In principle, they let an easy token stop after fewer steps and a difficult token continue for more steps. The appeal is simple: less compute for easy tokens and more compute for hard ones.

Development: Dynamic Depth Breaks Standard Batching

The problem is not the idea of variable depth. The problem is how inference servers pack work. Standard batching systems such as vLLM expect a uniform execution shape. If tokens with different numbers of loops must run together, they cannot share a normal forward pass because their computation does not line up at each step. A batch can only be efficient when the hardware can run the grouped work together.

This mismatch creates a practical barrier. A model that can choose its own depth may still be expensive to serve if each token is isolated, or if the system waits for the longest sequence in the batch. The result would be wasted compute, lower throughput, and weaker latency control. The question becomes: how can a serving system keep the benefits of looped computation while preserving batch efficiency?

Latest: CDB Forms Batches Between Loop Steps

The paper introduces continuous depth batching, abbreviated CDB, as the first efficient method for depth-adaptive looped LLMs. The core move is to stop thinking of a batch as one fixed forward pass. Instead, CDB forms new batches between loop steps. Tokens that are still in the same loop stage can be grouped together, while tokens that need more or fewer steps can move through the loop without forcing the entire batch to follow the same schedule.

This changes the shape of inference. The system no longer has to force every token into the same execution pattern. It can let depth remain adaptive while still grouping active tokens at each loop boundary. The approach targets the exact conflict between variable computation and high-throughput serving.

Current Stage: A Serving Mechanism For Looped LLMs

CDB is best understood as a scheduling and batching mechanism rather than a new benchmark number or a fully described product launch. Its value is in the timeline of inference cost: fixed forward passes made serving predictable but rigid, looped LMs made depth flexible but hard to batch, and CDB reconnects flexibility with efficiency by regrouping work between loop steps.

What follows next is whether depth-adaptive models can use this mechanism in real serving stacks. The key test is not only how many loop steps a model can skip, but how many tokens remain packed tightly enough at each boundary to keep accelerator utilization high. For readers tracking LLM infrastructure, the signal is clear: variable-depth computation is becoming a first-class inference problem, and batching is no longer a passive detail.