Decoupled Quantization: Tailoring LLM Prefill and Decode Stages

Disaggregated quantization is a technique for large language models, or LLMs, that treats prompt processing and token generation as two weight paths that can use different numeric precisions. Qwen3.8 gives the idea a concrete public artifact: a 27B NVFP4 prefiller, a paper identifier 2609.26333, and code under IST-DASLab. For readers following LLM serving, the change is that a single quantized weight copy no longer has to serve both speed and accuracy demands at once.

Background: One Quantized Model, Two Unequal Jobs

LLM inference usually has two phases. Prefill is the first pass over a prompt, where the model converts many input tokens into cached state. It can be compute-heavy and often benefits from parallelism. Decode is the generation loop, where one token is produced after another; it often depends more on moving weights and cached state than on doing many independent calculations at once.

Quantization lowers the numeric precision of weights and, in some systems, activations. That can reduce memory use and improve throughput. The difficulty is that one precision setting is shared by both phases. A setting that is aggressive enough to speed up prefill may also affect the accuracy of decode. A setting that protects accuracy may leave too much of the long-sequence serving cost on the table, especially for dense models, where all parameters participate in each step.

Development: Splitting Weights by Phase

The disaggregated approach separates the weights used for prefill from the weights used for decode, allowing each phase to run at a different precision. The stated benefit is not just one metric: it can improve both speed and accuracy. That means the system can choose a precision path for prompt processing and another for token generation instead of forcing both through the same quantized checkpoint.

The design also uses offloading. Offloading moves weights between memory tiers so the right precision path can be used when needed. For long sequences and dense models, the source describes this as producing zero additional overhead, meaning the phase split should not add extra runtime cost after the system is arranged.

A practical point is compatibility. The method is described as applicable to existing quantized checkpoints, which means it can be layered onto already compressed model files rather than requiring a new quantization baseline from scratch.

Latest Artifacts: Paper, Code, and Qwen3.8 NVFP4

The research record is listed as arXiv:2609.26333 at https://arxiv.org/abs/2609.26333. The implementation is public at https://github.com/IST-DASLab/disaggregated-quantization. The model-side artifact is a Qwen3.8 NVFP4 prefiller, available at https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-NVFP4-prefiller. The name 27B indicates the parameter scale of the prefiller, while NVFP4 refers to a 4-bit floating-point format used for that component.

The source does not provide calendar dates for release, so the clearest markers are the paper number, the repository, and the named Qwen3.8 27B NVFP4 prefiller. Together they show the idea moving from a paper and code path to a downloadable model artifact.

The current stage is public research code and a named prefiller, not a stated production product. What to watch next is whether inference stacks expose separate prefill and decode weights as a routine option, whether existing quantized checkpoints can adopt the split without accuracy surprises, and whether offloading keeps long-sequence dense serving free of added overhead outside the paper's setup.

Meta description: Qwen3.8 27B NVFP4 prefillers show how disaggregated quantization separates LLM prefill and decode weights to improve speed and accuracy on long sequences.

Tags: llm inference, quantization, Qwen3.8, NVFP4, disaggregated quantization

Featured image: abstract technology style, no real people, no company logos, no text, with two parallel inference pathways.