Modify the Multiplication, Retain the Parameters: Associative Algebraic Layers for Transformers

Title: Qwen DeepSeek Study Tests Cheaper Matrix Product in Transformers

Meta description: A study tests lower-rank products replacing Transformer matrix multiplication in Qwen and DeepSeek shapes, showing throughput gains and score drops.

Tags: Transformer projections, matrix multiplication, Qwen, DeepSeek, AI inference

Featured image: abstract technology style showing interconnected linear algebra blocks, no real people, no company logos, no text.

Background: Matrix multiplication in Transformer projections

In modern Transformers, the dominant arithmetic in dense layers is matrix multiplication. A projection is a linear map applied to hidden states: in attention blocks, projections transform queries, keys, and values; in feed-forward blocks, projections expand and contract the model dimension. Because these operations are used many times during training and inference, even a small change in their arithmetic can have a visible effect on speed and cost.

The study takes this standard assumption as its starting point. It asks whether the matrix multiplication used in Transformer projections can be replaced by a different, cheaper associative product while keeping the same weight bank and parameter count. An associative product is an operation whose grouping can be changed without changing the result, a property that matters when a computation is split across hardware or rewritten as a sequence of smaller operations.

Development: Lower-rank associative product and GPU kernels

The first step is to reduce the cost of the algebraic operation itself. The study compares the standard bilinear representation of matrix multiplication with a lower-rank associative product. In the q=2 case, the proposed product has bilinear rank 6, while the Strassen 2x2 product has rank 7. Lower rank generally means fewer multiplication components, so it can reduce the work required to compute the projection.

After establishing the algebraic form, the study moves to hardware testing. It evaluates GPU projection kernels across Transformer shapes, including shapes associated with Qwen and DeepSeek models. This stage matters because a theoretical rank reduction does not automatically produce faster end-to-end execution: memory movement, kernel efficiency, and the surrounding Transformer architecture can absorb or offset the arithmetic saving.

Latest findings: 110M-parameter training and throughput gains

The study then expands from kernels to trainable models. It uses two decoder-only language models with approximately 110M parameters, trained for 12.3B tokens. The two model variants differ only in the MLP multiplication law, which keeps the comparison focused on the effect of replacing the multiplication rule while holding the parameter count and weight bank structure intact.

In these measurements, the algebraic model achieves a 6.2-7.8% increase in end-to-end generation throughput across four prompt domains. That result gives the main positive signal: the lower-rank product can translate into measurable generation speed improvements in the tested settings.

The same results also include the main negative signal. The algebraic model obtains lower scores on GSM8K, MBPP, and IFEval, three benchmark areas used to probe reasoning, coding, and instruction-following style evaluation. The throughput gain and quality drop therefore appear together in the reported small-scale experiments.

Current stage: feasibility and trainability study

The work is described as a small-scale feasibility and trainability study. Its value is not a finished replacement for standard Transformers, but a demonstration that an associative, lower-rank multiplication law can be inserted into Transformer-shaped computations, trained on language modeling data, and evaluated in GPU kernels and generation throughput.

It also marks the point where the trade-off becomes concrete: faster generation in the measured domains is paired with lower scores on GSM8K, MBPP, and IFEval. The study therefore sits at the early experimental stage, where the key remaining questions are how the quality gap scales with model size, whether the throughput advantage persists across more hardware shapes, and how the algebraic product interacts with the full parameter bank in larger Qwen or DeepSeek-style systems.