Neural Spectral Capacity: Quantifying and Designing Architectures Using Only Network Specifications

Meta Description: Neural Spectral Capacity scores Transformer architectures from specs alone, enabling budget-aware search and LLaMA pruning; accepted at NeurIPS 2026 with code.

Tags: Neural Spectral Capacity, Transformer architecture search, NeurIPS 2026, LLaMA pruning, dynamic programming

Featured Image: Abstract technology style showing layered spectral bands and a dark grid, no real people, no company logos, no visible text.

NSC Scores Transformer Architectures Under Budget Without Training Data

Neural Spectral Capacity (NSC) assigns a score to a Transformer architecture from its specification alone, without instantiating the model, relying on data, or computing gradients. Because the score is additive across layers, budget-constrained architecture selection becomes an exact dynamic program rather than a black-box search. That reframing matters when teams need to choose Transformer variants quickly under a parameter or FLOPs budget.

Background: The Bottleneck in Transformer Architecture Evaluation

Architecture evaluation usually requires more work than a specification review. Candidate networks often must be built, initialized, and tested before a designer can tell whether they are better than alternatives. That process is costly in large Transformer search spaces, where dimensions, layer counts, attention patterns, and initialization settings interact.

NSC addresses that bottleneck by using a spectral property of random initialization. With standard random initialization, the Marchenko–Pastur law describes each weight matrix’s singular-value spectrum as a closed-form function of its shape and initialization variance. Summing those layer-level spectral quantities over the network produces NSC, a score that can rank candidates without running them.

Development: Turning Random-Initialization Spectra into a Budget Program

The score’s layer-by-layer structure gives NSC a computational advantage. If the total score is a sum over layers, maximizing it subject to a cost limit can be expressed as a bounded-knapsack dynamic program, called NSC-DP. This exact program returns the architecture that globally maximizes NSC under a parameter or FLOPs budget, instead of relying on sampled search trials.

The reported ranking result uses FlexiBERT, which contains 500 BERT architectures. NSC reaches τ = 0.695, compared with 0.485 for #Params and 0.635 for W-PCA. The difference becomes sharper when candidates have similar sizes. On pairs whose #Params differ by less than 10%, NSC keeps τ = 0.505, while #Params collapses to 0.082. The numbers show that parameter count loses ranking signal among size-normalized candidates, while NSC retains a stronger ordering signal.

Latest Progress: A 2-Second Search, LLaMA Pruning, and NeurIPS 2026

The search result demonstrates the method’s speed. On WikiText-103, NSC-DP found a Transformer-XL architecture with 23.087 PPL, beating the human-designed baseline at 23.279 PPL. In perplexity evaluation, lower PPL is better. The reported search took 2 seconds on one CPU core.

The pruning result extends NSC from architecture selection to model reduction. Applied to LLaMA-7B, NSC produced a 5.7B model that was evaluated across eight commonsense tasks without calibration data. The pruning process was reported as ~5,900× faster than the strongest training-free proxy baseline. The work is accepted at NeurIPS 2026, with code provided.

What remains to be tested is how consistently this spectral score predicts downstream quality outside the reported FlexiBERT, Transformer-XL, and LLaMA cases, especially as budgets are defined in parameters or FLOPs and architectures move across task families.