FuseReg: Layer-Fusion Regularization for Bridging the Reconstruction-Generation Gap in Representation Autoencoders

Meta description: FuseReg replaces fixed layer fusion in Representation Autoencoders with random layer-subset sampling, reducing gFID scores and improving robust generation.

Tags: FuseReg, Representation Autoencoders, layer fusion, DiT, gFID

Featured image: Abstract layered neural network fusion concept, no real people, no company logos, no text.

FuseReg addresses a mismatch inside Representation Autoencoders, models that fuse features from multiple encoder layers into a shared latent space. Reconstruction and generation pull the model in different directions, and the method changes training so the latent representation does not depend on one fixed layer recipe.

Background: two consumers, one latent space

Representation Autoencoders combine encoder-layer features into a shared latent space. The source says the decoder prefers shallow, pixel-rich features, while DiT, the generative branch referenced in the reported setup, prefers deeper, more structured semantic features. Those preferences create a reconstruction–generation mismatch. A latent code that is easy for one branch to use may be less useful for the other, because the two branches need different kinds of information from the same hierarchy.

This mismatch becomes a design problem when fixed fusion is chosen. If the latent space is built from one hand-picked set of layers, the model can become too dependent on that set. The source describes the resulting fragility in practical terms: fixed-fusion decoders degrade strongly outside their training fusion.

Development: random subsets replace fixed fusion

FuseReg replaces fixed layer fusion with random layer-subset sampling during training. For a sampled subset S, the latent vector is z_S = 1/|S| sum_{k in S} h_k. The normalized sampling keeps the latent mean unchanged: E[z_S | x] = h_bar. Under a linear squared-loss surrogate, the source shows that random subset sampling explicitly regularizes cross-layer disagreement. FuseReg discourages models from relying too strongly on any particular layer composition.

The change is small but consequential. Instead of treating layer selection as a static architecture choice, FuseReg treats it as a distribution. The model sees many possible layer combinations and learns a latent space that remains useful across them. This is why the method is framed as mitigation of a reconstruction–generation gap rather than a replacement for the encoder or decoder.

Validation: robustness across layer fusions

The reported robustness result is clear: a single FuseReg decoder remains effective across full, sparse, and even single-layer inputs. Fixed-fusion decoders degrade strongly outside their training fusion. The same FuseReg decoder can reconstruct from different layer subsets without retraining, including sparse and single-layer inputs.

For a reader maintaining a generative pipeline, that behavior matters because layer availability can change during experimentation or deployment. If a decoder can still reconstruct from different subsets, the system becomes less sensitive to one exact fusion rule. The source supports that conclusion with the single-decoder robustness claim and the observed reconstruction behavior.

Generation gains: lower gFID scores in reported comparisons

FuseReg can be applied independently to the decoder and DiT. The source reports three unguided gFID improvements. Decoder replacement only moves unguided gFID from 3.01 to 2.21. For DiT-Base, joint FuseReg moves unguided gFID from 13.96 to 9.93. For DiT-XL, unguided gFID moves from 2.91 to 2.38. The same trend also holds across different pretrained encoder families.

The numbers show two separate effects. On the decoder side, replacing fixed fusion with FuseReg improves unguided generation from 3.01 to 2.21. On the DiT side, joint use of FuseReg improves DiT-Base from 13.96 to 9.93, while DiT-XL improves from 2.91 to 2.38. These are the specific values tied to the reported comparisons.

What changes inside the encoder hierarchy

FuseReg distributes reconstruction information more evenly across encoder depth. The source lists three shifts: lower dependence on individual layers, stronger reconstruction from single layers, and more stable representations under changing layer compositions. Instead of searching for one optimal fusion, FuseReg trains the model to remain robust across a distribution of layer combinations.

Current stage: from fusion tuning to robust latent sampling

The key change is not a new architecture component but a training distribution. Representation Autoencoders keep their shared latent space, but FuseReg makes layer choice part of the optimization problem rather than a fixed design assumption. The reported numbers position FuseReg as a method that can improve generation while also making a single decoder usable across full, sparse, and single-layer inputs.