Transformer depth

DepthBench: Quantifying How Residual Connections Facilitate Greater Computational Depth

DepthBench tests 10 Transformers at fixed size and training, showing depth helps only when residual and normal…