LayerNorm drives this toward zero; RMSNorm preserves shift.
LayerNorm centers and rescales each token's features. RMSNorm rescales without subtracting the mean. Inject shifts and outliers, then compare the geometry, statistics, and residual behavior.
Open activation labTransformer residual streams repeatedly accumulate updates. Normalization makes each sublayer see a controlled activation magnitude, improving optimization across depth. The choice of centering, epsilon, learned gain, and placement affects what information survives.
LayerNorm drives this toward zero; RMSNorm preserves shift.
Root mean square after optional normalization and gain.
Largest absolute feature after processing.
Directional similarity between input and output vectors.
Subtracting the feature mean makes the vector translation-invariant. Dividing by standard deviation gives unit variance before learned gain and bias restore useful scale and offset.
y = γ ⊙ (x − μ) / √(σ² + ε) + βRMSNorm removes the centering operation and normalizes only by root mean square. It is cheaper and preserves information carried as a common shift.
y = γ ⊙ x / √(mean(x²) + ε)Pre-norm blocks normalize before attention or the MLP, leaving a direct residual path through depth. Post-norm blocks normalize after adding the sublayer output and can require more careful optimization.
pre-norm: xₗ₊₁ = xₗ + F(norm(xₗ))LayerNorm erases uniform offsets; RMSNorm retains their direction while controlling overall energy.
Choose based on what the residual stream encodes.A single large feature raises the denominator and compresses ordinary coordinates, even when global RMS is controlled.
Normalization is not clipping.Epsilon prevents division by nearly zero variance, but an oversized epsilon weakens scale normalization.
Numerical stability is part of the model.