Massive activations, residual-stream coordinates with magnitudes far larger than typical activations, are associated with attention sinks in transformers, but how their scale is regulated during training remains incompletely understood.
← Back to all developments · Signals are a development reported by more than one source. See the methodology.