Scaling Repeated-Data Pretraining: Batch Size, Regularization, and GPU-Time Tradeoffs

We study pretraining when compute is abundant but unique training data are finite, requiring repeated training on the same data.

← Back to all developments · Signals are a development reported by more than one source. See the methodology.