What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models.

arXiv:2610.37631v1 Announce Type: new Abstract: Language models pretrained on web corpora collected over multiple years learn both general and domain-specific knowledge. How these different kinds of knowledge are learned as pretraining progresses remains uncle