UltraBERT: Beyond the Limit of BERT Pre-Training.

arXiv:2610.15028v1 Announce Type: cross Abstract: Recent bidirectional encoders have been modernized with innovations from causal large language models (LLMs). However, the pretraining recipe itself remains inherited from small-scale conventions, leaving encod