Typhoon: Gradient-Based Task-Specific Masking for Masked Language Models.

arXiv:2303.15619v3 Announce Type: replace-cross Abstract: The choice of which tokens to mask is an under-examined design decision in masked language modeling (MLM). Standard pretraining masks tokens uniformly at random, whereas content-aware masking schemes ca