The Direction of Ignorance: LLMs Encode and Temper a Unigram Prior.

arXiv:2609.02959v2 Announce Type: replace-cross Abstract: What does a language model predict when it has few clues?The answer lies in its unembedding geometry: a single direction of the unembedding matrix encodes the unigram distribution of the training corpus