Head Dimension Matters in Language Models for Long-Context Retrieval.

arXiv:2610.32213v1 Announce Type: cross Abstract: Multi-head attention retrieves and combines context representations through query--key interactions and value aggregation, with head dimension constraining the per-head representation space. In long-context pro