Are Human-Aligned Models Models of Humans? A Turing-Test Gap in Preference Alignment.

arXiv:2609.23640v1 Announce Type: new Abstract: Human-feedback alignment has made language models useful assistants and is commonly described as aligning them with humans. However, the responses people prefer from an AI need not be the responses they themselve