CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF.

arXiv:2610.21138v1 Announce Type: new Abstract: Feedback noise is pervasive in preference data, arising from subjective human judgments, annotation mistakes, and imperfect LLM judges. Standard reward modeling trained on such feedback yields a biased estimate o