Can we trust LLMs to judge whether a research idea is novel?

A study called Old Ideas, Novel Problems tested how reliably large language models judge the novelty of research ideas, and found that small changes to the evaluation setup can produce very different verdicts.

Why it matters: It suggests LLM judgments of research novelty may be sensitive to how the evaluation is set up, which matters if such models are used to assess ideas.

Written by AI from Ai2 Blog's published text.