On Evaluating and Improving Conversational Agents in Production.

arXiv:2609.32092v1 Announce Type: cross Abstract: We present a framework for evaluating and improving a large-scale, multi-agent shopping assistant in production, and report lessons from its use. Offline evaluation of such a system faces three obstacles. (i) A