Anatomy of Off-Policy Policy Gradient: Importance Sampling, KL Regularization, and Baselines.

arXiv:2610.25725v1 Announce Type: cross Abstract: Reinforcement learning has become a central tool for large language model (LLM) post-training, where policy gradient methods are routinely deployed off-policy, even though vanilla policy gradient assumes on-pol