VinePPO

VinePPO introduces Monte-Carlo credit assignment that fixes value-network failure in PPO for LLM reasoning, computing unbiased per-step value estimates by rolling out from intermediate states rather than training a separate critic. A widely-cited RL-post-training method from McGill-NLP / Mila (Aaron Courville, Nicolas Le Roux, Siva Reddy) that improves both sample efficiency and final accuracy on math-reasoning benchmarks over standard PPO.