"Automatic AI Experiment Reproduction with Paper Lineage." A multi-agent framework that takes an ML paper and writes, runs and debugs the code for its core experiment end to end. The new idea is paper lineage: papers leave out the tacit implementation conventions of their subfield, so before writing code the agent walks the source paper's citations and the linked code repositories to recover them. The second piece is a sampling-based unit test: a dry run on sampled mini-batches that checks the pipeline before full training, and fixes errors with line-range EDIT commands instead of regenerating whole files. The work comes from the AI9Stars team with OpenBMB, Tsinghua, Xidian and UCAS; Zhiyuan Liu, Maosong Sun and Xu Han are among the authors. It was first posted in May 2025, revised in April 2026 (v4), and accepted to the ACL 2026 main conference.

ReproduceBench is the paper's own benchmark: 13 papers from 13 AI subfields, from knowledge distillation (DKD on CIFAR-100) to PDE solving (LSM on Darcy). For each one the authors refactored the official code and re-ran it to set the target number. Each method is averaged over three runs. With Gemini-2.5-Pro as the backbone, AutoReproduce's code runs 94.87% of the time and lands a normalised 19.72% from the reference result. The concurrent PaperCoder on o3-mini runs 17.94% of the time with an 89.23% gap, and ChatDev runs 2.56% of the time. The jump in execution rate is what the debug loop buys; on the o1-judged paper-level alignment score, PaperCoder (90.41) and AutoReproduce on o3-mini (90.86) are nearly tied. On OpenAI's PaperBench Code-Dev, AutoReproduce scores 48.5% on o3-mini, 49.6% with diagrams, and 44.1% without lineage. That compares with 45.1% for PaperCoder and 43.4% for OpenAI's IterativeAgent on o1-high.

Caveats: all of these are the authors' own runs on a 13-paper benchmark they built, with an LLM judge (o1) for the alignment scores. The backbones are 2025-vintage (GPT-4o, Claude 3.5 Sonnet, o3-mini, Gemini 2.5 Pro). The task covers a single experiment per paper, not a repository-level reproduction. Code is on GitHub with no license file.

Paper

Venue ACL 2026
Authors: Xuanle Zhao · Zilin Sang · Yuxuan Li · Qi Shi · Weilun Zhao · Shuo Wang · Duzhen Zhang · Xu Han · Zhiyuan Liu · Maosong Sun
agentsresearchcodebenchmark

Related