An assay-grounded protein-engineering benchmark that asks not whether a model can score a mutation but whether it can make the decisions a protein-engineering campaign actually requires. Built from 202 unique real assays, expanded to 293 assay–task instances and 607,269 candidate rows, it defines four task interfaces: T1 single-mutant generation over the full legal single-mutation space with no measurements supplied; T2 measurement-free multi-mutant ranking (zero-shot discrimination over combinations); T3 anchor-informed multi-mutant ranking, where a set of measured mutants and their fitness is given; and T4 single-mutant-informed multi-mutant ranking. Generation tasks are scored by recall, ranking tasks by Spearman correlation. Protein language models, general LLMs and LLM-based scientific agents are run through the same interfaces, which is the comparison most mutation-effect benchmarks cannot make. By Shanghai AI Lab with Tsinghua's Institute for AI Industry Research (AIR) and School of Pharmaceutical Sciences; released on HuggingFace in September 2026 and announced on 2026-10-30 as the science-intelligence arm of the lab's OpenCompass (司南) evaluation system.

The headline result is a negative one, and the lab presents it that way: current models cannot reliably carry out protein modification. Under a budget of 40 candidates, the best model averages fewer than two high-fitness mutations found, and performance falls as mutation order rises. The two model classes are complementary rather than ordered — protein language models are stronger at surfacing good mutations when no experimental data is available, while general LLMs and agents pull ahead once measurements are in hand, and the two tend to capture different mutations. Iterating on experimental feedback helps measurably: at the same total budget of 40 candidates, GPT-6 Astra's high-fitness recall rises from 2.54% to 4.65% across feedback rounds, which the authors read as evidence that closed-loop querying beats one-shot answering. The lab frames the whole result as support for its "通专融合" (general–specialist fusion) thesis: couple general models with specialist protein models into a hypothesis → experiment-selection → feedback loop. Dataset license is listed as "other".

Paper

Evaluation Details

Questions 293
Domains 2
Scoring Recall (generation tasks), Spearman correlation (ranking tasks)
Used in: OpenCompass (司南)
Domains: protein engineering, biology
benchmarkevaluationsciencebiologyagents

Related