AstaBrief-8B
modelYour notes
Report generation in Ai2's Asta platform ran on a Claude-backed ScholarQA pipeline: retrieve literature, summarize snippets, cluster them, then write the answer section by section. It worked, but it was slow, expensive to serve, and closed — which rules it out for institutions whose research questions touch unpublished or sensitive work. AstaBrief-8B is the attempt to replace it with an open-weights model small enough to self-host: it takes a research question plus retrieved literature excerpts and writes the entire cited report in one pass, skipping the summarization and clustering stages entirely. Across the full Asta pipeline, the resulting Fast mode averages 51.1 seconds per report against 178.5 for Thinking mode, about 3.5× faster.
It is post-trained from Qwen3-8B with SFT plus DPO and no RL — a deliberate choice Ai2 explains against their own prior results: RL-based methods had improved long-form report generation in DR Tulu, but "RL-based training can be unstable and expensive" and they wanted "a cheaper, more operationally manageable setup — one that's also easier to debug and iterate on." That bet pushes all the risk onto data quality. Real Asta/ScholarQA user queries were filtered for quality, relevance and privacy (beta-tester and bot traffic stripped, short queries dropped, an LLM pass removing non-English, non-scientific and PII-bearing prompts) down to a pool of 90K research-focused queries; SFT targets were generated through the multi-step ScholarQA pipeline backed by a mix of Claude 3.5 Sonnet, Claude 3.7 Sonnet, o3, o4-mini and GPT-4.1, yielding 47K usable examples. DPO pairs came from a disjoint query subset, pitting a ScholarQA report (usually Claude-backed) against one generated by o3, o4-mini, DeepSeek-V3 or DeepSeek-R1; GPT-4.1 and DeepSeek-R1 both judged each pair and only unanimous pairs were kept (the judges agreed with human preferences 95% of the time), leaving about 6K pairs.
The most transferable finding is negative. Four statistics-based filters were tried on the synthetic SFT reports — output-to-input token ratio, mean retrieval relevance of cited papers, citation density (share of statements carrying at least one citation), and citation diversity. "The strongest gains came from filtering out synthetic reports with low citation density; more aggressive filtering, filter combinations, and learning-rate sweeps didn't add meaningful gains." Ai2's reading: scientific specialization "isn't necessarily a matter of adding more scientific text to pretraining" — whether post-training data actually demonstrates grounding and attribution matters more than how elaborately it is selected.
On the ScholarQA-CS2 test split, DPO lifts the average from 77.3 for stock Qwen3-8B and 83.7 for the SFT checkpoint to 87.0, with citation precision moving 76.2 → 87.7 → 90.5 and citation recall 64.6 → 71.3 → 78.2. Against the systems it replaces it is close rather than ahead on absolute scores: SQA-CS2 dev 86.3 vs 87.6 for Asta ScholarQA and 86.5 for DR-Tulu-8B, and DeepScholarBench 53.50 vs 60.25 and 56.26 — its clearest loss. It does win the pairwise comparisons against Asta ScholarQA (55% on dev, 72% on test), and in a 14-question human study with three researchers DR Tulu wins overall preference while two of the three prefer AstaBrief on citation accuracy. Ai2 flags that most of the work was done in 2025, so the proprietary comparisons reflect that frontier, and that their metrics cover relevance, coverage and citation grounding but not whether a report preserves the evidentiary scope of its sources — turning a sample-specific finding into a generic claim, or a descriptive result into a recommendation. Apache 2.0, ~63 HuggingFace likes; DPO ran in open-instruct on 8×H100, and the SFT mix, DPO mix and prompts are all released.
Model Details
Benchmark Scores
| Benchmark | Score | Mode |
|---|---|---|
| ScholarQA-CS2 (test, average) | 87.0 | — |
| ScholarQA-CS2 (test, citation precision) | 90.5 | — |
| ScholarQA-CS2 (test, citation recall) | 78.2 | — |
| SQA-CS2 (dev) | 86.3 | — |
| DeepScholarBench | 53.50 | — |