The regional benchmark the SEA-LION models are built against, developed by AI Singapore with Stanford CRFM's HELM team and published at ACL 2025 Findings (Yosephine Susanto first author, William Chandra Tjhi senior author). SEA-HELM scores models along five pillars, NLP classics, LLM-specific tasks such as instruction following and chat, Southeast Asian linguistics (the LINDSEA diagnostics), SEA culture, and safety, in Filipino, Indonesian, Tamil, Thai, and Vietnamese at launch, with about 35,000 items across translation, reasoning, understanding, toxicity, and cultural tests; the 2026 revision used for v4.8 adds Malay and Burmese and folds in SEA-SafeguardBench. Its public leaderboard ranks open and closed models by language and task and is the source of the SEA-HELM numbers quoted across the SEA-LION entries.

SEA-HELM Audio (arXiv 2610.19869, 16 October 2026; blog 23 October) extends the benchmark to speech. Prompts from the SEA-HELM IFEval and Global-MMLU-Lite subsets are synthesized to speech in a single voice in Chinese, English, Indonesian, Malay and Tamil, run through cascaded ASR-LLM pipelines, and scored with the same rule-based verifiers as the text versions. The headline metric is the "speech gap" between text and speech accuracy. The paper reports drops of 9.00–18.50 points on multiple-choice reasoning and 20.00–42.86 points on instruction following, with Tamil above 40 points; in the blog SEA-LION-v4.5-27B loses 32–33 points on Indonesian and Malay instruction following. The authors note that synthetic speech lacks accents, noise and code-switching, so the gaps are likely a floor. Dataset, code (aisingapore/seahelm-audio) and a voice-agent leaderboard are public.

Paper

Venue ACL 2025 Findings

Evaluation Details

Domains 5
Scoring per-task scores averaged to per-language and overall Southeast Asian scores
Domains: NLP classics, instruction following and chat, Southeast Asian linguistics, Southeast Asian culture, safety
evalbenchmarkmultilingual

Related