ERNIE 5.5 Flash Base
modelYour notes
Baidu's first open-weight release of the ERNIE 5.x generation, and deliberately a pre-trained base checkpoint: it was taken at the end of pre-training, before any mid-training or post-training. Baidu says a post-trained ERNIE 5.5 Flash will follow. 280B total / 14B active MoE, pre-trained from scratch on 33.5T tokens with Muon in FP8 mixed precision. Pre-training context was extended in stages to 128K, and the model supports up to 1M tokens for fine-tuning and inference (max_position_embeddings 1,048,576). It is text-only. The hardware and GPU count are not disclosed.
The architecture borrows heavily from DeepSeek V4 Flash, which it is sized against (experimental_attention_variant: dsv4_hybrid in config.json), and adds several Baidu components:
- Shape: 43 layers at hidden size 4096 with 64 heads.
- Attention: hybrid. One layer in seven uses token-level MLA with DSA sparse selection (64-head indexer, top-2048). The other 37 use Heavily Compressed Attention, which caches one entry per 128 tokens plus a short window of recent tokens. Every attention layer is low-rank gated, and Virtual Head Attention mixes across heads.
- MoE: every layer after the first is an MoE block with 512 routed experts, top-10, plus one shared expert. The experts run in a 2048-d latent space. Routing uses a Multi-Router ensemble of sigmoid routers with auxiliary-loss-free balancing.
- Residuals and output: mHC with 4 residual streams, and a MultiMax output head.
- Multi-token prediction: ECHO replaces a separate MTP module by reusing the final backbone layer, which also serves as the speculative-decoding draft.
- Tokenizer: a new BPE tokenizer of about 200K tokens (201,216) that splits numbers right to left in groups of three.
Baidu also released FlashMask-4, an attention kernel for large head dimensions on SM90 and SM100. Three of the components have their own papers: Virtual Head Attention, Multi-View Latent MoE (the source of Multi-Router) and ECHO.
Baidu evaluated the base model against three other base models from their public checkpoints, with the same harness and prompts for all of them: DeepSeek V4 Flash Base (284B-A13B), Hy3 preview Base (295B-A21B) and MiMo V2.5 Base (310B-A15B). Baidu reports the best score on 25 of 30 benchmarks. Results, with DeepSeek V4 Flash Base in parentheses:
- Knowledge: MMLU 93.3 (89.4), MMLU-Pro 78.4 (69.2), SimpleQA 34.1 (29.1).
- Reasoning: GPQA 60.2 (55.9), BBH direct 77.1 (69.0).
- Code: HumanEval+ 82.3 (69.5), MBPP+ 81.5 (67.2), LiveCodeBench 40.1 (31.9).
- Long context: LongBench v2 48.7 (44.5), RepoQA 91.7 (81.0).
- Where it trails: ChineseSimpleQA, INCLUDE-lite, HELMET-Rerank and InfiniteBench (behind DeepSeek V4 Flash Base), and FullStackBench and CoT BBH (behind Hy3 preview Base).
Weights are FP8 and BF16 safetensors on HuggingFace. They load with Transformers and serve through SGLang and vLLM, and NVIDIA's Megatron-Bridge has a fine-tuning example. The ERNIE 5.5 Flash Base Model License Agreement v1.0 is not an OSI license. Its terms:
- Non-commercial use: free.
- Commercial use, related revenue up to US$50M a year: free, but requires a one-time notice to Baidu and a "Powered by ERNIE" attribution.
- Above US$50M: requires separate written authorization.
- Derivatives: the terms cover models distilled or trained on the base model's outputs.
- Territory: the license excludes the European Union, the United Kingdom and South Korea.
Model Details
Benchmark Scores
| Benchmark | Score | Mode |
|---|---|---|
| MMLU | 93.3 | base, 5-shot |
| MMLU-Pro | 78.4 | base, 5-shot |
| GPQA | 60.2 | base, 5-shot |
| HumanEval+ | 82.3 | base, pass@1 |
| LiveCodeBench | 40.1 | base, pass@1 |
| LongBench v2 | 48.7 | base, 0-shot |
Variants
| Name | Parameters | Notes |
|---|---|---|
| ERNIE-5.5-Flash-Base | 280B | FP8 base checkpoint |
| ERNIE-5.5-Flash-Base-BF16 | 280B | BF16 base checkpoint |