PanGu-Sigma

model paper
Huawei , PCL
Trillion-parameter sparse language model (1.085T) extending PanGu-alpha with Random Routed Experts (RRE). Trained on 329B tokens in 40+ languages on 512 Ascend 910 accelerators.
Paper (arXiv)

Outputs 2

PanGu-Sigma

model
Architecture MOE
Parameters 1.1T
Training tokens 329B

PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing

paper
Paper (arXiv)
Citations 7
arXiv HTML
nlpmoe

Related