TinyDeepSeek

From-scratch reproduction of the full DeepSeek-R1 pipeline at small scale — pretraining, SFT, and RL — replicating the DeepSeek architecture stack: Multi-head Latent Attention (MLA), auxiliary-loss-free load balancing, and Multi-Token Prediction. Ships 0.5B and 3.3B base models with intermediate checkpoints, training code, and data recipes, all Apache 2.0.

Tracked as a rarity rather than for adoption: a genuine university from-scratch pretrain of a modern DeepSeek-style stack, from Benyou Wang's FreedomIntelligence group (CUHK-Shenzhen). No arXiv paper yet (paper-backfill work-list).