DeepInfer

DeepSeek's own inference engine, open-sourced on October 16, 2026. The README describes it as "the high-performance inference framework that powers DeepSeek's large-scale production services". Until now the lab had released the parts (FlashMLA, DeepGEMM, DeepEP, EPLB, DeepSelect), and this is the engine they plug into. It supports one model, DeepSeek-V4.1-Flash, and the default checkpoint is the October 17 V4.1-Flash-1017 update. It runs on two hardware targets, NVIDIA SM100 (Blackwell) GPUs and Huawei Ascend 950 NPUs, which extends the dual stack of the September Ascend kernel ports from kernels to the whole runtime. DeepSeek says it was "originally built primarily on vLLM", with ideas taken from SGLang and TensorRT-LLM.

The feature list covers what V4.1-Flash's architecture needs. Parallelism is attention data-parallel with balanced context parallelism, plus MoE expert parallelism with EPLB. DSpark speculative decoding comes with confidence-scheduled verification. SWA bounded replay replays only a short suffix (140 tokens by default) at decoder prefill, and a zero-overhead asynchronous batch scheduler is on by default. It also offers batch-invariant inference and deterministic sampling (both NVIDIA-only), and regex, JSON-schema and EBNF structured outputs. Kernels come from the DeepSeek libraries above, TileKernels, and custom TileLang ops. The release is offline inference only. The README lists what a full service still needs and the release leaves out: a KV-cache system, VE/prefill/decode disaggregation, and load balancing across data-parallel replicas. The published throughput curves were measured on 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain, and are not independently reproduced. Apache 2.0. It had about 21.3K stars and 2.4K forks eight days after release.