On October 20, 2026, Chinese artificial intelligence laboratory DeepSeek released DeepInfer, an open-source, high-throughput inference engine representing the exact serving software behind its commercial production clusters. Published on GitHub under the repository deepseek-ai/DeepInfer, the framework provides the lower-level kernels, scheduling routines, and parallelization abstractions required to run DeepSeek-V4.1-Flash at peak hardware efficiency across both NVIDIA SM100 graphics processors and Huawei Ascend 950 neural processing units.
DeepInfer unifies DeepSeek's specialized kernel library into a cohesive runtime. The framework natively integrates FlashMLA for multi-head latent attention, DeepGEMM for low-precision matrix arithmetic, DeepEP for expert-parallel inter-device communication, and custom operations written in DeepSeek's domain-specific TileLang. The system supports complex parallel topologies, including attention data parallelism, balanced context parallelism, and mixture-of-experts expert parallelism combined with expert parallelism load balancing (EPLB).
To minimize serving latency during iterative generative loops, DeepInfer introduces two novel architectural scheduling primitives. First, DSpark implements multi-token prediction (MTP) governed by confidence-scheduled verification, allowing the model to draft speculative sequences up to 5 tokens in advance while dynamically scheduling verification checks based on intermediate confidence scores. Second, for sliding-window attention (SWA) layers, the engine enforces bounded replay prefill truncation. By retaining only a compact 140-token suffix covering the active attention window and maximum backtracking horizon, DeepInfer avoids recomputing massive historical context buffers.
In enterprise deployments, inference non-determinism presents major debugging obstacles. Addressing this, DeepInfer features batch-invariant inference on NVIDIA hardware. Under standard inference engines, variations in dynamic batch packing alter floating-point summation order, causing identical prompts to yield divergent outputs across concurrent requests. DeepInfer guarantees bitwise reproducible generations regardless of surrounding batch workloads, facilitating regression testing in mission-critical applications.
DeepSeek clarified that the current release addresses offline inference execution and single-instance serving. To assemble a complete commercial web API, operators must supplement DeepInfer with external distributed key-value cache storage layers and disaggregated prefill/decode routing balancers. The repository gained over 14,000 GitHub stars within 24 hours as infrastructure engineers began deploying the engine across private compute clusters.