DeepInfer: What DeepSeek Open-Sourced, and What It Left Out

Ships one model only

DeepInfer is the inference framework DeepSeek uses for its own large-scale production serving. It was open-sourced in October 2026 under Apache-2.0. The repository is deepseek-ai/DeepInfer. It is a different piece of software from DeepSeek Harness, which is the agent product, not this serving stack.

The release supports one model: DeepSeek-V4.1-Flash. The hardware list is two entries: an NVIDIA SM100 GPU, or a Huawei Ascend 950 NPU. The code is based mainly on vLLM, and it also draws on SGLang and TensorRT-LLM.

On the 2026-10-22 snapshot the repository had 17,962 stars. It was created on 2026-10-16, about 3,131 stars a day. The Hacker News thread that opened it sat at 2,302 points and 882 comments.

Kernels and scheduling

The kernels named in the release are:

• FlashMLA
• DeepGEMM
• DeepEP
• TileKernels
• DeepSelect
• DeepSeek's own TileLang kernels

Parallelism is split the same way the model is. Attention uses data parallel and balanced context parallel. Mixture-of-experts uses expert parallel plus EPLB. That scheduling is the serving-side cousin of how systems route a query to a model: the decision is which worker gets the tokens, not which product the user opened.

Speculative decoding is DSpark with confidence-scheduled verification. Set DEEPINFER_ENABLE_MTP_VARLEN=1. DEEPINFER_USE_NEXTN_DECODE sets the draft length. The default is 5.

Decoder sliding-window attention uses bounded replay. The documented settings are DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140. The kept suffix has to cover the sliding-window plus the maximum lookback.

Scheduling is asynchronous: CPU batch scheduling overlaps device execution, and that overlap is on by default. Batch-invariant mode is DEEPINFER_BATCH_INVARIANT=1, NVIDIA only. Deterministic sampling needs that flag, plus DEEPINFER_ENABLE_MTP_VARLEN=0, plus a fixed seq_seed. That path is NVIDIA only too. Structured output covers regex, JSON schema, and EBNF, passed through so_args on SamplingParams.

The published benchmark keeps one scale. Prefill is measured on 4 SM100 GPUs, decode on 64, and both stay inside the same scale-up domain.

What is missing

The README lists three things a complete inference service needs that this release does not include:

1. A KV cache system.
2. VE / Prefill / Decode separation. Most of the engine side is already implemented.
3. Load balancing of prefill and decode across data-parallel replicas.

Those three are the serving bill, which is the subject of AI agent cost control once a framework is asked to stay up for other people's traffic.

How to try it

The README's prerequisites are:

• Linux
• GCC 13 or newer
• Python 3.12
• CUDA 13 on NVIDIA, or CANN 9.2 on Ascend

The model stays DeepSeek-V4.1-Flash. The device stays an SM100 GPU or an Ascend 950 NPU. Nothing in the release adds a second model or a third device.

Bottom line

The README itself lists three things a complete inference service needs that this release does not include: a KV cache system, VE / Prefill / Decode separation (most of the engine side is already implemented), and load balancing of prefill and decode across data-parallel replicas.

What was open-sourced is the kernels, the scheduling, speculative decoding, and sampling control. It is not an inference server you can put in front of users as it stands. Filling those three gaps is the real work.

It does not support any other model, and it does not support consumer cards. The only hardware in the release is NVIDIA SM100 and Huawei Ascend 950.

how do AI agents work — return to the complete AI agent architecture guide.