DeepSeek DeepInfer: Open-Source AI Engine Runs on Huawei
Published October 21, 2026 · By Daniel Rao
On Monday, the US government told every federal civilian agency to rip DeepSeek out of their systems. On Tuesday, DeepSeek handed the world its crown jewels — the software that runs its own AI services — and said, in effect: here, take it, it is free. Either the timing is the best-judged piece of defiance in recent tech history, or it is the coincidence of the year. The honest answer is a little of both, and a lot of neither.
Update, Oct. 21: 15,000 stars, a vLLM thank-you, and the benchmarkers' hedges
The day after this post went up, the scoreboard kept moving. The DeepInfer repository passed 15,000 GitHub stars (systemdesignacademy/Medium, Oct. 21), and the vLLM project publicly thanked DeepSeek on X for the open-source credit, saying it would fold DeepInfer's ideas into its own DeepSeek-V4.1-Flash support (The Next Web via teckhustlers) — an unusually gracious truce from the project DeepSeek built on. Independent observers are split: Tim Dettmers judged DeepSeek's serving stack comparable to OpenAI's and better than Anthropic's at this user scale (latent.space AINews — his judgment, not a matched benchmark), while the skeptics note that no one outside DeepSeek has independently verified the 155-trillion-tokens-a-day figures and the benchmarks skip any comparison against vLLM or SGLang. One quiet gate worth knowing: the README supports NVIDIA SM100 and Huawei Ascend 950 only — teams on Hopper or older GPUs get nothing from this release yet (zyvop). DeepSeek says inference code for future models will land in the same repository (The Next Web via zyvop). The next proving ground is the first independent vLLM/SGLang head-to-head.
What landed on GitHub on October 20 was not another model. It was DeepInfer, DeepSeek's in-house inference framework — the code that turns model weights into fast, cheap answers for millions of concurrent users, at what the company claims is the most aggressive cost curve in the industry. It is the first time any frontier lab has opened up the whole engine rather than a few carefully chosen parts, and it arrived under the permissive Apache 2.0 licence. Eighteen months ago, DeepSeek said it would never do exactly this. It changed its mind.
This is not a model-release story. It is an economics story wearing an open-source trench coat — and a geopolitical one, because of a single line in the README.
One day after the purge order
The sequence of events is what makes this one hard to look away from. On Monday — DHS ordered federal civilian agencies to strip DeepSeek products, and anything built on them, from their IT systems, banning the lab from federal contracts. On Tuesday morning, @deepseek_ai posted a four-part thread on X: "Introducing DeepInfer, our in-house inference framework, now fully open source! Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We're sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable."
We opened the GitHub repo on October 21 to see what the fuss was about, and the numbers were already absurd: 16,446 stars, 1,634 forks, four commits — from a repository that had sat private since October 16 and only made its first public commit on the 20th. One of the most-liked replies landed the punchline the open-source crowd had been rehearsing for a year: OpenAI, "supposed to be open for the benefit of humanity, ended up staying closed."
Before going further: no, this was almost certainly not thrown together overnight as a middle finger to Washington. Repositories like this take months to clean up for public release — the April 2025 decision not to release, which we will get to, proves DeepSeek has been deliberating this for a year and a half. But they knew exactly what day it was. That timing is deliberate theatre, and the audience is everyone.
What DeepInfer actually is
An inference engine is the part of AI that happens after training: taking a finished model and serving answers to millions of people quickly and cheaply. The same model can be dramatically cheaper or more expensive to run depending on the software wrapped around it. DeepSeek's famously low prices have always rested on how well it does this part, and DeepInfer is the first time the lab has shown the whole engine instead of individual pieces.
The README, which we read line by line, is honest about what you get and what you do not. The supported model list has one entry: DeepSeek-V4.1-Flash. The mode is offline inference only. What you do get is the core machinery: custom low-level kernels (FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels), attention data parallelism, balanced context parallelism, mixture-of-experts expert parallelism with a load balancer called EPLB, speculative decoding, asynchronous scheduling that is on by default, deterministic sampling, and structured outputs. The acknowledgements say the framework was "originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM" — a credit to the open-source community that is doing real work here, not decoration.
The default checkpoint — deepseek-ai/DeepSeek-V4.1-Flash-1017, the refreshed weights DeepSeek put out over the weekend that Artificial Analysis calls one of the world's most powerful and cost-efficient open-weight models — downloads into the Hugging Face cache on first run. But here is the catch the hype skipped: the README lists what a complete inference service still needs, and this release does not have it — no KV cache system, no prefill/decode disaggregation across machines, no load balancing across data-parallel replicas. DeepSeek calls those "tightly coupled" to its internal infrastructure, and describes the smallest efficient deployment as 64 GPUs for prefill plus 64 for decoding. MarkTechPost puts it more bluntly: offline inference only; everything that makes it a live service is left to you.
The Huawei line in the README
And then there is the line that will be quoted in every geopolitics briefing this week. Supported hardware, per the README: NVIDIA SM100 GPUs — and Huawei Ascend 950 NPUs.
Read that again. A frontier AI lab, in its first-ever open-sourcing of production inference code, has put Huawei's silicon on equal billing with Nvidia's latest Blackwell chips. This is a first-party production endorsement of Chinese accelerators from the lab the US government is currently trying to purge from federal systems. DeepSeek's benchmarks all ran on Nvidia hardware — and two features, batch-invariant inference and deterministic sampling, are Nvidia-only for now — but the Ascend 950 is a supported target from day one. That is not nothing. It is the single most consequential line in the repository.
The context makes it louder. DeepSeek brought its TileLang kernel programming language to Huawei's chips on September 30, with Huawei's help, and Bloomberg reported that same month that the startup plans to deploy at least 160,000 Ascend 950DT accelerators at a new data centre in Inner Mongolia. Meanwhile the hardware economics of the whole region are shifting under Washington's feet — ByteDance, Alibaba and Tencent just placed a $35B order for Nvidia workstation GPUs that Nvidia has warned it may not be able to fill, precisely because the datacentre chips are banned and taxed. DeepInfer sharpens the tension: the software layer now treats Chinese and American silicon as interchangeable.
Inside the engine room
The two ideas worth your attention are DSpark and what DeepSeek calls SWA bounded replay.
Speculative decoding guesses several tokens ahead and checks them in one go — faster when the guesses land. DeepInfer's version, DSpark, has a draft model propose five tokens per step while a "confidence head" predicts which the main model will accept; verification compute goes only to the likeliest candidates. DeepSeek says this trims the draft tokens checked per step by about 30 per cent, for a 4 to 7 per cent throughput gain at its production load — with the honest caveat, per MarkTechPost's reading of the same deployment Q&A, that it can run slightly slower at low sequence counts.
SWA bounded replay is the prefill trick: instead of re-running a whole prompt back through the model, DeepInfer replays only a bounded suffix — the last 140 tokens under the default V4.1 settings. DeepSeek says that nearly halves total prefill computation. And then there is the batch-invariant mode, which guarantees identical inputs produce identical outputs regardless of what else happens to be in the batch — the kind of reproducibility feature that enterprise and regulated users actually need, and a genuine differentiator over stock vLLM.
The production numbers DeepSeek attaches to all of this are, to be clear, DeepSeek's own claims, and no independent lab has verified them yet: one cluster serving 155 trillion tokens a day, a 96 per cent input cache hit rate, average time-to-first-token around 400 milliseconds, output at 230 tokens per second — 300 for coding — and revenue of $38 per GPU per hour at off-peak pricing. The throughput charts on GitHub, measured on 4 SM100 GPUs for prefill and 64 for decode, show decode topping 25,000 tokens per second per GPU at high batch sizes — again, self-reported, with no direct comparison against vLLM or SGLang. That comparison is now the community's job. If those numbers survive independent benchmarks, every inference provider on earth has a problem.
The community's first verdict: thank-you from vLLM, hedges from everyone else
The vLLM project's public thank-you is the part nobody expected. DeepSeek built DeepInfer on vLLM's lineage — and the project responded by welcoming the production learnings home, folding DeepInfer's ideas into its own DeepSeek-V4.1-Flash support. Production insight is flowing back into the ecosystem instead of away from it. The independent voices are split from there: Tim Dettmers judged DeepSeek's serving stack comparable to OpenAI's, and better than Anthropic's, at its user scale (latent.space AINews — his judgment, not a matched benchmark). The skeptics' case is that no one outside DeepSeek has independently verified the 155-trillion-tokens-a-day figures, the benchmarks skip any matched comparison against vLLM or SGLang, and pieces like KV cache, disaggregation and routing are missing. For readers: if you're an inference engineer, the code is worth reading this week; if your fleet runs on Hopper or older GPUs, this release isn't for you yet — keep vLLM or SGLang.
What's missing — and why that matters
This is the part of the story the victory lap skipped. DeepInfer is not, today, something you can point at the internet and call a service. The missing pieces — a KV cache system, prefill/decode disaggregation, load balancing across replicas — are not garnish; they are the difference between a fast engine and a running one. DeepSeek is explicit about this in the README, which is commendable, but it means the real price of admission is a 128-GPU cluster and a team that knows what to do with it.
There is a quieter omission: the benchmarks were all measured on Nvidia hardware. Huawei support is in the README, but no Ascend numbers are in the repo. Treat the Huawei line as a signal of intent, not a benchmark.
Then there is deepseek-recipe, the companion release. It is an MIT-licensed set of Rust libraries with Python bindings that converts API requests — in OpenAI or Anthropic formats — into DeepSeek's internal conversation format and parses the output back. Its README is plain: the caller must provide model inference, tool execution and HTTP transport. Translation: DeepSeek just handed you its protocol layer too. What remains is serving — which is, not coincidentally, the part it kept.
Eighteen months ago, DeepSeek said no
In April 2025, DeepSeek published a note saying it had considered releasing its full inference engine and decided against it. The code, it said, was a heavily customised early fork of vLLM, "tightly coupled" with its internal cluster tools — and maintaining a large open-source project was more than "a small research team focused on developing better models" had bandwidth for. It promised to contribute pieces to existing projects instead, and to be fair, it did: FlashMLA, DeepGEMM and DeepEP all shipped as open kernel libraries, and DeepInfer uses them now.
Something changed. The README still credits vLLM as the base it was "originally built primarily on," so this is not a rewrite — it is the same lineage, decoupled enough to publish. What changed is presumably the strategic calculus: a small team decided that the community maintaining this code for them is worth more than the secrecy. DeepSeek has concluded the engine is not the moat — or, more precisely, that the moat is the scale at which the engine runs, and no one can clone 155 trillion tokens a day from a GitHub repo.
The part nobody else will show you
No Western lab will ever do this. OpenAI, Anthropic, Google — the serving stacks behind their $20-a-month products and nine-figure enterprise contracts are the most closely guarded software in the industry. DeepSeek just opened the part that determines whether the unit economics work at all, a day after being declared a national-security threat by the country whose labs it is now undercutting.
The lab's own people are not being subtle about the framing. Senior researcher Chen Deli wrote on social media on Tuesday that open-source and closed-source AI represent "a decisive battle between two futures, two destinies for humanity," adding: "However long they want to fight, we'll fight. Until total victory." Cui Tianyi, who leads DeepSeek's Harness team, wrote that the company hopes more businesses and organisations worldwide will "deploy DeepSeek models and profit from doing so."
Strip the rhetoric and what is left is colossally rational. Training a frontier model costs hundreds of millions; serving one to hundreds of millions of users, cheaply, is where the actual business is. DeepSeek's entire identity is built on doing that part better than anyone — remember, this is the same lab that just put $400M on a months-old stealth robot startup, and it did not get there by being sentimental. Open-sourcing DeepInfer commoditises the serving layer, which hurts whoever charges the most for serving — the closed labs — while DeepSeek's own costs were already the lowest. This is not charity. It is a price war fought with a licence file.
Whether it works depends on the one thing DeepSeek cannot open-source: trust. Washington just told its agencies the company's technology may funnel sensitive data to Beijing. But for every developer outside the federal government, the calculus just changed. The engine is free. The benchmarks are public. The Huawei line is in the README.
FAQ: What is DeepInfer?
DeepInfer is the inference framework DeepSeek says powers its large-scale production services — the software that serves its DeepSeek-V4.1-Flash model to users. DeepSeek open-sourced it on GitHub on October 20, 2026, under the Apache 2.0 licence. It covers offline inference with the core features needed for serving at scale.
FAQ: What hardware does DeepInfer need?
Nvidia SM100 (Blackwell) GPUs or Huawei Ascend 950 NPUs, on Linux, with GCC 13 or later, Python 3.12, and CUDA 13 or CANN 9.2. DeepSeek describes the smallest efficient deployment as 64 GPUs for prefill plus 64 for decoding.
FAQ: Why is the Huawei support such a big deal?
Because it is a frontier lab putting Chinese accelerators on equal billing with Nvidia's latest chips in production inference code — the first time DeepSeek has done so. It does not prove the Ascend path is fast (no Ascend benchmarks are in the repo), but as a signal of where DeepSeek's hardware strategy is going, it is the most consequential line in the release.
FAQ: What's missing from the release?
A KV cache system, prefill/decode disaggregation across machines, and load balancing across replicas — the pieces that turn an engine into a live service. DeepSeek says they are tightly coupled to its internal infrastructure. Batch-invariant inference and deterministic sampling work on Nvidia hardware only.
FAQ: Are the performance numbers real?
They are DeepSeek's own claims and have not been independently verified: 155 trillion tokens per day, 96 per cent input cache hit rate, ~400ms time-to-first-token, 230 tokens per second output (300 for coding), and $38 per GPU per hour at off-peak pricing. The charts were measured on Nvidia SM100 hardware, with no comparison against vLLM or SGLang.
FAQ: Does DeepInfer run on Hopper or older GPUs?
No. The README supports NVIDIA SM100 (Blackwell) GPUs and Huawei Ascend 950 NPUs only; teams on Hopper or older hardware should keep using vLLM or SGLang for now.
FAQ: Will future DeepSeek models land in the same repo?
DeepSeek says inference code for future models will land in the DeepInfer repository. Missing pieces like KV cache, disaggregation and load balancing have no release date yet.
Sources: DeepSeek's DeepInfer GitHub repository and README (read October 21, 2026); DeepSeek's deployment Q&A and @deepseek_ai X announcement thread (October 20); SiliconANGLE, South China Morning Post, MarkTechPost, Crypto Briefing, PYMNTS, Forkast, AI News Daily, Pillitteri, madrobot.blog (all October 20, 2026); Hugging Face model card for deepseek-ai/DeepSeek-V4.1-Flash-1017. Production figures are DeepSeek's own claims and have not been independently tested.
By Daniel Rao
Daniel Rao covers AI, semiconductors and the US–China technology race for TechPulse USA.
Comments
Post a Comment