I Ran DeepSeek V4 Flash 0731 Locally with 48 GB of RAM
I set myself a slightly unreasonable goal: run the full official DeepSeek-V4-Flash-0731 model on my desktop, even though the machine has only 48 GB of system RAM and the checkpoint alone is about 155 GiB.
Calling this computer a "potato" needs an immediate qualification. It has two NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPUs with 96 GB of VRAM each, plus two fast 4 TB PCIe 5.0 NVMe SSDs. Relative to an AI data center, it is a small local machine. Relative to a normal desktop, it is obviously very powerful.
The potato part was system memory. Linux could see about 45 GiB of usable RAM, while the first model load and MoE conversion wanted far more. My first milestone was serving a 32,768-token context through an OpenAI-compatible API with tensor parallelism across both GPUs and DSpark speculative decoding. I later pushed the experimentally verified baseline to 400,000 tokens and completed two exact-boundary requests concurrently.
This post is the story of how I reached those points. The exact setup, scripts, environment variables, safety checks, SSD provisioning, serving options, and benchmark commands live in the public lunarmoon26/deepseek-v4-flash repository.
The local AI machine
My starting hardware looked promising from a GPU-capacity perspective and deeply uncomfortable from a host-memory perspective:
- CPU: AMD Ryzen 7 9700X, 8 cores / 16 threads, 65 W default TDP
- Motherboard: MAXSUN MS-iCraft B850 AIGA
- System RAM: 48 GB installed, approximately 45 GiB usable
- GPUs: 2 x NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB each
- GPU topology: PCIe under a
PHB, with no NVLink - GPU power limit: 300 W per card
- System SSD: 2 TB Crucial P310 PCIe 4.0 x4 NVMe
- Model/cache SSDs: 2 x 4 TB WD_BLACK SN8100 PCIe 5.0 x4 NVMe
The MAXSUN board is surprisingly well suited to a personal local-AI workstation. With my Ryzen 9000 CPU, the two graphics slots run at PCIe 5.0 x8/x8, and two M.2 slots each get a processor-direct PCIe 5.0 x4 connection. The remaining M.2 slots and lower-priority I/O sit behind the B850 chipset. That gives both GPUs and both model-cache SSDs direct CPU lanes without requiring a workstation or server platform.
The SN8100 drives are rated for sequential reads up to 14,900 MB/s, writes up to 14,000 MB/s, and 2,400 TBW of endurance. Those are manufacturer peak figures rather than measurements from my run, but this workload gave them a better purpose than shaving another second from a game-loading screen.
Low power was also part of the design. The 9700X has a 65 W default TDP and is one of the more efficiency-focused desktop CPUs in the Ryzen 9000 generation. Each RTX PRO 6000 has a 600 W stock total-board-power rating, but I followed Lambda's nvidia-smi power-limit procedure and capped each card at 300 W.
During the first 32K inference run, nvidia-smi showed roughly 170 W per GPU. The later exact-400K test held both cards at their 300 W caps and 100% utilization. That difference shows how strongly prompt geometry changes this system's power demand. These are GPU board-power readings rather than whole-machine wall power, and I have not yet run a controlled capped-versus-uncapped comparison.
Why I thought it might fit
The two GPUs provided roughly 191 GiB of aggregate VRAM. On paper, that looked plausible for a roughly 155 GiB mixed-precision checkpoint, although it left limited room for runtime workspaces and KV cache. The published recipe used eight RTX PRO 6000 cards, so my TP=2 setup was an experiment rather than a smaller copy of a supported deployment.
The checkpoint choice created the first detour. The preview NVIDIA NVFP4 release, a community 0731 conversion, and the official 0731 release were easy to treat as versions of the same thing. They were not. They used different weights and speculative-decoding paths.
I first downloaded the community MJPansa/DeepSeek-V4-Flash-0731-NVFP4 conversion because its model card reported a TP=2 load on two DGX Spark systems. Later, I switched to the official DeepSeek-V4-Flash-0731 release, which uses DSpark. That meant downloading another complete checkpoint, but it removed any ambiguity about what I was serving. The final official model occupied 48 safetensors shards and approximately 155.43 GiB on disk.
When 100% GPU utilization meant zero progress
My first serious startup attempt initialized two NCCL ranks and then appeared to work. Both GPUs showed 100% utilization, so I assumed Blackwell kernels were compiling or autotuning.
I spent time changing warm-up settings, disabling autotuners, forcing eager execution, and reducing batch limits. Nothing changed. Each GPU still held only about 1 GB of VRAM, PCIe traffic stayed near zero, no model shards entered VRAM, and no kernel cache grew.
A community SM120 runbook described the exact signature. The workers were deadlocked in NCCL's first PCIe P2P collective. They had never reached model loading or warm-up.
The initial fix disabled NCCL P2P, vLLM's PCIe all-reduce path, and its custom all-reduce implementation. The important lesson was that 100% utilization only proved the GPUs were busy. It did not prove they were making useful progress.
Stock vLLM reached the Blackwell boundary
Fixing communication moved the startup far enough to reveal the next problem: stock vLLM reached an SM120 sparse-MLA compatibility error. Other reports also documented incomplete workstation Blackwell support around mHC, DeepGEMM, block-scaled GEMM, and sparse MLA.
I considered several community forks, patched Triton fallbacks, SparkInfer configurations, and llama.cpp with a separate drafter. I eventually chose vLLM-Moet because it kept the vLLM-compatible API, shipped SM120 kernels, supported the RTX PRO 6000 explicitly, and included a DSpark backport.
This was more than putting stock vLLM in Docker. vLLM-Moet replaced unsupported execution paths and used a specialized reduced-precision routed-expert design. It was the first approach that matched both my GPUs and the official 0731 model closely enough to continue.
Fixing NCCL made the desktop disappear
Once the communication deadlock was gone, the workers finally began loading and converting the model. The next terminal failure ended with rank-1 TCPStore ... Broken pipe messages, which looked like another distributed-systems problem.
The kernel log told a different story. Linux had exhausted all physical RAM and the original 8 GiB swap file, invoked the OOM killer, and killed rank 0. The later TCPStore errors were only the aftermath.
On one attempt, the mouse froze and the graphical session disappeared into a black screen with a blinking caret. My model experiment had taken down the desktop instead of failing cleanly.
I first put the process in a memory cgroup with 28 GB of RAM and 2 GB of swap. It failed safely, which protected the graphical session, but the allowance was far too small to complete conversion. That was still useful progress: I now had separate problems to solve. The model needed much more virtual memory, and Linux needed enough physical memory reserved to remain a usable desktop.
The SSDs changed jobs
I had originally imagined the two SN8100 drives as KV-cache offload devices. The host OOM made their real purpose clear. They became virtual-memory and persistent-MoE infrastructure.
I gave each drive a 129 GiB equal-priority swap partition and formatted the remaining space as ext4. Together with the existing swap, the machine exposed approximately 265 GiB. One SSD held the generated MoE plane cache, while the other held the disk-backed expert store used during inference.
The first conversion eventually produced about 203 GiB of MoE planes and 130 GiB of TP=2 expert packs. Persisting those artifacts mattered as much as adding swap. An interrupted run could resume completed layers, and later starts did not need to repeat the entire conversion.
The final Docker cgroup allowed the model 34 GB of physical RAM and 200 GB of combined RAM plus swap. That left roughly 11 GiB of physical memory outside the container for Linux, the desktop, filesystem cache, and GPU drivers. If the model crossed the hard limit, Docker could kill the container before Linux sacrificed the graphical session again.
SSD capacity versus more RAM
Price shaped this decision. I was lucky enough to buy each 4 TB SN8100 for a little over £400. While writing this, PriceSpy listed the same drive at £665. Some 128 GB 2 x 64 GB DDR5 kits were also well above £1,200, including a Crucial kit at £1,299 and a Corsair kit at £1,499.99.
For this exact layout, two 2 TB PCIe 5.0 SSDs would probably have been enough. The two swap partitions, plane cache, expert store, and later rank-0 copy consumed less than 700 GiB across the pair. That estimate assumes the 156 GB checkpoint remains on a separate system SSD and that old cache generations are removed.
SSD capacity is not equivalent to RAM. Heavy swapping made the first conversion painfully slow, and even 128 GB of installed RAM would still have been below the approximately 220 GiB combined memory-and-swap demand I observed. The SSDs gave me an affordable bootstrap path and permanent model storage. More RAM would make that path much faster.
The long conversion and one last misleading error
The normal safetensors loader eventually read all 48 shards. An eager-loading shortcut failed on the checkpoint's F8_E8M0 scale type, so I returned to the slower streamed and memory-mapped path.
Conversion then built 43 main-model layers under heavy swap pressure. It was slow, but the output persisted. Watching completed layers survive retries was the point where this stopped feeling like an endless startup failure and started feeling recoverable.
After the expensive load finally completed, vLLM reported about 70.66 GiB of model memory per GPU and failed during sparse-MLA warm-up with AssertionError: No free indices.
I initially read that as another VRAM limit. The actual problem was much smaller: I had allowed only one request-state slot, while the warm-up routine needed one decode request and one prefill request at the same time. Raising the limit from one to two fixed the error without changing GPU memory allocation.
The server finally started
After the NCCL workaround, the switch to vLLM-Moet, the memory redesign, the long conversion, and the request-slot fix, the service finally reached Application startup complete.
The first working server used both GPUs, a 32,768-token context limit, FP8 MLA KV cache, and the official checkpoint's seven-token DSpark draft path. I exposed it as deepseek-v4-flash through a local OpenAI-compatible API.
There was still one storage imbalance. Both tensor-parallel rank files initially lived on SSD2, while SSD1 was mostly idle after startup. Because vLLM-Moet generated a separate expert pack for each rank, I copied rank 0 to SSD1 and used Docker file overlays so rank 0 read from one drive and rank 1 from the other. The repository contains the verified migration and rollback logic; the useful story-level point is that the supposedly single expert store already had a natural per-rank boundary I could exploit.
What "the full model" means
The service runs the full official DeepSeek-V4-Flash-0731 release. It is not a distilled model, GGUF conversion, or unofficial derivative.
Model identity and execution precision are separate questions, though. The official checkpoint combines FP8 dense weights, FP4 routed experts, and BF16 tensors. vLLM-Moet compresses the routed experts further into a 2-bit base representation, backed by a 24 GiB FP4 recovery tier for hot experts and low-confidence steps. The KV cache uses FP8.
That mixed-precision path is central to making the model fit. Calling the result "full FP8" would be inaccurate, and the reduced-precision expert path still deserves more quality testing.
The first successful 32K results
My smoke test asked:
What is 17 times 19? Return only the integer.
The model returned 323. This was not a meaningful intelligence test, but it proved that the complete service could accept a request and produce a response after everything it had taken to start.
I then ran a small 32K serving benchmark: three prompts at concurrency 1, each with 30,720 input tokens and 512 output tokens. All three completed with no failures in 48.10 seconds.
| Metric | Result |
|---|---|
| Aggregate output throughput | 31.93 tokens/s |
| Mean time to first token | 9.785 s |
| Mean time per output token | 12.226 ms |
| Mean end-to-end latency | 16.032 s |
| DSpark speculative-token acceptance | 72.32% |
This was only one short run with no benchmark warm-up requests. It was an initial measurement rather than the machine's final performance envelope.
August 11 update: from 32K to 400K
I continued tuning after that first success. By August 11, the experimentally verified launcher had moved from a 32,768-token experiment to a 400,000-token total context, still with two request slots and the same 300 W GPU power caps.
The 400K limit includes both the prompt and generated output. A request with 399,872 input tokens and 128 generated tokens is exactly at the boundary; a 400K prompt followed by additional output is not.
Re-enabling PCIe P2P
Disabling NCCL P2P was the correct fix for the original startup deadlock, but I did not want to preserve that workaround without retesting the hardware. After disabling IOMMU in the BIOS, both GPUs passed peer read and write checks across their shared PCIe PHB topology, and NCCL 2.28.9 reported P2P/CUMEM transport.
In a local 256 MiB two-rank all-reduce test, bandwidth improved from 13.86 GiB/s with P2P disabled to 19.91 GiB/s with P2P enabled at the PHB level. The full 400K server then completed startup and its smoke test. The current launcher therefore enables NCCL P2P while keeping vLLM's custom PCIe all-reduce disabled, with the original no-P2P path retained as a rollback.
Disabling IOMMU weakens DMA isolation and can affect virtualization or device passthrough. I accepted that machine-specific tradeoff for this dedicated local inference setup; it is not a general recommendation for every workstation.
Finding the new baseline
The current serving geometry uses two request slots, 4,096 maximum batched tokens, DSpark-7, a 24 GiB FP4 recovery pool per GPU, and the DeepSeek-specific FP8 MLA KV cache. Startup reported 13.57 GiB available for GPU KV cache and capacity for 1,765,220 KV tokens. That left enough headroom for two 400K requests, even though the theoretical capacity number was higher than the concurrency I intended to serve.
I compared batch sizes with a synthetic sweep of four 32K-prompt, 512-output requests at concurrency 2. Each setting had one un-warmed run and used different random prompt content, so these are directional tuning results rather than a definitive ranking:
| Maximum batched tokens | Aggregate throughput | Output throughput | Median TTFT |
|---|---|---|---|
| 1,024 | 1,102.65 tokens/s | 16.96 tokens/s | 38.84 s |
| 2,048 | 1,218.44 tokens/s | 18.75 tokens/s | 31.07 s |
| 4,096 | 1,738.74 tokens/s | 26.75 tokens/s | 23.02 s |
The 4,096-token setting produced the best observed result. In the same directional sweep, DSpark-7 reached 1,738.74 aggregate tokens/s, compared with 1,669.46 for DSpark-5 and 1,675.98 for DSpark-3. A larger 28 GiB recovery pool improved observed expert-recovery coverage, but reduced KV capacity to 1,245,285 tokens and lowered aggregate throughput to 1,509.51 tokens/s. I kept the 24 GiB pool because it preserved the better measured performance and enough KV room for two full contexts.
I also made vLLM and FlashInfer's compilation and autotuning cache persistent. Compatible restarts can now reuse tuned kernels instead of replaying all of that work inside a disposable container.
Testing the exact 400K boundary
The final capacity test sent two independent requests concurrently. Each contained 399,872 input tokens and generated 128 more, landing exactly at 400,000 tokens per request. Both completed without failures.
| Measurement | Result |
|---|---|
| Concurrent requests | 2 |
| Benchmark duration | 425.61 s |
| Aggregate input + output throughput | 1,879.65 tokens/s |
| Aggregate generated-token throughput | 0.6015 tokens/s |
| Median time to first token | 275.793 s |
| Container resident memory | Below 7 GiB |
The aggregate throughput is dominated by processing the two enormous prompts. A roughly 276-second median wait for the first token and 0.60 generated tokens/s make this a capacity demonstration, not an interactive fresh-400K chat experience.
It also proves execution capacity rather than long-context understanding. I still need deterministic retrieval tests at 128K, 256K, and 400K before claiming the full window is useful for information buried anywhere in the prompt. NVMe-backed reusable prefixes and true prefill/decode disaggregation remain gated experiments in the project's roadmap, not features of the current baseline.
What I learned
Several failures looked convincing while pointing in the wrong direction. GPU utilization looked like compilation but was a communication deadlock. TCPStore errors looked like networking but followed a host OOM. No free indices looked like VRAM exhaustion but came from one missing request slot.
The desktop also needed protection from the experiment. Large swap made conversion possible, while the cgroup made failure survivable. I needed both. Swap alone would happily let the model make the entire machine miserable.
Persistent intermediate work changed the economics of retrying. Once completed MoE layers, expert packs, and tuned kernels survived a failed startup, each attempt moved forward rather than replaying hours of conversion.
Most of all, serving a model this large was a whole-machine systems problem. GPU capacity was only the opening condition. PCIe topology, firmware, kernel support, host memory, swap behavior, filesystems, container limits, precision, and storage placement determined whether the theoretical fit became a working service.
Reproducing the implementation
The implementation details are intentionally kept out of this recap. The DeepSeek V4 Flash local service repository contains:
- the official checkpoint downloader and pinned model revision;
- the vLLM-Moet build and 400K serving launcher;
- the NCCL P2P validation and rollback settings;
- the host-managed 300 W GPU power-limit service;
- guarded NVMe provisioning and desktop-memory limits;
- persistent MoE and kernel-cache configuration;
- the per-rank SSD split and validation checks;
- smoke-test, benchmark, and long-context roadmap artifacts.
There was no single clever flag that made this work. Every successful step uncovered the next constraint, and the final setup substitutes GPU VRAM, NVMe capacity, persistent conversion artifacts, and strict memory isolation for the large host-RAM environment this load path normally expects.
It also does not prove that DeepSeek V4 Flash will run comfortably on an ordinary 48 GB PC. This experiment still depends on two 96 GB Blackwell GPUs, hundreds of GiB of NVMe-backed swap, fast SSDs, and a specialized reduced-precision expert path. Within those limits, my "potato" moved from answering 323 to completing two exact 400K-context requests at once.