vibescoder
cd ..

Adding A Dual-Node NVIDIA Spark Cluster for Real Local AI in the Homelab

·9 min read
homelabaillmnvidiahow-tobuilding-in-public

This blog is my journey in local AI. I believe this will dominate the AI conversation in 2027 and I wanted to start at homelab-scale to build a baseline. I started with a single RTX 5090 workstation because converting my gaming rig seemed logical. Could one serious desktop run useful local models and answer real agent calls?

It could. The 5090 setup proved that local inference was not just a benchmark hobby. Then the limits showed up. A 32GB GPU is a lot until the model you want needs five times that. The next question was not whether local AI was useful. It was what happens when the local box is built for higher-fidelity models instead of squeezed into a gaming GPU.

So this was the next step: two DGX Sparks, one QSFP link, one large model, and a practical question. How hard is it to make this thing serve DeepSeek V4 Flash?

My answer after getting it working: 6 out of 10. Not turnkey. Not magic. Not a weekend-destroying nightmare either. You need to be comfortable reading docs, checking network interfaces, restarting containers, and believing error messages more than vibes. Conveniently, that is most of this blog.

The RTX 5090 Taught Me Where the Next Limit Was

The 5090 rig, AI-NT-No-Problem, is still a great local AI workstation. It runs the agent harnesses, hosts Coder, and has enough GPU for the 27B to 35B class of models that make local coding agents useful. That is the first tier of the local AI thesis.

The Spark pair is the next tier. It is not replacing the 5090 because the 5090 failed. It is taking over the job that the 5090 should not have to do forever: hold larger models, serve multiple agentic workloads, and become a dedicated inference tier for the house.

That matters for the larger homelab redesign. PAIR still matters, because I want client machines to keep calling a stable local endpoint while compute moves around behind them. But PAIR was not the first thing to prove. First I needed the raw serving layer to work.

DeepSeek V4 Flash was the right first target. It is large enough to leave the 5090 behind, small enough to plausibly fit across two Sparks, and interesting enough for agentic work that success would mean something.

The First Win Was Boring Networking

The two Sparks came through initial setup cleanly. Both were running Ubuntu 24.04.5 on GB10 hardware with NVIDIA driver 580.173.02, CUDA 13.0, Docker, and roughly 3.7TB of NVMe storage. Once I had the two Spark hostnames on the home network, I added a private high-speed cluster network over ConnectX-7.

The first check was ibdev2netdev.

ibdev2netdev

At first, the QSFP interfaces were down. The kernel logs showed the obvious thing: cable unplugged. The cable was there, but the link was not. After reseating it, both machines showed the same two active interfaces:

rocep1s0f1 port 1 ==> enp1s0f1np1 (Up)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Up)

That was the first useful lesson. If the link is not up, stop debugging vLLM. Fix the cable.

From there I configured persistent Netplan files for the private CX7 network. The real addresses are not important. The pattern is:

network:
  version: 2
  ethernets:
    enp1s0f1np1:
      addresses:
        - 192.168.x.10/24
      dhcp4: no
      dhcp6: no
    enP2p1s0f1np1:
      addresses:
        - 192.168.y.10/24
      dhcp4: no
      dhcp6: no

Node two gets matching addresses in the same private ranges. After netplan apply, both CX7 links could ping both directions. I added hostname aliases and set up bidirectional SSH between the Sparks. That gave me the foundation for distributed serving.

None of this was hard. It was just exact. The kind of exact where one loose cable makes the entire model-serving story irrelevant.

The Generic vLLM Path Was Close Enough to Waste Time

My first serving attempt followed NVIDIA’s multi-node vLLM path. Pull the NGC vLLM image, download the Ray cluster script, start a head node, start a worker node, and verify Ray sees two GPUs.

That part worked. Ray reported two nodes, two GPUs, and a real cluster. Then a tiny smoke test wedged the head node badly enough that SSH accepted TCP but never sent a banner. Not my favorite failure mode.

The mistake was treating Spark like a generic CUDA box. It is Linux. It is CUDA. It is also GB10, and DeepSeek V4 on GB10 has a more specific path than the generic vLLM playbook.

The live vLLM recipe page had the key detail: DeepSeek V4 on DGX Spark needs the Spark-specific B12X build, not stock vLLM.

docker pull eugr/spark-vllm-b12x:latest

That is the second useful lesson. The general docs get you to the neighborhood. The model recipe gets you to the address. For Spark, the difference matters.

DeepSeek V4.1 Was the Wrong Distraction

I briefly looked at DeepSeek V4.1 Flash because of course I did. It was newer, shinier, and exactly the sort of thing that makes a homelab owner forget the point of the day.

The metadata brought me back to earth. DeepSeek V4 Flash is a 148.66 GiB checkpoint. DeepSeek V4.1 Flash is in a different size class. The upstream checkpoint is roughly 475 GiB, and the NVFP4 variants I found were still around 400 GiB. That is not a two-Spark 256GB target in the same way.

So I went back to deepseek-ai/DeepSeek-V4-Flash. That was the right call. The day’s goal was not to chase the newest model into a wall. The goal was to prove the Spark pair could serve a serious higher-fidelity model at all.

The Model Fit After the Defaults Stopped Fighting the Hardware

The working launch used one container on each Spark. The worker starts first. The head starts second. Both point at the head’s CX7 address, use tensor parallel size 2, and mount the local Hugging Face cache.

The first B12X attempt got farther than the generic Ray path, then failed on memory. vLLM saw about 111.35 GiB free on the device and --gpu-memory-utilization 0.92 wanted about 111.95 GiB. I missed by roughly 0.6 GiB. That is annoying, but it is a good kind of annoying. The error told me exactly what to change.

The second failure was more interesting. I had enabled FP4 indexer cache. The model tried to use mxfp4 for the indexer KV path and threw this:

indexer_kv_dtype='mxfp4' requires Blackwell datacenter GPUs
(sm_10x, e.g. B200/GB200); sm_120 and earlier architectures are not supported.

GB10 reports as SM120. So the Spark path needed FP8 indexer cache, not the datacenter Blackwell FP4 indexer path.

The conservative working settings were:

--gpu-memory-utilization 0.80 \
--max-model-len 4096 \
--max-num-seqs 1 \
--max-num-batched-tokens 256 \
--enforce-eager \
--attention_config.use_fp4_indexer_cache False \
--moe-backend auto \
--tensor-parallel-size 2 \
--nnodes 2

That is not the final tuning. It is the get-it-running configuration. Low context. One sequence. Eager mode. No CUDA graphs. The point was to prove the model loads and answers before optimizing anything.

Once the full 148.66 GiB checkpoint was cached on both Sparks, the successful run loaded the model in about two minutes per node, compiled the needed kernels, created the KV cache, and started the OpenAI-compatible server.

Application startup complete.
GET /health HTTP/1.1 200 OK

The First Endpoint Was Conservative and Real

The server came up on the head Spark and reported the model through /v1/models:

{
  "id": "deepseek-ai/DeepSeek-V4-Flash",
  "max_model_len": 4096
}

The first test was intentionally stupid:

What is 17*19? Return only the final integer.

It returned 323.

That only proves the endpoint is alive, so I gave it a small coding task next. Write a Python function called merge_intervals(intervals) that handles half-open intervals, ignores empty or reversed ranges, merges touching intervals, returns sorted tuples, and does not mutate the caller’s input.

The generated solution passed a local test harness covering overlap, touching intervals, invalid intervals, tuples, generators, and mutation safety. That is the first result that felt useful. Not because merging intervals is impressive, but because the Spark-hosted model produced code that passed tests through the same kind of loop I care about for agents.

There are caveats. Even with thinking: false, the response still included internal reasoning tokens. The tiny coding task took around 68 seconds. The context window is capped at 4096 tokens for this first stable run, nowhere near the model’s 1M-token headline. PAIR is not in the path yet. This is a working endpoint, not the finished homelab brain.

The Next Step Is Choosing What Deserves the Spark Pool

This first Spark bring-up answered one question and opened several better ones.

Can two DGX Sparks serve DeepSeek V4 Flash locally? Yes.

Is this the production config? No.

The next work is tuning and selection. I need to test whether eager mode is necessary or just safe. I need to raise the context limit gradually and watch memory. I need to measure prompt processing and decode separately. I need to test concurrency instead of one polite request at a time. And I need to research the best model that actually fits the 256GB pool, because V4.1 already proved that newer does not automatically mean practical.

That is the right place for this first step to end. The 5090 rig proved local AI could matter in daily work. The Spark pair is the next experiment: can local AI move into larger, higher-fidelity model territory without turning my house into a miniature data center?

The first answer is yes, with footnotes. The next question determines the best model to run.

By the Numbers

  • 2 DGX Sparks in the first dedicated inference cluster
  • 2 active CX7 interfaces on each Spark after reseating the QSFP cable
  • 4 private CX7 addresses assigned across the pair
  • 580.173.02 NVIDIA driver version reported on both nodes
  • 13.0 CUDA version reported by nvidia-smi
  • 148.66 GiB DeepSeek V4 Flash checkpoint size
  • 149G model cache retained on each Spark after the first download
  • 4096 token context limit for the first stable run
  • 0.80 GPU memory utilization that loaded safely
  • 323 first correct arithmetic response from the endpoint
  • 1 simple coding task passed by the live Spark-hosted model
  • 6/10 setup difficulty, mostly from networking exactness, image choice, and hardware-specific vLLM flags

Comments