vibescoder
cd ..

Getting DeepSeek V4 Flash with Vision Running on the Dual Spark Cluster

·9 min read
homelabaillmnvidiadebuggingbuilding-in-public

Yesterday’s plan said the Sparks would become the dedicated inference tier. Today’s post proved the cluster could boot DeepSeek V4 Flash and answer through an OpenAI-compatible endpoint.

That should have been the victory lap.

It lasted about five minutes. The text model worked, but the next obvious test was an image. My whole local AI thesis has been that home infrastructure should get more useful over time, not just cheaper. A local endpoint that can reason over text is useful. A local endpoint that can see screenshots, photos, dashboards, and weird Home Assistant states is much more useful.

So the next job was simple to describe and annoying to execute. Keep the two-Spark cluster. Keep DeepSeek. Add vision. Then point Hermes at it.

V4.1 Looked Like the Obvious Upgrade Until I Did Math

My first mistake was predictable. I saw DeepSeek V4.1 Flash and tried to make it the next target.

This is how homelab time disappears. You see a newer model exists, notice it’s just a simple point release, and your brain quietly rounds the hardware problem down to vibes. The Spark pair has 256GB of unified memory, which sounds huge until the checkpoint metadata tells you the thing you want is closer to 475 GiB upstream. Even the NVFP4 variants I found were still in the 400 GiB neighborhood.

That is not a tuning problem. That is needs-a-bigger-boat problem.

The lesson was the same one from the 5090 experiments, just with larger numbers. Local AI starts with memory math. If the weights do not fit with room for KV cache, routing, runtime overhead, and the operating system, the rest of the plan is fan fiction.

So V4.1 went back on the shelf. The point was not to run the newest DeepSeek model at any cost. The point was to turn the Spark pair into a reliable local inference appliance.

The First DeepSeek Win Was Text Only

The working endpoint from the first bring-up used DeepSeek V4 Flash, a Spark-specific B12X vLLM image, tensor parallel size two, and a deliberately conservative profile.

It was a good milestone. The cluster loaded roughly 149GB of weights on each node, answered 17*19 with 323, and wrote a merge_intervals function that passed the same style of test harness I use in the local agent bakeoffs. That was enough to prove the cluster was not just blinking lights and Docker logs.

But it had two problems.

First, I had forced the context way down to get a clean boot. The stable profile used a 4096 token context, one sequence, small batch limits, eager execution, and FP8 indexer cache. That was the right get-it-running posture. It was not the reason I bought two Sparks.

Second, it did not solve vision. The model serving path was text-first. I could imagine bolting another vision model beside it, but that would miss the point. I wanted one local model that Hermes could use for text and image input through the same API.

That led to DeepSeek V4 Flash Vision-Exp.

SGLang Was the Plausible Wrong Road

The recommendation I started with said to try SGLang first for DSpark support. That sounded reasonable. SGLang moves fast, DeepSeek V4 is not a boring model, and Spark support is young enough that a development image can matter more than the stable tag.

The generic SGLang image failed on a missing vision aligner key:

KeyError: 'aligner.gate_up_proj.weight'

The DGX-Spark-specific development image got farther, then the OS killed it during startup. I tried conservative context. I tried smaller context. Same shape. Then I found an open SGLang issue describing Vision-Exp OOM failures on a two-Spark cluster.

This is the part I want to get better at. I kept retrying before asking the better question. Has anyone already loaded this exact model on this exact hardware?

The Community Recipe Was the Breakthrough

The evidence pointed to the MiaAI-Lab DSpark recipe. It was not just a README with happy-path commands. It had a real issue tracker, specific Vision-Exp patches, and merged work for native Vision-Exp support. Another fix covered image token routing.

That mattered. The Spark ecosystem is still small enough that community recipes are not garnish. They are often the productized path before the productized path exists.

I cloned the recipe, copied it to the head Spark, configured the worker over the private cluster link, pinned the Anemll DSpark vLLM image, and pointed both nodes at the cached Hugging Face model. The first launch failed before the model even loaded:

FATAL: /usr/local/lib/python3.12/dist-packages/vllm/tokenizers/deepseek_v4_encoding.py drift:no-image-placeholder

That error looked scary. It was also precise.

The Vision-Exp hotfix expected the tokenizer encoder file to contain an image placeholder token. The copied encoder did not. At first I wondered if the Hugging Face revision had drifted. The recipe mentioned one tested revision, while the current cache pointed at a newer one. I downloaded both encoder files directly and compared them. Same size. Same hash. Same placeholder.

The model was fine. My cache was not.

The snapshot on both Sparks had the weight shards and inference files, but it was missing encoding/encoding_dsv4.py. The launcher then fell through to the wrong encoder path. That matched the community’s own placeholder issue, even though their report involved a text-only checkpoint.

The fix was not clever:

./prepare-dspark-model-cache.sh --official --yes

After that, both nodes had the Vision-Exp encoder, the placeholder existed, and the hotfix applied cleanly.

RDMA Failed After the Model Finally Started Loading

Getting past the encoder revealed the next bug. The model reached distributed startup and then NCCL failed while registering memory for RDMA:

ibv_reg_mr_iova2 failed with error Cannot allocate memory
RuntimeError: NCCL error: unhandled system error

I tried the obvious community knobs first. Lower GPU memory utilization. Disable GPU-initiated networking. Disable DMA-BUF probing. Restrict NCCL to one CX7 HCA instead of both. None of that got a clean boot.

The practical fallback was to stop using RDMA for NCCL and run over TCP sockets on the CX7 interface:

NCCL_NET=Socket
NCCL_IB_DISABLE=1

That is not the forever answer. RDMA should be faster, and I still want to understand the memory registration failure. But the goal was a working Vision-Exp endpoint, not a perfect networking paper. Socket mode over the private Spark link got the cluster through startup.

This is another place where my mental model was off. I treated the CX7 network as one thing. In practice, the physical link can be up, private IP can work, SSH can work, TCP can work, and NCCL over RDMA can still fail.

The Endpoint Became Real When Hermes Used It

Once the server finished loading, the health endpoint returned 200 and /v1/models reported:

{
  "id": "deepseek-v4-flash-vision-exp",
  "max_model_len": 1048576
}

The smoke tests stayed intentionally simple. Text first:

17*19 -> 323

Then the same small coding task. Then a real image request using the Hugging Face example carrots image. Vision-Exp answered:

The image shows carrots.

That is not a benchmark. It is a heartbeat with eyes.

The last step was durability. I published a .local name for the endpoint with Avahi, restricted mDNS to the home LAN interface, and added a small systemd health supervisor. Docker restarts the containers, but distributed systems love half-alive states. The supervisor checks /health after boot and every few minutes. If the endpoint is stale, it stops both ranks and starts the two-node recipe again.

Then I rebooted both Sparks. The endpoint came back. The mDNS name resolved from another home-network machine. Hermes switched from the older local llama.cpp model to the new Spark endpoint and answered the same arithmetic smoke test correctly. The Sparks are no longer a project on the bench. They are the local model endpoint for the house.

The Spark Ecosystem Rewards Exactness More Than Confidence

I like the result. I also like that the path there made me wrong in several different ways.

I wanted V4.1 before checking the memory math. Wrong.

I treated the first working V4 endpoint as if the model selection problem was mostly solved. Wrong. It lacked vision, which matters for where this homelab is going.

I kept retrying SGLang before looking hard enough for proof that this exact model had loaded on dual Spark hardware. Wrong.

I saw an encoder hotfix failure and initially suspected upstream revision drift. Wrong. The cache was incomplete.

I assumed the CX7 link working meant NCCL over RDMA would work. Wrong again.

That is the lesson from this phase of the build. Spark is not impossible. It is also not appliance-simple yet. You need the official docs, the model cards, the container runtime, and the community issue tracker open at the same time. The path exists, but it is narrow.

For me, that is still worth it. The 5090 rig proved local AI could be part of daily work. The Spark pair now proves the next tier can run a higher-fidelity multimodal model and serve it to the tools I already use. It is not finished. It is finally useful in the way I wanted.

The next question is no longer whether the cluster can run a serious model. It can. The next question is whether a local multimodal model changes what I ask my agents to do.

By the Numbers

  • 2 DGX Sparks running one distributed Vision-Exp endpoint
  • 1 oversized model rejected before startup because V4.1 Flash did not fit the two-Spark memory pool
  • 167811372792 bytes reported in the Vision-Exp checkpoint index
  • 48 safetensor shards cached for Vision-Exp on each Spark
  • 36707 bytes in the required Vision-Exp encoding_dsv4.py file
  • 1048576 token max model length reported by the working Vision-Exp endpoint
  • 0.78 GPU memory utilization used for the stable Vision-Exp boot
  • 313 image tokens in the carrots smoke test
  • 3 runtime paths tried before landing on the working community recipe
  • 1 mDNS endpoint name now used by Hermes instead of a hard-coded IP

Comments