vibescoder

RDMA and a 500K Context Window Took the Spark Cluster From 32 to 61 Tokens a Second

·9 min read

My quest to use local AI hinges on recreating AI capabilities that are “near frontier”. What does that mean? I don’t entirely know. Subjectively it means fast streaming, larger context windows, and no looping or hallucinating.

That’s the experience I get with Sol or Opus. It’s not the experience I’m getting with DeepSeek. I often get in the 20s for tok/s and in longer agentic sessions I get stuck in tool call doom loops.

So, I set about another set of experiments to see if I can push towards the frontier.

Getting objective about what “near-frontier” actually means

As stated, I wanted the Spark cluster to serve coding agents at 40-50 tok/s. It was serving them at 32. I discovered two settings holding it back, and the tool-call loop I blamed on context size was a separate bug.

The cluster runs DeepSeek V4 Flash Vision-Exp across two DGX Sparks with tensor parallelism of two. It booted with NCCL over TCP sockets and a one-million-token context ceiling. Socket mode was the only way to get the model to start. The big context was the ceiling the checkpoint advertised. Neither setting matched the job I wanted. The context change moved single-stream decode from 32.8 to 34.3 tokens a second. RDMA did the rest.

The cluster had been running NCCL over TCP since the first boot

The earlier write-up ended with RDMA failing during distributed startup. NCCL tried to register memory for the RDMA path and got:

ibv_reg_mr_iova2 failed with error Cannot allocate memory
RuntimeError: NCCL error: unhandled system error

The fallback was NCCL_NET=Socket and NCCL_IB_DISABLE=1. That got the endpoint running. It also left the cluster talking over TCP on a link that could do far more.

The raw fabric check was the part I had not done. ib_write_bw over the CX7 RoCE device measured 12,995 megabytes a second, about 104 gigabits. TCP on the same interface measured roughly sixteen to nineteen gigabits. The cluster was using a fraction of the wire.

I also assumed the memory registration failure was a configuration problem. It was a kernel regression. NVIDIA’s DGX Spark forum thread on the NCCL RoCE regression documents that the 7.0.0-1019-nvidia kernel breaks NCCL’s RoCE path on the GB10 and that booting 6.17.0-1032-nvidia restores it. The older 6.17.0-1014-nvidia had no matching NVIDIA driver modules in the image, so I installed the kernel, driver modules, and nvidia-fs modules for 6.17.0-1032-nvidia on both nodes and set GRUB to that entry. Both nodes now boot 6.17.0-1032-nvidia with driver 580.178.04, and NCCL comes up over the RoCE device:

NCCL INFO NET/IB : Using [0]rocep1s0f1:1/RoCE [RO]
NCCL INFO Using network IB

The GPUDirect path is unavailable on this hardware. NVIDIA’s DGX Spark porting guide states that GB10 does not support GPU-direct RDMA, so NCCL registers host buffers instead of GPU memory. That is why the first attempt failed and why the bandwidth win still holds.

Cutting the context window made the KV pool practical and the fast path visible

The checkpoint advertises a 1M token ceiling. I had set MAX_MODEL_LEN to that value. Two things went wrong with it.

First, the prefix cache had to hold a much larger KV pool. At 500K the cache residency for real multi-turn coding sessions is more practical. Second, the serving stack logs the UE8M0 DeepGEMM path at 500K. The line that matters is:

Detected quantization_config.scale_fmt=ue8m0; enabling UE8M0 for DeepGEMM.

I did not test whether the 1M setting suppressed that path. I only confirmed it engages at 500K, and that the cluster keeps the fast quantized path with a 500K ceiling, which is more than any coding turn I run.

The context change alone was modest. On socket mode, single-stream decode went from 32.8 to 34.3 tokens a second. The concurrency case barely moved. That told me the context was a ceiling, not the bottleneck.

RDMA moved single-stream decode from 34 to 61 tokens a second

The measured matrix is the honest comparison. Every case used a unique cold prefix so no request reused another request’s cache, thinking was off, and output length was fixed at 128 tokens.

Case1M context, TCP500K context, TCP500K context, RDMA
Single stream, 256-token prompt, decode32.8 tok/s34.3 tok/s61.5 tok/s
Single stream, 256-token prompt, first token0.44 s0.50 s0.30 s
Single stream, 32K-token prompt, decode46.3 tok/s38.4 tok/s44.7 tok/s
Single stream, 32K-token prompt, first token32.7 s32.4 s18.3 s
Six streams, 4K-token prompt, per stream9.5 tok/s9.7 tok/s17.0 tok/s
Six streams, 4K-token prompt, aggregate25.5 tok/s27.0 tok/s45.6 tok/s
Six streams, 4K-token prompt, first token15.3 s14.5 s8.7 s

The single-stream lane is where the target landed. 61.5 tokens a second clears the forty to fifty range I set. The six-stream aggregate went from 27.0 to 45.6. The 32K prefill case cut its first-token wait from 32.4 seconds to 18.3 while its decode rate rose from 38.4 to 44.7.

One number I want to flag rather than smooth over. The 32K decode rate at 500K context on TCP was lower than the 1M baseline. The first-token time was essentially unchanged, so the drop was in generation, not prefill. I did not chase that down. It is a real measurement, not a clean win on every axis.

The tool-call doom loop was an encoder bug, not a context problem

I expected the smaller context to reduce the doom loop, the pattern where an agent keeps calling the same tool and the model returns an empty turn. It did not. I measured that directly before changing anything.

The DeepSeek checkpoint’s encoder appends a generation header only when the last message in the transcript is a user or developer turn. A transcript that ends on a closed assistant turn, or on a trailing reminder after one, ends on a bare end-of-sequence token. The model then produces an empty, no-op turn. That is the loop.

The fix is a backend hotfix, DSPARK_ENABLE_ASSISTANT_FINAL_HOTFIX=1, which patches the encoder and verifies itself on both ranks:

[OK] [assistant-final-hotfix] patched and verified

With it, empty turns went from 13 of 16 cases to zero of 32. The tool battery passed five of five, including single, required, multi-turn, parallel, and thinking-plus-tool cases. A probe run on the RDMA boot produced 15 cases with zero empty turns and no markup leakage.

I did not enable the DSML recovery flag. Thirty tool cases showed no leaked markup, so that flag would have added a recovery path with no evidence it was needed.

The throughput cost of the hotfix was not zero in every case. Single-stream decode stayed at 32.4 tokens a second against the 34.3 context-only baseline, which is within the run-to-run spread. The six-stream aggregate measured 22.0 against 27.0. Both runs were on the same TCP transport with the same context size, so the concurrency drop is a real comparison and not a cross-transport artifact. I am leaving it as an open question.

The router guard is a backstop, not the fix

The encoder hotfix handles the model side. The router handles the client side, because a client can still send a transcript that ends on a closed assistant turn.

I added a loop guard to Local Agent Router. It canonicalizes each assistant turn to a tool name plus sorted arguments. After three identical turns it appends a corrective nudge as a user turn. After five it returns an HTTP 409 with the repeated tool names and no backend call. A non-tool assistant turn resets the run.

The nudge has to be a user turn. A trailing system turn recreates the same dead state the encoder bug produces, and the router’s own text leaks into the model’s output. I also had to accept the latest_reminder role. The router rejected it before because its schema only allowed user, assistant, system, and tool. Accepting it matters because that role is what the encoder turns into the dead tail. Passing it through unchanged lets the backend hotfix handle it instead of the router rewriting the transcript.

The router’s Anthropic path now flattens tool-result content to a plain string. DeepSeek drops array content silently and re-calls the same tool, which is another loop source.

I verified both sides live. Five identical calls returned 409 without touching the backend. Three identical calls returned 200 with a clean answer and no nudge text in the output. The router suite passed 296 tests, and the change is deployed.

What I deliberately did not change

I left MAX_NUM_SEQS at six. There is no evidence that eight is safe at this context size, and the recipe documents no tested value. I left speculative decoding on. The measured loop cause was the encoder header, not the draft model. I left the DSML recovery flag off.

I also kept the policy engine. The Olla review already settled that question. The cluster is a fixed coding destination, and the router’s job is a short routing table plus a health check, not a fleet-discovery system.

By the Numbers

  • 61.5 tok/s single-stream decode at 500K context over RDMA, up from 34.3 on TCP
  • 45.6 tok/s aggregate across six streams, up from 27.0
  • 104 Gbps raw RoCE bandwidth from ib_write_bw, against 16 to 19 Gbps over TCP
  • 500000 token context ceiling reported by the endpoint, down from 1048576
  • 0 of 32 empty turns after the encoder hotfix, down from 13 of 16
  • 5 of 5 tool battery cases passing after the fix
  • 296 router tests passing with the loop guard
  • 3 identical tool calls before the router nudges, 5 before it returns 409
  • 1 kernel regression, 7.0.0-1019-nvidia, that broke NCCL’s RoCE path
  • 2 nodes rebooted onto 6.17.0-1032-nvidia with driver 580.178.04
  • 0 configuration changes to MAX_NUM_SEQS, speculative decoding, or the DSML recovery flag

Comments