Friday Fixes: Plugging the Spark

The Spark cluster finally crossed the line from project to infrastructure this week.

A working model endpoint is nice. A household model endpoint is different. It needs a stable name. It needs to come back after reboot. It needs the agents to use it without knowing which machine is doing the work. It needs enough benchmark data that I can tell whether the model is good at the job, not just alive.

That was this week’s work. Less glamorous than the first successful 323 response. More important.

The Endpoint Needed a Name That Was Not an IP Address

The first version of the Spark endpoint worked through a LAN address. That is fine for a bring-up session. It is not how I want a home full of agents to depend on something.

The head Spark can change IP after a reboot. A future router change could move the subnet. A human should not have to remember which Spark is the head node, which port vLLM is on, or whether a private cluster fabric address leaked into a client config.

So I gave the endpoint a name:

deepseek-spark.local

The head Spark publishes that name over mDNS with Avahi. I restricted the Avahi config to the home LAN interface so clients see the normal home-network address, not the private CX7 fabric addresses between the Sparks. The model stays bound on the head. The clients call the name.

That one change makes the endpoint feel like a real appliance. Hermes, a laptop, a test script, and a dashboard can all point to the same URL:

http://deepseek-spark.local:8888/v1

The gotcha was testing. This Coder workspace runs inside a container that is not always in the home LAN multicast domain, so .local resolution can fail from here even when it works from a real home-network machine. The homelab host resolved it correctly. That is the test that mattered.

Reboot Recovery Needed More Than Docker Restart Policies

Docker can restart containers after reboot. That is necessary. It is not sufficient for a two-node distributed model.

A single-node service is usually either up or down. A two-node inference pair has more entertaining failure modes. The head can be healthy while the worker is stale. The worker can restart before the head is ready. The HTTP port can be closed while both containers technically exist. Distributed systems enjoy making boolean health checks feel naive.

The fix was a tiny systemd health supervisor on the Spark head.

It checks /health after boot and every five minutes. If the endpoint is healthy, it exits. If the endpoint is stale, it stops both ranks and relaunches the two-node DSpark stack through the community launcher. Docker still has unless-stopped on both containers, but the health supervisor handles the half-alive cases Docker cannot reason about.

Then I rebooted both Sparks.

Both nodes came back. The containers came back. The mDNS name resolved from another home-network machine. The endpoint returned 200. That is the difference between “I got it running” and “I can depend on it tomorrow morning.”

Hermes Had to Stop Pointing at Yesterday’s Model

Once the Spark endpoint survived a reboot, the next obvious problem was Hermes.

Hermes still pointed at the older local model endpoint. That meant the dual Spark cluster was technically working but not carrying the actual household agent workload. Classic homelab problem. The new thing is live, but the old thing is still in every config that matters.

I added a named Hermes custom provider:

model: deepseek-v4-flash-vision-exp
base_url: http://deepseek-spark.local:8888/v1
api_mode: chat_completions
context_length: 1048576

I marked it as vision-capable, set it as the default, and restarted the Hermes gateway. A one-shot Hermes smoke test answered 17*19 with 323.

That put DeepSeek into the real path. Not just curl. Not just a benchmark script. The agent now uses the Spark endpoint.

That also clarified one boundary. The model does not have native web search. It can call tools, but it does not browse by itself. Hermes can use Brave, Firecrawl, and other search tools. Those tools cost money. The model endpoint is local. The web is not.

That is pushing me toward a shared, cost-aware search service for the household instead of handing every agent its own paid search keys. Search should be a shared tool boundary first, and maybe a research subagent later.

The Dashboard Needed a Tailnet Door

I also needed to reach the Hermes web UI while away from home.

The safe local dashboard bind is loopback. That works when I am sitting at the machine. It does nothing when I am somewhere else and want the browser to hit the homelab over Tailscale.

The fix was to start the Hermes dashboard bound to the homelab’s Tailscale address on port 9119, with dashboard auth still enabled. That made it reachable from tailnet devices without binding it to every interface or exposing it to the public internet.

It is a small fix, but it matters. Remote access is where homelab convenience often turns into accidental exposure. This version is the shape I want: reachable over the private tailnet, authenticated, and not advertised to the whole LAN wildcard.

The Benchmark Said DeepSeek Wins Broadly and Loses Specifically

The most useful work of the week was not operational. It was measurement.

The Spark endpoint passed the smoke tests, but smoke tests are not a model selection strategy. So I ran it through the same local-agent benchmark suite I used for the prior bakeoffs: Home Assistant, calendar, portfolio, to-do, and two coding tasks scored by actually executing generated Python.

DeepSeek V4 Flash Vision-Exp on the Spark cluster became the broad-suite leader:

That is the right kind of messy result.

DeepSeek wins the broad personal-assistant benchmark. Granite still wins Home Assistant by a lot. If I care about one score across all household-agent jobs, DeepSeek leads. If I care only about smart-home control, Granite remains the model to beat.

The most important failure mode was not hallucinated tools. DeepSeek often answered from fixture context instead of calling the authoritative tool. It saw the motion sensor state in the prompt and answered directly. That is understandable in a static benchmark. It is wrong for a live house.

That may be the biggest architecture lesson of the week. Prompt context is routing metadata. Tools are the source of truth.

Tuning Taught Me What Not to Touch

After the baseline, I tried to make the model show better on the same hardware.

The useful tuning profile was modest:

MAX_NUM_SEQS=6
GPU_MEMORY_UTILIZATION_TEXT=0.80
Issue 191 tool-call fail-closed enabled
thinking=false for tool-call requests

That candidate nudged the equal-weighted score from 0.854 to 0.857, improved sample-pooled from 0.770 to 0.785, and cut the run time from about 24.8 minutes per pass to 21.3 minutes. Not a revolution. A real improvement.

The tempting bad idea was the k=5 DSpark block unlock. On paper, it made sense. Vision-Exp’s trained DSpark block size is 5, and the stock serving path uses 6 because of a runtime divisibility rule. The community notes showed reasons to try it.

On this workload, it regressed hard:

k=6 candidate: 0.857 equal-weighted
k=5 candidate: 0.824 equal-weighted

It hurt Home Assistant, portfolio, and to-do. It also made the coding conflict task slower. That is exactly why I do not want to turn knobs by intuition. The knob that sounds closer to the model’s internals can still be wrong for the workload.

I paused with SP indexer enabled and healthy, ready for the next round of long-context testing. That one is more likely to matter for big prompts than for the current household-agent suite.

The Unit Cleanup Was Small but Worth Doing

One editorial fix snuck in at the end.

The Spark posts used a binary-unit label in a few places. The site convention is GB for reader-facing storage and memory numbers. I scanned active posts and normalized the Spark prose.

I left old MiB strings in command output and archived fodder alone. Literal logs are allowed to look like logs. Published prose should be consistent.

This is not the kind of fix anyone remembers. It is the kind that makes a site feel less random when you read three related posts in a row.

By the Numbers

• 1 durable endpoint name, deepseek-spark.local
• 2 Spark nodes rebooted and verified after durability work
• 1 Hermes default model moved to the Spark DeepSeek endpoint
• 1 Hermes dashboard exposed safely over Tailscale on port 9119
• 3 full DeepSeek benchmark passes from the homelab workstation
• 405 benchmark samples scored across those passes
• 0.854 DeepSeek’s baseline equal-weighted score, the new broad-suite leader
• 0.857 best measured tuning score so far with tool-call requests using thinking=false
• 0.824 k=5 block-unlock score, a clear regression
• 5 active binary-unit references normalized to GB