DeepSeek Wins the Homelab Agent Benchmark But Not the Smart Home

After getting DeepSeek V4 Flash with vision running on the Spark pair, I did the only reasonable thing. I pointed the local-agent benchmark at it and waited to see if the new dedicated inference tier earned its keep.

The benchmark has become my reality check for local AI. Not vibes. Not model-card claims. Not one impressive coding answer. The job is the same one my house asks of these systems every day: Home Assistant, calendars, a read-only portfolio, to-do lists, and a couple of coding tasks that execute the model’s generated Python instead of grading style.

DeepSeek V4 Flash Vision won the broad benchmark. It also did not win the part I most want a household assistant to win. That is the interesting result.

The Test Did Not Change For the Spark Cluster

The setup changed a lot. The model moved from the RTX 5090 tier to a dual DGX Spark cluster. The runtime moved from local llama.cpp and llama-swap to a two-node vLLM DSpark endpoint. The served model became DeepSeek V4 Flash Vision-Exp, exposed through the same OpenAI-compatible API shape Hermes can use.

The test did not change.

It is still the six-domain local-agent harness built on Inspect AI. It still runs one sample at a time. It still scores tool calls without executing real home actions. It still treats Home Assistant, calendar, portfolio, to-do, and coding as separate jobs instead of flattening everything into one pile and letting the biggest domain dominate.

The six domains were:

That is 135 samples per pass. I ran three full passes, because the earlier bakeoffs taught me not to trust one clean-looking run. Local models move around. Small domains move around more.

DeepSeek Takes the Equal-Weighted Lead

Here is the headline table.

On the broad assistant score, DeepSeek is the new leader. It lands at 0.854 equal-weighted, ahead of Granite 4.2 no-think at 0.837 and Granite 4.2 thinking at 0.832.

That is a real win, but not a sweep. Equal-weighted means each domain gets one vote. It matches how I think about the household assistant job. Home Assistant matters. So do calendars, portfolio questions, list management, and code. A model that dominates one job and falls apart on another should not hide behind sample count.

DeepSeek’s shape is balanced enough to win that version of the leaderboard.

Granite Still Owns the Smart Home

Sample-pooled tells a different story because Home Assistant has 80 of the 135 samples. Under that view, DeepSeek drops to 0.770 while Granite 4.2 no-think sits at 0.812.

That is not a scoring bug. It is the point of reporting both numbers.

If the question is “which model is the best broad personal assistant,” I care about equal-weighted. DeepSeek wins that. If the question is “which model should I trust most to run lights, thermostats, locks, covers, sensors, and household automations,” the Home Assistant domain deserves to dominate. Granite 4.2 no-think still wins that slice by a lot.

The Home Assistant gap is not subtle:

This is why I do not like single-number victory laps. DeepSeek is the new overall leader and still not the best smart-home model I have tested.

Both statements are true.

The Domain Breakdown Explains the Win

DeepSeek wins the aggregate by being good almost everywhere.

The coding tasks are saturated. DeepSeek scored 1.000 on both, every run. That sounds impressive until you remember I already suspected these tasks were aging out as differentiators. DeepSeek confirms it. The coding tasks still catch basic failure. They no longer separate strong models.

Calendar is the cleanest non-coding win. DeepSeek hit 0.875 three times in a row. Portfolio was solid at 0.778. To-do was noisy, as usual, but better than Granite’s weak 0.576.

That combination is enough to pass Granite overall, even while losing Home Assistant.

The Main Failure Was Not Hallucination

The failures were more interesting than the score.

DeepSeek did not mostly invent fake tools. It did not mostly emit malformed tool JSON. It often did something that looks helpful and is wrong for an agent.

It answered from context.

The benchmark prompt includes fixture state so the model can route tool calls. For example, the prompt may include that a motion sensor exists in the living room and its fixture state is off. When asked whether there is motion in the living room, DeepSeek often answered directly instead of calling HassGetState.

That is a reasonable answer if the prompt is ground truth. It is a bad answer if the prompt is a stale catalog and the tool is the live source of truth.

Across three runs, the failure classes looked like this:

The lesson is not “DeepSeek cannot use tools.” It can. The action path worked often. The lesson is that local household agents need an explicit architecture rule: context is routing metadata, tools are authority.

That becomes even more important once search, calendars, files, and Home Assistant all sit behind the same model. If a tool can answer live state, the model should call it. Prompt context should help choose the tool, not replace it.

The Spark Cluster Buys Capability Not Speed

The other caveat is latency.

Tool-call domains were usable. Home Assistant averaged 6.3 seconds per sample. Calendar averaged 7.8. Portfolio averaged 5.6. To-do averaged 7.1.

Coding was slow:

DeepSeek got every coding sample right, but it took its time. The Spark cluster is running a much larger multimodal model than the old 5090 candidates. The comparison is not fair if you only look at speed. It is also not fair if you ignore speed.

For a household assistant, I can live with 5 to 8 second tool calls while tuning continues. For interactive coding, this config is too slow. It might still be useful for background work. It is not a snappy pair-programmer yet.

The Winner Depends on the Job

This is where I land.

DeepSeek V4 Flash Vision-Exp on the dual Spark cluster is the best broad local-agent model I have tested so far. It wins the six-domain equal-weighted benchmark, runs through a durable household endpoint, and gives me multimodal capability the earlier text-only setup did not have.

Granite 4.2 no-think remains the model to beat for Home Assistant. If I were choosing only for smart-home control today, Granite still has the better number.

So the real answer is not a single model. It is routing.

The local AI stack I want probably uses DeepSeek as the general household brain and keeps a sharper, cheaper, faster model available for the narrow smart-home lane. That is the whole reason this homelab is turning into an inference tier instead of one heroic desktop. Different jobs deserve different models.

The Spark pair finally makes that kind of routing feel practical. It does not make the decision disappear.

By the Numbers

• 3 full DeepSeek passes through the benchmark suite
• 405 total benchmark samples across those passes
• 0.854 DeepSeek’s equal-weighted mean, the new broad-suite leader
• 0.770 DeepSeek’s sample-pooled mean, behind Granite 4.2 no-think
• 0.812 Granite 4.2 no-think’s Home Assistant mean, still the smart-home score to beat
• 1.000 DeepSeek’s score on both coding domains across all three runs
• 74.3 minutes total DeepSeek wall time across three passes
• 41 plain-text-instead-of-tool failures, the most important failure class
• 6.3 seconds average Home Assistant latency on the Spark endpoint