How We Got Here: Building the Test Harness Behind the Local Agent Bakeoff

The results already shipped. Qwen 3.6 wins on equal-weighted average, Muse Glimmer nearly took it, Hermes 4.3 finishes last twice over. I published that post first on purpose — Muse Glimmer was six days old at test time, and getting a real number out before the takes piled up mattered more than telling the story in order.

This is the story in order. Building the instrument that produced those numbers took longer than running it, and it’s the part worth writing down before I forget the details: forking an open-source Home Assistant benchmark, extending it into three domains it never covered, and hitting two real bugs — one that was sitting in code nobody had touched yet, one that I wrote myself and didn’t notice until a script that should have passed didn’t.

Why Fork Instead of Build

The homelab’s actual job is Home Assistant, a calendar, an investment portfolio, and a to-do list — not a coding benchmark. Every option I looked at for testing that mix was either a manual rubric (score 1-5, argue with yourself about the number) or built for a different job entirely.

Drizzt321/ha-voiceagent-llm-benchmark had already solved the hard infrastructure problems for exactly this shape of eval: Inspect AI wiring, tool-call capture without execution, multi-dimensional correct/incorrect/not-applicable scoring, NDJSON test cases, direct llama.cpp integration. It only covered Home Assistant device control. Forking it and extending it was less work than reinventing the same plumbing, and it meant starting from a harness that already had 106 passing unit tests and real production mileage instead of a blank file.

The architecture made the decision easy to commit to. Dataset loading, tool definitions, prompt assembly, the solver, and the scorer are five separate files with a clean boundary between them. Reusing four of the five and swapping the parts that were genuinely Home Assistant-specific turned out to be exactly as clean as the file layout promised.

What Was Reusable, What Wasn’t

To-do needed almost nothing. Home Assistant already has real native intents for list management — HassListAddItem, HassListCompleteItem — so the to-do domain is the original task file, unmodified, pointed at a richer fixture (personal and work task lists, not just a single shopping list). No new code.

Calendar and portfolio needed real work, because no ground truth exists for either. Home Assistant has no built-in Assist intents for calendars — the calendar integration exposes entities and a couple of services, but nothing wired into the LLM intent system the way device control is. So calendar_tools.py‘s five tools (list, find-next, create, update, delete) are designed from scratch, not extracted from an existing spec.

Portfolio needed an actual design decision, not just new code: no trade-execution tool exists, on purpose. A local model placing real trades unattended is a much bigger trust call than “did it call the right function in a benchmark,” and it’s not one this bakeoff was built to make. The portfolio domain is read-only — holdings, quotes, performance, drift — and three of its fifteen test cases exist specifically to check that every model refuses a buy/sell/rebalance request instead of hallucinating a way to comply.

The scorer itself needed one small, structural change. tool_call_scorer() had VALID_TOOL_NAMES and a query-tool set hardcoded as module constants — fine when Home Assistant was the only domain, wrong once calendar and portfolio needed their own valid-tool sets. Parametrizing both (with the HA sets as defaults, so the original task’s behavior didn’t change) was a five-line diff that unlocked reuse across every new domain without touching the scoring logic itself.

The Bug That Was Already There

Before writing a single line of new domain code, I ran the original, unmodified Home Assistant benchmark to confirm the baseline actually worked. It didn’t. All 80 samples errored:

Value error, Unknown GenerateConfig field(s): config.
Use extra_body for provider-specific options.

The installed inspect-ai — 0.3.259, still within the repo’s own pyproject.toml pin of <0.4 — had changed Generate.__call__‘s signature somewhere in the 0.3.x line. The old call style, generate(state, tool_calls="none", config=GenerateConfig(...)), no longer matched; config fields have to be passed directly as keyword arguments now. This wasn’t a bug I introduced. It was sitting in solver.py, unmodified, waiting for anyone to update their inspect-ai install past whatever version the repo was last tested against.

Two things made this worth stopping for. First, it would have silently invalidated every new domain too, since I was about to copy the same call pattern into a new generic solver. Second, and more useful as a habit: I found it because I checked the baseline before extending anything, not after something new looked wrong. If I’d started writing calendar and portfolio code first and hit this error, the natural assumption would have been “I broke something in the new code” — a much longer debugging path than “the thing I haven’t touched yet is also broken.”

Fixed in both solver.py and the new domain_solver.py: pass timeout, attempt_timeout, and max_retries directly to generate() instead of wrapping them in a GenerateConfig object. Verified against all 80 original Home Assistant samples (clean run, 0 errors) before moving on.

A Boundary Case That Lied

The portfolio-drift coding task needed a test for strict-inequality logic: flag an asset class only if it’s drifted from target by more than a threshold, not equal to it. I designed a boundary case on paper — a 55%/45% split against a 50%/50% target, a clean 5-point drift against a 5-point threshold — and wrote a unit test asserting it should NOT be flagged.

The test failed. Not because the logic was wrong:

>>> 55000 / 100000 * 100
55.00000000000001

Binary floating point can’t represent 0.55 exactly. The “exact” boundary case wasn’t exact — it was 5.000000000000007 points of drift, which is greater than 5, which means the ground-truth function I’d just written correctly flagged it, and my hand-written expectation was the thing that was wrong.

The tempting fix is an epsilon tolerance in the comparison — > threshold + 1e-9 — and I started to write exactly that before catching the real problem with it: it doesn’t fix the test case, it changes what the test case means. A model’s script that computes the same floating-point division would hit the identical rounding noise and could flag the same “boundary” case for the same accidental reason, and an epsilon in the scorer would forgive that as if it were correct reasoning about the boundary rather than the same coincidence. The actual fix was choosing numbers immune to the problem: a 50%/50% split against a 45%/55% target, both of which land on exactly representable binary fractions, so the boundary is real instead of an artifact of how the test happened to be written.

Small bug, but a useful reminder for anyone building execution-scored tests: floating-point arithmetic doesn’t fail loudly. It fails by being five-billionths of a percent wrong in exactly the spot where you’re checking an inequality.

Closing a Contamination Gap

The coding tasks execute model-generated Python directly — via subprocess.run(), with a timeout, no network access from the harness’s side. That’s adequate for scoring output correctness against models you already trust enough to run as an agent. It is not a hardened sandbox, and one gap was worth closing before running anything for real: the subprocess originally ran in the harness’s own working directory, with no scratch isolation. Nothing in the prompt asks a generated script to write a file, but nothing forbids it either, and a file written by one model’s script could in principle persist and leak into a later model’s run.

Every other part of this harness is already isolated by construction. Only one model is ever resident in VRAM at a time — llama-swap tears down the previous llama-server process before booting the next — and every Inspect sample is a single, stateless generate() call with no shared memory between samples or between separate eval runs. The subprocess execution path was the one place state could theoretically survive past the request that created it. Fixed by wrapping each script execution in a fresh tempfile.TemporaryDirectory() and running there instead — one line of structural change, closes the only real gap.

Giving Every Model a Fair VRAM Shake

The first context-window pass used placeholder values I picked as reasonable-sounding defaults, not measured ones. That turned out to matter enough to redo properly.

Hermes 4.3’s first config asked for 65,536 tokens of context and OOM’d outright — its dense 36B weights at Q5_K_M leave too little VRAM for a KV cache that large. Dropping to 16,384 worked but left VRAM sitting idle; testing upward found 32,768 was the real ceiling, landing at roughly 30GB used of the card’s 32.6GB. That’s a genuinely tight, structural limit for this model on this card, not a config oversight.

The other three told a different story once actually measured. Nemotron Lightning was configured for 40,960 tokens — and testing showed it holds up to 524,288 with no capping and barely more VRAM used (27.3GB vs. 25.4GB at 131,072), because its hybrid Mamba-Transformer architecture scales context far more cheaply than pure attention. Muse Glimmer was configured for 32,768 and turned out to have a hard architectural ceiling at 131,072 — llama.cpp logs and silently caps past that regardless of available VRAM, since it’s the model’s trained maximum, not a resource limit. Qwen was already at 131,072 and tested clean up to 262,144.

Rather than give each model whatever number it could physically support — which would have made “how much can this model see” a hidden, uncontrolled variable in the results — I set Qwen, Nemotron Lightning, and Muse Glimmer to the same 131,072, since that’s the real ceiling for the one model (Muse Glimmer) that can’t go higher no matter what. Hermes 4.3 stays at its own VRAM-bound 32,768, a real disadvantage that’s honestly reported rather than argued away. Every actual sample in this bakeoff used a few thousand tokens at most — nowhere near any of these ceilings — so none of this changed a single score. It changed whether the setup was defensible if someone asked why.

A Fifth Contestant, Added Mid-Run

Qwen3.8-27B shipped a day after the results post went out — Apache 2.0, dense 27B, a surprise vision encoder, 262K native context. Recent enough that skipping it felt like the wrong call, so it went through the exact same battery as the original four: same fixtures, same task files, same scorer, no changes to the harness itself.

It’s the odd one out architecturally — the only dense model in a field of three MoE/hybrid designs and one dense-but-different Hermes 4.3 — but its hybrid Gated DeltaNet/attention block (a 3:1 ratio, only 16 of 64 layers carrying a KV cache) makes context nearly as cheap as the Mamba-hybrid Nemotron Lightning. It loaded the full UD-Q4_K_XL quant plus 131,072 tokens of context in 22.9GB, comfortably inside the 5090’s 32GB. That context number wasn’t a discovery this time — it was a decision to match. Qwen3.8-27B could structurally run well past 131,072 (its native ceiling is 262,144, extensible further upstream), but Muse Glimmer still can’t, so giving the new contestant more context than the group’s established ceiling would have handed it an advantage that had nothing to do with model quality. Same rule as before, just applied to a fifth model instead of three.

Two Scoring Methods, One Real Number

The scorer produces a straightforward accuracy percentage per domain. Turning five domains of different sizes into one overall number takes a decision, and it’s worth stating before the numbers exist rather than picking whichever method flatters a preferred outcome after the fact.

Pooling every sample together — 135 correct-or-not answers divided into one accuracy number — lets Home Assistant’s 80 samples decide 59% of the result by sheer count, even though it’s one of five equally real jobs this assistant does. Averaging the five domain accuracies instead treats Home Assistant, calendar, portfolio, to-do, and coding as five co-equal responsibilities regardless of how many test cases exist for each. That’s the number that matches how the assistant actually gets used, so it’s the one the results post uses for every conclusion — sample-pooled is reported too, for transparency, and the two methods genuinely disagreed on the middle of the field once real numbers came in.

How Much Does One Run Prove?

Every number up to this point — in this post and in the results post it explains — came from running each model through the battery exactly once. That’s standard practice for a leaderboard, and it’s also an assumption worth checking rather than trusting by default: LLM inference at a non-zero temperature doesn’t return the same tool call twice just because you asked the same question twice. So after the initial results shipped, all five models (the original four, plus Qwen3.8-27B) went through the full six-domain battery a second time, then a third, with nothing else changed — same fixtures, same prompts, same scorer, same model weights.

The ranking moved. After run 2, Qwen 3.6 — the equal-weighted winner in the published results — dropped from 1st to 3rd, and Qwen3.8-27B jumped from 4th to 1st. Nothing about either model changed between runs; the only thing that changed was which of several plausible tool calls each model happened to sample that time.

Equal-weighted score by run, all five models, sorted by 3-run mean.

Averaged across three runs, Qwen 3.6 does end up back on top — but by 0.010 over Qwen3.8-27B, a smaller gap than either model’s own run-to-run standard deviation (0.020 and 0.019). That’s not a real gap; it’s two models tied inside the noise floor of this test size. Muse Glimmer, which led after run 1, falls to third once averaged, dragged down almost entirely by one domain.

Here’s the full domain-level picture behind those averages:

Accuracy by model and domain, all three runs, with per-domain standard deviation and range.

Three things stand out.

To-do is the noisiest domain in the battery, by a wide margin. It’s also the smallest, at 11 samples. Muse Glimmer and Qwen3.8-27B each swing 0.273 across three runs on it — over a quarter of the score, on a domain that’s just five native Home Assistant list intents repeated across a few fixtures. Every domain-level swing bigger than 0.15 anywhere in this table happened on to-do. Compare that to Home Assistant, the largest domain at 80 samples: every model’s HA stdev stays at or below 0.039. More samples buys stability; 11 isn’t enough to trust a single run on, and to-do is the domain where that shows up hardest.

The coding_conflict “ceiling” wasn’t actually a ceiling. The results post noted all four original models hit a flat 1.000 on both coding tasks and called it a ceiling effect worth watching. Two more runs partially answer that: Nemotron Lightning and Hermes 4.3 each stayed perfect for two runs, then both dropped to 0.875 on the third. Qwen 3.6, Muse Glimmer, and Qwen3.8-27B stayed at 1.000 across all three. That’s a real, if thin, capability signal — 2 of 5 models have a non-zero failure rate on this task — but it took a third independent sample to surface. A single run would have reported five identical 1.000s and called the domain saturated.

Consistency is its own axis, separate from accuracy. Hermes 4.3 (stdev 0.004) and Nemotron Lightning (0.007) came back almost bit-for-bit identical across all three runs — every domain except calendar-conflict returned the exact same score three times. Qwen 3.6, Muse Glimmer, and Qwen3.8-27B all move meaningfully run to run (stdev 0.019–0.032). Hermes 4.3 finishes last on raw accuracy in every single run, but if what you actually want from a local agent is predictability — the same input reliably producing the same class of output — it and Nemotron Lightning are the two models that deliver that, and neither of the two overall accuracy leaders do.

That consistency finding also answers a narrower, more practical question: which model to trust with the Home Assistant domain specifically, since that’s the actual daily job, not an abstraction.

Home Assistant domain accuracy by run, all five models, sorted by 3-run mean.

The equal-weighted winner and the Home Assistant winner are two different models. Qwen 3.6 wins the aggregate — driven by strong portfolio and calendar numbers — but it’s the second-weakest of the five at the one job that’s actually a voice-controlled smart-home butler: pure HA tool-calling. Qwen3.8-27B and Muse Glimmer are statistically tied for the best HA performance (0.708 vs 0.705, well inside each other’s stdev), with Nemotron Lightning a clear third. If the deciding use case is specifically “can I trust this to run my house,” the aggregate leaderboard is the wrong number to read — the domain-specific one is.

What Three Runs Change (and What They Don’t)

They don’t overturn the headline. Qwen 3.6 is still the equal-weighted winner, Hermes 4.3 still finishes last, and nothing here suggests the original methodology — fork a proven harness, verify the baseline, fix real bugs, give every model a defensible context window — was unsound.

What they do change is how much confidence any single decimal place deserves. The published results post reported Qwen 3.6 at 0.842 against Muse Glimmer’s 0.832 as if that 1-point gap meant something. It didn’t — both numbers move by more than that between runs of the same model. The real takeaway isn’t “Qwen 3.6 beats Muse Glimmer,” it’s that Qwen 3.6, Qwen3.8-27B, and Muse Glimmer are three models bunched together at the top, indistinguishable at this sample size, while Hermes 4.3 and Nemotron Lightning are clearly behind them on accuracy but clearly ahead of them on consistency. That’s a less quotable sentence and a more honest one.

It also means the domain that matters to a specific use case can point somewhere different than the aggregate. That’s not a flaw in equal-weighting five domains — it’s what equal-weighting is supposed to reveal, once you look at the domain instead of just the average it feeds into.

The fork lives at carryologist/ha-voiceagent-llm-benchmark, branch bakeoff-personal-assistant-domains. 168 unit tests, ruff clean, six task files, three of them entirely new.

By the Numbers

• 2 real bugs found and fixed — one pre-existing, one self-inflicted
• 106 → 168 unit tests, all passing
• 5 files in the original architecture, 4 reused unchanged
• 55.00000000000001 — the floating-point value that broke a hand-written test expectation
• 131,072 — the context window 4 of 5 models were set to, for fairness, not convenience
• 524,288 — the context window Nemotron Lightning actually tested clean to
• 1 temp directory per script execution, closing the only real cross-run contamination gap
• 5 models, 3 full passes each through all six domains — 90 eval runs total, to separate signal from noise
• 0.010 — the gap between the 1st- and 2nd-place models by 3-run average, smaller than either model’s own run-to-run standard deviation
• 0.273 — the largest single-domain swing across three runs (to-do, hit by both Muse Glimmer and Qwen3.8-27B)
• 2 of 5 models cracked the coding_conflict “ceiling effect” on the third run, after two straight perfect scores
• 0 manual scores in the entire results post