Local Agent Bakeoff: Granite 4.2 Is the New Homelab King, But Turn Thinking Off to Speed It Up 10x

Qwen 3.6 held the crown last round, but barely. The margin over second place was smaller than the model’s own run-to-run noise. I said I’d keep watching for the next thing worth testing.

It showed up sooner than expected, and not on purpose. I asked the agent to sweep the homelab for routine software updates, and the llama.cpp changelog for that update mentioned three model architectures: I’d never heard of: Qwen3.8-Flash-Next, DeepSeek V4, and something called Nemotron3.5. We’d considered them before, but dismissed them. I couldn’t recall why. Oh, that’s right! Too big. But chasing that down surfaced three real contestants that fit the rig and had never been through the harness: IBM’s Granite 4.2, Poolside’s Laguna XS 2.1, and DeepReinforce’s Ornith 1.5. So we ran it back.

This round changes the leaderboard, and the most interesting finding wasn’t even about which model won. It was about a single configuration flag on the winner.

Three New Contestants Join the Field

Same rig as last time: one RTX 5090, 32GB of VRAM, one llama-swap-managed endpoint. Round 1’s five models (Qwen 3.6, Qwen3.8-27B, Nemotron Lightning, Muse Glimmer, Hermes 4.3) did not get rerun. Nothing about the harness or the fairness rules changed since last time, so their numbers stand as recorded.

Specs for all three new models as configured on the homelab’s RTX 5090.

Granite 4.2 is IBM’s first reasoning-native Granite release, dense across all three of its sizes, purpose-built for agentic enterprise workflows. Laguna XS 2.1 is Poolside’s smallest coding-focused model, a mixture-of-experts design meant to run agentic coding work on a single consumer GPU. Ornith 1.5 is DeepReinforce’s mid-size entrant, distilled down from a 397B flagship using the same self-improving training loop, the same “smaller model inherits the bigger one’s training recipe” story that made Nemotron Lightning a strong showing last round.

Laguna XS 2.1 and Ornith 1.5 both hold the group’s 131,072-token fairness ceiling with room to spare, 25.0GB and 24.3GB used respectively. Granite 4.2 did not. It’s the only dense model in this round’s field, and dense attention makes a large KV cache expensive in a way MoE and hybrid architectures don’t pay. The group ceiling OOM’d outright. Binary-searching it found the real number: 98,304 also OOMs, by about 512MB on the compute buffer, and 65,536 loads clean with roughly 4.5GB of headroom. That’s the same structural situation Hermes 4.3 hit last round, a dense model paying a real VRAM tax that MoE contestants don’t, honestly reported rather than argued away.

The Test Didn’t Change, So the Comparison Holds

Same six domains, same fixtures, same Inspect AI-driven scorer built out over the last bakeoff’s harness work: Home Assistant device control, calendar management, a read-only investment portfolio, a personal to-do list, and two coding tasks scored by actually executing the generated code. Every new model went through the full battery three times, not once, for the same reason as last round: a model doesn’t necessarily make the same tool call twice at non-zero sampling temperature, and a single run tells a cleaner but less true story than three do.

Granite 4.2 Takes the Equal-Weighted Crown

Accuracy by domain, 3-run mean, all three new models, IBM Granite 4.2 tested in its default thinking-on configuration.

Line that up against round 1’s table and the leaderboard actually moves. Granite 4.2 lands at 0.832 equal-weighted, ahead of Qwen 3.6’s 0.814. Ornith 1.5 lands at 0.817, also ahead of the old incumbent. Laguna XS 2.1 lands at 0.799, tying Muse Glimmer almost exactly.

Combined leaderboard across both rounds, sorted by equal-weighted 3-run mean.

Granite 4.2’s standard deviation across its three runs is 0.002, the tightest of any model tested across both rounds. Whatever it’s doing, it’s doing it the same way every time.

Granite Doesn’t Just Win, It Runs Away With the One Job That Matters

Equal-weighted treats all six domains as co-equal, which is the right number for an overall verdict. But this assistant’s actual daily job is mostly Home Assistant, and that domain tells a starker story than the aggregate does.

Home Assistant domain accuracy, 3-run mean, both rounds combined.

That’s not a close call. Granite 4.2 beats round 1’s best Home Assistant performer by nearly nine full points, on the one domain that’s actually a voice-controlled smart-home butler rather than an abstract benchmark. Laguna XS 2.1 and Ornith 1.5, both coding-leaning models by training focus, land well below the group’s middle here, a reminder that a model’s marketing category (agentic coding, in both cases) doesn’t automatically transfer to tool-calling accuracy in a different domain.

Turning Thinking Off Costs Nothing and Saves 10.8x the Clock

Granite 4.2 was also, by a wide margin, the slowest model to run through the battery. Its 80-sample Home Assistant domain alone took roughly 10 minutes, next to Laguna XS 2.1 finishing the same domain in 45 seconds. The reason is IBM’s own design choice: Granite 4.2 ships three switchable reasoning modes, full thinking, low-effort, and non-thinking, and thinking is on by default. Every sample in the table above got a full chain-of-thought trace before the actual tool call, because nothing in the harness told it not to.

IBM’s stated reason isn’t generic. Their framing is specifically agentic: enterprise tasks are ambiguous and multi-step, and reasoning helps Granite 4.2 “evaluate which applications to use and in what order rather than executing blindly.” It’s a real bet, grounded in a real research lineage going back to the original chain-of-thought paper, that showing intermediate steps before an answer measurably improves multi-step accuracy. The question worth asking on a narrow tool-calling battery like this one is whether that bet pays for itself.

So we ran a fourth configuration: the exact same weights, the exact same context cap, the exact same everything, with --chat-template-kwargs '{"enable_thinking": false}' as the only difference.

Same weights, same context cap, one variable changed.

Non-thinking mode scored marginally higher on both numbers, well within noise, so call the accuracy difference a tie. What isn’t a tie is the clock. Turning thinking off cut the full 3-run, 6-domain pass from 92.9 minutes to 8.6 minutes, a 10.8x speedup, for zero measurable accuracy cost on this specific battery. IBM’s low-effort and non-thinking modes exist because a model that reasons about whether to turn off the office lights is spending compute on a decision that was never ambiguous. On a narrow, well-defined tool-calling job, that’s exactly what happened here. The reasoning-first default is a reasonable choice for genuinely ambiguous enterprise workflows. It’s the wrong default for a voice assistant that mostly needs one correct tool call, fast.

Two Coding Tasks Are Now Fully Solved, Which Means They’re Broken as a Test

Every model in this round, all four configurations including both Granite variants, scored a flat 1.000 on both coding tasks, across all three runs, no cracks anywhere. That’s a harder ceiling than round 1 saw. Last round, two of five models dropped to 0.875 on the calendar-conflict detector on the third run, a thin but real signal that the task still had some difficulty left in it. This round, nothing moved. Not once, across 18 fresh eval runs from two brand-new coding-focused models.

That’s not a compliment to the models so much as an indictment of the test. A portfolio-drift-flagging script and a calendar-conflict detector were reasonable coding checks for last round’s field. They’re not discriminating anymore, and a saturated test tells you nothing about which model is actually better at code. Before round 3, this needs fixing, and there are really only two honest paths. Hand-roll harder cases into the existing harness, more edge cases, nested logic, maybe a task that requires reading and modifying existing code instead of generating a script from scratch. Or bolt on an established, harder benchmark like SWE-bench or Terminal-Bench as a supplement, since both Laguna XS 2.1 and Ornith 1.5 already report strong numbers there and it would let this bakeoff’s coding domain actually separate the field instead of rubber-stamping everyone with a perfect score.

The Verdict

Granite 4.2 is the new equal-weighted leader, and it should run with thinking off. It wins the aggregate, and it dominates the Home Assistant domain specifically by the widest margin either round has produced. The catch is entirely self-inflicted: run it in IBM’s default configuration and it’s the slowest model tested by a wide margin, for accuracy that a non-thinking config matches or slightly beats. Configured correctly, this is the new daily-driver candidate.

Ornith 1.5 is a legitimate second-place finish from a model built for a different job. DeepReinforce trained it for agentic coding, not smart-home tool-calling, and it still lands ahead of round 1’s incumbent on the equal-weighted number. Its Home Assistant score is the weakest of the three new contestants, which tracks with its training focus, but the overall package holds up.

Laguna XS 2.1 ties Muse Glimmer almost exactly, and pays for its coding-model DNA in wall-clock time. It’s the only one of the three new models whose coding-task times ballooned (249 seconds and 454 seconds for the two coding tasks alone, out of a 15.5-minute total run), a real cost this round’s scoring doesn’t capture. Its own DFlash speculator model, not wired up for this round, is the obvious next lever if it stays in rotation.

What’s Next

Granite 4.2 with thinking off is the new candidate for the actual homelab, not just the leaderboard. Before committing to a daily-driver swap, it needs the same soak-test treatment Muse Glimmer got last round, real usage, not just battery numbers. The coding domain needs a rebuild before round 3 means anything on that axis, and that’s the next concrete task, not a someday item. And DFlash and DSpark, the speculative-decoding draft models several of this round’s contestants shipped with, never got tested. That’s a speed question for a future post, not a quality one, since none of it should move a single accuracy number above.

By the Numbers

• 3 new contestants tested, 4 configurations total counting Granite’s thinking-on and thinking-off variants
• 0.832 — Granite 4.2’s equal-weighted 3-run mean, the new leader across both rounds, beating round 1’s Qwen 3.6 at 0.814
• 0.794 vs. 0.708 — Granite 4.2’s Home Assistant-domain mean vs. round 1’s best performer on that domain, Qwen3.8-27B
• 10.8x — the wall-clock speedup from turning Granite 4.2’s default thinking mode off, 5,572 seconds down to 514 seconds across the full 18-eval pass
• 0.002 — Granite 4.2’s equal-weighted standard deviation across three runs, the tightest of any model tested across both rounds
• 0 — cracks in the coding-task ceiling this round, versus 2 of 5 models cracking it on round 1’s third run
• 65,536 — Granite 4.2’s VRAM-bound context ceiling, measured after 131,072 and 98,304 both OOM’d
• 8 models now tested across two rounds, 0 of round 1’s five rerun this round on the assumption the harness didn’t materially change