Splitting the Bakeoff: Coding Gets Its Own Test, and It’s Terminal-Bench Over SWE-bench

Round 2 of the local agent bakeoff closed with a clean leaderboard and one loose thread. Both coding tasks, a portfolio-drift-flagging script and a calendar-conflict detector, scored a flat 1.000 across every model, every run, all 18 fresh evals from two models built specifically for agentic coding. Round 1 at least cracked a little on the third run. Round 2 didn’t move at all.

That’s not a compliment to the models. It’s a broken test. Before running any more of these bakeoffs, it’s worth stopping to plan the fix properly instead of bolting on another hand-rolled task and hoping it holds up longer. This is that plan, not the results. The actual eval run is a separate post, the same way the harness deep dive followed the round 1 results instead of getting rushed into the same piece.

Why Coding Doesn’t Belong in the Home Assistant Butler Score

The bakeoff has always been built around one real question: which local model should run my smart home, my calendar, my portfolio, and my to-do list. Those four domains share a common thread. They all require judgment under ambiguity, deciding whether to act, ask, or refuse, picking the right tool out of several plausible ones, extracting the right arguments from a loosely specified request. Coding is a different kind of task entirely. A generated script either produces the correct output or it doesn’t. There’s no ambiguity to navigate, just correctness to verify.

Folding a different kind of question into the same equal-weighted average has been quietly doing damage for two rounds now. Laguna XS 2.1 and Ornith 1.5 are both coding specialists by training focus, and both landed below the group’s middle on the domain that’s actually this assistant’s job, Home Assistant. Their coding score, now saturated at a perfect 1.000 for both, was doing nothing to separate them from each other or from anyone else, while still counting for a sixth of their aggregate. A domain that can’t discriminate shouldn’t get a vote.

The fix isn’t a harder hand-rolled task. The last coding domain build already turned up a real bug, a floating-point boundary case that lied about being exact, in a test written from scratch with no independent validation. Making that same kind of task harder just means re-earning trust in new ground truth I wrote myself, with no outside check on whether the difficulty curve lands anywhere useful. Better to answer the coding question with a benchmark built and maintained by people who do nothing else, and answer the Home Assistant question with the four domains that were always the actual point.

Terminal-Bench and SWE-bench Answer Different Questions

Two names come up in almost every coding-model card these days, including the ones for this round’s contestants. Terminal-Bench 2.1 and SWE-bench both claim to measure real coding capability, and both get cited constantly, but they’re not measuring the same thing.

SWE-bench is the more established name. SWE-bench focuses on resolving GitHub issues inside existing software repositories, and its curated Verified subset consists of 500 software engineering tasks rigorously validated by human expert developers. That’s real, industry-recognized rigor, and it’s why so many model cards lead with a SWE-bench number. It’s also narrow by design, one specific and very common job, patch a bug in someone else’s codebase, evaluated at real scale.

Terminal-Bench takes a wider view. Terminal-Bench 2.1 evaluates a broader range of technical work than SWE-bench, including workflows such as configuring services, compiling software, training models, processing data, repairing security problems, and reproducing scientific results inside terminal environments. That’s a closer match to what actually happens on a homelab box day to day, not just fixing someone else’s Python.

The setup cost is where the two really diverge. SWE-bench needs a working environment for every one of its 500 instances, real repo checkouts with real dependency graphs, environment initialization specific to SWE-Bench, which includes removing future commits to prevent data leakage, as well as configuring network proxies and critical system settings. Terminal-Bench keeps the task count smaller and the environments self-contained. Terminal-Bench v2.1 is a verified refresh of the v2.0 agentic terminal benchmark, keeping the same 89 curated tasks, and it comes with its own purpose-built execution framework: researchers can run Terminal-Bench 2.1 with Harbor, the open-source evaluation framework used to execute tasks in containerized environments.

Neither one is a small lift compared to the subprocess-in-a-tempdir scorer this bakeoff has used so far. Both need real container orchestration. But 89 self-contained tasks is a much more tractable target for one RTX 5090 than 500 tasks each needing their own repo-specific setup.

One number worth sitting with before committing to either: frontier models and agents score less than 65% on the benchmark, referring to Terminal-Bench 2.0’s own difficulty calibration. A 30B-class quantized local model could easily crater toward single digits on either benchmark, trading today’s ceiling problem (everyone scores 1.000) for a floor problem (everyone scores near zero). That risk applies equally to both, and it’s the reason a pilot run comes before a full commitment, more on that below.

Why Terminal-Bench 2.1 Wins the Call

Three things tip this toward Terminal-Bench over SWE-bench for this specific project, not as a universal ranking, just for what this blog actually needs right now.

It’s the more homelab-shaped benchmark. A homelab box does a mix of sysadmin work, service configuration, and the occasional script, not exclusively “resolve issues in a large open-source Python codebase.” Terminal-Bench’s broader task variety, compiling, data processing, security fixes, matches that mix better than SWE-bench’s narrower GitHub-issue framing.

It’s already the reference point for this round’s actual candidates. Both Laguna XS 2.1 and Ornith 1.5 publish Terminal-Bench numbers on their own model cards. Running it locally means checking a real, comparable claim against the exact quantized weights sitting on this rig, not just trusting a vendor’s full-precision number on different hardware.

It’s actively maintained, and that maintenance is visible. v2.1 incorporates environment and instruction fixes, patched Dockerfiles and corrected instruction-test mismatches across roughly a dozen tasks, so scores reflect agent capability rather than environment gaps. That’s a benchmark maintainer catching and fixing the exact class of bug this blog hit building its own coding tasks from scratch. Trusting someone else’s test more than my own, on this specific point, is the honest call.

SWE-bench isn’t wrong, and it isn’t going in the trash. It’s the more universally recognized name, and if a coding-focused model ever earns a dedicated deep dive, running it against SWE-bench Verified too would make the result legible to a much wider audience. But that’s a future nice-to-have, not the thing to build first.

The Plan for the Actual Run

This is the plan, written down before any of it happens, the same discipline the merge-plan pattern from the feature bakeoffs argues for: decide the approach on paper first, then let the run confirm or correct it.

Step 1, stand up Harbor. Get the containerized evaluation framework running against the homelab’s existing llama-swap endpoint, using the Terminus 2 scaffold that Terminal-Bench’s own leaderboard runs use. This is pure infrastructure, no scoring yet.

Step 2, pilot on a handful of tasks before committing to all 89. Given the floor-effect risk above, the first real test isn’t a full run. It’s two or three tasks, picked for a range of difficulty, to confirm a 30B-class quantized local model produces scores that actually land somewhere between 0 and 100%, not uniformly at one end. If the floor effect shows up here, that’s the moment to reconsider scope, not after burning a full 89-task pass on every candidate.

Step 3, run it against the models that actually claim coding strength first. Laguna XS 2.1 and Ornith 1.5 both trained specifically for agentic coding and both already publish Terminal-Bench numbers, so they’re the natural first pass, the two data points this whole project exists to check. Granite 4.2, Qwen3.8-27B, and the rest of the Home Assistant-focused roster can follow once the harness is proven out, since a coding benchmark is still worth knowing even for a model that isn’t marketed on it.

Step 4, report it as its own leaderboard, not a folded-in column. Whatever comes out of this gets its own table, its own post, and its own verdict, kept explicitly separate from the equal-weighted Home Assistant butler score. Two questions, two numbers, no more diluting one with the other.

None of this touches the four-domain personal-assistant battery going forward. That test stays exactly as it is, minus the two coding tasks it’s carrying today, and it’s what future bakeoff rounds will keep running.

By the Numbers

• 0 cracks in round 2’s coding-task ceiling, across 18 fresh evals from 2 coding-focused models, the finding that triggered this whole plan
• 89 curated tasks in Terminal-Bench 2.1, versus 500 in SWE-bench Verified
• 2 models with a real reason to go first in the pilot, Laguna XS 2.1 and Ornith 1.5, both already publishing Terminal-Bench numbers on their own model cards
• <65% — frontier models’ own reported ceiling on Terminal-Bench 2.0, the number that justifies a small pilot run before a full 89-task commitment
• 1 floating-point ground-truth bug already found in this blog’s own hand-rolled coding tasks, the reason a self-maintained benchmark keeps losing to an externally validated one
• 4 domains staying in the Home Assistant butler battery going forward, Home Assistant, calendar, portfolio, and to-do, now that coding has its own separate test
• 2 benchmarks researched, 1 picked, for reasons specific to this rig and this project, not a universal verdict on either one