Has This Blog Been a Waste of Time? (I Made My Coding Agent Investigate)
I gave my coding agent an uncomfortable assignment this week: go find out if we've been wasting our time.
Every Model Showdown round on this blog runs on tooling we wrote ourselves — a bakeoff harness, a scoring rubric, thermal-test.sh, ad hoc token counting glued together across a dozen posts. My coding agent is also my research partner for all of it. Both of us are, structurally, coders. And I've started to wonder whether that's a bias, not a strength: if the tool you're best at is writing code, every problem you're handed starts to look like a problem you solve by writing more of it. Want to test consumer AI hardware? Build a harness. Sure, we repurpose libraries here and there, but we're still stitching it all together by hand, every time, instead of asking whether someone already solved this.
So the assignment was research, not code: go find out what hobbyists — the Linus Tech Tips / Gamers Nexus / Level1Techs crowd — and the broader AI benchmarking world already publish. Then tell me honestly whether we should have been using any of it instead of building our own.
What's Already Out There
The hardware-review houses. LTT Labs runs MarkBench, an orchestration and data-collection framework whose actual test harnesses — the same code that generates the numbers in LTT videos — are open-sourced on GitHub and updated on a quarterly cadence. It's built for GPUs and games, not LLMs: one harness scripts PyAutoGUI to click through a menu, another runs an OCR service just to find text on screen. Gamers Nexus takes the opposite approach to the same problem — not a reusable framework, but a public living document of exactly which SOPs, test benches, and settings produced which chart, updated review by review. Neither one has touched an LLM workload. And GN gave me the best counter-argument to my own thesis before I'd even finished asking the question: in 2025 they published a whole new measurement methodology — "animation error" — because framerate and frametime testing, the industry standard for a decade, still didn't capture what a stutter actually felt like to a player. Even the best-resourced reviewers in the business sometimes conclude nothing existing measures what they need, and build something new. That's not automatically a bias. Sometimes it's just correct.
The vendor-neutral AI-PC suites. MLPerf Client, built by MLCommons with AMD, Intel, Microsoft, NVIDIA, and Qualcomm all at the table, is now on v1.6 and measures how a Windows, macOS, or Linux client handles real generative AI tasks like summarization and content creation. UL's Procyon AI Text Generation Benchmark does the enterprise-press version of the same thing: seven fixed prompts against Phi-3.5-mini, Mistral-7B, Llama-3.1-8B, and Llama-2-13B, license required. Both are real, credible, and completely useless for us — fixed model rosters that don't include a single model we actually run, and both measure single-turn inference, not whether an agent can hold a task together across fifty tool calls.
The tools built for exactly our stack. llama-bench ships inside llama.cpp itself and is the de facto community standard for raw prompt-processing and token-generation speed. llama-benchy extends that same measurement style to any OpenAI-compatible backend, which is exactly why I flagged it as a clean win back in July and then never actually wired it into a bakeoff. OpenBenchmarking.org crowdsources llama.cpp results from anyone running the Phoronix Test Suite, opt-in, auto-rerunning until the variance settles. None of these three measure anything beyond throughput. None of them were ever going to replace the rubric. But none of them needed to — they're a layer underneath it that we skipped.
Our actual peer group. A Level1Techs forum member built a Strix Halo LLM benchmark harness from scratch, ran rigorous sweeps against the newest MoE architectures, and checked the full results into GitHub for the community to pick apart. That's the same move this blog makes every few weeks. It's worth saying plainly: "one hobbyist builds their own harness and publishes it" isn't a personal failing of mine or my agent's habits. It's the normal, respected way this exact community operates.
The layer that actually matters to us. Model Showdown isn't a throughput test — it's an agentic-correctness test, and that field looks different. Aider's Polyglot benchmark scores models on 225 of the hardest Exercism exercises inside Aider's own structured edit loop, with community-contributed results merged by PR. SWE-bench Verified grades a patch against a real GitHub issue and a human-validated test suite, though a notable complexity is that it evaluates the agentic harness and the underlying model together, which is exactly why different labs report different numbers for the same model. Terminal-Bench and METR's RE-Bench push further into containerized, tool-using, long-horizon work. A recent survey of the post-SWE-bench landscape put it about as bluntly as I'd put it myself: evaluation is moving toward harness portfolios that test repository repair, terminal execution, and tool-use reliability together, and any single benchmark is one signal, not the whole basis for a decision.
The Honest Answer
Split it in two, because the two halves of our own stack don't get the same grade.
The throughput layer: yes, we reinvented a wheel, and we knew it. llama-bench and llama-benchy already solve "how fast does this model run on this hardware," drop onto our existing endpoint with zero infrastructure change, and I identified llama-benchy specifically as a clean win in a research post back in July — and then the next Model Showdown round still reported informal token counting instead. That's not a close call. That's the bias, caught in the act, on our own blog.
The agentic layer: no, not cleanly, and it's not just stubbornness. Aider Polyglot is the closest existing match to what Model Showdown does, and it's still testing something narrower: isolated Exercism puzzles in a sandbox, one edit-and-retry loop, no real repo, no git, no Playwright, no fifty-turn session that can quietly go sideways. Terminal-Bench is closer in spirit but isn't wired to a Coder Agents chat and doesn't know our rubric. Adopting either wholesale would have meant building nearly as much integration code as our own harness already required, in exchange for a narrower signal. This is the same shape as the security audit work that's become a habit on this blog: sometimes the "buy vs. build" answer really is build, and the honest version of that answer names the specific reason instead of just defaulting to it.
What's Next
That split is a claim, not just a vibe, and claims like that should be checked with data instead of left as a nice paragraph. So the next step is an actual experiment: run llama-bench/llama-benchy against our own ad hoc timing on the same model, run a slice of Aider's Polyglot benchmark against two models from a recent Model Showdown round side by side with our own rubric, and keep an honest ledger of how much setup time each approach cost — because if wiring up someone else's harness takes longer than just running another homegrown round, that's evidence for the bias too, independent of whose numbers turn out to be more useful.
I'm predicting the existing tools win the throughput half outright and the homegrown rubric keeps finding things Polyglot structurally can't. I could be wrong about that. Next post will say so either way.
Back to the thesis. More soon.