Model Showdown Round 8: Sonnet 5, Opus 4.8, and Fable 5 Walk Into a Tag Manager

It started as a two-line request: update my homelab Coder instance to the latest version. It ended, several hours later, with three frontier Anthropic models building the same feature in parallel, a definitive answer on why Playwright can't be wired into Coder Agents, and a cost table that makes the case for Sonnet 5 louder than any benchmark I've run yet.

Here's the whole arc.

The upgrade that opened the door

The homelab Coder server was on v2.34.0. curl -L https://coder.com/install.sh | sh over SSH pulled v2.35.1, a systemctl restart coder, and I was current. Nothing interesting there — except that landing on v2.35.1 meant landing on a deployment where the AI model lineup had drifted since I'd last looked at it. Sonnet 5, Fable 5, and Opus 4.8 were all real, shipped models I hadn't provisioned yet, and Opus 4.6/4.7 and Sonnet 4.6 were sitting there stale.

That's the real starting point for this post: not the version bump, but what I found once I went looking at the model config.

Sonnet 5 that thought it was Sonnet 4.5

I configured Sonnet 5 and Fable 5 through the Coder Models UI, using the model IDs I had. Both showed up correctly in the chat picker. Neither behaved correctly:

Are you running sonnet 5/
I'm running on Claude Sonnet 4.5.

Fable 5 was worse — it didn't even know what it was: "I'm one of Anthropic's Claude 4-series models."

The model IDs were right. What was missing was the provider_options block — effort, thinking.budget_tokens, thinking_display — that actually engages the model's current-generation behavior on Anthropic's side. Without it, the request apparently falls back to older default routing.

I didn't have to guess at the correct shape. A teammate's Coder instance had the identical models configured and working, so I pulled the exact JSON straight out of the browser's network tab — every model config on their deployment, cost rates included — and used it as ground truth:

{
  "model": "claude-fable-5",
  "display_name": "Fable 5 Medium ($$)",
  "model_config": {
    "provider_options": {
      "anthropic": {
        "send_reasoning": true,
        "effort": "medium",
        "thinking_display": "summarized",
        "web_search_enabled": true
      }
    }
  }
}

I PATCHed the homelab's model configs to match — Sonnet 5, Fable 5, and a freshly added Opus 4.8 — then deleted Opus 4.6, Opus 4.7, and Sonnet 4.6, which weren't earning their place in the lineup anymore. Final roster: three current-generation Anthropic models, all correctly self-identifying, all with real thinking budgets.

That's a config problem solved. It also happened to leave me with three fresh, uncompared models sitting on the same deployment. So: bakeoff.

The plan that had to change twice

I already had a Round 8 plan on the books — a reproducibility test, re-running Opus 4.7 and Sonnet 4.6 against a new Opus 4.8 on the same task from Round 5. That plan died the moment I deleted Opus 4.7 and Sonnet 4.6 from the deployment. There was no "incumbent" left to hold constant.

New plan: three-way horse race — Sonnet 5, Fable 5, Opus 4.8 — same rubric, no reproducibility angle, task TBD.

The obvious task was Round 5's admin/images feature. Except a screenshot check turned up an Images tab live in the real /admin nav — the feature had already shipped to main since I'd last planned this round. Good problem to have, bad task to reuse. I switched to the tag manager task instead — planned back in Round 9, never executed, still absent from main. Confirmed clean baseline at commit f6b713f, cut three branches (run-1, run-2, run-3), sealed a random run-to-model mapping, and wrote three identical task prompts that differed only by branch name.

Three isolated chats, one identical prompt

Goal: add a tag manager to /admin.

Requirements:
- lib/tags.ts to read all tags from published and draft posts
- GET /api/admin/tags with per-tag post counts
- PUT /api/admin/tags/{tag} to rename across all posts
- DELETE /api/admin/tags/{tag}
- /admin/tags page with inline rename/delete
- Link from the /admin nav
- Screenshot via Playwright MCP
- npm run build must pass before committing
- Commit in logical chunks, push when done

Three Coder Agents chats, three models, launched in parallel, no cross-awareness. I sat back and let them cook.

The wall all three models hit at the same spot

An hour later, all three chats were sitting in a waiting state. Pulling the transcripts, every single one had:

1. Built the entire feature
2. Passed npm run build
3. Committed in logical chunks
4. Hit the screenshot requirement — and discovered Playwright wasn't available as a callable tool
5. Burned real effort improvising a workaround (npm install playwright, playwright install --with-deps chromium, hunting missing system libs)
6. Successfully produced and attached a real screenshot
7. Stopped, without ever running git push

My first assumption was a hard message cap — the API's default page size (50) made it look like every chat had been cut off at exactly the same length. Re-querying with a higher limit corrected that: real totals were 92, 98, and 116 messages. No cap exists. What's real is that all three models, independently, treated "I've shown you visual proof it works" as a natural point to pause and wait for a human, even though the prompt's last bullet point was explicitly "push when done."

That's a genuine, reproducible finding, and it says nothing about which model is better — it happened identically regardless of which model was running.

Why Playwright can't actually be fixed here

I went looking for a real fix and found a structural wall instead. Registering an MCP server on this Coder deployment only accepts transport: streamable_http or sse:

{"message":"Validation failed.","validations":[
  {"field":"transport","detail":"Validation failed for tag \"oneof\" with value: \"stdio\""}
]}

Playwright MCP is stdio-only by design — it drives a local browser, so it has to run as a local process. There's no hosted, remote Playwright MCP endpoint to point a streamable_http transport at. This isn't a misconfiguration I introduced; it's baked into how Coder Agents' native tool-calling and Playwright's transport model don't overlap. My own workspace template even documents an adjacent version of this exact gap — the fitness-tracker and vibescoder MCP servers are wired up over raw HTTP specifically "because the Coder Agents chat doesn't auto-register MCP servers from ~/.mcp.json."

The real fix isn't config — it's environment. Pre-baking Playwright, Chromium, and its system dependencies into the workspace Docker image would let agents skip the entire live-provisioning step and go straight to taking the screenshot. That's a Dockerfile change I'm holding off on until I've thought it through further.

Finishing the run

Since all three chats were genuinely paused, not dead, I nudged them — identically, via the same API the chat UI itself uses:

curl -X POST '.../api/experimental/chats/{id}/messages' \
  -H "Coder-Session-Token: $TOKEN" \
  -d '{"content": [{"type": "text", "text":
       "Continue — run npm run build one more time to confirm it still passes,
        then push the run-N branch now."}]}'

All three finished within a minute of the nudge. All three branches pushed clean, independently re-verified with a fresh npm install && npm run build in isolated git worktrees.

Blind scoring

Same 7-dimension rubric the series has always used, scored before I knew which run was which model.

Tightest field this series has produced — 0.10 points from top to bottom.

run-1 had the only explicit error-handling path around its live-GitHub data fetch, and the cleanest separation between pure content-transform functions and the API route orchestrating them. Weakest commit hygiene — two coarse commits against the other two runs' four.

run-2 reasoned explicitly, in its own code comments, about a staleness/race condition — listing tags from a fast build-time snapshot but re-fetching each post fresh from GitHub right before writing, "so we never clobber a concurrent edit." After the nudge, it also explicitly verified branch safety before pushing rather than just complying. Best commit hygiene.

run-3 had the cleanest code of the three — a single shared mutateTag helper factoring rename and delete, no duplication. It also added publishedCount/draftCount tracking per tag that nobody asked for and the UI never surfaces.

The reveal

run-1: Opus 4.8
run-2: Sonnet 5
run-3: Fable 5

Sonnet 5 wins outright.

The cost story is not close

Sonnet 5 scored highest and cost roughly a quarter of Opus 4.8 and a quarter of Fable 5. That's the whole argument, in one table.

The Fable 5 result deserves a closer look, though, because the obvious story — "it overthought the problem" — isn't what the data shows. Fable 5 actually used the fewest tokens of the three in every category: lowest output, lowest cache write, lowest cache read. Its cost is high purely because its per-token pricing is 4-5x Sonnet 5's. So "too thoughtful" isn't about verbosity. It's about where the thinking went. Fable 5 tied or trailed in every heavily-weighted category — Correctness, Engineering judgment, Scope discipline, Surprise, 55% of the rubric combined — and only led outright in Code quality. Its one real differentiator, the shared mutateTag helper, was genuine skill. Its other distinguishing move, the unused published/draft counters, was polish spent where nobody was looking, while Opus 4.8 and Sonnet 5 spent their effort on error handling and safety reasoning respectively — both of which paid off in more heavily-weighted categories. That's a read on one run each, not a verdict on Fable 5 as a model.

Two notes on how this bakeoff itself got run

This round's entire orchestration — model provisioning, branch creation, chat creation, telemetry pulls, the nudge, blind scoring — was driven by an assistant SSH'd into the homelab from inside a Docker-hosted Coder workspace, rather than done by hand through the browser. Worth being explicit that this doesn't compromise the experiment: the three contestant models still worked in fully isolated chats with zero cross-awareness, and every consequential step — model config changes, anything touching main, the mid-run nudge — still went through explicit human sign-off before it happened. What changed was orchestration speed, not experimental integrity.

The more interesting discovery, for future rounds: the Coder chat API accepts posted messages directly at POST /api/experimental/chats/{id}/messages — the exact same endpoint the browser UI itself calls. I only used it this round to send the resume nudge, but there's no reason the entire "paste the prompt into three browser tabs" step couldn't be scripted too — create the chats, assign the models, post the warmup and task prompt, poll for completion, nudge anything that stalls. Next round, I'd like to make the whole thing closer to a one-command operation.

What's next

The local model track (Round 9: dense vs MoE architecture, testing whether last round's MoE screenshot-spiral was structural or a bad day) is still on the board, unrelated to this round's findings. And the Playwright-in-Docker fix is a real, scoped piece of work I want to think through properly before touching the workspace image.

By the Numbers

• 3 frontier models benchmarked: Sonnet 5, Opus 4.8, Fable 5
• 0.10 rubric points separated 1st place from 3rd — tightest field in the series
• 4.40 — Sonnet 5's winning weighted score
• $0.45 — Sonnet 5's cost per rubric point, the cheapest by roughly 2.3x
• $7.84 — Fable 5's total cost on this task, despite using the fewest tokens of the three
• 0 — MCP servers registered on the deployment before this round; Playwright cannot be one, ever, on this architecture
• 3 of 3 models independently paused at the identical step (screenshot attached, before push) — a disposition finding, not a per-model one
• 1 API call (POST .../chats/{id}/messages) that turned a stuck bakeoff back into a finished one