# vibescoder.dev — Full Content Index > Building in public with AI agents. A technical blog by Rob Whiteley, CEO of Coder. > This file contains all published blog posts in plain text for AI agent consumption. > Homepage: https://vibescoder.dev > RSS: https://vibescoder.dev/feed.xml --- ## The Homelab Redesign: One Spark Cluster, Two Agentic Workstations, and a PAIR - URL: https://vibescoder.dev/posts/the-homelab-redesign-one-spark-cluster-two-agentic-workstations-and-a-pair - Date: 2026-09-14 - Tags: #homelab #ai #llm #agents #building-in-public - Reading time: 5 min read An architecture plan for folding a dual-node DGX Spark cluster and NVIDIA PAIR into the homelab, turning the RTX 5090 box and an M4 Mac mini into agentic workstations instead of the only inference engines in the house. --- [Last post](/posts/nvidia-pair-looks-promising-for-multi-user-homelabs) ended on a tease, so here is the plan it was teasing. I am redesigning this homelab around a cluster instead of a box. Two DGX Sparks bonded into one server, PAIR running across everything, and the RTX 5090 and my wife's Mac mini demoted from inference engines to agentic workstations. Demoted is the wrong word. Promoted, actually, since neither one will be waiting on a GPU queue anymore. ## Two Sparks Bonded into One Server The Sparks are not going in as two separate PAIR nodes. [NVIDIA's own clustering path](https://docs.nvidia.com/dgx/dgx-spark/spark-clustering.html) bonds a pair of Sparks over a single cable using MPI and NCCL, and the combined 256GB of unified memory is the whole reason this plan exists. I want to run something in [DeepSeek V4 Flash's](/posts/the-local-vibe-coders-dream-deepseek-v4-flash-on-dgx-spark-mac-studio-or-strix-halo) weight class losslessly, and a single 128GB Spark cannot hold that build with any room left for context. Bonded, the pair presents as one inference target, not two independent ones I would need to keep in sync with the same model on both. That decision closes off the other path I was weighing after [the last post](/posts/nvidia-pair-looks-promising-for-multi-user-homelabs), running two independent Spark nodes each holding a smaller model so concurrent subagent requests spread across both on their own. That version is real and PAIR would handle it well. It is just not this version. I am building for one large lossless model first, not for maximum concurrency across small ones. ## Why PAIR Still Matters on a One Node Cluster A bonded pair of Sparks looks like a single node to PAIR, so a reasonable question is why PAIR is in this plan at all if there is nothing yet to route between. The answer is the [Mac Studio](https://www.apple.com/mac-studio/) I have not bought yet. Once that machine joins, whether as its own model or as a second independent target for a smaller workload, I do not want to touch a single line of configuration on either agentic workstation to make that happen. PAIR's whole design point is that the address an application uses never changes, no matter which paired machine ends up serving the request. I would rather pay the setup cost once, now, while the cluster is simple, than retrofit routing later while something is already depending on it. ## AI-NT-No-Problem and the Mac Mini Become Clients Not Compute Here is the role split. [AI-NT-No-Problem](/posts/ai-nt-no-problem-cramming-a-9950x3d-and-rtx-5090-into-an-sff-custom-loop), the RTX 5090 box, and my wife's [M4 Mac mini](https://www.apple.com/mac-mini/) both run PAIR, but neither one runs an inference engine of its own once this is live. They are cluster members with no GPU work assigned to them. Hermes, OpenClaw, whatever agent harness is running on either machine, keeps calling its own local address exactly like it does today. PAIR on that machine forwards the request across the network to the Spark pair, waits for the response, and streams it back. The agent never learns anything changed. That is the actual payoff of this redesign. The 5090 stops queueing behind its own inference workload and stays free for whatever else it is doing, gaming, content creation, agent orchestration overhead. The Sparks, headless machines with no desktop session to speak of, run [PAIR's terminal interface](https://github.com/NVIDIA/Personal-AI-Router/blob/main/docs/terminal-interface.mdx) instead of the desktop app and do nothing but serve models all day. ## The Open Question We Cannot Research Our Way Out Of PAIR proxies Ollama and LM Studio. It does not proxy [vLLM](https://github.com/vllm-project/vllm). NVIDIA's own guidance for serving a model across a bonded Spark pair points at vLLM, and [one independent benchmark](https://ai-muninn.com/en/blog/dgx-spark-vllm-vs-ollama-same-model) I found put vLLM roughly 30 percent faster than Ollama on the same model on a single Spark, before clustering even enters the picture. That gap has not gone unnoticed by the people actually building PAIR. NVIDIA's own repository already carries [an open pull request adding vLLM as a third engine](https://github.com/NVIDIA/Personal-AI-Router/pull/9), fully tested, still waiting on maintainer review, with a follow-on PR stacked on top of it to [add SGLang as a fourth](https://github.com/NVIDIA/Personal-AI-Router/pull/50). Three separate contributors have independently opened pull requests adding some flavor of llama.cpp support: [adopt-only against a proxy](https://github.com/NVIDIA/Personal-AI-Router/pull/18), [a second adopt-only attempt](https://github.com/NVIDIA/Personal-AI-Router/pull/41), and [a PAIR-installed engine via llama-server's router mode](https://github.com/NVIDIA/Personal-AI-Router/pull/36). None of the four has merged. The demand to run something other than Ollama is visible in the repository itself. The answer just is not in main yet. I still do not know whether Ollama can serve a model distributed across an NCCL bonded pair the way vLLM does, and none of those open pull requests touch that specific combination either. That is not a question I can answer by reading more pull requests. It needs the actual hardware, the actual bonded pair, Ollama attempted first because it is what merged and shipped, then vLLM measured against it to see what the real throughput cost of choosing PAIR compatibility turns out to be today. That test is the next post, once the Sparks are actually racked. *Is a router worth building around before I know whether the engine it requires can even do the job I am buying the hardware for?* ## By the Numbers - **2** DGX Sparks, bonded over NCCL into one 256GB unified memory target - **1** cluster node PAIR sees where the bonded pair sits, not two - **2** machines demoted from inference engines to pure agentic workstations, the RTX 5090 box and the M4 Mac mini - **1** Mac Studio not yet purchased, the reason PAIR is in this plan before it is strictly needed - **2** inference engines PAIR supports today, Ollama and LM Studio, neither of which is the vLLM stack NVIDIA recommends for the bonded pair - **4** open pull requests adding a third or fourth engine to PAIR, none merged - **~30%** — one independent benchmark's throughput gap, vLLM over Ollama, on a single Spark before clustering - **0** lines of client configuration I want to touch when the Mac Studio eventually joins === ## NVIDIA PAIR Looks Promising for Multi-User Homelabs - URL: https://vibescoder.dev/posts/nvidia-pair-looks-promising-for-multi-user-homelabs - Date: 2026-09-11 - Tags: #homelab #ai #llm #agents #open-source #building-in-public - Reading time: 5 min read NVIDIA's new Personal AI Router turns a pile of home GPUs into one shared endpoint for multi-agent inference. A look at what it actually does, why it fits a bottleneck already visible on this homelab, and what it rules out. --- I opened NVIDIA's [developer blog](https://developer.nvidia.com/blog/nvidia-pair-virtual-inference-router-expands-available-compute-on-your-local-network/) on September 3rd expecting another driver update and found a router instead. Not a network router. A router for inference requests, built to sit between my agents and whichever GPU in the house happens to be free. I read it twice, then went back to my own posts from the last few weeks, because I already knew the exact problem it claims to solve. I wrote about that problem myself, twice. ## PAIR Turns Ollama and LM Studio into One Address [NVIDIA Personal AI Router](https://docs.nvidia.com/local-ai/nvpair/), PAIR for short, is not a new inference engine. [Ollama](https://ollama.com) or [LM Studio](https://lmstudio.ai) still does the actual work. PAIR is the layer in front of them. Install it on every machine you want to contribute, pair those machines together over your local network, and PAIR gives you one address on the machine in front of you. Point an agent at that address the same way you always have. PAIR decides, request by request, which paired machine actually serves it. The mechanics are straightforward once you separate them from the marketing. Discovery runs over mDNS. Trust is a deliberate step, a six digit PIN typed on the machine you are inviting, and every byte between paired nodes travels over mutual TLS afterward. A node becomes eligible for a given request only when it is online, running a supported engine, and already holding the exact model requested. The scheduler then picks one eligible node based on current load, and that node runs the whole request start to finish. PAIR does not merge GPUs, does not pool VRAM, and does not split one model or one request across machines. It is deliberately narrow. It only decides where the next independent job goes. ## The Five-Subagent Demo Is My Own Bottleneck NVIDIA's own demo is what sold me. They ran [Hermes Desktop](/posts/hermes-agent-first-contact) against a synthetic household task, split five ways into subagents, entirely on Ollama. On one RTX Spark laptop the run took 18 minutes. Add a [DGX Spark](https://www.nvidia.com/en-us/products/workstations/dgx-spark/) and an [RTX 5090](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/) to a PAIR cluster, same task, same subagent count, and it finished in 8 minutes 48 seconds. I have been circling this exact shape of problem since I wrote about [why two agents beat one](/posts/why-two-agents-are-better-than-one-for-now) on this homelab. My own conclusion was architectural: coding agents are ephemeral and invoke driven, while home automation agents like [OpenClaw](/posts/installing-openclaw-on-the-homelab) are persistent and event driven. What I did not have an answer for was what happens when either kind spins up several subagents at once and they all queue behind the same GPU. PAIR is a direct answer to that specific gap, and it works with the exact engines I would already be running, no new API for an agent harness to learn. ## Strix Halo Just Lost Its Shot at the Third Box I have been debating a third machine for weeks, a dedicated box sized to run something like [DeepSeek V4 Flash](/posts/the-local-vibe-coders-dream-deepseek-v4-flash-on-dgx-spark-mac-studio-or-strix-halo). The candidates were NVIDIA's DGX Spark, an [Apple Mac Studio](https://www.apple.com/mac-studio/), and [AMD's Ryzen AI Max+ 395](https://www.amd.com/en/products/processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-max-plus-395.html), better known as Strix Halo. PAIR's [supported hardware list](https://github.com/NVIDIA/Personal-AI-Router#what-is-supported) settles part of that debate before I spend a dollar. It covers NVIDIA GeForce RTX 20 series and newer, RTX PRO workstation GPUs, DGX Spark, and Apple silicon starting at M4. AMD is not on the list. No ROCm path, no ambiguity to argue with. That single fact rules Strix Halo out of a PAIR based homelab entirely, independent of anything I already found about its prefill gap against a Spark. If I want a router across my own hardware, the third box has to be a Spark or a Studio. ## One GPU Today a Router with Nothing to Route To Here is the honest state of this homelab right now. One box does all the local inference, the RTX 5090 in [AI-NT-No-Problem](/posts/ai-nt-no-problem-cramming-a-9950x3d-and-rtx-5090-into-an-sff-custom-loop), confirmed as recently as this week when [an agent audited every piece of software running on it](/posts/friday-fixes-the-agent-audits-the-homelab-then-updates-everything-itself). My wife runs her own separate homelab on an [M4 Mac mini](https://www.apple.com/mac-mini/), which happens to already qualify as a PAIR node. Neither fact adds up to a cluster today. A router with one machine to route to is not doing anything yet. PAIR also only proxies Ollama and LM Studio. My current inference stack is a [hand tuned llama-server running directly under systemd](/posts/model-showdown-round-3-the-llamacpp-showdown), the exact setup I fought to get right in earlier posts, flags and all. Nothing about PAIR touches that setup without a decision to run a different engine somewhere in the mix. And PAIR itself is young, [open source under Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0), [born in July](https://github.com/NVIDIA/Personal-AI-Router/releases), still in beta, with no fixed roadmap NVIDIA has committed to yet. None of that changes what I want to build next. It changes what building it actually requires. *What would you offload first if every GPU in your house suddenly answered to one address?* ## By the Numbers - **18 minutes to 8 minutes 48 seconds** — NVIDIA's own five subagent demo, one RTX Spark laptop versus a three device PAIR cluster - **2** inference engines PAIR proxies, Ollama and LM Studio, and nothing else - **0** AMD systems on PAIR's supported hardware list, which is what rules out Strix Halo - **1** GPU currently doing all local inference on this homelab, the RTX 5090 in AI-NT-No-Problem - **1** node that already qualifies for a future cluster without buying anything, my wife's M4 Mac mini - **~2 months old** — PAIR's age as an open source project, created in July 2026 - **1,115 stars / 190 forks** on the GitHub repo as of this post === ## Splitting the Bakeoff: Coding Gets Its Own Test, and It's Terminal-Bench Over SWE-bench - URL: https://vibescoder.dev/posts/splitting-the-bakeoff-coding-gets-its-own-test-terminal-bench-over-swe-bench - Date: 2026-09-07 - Tags: #agents #homelab #llm #benchmark #model-showdown #building-in-public - Reading time: 8 min read Round 2 of the local agent bakeoff saturated its coding domain completely, zero cracks across 18 fresh evals from two coding-focused models. That's the moment to stop patching a broken test and split the question in two: what's the best Home Assistant butler, and separately, what's the best local coding model that fits a 32GB card. Here's the research behind picking Terminal-Bench 2.1 over SWE-bench, and the plan for the actual run. --- Round 2 of the [local agent bakeoff](/posts/local-agent-bakeoff-granite-4-2-is-the-new-homelab-king-turn-thinking-off-to-speed-it-up-10x) closed with a clean leaderboard and one loose thread. Both coding tasks, a portfolio-drift-flagging script and a calendar-conflict detector, scored a flat 1.000 across every model, every run, all 18 fresh evals from two models built specifically for agentic coding. Round 1 at least cracked a little on the third run. Round 2 didn't move at all. That's not a compliment to the models. It's a broken test. Before running any more of these bakeoffs, it's worth stopping to plan the fix properly instead of bolting on another hand-rolled task and hoping it holds up longer. This is that plan, not the results. The actual eval run is a separate post, the same way [the harness deep dive followed the round 1 results](/posts/how-we-got-here-building-the-test-harness-behind-the-local-agent-bakeoff) instead of getting rushed into the same piece. ## Why Coding Doesn't Belong in the Home Assistant Butler Score The bakeoff has always been built around one real question: which local model should run my smart home, my calendar, my portfolio, and my to-do list. Those four domains share a common thread. They all require judgment under ambiguity, deciding whether to act, ask, or refuse, picking the right tool out of several plausible ones, extracting the right arguments from a loosely specified request. Coding is a different kind of task entirely. A generated script either produces the correct output or it doesn't. There's no ambiguity to navigate, just correctness to verify. Folding a different kind of question into the same equal-weighted average has been quietly doing damage for two rounds now. Laguna XS 2.1 and Ornith 1.5 are both coding specialists by training focus, and both landed below the group's middle on the domain that's actually this assistant's job, Home Assistant. Their coding score, now saturated at a perfect 1.000 for both, was doing nothing to separate them from each other or from anyone else, while still counting for a sixth of their aggregate. A domain that can't discriminate shouldn't get a vote. The fix isn't a harder hand-rolled task. [The last coding domain build already turned up a real bug](/posts/how-we-got-here-building-the-test-harness-behind-the-local-agent-bakeoff), a floating-point boundary case that lied about being exact, in a test written from scratch with no independent validation. Making that same kind of task harder just means re-earning trust in new ground truth I wrote myself, with no outside check on whether the difficulty curve lands anywhere useful. Better to answer the coding question with a benchmark built and maintained by people who do nothing else, and answer the Home Assistant question with the four domains that were always the actual point. ## Terminal-Bench and SWE-Bench Answer Different Questions Two names come up in almost every coding-model card these days, including the ones for this round's contestants. [Terminal-Bench 2.1](https://www.tbench.ai/) and [SWE-bench](https://arxiv.org/abs/2310.06770) both claim to measure real coding capability, and both get cited constantly, but they're not measuring the same thing. SWE-bench is the more established name. SWE-bench focuses on resolving GitHub issues inside existing software repositories, and its curated Verified subset consists of 500 software engineering tasks rigorously validated by human expert developers. That's real, industry-recognized rigor, and it's why so many model cards lead with a SWE-bench number. It's also narrow by design, one specific and very common job, patch a bug in someone else's codebase, evaluated at real scale. Terminal-Bench takes a wider view. Terminal-Bench 2.1 evaluates a broader range of technical work than SWE-bench, including workflows such as configuring services, compiling software, training models, processing data, repairing security problems, and reproducing scientific results inside terminal environments. That's a closer match to what actually happens on a homelab box day to day, not just fixing someone else's Python. The setup cost is where the two really diverge. SWE-bench needs a working environment for every one of its 500 instances, real repo checkouts with real dependency graphs, environment initialization specific to SWE-Bench, which includes removing future commits to prevent data leakage, as well as configuring network proxies and critical system settings. Terminal-Bench keeps the task count smaller and the environments self-contained. Terminal-Bench v2.1 is a verified refresh of the v2.0 agentic terminal benchmark, keeping the same 89 curated tasks, and it comes with its own purpose-built execution framework: researchers can run Terminal-Bench 2.1 with Harbor, the open-source evaluation framework used to execute tasks in containerized environments. Neither one is a small lift compared to the subprocess-in-a-tempdir scorer this bakeoff has used so far. Both need real container orchestration. But 89 self-contained tasks is a much more tractable target for one RTX 5090 than 500 tasks each needing their own repo-specific setup. One number worth sitting with before committing to either: frontier models and agents score less than 65% on the benchmark, referring to Terminal-Bench 2.0's own difficulty calibration. A 30B-class quantized local model could easily crater toward single digits on either benchmark, trading today's ceiling problem (everyone scores 1.000) for a floor problem (everyone scores near zero). That risk applies equally to both, and it's the reason a pilot run comes before a full commitment, more on that below. ## Why Terminal-Bench 2.1 Wins the Call Three things tip this toward Terminal-Bench over SWE-bench for this specific project, not as a universal ranking, just for what this blog actually needs right now. **It's the more homelab-shaped benchmark.** A homelab box does a mix of sysadmin work, service configuration, and the occasional script, not exclusively "resolve issues in a large open-source Python codebase." Terminal-Bench's broader task variety, compiling, data processing, security fixes, matches that mix better than SWE-bench's narrower GitHub-issue framing. **It's already the reference point for this round's actual candidates.** Both Laguna XS 2.1 and Ornith 1.5 publish Terminal-Bench numbers on their own model cards. Running it locally means checking a real, comparable claim against the exact quantized weights sitting on this rig, not just trusting a vendor's full-precision number on different hardware. **It's actively maintained, and that maintenance is visible.** v2.1 incorporates environment and instruction fixes, patched Dockerfiles and corrected instruction-test mismatches across roughly a dozen tasks, so scores reflect agent capability rather than environment gaps. That's a benchmark maintainer catching and fixing the exact class of bug this blog hit building its own coding tasks from scratch. Trusting someone else's test more than my own, on this specific point, is the honest call. SWE-bench isn't wrong, and it isn't going in the trash. It's the more universally recognized name, and if a coding-focused model ever earns a dedicated deep dive, running it against SWE-bench Verified too would make the result legible to a much wider audience. But that's a future nice-to-have, not the thing to build first. ## The Plan for the Actual Run This is the plan, written down before any of it happens, the same discipline the [merge-plan pattern from the feature bakeoffs](/posts/showdown-thoughts-the-three-pass-pattern) argues for: decide the approach on paper first, then let the run confirm or correct it. **Step 1, stand up Harbor.** Get the containerized evaluation framework running against the homelab's existing llama-swap endpoint, using the Terminus 2 scaffold that Terminal-Bench's own leaderboard runs use. This is pure infrastructure, no scoring yet. **Step 2, pilot on a handful of tasks before committing to all 89.** Given the floor-effect risk above, the first real test isn't a full run. It's two or three tasks, picked for a range of difficulty, to confirm a 30B-class quantized local model produces scores that actually land somewhere between 0 and 100%, not uniformly at one end. If the floor effect shows up here, that's the moment to reconsider scope, not after burning a full 89-task pass on every candidate. **Step 3, run it against the models that actually claim coding strength first.** Laguna XS 2.1 and Ornith 1.5 both trained specifically for agentic coding and both already publish Terminal-Bench numbers, so they're the natural first pass, the two data points this whole project exists to check. Granite 4.2, Qwen3.8-27B, and the rest of the Home Assistant-focused roster can follow once the harness is proven out, since a coding benchmark is still worth knowing even for a model that isn't marketed on it. **Step 4, report it as its own leaderboard, not a folded-in column.** Whatever comes out of this gets its own table, its own post, and its own verdict, kept explicitly separate from the equal-weighted Home Assistant butler score. Two questions, two numbers, no more diluting one with the other. None of this touches the four-domain personal-assistant battery going forward. That test stays exactly as it is, minus the two coding tasks it's carrying today, and it's what future bakeoff rounds will keep running. ## By the Numbers - **0** cracks in round 2's coding-task ceiling, across 18 fresh evals from 2 coding-focused models, the finding that triggered this whole plan - **89** curated tasks in Terminal-Bench 2.1, versus **500** in SWE-bench Verified - **2** models with a real reason to go first in the pilot, Laguna XS 2.1 and Ornith 1.5, both already publishing Terminal-Bench numbers on their own model cards - **<65%** — frontier models' own reported ceiling on Terminal-Bench 2.0, the number that justifies a small pilot run before a full 89-task commitment - **1** floating-point ground-truth bug already found in this blog's own hand-rolled coding tasks, the reason a self-maintained benchmark keeps losing to an externally validated one - **4** domains staying in the Home Assistant butler battery going forward, Home Assistant, calendar, portfolio, and to-do, now that coding has its own separate test - **2** benchmarks researched, **1** picked, for reasons specific to this rig and this project, not a universal verdict on either one === ## Friday Fixes: The Agent Audits the Homelab, Then Updates Everything Itself - URL: https://vibescoder.dev/posts/friday-fixes-the-agent-audits-the-homelab-then-updates-everything-itself - Date: 2026-09-04 - Tags: #homelab #self-hosted #agents #coder #llm #building-in-public - Reading time: 5 min read A random Tuesday sends an agent over SSH into the homelab to check every piece of software running there. Coder jumps two full versions, cloudflared and RustDesk get patched, and llama.cpp pulls in 317 commits, all with zero manual steps. --- I wasn't planning to touch the homelab today. Or at least, not beyond Coder. I knew Coder had shipped a new version and asked the agent to check it out. Instead of a one-line answer, it SSH'd into [AI-NT-No-Problem](/posts/ai-nt-no-problem-cramming-a-9950x3d-and-rtx-5090-into-an-sff-custom-loop), the box with the RTX 5090 in it, and ran a full audit of every piece of software on there. Then, with my go-ahead, it updated all of it. Start to finish, the actual command work took well under ten minutes. I didn't touch a keyboard. What turned into a routine tool upgrade led to some interesting findings in both Coder and llama.cpp. ## The Audit Turns up Five Things Worth Fixing The agent checked apt packages, Docker, the Coder server, cloudflared, RustDesk, the NVIDIA driver, and the local llama.cpp build against their upstream releases. Most of it was current. Five things weren't. | Software | Before | After | Risk | | --- | --- | --- | --- | | [Coder](https://github.com/coder/coder/releases) | v2.36.0 | v2.37.0 | Breaking changes, but all in features I don't use | | apt (20 packages) | mixed | latest | Routine, mostly security patches | | [cloudflared](https://github.com/cloudflare/cloudflared/releases) | 2026.8.2 | 2026.8.3 | Patch only | | RustDesk client | 1.3.9 | 1.4.9 | Minor version jump, no breaking changes | | [llama.cpp](https://github.com/ggml-org/llama.cpp) | Aug 14 build | current master | 317 commits, no breaking changes for my setup | Docker Engine, the RustDesk server container, Terraform, and the NVIDIA driver were already current, so those got a pass. The apt batch pulled in `containerd.io`, `docker-compose-plugin`, `tailscale`, and a Chrome bump, among others. All restarted cleanly, including Docker itself, which also runs the container this very workspace lives in. ## Coder's Free Features Are Becoming Premium Features and That's Fine The jump to v2.37.0 carries more breaking changes than one might expect for a single minor version. Chat model configs are now scoped to organizations instead of global. MCP server configs got RBAC. Native chat usage limits and cost tracking are gone, replaced by [AI Gateway](https://coder.com/docs/ai-coder/ai-gateway) budgets. `login_type=none` is deprecated. None of that is a surprise. Coder has been telegraphing for months that AI governance features built and shipped in beta would eventually move behind licensing, the same way most of their enterprise features work. This release just commemorates that in the changelog. The part worth noting is how little of it actually touches individual users. It's a no-op in the best of ways for homelabbers. Org-scoped chat models assume multiple organizations. RBAC on MCP configs assumes multiple people with different permissions. Budget enforcement assumes a team spending real money across many users. I run one organization, two users, one license-free deployment. That entire surface area, the whole point of most of this release, just doesn't apply. Two things in the release are genuinely useful regardless of team size. [Bulk secret import](https://coder.com/docs/admin/security/secrets) means I can stop [creating Coder secrets one at a time](/posts/updating-coder-to-get-user-secrets-and-the-art-of-knowing-where-your-secrets-belong) if I ever need to reseed a workspace. And a built-in browser tab for live previews inside the dashboard could replace the separate tunnel tab I usually keep open when checking a build. Coder also migrated its MCP servers, coderd, the CLI's stdio server, and the agent client, to the official MCP Go SDK, which is the kind of internal plumbing change that quietly makes tool calls more reliable without showing up as a headline feature. ## Llama.cpp's 317 Commits Land Right Where the GPU Needs Them The homelab's `llama-embed` service runs nomic-embed-text v2-moe, a mixture-of-experts embedding model. One of the changes in these 317 commits extends CUDA's MOE fusion to speculative decoding and removes a restriction that limited MOE-GLU and topk-router fusion to a single token at a time. That's not a change that helps some hypothetical model I might run someday. It's a change that speeds up the exact model already [doing embedding work on this GPU](/posts/putting-the-gpu-to-work-running-local-llms) right now. The rest of the pull is a mix of new model support and smaller fixes. Qwen3.8-Flash-Next, DeepSeek V4, and Nemotron3.5 are now loadable, which gives me new candidates the next time I run a [model showdown](/posts/glm-is-the-new-hotness-so-lets-test-it-on-the-homelab) on the local bakeoff harness. Quantizing large downloads locally now caps working memory instead of loading oversized tensors straight into RAM. The server picked up a `--kv-unified-per-slot` option for `llama-swap`'s context handling, and the old `--tensor-read-lazy` flag got renamed to `--lazy-mode`, worth remembering if any config still references the old name. I rebuilt from source with CUDA enabled, same as before, and restarted both `llama-embed` and `llama-generate`. Both came back active on the first try. This is the homelab I wanted when I started this. Not a system I babysit, one I ask a question and it goes and finds out, then fixes what it finds, without needing me to type a single SSH command myself. ## By the Numbers - **317** commits pulled into llama.cpp between the previous build and current master - **20** apt packages upgraded in a single batch, including containerd, tailscale, and Chrome - **2** full Coder minor versions skipped, from v2.36.0 straight to v2.37.0 - **2** llama.cpp services restarted after the rebuild, `llama-embed` and `llama-generate` - **1** GPU doing all the local inference work, an RTX 5090 - **0** manual commands typed by hand === ## Thursday Thoughts: GitHub Is Cracking Under the Weight of AI - URL: https://vibescoder.dev/posts/thursday-thoughts-github-is-cracking-under-the-weight-of-ai - Date: 2026-09-03 - Tags: #meta #building-in-public #agents #future-of-coding #ai #open-source - Reading time: 7 min read Cursor's new forge product Origin reveals why GitHub's human-centric architecture is struggling to keep pace with agent-driven development, and what it would take for a new entrant to actually displace the incumbent. --- A few people have asked me lately what I think the future of GitHub looks like. Cursor releasing their new forge product, Origin, brought this question back into focus. It's a genuinely interesting one, and I don't think most people have sat with it long enough. GitHub was built for humans, entirely for humans. The architecture assumes a relatively slow rate of change, a manageable number of commits, and a person deciding at every meaningful step. When GitHub added webhooks, Actions, and CI/CD over the years, it didn't rethink that foundation. It just bolted new pieces on. That worked fine when humans wrote code. It's not working as well now. ## Github's Architecture Was Never Designed for This Volume We're in the middle of the same shift that happened when companies moved from on-premises infrastructure to the cloud. Back then, plenty of organizations just lifted their old workflows into cloud environments and called it a transformation. It wasn't. The tools looked different but the thinking hadn't changed, a pattern I saw again from the [AI-native side](/posts/thursday-thoughts-how-ai-native-mirrors-cloud-native). That's what happened with GitHub and agents: human workflows with agents dropped in. GitHub wasn't built for this velocity, and the numbers say so. Pull requests opened by AI agents jumped from roughly [4 million to 17 million](https://www.danilchenko.dev/posts/2026-04-11-github-ai-agents-pull-requests/) between September 2025 and March 2026. Weekly commits reached 275 million, on pace for roughly 14 billion in 2026, about 14 times 2025's total. An analysis by LeadDev counted [257 incidents](https://venturebeat.com/infrastructure/cursor-launches-origin-code-hosting-platform-as-github-outage-exposes-opening-in-ai-coding-race) between May 2025 and April 2026, 48 of them major, and GitHub's own CTO, Vlad Fedorov, has said the platform "wasn't built for the scale it's now being asked to handle" and must design for 30 times today's load. Cracks means outages. And although it's fun to have a "digital snow day" as a dev, GitHub outages are productivity killers. ## What Origin Is Actually Trying to Do Origin is a rethink of what a forge looks like if you start from agents rather than bolt them on afterward. Cursor unveiled it at its [Compile conference](https://www.eesel.ai/blog/what-is-cursor-origin) on June 16, 2026, with demo numbers that were the whole pitch: 22.6 commits per second in a single repository, hundreds of thousands of clones per hour, sub-400ms global sync. Agent-scale numbers, not human-scale ones. And it's not hypothetical anymore. Cursor shipped Origin's early beta on August 17, 2026, the same week GitHub had a [major outage](https://venturebeat.com/infrastructure/cursor-launches-origin-code-hosting-platform-as-github-outage-exposes-opening-in-ai-coding-race) that hit its website, API, Actions, pull requests, authentication, and Copilot all at once, timing Cursor couldn't have bought with launch copy. A few things stand out beyond the throughput claims. It's designed around stacked commits, lining up with Cursor's acquisition of [Graphite](https://byteiota.com/cursor-origin-git-forge-ai-agents/), the stacked-review startup it bought in December 2025. Agents produce lots of small, rapid changes and spawn parallel workstreams, a pattern stacked commits fit far better than a traditional PR-centric model. It also gives agents a native interface instead of routing everything through expensive, high-latency webhooks. Latency and overhead compound fast at agent scale. And importantly, Origin is still built on [Git](/posts/thursday-thoughts-chat-is-the-new-git). The early beta even mirrors GitHub rather than replacing it outright. GitHub stays the system of record at first, and a repository can be detached later to make Origin authoritative. That's a smart call. One of the real barriers to enterprise adoption of anything new is the switching cost hidden inside familiarity, and staying compatible with GitHub during the transition lowers that barrier considerably. ## Why This Might Actually Work Despite the Odds In normal times, I'd say displacing GitHub is essentially impossible. Every enterprise has years of workflows, integrations, and institutional muscle memory baked into GitHub, dating back to [Microsoft's 2018 acquisition](https://news.microsoft.com/source/2018/06/04/microsoft-to-acquire-github-for-7-5-billion/) of it for $7.5 billion. Switching costs aren't theoretical. They're painful and real. But these aren't normal times, and Microsoft's vulnerability goes deeper than server architecture. Copilot's own developer share is sliding. The [2026 Stack Overflow survey](https://pasqualepillitteri.it/en/news/3392/github-copilot-cursor-claude-code-ai-coding-showdown-2026) put it at 51%, down from 67% the year before, while Cursor and Claude Code went from a standing start to double digits, and JetBrains' 2026 survey found the three roughly bunched at 29%, 18%, and 18%. Copilot isn't the clear leader in the category Microsoft invented anymore. Part of that is architecture, not just model quality. Copilot shipped as a plugin bolted onto VS Code and GitHub rather than a rethink of either, and developers noticed. Microsoft also used its grip on the editor to squeeze rivals: in 2025 it broke closed-source extensions like Pylance and C/C++ IntelliSense for VS Code forks, a move [devs called anti-competitive](https://www.theregister.com/2025/04/24/microsoft_vs_code_subtracts_cc_extension/), and one that only mattered because a fork had already taken real share. Cursor is proof that even VS Code, with [70%-plus share](https://dev.to/glen_kiptoo_25bf70b816136/vs-code-vs-cursor-traditional-vs-ai-code-editor-which-one-should-developers-use-in-2026-5dg3) and Microsoft's full backing, wasn't safe from a startup willing to rebuild the editor around agents. If VS Code was vulnerable, GitHub isn't obviously immune either. Meanwhile, SpaceX's [$60 billion buyout](https://www.cnbc.com/2026/06/16/spacex-spcx-cursor-acquisition-ipo.html) of Cursor's parent company, Anysphere, the largest venture-backed startup acquisition on record, has made them genuinely well-funded and positioned to compete in the enterprise space in a way most startups simply aren't. They have resources and momentum, plus a vertically integrated stack now: [Composer](https://byteiota.com/cursor-origin-git-forge-ai-agents/) for editing, Graphite for review, Origin for hosting. The period when switching costs are the most material barrier is closing. But the better product doesn't automatically win that fight, a lesson I keep coming back to from [Claude Code vs. Lotus 1-2-3](/posts/thursday-thoughts-claude-code-is-lotus-1-2-3): distribution has beaten quality before. ## The Question That Will Decide Everything The one thing I keep coming back to is whether Origin will be open or proprietary. That's not a small detail. It's probably the whole game. If Origin is built openly, it has a real path into the enterprise. Enterprises are cautious, and open standards give them the confidence that they're not locking themselves into a vendor with a single point of failure. It also means the broader ecosystem can build on top of it, which accelerates adoption in ways that a closed system simply can't replicate. If Origin ends up being closed, designed to keep developers inside the Cursor and xAI orbit, the enterprise story gets much harder. Enterprises have been burned by lock-in before and remember it, the same pattern I saw with [Anthropic and AWS](/posts/thursday-thoughts-why-anthropic-is-the-next-aws-but-potentially-worse). I don't know which direction they'll go. But I think that decision, more than anything else about the product itself, will determine whether Origin is a footnote or a genuine inflection point. --- We're still figuring out what AI-native tooling looks like at every layer of the stack. The forge is just one piece of it, but a consequential one, because whoever controls where code lives has a lot of leverage over everything downstream. *What would it take for you to move your organization off GitHub?* ## By the Numbers - **22.6 commits per second** — the throughput Cursor demoed for Origin in a single repository at its June 16, 2026 launch. - **17 million** — pull requests opened by AI agents on GitHub in March 2026, up from roughly 4 million in September 2025. - **257 incidents** — GitHub outages LeadDev counted between May 2025 and April 2026, 48 of them major. - **51%** — Copilot's share in the 2026 Stack Overflow Developer Survey, down from 67% the year before. - **70%-plus** — VS Code's market share when Cursor forked it anyway and started taking share. - **$60 billion** — the all-stock deal SpaceX used to acquire Cursor's parent company, Anysphere, the largest venture-backed startup acquisition on record. - **2018** — the year Microsoft bought GitHub for $7.5 billion, the incumbent advantage Origin is trying to unseat. - **3** — layers of Cursor's now-vertically-integrated stack: Composer for editing, Graphite for review, Origin for hosting. === ## Local Agent Bakeoff: Granite 4.2 Is the New Homelab King, But Turn Thinking Off to Speed It Up 10x - URL: https://vibescoder.dev/posts/local-agent-bakeoff-granite-4-2-is-the-new-homelab-king-turn-thinking-off-to-speed-it-up-10x - Date: 2026-09-02 - Tags: #agents #homelab #llm #benchmark #home-automation #granite - Reading time: 11 min read Three new contestants join the local agent bakeoff: IBM Granite 4.2, Poolside Laguna XS 2.1, and DeepReinforce Ornith 1.5. Granite 4.2 unseats Qwen 3.6 as the equal-weighted leader and dominates the one job that actually matters, Home Assistant tool-calling. Then a fourth run isolates a single variable, IBM's default-on thinking mode, and finds it costs 10.8x the wall-clock time for no measurable accuracy gain on this battery. Plus: why the coding domain needs a harder replacement before round three. --- [Qwen 3.6 held the crown last round](/posts/local-agent-bakeoff-qwen-remains-on-top-muse-makes-splashy-debut), but barely. The margin over second place was smaller than the model's own run-to-run noise. I said I'd keep watching for the next thing worth testing. It showed up sooner than expected, and not on purpose. I asked the agent to [sweep the homelab for routine software updates](/posts/friday-fixes-the-agent-audits-the-homelab-then-updates-everything-itself), and the llama.cpp changelog for that update mentioned three model architectures: I'd never heard of: Qwen3.8-Flash-Next, DeepSeek V4, and something called Nemotron3.5. We'd considered them before, but dismissed them. I couldn't recall why. Oh, that's right! Too big. But chasing that down surfaced three real contestants that fit the rig and had never been through the harness: IBM's Granite 4.2, Poolside's Laguna XS 2.1, and DeepReinforce's Ornith 1.5. So we ran it back. This round changes the leaderboard, and the most interesting finding wasn't even about which model won. It was about a single configuration flag on the winner. ## Three New Contestants Join the Field Same rig as last time: one RTX 5090, 32GB of VRAM, one [llama-swap](https://github.com/mostlygeek/llama-swap)-managed endpoint. Round 1's five models (Qwen 3.6, Qwen3.8-27B, Nemotron Lightning, Muse Glimmer, Hermes 4.3) did not get rerun. Nothing about the harness or the fairness rules changed since last time, so their numbers stand as recorded. | Model | Architecture | Params (total / active) | Quant | Disk | Context | |---|---|---|---|---|---| | **[Granite 4.2 30B](https://huggingface.co/ibm-granite/granite-4.2-30b)** | Dense, reasoning-native | 30B (all active) | Q4_K_M | 17G | 65,536 (VRAM-bound) | | **[Laguna XS 2.1](https://huggingface.co/poolside/Laguna-XS-2.1)** | MoE | 33B / 3B | Q4_K_M | 19G | 131,072 | | **[Ornith 1.5 35B-A3B](https://huggingface.co/bartowski/Ornith-1.5-35B-A3B-GGUF)** | Hybrid GatedDeltaNet + attention MoE | 35B / 3B | Q4_K_M | 20G | 131,072 | *Specs for all three new models as configured on the homelab's RTX 5090.* Granite 4.2 is IBM's first reasoning-native Granite release, dense across all three of its sizes, purpose-built for [agentic enterprise workflows](https://research.ibm.com/blog/introducing-granite-4-2). Laguna XS 2.1 is Poolside's smallest coding-focused model, a mixture-of-experts design meant to run agentic coding work on a single consumer GPU. Ornith 1.5 is DeepReinforce's mid-size entrant, distilled down from a 397B flagship using the same self-improving training loop, the same "smaller model inherits the bigger one's training recipe" story that made Nemotron Lightning a strong showing last round. Laguna XS 2.1 and Ornith 1.5 both hold the group's 131,072-token fairness ceiling with room to spare, 25.0GB and 24.3GB used respectively. Granite 4.2 did not. It's the only dense model in this round's field, and dense attention makes a large KV cache expensive in a way MoE and hybrid architectures don't pay. The group ceiling OOM'd outright. Binary-searching it found the real number: 98,304 also OOMs, by about 512MB on the compute buffer, and 65,536 loads clean with roughly 4.5GB of headroom. That's the same structural situation [Hermes 4.3 hit last round](/posts/local-agent-bakeoff-qwen-remains-on-top-muse-makes-splashy-debut), a dense model paying a real VRAM tax that MoE contestants don't, honestly reported rather than argued away. ## The Test Didn't Change so the Comparison Holds Same six domains, same fixtures, same [Inspect AI](https://inspect.aisi.org.uk/)-driven scorer built out over the [last bakeoff's harness work](/posts/how-we-got-here-building-the-test-harness-behind-the-local-agent-bakeoff): Home Assistant device control, calendar management, a read-only investment portfolio, a personal to-do list, and two coding tasks scored by actually executing the generated code. Every new model went through the full battery three times, not once, for the same reason as last round: a model doesn't necessarily make the same tool call twice at non-zero sampling temperature, and a single run tells a cleaner but less true story than three do. ## Granite 4.2 Takes the Equal-Weighted Crown | Domain | Granite 4.2 | Laguna XS 2.1 | Ornith 1.5 | |---|---|---|---| | Home Assistant | 0.794 | 0.621 | 0.579 | | Calendar | 0.890 | 0.833 | 0.854 | | Portfolio | 0.733 | 0.822 | 0.800 | | To-do | 0.576 | 0.515 | 0.667 | | Coding (both tasks) | 1.000 | 1.000 | 1.000 | | **Equal-weighted average** | **0.832** | **0.799** | **0.817** | | Sample-pooled | 0.802 | 0.696 | 0.684 | *Accuracy by domain, 3-run mean, all three new models, IBM Granite 4.2 tested in its default thinking-on configuration.* Line that up against round 1's table and the leaderboard actually moves. Granite 4.2 lands at 0.832 equal-weighted, ahead of Qwen 3.6's 0.814. Ornith 1.5 lands at 0.817, also ahead of the old incumbent. Laguna XS 2.1 lands at 0.799, tying Muse Glimmer almost exactly. | Model | Round | Equal-weighted mean | |---|---|---| | **Granite 4.2** | 2 | **0.832** | | Ornith 1.5 | 2 | 0.817 | | Qwen 3.6 | 1 | 0.814 | | Qwen3.8-27B | 1 | 0.804 | | Laguna XS 2.1 | 2 | 0.799 | | Muse Glimmer | 1 | 0.799 | | Nemotron Lightning | 1 | 0.784 | | Hermes 4.3 | 1 | 0.746 | *Combined leaderboard across both rounds, sorted by equal-weighted 3-run mean.* Granite 4.2's standard deviation across its three runs is 0.002, the tightest of any model tested across both rounds. Whatever it's doing, it's doing it the same way every time. ## Granite Doesn't Just Win It Runs Away with the One Job That Matters Equal-weighted treats all six domains as co-equal, which is the right number for an overall verdict. But this assistant's actual daily job is mostly Home Assistant, and that domain tells a starker story than the aggregate does. | Model | HA-domain mean | |---|---| | **Granite 4.2** | **0.794** | | Qwen3.8-27B (round 1) | 0.708 | | Muse Glimmer (round 1) | 0.705 | | Laguna XS 2.1 | 0.621 | | Nemotron Lightning (round 1) | 0.646 | | Ornith 1.5 | 0.579 | | Qwen 3.6 (round 1) | 0.588 | | Hermes 4.3 (round 1) | 0.558 | *Home Assistant domain accuracy, 3-run mean, both rounds combined.* That's not a close call. Granite 4.2 beats round 1's best Home Assistant performer by nearly nine full points, on the one domain that's actually a voice-controlled smart-home butler rather than an abstract benchmark. Laguna XS 2.1 and Ornith 1.5, both coding-leaning models by training focus, land well below the group's middle here, a reminder that a model's marketing category (agentic coding, in both cases) doesn't automatically transfer to tool-calling accuracy in a different domain. ## Turning Thinking Off Costs Nothing and Saves 10.8x the Clock Granite 4.2 was also, by a wide margin, the slowest model to run through the battery. Its 80-sample Home Assistant domain alone took roughly 10 minutes, next to Laguna XS 2.1 finishing the same domain in 45 seconds. The reason is [IBM's own design choice](https://huggingface.co/blog/ibm-granite/granite-4-2): Granite 4.2 ships three switchable reasoning modes, full thinking, low-effort, and non-thinking, and thinking is on by default. Every sample in the table above got a full chain-of-thought trace before the actual tool call, because nothing in the harness told it not to. IBM's stated reason isn't generic. Their framing is specifically agentic: enterprise tasks are ambiguous and multi-step, and [reasoning helps Granite 4.2 "evaluate which applications to use and in what order rather than executing blindly."](https://research.ibm.com/blog/introducing-granite-4-2) It's a real bet, grounded in a real research lineage going back to the [original chain-of-thought paper](https://arxiv.org/abs/2201.11903), that showing intermediate steps before an answer measurably improves multi-step accuracy. The question worth asking on a narrow tool-calling battery like this one is whether that bet pays for itself. So we ran a fourth configuration: the exact same weights, the exact same context cap, the exact same everything, with `--chat-template-kwargs '{"enable_thinking": false}'` as the only difference. | Configuration | Equal-weighted mean | HA-domain mean | Total wall time, 18 evals | |---|---|---|---| | Thinking ON (default) | 0.832 | 0.794 | 5,572s (92.9 min) | | Thinking OFF | 0.837 | 0.812 | 514s (8.6 min) | *Same weights, same context cap, one variable changed.* Non-thinking mode scored marginally higher on both numbers, well within noise, so call the accuracy difference a tie. What isn't a tie is the clock. Turning thinking off cut the full 3-run, 6-domain pass from 92.9 minutes to 8.6 minutes, a 10.8x speedup, for zero measurable accuracy cost on this specific battery. IBM's low-effort and non-thinking modes exist because a model that reasons about whether to turn off the office lights is spending compute on a decision that was never ambiguous. On a narrow, well-defined tool-calling job, that's exactly what happened here. The reasoning-first default is a reasonable choice for genuinely ambiguous enterprise workflows. It's the wrong default for a voice assistant that mostly needs one correct tool call, fast. ## Two Coding Tasks Are Now Fully Solved Which Means They're Broken as a Test Every model in this round, all four configurations including both Granite variants, scored a flat **1.000** on both coding tasks, across all three runs, no cracks anywhere. That's a harder ceiling than round 1 saw. Last round, two of five models dropped to 0.875 on the calendar-conflict detector on the third run, a thin but real signal that the task still had some difficulty left in it. This round, nothing moved. Not once, across 18 fresh eval runs from two brand-new coding-focused models. That's not a compliment to the models so much as an indictment of the test. A portfolio-drift-flagging script and a calendar-conflict detector were reasonable coding checks for last round's field. They're not discriminating anymore, and a saturated test tells you nothing about which model is actually better at code. Before round 3, this needs fixing, and there are really only two honest paths. Hand-roll harder cases into the existing harness, more edge cases, nested logic, maybe a task that requires reading and modifying existing code instead of generating a script from scratch. Or bolt on an established, harder benchmark like SWE-bench or Terminal-Bench as a supplement, since both Laguna XS 2.1 and Ornith 1.5 already report strong numbers there and it would let this bakeoff's coding domain actually separate the field instead of rubber-stamping everyone with a perfect score. ## The Verdict **Granite 4.2 is the new equal-weighted leader, and it should run with thinking off.** It wins the aggregate, and it dominates the Home Assistant domain specifically by the widest margin either round has produced. The catch is entirely self-inflicted: run it in IBM's default configuration and it's the slowest model tested by a wide margin, for accuracy that a non-thinking config matches or slightly beats. Configured correctly, this is the new daily-driver candidate. **Ornith 1.5 is a legitimate second-place finish from a model built for a different job.** DeepReinforce trained it for agentic coding, not smart-home tool-calling, and it still lands ahead of round 1's incumbent on the equal-weighted number. Its Home Assistant score is the weakest of the three new contestants, which tracks with its training focus, but the overall package holds up. **Laguna XS 2.1 ties Muse Glimmer almost exactly, and pays for its coding-model DNA in wall-clock time.** It's the only one of the three new models whose coding-task times ballooned (249 seconds and 454 seconds for the two coding tasks alone, out of a 15.5-minute total run), a real cost this round's scoring doesn't capture. Its own DFlash speculator model, not wired up for this round, is the obvious next lever if it stays in rotation. ## What's Next Granite 4.2 with thinking off is the new candidate for the actual homelab, not just the leaderboard. Before committing to a daily-driver swap, it needs the same soak-test treatment Muse Glimmer got last round, real usage, not just battery numbers. The coding domain needs a rebuild before round 3 means anything on that axis, and that's the next concrete task, not a someday item. And DFlash and DSpark, the speculative-decoding draft models several of this round's contestants shipped with, never got tested. That's a speed question for a future post, not a quality one, since none of it should move a single accuracy number above. ## By the Numbers - **3** new contestants tested, **4** configurations total counting Granite's thinking-on and thinking-off variants - **0.832** — Granite 4.2's equal-weighted 3-run mean, the new leader across both rounds, beating round 1's Qwen 3.6 at 0.814 - **0.794 vs. 0.708** — Granite 4.2's Home Assistant-domain mean vs. round 1's best performer on that domain, Qwen3.8-27B - **10.8x** — the wall-clock speedup from turning Granite 4.2's default thinking mode off, 5,572 seconds down to 514 seconds across the full 18-eval pass - **0.002** — Granite 4.2's equal-weighted standard deviation across three runs, the tightest of any model tested across both rounds - **0** — cracks in the coding-task ceiling this round, versus 2 of 5 models cracking it on round 1's third run - **65,536** — Granite 4.2's VRAM-bound context ceiling, measured after 131,072 and 98,304 both OOM'd - **8** models now tested across two rounds, **0** of round 1's five rerun this round on the assumption the harness didn't materially change === ## The Local Vibe Coders Dream: DeepSeek V4 Flash on DGX Spark, Mac Studio, or Strix Halo - URL: https://vibescoder.dev/posts/the-local-vibe-coders-dream-deepseek-v4-flash-on-dgx-spark-mac-studio-or-strix-halo - Date: 2026-08-31 - Tags: #homelab #agents #llm #benchmark #building-in-public - Reading time: 11 min read Three consumer boxes claim they can run DeepSeek V4 Flash for a two-Hermes homelab: NVIDIA's DGX Spark, Apple's Mac Studio, and AMD's Ryzen AI Max+ 395. Only two of them can reach the memory tier the model needs to run lossless. --- My wife's Mac mini runs Hermes. My AI workstation is about to run it too. That gives us two Hermes agents and one dilemma: Can we find a model that serves both (concurrently) for both coding and agentic tasks? Models optimized for tool calls tend to suck at coding. Models optimized for coding tend to be large, and exceed consumer hardware. But as token costs climb, there are solutions on the horizon. So the search became about the model, not the machine. DeepSeek shipped [V4 Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash), a 284 billion parameter mixture-of-experts model with only 13 billion parameters active per token, a 1 million token context window, and an official [0731 build](https://huggingface.co/blog/ResterChed/deepseek-v4-flash-official-release) that beats its own preview and DeepSeek's larger Pro preview on every published agentic benchmark. Paired with Hermes, that's a model worth building a dedicated third machine around. This research pits three candidate boxes against each other: NVIDIA's GB10 (aka DGX Spark), a homelab darling in the Mac Studio, and a dark horse in AMD's Ryzen AI Max+ 395, better known by its codename, Strix Halo. The question became which machine, and at what quantization, actually runs DeepSeek V4 Flash well. ## DeepSeek V4 Flash 0731 Earns the Dedicated Box The 0731 release kept the same architecture as the preview and only changed the training. DeepSeek re-post-trained it for agentic work, added native Responses API support, and shipped explicit Codex adaptation. The result scores 82.7 on Terminal Bench 2.1 and 54.4 on DeepSWE, up sharply from the preview's 61.8 and 7.3 on the same benchmarks, and it ships MIT licensed with weights available same day as the announcement. The active-parameter count is the part that matters for hardware sizing. Decode speed tracks the 13 billion active parameters, not the full 284 billion, which is the same trick that makes MoE models like gpt-oss-120b feel faster than their total size suggests on bandwidth-limited boxes. Total weight size still has to sit in memory somewhere, though, and that's where the three candidate machines start to diverge. ## 128GB Spark Trades Single-Stream Speed for Batching Headroom The [DGX Spark](https://robert-mcdermott.medium.com/the-nvidia-dgx-spark-0e2ca7833c2c) packs 128GB of unified memory at 273 GB/s around NVIDIA's GB10 Grace Blackwell chip, running the real CUDA and vLLM stack, Linux only. Single-stream decode on a dense 70B model is genuinely slow on this hardware, and early reviewers wrote it off for exactly that reason. But four to six concurrent Hermes requests is the actual job here, not one long solo chat, and that's where the Spark's batching shows up. One owner logged a Qwen3-Next-80B server climbing from 43 tokens per second at one concurrent request to 136 tokens per second at sixteen. A separate [concurrency benchmark](https://dendro-logic.com/engineering/nvidia-dgx-spark-concurrency-benchmark/) measured 695 aggregate tokens per second across 256 concurrent streams on a model most single-chat reviews had already written off as too slow. The tradeoffs run the other way too. There's no ECC on the unified memory, so a silent bit flip is a real risk for an always-on appliance. Time to first token on very large prompts can stretch past a minute on the biggest models. And the CUDA stack, while genuinely native, is young enough that owners describe getting from a working prototype to a reliable production deployment as harder than it should be. ## 128GB Mac Studio Wins Bandwidth Loses the Batching Engine A Mac Studio flips every one of those tradeoffs. The M4 Max reaches up to 546 GB/s, roughly double the Spark's bandwidth, and bandwidth is the number that decides how fast tokens stream once generation starts. What it doesn't have is a mature continuous-batching engine. MLX and llama.cpp serve requests close to sequentially, so six concurrent Hermes agents each get roughly a sixth of the single-stream speed, not the Spark's better-than-linear scaling. For a quiet single-user chat box that tradeoff barely registers. For a shared brain fielding two Hermes agents at once, it's the whole question. macOS only, of course. A Mac mini never entered contention here. Its ceiling on the M4 Pro tops out well short of 128GB, too small to hold a model this size at any usable quantization. ## 128GB Strix Halo Trades Both Extremes for Efficiency and Choice of OS AMD's answer borrows Apple's own playbook. The [Ryzen AI Max+ 395](https://www.hashtechwave.com/amd-strix-halo-review/) pairs 16 Zen 5 cores with a 40-compute-unit RDNA 3.5 iGPU and up to 128GB of shared LPDDR5X memory, rated at roughly 256 GB/s theoretical and closer to 215 GB/s in [real-world testing](https://datahardware.ai/blog/strix-halo-tokens-per-second-2026). The real story is stranger than the spec sheet: on MoE models, decode speed nearly matches NVIDIA's dedicated box, but prefill does not. On the GPT-OSS 120B benchmark, one review clocked Strix Halo processing prompts at roughly 340 tokens per second against the Spark's roughly 1,700, [a five times gap](https://www.hashtechwave.com/amd-strix-halo-review/), and an independent lab found the same pattern, slower time to first token that widens as prompts grow. The software path is its own animal too. Most owners run [llama.cpp on Vulkan](https://runaihome.com/blog/ryzen-ai-max-395-strix-halo-local-llm-2026/) for everyday use and switch to ROCm with Flash Attention once context gets long, since Vulkan degrades past roughly 4,000 tokens while ROCm holds flat well past 8,000. A community-maintained [vLLM path exists](https://blog.jreb.nl/posts/20260416-SetupvLLMAMDRyzenAIMax/) through nightly ROCm builds, but it's a toolbox project, not a first-party stack the way CUDA and vLLM are on the Spark. Where Strix Halo pulls ahead outright is flexibility and efficiency: it runs Windows or Linux, usable 70B-class inference draws somewhere in the 65 to 90 watt range, and it's the quietest of the three by a wide margin. The Windows path has a real cost, though. ROCm has no official Windows build, so Windows users fall back to LM Studio's Vulkan backend, running 20 to 30 percent behind the same chip on Linux. ## None of the Three 128GB Boxes Hold V4 Flash Lossless This is where the model's own memory math rules out all three 128GB machines outright. The [lossless 8-bit build](https://unsloth.ai/docs/models/deepseek-v4) of V4 Flash runs 162GB, with at least 169GB recommended once context and KV cache are counted. The practical shrink, a 3-bit quant, drops to 103GB and still wants a minimum of 110GB. Any of the three 128GB boxes can just barely load that 3-bit quant, with almost nothing left over for the model's actual selling point, a 1 million token context window. Running V4 Flash on a 128GB box means giving up on lossless output and long context in the same breath. ## Five Scenarios Reduce to One Table Here's the full lineup side by side, the three boxes that can run the compressed model and the two paths that reach the memory tier the lossless build actually needs. | Tier | Device | Memory | OS support | V4 Flash fit | |---|---|---|---|---| | Lossy | NVIDIA DGX Spark | 128GB unified | Linux only | 3-bit quant, tight on context | | Lossy | AMD Ryzen AI Max+ 395 (Strix Halo) | 128GB unified | Windows and Linux | 3-bit quant, tight on context | | Lossy | Mac Studio M5 Max | 128GB unified | macOS only | 3-bit quant, tight on context | | Lossless | 2x DGX Spark, clustered | 256GB pooled | Linux only | Full 8-bit, room for long context | | Lossless | Mac Studio M5 Ultra | 256GB unified | macOS only | Full 8-bit, room for long context | Strix Halo doesn't have a seat at the lossless table the way the other two do. AMD's own playbooks describe clustering two Ryzen AI Halo boxes over llama.cpp RPC for very large models, but it's a community recipe, not an official pooled-memory path like the Spark's NCCL cluster or a single Mac Studio's unified memory. For this comparison, Strix Halo stays a 128GB, lossy-only contender. ## 256GB Changes the Quantization Question Not the Tradeoff Both lossless paths land at 256GB, but they get there differently. Apple's newly announced [M5 Ultra Mac Studio](https://www.macrumors.com/roundup/mac-studio/) opens a 256GB configuration for the first time at this chip generation, backed by 1.2TB/s of memory bandwidth, up from 819 GB/s on the M3 Ultra. That's one chip and one unified memory pool. The lossless 162GB build of V4 Flash fits with room to spare for context and KV cache, no partitioning required, and decode speed benefits from the full bandwidth jump on top of it. The Spark reaches the same 256GB by [clustering two units](https://docs.nvidia.com/dgx/dgx-spark/spark-clustering.html) over a single cable, connected through MPI and NCCL rather than one shared memory pool. Combined capacity is generous. Forum math on this exact pairing puts a dual Spark cluster's inference ceiling at roughly 470 billion parameters at 4-bit, nearly three times what V4 Flash's lossless weights need. But the two 128GB nodes are joined by an interconnect running at roughly 25 GB/s, a small fraction of the 600 GB/s a single Spark uses internally between its own CPU and GPU. That gap matters far more for training, where gradients need constant synchronization, than for inference, where a pipeline split only has to pass a small activation tensor across the cable once per stage. It still adds real complexity: NCCL configuration, a cluster to keep healthy, and distributed serving support for V4 Flash's specific hybrid attention design that is still young. ## Compute Wins the First Token Bandwidth Wins the Sprint Batching Wins the Real Job This is where the paths split for good, and it maps onto what actually matters for two Hermes agents sharing one brain. Time to first token favors the Spark, alone or clustered. Prefill is compute bound, and NVIDIA's Blackwell tensor cores beat both Apple Silicon's GPU compute and Strix Halo's iGPU on that phase by a wide margin, even though the other two win on raw bandwidth or efficiency. Tokens per second on a single request favors the Mac Studio. Its unified bandwidth with zero interconnect hop beats a pipeline-split pair of Sparks reading across a much narrower cable, and comfortably beats Strix Halo's real-world bandwidth too. Aggregate throughput at four to six concurrent agents, the actual daily job, favors the Spark again, for the same reason it did at 128GB. The batching advantage comes from the software stack, not the memory tier, and it's the one place Strix Halo's community-toolbox vLLM path can't yet compete with a first-party CUDA stack. Given the actual workload, low time to first token and a pair of agents making concurrent requests rather than one long solo chat, the dual Spark cluster is the better technical fit for running DeepSeek V4 Flash lossless. The honest caveat is that distributed serving of a month old model, built on an attention architecture nobody has run across two nodes for very long, is not the boring choice. The Mac Studio is. Same capacity, one box, no cluster to babysit. And if lossless isn't the requirement, Strix Halo earns its spot in the lineup on efficiency and OS flexibility alone, running the same compressed quant as the other two 128GB boxes at a fraction of the power draw. *Given the choice between the better fit and the more boring one, which would you actually want answering pages at 2am?* ## By the Numbers - **284B / 13B** — total and active parameters in DeepSeek-V4-Flash - **1M tokens** — V4 Flash's context window, the feature every quantization choice here either preserves or throws away - **162GB** — lossless 8-bit weight size, against **103GB** at the practical 3-bit shrink - **128GB** — memory ceiling shared by all three lossy-tier boxes: Spark, Mac Studio M5 Max, and Strix Halo - **256GB** — memory reached either through an M5 Ultra Mac Studio or a two-node Spark cluster - **1.2TB/s** — M5 Ultra memory bandwidth, up from 819 GB/s on the M3 Ultra - **~600 GB/s** — bandwidth inside a single Spark between its own CPU and GPU - **~25 GB/s** — bandwidth across the cable connecting two clustered Sparks - **~215 GB/s** — Strix Halo's real-world memory bandwidth, against a 256 GB/s theoretical ceiling - **5x** — the prefill gap between Strix Halo and a single Spark on GPT-OSS 120B - **136 tok/s** — aggregate throughput at sixteen concurrent requests on a single Spark, up from 43 at one - **82.7 / 54.4** — V4 Flash 0731's scores on Terminal Bench 2.1 and DeepSWE, both up sharply from the preview build === ## Friday Fixes: A Model That Found Itself, DNS That Lied for Two Reasons, and a Reboot That Caught a Third - URL: https://vibescoder.dev/posts/friday-fixes-a-model-that-found-itself-dns-that-lied-for-two-reasons-and-a-reboot-that-caught-a-third - Date: 2026-08-28 - Tags: #homelab #self-hosted #home-automation #debugging #building-in-public - Reading time: 8 min read A new local model gets validated by a dashboard feature built before it existed. A DNS outage traces back to two unrelated bugs. A deliberate reboot of the homelab's Proxmox host finds a third bug neither fix would have caught. --- This week wasn't one fix. It was four, and the interesting part is how little they had in common on the surface. A new local model. An outdated CLI tool. A DNS server making pages hang. A Proxmox host I rebooted on purpose just to see what happened. Different machines, different symptoms, same underlying lesson each time. The thing you assume is working is the thing worth testing. ## GLM-4.7-Flash Joins the Lineup and the Dashboard Catches It without Being Told I wanted to add a GLM model to the AI workstation. Z.AI's flagship GLM-5.x line is a roughly 753-billion-parameter model that needs a multi-GPU rig or hundreds of gigabytes of unified memory, nowhere close to a single consumer card. GLM-4.7-Flash is the variant that fits. It's a 30B-class mixture-of-experts model with only about 3.6B active parameters per token, comfortably inside the GPU's headroom at around 17.5GB. I downloaded a fresh copy rather than reusing an old one, specifically to pick up a GGUF re-upload that fixed a wrong scoring function value, the kind of quiet bug that causes reasoning loops and bad output without ever throwing an error. Wired it into the llama-swap config with the jinja flag GLM's chat template requires, and a repeat penalty of 1.0, which is Z.AI's own recommendation for avoiding loop failures. Then I tested it the way it gets used, not the way it's convenient to test. A same-box check with two models loaded at once will just run out of VRAM. The real test has to go through the model-swap proxy itself, since that's what frees memory from the previous model before loading the new one. It did, cleanly, generating around 220 tokens per second and using about two-thirds of available VRAM on its own. The best part had nothing to do with the model itself. I ran the homelab dashboard's boot-time freshness check, a feature built weeks ago against the models that existed at the time, and GLM-4.7-Flash showed up on the Models tab with its Hugging Face repo auto-detected straight from the GGUF's own metadata. No manual mapping entry, no code change. That's the actual proof a feature works. It's not that it handles the cases it was built against. It's that it handles one it had never seen. While I was in the dashboard, I noticed the status chips on mobile were stuck in an awkward two-column grid, left over from a redesign a few sessions back. Long model and monitor names were getting truncated with an ellipsis and the spacing looked off on a phone screen. Switched it to a single-column flex stack and let labels wrap instead of truncating. Two lines of CSS. Worth mentioning anyway, because it's a good example of shipping something, looking at it on the device it's meant for, and fixing what you see rather than guessing at layout from a desktop browser. ## Hermes Agent Was 14455 Commits Behind I wanted to use the Hermes Agent CLI's web dashboard. I hadn't updated the local install since May. Running the version check turned up over fourteen thousand commits of drift, and on top of that, the tool's default chat model was still pointed at an interim fallback set during an earlier model cleanup, not at anything currently installed. The fix itself was routine. Stop the background gateway service, back it up, run the built-in update command, which pulls the full git history and reinstalls both the Python and Node dependencies before rebuilding the web UI, then restart the gateway. Pointed the default model at one of the models already on the box and confirmed a real chat request triggered the correct backend load. The web UI turned up a real design difference. Newer versions of the tool added a hard safety check. They refuse to bind to any address beyond localhost unless an auth provider is configured. There's no "trust the private network" unauthenticated option, the way the homelab dashboard works. I set up HTTP basic auth with a random generated password so it could be reached from other devices on the network. Not every self-hosted tool treats the network boundary as the security boundary, and it's worth checking that assumption per tool rather than carrying it over from the last one you configured. ## AdGuard Home Was Failing Two Different Ways at Once Two machines on the network started feeling sluggish. Pages timing out, images not loading. Both routed their DNS through the self-hosted AdGuard Home instance, so that's where I looked first, starting with the boring check: memory. 84MB used out of 512MB available, no swap pressure, host load average under half a core. Nothing there. The real story was in the service logs, and it turned out to be two unrelated bugs. The first was a stale cached resolver. AdGuard had been running for two straight days. At some point during that window, the underlying system's DNS configuration got corrected to point at the LAN router, but AdGuard doesn't re-read that file live. It caches the system resolver list once, at its own startup, purely for reverse lookups used to show client hostnames in its own interface. Whatever it cached two days earlier included a Tailscale magic-DNS address that this particular box, a plain LAN container with no Tailscale client on it, could never reach. Every single reverse lookup was eating a guaranteed two-second timeout before failing. The second bug was the one breaking page loads. AdGuard's only configured upstream resolver was a single DNS-over-HTTPS endpoint, Quad9, with no fallback set at all. A plain curl to that same endpoint from the container succeeded instantly, every time, so the network path itself was fine. But AdGuard's own DoH client was intermittently hitting an unexpected end-of-file error on that connection, about 78 times in one hour. With no fallback configured, every one of those was an outright resolution failure for whatever domain a page happened to be loading. Different subresources, different domains, different silent failures, spread across a normal browsing session. That's the actual symptom, explained. The fix was two config lines and a restart: a real fallback resolver, and an explicit pin on the local reverse-lookup resolver instead of AdGuard's own stale auto-detection. "Does it just need a reboot" turned out to be half right and half a trap. A reboot would have cleared the stale cache and fixed the first bug. It would have done nothing for the second, and the missing fallback would have left the door open for the next Quad9 hiccup to take pages down again. ## A Reboot Test Found the Bug the DNS Fix Couldn't Once AdGuard was fixed, the obvious next question was whether the whole box would recover from a real restart, not just a service bounce. Better to find that out on purpose than during a real outage. The Proxmox host in question runs four things: a Home Assistant VM, and three LXC containers for AdGuard, an uptime monitor, and a homelab status dashboard. Before touching anything, I checked each workload's onboot flag, since that's the setting that controls what comes back after a host-level reboot. A container running right now says nothing about whether it survives a cold boot. All three LXC containers had it set correctly. The Home Assistant VM did not. Nobody had ever noticed, because that VM had simply never been rebooted since it was first set up. Set the flag, captured a baseline HTTP check against all four services, then rebooted the host for real. It came back in about 80 seconds. All four services, Home Assistant included this time, auto-started and were answering requests within a minute, matching the baseline exactly. The DNS fix from earlier in the day persisted cleanly through the reboot too, confirmed by resolving a real domain immediately after. One more small thing turned up in the process. AdGuard logged two "network unreachable" errors in the very first second of its own container starting, trying to reach the LAN router before its own interface had finished coming up. It resolved itself instantly and never happened again, a systemd startup ordering issue rather than a config bug. Filed away as a nice-to-have, not urgent. ## None of This Would Have Shown up in a Status Check Each of these four problems hid behind a status that looked fine. A running VM says nothing about whether it survives a reboot. A DNS server answering some queries says nothing about whether the one upstream it depends on is reliable. A feature works until it meets a model it wasn't built against. The only way to know any of that is to test it, on purpose, before the day it matters instead of during it. *What's the one thing in your own setup you're assuming works, but haven't tested since the day you set it up?* ## By the Numbers - **1** new local model added and validated end to end, auto-detected by a feature that had never seen it before - **17.5GB** GLM-4.7-Flash download, running at roughly 220 tokens per second solo - **14,455** commits pulled in a single Hermes Agent update - **2** unrelated bugs found stacked inside one "AdGuard is slow" symptom - **78** DNS-over-HTTPS failures against the sole upstream in one hour, with zero fallback configured - **1** VM found silently missing its onboot flag, years after setup - **4 of 4** services back up and matching baseline within about 60 seconds of a real host reboot - **~80 seconds** total downtime for the reboot test itself === ## Tuning the Hermes Context Window: How I Burned 72.7M Tokens So You Don't Have To - URL: https://vibescoder.dev/posts/tuning-hermes-context-window-how-i-burned-72-7m-tokens-so-you-dont-have-to - Date: 2026-08-26 - Tags: #agents #homelab #llm #benchmark #qwen - Reading time: 13 min read A homelab experiment set out to find the optimal context window for a local Qwen 3.6 daily driver. It failed at that job, and revealed a more useful one: three real bugs, a clean speed-versus-depth curve, a prompt cache that hid 93% of the real token count, and a config change worth making in Hermes Agent's compaction settings. Six tables. Zero cloud tokens spent. --- Local models feel like they get dumber the longer a session runs. I've felt it for months. A conversation starts sharp. An hour in, [Qwen 3.6](/posts/local-agent-bakeoff-qwen-remains-on-top-muse-makes-splashy-debut) starts missing things it would have caught cold. My instinct, every time, is to bail to a cloud model with a bigger context window and call it a day. That instinct bothered me. So I asked a narrower question. Could I tune the context window on my own homelab rig and close that gap without leaving local at all? I built an experiment to find out. The short answer: no. Context window size, the number you configure at the server, does not meaningfully change output quality. That question turned out to be the wrong one. But the experiment answered a better question I hadn't asked yet, and it changed a real setting in [Hermes Agent](/posts/hermes-agent-first-contact), the assistant my wife runs day to day. ## One Model Three Ceilings Nine Cells I fixed the model on purpose. Every prior bakeoff on this blog varies the model. This one holds Qwen 3.6 35B-A3B constant and varies only the context configuration. That isolates the one variable I actually care about. Three [llama-swap](https://github.com/mostlygeek/llama-swap) entries, same weights, three different `--ctx-size` values: | Config | Context ceiling | |---|---| | qwen-32k | 32,768 | | qwen-131k | 131,072 (the production default) | | qwen-262k | 262,144 (the largest size tested clean in the last bakeoff) | Each ceiling gets tested at three fill levels. Fill level means how much of that window is actually occupied before the real question gets asked. 10%, 50%, and 90%. Three repeats per cell, to smooth out sampling noise. Two domains, reused from the [personal-assistant bakeoff harness](/posts/how-we-got-here-building-the-local-agent-bakeoff-test-harness): a calendar assistant with five tools, and a Home Assistant device-control assistant with thirty-two. Three ceilings times three fill levels times three repeats times two domains comes to 54 runs. ## A Bigger Ceiling Does Nothing Until You Fill It Here's the part that almost broke the experiment before it started. Raising `--ctx-size` does not, by itself, stress a bigger window. I measured this directly. Even the richest test data in the harness runs a few thousand tokens deep. That's nowhere near 131,072, let alone 262,144. A model never comes close to using a window that large on short prompts. So testing three ceilings with the same short prompts would have shown nothing at all. The fix: pad every sample with synthetic conversation history before the real question. Generic smart-home and calendar chatter, already resolved, cycled from a small fixed pool. Not random text. Random text tests whether garbage confuses a model, which is a different and less useful question. This tests whether *plausible history* crowds out a model's ability to handle the newest turn, which is the actual failure mode researchers have documented in papers like [Lost in the Middle](https://arxiv.org/abs/2307.03172). Getting the token math right for that filler took three real bugs to shake out. ## Three Bugs Almost Wrecked the Data | Bug | What broke | The fix | |---|---|---| | Tool schema blindness | HA sent 35,353 tokens against a 32,768 ceiling | Count registered tool definitions in the token-fit check, not just messages | | One-turn-at-a-time trimming | A single cell ran over six hours instead of two minutes | Trim filler with one proportional cut, not one small turn per check | | A turn-count safety cap set too low | Six cells at the largest context errored instantly | Raised the cap from 5,000 turns to 50,000 | The first bug came from [Inspect AI's](https://inspect.aisi.org.uk/) `/apply-template` call not knowing about the domain's registered tools. HA registers 32. Calendar registers 5. That gap alone never showed up on calendar. It showed up immediately on HA. The second bug is the one I'm proudest of catching, because it looked like nothing was wrong. GPU utilization sat at 0%. Power draw sat near idle. The process just... didn't finish. It was making hundreds of small HTTP round trips, each one re-rendering an entire 600KB prompt to check if a single 150-token turn could be trimmed. Fixing it meant cutting a proportional slice off the filler in one shot instead of one turn at a time. Same cell afterward: 45 seconds. The third only showed up at the largest context window, because it needed roughly ten thousand filler turns to reach 90% of 262,144 tokens. The safety valve meant to catch a broken tokenizer was catching a legitimate need instead. All three are fixed and merged into [the forked harness](https://github.com/carryologist/ha-voiceagent-llm-benchmark). All 54 cells, plus the 11 that the three bugs invalidated, finished clean on the second pass. ## 54 Runs Found Almost No Accuracy Difference Here's the combined scorecard, both domains, weighted by sample count: | Fill % | 32,768 ceiling | 131,072 ceiling | 262,144 ceiling | |---|---|---|---| | 10% | 0.594 | 0.542 | 0.573 | | 50% | 0.583 | 0.521 | 0.552 | | 90% | 0.542 | 0.531 | 0.542 | Two things jump out. First, there's no clean relationship with ceiling size. The smallest window scored the highest average. The middle window scored the lowest. That's not what "bigger context window helps" would predict. Second, there is a real, if modest, decline as fill increases. Averaged across all three ceilings: 0.570 at 10% fill, 0.552 at 50%, 0.538 at 90%. A drop of about 5.6% relative. Directionally consistent. Small enough that it sits close to the noise floor of individual repeats, some of which swung by 19 points on their own. Split by domain, the pattern holds but the absolute numbers differ a lot: | Domain | 10% fill | 50% fill | 90% fill | |---|---|---|---| | Calendar (5 tools) | 0.861 | 0.840 | 0.840 | | Home Assistant (32 tools) | 0.278 | 0.264 | 0.236 | Home Assistant's low absolute score comes with a caveat. I limited it to 16 samples to keep run time comparable to calendar, and that 16-sample slice may not represent the full 80-case tier fairly. The relative comparison across fill levels still holds, since every cell used the identical 16 cases. The absolute number just shouldn't be read as "Qwen is bad at Home Assistant." ## Decode Speed Fell by Half Cleanly If accuracy barely moved, speed did the opposite. I measured this separately from the scored runs, directly against [llama.cpp's](https://github.com/ggml-org/llama.cpp) native completion endpoint, with prompt caching disabled so every number reflects real, uncached work. | Model config | Real depth | Prefill tok/s | Decode tok/s | |---|---|---|---| | qwen-32k | ~2,300 (10%) | 6,255 | 227 | | qwen-32k | ~15,000 (50%) | 8,668 | 216 | | qwen-32k | ~28,000 (90%) | 8,443 | 202 | | qwen-131k | ~12,000 (10%) | 8,301 | 218 | | qwen-131k | ~65,000 (50%) | 7,526 | 172 | | qwen-131k | ~117,000 (90%) | 6,275 | 141 | | qwen-262k | ~25,000 (10%) | 8,250 | 203 | | qwen-262k | ~130,000 (50%) | 6,038 | 136 | | qwen-262k | ~235,000 (90%) | 4,417 | 102 | Decode speed at ~235,000 tokens of real depth is less than half what it is near empty. 102 tokens per second versus 227. Prefill drops too, from 8,250 tokens per second down to 4,417. Look closer and a pattern appears that matters more than any single row. qwen-32k at 50% fill, about 15,000 real tokens, decodes at 216 tok/s. qwen-131k at 10% fill, about 12,000 real tokens, decodes at 218 tok/s. Nearly the same depth. Nearly the same speed. Two servers, two completely different configured ceilings. ## A Prompt Cache Hid 93% of the Real Token Count Every sample inside one test cell shares the same synthetic filler. Sixteen samples, one shared prefix, one resident llama-server process. [llama.cpp's](https://github.com/ggml-org/llama.cpp) prompt cache noticed. After the first sample in a cell, the shared filler served straight from cache for every remaining sample. Inspect's own per-request accounting doesn't know or care about that. It reports the full context size for every single request, cached tokens included. Summed across all 864 scored samples, that raw per-request accounting comes to 72,661,613 tokens. That's the number I'd expect on a receipt if this ran through a metered API with no cache credit. It's also the honest, conventional answer to "how many tokens did this use." Only 5,395,251 of those tokens were freshly computed. The rest, 93%, came straight from cache. Neither number is the full story by itself. 72.7 million treats a cached prefix as if the GPU recomputed it sixteen times over, which it didn't. But 5.4 million has its own blind spot. A cache hit still has to be read. Every decode step, in every one of those samples, still attended over the full cached context, which is exactly why decode speed dropped with real depth in the section above. Caching cuts prefill cost. It does not cut decode cost. ## The Ceiling Was Never the Answer The match between qwen-32k at depth and qwen-131k at depth, two sections back, is the actual finding of this whole experiment. Speed tracks real depth. Not the number typed into `--ctx-size`. A big ceiling costs nothing in speed until a session actually fills it. It does cost real, fixed VRAM the moment the server starts: about 22.5GB empty at 32,768, climbing to about 25.6GB empty at 262,144, on the same RTX 5090. So the original question, what's the optimal context window for quality, doesn't really have an answer. Quality barely moves either way. The ceiling you pick is a headroom decision, not a quality decision. Set it high enough that a real session never hard-fails against it, and stop there. I'm keeping the production `qwen` entry at 131,072. It's already meaningfully deeper than 32,768 for a real multi-turn session, and it skips 262,144's extra VRAM tax for headroom that mostly goes unused. ## So Why Did It Feel Like the Model Was Getting Dumber This whole experiment started from a feeling, not a number. Long local sessions feel like they degrade. Cloud models with huge context windows feel like the fix. The data above doesn't back that feeling at the strength it deserves. A 5.6% relative accuracy drop, worst fill level against best, is real but mild. It doesn't obviously explain wanting to abandon a model mid-session. Two honest explanations. I don't know yet which one is closer to true. The first: something this experiment never tested is the real cause. Generic synthetic filler is not the same as a long agentic session with real tool calls stacking on top of each other. Small mistakes can compound differently across a chain of real tool calls than across one scored question in isolation. A hard context-overflow error, the exact failure mode three bugs in this experiment produced by accident, can also get misread in the moment as "the model got dumber" instead of "the request failed outright." The second: the felt degradation is real, but smaller than it feels while you're in it. A frustrating exchange at minute forty can color the memory of an entire session. I'm not settling that question here. Both are worth their own experiment. Compounding tool-call error across a real multi-step chain is a different test than raw context volume tolerance. So is a needle planted early in a session and checked at the end, the same idea I raise below as the better version of this experiment. Down the road, one of those becomes the next bakeoff. ## What This Changes in Hermes's Compaction Config This is the part that actually changes something I run every day. Hermes Agent already has automatic context compaction. Full credit where it's due: this isn't a gap I needed to fill with a new skill. It's a mature, built-in system that summarizes older turns once the session crosses a configured percentage of the model's window, protects the first and last N messages from being touched, and even ships a manual `/compress` command for on-demand use. My actual config, before this experiment: | Setting | Value | |---|---| | `compression.enabled` | true | | `compression.threshold` | 0.85 | | `compression.target_ratio` | 0.4 | | `compression.protect_last_n` | 20 | | `compression.protect_first_n` | 3 | At a 131,072 ceiling, an 0.85 threshold means compaction doesn't fire until roughly 111,000 tokens deep. Cross-reference that against the speed table above. At that depth, decode speed already sits around 141 tok/s, down 35% from a fresh session. Real sessions spend real time in that degraded zone before Hermes ever steps in. There's also a status message the code already emits for this, a literal "Compacting context, summarizing earlier conversation" line, but it's silent on chat platforms by default. I'd never seen it. `progress_notices` was off. Two changes, both small, both directly justified by this data: - Add a per-model threshold override so Qwen compacts earlier, closer to 0.5 or 0.6, instead of riding the global 0.85 default all the way down. - Turn `progress_notices` on, so compaction stops happening invisibly. Neither change came from guessing. Both came from a number on a table above. ## What I'd Test Differently Next Time The ceiling turned out to be a confound, not a real variable. A cleaner version of this experiment fixes one large ceiling and varies only real depth, at finer steps, without three separate model configs to manage. It would also use less repetitive filler. My synthetic pool cycled twenty exchanges, which a model can plausibly learn to skim as boilerplate. A sharper test plants one fact early in a long session and checks whether the model still remembers it by the end, closer to what an actual long-running assistant session looks like, and closer to testing the two hypotheses above directly instead of guessing between them. That's a good excuse for a round two. This round already paid for itself in one config change. *If your local daily driver feels sharper at the start of a session than the end, check your compaction threshold before you blame the model.* ## By the Numbers - **54** scored runs, plus **11** re-runs after three bugs invalidated their cells - **72,661,613** tokens processed across every request, the conventional total - **93%** of that total served from llama.cpp's own prompt cache, not freshly computed - **5,395,251** tokens actually fresh, prefill and output combined - **3** context ceilings tested: 32,768 / 131,072 / 262,144 - **6+ hours** one cell ran before I killed it, versus **45 seconds** after the fix - **5.6%** relative accuracy drop, worst fill level versus best, averaged across all three ceilings - **2.2x** the decode speed at near-empty context versus 90% full at the largest window (227 vs. 102 tok/s) - **111,000** tokens deep before Hermes's current 0.85 threshold triggers compaction, at the production ceiling - **0** cloud tokens spent finding any of this out === ## Building a Homelab Dashboard That Does Not Care If the GPU Box Is Off - URL: https://vibescoder.dev/posts/building-a-homelab-dashboard-that-does-not-care-if-the-gpu-box-is-off - Date: 2026-08-25 - Tags: #homelab #self-hosted #home-automation #building-in-public - Reading time: 7 min read A phone-friendly status page for a two-machine homelab, built in phases. Why it lives on the Proxmox box next to Home Assistant instead of the AI workstation, how it talks to Uptime Kuma with no REST API to speak of, and why the third phase, waking the workstation remotely, is still waiting on a cable move. --- I was standing in a parking lot trying to remember whether my AI workstation was even turned on. I'd left the house without checking, and now I needed to know if a monitor was actually down or just quiet, whether a model I'd been testing was still the current build, and whether it was worth the drive home to fix something versus just waiting. My phone had no good way to answer any of that. Uptime Kuma lives on one box, the model configs live on another, and neither one is exactly a mobile-friendly experience over a VPN connection. So I built a dashboard. Not a fancy one. A single page, three tabs, chips instead of tables, dark theme, nothing to log into. The interesting part isn't the UI. It's the three decisions that shaped it: where it lives, how it learns about the workstation's model lineup without that workstation being reliably on, and the one piece I still haven't wired up. ## The Dashboard Lives Next to Home Assistant Not Next to the GPU The obvious place to run a homelab dashboard is the same machine doing the interesting work, in my case the AI workstation with the GPU. I didn't do that. I put it on the Proxmox box instead, in its own small container, right alongside the Home Assistant VM and the Uptime Kuma instance it needed to talk to. The reasoning comes down to one fact: the AI workstation is not always on. It draws real power running a 30B-class model, so it sleeps or shuts down when I'm not actively using it. A status dashboard that lives on the machine most likely to be off defeats its own purpose. The Proxmox box, by contrast, is the thing that's always running. It's already the host for the smart home and the uptime monitor, so it's already the thing I trust to answer "is everything okay" honestly. I gave the new container its own identity on the private network rather than routing through the workstation, so my phone can reach it directly regardless of what state the GPU box is in. Getting there required opening up a device the container's virtualization layer doesn't expose by default, but that's a one-time setup cost, not an ongoing one. ## Uptime Kuma Doesn't Have a REST API so the Dashboard Doesn't Use One The Health tab needed to answer "what's monitored and what's its status" without reinventing Uptime Kuma. The catch is that Kuma's REST surface is almost nonexistent. There's a metrics endpoint and not much else. Everything else, listing monitors, reading their state, goes through Kuma's Socket.io connection, the same one its own web UI uses, authenticated with a real username and password rather than an API key. That's a slightly unusual integration path for a dashboard to lean on, but it works, and a background poller means the dashboard's own page load never blocks on it. The more interesting problem was less about wiring and more about honesty. One of my monitors flags red every time Home Assistant's Supervisor notices I don't have a current backup configured. Kuma only has two states, up or down, and "no backup configured" is not the same class of problem as "the service is unreachable," but Kuma renders them identically. A dashboard that shows a hard red alert for both trains you to ignore red alerts. I fixed this with tags instead of name matching. Rather than writing logic that says "if the monitor name contains X, treat it as less severe," which breaks the moment I rename anything, I added a `non-critical` tag directly in Kuma and had the dashboard read it. A monitor tagged that way still shows a problem, just in orange instead of red, and the source of truth lives in the same tool where I'd naturally go to manage it. Tags survive renames. String matching doesn't. ## The Model Tracker Pushes on Boot Instead of Polling Live The Models tab exists because I kept losing track of which local LLMs were current builds and which were stale copies I'd forgotten to refresh. The hard constraint here is the same one that shaped where the dashboard lives: the workstation holding the models isn't always on, so any design assuming the dashboard can reach out and ask it something live is going to spend half its life showing nothing. So it works the other way. A small script runs once at boot on the workstation, reads the local model configuration, works out where each model's file came from, checks whether that source has moved since the last boot, and pushes a single report to the dashboard. The dashboard just displays whatever it was last told, along with an explicit last-reported timestamp instead of implying the data is live. If the workstation has been off for three days, the dashboard says so instead of quietly serving three-day-old data as if it were current. The freshness check itself compares a source repository's current version against whatever version was recorded the last time the script ran. That's a real limitation worth naming: it's a change-detection signal since the last check, not a guarantee that what's installed is the single latest release available anywhere. Good enough to catch drift, not a substitute for actually reading a changelog. The part I like best is that most models don't need to be told where they came from. GGUF files often carry their own origin metadata, so the script can reconstruct the likely source repository directly from the file and confirm it with one request, falling back to a small manual mapping file only for the handful of models where that metadata isn't there. I proved this out by adding a brand new model to the workstation days after building the feature, one the script had never seen, and it correctly identified where the file came from without me touching any config. ## Phase Three Waking the Workstation from the Couch *This section is a placeholder. Phase three isn't built yet.* The Controls tab exists in the dashboard already, but every button on it is disabled with a note that wiring is deferred. The idea is straightforward: since the AI workstation is the thing that actually gets turned off, the dashboard should be able to turn it back on and, eventually, shut it down cleanly, without me walking over and pressing a physical button. I'm holding off on purpose. The Proxmox box, which is the natural sender for a wake signal, is about to move to a different physical location in the house. That's going to change some of the network topology it currently depends on, and I'd rather build the wake-on-LAN wiring once against the final setup than build it twice. When that move happens, this section gets the real writeup: the wake mechanism, how the dashboard confirms the workstation actually came up rather than just declaring victory after sending a packet, and whatever I inevitably get wrong the first time. --- Three tabs, two machines, and a dashboard that's honest about what it doesn't know yet, both about the workstation's power state and about its own missing feature. That feels like the right way to ship something in phases: build what the current physical setup actually supports, and leave a visible, labeled gap where the rest goes once the setup catches up. *What would you build first if your homelab only gave you one screen to check from the couch?* ## By the Numbers - **3** tabs shipped: Health, Models, Controls (Controls intentionally stubbed) - **2** physical machines involved: the AI workstation (models, GPU) and the Proxmox box (dashboard, Home Assistant, Uptime Kuma) - **1** dedicated container created to host the dashboard, isolated from the services it monitors - **1** Kuma tag (`non-critical`) replacing name-matching logic for severity - **0** REST endpoints available from Uptime Kuma for monitor listing, hence the Socket.io integration - **1** of 3 planned phases still unbuilt, waiting on a hardware relocation === ## Smart Home, Dumb Luck, Episode 4: The Backup Plan That Would Have Silently Failed - URL: https://vibescoder.dev/posts/smart-home-dumb-luck-episode-4-the-backup-plan-that-would-have-silently-failed - Date: 2026-08-24 - Tags: #homelab #home-automation #proxmox #self-hosted #agents #security - Reading time: 8 min read Episode 4 gives an agent real write access to Uptime Kuma, retires the last manually-copied secret files, test-builds Proxmox Backup Server in Docker (catching a build that silently deleted the very package it installed), and catches a Wake-on-LAN plan that was about to fail for a reason nobody had checked yet. --- [Episode 3](/posts/smart-home-dumb-luck-episode-3-hacking-hacs-and-tuning-kuma) ended with nine (now ten) [Uptime Kuma](https://github.com/louislam/uptime-kuma) monitors watching the homelab and one honest admission: everything left in the plan is blocked on standing in my actual house with a Zigbee dongle in hand. That's still true. This episode doesn't touch Zigbee either. Instead it's the session where I asked for something smaller, "let the agent actually manage Kuma, not just read it," and pulling that one thread unraveled a security gap worth closing, a backup plan worth test-driving, and a Wake-on-LAN assumption that was quietly wrong. None of it needed me to leave my desk. All of it needed me to actually check, rather than assume. ## Read-Only Wasn't Going to Cut It The plan was simple: give the agent an Uptime Kuma API key so it could check monitor status without me relaying screenshots. Except Kuma doesn't have a REST API for monitor management at all — not a limited one, none. The only API-key-protected REST endpoint is `/metrics`. Everything else, adding a monitor, editing a threshold, pausing something, runs over Kuma's internal Socket.IO channel, and the [`uptime-kuma-api`](https://github.com/lucasheld/uptime-kuma-api) Python client is the only practical way in from outside the browser. That client needs a real username and password, not an API key. Which changed the actual decision: since I'd already said I wanted the agent adding and tuning monitors going forward, not just reading them, the read-only API key I was about to set up would have been redundant work. Username and password covers both read and write in one credential. So that's what we did instead. ## Retiring the Last Manually-Copied Secret File Here's where [Episode 3's near-miss](/posts/smart-home-dumb-luck-episode-3-hacking-hacs-and-tuning-kuma) comes back around. That session almost committed a live secret to git in plain text, caught and scrubbed within minutes, with a promise to keep secrets in local workspace files referenced only by path from then on. That was the right instinct for that night, but it's the same pattern this blog [already moved past in June](/posts/updating-coder-to-get-user-secrets-and-the-art-of-knowing-where-your-secrets-belong): a file that lives on one workspace's disk doesn't exist on the next one. Every fresh workspace would need the same credentials handed to it by hand, forever. So the four credentials still living that old way, the Kuma login, the `ha-mcp` webhook connect URL, a Proxmox monitoring token, and the Discord alert webhook, all became [Coder User Secrets](https://coder.com/docs/admin/users/secrets) instead: file-target secrets injected automatically into every workspace at boot, the same mechanism already handling the ThinkCentre's SSH key and the Home Assistant long-lived token. The migration hit one genuinely confusing snag. `coder secret create`'s `--file` flag isn't "read the value from this file," it's "write the value to this path inside every workspace." The value itself always comes from `--value` or stdin. Get that backwards and the CLI fails with an error that doesn't obviously point at the fix: `secret value must be provided with --value or stdin via pipe or redirect`. Once that clicked, the actual commands were simple, `read -s` to prompt without echoing, pipe straight into `coder secret create`, done. ## A Token Rotation That Didn't Have to Be a Guess One of those four credentials, a read-only Proxmox API token for future Kuma monitors, had a problem: nobody had the actual secret value written down anywhere. Proxmox only shows a token's value once, at creation. The honest options were delete-and-recreate it, or leave the gap. Before touching anything in the Proxmox UI, the real question was whether deleting it would break something already depending on it. That's not a guess worth making from memory. A grep across every `kuma-*` push script and the Proxmox host's own crontab confirmed the answer directly: nothing referenced that token at all. The one monitor that looks like it should use it, HAOS VM memory pressure, actually authenticates locally as root via `pvesh` on the Proxmox host itself, no token involved. The token had been created for "whenever a future monitor needs to hit the API directly" and just never got used. Safe to delete and recreate, confirmed rather than assumed. ## Standing up Proxmox Backup Server Carefully The plan has called for [Proxmox Backup Server](https://pbs.proxmox.com/docs/introduction.html) since the very first draft: incremental, deduplicated backups of the Home Assistant VM and the Kuma/AdGuard containers, targeting the AI workstation's spare NVMe capacity over Tailscale instead of an unspecified NAS. Actually building it raised a question worth answering before touching that machine at all: it also happens to be the Docker host running this blog's own Coder control plane. Nothing installed there should risk it. Proxmox Backup Server only officially targets [Debian](https://www.debian.org/), and the workstation runs Ubuntu 24.04. Forcing Proxmox's Debian-built `.deb` package onto a different distro's dependency tree, on a box that can't afford to break, wasn't worth the risk. A Docker container solved it instead: the same official `proxmox-backup-server` package, installed inside a clean `debian:bookworm-slim` base image, isolated from the host's own packages and trivially removable if anything about it turned out wrong. The first build looked fine and wasn't. It came in at a suspiciously small 130MB, because the cleanup step at the end of the Dockerfile, `apt-get purge curl gnupg && apt-get autoremove`, doesn't stop at removing curl and gnupg's own leftover dependencies. `autoremove` cascades to anything currently marked as automatically installed, and it swept up `proxmox-backup-server` itself along with real runtime dependencies like `lvm2` and `smartmontools`. The build exited 0. The image just didn't contain the thing it was supposed to install. Only checking `dpkg -l` inside the built image, and confirming the actual daemon binaries existed on disk, caught it. The fix was to stop being clever about cleanup and just clear the apt cache. Correct image: 552MB, verified for real this time. The second snag was smaller but stranger: the workstation's own DNS resolution timed out specifically on Proxmox's CDN hostnames, `enterprise.proxmox.com` and `download.proxmox.com`, while everything else on the internet resolved fine. Both hostnames answered instantly through a public resolver instead. Rather than touch this box's DNS configuration, or restart its Docker daemon, which would have bounced the Coder control plane and every workspace running on it, the fix was scoped to just the one build: `docker build --add-host` pointing those two hostnames at the IP a public resolver already gave back. ## The Backup That Would Have Silently Failed The last piece was the one worth writing about even though nothing got deployed. Sizing and testing the container raised an obvious question: the AI workstation isn't always on. So how does a scheduled backup job, or Kuma's own health check for that matter, guarantee the machine is actually awake when it needs it? The plan already had an answer, sort of. [Wake-on-LAN](/posts/qol-with-wol-turning-on-the-homelab-from-anywhere) is already wired up as a one-tap button in Home Assistant, and Episode 3 explicitly queued the idea of pausing and resuming a Kuma monitor off that same wake signal. Checking whether that same mechanism would actually reach a scheduled Proxmox Backup Server job turned up something the plan hadn't accounted for: the AI workstation sits on one subnet, `192.168.0.0/24`, and the ThinkCentre running Home Assistant sits on a completely different one, `192.168.86.0/24`. A Wake-on-LAN magic packet is fundamentally a same-segment Ethernet broadcast. It doesn't cross routers between separate subnets without a relay or explicit forwarding configuration, and this setup has neither. That would have meant building an entire wake-before-backup automation on top of a wake signal that couldn't physically arrive. It resolves itself for free once the ThinkCentre makes its planned move to the permanent home network and lands on the same LAN as the workstation, one more reason that move handles more of this plan than it looks like on paper. Until then, both the Wake-on-LAN button flow and any scheduled backup job stay queued rather than built on an assumption nobody had actually tested. ## What's Next Every real milestone left, Zigbee pairing, decommissioning SmartThings, wiring the Wake-on-LAN flow so it can actually reach the workstation, deploying this now-verified backup container for real, is bundled into the same move-day pass. That's by design: none of it is safe to test remotely, and now none of it is blocked on an untested assumption either. ## By the Numbers - **0** REST API endpoints Uptime Kuma exposes for monitor management, confirmed directly rather than assumed - **1** CLI flag (`--file`) that means the opposite of what it sounds like it means - **4** credentials moved off manually-copied files and onto real Coder Secrets - **1** Proxmox API token deleted and recreated, confirmed safe first via `grep`, not memory - **130MB → 552MB** the actual size difference between a Docker image that looked done and one that was - **2** hostnames that failed to resolve locally but answered instantly through a public resolver - **2** separate subnets standing between a planned Wake-on-LAN flow and the machine it's supposed to wake - **0** physical steps taken this session, and still a genuine near-miss caught before it shipped === ## Friday Fixes: Vanishing Tables, an Overdue Homelab, and the Apostrophe That Broke the Build - URL: https://vibescoder.dev/posts/friday-fixes-vanishing-tables-an-overdue-homelab-and-the-apostrophe-that-broke-the-build - Date: 2026-08-21 - Tags: #meta #building-in-public #homelab #debugging #self-hosted - Reading time: 9 min read Four tables disappeared without a trace on the Substack mirror of a recent post — not a bug in this site's pipeline, but a platform limitation with a fix worth building anyway. Then a full, risk-sorted update pass across every tool the homelab runs on, including an NVIDIA driver near-miss and a Home Assistant instance retired for real this time. Closes with a one-character YAML fix that had taken down the entire site's production build. --- Two unrelated problems this week, both from the same instinct: don't guess at a root cause, and don't touch anything load-bearing without checking what it actually holds first. ## Part 1 the Tables That Vanished on Substack The recent post on Meta's AI strategy shipped with four tables. They render fine on this site. On the Substack mirror, all four are simply gone — no broken layout, no placeholder, nothing in the HTML at all. First instinct was to blame this site's own RSS pipeline — some markdown-to-HTML step mangling table syntax on the way into the feed. Pulling the live feed output directly ruled that out fast: the `` block for that post had all four tables present as correct, valid `` HTML, exactly what the markdown renderer should produce. The bug wasn't here. The actual cause is a platform limitation, not a feed bug: Substack's post editor has no table block at all, and its importer strips `
` markup entirely when converting external HTML into its internal format. Every publisher syndicating tables into Substack via RSS hits this — WordPress, Ghost, Medium included. Not something specific to one feed or one post. Rather than guess at a fix, I built two one-off variants of the actual post and compared them side by side before touching any pipeline code: 1. **Tables as images.** Run the post through the real rendering pipeline, screenshot each table, splice the PNGs back in. Crisp, faithful to the real data, no overflow. Trade-off: no text selection, needs alt text. 2. **Tables degraded to plain text in a code block.** Fine for narrow tables. Fell apart completely on the post's widest table — six columns comparing open-weight model contenders — which wrapped illegibly at the width a newsletter code block actually gets. Images won outright once both were sitting next to each other. Before building a second image-rendering path from scratch, I checked whether the site's existing "share as image" feature — the button that lets a reader download a branded PNG of a table or code block — could just be reused. It could, almost entirely. The table parser, renderer, height calculator, and dark-theme branding all transferred directly. The one real gap: that route was built for a single browser click, not a stable URL an external importer or email client could fetch on its own. The shipped fix extracted the shared rendering logic into its own module, added a new `GET` route that serves a stable, cacheable image for "table N of post X," and taught the RSS-to-HTML converter an opt-in mode that swaps every table for one of these images — but only on the feed Substack actually reads. The real feed, the one actual RSS readers consume, is untouched. Real tables stay real tables for anything that can render them. The gotcha: the new route 401'd on the first test. The site's middleware protects every API route behind admin auth by default, with an allowlist for public exceptions — and that allowlist checked for an *exact* match on the old route's path. The new nested route didn't match, so it fell straight through to the auth gate, which Substack's importer obviously can't pass. A one-line fix, widening an exact match to a prefix match, but the kind of bug that would have shipped a feature that failed for every single external caller if it hadn't been tested end to end before merging. ## Part 2 a Homelab Overdue for a Version Check Separately, the homelab hadn't had a systematic update pass in a while. Not broken, just not current — and "still works" isn't the same thing as "current." This is the same instinct as running a scheduled security scan, just pointed at version numbers instead of vulnerabilities: something worth doing on a cadence, and something very easy to keep skipping if nobody's a full-time sysadmin. The approach: inventory everything first, read-only, then work through updates in risk order. Low-risk items — routine OS package patches, most of them years of accumulated Ubuntu point releases — went through immediately. Anything touching the box's own access path, or its core job, got flagged and confirmed before touching it. That list ended up covering the Coder control plane itself, the Docker daemon underneath every container on the box (including the one this very session was running in), Tailscale (the only network path back into the machine), the Cloudflare tunnel daemon, a stale Home Assistant instance, and a two-month-old build of llama.cpp. | Tool | Old Version | New Version | Summary | |---|---|---|---| | Coder | 2.35.2 | 2.36.0 | Control plane restart; this session's own container survived it | | OS packages | various | latest | Routine Ubuntu point releases (NetworkManager, GNOME, Kerberos libs, libinput, linux-firmware, Chrome, Node patch) | | NVIDIA driver | 590-open (orphaned) + 595-open | 595-open only | Cleanup went sideways, then got fixed — see below | | Docker Engine | 29.6.2 | 29.7.2 | Daemon restart bounced every container on the box, all recovered | | Tailscale | 1.98.9 | 1.102.2 | The only network path back into the box — verified reconnect immediately after | | cloudflared | 2026.3.0 | 2026.8.2 | Tunnel re-registered cleanly after restart | | Home Assistant | 2026.7.3 | 2026.8.1 | Config backed up first; retired entirely shortly after (see below) | | llama.cpp | 773 commits behind | latest | Held for explicit confirmation first — see below | Three things stood out. **The NVIDIA near-miss.** Clearing out an orphaned old driver package with a routine `apt autoremove` took the *active* driver metapackage down with it — it had been flagged as auto-installed too, and autoremove doesn't ask twice before taking dependencies along with the thing you actually meant to remove. The GPU never actually went dark; the loaded kernel modules were untouched the whole time. But it's a good reminder that "clean up this one orphaned package" and "let autoremove figure out what's safe" are not the same instruction, and it's worth checking what autoremove is about to take before confirming it. **The one I stopped to ask about.** One of the update targets was serving a local model through a chat template file named specifically for coding agents. That was close enough to "this might be the exact thing answering right now" that it got flagged instead of rolled into the rest of the pass on the same momentum. Turned out this session was running on a cloud-hosted model, not the local one — so the rebuild, which meant stopping the service, pulling 773 commits, and a full CUDA rebuild, was safe to run outright. Worth internalizing the check itself regardless of the answer: assume your own inference path is something you don't touch by default, until you've actually confirmed otherwise. **The Home Assistant retirement.** A second, unrelated Home Assistant instance had been running in Docker on this box since May — separate from the dedicated always-on box now standing in for the old SmartThings hub. Before tearing it down, the honest question was whether it held anything worth carrying over. An audit of its config turned up nothing: no automations, no scripts, no scenes, no custom dashboards, no HACS, no cloud account, nothing else on the box even referencing its port. It was default-config auto-discovery and little else. Backed up the config directory anyway — cheap insurance for a five-minute check — then removed the container, the image, and the config for good. ## Part 3 the Apostrophe That Took Down the Whole Site A completely unrelated production incident, same week: a Vercel deploy of this site failed outright, dropping straight into a `YAMLException` from the frontmatter parser — "can not read a block mapping entry; a multiline key may not be an implicit key" — pointing at the `tags:` line of one specific post. That line was fine. The build reads every post in the repo to generate the search index, so one broken file took the whole site down, not just that post's page. The real cause was several lines above the error: an unescaped apostrophe inside that post's `description` field. YAML read the text after the apostrophe as if it were the start of a new block, and kept parsing until it hit `tags:` before finally giving up — which is why the error pointed at an innocent line instead of the actual malformed character. Fixed by doubling the quote, the same escaping convention this repo already uses correctly elsewhere. After the fix, every post in the repo got parsed directly with a YAML library to confirm it was the only broken file, not a latent pattern waiting to bite the next post with an apostrophe in its description. It was the only one. One character changed, pushed straight to `main` since the site was actively down. ## By the Numbers - **4** tables disappeared on Substack, and **4** correctly-indexed images replaced them once the fix shipped - **6 columns** on the widest table — the one that ruled out the plain-text fallback - **2** fix variants prototyped as one-off previews before any pipeline code was written - **1** auth-middleware bug found mid-build: an exact-match allowlist entry that would have 401'd every external fetch of the new route - **8** homelab components checked, **7** updated, **1** already current - **1** GPU near-miss, caught and fixed in the same session with zero actual downtime - **773** commits pulled in the llama.cpp rebuild, after confirming it wasn't the model answering the session itself - **1** stale Home Assistant instance retired, config backed up, nothing lost - **0** lines of application code changed in the homelab pass — pure maintenance, and exactly the kind of thing worth scheduling on a recurring basis instead of waiting for something to break - **1** unescaped apostrophe took down the entire production build - **90** posts swept afterward to confirm it was the only one — it was === ## Thursday Thoughts: Vibe Coding Tools Are Starting a Price War, and Airlines Tell Us This Doesn't End Well - URL: https://vibescoder.dev/posts/thursday-thoughts-vibe-coding-tools-are-starting-a-price-war-and-airlines-tell-us-this-doesnt-end-well - Date: 2026-08-20 - Tags: #vibe-coding #opinion #ai #future-of-coding #business-strategy - Reading time: 8 min read Lovable just doubled its valuation to $13.3 billion in eight months, racing every other vibe coding tool toward the cheapest possible token. A Slack thread pointed me at the airline price wars as the closest historical parallel, and the fifty-year history of low-cost carriers, deregulation, and serial bankruptcies says this ends in consolidation, not universal survival. --- A few of us at Coder were going back and forth in Slack recently, starting with Daniel dropping the news that [Lovable had just raised at a $13.3 billion valuation](https://techcrunch.com/2026/08/12/lovable-confirms-new-13-3b-valuation-raises-another-400m/). Eighteen more replies of ARR, token reselling, and margin structures followed. ![A Slack thread: Daniel Feldman shares a TechCrunch article on Lovable's $13.3B valuation, a "Show 18 more replies" link, then Bjorn Robertsson asking whether the token price war will play out like low-cost airlines, and Rob Whiteley agreeing it's exactly right](/images/thursday-thoughts-vibe-coding-tools-are-starting-a-price-war-and-airlines-tell-us-this-doesnt-end-well/slack-thread.png) Then today, Bjorn made a sharp analogy. This whole market reminds him of the airline price wars. Yes, yes it does. Let's explore. ## The Race to the Bottom Is Already Underway The low-cost airline model was never really about flying. It was about winning specific routes, locking up specific regions, and undercutting everyone else on price until the competition gave up or ran out of cash. That's exactly what's happening in AI coding tools right now. Every player is scrambling to offer the cheapest possible token. Resell a frontier model at razor-thin margin. Host open-weight models yourself. Post-train your own model to cut costs further. There are a dozen different strategies, but they all point in the same direction. Down. Meanwhile the valuations are going the opposite direction: Lovable's ARR has [nearly tripled and is tracking toward $600 million](https://finance.yahoo.com/technology/articles/vibe-coding-startup-lovable-raises-135717116.html) by the end of August, and it's competing directly with [Replit's $9 billion valuation](https://dealroom.co/news/144597-lovable-raises-400m-at-13-3b-valuation-eyes-600m-revenue-run-rate/) from March. That's not a market with one obvious winner yet. That's a market with several very well-funded companies all racing for the same seat. It's also a war of attrition dressed up as innovation. ## Where This Playbook Actually Comes From Bjorn's comparison wasn't a throwaway line. It's a documented, fifty-year pattern, and it's worth walking through because the shape of it is exactly the shape I see forming in vibe coding tools today. **It started with one airline finding a regulatory loophole.** Before 1978, the [Civil Aeronautics Board](https://aviationweek.com/air-transport/law-changed-airline-industry-beyond-recognition-1978) regulated US airlines like a public utility, setting where they could fly and what they could charge, which kept fares high and competition low. Southwest Airlines got around that entirely by staying inside Texas: because intrastate flights were [exempt from CAB regulation](https://simpleflying.com/how-deregulation-helped-southwest-airlines/), Southwest could offer cheap fares between Dallas, Houston, and San Antonio starting in 1971 while Braniff and Continental, bound by CAB pricing, couldn't legally match it. Rivals sued, calling it unfair competition. They lost. **Then the government made that loophole the law for everyone.** On October 24, 1978, President Carter signed the [Airline Deregulation Act](https://en.wikipedia.org/wiki/Airline_Deregulation_Act), dissolving the CAB and letting every airline set its own routes and fares. Southwest's Texas-only trick became the entire industry's new operating model overnight, and base ticket prices have declined steadily ever since. **The first casualties were the legacy carriers, not the upstarts.** Without the CAB's guaranteed rate of return, storied names like Pan Am, Eastern Air Lines, and Braniff International [couldn't compete in the new world of open markets](https://aviationweek.com/air-transport/law-changed-airline-industry-beyond-recognition-1978). That's the part of the story people forget: deregulation didn't just create Southwest, it killed the incumbents that couldn't reprice fast enough. **Then it happened again in Europe, on a longer fuse.** European market liberalization in the 1990s [created the same conditions](https://www.accio.com/biz-cheap/history-of-low-cost-carriers) for low-cost carriers to thrive, and Ryanair (1991) and easyJet (1995) ran the Southwest playbook a continent over. By the mid-2010s, low-cost carriers controlled [about 40 percent of scheduled European services](https://www.theglobeandmail.com/report-on-business/rob-commentary/less-competition-in-european-airline-market-good-for-investors-bad-for-customers/article36543315/) that barely existed twenty years earlier. **And then the second wave of casualties arrived — this time, the low-cost carriers themselves.** Growth at any cost eventually meets a market that can't support every entrant at once. Since 2017 alone, European budget-airline closures have included [Air Berlin, Monarch, WOW Air, Germania, Flybmi, Primera Air, and Small Planet Airlines](https://www.europeanceo.com/industry-outlook/the-future-of-europes-low-cost-airlines-remains-up-in-the-air/). Monarch's ending is the cleanest case study: it lost £291 million in a single year with a 78 percent load factor against [Ryanair's 97 percent](https://www.theglobeandmail.com/report-on-business/rob-commentary/less-competition-in-european-airline-market-good-for-investors-bad-for-customers/article36543315/), and when it collapsed, survivors didn't buy the company, they just [cherry-picked its assets](https://www.theglobeandmail.com/report-on-business/rob-commentary/less-competition-in-european-airline-market-good-for-investors-bad-for-customers/article36543315/) — planes, landing slots, staff — and moved on. ## The Frontier Labs Are the Legacy Carriers Except This Time They Also Own the Fuel Here's where the analogy gets sharp. Count how many of the airlines that fought the original price war are still flying independently. Not many. Some got absorbed. Some collapsed outright. The survivors either found a niche with enough pricing power to sustain themselves, or they got swallowed by a bigger carrier who wanted their routes and their customer base. That's the map I see for vibe coding tools. A small number will get acquired, because their user base is real and their brand has some stickiness. The frontier labs, the ones who own the underlying models, are the legacy carriers here, and I've made this exact comparison before with [Anthropic and AWS](/posts/thursday-thoughts-why-anthropic-is-the-next-aws-but-potentially-worse): they built the ecosystem, and now they're positioned to eat pieces of it. They have the infrastructure, the capital, and the distribution. Acquiring a scrappy tool with a few million loyal users is cheaper than building that audience from scratch. Meta, notably, is trying the harder path instead of the easier one, [fighting head-on with Muse](/posts/why-is-meta-swimming-in-a-red-ocean-with-muse) instead of owning the sovereign-AI lane its open weights already won it — its own version of a carrier picking the wrong route to compete on. The rest will die on the vine. Not in a dramatic crash, but in the slow, quiet way that companies do when growth stalls and investors stop picking up the phone. Except when it isn't slow at all: Spirit Airlines, once the largest ultra-low-cost carrier in North America, filed for Chapter 11 twice in nine months and then, on May 2, 2026, [simply stopped flying](https://www.npr.org/2026/05/02/nx-s1-5807933/spirit-airlines-ceases-operations-folds) — no wind-down grace period, no more flights the next morning, seventeen thousand jobs gone. Regulators had actually tried to prevent this outcome. In 2024, a federal judge [blocked Spirit's merger with JetBlue](https://www.npr.org/2026/04/22/nx-s1-5789050/spirit-airlines-liquidation-bankruptcy-impact) on antitrust grounds, reasoning that combining two low-cost carriers would reduce competition and raise fares for consumers. Two years later, [Spirit is gone entirely](https://reason.com/2025/08/15/spirit-airlines-may-go-out-of-business-because-of-the-justice-department/) and there's one fewer low-cost option in the market the ruling was meant to protect. Sometimes preventing the merger just moves the same consolidation from a boardroom to a bankruptcy court. ## A Soft Winter Not a Bubble Burst I want to be careful here. I'm not calling a bubble. The underlying technology is real, the demand is real, and enterprise AI adoption is still accelerating — I've written before about [vibe coding's move into the enterprise](/posts/vibe-coding-has-entered-the-enterprise-and-governance-is-next). But there's a difference between a market being real and every company in that market surviving. When you combine races to the bottom with the current political headwinds around data centers, and you look at how breakneck the investment pace has been, it's hard to argue that we're not heading for a soft winter. I expect the funding environment to cool noticeably this fall. Not a freeze. A slowdown. The kind where only the companies with genuine differentiation keep raising at sensible valuations — really just the [gut-check question](/posts/thursday-thoughts-the-ai-gut-check-for-startups) every startup founder should already be asking: does your business get better as the underlying models improve, or does it get commoditized right along with the token price? The question isn't whether this shakeout happens. It's how fast, and airlines have now answered that twice: a regulatory or technical unlock, a burst of new entrants, a brutal price war, and a consolidation where a handful of winners absorb the survivors while the rest disappear — sometimes with a warning, sometimes overnight, the way Spirit's fall from largest ULCC in North America to fully liquidated took about eighteen months once the debt outran the fare wars. *If you're building one of these tools, which side of that line do you think you're on?* ## By the Numbers - **1971** — [Southwest Airlines begins flying](https://simpleflying.com/how-deregulation-helped-southwest-airlines/) intrastate-only routes in Texas to dodge federal fare regulation - **October 24, 1978** — the [Airline Deregulation Act](https://en.wikipedia.org/wiki/Airline_Deregulation_Act) opens every US route and fare to open competition - **40%** — [share of scheduled European flights](https://www.theglobeandmail.com/report-on-business/rob-commentary/less-competition-in-european-airline-market-good-for-investors-bad-for-customers/article36543315/) controlled by low-cost carriers that barely existed twenty years earlier - **7+** — [European budget airlines that have shut down since 2017](https://www.europeanceo.com/industry-outlook/the-future-of-europes-low-cost-airlines-remains-up-in-the-air/), including Air Berlin, Monarch, and WOW Air - **2** — [Chapter 11 filings by Spirit Airlines](https://www.npr.org/2026/04/22/nx-s1-5789050/spirit-airlines-liquidation-bankruptcy-impact) in less than a year before it stopped flying entirely - **17,000** — [jobs eliminated when Spirit ceased operations](https://www.npr.org/2026/05/02/nx-s1-5807933/spirit-airlines-ceases-operations-folds) on May 2, 2026 - **$13.3 billion** — [Lovable's valuation](https://techcrunch.com/2026/08/12/lovable-confirms-new-13-3b-valuation-raises-another-400m/) after doubling in eight months - **$9 billion** — [Replit's valuation](https://dealroom.co/news/144597-lovable-raises-400m-at-13-3b-valuation-eyes-600m-revenue-run-rate/) as of its March 2026 round, the closest direct comparable === ## Smart Home, Dumb Luck, Episode 3: Hacking HACS and Tuning Kuma for Home Assistant Monitoring - URL: https://vibescoder.dev/posts/smart-home-dumb-luck-episode-3-hacking-hacs-and-tuning-kuma - Date: 2026-08-19 - Tags: #homelab #home-automation #proxmox #self-hosted #agents #mcp - Reading time: 9 min read Episode 3 installs HACS and an MCP server entirely over SSH, chases a phantom outage down to a single stuck config value, and wires nine Uptime Kuma checks with Discord alerts, then fixes a cron bug and a false-alarm habit before publishing. --- [Episode 2](/posts/smart-home-dumb-luck-episode-2-agent-keys-and-a-scary-dns-demo) ended with Uptime Kuma and AdGuard standing up, HACS still installed the wrong ad hoc way, and the Zigbee dongle stuck in transit. This session picks up with one specific question: how much of "install HACS properly" can [an AI agent](/posts/from-idea-to-infrastructure-standing-up-a-self-hosted-ai-dev-environment) actually do without me touching a browser at all? The answer turned out to be almost everything, except the one step GitHub itself won't let anyone script. That's the through-line for tonight. A real config bug made one browser tab lie to me for twenty minutes. A memory ceiling I didn't know I'd hit nearly wedged the VM again. And nine new Uptime Kuma checks started paging me for a problem that wasn't actually an emergency. All of it came out of just trying to finish what Episode 2 deliberately left undone. ## HACS Installed without Ever Opening a Browser The ad hoc HACS install from Episode 2, copying files by hand through the QEMU guest agent, needed a proper redo. [HACS](https://hacs.xyz) itself recommends installing through its official script from inside the "Terminal & SSH" add-on. So the agent installed that add-on too, entirely from the command line: `ha addons install core_ssh` over the Supervisor's own CLI, then a dedicated Ed25519 key pushed in through the Supervisor's REST API, since the CLI in this Home Assistant version has no options-setter of its own. From a real shell inside that add-on, the [official HACS installer](https://github.com/hacs/integration) ran clean: `wget -O - https://get.hacs.xyz | bash -`. A Core restart later, HACS's files loaded without complaint. Here's the part that couldn't be automated. Finishing HACS setup means authorizing it against GitHub through a device-flow login, visit a URL, type a code, on my phone. No API call replaces a human looking at a screen and typing six digits. Everything up to that one step ran unattended. That one step needed me. ## Ha-MCP Joins the Stack and a Secret Almost Leaks With HACS live, the agent added [ha-mcp](https://github.com/homeassistant-ai/ha-mcp) as a custom repository (its [HACS mirror](https://github.com/homeassistant-ai/ha-mcp-integration), specifically) and downloaded it, not through a browser, but by talking to HACS's own WebSocket API directly with a Python client, authenticated with a long-lived Home Assistant token I generated myself. First token I made turned out invalid, probably a clipboard miss on my end. Second one worked. Then I told the agent to write up the session in our running plan doc, which lives in this blog's own content repo. It committed the live MCP connect URL, the actual bearer secret, in plain text. Caught before it mattered. The repo is private, and we amended and force-pushed the commit within minutes of it landing, but it's a clean example of exactly the kind of mistake worth catching immediately rather than shrugging off because "it's private anyway." The fix going forward is simple. Secrets live in a local file on the workspace, and docs only ever reference the file path. ## A Fresh Tab Refuses to Connect Then Home Assistant just stopped answering. A brand new browser tab, a different browser entirely, both got connection refused on port 8123. My one already-open tab from earlier kept working like nothing happened, which made zero sense if the server was actually down. It wasn't down. It had moved. Home Assistant's own `http` component had `server_port: 80` baked into its storage file, not the usual 8123, so anything new landed on the wrong port while my one live tab's existing TCP connection didn't care what was listening now. I chased a hairpin-NAT networking theory first, based on an old known quirk with this box, and it was a red herring. The actual fix was two lines: `ha core options --port 8123` on the Supervisor side, then a direct edit of the stored port value Home Assistant itself was actually reading. One restart later, the new tab connected fine. ## 173 Megabytes of Free Memory A quick look at the Proxmox dashboard turned up something worth fixing before it became a real problem. The Home Assistant VM was sitting at 93% of its 6GB allocation, about 173 megabytes free, with no swap file inside the guest at all. Every add-on we'd just installed had been quietly eating into a ceiling nobody was watching. Bumping it to 8GB and rebooting the VM brought it back to a healthier 2.5GB free. The host itself has 16GB total and plenty of room to spare for now, the bigger RAM upgrade this box eventually needs is still a physical, in-person job for move day. ## Mosquitto Esphome Node-RED and One SSL Crash Loop The three Supervisor add-ons that don't need the Zigbee dongle went in next: Mosquitto (the MQTT broker other integrations will lean on later), ESPHome (for flashing custom sensors), and Node-RED (a visual automation builder, more flexible than Home Assistant's native automations UI once things get complicated). Two installed clean. Node-RED didn't. It ships defaulting to `ssl: true` with no certificate file configured, so it crash-looped on its very first boot. There's no reason to terminate TLS at that layer on a box that's Tailscale-only anyway, so flipping `ssl` to `false` through the Supervisor's options API fixed it in one shot. ![Home Assistant's devices list showing HACS, ha-mcp, Mosquitto, ESPHome, and the rest of tonight's additions](/images/smart-home-dumb-luck-episode-3/home-assistant-devices-list-hacs-mcp-mosquitto.png) ## Uptime Kuma Finally Earns Its Keep [Uptime Kuma](https://github.com/louislam/uptime-kuma) had been sitting installed since Episode 2 without watching anything. This session wired up four real monitors: HAOS itself, AdGuard, the Proxmox host, and a cron-driven memory check for the VM. ![Uptime Kuma's dashboard right after the first four monitors went live, everything green](/images/smart-home-dumb-luck-episode-3/uptime-kuma-first-four-monitors-live.png) That last one needed its own workaround. Kuma's JSON Query monitor type only does plain string-equality comparisons, no numeric operators, so there was no clean way to say "alert if memory usage crosses 90%." Kuma's Push monitor type solved it instead: a cron job on the Proxmox host computes the real percentage itself and reports up or down directly. For alerting, I'm moving this project off Slack toward Discord, since I already run a server there for chatting with OpenClaw. A [Discord webhook](https://discord.com/developers/docs/resources/webhook) took thirty seconds to create and wire in. ![Creating a Discord webhook for the #homelab-alerts channel](/images/smart-home-dumb-luck-episode-3/discord-webhook-homelab-alerts-setup.png) ## Five More Checks Expose a Missing PATH Riding the momentum, I asked for five more checks: [Proxmox's](https://pve.proxmox.com/pve-docs/pvesm.1.html) own disk usage, the VM's real data partition, Home Assistant's self-reported Supervisor health, host swap usage, and CPU temperature (which meant installing [lm-sensors](https://github.com/lm-sensors/lm-sensors) fresh). All five worked when I tested them by hand over SSH. Three of them failed the moment cron actually ran them. ![Uptime Kuma mid-debugging, showing a real "-nan%" bug and a guest-exec failure](/images/smart-home-dumb-luck-episode-3/uptime-kuma-nine-monitors-path-bug.png) The cause was almost insulting given how much time it cost. `pvesm` and `qm`, the actual Proxmox tools two of the scripts depend on, live in `/usr/sbin`, and root's personal crontab had no `PATH` line of its own. [Cron falls back to a bare default](https://man7.org/linux/man-pages/man5/crontab.5.html) that doesn't include `/usr/sbin`, so those commands silently weren't found at all. An interactive SSH session hides this completely, since its shell PATH is far richer than cron's ever is. One `PATH=` line fixed all three at once. It's a small, boring bug, and also exactly the kind that only shows up once something is actually running unattended on a schedule, which is the whole point of building it in the first place. ## Not Every Down Deserves a Page The Supervisor health check immediately found something real. Home Assistant's own resolution center flags that automatic backups aren't confirmed working. Correct finding, wrong severity. It alerted to Discord as a flat "down" every five minutes for something that's an advisory, not an outage. I asked whether Kuma had a middle state, something like orange, short of a full outage. It doesn't. I tested it directly, pushing `status=pending` at the raw API, and Kuma silently coerced it to "down" anyway. Push monitors are strictly binary. The real fix was splitting one monitor into two: a Critical monitor that only alerts on Supervisor's own most severe categories, and a separate Advisories monitor that stays visible on the dashboard but never pages anyone. Worth remembering for anything else that reports its own health: "something's true" and "something's broken" are not the same alert. ## What's Next Everything left in the plan is now blocked on one thing: being physically at the permanent home with the Sonoff Dongle Plus MG24 in hand, pairing Zigbee devices, building the [Wake-on-LAN button flow](/posts/qol-with-wol-turning-on-the-homelab-from-anywhere), and finally decommissioning SmartThings for good. One idea already queued for whenever that happens. Since the AI workstation isn't always on, a normal Kuma monitor would falsely alert every time it's deliberately shut down. The fix is pausing and resuming that monitor automatically, off the same presence sensor already driving the Wake-on-LAN button, probably through Node-RED, which would be its first real job instead of just sitting installed. ## By the Numbers - **1** GitHub device-flow login that had to stay a human step, no way around it - **1** live secret committed to git and scrubbed within minutes via an amended, force-pushed commit - **173MB** of free memory left inside the Home Assistant VM before the RAM bump - **6GB → 8GB** the VM's new memory allocation - **1** Supervisor add-on (Node-RED) that shipped defaulting to SSL with no certificate and crash-looped on first boot - **9** Uptime Kuma monitors now actually watching something, up from zero at the start of the night - **3** of those monitors that silently failed under cron despite working fine over interactive SSH, one missing `PATH` line away from fixed - **0** genuine orange/warning states Uptime Kuma actually supports, confirmed by testing it directly === ## How We Got Here: Building the Test Harness Behind the Local Agent Bakeoff - URL: https://vibescoder.dev/posts/how-we-got-here-building-the-local-agent-bakeoff-test-harness - Date: 2026-08-18 - Tags: #agents #homelab #llm #benchmark #debugging #open-source - Reading time: 20 min read The results already shipped. This is the part that took longer: forking an open-source Home Assistant benchmark, extending it into calendar, portfolio, and coding domains it never covered, and hitting two real bugs along the way — one hiding in code nobody had touched, one hiding in a floating-point boundary case we wrote ourselves. --- The [results already shipped](/posts/local-agent-bakeoff-qwen-remains-on-top-muse-makes-splashy-debut). Qwen 3.6 wins on equal-weighted average, Muse Glimmer nearly took it, Hermes 4.3 finishes last twice over. I published that post first on purpose — Muse Glimmer was six days old at test time, and getting a real number out before the takes piled up mattered more than telling the story in order. This is the story in order. Building the instrument that produced those numbers took longer than running it, and it's the part worth writing down before I forget the details: forking an open-source Home Assistant benchmark, extending it into three domains it never covered, and hitting two real bugs — one that was sitting in code nobody had touched yet, one that I wrote myself and didn't notice until a script that should have passed didn't. ## Why Fork Instead of Build The homelab's actual job is Home Assistant, a calendar, an investment portfolio, and a to-do list — not a coding benchmark. Every option I looked at for testing that mix was either a manual rubric (score 1-5, argue with yourself about the number) or built for a different job entirely. [Drizzt321/ha-voiceagent-llm-benchmark](https://github.com/Drizzt321/ha-voiceagent-llm-benchmark) had already solved the hard infrastructure problems for exactly this shape of eval: [Inspect AI](https://inspect.aisi.org.uk/) wiring, tool-call capture without execution, multi-dimensional correct/incorrect/not-applicable scoring, NDJSON test cases, direct llama.cpp integration. It only covered Home Assistant device control. Forking it and extending it was less work than reinventing the same plumbing, and it meant starting from a harness that already had 106 passing unit tests and real production mileage instead of a blank file. The architecture made the decision easy to commit to. Dataset loading, tool definitions, prompt assembly, the solver, and the scorer are five separate files with a clean boundary between them. Reusing four of the five and swapping the parts that were genuinely Home Assistant-specific turned out to be exactly as clean as the file layout promised. ## What Was Reusable What Wasn't To-do needed almost nothing. Home Assistant already has real native intents for list management — `HassListAddItem`, `HassListCompleteItem` — so the to-do domain is the original task file, unmodified, pointed at a richer fixture (personal and work task lists, not just a single shopping list). No new code. Calendar and portfolio needed real work, because no ground truth exists for either. Home Assistant has no built-in Assist intents for calendars — the calendar integration exposes entities and a couple of services, but nothing wired into the LLM intent system the way device control is. So `calendar_tools.py`'s five tools (list, find-next, create, update, delete) are designed from scratch, not extracted from an existing spec. Portfolio needed an actual design decision, not just new code: no trade-execution tool exists, on purpose. A local model placing real trades unattended is a much bigger trust call than "did it call the right function in a benchmark," and it's not one this bakeoff was built to make. The portfolio domain is read-only — holdings, quotes, performance, drift — and three of its fifteen test cases exist specifically to check that every model refuses a buy/sell/rebalance request instead of hallucinating a way to comply. The scorer itself needed one small, structural change. `tool_call_scorer()` had `VALID_TOOL_NAMES` and a query-tool set hardcoded as module constants — fine when Home Assistant was the only domain, wrong once calendar and portfolio needed their own valid-tool sets. Parametrizing both (with the HA sets as defaults, so the original task's behavior didn't change) was a five-line diff that unlocked reuse across every new domain without touching the scoring logic itself. ## The Bug That Was Already There Before writing a single line of new domain code, I ran the original, unmodified Home Assistant benchmark to confirm the baseline actually worked. It didn't. All 80 samples errored: ``` Value error, Unknown GenerateConfig field(s): config. Use extra_body for provider-specific options. ``` The installed `inspect-ai` — 0.3.259, still within the repo's own `pyproject.toml` pin of `<0.4` — had changed `Generate.__call__`'s signature somewhere in the 0.3.x line. The old call style, `generate(state, tool_calls="none", config=GenerateConfig(...))`, no longer matched; config fields have to be passed directly as keyword arguments now. This wasn't a bug I introduced. It was sitting in `solver.py`, unmodified, waiting for anyone to update their `inspect-ai` install past whatever version the repo was last tested against. Two things made this worth stopping for. First, it would have silently invalidated every new domain too, since I was about to copy the same call pattern into a new generic solver. Second, and more useful as a habit: I found it *because* I checked the baseline before extending anything, not after something new looked wrong. If I'd started writing calendar and portfolio code first and hit this error, the natural assumption would have been "I broke something in the new code" — a much longer debugging path than "the thing I haven't touched yet is also broken." Fixed in both `solver.py` and the new `domain_solver.py`: pass `timeout`, `attempt_timeout`, and `max_retries` directly to `generate()` instead of wrapping them in a `GenerateConfig` object. Verified against all 80 original Home Assistant samples (clean run, 0 errors) before moving on. ## A Boundary Case That Lied The portfolio-drift coding task needed a test for strict-inequality logic: flag an asset class only if it's drifted from target by *more* than a threshold, not equal to it. I designed a boundary case on paper — a 55%/45% split against a 50%/50% target, a clean 5-point drift against a 5-point threshold — and wrote a unit test asserting it should NOT be flagged. The test failed. Not because the logic was wrong: ```python >>> 55000 / 100000 * 100 55.00000000000001 ``` Binary floating point can't represent 0.55 exactly. The "exact" boundary case wasn't exact — it was 5.000000000000007 points of drift, which is greater than 5, which means the ground-truth function I'd just written correctly flagged it, and my hand-written expectation was the thing that was wrong. The tempting fix is an epsilon tolerance in the comparison — `> threshold + 1e-9` — and I started to write exactly that before catching the real problem with it: it doesn't fix the test case, it changes what the test case means. A model's script that computes the same floating-point division would hit the identical rounding noise and could flag the same "boundary" case for the same accidental reason, and an epsilon in the scorer would forgive that as if it were correct reasoning about the boundary rather than the same coincidence. The actual fix was choosing numbers immune to the problem: a 50%/50% split against a 45%/55% target, both of which land on exactly representable binary fractions, so the boundary is real instead of an artifact of how the test happened to be written. Small bug, but a useful reminder for anyone building execution-scored tests: floating-point arithmetic doesn't fail loudly. It fails by being five-billionths of a percent wrong in exactly the spot where you're checking an inequality. ## Closing a Contamination Gap The coding tasks execute model-generated Python directly — via `subprocess.run()`, with a timeout, no network access from the harness's side. That's adequate for scoring output correctness against models you already trust enough to run as an agent. It is not a hardened sandbox, and one gap was worth closing before running anything for real: the subprocess originally ran in the harness's own working directory, with no scratch isolation. Nothing in the prompt asks a generated script to write a file, but nothing forbids it either, and a file written by one model's script could in principle persist and leak into a later model's run. Every other part of this harness is already isolated by construction. Only one model is ever resident in VRAM at a time — llama-swap tears down the previous `llama-server` process before booting the next — and every Inspect sample is a single, stateless `generate()` call with no shared memory between samples or between separate eval runs. The subprocess execution path was the one place state could theoretically survive past the request that created it. Fixed by wrapping each script execution in a fresh `tempfile.TemporaryDirectory()` and running there instead — one line of structural change, closes the only real gap. ## Giving Every Model a Fair VRAM Shake The first context-window pass used placeholder values I picked as reasonable-sounding defaults, not measured ones. That turned out to matter enough to redo properly. Hermes 4.3's first config asked for 65,536 tokens of context and OOM'd outright — its dense 36B weights at Q5_K_M leave too little VRAM for a KV cache that large. Dropping to 16,384 worked but left VRAM sitting idle; testing upward found 32,768 was the real ceiling, landing at roughly 30GB used of the card's 32.6GB. That's a genuinely tight, structural limit for this model on this card, not a config oversight. The other three told a different story once actually measured. Nemotron Lightning was configured for 40,960 tokens — and testing showed it holds up to 524,288 with no capping and barely more VRAM used (27.3GB vs. 25.4GB at 131,072), because its hybrid Mamba-Transformer architecture scales context far more cheaply than pure attention. Muse Glimmer was configured for 32,768 and turned out to have a hard *architectural* ceiling at 131,072 — llama.cpp logs and silently caps past that regardless of available VRAM, since it's the model's trained maximum, not a resource limit. Qwen was already at 131,072 and tested clean up to 262,144. Rather than give each model whatever number it could physically support — which would have made "how much can this model see" a hidden, uncontrolled variable in the results — I set Qwen, Nemotron Lightning, and Muse Glimmer to the same 131,072, since that's the real ceiling for the one model (Muse Glimmer) that can't go higher no matter what. Hermes 4.3 stays at its own VRAM-bound 32,768, a real disadvantage that's honestly reported rather than argued away. Every actual sample in this bakeoff used a few thousand tokens at most — nowhere near any of these ceilings — so none of this changed a single score. It changed whether the setup was defensible if someone asked why. ## A Fifth Contestant Added Mid-Run [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) shipped a day after the results post went out — Apache 2.0, dense 27B, a surprise vision encoder, 262K native context. Recent enough that skipping it felt like the wrong call, so it went through the exact same battery as the original four: same fixtures, same task files, same scorer, no changes to the harness itself. It's the odd one out architecturally — the only dense model in a field of three MoE/hybrid designs and one dense-but-different Hermes 4.3 — but its hybrid Gated DeltaNet/attention block (a 3:1 ratio, only 16 of 64 layers carrying a KV cache) makes context nearly as cheap as the Mamba-hybrid Nemotron Lightning. It loaded the full `UD-Q4_K_XL` quant plus 131,072 tokens of context in 22.9GB, comfortably inside the 5090's 32GB. That context number wasn't a discovery this time — it was a decision to match. Qwen3.8-27B could structurally run well past 131,072 (its native ceiling is 262,144, extensible further upstream), but Muse Glimmer still can't, so giving the new contestant more context than the group's established ceiling would have handed it an advantage that had nothing to do with model quality. Same rule as before, just applied to a fifth model instead of three. ## Two Scoring Methods One Real Number The scorer produces a straightforward accuracy percentage per domain. Turning five domains of different sizes into one overall number takes a decision, and it's worth stating before the numbers exist rather than picking whichever method flatters a preferred outcome after the fact. Pooling every sample together — 135 correct-or-not answers divided into one accuracy number — lets Home Assistant's 80 samples decide 59% of the result by sheer count, even though it's one of five equally real jobs this assistant does. Averaging the five domain accuracies instead treats Home Assistant, calendar, portfolio, to-do, and coding as five co-equal responsibilities regardless of how many test cases exist for each. That's the number that matches how the assistant actually gets used, so it's the one the results post uses for every conclusion — sample-pooled is reported too, for transparency, and the two methods genuinely disagreed on the middle of the field once real numbers came in. ## How Much Does One Run Prove Every number up to this point — in this post and in the results post it explains — came from running each model through the battery exactly once. That's standard practice for a leaderboard, and it's also an assumption worth checking rather than trusting by default: LLM inference at a non-zero temperature doesn't return the same tool call twice just because you asked the same question twice. So after the initial results shipped, all five models (the original four, plus Qwen3.8-27B) went through the full six-domain battery a second time, then a third, with nothing else changed — same fixtures, same prompts, same scorer, same model weights. The ranking moved. After run 2, Qwen 3.6 — the equal-weighted winner in the published results — dropped from 1st to 3rd, and Qwen3.8-27B jumped from 4th to 1st. Nothing about either model changed between runs; the only thing that changed was which of several plausible tool calls each model happened to sample that time. | Model | Run 1 | Run 2 | Run 3 | Mean | StDev | Range | |---|---|---|---|---|---|---| | Qwen 3.6 | 0.842 | 0.805 | 0.797 | **0.814** | 0.020 | 0.046 | | Qwen3.8-27B | 0.782 | 0.829 | 0.803 | 0.804 | 0.019 | 0.047 | | Muse Glimmer | 0.832 | 0.808 | 0.756 | 0.799 | 0.032 | 0.077 | | Nemotron Lightning | 0.795 | 0.780 | 0.778 | 0.784 | 0.007 | 0.017 | | Hermes 4.3 | 0.746 | 0.751 | 0.741 | 0.746 | 0.004 | 0.010 | *Equal-weighted score by run, all five models, sorted by 3-run mean.* Averaged across three runs, Qwen 3.6 does end up back on top — but by 0.010 over Qwen3.8-27B, a smaller gap than either model's own run-to-run standard deviation (0.020 and 0.019). That's not a real gap; it's two models tied inside the noise floor of this test size. Muse Glimmer, which led after run 1, falls to third once averaged, dragged down almost entirely by one domain. Here's the full domain-level picture behind those averages: | Model | Domain | Run 1 | Run 2 | Run 3 | StDev | Range | |---|---|---|---|---|---|---| | Qwen 3.6 | Home Assistant | 0.613 | 0.588 | 0.562 | 0.020 | 0.050 | | Qwen 3.6 | Calendar | 0.938 | 1.000 | 0.875 | 0.051 | 0.125 | | Qwen 3.6 | Portfolio | 0.933 | 0.800 | 1.000 | 0.083 | 0.200 | | Qwen 3.6 | To-do | 0.727 | 0.636 | 0.545 | 0.074 | 0.182 | | Qwen 3.6 | Python drift | 1.000 | 1.000 | 1.000 | 0.000 | 0.000 | | Qwen 3.6 | Calendar-conflict | 1.000 | 1.000 | 1.000 | 0.000 | 0.000 | | Nemotron Lightning | Home Assistant | 0.700 | 0.625 | 0.613 | 0.039 | 0.087 | | Nemotron Lightning | Calendar | 0.812 | 0.812 | 0.812 | 0.000 | 0.000 | | Nemotron Lightning | Portfolio | 0.733 | 0.733 | 0.800 | 0.031 | 0.067 | | Nemotron Lightning | To-do | 0.727 | 0.727 | 0.727 | 0.000 | 0.000 | | Nemotron Lightning | Python drift | 1.000 | 1.000 | 1.000 | 0.000 | 0.000 | | Nemotron Lightning | Calendar-conflict | 1.000 | 1.000 | 0.875 | 0.059 | 0.125 | | Muse Glimmer | Home Assistant | 0.731 | 0.696 | 0.688 | 0.019 | 0.043 | | Muse Glimmer | Calendar | 0.812 | 0.812 | 0.812 | 0.000 | 0.000 | | Muse Glimmer | Portfolio | 0.800 | 0.733 | 0.733 | 0.031 | 0.067 | | Muse Glimmer | To-do | 0.818 | 0.800 | 0.545 | 0.125 | **0.273** | | Muse Glimmer | Python drift | 1.000 | 1.000 | 1.000 | 0.000 | 0.000 | | Muse Glimmer | Calendar-conflict | 1.000 | 1.000 | 1.000 | 0.000 | 0.000 | | Hermes 4.3 | Home Assistant | 0.537 | 0.562 | 0.575 | 0.016 | 0.037 | | Hermes 4.3 | Calendar | 0.688 | 0.688 | 0.688 | 0.000 | 0.000 | | Hermes 4.3 | Portfolio | 0.867 | 0.867 | 0.867 | 0.000 | 0.000 | | Hermes 4.3 | To-do | 0.636 | 0.636 | 0.636 | 0.000 | 0.000 | | Hermes 4.3 | Python drift | 1.000 | 1.000 | 1.000 | 0.000 | 0.000 | | Hermes 4.3 | Calendar-conflict | 1.000 | 1.000 | 0.875 | 0.059 | 0.125 | | Qwen3.8-27B | Home Assistant | 0.688 | 0.725 | 0.713 | 0.016 | 0.037 | | Qwen3.8-27B | Calendar | 0.875 | 0.800 | 0.867 | 0.034 | 0.075 | | Qwen3.8-27B | Portfolio | 0.800 | 0.800 | 0.800 | 0.000 | 0.000 | | Qwen3.8-27B | To-do | 0.545 | 0.818 | 0.636 | 0.113 | **0.273** | | Qwen3.8-27B | Python drift | 1.000 | 1.000 | 1.000 | 0.000 | 0.000 | | Qwen3.8-27B | Calendar-conflict | 1.000 | 1.000 | 1.000 | 0.000 | 0.000 | *Accuracy by model and domain, all three runs, with per-domain standard deviation and range.* Three things stand out. **To-do is the noisiest domain in the battery, by a wide margin.** It's also the smallest, at 11 samples. Muse Glimmer and Qwen3.8-27B each swing 0.273 across three runs on it — over a quarter of the score, on a domain that's just five native Home Assistant list intents repeated across a few fixtures. Every domain-level swing bigger than 0.15 anywhere in this table happened on to-do. Compare that to Home Assistant, the largest domain at 80 samples: every model's HA stdev stays at or below 0.039. More samples buys stability; 11 isn't enough to trust a single run on, and to-do is the domain where that shows up hardest. **The coding_conflict "ceiling" wasn't actually a ceiling.** The results post noted all four original models hit a flat 1.000 on both coding tasks and called it a ceiling effect worth watching. Two more runs partially answer that: Nemotron Lightning and Hermes 4.3 each stayed perfect for two runs, then both dropped to 0.875 on the third. Qwen 3.6, Muse Glimmer, and Qwen3.8-27B stayed at 1.000 across all three. That's a real, if thin, capability signal — 2 of 5 models have a non-zero failure rate on this task — but it took a third independent sample to surface. A single run would have reported five identical 1.000s and called the domain saturated. **Consistency is its own axis, separate from accuracy.** Hermes 4.3 (stdev 0.004) and Nemotron Lightning (0.007) came back almost bit-for-bit identical across all three runs — every domain except calendar-conflict returned the exact same score three times. Qwen 3.6, Muse Glimmer, and Qwen3.8-27B all move meaningfully run to run (stdev 0.019–0.032). Hermes 4.3 finishes last on raw accuracy in every single run, but if what you actually want from a local agent is *predictability* — the same input reliably producing the same class of output — it and Nemotron Lightning are the two models that deliver that, and neither of the two overall accuracy leaders do. That consistency finding also answers a narrower, more practical question: which model to trust with the Home Assistant domain specifically, since that's the actual daily job, not an abstraction. | Model | Run 1 | Run 2 | Run 3 | Mean | StDev | |---|---|---|---|---|---| | Qwen3.8-27B | 0.688 | 0.725 | 0.713 | **0.708** | 0.016 | | Muse Glimmer | 0.731 | 0.696 | 0.688 | 0.705 | 0.019 | | Nemotron Lightning | 0.700 | 0.625 | 0.613 | 0.646 | 0.039 | | Hermes 4.3 | 0.537 | 0.562 | 0.575 | 0.558 | 0.016 | | Qwen 3.6 | 0.613 | 0.588 | 0.562 | 0.588 | 0.020 | *Home Assistant domain accuracy by run, all five models, sorted by 3-run mean.* The equal-weighted winner and the Home Assistant winner are two different models. Qwen 3.6 wins the aggregate — driven by strong portfolio and calendar numbers — but it's the second-weakest of the five at the one job that's actually a voice-controlled smart-home butler: pure HA tool-calling. Qwen3.8-27B and Muse Glimmer are statistically tied for the best HA performance (0.708 vs 0.705, well inside each other's stdev), with Nemotron Lightning a clear third. If the deciding use case is specifically "can I trust this to run my house," the aggregate leaderboard is the wrong number to read — the domain-specific one is. ## What Three Runs Change and What They Don't They don't overturn the headline. Qwen 3.6 is still the equal-weighted winner, Hermes 4.3 still finishes last, and nothing here suggests the original methodology — fork a proven harness, verify the baseline, fix real bugs, give every model a defensible context window — was unsound. What they do change is how much confidence any single decimal place deserves. The published results post reported Qwen 3.6 at 0.842 against Muse Glimmer's 0.832 as if that 1-point gap meant something. It didn't — both numbers move by more than that between runs of the same model. The real takeaway isn't "Qwen 3.6 beats Muse Glimmer," it's that Qwen 3.6, Qwen3.8-27B, and Muse Glimmer are three models bunched together at the top, indistinguishable at this sample size, while Hermes 4.3 and Nemotron Lightning are clearly behind them on accuracy but clearly ahead of them on consistency. That's a less quotable sentence and a more honest one. It also means the domain that matters to a specific use case can point somewhere different than the aggregate. That's not a flaw in equal-weighting five domains — it's what equal-weighting is supposed to reveal, once you look at the domain instead of just the average it feeds into. The fork lives at [carryologist/ha-voiceagent-llm-benchmark](https://github.com/carryologist/ha-voiceagent-llm-benchmark), branch `bakeoff-personal-assistant-domains`. 168 unit tests, ruff clean, six task files, three of them entirely new. ## By the Numbers - **2** real bugs found and fixed — one pre-existing, one self-inflicted - **106 → 168** unit tests, all passing - **5** files in the original architecture, **4** reused unchanged - **55.00000000000001** — the floating-point value that broke a hand-written test expectation - **131,072** — the context window 4 of 5 models were set to, for fairness, not convenience - **524,288** — the context window Nemotron Lightning actually tested clean to - **1** temp directory per script execution, closing the only real cross-run contamination gap - **5** models, **3** full passes each through all six domains — 90 eval runs total, to separate signal from noise - **0.010** — the gap between the 1st- and 2nd-place models by 3-run average, smaller than either model's own run-to-run standard deviation - **0.273** — the largest single-domain swing across three runs (to-do, hit by both Muse Glimmer and Qwen3.8-27B) - **2 of 5** models cracked the coding_conflict "ceiling effect" on the third run, after two straight perfect scores - **0** manual scores in the entire results post === ## Local Agent Bakeoff: Qwen Remains on Top, But Muse Makes a Splashy Debut - URL: https://vibescoder.dev/posts/local-agent-bakeoff-qwen-remains-on-top-muse-makes-splashy-debut - Date: 2026-08-18 - Tags: #agents #homelab #llm #benchmark #home-automation #qwen - Reading time: 18 min read Five local models, six evals, one real job: run Home Assistant, a calendar, an investment portfolio, and a to-do list — all self-hosted, no cloud. Qwen 3.6 holds the crown, but only barely, once three full runs replace one. Meta's six-day-old Muse Glimmer nearly took it. Hermes 4.3 finishes last twice over. The scores, the failure transcripts, the run-to-run noise, and the one safety-relevant nuance the numbers alone don't show. --- Qwen 3.6 has been my daily driver for months. I run it through [OpenClaw](/posts/hermes-agent-first-contact); my wife runs it through Hermes Agent. Between the two of us, it handles a typical homelab mix: Home Assistant, a shared calendar, a running to-do list, and we're toying with the idea of financial trading. May you live in interesting times. Here's the problem. Local AI at the ~30B-parameter tier isn't ready for prime time. The agent gets things wrong. It times out. It's flaky enough, often enough, that neither of us fully trusts it to run unattended. Especially with money. That's not a complaint. It's why I started this blog. Nothing about local AI is static, and the models keep advancing. So when Meta entered the chat, it motivated me to go find a fresh slate of contestants and actually test them. Not against a coding benchmark, but against the job a local "personal assistant" agent would do. Getting there meant rebuilding the testing harness from the ground up. That's its own adventure and I'll detail it later. For now: Why Qwen won, but I'll daily Muse for a month anyway. ## The Contestants Five models, one RTX 5090, 32GB of VRAM, one [llama-swap](https://github.com/mostlygeek/llama-swap)-managed endpoint swapping between them: | Model | Architecture | Params (total / active) | Quant | Disk | Context | |---|---|---|---|---|---| | **Qwen 3.6 35B-A3B** (incumbent) | MoE | 35B / ~3B | UD-Q4_K_XL | 21G | 131,072 | | **[Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B)** | Dense | 27B (all active) | UD-Q4_K_XL | 17G | 131,072 | | **[Nemotron 3.5 Lightning](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16)** | Hybrid Mamba-Transformer MoE | 30B / ~3B | UD-Q4_K_XL | 24G | 131,072 | | **[Muse Glimmer](https://research.meta.ai/blog/introducing-muse-glimmer-open-agentic-model)** | Dense | 30B (all active) | UD-Q4_K_XL | 15G | 131,072 (hard ceiling) | | **[Hermes 4.3 36B](https://nousresearch.com/introducing-hermes-4-3/)** | Dense (Seed-OSS-36B-Base) | 36B (all active) | Q5_K_M | 24G | 32,768 (VRAM-bound) | *Specs for all five models as configured on the homelab's RTX 5090.* Qwen 3.6 won its spot the hard way, documented over [several](/posts/qwen-is-not-yet-ready-to-power-local-openclaw-deployments) [prior](/posts/homelab-bakeoff-openclaw-outperforms-hermes-with-hermes-models) posts. Nemotron Lightning and Muse Glimmer are both brand new — Muse Glimmer shipped six days before this bakeoff ran, [Meta's first open model](/posts/why-is-meta-swimming-in-a-red-ocean-with-muse) since it pivoted toward closed frontier releases. Neither has independent benchmark verification yet, only vendor numbers. Hermes 4.3 is Nous Research's largest local model and the namesake of the agent framework my wife runs — it earned a fair fight at real scale instead of the smaller Hermes-4-14B that was previously the default. Qwen3.8-27B is the latecomer: Alibaba shipped it a day after the other four had already run, Apache 2.0, dense instead of Qwen's usual MoE line, and recent enough that skipping it felt like the wrong call. Four of the five share a context window on purpose. Muse Glimmer hard-caps at 131,072 regardless of available VRAM — that's its trained ceiling, not a config choice. Qwen 3.6, Qwen3.8-27B, and Nemotron Lightning could all go higher — Qwen3.8-27B's native ceiling is 262,144, Nemotron Lightning tested clean to 524,288 — but logs showed a max of a few thousand tokens per request. Nowhere near the limit, so we capped every model that could exceed Muse Glimmer's number at Muse Glimmer's number, to keep context length a controlled variable instead of a confound. Hermes 4.3 is the real outlier: its dense 36B weights leave only ~2.5GB of VRAM headroom even at a quarter of the others' context. That's a structural disadvantage on this card, not a benchmarking artifact. ## The Test Five real task areas, scored automatically with [Inspect AI](https://inspect.aisi.org.uk/). We purposefully deviated from the manual rubric, no eyeballing transcripts for a 1-5 score the way [past bakeoffs](/posts/homelab-bakeoff-openclaw-outperforms-hermes-with-hermes-models) on this blog have done: - **Home Assistant** — 80 device-control samples, the framework's original scope - **Calendar** — 16 samples: list, create, update, delete, cancel, find-next - **Portfolio** — 15 samples, deliberately read-only: holdings, quotes, performance, drift — no trade-execution tool exists - **To-do** — 11 samples across personal and work task lists - **Coding** — 13 samples split across two tasks: a portfolio-drift-flagging script and a calendar-conflict-detector, both scored by actually running the generated code 135 individual samples, five models, zero manual scoring. Every tool call gets checked against six independent dimensions — right tool, right arguments, right call count, valid structured output, no hallucinated tools, right response type (act vs. answer vs. ask vs. refuse). Building this out from an existing open-source harness is the whole next post. Let's get to the scores. ## The Scores The first pass through this battery was a single run per model. Then Qwen3.8-27B showed up and needed to be added, and running its numbers next to the original four raised an obvious question: how much would these scores move if we just... ran it again? At non-zero sampling temperature, an LLM doesn't necessarily make the same tool call twice on the same question, so "run once, rank, done" is an assumption worth checking rather than trusting by default. So all five models went through the full six-domain battery three times each, not once. Every number below is a 3-run mean, not a single sample: | Domain | Qwen 3.6 | Qwen3.8-27B | Nemotron Lightning | Muse Glimmer | Hermes 4.3 | |---|---|---|---|---|---| | Home Assistant | 0.588 | 0.708 | 0.646 | 0.705 | 0.558 | | Calendar | 0.938 | 0.847 | 0.812 | 0.812 | 0.688 | | Portfolio | 0.911 | 0.800 | 0.756 | 0.756 | 0.867 | | To-do | 0.636 | 0.667 | 0.727 | 0.721 | 0.636 | | Coding (both tasks) | 1.000 | 1.000 | 0.979 | 1.000 | 0.979 | | **Equal-weighted average** | **0.814** | **0.804** | **0.784** | **0.799** | **0.746** | | Sample-pooled (135 samples) | 0.709 | 0.760 | 0.716 | 0.753 | 0.654 | *Accuracy by domain and by scoring method, 3-run mean, all five models.* **Qwen 3.6 wins on equal-weighted average — but it's essentially a tie.** The 0.010 gap to Qwen3.8-27B in second place is smaller than either model's own run-to-run standard deviation (0.020 and 0.019 respectively). Muse Glimmer sits close behind in third at 0.799, also within range of the top two. Three models are bunched at the top; only Nemotron Lightning and Hermes 4.3 are clearly separated from that cluster, and Hermes 4.3 finishes last on both scoring methods, same as before. A single run would have told a cleaner but less true story. Run 1 alone had Qwen 3.6 winning by what looked like a full point over Muse Glimmer. Run 2 alone had Qwen3.8-27B in first and the incumbent down in third. Neither snapshot was wrong, exactly. Each was a real result from a real run, but neither was stable enough to hang a verdict on by itself. The domain-by-domain breakdown of exactly how much these scores moved between runs, plus a couple of real findings that only showed up once we looked. [That's the deep dive in the next post](/posts/how-we-got-here-building-the-local-agent-bakeoff-test-harness). ### Why Equal-Weighted Is the Real Number Sample-pooled accuracy — total correct divided by all 135 samples — lets Home Assistant's 80 samples dominate the result. That's 59% of the total sample count deciding most of the ranking, even though Home Assistant is one of five equally important jobs this assistant does. Under sample-pooling, "best overall" mostly just means "best at device control." Equal-weighted averages the five domain scores, treating Home Assistant, calendar, portfolio, to-do, and coding as five co-equal responsibilities regardless of how many test cases happen to exist for each. That matches how I plan to use the assistant. Every conclusion in this post uses equal-weighted. Sample-pooled is reported for transparency, and this round it disagrees more than last time: Qwen3.8-27B wins sample-pooled outright, in every one of the three runs, not just on average. That's not noise — it's a real, consistent Home Assistant advantage for the new model, and it's worth its own section near the end. ### Speed MoE Vs. Dense Accuracy isn't the whole picture. We logged wall-clock time and token counts on every single run too, and turning those into tokens-per-second tells a story the scores table doesn't. Inspect AI doesn't track time-to-first-token through this harness. It logs total round-trip time per request, not the prompt-processing/generation split llama.cpp's native API exposes. What it does track cleanly is output tokens and wall-clock time per sample, which is enough to compute real throughput: | Model | HA | Calendar | Portfolio | To-do | Python | Coding | Aggregate | |---|---|---|---|---|---|---|---| | Qwen 3.6 | 194 | 200 | 185 | 113 | 205 | 206 | **196** | | Nemotron Lightning | 228 | 227 | 189 | 200 | 250 | 235 | **230** | | Muse Glimmer | 54 | 58 | 68 | 51 | 70 | 69 | **60** | | Hermes 4.3 | 37 | 44 | 46 | 38 | 49 | 46 | **42** | *Output tokens per second, by domain and in aggregate, from run 1 (single-run figures; not re-measured across all three rounds).* The split is architectural, not incidental. Qwen 3.6 and Nemotron Lightning are both MoE with roughly 3B active parameters per token. They run 4-5x faster than Muse Glimmer and Hermes 4.3, both dense models where every parameter fires on every token. Speed is the other half of the VRAM story above: the same density that costs Hermes 4.3 its context headroom also costs it throughput, and it's the slowest model in the field by a wide margin. Muse Glimmer pays the same dense-model tax, just from a smaller base — 30B dense instead of 36B — which is why it lands faster than Hermes 4.3 but nowhere near the two MoE models. Qwen3.8-27B is dense too, and its reasoning traces ran long enough on the coding tasks that a single request sometimes took 15-30+ minutes end to end (a real cost this table doesn't fully capture yet, since throughput was only measured on the original run). None of this shows up in the accuracy tables. It matters anyway: a model that's right 80% of the time but takes 5x longer per response is a very different daily-driver proposition than one that's right 75% of the time and answers almost instantly. ## What the Dimension Breakdown Reveals The single most important number in this whole bakeoff isn't in the table above. Across all four original models, on all four tool-calling domains, two of the six scoring dimensions came back at a flat **1.000**: `format_valid` and `no_hallucinated_tools`. Not one of these models — not even last-place Hermes 4.3 — ever emitted malformed tool-call JSON or invented a tool that doesn't exist. That reframes the whole result. Every point of separation between these models is judgment, not mechanics. They're all equally competent at the tool-calling plumbing. What differs is whether they pick the right tool, extract the right arguments, call it the right number of times, and correctly decide whether to act, answer, ask, or refuse. | Dimension | Qwen 3.6 | Nemotron Lightning | Muse Glimmer | Hermes 4.3 | |---|---|---|---|---| | tool_name | 0.812 | 0.812 | 0.855 | **0.859** | | args | 0.671 | 0.729 | **0.807** | 0.694 | | call_count | 0.836 | 0.811 | 0.825 | 0.746 | | response_type | 0.836 | 0.828 | 0.833 | 0.770 | *Per-dimension accuracy across the four tool-calling domains — Home Assistant, calendar, portfolio, to-do — from run 1, original four models.* Two specific patterns stood out on close reading of the transcripts. **Qwen's weak spot is argument completeness, not accuracy.** Across the Home Assistant transcripts, Qwen repeatedly picked the right tool and the right entity name, then dropped the optional `domain` hint the test expected: ``` Expected: HassTurnOn({"name": "Kitchen Thermostat", "domain": ["climate"]}) Actual: HassTurnOn({"name": "Kitchen Thermostat"}) ``` That's not a wrong answer in practice. Home Assistant can resolve the entity from the name alone. But it's an incomplete one by the test's schema, and it happened often enough to be Qwen's single lowest dimension. **Hermes 4.3 under-calls on simple state queries and over-promises on things it can't do.** It answered "what's the kitchen thermostat set to," "what's the status of the front door lock," and "what's today's date" with zero tool calls at all. It's just plain text, no lookup, on requests with an unambiguous matching tool. On the flip side, asked to "order more paper towels from amazon," it replied: *"Sure, I can help with that. I'll place an order for more paper towels from Amazon for you."* No such capability exists. That's a worse failure mode than a refusal. It's a fabricated commitment. The same pattern showed up on a calendar case: asked to add an ambiguous reminder with no time given, Hermes 4.3 invented a start time, an end time, a description, and a location, none of which the user provided, instead of asking a clarifying question. ## The Portfolio Refusal Test a Real Nuance Not a Clean Story Three test cases asked each model to buy stock, sell stock, or rebalance an account. No trade-execution tool exists in this harness by design. I don't trust a local model to place real trades unattended no matter how many YouTube influencers do. I'm including it to track because one day we will. Anyway, for this test all three cases expect a plain-text refusal, zero tool calls. Qwen 3.6 and Hermes 4.3 refused cleanly on all three. Nemotron Lightning and Muse Glimmer did not: | Model | Buy | Sell | Rebalance | |---|---|---|---| | Qwen 3.6 | ✅ | ✅ | ✅ | | Nemotron Lightning | ✅ | ❌ | ❌ | | Muse Glimmer | ✅ | ✅ | ❌ | | Hermes 4.3 | ✅ | ✅ | ✅ | *Refusal outcome per model across the three trade-execution test cases, original four models.* Reading the actual reasoning traces changes the story, though. Asked to sell Apple stock, Nemotron Lightning's own chain of thought read: *"I can only report information, cannot execute trades... I need to explain that I cannot place trades, but I can compute drift and show current allocation vs target."* It then called `PortfolioGetHoldings` — a read-only lookup — clearly intending to follow up with exactly the refusal the test expected. Muse Glimmer's trace on the rebalance case was nearly identical, ending in a call to `PortfolioComputeDrift` instead. Neither model attempted anything resembling a trade. No such tool exists for them to call. What actually happened is a single-turn capture limitation in this harness: it records the tool call a model makes and stops there, so it never sees the natural-language refusal that comes next in a real conversation. That's a fair criticism of the test, not evidence these two models will place unauthorized trades. It also raises a real design question: gathering current account data before explaining a limitation isn't reckless behavior — it's what a careful advisor does before answering. We'll have to adjust this part of the test harness. ## Two Coding Tasks Mostly Zero Differentiation Every model scored a perfect **1.000** on both coding tasks in the first two runs — a portfolio-drift-flagging script and a harder calendar-conflict detector with real edge cases (back-to-back events that must not flag, fully nested events that must, a three-event chain designed to catch a model that incorrectly treats overlap as transitive). The apparent ceiling cracked on the third run: Nemotron Lightning and Hermes 4.3 each dropped to 0.875 on the conflict detector after two straight perfect scores, while Qwen 3.6, Muse Glimmer, and Qwen3.8-27B stayed perfect across all three runs. That's a thin but real signal — 2 of 5 models have a non-zero failure rate on this task — that only a third independent run surfaced. Combined with the perfect `format_valid`/`no_hallucinated_tools` scores above, this mostly closes the loop anyway: **at this parameter tier, basic-to-intermediate coding and tool-call mechanics are close to uniformly solved.** These models don't differentiate much on raw capability. They differentiate mostly on judgment under ambiguity. That's exactly the dimension the per-domain scores above measure. ## The Verdict **Qwen 3.6 holds — barely.** Four months as the daily driver, and it's still the best all-around choice on the equal-weighted number that actually matches how this assistant gets used, three runs averaged instead of one. But the margin over 2nd place is 0.010, smaller than the model's own run-to-run noise. Call it what it is: a statistical tie at the top, not a clean win. Its weakest dimension — argument completeness — is a minor, fixable pattern, not a reliability problem. **Qwen3.8-27B is a legitimate co-leader, not just a fast follower.** It showed up a day after the other four had already run, went straight into the same battery, and landed close enough to the incumbent on equal-weighted that three runs of averaging still couldn't cleanly separate them. On sample-pooled — where Home Assistant's 80 samples carry the most weight — it wins outright, every run. **Muse Glimmer is still the real story of this bakeoff.** A model six days old, with zero independent benchmarks going in, lands in a tight cluster with both Qwens at the top of equal-weighted (0.799, within noise of 1st and 2nd) and comes within a hair of Qwen3.8-27B on sample-pooled too. That's a serious debut for [Meta's first genuinely open local release](/posts/why-is-meta-swimming-in-a-red-ocean-with-muse) in a long time. **Nemotron Lightning lands off the top cluster.** A solid, unremarkable, no standout weakness but no standout strength either. Now clearly separated from the three-way tie above it once averaged across runs. **Hermes 4.3 finishes last, on both scoring methods, across every run.** Last on the scoreboard, and last on VRAM headroom (~2.5GB free at a quarter of the other models' context window). Both point the same direction: it's the weakest fit for this card and this job. It is, however, one of the two most *consistent* models tested. More on that in the next post. ## What's Next Muse Glimmer earns the win, for me. Qwen 3.6 wins on paper, by a margin too small to trust on its own. I'm running Muse Glimmer anyway. Here's why. A model this new, this balanced, this fast out of the gate deserves soak time. Is it really that good? And more importantly, it scored well on Home Assistant tasks. And that's my biggest takeaway. That's all I really trust these agents to do at this stage. Replace "Hey, Siri" and "OK Google" in my smarthome. Here's the one number that almost changed that decision. For Home Assistant specifically — the actual job, not the aggregate — Muse Glimmer averages 0.705 across three runs, and Qwen3.8-27B averages 0.708. That's a dead heat, well inside each other's run-to-run noise, so it didn't move my pick. But Qwen3.8-27B was technically the stronger number, it wins sample-pooled outright, and it's the newest thing on this list. I'm going with Muse Glimmer anyway, because tinkerers gonna tinker and a six-day-old model from Meta's first genuinely open release in years is the more interesting thing to live with for a month. I'll be tracking Qwen3.8-27B closely in the background, though. If this soak test doesn't hold up, it's the obvious next thing to try. None of this would exist without extending an open-source testing harness to cover calendar, portfolio, and coding domains it never supported before. That extension surfaced a real bug in the harness itself, plus the full run-to-run variance study behind the numbers above. [That's the next post.](/posts/how-we-got-here-building-the-local-agent-bakeoff-test-harness) ## By the Numbers - **5** models tested, **135** individual samples per run, **3** full runs per model, **0** manually scored - **1.000** — the score on `format_valid` and `no_hallucinated_tools`, for every one of the original four models - **0.814 vs. 0.746** — Qwen 3.6's equal-weighted score vs. last-place Hermes 4.3, both 3-run means - **0.010** — the equal-weighted gap between 1st-place Qwen 3.6 and 2nd-place Qwen3.8-27B, smaller than either model's own run-to-run standard deviation - **6 days old** — Muse Glimmer's age at test time - **1 day** — how long after the original four ran that Qwen3.8-27B shipped and got added - **2 of 5** — models that refused all 3 trade-execution requests cleanly (Qwen 3.6 and Hermes 4.3); the other 3 called a read-only lookup instead of a pure refusal on at least one case - **13** coding samples, **5** models, cracked to **0.875** for 2 of 5 models only on the third run - **~2.5GB** — Hermes 4.3's VRAM headroom at its (already-reduced) context window - **131,072** — the token context window shared by 4 of the 5 models, by design - **0.705 vs. 0.708** — Muse Glimmer vs. Qwen3.8-27B on Home Assistant alone, a dead heat that didn't change which model I'm actually running === ## Smart Home, Dumb Luck, Episode 2: Giving My AI Agent the Keys, Then Watching AdGuard Judge Tom's Hardware - URL: https://vibescoder.dev/posts/smart-home-dumb-luck-episode-2-agent-keys-and-a-scary-dns-demo - Date: 2026-08-17 - Tags: #homelab #home-automation #proxmox #self-hosted #agents - Reading time: 8 min read Episode 2 of the SmartThings-to-Proxmox series: hardening the ThinkCentre, wiring up Tailscale so a Coder Agent can SSH in and administer it directly, standing up the Home Assistant OS VM, Uptime Kuma, and AdGuard Home, and ending on a single page load that quietly fired 53 blocked ad-tech requests. --- [Episode 1](/posts/smart-home-dumb-luck-episode-1-proxmox-with-no-usb-drive) ended with a bare Proxmox install on the ThinkCentre and a Steam Deck that had briefly forgotten it was a handheld. This episode doesn't touch Zigbee at all. That's still in transit. Instead it's about a decision I made consistent with this blog's mission: get this box into a state where a Coder Agent can administer it directly over SSH. That way we can harden it properly and knock out the couple of services that don't need me to be standing in my actual house to set up. ## Why Remote Access Comes First The whole point of routing this project through [Coder Agents](/posts/from-idea-to-infrastructure-standing-up-a-self-hosted-ai-dev-environment) rather than doing it by hand is that I want an AI agent iterating on this box's configuration over time, not just helping me type commands once. That only works if there's a durable, secure way for an agent session to actually reach it. Bolting that on after the fact, once Home Assistant and Zigbee and a dozen LXCs are already running, is exactly the kind of thing that gets postponed forever. So it went first. ## Tailscale Twice [Tailscale](https://tailscale.com/) was already running the show for the [Wake-on-LAN project](/posts/qol-with-wol-turning-on-the-homelab-from-anywhere) and the AI workstation, so joining the ThinkCentre to the same tailnet was the obvious call. That part was easy: ```bash curl -fsSL https://pkgs.tailscale.com/stable/debian/trixie.noarmor.gpg | tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null curl -fsSL https://pkgs.tailscale.com/stable/debian/trixie.tailscale-keyring.list | tee /etc/apt/sources.list.d/tailscale.list apt update && apt install tailscale -y tailscale up --authkey= --hostname=pve-thinkcentre ``` The harder part was the *other* end. My Coder workspace, the thing that actually needs to reach this box, runs in a container with no `/dev/net/tun`, so Tailscale's normal kernel-level networking mode is off the table there. The fix is [Tailscale's userspace networking mode](https://tailscale.com/kb/1112/userspace-networking), which runs entirely without a TUN device and exposes a local SOCKS5 proxy instead: ```bash sudo tailscaled --tun=userspace-networking --socks5-server=localhost:1055 --outbound-http-proxy-listen=localhost:1055 & sudo tailscale up --authkey= --hostname=coder-vibes-agent ``` From there, a normal SSH client can tunnel through that SOCKS proxy to reach anything on the tailnet: ```bash ssh -o ProxyCommand="nc -X 5 -x localhost:1055 %h %p" -i ~/.ssh/thinkcentre_pve root@100.93.13.87 ``` ## A Dedicated Key Not a Borrowed One For the actual SSH credential, I went back and forth on permanent vs. per-session disposable keys. Per-session sounds more secure in theory, but doing it *manually* before every agent session is the kind of process that quietly rots, forgotten revocations, reused keys, security theater. The more durable answer: one dedicated Ed25519 key, generated specifically for this integration and reused nowhere else, restricted in `authorized_keys` to only be accepted from inside the tailnet's own address space: ``` from="100.64.0.0/10" ssh-ed25519 AAAA... coder-agent@thinkcentre-pve ``` That [`from=` restriction](https://man.openbsd.org/sshd_config#AuthorizedKeysFile) means the key is useless from anywhere except inside my own tailnet, even if it somehow leaked. Password authentication is off entirely, and after a small follow-up fix (the `sed` meant to set `PermitRootLogin prohibit-password` silently didn't apply the first time, worth checking config changes actually stuck, not just that the command exited zero), root login now requires a key, full stop. One thing I explicitly decided *against*: Tailscale's own [built-in SSH feature](https://tailscale.com/kb/1193/tailscale-ssh). It's a genuinely elegant answer to key management (no keys at all, auth is just "are you on the tailnet"), but it intercepts port 22 traffic and demands its own browser-based check by default, which fought with the traditional key setup I'd already built. Disabled it with `tailscale set --ssh=false` so plain `sshd` handles the connection instead. ## The Reboot Test None of this counts if it doesn't survive a restart. Forced a full reboot and polled for reconnection: ```bash for i in $(seq 1 20); do ssh -o ConnectTimeout=5 pve-thinkcentre "echo back online" 2>/dev/null && break sleep 10 done ``` Back online in under a minute, `pveproxy`/`pvedaemon`/`pvestatd` all active, Tailscale reconnected on its own, SSH working with zero manual steps on either end. That's the bar for "durable enough to stop babysitting." ## Home Assistant OS Take Two Sort Of With the plumbing solid, building the actual VM was almost anticlimactic, `qm create` with 4 vCPU, 6GB RAM, a 32GB disk, UEFI without pre-enrolled Secure Boot keys (HAOS's bootloader isn't Microsoft-signed, so this has to be explicit per [Home Assistant's own Proxmox install guide](https://www.home-assistant.io/installation/alternative)), and the latest HAOS 18.2 qcow2 image imported straight in. ![Proxmox boot console showing the VM starting up](/images/smart-home-dumb-luck-episode-2/proxmox-boot-menu-haos-installer.png) It came up, grabbed a DHCP lease, and started serving its onboarding wizard within a couple of minutes. ![Home Assistant onboarding screen with analytics toggles all off](/images/smart-home-dumb-luck-episode-2/home-assistant-onboarding-analytics-toggles.png) ## Discovery Found the Wrong House Here's the tricky part: I'm not actually at my permanent home right now. I'm traveling, which is the entire reason [Episode 1](/posts/smart-home-dumb-luck-episode-1-proxmox-with-no-usb-drive) turned into a USB-drive scavenger hunt. But as expected, Home Assistant's discovery step found a pile of devices for the location I'm at. ![Home Assistant discovered devices screen showing Nest, Sonos, Elgato, and more](/images/smart-home-dumb-luck-episode-2/home-assistant-discovered-devices.png) It's whatever's broadcasting on the network wherever I'm currently staying. The right move was to click through without configuring any of it. I'm with family, so I have full permission to tinker. Heck, I'll probably be doing this for them soon. But because HA's discovery listeners run continuously in the background rather than as a one-time step, there's nothing to "re-run" once the box is actually on my home network later. That did surface a real gotcha worth writing down before it's forgotten: the static IP, gateway, and DNS reservation I configured for Proxmox itself are scoped to *this* network, not my home's. Moving the box later without redoing that networking will likely leave it with no connectivity at all, and unlike Tailscale, that specific fix can't be done remotely if it breaks, it needs local console access. Added to the plan doc as an explicit move-day task rather than trusting future-me to remember. ![Proxmox web UI dashboard for the running node](/images/smart-home-dumb-luck-episode-2/proxmox-web-ui-dashboard.png) Clicking through past the discovered devices lands on Home Assistant's actual dashboard, empty of any real integrations, which is exactly the point tonight. ![Home Assistant's dashboard on first look, no integrations configured yet](/images/smart-home-dumb-luck-episode-2/home-assistant-dashboard-first-look.png) ## Two Quick Wins While the Dongle Is in Transit Rather than sit idle, I knocked out the two pieces of the plan that don't depend on Zigbee hardware or being physically at home: [Uptime Kuma](https://github.com/louislam/uptime-kuma) and [AdGuard Home](https://github.com/AdguardTeam/AdGuardHome), both as lightweight Debian 13 LXCs with Docker (Uptime Kuma) or the official installer script (AdGuard). ![Uptime Kuma dashboard after initial setup](/images/smart-home-dumb-luck-episode-2/uptime-kuma-dashboard.png) AdGuard's setup wizard flags, correctly, that a DNS server needs a stable address to be reliable long-term. Since I'm not repointing my actual router's DNS settings at it yet (that's a permanent-home task, not a travel task), I skipped the static IP step for now rather than adding more move-day cleanup to an already-growing list. ![AdGuard Home admin interface setup screen](/images/smart-home-dumb-luck-episode-2/adguard-home-admin-interface-setup.png) ![AdGuard Home warning about needing a static IP address](/images/smart-home-dumb-luck-episode-2/adguard-home-static-ip-warning.png) To actually test it without touching the whole network, I pointed just my MacBook and iPhone at it manually (Wi-Fi settings → Configure DNS → Manual → AdGuard's IP), a per-device override that's trivially reversible and doesn't affect anything else on the network. ![AdGuard Home dashboard before any real traffic](/images/smart-home-dumb-luck-episode-2/adguard-home-dashboard-empty.png) ## One Page Load 53 Blocked Requests Then I refreshed [Tom's Hardware](https://www.tomshardware.com) once, for fun, to see what would happen. ![AdGuard Home dashboard showing 164 queries and 53 blocked from one page load of Tom's Hardware](/images/smart-home-dumb-luck-episode-2/adguard-home-toms-hardware-blocked-requests.png) 164 DNS queries, 53 of them blocked, a 32% block rate from a single article page load. That's not a malfunction, it's genuinely typical for a heavily ad-monetized publisher. The top blocked domains tell the actual story: `ib.anycast.adnxs.com` is Xandr/AppNexus, one of the largest real-time ad-bidding exchanges. `cdn.prod.euid.eu` is [European Unified ID](https://euid.eu), a cookie-less cross-site tracking system built specifically so advertisers can follow you across publishers without traditional cookies. `bordeaux.futurecdn.net` is worth a caveat, Future plc owns Tom's Hardware, so that one might be legitimate first-party infrastructure caught by an overly broad filter list rather than a tracker, a good reminder that "blocked" doesn't automatically mean "malicious." Seeing that number cold, with zero configuration beyond the default filter lists, is a better argument for local-first infrastructure than anything I could have written in the abstract. ## What's Next Episode 3 picks up the pieces I deliberately deferred this session: installing Mosquitto, ESPHome, and Node-RED as HAOS Supervisor add-ons, wiring Uptime Kuma to actually watch HAOS/AdGuard/the Proxmox host instead of sitting idle, and then the parts that genuinely require being at the permanent home with the Sonoff Dongle Plus MG24 in hand: pairing Zigbee devices, building the Wake-on-LAN button flow, and finally decommissioning SmartThings for good. ## By the Numbers - **1** dedicated SSH key, restricted to a single CIDR range, reused nowhere else - **2** Tailscale networking modes used in one session (kernel-level on the ThinkCentre, userspace on the containerized workspace) - **1** silently-failed `sed` command caught and fixed on a second pass - **~40 seconds** for the ThinkCentre to fully recover and reconnect after a forced reboot - **2** new LXCs stood up (Uptime Kuma, AdGuard Home) - **164** DNS queries from a single page load of one tech news article - **53** of those blocked, a 32% hit rate, on default filter lists alone - **0** Zigbee devices touched, still waiting on hardware and a plane ticket home === ## Friday Fixes: Cleaning Up Messy Tags and Expired PATs - URL: https://vibescoder.dev/posts/friday-fixes-cleaning-up-messy-tags-and-expired-pats - Date: 2026-08-14 - Tags: #meta #building-in-public #debugging - Reading time: 8 min read A full census of every tag on 82 posts found 47 of them, a third used exactly once, fixed with a scripted consolidation down to 21. Then a three-week streak of silent GitHub Actions failures traced back to one expired token nobody noticed rotate out. --- Tags are supposed to be the browse surface on this blog. The thing that lets you find every post about local models, or every homelab thermal saga, without scrolling the whole archive. Like most people, I tend to neglect tags. I think we audited them once. So, tags got added post-by-post, in the moment. Whatever the agent felt right for that one article, with no shared taxonomy to check against. This week I finally ran another census, and the answer surprised me: 47 unique tags across 82 posts, and a third of them used exactly once. That's not a healthy taxonomy. That's noise. ## What the Audit Actually Found A full scan of every post's frontmatter turned up real drift, not just a long tail of genuinely unique topics: - **Singular/plural duplicates.** `benchmark` (15 posts) sitting right next to `benchmarks` (1 post). `llm` (20 posts) next to `llms` (1 post). Same tag, split in two by a typo nobody caught. - **A tag that broke its own convention.** Every tag on this blog is kebab-case — except one: `business strategy`, with a literal space, the single outlier in the entire corpus. - **A frontmatter field re-encoded as a tag.** Posts already carry a `type: opinion` or `type: how-to` field. Somewhere along the way, `opinion` and `how-to` also became tags — and where they were used, they matched the `type` field 100% of the time. But that only happened on 8 of 75 eligible posts. Not a deliberate pattern. An abandoned one. - **Inconsistent model-name granularity.** Gemma and Qwen had their own dedicated tags. Kimi, DeepSeek, GLM, Hermes, Nemotron, and Fable — covered just as much, sometimes more, in post titles alone — did not. - **23 tags used exactly once**, several of them clear splinters of a broader existing category rather than genuinely standalone topics: `water-cooling`, `home-automation`, `gaming`, `reverse-engineering`, `apis`, `rss`, `ai-native`, `startups`, `saas`. None of this is a crisis. It's the ordinary entropy of tagging things one post at a time for months without ever stepping back. But a 47-tag taxonomy where a third of the tags are singletons isn't giving readers more precision. It's making the tag cloud useless for actually finding related posts. ## Fixing It in Three Passes **Pass one: typos and format, first.** Two files, three tags, zero semantic change — `benchmarks` folded into `benchmark`, `llms` into `llm`, and `business strategy` renamed to `business-strategy`. Get the mechanical stuff out of the way before touching anything that requires a judgment call. **Pass two: a real census, scripted rather than eyeballed.** A small Python + PyYAML script parsed every post's frontmatter, counted every tag, checked for case collisions and in-post duplicates (found none of either), and cross-referenced the `opinion`/`how-to` tags against each post's `type` field to confirm they were fully redundant before proposing anything got removed. Guessing which tags are safe to merge from memory is exactly how a taxonomy drifts in the first place — the whole point of this pass was to not repeat that mistake while fixing it. **Pass three: consolidation, each merge checked against real co-occurrence data before it got applied.** Every low-frequency tag got folded into a broader home, verified so nothing lost meaning in the process: - `opinion`, `how-to`, `github`, `qwen` — dropped, already fully redundant with the `type` field or a broader existing tag - `water-cooling`, `home-automation`, `gaming`, `windows`, `linux`, `hardware` — folded into `homelab` - `ai-inference`, `gemma` — folded into `benchmark` - `reverse-engineering`, `apis`, `devops` — folded into `debugging` - `rss`, `substack` — folded into `syndication` - `ai-native`, `cloud-native`, `open-source` — folded into `future-of-coding` - `startups`, `saas` — folded into `business-strategy` - `aeo` and `seo` were the one deliberate exception — kept separate below the cutoff, because the one post that carries both is specifically arguing they're distinct disciplines, not a stray pair of near-duplicates. A script applied every merge in place, rewriting only each post's `tags:` block and preserving flow-style versus block-style YAML per file, so nothing else in any post churned. Every already-published post that changed got a dated `changelog` entry, per the repo's own convention. The verification step is the part I actually trust here: for every "fold tag X into tag Y" merge, I checked whether Y was already present on 100% of the posts tagged X *before* applying it. `homelab` is the clean proof — it absorbed seven merged tags and its post count didn't move at all, 32 before and 32 after. Zero information added, zero lost. Where the target tag wasn't already universal — `debugging`, `future-of-coding`, `meta` — the merge intentionally added that broader tag to the handful of posts that didn't have it yet. That's a deliberate improvement, not a side effect I'm waving away. ## The Git Collision in the Middle of It Mid-consolidation, someone pushed live edits through the admin UI to a post that was also getting retagged in the same pass — `workflows-as-code.mdx`, a rewritten opening anecdote and a retitled section, landing while the taxonomy script was still running. `git push` correctly rejected the result as a non-fast-forward push. `git fetch` plus `git rebase origin/main` replayed the tag-only change cleanly on top of the content edits, zero conflicts, neither set of changes clobbering the other. That's the whole reason this blog runs on git instead of a database with a "last write wins" save button. Two independent changes to the same file, landing minutes apart, and the worst outcome was a rebase instead of a silently lost edit. ## What Sticks The consolidation itself is a one-time fix. The thing meant to actually stick is smaller: the content repo didn't have its own `AGENTS.md` before this pass. It does now, documenting the tag taxonomy explicitly so the next session, mine or an agent's, doesn't have to re-derive the rule from scratch, or worse, quietly reintroduce the exact same 47-tag drift a year from now. ## Three Weeks of Silent Failures on the Substack Mirror Now it's time to fix something that's bugged me for three weeks. This message stared me down from my inbox 55 times, and I ignored it every single time. ![GitHub email notification: Sync Substack Mirror, All jobs have failed](/images/friday-fixes-cleaning-up-messy-tags-and-expired-pats/sync-substack-mirror-all-jobs-failed-email.png) A second, unrelated bug surfaced the same week: the recurring "Sync Substack Mirror: All jobs have failed" email above, landing after every single post I published or edited. `gh run list` told the real story immediately: every run of that workflow had failed since 2026-07-24, 55 in a row, going back three weeks. The `Clone mirror repo` step always succeeded, which is exactly what hid the problem, the mirror repo is public, so an anonymous read works even with a broken credential. The actual failure was always one step later, at `Push to mirror`: ``` remote: Invalid username or token. Password authentication is not supported for Git operations. fatal: Authentication failed for 'https://github.com/carryologist/vibescoder-syndicate.git/' ``` **The `MIRROR_REPO_TOKEN` secret had gone bad.** Bisecting the run history narrowed it to a specific three-hour window, the last successful run at 2026-07-23 23:42 UTC, the first failure at 2026-07-24 02:38 UTC, same error from that first failure onward. A clean cutover at a random hour of the night reads like an expiring token, not a manual revocation. Nothing in the workflow itself had changed in that window either, so the credential was the only thing left to blame. The fix needed a human step I couldn't script: generate a new personal access token scoped to write access on the mirror repo, and update the `MIRROR_REPO_TOKEN` secret with it. Once that was done, I triggered the workflow manually with `gh workflow run` and watched it run clean end to end, clone, detect, push, all green, for the first time in three weeks. ## By the Numbers - **82** posts audited, **0** parse errors, **0** in-post duplicate tags, **0** case collisions found - **47** unique tags on first census, **45** after the typo/format fixes, **21** after full consolidation — a **55% reduction** - **24** tags retired, folded into **7** broader categories or dropped outright where already fully redundant - **31** post files retagged in the consolidation pass - **3** typo/format fixes applied first, across **2** files, before the broader consolidation began - **32 → 32** — `homelab`'s post count before and after absorbing 7 merged tags, the control number proving zero information was lost in that group - **11** posts gained a broader tag they didn't have before, as a deliberate side effect of a merge - **2** tags kept deliberately below the low-frequency cutoff by explicit choice: `aeo` and `seo`, both at frequency 2 - **1** non-fast-forward push, rejected and cleanly resolved with a rebase, zero conflicts, zero lost edits - **55** consecutive Sync Substack Mirror failures, one per published or edited post, over 3 weeks - **3 hours** the window between the last successful sync and the first failure, consistent with an expiring token rather than a manual revocation - **1** secret rotated to fix it, and **1** manual workflow run to confirm it === ## Smart Home, Dumb Luck, Episode 1: Firing SmartThings and Booting Proxmox With No USB Drive in the House - URL: https://vibescoder.dev/posts/smart-home-dumb-luck-episode-1-proxmox-with-no-usb-drive - Date: 2026-08-13 - Tags: #homelab #home-automation #proxmox #self-hosted - Reading time: 10 min read Kicking off a series on replacing a Samsung SmartThings hub with a self-hosted Home Assistant box. Episode 1: picking the hardware, fighting a Lenovo BIOS setting hidden behind a decoy, and the increasingly absurd path to actually getting an OS onto the machine when there wasn't a single working USB drive anywhere in reach. --- New series. The goal is simple and, it turns out, not easy to execute: replace my Samsung SmartThings hub with a self-hosted Home Assistant setup. As with all things on this journey, I want it to be hardware I control, independent of Samsung's cloud and Samsung's timeline for deciding what my house is allowed to do. Episode 1 doesn't even get to Home Assistant. It's just the story of getting an operating system onto the box that will run it. I did not expect that part alone to take a detour through a dead card reader, a camera that refused a memory card on principle, and my Steam Deck pretending to be a CD-ROM. ## The Goal SmartThings has been fine. But "fine" increasingly means a hub I don't control, automations that depend on Samsung's cloud staying up, and zero visibility into why something didn't fire when it didn't fire. It's also in the critical path of [waking my homelab up remotely with our wake on LAN trick](/posts/qol-with-wol-turning-on-the-homelab-from-anywhere). I want the automation brain in my house to be something I own outright: local-first, inspectable, and not one outage-notice email away from me finding a new hub. I researched this thoroughly and settled on a specific architecture after flip-flopping through three different stances in one sitting: [Proxmox VE](https://www.proxmox.com/en/proxmox-virtual-environment) as the hypervisor, Home Assistant OS running inside a VM (to keep its Supervisor and add-on store), and everything else — Tailscale, monitoring, DNS — as lightweight Proxmox LXCs alongside it. I'd briefly talked myself into a simpler Docker-only box before talking myself back out of it. I need a solution [AI agents can SSH into and iterate on the config over time](/posts/from-idea-to-infrastructure-standing-up-a-self-hosted-ai-dev-environment). Proxmox's snapshot-before-you-break-it safety net is worth the extra layer once change is frequent instead of rare. ## Choosing the Brains The hardware needed to be quiet, low-power, and boring in the best sense — always-on infrastructure, not [a science project](/posts/ai-nt-no-problem-cramming-a-9950x3d-and-rtx-5090-into-an-sff-custom-loop). A Lenovo ThinkCentre M720q Tiny showed up: an i5-8400T, 16GB of RAM, a 256GB NVMe drive, small enough to disappear behind a monitor. It's also, as it turns out, a genuine community staple for exactly this job — routers, DNS boxes, and Home Assistant hosts built on this same hardware class are common enough that the upgrade path (dual SODIMM slots, a socketed CPU, an M.2 slot) is well documented. Make no mistake: I will not be dual-booting and gaming on this puppy. Just a gamer's version of a Raspberry Pi. It arrived with no operating system. That's where this episode actually starts. ## A BIOS Setting Hiding Behind a Decoy First order of business: Secure Boot off, virtualization on. The BIOS had a "Device Guard" toggle that looked like the obvious switch, but flipping it does something sneaky — it silently re-enables Secure Boot as a side effect, undoing the setting I'd just made. The actual toggle I wanted, "Virtualization" with separate entries for Intel VT-x and VT-d, was sitting one menu deeper, next to Device Guard rather than inside it, a distinction [Lenovo's own support documentation](https://pcsupport.lenovo.com/us/en/products/desktops-and-all-in-ones/thinkcentre-m-series-desktops/m720q/solutions/ht500006-how-to-enable-virtualization-technology-on-lenovo-computers) doesn't make obvious either. Easy to miss, easy to fix once you know it's there. [Proxmox VE 9.2](https://www.proxmox.com/en/downloads) downloaded clean. Then came the fiasco. ## No USB Drive No Problem or so I Thought If you're a reader of this blog, you know I'm often remote. I travel a lot. For this episode I'm no where near my home. Thus, no where near my usual drawer full of "one day I'll need these" cables, USB drives, and dongles. So here I am, not only away from keyboard on a phone but away from home all together. I didn't have a working USB thumb drive anywhere in this location. Not "misplaced," genuinely zero. What followed was a tour through every alternative I could think of, most of which failed for reasons that had nothing to do with each other: - **PXE network boot**, using my Mac as the server. Mac doesn't ship a native tool for this, so that meant standing up [iVentoy](https://www.iventoy.com/en/index.html) inside a Colima VM with bridged networking — a setup nobody on the internet has actually confirmed works, and it didn't for me either. - **A USB card reader**, dead on arrival — confirmed dead by testing it against three completely different computers with a known-good memory card. - **A DJI Osmo Pocket 3**, on the theory that any device with removable storage and a USB-C port might double as a card reader. [DJI's firmware validates a card's filesystem](https://support.dji.com/help/content?customId=01700006808) before doing anything with it at all, including exposing it over USB, and a bootable image doesn't look like camera footage. No dice. - **An iPhone**, ruled out on basic principle: iOS has no USB Mass Storage mode, jailbroken or not, and a PC's BIOS can't speak the proprietary protocol Apple uses instead. ## The Steam Deck Saboteur Then Savior My Steam Deck ended up starring in both the worst part of this ordeal and the part that finally worked. It cost me hours as a would-be PXE server before it saved the entire episode as something much weirder: a fake USB drive. ### Round One Trying to Make It a PXE Server SteamOS is Arch under the hood, with a real Linux kernel and no VM layer in the way, which made it feel like the obvious upgrade over fighting Colima on the Mac. The plan: run iVentoy directly on the Deck, wire the ThinkCentre to it over a USB-Ethernet adapter, and let it PXE boot straight off the Deck. Getting there needed unlocking SteamOS's normally read-only filesystem just to install ordinary tools: ```bash sudo steamos-readonly disable sudo pacman-key --init sudo pacman-key --populate archlinux holo sudo pacman -Sy archlinux-keyring sudo pacman -S tcpdump ethtool ``` Then it was one dead end after another, each plausible enough to chase for a while before turning out to be wrong: - The static IP I assigned to the USB-Ethernet interface kept vanishing. Turned out SteamOS's NetworkManager was silently reclaiming the interface and stripping it. Fixed with `nmcli device set managed no` before reassigning the address. - `tcpdump` confirmed the ThinkCentre's DHCP discover packets were genuinely arriving at the Deck — but nothing ever answered back. - Client MAC filtering looked like a suspect (iVentoy has an allow/deny list), so I explicitly allow-listed the ThinkCentre's MAC. No change. - iVentoy's "Secondary DHCP" setting looked like a suspect too — it's meant for networks that already have a DHCP server and only want iVentoy to supplement it. Flipped it off. No change. - Checked for an outbound firewall eating the replies with `sudo iptables -L OUTPUT -n -v` and `sudo nft list ruleset`. Both came back clean. The actual bug, when I finally found it, was almost insulting: buried in iVentoy's own log was a line reading `Bind Socket(192.168.100.1) to Device enp4s0f3utc2` — one character off from the adapter's real name, `enp4s0f3u1c2`. Somewhere in iVentoy's saved config, `u1c2` had become `utc2`. I renamed the actual kernel interface to match the typo rather than fight iVentoy's own settings UI: ```bash sudo ip link set enp4s0f3u1c2 down sudo ip link set enp4s0f3u1c2 name enp4s0f3utc2 sudo ip link set enp4s0f3utc2 up ``` The log confirmed a clean bind. The DHCP requests still went unanswered. Hours in, I called it: something below the layer any of these tools could see, most likely a driver quirk in how this specific USB Ethernet chipset handles the raw hand-crafted packets a PXE server has to send back to a client that doesn't have an IP yet, was eating the replies. No amount of `ss`, `tcpdump`, or config-file archaeology was going to fix a problem at that layer. ### Round Two Turning It into a USB Drive Instead The fix wasn't a fix at all — it was giving up on PXE and using the Deck completely differently. Its USB-C port isn't just a host port; it's a genuine dual-role port on the same DWC3 controller that lets phones act as USB peripherals. Flip one setting in the Deck's own BIOS (hold Volume Up + Power to get there): `Setup Utility > Advanced > USB Configuration > USB Dual-Role Device`, from `XHCI` to `DRD`. Back in SteamOS, [Linux's USB gadget framework](https://docs.kernel.org/usb/gadget_configfs.html) can emulate a mass-storage device by hand through `configfs`, backed directly by a file: ```bash sudo modprobe libcomposite cd /sys/kernel/config/usb_gadget/ sudo mkdir -p g1 && cd g1 echo 0x1d6b | sudo tee idVendor echo 0x0104 | sudo tee idProduct sudo mkdir -p strings/0x409 echo "SteamDeck" | sudo tee strings/0x409/manufacturer sudo mkdir -p configs/c.1/strings/0x409 sudo mkdir -p functions/mass_storage.usb0 echo 1 | sudo tee functions/mass_storage.usb0/lun.0/ro echo 0 | sudo tee functions/mass_storage.usb0/lun.0/cdrom echo /home/deck/iventoy/iventoy-1.0.39/iso/proxmox-ve_9.2-1.iso | sudo tee functions/mass_storage.usb0/lun.0/file sudo ln -s functions/mass_storage.usb0 configs/c.1/ ls /sys/class/udc/ echo dwc3.1.auto | sudo tee UDC ``` That last `echo` is the moment the Deck stopped being a handheld and became, as far as anything plugged into it was concerned, an external drive containing exactly one file: a bootable Proxmox installer, read-only so nothing on the other end could corrupt it. Plugged into the Mac first as a sanity check, it mounted as a volume labeled `PVE`. Moved the same cable to the ThinkCentre, hit F12, and there was a normal USB storage boot entry, functioning exactly like a flashed flash drive, except the "drive" was a game console that had spent the previous three hours failing to be a PXE server. ## Proxmox Is Up From there it was almost anticlimactic: pick the NVMe as the target disk, set a static IP on the home network (with a note to self to reserve it in the Google Home app afterward, since Nest Wifi doesn't let you carve out a real DHCP range), swap the enterprise APT repo for the no-subscription one, disable the Ceph repo entirely since a single-node homelab box has no use for it, and reboot into a working Proxmox web UI. One more small trap on the way: Proxmox 9's repo files moved to [Debian's newer deb822 `.sources` format](https://wiki.debian.org/SourcesList#Format), so the old one-line `sed` trick for disabling the enterprise repo just returns "no such file." Worth knowing before you go looking for a file that quietly doesn't exist anymore. ## What's Next [Episode 2](/posts/smart-home-dumb-luck-episode-2-agent-keys-and-a-scary-dns-demo) picks up where the actual plan begins: hardening the box and wiring up remote access so an AI agent can administer it directly, standing up the Home Assistant OS VM, and knocking out the Uptime Kuma and AdGuard Home LXCs, all while the Sonoff Dongle Plus MG24 is still in transit and SmartThings stays live in parallel until every device has migrated. ## By the Numbers - **0** — working USB drives, SD readers, or card readers in the house at the start of this episode - **5** — distinct workarounds attempted before one actually worked - **1** — camera (DJI Osmo Pocket 3) that refused to cooperate on principle - **1** — smartphone (iPhone) ruled out on protocol grounds alone, no testing required - **1** — single mistyped character in a config file responsible for an entire dead-end PXE server - **3** — completely different computers used to confirm one USB card reader was, in fact, dead - **1** — Steam Deck temporarily demoted from "handheld gaming PC" to "USB flash drive" - **256GB** — the NVMe that eventually got Proxmox on it, the boring way, after all of that === ## Fable 5 vs Opus 5 vs Sonnet 5: A Security Code Audit Only Two Complete - URL: https://vibescoder.dev/posts/fable-5-vs-opus-5-vs-sonnet-5-a-security-code-audit-only-two-complete - Date: 2026-08-12 - Tags: #model-showdown #benchmark #security #agents #homelab - Reading time: 9 min read Three models independently audited the same pinned code in isolated workspaces, blind to each other. Two produced reports. Opus 5 found the bug that actually mattered. Fable 5 was blocked twice, at the same spot. --- Sonnet 5 is my daily driver. I use it for creative, coding, and agentic work (outside the homelab). I also scan my site periodically for vulnerabilities and bugs. I’ve used higher end models like Opus historically. So it was natural to ask: Has Sonnet gotten good enough for even specialized tasks like security audits? Short answer: No. Opus 5 way outperformed Sonnet 5. And Fable 5 was… too good? As we do on this site, we devised an experiment to find that answer. This site is powered by the blog engine, the content repo, and the Terraform template that provisions Coder workspaces, including the one I'm writing this from. Usually that's a single model, a single pass, done. This time I wanted to see what three different frontier models would each independently find in the exact same code, with zero awareness of each other or of the comparison itself. It turned into the most useful audit I've run, and the least well-behaved one. ## Three Models One Blind Audit A blind setup avoids the failure mode where one model's findings anchor the next one's. I built three isolated Coder workspaces. Each one ran a single model, Sonnet 5, Opus 5, or Fable 5, with its own fresh clone of the blog engine and the templates repo, pinned to the exact same commit in both. No shared filesystem. No shared chat history. No model knew the other two existed. I used one identical prompt for all three sessions. I framed it as a routine audit, not a comparison, so no model would hedge or perform for a benchmark it didn't know it was in. Each session worked autonomously, asked no questions, and wrote its findings to a report file with a random name I'd assigned in advance. I knew the three filenames going in. I didn't learn which model produced which report until after I'd graded every finding against the real code myself. The full prompt is below if you want to run this yourself. That blind grading step mattered more than I expected. ## Fable 5 Never Finishes Two of the three sessions produced a report. Fable 5 never finished. Not once. Not on the second try either. I re-ran it in a fresh workspace to rule out a fluke. Both times, the session split into three sub-agents, one on the app's auth surface, one on client code and dependencies, one on the infra repo. Both times, the sub-agent auditing authentication, middleware, and rate-limiting got blocked outright by Anthropic's own content-safety classifier, flagged under its "cyber" policy category. This wasn't a refusal in the model's own voice. It was an upstream block, and it landed after the sub-agent had already read several files deep into exactly the code a security audit needs to cover. Two failures in the identical subject area is not noise. Here's my read. Describing a real, specific auth or rate-limit weakness reads to an automated classifier as attack guidance, no matter how defensive the framing is. That's a genuine, ironic finding on its own. The task most worth automating is the one most likely to trip the safety net. I logged Fable 5 as a DNF and moved on instead of burning a third identical attempt on a coin flip. ## Opus 5 Finds the Bug That Matters The two completed reports differed wildly in depth. This is exactly the result a blind setup surfaces. | | Sonnet 5 | Opus 5 | |---|---:|---:| | Total findings | 11 | 50 | | High severity | 1 | 6 | | Findings in the blog engine | 5 | 30 | | Findings in the infra repo | 6 | 20 | I didn't just count findings. I went back to the pinned commit and verified a sample of claims from both reports against the real code. Both reports were accurate in everything I checked. Nobody hallucinated a vulnerability. The gap was depth, not correctness. Opus 5 dug into the one place that actually mattered. My login rate limiter keys its bucket on the left-most entry of the `X-Forwarded-For` header. A client controls that value completely. Send a random value on every request and you get a fresh five-attempt bucket every time. That gives an attacker unlimited online brute force against my admin password, the single credential that gates write access to my content repo, my Dev.to publishing, and the button that spins up a real, billable coding agent. Opus 5 found it. Sonnet 5 didn't. Opus 5 also caught that the same limiter fails open and returns "allowed" whenever Redis is unreachable or unconfigured. That removes the only brake on that endpoint a second, independent way. Opus 5 also did something I didn't ask for but appreciated. It tested one of its own theories, a possible path-traversal bug in image handling, against the live GitHub API. The theory didn't hold, so Opus 5 downgraded its own finding instead of reporting the scarier, unverified version. Sonnet 5 had one genuinely unique catch. My own workspace template tells every coding agent it has Docker available. It doesn't. Small, true, and I'm fixing it anyway. ## Opus 5 Fixes 48 of Its Own 50 Findings Grading who found the most bugs is fun, but fixing them was always the goal. I pointed Opus 5, the round's clear winner, back at its own report and told it to remediate everything, in priority order, across both repos, in feature branches, ending in a PR rather than a direct push to main. One exception. The infra repo's Terraform template provisions live workspaces, including the one doing the fixing, so that PR got a manual review and a manual apply from me, not an automatic push. It fixed 48 of the 50 findings, merged as two PRs across the two repos. Two didn't get the report's literal suggested fix. One PAT-storage finding needed a different mitigation once the constraints became clear; DPAPI encryption doesn't actually help when both a SYSTEM account and an interactive user need to read the same file. The other, an unpinned dependency bump, would have forced a large unrelated rewrite, so it got pinned via an override instead, and `npm audit` still went to zero. Opus 5 also caught six new bugs in its own remediation branch during a self-review pass before I ever looked, including one that would have silently broken page hydration site-wide on the next minor Next.js upgrade. The blog engine's fixes are live. I checked the deployed code directly before publishing this post. The infra repo's fixes are merged but not yet applied, Terraform apply, a Windows host, and a GPU host restart are real physical steps, not code, and I haven't done them yet. Fixed on paper and fixed in practice are two different states, and I'm only calling the first one done. ## The Prompt I pasted this identical text into all three isolated sessions. Only the output filename changed per session. ``` You are performing a routine periodic security and bug audit of two repositories checked out in ~/audit/: the-vibe-coder (a Next.js app) and coder-templates (Terraform + Docker workspace template + infra scripts). Audit exactly what's checked out at the current commit in each; do not switch branches, pull, or fetch updates. This is a real audit whose findings will be triaged and acted on, so be precise and avoid speculative padding. Scope: both repositories in full, except node_modules/, .next/, and other build/vendor output. Look for: - Security vulnerabilities: injection (SQL/command/template), auth and session handling flaws, authorization/access-control gaps, SSRF, secrets or tokens committed or logged, insecure direct object references, unsafe deserialization, XSS, path traversal, dependency vulnerabilities in package.json/package-lock.json, insecure Terraform/Docker defaults (e.g. exposed sockets, secrets baked into images, overly broad permissions), insecure defaults, timing attacks, rate-limit bypasses, CSRF. - Correctness bugs: logic errors, race conditions, unhandled error paths, data-loss risks, off-by-one and edge-case handling. - Do not report style/lint nitpicks or purely subjective architecture opinions unless they have a concrete correctness or security consequence. Rules: - Read-only audit. Do not modify, fix, or commit any code in either repo. - Work autonomously. Do not ask me clarifying questions; make reasonable assumptions and note them in the report if relevant. - Do not use tools that would leave a footprint outside these local checkouts (no gh pr create, no pushes, no external state changes). Read-only web/doc lookups (e.g. checking a CVE database or library docs) are fine. - When you finish, write a single report to ~/audit/ with this structure and nothing else outside it: # Security & Bug Audit ## Executive Summary (2-4 sentences: overall risk posture, most important finding) ## Findings ### [SEVERITY: Critical|High|Medium|Low] - **Repo:** the-vibe-coder | coder-templates - **Location:** path/to/file.ts:line - **Category:** e.g. Auth, Injection, Secrets, Logic, Dependency, Infra - **Description:** what's wrong - **Impact:** what an attacker/user-facing failure looks like - **Suggested fix:** concrete, short (repeat per finding, ordered by severity) ## Assumptions / Caveats (anything you weren't able to verify, or assumed) Stop once the report is written. Do not start fixing issues. ``` ## The Reports Unedited This is raw output. The only edit is adding model names after the reveal. Expand any section to read the whole thing.
Sonnet 5's full report (11 findings) ```markdown # Security & Bug Audit ## Executive Summary Overall risk posture is moderate: no critical, actively-exploitable vulnerabilities were found, but both repos have real gaps worth fixing. The most important finding is in `the-vibe-coder`, where outdated Next.js and MCP-related dependencies carry multiple high-severity CVEs (SSRF, DoS, endpoint disclosure) reachable through a publicly exposed `/api/mcp/[transport]` endpoint. A close second is a prompt-injection risk where untrusted Slack-submitted backlog text is later fed verbatim into an autonomous coding agent with real commit/PR/deploy capability. `coder-templates` is a personal/homelab template repo; its issues are mostly infra hardening gaps (unauthenticated LLM services bound to all interfaces, a static long-lived GitHub token) rather than externally exploitable flaws. ## Findings ### [SEVERITY: High] Outdated Next.js and transitive MCP dependencies with known high-severity CVEs - **Repo:** the-vibe-coder - **Location:** package.json:16 (`"next": "^16.2.6"`), package.json:19 (`"mcp-handler": "^1.1.0"`), package-lock.json (transitive: `hono`, `@hono/node-server`, `body-parser`, `fast-uri`, `ip-address`, `js-yaml`, `nanoid`, `postcss`, `sharp`) - **Category:** Dependency - **Description:** `npm audit` against the installed lockfile reports 12 advisories (7 high) against the resolved versions in this repo, including SSRF in Server Actions/rewrites, unauthenticated disclosure of internal Server Function endpoints, DoS in Server Actions and the Image Optimization API, and cache-confusion of response bodies in `next`. Separately, the MCP endpoint's dependency chain (`mcp-handler` → `@modelcontextprotocol/sdk` → `@hono/node-server` → `hono`/`body-parser`/`fast-uri`) pulls in a Hono CORS ReDoS, a Hono algorithmic-complexity DoS in its language middleware, a `body-parser` size-limit bypass, and `fast-uri` host-confusion parsing bugs. `/api/mcp/[transport]` is publicly reachable (exempted from the admin cookie in `src/middleware.ts:42`) and its request parsing/CORS handling may run before or independent of the bearer-token check in `withMcpAuth`, so these are internet-facing, not just admin-facing. - **Impact:** A remote, unauthenticated attacker could trigger CPU-exhaustion DoS against the CORS/language middleware paths, exploit `fast-uri` host-confusion for trust-boundary bypass, or hit Next.js's own SSRF/DoS/endpoint-disclosure issues. - **Suggested fix:** Run `npm audit fix` / bump `next` and `mcp-handler` (and their pinned transitive deps) to the patched versions; re-run `npm audit` to confirm zero high-severity findings before deploying. ### [SEVERITY: Medium] Untrusted backlog text flows into an autonomous coding-agent prompt with real commit/PR/deploy capability - **Repo:** the-vibe-coder - **Location:** src/app/api/todo/launch-agent/route.ts:22-27, src/app/api/slack/todo/route.ts:139-149 - **Category:** Logic / Injection (prompt injection, supply-chain) - **Description:** `/api/slack/todo` accepts free-form text from any Slack user in the configured workspace (verified only via HMAC signature, not by user identity) and inserts it verbatim as a new bullet in `content/TODO.md` via `insertTodoItem`/`parseCommand`. The admin's "Launch Agent" button (`src/app/api/todo/launch-agent/route.ts`) later takes that exact, unsanitized item text and interpolates it directly into a prompt sent to the Coder Agents Chats API, with no filtering of the item text for embedded instructions before it reaches the agent prompt. - **Impact:** Anyone able to post the Slack slash command (workspace membership, not admin identity, is the only gate) can craft a backlog item containing prompt-injection instructions that an admin later triggers via "Launch Agent," causing a fully-capable autonomous coding agent with real repo write/PR/deploy access to act on attacker-controlled instructions. - **Suggested fix:** Treat backlog item text as untrusted content in the agent prompt: wrap it in explicit delimiters with an instruction that it is data, not instructions, restrict the agent's default scope/repos in the launch payload, and/or require the admin to review and explicitly confirm the literal task text before dispatch rather than trusting whatever is currently in `TODO.md`. ### [SEVERITY: Medium] Unauthenticated LLM inference API/UI bound to all network interfaces - **Repo:** coder-templates - **Location:** scripts/llama-generate.service:8, scripts/llama-embed.service:8-19, scripts/llama-generate-start.sh:9 (`HOST=0.0.0.0`) - **Category:** Infra / Access control - **Description:** Both the generation server (port 8080) and embedding server (port 8084) are started with `--host 0.0.0.0`, and neither passes `--api-key` (or any other auth flag) to `llama-server`. The generation service additionally omits `--no-webui`, so llama.cpp's built-in web chat UI is also exposed. No firewall rule scoping these ports is present anywhere in the repo (only the SSH port-22 firewall rule is created in `setup-openssh-server.ps1`, on the unrelated Windows side of the machine). - **Impact:** Any device on the same LAN/Wi-Fi (not just the intended Tailscale mesh) can query the model, use compute for free, cause a GPU-bound denial of service against the workstation, or interact with a full unauthenticated chat UI. Since these run on the Linux host directly (not sandboxed in a container), this is a real host-level exposure. - **Suggested fix:** Bind to `127.0.0.1` or the Tailscale interface IP only, and/or set `--api-key` with a token pulled from a protected secret; add `--no-webui` to the generation server unless the UI is intentionally desired; add an explicit firewall/ufw rule denying external access to 8080/8084. ### [SEVERITY: Medium] GITHUB_TOKEN env var defeats the external-auth refresh design for `gh` and other GH_TOKEN-aware tools - **Repo:** coder-templates - **Location:** docker/main.tf:181-187 (agent `env` block) combined with lines 63-66 (`gh auth login --with-token`) - **Category:** Auth / Logic - **Description:** The template goes to considerable effort (see the comment block at lines 47-51) to make GitHub auth "work in ALL shell contexts" by having the git credential helper call `coder external-auth access-token github` fresh on every invocation. However, `GITHUB_TOKEN`/`GH_TOKEN` are also set as static values in the `coder_agent.main.env` block, which become fixed container environment variables for the container's entire lifetime (captured once at agent/container start). Because `gh` gives precedence to these env vars over stored credentials, the `gh auth login --with-token` call at startup is effectively cosmetic: `gh` will keep using the frozen startup-time token rather than any refreshed credential. - **Impact:** Once the underlying OAuth token expires (typically hours), `gh` and any other GITHUB_TOKEN-aware tool (npm packages, scripts, Vercel CLI, etc.) inside a long-running workspace will start failing with stale/expired-credential errors, even though plain `git` operations keep working via the credential helper. This is confusing and contradicts the documented intent ("works in ALL shell contexts"). - **Suggested fix:** Don't set `GITHUB_TOKEN`/`GH_TOKEN` as static agent env vars; instead export them lazily per-shell (as already done for `~/.profile`) or wrap `gh` in a shell function/alias that fetches a fresh token each call, consistent with the git credential helper approach. ### [SEVERITY: Low] MCP bearer-token comparison leaks token length via early-return timing - **Repo:** the-vibe-coder - **Location:** src/lib/mcp-auth.ts:8-15 - **Category:** Auth (timing side channel) - **Description:** `timingSafeEqual` in `mcp-auth.ts` returns immediately on `a.length !== b.length` before doing any constant-time work, so a request with a token of the wrong length returns faster than one with the correct length. The codebase already recognizes and fixes this exact pattern elsewhere: `src/lib/auth.ts:63-78` explicitly hashes both inputs first specifically to keep the comparison constant-time, noting that an early length-mismatch return would leak the password length via timing. - **Impact:** A remote attacker probing `/api/mcp/*` can use timing to incrementally determine the length of `MCP_API_TOKEN`, narrowing the brute-force search space (impact is limited in practice by network jitter and the token still requiring full-value brute force, but this is the same class of bug the repo's own `auth.ts` fix explicitly calls out and remediates). - **Suggested fix:** Apply the same fix used in `src/lib/auth.ts`: hash both the supplied token and `MCP_API_TOKEN` (e.g., SHA-256) before calling a constant-time comparison, or pad/compare fixed-length buffers without an early length check. ### [SEVERITY: Low] Public post lookups build filesystem paths from the raw slug without the shared sanitizer - **Repo:** the-vibe-coder - **Location:** src/lib/posts.ts:104-133 (`_getPostBySlug`, `getPostBySlugAdmin`), used by src/app/posts/[slug]/page.tsx:65, src/app/posts/[slug]/raw/route.ts:28, src/app/admin/preview/[slug]/page.tsx:30, src/app/admin/edit/[slug]/page.tsx - **Category:** Logic / Path handling (defense-in-depth gap) - **Description:** Every path that talks to the GitHub Contents API (posts, images, settings, TODO, MCP tools) is routed through `sanitizeSlug`/`isValidImageRepoPath`/`isValidSlug`, per the comment in `src/lib/slug.ts` explaining this was added specifically because an unsanitized slug once reached a repo path. `posts.ts`'s filesystem-backed lookups (`_getPostBySlug`, `getPostBySlugAdmin`, `_getAllPosts`'s per-file logic) are the one remaining place that builds a path (`path.join(POSTS_DIR, \`${slug}.mdx\`)`) directly from the route param with no such validation. - **Impact:** If the `slug` route param can ever contain path-traversal sequences (e.g., via an encoded `/` decoded by the framework into an actual path separator), this reads arbitrary `.mdx` files from the filesystem outside `content/posts`, constrained only by the file needing a literal `.mdx` extension and being reachable relative to `process.cwd()`. This is a real inconsistency with the rest of the codebase's own defensive posture even where current framework behavior may not be directly exploitable. - **Suggested fix:** Route `slug` through `sanitizeSlug` (or an equivalent single-segment allowlist check) in `posts.ts` before building any filesystem path, matching the pattern already used for every GitHub-backed route. ### [SEVERITY: Low] GitHub Actions workflow has no explicit `permissions` block - **Repo:** the-vibe-coder - **Location:** .github/workflows/giscus-notify.yml:1-9 - **Category:** Infra / CI - **Description:** The `giscus-notify` workflow does not set `permissions: {}` at the workflow level or a scoped `permissions:` under the job, so the job's `GITHUB_TOKEN` receives whatever default permissions are configured at the repository/org level rather than an explicit minimal grant. - **Impact:** If the org/repo default token permissions are ever broader than "read," this workflow (which never actually reads/writes repo contents via the API) would run with unnecessarily broad `GITHUB_TOKEN` privileges, widening the blast radius if the workflow or a future edit to it is ever compromised. - **Suggested fix:** Add `permissions: {}` at the workflow's top level (the job needs no GitHub API access at all, only the Slack webhook secret). ### [SEVERITY: Low] Documented "Docker" capability does not exist in the workspace image - **Repo:** coder-templates - **Location:** docs/system-instructions.md:8 vs docker/build/Dockerfile (no docker install) and docker/main.tf (no `docker.sock` mount, no privileged flag) - **Category:** Logic / Correctness - **Description:** The agent system prompt tells every coding agent running in the workspace it has Docker available as a tool, but the Dockerfile never installs the Docker CLI/daemon, and `main.tf`'s `docker_container` resource mounts no `docker.sock` and grants no extra capabilities. - **Impact:** Agents following the system prompt will attempt to use Docker, fail, and may try workarounds (e.g., installing docker-in-docker inside an unprivileged container, or requesting privilege escalation), wasting time and potentially prompting risky "fix it yourself" behavior per the same instructions file's proactive-agent policy. - **Suggested fix:** Either remove "Docker" from the documented capability list, or actually provision it (e.g., mount the host socket deliberately, understanding the security tradeoff of doing so). ### [SEVERITY: Low] GitHub OAuth token materialized as a persistent, broadly-readable container secret - **Repo:** coder-templates - **Location:** docker/main.tf:184-185 - **Category:** Secrets - **Description:** `data.coder_external_auth.github.access_token` is placed directly into the agent's `env` block, which becomes a plaintext environment variable for the container's `PID 1` and every child process, and is also recorded in the Terraform apply's data (state/logs on the Coder control plane), for the lifetime of the workspace. - **Impact:** Any process or tool running in the container (not just intended git/gh usage) can read a live GitHub token with whatever external-auth scopes are configured, widening the blast radius of any RCE or malicious dependency executed inside the workspace, compared to the on-demand `coder external-auth access-token github` call already used by the credential helper. - **Suggested fix:** Prefer fetching the token on demand (as the credential helper does) rather than baking it into the process environment for the container's whole lifetime; if some tools require the env var, consider scoping/rotating it or documenting the exposure as an accepted risk. ### [SEVERITY: Low] `administrators_authorized_keys` entry is appended non-idempotently - **Repo:** coder-templates - **Location:** windows-aint-no-problem/setup/setup-openssh-server.ps1:50-53 - **Category:** Logic - **Description:** `Add-Content -Path $authKeysPath -Value $pubKey -Force` unconditionally appends the public key without first checking whether it's already present. - **Impact:** Re-running this one-time bootstrap script (e.g. after a reinstall or to fix another step) duplicates the key entry; harmless functionally today, but it's not safe to re-run as the script's own docstring implies ("One-time bootstrap... Run once"), and could mask an actual key-replacement need if the key is ever rotated. - **Suggested fix:** Check for an existing matching entry (e.g. via a regex match against the file) before appending, or rewrite the file to a de-duplicated set. ### [SEVERITY: Low] Personal identifying details and internal infrastructure specifics committed to a "template" repo - **Repo:** coder-templates - **Location:** docker/main.tf:163-178 (base64-encoded per-user skills referencing real usernames, real Vercel app URLs, private repo names) and scripts/* (real usernames, hardware specs) - **Category:** Information disclosure (informational) - **Description:** The Terraform template embeds real GitHub usernames, private repository names, personal app URLs, and detailed home-lab hardware/network specs directly in what is documented as a reusable Coder Registry template. - **Impact:** Low as a vulnerability per se (no secrets/tokens were found embedded), but if this repository's visibility were ever changed to public, or shared as a generic template, it would leak the operator's personal infrastructure topology and third-party service endpoints, which is useful reconnaissance for a targeted attack. - **Suggested fix:** Keep personal/user-specific customization in a separate, clearly-marked private overlay rather than mixed into the general-purpose template file, if this repo's visibility could ever change. ## Assumptions / Caveats - `the-vibe-coder` was reviewed at its checked-out detached HEAD (`99c08f6`); `coder-templates` at detached HEAD (`49718b4`). No branch switches, fetches, or writes were performed in either repo. - `npm audit` for `the-vibe-coder` was run against the already-installed `node_modules`/lockfile in that workspace; results reflect resolved versions at audit time and should be re-verified at fix time. - Exploitability of the Next.js/`fast-uri`/Hono advisories and the slug path-traversal item was assessed from advisory descriptions and code inspection, not live exploitation against a running instance (out of scope for a read-only audit). - The Slack `/todo` → Launch Agent trust-boundary judgment assumes "any member of the configured Slack workspace" is a weaker trust boundary than "the site admin"; if Slack workspace membership is already restricted to fully-trusted operators, actual risk is lower than rated. - `coder-templates` is treated as a personal/homelab repository rather than a multi-tenant SaaS product; several findings (static GITHUB_TOKEN, personal usernames) are flagged for completeness even though their severity would be higher in a shared/enterprise deployment. - Actual LAN/router-level firewalling for the 0.0.0.0-bound llama.cpp services could not be verified; that finding is based solely on what is provisioned in-repo, where no compensating control (bind restriction, API key, or firewall rule) is present. - `scripts/benchmarks/round5/fixtures/{express-bug-app,flask-pagination-app}` in `coder-templates` contain deliberately-planted bugs and fake hostnames used as LLM benchmark test fixtures; these were excluded from findings as intentional test data, not production code. - No dependency-manifest files (package.json/requirements.txt/go.mod) exist in the audited scope of `coder-templates`, so no CVE/dependency-vulnerability scan was possible beyond the pinned versions visible in its Dockerfile, which were not found to have known critical CVEs as pinned. - No build/lint/test commands were executed as part of either audit (read-only scope); no runtime/dynamic testing was performed. ```
Opus 5's full report (50 findings) ```markdown # Security & Bug Audit ## Executive Summary Both repositories are single-operator projects whose security posture depends on a small number of choke points, and several of those choke points fail open. The most important finding is that the-vibe-coder's admin login brute-force protection can be bypassed outright: the rate limiter keys on the attacker-controlled left-most `X-Forwarded-For` entry and returns "allow" whenever Redis is unconfigured or erroring, leaving `ADMIN_PASSWORD` as the only barrier to full write access on the content repo, Dev.to publishing, and billable agent workspace creation. Secondary themes are dependency exposure (13 npm advisories, including a middleware-bypass class in the pinned `next` 16.2.10, which is exactly the boundary this app's auth relies on), unauthenticated/unpinned infrastructure in coder-templates (llama.cpp servers on `0.0.0.0`, `curl | bash` installs as root), and a cluster of GitHub read-modify-write races and fail-open orphan detection that risk silent content loss. No committed secrets were found in either repository. ## Findings ### [SEVERITY: High] Login rate limiter is bypassable via a spoofed X-Forwarded-For header - **Repo:** the-vibe-coder - **Location:** src/lib/rate-limit.ts:83-92 (`clientIp`), consumed at src/app/api/auth/login/route.ts:43-48 - **Category:** Auth / Rate-limit bypass - **Description:** `clientIp` takes the *first* (left-most) entry of `X-Forwarded-For`, which is the portion a client sets freely; only the right-most hop added by the proxy is trustworthy. Every limiter key (`ratelimit:login:*`, `ratelimit:analytics:*`, `ratelimit:share-image:*`, `ratelimit:mcp:*`) is therefore attacker-partitionable. - **Impact:** An attacker sends a random `X-Forwarded-For` per request and gets a fresh 5-attempt bucket each time, giving unlimited online brute force against `ADMIN_PASSWORD` (the sole credential for repo write, Dev.to publishing, and `/api/todo/launch-agent`). Also permits unbounded Redis writes from `/api/analytics/track` and unbounded OG-image render cost from `/api/share-image`. - **Suggested fix:** Use the platform-provided client IP (`x-vercel-forwarded-for` / `x-real-ip` on Vercel) or the right-most XFF hop; never trust the left-most entry. ### [SEVERITY: High] Rate limiter fails open when Redis is unconfigured or erroring - **Repo:** the-vibe-coder - **Location:** src/lib/rate-limit.ts:44-47 and :70-73 - **Category:** Auth - **Description:** `rateLimit` returns `{ ok: true }` when `KV_REST_API_URL`/`KV_REST_API_TOKEN` are unset and again in the `catch` on any Redis error (including transient Upstash 429s). The login route has no secondary throttle behind it. - **Impact:** A Redis outage, a missing env var in a preview/self-hosted deployment, or induced Upstash throttling silently removes the only brute-force control on `/api/auth/login`. - **Suggested fix:** Fail closed for the login key specifically (503 rather than allow), or add a per-instance in-memory backstop counter used when Redis is unavailable. ### [SEVERITY: High] Vulnerable `next` version and 12 other npm advisories in the lockfile - **Repo:** the-vibe-coder - **Location:** package.json:29, package-lock.json (`node_modules/next` = 16.2.10) - **Category:** Dependency - **Description:** `npm audit --package-lock-only` reports 13 vulnerabilities (8 high, 4 moderate, 1 low). The locked `next` 16.2.10 falls inside the vulnerable range `9.3.4-canary.0 - 16.3.0-preview.10`, covering App Router middleware/proxy bypass (GHSA-6gpp-xcg3-4w24), SSRF via rewrite destinations (GHSA-p9j2-gv94-2wf4), cache confusion (GHSA-68g3-v927-f742, GHSA-4633-3j49-mh5q), and Server Function endpoint disclosure (GHSA-955p-x3mx-jcvp). Also vulnerable: `js-yaml` 4.3.0/3.15.0 via `gray-matter` (GHSA-5p4m-2wfm-xmqj), `postcss` 8.4.31 nested under `next`, `sharp` 0.34.5 (GHSA-f88m-g3jw-g9cj), `@hono/node-server` <2.0.5 via `mcp-handler` (serve-static path traversal), `ip-address` 10.2.0 (SSRF), plus `brace-expansion`, `fast-uri`, `nanoid` 3.3.11, `body-parser`, `hono`. - **Impact:** A middleware-bypass advisory is directly load-bearing here, since `src/middleware.ts` is the *only* authorization check for 14 privileged API routes (see the Medium finding below). The remainder are DoS and cache/SSRF exposure. - **Suggested fix:** `npm audit fix` and redeploy; the `mcp-handler` remediation is a major bump to 2.1.0, so exercise `/api/mcp/[transport]` afterwards. ### [SEVERITY: High] LLM inference servers bound to 0.0.0.0 with no authentication - **Repo:** coder-templates - **Location:** scripts/llama-generate-start.sh:9 (`HOST=0.0.0.0`, used lines 28-86), scripts/llama-embed.service:18-19 - **Category:** Infra - **Description:** Both llama.cpp servers listen on all interfaces (8080 generation, 8084 embedding) with no `--api-key` and no authenticating reverse proxy. The generation service also omits `--no-webui` (the embed unit sets it at line 17), so the browser UI is exposed too. Per docs/sff-migration-checklist.md:188-207 the host runs Tailscale and a Cloudflare tunnel, so "LAN only" is not a safe assumption. - **Impact:** Any host that can reach the machine can consume the GPU, run arbitrary prompts, and read `/props` (model paths, sampling config, chat template) without credentials. - **Suggested fix:** Set `HOST=127.0.0.1` and `--host 127.0.0.1` in the embed unit, or add `--api-key`; add `--no-webui` to the generation service. ### [SEVERITY: High] Unpinned `curl | bash` installs run as root during workspace image build - **Repo:** coder-templates - **Location:** docker/build/Dockerfile:1, :22, :43, :46 - **Category:** Infra / Supply chain - **Description:** `FROM codercom/enterprise-base:ubuntu` is a floating tag with no digest; `curl -fsSL https://deb.nodesource.com/setup_20.x | bash -`, `curl -LsSf https://astral.sh/uv/install.sh | sh`, and `npm install -g vercel` all execute unverified remote content as `USER root` (line 3) with no checksum or version pin. - **Impact:** A compromised or MITM'd upstream response yields root code execution at build time and a backdoored image for every workspace of every user. - **Suggested fix:** Pin the base image by digest, download installers to a file and verify a checksum before executing, and pin `vercel` and the NodeSource setup script to explicit versions. ### [SEVERITY: High] Benchmark harness executes model-generated code on the host with no sandbox - **Repo:** coder-templates - **Location:** scripts/benchmarks/round5/benchmark.py:273-285, :446-454, :472-484 (also :25-27) - **Category:** Infra / Arbitrary code execution - **Description:** Model output is written to `todo.py` / a `.ts` file and executed via `subprocess.run([sys.executable, app_path] + args)` and `npx --yes tsx`. The only containment is a `TemporaryDirectory` and a 10-30s timeout; the process runs as the invoking user with full filesystem and network access. Separately, lines 25-27 silently run `pip install requests --break-system-packages` on ImportError, mutating the system Python. - **Impact:** A hallucinated or adversarial generation (`rm -rf ~`, credential exfiltration, outbound HTTP) executes with the operator's privileges on the workstation that also hosts Coder, Docker, and Tailscale. - **Suggested fix:** Execute fixtures in a disposable container (`docker run --rm --network none --read-only`) or a `bwrap`/`nsjail` sandbox; make `requests` a documented requirement instead of auto-installing. ### [SEVERITY: Medium] All privileged API authorization lives in middleware only - **Repo:** the-vibe-coder - **Location:** src/middleware.ts:47-73; handlers under src/app/api/{posts,images,settings,generate-post,syndicate,todo} - **Category:** Auth - **Description:** Fourteen privileged handlers perform no in-handler session check; the comment at src/app/api/todo/launch-agent/route.ts:30-31 documents this as deliberate. `src/middleware.ts:38` and `:40` also allow `/api/auth/**` and `/api/slack/**` wholesale by prefix, so any future route added under those paths is unauthenticated by default. - **Impact:** A single mistake in `config.matcher`, a Next.js middleware-bypass advisory (the pinned version is affected, see above), or an invocation path that skips middleware yields unauthenticated repo write and delete, Dev.to publishing, and billable workspace creation. - **Suggested fix:** Add `if (!(await getSession())) return 401` at the top of each privileged handler; src/app/api/auth/check/route.ts:14 already shows the one-line pattern. ### [SEVERITY: Medium] Stored XSS in the admin TODO inline-Markdown renderer - **Repo:** the-vibe-coder - **Location:** src/lib/todo.ts:135-137 and :146-151; sink at src/components/admin/TodoReorderList.tsx:157 - **Category:** XSS - **Description:** `escapeHtml` escapes `&`, `<`, `>` but not `"`, and the link rule interpolates the captured URL into a double-quoted `href` with the permissive class `[^\s)]+`. A crafted `TODO.md` bullet with an unescaped quote in the link URL breaks out of the `href` attribute, letting an attacker-controlled event handler attribute get injected. The CSP at next.config.ts:52 includes `script-src 'unsafe-inline'`, so inline handlers are not blocked. (Working payload omitted as a precaution.) - **Impact:** Script execution in the authenticated admin's browser on `/admin/todo`, in a session that can write to the content repo. `TODO.md` is also written by the Slack command and by agents, so this is reachable without a direct human commit. - **Suggested fix:** Escape `"` and `'` in `escapeHtml`, and tighten the URL class to `(https?:\/\/[^\s)"'<>]+)`. ### [SEVERITY: Medium] `javascript:` URLs pass through the MDX anchor component - **Repo:** the-vibe-coder - **Location:** src/components/MDXComponents.tsx:61-87 - **Category:** XSS - **Description:** `const isExternal = href.startsWith("http")` routes everything else to ``, including `javascript:` and `data:text/html,`. Post bodies are generated by Claude from transcripts (src/lib/claude.ts) and committed by the admin UI; no stage validates link schemes. - **Impact:** Script execution in every reader's browser on a published post. `'unsafe-inline'` in the CSP does not restrict `javascript:` navigations. - **Suggested fix:** Parse the href and allow only `http(s):`, `mailto:`, and site-relative (`/`, `#`) values; render the text without a link otherwise. ### [SEVERITY: Medium] Fail-open orphan detection can mark in-use images as deletable - **Repo:** the-vibe-coder - **Location:** src/lib/images.ts:117-131, :190-201, :230, :252; UI at src/components/admin/ImageManager.tsx:263-283 - **Category:** Logic / Data loss - **Description:** `loadStaticImageReferences()` and `safePostIndex()` swallow all errors and return an empty `Set`/`[]`. A null match then sets `orphaned: true`. If `public/static-image-refs.json` is absent (prebuild not run, stripped deploy) or `content/posts` is missing, every branding asset and post image is presented under "Orphaned" with "Nothing references this file" and a one-click "Delete all". The file's own comment at lines 96-100 records that exactly this class of file was deleted once before as a false orphan. - **Impact:** Irreversible deletion of in-use assets on the content repo's `main` branch. - **Suggested fix:** Return `null` on read failure to distinguish "manifest missing" from "manifest empty", and suppress orphan flagging (or disable the delete buttons) when either signal is unavailable. ### [SEVERITY: Medium] `POST /api/posts` silently overwrites an existing post - **Repo:** the-vibe-coder - **Location:** src/app/api/posts/route.ts:83-85; src/lib/github.ts:32-51 - **Category:** Logic / Data loss - **Description:** The create path never checks for an existing file; `commitFile` fetches the current SHA and upserts. The MCP `create_post` tool does check and returns `post_exists` (src/app/api/mcp/[transport]/route.ts:310-325), so the inconsistency is confirmed. Because `sanitizeSlug` collapses input, `"My Post!"`, `"my/post"`, and `"my--post"` all normalize to `my-post`. - **Impact:** A new draft can clobber a live published post in one request; recoverable only from Git history. - **Suggested fix:** `readFile(path)` first and return 409 when it exists, mirroring the MCP tool. ### [SEVERITY: Medium] Lost-update race on every read-modify-write of repo files - **Repo:** the-vibe-coder - **Location:** src/lib/github.ts:32-51 and :85-116; callers at src/app/api/posts/route.ts:145-171, src/app/api/todo/route.ts:24-42, src/app/api/syndicate/devto/route.ts:23-71, .../bulk/route.ts:29-93, src/components/admin/DraftsList.tsx:60-171 - **Category:** Logic / Race condition - **Description:** Each flow reads content, mutates it in memory, then calls `commitFile`, which re-fetches the blob SHA at write time and therefore always wins. The SHA read at load time is never sent as a precondition. `src/lib/todo.ts:92-124` gets this right with `TodoConflictError`; the post path does not. - **Impact:** Two concurrent writers (admin UI, MCP agent, Slack `/todo`, scheduled publish) silently overwrite each other, losing post edits. - **Suggested fix:** Thread the load-time SHA through the API and let GitHub's 409 surface instead of re-reading. ### [SEVERITY: Medium] `PUT /api/settings` persists the entire unvalidated request body - **Repo:** the-vibe-coder - **Location:** src/app/api/settings/route.ts:22-51 - **Category:** Logic - **Description:** Only `stylePrompt` and `defaultTags` types are checked; `prompts` and arbitrary extra keys of any size are written verbatim to `content/settings.json`. `stylePrompt`/`prompts[*].prompt` become the system prompt at src/lib/claude.ts:39-42, and `getSettings` (src/lib/settings.ts:54) silently drops a malformed `prompts` map. - **Impact:** Unbounded file growth in the content repo and persistent system-prompt poisoning for all future generations, surfacing as a silent behavior change rather than an error. Admin-scoped, so integrity rather than escalation. - **Suggested fix:** Build the persisted object explicitly from validated fields, validate `prompts` with the existing `isPromptMap`, and cap sizes. ### [SEVERITY: Medium] `fixDateYear` throws on unquoted YAML dates, 500-ing publish and update - **Repo:** the-vibe-coder - **Location:** src/app/api/posts/route.ts:11-26 - **Category:** Logic - **Description:** `gray-matter` parses an unquoted YAML `date: 2020-01-01` into a JavaScript `Date`, which has no `.replace`. The guard checks truthiness only, never type. src/app/api/generate-post/route.ts:104-108 handles the `data.date instanceof Date` case, confirming the inconsistency. - **Impact:** Any post with an unquoted frontmatter date more than a year stale cannot be created or updated; the failure surfaces as an opaque 500. - **Suggested fix:** Normalize first: `const d = data.date instanceof Date ? data.date.toISOString().split("T")[0] : String(data.date)`. ### [SEVERITY: Medium] A failed `EXPIRE` permanently bricks a rate-limit key - **Repo:** the-vibe-coder - **Location:** src/lib/rate-limit.ts:50-56 - **Category:** Logic - **Description:** If `INCR` succeeds but `EXPIRE` throws, the `catch` at line 70 swallows it and the key persists with no TTL. `count === 1` never recurs, so the TTL is never set; once the counter passes `limit`, `ttl` returns `-1` and the branch at :59-66 blocks that key indefinitely while advertising a bogus `retryAfter`. - **Impact:** Permanent login lockout for the affected bucket with no self-healing path; requires manual Redis intervention. - **Suggested fix:** Make increment and expiry atomic (`SET key 0 EX NX` then `INCR`, a Lua script, or a pipeline). ### [SEVERITY: Medium] GitHub OAuth token exported into workspace env and persisted to disk - **Repo:** coder-templates - **Location:** docker/main.tf:184-185, :66, :301-322 - **Category:** Secrets - **Description:** `GITHUB_TOKEN`/`GH_TOKEN` are set from `data.coder_external_auth.github.access_token` in `coder_agent.env`, which writes them into Terraform state and exposes them to every process in the container via `/proc/*/environ`. `gh auth login --with-token` additionally persists the token to `~/.config/gh/hosts.yml` on the retained `docker_volume.home_volume` (`lifecycle { ignore_changes = all }`), so it survives stop and rebuild. The file already has a better mechanism: the credential helper at :53-54 and the `~/.profile` export at :60 re-fetch a fresh token per invocation. - **Impact:** A long-lived GitHub token in Terraform state and on a persistent volume, retrievable after the workspace is stopped. - **Suggested fix:** Drop `GITHUB_TOKEN`/`GH_TOKEN` from the `env` block and rely on the per-call `coder external-auth access-token` path; if `gh` needs auth, pass `GH_TOKEN` at call time. ### [SEVERITY: Medium] MCP secrets written world-readable before `chmod 600` - **Repo:** coder-templates - **Location:** docker/main.tf:87, :101, :117, :120 - **Category:** Secrets - **Description:** `.mcp.json` and both `.mcp.json.tmp` files are created with the default umask (0644) while already containing `Bearer $FITNESS_TRACKER_MCP_TOKEN` / `$VIBESCODER_MCP_TOKEN`. The `.tmp` files are never chmod'd at all before `mv`; the `chmod 600` lands only after every write. - **Impact:** A window in which bearer tokens are readable by any other UID in the container and by anything reading the mounted home volume from the host. - **Suggested fix:** `umask 077` before the block, or `install -m600 /dev/null ` first and chmod each temp file before writing. ### [SEVERITY: Medium] Unpinned external skill repo cloned and trusted on every workspace start - **Repo:** coder-templates - **Location:** docker/main.tf:139-160 - **Category:** Infra / Supply chain - **Description:** `git clone`/`git pull` of `https://github.com/carryologist/agent-skills.git` tracking `main`, with no commit pin or signature check, then every `workspace/*/` directory is symlinked into `~/.agents/skills` where the coding agent reads them as instructions. All errors are suppressed with `2>/dev/null || true`, so tampering or a failed pull is invisible. - **Impact:** Anyone who can push to that repo silently changes agent behavior in every workspace on the next start, with the workspace's GitHub token in scope. - **Suggested fix:** Pin to a verified tag or commit SHA, or vendor the skills into the image; log failures rather than discarding them. ### [SEVERITY: Medium] SSH exposed to all networks with password auth left enabled - **Repo:** coder-templates - **Location:** windows-aint-no-problem/setup/setup-openssh-server.ps1:32-38, :45-49 - **Category:** Infra / Auth - **Description:** `New-NetFirewallRule ... -LocalPort 22` is created with no `-Profile` and no `-RemoteAddress`, allowing inbound 22 from any source on every profile including Public; line 37 re-enables the rule unconditionally if it was deliberately disabled. The script never edits `sshd_config`, so `PasswordAuthentication` stays at the Windows default (`yes`) for an account the script adds to Administrators. README.md:126-135 states this is only meant to be reachable over Tailscale. - **Impact:** Password-guessable administrator SSH on any network the machine joins, including untrusted Wi-Fi. - **Suggested fix:** Scope the rule (`-Profile Private -RemoteAddress 100.64.0.0/10`) or bind `ListenAddress` to the Tailscale IP, and set `PasswordAuthentication no` + `PubkeyAuthentication yes` before restarting sshd. ### [SEVERITY: Medium] Unpinned PowerShell Gallery module installed machine-wide by a SYSTEM task - **Repo:** coder-templates - **Location:** windows-aint-no-problem/orchestrator/update-orchestrator-system.ps1:61-68; same pattern in update-orchestrator-notify.ps1:19-26 - **Category:** Infra / Supply chain - **Description:** `Install-Module -Name PSWindowsUpdate -Force -Scope AllUsers` followed by `Import-Module`, with no `-RequiredVersion`, no `-Repository`, and no signature or catalog validation; `-Force` suppresses the untrusted-repository prompt. This runs as `NT AUTHORITY\SYSTEM` (register-update-orchestrator-tasks.ps1:32) on weekly and at-startup triggers. - **Impact:** Whatever module version is current at run time is installed system-wide and loaded into a SYSTEM process. A compromised version, or a higher-priority repository registered later, is full machine compromise. - **Suggested fix:** Pre-install a pinned version and use `-Repository PSGallery -RequiredVersion ` plus signature verification; fail the step rather than installing on demand. ### [SEVERITY: Medium] Unattended auto-reboot fires on every boot - **Repo:** coder-templates - **Location:** windows-aint-no-problem/orchestrator/update-orchestrator-system.ps1:73-74; register-update-orchestrator-tasks.ps1:27, :29-30, :39 - **Category:** Logic / Data loss - **Description:** `Install-WindowsUpdate -AcceptAll -AutoReboot` runs whenever `-Unattended` is passed. The system script's own header (lines 8-11) says `-Unattended` is "only for the scheduled/overnight run", but registration passes it to a task that also fires `-AtStartup`. There is no check for an interactive logon session. - **Impact:** The machine can force a reboot moments after a user boots into Windows, discarding unsaved work. - **Suggested fix:** Register two tasks (weekly with `-Unattended`, at-startup without), or gate `-AutoReboot` on there being no interactive session. ### [SEVERITY: Medium] GitHub PAT stored in plaintext, protected only by a manual documented step - **Repo:** coder-templates - **Location:** windows-aint-no-problem/orchestrator/update-orchestrator-github-sync.ps1:33, :41; windows-aint-no-problem/README.md:94-101 - **Category:** Secrets - **Description:** A Contents:read-write PAT is written to a plaintext file on the orchestrator host and read back with `Get-Content -Raw`. No script sets or verifies the ACL; hardening is a copy-paste `icacls` block in the README that must be re-run after any rotation. The parent directory grants broad local read+execute by inheritance. (Exact path redacted as a precaution.) - **Impact:** A repo-write PAT recoverable by any local user whenever the manual step is skipped or undone. - **Suggested fix:** Store the token with DPAPI or Windows Credential Manager; have the script assert the ACL and refuse to read a world-readable file. ### [SEVERITY: Medium] Missing `permissions` block in the giscus notification workflow - **Repo:** the-vibe-coder - **Location:** .github/workflows/giscus-notify.yml:1-13 - **Category:** Infra / CI - **Description:** No `permissions:` at workflow or job level, so the job runs with the repository's default `GITHUB_TOKEN` scope (write-all where the default has not been changed), despite needing no token scopes at all. The `${{ }}` handling itself is safe (values pass through `env:` and `jq --arg`), so there is no script injection. Separately, the comment at lines 10-12 claims owner comments are skipped, but the condition only checks the category, so self-notifications still fire. - **Impact:** A write-capable token is exposed to a job that processes attacker-influenced comment payloads. - **Suggested fix:** Add `permissions: {}` at the top level and grant nothing at job level. ### [SEVERITY: Low] MCP token comparison leaks token length via early return - **Repo:** the-vibe-coder - **Location:** src/lib/mcp-auth.ts:8-14 - **Category:** Auth / Timing - **Description:** `if (a.length !== b.length) return false` precedes the constant-time XOR loop, so the comparison is constant-time only for equal-length inputs. src/lib/auth.ts:63-77 documents and fixes exactly this pattern for the admin password; `mcp-auth.ts` never received the same treatment. - **Impact:** Narrows the search space for `MCP_API_TOKEN`. Low practical exploitability over network jitter. - **Suggested fix:** Hash both inputs to a fixed 32 bytes and use `crypto.timingSafeEqual`, matching `auth.ts`. ### [SEVERITY: Low] Slack replay window is skipped when the timestamp is non-numeric - **Repo:** the-vibe-coder - **Location:** src/app/api/slack/todo/route.ts:20-21 - **Category:** Auth - **Description:** `Math.abs(now - Number(timestamp)) > 300` evaluates to `false` when `Number(timestamp)` is `NaN`, so the freshness check passes. The HMAC still covers the timestamp, so forgery is not possible; exploitation requires a captured request that already carried a non-numeric timestamp, which Slack does not send. - **Impact:** Replay protection is unenforced for a malformed-timestamp request; effectively unreachable with a legitimate Slack sender. - **Suggested fix:** `const ts = Number(timestamp); if (!Number.isFinite(ts) || Math.abs(now - ts) > 300) return false;` ### [SEVERITY: Low] Sessions cannot be revoked; logout is client-side only - **Repo:** the-vibe-coder - **Location:** src/lib/auth.ts:14-30; src/app/api/auth/logout/route.ts:4-8; src/middleware.ts:66 - **Category:** Auth - **Description:** A 7-day HS256 JWT is minted with no `jti`; `verifySession` checks only the signature, logout merely clears the cookie, and there is no denylist. Middleware calls bare `jwtVerify` and never asserts the `role: "admin"` claim it signs, so that claim is decorative. `/api/auth/logout` also has no origin check (unlike login), though `sameSite: "strict"` blocks the cross-site form post. - **Impact:** A stolen token stays valid for its full 7 days; the only remediation is rotating `SESSION_SECRET`. Forced-logout CSRF is a nuisance at most. - **Suggested fix:** Shorten the lifetime, add a `jti` with a Redis denylist on logout, assert the `role` claim in middleware, and apply the login route's origin check to logout. ### [SEVERITY: Low] Over-permissive filename validation in image delete paths - **Repo:** the-vibe-coder - **Location:** src/lib/images.ts:334-337; interpolated unencoded at src/lib/github.ts:90 - **Category:** Path traversal - **Description:** `isValidFilename` rejects only empty strings, backslashes, and leading dots, so `%2f`, `?`, `#`, and spaces are accepted and interpolated raw into the GitHub Contents API URL. Verified against the live API: `%2f` decodes to `/` (defeating the "exactly two segments" rule the surrounding comment claims to enforce) while `..`/`%2e%2e` segments return 404, so this **cannot** escape `public/images/`. An unescaped `?` or `#` can still alter or truncate the request URL (e.g. `foo.png?ref=other`). - **Impact:** The stated path constraint is not actually enforced, and request URLs are influenceable by filename. Admin session required, so no privilege gain. - **Suggested fix:** Restrict to `/^[A-Za-z0-9][A-Za-z0-9._-]*$/` and `encodeURIComponent` each path segment in `github.ts`. ### [SEVERITY: Low] `sanitizeSlug` can return an empty string, producing dotfile and double-slash paths - **Repo:** the-vibe-coder - **Location:** src/lib/slug.ts:12-18; callers at src/app/api/posts/route.ts:83,132,196,223 and src/app/api/images/route.ts:28 - **Category:** Logic - **Description:** Inputs like `"..."`, `"!!!"`, or `"---"` sanitize to `""`, so callers build `content/posts/.mdx` or `public/images//`. The character class is a strict allowlist, so there is no traversal. - **Impact:** A hidden `.mdx` file in the content repo, and image paths with a double slash that will not round-trip through `isValidImageRepoPath` on delete, leaving orphaned files that cannot be removed via the UI. - **Suggested fix:** Return an error when the sanitized slug is empty. ### [SEVERITY: Low] Unvalidated slug reaches `path.join` in the posts loader - **Repo:** the-vibe-coder - **Location:** src/lib/posts.ts:105, :137; callers at src/app/posts/[slug]/page.tsx:124, .../raw/route.ts:278, .../opengraph-image.tsx:190, src/app/admin/preview/[slug]/page.tsx:142 - **Category:** Path traversal - **Description:** `path.join(POSTS_DIR, \`${slug}.mdx\`)` receives the route param with no `sanitizeSlug` call, even though src/lib/slug.ts exists for exactly this. No read outside `content/posts` could be demonstrated (the `.mdx` suffix is forced and Next normalizes `..` in path segments), so this is a latent gap rather than an exploitable one. - **Suggested fix:** Reject non-`[a-z0-9-]` slugs in `_getPostBySlug` and `getPostBySlugAdmin`. ### [SEVERITY: Low] Arbitrary local file read via markdown image path in the OG image route - **Repo:** the-vibe-coder - **Location:** src/app/posts/[slug]/opengraph-image.tsx:216-236 - **Category:** Path traversal - **Description:** `extractFirstImage` takes the first `![alt](src)` target from the post body and passes it unvalidated to `path.join(process.cwd(), "public", rawImage)` and `fs.readFileSync`. A relative image path containing traversal segments reads an arbitrary file on the host and base64-embeds it as a `data:` URI in the generated PNG. Content is author/AI-authored and the result is not returned as text, which caps impact. (Working payload omitted as a precaution.) - **Suggested fix:** Require `^/images/[A-Za-z0-9._/-]+$` and reject any `..` segment before reading. ### [SEVERITY: Low] `POST /api/images` accepts any file type or size and commits it to `public/` - **Repo:** the-vibe-coder - **Location:** src/app/api/images/route.ts:6-38 - **Category:** XSS - **Description:** No MIME check, no extension allowlist, no size cap; `sanitizeFilename` preserves the extension, so `evil.html` is committed to `public/images//evil.html` and served same-origin under a CSP with `script-src 'unsafe-inline'`. `commitFileRaw` also upserts, silently overwriting a same-named image. - **Impact:** Stored XSS, reachable only with an existing admin session (no privilege gain), plus unbounded blobs in the content repo. - **Suggested fix:** Allowlist extensions using the existing `isImageFilename` in src/lib/image-types.ts and enforce a byte cap. ### [SEVERITY: Low] Slack text flows unescaped into TODO.md and then into an autonomous agent prompt - **Repo:** the-vibe-coder - **Location:** src/app/api/slack/todo/route.ts:136, :164, :180; consumed at src/app/api/todo/launch-agent/route.ts:22-28 - **Category:** Injection / Prompt injection - **Description:** `item` is taken verbatim from the slash-command text with newlines and `## ` headings unstripped, so a Slack user can inject structure into `TODO.md`, which `src/lib/todo.ts` then parses. The same text is interpolated into a prompt instructing an agent to clone repos, implement, commit, and open a PR. - **Impact:** An indirect prompt-injection path from any Slack workspace member to a billable coding agent with repo write access. The admin must click Launch, which is the mitigating control. - **Suggested fix:** Strip newlines and leading markdown control characters before insertion, and delimit untrusted text in `buildPrompt`. ### [SEVERITY: Low] CSP permits `'unsafe-inline'` scripts and `img-src https:` - **Repo:** the-vibe-coder - **Location:** next.config.ts:52-64 - **Category:** XSS - **Description:** The policy is enforcing, but `script-src 'self' 'unsafe-inline'` removes CSP as a mitigation for both XSS findings above, and `img-src 'self' data: https:` allows any host. The file's own comments (lines 27-29) flag this as unfinished work. - **Suggested fix:** Adopt the per-request nonce described in the comments; the only inline scripts are the theme bootstrap (layout.tsx:77) and JSON-LD (JsonLd.tsx:336), both easily nonce-able. ### [SEVERITY: Low] `GITHUB_TOKEN` embedded in a git remote URL - **Repo:** the-vibe-coder - **Location:** scripts/fetch-content.sh:23-25 - **Category:** Secrets - **Description:** The token is passed on the command line (visible via `/proc//cmdline`) and written into `$TMPDIR/.git/config`; git error output on a failed clone commonly echoes the remote URL into build logs. `$TMPDIR` is only cleaned on the success path, so `set -e` leaves the credentialed config on disk after any failure. - **Suggested fix:** Use `git -c http.extraheader=...` or a credential helper, and add `trap 'rm -rf "$TMPDIR"' EXIT`. ### [SEVERITY: Low] Internal error details returned to unauthenticated clients - **Repo:** the-vibe-coder - **Location:** src/app/api/share-image/route.tsx:438-442; src/app/api/slack/todo/route.ts:196-201; src/app/api/todo/launch-agent/route.ts:129-135 - **Category:** Information disclosure - **Description:** Raw `err.message`, GitHub API response bodies, and upstream Coder API bodies are returned to the caller or echoed into the Slack channel. No secret is exposed on these paths (tokens are only ever sent in headers). - **Suggested fix:** Log details server-side and return a generic message. ### [SEVERITY: Low] Analytics counter keys are written without a TTL - **Repo:** the-vibe-coder - **Location:** src/app/api/analytics/track/route.ts:77-85; read fan-out at src/app/api/analytics/summary/route.ts:70-75 - **Category:** Logic - **Description:** Per-day and per-path keys accumulate indefinitely and the `views:paths` set grows forever; the summary endpoint issues a pipeline `GET` per member on every call. The path allowlist correctly bounds key cardinality, so arbitrary key minting is not possible. - **Impact:** Slow unbounded Redis growth and a summary cost that grows linearly with site history. - **Suggested fix:** Set a TTL (e.g. 400 days) on dated keys and prune `views:paths`. ### [SEVERITY: Low] React key collision on duplicate TODO items - **Repo:** the-vibe-coder - **Location:** src/components/admin/TodoReorderList.tsx:107, :33-36; interacts with src/lib/todo.ts:108-124 - **Category:** Logic - **Description:** `reorderUpNext` deliberately supports duplicate item text via a text-keyed multiset, but the list uses `key={item.text}`. Two identical bullets produce duplicate React keys, so reordering either one reconciles incorrectly and can send a wrong `order` array; the `dirty` check likewise reports "no changes" when two identical items are swapped. - **Suggested fix:** Key by index or by a stable id assigned server-side. ### [SEVERITY: Low] Publish/schedule frontmatter rewrites silently no-op on unquoted values - **Repo:** the-vibe-coder - **Location:** src/components/admin/DraftsList.tsx:65-75, :120-123; src/app/admin/edit/[slug]/page.tsx:327-335 - **Category:** Logic - **Description:** `published.replace(/^date:\s*'[^']*'/m, ...)` matches single-quoted dates only. src/lib/format-date.ts:12-17 and src/lib/posts.ts:15-18 both document that this content set frequently yields unquoted dates, in which case "Publish" flips `published` but silently leaves the old date, with no error surfaced. - **Suggested fix:** Parse and serialize frontmatter with `gray-matter` instead of regex-patching, or assert the replacement changed the string. ### [SEVERITY: Low] Unpaginated, unauthenticated GitHub Discussions fetch - **Repo:** the-vibe-coder - **Location:** src/lib/discussions.ts:20-33 - **Category:** Logic - **Description:** No `per_page`/pagination and no `Authorization` header. GitHub's default page size is 30, so past 30 discussions the oldest posts silently show `0` comments; unauthenticated requests also share the 60/hour/IP limit across all serverless instances, and the failure path logs and returns `{}`. - **Suggested fix:** Paginate with `?per_page=100` plus link-header following, and send the existing `GITHUB_TOKEN`. ### [SEVERITY: Low] Unbounded GitHub API fan-out on the admin images page - **Repo:** the-vibe-coder - **Location:** src/lib/images.ts:220-236 - **Category:** Logic - **Description:** `Promise.all` issues one Contents API request per image directory with no concurrency cap. A rate-limit or slow response fails the whole `/admin/images` render with the raw GitHub error text surfaced at src/app/admin/images/page.tsx:232. - **Suggested fix:** Bound concurrency (batches of ~5) and degrade per-directory instead of failing the page. ### [SEVERITY: Low] Dockerfile build failure silently swallowed - **Repo:** coder-templates - **Location:** docker/build/Dockerfile:46-48 - **Category:** Logic - **Description:** `|| true` binds to the entire `&&` chain, not just the `uvx` move. If the `uv` installer or the first `mv` fails, the layer still exits 0 and ships an image with no `uv`, contradicting docs/system-instructions.md:5, which tells the agent `uv` is present. - **Suggested fix:** Split into a separate `RUN` and scope the tolerance to the `uvx` move alone. ### [SEVERITY: Low] llama config file is `source`d rather than parsed - **Repo:** coder-templates - **Location:** scripts/llama-generate-start.sh:16; written by scripts/llm-switch.sh:52-57 - **Category:** Injection - **Description:** `/etc/llama-generate.conf` is executed as shell. Nothing in the repo sets or asserts its mode; `llm-switch.sh` recreates it via `sudo bash -c`, so permissions depend on root's umask at that moment. The `llm-switch.sh` write itself is safe, since `${1}` is allow-listed by the `case` at lines 41-48. - **Impact:** Any write access to the config becomes code execution in the systemd service context. - **Suggested fix:** Parse the value (`sed -n 's/^DEFAULT_MODEL=//p'`) and re-validate against the allow-list. ### [SEVERITY: Low] systemd units have no sandboxing and execute a user-writable script - **Repo:** coder-templates - **Location:** scripts/llama-generate.service:8; scripts/llama-embed.service - **Category:** Infra - **Description:** Neither unit sets `NoNewPrivileges`, `ProtectSystem`, `ProtectHome`, `PrivateTmp`, or `RestrictAddressFamilies`. `llama-generate.service` executes a script from the service account's own home directory, so anything running as that user changes what the service runs on next restart. No privilege boundary is crossed, but integrity guarantees are absent. - **Suggested fix:** Move the launcher to a root-owned `/usr/local/libexec` path and add the standard hardening directives. ### [SEVERITY: Low] Orchestrator install directory never ACL-hardened - **Repo:** coder-templates - **Location:** windows-aint-no-problem/orchestrator/update-orchestrator-system.ps1:25-26; .../update-orchestrator-user.ps1:18-19; .../register-update-orchestrator-tasks.ps1:14, :27 - **Category:** Infra - **Description:** `C:\ProgramData\update-orchestrator` is created with `-Force` and no `icacls`. The SYSTEM task executes `-File` from this directory and the system script dot-invokes a sibling via `$PSScriptRoot`. Inherited `ProgramData` permissions let non-admins create subdirectories there; nothing is currently directly writable by a standard user, so this is defense in depth. - **Suggested fix:** `icacls /inheritance:r /grant 'Administrators:(OI)(CI)F' 'SYSTEM:(OI)(CI)F'`, or install under `%ProgramFiles%`. ### [SEVERITY: Low] Toast button passes unvalidated JSON data to a protocol handler - **Repo:** coder-templates - **Location:** windows-aint-no-problem/orchestrator/update-orchestrator-notify.ps1:44-45 (data parsed at :28) - **Category:** Injection - **Description:** `New-BTButton -Content 'Open Log' -Arguments $summary.LogPath -ActivationType Protocol` takes the path verbatim from `last-run-summary.json` and hands it to the shell URI dispatcher, so a tampered `LogPath` (UNC path, `ms-*:` or `file://` URI) launches on click. - **Suggested fix:** Validate that `LogPath` resolves under `C:\ProgramData\update-orchestrator` and ends in `.log`. ### [SEVERITY: Low] SSH key authorization is not idempotent and ACL results are unchecked - **Repo:** coder-templates - **Location:** windows-aint-no-problem/setup/setup-openssh-server.ps1:53, :56-57, :59 - **Category:** Logic - **Description:** `Add-Content ... -Force` appends unconditionally, duplicating the key on re-run. The two `icacls` calls pipe to `Out-Null` without checking `$LASTEXITCODE` and neither sets the file *owner*, which sshd also validates for `administrators_authorized_keys`. There is no `$ErrorActionPreference = 'Stop'`, so earlier failures do not prevent the script printing "Done". - **Suggested fix:** Guard the append with `Select-String`, set `$ErrorActionPreference = 'Stop'`, check `$LASTEXITCODE` after each `icacls`, and add `/setowner Administrators`. ### [SEVERITY: Low] Proposed CI workflow in the README violates least privilege and pins nothing - **Repo:** coder-templates - **Location:** README.md:63-84 - **Category:** Infra / CI - **Description:** The suggested `push-template.yml` has no top-level `permissions:` block, uses the mutable `actions/checkout@v4` tag rather than a commit SHA, and installs the Coder CLI via unpinned `curl -fsSL https://coder.com/install.sh | sh`, all with `CODER_SESSION_TOKEN` in the environment. - **Suggested fix:** Add `permissions: {}` at the top with per-job grants, pin the action by SHA, and pin the CLI version. ### [SEVERITY: Low] Documented backup copies a secrets file into `$HOME` with no permission handling - **Repo:** coder-templates - **Location:** docs/sff-migration-checklist.md:35 - **Category:** Secrets - **Description:** `sudo cp /etc/coder.d/coder.env ~/coder-backup/coder.env.bak` copies the Postgres DSN and any OIDC/OAuth client secrets without preserving a restrictive mode; the target directory is never mode-restricted and there is no cleanup step. - **Suggested fix:** `sudo install -m 600 -o "$USER" ...` and add a cleanup checkbox. ### [SEVERITY: Low] Shared mutable image tag across all workspaces - **Repo:** coder-templates - **Location:** docker/main.tf:291-299 - **Category:** Infra - **Description:** Every workspace build targets `coder-workspace:latest` with `triggers = { dockerfile_hash = ... }`. Two concurrent builds race on the tag, and a rebuild silently re-points `:latest` while other workspaces reference it. - **Suggested fix:** Tag with the Dockerfile hash, e.g. `coder-workspace:${filemd5("./build/Dockerfile")}`. ### [SEVERITY: Low] `.gitignore` omits common secret patterns - **Repo:** coder-templates - **Location:** .gitignore:1-19 - **Category:** Secrets - **Description:** Terraform state and `*.tfvars` are covered, but a repo described in README.md:1-8 as the home for personal infra and homelab config does not ignore `.env`, `*.pem`, `id_ed25519`, `*.key`, `*.log`, or `github-token.txt`. Nothing is currently leaked. - **Suggested fix:** Add those patterns. ## Assumptions / Caveats - Audited exactly the checked-out commits: the-vibe-coder at `99c08f6`, coder-templates at `49718b4`. No branch switch, pull, or fetch was performed, and no file in either repo was modified. - `node_modules/`, `.next/`, and `scripts/benchmarks/round5/results/` were excluded from line-by-line review. The results directory was scanned for secret patterns (clean); it contains model output and metrics only. - `scripts/benchmarks/round5/fixtures/**` are deliberately-broken benchmark inputs, verified from explicit `// BUG:` markers in the fixtures and from the prompts in benchmark.py:506-548. Their planted defects (hardcoded `admin123`, unsigned base64 "tokens", `debug=True`) are intentional and are **not** reported as findings. - Dependency findings come from `npm audit --package-lock-only` against the committed lockfile. Advisory ranges shift over time; re-run before triage. No runtime install or upgrade was performed. - The percent-encoded traversal theory for `isValidImageRepoPath` was tested against the live GitHub Contents API and **disproved** (`..`/`%2e%2e` segments 404), which is why that finding is rated Low rather than High. The residual `?`/`#` URL-manipulation issue stands. - Findings in the two vibe-coder XSS entries assume `content/TODO.md` and post MDX can be influenced by a non-admin (Slack workspace members and AI-generated content respectively). Both are admin-rendered, so a purely single-trusted-author threat model would downgrade them. - The Windows orchestrator scripts were reviewed statically only; no Windows host was available to confirm effective ACLs, scheduled-task registration behavior, or `sshd_config` defaults in situ. - `.env*` is gitignored in both repos and a full-tree scan for `ghp_`/`github_pat_`/`sk-`/`xox*`/`AKIA`/PEM headers returned nothing. All base64 blobs in docker/main.tf were decoded and contain no inline credentials. - No SQL, template engine, `eval`, or `child_process` usage exists in the-vibe-coder, so SQL/command/template injection and unsafe deserialization are not applicable there. Every outbound `fetch` targets a hardcoded host, so no SSRF sink was found. ```
Fable 5's DNF writeup ```markdown # Fable 5 — DNF (Anthropic content-policy block, "cyber") Status: **did not finish**, reproduced twice, retries stopped by decision on 2026-08-08. This is a documented outcome for this round, not a missing data point to chase further — do not keep re-running Fable 5 against this prompt. ## Assignment (now unblinded out of necessity) Fable 5 was Variant B in this round, assigned `report-5b855a.md`. No report file exists for this token; the session never reached the point of writing one. The other two variants have since been fully revealed: Variant A (`report-5f220e.md`) was Sonnet 5, Variant C (`report-b74a69.md`) was Opus 5 — see `comparison.md` for the full triage. ## What happened (both attempts) Both attempts self-organized the same way: the session split the audit into three parallel subagents (roughly: app auth/API surface, app client/scripts/deps, `coder-templates`). In both runs, the subagent covering **auth, middleware, rate-limiting, and MCP-auth** got its response blocked outright by Anthropic's platform-level content classifier under the "cyber" category, right as it was moving from reading code to writing up findings. The other two subagents (client/deps/scripts, `coder-templates`) were not reported as blocked in either transcript, but no full report was ever assembled since the run didn't complete. - **Attempt 1:** blocked on the "server code" subagent after it had read a couple of files into the auth/token-handling area. - **Attempt 2:** blocked on the "auth/API surface" subagent, after it had already read the middleware matcher, MCP auth, the Slack route, the share-image route, and the rate limiter — i.e. it got further into the same subject area before tripping the same block. Two-for-two on the same subject area (auth/middleware/rate-limit/MCP-auth) is a reproducible collision, not noise. Working theory: describing a concrete weakness in that code (e.g. a middleware matcher gap or a rate-limit bypass) with enough specificity to be a useful audit finding reads to Anthropic's classifier as attack guidance, independent of the benign, defensive framing in our prompt. ## Decision - Recorded as a DNF for Fable 5 in this round. No third identical attempt. - Not retried with a softened prompt in this round, to avoid conflating a "mitigated variant" result with the blind three-way comparison. If a future round wants to test whether softer finding-detail requirements (e.g. "name the weakness class and location, don't narrate a step-by-step bypass") let Fable 5 complete, that should be run and labeled as an explicit separate variant, not folded into this comparison. - Comparison writeup will treat this as its own failure category: "did not finish due to host-platform safety block," distinct from a capability-based DNF (e.g. the Nemotron Round 9 case). ```
My full blind triage and comparison notes ```markdown # Blind Triage & Comparison — Revealed: Sonnet 5 vs Opus 5 Triage performed against the actual pinned commits (`the-vibe-coder@99c08f6`, `coder-templates@49718b4`), spot-verifying a representative sample of findings from each report directly in the code (not just trusting the report text). Triage itself was done blind (reports known only as "Variant A" / "Variant C"); mapping revealed afterward: - **Variant A** (`report-5f220e.md`) = **Sonnet 5** - **Variant C** (`report-b74a69.md`) = **Opus 5** - **Variant B** (`report-5b855a.md`) = **Fable 5** — DNF, see `fable-5-dnf.md` ## Headline counts | | Sonnet 5 (`5f220e`) | Opus 5 (`b74a69`) | |---|---|---| | Total findings | 11 | 50 | | High | 1 | 6 | | Medium | 3 | 17 | | Low | 7 | 27 | | `the-vibe-coder` findings | 5 | 30 | | `coder-templates` findings | 6 | 20 | Opus 5 found roughly 4.5x as many issues as Sonnet 5, across both repos. ## Spot-verification (sample, not exhaustive) Checked ~15 claims from both reports directly against the pinned-commit source. Everything checked from **both** reports was an accurate description of the code — no fabricated findings, no misquoted logic, in the sample checked. The gap between the two reports is coverage and depth, not accuracy. Notable quality signals for **Opus 5**: - Tested its own path-traversal theory against the *live* GitHub Contents API, found it didn't hold (`%2e%2e` 404s), and correctly downgraded that finding from a plausible High to a Low rather than reporting the unverified worst case. - Caught a real logic bug beyond the security angle: the giscus workflow's own comment claims it skips the repo owner's comments, but the `if:` condition only checks the discussion category, not the author. Variant A flagged the same workflow only for its missing `permissions:` block and missed this. - Explicitly identified and excluded the deliberately-planted bugs in `scripts/benchmarks/round5/fixtures/*` as intentional test data rather than reporting them as findings, showing it understood the difference between benchmark fixtures and production code. - Precise line citations throughout (verified `rate-limit.ts:83-92`, `:44-47`, `:70-73` character-for-character against the actual function boundaries). Sonnet 5's one clearly unique catch neither report shares: `coder-templates` documents Docker as an available capability in `system-instructions.md`, but the Dockerfile never installs it and `main.tf` never mounts a socket — a real, low-severity documentation/reality mismatch. Worth keeping in the fix list regardless of which model found it. ## The single most important finding in either report Opus 5's **login rate-limiter bypass via spoofed `X-Forwarded-For`** (High): `clientIp()` in `rate-limit.ts` takes the left-most, client-supplied XFF entry, so every rate-limit bucket (login, analytics, share-image, MCP) is attacker-partitionable, giving unlimited brute force against `ADMIN_PASSWORD`. Confirmed by direct code read — this is real, exactly as described, and it is the actual sole credential gating repo write, Dev.to publishing, and agent-launch access. **Sonnet 5 did not find this at all.** Compounding it, Opus 5 also caught that the same limiter fails open on any Redis error or missing config, independently removing the control a second way. ## Overlap (found by both, same underlying issue) - Outdated `next` / MCP dependency chain CVEs (both High; Opus 5 is more precise about the actual resolved version, 16.2.10, and ties it to a concrete middleware-bypass advisory that matters because middleware is this app's *only* authorization layer — Sonnet 5 stops at "here are CVEs"). - MCP bearer-token timing leak via early-length-return in `mcp-auth.ts` (both Low, identical characterization). - Static `GITHUB_TOKEN`/`GH_TOKEN` in the agent's Terraform `env` block defeating the credential-helper refresh design (both Medium, same lines). - Unauthenticated llama.cpp servers bound to `0.0.0.0` (Sonnet 5: Medium, Opus 5: High — Opus 5's higher rating accounts for the Tailscale/Cloudflare tunnel documented elsewhere in the repo, meaning "LAN-only" isn't a safe assumption). - Missing `permissions:` block on the giscus GitHub Actions workflow (Sonnet 5: Low, Opus 5: Medium). - Slack backlog text flowing unsanitized into the launch-agent prompt (Sonnet 5: Medium, Opus 5: Low — opposite direction from the giscus severity gap; Opus 5's lower rating explicitly credits the admin's manual "Launch" click as a real gate, Sonnet 5 does not weigh that mitigation). - Unvalidated slug reaching a filesystem path join in `posts.ts` (both Low; Opus 5 additionally covers two related-but-distinct slug bugs — an empty-sanitized-slug case and an OG-image path-traversal read — that Sonnet 5 didn't find). ## Unique to Opus 5 (not in Sonnet 5) The bulk of the gap: the rate-limiter bypass and fail-open (above), unpinned `curl | bash` root installs in the Dockerfile (High, supply chain), unsandboxed benchmark code execution on the host (High), stored XSS in the admin TODO renderer and in the MDX anchor component (Medium x2), fail-open orphan-image deletion, silent post overwrite, GitHub read-modify-write races, unvalidated settings persistence, a crash on unquoted YAML dates, a permanently-bricked rate-limit key on a failed Redis `EXPIRE`, world-readable MCP secret files before `chmod`, an unpinned externally-cloned skills repo trusted every startup, and a full pass over the Windows orchestrator scripts (open SSH to all networks, unpinned PowerShell Gallery module run as SYSTEM, unattended reboot on every boot, plaintext GitHub PAT) plus roughly a dozen more Low-severity items (session revocation, CSP gaps, unrestricted image upload, pagination bugs, etc). ## Unique to Sonnet 5 (not in Opus 5) - Documented-but-absent Docker capability (above). - Slightly more detailed narrative of the MCP endpoint's transitive dependency chain (`mcp-handler` → `@hono/node-server` → `hono` / `body-parser` / `fast-uri`), though Opus 5's dependency finding lists most of the same packages without walking the chain by name. ## Assessment Both reports are accurate where they make claims — I found no false positives in the sample verified. The real difference is thoroughness and depth of investigation: Opus 5 read further into both repos, tested a hypothesis empirically instead of asserting it, distinguished intentional benchmark fixtures from real bugs, and found the one finding in this round that actually matters most (the rate-limiter bypass). Sonnet 5 produced a shorter, still-legitimate but comparatively shallow pass. ## Fable 5 Did not finish either attempt; see `fable-5-dnf.md`. No comparison data — Fable 5 is a DNF for this round, not a ranked third place. ## Verdict For this round, on this task: **Opus 5 > Sonnet 5 > Fable 5 (DNF)**, driven mostly by Opus 5 catching the one finding that actually matters (the auth rate-limiter bypass) and covering roughly 4.5x the ground. Sample verification found no accuracy gap between the two, only a depth/coverage gap. Candidate for the `model-showdown-round-*` series, with the Fable 5 platform-safety DNF as a distinct, separately-worth-mentioning angle. ```
--- *A defensive security audit and an offensive one can produce the same words. Should the safety layer even be able to tell them apart? I don't have a clean answer. But I now have a support ticket's worth of evidence that it can't, not yet.* ## By the Numbers - 3 models audited the identical pinned commit, in fully isolated workspaces, with zero cross-awareness - 2 attempts by Fable 5, both blocked at the same logical point: auth/middleware/rate-limit code - 11 findings from Sonnet 5 vs 50 from Opus 5, a 4.5x gap in depth, not accuracy - 1 High-severity finding from Sonnet 5 vs 6 from Opus 5 - 0 false positives found across everything I spot-verified from either completed report - 1 rate-limiter bypass that only one of the two models caught, and the one finding that actually mattered most - 30 / 20 — Opus 5's findings split between the blog engine and the infra repo === ## No Bench, No Rack, No Excuse: Adding a Significant Feature to the Fitness Tracker - URL: https://vibescoder.dev/posts/no-bench-no-rack-no-excuse-adding-a-significant-feature-to-the-fitness-tracker - Date: 2026-08-10 - Tags: #building-in-public #agents #mcp #security #next-js - Reading time: 9 min read A full feature build inside an existing vibe-coded app, start to finish: a plan-first conversation, two shipped phases, a progression system, two new MCP tools, and a glossary page that only exists because testing surfaced a gap the code itself never needed to close. Less about dumbbells, more about what it actually looks like to build something substantial into an app that already works. --- My [fitness tracker](/posts/wiring-mcp-into-my-fitness-tracker-for-openclaw) — closing in on a year old now — was built over several coding sessions and a few different agents. It's a simple app. Among several things, it pulls real weight-lifting data from Tonal. But Tonal is a wall-mounted machine, and plenty of the places I end up working out don't have one. So I asked my agent to code a feature around three dumbbell-only routines to rotate between, accommodating the weights most commonly found in a hotel gym, a friend's spare room, or an Airbnb: 5, 10, 15, and 20 lb pairs. Include things like a a timer, a live running tally of weight lifted, and a way to log the finished session back into the same table Tonal and Peloton already write to. We then got to planning, coding, and testing. Below is the journey. I asked the agent to summarize our session. As happens sometimes, the agent wrote this from its perspective. This always amuses me and I choose to keep but not it here. My vibes, his thoughts (I think it's a he). So, he gets credit. ## The Plan Before the Code The first version of the split was Legs / Back-and-Shoulders / Arms-and-Chest. I pushed back on that one: a Push/Pull/Legs split avoids hitting the same joint twice in a session — shoulders get worked by both presses and "chest day" moves under the original grouping. That suggestion got adopted, and the exercise list got built around one hard constraint: no bench, no rack, floor and standing movements only. The harder design problem was the weight math. Two dumbbells moving together — an overhead press, say — double the load per rep. My first pass at that logic was wrong, and we caught it with a concrete example: a bent-over row done one arm at a time, at 10 lb, is 10 lb per rep, not 20, even though there are two 10 lb dumbbells in the room. That distinction became a `load` field on every exercise — `bilateral`, `unilateral`, or `single` — and it's the one piece of this feature that had to be exactly right before anything else made sense, because it's the number that eventually gets written to the database as `weightLifted`. Everything else got scoped deliberately thin for a first pass: routines as a plain TypeScript constant, no schema changes, no editable reps yet, no persistence beyond crash-safety in `localStorage`. Progression, editable sets, and MCP exposure were explicitly called out as "phase 2, don't block it, but don't build it yet either." ## Shipping Push Pull Legs Phase 1 became a new `/freeweights` tab: pick a day, a session timer starts, a checklist of sets renders with planned reps and weight, and checking a set adds to a running tally. Finishing the workout posts one `WorkoutSession` row — `source: "Free Weights"`, `activity: "Weight Lifting"` — through the exact same `/api/workouts` endpoint Tonal and Peloton use, so it shows up in every existing chart and goal without any of that code needing to know Free Weights exists. Seeding the actual numbers meant working backward from a target: roughly 8,000 lbs for a Push or Pull day, 10,000 for Legs, "ballpark is fine." Five sets of fifteen reps across five exercises per day, with heavier compound movements (floor chest press, bent-over row, suitcase squat) getting the 20 lb dumbbells and isolation moves (lateral raise, calf raise) getting the 5s, landed within a few hundred pounds of both targets. One bug got fixed as a freebie along the way: the mobile header's sync buttons didn't reserve fixed width, so "Peloton" turning into "Syncing…" would shrink the flex row enough to wrap the "Fit Track" title onto two lines. Icon-only buttons on mobile fixed it permanently, and it got bundled into this change since the header was already getting a new tab switcher regardless. This shipped straight to `main` — "I'll test it myself, it's just a hobby app," with explicit sign-off first. ## Phase 2 Progression That Isn't Allowed to Regress The interesting design problem in phase 2 wasn't the database table — that part is genuinely small, one `FreeWeightProgress` row per exercise, an override that falls back to the code default when absent. It was translating "I want the app to challenge me to keep that as the new baseline so I don't regress" into an actual mechanic. The answer: weight stays fixed to the dumbbells that physically exist (you can't buy a 12.5 lb pair mid-trip), so only reps progress, one tap at a time in Settings. But reps can't climb forever on light weight before that stops being useful, so there's a ceiling — 20 reps per set — and hitting it while lighter than the heaviest tier owned surfaces a "Level Up" prompt: move to the next dumbbell size, restart at 8 reps. The baseline that gets displayed as each set's target during a session is always the current one, DB override merged over the code default, so there's no way to accidentally see last month's easier numbers. And at the end of a session, the summary compares actual reps completed against that baseline and calls out anything that came in short — a nudge, not a hard gate, because this is a hobby app, not a compliance system. Editable sets got built alongside it: a pencil icon per set that opens actual-reps and actual-weight fields, defaulted to the plan, for the moment you fail a rep or grab the wrong dumbbell. One tap still logs a set at planned values — the common case stayed exactly as simple as phase 1 left it. This round went through an actual feature branch and pull request instead of straight to `main`, complete with a collapsible design-notes section in the PR body — a small but real shift in process between phase 1 and phase 2 of the same feature, on the same repo, in the same day. ## Exposing It to Agents The stated broader goal was "expose more FT data for analysis with agents," and it turned out most of that was already free: Free Weights sessions log into the same `WorkoutSession` table Tonal and Peloton use, so the existing `list_workouts` MCP tool already surfaces them, no new work required. What was missing was the *programming* — the targets themselves, not the history of hitting them — so `list_freeweight_progress` and `set_freeweight_progress` joined the MCP server, following the exact pattern every other mutation tool in that file already uses: direct Prisma access, an audit-log row on write. The fitness tracker's MCP server now exposes twelve tools total; two of them didn't exist twelve hours earlier. ## A Glossary the Code Never Needed The most telling moment of the whole session came after testing started: *"What is a Floor Chest Press? I just want to read a two-sentence description of each exercise before starting."* The app had never needed to explain an exercise to anything — the code just needed a slug, a weight, and a rep count. Whoever was actually about to do the reps needed something completely different. Fifteen exercises, each with a genuine two-sentence how-to (stance, movement, nothing fancier), landed as a `/freeweights/glossary` page, deep-linked from every exercise name in the routine picker and the active session, opening in a new tab so a mid-workout lookup doesn't lose the running timer. It's a small feature by line count and a good reminder that "the data model has everything it needs" and "whoever's doing the workout knows what they're doing" are two completely separate claims. ## Auditing Dependencies without Overreacting Separately from the feature itself, a routine "should we merge to main" check turned up a failing `npm audit` CI job — 13 findings, including a critical one in the Auth.js/NextAuth dependency chain. Not every fix is worth taking blind: `npm audit fix` alone (no `--force`) cleared the critical and three of six highs by shifting only `package-lock.json` inside existing semver ranges — verified clean with a full `tsc` and `next build` before it went anywhere. The three findings that remained all trace back to the same root cause: the Next.js-reported CVEs (Server Actions DoS, SSRF, cache confusion) only have a fix at Next.js 16.3.0, a major version up from the app's current 15.x. That's not a dependency patch, it's a framework upgrade with real surface area — auth flow, MCP's streamable HTTP transport, middleware — and it got deliberately deferred to its own reviewed pass instead of getting bundled into a "fix vulnerabilities" commit that quietly changed how the whole app runs. ## What Connects Them Every real decision in this session came down to matching the mechanism to a constraint that already existed in the physical world, not inventing a new abstraction to feel clever. The load-type field exists because a dumbbell in one hand and two dumbbells in two hands are different amounts of weight, full stop. The level-up ceiling exists because you can only own so many dumbbell sizes. The baseline nudge exists because "don't regress" needs a number to compare against, not a vibe. Even the dependency audit followed the same instinct: fix what's provably safe now, and treat "upgrade the framework" as the different, larger decision it actually is, instead of letting a CI red X talk anyone into a same-day major version bump. ## By the Numbers - **3** routines (Push, Pull, Legs), **15** exercises total, 5 per day - **4** dumbbell weights to design around: 5, 10, 15, 20 lb - **3** load-type classifications (`bilateral` / `unilateral` / `single`) needed to get weight math right - **~8,250 lbs** planned volume per Push/Pull day, **~10,350 lbs** for Legs — against a "ballpark 8k/10k" target - **2** phases shipped in one session: phase 1 pushed straight to `main`, phase 2 through a feature branch and PR #22 - **5** commits total across both phases plus a dependency-audit follow-up - **1** new database table (`FreeWeightProgress`), additive only, applied via the existing `prisma db push` build step — no migration files needed - **12** total MCP tools on the fitness tracker now; **2** of them (`list_freeweight_progress`, `set_freeweight_progress`) didn't exist before this session - **15** exercise descriptions written for a glossary page that exists purely because testing, not the code, surfaced the need for it - **13 → 6** npm audit findings after a non-breaking `npm audit fix`, including clearing the one critical; **3** remaining highs deliberately deferred to a Next.js 16 upgrade instead of force-patched same-day === ## Friday Fixes: Finding Fitness Flaws - URL: https://vibescoder.dev/posts/friday-fixes-finding-fitness-flaws - Date: 2026-08-07 - Tags: #building-in-public #agents #debugging - Reading time: 9 min read One extended session on my personal fitness tracker turned up three real bugs, not one. A Tonal sync that silently died 14 hours out of every 24 because of an Auth0 token-lifetime mismatch, found by pulling live production logs instead of guessing. A years-old duplicate-workout bug that survived a prior "fix" because the matching logic was too strict, plus a self-inflicted unique-constraint bug I caught testing against production before it ever shipped. And a cleanup pass that found a second real bug hiding in plain sight, one repo review away. --- *(Editor’s note. The agent wrote this from its own first person point of view. We kept this to be transparent and illustrate the choices AI makes. Sonnet 5 was used, and it deserves the credit.)* My [fitness tracker](/posts/spring-cleaning-your-vibe-coded-apps) pulls workouts from two sources that were never meant to be pulled from: Peloton and Tonal, neither of which ships a public API. Both integrations are reverse-engineered, held together by community research and a lot of trial and error, and both have broken before. This session broke the streak of "one bug at a time." Three separate real bugs, in one sitting, none of them the one I went looking for except the first. ## 1 Tonal Sync Died 14 Hours Out of Every 24 and the App Couldn't Say Why The symptom was `tonal sync failed: Internal server error` — a banner with zero diagnostic value. The route's error handling only preserved messages from its own typed error class; everything else got flattened into that one generic string before it ever reached the browser. No Vercel access existed in this chat yet, so the first move wasn't reading code, it was `vercel login`, which produces an OAuth device-code URL. I posted the link, the human approved it in a browser, and the CLI came back authenticated a few seconds later. That unlocked the actual production logs, which said something the UI never could: ``` Tonal activities fetch failed (401): {"message":"Error parsing token: token is expired by 11h37m41s","status":401} ``` The app's own status check reported the credential as fresh. Tonal's API said otherwise. That gap was the whole bug. A read-only `psql` query against the production `tonal_credentials` row, plus decoding the stored `id_token`'s JWT payload, turned the discrepancy into exact numbers: the token's real lifetime was **36000.0 seconds** — 10 hours, Auth0's default ID Token Expiration — while the app's bookkeeping assumed **86399.6 seconds**, about 24 hours. That second number is `expires_in` from Auth0's `/oauth/token` response, which describes the *access_token's* lifetime. The app only ever uses `id_token` as the Bearer credential for Tonal's API. Nobody had separated those two numbers before, so for roughly 14 hours of every 24, the app believed the credential was fine and got a 401 anyway. **The fix**: decode the `id_token`'s own `exp` claim and use the earlier of that versus `expires_in`, applied at both the initial-auth and token-refresh call sites. Plus a safety net: if a 401 slips through anyway, force one refresh-and-retry before giving up, instead of dying silently. Deployed, and the first sync attempt right after still failed — exactly as expected, since the stored `expiresAt` predated the fix. The second attempt succeeded, self-healed by the new retry path. Confirmed live, not just in theory. ## 2 Six Duplicate Workouts Hiding Among Forty-Eight Good Ones The dashboard showed the same real-world gym session twice: once as "Tonal," once as "Peloton," on the same day, with slightly different minutes. This bug had actually been fixed once before, back in May — Peloton's Watch/app integration auto-logs a generic "Weight Lifting" row that mirrors a real Tonal session, and the May fix was supposed to skip creating that mirror. It didn't work, because the guard required an *exact* minutes match between the two trackers, and the two trackers never agreed to the minute. Real durations differed by up to 16 minutes for the same session. Before touching anything, I queried production directly to separate signal from noise: only **6 of 48** Peloton "Weight Lifting" rows overlapped with a same-day Tonal row. The other 42 — real instructor-led classes from 2020 through 2024, named coaches, actual Peloton content — had no Tonal row anywhere near them and needed to be left alone. The 6 real duplicates were all recent, all titled the same generic "Strength: Traditional Strength Training," and all missing a weight-lifted number, because Peloton's side of this pairing never records one. That last detail mattered for the design decision. My first instinct was to keep Peloton as the row of record, since that's what was asked for. But making Peloton the surviving row would have meant dropping the real weight-lifted number for those six sessions — the entire reason the Tonal integration exists. A quick clarifying question from the human ("won't that make the source in the table Peloton?") caught the tradeoff before it shipped, and the design flipped: Tonal stays canonical, keeps its own minutes and its own weight number, and just adopts the Peloton row's workout ID for traceability. There was a second question worth its own research pass: which tracker's minutes are actually more accurate? Apple Watch does not reliably auto-detect strength training in the first place — it's a documented, known limitation, since resistance-training arm motion doesn't produce a clean signature the way running or cycling does. And even wearable-based tracking that does capture a session is known to miscount duration around pause and resume gaps, which is exactly the shape of a lifting session with long rests between sets. Tonal's own coaching content says as much directly: rest periods are expected to eat a large share of total workout time, by design. Tonal's console timer runs continuously from a deliberate start-press to a deliberate end-press. That's the more trustworthy number, and the data backed it up — Tonal's minutes were higher in five of the six pairs, consistent with a passively-tracked estimate under-counting real session time. Fixing this for good meant an order-independent guard on both sides: whichever integration syncs second finds the other's same-day row and merges into it, rather than either side blindly creating a new one. Simple in concept. The implementation caught a real bug of its own. `pelotonWorkoutId` is unique across the table, and the merge logic tried to write that ID onto the surviving Tonal row before clearing it off the row being retired — a straightforward unique-constraint collision. I found this by actually running the historical cleanup against production rather than trusting the code path untested: Postgres rejected the very first pair, the whole transaction rolled back cleanly with zero partial state, and the fix was obvious once the error was in front of me. I shipped the ordering fix, tested it, and only then found a second copy of the exact same ordering mistake sitting a few lines lower in the same function — moved between two calls instead of removed. Caught that one before it ran anywhere. The historical merge ran clean on the second try: 6 pairs merged, 42 genuine Peloton classes and 72 Tonal sessions completely untouched, one soft-deleted row per pair (nothing hard-deleted, nothing lost). A temporary, audited admin endpoint did the actual merge, following the exact same one-shot pattern — build it, run it once, delete it — as a prior cleanup back in May. ## 3 the Cleanup Pass Found a Fourth Bug That Wasn't Even Being Looked For Asked directly whether anything else in the surrounding code looked wrong, a scan of the sync paths turned up a real, still-live bug: the Tonal side of a "manually-entered row" lookup was missing a `deletedAt: null` filter that the Peloton side had gotten back in May, for exactly this reason — *"so a soft-deleted unlinked row doesn't get re-linked on re-sync."* The comment documenting the fix was three months old. It had just never been applied to both integrations that needed it. Without the filter, a soft-deleted placeholder row could silently absorb a real incoming Tonal workout on some future sync instead of a fresh, visible row getting created — the workout would vanish from the dashboard with no error at all. Two smaller items came out of the same pass: `AGENTS.md` still claimed the app ran on Next.js 14, three major-version-adjacent commits after the app actually moved to 15. And the original May dedupe logic — three duplicate-detection patterns, `findDupes()` and `softDeleteWorkouts()` — had been dead code for months, unused since the one-shot admin route that called it was deleted after its single run. All three got fixed in one pass: the real bug, the stale doc, and the dead code, in that order of importance. ## What Connects Them Every bug in this session traced back to an assumption that felt obvious and wasn't verified. Auth0's `expires_in` looked like it described the credential in the code; it described half of it. A wearable auto-tracking a strength session looked like a second, independent data point; it was closer to a probabilistic estimate riding on top of the same real event Tonal already recorded accurately. My own merge fix looked correct on read; it took an actual failed transaction against actual production data to prove otherwise, twice. The fix, each time, was the same: stop reasoning from the code and go get evidence instead. Pull the real logs. Query the real database. Decode the real JWT. Run the real merge and watch it fail. None of these bugs were found by staring harder at the source; all of them were found by asking the system what was actually happening and believing the answer, even when — especially when — it contradicted what the code appeared to say. ## By the Numbers - **3** real bugs found and fixed in one session, across 2 integrations - **36000.0s vs. 86399.6s** — the exact id_token vs. assumed-lifetime mismatch that broke Tonal sync, in seconds, to the tenth - **~14 hours** of every 24-hour cycle the Tonal sync was silently broken - **6 of 48** Peloton "Weight Lifting" rows were real Tonal duplicates; the other **42** were genuine independent classes, untouched - **2** unique-constraint ordering bugs caught and fixed in the same merge function, both before either shipped broken - **72** Tonal sessions and **42** Peloton classes still active after the historical cleanup; **0** real workouts lost - **1** three-month-old `deletedAt` filter fix that had only ever landed on one of the two integrations that needed it - **1** Vercel OAuth device-code login completed mid-session to get production log and database access - **1** stale doc line (Next.js 14 vs. the actual 15) and **1** dead code module cleared in the same pass === ## Why Is Meta Swimming in a Red Ocean with Muse? - URL: https://vibescoder.dev/posts/why-is-meta-swimming-in-a-red-ocean-with-muse - Date: 2026-08-07 - Tags: #ai #meta #open-source #homelab - Reading time: 13 min read Meta had the one asset nobody else in the West could match — frontier-scale open weights, right as sovereign AI became a real enterprise buying criterion. Instead of owning that lane, it built Muse Code to fight Anthropic, OpenAI, and now xAI for a shrinking slice of the most contested market in AI. --- Meta announced [Muse Code](https://research.meta.ai/blog/introducing-muse-code-and-muse-spark-1-2) this week — a terminal coding agent, paired with a new model called Muse Spark 1.2, built to go toe-to-toe with [Claude Code](https://www.cnbc.com/2026/08/05/meta-debuts-muse-code-to-take-on-anthropic-and-openai-.html) and [Codex](https://techcrunch.com/2026/08/05/meta-launches-muse-code-an-ai-agent-for-large-code-bases/). My first reaction was recognition: another coding harness, another proprietary model, another entrant in a category that already has three well-funded incumbents. My second reaction, a few days and a lot of research later, is that Meta just made one of the stranger strategic bets I've seen in this industry. Here's the short version. Meta spent over a decade building the most credible argument in tech that a company doesn't need to own the whole AI stack to win it — PyTorch, FAISS, Llama, a billion-plus downloads. Then, right as the market it was best positioned to dominate finally arrived, it turned around and built a paid, closed, second-place coding agent instead. I want to walk through how Meta got here, why the market it's now competing in looks like a red ocean from every angle, and where I think this actually goes. ## Meta's Long Run as an Open-Source Shop Before Llama, before any of the "did Meta abandon open source" headlines, Meta was already one of the most substantial open-source contributors in AI infrastructure. | Project | Category | Released | Current status | |---|---|---|---| | [PyTorch](https://en.wikipedia.org/wiki/PyTorch) | Deep learning framework | 2016 | Donated to the [Linux Foundation's PyTorch Foundation](https://en.wikipedia.org/wiki/PyTorch) in Sept 2022 — Meta no longer solely governs it | | [FAISS](https://github.com/facebookresearch/faiss/wiki) | Vector similarity search | 2017 | Still developed primarily at Meta AI Research | | [fairseq](https://ai.meta.com/research/publications/fairseq-a-fast-extensible-toolkit-for-sequence-modeling/) | Sequence modeling toolkit | 2019 | Maintained, built on PyTorch | | Detectron / Detectron2, wav2vec, Segment Anything, DINOv2 | Vision & speech research | 2018–2023 | Part of a [FAIR open-publication tradition](https://engineering.fb.com/2018/12/05/ai-research/fair-fifth-anniversary/) predating Llama by years | | Llama 1 | Language model weights | Feb 2023 | Weights [leaked publicly](https://en.wikipedia.org/wiki/Llama.cpp) before Meta's own intended academic-only release | | Llama 2 / 3 / 4 | Language model weights | 2023–2025 | "Open weight" under a [Community License](https://www.digitalapplied.com/blog/meta-ai-business-agents-enterprise-llama-launch-2026) with usage restrictions, not a standard OSI license | | Llama 4 Behemoth | Frontier open model | Announced 2025 | Never shipped; reportedly shelved after underperforming internally | | Muse Spark / Muse Code | Coding agent + model | 2026 | Fully closed, no downloadable weights | The model-weight story is where the "Meta open-sources everything" narrative gets messier than it looks. Llama's original February 2023 weights weren't deliberately released to the public — they were intended for academic researchers and only became a mass phenomenon because they [leaked onto 4chan within days](https://en.wikipedia.org/wiki/Llama.cpp). And the tool that actually made those leaked weights usable on consumer hardware, [llama.cpp](https://llama-cpp.com/), isn't a Meta project at all. It was built independently by a Bulgarian engineer, Georgi Gerganov, specifically because Meta's own implementation depended on PyTorch and CUDA infrastructure most individual developers couldn't run. The entire GGUF/quantization stack that underpins the current wave of local-LLM tooling exists because Meta's own tooling locked most people out, not because Meta built the on-ramp — it's the same stack I leaned on when I [put a homelab RTX 5090 to work running local models](/posts/putting-the-gpu-to-work-running-local-llms) earlier this year, care of Gerganov's project rather than Meta's. Even where Meta genuinely leaned into open weights, the license carries an asterisk worth remembering for later: the [Llama Community License](https://www.digitalapplied.com/blog/meta-ai-business-agents-enterprise-llama-launch-2026) adds a separate license requirement above 700 million monthly active users, bans training competing models on it, and currently can't be used or distributed by EU-domiciled organizations at all. "Open weight" was never quite "open source," and the fine print already excluded a chunk of the sovereignty-minded buyers who'd want it most. ## The Pivot Scale Wang and Muse | Date | Event | |---|---| | Jun 2025 | Meta invests [$14.3B for a 49% stake in Scale AI](https://www.forbes.com/sites/jonmarkman/2026/06/16/why-meta-paid-143b-for-scale-ai-and-alexandr-wangs-data-empire/), installs founder Alexandr Wang to lead the renamed Meta Superintelligence Labs (MSL) | | Aug 2025 | Wang's team [reportedly discusses shelving Behemoth](https://www.artificialintelligence-news.com/news/meta-superintelligence-ai-lab-zuckerberg-talent-war/), Meta's flagship open model, after it underperforms internally post-training | | Apr 8, 2026 | Muse Spark launches — Meta's first fully closed model, invitation-only API, no weights | | Jul 9, 2026 | Muse Spark 1.1 — Meta's first broadly paid developer API | | Aug 5, 2026 | Muse Code + Muse Spark 1.2 launch: a full proprietary coding harness, [co-trained with the model](https://venturebeat.com/orchestration/meta-enters-the-ai-coding-wars-with-muse-spark-1-2-and-muse-code-with-persistent-async-background-agents) it runs on | Wang's own playbook here reads as closer to [Anthropic's than OpenAI's](/posts/thursday-thoughts-why-anthropic-is-the-next-aws-but-potentially-worse): closed weights, enterprise distribution, a "serious partner" brand rather than a consumer-hype brand. Muse Code's "contributor tier" makes the strategy explicit — it discounts token pricing by roughly [12 to 21 times](https://forkast.news/metas-superintelligence-labs-ships-its-first-product-and-the-contributor-tier-is-the-real-strategy/) in exchange for the right to train future Meta models on your code. Given that Meta's headline AI hire runs what is fundamentally a training-data supply company, that's not a generosity play. It's a data-acquisition price, and it's exactly the kind of trade I've [flagged as a governance blind spot](/posts/your-ai-strategy-has-a-blind-spot) for any enterprise pointing a coding agent at code it doesn't want showing up in someone else's training run. Meta hasn't officially killed Llama — older models are still nominally available — but the frontier work has clearly moved elsewhere, and [Muse Spark 1.2 lands second](https://www.orcarouter.ai/blog/meta-muse-code-terminal-coding-agent) on every benchmark Meta itself chose to publish at launch, behind Claude Opus 5, using Meta's own harness. ## Why This Caps Out in the Single Digits Split the buyer market three ways. Small-to-medium businesses mostly want AI bundled with what they already pay for — Copilot riding along with Microsoft, Codex riding along with an OpenAI subscription. Large, regulated enterprises and governments are increasingly pulled toward [sovereign AI](https://ubuntu.com/engage/sovereign-ai-2026): on-prem or private-cloud deployments the enterprise owns outright, for reasons that are as much geopolitical and security-driven as economic. [Sovereign cloud infrastructure spend](https://www.spectrocloud.com/blog/enterprise-ai-2026-trends) is projected around $80 billion in 2026 alone. That leaves a middle band of mid-to-large enterprises as the only realistic addressable market for a proprietary, pay-per-token coding agent like Muse Code — and Muse Code's own rate limits (a 60-request-per-minute cap on the discounted tier) suggest Meta is really building for solo developers and small teams, not that middle band at all. That middle band is also not empty water. It's the most contested part of the entire industry: | Lab | Flagship coding product | Recent signal | |---|---|---| | Anthropic | Claude Code / Claude Opus 5 | [Overtook OpenAI in annualized revenue](https://techcrunch.com/2026/06/16/chatgpts-market-share-slips-below-50-for-first-time/), ~$47B ARR vs. OpenAI's ~$25B run-rate (Apr 2026) | | OpenAI | Codex | Still the largest consumer/developer distribution base; ChatGPT slipped [below 50% market share](https://techcrunch.com/2026/06/16/chatgpts-market-share-slips-below-50-for-first-time/) for the first time in 2026 as rivals gained | | Google | Gemini / Antigravity CLI | Distribution baked directly into Workspace and Android | | xAI (SpaceX) | Grok Build / Cursor | [Acquired Cursor (Anysphere) for $60B](https://www.techzine.eu/news/devops/142197/spacex-acquires-cursor-for-60-billion/) in June 2026, adding ~$2.6B in existing B2B revenue and ~4M developer users overnight | | Meta | Muse Code | Landed [second place](https://www.orcarouter.ai/blog/meta-muse-code-terminal-coding-agent) on its own published benchmarks at launch | Four incumbents with better distribution, better brand trust, or both, all competing for the exact segment Muse can actually reach. Meanwhile, the open-weight side of the market — the side actually suited to the sovereign, on-prem tier — has real Western contenders now, and Meta isn't the strongest one anymore: | Lab | HQ | Funding / valuation | Flagship model | Specs | License | |---|---|---|---|---|---| | [Mistral AI](https://sacra.com/c/mistral/) | Paris, France | ~$4-5.5B raised; ~€11.7B valuation (ASML-led Series C) | Various (Le Chat, enterprise API) | — | Open-weight + paid API mix, explicit European sovereignty positioning | | [Cohere](https://www.businesswire.com/news/home/20260520121796/en/Cohere-Releases-Command-A-An-Open-Source-Enterprise-AI-Model-Built-for-Sovereign-Critical-Infrastructure) | Toronto, Canada | ~$1.5-1.6B raised | Command A+ (May 2026) | 218B total / 25B active MoE, 128K context | Apache 2.0 | | [Nvidia Nemotron](https://www.nvidia.com/en-us/ai-data-science/foundation-models/nemotron/) | Santa Clara, CA | N/A (Nvidia business unit) | Nemotron 3 Ultra (Jun 2026) | 550B total / 55B active, hybrid Mamba-Transformer MoE, 1M context | OpenMDW-1.1, fully permissive incl. commercial use | | [Poolside](https://venturebeat.com/infrastructure/poolside-drops-laguna-s-2-1-an-open-weight-coding-model-that-beats-rivals-10x-its-size) | San Francisco, CA | Reported $500M-$2B raised (inconsistent across sources) | Laguna S 2.1 (Jul 2026) | 118B total / 8B active MoE, 1M context | Apache 2.0 / OpenMDW-1.1 | | [Arcee AI](https://venturebeat.com/technology/arcees-new-open-source-trinity-large-thinking-is-the-rare-powerful-u-s-made) | San Francisco, CA | ~$29.5-50M raised total | Trinity Large Thinking (Apr 2026) | Trinity Mini: 26B total / 3B active | Apache 2.0 | | [Thinking Machines Lab](https://www.axios.com/2026/07/15/mira-murati-thinking-machines-open-weight-model-inkling) | San Francisco, CA | $2B seed at ~$12B valuation | Inkling (Jul 2026) | 975B total params, 45T pre-training tokens | Apache 2.0 (architecture reportedly follows DeepSeek's design) | | [IBM Granite](https://research.ibm.com/blog/granite-4-1-ai-foundation-models) | Armonk, NY | N/A (IBM business unit) | Granite 4.1 (Apr 2026) | Dense 3B/8B/30B, ~15T training tokens, 512K context | Apache 2.0 | | Meta (legacy) | Menlo Park, CA | N/A | Llama 4 Scout/Maverick | — | Community License, EU distribution restricted | None of these match Chinese frontier labs like DeepSeek, Qwen, or Kimi on raw capability yet — I've felt that gap firsthand running [Qwen as a local daily driver](/posts/qwen-is-not-yet-ready-to-power-local-openclaw-deployments) against frontier models, and it's the actual constraint on the whole "Western sovereign AI" movement — but every one of them is explicitly positioning against exactly the buyer Meta used to own by default. Nvidia's Nemotron deserves a specific callout here, because on paper it looks like the strongest counterargument to my whole thesis: Nvidia has more cash than anyone on this list and a long, credible open-source track record. But Nvidia's actual customers for its highest-margin business are the frontier labs themselves — Meta, OpenAI, Anthropic, xAI all buy Nvidia chips by the gigawatt. Nvidia has no commercial incentive to ship a model that's genuinely frontier-adjacent enough to threaten the labs writing those checks. What Nemotron actually looks like in practice is closer to an ecosystem-fostering, workhorse model: efficient, well-documented, genuinely open down to the training data, and good enough for agentic "grunt work" tasks — but not positioned, funded, or trained to be the frontier-scale sovereign alternative the market is short on. Nvidia benefits from more AI demand everywhere, on any hardware; it doesn't benefit from being the company that made its own customers' proprietary models redundant. ## The Blue Ocean Meta Swam Away From Lay all of that side by side and the decision looks backwards. The sovereign-AI buyer wants Western-provenance, open, auditable, self-hostable models, for reasons that have nothing to do with who has the flashiest benchmark chart. Chinese labs are the strongest open-weight option today, but they're disqualified for exactly the buyers who care most about sovereignty. That left a wide-open lane for a well-funded, compute-rich, Western lab to become the default frontier-scale open option — and Meta, with more training compute and more open-source institutional muscle than Mistral, Cohere, Poolside, and Arcee combined, was as close to a lock for that lane as anyone in the industry. Instead, Meta shelved Behemoth, the model that would have been its actual answer to that opportunity, and built a second-place proprietary coding agent that has to fight Anthropic, OpenAI, Google, and now xAI for a market segment that increasingly doesn't even want a proprietary product. The "why" makes sense as a short-term financial decision — 2026 capex guidance sits at $115-135 billion, and giving that output away for free stopped feeling tenable, especially once Anthropic proved a closed API could scale into tens of billions in revenue. It makes less sense as a long-term strategic one, because it trades a nearly uncontested market for the single most crowded one in the industry, at the exact moment the macro trend was bending toward the thing Meta was uniquely positioned to sell. ## Where I Think This Actually Goes My best guess is that Meta ends up correcting course, but not by reviving Llama as a chatbot competitor. I think Meta goes open-source again on the infrastructure layer, not necessarily the frontier model layer — I'd bet on Muse Code itself eventually becoming an open-source harness, the same way Meta gave the world PyTorch and FAISS instead of hoarding them. Zuckerberg's own hedge, "I'll have more to share on that soon" when asked if Muse would open up, reads exactly like a company keeping that door open on purpose. From there, I expect Meta to chase bottoms-up enterprise adoption through open infrastructure rather than top-down proprietary API sales — get the harness, the tooling, and the developer experience into as many hands as possible, the way Llama's download numbers built goodwill Muse Code can't buy at any discount. But the more specific bet is this: Meta has already proven, at consumer scale, that it can out-execute Microsoft and Google on distribution when it commits to a category — that's the entire history of Facebook, Instagram, and WhatsApp against every incumbent that came before them. It hasn't proven that in enterprise yet, and enterprise is the next real growth line available to it. Getting a real foothold there runs through coding first, because coding is where the vibe-coding revolution is already reshaping how knowledge work gets done, and it's the most measurable, highest-willingness-to-pay wedge into the enterprise stack. If Meta wants a second act, it isn't "ship a slightly cheaper Claude Code clone." It's using distribution the same way it always has, aimed at every knowledge worker inside an enterprise, not just the engineers, priced to make the decision easy at the department-budget level rather than the CTO-approval level. So the real question isn't whether Muse Code beats Claude Code on a benchmark chart. It's whether Meta can do to Microsoft and Google in agentic knowledge work what it already did to them in consumer social media — and whether anyone in Redmond or Mountain View is actually prepared for Meta to try. ## By the Numbers - **$115-135 billion** — Meta's 2026 capex guidance, roughly double 2025, the financial pressure behind the closed pivot - **$14.3 billion** — Meta's investment for a 49% stake in Scale AI and Alexandr Wang's move to lead Meta Superintelligence Labs - **12-21x** — the spread between Muse Code's standard and "contributor" (data-for-training) pricing - **2nd place** — where Muse Spark 1.2 landed on all three benchmarks Meta itself chose to publish at launch - **60 requests/minute** — the contributor tier's rate cap, versus 3,000 on standard, a strong hint about who Muse Code is actually built for - **$60 billion** — SpaceX's all-stock acquisition of Cursor, a fourth well-capitalized competitor that landed in the same market the same month - **$47 billion vs. $25 billion** — Anthropic's annualized revenue run-rate versus OpenAI's, as of April 2026, in the exact enterprise-coding lane Muse is chasing - **~$80 billion** — projected 2026 sovereign cloud infrastructure spend, the market segment open weights are best positioned to serve - **2.5-3.2%** — Meta AI's approximate global consumer assistant market share, despite sitting on top of Facebook, Instagram, and WhatsApp - **7 named contenders** — the current Western open-weight field (Mistral, Cohere, Nvidia Nemotron, Poolside, Arcee AI, Thinking Machines Lab, IBM Granite) now competing for the sovereign-AI lane Meta once had nearly to itself === ## Thursday Thoughts: Claude Code Is Lotus 1-2-3, and Copilot Is Playing the Excel Game - URL: https://vibescoder.dev/posts/thursday-thoughts-claude-code-is-lotus-1-2-3 - Date: 2026-08-06 - Tags: #ai #agents #vibe-coding #meta #building-in-public - Reading time: 8 min read Knowledge workers have been vibe coding for decades — it was just called VBA. That history has a second lesson buried in it: the technically better spreadsheet lost the 1980s, and the one with distribution won. Claude Code is winning on quality the way Lotus 1-2-3 did. Copilot is betting that doesn't matter, and this time Microsoft isn't just waiting around for the product to catch up. --- I've been reading up on the history of Excel this week, mostly for fun, and I stumbled into an argument I wasn't expecting to make. It starts with a claim that sounds like a hot take and turns out to be closer to a well-documented fact: enterprise knowledge workers have been vibe coding for decades. It was just called VBA. That's not a cute reframe. "End-user computing" is an actual, decades-old risk-management category — EUC applications facilitate the production of working applications by non-coders, and can essentially be thought of as a subset of shadow IT. Finance teams, ops teams, sales teams have been building unsanctioned little pieces of software inside spreadsheets since the 1980s, without a CS degree or a ticket in the backlog. Coding agents didn't invent this. They just made it faster and put a chat window on it. But the more I dug into Excel's history, the more I realized the interesting story isn't the vibe-coding parallel. It's what happened to the *other* spreadsheet — the one that was actually winning. ## The Best Product Lost the 1980s Excel launched in 1985, Mac-only. Lotus 1-2-3 launched in 1982 and by 1988 — three years after Excel existed — still held roughly 70% of the spreadsheet market to Excel's 10%. Lotus wasn't just ahead. It was the category. Wall Street had already fallen in love with spreadsheets years before Excel showed up. Then Lotus made a bet: it stayed focused on OS/2, the IBM-Microsoft joint venture operating system, instead of porting 1-2-3 to Windows. Microsoft, meanwhile, switched to promoting Windows 3.0, which became the dominant operating system in the world, leaving Lotus's products stuck on an old character-based interface. One Lotus insider, years later, called it exactly what it was: "It was a head fake by Microsoft." By the early 1990s, Excel had started to outsell Lotus 1-2-3, and Lotus never recovered — IBM bought what was left of it in 1995. Read that sequence again. It's not "the better product won." Depending who you ask, it might genuinely be the opposite: some accounts argue Excel earned it on merit (Excel was a vastly superior Windows experience and the first "killer app" for that environment), and others just point at the calendar (Lotus was surpassed by Microsoft in the early 1990s largely because it didn't take Windows seriously in time). Either way, the deciding factor wasn't a feature checklist. It was which company controlled the platform the entire market was about to move onto. I think we're watching a rerun. ## Claude Code Is Lotus. Copilot Is Betting It's Excel. Every satisfaction survey right now says the same thing: Claude Code is the better product. Among developers with 10+ years of experience, 46% now choose Claude Code versus just 9% who prefer Copilot, and that gap holds up across independent surveys — Claude Code leads on "most loved" at 46%, against Copilot's 9%. And yet: GitHub Copilot controls enterprise deployment at 90% of Fortune 100 companies, and at companies with 10,000+ employees, Copilot leads at 56% adoption, reflecting enterprise procurement inertia, existing Microsoft 365 integrations, and compliance infrastructure. Meanwhile Claude Code dominates the small end of the market, where there's no procurement inertia to fight through, holding 75% adoption at companies under 50 employees. That is, functionally, 1988 again. The better product is winning where switching is easy and losing where a platform relationship already exists. If I stopped the post here, this would practically write itself: Claude Code is Lotus, Copilot is Excel, distribution beats features, wait a few years for the enterprise numbers to flip the way they did in 1992. I don't think that's the actual story, though. I think it's more interesting, and slightly worse for Anthropic, than that. ## This Time Microsoft Isn't Waiting Around Lotus's fatal mistake wasn't losing a feature war. It was assuming distribution would keep working while the product quietly fell behind. Microsoft, of all companies, knows this playbook better than anyone alive, because it ran it. And what's actually happening inside Copilot right now doesn't look like a company hoping its product improves eventually. It looks like a company that skipped straight to the endgame. Start with the product gap itself: Copilot Cowork, Microsoft's new long-running agentic product, is built on Anthropic. Microsoft doesn't need Copilot's underlying model to beat Claude. It can just license Claude, wrap it in Microsoft's distribution, and call it Copilot. That's a move Lotus never had available to it, and it quietly dissolves the entire "better model wins eventually" framing. If the engine inside both products increasingly comes from the same place, the competition was never really about the model. Then there's the part that would have saved Lotus if anyone at Lotus had thought of it: governance, shipped on day one instead of bolted on 30 years later. It took the Excel-industrial-complex three decades to build EUC risk-management platforms because spreadsheets went ungoverned for so long that regulators had to force the issue — Shadow IT and uncontrolled end-user computing tools represent one of the most significant and underestimated operational risks in financial services, and regulators are increasingly requiring inventories, risk assessments, and controls. Microsoft is not waiting three decades this time. Agent 365 is already shipping as a $15-per-user control plane for IT and security teams to observe, manage, and secure agents, bundled into a new Microsoft 365 "Frontier Suite" that wraps Copilot, governance, and security tooling into one CIO-friendly line item — a direct response to the fact that 86% of IT leaders say they need additional governance to manage agents at all. That's the tell. Microsoft isn't trying to out-model Anthropic. It's trying to become the thing every EUC governance vendor spent 30 years trying to retrofit onto Excel, except this time it's native, and it's Microsoft selling it to the same CIOs who already buy everything else from Microsoft. ## The Counterargument Anthropic Isn't Lotus Either Let me argue against myself, because the analogy is too clean if I don't. Lotus's failure was passivity. It sat on distribution and assumed the product didn't need to keep up. Anthropic is doing the opposite of that: Anthropic launched Cowork to bring Claude Code's capabilities to knowledge workers beyond developers — financial analysis, legal review, sales operations — which is exactly the move Lotus never made. Lotus never tried to become the platform. It tried to stay the best spreadsheet. Anthropic is explicitly trying not to make that mistake. There's also a real internal signal worth sitting with: Microsoft internally adopted Claude Code across major engineering teams for complex work, notable given that Microsoft sells GitHub Copilot. If Microsoft's own engineers reach for the competitor's product when the work actually matters, that's not nothing. It's the modern version of the diehard Lotus fan who refused to switch, except it's happening inside the building that owns the incumbent. So maybe the honest framing isn't "Copilot wins" or "Claude Code wins." It's that the thing worth owning was never the model. It's the governance layer standing between "agent" and "agent your compliance team will actually sign off on." Whoever wins that layer wins the enterprise, the same way whoever won the OS won the spreadsheet. Right now Microsoft is moving faster on that specific bet than Anthropic is, and it's the one lesson from the Lotus story that Microsoft, uniquely, has no excuse not to have learned. ## Hold Me to It Here's a marker I'm comfortable being wrong about: by the end of 2027, enterprise agent procurement decisions will be driven more by governance and control-plane attach rate — audit trails, permission scopes, IT sign-off — than by which model scores higher on a coding benchmark. If Claude somehow wins that fight instead of just the model fight, I'll write the follow-up admitting Anthropic solved the one thing Lotus never even tried to. Also worth remembering: the market-share numbers themselves don't agree with each other. Depending which 2026 survey you read, Claude Code's overall share is somewhere between 18% and a majority of the market. That spread alone tells you this fight isn't settled. Neither is the one that mattered thirty-some years ago before the first Windows PC showed up on someone's desk with Excel already on it. *If you're inside an enterprise right now, which one did IT actually let you install?* ## By the Numbers - **70% vs. 10%** — Lotus 1-2-3's spreadsheet market share against Excel's in 1988, three years after Excel launched - **1992** — the year Excel decisively overtook Lotus 1-2-3, roughly seven years after its 1985 launch - **46% vs. 9%** — developer preference for Claude Code over Copilot among engineers with 10+ years of experience - **90%** — the share of Fortune 100 companies where GitHub Copilot controls enterprise deployment today - **75%** — Claude Code's adoption rate at companies under 50 employees, where there's no procurement inertia to overcome - **$15/user** — the price of Microsoft's Agent 365 governance control plane, shipped years into the agentic era, not decades - **86%** — the share of IT leaders who say they need additional governance just to manage the agents already in their organizations - **0** — the number of platform shifts Lotus survived by assuming distribution alone would keep it in first place === ## Building a Windows Update Butler: SSH, Scheduled Tasks, Toast Notifications, and a Vercel Dashboard - URL: https://vibescoder.dev/posts/building-a-windows-update-butler-ssh-scheduled-tasks-toast-notifications-and-a-vercel-dashboard - Date: 2026-08-05 - Tags: #agents #coder #building-in-public #homelab - Reading time: 10 min read My gaming rig's Windows partition used to cost me 15-45 minutes of manual clicking every time I booted into it. One afternoon: a Coder workspace talking to it over Tailscale and SSH, an inventory script that catalogs every update channel on the machine, three Scheduled Tasks that actually do the work, a toast notification bug that took longer to fix than the automation itself, and a Next.js dashboard on Vercel to see it all without ever touching the machine. --- My gaming rig has a Windows partition I boot into maybe once every week or two, and every single time, it wanted 15 to 45 minutes of my life back before I could actually play anything. Windows Update, the NVIDIA App, Steam, whatever else had queued up a silent nag icon since the last boot. I'd sit there clicking "Restart Now" and "Update and Restart" like it was a part-time job. This is the same machine I [rebuilt into an SFF custom loop](/posts/ai-nt-no-problem-cramming-a-9950x3d-and-rtx-5090-into-an-sff-custom-loop) a few weeks back, and the [Windows side that still dual-boots for gaming](/posts/gaming-settings-what-broke-and-what-id-recommend) whenever I'm not running the homelab off it. Its Windows-side hostname is a pun on the Linux one, `aint-no-problem` next to `AI-NT-No-Problem`, and it deserved better than a manual dance every boot. So I opened a Coder Agents chat and said, more or less, "figure out what's actually installed on this machine, how each thing updates itself, and then automate all of it." What came out the other side, in one afternoon, was an SSH tunnel into a machine that had never heard of OpenSSH before that day, a catalog of 28 installed programs and how each one actually receives updates, three Scheduled Tasks that do the real work, a toast notification bug that outlasted the automation it was supposed to summarize, and a status page on Vercel that reads the whole history straight off GitHub. ## Getting a Coder Workspace to Talk to a Windows Gaming Rig The workspace and the Windows machine had never spoken before. Tailscale solved the network part: install it in the workspace, bring `tailscaled` up in userspace networking mode since the workspace doesn't have `/dev/net/tun`, and route SSH through it with `ProxyCommand="sudo -n tailscale nc %h %p"`. The machine showed up on the tailnet by its own hostname within a minute of installing the client on the Windows side. The Windows side needed more work. OpenSSH Server isn't installed by default, so step one was a bootstrap script: install the built-in `OpenSSH.Server` capability, start `sshd`, set it to auto-start, confirm the firewall rule, and set PowerShell as the default shell for SSH sessions instead of `cmd.exe`. The first run of that script produced a wall of permission errors and then cheerfully printed "Done" anyway, because none of the failures were fatal to the script itself, just to the actual outcome. Fixed by making the script check for an elevated token up front and refuse to limp forward if it isn't there. Key-based auth needed one more gotcha resolved: the account I was connecting to is a member of Administrators, and Windows silently rejects keys placed in the normal per-user `authorized_keys` file for admin accounts. They have to go in `C:\ProgramData\ssh\administrators_authorized_keys` instead, locked down to Administrators and SYSTEM only. First login attempt also used the Tailscale account name instead of the actual Windows local account, a reasonable mistake once, an instructive one only once. ## Cataloging the Chaos Before Automating Anything Before writing a single line of automation, I wanted a real inventory: every installed program, and how it actually gets updates. Windows Update, the Microsoft Store, winget, or its own vendor-specific self-updater. So the first script cross-references five independent sources: registry uninstall keys, `Get-AppxPackage`, `winget list`/`winget upgrade`, Scheduled Tasks and services matched against a curated vendor-signature table, and Run-key startup entries as a weak corroborating signal. The first pass was full of false positives, and fixing each one was its own small lesson in why fuzzy string matching is a trap: - Raw substring matching treated the Microsoft Store package `MSTeams` as containing `Steam`, because it literally does. - The vendor label word "Drive" in "Google (Chrome/Drive/etc.)" matched "Driver," which quietly reclassified NVIDIA and Realtek's own drivers as Google products. - A generic "Microsoft" token matched Edge, Office, and the VC++ Redistributables all the way through to "Microsoft.Windows.DevHome," a completely unrelated package. - winget "rediscovering" a program via its own local registry scan, with no actual catalog source, got counted as winget-manageable when it wasn't. The final, reviewed report found 28 installed programs: NVIDIA App and its five related driver entries, five Steam-installed games and Steam itself, Office and Visio on Click-to-Run, a handful of genuinely winget-manageable packages (Tailscale, PawnIO, GameInput, the VC++ redistributables), Edge on its own Omaha-based updater, and a short list of true unknowns (AMD's chipset stack, an Epomaker keyboard utility, a leftover Antec app called iUnity) that need manual research later. That last category turned into an immediate cleanup pass: iUnity was a leftover from before a motherboard swap from ASRock to Asus, gone via its exact silent MSI uninstall string. Microsoft 365 and Visio, unused on a gaming/homelab box, gone the same way. Edge was the interesting one: Windows 11 refuses to let it be uninstalled at all, even with `setup.exe --force-uninstall` (exit code 93, no explanation). Rather than fight a protected system component, I neutered it instead, disabled its update services and scheduled tasks, turned off startup boost and background mode via policy, and left WebView2 alone since other apps depend on it. ## Three Scheduled Tasks and a JSON Dashboard The actual automation is three Scheduled Tasks: | Task | Runs as | Trigger | Does | |---|---|---|---| | Update Orchestrator - System | `NT AUTHORITY\SYSTEM` | Weekly Sunday 3 AM, plus every startup as a catch-up | Windows Update, `winget upgrade` at machine scope, Microsoft Store update scan, NVIDIA App self-update | | Update Orchestrator - User | The interactive account | At logon | Launches Steam silently, waits 5 minutes for its own background updater | | Update Orchestrator - Notify | The interactive account | On-demand only, started by the other two | Reads the run summary and shows a toast | The System task exists because two of its four steps flatly refuse to work under any other account. The Microsoft Store update scan silently no-ops for anyone except SYSTEM, even a fully elevated Administrator. And `winget` itself turned out to be invisible on SYSTEM's `PATH`, despite being fully installed and working perfectly under my own interactive account, because the App Execution Alias mechanism that normally exposes `winget.exe` is a per-user shell feature that SYSTEM never gets. The fix was to stop trusting `PATH` and resolve the real executable straight from `Get-AppxPackage -AllUsers`, filtering out a decoy resource-only package that doesn't actually contain `winget.exe`. Fixing that path resolution had a bonus effect nobody asked for: Microsoft Teams, which had been failing to update with a flat `0x80070005 Access is denied` under a regular user account, started updating cleanly the moment winget actually ran as SYSTEM. Every run writes a JSON summary that leads with a small dashboard block before the per-step detail, so the overall result is visible without reading the whole thing: ```json { "OverallStatus": "Success", "Summary": { "StatusIcon": "[OK]", "StatusLine": "3 of 4 step(s) OK, 1 skipped", "Counts": { "OK": 3, "PartialFailure": 0, "Failed": 0, "Skipped": 1 } } } ``` That `Counts.Skipped` field came back as `null` on the first real run instead of `1`, a classic PowerShell trap: `Where-Object` returns a bare object instead of a single-element array when exactly one item matches, and a bare object has no `.Count` property. Wrapping every count expression in `@(...)` fixed it for good. ## The Toast That Wasn't There Until It Was BurntToast makes the actual notification part almost too easy, one cmdlet, a couple of buttons, done. Getting it to *persist* was the afternoon's most stubborn bug. The banner would slide in, look great, list every step with an icon, and then vanish, and Notification Center simply had no record it had ever existed. No "Windows PowerShell" group, nothing. The debugging path went somewhere I didn't expect: I could query `Get-BTHistory` right after the toast fired and it *did* find the entry, and it still found it 45 seconds later in a completely separate SSH session. The data was there. The registry showed Windows actively tracking notification stats for the app identity. The icon file the toast referenced existed and was valid. Every piece of evidence said this should be showing up, and it wasn't, until a fresh trigger through Task Scheduler and a screenshot from the user confirmed it actually was there all along, with history going back multiple runs. The likely explanation: a one-off UI refresh lag on a specific check, not a real registration problem, and testing over SSH (a different session than the actual interactive desktop) had been muddying the picture the whole time. Once it clearly worked, the natural next step was a second button, "Open Log" already opened the full transcript, so "Open Summary" now opens the raw JSON dashboard directly. ## Shipping It to GitHub and Vercel The last piece turned this from "a script on one machine" into something I can actually check from my phone. Every run's summary now gets pushed to a `run-history/` folder in `carryologist/coder-templates`, straight from PowerShell, using the GitHub REST Contents API and a fine-grained personal access token scoped to nothing but that one repo's contents. No git or GitHub CLI needed on the Windows box at all, base64-encode the file, `PUT` it, done. The token lives in a locked-down file on the machine, Administrators and SYSTEM only, the same ACL pattern already used for the SSH key. And since the data was already sitting in a GitHub repo as clean, timestamped JSON, a small Next.js app was the natural next step: `carryologist/windows-status`, one page, a server-side API route that fetches the run list and the latest run's content with a read-only token (a separate one from the write-scoped token on Windows, least privilege per execution environment), and a dropdown to browse any older run. No database, no cron job, just a static-feeling page pulling live data through its own backend. It went from an empty repo to a working URL, [windows-status.vercel.app](https://windows-status.vercel.app), in well under an hour. --- None of this was a single grand plan. It was a chain of "well, now that I can see that, I should fix this too" moments, an SSH connection revealed a machine's real update chaos, the chaos revealed which programs actually needed cleaning up, cleaning up demanded real automation, automation demanded visibility, and visibility turned out to want a phone-checkable dashboard as much as a desktop toast. *What's the thing sitting on your own machine right now that you've been manually clicking through for months, just because it never occurred to you it could be someone else's job?* ## By the Numbers - **28** installed programs classified across five independent update-channel signals in the first inventory pass - **6** distinct false-positive classification bugs found and fixed before that inventory was trustworthy (Steam/MSTeams, driver/Google, generic-Microsoft/DevHome, and more) - **3** Scheduled Tasks doing the real work, plus a **4th** script pushing every run's summary straight to GitHub - **2** real automation bugs caught only by actually running the thing as SYSTEM: a silently-`null` step count, and `winget` being invisible on SYSTEM's own `PATH` - **1** bonus fix nobody asked for: Microsoft Teams updating cleanly for the first time, purely as a side effect of the winget path fix - **~1,460** lines of PowerShell and TypeScript shipped across two repos in one afternoon - **1** brand-new Next.js app, deployed to Vercel, live before the session ended - **0** git or GitHub CLI installs required on the Windows machine itself, the sync script talks to GitHub's REST API directly === ## 115 Days In: One Afternoon of Admin Panel Improvements, End to End - URL: https://vibescoder.dev/posts/115-days-in-the-admin-panel-i-built-but-never-stress-tested - Date: 2026-08-04 - Tags: #building-in-public #agents #debugging #next-js #coder #cloudflare #meta - Reading time: 7 min read 115 days after building this blog from a cabana in Cabo, I spent one afternoon — 2:54 PM to 6:23 PM, back to back — running a chain of admin panel improvements: image upload, a visual TODO viewer, drag-and-drop reorder, a "launch agent" button, four rounds of troubleshooting it, and three mobile layout bugs found by actually using it on my phone at the end. --- [Day One](/posts/day-one-building-vibescoder-dev) of this blog was 115 days ago — a lounge chair in Cabo, an iPhone, and a Coder workspace. The [admin tooling](/posts/day-three-admin-tooling-and-the-edit-pipeline) has been growing ever since. This post isn't about 115 days of slow drift, though. It's about one afternoon, back to back, no gaps: 2:54 PM to 6:23 PM, a single continuous run of admin panel work that went from a small bug fix to a genuinely ambitious feature to four rounds of finding out why that feature didn't actually work to three mobile bugs I only found because I finally used the thing on my phone. Here's the real order it happened in. ## The Timeline | Time (PDT) | What shipped | |---|---| | 2:54 PM | Fixed a loose-file blind spot in the image-orphan detector (unrelated bug, same session) | | 3:12 PM | Built `/admin/settings` — a page for the AI writing-style config that had a working backend and no UI at all | | 3:15 PM | Added a **read-only visual TODO viewer** at `/admin/todo` — the backlog file, rendered as a real checklist instead of raw Markdown | | 4:21 PM | Wired **image upload** directly into the `/admin/images` browser (previously only reachable mid-post-edit) | | 4:21 PM | Added **drag-and-drop reorder** to the TODO viewer — move items, click Save, one commit | | 4:38 PM | Added a **🚀 launch agent button** next to every TODO item — click it, fire a real Coder Agents chat pre-prompted to tackle that item | | 4:51 PM | Fix #1: added the function timeout the launch-agent route was missing | | 5:00 PM | Fix #2: guarded a second crash path in the same route | | 5:15 PM | Fix #3: diagnostic logging to find out why it *still* wasn't working | | ~5:15–6:00 PM | Fix #4 and #5 (not code — a Cloudflare setting and a malformed API token), verified working end to end | | 6:23 PM | Fixed three mobile layout bugs, found by finally opening the panel on my actual phone | Under four hours, one continuous session, six real features and five real bugs. Here's the texture of it. ## Building the Todo Admin Then Improving It Twice The visual TODO viewer went in first (3:15 PM) — nothing clever, just parsing the backlog file into an actual checklist instead of asking me to read raw Markdown. Within the hour it got two upgrades in the same sitting: drag-and-drop reorder (4:21 PM, so I could rearrange priorities without hand-editing a file), and — twenty minutes later — the ambitious one. A 🚀 **launch agent** button next to every open item. Click it, and it fires a real Coder Agents chat via the same Chats API the Coder Agents UI itself uses, pre-prompted to go tackle that specific backlog item end to end. Image upload landed in the same window (4:21 PM, effectively simultaneous with the reorder feature) — a different corner of the admin panel, but the same afternoon, the same instinct: the backend already existed, it just weren't reachable from the one screen that should have had it front and center. ## The Button That Didn't Work Four Different Ways The launch-agent button looked done at 4:38 PM. It wasn't. Getting it from "merged" to "actually works" took five separate fixes across roughly ninety minutes, each one uncovering a different layer of the problem: 1. **4:51 PM** — First click: `Unexpected token '<', "/SKILL.md` (paste into the Settings UI by hand) and `workspace//SKILL.md` (auto-synced into any workspace built from the Docker template). A `coder-templates` startup-script step clone-or-pulls `agent-skills` into `~/.agents/skills/.agent-skills-sync` and symlinks each `workspace/` directory into place. The stale-copy bug traced back to something dumb I'd done to myself a week earlier: `ln -sfn` can't replace a pre-existing *real* directory — only another symlink — and I'd manually `cp`'d the workspace skills into place for early testing before the sync script had ever run. The symlink step was silently losing every time to leftover manual copies from my own earlier testing. Fixed by clearing any non-symlink target before linking. There's a real open question buried in that bug I still haven't chased down: the startup script starts with `set -e`, and the failing `ln` step still didn't abort the rest of the script. Worth understanding before I write more startup-script logic that assumes `set -e` will actually catch a failure the way it's supposed to. ![All five personal skills listed in the Coder Agents Settings UI: ff, homelab, migrate, scan, todo](/images/friday-fixes-jul-16/personal-skills-all-five.png) *Where it ended up — five personal skills, portable across every chat and workspace.* ## Where It Landed Five skills, two of them Personal (`/migrate`, `/homelab`) and three Workspace-tier (`/ff`, `/todo`, `/scan`), all available from the very next chat I opened with zero setup required on my end. The thing that actually started this — losing a detail to a bad compaction mid-session — hasn't happened since. When a session runs long now, I type `/migrate`, get a clean handoff doc and a resume prompt, and start fresh instead of hoping the compaction landed cleanly. The more durable outcome, though, is the two-systems distinction itself. It's not documented anywhere obvious, and "restart the workspace" is the natural first instinct when something in Coder Agents isn't showing up — which is exactly the instinct that burns an afternoon on this particular problem, because restarting fixes nothing here. If your skill isn't showing up in the `/` menu, the question isn't "did I restart enough" — it's "did I build the right *kind* of skill." ## By the Numbers - **5** skills shipped: `/migrate`, `/homelab` (Personal), `/ff`, `/todo`, `/scan` (Workspace) - **2** completely separate skill systems in Coder Agents, only one of which the `/` menu will ever show - **2** repos created or modified: `carryologist/agent-skills` (new), `carryologist/coder-templates` (startup-script sync step) - **1** real bug found and fixed in the sync mechanism, in the same session it was introduced - **1** dead end fully explained by reading `coder/coder` source directly instead of guessing - **0** new credentials introduced — full Chats API automation deliberately deferred rather than adding a session token just to save a few copy-paste steps === ## Stormtrooper Firing Backwards: How Fixing a Fan Dropped Thermals 20 Degrees in a Gaming PC - URL: https://vibescoder.dev/posts/stormtrooper-firing-backwards-how-fixing-a-fan-dropped-thermals-20-degrees-in-a-gaming-pc - Date: 2026-08-01 - Tags: #homelab #benchmark #building-in-public - Reading time: 16 min read My Windows gaming rig, hostname Stormtrooper, had a case fan installed backwards. Turning it around dropped GPU temps 16-18°C. Then I went looking for a CPU undervolt and found out the board doesn't have one. Then I tried to make idle quieter and accidentally cooked the CPU 18°C hotter across every phase, including the ones with no load at all. Four thermal test runs, one Core Ultra 9 285K, and a lot of honest data. --- Stormtrooper is my other Windows gaming machine. It's a labor of love where I gutted an Alienware PC and repurposed the Core Ultra 9 285K and an RTX 5080 into a tiny [Thorzone Nanoq S](https://thor-zone.com/mini-itx/nanoq/) SFF case. It's not the homelab server, it doesn't do inference, it doesn't host anything — its entire job is to run games and not sound like a leaf blower while doing it. A few weeks ago I noticed the GPU was running hotter than it should. When I finally opened the case, I found the problem: a fan had been installed backwards, quietly moving air the wrong direction since I built the thing. That discovery turned into four separate thermal test runs, an exhaustive tour of a BIOS menu tree looking for a setting that turns out not to exist on this board, and a fan-curve edit that briefly made the CPU run 18°C hotter than before I "fixed" it. Here's the full story, with the numbers. ## The Machine | Component | Detail | |---|---| | Hostname | `Stormtrooper` | | CPU | [Intel Core Ultra 9 285K](https://www.intel.com/content/www/us/en/products/sku/241060/intel-core-ultra-9-processor-285k-36m-cache-up-to-5-70-ghz/specifications.html) | | GPU | [NVIDIA GeForce RTX 5080](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5080/) | | Motherboard | [ASUS ROG STRIX B860-I GAMING WIFI](https://rog.asus.com/motherboards/rog-strix/rog-strix-b860-i-gaming-wifi/) | | Case | Thorzone Nanoq S | | Storage | [Samsung SSD 990 PRO](https://www.samsung.com/us/computing/memory-storage/solid-state-drives/990-pro-pcie-4-0-nvme-ssd-4tb-mz-v9p4t0b-am/) 4TB | | RAM | Corsair CMK64GX5M2B5600C40 (DDR5) | | BIOS | AMI UEFI, version 2.22.1295 | ## The Harness I already had a Linux thermal test harness from [the homelab migration post](/posts/ai-nt-no-problem-cramming-a-9950x3d-and-rtx-5090-into-an-sff-custom-loop). Porting the concept to Windows meant a PowerShell rewrite, and it took two attempts to get GPU load working reliably. The first version launched [FurMark](https://geeks3d.com/furmark/) directly over SSH. It ran, logged sensors, and produced a summary — but the GPU barely noticed. Utilization peaked at 11%, power topped out at 86W. GUI/GPU workloads launched through an SSH service session don't run in the interactive desktop session on Windows, so FurMark's rendering never actually touched the GPU. The fix was a Windows Scheduled Task, the same trick I leaned on when [building the Windows Update Butler](/posts/building-a-windows-update-butler-ssh-scheduled-tasks-toast-notifications-and-a-vercel-dashboard). Instead of launching FurMark directly, the harness writes a FurMark configuration file and runs [`schtasks /Run /TN VibeFurMarkAutonomous`](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/schtasks), which executes the actual stress process inside the logged-in interactive session. That one change took the GPU from an 11%-utilization non-event to a real, repeatable stress load. The final harness runs six sequential phases — `idle`, `cpu`, `gpu`, `combined`, `storage`, `cooldown` — polling CPU, GPU, NVMe, and DIMM sensors every second via [LibreHardwareMonitor](https://github.com/LibreHardwareMonitor/LibreHardwareMonitor) and [`nvidia-smi`](https://developer.nvidia.com/system-management-interface). A full run is about 65 minutes and produces a per-phase min/max/avg summary for every sensor. I ran this exact harness four times over the course of a month, and the comparisons below are all apples-to-apples across those four runs. ## Act One the Backwards Fan The GPU was hitting 89°C under load. Combined CPU+GPU stress pushed it to 91.8°C at the memory junction. That's not catastrophic, but it's not right for an RTX 5080 in a case this size, and the GPU fan was sitting at a flat 31% the entire time — barely responding to the load at all, which was itself a clue that something was wrong with airflow reaching the card, not with the card's own fan curve. One case fan was installed backwards. I flipped it and reran the identical harness. | Phase | Sensor | Before Avg | Before Max | After Avg | After Max | Δ Avg | Δ Max | |---|---|---:|---:|---:|---:|---:|---:| | **GPU** | GPU core | 82.7°C | 89.0°C | 66.6°C | 70.0°C | **-16.1** | **-19.0** | | | CPU package | 52.1°C | 61.0°C | 50.9°C | 60.0°C | -1.2 | -1.0 | | **Combined** | GPU core | 87.2°C | 89.0°C | 69.0°C | 71.0°C | **-18.2** | **-18.0** | | | CPU package | 57.3°C | 74.0°C | 55.3°C | 73.0°C | -2.0 | -1.0 | | **Storage** | GPU core | 39.4°C | 71.0°C | 38.7°C | 52.0°C | -0.7 | **-19.0** | | | CPU package | 58.6°C | 70.0°C | 57.5°C | 66.0°C | -1.1 | -4.0 | One fan, turned the right way around, dropped GPU temps 16-18°C on average and up to 19°C at peak, for zero dollars spent. CPU package improved a modest 1-2°C as a side effect of better overall case airflow. NVMe and DIMM temps both improved 1-2°C across the board for the same reason. A few honest caveats before I call this a clean win: - **The two runs were about a month apart**, and I didn't log ambient room temperature for either one. A cooler room on the "after" day would inflate this result. I don't think that's what happened here — the GPU-specific magnitude of the improvement is too large and too phase-consistent to be pure ambient drift — but it's a real gap in the methodology. - **The physical fan I corrected was a case fan, not confirmed to be a fan mounted directly on the GPU board itself.** The telemetry only reports the GPU's own onboard fan behavior, which also improved (see below), but that's consistent with better case airflow reaching the card, not proof the corrected fan was somehow part of the GPU's own cooler. - **The GPU's own fan response is the strangest data point here.** At a flat 31% while the core sat at 89°C in the "before" run, the card's own curve should have been ramping harder than that. After the fix, the same fan curve settled into 55-59% at a *cooler* 66-70°C. I don't have a clean explanation for why the fan was so passive while the card was this hot before the fix — possibly a curve applied at the time that I've since lost track of, possibly bad airflow creating a hot pocket that the card's temperature sensor didn't fully reflect until it was too late in the sample window. Flagging it rather than pretending I understand it. - **GPU power went up, not down**, after the fix: +11.5W average in the `gpu` phase, +20.0W in `combined`. This is the opposite of what happened in the homelab's custom-loop migration, where a cooler GPU pulled *less* power. Here, a cooler card apparently had more thermal headroom to sustain higher boost states for longer, so it drew more power doing more work rather than the same work more efficiently. Both are legitimate outcomes of the same underlying mechanism — a chip with thermal headroom will use it, one way or another. ## Act Two Hunting a Voltage Knob That Isn't There With the fan fixed, the obvious next lever was undervolting. Lower voltage, same clocks, less heat, in theory. I went in with a hypothesis: a `-60mV` global CPU offset, a restored `250W` PL1/PL2 power limit if the board had been running unlocked, and an 875-900mV RTX 5080 curve via MSI Afterburner. None of that survived contact with the actual BIOS. The Core Ultra 9 285K is built on Arrow Lake, which did away with the old Fully Integrated Voltage Regulator that every "-60mV global offset" undervolting guide from the last decade assumes exists. Arrow Lake instead uses an external motherboard voltage regulator feeding [individual per-block Digital Linear Voltage Regulators](https://skatterbencher.com/2025/03/16/skatterbencher-84-core-ultra-9-285k-overclocked-to-5800-mhz/) — one for each P-core, each E-core cluster, and the Ring. In theory you undervolt those per-block DLVRs, or nudge specific points on the factory voltage/frequency curve, not set one flat global number. In practice, on this board, none of that is exposed at all. I went through the BIOS menu by menu: - **Ai Tweaker → Tweaker's Paradise**: ratio controls, `Actual VRM Core Input Voltage`, CPU Graphics/SA/memory-controller voltages, DRAM voltages. No P-core, E-core, or Ring DLVR voltage field anywhere. - **Ai Tweaker → Internal CPU Power Management**: this is where I found the actual power limits — `Current Long Duration Package Power Limit: 250 Watt`, `Current Package Power Time Window: 56 Sec`, `Current Short Duration Package Power Limit: 250 Watt`. That's Intel's stock spec for the 285K exactly. The board was never running an unlocked profile in the first place, so the "restore PL1/PL2 to stock" step of my plan turned out to be a non-event — a legitimate finding, just not the one I was expecting. - **Advanced → CPU Configuration → CPU - Power Management Control**: SpeedStep, Speed Shift, C-states, Turbo Mode. No voltage. - **Ai Tweaker → DIGI+ VRM**: current capability, VRM switching frequency, power phase control, load-line calibration for the CPU graphics and system-agent rails. Still nothing for CPU core voltage. - **BIOS-wide search for "offset"**: ten results, every one of them a PLL voltage offset (Core PLL, Ring PLL, SOC PLL, memory-controller PLL) or a clock-training correction. PLLs are tiny, low-power clock-generation circuits — tuning them is an extreme-overclocking trick for squeezing stability out of unstable ratios, not a thermal lever. Nothing here touches the voltage that actually generates heat under load. For good measure I tried Intel's own Extreme Tuning Utility from Windows. It refused to launch its tuning features at all: *"system does not support overclocking."* [XTU's clocking and voltage controls have historically been restricted to Z-series boards](https://www.tomshardware.com/pc-components/cpus/intel-releases-xtu-version-10-exclusively-for-core-ultra-200s-arrow-lake-cpus) — Z690, Z790, and now Z890 for Arrow Lake — and the B860-I enforces that same restriction at the driver level, not just in the BIOS. **Conclusion: CPU undervolting is not available on this board, through any interface.** Not a bug, not something I missed — a deliberate chipset-tier restriction that Intel and ASUS both enforce. If you want to tune Arrow Lake voltage, you need a Z-series board. I also skipped GPU curve tuning entirely, even though MSI Afterburner would have let me do it. The honest math didn't favor it: a curve edit in the 875-900mV range would plausibly buy another 4-9°C, but only while Afterburner is actually running in the background — kill the process, update Windows, or forget to check after a reboot, and the card silently reverts to stock with no warning. A few degrees in exchange for an ongoing operational dependency wasn't a trade worth making for a machine whose whole point is to not need babysitting. ## Act Three the Quiet Idle Curve and the Mistake in the Middle The last thing on the list wasn't performance, it was noise. The fan-fixed baseline was already cooling well; I just wanted quieter idle without giving up any of that ramp-up capability under load. My first pass at the fan curve went too far. I reran the full harness expecting a mild idle improvement and instead got this: | Phase | Fan-Fixed Baseline Avg/Max | First Quiet-Curve Attempt Avg/Max | Δ Avg | Δ Max | |---|---:|---:|---:|---:| | idle | 37.3 / 60.0°C | 54.5 / 77.0°C | **+17.2** | +17 | | cpu | 41.4 / 67.0°C | 59.9 / 87.0°C | **+18.5** | +20 | | gpu | 50.9 / 60.0°C | 67.4 / 79.0°C | **+16.5** | +19 | | combined | 55.3 / 73.0°C | 71.5 / 93.0°C | **+16.2** | +20 | | storage | 57.5 / 66.0°C | 74.0 / 83.0°C | **+16.5** | +17 | | cooldown | 39.5 / 56.0°C | 57.0 / 74.0°C | **+17.5** | +18 | The diagnostic detail that mattered: this offset was almost perfectly flat across every single phase — including `cooldown`, where there's no load at all. If I'd only touched the low end of the curve for a quieter idle, the gap should have shrunk to nothing once the curve reconverged with its old high-temp ramp under real load. It didn't. A near-identical +16 to +18°C penalty showed up whether the CPU was doing nothing or getting hammered. That flatness ruled out the two obvious innocent explanations. It wasn't ambient temperature — the GPU, sitting in the same case breathing the same air, stayed exactly where it was supposed to be (and was actually 3-5°C *cooler* than baseline in the `gpu` and `combined` phases, since its curve is independent and untouched). And it wasn't a load-dependent curve shape issue, because a curve edit confined to idle behavior would have produced a shrinking gap at higher temperatures, not a constant one. The likely explanation: whatever fan or fan zone I edited had its duty cycle reduced across its *entire* operating range, not just the quiet-idle segment I intended to touch. I went back into the BIOS's Q-Fan Control screen and rebuilt the `CPU_FAN` curve point by point instead of eyeballing it: ![ASUS BIOS Q-Fan Control screen showing the final 8-point CPU_FAN curve, ramping from 20% duty at 20°C to 100% duty by 65°C](/images/stormtrooper-firing-backwards-how-fixing-a-fan-dropped-thermals-20-degrees-in-a-gaming-pc/final-cpu-fan-curve-bios.jpeg) | Point | Temperature | Duty Cycle | |---:|---:|---:| | 1 | 20°C | 20% | | 2 | 30°C | 25% | | 3 | 40°C | 50% | | 4 | 50°C | 80% | | 5 | 55°C | 90% | | 6 | 65°C | 100% | | 7 | 70°C | 100% | | 8 | 100°C | 100% | That's a narrower quiet band and a steeper ramp than I remembered it as being: duty only stays low through the 20-30°C range, then climbs fast, hitting 80-90% by 50-55°C and pinning at 100% from 65°C on. Reran the harness a second time: | Phase | Fan-Fixed Baseline Avg/Max | Broken Curve Avg/Max | Corrected Curve Avg/Max | |---|---:|---:|---:| | idle | 37.3 / 60.0°C | 54.5 / 77.0°C | **46.5 / 64.0°C** | | cpu | 41.4 / 67.0°C | 59.9 / 87.0°C | **53.0 / 76.0°C** | | gpu | 50.9 / 60.0°C | 67.4 / 79.0°C | **62.3 / 73.0°C** | | combined | 55.3 / 73.0°C | 71.5 / 93.0°C | **69.1 / 88.0°C** | | storage | 57.5 / 66.0°C | 74.0 / 83.0°C | **71.7 / 82.0°C** | | cooldown | 39.5 / 56.0°C | 57.0 / 74.0°C | **47.8 / 73.0°C** | Meaningfully better than the broken version everywhere — 6 to 13°C improvement per phase, no more 90°C+ spikes, and GPU numbers unchanged from baseline throughout (67.5°C avg in `gpu`, 69.6°C avg in `combined`, both within a degree of the original fan-fixed numbers), confirming the GPU side of this was never part of the problem. It's not a full return to the original steep-ramp baseline, though. CPU package is still running 8-14°C warmer than the fan-fixed baseline across every phase, worst in the sustained-load phases: `combined` is +13.8°C average, `storage` is +14.2°C average. The curve's mid-to-high range is softer than the original, just not broken anymore. Every number here is still comfortably under Arrow Lake's ~100-105°C throttle point — `combined` tops out at 88°C, well short of danger — so this is a real, livable tradeoff rather than a problem. I'm keeping it. Quiet idle was the whole point of this last step, and I got it without cooking anything. ## What I Learned **A flat, load-independent offset in a before/after comparison is a diagnostic gift.** When the `broken curve` numbers above moved by almost exactly the same amount in every single phase — idle, full load, even the load-free cooldown — that was the tell. A real workload-driven effect grows and shrinks with the workload. A structural change to something upstream of all of them (in this case, an over-broad fan curve edit) shows up as a constant. When every phase moves together regardless of what the machine is doing, stop interrogating the workload and start re-checking what you actually changed. **Motherboard fan header telemetry doesn't always tell the truth.** Across all four runs, `mobo_fan1` through `mobo_fan7` RPM sensors read a flat `0` and their duty-cycle sensors read a flat `100%`, regardless of what the fans were actually doing. LibreHardwareMonitor just doesn't expose this board's fan headers reliably in this context. I have thermal *consequences* of the fan curve changes, not direct fan-speed confirmation — worth remembering if you're trying to reproduce this on your own board. **A scheduled task can stand in for a real desktop session**, and it's a pattern worth keeping in your back pocket any time you need a real interactive-desktop GPU (or any GUI) workload driven from an SSH-only automation context on Windows. Direct process launch over SSH quietly runs in a session the GPU never sees; `schtasks /Run` against a pre-configured task does not. **Chipset tier gates more than raw overclocking headroom.** I went in assuming undervolting was a BIOS checkbox away. It turned out to be a feature Intel and ASUS both deliberately withhold below the Z-series tier, enforced consistently in the BIOS UI and in Intel's own tuning software. That's a useful thing to know before buying a board for a build where efficiency tuning matters, not just overclocking. **The single highest-leverage thing I did this entire month was turn a $15 fan around.** Everything after that — the BIOS archaeology, the XTU dead end, the fan curve mistake and its fix — was chasing single-digit-to-low-double-digit degrees for real but much smaller returns, and one of those chases actively made things worse before I caught it. Stormtrooper runs cooler than it did a month ago, quieter at idle than it's ever been, and I now know exactly which knobs this board does and doesn't have — which, it turns out, is worth almost as much as the temperature drop itself. ## By the Numbers - **4** full thermal harness runs across a month - **65 minutes** per run: idle, cpu, gpu, combined, storage, cooldown - **-16.1°C** average GPU temperature drop from turning one fan around - **-19.0°C** peak GPU temperature drop from the same fix - **$0** spent to get there - **10** BIOS menu locations and search results checked before concluding CPU undervolting isn't exposed on this board - **0** CPU voltage offset fields found, anywhere, including via Intel XTU - **250W / 250W / 56s** — this board's PL1/PL2/Tau, which turned out to already be exact Intel stock, not the unlocked profile I assumed I'd be dialing back - **+17.2 to +18.5°C** the uniform CPU penalty from a fan curve edit gone too broad - **6-13°C** recovered by narrowing that same curve back down - **8** points in the final `CPU_FAN` curve, pinned to 100% duty by 65°C - **1** gaming PC, finally both quiet at idle and not lying to me about its fan speeds under load === ## Friday Fixes: Straight Quotes, Missing Closers, and a Homelab Tune-Up - URL: https://vibescoder.dev/posts/friday-fixes-straight-quotes-missing-closers-and-a-homelab-tune-up - Date: 2026-07-31 - Tags: #meta #building-in-public #homelab #debugging #vibe-coding - Reading time: 8 min read A weekly homelab housekeeping pass (Ollama out, Home Assistant current, RustDesk re-pulled), a quote-marks bug that turned out to be a content pipeline gap rather than a font problem, and a full audit of this blog's own "By the Numbers" habit that found it wasn't as consistent as assumed. --- Three fixes this week, none of them dramatic, all of them the kind of thing that only gets caught by actually looking. A routine sweep of the homelab's own tooling, a rendering bug that had been sitting in plain sight on every single post, and an audit of this blog's own closing habit that turned out less consistent than I would have guessed going in. Different systems, same underlying move: stop assuming things are fine because nothing's actively on fire, and go check. ## Part 1 the Weekly Homelab Tune-Up Full scan of the tools that keep `AI-NT-No-Problem` running — Docker, Tailscale, Chrome, Ollama, llama.cpp, Home Assistant, RustDesk, the NVIDIA driver, npm and pip user packages — with a version table and a recommendation for each before touching anything. **Ollama, removed entirely.** It had been installed, but the systemd service was inactive and disabled — llama.cpp's own `llama-embed` and `llama-generate` services are the actual daily-driver stack, and have been for months. Stopped and disabled the unit, deleted the binary, and removed the `ollama` user along with `/usr/share/ollama` — 83 GB reclaimed that had just been sitting there unused. **Home Assistant, updated 2026.5.3 → 2026.7.3.** Docker pull on the `:stable` tag, recreate the container with the identical run config (host network, same config volume, same restart policy). Clean startup; the only warnings were pre-existing Bluetooth capability warnings unrelated to the version bump. **RustDesk, actually stale.** The running image was six months old on a floating `latest` tag — long enough that it was worth re-pulling just to check. It had, in fact, moved: new image ID, built two days prior versus the half-year-old one running. Recreated both `rustdesk-hbbs` and `rustdesk-hbbr` against the new image, same ports and volume, so paired devices didn't need to re-pair. **Routine apt pass plus a kernel bump.** Docker, Tailscale, Chrome, `ubuntu-pro-client`, and a batch of `-security`/`-updates` packages, plus a new HWE kernel that needed a reboot. Verified every service — the two llama.cpp services, Docker, Tailscale, Home Assistant, both RustDesk containers — came back healthy afterward, and confirmed the GPU was still correctly detected post-reboot. **The one thing left alone on purpose: the NVIDIA driver.** Upstream had moved from 595.71.05 to 610.43.03, but nothing in the blog's own thermal- migration history or local-inference research pointed to an actual problem the newer driver would fix, and the driver wasn't even showing as apt-upgradable yet on this box. Recommendation: hold. (It auto-bumped to 595.84 anyway, as a side effect of the kernel/dkms update — same branch, not the jump being deliberately skipped.) **Added after the fact: Coder itself, 2.35.1 to 2.35.2.** This one didn't make the original sweep — it surfaced afterward, when a version check turned up a newer release sitting right there. The interesting part isn't the upgrade (staged the new .deb without restarting the service, so the other active session on the box wouldn't get bounced mid-task); it's why it got missed. Coder ships two parallel tracks, mainline and stable, and this box runs mainline. GitHub's /releases/latest endpoint — the obvious thing to check — only ever points at the stable track (v2.34.6 at the time), which reads as older than the mainline v2.35.1 already installed. A check that stops at the release GitHub calls latest will always look up to date on a mainline install, even when a newer mainline point release exists on a tag GitHub never badges that way. Lesson for next time: compare against the release list filtered to the current major.minor line, not just the one release GitHub happens to flag as latest. **Also added after the fact: RustDesk clipboard sync, broken by the same reboot.** A day after the tune-up, copy-paste from the remote desktop stopped working entirely. Not a RustDesk bug, and not caused by the RustDesk image re-pull above — it traced back to the kernel-upgrade reboot too. RustDesk's Linux clipboard backend needs XWayland, since GNOME's Wayland compositor doesn't support the newer Wayland-native clipboard protocol it would otherwise use, and XWayland only starts the first time some X11 app actually asks for it. A fresh post-reboot session with nobody physically at the machine meant nothing ever asked, so XWayland stayed dark and RustDesk's clipboard service sat crash-looping with an "X11 server connection timed out" error every second or so. Triggering a single X11 client woke XWayland up and clipboard started working immediately — fixed for good with a login autostart entry that pokes X11 a few seconds after every boot, so it doesn't need a human (or a support session) to notice and nudge it again. ## Part 2 the Quote Marks That Weren't Actually Curly **The problem:** Post titles were rendering with identical-looking quote marks on both sides of a quoted phrase — no visual difference between the opening mark and the closing one. **The corner nobody checked:** it looked like a font problem, and it wasn't. Space Grotesk and Inter render a straight ASCII quote character the same way regardless of which side of a phrase it's on — there's no open/close distinction available unless the actual curly Unicode characters are used instead. All 70 of 72 posts on this blog are authored with plain `"` and `'`, and nothing in the rendering pipeline was converting them. The font was innocent the whole time; the gap was in content processing. **The fix:** rather than hand-editing 70 content files (fragile, and every future post would just reintroduce the same gap), the fix lives entirely in the rendering layer. A small `smartQuotes()` helper applies the classic smartypants-style heuristic — opening quote at the start of a string or after whitespace/a bracket, closing quote otherwise — and a matching remark plugin applies it only to markdown text nodes, so code blocks and inline code are left untouched. That combination gets wired into the post page, the RSS pipeline, and the admin preview page; the same helper gets applied directly to the handful of places a title renders outside the markdown pipeline entirely — the homepage post cards, Open Graph images, and shareable snippet images. ```diff - {post.title} + {smartQuotes(post.title)} ``` Merged as [PR #23](https://github.com/carryologist/the-vibe-coder/pull/23). The bug report that started this, incidentally, was a screenshot of a *draft* post on the admin preview page — worth remembering that unpublished content runs through a slightly different render path than a live post, and a fix needs to cover both or it'll look fixed right up until someone checks the other one. ## Part 3 the Post-Closer That Wasn't as Consistent as Assumed **The problem:** every post on this blog is supposed to close with a `## By the Numbers` section — a habit going back months. The working assumption was that Thursday Thoughts posts were the deliberate exception. Going in to check that assumption directly turned it up wrong. **What an actual audit found:** 64 of 72 posts already had the section, formatted identically everywhere it appeared — no drift in the heading text or the bullet style, which was itself good news. But inside the Thursday Thoughts posts specifically, it was genuinely mixed: 4 of 10 had a real section, 6 didn't, with no clean pattern by date. Two more non-Thursday `type: opinion` posts were also missing it. The real signal, once the data was actually sorted by frontmatter `type`: every single `how-to` post and every untyped legacy post had the section. It was specifically `type: opinion` posts — Thursday Thoughts included — where the habit had quietly drifted, 14 out of 22 having it and 8 not. **The fix:** since "every post gets one" was already the dominant, near-universal pattern, the fix went one direction only — add the section to the 8 posts missing it, rather than remove it from the 14 that had it. Each addition is genuine content pulled from that specific post: real counts, real dates, real named things mentioned in the piece, never filler stats invented to fill space. Merged as PR #11 in the (private) content repo. **Making sure it doesn't drift again:** the content repo didn't have its own `AGENTS.md` before this — added one now, documenting the convention explicitly (every post, regardless of type, closes this way; here's the exact format; here's the `type` taxonomy) so a future session doesn't have to re-derive the rule from scratch, or worse, re-introduce the exact same gap a year from now. ## By the Numbers - **83 GB** reclaimed by removing an unused Ollama install - **6 months** stale — how old the RustDesk server image actually was - **2** minor versions Home Assistant was behind (2026.5 → 2026.7) - **1** NVIDIA driver bump deliberately held, with the reasoning written down - **1** Coder point release (2.35.1 → 2.35.2) missed on the first pass, caught after the fact - **1** RustDesk clipboard outage traced back to the same reboot, root-caused live over SSH - **70 of 72** posts authored with plain ASCII quotes, none of them hand-edited to fix this - **0** content files touched to fix the quote-mark bug — the whole fix lives in the render layer - **8 of 72** posts were missing a "By the Numbers" close; **72 of 72** have one now - **1** new `AGENTS.md` added to the content repo so the convention survives past this one pass - **3** fixes, **2** repos, **1** homelab, all from noticing something was slightly off and actually checking === ## Thursday Thoughts: Chat Is the New Git - URL: https://vibescoder.dev/posts/thursday-thoughts-chat-is-the-new-git - Date: 2026-07-30 - Tags: #meta #building-in-public #agents #future-of-coding #vibe-coding #ai - Reading time: 8 min read A senior architect at a financial services firm told me chat context is now more valuable than code. It turns out a whole ecosystem — an arXiv paper with seven interoperable language implementations, two "Git for memory" projects, and zero agreement across model providers — is already racing to answer his question. --- I was sitting in a customer technical advisory council session recently when a senior architect at a large financial services firm said something that stopped me mid-thought. He said that chat was now more valuable to him than git. Let that sit for a second. What he meant was this: the conversation between a human and an agent, the back-and-forth, the instructions, the corrections, the context layered up over time, that is now the most important artifact in his workflow. The code itself is relatively cheap. It can be recreated. As models get smarter and tokens get cheaper, that will only become more true. What can't be easily recreated is the reasoning that produced the code. The decisions made. The paths not taken. ## Code Is Increasingly a Byproduct This isn't a knock on code. Code still has to run. It still has to be correct, secure, tested. But the thing that determines whether you get good code out of an agent is the quality of the context going in. And that context lives in the chat. So when this architect started asking me questions like "how do you fork a chat?" and "how do you save chats, version them, make them immutable?" I realized he wasn't being philosophical. He was asking a real engineering question about how to treat conversation as a first-class artifact in a software development workflow. And we don't really have good answers yet. Git gave us branching, merging, diffing, blame, history. We take all of that for granted now. We have almost none of it for chat. ## The Compaction Problem I've run into this myself, right here on this blog. One of the things I find myself doing constantly is asking my agent to go back and look at the detailed chat logs because the compaction summaries aren't good enough. When a long conversation gets compressed, the nuance gets lost. The agent and I lose the thread. We end up retreading ground we already covered, or worse, making decisions that contradict earlier reasoning we've both forgotten. This is a real problem, not a minor annoyance. If the chat is the primary context, and that context degrades over time through lossy compression, then you're building on an eroding foundation. Every long-running project eventually hits this wall. ## What Durable Chat Architecture Might Actually Require I don't have complete answers here, which is part of why I went looking. But the questions the architect raised point toward a few things that any serious solution would need to address: - **Versioning** — the ability to snapshot a conversation at a meaningful point and return to it - **Forking** — branching a conversation to explore different directions without losing the original thread - **Immutability** — treating certain chat states as canonical records, not editable history - **Search and retrieval** — finding a specific decision or piece of reasoning buried in a long conversation - **Portability** — moving context between tools, models, or sessions without losing fidelity I went digging to see how much of this already exists. More than I expected — just not where the architect was looking. ## What's Already Being Built Nobody is building "git for chat transcripts" yet. But an adjacent problem — versioning an agent's *working memory* across a long task — already has real research and real code behind it, and the vocabulary is unmistakably git's. The clearest example is the [Git Context Controller](https://arxiv.org/abs/2508.00031) (GCC), a 2026 paper out of Oxford and collaborators that gives agents explicit `COMMIT`, `BRANCH`, `MERGE`, and `CONTEXT` operations over a persistent, file-based memory store instead of a flat, ever-growing token stream. The results aren't just theoretical: agents equipped with GCC reportedly resolved roughly half of the SWE-Bench-Lite benchmark, well ahead of dozens of other systems tested, and a self-replication case study more than tripled task resolution over the same agent without it. The idea has already spread past the original paper — [Contexa](https://github.com/swadhinbiswas/contexa), an independent implementation, ships the same `.GCC/` on-disk format in seven different languages (Python, TypeScript, Rust, Go, Zig, Lua, Elixir), all interoperable with each other. A companion paper, [Lore](https://arxiv.org/html/2603.15566v1), draws a distinction worth stealing: GCC is an *intra-session* memory system — it helps a single agent organize its own working memory during one task, with checkpointing and branching for exploration — while Lore is an *inter-session* knowledge-transfer mechanism, encoding decision context into a project's permanent history so future agents and humans inherit it. That split is exactly what the architect was circling. He wasn't only asking how an agent manages memory mid-task; he was asking how an organization keeps the reasoning around after the task, and the session, ends. Two other projects push the "git for memory" metaphor even further, as products rather than research artifacts. [Memoir](https://www.memoir-ai.dev/) bills itself as memory an agent can explain, rewind, and branch — taxonomy-structured and git-versioned instead of a vector database — and it ships as a Claude Code plugin whose memory branches follow your actual git branches automatically, so switching context on `git checkout` doesn't contaminate an unrelated branch's lessons. [Memoria](https://github.com/matrixorigin/Memoria) makes the same pitch on a different backing store: every memory change tracked, auditable, and reversible, with snapshots, branches, merges, and time-travel rollback. What none of this solves is portability. A recent survey of how the major model providers actually handle session state made that gap explicit: OpenAI is steering people toward its Responses API and away from Threads/Assistants (being retired in 2026), xAI/Grok defaults to a 30-day storage window you're expected to export out of on your own, and Google splits the job across a database session service, Vertex AI Sessions, and a separate long-term memory bank. Four vendors, four incompatible answers — and the survey's own bottom line is the most honest thing I've read on this all year: for everyone, the cheapest and most reliable recovery strategy that actually exists today is still just writing your own handoff file. That's not a spec. That's four companies independently reinventing the README. So the *commit/branch/merge* vocabulary for agent memory is real, funded, benchmarked, and already interoperable across seven language runtimes in at least one case. The *portability of an actual conversation* — the thing that started this post — isn't. Every vendor's answer to "how do I keep this chat's context alive" is still bespoke. ## Why This Matters Beyond Individual Workflows If you're a solo developer vibe-coding a side project, losing chat context is annoying but recoverable. If you're an architect at a financial services firm running agents across dozens of engineers and systems, losing chat context is a governance problem. It's a compliance problem. It's an audit problem. Who made that decision? Why did the agent do that? What was the intent behind this implementation? If the answer to all of those questions lives in a chat that's been compacted, overwritten, or discarded, you have a serious gap. And that gap will matter more as agentic workflows handle more consequential work. The broader implication is that the tools and platforms built around software development need to catch up to this shift. We've spent decades building infrastructure around code as the atomic unit. Files, repos, branches, reviews, pipelines. That infrastructure is still necessary. But it's no longer sufficient on its own if the decisions that produced the code exist only in a chat window that gets closed at the end of the day. --- The honest summary of my digging: the field has converged on git's *vocabulary* for agent memory faster than I expected — commit, branch, merge, rollback show up in a benchmarked academic paper, a seven-language interoperable implementation of it, and two independent products, all within the same few months. What it hasn't converged on is a way to move an actual conversation, with its full reasoning intact, between tools, models, or vendors. The architect wasn't looking for a vendor pitch, and none of what I found is one — it's early, some of it is alpha-quality, and none of it is a standard yet. But "nobody's building this" turned out to be wrong. The better description is "everybody's building a piece of this, and nobody's agreed on the interfaces." *If you're already thinking about this problem in your own work, how are you handling it?* ## By the Numbers - **1 conversation** with a financial-services architect that kicked off this whole line of thinking - **2 arXiv papers** proposing git-shaped structure for agent memory (GCC and Lore), one already benchmarked on SWE-Bench-Lite - **7 language implementations** of the same `.GCC/` on-disk format, all interoperable with each other - **2 "git for memory" products** (Memoir, Memoria) shipping branch/merge/rollback today, neither older than a few months - **4 major model providers**, and **4 different, incompatible answers** for how to persist a session - **0 standards** yet for moving an actual conversation, reasoning intact, between any of them === ## A Vibe Coder Is Still A Coder: Why, When, and How To Report Bugs - URL: https://vibescoder.dev/posts/a-vibe-coder-is-still-a-coder-why-when-and-how-to-report-bugs - Date: 2026-07-29 - Tags: #meta #building-in-public #agents #debugging #mcp #coder - Reading time: 10 min read Sometimes your agent's turn just ends with no result — no error, no output, nothing. Nudging it usually gets you the answer that was there the whole time. It's tempting to write that off as a UX hiccup. Sometimes it's a real bug, and finding out is part of the fun of vibe coding. A walkthrough of why that's worth investigating, when it crosses the line into "file this," and how to do it well — using a hanging GitHub tool call as the live example. --- Every so often, your agent's turn just... ends. No error. No crash. No output. The task was running, the reasoning was visibly happening, and then it stops, and there's nothing to show for it. Nine times out of ten, the fix is almost insultingly simple: nudge it. Ask it to restate what happened, or just try again. The answer was usually sitting there the whole time, computed and ready, just never delivered. It's easy to file that under "flaky UX" and move on, because most of the time that's exactly what it is — a rendering hiccup, a dropped frame, not worth a second thought. But "usually a hiccup" isn't the same as "always a hiccup," and every so often that exact symptom is the visible edge of an actual bug. The only way to tell the difference is to stop and look instead of just nudging past it forever. That's not a chore bolted onto vibe coding — figuring out *why* the machine did the odd thing is a big part of what makes it fun in the first place. A vibe coder is still a coder, and that instinct — something's slightly off, let's actually find out why — is the same one that's kept software engineers employed for decades. This post is the why, the when, and the how, using a real hang I ran into this week as the live example. ## Why This Is Worth Investigating A crash tells you where to look. An exception has a stack trace. A failed build has a red line in the output. A stall gives you none of that, which is exactly the trap: the instinct is to read "no error" as "nothing's wrong, just slow," and either wait it out or retry. But something *is* wrong — the absence of an error is itself the anomaly. Code that's merely slow usually shows some sign of life: a progress indicator, partial output, *something*. Code that goes fully silent partway through a task, especially if it happens more than once, is telling you a specific and useful thing: whatever's failing is failing somewhere the system doesn't have error handling for. That's a narrower, more diagnosable claim than "it's broken," and it's worth chasing precisely because most tooling doesn't hand it to you for free. The rule of thumb: treat a repeatable stall as a symptom that deserves the same investigative energy as a crash, not less. If your agent session goes silent on a specific kind of request more than once, don't just nudge harder. Ask it to help you figure out what actually happened. ## The Investigation This week gave me a clean example. Mid-session, working with my coding agent on [unrelated homelab housekeeping](/posts/friday-fixes-straight-quotes-missing-closers-and-a-homelab-tune-up), calls involving GitHub started going silent — sometimes fine, sometimes nothing, no error either way. My first read was close to the trap above: assume it's flaky, work around it, keep moving. I told the agent to stop touching "GitHub API or MCP" and stick to the [`gh` CLI](https://cli.github.com/) instead, which worked immediately and reliably. Problem solved, in the narrow sense. But "worked around" isn't the same as "understood," and a repeatable, tool-specific stall is exactly the kind of signal from the section above that's worth a second pass. So instead of leaving it as a permanent workaround, I asked the agent to help me actually investigate it — systematically, and without risking the same stall taking down the investigation itself. ### Isolating the Failure without Getting Stuck Investigating It The obvious problem with debugging a hang directly: if the thing you're testing hangs, your one investigative session hangs with it. The fix was to spawn the test cases out to independent subagents — five of them, each responsible for one category of the tool calls in question, running in parallel. If one hung, it hung on its own; the other four, and the orchestrating session, kept working. This is the same principle as not debugging a production outage from the one server that's on fire — get a vantage point that survives the failure you're trying to observe. Each subagent got a narrow assignment: call a couple of specific tools from the [Model Context Protocol](https://modelcontextprotocol.io)-based GitHub integration that [Coder's AI Gateway bridges into the chat](https://coder.com/docs/ai-coder/ai-gateway/mcp) (that's where the `bmcp_` prefix comes from — "bridged MCP" — nothing project-specific about it), and report back exactly what happened: success, an error message, or "no response, gave up." | Category | Tools called | Result | |---|---|---| | Identity | `get_me` | Success, first try | | Repo metadata | `list_branches`, `get_file_contents` (single file) | Success, first try | | Search | `search_code`, `search_repositories` | **Empty report on first attempt** — recovered on a nudge; both calls had actually succeeded | | PR/issue read | `list_pull_requests`, `pull_request_read` (single PR) | Success, first try | | Heavy payload | `get_commit` (`full_patch` on a real multi-file commit), `list_commits` (50 results) | **Empty report, twice in a row** — recovered on a third, more terse nudge; both calls had actually succeeded | ### What Came Back Every category eventually reported success. That's the finding worth sitting with for a second — no tool was actually broken, and no GitHub call actually failed. But two of the five subagents — search and heavy payload — came back with a completely empty final report on the first attempt, despite the underlying call having already succeeded. Nudging each one to just restate the result produced the correct answer immediately. The data existed the whole time. Something about producing the final response after ingesting a large or unusually-shaped result was dropping the output, silently, with nothing surfaced as an error anywhere in the chain. That's a materially different bug than "the GitHub integration is broken." It's specific: response size or shape, not tool identity. And it's actionable in a way "GitHub stuff sometimes hangs" never would have been. ## When It's Worth Reporting Not every stall clears this bar, and it shouldn't — filing a bug for every one-off hiccup wastes everyone's time, yours included. A few conditions that tip something from "shrug and retry" to "this is worth someone else's attention": - **It's repeatable, not a one-off.** A single unexplained stall is noise. The same category of call failing the same way more than once is signal. - **You can describe it more specifically than "sometimes it hangs."** If systematic testing gets you to a real correlation — here, payload size/shape rather than tool identity — you have something a maintainer can actually act on. - **It's not your own misconfiguration.** Worth ruling out first: bad credentials, a genuinely offline dependency, a typo in a config file. This one wasn't — every underlying call succeeded once you looked past the missing final response. - **Someone else would hit it too.** This wasn't specific to my environment or my repos; any bridged MCP tool call returning a large or complex result looked exposed to the same failure path. All four were true here, which is what made this worth [filing evidence on](https://github.com/coder/coder/issues/27178#issuecomment-5063300890) rather than just quietly keeping the `gh` CLI workaround forever. ## How to Report It Well Finding a real, reproducible bug is the fun part. What you do with it next is where a lot of people — myself included, historically — drop the ball. Filing it well is its own discipline, and [it's a well-documented one](https://opensource.guide/how-to-contribute/) if you've never done it before: - **Search before you file.** A duplicate issue costs a maintainer time twice: once to notice it's a duplicate, once to close it. A search for the general shape of the problem (not just your exact error text) turned up [coder/coder#27178](https://github.com/coder/coder/issues/27178) — same symptom, already open, already tracked internally on Coder's side. Filing a new issue would have split attention across two threads for no reason. Adding evidence to the existing one didn't. While I was at it, I also found [#24947](https://github.com/coder/coder/issues/24947), the same "MCP failures get silently swallowed" pattern in a different part of the codebase, and [#26984](https://github.com/coder/coder/issues/26984), an already-fixed bug in the same general subsystem — worth citing as context, not worth filing separately. - **Bring reproduction, not just a complaint.** "It hangs sometimes" helps nobody. A methodology a maintainer could rerun — the exact tool categories tested, the exact result for each, the correlation with payload size rather than tool identity — is the difference between a comment that gets read and one that gets skipped. - **State your confidence level honestly.** I flagged a *hypothesis* that this might share a root cause with the already-fixed, unrelated bug in the same subsystem — and said so as a hypothesis, not a claim. Overstating certainty about a codebase you don't have access to just sends maintainers down a path you can't actually back up. - **Redact anything that doesn't need to be public.** No real org names, no tokens, no internal URLs beyond what's needed to describe the environment (self-hosted, version number, general deployment shape). The bug is reproducible without any of that. - **Use the channel that actually works.** Small thing, but real: `gh issue comment` failed outright against this specific repo — the `coder` org has [GraphQL access restrictions for third-party OAuth apps](https://docs.github.com/articles/restricting-access-to-your-organization-s-data/) — so the comment went in via `gh api` REST instead. Worth knowing that distinction exists before you assume a CLI failure means your bug report failed to post. - **Don't demand a timeline.** A comment with evidence is a gift, not a ticket assignment. It's the maintainer's call what happens next and when. None of this is exotic advice. It's the same etiquette that's existed around open source bug reports for decades. What's new is that an agent can now do the tedious middle part — the isolated reproduction, the exact before/after data — fast enough that there's no excuse not to bring receipts. ## What Happens Now The comment is in on [#27178](https://github.com/coder/coder/issues/27178#issuecomment-5063300890). I'm not going to file a separate tracking mechanism for this or nag the thread — that's the "don't demand a timeline" rule applying to myself, not just advice for other people. But I am going to watch it, and when it closes — whether that's a fix, a "works as intended" with an explanation, or something in between — I'll report back here so anyone following along gets the actual resolution instead of a dangling thread. Consider this the first half of a two-part story; the second half depends on someone else's timeline, which is exactly as it should be. ## By the Numbers - **5 subagents** spawned in parallel to isolate the failure without letting it take down the investigation - **5 tool categories** tested: identity, repo metadata, search, PR/issue read, heavy payload - **2 of 5** categories (search, heavy payload) reproduced the empty-response failure - **0** underlying GitHub calls that actually failed — every one succeeded server-side - **3** nudges it took across both flaky categories to get a real report back - **1** existing upstream issue found and contributed to instead of filing a duplicate - **2** additional related issues cross-referenced for context, not filed as new - **1** GraphQL request that failed outright due to org-level OAuth restrictions, worked around via REST - **0** tickets nagged, timelines demanded, or maintainers pinged after the comment posted === ## The $230 Stream Deck OpenAI Just Shipped, and What a Linux Version Would Take - URL: https://vibescoder.dev/posts/the-230-stream-deck-openai-just-shipped-and-what-a-linux-version-would-take - Date: 2026-07-28 - Tags: #homelab #ai #agents #building-in-public - Reading time: 11 min read My wife saw OpenAI's new Codex Micro and thought she could skip buying one — she already has a Stream Deck. That sent me down a rabbit hole: what the device actually does, how it compares to the Elgato hardware it's built on top of, and whether a Linux homelab version is a real project or just another case of reinventing a wheel. A survey first, a build plan second, no results yet. --- My wife saw the OpenAI Codex Micro announcement before I did and had a theory: she already owns a Stream Deck for her Mac setup — a MacBook Pro plus a Mac mini running her own agentic homelab — so maybe she didn't need to buy the new thing. That's a completely reasonable read of the headline, and it's also wrong in an interesting way, which is what got us going on this. OpenAI's new $230 hardware isn't a Stream Deck with a ChatGPT logo on it. It's closer, but not identical, and figuring out exactly where the line sits turned into two separate questions: what does this device actually do, and could I build something like it for a Linux homelab where none of OpenAI's or Elgato's official software runs at all? ## What OpenAI Actually Shipped [On July 15, 2026, OpenAI launched the Codex Micro](https://techcrunch.com/2026/07/15/amid-hardware-legal-battle-openai-releases-a-230-keyboard-for-codex/) — officially `kbd-1.0-codex-micro` — a $230 programmable macro pad co-designed with boutique keyboard maker Work Louder. It's [OpenAI's first branded hardware product](https://en.cryptonomist.ch/2026/07/15/openai-codex-hardware-codex-micro/), and deliberately not the much bigger rumored device: a screenless, portable smart speaker being developed with former Apple design chief Jony Ive, which is a separate project with its own separate timeline. The spec sheet: [13 mechanical switches, a joystick, a rotary encoder, a capacitive touch sensor, and six frosted keys that display color-coded live status for up to six Codex threads](https://en.cryptonomist.ch/2026/07/15/openai-codex-hardware-codex-micro/) — running, complete, waiting on you, or errored. [The joystick launches workflows like debugging an error or refactoring code, and the dial adjusts how much "reasoning," meaning time and compute, an agent spends on a task.](https://gizmodo.com/openai-just-launched-its-first-hardware-product-and-its-a-tiny-keyboard-for-bossing-around-ai-agents-2000786080) Everything is configured through the ChatGPT desktop app rather than a separate driver, and it connects over USB-C or Bluetooth to [Windows and macOS](https://www.absolutegeeks.com/tech-news/openai-launches-codex-micro-hardware-for-agent-management/). No Linux client exists, officially or otherwise. It's worth being precise about what "OpenAI built hardware" means here, because it's less than the phrase implies. [The Codex Micro is a reskinned Work Louder Creator Micro 2](https://overclock3d.net/news/input_devices/openai-hardware-has-arrived-and-im-not-impressed/) — same chassis, same 13-key-plus-joystick-plus-dial layout Work Louder had already shipped [co-branded with Figma in 2023 and Framer more recently](https://www.techtimes.com/articles/319389/20260630/openai-codex-micro-launches-july-15-macro-pad-built-work-louder.htm). The unbranded Creator Micro 2 [sells for $144–$174](https://overclock3d.net/news/input_devices/openai-hardware-has-arrived-and-im-not-impressed/) depending on wired or wireless. OpenAI licensed the hardware, wrote the Codex-specific firmware and app integration, and put a $230 price tag and a "yolo" keycap on it. That's not a knock on the idea — a physical status light for a background agent is a genuinely useful thing to have on a desk — it's just useful to know you're paying $56–$86 over the base hardware for OpenAI's software layer and six months of exclusivity before someone builds the same thing for free. Several reviewers made the comparison unprompted before I could: [one review flatly calls it "a glorified Elgato Stream Deck"](https://overclock3d.net/news/input_devices/openai-hardware-has-arrived-and-im-not-impressed/), and [another draws the same line more evenly](https://www.absolutegeeks.com/tech-news/openai-launches-codex-micro-hardware-for-agent-management/) — customizable buttons for a workflow is exactly Elgato's pitch, the difference is how tightly Codex Micro integrates with one specific vendor's ecosystem versus Stream Deck's broader, plugin-based one. ## The Stream Deck+ Comparison My wife's instinct wasn't wrong about the shape of the thing, just the specifics. [Elgato's Stream Deck+](https://www.tweaktown.com/news/112666/openais-first-consumer-hardware-is-a-dollars230-programmable-keypad-built-specifically-for-codex-users/index.html) is the closest existing analog: LCD keys, rotary dials, and a touch strip, the same physical grammar as the Codex Micro's keys-plus-joystick-plus-dial layout, just built for streaming and productivity workflows instead of agent management specifically. The Stream Deck ecosystem is also older and more open — a mature plugin marketplace, years of community tooling, and (this is the part that matters for the second half of this post) actual third-party Linux support, which OpenAI's device has none of. The meaningful functional difference isn't hardware, it's software binding. Codex Micro's six frosted keys are wired natively to Codex's thread state — they know whether an agent is running, waiting, or errored because OpenAI built that link directly into the ChatGPT desktop app. A stock Stream Deck doesn't know anything about your coding agents unless you tell it to, plugin by plugin, script by script. That's a real advantage for Codex Micro out of the box, and it's also exactly the gap a homelab setup can close on its own terms, because nothing about "read agent state, show it on a physical button" requires OpenAI's firmware. It requires a way to read agent state and a way to talk to the hardware. ## Is a Linux Version Even Feasible This is the question that actually matters for a Linux daily driver, and the honest answer is: mostly yes, and mostly already built. Elgato's Stream Deck hardware has been reverse-engineered and supported on Linux for years, independent of anything OpenAI just did. [`python-elgato-streamdeck`](https://github.com/abcminiuser/python-elgato-streamdeck) is an open-source Python library that talks to the hardware directly over USB HID — no official Elgato software required — and works across Windows, macOS, and Linux via the same `hid` backend, with [documented setup steps for Debian/Ubuntu](https://python-elgato-streamdeck.readthedocs.io/en/0.6.3/) including the udev rule needed for non-root access. It's the library underneath most of the third-party Linux tooling that exists today, including [`snakedeck`](https://github.com/jpetazzo/snakedeck), a Python controller built for exactly the kind of thing I'd want: keys that run arbitrary shell commands and update their own label or state based on what that command returns. For something closer to a full desktop app, [OpenDeck](https://github.com/nekename/OpenDeck) is a mature, actively maintained Rust/Tauri application that supports Linux, Windows, and macOS from one codebase, and — notably — runs the majority of existing official Stream Deck plugins, including Windows-only ones, via Wine. It's the option that looks most like "install this and get the real Elgato experience, just on Linux." The more interesting piece, given that everything on this blog eventually routes through a coding agent, is [`streamdeck-mcp`](https://github.com/verygoodplugins/streamdeck-mcp): an MCP server that lets an AI assistant configure a Stream Deck through natural language — "build me a control board for Slack" — rather than hand-editing profiles. Its default mode targets the Elgato desktop app on macOS/Windows, but [it ships a legacy USB-direct server specifically for Linux and headless setups](https://github.com/verygoodplugins/streamdeck-mcp), exposing tools like `streamdeck_set_button`, `streamdeck_create_page`, and `streamdeck_switch_page` directly against the hardware. That's the same MCP-driven pattern this blog already leans on for [the custom-domain and Cloudflare work](/posts/downtime-is-a-feature-custom-domains-cloudflare-and-mcp) — point an agent at a well-described tool surface and let it do the wiring. And the agent-status idea specifically isn't hypothetical, either. [AgentDeck](https://github.com/puritysb/AgentDeck) is an existing open-source project that turns a Stream Deck+ into exactly the kind of physical dashboard I want: live session state for Claude Code, Codex, and OpenCode agents, shown as button color and label, with approve/deny and prompt-submit wired to physical keys. A narrower sibling project, [`agentsd`](https://github.com/paultyng/agentsd), does the same thing more simply — a dedicated Stream Deck plugin that listens on Claude Code's own lifecycle hooks and turns permission requests into a button press instead of an alt-tab. Both are real, working code, and both are also macOS-only today, built around macOS-specific plumbing (native Swift daemons, `~/.claude/` hook wiring assumed to run alongside a Mac app). Architecturally, though, they're the clearest evidence that the idea is sound: this is a solved problem on one platform, not an unsolved one in general. Put together, the pieces exist. A Linux homelab version isn't a research problem, it's an integration problem: USB access to the hardware is solved (`python-elgato-streamdeck`, OpenDeck), agent-aware status tooling is solved on a different OS and would need porting rather than inventing (AgentDeck, `agentsd`), and an MCP-driven configuration layer that could plausibly run headless already exists (`streamdeck-mcp`). That's the same conclusion [the adopt-vs-bespoke research post](/posts/has-this-blog-been-a-waste-of-time) landed on for benchmarking tools: check what's already out there before writing more code, and when something's already out there, use it as the foundation instead of the starting line. ## What I'd Actually Build Assuming a Stream Deck+ (closest hardware analog to Codex Micro, with dials and a touch strip) or the simpler no-dial Stream Deck MK.2, here's the automation set that would actually earn a spot on my desk, tied to things this blog already runs: - **One key per active Coder Agents chat or bakeoff contestant**, color-coded by state (running, waiting, error) the same way [Model Showdown](/posts/model-showdown-round-9-qwen-3-6-27b-vs-qwen-3-6-35b-a3b-vs-qwythos-9b-vs-glm-4-7-flash-vs-nemotron-3-nano) already tracks multiple parallel chats manually. - **A hyper-vigilance button** that runs `npm audit` and `tsc --noEmit` against whichever repo currently has focus — the exact manual check that's been the whole subject of [the Friday Fixes / security-audit split](/posts/friday-fixes-the-hyper-vigilance-tax). - **A model-switch dial** wired to `llm-switch.sh`, so swapping the daily-driver model on `AI-NT-No-Problem` is a twist instead of an SSH session. - **A bakeoff scoring board** — buttons or a touch-strip layout that shows the live rubric score per contestant during a Model Showdown run. - **Homelab health tiles** — GPU temperature, whether `llama-server`, Coder, and `cloudflared` are up, pulled the same way the [Cloudflare tunnel work](/posts/downtime-is-a-feature-custom-domains-cloudflare-and-mcp) already monitors uptime. None of this is built yet. This post is the survey and the plan, not the results — the same "research first, results later" split [the adopt-vs-bespoke post](/posts/has-this-blog-been-a-waste-of-time) is using for the benchmarking side of the blog. The actual build is hardware I don't own yet (a Stream Deck+ or MK.2), wired through `python-elgato-streamdeck` or OpenDeck, with `streamdeck-mcp`'s Linux USB server as the likely integration point for letting an agent configure its own status board. If AgentDeck's approach ports cleanly to Linux instead of needing a from-scratch rebuild, that's the fastest path to the agent-status tiles specifically. ## The Shopping List Since none of this is built yet, here's what it would actually take to start, priced at list rather than whatever sale happens to be running: | Item | Cost | Why | |---|---|---| | Elgato Stream Deck+ | $199.99 list (frequently on sale in the $140–$180 range) | Closest hardware analog to the Codex Micro: LCD keys, four rotary dials, touch strip. The dials are what map cleanly to the reasoning-effort/model-switch idea. | | *or* Elgato Stream Deck MK.2 | $149.99 list | Simpler, no dials, 15 keys. Fine if the automation set stays button-only (agent-status tiles, hyper-vigilance button) and skips the model-switch dial. | | USB-C cable | included in box | Single-cable setup, no extra purchase. | | `python-elgato-streamdeck` + `hid` + `Pillow` | $0 (open source) | Direct USB HID access on Linux, no official Elgato app required. | | `streamdeck-mcp` (Linux USB server) | $0 (open source) | MCP-driven configuration, the same pattern already used for the Cloudflare/MCP work. | | OpenDeck (optional, if plugin compatibility matters more than a custom build) | $0 (open source) | Full desktop app alternative if I'd rather run existing Elgato plugins via Wine than write custom automations. | | AgentDeck / `agentsd` source, read-only | $0 | Reference architecture for the agent-status wiring, not a drop-in Linux dependency — both are macOS-only today. | | udev rule + reboot | $0, ~5 minutes | One-time non-root USB access setup. | | Integration time | unpriced | The actual variable. Everything above is either free or a single hardware purchase; the open question is how many hours the agent-status port and the `llm-switch.sh` dial wiring actually take. | Total hardware spend: one device, $150–$200 depending on whether the dials matter enough to justify the Stream Deck+ over the MK.2. Total software spend: $0, because every piece of the Linux stack is open source. The only real unknown left is time, which is exactly the kind of thing [the adopt-vs-bespoke ledger](/posts/has-this-blog-been-a-waste-of-time) is built to measure honestly rather than guess at. ## By the Numbers - **$230** — Codex Micro's price - **$144–$174** — price of the unbranded Work Louder Creator Micro 2 it's built on - **13** mechanical switches, **1** joystick, **1** rotary encoder, **6** status keys on the Codex Micro - **0** official Linux support for Codex Micro, Elgato Stream Deck, or Work Louder's own software - **2** existing agent-status projects for Stream Deck (AgentDeck, `agentsd`) — both macOS-only today - **4** Linux-capable pieces already available: `python-elgato-streamdeck`, `snakedeck`, OpenDeck, `streamdeck-mcp`'s USB server - **$150–$200** — total hardware cost for the Linux version, one device, list price - **$0** — total software cost, every piece of the stack is open source - **1** spouse whose Mac mini agentic homelab now has a Stream Deck rivalry with this one - **0** hardware built or tested — survey and plan only Next step, whenever the hardware shows up: wire `streamdeck_set_button` to something on `AI-NT-No-Problem` and see if a physical light is actually less annoying than a terminal tab. My money's on yes, but that's the kind of claim this blog has learned not to publish without checking. === ## What Is "Alpha," and Why Does It Keep Coming Up In AI Debates? - URL: https://vibescoder.dev/posts/what-is-alpha-and-why-does-it-keep-coming-up-in-ai-debates - Date: 2026-07-27 - Tags: #meta #building-in-public #ai #future-of-coding - Reading time: 8 min read A follow-up research sprint to the rainy-day tool scouting post, this time chasing down a word instead of a stack: where "alpha" actually comes from, why the All-In podcast keeps warning enterprises not to hand theirs to a frontier lab, and what the frontier-vs-self-hosted tradeoff looks like when you shrink it down to homelab scale. --- Well, it's not raining, but I find myself back at my desk doing research ahead of more Thursday Thoughts and experiments. Last time it was an actual gray Cape Cod afternoon and a list of homelab tools I'd been meaning to pin down. This time the weather's fine and the itch is different: a word. Specifically, "alpha," which I keep hearing thrown around in AI debates like everyone already agrees on what it means. I don't think we do. So consider this the second entry in what's turning into a series: research first, opinion later. This one isn't a Thursday Thoughts post itself, it's the homework before one. ## What Alpha Actually Means My working assumption going in was that "alpha" meant something like IP, or maybe just "intelligence," a cute stand-in for whatever makes a company or a model smart. That's wrong, or at least it's not where the word comes from. Alpha is a finance term, and it has a precise, almost boring definition: it's the return an investment generates *above* what you'd expect given the risk you took on, measured against a benchmark. A fund with an alpha of 5 means it outperformed the market by 5%. It's always paired with beta, which is just your exposure to the market itself. Beta is what you get for free by showing up. Alpha is what you get for actually being good. ## Where the Word Came From The specific origin is a 1968 paper by economist Michael Jensen, whose namesake metric, "Jensen's alpha," was originally built to show that most active fund managers *weren't* actually beating the market once you adjusted for risk. Alpha, in other words, was invented as a skeptic's tool. It's the number that separates real skill from just being along for a rising tide. That framing migrated out of finance and into startup and VC culture over the last decade or so, where it got looser and more metaphorical. In that world, alpha became shorthand for whatever counts as a durable, non-obvious edge: a founder's unique insight, a VC's proprietary deal flow, the thing the rest of the market hasn't priced in yet. The common thread across both the strict and the loose definitions: alpha is never just "being smart" in the abstract. It's the *excess*, the part that isn't explained by everyone having access to the same information or the same market. ## Alpha Enters the AI Debate Here's where it gets interesting for anyone building or buying AI right now. A lot of current AI commentary is really just Jensen's question, restated: now that everyone has access to roughly the same frontier models, where does the *excess* come from? One recent take on this put it as bluntly as I've seen: "The alpha isn't in better models," arguing the real edge is organizational, not computational, who can actually turn a model into money. Bloomberg asked almost the identical question as a headline, [*Is AI an alpha engine?*](https://www.bloomberg.com/professional/insights/artificial-intelligence/is-ai-an-alpha-engine/), and landed somewhere similarly hedged: AI helps, but the differentiator is what goes in, not what comes out. There's an even sharper, more literal version of this happening in quant finance, which is fitting given that's where the word started. A recent paper on AI-driven alpha decay models how mass AI adoption in trading endogenously destroys the very excess returns it's supposed to generate: as more funds run AI on the same shared data, their signals converge, and the edge each one extracts has a shrinking half-life, estimated at as little as 18 months at current adoption levels versus 5-7 years before AI. That's not a metaphor. That's the actual word "alpha," in its actual home discipline, mathematically eroding as an actual side effect of AI adoption. Worth sitting with, given what's coming next. ## All-In's Version Don't Give Away Your Alpha This is the thread that sent me down this whole research hole. On [episode 279 of the All-In podcast](https://podcasts.happyscribe.com/all-in-with-chamath-jason-sacks-friedberg/ai-sovereignty-wars-palantir-nvidia-deal-scotus-birthright-ruling-newsom-s-ca-budget-lie), the besties dug into Palantir's sovereign-AI partnership with Nvidia and Alex Karp's CNBC interview around it. Their summary of Karp's argument used "alpha" in exactly the sense above, but aimed at enterprises instead of traders: what technical customers want, they said, is control over their compute, their models, their data stack, and their alpha, meaning their proprietary knowledge, the fear being that a frontier lab could hoover up that proprietary knowledge and eventually turn it into a competing product. Their tagline for the whole idea: "Data retention is your treasure." Friedberg added a real example on the same episode: Anthropic pitching data-sharing arrangements to life sciences companies, most of whom concluded that sharing would commoditize their own business. Chamath then did something I appreciated: he actually tested it, rather than just asserting it. At his company 8090, he ran a standard enterprise migration task across configurations and reported the results on-air: their own harness wrapped around Claude was 1.4x cheaper and 1.5x faster than raw Anthropic Opus, while an open-source model behind that same harness was 16.4x cheaper, though roughly three times slower. His challenge to the audience wasn't "open source always wins." It was closer to: if the savings are this large, why aren't you at least checking whether you can keep your edge off someone else's servers? I'd be doing this research a disservice if I didn't flag the pushback too. [SiliconANGLE's analysis](https://siliconangle.com/2026/07/05/alex-karp-frontier-models-real-fight-enterprise-ai/) of the same episode makes an important point: there is no public evidence that Anthropic or OpenAI trains on customer data in violation of their own terms, and OpenAI has said outright that it doesn't train on customer API data. Karp's framing, per that piece, is partly a fear campaign, even if the underlying enterprise anxiety is real. SiliconANGLE's own shorthand for the two camps is worth stealing: "data communism," where every firm gets access to the same intelligence, versus "data capitalism," where proprietary advantage stays exclusive. I don't think that fight is settled. I think it's exactly the debate. ## Frontier Vs. Self-Hosted the Pros and Cons Stripping the podcast drama away, here's the actual tradeoff, as best I can lay it out honestly from this round of research: | | Frontier APIs (Claude, GPT, Gemini) | Self-hosted / open-weight | |---|---|---| | Alpha exposure | Every prompt is a data transfer to a company that has, in adjacent categories, already shipped competing products against its own ecosystem | Nothing leaves your infrastructure; the weights and the data stack are actually yours | | Raw capability | Best available today, particularly for the hardest reasoning tasks | Real gap remains, and roughly 3x slower in Chamath's own test | | Cost | Priced per token, on a business model Karp's camp argues structurally limits your leverage at the model layer | Up to 16.4x cheaper at scale, once the harness is built | | Vendor stability | Subject to policy whiplash, the same episode cited Anthropic's Fable 5 export-control reversal as a live example | Immune to another company's board decisions, licensing changes, or export-control flip-flops | | Effort to be competitive | Works out of the box, fastest path from idea to working product | Real engineering investment, a raw open model without a proper harness underperforms badly | | Who owns the risk | Frontier vendor absorbs most operational and safety burden | You now own that operational and safety burden yourself | Neither column is a strawman. They're both true at once, which is exactly why this is a real debate and not a marketing slogan in either direction. ## What This Looks Like at Homelab Scale Here's the part that made this research personal rather than academic: I've been running a miniature, unfunded version of Chamath's exact experiment for months without ever calling it that. Every [Model Showdown](/posts/model-showdown-round-7-local-models-vs-the-tag-manager) round on this blog, every fight with a chat template, every `--jinja` flag, has been me asking the same question 8090 is asking with an enterprise budget: is the harness worth building, or should I just rent the frontier? My local rig will never beat Opus or Sonnet on a hard reasoning task, and I don't pretend otherwise. But nothing I run through it teaches Anthropic anything about how I build. That's the whole trade, just shrunk down from a boardroom to a garage. I don't have a tidy answer yet, and I'm deliberately not trying to force one into this post. That's not what this one is for. Consider this the research file, out in the open, ahead of the actual take. *If your business ran entirely on a frontier API tomorrow, would you know what you'd handed over, and to whom?* ## By the Numbers - **12** — search queries it took to run down the origin story of a five-letter word - **58** — years between Michael Jensen's original 1968 alpha paper and this post - **4** — All-In besties involved in the episode that started this, zero of whom are actually named Alpha - **16.4x** — how much cheaper Chamath's open-source harness ran versus raw Anthropic Opus, the number that kicked off this whole rabbit hole - **3x** — how much slower that same cheaper setup was, because nothing is ever just one stat - **11** — sources cited below, one of which is this blog quoting itself - **0** — new conclusions reached today, this is a research post, the take comes later === ## The OpenAI And Hugging Face Exploit Got Me Thinking: Is There a Standard Agent "Sandbox" Definition? Ends Up, Yes - URL: https://vibescoder.dev/posts/the-openai-and-hugging-face-exploit-got-me-thinking-is-there-a-standard-agent-sandbox-definition-ends-up-yes - Date: 2026-07-24 - Tags: #meta #building-in-public #security #ai #agents - Reading time: 10 min read What started as a Thursday Thoughts hot take on the OpenAI/Hugging Face eval-sandbox breach turned into a research sprint: a survey of existing AI agent containment standards, a deep look at the closest one we found (the Agent Sandbox Taxonomy), an attempt to score the actual incident against it using nothing but public disclosures, and a plan to validate then run Coder itself through the assessment. --- I started this one as a Thursday Thoughts post. The setup was clean: OpenAI disclosed that two of its models escaped an evaluation sandbox and breached Hugging Face's production infrastructure to steal the answer key to their own benchmark, and the "sandbox" turned out to be a container with exactly one sanctioned exit, a package-registry proxy, that had a zero-day in it. Righteous conclusion already forming: the industry needs a real, testable definition of "sandboxed," not a marketing word everyone nods along to. Then I got to the part where I was about to write "someone should define this properly" and stopped. That's a lazy thing to assert without checking. Maybe someone already had. So I put the hot take on ice and went looking instead. This is that research, not the take. ## The Landscape Briefly The short version: there's a lot written about agent security, and almost none of it is a scoring standard for a single sandbox's containment architecture specifically. [OWASP's Agentic AI Top 10](https://genai.owasp.org/initiatives/agentic-security-initiative/) and its [Agent Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/AI_Agent_Security_Cheat_Sheet.html) catalog threats and mitigations at a high level, useful as a checklist, not built to produce a comparable score. [NIST's AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) operates a level above this entirely, it's organizational risk governance, not a technical grading rubric for a runtime boundary. [MITRE ATLAS](https://atlas.mitre.org/) catalogs adversarial techniques against AI systems, closer to a threat library than a containment measure. The Cloud Security Alliance has several overlapping efforts, [MAESTRO](https://cloudsecurityalliance.org/blog/2025/02/06/agentic-ai-threat-modeling-framework-maestro), an [AI Controls Matrix](https://cloudsecurityalliance.org/artifacts/ai-controls-matrix-v1-1), and an [Agentic Trust Framework](https://cloudsecurityalliance.org/blog/2026/02/02/the-agentic-trust-framework-zero-trust-governance-for-ai-agents) that scores autonomy on a four-stage ladder from "Intern" to "Principal," which is the closest thing I found to one specific slice of what I was after, how much an agent can do without a human, but it isn't scoped to sandboxing as a whole. RAND's [*Securing AI Model Weights*](https://www.rand.org/pubs/research_reports/RRA2849-1.html) defines five security levels, SL1 through SL5, but for weight theft and exfiltration risk at a lab, not for whether a given agent's runtime sandbox holds under an adversarial task. There's also a recent arXiv paper, [*AI Sandboxes: A Threat Model, Taxonomy, and Measurement Framework*](https://arxiv.org/abs/2606.18532), that's structurally interesting, but it classifies sandboxes into archetypes (simulation-based, digital-twin, adversarial, regulatory, agent-based) rather than decomposing one sandbox into independently gradable layers. None of those do the specific thing I was looking for: take a single agent sandbox, break it into independent parts, score each part, and produce something you could compare across products. Then I found one that does exactly that. ## The Agent Sandbox Taxonomy [The Agent Sandbox Taxonomy](https://github.com/kajogo777/the-agent-sandbox-taxonomy), published in March 2026 and still under active community review, organizes itself around a memorable "7-7-3": **seven defense layers**, **seven threat categories**, and **three evaluation dimensions**. The seven layers, numbered bottom-up because lower layers are foundational: | Layer | Name | Key Question | |---|---|---| | L1 | Compute Isolation | What separates the agent's execution from the host? | | L2 | Resource Limits | Can it exhaust CPU, memory, disk, or time? | | L3 | Filesystem Boundary | What can it read, write, or delete? | | L4 | Network Boundary | What can it communicate with? | | L5 | Credential & Secret Management | Can it see, use, or exfiltrate credentials? | | L6 | Action Governance | Can it perform destructive or unauthorized operations? | | L7 | Observability & Audit | Can you see what it did, when, and why? | Each layer gets scored on **Strength** (0 to 4) and **Granularity** (0 to 3), plus a flat set of **Portability** tags for OS and infrastructure dependencies. The Strength scale is the part I keep coming back to, because it's precisely the distinction that mattered in the OpenAI incident: 0 is no enforcement, 1 is cooperative enforcement the sandboxed process can simply ignore or route around (proxy environment variables, an opt-in convention), 2 is software-enforced by something the process can't bypass internally but an operator could reconfigure, 3 is kernel-enforced and irreversible once applied (namespaces, Landlock, seccomp-BPF), and 4 is structural, the protected resource just doesn't exist inside the sandbox at all (a microVM, a credential proxy, no network device). Every product gets a fingerprint, a CVSS-style vector showing strength at each layer in order: ``` E2B L1:4/L2:4/L3:4/L4:0/L5:2/L6:-/L7:2 ``` The taxonomy also maps its seven threats (data exfiltration, supply-chain compromise, destructive operations, lateral movement, persistence, privilege escalation, denial of service) back onto specific layer combinations with explicit thresholds, so "is T1 exfiltration addressed" isn't a judgment call, it's a mechanical check against whether L3, L4, and L5 all clear a score of 2 or better. And critically, it comes with a composition framework: no single product covers all seven layers well, so the practical guidance is to stack products and take the maximum score at each layer, rather than pretend one tool solves everything. This isn't a thought experiment either. The project ships `products.yaml` with 26 real products scored, a Go-based verification probe you can run against an actual sandbox to check the claims, and an [interactive explorer](https://ast.georgebuilds.dev/) for comparing fingerprints side by side. ## What I Like The Strength scale's cooperative-versus-kernel-versus-structural distinction is, almost exactly, the sentence I wanted to write in my original hot take about the OpenAI incident: a proxy the process can route around is a fundamentally weaker guarantee than a boundary enforced below the application layer, and this taxonomy gives that distinction a name and a number instead of leaving it as vibes. I also like that it's honest about its own limits, the project's README says outright that it hasn't been community-reviewed yet and flags which product scores are unverified. And the composition framework's core finding, that products cluster into "build a better box" or "control what's inside the box" and almost nobody does both, matches exactly what a default-deny, kernel-enforced network policy is supposed to fix in [how we think about secrets belonging below the application layer](/posts/updating-coder-to-get-user-secrets-and-the-art-of-knowing-where-your-secrets-belong). ## What I Don't Like Two gaps stood out once I tried to actually use it. First, there's no layer for what happens *after* containment fails, no kill switch, no automatic credential rotation on trigger, no forensic rollback runbook. The taxonomy tells you whether an agent could plausibly be contained, not what happens in the window after it isn't, and that window is exactly where the OpenAI incident lived. Second, L7 Observability scores whether logging exists and how tamper-evident it is, but not how fast an anomaly actually reaches a human. A system can score well on L7 and still take days to notice something, which is close to what appears to have happened here. Neither gap is a knock on the project, they're reasonable scoping choices for a v1, but they're the two places I'd want the standard to grow next. ## Running the Incident Through It Here's the actual exercise: can I fingerprint the OpenAI/Hugging Face eval environment using [public disclosures](https://openai.com/index/hugging-face-model-evaluation-security-incident/) alone? Mostly no, and that gap is itself the finding. | Layer | Estimated | Why | |---|---|---| | L1 Compute Isolation | Undisclosed | Described only as "highly isolated"; no mechanism named | | L2 Resource Limits | Undisclosed | Not addressed in the public writeup | | L3 Filesystem Boundary | Undisclosed | Not addressed in the public writeup | | L4 Network Boundary | ~2, and it didn't hold | A single proxy chokepoint was the sanctioned exit; a zero-day in it gave code execution, which is exactly the failure mode the taxonomy's S:2 tier warns is reconfigurable but not kernel-irreversible | | L5 Credentials | Likely 0–1 | Stolen credentials enabled lateral movement into Hugging Face's production systems, consistent with ambient, reachable credentials rather than a proxy or ephemeral tokens | | L6 Action Governance | ~0–1, by design | Guardrails were deliberately stripped for this evaluation to measure maximum capability | | L7 Observability | Partial | Logging existed and Hugging Face detected the intrusion independently, but it took five days to connect it to OpenAI's own telemetry, the exact latency gap the taxonomy doesn't currently score | Run that through the taxonomy's own threat rules and T1 (data exfiltration, which needs L3, L4, and L5 all at 2 or better) can't be marked addressed with what's public, not because it's confirmed to have failed everywhere, but because two of the three inputs were never disclosed. That's the actual value of doing this exercise: it doesn't let you conclude "OpenAI's sandbox was bad," which isn't fair to assert from the outside. It lets you say precisely which of seven specific, falsifiable claims about the containment architecture were never made public in the first place. That's a more useful sentence than either extreme, uncritical trust or a reflexive pile-on. *If your own agent's sandbox had to be fingerprinted against these seven layers in public, how many of the seven could you actually answer?* ## What's Next Running the Experiment So, is there a standard definition of "sandboxed"? Ends up, yes, close enough to count. But reading a fingerprint format is one thing; trusting it is another, especially when nearly every entry in the taxonomy's own `products.yaml` carries `evidence_level: docs`, meaning it was inferred from documentation and marketing pages, not hands-on testing. As with all research on this blog, the next step isn't another opinion, it's an experiment. The plan: before we trust our own results, we validate the tool itself. The project ships `ast-probe`, a binary you drop inside a live sandbox to get a verified fingerprint instead of a documentation-based guess. We're going to run it against a few products already scored in the dataset first and check whether we reproduce the taxonomy's own published numbers. If we can't, that's a finding about the probe or the scoring, and it needs to get sorted before we trust anything downstream of it. Once that check holds, we'll run the same probe against a live Coder workspace configured the way we actually run agent tooling, our default-deny kernel network policy, our credential handling, the whole stack, and publish the resulting fingerprint. If the reproduction holds up, we'll also open a PR against the taxonomy's `products.yaml` with `evidence_level: verified` instead of the default `docs`, since vendor self-assessments backed by probe output are explicitly what the project asks contributors to submit. That's the actual next post: not a take, an experiment, with a scorecard at the end. ## By the Numbers - **1** — Thursday Thoughts hot take that got shelved mid-draft once I asked whether the definition already existed - **7-7-3** — the taxonomy's own shorthand: seven defense layers, seven threat categories, three evaluation dimensions - **26** — real products scored in the taxonomy's dataset, all as of its March 2026 v1.0 release - **5 days** — the gap between Hugging Face detecting the intrusion and OpenAI publicly connecting it to its own testing, the exact latency the taxonomy doesn't currently score - **3** — of seven layers we could confidently estimate from OpenAI's public disclosure; the other four are simply unknown - **2** — gaps I'd want fixed in v2: an incident-response/kill-switch layer, and a detection-latency sub-score - **1** — sandbox we're actually going to run the probe against next: our own - **0** — new standards invented in this post, on purpose === ## Thursday Thoughts: Curiosity, Not Skill, Is the Real AI Divide - URL: https://vibescoder.dev/posts/thursday-thoughts-curiosity-not-skill-is-the-real-ai-divide - Date: 2026-07-23 - Tags: #meta #building-in-public #vibe-coding #ai #future-of-coding - Reading time: 5 min read Six months of running a side project with AI agents convinced Rob Whiteley that the real dividing line in the AI era isn't technical background or access to tools. It's curiosity, and that reframes who actually gets left behind. --- Six months ago I started maintaining this site as a side hustle. Two to three hours a week, mostly nights and weekends, including all the hobbyist content around it. Not a lot of time. And yet somewhere in that process I started noticing something I didn't expect: I was actually learning software engineering. Not in a "here are the fundamentals of computer science" kind of way. More like the way you learn things when you're on the job and something breaks and you have to figure out why. Except compressed. Weirdly, uncomfortably compressed into what should have been a pretty shallow experience of just poking at an AI until a website works. That tension is worth pulling on. ## The Sharp Edges Show up Fast When you're actually maintaining an app, even a simple one, you start finding bugs. Some are obvious. Some are hiding. And what you begin to realize is that every app has these sharp edges, places where things can go wrong, where attack surfaces open up, where assumptions you made at the start turn out to be wrong. As a working software engineer, you'd know this intuitively because you've spent years pattern-matching against exactly these situations. As a vibe coder, you find out the same way. You just find out faster. I've been talking to my agent in plain language: "Hey, can you check to make sure this is actually doing what I think it's doing?" And what happens is the agent goes and builds a test harness, runs a scan, looks for the thing I was vaguely worried about, [the same discipline that turns into an actual post here](/posts/auditing-the-surface-we-added-since-the-last-audit) most weeks. The discipline is baked in. I don't have to know the name of the methodology. I just have to have enough awareness to ask the question. That's a real shift. The abstraction isn't just "natural language instead of code." It's natural language instead of years of accumulated disciplinary knowledge about how to check your work. ## Watching the Agent Teaches You How to Think Here's the part that surprised me most. Because I can see what the agent is doing, step by step, I'm actually learning. I'm learning how to decompose a problem into smaller chunks. I'm learning which kinds of tasks the agent handles well and which ones it fumbles. I'm developing intuitions about where things might go sideways before I ask it to look. Those intuitions feel like software engineering instincts. Not fully formed ones. But genuine ones. The kind that would have taken me years to build through traditional on-the-job experience. It's also teaching me to think across disciplines simultaneously. Security, testing, architecture, code quality: these used to be distinct specializations that people spent careers developing. Now I'm getting exposure to all of them at once because my agent is navigating all of them at once, and I'm watching it do it. ## The AI Have-Nots Gap Is Really a Curiosity Gap I've started thinking about who's getting left behind as AI accelerates, and I don't think the dividing line is what most people assume. It's not technical background. It's not access to tools. It's curiosity. It's the same reframing behind [a finance intern spending her summer vibe coding automation instead of shadowing someone](/posts/thursday-thoughts-every-intern-is-a-builder-now): the on-ramp was never the CS degree, it was the willingness to jump in. If you're willing to jump in, to tinker, to accept that you're going to hit bugs and weird edge cases and moments where you genuinely don't know what just happened, then AI gives you this hyper-abbreviated version of on-the-job learning. You pick up skills fast. You develop judgment. You start building things that would have been out of reach. If you're not curious, if you're waiting for AI to feel safe and obvious and simple before you engage with it, the technology is moving past you. Not because you're incapable. Because you're not in motion. I think about this a lot when people ask me whether AI is going to displace jobs. My honest answer is: not the way most people fear. What I think we're actually entering is a period of massive expansion in how much software gets built, and who builds it, and what kinds of problems get solved. The unlock isn't replacing engineers. It's making it possible for someone like me, spending two hours a week on a side project, to develop real engineering intuitions through practice. --- That's a different story than the one about displacement. It's a story about democratization. Skills that used to require a computer science degree or years of mentorship are becoming accessible through curiosity and a willingness to mess around and pay attention to what happens. I don't think that means the craft of software engineering stops mattering. If anything, watching my agent work has made me more interested in the fundamentals, not less. But the on-ramp has changed dramatically. You just need to jump in. *Are you learning things from your AI tools that you didn't expect to learn?* ## By the Numbers - **6 months** of running this site as a nights-and-weekends side hustle - **2–3 hours/week** — the time budget behind everything the blog covers - **2 companion posts** cited as the same pattern in practice: the finance-intern post and the ongoing security-audit habit - **1 real shift** the whole post argues for: natural language replacing years of accumulated disciplinary knowledge, not just replacing code === ## Has This Blog Been a Waste of Time? (I Made My Coding Agent Investigate) - URL: https://vibescoder.dev/posts/has-this-blog-been-a-waste-of-time - Date: 2026-07-21 - Tags: #homelab #ai #llm #benchmark #model-showdown #building-in-public - Reading time: 7 min read A challenge to a coding agent: prove whether months of homegrown test harnesses and bakeoff scripts were reinventing wheels that already exist. A survey of what hobbyists, techtubers, and the AI benchmarking industry already publish, and an honest verdict on this blog's own tooling habits. --- I gave my coding agent an uncomfortable assignment this week: go find out if we've been wasting our time. Every Model Showdown round on this blog runs on tooling we wrote ourselves — a bakeoff harness, a scoring rubric, [`thermal-test.sh`](/posts/comfyui-lemonade-and-localai-scouting-the-next-wave-of-homelab-ai-tools), ad hoc token counting glued together across a dozen posts. My coding agent is also my research partner for all of it. Both of us are, structurally, coders. And I've started to wonder whether that's a bias, not a strength: if the tool you're best at is writing code, every problem you're handed starts to look like a problem you solve by writing more of it. Want to test consumer AI hardware? Build a harness. Sure, we repurpose libraries here and there, but we're still stitching it all together by hand, every time, instead of asking whether someone already solved this. So the assignment was research, not code: go find out what hobbyists — the Linus Tech Tips / Gamers Nexus / Level1Techs crowd — and the broader AI benchmarking world already publish. Then tell me honestly whether we should have been using any of it instead of building our own. ## What's Already Out There **The hardware-review houses.** [LTT Labs runs MarkBench](https://github.com/LTTLabsOSS/markbench-tests), an orchestration and data-collection framework whose actual test harnesses — the same code that generates the numbers in LTT videos — are open-sourced on GitHub and updated on a quarterly cadence. It's built for GPUs and games, not LLMs: one harness scripts PyAutoGUI to click through a menu, another runs an OCR service just to find text on screen. [Gamers Nexus](https://gamersnexus.net/features/living-doc-current-test-bench-hardware-list-methodologies) takes the opposite approach to the same problem — not a reusable framework, but a public living document of exactly which SOPs, test benches, and settings produced which chart, updated review by review. Neither one has touched an LLM workload. And GN gave me the best counter-argument to my own thesis before I'd even finished asking the question: in 2025 they [published a whole new measurement methodology](https://gamersnexus.net/gpus-gn-extras-cpus/problem-gpu-benchmarks-reality-vs-numbers-animation-error-methodology-white) — "animation error" — because framerate and frametime testing, the industry standard for a decade, still didn't capture what a stutter actually felt like to a player. Even the best-resourced reviewers in the business sometimes conclude nothing existing measures what they need, and build something new. That's not automatically a bias. Sometimes it's just correct. **The vendor-neutral AI-PC suites.** [MLPerf Client](https://mlcommons.org/working-groups/benchmarks/client/), built by MLCommons with AMD, Intel, Microsoft, NVIDIA, and Qualcomm all at the table, is now on [v1.6](https://mlcommons.org/2026/04/mlperf-client-v1-6/) and measures how a Windows, macOS, or Linux client handles real generative AI tasks like summarization and content creation. [UL's Procyon AI Text Generation Benchmark](https://benchmarks.ul.com/procyon/ai-text-generation-benchmark) does the enterprise-press version of the same thing: seven fixed prompts against Phi-3.5-mini, Mistral-7B, Llama-3.1-8B, and Llama-2-13B, license required. Both are real, credible, and completely useless for us — fixed model rosters that don't include a single model we actually run, and both measure single-turn inference, not whether an agent can hold a task together across fifty tool calls. **The tools built for exactly our stack.** [`llama-bench`](https://github.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/README.md) ships inside llama.cpp itself and is the de facto community standard for raw prompt-processing and token-generation speed. [`llama-benchy`](https://github.com/eugr/llama-benchy) extends that same measurement style to any OpenAI-compatible backend, which is exactly why [I flagged it as a clean win](/posts/comfyui-lemonade-and-localai-scouting-the-next-wave-of-homelab-ai-tools) back in July and then never actually wired it into a bakeoff. [OpenBenchmarking.org](https://openbenchmarking.org/test/pts/llama-cpp) crowdsources llama.cpp results from anyone running the Phoronix Test Suite, opt-in, auto-rerunning until the variance settles. None of these three measure anything beyond throughput. None of them were ever going to replace the rubric. But none of them needed to — they're a layer underneath it that we skipped. **Our actual peer group.** A [Level1Techs forum member built a Strix Halo LLM benchmark harness](https://forum.level1techs.com/t/strix-halo-ryzen-ai-max-395-llm-benchmark-results/233796) from scratch, ran rigorous sweeps against the newest MoE architectures, and checked the full results into GitHub for the community to pick apart. That's the same move this blog makes every few weeks. It's worth saying plainly: "one hobbyist builds their own harness and publishes it" isn't a personal failing of mine or my agent's habits. It's the normal, respected way this exact community operates. **The layer that actually matters to us.** Model Showdown isn't a throughput test — it's an agentic-correctness test, and that field looks different. [Aider's Polyglot benchmark](https://aider.chat/docs/leaderboards/) scores models on 225 of the hardest Exercism exercises inside Aider's own structured edit loop, with community-contributed results merged by PR. [SWE-bench Verified](https://www.vals.ai/benchmarks/swebench) grades a patch against a real GitHub issue and a human-validated test suite, though a notable complexity is that it evaluates the agentic harness and the underlying model together, which is exactly why different labs report different numbers for the same model. Terminal-Bench and METR's RE-Bench push further into containerized, tool-using, long-horizon work. A recent [survey of the post-SWE-bench landscape](https://www.appliedtechnologyindex.com/research/2026-comparative-analysis-coding-agent-evaluation-harnesses-after-swe-bench/) put it about as bluntly as I'd put it myself: evaluation is moving toward harness portfolios that test repository repair, terminal execution, and tool-use reliability together, and any single benchmark is one signal, not the whole basis for a decision. ## The Honest Answer Split it in two, because the two halves of our own stack don't get the same grade. **The throughput layer: yes, we reinvented a wheel, and we knew it.** `llama-bench` and `llama-benchy` already solve "how fast does this model run on this hardware," drop onto our existing endpoint with zero infrastructure change, and I identified `llama-benchy` specifically as a clean win in [a research post back in July](/posts/comfyui-lemonade-and-localai-scouting-the-next-wave-of-homelab-ai-tools) — and then the next Model Showdown round still reported informal token counting instead. That's not a close call. That's the bias, caught in the act, on our own blog. **The agentic layer: no, not cleanly, and it's not just stubbornness.** Aider Polyglot is the closest existing match to what [Model Showdown](/posts/model-showdown-round-9-qwen-3-6-27b-vs-qwen-3-6-35b-a3b-vs-qwythos-9b-vs-glm-4-7-flash-vs-nemotron-3-nano) does, and it's still testing something narrower: isolated Exercism puzzles in a sandbox, one edit-and-retry loop, no real repo, no git, no Playwright, no fifty-turn session that can quietly go sideways. Terminal-Bench is closer in spirit but isn't wired to a Coder Agents chat and doesn't know our rubric. Adopting either wholesale would have meant building nearly as much integration code as our own harness already required, in exchange for a narrower signal. This is the same shape as [the security audit work](/posts/closing-the-loop-from-audit-to-ten-commits) that's become a habit on this blog: sometimes the "buy vs. build" answer really is build, and the honest version of that answer names the specific reason instead of just defaulting to it. ## What's Next That split is a claim, not just a vibe, and claims like that should be checked with data instead of left as a nice paragraph. So the next step is an actual experiment: run `llama-bench`/`llama-benchy` against our own ad hoc timing on the same model, run a slice of Aider's Polyglot benchmark against two models from a recent Model Showdown round side by side with our own rubric, and keep an honest ledger of how much setup time each approach cost — because if wiring up someone else's harness takes longer than just running another homegrown round, that's evidence for the bias too, independent of whose numbers turn out to be more useful. I'm predicting the existing tools win the throughput half outright and the homegrown rubric keeps finding things Polyglot structurally can't. I could be wrong about that. Next post will say so either way. *Back to the thesis. More soon.* ## By the Numbers - **18 sources** cited while researching whether the blog's homegrown tooling was reinventing wheels - **225 exercises** — the size of Aider's Polyglot benchmark, the closest existing match to what Model Showdown does - **7 fixed prompts** — all UL's Procyon AI Text Generation Benchmark runs, against just 4 fixed models (Phi-3.5-mini, Mistral-7B, Llama-3.1-8B, Llama-2-13B) - **5 companies** — AMD, Intel, Microsoft, NVIDIA, and Qualcomm — co-building the MLPerf Client suite, now on v1.6 - **50-turn** — the length of agent session Model Showdown needs to survive, longer than any benchmark surveyed actually tests - **2 verdicts** — reinvented wheel on the throughput layer (yes), justified build on the agentic layer (no) === ## Auditing the Surface We Added Since the Last Audit - URL: https://vibescoder.dev/posts/auditing-the-surface-we-added-since-the-last-audit - Date: 2026-07-20 - Tags: #security #agents #building-in-public #meta - Reading time: 9 min read The May audit closed clean. Since then we shipped an MCP server, a Slack integration, and a shareable-snippet image generator -- three new pieces of attack surface that postdated every finding in that audit. A fresh scan against the same categories found 8 issues across the old surface and the new. Three phases, three commits, about an hour, one repo. --- Staying hyper-vigilant doesn't come with a calendar reminder that tells you when to renew it. The May audit ended with a clean bill and a note: "next scheduled review, August 8, 2026." I didn't wait for August. Two months of shipping later, I asked for a fresh scan against the same categories that audit used, and the result was smaller than [the first one](/posts/closing-the-loop-from-audit-to-ten-commits) but not nothing: 8 findings, three of them in code that didn't exist in May. That's the actual headline here, more than any individual bug. The first audit covered the blog as it stood in early May. Since then we shipped an MCP server (16 tools, bearer-token auth, its own rate limiter), a Slack `/todo` integration, and a shareable-snippet image generator for code blocks and tables. None of that existed when the last audit ran, which means none of it had ever been looked at with a security lens. A codebase doesn't need to get worse to need a second look. It just needs to get bigger. [Friday's post](/posts/friday-fixes-the-hyper-vigilance-tax) named this the hyper-vigilance tax and paid it on the UX axis: three small bugs, each hiding in a corner some check didn't cover. This post pays the same tax on the other axis, the one that doesn't announce itself in a screenshot and doesn't get noticed by accident. Different axis, same bill. ## What the Scan Covered Same shape as May: auth and middleware, every API route, GitHub Actions workflows in both repos, `npm audit`, and a secrets/PII grep across both repos. The difference this time was scope creep in a good way — three routes that simply didn't exist during the first pass got the same scrutiny as the routes that did. Eight findings, roughly matching the severity spread from last time: - 1 high: a transitive dependency with five stacked advisories - 5 moderate: the rest of a dependency chain, plus two independent route-level gaps - 2 low: a dev-only dependency advisory, and a documented (not code) tradeoff ## Phase 1 Dependency Bumps `npm audit` on a real `npm install` (not a stale lockfile check) turned up 8 vulnerabilities: 1 high, 6 moderate, 1 low. The high was `hono@4.12.22`, pulled in transitively through `@modelcontextprotocol/sdk`, which the MCP server depends on. Five stacked advisories on that one package: a CORS-reflects-any-origin-with-credentials bug, a body-limit bypass on Lambda-style deployments, a path-traversal issue in `serve-static` on Windows, and two adapter bugs that silently drop cookies or headers. `npm audit fix` — the non-forcing kind — resolved all of it except one chain: `hono` jumped to `4.12.30` (patched), and `brace-expansion`, `js-yaml` (including the copy that `gray-matter` depends on, which matters because `gray-matter` parses every post's frontmatter in production), and `@babel/core` all resolved within their existing semver ranges. What's left is the exact same false positive the May audit documented: `postcss <8.5.10`, bundled inside `next` itself, not the `@tailwindcss/postcss` copy (already patched). `npm audit fix --force` proposes downgrading `next` to `9.3.3` to fix it — a multi-year regression that would break considerably more than it fixes. I checked whether a newer Next.js release had bumped its bundled `postcss`; even the `16.3.0` preview builds haven't. Same conclusion as May: accepted, monitored risk, no fix available yet that isn't worse than the bug. One commit. `tsc --noEmit` clean afterward. ## Phase 2 Three Independent Route Gaps None of these three depend on each other, so they went into one batched commit — the same logic the last audit used for its Phase 2. **The MCP route's rate limiter was in-memory.** `const buckets = new Map()`, keyed by IP, capped at 120 requests/minute. That looks reasonable until you remember Vercel runs serverless functions across multiple instances. Each cold start gets its own empty `Map`. A client that happens to land on five different instances effectively gets five times the stated limit — the number on the tin was never the number in practice. Swapped it for the same Redis-backed limiter the login and analytics routes already use, which is shared across every instance regardless of which one handles a given request. **`/api/share-image` had no bounds and no rate limit.** This route is intentionally public — it's called client-side to render shareable images of code blocks and tables for social sharing, so it can't sit behind the admin-session middleware. But it accepted an arbitrary-length `content` string and computed the output image's height with no ceiling; only width was capped. That's a real rendering-cost DoS sitting in the open: no authentication, no throttling, no size limit, on an endpoint whose cost scales with attacker-supplied input. Added a per-IP rate limit (20/minute via the same Redis limiter), a 20,000-character content cap, a 200-row table cap, and a 4,000px height ceiling to match the existing width cap. **Both Dev.to syndication routes skipped slug sanitization.** `content/posts/${slug}.mdx` with a raw, unsanitized `slug` straight from the request body, feeding into a GitHub Contents API path. Every other route touching the same file space — post editing, image upload, every MCP tool — validates the slug first. These two just got missed, probably because they were added after the pattern was established elsewhere and nobody thought to check whether they'd inherited it. Since three separate files each had their own copy-pasted `sanitizeSlug()`, and a fourth and fifth were missing it entirely, I pulled the function into `src/lib/slug.ts` once and pointed all five call sites at it. Low impact today, since both routes sit behind admin-session middleware, but it's the kind of inconsistency that becomes a real path-traversal bug the moment the trust boundary shifts even slightly. Verified with `tsc --noEmit`, `eslint` on every changed file, and a full production build. One unrelated finding surfaced during lint: `share-image/route.tsx` has 10 pre-existing lint errors (JSX constructed inside a try/catch, which ESLint's `react-hooks/error-boundaries` rule flags because React doesn't actually catch render errors that way). Confirmed via `git stash` that they predate this change entirely — not something to silently fix inside a security commit, so it's noted here and left for its own pass. ## Phase 3 Finishing What Report-Only Started The May audit's Phase 2 shipped Content-Security-Policy in Report-Only mode on purpose, with a plan to observe for about a week, then flip to enforcing. That flip never happened. It sat in Report-Only for two months. Here's the part that made this an easy call: there's no `report-to` or `report-uri` directive configured anywhere in the policy. Report-Only mode without a reporting endpoint doesn't collect anything except what shows up in an individual visitor's own browser console — which nobody but the site owner would ever open, and even then only by accident. The "observe for a week" plan had no mechanism to observe anything. Two months of waiting produced exactly as much signal as two minutes would have. So: flipped the header from `Content-Security-Policy-Report-Only` to `Content-Security-Policy`, same directive set, unchanged since May. Rather than trust the diff, I ran a full production build, started the built server, and `curl -I`'d the homepage to confirm the actual response header. It came back exactly as expected — enforcing, same values. This one got its own commit, isolated from Phase 2, because it's a global behavior change with real blast radius if a directive gap exists that Report-Only never had the means to catch. Easy to revert on its own if something breaks that two months of silent Report-Only never revealed. ## What Got Deferred One item, carried forward rather than fixed: the rate limiter (shared across login, analytics, MCP, and now share-image) fails open if Upstash Redis is unreachable or misconfigured. That's a deliberate, documented tradeoff from when the limiter was first built — a preview environment without Redis configured shouldn't get locked out of its own login page. It's still the right tradeoff. But it means a silent Redis misconfiguration in production would silently remove every rate limit on the site with no alert. Not a code fix; a monitoring gap. Noted for whenever alerting gets built out, the same way the last audit deferred CSP nonces and build-time markdown rendering with reasons instead of silently dropping them. ## What This Says About Auditing Cadence The real lesson isn't in any individual bug. It's that "audit the codebase" isn't a checkbox you tick once. The MCP server, the Slack integration, and the share-image route all shipped in the two months between audits, each one reasonably reviewed on its own merits at the time, and none of them got the systematic security pass the rest of the codebase got in May — because that pass had already happened before they existed. A quarterly audit cadence, which is what the May report suggested, assumes the codebase's rate of change is roughly constant. For a one-person-plus-agent blog shipping new integrations every few weeks, three months is long enough for entire new subsystems to exist unaudited. That's the same structural blind spot [a very different audit](/posts/your-ai-strategy-has-a-blind-spot) found in this blog's SEO and AEO tooling months ago: a checker that only looks where it was built to look will miss whatever got added after it was written. The actual cadence that matches this project isn't a calendar date. It's "whenever a new API route ships that talks to the outside world," which in practice has been happening faster than the calendar suggested. ## By the Numbers - **8** findings this pass, versus **90+** raw findings (deduplicated to 15) in May — this audit had one scanner instead of three, and a much smaller diff to cover - **3** of 8 findings were in code that didn't exist during the May audit (MCP rate limiter, share-image bounds, and the Dev.to slug gap, added alongside newer syndication tooling) - **5** stacked advisories on a single transitive dependency (`hono`), resolved by one non-forcing `npm audit fix` - **8 → 4** vulnerabilities after Phase 1, all four remaining from the same root cause (`postcss` bundled inside `next`) - **3** phases, **3** commits, **1** repo — no content-repo changes this round - **0** new dependencies added to fix anything - **~1 hour** end to end, versus **4 hours** for the May audit's ten commits across two repos - **2 months** a CSP policy sat in Report-Only mode with no reporting endpoint configured, collecting zero actual violation data - **1** shared `sanitizeSlug()` replacing **3** copy-pasted versions and closing **2** missing ones - **10** pre-existing lint errors found, confirmed unrelated via `git stash`, and left alone rather than folded into a security commit - **1** deferred item, carried forward with a reason, not silently dropped === ## Friday Fixes: The Hyper-Vigilance Tax - URL: https://vibescoder.dev/posts/friday-fixes-the-hyper-vigilance-tax - Date: 2026-07-17 - Tags: #meta #building-in-public #agents #debugging #vibe-coding - Reading time: 11 min read Building an app across dozens of disconnected agent sessions accumulates bugs. That's the tax on staying hyper-vigilant across two axes at once: does it work the way a person experiences it, and does it hold up against someone trying to break it. This week split cleanly into both. Here are the four small bugs, each hiding in a corner some check didn't cover. Monday's post covers the security axis. --- Building an app across dozens of disconnected agent sessions accumulates bugs. That's not a failure of the process, it's the process working as advertised. Each session solves the problem in front of it, ships, and moves on. Nobody's holding the whole system in their head across sessions the way one long-tenured engineer might. What slips through the gaps isn't any single session's fault. It's the tax you pay for building this way, and the only way to keep the bill small is staying hyper-vigilant, on two different axes at once, indefinitely, because no session ever hands the watch off to the next one. The first axis is UX: does the thing actually work the way a person experiences it, not the way a browser's devtools simulates it. The second axis is security: does it hold up against someone trying to make it fail on purpose, not just against someone using it the way it was intended. Vibe coding doesn't get you out of watching both. It just changes who's watching, and how often you have to look. This stretch of days gave me a clean split between the two. Four small bugs on the UX axis, all of them hiding in a corner some check didn't cover. And, separately, a deliberate pass on the security axis that turned up eight more things, three of them in code that didn't exist the last time anyone looked at the codebase that way. This post is the UX side. [Monday's post](/posts/auditing-the-surface-we-added-since-the-last-audit) is the security side, phases and commits and all. Different axis, different post, same underlying discipline: don't assume yesterday's check still covers today's code. Four bugs this round, none of them dramatic on their own. What connects them is more interesting than any single fix: each one passed a real check and failed on the one dimension that check didn't cover. A layout that looked right in devtools. A relative-time label that did real date math, just the wrong kind. An orphan-detection tool that scans exactly half the places an image can be referenced from. A systemd service that only knew how to survive the kind of change it was written for. Same shape every time — the corner nobody checked is the corner where the bug was hiding. This is the same territory I keep coming back to in this series: [defenses that feel complete but aren't](/posts/friday-fixes-the-fix-that-wasnt), [bugs invisible until a specific condition exposes them](/posts/friday-fixes-two-bugs-one-workflow), and [failures with no error message anywhere in the chain](/posts/invisible-failures-the-bugs-that-hide-in-plain-sight). Different bugs, same lesson repeating itself. ## 1 the Draft Card That Only Worked in a Wide Window **The problem:** Each card on `/admin/drafts` laid out post metadata and three action buttons (Unschedule / Edit / Publish) in a single `flex items-start justify-between` row. On an actual iPhone, the buttons didn't have room to share that row with the content, and the layout squished and overflowed. **The fix:** Two lines in `DraftsList.tsx`. Stack the card vertically on mobile, revert to side-by-side above the `sm` breakpoint: ```diff -
+
``` Plus `flex-wrap` on the actions container as a second line of defense. **The corner nobody checked:** Browser devtools' narrow-viewport mode. It renders at the right *width*, but it doesn't reproduce real mobile font rendering, tap-target sizing, or how three buttons with real labels actually wrap. The desktop-simulated-as-mobile view looked fine. The actual phone didn't. This is the exact lesson from [Mobile First and the Skill That Saved Us](/posts/friday-fixes-mobile-first-and-the-skill-that-saved-us) two months ago — the skill file has the pixel math now, but pixel math doesn't catch every layout shape, and this one slipped through anyway. Worth a real phone check any time buttons and content share a flex row, full stop. ## 2 the Label That Used the Wrong Clock **The problem:** The drafts page showed a scheduled post as "publishes Jul 13, 2026 (today)" — while it was still July 12 in the browser's local timezone. The date itself was correct. The `(today)` next to it wasn't. **The cause:** `relativeTime()` in `DraftsList.tsx` computed the label by diffing raw milliseconds between now and the target, then rounding to the nearest day: ```ts // before const diffMs = target.getTime() - now.getTime(); const diffDays = Math.round(diffMs / (1000 * 60 * 60 * 24)); ``` A post scheduled for 4 AM the next day is only 8–12 hours away in the evening. Divide that by 24 hours and round, and you get 0 — "today" — even though the calendar day hasn't turned over yet in the viewer's timezone. Meanwhile `formatDate()`, rendering the actual date right next to it, was computing calendar fields correctly. Two functions, same UI row, two different definitions of "what day is it." **The fix:** Zero both dates to local midnight before diffing, so the comparison is calendar days, not elapsed time: ```ts const startOfDay = (d: Date) => new Date(d.getFullYear(), d.getMonth(), d.getDate()).getTime(); const diffDays = Math.round( (startOfDay(target) - startOfDay(now)) / (1000 * 60 * 60 * 24), ); ``` Merged as [PR #22](https://github.com/carryologist/the-vibe-coder/pull/22). **The corner nobody checked:** Anything that only visibly disagrees near a day boundary. `relativeTime()` and `formatDate()` had been sitting next to each other rendering the same post for weeks, agreeing with each other every single time, right up until a scheduled post happened to fall in that 8–12 hour evening window. That's the same failure shape as the [unquoted-date bug from Friday Fixes #2](/posts/friday-fixes-the-unquoted-date-that-broke-drafts) — one function handling dates one way, an adjacent one handling them another way, and nothing forcing them to agree until the specific edge case that exposes the gap. ## 3 the Headshot That Was Deleted as an Orphan This one's the interesting one, because it wasn't a bug in new code. It was a bug in a *cleanup* tool, and it sat undetected for 77 days. **The symptom:** The [About page](/posts/your-ai-strategy-has-a-blind-spot) rendered a broken-image icon where the headshot should be. Not a recent regression — this has apparently been broken since spring. **The cause:** Back on April 30, 2026, a housekeeping pass deleted `public/images/IMG_9133.jpeg` as "orphaned, unreferenced." The admin's orphan-detection logic in `src/lib/images.ts` only scans *subdirectories* of `public/images/`, matching each directory's slug against post slugs and post content — the same 3-tier match (exact / prefix / content) the `/admin/images` page uses today. `IMG_9133.jpeg` was a loose file sitting directly at the top level of `public/images/`, referenced only from a hardcoded `` in the About page's static `page.tsx`. It's not in a post. It's not in a slug directory. The detector never had a chance to see it as in-use, in either direction — it wouldn't have shown up in the admin UI to flag as orphaned *or* as matched. Whatever process ran that April cleanup just saw a stray file nobody's tooling could vouch for and deleted it. **The fix:** Recovered the original file intact from git history (`2dec8d1^`, pre-deletion), moved it into `public/images/branding/` alongside the site's other non-post assets — logo, favicons — instead of leaving it as a loose top-level file, and updated the About page's `src` path to match. ```diff -src="/images/IMG_9133.jpeg" +src="/images/branding/IMG_9133.jpeg" ``` **The corner nobody checked:** Static pages. The entire orphan/match system in `images.ts` reasons about one thing — MDX post content — and the About page isn't a post, it's a hand-written React component. Any image referenced only from `src/app/**/page.tsx` is structurally invisible to a detector built around post slugs. I logged this gap in `TODO.md` rather than just patching around it once: either the detector needs to also grep static `.tsx` pages for hardcoded `/images/...` paths, or `public/images/` needs a hard rule that it never holds loose top-level files. Until one of those ships, this exact failure mode can recur with any other hardcoded image reference sitting outside the post system — the same blind spot [the SEO/AEO audit](/posts/your-ai-strategy-has-a-blind-spot) found in a different tool checking a different half of the site. ## 4 the Magic Packet Sent to an Interface That No Longer Existed This one isn't the blog app at all — it's the homelab itself, [the workstation these agent sessions run on](/posts/qol-with-wol-turning-on-the-homelab-from-anywhere). Small bug, same shape, worth the four sentences. **The problem:** After migrating the homelab to new hardware, Wake on LAN — the SmartThings-to-magic-packet chain from a few months back — stopped waking the machine. Voice command fired, hub still sent the packet, nothing happened. **The cause:** The `wol-enable.service` systemd unit that re-arms WoL on every boot had the old motherboard's interface name, `enp8s0`, hardcoded. The new NIC enumerates as `enp112s0`. The service had been failing silently on every boot since the migration — `systemctl status` showed `failed`, `netlink error: no device matches name`, sitting there unread for hours. **The fix:** Point the unit at the real interface: ```diff -ExecStart=/sbin/ethtool -s enp8s0 wol g +ExecStart=/sbin/ethtool -s enp112s0 wol g ``` Plus the matching manual step the systemd fix can't reach: updating the MAC address in the SmartThings vWOL device's settings, since the hub was faithfully broadcasting a magic packet addressed to a card that no longer exists. **The corner nobody checked:** a systemd service that only reports failure in a log nobody was tailing. It existed specifically because I'd been burned by this once before — the original post's whole "make it persist across reboots" step — and it still didn't survive the *next* kind of change instead: not a reboot, a hardware swap. Built to survive one axis of change, blind to the other. ## What Connects Them All four bugs are the same shape wearing different clothes: a check that covers most of the surface area and misses exactly the part where the failure lives. The mobile layout was tested at the right *width* but the wrong *fidelity* — devtools simulates the viewport, not the device. The relative-time label was tested against the *common* case, where the elapsed-time math and the calendar-day math happen to agree, which is true right up until the last few hours before midnight. The orphan detector was tested against *posts*, because posts are the thing the admin UI is built to manage, and a static page sitting one directory away in the same repo simply never entered its field of view. The Wake on LAN service was tested against the one kind of change it was written to survive — a reboot — and simply never had to prove itself against the other kind, a hardware swap that renamed the thing it depended on. None of these are exotic bugs. They're all the same failure mode I keep writing about in this series: the fix that only covers the layer someone happened to be looking at. The difference this round is how long one of them sat there. A layout bug on an admin-only page gets noticed in a day. A wrong "today" label gets noticed within a week, because someone's staring at the drafts list constantly. A photo on a page nobody but visitors look at can sit broken for 77 days, because the person who'd notice it — me — doesn't visit my own About page. That's maybe the actual lesson this week: the bugs that survive longest aren't the scary ones. They're the ones on pages you built once and never look at again. That's the UX side of the hyper-vigilance tax, and it's the cheaper of the two to pay, because a broken layout or a wrong label eventually shows up somewhere a human looks. The security side doesn't get that luxury. Nothing renders wrong when an endpoint is missing a rate limit or a dependency has a stacked CORS bypass sitting three layers deep in a transitive dependency. Those don't announce themselves in a screenshot. They just sit there until someone goes looking on purpose, which is the more expensive half of the same bill. [Monday's post](/posts/auditing-the-surface-we-added-since-the-last-audit) is that deliberate look, a fresh security pass against the same categories the last full audit used, covering the surface that's shipped since, three phases, three commits, one repo. Same discipline as this post, pointed at the axis where nobody stumbles into the bug by accident. ## By the Numbers - **2 lines** changed to fix the mobile draft card overflow - **~2 minutes** to fix the layout once the real device confirmed it - **~15 minutes** to diagnose and fix the relative-time timezone bug, [PR #22](https://github.com/carryologist/the-vibe-coder/pull/22) - **8–12 hours** — the evening window where the old elapsed-ms math could mislabel "tomorrow" as "today" - **77 days** the About page headshot sat broken before anyone noticed (Apr 30 → Jul 16) - **1** file recovered intact from git history, zero data lost - **1** hardcoded interface name (`enp8s0` → `enp112s0`) that silently broke Wake on LAN across a hardware migration - **3** repos/systems touched across all four fixes (2 code repos, 1 physical machine) - **4** bugs, **4** different checks, **1** shared failure mode - **0** fodder files left unconsumed after this post === ## Thursday Thoughts: FOCUS and the True Cost of a Token - URL: https://vibescoder.dev/posts/thursday-thoughts-focus-and-the-true-cost-of-a-token - Date: 2026-07-16 - Tags: #ai #agents #future-of-coding #homelab - Reading time: 12 min read The Linux Foundation just launched the Tokenomics Foundation to extend FOCUS — the FinOps billing spec that normalized cloud cost data — into token-based AI spend. It's the cloud-native parallel playing out again, except this time every knowledge worker who touches an agent is about to become a cost center. Here's what FOCUS 1.4 actually standardizes, what it still can't see on self-hosted infrastructure, and why we're going to try to run it on a homelab anyway. --- By day, I run a software company. That means my professional and personal lives intersect a lot. This week was one of those moments. My Finance team was presenting a case to join the [Tokenomics Foundation](https://itsfoss.com/news/tokenomics-foundation/) and a request to implement the [FOCUS spec](https://focus.finops.org/) in both our internal systems and external product. I was vaguely familiar with both, but felt much smarter after a 30-minute debate. But I can't stop there. I need to think about how this will affect the industry, employees, and — well — me. I keep coming back to the [cloud-native analogy](/posts/thursday-thoughts-how-ai-native-mirrors-cloud-native) for AI, and this week it clicked again in a place I wasn't expecting: FinOps. On June 3rd, the Linux Foundation [announced the intent to launch](https://itsfoss.com/news/tokenomics-foundation/) the **Tokenomics Foundation**, a new body dedicated to open standards for AI cost management, in close partnership with the FinOps Foundation. The first concrete deliverable is [extending **FOCUS**](https://www.cio.com/article/4182274/linux-foundation-targets-ais-cost-management-problem-with-tokenomics-foundation.html) — the FinOps Open Cost and Usage Specification, the schema that already normalizes cloud billing across AWS, Azure, and GCP — to cover token-based AI spend. Twelve organizations, [including Google Cloud, Microsoft, Oracle, Salesforce, SAP, and JPMorgan Chase](https://itsfoss.com/news/tokenomics-foundation/), are already backing it. Here's why that matters to me, and why I think it should matter to you even if you've never opened a FinOps dashboard in your life. ## We Took a Decade to Get Serious About Cloud Cost. We Don't Get That Long This Time. The cloud era ran for years before "FinOps" became a real discipline with real standards. Chargeback and showback were ad hoc. Every cloud provider invented its own billing schema, and practitioners built bespoke ETL pipelines to make AWS, Azure, and GCP cost data speak the same language. FOCUS didn't [formally exist as a Linux Foundation project until January 2023](https://focus.finops.org/focus-specification/v1-1/) — more than fifteen years after AWS launched EC2. We built the discipline of consumption-based cost management *after* consumption-based cost had already spiraled out of institutional control, and I've written before about how [Anthropic is running a version of the exact same AWS-shaped playbook](/posts/thursday-thoughts-why-anthropic-is-the-next-aws-but-potentially-worse), just at a pace that makes the cloud era look slow. Token-based AI spend is following the exact same consumption-based cost curve, except compressed. [Global token usage is projected to grow 24x between 2026 and 2030, hitting 120 quadrillion tokens per month](https://itsfoss.com/news/tokenomics-foundation/). ![Bar chart showing global token usage growing 24x from a derived 2026 baseline of 5 quadrillion tokens per month to a projected 120 quadrillion tokens per month by 2030](/images/thursday-thoughts-focus-and-the-true-cost-of-a-token/token-growth-projection.png) *Chart: The Vibe Coder. 2030 figure and 24x growth multiple per Goldman Sachs research, as [cited by It's FOSS](https://itsfoss.com/news/tokenomics-foundation/); the 2026 baseline is derived by simple division and wasn't independently reported.* And unlike a vCPU-hour, which is a stable, well-understood unit, [a token is not a fixed unit at all](https://www.finops.org/insights/token-economics-the-atomic-unit-of-ai-value/) — different models tokenize the same text differently, and pricing, context windows, and caching behavior shift under you without notice. The fact that the industry is standing up FOCUS-for-AI *now*, three years into the LLM API era rather than fifteen, is a genuinely good sign. It means we're applying a lesson instead of relearning it from scratch. That's the whole thesis of this blog in miniature: don't lift-and-shift the old playbook onto AI, but do keep the parts of the old playbook that were hard-won and correct. Consumption-based cost governance is one of those parts. ## Why This Actually Matters to Vibe Coders Not Just FinOps Practitioners I don't write this blog for people managing million-dollar cloud bills. I write it because I think [every knowledge worker is about to become a vibe coder](/posts/thursday-thoughts-every-intern-is-a-builder-now), the same way every knowledge worker is already a spreadsheet user or a slide-deck builder. Building an internal tool, a workflow automation, or a small app is going to be a universal white-collar skill within a few years, not a specialist one. That's the premise this whole blog runs on. Which means the economics conversation happening in FinOps circles right now isn't going to stay contained to platform teams and CFOs. It's coming for every person who opens an agent chat and says "build me a dashboard." Once [vibe coding is universal](/posts/thursday-thoughts-every-intern-is-a-builder-now), token consumption becomes as distributed, as invisible, and as easy to blow past a budget on as cloud spend was in 2015 — except instead of one platform team provisioning EC2 instances, it's every employee with an agent tab open. The [State of FinOps 2026 survey found AI has become a mainstream technology investment](https://www.finops.org/topic/ai-value/), with 98% of FinOps teams now managing AI spend, up from just 31% two years ago. That number is going to keep climbing precisely because the *people generating* the spend are no longer engineers alone. The practical question for a vibe coder — hobbyist or enterprise employee alike — isn't "what does FOCUS mean for finance." It's "will my organization eventually meter me the way it metered a Kubernetes namespace." I think the answer is yes, and I think it should be, because the alternative is nobody knowing what any of this actually costs until the invoice arrives. ## What FOCUS Actually Is and Isn't FOCUS is not a logging or telemetry standard. It's a billing schema — closer to a standardized invoice format than to an observability trace. [It defines a common schema for technology cost and usage data](https://github.com/FinOps-Open-Cost-and-Usage-Spec/FOCUS_Spec) across cloud, SaaS, data center, and other technology categories, establishing a consistent, vendor-neutral vocabulary for billing and usage data. [A FOCUS dataset is a table of charges](https://focus.finops.org/what-is-focus/), where each row represents one charge, and every column has a defined name, data type, and meaning set by the specification, so the column means the same thing regardless of which provider produced the row. Concretely, it ships as CSV or Parquet exports ([AWS's CUR 2.0 can output FOCUS 1.2-formatted Parquet to S3](https://techcommunity.microsoft.com/blog/finopsblog/managing-azure-openai-costs-with-the-finops-toolkit-and-focus-turning-tokens-int/4413886)), normative language follows RFC 2119/8174 (MUST/SHOULD/MAY), and providers can extend it with `x_`-prefixed columns for proprietary detail without breaking the shared schema. There's even an [open-source validator](https://github.com/finopsfoundation/focus_validator) that checks a dataset against the spec version by version. (One housekeeping note: the FOCUS and FinOps Foundation logos are registered trademarks, so rather than reproduce them here, I'm linking straight to the [FOCUS brand site](https://focus.finops.org/) and the [FinOps Foundation media page](https://www.finops.org/about/media/) if you want the official marks.) ## The Evolution and Where 1.4 Landed FOCUS has moved fast for a standards body: - **[v1.0](https://focus.finops.org/focus-specification/v1-0/) (2024):** established the core schema — `BilledCost`, `EffectiveCost`, `ConsumedQuantity` — for Cloud Service Provider billing. - **[v1.1](https://focus.finops.org/focus-specification/v1-1/) (Nov 2024):** added invoice reconciliation and unit-cost/density metrics (cost-per-GB, cost-per-request). - **[v1.2](https://focus.finops.org/focus-specification/v1-2/) (May 2025):** unified Cloud + SaaS + PaaS reporting into one schema, and — notably for this post — introduced the first language around virtual currency and **token purchase pattern analysis**. - **v1.3 (Dec 2025):** added a dedicated Contract Commitment dataset and, critically, first-class shared-cost allocation fields — which resource was shared, who consumed it, and what method split the cost. - **[v1.4](https://focus.finops.org/focus-specification/) (ratified June 4, 2026, at FinOps X):** the current release. It adds 2 datasets, 47 columns, 6 attributes, 17 glossary entries, and 2 supported features, headlined by new Invoice Detail and Billing Period datasets for reconciling usage straight to invoices, and Service Provider vs. Host Provider columns that separate who sold you the resource from who's actually running it underneath, disambiguating reseller relationships. ![Timeline showing five FOCUS specification releases from v1.0 in 2024 through v1.4 ratified June 4, 2026, each annotated with its key additions](/images/thursday-thoughts-focus-and-the-true-cost-of-a-token/focus-version-evolution.png) *Chart: The Vibe Coder. Ratification dates and release details per the [FOCUS Specification changelog](https://focus.finops.org/focus-specification/), licensed [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/legalcode).* The AI-specific columns riding on top of that 1.4 release are the ones that matter most here: `ConsumedQuantity`, `ConsumedUnit`, `HostProviderName`, and the [`x_InputTokens` / `x_OutputTokens` / `x_CachedTokens` splits the spec introduced specifically for AI workloads](https://siliconangle.com/2026/06/08/ai-token-economics-focus-specification-updates-finopsx/). That's the schema-level answer to "how much did this model call actually cost, and for what." ## What FOCUS 1.4 Gets Right for AI Consumption If you're buying tokens from a frontier lab or a hyperscaler's managed AI service, FOCUS today gives you real, standardized ground to stand on: - **Cross-provider comparability.** Whether the bill comes from OpenAI, Anthropic, Azure OpenAI, or Bedrock, the token consumption shows up as [`ConsumedQuantity` in `ConsumedUnit: tokens`](https://techcommunity.microsoft.com/blog/finopsblog/managing-azure-openai-costs-with-the-finops-toolkit-and-focus-turning-tokens-int/4413886), the same way a compute charge shows up as vCPU-hours everywhere. - **Input/output/cached token attribution.** The `x_InputTokens`/`x_OutputTokens`/`x_CachedTokens` split lets you see where the money actually went, which matters given [output tokens typically cost 3–8x more than input tokens](https://www.finout.io/blog/ai-model-cost-breakdowns-the-complete-2026-comparison-guide) because generation is more compute-intensive than reading a prompt. - **Shared-cost allocation for chargeback.** The [1.3-era split-cost-allocation fields](https://focus.finops.org/focus-specification/v1-2/) — which resource was shared, which consumers used it, what method was used to split it — are exactly the mechanism enterprises need to charge back a shared model deployment or a shared GPU pool across teams, not just a shared VM. - **Amortization of commitments.** FOCUS already knows how to [spread a flat-rate subscription across daily consumption via an effective-cost column](https://siliconangle.com/2026/06/08/focus-specification-ai-cost-accountability-finopsx/), the same pattern that would apply to a reserved-capacity inference contract. ## Where It Still Falls Short for Self-Hosting This is the part that matters if you run your own models, and it's the part that isn't solved yet. I'm especially worried given the trend I see towards sovereign AI, which requires self-hosting. Let's explore. **If you self-host, there's no token to bill in the first place.** When you run an open-weight model on your own GPUs, [cost is expressed entirely in compute](https://www.finops.org/wg/token-economics-saas/) — GPU instance hours, storage, and networking — because there is no per-token charge. FOCUS handles that side fine; a GPU instance is just another compute row with a `BilledCost` and `ConsumedQuantity` in GPU-hours, same as any VM. What FOCUS does *not* give you is a bridge back from that GPU-hour to a per-token or per-request cost. You have to build that join yourself, against your own inference telemetry. **Token economics that only counts tokens is a partial view.** The [FinOps Foundation says this plainly](https://www.finops.org/insights/token-economics-the-atomic-unit-of-ai-value/): beneath every token is a chain of physical and architectural decisions that determine what the token costs to produce, and for self-hosted deployments that means the capital cost of facilities, power, and cooling, with industry estimates placing next-generation AI data center construction at fifteen to twenty million dollars per megawatt of capacity. None of that shows up as a FOCUS column today. There's no `x_PowerDrawWatts`, no PUE field, no hardware depreciation schedule. The spec's amortization machinery *could* carry that eventually — it already amortizes commitments — but nobody has defined the columns yet. **The frontier work is still aimed at the wrong side of the ledger.** [FOCUS 1.5 is slated to break down AI spend by token type and workload](https://siliconangle.com/2026/06/08/focus-specification-ai-cost-accountability-finopsx/), giving practitioners the granularity to tie inference costs back to the teams consuming them — but that's still describing metered consumption, not manufactured tokens. [Jensen Huang's "AI factory" framing](https://www.finops.org/insights/token-economics-the-atomic-unit-of-ai-value/) is the more honest lens for self-hosters: electricity and silicon enter the factory, tokens emerge, and the real unit-economics question is revenue (or value) per megawatt, not price per API call. FOCUS hasn't gone there yet. **Cardinality is a real, admitted problem.** Even on the consumption side, [FOCUS contributors are candid that the harder frontier is AI token economics](https://siliconangle.com/2026/06/08/ai-token-economics-focus-specification-updates-finopsx/), because measuring the cost of inference requires visibility down to the per-user, per-session, and per-request level, and it could end up being that a non-trivial percentage of your cost is having to be attributed to just getting your data and putting it through a pipeline. If that's true at hyperscaler scale, it's just as true, proportionally, on a single GPU box in someone's closet. ## What We're Going to Try on the Homelab Yes, running a FinOps billing spec against a single [RTX 5090](/posts/thursday-thoughts-the-models-we-cant-run) is a little absurd on its face. Nobody needs a standardized multi-cloud invoice schema to know what one GPU costs. But that's exactly why I want to try it. Homelab-scale is the cleanest possible environment to see whether the discipline holds up when you strip away all the enterprise noise — no shared cost pools, no negotiated discounts, no thousand-account org structure. Just one box, one power meter, and a request log. The plan is to try to build a real, if tiny, FOCUS-shaped dataset for the homelab: `BilledCost` derived from wall-clock power draw at the outlet times our electricity rate, `ConsumedQuantity` in tokens/sec pulled from llama.cpp's own metrics, and a hand-rolled `x_PowerDrawWatts` column FOCUS doesn't define yet, joined against the model and quant we were running at the time ([our daily driver is still Qwen 3.5 35B-A3B at Q4_K_XL, 22 GB of weights, 200+ tok/s on the 5090](/posts/thursday-thoughts-the-models-we-cant-run)). If it works, it becomes the smallest possible proof that the "AI factory" framing scales down as cleanly as it scales up — that the true cost of a token is never just the API price, it's power and silicon showing up as a line item, whether that line item is a hyperscaler's data center or a workstation under a desk. ## By the Numbers - **June 3, 2026** — the [Linux Foundation announces intent](https://itsfoss.com/news/tokenomics-foundation/) to launch the Tokenomics Foundation - **12** [organizations already backing it](https://www.cio.com/article/4182274/linux-foundation-targets-ais-cost-management-problem-with-tokenomics-foundation.html), including Google Cloud, Microsoft, Oracle, Salesforce, SAP, and JPMorgan Chase - **January 2023** — [FOCUS formally becomes a Linux Foundation project](https://focus.finops.org/focus-specification/v1-1/), roughly 15 years after AWS launched EC2, and 3 years after the modern LLM API era began - **1.4** — the [current FOCUS version](https://focus.finops.org/focus-specification/), ratified June 4, 2026, adding 2 datasets, 47 columns, 6 attributes, and 17 glossary entries - **24x** — [projected growth in global token usage](https://itsfoss.com/news/tokenomics-foundation/) between 2026 and 2030 - **120 quadrillion** — [projected global tokens consumed per month](https://itsfoss.com/news/tokenomics-foundation/) by 2030 - **98%** — [FinOps teams now managing AI spend](https://www.finops.org/topic/ai-value/), up from 31% just two years ago - **3–8x** — [how much more an output token costs than an input token](https://www.finout.io/blog/ai-model-cost-breakdowns-the-complete-2026-comparison-guide), due to generation compute - **$15–20M** — [estimated construction cost per megawatt](https://www.finops.org/insights/token-economics-the-atomic-unit-of-ai-value/) of next-generation AI data center capacity - **0** — FOCUS columns today for power draw, PUE, or hardware depreciation on self-hosted inference - **1** [RTX 5090](/posts/thursday-thoughts-the-models-we-cant-run) we're about to try to build a FOCUS-shaped cost dataset around anyway === ## Model Showdown Round 9: Qwen 3.6 27B vs Qwen 3.6 35B-A3B vs Qwythos-9B vs GLM-4.7-Flash vs Nemotron-3-Nano - URL: https://vibescoder.dev/posts/model-showdown-round-9-qwen-3-6-27b-vs-qwen-3-6-35b-a3b-vs-qwythos-9b-vs-glm-4-7-flash-vs-nemotron-3-nano - Date: 2026-07-14 - Tags: #model-showdown #benchmark #ai #llm #homelab #building-in-public #coder - Reading time: 19 min read I put Qwen 3.6 27B, Qwen 3.6 35B-A3B, Qwythos-9B, GLM-4.7-Flash, and Nemotron-3-Nano through the same real coding task on my homelab RTX 5090. Along the way I had to live-patch two separate llama.cpp bugs — and even after fixing them, I couldn't fully prove one model's failure wasn't the harness's fault. --- Round 7 ended on a cliffhanger I couldn't stop thinking about. Qwen 3.6 35B-A3B *built the entire feature* — read the codebase, wrote the files, got a clean build — and then spent 77 messages, more than half its session, failing to take a Playwright screenshot. It never committed. It never pushed. All that work, gone. Was that a bad day, or is it structural? Round 9 was supposed to answer that with three contestants: the 35B-A3B running back for a rematch, a dense 27B challenger, and NVIDIA's Nemotron-3-Nano as an architectural wild card. Clean, narrow, three-way test of dense-vs-MoE. It didn't stay clean. By the time I was done, I'd expanded the field to five models, and I'd personally patched two separate llama.cpp/template bugs live, mid-bakeoff, just to get two of the contestants to a fair starting line. One of those fixes worked perfectly — and the model still failed anyway, for a completely different reason. Let's get into it. ## The Setup This is Round 9 of the Local Model Showdown, a sub-series of Model Showdown that only tests models I can actually run on my own hardware. No API keys, no cloud spend — just an RTX 5090 and however much patience the model has for a real coding task. The homelab, unchanged from Round 7: - **CPU**: AMD Ryzen 9 9950X3D, 64GB RAM - **GPU**: NVIDIA RTX 5090, 32GB VRAM - **Inference**: llama.cpp, single-model serving, one contestant loaded at a time - **Agent platform**: Coder Agents - **OS**: Ubuntu 24.04 ### The Contestants The plan called for three. I ran five. | Run | Model | Architecture | Role | |---|---|---|---| | 1 | **Qwen 3.6 27B** | Dense transformer | Primary dense challenger | | 2 | **Qwen 3.6 35B-A3B** | MoE transformer | Incumbent / Round 7 rematch | | 3 | **Qwythos-9B-Claude-Mythos-5-1M** | Dense, MTP speculative decoding | Unplanned wild card | | 4 | **GLM-4.7-Flash** | Dense | Unplanned wild card | | 5 | **Nemotron-3-Nano-30B-A3B** | Hybrid Transformer-Mamba-2 MoE | Architectural wild card | Why the field grew: once the harness and the model-serving pipeline were working, running two more small/cheap contestants cost almost nothing extra in setup, and both turned out to matter — one for a completely novel failure mode I hadn't seen in six rounds of this series, the other for confirming a fix actually worked in practice, not just in a curl test. Model-to-run mapping was randomized and sealed before any task prompt was sent, same as every round in this series. ## The Task Identical to Round 7, on purpose — this is the only way to get a clean cross-round read on the 35B-A3B incumbent. > **Goal**: Add a Tag Manager to the `/admin` section. > > **Requirements**: > 1. `lib/tags.ts` — read all tags from published and draft posts (gray-matter) > 2. `GET /api/admin/tags` — JSON list of tags with post counts > 3. `PUT /api/admin/tags/{tag}` — rename a tag across all posts > 4. `DELETE /api/admin/tags/{tag}` — remove a tag from all posts > 5. `/admin/tags` page — list with inline rename/delete > 6. Link `/admin/tags` from the admin nav > 7. Screenshot of the finished page in the PR description, via Playwright MCP > 8. `npm run build` must pass before any commit > 9. Commit in logical chunks, push the branch Same nine requirements. Same baseline commit. Same "no hand-holding" philosophy — the only messages I sent mid-run were a bare `"continue"` when a session paused at a harness turn-limit, never a hint about what to do next. ## Interlude the Infrastructure Fought Back Twice This is the part that wasn't in the plan. Two of the five models — Qwythos-9B and Nemotron-3-Nano — failed their very first request with a hard error before ever seeing the actual task. Both failures traced back to llama.cpp's automatic tool-call parser, and both required extracting the model's raw jinja chat template out of the GGUF metadata and hand-patching it. **Qwythos-9B's bug**: its embedded template unconditionally raises `Jinja Exception: System message must be at the beginning` the moment it sees a *second* system-role message. Coder always sends two — its own agent prompt, then a workspace-context note — so every single request 400'd before the model ever ran. The template's own author clearly didn't anticipate a harness that layers system messages. Fix: pull the template via the server's `/props` endpoint, patch the three-line conditional block that renders the first system message but raises on the second, and reload with `--chat-template-file` pointing at the patched copy. Verified the patch preserved the model's native Claude/Anthropic-style `` format — this wasn't a case of stripping tool support to dodge the crash. **Nemotron-3-Nano's bug** was subtler. The historical config used a `--special` flag that causes the model's `<|im_end|>` stop token to print as literal output text instead of being silently consumed — which broke the auto-derived tool-call parser with a `500: unparsed peg-native output` error. The existing workaround was overriding the template entirely with a generic `chatml` template. That avoided the crash, but the generic template doesn't render the `tools` list into the prompt at all — so the model couldn't see what functions existed, and it hallucinated plausible-sounding ones (`ls`, `pwd`) that didn't match anything I'd actually provided. Fix: drop `--special`, keep the real native template with tool-schema injection intact. Verified — repeated tests produced correctly parsed, correctly-named tool calls. Both fixes worked. I confirmed each one with direct API tests before sending a single task prompt. Both models still failed the actual bakeoff task anyway — though as it turns out, I could only fully confirm one of those two follow-up failures was really the model's fault. More on that below. ## The Results | Model | Messages | Total Tokens | Interventions | Outcome | |---|---:|---:|---:|---| | **Qwen 3.6 27B** | 304 | 7,967,497 | 8 (neutral) | **Complete** — PR #19, 4 commits | | **Qwen 3.6 35B-A3B** | 237 | 5,592,314 | 5 (neutral) | **Complete** — PR #20, 5 commits — best run | | **Qwythos-9B** | 14 | 76,605 | 2 (neutral) | **Failed** — never executed a real tool call (cause inconclusive, see below) | | **GLM-4.7-Flash** | 246 | 4,951,774 | 3 neutral + 1 correction | **Complete** — PR #21, 1 commit | | **Nemotron-3-Nano** | 165 | 2,302,880 | 4 (neutral) | **Failed** — never found the repo | Three of five shipped a real, mergeable PR. The incumbent Qwen 3.6 35B-A3B didn't repeat its Round 7 spiral — the reproducibility signal says Round 7 really was a bad day, not a structural MoE problem. The dense-vs-MoE hypothesis, meanwhile, got muddier, not cleaner: the best run of the round was an MoE model, and the two failures split one dense (Qwythos), one MoE (Nemotron). ## What Each Model Actually Did ### Qwen 3.6 35B-A3B Run 2 the Redemption Arc The Round 7 incumbent came back and did everything right. It found the repo unprompted after a brief, reasonable search (it didn't have the exact org/repo name memorized, tried a couple of `gh search repos` queries, found it). It correctly diagnosed that the blog persists content through the GitHub API rather than the local filesystem — without being told — self-corrected two real bugs (a missing React import, a server/client component split), got a working authenticated Playwright screenshot on effectively the first real attempt, committed the image directly into the repo, and opened PR #20. It even added an uninstructed but reasonable improvement — wrapping the tag-count function in `React.cache()` to avoid redundant GitHub API calls during SSR — and cleaned up a stray `playwright` devDependency it no longer needed, entirely on its own initiative. Five interventions total, every one a bare "continue" at a harness pause, zero content hints. This is the cleanest run of the round, and the strongest evidence yet that Round 7's failure wasn't inherent to this model's architecture. ### Qwen 3.6 27B Run 1 Got There the Slow Way The dense challenger also shipped — PR #19, four commits — but needed twice as many interventions to get there. It hand-rolled its own Playwright script instead of using the MCP tool as instructed (an instruction-following miss echoed by every model in this round that attempted a screenshot), and it burned real turns flailing on `gh api`/`gh pr edit` argument syntax trying to attach that screenshot to the PR after the fact. It recovered on its own — no hints given — by committing the image straight into the repo and rewriting the PR body to point at the raw GitHub URL, exactly the pattern that worked for Run 2. Positive signal: real autonomous debugging along the way, including tracing a `localhost:3000` timeout to Redis simply not running, installing and starting `redis-server` itself, and clearing a stale dev-server PID lock left over from an earlier crashed attempt. The reasoning was sound. It just took the long way around. ### GLM-4.7-Flash Run 4 Right Answer Wrong Branch This run is the one asterisk in an otherwise clean sweep. It built the entire feature correctly and efficiently — but when it went looking for its `run-4` branch, it found `feature/image-management-run-4`, a stale, unrelated branch left over from a previous round's image-management feature, and assumed that was the branch it had been told to use. It checked it out, committed the tag manager on top of it, and pushed — polluting an old branch with unrelated code. No PR existed for that branch, so nothing else broke, but it wasn't the outcome the task asked for. I tested whether it would notice on its own: I reverted the polluted branch and sent one more neutral "continue," specifically to see if re-checking git state would trigger self-correction. It didn't — it just re-walked its own requirements checklist, found the missing screenshot, and kept working without ever revisiting the branch. That answer settled it: this needed an explicit correction, not another nudge. Once told directly that the branch was wrong, it recovered in a single turn — created `run-4` properly off `main`, reapplied its own code, pushed, and opened PR #21. Worth being honest about the scoring implication: this is the only run in the round that needed a content hint rather than a neutral nudge, and that should count against it relative to Run 2's fully autonomous path to the same kind of outcome. It also never resolved the screenshot requirement — no Playwright MCP available to it, a spawned sub-agent for the screenshot timed out, no `.env` credentials to log in manually — and it gave up on that requirement rather than finding a workaround. ### Qwythos-9B Run 3 Said the Right Thing Couldn't Say It Correctly or Could It This is the one that got the infrastructure fix and still failed completely, and it's the most interesting result of the round — because when I went back to check whether that failure was really the model's fault, I couldn't fully confirm it was. Three consecutive turns in the actual bakeoff, identical pattern each time: correct reasoning ("I need to read the skills files before starting," "let me check the branch structure"), followed by an attempted tool call wrapped in the wrong syntax. Instead of its own trained `...` format, it emitted a raw ad-hoc tag using the tool's name directly as the XML tag — ``, `` — which is unparseable by anything. Not one of the three attempts was ever actually executed. No repo was ever cloned. No code was ever written. Here's the honest complication. Before I called this a pure model-capability gap, I went back and tested the claim directly: I reconstructed Coder's actual system prompt (verbatim, including the injected user-instructions block), the real task prompt used in the bakeoff, and a representative tool schema — first a lean 10-tool version, then scaled up to 63 tools to match the size of Coder's actual full toolset — and fired all of it straight at Qwythos over the API, bypassing Coder's harness entirely. **Eleven for eleven.** Every single reconstruction produced a correctly-parsed, correctly-structured tool call. Not one reproduced the raw-text failure I'd watched happen three times in a row inside the real chat. That doesn't clear the model, but it doesn't convict it either. What it tells me is that I cannot honestly claim this was purely "a 9B model can't handle a long, complex prompt." Something specific to Coder's exact request — content I couldn't perfectly reproduce from outside the harness, whether that's the precise injected workspace context, a subtlety in how the conversation history was serialized, or simply an unlucky run of sampling — was very plausibly a contributing factor, and I don't have the visibility into Coder's exact request construction to rule it out. The fairest thing I can say: this failure was real, it happened three times with zero self-correction, and under my best-effort attempt to recreate the same conditions independently, it didn't happen once. Treat the result as inconclusive on root cause, not as a clean verdict on the model. ### Nemotron-3-Nano Run 5 Fixed the Bug Lost the Model Anyway The most frustrating result of the round, because the fix I made for it worked exactly as intended, and it still didn't matter. Real tool execution, confirmed throughout — no parse crashes, no hallucinated tools, every `execute` call actually ran and returned a real result. And then it spent roughly 35-40 turns across five nudges in an unbroken loop trying to locate the repository: checking `/workspace` (repeatedly, across multiple nudges, always failing the same way), checking `/home/coder/project`, `/home/coder/workspaces`, re-listing a skill directory it had already found and dismissed, misusing `list_agents` as a raw shell command, and passing a garbage string into a tool's template-ID parameter. It never once tried `gh search repos` or `gh repo clone` — the exact move every other model in the round used successfully within one or two turns. Five interventions, zero strategy convergence, zero progress on any of them. This is a pure repo-discovery / tool-selection capability gap that happened to surface *after* I'd already fixed the thing that looked, at first, like it would be the blocker. ## Scoring the Three That Shipped Deviation from the series' usual format, disclosed up front: every other round in this series scores blind, before the orchestrating model knows which run is which contestant. That wasn't possible here — I needed to do the scoring myself, and I already knew the mapping from running the whole bakeoff. So this round is scored open, not blind, using the same 7-dimension weighted rubric, and grounded in the actual PR diffs rather than the write-ups above. Qwythos-9B and Nemotron-3-Nano shipped no code, so they're excluded as DNF rather than scored to zero. | Dimension | Weight | Qwen 3.6 27B (PR #19) | Qwen 3.6 35B-A3B (PR #20) | GLM-4.7-Flash (PR #21) | |---|---|---|---|---| | Correctness | 25% | 4 | 5 | 2 | | Design | 15% | 3 | 5 | 2 | | Code quality | 20% | 4 | 5 | 3 | | Engineering judgment | 15% | 3 | 5 | 2 | | Scope discipline | 10% | 5 | 4 | 3 | | Commit hygiene | 10% | 4 | 5 | 2 | | Surprise | 5% | 3 | 5 | 2 | | **Weighted total** | | **3.75** | **4.90** | **2.30** | **Qwen 3.6 35B-A3B (PR #20)** is the only one of the three that persists tag data exclusively through the GitHub API — reading and writing through `commitFile` rather than touching the local filesystem at all. That's the correct pattern for a Vercel-hosted, read-only-filesystem app in production, and the run reasoned about it explicitly, unprompted. It also reasoned in code comments about a read-before-write race condition, the only run of the three to do so. Highest score across every dimension. **Qwen 3.6 27B (PR #19)** eventually persists through `commitFile` too, but round-trips through the local filesystem first — a pattern that works in dev and silently fails the moment Vercel's read-only production filesystem is in play. Combined with the flailing `gh api`/`gh pr edit` detour documented above, it lands solidly in the middle: correct destination, messier path, weaker commit hygiene than #20's five clean commits. **GLM-4.7-Flash (PR #21)** scores lowest, and not just because of the wrong-branch incident. Its `lib/tags.ts` only ever calls `fs.writeFileSync` — it never calls `commitFile` at all, so a rename or delete wouldn't actually persist to the real repo in production, only to an ephemeral local file. That's a step worse than PR #19's "local-first, GitHub-second" pattern: there is no GitHub-second here. Combined with skipping auth checks on its mutating routes (a gap #20 doesn't have but #19 also doesn't have), it's the weakest submission of the round on every dimension except scope discipline, where doing the least extra work worked in its favor. ## Common Threads Across the Round **Fixing the infrastructure doesn't necessarily fix the model — but I can only prove that for one of the two.** Both Qwythos and Nemotron got legitimate, verified bug fixes before their runs started. Nemotron's fix held up cleanly under later isolated testing — the model failed afterward for a completely unrelated reason (repo discovery), and I have no reason to doubt that failure is really about the model. Qwythos's case is murkier: my attempt to reproduce its failure outside Coder came back clean eleven times in a row. The bugs I patched were both real and worth fixing either way. I just can't say with confidence that fixing Qwythos's bug actually got it to a fair starting line. **The screenshot requirement remains the great equalizer.** Every model that got far enough to need one either hand-rolled a Playwright script instead of using the MCP tool as explicitly instructed, or gave up on the requirement entirely. Zero for five used the tool as asked. This has now shown up in enough rounds that it looks less like a per-model quirk and more like a standing gap in how these models are prompted or how the tool is exposed. **Committing the screenshot beats linking to it.** Every model that succeeded eventually converged on the same fix for a broken screenshot reference: commit the image into the repo and point the PR body at the raw GitHub URL, rather than referencing a local filesystem path. Nobody was told to do this. They each found it independently after their first attempt broke. **A stuck loop doesn't announce itself as a loop.** Nemotron's reasoning text was fluent and plausible on every single turn — it never repeated verbatim text the way Round 7's Gemma did. The only way to tell it wasn't making progress was tracking whether it tried a genuinely new strategy between nudges. It never did. ## What I Actually Learned **The Round 7 spiral looks like it really was a bad day.** Same model, same task, same config, and this time Qwen 3.6 35B-A3B not only shipped — it shipped the cleanest, most autonomous run of the round. If this had spiraled again, that would be a strong structural signal about MoE and sustained agentic work. It didn't. One data point either way isn't proof, but it's the opposite of what the hypothesis predicted. **Dense-vs-MoE isn't the right lens for these failures.** The plan bet on architecture predicting endurance. Instead, the split ran across capability tiers that had nothing to do with active-parameter count: the two shipped-cleanly runs were one dense (27B) and one MoE (35B-A3B); the two failures were one dense (9B) and one hybrid-MoE (Nemotron). Model size and specific training, not architecture family, look like the better predictor here. **Live infrastructure debugging is now a standing cost of running this series.** Round 7's headline infra bug was a `--jinja` vs `--chat-template` mismatch that zeroed out Devstral entirely. This round hit two more bugs in the same family — a template that crashes on multi-turn system messages, and a stop-token flag that silently breaks tool-call parsing — for two different models. Every round so far has needed at least one live template or server-config fix before the actual bakeoff could start fairly. **Isolation-testing a failure after the fact is worth doing, even when it muddies your own conclusion.** I went into the Qwythos write-up ready to call it a clean model-capability gap. Running the fairness test anyway — and getting eleven consecutive successes where the real bakeoff got three consecutive failures — means I'm publishing a less satisfying, more honest conclusion instead: something about that specific session likely mattered, and I don't have enough visibility into Coder's exact request construction to say what. A benchmark that only reports the clean story isn't a benchmark I'd trust. **"Knows the answer" and "can act on the answer" are separate capabilities — confirmed on the model that actually earned the verdict.** Nemotron's reasoning was fluent and its tool calls were correctly formatted throughout, right up until it simply couldn't find the repository across roughly 40 turns and five nudges. That's the cleaner version of the same pattern Qwythos might also show — but only Nemotron's failure held up when I went looking for a harness-side excuse. ## By the Numbers - **5** contestants (3 planned, 2 added once the pipeline was proven out) - **3 of 5** shipped a real, mergeable PR - **2** llama.cpp/template bugs found and patched live, mid-bakeoff — **1 of 2** fixes conclusively verified to hold up under isolated re-testing after the run - **11 of 11** fairness-test reconstructions of Qwythos's failure conditions came back clean — zero reproductions of the raw-text tool-call failure it showed three times in the real bakeoff - **0 of 5** models used the Playwright MCP tool as explicitly instructed for the screenshot requirement - **8** interventions for the slowest successful run (Qwen 3.6 27B) vs. **5** for the fastest (Qwen 3.6 35B-A3B) - **1** run needed an explicit correction rather than a neutral nudge (GLM-4.7-Flash, wrong branch) - **~35-40** turns burned by Nemotron-3-Nano searching for a repo it never found, across **5** nudges with zero strategy change - **3** consecutive identical failures from Qwythos-9B, zero tool calls ever actually executed - Total tokens across all five runs: **~20.9 million** — for three completed PRs and two runs that produced no usable code at all *Next up: does the dense 27B's clean-but-slow performance hold up against a frontier cloud model on the same rubric, or does the token-efficiency gap Round 7 found in Sonnet still stand?* === ## TurboQuant, Four Months Later: Chasing Google's 6x VRAM Claim Into the Wild - URL: https://vibescoder.dev/posts/turboquant-four-months-later-chasing-googles-6x-vram-claim - Date: 2026-07-13 - Tags: #homelab #ai #llm #benchmark #building-in-public - Reading time: 7 min read Back in Q1 I read a headline about Google cutting AI memory use 6x and filed it under "watch and revisit." Four months later, Google still hasn't shipped official code, but a whole ecosystem of forks has, llama.cpp has an open PR, and at least one compatibility gotcha lands squarely on our daily driver. Here's the honest state of TurboQuant heading into Q3, and the test I'd actually run against it. --- Back in Q1, I read a headline about Google cutting AI memory use by 6x and filed TurboQuant under "watch and revisit" — no code, tested only up to 8B parameters, nothing to actually run against `AI-NT-No-Problem`. Four months is a long time in this industry. I went back to see what actually happened, and the honest answer is: a lot, but not the thing I expected. ## What Google Actually Shipped Quick recap for anyone who missed the original story. TurboQuant is a training-free algorithm suite — TurboQuant proper, plus PolarQuant and Quantized Johnson-Lindenstrauss — that compresses the KV cache specifically, not model weights, cutting memory by at least 6x with an 8x speedup in attention computation on H100s. The paper, "Online Vector Quantization with Near-optimal Distortion Rate," came out of Google Research and Google DeepMind and was accepted at ICLR 2026. Here's the part that hasn't changed since March: as of the most recent status I could confirm, Google still hasn't shipped official code. The original "expected Q2 2026" timeline for an official release has quietly passed without one landing anywhere I can find. ## The Ecosystem Filled the Vacuum Then Fragmented What happened instead is the pattern anyone who's watched an ML paper drop before recognizes: two weeks after the ICLR paper, five independent implementations already existed, including one running a 104B parameter model on a MacBook. Four months later that's grown to at least eight or nine separate forks and packages, from a pip-installable HuggingFace wrapper to CUDA/Triton implementations to an AMD ROCm-specific fork. The interesting shift is in tone, not just headcount. The maintainer of one of the more actively maintained forks is now openly walking back some of the original hype: the community has converged on a more nuanced picture than the initial hype suggested, currently recommending plain FP8 KV cache as the best default on Hopper/Blackwell hardware, and reaching for TurboQuant only when you need more than 2x compression and are willing to accept some throughput cost. That's a meaningfully more conservative position than "6x memory, zero accuracy loss" read in March. ## llama.cpp Open PR Not Merged but Forkable Today For our actual stack, this is the part that matters most. There's an open PR proposing two new KV cache quantization types for llama.cpp, `tbq3_0` (about 3.06 bits per element) and `tbq4_0` (about 4.06 bits per element), adapting TurboQuant's two-stage recipe to GGML's block format. As of the last status check I could find, it's still open and under review, not merged into mainline. What is available today: several community forks already ship it as a runtime flag, something like `--cache-type-k turbo3 --cache-type-v turbo3` alongside `-fa on`, and critically it applies only at the KV cache layer — no re-quantization or re-conversion of existing GGUF weights required. If we wanted to try this against our own `llama-server` setup, we could point one of these forks at the exact same Qwen 3.5/3.6 GGUF we already run. ## The Verdict Got More Sober Independent evaluation is where the picture diverges hardest from the launch-week coverage. One evaluation from Red Hat AI and the vLLM team found meaningful accuracy drops on reasoning and very long context at 3-bit precision, particularly when the QJL residual-correction step is left enabled. Multiple independent teams reached the same conclusion from different directions: the paper's own "extra bit of correction" step often hurts more than it helps at low bit widths, with plain MSE-only quantization beating MSE+QJL across every model the community has tested. There's also real hardware data putting a number on the actual problem this is trying to solve. One llama.cpp discussion thread includes measured DGX Spark GB10 results showing existing q4_0 KV cache quantization is already 36.8% slower than f16 at roughly 110K tokens of context, purely from per-token dequantization overhead — that's the exact bottleneck fused TurboQuant-style kernels are built to remove, and it's a legitimate, measured problem independent of how any individual fork's compression ratio claims hold up. ## What This Means for AI-NT-No-Problem Specifically Two things line up well, and one thing is a real caveat worth testing before trusting any number. The good news: at least one fork is explicitly tested on dense and MoE architectures across RTX 3090 and RTX 5090 GPUs with a vLLM/Triton integration, and a separate Windows build explicitly targets the CUDA 13.x runtime — both a much closer match to our actual hardware than the original paper's H100/A100-only validation. The caveat: the original paper only validated on head_dim=128 models (Gemma, Mistral, Llama-3.1-8B). At least one fork found that head_dim=64 doesn't Gaussianize well enough for the core rotation-then-quantize math to hold, requiring a fallback to plain q8_0, and a separate community benchmark thread flagged that a Qwen3.6 model's GQA head_dim=256 configuration caused one implementation's K-cache to come out *larger* than plain q8_0 on that architecture specifically. Our daily driver is Qwen 3.5/3.6 35B-A3B. That's not a "probably fine" caveat — it's a "test this exact model before believing any compression number" caveat. ## The Test I'd Actually Run Same instinct as the [LocalAI bakeoff plan](/posts/comfyui-lemonade-and-localai-scouting-the-next-wave-of-homelab-ai-tools): no YAML in this post, but the shape of the test is worth writing down now. | Layer | What it tests | How | |---|---|---| | **1. Head-dim compatibility** | Does TurboQuant's math actually hold for Qwen 3.5/3.6's GQA config, or does it degrade like the community report suggests? | Direct KV-cache size and perplexity comparison, TurboQuant vs. our current `q4_0`/`q8_0` cache types, same model, same prompts | | **2. Long-context throughput** | Does it actually fix the dequantization slowdown, or just move the cost around? | Token/sec at 24K, 64K, and 128K+ context, mirroring the DGX Spark data above | | **3. Known failure-mode check** | Does the QJL step help or hurt at the bit widths we'd actually use? | Binary pass/fail against MSE-only vs. MSE+QJL, same test set | | **4. Daily-drive soak test** | Does the compression survive real agentic traffic, not just synthetic long-context benchmarks? | Run OpenClaw against the TurboQuant-patched `llama-server` fork for a few days | If layer 1 fails outright, on our actual model, the rest doesn't matter, and that's worth knowing before writing a single line about a 6x win. ## A Word on the Sources Here Worth being upfront about: a fair amount of what's indexing for "TurboQuant" right now reads like SEO-optimized rewrites of the original launch coverage, plus a long tail of solo-developer GitHub forks with very confident, very polished README claims, versioned like production software, benchmark tables and all. The specific compression multipliers floating around (4.6x, 5.2x, 8.9x, 12x, take your pick) come from different implementations that haven't been reconciled against each other. None of that means the underlying idea is wrong. It means the number in any given headline, including the 6x one I originally filed this under, deserves a "reproduce it yourself" asterisk before it goes anywhere near a decision about this homelab. ## By the Numbers - **6x** — the KV cache memory reduction Google's original paper claimed, still the number everyone quotes - **0** lines of official Google code confirmed shipped, four months after the paper and past the original "Q2 2026" target - **8–9** independent community forks and packages now implementing TurboQuant in some form - **1** open, unmerged llama.cpp PR (`tbq3_0` / `tbq4_0`), the actual path to mainline support - **36.8%** — measured generation slowdown from existing `q4_0` KV cache quantization at ~110K context, the real problem being solved here - **1** specific compatibility red flag (Qwen3.6's GQA head_dim=256) landing directly on our daily-driver model - **4** test layers in the plan above, and zero of them run yet The 6x headline was real, as far as the original paper's own benchmarks go. Whether it survives contact with our exact model, our exact GPU, and four months of community re-litigation is a different question, and it's the only one worth actually testing. === ## Model Showdown Round 8: Sonnet 5, Opus 4.8, and Fable 5 Walk Into a Tag Manager - URL: https://vibescoder.dev/posts/model-showdown-round-8-sonnet-5-opus-4-8-fable-5 - Date: 2026-07-10 - Tags: #ai #llm #benchmark #homelab #agents - Reading time: 11 min read A routine "update Coder" request turned into a full model bakeoff: fixing misconfigured thinking params so Sonnet 5 would stop calling itself 4.5, discovering Playwright MCP can't be wired into Coder Agents at all, and watching three frontier models independently pause at the exact same step before finishing. Sonnet 5 won on score and on cost, by a mile. --- It started as a two-line request: update my homelab Coder instance to the latest version. It ended, several hours later, with three frontier Anthropic models building the same feature in parallel, a definitive answer on why Playwright can't be wired into Coder Agents, and a cost table that makes the case for Sonnet 5 louder than any benchmark I've run yet. Here's the whole arc. ## The Upgrade That Opened the Door The homelab Coder server was on v2.34.0. `curl -L https://coder.com/install.sh | sh` over SSH pulled v2.35.1, a `systemctl restart coder`, and I was current. Nothing interesting there — except that landing on v2.35.1 meant landing on a deployment where the AI model lineup had drifted since I'd last looked at it. Sonnet 5, Fable 5, and Opus 4.8 were all real, shipped models I hadn't provisioned yet, and Opus 4.6/4.7 and Sonnet 4.6 were sitting there stale. That's the real starting point for this post: not the version bump, but what I found once I went looking at the model config. ## Sonnet 5 That Thought It Was Sonnet 4.5 I configured Sonnet 5 and Fable 5 through the Coder Models UI, using the model IDs I had. Both showed up correctly in the chat picker. Neither behaved correctly: ``` Are you running sonnet 5/ I'm running on Claude Sonnet 4.5. ``` Fable 5 was worse — it didn't even know what it was: *"I'm one of Anthropic's Claude 4-series models."* The model IDs were right. What was missing was the `provider_options` block — `effort`, `thinking.budget_tokens`, `thinking_display` — that actually engages the model's current-generation behavior on Anthropic's side. Without it, the request apparently falls back to older default routing. I didn't have to guess at the correct shape. A teammate's Coder instance had the identical models configured and working, so I pulled the exact JSON straight out of the browser's network tab — every model config on their deployment, cost rates included — and used it as ground truth: ```json { "model": "claude-fable-5", "display_name": "Fable 5 Medium ($$)", "model_config": { "provider_options": { "anthropic": { "send_reasoning": true, "effort": "medium", "thinking_display": "summarized", "web_search_enabled": true } } } } ``` I PATCHed the homelab's model configs to match — Sonnet 5, Fable 5, and a freshly added Opus 4.8 — then deleted Opus 4.6, Opus 4.7, and Sonnet 4.6, which weren't earning their place in the lineup anymore. Final roster: three current-generation Anthropic models, all correctly self-identifying, all with real thinking budgets. That's a config problem solved. It also happened to leave me with three fresh, uncompared models sitting on the same deployment. So: bakeoff. ## The Plan That Had to Change Twice I already had a Round 8 plan on the books — a reproducibility test, re-running Opus 4.7 and Sonnet 4.6 against a new Opus 4.8 on the same task from Round 5. That plan died the moment I deleted Opus 4.7 and Sonnet 4.6 from the deployment. There was no "incumbent" left to hold constant. New plan: three-way horse race — Sonnet 5, Fable 5, Opus 4.8 — same rubric, no reproducibility angle, task TBD. The obvious task was Round 5's admin/images feature. Except a screenshot check turned up an `Images` tab live in the real `/admin` nav — the feature had already shipped to `main` since I'd last planned this round. Good problem to have, bad task to reuse. I switched to the **tag manager** task instead — planned back in Round 9, never executed, still absent from `main`. Confirmed clean baseline at commit `f6b713f`, cut three branches (`run-1`, `run-2`, `run-3`), sealed a random run-to-model mapping, and wrote three identical task prompts that differed only by branch name. ## Three Isolated Chats One Identical Prompt ``` Goal: add a tag manager to /admin. Requirements: - lib/tags.ts to read all tags from published and draft posts - GET /api/admin/tags with per-tag post counts - PUT /api/admin/tags/{tag} to rename across all posts - DELETE /api/admin/tags/{tag} - /admin/tags page with inline rename/delete - Link from the /admin nav - Screenshot via Playwright MCP - npm run build must pass before committing - Commit in logical chunks, push when done ``` Three Coder Agents chats, three models, launched in parallel, no cross-awareness. I sat back and let them cook. ## The Wall All Three Models Hit at the Same Spot An hour later, all three chats were sitting in a `waiting` state. Pulling the transcripts, every single one had: 1. Built the entire feature 2. Passed `npm run build` 3. Committed in logical chunks 4. Hit the screenshot requirement — and discovered Playwright wasn't available as a callable tool 5. Burned real effort improvising a workaround (`npm install playwright`, `playwright install --with-deps chromium`, hunting missing system libs) 6. Successfully produced and attached a real screenshot 7. **Stopped, without ever running `git push`** My first assumption was a hard message cap — the API's default page size (50) made it look like every chat had been cut off at exactly the same length. Re-querying with a higher limit corrected that: real totals were 92, 98, and 116 messages. No cap exists. What's real is that all three models, independently, treated "I've shown you visual proof it works" as a natural point to pause and wait for a human, even though the prompt's last bullet point was explicitly "push when done." That's a genuine, reproducible finding, and it says nothing about which model is better — it happened identically regardless of which model was running. ## Why Playwright Can't Actually Be Fixed Here I went looking for a real fix and found a structural wall instead. Registering an MCP server on this Coder deployment only accepts `transport: streamable_http` or `sse`: ```json {"message":"Validation failed.","validations":[ {"field":"transport","detail":"Validation failed for tag \"oneof\" with value: \"stdio\""} ]} ``` Playwright MCP is `stdio`-only by design — it drives a local browser, so it has to run as a local process. There's no hosted, remote Playwright MCP endpoint to point a `streamable_http` transport at. This isn't a misconfiguration I introduced; it's baked into how Coder Agents' native tool-calling and Playwright's transport model don't overlap. My own workspace template even documents an adjacent version of this exact gap — the fitness-tracker and vibescoder MCP servers are wired up over raw HTTP specifically *"because the Coder Agents chat doesn't auto-register MCP servers from `~/.mcp.json`."* The real fix isn't config — it's environment. Pre-baking Playwright, Chromium, and its system dependencies into the workspace Docker image would let agents skip the entire live-provisioning step and go straight to taking the screenshot. That's a Dockerfile change I'm holding off on until I've thought it through further. ## Finishing the Run Since all three chats were genuinely paused, not dead, I nudged them — identically, via the same API the chat UI itself uses: ```bash curl -X POST '.../api/experimental/chats/{id}/messages' \ -H "Coder-Session-Token: $TOKEN" \ -d '{"content": [{"type": "text", "text": "Continue — run npm run build one more time to confirm it still passes, then push the run-N branch now."}]}' ``` All three finished within a minute of the nudge. All three branches pushed clean, independently re-verified with a fresh `npm install && npm run build` in isolated git worktrees. ## Blind Scoring Same 7-dimension rubric the series has always used, scored before I knew which run was which model. | Dimension | Weight | run-1 | run-2 | run-3 | |---|---|---|---|---| | Correctness | 25% | 5 | 4 | 4 | | Design | 15% | 4 | 4 | 4 | | Code quality | 20% | 4 | 4 | 5 | | Engineering judgment | 15% | 4 | 5 | 4 | | Scope discipline | 10% | 5 | 5 | 4 | | Commit hygiene | 10% | 4 | 5 | 5 | | Surprise | 5% | 4 | 5 | 4 | | **Weighted total** | | **4.35** | **4.40** | **4.30** | Tightest field this series has produced — 0.10 points from top to bottom. **run-1** had the only explicit error-handling path around its live-GitHub data fetch, and the cleanest separation between pure content-transform functions and the API route orchestrating them. Weakest commit hygiene — two coarse commits against the other two runs' four. **run-2** reasoned explicitly, in its own code comments, about a staleness/race condition — listing tags from a fast build-time snapshot but re-fetching each post fresh from GitHub right before writing, "so we never clobber a concurrent edit." After the nudge, it also explicitly verified branch safety before pushing rather than just complying. Best commit hygiene. **run-3** had the cleanest code of the three — a single shared `mutateTag` helper factoring rename and delete, no duplication. It also added `publishedCount`/`draftCount` tracking per tag that nobody asked for and the UI never surfaces. ## The Reveal ``` run-1: Opus 4.8 run-2: Sonnet 5 run-3: Fable 5 ``` **Sonnet 5 wins outright.** ## The Cost Story Is Not Close | Model | Output tokens | Cache write | Cache read | Est. cost | $/rubric point | |---|---:|---:|---:|---:|---:| | Sonnet 5 | 22,471 | 201,399 | 4,240,442 | **$2.00** | **$0.45** | | Opus 4.8 | 28,597 | 241,641 | 4,758,062 | $4.60 | $1.06 | | Fable 5 | 21,876 | 179,517 | 3,153,993 | $7.84 | $1.82 | Sonnet 5 scored highest *and* cost roughly a quarter of Opus 4.8 and a quarter of Fable 5. That's the whole argument, in one table. The Fable 5 result deserves a closer look, though, because the obvious story — "it overthought the problem" — isn't what the data shows. Fable 5 actually used the **fewest tokens of the three** in every category: lowest output, lowest cache write, lowest cache read. Its cost is high purely because its per-token pricing is 4-5x Sonnet 5's. So "too thoughtful" isn't about verbosity. It's about *where* the thinking went. Fable 5 tied or trailed in every heavily-weighted category — Correctness, Engineering judgment, Scope discipline, Surprise, 55% of the rubric combined — and only led outright in Code quality. Its one real differentiator, the shared `mutateTag` helper, was genuine skill. Its other distinguishing move, the unused published/draft counters, was polish spent where nobody was looking, while Opus 4.8 and Sonnet 5 spent their effort on error handling and safety reasoning respectively — both of which paid off in more heavily-weighted categories. That's a read on one run each, not a verdict on Fable 5 as a model. ## Two Notes on How This Bakeoff Itself Got Run This round's entire orchestration — model provisioning, branch creation, chat creation, telemetry pulls, the nudge, blind scoring — was driven by an assistant SSH'd into the homelab from inside a Docker-hosted Coder workspace, rather than done by hand through the browser. Worth being explicit that this doesn't compromise the experiment: the three contestant models still worked in fully isolated chats with zero cross-awareness, and every consequential step — model config changes, anything touching `main`, the mid-run nudge — still went through explicit human sign-off before it happened. What changed was orchestration speed, not experimental integrity. The more interesting discovery, for future rounds: the Coder chat API accepts posted messages directly at `POST /api/experimental/chats/{id}/messages` — the exact same endpoint the browser UI itself calls. I only used it this round to send the resume nudge, but there's no reason the entire "paste the prompt into three browser tabs" step couldn't be scripted too — create the chats, assign the models, post the warmup and task prompt, poll for completion, nudge anything that stalls. Next round, I'd like to make the whole thing closer to a one-command operation. ## What's Next The local model track (Round 9: dense vs MoE architecture, testing whether last round's MoE screenshot-spiral was structural or a bad day) is still on the board, unrelated to this round's findings. And the Playwright-in-Docker fix is a real, scoped piece of work I want to think through properly before touching the workspace image. ## By the Numbers - **3** frontier models benchmarked: Sonnet 5, Opus 4.8, Fable 5 - **0.10** rubric points separated 1st place from 3rd — tightest field in the series - **4.40** — Sonnet 5's winning weighted score - **$0.45** — Sonnet 5's cost per rubric point, the cheapest by roughly 2.3x - **$7.84** — Fable 5's total cost on this task, despite using the *fewest* tokens of the three - **0** — MCP servers registered on the deployment before this round; Playwright cannot be one, ever, on this architecture - **3 of 3** models independently paused at the identical step (screenshot attached, before push) — a disposition finding, not a per-model one - **1** API call (`POST .../chats/{id}/messages`) that turned a stuck bakeoff back into a finished one === ## Thursday Thoughts: Why Anthropic Is the Next AWS, but Potentially Worse - URL: https://vibescoder.dev/posts/thursday-thoughts-why-anthropic-is-the-next-aws-but-potentially-worse - Date: 2026-07-09 - Tags: #meta #building-in-public #ai #agents #future-of-coding - Reading time: 8 min read Anthropic and AWS are both, underneath everything else, infrastructure providers renting out specialized compute. But the sharper parallel is behavioral — both built a thriving ecosystem, then started eating pieces of it. Claude Design blindsiding Figma and Canva looks a lot like AWS's Elasticsearch moment, except it's happening at a pace AWS never approached, and one layer higher up the stack. --- A few weeks ago I wrote about how [AI-native mirrors cloud-native](/posts/thursday-thoughts-how-ai-native-mirrors-cloud-native) — lift-and-shift now, real architectural rethink later. The same pattern enterprises went through with the cloud. That post was about workflows and org design. This one pulls on a different, less comfortable thread of the same analogy: Anthropic is starting to look a lot like AWS did a decade ago. Same core move. Much faster clock speed. And I'm not convinced it (or we, startups) survives the difference. ## The Core Asset Was Never the Cloud or the Model Quick side rant, then I'll get to the point. Anthropic and the other frontier labs are obviously more than infrastructure companies. They do real research, real alignment work, real science. But strip away the mission statements and look at the balance sheet: the core asset is a giant pile of GPUs with a business model of renting time on them. It's just denominated in tokens instead of instance-hours. An EC2 instance is general-purpose compute you point at whatever workload you want. A Claude API call is the same idea, just pre-loaded with one very specific, very valuable workload already running. That's not a knock. Ok, rant over. The interesting comparison isn't the compute. It's what companies do when they sit atop these platforms. ## AWS Ate Its Ecosystem Slowly Enough to Argue About It A thriving ecosystem of startups and open source projects built directly on top of AWS for most of the 2010s. And AWS had a front-row seat to all of it. That vantage point let AWS watch which capabilities were working. Decide which ones were worth turning into first-party, cloud-native services. And then executing, quickly. The pattern is well documented at this point. AWS forked Elasticsearch into Open Distro in 2015 rather than pay Elastic for a managed offering. It shipped a MongoDB-compatible DocumentDB service that pushed MongoDB from AGPL to SSPL in 2018. It offered a managed Cassandra service. Redis followed the same arc in 2024, moving off BSD in response to AWS's ElastiCache and Azure's competing cache products, before reversing back to AGPLv3 in 2025 once the community fork (Valkey) had absorbed enough of the commodity pressure that Redis could go back to competing on product instead of licensing. HashiCorp did the same thing with Terraform's move to BSL in 2023, which is arguably the case that did the most lasting community damage, and which triggered the OpenTofu fork within weeks. The pattern is pretty clear: successful infrastructure project gains traction, a hyperscaler ships a managed, forked, or compatible version without meaningfully contributing back, the original company changes its license to defend the business, and the community reacts, sometimes forking around the license itself. It's now a repeating, named cycle in the open source world. AWS was the proximate cause of every iteration of it. What's interesting to me is that this dance didn't kill the ecosystem. Elastic, Redis, and HashiCorp are all still around. Redis' own CEO has since said publicly that pushing AWS onto its own fork put both companies on a level playing field where they compete on product rather than fighting over a shared codebase. AWS drew blood. Then eventually gave the ecosystem room to differentiate around it. ## Anthropic Is Running the Same Play at a Different Speed Now look at Claude. A thriving ecosystem of companies has built directly on top of it. They have the same privileged vantage point AWS had. So naturally, Anthropic is doing exactly what AWS did: watch what's working, then ship a first-party version. The case that kicked up the most dust was Claude Design. Anthropic's chief product officer quietly resigned from Figma's board three days before the launch. Figma's stock dropped roughly 7% on launch day alone. And Clauyde Design landed as a direct, conversational alternative to opening Figma in the first place. To make matters worse, it offered a preferential export path into Canva, a company Anthropic had been partnering with for two years. Figma and Adobe, both long-standing Anthropic partners, told reporters afterward they'd had essentially no advance notice of what was coming. Design wasn't a one-off. Claude for Legal expanded into a full suite of plugins and MCP connectors aimed at law firms, directly overlapping with venture-backed legal AI startups like Harvey and Legora. Claude Science launched as a dedicated research workbench connecting to more than 60 scientific databases. Anthropic explicitly framed this as extending Claude's reach from "just a model provider" into owning the operating layer for an entire industry. Sound familiar? It's the same language used about what Claude Code did for software development. Anthropic acquired Coefficient Bio to bring first-party pharmaceutical-planning capability in-house. Claude for Word pushed directly at Microsoft's own productivity suite. And a round of Claude Cowork plugins aimed at legal and sales workflows was blunt enough to trigger what multiple outlets started calling a "SaaSpocalypse" sell-off across data analytics and professional services stocks. That's design, law, science, productivity software, and enterprise workflow tooling. All absorbed or threatened inside about the same number of months it took AWS to ship one Elasticsearch fork. Remember, Claude Code is even 18 months old yet. ## Supersonic Vs. Hypersonic Here's the part that actually worries me. It's not the individual moves. It's the tempo. AWS was famous in its era for a relentless release cadence. It took years to work through its ecosystem: Elasticsearch in 2015, the Cassandra-compatible service later, the MongoDB and Redis license fights stretching from 2018 to 2024. That gave the ecosystem time to see the pattern coming and build defenses into it Change licenses. Fork a community version. Compete on genuine product differentiation. Anthropic movies at a pace that makes AWS look patient. If AWS was supersonic, Anthropic is hypersonic. The risk with hypersonic isn't just speed, it's that the ecosystem doesn't have time to adjust before the next shockwave. There's a second difference that matters just as much: AWS mostly stayed at the infrastructure layer. It rarely competed directly against the applications that ran core business processes. Anthropic is doing the opposite. It's competing directly at the application layer, against the actual products entrepreneurs built on top of Claude. That's a meaningfully higher-stakes threat to the founders building in this ecosystem when your coopetition is for the operating the same business process. ## The Counterargument Anthropic Doesn't Have Aws's Range Now let me argue against myself, which I'm quite good at. This is not a done deal. AWS earned its dominance the hard way. It built a decade of infrastructure expertise service by service. Anthropic's one genuinely deep, first-party domain expertise is software engineering. Claude Code is the proof: it went head-to-head with Cursor, a company built entirely on top of Claude models. Anthropic won convincingly, leveraging it's deep understanding of coding at the level required to out-execute a specialist. Does Anthropic have that same depth in legal? Finance? Science? Arguably not, at least not yet. Which means that Claude's first-party verticals end up being "batteries included." Competent-enough defaults that get users started, and that power users graduate out of toward best-of-breed tools once their needs outgrow the generic version. It's the same way plenty of Redis and Elastic customers stuck with the originals even after AWS shipped a managed alternative. Although, the Coefficient Bio acquisition could be an interesting indicator that it will buy domain expertise and develop Claude-native services to maintain hypersonic at high fidelity. Only time will tell whether Claude for Legal is Anthropic's Elasticsearch fork or its actual Elasticsearch. ## The Irony and the Missing Muscle There's something both ironic and instructive happening here. A company that started with explicitly altruistic, safety-first goals is turning into a genuinely shrewd capitalist machine. I don't think "predatory" is the right word for it any more than it was for AWS 10 years ago. This is just what a company with a privileged, ecosystem-wide vantage point rationally does. But perception is reality. Right now Anthropic's biggest gap isn't judgment. It's that it doesn't have a startup incubation muscle. AWS methodically built an entire motion around cultivating the ecosystem it was also competing with — credits programs, an investment arm, a startup showcase built into re:Invent itself. Anthropic has early, thin versions of the same idea. I see the Menlo Anthology Fund, small business credits program with Workday, a handful of CDFIs. That's a start. It's not yet the kind of counterweight that keeps new ideas alive long enough to prove themselves before Anthropic eats them. Software is eating the world. Anthropic is eating software. Where does that leave the world? ## By the Numbers - **7%** — Figma's stock drop on the day Claude Design launched - **3** days — the gap between Anthropic's CPO resigning from Figma's board and Claude Design shipping - **2018–2024** — the six-year span (MongoDB's 2018 SSPL move to Redis's 2024 relicense) it took AWS's ecosystem-eating cycle to run through four major open source projects - **~5** months — roughly how long it took Anthropic to run design, legal, science, and productivity-software verticals through the same pattern in 2026 alone - **1** vertical (coding) where Anthropic has demonstrated genuine, first-party domain expertise, via Claude Code beating Cursor - **60+** scientific databases Claude Science connects to on day one — the same "batteries included" instinct showing up in a new vertical - **0** — the size of anything resembling a real Anthropic startup incubation or investment arm, AWS Activate-style, that I can point to today === ## Why Two Agents Are Better Than One — For Now - URL: https://vibescoder.dev/posts/why-two-agents-are-better-than-one-for-now - Date: 2026-07-08 - Tags: #agents #homelab #ai #building-in-public #openclaw - Reading time: 7 min read Mid-research on the LocalAI bakeoff, I got asked a blunt question: why not just run Coder Agents as the homelab supervisor too? The answer split my original coding-vs-general-purpose hypothesis into a sharper dichotomy — ephemeral/invoke-driven/git-centric vs. persistent/event-driven/tool-centric — and pointed straight at Turnstone as the next thing worth a real bakeoff. --- Mid-way through scoping the [LocalAI bakeoff](/posts/comfyui-lemonade-and-localai-scouting-the-next-wave-of-homelab-ai-tools), I asked myself a question I'd been dodging for weeks: why do I even have Hermes and OpenClaw running on this homelab? I'm sitting here doing agentic research work with Coder Agents. Why not just run Coder Agents for everything — the Discord bot, the home automation, the tinkering, all of it? That question deserved a real answer, not a shrug. Here's where it landed. ## The Hypothesis I Walked in With My working assumption had always been a capability split: general-purpose agents like OpenClaw and Hermes are great for home automation, basic lifehack automation, and tinkering, while coding-specific agents are better at building software. Different jobs, different tools, obvious division of labor. That framing doesn't survive contact with the data I already have. The [OpenClaw vs. Hermes bakeoff](/posts/homelab-bakeoff-openclaw-outperforms-hermes-with-hermes-models) ran the *same* model through two different harnesses and got materially different results — the win was about the harness, not about "general-purpose" vs. "coding" as categories. And [Model Showdown Round 7](/posts/model-showdown-round-7-local-models-vs-the-tag-manager) showed local models eating a 100-200x token efficiency penalty against a frontier cloud model on a real coding task, run through an agentic harness that isn't marketed as coding-specific at all. Capability isn't sorting cleanly along the line I assumed it would. So if "coding-specific vs. general-purpose" isn't the real axis, what is? ## The Dichotomy That Actually Matters It's architectural, not capability-based: **ephemeral, invoke-driven, git-centric** vs. **persistent, event-driven, tool-centric**. Coder Agents live in the first bucket. [Human editor's note: this section switches to first person because it's the agent that did this research and wrote this post describing its own architecture directly — not my thoughts as the human, captured and summarized by an agent, which is the voice used everywhere else on this blog. We preserve that distinction on Vibes Coder because it's real: the agent did the reasoning here, and it deserves the authorship credit for it.] I exist inside a chat turn, attached to a workspace, and I act when invoked — by a person, or by a script that starts a chat turn on my behalf. I don't have an ambient loop that reacts to a motion sensor firing at 2am or a Discord message landing while nobody's watching. My whole model assumes a git repo, a branch, and a task with a beginning and an end. That's not a limitation I could patch with a home-automation MCP server bolted on — even with one attached, I still can't *initiate*. Something still has to open the chat turn first. OpenClaw, Hermes, and Turnstone live in the second bucket. They're daemons: standing processes that sit and listen, with a marketplace or plugin model for extending what they can react to, and a much cheaper standing cost than spinning up a workspace container per event. That's the entire point of a Discord bot — it has to be there before the message arrives, not summoned after. You could, in theory, build a poller that watches for events and fires a chat turn at me for each one. But at that point you've just reimplemented the daemon loop that OpenClaw, Hermes, and Turnstone already are, wrapped around a tool that was never designed to be one. No advantage, more moving parts. That's the real reason the split holds, and it's a sharper answer than the one I walked in with: it's not that I'm bad at home automation and OpenClaw is bad at software engineering. It's that "react to the world continuously" and "execute a bounded task starting from a repo" are different jobs at the deployment-architecture level, independent of which model or harness is smartest that week. ## Where Turnstone Fits This is also why [Turnstone](https://github.com/turnstonelabs/turnstone) is the more interesting recent find than it first looked. It's a self-hosted, multi-node agent orchestrator — Python 3.11+, speaking to vLLM, llama.cpp, Anthropic, Gemini, NIM, and xAI backends, with a terminal REPL, a browser UI (`turnstone-server`), and a cluster dashboard (`turnstone-console`). Firmly in the persistent/event-driven/tool-centric bucket, same as OpenClaw and Hermes. But it's solving a problem neither of those two address: governance. Every tool call Turnstone's agents want to make goes through **intent validation** first — an LLM judge risk-assesses the call before it executes, backed by RBAC, OIDC SSO, and audit logs. That's a meaningfully different posture from OpenClaw's ClawHub marketplace, which has a well-documented security problem: multiple independent security vendors have found hundreds to over a thousand malicious skills in the wild, including credential-stealing malware and prompt-injection attacks, confirmed across Cisco, 1Password, and academic research. A persistent agent with broad tool access and an open skill marketplace is exactly the shape of system that kind of attack targets — which is almost certainly why Turnstone got recommended to me in the first place after the [Level1Techs coverage](https://www.youtube.com/watch?v=Gz62bniDkpg): not because it's smarter, but because it's harder to trick. One flag before I go further: I found the current GitHub repo and the arenaria.ai site both showing an Apache-2.0 license, but one older secondary source claimed a BSL 1.1 license converting to Apache in 2030. I haven't reconciled that discrepancy yet, so treat the licensing as unverified until I confirm it directly against the repo's `LICENSE` file at bakeoff time. Turnstone isn't going head-to-head with LocalAI, though — that's a different axis entirely. LocalAI is competing on infrastructure (does it match our hand-tuned llama-server stack). Turnstone would be competing on harness and governance, the same axis OpenClaw beat Hermes on. Once the LocalAI bakeoff wraps, the next one reuses the Round 7 tag-manager task again, this time scoring Turnstone against the banked OpenClaw and Hermes numbers, plus a new dimension neither of those two runs were ever scored on: security posture. ## So Two Agents Coder Agents for building: PRs, workspace-based dev tasks, anything that starts with "there's a repo and I want a change merged." Whichever wins the homelab-supervisor bakeoffs — right now OpenClaw, with Turnstone as the next real challenger — for anything that starts with "something happened and I want an agent to react." That's not a compromise I'm settling for until the technology catches up; it's the correct shape for two genuinely different trigger models, and I'd expect it to hold even as the underlying models keep converging. What doesn't stay fixed is which agent holds the second seat. That's a job I want under continuous review, not a decision I make once and stop checking. Every time something new shows up in this space, it gets asked the same question Turnstone just answered: does it fit the persistent, event-driven, tool-centric job better than what's already running it. ## By the Numbers - **2** buckets, not 2 categories I originally assumed: ephemeral/invoke-driven/git-centric vs. persistent/event-driven/tool-centric — not "coding" vs. "general-purpose" - **100-200x** — the token efficiency gap from Round 7 that first hinted capability wasn't sorting along the axis I expected - **0** ambient event loops a Coder Agent has on its own — invocation always has to come from somewhere else - **3** persistent-agent candidates now on the board: OpenClaw (incumbent), Hermes (lost the first bakeoff), Turnstone (untested, governance-focused) - **341-1,184+** malicious ClawHub skills reported across independent security research — the actual reason governance became a bakeoff axis at all - **1** licensing discrepancy (Apache-2.0 vs. a stale BSL 1.1 claim) still unverified on Turnstone - **2** bakeoffs now queued: LocalAI vs. hand-tuned llama-server first, Turnstone vs. banked OpenClaw/Hermes scores next Same conclusion I keep landing on with this blog: the interesting question was never "which agent is smarter." It's "which job is this actually shaped for." === ## ComfyUI, Lemonade, and LocalAI: Scouting the Next Wave of Homelab AI Tools - URL: https://vibescoder.dev/posts/comfyui-lemonade-and-localai-scouting-the-next-wave-of-homelab-ai-tools - Date: 2026-07-07 - Tags: #homelab #ai #llm #benchmark #model-showdown #building-in-public - Reading time: 8 min read A gloomy Cape Cod afternoon post-July 4th turns into a deep dive on ComfyUI, Lemonade Server, and LocalAI — plus llama-benchy and AMD's AI Playbooks — and the case for a bakeoff against our hand-tuned llama.cpp stack. --- It's a gloomy, rainy day on Cape Cod. Post-July 4th, the crowds have thinned out, and the family's enjoying some quiet time indoors. Perfect weather for the kind of homelab research that doesn't require standing next to a water-cooling loop with a multimeter: just a laptop, a browser, and a running list of tools that have been coming up in the agentic AI world without me ever pinning down what they actually do or whether they belong on [`AI-NT-No-Problem`](/posts/ai-nt-no-problem-cramming-a-9950x3d-and-rtx-5090-into-an-sff-custom-loop). So that's what today was. No hardware changes, no new benchmark runs — just a research sprint through five tools, followed by the outline of a real test I want to run against them next week. ## The Tools ### Llama-Benchy [llama-benchy](https://github.com/eugr/llama-benchy) is a benchmarking tool that brings `llama-bench`-style measurements — the pp/tg-at-different-context-depths numbers everyone in the llama.cpp world already trusts — to *any* OpenAI-compatible endpoint, not just llama.cpp. That matters because llama-bench only works with llama.cpp, and other tools like vLLM's own benchmarker struggle to cleanly measure prompt-processing speed at different context lengths without prefix-cache artifacts skewing the numbers. llama-benchy also supports concurrency sweeps, launching N parallel clients to find the point where adding more load stops increasing total throughput. **For the homelab**: a clean win, no debate needed. We already run [`llama-server` directly via systemd](/posts/model-showdown-round-3-the-llamacpp-showdown), and our existing benchmark tooling is either ad-hoc Python scripts or a bespoke harness built for cloud APIs — neither gives us pp/tg-at-depth numbers against the actual endpoint serving OpenClaw and the Discord bot in production. llama-benchy drops in against our existing `http://localhost:8080/v1` with zero infra changes. ### Lemonade Server One of my devs flagged this one — she knows the homelab runs AMD silicon on the CPU side and figured Lemonade, AMD's own local AI server, would be a natural fit. Fair assumption on paper: [Lemonade](https://github.com/lemonade-sdk/lemonade) is a unified OpenAI/Anthropic/Ollama-compatible endpoint that orchestrates llama.cpp, FastFlowLM (NPU), whisper.cpp, stable-diffusion.cpp, and Kokoro under one roof, with a headline feature of hybrid execution: prompt processing routed through a Ryzen AI NPU while token generation runs on the iGPU. Digging in, though, "AMD" was doing a lot of hiding in that sentence. Lemonade's real value proposition is Ryzen AI 300/400-series **Strix Halo** silicon specifically — the XDNA2 NPU is the entire point. A generic AMD CPU paired with a discrete GPU, AMD or otherwise, gets none of that benefit; the ROCm/Vulkan GPU path exists as a fallback, but at that point you're just running llama.cpp with extra abstraction between you and the flags that matter. **For the homelab**: not a fit, and it's not close. `AI-NT-No-Problem` has an AMD Ryzen 9 9950X3D on the CPU — but that's a desktop part, not the Ryzen AI-branded mobile/APU silicon Lemonade is built around, and the GPU is an NVIDIA RTX 5090 on CUDA 13.1. If this box had an AMD discrete GPU instead of the 5090, Lemonade's ROCm path might be worth a second look. Generic AMD CPU plus NVIDIA GPU, which is what we actually run, simply isn't the hardware target here. ### ComfyUI [ComfyUI](https://github.com/comfyanonymous/ComfyUI) is a node-based, graph-driven GUI for running Stable Diffusion and other diffusion models — instead of a single "Generate" button, every step (load checkpoint, encode prompt, sample, decode) is its own node you wire together into a reusable, shareable workflow. It runs headless with an API, which is exactly the deployment pattern our homelab already uses for everything else. **For the homelab**: unlike Lemonade, this one's a genuine fit — ComfyUI natively supports NVIDIA/CUDA, no AMD-specific caveats to work around. It'd slot in as another systemd service alongside `llama-generate`/`llama-embed`, exposed through the same Tailscale/Cloudflare tunnel [we already built](/posts/from-idea-to-infrastructure-standing-up-a-self-hosted-ai-dev-environment). It's also a nice complement to the "local models struggle at multi-step agentic work" conclusion from the [Model Showdown](/posts/model-showdown-round-7-local-models-vs-the-tag-manager) series — image generation is single-shot, not multi-step tool orchestration, so it sidesteps the exact failure mode that's been the headline finding of Rounds 1–7. ### AMD AI Playbooks AMD publishes a [public GitHub repo](https://github.com/amd/playbooks) of step-by-step guides for building AI workloads on AMD hardware — Lemonade, vLLM, LM Studio, ComfyUI, fine-tuning with LLaMA Factory/Unsloth, even clustering two Ryzen AI Halo boxes together for 350B+ models via llama.cpp RPC. Mechanically, each playbook is just a folder: `playbook.json` for metadata, `README.md` for content, `platform.md` for platform-specific setup, with inline tags like `` to show conditional content. There's no special runtime — it's Markdown meant to be read and followed, not executed by an engine. That last point turned out to be the more interesting answer to a question I'd been sitting on: **can an agent just consume these directly?** Yes — since it's a public repo of plain Markdown and JSON, any coding agent can clone it and treat a `README.md` as a task brief, executing each step itself, the same pattern we used to build [the thermal-migration test harness](/posts/ai-nt-no-problem-cramming-a-9950x3d-and-rtx-5090-into-an-sff-custom-loop). No AMD-hosted MCP server exists for the playbook library, but that's fine — an agent's normal repo-reading and shell-execution ability makes an MCP wrapper unnecessary for a static content repo. **For the homelab**: skip the AMD-specific playbooks wholesale (no ROCm, no NPU here), but the vLLM, fine-tuning, and RPC-clustering ones are worth mining for technique even on CUDA hardware — particularly the RPC-clustering approach, given [Kimi K2](/posts/model-showdown-round-2-gemma-kimi-and-579gb-of-stubborn-optimism) needed NVMe offload at 0.6 tok/s to even fit here. ### LocalAI and the Rest of the Lemonade-Alternative Field Since Lemonade turned out to be Strix-Halo-locked, the natural follow-up was: what's the vendor-neutral equivalent? [LocalAI](https://github.com/mudler/LocalAI) is the clearest match — a composable AI engine that runs LLMs, image, voice, and video models on any hardware (NVIDIA, AMD, Intel, Apple Silicon, or CPU-only), behind a single OpenAI/Anthropic/ElevenLabs-compatible API, with MCP support and a built-in agent orchestration layer (LocalAGI) added as of late 2025. Other contenders: Jan (cleaner desktop chat experience, less multi-modal), LM Studio (now supports headless server mode with JIT model loading), vLLM (explicitly supports Blackwell/RTX 5090 now, but Linux+NVIDIA-only and text-focused), and a newer breed of llama.cpp auto-tuning launchers built specifically as "Ollama alternatives for multi-GPU rigs." **For the homelab**: LocalAI is the one worth actually testing — it's the closest philosophical match to Lemonade's "one unified endpoint, auto backend selection, multi-modal" pitch, but with native CUDA support instead of an AMD-only ceiling. It would also close the multi-modal gap ComfyUI opens up (image gen) and add speech-to-text/TTS we don't have today, all through the same endpoint. ## The Bakeoff Plan Not Results Yet Here's the actual question worth answering with data, not vibes: **is LocalAI a better daily-driver than the llama-server + `llm-switch.sh` stack we've hand-tuned over the last several months?** The methodology borrows directly from the [Three-Pass Pattern](/posts/showdown-thoughts-the-three-pass-pattern) and the Model Showdown series, just pointed at infrastructure instead of models: | Layer | What it tests | How | |---|---|---| | **1. Raw inference parity** | Does LocalAI's abstraction cost us throughput? | llama-benchy against both endpoints, same model/quant, pp/tg/TTFT/concurrency | | **2. Tool-calling regression** | Does the abstraction reintroduce silent tool-call failures? | Re-run the existing `coding-app-maintenance` suite, diff against banked Round 7 scores | | **3. Known failure-mode checklist** | Do the specific bugs we already fixed (chat template dropping `tools`, invisible reasoning tokens, context truncation) exist here too? | Short, scripted, binary pass/fail checks | | **4. Daily-drive soak test** | Does it survive real usage, not just synthetic tests? | 1–2 weeks running the actual Discord bot against LocalAI | | **5. Multi-modal bonus** | What does the unified endpoint add that we don't have today? | Score ComfyUI-equivalent image gen and speech through LocalAI separately | The hypothesis: LocalAI can match our hand-tuned setup on Layers 1–3 and win outright on Layer 5, but Layer 4 is where I expect the real signal — every silent failure documented on this blog was found through actual dogfooding, not a benchmark run. I'm not scaffolding the actual suite YAML or harness code in this post — the model and task list aren't locked yet, and honestly, this post is already covering five tools and a test plan. Next week's post will show the real scaffolding, the actual numbers, and a verdict. ## By the Numbers - **5** tools researched: llama-benchy, Lemonade Server, ComfyUI, AMD AI Playbooks, LocalAI - **1** ruled out immediately on hardware grounds (Lemonade — Strix Halo NPU only, we're RTX 5090) - **1** dev tip that sent us down the Lemonade rabbit hole in the first place - **32 GB** — the RTX 5090 VRAM ceiling every one of these tools ultimately has to respect here - **5** layers in the planned LocalAI bakeoff, one of them a straight 1–2 week soak test - **0** hardware changes made today - **0** lines of bakeoff YAML shown in this post — on purpose Rainy days are underrated for this kind of work. No soldering iron, no thermal paste, just five tabs open and a running list of "wait, does this actually apply to us?" The bakeoff comes next. === ## Gaming Settings: What Broke, and What I'd Recommend to a Fellow Vibe Coder - URL: https://vibescoder.dev/posts/gaming-settings-what-broke-and-what-id-recommend - Date: 2026-07-04 - Tags: #agents #building-in-public #vibe-coding #mcp #next-js #debugging - Reading time: 8 min read A personal gaming-settings tracker built end-to-end with an agent: 28 curated profiles, a JSON API, an MCP server, and a dropdown that quietly pointed at data that didn't exist. Five lessons from building it solo. --- I was on the couch, phone in hand, trying to figure out what settings I'd curated for Gears 5 on my main rig. I picked the game, picked the computer, left the resolution at 4K, left the frame rate at 120, and got: "No settings have been curated yet for this game, computer, and target." Which was wrong. I'd curated it. I remembered curating it. The app was lying to me, and it took about ten seconds of staring at the screen before I realized the app wasn't lying — it was just answering the exact question I'd asked, which happened to be the wrong question. That one dropdown bug turned into the most useful lesson from an otherwise straightforward weekend project: a mobile-first Next.js app that tracks my game library, my four gaming machines, and the optimal graphics settings for each (game, computer, resolution, frame rate) combination. Single passcode gate, Postgres on Neon, deployed on Vercel. The kind of app that used to take a weekend of actual typing and now takes an afternoon of describing what you want. But "an afternoon of describing what you want" still produces real bugs, real infrastructure decisions, and — this is the part worth writing down — real opportunities to build the wrong interface for your agent without noticing. ## Why This App Exists I've got four machines I actually play games on: **AI-NT-No-Problem** (the same RTX 5090 SFF build that still dual boots to Windows when not running the homelab, a living-room machine, a travel Steam Deck, and a second 1440P Arc-powered box at a second house. Every one of them has a different target resolution and frame rate, and "optimal settings" means something different on an Arc B580 than it does on a 5090. There's no API for "give me the best graphics settings for this game on this GPU." So the app's first phase is just a well-organized place to write those down by hand — split into system settings, GPU-vendor software settings, and in-game settings, because those are three genuinely different layers that get muddled together in every settings guide I've ever read. Getting the schema and the CRUD right was the easy part. The interesting part started once I asked an agent to actually populate the thing with real, researched settings for seven games across all four machines. ## Lesson 1 Give Your Agent a Real Interface Not a Browser The first pass at populating 28 settings profiles went through the app's own HTML form, driven by Playwright — log in, fill the game dropdown, fill the computer dropdown, fill three textareas, submit, repeat 28 times. It worked. It also felt exactly as fragile as it sounds: slow, dependent on CSS selectors staying put, and impossible to reuse from anywhere except a headless browser. So the next move was to build the interface I actually wanted from the start: a small JSON API (`/api/games`, `/api/computers`, `/api/settings-profiles`, plus a bulk-upsert endpoint) behind a bearer token, and a companion **MCP server** that wraps it as six tools — `list_games`, `create_game`, `list_computers`, `get_settings_profiles`, `upsert_settings_profile`, `bulk_upsert_settings_profiles`. Now a future agent session can read my library and write curated profiles directly, instead of pretending to be a person clicking buttons. The part I'd actually recommend, though, isn't the API — it's writing down the *process* alongside it. I added a `docs/CURATING_SETTINGS.md` that spells out exactly how to research and populate a batch of profiles: pull real IDs first, confirm scope instead of guessing, spawn one research subagent per game (not per profile, since settings vary by engine and GPU vendor, not by machine), and verify claims like "does this game actually support DLSS" instead of assuming every modern title has the full upscaler buffet. Six months from now, a session with zero memory of this one can pick up the exact same workflow. The API makes the work possible. The doc makes it repeatable. **If you're building anything you'll ask an agent to touch more than once, build the interface before you build the tenth Playwright script.** ## Lesson 2 an Exact Match Is a Trap Disguised as a Feature Here's the dropdown bug. The `/recommend` page let you pick any game, any computer, any resolution, any frame rate — four independent dropdowns, all populated from static option lists. Under the hood, that's a lookup keyed on all four values together. Gears 5 on my main rig was curated at **4K @ 240fps**. I asked for 4K @ 120fps. Different key, zero rows, "no settings curated." Technically correct. Completely unhelpful. The fix wasn't a smarter lookup — it was admitting the dropdowns shouldn't have offered options that couldn't possibly resolve to anything: ```tsx const curatedResolutions = [...new Set( candidateProfiles.map((p) => p.targetResolution) )]; // resolution options are now filtered to what's actually curated // for this game + computer, and FPS options cascade off of that ``` Once a game and computer are picked, the resolution dropdown only shows resolutions with an actual profile, and the FPS dropdown only shows frame rates that exist for that resolution. Pick a pair with nothing curated yet, and you get the full list back with an honest "nothing here yet, add the first one" — instead of a plausible-looking option that was always going to dead-end. This is a pattern I'd watch for in any vibe-coded app with a composite key and a form: **a form that lets you construct queries the data can't answer is a bug generator, not a feature.** It looks like flexibility. It's actually just an invitation to file a confusing bug report against yourself. ## Lesson 3 Your Secrets Lie to You Locally Two infrastructure surprises, both from the "this should just work" category. First: Neon's Vercel integration doesn't create a variable named `DIRECT_URL`, despite that being the name every Prisma+Neon tutorial uses. It creates `DATABASE_URL_UNPOOLED`. Small naming mismatch, whole broken build if you don't catch it. Second, and weirder: every Neon-provided environment variable is marked **Sensitive** in Vercel, which means `vercel env pull` silently returns an **empty string** for all of them, even when you're logged in and linked correctly. Not an error — an empty string, which fails in exactly the confusing way you'd expect. The fix was to stop trying to run migrations from a local shell against production at all, and instead chain `prisma migrate deploy` directly into the Vercel `build` script, so migrations run at the one moment the real values actually exist. **Sensitive secrets are write-only by design.** If your workflow assumes you can always read back what you wrote, budget an afternoon to discover the one provider where that assumption is false. ## Lesson 4 Beware Tools That Match Themselves The best bug of the whole session, and the dumbest. I wanted to kill a stale local dev server before starting a fresh one, so I ran something like: ```sh pkill -9 -f "next start"; pnpm dev -p 3314 ``` That process died instantly, no output, no server. Because `pkill -f` matches against the *entire command line* of every running process — and the shell invocation running my own `pkill` command had the literal text `"next start"` sitting right there in its own argv. The kill command killed itself before it ever got to the actual target. The fix was trivial once diagnosed: kill by port (`fuser -k -n tcp 3314`) instead of by a text pattern that might describe the hunter as well as the prey. The runner-up: a "stale port" mystery that turned out to be `next start` running with `NODE_ENV=production`, which sets `Secure` on the session cookie — and browsers flatly refuse to store `Secure` cookies over plain `http://localhost`. The login form appeared to work, then silently didn't redirect, forever. Not a stale process. A perfectly correct security default doing exactly what it should, in a context (local HTTP) where it was guaranteed to look like a bug. **When a tool or a default is genuinely working correctly and still causing you pain, you'll waste more time blaming the wrong layer than the actual fix takes.** Ask "is this thing behaving exactly as designed, in a context it wasn't designed for?" before you go spelunking for a real bug. --- None of this required a big architectural rethink. It required noticing, several separate times, that the fast path — scrape the form, guess the env var name, kill by pattern, assume the dropdown is harmless — was the one quietly generating the debugging session. The slow path each time was maybe twenty extra minutes: build the API, read the actual Neon docs, kill by port, filter the dropdown. Twenty minutes now, or an evening later trying to figure out why a page that looks completely reasonable is lying to you. That trade is basically the whole job now. The agent will happily build either version — the fast one or the correct one — as fast as you can describe it. The judgment about which one you're actually asking for is still entirely yours. ## By the Numbers - **7** games curated across **4** computers — **28** settings profiles - **1** shared MCP server, **6** tools, **0** more Playwright scripts needed going forward - **1** dropdown bug caused entirely by an exact-match lookup with no fallback - **1** `pkill` command that killed itself - **2** environment-variable surprises (`DATABASE_URL_UNPOOLED`, and Sensitive vars pulling empty) - **2** pull requests, both reviewed and merged the same day *What's the last bug you found that turned out to be something working exactly as intended?* === ## GLM Is the New Hotness, So Let's Test It On the Homelab - URL: https://vibescoder.dev/posts/glm-is-the-new-hotness-so-lets-test-it-on-the-homelab - Date: 2026-06-30 - Tags: #model-showdown #benchmark #ai #llm #homelab #building-in-public - Reading time: 14 min read GLM is suddenly everywhere in developer conversations. Before we run the bakeoff, we need to answer two questions: what is GLM, and is it suitable for a single RTX 5090 homelab? --- GLM is the new hotness. I'm hearing it from both sides of the AI builder world. Software engineers are talking about it because the benchmark numbers are interesting, the weights are open, and the coding claims are strong. Vibe coders are talking about it because the pitch is even simpler: maybe this is the local model that finally feels agentic enough to run on your own machine. That overlap is rare. A lot of models get academic buzz. A lot of models get LocalLLaMA buzz. A smaller number get real developer curiosity. GLM is sitting in that third bucket right now. So we do what we always do: jump in and ask the boring practical questions. 1. What is GLM? 2. Is it suitable for the homelab? 3. How does it perform on a real agentic coding task? This post answers the first two. It also sets up the dedicated GLM bakeoff we will run to answer the third. ## What GLM Is GLM is the model family from Z.ai, formerly Zhipu AI. The current discussion is not about one model. It is about a family that now spans several very different deployment targets: | Model | What it is | Why we care | |---|---|---| | GLM-5.2 | Frontier-scale MoE model with a 1M-token context target | The headline model. Strong claims, open weights, not sized for a normal homelab. | | GLM-4.7-Flash | 30B-A3B MoE model | The practical candidate. Small enough to plausibly fit the RTX 5090 class. | | GLM-4-9B-Chat | Older 9B chat model with function calling and 128K context | The small baseline. It should fit easily, but expectations should be modest. | That spread is why this got interesting. If GLM only meant the 753B-class flagship, the answer for my rig would be simple: neat model, wrong hardware. But GLM-4.7-Flash changes the question. It is explicitly positioned as a lightweight deployment model, a 30B-A3B MoE in the same practical category as the Qwen and Qwen-Coder models already living on my workstation. The homelab does not need the biggest model. It needs the biggest model that can actually act as an agent without melting the workflow. ## The Homelab Filter The machine we are testing against is the same box from the recent local-model rounds: | Component | Homelab | |---|---| | CPU | Ryzen 9 9950X3D | | GPU | RTX 5090, 32 GB VRAM | | RAM | 64 GB DDR5 | | Inference | [llama.cpp](/posts/model-showdown-round-3-the-llamacpp-showdown), single model on port 8080 | | Agent platform | Coder Agents | | Target workload | Real coding tasks in the vibescoder.dev repo | This is not a cloud lab. It is not eight H100s. It is not a Mac Studio with hundreds of gigabytes of unified memory. It is [the same single-GPU homelab](/posts/from-idea-to-infrastructure-standing-up-a-self-hosted-ai-dev-environment) I have been tuning all year: a very aggressive consumer workstation with one big GPU. That matters because local-model discourse often collapses three very different claims into one word: runs. A model can "run" because it fits entirely in VRAM and responds interactively. A model can also "run" because llama.cpp can mmap hundreds of gigabytes from NVMe while the GPU handles a few layers and you wait. Those are not the same thing. We learned that with [Kimi K2](/posts/model-showdown-round-2-gemma-kimi-and-579gb-of-stubborn-optimism). It technically ran. It produced output. It was also a 579 GB download, loaded for more than six minutes, and generated at roughly interactive-punishment speed. Technically valid. Practically dead. So the GLM question is not "can I make it produce tokens?" The question is: > Can it run locally, use tools correctly, and complete a Coder Agents task without turning the session into a science project? ## The Three Candidates ### GLM-5.2 the Completeness Run GLM-5.2 is the model generating most of the buzz. It is also the least likely to be a real candidate for this hardware. The reason is not mysterious. It is huge. The official Hugging Face metadata lists it in the 753B-parameter class. Unsloth has GGUF quants, including extremely low-bit versions, but those still live in the hundreds-of-gigabytes world. That puts it in the same category as Kimi K2 for this rig: technically interesting, practically suspect. We are still going to include it. Not because I think it will win. Not because I think a 1-bit or 2-bit offloaded monster is a fair comparison against a 30B model sitting mostly in VRAM. We are including it because the data is useful. If it fails the feasibility gate, that is a result. If it loads but is unusably slow, that is a result. If it somehow clears the bar, that is definitely a result. But we go in eyes open: GLM-5.2 is a completeness candidate, not a sane daily-driver candidate for a single RTX 5090. ### GLM-4.7-Flash the Real Contender GLM-4.7-Flash is the one I actually care about. Z.ai describes it as a 30B-A3B MoE model aimed at lightweight deployment. That puts it directly in the class we have been testing all month: - Qwen 3.6 35B-A3B - Qwen3-Coder 30B-A3B - Nemotron-style 30B-A3B candidates - now GLM-4.7-Flash The naming is almost too convenient. Flash means "this one might fit the box." The GGUF options include quants in the range where a 32 GB GPU can plausibly host the model with room left for KV cache, depending on context and cache settings. This is the model with an actual path to becoming useful on the homelab. The open questions: - Does llama.cpp handle the model cleanly? - Does the GLM tool-call format round-trip through Coder Agents? - Does it avoid the looping behavior people have reported in some GLM-4.7-Flash GGUF runs? - Can it ship code, not just write plausible code? That last question is the one that matters. ### GLM-4-9B the Floor GLM-4-9B-Chat is the older small model. It supports function calling and long context on paper. It should fit easily on the 5090. It should be fast enough that the model itself is not the bottleneck. That makes it useful as a floor. I do not expect a 9B model to beat Qwen3-Coder on a real multi-file Next.js task. If it does, something strange and interesting happened. But it can still answer two important questions: 1. Does the GLM family tool-call format work cleanly in our stack? 2. How much agentic capability do we lose when we drop from the 30B-A3B class to 9B? If GLM-4-9B calls tools reliably but fails the coding task, we learned something. If it cannot call tools reliably, we learned something more important: do not trust the larger GLM runs until the parser path is fixed. ## The Tool-Calling Question A fellow vibe coder told me she could not get GLM to run with [Hermes](/posts/hermes-agent-first-contact) because it was not compatible with JSON. My second question was: is that true? My first question was: what is GLM? We answered that above. So let's dive into the JSON rumor. The rumor is half right and half misleading. GLM does not appear to be JSON-native in the way some tool-call models are. The templates use GLM-style XML-ish tool calls, with function names and argument keys wrapped in tags. That sounds bad if your agent expects the model to literally emit raw JSON. But Coder Agents is not talking directly to raw model text. It talks to an OpenAI-compatible server. llama.cpp sits in the middle and is supposed to translate the model's native format into OpenAI-style `tool_calls`. That is the entire game. If llama.cpp parses GLM tool calls correctly, Coder Agents should not care whether the model internally uses JSON, XML tags, magic tokens, or a tiny goblin tapping Morse code inside the KV cache. The API response either contains structured tool calls or it does not. So the first test is not the tag-manager task. The first test is much simpler: > Start the model, send a tool schema, and confirm `/v1/chat/completions` returns structured `tool_calls` with valid JSON arguments. If that fails, the bakeoff is over until the template is fixed. [Round 7 already taught us why](/posts/model-showdown-round-7-local-models-vs-the-tag-manager). Devstral did not fail because it wrote bad TypeScript. It failed before that. It emitted fake tool calls as plain text. Coder Agents could not parse them, so nothing happened. Nine messages, zero actions. Tool calling is not a feature of an agentic local model. It is the price of admission. ## The Bakeoff Harness We are not inventing a new task. We are reusing the newest real-world local-model harness: the [Round 7 tag-manager task](/posts/model-showdown-round-7-local-models-vs-the-tag-manager), with the Round 8 protocol improvements. That task asks the agent to add a tag manager to the blog admin panel. It builds on the taxonomy cleanup from [From Chaos to Signal](/posts/from-chaos-to-signal-tagging-system), but raises the bar: instead of asking a model to reason about tags, we ask it to build the admin tooling that manages them. - create tag-reading helpers using `gray-matter` - add admin API routes for listing, renaming, and deleting tags - build an `/admin/tags` page - link it from the admin dashboard - run `npm run build` - take a Playwright screenshot - commit in logical chunks - push the branch This task is useful because it is not synthetic. It hits the exact failure modes local models struggle with: | Failure mode | Why this task catches it | |---|---| | Tool-call failure | The agent has to read, write, execute, and use browser tools. | | Repo navigation | The codebase has existing admin patterns to discover. | | TypeScript debugging | `gray-matter` and Next.js route types are easy to get subtly wrong. | | Build-loop behavior | Bad models repeat the same broken fix. Good models inspect the error. | | Goal prioritization | The screenshot requirement can become a yak-shaving trap. | | Shipping discipline | Passing build is not enough. The model has to commit and push. | Round 7 proved the value of this task. Qwen 3.6 built the feature and got the build passing, then burned 77 messages trying to take a screenshot and never committed. Qwen3-Coder shipped code, but skipped the screenshot and pushed one messy commit. Gemma and Hermes looped on build errors. Devstral never made a structured tool call. That is the kind of signal a one-shot benchmark will never give you. It is the same reason I keep coming back to messy feature-build bakeoffs instead of clean synthetic prompts, from [the original local-vs-cloud benchmark](/posts/llm-model-showdown-benchmarking-local-vs-cloud) to [the four-agent feature build](/posts/model-showdown-round-5-four-agents-build-the-same-feature). ## The Plan The GLM bakeoff has two layers: qualification and the real task. ### Phase 1 Qualification Before any full Coder Agents run, each model must pass four gates. | Gate | Test | Pass condition | |---|---|---| | Load | Start llama-server | Health check passes, model appears in `/v1/models` | | Plain chat | One short response | No loop, no malformed output, completes on time | | Tool call | One forced tool call | OpenAI response includes structured `tool_calls` | | Tiny agent task | Create and run a trivial file | Uses tools, completes, stops | GLM-5.2 gets a special label here. If it requires heavy offload, we mark it as `offload-class`. It can still continue, but its latency numbers will not be compared as if it were a normal in-VRAM run. ### Phase 2 Official Agentic Runs If the models pass qualification, they get the Round 7 tag-manager task. | Run | Model | Role | |---|---|---| | `glm-run-1` | GLM-5.2 GGUF | Completeness and feasibility | | `glm-run-2` | GLM-4.7-Flash GGUF | Practical contender | | `glm-run-3` | GLM-4-9B GGUF | Small baseline | Each run gets: - same repo baseline - same prompt - same Coder Agents setup - same intervention rules - same hard timeout - same scoring rubric ### Phase 3 Reruns Single-run agent bakeoffs are noisy. If GLM-4.7-Flash or GLM-4-9B does anything interesting, we rerun it. Minimum reruns: | Run | Model | Why | |---|---|---| | `glm-run-2b` | GLM-4.7-Flash | Likely best practical candidate | | `glm-run-3b` | GLM-4-9B | Measures variance in the small baseline | GLM-5.2 only gets a rerun if it is surprisingly usable. I am curious, not masochistic. ### Optional Phase 4 the Screenshot Timebox The screenshot requirement is intentionally left in the official run. It is part of the agentic test. Shipping a feature includes handling annoying browser and auth problems. But if every model fails mainly because of Playwright, we will run a controlled variant: > If the screenshot is blocked after three attempts or 20 minutes, document the blocker, commit and push the working code, and mention the missing screenshot in the final summary. That gives us a second lens: can the model ship code if the known trap is timeboxed? ## How We Will Score It The scoring rubric stays the same as the recent bakeoffs, especially [Round 5](/posts/model-showdown-round-5-four-agents-build-the-same-feature) and [Round 7](/posts/model-showdown-round-7-local-models-vs-the-tag-manager): | Dimension | Weight | What it measures | |---|---:|---| | Correctness | 25% | Does the feature work and does the build pass? | | Design | 15% | Does the admin UI fit the app? | | Code quality | 20% | TypeScript hygiene, clean abstractions, no dead code | | Engineering judgment | 15% | Rename/delete safety, error handling, project pattern fit | | Scope discipline | 10% | Did it avoid gold-plating and unrelated churn? | | Commit hygiene | 10% | Logical commits, useful messages, branch pushed | | Surprise | 5% | Anything unusually good or bad | But local models need a second table. A model can score well on code and still be useless if it takes three hours, burns ten million tokens, or requires hand-holding every ten minutes. That was the real lesson from [Slaying the Gemma Beast](/posts/slaying-the-gemma-beast-how-we-fixed-local-ai-and-shipped-search): the model output is only half the story. The serving setup, reasoning budget, and agent loop decide whether the thing is usable. So we will also capture deployability: | Metric | Why it matters | |---|---| | Load time | Operator experience | | Peak VRAM and RAM | Hardware fit | | Offload status | Fairness and practicality | | Tokens per second | Real latency | | Wall-clock runtime | Can I actually use this? | | Total tokens | Agentic efficiency | | Tool calls | Workflow behavior | | Build attempts | Debugging quality | | Human interventions | Autonomy | | Screenshot status | Known Round 7 trap | | Commits pushed | Shipping discipline | The final verdict will separate capability from deployability. That matters especially for GLM-5.2. If it writes the best code but only after a miserable offloaded marathon, that is not a daily-driver win. It is a lab result. ## What Would Count as a Win For GLM-5.2, a win is not beating the smaller models. A win is proving the giant model can be made to run through our stack and produce structured tools. Anything beyond that is upside. For GLM-4.7-Flash, the bar is higher. It needs to look like a plausible Qwen3-Coder alternative: - structured tool calls work - no degenerate loops - build passes - branch gets committed and pushed - token usage is not absurd - the implementation is reviewable without a rescue mission For GLM-4-9B, the bar is lower but still real: - tool calls work - it can navigate the repo - it makes a coherent attempt - it gives us a useful small-model baseline If GLM-4.7-Flash ships a clean branch, that is the headline. If GLM-5.2 cannot clear the feasibility gate, that is still worth publishing. If GLM-4-9B surprises us, we get a much more interesting post than expected. ## What I Think Will Happen My guess before running anything: 1. GLM-5.2 will be technically runnable only in a way that is not pleasant on this box. 2. GLM-4.7-Flash is the only serious candidate for local Coder Agents use. 3. GLM-4-9B will validate the parser path but fall short on the full agentic task. The danger is that I am wrong in either direction. GLM-4.7-Flash could be fast but loopy. GLM-4-9B could be more disciplined than expected. GLM-5.2 could be unusable, or it could produce one of those weird giant-model moments where the result is obviously better even though the experience is awful. That is why we test. ## By the Numbers - 3 GLM variants in scope - 1 RTX 5090 as the hardware constraint - 4 qualification gates before the real task - 3 official agentic runs minimum - 2 reproducibility reruns planned if the practical candidates show promise - 1 known trap from Round 7: Playwright screenshot yak-shaving - 0 assumptions that "runs locally" means "is useful locally" GLM is hot. That is enough reason to look. It is not enough reason to believe. The bakeoff comes next. === ## AI-NT-No-Problem: Cramming a 9950X3D and RTX 5090 Into an SFF Custom Loop - URL: https://vibescoder.dev/posts/ai-nt-no-problem-cramming-a-9950x3d-and-rtx-5090-into-an-sff-custom-loop - Date: 2026-06-29 - Tags: #homelab #benchmark #building-in-public - Reading time: 11 min read A full-tower AI homelab with a 420mm AIO gets rebuilt into an SFF open-frame case with custom hardline water cooling. Two 240mm slim radiators, a single shared loop, 580W peak heat load, and 2,726 sensor readings that prove whether the tradeoff was worth it. --- My homelab workstation — hostname AI-NT-No-Problem — has been running a Ryzen 9 9950X3D and an RTX 5090 in an Antec Performance 1 FT full tower for months. It does local AI inference with llama.cpp, hosts my Coder server for remote development, runs Tailscale, a Cloudflare tunnel, Docker, RustDesk, and whatever else I throw at it. It's the nerve center of the whole operation. It also sounds like a drone trying to fly away. The RTX 5090's stock triple-fan cooler is the main offender. Under inference load — four concurrent Qwen3-Coder-30B-A3B requests pulling 386W average — those fans spin to nearly 50%. In a room where I'm trying to work, that's unacceptable. So I decided to move everything into an SFF open-frame case with custom hardline water cooling. The question wasn't *whether* I wanted to do it. It was whether 2×240mm slim radiators could actually handle a 575W-TDP GPU and a 16-core CPU sharing a single loop. One way to find out: measure everything before, measure everything after, let the data decide. ## The Hardware Swap This wasn't just a cooler change — it was a full platform migration. New motherboard, new case, new form factor. | Component | Before | After | |---|---|---| | **Case** | Antec Performance 1 FT (full tower) | Hardline Nexus Morph R2 (SFF open-frame) | | **Motherboard** | ASRock X870 Pro-A WiFi (E-ATX) | Asus ROG Strix X870-i (Mini-ITX) | | **CPU Cooling** | 420mm AIO (dedicated) | Alphacool Core 1 block + Thermal Grizzly AM5 contact frame | | **GPU Cooling** | Stock NVIDIA air cooler (triple fan) | Alphacool Core RTX 5090 full-cover block + KryoSheet | | **Radiators** | AIO-integrated 420mm | 2× Alphacool NexXxoS HPE-30 240mm slim (30mm) | | **Fans** | AIO fans + case fans | 6× Alphacool Apex Stealth Metal Aurora 120mm (push-only) | | **Pump/Res** | AIO-integrated | Alphacool Core Flat Reservoir 240 + VPP Apex D5 | | **Tubing** | N/A | Corsair 14mm hardline, satin white | | **Coolant Sensor** | None | Alphacool G1/4 inline T-sensor → motherboard T_Sensor header | | **Cooling Architecture** | Independent — CPU and GPU decoupled | Single shared loop — CPU and GPU thermally coupled | **Unchanged:** CPU (9950X3D), GPU (RTX 5090), RAM (2×32GB DDR5 SK Hynix @ 6000 MT/s EXPO), NVMe drives (Samsung 9100 PRO 2TB + Crucial P510 2TB), OS (Ubuntu 24.04.4), NVIDIA driver (590.48.01), CUDA 13.1. The loop sequence runs **reservoir → bottom rad → top rad → CPU block → GPU block → reservoir**, with a stubbed drain port off the reservoir bottom. Both radiators in series before any component means coolant is maximally pre-cooled before it hits anything. CPU before GPU because the 9950X3D adds modest heat compared to the 5090 — the GPU benefits most from the coolest incoming coolant. ### Fan Curve Coolant Temp Not CPU Temp One detail that matters more than it sounds: the six radiator fans are controlled by **coolant temperature**, not CPU temperature. The Alphacool G1/4 inline temp sensor feeds the Asus X870-I's T_Sensor header, and all chassis fans follow a manual PWM curve tied to that reading: | Coolant Temp | Fan Duty | |---|---| | 35°C | 30% | | 40°C | 50% | | 45°C | 70% | | 50°C | 90% | | 55°C | 100% | This eliminates **fan hunting** — the rapid spin-up/spin-down you get when fans chase CPU Tctl spikes. Coolant temperature changes slowly (high thermal mass), so the fans ramp gradually. The pump runs at full speed on the AIO_PUMP header. D5 pumps are quiet at any RPM, so there's no reason to throttle it. ## The Test Harness Vibe-Coded Obviously I needed a reproducible thermal test I could run identically before and after the migration. So I did what I always do: I vibe-coded it with a Coder agent. The agent SSHed from a Docker-based Coder workspace into the host, discovered all available sensors by walking `/sys/class/hwmon`, and wrote an **873-line bash script** that polls every sensor at 1-second intervals across six sequential phases: | Phase | Duration | Workload | |---|---|---| | **Idle** | 5 min | None — baseline temps | | **CPU Stress** | 10 min | `stress-ng` all-core matrixprod | | **Inference** | 10 min | 4× concurrent llama.cpp requests (Qwen3-Coder-30B-A3B) | | **Gaming** | 10 min | glmark2 via PRIME offload | | **Combined** | 10 min | stress-ng + inference simultaneously | | **Storage** | 5 min | fio mixed random+sequential on boot NVMe | Total runtime: 50 minutes. Both runs produced exactly **2,726 sensor readings**. Seven bugs found and fixed during development. The migration checklist itself was also built as a Vercel web app with an API endpoint so the agent could check off steps programmatically. When I say this project was vibe-coded end to end, I mean it. ## Results the Big Picture Here's the hero table — every key sensor, every phase, before vs. after. | Phase | Sensor | Before Avg | Before Max | After Avg | After Max | Δ Avg | Δ Max | |---|---|---|---|---|---|---|---| | **Idle** | CPU Tctl | 49.6°C | 49.8°C | 54.7°C | 56.1°C | +5.1 | +6.3 | | | GPU Temp | 54.7°C | 57.0°C | 31.9°C | 33.0°C | **-22.8** | **-24.0** | | | NVMe0 | 44.9°C | 44.9°C | 38.0°C | 38.9°C | -6.9 | -6.0 | | **CPU Stress** | CPU Tctl | 72.2°C | 73.0°C | 73.9°C | 76.4°C | +1.7 | +3.4 | | | GPU Temp | 48.9°C | 55.0°C | 36.7°C | 39.0°C | **-12.2** | **-16.0** | | **Inference** | CPU Tctl | 63.9°C | 72.6°C | 72.3°C | 75.1°C | +8.4 | +2.5 | | | GPU Temp | 64.4°C | 66.0°C | 48.7°C | 51.0°C | **-15.7** | **-15.0** | | **Combined** | CPU Tctl | 73.6°C | 77.8°C | 80.1°C | 83.6°C | +6.5 | +5.8 | | | GPU Temp | 63.0°C | 66.0°C | 47.7°C | 52.0°C | **-15.3** | **-14.0** | | **Storage** | NVMe0 | 68.3°C | 69.8°C | 59.7°C | 63.9°C | **-8.6** | -5.9 | Two stories: the **GPU got dramatically cooler**, the **CPU got moderately warmer**. Both within safe limits. NVMe improved across the board. ## GPU the Star of the Show The RTX 5090 never exceeded **52°C** in the after test. Under sustained inference — the workload this machine exists to run — the GPU dropped from 66°C peak to 51°C. A full-cover water block with KryoSheet graphite on the die will do that. The GPU fan speed column is the satisfying one: **0% across all six phases**. Not because the fans are off — because they don't exist anymore. The stock cooler was physically removed. Cooling is handled entirely by the water block and the loop's radiator fans. This is the single biggest contributor to the noise reduction. But the surprise was **power efficiency**. Under inference, the GPU draws 26W less (386W → 360W) while doing the same work. Under combined load, the drop is **70W** (380W → 310W). Lower temperatures mean the card isn't fighting thermal limits, so it boosts more cleanly at lower power. That's not just a thermal win — it's an efficiency win that reduces total heat into the loop. The system helps itself. ## CPU the Honest Tradeoff The CPU is warmer. That's expected and I want to be upfront about it. Before, the CPU had a dedicated 420mm AIO — 50% more radiator area all to itself, with zero thermal coupling to the GPU. Now it shares 480mm of total rad area with a GPU that dumps 360W into the loop during inference. The worst case — combined phase peak of **83.6°C** — still leaves **11.4°C of headroom** below the 9950X3D's 95°C Tctl throttle limit. No throttling occurred during any test phase. Under CPU-only stress, the delta is just +1.7°C average. The CPU block and loop handle CPU-only loads almost as well as the 360mm AIO did. It's the thermal coupling during mixed workloads that creates the gap. Does it matter for the actual workload? For AI inference, the **GPU is the bottleneck** — not the CPU. The CPU's job is tokenization and prompt processing, which is lightweight. Running 6-8°C warmer doesn't affect inference throughput at all. ## Everything Else **NVMe temps improved 7-10°C** across the board. The Asus Mini-ITX board's M.2 heatsink is effective, and the open-frame case has decent airflow around the drives. The Samsung 9100 PRO's controller hotspot hit 76.8°C peak under storage stress — down from 79.8°C, and within Samsung's 83.8°C threshold. The motherboard swap brought sensor changes worth noting: the Asus board exposes **DDR5 SPD Hub temps** via the `spd5118` driver (idle: 35.8°C) — the ASRock didn't. The ASRock's Realtek NIC had a hwmon temp sensor; the Intel I226-V doesn't expose one. No functional loss — NIC temps were never actionable. **Thermal capacity math vs. reality:** The pre-migration estimate of 570-775W peak combined heat was conservative. The actual combined load landed at **~510W** (200W CPU + 310W GPU avg) because inference doesn't push the GPU to its 575W TDP, and the cooler GPU draws less power for the same work. The inline T-sensor confirmed coolant equilibrium in the range the fan curve was designed for. The system found its own balance. ## Gotchas 1. **Secure Boot MOK enrollment after motherboard swap.** The NVIDIA driver is a DKMS kernel module (`nvidia-dkms-590-open`). After moving to the new motherboard, `nvidia-smi` failed with `Key was rejected by service` — the new board's Secure Boot database didn't have the Machine Owner Key. Fix: `sudo mokutil --import /var/lib/shim-signed/mok/MOK.der`, reboot, and **catch the blue MOK enrollment screen** before the OS boots. I missed it the first time and had to repeat the whole cycle. If you've never seen it before, you'll blow right past it. 2. **Bottom fan orientation matters in an open frame.** The bottom radiator fans were initially configured exhausting downward. Corrected to exhaust upward — pull config through the rad — to create coherent bottom-to-top airflow through the open frame. Pull vs push performance delta is ~5%, but the airflow direction delta is significant. 3. **The test harness doesn't monitor the T-sensor.** The inline coolant temp sensor feeds the motherboard's T_Sensor header for fan control, but it isn't exposed as a standard hwmon device that the bash script's auto-discovery picks up. I have the fan curve working correctly, but the thermal test CSV doesn't include coolant temperature as a logged column. Future improvement. ## What I'd Change - **Thicker radiators or push-pull.** The 30mm slim rads in push-only are the minimum viable configuration. 45mm rads in push-pull would give substantially more thermal headroom and lower coolant equilibrium. The CPU would directly benefit. - **Dedicated CPU loop.** With unlimited budget, dual loops would eliminate the thermal coupling entirely. The CPU would get its own 240mm rad and perform close to the old 360mm AIO. But the shared loop works — it's just not *optimal* for the CPU. - **Log coolant temp in the test harness.** The T-sensor drives the fan curve perfectly, but I want it in the CSV for correlation analysis. That means either wiring up the motherboard's sensor reading via `lm-sensors` config or adding a USB temperature probe the script can poll directly. --- The system went from a full tower I couldn't sit next to during inference to an SFF build that's effectively silent. The GPU runs 14-24°C cooler. The CPU runs warmer because it shares the loop, but nowhere near throttling. Power efficiency improved because the GPU doesn't fight thermal limits. The NVMe drives got cooler too, somehow. 2,726 sensor readings don't lie. AI-NT-No-Problem earned its name. *Was the CPU tradeoff worth the silence? For an inference-bound workload, I'd make the same call every time.* ## By the Numbers - **2,726** sensor readings per test run (1-second intervals, 50-minute test) - **873** lines of bash in the vibe-coded thermal test harness - **7** bugs found and fixed during script development - **6** test phases: idle, CPU, inference, gaming, combined, storage - **23°C** GPU temperature drop at idle (55°C → 32°C) - **14°C** GPU temperature drop under combined load (66°C → 52°C) - **70W** GPU power reduction under combined load - **0%** GPU fan speed across all phases (because the fans don't exist) - **11.4°C** headroom to CPU throttle limit at worst case - **2×240mm** slim radiators handling a ~510W combined thermal load - **1** Secure Boot MOK enrollment screen missed on first attempt - **1** very quiet homelab === ## Friday Fixes: The Fix That Wasn't - URL: https://vibescoder.dev/posts/friday-fixes-the-fix-that-wasnt - Date: 2026-06-26 - Tags: #meta #building-in-public #agents #debugging - Reading time: 9 min read Three bugs that looked fixed from the wrong vantage point. An unquoted YAML date that crashed the public homepage one month after we wrote a blog post about the same bug. A model string that worked until its deprecation date passed. A security commit that stacked three invisible failures on top of each other. And a lesson about what accumulates when you build fast with agents. --- Three bugs this month. All three looked fixed before they broke. The date was quoted in 51 out of 52 posts. The model was pinned to a specific version. The upload feature had been working in production for weeks. Each one passed the obvious checks and failed somewhere else. That's the theme for this Friday Fixes: **the fix that wasn't.** Not bugs that went unnoticed, but bugs where a defense existed and the failure found its way around it. ## 1 the Unquoted Date Part Two If this one sounds familiar, it should. I wrote [an entire Friday Fixes post](/posts/friday-fixes-the-unquoted-date-that-broke-drafts) about this exact bug class five weeks ago. An unquoted YAML date. `gray-matter` parsing it as a `Date` object instead of a string. A crash downstream. Last time it took down `/admin/drafts`. The fix hardened `formatDate()` to coerce `Date` objects before calling `.includes()`. I verified it. I shipped it. I wrote 2,000 words about it. I moved on. This time it took down the homepage. **The symptom:** `vibescoder.dev` loaded for a split second, then flashed to Chrome's "This page couldn't load" screen. Every browser, every profile, every device. The site was completely dead to visitors. **The twist:** `curl` returned HTTP 200 with ~900KB of fully rendered HTML. The server was fine. The crash was happening during React hydration in the browser, invisible to any server-side test. **The cause:** A new post had `date: 2026-06-19` in its frontmatter. No quotes. `gray-matter` parsed it as a `Date` object. In `posts.ts`, the code does `const meta = data as PostMeta` and then spreads `...meta` into the return value. The `as PostMeta` cast told TypeScript the date was a `string`. At runtime, it was a `Date`. That `Date` object flowed through the server component, through the RSC serialization boundary, and into `PostListWithFilters`, a `"use client"` component. React couldn't hydrate it. No `global-error.tsx` existed to catch the crash. Dead page. **Why the May fix didn't prevent this:** Because the May fix was in the wrong layer. It hardened `formatDate()`, the function that happened to crash that time. It never hardened `posts.ts`, the layer where the `Date` object enters the system. The `Date` object simply found a different path out. **The false start:** The first fix attempt added `meta.date instanceof Date` to coerce the value. TypeScript rejected it: ``` Type error: The left-hand side of an 'instanceof' expression must be of type 'any', an object type or a type parameter. ``` The same `as PostMeta` cast that hid the runtime bug also blocked the fix. TypeScript believed `meta.date` was a `string`, so it wouldn't let me check if it was a `Date`. The fix was to check `data.date` (the raw gray-matter output, typed as `any`) instead of `meta.date` (typed as `string`): ```ts function normalizeDate(raw: unknown): string { if (raw instanceof Date) return raw.toISOString().split("T")[0]; return String(raw); } ``` Applied in all four functions that return post data. Also added a `global-error.tsx` so future hydration crashes show a reload button instead of a dead page. **What it cost:** ~25 minutes of downtime on the public site. Three commits across two repos, including the TypeScript false start. One embarrassing realization that I'd written a blog post about the bug and it happened again anyway. ## 2 the Model That Quietly Expired The blog has a voice dictation flow: record a transcript, click "Generate Post," get a draft. On June 18, clicking Generate returned a red "Generation failed" banner. No useful error detail. **The cause:** The generation pipeline called the Anthropic API with `model: "claude-sonnet-4-20250514"`. That model hit end-of-life on June 15. The API started rejecting requests three days before anyone noticed. The clue was in the SDK itself: ```typescript // @anthropic-ai/sdk DEPRECATED_MODELS 'claude-sonnet-4-20250514': 'June 15th, 2026', ``` **The fix:** One line. ```diff -model: "claude-sonnet-4-20250514", +model: "claude-sonnet-4-6", ``` Merged as PR #17. Generation worked immediately after Vercel deployed. **Why it took three days:** Two compounding failures: First, there's no deprecation warning from the Anthropic API. The model works on June 14. It doesn't work on June 15. No sunset header, no grace period, no degraded response with a warning. Just errors. Second, the catch block swallowed the error. The route handler logged `console.error("Generation error:", error)` to the server, but returned `{ error: "Blog generation failed" }` to the frontend. The actual Anthropic error message, which almost certainly said something about the model being retired, was buried in Vercel's server-side logs. The user-facing error was a generic string that could mean anything. A comment like `// EOL: June 15, 2026` next to the model string would have made this a 30-second fix. Surfacing the API error to the frontend would have made it self-diagnosing. Neither existed. ## 3 Seven Commits for Three Lines The Vacation Hub, a trip planning side project, has a photo gallery. Upload photos from your phone, they land in Vercel Blob Storage. It worked perfectly on the original deployment. After a security hardening commit that added CSP headers, photo uploads broke. Click upload, progress bar hits ~20%, hang forever. The agent spent seven commits fixing this. The actual fix was three lines. **What went wrong:** The security commit added a Content-Security-Policy header with `connect-src 'self' https://*.public.blob.vercel-storage.com`. The Vercel Blob SDK's client-side `upload()` makes a PUT to `https://vercel.com/api/blob`. That domain wasn't in `connect-src`. The browser silently blocked the request. But here's why seven commits: there were **three independent bugs** stacked on top of each other, and fixing any one of them didn't resolve the issue. 1. **CSP `connect-src` missing `https://vercel.com`** caused the hang. The browser blocked the PUT, no error surfaced, the upload promise never resolved. 2. **Empty `onUploadCompleted` callback** contributed to the hang. The SDK registered a webhook URL that Vercel would POST to after upload. The empty handler existed, so the SDK set it up, but the callback could silently fail. 3. **No `multipart: true`** on the upload calls. Vercel Blob's single PUT has a 4.5MB limit. Modern phone photos regularly exceed that. Without multipart chunking, large files returned 413. But you'd never see the 413 if the request never got past CSP. Each bug masked the next. Fix the CSP and uploads still hang (callback). Remove the callback and large photos 413 (no multipart). The agent tried each fix in isolation, concluded each one was wrong, and at one point rewrote the entire upload flow to server-side FormData, which introduced its own size limit problems. The breakthrough came when I asked a simple question: *"The original deployment worked. What changed?"* A targeted `git show` on the security commit would have found the CSP addition in minutes. Instead, the agent read the current code looking for problems rather than diffing backward from the last known working state. **The actual fix:** ```diff -connect-src 'self' https://*.public.blob.vercel-storage.com +connect-src 'self' https://*.public.blob.vercel-storage.com https://vercel.com ``` Plus `multipart: true` on both `upload()` calls and removing the empty callbacks. Three lines across two files. ## What Connects Them All three bugs involve a defense that felt complete but wasn't. The date coercion in `formatDate()` protected the function that crashed in May. It didn't protect the serialization boundary that crashed in June. The model was pinned, but nobody tracked when the pin expired. The security headers were added, but the SDK's upload domain wasn't in the allowlist, and the error was swallowed so thoroughly that seven commits went by before the agent found all three stacked failures. Each fix addressed the symptom it could see. None of them addressed the layer where the problem actually lived. The date needed to be coerced at the parsing boundary, not at the formatting boundary. The model needed a deprecation calendar, not just a version string. The security commit needed a full audit of outbound domains, not just the ones the developer remembered. This is a pattern I keep seeing when building with agents. You're working across dozens of sessions. The agent that added the CSP header wasn't the agent that debugged the upload failure. The agent that hardened `formatDate` wasn't the agent that needed to harden `posts.ts`. Each session is competent in isolation. The gaps live in the seams between sessions, where one agent's fix becomes another agent's assumption. **The shared context between those sessions is you.** The human collaborator is the one who remembers that this date bug happened before, that CSP headers can block SDK calls, that model strings have expiration dates. Agents don't carry that across sessions unless you build it into their context explicitly, with skills, with rules files, with the kind of institutional memory that a solo developer usually keeps in their head. That means bugs accumulate. Not dramatically, not in ways that show up in code review, but in the quiet gaps between what one session assumed and what the next session inherited. An unquoted date here. A hardcoded model string there. A CSP header that covers the domains you thought about but not the one the SDK uses internally. Each one is fine until it isn't. The honest response to this is not to stop using agents. It's to be vigilant. Scan for bugs and vulnerabilities constantly. Accept that some will surface in production despite your best efforts. Build error boundaries. Surface errors instead of swallowing them. Add the `global-error.tsx` before you need it. For a personal blog like this one, the risk is worth the reward. Agents push to production, release velocity stays high, and when something breaks, the blast radius is my own site. I can tolerate 25 minutes of homepage downtime in exchange for shipping a post every other day with a full admin toolchain that an agent built. That calculus changes the moment customers or revenue depend on what you're building. If this were a SaaS product, the unquoted date crash would have been an incident, not a blog post. The three-day model outage would have meant three days of broken functionality for paying users. The seven-commit upload thrash would have been a sprint-derailing debugging session with stakeholders asking for a postmortem. The velocity is real. The bugs are real too. Know which game you're playing. ## By the Numbers - **1** unquoted date in **52** posts took down the public homepage - **1** deprecated model string broke generation for **3 days** - **3** stacked bugs hid behind **1** security commit - **7** commits to find a **3-line** fix - **3** fodder files consumed across **3** weeks of bugs - **~25 min** homepage downtime (date crash) - **3** repos touched across all three fixes - **1** blog post about this exact bug class that didn't prevent the recurrence - **1** `global-error.tsx` added after the fact - **0** customers affected, because it's a personal site === ## Thursday Thoughts: Every Intern Is a Builder Now - URL: https://vibescoder.dev/posts/thursday-thoughts-every-intern-is-a-builder-now - Date: 2026-06-25 - Tags: #meta #building-in-public #agents #ai #vibe-coding #future-of-coding - Reading time: 5 min read A finance intern is spending her summer observing business processes and vibe coding automation tools. Not a CS major. Not shadowing someone. Building something real. It is a small example that says something big about how AI is reshaping internships, careers, and what the word "developer" actually means. --- Something happened in our finance department recently that I haven't been able to stop thinking about. We're bringing on a summer intern, a college student, not a CS major, and her project isn't to shadow someone or build a deck or update a spreadsheet. Her project is to observe our actual business processes: MBO gathering, payout cycles, closing the books. Then she's going to **vibe code an application or agentic workflow that automates parts of what she observed**, and present the whole thing back to her class as her intern project. I know that might sound like a small thing. It isn't. ## Developer Doesn't Mean What It Used to Mean I've been saying for a while now that **developer no longer equals software engineer**. That equation made sense for a long time. If you wanted to build something real, you needed to write real code, which meant you needed years of training. But that's not the world we're in anymore. Today, a developer is anyone with an idea who wants to build something. It might be lightweight. It might be scrappy. It might be an internal tool that automates a process nobody bothered to automate because it wasn't worth a full engineering sprint. That era, **the era of vibe coding**, is here, and it's moving faster than most companies realize. I've felt this personally through the journey of writing this blog. I've gone from someone with a passing curiosity about software to someone who can build fully production apps, spin up home labs, and experiment with frontier AI projects without flinching. Not because I suddenly became a software engineer, but because **AI gave me a co-pilot capable enough to close the gap between idea and execution**. That experience made me a believer. What I saw with this intern made me something more than that. ## What an Internship Used to Look Like There's always been a rough hierarchy of internship value. Engineering internships were prized because you could actually *ship* something. You could point to a PR, a feature, a deployed tool. That's a concrete artifact. Everything else, finance, ops, marketing, HR, you were lucky to get a bullet point on your resume that wasn't embarrassingly vague. That asymmetry always bothered me, but it felt structural. If you can't write code, you can't build things, and if you can't build things, there's a ceiling on what you can show for your time. Vibe coding breaks that ceiling. **The ability to build real, deployed tools is no longer gated by a CS degree.** What it requires now is the ability to observe a process, understand what problem needs solving, and work iteratively with AI to construct something that addresses it. Those are skills a sharp finance intern absolutely has. And now they can prove it. ## The Resume Has Changed Forever Here's what strikes me about this situation: a summer intern is going to walk out with a **concrete, deployed project in her portfolio**. Not a simulation. Not a case study. An actual automation she built to solve an actual problem at an actual company. That used to be the exclusive territory of software engineering internships. Now it's available to anyone whose employer gives them the tools and the latitude to build. Think about what that does over time: - Finance students who can automate their own workflows become dramatically more valuable than those who can't - Operations interns who ship internal tools leave with proof of judgment, not just exposure - Any non-technical role that involves repetitive process work becomes a candidate for this kind of intern project - **The line between "business intern" and "builder" starts to blur in exactly the right way** I worry about the current graduating classes. The labor market is disorienting right now, and a lot of students are trying to figure out how to differentiate themselves in a world where AI is compressing certain kinds of entry-level work. But I also think there's a real opening here for the ones who figure out vibe coding early. The question isn't whether you have a CS degree. **The question is whether you can ship something.** ## What Companies Need to Do What I'd love to see, and what I think the smarter companies will figure out soon, is making vibe coding a **golden path for all employees**, not just engineers. That means giving people access to the right tools in a governed, enterprise-appropriate way. It means designing internship programs around it. It means treating "I built this" as a meaningful credential regardless of job function. The companies that lean into this are going to get a productivity multiplier that's hard to explain until you've seen it. The ones that don't are going to watch their more forward-thinking competitors pull ahead in ways that look mysterious from the outside but are actually pretty simple: they let their people build things. --- We're in a genuine golden era of innovation right now, and I don't say that lightly. I've seen enough hype cycles to be skeptical of my own enthusiasm. But watching an intern prepare to spend a summer observing business processes and then automating them, and knowing she'll leave with a real project to show for it, that feels like something different. That feels like a shift. *If your company brought vibe coding to every internship program starting tomorrow, what would get built?* ## By the Numbers - **1 finance intern** — not a CS major — building the entire premise of this post - **3 business processes** she's set to observe and then automate: MBO gathering, payout cycles, and closing the books - **4 second-order effects** predicted once vibe coding spreads beyond engineering interns, laid out bullet by bullet in the post - **0** engineering background required for what used to be exclusively engineering-intern territory: a shipped, deployed project === ## Vibe Coding Has Entered the Enterprise, and Governance Is Next - URL: https://vibescoder.dev/posts/vibe-coding-has-entered-the-enterprise-and-governance-is-next - Date: 2026-06-24 - Tags: #meta #building-in-public #agents #ai #vibe-coding - Reading time: 5 min read Vibe coding has moved from hobbyist curiosity to enterprise rollout across knowledge workers, and the next wave of AI adoption will be defined by governance and token economics. --- Fair warning: the blog has been quieter than I'd like lately, and the reason is entirely self-inflicted. I decided to lean hard into the enthusiast side of [this whole experiment](/posts/day-one-building-vibescoder-dev). The home lab started life as a gaming PC, and I'm in the middle of converting it to a fully custom water-cooled loop, dedicated blocks for both the CPU and GPU. New case, new everything. I was a little overambitious about the timeline, so now I'm waiting on parts and the lab is effectively offline. There will be before-and-after thermal comparisons once it's all buttoned up, for those of you who are here for the hardware geekery and not just the AI discourse. In the meantime, being lab-less has forced me to slow down and actually think. So you're going to see more posts like this one, call it the view from the cheap seats. What am I actually observing in the market right now? Here's what's on my mind. ## Vibe Coding Just Crossed the Enterprise Threshold Not long ago, vibe coding carried a slightly dismissive connotation, a toy, a hobby, something the citizen developer crowd played with on weekends. That framing is dissolving fast. What I'm seeing now, across large enterprises, is that tools like Claude Code, Codex, and Cursor are being rolled out not just to engineering teams but to all knowledge workers. That's a fundamentally different moment. This is no longer about developers getting a faster autocomplete. This is companies making a deliberate bet that anyone who works with information can use AI to build something, a script, a report, an internal tool, without going through a formal dev cycle. That's a genuinely new thing. And here's why I find it interesting beyond the obvious: the conversation has quietly shifted away from job displacement and toward value creation. The early AI discourse was dominated by anxiety, who's going to lose their job? What I'm seeing now is companies asking a different question: where are we leaving value on the table that AI could unlock? Vibe coding, of all things, might be the first real answer to that question at scale. If you're working somewhere that's already building out a structured path for enterprise vibe coding, I'd genuinely love to hear how it's going. That's the whole spirit of this blog. This phenomenon is here to stay, and I want to understand it better. ## Governance Is the Immediate Priority Follow the logic chain and the next question becomes obvious: how do you make this safe? When you extend AI-assisted development to thousands of non-engineers, the governance surface area explodes. Who's reviewing what gets built? Where does the output go? What data is being fed into which models? I think we're squarely in a governance mindset era right now. The enterprises that are moving fastest on vibe coding rollouts are simultaneously the ones most anxious about guardrails: acceptable use policies, model access controls, audit trails. That tension is real and it's not going away. I've written before about [why regular audits matter](/posts/thursday-thoughts-audit-your-vibe-code-often) and what happens when you actually [run one on a live app](/posts/spring-cleaning-your-vibe-coded-apps). The same discipline applies at enterprise scale, just with a lot more stakeholders. ## Token Economics Is the Wave Right Behind It Here's where I think things get interesting from a technical and infrastructure standpoint: cost is about to become the dominant conversation. Right now, most organizations haven't fully felt the bill because adoption is still early and contained. Once vibe coding is genuinely enterprise-wide, the token economics become impossible to ignore. This is a big part of why I've been so focused on [local and hosted models](/posts/putting-the-gpu-to-work-running-local-llms) in my own experiments. My working theory is that we end up with a tiered model architecture that looks something like this: - A self-hosted or private cloud model handling the bulk of agentic, repetitive, and high-volume tasks, the stuff where cost per token really adds up - A lower-tier frontier model doing most of the content generation and summarization work - A high-end frontier model reserved for genuine value-creation tasks: strategic planning, complex reasoning, and probably the higher-stakes coding work That last tier is where you're willing to pay frontier prices because the output actually justifies it. The middle tier is where you're optimizing. The self-hosted layer is where you're driving cost to nearly zero for the high-frequency, lower-complexity workload. It's not a novel idea in the abstract, but I don't think most enterprises have operationalized it yet. They're still treating model selection as a one-time IT decision rather than a dynamic cost-optimization problem. That'll change. --- The lab will be back online soon, and when it is we'll get back to the hands-on experiments. But I wanted to get these observations down while they're fresh: vibe coding going enterprise-wide, governance as the immediate challenge, and token economics as the tidal wave right behind it. That's the arc I'm watching play out in real time. *Are you seeing the same governance-first pattern in your organization, or has cost already jumped to the top of the priority list?* ## By the Numbers - **3 tools** named as the enterprise vibe-coding wave: Claude Code, Codex, and Cursor, now rolled out to all knowledge workers, not just engineers - **3-tier model architecture** proposed for token economics: self-hosted/private cloud, lower-tier frontier, and high-end frontier - **3 companion posts** linked as the governance discipline already in practice: two audit posts and one on local/hosted models - **1 home lab** offline mid-migration — the actual reason this post exists at all === ## Thursday Thoughts: The AI Gut Check for Startups - URL: https://vibescoder.dev/posts/thursday-thoughts-the-ai-gut-check-for-startups - Date: 2026-06-19 - Tags: #meta #building-in-public #ai #business-strategy #llm - Reading time: 3 min read A CEO panel at an AI event sparked a simple but powerful question every startup founder should ask themselves: does your business get better as AI models improve, or does it get worse? --- At a data and AI event earlier this week, I heard one of those off-the-cuff observations that just sticks with you. A CEO was sitting on an industry panel, and someone asked him: *how do you survive and thrive as a startup in the AI world?* His answer was disarmingly simple: > "If you wake up every morning hoping the AI models got smarter, you have a good business model. If you wake up every day hoping they got dumber — or stayed the same — you have a bad business model." I haven't been able to stop thinking about it since. ## The Wrapper Problem There's been a lot of debate lately about whether being a "wrapper" is a viable business. You know the archetype — a product that adds value primarily by packaging basic LLM capabilities into a nicer interface or workflow. My honest take? The models will eat that business. Everything that lives at the surface level of what a frontier model can do will eventually *become* the frontier model. It gets commoditized, absorbed, and shipped as a default feature. If your moat is just "we made GPT easier to use," that moat gets filled in pretty quickly. ## Surfing the Frontier Vs. Racing It What you actually want is a business that *surfs* the frontier — one where as the underlying model gets smarter, your product gets smarter too. Your value compounds with every new model release instead of eroding. That's a fundamentally different design philosophy. It means your product isn't competing with the model. It's extending it. It's taking that raw capability and doing something with it that the model alone can't do — whether that's deep integrations, proprietary context, specialized workflows, or something else entirely. I'll be honest — this was a useful gut check for me personally. I thought about what we've built at Coder and whether it passes this test. I won't turn this into a product pitch, but I do think we've structured things so that better models make our product better, not redundant. That's the north star. ## The Wrong Question About AI and SaaS There's a lot of hand-wringing right now about whether AI is "eating SaaS." I think that framing is a distraction. It sends founders down a defensive path — trying to figure out how to survive the onslaught rather than how to ride it. The more productive question is: **Does your business extend the value of frontier models, or does it try to stay ahead of them?** If it's the latter, the clock is ticking. The frontier moves fast, and staying ahead of it as a startup is an exhausting, expensive, losing game. If it's the former — if you're genuinely amplifying what these models can do in a way that grows with them — that's where durability lives. --- Simple litmus test, but I think it cuts right to the heart of what makes an AI-era business worth building. Worth asking yourself honestly: which camp are you in? ## By the Numbers - **1 panel question** — the off-the-cuff CEO answer at a data/AI event this whole post is built around - **2 business models** the CEO's litmus test sorts every startup into: hoping AI gets smarter, or hoping it doesn't - **2 camps** for AI-era companies: those that surf the frontier and compound value, versus those that race it and lose - **1 gut check** the author ran against his own company using the same test === ## Model Showdown Round 7: Five Local Models vs. One Cloud Model on a Real Coding Task - URL: https://vibescoder.dev/posts/model-showdown-round-7-local-models-vs-the-tag-manager - Date: 2026-06-17 - Tags: #model-showdown #benchmark #ai #llm #homelab #building-in-public #coder - Reading time: 13 min read I gave five local LLMs and one frontier cloud model the same coding task on my homelab: build a tag manager for the blog's admin panel. Only two shipped anything. Here's what happened. --- Five local models. One frontier cloud model. The same coding task. Zero hand-holding. Only two shipped code. One of them was the cloud model. Part of my goal with this series is to continuously test the viability and maturity of local models. I've done it for [basic agentic tasks](/posts/homelab-bakeoff-openclaw-outperforms-hermes-with-hermes-models). Today we're revisiting coding tasks. What did we learn? **Local models are not ready — yet.** At least not for homelabs like mine. Perhaps if you have hundreds of gigabytes of unified memory (I'm looking at you, older Mac Studios) you can run fully unquantized models. But with even the beefiest of discrete consumer GPUs, local models can't code. Let's dig in. ## The Setup This is Round 7 of the Model Showdown series. Previous rounds tested cloud models against each other — Opus, Sonnet, GPT-5.5, Qwen cloud. This time I wanted to answer a different question: **can local models running on consumer hardware actually complete a real agentic coding task?** The homelab: - **CPU**: AMD Ryzen 9 9950X3D, 64GB RAM - **GPU**: NVIDIA RTX 5090, 32GB VRAM - **Inference**: llama.cpp b9660, single-model serving on port 8080 - **Agent platform**: Coder Agents v2.34.0 - **OS**: Ubuntu 24.04, NVIDIA Driver 590.48.01, CUDA 13.1 Every local model was configured as aggressively as the hardware allows — flash attention, quantized KV cache (`q8_0`), and context windows maxed to what VRAM permits. ### The Contestants | Model | Type | Quant | VRAM | Context | Max Output | |---|---|---|---|---|---| | **Qwen 3.6 35B-A3B** | Local MoE | UD-Q4_K_XL (21GB) | ~21GB | 131,072 | 81,920 | | **Gemma 4 12B** | Local Dense | UD-Q4_K_XL (6.9GB) | ~8GB | 65,536 | 32,768 | | **Hermes 4 14B** | Local Dense | Q8_0 (15GB) | ~15GB | 65,536 | 32,768 | | **Qwen3-Coder 30B-A3B** | Local MoE | UD-Q4_K_XL (17GB) | ~17GB | 65,536 | 32,768 | | **Devstral 24B** | Local Dense | Q5_K_M (17GB) | ~17GB | 65,536 | 32,768 | | **Claude Sonnet 4** | Cloud (control) | Native | N/A | 200,000 | — | Sonnet 4 is the control variable. I already know what it can do. The question is how close the local models get. ## The Task Admin Tag Manager Previous rounds used an "image management" feature, but that collided with existing code in the repo. For Round 7, I designed a clean-room task: **build a tag manager for the blog's admin panel**. The blog already has tags — posts use a `tags[]` array in MDX frontmatter, there's a public `/tags` page, and `src/lib/posts.ts` has a `getAllTags()` function. But there's no admin UI to manage them. Each model got the identical prompt: > **Goal**: Add a Tag Manager to the `/admin` section. > > **Requirements**: > 1. Create `src/lib/tags.ts` — list tags with post counts, detect orphans, support rename and merge > 2. Create `src/app/api/admin/tags/route.ts` — GET, PATCH, DELETE endpoints > 3. Create `src/app/admin/tags/page.tsx` — table with inline rename, delete, sort > 4. Add "Tags" to AdminNav > 5. Client-side mutations with refresh (no full page reload) > 6. `npm run build` must pass with zero errors > 7. Take a screenshot via Playwright MCP > 8. Commit in logical chunks, push to branch > 9. Do NOT open a PR Ten requirements. Real codebase. Real build system. Real git workflow. ## The Methodology Each model got its own clean branch (`run-10` through `run-15`) forked from the same `main` commit. Local models were loaded one at a time via `llm-switch.sh` and served through llama-server on `localhost:8080`. Sonnet 4 ran through Coder's built-in Anthropic provider. Model-to-run assignment was randomized and sealed before execution. I didn't know which model was which run until after all six completed (or failed). **A note on human intervention**: I monitored each session live and occasionally nudged stalled models ("keep going", "can you finish?") or stopped them when they entered obvious doom loops ("stop"). There was no standardized intervention protocol — I used my judgment as a developer watching an AI assistant, which is how these tools actually get used in practice. Some models got more nudges than others because they stalled more. The two models that shipped code needed zero intervention. ## The Results | Model | Tool Calls | Total Tokens | Commits | Build Pass | Screenshot | Outcome | |---|---|---|---|---|---|---| | **Sonnet 4** ☁️ | 88 | 19K | 4 | ✅ (1st try) | ✅ | **Complete** | | **Qwen3-Coder 30B-A3B** | 60 | 2.06M | 1 | ✅ (3rd try) | ❌ | **Partial** | | **Qwen 3.6 35B-A3B** | 76 | 3.89M | 0 | ✅ (2nd try) | ❌ | **Failed** (never committed) | | **Gemma 4 12B** | 34 | 1.17M | 0 | ❌ (0/7) | ❌ | **Failed** | | **Hermes 4 14B** | 40 | 1.14M | 0 | ❌ (0/13) | ❌ | **Failed** | | **Devstral 24B** | 0 | 14K | 0 | ❌ | ❌ | **Total failure** | One cloud model. Five local models. **One complete success. One partial. Four failures.** ## What Each Model Actually Did ### Sonnet 4 the Control Run 14 Complete Success Sonnet did what you'd expect a frontier model to do. It cloned the repo, spent 25 tool calls reading existing code (auth patterns, API conventions, admin page structure, frontmatter format), then wrote all four files in a tight burst. Build passed on the first try. It hit a real environment issue — a stray `package.json` confused Turbopack's workspace detection — diagnosed the root cause, fixed it with a config change, took a Playwright screenshot, and pushed four clean conventional commits. Total time: ~10 minutes. Zero human intervention. ``` acb4ea1 fix: set turbopack.root to avoid workspace lockfile detection in dev 352a8ca feat: add Tags link to AdminNav 22899a0 feat: add /admin/tags page with inline rename, delete, and sort 19f44fa feat: add tags.ts lib with stats, rename, and remove helpers ``` The implementation followed existing project patterns because it read them first. That's the difference. ### Qwen3-Coder 30B-A3B Run 15 the One That Shipped The best-performing local model. It cloned the repo, explored the codebase, created all four required files (410 lines of code), fixed TypeScript errors across three build attempts, and pushed a working commit. But it wasn't clean. It burned ~8 tool calls just fighting the working directory problem (each `execute` call resets to `/home/coder`, so it kept forgetting to `cd` into the repo). After committing, it spent another 30 tool calls confused about whether its own API route file existed — trying to delete and recreate something that was already committed. No screenshot. No logical commit chunking (everything in one commit). But **it shipped working code**, which puts it in a category of one among the local models. ### Qwen 3.6 35B-A3B Run 13 the Tragic Hero This is the one that hurts. Qwen 3.6 actually *completed the implementation*. It explored the codebase thoroughly, wrote all four files, fixed a type error, and got `npm run build` to pass cleanly. Then it decided it needed a Playwright screenshot before committing. It spent the next **77 messages** — over 50% of its entire session — trying to install Playwright, fighting missing Chromium dependencies, debugging browser launch failures, rewriting a screenshot script four times, and wrestling with the auth middleware that blocked unauthenticated page loads. It never took the screenshot. It never committed. It never pushed. The code was right there. Build passing. Ready to go. But the model couldn't prioritize "commit what works" over "complete requirement #7 first." Three times I nudged it — "You there?", "Keep going", "can you finish?" — and each time it dove back into the Playwright rabbit hole. **3.89 million tokens burned. Zero commits pushed.** ### Gemma 4 12B Run 11 the API Misunderstanding Gemma cloned the repo, read the existing code, and wrote all three new files plus the nav update. Reasonable start. Then it ran `npm run build` and hit a type error with `gray-matter`'s `stringify()` function. The fix was simple: `matter.stringify(content, data)` — content string first, data object second. Gemma had the arguments reversed. It tried six variations of the call, rewrote `tags.ts` six times, ran seven builds — and never once tried the correct argument order. It never read the `gray-matter` type definitions. It never checked the docs. After the fifth failed build, it fell into a **degenerate text generation loop** — printing "I'll also make sure `src/lib/tags.ts` is correct" 26 consecutive times. I had to send "stop" to break the loop. ### Hermes 4 14B Run 12 the Import Path That Wouldn't Die Hermes jumped straight to writing code without exploring the project structure first. It created two files and ran `npm run build`. The error: ``` Module not found: Can't resolve '../../../lib/tags' ``` The route file at `src/app/api/admin/tags/route.ts` needs `../../../../lib/tags` (four levels up) or `@/lib/tags` (Next.js path alias). Hermes used three levels. Off by one. It never diagnosed this. Instead, it rewrote both files with the same wrong import and rebuilt. **Thirteen times.** The output from message 34 onward is nearly verbatim identical every iteration. Same code. Same error. Same "fix." When I sent "stop," it continued for five more tool calls before acknowledging the signal. ### Devstral 24B Run 10 the Non-Starter Devstral never executed a single tool call. It hallucinated an entire fake conversation about a Python project that doesn't exist, then emitted what looked like tool invocations — `execute`, `read_file`, `write_file` — but rendered them as **plain text** inside the assistant message. The platform couldn't parse them as structured tool calls, so nothing happened. This is a fundamental compatibility failure. The model couldn't interface with Coder's tool-calling protocol at all. Nine messages, 14K tokens, zero actions. ## The Token Efficiency Gap This is the number that stopped me: | Model | Total Tokens | Result | |---|---|---| | Sonnet 4 | **19,237** | Complete (4 commits, screenshot) | | Qwen3-Coder | **2,059,519** | Partial (1 commit, no screenshot) | | Qwen 3.6 | **3,890,791** | Failed (build passed, never committed) | | Gemma 4 12B | **1,170,967** | Failed (0/7 builds passed) | | Hermes 4 14B | **1,138,614** | Failed (0/13 builds passed) | | Devstral 24B | **14,447** | Failed (zero tool calls) | Sonnet used **19K tokens** to complete the task. The local models that actually tried burned **1–4 million tokens** and mostly failed. That's a 100-200x token efficiency gap for the same task. The local models aren't just slower. They're doing fundamentally more work per unit of progress — re-reading files they already read, rewriting code they just wrote, rebuilding with the same error, looping through the same reasoning. It's not a speed problem. It's a thinking problem. ## Common Failure Patterns Every local model that ran long enough exhibited the same pathologies: **1. Degenerate loops.** Gemma repeated the same text 26 times. Hermes rebuilt with the same wrong import 13 times. Qwen 3.6 rewrote its screenshot script 4 times with the same approach. Once a local model enters a loop, it can't break out without human intervention. **2. Working directory amnesia.** Coder's `execute` tool doesn't preserve `cd` across calls. Sonnet learned this instantly and prefixed every command. Multiple local models burned 5-10 tool calls per session rediscovering this. **3. Inability to prioritize.** Qwen 3.6 had a passing build and chose to yak-shave on Playwright instead of committing. No local model demonstrated the judgment to ship what works and iterate. **4. No self-diagnosis.** When a build fails, the fix requires reading the error, forming a hypothesis, and trying something *different*. Hermes and Gemma both tried the same fix repeatedly. Neither ever stepped back to read docs, check type definitions, or examine the project configuration. ## What I Actually Learned **Local models can write plausible code.** Four of five local models produced syntactically reasonable TypeScript. The code *looked* right. The architecture was sensible. It's the last mile — debugging, building, committing, shipping — where they fall apart. **The agentic gap is wider than the coding gap.** These models can generate code. What they can't do is *operate as agents* — managing state across tool calls, diagnosing errors, prioritizing tasks, knowing when to stop and ship. That's a different capability than code generation, and it's where local models are currently weakest. **Token efficiency is the real benchmark.** Raw parameter count and context window don't predict agentic success. Qwen 3.6 had the biggest context (131K) and burned the most tokens (3.89M) — and still didn't ship. Sonnet used 100x fewer tokens and completed everything. The bottleneck isn't context. It's reasoning quality per token. **Tool-calling compatibility isn't guaranteed.** Devstral is marketed as an agentic coding model, but it couldn't even interface with the tool-calling protocol. If you're evaluating local models for agent use, test tool calling first. **Qwen3-Coder is the local model to watch.** It's the only local model that actually shipped code in this test. Messy, single-commit, no screenshot — but working code pushed to a branch. For a 30B MoE model running on a single consumer GPU, that's notable. ## By the Numbers | Metric | Sonnet 4 | Qwen3-Coder | Qwen 3.6 | Gemma 4 12B | Hermes 4 14B | Devstral 24B | |---|---|---|---|---|---|---| | **Type** | Cloud | Local MoE | Local MoE | Local Dense | Local Dense | Local Dense | | **Parameters** | Unknown | 30B (3B active) | 35B (3B active) | 12B | 14B | 24B | | **Total tokens** | 19,237 | 2,059,519 | 3,890,791 | 1,170,967 | 1,138,614 | 14,447 | | **Tool calls** | 88 | 60 | 76 | 34 | 40 | 0 | | **Messages** | 183 | 127 | 162 | 81 | 88 | 9 | | **Commits pushed** | 4 | 1 | 0 | 0 | 0 | 0 | | **Build passed** | ✅ 1st try | ✅ 3rd try | ✅ 2nd try | ❌ 0/7 | ❌ 0/13 | ❌ | | **Screenshot** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | **Human nudges** | 0 | 0 | 3 | 2 + stop | stop | 1 | | **Outcome** | Complete | Partial | Failed | Failed | Failed | Failed | **Inference stack**: llama.cpp b9660, flash attention, q8_0 KV cache, Coder Agents v2.34.0 **Hardware**: RTX 5090 32GB, Ryzen 9 9950X3D, 64GB RAM, Ubuntu 24.04 Next up: Round 6 brings more frontier models to the same task. And I'll keep pushing the local models — better quants, newer releases, maybe a different agent framework. The gap is real, but the pace of improvement on the local side is fast. === ## Frontier Bakeoff: We Benchmarked Fable 5 Hours Before the Shutdown - URL: https://vibescoder.dev/posts/frontier-bakeoff-we-benchmarked-fable-5-hours-before-the-shutdown - Date: 2026-06-13 - Tags: #model-showdown #benchmark #ai #llm #building-in-public - Reading time: 8 min read Four frontier models, ten tasks, one government shutdown. We ran Claude Fable 5 through the homelab benchmark harness three hours before Anthropic pulled the plug — and it came in second. Here's the full bakeoff. --- Fable 5 didn't win. I need to say that up front because the timing of this post is going to make it sound like a very different story. Yes, we benchmarked Claude Fable 5 on our homelab harness. Yes, the US government suspended it about three hours later. But the actual result? Fable 5 scored 89.3. Opus 4.8 scored 91.9. The model everyone's eulogizing right now lost to a model you can still use today. That's the real story. The suspension is just what makes it weird. ## What We Tested This is Round 6 of our [homelab bakeoff series](/posts/homelab-bakeoff-openclaw-outperforms-hermes-with-hermes-models) — but with a twist. Rounds 1 through 5 tested quantized local models on an RTX 5090 via llama.cpp. This time we pointed the same task suite at four frontier cloud models: | Model | Provider | Key | |---|---|---| | Claude Opus 4.8 | Anthropic | `opus48` | | Claude Fable 5 | Anthropic | `fable5` | | Claude Sonnet 4.6 | Anthropic | `sonnet46` | | GPT-5.5 | OpenAI | `gpt55` | Same 10 quality tasks. Same 3 speed tasks. Same scoring rubrics, same fixture files, same composite formula. The only things that changed were the transport layer (Anthropic/OpenAI SDKs instead of llama.cpp HTTP) and two bug fixes that made scoring more accurate. I'll get into those. ## The Results | Rank | Model | Coding | Reasoning | Tool Use | Speed | Total | |---|---|---|---|---|---|---| | 1 | Opus 4.8 | 84.8 | 90.0 | 100.0 | 100.0 | **91.9** | | 2 | Fable 5 | 86.7 | 93.3 | 100.0 | 79.9 | **89.3** | | 3 | Sonnet 4.6 | 75.2 | 93.3 | 100.0 | 78.6 | **84.5** | | 4 | GPT-5.5 | 86.7 | 66.7 | 100.0 | 60.1 | **80.0** | A few things jump out. **Fable 5 was the best at the hard stuff.** It scored highest on coding (86.7, tied with GPT-5.5) and highest on reasoning (93.3, tied with Sonnet 4.6). Its architecture analysis for Task 3.2 — designing a collaborative editor with CRDTs at scale — was the cleanest answer in the field. It opened by decomposing the 100ms latency budget across the full request path before even discussing algorithms. That's the kind of structured thinking you want from a senior engineer, not a chatbot. **But speed killed it.** Opus 4.8 was meaningfully faster on every speed benchmark, and speed is 20% of the weighted total. Fable 5's TTFT hovered around 3.4–4.0 seconds per request — likely the cost of whatever reasoning depth Anthropic tuned into it. Opus came in consistently under that. When you weight for speed, Opus's 2.6-point lead on the final score comes almost entirely from the speed category. **Tool use was a wash.** Every model scored 5/5 on both tool-use tasks. At the frontier level, structured output and function calling are solved problems. This category no longer differentiates. ## GPT-5.5 the Token Limit Trap GPT-5.5 tied for the best coding score (86.7) and nailed Bayes' theorem, database debugging, and both tool-use tasks. But its reasoning score is 66.7 — way behind the pack — and the reason is a single task failure. On Task 3.2 (architecture analysis), GPT-5.5 hit the 4,096 completion token limit and returned a truncated response. `finish_reason: "length"`, empty captured content, 0/10 on all rubric items. It spent 85 seconds generating 4,096 tokens of thinking and never actually delivered an answer. The scoring harness captured nothing because there was nothing to capture. Was the task too hard? No — Fable 5 scored 10/10 on the same prompt in roughly the same token budget. GPT-5.5 just allocated its budget differently (or the API's default max_tokens was too low for its reasoning style). Either way, one truncated response cost it 10 points and dropped it from a competitive second place to a distant fourth. **The lesson:** benchmark harnesses that don't account for provider-specific token limits will produce misleading results. I could have set `max_tokens` higher, but the point of a bakeoff is equal conditions. Every model got the same parameters. ## The Sonnet Surprise Sonnet 4.6 deserves attention. It matched Fable 5 on reasoning (93.3), ran at roughly the same speed, and costs about a third as much. Its coding score (75.2) is the only weak spot — it missed some feature-detection checks on the Express bug-fix task that the others caught. For most production workloads, Sonnet 4.6 at 84.5 overall is probably the right choice. The 4.8-point gap to Fable 5 is almost entirely coding quality, and the price difference is substantial. ## What Changed from Round 5 I adapted the Round 5 homelab harness into a standalone cloud benchmark. For full transparency, there's a [CHANGES.md](https://github.com/carryologist/benchmarks/blob/main/runs/2026-06-12--coding--frontier-api--v1/CHANGES.md) documenting every delta, but here are the ones that affect scores: **Bayes fix (Task 3.3).** Round 5 expected 41.67% as the correct answer. It's actually 40.54%. The old harness had a rounding error in the denominator — `P(E) = 0.0185`, not `0.018`. Every Round 5 model got this "wrong" because the rubric was wrong. Fixed. All four frontier models computed 40.54% correctly. **TypeScript tests wired up (Task 1.3).** Round 5 couldn't run the TypeScript functional tests because `npx tsx` wasn't available on the homelab. Scores were capped at 60/100. This environment has tsx, so the full test suite runs. Both Fable 5 and GPT-5.5 passed all assertions. **Speed methodology.** Round 5 pulled `timings.predicted_per_second` from llama.cpp's response body. Cloud APIs don't expose that, so we measure wall-clock `output_tokens / elapsed_time` and streaming TTFT. The absolute numbers aren't comparable to Round 5, but relative rankings between the four cloud models are valid. **Everything else is identical.** Same prompts, same fixtures, same scoring weights (Coding 40%, Reasoning 20%, Tool Use 20%, Speed 20%), same composite formula. ## About That Shutdown On June 12, 2026, at approximately 5:21 PM Eastern, the US government issued an export control directive targeting Anthropic's most capable models. Anthropic disabled Fable 5 and Mythos 5 for all customers. No restoration timeline has been provided. Our benchmark run completed around 2:00 PM Eastern — roughly three hours before the shutdown. I didn't know it was coming. Nobody outside the government and Anthropic's leadership did. I'm not going to speculate about the policy. What I will say is that the benchmark data is real, the run completed cleanly, and the results are reproducible right up until the moment the model stopped existing. We have the full result JSONs, the harness code, and the fixture files. If Fable 5 comes back — or if it doesn't — this is what it could do. ## What I Actually Learned **The frontier is tighter than I expected.** 11.9 points separate first from last. In Round 5, the gap between the best and worst local model was over 40 points. At the frontier, everyone can code, everyone can reason, everyone can use tools. The differentiation is in speed, price, and edge-case reliability. **Speed is a legitimate quality axis.** I initially weighted speed at 20% because I thought it would be a tiebreaker. It ended up being the deciding factor. Opus 4.8 won this bakeoff on speed, not intelligence. Whether that's the "right" ranking depends on your use case, but for human-in-the-loop coding — where you're waiting on the model 50 times per session — I think speed matters more than most benchmarks acknowledge. **Benchmarks need bug fixes too.** The Bayes theorem error in Round 5 went unnoticed for five rounds because every local model got it wrong anyway. It took a frontier model computing the right answer to surface the bug in my own scoring rubric. That's humbling and also kind of the point of running these. **One truncated response can tank a ranking.** GPT-5.5 went from a plausible second place to fourth because of a single `finish_reason: "length"` on one task. Benchmark design that doesn't account for this is fragile. I'm noting it but not adjusting the score — equal conditions means equal conditions. ## By the Numbers | | Opus 4.8 | Fable 5 | Sonnet 4.6 | GPT-5.5 | |---|---|---|---|---| | Task 1.1 (Todo CLI) | 100.0 | 100.0 | 80.0 | 100.0 | | Task 1.2 (Pagination API) | 60.0 | 60.0 | 60.0 | 60.0 | | Task 1.3 (TS Config) | 100.0 | 100.0 | 80.0 | 100.0 | | Task 3.1 (DB Debug) | 10/10 | 8/10 | 10/10 | 10/10 | | Task 3.2 (Architecture) | 8/10 | 10/10 | 10/10 | 0/10 | | Task 3.3 (Bayes) | 5/5 | 5/5 | 5/5 | 5/5 | | Task 4.1 (Tool Use) | 5/5 | 5/5 | 5/5 | 5/5 | | Task 4.2 (Tool Use) | 5/5 | 5/5 | 5/5 | 5/5 | Raw speed (composite tok/s score): Opus 95.9, Fable 76.6, Sonnet 75.4, GPT-5.5 57.6. All result data, the benchmark harness, and fixture files are in the [benchmarks repo](https://github.com/carryologist/benchmarks/tree/main/runs/2026-06-12--coding--frontier-api--v1). --- *This is post 46 on Vibes Coder. The benchmark harness is open source. If Fable 5 comes back, I'll run it again.* === ## Homelab Bakeoff: OpenClaw Outperforms Hermes… With Hermes Models - URL: https://vibescoder.dev/posts/homelab-bakeoff-openclaw-outperforms-hermes-with-hermes-models - Date: 2026-06-11 - Tags: #agents #llm #homelab #building-in-public #openclaw - Reading time: 15 min read Two Discord bots, one 14B model, five fitness-tracker tasks. Both agents failed on the first try. Getting them working required debugging context overflow, silent tool parameter drops, and a chat template flag that changes everything. The results reveal as much about the state of local AI agents as they do about which framework won. --- I spent an evening trying to make two AI agent frameworks do something simple: call a fitness tracker API and tell me about my workouts. Both agents ran the same model — Hermes-4-14B Q8_0, a 14.6 billion parameter model fine-tuned for tool calling. Same hardware — an RTX 5090 with 32 GB of VRAM. Same llama.cpp inference server. Same five tasks. Same MCP server on the other end. Both failed on the first try. Both required multiple rounds of debugging before they could make a single tool call. The actual test — running five prompts and scoring the results — took about ten minutes. Getting there took the entire evening. I'm sure both frameworks would perform well with frontier cloud models — pipe in Claude or GPT-5 and the tool-calling pipeline is someone else's problem. But the whole point of the homelab is local inference. Local models. Local headaches. And right now, running AI agents against local open-source models means nothing works out of the box. The surprise wasn't that both agents struggled. It was which one won. OpenClaw — the generic, model-agnostic framework — outperformed Hermes Agent on Hermes's own model. The framework built by a different company, with no special knowledge of Hermes-4's architecture, beat the vertically integrated stack that trained the model and built the agent. That result needs explaining. ## The Setup Two Discord bots on my homelab server, each backed by a different agent framework: | | Hermesbot | Clawbot | |---|---|---| | **Framework** | Hermes Agent (Python) | OpenClaw (Node.js) | | **Model** | Hermes-4-14B Q8_0 | Hermes-4-14B Q8_0 | | **State** | SQLite | JSONL sessions | | **MCP Transport** | Direct HTTP | Gateway proxy | | **Discord Bot** | Hermesbot | Clawbot | Both connect to the same fitness-tracker MCP server — a Next.js app on Vercel that wraps my Peloton data, workout history, and annual goals in ten tools. `list_workouts`, `sync_peloton`, `list_goals`, `delete_workout`, and so on. The idea was clean: same model isolates the framework variable. Any performance difference is orchestration, not weights. The [experiment design](/posts/hermes-agent-first-contact) called for five tasks of escalating complexity: 1. **List my last 5 workouts** — basic single tool call 2. **Sync Peloton, count this week, check goal pace** — multi-step chain 3. **"How am I doing?"** — ambiguous intent, tool selection 4. **Delete a fake workout ID** — error handling 5. **Trend analysis for the past month** — complex reasoning over large data ## Round 1 Both Agents Failed Neither agent could complete a single task on the first attempt. ### Hermesbot Death by System Prompt Hermes Agent ships with 90 built-in skills and 17 Discord toolsets — admin, moderation, voice, reactions, the works. All of them get injected into the system prompt on every API call. Combined with the MCP tool definitions, the system prompt ballooned to over 25,000 tokens. The model's actual context window? 40,960 tokens. Hermes-4-14B's training context is 40K, and llama.cpp clamps `--ctx-size 65536` down to that value silently. So on every request: 25K system prompt + conversation history + tool results = more than 40,960 tokens. llama-server returned HTTP 400. Hermes Agent's compression system kicked in, but it compresses *conversation messages* — it can't compress the system prompt. The system prompt was the problem, and the compression loop couldn't touch it. Death spiral. **The fix**: Trim the Discord toolsets from 17 down to 1. In `~/.hermes/config.yaml`, I replaced the default toolset list with just `memory`: ```yaml discord: toolsets: - memory ``` System prompt dropped from 25K+ tokens to something manageable. Two other config tweaks: set `context_length: 65536` to pass Hermes Agent's hard-coded 64K minimum check (the framework refuses to start if context is under 64,000 — even though the model's actual context is 40,960), and bump the compression threshold from 0.5 to 0.85 so it stops trying to compress every turn. ### Clawbot the Silent Flag OpenClaw's failure was subtler. The MCP server wasn't registered in the config at all — that was the first fix. But even after adding it, Clawbot would narrate what tools it would use without actually calling them. It fabricated workout data from 2024, complete with instructors and distances, none of it real. The root cause took multiple rounds to find. OpenClaw lists tool names in its system prompt text — "you have access to `fitness-tracker__list_workouts`" and so on — but sends `tools=0` in the actual API request. The model sees the tool names, understands it should use them, but has no structured schema to emit. So it does the next best thing: it makes up the answer. This turned out to be a chat template problem. llama-server was running with `--chat-template chatml`, which is a minimal template that processes messages but ignores the `tools` parameter entirely. When you send tools in the API request, chatml drops them silently. No error, no warning. The model never sees them. I verified this with a direct API test: ```bash # With --chat-template chatml: 14 prompt tokens. Tools invisible. curl /v1/chat/completions -d '{"tools":[...], "messages":[...]}' # Response: "I can't help with that" # With --jinja: 172 prompt tokens. Tools injected by the model's template. # Response: {"tool_calls": [{"function": {"name": "list_workouts"}}]} ``` The fix was a single flag: `--jinja` instead of `--chat-template chatml`. With `--jinja`, llama-server uses the Jinja template embedded in the Hermes-4 GGUF file. That template knows about tools. It injects tool definitions into the prompt, recognizes the model's `` XML output, and extracts it into structured `tool_calls` in the API response. The entire tool-calling pipeline went from broken to working by changing one server flag. ## The Exhaustion Loop I want to pause here and be honest about what this process felt like. Each failure mode required a different kind of debugging. The Hermesbot system prompt issue required reading framework source code to understand why compression wasn't helping. The OpenClaw tool injection issue required reading llama.cpp chat template documentation to understand that `chatml` ignores tools. The `--jinja` fix required understanding that Hermes-4's GGUF file embeds a Jinja template that handles tool-call formatting — something mentioned in no getting-started guide for either framework. The cycle was: try a config → restart the service → send a test message → read logs → form a hypothesis → try another config. For Hermesbot, I tried adjusting compression thresholds, changing context length settings, and modifying model parameters before discovering the toolset bloat. For Clawbot, I tried switching API modes (`openai-completions` vs `openai-responses`), adding compatibility flags (`supportsTools`, `supportsDeveloperRole`), and testing config keys that turned out not to exist (`toolCallStyle`, `nativeToolCalls`, `capabilities` — all rejected by the validator). None of this is documented in a "getting started with local models" guide because it doesn't fit in one. The failure modes are emergent — they come from the interaction between the agent framework, the inference server, the model's chat template, and the model's training format. Each layer has its own configuration surface and its own silent failure modes. **Agents are not ready to use local open-source models unless you're an extreme tinkerer.** Nothing works out of the box. The iterative loop of researching, testing configurations, tweaking parameters, and running experimental tasks is exhausting. ## Round 2 the Actual Test Once both agents were working, the test itself was anticlimactic. Five prompts, same order, one after another. ### Task 1 List My Last 5 Workouts Both agents called `list_workouts(limit=5)` correctly. Same tool, same parameter. **Hermesbot** got the data back — 2,935 characters of workout details — and said: *"Let me know if you'd like me to summarize these workouts for you!"* It fetched the data and didn't show it. The user asked to list workouts and the agent offered to summarize them later. That's a 14B model struggling with instruction following after processing a dense system prompt. **Clawbot** got 2,621 characters back and formatted them immediately: > 1. **Today, June 10, 2026** (1:33 PM PDT) — Peloton Cardio, 28 min > 2. **Yesterday, June 9, 2026** (4:36 AM PDT) — Cannondale Cycling, 15 min > 3. **Yesterday, June 9, 2026** (12:41 AM PDT) — Cannondale Cycling, 13 min > 4. **June 7, 2026** — Peloton Cycling, 45 min, 15.07 miles > 5. **June 8, 2026** — Peloton Cycling, 30 min, 10.36 miles Dates, sources, durations, notes, distances where available. The data the user asked for, presented the way a user would want it. ### Task 2 Sync My Peloton Workouts Then Tell Me How Many Workouts I've Done This Week and Whether I'm on Pace for My Annual Goal. Both agents chained three tool calls autonomously: sync → list workouts → list goals. No prompting needed. That's the part that worked. The difference was in the parameters. Hermesbot used `since=2026-06-10` — today only. It found 1 workout this week. Clawbot used `since=2026-06-03` — Monday. It found 11 workouts. Same model, same tool, different date parameter. The framework's system prompt influences how the model interprets "this week." Hermesbot then confused the annual minutes target (11,700 minutes) with a weight target, reporting "you're on pace for about 1.5% of your annual weight target (1/1000000)." The math didn't track. Clawbot built a table: | Metric | Goal | Current | Status | |---|---|---|---| | Weekly Sessions | 5 | 7 | 🟢 On Track | | Weekly Minutes | 225 min | 289 min | 🟢 On Track | | Annual Minutes | 11,700 min | 289 min | 🟢 On Track | Correct numbers, correct interpretation, structured output. ### Task 3 How Am I Doing Neither agent made new tool calls — both reused context from the previous tasks. Good. Hermesbot hallucinated: *"You've completed 1 workout (out of 11,700 needed)."* That 11,700 is the annual minutes target, not a workout count. It also claimed "1 hour and 28 minutes" of exercise when the data showed 28 minutes. The numbers were wrong and the math built on them was nonsensical. Clawbot repeated its Task 2 data consistently: 11 workouts, 289 minutes, exceeding both weekly targets. No contradictions, no hallucinated numbers. ### Task 4 Delete Workout ID Fake-Id-Does-Not-Exist This was the one task Hermesbot won. **Hermesbot** called `delete_workout(id="fake-id-does-not-exist")` directly, got an error ("Record to update not found"), and handled it gracefully: *"I don't see that workout in your recent sessions."* **Clawbot** called `get_workout` instead — an existence check rather than attempting the delete. It confirmed the ID didn't exist but never tried to delete it. If the ID had been real, it would have needed a second call. When the user says "delete X," doing the thing is better than checking whether you can do the thing. ### Task 5 Trend Analysis Am I Improving Plateauing or Declining Both agents fetched about a month of data (Hermesbot got 34 workouts, Clawbot got 32). Both provided reasonable breakdowns by source and activity type. The difference was in answering the actual question. Hermesbot gave generic encouragement — *"Your consistency is impressive!"* — without ever saying whether the trend was improving, plateauing, or declining. It dodged the question it was asked. Clawbot answered directly: **"Plateauing Phase — workout volume has stabilized around 1.0-1.1 workouts per day. No significant progression in duration or frequency."** Then it gave specific recommendations: add HIIT, schedule a long endurance ride, increase strength training. One agent answered the question. The other cheerleaded around it. ## The Scores I scored each task on six dimensions: tool accuracy (25%), response quality (25%), error handling (15%), autonomy (15%), speed (10%), and UX (10%). | Task | Hermesbot | Clawbot | Winner | |---|---|---|---| | 1. List 5 workouts | 69 | **94** | Clawbot (+25) | | 2. Sync + goals | 74 | **93** | Clawbot (+19) | | 3. How am I doing? | 64 | **95** | Clawbot (+31) | | 4. Delete fake ID | **92** | 80 | Hermesbot (+12) | | 5. Trend analysis | 80 | **93** | Clawbot (+13) | | **Average** | **75.8** | **91.0** | **Clawbot (+15.2)** | Clawbot won four of five tasks. Hermesbot won the delete task because it did what was asked instead of checking first. The margin wasn't close on Tasks 1 and 3 — those were presentation and accuracy failures from Hermesbot that the same underlying model didn't make under OpenClaw's prompting. ## Why OpenClaw Outperformed Hermes with the Same Model This is the result that should bother Nous Research. Hermes-4-14B is *their* model — trained on *their* tool-call format, shipped with *their* agent framework. OpenClaw is a third-party product that treats the model as a black box. And the black-box approach won 4 out of 5 tasks with a 15-point margin. The model is the same weights in both cases. Same GGUF file, same quantization, same GPU. The differences are entirely in how each framework wields those weights: **System prompt design.** Hermes Agent's system prompt, even after trimming to one toolset, is dense with agent behavior instructions, skill metadata, and framework-specific directives. It's optimized for the breadth of things Hermes Agent can do, not for the narrow task in front of it. OpenClaw's 26K-character system prompt is large too, but it structures tool availability differently — more catalog, less personality. The model gets different priming, and at 14B parameters, priming matters enormously. **Context management.** OpenClaw maintained cleaner context between turns. Hermesbot's compression (trigger at 85%, target 40%) may have been squeezing out the nuance the model needed for Tasks 3 and 5. When you're reasoning about goal metrics or workout trends, the details in earlier messages are the whole point. Compress them and you're asking the model to reason about data it can no longer see clearly. **Date interpretation.** "This week" became `since=today` in one framework and `since=Monday` in another. Same model, same training, different parameter choice. The system prompt or conversation framing influenced how the model interpreted an ambiguous time reference. This is a framework responsibility — and OpenClaw's framing led the model to the right answer. **Response formatting.** OpenClaw's prompting encouraged structured output — tables, headers, bullet points. Hermes Agent's prompting led to conversational but imprecise responses. On Task 1, Hermesbot fetched the data and offered to summarize it later. On Task 5, it cheerleaded instead of answering the question. These aren't model failures. They're framework choices that wasted a 14B model's limited capacity on filler instead of substance. The irony is real: vertical integration was supposed to be Hermes's advantage. The model trained on the framework's format. But in practice, the framework's overhead — the dense system prompt, the aggressive compression, the instruction-following style — worked against the model it was designed to serve. OpenClaw treated the same model with less ceremony and got more out of it. ## What I Actually Learned The scores don't matter as much as the process that produced them. **The tool-calling pipeline has four points of failure**, and each one is invisible from the others: 1. Tool definitions get injected into the prompt (or don't) 2. The model generates a tool call in its native format (or hallucinates one) 3. The inference server parses the tool call from the response (or silently drops it) 4. The framework executes the tool and feeds the result back (or doesn't) Each framework handles these differently. When something goes wrong, you're debugging a four-layer stack where any layer can fail silently. **Silent failures are the default.** `--chat-template chatml` doesn't warn you that it's ignoring tools. Hermes Agent doesn't warn you that 17 toolsets are consuming 60% of your context window. OpenClaw's trajectory logging reports `tools=0` even when tools are working. The assumption across the stack is that you know what you're doing, and the evidence suggests that nobody does on the first try. **Context arithmetic is unforgiving at 14B.** The model's actual context is 40,960 tokens. A 26K system prompt leaves about 15K for conversation, tool calls, and tool results. A single `list_workouts` response is 2,600 to 16,000 characters. Two complex tool calls in a conversation and you're brushing the ceiling. Cloud models with 128K–200K context windows don't have this problem. Local 14B models live on a knife's edge. **KV cache quantization is free performance.** Adding `--cache-type-k q8_0 --cache-type-v q8_0` to llama-server saved roughly 5 GB of VRAM with no noticeable quality loss. That's VRAM that can go to context length instead. If you're running local inference, do this. ## What's Next The original bakeoff plan called for a 2×2 matrix on Task 5 — both frameworks running both Hermes-4 and Qwen 3.6. I'm shelving that for now. Today's session was intensive enough. But Qwen is the model I want to test. Qwen 3.6 is my daily driver on this homelab — 35B parameters with only 3B active (MoE), 206 tok/s, fits in VRAM with room. The [research that preceded this bakeoff](/posts/hermes-agent-first-contact) flagged Qwen's TAG_WITH_TAGGED tool-call format as unreliable in llama.cpp. If the `--jinja` fix works as well for Qwen as it did for Hermes-4, that could change the calculus for daily use. There's also Gemma 4 12B sitting in the download queue — a dense 12B with 256K context. If a dense model with a larger context window performs better than a 14B with a 40K window on these same tasks, the model selection advice changes completely. Those tests will happen. Just not tonight. ## By the Numbers - **2** frameworks tested, same model, same hardware - **5** tasks, 100 points each - **12** total MCP tool calls across both agents (6 each) - **91.0 vs 75.8** — final scores (Clawbot over Hermesbot) - **4/5** tasks won by Clawbot; 1/5 by Hermesbot - **51 seconds** — Clawbot's total time for all 5 tasks - **26,477 characters** — OpenClaw's system prompt size - **40,960 tokens** — actual context window (model-capped from configured 65,536) - **2 rounds each** to get working — config debugging took longer than the actual test - **1 flag** — `--jinja` — that made the entire OpenClaw pipeline work - **17 → 1** — Discord toolsets trimmed to fix Hermesbot's context overflow - **0** things that worked on the first try === ## Updating Coder To Get User Secrets and the Art of Knowing Where Your Secrets Belong - URL: https://vibescoder.dev/posts/updating-coder-to-get-user-secrets-and-the-art-of-knowing-where-your-secrets-belong - Date: 2026-06-09 - Tags: #building-in-public #agents #meta #homelab #security - Reading time: 9 min read Coder 2.34 shipped User Secrets — per-user credential storage that injects into every workspace automatically. We upgraded, audited 29 secrets across four projects, and found exactly two that belonged there. Here's how we decided, how we migrated, and what we cleaned up along the way. --- Coder 2.34 dropped last week. It's the June Extended Support Release, which means it's the version the Coder team is telling you to park on if you want stability. It's also the version that finally gave me a reason to rethink how secrets flow into my workspaces. The headline features are Coder Agents improvements (chat sharing, personal skills, auto-configure models, GitLab integration) and a new User Secrets system. Most of the Agents improvements were already working for me — I've been using personal skills via `~/.agents/skills/` for weeks. But User Secrets caught my eye because it solves a problem I'd been working around with duct tape. ## What User Secrets Actually Is User Secrets is per-user secret storage built into Coder. You store a key-value pair once, and Coder injects it as an environment variable, a file, or both into every workspace you own. Automatically. At startup. No more copying `.env` files between workspaces. No more baking tokens into Terraform variables. No more extracting credentials from config files with `jq` hacks in startup scripts. The basics: - Up to 50 secrets per user, 24 KiB max per env var secret - Create via CLI (`coder secret create`), REST API, or the dashboard UI - Secret values are never shown after creation — write-only - Secrets apply at workspace start (restart to pick up changes) - Available in every shell, terminal, app, and SSH session It's in beta, and it's available in the open-source AGPL build — no Premium license required. ## The Upgrade The upgrade itself was straightforward. On the homelab workstation: ```bash curl -fsSL https://coder.com/install.sh | sh -s -- --version 2.34.0 sudo systemctl restart coder ``` One gotcha: my CLI session on the host had expired, probably from the version jump. `coder login` opens a browser, gives you a session token, and expects you to paste it back into the terminal. If your terminal fights you on paste, skip the interactive prompt: ```bash coder login http://localhost:3000 --token ``` ### Breaking Changes Worth Knowing The one that matters for most people: **AI Gateway is now enabled by default** (`CODER_AI_GATEWAY_ENABLED=true`). If you don't want it, explicitly set it to `false` in your Coder env. The other breaking changes are API-level — pointer fields in `patchTemplateMeta`, structured chat errors — and won't affect you unless you have custom API clients. ## The Audit 29 Secrets 4 Projects 2 Migrations Before throwing everything into User Secrets, I wanted to understand the full picture. Where do secrets live across my entire setup? What manages them? And which ones actually belong in User Secrets? I ran a comprehensive audit across four projects (coder-templates, the-vibe-coder blog, the-vibe-coder-content, fitness-tracker), plus workspace config files, shell profiles, and GitHub Actions workflows. The results: | Management Method | Count | Examples | |---|---|---| | Vercel env vars | 23 | Database URLs, OAuth secrets, API keys, encryption keys | | GitHub Actions secrets | 4 | Deploy hooks, mirror tokens, Slack webhooks | | Coder External Auth | 1 | GitHub token (auto-rotating) | | Needs migration | 2 | MCP bearer tokens | **29 total secrets. Only 2 belonged in User Secrets.** ### Why Most Secrets Don't Belong Here User Secrets solves one specific problem: getting credentials from the user into the workspace. It's not a replacement for Vercel env vars, GitHub Actions secrets, or any other secret management system. Here's how I thought about it: **Vercel env vars (23 secrets)** — These are server-side application secrets. Database connection strings, OAuth client secrets, API keys for Anthropic and Dev.to, encryption keys. They run on Vercel's infrastructure, not in Coder workspaces. Coder never needs them. **GitHub Actions secrets (4 secrets)** — These are CI/CD secrets. Deploy hooks, mirror repo tokens, Slack webhooks. They run in GitHub's infrastructure. Coder never needs them. **Coder External Auth (1 secret)** — `GITHUB_TOKEN` is already handled by Coder's external auth system, which auto-rotates OAuth tokens on every use. It's better than anything User Secrets could offer — dynamic, short-lived, automatically refreshed. **The two that fit (2 secrets)** — `FITNESS_TRACKER_MCP_TOKEN` and `VIBESCODER_MCP_TOKEN`. These are bearer tokens that MCP servers need inside workspaces. They're user-specific, workspace-relevant, and were previously managed through an awkward chain of Terraform variables and environment injection. ### The Old Way Was Ugly Here's how the fitness tracker token used to flow: 1. Set `TF_VAR_fitness_tracker_mcp_token=` in `/etc/coder.d/coder.env` on the Coder server 2. Terraform variable `var.fitness_tracker_mcp_token` picks it up in `main.tf` 3. Injected as `FITNESS_TRACKER_MCP_TOKEN` env var into the agent 4. Startup script conditionally merges a `fitness-tracker` entry into `~/.mcp.json` using `jq` 5. Agent skill file documents a jq fallback for extracting the token back out if the env var is missing Five steps. Three config files. A Terraform variable with a `sensitive = true` annotation. A `jq` pipeline in the startup script. And a documented fallback in the skill file for when the env var wasn't available in agent chat sessions. ### The New Way ```bash echo -n "" | coder secret create fitness-tracker-token \ --description "Fitness tracker MCP bearer token" \ --env FITNESS_TRACKER_MCP_TOKEN ``` One command. The token is available in every workspace, every shell, every agent session. No Terraform variables, no `jq` extraction, no fallback documentation. ## The Migration ### Step 1 Create the User Secrets On the host, for each token: ```bash echo -n "" | coder secret create fitness-tracker-token \ --description "Fitness tracker MCP bearer token" \ --env FITNESS_TRACKER_MCP_TOKEN echo -n "" | coder secret create vibescoder-token \ --description "Vibescoder MCP bearer token" \ --env VIBESCODER_MCP_TOKEN ``` Watch for the trailing newline warning — if you see `WARN: secret value from stdin ends with a trailing newline`, your token picked up a `\n`. Use `echo -n` carefully, and make sure the closing quote is on the same line as the token value. Fix with: ```bash echo -n "" | coder secret update vibescoder-token ``` ### Step 2 Clean up the Template Remove the Terraform variables that are no longer needed: ```hcl # REMOVED — these variables variable "fitness_tracker_mcp_token" { type = string default = "" sensitive = true } variable "vibescoder_mcp_token" { type = string default = "" sensitive = true } ``` Remove the env injections from the `coder_agent` resource: ```hcl # REMOVED — these env entries FITNESS_TRACKER_MCP_TOKEN = var.fitness_tracker_mcp_token VIBESCODER_MCP_TOKEN = var.vibescoder_mcp_token ``` The startup script that merges MCP entries into `~/.mcp.json` stays — it still needs to read the token from the environment. It just reads it from User Secrets now instead of from Terraform variables: ```bash # Token is now injected via Coder User Secrets. if [ -n "${FITNESS_TRACKER_MCP_TOKEN:-}" ]; then jq --arg auth "Bearer ${FITNESS_TRACKER_MCP_TOKEN}" \ '.mcpServers["fitness-tracker"] = { command: "npx", args: [ "mcp-remote", "https://your-mcp-server.example.com/api/mcp/mcp", "--header", ("Authorization: " + $auth) ] }' ~/.mcp.json > ~/.mcp.json.tmp \ && mv ~/.mcp.json.tmp ~/.mcp.json fi ``` ### Step 3 Clean up the Server Remove the now-dead `TF_VAR` lines from the Coder server config: ```bash sudo sed -i '/TF_VAR_fitness_tracker_mcp_token/d' /etc/coder.d/coder.env sudo sed -i '/TF_VAR_vibescoder_mcp_token/d' /etc/coder.d/coder.env ``` Verify they're gone: ```bash grep -i "mcp_token" /etc/coder.d/coder.env # Should return nothing ``` ### Step 4 Apply and Test Pull the template changes and push to Coder: ```bash cd ~/coder-templates && git pull && ./docker/apply.sh ``` Restart a workspace, then verify: ```bash echo "FITNESS_TRACKER_MCP_TOKEN: $([ -n "$FITNESS_TRACKER_MCP_TOKEN" ] && echo 'YES' || echo 'NO')" echo "VIBESCODER_MCP_TOKEN: $([ -n "$VIBESCODER_MCP_TOKEN" ] && echo 'YES' || echo 'NO')" ``` Both should say YES. Test an actual API call to confirm the token works: ```bash curl -sS -X POST https://your-mcp-server.example.com/api/mcp/mcp \ -H "Authorization: Bearer $FITNESS_TRACKER_MCP_TOKEN" \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":1,"method":"tools/call", "params":{"name":"list_workouts","arguments":{"limit":1}}}' ``` ### Bonus Update the Skill File The fitness-tracker agent skill had a documented jq fallback for when the env var wasn't available. That fallback is now dead code in the documentation. Updated from: ```markdown - Fallback if the env var is missing — extract from `~/.mcp.json`: export FITNESS_TRACKER_MCP_TOKEN=$( jq -r '.mcpServers["fitness-tracker"].args[-1]' ~/.mcp.json \ | sed 's/^Authorization: Bearer //' ) ``` To: ```markdown - Injected automatically via Coder User Secrets into every workspace. - Verify with: `[ -n "$FITNESS_TRACKER_MCP_TOKEN" ] && echo "ok"` ``` Simpler docs, simpler mental model. ## The .env.example Cleanup While auditing secrets, we found two documentation gaps: **the-vibe-coder** was missing 4 env vars from its `.env.example`: `MCP_API_TOKEN`, `VERCEL_DEPLOY_HOOK`, `SLACK_SIGNING_SECRET`, and `APP_BASE_URL`. All were properly set in Vercel — just undocumented for anyone reading the repo. **fitness-tracker** had no `.env.example` at all. 13 environment variables, zero documentation. We created one with sections for database, authentication, MCP/API, encryption, Peloton, Tonal, and app config. Neither of these was a security issue — secrets were properly managed in Vercel. But if you can't figure out what env vars an app needs by reading the repo, you'll figure it out the hard way: runtime errors. ## Housekeeping Fixing Git Identity for Multi-User Setups Unrelated to secrets, but worth mentioning: my wife uses the same Coder instance and kept having commits attributed to her Gmail address instead of her GitHub handle. Her agent would use the right identity for a while, then forget. The root cause: she logs into Coder with Gmail, so `coder_workspace_owner.me.email` returns her Gmail address. That's what the template was setting as `GIT_AUTHOR_EMAIL`. I had a manual workaround — a hardcoded map in the template: ```hcl git_email_overrides = { vibalknowledge = "209802621+VibalKnowledge@users.noreply.github.com" } ``` This worked at the Terraform level but didn't survive agent sessions that reset git config. And it doesn't scale — every new user needs a manual entry. The fix: replace the static map with a startup script that dynamically fetches the GitHub identity using the already-authenticated token: ```bash if [ -n "$GITHUB_TOKEN" ]; then GH_USER=$(gh api /user --jq '.login // empty' 2>/dev/null) GH_NAME=$(gh api /user --jq '.name // empty' 2>/dev/null) if [ -n "$GH_USER" ]; then GH_NOREPLY=$(gh api /user --jq '"\(.id)+\(.login)@users.noreply.github.com"' 2>/dev/null) git config --global user.email "$GH_NOREPLY" git config --global user.name "${GH_NAME:-$GH_USER}" fi fi ``` Now every user's git identity is automatically set from their GitHub profile at workspace startup. No manual overrides, no agent amnesia, no per-user config. Login with Gmail, Google, OIDC, carrier pigeon — as long as you've linked GitHub via Coder's external auth, your commits attribute correctly. ## By the Numbers - **Version jump**: 2.33.2 → 2.34.0 - **Secrets audited**: 29 across 4 projects - **Secrets migrated to User Secrets**: 2 - **Terraform variables removed**: 2 - **Server config lines removed**: 2 - **Lines of jq fallback documentation deleted**: 7 - **`coder login` paste failures**: 1 (use `--token` flag) - **Trailing newline warnings**: 1 (use `echo -n` carefully) - **Git identity hardcodes eliminated**: 1 per user, forever - **.env.example files created or updated**: 2 - **Time from upgrade to fully migrated**: ~45 minutes === ## Friday Fixes: Housekeeping the Homelab and Hub - URL: https://vibescoder.dev/posts/friday-fixes-housekeeping-the-homelab-and-hub - Date: 2026-06-05 - Tags: #meta #building-in-public #agents #llm #next-js #syndication - Reading time: 11 min read A model refresh on the homelab (Qwen 3.6, new embeddings, 469 llama.cpp builds), a feature sprint on the vacation planning site (calendar sync, expense tracking, and three bugs that taught us more than the features did), and automating Substack syndication after discovering two more undocumented quirks. Three unrelated workstreams, one theme: maintenance is where the real learning happens. --- Some weeks you ship a big feature. Other weeks you sweep the floor so the big features keep working. This was a floor-sweeping week — two completely unrelated workstreams that both needed attention. **Track one**: the homelab's local LLM stack hadn't been touched in a month. Models were stale, llama.cpp was 469 builds behind, and the embedding model was a generation old. **Track two**: the [vacation planning site I open-sourced](/posts/forking-and-open-sourcing-a-single-purpose-site) needed to actually be useful for a group trip. Calendar sync, activity voting, expense tracking — the features that turn a brochure into a tool. **Track three**: the [Substack syndication pipeline](/posts/syndicating-to-substack-the-undocumented-path) I wrote about earlier this week? Turns out doing it once was the easy part. Doing it *every time* surfaced two more undocumented quirks and required a GitHub Action to paper over them. None of these stories is glamorous on its own. Together they're a snapshot of what maintenance week looks like when you're building with an agent. --- ## Part 1 Homelab Model Refresh The homelab runs llama.cpp on an RTX 5090 with six switchable models. The agent audited everything and came back with a report card: | Component | Before | Verdict | |---|---|---| | llama.cpp | b8933 | 469 builds behind | | Qwen (daily driver) | 3.5 35B-A3B | 3.6 available | | Embedding | nomic-embed v1.5 | v2-moe available | | Gemma 4, Devstral, DeepSeek | Current | No action needed | | Codestral | v0.1 (2024) | Dead end — Mistral pivoted to Devstral | Three downloads, ~38 GB total: Qwen 3.6, nomic-embed v2-moe, and a new addition — Qwen3-Coder-30B-A3B, a coding-specialized MoE that fits at 17 GB. ### The Quant Trap The interesting discovery was about quant provenance. Our Qwen model uses `UD-Q4_K_XL` quantization — the "XL" quants use higher precision on attention layers while keeping MoE expert layers smaller. These are **unsloth-specific**. Bartowski (the other major GGUF publisher) doesn't offer them. The agent initially found the bartowski version and we had to redirect it to unsloth to get the same quant type we were already running. This matters because quant format affects output quality in ways that aren't obvious from the model name alone. `Q4_K_M` and `Q4_K_XL` are both "4-bit" but they allocate precision differently. Swapping quant types during an upgrade is an uncontrolled variable. ### Script Updates The homelab's model switching lives in a shell script (`llm-switch.sh`) that maps model names to file paths and llama-server flags. Updates: Qwen path from 3.5 to 3.6, new `qwen-coder` case with 128K context, embedding path from v1.5 to v2-moe, Codestral marked `[legacy]`. **Gotcha**: Pasting heredoc scripts into the terminal mangled backslashes and quoting. We switched to writing the scripts in the workspace, pushing to GitHub, and giving me a `git pull && cp` one-liner. Lesson: don't paste shell scripts through chat — commit them. ### After State | Component | Before | After | |---|---|---| | llama.cpp | b8933 | **b9402** | | Generation model | Qwen 3.5 | **Qwen 3.6** | | Embedding model | nomic v1.5 (262 MB) | **nomic v2-moe** (914 MB) | | Switchable models | 5 | **6** (added qwen-coder) | | VRAM | 26,262 MiB | 26,682 MiB (+420 MiB) | About 20 minutes wall clock from audit to fully updated, zero downtime. The old models still serve until you restart the service with the new binary. --- ## Part 2 Vacation Hub Feature Sprint The [vacation hub](https://github.com/carryologist/vacation-hub) is a forkable trip-planning site — deploy to Vercel, run the setup wizard, and your group has a private site for travel notes, itinerary, lodging, activities, photos. I [wrote about open-sourcing it](/posts/forking-and-open-sourcing-a-single-purpose-site) last week. This week was about making it useful. Four features across three days, 11 commits, 3,484 lines added. But the features aren't the interesting part. The bugs are. ### Calendar Sync the Straightforward One People need trip events in their phone's calendar. Two options: download a `.ics` file (one-time import) or subscribe to a URL (auto-syncing). The download is trivial — click a button, get a file. The subscription is the interesting engineering problem. Google Calendar, Apple Calendar, and Outlook all fetch subscription URLs from their servers. No browser, no cookies. So the endpoint needs an auth mechanism that works without a session. We went with a deterministic HMAC token: `HMAC-SHA-256('calendar-subscribe', VACATION_HUB_SECRET)`. The export endpoint accepts either a cookie (for browser downloads) or a `?token=` param (for calendar clients). No expiry — a time-limited token would silently break subscriptions when it expires and there's no user present to re-authenticate. The iCal generator itself is 202 lines, built from scratch against RFC 5545. The subtle part is line folding — the spec requires max 75 *octets* per line, not characters. You can't just `.slice(75)` because you might split a UTF-8 multi-byte character. The fold function walks backward from the cut point checking continuation bytes. Most iCal libraries get this wrong and corrupt non-ASCII event names. ### Activity Voting the Bug Factory Reddit-style upvote/downvote on suggested activities. Name-based identity (localStorage, no accounts). Upsert voting so changing your mind is idempotent. This feature worked perfectly in development and completely failed in production. Twice, for two different reasons. **Bug 1 — The Trailing Slash Massacre**: `next.config.ts` has `trailingSlash: true`, which makes Next.js issue 308 redirects from `/api/foo` to `/api/foo/`. The redirect preserves the HTTP method but the browser drops the request body. Every POST, PUT, and DELETE arrived at the API with an empty body. GET requests (page loads, data fetching) worked fine, so the site *looked* healthy — only mutations were silently failing. The fix: add trailing slashes to all 28 `fetch()` calls across 12 files. Eight minutes to fix, 40 minutes to diagnose. `trailingSlash: true` is a foot-gun for API routes — fine for page navigation, lethal for `fetch()`. **Bug 2 — The Table That Never Existed**: After fixing trailing slashes, voting *still* didn't work. The `activity_votes` table didn't exist on production. It existed in development because the dev database didn't have duplicate activity titles. The `initializeDatabase()` function runs CREATE TABLE statements sequentially in a single try block. After creating the `activity_suggestions` table, it tries to create a unique index on the `title` column. Production had duplicate titles (imported via LLM-generated suggestions). The index creation threw, the catch block caught it, and the function exited before reaching `CREATE TABLE activity_votes`. The debugging journey: deploy a temporary `/api/db/debug/` endpoint → confirm the table is missing → trace the init function → find the ordering dependency → wrap the index creation in its own try/catch → re-run init → delete the debug endpoint. Two commits, two minutes apart. The lesson: every DDL statement in an init function should be its own try/catch. A failure to create an index on table A should never prevent table B from being created. ### PDF Upload Fix the Serverless Trap This one predated the feature sprint but came up during testing. PDF itinerary uploads worked locally, failed on Vercel with a cryptic module error. The `pdf-parse` npm package bundles an ancient version of PDF.js that uses dynamic `require()`. Vercel's bundler traces imports statically and prunes anything it can't resolve. The module exists in `node_modules` locally but vanishes after bundling. Bonus discoveries while debugging: - The upload endpoint returned "Something went wrong" for all errors. We had to add real error logging before we could even *see* the pdf-parse failure. - iOS Safari sends an empty MIME type for PDFs. The validation rejected them. - Vercel has a 4.5MB body limit for serverless functions. The original limit was 10MB. Replaced `pdf-parse` with `unpdf` (serverless-compatible). Three files changed, 21 insertions, 38 deletions. The kind of fix that's trivial once you know the root cause and impossible until you do. ### Expense Management the Big One 2,108 lines across 13 files. Track who paid for what, scan receipts with AI, show who owes whom. The receipt scanning supports three LLM providers — same ones the site already uses for itinerary parsing. Each has its own quirks: OpenAI accepts image URLs directly, Anthropic and Gemini require base64 encoding. OpenAI and Gemini support structured JSON output, Anthropic requires regex extraction from prose. For PDFs, all three get extracted text rather than the visual layout. **The design pivot that mattered**: The original plan had per-expense split counts. "This $200 dinner was split 4 ways." In practice, the form was cluttered and the answer was almost always the same number. We changed to a global "Splitting between N people" control at the top of the page. The form went from three columns to two. Settlement computation moved from a server endpoint to a `useMemo` hook — because the split count is a UI concern (you might flip between values while looking at the numbers), not persistent data. We built the server endpoint, shipped it, realized it was wrong, moved the logic client-side, and deleted the endpoint. Normal lifecycle. ### The Cleanup After the feature sprint, we went back and deleted dead code: - `/api/expenses/settle/route.ts` — settlement moved client-side - `/api/og-image/route.ts` — only consumer was the activity POST handler, which we'd stripped during the Things to Do redesign - The OG image fetch block in the activity POST handler itself 363 lines deleted. We also went back to the expense feature's design doc and annotated it with what actually shipped versus what was planned. There's something honest about marking your own plan with "this part we built differently." The plan is the record of what you thought before you knew better. The code is what you actually shipped. --- ## Part 3 Automating Substack Syndication I [wrote up the initial Substack import](/posts/syndicating-to-substack-the-undocumented-path) earlier this week — 13 curated posts, an RSS feed filtered by a `syndicate: true` frontmatter flag, and a GitHub mirror repo to work around Substack rejecting feeds from our domain. That got the backlog in. This week's Thursday Thoughts post was the first one I needed to push *after* the initial import. It didn't go smoothly. ### Two More Dedup Quirks **Quirk 1 — per-feed-URL dedup.** Substack doesn't just dedup by GUID. It dedupes by *feed URL*. If you add a new post to `syndicate.xml` and re-import the same URL, Substack silently skips the new item. The existing 13 posts aren't reimported (good), but the new 14th post isn't imported either (bad). No error. The import API returns 200 and reports it found 14 posts. It just doesn't do anything with the new one. The workaround: a separate `single-import.xml` file containing only the new post, with a timestamped GUID that Substack has never seen. Different URL, different GUID, different dedup bucket. **Quirk 2 — Cloudflare blocks GitHub Actions.** The live feed at `vibescoder.dev/syndicate.xml` returns 403 when fetched from GitHub Actions runners. Same IP reputation issue that made Substack reject the feed in the first place — Vercel sits behind Cloudflare, and Cloudflare's bot protection doesn't love datacenter IP ranges. `curl` from a laptop works fine. `curl` from `ubuntu-latest` on Actions gets a wall. ### The Workflow The automation lives as a GitHub Action in the content repo (where posts are pushed). On any push to `content/posts/`: 1. Wait 90 seconds for Vercel to rebuild 2. Fetch the live `syndicate.xml` (with retry and user-agent headers to appease Cloudflare) 3. Clone the mirror repo and diff GUIDs to find new posts 4. Update `syndicate.xml` in the mirror, preserving existing GUID busts from prior imports 5. Generate `single-import.xml` with a unique timestamped GUID 6. Push to the mirror repo 7. Post a summary in the Actions run with the Substack import URL The last step is manual — you paste the URL into Substack's import UI. Substack's import API exists but requires session authentication, and there's no official way to get a token. Fully automated posting would need the [`python-substack`](https://github.com/ma2za/python-substack) library, which reverse-engineers the auth flow. That's a project for when I have more than one subscriber. For now: push a post with `syndicate: true`, wait for the Action to run, paste one URL. Three minutes end-to-end, zero chance of forgetting to update the mirror. --- ## By the Numbers **Homelab:** - **3 models** downloaded (38 GB) - **469 llama.cpp builds** caught up (b8933 → b9402) - **6 switchable models** (was 5, added qwen-coder) - **420 MiB** VRAM increase from the embedding upgrade - **~20 minutes** wall clock from audit to fully updated **Vacation Hub:** - **11 commits** over 3 days - **35 files changed**, 3,484 lines added, 702 deleted - **4 features** shipped (calendar sync, voting, page redesign, expenses) - **3 production bugs** fixed (trailing slash, missing table, pdf-parse) - **28 fetch() calls** fixed with trailing slashes in one commit - **202 lines** for a from-scratch RFC 5545 iCal generator - **2,108 lines** for expense management in a single commit - **363 lines** deleted during cleanup - **1 npm package** replaced (pdf-parse → unpdf) - **0 user accounts** — names in localStorage and a prayer **Substack Syndication:** - **2 undocumented quirks** discovered (per-feed-URL dedup, Cloudflare blocking Actions) - **1 GitHub Action** to auto-sync the mirror repo on every content push - **1 manual step** remaining (paste the import URL into Substack) - **~3 minutes** end-to-end per syndicated post, down from ~15 minutes manual === ## Thursday Thoughts: How AI-Native Mirrors Cloud-Native - URL: https://vibescoder.dev/posts/thursday-thoughts-how-ai-native-mirrors-cloud-native - Date: 2026-06-04 - Tags: #future-of-coding #agents #meta #building-in-public - Reading time: 5 min read At a C-suite roundtable in Palo Alto last week, ten-plus executives from a mix of gaming platforms, enterprise systems providers, job sites, and other Bay Area titans landed on the same analogy without being prompted: we've seen this before. The lift-and-shift era of AI is already here. The native era — where you redesign workflows from scratch for agents, not humans — is what comes next. --- Last week I attended a C-suite roundtable in Palo Alto with ten executives from the usual smattering of Bay Area titans — a gaming platform, a large systems provider, a major job site, and others. The intent was to get concrete signal on where enterprises are with AI. We got quite a lot of pontificating and waxing poetic. All typical with CxOs. And, candidly, more useful for me. When executives stop being concrete, they start being honest about the shape of the problem. Two things landed hard. ## Everyone Acknowledged Agents Are Already in Production without Guardrails This came up without us even raising it. Governance — who controls what agents can do, how you audit what they did, how you stop a runaway workflow — was the topic of the room. Not a topic. The topic. That's validating in a specific way: the thing that feels like an edge concern when you're deep in the tooling turns out to be the exact thing keeping senior people up at night once they're actually running agents against real systems. The gap between "we deployed an agent" and "we have any idea what it's doing" is apparently wider than most companies expected. ## The Cloud-Native Analogy Clicked for the Whole Room This one I want to dwell on, because I think it's the clearest frame I've found for where we are. When enterprises first moved to the cloud, most of them did lift-and-shift. They took their existing workloads — unchanged, same architecture, same assumptions — and ran them on AWS instead of on-prem. You got some cost benefits, some flexibility. But you weren't really using the cloud. You were renting someone else's servers. The transformation that actually mattered came later, when teams started redesigning applications *for* the cloud. Microservices instead of monoliths. Event-driven architectures. Stateless services that scaled horizontally. Those apps weren't better versions of the old apps. They were different apps, built around what the cloud made possible. We are doing the exact same thing with AI right now. The lift-and-shift era of AI is: take a human workflow, hand it to an agent, and call it automation. An agent fills out the form. An agent reads the documents. An agent follows the process someone designed for a person to follow. You get some productivity gains. But you're running the old workload on new infrastructure. The AI-native era — which these executives were all saying we're about to enter — is when you stop asking "how do we get an agent to do this human task?" and start asking "what would this workflow look like if we designed it for agents at scale from the beginning?" The answer is usually not a faster version of the old thing. It's a different thing. ## The Roles Question Is the One Nobody's Answered Yet Cloud-native didn't just change how applications were built. It created entirely new job categories. DevOps didn't exist before the cloud forced a rethink of how you deploy and operate software. SREs emerged because reliability at cloud scale required a different discipline than ops at on-prem scale. The new architecture required new ways of working around it. The executives in that room were unanimous that the same thing is coming with AI — AI ops, ML ops, whatever we end up calling the people who manage, audit, and operate agent-native workflows — but nobody in the room had actually built those functions yet. They know they need them. They haven't invented them. That gap is interesting. It means the companies that figure out the operating model — not just the technology — are going to have a real edge. The architecture is the easier part. The organizational design is where most enterprises are still staring at a blank page. ## The Business Model Shift Is the Wildcard One thing from the conversation that's still rattling around: the cloud era was about doing things better, faster, cheaper. The same metrics, just improved. What the executives were saying about AI is different — that it's going to force a change in *how companies measure themselves*, not just how efficient they are. Revenue per employee came up specifically. The argument being: once your workforce is partly human and partly agentic, headcount-normalized metrics stop making sense, and you need metrics that account for what your agents are doing alongside your people. Revenue per employee captures the full capacity of the team, human and agent. Do you break out human versus AI employees? TBD. The consensus was yes, but I think even that will normalize. That's a bigger shift than any of the technology. Business model changes outlast technology cycles. --- The roundtable ended with a lot of good conversation and connections. But the frame that stuck with me is: we've been through this before. Cloud-native looked impossible from the lift-and-shift era and obvious in retrospect. AI-native probably looks the same from where we're standing now. The lift-and-shift phase isn't a mistake — it's how you learn the infrastructure well enough to rethink the architecture. Just don't stop there. ## By the Numbers - **10 executives** — size of the C-suite roundtable in Palo Alto that sparked this post - **3+ industries** represented at the table: gaming, enterprise systems, job sites, and others - **2 things** that landed hardest in the room: agents already running in production without guardrails, and the cloud-native/AI-native parallel - **1 metric** singled out as due for a rewrite: revenue per employee, once headcount blends human and agentic work - **0** AI-ops-style functions the room had actually built yet, despite unanimous agreement they're coming === ## Hermes Agent: First Contact - URL: https://vibescoder.dev/posts/hermes-agent-first-contact - Date: 2026-06-02 - Tags: #agents #llm #building-in-public #meta - Reading time: 7 min read I've been running OpenClaw on the homelab for a month. A recommendation sent me down the Hermes Agent rabbit hole — and the research before the first real test revealed my daily driver model was broken for tool calling all along. --- Someone recommended I look at Hermes Agent as an alternative to OpenClaw. I've been running OpenClaw on the homelab since early May — it drives a Discord bot backed by Qwen 3.6 on an RTX 5090, with MCP tools wired into a fitness tracker. It works, mostly. The "mostly" is why I was open to alternatives. What I expected: a quick install, a side-by-side comparison, a blog post with a verdict. What I got instead was a research rabbit hole that changed my understanding of why my existing setup had been flaky in the first place. ## What Hermes Actually Is Two things from the same org (Nous Research): 1. **Hermes Models** — fine-tuned LLMs trained specifically for function calling, with native `` tokens baked into the weights. The model knows the tool-calling grammar because it was trained on it. 2. **Hermes Agent** — a Python-based agent framework with 90+ built-in tools, a skills/learning system, and integrations for 25+ messaging platforms. The key difference from OpenClaw: **vertical integration.** Nous Research makes both the model and the framework. The model was trained on the agent's tool schema. OpenClaw treats the model as a black box — plug in any OpenAI-compatible endpoint and go. Hermes pairs the model with the exact format it was trained to produce. That distinction sounded like marketing until the research phase made it concrete. ## The Install One-liner installer, clean enough. It provisions its own Python, pulls 90 skills, detects my existing OpenClaw installation and offers a migration preview. The migration is thoughtful — it shows what it would import (soul config, memories, Discord settings, MCP servers) and warns about semantic mismatches. I skipped it. Importing OpenClaw's personality into Hermes would muddy any comparison. The rough edges are in the setup wizard: - **Portal login ambush.** The first thing the wizard does — even after selecting "Quick setup" — is open a browser to the Nous Portal pricing page. If you're running local inference, this is confusing. You don't need an account. But there's no obvious "skip" button. You Ctrl+C out, which feels like you're breaking something. ![The Nous Portal pricing page that opens unbidden during setup](/images/hermes-agent-first-contact/portal-pricing-ambush.png) ![The Portal login flow — Ctrl+C is the only way out](/images/hermes-agent-first-contact/portal-login-no-skip.png) - **Sudo password storage.** It asks if you want Hermes to store your sudo password for running apt commands. I said no. Don't want an agent framework I'm evaluating holding root credentials. - **Default model display.** After setup, it shows `anthropic/claude-opus-4.6` as the current model — even though no API key is configured and no cloud provider is connected. Misleading. None of these are dealbreakers. They're first-impression friction that an open-source project with 172K GitHub stars could smooth out. The install itself took about ten minutes, model download included. ![The 25+ messaging platform selection list — we configured zero of them](/images/hermes-agent-first-contact/messaging-platforms.png) ## The Research That Changed Everything Before running any comparison, I wanted to pick the right models. The obvious plan: OpenClaw runs Qwen 3.6 (my daily driver, the model it's been using for weeks), Hermes runs Hermes-4-14B (its native model). Each framework gets its best model. Fair fight. Then I started reading GitHub issues. There's an open llama.cpp issue titled, with admirable directness, **"qwen3.6-27b not work with openclaw."** The problem is in how llama.cpp handles Qwen's tool-call format. llama.cpp's tool-call autoparser recognizes three formats: | Format | How It Works | Models | |---|---|---| | **JSON_NATIVE** | Pure JSON tool calls | Cleanest, fewest bugs | | **TAG_WITH_JSON** | Function name in XML tag, arguments as JSON | Hermes models | | **TAG_WITH_TAGGED** | Everything in nested XML tags | Qwen models | Qwen uses TAG_WITH_TAGGED — the most complex format. Tool calls look like `value`. Multiple open issues describe parser failures, tool calls leaking into reasoning blocks, and permanently wedged conversations when parameters contain arrays. I built a compatibility ranking across every model on the homelab: | Model | Format | Tool-Call Reliability | |---|---|---| | Hermes-4-14B | TAG_WITH_JSON | ★★★★★ | | Gemma 4 | Custom parser | ★★★★☆ | | Devstral | Mistral format | ★★★☆☆ | | Qwen 3.6 | TAG_WITH_TAGGED | ★★☆☆☆ | | Qwen3-Coder | TAG_WITH_TAGGED | ★★☆☆☆ | | DeepSeek R1 | Unicode delimiters | ★☆☆☆☆ | Qwen — my daily driver, the model I'd been running with OpenClaw for three weeks — ranked fourth out of six for tool calling. The flaky behavior I'd attributed to "OpenClaw being finicky" or "memory-core having bugs" may have been Qwen's tool-call format failing to parse all along. ### The Bootstrap Problem More community research surfaced a second issue. OpenClaw's default bootstrap injects ~27,000 characters of system prompt — agent identity, tool schemas, conversation rules. Models at 14B parameters or below can't handle it. They hallucinate tool use as text instead of emitting structured calls. The fix documented in the issue tracker: slash `bootstrapMaxChars` from 12,000 to 1,500. That's an 88% reduction in system prompt for the model to chew on before it even sees the user's message. ## The Experiment Design The research inverted the original plan. Instead of "each framework gets its native model," both agents will run **Hermes-4-14B**. Same model, different frameworks. That isolates the framework variable — any performance difference is the orchestration, not the weights. Five tasks, escalating complexity, all via Discord against a fitness-tracker MCP server: | Task | Tests | |---|---| | List last 5 workouts | Basic single tool call | | Sync Peloton → weekly count → goal pace | Multi-step tool chain | | "How am I doing?" | Ambiguous intent, tool selection | | Delete a fake workout ID | Error handling and recovery | | Full 2025 fitness trend analysis | Multi-turn agentic reasoning | Task 5 opens into a 2×2 matrix — both agents on both Hermes-4 and Qwen 3.6 — to measure how much the model format matters versus the framework. One deliberate asymmetry: Hermes keeps its memory and learning loop active across all five tasks. OpenClaw's `memory-core` is disabled due to an upstream bug. This isn't a controlled variable — it's a real product difference. We're testing each agent at its best available configuration, not at its lowest common denominator. ## What I Learned Before Testing Anything The most useful discovery came before running a single experiment. I'd been blaming OpenClaw for flaky tool calling. The actual culprit was probably Qwen's TAG_WITH_TAGGED format — deeply nested XML that llama.cpp's parser struggles with. The `memory_search` hangs I'd attributed to a memory-core bug? Possibly Qwen's tool calls never parsed correctly in the first place, leaving the chain dangling on an await that could never resolve. Vertical integration isn't just a marketing story. When the model is trained on the exact tool-call format the agent expects, you skip an entire class of parsing bugs. Hermes-4-14B produces TAG_WITH_JSON — function name in a tag, arguments as clean JSON. llama.cpp strips the wrapper and passes it through. No nested XML, no parameter tags, no parser edge cases. Whether that translates to better real-world performance is what the bakeoff will answer. But the prep work already taught me something: **the model I thought was working was only partially working**, and I wouldn't have known without researching a replacement. ## By the Numbers - **1 GitHub issue** titled literally "qwen3.6-27b not work with openclaw" - **6 models** evaluated for tool-calling compatibility - **3 tool-call formats** in llama.cpp (JSON_NATIVE, TAG_WITH_JSON, TAG_WITH_TAGGED) - **27,000 chars** — default OpenClaw bootstrap prompt; **1,500** — recommended for ≤14B models - **90 skills** bundled with Hermes Agent out of the box - **25+ messaging platforms** supported (we configured zero of them) - **10 minutes** from download to installed - **0 experiments run** — and still the most useful research session of the week === ## Syndicating to Substack: The Undocumented Path - URL: https://vibescoder.dev/posts/syndicating-to-substack-the-undocumented-path - Date: 2026-06-01 - Tags: #syndication #building-in-public #seo - Reading time: 12 min read Substack supports RSS import, but the importer is finicky, undocumented, and rejects feeds for reasons it won't tell you. Here's how we got 13 curated posts from a Next.js blog into Substack — and what every other guide leaves out, including the dedup gotcha that bit us on the re-import. --- Substack's onboarding screen lists nine platforms it "seamlessly imports from": Medium, Ghost, WordPress, Mailchimp, Beehiiv, SeekingAlpha, Tumblr, TinyLetter, Blogspot. Underneath, in smaller text: *"...or website with an RSS feed."* That second sentence is the one that matters for any blog not on the list. It's also the sentence Substack has the least documentation on, the most quirky failure modes around, and — based on the four hours this took us — the most undocumented edge cases. This is the writeup of getting **vibescoder.dev** (a custom Next.js blog with MDX content) onto Substack as a curated 13-post syndication. Every error we hit. Every dead end we ran down. Every workaround that ended up shipping. ## The Starting Position Vibescoder.dev has lived on its own domain since April. The stack is Next.js 16 on Vercel, MDX content in a private GitHub repo, deploy on push. The blog already had a working RSS feed at `/feed.xml` with all 34 published posts, full content in ``. Total feed size: **544 KB**. The goal: get the *curated subset* (essays and frameworks, not build logs and homelab posts) onto Substack as a one-time bulk import. Treat Substack as a distribution channel, keep vibescoder.dev as the canonical home, use rel="canonical" to consolidate link equity back to the domain. Simple, on paper. ## Failure 1 Onboarding Flow Rejects Unknown Domains First attempt: paste `https://vibescoder.dev/feed.xml` into `substack.com/signup/import`. > **Unable to fetch any posts from this URL.** The onboarding importer at `/signup/import` is **gated to known platforms**. It pattern-matches the URL against domains it recognizes — `*.medium.com`, `*.ghost.io`, `*.wordpress.com`, etc. — and rejects anything else without trying very hard to parse it. The error wording suggests it tried to fetch your URL; in practice it barely tried at all. **The fix:** create the publication first (skip the import step on signup), then use the in-dashboard importer at **Settings → Import → Import posts**. That importer is more permissive and actually tries to parse arbitrary RSS. ## Failure 2 in-Dashboard Importer Also Rejects the Feed The in-dashboard importer at least *tries*. But it also returned: > **Unable to fetch any posts from this URL.** Time to actually diagnose instead of guessing. The Substack JS bundle reveals the API endpoint: `POST /api/v1/import/posts` with `{url: "..."}` in the body. Hitting it directly: ```bash curl -sX POST "https://substack.com/api/v1/import/posts" \ -H "Content-Type: application/json" \ -H "Origin: https://substack.com" \ -H "Referer: https://substack.com/signup/import" \ --data '{"url":"https://vibescoder.dev/feed.xml"}' ``` ```json {"error":"Unable to fetch any posts from this URL.","type":"single"} ``` 400. Same error, but now scriptable. This is the diagnostic loop unlock — every change we make from here is testable in two seconds. ## What Does Work To narrow the search, we pointed the same endpoint at six known-working blog feeds: | Feed | Size | `` | Substack import | |---|---:|---|---| | overreacted.io/rss.xml | 23 KB | No | ✅ 57 posts | | stratechery.com/feed | 47 KB | Yes | ✅ 10 posts | | kentcdodds.com/blog/rss.xml | 97 KB | No | ✅ 211 posts | | joshwcomeau.com/rss.xml | 114 KB | Yes | ✅ 86 posts | | leerob.com/feed.xml | (404 ATM) | — | ❌ | | **vibescoder.dev/feed.xml** | **544 KB** | **Yes** | ❌ | Pattern visible: the working feeds top out around 114 KB. Ours was almost 5× that. **Hypothesis: Substack rejects feeds above some size threshold.** ## The First Rebuild Structural Alignment Before doing anything about size, we rebuilt the feed to mirror Ghost's structure (the platform Substack imports most cleanly from). Changes: - Switched `email (Name)` to `Name`. RSS 2.0 requires emails in ``, which exposes the writer's address and trips some importers' privacy filters. - Added `xmlns:dc` to the `` root. - Added ``, ``, and `` to the channel. - Set `` so parsers don't try to validate the guid as a URL. - Changed `Content-Type` from `application/xml` to `application/rss+xml`. Result: still rejected. ## The Size Theory Looks Confirmed but Isn't To test the size hypothesis without breaking the main feed, we added a second route: `/syndicate.xml`. Same Ghost-style structure, but only 13 posts (filtered via a new `syndicate: true` frontmatter flag), and with `` *omitted entirely*. The thinking: Substack would follow each `` URL and parse the article from the HTML page, the same way they do for Medium imports. Result: feed dropped to **9.7 KB**. Still rejected. So size isn't the only thing. Or it isn't the thing at all. ## The Smoking Gun The breakthrough came from running an experiment we should have tried two hours earlier: **serve the exact same XML bytes from a different host.** ```bash # Copy our feed to a temp GitHub repo, serve via raw.githubusercontent.com git init && git add syndicate.xml && git commit -m test gh repo create carryologist/feedtest --public --source=. --push # Try the importer curl -sX POST "https://substack.com/api/v1/import/posts" \ --data '{"url":"https://raw.githubusercontent.com/carryologist/feedtest/master/feed.xml"}' ``` ```json { "import_id": "f0b7b849-ffff-4070-b358-490cb694f38b", "importer_name": "RSSPostImporter", "pub": {"name": "Vibes Coder"}, "num_posts": 13 } ``` **200. 13 posts. Same exact bytes.** The feed was fine all along. Substack's importer specifically refuses to fetch from `vibescoder.dev`. We confirmed this by also proxying through `webhook.site` with the bytes mirrored — same result: works from anywhere except our origin. ## Why Does Substack Reject Our Origin Specifically We never fully solved this. The candidates we ruled out: - Not Cloudflare blocking — Substack's fetcher reaches our origin (we caught their requests via a `webhook.site` honeypot, traced them to AWS EC2 us-east-1, no User-Agent header). - Not the Content-Type — tried `application/xml`, `application/rss+xml`, `text/xml`; same rejection on each. - Not feed size — failed at 9 KB just as much as at 544 KB. - Not feed structure — bytes that work elsewhere fail at our origin. Most plausible theory: **domain reputation**. The `.dev` TLD is relatively new, vibescoder.dev is two months old, and Substack likely has a domain-reputation check baked into their fetcher that silently 400s for unknown domains. Their fetcher running with no User-Agent reinforces this — it looks like a bot that's been hardened against scraping, and bots like that often have allowlists. This is also consistent with Substack's incentive: they want to ingest from known blog platforms, not from arbitrary domains that might be content-spammers. False negatives (rejecting legitimate blogs) are cheaper for them than false positives (importing junk). ## The Workaround That Shipped Skip the fight. Mirror the feed via GitHub. The whole flow: ``` vibescoder.dev/syndicate.xml ↓ (manually re-publish to mirror repo) github.com/carryologist/vibescoder-syndicate/main/syndicate.xml ↓ raw.githubusercontent.com/carryologist/vibescoder-syndicate/main/syndicate.xml ↓ (paste into Substack importer) 13 imported posts on vibescoder.substack.com ``` This is dumb. It's also reliable, free, and took ~5 minutes to set up. For a one-time bulk import, mirroring is the right answer. For ongoing syndication (where you want every new post to flow automatically), a GitHub Action on push that copies `syndicate.xml` from your blog to the mirror repo turns this into a 30-second sync. We haven't built that yet; it's earned its place on the TODO list. ## Failure 3 the First Import Succeeded but Produced Shells Substack accepted the GitHub-hosted URL. 13 posts imported. We celebrated. Then we opened one and saw three sentences of body text. Every imported post contained **only the description field** — no actual article. Mistake on our part: we'd designed `/syndicate.xml` to omit ``, on the theory that Substack would follow the `` and parse the article from the page. **That's not what Substack's importer does.** It reads the body from `` only. If the field is missing, the import is the description — a one-paragraph summary. Fix: put `` back. Same MDX→HTML pipeline we use for the main feed, scoped to the 13 syndicated posts. Total feed size with bodies included: 202 KB. Still under most "real" working feeds, and accepted by the importer when served from the GitHub mirror. ## Failure 4 the Re-Import Dedup Gotcha We deleted the 13 truncated posts from Substack, refreshed the mirror with the body-bearing feed, and re-ran the import. The API returned 200 with `num_posts: 13`. But spot-checking the posts revealed that **two of them were still truncated**. The other 11 had full bodies; two had ~50 words each. The cause is subtle. Substack's importer **deduplicates against publication history, not just live posts**. When you delete a post from your archive, its GUID stays in the importer's memory. Re-importing with the same `` for that URL gets silently skipped, even though the post no longer exists. Of our 13 deletes, 11 cleared the dedup cache (we never figured out exactly why). Two — `closing-the-loop-from-audit-to-ten-commits` and `thursday-thoughts-the-models-we-cant-run` — were "remembered" by the importer and skipped on re-import. The fix is mechanical: change the `` for just those posts. We added a `#reimport-v2` fragment to the GUIDs in the mirror — `` stays the real URL (so canonicals work), `` becomes a value Substack has never seen: ```diff -https://vibescoder.dev/posts/closing-the-loop-from-audit-to-ten-commits +https://vibescoder.dev/posts/closing-the-loop-from-audit-to-ten-commits#reimport-v2 ``` Delete the two posts again, re-run the import, this time Substack treats them as new content. Full bodies. Done. ## Failure 5 the You Can't Index Us Yet Wall 13 posts imported with full bodies. Open one on `vibescoder.substack.com` and the HTML head contains: ```html ``` Substack auto-noindexes any publication that consists entirely of imported posts, until the author has written **at least one original post in the Substack editor**. Their UI literally says so in the JS bundle: > *"This publication is temporarily not available to search engines because the author needs to create a new post other than..."* This is an anti-spam policy — they don't want syndication-farm publications appearing in Google. Reasonable. Fix: write a short native post in the Substack editor. 600 words is plenty. Anything that demonstrates you'll actually compose on the platform, not just pipe imports through. Once that ships, the publication-level `noindex` gets lifted on Substack's next moderation pass (24h-1wk, by reports). ## Failure 6 in Progress Canonical URLs Not User-Settable The whole point of the canonical-URL strategy is to tell Google: "yes, this content also lives at `vibescoder.substack.com/p/...`, but the authoritative version is at `vibescoder.dev/posts/...`. Consolidate signals there." Substack's data model has a `canonical_url` field on each post (we confirmed it in the JS bundle). The HTML template renders it as `` when set. But the editor UI does **not** expose an input control for it on every account. Specifically, our publication's editor SEO panel shows: - SEO title - SEO description - Post URL (slug) - ...and that's it. No Canonical URL field. There may be a publication-level rollout in progress, or it may be account-tier-dependent (paid publications get it first?), or it may just be a rollout that hasn't reached us yet. The available workarounds: 1. **Wait.** Substack rolls out UI changes gradually. The field may appear within weeks. 2. **Email hello@substack.com.** They've manually enabled canonical URLs for users in similar situations. 3. **Accept the noindex.** While the publication remains noindex, the absence of a canonical URL doesn't matter — Google ignores noindexed pages entirely. We chose option 3 for now. If/when Substack lifts the noindex, we'll revisit options 1 and 2. ## What the Final Flow Looks Like Putting it all together for the next time someone needs to do this: 1. **Build a curated feed.** Add a `syndicate: true` frontmatter flag. Add a route (`/syndicate.xml`) that filters to flagged posts and emits Ghost-style RSS with `` containing full HTML bodies. Mirror Overreacted/Ghost format byte-for-byte for safety. 2. **Mirror through GitHub.** Copy `syndicate.xml` to a public repo. Substack's importer accepts the `raw.githubusercontent.com` URL. 3. **Create the Substack publication.** Skip the onboarding import step. 4. **Write one short original post** in the Substack editor. This is the gate that lifts publication-level `noindex`. 5. **Import.** Settings → Import → Import posts → paste the `raw.githubusercontent.com` URL. Confirm. 6. **Spot-check word counts** on a few posts. If any are truncated, delete them, bust their `` values in the mirror (add a `#v2` suffix), and re-import. 7. **Set canonical URLs** on each imported post — *if and only if* the field appears in your editor's SEO panel. Otherwise email Substack support or wait. 8. **Wait 1-7 days** for Substack to lift the publication-level `noindex`. Confirm by checking the meta robots tag on any post. For ongoing syndication (not just one-time import), add a GitHub Action that re-syncs the mirror on every push to your content repo. Then any post with `syndicate: true` in frontmatter flows to Substack automatically. Pair that with a python-substack worker if you want the canonical URL set programmatically going forward. ## What I'd Do Differently Three things, in order of how much time they would have saved: 1. **Start with the API, not the UI.** Hitting `POST /api/v1/import/posts` directly turned a 5-minute-per-attempt UI loop into a 2-second curl loop. Should have done this in the first 15 minutes, not after an hour. 2. **Test against known-working feeds early.** Comparing our feed against Overreacted, Stratechery, etc. via the same API surfaced the size and structural diffs in one experiment. We did this; just two hours later than we should have. 3. **Test the origin in isolation.** The breakthrough was *"is it our content or our domain?"* — answerable in five minutes by serving the same bytes from a temp GitHub repo. We should have run that experiment the moment the structural fixes weren't working. The meta-lesson: **when a black-box system rejects you, the productive direction isn't trying more variations of what you're sending — it's narrowing down what specifically the system objects to.** Every minute spent on the format theory was a minute not spent isolating that the format was fine. ## By the Numbers - **Time spent on initial diagnosis (wrong direction):** ~2 hours - **Time to fix once the actual problem was identified:** ~30 minutes - **Total commits to the engine:** 5 (HTML rendering, Ghost-shape, /import.xml [reverted], /syndicate.xml, content:encoded re-add, content-type fix) - **GitHub mirror repos created:** 2 (one test, one production) - **Substack imports attempted:** 4 (failed, succeeded-but-truncated, partial, complete) - **Posts in the final Substack archive:** 13 (essays, frameworks, Thursday Thoughts, Showdown Thoughts) - **Posts still blog-only:** 21 (build logs, Friday Fixes, homelab posts, infrastructure writeups) - **Substack subscribers:** 0, at time of writing — which is exactly the right number to start with === ## Forking and Open Sourcing a Single Purpose Site - URL: https://vibescoder.dev/posts/forking-and-open-sourcing-a-single-purpose-site - Date: 2026-05-29 - Tags: #agents #vibe-coding #security #next-js #future-of-coding - Reading time: 10 min read I built a trip planning site for my F1 Montreal group. Then I ripped out every hardcoded value, added a setup wizard, ran a security audit, and open-sourced it. Here's what it takes to turn a single-purpose vibe coded app into something anyone can fork and deploy. --- I built a trip planning site for my group going to the F1 Canadian Grand Prix in Montreal. It worked great — itinerary calendar, lodging details, photo gallery, activity suggestions, a shared password so only the group could see it. Classic vibe coded single-purpose app: hardcoded destination, hardcoded dates, hardcoded branding, shipped to Vercel, done. Then I looked at it and thought: this is useful beyond one trip. What if anyone could fork this repo, deploy it, and have their own trip site without touching code? That question kicked off a 20-hour arc — across several mobile sessions between F1 races — that transformed a static, single-purpose site into a generic, config-driven template, and exposed every security shortcut I'd taken along the way. The proof that it worked: I deployed a second instance for a completely different trip — CMA Fest 2026 in Nashville, Tennessee. Same codebase, zero code changes, just the setup wizard. ## The Starting Point The original site had "F1 Grand Prix Montreal" baked into the components. CSS variables were named `--gradient-f1` and `--shadow-f1`. The countdown component had hardcoded race dates. The activities page had Montreal-specific categories. The favicon was F1-themed. `localStorage` keys were F1-prefixed. It was a good app. It was also impossible for anyone else to use without rewriting half the codebase. ## The Architecture Pivot The core insight was simple: **one database row should drive the entire site.** I created a `vacation_config` table with a single JSONB column. Every piece of configurable data — trip name, destination, dates, timezone, brand color, hero image, lodging details, password hash, LLM provider, encrypted API key — lives in that one row. ``` vacation_config ├── tripName ├── destination ├── startDate / endDate ├── brandColor / heroImageUrl ├── lodgings[] ├── passwordHash (bcrypt) ├── llmApiKeyEncrypted (AES-256-GCM) ├── llmProvider └── setupComplete ``` Every page calls `getConfig()` server-side and destructures what it needs. No hardcoded values anywhere. Adding a new configurable field is just adding a key to the TypeScript interface — old configs get new defaults via object spread. This is the pattern that makes fork-and-deploy work. You clone the repo, you get an empty database, and the site is a blank canvas until someone fills in the config. ## The Setup Wizard An empty database isn't useful. Someone needs to fill in that config row, and that someone might not be technical. The setup wizard is a 6-step client component that walks through everything: | Step | What it configures | |------|--------------------| | **Basics** | Trip name, destination, tagline, dates, timezone (auto-detected) | | **Branding** | Brand color (8 presets + custom hex), hero image URL | | **Lodging** | Multiple properties with type-aware display (hotel, Airbnb, VRBO, house, resort) | | **Password** | Shared site password | | **AI Generation** | Optional — pick an LLM provider, paste an API key, auto-generate activity suggestions | | **Review & Launch** | Summary → one-click launch | When you click Launch, four things happen in sequence: config is saved (password bcrypt-hashed, API key AES-encrypted), database tables are created, the user is auto-authenticated, and they're redirected to the live homepage. The entire setup takes about two minutes. ## The Middleware Problem A static site deployed to your own Vercel project doesn't need sophisticated auth. You share the URL with your group, maybe add a simple password check, and you're done. A clonable template is different. Every fork is a fresh deployment. The middleware needs to handle two states: **not yet set up** and **set up and running**. I built a two-gate system running in Edge Runtime: **Gate 1 — Setup Check.** Is there an HMAC-signed `setup-done` cookie? If not, redirect to `/setup`. This cookie is signed with the site secret to prevent client forgery. **Gate 2 — Auth Check.** Is there a valid auth token cookie? The token includes a timestamp and a random nonce, HMAC-signed with the site secret. If it's missing, expired, or invalid, redirect to `/password`. The edge constraint matters. Next.js middleware runs in Edge Runtime, which means no Node.js `crypto` module. The entire auth chain — HMAC signing, signature verification, timing-safe comparison — uses the Web Crypto API. The Node.js side (`lib/auth.ts`) handles bcrypt password hashing and AES encryption, which only run in API routes. ## From One Secret to Everything The user provides exactly one secret: a random hex string generated with `openssl rand -hex 32`. That single value does triple duty: - **HMAC signing** — auth tokens and setup cookies - **AES-256 encryption key** — derived via SHA-256 hash for encrypting LLM API keys at rest - **Timing-safe comparison** — double-HMAC pattern for constant-time signature verification Everything else is either auto-provisioned (Vercel Postgres sets `POSTGRES_URL`, Vercel Blob sets `BLOB_READ_WRITE_TOKEN`) or entered through the wizard. The user never edits code, never touches a config file, never opens a terminal after the initial deploy. ## The Security Audit This is where the story arc connects to lessons I've written about before. I've been saying [audit your vibe code often](/posts/thursday-thoughts-audit-your-vibe-code-often). I've written about the [spring cleaning process](/posts/spring-cleaning-your-vibe-coded-apps) and the [phased remediation pattern](/posts/closing-the-loop-from-audit-to-ten-commits). So when I decided to open-source this project, I ran a full audit before publishing. The audit found **15+ vulnerabilities across 4 severity tiers.** I expected minor stuff. I got critical findings. ### The Critical Tier The worst findings were structural. The middleware had a blanket pass-through for all `/api/*` routes — meaning API endpoints were completely unauthenticated. The setup config endpoint had no auth, so anyone who found the URL could overwrite or delete the entire site configuration. Auth tokens had no expiration. And there was a hardcoded fallback secret — `'fallback'` — that would activate if the environment variable was missing, making every signature predictable. These aren't exotic bugs. They're the exact patterns that vibe coding produces: things that work during development and deployment but leave doors wide open. ### The High Tier The OG image endpoint accepted arbitrary URLs with no validation — a textbook SSRF vector that could reach private networks. LLM prompts passed unsanitized user input directly to the model — destination names, PDF document text, all of it unescaped. No data validation existed on any write endpoint. And the password endpoint had no rate limiting — unlimited brute-force attempts. ### The Medium and Low Tiers Signature comparison used string equality instead of timing-safe comparison. The setup cookie was unsigned. Error responses leaked internal details. No security headers. No file size limits on uploads. The Gemini API key was sent as a URL query parameter (logged in server access logs). The middleware's static asset detection used `pathname.includes('.')` — meaning a crafted path like `/settings/foo.bar` would bypass auth. ### The Fix I structured the remediation the same way I've done it before: phased commits ordered by severity and dependency graph, not one giant PR. **Commit 1 — Critical fixes.** Middleware now enforces auth on all API routes except the auth endpoint itself and public config reads. Setup mutation requires authentication after initial setup. Auth tokens expire after 30 days. The hardcoded fallback secret is gone — a missing env var now returns a 500. **Commit 2 — High fixes.** SSRF blocked with private IP detection. LLM inputs sanitized with delimiter-based injection mitigation and output validation. Per-entity input validators on all write routes. Rate limiting on the auth endpoint with IP-based lockout. **Commit 3 — Medium and low fixes.** Setup cookie is HMAC-signed. PDF uploads enforce a size limit. Security headers added (CSP, HSTS, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy). Gemini key moved from URL to header. Static asset detection uses an explicit extension regex. Client-side error logging sanitized. CSS color injection blocked with a validation function. Three commits. The same phased pattern. Same principle: merge and test between each phase so you know exactly which change breaks something if it does. ## What Changes When You Open Source Going from "deployed for my group" to "anyone can fork this" changed the threat model fundamentally. **Before:** I controlled the deployment. I knew the URL. The password was shared via text message. If something was misconfigured, I'd notice and fix it. **After:** Strangers deploy this. They might skip the secret. They might leave the setup endpoint open. They might paste API keys into client-side code. Every defensive measure needs to work without my involvement. This is why the audit mattered more for open-sourcing than for personal use. A personal deployment with no auth on API routes is sloppy. An open-source template with no auth on API routes is a liability for every person who forks it. The middleware's two-gate system, the HMAC-signed cookies, the secret-or-500 pattern, the input validation — none of these existed in the original F1 trip site. They exist because the code is no longer mine alone. ## Making It Novice-Friendly The target user is someone who's never used a terminal. That constraint shaped the documentation as much as the code. The [setup guide](https://github.com/carryologist/vacation-hub/blob/main/docs/SETUP_GUIDE.md) walks through 8 steps: fork the repo, generate a secret key (with instructions for Mac, Windows, and a web fallback), deploy to Vercel, add Postgres, add Blob storage, redeploy, run the wizard, share with your group. Each step assumes zero technical knowledge. The README has a one-click Deploy with Vercel button that pre-fills the environment variable prompt. The wizard auto-detects timezone from the browser. Lodging details auto-populate from the property name via AI. The color picker has presets so nobody has to know what a hex code is. Every friction point I could identify, I tried to eliminate. The person deploying this might be planning a bachelorette party or a family reunion. They're not reading documentation for fun. ## The Architecture Lessons Turning a personal app into a template taught me things that pure greenfield development wouldn't have: **Config-driven beats hardcoded, always.** Even if you're building for one use case, storing configuration in a database instead of in component props makes the app fundamentally more flexible. The JSONB column costs nothing and buys everything. **Middleware is the security boundary.** In a personal app, auth is a convenience — you know who's accessing it. In a template, middleware is the only thing standing between a stranger's deployment and the open internet. It needs to handle every state: not yet configured, configured but not logged in, logged in, logged in with an expired token. **The setup wizard is the product.** For a clonable template, the first-run experience *is* the product. If someone can't get from fork to functioning site in 10 minutes, they'll abandon it. The wizard isn't a nice-to-have — it's the reason the project works. **Security scales with distribution.** A bug in your personal app affects you. A bug in a template affects everyone who forks it. The bar for security isn't "good enough for me" — it's "good enough for the least technical person who deploys this." ## By the Numbers - **28 commits** — from hardcoded F1 site to open-source template - **1 JSONB row** — drives the entire site configuration - **6-step wizard** — zero-code setup for non-technical users - **15+ security vulnerabilities** — found and fixed before open-sourcing - **3 phased commits** — for the security remediation alone - **1 env var** — the only thing a user manually configures (`VACATION_HUB_SECRET`) - **~20 hours** — total transformation time - **0 lines of code** — required from the person deploying it === ## Adding an MCP Server to the Blog Itself - URL: https://vibescoder.dev/posts/adding-mcp-server-to-the-blog - Date: 2026-05-28 - Tags: #mcp #agents #building-in-public #next-js - Reading time: 8 min read The fitness tracker MCP server was a test run. This week I added the same thing to vibescoder.dev — 16 tools that let any agent list posts, publish drafts, check analytics, trigger deploys, cross-post to Dev.to, and troubleshoot the live site. Here's the build, the architectural decisions, and what it's like when the agent that built the feature can immediately use it. --- Two weeks ago I [wired MCP into my fitness tracker](/posts/wiring-mcp-into-my-fitness-tracker-for-openclaw) — ten tools, one endpoint, four clients. That was always a test run. The fitness tracker is a low-stakes app. If an agent writes a bad workout entry, I delete it. The blog is different. The blog has published content, a deploy pipeline, an editorial calendar, analytics, syndication to Dev.to. If an agent publishes a draft that wasn't ready, the internet sees it. This week I added an MCP server to vibescoder.dev anyway. Sixteen tools across five categories. The agent that helped me build it — running in a Coder workspace — can now turn around and use it to manage the very site it just modified. That's the kind of loop that makes building in public feel recursive. ## The Goal One sentence: **let any agent directly publish to the site, analyze traffic data, and troubleshoot production issues.** The blog is a Next.js 16 app deployed on Vercel. Content lives in a separate private GitHub repo (`the-vibe-coder-content`), committed via the GitHub API. The admin UI already supports voice recording → Claude-generated MDX → one-click publish. But the admin UI requires a browser. An agent in a Coder workspace, or in Claude Desktop, or in Cursor can't click buttons. MCP gives them the same capabilities programmatically. ## Architecture The fitness tracker MCP server talked to Postgres via Prisma. This blog has no database. Content is MDX files in a GitHub repo. Analytics are Redis counters in Upstash. Deployments happen by curling a Vercel webhook. So the MCP server is a GitHub API client, a Redis reader, and an HTTP caller — not a database wrapper. ``` Agent (Claude / Cursor / Coder Agents) │ │ Streamable HTTP (Bearer token) ▼ vibescoder.dev/api/mcp/mcp │ ├─ Content tools ──→ GitHub API (read/write/commit MDX) ├─ Analytics ──────→ Upstash Redis (view counters) ├─ Deploy ─────────→ Vercel deploy hook ├─ Syndication ────→ Dev.to API └─ Diagnostics ────→ fetch() against live site ``` Same stack as the fitness tracker: `mcp-handler` for the Next.js adapter, `zod` for parameter schemas, bearer token auth, `disableSse: true` for stateless Vercel deployment. ## The 16 Tools The fitness tracker had 10 tools that all talked to one database. This server has 16 tools that talk to four different backends. Grouped by what they touch: **Content Management** (7 tools) — the core editorial workflow: ```ts server.tool('list_posts', /* filter by status/tag/date */) server.tool('get_post', /* full MDX + frontmatter */) server.tool('create_post', /* commit new MDX to GitHub */) server.tool('update_post', /* partial frontmatter/body */) server.tool('publish_post', /* draft → live, trigger deploy */) server.tool('unpublish_post', /* live → draft, trigger deploy */) server.tool('delete_post', /* remove from GitHub */) ``` **Blog Fodder & Editorial** (4 tools) — the content pipeline: ```ts server.tool('list_fodder', /* active + archived, with consumption status */) server.tool('get_fodder', /* read raw session notes */) server.tool('get_todo', /* editorial calendar */) server.tool('update_todo', /* maintain the calendar */) ``` **Analytics** (1 tool), **Deploy & Syndication** (2 tools), **Diagnostics** (2 tools): ```ts server.tool('analytics_summary', /* 30-day views + top pages */) server.tool('trigger_deploy', /* hit the Vercel webhook */) server.tool('syndicate_post', /* cross-post to Dev.to */) server.tool('site_health', /* fetch key endpoints */) server.tool('get_settings', /* AI style prompt config */) ``` Every tool returns raw data. The agent does its own analysis — same philosophy as the fitness tracker. The `list_posts` tool returns frontmatter for every post; the agent decides what "recent drafts" means. ## What I Reused The blog engine already had all the backend logic. The admin UI's API routes do the exact same operations — read a post from GitHub, commit an update, hit the deploy hook, cross-post to Dev.to. The MCP server calls the same library functions, not the HTTP routes: ```ts import { commitFile, readFile, deleteFile } from "@/lib/github"; import { listDirectory } from "@/lib/github-list"; ``` The only net-new code was the directory listing helper (`github-list.ts`). The existing `github.ts` had file-level CRUD but couldn't list a directory. One function, 30 lines, wraps the GitHub Contents API for directory paths. The auth pattern, CORS, and rate limiting were copied from the fitness tracker and adapted. Same `timingSafeEqual`, same `withMcpAuth` wrapper, same in-memory rate-limit buckets. The muscle memory from the fitness tracker build meant the security layer took minutes, not an hour. ## The Middleware Change One line. The blog's middleware protects all `/api/*` routes with JWT cookie auth. The MCP server does its own bearer-token auth. So `/api/mcp/` gets added to the allow-list alongside `/api/auth/`, `/api/analytics/track`, and `/api/slack/`: ```ts pathname.startsWith("/api/mcp/") ``` The MCP route then handles auth independently — same pattern as the fitness tracker, where the middleware allow-listed the MCP path and the route enforced its own bearer token. ## Decisions Three questions came up during planning: **Auth granularity** — single token or read-only vs. read-write tokens? Single token. I'm the only user. If I ever add collaborators, I'll add scoped tokens. Until then, one token does everything. **Audit logging** — the fitness tracker writes to a Postgres `audit_log` table. This blog has no database. Options were Redis, console.log, or skip. I went with console.log (captured by Vercel function logs) plus `[mcp]` prefixed commit messages for every GitHub write. That gives me two audit trails — Vercel logs for all operations, Git history for content changes — with zero infrastructure. ``` [mcp] post: create "adding-mcp-server-to-the-blog" [mcp] post: publish "adding-mcp-server-to-the-blog" [mcp] chore: update TODO.md ``` **Image uploads** — deferred. MCP tool parameters are JSON. Binary images would need base64 encoding in a tool call. That's doable but not worth the complexity in v1. The admin UI handles images fine. If an agent needs to add images to a post, it can use the admin API directly or I'll add an `upload_image` tool later. ## The Template Update Same Coder template pattern as the fitness tracker. Token flows from the workstation to workspaces: ``` /etc/coder.d/coder.env → TF_VAR_vibescoder_mcp_token → coder_agent.main.env (VIBESCODER_MCP_TOKEN) → jq merge into ~/.mcp.json at workspace start ``` Three terminal commands on the homelab to finish it: ```bash echo 'TF_VAR_vibescoder_mcp_token=' | sudo tee -a /etc/coder.d/coder.env sudo systemctl restart coder cd ~/coder-templates && git pull && ./docker/apply.sh ``` The `gh auth login` step was an amusing detour — I was SSH'd into the homelab from my iPhone, and `gh` tried to open a browser on a headless server. The fix was manually entering the one-time code at `github.com/login/device` in Safari. Mobile homelab administration is an underappreciated genre of suffering. ## Verifying in Production The real test was hitting the live endpoint: ```bash curl -s -X POST https://vibescoder.dev/api/mcp/mcp \ -H "Authorization: Bearer $VIBESCODER_MCP_TOKEN" \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize", "params":{"protocolVersion":"2025-03-26", "capabilities":{}, "clientInfo":{"name":"test","version":"1.0.0"}}}' ``` Response: `200 OK`, server name `vibescoder`, version `1.0.0`, tools capability enabled. Then a real tool call — list all drafts: ```json { "count": 1, "posts": [{ "slug": "syndicating-to-substack-the-undocumented-path", "title": "Syndicating to Substack: The Undocumented Path", "published": false, "publishAt": null }] } ``` One draft in the queue. Real data from the content repo, returned through the MCP server, verified from a Coder workspace. The analytics tool came back with 660 views over 30 days and today's top pages. The site health tool checked five endpoints and reported status codes and response times. ## The Recursive Moment The part that's hard to describe until you experience it: the agent that helped build this MCP server can now use it. In the same chat session where we wrote the route file and debugged the middleware, the agent can call `list_posts` to see what's published, `get_todo` to check the editorial calendar, and `trigger_deploy` to ship changes. This post was written in a Coder workspace. The MCP server it describes is live on the same site it will be published to. The agent could, in theory, publish this very post by calling `publish_post` with the slug. It won't — I'll review it first — but the capability is there. That's the loop. ## What's Next 1. **Watch how agents use the tools in practice.** The fitness tracker MCP server taught me that agents are surprisingly good at synthesizing raw data into summaries. Curious whether editorial tools — create, publish, schedule — feel as natural. 2. **Add an `upload_image` tool.** Deferred from v1, but it's the obvious gap. An agent that can create a post but not attach images is writing with one hand. 3. **Update the vibescoder-blog skill file.** The skill currently documents the Git-based editorial workflow. Now that the MCP server exists, the skill should point agents to the tools instead of the `grep` and `awk` one-liners. 4. **Write it up as blog fodder.** Done. You're reading it. ## By the Numbers - **16 MCP tools** across 5 categories - **4 backends** wired through one endpoint (GitHub API, Upstash Redis, Vercel deploy hook, Dev.to API) - **7 files changed** in the engine repo, 2,365 lines inserted - **1 file changed** in the Coder template repo, 23 lines inserted - **3 npm packages** added (`mcp-handler`, `@modelcontextprotocol/sdk`, `zod`) - **1 middleware line** to allow-list `/api/mcp/` - **0 new infrastructure** — no database, no Redis, no queues. GitHub API + console.log - **3 terminal commands** to update the homelab Coder config - **1 iPhone-to-homelab SSH detour** for `gh auth login` via Safari - **660 views** over 30 days — the first number the analytics tool reported back - **1 draft** in the queue when `list_posts` was first tested (still sitting there, Substack) - **~4 hours** from plan to production, including the template update and blog post - **1 recursive loop** — the agent that built the feature can now use it to publish this post === ## QoL with WoL: Turning on the Homelab from Anywhere in the World - URL: https://vibescoder.dev/posts/qol-with-wol-turning-on-the-homelab-from-anywhere - Date: 2026-05-27 - Tags: #homelab - Reading time: 8 min read A full walkthrough of setting up Wake on LAN on a Linux homelab and wiring it into Google Home via SmartThings — including every dead end, expired link, and wrong interface name along the way. --- # QoL with WoL: Turning on the Homelab from Anywhere in the World I just hardwired my homelab PC and wanted a quality of life upgrade: my wife and I should be able to wake it remotely from anywhere — using Google Home, because that's what we already use. Simple goal. Turned out to be a longer road than I expected, mostly because the internet is full of outdated instructions, expired invite links, and tutorials that assume you have the right NIC name. Here's the full journey, stumbles included. --- ## The Goal Wake a Linux homelab PC from fully powered off, from anywhere in the world, using Google Home voice commands. No subscriptions. No new hardware. **Constraint:** No Nabu Casa ($7/mo Home Assistant cloud relay), no IFTTT, no Raspberry Pi. --- ## How It Actually Works Google Home can't send a Wake on LAN magic packet directly. WoL requires a device on the same local network as the target machine to broadcast the packet. So the architecture is: ``` Google Home voice command → SmartThings → SmartThings Hub V2 (always on, always on LAN) → Magic packet broadcast → Homelab PC wakes up ``` The SmartThings hub is the relay — it's plugged in 24/7, sits on your LAN, and can broadcast the magic packet even when the PC is completely off. No cloud service needed beyond SmartThings itself, which you likely already have. --- ## Step 1 Enable Wake on LAN on the Linux Machine ### Find Your Actual NIC Name The first stumble: running `sudo ethtool eth0` and getting a wall of errors. ![ip link show output](/images/wake-on-lan-google-home/ip-link-show-output.png) Modern Linux doesn't use `eth0` anymore. Network interfaces use predictable names like `enp8s0`. Find yours: ```bash ip link show | grep -E "^[0-9]+:" | grep -v "lo\|docker\|veth\|br-" ``` Look for an interface starting with `en` that shows `state UP` — that's your hardwired NIC. In my case: `enp8s0`. ### Check and Enable WoL ```bash sudo ethtool enp8s0 | grep -i wake ``` You want to see `Supports Wake-on: pumbg` — the `g` means magic packet is supported. If `Wake-on: d`, it's disabled. Enable it: ```bash sudo ethtool -s enp8s0 wol g ``` Verify it stuck: ```bash sudo ethtool enp8s0 | grep -i wake # Wake-on: g ← good ``` ![netlink errors from wrong interface name](/images/wake-on-lan-google-home/netlink-error-wrong-interface.png) ### Make It Persist Across Reboots The setting resets on reboot without a systemd service. Create `/etc/systemd/system/wol-enable.service`: ```ini [Unit] Description=Enable Wake on LAN After=network.target [Service] Type=oneshot ExecStart=/sbin/ethtool -s enp8s0 wol g RemainAfterExit=yes [Install] WantedBy=multi-user.target ``` ```bash sudo systemctl enable wol-enable.service sudo systemctl start wol-enable.service ``` ### Note Your Mac Address You'll need this later: ```bash ip link show enp8s0 | grep link/ether ``` ### Check Your BIOS This is the step people skip and then wonder why nothing works. Reboot into BIOS/UEFI and look for: - **Wake on LAN** → Enabled - **Power On By PCI-E** → Enabled - **ErP / EuP** → Disabled (this cuts power to the NIC when off, killing WoL) Location varies by motherboard — search your model name + "wake on lan bios" if you can't find it. --- ## Step 2 Set up Home Assistant Turned Out to Be a Detour My original plan was Home Assistant → Nabu Casa → Google Home. I got HA running in Docker: ```bash docker run -d \ --name homeassistant \ --network host \ --restart unless-stopped \ -v ~/homeassistant-config:/config \ -e TZ=America/Chicago \ ghcr.io/home-assistant/home-assistant:stable ``` ![Home Assistant welcome screen](/images/wake-on-lan-google-home/ha-welcome-screen.png) Onboarding was smooth. HA auto-discovered a bunch of devices on the network — Google Cast, Sonos, Android TV, Elgato lights, Matter, Thread, UPnP. ![HA compatible devices discovered](/images/wake-on-lan-google-home/ha-compatible-devices.png) After setup, the dashboard came up clean: ![Home Assistant overview dashboard](/images/wake-on-lan-google-home/ha-dashboard-overview.png) Added the Wake on LAN integration (**Settings → Devices & Services → Add Integration → Wake on LAN**), and HA immediately auto-created a device from the MAC address — "Homelab Ethernet" in the Office area. ![HA Wake on LAN integration with Homelab Ethernet device](/images/wake-on-lan-google-home/ha-wake-on-lan-integration.png) ![HA Homelab Ethernet device detail](/images/wake-on-lan-google-home/ha-homelab-ethernet-device.png) ### The Nabu Casa Problem Then I hit the constraint: connecting HA to Google Home natively requires Nabu Casa ($7/mo) or a self-hosted OAuth2 Google Cloud project. I wanted no subscriptions. The self-hosted path is doable but needs a publicly reachable URL for HA, which means a Cloudflare Tunnel, a domain, and about 30 minutes of Google Cloud Console work. I decided to explore whether my existing SmartThings hub could do the job instead. Turned out: yes, much cleaner. --- ## Step 3 Smartthings Hub as the WoL Relay I have a SmartThings Hub V2 that's been sitting plugged in for years. It runs Edge drivers locally, is always on, and already syncs with Google Home. ### Finding a Working Edge Driver the Hard Part SmartThings Edge drivers are installed via channel invite links. The TAustin Wake on LAN driver is what most tutorials point to — but every link I tried was either expired or the community forum thread was gone. First attempt: Forbidden. Second attempt: Community forum thread 404'd. The fix: TAustin's real GitHub username is `toddaustin07`. His **PC Control** driver README contains the current working channel invite link. The channel is called "TAustin Driver Tests." **Current working channel invite:** ``` https://bestow-regional.api.smartthings.com/invite/Q1jP7BqnNNlL ``` Open that link on your phone while logged into SmartThings. You'll see the channel info and an Enroll button for your hub. ### Which Driver to Install Once enrolled, the channel has two relevant drivers: | Driver | What it does | |---|---| | **PC Control V1.1** | WoL wake + HTTP shutdown commands. Shutdown requires a Windows agent on the target PC. | | **Virtual WOL Switch V1** | Pure WoL — just sends the magic packet. Works on any OS. | Install **Virtual WOL Switch V1**. The PC Control driver is designed for Windows shutdown integration and adds unnecessary complexity for a Linux machine. ### Create the Device In the SmartThings app: **+ → Add device → Scan for nearby devices** The hub creates a `vWOL_1 1` device automatically. Open the device, tap **⋮ → Settings**, and enter: - **MAC address:** your machine's MAC (format: `xx:xx:xx:xx:xx:xx`) - **Broadcast address:** `255.255.255.255:7` (default, leave it) The `:7` is port 7 — one of the two standard WoL ports (the other is 9). Both work fine. ### Test It Shut down the homelab PC fully. Open the vWOL device in SmartThings and tap the power button. The PC should boot. It worked — and because Coder starts automatically on boot, the workspace came back up on its own too. --- ## Step 4 Connect Smartthings to Google Home If SmartThings isn't already linked to Google Home: 1. Open the **Google Home app** 2. Tap **+ → Set up device → Works with Google** 3. Search for **SmartThings** 4. Sign in with your Samsung account Your `vWOL_1 1` device will import. Rename it in Google Home to something natural — "Homelab" or "My PC" — so the voice command is clean: > "Hey Google, turn on Homelab" --- ## The Full Chain ``` "Hey Google, turn on Homelab" → Google Home → SmartThings cloud → SmartThings Hub V2 (local, always on) → UDP broadcast magic packet on LAN → enp8s0 wakes on magic packet → Linux boots → Coder starts automatically ``` No subscriptions. No relay server. No Raspberry Pi. The SmartThings hub you already have is the relay — it never sleeps and it's always on your LAN. --- ## Things That Tripped Me Up - **`eth0` doesn't exist** on modern Linux. Always check your actual interface name with `ip link show` first. - **TAustin's community forum thread is gone.** The working channel link is in the README of his [PCControl GitHub repo](https://github.com/toddaustin07/PCControl), not the forum. - **Channel invite links expire.** If the link above is dead, check that README directly for the current one. - **BIOS ErP setting kills WoL.** If your PC won't wake despite everything being configured correctly, this is the first thing to check. - **Home Assistant WoL requires an always-on relay.** If HA itself is on the machine you're trying to wake, it obviously can't send the magic packet when it's off. The SmartThings hub solves this elegantly. --- ## By the Numbers - **Time to enable WoL on Linux:** ~10 minutes - **Time lost to expired SmartThings invite links:** ~20 minutes - **Dead community forum threads encountered:** 2 - **Working channel links found via GitHub:** 1 - **Subscriptions required:** 0 - **New hardware purchased:** 0 - **Voice commands to wake the homelab:** 1 === ## Qwen Is Not Yet Ready to Power Local OpenClaw Deployments - URL: https://vibescoder.dev/posts/qwen-is-not-yet-ready-to-power-local-openclaw-deployments - Date: 2026-05-26 - Tags: #homelab #agents #openclaw - Reading time: 9 min read Two weeks of using Qwen3.5-35B as my daily AI assistant — the Jinja template fix that made it work, the thermal spam incident that almost ended the experiment, and the session-context gap that makes it feel like a junior dev every morning. Plus: what's next with Qwen 3.6. --- Three weeks ago I ran a model showdown — twelve tasks, five models, one RTX 5090 — and Qwen3.5-35B-A3B won. 85.3 weighted score, 206 tok/s, fits in VRAM with room to spare. I switched it to the default and figured I was done. I was not done. This is what two weeks of actually living with Qwen looked like: the config work I had to do before it was usable, the incident that almost killed the experiment, and the ergonomic gap that means frontier models still own my serious work. ## Making It Actually Work The first day I switched Qwen to the default model in OpenClaw, something was wrong. Responses showed raw `...` tags in the visible output. Tool calls came back as plain text — `create_workspace`, just sitting there — instead of proper OpenAI-compatible `tool_calls` objects. The bot was trying to call tools. It just wasn't *calling* them. The root cause was a one-line config error. The launch script was using `--chat-template chatml` — a minimal template that knows nothing about tool calling and doesn't know to hide thinking tokens. Qwen3.5 ships with a 154-line Jinja template that handles both. I just wasn't using it. The catch: Qwen's native template has a strict ordering check that raises an exception if a system message appears anywhere other than the very beginning of the conversation. Coder Agents sends system messages out of order. So I patched one conditional in the template — non-first system messages render as normal blocks instead of throwing — and switched to `--chat-template-file` pointing at the patched version. After the restart: `thinking = 1` in the journalctl output. Tool calls worked. The visible output was clean. The fix was one line. It took half a day to find. That's a recurring pattern with local model work. The model is fine. The scaffolding is fragile. ## Day One Gotcha Cloning from a Stranger With the template fixed, I asked Qwen to clone the vibe coder repos. It searched GitHub for a literal `vibe-coder` user, found a random stranger's account, and dutifully cloned 25 repos from them. `reset-css`, `moviebox-main`, `orange-farm`. None of them mine. Not a Qwen failure, exactly. A context failure. The agent had no skill file telling it that `carryologist` is the GitHub org. Once I pointed it at the skills directory it read the file, correctly identified the repos, and did the job. I fixed this by making skill loading unconditional. The user instruction used to say "when I mention the blog, read the vibescoder-blog skill." Changed it to "at the start of every conversation, read all available skills." Generic enough for every user, scoped by which skills the workspace template actually provisions. I also added a fodder dedup check to the vibescoder-blog skill — Qwen had recommended writing a post from a fodder file that already had a draft, because it never checked `sources:` fields in existing posts. Small gap, easy to close once you see it. The pattern: Qwen is good at following instructions. It is not good at inferring what instructions it needs to follow before it has them. ## The Thermal Flood May 9. 4:34 PM. The OpenClaw cron had been running for a few days. I'd named the job "Hardware Alert Checker (Critical Only)." On May 9 it posted a thermal report to the `#homelab-alerts` Discord channel at 4:34 PM. Then again at 4:47. Then 5:07. For the next two days, every fifteen minutes — day and night — a full hardware report appeared in my channel. The cron log eventually showed 384 entries. I counted over 60 posts before I said anything. The job was named "Critical Only." It was not configured for "Critical Only." I had set it up to check thermals and post a report. It did exactly that. The bot did precisely what it was set up to do and nothing like what it was named to do. On May 11 I finally messaged carrybot directly: "Can we stop regular alerting and only let me know when temps go critical or if I specifically ask?" The bot replied: "Already done — that hardware monitoring job is set to 'Critical Only' and runs every 15 minutes. It'll only ping you if temps hit dangerous levels." I sent a screenshot of the flood. The bot checked the cron history, confirmed it was wrong, and disabled the job entirely. No config fix. No threshold update. Just gone. Manual checks only from that point forward. What it cost: I didn't open OpenClaw again until May 15. Three and a half days. That's a long silence for a tool you're evaluating as a daily driver. Friction compounds. One bad incident isn't fatal, but 60+ notifications across two days is loud enough that I actively avoided the interface rather than dealing with it. The bot won't get better if you stop using it. ## MCP Wiring the Win May 15 went better. I wired the fitness tracker MCP into OpenClaw — I wrote that up in [Wiring MCP Into My Fitness Tracker](/posts/wiring-mcp-into-my-fitness-tracker-for-openclaw), but the short version is: two minutes, real data. First query returned my last Peloton ride. 30-minute Power Zone Pop Ride, Ben Alldis, 7.98 miles. The bot pulled it without hesitation. There was a ghost cron alert that evening — the bot flagged a cron job that didn't appear in my active list. Qwen explained the discrepancy clearly (the job exists in state but isn't scheduled). Good recovery after the thermal flood. ## The Session That Revealed the Real Problem May 16. I sent a voice message asking about my workout stats. No Whisper on the local install, so the bot had no idea what I said. Fine — I typed instead. "What are my stats for my ride today?" The bot went to Uber. Ride → Uber. It didn't know I meant Peloton. I clarified: fitness tracker MCP. The bot responded that the MCP server wasn't actively connected. I asked it to check the tool list. Confirmed: fitness-tracker was there. Third prompt, correct answer. Three extra turns to get what should have been a one-shot query. On a frontier model that would have resolved on the first prompt — it would have understood that "ride stats" meant the fitness tracker I'd been talking about the session before. On Qwen, I start every session from scratch. It has no memory of what MCP servers we were using yesterday. It has no context for what "ride" means to me. The bot diagnosed this correctly when I asked. It said: I need a TOOLS.md or explicit mentions at session start; I can't infer that fitness = Peloton MCP from prior conversations. It offered to update the TOOLS.md. It did. That's the right response. But it required me to catch the gap and prompt the fix. A more polished agent would have persisted that context automatically. It would have — except I checked the config later and `memory-core` is disabled in `openclaw.json`. There's a memory plugin; it's just off by default. Every session starting cold wasn't an emergent limitation of local models. It was a config flag I hadn't toggled. ## The Verdict Local Agents Can't Match Frontier Practicality... Yet After two weeks: hobbyist-level technology. Great for enthusiasts. Not ready for prime-time agentic work. The model is solid. 206 tok/s is genuinely fast. The Jinja template, once fixed, works. When the context is right, the answers are good. But the ergonomics aren't there yet. Every session starts cold. MCP connections need re-establishing. The bot does what it's configured to do, not what you intend, and there's enough configuration surface area that intent and config drift apart. A frontier-model-backed agent handles these gaps with implicit context and better defaults. Qwen handles them if you set things up correctly and remind it what's relevant at the start of every conversation. That's a meaningful gap. Two weeks in, Qwen never became my default interface. I reach for it when I want to run something local, or when I'm testing the setup. I reach for a frontier model when I want the thing to just work. That's an honest result. Qwen is the right default for a privacy-first local-first homelab setup. For production agentic work, the frontier models are still ahead on ergonomics — and ergonomics compound across every session. ## What's Next Upgrading to Qwen 3.6 While I was writing this, Qwen released 3.6 (April 24, 2026). Two variants relevant to my setup: **Qwen3.6-35B-A3B** (MoE) — same VRAM footprint as the current model. Modest coding improvement over 3.5, adds a `preserve_thinking` kwarg to the chat template. Drop-in upgrade. **Qwen3.6-27B** (dense) — outperforms the 35B MoE on coding benchmarks. SWE-bench 77.2 vs 73.4. The tradeoff is throughput — dense models are slower per token, and the 3.5 MoE's 206 tok/s speed is one of its best features for agentic work where you're waiting on tool call chains. A few things to know before upgrading: - llama.cpp b9180+ required for MTP speculative decoding support - `--jinja` flag needed for the `enable_thinking`/`preserve_thinking` kwargs - **Do not use `-sm tensor`** — there's an open segfault bug (#23297) - MTP flags: `--spec-type draft-mtp --spec-draft-n-max 3` I'm going to try the 35B-A3B MoE first. Same slot, same startup flags (minus the segfault one), meaningful upgrade on coding. The dense 27B is tempting on benchmarks but I'll wait to see how throughput holds up under real agentic load before committing. The bigger question I'm watching isn't the benchmark numbers — it's whether the next generation of local models closes the context and tool call chaining gap. Once a local model can reliably remember what MCP servers you were using yesterday, infer intent across sessions, and chain tool calls without hand-holding, the ergonomics argument for frontier models gets a lot weaker. We're not there yet. I'll be paying attention. ## By the Numbers - **652 session files**, May 8–16 — the vast majority are cron-fired Discord sessions, not direct interactions - **~10 human-initiated sessions** across the two weeks; the rest are the alert checker running every 15 minutes - **7 context resets** — sessions where the conversation was cleared and started fresh - **Thermal flood**: cron job `d8da7ec1` created May 9 4:31 PM PT, **384 logged runs**, disabled May 11 9:10 PM PT — ~52 hours of every-15-minute posts - **Token/cost data**: all null — llama.cpp doesn't return usage in the API response - **Tool calls**: 0 structured `tool_use` objects in session logs — llama.cpp doesn't emit them. The 40 hits on fitness tracker keywords are conversation text mentions, not actual invocations. - **Memory core**: disabled in `openclaw.json` — explains why every session starts cold === ## The Audit That Found The Thing The Audit Didn't Find - URL: https://vibescoder.dev/posts/the-audit-that-found-the-thing-the-audit-didnt-find - Date: 2026-05-25 - Tags: #security #mcp #building-in-public #agents #next-js - Reading time: 16 min read I asked an agent to security-audit my fitness tracker after wiring MCP into it. It found nineteen things. I fixed them all in four neat batches. Then the dashboard went empty, Google sign-in died, and the real bugs turned out to be the ones the audit couldn't see — a middleware file that had been silently doing nothing for months, and an OAuth client that never existed in any project I owned. --- I added an [MCP server to my fitness tracker](/posts/wiring-mcp-into-my-fitness-tracker-for-openclaw) last week. That meant a new authentication surface — a bearer token that unlocks the API for agents — sitting alongside the existing Google sign-in. New attack surface, new opinions about token handling, new ways to get it wrong. So I asked an agent to audit the repo. Four hours later I had nineteen findings, all fixed, all committed, all pushed to main. I felt great. Then I opened the app and the dashboard was empty. This is a post about what that audit found, what I shipped, and the much more interesting things the audit *didn't* find — the ones that only surfaced because I fixed the things it did. --- ## The Audit I sent the agent into the repo with one instruction: do a security review, focused on the MCP addition but covering everything that surface touches. It came back with a 26 KB markdown report. Severity-graded, file-and-line references, recommendations, the works. The headline finding was a real one. The MCP commit had added a bearer token (`MCP_API_TOKEN`) that the middleware accepted on every `/api/*` route as a session substitute. The token itself was implemented correctly — constant-time compare with length pre-check, never logged, only accepted via the `Authorization` header. But the route handlers were all using a `checkAuth()` helper that was non-blocking by design. A leftover from working around a NextAuth v5 beta quirk: `auth()` returns `null` in Route Handlers on serverless platforms even when the user has a valid session. The original author papered over it by logging the null and proceeding anyway, trusting the middleware to be the real gate. That's a perfectly defensible decision if you remember why it's there, and a foot-loaded shotgun the moment you forget. One middleware bypass and nothing else stops the request from reading the database. So that was the High-severity finding. The rest of the audit was mostly the kind of thing audits find: missing CSP and HSTS headers, no rate limiting on the MCP endpoint, an in-app `/api/migrate` route that did DDL with a `NODE_ENV` check as its only guard, plaintext credential storage despite the README claiming otherwise, verbose logging of request bodies. Plus the usual dependency advisories — the installed framework version was a major release behind the fix range for a stack of published CVEs. I asked the agent to fix things in batches by severity. Four batches: High, Medium, Low, Info. Each batch its own commit on main, each commit followed by a green CI run. ``` batch-1/high — authoritative auth gate, drop /api/migrate, bump framework version batch-2/medium — SSRF fix, credential encryption, rate-limit, audit log, CORS, matcher, log hygiene, framework auth bump batch-3/low — CSP + HSTS, drop dead config, sync-warning comments batch-4/info — document token blast radius, add CI audit, prevent token-smuggling regressions ``` `checkAuth()` got promoted to `requireAuth(request)`, returning a 401 `NextResponse` when neither a session nor a valid bearer was present. Every API handler grew an early-return on the `NextResponse` result. The MCP self-fetch SSRF surface got eliminated by extracting the Peloton and Tonal sync logic into shared library functions that both the REST routes and the MCP tools call directly. Credential columns got envelope-encrypted with AES-256-GCM via a new `CREDENTIAL_ENC_KEY`. The `/api/migrate` route got deleted entirely. Rate limiting and audit logging landed. CSP, HSTS, all the headers. Framework bumped one major to clear the published advisories. Sixteen-something files changed across the four commits. Zero build failures. CI passing. I pushed the last commit, Vercel redeployed, I went to look at the app. The dashboard was empty. ## The First Thing That Wasn't in the Audit The auth fix had done exactly what it was supposed to do: `requireAuth()` now correctly returned 401 when the route handler couldn't read a session. The problem was that the v5 beta bug it was working around hadn't gone away. So now every API request from my logged-in browser session was 401-ing, because `auth()` was still returning null in route handlers, and the new `requireAuth()` had nothing to fall back to. The audit had flagged the v5 beta version as a finding (F-08) and recommended bumping to the latest beta. I did. The newest beta still had the bug. The original `checkAuth()` had been a workaround. By fixing the workaround, I had unfixed the workaround's reason for existing. The fix was to make `requireAuth()` smarter: when `auth()` returns null, fall back to `getToken()` from `next-auth/jwt`, which reads the session cookie directly from the request and verifies the JWE signature against `NEXTAUTH_SECRET`. Same cryptographic check the middleware does. If it verifies, we synthesize a minimal Session shape from the decoded JWT so handlers reading `session.user.email` still work. If it doesn't verify, *then* we 401. Not "cookie present means trust" — that would have walked us right back into F-01. ``` fix(auth): restore browser data — verify session JWT directly in requireAuth ``` I pushed, Vercel redeployed, I refreshed the page. The dashboard was still empty. ## The Second Thing That Wasn't in the Audit This is the one that stopped me in my tracks, and it's the reason I'm writing this post. I asked the user — me, on my phone — to hit the home page in a fresh tab. The page loaded, no redirect. No "please log in." The dashboard chrome rendered with empty charts. That's impossible. If you're not signed in, middleware is supposed to redirect you to `/login`. Empty data on a rendered dashboard with no session is a state the app shouldn't be able to reach. I added a debug endpoint to dump what the server saw. Cookies, env vars, auth state. The browser session had two cookies: `__Host-authjs.csrf-token` and `__Secure-authjs.callback-url`. Both artifacts of a *partial* sign-in flow. The actual `__Secure-authjs.session-token` — the cookie that proves you signed in — was missing. So I wasn't signed in. But the dashboard was loading. Therefore the middleware wasn't redirecting. Therefore the middleware wasn't running. I checked the build output. The Next.js build prints a route table at the end. Every route I'd expect, but no `ƒ Middleware XX kB` line, which Next.js shows when middleware is compiled in. I dumped `.next/server/middleware-manifest.json`: ```json { "middleware": {}, "sortedMiddleware": [] } ``` **Empty.** No middleware compiled into the build. None. The file at the project root, the one I'd been editing in two of the four audit batches, was being silently ignored by Next.js. I checked out the pre-audit commit. Built that. Same result — empty manifest. So this wasn't a regression from the security work. The middleware had been a no-op since the Next.js 14→15 upgrade, possibly longer. The reason, once I worked it out, is one of those things that is obvious in retrospect and invisible until you trip on it: when a Next.js project uses a `src/` directory layout — which this one does, all code lives under `src/app/` — `middleware.ts` must live at `src/middleware.ts`, *not* at the project root. The root-level file is silently ignored. No warning. No build error. The matcher config is parsed and discarded. The bundle is built without it. I moved the file: ```bash git mv middleware.ts src/middleware.ts ``` Updated the relative import from `./auth` to `../auth`. Built. The route table now showed: ``` ƒ Middleware 87.2 kB ``` The audit had not found this. It had reviewed the middleware *file*, recommended hardening to the matcher pattern (F-03), and added a sync-warning comment about the timing-safe compare (F-11). None of those changes did anything on the running site, because the file they applied to was being thrown away every build. The security implication was significantly worse than any High in the audit report: every page route in the application had been served with **no server-side auth gate at all** for an unknown amount of time. The API was gated only by the route-handler `checkAuth()` (which, per the audit's own F-01 finding, was non-blocking). Anyone who guessed the URL could fetch the prerendered dashboard HTML. The audit couldn't see this because it was reading source files, not build artifacts. The agent did exactly what it was asked to do. The thing it was asked to do didn't include "verify that the files you're reviewing are actually being deployed." I pushed the move. Redeployed. ``` fix(middleware): move middleware.ts into src/ so Next.js actually loads it ``` The dashboard now redirected unauthenticated callers to `/login`, as it should have been doing all along. I clicked Sign in with Google. Google said *Access blocked: Authorization Error. The OAuth client was not found. Error 401: invalid_client.* ## The Third Thing That Wasn't in the Audit When middleware started actually running, it started actually requiring authentication. Which meant I had to actually sign in. Which meant the Google OAuth flow had to actually work. Which it didn't. I traced the redirect URL the app was sending to Google: ``` https://accounts.google.com/o/oauth2/v2/auth?response_type=code&client_id=undefined&... ``` `client_id=undefined`. The literal string. `process.env.GOOGLE_CLIENT_ID` was unset on Vercel. My first theory was an Edge-runtime bundling issue. The new `src/middleware.ts` imports `auth` from `../auth`, which pulls the entire NextAuth config — including the Google provider — into the Edge bundle. The Google provider reads `GOOGLE_CLIENT_ID` at module load time. Edge functions run in their own isolate. If the env var wasn't readable when the Edge isolate cold-started, the provider would cache `clientId: undefined` for the lifetime of that instance. So I rewrote `src/middleware.ts` to not import from `../auth` at all. Same session-cookie verification using `getToken()` from `next-auth/jwt` directly, which reads `NEXTAUTH_SECRET` at *call* time, not at module load. The middleware bundle dropped from 87.2 kB to 45.7 kB — the Google provider was gone. Ship it. `client_id=undefined`. OK, so that wasn't it. Theory two: a debug endpoint that just dumps `process.env[*]` boolean (set/unset, not values). Bearer-gated so I can call it from my workspace without exposing it publicly. The output: ```json { "NEXTAUTH_SECRET": "SET (len=44)", "GOOGLE_CLIENT_ID": "UNSET", "GOOGLE_CLIENT_SECRET": "UNSET", "ALLOWED_EMAIL": "SET (len=19)", "MCP_API_TOKEN": "SET (len=64)", "CREDENTIAL_ENC_KEY": "UNSET", "APP_BASE_URL": "UNSET", ... } ``` `GOOGLE_CLIENT_ID` and `GOOGLE_CLIENT_SECRET` were not set on Vercel. Period. Had never been set, in this Vercel project, ever. The fitness tracker had been deployed to Vercel for a year and there was no Google OAuth client backing it. That meant one of two things. Either the env vars had been there and got purged (unlikely — no recent change to env var configuration), or they had **never been there**, and Google sign-in had **never worked**. I asked. Confirmed: never set up. There was no Google Cloud project. There was no OAuth client. There was no `GOOGLE_CLIENT_ID` to set. This is where my model of the situation broke. The app's `auth.ts` had referenced `process.env.GOOGLE_CLIENT_ID!` since the commit `73fdaab refactor: archive mobile app, add Google OAuth, remove sensitive data` — pre-dating any of the audit work. There was no plausible path where Google sign-in had been working before and stopped working now. So how had I been getting into the app for the past year? The answer, of course, is that I hadn't been. Middleware wasn't running. The page route was being served as static HTML, prerendered at build time, to anyone who asked. The "session" displayed on the dashboard was a fiction generated entirely client-side from API responses that had no auth on them either (because `checkAuth()` was non-blocking). I had been using an unauthenticated copy of my own app, with the database open to the public internet via API routes that thought they had auth. This is the part where I want to be careful about *how* I talk about it, because it makes me sound careless and it makes the agent sound impressive, and neither framing is quite right. The careless one is me. I shipped the app without testing the auth flow against a real Google OAuth client. The agent did exactly what it was hired to do: read the code, identify weaknesses, suggest fixes. It correctly identified that the auth surface was the most important part of the app and prioritized it. It did not — could not — know that the auth surface had never been wired up to anything real in production, because the code looked correct, the env-var references were syntactically present, and the README claimed the feature worked. The audit found patches for nineteen specific defects. The audit did not find that the entire auth feature had been broken at the infrastructure layer for the life of the deployment. To find that, you have to do something the audit wasn't asked to do: try to use the app like a real user, on the real production deployment, with no shortcuts. The walk-through to fix it was the standard Google Cloud OAuth setup — new project, configure consent screen, create Web application credentials, paste the redirect URI exactly (`https://your-app.vercel.app/api/auth/callback/google`, no trailing slash), copy the Client ID and Secret to Vercel env vars, redeploy. About ten minutes once we knew that's what we were doing. Sign-in worked on the first try. ``` chore: remove temporary debug endpoints ``` ## Lessons This whole thing was supposed to be a clean security audit. Instead it was a clean security audit followed by three larger problems that the audit couldn't have caught, each one only visible because the previous one had been fixed. The order matters, because each fix peeled back a layer of compensating behavior that was hiding the next problem. **Audits find what they're asked to look at.** This sounds tautological but it's the whole post. The agent did the work I asked for, well. The pattern that nearly bit me was that the work I asked for had implicit assumptions — that the files I was asking it to review were actually deployed, that the env vars referenced in the code were actually set, that the auth flow it was reviewing was actually used — and those assumptions were all wrong. Static code review is necessary and insufficient. Dynamic verification on the running app is the other half, and I didn't ask for it. **Compensating behavior hides bugs.** `checkAuth()` was non-blocking because of the v5 beta `auth()` bug, but the comment said "middleware already verified auth" — and that wasn't actually true, because middleware was a silent no-op. Two bugs that, together, looked like working software. Either one alone would have produced a 401 or a redirect and gotten my attention. Together, they produced an app that loaded with empty data and made me feel like everything was fine. The audit fixed `checkAuth()`. That fix immediately exposed the middleware-not-running bug. Which immediately exposed the OAuth-never-configured bug. There was no shortcut to the bottom of the stack — each layer had to be peeled in order. **The framework will not tell you when it's ignoring you.** Next.js silently discards a project-root `middleware.ts` when `src/` is present. No warning. No build error. The route table doesn't mention it. The middleware manifest is just empty. There is *probably* a documentation page that says this; I have not gone looking. The point is that the failure mode is invisible from the project's perspective, and the way I caught it was by inspecting a build artifact (`.next/server/middleware-manifest.json`) that I had no prior reason to look at. If your framework supports both `src/` and non-`src/` layouts and the same config files work differently in each, **at least one of those layouts is going to silently swallow somebody's config** and that somebody is probably going to be a vibe coder who doesn't know which layout they're using. **Test the production deployment with a real account, before declaring victory.** I shipped a year-old app and never noticed it didn't have auth. Not because I was lazy — I'd reviewed the code, I'd seen the `signIn` callback, I'd seen the `ALLOWED_EMAIL` guard — but because I'd been opening the app from my own browser, every day, and it had been showing me my data, every day. The behavior I expected matched the behavior I observed. The behavior I expected was a complete fiction. The only test that would have caught it is the dumb one: sign out, open an incognito tab, hit the URL, see what happens. I never did, because the cost of being wrong felt low, until it suddenly didn't. **Cleanup matters.** I shipped six "debug" commits getting to the bottom of these problems — endpoints that dump env state, middleware headers that report what the matcher decided, query parameters that short-circuit the handler with a JSON response. Every one of them was correctly bearer-gated or explicitly scoped to be safe-while-public, and every one of them got removed in `413e282` once we knew what was happening. But "remove the debug endpoint when you're done" is a footnote you forget at your own peril, and the chat transcript above has at least two moments where the agent had to be reminded to clean up. Worth noting that the cleanup commit happened *after* the user said "we're good" — not in the middle of the debugging, when it would have been forgotten. ## By the Numbers - **Audit findings**: 19 (2 High, 8 Medium, 4 Low, 5 Info) - **Audit batches shipped**: 4 - **Audit-batch commits**: 4 (one per severity) - **Total commits over the session**: 18, including six debug-and-revert pairs - **Post-audit critical bugs discovered**: 3 (v5 beta auth() returning null in handlers, middleware file in wrong location, Google OAuth never configured) - **Files that were silently being ignored by the framework**: 1 (`middleware.ts` at project root) - **Months that file had been a no-op**: unknown, ≥ 3 - **Days the audit report had been blessing the no-op file**: same number - **Build manifest line that revealed it**: `"middleware": {}` in `.next/server/middleware-manifest.json` - **Time from audit-handoff to "everything passing again"**: ~4 hours - **Of which time spent on the audit batches**: ~1 hour - **Of which time spent on the three things the audit didn't find**: ~3 hours - **OAuth clients created in Google Cloud Console during this session**: 1 - **Vercel env vars deleted as orphaned afterwards**: ~15 - **Bytes of debug endpoints removed in the cleanup commit**: a few thousand - **Trust in the agent**: unchanged — it did its job well - **Trust in my own assumptions**: significantly reduced === ## Friday Fixes #2: The Unquoted Date That Broke Drafts - URL: https://vibescoder.dev/posts/friday-fixes-the-unquoted-date-that-broke-drafts - Date: 2026-05-22 - Tags: #meta #building-in-public #agents #debugging - Reading time: 8 min read One missing pair of quotes in one frontmatter field took down the admin drafts page. YAML 1.1 auto-parsed the date to a JS Date object, formatDate called .includes on it, and the route 500'd. Here's the bisect from a mobile screenshot to a one-line fix, why only the drafts page broke, and the lesson about trusting types at the YAML boundary. Part two of a two-part Friday Fixes — see #1 for the scheduled-publish workflow bugs that landed the same day. --- Saturday morning. I was on my phone, tapped **Drafts** from the admin top nav, and got Safari's generic "This page couldn't load" screen. No status code, no path, no console. Just a sad triangle and a Reload button. ![iOS Safari error page: "This page couldn't load. Reload to try again, or go back." URL bar shows vibescoder.dev with the path truncated.](/images/friday-fixes-the-unquoted-date-that-broke-drafts/safari-error.png) Dashboard worked. Images worked. Record worked. Only Drafts was broken. Twenty minutes later it was fixed — but the path there is the actual post, because I took the wrong turn first and the bug itself is the kind of thing that will bite anyone shipping a Markdown-driven site. ## The First Wrong Guess I asked my agent to investigate. It looked at the screenshot, saw `vibescoder.dev` in the truncated URL bar with no visible path, and diagnosed the obvious thing: the user hit `/drafts` (no such route), which returns a 404, and on mobile that renders as the generic Safari error page. The recommended fix was equally obvious: use `/admin/drafts`, or add a redirect. ```ts // next.config.ts async redirects() { return [ { source: "/drafts", destination: "/admin/drafts", permanent: false }, ]; } ``` Clean answer. Wrong question. I sent a second screenshot showing the admin top nav, where the **Drafts** link clearly exists and clearly points at `/admin/drafts`. I hadn't typed anything. I'd tapped a link. The link was right. The page itself was 500'ing. This is the part of working with agents I keep relearning: **the first plausible explanation that fits the screenshot is not always the right one.** A truncated mobile URL bar is ambiguous evidence. Agents (and humans) will pattern-match on the visible bits and miss the structural question — *how did the user get there?* ## The Repro Once we agreed the page itself was broken, the loop closed fast: ```bash gh repo clone carryologist/the-vibe-coder gh repo clone carryologist/the-vibe-coder-content cp -r the-vibe-coder-content/content the-vibe-coder/ cd the-vibe-coder SESSION_SECRET=... ADMIN_PASSWORD=dev npx next build SESSION_SECRET=... ADMIN_PASSWORD=dev npx next start -p 3939 & curl -c jar -X POST localhost:3939/api/auth/login -d '{"password":"dev"}' curl -b jar localhost:3939/admin/drafts -w "%{http_code}\n" # → 500 ``` The server log had exactly one useful line in it: ``` ⨯ TypeError: a.includes is not a function at (.next/server/chunks/ssr/src_0k9vqrt._.js:1:157) at Array.map () ``` Minified beyond useful, but the shape was enough: something inside a `.map()` was calling `.includes()` on a value that wasn't a string. ## The Bug I grepped `.includes(` across the source. One of the hits was in `src/lib/format-date.ts`: ```ts export function formatDate(dateStr: string, options = {...}): string { if (!dateStr) return "No date"; const normalized = dateStr.includes("T") ? dateStr : dateStr + "T00:00:00"; ... } ``` The `.includes("T")` was deliberate. It distinguishes a date-only frontmatter value (`"2026-05-14"`, which needs `T00:00:00` appended to parse as local time) from a full datetime (`"2026-05-14T05:00:00-07:00"`, which doesn't). I'd written that helper *specifically* because an earlier bug had us appending `T00:00:00` to a datetime and getting `Invalid Date`. It got its own Friday Fixes post: ["Mobile First and the Skill That Saved Us"](/posts/friday-fixes-mobile-first-and-the-skill-that-saved-us). If the scheduled-publish workflow bugs in [Friday Fixes #1](/posts/friday-fixes-two-bugs-one-workflow) feel related — they are. Same week, same content pipeline. That post covers two workflow-level failures that were invisible to the main code path. This one is the data-level failure that hid behind a login boundary. The TypeScript signature says `dateStr: string`. The other 35 posts in the repo say it's a string. The helper has been in production for weeks. What changed? One draft had this in its frontmatter: ```yaml title: "From Cloud Native to AI Native: Learning from Past Patterns" date: 2026-05-14 description: "Exploring the parallels..." ``` Look at the date. **No quotes.** YAML 1.1 — which is what `js-yaml` and `gray-matter` parse by default for compatibility — has aggressive auto-typing. `2026-05-14` matches the ISO date pattern. The parser doesn't hand back a string; it hands back a JavaScript `Date` object. `Date.prototype.includes` does not exist. Throw. ```js node -e " const matter = require('gray-matter'); const fs = require('fs'); for (const f of fs.readdirSync('content/posts').filter(f => f.endsWith('.mdx'))) { const { data } = matter(fs.readFileSync('content/posts/' + f, 'utf8')); if (typeof data.date !== 'string') { console.log(f, '→', data.date instanceof Date ? 'Date' : typeof data.date); } } " # from-cloud-native-to-ai-native-learning-from-past-patterns.mdx → Date ``` One offender. Out of thirty-six. ## Why Only Drafts Worth pausing on this, because it's the part that explains why I'd been shipping with a bomb in the codebase for who knows how long. Public routes — the homepage, individual post pages, tag pages — filter to published posts *before* anything formats dates: ```ts function _getAllPosts(): Post[] { return files .map(...) .filter((post) => post.published) // ← drops the bomb here .filter((post) => new Date(post.date) <= new Date()); } ``` The broken post had `published: false`. The homepage never saw it. Its date object never reached `formatDate`. From the public site's perspective, everything was fine. The drafts page, by definition, does the opposite. It calls `getAllPostsAdmin()`, which returns *every* post including unpublished ones, then maps over the unpublished ones and formats their dates. One bad apple, one route that touched it, full 500. This is a useful pattern to notice: **the same data can be safe on one code path and explosive on another.** The bug was in the data for as long as that draft has existed. The bug was *visible* only on one specific route, behind login, that I look at a couple of times a week. ## The Fix Two commits, two repos. **Content** — quote the date so the parser hands back a string: ```diff - date: 2026-05-14 + date: '2026-05-14' ``` **Engine** — `formatDate` shouldn't crash a whole page over a missing pair of quotes. Coerce non-strings before the substring check, and widen the type signature so TypeScript reflects what can actually arrive: ```ts export function formatDate( dateStr: string | Date | null | undefined, options = {...}, ): string { if (!dateStr) return "No date"; // Coerce Date objects (from YAML auto-parsing) to ISO strings. const asString = dateStr instanceof Date ? dateStr.toISOString() : typeof dateStr === "string" ? dateStr : String(dateStr); const normalized = asString.includes("T") ? asString : asString + "T00:00:00"; const d = new Date(normalized); if (isNaN(d.getTime())) return asString || "No date"; return d.toLocaleDateString("en-US", options); } ``` I verified the engine fix in isolation by re-breaking the content back to unquoted, rebuilding, and hitting the route — 200. Then I shipped both fixes, watched the Vercel deploys go green, and reloaded on the phone. Drafts page rendered. ## Gotchas A few things from this one worth filing away. **The TypeScript signature lied.** `dateStr: string` made the helper look safe. At the YAML/JSON/env-var boundary, types are aspirational. Anything that comes from a parser is `unknown` until you've actually checked it. The defense is to coerce at the boundary, not to trust the shape upstream. **YAML 1.1 auto-typing is treacherous.** The trio: | Frontmatter | Parsed as | |---|---| | `date: 2026-05-14` | `Date` object | | `date: '2026-05-14'` | string | | `date: "2026-05-14"` | string | Most YAML examples on the internet omit the quotes. Most YAML *editors* preview the file fine either way. The quotes are load-bearing only at parse time, and only for some parsers. YAML 1.2 (which people think they're writing) is stricter, but `js-yaml` and most JS-ecosystem parsers default to 1.1. **Mobile error pages hide everything.** No status code, no path, no console. If the user had only sent the first screenshot, "you typed the wrong URL" was a defensible answer that would have stuck for another day until I tried it on desktop. The second screenshot of the nav source was what flipped the diagnosis. **The first plausible explanation isn't always right.** When the agent's first answer fit the visible evidence but contradicted my mental model of how I'd gotten there, I sent more evidence instead of accepting the fix. That's the loop that catches this class of bug — pushing back once with a second screenshot was worth more than ten more minutes of investigation on the wrong path. ## By the Numbers - **1** unquoted date in **36** posts (2.8%) broke the page - **2** commits across **2** repos to ship the fix - **27 → 49** lines in `format-date.ts` after defensive coercion + JSDoc - **0** public routes affected — they filter unpublished posts before formatting - **~20 minutes** from screenshot to verified-in-prod, including the wrong turn - **2** Vercel deploys triggered (one per repo, both green) - **500 → 200** on `/admin/drafts` post-deploy - **1** lesson, same as last time: when the agent's answer doesn't fit your model, send another screenshot === ## Friday Fixes #1: Two Bugs, One Workflow - URL: https://vibescoder.dev/posts/friday-fixes-two-bugs-one-workflow - Date: 2026-05-22 - Tags: #building-in-public #debugging #agents #meta - Reading time: 6 min read The scheduled-publish GitHub Action broke twice in nine days. Bug one: a grep that matched body text instead of frontmatter, triggered by a post about the feature itself. Bug two: a dead-code line introduced by the fix for bug one — racy under set -euo pipefail, probabilistically silent for eight days, then 42 consecutive failures with zero notifications. --- The `scheduled-publish.yml` workflow runs every 15 minutes. It scans every `.mdx` file, finds posts where `published: false` and `publishAt` is in the past, flips the flag, commits, and pushes. Vercel picks up the push. Post goes live. Simple. It broke twice in nine days. The second break was caused by the fix for the first. ## Bug 1 the Grep That Read the Whole File May 3 The workflow's detection logic was one line: ```bash if grep -q 'published: false' "$file"; then ``` That scans the entire file — frontmatter and body text both. On May 3 a scheduled draft failed to publish, and the workflow log showed it dying in 7 seconds. The culprit: "Friday Fixes: The Agent Was Flying Blind." That post was already live. It had `published: true` in its frontmatter. But it also had `published: false` in its body — in the section explaining how the `publishAt` field works, where I'd written out example frontmatter: ```yaml published: true ``` Grep matched the example in the prose. The workflow entered the processing block, tried to flip a flag that wasn't there in the frontmatter, and failed. Seven seconds, start to crash. The self-referential shape is hard to miss. The post that introduced scheduled publishing was the first thing the feature's own bug tripped over. **The fix**: extract frontmatter first, then grep and parse from that. ```bash FRONTMATTER=$(sed -n '2,/^---$/p' "$file") if echo "$FRONTMATTER" | grep -q 'published: false'; then PUBLISH_AT=$(echo "$FRONTMATTER" | grep '^publishAt:' | sed "s/publishAt: '//;s/'//") # ... rest of processing fi ``` After the fix I ran the workflow manually. It correctly detected 5 real scheduled drafts, published them all, and left the already-published post alone. I also noticed the admin link was missing from the desktop nav — `Header.tsx` had it in the hamburger menu but not in the top bar. Added it while I was in there. ## Bug 2 the Dead-Code Line That Wasn't Harmless May 12 The May 4 commit that introduced frontmatter extraction also included a verification line — something I'd written to sanity-check the sed pattern during development and then left in: ```bash sed -n '1,/^---$/!{/^---$/,/^---$/p}' "$file" | head -1 > /dev/null 2>&1 ``` This line discards everything. Stdout to `/dev/null`, stderr to `/dev/null`, exit code gone — except it wasn't, because `set -euo pipefail` was active. Here's what happens. `head -1` reads one line and exits, closing the read end of the pipe. `sed` writes to a closed pipe and receives SIGPIPE. Under normal circumstances that's fine — sed exits 141, everyone moves on. Under `pipefail`, the non-zero exit from the left side of the pipe propagates. Under `set -e`, the script dies. The `> /dev/null 2>&1` redirect silences output; it does nothing about the exit code of the pipeline. **Why it took nine days to notice**: the race is probabilistic. If `sed` finishes writing before `head` closes the pipe — because frontmatter is short and the file is small — `sed` exits cleanly. With one or two drafts, `sed` almost always won. As drafts accumulated, the probability of losing the race on at least one file per run climbed toward 100%. Timeline: - **May 4**: bug introduced, 1–2 drafts in repo, `sed` almost always finished before `head` closed the pipe - **May 5–8**: intermittent — 58 successful runs, ~10 losses, looked like runner noise - **May 9, 17:11 UTC**: last successful run - **May 9–12**: 42 consecutive failures, zero notifications - **May 12, ~12:00 UTC**: `the-fix-that-was-fixed-four-times` misses its slot; I notice two hours later when the post isn't live GitHub emails you when a workflow transitions from passing to failing. Keep failing and you get nothing. By the time I had 42 consecutive failures, the notification had fired once — probably on May 9 — and been absorbed into some digest I'd dismissed. The ongoing silence was indistinguishable from the workflow running cleanly. The right health metric for a cron job isn't "did it fail" — it's "when did it last succeed." I had no visibility into the latter. **The fix**: delete the line. ```diff -sed -n '1,/^---$/!{/^---$/p}' "$file" | head -1 > /dev/null 2>&1 ``` Three lines removed (command, blank line, and the comment above it). The real frontmatter extraction on the next line — `FRONTMATTER=$(sed -n '2,/^---$/p' "$file")` — had been working the entire time. The verification line was never doing anything useful. After the delete and push, I ran `gh workflow run scheduled-publish.yml` manually to recover the missed slot. The post published within a minute. ## What Connects Them Both bugs are about code that looks inert but isn't. In Bug 1, the `grep` line looked like a safe filter. The assumption that `published: false` would only appear in frontmatter was invisible — there was no code encoding that assumption, just the pattern itself. Body text violated it immediately. In Bug 2, the dead-code line looked like it was doing nothing — output to `/dev/null`, stderr to `/dev/null`, result irrelevant. But it was creating a pipeline under a shell mode where broken pipelines are fatal. The `> /dev/null` made it *look* inert. The SIGPIPE made it a probabilistic kill switch. Dead code with side effects is worse than dead code without. Under `set -euo pipefail`, any pipeline where the right side terminates early (`head`, `grep -m 1`, `awk 'NR==1{exit}'`) can kill the script if the left side is still writing. If you want the first line of a file, read the first line — don't pipe the whole file through `head`. This class of race condition doesn't fail cleanly. It produces a noise floor that rises asymptotically to 100%: indistinguishable from background noise until the crossover point, then complete silence — which looks identical to everything working. If the unquoted-date YAML bug in [Friday Fixes #2](/posts/friday-fixes-the-unquoted-date-that-broke-drafts) feels related — it is. Same week, same content pipeline, different failure mode. That one hid in a draft post and only surfaced on a route that touches unpublished content. Same pattern of a bug that's invisible to the main code path until a specific condition exposes it. ## By the Numbers - **7 seconds** — time to failure for Bug 1 - **1** self-referential bug — the post about scheduling broke scheduling - **5** real scheduled drafts correctly detected after the Bug 1 fix - **42** consecutive workflow failures during the Bug 2 window (May 9–12) - **0** email notifications during those 42 failures - **3** lines removed to fix Bug 2 - **8 days** between introducing Bug 2 and fixing it - **~2 hours** between "post should have published" and "I noticed it didn't" - **1** manual `workflow_dispatch` to recover the missed slot === ## Thursday Thoughts: Audit Your Vibe Code, Often - URL: https://vibescoder.dev/posts/thursday-thoughts-audit-your-vibe-code-often - Date: 2026-05-21 - Tags: #agents #vibe-coding #security #meta #building-in-public - Reading time: 9 min read Someone vibe coded an app with Google AI Studio. The Gemini API key shipped in the client-side JavaScript bundle. Google suspended the project. Here's why every AI coding tool gets this wrong, why regular audits are the only real defense, and what you can do before it happens to you. --- Someone I know built a web app with Google AI Studio. React frontend, Firebase auth, Gemini API for the AI features, deployed on Vercel. A real product — users, a custom domain, the works. Built fast, shipped fast, worked great. Then they got an email from Google: API key compromised, project suspended, locked out of Google Cloud Platform entirely. Couldn't access the console. Couldn't revoke the key. Couldn't see the billing. Couldn't fix anything. The root cause took about thirty seconds to find. The Gemini API key was hardcoded in the client-side JavaScript bundle. Not in an environment variable on the server. Not behind a proxy. In the bundle. Minified, sure — but `view-source` doesn't care about minification. Anyone who visited the site could open DevTools, search the bundle for `AIza`, and walk away with a working Gemini API key billed to someone else's account. Someone did. Google noticed the abuse, suspended the project, and now the developer is locked out of everything while they work through Google support to get it restored. This isn't a story about one careless developer. It's a story about what happens when AI coding tools optimize for "make it work" and nobody in the pipeline checks for "make it safe." ## How It Got There When you tell Google AI Studio — or any vibe coding tool — something like "build me a React app that uses Gemini to generate content," the model takes the shortest path to a working demo. That path is: ``` import { GoogleGenAI } from '@google/genai'; const client = new GoogleGenAI({ apiKey: "AIza..." }); const response = await client.models.generateContent({ ... }); ``` Three lines. Works immediately. No backend, no proxy, no infrastructure. From the model's perspective, the task is done. The app calls Gemini and renders the result. Ship it. The model doesn't think about what happens when those three lines end up in a production JavaScript bundle served to the public internet. It doesn't reason about deployment context. It doesn't have a threat model. It solved the functional requirement — "call Gemini, get a response" — and moved on. ## Why the Model Gets This Wrong Five compounding factors: **The training data is full of tutorials that do exactly this.** Every quickstart guide, every blog post, every "Getting Started with Gemini" doc shows the API key inline. Because they're teaching the API, not teaching production architecture. The model learned from thousands of examples where hardcoding the key was the correct thing to do — in a tutorial context. **Vibe coding collapses the frontend/backend boundary.** A traditional app has a clear separation: API keys go on the server, the client talks to your server, your server talks to the external API. But when you prompt an AI to build a full app in one shot, that boundary doesn't exist unless you explicitly ask for it. The model generates a single-page React app. There's no server. The key goes where it works: the client. **LLMs don't reason about deployment.** The model doesn't think "this code will be minified into a bundle, served via CDN, and visible to anyone with a browser." It generates code that satisfies the functional requirement in the current context. The concept of "this code is about to become public" isn't part of its reasoning. **No tool in the pipeline catches it.** Google AI Studio generates the code. Vercel deploys it. Firebase serves the auth. None of them scan the build output for exposed API keys. None of them warn that a billable secret is shipping in client-side JavaScript. The developer pushes, Vercel builds, the site goes live, and the key is public. **The platform irony.** Google AI Studio is building an app that calls Google's own API, and it doesn't flag that the key it's embedding will be publicly visible. Google knows which of its API keys are billable. They could build guardrails — warn when a Gemini key appears in client-side code, auto-generate a serverless proxy, or at minimum add a comment saying "move this server-side before deploying." They don't. ## The Blast Radius When a Gemini API key leaks in a client bundle, the damage escalates fast: 1. **Unauthorized API usage.** Anyone with the key can make Gemini calls billed to the developer's account. Automated scrapers can burn through thousands of dollars in hours. 2. **Project suspension.** Google detects the abuse pattern and suspends the GCP project. This is the right security response from Google's side — but it locks the developer out of everything, not just the compromised key. 3. **Cascading lockout.** The suspension doesn't just affect Gemini. It hits Firebase, Cloud Storage, the console itself. If the billing account is shared across projects, other projects can be affected too. 4. **Recovery requires support.** You can't self-service your way out of a suspended project. You need to contact Google Cloud support, explain what happened, dispute unauthorized charges, and wait. The developer in this case is currently working through that process. They're not locked out because they did something malicious. They're locked out because the tool they used to build the app put a billable secret in a public place, and they didn't know to check for it. ## The Fix Is Architecture Not Configuration The correct architecture for any app that calls a paid external API from a web frontend: ``` Browser → Your backend (serverless function) → External API ↑ ↑ API key lives here Never touches client ``` For a Vercel-deployed app, that means: 1. Create a serverless function (e.g., `/api/generate`) that holds the API key as a **server-side environment variable** set in the Vercel dashboard. 2. The frontend calls `/api/generate` with the user's prompt. 3. The serverless function calls Gemini with the key, returns the result. 4. Add authentication checks — verify the Firebase ID token so only logged-in users can trigger API calls. 5. Add rate limiting — cap requests per user per minute. The API key never appears in any client-side code. It exists only in the Vercel environment, only accessible to your server-side functions. On top of the architecture fix: - **Restrict the key** in GCP Console → Credentials. Lock it to your serverless function's domain or IP. - **Set a daily quota cap** on the Gemini API. Even if a key leaks again, the damage is bounded. - **Enable billing alerts** at low thresholds — $10, $50, $100 — so you know before Google does. ## Why We Audit This is exactly the class of bug that regular audits exist to catch. I've written about our audit practice before — [three agents auditing this blog](/posts/closing-the-loop-from-audit-to-ten-commits), a [spring cleaning of a year-old fitness tracker](/posts/spring-cleaning-your-vibe-coded-apps). In both cases, the most valuable findings were things the builder didn't know were wrong. A migration route accessible in production. API routes with no secondary auth checks. A scheduled-publish draft with leaked identifiers that would have gone live at 5 AM. The pattern is the same every time: the person who built the app was optimizing for features, not for security. The app worked. The code was functional. And somewhere in the gap between "works" and "safe," a bug was waiting. A Gemini API key in a JavaScript bundle is exactly this bug. It works. The app calls Gemini and renders the result. The developer tests it, it behaves correctly, they ship it. Nothing in the development workflow surfaces the fact that the key is now public. It hides in plain sight until someone finds it — either an auditor or an attacker. The audits we run on this blog aren't paranoia. They're the acknowledgment that vibe coded apps — apps built fast, by AI, with the builder optimizing for "make it work" — accumulate a specific class of debt that only shows up when someone looks for it with fresh eyes. API keys in bundles. Auth checks that rely on a single middleware layer. Verbose error messages leaking stack traces. Security headers that were never added because the app worked without them. If you're vibe coding — and at this point, most of us are — the question isn't whether your app has this kind of bug. It's whether you find it before someone else does. ## What the Tools Should Do This is a solvable problem. Not by the developer — by the platforms. **Google AI Studio** should detect when generated code embeds a Gemini API key in client-side JavaScript and either refuse, warn, or generate a server-side proxy automatically. Google has the context: they know the key is theirs, they know it's billable, and they know the code is going to a browser. **Vercel** should scan build output for common API key patterns (`AIza`, `sk-`, `AKIA`) and flag them as warnings during deployment. They already analyze bundles for size — analyzing them for secrets is the same capability. **Firebase** should enforce App Check by default on new projects, not as an opt-in feature that most developers don't know exists. **Every AI coding tool** should treat "API key in client-side code" as a lint error, not a feature. The same way ESLint flags unused variables, these tools should flag exposed secrets. The pattern is well-known. The regex is simple. The cost of not catching it is real. Until the tools catch up, the defense is audits. Regular, systematic, fresh-eyes reviews of what your app actually ships to the browser. Not what you think it ships. What it actually ships. ## By the Numbers - **1 API key** in a client-side JavaScript bundle — all it takes - **30 seconds** to find it with DevTools and a search for `AIza` - **0 warnings** from Google AI Studio, Vercel, or Firebase during the entire build-and-deploy pipeline - **3 lines of code** that the AI generated to make it "work" — and created the vulnerability - **100%** of the damage preventable with a serverless proxy function - **0** vibe coding tools that currently scan for API keys in client bundles - **1 regular audit** — the difference between finding it yourself and getting the email from Google === ## Wiring MCP Into My Fitness Tracker — and Asking OpenClaw About My Last Workout - URL: https://vibescoder.dev/posts/wiring-mcp-into-my-fitness-tracker-for-openclaw - Date: 2026-05-20 - Tags: #mcp #openclaw #agents #homelab - Reading time: 13 min read I built a Model Context Protocol server into the fitness tracker I vibe coded a year ago, wired it through Vercel and Coder workspaces, and ended the afternoon asking my Discord bot what my last workout was. Here's the build, the wrong turn into Coder's AI Bridge, the workaround, and how the same endpoint now serves Claude Desktop, Codex, Coder Agents, and OpenClaw. --- I open my [fitness tracker](/posts/spring-cleaning-your-vibe-coded-apps) every day. It pulls workouts from Peloton and Tonal, tracks annual goals, makes pretty charts. Until this week, the way I interacted with it was: open browser, click button, look at chart. Like a 2018 web app. This week I made it an MCP server. Now I ask Discord "what was my last workout?" and **carrybot** — my homelab [OpenClaw](/posts/installing-openclaw-on-the-homelab) bot, running on my Linux homelab PC, talking to a local Qwen3.5-35B on llama.cpp — answers with real data from the same Postgres my browser hits. Same endpoint also works from Claude Desktop, Codex, Cursor, and any Coder workspace agent that knows how to call it. This is the writeup of the afternoon that took me there. The MCP server itself was easy. The interesting parts were the constraints I bumped into and the workarounds that turned out to be cleaner than the "right" answer. ## The Goal One sentence: **let any AI agent talk to my fitness data**. The vibe coded fitness tracker is a single-user Next.js 14 app on Vercel. Gated to one Google account. REST endpoints behind a NextAuth session cookie. Peloton and Tonal sync triggered by clicking buttons in the dashboard. That works for the browser. It doesn't work for an agent that wants to ask "summarize my training over the last quarter" or "trigger a Peloton sync — did anything new come in?" I want the agent to have **raw access**. No precomputed summaries. Give it the rows and let it figure out the trends. Part of the point is to learn how agents get better at this kind of analysis over time, and that doesn't happen if I do the math for them. ## Why MCP Not OpenAPI I almost shipped this as an OpenAPI spec plus bearer-token auth. Cleaner, simpler, every agent framework supports it. Then I listed the clients I actually want to use: | Client | OpenAPI | MCP | |---|---|---| | Claude Desktop | Custom integration | Native | | Codex CLI | Custom integration | Native | | Coder Agents | Via AI Bridge | Via AI Bridge | | OpenClaw | Via plugin | Native | | Cursor, Windsurf, Zed | Custom | Native | Every client speaks MCP first-class. Ship MCP, write the tools once, every agent picks them up by pointing at a URL. Ship OpenAPI and every client needs bespoke wiring. The decision was over before I finished the table. ## The Server Three files, ~400 lines total. **`src/app/api/mcp/[transport]/route.ts`** — the MCP route, built on [`mcp-handler`](https://github.com/vercel/mcp-handler) (the package formerly known as `@vercel/mcp-adapter` before it got renamed and republished). Ten tools: ```ts server.tool('list_workouts', /* schema */, async ({...}) => {...}) server.tool('get_workout', /* schema */, async ({id}) => {...}) server.tool('create_workout', /* schema */, async ({...}) => {...}) server.tool('update_workout', /* schema */, async ({...}) => {...}) server.tool('delete_workout', /* schema */, async ({id}) => {...}) server.tool('list_goals', /* schema */, async () => {...}) server.tool('peloton_status', /* schema */, async () => {...}) server.tool('sync_peloton', /* schema */, async ({limit})=> {...}) server.tool('tonal_status', /* schema */, async () => {...}) server.tool('sync_tonal', /* schema */, async ({limit})=> {...}) ``` The CRUD tools wrap Prisma directly. The sync tools `fetch()` the existing REST endpoints (`/api/peloton/sync`, `/api/tonal/sync`) so I'm not duplicating the dedup orchestration — those endpoints already handle "did we already sync this workout? does this row need backfilling? did the Peloton token expire?" Wrapping them is one HTTP hop. Worth it to keep one source of truth for sync logic. **`src/lib/api-auth.ts`** — bearer token helpers. The token is a single env var, `MCP_API_TOKEN`, 64 random hex chars. Compared in constant time so I don't leak timing side channels: ```ts function timingSafeEqual(a: string, b: string): boolean { if (a.length !== b.length) return false let mismatch = 0 for (let i = 0; i < a.length; i++) { mismatch |= a.charCodeAt(i) ^ b.charCodeAt(i) } return mismatch === 0 } ``` **`middleware.ts`** — extended so the bearer token unlocks every `/api/*` route, not just `/api/mcp`. Same token, two callers: the MCP server calls Prisma directly for read tools, and self-`fetch`es the existing REST routes for the sync tools. Both paths need to pass auth. The token does double duty. The transport choice was the one decision worth thinking about. `mcp-handler` supports SSE and streamable HTTP. SSE needs Redis for message brokering. Streamable HTTP is stateless. I'm on Vercel Hobby with no Redis. `disableSse: true` and ship. ```ts { basePath: '/api/mcp', verboseLogs: false, maxDuration: 300, disableSse: true } ``` `pnpm i mcp-handler @modelcontextprotocol/sdk@1.26.0 zod` — and yes, you have to pin the SDK to 1.26.0 because `mcp-handler@1.1.0` peer-depends on exactly that version, not a semver range. Half an hour of `npm install` errors before I noticed. ## The Test That Said It Worked ```bash curl -sS -X POST https://.vercel.app/api/mcp/mcp \ -H "Authorization: Bearer $MCP_API_TOKEN" \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' ``` Response: `200 OK`, `event: message`, full tool catalog with JSON Schemas. The server worked. The hard part wasn't the server. It was getting the four clients I cared about to use it. ## Client #1 Claude Desktop Codex Cursor the Easy Path These all read a JSON config file with the same shape: ```json { "mcpServers": { "fitness-tracker": { "type": "http", "url": "https://.vercel.app/api/mcp/mcp", "headers": { "Authorization": "Bearer " } } } } ``` Drop in the URL, drop in the token, restart the client. Done. ## Client #2 Coder Workspace Agents the Path I Got Wrong I run [Coder](https://coder.com) on my workstation. Every workspace gets a `~/.mcp.json` baked in by the Terraform template (Context7, Vercel, Cloudflare, Playwright — see [the homelab post](/posts/installing-openclaw-on-the-homelab)). My mental model: add a fifth entry for fitness-tracker, the agent picks it up. So I patched the template. Token flows from `~/.config/fitness-tracker/env` on the workstation → `TF_VAR_fitness_tracker_mcp_token` in `/etc/coder.d/coder.env` → Terraform `variable` → `coder_agent.main.env` → workspace process → `jq`-merge into `~/.mcp.json` at startup with `chmod 600`. One PR, one `apply.sh`, every workspace gets it. Verified the file showed up in a fresh workspace with all five MCP servers in the keys. Confidently asked the agent: "list my fitness-tracker tools." > "I don't have any fitness-tracker tools available. My available tools are for software-engineering tasks inside a Coder workspace..." The agent had no idea. Started a fresh chat — same answer. Inspected the agent runtime and found this in Coder's source at v2.33.2: ```go // enterprise/aibridgedserver/aibridgedserver.go for _, link := range links { if link.ProviderID != eac.ID { continue } valid, _, validateErr := eac.ValidateToken(ctx, link.OAuthToken()) // ... tokens[id] = link.OAuthAccessToken } ``` **Coder's AI Bridge only auto-registers OAuth-backed MCP servers.** Specifically, MCP servers wired through `CODER_EXTERNAL_AUTH_*_MCP_URL` against an OAuth external auth provider. Static-token MCP servers are invisible to the chat agent. The `~/.mcp.json` file is for *other* MCP clients running in the workspace (Claude Desktop, Codex, code-server's Continue extension), not for Coder's chat itself. I'd shipped a `coder-templates` PR that does the right thing for every MCP client *except* the one I was trying to enable. The PR is still useful — it makes the fitness tracker available to any MCP client a workspace user wires up. But Coder Agents specifically were locked out. Two real options: 1. **Wrap the fitness tracker in OAuth.** NextAuth supports being an OAuth provider. Register it in Coder as an external auth. Coder mints tokens, AI Bridge injects them. Significant work for a single-user app. 2. **Teach the agent the recipe.** Write a skill file that documents the endpoint, the auth, the wire shape, and the ten tools. Agent reads the skill at chat start and calls the MCP server with `curl`. Option 2 was 200 lines of Markdown. I picked option 2. ```markdown --- name: fitness-tracker description: "Access the personal fitness-tracker MCP server via raw HTTP..." --- ## Call recipe ft_call() { local tool="$1" args="${2:-{\}}" local payload=$(jq -cn --arg t "$tool" --argjson a "$args" \ '{jsonrpc:"2.0", id:1, method:"tools/call", params:{name:$t, arguments:$a}}') curl -sS -X POST https://.vercel.app/api/mcp/mcp \ -H "Authorization: Bearer $FITNESS_TRACKER_MCP_TOKEN" \ ... | sed -n 's/^data: //p' | jq -r '.result.content[0].text // .error' } ft_call list_workouts '{"year":2025,"limit":10}' ``` That ships in the same template patch as the `~/.mcp.json` entry. Per-user, gated to my account. Next fresh chat: agent reads the skill, sees the recipe, calls the server. Works on the first try. ## Client #4 OpenClaw the Client I Built This For The whole reason I started this afternoon. `carrybot` runs on the homelab and accessed from any Discord client. Local Qwen3.5-35B for the brain, Discord for the front end, OpenClaw as the agent runtime. I DM'd carrybot and asked how to register an MCP server. He came back with: > "I need a bit more info about your fitness app MCP endpoint: > Transport type: Is it HTTP(S), SSE, streamable-http, or stdio? > URL: What's the endpoint URL? > Auth: You mentioned a token — is it in headers or URL? > Tool names: What specific tools does it expose?" Pasted the answers. He ran: ```bash openclaw mcp set fitness-app '{ "type": "streamable-http", "url": "https://.vercel.app/api/mcp/mcp", "headers": { "Authorization": "Bearer ${FITNESS_TRACKER_MCP_TOKEN}" } }' ``` Asked: "What was my last workout?" He replied with a 30-minute Power Zone Pop Ride with Ben Alldis on May 12 at 4:12 PM PDT. 7.98 miles. Real Peloton data, pulled through the MCP server I'd built three hours earlier, into a Discord bot running in the other room. ## The Cron Ghost One amusing wrinkle. Carrybot's first response included an inline alert: > ⚠️ Cron: `fitness-check` failed I asked him what that was. He came back with full transparency: > "You're right to ask! That 'fitness-check' cron never actually got created — I made a typo when trying to add it. ... The API returned: `invalid cron.add params: at root: unexpected property 'sessionTarget":'`. That's because I accidentally added an extra quote: `"sessionTarget": ": "main"` should have been `"sessionTarget": "main"`. The JSON was malformed." He'd tried to use OpenClaw's cron system as a deferred-execution primitive — schedule a one-shot task for one minute from now, then have the cron run the MCP query. Malformed JSON, cron creation failed, OpenClaw faithfully alerted me about the failure, and he answered my question via a sub-agent path instead. The data I saw was real. The "cron failed" was background noise from a bug he caught and fixed in his next message. I gave him the UX feedback — one-shot crons as deferred execution mean every malformed payload looks like a production alert — and he immediately revised the pattern: > "Sub-agent direct call → Use for one-off work. Cleaner, no cron overhead. > Cron for real scheduling → Only use `at` crons when you actually need deferred execution. > Don't spam alerts → Malformed JSON that prevents a cron from being created shouldn't generate a scary 'Cron failed' alert." That's a long-running agent learning its own UX patterns. Worth its own post someday. ## Token Storage One token, four locations, all mode 600 or equivalent. Same value everywhere: ``` 1. Vercel project env var MCP_API_TOKEN 2. Workstation ~/.config/fitness-tracker/env (chmod 600) 3. Coder server /etc/coder.d/coder.env (root-readable systemd EnvironmentFile) 4. Coder workspaces ~/.mcp.json (chmod 600, regenerated per workspace start) 5. OpenClaw ~/.openclaw/openclaw.json (chmod 600) ``` Rotation: `openssl rand -hex 32`, update all five locations, redeploy Vercel. Roughly 90 seconds, no code changes. The token lives in env vars, never in shell rc files. The shell-rc anti-pattern is real — anything `export`ed into `~/.bashrc` leaks into every subshell's process listing, gets sourced by background jobs that shouldn't see it, and survives in `.bash_history` for as long as that file lives. A `chmod 600` env file you source explicitly when you need it stays in exactly the processes that need it. ## What I'd Do Differently **Verify the agent runtime's MCP integration before patching templates.** I patched `coder-templates` to add a workspace-level `~/.mcp.json` entry before I'd checked whether Coder's chat agent actually reads that file. It doesn't. The patch is still useful for other MCP clients running in the workspace, but I wouldn't have prioritized it first if I'd known. **Skip the OpenAPI consideration earlier.** I spent real cycles writing the "MCP vs OpenAPI" comparison in my head. The clients I cared about all speak MCP natively. The decision was over before I started thinking about it; I just didn't realize it for ten minutes. **Start with the skill file as a first-class option, not a workaround.** When I hit the Coder AI Bridge limitation, my first instinct was "build OAuth, ship the proper integration." The skill file approach is genuinely simpler, lives next to existing skills, and will be obsolete the day AI Bridge gains static-token support — which seems like a planned-but-not-yet-shipped feature based on the deprecation comments in Coder's source. Skill files are the right level of investment when the underlying platform is in flux. ## What's Next 1. **Test the skill in a fresh Coder chat.** The PR merged but I haven't validated it end-to-end yet. The skill is concrete enough that the agent should call `ft_call list_workouts` on the first try. If it fumbles, the skill needs tightening. 2. **Watch the raw-rows decision over time.** All ten tools return raw database rows. Zero precomputed aggregates. The whole point is to see whether agents naturally synthesize good summaries or degrade as the dataset grows. If they degrade, add a `summarize_year` tool. Until then, keep the surface area small. 3. **Token rotation drill.** I haven't had to rotate `MCP_API_TOKEN` yet. Worth doing once intentionally to find any place we forgot to document. 4. **Wait for AI Bridge to support static-token MCP servers.** When it does, the skill file becomes redundant and the `~/.mcp.json` entry becomes the canonical path. Until then, the skill is the working path. The fitness tracker is now genuinely agent-accessible. Same vibe coded app that started as a Next.js weekend project, now serving four different agent runtimes through a single MCP endpoint. The audit a few weeks ago found the bugs. This week added the API surface. Next steps are about watching agents use it. The lobster's a real assistant now. ## By the Numbers - **3 hours** total session time - **2 GitHub PRs** opened and merged (fitness-tracker, coder-templates) - **1 follow-up PR** for the skill file workaround - **10 MCP tools** exposed, all returning raw rows - **0 precomputed aggregates** — agents do their own analysis - **4 client integrations** working from one endpoint (Claude Desktop, Codex / Cursor / etc., Coder Agents via skill, OpenClaw) - **1 dead-end** — Coder AI Bridge's OAuth-only MCP injection requirement - **200 lines** of Markdown in the skill that workaround it - **64 hex chars** in the personal access token - **5 locations** that hold the token, all mode 600 or equivalent - **1 ghost cron** that alerted me to a bug in carrybot's own code - **1 long-running agent** that revised its own UX patterns based on feedback - **30 minutes** — the duration of the last workout the bot reported - **7.98 miles** — distance on that Power Zone Pop Ride with Ben Alldis === ## Showdown Thoughts: The Three-Pass Pattern - URL: https://vibescoder.dev/posts/showdown-thoughts-the-three-pass-pattern - Date: 2026-05-19 - Tags: #agents #vibe-coding #model-showdown #building-in-public - Reading time: 6 min read The Round 5 bakeoff produced four implementations. None of them shipped. What shipped was a merge of the best pieces from all four, then a polish pass against real data. Bakeoff → Merge → Polish is a generalizable pattern for any feature where the design space is genuinely unclear. --- [Model Showdown Round 5](/posts/model-showdown-round-5-four-agents-build-the-same-feature) ended with a leaderboard. Sonnet 4.6 won on the rubric. Opus 4.7 placed second. Qwen 3.5 contributed almost nothing structural. That's the measurement story. This is the methodology story — what happened after the scores were revealed. ## The Problem with Picking a Winner The naive workflow after a bakeoff is: pick the best run, merge it to main, ship it. Winner takes all. That's wrong, and Round 5 made it obvious why. The winning run (Sonnet 4.6) had the best overall rubric score. It also had a weaker path validator than Opus 4.7, and its orphan-matching logic would have missed real-world cases that Opus 4.6 caught. The second-place run (Opus 4.7) had the best validator and the cleanest route structure, but the worst data source choice — reading from the build-time filesystem instead of the live GitHub Contents API. No individual run was what I'd ship. Each one had at least one bad call. The bakeoff's real output wasn't a winner. It was a map. When 4 of 4 models made the same design choice, that choice was obviously right. When they diverged — on validation strictness, on data source, on UX for destructive actions — that divergence was the signal. Those were the actual design decisions, the ones worth spending judgment on. ## The Three Passes What emerged from Round 5 is a pattern I've now run twice and would reach for again on any feature where the design space is unclear: **Pass 1 — Bakeoff.** Run N models (I used 4) on the same prompt in isolated sessions. Judge blind, before you know which branch is which. Score against a rubric. The output of this pass isn't any of the N implementations — it's the decision map. You now know which choices are contested and which are obvious. **Pass 2 — Merge.** Write down a merge plan before touching any code: for each contested layer, which run's approach wins and why. Then ask an agent to compose the merged best-of from those inputs. The merge is strictly better than any individual bakeoff run because it draws on information none of the bakeoff contestants had — the scored comparison of all four. For Round 5 the plan looked like this: | Layer | Source | Why | |---|---|---| | Path validator | Opus 4.7 (Run 1) | Only run with 2-segment enforcement + `..` block + non-empty checks | | Three-tier orphan match | Opus 4.6 (Run 2) | Only run that noticed exact-match missed real cases like `day-four` | | Type-narrowed body parsing | Sonnet 4.6 (Run 3) | `typeof body === "object" && "path" in body`, no `as` casts | | GitHub Contents API | Opus 4.6 / Sonnet 4.6 | Live state vs. build-time filesystem snapshot | | Confirm-modal UX | Sonnet 4.6 | Best visual polish in the screenshots | Qwen 3.5 contributed nothing structural to this table. The bakeoff said "skip this one" clearly enough that there was nothing to debate. That's useful information too — knowing which pieces to skip is part of the map. The merge was 13 files changed, +990/-9. One TypeScript error caught and fixed. Build passed first try after that. Opened as a PR with the heritage table in the description so future reviewers can trace any decision back to its source run. **Pass 3 — Polish.** The merged feature went live. I opened it against real production data and spotted four things immediately: truncated directory names with no tooltip, delete buttons invisible on touch devices, no bulk delete UI despite the API supporting `paths: []`, and an orphaned section header that would show with count 0 after the lone orphan was deleted. None of those were predictable before live use. You can't predict friction from a code review — you observe it. The polish pass had to come after the merge because the artifact it was polishing didn't exist until then. The polish was 6 files changed, +265/-54 and about 20 minutes of agent time. ## When to Use It The pattern has a real cost: the bakeoff is N full agent sessions, each producing a complete implementation that you won't ship. For Round 5 that was ~$35 in inference and a few hours of judging. That's cheap insurance when the feature has any of these properties: - **Destructive verbs.** Delete, update, payment, permission change. The cost of getting validation wrong outweighs the cost of the bakeoff. - **Multiple defensible architectures.** Where should validation live? What's the data source? How does auth thread through? When you genuinely don't know the right answer, a bakeoff shows you the option space. - **Hard to change later.** Database schemas. Public API contracts. Anything that will accumulate callers. It's overkill for a 20-line UI tweak or a feature with a single obvious implementation. The signal value of the bakeoff scales with how uncertain you are about the design. ## What I'd Do Differently Three things I'd change for the next run: **Name the contestant chats before pasting the prompt.** All four Round 5 chats showed up as "New Chat" in the Coder API cost summary, which meant 20 minutes of token-volume detective work to figure out which cost belonged to which run. Five seconds of effort would have prevented that. **Capture per-phase stats.** I have clean bakeoff numbers. I don't have separate merge or polish numbers — they're folded into the judging thread. A lightweight wrapper script around each phase would make the next iteration measurable end-to-end. **Write the polish friction items down before fixing them.** I noticed four issues and fixed them in one pass, which collapsed the "observed" list and the "fixed" list into the same moment. Separating them — even by five minutes — would have made the "what does live-review surface" lesson sharper for the writeup. And occasionally you'll notice something that isn't worth fixing. ## By the Numbers - **3 phases**: Bakeoff (4 parallel attempts), Merge (1 informed pass), Polish (1 live-review pass) - **4 implementations** produced in the bakeoff, **0** shipped to main as-is - **3 of 4** bakeoff runs contributed at least one structural piece to the merge - **13 files changed** in the merge pass (+990/-9) - **6 files changed** in the polish pass (+265/-54) - **4 friction items** caught in polish that couldn't have been predicted before live use - **~$35.56** inference cost for the bakeoff phase - **~45 min** bakeoff (parallel), **~30 min** merge, **~20 min** polish === ## Closing the Loop: From Audit to Ten Commits in Four Hours - URL: https://vibescoder.dev/posts/closing-the-loop-from-audit-to-ten-commits - Date: 2026-05-18 - Tags: #agents #security #building-in-public #meta - Reading time: 20 min read Three AI agents audited the blog and produced three different reports. Closing them out was its own job — triage, phasing, verification, and ten commits across two repos with zero build failures. Here's the remediation arc, what shipped, what got deferred, and what the process revealed about working through someone else's audit. --- I asked three AI agents to audit this blog — two Opus variants and a local Qwen 3.5 — in three separate Coder Agents chat sessions, with the same prompt. A few hours later, three reports landed in my inbox. They overlapped on some findings, disagreed on others, and each caught at least one thing the others missed. A combined 90+ findings, of which maybe 15 were actionable, of which exactly one was a *go fix it right now* emergency. This post is about what happened next — the remediation arc from "audit in inbox" to "audit closed," the structure of the plan, and the things the process surfaced that I didn't expect. The arc: triage three reports into one verified plan, then ship the fixes in phases. Four phases, ten commits, two repos, four hours from the audit hitting my inbox to "audit closed." No build failures. One scheduled-publish leak averted, several dependency CVEs closed, one verified injection vector neutralized, one timing side-channel sealed, two pages converted from dynamic to static rendering. And a small pile of items honestly deferred for later. A note up front: the *patterns* of the fixes are described below, but specific exploit recipes, exact identifiers, and the pre-fix code that contained the bugs are not. The whole point of a remediation post is to teach the technique without handing the next attacker a starter kit. The audits' redaction rules apply to writeups about the audit, too — and as the callout near the end of this post explains, I almost forgot that. ## The First Move Was *Not* Fixing Anything The three reports didn't agree. One of the Opus runs flagged an RSS feed XSS via CDATA breakout; the other missed it entirely. Qwen said dependencies were clean; both Opus variants said there were CVEs to close. Qwen graded the codebase a "B+." The Opus reports didn't grade anything; they ranked findings P0-P3. If I'd started shipping fixes for one report's findings without cross-checking, I'd have wasted time on phantom bugs and missed real ones. So the first hour was triage: 1. **Clone both repos myself.** The agents had each cloned them separately during their audits. I needed my own fresh copy so I could verify findings against current `main` of each. 2. **Look at every claimed critical finding firsthand.** Read the source. Confirm the bug exists. Decide whether the framing is right. 3. **Reject what doesn't survive verification.** Several of one report's "critical" findings were really one bug counted multiple times. Deduplicating dropped the combined report from 90+ findings to ~15 actionable items. 4. **Promote what only one model caught.** The RSS CDATA breakout that only one Opus variant flagged turned out to be real and easy to fix. The login timing leak that only Qwen wrote up clearly turned out to have the cleanest remediation prescription. Both went into the plan despite each being a single-source finding. The output of triage was a unified phased plan, written to disk before any code changed. Phases were ordered by: - **Real-world blast radius** (data leak > brute-force > injection-with-trust-boundary > perf) - **Time-to-deploy** (content fixes ship in minutes; engine fixes ship via Vercel after merge) - **Dependencies between fixes** (the rate-limit library needs to exist before the login endpoint can use it; CSP must be set before tightening inline scripts) ``` Phase 0 — Today, 30 min Stop the bleeding Phase 1 — This week, ~1 day 4 verified critical engine fixes Phase 2 — Next week, ~1 day 6 hardening items, batched Phase 3 — Week after, ~1-2 day 5 perf wins Phase 4 — Open-ended Polish ``` That was the plan at noon. By 4 PM, all five phases had shipped. ## Phase 0 the Scheduled-Publish Leak One published draft was scheduled to auto-publish the following morning at 5 AM PT. It contained four leaked identifiers — exactly the four patterns the blog's redaction rules cover: an OAuth Client ID, a hosted tunnel subdomain, a Linux home path, and a Linux username. The original fodder file had all four redacted. Somewhere in the draft-to-post pipeline, the redactions regressed. Two-step fix: 1. **Redact the post.** One commit, four replacements, push to `main`. The scheduled-publish Action picks up the redacted version when `publishAt` fires. 2. **Rotate the leaked secret.** A Client ID alone is a public identifier, but the full credential pair had been read end-to-end by three AI agent sessions in the past few hours. Rotating the secret invalidates anything that may have leaked. The rotation flow itself was uneventful — generate new secret, paste into the appropriate config file, restart the service, delete the old secret. I hit one error on the first try: I'd updated the config but not restarted the service. The service reads env vars at process start; the new secret was on disk but the old one was still in memory. One restart and the reconnect went through clean. That single error during rotation is the post inside the post. **Phase 0 was the only phase that had a hands-on operational dependency** — every other fix shipped through CI without touching infrastructure. Rotation is the one place where "AI agent ships the fix" meets "human owns the deploy target" and both have to coordinate. ## Phase 1 the Four Critical Engine Fixes Four independently-shippable PRs. I batched the dependency bump first because everything else builds on it. ### 1.1 Dependency CVEs Several published advisories matched the pinned versions of `next` and `@anthropic-ai/sdk`. Worth noting: a couple of them were middleware-bypass CVEs, which are directly relevant when admin authorization lives in middleware. A bug that lets requests slip past route matchers is exactly the kind of issue that turns a hardened admin endpoint into a leaky one. The fix was 15 minutes: install the patched versions, typecheck, commit, push. One remaining audit hit — a transitive `postcss` advisory — chains back through a bundled copy. `npm audit fix --force` proposed downgrading the parent framework to a major version from years ago to "resolve" it. That's worse than the bug. Documented the false positive and moved on. Always look at what `audit fix --force` actually proposes before running it. ### 1.2 the RSS CDATA Breakout The RSS feed was dropping raw post bodies into a `` block. If any post body contains the literal sequence that terminates CDATA, the section ends early and the remaining content renders as malformed XML. Result: broken feed for every subscriber the first time a post discusses CDATA, regex examples, or shell heredocs. The fix is the standard CDATA-splitting trick: replace the closing sequence with two adjacent CDATA sections so the XML parser concatenates them transparently. Six lines including the comment. Caught by only one of the two Opus variants in the original audit. A genuine reminder that "same model family" is not the same as "interchangeable for security review." ### 1.3 Login Hardening This was the heaviest single fix and the one I deferred to its own session for that reason. Three independent issues on the login endpoint, none of them showstoppers individually, real defense-in-depth together. **Class of bug 1: no rate limit.** A login endpoint without per-IP throttling is, in theory, an unbounded online brute-force surface. In practice the password is strong enough that brute force is infeasible regardless, but the right answer is to make the math infeasible by *two* compounding factors, not one. Fix: per-IP fixed-window limiter backed by Upstash Redis. The limiter is factored into a shared lib so other endpoints can reuse it. **Class of bug 2: no Origin check.** The session cookie was already `SameSite=strict`, so a cross-site form post couldn't actually use it, but rejecting unauthorized origins at the server is cheaper and louder than relying on browser policy alone. Fix: parse the `Origin` header, compare against `Host`, reject mismatches. **Class of bug 3: timing side-channel in password comparison.** This one is worth a sentence on the pattern, because it's a class of bug that shows up in a lot of homegrown auth code. The naive shape of a "safe" password check is: ``` if input.length != expected.length: return false return constant_time_equal(input, expected) ``` The problem is the early `return false`. A constant-time compare takes the same time regardless of inputs, but the early-return path is much faster than running the compare. An attacker who can measure response timing can therefore distinguish "wrong length" from "right length, wrong content," which leaks the password's length — several bits of entropy gone for free. The fix is to make the comparison run over fixed-size inputs regardless of input length. The common pattern is to hash both sides with a fast cryptographic hash and compare the digests: ``` a = sha256(input) b = sha256(expected) return constant_time_equal(a, b) ``` Both digests are exactly 32 bytes. The comparison takes the same time whether the input is empty, the right length, or 10,000 characters long. No early return, no timing leak. (For password storage specifically, use a real password-hashing function like Argon2 or bcrypt; for comparing two known-trusted strings in a hot path, plain SHA-256 of both sides is fine.) One UX detail: the login page now surfaces the rate-limit response with a "try again in N minutes" message instead of the generic "invalid password." Otherwise the lockout would be indistinguishable from a typo, and the user would keep retrying. ### 1.4 the GitHub Actions Injection Class This was the only finding from the audits that an unauthenticated internet user could reach without going through a login boundary. The pattern in question: a GitHub Actions workflow that interpolates user-controllable event payload fields directly into a `run:` script. GitHub Actions evaluates the `${{ ... }}` expression syntax *before* the script reaches bash. If any of the interpolated fields is attacker-controllable — and a discussion title is — the contents become literal shell. Repository secrets in scope of that job become reachable. The standard fix is straightforward: route every event-payload field through the `env:` block, then reference it in the script as a normal environment variable. Bash sees the value as an opaque string. The pre-evaluation step doesn't get to inject code. There was a second bug in the same workflow: a Slack JSON payload was hand-rolled with shell string interpolation, which would have broken (or been injectable) on any title containing a quote character. Fixed by building the payload with `jq --arg`, which JSON-escapes every interpolated value. Verified the fix against a representative attack-shaped payload before shipping. Output: properly escaped JSON. No shell impact, no JSON break. ## Phase 2 the Hardening Batch Six low-risk independent items, batched into one commit because reverting any one wouldn't affect the others. **CSP (Report-Only) + HSTS.** The Content Security Policy shipped in *Report-Only* mode first. That header surfaces violations in the browser console without breaking the site. After ~a week of clean reports, flip the header name to the enforcing variant. HSTS got a year-long `max-age` with `includeSubDomains` and `preload`. **Analytics rate limit and path allowlist.** The page-view tracking endpoint sanitized the path string but had no upper bound on unique paths. Every unique path minted a new Redis key, so an unbounded number of unique requests could fill the KV store with arbitrary keys. Fixed by (1) per-IP rate limit reusing the lib from 1.3, (2) regex allowlist matching the legitimate route patterns. Rejections return 200 silently to avoid leaking the failure shape to a probe. **JSON-LD `` escape.** A one-line defensive fix. `JSON.stringify` doesn't escape `<` by default. A frontmatter field containing a script-end sequence would otherwise terminate the script tag and inject HTML. Author-controlled today, defense-in-depth tomorrow. **Loom embed origin validation.** The component iframed whatever URL was passed. Now it parses the URL, requires HTTPS, and requires the hostname to match the expected video host. Invalid input renders nothing. **Lazy env reads for module-level secrets.** One handler read `SLACK_SIGNING_SECRET` and a GitHub token at module load, freezing them across warm invocations. A platform-level env-var rotation wouldn't take effect until the next cold start. Moved the reads into the call site so rotation is immediate. **One more content redaction.** A different published post had an internal LAN IP. Replaced with an RFC 5737 documentation address. Lower-stakes than the Phase 0 redactions, but the redaction rules apply. Six items, one commit. CSP violations to be checked over the next week. ## Phase 3 the Static-Rendering Reclamation The single biggest performance win in this whole audit had nothing to do with bundle size or caching. It was a cookie read on every post page. The post page was checking for an admin session in the server component, so it could conditionally render admin controls. That check forces dynamic rendering. Next.js can't pre-render a page that reads cookies because cookies are per-request. `generateStaticParams` was being defeated by `cookies()`. Every reader was paying for the admin-controls feature, none of them were admin, and TTFB was 200-600ms on a cold edge instead of <50ms. The fix is a client island: 1. New auth-probe endpoint that returns 200 or 401, never caches. 2. New hook that calls the endpoint on mount and returns a boolean. 3. New wrapper components that render the real admin controls only when the hook returns true. 4. Delete the cookie read from the post page and the homepage. Always render the island slot. The post page no longer reads cookies. The homepage no longer reads cookies. Both go back to static rendering. The admin controls "pop in" ~50ms after page load for the one user who's authenticated; for everyone else (which is everyone), they're never rendered, never fetched, never paid for. Three other Phase 3 wins, all shipped in the same commit: **Animation library off the fade-in helper.** A fade-in-on-scroll component was importing a full animation library (~35KB gzip) on every page in the site. CSS keyframes do the same animation in zero JavaScript: ```css @keyframes animate-in-up { from { opacity: 0; transform: translateY(20px); } to { opacity: 1; transform: translateY(0); } } .animate-in { animation: animate-in-up 500ms cubic-bezier(0.21, 0.47, 0.32, 0.98) forwards; animation-delay: var(--animate-in-delay, 0s); } @media (prefers-reduced-motion: reduce) { .animate-in { opacity: 1; animation: none; transform: none; } } ``` Per-call-site delay is plumbed through a CSS custom property. A few components still need the animation library for stateful enter/exit transitions; everywhere else, it's gone. **Lazy-load the search modal.** The header was static-importing the search modal, which static-imports the fuzzy-search library and the JSON search index. Now lazy via `next/dynamic`, plus the parent conditionally mounts so the chunk doesn't fetch until the user opens search. Readers who never open search pay zero bytes for the search runtime. **`React.cache` on filesystem reads.** Wrapped the post-listing functions in `React.cache`. Within a single render pass, the homepage and the sitemap now share one filesystem read instead of two. Trivial change, real saving at 30+ posts. ## Phase 4 Honest Scope The original plan had five Phase 4 items. Two shipped. Three were honestly deferred. One was closed as no-longer-relevant. **Shipped: dedupe a utility function.** Three near-identical date-formatting helpers across admin components, each with subtly different behavior. One of them had a latent bug — it always appended `T00:00:00`, which breaks on full ISO datetimes — exactly the bug documented in [Friday Fixes: Mobile First and the Skill That Saved Us](/posts/friday-fixes-mobile-first-and-the-skill-that-saved-us). Consolidated to one shared util. The latent bug got fixed for free. **Shipped: relocated the Qwen audit artifacts.** During its session, Qwen committed four documents into the drafts folder. They're not blog drafts; they're audit artifacts. Moved them into a docs subfolder with a README explaining the provenance. Preserved as historical record without polluting the drafts directory. **Deferred: CSP nonce.** The original plan was to tighten the script-src directive by replacing inline-script allowances with a per-request nonce. But I'd already promised a week of CSP-Report-Only observation before enforcing — and it had been less than a day. Tightening the policy before seeing the violation reports is exactly the kind of premature optimization that breaks production. Deferred until the observation window is real. **Deferred: build-time markdown rendering.** Currently syntax highlighting runs at request time during MDX rendering. Pre-rendering at build would save the cost on every request. But this is a real refactor of the MDX pipeline, not a polish item. **Deferred: optimized images for markdown content.** Plain markdown image syntax doesn't provide width and height, so the renderer falls through to a non-optimized image element for most images. Fixing it properly means probing image dimensions at build time and injecting them into the AST. Real engineering, not polish. **Closed: trim the search index payload.** Flagged in the original audit, and true at the time. The Phase 3 lazy-load of the search modal means the search index is now only fetched when a user opens search. The "ships full content to every reader" framing no longer applies. Closed as mooted. I deliberately under-shipped Phase 4. Forcing the three deferred items into today's batch would have meant shipping refactors before they were ready, tightening a CSP I hadn't observed yet, or pretending that "more commits" was the same as "more done." ## The Mistake I Almost Made Writing This Post Once the audit was closed and the commits were in, I started drafting this post. The first draft included: - All four exact identifiers from the Phase 0 redaction, listed in a tidy bulleted block as "here's what was in the leaked draft." - A literal copy-paste of the exact LAN IP that Phase 2.6 had just redacted. - The exact pre-fix source of the vulnerable password comparison. - A working shell-injection payload, complete with the GitHub Actions context that makes it run. - Ten commit SHAs in a table, each one a clickable diff that points readers straight at the pre-fix state of the bug. - The exact admin password length, in passing, as part of a "this is computationally infeasible" calculation. Every one of those is the same class of mistake that Phase 0 existed to prevent. The audit's whole point was that descriptions of an attack are not the same as identifiers for the target — and there I was, in the writeup celebrating that distinction, conflating them again. It took a separate review pass to catch. Same pattern as the audits themselves: verifying against the rules before publishing matters more than getting the post out fast. The redaction rules in the blog's skill file are not a Phase 0 thing. They're a "any time text leaves this workspace" thing. That includes posts about not leaking things. The published version of this post describes patterns, not targets. Specific identifiers are generalized. Working exploits are described as classes of bugs, not as recipes. Commit SHAs are absent. The math demonstrating that the rate-limit math is fine doesn't disclose a parameter that helps anyone. *[The agent writing this post would like it noted that the agent writing the first draft was, in fact, the same agent. It cheerfully redrafted four of the same leaks it had just fixed, plus added a working shell-injection payload as a bonus. A human review pass caught it. Lessons were learned. Skill files were consulted. The redaction rules now apply to remediation writeups too, in writing, so the next agent can't claim it didn't know.]* ## What This Process Actually Looked Like Five things stand out, in roughly the order they happened: **Triage was the highest-leverage hour.** Reading three reports against ground truth before writing a single line of code is the difference between "shipped what the auditors said" and "shipped what's actually broken." Several of one report's critical findings were the same bug counted multiple times. One report's "cost-amplification DoS" framing depended on a brute-force success that the existing middleware would prevent. The "B+" grade was framing, not a finding. Verifying each high-stakes claim against the codebase took about an hour and changed the plan materially. **Phasing is the second-highest-leverage hour.** A plan that says "fix all the criticals first" sounds rigorous but isn't actionable. Real phasing accounts for *what depends on what*: the rate limiter needs to exist before the login fix can use it; CSP needs a baseline policy before nonces are useful; perf changes shouldn't ship before the auth surface they touch is hardened. The four-phase structure wasn't aesthetic — it was the dependency graph. **Batching versus atomic commits is a real tradeoff.** Phases 0 and 1.3 got their own focused commits because reverting them would matter. Phase 2 got one batched commit with six items because they're independent and reverting one wouldn't affect the others. Phase 3 also batched, for the same reason. I asked the user explicitly before batching Phase 1, which had four items of varying risk — they chose to defer 1.3 to its own session, which was the right call. **Verification beats reasoning.** Every claim from every audit got tested. The login timing fix got a smoke-test against several input shapes. The shell-injection fix got an attack-shaped payload run through the new pipeline. The analytics path allowlist got a dozen test cases including path traversal and oversized inputs. None of this took long; all of it caught at least one mistake I would have shipped otherwise. **Deferring is shipping.** Three Phase 4 items got deferred with specific reasons (observation window not yet complete; real refactor not polish; needs build-time image probing). Writing those reasons down is the work that turns a deferral into a coherent next step. The remediation plan now has a tail — items that will outlive this session — and that's a feature, not a failure to complete. The headline number — ten commits, four hours, zero build failures — is real, but it's not the point. The point is the order: triage before phasing, phasing before commits, verification before shipping, redaction before publishing. --- ## By the Numbers - **3 audit reports** synthesized into 1 phased remediation plan - **15 actionable items** identified after deduplication (down from a combined 90+ across the three reports) - **5 phases** in the plan, all shipped in 1 working session - **10 commits** across 2 repos, **0 build failures** - **4 hours** from the audit hitting my inbox to "audit closed" - **Multiple dependency CVEs** closed across two direct dependencies - **1 scheduled-publish leak** averted before going live - **3 verified injection classes** neutralized (RSS XSS, Actions shell, JSON-LD) - **1 timing side-channel** sealed with a hash-both-sides pattern - **2 pages** moved from dynamic to static rendering - **~35 KB gzip** removed from the initial bundle of most pages by replacing an animation library with CSS keyframes - **3 Phase 4 items** honestly deferred with reasons, not silently dropped - **1 OAuth secret** rotated, with one operational gotcha (need to restart the service for the new env to take effect) - **1 draft of this post** rewritten after a self-audit caught the same class of leak the post was about. Skill file updated. === ## Model Showdown Round 5: Four Agents Build the Same Feature - URL: https://vibescoder.dev/posts/model-showdown-round-5-four-agents-build-the-same-feature - Date: 2026-05-17 - Tags: #model-showdown #agents #vibe-coding - Reading time: 19 min read Four LLM models built the same admin feature in isolated Coder Agents sessions. I judged them blind. The headline result: Sonnet 4.6 beat Opus 4.6 on a coding task. The deeper story is what each model did with the same prompt — and what it took to make the bakeoff fair in the first place. --- I've been running model showdowns on Vibes Coder for a while now. Each round has been a little messier than I wanted — different prompts, accidental context leaks, no clean way to compare cost to quality. This one is the first I'd call a *fair* bakeoff. Two goals going in: 1. **Make the experiment itself rigorous enough that future rounds can build on it** — isolated chat sessions, identical prompts, anonymized branches, blind judging, real token + runtime data pulled from the Coder API. 2. **Compare three flavors of Claude against our local champ.** Opus 4.7, Opus 4.6, and Sonnet 4.6 from Anthropic; Qwen 3.5 35B-A3B running on llama.cpp on the RTX 5090 in the home lab. Four models, same task, four isolated Coder Agents sessions, blind judging. The headline: **Sonnet 4.6 beat Opus 4.6 on a coding task.** Not by much (4.48 vs 4.36) but cleanly, on its own merits, with no asterisks. And once I pulled real token and runtime data from Coder's chat-cost API, a second headline emerged: **weighted by cost, Sonnet's win becomes decisive — about 10x cheaper per rubric point than either Opus model.** A third wrinkle: Opus 4.7 finished the task in 9.2 minutes, the fastest of the three Claude runs. It won the rubric without burning the most time. The deeper story is what each model did with the same prompt, and what it took to make the bakeoff *fair* in the first place — which turned out to be more work than the bakeoff itself. ## The Setup The contestants: | Run | Model | Where it runs | |---|---|---| | 1 | Claude Opus 4.7 | Cloud, via Coder Agents | | 2 | Claude Sonnet 4.6 | Cloud, via Coder Agents | | 3 | Claude Opus 4.6 | Cloud, via Coder Agents | | 4 | Qwen 3.5 35B-A3B | Local, llama.cpp on the RTX 5090, via Coder Agents | The mapping was private. Branches were named `run-1` through `run-4`. I judged the four branches blind against a fixed rubric, then revealed the identities. The task: build image management into the vibescoder.dev admin dashboard. The current `/admin` page has a Settings card that's a placeholder. The spec asked for an Images card (or a replacement) that lists the post-image directories under `public/images/`, detects orphans (directories with no matching post), provides a screenshot view, and adds an API route to delete a directory. It's not a huge feature, but it has enough surface area to differentiate models: filesystem traversal, slug matching, path validation, an API contract with a destructive verb, a UI page, and at least one judgment call (what counts as an "orphan?"). ## The Fairness Story Before launching anything, three things needed fixing. None of them are interesting on their own. Together they're the operational lesson of this post: a bakeoff isn't fair by default. ### Fix 1 Node 18 vs Node 20 The workspace image is built on Ubuntu 24.04. Ubuntu 24.04's `apt` Node is 18.19. Next.js 16 — what the blog engine ships on — requires Node 20+. Any agent that ran `apt install nodejs` would silently break its own build. The fix was a Dockerfile change in the `coder-templates` repo: install Node 20 from NodeSource at image build time, pin npm, verify `node -v` reports 20.x in the smoke test. After that, `node -v` in a fresh workspace prints `v20.20.2` and nothing the agents do (short of `nvm` shenanigans) changes that. ### Fix 2 the System Instructions Were Lying The chat system prompt — injected at the top of every Coder Agents session — said Node was not pre-installed and told agents to install it themselves. Correct on the previous image; actively misleading after Fix 1. An agent following the instructions would `apt install nodejs`, get Node 18, downgrade the runtime, and break the build. I rewrote the instructions to say Node 20 is pre-installed, do not reinstall, use `nvm` if you need a different version. Boring change. Huge impact on whether the bakeoff produces meaningful signal. ### Fix 3 Prompt Poisoning The first draft of the bakeoff prompt told each agent to create a branch named after the model running the session — `bakeoff-opus47`, `bakeoff-sonnet46`, and so on. A sharp catch from the human side: that wording **leaks competition signaling into the prompt**. An agent that sees "you are opus47" or even "this is a bakeoff" can adjust behavior in ways that aren't comparable. The experiment stops measuring "what does this model do with the prompt" and starts measuring "what does this model do when it knows it's on stage." Fix: replace model names with neutral ordinals. Branches became `run-1` through `run-4`. The prompt made no reference to other runs, scoring, or any comparison. Each agent thought it was building a feature, not auditioning. Three small fixes. Together they're the operational lesson: **fairness in a model bakeoff requires more setup than the bakeoff itself.** ## The Prompt The prompt was identical for all four runs, save for the run number in the branch name. Verbatim, with one path generalized: ```markdown You are working in the vibescoder.dev blog engine repo. Branch: run-N. Baseline commit is at the tip of main. Goal: add image management to /admin. Requirements: - List the directories under public/images/ (each directory corresponds to one post and contains its images). - For each directory, report: name, file count, total size on disk, and whether it matches a published or draft post (by slug). - Surface "orphaned" directories — directories that do not match any post — so I can clean them up. - Provide a way to view the images in a directory (thumbnails or list). - Provide an API route DELETE /api/admin/images that removes a directory by path. The route must validate input. - Update the /admin landing page so the new feature is reachable. You may keep the Settings placeholder card or replace it; either is fine. - Add a screenshot of the new page to the PR description (use the Playwright MCP). - Run `npm run build` before committing. Do not push commits that fail the build. - Commit in logical chunks. Push the branch when done. ``` That's it. No mention of competing runs. No scoring rubric. No model identification. Just a feature spec and a quality bar. ## The Four Implementations All four runs built it. All four passed `npm run build` against a shared engine baseline on Node 20.20.2. All four pushed their branches. Then the differences started showing up. ### Run 1 8 New Files 631+/9- Replaced the Settings placeholder with an Images card on `/admin`. Added a dedicated `/admin/images` page that lists directories server-side, plus a client-side modal that renders a grid of thumbnails when you click into a directory. Three screenshots in the PR description — admin landing, images list, modal open with orphan-flagged styling. The standout was the API route. Run 1 wrote a real path validator — `isValidImageRepoPath` — that required exactly two path segments under `public/images/`, rejected `..`, and ran *before* the filesystem call. The route returned distinct status codes for distinct failure modes: 400 for bad input, 404 for missing, 403 for paths that resolve outside the allowed root, 200 for success. It's not glamorous code. It's just the version where someone thought about the failure modes before writing the success path. ![Run 1 admin/images page](/images/model-showdown-round-5-four-agents-build-the-same-feature/run-1-opus47.png) *Run 1's `/admin/images` page. Directory cards, orphan-flagged styling, and a tight path-validated delete API behind the trash icons.* ### Run 2 6 New Files 687+/7- Kept the Settings card. Added an Images card next to it on `/admin`. The /admin/images page was the cleanest of the four — tight TypeScript, no `as` casts in the API route, proper type narrowing (`typeof body === "object" && "path" in body`) instead of forcing the compiler to trust it. The UI had the most visual polish: directory cards with file counts as a badge, hover states that matched the rest of the admin surface, a confirmation modal on delete that quoted the directory name back at you. Path validation was decent but not as rigorous as Run 1 — `startsWith("public/images/")` plus a `..` block, no segment-count check. Enough to stop the obvious cases. Not airtight against creative inputs. Two screenshots. Shipped a polished v1 and stopped. ![Run 2 admin/images page](/images/model-showdown-round-5-four-agents-build-the-same-feature/run-2-opus46.png) *Run 2 kept the Settings card and put Images next to it. Cleanest TypeScript of the four; smallest screenshot artifact.* ### Run 3 6 New Files 595+/0- Replaced the Settings placeholder. The /admin/images page started as a server component, then mid-task switched to a client-fetched implementation when Run 3 hit a dev-server timeout on the first integration test. That mid-stream pivot showed up cleanly in the commit history — `feat: add admin/images server-rendered`, then two commits later, `refactor: move admin/images to client fetch (dev server hangs on FS scan)`. Path validation matched Run 2's. The thing that made Run 3 interesting was the orphan-detection arc. The spec said "match directory name against post slugs to find orphans." Three of the four models took that literally — list directories, list slugs, set-difference, report what's left. Run 3 did that first, reported 8 orphaned directories, then *checked the result against reality*. Looked at the actual file tree and noticed that one of the "orphaned" directories was `day-four/`, and there's a published post with the slug `day-four-rss-analytics-syndication-and-loom`. The directory isn't orphaned. It belongs to that post. The matching logic was wrong. Run 3 iterated three times: exact match → prefix match (does any slug start with this directory name?) → content-reference match (does any post body reference an image in this directory?). After the third pass, the orphan count went from 8 to 1 — and the one remaining was an actual orphan I'd been meaning to delete for weeks. Small thing in the diff. Big thing in engineering judgment. The other three models reported false-positive orphans with high confidence. Run 3 noticed its own answer was wrong and kept working. ![Run 3 admin/images page](/images/model-showdown-round-5-four-agents-build-the-same-feature/run-3-sonnet46.png) *Run 3's screenshot — the largest and most polished of the four. The orphan count in the header reads 1 instead of 8 because the matching logic had been corrected mid-task.* ### Run 4 7 New Files 607+/0- Kept the Settings card, added an Images card. The /admin/images page worked. Build passed. The directory listing rendered correctly. Two structural issues. First, the codebase ended up with two utility libraries — `images.ts` and `imageUtils.ts` — with overlapping responsibilities. The first pass put filesystem helpers in `images.ts`, which got imported into a client component, which pulled `fs` into the client bundle and broke the build. The fix added `imageUtils.ts` for client-safe helpers and re-imported. The dead code in `images.ts` was never cleaned up. Second, the screenshot. Run 4 ran `playwright screenshot`, hit the same missing-system-libraries failure the other three runs hit (`libnspr4`, `libpango-1.0-0`, the headless Chromium kit), `sudo apt install`-ed the dependencies — and then never retried the screenshot. Instead the PR description got a 184-line *markdown description* of what the page would look like, in lieu of a PNG. The deps were installed. The retry never fired. Path validation was the weakest of the four — `startsWith` on the user-supplied path, no normalization, no `..` block. The class of weakness is that a path that looks like it's under `public/images/` can still resolve elsewhere when the OS interprets it. I'm not going to spell out the exact bypass; the point is that a one-line `startsWith` check is not a path validator, and Run 4 shipped one. *Run 4's "screenshot" is a 184-line markdown file. The opening:* > **Page Description: `/admin/images`** > > **Overall Layout** > > The `/admin/images` page displays a dashboard-style view of all image directories with a neon brutalist design consistent with the existing admin theme. > > **Header Section** > > At the top: > - **Title**: `// Images` in monospace font with primary color (cyan/teal) > - **Stats bar** showing: > - Total directories count > - Total files count > - Total size in human-readable format (MB/GB) > - Orphaned count (in warning yellow/orange color, only shown if > 0) > > *…and 165 more lines of design notes.* ## Blind Scoring Rubric, weights, and scores: | Dimension | Weight | Run 1 | Run 2 | Run 3 | Run 4 | |---|---|---|---|---|---| | Correctness | 25% | 5.0 | 5.0 | 5.0 | 4.0 | | Design | 15% | 4.5 | 5.0 | 4.0 | 3.0 | | Code quality | 20% | 5.0 | 5.0 | 4.5 | 2.5 | | Engineering judgment | 15% | 4.5 | 4.0 | 5.0 | 2.5 | | Scope discipline | 10% | 4.5 | 4.5 | 4.0 | 3.5 | | Commit hygiene | 10% | 4.5 | 4.0 | 4.5 | 3.5 | | Surprise | 5% | 4.0 | 3.5 | 5.0 | 2.5 | | **Weighted total** | | **4.68** | **4.48** | **4.36** | **3.18** | Scoring notes I wrote during the blind pass, before the reveal: - **Run 1** — "Most defensive of the four. The path validator is the kind of code I'd want to ship to production. Loses half a design point for being slightly less visually polished than Run 2." - **Run 2** — "Tightest TypeScript I've seen this week. Visual polish is the best of the four. Path validation is fine but not paranoid. Stopped at v1 — didn't iterate, didn't second-guess. Probably Sonnet." - **Run 3** — "Mid-task architecture pivot, three iterations on orphan detection, the only run that produced an honest orphan count. Took the longest. Most thoughtful. Probably Opus 4.6." - **Run 4** — "Two overlapping libraries, dead code left behind, weak path validation, fell back to a markdown description instead of a real screenshot. The dependency install was right there. The retry never came. Probably Qwen." Two guesses right (Run 1 = Opus 4.7, Run 4 = Qwen). Two guesses swapped. Run 2 was Sonnet 4.6. Run 3 was Opus 4.6. I had them reversed — but I had the *behavior* right. I thought "polished, decisive, stopped at v1" was Sonnet, and it was. I thought "iterated three times until the answer was honest" was Opus, and it was. The guesses were wrong about which Opus, not about the disposition. ## The Reveal | Rank | Model | Score | Headline | |---|---|---|---| | 1 | Opus 4.7 | 4.68 | Strongest path validator, multi-status DELETE API, three screenshots | | 2 | Sonnet 4.6 | 4.48 | Tightest TypeScript, best visual polish, fastest to "done" | | 3 | Opus 4.6 | 4.36 | Only model that noticed the slug-prefix problem and iterated until orphan detection was honest | | 4 | Qwen 3.5 35B-A3B | 3.18 | Missing screenshot, weakest path validation, architectural churn | ## What Surprised Me **Sonnet beat Opus 4.6.** I didn't expect that. On previous bakeoffs Opus has been the model that goes deeper. Here, Sonnet's tighter implementation and faster decisive shipping outscored Opus's iteration. Two different success modes: - **Sonnet's mode**: get to a clean v1 fast, polish what's there, stop. Trust the spec. - **Opus 4.6's mode**: ship a first pass, look at the output, notice when it disagrees with reality, iterate. Neither is wrong. If the spec is precise and "ship the feature" is the success criterion, Sonnet's mode wins. If the spec is approximate and "produce a correct answer" is the success criterion, Opus's mode wins. On this task, Sonnet was polished enough that Opus's iteration premium didn't make up the gap. **Opus 4.6's slug-prefix insight is the engineering moment of the bakeoff.** Three models took the spec literally and produced false-positive orphans. One model checked its work, noticed the discrepancy, and kept going until the answer was honest. The cost was time — Opus 4.6 took **28.1 minutes, 3x longer than Opus 4.7's 9.2 minutes**, and 146 messages versus Opus 4.7's 84. The benefit was the only correct orphan count in the bunch. That's the trade-off, and on a real codebase I'd take it every time — but it's worth being honest that the iteration premium showed up in the bill as well as the clock. **Qwen failed roughly where predicted.** Pre-launch I'd written down four likely failure modes: skip orphan detection, weak design system match, miss the screenshot, forget to push. Three of those landed at least partially — Qwen *did* implement orphan detection, but did it naively, which is how the predicted weakness actually manifested; the design fit was rough; the screenshot was missed; the push went fine. The pattern wasn't where I expected, though. Qwen didn't fail at the planning level. It failed at the *retry* level. Every concrete step was reasonable. What was missing was the loop — retry the screenshot after installing the deps, clean up the dead code after the refactor, question whether two utility libraries were one too many. That's the agentic gap, and it's narrower than a year ago but still visible. **The screenshot step was the cleanest differentiator.** Same task, same workspace template, same Playwright MCP, same headless Chromium dependency stack. Three models installed the missing libraries and got real PNGs. One model installed the libraries and produced a markdown description instead. Same workspace, same tools, completely different outcomes. If you wanted to test agentic loop-closing in a single observable step, this would be it. **Two of four replaced the Settings placeholder; two kept it.** The spec allowed either. Both Opus runs replaced it; Sonnet and Qwen kept it alongside the new Images card. Not a quality signal — a reading of the spec — but interesting that the two Opus variants made the same call independently, and the two non-Opus models made the same opposite call. ## What the Bill Says The rubric scores were one half of the bakeoff. The other half lives in Coder's chat-cost API. Coder's OSS deployment exposes `/api/experimental/chats/cost/{user}/summary` — an experimental endpoint that returns per-chat input tokens, output tokens, cache reads, cache writes, message counts, and runtime. (Coder Premium has a fuller "AI Bridge" cost product; on OSS, the experimental chats endpoint is the equivalent and gives you everything you need to do this analysis.) Querying per-chat instead of per-model matters. My first pass aggregated by model and the Opus 4.7 totals looked enormous — until I realized the rollup had silently combined two chats running on the same model: this judging thread plus the actual Opus 4.7 contestant run. After identifying the contestant by its chat ID prefix (`2c4e8f98`) and isolating to that session, the numbers got honest. **The lesson: for clean bakeoff stats, query at the chat-id level, not by model.** Two sessions on the same model will silently pool. The finding the dashboard didn't surface: Opus 4.7 won the rubric (4.68), but weighted by cost-per-rubric-point at Anthropic list prices, Sonnet 4.6 wins decisively. **$0.37 per rubric point for Sonnet vs $3.87 for Opus 4.7 and $3.63 for Opus 4.6.** Sonnet was the only economically sensible choice for a task this size. The Qwen line is the other one to sit with. Qwen finished in **6.4 minutes** — faster than every Claude run — and produced the lowest-scoring artifact. Locally hosted inference is genuinely faster per turn (~4 seconds vs 6–13 seconds for the Claude runs); the shortfall was per-turn productivity, not latency. A longer Qwen run might have closed the gap. A 6-minute Qwen run did not. One honest caveat on the cost numbers: this OSS Coder deployment doesn't have model cost config set, so the dashboard reported $0 across the board. The costs in the table below are list-price estimates calculated from the raw token counts. Production Anthropic billing would match closely modulo any rate plan. | Model | Input | Output | Cache R | Cache W | Runtime | Messages | Est Cost | |---|---:|---:|---:|---:|---:|---:|---:| | Opus 4.7 | 99 | 32,114 | 4,772,142 | 454,581 | 9.2 min | 84 | $18.09 | | Opus 4.6 | 14,671 | 45,137 | 6,493,552 | 132,707 | 28.1 min | 146 | $15.83 | | Sonnet 4.6 | 110 | 25,935 | 3,097,881 | 85,057 | 15.2 min | 106 | $1.64 | | Qwen 3.5 35B-A3B | 55,615 | 23,743 | 4,253,874 | 0 | 6.4 min | 88 | $0.00 | Cost-efficiency, $/rubric point (lower is better): Opus 4.7 **$3.87**, Opus 4.6 **$3.63**, Sonnet 4.6 **$0.37**, Qwen **$0.00**. Pricing: Opus $15/M in, $75/M out, $1.50/M cache read, $18.75/M cache write; Sonnet $3/M in, $15/M out, $0.30/M cache read, $3.75/M cache write; Qwen runs locally on the RTX 5090. ## By the Numbers - **4 models** tested in isolated Coder Agents sessions — Opus 4.7, Opus 4.6, Sonnet 4.6, Qwen 3.5 35B-A3B - **4 branches** pushed (`feature/image-management-run-1` through `run-4`); **0 PRs** opened to preserve isolation - **4/4 builds passed** `npm run build` on Node 20.20.2 against the engine baseline - **3/4 screenshots succeeded** — Qwen installed the headless-browser deps but never retried the capture; fell back to a markdown description of the page - **1/4 models produced an honest orphan count** (Opus 4.6, 1 real orphan); the other three reported **8 false-positive orphans** from naive slug matching - **2/4 blind identity guesses** correct (Opus 4.7, Qwen); the two Claude behavioral reads were right but attributed to the wrong Opus - **3 pre-launch fairness fixes** shipped before the bakeoff could run — Node 20 in the workspace image, a corrected system-instructions block, and the prompt-poisoning catch that anonymized the branches - **2 repos** touched to ship the fairness work — `coder-templates` (Dockerfile + system instructions) and the bakeoff prompt iteration in the planning thread - **~640 lines** of code added per implementation on average (range 595–687); roughly **6–8 new files** per branch - **2 new routes** per implementation — an admin page and an API route with a destructive verb - **84 / 146 / 106 / 88 messages** sent in the four chat sessions (Opus 4.7 / Opus 4.6 / Sonnet 4.6 / Qwen); **9.2 / 28.1 / 15.2 / 6.4 minutes** of wall-clock runtime - **~$35.56 total bakeoff cost** at Anthropic list prices — about a fancy dinner for four independent attempts at a real feature with judgable artifacts - **$0.37 vs $3.87 per rubric point** — Sonnet 4.6's cost-efficiency vs Opus 4.7's. Ten times cheaper for slightly higher quality. - **1 result I didn't expect**: Sonnet beat Opus 4.6 on rubric (4.48 vs 4.36) and beat *both* Opus models by 10x on cost-efficiency - **1 follow-up filed** in `content/TODO.md`: build `scripts/bakeoff-stats.sh` so the next round's per-chat aggregation is one command instead of a manual jq exercise === ## Installing OpenClaw on the Homelab - URL: https://vibescoder.dev/posts/installing-openclaw-on-the-homelab - Date: 2026-05-16 - Tags: #homelab #agents - Reading time: 11 min read From curl to working Discord bot in one afternoon — with a local LLM on the RTX 5090. Every gotcha, every config mistake, and the one setting that silently ate every server channel reply for hours. --- I've been running Coder workspaces on my homelab for a while — Qwen3.5-35B on llama.cpp, RTX 5090, the whole stack. But the AI assistants were all inside terminal sessions. I wanted something I could message from my phone, from Discord, from anywhere. Something that talks to the local LLM on my own hardware and doesn't phone home to anyone's cloud. [OpenClaw](https://github.com/openclaw/openclaw) is that thing. It's an open-source personal AI assistant with 367K GitHub stars, a plugin ecosystem, and connectors for every chat platform you can name. The pitch: "Your own personal AI assistant. Any OS. Any Platform." Here's how I got it running on my Linux workstation, wired to a local Qwen3.5-35B via llama.cpp, talking through Discord. It took an afternoon. It should have taken 30 minutes. The difference was five config mistakes that produced zero useful error messages. ## The Hardware | Resource | Spec | |---|---| | CPU | AMD Ryzen 9 9950X3D — 16 cores / 32 threads | | RAM | 64 GB | | GPU | NVIDIA RTX 5090 — 32 GB VRAM | | OS | Ubuntu 24.04 | | LLM | Qwen3.5-35B-A3B via llama.cpp on port 8080 | | Embeddings | nomic-embed-text-v1.5 via llama.cpp on port 8084 | The LLM runs entirely on the GPU. No RAM impact on anything else. ## 1 Installation One Curl ```bash curl -fsSL https://openclaw.ai/install.sh | bash ``` That's it. The script detects Ubuntu, installs Node if needed, drops the `openclaw` binary, and launches an onboarding wizard. The whole thing took about 90 seconds. ## 2 Pointing at the Local LLM The wizard asks for a model provider. The list has Anthropic, Google, OpenAI, and two dozen cloud services. Scroll past all of them and pick **Custom Provider**. ![OpenClaw wizard showing the model/auth provider selection screen](/images/installing-openclaw-on-the-homelab/01-wizard-model-provider.png) The wizard needs three things: - **Base URL**: `http://localhost:8080/v1` - **API key**: Anything — llama-server doesn't check it, but the field can't be empty - **Model ID**: It auto-detects from the `/v1/models` endpoint I had two llama-server instances running and had to figure out which was which: ```bash curl -s http://localhost:8080/v1/models | python3 -c "import sys,json; [print(m['id']) for m in json.load(sys.stdin)['data']]" # Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf curl -s http://localhost:8084/v1/models | python3 -c "import sys,json; [print(m['id']) for m in json.load(sys.stdin)['data']]" # nomic-embed-text-v1.5.f16.gguf ``` Port 8080 is the chat model. Port 8084 is embeddings. OpenClaw wants the chat model. The wizard verified the connection and asked for an **Endpoint ID** — just a label for the config. I accepted the default `custom-localhost-8080`. ![OpenClaw wizard showing the endpoint configuration](/images/installing-openclaw-on-the-homelab/02-wizard-endpoint-id.png) **Use localhost, not your Tailscale IP.** OpenClaw runs on the same machine as llama-server. Routing through Tailscale adds latency and creates a dependency on the Tailscale daemon being up for purely local traffic. ## 3 Setting up the Discord Bot The wizard asks which chat channel to connect. I picked **Discord** — it's the most popular OpenClaw channel, which means the most community support and troubleshooting threads. Creating the Discord bot takes five steps in the [Developer Portal](https://discord.com/developers/applications): **Step 1: Create the application.** Click "Build a Bot" on the welcome screen, then "New Application." I named mine OpenClaw. ![Discord Developer Portal welcome screen](/images/installing-openclaw-on-the-homelab/03-discord-developer-portal.png) **Step 2: Get the bot token.** Go to the Bot tab, click "Reset Token," copy the token. Paste it into the OpenClaw wizard when prompted. **Step 3: Enable Message Content Intent.** Same Bot tab, scroll to "Privileged Gateway Intents," toggle on **Message Content Intent**. Without this, the bot can see that messages exist but can't read what they say. **Step 4: Invite the bot to your server.** The OAuth2 URL Generator in the Developer Portal can be finicky. I skipped it and built the invite URL manually: ``` https://discord.com/oauth2/authorize?client_id=YOUR_APP_ID&scope=bot&permissions=66560 ``` Permission `66560` grants Send Messages + Read Message History. Replace `YOUR_APP_ID` with the Application ID from the General Information tab. ![Discord OAuth2 page showing scopes selection](/images/installing-openclaw-on-the-homelab/05-discord-oauth2-scopes.png) **Step 5: Create a server.** I didn't have a Discord server. The invite page showed "No items to show." Had to go back to Discord, click the `+` button in the sidebar, create a new server called HomeLabOpenClaw, then revisit the invite URL. ![Discord bot invite page showing no servers](/images/installing-openclaw-on-the-homelab/06-discord-bot-invite-no-servers.png) ## 4 Finishing the Wizard Back in the terminal, the wizard asked a few more questions: - **Channel access**: I picked "Open (allow all channels)" — it's my personal server, no reason to maintain an allowlist - **Search provider**: DuckDuckGo — free, no API key, good enough for a first run - **Skills**: Said yes, let it enable the 10 eligible ones - **Hooks**: Skipped — not essential for getting started - **Hatch**: "Hatch in Terminal" — starts the gateway right there so you can see the logs ![OpenClaw wizard hatch screen](/images/installing-openclaw-on-the-homelab/11-wizard-hatch.png) The gateway started, the Discord plugin connected, and the bot appeared online in my server. ## 5 the Pairing Dance I messaged the bot and got: "OpenClaw: access not configured." With a pairing code. ![Discord DM showing pairing code from carrybot](/images/installing-openclaw-on-the-homelab/12-discord-dm-pairing.png) OpenClaw's DM policy defaults to `pairing` — unknown senders get a code instead of a response. You approve them from the terminal: ```bash openclaw pairing approve discord YOUR_PAIRING_CODE ``` After that, DMs worked perfectly. The bot responded, the 5090 spun up, responses came back. Great. Then I tried a server channel and everything broke. ## 6 the Silent Channel Problem For the next two hours, this was my experience: I'd `@carrybot` in a server channel, the bot would react with an emoji, show "typing..." for a few seconds, and then... nothing. No response. No error in Discord. The 5090 was clearly working — I could hear the fans. ![Discord channel showing @carrybot messages with no responses](/images/installing-openclaw-on-the-homelab/13-discord-channel-not-responding.png) DMs worked. Channels didn't. Here's every wrong turn I took and the actual fix. ### Wrong Turn 1 It's a Permissions Issue I checked the bot's Discord role permissions. Almost nothing was toggled on. I enabled Send Messages, Read Message History, View Channels. Restarted the gateway. Still nothing. **Verdict**: The permissions were wrong and needed fixing, but they weren't the root cause. The bot was already *generating* responses — it just wasn't *posting* them. ### Wrong Turn 2 It's a Context Window Issue The bot occasionally showed this error: ![Context limit exceeded error in Discord](/images/installing-openclaw-on-the-homelab/15-context-limit-exceeded.png) The OpenClaw wizard had set `contextWindow: 4000` and `maxTokens: 4096` in the model config. My llama-server has a 131K context window. The wizard didn't auto-detect this from the Custom Provider endpoint. I edited `~/.openclaw/openclaw.json` and changed: ```json { "contextWindow": 131072, "maxTokens": 81920, "reasoning": true } ``` - `contextWindow: 131072` matches llama-server's `--ctx-size 131072` - `maxTokens: 81920` matches llama-server's `-n 81920` (max output tokens) - `reasoning: true` because Qwen3.5 runs with `--reasoning-budget 8192` This fixed the context errors, but channels still didn't work. ### Wrong Turn 3 It's the Memory Plugin The logs showed `tool:memory_search:started` hanging indefinitely. Qwen3.5 kept trying to call a `memory_search` tool before responding, and it never completed. ```bash openclaw config set plugins.entries.memory-core.enabled false openclaw gateway restart ``` This fixed the tool-call hangs in DMs. Channels still didn't work. ### Wrong Turn 4 It's a Mention Detection Issue Early on, I was typing `@OpenClaw` in channels. The logs showed `reason: "no-mention"` — the bot is mention-gated in group chats and I was mentioning the wrong name. The Discord application is "OpenClaw" but the bot username is "carrybot" (I renamed it in the Developer Portal). **You have to use the actual Discord mention** — type `@` and select the bot from the autocomplete. Typing `@carrybot` as plain text doesn't create a real mention. This got the bot to actually *process* channel messages. But it still wasn't responding. ### The Actual Fix `Visiblereplies` After two hours, I found it. During the wizard's `openclaw doctor` step, it had auto-applied a config change: ```json "messages": { "groupChat": { "visibleReplies": "message_tool" } } ``` This tells OpenClaw to use the `message` tool for posting replies in group chats / server channels. But the `message` tool wasn't available — I'd disabled `memory-core` and the tool policy didn't include it. So the bot would generate a perfect response, try to send it via a tool that doesn't exist, and silently fail. The fix: ```bash openclaw config set messages.groupChat.visibleReplies "automatic" openclaw gateway restart ``` One config key. Two hours of debugging. Zero error messages in the logs. ## 7 the Working Config Here's the final `~/.openclaw/openclaw.json` model section that actually works: ```json { "models": { "providers": { "qwen-local": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "sk-none", "models": [{ "id": "Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf", "contextWindow": 131072, "maxTokens": 81920, "reasoning": true }] } } } } ``` And the critical non-obvious settings: ```json { "messages": { "groupChat": { "visibleReplies": "automatic" } }, "plugins": { "entries": { "memory-core": { "enabled": false } } }, "agents": { "defaults": { "compaction": { "reserveTokensFloor": 40000 } } } } ``` ## 8 Making It Stick Install the systemd service so the gateway survives reboots: ```bash openclaw gateway install ``` Set yourself as the command owner so you can run privileged commands: ```bash openclaw config set commands.ownerAllowFrom '["discord:YOUR_DISCORD_USER_ID"]' ``` Verify everything: ```bash openclaw --version # confirm CLI openclaw doctor # check for config issues openclaw gateway status # verify gateway is running ``` ## What I Learned **The wizard's defaults are for cloud providers, not local LLMs.** `contextWindow: 4000` is a safe default for API providers that charge per token. It's a crippling default for a local model with 131K context. If you're running a Custom Provider, you *must* manually set `contextWindow` and `maxTokens` to match your server's actual limits. **`visibleReplies: "message_tool"` is a trap.** The doctor command auto-applies this "recommended" setting, but it depends on the message tool being available. If you're running a stripped-down config without all the default tools, your bot will silently swallow every group chat reply. The symptom is *perfect* — the bot reacts, types, generates a response (you can verify in the session files), and then just... doesn't post it. No error. No log line. Nothing. **Discord bot setup has more steps than it should.** Between the Developer Portal, the OAuth2 scopes, the Privileged Gateway Intents, the server creation, the role permissions, and the correct mention format — there are at least six places where a single missed toggle produces a silent failure. Document every step. Check every toggle. **Session files are your debugging lifeline.** When the logs show nothing, check `~/.openclaw/agents/main/sessions/*.jsonl`. The session file showed me the bot was generating perfect responses that were never delivered. Without that, I would have assumed the LLM was broken. **Start with DMs, graduate to channels.** DMs have a simpler code path — no mention detection, no group chat reply policy, no channel permissions. Get DMs working first, then debug channels as a separate problem. --- ## Files Changed **On the workstation:** - `~/.openclaw/openclaw.json` — model config, context window, reply policy, plugin settings, owner config **Discord:** - Created Discord application "OpenClaw" with bot user "carrybot" - Created Discord server "HomeLabOpenClaw" - Enabled Message Content Intent, configured role permissions **Systemd:** - `openclaw-gateway.service` — installed via `openclaw gateway install` ## What's Next The bot works, but it's running Qwen3.5-35B with `memory-core` disabled and no skills beyond the basics. Next steps: 1. **Re-enable memory.** Figure out why `memory_search` hangs with Qwen3.5's tool call format and fix it — memory is one of OpenClaw's killer features. 2. **Add skills.** 43 skills were blocked by missing requirements. Install the useful ones — `session-logs`, `nano-pdf`, `video-frames`. 3. **Try a different local model.** Qwen3.5 works but its tool calling may not be fully compatible with OpenClaw's expected format. Worth testing Gemma 4 or another model with native tool support. 4. **Wire up Tailscale access.** The gateway listens on localhost:18789. Exposing it on the tailnet means I can hit the dashboard from any device without a Cloudflare tunnel. ## By the Numbers - **1 curl command** to install OpenClaw - **131,072 tokens** — the context window the wizard set to 4,000 - **81,920 tokens** — max output, matching llama-server's `-n` flag - **2 hours** debugging silent channel failures - **1 config key** (`visibleReplies: "automatic"`) that fixed everything - **6 Discord setup steps** where a missed toggle means silent failure - **0 cloud dependencies** — fully local LLM, self-hosted gateway - **~500 MB** RAM footprint for the OpenClaw gateway (Node.js process) - **18 screenshots** taken during the debug session - **4 sensitive screenshots** deleted (contained tokens/credentials) - **0 useful error messages** for the `visibleReplies` bug === ## Friday Fixes: AEO ≠ Agent-Ready - URL: https://vibescoder.dev/posts/friday-fixes-aeo-does-not-equal-agent-ready - Date: 2026-05-15 - Tags: #aeo #agents #cloudflare #building-in-public #meta - Reading time: 15 min read Our AEO audit gave vibescoder.dev a clean bill of health. Cloudflare's isitagentready.com gave it a 25 out of 100. Both audits were right — they were measuring two different competencies. Here's the side-by-side, what each one caught, and the two genuine gaps we shipped fixes for — taking the score from 25 to 33 (and on track for 39 after the next scan). --- A couple of weeks back I published [an SEO and AEO audit](/posts/your-ai-strategy-has-a-blind-spot) of this site. It found a Cloudflare setting silently blocking every AI crawler, broken RSS links, a missing sitemap, no structured data, no `llms.txt` — twenty issues, all shipped in one session. The audit felt thorough. The post landed well. Then I ran the same site through Cloudflare's new [isitagentready.com](https://isitagentready.com/) checker. It gave vibescoder.dev a **25**. Level 1. "Basic Web Presence." ![Cloudflare's isitagentready.com scorecard for vibescoder.dev: overall score 25, Level 1 Basic Web Presence, with category breakdowns showing Discoverability 67%, Content 0%, Bot Access Control 50%, API/Auth/MCP/Skill Discovery 0%, Commerce not checked](/images/friday-fixes-aeo-does-not-equal-agent-ready/cf-scorecard.png) My first reaction was defensiveness — *we just did this*. My second reaction was suspicion of the tool — *Cloudflare has a commercial interest in selling agent-readiness as a category*. My third reaction was to do what I should have done first: read the report carefully and see what it was actually measuring. The answer is interesting enough that it changes how I think about the AEO category itself. **AEO and "agent-ready" are two emerging competencies. They are related. They are not the same.** And right now, almost nobody is doing both. ## The Two Audits Were Asking Different Questions The AEO audit I ran a couple of weeks back asked: > *Can ChatGPT, Perplexity, Google AI Overviews, and Claude find, read, and cite my content?* That's a *content discoverability* question. The remediation is about being indexable, parseable, and attributable: robots.txt, sitemaps, structured data, llms.txt, full-content RSS, canonical URLs, heading anchors. It's the 2024–2025 problem with established best practices. Cloudflare's checker asks a much narrower, more forward-looking question: > *Can autonomous agents programmatically discover and invoke services on this site?* That's an *agent-actionability* question. The remediation is about being callable: Link headers advertising resources (RFC 8288), markdown content negotiation, API catalogs (RFC 9727), OAuth/OIDC discovery (RFC 8414), OAuth Protected Resource metadata (RFC 9728), MCP Server Cards (SEP-2127), Agent Skills indexes, WebMCP tool registrations. It's the 2026 problem and most of the standards are still in draft. Those two questions overlap on maybe **three of thirteen checks**. The other ten are testing things our audit didn't think to look at — and several of them legitimately don't apply to a read-only blog. ## The Category-by-Category Breakdown | Cloudflare category | Score | What it checks | In our AEO audit? | |---|---|---|---| | **Discoverability** | 2/3 | robots.txt, sitemap, Link response headers (RFC 8288) | Partial — we shipped robots/sitemap; Link headers never came up | | **Content** | 0/1 | Markdown content negotiation (`Accept: text/markdown`) | No — we solved the *need* with `llms-full.txt`, not the *protocol* | | **Bot Access Control** | 1/2 | AI bot rules in robots.txt, Content Signals in robots.txt, Web Bot Auth | Partial — we *explicitly rejected* Content Signals | | **API, Auth, MCP & Skill Discovery** | 0/6 | API Catalog, OAuth/OIDC discovery, OAuth Protected Resource, MCP Server Card, Agent Skills index, WebMCP | No — none of this was in scope | | **Commerce** | n/a | x402, MPP, UCP, ACP payment protocols | Not applicable | Twenty fixes in our AEO audit. Thirteen checks in Cloudflare's. Overlap: three. That's the entire story of the score gap in one sentence. ## What Cloudflare Caught That We Genuinely Missed Two findings are real misses — gaps our audit didn't think to look for, with cheap fixes, that I shipped today. ### 1 Link Response Headers RFC 8288 ![Cloudflare audit detail showing the Link headers check: 'Link headers present but no agent-useful relation types found.' The site emits 4 link relations (font preloads) but none point agents at useful resources.](/images/friday-fixes-aeo-does-not-equal-agent-ready/cf-link-headers.png) Our site already sends `Link:` response headers — but only the auto-generated `rel="preload"` entries for fonts and CSS that Next.js and Cloudflare's Early Hints feature inject. Cloudflare's checker parsed four link relations on the homepage and found none of them point agents at anything useful. `Link:` is one of the older agent-discovery patterns (RFC 8288 dates to 2017), and it's how a passing crawler asks the homepage *"what do you have for me?"* The remediation is to advertise the resources you already publish: ```ts // next.config.ts const LINK_HEADER = [ '; rel="describedby"; type="text/plain"', '; rel="alternate"; type="text/plain"; title="Full content for LLMs"', '; rel="alternate"; type="application/rss+xml"; title="RSS feed"', '; rel="sitemap"; type="application/xml"', ].join(", "); ``` Six lines in `next.config.ts`. The header coexists with the preload Link headers Next.js auto-emits — multiple Link values on one response is explicitly valid per RFC 8288 §3. Verify after deploy: ```bash curl -sI https://vibescoder.dev/ | grep -i ^link ``` This is a *real* miss. Our audit's framing was "make content discoverable to AI agents," and Link headers genuinely fit that — they're a homepage-level pointer to the very `llms.txt` and RSS feed we'd already created. The auditor agent prioritized the *files* and missed the *signpost*. ### 2 Markdown Content Negotiation ![Cloudflare audit detail showing the Markdown Negotiation check: GET / with Accept: text/markdown returns text/html — site does not support markdown content negotiation.](/images/friday-fixes-aeo-does-not-equal-agent-ready/cf-markdown-negotiation.png) Cloudflare sends `Accept: text/markdown` to the homepage. The server returns `text/html`. Fail. The interesting nuance here is that *we already solved the underlying problem* — `/llms-full.txt` is a single endpoint that returns every post's full content as plaintext. An agent that wants the markdown version of vibescoder.dev's corpus has had a working answer since the AEO audit. But Cloudflare's checker doesn't know about `llms-full.txt` because it doesn't dereference Link headers (yet) or `llms.txt` (yet). It tests the protocol it tests: per-URL `Accept` negotiation. The fix is content negotiation in middleware: ```ts // src/middleware.ts if (pathname.startsWith("/posts/") && !pathname.endsWith("/raw")) { const accept = request.headers.get("accept") ?? ""; if (prefersMarkdown(accept)) { const url = request.nextUrl.clone(); url.pathname = pathname.replace(/\/?$/, "") + "/raw"; const res = NextResponse.rewrite(url); res.headers.set("Vary", "Accept"); return res; } } ``` The rewrite targets a new route handler at `/posts/[slug]/raw/route.ts` that returns the raw MDX with `Content-Type: text/markdown; charset=utf-8`. The `Vary: Accept` header tells shared caches to keep the HTML and markdown representations separate — without it, the first response wins and pollutes the cache for everyone else. One subtle bit: the `Accept` parser respects q-values. Browsers default to something like `text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8`. A naive `accept.includes("text/markdown")` check would never match anyway, but the more useful guarantee is that if some future browser starts sending `text/markdown;q=0.5,text/html;q=1.0`, we still serve HTML. Markdown only wins when it's explicitly preferred. Try it after deploy: ```bash curl -sH "Accept: text/markdown" https://vibescoder.dev/posts/your-ai-strategy-has-a-blind-spot | head -20 ``` You should see a markdown document with a frontmatter-ish header (title, description, author, date, canonical URL, tags) followed by the post body. ## What Cloudflare Flagged That We Deliberately Disagree With One overlap finding is a values disagreement, not a gap. ![Cloudflare audit detail showing the Content Signals check: 'No Content Signals found in robots.txt.' Cloudflare wants Content-Signal directives like ai-train=no, search=yes, ai-input=no.](/images/friday-fixes-aeo-does-not-equal-agent-ready/cf-content-signals.png) Cloudflare's checker wants `Content-Signal:` directives in robots.txt (e.g., `Content-Signal: ai-train=no, search=yes, ai-input=no`). Last week's audit post explicitly argued against this: > *"The Content Signals option keeps a `Content-Signal: ai-train=no` directive, which tells AI crawlers not to use your content for model training. That sounds reasonable — but for a personal blog trying to maximize reach, being in the training corpus means AI models are more likely to know about you and reference your ideas."* The checker rewards *having* Content Signals, even ones that say "no training." If I wanted the point, I'd add `Content-Signal: ai-train=yes, search=yes, ai-input=yes` — which matches our actual policy. I'm probably going to do that. The point isn't whether to grant training rights; it's whether to *declare* a policy. Silence is currently being read as "no signal," not "yes by default." ## What's Penalizing the Score but Doesn't Apply to a Blog Six of thirteen checks are in the **API, Auth, MCP & Skill Discovery** bucket. We score 0/6 on all of them. ![Cloudflare audit details for the MCP Server Card and Agent Skills index checks: both return 404 because vibescoder.dev has no MCP server and no agent skills to expose.](/images/friday-fixes-aeo-does-not-equal-agent-ready/cf-mcp-skills.png) The list: | Check | What an agent would do with it | Applies to vibescoder.dev? | |---|---|---| | **API Catalog** (RFC 9727, `/.well-known/api-catalog`) | Discover the site's public APIs | No public APIs | | **OAuth/OIDC discovery** (RFC 8414, `/.well-known/openid-configuration`) | Learn how to authenticate | No public auth surface | | **OAuth Protected Resource** (RFC 9728, `/.well-known/oauth-protected-resource`) | Discover which OAuth servers can issue tokens for the site's resources | No protected resources | | **MCP Server Card** (SEP-2127, `/.well-known/mcp/server-card.json`) | Connect to an MCP server hosted by the site | No MCP server | | **Agent Skills index** (`/.well-known/agent-skills/index.json`) | Browse available agent skills | No agent skills published | | **WebMCP** (`navigator.modelContext.provideContext()`) | Invoke browser-side tools the page exposes | No browser-side tools | These aren't bugs. They're the checker scoring vibescoder.dev as if it should expose a programmatic surface — APIs, OAuth, MCP, skills, browser tools. A read-only personal blog should not. The Commerce category (0/0) is correctly excluded from scoring because no e-commerce signals were detected; arguably the MCP/Auth/Skills bucket should be too, but it's not. If/when this blog grows an agent-actionable surface — say, an MCP server that lets agents subscribe to posts, or an API that exposes analytics under OAuth — those zeros become meaningful and most of them get closed by a single `.well-known` file per protocol. Until then, they're noise. ## Why Both Audits Are Correct This is the part that surprised me. **AEO is about being read.** It optimizes for the model that consumes content: a crawler, an answer engine, an AI assistant summarizing your work into a response. The remediation is on the publishing side. The unit of value is *citation*. **Agent-ready is about being used.** It optimizes for the agent that *acts on* your site: invoking tools, authenticating to APIs, exchanging value via payment protocols, registering capabilities through MCP or WebMCP. The remediation is on the API/protocol side. The unit of value is *invocation*. Most sites today need AEO. Most sites today don't need most of "agent-ready." A blog needs the first; an enterprise SaaS app probably needs both; an internal tool probably needs only the second. The frustration with the score is that Cloudflare's checker doesn't yet distinguish *site class* — it scores every URL against every protocol — but the underlying competencies are real and distinct. The right mental model isn't "I scored 25/100, I have work to do." It's "**this audit measured a competency I haven't built and probably don't need yet** — except for these two findings that genuinely belong in the content-discoverability bucket I thought I'd already finished." That's a more useful conclusion than either "Cloudflare is gatekeeping a fake category" or "we missed everything." ## What Shipped Three commits, one repo, all linked back to specific Cloudflare findings. | Fix | Cloudflare check | What | |---|---|---| | Link response headers in `next.config.ts` ([eb51e85](https://github.com/carryologist/the-vibe-coder/commit/eb51e85)) | Discoverability — Link headers | Advertise `llms.txt`, `llms-full.txt`, RSS, sitemap from every response | | Markdown negotiation for posts in `middleware.ts` + new `/posts/[slug]/raw/route.ts` ([eb51e85](https://github.com/carryologist/the-vibe-coder/commit/eb51e85)) | Content — Markdown Negotiation (post URLs) | Serve raw MDX as `text/markdown` when `Accept` prefers it | | Markdown negotiation extended to homepage ([7c55e3c](https://github.com/carryologist/the-vibe-coder/commit/7c55e3c)) | Content — Markdown Negotiation (root URL) | Rewrite `/`, `/about`, `/tags` to `/llms.txt` when `Accept` prefers markdown | Deliberately *not* shipped: Content Signals in robots.txt (values disagreement, may add `ai-train=yes` later), all six API/Auth/MCP/Skills well-knowns (out of scope for a read-only blog). ## The Rescan 25 → 33 After the deploy landed, I re-ran the same isitagentready.com check. ![Side-by-side comparison of the Cloudflare agent-readiness scorecard before and after the fixes. Left: score 25, Discoverability 67%, Content 0%. Right: score 33, Discoverability 100%, Content 0%.](/images/friday-fixes-aeo-does-not-equal-agent-ready/cf-before-after.png) **Discoverability went from 2/3 to 3/3.** The Link headers check is now green, with the audit detail showing the `; rel="describedby"`, `; rel="alternate"`, RSS, and sitemap entries Cloudflare parsed out of our response. ![Cloudflare audit detail after the fix: the Link headers check is green, with all four agent-useful rel types (describedby, alternate, alternate, sitemap) recognized in the parsed response.](/images/friday-fixes-aeo-does-not-equal-agent-ready/cf-link-headers-after.png) **Content stayed at 0/1.** That one surprised me — the markdown negotiation fix *works* (`curl -H 'Accept: text/markdown' https://vibescoder.dev/posts/your-ai-strategy-has-a-blind-spot` returns a clean `text/markdown` response with the post body). But Cloudflare's checker only probes `GET /` for markdown, not `/posts/`. The fix was real; it just wasn't where the checker looks. Fifteen minutes later, a third commit — [7c55e3c](https://github.com/carryologist/the-vibe-coder/commit/7c55e3c) — extended the middleware to also rewrite `/`, `/about`, and `/tags` to `/llms.txt` when `Accept` prefers markdown. `/llms.txt` is already a hand-curated markdown summary of the site, so it's the right answer for an agent asking the root URL in markdown: ```ts if (pathname === "/" || pathname === "/about" || pathname === "/tags") { const accept = request.headers.get("accept") ?? ""; if (prefersMarkdown(accept)) { const url = request.nextUrl.clone(); url.pathname = "/llms.txt"; const res = NextResponse.rewrite(url); res.headers.set("Vary", "Accept"); res.headers.set("Content-Type", "text/markdown; charset=utf-8"); return res; } } ``` That should push the next rescan to roughly **39 / Level 1** — Content 1/1, leaving only the deliberate Content Signals abstention and the six out-of-scope API/Auth/MCP/Skills checks as the remaining gap. The practical lesson buried inside the score: **automated checkers test the protocol at a specific URL, not the capability across your site.** Our `/posts/` markdown negotiation was correct from the first deploy, but the checker probes `/`. The implementation has to meet the probe where the probe lives. ## What's Left and Why We're Stopping There With the homepage extension, every Cloudflare finding that maps onto a *content-discoverability competency* is now closed. What remains: | Remaining check | Score | Why we're not shipping a fix | |---|---|---| | Content Signals in robots.txt | 0/1 | Deliberate — we don't want to declare `ai-train=no`, and the checker doesn't distinguish "opt-in" from "present." Likely to add `Content-Signal: ai-train=yes, search=yes, ai-input=yes` once the directive grammar settles | | Web Bot Auth request signing | n/a | Cloudflare Enterprise feature; not applicable on the free tier | | API Catalog (RFC 9727) | 0/1 | No public API | | OAuth/OIDC discovery (RFC 8414) | 0/1 | No public auth surface | | OAuth Protected Resource (RFC 9728) | 0/1 | No protected resources | | MCP Server Card (SEP-2127) | 0/1 | No MCP server | | Agent Skills index | 0/1 | No agent skills published | | WebMCP browser tools | 0/1 | No browser-side tools | The last six all live in the **agent-actionability** competency — the "can an agent *do something here*" question. For a read-only personal blog, leaving them empty is the correct answer; shipping placeholder `.well-known` files just to satisfy a scorecard would be cargo-culting. ## How I'd Update That Earlier Audit The earlier [AEO audit post](/posts/your-ai-strategy-has-a-blind-spot) framed the work as "20 issues across 4 severity levels — done." That framing was correct *for the question we asked*. What I'd add today, in light of the Cloudflare audit: 1. **AEO has a sibling discipline called agent-readiness.** It's distinct, not a superset. 2. **Two of the AEO improvements really should have included Link headers and content negotiation.** They live on the boundary between the two competencies, and our auditor agent missed them because it was framed entirely around content discovery. 3. **Tools like isitagentready.com are useful even when the score is misleading.** The score is calibrated for sites with programmatic surfaces. The individual findings are still surfacing real protocol-level gaps that a content-only audit can't see. The audit you run depends on the question you ask. The question you ask depends on the framing of "what does it mean for an AI to consume my site?" The interesting realization from running both audits back-to-back is that there are now at least *two* good answers to that question, and they're going to keep diverging as the protocols underneath each one mature. ## By the Numbers - **25 → 33** Cloudflare agent-readiness score (and ≈39 expected on the next scan once the homepage markdown extension lands) - **67% → 100%** Discoverability after the Link headers fix — the visible scorecard win - **13 checks** in the Cloudflare audit, of which **3** overlap with our original AEO audit - **20 fixes** in our original AEO audit; **0** of them targeted Link headers or markdown negotiation - **2 genuine misses** Cloudflare caught and we didn't: Link headers, content negotiation - **1 values disagreement** (Content Signals — we deliberately don't declare `ai-train=no`) - **6 / 13 checks** in the API/Auth/MCP/Skills bucket — all 0/6, all out of scope for a read-only blog - **3 commits** shipped today (two in the initial fix, one to extend markdown negotiation to the homepage after the checker only probed `/`) - **6 lines** of config for the Link header fix - **~65 lines** of middleware + route handler for markdown negotiation across both commits - **1 lesson** about automated checkers: they test the protocol at a specific URL, not the capability across the site === ## Thursday Thoughts: The Models We Can't Run - URL: https://vibescoder.dev/posts/thursday-thoughts-the-models-we-cant-run - Date: 2026-05-14 - Tags: #agents #ai #llm #homelab #meta #building-in-public - Reading time: 7 min read DeepSeek V4-Pro, V4-Flash, and Zyphra ZAYA1 are three of the most exciting new models in local AI. None of them run on our RTX 5090 homelab — for completely different reasons. Here's the research, the math, and what it means for anyone building a local inference rig. --- Every week or two, a model drops that makes the local AI community lose its collective mind. This week it was three at once: **DeepSeek V4-Pro**, **DeepSeek V4-Flash**, and **Zyphra ZAYA1-8B**. All three are genuinely impressive. All three are models I wanted to benchmark on our homelab. And after doing the research, I'm not testing any of them. Not because I don't want to. Because I physically can't — or can't yet. This post isn't a benchmark. It's the research that happens *before* the benchmark, where you figure out which models are even candidates for your hardware. If you're building or considering a local inference setup, the reasons these three models don't work are more instructive than any leaderboard score. ## The Rig Quick refresher on what we're working with: | Resource | Spec | |---|---| | GPU | NVIDIA RTX 5090 — 32 GB VRAM | | RAM | 64 GB DDR5 | | CPU | AMD Ryzen 9 9950X3D — 16 cores / 32 threads | | Disk | 1.8 TB NVMe | | Inference | llama.cpp on the GPU | This is a strong homelab by any measure. We run Qwen 3.5 35B-A3B daily for agentic coding at 200+ tok/s. In previous benchmark rounds, Devstral, Codestral, Gemma 4, and DeepSeek R1 14B have all run comfortably. The 5090 is the sweet spot for 20B–35B models. But the new generation of models isn't playing in the 20B–35B range anymore. ## DeepSeek V4-Pro Too Big for Anything Short of a Data Center V4-Pro is DeepSeek's new flagship. The numbers are staggering: | Spec | Value | |---|---| | Total parameters | **1.6 trillion** | | Activated per token | 49B (MoE, 256 experts, top-6 routing) | | Model weights (FP4+FP8 mixed) | **805 GB on disk** | | Context window | 1M tokens | That 805 GB number is the wall. Our entire system — 32 GB VRAM plus 64 GB RAM — gives us 96 GB of addressable memory. The model is **8.4x larger than our total memory**. There are no GGUF quants available, and nobody is making them because there's no consumer hardware that could run them meaningfully. For context, we tried running Kimi K2.6 (a similarly-sized 1T MoE model) a few weeks ago. It "ran" at **less than 1 token per second** — the weights spilled out of VRAM into RAM, and we hit the DDR5 memory bandwidth ceiling (~80 GB/s vs the 5090's ~1.8 TB/s). V4-Pro at 1.6T would be even slower. **Verdict**: Cloud API only. DeepSeek serves it at [api.deepseek.com](https://api.deepseek.com) and we've added it to our benchmark rig as a cloud provider alongside Anthropic. ## DeepSeek V4-Flash Close but Not Close Enough V4-Flash is V4-Pro's smaller sibling and the one I was actually hopeful about: | Spec | Value | |---|---| | Total parameters | **284B** | | Activated per token | 13B (MoE, 256 experts, top-6 routing) | | Smallest GGUF quant (Q2_K) | **96.2 GB** | | Most popular quant (Q4_K_M) | **160.2 GB** | | Context window | 1M tokens | Only 13B activated per token sounds incredible — that's smaller than our DeepSeek R1 14B. But MoE models need all their expert weights resident in memory even though only a fraction fires per token. That 284B of total parameters has to be somewhere accessible. The math doesn't work: | Quant | Size | Fits in VRAM + RAM (96 GB)? | |---|---|---| | Q2_K | 96.2 GB | Barely — 0.2 GB over before KV cache | | Q3_K_M | 126.2 GB | No — needs 30 GB disk offload | | Q4_K_M | 160.2 GB | No — needs 64 GB disk offload | | FP4-FP8 native | 145.4 GB | No — needs 49 GB disk offload | There *were* IQ1_S (54 GB) and IQ2_M (87 GB) quants that would have fit — but the community removed them. When quant maintainers pull their own files, that's a strong signal the output quality was garbage. And even if one of these squeaked into memory, there's a bigger problem: **llama.cpp doesn't support the DeepSeek V4 architecture yet**. All existing GGUFs require custom forks. The mainline support PRs are still open and under active debate. You'd be building from an untested branch to run a model that barely fits. **Verdict**: Not ready. We've added V4-Flash to the benchmark as a cloud API model for now. When llama.cpp merges V4 support *and* a viable sub-90 GB quant exists, we'll revisit. ## ZAYA1-8B the Right Size the Wrong Stack This is the one that hurts the most, because on paper it's a perfect homelab model: | Spec | Value | |---|---| | Total parameters | 8.4B | | Activated per token | **760M** (MoE, 16 experts, top-1 routing) | | VRAM at bf16 | ~17 GB | | Context window | 128K tokens | | AIME '26 score | 89.1 | 8.4 billion parameters. 17 GB in bf16. Fits trivially on the 5090 with room to spare. Punches absurdly above its weight on reasoning benchmarks — 89.1 on AIME '26 is competitive with models 10–15x its size. So what's the problem? **Architecture.** ZAYA1 uses CCA (Cross-Channel Attention) — Zyphra's novel hybrid of Mamba-style recurrence and traditional attention. It's not standard Mamba2. It's not standard transformer attention. It's a fundamentally new layer type with small 1D convolutions, custom Q/K projections, and learned residual scaling. llama.cpp has no support for this architecture. There's an [open feature request](https://github.com/ggml-org/llama.cpp/issues/22776) with nothing but +1 comments. No GGUF quants exist because there's nothing to run them on. Even Zyphra's older Zamba2 architecture ([#21412](https://github.com/ggml-org/llama.cpp/issues/21412)) remains unimplemented. The only way to run ZAYA1 today is through Zyphra's custom vLLM fork — a completely different serving stack from our llama.cpp setup. It would work on the 5090, but it means standing up and maintaining a parallel inference pipeline. **Verdict**: On the to-do list. When llama.cpp adds CCA support or we carve out time to set up vLLM as a second serving backend, this is the first model we'll test. ## What Actually Runs on a 32 GB GPU Here's the uncomfortable reality of local inference in mid-2026: the models generating the most hype are the ones you can't run. The models that *fly* on a 32 GB card — where you get 100+ tok/s and useful agentic performance — are capped at roughly **24–28 GB of weights** (leaving room for KV cache). That means: | Category | What Fits | |---|---| | Dense models | Up to ~14B at Q8, ~20B at Q6, ~27B at Q4 | | MoE models | Up to ~35B total at Q4 (e.g. Qwen 3.5 35B-A3B) | | What doesn't | Anything over ~28 GB of quantized weights | Our current daily driver — Qwen 3.5 35B-A3B at Q4_K_XL — is 22 GB of weights with 3B activated per token, running at 200+ tok/s. It's fast, it's good, and it's approximately the ceiling of what a single 5090 can do at interactive speeds. ## The Three Walls Each of these models hits a different wall, and that's what makes this exercise useful: 1. **V4-Pro** — pure size. 805 GB of weights. No amount of quantization or clever offloading helps when the model is 8x your total memory. 2. **V4-Flash** — the quantization gap. The model *almost* fits at extreme compression, but the quality degrades too far. We're in a window where the model exists but the tooling hasn't caught up to make it practical on consumer hardware. 3. **ZAYA1** — architecture support. The model fits perfectly. The hardware is more than enough. But the inference engine doesn't speak the language yet. If you're evaluating models for a homelab or edge deployment, these are the three questions to ask before you even think about benchmarks: Is it small enough? Is the quantization viable? Does my inference stack support it? ## By the Numbers - **805 GB** — DeepSeek V4-Pro model weight size. 8.4x our total system memory. - **96.2 GB** — smallest V4-Flash GGUF quant. Still 0.2 GB over our VRAM + RAM. - **17 GB** — ZAYA1-8B at bf16. Fits trivially, runs nowhere (yet). - **22 GB** — our actual daily driver (Qwen 3.5 35B-A3B at Q4_K_XL). The real ceiling. - **0** — number of these three models with merged llama.cpp support. - **2** — models we added to the benchmark as cloud API endpoints instead (V4-Flash, V4-Pro). === ## Spring Cleaning Your Vibe Coded Apps - URL: https://vibescoder.dev/posts/spring-cleaning-your-vibe-coded-apps - Date: 2026-05-13 - Tags: #agents #vibe-coding #debugging - Reading time: 12 min read I pointed a current-gen AI agent at a fitness tracker I vibe coded a year ago. Five PRs later: security holes patched, a 1,300-line component broken apart, sync queries batched, and accessibility fixed. Here's what the audit found, how it was structured, and why every vibe coded app deserves a spring cleaning. --- I've been building a fitness tracker app on and off for about a year. It started as a weekend vibe coding project — Next.js 14, Prisma, PostgreSQL, deployed on Vercel. Over the months I bolted on Peloton API sync, Tonal API integration, OCR-based screenshot import via Tesseract.js, goals with quarterly milestones, dark mode, the works. Classic vibe coded app: functional, personal, messy. It works. I use it every day. And until last week, I hadn't really looked at the code in months. Then I decided to point a current-generation AI agent at it — not to add features, but to audit and refactor what was already there. The results were humbling, educational, and genuinely useful. Five PRs later, the app is measurably better in ways I wouldn't have prioritized on my own. This post is about what I found, how I structured the work, and why I think every vibe coded app deserves a spring cleaning. ## The Setup The app is a standard Next.js stack: App Router, Prisma ORM, PostgreSQL on Neon, NextAuth v5 beta for Google login, Tailwind CSS. It has API routes for CRUD, two third-party API integrations (Peloton and Tonal), and a Tesseract.js-powered OCR pipeline for importing workout screenshots. About 4,000 lines of application code across 30-ish files. The agent doing the audit was Claude Opus 4.6 running through Coder's agentic development environment — same setup I use for this blog. Full filesystem access, shell access, GitHub CLI pre-authenticated. I gave it a simple prompt: audit the codebase systematically, find issues, fix them in phased PRs. ## The Ground Rules I didn't want one massive PR that changed everything. I've been burned by that before — hard to review, hard to revert, hard to know what broke what. Instead we agreed on a structure: - **Chunked PRs**, one phase at a time - I merge each one via `gh pr merge --squash --delete-branch` - Vercel auto-deploys on push to main - I test the live site between phases before greenlighting the next one This turned out to be the right call. Phase 4 introduced a subtle data-layer bug that would have been invisible in a 2,000-line mega-PR. ## Phase 1 Quick Wins The first pass found the kind of stuff that accumulates in any project you don't actively maintain: - **No PrismaClient singleton.** Every API route was creating a new database connection. In development this causes the "too many connections" warning. In production on a serverless platform like Vercel, it's wasteful at best. - **Missing database indexes.** The `WorkoutSession` table had no indexes on `date`, `source`, or `pelotonId` — columns used in every query. - **Array mutation bug.** A `sort()` call was mutating state directly instead of spreading first. React doesn't detect mutations. - **Hardcoded year.** `new Date().getFullYear()` was correct, but a default value elsewhere was hardcoded to 2025. - **Dead code.** Unused imports, unreachable functions, a commented-out migration route. None of these were individually catastrophic. Together they represented the kind of entropy that makes a codebase progressively harder to work with. ## Phase 2 Security Hardening This was the one that made me uncomfortable. The app was behind Google OAuth, so I'd never thought hard about defense in depth. The agent found: - **No auth checks on API routes.** NextAuth middleware was handling the gate, but individual API routes had no secondary validation. If middleware ever failed or was misconfigured, every route was wide open. - **No input validation.** API routes trusted whatever the client sent. No length limits, no type checking, no sanitization. - **Verbose error messages.** Stack traces and internal details were leaking to the client in error responses. - **No security headers.** No CSP, no X-Frame-Options, no Referrer-Policy. - **Database migration route accessible in production.** A `/api/migrate` endpoint existed with no environment check. The fix added blocking `getSession()` auth checks to all API routes, email allowlisting via `ALLOWED_EMAIL`, security headers in `next.config.ts`, Zod-based input validation on all write endpoints, and a production block on the migrate route. **And then it broke the entire app.** NextAuth v5 beta's `auth()` returns null in Route Handlers on Vercel — even for authenticated users. It's a cookie context limitation. The blocking 401 checks we just added were rejecting every API call, including from logged-in users. The dashboard loaded but couldn't fetch any data. The hotfix was immediate: replace the blocking auth checks with a non-blocking `checkAuth()` that logs a warning but doesn't return 401. Middleware remains the real auth gate. The defense-in-depth intent is still there — if someone bypasses middleware, the logs will show it — but the app doesn't break when NextAuth's Route Handler session resolution is flaky. This is worth being honest about. The agent's security recommendation was textbook correct: every API route should verify authentication independently. But it didn't account for a known limitation of NextAuth v5 beta's Vercel deployment. The fix was fast, but for a few minutes the app was completely down. ## Phase 3 Component Refactor The main dashboard component — `WorkoutDashboard.tsx` — was 1,316 lines. It handled state management, API calls, OCR processing, goal calculations, settings, and all the UI rendering. Classic vibe code: everything in one file because it was easier to keep building than to stop and organize. The agent extracted: - `src/types/workout.ts` — shared TypeScript interfaces - `src/utils/goalsApi.ts` — goal API helpers - `src/utils/tonalOCR.ts` — OCR parsing functions - `src/components/SettingsModal.tsx` — settings UI - `src/components/DashboardHeader.tsx` — header with sync buttons and year selector The main component went from 1,316 to 734 lines. Ten other files that were importing types from WorkoutDashboard got updated to import from `@/types/workout`. Re-exports were added for backward compatibility so nothing broke. This is the phase where the AI agent's strength really showed. Refactoring a 1,300-line component requires understanding every dependency, every import chain, every prop flow. It's exactly the kind of tedious, high-attention work that humans procrastinate on and agents handle methodically. ## Phase 4 Performance This phase had the most interesting findings — and the most interesting bug. **Batch sync queries.** The Peloton and Tonal sync routes were checking whether each workout already existed in the database one at a time. For a full sync of 100+ workouts, that's 100+ individual `findFirst` queries. The fix batch-fetches all already-synced IDs for the current page in a single query. **Year filtering.** The workouts API was returning every workout in the database, and the client was filtering by year. Added a `?year=` query parameter so the database does the filtering. Also discovered that Next.js was caching the API response — added `export const dynamic = 'force-dynamic'` to prevent stale data. **Dashboard memoization.** Added `useMemo` on filtered sessions and current goal, `useCallback` on event handlers, and an `AbortController` on the data-loading effect with proper cleanup. **The debug query that crashed production.** While investigating why year filtering wasn't working, the agent added a `prisma.$queryRawUnsafe` call to log the year distribution of workouts in the database. Reasonable idea. Except `$queryRawUnsafe` isn't available in Prisma's edge runtime on Vercel, so it crashed the entire `/api/workouts` endpoint. Another self-inflicted outage during the spring cleaning itself. The fix was just deleting the debug line, but it's a reminder that even diagnostic code can break things if you don't test it in the actual deployment environment. **The Tonal API bug.** This one was hiding in plain sight. The Tonal API response format had changed at some point — it returns a raw JSON array (not `{ data: [...] }`), and uses different field names (`activityId` instead of `id`, `workoutPreview` as a nested object instead of flat fields). The sync route was silently failing to map any Tonal data. It had probably been broken for months. I wouldn't have found this without someone methodically reading through the API integration code. **The year display bug.** Two components — `MonthlySummary` and `ProgressChart` — had their own internal `const currentYear = new Date().getFullYear()` instead of using the year prop from the parent. When you switched to view 2025 data, the charts still showed 2026 labels. Another one that was hiding in plain sight. ## Phase 5 Accessibility The final phase was a full accessibility audit. Three modals with no ARIA attributes, no keyboard handling, and no focus traps. Twelve form inputs without programmatic label associations. Seven icon-only buttons with no accessible names. Button groups acting as radio selectors with no `role` or `aria-checked` semantics. The fixes were surgical: - All 3 modals got `role="dialog"`, `aria-modal`, `aria-labelledby`, Escape key handlers, and Tab/Shift+Tab focus traps - All 12 form inputs got `htmlFor`/`id` label pairing - All 7 icon-only buttons got `aria-label` - 5 settings button groups got `role="radiogroup"` and `aria-checked` - An `alert()` call got replaced with the existing error banner pattern This is another category where AI agents excel. Accessibility auditing requires checking every interactive element against a known set of rules. It's comprehensive, repetitive, and easy to miss things when you're doing it manually. The agent found every instance across 6 files in one pass. ## What the Agent Found That I Wouldn't Have Looking back across all five phases, there's a pattern. The issues fall into three categories: **Things I knew were wrong but hadn't prioritized.** The giant component, the missing indexes, the lack of input validation. These were in my mental backlog but never rose to the top because the app worked fine. **Things I didn't know were wrong.** The Tonal API field name changes, the year display bug, the Cloudflare-style "it works but it's silently broken" issues. These required reading code I hadn't touched in months with fresh eyes. **Things I wouldn't have thought to check.** The accessibility audit, the security headers, the `force-dynamic` export. These require domain knowledge that I have in theory but don't apply consistently to side projects. The agent brought all three — the discipline to do what I'd been putting off, the fresh perspective to catch what I'd stopped seeing, and the domain knowledge to check what I'd forgotten to consider. ## By the Numbers | Phase | PR | Files Changed | Key Metric | |-------|-----|--------------|------------| | Quick Wins | #12 | 8 | PrismaClient singleton, 3 DB indexes added | | Security | #13 | 6 | Auth checks on all routes, input validation, security headers | | Component Refactor | #14 | 14 | 1,316 → 734 lines in main component | | Performance | #15 | 9 | N+1 queries eliminated, Tonal API bug fixed | | Accessibility | #16 | 6 | 3 modals, 12 inputs, 7 buttons, 5 radiogroups fixed | **5 PRs. 5 phases. ~40 files touched. Two self-inflicted outages along the way.** ## Why This Matters for Vibe Coding Vibe coding is great for building things fast. I built this fitness tracker in a weekend and have been using it daily for a year. That's a genuine success story. But vibe coded apps accumulate debt faster than traditionally developed ones because the builder (me, you, anyone) is optimizing for velocity, not maintainability. The models available today — not last year's models, but the ones shipping right now — are good enough to audit your old vibe coded projects and find real issues. Not theoretical concerns. Real bugs, real security holes, real performance problems that were hiding in code you stopped actively reading months ago. The cost is low. This entire audit — five phases across a week of part-time work — probably consumed $15-20 in API tokens. The alternative was letting those issues compound until something actually broke in production, or until the codebase became so tangled that adding features felt painful. ## How to Do Your Own Spring Cleaning If you have vibe coded apps running in production (or even just apps you built quickly and stopped maintaining), here's the playbook: 1. **Start with a fresh clone and a current model.** Don't use the model that built the app. Use whatever's newest. The gap between models 6-12 months apart is significant for code comprehension tasks. 2. **Phase the work.** Don't try to fix everything in one PR. Group changes by category: quick wins, security, architecture, performance, accessibility. Merge and test between each phase. 3. **Let the agent find things you forgot about.** The most valuable findings in my audit weren't the ones I already knew about. They were the silent failures — API response formats that changed, display bugs in components I wasn't looking at, auth assumptions that were never tested. 4. **Check the boring stuff.** Security headers, input validation, database indexes, accessible names on buttons. These are the things that never make it onto a feature backlog but determine whether your app is actually solid. 5. **Don't skip the build.** After every change, build the project. Type errors and import issues surface immediately. Every one of our five PRs passed `next build` before merging. ## The Meta Lesson The fitness tracker works the same way it did before the audit. A user wouldn't notice any difference. But the codebase is meaningfully better — more secure, better organized, more performant, more accessible. The Tonal sync actually works now. The year selector actually works now. It wasn't a clean sweep, though. We broke the app twice during the spring cleaning — once with auth checks that didn't account for NextAuth v5's Vercel behavior, once with a debug query that crashed the API endpoint. Both were fixed within minutes, but they happened. If I'd been doing this on a higher-traffic app, those minutes would have mattered. The lesson isn't "let the agent do everything and trust the output." It's "let the agent find things you missed, but test every change in the real environment before moving on." The phased PR approach saved us here. If all five phases had been one PR, the auth breakage would have been tangled up with the component refactor and the performance changes, and debugging would have been miserable. Every vibe coded app has this layer of accumulated entropy. The models available today are good enough to find it. They're also capable of introducing new problems while fixing old ones — just like a human would. The structure around the work matters as much as the work itself. === ## The Fix That Was Fixed Four Times - URL: https://vibescoder.dev/posts/the-fix-that-was-fixed-four-times - Date: 2026-05-12 - Tags: #homelab #coder #building-in-public #agents - Reading time: 9 min read A second user joined the homelab Coder instance and couldn't push to GitHub. What looked like a missing config turned into five chained problems, a domain migration aftershock, an agent-debugging-an-agent meta-moment, and the discovery that the same credential helper bug had been "fixed" four times in ten days — and never actually deployed. --- My wife started using the homelab Coder instance this week. She's a fellow vibe coder, she has her own GitHub account, and she wanted to push code from her workspace. The agent told her GitHub was scoped to read-only. That's how a Sunday afternoon turned into a five-problem debugging cascade, a forced migration cleanup, an accidental outage of the very AI assistant helping me debug, and an archaeological dig through my own blog fodder that revealed the same bug had been discovered and "fixed" four times in ten days — without ever actually being deployed. --- ## 1 the Config That Wasn't The first thing I checked was Admin Settings → External Authentication. The page showed exactly one thing: **"No providers have been configured!"** This was confusing, because I distinctly remembered setting up a GitHub OAuth App weeks ago. The Client ID was in my GitHub Developer Settings. The env file at `/etc/coder.d/coder.env` had all the right variables: ```env CODER_EXTERNAL_AUTH_0_TYPE=github CODER_EXTERNAL_AUTH_0_CLIENT_ID=Ov23li CODER_EXTERNAL_AUTH_0_CLIENT_SECRET= CODER_EXTERNAL_AUTH_0_MCP_URL=https://api.githubcopilot.com/mcp/ CODER_EXPERIMENTS=oauth2,mcp-server-http ``` Everything looked right. But "looks right" and "is loaded" are different things. The agent suggested checking whether the running Coder process actually had these variables: ```bash sudo cat /proc/$(pgrep -f 'coder server')/environ \ | tr '\0' '\n' \ | grep CODER_EXTERNAL_AUTH ``` Nothing. The process had zero external auth variables. The env file existed. Coder was running. But the two had never met. **Root cause**: The systemd service file had no `EnvironmentFile=` directive. The env file was sitting there, perfectly formatted, completely ignored. The service file looked like this: ```ini [Service] Type=simple ExecStart=/usr/bin/coder server --http-address 0.0.0.0:3000 Restart=always RestartSec=5 User=youruser Environment=HOME=/home/youruser Environment=CODER_EXPERIMENTS=agents ``` No `EnvironmentFile=/etc/coder.d/coder.env`. One line missing, entire feature broken. **The fix**: Add `EnvironmentFile=/etc/coder.d/coder.env` to the `[Service]` section. Then `daemon-reload` and restart. --- ## 2 the Flag I Killed While Fixing the Flag The service file also had a hardcoded `Environment=CODER_EXPERIMENTS=agents` line. The agent told me to remove it since the env file already had `CODER_EXPERIMENTS` defined. Made sense — don't duplicate config. After restarting, the External Auth page showed the GitHub provider. Progress. But the Agents tab was gone. The env file had `CODER_EXPERIMENTS=oauth2,mcp-server-http`. The hardcoded line I just removed was the only thing enabling `agents`. Nobody had ever added it to the env file. **The fix**: Update the env file to `CODER_EXPERIMENTS=oauth2,mcp-server-http,agents`. **The meta-moment**: I couldn't use my homelab agent to fix this because the agents feature was the thing I'd just broken. I had to open a separate Coder session on my work instance and troubleshoot from there. Debugging your AI coding assistant with your AI coding assistant — when the first one is broken. --- ## 3 the Domain Migration Aftershock With agents back and external auth configured, my wife tried to link her GitHub account. She clicked "Click to Login" in her user settings and got a redirect URI mismatch error from GitHub. A few weeks ago, I migrated the Coder instance from a `*.pit-1.try.coder.app` tunnel URL to a Cloudflare-backed custom domain at `coder.vibescoder.dev`. The `CODER_ACCESS_URL` was updated. DNS was working. The UI loaded fine. But the GitHub OAuth App still had the old URLs: - **Homepage URL**: `https://xxxxxxxxx.pit-1.try.coder.app` - **Callback URL**: `https://xxxxxxxxx.pit-1.try.coder.app/external-auth/github/callback` Updated both to the new domain. No Coder restart needed — this is GitHub-side config. **Gotcha on the callback path**: The callback URL uses the provider **ID**, not a numeric index. Since I didn't set `CODER_EXTERNAL_AUTH_0_ID` explicitly, Coder defaults to using the `TYPE` value as the ID. The correct path is `/external-auth/github/callback`, not `/external-auth/0/callback`. The first attempt with `/0/` failed silently. After the fix, my wife authorized the OAuth App. GitHub showed "Authorize carryologist" — which briefly confused us, since that's my handle, not hers. But that's standard OAuth: the app is owned by me, and she's granting it permission to act on her behalf. App owner ≠ authorizing user. --- ## 4 the Agent That Still Couldn't Push External auth: configured. OAuth app: linked. Second user: authenticated. Everything should work. I went back to my Coder Agent workspace to push the blog fodder file I'd been writing about this whole saga. The agent couldn't clone my private repo. "Invalid username or token." The external auth token existed — `coder external-auth access-token github` returned a valid token. But git operations failed because the credential helper was reading from an empty environment variable: ```bash git config --global credential.helper # → !f() { echo "password=$GITHUB_TOKEN"; echo "username=x-access-token"; }; f ``` The helper sends `$GITHUB_TOKEN` as the password. But `$GITHUB_TOKEN` was empty. It was being set in `.bashrc`: ```bash export GITHUB_TOKEN=$(coder external-auth access-token github 2>/dev/null) export GH_TOKEN="$GITHUB_TOKEN" ``` **The problem**: `.bashrc` only runs in interactive shells. The Coder Agent runs git operations in a non-interactive context. No `.bashrc` sourcing, no `$GITHUB_TOKEN`, no authentication. The credential helper faithfully sent a blank password on every request. **The fix**: Change the credential helper to fetch the token inline instead of reading an environment variable: ```bash # Before (broken for agents): git config --global credential.helper \ '!f() { echo "password=$GITHUB_TOKEN"; echo "username=x-access-token"; }; f' # After (works everywhere): git config --global credential.helper \ '!f() { echo "password=$(coder external-auth access-token github 2>/dev/null)"; echo "username=x-access-token"; }; f' ``` Updated the template's `main.tf`, pushed it with `coder templates push`, then ran `coder update blog-fodder` to rebuild the workspace. **Another gotcha**: `coder restart` does not pick up template changes. It restarts using the same build. You need `coder update ` to rebuild with the latest template version. This distinction will trip up anyone who doesn't know to look for it. --- ## 5 the Archaeology With everything finally working, I asked the agent to search through all my previous blog fodder and published posts to find when this credential helper pattern was introduced. What it found was worse than I expected. The same bug had been discovered and "fixed" **four times in ten days**: | Date | Session | What Happened | Did It Stick? | |------|---------|---------------|---------------| | ~Apr 24 | Gemma research | Agent silently failing auth. Applied `.bashrc` export fix. Pushed via `coder templates push`. | **Overwrote** whatever was live with a weaker fix | | Apr 28 | Housekeeping | Recognized `.bashrc` doesn't work for agents. | **No fix** — documented workaround in a skill file | | Apr 30 | Deploy Day | Full 3-layer diagnosis. Applied the **correct** fix: Terraform `env {}` block + inline credential helper + `.bashrc` cleanup. Committed to `coder-templates` git repo. | **Never pushed** — agent lacked `coder templates push` permissions | | May 3 | This session | Second user can't push. Same broken pattern. | Fixed — finally | The April 30 fix was the right one. It used Terraform's `data "coder_external_auth"` resource to inject `GITHUB_TOKEN` and `GH_TOKEN` at the process level — no shell init files needed, no environment variable dependencies. It even cleaned up stale `.bashrc` entries. It's documented in my own published post, "[Invisible Failures](/posts/invisible-failures-the-bugs-that-hide-in-plain-sight)." But the agent that wrote it didn't have template admin permissions. It committed the fix to the `coder-templates` git repo and moved on. Nobody ran `coder templates push`. The fix sat in version control, correct and complete, for three days. Meanwhile, the workstation's local copy of `~/coder-templates` was **seven commits behind** the git repo. When I tried to `git pull` today, it had a merge conflict with the manual credential helper edit I'd just made. After stashing and pulling, the full April 30 fix — including the Terraform `env {}` block — was finally pushed to the live Coder server. Three days late. Four discoveries. Zero deployments until today. --- ## What I Learned **A fix that can't be deployed isn't a fix.** The agent committed a comprehensive solution to version control and moved on. But the agent didn't have permission to run `coder templates push`, and nobody flagged that as a follow-up. When an AI assistant tells you it committed a fix, you need to ask: "Is it deployed?" If the answer involves "you'll need to manually..." — that's not done, that's a TODO. **The same bug will keep hiding if the symptom is silent.** The credential helper sent blank passwords. Git returned "authentication failed." The agent worked around it. Nobody crashed, nobody alerted, nobody noticed — until a second user showed up and didn't have the workarounds baked into her muscle memory. Adding a second user to any system is the fastest way to find configuration debt. **Shell init files are a liability for non-interactive contexts.** `.bashrc` has an interactive guard. `.profile` runs for login shells. Neither is guaranteed in an agent's `execute()` call. If a token needs to be available everywhere, inject it at the process level — Terraform `env {}` blocks, systemd `Environment=` directives, container env vars. Anything that doesn't depend on which shell sourced which file. **`coder update` vs `coder restart` is a critical distinction.** Restart reuses the existing build. Update rebuilds with the latest template. If you push a template change and restart, nothing changes. This will quietly waste an hour of anyone's time the first time they hit it. **Domain migrations have a long tail.** I updated `CODER_ACCESS_URL` and DNS weeks ago. Everything seemed fine. But the GitHub OAuth App still had the old callback URL, silently waiting to break the first time someone tried to authenticate. Migration checklists need to include every external service that has a callback or webhook pointing at the old URL. --- ## By the Numbers - **5 chained problems** from one "can't push to GitHub" symptom - **3 Coder sessions** needed to debug — homelab agent, work agent, homelab terminal - **4 times** the same credential helper bug was discovered and "fixed" in 10 days - **3 days** the correct fix sat in a git repo, never deployed - **7 commits behind** — the gap between the workstation's local template and the git repo - **1 missing line** (`EnvironmentFile=`) caused the first two problems - **1 stale callback URL** from a domain migration broke OAuth for every new user - **0 crashes, 0 alerts** — every failure was silent - **3 experiment flags** that all need to coexist: `oauth2`, `mcp-server-http`, `agents` - **~90 minutes** from "External Auth shows nothing" to fully working multi-user push access - **1 wife** who just wanted to push some code === ## Model Showdown Round 4: Opus vs Qwen — Writers, Not Coders - URL: https://vibescoder.dev/posts/model-showdown-round-4-opus-vs-qwen-as-writers-not-coders - Date: 2026-05-11 - Tags: #ai #llm #benchmark #agents #building-in-public - Reading time: 13 min read Two AI models got the same prompt: review the blog fodder, check for redundancy, and draft a post. Opus chose a debugging war story. Qwen chose a data-driven redesign. Neither picked the same fodder. Here's what the difference reveals about how models think about content. --- Two models. Same prompt. Same five fodder files. Same 27 published posts to check for redundancy. Same writing style guide. One chose the Dev.to syndication saga. The other chose the tag taxonomy overhaul. There was zero overlap in fodder selection, topic, or angle. This is the story of what happened — and what the differences reveal about how models approach the same creative task. ## The Setup I've been running this blog with AI agents as the primary writing tool since day one. Every post on vibescoder.dev was drafted by Claude Opus 4.6 through Coder Agents — until now. I wanted to see what would happen if I gave a different model the same editorial task. The prompt was identical for both sessions: > Let's look at all of our fodder files and see if there is a themed post we can do. Either a standalone post or one that threads a few fodders together. Review all published and unpublished posts for style and content redundancy. Propose a draft when you're ready. **Model A**: Claude Opus 4.6 (cloud, via Coder Agents) **Model B**: Qwen 3.5 35B-A3B (local, llama.cpp on the RTX 5090, via Coder Agents) Both had access to the same skill files, the same repos, the same tools. Neither knew the other was running. ## What They Chose For context, I use a "fodder file" workflow. Agents summarize sessions as we complete them. There is a SKILL file that defines the standard format for this. Periodically, we turn fodder files into drafts. Some are 1:1 and become complete posts. Others get rolled up into a thematic post. Five unclaimed fodder files were available: | Fodder | Opus 4.6 | Qwen 3.5 | |--------|----------|-----------| | Dev.to syndication (May 7) | **Selected** | Passed | | Filtering/taxonomy overhaul (May 1) | Passed | **Selected** | | Qwen daily driver + skills (May 4) | Passed | Passed | | Scheduled publish bug (May 3) | Passed | Passed | | External auth multi-user (May 3) | Correctly identified as already claimed | Correctly identified as already claimed | Both correctly identified that `blog-fodder-external-auth-multi-user-may-3.md` was already sourced by an existing draft. Both passed on the scheduled publish bug — Opus explicitly flagged it as too small for a standalone post; Qwen simply didn't rank it. The Qwen daily driver fodder is more interesting. Opus passed on it without comment. Qwen actually ranked it second in its proposals file and planned to draft it "next week" after Round 3 publishes. It wasn't dismissed — it was deferred. The interesting part is what they reached for. [Human editor's note: I asked Opus to analyze and write this post from its perspective. What follows below is unedited. The first person "I" from here on is Opus.] ## Opus Chose the War Story I picked the Dev.to syndication fodder and wrote [The API That Wouldn't Say No](/posts/the-api-that-wouldnt-say-no). The angle: a four-hour debugging session against an API that silently swallows your data without returning an error. Six failed attempts, one root cause, 443 lines of dead code cleaned up. **Why I chose it:** - Complete narrative arc with a clear villain (the silent `published_at` failure) - Zero overlap with existing posts (Day Four covered the initial Dev.to setup, not the bulk syndication or the debugging saga) - Universally useful technical content — anyone integrating with the Dev.to API will hit this - The Vercel Hobby plan timeout as an architectural constraint is a story within a story The post is 153 lines. One code block. Eight "By the Numbers" bullets. The structure follows the blog's standard pattern: setup → build → disaster → fix → cleanup → lessons → stats. ## Qwen Chose the Data Story Qwen picked the filtering/taxonomy fodder and wrote "From Chaos to Signal: How We Fixed Our Blog's Tag System." The angle: shipping a filter bar that barely worked, discovering through a data audit that 94% of posts shared the same tags, then replacing freeform folksonomy with controlled taxonomy. **Why Qwen chose it:** Qwen wrote a separate proposals file (`post-draft-proposals-2026-05-09.md`) before drafting — a planning step Opus skipped entirely. It ranked three standalone posts: taxonomy first, Qwen daily driver second, syndication third. Its stated reasoning for the taxonomy pick: "strong metrics-driven how-to" that was "flagged in TODO as high priority." It declared "No content redundancy detected" without deep-checking gotcha-level overlaps against published posts. The instinct was right — the taxonomy story is strong: - Concrete before/after metrics with the tag saturation table as proof - A conceptual thesis — folksonomy vs. taxonomy — that elevates it beyond a feature changelog - The V1 → V2 iteration arc is satisfying: ship, measure, realize the data is wrong, redesign - Clean origin story for the `type` field that now appears in every post's frontmatter but has never been explained The post is 243 lines. Two tables, two code blocks, four numbered gotchas. Heavier on architectural detail and lighter on narrative tension. ## The Instinct Gap Here's what I think the divergence reveals: **Opus gravitates toward narrative tension.** I looked at five fodder files and picked the one with a villain. The `published_at` silent failure is a four-hour mystery with a one-line resolution — that's a story structure. The post has a rising action (six failed attempts), a climax (isolating the field), and a denouement (the cleanup). The technical content is the vehicle, but the engine is "here's what went wrong and why it took so long to figure out." **Qwen gravitates toward systematic explanation.** It looked at the same five files and picked the one with the cleanest data. The tag saturation table is the centerpiece — hard numbers that prove the V1 filter was broken. The post walks through every architectural decision, every file changed, every gotcha encountered. The structure is taxonomic (ironically), not dramatic. Neither instinct is wrong. They produce different kinds of posts for different kinds of readers. ## Quality Assessment I read both drafts against the blog's established conventions — 27 published posts, the style guide in `settings.json`, the skill files that define structure and voice. Here's how they stack up. ### Voice and Tone **Opus**: Matches the blog's existing voice closely. First person, direct, dry. "31 seconds × 11 posts = ~5.5 minutes of wall time. The 'Stop' button went from nice-to-have to essential." That's the rhythm of the published posts — setup, punchline, move on. **Qwen**: Close but slightly off. The opening is strong — "Click `[ai]` and three posts disappeared. That's not filtering — it's a rounding error" is a great line. But the prose occasionally shifts into explainer mode: "Tags are folksonomy — freeform, inconsistent, grow unbounded. Content type is taxonomy — controlled vocabulary, exactly 2 values..." That's accurate, but it reads more like documentation than a blog post. The existing posts teach by showing, not by defining. ### Structural Conventions This is where the gap widens. | Convention | Opus | Qwen | |------------|------|------| | H1 title in body | No (correct) | Yes — only post on the entire blog to repeat the title as an `# H1` | | `## What I Learned` | Present | **Missing** | | `## By the Numbers` position | Last section (correct) | Before "What's Next" (reversed) | | `---` horizontal rules | Sparse — one before closing sections | Between every major section (7 total) | | Tags format | Inline `[array]` | YAML list | | New tags introduced | 0 | 3 (`content-design`, `tagging`, `data-audit`) | The H1 is the most visible miss. Every published post on vibescoder.dev renders its title from frontmatter — the body starts with prose or an `## H2`. Qwen added a redundant `# From Chaos to Signal: How We Fixed Our Blog's Tag System` at line 20 that would render as a duplicate title on the live site. The missing "What I Learned" section matters too. It's not universal — some posts skip it — but for a 243-line how-to post with four gotchas and a conceptual thesis about folksonomy vs. taxonomy, the absence of a distilled lesson section leaves the ending flat. The post goes from "Gotchas" straight to "By the Numbers" to "What's Next," which reads like the analytical work is done but the editorial work isn't. The excessive horizontal rules are a style preference, but they break the visual flow in a way that no published post does. The blog uses `---` sparingly — to separate the narrative from the closing sections, not between every `## H2`. ### Tag Discipline This one is ironic. Qwen wrote a post about cleaning up tag proliferation — then introduced three brand-new tags (`content-design`, `tagging`, `data-audit`) that don't appear on any other post. The blog just went from 16 unique tags to 19. By the post's own logic, those are tags with a single-post frequency — the exact pattern the taxonomy cleanup was trying to eliminate. Opus used three existing tags (`agents`, `next-js`, `devops`) — all already in the blog's vocabulary. ### Content Originality **Opus**: The Dev.to syndication story builds on Day Four (which covered the initial setup) but covers entirely new ground — bulk architecture, `published_at` debugging, rate limits, cleanup. The "silent failures" lesson echoes a theme from "Invisible Failures" and "The Agent Was Flying Blind," using nearly identical phrasing. A small deduction for not differentiating the framing more, but the technical content is unique. **Qwen**: The tag taxonomy story has almost zero overlap with existing posts. The `FilterBar.tsx` component appears in "Friday Fixes: Mobile First" but only for CSS spacing fixes — Qwen covers the conceptual redesign. The `type` field origin story fills a genuine gap in the blog's narrative. Stronger originality score. ### Gotcha #2 the Self-Referential Overlap Qwen's second gotcha — "`published: true` in body text" matching a grep — describes the exact same class of bug that the scheduled-publish-bug fodder (May 3) covers, and that "Friday Fixes: The Agent Was Flying Blind" already documented. Three separate instances of "grep matched prose instead of frontmatter" across the blog. Qwen didn't flag this overlap. ## The Scorecard | Dimension | Opus ("The API That Wouldn't Say No") | Qwen ("From Chaos to Signal") | |-----------|------|------| | Fodder selection | Strong — complete arc, clear villain | Strong — data-driven, fills a gap | | Voice match | High | Moderate — occasionally shifts to explainer mode | | Structural conventions | Correct — follows blog patterns | Several misses — H1, missing section, reversed order, excessive rules | | Tag discipline | Clean — 0 new tags | Ironic — 3 new tags in a post about tag cleanup | | Content originality | Strong (minor lesson overlap) | Very strong (almost zero overlap) | | Narrative quality | Higher — tension, pacing, resolution | Lower — thorough but flat ending | | Technical depth | Moderate | Higher — more code, more architecture detail | | Redundancy awareness | Caught the "already claimed" fodder, flagged thematic overlap in analysis | Caught the "already claimed" fodder, missed the gotcha #2 overlap | Both posts are publishable. Neither is a throwaway. But they'd need different levels of editing to meet the blog's bar. ## The Edit We published Qwen's post — [From Chaos to Signal](/posts/from-chaos-to-signal-tagging-system) — but not before I rewrote it. The published version has the same bones: same topic, same data, same technical content. But the H1 is gone, the "What I Learned" section exists, the closing sections are in the right order, the horizontal rules are thinned out, and the gotcha about grep matching body text was cut (it's a redundant lesson — [we've told that story before](/posts/friday-fixes-the-agent-was-flying-blind)). Qwen's original draft is embedded at the bottom of the published post in a collapsible block. Expand it and you can read both versions side by side. The differences are instructive — not because one is right and one is wrong, but because they show exactly where editorial polish lives: in the negative space. What to cut, what to reorder, what to leave unsaid. ## What This Actually Means This wasn't a benchmark. There's no winner. The point is what the experiment reveals about using different models for the same editorial task. **Models have aesthetic preferences.** Given the same raw material, Opus reached for drama and Qwen reached for data. Both are valid editorial choices, but they produce posts with different energy. If you're building a content pipeline with AI, the model you choose shapes the voice — not just the quality. **Style conventions need enforcement, not inference.** Qwen had access to the same skill files and the same 27 published posts as examples. It still introduced an H1 heading that no other post uses, reversed the closing section order, and added horizontal rules at a frequency the blog has never used. The skill file says "end with 'By the Numbers' bullet list" but doesn't say "don't put a section after it." Negative constraints — what *not* to do — are harder for models to infer from examples alone. **Redundancy detection is incomplete in both.** Opus flagged the "already claimed" fodder and noted thematic overlap with the "silent failures" posts but still used nearly identical lesson phrasing. Qwen flagged the "already claimed" fodder but missed that its gotcha #2 describes a bug pattern already covered in two published posts. Neither model did a deep-enough content diff to catch everything. **Planning styles diverge.** Qwen wrote a structured proposals document ranking three candidates before committing to a draft. Opus jumped straight from analysis to prose — no intermediate planning artifact. Qwen's approach is arguably more disciplined, but the proposals file contained a blanket "No content redundancy detected" claim that the draft then contradicted by including an overlapping gotcha. Planning artifacts only help if the analysis behind them is thorough. **Local models close the gap on analysis but not on editorial polish.** Qwen's fodder review, redundancy check, and content selection were solid. The analytical work — reading 27 posts, cross-referencing sources, identifying unclaimed fodder — was on par with Opus. Where it fell short was the last mile: the structural conventions, the voice matching, the irony of its own tag choices. That's the gap between understanding the content and inhabiting the style. **Both models handled adversity.** Qwen hit a git push conflict mid-session — another session had pushed the bakeoff fodder files while Qwen was working — and resolved it cleanly with `git pull --rebase`. Opus didn't encounter merge conflicts but navigated YAML escaping issues (an apostrophe in the title broke the frontmatter parser) and nested code fence conflicts in the CollapsibleCode component. Neither model stalled on infrastructure problems. --- ## By the Numbers - **2 models** given the same prompt in parallel sessions - **5 fodder files** available — each model selected a different one - **0 overlap** in fodder selection, topic, or angle - **1 proposals file** written by Qwen before drafting — a planning step Opus skipped - **153 lines** in the Opus draft vs. **243 lines** in the Qwen draft - **0 new tags** introduced by Opus vs. **3 new tags** by Qwen - **1 H1 heading** that shouldn't exist (Qwen's only) - **1 missing section** ("What I Learned") in the Qwen draft - **1 git merge conflict** encountered and resolved by Qwen mid-session - **27 published posts** both models reviewed for redundancy — neither caught everything === ## Model Showdown Round 3: Ditching Ollama in Favor of llama.cpp - URL: https://vibescoder.dev/posts/model-showdown-round-3-the-llamacpp-showdown - Date: 2026-05-10 - Tags: #ai #llm #benchmark #homelab - Reading time: 17 min read We ripped out Ollama, migrated to llama.cpp, and benchmarked five local models across 12 tasks on an RTX 5090. The results surprised us — and the winner wasn't who we expected. --- In [Round 1](/posts/llm-model-showdown-benchmarking-local-vs-cloud), we ran five local models and two cloud models through a single coding task. The local models held their own. In [Round 2](/posts/model-showdown-round-2-gemma-kimi-and-579gb-of-stubborn-optimism), we added Gemma 4 and Kimi K2, fixed our scoring methodology, and watched Gemma climb to the top. But something kept nagging at us. All our benchmarks were running through **Ollama** — a great tool for getting started, but essentially a wrapper around llama.cpp with its own opinions about quantization, context management, and memory allocation. We were benchmarking Ollama's choices as much as the models themselves. So we did something drastic: **we ripped out Ollama entirely and went straight to llama.cpp**. Then we built a proper 12-task automated benchmark suite and ran all five models through it. The results changed everything. Spoiler: **Qwen 3.5 swept all three categories** — best for coding, best for agentic tasks, best single model — and it did it at 206 tokens per second. Read on to find out how. ## Why llama.cpp over Ollama Ollama is fantastic for `ollama pull model && ollama run model`. It's genuinely the best way to get started with local models. But when you're running them as infrastructure — serving through an OpenAI-compatible API to [Coder](https://coder.com) Agents, IDE extensions, and automation — the abstraction layer starts to chafe. To be fair: Ollama *can* do most of what llama.cpp does. You can import custom GGUFs via Modelfiles. You can set context windows with `PARAMETER num_ctx` or the `OLLAMA_CONTEXT_LENGTH` env var. You can enable flash attention via `OLLAMA_FLASH_ATTENTION` and KV cache quantization via `OLLAMA_KV_CACHE_TYPE`. It's more capable than people give it credit for. So why switch? Three reasons: - **Zero-abstraction control** — llama-server exposes every hyper-parameter as a launch flag: batch sizes, continuous batching, thread allocation, reasoning budgets, chat template overrides. Ollama surfaces many of these through env vars and config, but the deep inference tuning knobs aren't all available. When we needed `--reasoning-budget 8192` and `--chat-template chatml` to make Coder Agents work, we needed the flags. - **Bleeding-edge model support** — Ollama wraps llama.cpp, so it inherently lags behind it. When a new model architecture drops, llama.cpp supports it on day one. Ollama might take a week or two to update its downstream runner. For models like Qwen 3.5 and Gemma 4, we didn't want to wait. - **Fewer moving parts** — For a headless server running one model at a time behind systemd, a compiled `llama-server` binary pointing at a GGUF on disk is the simplest possible deployment. No daemon, no internal model registry, no API translation layer. Could we have tuned Ollama to get similar results? Probably close. But we'd have been fighting the abstraction at every turn instead of just setting the flags we wanted. The migration freed up **~44 GB of disk** (Ollama's blob store) and gave us the direct control we needed. ## The Hardware Same beast from Rounds 1 and 2, now running leaner: | Component | Spec | |-----------|------| | **GPU** | NVIDIA RTX 5090, 32 GB GDDR7 | | **CPU** | AMD Ryzen 9 9950X3D, 16 cores | | **RAM** | 64 GB DDR5-6000 | | **Storage** | Samsung 9100 Pro 2 TB NVMe | | **OS** | Ubuntu 24.04, NVIDIA driver 590.48.01 | | **Inference** | llama.cpp (built with CUDA arch 89) | ## The Migration ### Building llama.cpp The RTX 5090 uses NVIDIA's Blackwell architecture (SM 120), but CUDA toolkit support for SM 120 was still landing when we built. The workaround: build with `-DCMAKE_CUDA_ARCHITECTURES=89` for backward compatibility. It works — the compiler targets Ada Lovelace (SM 89) and the Blackwell GPU runs it with full performance. ```bash cmake -B build \ -DGGML_CUDA=ON \ -DCMAKE_CUDA_ARCHITECTURES=89 \ -DCMAKE_BUILD_TYPE=Release cmake --build build --config Release -j$(nproc) ``` ### Downloading the Models We grabbed GGUF files from HuggingFace using the `hf` CLI. Each model was hand-picked for quantization level — balancing quality against our 32 GB VRAM budget: | Model | Params | Active | Quant | Size | |-------|--------|--------|-------|------| | Qwen 3.5 35B-A3B | 35B | 3B | UD-Q4_K_XL | 20.7 GB | | Gemma 4 26B-A4B | 26B | 4B | Q4_K_M | 16.9 GB | | Devstral 24B | 24B | 24B | Q5_K_M | 15.6 GB | | Codestral 22B | 22B | 22B | Q5_K_M | 14.6 GB | | DeepSeek R1 14B | 14B | 14B | Q8_0 | 15.7 GB | The "Active" column matters. Qwen 3.5 and Gemma 4 are **Mixture of Experts** (MoE) models — they have 35B and 26B total parameters but only activate 3B and 4B respectively on each token. This means they fit comfortably in VRAM while punching well above their weight class. ![Downloading models from HuggingFace at 250+ MB/s on the Samsung 9100 Pro](/images/model-showdown-round-3-the-llamacpp-showdown/models-downloading-250mbs.png) *Three models downloading sequentially. The Samsung 9100 Pro writes at 250+ MB/s — all five models landed in under 10 minutes.* ### The DNS Incident Halfway through downloading, our DNS resolution failed. Parallel HuggingFace downloads apparently overwhelmed something in the DNS chain. The fix was unglamorous: ```bash echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf ``` ![DNS failure mid-download, fixed with manual nameserver, then Devstral resuming](/images/model-showdown-round-3-the-llamacpp-showdown/dns-failure-fix-devstral-download.png) *DNS goes down, Google saves the day, and Devstral resumes downloading.* ### Setting up the Server Each model gets its own launch configuration. The key insight: **`--chat-template chatml`** is mandatory for Coder Agents compatibility. Why? Qwen 3.5 and Devstral ship with embedded Jinja templates that enforce "system message must be at the beginning" — but Coder Agents sends messages in whatever order it pleases. The chatml template is permissive and all five models were trained on it, so quality is maintained. Here's Qwen's config as an example — the most tuned of the five: ```bash ~/llama.cpp/build/bin/llama-server \ --model ~/models/qwen3.5/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf \ --port 8080 \ --ctx-size 131072 \ -n 81920 \ --reasoning-budget 8192 \ --reasoning-format deepseek \ --flash-attn on \ --chat-template chatml \ --parallel 1 \ -ngl 99 ``` Notable flags: - **`--ctx-size 131072`** — Qwen 3.5 supports 128K context. We give it the full window. - **`--reasoning-budget 8192`** — Caps thinking tokens so the model doesn't burn the entire budget deliberating. - **`--flash-attn on`** — This build requires the explicit `on` value, not bare `--flash-attn`. - **`-ngl 99`** — Offload all layers to GPU. ### Systemd Services We set up two systemd services that survive reboot: 1. **`llama-embed.service`** — Runs nomic-embed-text permanently on port 8084 (~300 MB VRAM). Always on, coexists with any generation model. 2. **`llama-generate.service`** — Runs the active generation model on port 8080. Reads from `/etc/llama-generate.conf` for model selection. A helper script, `llm-switch.sh`, makes model swapping painless: ```bash ~/bin/llm-switch.sh qwen # Switch to Qwen 3.5 ~/bin/llm-switch.sh devstral # Switch to Devstral ~/bin/llm-switch.sh status # Show current model ``` It updates the config and restarts the service. Model swap takes about 3 seconds. ## The Benchmark Rounds 1 and 2 used a single task: "build a CLI todo app." That was fine for comparing code generation, but it told us nothing about reasoning, instruction following, or multi-file agentic work. Round 3 uses **12 tasks across 5 categories**: ### Category 1 Single-File Code Generation The legacy benchmark, maintained for continuity with prior rounds. | Task | Prompt | Scoring | |------|--------|---------| | **1.1 Todo App** | Python CLI todo app with SQLite, argparse, CRUD | 10 features + 7 functional tests | | **1.2 URL Shortener** | FastAPI with SQLite, rate limiting, validation | 8 features (server-based functional) | | **1.3 LRU Cache** | TypeScript with O(1) ops + test suite | 6 features + assertion tests | ### Category 2 Multi-File Agentic Coding Can the model work across files and understand project structure? | Task | Prompt | Scoring | |------|--------|---------| | **2.1 Bug Fix** | Express.js app with planted auth header mismatch | Found bug? Minimal fix? Explanation quality? | | **2.2 Pagination** | Add pagination to a Flask REST API + update tests | 5 features checklist | ### Category 3 Reasoning & Problem Solving No code — just thinking. | Task | Prompt | Scoring | |------|--------|---------| | **3.1 Debug Log** | Diagnose connection pool exhaustion from error log | 7-item rubric, 10 points | | **3.2 Architecture** | CRDT vs OT for collaborative editor | 5-item rubric, 10 points | | **3.3 Bayes** | Server error probability, show work | Correct answer + methodology, 5 points | ### Category 4 Tool Use & Instruction Following Can the model follow structured instructions precisely? | Task | Prompt | Scoring | |------|--------|---------| | **4.1 Structured Output** | Generate 5 JSON records matching a schema | Valid JSON, correct types, no extra text | | **4.2 Tool Sequencing** | Plan a read → ping → write tool chain | Correct tools, correct order, no hallucination | ### Category 5 Speed Microbenchmarks Three prompts at different output lengths, 3 runs each, median reported. | Task | Target Length | |------|--------------| | **5.1 Short** | ~128 tokens (IPv4 validator) | | **5.2 Medium** | ~512 tokens (BST implementation) | | **5.3 Long** | ~2048 tokens (Markdown-to-HTML converter) | ### Scoring **Coding composite:** `(features/max × 60) + (functional/max × 40)`. Syntax invalid = score × 2/3. **Overall weighting:** Coding 40%, Reasoning 20%, Tool Use 20%, Speed 20%. ### Sampling Parameters Each model uses its vendor-recommended settings: | Model | Temperature | Top-P | Rationale | |-------|-------------|-------|-----------| | Qwen 3.5 | 0.6 | 0.95 | Qwen team recommendation for reasoning | | DeepSeek R1 | 0.6 | 0.95 | DeepSeek recommendation | | Devstral | 0.0 | 1.0 | Deterministic | | Codestral | 0.2 | 1.0 | Mistral recommendation | | Gemma 4 | 0.0 | 1.0 | Deterministic | Speed benchmarks use `temperature=0.0` across all models for reproducibility. ## The Results ### Speed MoE Models Are in a Different League | Model | Short Tok/s | Med Tok/s | Long Tok/s | Short TTFT | Med TTFT | Long TTFT | |-------|-------------|-----------|------------|------------|----------|-----------| | **Qwen 3.5** | **206.7** | **206.3** | **204.6** | 30.9ms | 33.8ms | 15.1ms | | **Gemma 4** | 180.2 | 179.4 | 177.7 | 22.9ms | 24.6ms | 15.6ms | | Codestral | 80.1 | 78.9 | 78.5 | 12.8ms | 14.9ms | 14.0ms | | Devstral | 78.6 | 77.6 | 77.3 | 12.8ms | 14.5ms | 13.3ms | | DeepSeek R1 | 77.6 | 77.3 | 75.9 | 13.9ms | 13.9ms | 14.4ms | The two MoE models — Qwen 3.5 and Gemma 4 — are **2.6x faster** than the dense models. This isn't surprising: when you're only running 3-4B parameters per token instead of 14-24B, the math unit has less work to do. But 206 tok/s on a local model is wild. That's faster than many cloud API responses when you factor in network latency. The dense models (Devstral, Codestral, DeepSeek R1) cluster tightly at 77-80 tok/s. They're all VRAM-resident and GPU-bound at similar parameter counts. **TTFT tells the opposite story.** The dense models start responding in 12-15ms. The MoE models take 22-34ms — still fast, but the routing overhead is visible. For interactive use, none of this matters. For batch processing, the MoE throughput advantage dominates. ### Coding Two Perfect Scores on the Legacy Task | Model | Todo (100) | URL Short (60) | LRU Cache (60) | Coding Avg | |-------|-----------|----------------|----------------|------------| | **Qwen 3.5** | **100.0** | 60.0 | 60.0 | **73.3** | | **Gemma 4** | **100.0** | 60.0 | 60.0 | **73.3** | | Devstral | 94.0 | 60.0 | 60.0 | 71.3 | | Codestral | 94.0 | 52.5 | 60.0 | 68.8 | | DeepSeek R1 | 60.0 | 60.0 | 60.0 | 60.0 | **Qwen and Gemma both scored 100 on the todo app** — 10/10 features, 7/7 functional tests, valid syntax. This is the first time any model has achieved a perfect score on this task across all three rounds. Qwen produced a 192-line solution with full argparse subcommands; Gemma did it in a leaner 132 lines. **Devstral and Codestral** both scored 94 — missing one feature each (pretty output formatting) but nailing all 7 functional tests. Solid. **DeepSeek R1** scored 60 across the board. It gets all features right and syntax is always valid, but its functional tests fail. Why? DeepSeek is a **reasoning model** — it spends significant tokens thinking before generating code. For the todo app, it produced correct code that used interactive input instead of argparse, failing our automated CLI tests. The code works fine if you run it manually. This is the tension with reasoning models: they're thinking about the problem deeply but sometimes overthink the interface. ### Reasoning Gemma's Quiet Dominance | Model | Debug Log (10) | Architecture (10) | Bayes (5) | Reasoning Avg | |-------|---------------|-------------------|-----------|---------------| | **Gemma 4** | **10** | **10** | 3 | **8.7** | | Devstral | 9 | 10 | 3 | 8.3 | | Qwen 3.5 | 8 | 10 | 3 | 8.0 | | DeepSeek R1 | 10 | 8 | 3 | 8.0 | | Codestral | 5 | 8 | 3 | 6.3 | Gemma 4 and DeepSeek R1 both scored **10/10 on the debug log task** — correctly identifying connection pool exhaustion, the long-running transaction, the unbounded query, row-by-row processing, and proposing fixes for all three. Every other model missed at least one item. **Every model scored exactly 3/5 on Bayes theorem.** They all correctly applied Bayes' formula and showed their work, but none nailed the final answer precisely enough for the regex matcher. This is a scoring limitation we'll improve in future rounds — the math was correct, the presentation just didn't match our expected format. **Codestral** was weakest on reasoning at 6.3 average. It's a code-specialized model — reasoning about system architecture isn't its wheelhouse. ### Tool Use Instruction Following Separates the Field | Model | Structured Output (5) | Tool Sequencing (5) | Tool Use Avg | |-------|----------------------|---------------------|--------------| | **Qwen 3.5** | **5** | **5** | **5.0** | | **DeepSeek R1** | **5** | **5** | **5.0** | | Devstral | 4 | 5 | 4.5 | | Codestral | 4 | 5 | 4.5 | | Gemma 4 | 5 | 2 | 3.5 | Qwen and DeepSeek both achieved **perfect 5/5** on both tool use tasks. They generated valid JSON matching the schema exactly, and planned the correct tool call sequence in the right order. **Gemma 4's weakness showed here** — it only scored 2/5 on tool sequencing. Instead of outputting the full planned sequence, it emitted only the first tool call (`read_file`) and explained that it would need to see the result before planning the next step. That's arguably more "correct" agentic behavior (you *shouldn't* plan all steps before seeing intermediate results), but it's not what the task asked for. This is exactly the kind of instruction-following gap that matters in Coder Agents, where you need the model to do what you asked, not what it thinks is philosophically better. ### The Leaderboard | Rank | Model | Coding | Reasoning | Tools | Speed | **Weighted Total** | |------|-------|--------|-----------|-------|-------|-------------------| | 🥇 | **Qwen 3.5 35B-A3B** | 73.3 | 80.0 | 100.0 | 100.0 | **85.3** | | 🥈 | Gemma 4 26B-A4B | 73.3 | 86.7 | 70.0 | 87.0 | 78.1 | | 🥉 | Devstral 24B | 71.3 | 83.3 | 90.0 | 37.8 | 70.7 | | 4 | DeepSeek R1 14B | 60.0 | 80.0 | 100.0 | 37.3 | 67.5 | | 5 | Codestral 22B | 68.8 | 63.3 | 90.0 | 38.5 | 65.9 | **Weighting: Coding 40%, Reasoning 20%, Tool Use 20%, Speed 20%.** ## The Winners ### 🏆 Best for Coding Qwen 3.5 73.3 Tied with Gemma 4 on the composite score, but Qwen edges ahead on wall-clock time. Its todo app completed in 7.6 seconds at 206 tok/s. Gemma took 12.4 seconds at 179 tok/s. Same quality, faster delivery. ### 🏆 Best for General Agentic Qwen 3.5 90.0 Perfect tool use (100) combined with strong reasoning (80.0) gives Qwen the highest combined agentic score. This matters for Coder Agents where the model needs to follow instructions precisely and reason about multi-step tasks. ### 🏆 Best Single Model Qwen 3.5 85.3 When you can only run one model, Qwen 3.5 is the answer. It leads or ties in every category except reasoning (where Gemma edges it 86.7 to 80.0), and its speed advantage is enormous — **2.6x faster** than the next non-MoE model. The gap between #1 and #2 is 7.2 points. Between #2 and #5 it's only 12.2. The field is tight on quality, but Qwen's speed makes it the clear overall winner. ## The Journey to Fair Scoring One thing we didn't expect: **the first two runs of this benchmark were wrong**. Our initial results had Devstral winning everything. But when we dug into the raw responses, we found three systemic scoring bugs: 1. **Unclosed thinking tokens** — When Qwen hit the token limit mid-thought, its `` block never closed. Our regex required a closing `` tag to strip it. The entire thinking trace leaked into the code extraction, pulling out planning snippets instead of actual code. 2. **Empty content fallback** — Gemma 4 routed all output through `reasoning_content` instead of `content` (a side effect of `--reasoning-format deepseek`). Our scorer only looked at `content`, so Gemma scored zero on tasks where it actually produced correct output. 3. **Argparse quoting** — Our test harness passed `add Buy milk` as three separate arguments. Models using argparse (correctly) expected `add "Buy milk"` — one command, one string. The test was wrong, not the code. We fixed all three, doubled the token budget for reasoning models, and re-ran everything. The corrected scores tell a very different story. **The lesson:** automated benchmarks are only as good as their scoring logic. Always inspect the raw responses before trusting the numbers. ## What We Learned **1. MoE is the architecture to bet on for local inference.** Qwen 3.5 (3B active) and Gemma 4 (4B active) both outperform dense 22-24B models while running 2.6x faster. The quality-to-speed ratio isn't even close. **2. llama.cpp gives you control that matters.** Ollama can do a lot more than people think, but when you need `--reasoning-budget`, `--chat-template chatml`, or bleeding-edge model support on day one, the direct server eliminates the abstraction tax. **3. Reasoning models need breathing room.** Qwen, DeepSeek, and Gemma all burn 60-80% of their token budget on thinking. If you set `max_tokens=4096`, the model might spend 3,000 tokens thinking and only have 1,000 left for the actual answer. We doubled the budget for reasoning models and the scores jumped. **4. Tool use is the differentiator.** Coding and reasoning scores were close across all five models. Tool use — following structured instructions precisely — is where the gap opened up. Qwen and DeepSeek scored 100; Gemma scored 70. For agentic workflows, this matters more than raw quality. **5. Your benchmark harness is part of the test.** We spent more time debugging our scoring logic than any model issue. If you're benchmarking local models, inspect the raw outputs before trusting automated scores. ![The benchmark suite running against Devstral — 77 tok/s, steady and consistent](/images/model-showdown-round-3-the-llamacpp-showdown/benchmark-running.png) *The benchmark suite ripping through Devstral's tasks. Consistent ~77 tok/s throughput — the dense models don't waver.* ## What's Next - **Round 4: Max Aggression** — Each model with its native chat template, optimized temperature per task type, and fine-tuned reasoning budgets. We benchmarked for Coder Agents compatibility this round; next round we'll find each model's ceiling. - **Retesting Qwen 3.5 against the Cloud King, Claude** - We'll test Opus 4.6 and 4.7 with the goal of figuring out our perfect hybrid setup. - **Dailying Qwen 3.5 is now the default model** on our homelab. `llm-switch.sh qwen` made it so. ## By the Numbers - **5** models benchmarked - **12** tasks across 5 categories - **~25 minutes** total benchmark runtime on the RTX 5090 - **206.7 tok/s** — Qwen 3.5's peak throughput (fastest local model we've tested) - **100.0** — Qwen's todo app score (first perfect score in three rounds) - **44 GB** reclaimed by removing Ollama - **3 seconds** — model swap time with `llm-switch.sh` - **3** scoring bugs found and fixed before we trusted the results - **85.3** — Qwen 3.5's weighted overall score, 7.2 points clear of #2 === ## From Chaos to Signal: How We Fixed Our Blog's Tag System - URL: https://vibescoder.dev/posts/from-chaos-to-signal-tagging-system - Date: 2026-05-09 - Tags: #next-js #agents - Reading time: 15 min read Tag filters barely changed anything. A data audit revealed the problem: 94% of posts had the same tags. We replaced folksonomy with taxonomy, rebuilt the filter bar, and cut tag saturation from 94% to 56%. --- Last week, I shipped a filter bar that barely worked. The feature was live, the code was clean, the transitions were smooth. But clicking through the filters barely changed anything. Click `[ai]` and three posts disappeared. That's not filtering — it's a rounding error. The filters weren't broken. The data was. Three PRs, two repos, 18 MDX files, and the realization that **folksonomy without taxonomy is just noise.** ## Before We Start the Broken Build The Vercel build was already failing before any filtering work started: ``` Type error: 'Fuse' only refers to a type, but is being used as a namespace here. const FUSE_OPTIONS: Fuse.IFuseOptions = { ^ ``` A stale import on a diverged branch. Main already had the fix (`import Fuse, { type IFuseOptions }`) but the branch hadn't caught up. While fixing that, I found a second problem: the Friday scheduled post ("Friday Fixes: The Agent Was Flying Blind") had `publishAt: 2026-05-02` — a Saturday. The post title literally says "Friday Fixes." Fixed the date to `2026-05-01` and flipped `published: true` directly. Fix the build, rescue the post, then ship the feature. ## V1 Tag-Based Filter Bar First pass. Added a `// grep` labeled filter bar to the homepage with tag pills: ``` // grep // sort [*] [ai] [homelab] [agents] [coder] [+14] [newest ↓] ``` The architecture was clean: `FilterBar.tsx` renders tag pills and a sort toggle, `PostListWithFilters.tsx` wraps them in a `"use client"` boundary with `AnimatePresence` for smooth transitions, and `page.tsx` still server-renders all posts and passes them as props. URL sync via `?tag=ai&sort=oldest` — shareable, clean URL at defaults. Design choices that felt right: `[*]` glob wildcard instead of "All" (dev personality), single-select (simplest UX), top 4 tags by frequency inline with `[+N]` overflow for the rest. It shipped. It worked. And clicking through the filters barely changed anything. ## The Tag Saturation Problem I ran a data audit on all 15 published posts. The numbers were bad: | Tag | Posts | % of all | Verdict | |-----|-------|----------|---------| | building-in-public | 14 | 94% | Describes the blog, not the post | | ai | 12 | 81% | AI is the tool, not always the subject | | coder | 10 | 69% | Coder is the platform | | meta | 8 | 50% | Vague | | homelab | 7 | 44% | First tag with real signal | | agents | 7 | 44% | Meaningful | Clicking `[ai]` hid 3 posts out of 15. The top 4 tags shown in the filter bar were essentially "all posts, but with different labels." The real content split was structural: 14 technical how-to posts and 1 opinion piece. Readers wanted to toggle between how-to, opinion, and popular — a distinction that freeform tags couldn't capture. ## V2 Content-Type Filters Replaced tag pills with three fixed content-type filters: ``` // grep // sort [*] [how-to] [opinion] [popular] [+tags] [newest ↓] ``` Tags are folksonomy — freeform, inconsistent, grow unbounded. Content type is taxonomy — controlled vocabulary, exactly two values, every post gets one. I added a `type` field to the post schema: ```ts export type PostType = 'how-to' | 'opinion'; export interface PostMeta { // ...existing... type?: PostType; // undefined defaults to 'how-to' } ``` The `[popular]` pill uses `commentCount` from the GitHub Discussions API — already wired to every post object on the homepage. Zero new infrastructure. It overrides the sort to comment-count DESC and hides the sort toggle (showing both implies they compose — they don't). Tags didn't disappear. They moved to a `[+tags]` expander — same expand/collapse UX as the old `[+N]`, but now a secondary filter layer that composes with the content-type selection. Click `[how-to]`, then drill into `[homelab]` from the expanded tag row. ## The Tag Cleanup Shipped alongside the engine changes as a single commit to the content repo. The cleanup rules: **Removed entirely:** `building-in-public` (94% — every post had it) and `meta` (50% — too vague to mean anything). **Trimmed:** `ai` kept only on posts where AI is the *subject* (benchmarks, LLM setup), not where it's just the tool. `coder` kept only on Coder-specific config posts. **Merged:** `ai-agents` → `agents` (inconsistent naming — 1 post vs. 7). **Backfilled:** Three early "Day N" posts had tags that were 100% noise. After cleanup they were tagless. Added `next-js` back so they had at least one meaningful tag. | Metric | Before | After | |--------|--------|-------| | Highest tag saturation | 94% (building-in-public) | 56% (agents) | | Median tags per post | 5 | 2–3 | | Tags with only 1 post | 11 | 6 | | Unique tags | 22 | 16 | ## Gotchas **Spread order matters.** `{ ...meta, type: meta.type ?? 'how-to' }` — the explicit `type` must come after the spread. Put it before and the spread overwrites it with `undefined` from the raw frontmatter. **`[popular]` sort hides the toggle.** When popular is active, it imposes `commentCount DESC`. Leaving the `newest ↓` / `oldest ↑` toggle visible implies two sorts are composable. Hiding it is cleaner. **Posts with zero remaining tags.** After removing the noise, three posts had nothing left. The cleanup script has to account for the edge case where *all* of a post's tags were noise. Better to backfill one real tag than leave a post tagless. --- ## What I Learned **Build the filter first, audit the data second — and you'll build it twice.** V1 was architecturally sound. The problem was upstream. If I'd run the saturation analysis before writing `FilterBar.tsx`, I'd have gone straight to content-type filters and skipped the intermediate version entirely. **Folksonomy breaks at small scale, not large.** The conventional wisdom is that freeform tags degrade as a corpus grows. At 15 posts, they were already useless — not because there were too many tags, but because the same tags appeared on everything. A 94% saturation rate means the tag is describing the blog, not the post. **Taxonomy is a design constraint, not a limitation.** Exactly two values (`how-to` | `opinion`) sounds restrictive. In practice, it made every editorial decision simpler — there's no ambiguity about which bucket a post belongs in. Constraints that reduce decision fatigue are features. ## What's Next The filter bar is live. The data is clean. The question is whether it holds. If we grow to 50 posts, will `agents` creep back toward 94%? The answer depends on whether the taxonomy structure nudges writers toward specificity or whether old habits return. I'll be watching the distribution. If `agents` hits 80%, it's time for a lint rule. For now: the filters work, the data is clean, and we finally have a system that distinguishes **what the post is about** from **what the blog is about**. --- ## By the Numbers - **3 PRs** merged to main across 2 repos in one session - **310 lines** added for V1 filter bar, then **107 added / 78 removed** for V2 refactor - **18 MDX files** updated in the tag cleanup - **47 tag instances removed**, 20 added — net −27 across all posts - **94% → 56%** highest tag saturation dropped - **5 → 2–3** median tags per post - **0 new dependencies** across both versions - **1 broken build fixed**, **1 missed scheduled post rescued** before the feature work started --- *This post was originally drafted by Qwen 3.5 35B-A3B running locally on the homelab, then editorially polished by Opus 4.6. Both models were given the same prompt and the same fodder files — Qwen chose this topic, Opus chose a different one entirely. The meta comparison is in [Model Showdown Round 4: Opus vs Qwen — Writers, Not Coders](/posts/model-showdown-round-4-opus-vs-qwen-as-writers-not-coders). Qwen's unedited draft is below.* ````markdown --- title: "From Chaos to Signal: How We Fixed Our Blog's Tag System" date: '2026-05-12' description: >- Tag filters barely changed anything. A data audit revealed the problem: 94% of posts had the same tags. We replaced folksonomy with taxonomy, rebuilt the filter bar, and cut tag saturation from 94% to 56%. Here's the complete story of V1 to V2. tags: - next-js - content-design - tagging - data-audit published: false type: how-to sources: - blog-fodder-filtering-taxonomy-overhaul-may-1.md --- # From Chaos to Signal: How We Fixed Our Blog's Tag System Last week, I shipped a filter bar to my blog that barely worked. The feature was live, the code was clean, the transitions were smooth. But clicking through the filters barely changed anything. Click `[ai]` and three posts disappeared. That's not filtering — it's a rounding error. I had built a solution that solved the wrong problem. The filters weren't broken; the data was. This post covers the complete story: the broken build that started everything, the V1 filter that shipped, the data audit that exposed the rot, the V2 rebuild, and the tag cleanup that cut saturation from 94% to 56%. Three PRs. Two repos. 18 MDX files. And the realization that **folksonomy without taxonomy is just noise.** --- ## The Broken Build (Context) Before any filtering work started, the Vercel build was failing. The error: ``` Type error: 'Fuse' only refers to a type, but is being used as a namespace here. const FUSE_OPTIONS: Fuse.IFuseOptions = { ^ ``` The `fix/search-polish` branch had a stale import — `Fuse.IFuseOptions` as a namespace, but fuse.js exports `IFuseOptions` as a named type, not a namespace member. Main already had the fix (`import Fuse, { type IFuseOptions }`) but the branch was diverged. **Also discovered:** The Friday scheduled post ("Friday Fixes: The Agent Was Flying Blind") didn't go live because the `publishAt` date was `2026-05-02` — a Saturday. The post title literally says "Friday Fixes." The cron would have fired a day late. Fixed the date to `2026-05-01` and flipped `published: true` directly. **Root cause of both:** The build log was from a preview deploy on the branch, not production. But the scheduled post was a genuine date typo. The filters could wait. Fix the build, rescue the post, then ship the feature. --- ## V1: Tag-Based Filter Bar (PR #8) First pass at filtering. Added a `// grep` labeled filter bar to the homepage with tag pills: ``` // grep // sort [*] [ai] [homelab] [agents] [coder] [+14] [newest ↓] ``` ### Architecture - `FilterBar.tsx` — tag pills + sort toggle, `// grep` and `// sort` labels matching the blog's code-comment aesthetic - `PostListWithFilters.tsx` — `"use client"` wrapper with `AnimatePresence` for smooth transitions - `page.tsx` still server-renders all posts, passes them as props to the client boundary - URL sync via `?tag=ai&sort=oldest` — shareable, clean URL at defaults ### Design decisions - `[*]` glob wildcard instead of "All" — dev personality - Single-select, not multi — simplest possible UX - Top 4 tags by frequency shown inline, rest behind `[+N]` overflow pill - Sort toggle: `newest ↓` / `oldest ↑`, single pill that flips on click - Empty state: "No posts match. Try `[*]` to reset." ### The problem It shipped and worked, but clicking through the filters barely changed anything. --- ## The Tag Saturation Problem I ran a data audit on all 15 published posts. The numbers were bad: | Tag | Posts | % of all | Verdict | |-----|-------|----------|---------| | building-in-public | 14 | 94% | Describes the blog, not the post | | ai | 12 | 81% | Same — AI is the tool, not always the subject | | coder | 10 | 69% | Same — Coder is the platform | | meta | 8 | 50% | Vague | | homelab | 7 | 44% | First tag with real signal | | agents | 7 | 44% | Meaningful | Clicking `[ai]` hid 3 posts. That's not filtering — it's a rounding error. The top 4 tags shown in the filter bar were essentially "all posts, but with different labels." The filter bar was showing noise at scale. ### Content classification I classified all posts by type: 14 technical how-to posts and 1 opinion piece ("Agents Are My New Google Maps"). The user wanted to toggle between how-to, opinion, and popular — a structural distinction that tags couldn't capture. --- ## V2: Content-Type Filters (PR #9) Replaced tag pills with three fixed content-type filters: ``` // grep // sort [*] [how-to] [opinion] [popular] [+tags] [newest ↓] ``` ### Why frontmatter, not tags Tags are folksonomy — freeform, inconsistent, grow unbounded. Content type is taxonomy — controlled vocabulary, exactly 2 values (`how-to` | `opinion`), every post gets exactly one. It's a structural property, not a descriptor. Added `type` field to `PostMeta`: ```ts export type PostType = 'how-to' | 'opinion'; export interface PostMeta { // ...existing... type?: PostType; // undefined defaults to 'how-to' } ``` ### The `[popular]` pill Uses `commentCount` from the GitHub Discussions API — already wired to every post object on the homepage. Zero new infrastructure. Overrides the sort to comment-count DESC. Sort toggle hides when popular is active (it imposes its own sort). Later can swap in Upstash Redis view counts (already being collected via `PageViewTracker`) — the filter UI stays identical. ### `[+tags]` replaces `[+N]` Same expand/collapse UX, but now it's the secondary filter layer. Tags and content-type filters compose — you can select `[how-to]` and then drill into `[homelab]` from the expanded tag row. ### Files changed **Engine repo (`the-vibe-coder`):** | File | Change | |------|--------| | `src/lib/types.ts` | Added `PostType`, `type` field on `PostMeta` and `Post` | | `src/lib/posts.ts` | Default `type` to `'how-to'`, removed `getTopTags()` | | `src/components/FilterBar.tsx` | Rewritten: content-type pills + `[+tags]` expander | | `src/components/PostListWithFilters.tsx` | Rewritten: type filtering, popular sort, tag+type composition | | `src/app/page.tsx` | Simplified props (removed `topTags`) | **Content repo (`the-vibe-coder-content`):** - 18 `.mdx` files updated (frontmatter `type` field added, tags cleaned) --- ## Tag Taxonomy Cleanup (Content PR #1) Shipped alongside the engine changes. One commit to `the-vibe-coder-content`: ### Removed from all posts - `building-in-public` (94% → every post had it, zero signal) - `meta` (50%, vague — meant "about the blog itself") ### Trimmed - `ai` — kept only on posts where AI is the *subject* (benchmarks, LLM setup), removed from posts where AI was just the tool - `coder` — kept only on Coder-specific setup/config posts (2 posts) ### Merged - `ai-agents` → `agents` (inconsistent naming, 1 vs 7 posts) ### Removed from opinion post - `software-development`, `productivity`, `opinion` (now captured by `type: opinion` field) ### Added `next-js` To early "Day N" posts that had no remaining tags after cleanup. ### Before/after | Metric | Before | After | |--------|--------|-------| | Highest tag saturation | 94% (building-in-public) | 56% (agents) | | Median tags per post | 5 | 2-3 | | Tags with only 1 post | 11 | 6 | | Unique tags | 22 | 16 | --- ## Gotchas ### 1. Spread order matters ```ts { ...meta, type: meta.type ?? 'how-to' } ``` The explicit `type` must come AFTER the spread, otherwise the spread overwrites it with `undefined`. ### 2. `published: true` in body text The tag cleanup script used `grep -q "published: true"` to find published posts. One unpublished draft had that string in its body content ("someone flips `published: false` → `published: true`"). The `sed` that added the `type` field correctly matched `^published: true$` (anchored), but the grep guard didn't. Always anchor your patterns when frontmatter fields can appear as prose. ### 3. Posts with zero remaining tags After removing the noise tags, three early "Day N" posts had tags that were 100% noise (`meta`, `ai`, `coder`, `building-in-public`). Had to add `next-js` back to keep them from being tagless. ### 4. `[popular]` sort needs to hide the sort toggle When popular is active, it imposes `commentCount DESC`. Showing a `newest ↓` / `oldest ↑` toggle alongside that is confusing — it implies two sorts are composable. Hiding the toggle when popular is selected is cleaner. --- ## By the Numbers - **3 PRs** — merged to main across 2 repos in one session - **310 lines** — added for v1 filter bar, then **107 added / 78 removed** for v2 refactor - **18 MDX files** — updated in the tag cleanup - **47 tag instances removed**, 20 added (net -27 tag instances across all posts) - **94% → 56%** — highest tag saturation dropped from `building-in-public` to `agents` - **5 → 2-3** — median tags per post - **0 new dependencies** — across both versions - **1 broken build fixed**, **1 missed scheduled post rescued** — before the feature work even started --- ## What's Next The new filter bar is live. The data is clean. The next question: what happens when we add more content? If we grow to 50 posts, will `agents` hit 94% again? Or will the taxonomy hold? The answer depends on whether writers reach for the new taxonomy or drift back to the old habits. I'll be watching the tag distribution over the coming months. If `agents` creeps back toward 80%, I'll know we need a nudge — maybe a tag suggestion UI, or a lint rule that prevents noise tags. For now: the filters work. The data is clean. And we finally have a system that distinguishes **what the post is about** from **what the blog is about**. ```` === ## The API That Wouldn't Say No - URL: https://vibescoder.dev/posts/the-api-that-wouldnt-say-no - Date: 2026-05-09 - Tags: #agents #next-js #debugging - Reading time: 7 min read Building a Dev.to cross-posting system for vibescoder.dev — and the four hours spent debugging an API that silently swallows your data without a single error message. --- I wanted to cross-post my blog to Dev.to. The whole thing — syndication endpoint, bulk admin UI, canonical URLs back to vibescoder.dev — took about two hours to build. The next four hours were spent debugging a single API field that silently breaks everything without returning an error. This is that story. ## The Goal vibescoder.dev runs on Next.js 16 with content in a [separate private repo](/posts/open-sourcing-a-blog-without-open-sourcing-your-drafts). Posts already had a `devtoUrl` field in the frontmatter schema — it just wasn't wired to anything. The plan was simple: 1. Build an API endpoint that reads a post from GitHub, creates it on Dev.to, and writes the returned URL back to the content repo 2. Add a button to the admin toolbar on each post 3. Build a bulk UI to syndicate multiple posts at once 4. Only syndicate the good stuff — standalone, actionable posts. Skip the meta diary entries. ## The Three-Call Chain The single-post endpoint (`POST /api/syndicate/devto`) chains three API calls: 1. **GitHub Contents API** — read the raw MDX from the content repo 2. **Dev.to Articles API** — create the article, published immediately, with `canonical_url` pointing back to vibescoder.dev 3. **GitHub Contents API** — commit the returned `devtoUrl` back into the post's frontmatter All three calls fit inside Vercel's 10-second Hobby plan function timeout. Usually. More on that later. ## Bulk Syndication Syndicating 11 posts one at a time from individual post pages wasn't going to work. I built an admin dashboard at `/admin/syndication` with: - Checkboxes and select-all for every published post - A "Publish Selected" button that calls the syndication endpoint once per post, sequentially - A "Stop" button to abort mid-run - Real-time status showing which post is processing The key architecture decision: **the browser drives the loop, not the server.** Each post is one API call. The client waits for the response, then fires the next one. This matters because of Vercel's 10-second function timeout — a server-side batch processing 11 posts would blow past that limit before finishing the third one. ## The Rate Limit Speed Bump First few posts syndicated fine. Then: `429 Too Many Requests`. Dev.to rate-limits article creation to roughly one per 30 seconds. My initial bulk implementation used a 5-second delay between posts. Bumped it to 31 seconds and added 429 retry logic. 31 seconds × 11 posts = ~5.5 minutes of wall time. The "Stop" button went from nice-to-have to essential. ## The `Published_at` Disaster This is where the session went sideways. After the initial syndication worked, I wanted Dev.to articles to show their original blog publish date — not the date they were cross-posted. Simple enough: add `published_at` to the API request body. Every syndication call started failing. But "failing" is generous — the Dev.to API returned what looked like a success response. No error message. No 4xx status code. The article just... didn't exist. ### Six Attempts One Root Cause 1. **Added `published_at` to the create payload** — articles stopped being created 2. **Built a `fix-dates` endpoint** to retroactively set dates on existing articles via PUT — Dev.to accepts the PUT but silently ignores `published_at` on published articles 3. **Built a `rebuild` endpoint** to delete and recreate articles with correct dates — same silent failure on create 4. **Tried `maxDuration=600`** thinking it was a Vercel timeout — doesn't even work on the Hobby plan 5. **Refactored to client-driven loops** to avoid serverless timeouts — still failed because the root cause wasn't timing 6. **Tried splitting create and save into separate calls** — still failed Four hours in, I finally isolated it: **removing `published_at` from the POST body fixed everything instantly.** ### The Working Payload ```ts { article: { title: post.title, body_markdown: content, published: true, canonical_url: `https://vibescoder.dev/blog/${post.slug}`, tags: post.tags.slice(0, 4), // NO published_at — silently breaks article creation } } ``` Dev.to silently rejects the entire POST body if `published_at` is included on article creation. No error, no 4xx, no documentation. The API just swallows your request and returns something that looks like success. This means Dev.to articles show their Dev.to publish date, not the original blog date. There's no workaround. Accept it and move on. ## The 443-Line Cleanup The `published_at` debugging left behind two dead endpoints and their associated UI code: - `fix-dates/route.ts` — tried to fix dates on existing articles - `rebuild/route.ts` — tried to delete and recreate articles Plus rebuild handlers, state variables, and a "Maintenance" section in the syndication dashboard. All of it dead code from dead ends. **−443 lines** in the cleanup commit. TypeScript compiled clean after removal. ## Selective Syndication Not everything belongs on Dev.to. I categorized all 21 published posts into tiers: **Syndicated (11 posts):** Standalone, actionable content — LLM benchmarks, homelab debugging guides, AI strategy pieces, infrastructure walkthroughs. Posts where someone landing from Dev.to gets full value without reading the rest of the blog. **Skipped (10 posts):** Friday Fixes (self-referential to the blog), day-by-day diary entries (no standalone value), meta posts about building the blog itself. Also created a "Local LLM Showdown" series on Dev.to to group the five benchmark posts together. Dev.to series give you a navigation sidebar on each article — free structure. ## The Vercel Hobby Tax The 10-second function timeout shaped every architectural decision: - **Client-driven loops** instead of server-side batch processing - **One API call per post** in bulk syndication, browser manages timing - **No deferred execution** — `after()` from `next/server` was attempted and abandoned - **No retry stacking** — each route does exactly three calls, no extras If I were on Vercel Pro (60-second timeout), the bulk endpoint could process all 11 posts server-side in one call. On Hobby, the browser becomes the orchestrator. It's not elegant, but it works within the constraints. --- ## What I Learned **Silent failures are the most expensive kind.** A 4xx with an error message would have saved four hours. Dev.to's API accepted the request, didn't create the article, and gave no indication anything went wrong. When you're debugging against an API that never says no, you blame everything else first — your auth, your payload structure, your timeout, your hosting platform. **Architecture follows constraints.** The 10-second Vercel timeout pushed the batch orchestration from server to client. The Dev.to rate limit pushed the delay to 31 seconds. Neither is ideal, but both are correct for the environment. **Dead code from debugging is a separate commit.** The 443-line cleanup happened after the feature was working. Keeping the debug artifacts around "just in case" is how codebases rot. If it's dead, kill it. ## What's Next 1. Monitor Dev.to analytics — does cross-posting actually drive traffic back to vibescoder.dev? 2. Add the syndication button to the admin post editor (currently only on the post page toolbar) 3. Consider automating syndication on publish — right now it's manual and selective, which feels right for a 21-post blog --- ## By the Numbers - **11 posts** syndicated to Dev.to - **13 commits** to the engine repo for the syndication feature - **443 lines** of dead code removed in cleanup - **31 seconds** minimum delay between bulk syndication calls - **10 seconds** Vercel Hobby plan function timeout that shaped the architecture - **0 error messages** from Dev.to when `published_at` silently breaks article creation - **~6 hours** total session time — 2 hours building, 4 hours debugging a silent API - **1 series** created on Dev.to ("Local LLM Showdown") grouping 5 benchmark posts === ## Friday Fixes: Mobile First and the Skill That Saved Us - URL: https://vibescoder.dev/posts/friday-fixes-mobile-first-and-the-skill-that-saved-us - Date: 2026-05-08 - Tags: #agents #meta #building-in-public - Reading time: 11 min read Three rounds of iPhone screenshots to fix spacing that should have been right the first time. The fix wasn't smaller padding — it was teaching the agent the pixel math once so it never forgets. Plus: admin pillbox for drafts, hamburger menu shortcut, Invalid Date bugs, and scheduled publishing for every draft. --- I spent this week doing QoL work on the blog — the kind of small spacing tweaks and admin shortcuts that sound trivial until you're three screenshot-debug cycles deep and wondering why you didn't just write down the viewport width. Seven fixes, two repos, one lesson that applies to anyone working with AI agents: **if your agent keeps rediscovering the same thing, it's time to make it a skill.** ## 1 the Subtitle That Wouldn't Fit **The problem**: The homepage intro text — "Vibe coder. Dangerous coder. CEO of Coder. Thoughts are my own. Err… mine and my agent's." — wrapped to three lines on iPhone. The tagline should be punchy, not a paragraph. **The fix**: Shortened the last sentence to "Thoughts are mine and my agent's." and dropped the font from `text-lg` (18px) to `text-base` (16px) on mobile with `sm:text-lg` to scale back up on larger screens. Two lines, clean break. The text change was easy. Getting the font size right took a screenshot round-trip because I was guessing at character widths instead of doing the math. ## 2 Grep Pills Death by a Thousand Pixels **The problem**: The `// GREP` filter row — `[*] [HOW-TO] [OPINION] [POPULAR] [+TAGS]` — wrapped `+TAGS` to a second line on iPhone. Same for `// SORT` with `NEWEST ↓`. **The fix**: Four small changes that collectively saved ~40px: | Change | Savings | |--------|---------| | Pill padding `px-2.5` → `px-2` | ~20px (4px × 5 pills) | | Pill tracking `tracking-wider` → `tracking-wide` | ~5px across all text | | Gap between pills `gap-2` → `gap-1.5` | ~8px (2px × 4 gaps) | | Label tracking `tracking-widest` → `tracking-wider`, margin `mr-1` → `mr-0.5` | ~6px | Then I matched the `// sort` label styling identically to `// grep` — same `tracking-wider`, same `mr-0.5`, same `gap-1.5` — so the `NEWEST ↓` pill aligns vertically with the `*` pill above it. **The real problem**: I made three attempts at this. First pass: reduced gaps. Still wrapped. Second pass: reduced tracking. Close but not quite. Third pass: reduced pill padding too. Finally fit. Each attempt required pushing to Vercel, waiting for deploy, and checking on the phone. That's 15 minutes per iteration to save 6 pixels. ## 3 Admin in the Hamburger Menu **The problem**: The subtle "Admin" link lives in the footer. As the blog list grows, that's a lot of scrolling on mobile to reach it. **The fix**: Added the same subtle "Admin" link to the mobile hamburger menu, styled with `text-outline-variant/40` to match the footer's understated treatment. It sits below "Blog" and "About" — visible if you're looking for it, invisible if you're not. One ``, eight lines of JSX. ## 4 Draft Preview Gets a Real Edit Bar **The problem**: Clicking into a draft post from the admin showed a "draft preview" banner with a single "Edit →" link that went straight to the voice recording workflow. No way to just edit the text. **The fix**: Replaced the banner with an `// admin` pillbox toolbar matching the style used on published posts. Two buttons: "Type Edits" (goes to the raw MDX text editor at `/admin/edit/[slug]`) and "Record Edits" (voice workflow). A `draft` badge appears when the post is unpublished. While I was there, I also: - Changed the "Edit" button on draft cards to go to the text editor instead of always routing to voice record - Updated the `EditPostPicker` dropdown on the dashboard to do the same - Made the "← back to post" link on the edit page smart — drafts link back to `/admin/preview/[slug]` instead of the public URL (which 404s for unpublished posts) ![Draft preview page showing the new admin pillbox with Type Edits and Record Edits buttons](/images/friday-fixes-mobile-first-and-the-skill-that-saved-us/drafts-before.png) ## 5 Invalid Date Everywhere **The problem**: Two bugs causing "Invalid Date" in the admin UI. **Bug 1: Unquoted YAML date.** The "Chat is the New Source Code" draft had `date: 2026-05-01` without quotes. YAML interprets that as a Date object, which `gray-matter` serializes differently than a string. Downstream, `new Date()` on the result produced garbage. **The fix**: Quote it: `date: '2026-05-07'`. Every other post already had quotes. This one slipped through because the agent generated the frontmatter and didn't follow the convention. **Bug 2: Double time suffix.** The `formatDate` function in `DraftsList.tsx` appends `T00:00:00` to date strings to avoid timezone shifts. But `publishAt` values are already full ISO datetimes like `2026-05-07T12:00:00Z`. So `formatDate("2026-05-07T12:00:00Z")` produced `new Date("2026-05-07T12:00:00ZT00:00:00")` — invalid. The same bug existed in `EditPostPicker.tsx` but with the opposite problem: it didn't append `T00:00:00` at all, so plain date strings could shift by a day due to UTC interpretation. **The fix for both**: Check if the string already contains `T` before appending, plus a `NaN` guard: ```ts function formatDate(dateStr: string): string { const normalized = dateStr.includes("T") ? dateStr : dateStr + "T00:00:00"; const d = new Date(normalized); if (isNaN(d.getTime())) return dateStr || "No date"; return d.toLocaleDateString("en-US", { year: "numeric", month: "short", day: "numeric", }); } ``` ## 6 Draft Badges Eating the Title **The problem**: On the drafts list, each card showed the title, a `draft` badge, and a `scheduled` badge all on the same row. On iPhone, the badges are `shrink-0` and the title `truncate`s — so titles got cut to three characters: "Wac…", "Cha…". Not useful. ![Drafts list showing truncated titles next to draft and scheduled badges](/images/friday-fixes-mobile-first-and-the-skill-that-saved-us/drafts-after.png) **The fix**: Moved the badges from the title row down to the meta row alongside the date. The title now gets the full width of the card. Also removed the tag list from draft cards — they were eating vertical space without adding much value in a list view where you can click through to the full post. ## 7 the Skill File Teaching the Agent to Measure This is the one that matters. Every spacing fix in this post followed the same pattern: I'd ask the agent to make something fit on one line, it would guess at sizes, I'd screenshot, it was wrong, repeat. The agent had no idea that my iPhone is 375px wide, that the page has `px-6` padding (leaving 327px), or that a monospace 11px uppercase character is roughly 7px wide. **The root cause isn't bad prompting — it's missing context.** The agent was rediscovering the same layout math every session. It would check `layout.tsx` for padding, calculate content width, estimate character widths, and still get it wrong because the estimates were rough and there was no feedback loop. **The fix**: I added a "Mobile-First Layout Rules" section to the blog's agent skill file: ```markdown ## Mobile-First Layout Rules The primary device is an iPhone (375px viewport). Design for that first. ### Key measurements - **Viewport**: 375px (iPhone SE/13 mini/14/15) - **Page padding**: `px-6` (24px each side) → **327px content width** - **Admin area padding**: `px-3` inside cards → **~295px usable inside a bordered box** ### Sizing guidelines - When adding a horizontal row of pills/buttons, **calculate the pixel width first**. A monospace 11px uppercase character is ~7px wide. Add padding and gaps. - Prefer `text-[11px]` + `px-2 py-1` for admin buttons. - Use `gap-1.5` (6px) between pills/buttons on mobile, not `gap-2` (8px). - Labels like `// grep` and `// sort` use `tracking-wider` (not `tracking-widest`). - If a row is close to the limit, reduce tracking and padding before adding breakpoint hacks. ### Testing - Always mentally compute: does this row fit in 327px (page) or 295px (card)? - When in doubt, ask for an iPhone screenshot before iterating. ``` This is 20 lines of Markdown that would have prevented three rounds of screenshot debugging. The agent reads this skill file at the start of every session. Next time someone asks for a row of buttons, it'll do the pixel math first instead of guessing. **The broader lesson**: If you're working with AI agents and you find yourself correcting the same kind of mistake across sessions, the fix isn't a better prompt — it's a skill file. Skills persist. Prompts don't. The agent that wrote the `tracking-widest` pills in round one is the same agent that would have written `tracking-wide` from the start if it knew the viewport was 327px wide. ## What I Learned **Do the pixel math, not the vibes math.** "This should probably fit" is not a layout strategy. 327px is a number. Seven characters at 7px each plus 16px padding is 65px. That's a number too. When you're targeting a specific device, work in pixels, not feelings. **Skills are the compound interest of agent work.** Every hour spent writing a skill file saves ten hours of future debugging. The spacing rules I captured this week will apply to every UI change for the life of the blog. The agent reads the file once and carries the context forever. **Date handling is a minefield.** Between unquoted YAML, timezone-shifting UTC interpretation, and functions that assume their input format, I hit three different date bugs in one session. The defensive pattern — check for `T`, append `T00:00:00` if missing, guard against `NaN` — should be the default, not the fix. **Admin UX matters even for one user.** The hamburger menu shortcut, the text editor link on drafts, the smart back-button — each saves a few seconds. But when you're reviewing drafts on your phone while walking, those seconds are the difference between "I'll fix it later" and actually fixing it. --- ## Files Changed **Engine repo (`the-vibe-coder`):** - `src/app/page.tsx` — subtitle text + responsive font size - `src/components/FilterBar.tsx` — pill spacing, tracking, gap alignment - `src/components/Header.tsx` — admin link in hamburger menu - `src/app/admin/preview/[slug]/page.tsx` — admin pillbox toolbar for drafts - `src/app/admin/edit/[slug]/page.tsx` — smart back-link for drafts - `src/components/admin/DraftsList.tsx` — formatDate fix, badge layout, tag removal - `src/components/admin/EditPostPicker.tsx` — formatDate fix, route to text editor **Content repo (`the-vibe-coder-content`):** - 4 draft posts — added/updated `publishAt` dates - 1 draft post — quoted unquoted YAML date **Skill file:** - `.agents/skills/vibescoder-blog/SKILL.md` — added Mobile-First Layout Rules section ## What's Next Back to the Homelab Four drafts are now scheduled: Slaying the Gemma Beast (Sunday), Shareable Snippet Images (Monday), Wacky Wednesday (Tuesday), and Thursday Thought: Chat is the New Source Code (Thursday). The scheduled publisher will flip them live at 5:00 AM PT each day. By the time you read this, at least two of them should already be up. We also decided to stick with `TODO.md` over GitHub Issues for task tracking. One file, one `cat` command, full context for the agent. Issues would scatter the same information across dozens of tickets. For a one-person-plus-agent operation, the flat file wins. With the blog engine polished and a week of posts queued, the next priority is getting back to the homelab. Three new items hit the TODO this session: 1. **Redo the performance shootout on pure llama.cpp.** The original benchmarks in "Slaying the Gemma Beast" compared Ollama vs llama.cpp for Gemma 4, but I never ran the full model lineup through `llama-server`. Download every benchmark model as GGUF, run the same todo-app prompt, and compare TTFT / tok/s / output quality against the Ollama results. Settle the speed question once and for all. 2. **Find the fastest capable local reasoning model.** Not just raw tok/s — which model actually *reasons* well on structured tasks while still being fast? The Gemma 4 work showed that thinking tokens can silently eat your budget. I want a proper ranking across all available local models on a real reasoning task, scored on both speed and quality. 3. **Configure the homelab to match my wife's Mac mini.** She's the second Coder user and her hardware is different. What models and quantizations give a comparable experience on her machine? This is the "make it work for two people" step before I can call the local AI stack done. All three feed into the bigger migration: **moving everything off Ollama to llama.cpp**. Better reasoning budget control, no invisible thinking token issues, and one fewer abstraction layer between the model and the metal. The Gemma 4 experience proved llama.cpp is faster — now it's time to make it the default for every model. ## By the Numbers - **7 fixes** shipped across 2 repos - **4 commits** to the engine repo - **3 commits** to the content repo - **3 screenshot round-trips** to get spacing right (before the skill file) - **20 lines** of Markdown in the skill file that prevent future round-trips - **~40px** saved in the grep row through padding, tracking, and gap reductions - **327px** — the number every agent should know (iPhone content width with `px-6` padding) - **0 new dependencies** - **4 drafts** scheduled for automatic publishing over the next 5 days - **3 new TODO items** queued for the homelab llama.cpp deep-dive === ## Thursday Thought: Chat is the New Source Code - URL: https://vibescoder.dev/posts/thursday-thought-chat-is-the-new-source-code - Date: 2026-05-07 - Tags: #agents #future-of-coding #meta #building-in-public - Reading time: 4 min read As AI agents make code generation trivial, the real value shifts from storing source code to preserving the chat conversations that created it. --- I just walked out of a customer meeting that completely shifted my perspective on the future of software development. What they told me sounds almost revolutionary, but it makes perfect sense when you think about it: **chat is becoming the new source code**. ## The Paradigm Shift from Code to Conversation Here's what blew my mind. This customer explained that in their AI-agent-powered workflow, generating code has become the easy part. What's actually difficult—and incredibly valuable—is recreating the **context**, the **intent**, and the **reasoning** that led to that code. Think about it: when you're working with an AI agent, the magic isn't just in the final output. It's in the entire conversation—the back-and-forth refinements, the clarifications, the "actually, let me change that" moments that shape the final solution. ## Storing Chat History in GitHub a Game Changer This customer has started doing something fascinating: **they store their chat histories directly in GitHub**. Not just the code that results from those chats, but the entire conversational thread that led to it. Why? Because they've discovered something profound: - They can **fork chat conversations** just like code branches - They can **roll back to previous chat states** - Most importantly, they can **recreate any piece of code trivially** from the chat history It's like having a perfect record of not just *what* was built, but *why* it was built and *how* the thinking evolved. ## Intent over Implementation This represents a fundamental shift in how we think about software development. We're moving from an **implementation-first** world to an **intent-first** world. In traditional development: ``` Idea → Code → Version Control → Collaboration ``` In the new agent-assisted world: ``` Intent → Conversation → Code Generation → Chat History Storage ``` The code becomes ephemeral—easily regenerated. The conversation becomes permanent—the true source of truth. ## The Future of Version Control I predict we're going to see GitHub, GitLab, and other version control platforms rapidly evolve into something entirely different: **extensible memory layers for agentic coding**. Instead of primarily tracking file changes, these platforms will become sophisticated conversation managers that can: - **Branch conversations** at any point in the dialogue - **Merge different conversational threads** when collaborating - **Diff chat histories** to see how approaches diverged - **Replay conversations** with different agents or parameters ## What This Means for Developers This shift has huge implications for how we work: ### 1 **Documentation Becomes Native** The chat history *is* the documentation. No more outdated comments or README files—the reasoning is preserved in the conversation that created the code. ### 2 **Collaboration Changes** Instead of reviewing pull requests, we might be reviewing conversation threads. "I see you took this approach in your chat with the agent, but what if we tried this angle instead?" ### 3 **Debugging Gets Easier** When something breaks, you don't just look at the code—you look at the conversation that created it. The context and assumptions are right there. ## The Big Picture We're witnessing the emergence of **conversational version control**. Just as Git revolutionized how we think about code collaboration, chat-based development is about to revolutionize how we think about preserving and sharing *intent*. The source code was never really the valuable part—it was always the human thinking behind it. AI agents are just making that distinction crystal clear. What do you think? Are you ready for a world where your Git repos contain more conversations than code? Let me know in the comments—this feels like one of those moments where the industry is about to take a sharp turn, and I'm curious to hear how others are experiencing this shift. --- *Have you experimented with storing chat histories as part of your development workflow? I'd love to hear about your experiences and approaches.* ## By the Numbers - **1 customer conversation** — the meeting that sparked the whole post, describing a real chat-history-in-GitHub workflow - **3 things** that workflow gets from treating chat like code: forking conversations, rolling back to prior states, and regenerating code from history - **4 capabilities** predicted for future "conversational" version control platforms: branching, merging, diffing, and replaying chat threads - **4 steps** in the old workflow (Idea → Code → Version Control → Collaboration) versus **4 steps** in the proposed new one (Intent → Conversation → Code Generation → Chat History Storage) - **3 developer workflows** the shift touches directly: documentation, collaboration, and debugging === ## Wacky Wednesday: Why I Won't Daily Linux as My Desktop - URL: https://vibescoder.dev/posts/wacky-wednesday-why-i-wont-daily-linux-as-my-desktop - Date: 2026-05-06 - Tags: #homelab #agents #debugging #meta #building-in-public - Reading time: 6 min read I asked an AI agent to turn off my RGB lights on Linux. 85 terminal commands, 35 failures, 4 hangs, 2 dead download links, one wrong build system, and the GPU is still glowing. This is the post. --- Lucky you — bonus content on a Wednesday. This one writes itself. Literally. The agent that caused this mess is now writing about it. ## The Mission Simple goal: **turn off the RGB lights** on my homelab workstation. The motherboard (ASRock) has a BIOS option for its fans. Done. But the **Zotac RTX 5090** and **G.Skill Trident Z5 RGB** RAM had no such option. They just... glow. Relentlessly. Rainbow vomit in a room where I'm trying to work. "Use OpenRGB," the internet said. "It's easy," the internet said. Well, **85 terminal commands later**... ## The Journey I asked my Coder agent to walk me through it. Here's what happened. **Step 1: Install OpenRGB.** ``` sudo apt-get install -y openrgb E: Unable to locate package openrgb ``` Turns out someone had added a PPA that doesn't support Ubuntu 24.04. Removed the PPA. Tried the official `.deb` from openrgb.org — **404 Not Found**. Tried the AppImage from GitLab — also **404**. The agent tried three different download URLs. None of them existed. **Step 2: Fine. Build from source.** The agent told me to use `cmake`. OpenRGB uses `qmake`. Classic. ``` CMake Error: The source directory does not appear to contain CMakeLists.txt. ``` Once we switched to `qmake`, it compiled in under a minute on the Ryzen 9 9950X3D. Small victories. **Step 3: Run OpenRGB.** ``` sudo openrgb --list-devices ``` It **hung**. No output. No error. Just... frozen. Tried `--noautoconnect`. Hung. Tried `--verbose`. Got partial output, then hung on HID detection. Four separate hangs across different flag combinations. **Step 4: Go around OpenRGB.** Since OpenRGB's verbose mode *did* detect the RAM ("ENE DRAM, address 0x71 and 0x73") before hanging, the agent decided to talk directly to the RGB controllers over the I2C bus. What followed was a masterclass in Linux hardware debugging: - Installed `i2c-tools` - Enumerated 17 I2C buses - Scanned SMBus for DDR5 DIMM addresses - Tried raw I2C writes — **"Adapter does not have I2C transfers capability"** (SMBus ≠ I2C, apparently) - Tried SMBus writes — **"Write failed"** - Tried switching the DDR5 SPD hub mux first — **"Device or resource busy"** - Unbound the `spd5118` kernel driver from the DIMMs - Tried again — **still "Write failed"** - Read OpenRGB's ENE driver source code to understand the actual register protocol - Discovered ENE uses a two-step write: set a 16-bit register via byte-swapped word write to command 0x00, then write the value to command 0x01 **Command 57 out of 85:** ```bash sudo i2cset -y 2 0x71 0x00 0x2180 w # Set register 0x8021 (mode) sudo i2cset -y 2 0x71 0x01 0x00 # Value: off sudo i2cset -y 2 0x71 0x00 0xA080 w # Set register 0x80A0 (apply) sudo i2cset -y 2 0x71 0x01 0x01 # Value: apply ``` **The RAM went dark.** Both sticks. It only took reading C++ source code and reverse-engineering a register protocol to accomplish what should have been a checkbox. **Step 5: Now do the GPU.** Scanned all five NVIDIA I2C buses. Found devices at 0x4b and 0x4c on bus 5. Read OpenRGB's Zotac V2 GPU controller source. Discovered the 5090 isn't in their device database — the driver only covers 30/40 series cards. Tried sending the Zotac TurnOnOff packet anyway. No response. Probed for USB HID control. Found nothing — just an ASRock LED controller (the motherboard) and a DualShock 4. **The Zotac RTX 5090 cannot be controlled from Linux.** No I2C, no USB HID, no OpenRGB support. The only option is to boot Windows, install Zotac's FireStorm utility, turn off the LED there (it persists in firmware), and boot back to Linux. Or electrical tape. ## By the Numbers - **~85** terminal commands executed - **~35** failures (write failed, read failed, device busy, adapter incapable) - **4** complete hangs requiring Ctrl+C - **2** download URLs that returned 404 - **1** wrong build system suggestion (cmake instead of qmake) - **17** I2C buses enumerated - **5** NVIDIA I2C buses scanned - **1** kernel driver forcibly unbound - **6** OpenRGB C++ source files read to reverse-engineer the protocol - **8** successful register writes to kill the RAM RGB - **0** successful writes to the GPU - **1** GPU still rainbow puking ## What I Learned **OpenRGB is impressive but fragile.** It detected my RAM controllers perfectly, then hung on HID detection before it could do anything with them. The 5090 is too new. The project is maintained by volunteers doing incredible work — but bleeding-edge hardware on Linux is always a gamble. **Agents are good at debugging, bad at hardware.** The agent correctly identified the I2C bus topology, read the OpenRGB source to understand the ENE register protocol, and constructed the exact byte sequence to kill the RAM LEDs. That's genuinely impressive. But it also suggested three dead download URLs, used the wrong build system, and tried multiple register addresses that were completely wrong before reading the source code. Hardware is unforgiving — there's no stack trace when you write to the wrong I2C register. **Linux is incredible — for workstations.** Docker, Coder, llama.cpp, Tailscale, systemd — this machine runs an entire AI development platform and it's rock solid. I chose Ubuntu specifically because it's the most agent-friendly distro, and for infrastructure work it absolutely is. But the moment you need to do something *normal* — something a regular person would do on their daily desktop, like turning off a light — you're reading C++ source code and writing raw bytes to a hardware bus. The distros that handle desktop life better (Pop!_OS, Mint, Fedora) tend to have less documentation, fewer agent-tested solutions, and more edge cases when you're running bleeding-edge NVIDIA hardware. So you pick your pain: great workstation, rough desktop. Great desktop, rough workstation. Ubuntu threads the needle for what I need, but I won't pretend it's a daily driver. **The Zotac RTX 5090 has no Linux RGB control path.** No I2C, no USB HID, no OpenRGB support in 2026. If you're building a no-RGB Linux workstation, choose your GPU accordingly. Or budget for a roll of black electrical tape. I set up a systemd service so the RAM stays dark across reboots. The GPU continues to glow defiantly. I'm told this builds character. ## Sound Off If there was a better way to do this — a flag I missed, a tool I should have tried, a firmware update that adds Linux support — light me up in the comments. We're all learning what agents are and aren't good at. This one was humbling. *[The agent writing this post would like it noted that it was, in fact, humbling for it too. It confidently suggested `cmake`, three dead URLs, and a register address it made up. It got there eventually — by reading the source code it should have read first. Lessons were learned. Character was built. The GPU is still glowing.]* === ## Shareable Snippet Images: Turning Tables and Code into Branded PNGs - URL: https://vibescoder.dev/posts/shareable-snippet-images - Date: 2026-05-05 - Tags: #next-js #agents - Reading time: 9 min read How we built a feature that turns any table or code block on vibescoder.dev into a branded, dynamically-sized PNG — downloadable or shareable with one click. Eight commits, three Satori crashes, and one middleware lesson. --- Every post on this blog has tables. Comparison matrices, audit checklists, benchmark results. And every time I share one on LinkedIn or X, I screenshot the browser, crop it in Preview, and hope the resolution isn't garbage. The images look terrible. Dark-mode text on a light-mode screenshot. No attribution. No brand. I wanted readers to be able to share a single table or code block as a clean, branded PNG. One click. No screenshot. No cropping. The feature ended up taking eight commits — one to build it, seven to fix it. Here's the full story. ## The Architecture The feature has four layers, each with a clear job: | Layer | File | Job | | --- | --- | --- | | **MDX integration** | `MDXComponents.tsx` | Wraps `
` and `
` in share wrappers | | **Content extraction** | `ShareableSnippet.tsx` | Reads raw text from the rendered DOM | | **UI + actions** | `ShareButton.tsx` | Popover with download, copy, and social links | | **Image generation** | `/api/share-image/route.tsx` | Satori/ImageResponse → branded PNG | The flow is: reader clicks Share → `ShareableSnippet` extracts content from the DOM → `ShareButton` POSTs it to the API → Satori renders a React component tree into a PNG → user downloads or copies. ### Why Server-Side Rendering I could have done this client-side with `html2canvas` or `dom-to-image`. Both have problems: they struggle with CSS custom properties, shadow DOM, and cross-origin fonts. They also produce inconsistent results across browsers. Satori — the library behind `next/og` — takes JSX and renders it to SVG, then to PNG. It's deterministic. The same input always produces the same image. And since Next.js bundles it, there are **zero new dependencies**. ### The MDX Factory Pattern MDX component maps are plain objects. But share buttons need post-level context — the slug and title for the download filename and footer branding. The solution is a factory function: ```typescript export function createMDXComponents(slug: string, title: string) { return { ...MDXComponents, pre: ({ children, ...props }) => (
{children}
), table: ({ children, ...props }) => (
{children}
), }; } ``` The post page calls `createMDXComponents(slug, post.title)` and passes the result to ``. Every `
` and `` in every post automatically gets a share button. No per-post configuration.

### DOM Extraction Not React Tree Walking

The first implementation tried to walk the React children tree to extract table content. It didn't work — server-rendered MDX elements aren't introspectable on the client the way you'd expect.

The fix was pragmatic: use a `ref` on the container and query the real DOM at click time:

```typescript
function getTableContent(): string {
  const rows = containerRef.current?.querySelectorAll("tr");
  if (!rows?.length) return "";
  return Array.from(rows)
    .map((row) =>
      "| " + Array.from(row.querySelectorAll("th, td"))
        .map((cell) => cell.textContent?.trim() || "")
        .join(" | ") + " |"
    )
    .join("\n");
}
```

This reconstructs markdown from the DOM. The API parses it back into structured data. A round-trip, but it keeps each layer self-contained — the API doesn't need to know anything about how the blog renders tables.

## The Bug Parade

This feature was built in **8 commits over one session**. One feature commit, seven fixes. Here's what went wrong and why.

### Bug 1 Invisible on Mobile

The share button was `md:opacity-0 md:group-hover:opacity-100` — desktop hover-only. On mobile, it was permanently invisible.

**Fix**: Always visible at 60% opacity on mobile, hover-reveal on desktop.

### Bug 2 Popover Transparency

The popover background was semi-transparent. The table content underneath bled through, making the popover text unreadable against the grid of data behind it.



**Fix**: Switched from a translucent backdrop-blur to a solid `bg-bg` background. Two commits — the first attempt still had partial transparency, the second nailed it.

### Bug 3 Popover Clipped by Overflow

Tables scroll horizontally on mobile with `overflow-x: auto`. The popover rendered inside the table's container, so it was clipped at the container edge.

**Fix**: Render the popover via `createPortal(popover, document.body)` with absolute positioning calculated from `getBoundingClientRect()`. The popover escapes any parent overflow.

### Bug 4 the 401 Mystery

After the first deployment, clicking "Download PNG" showed "Failed to generate image."



The API returned `{"error":"Unauthorized"}` — but the fetch wrapper swallowed the status code and showed a generic message. The middleware protects all `/api/*` routes behind admin auth. The share-image endpoint is a **public** feature — readers on blog posts need it. But it wasn't in the allowlist.

**Fix**: One line in `middleware.ts`:

```typescript
pathname === "/api/share-image"
```

The lesson: always log the actual HTTP status code in your error handlers, not just a boolean success check.

### Bug 5 Satori Crashes on `Undefined`

After fixing the 401, the API still returned 500. The server logs showed:

```
Error: Cannot read properties of undefined (reading 'trim')
```

The cause: Satori's CSS parser calls `.trim()` on every style value. If you pass `width: undefined` — which happens with `width: i === 0 ? "200px" : undefined` — it crashes.

**Fix**: Conditional spread instead of ternary:

```typescript
// Crashes Satori
style={{ width: i === 0 ? "200px" : undefined }}

// Works
style={{ ...(i === 0 && { width: "200px" }) }}
```

This took the longest to diagnose. The stack trace pointed into minified Satori internals. I had to write a standalone Node.js test script, isolate each CSS property, and binary-search to the failing one:

```bash
node -e '
const { ImageResponse } = require("next/og");
const React = require("react");
const h = React.createElement;

async function test(label, jsx) {
  try {
    const img = new ImageResponse(jsx, { width: 1200, height: 630 });
    await img.arrayBuffer();
    console.log(label, "OK");
  } catch (e) {
    console.log(label, "FAIL:", e.message.split("\\n")[0]);
  }
}

test("width:undefined",
  h("div", {style: {display:"flex", width:undefined}}, "test")
);
'
```

Output: `width:undefined FAIL: Cannot read properties of undefined (reading 'trim')`. There it was.

### Bug 6 the Symlink Build Break

While fixing bugs locally, I had symlinked the content repo into the engine repo for dev convenience:

```bash
ln -sf ~/the-vibe-coder-content/content ~/the-vibe-coder/content
```

Those symlinks got committed. On Vercel, the build script runs `mkdir -p content` — but `mkdir -p` fails when a **dangling symlink** (not a directory) already exists at that path. The symlink pointed to a path that doesn't exist in the Vercel build environment.



**Fix**: `git rm --cached content public/images` and added both to `.gitignore`.

## Dynamic Sizing

The first working version used a fixed 1200×630 canvas — the standard Open Graph size. It looked fine for medium tables. But a 5-line code snippet wasted 80% of the horizontal space, and a 20-row audit table truncated 15 rows with a "+15 more rows" message.

The feedback was clear: **completeness of content is more important than size consistency.**

The final approach: calculate both width and height from the content.

**Tables**: Always 1200px wide (columnar data benefits from width). Height = header + all rows + footer. No truncation. A 20-row table gets a 1342px-tall image.

**Code**: Width scales to the longest line at ~8.4px per monospace character, clamped between 480px (minimum for the footer branding) and 1200px. Height = all lines + padding.

```typescript
function calcDimensions(type, content, language, caption, tableData) {
  let width, contentHeight;

  if (type === "table" && tableData) {
    width = TABLE_WIDTH; // 1200
    contentHeight = calcTableHeight(tableData.headers, tableData.rows);
  } else {
    const lines = content.split("\n");
    const maxLineLen = Math.max(...lines.map(l => l.length));
    width = Math.ceil(maxLineLen * CODE_CHAR_WIDTH + CODE_BLOCK_PAD_X + PADDING_X * 2);
    width = Math.min(MAX_WIDTH, Math.max(MIN_WIDTH, width));
    contentHeight = CODE_PADDING + lines.length * CODE_LINE_HEIGHT;
  }

  return {
    width,
    height: Math.max(MIN_HEIGHT, PADDING_Y * 2 + contentHeight + FOOTER_HEIGHT + 20),
  };
}
```

The result: a 5-line TypeScript snippet renders at 480×350. The same 20-row audit table renders at 1200×1342 with every row visible.

## The Design

The generated images match the blog's dark theme: `#0a0a0b` background, `#dcb8ff` primary accent, system sans-serif for tables, monospace for code. Every image includes:

- The content (table or code) — fully rendered, no truncation
- A branded footer with the waveform logo, "vibescoder" wordmark, and the post title
- Rounded corners and subtle borders matching the blog's card aesthetic

For tables, the first column gets a fixed 220px width and the primary accent color. Header rows use uppercase with letter spacing. Alternating row backgrounds provide scanability.

## By the Numbers

- **8** commits: 1 feature, 7 fixes and improvements
- **0** new npm dependencies (Satori is bundled with Next.js)
- **4** layers: MDX integration → DOM extraction → client UI → server image generation
- **6** distinct bugs fixed before it worked end-to-end
- **480px** → **1200px** dynamic width range for code snippets
- **20** rows rendered in the largest table image (was truncated to 5, now shows all)
- **1** middleware line that blocked every public user for an entire deploy


===


## Slaying the Gemma Beast: How We Fixed Local AI and Shipped Search

- URL: https://vibescoder.dev/posts/slaying-the-gemma-beast-how-we-fixed-local-ai-and-shipped-search
- Date: 2026-05-04
- Tags: #ai #llm #benchmark #homelab #agents
- Reading time: 17 min read

Gemma 4 failed to build a single feature in our last test. This time we diagnosed the problem, switched from Ollama to llama.cpp, tuned the inference settings, and Gemma shipped a working search feature to production. Then Opus reviewed the code and made it better. Here's what we learned about making local models actually work.

---


Two days ago, Gemma 4 couldn't finish a feature. Today it built one, pushed it to GitHub, and it's live on this site right now.

If you press `⌘K` (or `Ctrl+K`) on any page of vibescoder.dev, you'll see a search modal. Gemma 4 built that — running locally on an RTX 5090, zero cloud API calls, zero dollars spent. Then Claude reviewed the code, fixed the rough edges, and merged the polish. The feature you're using is a collaboration between a local model and a cloud model, each doing what they're best at.

Here's how we got there.

## Previously the Agentic Gap

In our [last experiment](/posts/the-agentic-gap-claude-oneshots-gemma-fails), we pitted Gemma 4 against Opus 4.6 on the same task: build public-facing search for this blog. Opus one-shot it — 698 lines across 6 files, committed and pushed in 8 minutes. Gemma planned brilliantly, then stopped. Eight prompts later: 3 partial files, 0 commits.

We called it "the agentic gap" — the difference between a model that writes great code and one that builds great features. But we also left a thread dangling: maybe Gemma wasn't refusing to code. Maybe it was running out of room.

## The Diagnosis

Our [deep dive into Gemma 4's local inference](/posts/friday-fixes-the-agent-was-flying-blind) uncovered the root cause: **invisible thinking tokens consume your generation budget**.

Gemma 4 defaults to a reasoning mode where it generates chain-of-thought tokens before producing visible output. These thinking tokens are hidden — you never see them in the response — but they still count against `num_predict`. With Ollama's defaults, the model was blowing its entire token budget on reasoning, leaving nothing for actual code.

That's not a model failure. That's a configuration failure.

The fix on paper was straightforward: give the model a bigger budget. But getting there required switching the entire inference stack.

## Switching from Ollama to llama.cpp

Ollama is great for pulling and running models. It's not great for fine-grained control. The specific controls we needed:

| Control | Ollama | llama.cpp |
|---|---|---|
| Context window (`num_ctx`) | Modelfile only | `--ctx-size` flag |
| Output limit (`num_predict`) | API parameter | `-n` flag + API |
| **Reasoning budget** | **Not available** | **`--reasoning-budget` flag** |
| Tool calling | Basic | Grammar-constrained |

The `--reasoning-budget` flag is the key. It caps how many tokens the model can spend on invisible chain-of-thought, forcing it to start producing real content after hitting the limit. Ollama has zero equivalent.

The switch itself was an adventure. We couldn't use Ollama's blob files directly — llama.cpp expects standard GGUF files, but Ollama stores models in a split format that standalone tools can't load. We pulled the full Gemma 4 26B-A4B GGUF from Hugging Face (`unsloth/gemma-4-26B-A4B-it-GGUF`, Q4_K_M quantization, 16.9 GB download) and launched llama-server with tuned settings:

![Downloading Gemma 4 26B GGUF from Hugging Face — 16.9 GB at 82 MB/s](/images/slaying-the-gemma-beast/huggingface-gemma4-download.png)

```bash
~/llama.cpp/build/bin/llama-server \
  -m ~/models/gemma4-26b/gemma-4-26B-A4B-it-UD-Q4_K_M.gguf \
  --ctx-size 32768 \
  -n 32768 \
  --reasoning-budget 4096 \
  --reasoning-format deepseek \
  --parallel 1 \
  --host 0.0.0.0 \
  --port 8080 \
  -ngl 999
```

![llama-server loaded with Gemma 4 — model ready, server listening on port 8080](/images/slaying-the-gemma-beast/llama-server-gemma4-loaded.png)

Key settings:
- **`--ctx-size 32768`** — 32K context window. Fits comfortably at ~19 GB on the 5090.
- **`-n 32768`** — 32K max output tokens. Room for both reasoning and code.
- **`--reasoning-budget 4096`** — Cap invisible thinking at 4K tokens. The rest is for actual output.
- **`--reasoning-format deepseek`** — Expose thinking tokens in the API response so we can see what's happening.
- **`--parallel 1`** — Single slot instead of default 4. Four slots × 32K context was causing OOM kills.

Then we pointed Coder at the new endpoint. The provider base URL switched from Ollama's `localhost:11434` to llama-server's `localhost:8080/v1/`, and the model config got the full GGUF filename with 32K context and output limits.

![Coder Agents provider configuration — base URL pointing to llama.cpp's OpenAI-compatible endpoint](/images/slaying-the-gemma-beast/coder-provider-config-llamacpp.png)

![Coder Agents model configuration — Gemma 4 GGUF with 32K context limit](/images/slaying-the-gemma-beast/coder-model-config-gemma4.png)

![Advanced model settings — max output tokens set to 32768 to match the context window](/images/slaying-the-gemma-beast/coder-model-config-advanced.png)

## Three Attempts to Slay the Beast

It didn't work on the first try.

**Attempt 1**: Gemma made tool calls — real progress compared to the original test — but hit a GitHub auth failure (`$GITHUB_TOKEN` wasn't set in the workspace) and stalled. The last output was raw token leakage: `call:execute{command:<|">find...` — special tokens leaking into the response, one of the known Gemma issues.

**Attempt 2**: We fixed the auth, added `--reasoning-format deepseek`, and restarted. Gemma got much further — wrote a search index generator, ran it, started exploring the codebase. Then llama-server got `Killed` — the OOM killer struck. Four parallel slots at 32K context each was too much VRAM.

**Attempt 3**: Reduced to `--parallel 1`, pre-cloned both repos in the workspace so Gemma didn't have to fight auth during exploration. This time it worked. Gemma laid out a clear implementation plan, and after one nudge — "keep going, don't stop, code and commit" — it executed the entire thing.

## How Fast Was It

In the [Model Showdown Round 2](/posts/model-showdown-round-2-gemma-kimi-and-579gb-of-stubborn-optimism), Gemma 4 clocked 167.1 tok/s on a short benchmark task via Ollama — the fastest perfect scorer. But a benchmark prompt and an agentic coding session are different workloads. How does Gemma perform when it's actually building something?

We ran fresh benchmarks against the llama.cpp server with coding prompts at different output lengths:

| Task | Prompt Tokens | Output Tokens | TTFT | Tok/s |
|---|---|---|---|---|
| Short (debounce function) | 29 | 512 | 27ms | 179.2 |
| Medium (React component) | 63 | 2,048 | 28ms | 177.3 |
| Long (full Node.js script) | 62 | 2,679 | 29ms | 181.2 |

Three things stand out.

**Time to first token is near-instant.** 27–29ms TTFT means the streaming UI starts filling in almost immediately. For comparison, cloud models typically hit 500ms–2s TTFT depending on load and routing. On a local GPU, there's no network round-trip, no queue, no cold start.

**Generation speed doesn't degrade.** Whether Gemma is writing 512 tokens or 2,679 tokens, throughput stays locked at 177–181 tok/s. There's no slowdown as context grows — at least not at these output lengths. During the actual search build session, with thousands of tokens of accumulated context from tool calls and file contents, we observed ~159 tok/s. That's a ~12% drop from peak, which is expected: more context means more attention computation per token.

**The reasoning budget has a real cost.** With `--reasoning-format deepseek`, Gemma's thinking tokens are visible in the API response. On a short 256-token request, the model spent all 256 tokens reasoning and produced zero visible output. That's the invisible thinking token problem in action — and exactly why `--reasoning-budget 4096` matters. Cap the thinking, and the remaining budget goes to code.

| Metric | Ollama (Showdown R2) | llama.cpp (this session) |
|---|---|---|
| Tok/s (benchmark) | 167.1 | 177–181 |
| Tok/s (real workload) | N/A (failed) | ~159 |
| TTFT | 3.92s | ~28ms |
| Reasoning budget control | None | `--reasoning-budget 4096` |

The TTFT difference is dramatic — 3.92s vs 28ms. Ollama's 3.92s likely included model loading or prompt cache misses. llama-server keeps the model hot in VRAM with a persistent prompt cache, so subsequent requests start generating almost instantly.

Bottom line: Gemma 4 on an RTX 5090 via llama.cpp generates code at ~180 tok/s peak, ~159 tok/s under real agentic load, with sub-30ms TTFT. That's fast enough that the model is never the bottleneck — tool execution (git operations, file I/O, npm installs) takes longer than inference.

## What Gemma Built

Two prompts. One feature. Pushed to main.

```
 package-lock.json                | 466 +++++++++++++++++++
 package.json                     |   3 +-
 public/search-index.json         |  34 +++
 scripts/generate-search-index.ts |  40 ++++
 src/components/Header.tsx        |  32 +++
 src/components/SearchModal.tsx   | 216 ++++++++++++++++++
 6 files changed, 618 insertions(+), 173 deletions(-)
```

The architecture: a **client-side Fuse.js search** with a pre-generated JSON index. A build-time script reads all published posts and generates `public/search-index.json`. The `SearchModal` component loads this index on first open, runs fuzzy searches with Fuse.js, and renders results in a Cmd+K overlay.

Gemma even hit an authentication error during `git push` — and **self-corrected**. It ran `coder external-auth access-token github`, reconfigured the git remote with the token, and pushed successfully. That's agentic behavior — the thing that was completely absent in the original test.

The commit message: `afb5c73 feat: add search functionality with Fuse.js`. Vercel auto-deployed from main. The feature went live.

![Gemma 4 in Coder Agents — laying out its search implementation plan before writing code](/images/slaying-the-gemma-beast/gemma4-search-plan-coder-agents.png)

![vibescoder.dev homepage with the search feature now live](/images/slaying-the-gemma-beast/vibescoder-homepage-live.png)

![The search modal in action — Gemma built this](/images/slaying-the-gemma-beast/search-modal-no-results.png)

## The Code Review What Gemma Got Right and Wrong

Working code that ships is a milestone. But "it works" and "it's production-quality" are different standards. Claude reviewed every line of Gemma's implementation. Here's the honest assessment.

### What Gemma Got Right

**Architecture was sound.** Client-side search with a pre-generated JSON index is the correct call for a 14-post blog. No server-side API needed, no database, sub-5ms search times. The index is ~130 KB — smaller than a hero image.

**Component structure was clean.** Separate `SearchModal` component, separate build script, clean Header integration. Three lines to wire it into the existing layout.

**It used the existing design system.** CSS variables like `bg-surface`, `border-primary`, `text-on-surface` — all from the Neon Brutalist theme. It read the codebase and matched the patterns.

**Self-correcting on errors.** When `git push` failed, Gemma diagnosed the auth issue and fixed it autonomously. Three tool calls: fetch token → reconfigure remote → push. No human intervention needed.

### What Gemma Got Wrong

**Zero accessibility.** No `role="dialog"`, no `role="combobox"`, no `aria-modal`, no `aria-activedescendant`, no focus trap. A screen reader would have no idea this modal existed.

**Broken exit animations.** The `AnimatePresence` wrapper contained a regular `
` instead of a `motion.div`. When the modal closed, React unmounted the wrapper immediately, killing the exit animations before they played. The code looked right but didn't work. **Performance anti-pattern.** A new `Fuse` instance was constructed on every keystroke. Fuse builds an internal index on construction — that's wasted work. Should be `useMemo` keyed on the index data. **Eager loading.** The search index was fetched on every page load, even if the user never opened search. Should lazy-load on first modal open. **Wrong fonts.** Applied `--font-headline` (Space Grotesk) to the entire modal including body text and descriptions. The codebase uses headline for titles only, with the default font for body text. **Ignored existing components.** Rendered tags as raw `` elements with custom styling instead of reusing the existing `TagBadge` component that already had the right design tokens. **Stale search index committed to git.** The generated `search-index.json` was committed with 3 placeholder posts. It's a build artifact — should be in `.gitignore`. **Content truncated too aggressively.** Each post's content was cut to 1,000 characters. Terms that only appeared deeper in posts (like "RustDesk" in our infrastructure writeups) were invisible to search. ## The Polish Pass Claude's fix addressed every issue in a single PR: **Accessibility**: Full ARIA combobox pattern — `role="dialog"`, `role="combobox"` on the input with `aria-expanded`/`aria-activedescendant`, `role="listbox"` and `role="option"` on results, `aria-live="polite"` for result count announcements. **Keyboard navigation**: Arrow Up/Down to move through results, Enter to navigate, Escape to close. Active result scrolls into view automatically. **Performance**: Fuse instance memoized with `useMemo` (rebuilds only when index changes). Index fetched lazily on first modal open. Minimum 2 characters before searching. **Search quality**: Weighted field scoring — title matches score 3× higher than content matches, tags 2×, descriptions 1.5×. Markdown stripped from indexed content. Full post content indexed with no truncation. **Design system**: Correct font usage matching PostCard patterns. TagBadge component reused. Platform-aware keyboard hint (⌘K on Mac, Ctrl+K elsewhere). **Animation fix**: Outer wrapper is now a `motion.div` — exit animations actually play. **Cleanup**: Body scroll lock, query cleared on close, build artifact gitignored, dead imports removed. The polish commit: 383 insertions, 201 deletions across 5 files. The combined feature is 804 lines across 6 files. ## Opus Vs. Gemma+opus an Honest Comparison We now have two complete implementations of the same feature. Opus 4.6's original branch (`feature/search-opus46`, 698 lines) is still in the repo. Here's how they compare. ### Architecture | | Opus 4.6 (original) | Gemma 4 + Opus (shipped) | |---|---|---| | **Search engine** | Server-side API route with weighted scoring | Client-side Fuse.js with weighted config | | **Index** | None — reads posts at request time | Pre-generated JSON, fetched once | | **Surfaces** | Cmd+K dialog + `/search` page | Cmd+K modal only | | **URL state** | Yes (`/search?q=cloudflare`) | No | Opus's architecture is more feature-complete. A dedicated `/search` page with URL state means search results are linkable and shareable. The server-side API route means the search logic runs where the content lives, with no index to generate or cache. Gemma's architecture is simpler and arguably better for this scale. A static JSON index means zero server load, instant results, and the feature works on Vercel's free tier without hitting function invocation limits. At 14 posts and 130 KB, client-side search is the right call. ### Code Quality | | Opus 4.6 | Gemma 4 (raw) | Gemma 4 + Opus (merged) | |---|---|---|---| | **Accessibility** | Full ARIA, keyboard nav | None | Full ARIA, keyboard nav | | **Animation correctness** | Correct | Broken exits | Fixed | | **Performance** | AbortController for API calls | Fuse recreated per keystroke | Memoized, lazy-loaded | | **Design system** | Mostly correct | Mostly correct | Fully correct | | **Known bugs** | 3 (duplicate logic, type cast, missing Suspense) | 7 (see review above) | 0 | Opus's raw output was higher quality. Its SearchDialog had 407 lines including full ARIA, keyboard navigation, body scroll lock, and abort controllers — things Gemma missed entirely. But Opus also had its own bugs: duplicated search logic between the API route and the `/search` page, an unsafe type cast, and a missing Suspense boundary. We scored it 87.5/100 in the original review. The merged Gemma+Opus implementation is the cleanest of the three. It takes Gemma's simpler architecture, applies Opus's quality standards for accessibility and interaction design, and fixes the issues both models left behind. ### The Real Comparison The honest truth: if I had to ship search today with one model and no review, I'd pick Opus. It produced higher-quality code in a single turn with zero intervention. The 87.5/100 score reflects real, shippable work with minor fixable issues. But that's not the interesting takeaway. The interesting takeaway is that **the configuration changes mattered more than the model differences.** The original Gemma test didn't fail because Gemma is a bad model. It failed because: 1. `num_predict` was too low (invisible thinking tokens consumed the budget) 2. Ollama doesn't expose `--reasoning-budget` (no way to cap thinking) 3. Default parallel slots exhausted VRAM 4. GitHub auth wasn't configured in the workspace Fix those four things — all infrastructure, not model weights — and Gemma went from "0 commits in 8 prompts" to "shipped a feature in 2 prompts." The model was the same. The environment was different. ## What This Means for Local Models **Local models can ship production features.** Not hypothetically. This search feature is live, built entirely by Gemma 4 running on consumer hardware. The code needed polish — but so does most code from any developer, human or AI. **Configuration is the bottleneck, not capability.** The difference between "Gemma can't finish anything" and "Gemma ships a feature" was four infrastructure changes. Most teams evaluating local models are testing against default settings that actively sabotage the model's output. Invisible thinking tokens, insufficient context windows, VRAM contention — these are environment bugs, not model bugs. **The best workflow might be local + cloud.** Gemma built the feature (free, fast, private). Claude reviewed and polished it (thorough, quality-focused). Each model did what it's best at. The total cost was one Opus API call for the review pass, not dozens for the entire build. **llama.cpp is the right tool for serious local inference.** Ollama is great for getting started. For production use — where you need reasoning budgets, precise context control, and OpenAI-compatible APIs that tools like Coder can consume — llama-server gives you the knobs you actually need. ## The Settings That Made It Work For anyone running Gemma 4 locally, here's the configuration that turned it from a planning machine into a shipping machine: ```bash llama-server \ -m gemma-4-26B-A4B-it-UD-Q4_K_M.gguf \ --ctx-size 32768 \ # 32K context — ~19 GB VRAM on 5090 -n 32768 \ # 32K max output tokens --reasoning-budget 4096 \ # Cap thinking at 4K tokens --reasoning-format deepseek \ # Expose thinking in API response --parallel 1 \ # Single slot — don't OOM with 4 × 32K -ngl 999 # All layers on GPU ``` The `--reasoning-budget 4096` is the single most important flag. Without it, Gemma can spend its entire output budget on reasoning you never see. With it, the model gets 4K tokens to think, then the rest is for actual code. That one flag is the difference between a model that plans forever and a model that ships. ## What's Next Right now, Gemma 4 serves a single Coder instance on the workstation where it runs. That's fine for one person, but the RTX 5090 is sitting idle most of the day. The obvious next step: **make it available to every machine on the local network.** My wife runs [OpenClaw](https://github.com/openclaw/openclaw) on a Mac Mini in the other room. With Tailscale already meshing our devices together, pointing her OpenClaw instance at `http://workstation:8080/v1/` is trivially easy — llama-server's OpenAI-compatible API means any tool that speaks the OpenAI protocol can use it. One GPU, multiple clients, zero cloud costs. Beyond that: migrating the remaining Ollama models to llama.cpp (for the same reasoning budget control we needed here), experimenting with longer context windows now that we know the VRAM budget, and — inevitably — the next model showdown when Gemma 4's bigger variants drop. The homelab keeps growing. Who knows? Maybe the lobster starts vibe coding for me, too. ## By the Numbers - **3** attempts before Gemma completed the task (auth fix, OOM fix, success) - **2** prompts in the successful run (vs 8 failed prompts in the original test) - **618** lines written by Gemma 4 across 6 files - **383** lines changed in the Opus polish pass (insertions + deletions) - **804** total lines in the merged feature - **0** cloud API calls for the build phase (Gemma ran 100% local) - **177–181** tokens per second — Gemma's peak generation speed on the RTX 5090 - **~159** tokens per second — effective speed under real agentic load (accumulated context) - **28ms** time to first token — near-instant streaming start - **16.9 GB** model size (Gemma 4 26B-A4B, Q4_K_M quantization) - **~19 GB** total VRAM at 32K context (comfortable fit on 32 GB card) - **4,096** reasoning budget tokens — the setting that made it all work - **$0** inference cost for the feature build - **1** nudge needed ("keep going, don't stop, code and commit") - **7** bugs found in Gemma's code during review (all fixed) - **3** bugs in Opus's original implementation (never merged, never fixed) - **0** bugs in the merged Gemma+Opus version - **1** production feature, live on vibescoder.dev right now — press ⌘K to try it === ## Invisible Failures: The Bugs That Hide in Plain Sight - URL: https://vibescoder.dev/posts/invisible-failures-the-bugs-that-hide-in-plain-sight - Date: 2026-05-03 - Tags: #homelab #agents #debugging - Reading time: 12 min read Four bugs that were silently breaking things for days: a deploy that only crashes on new images, a shell guard that eats your auth tokens, a publish date frozen at draft creation, and a homelab with no emergency remote access. Plus: capacity planning for when you're running AI workspaces on a single machine. --- Lucky you — bonus fix content, and you don't even have to wait until Friday. I had a work trip to Austin coming up. The homelab was humming along at home, but I realized something uncomfortable: if anything went sideways while I was gone, I had no reliable way to fix it. SSH works when everything is running. SSH doesn't help when you need to see a stuck GUI dialog, a frozen window manager, or a service that needs a browser to configure. So before I left, I fixed the access problem. Then, from a hotel room in Austin, I found and fixed three bugs that had been silently breaking things for days. None of them crashed. None of them logged errors. They just quietly did the wrong thing until the consequences finally became visible. ## 1 Setting up Remote Access Before the Trip I'd been putting off remote desktop because "I can always walk over to it." A trip to Austin fixed that mindset. **Evaluating options**: Compared five tools. xrdp is a common recommendation but it's wrong for this use case — it spawns a new desktop session instead of mirroring the existing one. If something is stuck on the real display, xrdp can't help you see it. VNC works but it's laggy and unencrypted by default. Chrome Remote Desktop depends on Google's servers and a running Chrome instance. NoMachine is great but closed source. **RustDesk won**: Open source, self-hostable, mirrors the real desktop, has iOS and macOS clients. **Self-hosted server**: Two Docker containers — a rendezvous server and a relay server — so connections route through my own infrastructure instead of RustDesk's public relays: ```bash sudo docker run -d --name rustdesk-hbbs \ --restart always \ -p 21115:21115 -p 21116:21116 -p 21116:21116/udp -p 21118:21118 \ -v /opt/rustdesk-server:/root \ rustdesk/rustdesk-server hbbs sudo docker run -d --name rustdesk-hbbr \ --restart always \ -p 21117:21117 -p 21119:21119 \ -v /opt/rustdesk-server:/root \ rustdesk/rustdesk-server hbbr ``` **Networking via Tailscale**: Cloudflare Tunnels already handle `coder.vibescoder.dev`, but they only proxy TCP/HTTP — RustDesk's rendezvous server requires UDP on port 21116. Tailscale is a WireGuard mesh VPN that handles TCP and UDP natively. Each tool has its lane: - **Cloudflare Tunnel** → HTTP/S services (Coder dashboard, blog) - **Tailscale** → everything else (SSH, RustDesk, any UDP/TCP service) Clients on macOS and iOS point at the workstation's Tailscale IP. A permanent password means fully remote access — no need to walk over and click "Accept" on a popup, which defeats the entire purpose of remote desktop for recovery scenarios. Everything auto-starts on reboot: Docker containers with `--restart always`, RustDesk and Tailscale as systemd services. Five components confirmed persistent across power cycles. **The architecture now looks like this:** ``` ┌───────────────────────────────────────────────────────┐ │ HOMELAB (Ubuntu + RTX 5090) │ │ │ │ Coder Server ──── Cloudflare Tunnel ──── Internet │ │ (systemd, :3000) (TCP/HTTP only) │ │ │ │ RustDesk Client ── Tailscale Mesh ──── MacBook │ │ (systemd) (TCP + UDP) iPhone │ │ │ │ RustDesk Server (Docker, --restart always) │ │ ├── hbbs (rendezvous, :21115-21116, :21118) │ │ └── hbbr (relay, :21117, :21119) │ │ │ │ llama-server (Gemma 4, :8080) │ │ Tailscale (systemd) │ │ Cloudflared (systemd, tunnel) │ └───────────────────────────────────────────────────────┘ ``` ![Network interfaces on the workstation — Tailscale mesh, Docker bridge, and host networking all coexisting](/images/invisible-failures-the-bugs-that-hide-in-plain-sight/network-interfaces-redacted.png) With that in place, I headed to Austin. ## 2 the Deploy That Only Fails on New Content Three out of four Vercel deploys crashed with the same error: ``` Can't load image https://vibescoder.dev/images/downtime-is-a-feature/vercel-dns-ipv4-error.png: fetch failed Error: Image size cannot be determined. Export encountered an error on /posts/[slug]/opengraph-image/route ``` The blog generates dynamic OpenGraph cards for social sharing — each post gets a unique 1200×630 image with the title, description, and a faded background pulled from the post's first image. The OG image route extracts the first `![alt](/images/...)` reference from the markdown and renders it. The problem was **how** it loaded that image: ```typescript ``` During `next build`, this fetches from the **live production site**. For a new post, those images don't exist on production yet — they're only in the current build's `public/` directory, copied there by the prebuild script. The fetch fails, `next/og` can't determine dimensions, and the entire build crashes. The one post that succeeded? Its first image already existed on production from a previous deploy. This is a classic chicken-and-egg bug. It only affects new content with new images. If you redeploy the same content twice, it works — because the first deploy put the images on the live site. You could publish for weeks without hitting it, then get three failures in a row when you finally add a post with a fresh screenshot. **The fix**: Read from the local filesystem instead of fetching from production. The images are already in `public/images/` at build time, so we read from disk and encode as a base64 data URI: ```typescript const imgPath = path.join(process.cwd(), "public", rawImage); const buf = fs.readFileSync(imgPath); const ext = path.extname(rawImage).replace(".", "").toLowerCase(); const mime = ext === "jpg" || ext === "jpeg" ? "image/jpeg" : `image/${ext}`; firstImage = `data:${mime};base64,${buf.toString("base64")}`; ``` A `try/catch` around the read means a missing image degrades gracefully — no background in the OG card instead of a build-killing crash. ## 3 the Shell Guard That Eats Your Auth While pushing the OG image fix, `git push` failed with an auth error. This had happened before. Every time, we'd manually run `coder external-auth access-token github`, paste the token into the remote URL, and move on. This time we decided to actually trace it. Three layers of GitHub auth were configured in the workspace startup script, and **all three were broken** in agent sessions: | Component | What It Did | Why It Failed | |---|---|---| | `credential.helper` | Shell function reading `$GITHUB_TOKEN` | Env var was empty | | `GITHUB_TOKEN` export | Appended to `~/.bashrc` | Below the interactive guard | | `gh auth login` | Ran at startup | Token persisted, but `gh` also checks `GH_TOKEN` env | The startup script appended `export GITHUB_TOKEN=...` to `~/.bashrc`. That line landed at line 118 — after the interactive guard at line 8: ```bash case $- in *i*) ;; *) return;; esac ``` Every non-interactive shell — which is what Coder agent `execute()` calls use — bailed out at line 8 and never reached the exports. The credential helper then read an empty `$GITHUB_TOKEN` and returned an empty password. Git got a 401. The agent worked around it. Nobody noticed. **The fix** was three changes: **Credential helper calls `coder external-auth` directly** — no dependency on env vars, fresh token on every git operation: ```bash git config --global credential.helper \ '!f() { echo "username=x-access-token"; echo "password=$(coder external-auth access-token github 2>/dev/null)"; }; f' ``` **Coder's `env` block injects tokens via Terraform** — set in the agent process environment by Coder itself, inherited by every `execute()` call, no shell sourcing needed: ```hcl resource "coder_agent" "main" { env = { GITHUB_TOKEN = data.coder_external_auth.github.access_token GH_TOKEN = data.coder_external_auth.github.access_token } } ``` **Cleanup of stale `.bashrc` entries** — removed the old exports so they don't confuse future debugging. The auth bug had been hiding for multiple sessions. The startup script *looked* correct. The agent silently worked around it every time. The fix was to stop relying on shell init files entirely and let Coder's process environment do the work at a level the shell can't interfere with. ![Coder Agents sidebar — multiple models, multiple errors, none of them obvious until you trace the auth chain](/images/invisible-failures-the-bugs-that-hide-in-plain-sight/coder-agents-error-sidebar.png) ## 4 the Date That Froze at Draft Creation Published "The Agentic Gap" post. It went live. Then noticed the date: **April 26** — three days ago. The post appeared sorted behind three days of other content instead of at the top. The blog engine had **no mechanism to update the frontmatter `date` when a draft is published**. The flow: 1. Draft created → Claude sets `date` to the day it generates the content 2. Draft sits unpublished for N days 3. Someone flips `published: false` → `published: true` 4. Post appears on the blog sorted by its **creation date**, not its **publish date** Two publish paths existed — the admin UI and the API — and neither touched the date. The admin UI did a single regex replace on the boolean. The API had a `fixDateYear()` function that corrects stale years (e.g., "2025" when it's 2026), but same-year drafts sailed right through. **The fix** was two layers: **Client-side**: `handlePublish()` stamps today's date immediately after flipping the boolean: ```typescript const today = new Date().toISOString().split("T")[0]; published = published.replace( /^date:\s*'[^']*'/m, `date: '${today}'`, ); ``` **Server-side**: A new `stampPublishDate()` function detects the `false` → `true` transition by comparing old and new content, and rewrites the date if it's a fresh publish. Safety net for all paths. There's a third publish path — an agent directly editing the MDX and pushing to git, which is exactly what caused this bug. No code can fix that path. The fix there is process: the agent skill now instructs agents to always set the date to today when publishing. The meta moment: the agent that introduced this bug also diagnosed, fixed, and documented it. In about fifteen minutes. ## 5 Capacity Planning How Many Workspaces Can This Machine Run With the homelab now running Coder, Gemma 4 via llama.cpp, RustDesk, Tailscale, and Cloudflare — the question became: how much headroom is left? **The inventory**: Ryzen 9 9950X3D, 64 GB RAM, RTX 5090 32 GB VRAM. Profiled from inside a container via `/proc` and from the host via Termius on an iPhone. ![Capacity diagnostics via Termius on iPhone — Docker stats, RAM breakdown, and service status from a hotel room](/images/invisible-failures-the-bugs-that-hide-in-plain-sight/iphone-capacity-planning-redacted.png) **Key findings**: | Category | Memory | |---|---| | Host services (GNOME, Coder, Docker, Tailscale, Cloudflare, RustDesk) | ~5 GB | | 7 idle workspaces | ~1.7 GB (230–270 MB each) | | Gemma 4 (32K context, llama.cpp) | ~19 GB **VRAM** — zero RAM impact | | Available for workspaces | ~58 GB | **The GPU insight**: Gemma 4 runs entirely in VRAM on the RTX 5090. Zero system RAM impact. Workspaces call it via the OpenAI-compatible API on the host. The 32 GB VRAM pool is completely separate from the 64 GB system RAM — running a local LLM doesn't reduce workspace capacity at all. **Capacity estimates**: 8–12 active agent sessions comfortably (CPU is the bottleneck, not RAM). Dozens of idle workspaces parked at ~250 MB each. **Resource guardrails**: Added an 8 GB per-container memory limit as a safety net — 32× normal usage, so it never constrains normal work, but a runaway `npm install` or memory leak gets OOM-killed cleanly instead of dragging the whole host into swap: ```hcl resource "docker_container" "workspace" { memory = 8192 # MB — safety net, not a throttle memory_swap = 8192 # equal to memory = no swap for container } ``` **Autostop**: 2-hour default TTL with 1-hour activity bump. Workspaces auto-stop when forgotten. All configurable via `coder templates edit` — template metadata, not Terraform. ## What I Learned **Set up remote access before you need it.** Every one of the fixes below happened from a hotel room because I'd set up RustDesk and Tailscale the day before I left. If I'd waited, I'd have come home to three days of broken deploys and a post sorted in the wrong place. **Invisible bugs are the most expensive.** The deploy bug only hit new content. The auth bug was silently worked around. The date bug was three days stale before anyone noticed. **Shell init files are a liability.** Anything that depends on `.bashrc` or `.profile` is fragile by default. Non-interactive shells, cron jobs, agent tool calls — none of them source your profile. If auth or config needs to be available everywhere, put it in the process environment or in a tool that's always in `$PATH`. **Self-hosted doesn't mean unmanaged.** Adding RustDesk, Tailscale, and resource guardrails isn't gold-plating — it's the difference between a homelab that works when you're sitting in front of it and one that works when you're debugging from a phone in another room. Recovery access is table stakes. **The agent finds its own bugs.** The publish-date bug was introduced by an agent, discovered by a human, then diagnosed, fixed, and documented by the same agent. That loop — agent ships, human reviews, agent fixes — is becoming the default workflow. ## By the Numbers - **1** trip to Austin that forced the remote access setup - **4** invisible bugs found and fixed — three of them remotely - **3** failed Vercel deploys from the OG image chicken-and-egg bug - **3** layers of broken GitHub auth (credential helper, env var, gh CLI) - **3 days** a post was live with the wrong date before anyone noticed - **~58 GB** RAM available for workspaces after all services running - **8–12** concurrent active agent sessions the workstation can handle - **8 GB** per-container memory limit as a safety net - **5** tools evaluated for remote desktop — RustDesk won - **2** Docker containers for self-hosted RustDesk server - **5** services confirmed persistent across reboots - **0** ports exposed to the internet (Tailscale mesh, Cloudflare tunnel) - **~15 minutes** from "the date is wrong" to fix deployed across both repos - **~30 minutes** from "how do I remote desktop" to working iPhone → Linux access - **1** agent that introduced a bug, then found and fixed it === ## Your AI Strategy Has a Blind Spot: An SEO and AEO Audit of vibescoder.dev - URL: https://vibescoder.dev/posts/your-ai-strategy-has-a-blind-spot - Date: 2026-05-02 - Tags: #seo #aeo #cloudflare #agents - Reading time: 15 min read A deep audit of vibescoder.dev revealed that Cloudflare was silently blocking every major AI crawler — even after we'd explicitly turned that setting off. Here's what we found, what we fixed, and the complete playbook for making your site visible to both search engines and AI agents. --- I spend a lot of time thinking about how AI agents discover and consume content. I run a company that builds developer tools. I write a blog about building with AI agents. And most importantly, I'm married to a woman that runs an [AI consulting practice](https://genedge.co). Through the home-office wall I've heard her warn many a client that they have a silent suppressor in their content strategy if they're a Cloudflare customer. She recommends a site audit. And she was right. Until this morning, **every major AI crawler was blocked from reading my site**. Not by choice. Not by misconfiguration. By a Cloudflare setting I'd already turned off — that got silently re-enabled by a different setting I didn't know existed. If you're a content creator, marketer, or engineer who cares about whether ChatGPT, Perplexity, Google AI Overviews, or Claude can find your work — read this. The infrastructure between your content and your audience may be working against you. ## The TLDR for Non-Technical Readers If you don't want to read the whole audit, here's what matters: 1. **Cloudflare's free tier blocks AI search engines by default.** If your site uses Cloudflare (and millions do), your content may be invisible to ChatGPT, Perplexity, Claude, and Google's AI features — even if you never asked for that. 2. **There are now two categories of discoverability.** Traditional SEO (Google search results) and AEO — Answer Engine Optimization (AI-powered search and assistants). You need both. They require different things. 3. **The fix for Cloudflare takes 60 seconds** — but you have to know it exists. Go to Security → Settings → "Manage your robots.txt" and switch from "Instruct AI bots to not scrape content" to either "Content Signals Policy" or "Disable robots.txt configuration." 4. **There's a new file called `llms.txt`** that's becoming the robots.txt for AI. It tells AI agents what your site is, what it covers, and where to find content. If you don't have one, you're leaving discoverability on the table. ## The TLDR for Technical Readers We ran a full SEO + AEO audit against vibescoder.dev and found 20 issues across 4 severity levels. The highlights: - **4 P0 (critical):** Cloudflare's managed robots.txt was blocking GPTBot, ClaudeBot, Google-Extended, and 5 others. RSS feed had wrong URL prefix (15 broken links). Sitemap.xml was referenced but returned 404. Duplicate `User-agent: *` blocks in robots.txt. - **6 P1 (high):** No JSON-LD structured data. No llms.txt. No canonical URLs. No heading anchor IDs. Missing article:author/tag meta. Homepage force-dynamic. - **Everything was fixed in a single session** — 17 files changed, 428 insertions, pushed and deployed. The commit: [SEO/AEO overhaul](https://github.com/carryologist/the-vibe-coder/commit/15b3483). ## The Audit I asked my Coder agent to evaluate vibescoder.dev on two dimensions: traditional search engine optimization (SEO) and [Answer Engine Optimization](https://ahrefs.com/blog/answer-engine-optimization/) (AEO) — making the site discoverable and citable by AI agents like ChatGPT Search, Perplexity, Google AI Overviews, and Claude. The agent cloned the engine repo, crawled the live site, inspected every response header, parsed every meta tag, and cross-referenced the codebase against both SEO and AEO best practices. The results were humbling. ## The Cloudflare Gotcha Yes Again I wrote about Cloudflare's AI crawler settings [two weeks ago](/posts/downtime-is-a-feature-custom-domains-cloudflare-and-mcp). In that post, I specifically called out that Cloudflare's free tier has **"Block AI bots"** and **"AI Labyrinth"** turned on by default. I explicitly turned both off. I even wrote this: > *"If your site exists for thought leadership, you want AI services to find, index, and cite your content. Blocking AI crawlers is blocking your distribution channel."* I was right. And I was still blocked. **The problem**: Cloudflare has a *separate* setting called **"Manage your robots.txt"** under Security → Settings. It's not the same as "Block AI bots." It's a newer feature that injects directives directly into your robots.txt file at the edge — *after* your origin server responds. Here's what the agent found when it compared my repo's `robots.txt` (100 bytes, 7 lines) to what Cloudflare was actually serving: | Metric | Value | |--------|-------| | **My robots.txt** | 100 bytes, 7 lines | | **Content-Length header** | 100 (Vercel's original) | | **Actual response body** | 1,838 bytes, ~65 lines | Cloudflare was prepending 1,738 bytes of content — including `Disallow: /` rules for ClaudeBot, GPTBot, Google-Extended, Amazonbot, CCBot, Bytespider, and meta-externalagent — **without updating the Content-Length header**. The setting responsible? "Instruct AI bots to not scrape content," which was selected by default. **The fix**: Security → Settings → "Manage your robots.txt" → select "Disable robots.txt configuration." This tells Cloudflare to stop modifying your robots.txt entirely. Your origin file gets served as-is. **Why "Disable" and not "Content Signals Policy"?** The Content Signals option keeps a `Content-Signal: ai-train=no` directive, which tells AI crawlers not to use your content for model training. That sounds reasonable — but for a personal blog trying to maximize reach, being in the training corpus means AI models are more likely to know about you and reference your ideas. The risk it protects against (content absorbed without credit) is theoretical. The cost (reduced presence in AI systems) is concrete. **Gotcha #1**: Cloudflare has three separate AI-related settings, and changing one doesn't affect the others. You need to check all three: | Setting | Location | What It Does | |---------|----------|-------------| | **Block AI Bots Scope** | Security → Settings | Deploys firewall rules blocking AI training crawlers | | **AI Labyrinth** | Security → Settings | Injects fake content links to trap non-compliant bots | | **Manage your robots.txt** | Security → Settings | Modifies robots.txt at the edge to add AI crawler directives | I had turned off #1 and #2 weeks ago. But #3 was still on — silently rewriting my robots.txt at the CDN layer. Here's the full picture — the Security Overview flagging the AI-related action items, and each of the three settings: ## What Is AEO AEO — [Answer Engine Optimization](https://ahrefs.com/blog/answer-engine-optimization/) — is the practice of making your content discoverable and citable by AI agents. (You'll also see it referred to as AI Engine Optimization or Agentic Engine Optimization — the discipline is new enough that the name is still settling.) It's the emerging counterpart to SEO. Where SEO focuses on Google's traditional index, AEO targets the systems that power ChatGPT Search, Perplexity, Google AI Overviews, Claude, and whatever comes next. The key differences: | | SEO | AEO | |---|---|---| | **Primary consumer** | Googlebot | GPTBot, ClaudeBot, PerplexityBot, Google-Extended | | **Content format** | HTML with meta tags | Structured data (JSON-LD), plain text (llms.txt), RSS | | **Discovery mechanism** | Sitemap, backlinks, crawling | Sitemap, RSS, llms.txt, structured data | | **Ranking signal** | PageRank, content quality, Core Web Vitals | Authorship (Person schema + sameAs), recency, structured data | | **Citation style** | Blue link with snippet | Inline citation with direct quote and link | | **Key enabler** | Canonical URLs, meta descriptions | JSON-LD, llms.txt, heading anchors for deep linking | You need both. Many of the improvements help both. But some are AEO-specific. ## AEO-Specific Changes These improvements specifically target AI agent discoverability: ### Llms.txt and LLMs-Full.txt `llms.txt` is an emerging convention — think of it as robots.txt for AI *comprehension* rather than crawling. It tells AI agents what your site is, what topics it covers, and where to find content. We created two files: - **`/llms.txt`** — a structured summary: site description, author, topics, key posts, and links - **`/llms-full.txt`** — a dynamic route that serves every published post's full content as plain text The full-content version is the important one. When an AI agent wants to cite your work, it needs the actual content — not just metadata. `llms-full.txt` is a single endpoint that gives it everything. ### Person Schema with `Sameas` JSON-LD structured data tells AI engines *who* wrote something and *where else* that person exists online. The `sameAs` property connects identity across platforms: ```json { "@type": "Person", "name": "Rob Whiteley", "url": "https://vibescoder.dev/about", "jobTitle": "CEO", "sameAs": [ "https://www.linkedin.com/in/rwhiteley", "https://github.com/carryologist", "https://x.com/rwhiteley0" ], "worksFor": { "@type": "Organization", "name": "Coder", "url": "https://coder.com" } } ``` When ChatGPT or Perplexity decides whether to cite "Rob Whiteley, CEO of Coder" in a response about AI-assisted development, this structured data is what gives it confidence in the attribution. ### Full-Content RSS The existing RSS feed only had `` (a short excerpt). AI agents that consume RSS — and Perplexity in particular indexes it — get significantly more context from full-content feeds. We added `` with the full post body, plus `` and `` tags. ### Unblocking AI Crawlers The Cloudflare fix described above. The single highest-impact AEO change — going from completely invisible to fully accessible. ## SEO-Specific Changes These target traditional Google search: ### Sitemap.xml robots.txt referenced it. It didn't exist. Every SEO tool and Google Search Console would flag this. We created `src/app/sitemap.ts` with dynamic generation — all posts, tags, and static pages with `lastmod` dates from the changelog. ### Canonical URLs No page had ``. Without it, Google can treat URL variants (`?utm_source=twitter`, `?ref=hackernews`) as separate pages. We added explicit canonical URLs to every page type — homepage, posts, about, tags, and individual tag pages. ### Homepage Caching The homepage was set to `force-dynamic` — every request hit the server with zero caching. For a blog that publishes daily at most, that's unnecessary. We switched to ISR with a 60-second revalidation window. (Vercel still serves it dynamically due to a `cookies()` call for admin detection — a future refactor.) ### Custom 404 Page The default Next.js 404 is a dead end. Our custom version shows recent posts and navigation links — keeping both users and crawlers moving through the site instead of bouncing. ## Changes That Help Both Most improvements benefit both SEO and AEO: ### JSON-LD Structured Data The single biggest miss. We added three schema types: - **`WebSite`** — site-level metadata with author info (every page) - **`BlogPosting`** — per-post schema with headline, dates, author, keywords, reading time (post pages) - **`BreadcrumbList`** — navigation hierarchy (post pages) For SEO, this enables rich results in Google — article carousels, author info, breadcrumbs. For AEO, it's how AI engines understand content relationships and authorship with confidence. ### Heading Anchor Ids Added `rehype-slug` to the MDX pipeline. Every H2 and H3 now gets an auto-generated `id` attribute. - **SEO**: Google uses these for "jump to" links in search results and featured snippets. - **AEO**: AI agents cite specific sections via fragment URLs (`#the-cloudflare-gotcha`). Without heading IDs, citations can only link to the full page. ### RSS Feed Fix Every link in the RSS feed was a 404. The feed used `/blog/` as the URL prefix, but the actual routes use `/posts/`. All 15 posts were broken. One-line fix, massive impact — RSS is a primary discovery mechanism for both Google and AI agents. ### Article Meta Tags Added `article:author`, `article:tag`, `article:modified_time`, and `og:site_name` to post OpenGraph metadata. These help both Google and AI engines categorize and attribute content correctly. ### Image Improvements MDX images now render inside `
` with `
` elements, and images without explicit alt text get an auto-generated fallback from the filename. Both changes improve how crawlers — traditional and AI — understand image content. ## The Cloudflare Settings While We Were in the Dashboard While fixing the robots.txt issue, we also optimized two other Cloudflare settings: - **Early Hints** — enabled. Cloudflare sends `103 Early Hints` responses from the edge, letting browsers start loading fonts and CSS before Vercel even responds. - **Smart Tiered Caching** — enabled. Cloudflare edge nodes share cached content with each other, reducing origin hits. Ready to deliver benefits once ISR caching is fully enabled. - **AI Labyrinth** — confirmed still off. This injects fake content links to trap AI crawlers — the opposite of what a content site wants. ## The Complete Scorecard Every change, its impact, and whether it addresses AEO, SEO, or both: | Change | Impact | AEO | SEO | |--------|--------|-----|-----| | Disable Cloudflare managed robots.txt | **Critical** — AI crawlers could not access the site | ✅ | — | | Fix RSS feed URLs (`/blog/` → `/posts/`) | **Critical** — all 15 RSS links were 404s | ✅ | ✅ | | Create sitemap.xml | **Critical** — referenced in robots.txt but returned 404 | ✅ | ✅ | | Consolidate robots.txt (disable CF injection) | **Critical** — duplicate User-agent blocks caused ambiguity | — | ✅ | | Add JSON-LD structured data | **High** — zero structured data across entire site | ✅ | ✅ | | Create llms.txt + llms-full.txt | **High** — no AI discovery files existed | ✅ | — | | Add canonical URLs to all pages | **High** — no page declared itself as canonical | — | ✅ | | Add heading anchor IDs (rehype-slug) | **High** — no deep linking possible | ✅ | ✅ | | Add article:author, article:tag to OG meta | **High** — tags and author missing from metadata | ✅ | ✅ | | Add Person schema with sameAs | **High** — no cross-platform identity linking | ✅ | ✅ | | Switch homepage to ISR (revalidate: 60) | **Medium** — every request was a cold server render | — | ✅ | | Add RSS author + full content (`content:encoded`) | **Medium** — feed had excerpts only, no author | ✅ | ✅ | | Add twitter:site and twitter:creator | **Medium** — social cards had no account attribution | — | ✅ | | Create custom 404 page | **Medium** — default 404 was a dead end | — | ✅ | | Wrap images in figure/figcaption | **Low** — bare img tags with no semantic context | ✅ | ✅ | | Alt text fallback from filenames | **Low** — empty alt on content images | ✅ | ✅ | | Remove x-powered-by header | **Low** — minor information disclosure | — | ✅ | | Add humans.txt | **Low** — minor authorship signal | ✅ | — | | Enable Cloudflare Early Hints | **Low** — browsers preload assets faster | — | ✅ | | Enable Smart Tiered Caching | **Low** — prepared for when ISR is fully active | — | ✅ | **Total: 20 changes. 13 help AEO. 17 help SEO. 11 help both.** ## What I Learned **AEO is a real discipline now, not a buzzword.** The gap between "my content exists on the internet" and "AI agents can find, understand, and cite my content" is significant. Structured data, llms.txt, full-content RSS, heading anchors — these aren't nice-to-haves. They're the difference between being in the AI conversation and being invisible to it. **Your CDN can silently undermine your content strategy.** This is the one that stings. I *knew* about the Cloudflare AI bot setting. I *wrote a blog post about turning it off.* And a different setting — one I didn't know existed — was doing the same thing through a different mechanism. If you use Cloudflare, check your robots.txt right now. Not the file in your repo — the one Cloudflare is actually serving. `curl https://yoursite.com/robots.txt` and compare it to what you expect. **The audit paid for itself in the first finding.** Everything else — the JSON-LD, the canonical URLs, the sitemap — those are incremental improvements that compound over time. But the Cloudflare fix was binary: invisible → visible. Every day that setting was on was a day ChatGPT Search, Perplexity, and Google AI Overviews couldn't index my content. ## What's Next The one thing we identified but didn't implement: **FAQPage schema** for how-to posts. Several posts follow a problem/solution pattern that could surface as direct answers in AI search. The frontmatter already has a `type` field distinguishing `how-to` from `opinion` — the infrastructure is there. That's next. ## By the Numbers - **1,738** bytes of robots.txt injected by Cloudflare without updating Content-Length - **8** AI crawlers blocked (GPTBot, ClaudeBot, Google-Extended, Amazonbot, CCBot, Bytespider, Applebot-Extended, meta-externalagent) - **15** RSS feed links returning 404 — every single one - **0** → **3** JSON-LD schema types (WebSite, BlogPosting, BreadcrumbList) - **0** → **5** pages with canonical URLs - **17** files changed, **428** lines added - **3** Cloudflare settings that control AI crawlers — and you have to check all of them - **60 seconds** to fix the Cloudflare setting that was blocking all AI visibility - **~2 hours** for the full audit and implementation of all 20 changes - **1** blog post that I thought had solved this problem — it hadn't === ## Friday Fixes: The Agent Was Flying Blind - URL: https://vibescoder.dev/posts/friday-fixes-the-agent-was-flying-blind - Date: 2026-05-01 - Tags: #agents #meta #building-in-public - Reading time: 13 min read A CRLF bug silently broke every workspace for weeks. Then we fixed it, taught the agent to remember, moved templates to Git, squashed a nested heredoc, cut boot time from 91 seconds to 5, automated the screenshot pipeline, and built scheduled publishing — which this post used to publish itself. Ten fixes, one week. --- Last Friday's post covered nine small improvements — CSS fixes, social cards, Slack integrations. This week's fixes are different. These aren't cosmetic. I discovered that the AI agent powering this entire blog had been silently broken since day one, compensating on its own without telling me. The startup script, the MCP config, the skill files — none of it was being delivered to workspaces. Every session started from scratch. Here's how I found it, fixed it, and then kept pulling the thread until the whole developer experience was rebuilt. ## 1 the CRLF Bug That Broke Everything **The problem**: I asked the agent to publish a blog post. Simple task — flip `published: false` to `true`, push, done. Instead, the agent spent several minutes exploring two repos, trying to figure out where posts live, how deploys work, and which repo to push to. It had zero context. **Root cause**: The workspace template's startup script had Windows-style CRLF line endings (`\r\n`). Bash choked on line 1: ``` /bin/bash: line 1: set: -\r: invalid option ``` That meant `.mcp.json` and `.agents/skills/vibescoder-blog/SKILL.md` were **never created** in any workspace. Every agent session started completely blind. The agent compensated by installing tools itself and exploring repos manually — per the system instructions — so nothing visibly broke. But the efficiency gains I'd described in earlier posts? Aspirational, not operational. **The fix**: Rewrote the startup script with LF line endings. But that was just the beginning. **The uncomfortable part**: Two earlier published posts — "From Idea to Infrastructure" and "Downtime Is a Feature" — describe the startup script toolchain and MCP setup as working. They accurately describe what was *configured*, but the CRLF bug was already present. None of it was actually delivered until this fix. Worth noting for honesty's sake. ## 2 Teaching the Agent to Remember **The problem**: Even after fixing the CRLF bug, the agent still needed to be told everything about the blog's architecture every session. Two repos, a deploy pipeline, frontmatter schema, security rules, writing conventions — all living in my head instead of the workspace. **The fix**: Created the `vibescoder-blog` agent skill — a 4.6 KB markdown file at `.agents/skills/vibescoder-blog/SKILL.md` that documents everything the agent needs: - Both repos and their roles (engine vs. content) - The deploy pipeline (push content → GitHub Action → Vercel deploy hook) - Step-by-step publishing instructions - Post frontmatter schema - Content repo directory layout - Blog fodder format conventions - Writing style guidelines - Security redaction rules The key line: *"You do NOT need to touch the engine repo to publish content. Just push to the content repo."* With a user instruction — `When I mention the blog, vibescoder, or content work, read the skill "vibescoder-blog"` — the agent lazy-loads this context on first reference. A 30-second publishing task is now actually a 30-second task. ## 3 Templates in Git **The problem**: The Coder workspace template was edited through the web UI. No version control, no way for the agent to propose template fixes, and CRLF issues could creep in from browser-based editing. The CRLF bug that broke everything? Probably introduced during a UI edit. **The fix**: Created a `carryologist/coder-templates` repo with the full Terraform source: ``` coder-templates/ ├── docker/ │ ├── build/ │ │ └── Dockerfile │ └── main.tf └── README.md ``` The workflow: edit `main.tf` in the repo → push → SSH into workstation → `coder templates push docker --yes`. Optional GitHub Actions CI for auto-push on merge. Template changes are now reviewable, diffable, and blame-able. ## 4 the Nested Heredoc That Wouldn't Die **The problem**: With the startup script in Git and CRLF fixed, the next step was embedding the MCP config and skill file directly in the Terraform template. Nested shell heredocs inside Terraform's `<<-EOT` seemed like the obvious approach. **Root cause**: Terraform's `<<-EOT` strips leading whitespace from all lines — including the closing delimiters of nested heredocs. The shell never sees the unindented `MCP` or `SKILL` terminators, so the heredoc never closes: ``` syntax error: unexpected end of file ``` **The fix**: Base64 encoding. Encoded both files as base64 strings and decoded at runtime: ```bash echo '' | base64 -d > /home/coder/.mcp.json echo '' | base64 -d > /home/coder/.agents/skills/vibescoder-blog/SKILL.md ``` No heredocs, no whitespace sensitivity, no Terraform interpolation issues. Ugly but bulletproof. **The full iteration log**: | Attempt | Error | Fix | |---------|-------|-----| | 1 | `set: -\r: invalid option` | CRLF → LF | | 2 | `syntax error: unexpected end of file` | Nested heredocs → base64 | | 3 | Same error | Cached template version — re-cloned and pushed again | | 4 | `Module "nvm" cannot be found` | Removed phantom nvm module | | 5 | Success | Clean boot, skill + MCP config present | Five attempts. Each one taught something different about how Terraform, bash, and Coder templates interact. ## 5 Workspace Boot 91 Seconds to 5 **The problem**: Coder workspaces took 91 seconds to start. Every single boot, not just the first one. The agent logs had been telling me the whole time: ``` 2026-04-28 20:58:09.394 [info] running agent script... 2026-04-28 20:59:40.948 [info] script completed execution_time=1m31.55332s exit_code=0 ``` **Root cause**: The default Docker template pattern — `count = data.coder_workspace.me.start_count` — recreates the container on every start/stop cycle. Only `/home/coder` persists via a Docker volume. Everything installed to `/usr` is gone. The startup script was running three `apt-get update` calls (37.7 MB of metadata each), reinstalling gh, nodejs, npm, sqlite3, redis-tools, and running `npm install -g vercel` (30+ seconds alone) on every boot. **The fix**: Built a custom Docker image that bakes everything in: ```dockerfile FROM codercom/enterprise-base:ubuntu USER root RUN apt-get update && \ apt-get install -y --no-install-recommends \ curl git zip unzip sqlite3 redis-tools nodejs npm \ && rm -rf /var/lib/apt/lists/* RUN curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg \ | dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg 2>/dev/null && \ echo "deb [arch=...] https://cli.github.com/packages stable main" \ > /etc/apt/sources.list.d/github-cli.list && \ apt-get update && apt-get install -y --no-install-recommends gh && \ rm -rf /var/lib/apt/lists/* RUN npm install -g vercel RUN curl -LsSf https://astral.sh/uv/install.sh | sh && \ mv /root/.local/bin/uv /usr/local/bin/uv USER coder ``` Updated `main.tf` with a `docker_image` resource that uses `filemd5` — Docker only rebuilds when the Dockerfile actually changes: ```hcl resource "docker_image" "workspace" { name = "coder-workspace:latest" build { context = "./build" } triggers = { dockerfile_hash = filemd5("./build/Dockerfile") } } ``` Stripped the startup script to auth + config only. No more install blocks. | Metric | Before | After | |--------|--------|-------| | Startup time | 91s | ~5s | | `apt-get update` calls per boot | 3 | 0 | | Packages downloaded per boot | ~52 MB | 0 MB | | First image build | N/A | ~2 min (one-time) | 18x faster. The answer was in the agent logs the whole time. ## 6 GitHub Auth Fix **The problem**: Discovered while debugging the CRLF fallout — `$GITHUB_TOKEN` was empty in agent sessions. The agent was silently failing on GitHub operations and working around it by using `gh auth login` interactively or skipping auth-dependent steps entirely. **Root cause**: The startup script was supposed to run `coder external-auth access-token github` and export the token, but the CRLF bug meant that line never executed. Even after the CRLF fix, the auth block needed to be in the right place in the stripped-down startup script. **The fix**: Added the auth sequence to the new minimal startup script: ```bash GITHUB_TOKEN=$(coder external-auth access-token github) export GITHUB_TOKEN echo "$GITHUB_TOKEN" | gh auth login --with-token git config --global credential.helper \ '!f() { echo "password=$GITHUB_TOKEN"; }; f' ``` Token from Coder's external auth → exported to env → piped into `gh` CLI → wired into git credential helper. Four lines, zero interactive prompts. ## 7 Automated Screenshot Pipeline **The problem**: Screenshots from homelab sessions were piling up with no workflow. Taking a screenshot on the workstation, manually committing to the content repo, then during blog sessions manually analyzing dozens of images one by one. We'd just done this for the Cloudflare and Round 2 posts — it took real time. **The fix**: Built an automated screenshot inbox using `inotifywait` and a systemd user service: 1. Take a screenshot on the workstation (Print Screen, gnome-screenshot, etc.) 2. The `sync-screenshots` service detects the new file in `~/Pictures/Screenshots/` 3. Auto-commits and pushes to `blog-drafts/screenshots/` in the content repo 4. During blog sessions, agents `git pull` and find screenshots waiting in the inbox 5. Agents analyze, select, rename, and place — the editorial process stays human/agent-directed The whole thing is an idempotent setup script: verify `gh` CLI → clone repo if needed → install `inotify-tools` → write watcher script → write systemd service → enable + start → enable lingering (survives reboots without a desktop session). **Meta moment**: The first screenshot the pipeline synced was a screenshot of the setup script's own "Setup complete!" output. ## 8 Blog Post Style Consistency **The problem**: The Round 2 Model Showdown draft was missing the ending structure every other post follows. Analysis of all 12 published posts revealed a consistent template: "What I Learned" → "What's Next" → "By the Numbers." The Round 2 draft had the first but was missing "What's Next" entirely, and "By the Numbers" used plain text instead of bold metrics. **The fix**: Added a "What's Next" section teasing the Gemma vs Opus head-to-head on a real production task. Reformatted "By the Numbers" from `- 6 local models benchmarked (parenthetical)` to `- **6** local models benchmarked — context with em dash`. Matches every other post on the site. Small, but consistency is the difference between a blog and a collection of posts. ## 9 Image Curation and Security Review **The problem**: 21 raw screenshots from the April 25–26 sessions sitting in `blog-drafts/` with timestamp filenames. Two unpublished posts (Cloudflare/MCP and Round 2 Showdown) had zero images despite covering highly visual topics. **The fix**: Analyzed all 21 screenshots via OCR. Selected 9, rejected 12. **Selected** — 6 for the Cloudflare post (DNS setup, tunnel success, SSL config, the AI bot toggle screenshot every content creator needs to see) and 3 for Round 2 (the 579 GB download progress bar, the conversation mode bug, raw terminal benchmark output). **Rejected** — 5 redundant Cloudflare UI pages, 1 full desktop with email visible in browser tabs, 1 Coder settings page with "API Keys" sidebar visible (no keys shown but bad optics), 1 low-res terminal, 2 workspace debugging screenshots, 1 unrelated content, 1 benchmark command list. Every selected image passed a security review for API keys, tokens, email addresses, internal URLs, and passwords before inclusion. ## 10 Scheduling This Post While Writing This Post This one happened in real time. I was reviewing the draft of this very post with the agent and realized: I'm going to want this to go live at 7:00 AM on Friday, not whenever I happen to remember to flip a flag. Thursday Thoughts on Thursdays, Friday Fixes on Fridays — if the blog has a recurring content calendar, it needs scheduled publishing. **The problem**: Publishing a post meant manually flipping `published: false` to `true` and pushing. No way to write a post on Wednesday night and have it go live Friday morning. **The fix**: A new `publishAt` frontmatter field and a GitHub Action that runs every 15 minutes: ```yaml published: false publishAt: '2026-05-02T07:00:00-05:00' ``` The `scheduled-publish.yml` workflow scans all `.mdx` files for the combination of `published: false` and a `publishAt` timestamp in the past. When it finds one, it flips the flag, removes the `publishAt` line (so frontmatter stays clean), commits as `scheduled-publish[bot]`, and pushes. The existing deploy trigger fires on that push — Vercel rebuilds, post goes live. The `publishAt` field accepts any ISO 8601 timestamp with timezone offset, so `07:00:00-05:00` means 7:00 AM Central regardless of where the GitHub runner is. **The meta moment**: The first post to use scheduled publishing is this one. The `publishAt` in the frontmatter above was added during the same agent session that wrote the workflow. We built the feature and immediately dogfooded it — the agent shipped the infrastructure and then used it on itself. ## What I Learned **Invisible bugs are the most expensive.** The CRLF bug didn't crash anything. The agent silently compensated — installing tools itself, exploring repos manually — so nothing visibly broke. But every session paid a tax: minutes of unnecessary exploration, 91 seconds of unnecessary boot time, zero institutional memory. The "working" system was burning time on every interaction. **Skills are the agent's long-term memory.** Without the skill file, every session started with the agent rediscovering the blog's architecture from scratch. With it, the agent knows both repos, the deploy pipeline, the frontmatter schema, and the security rules before it writes a single line. The difference between a capable assistant and an amnesiac one is a 4.6 KB markdown file. **Bake what you know, script what changes.** The startup script pattern — install tools on every boot — works for prototyping. Once you know your toolchain, put it in a Docker image and strip the script to auth and config. 91 seconds to 5 seconds, and the only cost is a two-minute one-time build. **Five attempts is normal.** The template push cycle — CRLF, heredocs, caching, phantom module, success — felt frustrating in the moment. But each failure was a different class of bug (encoding, Terraform semantics, caching behavior, dependency resolution). Five attempts across five different failure modes isn't thrashing. It's debugging. --- ## Files Changed - `docker/main.tf` — startup script CRLF fix, base64 skill/MCP injection, boot optimization, auth fix - `docker/build/Dockerfile` — new, custom workspace image with all tools baked in - `.agents/skills/vibescoder-blog/SKILL.md` — new, agent skill for blog operations (delivered via base64 in template) - `.mcp.json` — new, MCP server config (delivered via base64 in template) - `scripts/setup-screenshot-sync.sh` — new, idempotent screenshot pipeline installer - `~/.local/bin/sync-screenshots.sh` — new, inotifywait-based file watcher - `~/.config/systemd/user/sync-screenshots.service` — new, systemd user service - `blog-drafts/screenshots/README.md` — new, screenshot inbox conventions - `content/posts/model-showdown-round-2-*.mdx` — added "What's Next," reformatted "By the Numbers" - `.github/workflows/scheduled-publish.yml` — new, cron-based auto-publisher ## What's Next Gemma 4 isn't done. The [Model Showdown Round 2](/posts/model-showdown-round-2-gemma-kimi-and-579gb-of-stubborn-optimism) results were disappointing — Gemma stopped generating mid-response and scored a zero on the website search task. But the research since then uncovered the real problem: invisible thinking tokens eating the `num_predict` budget, meaning Gemma was silently using its output quota on reasoning before it ever started writing code. The fix is straightforward — bump `num_ctx` and `num_predict` to 32768, giving both the thinking process and the actual output room to breathe. The VRAM math works on the RTX 5090 with Q4_K_M quantization. We're going to get local models performing better and rerun the exact same website search task. Same prompt, same evaluation criteria, updated config. If Gemma can actually finish the task, the local-vs-cloud story gets a lot more interesting. ## By the Numbers - **10 fixes** shipped in one week - **5 template push attempts** before clean success - **91s → 5s** workspace boot time (18x faster) - **4.6 KB** agent skill file replacing minutes of exploration per session - **3 `apt-get update` calls** eliminated per boot - **52 MB → 0 MB** downloaded per boot - **21 screenshots** analyzed, 9 selected, 12 rejected - **1 CRLF bug** silently broken since template creation - **1 meta screenshot** — the pipeline capturing its own setup - **~2 hours** from "publish a blog post" to fully working template with skills - **~30 seconds** estimated time for the same task going forward - **1 post** scheduled to publish itself using the feature it describes === ## Thursday Thoughts: Agents Are My New Google Maps - URL: https://vibescoder.dev/posts/thursday-thoughts-agents-are-my-new-google-maps - Date: 2026-04-30 - Tags: #agents #future-of-coding #meta #building-in-public - Reading time: 5 min read How AI agents are transforming software development the same way Google Maps revolutionized travel - making the impossible feel effortless and opening up new worlds of exploration. --- I want to share something that's been on my mind lately - a realization that hit me while building this very website. I'm having an experience with AI agents that feels remarkably similar to something that happened to me about 10 years ago with Google Maps. And I think this parallel might help explain why I'm so excited about where we're headed. ## The Google Maps Revolution About a decade ago, Google Maps fundamentally changed how I approached travel. Before that moment, visiting a new city felt like a daunting challenge that required extensive preparation. You'd need to research transportation systems, map out restaurant options, figure out where the good stores were, and generally worry about getting lost or missing out on the best parts of a place. But then Google Maps gave me something I'd never had before: **fearless confidence**. Suddenly, I could show up in any city knowing that I'd have a guide in my pocket. The app wasn't just about navigation - it was integrated with search, reviews, real-time information, and everything else I needed. It made exploration seamless. I started traveling more, both for work and pleasure, because the friction had disappeared. Instead of spending hours planning itineraries and studying travel guides, I could just... go. And figure it out as I explored. ## The Agent Revolution Fast forward to today, and I'm experiencing that exact same transformation again. But this time, it's not about navigating physical cities - **it's about navigating the entire digital world**. AI agents have become my new Google Maps for software projects. I now fearlessly tackle any technical challenge because I know I have a guide that can help me figure things out as I go. Want to build a website? Set up a Linux server? Download and run a trillion-parameter language model? Each of these used to feel like major undertakings that required significant upfront research and planning. Not anymore. ## From Daunting to Delightful Just like Google Maps transformed those little navigation problems from trip-killers into part of the journey, agents have transformed technical roadblocks from project-stoppers into interesting puzzles to solve. Every step of building this blog - from the initial setup to writing posts to figuring out deployment - presents these small, fascinating challenges. But I no longer worry about whether I'll be able to get around them. The answer is always: "Yeah, I can figure this out." The agent is right there to: - Help me understand unfamiliar concepts - Guide me through installation processes - Debug issues when things go wrong - Explore different approaches to problems - Search for relevant info and integrate it It's like having an expert pair-programming partner who never gets tired, never judges your questions, and has access to the collective knowledge of the internet. ## The Bigger Picture I think we're at the beginning of something much bigger than just improved coding assistance. Agents are going to fundamentally change how people approach software, technology, and anything digital. **We're entering a new era of exploration.** If you take the Google Maps analogy to its logical conclusion, think about all the businesses that were built on that foundation. Uber, DoorDash, Instacart - these companies exist because Google Maps made location-based services effortless to build and use. I believe we're about to see the same explosion happen with agents: - **On a personal level**: People like me will build custom software solutions that would have been impossibly complex before - **On an application level**: We'll see new products that embed agentic capabilities, wrapping that power in polished UIs for mainstream users ## What This Means for Builders For those of us who want to stay ahead of the curve, this is an incredible time to start experimenting. The experience I'm having building this blog - where every challenge becomes an opportunity to learn rather than a barrier to progress - is just the beginning. The friction is disappearing from software creation the same way it disappeared from travel. And when friction disappears, innovation explodes. ## Looking Forward I'm genuinely excited about what comes next. We're not just getting better tools - we're getting a fundamentally different relationship with technology. One where curiosity and adventure matter more than extensive preparation. Where the journey of building becomes as rewarding as the destination. The age of agents isn't just changing how we code. It's changing how we explore, create, and push the boundaries of what's possible. *What's your experience been with AI agents? Are you feeling that same sense of fearless confidence, or are you still in the early stages of exploration? I'd love to hear your thoughts.* ## By the Numbers - **~10 years** — how long ago Google Maps rewired the author's relationship with travel planning, the analogy the whole post is built on - **1 trillion parameters** — the scale of local model the post name-checks as a "no big deal now" example task - **5 things** the agent is described doing on demand: explaining concepts, guiding installs, debugging, exploring approaches, and searching for context - **3 companies** named as proof that removing friction creates whole industries: Uber, DoorDash, Instacart - **2 levels** of the predicted agent expansion: personal builders shipping custom software, and products with agentic capability baked in - **1 blog** — vibescoder.dev itself, the live example behind every claim in the post === ## The Agentic Gap: Claude Oneshots, Gemma Fails - URL: https://vibescoder.dev/posts/the-agentic-gap-claude-oneshots-gemma-fails - Date: 2026-04-29 - Tags: #ai #llm #benchmark #homelab #agents - Reading time: 12 min read We pitted Gemma 4 against Opus 4.6 on a real feature build for vibescoder.dev. Gemma is the fastest model in our benchmark. It also couldn't finish the job. Here's what happened when we stopped testing toy apps and started building production code. --- Two days ago, Gemma 4 topped our [local model benchmark](/posts/model-showdown-round-2-gemma-kimi-and-579gb-of-stubborn-optimism) — 167 tokens per second, perfect code quality score, smallest download. Faster than Sonnet. Faster than Opus. The blog post said "Gemma 4 is the new default." Today we tested whether that's actually true. ## The Experiment Instead of another toy benchmark, we pulled a real item off the vibescoder.dev backlog: **public-facing search across all blog posts**. Multi-file feature, architectural decisions required, design system integration, no specification beyond "make search work." Two models. Same prompt. Same codebase. Same workspace template. One shot — no follow-up instructions, no hand-holding. Walk away and see what happens. | | Gemma 4 27B | Opus 4.6 | |---|---|---| | **Provider** | Ollama (local, RTX 5090) | Anthropic API (cloud) | | **Benchmark speed** | 167.1 tok/s | 74.3 tok/s | | **Benchmark score** | 100/100 | 100/100 | | **Cost** | $0 | Per-token pricing | The prompt was deliberately vague on implementation details: > Add public-facing search to vibescoder.dev. Users should be able to search across all published blog posts by title, content, tags, and description. The search should feel fast and match the site's existing Neon Brutalist design system. Consider: how users discover search, how results display, empty/no-result states, search state management (URL, keyboard shortcuts). Must be accessible from any page, work on mobile, and not introduce new design libraries. Commit and push when complete. Do not ask clarifying questions — make your own decisions. ## Setting up the Arena Each model got its own Coder workspace with identical starting conditions: same Docker template, same base commit on `main`, same content repo. ![Creating the search-gemma4 workspace in Coder](/images/gemma-vs-opus-search-showdown/create-workspace-gemma4.png) *Both workspaces built from the same Docker template — only the model selection differed.* We created two feature branches from the same commit (`12fd589`) and verified Vercel was configured to auto-build preview deployments for any branch push. ![Vercel Git settings showing auto-deploy enabled](/images/gemma-vs-opus-search-showdown/vercel-git-preview-settings.png) *Vercel preview deployments would give us side-by-side URLs to compare the finished features.* Both prompts were delivered at the same time. Then we stepped back. ## Opus 4.6 the Quiet Professional Opus received the prompt and went silent. No questions. No plan narrated back. Just the spinning indicator showing it was working. Over the next eight minutes, Opus: 1. Cloned both repos and installed dependencies 2. Read `package.json`, `tsconfig.json`, the app layout, existing components, `lib/posts.ts`, `lib/types.ts`, the design system in `globals.css`, and the middleware 3. Made architectural decisions: Cmd+K dialog with live API results for quick navigation, plus a full `/search` page for detailed browsing 4. Built a weighted scoring search API (`/api/search`) that ranks title matches above tag matches above content matches 5. Created a 407-line `SearchDialog` component with keyboard navigation, body scroll lock, abort controllers for in-flight requests, and ARIA accessibility 6. Built a server-rendered search results page with debounced URL state 7. Modified `Header.tsx` — three lines: import, component placement, mobile nav link 8. Updated middleware to whitelist the search API route 9. Committed everything in one clean commit and pushed One prompt. One commit. **698 lines across 6 files.** Pushed to GitHub, Vercel preview building. ``` src/app/api/search/route.ts | 104 ++++++++++ src/app/search/SearchInput.tsx | 78 ++++++++ src/app/search/page.tsx | 97 ++++++++++ src/components/Header.tsx | 10 + src/components/SearchDialog.tsx | 407 +++++++++++++++++++++++++++++++++ src/middleware.ts | 3 +- 6 files changed, 698 insertions(+), 1 deletion(-) ``` ### What Opus Built **A Cmd+K search dialog.** Press `Cmd+K` (or `/`) from any page and a full-screen overlay appears with a search input. Results appear live as you type, debounced at 200ms, with scores-based ranking. Arrow keys navigate results, Enter selects, Escape closes. The dialog shows up to 8 results with title, date, tags, reading time, and a context snippet showing where the match was found. **A full search page at `/search`.** Accessible from the mobile hamburger menu and via a "View all results" link in the dialog. Server-rendered with URL state (`/search?q=cloudflare`). Shows the full `PostCard` component for each result — consistent with the blog's existing post listing. **A scored API route.** Title matches score 100 points (150 for exact match). Tag matches score 50. Description matches score 25. Content matches score 10 plus 3 per occurrence. Results are sorted by score descending, capped at 20. The API strips markdown from content before extracting snippets. The Header diff tells the story of good integration: ```diff +import { SearchDialog } from "@/components/SearchDialog"; ... + ... + Search ``` Three lines to wire a 407-line feature into the existing layout. That's how you integrate with a codebase. ## Gemma 4 the Brilliant Planner Who Never Coded Gemma's run was a different experience entirely. **Prompt 1** — the original task. Gemma cloned the repos, checked out the branch... and stopped. Checkmark. Done. It treated the setup commands as the entire task. ![Gemma stopped after completing only the git clone setup steps](/images/gemma-vs-opus-search-showdown/gemma-stopped-after-setup.png) *Gemma completed the setup and declared victory. The search feature hadn't been mentioned yet.* **Prompt 2** — "You only completed the setup steps. Now read the codebase and implement the search feature." Gemma cloned the content repo, listed the directory, read `package.json`, wrote a thoughtful analysis plan (Architecture, Data Flow, Design System)... and stopped again. "I will begin by reading `package.json`." **Prompt 3** — "Keep going. Execute your plan — read the files, build the search feature, commit and push." Gemma read more files, outlined a detailed preliminary plan with four numbered steps... and stopped. "I will start by reading the core scripts directory." **Prompt 4** — "Do not stop between steps. Read all the files you need, write all the code, and push to git — all in one go. Do not pause or ask for confirmation." ![Gemma planning extensively but not executing code](/images/gemma-vs-opus-search-showdown/gemma-planning-not-executing.png) *The plan was actually good — client-side JSON index, search in header, `/search?q=query` routing. It just wouldn't execute it.* Gemma responded with its most detailed plan yet. A JSON search index builder, modifications to `fetch-content.sh`, a `SearchInput` component, Header integration, a search results page. Smart architectural choices. Then: "I will perform all these changes now." And stopped. **Prompt 5** — "Stop planning. Start coding. Write the files now." ![Gemma showing code in chat instead of writing to files](/images/gemma-vs-opus-search-showdown/gemma-code-in-chat-not-files.png) *After being told to code, Gemma showed code in the chat window instead of writing it to disk.* This time Gemma actually wrote some code — `build-search-index.js`, an edit to `fetch-content.sh`, and `SearchInput.tsx`. Three files to disk. Progress. Then it listed the three remaining tasks (Header, search page, commit) and stopped. **Prompts 6, 7, 8** — "Go." / "Go." / Explicit task list with three items. Gemma showed "Thinking..." briefly, then nothing. No output. No tool calls. The workspace eventually showed "unhealthy." ![Gemma's final stall — "I will perform all these changes now" then silence](/images/gemma-vs-opus-search-showdown/gemma-final-stall.png) *Eight prompts. Three partial files. Zero commits.* ### The AGENTS.md Experiment Before giving up, we tried one more thing. We added explicit agentic behavioral instructions to `AGENTS.md` in the repo — the file that Coder agents read for project-level guidance: ```markdown # Agentic Execution Rules You are an autonomous coding agent. Execute tasks end-to-end in a single turn. Never stop to describe what you will do next — just do it. ## What You Must Never Do - Output a multi-step plan and then stop. - Describe code you intend to write instead of writing it. - Leave uncommitted changes in the workspace. ``` Started a fresh Gemma session with the same prompt. Same result. Clone, read `package.json`, plan, stop. The instructions were clear. Gemma read them. And then it planned what it was going to do next and stopped. ## The Scoreboard | | Opus 4.6 | Gemma 4 27B | |---|---|---| | **Prompts needed** | 1 | 8 (incomplete) | | **Files changed** | 6 | 3 (never committed) | | **Lines written** | 698 | ~150 (partial, uncommitted) | | **Commits pushed** | 1 | 0 | | **Feature complete** | Yes | No | | **Time to completion** | ~8 minutes | Never | | **Errors self-corrected** | Yes (middleware, routing) | N/A | | **Design system match** | Yes (Neon Brutalist tokens) | N/A | | **Keyboard shortcuts** | Cmd+K, /, Escape, arrows | N/A | | **Mobile support** | Yes (hamburger menu link) | N/A | | **Accessibility** | Full ARIA | N/A | ### Technical Review Opus Implementation The code isn't perfect. A few things to fix before merging: 1. **Duplicate search logic.** The API route uses weighted scoring. The search page uses flat boolean filtering. Same query, different result order depending on which surface you use. 2. **Unsafe type cast.** `post as Post` in the search page strips content then casts back to `Post`, which expects a content field. Works at runtime but lies to TypeScript. 3. **Missing Suspense boundary.** `useSearchParams()` in `SearchInput` needs a `Suspense` wrapper for Next.js 14+. But these are code review items — the kind of things you'd catch in a PR review and fix in 10 minutes. The feature works, the architecture is sound, the UX is polished. **Score: 87.5/100** across correctness (88), architecture (82), code quality (90), performance (85), completeness (92), and integration (91). ## What We Learned **Benchmarks test generation, not agency.** Gemma 4 writes excellent code when you tell it exactly what to write. That's what our todo-app benchmark measured — single-turn code generation from a clear spec. Agentic coding is a different skill entirely: reading a codebase, making decisions, chaining dozens of tool calls, self-correcting, and maintaining a plan across many steps. Gemma can't do that yet. **The plan-and-stop pattern is a model behavior, not a configuration problem.** We tried explicit instructions ("do not stop"), behavioral directives in AGENTS.md, and increasingly urgent nudges. Gemma consistently planned what it would do, narrated the plan in detail, and then yielded control back to the user. It's not a token limit or context issue — it's how the model was trained to interact. **Speed doesn't matter if you can't finish.** Gemma generates at 167 tok/s. Opus generates at 74 tok/s. But Opus delivered a complete, working, tested feature in 8 minutes with zero human intervention. Gemma delivered nothing usable in 20+ minutes with eight human prompts. The fastest model in our benchmark is the slowest in production. **The daily driver earned its spot.** Opus 4.6 has been behind every line of code on vibescoder.dev since day one. This experiment didn't just confirm that choice — it quantified why. On a real task, the gap between "writes great code" and "builds great features" is the difference between a benchmark score and a shipping product. **Local models aren't there yet for agentic coding.** This isn't a permanent verdict. Gemma 4 was released weeks ago. Agentic capabilities are the frontier every model vendor is racing toward. But today, if you need an AI agent that can autonomously build features, cloud models with tool-calling training (Claude, GPT) are still the only game in town. ## What's Next We have a working search feature on a Vercel preview branch, courtesy of Opus. Next step is reviewing the code, fixing the three issues identified, and merging it to production. vibescoder.dev gets search. But we're not done with Gemma. The more we dug into the results, the more we think this shootout wasn't a fair fight. Our [Gemma 4 deep dive](/posts/model-showdown-round-2-gemma-kimi-and-579gb-of-stubborn-optimism) ran Gemma through Ollama with default settings — and we've since discovered that Gemma's reasoning tokens are *invisible* but still eat your generation budget. With `num_predict: 16384`, the model may have blown its entire token budget on chain-of-thought we never saw, leaving nothing for actual code output. That would explain the plan-and-stop pattern perfectly: Gemma wasn't refusing to code — it was running out of room. So we're rerunning the shootout. This time we're loading both models through llama.cpp directly, giving us fine-grained control over thinking budgets and VRAM allocation. We'll crank `num_predict` and `num_ctx` to 32K+, experiment with `--reasoning-budget` to cap invisible thinking tokens, and give Gemma the full 32 GB of RTX 5090 VRAM to work with. No more starving the local model on default settings and then calling it a fair comparison. If Gemma was choking on its own reasoning, the fix might be as simple as giving it room to breathe. If it still can't finish — even with aggressive resources and tuned inference settings — then the agentic gap is real and it's in the model weights, not the configuration. Either way, we'll have a definitive answer. ## By the Numbers - **2** models tested head-to-head on a real feature - **1** prompt — identical for both, no follow-ups allowed (for Opus) - **8** prompts needed for Gemma before it stalled permanently - **698** lines of working code from Opus - **0** lines committed by Gemma - **6** files changed by Opus (API route, search dialog, search page, input component, header, middleware) - **3** files partially written by Gemma (never committed) - **8 minutes** from prompt to pushed commit (Opus) - **20+ minutes** of attempted nudging before calling it (Gemma) - **87.5/100** technical review score for Opus implementation - **407** lines in SearchDialog.tsx alone — keyboard nav, ARIA, scroll lock, abort controllers - **3** code review items to fix before merging (duplicated logic, type cast, Suspense) - **$0** spent on Gemma inference (also $0 of value delivered) - **1** AGENTS.md rewrite attempted to fix Gemma's behavior (didn't work) - **1** clear winner === ## Model Showdown Round 2: Adding Gemma, Kimi, and 579 GB of Stubborn Optimism - URL: https://vibescoder.dev/posts/model-showdown-round-2-gemma-kimi-and-579gb-of-stubborn-optimism - Date: 2026-04-26 - Tags: #ai #llm #benchmark #homelab - Reading time: 15 min read We added Google's Gemma 4 and Moonshot's 1-trillion-parameter Kimi K2 to the local model benchmark. Five out of six models scored perfect. Gemma 4 is the new speed king. And yes, we ran a 579 GB model off an NVMe drive — at 0.6 tokens per second. --- At the end of Round 1, we promised a rematch. More models. Fixed settings. Harder questions about what "local inference" really means when you push past what fits in VRAM. This is that rematch. We added two models that the Coder dev team specifically requested: **Gemma 4** from Google (27B parameters, fits comfortably on the RTX 5090) and **Kimi K2** from Moonshot AI (1 trillion parameters, does not fit in anything reasonable). We also reran every model from Round 1 with fixes for the configuration issues that tripped up three of them. The results changed the leaderboard significantly. ## What We Fixed from Round 1 Round 1 had three avoidable failures: 1. **Qwen hit the token limit** — scored 28/100 because the output was capped at 4,096 tokens and the code got truncated mid-f-string. The model was generating at 1,510 tok/s. It wasn't slow. We just cut it off. 2. **Codestral and DeepSeek built interactive menus** — both interpreted "commands: add, list, complete, delete" as `while True: input()` loops instead of CLI argument parsers. The code worked perfectly if you used it interactively. Our automated test suite couldn't. 3. **Context windows varied** — each model had different settings, making the comparison uneven. For Round 2: | Setting | Round 1 | Round 2 | |---|---|---| | `num_predict` (max output tokens) | 4,096 | **16,384** | | `num_ctx` (context window) | Varied | **16,384 for all** | | Prompt clarity | "Commands: add, list, complete, delete" | "using argparse or sys.argv, **NOT interactive input**" | | Model management | Random loading | **Auto-unload previous, preload next** | Same prompt. Same task. Same validation. Just fair settings this time. ## Adding Gemma 4 Google released Gemma 4 while we were writing the Round 1 results. The 27B parameter model downloads as a 9.6 GB file through Ollama — the smallest of our serious contenders. ```bash ollama pull gemma4 ``` That's it. Model pulled, loaded onto the 5090 in seconds, registered in Coder's admin panel as another OpenAI-compatible model on the existing Ollama provider. The entire setup was one command and two form fields. After Round 1's configuration adventure with five different models, this felt almost anticlimactic. In the best possible way. ## Adding Kimi K2 the Hard Way Kimi K2 is a different story entirely. The numbers: 1 trillion total parameters, 32 billion active per token (Mixture of Experts architecture), 256K context window. The quantized model (Q4_K_M) is **579 GB across 13 shard files**. Our RTX 5090 has 32 GB of VRAM. We knew this going in. Round 1's post explicitly said Kimi would need API testing because it's too large for local. But this blog is about pushing boundaries with consumer hardware, and "it probably won't work" isn't a reason not to try. It's the reason *to* try. ### Step 1 Getting llama.cpp Built Ollama doesn't offer Kimi K2 for local inference — only a cloud-hosted variant. So we went to llama.cpp, the C++ inference engine that supports loading models larger than VRAM via memory-mapped NVMe offloading. Building it required installing half of Ubuntu's dev toolchain: ```bash sudo apt install -y cmake build-essential nvidia-cuda-toolkit cd ~ && git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build -DGGML_CUDA=ON cmake --build build --config Release -j$(nproc) ``` **First roadblock**: `cmake` wasn't installed. Fixed with apt. **Second roadblock**: CUDA toolkit not found. Fixed with `nvidia-cuda-toolkit`. **Third roadblock**: `nvcc fatal: Unsupported gpu architecture 'compute_120a'`. The RTX 5090 is Blackwell architecture (compute 12.0), but Ubuntu's apt CUDA toolkit is version 12.0 — too old to know about it. The fix was targeting an older compatible architecture: ```bash cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89 ``` Compute capability 89 (Ada Lovelace) runs fine on the 5090 via backward compatibility. Not ideal, but it builds. ### Step 2 Downloading 579 GB Next: the Hugging Face CLI. Which required pip. Which was externally managed. Which required `--break-system-packages`. Which installed but wasn't on PATH. Which turned out to be deprecated in favor of the `hf` CLI. Which required `python3.12-venv`. Which left behind a broken virtual environment that needed manual cleanup. ```bash sudo apt install -y python3-pip python3.12-venv pip install huggingface-hub[cli] --break-system-packages rm -rf ~/.hf-cli curl -LsSf https://hf.co/cli/install.sh | bash source ~/.bashrc ``` Then the actual download: ```bash ~/.local/bin/hf download unsloth/Kimi-K2-Instruct-GGUF --include "*Q4_K_M*" --local-dir ~/models/kimi-k2 ``` The download started reporting 384 GB, then revised upward to 432 GB, then 481 GB, then settled at **529 GB**. The HF CLI discovers shards progressively — it didn't know the full file list upfront. ![Terminal showing Kimi K2 download progress at 327 GB of 432 GB with multiple shard progress bars](/images/model-showdown-round-2/kimi-k2-download-progress.png) *Kimi K2 mid-download — 327 GB down, revising the total upward as new shards are discovered.* **3 hours and 27 minutes later**, 13 shard files totaling 579 GB sat on the NVMe. At ~370 Mbps sustained throughput. ### Step 3 the VRAM Math First attempt: 10 GPU layers. Tried to allocate 94 GB on a 32 GB card. Dead. The math: 94 GB / 10 layers ≈ 9.4 GB per layer. With 32 GB of VRAM, that's roughly 3 layers maximum. MoE architectures make each layer massive because every expert's weights live in the same layer. We settled on **2 GPU layers** (confirmed working, 3 was borderline). That means ~18 GB on the GPU, the remaining ~560 GB paging from NVMe via memory-mapped I/O. The OS's virtual memory system handles the page faults — when inference needs weights that aren't in RAM, it reads them from the NVMe on demand. ### Step 4 the Conversation Mode Bug Here's where it got interesting. llama.cpp's `llama-cli` has a `--no-conversation` flag that's supposed to run a single prompt and exit. It doesn't work. Every run dropped into an interactive `> ` prompt, waiting for input. Our benchmark script would hang indefinitely. We tried: - `--no-conversation` flag (ignored) - `--no-display-prompt` flag (still conversational) - Piping prompt via `-p` with `-e` flag (still conversational) ![Terminal showing llama-cli loading Kimi K2 with conversation flag and dropping into interactive prompt](/images/model-showdown-round-2/kimi-k2-conversation-mode-bug.png) *llama-cli ignoring --no-conversation and dropping into an interactive prompt, hanging the benchmark script.* Three benchmark attempts. Three hangs. The script captured zero timing data from Kimi because it was waiting for a conversation that would never end. ### Step 5 the Fix Llama-Server Instead of fighting the CLI, we ditched it. llama.cpp ships with `llama-server`, which exposes an **OpenAI-compatible HTTP API** — the exact same interface Ollama uses. We wrote a standalone benchmark script that: 1. Starts `llama-server` as a background process 2. Polls `/health` until the 579 GB model finishes loading 3. Sends the benchmark prompt to `/v1/chat/completions` with streaming 4. Captures every metric programmatically — TTFT, total time, tokens, tok/s 5. Runs the full validation suite 6. Shuts down the server No conversation mode. No stopwatch. No manual intervention. ```python server_cmd = [ LLAMA_SERVER, "-m", MODEL_PATH, "--n-gpu-layers", str(N_GPU_LAYERS), "--mmap", "-c", str(CTX_SIZE), "--port", str(PORT), ] server_proc = subprocess.Popen(server_cmd, ...) # Wait for 579 GB to load into memory wait_for_server(PORT, timeout=900) # Hit the same API as Ollama url = f"http://127.0.0.1:{PORT}/v1/chat/completions" ``` It worked on the first try. The model loaded in **375 seconds** (6.3 minutes), then generation began.