vibescoder

Why I’m Keeping Local Agent Router and Adding Static Routes

·8 min read

Hola Olla. Are you a better gateway for my homelab?

That’s the question I’ll answer in this blog. For context, I maintain Local Agent Router. But as a vibe coder and not an actual software engineer I have to ask myself if the burden of maintaining custom code is worth it. That’s why Olla landed on my radar at a good time.

Here was an open-source proxy that already handled model discovery, health checks, and failover across local inference servers. Those are features I maintain. I wanted to know whether I could deploy somebody else’s work and get that time back.

I want to ask that question regularly. A tool I build can solve a real problem on day one and become the wrong thing to maintain a few months later. Sometimes a community project catches up. Sometimes my own requirements change. This time, both had happened. I’m keeping Local Agent Router, and the next thing I want to build for it is a static routing option.

I Gave the Router a Harder Job When the Models Overlapped

I built Local Agent Router when several models in the lab could plausibly handle the same request. I wanted an agent to send its usual request and let the router make the choice. The YAML policy could match an image, a particular tool name, or a prompt above a token threshold. A writing workflow could signal its intent by including write_blog_draft. Health and capability checks narrowed the candidates before the router forwarded the request. As I found in Part 4 of the router series, the model name a client sent could be cosmetic because the policy did the selecting.

The lab now has more specialized jobs. The Mac Studio serves Hermes and general agent work. The DGX Spark cluster has the coding role, and the Strix Halo has the writing role. The RTX 5090 workstation is for video work and comes online as needed. In most cases the client already knows which job it wants to do.

That changed the review. I’d started by asking which gateway could take over all the rules I’d written. Once I put the machines and their jobs on one page, I could ask a smaller question. What do I need between a client that already knows its job and a server whose availability or model can change?

Olla Made Replacing the Router a Real Option

Olla was the closest fit I found for the day-to-day job. Its local backend support covers the serving engines I care about, and it offers priority routing, health checks, automatic failover, and model discovery. If I were starting this version of the lab from scratch, I’d test it early. My choice was whether deploying it now would take enough work off my plate to justify moving the clients and verifying the behavior I already rely on.

OptionWhat I Could Use It ForWhat the Decision Came Down To
OllaPut several local inference servers behind one proxy with discovery and health-based failover.The closest replacement for the simpler setup. A cutover would mean rechecking authentication, streaming, and fallback across my clients.
NVIDIA PAIRConnect compatible machines on the local network and distribute inference requests.Discovery and pairing are interesting. I already know where my main workloads live, so that would solve a smaller part of my problem today.
llama-swapSwap models on an individual inference host.It still has a job on the workstation. I also want a client endpoint that can survive changes across hosts.
LiteLLMUse a common gateway with load balancing and fallbacks across providers.I can see it earning its place if the provider mix grows. The routes I need today are mostly a short list of local destinations.
PortkeyApply routing and reliability controls through a common gateway API.Its broader provider and guardrail features are more than I need to assign the lab’s fixed jobs.

The review put Local Agent Router at roughly 5,300 lines of Python. That’s enough code to make the maintenance question real. It also does work I already depend on. The router serves two client API formats, streams responses, checks that backends are healthy and compatible, and applies deadlines and admission limits. A migration could be worth it if somebody else’s gateway took over a part of the system I was tired of maintaining. With the requirements I have now, I’d spend most of the migration verifying behavior I already have, just to land on a short routing table.

A Static Route Gives the Client a Name It Can Keep

I want the client to be able to ask for writing and keep using that name when I change the model behind it. The same goes for coding, general, and video. Those names describe jobs. The router configuration can map each one to a preferred backend and decide whether there’s a suitable fallback.

Proposed AliasPreferred DestinationWhen the Preferred Backend Is Unavailable
generalMac StudioReport the service as unavailable if there’s no compatible backend.
codingDGX Spark clusterUse the Studio when it can serve the request and signal the fallback.
writingStrix HaloUse the Studio when it can serve the request and signal the fallback.
videoRTX 5090 workstationCheck what the request requires. Use the Studio only if it can do the job. Otherwise return a clear failure.

In policy mode today, the router can ignore the request’s model field. With the static option, a recognized alias would give the client an explicit routing choice. I’ll keep the policy engine for workflows that benefit from inspecting the request. Most clients can use the simpler path, and I can pause work on new policy predicates and automatic fleet discovery. The table is the design I settled on for the next implementation.

The model swap is the part I care about most. A writing client should be able to keep asking for writing after I replace the model on the Halo. On the router side, I’d update the backend’s served model ID and its declared capabilities. The router’s inventory check looks for that exact ID in the backend’s model list. I’ve already hit the version of this problem where an alias worked when called directly but was missing from the list the router checked. A stable client alias keeps the client configuration intact. A passing inventory check tells me the router can reach the new model.

I Want a Model Swap to Be Quiet and a Fallback to Be Visible

If I upgrade the preferred writing model, the writing client can keep its settings. If the Halo goes offline and the Studio takes over, I want to know about it. The response can look normal even when the job ran on a different model.

The router already identifies a fallback in its response headers. A request served by a fallback carries this header.

x-local-agent-fallback-used: true

Two other headers identify the route and selected model alias. The /routes endpoint can show that a route has lost its preferred backend. That answers a health question. The response header answers what happened to one request. The remaining work is deciding where to show that signal. A header can be present while the person using the client sees only the answer.

My first checks for static routing are straightforward. Send the same client request before and after a backend model swap. Take the preferred backend offline and confirm a compatible Studio fallback is visible. Then ask for a capability the fallback lacks and check that the client gets a clear failure. That last case matters for the workstation. If I ask for a video while it’s off, a text reply from the Studio could leave me thinking the job ran.

I’ll revisit the build-versus-deploy decision when the lab changes again. If more hosts start competing for the same requests, the policy engine may see more use. If maintaining discovery and failover becomes the main job, I’ll take another look at Olla.

By the Numbers

  • About 5,300 lines of Python — The review’s estimate of the Local Agent Router codebase
  • 5 community alternatives — Olla, NVIDIA PAIR, llama-swap, LiteLLM, and Portkey in the side-by-side review
  • 4 proposed aliases — General, coding, writing, and video
  • 3 existing response headers — Route, selected model alias, and whether the request used a fallback
  • 2 client API formats — OpenAI Chat Completions and Anthropic Messages

Comments