vibescoder
cd ..

Extending DeepSeek V4 Flash Led Us to Planning a Local AI Policy Layer

·10 min read
homelabaiagentsllmself-hostedbuilding-in-public

The Spark cluster solved the model problem and immediately created an architecture problem.

I spent the week getting DeepSeek V4 Flash with vision running on the dual Spark cluster, wiring it into Hermes, and benchmarking it against the local personal-assistant suite. It worked. It even won the broad benchmark.

Then the result got awkward. DeepSeek won the broad household-agent score. Granite 4.2 no-think still won the smart-home slice. DeepSeek has vision. Granite is faster for Home Assistant. The 5090 still matters. The Sparks matter. A 256GB Mac Studio is likely coming. A 128GB Strix Halo box could become another Linux node.

So the question changed. It is no longer, “Can I run a serious local model?” It is, “Who decides which local model gets the next request?”

Future-state local-agent-router architecture showing household AI clients, a policy routing layer, optional PAIR and LiteLLM infrastructure paths, local model nodes, and shared budgeted tools.
Future-state local-agent-router architecture showing household AI clients, a policy routing layer, optional PAIR and LiteLLM infrastructure paths, local model nodes, and shared budgeted tools.

The future-state homelab architecture I want to build toward: agents call one policy layer, and the router decides whether a request belongs on DeepSeek Vision, Granite, a future Mac Studio, a future Strix Halo node, or a budgeted shared tool.

The Obvious Pieces All Solved Different Problems

I started where I always should start. I looked for the wheel before building one.

NVIDIA PAIR is the first obvious piece. I already liked it enough to write about why PAIR belongs in this homelab. It discovers local nodes, manages local inference engines, and exposes OpenAI-compatible endpoints. That is exactly the kind of infrastructure a multi-machine local AI house needs.

But PAIR is an inference router, not a household policy layer. Its own README makes the boundary clear: it routes independent requests to eligible nodes. It does not pool GPU memory, shard one model across machines, or decide that a Home Assistant state query should prefer Granite while a screenshot should prefer DeepSeek Vision.

LiteLLM is another strong candidate. It is a mature AI gateway. It handles many providers, fallbacks, budgets, logs, guardrails, and even auto routing. If the problem were only “unify a bunch of model APIs,” LiteLLM would be the short answer.

But this problem is not only provider routing. It is household intent routing. LiteLLM can help with the gateway layer, but it does not know that HassGetState is authoritative live state while a prompt inventory is only routing metadata.

Home-LLM is the closest thing I found for the smart-home slice. It exists specifically to control Home Assistant with local models. That makes it very relevant. It may have prompting, parsing, and integration ideas I should steal with both hands.

It still only solves one domain.

Open WebUI is a full self-hosted AI UI with agents, model integrations, tools, MCP/OpenAPI support, RAG, and web search providers. It is excellent as a user-facing AI workbench. It is not the always-on household daemon I want Hermes to be.

Turnstone was the most interesting full-platform alternative. It is local-first, tool-using, multi-node, and includes a routing proxy, MCP support, built-in tools, and intent validation. That is close to the mental model I want.

But adopting Turnstone would be a platform choice, not a small missing piece. I am not trying to replace Hermes today. I am trying to stop Hermes from pretending one model should answer everything.

The Hermes Plugins Were Close but Not Quite It

I also looked at the Hermes ecosystem itself.

Cobalt Agent advertises model routing, tool guards, skill injection, and orchestration for Hermes. That is the closest phrase match to what I want. The catch is that it appears to patch Hermes delegation internals. That may be fine for an experimental system. It is not the abstraction I want to publish first.

Hermes Archetype Router is conceptually useful. It adds specialist delegate tools, each with its own model, persona, toolset, and skill context. That is the right shape for some work. The parent agent chooses a specialist and the specialist runs with the right model.

That still leaves the parent model making the routing decision. It turns routing into a tool choice, not a policy layer every client can use.

hermes-jev-router solves a different but real problem: saving expensive main-model calls after tool results. I like it. It is not a fleet router.

The pattern was clear. There are useful wheels. They are not the wheel I seek.

The Missing Layer Is Policy Not Another Agent Framework

The missing thing is not an agent. It is not a UI. It is not a model server. It is not even a router in the generic load-balancer sense.

The missing thing is policy.

A local AI household needs to answer questions like these:

  • Does this request include an image?
  • Does this request include Home Assistant tools?
  • Is the request about live state?
  • Is the prompt context a stale snapshot or authoritative data?
  • Is this a side-effecting action?
  • Does the request require paid search?
  • Is the fast specialist good enough?
  • Should the big model handle it instead?
  • Is a node healthy right now?
  • Is this a fallback or the first choice?

That is the layer I do not see as a clean open-source project.

So for now I am calling it local-agent-router. The name is descriptive, not final. If this turns into a real open-source project, naming is part of the open-source process. First we build, then we polish.

The Router Should Know Capabilities Not Computer Names

The architecture only scales if every machine joins as a capability node.

The current and future fleet looks something like this:

nodes:
  spark-cluster:
    memory: 256GB
    strengths:
      - vision
      - long_context
      - large_models
      - background_coding
 
  rtx-5090:
    memory: 32GB_vram
    strengths:
      - fast_tool_calls
      - home_assistant
      - embeddings
 
  mac-studio:
    memory: 256GB_unified
    strengths:
      - high_memory_local_models
      - quiet_always_on
      - future_capacity
 
  strix-halo:
    memory: 128GB_unified
    strengths:
      - low_power
      - efficient_local_models
      - overflow_capacity

The router should not know that I love the Spark box or that the 5090 has a stupid name. It should know capabilities.

A model registry might look like this:

models:
  granite-home:
    endpoint: http://ai-workstation.local:8080/v1
    model: granite-4.2-30b
    capabilities:
      text: true
      vision: false
      tool_calling: true
      max_context: 65536
    tags:
      - home_assistant
      - fast
      - local
 
  deepseek-vision:
    endpoint: http://deepseek-spark.local:8888/v1
    model: deepseek-v4-flash-vision-exp
    capabilities:
      text: true
      vision: true
      tool_calling: true
      max_context: 1048576
    tags:
      - vision
      - long_context
      - local

Then the policy becomes readable:

routes:
  - name: vision
    when:
      has_image: true
    prefer:
      - deepseek-vision
 
  - name: home_assistant
    when:
      tools_include:
        - HassGetState
        - HassTurnOn
        - HassTurnOff
    prefer:
      - granite-home
    fallback:
      - deepseek-vision
 
  - name: long_context
    when:
      input_tokens_gt: 50000
    prefer:
      - deepseek-vision

That is simple enough to explain and concrete enough to build.

Tool Authority Is the Real Lesson From the Benchmark

The DeepSeek benchmark exposed the problem that keeps bothering me.

DeepSeek did not mostly hallucinate tools. It often read fixture state and answered directly. A motion sensor appeared in the prompt with a state. The user asked if there was motion. The model answered from the prompt instead of calling the query tool.

That is understandable. It is also wrong in a live house.

The router needs tool metadata:

tools:
  HassGetState:
    authority: live_state
    side_effect: false
    prefer_model: granite-home
 
  HassTurnOn:
    authority: actuator
    side_effect: true
    prefer_model: granite-home
    require_clarification_if_ambiguous: true
 
  web_search:
    authority: external_current_info
    side_effect: false
    cost: paid
    budget_group: search

The general rule is portable:

Prompt context is routing metadata.
Tools are authoritative for live state.

That applies to Home Assistant, calendars, portfolios, search, email, files, and anything else with a live system behind it.

Search Needs Budget Policy Too

This came up because Hermes can use Brave and Firecrawl. Those tools are useful. They also cost money.

Giving every agent direct access to paid search is the local AI version of leaving the garage door open because the model might need a rake.

A shared search policy could start simple:

budgets:
  search:
    daily_usd: 2.00
    order:
      - cache
      - searxng
      - brave
      - static_fetch
      - firecrawl

Start with cache. Try SearXNG or local search. Use the Brave Search API when freshness matters. Use Firecrawl when extraction needs a real crawler. Log the choice.

That is not a model problem. It is a household budget problem.

The MVP Should Be Boring

The first version of local-agent-router should be boring on purpose.

It should expose:

POST /v1/chat/completions
GET /v1/models
GET /health
GET /routes

It should load YAML. It should health-check backends. It should route on simple facts: images, tool names, context size, default. It should forward requests to OpenAI-compatible endpoints and log why it made the choice.

A log entry should look like this:

{
  "route": "home_assistant",
  "selected_model": "granite-home",
  "reason": "tools_include HassGetState",
  "fallback_used": false
}

That is enough to learn.

Notably, I do not want to start by building a giant LangGraph workflow. LangGraph is powerful. That is exactly why I want to avoid it at the start. The first version should be a policy router, not a general reasoning engine.

This Might Be a Real Open-Source Niche

I am not claiming nobody has built parts of this. The research says the opposite. The ecosystem is full of parts.

But I do not see a small, local-first, homelab-friendly policy router that assumes a household might have:

  • a GPU workstation
  • a Spark or Mac Studio class high-memory node
  • a low-power secondary box
  • Home Assistant
  • local models
  • a few paid tools
  • a preference for keeping private data local
  • more than one agent framework over time

That is a real shape. It is only going to get more common.

The community helped us get the Spark side running. The right way to pay that back might be to build the missing layer we needed once it worked.

The Plan From Here

I wrote a separate planning doc in this repo so future Coder Agents sessions can pick this up without re-reading this whole arc:

docs/experiments/local-agent-router-plan.md

The build order is straightforward:

  1. Write the spec and non-goals.
  2. Build a tiny OpenAI-compatible router.
  3. Add model and node registries.
  4. Add rule-based routing.
  5. Add tool authority metadata.
  6. Add route logs.
  7. Add paid search budget policy.
  8. Wire Hermes through it.
  9. Test with Granite on the 5090 and DeepSeek Vision on the Spark cluster.
  10. Package the result so another homelab can run it.

If this works, the next blog post will not be about the idea. It will be about the first ugly build.

That is the right order anyway.

First we build. Then we polish.

By the Numbers

  • 9 projects checked before deciding the missing piece was policy, not another model server
  • 4 current or future homelab nodes in the target architecture: Spark, RTX 5090, Mac Studio, and Strix Halo
  • 2 routing layers separated by design: infrastructure routing and household policy routing
  • 1 working project name for now: local-agent-router
  • 4 initial API surfaces in the MVP: chat completions, models, health, and routes
  • 5 search tiers in the proposed budget chain: cache, SearXNG, Brave, static fetch, Firecrawl
  • 10 build steps in the first implementation plan
  • 0 plans to replace Hermes, PAIR, or LiteLLM on day one

Comments