ComfyUI, Lemonade, and LocalAI: Scouting the Next Wave of Homelab AI Tools

It's a gloomy, rainy day on Cape Cod. Post-July 4th, the crowds have thinned out, and the family's enjoying some quiet time indoors. Perfect weather for the kind of homelab research that doesn't require standing next to a water-cooling loop with a multimeter: just a laptop, a browser, and a running list of tools that have been coming up in the agentic AI world without me ever pinning down what they actually do or whether they belong on AI-NT-No-Problem.

So that's what today was. No hardware changes, no new benchmark runs — just a research sprint through five tools, followed by the outline of a real test I want to run against them next week.

The Tools

llama-benchy

llama-benchy is a benchmarking tool that brings llama-bench-style measurements — the pp/tg-at-different-context-depths numbers everyone in the llama.cpp world already trusts — to any OpenAI-compatible endpoint, not just llama.cpp. That matters because llama-bench only works with llama.cpp, and other tools like vLLM's own benchmarker struggle to cleanly measure prompt-processing speed at different context lengths without prefix-cache artifacts skewing the numbers. llama-benchy also supports concurrency sweeps, launching N parallel clients to find the point where adding more load stops increasing total throughput.

For the homelab: a clean win, no debate needed. We already run llama-server directly via systemd, and our existing benchmark tooling is either ad-hoc Python scripts or a bespoke harness built for cloud APIs — neither gives us pp/tg-at-depth numbers against the actual endpoint serving OpenClaw and the Discord bot in production. llama-benchy drops in against our existing http://localhost:8080/v1 with zero infra changes.

Lemonade Server

One of my devs flagged this one — she knows the homelab runs AMD silicon on the CPU side and figured Lemonade, AMD's own local AI server, would be a natural fit. Fair assumption on paper: Lemonade is a unified OpenAI/Anthropic/Ollama-compatible endpoint that orchestrates llama.cpp, FastFlowLM (NPU), whisper.cpp, stable-diffusion.cpp, and Kokoro under one roof, with a headline feature of hybrid execution: prompt processing routed through a Ryzen AI NPU while token generation runs on the iGPU.

Digging in, though, "AMD" was doing a lot of hiding in that sentence. Lemonade's real value proposition is Ryzen AI 300/400-series Strix Halo silicon specifically — the XDNA2 NPU is the entire point. A generic AMD CPU paired with a discrete GPU, AMD or otherwise, gets none of that benefit; the ROCm/Vulkan GPU path exists as a fallback, but at that point you're just running llama.cpp with extra abstraction between you and the flags that matter.

For the homelab: not a fit, and it's not close. AI-NT-No-Problem has an AMD Ryzen 9 9950X3D on the CPU — but that's a desktop part, not the Ryzen AI-branded mobile/APU silicon Lemonade is built around, and the GPU is an NVIDIA RTX 5090 on CUDA 13.1. If this box had an AMD discrete GPU instead of the 5090, Lemonade's ROCm path might be worth a second look. Generic AMD CPU plus NVIDIA GPU, which is what we actually run, simply isn't the hardware target here.

ComfyUI

ComfyUI is a node-based, graph-driven GUI for running Stable Diffusion and other diffusion models — instead of a single "Generate" button, every step (load checkpoint, encode prompt, sample, decode) is its own node you wire together into a reusable, shareable workflow. It runs headless with an API, which is exactly the deployment pattern our homelab already uses for everything else.

For the homelab: unlike Lemonade, this one's a genuine fit — ComfyUI natively supports NVIDIA/CUDA, no AMD-specific caveats to work around. It'd slot in as another systemd service alongside llama-generate/llama-embed, exposed through the same Tailscale/Cloudflare tunnel we already built. It's also a nice complement to the "local models struggle at multi-step agentic work" conclusion from the Model Showdown series — image generation is single-shot, not multi-step tool orchestration, so it sidesteps the exact failure mode that's been the headline finding of Rounds 1–7.

AMD AI Playbooks

AMD publishes a public GitHub repo of step-by-step guides for building AI workloads on AMD hardware — Lemonade, vLLM, LM Studio, ComfyUI, fine-tuning with LLaMA Factory/Unsloth, even clustering two Ryzen AI Halo boxes together for 350B+ models via llama.cpp RPC. Mechanically, each playbook is just a folder: playbook.json for metadata, README.md for content, platform.md for platform-specific setup, with inline tags like <!-- @os:windows --> to show conditional content. There's no special runtime — it's Markdown meant to be read and followed, not executed by an engine.

That last point turned out to be the more interesting answer to a question I'd been sitting on: can an agent just consume these directly? Yes — since it's a public repo of plain Markdown and JSON, any coding agent can clone it and treat a README.md as a task brief, executing each step itself, the same pattern we used to build the thermal-migration test harness. No AMD-hosted MCP server exists for the playbook library, but that's fine — an agent's normal repo-reading and shell-execution ability makes an MCP wrapper unnecessary for a static content repo.

For the homelab: skip the AMD-specific playbooks wholesale (no ROCm, no NPU here), but the vLLM, fine-tuning, and RPC-clustering ones are worth mining for technique even on CUDA hardware — particularly the RPC-clustering approach, given Kimi K2 needed NVMe offload at 0.6 tok/s to even fit here.

LocalAI (and the rest of the Lemonade-alternative field)

Since Lemonade turned out to be Strix-Halo-locked, the natural follow-up was: what's the vendor-neutral equivalent? LocalAI is the clearest match — a composable AI engine that runs LLMs, image, voice, and video models on any hardware (NVIDIA, AMD, Intel, Apple Silicon, or CPU-only), behind a single OpenAI/Anthropic/ElevenLabs-compatible API, with MCP support and a built-in agent orchestration layer (LocalAGI) added as of late 2025. Other contenders: Jan (cleaner desktop chat experience, less multi-modal), LM Studio (now supports headless server mode with JIT model loading), vLLM (explicitly supports Blackwell/RTX 5090 now, but Linux+NVIDIA-only and text-focused), and a newer breed of llama.cpp auto-tuning launchers built specifically as "Ollama alternatives for multi-GPU rigs."

For the homelab: LocalAI is the one worth actually testing — it's the closest philosophical match to Lemonade's "one unified endpoint, auto backend selection, multi-modal" pitch, but with native CUDA support instead of an AMD-only ceiling. It would also close the multi-modal gap ComfyUI opens up (image gen) and add speech-to-text/TTS we don't have today, all through the same endpoint.

The Bakeoff (Plan, Not Results — Yet)

Here's the actual question worth answering with data, not vibes: is LocalAI a better daily-driver than the llama-server + llm-switch.sh stack we've hand-tuned over the last several months?

The methodology borrows directly from the Three-Pass Pattern and the Model Showdown series, just pointed at infrastructure instead of models:

The hypothesis: LocalAI can match our hand-tuned setup on Layers 1–3 and win outright on Layer 5, but Layer 4 is where I expect the real signal — every silent failure documented on this blog was found through actual dogfooding, not a benchmark run.

I'm not scaffolding the actual suite YAML or harness code in this post — the model and task list aren't locked yet, and honestly, this post is already covering five tools and a test plan. Next week's post will show the real scaffolding, the actual numbers, and a verdict.

By the Numbers

• 5 tools researched: llama-benchy, Lemonade Server, ComfyUI, AMD AI Playbooks, LocalAI
• 1 ruled out immediately on hardware grounds (Lemonade — Strix Halo NPU only, we're RTX 5090)
• 1 dev tip that sent us down the Lemonade rabbit hole in the first place
• 32 GB — the RTX 5090 VRAM ceiling every one of these tools ultimately has to respect here
• 5 layers in the planned LocalAI bakeoff, one of them a straight 1–2 week soak test
• 0 hardware changes made today
• 0 lines of bakeoff YAML shown in this post — on purpose

Rainy days are underrated for this kind of work. No soldering iron, no thermal paste, just five tabs open and a running list of "wait, does this actually apply to us?"

The bakeoff comes next.