Tuning the Hermes Context Window: How I Burned 72.7M Tokens So You Don’t Have To

Local models feel like they get dumber the longer a session runs. I’ve felt it for months. A conversation starts sharp. An hour in, Qwen 3.6 starts missing things it would have caught cold. My instinct, every time, is to bail to a cloud model with a bigger context window and call it a day.

That instinct bothered me. So I asked a narrower question. Could I tune the context window on my own homelab rig and close that gap without leaving local at all? I built an experiment to find out.

The short answer: no. Context window size, the number you configure at the server, does not meaningfully change output quality. That question turned out to be the wrong one. But the experiment answered a better question I hadn’t asked yet, and it changed a real setting in Hermes Agent, the assistant my wife runs day to day.

One Model, Three Ceilings, Nine Cells

I fixed the model on purpose. Every prior bakeoff on this blog varies the model. This one holds Qwen 3.6 35B-A3B constant and varies only the context configuration. That isolates the one variable I actually care about.

Three llama-swap entries, same weights, three different --ctx-size values:

Each ceiling gets tested at three fill levels. Fill level means how much of that window is actually occupied before the real question gets asked. 10%, 50%, and 90%. Three repeats per cell, to smooth out sampling noise. Two domains, reused from the personal-assistant bakeoff harness: a calendar assistant with five tools, and a Home Assistant device-control assistant with thirty-two.

Three ceilings times three fill levels times three repeats times two domains comes to 54 runs.

A Bigger Ceiling Does Nothing Until You Fill It

Here’s the part that almost broke the experiment before it started. Raising --ctx-size does not, by itself, stress a bigger window. I measured this directly. Even the richest test data in the harness runs a few thousand tokens deep. That’s nowhere near 131,072, let alone 262,144. A model never comes close to using a window that large on short prompts. So testing three ceilings with the same short prompts would have shown nothing at all.

The fix: pad every sample with synthetic conversation history before the real question. Generic smart-home and calendar chatter, already resolved, cycled from a small fixed pool. Not random text. Random text tests whether garbage confuses a model, which is a different and less useful question. This tests whether plausible history crowds out a model’s ability to handle the newest turn, which is the actual failure mode researchers have documented in papers like Lost in the Middle.

Getting the token math right for that filler took three real bugs to shake out.

Three Bugs Almost Wrecked the Data

The first bug came from Inspect AI’s /apply-template call not knowing about the domain’s registered tools. HA registers 32. Calendar registers 5. That gap alone never showed up on calendar. It showed up immediately on HA.

The second bug is the one I’m proudest of catching, because it looked like nothing was wrong. GPU utilization sat at 0%. Power draw sat near idle. The process just... didn’t finish. It was making hundreds of small HTTP round trips, each one re-rendering an entire 600KB prompt to check if a single 150-token turn could be trimmed. Fixing it meant cutting a proportional slice off the filler in one shot instead of one turn at a time. Same cell afterward: 45 seconds.

The third only showed up at the largest context window, because it needed roughly ten thousand filler turns to reach 90% of 262,144 tokens. The safety valve meant to catch a broken tokenizer was catching a legitimate need instead.

All three are fixed and merged into the forked harness. All 54 cells, plus the 11 that the three bugs invalidated, finished clean on the second pass.

54 Runs Found Almost No Accuracy Difference

Here’s the combined scorecard, both domains, weighted by sample count:

Two things jump out. First, there’s no clean relationship with ceiling size. The smallest window scored the highest average. The middle window scored the lowest. That’s not what “bigger context window helps” would predict.

Second, there is a real, if modest, decline as fill increases. Averaged across all three ceilings: 0.570 at 10% fill, 0.552 at 50%, 0.538 at 90%. A drop of about 5.6% relative. Directionally consistent. Small enough that it sits close to the noise floor of individual repeats, some of which swung by 19 points on their own.

Split by domain, the pattern holds but the absolute numbers differ a lot:

Home Assistant’s low absolute score comes with a caveat. I limited it to 16 samples to keep run time comparable to calendar, and that 16-sample slice may not represent the full 80-case tier fairly. The relative comparison across fill levels still holds, since every cell used the identical 16 cases. The absolute number just shouldn’t be read as “Qwen is bad at Home Assistant.”

Decode Speed Fell By Half, Cleanly

If accuracy barely moved, speed did the opposite. I measured this separately from the scored runs, directly against llama.cpp’s native completion endpoint, with prompt caching disabled so every number reflects real, uncached work.

Decode speed at ~235,000 tokens of real depth is less than half what it is near empty. 102 tokens per second versus 227. Prefill drops too, from 8,250 tokens per second down to 4,417.

Look closer and a pattern appears that matters more than any single row. qwen-32k at 50% fill, about 15,000 real tokens, decodes at 216 tok/s. qwen-131k at 10% fill, about 12,000 real tokens, decodes at 218 tok/s. Nearly the same depth. Nearly the same speed. Two servers, two completely different configured ceilings.

A Prompt Cache Hid 93% of the Real Token Count

Every sample inside one test cell shares the same synthetic filler. Sixteen samples, one shared prefix, one resident llama-server process. llama.cpp’s prompt cache noticed. After the first sample in a cell, the shared filler served straight from cache for every remaining sample. Inspect’s own per-request accounting doesn’t know or care about that. It reports the full context size for every single request, cached tokens included.

Summed across all 864 scored samples, that raw per-request accounting comes to 72,661,613 tokens. That’s the number I’d expect on a receipt if this ran through a metered API with no cache credit. It’s also the honest, conventional answer to “how many tokens did this use.”

Only 5,395,251 of those tokens were freshly computed. The rest, 93%, came straight from cache.

Neither number is the full story by itself. 72.7 million treats a cached prefix as if the GPU recomputed it sixteen times over, which it didn’t. But 5.4 million has its own blind spot. A cache hit still has to be read. Every decode step, in every one of those samples, still attended over the full cached context, which is exactly why decode speed dropped with real depth in the section above. Caching cuts prefill cost. It does not cut decode cost.

The Ceiling Was Never the Answer

The match between qwen-32k at depth and qwen-131k at depth, two sections back, is the actual finding of this whole experiment. Speed tracks real depth. Not the number typed into --ctx-size. A big ceiling costs nothing in speed until a session actually fills it. It does cost real, fixed VRAM the moment the server starts: about 22.5GB empty at 32,768, climbing to about 25.6GB empty at 262,144, on the same RTX 5090.

So the original question, what’s the optimal context window for quality, doesn’t really have an answer. Quality barely moves either way. The ceiling you pick is a headroom decision, not a quality decision. Set it high enough that a real session never hard-fails against it, and stop there.

I’m keeping the production qwen entry at 131,072. It’s already meaningfully deeper than 32,768 for a real multi-turn session, and it skips 262,144’s extra VRAM tax for headroom that mostly goes unused.

So Why Did It Feel Like the Model Was Getting Dumber

This whole experiment started from a feeling, not a number. Long local sessions feel like they degrade. Cloud models with huge context windows feel like the fix.

The data above doesn’t back that feeling at the strength it deserves. A 5.6% relative accuracy drop, worst fill level against best, is real but mild. It doesn’t obviously explain wanting to abandon a model mid-session.

Two honest explanations. I don’t know yet which one is closer to true.

The first: something this experiment never tested is the real cause. Generic synthetic filler is not the same as a long agentic session with real tool calls stacking on top of each other. Small mistakes can compound differently across a chain of real tool calls than across one scored question in isolation. A hard context-overflow error, the exact failure mode three bugs in this experiment produced by accident, can also get misread in the moment as “the model got dumber” instead of “the request failed outright.”

The second: the felt degradation is real, but smaller than it feels while you’re in it. A frustrating exchange at minute forty can color the memory of an entire session.

I’m not settling that question here. Both are worth their own experiment. Compounding tool-call error across a real multi-step chain is a different test than raw context volume tolerance. So is a needle planted early in a session and checked at the end, the same idea I raise below as the better version of this experiment. Down the road, one of those becomes the next bakeoff.

What This Changes in Hermes’s Compaction Config

This is the part that actually changes something I run every day.

Hermes Agent already has automatic context compaction. Full credit where it’s due: this isn’t a gap I needed to fill with a new skill. It’s a mature, built-in system that summarizes older turns once the session crosses a configured percentage of the model’s window, protects the first and last N messages from being touched, and even ships a manual /compress command for on-demand use.

My actual config, before this experiment:

At a 131,072 ceiling, an 0.85 threshold means compaction doesn’t fire until roughly 111,000 tokens deep. Cross-reference that against the speed table above. At that depth, decode speed already sits around 141 tok/s, down 35% from a fresh session. Real sessions spend real time in that degraded zone before Hermes ever steps in.

There’s also a status message the code already emits for this, a literal “Compacting context, summarizing earlier conversation” line, but it’s silent on chat platforms by default. I’d never seen it. progress_notices was off.

Two changes, both small, both directly justified by this data:

• Add a per-model threshold override so Qwen compacts earlier, closer to 0.5 or 0.6, instead of riding the global 0.85 default all the way down.
• Turn progress_notices on, so compaction stops happening invisibly.

Neither change came from guessing. Both came from a number on a table above.

What I’d Test Differently Next Time

The ceiling turned out to be a confound, not a real variable. A cleaner version of this experiment fixes one large ceiling and varies only real depth, at finer steps, without three separate model configs to manage. It would also use less repetitive filler. My synthetic pool cycled twenty exchanges, which a model can plausibly learn to skim as boilerplate. A sharper test plants one fact early in a long session and checks whether the model still remembers it by the end, closer to what an actual long-running assistant session looks like, and closer to testing the two hypotheses above directly instead of guessing between them.

That’s a good excuse for a round two. This round already paid for itself in one config change.

If your local daily driver feels sharper at the start of a session than the end, check your compaction threshold before you blame the model.

By the Numbers

• 54 scored runs, plus 11 re-runs after three bugs invalidated their cells
• 72,661,613 tokens processed across every request, the conventional total
• 93% of that total served from llama.cpp’s own prompt cache, not freshly computed
• 5,395,251 tokens actually fresh, prefill and output combined
• 3 context ceilings tested: 32,768 / 131,072 / 262,144
• 6+ hours one cell ran before I killed it, versus 45 seconds after the fix
• 5.6% relative accuracy drop, worst fill level versus best, averaged across all three ceilings
• 2.2x the decode speed at near-empty context versus 90% full at the largest window (227 vs. 102 tok/s)
• 111,000 tokens deep before Hermes’s current 0.85 threshold triggers compaction, at the production ceiling
• 0 cloud tokens spent finding any of this out