Benchmarking Hermes Tool Calls to Improve Speed, Freshness, and Accuracy
In my last post, I detailed who and why I connected Hermes to DeepSeek on the Mac Studio.
This post goes a bit deeper on the Hermes testing I did with my agent, Finn. Why? Because the out-of-the-box UI/UX for Hermes kinda sucks. I was prepared to write Hermes off (or any local agent running on local AI). I had concluded that at sub 100 token/s the usability just wasn’t there. But something was off. My Coder Agent performs adequately on the same inference server. So it’s a Hermes thing? Let’s find out.
I started by asking Finn to summarize a week of workouts and list my open chores. The first answer took almost 80 seconds. I asked again in the same chat and got an answer in 12 seconds. If I’d stopped at the stopwatch, I would have called that a sixfold speedup.
The trace showed what had changed. On the first run, Finn read Fitness Tracker and Honey-Do. On the repeat, he used the results already in the conversation. That might be exactly what I want when I ask him to rephrase a summary. If I ask what’s open now, I want him to look again. The faster run answered from the earlier snapshot. I had no evidence that either app had changed in the meantime.
Finn is my Hermes agent on the Mac Studio. My wife’s agent, Mia, runs on a Mac Mini and uses the same inference server. I’d already measured how batch size and prompt reuse affect DeepSeek on the Studio. Before taking a change from Finn to Mia, I needed a benchmark that could tell me whether the assistant had actually done the job.
I define the job before I start the timer
My test asked for workouts from the past seven days and the open items in Honey-Do. That gives me two reads, a date range, a filter, and a final answer I can check. The two timings that started the experiment were useful once I accounted for what Finn had read.
| Observed run | Answer recorded | What Finn read | What the timing covers |
|---|---|---|---|
| First request in a fresh chat | 79.6 s | Fresh results from both apps | Two app reads, four extra code calls, and the answer |
| Same request repeated in that chat | 12.0 s | Earlier results from the conversation | An answer using the existing snapshot |
A fresh chat can still get a warm model prompt. The first request in that 79.6-second run reused 14,042 of 14,046 opening tokens. It still spent 28.7 seconds processing new input after the apps responded. I now record two things for every run. Did the model reuse previously evaluated prompt tokens? Did the assistant go back to the source? Those can both be true on the same turn.
For a repeatable benchmark, I’d put a few known records in test accounts or serve a saved set of responses, then write down the expected answer before running the agent. To test freshness, I’d use a second version of the fixture that adds an open chore and marks another complete. I’d ask again in the same chat with the second version active. The trace should show a new read and the reply should reflect both changes. I’d also run a rephrasing request on the first version, where reusing the earlier data is appropriate. Those are two jobs with different pass conditions. The fixture change is how I’d extend the live runs described here.
A passing tool call can still produce a wrong answer
In a native Hermes Desktop run, Finn called both apps and reported 12 open chores. Honey-Do returned 11. The workout totals matched, but Finn also repeated the workout lookup. In the fresh CLI runs, the workout totals matched and 10 of 11 chore titles appeared verbatim. A title can be paraphrased, so I’d check the missing string against the actual answer before calling it an omission.
Here’s the scorecard I’d use for the two-app job. I’d save the actual tool responses alongside the answer so I can tell which check failed.
| Check | What passes |
|---|---|
| Freshness | Both apps get a new read when the request asks for current data. A second fixture version appears in the answer when it’s active. |
| Tool arguments | The workout date range matches the test’s stated clock and time zone. Honey-Do uses includeDone: false for open chores. |
| Facts | Workout totals and the chore count agree with the tool responses. |
| Coverage | Every expected open item is present, allowing for paraphrases. Completed items stay out of the list. |
| Extra work | Record tool calls, retries, and duplicate lookups. Check the reason for each extra call when assessing latency. |
I’d report timing alongside pass counts and show the failures separately. Otherwise, an answer that drops a chore can improve the average. In a test with changing live data, I’d score against the responses from that run. With fixed fixtures, I can compare configurations against the same expected answer.
I use the request trace to explain the wait
I record when the request starts, when the first useful answer text appears, and when the answer is complete. I also record the first tool result and the model’s prompt and generation timings for each request. A streaming model can emit reasoning or a tool call before any answer text appears. A tool result is another milestone. Both can be fast while the person on Telegram is still waiting.
| What to log | What it helps me check |
|---|---|
| Profile, interface, exact prompt, and new or resumed chat | Whether two runs exercised the same path |
| Cached and newly evaluated tokens on each model request | Whether the wait came from reading the opening prompt or processing later inputs |
| Tool name, arguments, start and end times, and result size | Whether the tool fetched the right data and whether the wait was in the tool or the next model request |
| First visible answer text and completed answer | How long the user waited to see an answer and how long the job took |
| Result snapshot and scorecard | Whether faster output was fresh, complete, and accurate |
That trace helped me avoid overstating one of the best-looking results. My explicit Honey-Do test asked Finn to fetch only open items and summarize directly without running code. The answer went from 79.6 to 26.9 seconds. I’d changed two things at once. When I added the open-items rule to Finn and went back to the original wording, the answer took 55.3 seconds. Processing after the tools returned fell from 28.7 to 7.4 seconds in both filtered runs. The run with the standing rule also spent 29 seconds reading its opening prompt. Watching the whole turn gave me three timings. Watching the stages told me which change was worth keeping.
I test the cache on the path I plan to use
My 156K synthetic replay showed what the Studio can do when it can reuse a long prefix. At the final batch setting, a new long prompt took 317.75 seconds to reach visible text. Its cached follow-up took 0.84 seconds. I’d use that replay to check the server, then run the actual job through the interface I plan to use.
| Test through Finn | Opening tokens reused | Time to first visible answer | What the run exercised |
|---|---|---|---|
| Second fresh CLI chat | 1,807 of 1,811 | 13.4 s | Two fresh app reads on a warm opening prompt |
| CLI chat resumed after an unrelated request | 1,778 of 5,044 | 36.3 s | Returning to a conversation with more history |
| Gateway probe after an unrelated CLI request | 5,531 of 5,557 | 0.89 s | A one-word reply through the Desktop-mode gateway path |
| Native Desktop app chat | 27 of 5,534 | Unmeasured | Two live apps, with a final answer recorded at 47.1 s |
I’d label the gateway result as a probe every time I use it. It tells me the gateway could deliver a small response quickly with that prompt already in cache. The native Desktop run is the one with the extra workout lookup and the wrong chore count. Its final-answer timing belongs to that task. I’d collect the first-visible time separately if I needed to compare how responsive Desktop and Telegram feel.
For the next comparison I’d hold the model, profile, tool definitions, and fixture steady while testing the same job in a new chat, as a follow-up, and after an unrelated request. I’d repeat each case, vary which configuration goes first, and report the median and range with the pass count. I’d group runs by the actual cached-token counts from the server. The order I ran the tasks in is a test condition. The logs tell me what got reused.
Mia’s activity changed what I wanted to benchmark
I asked Mia which tools she used most. She gave me an inventory drawn from her settings, scripts, and scheduled jobs. To decide what Finn should replay, I wanted to see when those tools actually ran.
I wrote a Python collector for Mia to run on the Mini. It opened the Hermes SQLite databases in read-only mode and counted tool calls from the assistant’s structured records, so a call and its result didn’t count twice. It packaged the settings, tool counts, and job outcomes into a ZIP and left out chat text, tool arguments, and credentials. The --prepare-review flag also writes a separate file with workflow text for Mia to review locally.
python3 ~/Downloads/collect-mia-hermes.py --days 14 --prepare-review| What the export showed | What I’d replay in Finn |
|---|---|
| Scheduled jobs accounted for 674 of 712 recorded tool calls, including every named web search and extraction call. | Run the daily digest against a fixed set of sources. Check which items it includes, whether it repeats any, and how long it takes. |
| The one Telegram session in the export accounted for 38 calls and leaned on code execution, terminal commands, and file writes. | Run a multi-step chat task that writes a file, then ask a follow-up in the same chat. One session gave me a starting point. |
I used Mia’s settings to build finn-mia-test. It uses her Chat Completions route and matches her context and output limits, compression settings, and tool timeouts. For the first replay, I gave Finn synthetic sources and asked him to save a digest locally. The job completed, but the draft repeated source IDs and included an unrelated item. I made the output requirements explicit and ran it again. The second draft passed.
My next test is to send a Telegram task Mia would recognize, first with the Studio idle and then while Finn’s digest is running. That will tell me what the person waiting for a reply sees when both jobs share one inference slot.
The first three questions I’ll ask when I see a faster run are whether the assistant read fresh data, whether the answer matches the source, and where the seconds went. I can answer all three from the same trace if I know what the job was supposed to do before I start the timer.
By the Numbers
- 79.6 seconds — First two-app answer with fresh reads
- 12 seconds — Same-chat repeat using the earlier results
- 11 open chores — Items in the tool response when the Desktop answer reported 12
- 28.7 to 7.4 seconds — Processing time after the app results when Finn fetched only open chores
- 0.89 seconds — First visible answer in the one-word gateway probe after an unrelated request