How I Got DeepSeek V4.1 Flash Running on a 256GB Mac Studio
The Mac Studio finally arrived. That gave me another 256 GB of unified memory in the homelab, this time in one Apple Silicon box alongside my two-node DGX Spark cluster. I had compared the machines on paper. Now I could start giving them different jobs.
The Sparks are where I want to run coding models. I can try one model spanning both nodes or a model per node, depending on the task. A million tokens or more is the target for that side of the lab. The Studio has a different job. I want it to run the most well-rounded model I can find for chat, tool calls, and the everyday work my Hermes agents do. I picked 256K as the starting window. That sounds generous until you see a household Telegram chat reach 119K tokens.
DeepSeek V4 Flash had treated me well on the Sparks, so it was the natural place to start. By the time I was setting up the Studio, V4.1 Flash had a quantized build that could run in this memory tier and still handle images. The first Studio deployment used MLX. This is the story of replacing that backend with the full quantized GGUF on llama.cpp and Metal, then tuning it for the two people who would use it.
Finn is my Hermes agent on the Studio. He helps with personal tasks and household chores. My wife’s agent, Mia, does the same for her from a Mac Mini. Both can use the Studio for inference. I have a CLI and a desktop app, but Telegram is where most of the conversations happen. Setting up Finn deserves its own post. Here I wanted to see what it takes to make the model behind both agents usable.
The sixth shard stayed on the SSD
I already had DeepSeek V4.1 Flash 1017 installed on the Studio. My first setup used an iQ-MLX 3.0 bpw checkpoint with mlx-serve. It had passed 128K retrieval tests and was configured for 262,144 tokens. That was the rollback option, and I kept its files on disk. The new target was ISTA-DASLab’s GSQ-RCO IQ2_XXS, the general-purpose, unpruned quant. I wanted vision and the full model for the mix of work Finn and Mia get. I left the pruned Coder variant out of this deployment.
| What I had on the Studio | MLX deployment | New GGUF deployment |
|---|---|---|
| Model | V4.1 Flash 1017, iQ-MLX 3.0 bpw v1.1 | V4.1 Flash 1017, GSQ-RCO IQ2_XXS, full unpruned |
| Serving runtime | mlx-serve 26.10.3 | llama.cpp with Metal, first b12314 and later b12359 |
| Model size on SSD | 427.30 GB | 248.84 GB, plus 0.97 GB for the projector |
| Engram | Original-precision tables on SSD | Shared IQ4_NL tables, mapped from SSD with lazy reads |
The smaller download made this deployment easier to stage alongside MLX. In an earlier MLX run, a synthetic 131K-token retrieval reached visible text in 132.9 seconds and decoded at about 80 tokens per second. That setup used DSpark with thinking off. I kept it as a reference because the runtime, quantization, and settings differed. The new build had five backbone shards totaling 138.24 GB and an Engram file of 110.60 GB. The Engram is a set of lookup tables. llama.cpp can ask the SSD for the rows it needs, while macOS can cache pages it has read. The working set depends on the request and what else the Mac has been doing.
I pinned the Hugging Face revision, checked the byte count and SHA-256 of each file, and only then let the launcher proceed. From the model directory I made the Engram look like the sixth shard. The names matter because the loader finds the other shards from the first filename. Publisher’s llama.cpp instructions
ln -s DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf \
DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00006-of-00006.ggufI set --load-mode mmap --lazy-mode auto and asked for full Metal offload. The startup log showed 41 of 41 layers on the GPU and lazy reads from the Engram. I tracked process RSS, page-ins, disk reads, and system swap during the tests. A page-in or a disk read can come from several model mappings. The loader log and virtual memory map were how I checked that the Engram itself was mapped. RSS is the server’s resident memory at the time of sampling. It doesn’t capture all the unified memory used by the machine.
I worked up to 256K before enabling vision
I started with one request slot, 16K context, and the embedded chat template. I tried ordinary questions, a reasoning response, a structured tool call, and retrieval from a long prompt. I increased the context only after the current stage passed. Codex couldn’t see the Metal device from its own session, so I launched the test supervisor from my Mac Terminal and let it run in the background.
| Configured context | Initial validation | Time to first token |
|---|---|---|
| 16,384 | Retrieval from a roughly 15K prompt | 28.43 s |
| 32,768 | Near-limit retrieval | 61.11 s |
| 65,536 | Near-limit retrieval | 131.36 s |
| 131,072 | Near-limit retrieval | 291.52 s |
| 262,144 | Retrieval from a 146,424-token synthetic Hermes history with a tool call and result | 339.46 s |
| 262,144 plus vision | Separate image probe with the BF16 projector | 1.91 s |
The last two rows use different requests. The 256K text test used a 146K input, and the image probe tested vision separately. These runs also found a bug in my checker. The first 256K answer contained the facts I asked for, but the checker tripped over the leading thinking block. I fixed the check and reran the stage.
On the 146K text test, the server generated at about 42 tokens per second and the sampled RSS peaked at 160.1 GiB, with no new system swap-outs. The next request reused the long prefix and produced its first token in 0.37 seconds. The 262,144-token window gives the prompt, history, tool results, reasoning, and answer one shared budget. I kept one request slot so the tested configuration stayed predictable. If both agents ask it something at once, one request can wait behind the other.
Mia found the missing LAN listener
Mia’s first Telegram test was a request to look at her honey-do list and summarize where things stood. Telegram reported that the provider was temporarily unavailable and would retry after 15 seconds. I had checked the new server from the Studio itself. Mia lived on the Mini, and my first cutover had restored the old port on loopback only.
I restored the Studio’s LAN listener on port 11234, kept the existing API key, and added the model alias the old client already sent. I tested Chat Completions and the Anthropic Messages compatibility route. Mia uses Chat Completions in her current configuration. The next Telegram try reached the Studio and answered with her existing Hermes settings.
| What the Studio logged for that Telegram turn | Time |
|---|---|
| Process the first 54,508 uncached input tokens | 122 s |
| Process new input across the six follow-up requests | 11 s total |
| Generate tokens across all seven model requests | 165 s total |
| First model request to last completion | 5 min 2 s |
Those numbers come from the server. I didn’t time when text first appeared in Telegram. The second request processed just 638 new tokens in about two seconds. The follow-up requests benefited from the cached conversation, but the model still had to emit tool calls and generate answers. That gave me two different delays to work on. A larger prompt batch could shorten the first pass over a long history, and a smaller request or one fewer tool round trip could shorten the whole turn.
A bigger prompt batch saved 46 seconds
I replayed a synthetic Hermes-style archive of roughly 156K tokens. It included earlier tool history and questions with exact expected answers. I measured the first emitted token and the first visible answer separately. A stream can contain reasoning before there’s any answer text to show. I also asked a follow-up, sent an unrelated chat, and asked another follow-up to see whether the original prompt prefix could still be reused.
The first experiment was the runtime. At a physical batch of 512, b12314 took 370.49 seconds to first visible text and llama.cpp 1.0.0 took 365.02 seconds. The older run logged 15.26 GB of process reads. The newer build ran second and logged just 80 KiB. I upgraded to keep the lab current after 1.0.0 passed retrieval, tool calls, vision, and the API checks. I couldn’t isolate a runtime speedup from that pair of runs. llama.cpp 1.0.0 release
Then I held the runtime at 1.0.0 and compared --ubatch-size 512 with 1024. The logical batch stayed at 2048. The physical batch determines how many prompt tokens the runtime can process together. Both runs had a warm macOS file cache. The first request in each run had almost no reusable model prompt, which is a different cache.
| Same runtime, roughly 156K input | Physical batch 512 | Physical batch 1024 |
|---|---|---|
| Time to first token | 363.46 s | 316.90 s |
| Time to first visible answer | 364.07 s | 317.75 s |
| Prompt processing | 429.23 tokens/s | 492.40 tokens/s |
| Generation | 42.18 tokens/s | 42.59 tokens/s |
| Warm follow-up, first visible answer | 0.87 s | 0.84 s |
| Follow-up after an unrelated chat, first visible answer | 0.83 s | 0.80 s |
| Peak sampled server RSS | 157.0 GiB | 157.5 GiB |
| New system swap-outs | 0 | 0 |
The larger batch cut 46.32 seconds from the wait for visible text, about 13%, while adding about half a GiB to sampled RSS. It barely moved generation or warm times. All three retrievals passed at 1024, along with structured tool calling and vision. I kept it based on this pair of runs. I’d need repeated runs and a wider set of histories to say how much it usually saves.
| Other server experiment | What I observed | What I kept |
|---|---|---|
| Reasoning effort 25 vs. 50 | Both rendered correctly on Chat Completions, passed three exact synthetic tool turns, and took a median of 2.63 s to visible text. | Effort 50. The simple task gave me no reason to lower it. Finn’s Anthropic route has a separate thinking-budget translation. |
| Thinking off | Hermes’s CLI switch alone left DeepSeek thinking on the tested Anthropic route. An explicit template flag disabled it on a separate route. The two-app comparison still landed at about 43 s to visible text either way, and a separate arithmetic check missed the expected answer with thinking off. | Thinking on by default. I kept the explicit no-thinking route as an option in Finn. |
| Prefix reuse | The larger-batch run got under a second to visible text on both long follow-ups. Each evaluated just 11 new prompt tokens. | Existing cache behavior. The experiment shows reuse is possible after an unrelated request. It doesn’t establish how often it happens on real chats. |
| DSpark and FP8 KV cache | I had used DSpark on MLX and considered the matching draft model for GGUF. FP8 KV cache was unavailable for this architecture on the tested build. | Standard decoding and default KV types. I left DSpark untested on this GGUF. It would target generation, while the batch test pointed to prompt processing as the larger cost on a fresh long turn. |
Finn got faster when he sent less to the model
Finn gave me a way to change the client while Mia kept using the same gateway settings. I asked him to summarize a week of workouts from Fitness Tracker and list my open chores from Honey-Do. Both apps have MCP servers. On one fresh-chat run, Honey-Do returned 33,163 characters and Finn made four extra code calls. A repeat in the same conversation finished in 12 seconds by reusing the previous app results. That was a fast answer to a different freshness requirement, so I used fresh app reads for the comparisons.
The default Hermes setup sent DeepSeek a long list of tool definitions and a skills index before Finn made a single tool call. That bloated even a short request. I built finn-lean to see how much of the wait came from that opening prompt. I gave it only the two read tools the task needed, skipped the bundled skills, and ran the same task against the same DeepSeek model with Tool Search still enabled. The uncached opening prompt fell from 13,281 tokens to 2,063. Reading it took 4.26 seconds, down from 27.31 seconds. Then I tested whether giving Finn those two tools directly would save the discovery round trip.
| What I tried | What changed | Decision |
|---|---|---|
| Ask for open chores at the tool | Honey-Do dropped from 33,163 to 5,676 characters. Prompt processing after the tools returned fell from 28.7 s to 7.4 s. The explicit test also asked Finn to skip code. | Keep the open-items rule in Finn. A later run with the original plain-language prompt chose the filter and made no extra code calls. |
| Give the test profile two read tools and skip bundled skills | With Tool Search in both runs, the first app result arrived in 11.59 s, down from 35.76 s. | Keep finn-lean for the workout and open chores workflow. |
| Expose the two tools directly | First app result at 9.43 s, versus 11.59 s with Tool Search. The total CLI runtime was about 30 s either way in that pair. | Direct exposure in the two-tool profile. A larger catalog has different trade-offs. |
| Use instant chat titles | Removed a separate inference request. With direct tools, two scratch runs finished in 27–28 s instead of 30.24 s in the earlier run. | Keep instant titles in both regular and lean Finn. |
| Upgrade Hermes to 0.23.1 | The full opening prompt dropped to 11,822 tokens. Time to the recorded answer went from 52.29 s to 53.99 s. The lean answer stayed near 23 s. | Keep the upgrade. The first runs showed smaller prompts and no clear answer-time gain. |
| Manually compact a synthetic long chat | Input fell from 44.3K to 38.3K tokens. All five test facts survived. Compaction took 212.5 s and the next request spent 78.8 s reading the rewritten prompt. | Leave automatic compaction settings alone. One manual test says little about automatic compaction in Mia’s longer chats. |
Two fresh CLI chats in the same lean profile showed how much the opening cache can change the wait. The second reused 1,807 of 1,811 opening tokens. Time to first answer text fell from 25.1 to 13.4 seconds. Resuming after an unrelated request reused only 1,778 of 5,044 opening tokens in another test, and the first answer text took 36.3 seconds. I kept those measurements separate from the controlled 156K-token replay. In a native Desktop run, Finn still called both apps but reported 12 open chores when Honey-Do returned 11. That’s a concrete reason to keep testing before recommending a lean setup for Mia.
I’ll keep a normal profile for general-purpose Hermes sessions and create lean profiles around recurring jobs. Each can carry just the tools and skills the job needs. A future finn-research profile could have search and page extraction tools plus writing skills. The workout and chores test gave me a way to check whether a profile like that saves time and still gets the job right.
I also left the Spark routing comparison for later. A useful comparison has to run the same tools and score the actual answers.
The Studio now starts the pinned build at login
A launch script I could run from Terminal was enough to benchmark. For everyday use I installed a per-user macOS LaunchAgent with RunAtLoad and KeepAlive. The first attempt couldn’t open its launcher under Documents, so I moved the service files into Application Support. Hard links let the installed service use the existing GGUF files without a second copy. The agent starts on login and can restart the server when it exits. I verified the agent serving requests after a restart.
The first time I promoted batch 1024, the server started at 1024 and the verifier crashed trying to decode the startup log as UTF-8. The script rolled back to 512. I changed the verifier to match the startup markers as bytes, checked both logs, then reran the promotion and verified the setting against the running service. This is the configuration I ended up with.
| Setting | Running configuration |
|---|---|
| Host and runtime | 256 GB M5 Ultra Mac Studio, llama.cpp 1.0.0 Apple Silicon build b12359, commit b1b770d90 |
| Checkpoint | Full ISTA-DASLab V4.1 Flash 1017 GSQ-RCO IQ2_XXS, revision 65d53bc8be65123e255ff02f3ded958b6dbf54fd |
| GPU and Engram | Metal MTL0, all 41 layers offloaded, memory-mapped weights and lazy IQ4_NL lookups |
| Context and batching | 262,144 total tokens, one slot, logical batch 2048, physical batch 1024 |
| Template and reasoning | Embedded Jinja template, reasoning on, server effort 50 |
| Vision | BF16 projector enabled after a passing image probe |
| Endpoint and aliases | Authenticated loopback and LAN on port 11234, deepseek-v4.1-general and model |
| Lifecycle | Pinned login service with automatic restart and idle sleep prevention while serving |
Here’s the server command with generic paths. On the shared service I bind the Studio’s LAN address alongside loopback and authenticate the clients using the key file.
caffeinate -i "$LLAMA_SERVER" \
--model "$MODEL_DIR/DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00001-of-00006.gguf" \
--device MTL0 -ngl 99 --fit off --load-mode mmap --lazy-mode auto \
--ubatch-size 1024 --parallel 1 --ctx-size 262144 \
--host 127.0.0.1 --port 11234 --cors-origins localhost \
--api-key-file "$API_KEY_FILE" \
--alias deepseek-v4.1-general,model --jinja \
--reasoning on --reasoning-effort 50 --reasoning-format deepseek \
--metrics --log-timestamps -lv 4 \
--mmproj "$MODEL_DIR/mmproj-DeepSeek-V4.1-Flash-1017-BF16.gguf"The Studio now does the job I bought it for. It can pick up a long Hermes conversation, handle an image, and return a tool call on the same endpoint Mia already used. A new long history can still make her wait five minutes. That’s why the next post starts with Finn’s actual workload. I want to know which conversations can keep their cache, which tools are worth putting in every prompt, and which jobs would feel better on the Sparks.
By the Numbers
- 248.84 GB — Backbone and Engram on disk, before the 0.97 GB projector
- 41 of 41 layers — Offloaded to Metal
- 262,144 tokens — Total context available to one request slot
- 46.32 seconds — Reduction in time to visible text from batch 512 to 1024 in the roughly 156K-token run
- 0.84 seconds — Time to visible text on that run’s warm follow-up
- 5 minutes and 2 seconds — Server-side elapsed time for the observed seven-request Telegram turn