Every Mac owner who has run a local LLM knows the feeling: the model downloads fine, the first answer looks great, and then you watch it type like it is on dial-up. The instinctive reaction is to blame the chip. The data says otherwise. This spring, Ollama swapped its inference engine on Apple Silicon and doubled one model’s output speed — from 58 to 112 tokens per second — on identical hardware. Nothing about the silicon changed. Once you understand why, every real optimization for local inference on a Mac falls into place, and most of them are free.
This guide walks the whole ladder: the formula that determines your ceiling, six fixes ordered from free to expensive, the myths that waste money, and the one spec worth paying for if you are buying a new Mac. Everything is cross-checked against Ollama’s own benchmarks, Apple’s ML research posts, and independent testing.
The One Formula That Explains Everything
A language model writes one token at a time, and for every single token it must read through all of its weights from memory. Not once per answer — once per token. Think of it as a book: to write each word, the model reads the entire book cover to cover. That makes generation speed a function of two things: how thick the book is (model size in gigabytes) and how fast you can turn the pages (memory bandwidth in GB/s). The ceiling formula is simple:
max tokens/sec ≈ memory bandwidth (GB/s) ÷ model size (GB)What is missing from this formula is just as important: GPU core count and chip marketing names. For token generation, a faster chip with the same memory bandwidth is barely faster. It is not the chip — it is the pipe.
Here is the current bandwidth landscape (Apple’s published specs): the base M6 in the Mac Mini does 170 GB/s (the 16 GB config actually runs at 153 GB/s — the bigger-memory variants get the wider pipe), the M5 Pro does 307 GB/s, the M5 Max 614 GB/s, and the M5 Ultra in the Mac Studio around 1,200 GB/s. For comparison, an RTX 5090 pulls about 1,800 GB/s — ten times the Mac Mini.
Run the numbers for a popular dense model — a 27B at 4-bit quantization is roughly 17 GB. On the Mac Mini: 170 ÷ 17 ≈ 10 tokens/second. That is the hard ceiling, no matter what software you run. On an M5 Max: 614 ÷ 17 ≈ 36 tokens/second. Independent measurements land at 85–95% of these ceilings in practice (one test measured 31 tok/s against a predicted 36), so treat the formula as a ceiling you will approach, not exceed.
So why run AI on a Mac at all? Because the pipe is slower but the shelf is enormous. A Mac has unified memory measured in the hundreds of gigabytes on the high end — a gaming GPU has a fast pipe and a tiny shelf. Keep that asymmetry in mind; it decides which optimizations matter most.
Fix 1: Switch to an MLX Engine (Free)
Most Mac users start with Ollama, which for years ran on llama.cpp — an engine built to run everywhere, and therefore optimized for nothing in particular. Apple has its own framework, MLX, written specifically for its silicon and its unified memory. In March 2026, Ollama started switching its Apple Silicon path to MLX; their measured test on a 35B MoE model went from 58 tok/s to 112 tok/s for generation, and prompt reading jumped from about 1,100 to 2,800 tokens per second. Part of that jump came from a file-format change bundled in the same update, so take the headline with a grain of salt.
Independent tests put MLX roughly 1.5× llama.cpp for generation — but with sharp edges. Zach Rattner’s 54-run bake-off on an M1 Ultra found MLX 41% faster on a dense 27B at 4-bit, the lead vanishing at 8-bit, and Ollama actually winning by 22% on a mixture-of-experts model. Apple’s own community benchmarks show a 15–35% MLX advantage depending on chip and model. Community testing (willitrunai.com) also found MLX uses roughly 10% less memory thanks to native unified-memory handling.
Verdict: yes, try it first. If your app offers an MLX version of the model, pick it — it is the closest thing to free speed on a Mac. The hint is right in the name on Hugging Face: the mlx-community organization maintains converted versions of popular models. In Ollama, look for the -mlx suffix (ollama run gemma4:12b-mlx). Benchmark both engines on your model if it matters — the answer moves more than the marketing suggests. As of the June 2026 update, Ollama’s MLX engine also supports NVIDIA’s NVFP4 format, which roughly halves the quality loss of 4-bit quantization compared to the common q4_K_M format, and runs about 20% faster.
Fix 2: Pick a Bigger Model — a Mixture of Experts (Free, and It Sounds Backwards)
This is the fix that should not work: choose a bigger model and go faster. A mixture-of-experts (MoE) model is a huge book split into chapters, where each token only reads the few chapters it needs. The 35B MoE model from Ollama’s test has 35 billion parameters total but only about 3 billion active per token. The file is ~21 GB, but the model only reads a small slice of it for every word it writes.
The practical results are dramatic. The dense 27B model on an M5 Max managed 31 tok/s. The bigger MoE model on an older, weaker M4 Max was measured at 70–90 tok/s — a bigger model on a worse chip, two to three times faster. On the Mac Mini, the ceiling goes from ~10 tok/s to something like 80 on paper (you will see less in practice, but you will feel it immediately).
The catch: the whole book still has to fit on the shelf. You read 3 billion parameters per token, but all 35 billion sit in memory. This is exactly why MoE and Macs are made for each other — a model that is big on the shelf but light on reading is the perfect fit for a slow-pipe, huge-shelf machine. Community benchmarks of the Qwen 30B-A3B class on a 36 GB M4 Max show 30–55 tok/s depending on engine, versus OOM errors or single digits for dense models of similar total size.
Fix 3: Quantization — Make the Pages Thinner (Free)
Every parameter is stored as a number, and you can store numbers with fewer bits. At 8 bits, a 27B model is ~29 GB; at 4 bits, ~17 GB. Since speed is bandwidth divided by size, half the size means roughly double the speed. On the Mac Mini, the 27B ceiling is ~6 tok/s at 8 bits versus ~10 at 4 bits. Same model, same Mac, nearly 2× from a download choice.
But 4-bit is not free: quality degrades a little, and below 4 bits it degrades a lot. Four-bit is the sweet spot for most people; squeezing further to buy speed is usually a bad trade. If you want the full picture of the format landscape — GPTQ, AWQ, GGUF, MXFP4, and now NVFP4 — see the earlier post on choosing between 4-bit formats. The short version: prefer MLX 4-bit or NVFP4 builds when available; both measurably preserve quality better than older formats.
Fix 4: The GPU Memory Cap (Free, and It Decides Fit — Not Speed)
A problem most people do not know they have: macOS does not let the GPU use all of unified memory. The Metal driver exposes a recommendedMaxWorkingSetSize of roughly 65–75% of RAM, and inference tools treat it as a hard ceiling. A 32 GB Mac Mini really offers about 21–24 GB for the model. Our 21 GB MoE fits — barely, with almost nothing left for the conversation itself.
Power users raise the cap with a hidden kernel parameter (sudo sysctl iogpu.wired_limit_mb=...), and it works — but it is undocumented, and overshoot freezes the whole machine. People who do it leave 4–8 GB for macOS. Verdict: this is not a speed boost; it decides whether the model loads at all. When the model does not fit, it does not get slower — it refuses to load, or macOS starts swapping to the SSD and everything grinds to a halt (an SSD is many times slower than RAM; that is how 10 tok/s becomes 1).
Two myths worth killing while we are here:
- Myth: more memory makes models faster. Mostly no — memory decides what fits; bandwidth decides speed. One funny exception: the base 16 GB Mac Mini runs at 153 GB/s while the 24/32 GB variants get 170 GB/s, so on that specific machine the memory upgrade really does buy a slightly wider pipe.
- Myth: it does not matter what else is open. It does. If the model plus your browser tabs exceed RAM, macOS pushes pages to the SSD — and swap is where generation speed goes to die.
Fix 5: Tame the KV Cache (Free)
Everything in your conversation — the question, pasted documents, every answer so far — is also stored in memory as the KV cache, and it grows. One documented case: a developer ran Gemma 4 31B on a 64 GB Mac, raised the memory cap, and the machine still froze — the model’s default 256K-token context needs about 21 GB of cache on its own. And it is not just space: for every new token the model reads the weights and the whole cache, so long chats get progressively slower per token.
Two fixes. First, do not set the context length longer than you need — a 4K context costs a fraction of a 256K one. Second, many apps (LM Studio, Ollama with the right flags) can store the KV cache at 8-bit instead of 16-bit, which roughly halves its size; tests show the quality cost is tiny.
If you run agentic workloads, note that Ollama’s June 2026 MLX update added a snapshot system that saves model state at conversation branch points, so agent sessions that resend tens of thousands of tokens of system prompt and tool definitions per call do not reprocess all of it every request. For agent workflows, prompt processing — not generation — dominates the wait, which leads to the one place the chip matters.
Where the Chip Actually Matters: Prompt Processing
Token generation is memory-bound, but reading your prompt before the first token is compute-bound — and that is where GPU cores and Apple’s new Neural Accelerators earn their keep. The M5 chips added matrix-multiplication units inside the GPU, exposed through Metal 4’s TensorOps. Apple’s benchmarks show time-to-first-token improving 3.5–4× over M4 (a 14B dense model’s first token arrives in under 10 seconds; a 30B MoE’s in under 3), while generation speed after that improved only ~20–27% — mostly from faster memory, not the accelerators.
Real-world impact: one reviewer measured a long-prompt workload at 81 seconds before the first answer on an M4 iPad versus 18 seconds on the M5. An analyst ran an 8B model with a 19,000-token prompt on an M5 — prompt reading went from ~160 tok/s with accelerators off to ~580 with them on, turning about 2 minutes of waiting into 30 seconds. If you paste long documents or run coding agents that ship huge prompts, a newer chip genuinely helps. If you mostly chat, memory bandwidth is the spec that matters. This distinction grows more important over time: coding assistants and AI agents send prompts of tens of thousands of tokens, and every tool call resends the whole transcript.
Fix 6: Speculative Decoding (Free, Model-Dependent)
The last trick feels like cheating. Speculative decoding pairs your big model with a tiny “draft” model from the same family (same vocabulary is required). The draft quickly guesses the next few tokens; the big model verifies all of them in a single pass — cheaper than generating them itself. Correct guesses cost one pass instead of several; wrong ones get thrown away, and the output is exactly what the big model would have written anyway. Quality is untouched by construction.
LM Studio (which supports this in its UI) reports 1.5–3× speedups in good cases. The catch is in “good cases”: if the draft guesses wrong more often than right, you get slower, because you are now running two models. It shines on predictable text — code, boilerplate, structured output — and struggles on creative prose. Keep the draft model small relative to the main one: LM Studio’s guidance says ~1B drafts for 7B mains, 3B for 14B, 7B for 32B.
Climbing the Ladder: One Mac Mini, End to End
Take the concrete machine from the video: a Mac Mini M6 with 32 GB, about $1,300. Stock, the dense 27B at 4-bit limps along under 10 tok/s — and at 8-bit it does not even load within the GPU cap. Then the ladder, none of which costs a dollar:
- Switch to the 35B MoE model → ceiling jumps ~8× (only ~3B parameters read per token)
- Run the MLX build → extract the most from that ceiling, plus lower memory overhead
- Keep the context short and the KV cache at 8-bit → no progressive slowdown, room for the conversation itself
- Add a ~1B draft model for speculative decoding → noticeably faster again on code
Same box, and it goes from painfully slow to genuinely pleasant.
If You Are Buying: Pay for Bandwidth, Not Cores
Buying new for local AI? The one spec worth paying for is memory bandwidth. A Mac Mini with the M5 Pro has 307 GB/s — nearly double the base M6’s 170 — which translates to almost double the generation speed on every model you will ever run. Step up to M5 Max (614 GB/s) or M5 Ultra (~1,200 GB/s) and the ceilings scale linearly. Memory capacity matters second: it decides which models fit at all, and the MoE trick needs headroom. Neural Accelerators matter third, and only if your workload is prompt-heavy.
One honest warning to close: none of these fixes make a model smarter — they make it faster. If the answers themselves are not good enough, that is a model-choice problem, and no sysctl or quantization setting will fix it.
The Receipt
| Fix | Cost | Speedup | Trade-off |
|---|---|---|---|
| MLX engine | Free | ~1.2–1.5× generation (model-dependent; MoE can favor llama.cpp) | Engine/model pairing needs a quick benchmark |
| MoE model | Free (needs RAM headroom) | 2–8× vs dense of similar quality | Whole model must fit in memory |
| 4-bit quantization | Free | ~2× vs 8-bit | Slight quality loss; below 4-bit falls off a cliff |
| Raise GPU memory cap | Free | Enables larger models (not a speed boost) | Undocumented; overshoot freezes the machine |
| Short context + 8-bit KV cache | Free | Prevents progressive slowdown; ~2× cache savings | Shorter memory window; negligible quality cost |
| Speculative decoding | Free (extra RAM for draft) | 1.5–3× on predictable text | Can be slower on creative prose or bad draft pairing |
| Buy M5 Pro tier+ | $$$ | ~2× per bandwidth tier, forever | Money; prompt-heavy workloads also gain TTFT from Neural Accelerators |
The formula to remember: speed = bandwidth ÷ the size you read per token. Switch to MLX for up to ~1.5×. Pick a mixture-of-experts model for several times more. Quantize to 4 bits for another ~2× over 8-bit. Keep the context short, and put a draft model on top for code. Your Mac really is slow at local AI — and it really is not the chip.