Local MoE: tuning the build, not the model
I run a 125-billion-parameter language model at home, on a gaming PC with a 24 GB graphics card and 64 GB of DDR5 RAM. It's a mixture-of-experts (MoE) model, each layer has 512 experts, and each token only uses 10 of them. It's an architecture I've grown fond of, because it keeps my room cooler. Alibaba have been utilising this architecture for their Qwen open model range, it has some distinct advantages:
- It fits on consumer hardware! Most of the model sits idle at any moment, so it doesn't need to live in GPU memory. The parts every token uses stay on the GPU, and the experts sit in RAM until they're needed. Held entirely in GPU memory, a model this size would need several graphics cards.
- Less work per token, each token only runs through a small fraction of the weights, so it does far less maths than a model of the same size that uses all of them.
- Less power and heat, when you have a 7900 XTX, you very much appreciate this.
The catch is speed. Out of the box it wrote 6.1 tokens a second...
I'm not an inference specialist, so I treated it like any performance problem: measure, find the bottleneck, change one thing at a time. Coding agents did the hands-on tuning, and two changes took it to 27.6 tokens a second: 4.5 times faster, on the same model file and the same hardware.
Neither change touched the model. Both were in the build.
1. A build for this CPU
The first measurement had all six CPU cores at 100% and the GPU 24% busy. With the experts in RAM, the CPU does most of the work for each token. The CPU was the bottleneck.
The agent's first theory was wrong. It measured how fast the CPU got through each of the model's number formats, saw the compressed ones running at a third of the speed of the rest, and suggested switching to a different model file. Then it found a report of the same setup running at 17-20 tokens a second, nearly three times what I had. Something else was going on.
It checked the CPU library bundled with the app. AVX2, AVX-512 and FMA, the instructions that let a modern CPU do many multiplications at once, were all switched off. The app ships a generic build that runs on any x86 CPU, so it uses none of them.
| Build | Tokens/s |
|---|---|
| Bundled with the app | 6.1 |
| llama.cpp compiled for this CPU | 17.9 |
The same measurement explained the wrong theory. On the new build, every format ran 2.4-7.4 times faster, and the compressed formats gained the most. It was that the default build didn't work for my hardware.
2. Caching the busiest experts
Each token uses 10 of 512 experts as mentioned, but they aren't picked evenly. Some are used far more than others. An open pull request to llama.cpp (#27861) adds a cache: the most-used experts get a copy in GPU memory, and the rest stay in RAM.
The agent applied it, porting two parts by hand that didn't apply cleanly, then tested cache sizes until GPU memory ran close to full:
| Cache slots | Tokens/s | Free GPU memory |
|---|---|---|
| None | 18.45 | 3.7 GiB |
| 32 | 19.7 | 11.7 GiB |
| 96 | 25.2 | 5.6 GiB |
| 120 | 27.6 | 3.4 GiB |
That's +50%! Each slot costs about 97 MiB of GPU memory, and 120 is as far as it goes. Any more, and the GPU ran out of memory and took my desktop session down with it.
I created a workbench to test different context lengths. At 50,000 tokens it went from 15.9 tokens a second without the cache to 23.5 with it.
What I could check
Now this is an n=1 scenario, I don't need to cater for
all the drivers, hardware, OS and so on. Therefore it wasn't about
trying to be in the detail but about ensuring any regression for
different scenarios like context windows was caught, and genuine
progression was built on top of. The principle of KISS, incremental
changes and ensuring a short feedback loop enabled this progress. To
name some truths:
- One change at a time - Same model file, same settings, before and after, just like an A/B test except I could control for the variables
- Fail fast - If something wasn't heading in the right direction, try something else, only going deep if there was a lack of understanding
- Tested where it's used. A cache that helps a short chat might not help a long session, so it was measured at 20,000 and 50,000 tokens as well
- The same answers. A faster model that answers differently isn't an improvement. With randomness switched off, the cached and uncached builds gave the same answers word for word, until a minor wording difference of the kind normal floating-point variation produces