Local MoE: tuning the build, not the model

· 4 min

I run a 125-billion-parameter language model at home, on a gaming PC with a 24 GB graphics card and 64 GB of DDR5 RAM. It's a mixture-of-experts (MoE) model, each layer has 512 experts, and each token only uses 10 of them. It's an architecture I've grown fond of, because it keeps my room cooler. Alibaba have been utilising this architecture for their Qwen open model range, it has some distinct advantages:

The catch is speed. Out of the box it wrote 6.1 tokens a second...

I'm not an inference specialist, so I treated it like any performance problem: measure, find the bottleneck, change one thing at a time. Coding agents did the hands-on tuning, and two changes took it to 27.6 tokens a second: 4.5 times faster, on the same model file and the same hardware.

Neither change touched the model. Both were in the build.

1. A build for this CPU

The first measurement had all six CPU cores at 100% and the GPU 24% busy. With the experts in RAM, the CPU does most of the work for each token. The CPU was the bottleneck.

The agent's first theory was wrong. It measured how fast the CPU got through each of the model's number formats, saw the compressed ones running at a third of the speed of the rest, and suggested switching to a different model file. Then it found a report of the same setup running at 17-20 tokens a second, nearly three times what I had. Something else was going on.

It checked the CPU library bundled with the app. AVX2, AVX-512 and FMA, the instructions that let a modern CPU do many multiplications at once, were all switched off. The app ships a generic build that runs on any x86 CPU, so it uses none of them.

Build Tokens/s
Bundled with the app 6.1
llama.cpp compiled for this CPU 17.9

The same measurement explained the wrong theory. On the new build, every format ran 2.4-7.4 times faster, and the compressed formats gained the most. It was that the default build didn't work for my hardware.

2. Caching the busiest experts

Each token uses 10 of 512 experts as mentioned, but they aren't picked evenly. Some are used far more than others. An open pull request to llama.cpp (#27861) adds a cache: the most-used experts get a copy in GPU memory, and the rest stay in RAM.

The agent applied it, porting two parts by hand that didn't apply cleanly, then tested cache sizes until GPU memory ran close to full:

Cache slots Tokens/s Free GPU memory
None 18.45 3.7 GiB
32 19.7 11.7 GiB
96 25.2 5.6 GiB
120 27.6 3.4 GiB

That's +50%! Each slot costs about 97 MiB of GPU memory, and 120 is as far as it goes. Any more, and the GPU ran out of memory and took my desktop session down with it.

I created a workbench to test different context lengths. At 50,000 tokens it went from 15.9 tokens a second without the cache to 23.5 with it.

What I could check

Now this is an n=1 scenario, I don't need to cater for all the drivers, hardware, OS and so on. Therefore it wasn't about trying to be in the detail but about ensuring any regression for different scenarios like context windows was caught, and genuine progression was built on top of. The principle of KISS, incremental changes and ensuring a short feedback loop enabled this progress. To name some truths: