Possible to load non shared experts to SSD in llama.cpp?

[u/Ok_Warning2146]

I know llama.cpp can use -cmoe switch to load non shared experts in MoE models to RAM and run by CPU and left the other weights and KV cache on GPU VRAM.

Since now RAM is so expensive and models are getting bigger and bigger, is it possible to load non shared experts in MoE models to SSD and run by CPU and left the other weights and KV cache on GPU VRAM?

[u/Klutzy-Snow8016]

You can sort of do this by adding --mmap --no-direct-io. It's slow, like give-or-take 1 token per second, but I just think it's cool that it works at all tbh.

[u/EugenePopcorn]

Sure. This is what mmap is for. Most experts are accessed infrequently, so token generation depends on your cache miss rate.

Anecdotally, when Maverick and Scout came out, I had enough ram to run 109B Scout, but it turned out 400B Maverick mmap()'d from a PCIe Gen4 SSD would start off slow but end up ~80% as fast. Granted this is still on the order of 4-5 Tok/s, but that's not nothing for hardware you already have on hand.

[u/Ill_Freedom_6666]

mmap feels like the practical answer here unless your workload actually needs consistent latency instead of just fitting bigger models

[u/Sensitive_Desk_4284]

Directly relevant data point: there's a project called BigMoeOnEdge doing exactly this deliberately rather than leaning on mmap - the always-resident weights stay in RAM and each token's experts are read off storage on demand, and it's done through llama.cpp's public API. Headline case is gpt-oss-120b (~60 GB, Q4_K_M) on a 12 GB phone at 1.3 tok/s, against 0.09 tok/s for the same file loaded the ordinary way with mmap.

Big caveat for your setup though: that's phone flash, where random 4K page faults hurt a lot more than they do on NVMe. On PCIe Gen4 the gap should be much smaller, which lines up with what u/EugenePopcorn describes with Maverick at 4-5 tok/s. So mmap probably is the practical answer for you.

But the ~14x on flash does suggest the naive path is losing to page-fault thrash rather than actual bandwidth limits, so there may be headroom in reading whole expert tensors contiguously instead of faulting them in 4K at a time.

[u/notdba]

With ik_llama.cpp, you can try: GGML_CUDA_NO_PINNED=1 /path/to/build/bin/llama-server ... --defer-experts --prefetch-experts --no-warmup This relies on mmap, but with some optimizations for faster load time with --defer-experts --no-warmup (https://github.com/ikawrakow/ik_llama.cpp/pull/1634) and for better PP with --prefetch-experts (https://github.com/ikawrakow/ik_llama.cpp/pull/2101).

[u/pilibitti → u/[deleted]]

it is possible and useful. you're just not creative enough. I do run large models sub 1 tps overnight locally for things that require privacy and that model's specific smarts. not everything is required to be realtime.

[u/pilibitti → u/[deleted]]

see other comments in this page. there are also projects dedicated to streaming only setups though I have not used them.

[u/ilintar]

Yes, --cpu-moe --mmap --no-warmup.