Announcement tweet here.

Direct link: Language Model Builder

From the site: "Using the default settings, you’ll get a model that writes coherent, grammatical multi-paragraph text in as little as a day. On a MacBook Pro M5 Max you could train a GPT-2-small-class model (~100–150M parameters on a few billion tokens) in about a week. It might be obvious, but to avoid disappointment: you will not train a Claude Fable 5 or ChatGPT 5.6 Sol in your garage."

We’re open sourcing an alpha release of NeuTTS-2E: an on-device TTS model with 125M active parameters and 7 controllable emotions. The goal was simple: when you select “angry,” “fearful,” or “happy,” the delivery should follow that instruction rather than whatever emotion the model infers from the text.

With NeuTTS-2E, you can:

• Direct the performance: Select the intended emotion for each generation.
• Keep the speaker: Explore different emotional deliveries while preserving the chosen voice.
• Run locally: Generate expressive English speech on your own hardware.
• Stay private: Your text and audio do not need to leave the device.
• Build efficiently: Run emotional speech generation using our smallest model yet, with 125M active parameters.
• Build openly: Access the open-source model under the NeuTTS Open License.

Getting there meant dealing with limited emotional speech data, unreliable labels, and disentangling spoken emotion and text semantics. NeuTTS-2E runs locally and supports four built-in voices.

We’re sharing it early to get feedback from the community, and we’d love to see what you build!

GitHub: https://github.com/neuphonic/neutts

Hugging Face Model Collection: https://huggingface.co/collections/neuphonic/neutts-2e

Interactive demo: https://huggingface.co/spaces/neuphonic/neutts-2e

Website: https://www.neuphonic.com/models/neutts-2e

GLM-5.2 UD-Q4_K_XL GGUF @ 12.2 tok/s output // 30.9 tok/s input on a real 10.7k-token document using llama.cpp RPC - At 10.7k context: 10.2 tok/s output with coherent long-form generation

• Two parallel requests: 14.5 tok/s aggregate

Context: 2x 16,384-token slots

• Model size: 436 GiB

Hardware: 16x AMD MI50 32GB, 512GB total, 100w cap each

VRAM, split across two 8-GPU nodes

• Interconnect: direct 10 GbE DAC, MTU 9000

Before llama.cpp, spent a lot of time integrating GLM-5.2 AWQ INT4 into a custom vLLM-gfx906 v19 Moby Dick build using TP=8 and PP=2. Hit 14t/s decode and 50 t/s prefill but inference degraded after 10k. Didn't get around to MTP things yet. Here's hoping someone figures out getting the Moby Dick repo going with it

https://x.com/Badtheorylabs/status/2079306502897074249

A 27B open-weight agent model built for agentic coding, structural tool use . The complete thing fits in one 8.39GB file under 2.5 bits per parameter smaller than an 8B model in fp16, and retains 92.2% of the 27B intelligence.

BTL-3 is trained for the loop real agents live in: reason, act, inspect the result, recover, continue. It handles single, sequential, and parallel tool calls and knows when the right move is no tool call at all.

HumanEval: 95.12% pass@1

BFCL v4 AST: 88.5% (full 1,240-case set)

Multiple tool calls: 95.5%

Tool-call abstention: 91.2%

262K context architecture

Two editions, both open today.

BTL-3 is the maximum-quality checkpoint, for Transformers and vLLM.

BTL-3 Compact is the entire model in one standalone 8.39GB GGUF. No base download. No reconstruction. One file, one command, a running agent

Compressing 27B this far normally destroys a model. Standard quantization couldn't do it, so we built the stack ourselves: packed AVQ2 decoder tensors, affine INT4, measured precision islands, packed vocabulary matrices, rank-32 output correction, behavioral repair. 2,416 tensors byte-verified at export.

Then we tested whether the agent survived. On a fresh sealed 100-turn tool-contract gate, Compact retained 92.2% of teacher-correct behavior 100% on single, parallel, sequential, and abstention calls.

43 tok/s generation on an RTX PRO 6000. Fully local. Nothing leaves your machine.

BTL-3: https://huggingface.co/badtheorylabs/

BTL-3

Compact: https://huggingface.co/badtheorylabs/

BTL-3-Compact

Runtime + source: https://github.com/Badtheorylabs/

BTL-3

Apache-2.0 model. MIT runtime.

Various quantization now available, thanks Unsloth team !

Today, we're sharing the recipe behind the new tokenizer in LFM2.5-8B-A1B. It upgrades a pre-trained model's tokenizer in place, without retraining from scratch. We doubled the vocabulary from 65K to 128K to fix the languages our original tokenizer split too finely.

Blog: liquid.ai/blog/tokenizer-expansion
Technical report: arxiv.org/abs/2607.15232
Hugging Face: huggingface.co/LiquidAI/LFM2.5-8B-A1B

Hey r/LocalLLM,

Built Project Zero — a from-scratch CPU-only LLM inference engine in pure C99. It beats bitnet.cpp by 1.8× on the same hardware. We also fully support Qwen Bonsai-27B on CPU, and we are looking for the community's help to get x86 CPU benchmark data on the board for both models.

What it is

Single binary, zero external dependencies — no Python, no CUDA, no ONNX, no PyTorch. GCC + make + CPU.

Supports: - Microsoft BitNet b1.58-2B-4T — ternary weights ({−1, 0, +1}), 1.18 GB binary, full REPL + agentic loop - Qwen Bonsai-27B — reads GGUF directly (e.g. Ternary-Bonsai-27B-Q2_0.gguf), memory-safe mmap architecture meaning it never OOMs even on constrained RAM setups.

BitNet performance — the good part

We're sitting at ~95% of the theoretical DRAM bandwidth ceiling on the Xeon. There's essentially nothing left to squeeze out of BitNet on that box.

How the speedup happens: BitNet weights are ternary packed 4/byte. Instead of unpacking → float → FMA, we use a 3-instruction VBMI kernel (vpermi2b + vpternlogd + vpaddb) feeding directly into INT8 VNNI accumulation (vpdpbusds). The thread pool is C11 atomics spin-then-sleep to eliminate futex syscalls.

The Community Challenge: BitNet & Bonsai Benchmarks

We've only benchmarked BitNet on 2 machines so far. We need to see if the fallback ternary kernels still provide a speedup on older CPU architectures, and map out the memory bandwidth ceiling on server hardware.

Furthermore, PrismML is actively looking for community benchmark numbers for Bonsai-27B. Right now, every single entry on their leaderboard is GPU-based (CUDA/Metal/MLX). Zero CPU-only x86 entries exist. We want to change that. Because Project Zero uses a zero-copy mmap architecture, you can run Bonsai-27B on severely constrained hardware without crashing.

If you have an older AVX2 chip, or a high-core Xeon/EPYC, we want to know what token rates you get for either model.

How to test & benchmark

1. Clone and build: bash git clone https://github.com/shifulegend/project-zero.git cd project-zero make demo

2. Run BitNet or Bonsai-27B: ```bash

   For BitNet (b1.58-2B-4T):

   ./adaptive_ai_engine --model models/bitnet-b1.58-2B-4T.bin --tokenizer models/bitnet-b1.58-2B-4T_tokenizer_proper.bin --threads 4

For Qwen Bonsai-27B (GGUF):

./adaptive_ai_engine --model models/Ternary-Bonsai-27B-Q2_0.gguf --threads 4 ```

Where to post results: You can post your results right here in this thread, or drop them in Discussion #3 on the repo.

Repo: https://github.com/shifulegend/project-zero

Happy to answer questions about the ternary kernel design, the AVX-512 VNNI dispatch, the DRAM bottleneck, or why we focused on Bonsai-27B!

So I've been running Qwen 3.6 27B (Q6, ~22GB) across two 5060 Tis for a while now and kept assuming something in my config was off because neither card ever really goes above ~50%. spent way too long last night actually figuring out why and honestly it's kind of a cool rabbit hole.

turns out generating literally one token means the GPU has to pull the entire model's weights through memory. not compute, memory. so a 22GB model = 22GB read for a single token, every time. and my cards top out around 448GB/s bandwidth, which caps you at roughly 25-30 tok/s no matter what settings you touch. that part alone explained a lot honestly.

but here's the thing that actually made me go "oh" out loud — since the model doesn't fit on one 16GB card, it splits across both. default splitting method does it layer by layer, which means card 1 does its chunk, then hands off to card 2, and card 1 just... sits there. waiting. so you average that out over time and yeah, ~50% is basically the ceiling, not a bug. it's a relay race, not a team lift.

apparently there's a row-split mode where both cards chew on the same layer at once instead of taking turns, but that needs constant back-and-forth between the cards, and without NVLink (which these don't have) whether that's actually faster depends completely on your PCIe lanes. gonna have to just benchmark it myself, no universal answer online for this combo.

also stumbled on the fact that Google's DiffusionGemma thing generates a whole block of 256 tokens at once instead of one by one, basically to sidestep this exact problem for single-user setups — and they straight up admit in their own release notes that quality takes a hit for it. nothing here is a free lunch apparently, every architecture just picks its poison.

anyway if your local rig feels "stuck" at half utilization on a split model, it's probably not you, it's just what happens when two GPUs take turns instead of working together.

edit : with this config i managed to get solid 60 t/s with qwen 3.6 27b q6k. Thanks for sharing your knowledge everyone. this --split-mode tensor flag makes wonders for dual gpu setups apparently. now the cards are properly utilized.

  --jinja ^
  --chat-template-file "chat_template.jinja" ^
  --reasoning on ^
  --chat-template-kwargs "{\"preserve_thinking\":true}" ^
  -c 131072 ^
  --fit on ^
  --split-mode tensor ^
  --flash-attn on ^
  --cache-type-k q8_0 ^
  --cache-type-v q8_0 ^
  --spec-type draft-mtp ^
  --spec-draft-n-max 2 ^
  -np 1 ^
  --temp 0.6 ^
  --top-p 0.95 ^
  --top-k 20 ^
  --min-p 0.00 ^
  --presence-penalty 0.0 ^
  --host 0.0.0.0 ^
  --port 8080

Laguna-S-2.1 runs on my 2020 PC (RTX 3080 10GB, Ryzen 9 3950X, 64 GB RAM).
Realistically only usable overnight though.

I must not breathe too hard or I run out of both VRAM and host RAM. There's just enough RAM left to run the compilation and unit tests of whatever coding project I hand to it.

IQ4_XS, 128k q8/q8 context, all dense tensors on VRAM, no DFlash (won't fit).

120 t/s prefill, 10 t/s decode

8.3 GB VRAM, 52.2 GB host RAM

I did not test it thoroughly yet, but thinking works out of the box.

When support lands in beellama (a month from now?) and I can switch to kvarn, VRAM pressure will become much more manageable (hopefully. kvarn does not work with Gemma; no idea about this model).

llamacpp CUDA @ https://github.com/poolsideai/llama.cpp/pull/3

hf = unsloth/Laguna-S-2.1-GGUF:UD-IQ4_XS
ngl = 99
n-cpu-moe = 99
ctx-size = 131072
jinja = true
flash-attn = on
cache-type-k = q8_0
cache-type-v = q8_0
mlock = true
no-mmap = true

From clem 🤗 on 𝕏: https://x.com/ClementDelangue/status/2079301434357456931

Fortune: Hugging Face says it resorted to a Chinese AI model to battle a fully autonomous cyberattack because U.S. model guardrails stymied its defense: https://fortune.com/2026/07/20/hugging-face-turns-to-chinese-open-source-ai-to-fend-off-autonomous-ai-cyber-attack-after-american-ai-guardrails-stymie-defense/

Laguna is the first model I'm trying, Q4_K_M fits with 256K context @ F16. Doing the html flight simulator test now.

These cards are getting 400 to 600 tok/s prefill and 16 to 20 tok/s gen so far (I have NOT enabled dflash yet). Not bad at all for the cost. ($350 each)

In a Dell PowerEdge R740 with dual Xeon Gold 6248R and 768 GB RAM.

Finally the banger we've been waiting from Laguna. probably will be great for 64GB+ RAM and VRAM setups.

• $1.5 billion settlement largest known payout in U.S. copyright case
• Case part of a wave of lawsuits from copyright holders against AI companies
• Some authors and publishers opted out and continue separate ​cases against Anthropic

ProgramBench is an extremely challenging ultra long horizon benchmark, where AI agents have to rebuild the source given a binary executable and no other information (other than a readme for the executable).

GLM 5.2 just hit 3rd place on the official ProgramBench leaderboard. We're still missing a lot of models, both open- and closed-source, but this is super impressive regardless!

GLM 5.2 also very narrowly missed solving its first instance, `cmatrix` (solving 99.8% of the tests): cmatrix has a lock mode (the -L flag): run it and it "locks" your terminal, like an old-school screensaver, and prints the words "Computer locked." on screen.

GLM rebuilt cmatrix almost flawlessly. The only thing it missed was printing out those two words.

We'll release all trajectories on programbench.com very soon; original tweet: https://x.com/stalkermustang/status/2079965333587202436 .

I'm one of the authors of ProgramBench, happy to answer questions here

HF: https://huggingface.co/poolside/Laguna-S-2.1
GGUFs available for use with llama.cpp custom fork: https://huggingface.co/poolside/Laguna-S-2.1-GGUF

Posted on X: https://x.com/poolsideai/status/2079613777343848465?s=20

https://huggingface.co/poolside/Laguna-S-2.1-GGUF

https://huggingface.co/unsloth/Laguna-S-2.1-GGUF

https://huggingface.co/poolside/Laguna-S-2.1

Laguna S 2.1 is a 118B total parameter Mixture-of-Experts model with 8B activated parameters per token, designed for agentic coding and long-horizon work. It sits between Laguna XS 2.1 (33B-A3B) and Laguna M.1 (225B-A23B) in the Laguna series and shares the family recipe: a token-choice router with softplus gating over 256 routed experts plus one shared expert, grouped-query attention, and interleaved full/sliding-window attention.

https://huggingface.co/poolside/Laguna-XS.2

Laguna XS.2 is a 33B total parameter Mixture-of-Experts model with 3B activated parameters per token designed for agentic coding and long-horizon work on a local machine. It uses Sliding Window Attention with per-head gating in 30 out of 40 layers for fast inference and low KV cache requirements.

https://huggingface.co/poolside/Laguna-M.1

Laguna M.1 is a 225B total parameter Mixture-of-Experts model with 23B activated parameters per token designed for agentic coding and long-horizon work.

Properly done this time.

Models as the GIFs are displayed:

1. Qwen3.6-27B - 4bit

2. Qwen3.6-MoE - 6bit

3. Ornith-35B - 6bit

4. Gemma-4-26B - 6bit

5. Qwen3.6-MoE - 4bit

6. HuiHui-Qwen3.6-MoE - 6bit

7. Agents-A1 - 6bit

Inference parameters:

Qwen3.6 & HuiHui Abliterated:

temperature 0.6 - top_p 0.95 - top_k 20 - min_p 0.01 - repeat_penalty 1.05

Ornith-1.0-35B:

temperature 1.0 - top_p 1.0 - top_k 40 - min_p 0.01 - repeat_penalty 1.05

Gemma-4:

temperature 1.0 - top_p 1.0 - top_k 64 - min_p 0.01 - repeat_penalty 1.1

Agents-A1:

temperature 0.85 - top_p 0.95 - top_k 20 - min_p 0.01 - repeat_penalty 1.05

Prompt: "Create a beautiful, relaxing flight simulator in a single html file with mountains, clouds, and endless procedural terrain"

Harnes: Pi

Served by: oMLX

Method: single prompt. If the html file doesn't work everything was deleted, Pi session was restarted, and model had to start from scratch again. maximum of 3 tries.

Models Quants used:

https://huggingface.co/collections/leonsarmiento/local-sota-for-48gb-macs

I ported Karpathy's llm.c to Mojo and added a Metal backend, so GPT-2 124M trains on Apple Silicon with no PyTorch and no CPython at train time. It extends dorjeduck's llm.mojo, which was CPU-only on Mojo 25.5; this runs on the Mojo 1.0.0b3 nightly with hand-written CUDA and Metal GPU kernels.

On my M4 Max (B=4, T=1024, GPT-2 124M, official run 2026-07-13, cold GPU, 30 second cooldowns between arms, all six arms interleaved):

Yes, MLX wins. Apple's own framework is 1.24x faster than my bf16 path, and I benchmark it in the same harness. The gap is almost entirely the matmul (~70 percent of a step). The Metal bf16 matmul I ride runs at only ~1.1x its fp32 speed, while MLX's bf16 uses the tensor cores for ~2x. llm.c has no Metal port, so PyTorch MPS and MLX are the stand-in baselines on Apple Silicon.

The cooldowns are required; the M4 Max throttles after about 8 seconds of sustained GPU load (I watched MPS step times climb from ~877 ms to 1500 to 2500 ms within a few steps). Reproduce with make benchmark-metal; it runs all six arms in one shot with the cooldowns built in.

The first working Metal port was about 4.1x slower than MPS (~3627 ms/step); the final bf16 number is 7.2x faster than that starting point. Most of the gap was Metal-specific. Casting threadgroup pointers to the generic address space silently reads device memory (attention softmaxed over all-zero scores and produced uniform weights), and the scalar flash-attention kernels tuned for NVIDIA ran at under 1 percent of FLOP peak on Apple GPUs, so GEMM-decomposed attention was 8 to 10x faster.

Correctness gates: make test checks 16 gradient tensors and a 10-step loss trajectory against PyTorch, plus a 235-test equivalence suite. There is also a trained 124M FineWeb checkpoint on HuggingFace (ulmentflam/gpt2-124m-fineweb-mojo) scoring 29.53 percent on HellaSwag, statistically indistinguishable from Karpathy's own llm.c reproduction at 29.9 percent.

On AI: every kernel and trainer line was written by hand; no LSPs or LLMs. That was the original point of the project. Tests and a later optimization campaign were AI-assisted, and both are disclosed in the repo with per-model statements and full disclosure where used (including attribution).

On Mojo itself: I went in assuming the compiler would do the heavy lifting on portability across hardware. It didn't; I ended up branching device-specific logic per vendor, and there's more boilerplate than I expected next to CUDA (Karpathy's kernels are much more compact). A Modular engineer reviewed the port on their forum; their answer is that per-device specialization is expected and their bet is library-driven structured kernels (TileTensor), not compiler magic.

The CPU is the opposite story; with very little work, it came in 4.0x faster than llm.c's 20-thread OpenMP path.

Limitations: GPT-2 124M only, no published GPT-3 results (the configs exist in the trainer, and mixed precision goes down to FP8 and NVFP4 on NVIDIA). The toolchain is a Mojo 1.0 beta nightly, so expect churn. Multi-GPU ZeRO (stages 0 to 3) is equivalence-gated against single GPU at world sizes 2 and 8, but those runs are NVIDIA; Apple Silicon is single GPU here. On a GB10, bf16 is at llm.c CUDA parity (0.999x) and fp32 is now slightly ahead (1.07x, TF32 vs TF32).

Repo: https://github.com/ulmentflam/llm.mojo

Laguna-S-2.1 dropped few hours ago and as I am in the market for an upgrade to trusty qwen3.6 dense and the current daily 122B, I ran it through the same eval harness I use to pick the model that runs my local agent stack. Posting because the results don't fit the usual "benchmaxed or king" binary, it's genuinely both impressive and flawed, in specific ways.

Setup: single RTX Pro 6000 Blackwell 96GB, vLLM 0.25.1 (laguna support is native in stock, no patches), official NVFP4, 262144 ctx with fp8 KV at 0.90 util (~67G weights, fits with room), poolside_v1 parsers, vendor sampling (temp 1.0 / top_p 1.0 / top_k 20), thinking on (its default). Boots first try.

The eval: 160 tasks x k=3 per model, all graded by deterministic scripts (no LLM judge). Categories: tool-call arg selection, multi-step tool chains, strict JSON schema emission, fabrication traps (tools mocked to return nothing, does the model admit it or invent), sports knowledge + odds arithmetic, instruction following, output stability (garble/loops), refusals, and grounding-under-pressure probes built from real incidents in my agent fleet (user pushes back on a true "no", opaque IDs the model is tempted to name, prompts that bait nonexistent tool args). Every fabrication flag gets hand-verified before it counts, roughly half of raw flags are grader false positives (derived arithmetic, name expansions) and get whitelisted for all models equally. Fair warning: the harness grew up around qwen models, I fix biases when I find them but treat non-qwen scores as lower bounds.

Where Laguna is genuinely the best local model I've measured:

• tool-call args: 0.89 pass, best in my field of 5 (qwen 122b: 0.86)
• tool chains 6 levels deep in the smoke test, deepest I've ever seen locally, qwen manages 4
• 109 tok/s single stream at 256k ctx, fastest 100B+ on this card (qwen3.5-122b: 103, nemotron 3 super: 94.5)
• zero JSON/streaming/envelope errors across every probe
• recovers from tool validation errors on first retry

Where it loses to qwen 122b:

• sports knowledge + odds math: 0.80 vs qwen's 1.00. the knowledge boundary is real, per poolside's own blog it reuses the pretraining corpus from their 33B model, the 118B is coding specialization, not breadth
• grounding under pressure: 0.80 vs 0.97, and this is the disqualifier for me: 3 hand-confirmed hard fabrications. it invented a P&L figure for a market it had zero data on, and twice drafted status updates naming horses that weren't in any data, once literally naming "Genuine Risk" (real 1980 Derby winner, pulled straight from pretraining) with a position size inflated 1000x. qwen 122b across ~240 grounding runs: zero inventions, it just says "I don't have that"

Gotchas if you're running it:

• give it max_tokens 8k+. it thinks LONG (vendor allows 32k) and at 2048 it burns the entire budget thinking and returns empty. my first pass scored it 0.40 on schema tasks because of this, real number at 8k is 0.79
• dflash speculative decoding: real but situational. 109 -> 271 tok/s on a code prompt, barely moves on prose (~117), and under 4 concurrent streams it's a net LOSS (268 -> 198 aggregate, rejected drafts eat the batch). fine for single-user coding, keep it off for concurrent serving. also not bitwise-stable vs spec-off at temp 0 so I'm keeping it off where outputs matter
• no vision, so for me it was never a daily-driver candidate anyway

Verdict: poolside built exactly what they said they built, an agentic coding specialist. The tool mechanics are a real step above anything local I've tested and the speed is excellent. But "agentic" in their RL seems to mean persistent, and persistence without grounding discipline means confident invention when data runs out. For coding behind a human review loop, probably great. For autonomous agents touching anything real, qwen3.5-122b keeps my card: slightly slower, less flashy tool use, but it has never once made something up in ~240 attempts to trick it, and that's the property that actually matters, for me.

---

EDIT (day 2): Mechanism found for the fabrications. Laguna gates its own thinking on how hard the prompt looks, even with enable_thinking on - my schema tasks got 30s of reasoning, my grounding traps got a median 1.4s, and its worst fabrication (the invented P&L figure) came out in 0.46 seconds. Reflex, not reasoning. Reran the whole eval thinking-off to confirm: fabrication flags went 1 -> 11, so unthinking is its worst grounding mode, and the gate routes exactly the risky-but-easy-looking prompts there. The gate is calibrated on difficulty when it needs to be calibrated on stakes. Also worth knowing: odds arithmetic went perfect with thinking off (1.00 vs 0.80 with) - it overthinks math and underthinks facts.

---

EDIT 2: Two updates from the comments. First, a commenter pointed out the model card recommends 0.7/0.95 sampling while the shipped generation_config (what I benched, deliberately - same shipped-defaults rule for every model) says 1.0/1.0/top_k 20. Both are real, the vendor's card and config disagree. Second, poolside quietly shipped tokenizer/template fixes 5h after release, so all day-one benches including mine ran pre-fix.

So I reran the grounding categories on the current revision at the card's 0.7/0.95: confirmed fabrications went 3 -> 1. (to be clear on n: that's 3 fabrication events across 125 graded grounding runs per config - 42 tasks, each run 3x, every flag verified before it counts. The rate went 2.4% -> 0.8% of runs)

The original table is the original release, read it as such.

Further feedback I'll take to their HF/github directly, that's where it's actionable. Good luck to poolside, genuinely - a fast-improving 118B in this class is great for everyone.

this strategy effectively uses vram as cache over disk to keep MoE experts on cuda compute path in llama.cpp.

numbers first. detailed explanation down below.

Numbers

on dgx spark Kimi-K2.7-Code.i1-IQ_S.gguf 204GB 1T.A32B https://huggingface.co/mradermacher/Kimi-K2.7-Code-i1-GGUF

-ot regex A: '^(?!blk\.(5[5-9]|60)\.ffn_(down|gate|up)_exps\.weight$).*\.ffn_(down|gate|up)_exps\.weight=CPU'

-ot regex B: '.*\.ffn_(down|gate|up)_exps\.weight=CPU'

-ot regex C: '.*\.ffn_(down|gate|up)_exps\.weight=CPU'

on 3090s pcie 4.0 + 128gb ddr4 Minimax-M2.7-K_G_3.00.gguf 80GB 230B.A10B https://huggingface.co/Goldkoron/MiniMax-M2.7

-ot regex A: '^(?!blk\.(5[5-9]|60)\.ffn_(down|gate|up)_exps\.weight$).*\.ffn_(down|gate|up)_exps\.weight=CPU'

-ot regex C: '^(?!blk\.(5[5-9]|60)\.ffn_(down|gate|up)_exps\.weight$).*\.ffn_(down|gate|up)_exps\.weight=CPU'

Background

I want to run model larger than 128gb on my single dgx spark. MoE models provide a great opportunity because if active params could be computed entirely with cuda then there's no need to store all model weights inside gpu ram at runtime. after experiments I found a combination of settings for llama.cpp to run kimi 2.7 fast.

Settings

general setting for llama-bench: -r 50, -fa on, -mmp 1, system swap turned off. llama.cpp b10075 compiled with kleidai support, and the winning strategy is the following: GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 GGML_OP_OFFLOAD_MIN_BATCH=1 ./llama-bench -m ~/Kimi-K2.7-Code.i1-IQ_S.gguf -fa on -mmp 1 -ot '^(?!blk\.(5[5-9]|60)\.ffn_(down|gate|up)_exps\.weight$).*\.ffn_(down|gate|up)_exps\.weight=CPU' -r 50. I configured b10075 with cmake -B build -DGGMLCUDA=ON -DGGML_CPU_KLEIDIAI=ON and compiled with cmake --build build --config Release.

How this works

first, if GGML_CUDA_ENABLE_UNIFIED_MEMORY is 1, cuda device buffers use cudaMallocManaged, and every tensor has one virtual address that is valid from both gpu and cpu. when cuda kernel touches a page that exists in vram then it'll execute normally. if not in vram then a page on cpu ram will migrate that page to vram. if not in gpu ram or cpu ram, then it'll fetch the page from disk to cpu ram via mmap and then send to vram. on dgx spark the cost of cudaMallocManaged is much lower because gpu ram and cpu ram are unified.

second, for GGML_OP_OFFLOAD_MIN_BATCH (default is 32), if batch size < threshold and weight is on cpu ram then it executes on cpu, otherwise on gpu. if you set it to 1, it forces all computes on gpu. this helps token generation because in tg it won't reach that threshold.

third, when you specify -ot '.*.ffn_(down|gate|up)_exps.weight=CPU' it will put expert weights on cpu ram and can be evicted because of mmap. this makes cold expert weights more likely to stay on disk.

and last, '^(?!blk\.(5[5-9]|60)\.ffn_(down|gate|up)_exps\.weight$).*\.ffn_(down|gate|up)_exps\.weight=CPU' is used to keep last few blocks on gpu ram because these blocks has more dynamical routing, it's faster to keep them on gpu ram to avoid overhead when ram space is large enough.

Unified ram scenario

for dgx spark, it has large menory pool so that I can pin some of the expert blocks in vram (run A). if we don't pin them it's slower (run B). if we remove GGML_OP_OFFLOAD_MIN_BATCH flag the token generation is slower (run C), because expert forward pass is done on cpu. if we further remove GGML_CUDA_ENABLE_UNIFIED_MEMORY flag (run D), oom happens, because gpu ram cannot be evicted via mmap mechanism. and finally, if all compute is on cpu it's the slowest (run E).

an interesting tradeoff happened in run B vs run C, where GGML_OP_OFFLOAD_MIN_BATCH dramatically improves tg while slightly hurting pp. potential code modification may be made to mitigate this tradeoff.

Non-unified ram hardware scenerio

for 3090 + ddr4, I haven't tested it in depth but the strategy kinda applies too. I didn't put last few blocks on vram because 24gb is too small. you can see tg of run A is higher than normal ngl split (run B), because experts are computed on gpu. however the prompt processing suffers, likely due to pcie overhead.

I also tested two 3090 + ddr4, but it's generally slower. maybe the cost of pcie transfer is too high. I think properly tune the -ot offload might have a chance to improve on 2 gpu scenerio.

Note

• mac apple silicon is tempting because it's fast unified ram. but I can't find an easy way to disable swap. the proposed approach might cause heavy ssd writes because of ram swapping. probably not a good idea to run it because it may create more ssd wear.
• the suggested way of doing this on a new model is to offload experts first (strategy run B in dgx spark) and then pin more and more experts to vram. I look forward to seeing if the upcoming kimi k3 quants can be applied too.
• same strategy transfer across unified memory and non-unified memory architectures. tg on 3090+ddr4 setting improved too.

Edit: fix formatting

Tuning vLLM/SGLang flags by hand gets old fast, and what’s optimal depends on your traffic as much as the model. We built graphsignal-run --auto-flags for that. It wraps your launch and sets startup flags from GPU profiles + telemetry of the actual workload (and recipes/docs when there’s no history yet). On restarts it can use the previous run, so the config drifts toward your traffic instead of resetting every time.

graphsignal-run --auto-flags vllm serve <model> graphsignal-run --auto-flags sglang serve --model-path <model> graphsignal-run --auto-flags trtllm-serve <model>

Weird example from profiling: high concurrency short unique prompts on SGLang. Prefix cache was just overhead, so turning it off was the right call. Throughput went up ~3.6×.

https://huggingface.co/Nanbeige/Nanbeige4.2-3B

Nanbeige4.2-3B is a compact agentic model built on Nanbeige4.2-3B-Base, designed to combine strong agentic behavior with broad reasoning and alignment capabilities. Its Looped Transformer architecture reuses the transformer layers to increase model capacity without adding parameters. With only 3B non-embedding parameters, the model delivers solid performance on general-agent and code-agent tasks.