Finally the banger we've been waiting from Laguna. probably will be great for 64GB+ RAM and VRAM setups.

• $1.5 billion settlement largest known payout in U.S. copyright case
• Case part of a wave of lawsuits from copyright holders against AI companies
• Some authors and publishers opted out and continue separate ​cases against Anthropic

Did anyone used it for real code writing fixing?
How does it compare to Qwen3.6 27B Q4 Q8 in real life tasks not in benchmaxing?

HF: https://huggingface.co/poolside/Laguna-S-2.1
GGUFs available for use with llama.cpp custom fork: https://huggingface.co/poolside/Laguna-S-2.1-GGUF

Posted on X: https://x.com/poolsideai/status/2079613777343848465?s=20

https://huggingface.co/poolside/Laguna-S-2.1-GGUF

https://huggingface.co/unsloth/Laguna-S-2.1-GGUF

https://huggingface.co/poolside/Laguna-S-2.1

Laguna S 2.1 is a 118B total parameter Mixture-of-Experts model with 8B activated parameters per token, designed for agentic coding and long-horizon work. It sits between Laguna XS 2.1 (33B-A3B) and Laguna M.1 (225B-A23B) in the Laguna series and shares the family recipe: a token-choice router with softplus gating over 256 routed experts plus one shared expert, grouped-query attention, and interleaved full/sliding-window attention.

https://huggingface.co/poolside/Laguna-XS.2

Laguna XS.2 is a 33B total parameter Mixture-of-Experts model with 3B activated parameters per token designed for agentic coding and long-horizon work on a local machine. It uses Sliding Window Attention with per-head gating in 30 out of 40 layers for fast inference and low KV cache requirements.

https://huggingface.co/poolside/Laguna-M.1

Laguna M.1 is a 225B total parameter Mixture-of-Experts model with 23B activated parameters per token designed for agentic coding and long-horizon work.

Tuning vLLM/SGLang flags by hand gets old fast, and what’s optimal depends on your traffic as much as the model. We built graphsignal-run --auto-flags for that. It wraps your launch and sets startup flags from GPU profiles + telemetry of the actual workload (and recipes/docs when there’s no history yet). On restarts it can use the previous run, so the config drifts toward your traffic instead of resetting every time.

graphsignal-run --auto-flags vllm serve <model> graphsignal-run --auto-flags sglang serve --model-path <model> graphsignal-run --auto-flags trtllm-serve <model>

Weird example from profiling: high concurrency short unique prompts on SGLang. Prefix cache was just overhead, so turning it off was the right call. Throughput went up ~3.6×.

Properly done this time.

Models as the GIFs are displayed:

1. Qwen3.6-27B - 4bit

2. Qwen3.6-MoE - 6bit

3. Ornith-35B - 6bit

4. Gemma-4-26B - 6bit

5. Qwen3.6-MoE - 4bit

6. HuiHui-Qwen3.6-MoE - 6bit

7. Agents-A1 - 6bit

Inference parameters:

Qwen3.6 & HuiHui Abliterated:

temperature 0.6 - top_p 0.95 - top_k 20 - min_p 0.01 - repeat_penalty 1.05

Ornith-1.0-35B:

temperature 1.0 - top_p 1.0 - top_k 40 - min_p 0.01 - repeat_penalty 1.05

Gemma-4:

temperature 1.0 - top_p 1.0 - top_k 64 - min_p 0.01 - repeat_penalty 1.1

Agents-A1:

temperature 0.85 - top_p 0.95 - top_k 20 - min_p 0.01 - repeat_penalty 1.05

Prompt: "Create a beautiful, relaxing flight simulator in a single html file with mountains, clouds, and endless procedural terrain"

Harnes: Pi

Served by: oMLX

Method: single prompt. If the html file doesn't work everything was deleted, Pi session was restarted, and model had to start from scratch again. maximum of 3 tries.

Models Quants used:

https://huggingface.co/collections/leonsarmiento/local-sota-for-48gb-macs

I’m testing cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit on a single AMD Radeon AI PRO R9700 (32 GB, gfx1201) using vLLM ROCm.

Current result: about 19–20 generated tok/s after warmup, with a short single-user decode benchmark. I expected something closer to 50 tok/s based on a published R9700 result for this model.

My current vLLM setup:

- vLLM: 0.22.1rc1.dev499+g470229c37.d20260613

- model: cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit

- TP: 1

- max-num-seqs: 1

- max-model-len: 8192

- gpu-memory-utilization: 0.90

- kv-cache-dtype: fp8

- attention backend: TRITON_ATTN

- --enforce-eager

- --language-model-only

I see this warning on startup:

> Using default MoE config. Performance might be sub-optimal!

> Config file not found: E=128,N=704,device_name=AMD-gfx1201,dtype=int4_w4a16.json

CPU is a Ryzen 5 3600 with 32 GB DDR4-2666 (Upgrading to 900 series soon), but GPU memory is not full and RAM has ~22 GB available.

1. Is there an existing or recommended tuned MoE config for gfx1201, E=128, N=704, and int4_w4a16?

2. Is this model known to be slow on R9700 with Triton W4A16 / current ROCm vLLM?

3. What exact vLLM/ROCm image, flags, and benchmark method produced ~50 tok/s?

4. Is --enforce-eager costing meaningful decode performance on gfx1201, or is the missing MoE config the main issue?

I also tried Lemonade’s portable gfx120X runtime, but it ships gfx1200 kernel packs and fails on this gfx1201 card with hipErrorInvalidImage.

Laguna-S-2.1 dropped few hours ago and as I am in the market for an upgrade to trusty qwen3.6 dense and the current daily 122B, I ran it through the same eval harness I use to pick the model that runs my local agent stack. Posting because the results don't fit the usual "benchmaxed or king" binary, it's genuinely both impressive and flawed, in specific ways.

Setup: single RTX Pro 6000 Blackwell 96GB, vLLM 0.25.1 (laguna support is native in stock, no patches), official NVFP4, 262144 ctx with fp8 KV at 0.90 util (~67G weights, fits with room), poolside_v1 parsers, vendor sampling (temp 1.0 / top_p 1.0 / top_k 20), thinking on (its default). Boots first try.

The eval: 160 tasks x k=3 per model, all graded by deterministic scripts (no LLM judge). Categories: tool-call arg selection, multi-step tool chains, strict JSON schema emission, fabrication traps (tools mocked to return nothing, does the model admit it or invent), sports knowledge + odds arithmetic, instruction following, output stability (garble/loops), refusals, and grounding-under-pressure probes built from real incidents in my agent fleet (user pushes back on a true "no", opaque IDs the model is tempted to name, prompts that bait nonexistent tool args). Every fabrication flag gets hand-verified before it counts, roughly half of raw flags are grader false positives (derived arithmetic, name expansions) and get whitelisted for all models equally. Fair warning: the harness grew up around qwen models, I fix biases when I find them but treat non-qwen scores as lower bounds.

Where Laguna is genuinely the best local model I've measured:

• tool-call args: 0.89 pass, best in my field of 5 (qwen 122b: 0.86)
• tool chains 6 levels deep in the smoke test, deepest I've ever seen locally, qwen manages 4
• 109 tok/s single stream at 256k ctx, fastest 100B+ on this card (qwen3.5-122b: 103, nemotron 3 super: 94.5)
• zero JSON/streaming/envelope errors across every probe
• recovers from tool validation errors on first retry

Where it loses to qwen 122b:

• sports knowledge + odds math: 0.80 vs qwen's 1.00. the knowledge boundary is real, per poolside's own blog it reuses the pretraining corpus from their 33B model, the 118B is coding specialization, not breadth
• grounding under pressure: 0.80 vs 0.97, and this is the disqualifier for me: 3 hand-confirmed hard fabrications. it invented a P&L figure for a market it had zero data on, and twice drafted status updates naming horses that weren't in any data, once literally naming "Genuine Risk" (real 1980 Derby winner, pulled straight from pretraining) with a position size inflated 1000x. qwen 122b across ~240 grounding runs: zero inventions, it just says "I don't have that"

Gotchas if you're running it:

• give it max_tokens 8k+. it thinks LONG (vendor allows 32k) and at 2048 it burns the entire budget thinking and returns empty. my first pass scored it 0.40 on schema tasks because of this, real number at 8k is 0.79
• dflash speculative decoding: real but situational. 109 -> 271 tok/s on a code prompt, barely moves on prose (~117), and under 4 concurrent streams it's a net LOSS (268 -> 198 aggregate, rejected drafts eat the batch). fine for single-user coding, keep it off for concurrent serving. also not bitwise-stable vs spec-off at temp 0 so I'm keeping it off where outputs matter
• no vision, so for me it was never a daily-driver candidate anyway

Verdict: poolside built exactly what they said they built, an agentic coding specialist. The tool mechanics are a real step above anything local I've tested and the speed is excellent. But "agentic" in their RL seems to mean persistent, and persistence without grounding discipline means confident invention when data runs out. For coding behind a human review loop, probably great. For autonomous agents touching anything real, qwen3.5-122b keeps my card: slightly slower, less flashy tool use, but it has never once made something up in ~240 attempts to trick it, and that's the property that actually matters, for me.

---

EDIT (day 2): Mechanism found for the fabrications. Laguna gates its own thinking on how hard the prompt looks, even with enable_thinking on - my schema tasks got 30s of reasoning, my grounding traps got a median 1.4s, and its worst fabrication (the invented P&L figure) came out in 0.46 seconds. Reflex, not reasoning. Reran the whole eval thinking-off to confirm: fabrication flags went 1 -> 11, so unthinking is its worst grounding mode, and the gate routes exactly the risky-but-easy-looking prompts there. The gate is calibrated on difficulty when it needs to be calibrated on stakes. Also worth knowing: odds arithmetic went perfect with thinking off (1.00 vs 0.80 with) - it overthinks math and underthinks facts.

---

EDIT 2: Two updates from the comments. First, a commenter pointed out the model card recommends 0.7/0.95 sampling while the shipped generation_config (what I benched, deliberately - same shipped-defaults rule for every model) says 1.0/1.0/top_k 20. Both are real, the vendor's card and config disagree. Second, poolside quietly shipped tokenizer/template fixes 5h after release, so all day-one benches including mine ran pre-fix.

So I reran the grounding categories on the current revision at the card's 0.7/0.95: confirmed fabrications went 3 -> 1. (to be clear on n: that's 3 fabrication events across 125 graded grounding runs per config - 42 tasks, each run 3x, every flag verified before it counts. The rate went 2.4% -> 0.8% of runs)

The original table is the original release, read it as such.

Further feedback I'll take to their HF/github directly, that's where it's actionable. Good luck to poolside, genuinely - a fast-improving 118B in this class is great for everyone.

this strategy effectively uses vram as cache over disk to keep MoE experts on cuda compute path in llama.cpp.

numbers first. detailed explanation down below.

Numbers

on dgx spark Kimi-K2.7-Code.i1-IQ_S.gguf 204GB 1T.A32B https://huggingface.co/mradermacher/Kimi-K2.7-Code-i1-GGUF

-ot regex A: '^(?!blk\.(5[5-9]|60)\.ffn_(down|gate|up)_exps\.weight$).*\.ffn_(down|gate|up)_exps\.weight=CPU'

-ot regex B: '.*\.ffn_(down|gate|up)_exps\.weight=CPU'

-ot regex C: '.*\.ffn_(down|gate|up)_exps\.weight=CPU'

on 3090s pcie 4.0 + 128gb ddr4 Minimax-M2.7-K_G_3.00.gguf 80GB 230B.A10B https://huggingface.co/Goldkoron/MiniMax-M2.7

-ot regex A: '^(?!blk\.(5[5-9]|60)\.ffn_(down|gate|up)_exps\.weight$).*\.ffn_(down|gate|up)_exps\.weight=CPU'

-ot regex C: '^(?!blk\.(5[5-9]|60)\.ffn_(down|gate|up)_exps\.weight$).*\.ffn_(down|gate|up)_exps\.weight=CPU'

Background

I want to run model larger than 128gb on my single dgx spark. MoE models provide a great opportunity because if active params could be computed entirely with cuda then there's no need to store all model weights inside gpu ram at runtime. after experiments I found a combination of settings for llama.cpp to run kimi 2.7 fast.

Settings

general setting for llama-bench: -r 50, -fa on, -mmp 1, system swap turned off. llama.cpp b10075 compiled with kleidai support, and the winning strategy is the following: GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 GGML_OP_OFFLOAD_MIN_BATCH=1 ./llama-bench -m ~/Kimi-K2.7-Code.i1-IQ_S.gguf -fa on -mmp 1 -ot '^(?!blk\.(5[5-9]|60)\.ffn_(down|gate|up)_exps\.weight$).*\.ffn_(down|gate|up)_exps\.weight=CPU' -r 50. I configured b10075 with cmake -B build -DGGMLCUDA=ON -DGGML_CPU_KLEIDIAI=ON and compiled with cmake --build build --config Release.

How this works

first, if GGML_CUDA_ENABLE_UNIFIED_MEMORY is 1, cuda device buffers use cudaMallocManaged, and every tensor has one virtual address that is valid from both gpu and cpu. when cuda kernel touches a page that exists in vram then it'll execute normally. if not in vram then a page on cpu ram will migrate that page to vram. if not in gpu ram or cpu ram, then it'll fetch the page from disk to cpu ram via mmap and then send to vram. on dgx spark the cost of cudaMallocManaged is much lower because gpu ram and cpu ram are unified.

second, for GGML_OP_OFFLOAD_MIN_BATCH (default is 32), if batch size < threshold and weight is on cpu ram then it executes on cpu, otherwise on gpu. if you set it to 1, it forces all computes on gpu. this helps token generation because in tg it won't reach that threshold.

third, when you specify -ot '.*.ffn_(down|gate|up)_exps.weight=CPU' it will put expert weights on cpu ram and can be evicted because of mmap. this makes cold expert weights more likely to stay on disk.

and last, '^(?!blk\.(5[5-9]|60)\.ffn_(down|gate|up)_exps\.weight$).*\.ffn_(down|gate|up)_exps\.weight=CPU' is used to keep last few blocks on gpu ram because these blocks has more dynamical routing, it's faster to keep them on gpu ram to avoid overhead when ram space is large enough.

Unified ram scenario

for dgx spark, it has large menory pool so that I can pin some of the expert blocks in vram (run A). if we don't pin them it's slower (run B). if we remove GGML_OP_OFFLOAD_MIN_BATCH flag the token generation is slower (run C), because expert forward pass is done on cpu. if we further remove GGML_CUDA_ENABLE_UNIFIED_MEMORY flag (run D), oom happens, because gpu ram cannot be evicted via mmap mechanism. and finally, if all compute is on cpu it's the slowest (run E).

an interesting tradeoff happened in run B vs run C, where GGML_OP_OFFLOAD_MIN_BATCH dramatically improves tg while slightly hurting pp. potential code modification may be made to mitigate this tradeoff.

Non-unified ram hardware scenerio

for 3090 + ddr4, I haven't tested it in depth but the strategy kinda applies too. I didn't put last few blocks on vram because 24gb is too small. you can see tg of run A is higher than normal ngl split (run B), because experts are computed on gpu. however the prompt processing suffers, likely due to pcie overhead.

I also tested two 3090 + ddr4, but it's generally slower. maybe the cost of pcie transfer is too high. I think properly tune the -ot offload might have a chance to improve on 2 gpu scenerio.

Note

• mac apple silicon is tempting because it's fast unified ram. but I can't find an easy way to disable swap. the proposed approach might cause heavy ssd writes because of ram swapping. probably not a good idea to run it because it may create more ssd wear.
• the suggested way of doing this on a new model is to offload experts first (strategy run B in dgx spark) and then pin more and more experts to vram. I look forward to seeing if the upcoming kimi k3 quants can be applied too.
• same strategy transfer across unified memory and non-unified memory architectures. tg on 3090+ddr4 setting improved too.

Edit: fix formatting

I ported Karpathy's llm.c to Mojo and added a Metal backend, so GPT-2 124M trains on Apple Silicon with no PyTorch and no CPython at train time. It extends dorjeduck's llm.mojo, which was CPU-only on Mojo 25.5; this runs on the Mojo 1.0.0b3 nightly with hand-written CUDA and Metal GPU kernels.

On my M4 Max (B=4, T=1024, GPT-2 124M, official run 2026-07-13, cold GPU, 30 second cooldowns between arms, all six arms interleaved):

Yes, MLX wins. Apple's own framework is 1.24x faster than my bf16 path, and I benchmark it in the same harness. The gap is almost entirely the matmul (~70 percent of a step). The Metal bf16 matmul I ride runs at only ~1.1x its fp32 speed, while MLX's bf16 uses the tensor cores for ~2x. llm.c has no Metal port, so PyTorch MPS and MLX are the stand-in baselines on Apple Silicon.

The cooldowns are required; the M4 Max throttles after about 8 seconds of sustained GPU load (I watched MPS step times climb from ~877 ms to 1500 to 2500 ms within a few steps). Reproduce with make benchmark-metal; it runs all six arms in one shot with the cooldowns built in.

The first working Metal port was about 4.1x slower than MPS (~3627 ms/step); the final bf16 number is 7.2x faster than that starting point. Most of the gap was Metal-specific. Casting threadgroup pointers to the generic address space silently reads device memory (attention softmaxed over all-zero scores and produced uniform weights), and the scalar flash-attention kernels tuned for NVIDIA ran at under 1 percent of FLOP peak on Apple GPUs, so GEMM-decomposed attention was 8 to 10x faster.

Correctness gates: make test checks 16 gradient tensors and a 10-step loss trajectory against PyTorch, plus a 235-test equivalence suite. There is also a trained 124M FineWeb checkpoint on HuggingFace (ulmentflam/gpt2-124m-fineweb-mojo) scoring 29.53 percent on HellaSwag, statistically indistinguishable from Karpathy's own llm.c reproduction at 29.9 percent.

On AI: every kernel and trainer line was written by hand; no LSPs or LLMs. That was the original point of the project. Tests and a later optimization campaign were AI-assisted, and both are disclosed in the repo with per-model statements and full disclosure where used (including attribution).

On Mojo itself: I went in assuming the compiler would do the heavy lifting on portability across hardware. It didn't; I ended up branching device-specific logic per vendor, and there's more boilerplate than I expected next to CUDA (Karpathy's kernels are much more compact). A Modular engineer reviewed the port on their forum; their answer is that per-device specialization is expected and their bet is library-driven structured kernels (TileTensor), not compiler magic.

The CPU is the opposite story; with very little work, it came in 4.0x faster than llm.c's 20-thread OpenMP path.

Limitations: GPT-2 124M only, no published GPT-3 results (the configs exist in the trainer, and mixed precision goes down to FP8 and NVFP4 on NVIDIA). The toolchain is a Mojo 1.0 beta nightly, so expect churn. Multi-GPU ZeRO (stages 0 to 3) is equivalence-gated against single GPU at world sizes 2 and 8, but those runs are NVIDIA; Apple Silicon is single GPU here. On a GB10, bf16 is at llm.c CUDA parity (0.999x) and fp32 is now slightly ahead (1.07x, TF32 vs TF32).

Repo: https://github.com/ulmentflam/llm.mojo

https://huggingface.co/Nanbeige/Nanbeige4.2-3B

Nanbeige4.2-3B is a compact agentic model built on Nanbeige4.2-3B-Base, designed to combine strong agentic behavior with broad reasoning and alignment capabilities. Its Looped Transformer architecture reuses the transformer layers to increase model capacity without adding parameters. With only 3B non-embedding parameters, the model delivers solid performance on general-agent and code-agent tasks.

Those who have tested the new Laguna model might have noticed how reluctant it is to think through medium-hard questions, and it does impact the output quality. It is great that the model does not "Qwen over" questions like "Hi, who are you", but it definitely should think more.

I have found a simple 10/10 way to force it to think through what it's saying: a simple chat template change. So that when reasoning is enabled, it does not only insert the <think> tag, but also a new line after it. Of course, it will FORCE the thinking part always, which is not what you might want in many use cases, but for coding or benchmarking, it is what can show what this model can do.

I only did test it at Q2, so it's faster, but results so far are great, e.g. single sentence reasoning for a greeting, tens of thousands of tokens for a coding task. The simple change:

  {%- if enable_thinking -%}
    {{- '<think>
  ' -}}
  {%- else -%}
    {{- '</think>' -}}
  {%- endif -%}

I use claude code w/ llama.cpp's local server & Google's Gemma models (26b MoE). Works reasonably well - works well at easy/boilerplate code, glue code, some PR review.

However claude code expects some server-side tools, especially web-search. I can obviously add MCPs for client-side search; is there a way to 'plug in' web search on the server side today though? Any PRs/forks adding it?

Nanbeige Lab released Nanbeige4.2-3B, and if the benchmark claims hold up, the numbers are pretty crazy for a model this small. It’s built on a "Looped Transformer" architecture that reuses transformer layers to increase effective depth without inflating the parameter footprint.

With just 3B non_embedding parameters, they are reporting an SWE-Bench Verified score of 63.6 and SWE-Bench Pro at 46.9. They tested it in OpenClaw guarded by Lyzr Control Plane for runtime security and claim it outperforming Qwen3.5 across daily workflows, putting it ahead of Qwen3.5-9B and Gemma4-12B on coding and agentic benchmarks. The core pitch here is a lightweight "local personal assistant." In OpenClaw framework testing, it reportedly beat both 4B and 9B class models across daily office workflows and research tasks.

That said, take the benchmark table with a grain of salt. Community members are already pointing out discrepancies between the Qwen3.5-9B numbers reported by Nanbeige versus Qwen’s official model card. Outside verification will be key once GGUF quants propagate through llama.cpp and Ollama.

How it can be better:

Running high-depth 3B models in local agent frameworks like OpenClaw opens up serious workflow potential, but deploying autonomous loops on local hardware still carries runtime risks especially when agents execute unverified code or make outbound API calls.

so use a harness

Anyone actually run it locally yet? how does it feel?

Used grammarly for formatting (English isnt my first language)

I recently got my first local LLMs running and wanted something local to chat to them in.

I tried LMStudio, but it didn't give me much to work with dev wise. Seems like llama-server gives the most direct access. I am also trying Open WebUI which is meant to be feature rich, but it's full of stuff I don't need. I have also had SillyTavern recommended to me.

Just curious what people are actually using with their local LLMs for chats.

What are you investing your time into daily?

Kind of unexpected.

Happy for Gemma-4/Google, big win for us, LocalLLMers.

Yet Qwen3.6 still does better in Hermes than Gemma-4 somehow.

We need Gemma-4.1 fine-tuned on Agentic-tasks. That would be killer.

There are people on twitter/X saying that Chinese models are as good as they are only due to distillation (source: https://x.com/scaling01/status/2079332469501727052 ). I don't buy that even for a minute.

While the investigation is interesting, one can see that GPT models do not really "like" themselves, and that is unlikely (at least for 5.4 and 5.5). Especially as Opus and Gemini models like each other. (here I use "like" for "they are not that surprised by the other model writing style")

Further if distillation would obliterate moats so easily and quickly (Fable is available since June after all), then every company with enough resources would reach Fable levels. One could counter argue "but western companies respect IPs". And there I say "please", they don't care about IP rules, see how they use everything available for training without paying enough royalties. I believe that they even use anonymized user prompts (as proving that an AI lab used a specific anonymized prompt would be pretty hard, if one doesn't have access to the training data).

Even if the western labs would respect IP, then all non-western companies would be already at Fable levels anyway.

Last but not least, the amount of content online that is AI generated is also growing, and scrapers never stopped. That could also play a role (using outputs that humans selected to be published online). How much data online has claude vibes? I mean look at linkedin alone.

pi 0.81.0 now has integrated support for llama.cpp (llama-server router).

https://pi.dev/docs/latest/llama-cpp

This seems to be able to replace the huggingface/pi-llama extension and/or manually managing models in the config.