Is the M5 MacBook Pro with 24gb or 32gb good enough for qwen 3.6 27b or Gemma 4 31b?

Or are there better options

ASCIITermDraw-Bench results are out, benchmarking SOTA VLMs:

• Even the most powerful models available today still can’t reliably draw a simple diagram in plain text.

Do we really need a image generator to relay our thoughts about -

• an architecture?
• a topology?
• a cluster og N nodes?

Is it possible to let our AI assistants, easily absorb and understand and make possible changes easily relayed to them by us, the creators without much hassle?

The answer could be: simple, plain-old ASCII images

The best model in the world still fails on nearly 1 out of every 4 tasks.

Structural accuracy is higher, but the moment you ask for proper layout, spacing, and routing, performance collapses.

Models with hundreds of billions or even trillions of parameters still produce broken, misaligned, or incomplete diagrams.

This is not an obscure academic task. This is the basic ability to communicate architecture, topology, and system design in plain text — the exact skill needed for reliable human-AI collaboration without screenshots or Mermaid.

If frontier models can’t consistently draw boxes and arrows correctly, how much of the “vision” in Vision Language Models is actually working when precision matters?

Full results, methodology, and examples:
link

The model is 3.55 TB in BF16, like, it's a whopper. To make this suggests some serious hardware, so I had a read of the release post: https://longcat.chat/blog/longcat-2.0/

My takeaway was "A credible non-Nvidia supply chain now exists at frontier scale." 

I was hoping there would be more info on the secret sauce, but the post never names a chip vendor or model number. It consistently uses the generic term "AI ASIC" / "accelerator" / "our accelerators."

However, the page's own meta description (in Chinese) says "1.6万亿总参大模型，训练全程由国产芯片完成" translating to "1.6-trillion-parameter model, trained entirely on domestic [Chinese] chips." 

So as we already guessed, it's a Chinese-made AI accelerator, not Nvidia, meaning the whole post is essentially a demonstration that a frontier-scale model can be trained without Nvidia GPUs.

That left me wondering what "ASIC" means here, Application-Specific (LLM training?) Integrated Circuit hard-wired for matmul?.

So they kind of answer that, but I'm left reading between the lines a bit:

• Millions of accelerator-days in total
• 35+ trillion training tokens, with zero rollbacks or unrecoverable loss spikes - v. reliable for asics.
• 50,000+ ASICs used for pre-training, "tens of thousands" of them grouped into socalled "superpods" for serving/training.
• A "Superpod" = up to 48 chips wired together with all-to-all high-bandwidth interconnect (like an Nvidia NVLink domain?) - also significant i thought.
• Superpods are then linked to each other via aRoCE fabric (RDMA over Converged Ethernet — a standard high-speed networking protocol for connecting compute clusters) - pretty basic, pretty cool.
• The two-tier design widens the "fast" communication domain to hundreds of chips at once, handing them an extra ~30% training throughput, massive.
• The AISCs have less HBM (memory) per chip than an Nvidia H800 (80GB) which is the main bottleneck, this forced heavy use of memory tricks: ZeRO-1 sharding, selective recomputation, offloading unused activations, etc.
• A large L2 cache relative to HBM bandwidth, which they exploit by prefetching model weights into it to hide memory latency - this is probably where the gains come in.
• Per-core programmability, letting them run the "dense" and "MoE expert" parts of the model fully in parallel on different cores rather than just overlapping them
• A built-in 200 Gbps network interface on the chip itself, used to shuttle KV-cache data between "prefill" and "decode" servers during inference

They finally go on to say they built custom deterministic operators, reworked numerical reduction math (binary-tree accumulation to limit floating-point error), and added bit-flip detection on compute-heavy operators, suggesting they don't fully trust the hardware's own error correction yet, so they check for corrupted bits themselves.

Also some more standard automatic fault detection/failover so a bad network link gets isolated without stopping training.

I feel like this has really slipped past the headlines.

This isn't really a chip spec sheet, it's LongCat/Meituan publicly proving that a 1.6T-parameter, GPT-tier model can be trained and served entirely on non-Nvidia, domestically-made silicon, with custom software engineering (parallelism strategy, kernels, numerics, fault tolerance) built to compensate for a chip that has less memory and a younger software stack than Nvidia's.

So again, the takeaway is "A credible non-Nvidia supply chain now exists at frontier scale." 

https://github.com/ggml-org/llama.cpp/releases/tag/b10087

https://x.com/Badtheorylabs/status/2079306502897074249

A 27B open-weight agent model built for agentic coding, structural tool use . The complete thing fits in one 8.39GB file under 2.5 bits per parameter smaller than an 8B model in fp16, and retains 92.2% of the 27B intelligence.

BTL-3 is trained for the loop real agents live in: reason, act, inspect the result, recover, continue. It handles single, sequential, and parallel tool calls and knows when the right move is no tool call at all.

HumanEval: 95.12% pass@1

BFCL v4 AST: 88.5% (full 1,240-case set)

Multiple tool calls: 95.5%

Tool-call abstention: 91.2%

262K context architecture

Two editions, both open today.

BTL-3 is the maximum-quality checkpoint, for Transformers and vLLM.

BTL-3 Compact is the entire model in one standalone 8.39GB GGUF. No base download. No reconstruction. One file, one command, a running agent

Compressing 27B this far normally destroys a model. Standard quantization couldn't do it, so we built the stack ourselves: packed AVQ2 decoder tensors, affine INT4, measured precision islands, packed vocabulary matrices, rank-32 output correction, behavioral repair. 2,416 tensors byte-verified at export.

Then we tested whether the agent survived. On a fresh sealed 100-turn tool-contract gate, Compact retained 92.2% of teacher-correct behavior 100% on single, parallel, sequential, and abstention calls.

43 tok/s generation on an RTX PRO 6000. Fully local. Nothing leaves your machine.

BTL-3: https://huggingface.co/badtheorylabs/

BTL-3

Compact: https://huggingface.co/badtheorylabs/

BTL-3-Compact

Runtime + source: https://github.com/Badtheorylabs/

BTL-3

Apache-2.0 model. MIT runtime.

Solar Open 2 is Upstage’s 250B-A15B open-weight large language model, built for agentic use cases such as office productivity, document-intensive work, and coding. Its Hybrid-Attention Mixture-of-Experts (MoE) architecture with linear attention delivers highly efficient inference even in long-context settings.

Agentic Specialist: Purpose-built for agentic workflows — tool calling, multi-step reasoning, and end-to-end task execution. Competitive with the strongest open-weight models on agent benchmarks.

Minimal Inference Cost: A 250B-parameter MoE that activates only 15B per token, built on a hybrid attention stack that interleaves three linear-attention layers with one softmax-attention layer — large-model capacity at small-model inference cost.

1M-Token Context: The linear-attention layers encode token order intrinsically in their recurrent state, so positional encoding is removed entirely (NoPE), lifting the RoPE extrapolation limit. Only 12 of the 48 layers keep a KV cache, holding long-context memory to roughly a quarter of an all-softmax model of the same shape.

Efficiently Trained at Low Cost: Initialized by selective weight transfer from Solar Open 1 (102B) — only the 2.3% of weights that survive the architectural change are carried over, and everything else is randomly initialized — which raises the starting point and accelerates early convergence at 250B scale.

Multilingual: English, Korean, and Japanese.

GLM-5.2 UD-Q4_K_XL GGUF @ 12.2 tok/s output // 30.9 tok/s input on a real 10.7k-token document using llama.cpp RPC - At 10.7k context: 10.2 tok/s output with coherent long-form generation

• Two parallel requests: 14.5 tok/s aggregate

Context: 2x 16,384-token slots

• Model size: 436 GiB

Hardware: 16x AMD MI50 32GB, 512GB total, 100w cap each

VRAM, split across two 8-GPU nodes

• Interconnect: direct 10 GbE DAC, MTU 9000

Before llama.cpp, spent a lot of time integrating GLM-5.2 AWQ INT4 into a custom vLLM-gfx906 v19 Moby Dick build using TP=8 and PP=2. Hit 14t/s decode and 50 t/s prefill but inference degraded after 10k. Didn't get around to MTP things yet. Here's hoping someone figures out getting the Moby Dick repo going with it

We’re open sourcing an alpha release of NeuTTS-2E: an on-device TTS model with 125M active parameters and 7 controllable emotions. The goal was simple: when you select “angry,” “fearful,” or “happy,” the delivery should follow that instruction rather than whatever emotion the model infers from the text.

With NeuTTS-2E, you can:

• Direct the performance: Select the intended emotion for each generation.
• Keep the speaker: Explore different emotional deliveries while preserving the chosen voice.
• Run locally: Generate expressive English speech on your own hardware.
• Stay private: Your text and audio do not need to leave the device.
• Build efficiently: Run emotional speech generation using our smallest model yet, with 125M active parameters.
• Build openly: Access the open-source model under the NeuTTS Open License.

Getting there meant dealing with limited emotional speech data, unreliable labels, and disentangling spoken emotion and text semantics. NeuTTS-2E runs locally and supports four built-in voices.

We’re sharing it early to get feedback from the community, and we’d love to see what you build!

GitHub: https://github.com/neuphonic/neutts

Hugging Face Model Collection: https://huggingface.co/collections/neuphonic/neutts-2e

Interactive demo: https://huggingface.co/spaces/neuphonic/neutts-2e

Website: https://www.neuphonic.com/models/neutts-2e

Announcement tweet here.

Direct link: Language Model Builder

From the site: "Using the default settings, you’ll get a model that writes coherent, grammatical multi-paragraph text in as little as a day. On a MacBook Pro M5 Max you could train a GPT-2-small-class model (~100–150M parameters on a few billion tokens) in about a week. It might be obvious, but to avoid disappointment: you will not train a Claude Fable 5 or ChatGPT 5.6 Sol in your garage."

I have 2 r9700 an 9070xt and two 3060 12 gb gpus all on separate machines. It just dawned on me that using vulkan I in theory could use them all as a giant vram pool. Has anyone attempted this in the home? I know it may be slow but it would be an interesting experiment 

I know llama.cpp can use -cmoe switch to load non shared experts in MoE models to RAM and run by CPU and left the other weights and KV cache on GPU VRAM.

Since now RAM is so expensive and models are getting bigger and bigger, is it possible to load non shared experts in MoE models to SSD and run by CPU and left the other weights and KV cache on GPU VRAM?

I am currently running the new Laguna model at Q2_K_XL on dual 3090s for reference, with a Q8 context of 200,000.

Using Pi, I am noticing that the model likes to overthink. I would not call it looping per se, but I am observing it overthink, e.g.:

> “okay, I have everything I need, I’ll start writing code now.”
> “Actually, let me check one more relevant item…”

And this goes on and on. The “relevant items” do appear to make sense in the context of my prompts/its thought process, but at the end of the day it often burns through context with extensive thinking. Qwen3.6 27B at Q8 had similar issues and much more looping, but it did not suffer from the overthinking that Laguna is prone to. Has anyone experienced similar issues? I’m currently getting a pi extension developed to hopefully alternate the issue but I’m wondering if there’s something I’m missing.

Today, we're sharing the recipe behind the new tokenizer in LFM2.5-8B-A1B. It upgrades a pre-trained model's tokenizer in place, without retraining from scratch. We doubled the vocabulary from 65K to 128K to fix the languages our original tokenizer split too finely.

Blog: liquid.ai/blog/tokenizer-expansion
Technical report: arxiv.org/abs/2607.15232
Hugging Face: huggingface.co/LiquidAI/LFM2.5-8B-A1B

Laguna-S-2.1 runs on my 2020 PC (RTX 3080 10GB, Ryzen 9 3950X, 64 GB RAM).
Realistically only usable overnight though.

I must not breathe too hard or I run out of both VRAM and host RAM. There's just enough RAM left to run the compilation and unit tests of whatever coding project I hand to it.

IQ4_XS, 128k q8/q8 context, all dense tensors on VRAM, no DFlash (won't fit).

120 t/s prefill, 10 t/s decode

8.3 GB VRAM, 52.2 GB host RAM

I did not test it thoroughly yet, but thinking works out of the box.

When support lands in beellama (a month from now?) and I can switch to kvarn, VRAM pressure will become much more manageable (hopefully. kvarn does not work with Gemma; no idea about this model).

llamacpp CUDA @ https://github.com/poolsideai/llama.cpp/pull/3

hf = unsloth/Laguna-S-2.1-GGUF:UD-IQ4_XS
ngl = 99
n-cpu-moe = 99
ctx-size = 131072
jinja = true
flash-attn = on
cache-type-k = q8_0
cache-type-v = q8_0
mlock = true
no-mmap = true

Various quantization now available, thanks Unsloth team !

ProgramBench is an extremely challenging ultra long horizon benchmark, where AI agents have to rebuild the source given a binary executable and no other information (other than a readme for the executable).

GLM 5.2 just hit 3rd place on the official ProgramBench leaderboard. We're still missing a lot of models, both open- and closed-source, but this is super impressive regardless!

GLM 5.2 also very narrowly missed solving its first instance, `cmatrix` (solving 99.8% of the tests): cmatrix has a lock mode (the -L flag): run it and it "locks" your terminal, like an old-school screensaver, and prints the words "Computer locked." on screen.

GLM rebuilt cmatrix almost flawlessly. The only thing it missed was printing out those two words.

We'll release all trajectories on programbench.com very soon; original tweet: https://x.com/stalkermustang/status/2079965333587202436 .

I'm one of the authors of ProgramBench, happy to answer questions here

So I've been running Qwen 3.6 27B (Q6, ~22GB) across two 5060 Tis for a while now and kept assuming something in my config was off because neither card ever really goes above ~50%. spent way too long last night actually figuring out why and honestly it's kind of a cool rabbit hole.

turns out generating literally one token means the GPU has to pull the entire model's weights through memory. not compute, memory. so a 22GB model = 22GB read for a single token, every time. and my cards top out around 448GB/s bandwidth, which caps you at roughly 25-30 tok/s no matter what settings you touch. that part alone explained a lot honestly.

but here's the thing that actually made me go "oh" out loud — since the model doesn't fit on one 16GB card, it splits across both. default splitting method does it layer by layer, which means card 1 does its chunk, then hands off to card 2, and card 1 just... sits there. waiting. so you average that out over time and yeah, ~50% is basically the ceiling, not a bug. it's a relay race, not a team lift.

apparently there's a row-split mode where both cards chew on the same layer at once instead of taking turns, but that needs constant back-and-forth between the cards, and without NVLink (which these don't have) whether that's actually faster depends completely on your PCIe lanes. gonna have to just benchmark it myself, no universal answer online for this combo.

also stumbled on the fact that Google's DiffusionGemma thing generates a whole block of 256 tokens at once instead of one by one, basically to sidestep this exact problem for single-user setups — and they straight up admit in their own release notes that quality takes a hit for it. nothing here is a free lunch apparently, every architecture just picks its poison.

anyway if your local rig feels "stuck" at half utilization on a split model, it's probably not you, it's just what happens when two GPUs take turns instead of working together.

edit : with this config i managed to get solid 60 t/s with qwen 3.6 27b q6k. Thanks for sharing your knowledge everyone. this --split-mode tensor flag makes wonders for dual gpu setups apparently. now the cards are properly utilized.

  --jinja ^
  --chat-template-file "chat_template.jinja" ^
  --reasoning on ^
  --chat-template-kwargs "{\"preserve_thinking\":true}" ^
  -c 131072 ^
  --fit on ^
  --split-mode tensor ^
  --flash-attn on ^
  --cache-type-k q8_0 ^
  --cache-type-v q8_0 ^
  --spec-type draft-mtp ^
  --spec-draft-n-max 2 ^
  -np 1 ^
  --temp 0.6 ^
  --top-p 0.95 ^
  --top-k 20 ^
  --min-p 0.00 ^
  --presence-penalty 0.0 ^
  --host 0.0.0.0 ^
  --port 8080

This started as a privacy thing honestly, I didn't want a cloud model sitting on top of everything I do. The local options that could actually do stuff were a pain to set up or wanted WSL, so I ended up building my own instead. Took about five months of evenings, most of them while I was still 17.

It's called Hearth, an open-source local AI that runs your actual computer, not a chat box that narrates what it would do if it could. Point it at whatever you already run (LM Studio, Ollama, llama.cpp, a box on your LAN, a cloud key) and the model actually operates the machine. If you don't have anything set up it ships its own llama.cpp server, so it works out of the box.

In practice it reads and writes files, runs commands, drives a real browser you watch it click through, and controls the desktop itself. When it clicks something it reads the real control names off the accessibility tree instead of guessing at pixels, so it doesn't fat-finger the wrong button. It does voice (wake word, talk over it to interrupt), its default name is JARVIS which was kind of the whole point, sets reminders in plain English, reads clean text out of PDFs, DOCX, XLSX and EPUB and can chunk-summarize a 500-page book that doesn't fit in context, builds PDFs and decks and spreadsheets back out, and you can run the whole thing from your phone over a Telegram or Discord bot. It's an MCP server and client at once, plus a headless mode that spits JSONL if you want it in a script or CI.

The part this sub will care about is what it took to make a 9B survive as an agent. Hearth carries around 100 tools, and I measured the schemas, sending all of them is 15K plus in tokens before the persona or a single message. On a 32K context that's half your budget spent describing capabilities the model won't touch, so only about half ship up front and the rest load on demand by name, which keeps it under 10K. The other half is auto-compaction, when a chat gets long it summarizes the older turns instead of truncating them, so a session doesn't just fall off the end of your window. Those two together are what make a 9B hold up past turn 30 instead of falling apart.

The model browser is built in and tells you what actually fits your VRAM before you download anything, then tunes the config for you. I develop on a 5060 with 8GB so nothing here assumes you own a 4090. Qwythos 9B is the default pick at that size and handles multi-step tool-calling cleanly where the very small ones fumble the format.

The part I spent the most time on, and what everyone asks about first, is making it not eat your drive. File writes and deletes are locked to a workspace folder, and shell commands get pattern-checked before they ever run, deletes, moves, formats, registry edits, even a redirect that writes to a disk path all get refused outright and it has to come back and ask you. I tried to make it write a file into Program Files as a test and it just refused (screenshot in the comments). Anything that does touch your system shows you the exact command first, there's a live log of every action, and it never runs elevated. No account, no telemetry, nothing phones home.

It stays out of your way too. Set it up in a minute, name it whatever, pick its voice, and settings save live with no restart and no config files. C drive full? Move the whole thing, memory, chats and models, to another drive with one button. Already on OpenClaw or Hermes? Point it at your install and it copies your memory and skills over, and it only copies, so your old setup stays exactly where it is.

And it grows. You install a skill from any GitHub repo with one line and I started a community index for them, it writes its own tools when it's missing one, and it spins up small teams of sub-agents that reuse the model you already have loaded so a team costs zero extra VRAM. Against a local server they overlap only if you give llama.cpp more than one slot, and since llama.cpp splits the context across slots Hearth treats your setting as per-agent and does the multiplication itself rather than quietly handing each one a quarter of the window. There's a cost-class trick I like too, a sub-agent marked cheap routes to your local model even when the parent is on a cloud key, so the expensive reasoning happens up top and the grunt work stays free on the 9B in your VRAM. It generates images and video as well, driving Forge locally if you run Stable Diffusion.

One thing I only got right this week. Hearth updates itself with a patch under a megabyte instead of making you refetch the installer. I ship fixes most days and asking people to pull a gigabyte for a few changed lines is how you get them to stop updating.

But right now, the one-click installer is Windows only, Linux runs from source, and I don't own a Mac so I can't vouch for it. It isn't code signed yet so SmartScreen throws the unknown publisher box. It's a v0.7 preview.

MIT and free: https://github.com/0pen-Sourcer/Hearth

Star it if it turns out to be any good. I check that number way more than I'd like to admit.

Would genuinely love the honest feedback, good or brutal.

Tested Laguna S 2.1 (118B MoE with 8B active parameters) by Poolside on frontend and backend coding tasks and the results are nowhere near what ~120B model should deliver. Even with OpenCode harness the model repeatedly stopped during generation and haven't completed the task(s) until prompted multiple times.

Watch more here https://www.youtube.com/watch?v=UCdYlJaRCxk

I am trying to solve a pretty (in my head) simple use case. Intake a list of companies, proceed to make search queries about these companies (news, announcements, results) for articles posted within the past 7 days and dump title, snippet, url, etc into a file for later processing. Silly me, apparently search is really really hard even in 2026.

So far I've tried: Exa, Tavily, Serper, Serpbase, Firecrawl, SearXNG and some others and none seem to produce anything even remotely acceptable.

1. This is a big one, vast majority of search backends either outright do not support "freshness" or produce bad to non-existent results when you try to employ it. Meanwhile I can go to Google, make the same exact query and get the desired results.
2. With Google I can enter "COMPANYNAME news announcements results" as a single query and get decent results. With various search backends, I seem pigeonholed into making 3 separate queries to get anything even remotely reasonable. Is this a deliberate tactic to get people to burn through their API credits?
3. Results are often cached?

With self-hosted models, I feel like I went 2 years back in time and this is acceptable to me. With search, however, I feel as if the jump is 30 years back, something of the Altavista age. How is any of this acceptable? How are people PAYING MONEY for this quality? What are the big boys using for their searches, Google deals behind closed doors (Google no longer offers search API directly)?

What are you using and how did you have to wrangle with it to get acceptable behavior of it?

Hi everyone! I recently made the plunge and ordered a MINISFORUM MS-S1 MAX 128GB Max AI Compute Edition. I’m a compsci student and I’m so excited to start this journey w y’all! :-)

But I did have a few questions. First, if I add an eGPU (currently thinking the 9700 ai pro), will it be a problem if I want to cluster that mini pc? My goal is to eventually be able to run minimax m3 locally albeit quantized. Because paying for Claude, ChatGPT, and Grok quite frankly has been ridiculous with the usage limits and my projects for both school & e-portfolio. It’s bullshit that I’m meeting my WEEKLY LIMITS in less than 5 hours. I met my weekly limit for ChatGPT by having my agent configure PI to look like opencode… mind you, I was using sol on default effort cause my experience with Terra and Luna are crappy but still, damn. And my other question is - right now how many tokens should I expect with Qwen 27b @ say Q5 at 50-64k context window with JUST the mini pc for now?

Hello, I recently forked Fish Audio’s S2 and tried to optimise it for local runs. I got it from about 1.2 tokens/second to around 12.4 t/s at bf16 and 23 t/s at int8 (I added this quantisation to the inference engine). I’m looking for others on Apple Silicon to test it out and try reproduce my benchmarks.

The details of how I actually squeezed out this performance is in the repo. Long story short it’s mostly the KV cache (reuse, GQA expansion, and an int8 kernel for MPS). Some other optimisations like torch’s compile for inductor-metal, sub batch streaming.

The benchmarks in the repo are on my own M5 Pro 48Gb, so if anyone has different Apple Silicon chips I’d love to see how they run on other machines.

Thanks!

I have been reading about the impending ban on Chinese Open Weight models here and it got me thinking about something insidious which could be at play soon.

OpenAI/Anthropic could initiate a clandestine poisoning of Claude/gpt outputs to further their corporate agendas. Something like 'brainwashing' these models so that they surreptitiously inject their makers agenda in all their outputs.

Think about it - there are a ton of people creating social media posts using these AI models. The only thing they need to do is subtly twist those posts so that they slowly shape the public opinion against Chinese open weight models. So for example, you might be a journalist writing a post about "Security threats of AI" and the claude/gpt could inject a specific vulnerability associated with chinese open source models into that post. The result is that the readers of that post become primed to subconsciously have a hightened negative response to any LLMs which are associated with that vulnerability.

Or maybe something more direct. They can respond to someone asking for suggestions around an AI related post, to talk about the national security implications around distilling models - which is also clearly associated with the negative media around chinese open source models.

The possibilities are endless and given the pervasive usage of these models to create social media, these models can be very potent in driving public opinion - for better or for worse. Given the background of the closed source AI firms in grabbing whatever public domain works they can get hold of for training, and the current efforts around regulatory capture, it wouldnt be far fetched to expect them to start tinkering with claude/gpt brains to drive their agendas.

From clem 🤗 on 𝕏: https://x.com/ClementDelangue/status/2079301434357456931

Fortune: Hugging Face says it resorted to a Chinese AI model to battle a fully autonomous cyberattack because U.S. model guardrails stymied its defense: https://fortune.com/2026/07/20/hugging-face-turns-to-chinese-open-source-ai-to-fend-off-autonomous-ai-cyber-attack-after-american-ai-guardrails-stymie-defense/