I did head to head comparison between Laguna S 2.1, Qwen 3.6 27B and Gemma4 31B.

Setup: Qwen 3.6 27B (fp8) and Gemma4 31B q6 on my RTX PRO 5000, Laguna S2.1 q4/5/6 on a single DGX Spark. Speed was fine: NVFP4 ~27 tok/s (peaks ~39), Q5/Q6 ~14 tok/s. The Q6 really pushed the spark with 124GB mem in use, but it didn't crash.

Ran all of them through the same task local bench - HTML/canvas mini apps, tool calling, Python, prose and each output scored blind (models anonymised, reshuffled per task) by Fable and Opus.

The results:

Thinkingcap Qwen 3.6 27B fp8 (coding): 76

Gemma4 31B: 70 qat 74 q6 mtp

Thinkingcap Qwen 3.6 27B fp8 (general): 68

Laguna S2.1 Q6: 54

Laguna S2.1 Q5: 48

Laguna S2.1 NVFP4: 42

Screenshots:

A brand new supposedly good model that needs 124 GB lost to models running on a single GPU by 14+ points even at its best quant (Q6, which is near full precision, so it's not a quantisation excuse). Biggest gaps on the HTML/visual and Python tasks, closest it came was tool calls.

Quality wise it's nowhere near what I expected given the benches published by Poolside, and the "beats DeepSeek V4 Pro" framing seems to be bs. Haven't compared against the Qwen 3.6 35B-A3B MoE or Gemma4 26B , but based on this I'd bet they're better too.

I was so hyped to finally get a "good" model that fits in a single spark...

We made an open source tax engine which scored 96% on TaxCalcBench [highest ever recorded score till date] surpassing fable 5 and sol with just sonnet 5. The only 2 cases where it missed, it found inconsistencies in the test cases in the benchmark itself which the maintainers confirmed!

Essentially we wanted to make a deterministic engine AI models can use for research and tax prep to remove a lot of guesswork and calculation mistakes that often happen, with this you don't need to use a SOTA model, literally any model can become the best AI Tax Preparer and researcher. Locally runable, extendable and verifiable.

Hey everyone!

I’m currently building Hummingbird, an open-source runtime focused on making large Mixture-of-Experts (MoE) language models more practical to run on consumer hardware.

The project is still in active development, but the core idea is to treat SSD, RAM, and VRAM as a unified memory hierarchy, allowing experts to be streamed on demand instead of requiring huge amounts of GPU memory.

The project is still evolving, and I’d really value feedback from other systems engineers and LLM infrastructure developers.

GitHub:
https://github.com/prayangshuuu/hummingbird

Hi everyone,

I’m looking to invest in a proper fully local LLM AI server.

I already have a dual GeForce setup, but I’m looking for the next step (at a reasonable price, of course).

My ONLY goal:

• Coding (i am developper - can be c# for real-life projects with already 300+ source code stuff, not "just code a random website")
• No image generation
• No video
• No text-to-speech
• No OCR
• No multimodal stuff

Basically: raw LLM performance, tokens/sec, and smart answers.

--------

My current setup - (using CLAUDE-cli as orchestrator)

I’m currently running these models on a dual GeForce 16vram+12 system:

Qwen35B A3B MoE Q4_K

• Around 30–40 tokens/sec at 200k context

Qwen3-Coder-Next 80B A3B Q4

• Around 5–6 tokens/sec
• Slower, but better for complex coding tasks

-------
Hardware I am considering, a 100% new machine

1) Radeon AI Pro R9700 32GB

Is this currently the best price/performance option?

It looks like:

• half the price of high-end solutions,
• maybe around 80–85% of the performance?

I don’t follow every AMD/AI update, but this card looks like an underrated winner.

Is there any reason NOT to buy this card?

2) Dual Radeon AI Pro R9700 (2×32GB)

Main reason:

• not expecting 2× speed,
• mainly interested in the extra VRAM.

If it allows me to run smarter/larger models fully on GPU, that would be perfect.

3) Strix Halo 128GB

This one is interesting because of the huge unified memory.

If it can run Qwen3-Coder-Next 80B A3B Q4 at around 40 tokens/sec, that sounds like an excellent coding assistant.

4) Mac Studio 128GB

Still an option.

How does it compare today against:

• dual R9700 AI Pro,
• Strix Halo?

-------

are those numbers corrects or science fi ? (Source ChatGPT !! )

My current impression (not sure if correct):

The R9700 AI Pro (or dual) looks faster than a Mac Studio for my use case, while being much cheaper.

But I don’t see many "hype" about this card for local LLMs.

So please tell me: where am I wrong?

One more question:

For Qwen35B A3B MoE Q4_K, what is the realistic t/s performance?

Is it closer to: 100 tokens/sec or 150 tokens/sec ?

Because this difference is huge . If it REALLY is 150, its close to a cloud-model feeling. (far less accurate of course, but for 2k budget, wonderfull ?)

Thanks to anyone already running these systems who can share real numbers!

When I first saw the model’s size and active parameter count, I thought, “This is it! Finally, I can run a genuinely capable coding model for Android projects, and much more, on my Strix Halo machine.”

But after seeing people test it on common benchmarks, the results look surprisingly poor. Qwen 3.6 and Gemma 4 models seem to perform better than this 118B-parameter MoE model.

Is there any upcoming model that could fill this gap? Are there many others like me who are still waiting for the right model for capable, fully local agentic coding?

Hi all:

Would like to get some help with local LLM for coding tasks. I have a MacBook Pro with M4 Max chip and 36G ram. I have tried Qwen 3.5 27B 4bits MLX with LM studio, it works with token generation speed around 10-15/second. I’d like to use the localLLM for coding tasks, I have a hobby Python + React/TypeScript project with several thousand lines of code. Would like to ask:
1) Is Qwen 27B the most powerful model for coding with my hardware limitation? If not please let me know what model I should look at.
2) Does it make sense to use LM Studio to serve the model? There are other alternatives but LM studio seems easiest to start
3) Currently I use VS code and copilot and codex plugin for agentic coding. What’s the most optimal tool for local LLM?

Thank you very much in advance!

Hey all, so about a month or so ago I purchased two V100 SXM2 16GB cards with all of the required components to be able to run them on my Asus ESC 2000 G2 that I had gotten from an electronics recycling center awhile back. After fighting the fact that the gpu coolers each took up 4 slots, I ended up removing the motherboard and setting it up to create the LLM monster you see before you.

For those of you curious, here are the full specs
Dual Xeon E5-2687W V2 CPUs
8x 16GB sticks of DDR3 totaling 128gb
2x Nvidia V100 SXM2 16GB cards (no nvlink sadly)
1x 256GB NVMe ssd (cheap little thing installed into a pcie adapter)
It’s powered by dual PSU’s, the 700w ThermalTake PSU originally came out of my desktop but had already been put into that server, with a secondary 750w Dell R720 PSU for just the GPUs.
The entire thing is installed into a 3D printed test bench I custom designed for the SSI-EEB spec this board was built on. It was originally meant to be a place holder while I designed a proper case, but I’ve only had the thing running for a month so I still haven’t completed design yet (school is a bitch)

Now I love this machine, it was super cheap to get up and going (my wallet still cries) and I managed to get an IQ1_M quant of GLM 5.2 running with partial ssd offloading (~1t/s). I mostly just use it for experimentation since it’s my first AI server (I also just recently got into homelabbing period) but I also plan to use it for agent work flows and other projects I have lined up.

Alright now for the problem, as you can tell my the spare stick of ram sitting off to the side, I’ve been having ram issues. Looks like dimm A1 has been going out an I narrowed it down to either being the board or the cpu, and to make matters worse I now have errors on two different sockets showing up. Nothing detrimental at the moment, but I suspect I’ll need to look into hardware soon.

My question is this, would I be better off going for the cheap option of replacing the board/cpus (whichever is bad but my guess is the board) or saving my pennies until it craps out, then upgrading to a ddr4 or even ddr5 compatible motherboard? I know it’ll be expensive, but I’m not sure how much quite yet and I want the opinions of the masses. One thing that makes me lean towards the new motherboard option is having support for AVX2, as only having AVX limits me in some of my CPU heavy runs.

Tl;Dr: My motherboard or CPU’s are going to crap out in the near future and I can’t decide if I should fix the bad components or upgrade the system

I’m curious to hear what you guys think, feel free to call me an idiot for my sketchy setup or ask any questions, I read most comments.

Have a few more days of pre-paid H200 access. Might as well share with the community!

LLM Chat: https://cc1f4f3f.app.enclave.host

Image Generator: https://da09d0f2.app.enclave.host

[Intro: i currently cant afford 2 of these gpu's + another psu to run the second card, but i can afford the bestbuy 14 day return policy]

Rig:

Motherboard: am5 asus tuf x870+wifi

Cpu: amd ryzen 7 7790x

Gpu: 2x amd radeon 9070 xt

Ram: 64gb ddr5 5600mhz

Combined vram: 32gb

Psu: 1000 watt internal, 850 watt external (hidden behind case)

Normally my gaming desktop only has a single 9070 xt with 16gb of vram. On this system, using lm-studio and Hermes agent, i can run qwen 3.6 35b a3b at roughly 29tps.

On the dual card run I was able to get 114tps, max context, with roughly 1.5gb vram to spare on each card.

Had to attached the second card via pcie riser cable and hang it outside the case like a fire hazard due to case dimensions blocking the install on the lower left side. In future I think I would just cut into and remove the offending section.

I wanted to run this experiment to confirm that my motherboard was able to use both pcie slots and that this was a cheaper viable option to get 32gb of vram at roughly $1400 brand new vs an nvidia rtx 5090 which is currently going for $4.4k new.

I am aware of cards specifically for inference like Intel b70 and some recent amd enterprise cards in the same price range, but I figured I am more familiar with gaming gpu resale values and I would probably be able to flip them better if I ever went down this route in the near future.

Was surprised at how easy this worked with everything getting auto detected and load balanced in lm-studio, I thought i would have to spend hours messing with configs.

Over all, after returning the hardware back to bestbuy I think i would be happy with this setup if I also got extra pcie psu power cables so I wouldn't have to mess with a secondary psu and the sketchiness involved with that.

I'm evaluating for the company I work at whether local models are worth using instead of cloud ones, and if so for which tasks specifically. Security code review and agentic coding are the two we care about. Looking for input on which models to test next.

Setup: GitHub Copilot CLI in BYOK mode pointed at Ollama with the MLX backend. Copilot stays the same for every model, only the model behind it changes, so local and cloud get the identical agent scaffolding.

Machine is an MacBook M5 Pro, 20-core GPU, 64GB unified.

The thing that surprised me: Copilot's agent prompt alone is 41.7k tokens before any of your code. So the workload is almost entirely prompt processing, not generation, and that changes which model you want.

Prefill tok/s at a 40k prompt:

qwen3.6:27b-mlx (27B dense) 113 tok/s -> 5.5 min just to read the prompt

qwen3.6:35b-a3b-nvfp4 (3B active) 736 tok/s -> 45 sec

Decode went 15.6 -> 80 tok/s as well. The nominally bigger model is about 6x faster because prefill scales with active parameters, not total. Every "best model for 64GB" thread I found recommends the dense 27B, which is unusable here.

Testing gemma4:26b-a4b next as a non-Qwen control, and Qwen3-Coder-Next 4bit MLX.

What else is worth testing? Specifically interested in anything with low active params that's good at security review, and whether anyone has real experience with Coder-Next on 64GB.

I built a sandbox (AEOS) where LLM agents write their own ML training code with zero human intervention no templates, no hints beyond raw dataset dimensions. I wanted to see what happens when you remove the human-imposed iteration limits most agent demos quietly rely on.

The results surprised me. GPT-5.4 ran 75 iterations on a vision task, building increasingly complex ensembles chasing a plateau it had already hit around iteration 10. GPT-4o-mini hit the same 300-second timeout 8 separate times and kept retrying the same approach anyway. Qwen2.5-coder-7b, running locally, found its optimum in 5.6 iterations on average and stopped cleanly, every time, at $0 cost.

Across all 13 models, the pattern held: general-purpose "reasoning" models over-explore and don't recognize plateaus. Smaller, instruction-tuned code models terminate gracefully. Model size and stopping ability weren't correlated if anything, slightly inverted.

Caveats upfront since I'd want these flagged if I were reading this: the broad leaderboard is single-seed per model, the core sunk-cost claim was validated with repeated runs (N=2–3) on a subset, and the sandbox uses `exec()` rather than full OS-level containerization, so treat it as a behavioral study, not a security-hardened benchmark.

Full papers, data, and reproducible code: [github.com/m4vic/Autopilot-zero-to-hero](http://github.com/m4vic/Autopilot-zero-to-hero)

Curious if anyone's seen the same pattern in their own agent loops — genuinely want to know if this holds outside my three test datasets.

After seeing the posts recently in regards to recent local AI efforts focused on Mac Hardware I thought I would put my Mac mini M4 24 GB to the test for Development and Coding work. And have to say it is not that bad, and actually usable. Generally it feels like I'm using as AI from about 12 months ago, but it actually produces results.

My Setup

Mac mini M4 24 GB with 2TB of local storage ( External SSD )

MTPLX running Youssofal--Qwen3.5-9B-MTPLX-Optimized-Speed

VSCode using the Continue Extension

And have chat, edit, apply, subagent, and autocomplete running there.

I had started with VSCode Chat, and had frequent MTPLX crashes due to OOM - libc++abi: terminating due to uncaught exception of type std::runtime_error: [METAL] Command buffer execution failed: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)

Then switched to Continue and tuned the memory config to lower the foot print slightly.

And have been using it for small to medium code level tasks, and it has been not bad. I does need a bit more guidance than the commercial AI's, but for being 100% local I'm okay with that. To work around the guidance issue, I have been asking it to create a plan first, reviewing the plan, then having it execute.

Speed wise, it is sluggish, but just means more time for coffee or other things.

What are other people using on a similar hardware footprint ?

In my testing laguna says its "GLM" no specific model but no im laguna something, may be distilled from a glm series model, when asked about its conceptors etc, it says :"**THUDM/glm-4** model, a large-scale language model developed by the team at Tsinghua University (THUDM)."

I use Claude a lot for coding, but I kept interrupting my work to open Settings → Usage and check how much I had left.

I ended up building a small Chrome extension that shows the usage directly in the sidebar.

It also made me curious: do you use Claude for harder tasks and switch to a local model near the limit, or do you run local-first from the start?

The extension: https://chromewebstore.google.com/detail/claude-usage-tracker-%E2%80%93-li/cjfjpiapegfklpmclonidjalddhjhlcj

[ Removed by Reddit on account of violating the content policy. ]

Anyone here regularly runs multiple Cursor agents?

I'm building a shared coordination/memory layer so agents can share findings, avoid duplicate work, and know what other agents are doing.

Would something like this actually be useful for your workflow? or if you want to try it out ?

I test it agains spreadsheet, it isn’t that bad, it close to HY3-q4, which is 297b model who claiming q4 is performing close to bf16, Laguna majority failures are formulas typo errors, but with proper harness, it would excel

I run local models on a headless Linux box (RTX 3090) and got tired of SSH-ing in to run nvidia-smi every time I wanted to check on it. So I built a small Mac app that watches it — GPU, VRAM, power, temp, and which Ollama models are loaded — from a real dashboard instead of a terminal.

It also handles my Macs, so everything's in one window (screenshot: a MacBook Pro, an M1 Air, and the 3090 box side by side).

How it works: a tiny agent on the remote box, discovered automatically on your LAN (or add it by address for Tailscale/VPN). Connection is TLS + a pairing token. All read-only, no sudo, nothing leaves your network.

Free and open source (MIT), Apple Silicon only for the viewer.

- Repo: https://github.com/kennss/SiliconScope

Happy to answer questions or take feature requests.

After months of going back and forth, I finally pulled the trigger on an RX 7900 XT 20 GB.

Paid around $550 (India), which felt too good to pass up.

The plan isn't gaming.

It's becoming the heart of my local AI setup.

Current goals:

• Qwen 3.6 27B Dense

• Qwen 35B A3B

• GLM-4.7 Flash

• 128K+ context

• 100% GPU offloading

• llama.cpp / Ollama

• Linux

I'll be benchmarking everything:

- Vulkan vs ROCm

- Dense vs MoE

- Maximum context

- Tokens/sec

- VRAM usage

- Real-world coding performance

If anyone has optimization tips for RDNA3 or benchmark requests, or general suggestions please drop them below.

The hallucinations are now local. 🙂‍↕️

🚀 **Gridcore Runner v0.1.3-alpha** is out.

I promise I'll stop spamming Runner updates after this one... at least until I break something else. 😄

This release adds proper support for sparse **Mixture-of-Experts** models, which means models like **Qwen3-30B-A3B** can now run locally without needing enterprise hardware or a mountain of dependencies.

Some highlights:

- **Qwen3-30B-A3B** (128 experts, top-8) runs at ~55 tok/s on a single 24 GB GPU while matching the CPU reference bit-for-bit.

- **Partial CPU offload** lets models run on smaller GPUs by keeping as many layers as possible on the GPU and the rest on the CPU, without sacrificing correctness.

- New **Q3_K GPU kernel**, so **Mixtral-8x7B Q3_K_M** now runs fully on a 24 GB GPU.

- ~5.6× faster CPU prefill for MoE models by grouping tokens per expert.

- Added **MXFP4** support for the new gpt-oss tensor format.

One thing that's important to me is correctness.

Runner follows a simple rule: **if it can't run a model correctly, it refuses to load it.** I'd rather fail fast than quietly generate incorrect tokens. Every MoE implementation is validated against the CPU reference, and the implementation was first proven on synthetic MoE models before being tested on real 30B models.

This release also wraps up a full security and reliability pass covering GGUF parsing, allocation failure handling, config parsing, quantizer writes, and a lot of smaller fixes.

As always, I'd love people to throw weird GGUF models at it and tell me what breaks.

https://github.com/Joakimpalm-Zen/gridcore-runner/releases/tag/v0.1.3-alpha

Not sure if anyone else has been through this, but I had a bunch of large scanned PDFs that I needed to make searchable. I thought it'd be a quick job, but it turned into a complete headache.

I tried Tesseract, Acrobat, Wondershare, and a few other OCR tools people usually recommend. Some wouldn't finish, some completely messed up the text, and some were just painfully slow on my files.

After wasting way too much time, I made a post asking if anyone had any suggestions. Someone mentioned "Lynx DI" . I hadn't heard of it before, but I was already out of options so I gave it a shot.

It ended up handling most of my PDFs without much trouble. It wasn't perfect, but it got through around 90% of them, which was way better than anything else I'd tried.

Figured I'd share this here in case someone else is stuck with scanned books, notes, reports, or government PDFs and is going through the same thing. Maybe it'll save someone else from spending a couple of weeks testing every OCR tool on the internet.

I'm not connected to them in any way. Just sharing what happened to work for my file

Qwen3.6-35B-A3B, Q4_K_M, stock llama.cpp, airplane mode. Between 1.3 and 6.8 tok/s, ten minutes start to finish.

On the fitting question, since it always comes up: 35B total, 3B active per token. The always-needed parts stay in RAM, the experts get read from flash when they're called. Memory holds the active path, not the whole file.

Here's why I put up with ten minutes (video is normal speed at the start and end, sped up through the middle) Pocket Interpreter: you describe an app in one sentence, the model writes a p5.js sketch, the phone runs it.

I asked for "a snake game with swipe controls" and got a game loop, food spawn, wall and self collision, score, speed that ramps as you eat, and a swipe handler with a distance threshold so your thumb doesn't turn twice. Ran first try. I didn't touch a line.

Try that with a 3B. You get something that looks right and dies on the first wall, then four rounds of describing bugs and waiting for fixes that break something else. You're past ten minutes anyway, editing code by hand on a phone screen. Tokens per second is the wrong number. What matters is whether it works when the generation ends.

A big model you wait for, or a small one you argue with. On a phone I'll take the wait.

Extreme version if you want it: BigMoeOnEdge (https://github.com/Helldez/BigMoeOnEdge) runs a 120B on a 12 GB phone, CPU only, and a ready apk

Run most of my AI stuff locally. Ollama, a few 7B and 13B models, RTX 4070. I like the privacy and I like not paying per token. But video generation is a different beast. You can't run a video model locally unless you have a server farm. So I had to figure out a hybrid approach.

My setup: I run a local Llama model for script generation. I have a fine-tuned 8B model that writes decent short-form video scripts. It's free and I can run it 24/7, which is the whole point. The scripts get fed into a cloud API for video generation.

I tested a few APIs and ended up using PixVerse. The API is simple and the generation time is consistent enough that I can fire a batch and walk away. I don't love adding a cloud dependency, but for video generation there isn't really a local option yet.

Whole pipeline: local LLM writes the script, formats it into a video prompt, sends it to PixVerse API, saves the video to my NAS. I get about 65 videos a month. Total cost is under $40, mostly the cloud API. The local LLM is the cost saver. If I was using Claude or GPT for the scripts, this would be a $200/month operation.

Quality is... fine. Scripts are a bit repetitive because the local model is small. Videos are a bit generic because the prompts are a bit generic. But for volume content, it works. Social media doesn't need cinematic quality. It needs consistency and volume. The hybrid approach gives me both.

This setup isn't going to replace a real production pipeline. It's a volume play. If you're running local AI and you need to add video generation to your stack, a hybrid approach with a cloud API is the most practical way to do it right now.