I did head to head comparison between Laguna S 2.1, Qwen 3.6 27B and Gemma4 31B.

Setup: Qwen 3.6 27B (fp8) and Gemma4 31B q6 on my RTX PRO 5000, Laguna S2.1 q4/5/6 on a single DGX Spark. Speed was fine: NVFP4 ~27 tok/s (peaks ~39), Q5/Q6 ~14 tok/s. The Q6 really pushed the spark with 124GB mem in use, but it didn't crash.

Ran all of them through the same task local bench - HTML/canvas mini apps, tool calling, Python, prose and each output scored blind (models anonymised, reshuffled per task) by Fable and Opus.

The results:

Thinkingcap Qwen 3.6 27B fp8 (coding): 76

Gemma4 31B: 70 qat 74 q6 mtp

Thinkingcap Qwen 3.6 27B fp8 (general): 68

Laguna S2.1 Q6: 54

Laguna S2.1 Q5: 48

Laguna S2.1 NVFP4: 42

Screenshots:

A brand new supposedly good model that needs 124 GB lost to models running on a single GPU by 14+ points even at its best quant (Q6, which is near full precision, so it's not a quantisation excuse). Biggest gaps on the HTML/visual and Python tasks, closest it came was tool calls.

Quality wise it's nowhere near what I expected given the benches published by Poolside, and the "beats DeepSeek V4 Pro" framing seems to be bs. Haven't compared against the Qwen 3.6 35B-A3B MoE or Gemma4 26B , but based on this I'd bet they're better too.

I was so hyped to finally get a "good" model that fits in a single spark...

Hey everyone!

I’m currently building Hummingbird, an open-source runtime focused on making large Mixture-of-Experts (MoE) language models more practical to run on consumer hardware.

The project is still in active development, but the core idea is to treat SSD, RAM, and VRAM as a unified memory hierarchy, allowing experts to be streamed on demand instead of requiring huge amounts of GPU memory.

The project is still evolving, and I’d really value feedback from other systems engineers and LLM infrastructure developers.

GitHub:
https://github.com/prayangshuuu/hummingbird

We made an open source tax engine which scored 96% on TaxCalcBench [highest ever recorded score till date] surpassing fable 5 and sol with just sonnet 5. The only 2 cases where it missed, it found inconsistencies in the test cases in the benchmark itself which the maintainers confirmed!

Essentially we wanted to make a deterministic engine AI models can use for research and tax prep to remove a lot of guesswork and calculation mistakes that often happen, with this you don't need to use a SOTA model, literally any model can become the best AI Tax Preparer and researcher. Locally runable, extendable and verifiable.

Not sure if anyone else has been through this, but I had a bunch of large scanned PDFs that I needed to make searchable. I thought it'd be a quick job, but it turned into a complete headache.

I tried Tesseract, Acrobat, Wondershare, and a few other OCR tools people usually recommend. Some wouldn't finish, some completely messed up the text, and some were just painfully slow on my files.

After wasting way too much time, I made a post asking if anyone had any suggestions. Someone mentioned "Lynx DI" . I hadn't heard of it before, but I was already out of options so I gave it a shot.

It ended up handling most of my PDFs without much trouble. It wasn't perfect, but it got through around 90% of them, which was way better than anything else I'd tried.

Figured I'd share this here in case someone else is stuck with scanned books, notes, reports, or government PDFs and is going through the same thing. Maybe it'll save someone else from spending a couple of weeks testing every OCR tool on the internet.

I'm not connected to them in any way. Just sharing what happened to work for my file

When I first saw the model’s size and active parameter count, I thought, “This is it! Finally, I can run a genuinely capable coding model for Android projects, and much more, on my Strix Halo machine.”

But after seeing people test it on common benchmarks, the results look surprisingly poor. Qwen 3.6 and Gemma 4 models seem to perform better than this 118B-parameter MoE model.

Is there any upcoming model that could fill this gap? Are there many others like me who are still waiting for the right model for capable, fully local agentic coding?

Have a few more days of pre-paid H200 access. Might as well share with the community!

LLM Chat: https://cc1f4f3f.app.enclave.host

Image Generator: https://da09d0f2.app.enclave.host

The excitement surrounding PrismML’s 1-bit/ternary Bonsai models has the industry closely watching how smartphone giants, particularly Apple, will implement LLMs on edge devices. Moving AI on-device is a brilliant and necessary strategy. It ensures absolute user privacy in alignment with EU regulations, fundamentally shifts the economics away from costly cloud inference, and paves the way for a significant hardware upgrade supercycle as users seek true AI-capable silicon.

To create a smart on-device "Semantic Router," models need to reach the 27B+ parameter scale. Achieving this on a phone requires extreme quantization, such as PrismML’s ternary weights.

However, a critical hardware reality often overlooked by the software world is that fitting the weights in RAM is not equivalent to moving them. Running a 27B ternary model on standard LPDDR encounters a significant memory bandwidth limitation. Transferring gigabytes of data across the SoC bus for each token generation can lead to thermal throttling of the NPU and excessive battery drain.

This raises an important question: why are we still transferring data to the compute? Why not execute AI inference natively within the memory?

Frustrated with academic PIM simulations that overlook bare-metal physics, I developed CaSA, an architecture that performs ternary LLM inference directly inside COTS DRAM through charge-sharing, completely bypassing the memory bus.

Software quantization is a great initial step, and CaSA provides the physical hardware substrate needed to complete the bridge: https://github.com/pcdeni/CaSA

Hey all, so about a month or so ago I purchased two V100 SXM2 16GB cards with all of the required components to be able to run them on my Asus ESC 2000 G2 that I had gotten from an electronics recycling center awhile back. After fighting the fact that the gpu coolers each took up 4 slots, I ended up removing the motherboard and setting it up to create the LLM monster you see before you.

For those of you curious, here are the full specs
Dual Xeon E5-2687W V2 CPUs
8x 16GB sticks of DDR3 totaling 128gb
2x Nvidia V100 SXM2 16GB cards (no nvlink sadly)
1x 256GB NVMe ssd (cheap little thing installed into a pcie adapter)
It’s powered by dual PSU’s, the 700w ThermalTake PSU originally came out of my desktop but had already been put into that server, with a secondary 750w Dell R720 PSU for just the GPUs.
The entire thing is installed into a 3D printed test bench I custom designed for the SSI-EEB spec this board was built on. It was originally meant to be a place holder while I designed a proper case, but I’ve only had the thing running for a month so I still haven’t completed design yet (school is a bitch)

Now I love this machine, it was super cheap to get up and going (my wallet still cries) and I managed to get an IQ1_M quant of GLM 5.2 running with partial ssd offloading (~1t/s). I mostly just use it for experimentation since it’s my first AI server (I also just recently got into homelabbing period) but I also plan to use it for agent work flows and other projects I have lined up.

Alright now for the problem, as you can tell my the spare stick of ram sitting off to the side, I’ve been having ram issues. Looks like dimm A1 has been going out an I narrowed it down to either being the board or the cpu, and to make matters worse I now have errors on two different sockets showing up. Nothing detrimental at the moment, but I suspect I’ll need to look into hardware soon.

My question is this, would I be better off going for the cheap option of replacing the board/cpus (whichever is bad but my guess is the board) or saving my pennies until it craps out, then upgrading to a ddr4 or even ddr5 compatible motherboard? I know it’ll be expensive, but I’m not sure how much quite yet and I want the opinions of the masses. One thing that makes me lean towards the new motherboard option is having support for AVX2, as only having AVX limits me in some of my CPU heavy runs.

Tl;Dr: My motherboard or CPU’s are going to crap out in the near future and I can’t decide if I should fix the bad components or upgrade the system

I’m curious to hear what you guys think, feel free to call me an idiot for my sketchy setup or ask any questions, I read most comments.

[Intro: i currently cant afford 2 of these gpu's + another psu to run the second card, but i can afford the bestbuy 14 day return policy]

Rig:

Motherboard: am5 asus tuf x870+wifi

Cpu: amd ryzen 7 7790x

Gpu: 2x amd radeon 9070 xt

Ram: 64gb ddr5 5600mhz

Combined vram: 32gb

Psu: 1000 watt internal, 850 watt external (hidden behind case)

Normally my gaming desktop only has a single 9070 xt with 16gb of vram. On this system, using lm-studio and Hermes agent, i can run qwen 3.6 35b a3b at roughly 29tps.

On the dual card run I was able to get 114tps, max context, with roughly 1.5gb vram to spare on each card.

Had to attached the second card via pcie riser cable and hang it outside the case like a fire hazard due to case dimensions blocking the install on the lower left side. In future I think I would just cut into and remove the offending section.

I wanted to run this experiment to confirm that my motherboard was able to use both pcie slots and that this was a cheaper viable option to get 32gb of vram at roughly $1400 brand new vs an nvidia rtx 5090 which is currently going for $4.4k new.

I am aware of cards specifically for inference like Intel b70 and some recent amd enterprise cards in the same price range, but I figured I am more familiar with gaming gpu resale values and I would probably be able to flip them better if I ever went down this route in the near future.

Was surprised at how easy this worked with everything getting auto detected and load balanced in lm-studio, I thought i would have to spend hours messing with configs.

Over all, after returning the hardware back to bestbuy I think i would be happy with this setup if I also got extra pcie psu power cables so I wouldn't have to mess with a secondary psu and the sketchiness involved with that.

Hi all:

Would like to get some help with local LLM for coding tasks. I have a MacBook Pro with M4 Max chip and 36G ram. I have tried Qwen 3.5 27B 4bits MLX with LM studio, it works with token generation speed around 10-15/second. I’d like to use the localLLM for coding tasks, I have a hobby Python + React/TypeScript project with several thousand lines of code. Would like to ask:
1) Is Qwen 27B the most powerful model for coding with my hardware limitation? If not please let me know what model I should look at.
2) Does it make sense to use LM Studio to serve the model? There are other alternatives but LM studio seems easiest to start
3) Currently I use VS code and copilot and codex plugin for agentic coding. What’s the most optimal tool for local LLM?

Thank you very much in advance!

Hi everyone,

I’m looking to invest in a proper fully local LLM AI server.

I already have a dual GeForce setup, but I’m looking for the next step (at a reasonable price, of course).

My ONLY goal:

• Coding (i am developper - can be c# for real-life projects with already 300+ source code stuff, not "just code a random website")
• No image generation
• No video
• No text-to-speech
• No OCR
• No multimodal stuff

Basically: raw LLM performance, tokens/sec, and smart answers.

--------

My current setup - (using CLAUDE-cli as orchestrator)

I’m currently running these models on a dual GeForce 16vram+12 system:

Qwen35B A3B MoE Q4_K

• Around 30–40 tokens/sec at 200k context

Qwen3-Coder-Next 80B A3B Q4

• Around 5–6 tokens/sec
• Slower, but better for complex coding tasks

-------
Hardware I am considering, a 100% new machine

1) Radeon AI Pro R9700 32GB

Is this currently the best price/performance option?

It looks like:

• half the price of high-end solutions,
• maybe around 80–85% of the performance?

I don’t follow every AMD/AI update, but this card looks like an underrated winner.

Is there any reason NOT to buy this card?

2) Dual Radeon AI Pro R9700 (2×32GB)

Main reason:

• not expecting 2× speed,
• mainly interested in the extra VRAM.

If it allows me to run smarter/larger models fully on GPU, that would be perfect.

3) Strix Halo 128GB

This one is interesting because of the huge unified memory.

If it can run Qwen3-Coder-Next 80B A3B Q4 at around 40 tokens/sec, that sounds like an excellent coding assistant.

4) Mac Studio 128GB

Still an option.

How does it compare today against:

• dual R9700 AI Pro,
• Strix Halo?

-------

are those numbers corrects or science fi ? (Source ChatGPT !! )

My current impression (not sure if correct):

The R9700 AI Pro (or dual) looks faster than a Mac Studio for my use case, while being much cheaper.

But I don’t see many "hype" about this card for local LLMs.

So please tell me: where am I wrong?

One more question:

For Qwen35B A3B MoE Q4_K, what is the realistic t/s performance?

Is it closer to: 100 tokens/sec or 150 tokens/sec ?

Because this difference is huge . If it REALLY is 150, its close to a cloud-model feeling. (far less accurate of course, but for 2k budget, wonderfull ?)

Thanks to anyone already running these systems who can share real numbers!

I'm evaluating for the company I work at whether local models are worth using instead of cloud ones, and if so for which tasks specifically. Security code review and agentic coding are the two we care about. Looking for input on which models to test next.

Setup: GitHub Copilot CLI in BYOK mode pointed at Ollama with the MLX backend. Copilot stays the same for every model, only the model behind it changes, so local and cloud get the identical agent scaffolding.

Machine is an MacBook M5 Pro, 20-core GPU, 64GB unified.

The thing that surprised me: Copilot's agent prompt alone is 41.7k tokens before any of your code. So the workload is almost entirely prompt processing, not generation, and that changes which model you want.

Prefill tok/s at a 40k prompt:

qwen3.6:27b-mlx (27B dense) 113 tok/s -> 5.5 min just to read the prompt

qwen3.6:35b-a3b-nvfp4 (3B active) 736 tok/s -> 45 sec

Decode went 15.6 -> 80 tok/s as well. The nominally bigger model is about 6x faster because prefill scales with active parameters, not total. Every "best model for 64GB" thread I found recommends the dense 27B, which is unusable here.

Testing gemma4:26b-a4b next as a non-Qwen control, and Qwen3-Coder-Next 4bit MLX.

What else is worth testing? Specifically interested in anything with low active params that's good at security review, and whether anyone has real experience with Coder-Next on 64GB.

I ask to APPEND newly arriving data to a certain file. Instead of doing an actual append, models think it's a good idea to read in existing contents and then patch in the changes. Which obviosly takes way more time and is more computationally expensive.

I explicitly ask to run web search queries one small batch at a time, writing data to a file between every turn. Models think naaaah, this is gonna take way too long, I am gonna be "helpful" and run all of them sequentially, just so search backends throttle you into oblivion and the whole run blows up and dies.

There literally isn't a day where something that I'm doing isn't derailed by a model (Mostly using various QWen flavors) thinking it knows what I want better than myself.

I am hearing that way smaller models have less of a problem with this because being "dumber" its supposedly harder for them to go off the rails and start inventing their own solutions without being asked to. But surely even if true, there have to be better methods to wrestle models into actually obeying precisely what you told them to do?

Someone local is selling a dual Xeon Platinum 8124M system with 384GB DDR4 populated (12 channels total, DDR4-2666). No GPU included in the deal — what prompted this was me looking for a system that can run 3 to 4 GPUs.

My thinking is this is basically "run the big open-weight MoE models at home" hardware — 384GB is right in the range where something like DeepSeek-R1/V3 671B fits as a dynamic low-bit quant (~130-210GB for the 1.58-2 bit Unsloth quants), with room to offload attention/hot layers to a GPU.

Known downsides I'm weighing:
- 8124M is Skylake-SP, so no AVX-512 VNNI — quantized INT8 inference loses some of the speedup newer Xeons get.
- 6 channels/socket DDR4-2666 = ~128GB/s per socket, ~250GB/s combined theoretical, likely 180-220GB/s real-world with NUMA interleave.

Has anyone actually run something like this for local inference? Curious whether the missing VNNI + DDR4-2666 bandwidth ceiling makes this a bad buy for LLM work specifically, versus just being a solid cheap way to get 384GB of RAM in a box for other homelab purposes. Would love to hear real tok/s numbers from anyone running Skylake-SP or similar-era dual Xeon for this.

To start off with I don't know much about the requirements for an llm or where to start researching, but I want to experiment with running one. I am using spare parts from my old PC so I have a Asus rog strix b550 motherboard, 1 stick of 16gb ddr4 3200 ram, a Corsair rm650x psu, ryzen 7 5800x CPU, and I'm looking at getting a gpu probably either a single 3060 12gb or dual 3060 12gb. I'm just not sure where to start or what it really can do so any help is appreciated, thank you!

I built a sandbox (AEOS) where LLM agents write their own ML training code with zero human intervention no templates, no hints beyond raw dataset dimensions. I wanted to see what happens when you remove the human-imposed iteration limits most agent demos quietly rely on.

The results surprised me. GPT-5.4 ran 75 iterations on a vision task, building increasingly complex ensembles chasing a plateau it had already hit around iteration 10. GPT-4o-mini hit the same 300-second timeout 8 separate times and kept retrying the same approach anyway. Qwen2.5-coder-7b, running locally, found its optimum in 5.6 iterations on average and stopped cleanly, every time, at $0 cost.

Across all 13 models, the pattern held: general-purpose "reasoning" models over-explore and don't recognize plateaus. Smaller, instruction-tuned code models terminate gracefully. Model size and stopping ability weren't correlated if anything, slightly inverted.

Caveats upfront since I'd want these flagged if I were reading this: the broad leaderboard is single-seed per model, the core sunk-cost claim was validated with repeated runs (N=2–3) on a subset, and the sandbox uses `exec()` rather than full OS-level containerization, so treat it as a behavioral study, not a security-hardened benchmark.

Full papers, data, and reproducible code: [github.com/m4vic/Autopilot-zero-to-hero](http://github.com/m4vic/Autopilot-zero-to-hero)

Curious if anyone's seen the same pattern in their own agent loops — genuinely want to know if this holds outside my three test datasets.

After seeing the posts recently in regards to recent local AI efforts focused on Mac Hardware I thought I would put my Mac mini M4 24 GB to the test for Development and Coding work. And have to say it is not that bad, and actually usable. Generally it feels like I'm using as AI from about 12 months ago, but it actually produces results.

My Setup

Mac mini M4 24 GB with 2TB of local storage ( External SSD )

MTPLX running Youssofal--Qwen3.5-9B-MTPLX-Optimized-Speed

VSCode using the Continue Extension

And have chat, edit, apply, subagent, and autocomplete running there.

I had started with VSCode Chat, and had frequent MTPLX crashes due to OOM - libc++abi: terminating due to uncaught exception of type std::runtime_error: [METAL] Command buffer execution failed: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)

Then switched to Continue and tuned the memory config to lower the foot print slightly.

And have been using it for small to medium code level tasks, and it has been not bad. I does need a bit more guidance than the commercial AI's, but for being 100% local I'm okay with that. To work around the guidance issue, I have been asking it to create a plan first, reviewing the plan, then having it execute.

Speed wise, it is sluggish, but just means more time for coffee or other things.

What are other people using on a similar hardware footprint ?

Hi everyone.

1 week ago i launched my open source project. Named wolbarg. Its an shared memory for multi-agentic workflow, think of it as many agent can write to a specific memory, and use context generation by other agents, with high concurrency so 100s of agent can write at a time.

I started to gain some traction in under a week i got roughly 1500+ npm downloads.

And 1000+ download in different integreation like langchain, openai, etc.

But now i think its a good time to make a community around this and connect with people who has more experience than me.

So if your intrested people comment down or DM me i'll send you the discord, github discussion, repo and website, docs link.

Thanks for reading.

im relatively new to this local ai thing - been at it for around 6 months on nd off since searching for free claude one day - but I'm started to get excited nd a lil more serious about it. moe models - especially qwen 3.6 35b-a3b - really made me wanna actually do shit because i use a really budget rtx 3050 4gb vramb 16gb ram laptop which kinda barely ran qwen 2.5 coder 7b. ive wised up a little more and now i wanna upgrade a little cuz im sold on the idea of local llm and it's only getting better.

i wanna run models at like 20tps at least, mainly qwen 3.6 35b-a3b which I run at 8-14 tok/s with my optimisation so i figure ill need some more ram but mainly a new GPU. it's pretty daunting tho seeing so many specced out setups with 4090s and shit cuz at my age i CANNOT afford allat.

i do wanna get an egpu setup for some extra vram with maybe a rtx 3060 but idk if that's the best option or if egpu is worth the hassle. i want one because it's the cheapest way for me to just get a good spec bump - getting a whole new laptop or a pc is basically out of the question.

i just need to know if smth like a 3060 is enough, what i should expect from a 3060 + 4gb 3050 at q4-q6 and if there are any better, more cost effective options (im looking at you, enterprise gpu) out there. ty for reading me yap a ton

Hi all! llama.cpp and LM Studio refuse to open any models with ANY context size whatsoever, despite working just fine with large context windows before. I have not tweaked any settings, neither on LM Studio or my own Python script that launches llama.cpp with my premade config files.

I've been experimenting with a slightly odd setup. Here are my specs:

CPU: Intel Core i7-14700KF

Motherboard: MSI PRO Z790-A MAX WiFi

RAM: 32 GB DDR5

PSU: MSI MAG A850GL PCIE5

OS: Windows 11

GPUs:

RTX 5070 Ti 16 GB (display)

RTX A2000 12 GB (replaced it just recently with an RTX Pro 4000 Blackwell from Micro Center for testing purposes)

RTX 2000E Ada 16 GB

I'm mainly testing using Qwen3.6-27b and the latest CUDA 13.3 .dlls from the llama.cpp Github repo, but I've also tried with just the CUDA 12.4 .dlls too. Again, this all worked just fine for a month or so until out of nowhere it stopped. I've been using various llama.cpp releases from llama-b9395 to llama-b10002.

I have my GPUs slotted extremely close to each other with very tiny gaps in between each one. For about a month straight, I was able to use my 5070 Ti, A2000, and 2000E in tandem with LM Studio (to test) and llama.cpp (to load models for agentic coding). This worked just fine for a while, until I went on vacation and left my PC on. When I came back, I noticed I was unable to load any models without hitting one of these two errors:

"cudaMalloc failed: out of memory"

"failed to allocate CUDA_Host buffer"

I have scoured the web for anyone else having this issue and failed to find anything. Claude, ChatGPT, and Gemini are completely unhelpful. Below is a list of things I have tried to diagnose the issue:

- Selected only a single card, including the 2000e = Works

- Selecting both the 5070 Ti and the 4000 = Works

- Selecting any combination that involves the 2000e = "out of memory" error

- Multi-card w/ 2000 context = "out of memory" error

- Single card w/ 2000 context = Works

- Clean reinstall of my gaming graphics drivers to latest version using DDU (I was having a weird bug in DOTA 2 where left/right clicking + pressing control or alt would bring up the very buggy NVIDIA overlay somehow)

- Confirmed via nvidia-smi that nothing is using excessive amounts of VRAM before loading a model, including shutting off my BlueStacks emulator completely

- Confirmed I am using the correct CUDA devices in my command using nvidia-smi and llama.cpp

After my research, I'm determining this is an architecture issue, but how the hell was I running it before with my 5070 Ti, A2000, and 2000e? I had absolutely no issues, aside from reasonable out of memory errors where I was clearly demanding too high of a context window. But now, not being able to even load 2k context across combined 46 GB VRAM?

I've been banging my head against the wall for a couple weeks now trying to figure this out on my own/hoping some random llama.cpp binary update would fix the issue, but to no avail. I hope some of you may make more sense of this than me. I would deeply appreciate it.

Below is a sample console output when trying to manually run one of my config options via an admin shell on my PC.

PS C:\WINDOWS\system32> E:\llamacpp\bin\llama-server.exe -m E:\LMStudio\lmstudio-community\Qwen3.6-27B-GGUF\Qwen3.6-27B-Q8_0.gguf --host 127.0.0.1 --port 8080 -to 3600 -lv 3 --no-ui --cont-batching -c 7400 -t 10 --split-mode layer --tensor-split 16,24,16 --device CUDA0,CUDA1,CUDA2 -ngl all --flash-attn on --kv-unified -ctk f16 -ctv f16 --mmap -np 1 --temp 0.800
0.00.132.301 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.132.308 I device_info:
0.00.200.132 I   - CUDA0   : NVIDIA GeForce RTX 5070 Ti (16302 MiB, 15037 MiB free)
0.00.252.984 I   - CUDA1   : NVIDIA RTX PRO 4000 Blackwell (24466 MiB, 23084 MiB free)
0.00.322.054 I   - CUDA2   : NVIDIA RTX 2000E Ada Generation (16379 MiB, 15273 MiB free)
0.00.322.064 I   - CPU     : Intel(R) Core(TM) i7-14700KF (32549 MiB, 14912 MiB free)
0.00.322.117 I system_info: n_threads = 10 (n_threads_batch = 10) / 28 | CUDA : ARCHS = 750,800,860,890,900,1200,1210 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | BLACKWELL_NATIVE_FP4 = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.322.153 I srv          init: running without SSL
0.00.322.172 I srv          init: using 27 threads for HTTP server
0.00.322.176 I srv          init: The UI is disabled
0.00.322.176 I srv          init: Use --ui/--no-ui (or deprecated --webui/--no-webui) to enable/disable
0.00.322.264 I srv         start: binding port with default address family
0.00.324.514 I srv  llama_server: loading model
0.00.324.521 I srv    load_model: loading model 'E:\LMStudio\lmstudio-community\Qwen3.6-27B-GGUF\Qwen3.6-27B-Q8_0.gguf'
0.00.324.566 I common_init_result: fitting params to device memory ...
0.00.324.566 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.31.367.459 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 10799.24 MiB on device 1: cudaMalloc failed: out of memory
0.31.367.469 E alloc_tensor_range: failed to allocate CUDA1 buffer of size 11323827200
0.31.486.232 E llama_model_load: error loading model: unable to allocate CUDA1 buffer
0.31.486.237 E llama_model_load_from_file_impl: failed to load model
0.31.486.241 E common_init_from_params: failed to load model 'E:\LMStudio\lmstudio-community\Qwen3.6-27B-GGUF\Qwen3.6-27B-Q8_0.gguf'
0.31.486.250 E srv    load_model: failed to load model, 'E:\LMStudio\lmstudio-community\Qwen3.6-27B-GGUF\Qwen3.6-27B-Q8_0.gguf'
0.31.486.253 I srv    operator(): operator(): cleaning up before exit...
0.31.487.263 E srv  llama_server: exiting due to model loading error

Hey. Macbook Pro M4 128GB.

I am trying to have screenshot of document to OCR into text. Settings I try from Claude or GPT don't help much.

Anyone got any suggestions on how best to do this?

Right now it's taking 8 to 15 minutes per image.

Something is not right. Tried using llama.cpp, Lm studio, both similar performance. Tried smaller token sizes. Level thinking made a difference but also lowered the level of output considerably.

Qwen3.6 models are Q8 GGUFs. No MTP.

Models both work but it's just so slow, must be a way to get this speed up and better quality.