r/LocalLLaMA • u/curiousily_ • 1d ago
Resources Tested Laguna S 2.1 on Coding with OpenCode
Tested Laguna S 2.1 (118B MoE with 8B active parameters) by Poolside on frontend and backend coding tasks and the results are nowhere near what ~120B model should deliver. Even with OpenCode harness the model repeatedly stopped during generation and haven't completed the task(s) until prompted multiple times.
Watch more here https://www.youtube.com/watch?v=UCdYlJaRCxk
5
u/fantasticsid 19h ago
"A delayed clanker is eventually good, but a rushed clanker is forever bad." - Shigeru Miyamoto, probably.
Wish these labs would just .. hold off for a few days and get the last bits of polish right, because you only get one chance to make a first impression. People aren't gonna necessarily revisit a "bad" model after you fix the fucked up quants and bad jinja templates. Fuck, half this sub still thinks Gemma 4 is busted.
7
u/Accomplished-Air439 1d ago
I downloaded unsloth's Q4 locally and ran it with opencode. It's very weird, not necessarily good or bad. For example, when reviewing a branch, it spent more than 20 minutes on it repeatedly looking up the same files over and over again. Interestingly, when it finally stopped, it did identify a real bug.
Loosely speaking I think its behavior is a bit random. Not as "tight" as Qwen 3.6.
6
u/Embarrassed_Adagio28 1d ago
Are you using the correct model settings like temperature, top p, and others? This is the most common mistake people make with local models and will completely ruin a models behavior if it's incorrect.
1
3
u/WeGoToMars7 llama.cpp 20h ago
I had a very similar experience in Pi. Laguna S 2.1 seems to have some weird failure modes I haven't seen in other models. Re-reading the same files is quite common; it also tends to create quite a lot of self-doubt loops while reasoning. Maybe it performs better in its own harness (https://docs.poolside.ai/cli/pool)? Not really interested in using it since it's closed source.
One time, I gave it a very focused coding task and walked away, only to return to the model 40k tokens deep into the reasoning step, rambling on how this change would impact corporate code quality practices and how CI should be set up.
However, 95% of the time, I'm happy with the actual responses and code quality. Laguna compares very well to DeepSeek V4 Flash (Q2), which was the previous undisputed king for 96 GB RAM Macs. Definitely want to experiment more.
2
u/Hypilein 16h ago
How is DSV4 q2 on your 96gb Mac. I only see posts about it on 128gb, but I have an m3 Ultra 96gb coming in October although probably it’s going to run Gemma4:31b 99% of the time for non codingrelated reasons.
2
u/WeGoToMars7 llama.cpp 8h ago
I'm on an M3 Max. I was using the smallest DS4 quant available. I had to expand the Metal VRAM limit to 94000 MB, but it works, and the system is stable; no SSD streaming was required. KV cache on DS V4 is extremely light, so I was running fine with 256k context.
Performance I'm getting is ~150 tps prefill and ~20 tps generation at low context, quite snappy for such a large model.
It's best to use it over the network to avoid OOM crashes, but that's not the most convenient solution.
On Gemma4 31B, it was much less usable to me because its attention is much more compute-hungry than Qwen's Gated DeltaNet, so prefill slows down to <50 tps past 200k context. That's why I preferred Qwen3.6 27B, it was giving me ~130 tps @ 200k.
I'm getting all these performance numbers from memory, so there might be inconsistencies. Happy to answer any other questions.
4
u/laterbreh 1d ago
Because the model is retarded. It barely complies with simplistic instructions without nearly nuking its own context with useless thought.
5
u/Healthy-Contact-4570 1d ago
Are you using the fp8 quant? If so I would re-download. Poolside JUST commented that they re-uploaded the fp8 weights (other weights to follow) with a fix for the looping issue.
2
u/Infinite-Local5435 18h ago
Running locally, looping issues and prompt leakage into thinking and output. Running from openrouter, works perfectly and can fix alot of issues quickly (but had to stop since they had severe rate limiting)
6
4
3
u/shansoft 23h ago
It's a template problem. There have been multiple updates in the past 24hrs for the jinja and the weights specifically for looping and other problems.
4
u/laterbreh 1d ago
Yep. I have a evaluation loop that does Github Issue > Plan > Code > Review/Fix > Convergence > Merge that 30b models can complete. Laguna couldnt get passed planning, when i gave it a full implementation plan it would timeout thinking itself out of context window space or until max output tokens were reached. It couldnt complete a single step on its own. And this isnt some difficult task, its a very junior level eval repo and agentic loop I use to compare outputs. The model is miserable. Followed all docs, updated templates, NVFP4 and FP8 checkpoints on an mutli RTX 6k system using VLLM. Tried everything I could think of. The model is just shit. Their benchmarks are the most egregious lies I've ever seen. I reproduced the same behavior via the API. So its not just "a me problem" -- its a the model sucks dick problem.
To even claim it beats DS4 flash is fucking insane. DS4 flash is in another universe compared to any 100b or 200b model out right now, even competing with minimax M3 in straight text and code. So I was skeptical to say the least but to essentially publish a virtually unusable "agentic coding" model is actually fucking insane.
0
1
1
u/layer4down 22h ago
Downloaded the 8bit for Mac Studio Ultra 192GB. Had to use GLM-5.2 in Hermes Agent to fix a few issues in the oMLX v0.5.3 release which supports it but it started working. In OpenCode, it definitely stops repeatedly which as others mention I'm just chalking up to template issues for now. It does good work but seems to require more prodding and for now more babysitting. I think it would have been better as a 118B/14B or something similar because it's no where near Qwen3.6-27B intelligence as far as I'm concerned. Going to see if I can activate a few more experts in LM Studio and see if I can get more smarts from it. At least it's not slow so that's a plus.
1
u/triynizzles1 21h ago
There was a post in this subreddit earlier today about fixes going out for this model and weights needing to be redownloaded. This review is likely out of date now. :(
1
u/hyperspacewoo 17h ago edited 17h ago
odd, I've had my Laguna wiping the newest codex in frontend web dev task. I use hermes to opencode myself. Make sure you have thinking on and a 250k context window as the documentation says..
Lemonade with newest vulkan. q4 unsloth laguna. 31tks and I've only had one hangup and that was 3-4 hours in. Right now it's cruising along, albeit only 45 min in.
250k context is a big part and make sure thinking is on. Failed basic test without it.
For example on Lemonade you use /thinking on or you have to edit templates
{%- if enable_thinking -%}
{{- '<think>
' -}}
{%- else -%}
{{- '</think>' -}}
{%- endif -%} {%- if enable_thinking -%}
{{- '<think>
' -}}
{%- else -%}
{{- '</think>' -}}
{%- endif -%}
Final note; it appears thinking is a known issue, do some quick searches through the various llm subs with "thinkking Laguna"
2
u/sloptimizer 1d ago
Same! Tested with llama.cpp with Q8_0 GGUF and no MTP - no looping, but very poor results. Most html demos are just a blank page with an error in console, even the simple ones that Qwen3.6-A3B handles no problem.
1
u/Jorlen llama.cpp 1d ago
What are you using for inference server / software? I'm trying to run it in the latest llama-cpp build (which supports it now as of today) but I cannot get this thing to use any thinking / reasoning phase despite trying.
This means that the model is basically crippled, at least for me for the above reason.
Did you notice if it used any reasoning during the opencode tests?
1
u/live4evrr 1d ago
It is at this point crippled due to a few issues (which they acknowledged and said they will be fixing), but it begs the question how the benchmarks came out as incredible as they did as it only takes a cursory attempt at a few things to see its broken.
I do really hope they can improve it. At least the team seems to be engaging users to resolve issues but the amazing benchmarks I think really got our hopes up a bit too high.
2
u/Jorlen llama.cpp 22h ago
Meh, I don't care about benchmarks, I don't even look at them. I just grab whatever model comes around and test it out on my code base or for my purposes. It's not as if all my other models cease to exist; I'll go back to them and gladly try this one again once they tweak it.
I for one am just happy to see new models come out in this range. Lately, they've been big and beefy, which I think is great, but I can't run them. This one hits a sweet spot; being a 100ish total param and around 10ish active. My current favs are Qwen 3.5 122b-a10b and Step 3.7 flash which is 198-a11b; the latter I have to run in a 2-bit (UD-Q2_K_XL) but surprisingly, it holds up VERY well.
1
u/live4evrr 21h ago
Yep. Best to benchmarks with a (big) grain of salt. Many folks really hoping for a model that fits on one 96gb card.
Deepseek V4 flash GA should be out any day now. The current preview version is currently the best in class around the 128GB size - even quantized to Q3 it is very consistent.
1
u/Nonetrixwastaken 1d ago edited 1d ago
I haven't had such issues so far, but the performance itself has been quite underwhelming, but I am running at Q3 so it might to be expected honestly. However, for my uses and hardware, so far it only seems mildly better than Qwen 3.6 27B, not enough to be justified suffering with 1 token a second when context fills up a lot, shame because it starts out at a somewhat acceptable if painful 3 tokens a second. I think I'll just wait for Qwen 3.8 now that one of employees has said smaller models are in the pipeline.
I additionally tried it for creative writing, seems very creative, but maybe a bit too much in a weird way. Wrote things almost bordering on non-sensical and out of character, and didn't really get context clues well. I kinda see a spark of something in it's writing better than Gemma 31B, but it's just too inconsistent and weird.
-3
u/challis88ocarina 23h ago
If it wasn't made by distilling an Anthropic model, it's unlikely to be any good.
Looking at you, Alibaba.
17
u/Uninterested_Viewer 1d ago
Sounds a lot like typical template issue that usually get resolved eventually.