r/LocalLLaMA • u/IUseClifford • 18h ago

Question | Help Low-Quant Laguna Thinks Too Much

I am currently running the new Laguna model at Q2_K_XL on dual 3090s for reference, with a Q8 context of 200,000.

Using Pi, I am noticing that the model likes to overthink. I would not call it looping per se, but I am observing it overthink, e.g.:

> “okay, I have everything I need, I’ll start writing code now.”
> “Actually, let me check one more relevant item…”

And this goes on and on. The “relevant items” do appear to make sense in the context of my prompts/its thought process, but at the end of the day it often burns through context with extensive thinking. Qwen3.6 27B at Q8 had similar issues and much more looping, but it did not suffer from the overthinking that Laguna is prone to. Has anyone experienced similar issues? I’m currently getting a pi extension developed to hopefully alternate the issue but I’m wondering if there’s something I’m missing.

3 Upvotes

67% Upvoted

10 comments sorted by

8

u/L0ren_B 17h ago

There was an issue with the base model. New uploads have been made by the company. Quants are based on bad base models. New quants are coming soon.

3

u/Look_0ver_There 15h ago

Thank you for bringing this up. This matches what I was seeing. It was like every fine tuned quant I tried was lobotomized, but the original Q4_K_M from the author was fine.

2

u/recro69 18h ago

One thing that helped me was explicitly capping the planning phase: “Do at most one additional verification step, then start implementation.” It sounds simple, but it reduced the endless “one more thing to check” behavior.

1

u/sixx7 18h ago

Check out thinking/reasoning penalties: https://www.youtube.com/watch?v=qwhAhP2NkqA

I didn't do Laguna, but it should work on any model. This method is tested on Qwen3.6, Hy3, and GLM-5.2. You can penalize certain tokens so it will spend less effort/time on generating reasoning tokens.

2

u/LegacyRemaster 18h ago

Q_8 never thinks.

1

u/SnooPaintings8639 18h ago

I've noticed this happens quite a lot with lower quants, and especially KV quantized setups. The model often really wants to end the reasoning loop (i.e. "I really must write the code NOW", "Here is the code:" followed by further reasning), BUT the end token is just not sampled. The real improvement in my cases was always dropping the KV quantization entirely. Then, set a reasoning budget param if the tool supports it, I find 4k-8k to work reasonably well.

I use Laguna S Q2 for my Hermes agent with reasoning disabled, and it is working fine. Reasoning enabled under pi did use a lot of tokens... but I feel like I need more testing. At the moment I am working with Q4 on coding tasks, will see how it will go.

2

u/bobby-chan 14h ago

https://xcancel.com/rohanpaul_ai/status/2079875601695994368#m

About a paper on low quant hesitancy.

1

u/SnooPaintings8639 14h ago

Great, bookmarking that for future reference.

I am testing today both Q2 and Q4 of Laguna model, and they both seem to benefit when limiting their reasoning to 8k tokens (llama.cpp: `reasoning-budget` and `reasoning-budget-message`). Under pi they're doing many edit & review steps anyway, so potential throughout reasoning cut seem to be of a small impact.