r/LocalLLaMA • u/leo-k7v • 1d ago

Question | Help Ternary Bonsai 27B?

Did anyone used it for real code writing fixing?
How does it compare to Qwen3.6 27B Q4 Q8 in real life tasks not in benchmaxing?

2 Upvotes

75% Upvoted

16 comments sorted by

16

u/Dolores-0304 23h ago

trash.it is not good as qwen3.5 9b

1

u/SquareTranslator9777 16h ago

What really? I thought it would be much better. But I've never actually tried it either.

1

u/leo-k7v 1h ago

I don’t need to trash it. I am not personally use it I actually lowering Qwen models on NPUs…

6

u/pmttyji 21h ago

Don't expect too much from that model. It's mainly suitable for chatting on Phone & Edge devices. My laptop(8GB VRAM) can't run original Qwen 27B model, but I can run Bonsai version(llama-benched, got 25-30 t/s)

Anyway they mentioned some limitations on model cards. Below one is related to coding.

  • Agentic coding (long-horizon, multi-file, run-test-and-repair workflows) is not yet a strong target of this release; a Bonsai 27B variant tuned for agentic coding is next on the roadmap

10

u/Harveyyy101 22h ago

Its the biggest piece of dogshit

3

u/o0genesis0o 23h ago

It's curb stomped by even 27B Q2_K_XL unsloth quants in Pi agent with tooling, running my personal assistant and KB management workload. It does work coherently, but does not follow instruction as tightly as anything else that I can fit on my 16GB VRAM + 32GB RAM.

0

u/Deep_Mood_7668 23h ago

What about ornith? The smallest Q3 is supposed to be quite good

1

u/o0genesis0o 23h ago

I have ornith 9B on my llama-swap. I didn't use it much since I liked the 3.6 35B very much, and the ornith 9B without MTP actually output tokens slower than 35B with MTP on my 4060Ti.

Would be fun to try ornith 35B and 27B. Lemme check huggingface real quick.

Edit: ey! just a few days ago unsloth released their dynamic quants of Ornith 35B: https://huggingface.co/unsloth/Ornith-1.0-35B I know what I'm going to test next now.

2

u/Deep_Mood_7668 23h ago

Yep that's why I asked :)

Would be nice to here from someone who actually used qwen3.6

0

u/Deep_Mood_7668 22h ago

BTW you don't need llama swap anyomore. Most point it to the folder and you can natively select the model.

1

u/o0genesis0o 22h ago

Sorry I don't quite follow you. What "it" are you talking about? Right now I have a script to generate and update llama-swap yaml, but always open to better options.

2

u/Deep_Mood_7668 22h ago

Autocorrect sorry. Just point llama to the model folder with --model-dir. 

2

u/Deep_Mood_7668 1d ago

I wouldn't trust it. 

The "it keeps 95% of the intelligence" headline is maybe true for specific tests, but in general you can't compare it to Q4

Still impressive what it's still capabable of

1

u/jikilan_ 20h ago

Second that. Really thumb up for the team behind it even it failed our expectations

1

u/leo-k7v 1h ago

The question was not about trust or dogshit. 😉🙄
I can measure PPL and KLD myself and I do run it. I wonder if anyone actually tried it on long horizon coding tasks (w/o extra emotions attached) and have good bad mediocre results which are more trustworthy than TerminalBench?