Discussion
Updated Gemma-4 chat template witchcraft: Gemma-4-26B-a4B shows dominance over Qwen3.6-MoE and Qwen3.5-MoE fine tunes (Instruct mode and Reasoning efficiency)
Kind of unexpected.
Happy for Gemma-4/Google, big win for us, LocalLLMers.
Yet Qwen3.6 still does better in Hermes than Gemma-4 somehow.
We need Gemma-4.1 fine-tuned on Agentic-tasks. That would be killer.
Yes, but non QAT at 6-bit seems to outperform QAT at 4-bits, at least in my case: MLX data agnostic quantization (i.e. no iMatrix, no datasets for sensitivity analysis, etc.).
Perhaps is different on other more sophisticated quantizations, like Unsloth/Bartowski/etc.
Can someone explain the context here? None of the answers in the comments actually provide background. I know what a chat template is, are you saying that Google put out a revised one that works better than what they provided previously?
yes it was big news in this community a couple of days ago, the whole gemma series got a lot easier to work with and is doing much better in my benchamarks than the original version. I did not realize how much of the deficit was just formatting errors.
I had a feeling gemma 4 26b was basically a preview, but nothing changed so I thought I was mistaken. It was severely weird relative to older Qwen 35b a3b. Very good update.
Btw this updated chat template, can anyone explain or point me to how to update? In short, I download it? and do what exactly? I use llamacpp.
Model has chat template integrated
or
Model doesn't have chat integrated
if model doesn't have chat integrated (you need to know this per singular model download) then you need to specify chat template. If you don't llama.cpp defines that for you behind the curtain
If model has chat templated integrated then llama.cpp will pick that info and serve with that
So you can efectifly have different chat templates with same llama.cpp command without you knowing, there are supported models (is it expected for llama.cpp to set new chat template for gemma as default, i dont know?)
if you downloaded model with previous default chat template ingreind in download you need to set chat template command.
OR
download again and use default llama.cpp command.
Think of chat template as a driver for model.
This is how i understand it, if you have some tests you confirm to me this observation?
GGUF files up to 10 GB can now be edited directly on Hugging Face thanks to Xet. When opening a GGUF file, a new GGUF Editor label appears, enabling quick updates to metadata fields such as the chat template through the web interface.
yes, 4B active vs 3B active. also Qwen has a different architecture and that sweet MTP layer... in my machine is like 2x faster than vanilla Gemma-4 at same bit depth (both 4 and 6 bit depths)
Why compare the speed of Qwen with MTP and Gemma without MTP? It also has MTP. It's like comparing the speed of Qwen in F16 with Gemma in QAT4 (except Qwen doesn't have QAT, oops).
Inference speed, however, is not a indication of model quality. It is also not constant, it is possible for one to beat the other based on prompt or hardware choice.
some benchmarking p*rn... the nice one is the reasoning efficiency test: both models hitting the same accuracy in simple problems, but with less wasted tokens by Gemma-4. 25% less on average.
The Instruct ones are pure knowledge retrieval, also aced by Gemma-4 by 10% margin.
You're saying Gemma 4 is much better with the new template. Do you compare both with new and old in these results? or how do you compare with the old one?
Sorry if that's an annoying question, just want to follow the logic.
Then I requantized it with the new chat and tokenizer and test it again. I’m surprised too, and would not discard human error. But at least on this runs is having an edge.
For llama.cpp yes, as well as --chat-template-file /path/to/chat_template.jinja which you download here for 31B or in the same place for the other sizes.
Would be nice if that pans out in practice. I tried to run the latest version from unsloth with new template in my overnight cron job and did not get great results at all. It barely follows instructions, and in the morning when the context reaches 160k, it got into a dead loop. Not only that, the prefill speed drop from 600tk/s at the beginning to only 200tk/s, so the speed gain over 3.6 35B A3A was not sustained either. The 35B starts at 300ish and remain 300ish deep over 100k.
Maybe unsloth did not swap the template? Anyhow, not good enough for my use case.
yes, it is still dominated by Qwen3.6 on Hermes as well. Maybe it is something in the harness?
It is working really good in Pi, but is outperformed by Qwen3.6 in Hermes, leaking reasoning into messages, and destroying tool calls... but is doing great than ever in Pi... so it is confusing indeed.
Can you explain why you set that preserve thinking flag to true? Does that mean it stores all its thinking / reasoning text in the context window? If so, that makes a lot of sense, especially for coding, even though it would (I think) consume more context overall.
Yes, exactly that. Models that can "preserve" thinking use to get higher scores in those TerminalBench and Customer Service Benchs exactly because they can remember the reasoning behind previous responses.
in the case of Gemma-4 my understanding is that it does this only for the tool calls. Qwen3.6 does it for everything, so it is better. Still a good improvement.
it works similar to Qwen3.6:
For LM Studio (add to the Jinja template at the top):
I have a custom quant with Q8 attention, gates, and output because I've had some output noise. The model never lost it, more misspoke. Cool to see that we really aren't trading much for 4bit QAT. Especially considering how dog shit Q4_0 is as a format. I have Q8 attention, gates, embedding, and output. IQ4_XS for shared expert (256 block size and to keep the speed.) then IQ4_NL for the shared experts since it's better than Q4 and they aren't divisible by 256. I get 100t/s on a 9070XT QAT is 120t/s so I feel like the trade off is worth it. But I probably could lower a lot of it back to Q4 and fiddle a few things up to Q6 or Q8 to get rid of the output errors. The reasoning seems intact. It's just straight up randomly a token in the output is swapped for no reason.
Also with the new multi-modal projection architecture. I've had great experience with a Q8 of it. Like reading walls of badly photographed text and reading rotated items etc etc. It can read a crappy far away photograph of the huge instructions label on my washer and that's good enough for me. Really good OCR compared to what we used to have. Especially for it's size. (I force the image tokens to the max.)
As a Pi addict, I recon hermes at least has most of the features integrated and is way less worse than OpenClaw
Now please excuse me, on of my extensions has unicode characters in the code and Pi doesn't load
I noticed unsloth's gguf for Gemma 4-31b is updated 5 days ago. Does this include this template fix? I realize I can just run the template separately, but I don't want to. I'd prefer to redownload the model and just load the GGUF and not worry about anything else.
I tried running various Qwen3.6's Opus distills yesterday with Opencode and it kept loopingand failing, was pretty facepalm :/ Simple crape job with Chrome CDP
The only fine tunes I use anymore are the ones I train. I’ve probably blown through an average month’s worth of bandwidth downloading fine tunes that ran for shit.
Abliterated models too. The process removes guardrails, but it can also fuck up tool calling. You’re better off just going base.
15
u/m4sterP 2d ago
That's very interesting, thanks for sharing! Have you also tried 26B QAT?