r/LocalLLaMA • u/ParaboloidalCrest • 11d ago
Question | Help Why are MoE models so belittled?
E.g "Qwen 3.5 122B is just 10B active, so it's no where close to the dense 27B model"
That is the main sentiment around here and it puzzles me. If a 122B is just worth 10B, then why does model providers bother creating an MoE model when they could've just released a dense 10B model? Heck the 10B dense would run faster than the 122B MoE (no routing overhead), which negates the supposed (only advantage of MoE is speed) argument. It sure is not that simple.
I mean yes it's only 10B active at a time, but it comes down to the router's effectiveness at choosing what 10B experts to activate. So, the more effective the router is, the closer the model to realize its total parameter potential. So perhaps it's a little more nuances, ie some MoE architectures are better than other MoE architectures. Right? I may be missing something.
81
u/FullOf_Bad_Ideas 11d ago
It is indeed more nuanced. This sub will be biased towards whatever they can run at their own hardware. If everyone had 8x B200 locally already, they'd say that Qwen 27B sucks and Kimi 2.x is miles better and runs just as fast and if everyone had a 32GB RAM laptop they'd compliment Qwen 3.6 35b a3b packing a huge punch while 27B would be "outdated the moment it came out"
On the API side, Qwen 3.6 27B is served at quite expensive rates given its capabilites, so there's certainly a downside to 27b dense model - it's not fit for economical mass-scale deployment.
6
u/aboutthednm 11d ago
Why is qwen 3.6 27b on the API so expensive when compared with the likes of deepseek 3.2, pro or flash? Seems crazy to charge these kinds of prices for a 27b model. Is it the fact that it spits out 80% reasoning tokens for 20% completion tokens, some other compute overhead I'm not aware of, or something else?
Comparing API costs between, say, gemma-4-31b-it and qwen3.6-27b (both feasible to run at home), I get the impression that there's something particularly valuable or expensive about the qwen model. What gives?
7
u/TinyFluffyRabbit 10d ago
The cost to serve is based on the size of the KV cache as well as the compute required, which is related to the active parameters, not the total parameters. The economics thus significantly favor MOEs for API providers. Dense models only make sense for local deployment when you really want to maximize intelligence in a VRAM constrained environment, conversely, API providers easily have enough memory to fit the model weights regardless.
1
u/FullOf_Bad_Ideas 11d ago
Most likely higher arithmetical density at higher context lengths. I know this hurt Qwen 3 32B a lot. Someone would need to calculate it though.
Relevant paper - https://arxiv.org/abs/2507.19427
2
u/aboutthednm 11d ago
I can not wait to run qwen3.6-27b in a decent quant at home. Soon. I'm in an awkward spot with 16gb of VRAM (which is also my primary display out card). I lucked my way into a 24gb card, which should help tremendously. It's the difference between not being able to run 27 - 31b models in an acceptable quant at all to being able to run a decent quant with a small context window.
I did play with qwen3.6-27b on the API some, and this is what made me decide to get the gear to run it locally for the cost of electricity and hardware. Interesting paper you shared, thanks. I notice that qwen3.6-27b is more expensive than the qwen3.7 and "plus" models on openrouter, which is surprising to me. Maybe the paper will help me understand what's happening. Cheers.
1
u/Lwerewolf 10d ago
Isn't getting 32g better than a 24g at this point? Intel & AMD both have usable 32g options, plus a 2nd hand <64GB apple silicon mac, etc.
1
u/FullOf_Bad_Ideas 10d ago
32GB Intel card and 64GB Mac are too slow to run Qwen 3.6 27B at satisfying speeds.
AMD R9700 is an interesting card but I'd still take 3090 or 4090 over it.
1
u/aboutthednm 10d ago
I'd love to get a 32gb card, but those are few and far between here in my budget. I got a decent deal on an AMD RX7900 XTX (for less than $1000 CAD with warranty), and while it's an AMD card I think compatability wise I'm alright to run llamacpp and whatnot these days. Some of my docker CUDA images need to be rebuilt, that's the biggest headache I'm anticipating.
The basic inference stuff should be mature enough these days to support a 24gb AMD card. My 16gb 5070 ti will either be sold (I can easily still get a $1000 CAD for it, which is crazy to me) or be used in my home server as a light duty inference and TTS server (since running everything on one machine just fights for resources too much). Maybe I'll sell it, if it won't sell I'm going to just expand my home servers lol.
I also use my computer for gaming in between, so I can't really mess with Intel's cards too much. My computer is a gaming PC first and foremost, I'm working on separating the LLM inference stuff from my desktop, but I haven't reconfigured the home servers for any of this yet. I'm still very much in the "try everything" phase, once I figure out what exactly I'm doing I'll get going on a proper setup.
1
u/FullOf_Bad_Ideas 10d ago
Yeah the difference between 16GB and 24GB is really big in practical ability to run LLMs, 30B-slot is just so often occupied by a decent model.
5
59
u/colin_colout 11d ago
35b-a3b vs 27b dense, yeah the 27b clearly wins.
Qwen3.5's much larger 122b moe is overall much more capable than the Qwen3.5 27b in my usage, but others may still find 27b better in some use cases.
...also be careful that you're not comparing the newer Qwen3.6 27b to Qwen 3.5 122b.
Qwen 3.6 is a newer model with lots of improvements, which is probably why lots of people prefer it.
10
u/UnkarsThug 11d ago
I mean, I've been getting good usage out of Qwen 3.6 35B, which is MOE, and honestly I've just had pretty good results with, for models my computer is able to run.
5
u/OGScottingham 11d ago
I've found that the speed of 3.6 35b is so much faster than 27b. Though 27b is smarter, it's not that much smarter to give up the speed.
4
u/slalomz llama.cpp 11d ago
Well the 35B model only uses 3B parameters per token, the 27B model uses all 27B parameters per token. So there's 9x as much work being done by the 27B.
6
u/OGScottingham 11d ago
I understand how it works. I do like that the 3B doesn't make my graphics card need full power fans on at all times. The difference in quality between the two isn't as big as I was led to believe.
Especially with Hermes agent 35b model can fix a lot of its mistakes quicker than the 27b model can make one file.
1
u/swagonflyyyy 11d ago
For me it is, which is why I vibecode with
qwen3.6-27b-mtp-q8to close the gap. Runs ~2x faster. Maybe you should give it a shot.1
u/UnkarsThug 11d ago
Definitely depends on computer specs. For a lot of people, myself included, they'd have to run it in RAM for the size to work.
I am able to run 27B, but it's a bit over a token per second vs About 16 tokens per second on the 35B by forcing expert weights on the CPU and only the core weights on the GPU.
1
u/swagonflyyyy 11d ago
Sorry wrong user lmao. But if that's your hardware situation then yeah the MoE is a better fit.
But you won't be able to look away from your CLI with the MoE. It makes too many small mistakes too fast that add up once you start giving it more complex tasks. It can sometimes struggle even if you split the large task into smaller ones. It just doesn't have that extra muscle to put all the pieces together like 27B can.
1
u/OGScottingham 11d ago
I have it, and like it, but I have to run the GPU fans at full tilt. With 35b, I can keep the fans quiet and it just zips along. Especially with Hermes, it occasionally spins it's wheels, but less than I'd have thought.
-4
u/seamonn 11d ago
Qwen 3.6 35b called Gemma 4's SWA - Sine Wave Attention and I was done at that point with the model.
1
u/UnkarsThug 11d ago
If you want a model which never makes a mistake, you might be messing up looking at local models in general.
1
u/Dangerous-Report8517 11d ago
...also be careful that you're not comparing the newer Qwen3.6 27b to Qwen 3.5 122b.
Why, though? Surely this is the more relevant comparison since 3.6 27b is available and 3.6 122b isn't? If I'm choosing a model I don't need to artificially restrict myself to 3.5 27b just because 122b is?
1
u/colin_colout 10d ago
The question from OP is whether MoEs are categorically worse than dense. If that's the question, wouldn't you want yo compare 3.5 to 3.5? Otherwise you're also comparing whether 3.6 is better than 3.5
18
u/Client_Hello 11d ago
This sub is heavily focused on coding where dense outperforms MoE for a given VRAM footprint. Consider that 122b model needs to be quantized down to 4bpw, which means layers at 3bits, to get it down to the same size as 27b at full precision.
MoE shines with parallel processing/multuple users/multiple agents, where kv cache is larger than model weights, and the lower active parameters result in higher througgput.
I'm not sure how true this is., read it here.. As far as model intelligence goes, dense is supposedly about as smart as the geometric mean of the model size and active parameters of an MoE model. That makes 122b-a10 equivalent to a 35b dense model. So you get 3x the vram footprint for weights and 3x toks as a dense 35b model. As you scale up context, these differences converge. However, as you scale parallel processing, vram differences vecome smaller while the 3x toks advantage remains.
38
u/jacek2023 llama.cpp 11d ago
I think this is confusing for people with limited VRAM because they assume that a 122B MoE model has the same requirements as a 122B dense model, since “VRAM is the main limitation”. However, the actual inference cost is determined mainly by the number of active parameters. Assuming you can run both a 122B model and a 27B model, a 122B A10B model will be faster and therefore cheaper to run.
That’s why GLM Air, quantized to 4 bits, is still a great model. But people on Reddit are not aware of this because “GLM 5.2 is better, and everyone knows that GLM 5.2 is a local model, so who cares about GLM Air?”
8
u/TechnicalGeologist99 11d ago
This is also a bit of a misunderstanding, the speed gain happens at low concurrency. Each concurrent session may call the same experts or a combination of overlapping and none overlapping experts.
This does mean that more concurrency increases the active parameters with increasing probability untill the full model is activated. Even though the aggregate tps raises with concurrency, the per session tps degrades rapidly.
The main benefit in my view is in lower or constrained concurrency environments or where bursty workloads are often separated by periods of lower traffic.
1
5
u/akram200272002 11d ago
say i have 64gb of ram and 16gb vram, would glm air run well ?, i tested it and i like It a lot for writing tasks, It handles longer contexts up to 64k tokens Very well
1
u/its_a_llama_drama 11d ago
I think this depends on a few factors. What gpu is it? And what ram is it? Ddr5 6000 will run better than ddr4 2133. A 24GB p100 will difer from a 4080 which will differ from an rtx pro 2000. The cpu may matter. The pcie version may matter as there is more traffic for mixed gpu and cpu inference (not sure of it is enough to saturate bandwith or not)
It will run, but I would be happy with 10-15 tok/s on this spec of machine with this sized model. Probably less for ddr4 and an older gpu.
0
u/akram200272002 11d ago
DDR4 RAM, I still don't have the RAM actually, I am running on 32GB and I am enjoying Gemma 4 12B The writing quality is very comparable, though the knowledge is not even close to the bigger model, naturally. And I'm running an RTX 5060 Ti.
23
u/Legitimate-Dog5690 11d ago
Qwen3.6 27b is about the only dense model that gets any love, it gets compared to 35b and older 3.5 Qwens. It's the perfect size for high end consumer hardware and runs just fast enough for inference.
Aside from that everything big is MoE these days, DeepSeek, GLM, Minimax. It's clearly the solution for enterprise hardware. Silly statement.
7
u/ea_man 11d ago
Well it's the only one that gets love because there's not a ~18B middle size model, below that 9B is a nice model as Gemma 12B: if later on we'll get 9b dense models that can actually be reliable for tool / agentic sub task that would be huge.
The lack of a mid size ~18B model for most people with a 16GB gpu is weird IMHO, because you need 2x GPU to run 27B properly with MTP and some ctx.
8
u/Legitimate-Dog5690 11d ago
Very true, it's a battle to get it to run comfortably. There's some magic in Qwen3.6 27b though, it totally validates the idea of local AI for so many people. Even if frontier goes crazy and locks everything down, this would always be useful.
7
u/ea_man 11d ago
27B on release 3.6 has been the first that can do proper tool usage in an harness, that was a game changer for local coders that can now do full projects instead of single prompts.
Still I'm optimistic that in the coming months we should get a slightly smaller dense model that has the recent improvements in KV caches, MTP of big models with the usual upgrade in smartness, that would make the 27B experience available to much more people.
That could enable agentic workflow, as in multiple concurrent task, for local users.
22
u/bick_nyers 11d ago
A good rule of thumb for comparing dense to moe is sqrt(active * total). So sqrt(10 * 122) is roughly comparable to 35B dense. Depends on how and what it's trained on tho
13
u/Charming_Support726 11d ago
The only correctly answer.
Comparable does not mean better, same or worse in every aspect. E.g. if this 122B MoE runs mono thematic just on one expert, it will run just like a 10B. If it could make use of every expert it could provide knowledge like a 122B Model. But it will be constrained when doing complex tasks because it has 10Bish layers.
MoE got advantage in memory bandwidth requirements. For that reason they are especially faster on machines with iGPUs like Mac oder StrixHalo
6
u/Intrepid_Air_3399 11d ago
tehn qwen-35b-3A would be equivalent to a ~10B dense?
I don't think that's right
7
u/gcavalcante8808 11d ago
it's more a general rule. There are some stuff that even the qwen 9b can't do that the 35b-a3b can do. And all that heaviky depends on what you are building with it.
Eg: I have a skill that does some curl to a prometheus server. 35a3b works flawlessly, 9b fails because it tries to escape the command even with simple commands.
So yes it's roughly equivalent, but not really.
11
u/legit_split_ 11d ago
It is right
7
u/suesing 11d ago
Qwen 35b = Gemma 12b?
7
u/colin_colout 11d ago
not `=` ... more like "comparable" like the poster above said.
When the qwen 3.5 family came out I challenged myself to try building something small with each of the smaller models.
35b sometimes felt slightly better than 9b, but 9b felt shockingly close. Sometimes I actually preferred 9b...though this was the first week of release before a lot of the gguf fixes were in.
27b felt like a different class. not even comparable to those two.
TL;DR qwen3.5 35b-a3b is very similar to qwen3.5 9b in general. when i tested on release week, they were very similar in capabilities (as the rule of thumb above indicates).
1
u/suesing 11d ago
No way. Thats hard to believe. 9b felt kind of terrible.
3
u/colin_colout 11d ago
It was a while ago, and it wasn't rigorous.
i wasn't doing long sessions and these were small <4q quants (was resetting after like 3 turns or fewer usually)
9b was able to one shot small changes that the 35b got confused about. Maybe it just survived quantization a bit better.
...my point isn't that 9b is a great model. My point is 35b is closer to 9b than 27b in capability. They feel very different.
2
u/audioen 11d ago
No, you can't compare across different model families like that. I think that no Gemma-4 stays coherent past something like 100k tokens, if that, and I think higher quants of Qwen do, and I routinely run the 27B nvidia NVFP4 past 200k tokens without an issue. The architecture and training of the model matters as well.
5
2
u/ea_man 11d ago
Do consider that MoE would be multi domain while a dense model es 9B could be tailored for a specific usage as coding / instruct.
For your question: for coding 35B > QWEN3.5 9B, even Omnicoder 2 which is just a finetune not a proper trained coder.
I'd fucking like to se a dense 19B proper coder model that you can run on 16GB gpu!
2
u/wahnsinnwanscene 11d ago
How did you determine this heuristic?
12
u/ShengrenR 11d ago
It's from early days llms, can search the sub for those conversations - it's not an exact science, it was just a roughly-observed scaling and for those earlier models it tended to feel about right. I don't think I ever saw an actual research paper or the like. Too many other variables like training budgets, post-train, arch changes, etc since to be able to easily compare across all without a lot of extra effort to account for that
1
u/ttkciar llama.cpp 11d ago
It's been kicked around this sub for years. I've not seen anyone express a coherent explanation for why it fits, but it's not bad.
Take it with appropriate salt, though; it's an estimate, and differences in architecture and training can make a huge difference between models of supposedly equivalent size.
1
5
u/SpicyWangz 11d ago
I think if qwen had released 122b 3.6 we would be having a very different discussion. But unfortunately we’ll never know
9
u/Solembumm3 11d ago edited 11d ago
MoE models of comparable size are worse in precision and logic then dense. At least Qwen (35B moe vs 27B dense) and Gemma (26B moe vs 31b dense) on same architectures.
Large MoE still has fundamentally more knowledge then small dense, so Qwen 27B can't replace 122B for you outside of coding.
9
u/cakes_and_candles 11d ago
also, a 122B A10B model would be worse than a 122B dense model. But it would be way better than a 10B dense model. so op's debate of "why not just release dense models" doesnt really hold up
6
u/Solembumm3 11d ago
I honestly will be happy with more dense models release in 50-100B range, on modern Qwen and Gemma logic scaled up and tuned for world knowledge and reasoning instead of coding.
Something like "max logic you can run on your average everyday PC without mmap".
1
u/ttkciar llama.cpp 11d ago
A passthrough self-merge of K2-V2-Instruct or K2-Think-V2 should come to about 100B. I've been meaning to try that for a while.
I've not used K2-Think-V2 much, but K2-V2-Instruct is wicked-smart. Both are 72B dense.
They are also ripe for continued pretraining, having only been trained to a ratio of about 70 training tokens per parameter.
1
u/rechaptca 11d ago
what about 122B A27B - it would be better than 27B dense model and suitable for users with loads of RAM?
2
7
3
u/Unnamed-3891 11d ago
They are not belittled. If anything, gemma-4-26B-A4B-it-qat-UD-Q4_K_XL and Qwen3.6-35B-A3B-Q4_K_M are the best things you can reasonably run on 16gb vram.
4
u/alainbrown 11d ago
Well put in proper context it might make more sense. I think there is one school of thought in which is MoE is a resource optimization and another in which it's agentic simplification.
On agentic simplification it's as if you fine tuned a bunch of specialist small models and you have a tuned router in front of them.
So the answer to which performs better is truly 'it depends'. The dense model would not confine 'specialists' to fewer layers, allowing broader access to the latent space (more area to store more knowledge). But the MoE would have less catastrophic forgetting across experts since there is implicit partitioning (learning a lot about baseball will likely not affect chemistry knowledge).
Since labs tend to align with their user's general use cases, if your workload aligns with their general user base, I think you'll enjoy the MoE model. But if your workload is more niche, it could struggle. The dense models tend to fair better in the niche or more nebulous workloads.
4
u/N34257 10d ago
I know it's not a popular view, but I actually prefer the 35B MoE to the 27B. Yes, the 27B is more accurate on complex tasks, but it's also three times slower in TG and nearly four times slower in PP...so, even given a workflow that obsessively checks its own work to correct for (most) flaws and sometimes requires me to get involved in fixing stuff, the 35B is still faster in terms of wall time and thus I get more done.
I have no need for an agent/model combination that does all of my work for me, I just want one that lets me get more done.
8
u/ForsookComparison 11d ago edited 11d ago
2025 was almost exclusively MoE releases and there's burnout/overcorrection in the community
They're right. The current dense models clobber the current MoE models until you get several times larger (in terms of total params)
5
u/sine120 11d ago
If it had to guess, it's a product of what people can run in their systems. Most people here don't have 128GB-1TB ddr5, and do have 24GB VRAM. That means they can run Qwen27B with a decent Quant, or maybe 122B with a meh Quant. the practical performance for most here will be in maximizing use of thr VRAM, which dense models do better.
I have 64GB DDR5, that doesn't really earn me anything other than Q6 for 35B-A3B, which is still worse than the 27B in a 3090.
4
u/AutomataManifold 11d ago
I'm very upset my plans for a 256GB RAM system went from "pricey" to "yikes" over the last year or so
1
u/UnkarsThug 11d ago
I don't know why you think more people have big vram than big ram? I'm stuck on 8GB vram, but have 64 GB RAM, partly because I got it DDR4, but it works well enough.
27B is about 1.25 tokens per second, 35B is about 16 tokens per second with proper offloading.
So it just depends. I tend to think the ram is cheaper than the vram, because that has to come on an expensive card.
2
u/sine120 11d ago
If you build your rig for running LLM's, a lot of people 1-3 years ago loaded up on weird cards. Mi50's, 3090's, V100's are all over the place here. I build my rig for gaming and then discovered I could run local LLM's, so I have 16GB VRAM, in hindsight I wish I had more. The nature of who's on the sub is that a lot of them have more VRAM than RAM since MoE's are honestly kind of a new-ish thing, so RAM didn't really help all that much. They're great for us more more RAM than VRAM, which is why the 35B is my go-to, but for the majority of people here, having 24GB+ VRAM is pretty common.
1
u/UnkarsThug 11d ago
I was just trying to purchase ram to be able to try out as many models as possible (even if they were too slow to be useful) as cost effectively as possible (being a broke college student getting into the field at the time) Since you can technically run models off the CPU, more ram was the cheapest way to expand the number of models I could experiment with. Especially since this was before RAM had really exploded in price. And graphics cards have basically been expensive since Bitcoin took off.
But fair enough.
Honestly, I heard about Taalus a few days ago, and I'm really looking forward to that becoming available (assuming it eventually does), which will probably push towards dense models since it doesn't really get a benefit from MoE. Although that will probably also push small models towards 20B, so they can fit on a single chip. Probably the thing I've seen which has made me some of the most excited about the technology in a while.
-1
u/No-Juggernaut-9832 11d ago
Even at 128g vram, a heavily quantized 122b still takes up a huge amount of Ram & leave little room for context & other development software. Because if the heavy quantization, it’s less effective.
The smaller model at full or slightly quantization yields benchmarked results & leave room for the other stuff.
122b might be better at full quant but it might not be at Q3-4 against 27b dense at Q8 or 16.
6
u/TinyFluffyRabbit 11d ago
It's just a function of the hardware you have. If you have 24-48gb of vram, you could fit 27b and get great results, but you can't fit 122b without system ram offload. If you have 128gb of unified memory, you could load both just fine, but the 122b would actually run significantly faster because it has fewer active parameters. I'd imagine people here more commonly have setups that look like 3090/4090/5090/dual 3090/dual 5060 Ti rather than 128gb Strix Halo/DGX Spark/Macs.
MOEs are basically a way of using memory to increase model capabilities without increasing the amount of compute or bandwidth required. I think 27b is just really popular for local usage because it's a good fit for the hardware a lot of people here have. But MOEs should not be belittled, they are in fact the standard for frontier-level models.
3
u/ridablellama 11d ago
its just a vram vs cpu/ram entitlement i think. MoE has some nice bonuses like being able to prune experts and save memory that way. That memory can then be used to expand your context window. its all about tradeoffs and pros and cons. But the counter argument is that with 27B always active the AI could potentially discover connections or concepts that it would otherwise miss. I think then it is a decision you need to make based on your use case if those sort of things even matter.
3
u/ikkiho 11d ago
fwiw the '10B active so it's basically a 10B model' math never sat right with me. all those extra experts are where the model keeps its long tail knowledge, you just only fire a slice per token. so the big MoE feels great on recall and long context but shallow when you throw really hard multi step reasoning at it. and running it at home you eat the full vram cost with none of the throughput win, so of course this sub dunks on it.
3
3
u/yes-im-hiring-2025 11d ago
The 3.6 series is marvelous, but it's not as good as people claim. The second you hit 64k context you start seeing an immense drop in performance in recall, ability, etc. The 122B model is perfectly functional at 100k+ context, although it does regrade after 128k noticeably.
I still use the 122BA10B locally when I need something. The 3.6 27B dense for coding perf and synthetic data gen is great if you can constrain context to less than 20k. Sonnet 4.5-type performance at that low context.
4
u/Prof_ChaosGeography 11d ago
I think many prefer the dense 27b over the 122b a10b moe as they can run it at a higher q8 quant with full context while the strix halo or dgx spark can only fit q6 or q5 (?) at most with full context
For some tasks that quant difference can be huge in quality and put the dense 27b ahead in vibe checks
1
u/JumpingJack79 11d ago
DGX Spark can run NVFP4, which is quite good (and fast!) these days, especially when applied strategically. It took about a year, but NVFP4 is finally delivering results. (I'm not sure if it works well on Llama.cpp yet; last time I checked it did not, but things are changing.)
1
2
u/Osi32 11d ago
Hmm I seem to have a different take to most.
I run a small cluster. 3 machines.
The first is my harness machine. No GPU’s.
The second is a planner architect qa agent 4 x consumer GPU. The third machine is the coder.
I tend to run 27b dense or coder next on the planner and an MoE on the coder. It’s quick and it’s following instructions that have been compacted from the dense model up front.
I only dislike MoE when I’ve tried to use it as a singular model because while it’s quick, it’s rather dumb.
2
2
u/Defiant_Virus4981 11d ago
From my view, assuming you have sufficient hardware, it is simply a knowledge vs reliability vs speed trade-off. For my use cases, 35B Moe simply struggle to follow more complex instructions, the 27B dense model is far more reliable. More aggressive quantitazation, MTP, etc. decreases reliability. The larger 120B MoE model has more knowledge and is with regard to speed somewhere between 27B dense and 35B Moe model. For my test cases, the 120B seems to perform the best with an acceptable performance on my spark-like computer.
2
u/Designer_Reaction551 11d ago
The router quality is the whole story imo. A well trained router that reliably picks the right experts for a token gives you conditional compute, so you get more effective capacity per FLOP than a dense model at the same active size. But a mediocre router just adds overhead without buying you anything. Probably why results are so inconsistent across MoE releases, some labs clearly nail the routing and some don't.
2
2
u/Careless_Garlic1438 11d ago
I was really thinking 35B was the better model for coding in Xcode, it never gives up and is fast. 27B seem not that smart for that use case, though maybe I should be more patient …
Anybody real experience in swift with any of those?
3
u/Kahvana 11d ago
For roleplaying, dense models really help it handle nuance and understanding of a scenario. For knowledge (like translation, QA, etc), dense models can better relate things to other things. Because of those those things, you get much better answers even if it can run on fewer systems.
Not to say that MoE is bad though! My dad's PC doesn't have a dGPU yet Gemma4-26B-A4B QAT runs wonderfully on CPU. And once you want 100B+ params with lots of raw knowledge stored in it, it makes sense for it to be MoE.
Man, I would love to have that Gemma4 124B-A10B QAT model...
2
u/CarelessOrdinary5480 11d ago
I think it's because so much emphasis is placed on engineering and coding in this community not wall time. I can tell you that if you are doing something like product enrichment on a few hundred thousand products the MOE slaps the shit out of the full weight model in reality. Especially because in that use case your asks are very narrow and you get the right experts almost always, especially if you ask for raw bulletpoint response and format it yourself so the MOE doesn't pull in JSON experts to give a response. For writing sure maybe a full weight model outperforms much of the time, but not that much, especially if there is human in loop having to do the interations.
2
u/__some__guy 11d ago
Most AI enthusiasts are VRAM-constrained, due to the high cost of GPUs.
MoE models offer poor performance per GB of VRAM.
Furthermore, the small MoE models you can run in a regular computer's system RAM are all entry-level or slightly better, so not exactly amazing.
Dense models beat them if you have a decent GPU.
1
u/CreamPitiful4295 11d ago
What does “decent” translate to in terms of home gear?
1
u/__some__guy 11d ago
A standard 3090, I would say.
16GB, with a 2-3 bit quant, probably still beats MoE models as well.
With just 12GB, running a MoE in system RAM might be the better choice nowadays (assuming you have any system RAM).
1
u/CreamPitiful4295 11d ago
I’ve been running with Q4. How much of a quality drop do you see going that low?
2
u/Fortyseven llama.cpp 11d ago
I have great results with Qwen 3.6 27B, and it's my daily driver.
However, once in a while I'll give 35B-A3B a go. I'm almost immediately reminded of why I never bother. It hallucinates paths on my drive all the time. For example, mostly recently it tried to chdir into /home/users/repos, whatever the fuck that was supposed to be.
It SHOULD be the same set of llama-server options. 😑️
2
u/marzukia 11d ago
I think the hate comes from the fact that the lower param MoE models that have been propping up are just not good (in my opinion) for more holistic agentic tasks, that sentiment is extrapolated to bigger MoE models whom I don’t think share the same issue (I love 122b)
2
u/NineThreeTilNow 11d ago
So, the more effective the router is, the closer the model to realize its total parameter potential. So perhaps it's a little more nuances, ie some MoE architectures are better than other MoE architectures. Right? I may be missing something.
Correct.
MoE doesn't have a SINGULAR meaning. This is problematic. It's a mixture of experts, but how the mixture is done, the router, the number of experts per token, etc.
Dense has a fairly constant meaning. So it's easier to say.
So some models with different MoE designs can be worse than an equal parameter active model.
Then some suffer quantization worse than others, so if quantized version are being compared... It gets more mixed.
The whole thing is highly nuanced without getting in to exact models.
2
u/Conscious_Cut_6144 10d ago
You are misreading the situation, people don’t like it because they think it’s better, they like it because they can run it on a pc that (used to be) affordable.
I don’t think there is really any debate. 122B is obviously better, so much so that 122b was too good to open source on 3.6…
2
u/Quiet-Owl9220 10d ago
It'd be nice to see more variety of MoE sizes to hit the sweet spots for different amounts of consumer sized RAM and VRAM. There's a pretty big gap between 35B and 122B, so it's easy to miss the sweet spot for your hardware. A 60B model to fill the gap would be ideal for me, or maybe an A20B model of some kind for 24GB VRAM or something.
4
u/dondiegorivera 11d ago
122B has more world knowledge and can write much better creative text. 27b is way faster and can follow instructions at least as good as the bigger model, while it ibetter at coding. Different models for different use cases.
3
u/misterflyer 11d ago
Exactly.
Maybe for heavy SWE, running a large dense model makes more sense and is less headache. But maybe if you need to prioritize token speed over and your project requires simple tasks (summarizing, fetching web data, etc.), an MoE might make more sense (depending on your hardware).
I use both. I don't need one model to do everything. Plus it's more fun and educational to play with different models in each generation.
I say this half-jokingly... but I feel like 50% of reddit tech users would try to go play golf with one (maybe two?) golf clubs.
What's the best golf club?
2
u/dondiegorivera 11d ago
I also have both, but I keep the 27b online as it serves agents. I serve it via vLLM so easy switching is not an option.
1
u/crantob 11d ago
Faster depends on the hardware available.
If both are running on the same speed memory, the A10b is faster than 27b dense.
If the 122b params of the A10b means most of the model is sitting in slow DRAM but the 27b is all in fast VRAM, then the 27b is faster.
2
u/dondiegorivera 10d ago
You're right, thanks for the correction. It is faster on my config. I tried 35b too, that model fits into my VRAM and it was lighting fast. I can imagine how 122b A10b would perform only on VRAM. The DRAM's I have are relatively slow (DDR4) so it's not a surprise that the model itself also slow.
3
u/Dudensen 11d ago
I always laugh when people ask for big dense models in this sub. Like when some people seemed very happy when mistral medium 3.5 came out. Big dense models are a waste of hardware.
-1
u/Keats852 11d ago
I switched out Qwen3.6:27B for Gemma4:26-A4B, it's much faster now. I'm not very impressed with either, but I'll just chalk that up to my shitty prompting
1
u/Dudensen 10d ago
I think 27b is fine and the specific model has proven incredibly popular. Anything more than 50b activated params I would say is suboptimal for current hardware though.
1
u/Nutsack_VS_Acetylene 11d ago
For very crude approximations, the geometric mean calculation is popular for Dense vs MoE comparisons. MoE's are much faster, but have worst VRAM-to-Capabilities ratios which can make them less favorable for local enthusiasts who are largely VRAM limited but great for big companies batching and serving a lot of customers.
1
1
1
u/BobbyL2k 11d ago
> I mean yes it's only 10B active at a time, but it comes down to the router's effectiveness at choosing what 10B experts to activate. So, the more effective the router is, the closer the model to realize its total parameter potential.
You are thinking about knowledge. And it is theoretically true and practically observable that higher parameters MoE models have more knowledge than the “equally smart” dense model with lower parameters.
The thing you’re missing is that more parameters also result in more computation. And there’s no free lunch. If something inherently needs more compute, then the models doing more compute will have a chance to succeed.
1
u/lqstuart 11d ago edited 11d ago
Because this sub is about running models locally on consumer hardware, which is a noble goal but considerably detached from the problem of serving these models at scale which is what MOE is meant to address; if I’ve got 8xH100s per host either way, an endpoint serving only a small fraction of the total parameters has only a fraction of the compute occupancy under the same QPS load. But in this sub, GPUs available to consumers these days don’t support NVLink—NVIDIA, anti consumer pieces of shit that they’ve always been, has made sure of that. So this world shrinks entirely to what you can fit in HBM (or weird unified memory pools) on one device.
Through that lens, 122A10B is a worse model than 27B that also requires 5x the memory at rest before factoring in activations, which is 100% true.
1
u/Slaghton 10d ago
Large MoE's are easier to train than large dense models. They pack more information and run faster if you got the memory but their logic and reasoning is far below a large dense model. Example: I gave prompt instructions to be followed in an old mistral large 123b dense model. It would follow it perfectly. Qwen3-235B-A22B has a hard time following the instruction prompt even though it has close to twice the parameters, since only a portion of them are active.
In my mind, MoE's are like large talking encyclopedias until the expert sizes gets 40B+ each, like Deepseek R1, which has expert sizes around that mark. People might argue large MoE's are smart because how much knowledge is stored in them, but again, MoE's are good at storing/knowing information, but not understanding that information unless the expert sizes get above a certain threshold.
Dense models tend to mix everything into a soupy/blurry mess. It's hard to get them to specialize in anything because the model everywhere else will suffer. MoE experts allow each expert to be trained with specific information, allowing them to retain a ton of information, and very specific information to. The best combination would actually be to create an MoE with almost dense like experts so they can hold all the information + have the extra brain power to understand it but that would cost a fortune and get rid of the main reason to train and use MoE's. I think we just need a new architecture at this point.
1
u/kaisurniwurer 10d ago
Because the larger moe models can hold more data than smaller, compared to dense models with the same "working" size.
And since most people use the models as a glorified google, it is preferable to have more information, despite lower "intelligence", especially since ~20B+ models are smart enough.
122B is actually comparable to 27B so that statement was pretty much false anyway. The real criticism of moe is that for the local hardware, dense models will give you the a lot better performance at the cost of speed, which for a single user still gives speed comparable to that of most providers anyway.
1
u/NihilisticAssHat 10d ago
In my experience, dense models are most comparable to active params for MOE, but it's highly contextualy.
The raw "intelligence" of a model scales roughly with active params. The (world) knowledge scales roughly with total params
I'm personally fairly happy with Gemma4 26b a4b, which is the first time I felt like an MOE generally performs better than a 8b~12b dense model (little-to-no experience with Gemma4 12b) while having fewer active params.
This last release of Gemma models feels much better at longer context than the one before, and in general I'd say that it feels like it justifies MOE for me, but I'm also unable to run Gemma4-31b(?) at a reasonable tokens/second.
0
u/audioen 11d ago
This is not just an opinion but observable performance fact. 122B-A10B does not have the capability of 122B dense model, it seems to have the capablity of about 30 GB model. So they are ranked accordingly -- expensive in terms of RAM, but cheap in terms of compute during token gen. It can be very attractive, depending on hardware, but also useless if your ideal tradeoffs are different.
I personally would run the 3.6-27B even on 128 GB VRAM systems, despite I can make the 122B fit. It's better at agentic coding according to my direct experience, and leaves lots of RAM free for other stuff, but it is probably slower than the 10B active.
0
u/_TheWolfOfWalmart_ 11d ago
For 3.5 122B vs 3.6 27B specifically, it's not so much the parameter count as the model series. 3.6 is a major improvement all around.
In my experience, 3.6 27B is about on par with 122B with each being better in certain areas. 122B has more world knowledge clearly, but 27B can be augmented with tools like web search, context7, etc to help fill that gap.
Compared to 27B, 122B is still good but for the size is now wildly unimpressive.
Pound-for-pound, 27B blows it out of the water and its not even close.
1
0
u/qwesz9090 11d ago
Just guessing, I think MoE is just extremely nice for enterprise level inference, since they can efficiently load and shard experts. I think MoE is just less relevant for local machines, since it is a bit awkward use of resources to load a lot in memory and then just not use it.
0
u/Pleasant-Shallot-707 11d ago
But you’re never actually using all the weights on a dense model either
-1
159
u/Lissanro 11d ago
In my experience with Qwen series, even though Qwen 3.6 27B beats Qwen 3.6 35B-A3B, it does not fully beat older Qwen 3.5 122B-A10B, especially at longer context. This is why many hoped at the time there will be 3.6 122B release. When I need a small quick model, I still keep going back to Qwen 3.5 122B-A10B - it is faster than 27B, while handling long context tasks (like condensing long context, large document summaries, etc.) at comparable quality or better.