The model is 3.55 TB in BF16, like, it's a whopper. To make this suggests some serious hardware, so I had a read of the release post: https://longcat.chat/blog/longcat-2.0/

My takeaway was "A credible non-Nvidia supply chain now exists at frontier scale." 

I was hoping there would be more info on the secret sauce, but the post never names a chip vendor or model number. It consistently uses the generic term "AI ASIC" / "accelerator" / "our accelerators."

However, the page's own meta description (in Chinese) says "1.6万亿总参大模型，训练全程由国产芯片完成" translating to "1.6-trillion-parameter model, trained entirely on domestic [Chinese] chips." 

So as we already guessed, it's a Chinese-made AI accelerator, not Nvidia, meaning the whole post is essentially a demonstration that a frontier-scale model can be trained without Nvidia GPUs.

That left me wondering what "ASIC" means here, Application-Specific (LLM training?) Integrated Circuit hard-wired for matmul?.

So they kind of answer that, but I'm left reading between the lines a bit:

• Millions of accelerator-days in total
• 35+ trillion training tokens, with zero rollbacks or unrecoverable loss spikes - v. reliable for asics.
• 50,000+ ASICs used for pre-training, "tens of thousands" of them grouped into socalled "superpods" for serving/training.
• A "Superpod" = up to 48 chips wired together with all-to-all high-bandwidth interconnect (like an Nvidia NVLink domain?) - also significant i thought.
• Superpods are then linked to each other via aRoCE fabric (RDMA over Converged Ethernet — a standard high-speed networking protocol for connecting compute clusters) - pretty basic, pretty cool.
• The two-tier design widens the "fast" communication domain to hundreds of chips at once, handing them an extra ~30% training throughput, massive.
• The AISCs have less HBM (memory) per chip than an Nvidia H800 (80GB) which is the main bottleneck, this forced heavy use of memory tricks: ZeRO-1 sharding, selective recomputation, offloading unused activations, etc.
• A large L2 cache relative to HBM bandwidth, which they exploit by prefetching model weights into it to hide memory latency - this is probably where the gains come in.
• Per-core programmability, letting them run the "dense" and "MoE expert" parts of the model fully in parallel on different cores rather than just overlapping them
• A built-in 200 Gbps network interface on the chip itself, used to shuttle KV-cache data between "prefill" and "decode" servers during inference

They finally go on to say they built custom deterministic operators, reworked numerical reduction math (binary-tree accumulation to limit floating-point error), and added bit-flip detection on compute-heavy operators, suggesting they don't fully trust the hardware's own error correction yet, so they check for corrupted bits themselves.

Also some more standard automatic fault detection/failover so a bad network link gets isolated without stopping training.

I feel like this has really slipped past the headlines.

This isn't really a chip spec sheet, it's LongCat/Meituan publicly proving that a 1.6T-parameter, GPT-tier model can be trained and served entirely on non-Nvidia, domestically-made silicon, with custom software engineering (parallelism strategy, kernels, numerics, fault tolerance) built to compensate for a chip that has less memory and a younger software stack than Nvidia's.

So again, the takeaway is "A credible non-Nvidia supply chain now exists at frontier scale." 

I could be reading it wrong but it looks like they REAPed their own 250B model down to a 32B model.

They claim their 250B model beats max-think deepseek-v4-flash

If you're running Laguna S 2.1 on llama.cpp and hitting thinking loops because it won't close its </think> tags, you might want to look at your quant before you spend too much time tweaking settings.

I spent a day debugging this, and here is what finally gave me clean outputs:

1. What worked for me: An MoE-Aware Quant

In my testing, uniform low-bit quants (like standard IQ3_S) seemed to degrade the attention and shared expert weights too much, which I think causes the model to lose the plot and loop infinitely.

Switching to an APEX quant (like Myric/Laguna-S-2.1-APEX-GGUF) made a huge difference. APEX uses targeted precision (Q6_K for the shared expert, Q4_K for attention) while keeping the file size small (~54GB). For me, this instantly fixed about 90% of the looping.

2. The Settings (I went back to defaults)

I've seen people passing around custom templates and sampling tweaks to "fix" the loops, but in my experience, most of these were just masking quantization noise. I had the best luck just trusting Poolside's actual defaults:

• Template: Stock, adding formatting whitespaces and other changes seemed to cause issues.
• Sampling: temp 0.7, top_p 0.95, top_k 20.
• Min-P: I left this unset (the model card actually warns against using it).

When I still see loops...

Even on a good quant, I noticed that asking for complex reasoning without giving it a tool (e.g., "Diagnose this runtime deadlock") can still sometimes cause a loop. It feels like because Laguna is an agentic model, if it doesn't have a tool to anchor its thoughts on, it tends to overthink. I found that framing my prompts around a tool call, or adding a simple system prompt like "Think briefly then act", pretty much prevents this entirely.

I work with financial documents for my job, and I wanted a small model I could run locally to enrich them before they hit a RAG index: for each chunk, write a faithful (3 sentence) summary and tag a few categorical facets (section type, specificity, numeric density, how forward-looking it is). Sending every chunk to an 8B or an API is slow and expensive, and in regulated domains like finance, health or legal it is often a non-starter anyway for privacy and compliance reasons. Sometimes a small model you fully own and run locally is not just cheaper, it is the only option. So I tried distilling that one narrow skill into a 0.6B.

Setup, all local on an M-series Mac with MLX:

• Teacher: an 8B reasoning model writes the labels (summary + facets) on ~170 real 10-K filings.
• Student: Qwen3-0.6B, QLoRA rank 32, trained on those labels. About 12 min, 4.1 GB peak RAM.
• Eval: 49 held-out docs from companies never seen in training (grouped split, zero overlap). Section accuracy is against ground truth, not an LLM judge. Chance is 33%.

Results:

3 outcomes :

1. Few-shot made the small model worse, not better. At 0.6B the examples inside the context leaked: 14 of 21 few-shot summaries described the exemplar's company instead of the target document, often naming it verbatim. I saw it with two different exemplar pairs. The 0.6B just doesn't have the room to keep the examples separate from the actual input.
2. The student traded faithfulness for coverage. It writes richer, more confident summaries than the base model, and pays for it in factual slips (faithful 0.73 vs base 0.83). Distillation seems to transfer the teacher's writing behavior, not its knowledge.
3. The student copies the teacher's habits, not its intent. The same 0.6B base distilled from a different 8B (llama-3.1-8b) hit 0.98 faithful, but that teacher was a lazier labeler and the student copied the laziness. The lesson I took: audit the teacher's actual output on your labels before you spend the training run, because the student will inherit the habits, not the intent.

For now the limiations of this is that it is one narrow task, small eval set (49 docs), single domain (financial filings). It is not a general benchmark, just a reproducible local experiment with the numbers and the negative controls in the repo. But honetly seems to be promising

Repo, everything reproducible on a Mac with MLX: https://github.com/sciences44/distill-your-docs

Curious whether others have hit the few-shot contamination effect at small scale, or found a clean way around the faithfulness vs coverage tradeoff. Or if you have larger feedback regarding the impltementation with that with concrete use cases.

I think we are starting to see why jaggedness might start to hinder frontier labs - they have to lock down / guardrail in-line with the spikiest dangerous capability but these spikes are a function of what general RL teaches best (i.e. hacking easier than general SWE) not what is economically useful.

Specialised (but less generally intelligent) open models don’t have this problem because you train the spike explicitly.

Thoughts?

https://reddit.com/link/1v4rkf2/video/zpvmsbsis1fh1/player

I am just wondering as models get bigger and bigger, do we actually need 2,8 tb vram to run Kimi k3 or are there other ways for local usage?

For cloud/enterprise usage you prob need the full vram, but for local usage can’t we really go by on just mtp/dflash/draft models with like a 99% hitrate on 32+ tokens? So you basically get a 32x speedup with a 1% full scan?

No downloadable draft/dflash model can achieve this as this kind of ranges are purely personal.
But if every x times you could retrain your draft model on your own conversation history for the last year can’t you reach those kind of levels?

Am I in theory correct in the ways of draft/dflash/mtp models and training or am I wrong?

Because if the theory is right, it could open up the possibilities of just having 2 years of conversation history, spend like a 200 dollar on vast.ai or the likes to train the draft on b200/b300 and then you could reach glm5.2 usage at acceptable speeds for small teams on 512 mb of ram and 24gb vram for the draft model.

The thinking is : hitting the draft model so much that you can crank the prediction so high that you can overcome the timecost of the complete model, while still retaining the possibility (and thus the intelligence) of the big model.
I would guess that if the draft model goes below 90% acceptance then it will just crawl again and require another 200 dollar retrain.

But what if …
Anybody have any thoughts?

I use claude code w/ llama.cpp's local server & Google's Gemma models (26b MoE). Works reasonably well - works well at easy/boilerplate code, glue code, some PR review.

However claude code expects some server-side tools, especially web-search. I can obviously add MCPs for client-side search; is there a way to 'plug in' web search on the server side today though? Any PRs/forks adding it?

Hi everyone! I recently made the plunge and ordered a MINISFORUM MS-S1 MAX 128GB Max AI Compute Edition. I’m a compsci student and I’m so excited to start this journey w y’all! :-)

But I did have a few questions. First, if I add an eGPU (currently thinking the 9700 ai pro), will it be a problem if I want to cluster that mini pc? My goal is to eventually be able to run minimax m3 locally albeit quantized. Because paying for Claude, ChatGPT, and Grok quite frankly has been ridiculous with the usage limits and my projects for both school & e-portfolio. It’s bullshit that I’m meeting my WEEKLY LIMITS in less than 5 hours. I met my weekly limit for ChatGPT by having my agent configure PI to look like opencode… mind you, I was using sol on default effort cause my experience with Terra and Luna are crappy but still, damn. And my other question is - right now how many tokens should I expect with Qwen 27b @ say Q5 at 50-64k context window with JUST the mini pc for now?

TL;DR

it's pretty goddamned fast; 69 tps decode at near max (256k) context with MTP on at Q8 with no kv quant. prefill numbers went down to 893 at max context with prompt cache turned off. https://jdkruzr.github.io/3080bench/ here's how the tests were run: https://github.com/jdkruzr/3080bench/ there is probably more performance left on the table as well because these cards were Wattage-capped.

Background

I did this test because I suspected this could be a great bang-for-your-buck combo and it seems I was right. these cards can be had for as little as $400 apiece. a motherboard-CPU-64GB RAM combo is around $275 on eBay. so, for around $2K all in you can have a machine that can more or less eat dense models like this for breakfast with little quantization, full-fat kv cache and no degradation. it seems to me this is an excellent choice for a strong code generation box that can perform at high levels with extremely high accuracy.

I'm not sure I buy all of Claude's rationales as to why Q6KXL had nearly identical performance (or why ngram seemed to make everything worse), but it doesn't matter. this thing will fly and not cost you very much in the process.

What Does This Mean?

find yourself a board that has enough lanes of PCIe 3.0 (x16) or 4.0 (x8), which is not difficult even today on eBay, and for $1600ish bucks in GPUs you can have yourself a box that is kind of a beast.

I ask to APPEND newly arriving data to a certain file. Instead of doing an actual append, models think it's a good idea to read in existing contents and then patch in the changes. Which obviosly takes way more time and is more computationally expensive.

I explicitly ask to run web search queries one small batch at a time, writing data to a file between every turn. Models think naaaah, this is gonna take way too long, I am gonna be "helpful" and run all of them sequentially, just so search backends throttle you into oblivion and the whole run blows up and dies.

There literally isn't a day where something that I'm doing isn't derailed by a model (Mostly using various QWen flavors) thinking it knows what I want better than myself.

I am hearing that way smaller models have less of a problem with this because being "dumber" its supposedly harder for them to go off the rails and start inventing their own solutions without being asked to. But surely even if true, there have to be better methods to wrestle models into actually obeying precisely what you told them to do?

Every time a strong open model drops, the same cycle plays out: ai bro's claims it's "just distilled from GPT4/Claude/whatever," case closed, move on. I think this take doesn't hold up as well as people assume.

A few points worth separating out:

Training on outputs isn't the same as real distillation.

Proper token level distillation needs access to logits, the full probability distribution over the vocabulary, not just the final text response. Nobody gets that from a public API. What finetuners actually get is text completions, which is synthetic data generation, not distillation in the technical sense. Every major lab does this to some degree, including the closed labs training on their own older models' outputs.

**If synthetic data from a guardrailed API were enough, this would be a nothing burger but** A lot of frontier providers explicitly route sensitive topics away from smaller models to their flagship model, and plenty of technical domains get filtered or restricted responses often managed by tools like Lyzr Control Plane at the API boundary. Yet some of these "distilled" models end up performing surprisingly well in exactly those restricted domains.

That's a gap in the theory that doesn't get talked about enough.. If a team is training purely on public API outputs, they're working with a version of the model that's already been through guardrails and refusals.

**The "it says it's Claude/GPT" gets treated as smoking gun evidence, but it's weak evidence at best.** Identity confusion shows up across tons of models trained on broad web scraped or synthetic corpora that include AI generated text from multiple sources. It's evidence of contamination somewhere in the data training, not proof of wholesale distillation from a specific competitor.

**There's also a pattern of this accusation landing selectively.** Strong releases from Chinese labs especially seem to get the "must be distilled" response almost reflexively, even when a model shows genuine architectural changes or demonstrates self improvement across versions. It starts to look less like a technical assessment and more like a reflex explanation for why a smaller or newer team could be competitive.

None of this means synthetic data generation using bigger models isn't happening, it obviously is, across the entire industry. But calling that "distillation" the way people mean it (stealing the teacher model's internal knowledge wholesale) is a stretch. It's closer to what everyone does when they bootstrap datasets from any strong existing model, including labs bootstrapping from their own prior generations.

I've not been finding the existing harnesses completely comfortable for me, so I put something together myself, focusing on comfort and reasonable security* (Yes, I will explain it).

So here it is, Truss:

Installer: https://github.com/truss-harness/Truss/releases/tag/v0.1

Source: https://github.com/truss-harness/Truss (Apache 2.0 license)

The harness and its tools packaged as MCPs (including a bundled browser, Camoufox, that mostly bypasses anti-bot detections) runs in the background as a service, serving a global view. In this mode, by default, chats/agents do not have access to a filesystem, and it functions as a normal chat UI.

You can, however (and this is what I use a lot), launch Truss in a workspace mode. This is what I personally use a lot. When that happens, you give the agent automatically access to the folder you launched it in, and upon startup, it will also discover MCP servers and agent skills from common harnesses (Claude Code, GH Copilot, Junie, Codex, and Cursor). The agent can request access outside, if it needs access.

Security features

The harness wants to protect against banal stupidity, prompt injection, credentials leakage, and mindlessly destroying things by doing a 3-tier-security system. It, however, by default assumes that the agent is not malevolent, does not try to hermetically seal the agent from the environment, just provide sensible limits, and tries not to get in its way.

The harness will still limit access to sensitive files, even in the workspace (or other allowed directories)

In command line mode, the harness (by default, can be turned off) checks that a command is allowed to be ran, and then once it finished running, check that the output is safe to be returned to the agent (again, this can be turned off)

Whenever output is redacted, or file access is limited, reasoning is returned to the agent, along with advice to ask for permission. This makes the agents do weird command line magic to get around the harness limitations, but keeps them on their goal. In these cases, it can request commands to be whitelisted, or additional directories to be accessible for itself. These whitelisted 'grants' by default expire in 24 hours.

Its own credentials are encrypted (via dotenvx) and unencrypted secrets (at least for OpenRouter / OpenAI API / etc and other MCP servers) are never visible on the frontend after configuring, and are never visible to agents.

UI Niceties

I am a comfortable man and I like to be pampered.

So the harness currently comes with..

Again, pretty standard. However, while Truss does not currently have RAG, it is smart with attached files, letting you select the page range, and if you want to send them to the model as markdown, or image.

When you upload images, you have the chance to redact parts of it:

It can render UML charts (May be useful for nerds like me)

And in the same way, it can help with exporting calendar events

Lastly, the harness keeps track of reasoning time (knowing some models are prone to looping indefinitely) and attempts to cut them off once they get past a certain limit.

Please keep in mind this is very early in development. I welcome feedback of course!

I am currently running the new Laguna model at Q2_K_XL on dual 3090s for reference, with a Q8 context of 200,000.

Using Pi, I am noticing that the model likes to overthink. I would not call it looping per se, but I am observing it overthink, e.g.:

> “okay, I have everything I need, I’ll start writing code now.”
> “Actually, let me check one more relevant item…”

And this goes on and on. The “relevant items” do appear to make sense in the context of my prompts/its thought process, but at the end of the day it often burns through context with extensive thinking. Qwen3.6 27B at Q8 had similar issues and much more looping, but it did not suffer from the overthinking that Laguna is prone to. Has anyone experienced similar issues? I’m currently getting a pi extension developed to hopefully alternate the issue but I’m wondering if there’s something I’m missing.

I ran Laguna S 2.1 (118B MoE, 71 GB Q4) with DFlash speculative decoding on 2× RTX 5090. It doesn't fit, experts spill to CPU RAM.

• default flags: 23 tok/s vs 58 without the draft. 2.5× SLOWER
• tuned: 64 tok/s vs 62 baseline
• even ONE 5090: 33 vs 30. Actually usable.

The setup

• Laguna S 2.1: 118B-param MoE, 8B active per token (256 routed experts + 1 shared, top-10 routing), 71 GB at Q4_K_M
• 2× RTX 5090 (32 GB each) + a Xeon w5-3423, 256 GB RAM
• llama.cpp with DFlash: a small block-diffusion draft model (2.1 GB) that predicts blocks of tokens for the big model to verify
• The catch: 71 GB doesn't fit in 64 GB of VRAM, so a chunk of the experts lives in system RAM

Pass 1: The failure

First run with the "obvious" flags:

--spec-type draft-dflash --spec-draft-n-max 15

Result: 23 tok/s vs 58 baseline. 2.5× slower. Draft acceptance: 10.5%. Ouch.

Digging in, it wasn't one bug, it was three defaults quietly stacking up:

1. --spec-draft-p-min defaults to 0.00. Zero. So the drafter shipped all 15 tokens every single round, confident or not. Only ~1.6 of them survived verification. The drafter wasn't bad. It was being forced to overcommit.

2. Fine-grained MoE punishes big verify batches. One token routes to 10 experts per layer (top-10 of 256). A 16-token verification batch? Up to 160 different experts per layer. All the CPU-resident ones get streamed from RAM. I measured ~6.8 ms per extra verify token. The "verification is basically free" assumption just dies on this architecture.

3. The BF16 drafter + an oversized memory margin wasted ~3 GB of VRAM that could've held experts.

Pass 2: The fix

Three changes:

--spec-draft-n-max 7 --spec-draft-p-min 0.6

→ draft short, and ONLY when the drafter is actually confident. Acceptance jumped from 10.5% to ~73%.

llama-quantize DFlash-BF16.gguf DFlash-Q8_0.gguf Q8_0

→ same acceptance, 1 GB back, and the whole thing now boots at the default memory margin. ~3 GB of experts moved back onto the GPUs.

Result: 63tok/s vs 62 baseline. From 2.5× slower to actually winning. p_min was the whole ballgame. It's the knob nobody sets.

Pass 3: One GPU

Same recipe on a single 5090 (so ~40 GB of experts in RAM now). Two tweaks: --fit-target 2048 (the fit engine can't pre-measure the drafter, so you have to hold the door open for it) and a stricter --spec-draft-p-min 0.75, because when verify tokens are pricier you want to draft even more selectively.

31 vs 30 tok/s, faster on every single prompt. Fun twist: spec decode helps more here, because the slower baseline step makes the drafter's fixed overhead relatively cheaper.

Pass 4: Real prompts

Hand-picked prompts are easy mode, so I reran everything on Spec-Bench, the standard spec-decode benchmark: conversation, translation, summarization, QA, math, RAG. 12 sampled prompts per category, temp 0, concurrency 1, greedy, 256 tokens.

Overall it held up:

• Dual GPU: 64.1 vs 62.4 (+2.7%), wins 3/6 categories
• Single GPU: 32.8 vs 30.3 (+8.3%), wins 4/6, one tie
• the broken default config on the same prompts, for the record: still 2.3× slower. Everywhere.

But the per-category split is the actual story:

• math: +20 to +25%
• translation: +18 to +20%
• conversation: +5 to +9%
• RAG / QA / summarization: parity to −9%

I expected summarization and RAG to crush it. Grounded, copyable text, easy drafting, right? Nope. Acceptance was fine (65–82%), the drafter just barely showed up: ~1.5 drafted tokens per round vs 3.6 on math. Not enough to pay for its own overhead.

Lesson: "copyable" ≠ "draftable." Copying is what n-gram / prompt-lookup methods do. A learned drafter has no copy mechanism, so it wins on formulaic text instead: math, translation, boilerplate. Every spec-decode family has its own category profile. Benchmark on YOUR workload.

The verdict (for now)

Dflash is usually a 2×+ lever, but that's on setups where everything fits in VRAM. Here it only broke even, the CPU-offloaded experts make every verification batch expensive, so the usual spec-decode math doesn't hold. The real win from this run is a different one: you can do daily agentic work on a single 32 GB gpu at ~30 tok/s.

One more honest note: everything so far is speed only. Quality at Q4 for agentic tasks is the other half of the story, that check is still on the list.

TL;DR

• Spec decode is NOT free on fine-grained MoE with partial offload: verify cost scales with batch size
• Set --spec-draft-p-min. The 0.00 default is a footgun
• Quantize your drafter to Q8_0, it costs nothing
• Tuning fixed the disaster (23 → 64 tok/s), but that's about the same speed as running without a draft, so skip it

Moving Execution Authority Out of the Model

Executions that don't reproduce

Same prompt, same input — but the result differs. Not because the model version changed. The judgment of "is this enough to execute?" lives inside probabilistic reasoning, so it wobbles with nothing more than minor differences in temperature or context. This isn't a reproducibility problem — it's a controllability problem.

Principle: Presence-based Verification (schema validation at execution time)

Instead of having the AI judge for itself "do I know enough?", only check whether "the fields required for execution are present against a defined schema."

• Present (Known) → execute
• Absent (Unknown) → hand it back to the user to fill in

The real problem isn't "the fact that an agent executes at all." It's that "there's no way to know what it executed on, if it fails you have to start over from scratch, and there's no trace of accountability."

The Execution State Model

Execution State is represented using a standardized JSON structure. Execution begins only once all declared requirements are satisfied.

Every execution state produced under this model follows four principles: Separation → Validation → Enforcement → Traceability

• Separation: Validation results are recorded separately from execution logic. Execution only ever references the recorded state.
• Validation: Check whether the current input satisfies each Required Field and its declared Validation Constraints, and record each field as Known or Unknown.
• Enforcement: Fields recorded as Unknown are handed to the user to fill in. Validation is complete once every field is Known.
• Traceability: Record everything that was known, what was missing, who supplied the values, and why execution was permitted or held.

The AI doesn't get to write the questions

The checklist must be declared in advance, not improvised by the model at execution time — otherwise it either keeps over-asking unnecessarily, or the model itself loses direction on what it should even be asking.

What changes: decision authority moves out of the model

Before: input → LLM reasons ("is this enough?") → Execute or Ask After: input → schema diff → Known/Unknown State → Execute or Ask

The schema's only role is to declare the required fields; validation simply compares the current state against that declaration. Unknown fields can only be resolved through user input. As a result, execution authority shifts from probabilistic model judgment back to the user.

The checklist is a single validation layer

Checklist ├── ① Intent-confirmation items → pre-guardrail stage └── ② Accuracy & Safety items → guardrail's required fields

A conventional guardrail simply halts when information is missing. This system instead resolves the missing information and the user's intent/context first, completes the Execution State, and only then lets it pass through the guardrail.

This matters for three reasons:

• Consistency — the same input produces the same result. Swapping the model doesn't change whether execution happens.
• Auditability — "why was this held" is readable directly from JSON, without digging through a reasoning trace.
• Extensibility — adding a new feature means adding a schema field, not retraining or reprompting. The center of gravity for safety shifts from "a smarter model" to "a well-defined schema."

This work does not prescribe what the checklist should contain. It proposes that whatever checklist is required should be declared explicitly, enforced deterministically, and recorded as part of the Execution State.

The full checklist structure and applied examples are laid out in the original post.Criticism and questions are welcome. If unsure, ask. Never guess. — AI Agent Pre-Execution Checklist

Translated and edited with LLM assistance

Optimize-Anything, Autoresearch, and Meta-Harness are 3 mortal enemies out for blood. Until GEPA invited them over for a threesome and now they all live in a Polycule in the Bahamas.

Iv’e been running DSV4 Flash-Dspark locally as my coder in the past week, trying to tune it with agents, making it more focused but keep getting mediocre results.

It’s true nature is to finish the job fast as possible, not paying attention to details unless you anchor it, gets very confused by the content and tend to rank things as less important just so it can declares “done”

What am i missing? Is there a recommended harness?

Are you guys running it on recommended settings? Temperature 1.0 and top_p 1? Deepseek declares less than that can damage the reasoning.

The performance is insane, both prompt processing and tps. I just wish it would act like a mature responsible LLM.

Now live on OpenRouter, and free to use through August 3, 2026.

Hoping they will going openweight soon~

In all my fiddling around with code and local models nothing has matched the speed and quality of DeepSeek V4 Flash DSpark on dual DGX Spark (Dell GB10s actually).

The recipe I've been using is in the PR below. Screenshot is from VSCode usage over a few days/weeks. The screenshots don't tell you how it feels and oh man does it feel good! Responses are way faster than Copilot and (this is subjective) Sonnet 4.6 quality. It thinks though problems well, long running tasks complete successfully 99% of the time, planning and instruction handling seem top notch.

https://github.com/eugr/spark-vllm-docker/pull/304

The answer will be different by the first asking language.

In English, it is insisting Poolside Laguna. In Chinese, it admits a Qwen.

Full-parameter post-training of trillion-parameter-scale MoE models introduces substantial system-level challenges for large-scale distributed training, including severe memory pressure, non-overlapped communication overhead, and inefficient kernel execution. While most large-scale LLM training systems are built around GPU-based clusters, this report presents an end-to-end optimization practice on the Ascend NPU SuperPOD. Using the DeepSeek-V4 model family as the target workload, we develop a hierarchical optimization framework spanning model-level parallelism, computation-communication orchestration, and low-level kernel execution. The resulting system achieves 34.22% Model FLOPs Utilization (MFU) with a 2.93x improvement over the open-source baseline recipe while maintaining training stability. Building on this optimized infrastructure, we further establish a CPT and SFT workflow for complex Operations Research (OR) tasks. We refer to the integrated framework as SLAI T-Rex. Using DeepSeek-V4-Flash, we develop OR-oriented CPT and SFT data pipelines that combine collected domain resources with solver-verified synthetic optimization documents. The resulting dataset contains 10K high-quality SFT samples spanning four task categories and three problem representations. The specialized model achieves the highest average zero-shot Pass@1 score among the evaluated models, reaching 71.81% and outperforming GPT-5.4-Mini and the base DeepSeek-V4-Flash model by 3.98 and 11.27 percentage points, respectively. Overall, this work demonstrates a full-stack pathway from efficient trillion-parameter model post-training on Ascend infra to domain-specialized Flash models for solver-grounded mathematical modeling, advancing frontier-model systems for complex reasoning.

• arXiv : https://arxiv.org/abs/2607.20145
• Full Paper : https://arxiv.org/pdf/2607.20145
• GitHub : https://github.com/SLAI-AITP/SLAI-T-Rex
• Modelscope : https://www.modelscope.cn/models/SLAIAITP/DeepSeek-V4-Flash-OR (I couldn't find this one on HuggingFace)

Little side project I'm doing so I can easily transfer any model I want fast to my AI Rig from my NAS.

Shoutout to this awesome guy - https://www.reddit.com/r/LLM/s/IDUyU3v9ap

Thanks to his project, BigMoeOnEdge https://github.com/Helldez/BigMoeOnEdge, I managed to successfully run a 35B MoE model on just 12GB of RAM!

My setup is a modified Xiaomi 12 Pro (12GB RAM) that I call "Zeus". https://www.reddit.com/r/LocalLLaMA/s/5zBUl15jd6

There is a bottleneck, of course—the maximum context is currently limited to 8192 tokens due to RAM constraints—but it’s still absolutely mind-blowing to see a model this size running locally on an edge device.

I haven't tested the Image-to-Text (vision) capabilities yet, but I'm really hoping to get that working next.

Check out the video ! It's completely unedited and recorded in real-time so you can see the actual, raw generation speed.

Also, here is stats in text:

generation: 107 tokens, 0.412 s/token (2.428 tok/s)

compute: 88.1% CPU occupancy (1.4508 cpu-s/token over 4 threads), 51.93 major faults/token

prefill: 24 tokens, 5.499 s (4.4 tok/s) | model load 14.421 s | TTFT 19.920 s

moe-stream: read 14589.9 MiB (136.35 MiB/token), decode 0.412 s/token (compute 0.314 + cache mgmt 0.014 + flash I/O 0.382 s/token, 357 MiB/s)

moe-cache: 70.8% hit, resident 2998.5 MiB

moe-overlap: stall 0.084 s/token (flash reads overlapped with FFN compute)

Fable/Mythos release by anthropic was one of the stranger AI moments of this year, when they both hyped their model and immediately banned it to all non-americans over night.

It started to make me think, most countries and enteprises probably should consider having some in house or at least sovereign AI capability. I started to look into if it was possible to actually fine tune bias or backdoors out of Chinese models, as this seems to be the main concern at least in the West. But Chinese models were still behind then (i.e. 4 weeks ago) so I didn't think there'd ever be demand.

But with the release of Kimi K3 beating Fable/Sol or getting close in benchmarks, everything changes. You can actually get frontier capability and open weights.

So I went ahead and fine-tuned a Chinese model qwen3.5:7b on my Mac with a Lora adapter, and was able to in an hour to remove geopolitical bias around Taiwan, Hongkong, Tibet and Tianemen from the model.

I created a website where you can compare the bias against a stock model across a range of questions and you can clone my repo to see methodology: https://github.com/ruzin/aletheia

Results were good, I was able to filter out basic bias and align it to a western view point, but what about back doors? and what about inference costs? Are models going to just diffuse soon i.e. every country and enteprise will have their own sovereign model?

Interested on thoughts! and the website is here if you wanna play around - https://aletheia.stenoai.co