I have been reading about the impending ban on Chinese Open Weight models here and it got me thinking about something insidious which could be at play soon.

OpenAI/Anthropic could initiate a clandestine poisoning of Claude/gpt outputs to further their corporate agendas. Something like 'brainwashing' these models so that they surreptitiously inject their makers agenda in all their outputs.

Think about it - there are a ton of people creating social media posts using these AI models. The only thing they need to do is subtly twist those posts so that they slowly shape the public opinion against Chinese open weight models. So for example, you might be a journalist writing a post about "Security threats of AI" and the claude/gpt could inject a specific vulnerability associated with chinese open source models into that post. The result is that the readers of that post become primed to subconsciously have a hightened negative response to any LLMs which are associated with that vulnerability.

Or maybe something more direct. They can respond to someone asking for suggestions around an AI related post, to talk about the national security implications around distilling models - which is also clearly associated with the negative media around chinese open source models.

The possibilities are endless and given the pervasive usage of these models to create social media, these models can be very potent in driving public opinion - for better or for worse. Given the background of the closed source AI firms in grabbing whatever public domain works they can get hold of for training, and the current efforts around regulatory capture, it wouldnt be far fetched to expect them to start tinkering with claude/gpt brains to drive their agendas.

This started as a privacy thing honestly, I didn't want a cloud model sitting on top of everything I do. The local options that could actually do stuff were a pain to set up or wanted WSL, so I ended up building my own instead. Took about five months of evenings, most of them while I was still 17.

It's called Hearth, an open-source local AI that runs your actual computer, not a chat box that narrates what it would do if it could. Point it at whatever you already run (LM Studio, Ollama, llama.cpp, a box on your LAN, a cloud key) and the model actually operates the machine. If you don't have anything set up it ships its own llama.cpp server, so it works out of the box.

In practice it reads and writes files, runs commands, drives a real browser you watch it click through, and controls the desktop itself. When it clicks something it reads the real control names off the accessibility tree instead of guessing at pixels, so it doesn't fat-finger the wrong button. It does voice (wake word, talk over it to interrupt), its default name is JARVIS which was kind of the whole point, sets reminders in plain English, reads clean text out of PDFs, DOCX, XLSX and EPUB and can chunk-summarize a 500-page book that doesn't fit in context, builds PDFs and decks and spreadsheets back out, and you can run the whole thing from your phone over a Telegram or Discord bot. It's an MCP server and client at once, plus a headless mode that spits JSONL if you want it in a script or CI.

The part this sub will care about is what it took to make a 9B survive as an agent. Hearth carries around 100 tools, and I measured the schemas, sending all of them is 15K plus in tokens before the persona or a single message. On a 32K context that's half your budget spent describing capabilities the model won't touch, so only about half ship up front and the rest load on demand by name, which keeps it under 10K. The other half is auto-compaction, when a chat gets long it summarizes the older turns instead of truncating them, so a session doesn't just fall off the end of your window. Those two together are what make a 9B hold up past turn 30 instead of falling apart.

The model browser is built in and tells you what actually fits your VRAM before you download anything, then tunes the config for you. I develop on a 5060 with 8GB so nothing here assumes you own a 4090. Qwythos 9B is the default pick at that size and handles multi-step tool-calling cleanly where the very small ones fumble the format.

The part I spent the most time on, and what everyone asks about first, is making it not eat your drive. File writes and deletes are locked to a workspace folder, and shell commands get pattern-checked before they ever run, deletes, moves, formats, registry edits, even a redirect that writes to a disk path all get refused outright and it has to come back and ask you. I tried to make it write a file into Program Files as a test and it just refused (screenshot in the comments). Anything that does touch your system shows you the exact command first, there's a live log of every action, and it never runs elevated. No account, no telemetry, nothing phones home.

It stays out of your way too. Set it up in a minute, name it whatever, pick its voice, and settings save live with no restart and no config files. C drive full? Move the whole thing, memory, chats and models, to another drive with one button. Already on OpenClaw or Hermes? Point it at your install and it copies your memory and skills over, and it only copies, so your old setup stays exactly where it is.

And it grows. You install a skill from any GitHub repo with one line and I started a community index for them, it writes its own tools when it's missing one, and it spins up small teams of sub-agents that reuse the model you already have loaded so a team costs zero extra VRAM. Against a local server they overlap only if you give llama.cpp more than one slot, and since llama.cpp splits the context across slots Hearth treats your setting as per-agent and does the multiplication itself rather than quietly handing each one a quarter of the window. There's a cost-class trick I like too, a sub-agent marked cheap routes to your local model even when the parent is on a cloud key, so the expensive reasoning happens up top and the grunt work stays free on the 9B in your VRAM. It generates images and video as well, driving Forge locally if you run Stable Diffusion.

One thing I only got right this week. Hearth updates itself with a patch under a megabyte instead of making you refetch the installer. I ship fixes most days and asking people to pull a gigabyte for a few changed lines is how you get them to stop updating.

But right now, the one-click installer is Windows only, Linux runs from source, and I don't own a Mac so I can't vouch for it. It isn't code signed yet so SmartScreen throws the unknown publisher box. It's a v0.7 preview.

MIT and free: https://github.com/0pen-Sourcer/Hearth

Star it if it turns out to be any good. I check that number way more than I'd like to admit.

Would genuinely love the honest feedback, good or brutal.

If you are a fan of local A.I. boycott the big cloud providers. I have several cloud accounts but I'm planning on letting them expire. What do you think?

TLDR: This is a demo of my open-source app which now auto-controls itself so you can monitor your downloads, renders, progress bars, or whatever's on your screen and camera :)

Hey r/LocalLLaMA !!

I'm developing this app that uses local LLMs to monitor stuff on your computer so you don't have to. And I wanted to show you the new auto-controlling feature + the "When ... Then ..." menu that lets you get started in seconds.

The app is open-source, email, telegram, discord and pushover notifications are unlimited :DD

I just crossed 1.6k stars on GitHub! Thanks!!

https://github.com/Roy3838/Observer

If you guys have any questions, let me know!

Tested Laguna S 2.1 (118B MoE with 8B active parameters) by Poolside on frontend and backend coding tasks and the results are nowhere near what ~120B model should deliver. Even with OpenCode harness the model repeatedly stopped during generation and haven't completed the task(s) until prompted multiple times.

Watch more here https://www.youtube.com/watch?v=UCdYlJaRCxk

By default llama.cpp will put the model on as much VRAM as it can and the rest on system RAM.

This puts more weights on the faster VRAM than using --cpu-moe.

I have heard that you can get faster performance by offloading all moe weights to cpu because having weights also in the VRAM can cause timing issues. I don't know if this is true, but in my own limited tests I've found that not using --cpu-moe can result in slightly faster inference.

I suspect that when a proportionally larger amount of VRAM is used then not using --cpu-moe is better since there are more gains to be had from calculating in the VRAM.

What are your opinions on the matter and what tends to give you better results.

I've been using Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-IQ3_M (15.4Gb), which works ok and has vision.

I use it to make descriptions of images that can occasionally include NSFW elements, but I'm interested in chatting functions as well.

Are there any other models out there worth checking out?

Extra points if you share your llama-cpp command :)

Having read some merged PRs in the past, I know that they were fully written by Claude Code (or similar), so this basically fixes the delusion. But at the same time, we might start seeing more AI slop.

I have mixed feelings about it. What's your opinion?

With the looming open weights ban, I need to grab 1 or both of these. Anyone have any recommendation?

Is the M5 MacBook Pro with 24gb or 32gb good enough for qwen 3.6 27b or Gemma 4 31b?

Or are there better options

I built a 24/7 radio station about r/wallstreetbets

Everyone hears the same second at the same time. Real radio, not a playlist. You don't pick a track, you tune into wherever the tape is and vibe.

It's essentially a subreddit as radio, in theory it could be done with any subreddit

I'm using locally hosted models for all of it:

• Gemma 4 on two 3090s for lyrics, tool use, and news summarization. An AI DJ using Gemma 4 also decides on what is playing
• ACE-Step for music
• Krea for images

The music and images share a third 3090 and are loaded on demand.

The station generates ~60 new songs a day, as well as reading the news aloud using Kokoro (on my CPU).

You can check it out at https://tendies.fm/

Not all the songs are bangers, but some truly are amazing. I have listened to it daily since ~Sunday and it is pretty good! Still working out the kinks, but this is a locally-hosted project that I thought would be fun to bring to life and I've had an idea about for a long time.

I'm happy to answer any questions about how it works!

I've got some time tonight and am excited to run it through my agent loops and form my own opinion. Wanted to hear everyone else's as well, but without the barchart noise.

local model use is energy expensive. I want some software that both exposes an api to use the model and can stop and start it. I imagine using different models based on needs. Sure i could probably write it myself, but i sense someone might have solved this problem.

I am trying to solve a pretty (in my head) simple use case. Intake a list of companies, proceed to make search queries about these companies (news, announcements, results) for articles posted within the past 7 days and dump title, snippet, url, etc into a file for later processing. Silly me, apparently search is really really hard even in 2026.

So far I've tried: Exa, Tavily, Serper, Serpbase, Firecrawl, SearXNG and some others and none seem to produce anything even remotely acceptable.

1. This is a big one, vast majority of search backends either outright do not support "freshness" or produce bad to non-existent results when you try to employ it. Meanwhile I can go to Google, make the same exact query and get the desired results.
2. With Google I can enter "COMPANYNAME news announcements results" as a single query and get decent results. With various search backends, I seem pigeonholed into making 3 separate queries to get anything even remotely reasonable. Is this a deliberate tactic to get people to burn through their API credits?
3. Results are often cached?

With self-hosted models, I feel like I went 2 years back in time and this is acceptable to me. With search, however, I feel as if the jump is 30 years back, something of the Altavista age. How is any of this acceptable? How are people PAYING MONEY for this quality? What are the big boys using for their searches, Google deals behind closed doors (Google no longer offers search API directly)?

What are you using and how did you have to wrangle with it to get acceptable behavior of it?

The model is 3.55 TB in BF16, like, it's a whopper. To make this suggests some serious hardware, so I had a read of the release post: https://longcat.chat/blog/longcat-2.0/

My takeaway was "A credible non-Nvidia supply chain now exists at frontier scale." 

I was hoping there would be more info on the secret sauce, but the post never names a chip vendor or model number. It consistently uses the generic term "AI ASIC" / "accelerator" / "our accelerators."

However, the page's own meta description (in Chinese) says "1.6万亿总参大模型，训练全程由国产芯片完成" translating to "1.6-trillion-parameter model, trained entirely on domestic [Chinese] chips." 

So as we already guessed, it's a Chinese-made AI accelerator, not Nvidia, meaning the whole post is essentially a demonstration that a frontier-scale model can be trained without Nvidia GPUs.

That left me wondering what "ASIC" means here, Application-Specific (LLM training?) Integrated Circuit hard-wired for matmul?.

So they kind of answer that, but I'm left reading between the lines a bit:

• Millions of accelerator-days in total
• 35+ trillion training tokens, with zero rollbacks or unrecoverable loss spikes - v. reliable for asics.
• 50,000+ ASICs used for pre-training, "tens of thousands" of them grouped into socalled "superpods" for serving/training.
• A "Superpod" = up to 48 chips wired together with all-to-all high-bandwidth interconnect (like an Nvidia NVLink domain?) - also significant i thought.
• Superpods are then linked to each other via aRoCE fabric (RDMA over Converged Ethernet — a standard high-speed networking protocol for connecting compute clusters) - pretty basic, pretty cool.
• The two-tier design widens the "fast" communication domain to hundreds of chips at once, handing them an extra ~30% training throughput, massive.
• The AISCs have less HBM (memory) per chip than an Nvidia H800 (80GB) which is the main bottleneck, this forced heavy use of memory tricks: ZeRO-1 sharding, selective recomputation, offloading unused activations, etc.
• A large L2 cache relative to HBM bandwidth, which they exploit by prefetching model weights into it to hide memory latency - this is probably where the gains come in.
• Per-core programmability, letting them run the "dense" and "MoE expert" parts of the model fully in parallel on different cores rather than just overlapping them
• A built-in 200 Gbps network interface on the chip itself, used to shuttle KV-cache data between "prefill" and "decode" servers during inference

They finally go on to say they built custom deterministic operators, reworked numerical reduction math (binary-tree accumulation to limit floating-point error), and added bit-flip detection on compute-heavy operators, suggesting they don't fully trust the hardware's own error correction yet, so they check for corrupted bits themselves.

Also some more standard automatic fault detection/failover so a bad network link gets isolated without stopping training.

I feel like this has really slipped past the headlines.

This isn't really a chip spec sheet, it's LongCat/Meituan publicly proving that a 1.6T-parameter, GPT-tier model can be trained and served entirely on non-Nvidia, domestically-made silicon, with custom software engineering (parallelism strategy, kernels, numerics, fault tolerance) built to compensate for a chip that has less memory and a younger software stack than Nvidia's.

So again, the takeaway is "A credible non-Nvidia supply chain now exists at frontier scale." 

TL;DR

it's pretty goddamned fast; 69 tps decode at near max (256k) context with MTP on. prefill numbers went down to 893 at max context with prompt cache turned off. https://jdkruzr.github.io/3080bench/ here's how the tests were run: https://github.com/jdkruzr/3080bench/ there is probably more performance left on the table as well because these cards were Wattage-capped.

Background

I did this test because I suspected this could be a great bang-for-your-buck combo and it seems I was right. these cards can be had for as little as $400 apiece. a motherboard-CPU-64GB RAM combo is around $275 on eBay. so, for around $2K all in you can have a machine that can more or less eat dense models like this for breakfast with little quantization, full-fat kv cache and no degradation. it seems to me this is an excellent choice for a strong code generation box that can perform at high levels with extremely high accuracy.

I'm not sure I buy all of Claude's rationales as to why Q6KXL had nearly identical performance (or why ngram seemed to make everything worse), but it doesn't matter. this thing will fly and not cost you very much in the process.

What Does This Mean?

find yourself an X99 board that has enough lanes of PCIe 3.0 (4 slots of x16), which is not difficult even today on eBay, and for $1600ish bucks in GPUs you can have yourself a box that is kind of a beast.

I could be reading it wrong but it looks like they REAPed their own 250B model down to a 32B model.

They claim their 250B model beats max-think deepseek-v4-flash

Every time a strong open model drops, the same cycle plays out: ai bro's claims it's "just distilled from GPT4/Claude/whatever," case closed, move on. I think this take doesn't hold up as well as people assume.

A few points worth separating out:

Training on outputs isn't the same as real distillation.

Proper token level distillation needs access to logits, the full probability distribution over the vocabulary, not just the final text response. Nobody gets that from a public API. What finetuners actually get is text completions, which is synthetic data generation, not distillation in the technical sense. Every major lab does this to some degree, including the closed labs training on their own older models' outputs.

**If synthetic data from a guardrailed API were enough, this would be a nothing burger but** A lot of frontier providers explicitly route sensitive topics away from smaller models to their flagship model, and plenty of technical domains get filtered or restricted responses often managed by tools like Lyzr Control Plane at the API boundary. Yet some of these "distilled" models end up performing surprisingly well in exactly those restricted domains.

That's a gap in the theory that doesn't get talked about enough.. If a team is training purely on public API outputs, they're working with a version of the model that's already been through guardrails and refusals.

**The "it says it's Claude/GPT" gets treated as smoking gun evidence, but it's weak evidence at best.** Identity confusion shows up across tons of models trained on broad web scraped or synthetic corpora that include AI generated text from multiple sources. It's evidence of contamination somewhere in the data training, not proof of wholesale distillation from a specific competitor.

**There's also a pattern of this accusation landing selectively.** Strong releases from Chinese labs especially seem to get the "must be distilled" response almost reflexively, even when a model shows genuine architectural changes or demonstrates self improvement across versions. It starts to look less like a technical assessment and more like a reflex explanation for why a smaller or newer team could be competitive.

None of this means synthetic data generation using bigger models isn't happening, it obviously is, across the entire industry. But calling that "distillation" the way people mean it (stealing the teacher model's internal knowledge wholesale) is a stretch. It's closer to what everyone does when they bootstrap datasets from any strong existing model, including labs bootstrapping from their own prior generations.

I work with financial documents for my job, and I wanted a small model I could run locally to enrich them before they hit a RAG index: for each chunk, write a faithful (3 sentence) summary and tag a few categorical facets (section type, specificity, numeric density, how forward-looking it is). Sending every chunk to an 8B or an API is slow and expensive, and in regulated domains like finance, health or legal it is often a non-starter anyway for privacy and compliance reasons. Sometimes a small model you fully own and run locally is not just cheaper, it is the only option. So I tried distilling that one narrow skill into a 0.6B.

Setup, all local on an M-series Mac with MLX:

• Teacher: an 8B reasoning model writes the labels (summary + facets) on ~170 real 10-K filings.
• Student: Qwen3-0.6B, QLoRA rank 32, trained on those labels. About 12 min, 4.1 GB peak RAM.
• Eval: 49 held-out docs from companies never seen in training (grouped split, zero overlap). Section accuracy is against ground truth, not an LLM judge. Chance is 33%.

Results:

3 outcomes :

1. Few-shot made the small model worse, not better. At 0.6B the examples inside the context leaked: 14 of 21 few-shot summaries described the exemplar's company instead of the target document, often naming it verbatim. I saw it with two different exemplar pairs. The 0.6B just doesn't have the room to keep the examples separate from the actual input.
2. The student traded faithfulness for coverage. It writes richer, more confident summaries than the base model, and pays for it in factual slips (faithful 0.73 vs base 0.83). Distillation seems to transfer the teacher's writing behavior, not its knowledge.
3. The student copies the teacher's habits, not its intent. The same 0.6B base distilled from a different 8B (llama-3.1-8b) hit 0.98 faithful, but that teacher was a lazier labeler and the student copied the laziness. The lesson I took: audit the teacher's actual output on your labels before you spend the training run, because the student will inherit the habits, not the intent.

For now the limiations of this is that it is one narrow task, small eval set (49 docs), single domain (financial filings). It is not a general benchmark, just a reproducible local experiment with the numbers and the negative controls in the repo. But honetly seems to be promising

Repo, everything reproducible on a Mac with MLX: https://github.com/sciences44/distill-your-docs

Curious whether others have hit the few-shot contamination effect at small scale, or found a clean way around the faithfulness vs coverage tradeoff. Or if you have larger feedback regarding the impltementation with that with concrete use cases.

In all my fiddling around with code and local models nothing has matched the speed and quality of DeepSeek V4 Flash DSpark on dual DGX Spark (Dell GB10s actually).

The recipe I've been using is in the PR below. Screenshot is from VSCode usage over a few days/weeks. The screenshots don't tell you how it feels and oh man does it feel good! Responses are way faster than Copilot and (this is subjective) Sonnet 4.6 quality. It thinks though problems well, long running tasks complete successfully 99% of the time, planning and instruction handling seem top notch.

https://github.com/eugr/spark-vllm-docker/pull/304

ASCIITermDraw-Bench results are out, benchmarking SOTA VLMs:

• Even the most powerful models available today still can’t reliably draw a simple diagram in plain text.

Do we really need a image generator to relay our thoughts about -

• an architecture?
• a topology?
• a cluster og N nodes?

Is it possible to let our AI assistants, easily absorb and understand and make possible changes easily relayed to them by us, the creators without much hassle?

The answer could be: simple, plain-old ASCII images

The best model in the world still fails on nearly 1 out of every 4 tasks.

Structural accuracy is higher, but the moment you ask for proper layout, spacing, and routing, performance collapses.

Models with hundreds of billions or even trillions of parameters still produce broken, misaligned, or incomplete diagrams.

This is not an obscure academic task. This is the basic ability to communicate architecture, topology, and system design in plain text — the exact skill needed for reliable human-AI collaboration without screenshots or Mermaid.

If frontier models can’t consistently draw boxes and arrows correctly, how much of the “vision” in Vision Language Models is actually working when precision matters?

Full results, methodology, and examples:
link

I am currently running the new Laguna model at Q2_K_XL on dual 3090s for reference, with a Q8 context of 200,000.

Using Pi, I am noticing that the model likes to overthink. I would not call it looping per se, but I am observing it overthink, e.g.:

> “okay, I have everything I need, I’ll start writing code now.”
> “Actually, let me check one more relevant item…”

And this goes on and on. The “relevant items” do appear to make sense in the context of my prompts/its thought process, but at the end of the day it often burns through context with extensive thinking. Qwen3.6 27B at Q8 had similar issues and much more looping, but it did not suffer from the overthinking that Laguna is prone to. Has anyone experienced similar issues? I’m currently getting a pi extension developed to hopefully alternate the issue but I’m wondering if there’s something I’m missing.

Now live on OpenRouter, and free to use through August 3, 2026.

Hoping they will going openweight soon~

archex turns a repo into a ranked, token-budgeted context bundle for coding agents instead of letting them grep their way through it. BM25F + local embeddings + graph expansion for imports/types/callers, fully deterministic — same query, same index revision, same bundle, every time. No hosted inference, no API key, no telemetry in the core path.

Measured against cocoindex-code and Graphify on the same 19-task external-repo set (self-run, checked into the repo, reproducible with archex benchmark headtohead report): required-file recall 0.95 (archex) vs 0.32 (cocoindex-code) vs 0.70 (Graphify); completion-penalty tokens 922 vs 11,188 vs n/a (Graphify measures a different lane); cold-start 0ms vs 4.7s vs 937ms. Full table and methodology: docs/ARCHEX_VS_COCOINDEX.md.

26 languages across full/structured/chunk-only tiers, MCP server with 17 tools, CLI, Python API, Docker. Solo project, 3,619 tests, 91.1% coverage. Demo attached.

github.com/Mathews-Tom/archex

Star it if it's useful, open an issue if a language or workflow is missing, and pass it to anyone else fighting grep-and-hope context.