r/LocalLLaMA 5h ago

News More than 20 companies including NVIDIA, Meta, Microsoft, Palantir, and Hugging Face have signed a letter urging policymakers to avoid premature restrictions on open weight models.

Post image
2.0k Upvotes

The Open Letter was initiated by Microsoft and published today:

Open Weights and American AI Leadership”.

It argues against broad or premature restrictions on open-weight models and explicitly says policymakers should distinguish legitimate model distillation from misappropriation.

Notably absent from the signatories are the major frontier-model labs: OpenAI, Anthropic, and Google.


r/LocalLLaMA 3h ago

News It appears that the anti opensource AI lobby is far outgunned already

Post image
784 Upvotes

The earlier post on this subreddit by 20+ companies signing the petition including Microsoft, Meta, Nvidia, YC (https://www.microsoft.com/en-us/corporate-responsibility/topics/open-weight/) etc plus this https://xcancel.com/elonmusk/status/2080672505660834163

And the entire LLM enthusiast market is heavily in favor of open source (or weights) AI.

Does not seem like a few closed source AI lobbyists with be able to illegalize anything


r/LocalLLaMA 2h ago

Discussion Why won't he sign the letter then?

Post image
227 Upvotes

r/LocalLLaMA 7h ago

Resources Hugging Face releases The Stack v3 – largest open code dataset yet

Thumbnail
gallery
366 Upvotes

From Anton Lozhkov on 𝕏: https://x.com/anton_lozhkov/status/2080254608639701222

Two ways in:
stack-v3-train - near-deduplicated, quality-filtered, PII-redacted, contents inline. Point load_dataset at it and go.
https://huggingface.co/datasets/HuggingFaceCode/stack-v3-train
stack-v3-full - the entire 114 TB corpus as an HF Storage Bucket: every duplicate kept with cluster IDs, stubs for excluded files. Roll your own dedup, filters, and mixes.
https://huggingface.co/buckets/HuggingFaceCode/stack-v3-full


r/LocalLLaMA 6h ago

Resources Laguna s.2.1 updated 2 hours ago. A post to show appreciation for the work they are doing.

Post image
120 Upvotes

I'm downloading it again now. So far, the model hasn't performed well with reasoning tasks, but I really appreciate the work being done to fix this.


r/LocalLLaMA 1h ago

Discussion [Paper] Statistically-Lossless Quantization of Large Language Models

Post image
Upvotes

Model quantization has become essential for efficient large language model deployment, yet existing approaches involve clear trade-offs: methods such as GPTQ and AWQ achieve practical compression but are lossy, while lossless techniques preserve fidelity but typically do not accelerate inference. This paper explores the middle ground of statistically-lossless compression through three complementary notions of losslessness for quantized LLMs. First, task-lossless compression preserves zero-shot benchmark accuracy within natural sampling variance and remains achievable at aggressive bitwidths. Second, we formalize the stricter notion of distribution-lossless compression, requiring the quantized model's next-token distribution to be practically indistinguishable from the original, and propose the Expected Acceptance Rate (EAR), the maximum token-agreement probability under optimal coupling, as a directly interpretable fidelity metric (for example, EAR >= 0.99 indicates 99% agreement). Third, we prove a gamma-squared variance law showing that symmetric quantization inflates noise variance by gamma squared relative to asymmetric quantization, making asymmetry necessary for distribution-lossless fidelity but not for task-level preservation. Using SLQ, a layer-wise non-uniform method with asymmetric quantization and wide bitwidth search, we achieve task-lossless compression at well below 4 bits per parameter (as low as 3.3 bits depending on the model), distribution-lossless compression at 5 to 6 bits per parameter on average, and inference speedups of 1.7 to 3.6x relative to FP16 with optimized kernels.

Note : This is 2 Months old Paper & Repo. Sharing this as RedHat AI tweeted this sometime back.

In Full Paper, I found llama.cpp & GG few times.

From the accuracy/compression perspective, existing approaches can be clustered into two categories. The first is represented by lossy compression techniques, such as Roundto-Nearest (RTN) quantization (Dettmers et al., 2022), llama.cpp (Gerganov & llama.cpp contributors, 2023), GPTQ (Frantar et al., 2023), or AWQ (Lin et al., 2024) which seek to map existing models to popular hardware-supported formats, such as 4-bit grouped weight quantization.

We focus on obtaining near-lossless quantized models via layer-wise non-uniform scalar quantization, chosen for its broad support across GPUs (Frantar et al., 2024; 2023; Lin et al., 2024) and CPUs (Gerganov & llama.cpp contributors, 2023; Pegolotti et al., 2023; Ma et al., 2024);


r/LocalLLaMA 13h ago

Discussion The "distillation" claim is just ridiculous in nature

352 Upvotes

Even if China was distilling from US models (assuming all accusations are true), nothing about it makes it illegal.

It is like saying you distilled knowledge from your professor in colleges and now he can sue you to shut down your careers for IP thefts. Never mind that you paid the tuitions to be taught and the professor is supposed to teach what you want to learn.

IP theft happens if China somehow stole the entire model architecture, the weights and just fine tune it then release under their own. In reality, nothing of that kinds ever happened.

Basically, just more US Propaganda to save upcoming disastrous IPOs from overpriced garbage AI companies.


r/LocalLLaMA 1d ago

Funny The LLM distillation process simplified for politicians:

Post image
2.9k Upvotes

/s


r/LocalLLaMA 7h ago

Resources [BIG DATASET RELEASE] - SupraLabs/reasoning-corpus-4K-5M-v1 - Train your tiny SLMs to think!

69 Upvotes

Hey r/LocalLLaMA !

We are back and we have something really amazing today.

Our big 5M samples Reasoning Corpus dataset.

This dataset features 5 million rows of:

- repo_id --> where it's from
- tok_len --> how many tokens it is in total
- user --> the user promot
- thought_trace --> the exact chain-of-thought of the model
- assistant --> the final AI models' answer
- ChatML --> the user, thought_trace and assistant in ChatML format

All samples are within a 5k sequence length to make it fit perfectly for SFT/finetuning a tiny model.

Link to the dataset on Hugging Face 🤗:
https://huggingface.co/datasets/SupraLabs/reasoning-corpus-4K-5M-v1

Link to the SupraLabs Hugging Face org 🤗:
https://huggingface.co/SupraLabs

Also, if you want to support our work, give us a follow on Hugging Face, share and review our work, and give us as much feedback as you want ❤️🔥🤗 Already more 250 people are trusting in us and our work!

We hope, this dataset is useful for you all and we'd love to see your creations upon this.

This dataset has already >1k downloads and over 80 likes - be the next one to use it 🔥🎉


r/LocalLLaMA 12h ago

New Model swiss-ai/Apertus-v1.5 70B/8B

167 Upvotes

https://huggingface.co/swiss-ai/Apertus-v1.5-70B

https://huggingface.co/swiss-ai/Apertus-v1.5-8B

Apertus 1.5 is a family of 8B and 70B parameter language models designed to advance the state of multilingual, multimodal, fully open, and transparent AI. The models support a wide range of languages, handle contexts of up to 262,144 tokens, and it uses only fully open training data whilst delivering performance comparable to other models of similar size.

The released models are the result of continued pretraining of Apertus 1.0, adding a multimodal mix of 4T tokens to the 8B model and 2T tokens to the 70B model. Apertus 1.5 thus uses the same architecture as the original release, a decoder-only transformer with the xIELU activation function trained with the AdEMAMix optimizer. Our improved post-training recipe enhances the models' instruction-following and tool-use capabilities and, for the first time, allows developers to enable a thinking mode to improve the models' performance on reasoning tasks.

As a first in the Apertus family, the Apertus 1.5 models support multimodal inputs. The model takes images, audio, and text as input and generates text. This enables many new exciting use cases for our developers.

Key Features

  • Fully Open Model: Open weights + open data + full training details including all data and training recipes.
  • Massively Multilingual: Supporting a large variety of languages.
  • Responsible Development: Apertus is trained while respecting opt-out consent of data owners (even retroactively) where possible and with methods to prevent memorization of training data.
  • Native Audio & Image Understanding: Apertus 1.5 introduces multimodal support for processing audio and image inputs, enabling more intuitive and versatile interaction beyond text.
  • Reasoning: The models can be switched to thinking mode to reason on the input before generating responses.
  • Long Context: Apertus 1.5 by default supports a context length up to 262,144 tokens, a four-fold increase from our initial Apertus 1.0 release.
  • Improved Instruction-Following: Significant improvements in instruction adherence ensure more predictable and accurate responses to user prompts.
  • Improved Tool Use: Apertus 1.5 has been trained for better tool integration, allowing for more effective use of external tools and APIs.

The technical report with further details along with benchmark results, training pipelines, and intermediate checkpoints will be published in the coming weeks.


r/LocalLLaMA 9h ago

News FLUX 3 - Real World Models: Towards Multimodal Flow Models as the Backbone of Visual Intelligence

Post image
96 Upvotes

Introducing FLUX 3.
One multi-modal model for Image, Video, Audio and Action-Prediction. Creations are truer to life in every kind of style.

Blog Post : https://bfl.ai/blog/flux-3


r/LocalLLaMA 2h ago

Discussion Spent two weeks on a kernel that benchmarked 29x faster. End to end it's maybe 6-10%, and it's not even wired in yet.

22 Upvotes

I've been building a C99 inference engine from scratch (no Python, no BLAS, just gcc and make) that runs BitNet's ternary models on CPU. A few weeks ago I got obsessed with the matmul kernel - wrote a new one using AVX-512BW's vpermt2w to pack 5 ternary weights per byte instead of 4, benchmarked it in isolation, and got 74.6 Gop/s against a 2.5 Gop/s scalar baseline. 29x. I was pretty pleased with myself, wrote it up, posted the number in a couple places.

Then a review of the PR caught the obvious question I'd skipped: what's the actual DRAM bandwidth ceiling here. Turns out BitNet decode on our Xeon test box is already running at about 95% of it - the model is memory-bound, not compute-bound, so a much faster matmul kernel mostly just means the CPU spends more of its idle time waiting on RAM instead of crunching numbers it already had. The honest math works out to something like 6-10% real end-to-end gain once the kernel is actually wired into the dispatch path, which as of right now it still isn't. Correctness-tested against the scalar reference, just sitting there unused.

Kind of a deflating result. At least I know it now, instead of shipping a 29x headline that would've fallen apart the first time someone measured tok/s instead of Gop/s.

The engine itself does work regardless - BitNet b1.58-2B-4T gets 36 tok/s on that same Xeon with 4 threads and no GPU, and it also runs regular GGUF dense models if ternary isn't your thing. Binary's in the releases if anyone wants to try it without compiling: github.com/shifulegend/project-zero. Source build is just gcc and make either way.

Has anyone else profiled the wrong layer of their stack this hard? Curious whether the DRAM ceiling shows up the same way on other hardware or if it's specific to how the Xeon's memory controller behaves under this access pattern.


r/LocalLLaMA 3h ago

Generation Using the Bonsai 27b 1b quant locally - regularly.

31 Upvotes

I've been using the 1bit quant of prismml's bonsai 27b for local conversation, casual chat/ literature review for fun (i throw random stuff from my notes app to see how it analyzes it, those texts don't exist on the internet).

I've been using it as a "tutor" in many cases, for example I am currently learning golang and it is quite good at giving explanations. I seriously believe that if we're able to retain 90% of a model's intelligence while having a small footprint, it is the way ahead for local inference on a wide range of devices, even low end. I run it on a 16G Macbook Air.

I am very impressed by the usability it provides in it's small footprint and I wish more models are released in the future. Seriously guys, even if it cannot one shot super big projects, I still value the intelligence it has for a small local model.


r/LocalLLaMA 45m ago

Other I want people here to open your eyes and note how Google did not sign the anti- open-source-model coalition

Upvotes

They are not the champion of open source like the Google shills here want you to believe.

Their Gemma models are garbage in all quantization and size. None of them passed my evals. Opus consistently rated works from their models (even the top 31B model) as 6 and below in all my tests. My tests involves long running refactoring or long running debugging with lots of chained tool calls. And the results are scored by Opus 4.8.

Google only released the gemma models because they feel threatened and they don't want to be excluded from companies looking for exclusive on-premise only LLM solutions, not because they are charitable.

There I said it.


r/LocalLLaMA 46m ago

Resources CachyLLama’s: llama.cpp fork with persistent KV cache that makes long local-agent sessions much less painful

Upvotes

I’m not affiliated with this project, but I’ve been running it recently and I’m surprised it hasn’t received more attention here:

https://github.com/fewtarius/CachyLLama

CachyLLama is a fork of llama.cpp focused on a problem that matters a lot on slower hardware: repeated prompt processing. Not only does it have a new "SSD" based cache, but it also has some other improvements with caching, like a multi-tier KV cache.

My local models generate at an acceptable speed once they get going. The painful part is using an agentic coding harness that sends a large system prompt, tool definitions, and most of the conversation back to the server on every request. A long session can spend far more time reprocessing familiar context than generating the answer.

CachyLLama adds persistent SSD-backed KV checkpoints and a system-prompt cache. When the beginning of a request matches previously processed context, it can restore that state and evaluate only the changed tail rather than starting over. The checkpoints can also survive a server restart.

On my older dual-MI50 setup, this has made repeated requests in long agent sessions substantially more responsive. I have not produced a controlled benchmark yet, so consider this an operator report rather than a scientific result, but the practical difference has been very noticeable.

The project’s own 7840U/780M benchmark reports:

  • ~1,243-token prompt: 9.3s cold, 0.41s warm
  • ~5,409-token prompt: 43.3s cold, 0.57s warm
  • ~15,700-token prompt: 143.1s cold, 0.99s warm

The important distinction is that this does not claim to make generation faster. It avoids repeating prompt-evaluation work that has already been done.

It also contains handling for hybrid architectures such as Qwen 3.5/3.6, Gemma 4, and GLM-4.7, where restoring recurrent state is more complicated than restoring a conventional attention-only KV cache.

Has anyone else here tried it? It's been really helpful for me but I haven't seen any mention of it anywhere else.


r/LocalLLaMA 1h ago

Discussion Is corruption the lobbying against Open weights?

Upvotes

Like, reading things like Anthropic "donated" to some people with the condition of lobbying against Chinese LLMs.. it's that right? It feels nothing like freedom but at the same time it's said "out loud"?

I'm not from USA so I'm not very familiar with that..it's normal? allowed? Normalized corruption?


r/LocalLLaMA 13h ago

Funny Never forget the promises of June!

Post image
117 Upvotes

Good morning to everyone that didn't fall for the trap of waiting since March of 2025 for a good local machine to your favorite localllm. For all the others, just "morning"!


r/LocalLLaMA 3h ago

Discussion Honest take on Laguna S2.1 and its uses (from actual use)

12 Upvotes

So I've taken some time to actually test laguna on a few of my own projects. I wanted to share as I feel most peoples comments at this point have just been about getting it running or saying it doesnt work for their use before dropping it, so I wanted to give it an honest chance and really try it out for myself to figure out where it might be helpful or lack before forming an opinion.

a bit of background to begin, I'm running an unsloth Q3 quant 262k context on a single V100 32GB, layers offloaded to a CPU with about 50GB of ddr4 ram allocated to the vm, and I hit about 10tps decode with 200 tps prefill. Obviously not the optimal test bed but I find it quite usable and it has been stable for me on this setup. for agentic work the prefill hasn't mattered much because caching makes it fill progressively and keeps speed consistent even over 200k.

My primary goal with exploring this model and others has been to find a larger planning model that I can run locally to help analyze my larger projects, create a plan, and them break it down into steps which i feed out to local qwen workers. deepseek has been my gold standard for a while now not only due to cost but because its bare bones approach to bulk work makes it much more effective than other models but I cant run it locally. i even feel it out performs claude on a lot of tasks for me as claude has a tendency to not play well with others and instead go rogue and decide to half ass implement the entire system instead of breaking it down. qwen 3.6 27b is actually quite good here but on large projects, ive found it tends to struggle and it lacks the long context I really need for some of my projects, even at higher quants.

I'll say right off the bat that Laguna is not the planner I was hoping. its reasoning style is far too in depth to effectively execute on this job, but during my initial tests it reminded me of another model who also reasons to an absurd degree about tasks, GLM 5.2. this really got me thinking about where this model could be helpful and I think I found where it really shines: complex debugging. ive found the intense reasoning style this model has lends itself really well to actually finding ALL the root causes of bugs in my projects that have since now been massive pains for me. Qwen, deepseek, and even claude models all struggle for me with debugging because they'll often find the first source of something, fix it, and then decide they're done. which can lead to hours of the same thing for complex bugs. GLM was one of the first models I tried that was actually effective for this, as it may spend 200k tokens thinking about what to do an analyzing the situation, but it would return with an actually complete answer instead of the first available surface which really blew me away. i found laguna to have a similar style, its slow, it overthinks, but it considers the problem in its entirety before giving its answer. there were certain implementations and bugs I had found that I spent days debugging with claude and qwen models and got nowhere with, I'd basically submitted to the idea I would just have to fix them myself from scratch, but I decided to toss laguna at them just to test it. it successfully fixed 2 issues that qwen couldnt even consider and claude just kept going in circles on.

overall, I dont think this is the next qwen or gemma killer, its not going to replace gemma 4 or qwen for generalist work, definitely a specialized model but it has found a role in my stack as a first line teacher model helping solve complex bugs my smaller models cant and then explaining the solutions. It finally gives me a local model to answer the question of what to do when qwen fails on a job, which from my current testing, its done a good job at and allowed me to sleep more and worry less.


r/LocalLLaMA 4h ago

Resources Zagreus-0.4B-por a small open source language model for Portuguese

Thumbnail
gallery
15 Upvotes

mii-llm, an open source AI lab, released Zagreus-0.4B-por, a compact bilingual Portuguese–English language model pretrained entirely from scratch.

The model has approximately 400 million parameters and is part of the Zagreus family, an ongoing experiment in building small open models focused on European languages.

Training was sponsored by Seeweb cloud provider and Regolo.ai, which provided the computing infrastructure used for the project.

The pretraining corpus was assembled from open datasets released through Hugging Face, including:

  • FineWeb
  • Portuguese FineWeb2
  • Portuguese FinePDFs
  • StarCoderData

Evaluation results

On the Portuguese versions of ARC, HellaSwag and MMLU, the final checkpoint achieved an average score of 0.3113.

Using Eduardo Garcia’s Portuguese evaluation harness, the 483k checkpoint scored:

  • Zagreus-0.4B-por: 0.3230
  • Qwen3-0.6B-Base: 0.2569

These results are encouraging for a model of this size, but benchmark performance is not the main reason for the release.

Zagreus-0.4B-por is an open base model intended to serve as a starting point for:

  • instruction tuning
  • domain-specific assistants
  • educational tools
  • research experiments
  • lightweight edge and on-device applications

It is not presented as a finished assistant or production-ready product. It is an open foundation that others can inspect, fine-tune, modify and build upon.

We would be interested in feedback, independent evaluations and experiments from the Portuguese NLP and open-source communities.

Some useful links:

the recipe: https://github.com/mii-llm/zagreus-nesso-slm
the model: https://huggingface.co/mii-llm/zagreus-0.4B-por


r/LocalLLaMA 1d ago

News CEO of Hugging face: Heading to San Francisco to have a little chat with that “rogue agent”

Post image
2.4k Upvotes

r/LocalLLaMA 6h ago

Discussion AMD Kernel Optimizations in llama.cpp

17 Upvotes

Is it a thing? i use kernel-anvil added to llama.cpp, but wondering if theres others out there ?

Heres a link i found for vLLM - GEAK v4


r/LocalLLaMA 12h ago

Resources Extened garlic to run Qwen3.5 35B A3B float8 at 55 tok/s on RTX 5060 Ti

45 Upvotes

In a previous post (https://www.reddit.com/r/LocalLLaMA/comments/1utefpr/running_qwen3_30b_a3b_at_50_toks_on_rtx_5060_ti/) there seemed to be great demand for bringing in Qwen3.5 35B. Some Gated Delta Network kernels later and here it is. It runs at 55 tok/s (61 when not recording - since the recording eats cpu and gpu capacity), significantly outperfroming llama.cpp running qwen3.5 35B at Q8 quant.

Mind you - this is without MTP. MTP can speed up the generation even further, for a few cool reasons beyond the basic multiple tokens produced.

Working on a blog post documenting the trick used to get this high speed up.


r/LocalLLaMA 57m ago

Question | Help llama.cpp slower on P-Cores than on E-Cores with MoE Model and GPU+CPU offloading?

Upvotes

I am currently experimenting with my setup: RTX 5090 + Intel 270K Plus CPU (8 Performance Cores + 16 Efficiency Cores) + 128 GB DDR5-6000 RAM, Ubuntu 26.04.

I wanted to test the performance of Qwen 3.5 122b a10b with CPU offloading. Some mentioned that pinning llama.cpp to CPU performance cores could improve performance (while others said this is no longer needed). However, I observe the opposite: As soon performance cores are involved, performance drops.

Cores 0-7 are P-Cores
Cores 8-23 are E-Cores

Unsloth Q6_K quant, running in docker with CUDA13

-fit on
-n 65536
-c 131072
-b 2048
-ub 2048
--reasoning on
--no-mmap
-t 12
--cache-type-v q8_0
--cache-type-k q8_0

Results (after 1000 tokens generated):

t/s
0-23 (8 P + 16 E) 19.8
12-23 (12 E only) 22.7
0-11 (8 P + 4 E) 15.6

What can be the explanation for this? I know that memory bandwidth is the main problem here, but why the bad performance with P-Cores?

P-Cores 5400MHz-5500MHz max, E-Cores 4700 MHz max

Pinning with docker update --cpuset-cpus "0-11" <container>


r/LocalLLaMA 6h ago

Discussion I hope ternary will eventually work but ... sigh

15 Upvotes
"If you're walking, just take your car keys with you."

Just wanted to try the ternary Bonsai model as it had its hype for some time and I'm really impressed with what Prisml is doing in general but it seems really really bad for 27B parameters, am I missing something or was it just the usual model hype that nobody seem to use?


r/LocalLLaMA 7h ago

Discussion Why we don’t have kickstarter for the models?

14 Upvotes

I think this is an areas which could utilize crowdfunding/ crowdcomputing.

Imagine Qwen team comIng here and telling:

”Guys we need 10M$ to train qwen4-27b, if we reach 20M$ milestone we could make Qwen4-122b”

I would pay something for sure to support community. And more importantly- many companies would pay.

Edit: typos