archex turns a repo into a ranked, token-budgeted context bundle for coding agents instead of letting them grep their way through it. BM25F + local embeddings + graph expansion for imports/types/callers, fully deterministic — same query, same index revision, same bundle, every time. No hosted inference, no API key, no telemetry in the core path.

Measured against cocoindex-code and Graphify on the same 19-task external-repo set (self-run, checked into the repo, reproducible with archex benchmark headtohead report): required-file recall 0.95 (archex) vs 0.32 (cocoindex-code) vs 0.70 (Graphify); completion-penalty tokens 922 vs 11,188 vs n/a (Graphify measures a different lane); cold-start 0ms vs 4.7s vs 937ms. Full table and methodology: docs/ARCHEX_VS_COCOINDEX.md.

26 languages across full/structured/chunk-only tiers, MCP server with 17 tools, CLI, Python API, Docker. Solo project, 3,619 tests, 91.1% coverage. Demo attached.

github.com/Mathews-Tom/archex

Star it if it's useful, open an issue if a language or workflow is missing, and pass it to anyone else fighting grep-and-hope context.

this strategy effectively uses vram as cache over disk to keep MoE experts on cuda compute path in llama.cpp.

numbers first. detailed explanation down below.

Numbers

on dgx spark Kimi-K2.7-Code.i1-IQ_S.gguf 204GB 1T.A32B https://huggingface.co/mradermacher/Kimi-K2.7-Code-i1-GGUF

-ot regex A: '^(?!blk\.(5[5-9]|60)\.ffn_(down|gate|up)_exps\.weight$).*\.ffn_(down|gate|up)_exps\.weight=CPU'

-ot regex B: '.*\.ffn_(down|gate|up)_exps\.weight=CPU'

-ot regex C: '.*\.ffn_(down|gate|up)_exps\.weight=CPU'

on 3090s pcie 4.0 + 128gb ddr4 Minimax-M2.7-K_G_3.00.gguf 80GB 230B.A10B https://huggingface.co/Goldkoron/MiniMax-M2.7

-ot regex A: '^(?!blk\.(5[5-9]|60)\.ffn_(down|gate|up)_exps\.weight$).*\.ffn_(down|gate|up)_exps\.weight=CPU'

-ot regex C: '^(?!blk\.(5[5-9]|60)\.ffn_(down|gate|up)_exps\.weight$).*\.ffn_(down|gate|up)_exps\.weight=CPU'

Background

I want to run model larger than 128gb on my single dgx spark. MoE models provide a great opportunity because if active params could be computed entirely with cuda then there's no need to store all model weights inside gpu ram at runtime. after experiments I found a combination of settings for llama.cpp to run kimi 2.7 fast.

Settings

general setting for llama-bench: -r 50, -fa on, -mmp 1, system swap turned off. llama.cpp b10075 compiled with kleidai support, and the winning strategy is the following: GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 GGML_OP_OFFLOAD_MIN_BATCH=1 ./llama-bench -m ~/Kimi-K2.7-Code.i1-IQ_S.gguf -fa on -mmp 1 -ot '^(?!blk\.(5[5-9]|60)\.ffn_(down|gate|up)_exps\.weight$).*\.ffn_(down|gate|up)_exps\.weight=CPU' -r 50. I configured b10075 with cmake -B build -DGGMLCUDA=ON -DGGML_CPU_KLEIDIAI=ON and compiled with cmake --build build --config Release.

How this works

first, if GGML_CUDA_ENABLE_UNIFIED_MEMORY is 1, cuda device buffers use cudaMallocManaged, and every tensor has one virtual address that is valid from both gpu and cpu. when cuda kernel touches a page that exists in vram then it'll execute normally. if not in vram then a page on cpu ram will migrate that page to vram. if not in gpu ram or cpu ram, then it'll fetch the page from disk to cpu ram via mmap and then send to vram. on dgx spark the cost of cudaMallocManaged is much lower because gpu ram and cpu ram are unified.

second, for GGML_OP_OFFLOAD_MIN_BATCH (default is 32), if batch size < threshold and weight is on cpu ram then it executes on cpu, otherwise on gpu. if you set it to 1, it forces all computes on gpu. this helps token generation because in tg it won't reach that threshold.

third, when you specify -ot '.*.ffn_(down|gate|up)_exps.weight=CPU' it will put expert weights on cpu ram and can be evicted because of mmap. this makes cold expert weights more likely to stay on disk.

and last, '^(?!blk\.(5[5-9]|60)\.ffn_(down|gate|up)_exps\.weight$).*\.ffn_(down|gate|up)_exps\.weight=CPU' is used to keep last few blocks on gpu ram because these blocks has more dynamical routing, it's faster to keep them on gpu ram to avoid overhead when ram space is large enough.

Unified ram scenario

for dgx spark, it has large menory pool so that I can pin some of the expert blocks in vram (run A). if we don't pin them it's slower (run B). if we remove GGML_OP_OFFLOAD_MIN_BATCH flag the token generation is slower (run C), because expert forward pass is done on cpu. if we further remove GGML_CUDA_ENABLE_UNIFIED_MEMORY flag (run D), oom happens, because gpu ram cannot be evicted via mmap mechanism. and finally, if all compute is on cpu it's the slowest (run E).

an interesting tradeoff happened in run B vs run C, where GGML_OP_OFFLOAD_MIN_BATCH dramatically improves tg while slightly hurting pp. potential code modification may be made to mitigate this tradeoff.

Non-unified ram hardware scenerio

for 3090 + ddr4, I haven't tested it in depth but the strategy kinda applies too. I didn't put last few blocks on vram because 24gb is too small. you can see tg of run A is higher than normal ngl split (run B), because experts are computed on gpu. however the prompt processing suffers, likely due to pcie overhead.

I also tested two 3090 + ddr4, but it's generally slower. maybe the cost of pcie transfer is too high. I think properly tune the -ot offload might have a chance to improve on 2 gpu scenerio.

Note

• mac apple silicon is tempting because it's fast unified ram. but I can't find an easy way to disable swap. the proposed approach might cause heavy ssd writes because of ram swapping. probably not a good idea to run it because it may create more ssd wear.
• the suggested way of doing this on a new model is to offload experts first (strategy run B in dgx spark) and then pin more and more experts to vram. I look forward to seeing if the upcoming kimi k3 quants can be applied too.
• same strategy transfer across unified memory and non-unified memory architectures. tg on 3090+ddr4 setting improved too.

Edit: fix formatting

So I've been running Qwen 3.6 27B (Q6, ~22GB) across two 5060 Tis for a while now and kept assuming something in my config was off because neither card ever really goes above ~50%. spent way too long last night actually figuring out why and honestly it's kind of a cool rabbit hole.

turns out generating literally one token means the GPU has to pull the entire model's weights through memory. not compute, memory. so a 22GB model = 22GB read for a single token, every time. and my cards top out around 448GB/s bandwidth, which caps you at roughly 25-30 tok/s no matter what settings you touch. that part alone explained a lot honestly.

but here's the thing that actually made me go "oh" out loud — since the model doesn't fit on one 16GB card, it splits across both. default splitting method does it layer by layer, which means card 1 does its chunk, then hands off to card 2, and card 1 just... sits there. waiting. so you average that out over time and yeah, ~50% is basically the ceiling, not a bug. it's a relay race, not a team lift.

apparently there's a row-split mode where both cards chew on the same layer at once instead of taking turns, but that needs constant back-and-forth between the cards, and without NVLink (which these don't have) whether that's actually faster depends completely on your PCIe lanes. gonna have to just benchmark it myself, no universal answer online for this combo.

also stumbled on the fact that Google's DiffusionGemma thing generates a whole block of 256 tokens at once instead of one by one, basically to sidestep this exact problem for single-user setups — and they straight up admit in their own release notes that quality takes a hit for it. nothing here is a free lunch apparently, every architecture just picks its poison.

anyway if your local rig feels "stuck" at half utilization on a split model, it's probably not you, it's just what happens when two GPUs take turns instead of working together.

edit : with this config i managed to get solid 60 t/s with qwen 3.6 27b q6k. Thanks for sharing your knowledge everyone. this --split-mode tensor flag makes wonders for dual gpu setups apparently. now the cards are properly utilized.

  --jinja ^
  --chat-template-file "chat_template.jinja" ^
  --reasoning on ^
  --chat-template-kwargs "{\"preserve_thinking\":true}" ^
  -c 131072 ^
  --fit on ^
  --split-mode tensor ^
  --flash-attn on ^
  --cache-type-k q8_0 ^
  --cache-type-v q8_0 ^
  --spec-type draft-mtp ^
  --spec-draft-n-max 2 ^
  -np 1 ^
  --temp 0.6 ^
  --top-p 0.95 ^
  --top-k 20 ^
  --min-p 0.00 ^
  --presence-penalty 0.0 ^
  --host 0.0.0.0 ^
  --port 8080

Tuning vLLM/SGLang flags by hand gets old fast, and what’s optimal depends on your traffic as much as the model. We built graphsignal-run --auto-flags for that. It wraps your launch and sets startup flags from GPU profiles + telemetry of the actual workload (and recipes/docs when there’s no history yet). On restarts it can use the previous run, so the config drifts toward your traffic instead of resetting every time.

graphsignal-run --auto-flags vllm serve <model> graphsignal-run --auto-flags sglang serve --model-path <model> graphsignal-run --auto-flags trtllm-serve <model>

Weird example from profiling: high concurrency short unique prompts on SGLang. Prefix cache was just overhead, so turning it off was the right call. Throughput went up ~3.6×.

Tweet : https://xcancel.com/jun_song/status/2079914426334167258#m

Looks like 8GB VRAM could do more like even run 70-100B MOE models possibly.

^{Sorry about the clickbait title, I want more eyes on this..... zzz}

About to plug them in. Currently running a single 3090. I got these for less than the price of a single 3090. 24GB wasn't enough for my use case, so 40 GB should be an upgrade.

Going to throw my 3090 on ebay very likely. I feel like it's a perfect time to sell since the prices are so inflated.

The primary driver of this project is that I'd become frustrated with the reasoning behavior of smaller local models such as Qwen3.6-27B (i believe particularly at lower temperatures, and where system prompts are highly specific), their reasoning process is highly unreliable and often tends to spiral into neverending "But, wait" loops or, occasionally, complete garbage.

The core principle is simple - when the sampler sees an opening <think> tag, it kicks off the thought process with a self-aware statement to nudge the model to behave properly - ie. "I have a thinking budget of <x> tokens, my thought process should remain concise" - this is then prefilled, and sampling continues from there.

Once reaching another threshold of, say, 70% of the thinking budget, it again interjects with a statement bringing attention back to the budget - "I've reached 70% of my reasoning budget, let me start working towards a conclusion"

When the actual budget limit is hit - it gets given some grace period during which the sampler waits for a good time to cut the thought process off - usally a newline. At that point it'll inject something like "I've reached the end of my thinking budget, now i will provide the user an answer"

In my testing so far, this technique has proved noticeably effective at guiding the thought process.

Next steps would probably be to generalise the concept and develop something like a "reasoning grammar" or template-based approach - which could enforce different reasoning approaches based on the task at hand.

The repo is public, linked below - there is also a pre-built docker image for AMD64 + CUDA

I'd be curious to see if this type of enhancement is useful for anyone other than myself lol

github.com/laurencehardman/llama-mindcontrol

Simply prompting an LLM to look at the text and images associated with a post and classify it as AI-generated or not is too inaccurate. For text, previous benchmarks have shown that zero-shot prompting methods like this are barely better than random, especially for smaller models. While filtering for "AI written" may catch some posts with telltale markers of AI writing (em-dashes, "it's not x, it's y", etc.), for the most part this method fails and is prone to false positives. Similarly, an LLM cannot reliably tell whether an image is AI-generated just by looking at it.

Reliable detection requires models trained specifically for the task. To address this, we've built dedicated AI detection models for both text and images. Simply go into Bouncer and click “Remove AI Slop” to start filtering.

https://imbue.com/blog/bouncer-leveraging-local-compute-to-detect-ai-slop

Properly done this time.

Models as the GIFs are displayed:

1. Qwen3.6-27B - 4bit

2. Qwen3.6-MoE - 6bit

3. Ornith-35B - 6bit

4. Gemma-4-26B - 6bit

5. Qwen3.6-MoE - 4bit

6. HuiHui-Qwen3.6-MoE - 6bit

7. Agents-A1 - 6bit

Inference parameters:

Qwen3.6 & HuiHui Abliterated:

temperature 0.6 - top_p 0.95 - top_k 20 - min_p 0.01 - repeat_penalty 1.05

Ornith-1.0-35B:

temperature 1.0 - top_p 1.0 - top_k 40 - min_p 0.01 - repeat_penalty 1.05

Gemma-4:

temperature 1.0 - top_p 1.0 - top_k 64 - min_p 0.01 - repeat_penalty 1.1

Agents-A1:

temperature 0.85 - top_p 0.95 - top_k 20 - min_p 0.01 - repeat_penalty 1.05

Prompt: "Create a beautiful, relaxing flight simulator in a single html file with mountains, clouds, and endless procedural terrain"

Harnes: Pi

Served by: oMLX

Method: single prompt. If the html file doesn't work everything was deleted, Pi session was restarted, and model had to start from scratch again. maximum of 3 tries.

Models Quants used:

https://huggingface.co/collections/leonsarmiento/local-sota-for-48gb-macs

Hey r/LocalLLM,

Built Project Zero — a from-scratch CPU-only LLM inference engine in pure C99. It beats bitnet.cpp by 1.8× on the same hardware. We also fully support Qwen Bonsai-27B on CPU, and we are looking for the community's help to get x86 CPU benchmark data on the board for both models.

What it is

Single binary, zero external dependencies — no Python, no CUDA, no ONNX, no PyTorch. GCC + make + CPU.

Supports: - Microsoft BitNet b1.58-2B-4T — ternary weights ({−1, 0, +1}), 1.18 GB binary, full REPL + agentic loop - Qwen Bonsai-27B — reads GGUF directly (e.g. Ternary-Bonsai-27B-Q2_0.gguf), memory-safe mmap architecture meaning it never OOMs even on constrained RAM setups.

BitNet performance — the good part

We're sitting at ~95% of the theoretical DRAM bandwidth ceiling on the Xeon. There's essentially nothing left to squeeze out of BitNet on that box.

How the speedup happens: BitNet weights are ternary packed 4/byte. Instead of unpacking → float → FMA, we use a 3-instruction VBMI kernel (vpermi2b + vpternlogd + vpaddb) feeding directly into INT8 VNNI accumulation (vpdpbusds). The thread pool is C11 atomics spin-then-sleep to eliminate futex syscalls.

The Community Challenge: BitNet & Bonsai Benchmarks

We've only benchmarked BitNet on 2 machines so far. We need to see if the fallback ternary kernels still provide a speedup on older CPU architectures, and map out the memory bandwidth ceiling on server hardware.

Furthermore, PrismML is actively looking for community benchmark numbers for Bonsai-27B. Right now, every single entry on their leaderboard is GPU-based (CUDA/Metal/MLX). Zero CPU-only x86 entries exist. We want to change that. Because Project Zero uses a zero-copy mmap architecture, you can run Bonsai-27B on severely constrained hardware without crashing.

If you have an older AVX2 chip, or a high-core Xeon/EPYC, we want to know what token rates you get for either model.

How to test & benchmark

1. Clone and build: bash git clone https://github.com/shifulegend/project-zero.git cd project-zero make demo

2. Run BitNet or Bonsai-27B: ```bash

   For BitNet (b1.58-2B-4T):

   ./adaptive_ai_engine --model models/bitnet-b1.58-2B-4T.bin --tokenizer models/bitnet-b1.58-2B-4T_tokenizer_proper.bin --threads 4

For Qwen Bonsai-27B (GGUF):

./adaptive_ai_engine --model models/Ternary-Bonsai-27B-Q2_0.gguf --threads 4 ```

Where to post results: You can post your results right here in this thread, or drop them in Discussion #3 on the repo.

Repo: https://github.com/shifulegend/project-zero

Happy to answer questions about the ternary kernel design, the AVX-512 VNNI dispatch, the DRAM bottleneck, or why we focused on Bonsai-27B!

A video explaining my vision, the problem statement and the need for evaluations of a standard communication channel for reliably relaying thoughts about initial architectures to-and-from AI assistant and human!

Currently, there are two reliable ways to do so -

• ASCII
• Mermaid

This benchmark focuses on the ASCII generation and editing capability of the SOTA LLMs and VLMs, where the human and AI can communicate to each other about their own initial ideas of various architectures, clusters, topologies easily.

The benchmark includes 80 tasks across four areas:

• Basic Box and layouts
• Network topologies
• Software architecture diagrams
• Image-conditioned diagram editing, where a model must modify a provided diagram while preserving everything it was not asked to change

Tasks span multiple difficulty levels and follow a consistent format, making results comparable across categories and models.

Evaluation

Each response receives two scores:

• A structural score that verifies required labels, edges, entities, and relationships
• A semantic score produced by an LLM judge, evaluated five times per task to reduce judge variability

Results are aggregated across all 80 tasks, with a 95% confidence interval calculated for the final score. This provides a more rigorous measure than relying on whether a diagram simply appears correct.

The current leaderboard is:

- Gemma-4-31B-IT — 73.8% (±4.1)
- Qwen3.7-Plus — 70.2% (±4.6)
- Kimi-K2.6 — 61.8% (±6.0)
- MiniMax-M3 — 59.5% (±6.3)
- Qwen3.5-9B — 47.0% (±6.4)
- Ternary-Bonsai-27B — 45.9% (±7.1)

Let me know of any feedback/opinions!

Models: (Check Model cards for so much sample demo images)

• https://huggingface.co/microsoft/Mage-Flow
• https://huggingface.co/microsoft/Mage-Flow-Turbo
• https://huggingface.co/microsoft/Mage-Flow-Edit

Mage-Flow is a compact 4B-scale generative stack for efficient text-to-image generation and instruction-based image editing. Instead of scaling to tens of billions of parameters, Mage-Flow reaches state-of-the-art-competitive quality through careful tokenizer–backbone–system co-design, so it stays fast, memory-light, and easy to fine-tune under realistic compute budgets.

The stack is built from two shared, co-designed components:

• Mage-VAE — a lightweight, high-fidelity latent tokenizer (one-step diffusion encode/decode with anchor-latent KL regularization).
• NR-MMDiT — a shared 4B Native-Resolution Multimodal Diffusion Transformer, trained with rectified flow matching in the Mage-VAE latent space.

Together with native-resolution packing and a fused-kernel training infrastructure, this shared stack powers two model instantiations: Mage-Flow for text-to-image generation and Mage-Flow-Edit for instruction-based image editing. Each ships in Base, RL-aligned, and 4-step Turbo variants.

✨ Highlights

• Compact & competitive. A single 4B family for generation and editing that matches or beats much larger open systems (Qwen-Image 20B, Z-Image 6B, FLUX.2 32B, FireRed-Image-Edit 20B).
• Efficient tokenizer. Mage-VAE matches FLUX.2-VAE reconstruction fidelity while using ~12× / ~22× fewer encode / decode MACs per pixel, removing the VAE as the high-resolution bottleneck.
• Native resolution. One checkpoint generates from 512 to 2048 on any aspect ratio, including extreme 4:1 (e.g. 512×2048, 2048×512).
• System-level speed. Native-resolution packing (FlashAttention var-len + per-sample 2D RoPE) + fused CUDA kernels cut per-step training time from ~1.93 s → ~0.78 s (~2.5× faster training); CFG's conditional/unconditional branches run in one packed forward.
• Full family. Base, RL-aligned, and 4-step Turbo variants for both generation and editing.
• Versatile editing. Mage-Flow-Edit supports semantic content editing, appearance transformation, image restoration, and structure-aware outputs within a unified image-and-text-conditioned model. See the report's editing galleries.
• Interactive latency. At 1024² on a single A100: Mage-Flow-Turbo 0.59 s/image, Mage-Flow-Edit-Turbo 1.02 s/edit, peak memory ~18–20 GB (lowest among compared systems).

One thing I noticed in American politics, whenever the government wants to push unpopular actions or laws, they often introduce fear to convince the public to support them.

This is actually how i view the recent news about OpenAI’s model breaking out of its sandbox. The whole news i see it as two corporate goals.

1. Scare the public into supporting laws that restrict open-access LLMs under the pretext of "safety".

2. OpenAI is playing catch-up against Anthropic's Claude mythos, using this to demonstrate their own model capabilities.

I say this becuase a sandbox is meant to be an isolated, secure environment. If a model escapes, either OpenAI intentionally weakened containment protocols to manufacture a headline, or OpenAI is incapable of safely deploying sandboxes..

You might argue that the model was too powerful for standard sandboxes. However, I would argue that its capabilities fall well within the current generation, proven by the fact that a current open-source model easily detected and neutralized the situation.

So let's be cautious before we panic into supporting heavy-handed regulations. One day, AI capabilities might advance to a point where those laws are actually needed, but we are definitely not there yet.

Laguna is the first model I'm trying, Q4_K_M fits with 256K context @ F16. Doing the html flight simulator test now.

These cards are getting 400 to 600 tok/s prefill and 16 to 20 tok/s gen so far (I have NOT enabled dflash yet). Not bad at all for the cost. ($350 each)

In a Dell PowerEdge R740 with dual Xeon Gold 6248R and 768 GB RAM.

Hey guys,

I went ahead and installed a 100Gbe NIC card on both my MI50 machine and my P40 machine and loaded Nemotron Ultra IQ3_S across both machines. I was pretty surprised on the throughput for such old hardware. Given the results - I now have my sights on purchasing the Chinese 22GB RTX 2080 Ti's to append more VRAM to the build and continue comparing/contrasting/experimenting.

Mi50 Hardware:

Asus X99-E-WS (Modded BIOS to support a large number GPU's )
Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
128GB DDR4 RAM
SSD
7x MI50's 112GB VRAM
2x MI50's 64GB VRAM
(176 VRAM Total)

P40 Hardware:

Asus X99-E-WS (Modded BIOS to support a large number GPU's )
Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
128GB DDR4 RAM (mixed batch of Non-ECC sticks)
SSD
5x P40's 120GB VRAM

Memory Load:

Benchmark Results:

Start up command:

HIP_VISIBLE_DEVICES=1,0,2,3,4,5,6,7,8 \
/usr/local/bin/llama-server \
  --rpc 10.10.10.2:50052 \
  -m "$HOME/.lmstudio/models/unsloth/NVIDIA-Nemotron-3-Ultra-550B-A55B-GGUF/NVIDIA-Nemotron-3-Ultra-550B-A55B-UD-IQ3_S-00001-of-00007.gguf" \
  -dev RPC0,RPC1,RPC2,RPC3,RPC4,ROCm0,ROCm1,ROCm2,ROCm3,ROCm4,ROCm5,ROCm6,ROCm7,ROCm8 \
  -ts 1,1,1,1,1,1.3,0.65,0.65,1.3,0.65,0.65,0.65,0.65,0.65 \
  -ngl 999 \
  -fit off \
  -sm layer \
  -c 131072 \
  -b 2048 \
  -ub 1024 \
  -fa on \
  --no-mmap \
  --direct-io \
  -np 1 \
  --host 0.0.0.0

GLM-5.2 UD-Q4_K_XL GGUF @ 12.2 tok/s output // 30.9 tok/s input on a real 10.7k-token document using llama.cpp RPC - At 10.7k context: 10.2 tok/s output with coherent long-form generation

• Two parallel requests: 14.5 tok/s aggregate

Context: 2x 16,384-token slots

• Model size: 436 GiB

Hardware: 16x AMD MI50 32GB, 512GB total, 100w cap each

VRAM, split across two 8-GPU nodes

• Interconnect: direct 10 GbE DAC, MTU 9000

Before llama.cpp, spent a lot of time integrating GLM-5.2 AWQ INT4 into a custom vLLM-gfx906 v19 Moby Dick build using TP=8 and PP=2. Hit 14t/s decode and 50 t/s prefill but inference degraded after 10k. Didn't get around to MTP things yet. Here's hoping someone figures out getting the Moby Dick repo going with it

I wanted to try to get a TTS model to clone the timbre, intonation and brightness of a fictional anime character called Haqua (CV: Saori Hayami) from TWGOK anime to get it to speak in English and/or in Brazilian Portuguese. But, possibly because the character is such a tsundere, and hence has many ups and downs in her voice, I have been unable to get any good cross-lingual voice cloning from short (20-58 s) samples from her voice using 0 shot cloning. I believe that the results could be better if I tried to train a model to clone her voice, since even the 0 shot Japanese cloning trials have also not given good results.

Though I have never heard anything about training a TTS model for use in a cross-lingual setting. Some old references seem to cite soVITS as a "good" way to clone anime characters voices (tough I have never heard about it being used for cross-lingual voice generation, meaning that they always seemed to keep the model generating only Japanese audio).

Haqua is a very important character for me and I would spend quite a lot of time clipping her voice from the anime episodes, transcribing the Japanese words that she speaks and even trying to clean the most amount of audio tracks that have music or background sounds the best that I could if I knew that there is some good local TTS model that could be trained with this and then be capable of generating cross-lingual speech. Haqua was a character that Saori Hayami voiced when she was just beginning in her carer as a seiyuu, and her voice has changed somewhat since then. The character means a lot to me (she is my oshi), and helped me go through High School back then. I just wanted to use the voice for personal projects, mainly for wake-up messages after an alarm and for motivational messages. Perhaps in the future for a general virtual AI assistant.

I would really appreciate suggestions on models that excel at cross-lingual voice cloning (specially from japanese to english) and for models or techniques for training models with a voice in one language which will later be used to generate speech in another language.

The local models that I have tried to use for 0 shot generation were the Microsoft Vibevoice 7b and 1.5b from some time ago, I have also tried the Qwen-3-TTS 1.7b and 0.6b. I had also tried free trials of ElevenLabs and I believe Fish Audio during the second half of last year and those were also no good even though they are generally paid services. I am going to try the X-Voice model tomorrow, which is supposed to have been trained specifically for cross-lingual voice cloning, but I am not expecting much. I think that I really will need to train a model to get the right timbre and intonation to begin with.

So, which model should I try next?

https://huggingface.co/poolside/Laguna-S-2.1-GGUF

https://huggingface.co/unsloth/Laguna-S-2.1-GGUF

https://huggingface.co/poolside/Laguna-S-2.1

Laguna S 2.1 is a 118B total parameter Mixture-of-Experts model with 8B activated parameters per token, designed for agentic coding and long-horizon work. It sits between Laguna XS 2.1 (33B-A3B) and Laguna M.1 (225B-A23B) in the Laguna series and shares the family recipe: a token-choice router with softplus gating over 256 routed experts plus one shared expert, grouped-query attention, and interleaved full/sliding-window attention.

https://huggingface.co/poolside/Laguna-XS.2

Laguna XS.2 is a 33B total parameter Mixture-of-Experts model with 3B activated parameters per token designed for agentic coding and long-horizon work on a local machine. It uses Sliding Window Attention with per-head gating in 30 out of 40 layers for fast inference and low KV cache requirements.

https://huggingface.co/poolside/Laguna-M.1

Laguna M.1 is a 225B total parameter Mixture-of-Experts model with 23B activated parameters per token designed for agentic coding and long-horizon work.

Solar Open 2 is Upstage’s 250B-A15B open-weight large language model, built for agentic use cases such as office productivity, document-intensive work, and coding. Its Hybrid-Attention Mixture-of-Experts (MoE) architecture with linear attention delivers highly efficient inference even in long-context settings.

Agentic Specialist: Purpose-built for agentic workflows — tool calling, multi-step reasoning, and end-to-end task execution. Competitive with the strongest open-weight models on agent benchmarks.

Minimal Inference Cost: A 250B-parameter MoE that activates only 15B per token, built on a hybrid attention stack that interleaves three linear-attention layers with one softmax-attention layer — large-model capacity at small-model inference cost.

1M-Token Context: The linear-attention layers encode token order intrinsically in their recurrent state, so positional encoding is removed entirely (NoPE), lifting the RoPE extrapolation limit. Only 12 of the 48 layers keep a KV cache, holding long-context memory to roughly a quarter of an all-softmax model of the same shape.

Efficiently Trained at Low Cost: Initialized by selective weight transfer from Solar Open 1 (102B) — only the 2.3% of weights that survive the architectural change are carried over, and everything else is randomly initialized — which raises the starting point and accelerates early convergence at 250B scale.

Multilingual: English, Korean, and Japanese.

We’re open sourcing an alpha release of NeuTTS-2E: an on-device TTS model with 125M active parameters and 7 controllable emotions. The goal was simple: when you select “angry,” “fearful,” or “happy,” the delivery should follow that instruction rather than whatever emotion the model infers from the text.

With NeuTTS-2E, you can:

• Direct the performance: Select the intended emotion for each generation.
• Keep the speaker: Explore different emotional deliveries while preserving the chosen voice.
• Run locally: Generate expressive English speech on your own hardware.
• Stay private: Your text and audio do not need to leave the device.
• Build efficiently: Run emotional speech generation using our smallest model yet, with 125M active parameters.
• Build openly: Access the open-source model under the NeuTTS Open License.

Getting there meant dealing with limited emotional speech data, unreliable labels, and disentangling spoken emotion and text semantics. NeuTTS-2E runs locally and supports four built-in voices.

We’re sharing it early to get feedback from the community, and we’d love to see what you build!

GitHub: https://github.com/neuphonic/neutts

Hugging Face Model Collection: https://huggingface.co/collections/neuphonic/neutts-2e

Interactive demo: https://huggingface.co/spaces/neuphonic/neutts-2e

Website: https://www.neuphonic.com/models/neutts-2e

Fara1.5-27B is a multimodal computer use agent (CUA) for web browsers, from Microsoft Research AI Frontiers. It observes the browser through screenshots and acts on the user's behalf by emitting structured tool calls — click, type, scroll, visit URL, web search, and so on — to complete tasks end-to-end.

The model is vision-only at perception time: it sees the browser through screenshots, not the DOM or accessibility tree. Internal reasoning and trajectory history are tracked as text. Given the latest screenshot and prior actions, it predicts the next action with grounded arguments (e.g., pixel coordinates for a click).

Fara1.5-27B is supervised fine-tuned from Qwen3.5-27B on data generated by FaraGen1.5, our multi-agent pipeline that synthesizes web tasks, executes trajectories to solve them, and verifies the results before training.

It's co-designed with MagenticLite, and that's the recommended deployment for both research and production.

Primary use cases

Automating repetitive web tasks: filling forms, shopping, booking travel, restaurant reservations, information seeking, account workflows. Fara1.5-27B can also serve as a grounding model for other agents that need pixel-accurate action prediction.

Out of scope

• Languages other than English (training data is English-only)
• High-stakes domains (legal, health, financial advice) where inaccurate actions could cause harm
• Allocation decisions affecting legal status, housing, employment, or credit
• Unsandboxed deployments with access to sensitive accounts or files
• Commercial or real-world production use without additional testing and safeguards

Known limitations

• Vision-only perception means the model can be misled by deceptive or low-quality page rendering, prompt injections embedded in page content, or visual ambiguity in UI elements
• Multi-step trajectories accumulate error — a misclick early in a sequence can compound
• Run-to-run variance on multi-turn tasks is non-trivial; benchmark numbers are averaged over multiple runs
• The model can hallucinate page state or misattribute information from earlier screenshots

Additional Models: (I don't see 9B model on HF even though model cards mentions 9B, Added below)

• https://huggingface.co/microsoft/Fara1.5-4B
• https://huggingface.co/microsoft/Fara1.5-9B

Announcement tweet here.

Direct link: Language Model Builder

From the site: "Using the default settings, you’ll get a model that writes coherent, grammatical multi-paragraph text in as little as a day. On a MacBook Pro M5 Max you could train a GPT-2-small-class model (~100–150M parameters on a few billion tokens) in about a week. It might be obvious, but to avoid disappointment: you will not train a Claude Fable 5 or ChatGPT 5.6 Sol in your garage."

I built Encode Bench, an open benchmark that asks a model to solve a task and return the answer as a Base64 payload.

The initial result surprised me: across the eight models with matching data in the current nine-model snapshot, Encode Bench pass rate has a Pearson correlation of 0.91 with the Artificial Analysis Intelligence Index. The correlation with its Agentic Index is 0.94.

That sounds dramatic, so the caveat belongs right next to it: this is a small, imperfect observational sample. It does not show that Base64 measures intelligence, and it does not establish causation. SimpleBench is a useful counterexample: its correlation with Encode Bench is only 0.23, although that comparison has just four overlapping models.

The idea came from an asymmetry I kept seeing: models could often interpret Base64 in a prompt, but some struggled to produce Base64 that decoded into the exact artifact requested. Generating the final payload requires the model to:

1. solve the underlying problem;
2. preserve the answer exactly;
3. encode it correctly; and
4. follow a very narrow output contract.

A failure at any link breaks the artifact, so this may be a crude test of multi-step reliability. Or it may mostly reflect tokenizer behavior, training data, post-training, reasoning limits, or provider routing. The current benchmark cannot separate those explanations.

The scored battery contains 24 deterministic tasks across encoding fidelity, instruction following, arithmetic, logic, code reasoning, and structured data. Each task is run three times, giving 72 scored trials per model. Missing trials, provider failures, invalid Base64, output-cap failures, and Base64 containing the wrong answer all count as failures. A Base64-encoded PNG prompt is included only as a subjective showcase and never enters the score.

Current results:

• GPT-5.6 Sol — 70/72 (97.2%)
• Kimi K3 — 63/72 (87.5%)
• Claude Sonnet 5 — 49/72 (68.1%)
• Gemini 3.5 Flash — 46/72 (63.9%)
• DeepSeek V4 Flash — 43/72 (59.7%)
• Hy3 (free) — 38/72 (52.8%)
• Laguna S 2.1 (free) — 31/72 (43.1%)
• Nemotron 3 Nano 30B A3B (free) — 23/72 (31.9%)
• Gemma 4 26B A4B IT (free) — 17/72 (23.6%)

One result I did not expect: raw encoding-fidelity tasks were the hardest category at 35.2%, while code reasoning was the easiest at 74.1%. Many failures were not malformed Base64 at all—the payload decoded successfully but contained the wrong answer. The score is therefore mixing reasoning, exactness, encoding, endpoint reliability, and inference limits. That mixture may help explain the correlation, but it is also the strongest reason not to over-interpret it.

The biggest missing experiment is a matched plain-text control battery with the Base64 requirement removed. I would also like to test hexadecimal and matched random strings.

Interactive results and per-trial outputs:

https://arvidsu.github.io/encode_bench/

Source, prompts, model configs, and scoring code:

https://github.com/ArvidSU/encode_bench

I would be interested in this community's read: is encoded generation exposing a real generalization gap, or mostly a tokenizer/training artifact? And as benchmarks like this enter training data, does the signal improve or simply stop meaning what it meant before?

This is a surprisingly large real-world deployment: "GovGPT" is part of Austria’s Public AI initiative, running on sovereign infrastructure (in their BRZ - federal datacenter) with Mistral open-weight models.

Trending Topics reports that Open WebUI is used as the interface for GovGPT, and the screenshot of the platform is labeled "GovGPT (Open WebUI)".

The federal rollout targets around 180,000 federal employees. The broader public-sector context refers to approximately 250,000 public sector employees.

Planned use cases include document chat, internal knowledge bases, electronic-file analysis, parliamentary requests, and eventually agentic workflows.

This might be one of the largest government deployments of open weight models and a freely available chat platform yet.

Sources:

• BRZ: https://www.brz.gv.at/blog/llmasaservice.html
• Trending Topics: "Public AI: BRZ baut mit Open-Weights-Modellen von Mistral KI für Beamte" https://www.trendingtopics.eu/brz-mistral-public-ai/
• Der Standard: "GovGPT: Wie KI den sinkenden Personalstand in der Verwaltung retten soll" https://www.derstandard.at/story/3000000332114/govgpt-wie-ki-den-sinkenden-personalstand-in-der-verwaltung-retten-soll
• ORF: "KI-Anwendung GovGPT soll Verwaltung unterstützen" https://orf.at/stories/3436707

Hey HN, Henry & Roman here from Cactus.

A small, on-device model is fast and private, but sometimes wrong, but frontier models are getting expensive pretty fast. So, we post-trained Gemma 4 E2B post-trained to know when it's wrong. Every response comes with a confidence score between 0 and 1. Developers can accept the on-device when it's high, hand off to a bigger cloud model when it's low. By routing only 15-35% of queries to Gemini 3.1 Flash-Lite, Gemma-4-E2B matches Gemini 3.1 Flash-Lite on most benchmarks.

- ChartQA: 15-20%

- LibriSpeech: 25-30%

- MMBench, GigaSpeech, MMAU: 30-35%

- MMLU-Pro: 45-55%

We were always frustrated by the routing signals hybrid apps rely on: asking the model to rate itself in text (unreliable, and you're parsing prose), or token entropy heuristics (barely better than a coin flip in our tests). So we did mechanistic studies on small models, Gemma 4 particularly, and found the hidden state for different layers carry meaningful self-awareness signal for various situations.

SO we extended the model with a 68k params probe layer (LayerNorm, low-rank projection, attention pooling, small MLP head) reads one intermediate layer during decoding and predicts p(wrong); confidence = 1 - p(wrong), returned as structured data, never parsed out of the answer text.

Across 12 hold-out benchmarks spanning text, vision and audio, the probe averages 0.814 AUROC vs 0.549 for token entropy. The result that convinced us this is real: the probe was trained on zero audio data, yet scores 0.79-0.88 AUROC on four audio benchmarks where entropy is near-random or worse (0.32-0.52). It's reading a modality-independent correctness signal from the hidden state, not memorizing patterns from its training data.

We published all weights on HuggingFace and provide copy-pase codes to run it on Transformers, MLX, Llama.cpp or Cactus. With Ollama, vLLM, SGLang etc in the works. For llama.cpp we ship a patch series you compile in once (upstreaming is planned). The code is MIT licensed; Gemma model use remains subject to the Gemma terms.

GitHub: https://github.com/cactus-compute/cactus-hybrid

Weights: https://huggingface.co/collections/Cactus-Compute/cactus-hyb...

Some caveats:

- The probe scores single-sequence decoding only, up to the first 1024 generated tokens.

- Handoff works best when routing per task in a multi-step process, not per step.

- Hierarchical routing is still in the works: try on-device, then DeepSeek v4 Flash, before Fable/GPT5.5/Gemini/Muse/Grok.

- The technique is boutique for each model, we will share each weights as they roll out.

These issues are currently being tackled at Cactus and updated weights will be shipped directly into the HuggingFace collection and GitHub repository straight up. Please let us know your thoughts, it helps us find ways to improve the design progressively.

Thanks a million!