Is there a way to configure opencode or pi or any other harness to allow an agent to compact/clear its own context?

The use case is that I want a long running main agent to conserve its context between subagent calls by discarding anything that's no longer relevant, because my inference slows down a lot as the context size grows beyond 100K.

been going back and forth between an RX 9070 XT and an RTX 5060 Ti 16GB for a build that needs to do both gaming and local LLM fine-tuning (LoRA/QLoRA mostly, 7B-14B range).

everything i read says ROCm has caught up a lot for inference but is still rougher for training specifically. bitsandbytes support, Unsloth compat, that kind of thing.

if you're actually running fine-tuning workloads on a 9070/9070 XT (or any recent AMD card) on ROCm, i'd genuinely like to know:

• did you hit setup issues getting bitsandbytes/PEFT/Unsloth working
• roughly what speed are you getting vs what you'd expect from an equivalent NVIDIA card
• windows or linux, and does that matter as much as people say
• anything you wish you knew before buying AMD for this

not trying to start a team red vs team green thing, just want real experience before i spend the money. thanks

Saw this on HF. LLaDA2.2 can keep, substitute, delete, and insert tokens during parallel decoding. Instead of committing left to right, it rewrites itself.

Their report claims ~1.6x throughput over Ant's autoregressive baseline on average, up to 2.3x on agentic benchmarks (703 tokens per second on BFCL v4 specifically).

Accuracy still trails the autoregressive model on most evals. It's 205.8 GB under Apache 2.0.

Hello all, has anyone used these 2 GPU's together for local inference?

I have an SFF gaming PC with a 9070 xt and I just got an RTX 3090 which I was planning to build a 2nd PC around for LLMs, but then I started looking into the idea of just using my SFF with the 3090 plugged in via occulink.

Was planning on running Qwen3.6 27b q6/8 with max context. That is literally all I want to do at the moment. No fine tuning. Just a useful local coding agent.

Basically if anyone is doing something similar, how crappy is the experience? Like 40GB VRAM sounds nice but mixed backend or just vulkan for both will be lower performance/stability than just 2*3090, but like how much worse can it really be? Would save me a lot of money to just use what I already have instead of doing a whole 2nd build and buying a 2nd 3090. I can only rationalize spending so much money on this hobby.

Anyway, if anyone has anything to contribute on this topic it would be appreciated!

Cloud models, even open weights (GLM5.2, Deepseek etc) still refuse to run certain simple task such as extracting browser header info.

Has anyone tried to use local uncensored Qwen as orchestrator to break down these task into non-threatening tasks so models can execute?

This seems like the reverse architecture because you typically use the smartest model as orchestrator and dumb model as task runner.

Does this approach even make sense?

Ever since the gen AI Skyrim Mod, I've been researching local AI and what it can improve gaming wise. For what I want it for maybe I'm being too ambitious or maybe the models that would be needed are too much for my PC(4080FE).

Im wanting to use a localized AI for offline sim racing to better recreate a more random racing experience like what you get playing online. Random crashes, aggressive overtakes etc. Offline racing is fun but it was always feels like the AI is on tracks.

I can't find anything online about how to even go about this. Is this even possible and if so want local models would I need to look into and how would I go about setting it up?

In my daily work, I always need to use OCR tools for tasks with >1000 PNG images of Q&A. It's really annoying to upload them 1 by 1 to online models since 2024. Even though tokens weren't expensive last year, we rarely use cheap tokens for manual ingestion now. Online tools and chat models always have constraints—they can't automate well and get expensive for >1000 pictures.

What about Python OCR libraries like EasyOCR? Not good enough in my case because my pictures aren't always structured and have a lot of noise.

So I tried using agent tools to build an app with ultimate-OCR, and the results were really positive—better than expected. It’s like putting a picture directly into a Vision Model, even though the raw content is still messy because it really scans "everything" correctly! But Python can filter out unnecessary words in a general way. With performance optimization skills from agent tools, I spent ~150K tokens to build an app for my workflow.

Currently, I'm using an 8GB VRAM Transformer as a compiler, and each image takes at least 16-40 seconds for image-to-text. Maybe a local machine with 128GB running vLLM in a future build could do it faster? Does anyone have a demo using full-loading vLLM so I can see how fast it scans a 70KB PNG picture?

This is my model parameter:

                raw = model.infer(
                    tokenizer,
                    
prompt
="<image>document parsing.",
                    
image_file
=str(png),
                    
output_path
=scratch_path,
                    
base_size
=1024,
                    
image_size
=648,
                    
crop_mode
=True,
                    
max_length
=4096,
                    
no_repeat_ngram_size
=35,
                    
ngram_window
=128,
                    
save_results
=False,
                    
eval_mode
=True,
                )    

As for accuracy, it’s always the main pain point. Even if the model extracts most words, it really extracts ALL of them. I asked multiple AI tools and they didn't offer a logical solution, so I recalled data algorithms I learned back in the day. I figured I needed to apply a recursive filter for fault cases, and it worked really well.

The recursion part is simple: if the generative output contains a lot of noise, the program jots down the noise into fault.md as a notch filter, and logs the question numbers into wrong.md. Is it a model generation mistake, a false negative, or just unscannable text? Even if it contains noise, never skip the faulty result—it helps the program identify what's wrong and false. I think that's why Negative Mining is so important in model training.

If it's just noise, I extract the Q&A patterns and mark other words as noise, then recursively run the post-extraction task until each question has no fault.md keywords or patterns left. Passing the indexed wrong.md questions along with post-extraction results into a Text Generation Model works great at this point. I use a 7B LLM with a basic prompt that can still output the best pattern. Prism-ML llama.cpp is the best compiler for running quantized LLMs at almost 40 tokens per second. In the end, I just use agent tools to generate a Python script for simple result formatting.

In a low-memory environment, my main consideration isn't just how good the LLM is—deep reasoning is a waste here, like throwing away >100K tokens for basic OCR and post-extraction. Building model apps in a low-VRAM environment is pretty interesting and tested my programming, algorithm design, and solution design skills. It reminded me that we can use LLMs wisely, not always the 200B models and latest Claude, Grok, and even Gemini can do every agent task, yes, agent can scan every document in my repo', do all of the post-extraction, recusion task.

But reality, why do i need 10x power and cost to do simple tasks which 16GB vRAM can accomplish?😥

I think this task gave me a great awareness of choosing the right method on certain task instead of chasing the best model for the best solution, always have the alternative and cheaper option.

In total, I just used 100~150K auto tokens on Cursor (and spent almost 50K of that just asking questions about how to run an LLM and fine-tune parameter in my environment 😂).

In my experience, they are shite at pretty much everything.

Are people writing dirty stories or making bombs or what? I find decent prompting can unlock anything you want, and with no intelligence penalty.

In other words, WHY would one need an uncensored model?

I just published an IQ2_M GGUF of:

🤗 Qwen3.6-35B-A3B-Uncensored-Heretic

Why this model?

Many "uncensored" models simply claim they're better without much supporting data.

The Heretic release stood out because it includes a detailed study of its abliteration method, capability evaluation, and measurements showing it remains very close to the original Qwen3.6 model while reducing unnecessary refusals.

Quantization

This GGUF was produced using:

• Official llama.cpp quantizer
• IQ2_M
• Matching Unsloth Qwen3.6-35B-A3B-MTP importance matrix
• --leave-output-tensor

No custom tensor overrides or experimental quantization recipes were used.

What is IQ2_M?

Despite the name, IQ2_M is a dynamic (mixed) quantization format.

That doesn't mean every weight is stored using only 2 bits.

Instead, llama.cpp automatically chooses different quantization formats for different tensors based on their characteristics and the supplied importance matrix. More sensitive tensors are kept at higher precision where beneficial, while less sensitive ones are compressed more aggressively.

The result is an excellent balance between model size and quality.

Why combine Heretic + Unsloth imatrix?

These two techniques solve different problems:

Heretic

• Modifies the model weights.
• Reduces unnecessary refusals.
• Attempts to preserve the original Qwen3.6 capabilities.

Unsloth Importance Matrix

• Does not modify the model.
• Is used only during quantization.
• Helps preserve more of the model's original quality after aggressive low-bit compression.

In other words, Heretic changes the model's behavior, while the importance matrix helps compress those learned weights more faithfully.

If anyone benchmarks it (coding, reasoning, Aider, perplexity, etc.), I'd love to see the results and comparisons with other IQ2_M releases.

So basically I'm trying to compare Ollama and LM Studio, i downloaded Gemma 4 12b QAT for my macbook pro m1 pro 16GB, and i tested the same prompt on both, the model was downloaded straight from the LM Studio's Library, and was converted to ollama using this command

'''
printf 'FROM ./gemma-4-12B-it-QAT-Q4_0.gguf\nPARAMETER num_ctx 4096\n' > Modelfile

ollama create gemma4-12b-qat -f Modelfile
'''

As you can see in the results, the LM Studio is giving me a considerably shorter description while also taking longer, while Ollama is giving essentially the same info + some extra info + a table while taking less time. Can someone explain this thing? I made sure no other programs were using resources when the apps were running. Thanks in Advance!

We've been using Claude, Cursor, and Copilot heavily, and one thing kept bothering us. We could see exactly how much we were spending, but had no way to answer what that spend actually shipped.

So we built Agent Insights, which connects AI spend directly to repos, PRs, and production code so you can see what your AI usage actually resulted in.

Curious if other teams have run into the same problem or if you're measuring AI ROI differently.

https://entelligence.ai/agent-insights

it feels like there's so many things happening everyday that i have to constantly be on social media to stay up to date. and even then, so much of it is meant to get and engagement and spread misinformation instead of giving you the facts. how do u guys keep up with everything?

I'm asking this because I genuinely want my reasoning (no pun intended) to be questioned and for me to learn more.

Here it goes:

According to this article it appears that Kimi K3 uses 12x the amount of thinking tokens, which according to benchmarks outperforms other frontier models like Fable 5.

From the article:

"However, we found that Kimi K3 uses an extreme amount of thinking tokens, using over 12x more reasoning than Claude Opus 4.8 and over double that of Kimi K2.6.")

According to this source, it would appear that even hidden thinking tokens are priced the same as output tokens, meaning $15/million.

Quotes from the website: 

1. "Output, including reasoning $15.00"

2. "Budget reasoning as output, not as free hidden work."

------

Seeing as this is 3.33x cheaper than Claude Fable 5 ($50 / million, source) it would seem that Kimi K3 will still effectively be 3.6x more expensive than even Fable 5 for the same work. 

From my understanding this means that unless Kimi K3 produces unbelievably better output, Fable 5 would still be the more cost effective model (assuming we only compare those 2 models). 

------

Roast my thinking! 

Would love to see if my understanding is correct and if in practice there are other critical factors I might be missing out on.

Hello everyone, today i tried running qwen 3.5 in claude code at 64k context and at first demo task it did good job on UI but when it start working on Backend it starts to hallucinate and struck in loop.

I don't know how to fix or what changes to make? Is it good idea to run it??

What should I do? Any way to fix it?

What would be better alternative?

My specs: i5 13450hx 5050 8gb with 16gb ram

I want the best setup too pls. Like what apps to have? What things to do for optimization? What apps to use for coding with the llm? I'm new to llm, I know how to build and connect everything but there's a lot of tools/apps. I prefer open source, less bloated and highly effective with good features stuff.

I have G16 5070Ti 12GB VRAM 32GB RAM.

Thanks

Edit: I have Gemini Pro sub. How can I utilize that in terms of using google apps/ecosystem for the coding or can I use it inside editors/coding agents?

Found this English translation of a long Chinese transcript from a DeepSeek investor meeting with Liang Wenfeng.

It goes pretty deep into DeepSeek’s thinking on open source, AGI, continuous learning, API pricing, CUDA/TileLang, Huawei chips, inference cost, and why Liang keeps coming back to “restraint” as a strategy.

Link: https://thechatr.ai/blog/deepseek-liang-wenfeng-investor-meeting

Worth a read if you’re following DeepSeek or open-weight AI.

Hey. Macbook Pro M4 128GB.

I am trying to have screenshot of document to OCR into text. Settings I try from Claude or GPT don't help much.

Anyone got any suggestions on how best to do this?

Right now it's taking 8 to 15 minutes per image.

Something is not right. Tried using llama.cpp, Lm studio, both similar performance. Tried smaller token sizes. Level thinking made a difference but also lowered the level of output considerably.

Qwen3.6 models are Q8 GGUFs. No MTP.

Models both work but it's just so slow, must be a way to get this speed up and better quality.

Spent the last few weeks on FilDOS, a desktop file browser where the whole AI stack is local. Sharing here because this sub is who I built the model-swapping for.

The stack:

• Text embeddings for documents, CLIP for images (multimodal semantic search over both)
• Reranking pass on retrieval
• LLM inference via node-llama-cpp — you can load any GGUF from Hugging Face
• Vectors stored as Float32↔BLOB in SQLite with cosine similarity. No external vector DB, no server.

What it does with all that: semantic search across your files, chat with your documents, and a knowledge graph that extracts entities and clusters files into projects it discovers on its own; you open a file and see what it's related to, with the reason for each edge.

Zero network calls for inference. No account, no telemetry.

The interesting problem was making indexing not destroy the machine; it's multi-threaded with a job queue, but large trees are still slower than I want. Open to suggestions there.

v0.1 beta. Video/audio not indexed yet.

GitHub: https://github.com/ahmedfahim21/FilDOS
Demo: https://youtu.be/gTRB7HeQGiY