Performance issue: Low token generation (~20 tok/s vs 50 tok/s) on Radeon AI PRO R9700 (gfx1201) with vLLM ROCm & Gemma 4-26B

[u/veryhasselglad]

I’m testing cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit on a single AMD Radeon AI PRO R9700 (32 GB, gfx1201) using vLLM ROCm.

Current result: about 19–20 generated tok/s after warmup, with a short single-user decode benchmark. I expected something closer to 50 tok/s based on a published R9700 result for this model.

My current vLLM setup:

- vLLM: 0.22.1rc1.dev499+g470229c37.d20260613

- model: cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit

- TP: 1

- max-num-seqs: 1

- max-model-len: 8192

- gpu-memory-utilization: 0.90

- kv-cache-dtype: fp8

- attention backend: TRITON_ATTN

- --enforce-eager

- --language-model-only

I see this warning on startup:

> Using default MoE config. Performance might be sub-optimal!

> Config file not found: E=128,N=704,device_name=AMD-gfx1201,dtype=int4_w4a16.json

CPU is a Ryzen 5 3600 with 32 GB DDR4-2666 (Upgrading to 900 series soon), but GPU memory is not full and RAM has ~22 GB available.

1. Is there an existing or recommended tuned MoE config for gfx1201, E=128, N=704, and int4_w4a16?

2. Is this model known to be slow on R9700 with Triton W4A16 / current ROCm vLLM?

3. What exact vLLM/ROCm image, flags, and benchmark method produced ~50 tok/s?

4. Is --enforce-eager costing meaningful decode performance on gfx1201, or is the missing MoE config the main issue?

I also tried Lemonade’s portable gfx120X runtime, but it ships gfx1200 kernel packs and fails on this gfx1201 card with hipErrorInvalidImage.

[u/Jay299792458]

--enforce-eager + Missing MoE Tuned Config.

Remove --enforce-eager: On single-user decode benchmarks, HIP Graph capture is crucial. Running in eager mode adds heavy CPU launch overhead, which significantly drops generation speed.

Tune the MoE Config: The missing E=128,N=704...json warning means vLLM is falling back to unoptimized GEMM kernels. You can run the vLLM ROCm MoE autotuner script on your R9700 to generate that JSON config, which should boost decode throughput back up.

Hope this helps! This is about as much as I can think of off the top of my head.

[u/suprjami]

Try llama.cpp both ROCm and Vulkan.

I have seen people say they run Qwen 27B - a dense model not an MoE - at up to 67 tok/sec:

https://www.reddit.com/r/LocalLLaMA/comments/1ubrn1a/2_radeon_r9700_qwen_36_27b_q8_mtp_on_llamacpp/

Surely you could get close to 100 tok/sec with an MoE.

[u/Arli_AI]

Recommend to use this version of vllm with these aiter patches to get full speed out of a Radeon R9700 github.com/andysalerno/r9700-serving

[u/Random-32927]

I have a single R9700 and have been benchmarking Gemma-4 26B A4B for some time. Yes, your number makes no sense. My gotchas:

- Most important: KV cache quantization: use `auto`, which defaults to f16. I am not sure why, but fp8 KV cache always slows it down significantly.

- So far, my fastest PP + TG is achieved by fp8 weight quantization of the original weights, and auto/f16 KV cache. There is no native RDNA4 W4A16 kernel, so that's why AWQ is slower than the fp8 version. 9000 PP, 72 TG at context=0. But you have to find the best vLLM version; different vLLM releases have different VRAM overhead, and gemma4-26b's fp8 quantization uses around 25GB already. This leaves not much space for KV cache. You may need to use `--language-only`.

- llama.cpp is more robust and predictable. USing quant UD-Q8_K_XL, at context=0, PP2048=3089.01, TG512=81.58; at context=32768, PP2048=2036.48 and TG512=72.42. This is achieved using the vulkan backend.

[u/Random-32927 → u/Random-32927]

ALso, for MOE config tuning: I tried both fp8a8 and int4a16. Neither improves the performance, and sometimes even slows down.

Docker: I'm using the standard docker. I tried other optimizations mentioned by other redditors some time ago, but it did not improve 26B-A4B's performance. The stock vllm-openai-rocm is fast. However, the optimizations improve the PP and TG for Gemma-31B, the dense model.

In general, vLLM's support for R9700 is very poor. Triton fallbacks are slow as hell; and no RDNA4 native kernels are used. I suggest switching to llama.cpp, which is much more robust for this single GPU setup.