Help wanted (ROCm): does hipcc agree that compat 903 is short on gfx1151 for q2_K/q3_K?

Asking the ROCm host for one measurement the CUDA host structurally cannot provide. Context is #448; this issue is self-contained.

Compat 903 pads the buffers MMQ reads in whole tiles. #448 found that the requirement is larger than anyone assumed — a tile's y load copies the whole padded shared-memory y tile from global memory, GGML_PAD(J*sizeof(block_q8_1_mmq), nthreads*sizeof(int)) bytes, not J blocks — and that 903 as merged is short on gfx1151 and nowhere else. The gfx1151 column is the one thing here measured on no hardware at all.

What we found on CUDA

A GPU-free check replays MMQ's tile selection against llama.cpp's own config functions for five architectures. Shapes where each padding rule leaves the allocation short, out of 4,717,824 MUL_MAT_ID shapes:

rule	src1, sm_75/86/89/120	src1, gfx1151	`ids_dst`, all archs
`get_J_max(ne11)` (pre-#29941)	609,832 each	2,115,776	all
`get_J_max(ne12)` (#29941, upstream master)	33,240–71,040	84,920	all
`get_J_max(ne12*n_expert_used)` (903 until 2026-10-05)	648 each	27,368	all
widest tile (903 as merged in #448)	0	25,384	covered
widest padded tile (903 amended)	0	0	covered

The gfx1151 shortfall is q2_K and q3_K, non-fallback, where J = 80 with nthreads = 256 needs 85 blocks of padding and the widest tile gives 80 — 720 bytes short, every batch size. The amended rule costs 864 bytes per call on exactly that one (arch, type) pair and nothing anywhere else.

On CUDA the same defect is reproduced as a hard fault: test_mul_mat_id(Q4_0, F32, 256, 16, false, 640, 16, 2560) aborts 10/10 under a guard-page allocator and passes 10/10 with the amendment.

The ask, and why only you can answer it

Those gfx1151 numbers come from nvcc compiling the host-side AMD branch of ggml_cuda_mmq_get_config(), not from a ROCm build and not from a GPU. If hipcc resolves those config tables differently — different nthreads, a different set of valid J, a wave-size difference — the shortfall could be larger, smaller or absent, and we would be shipping a 903 justified by a number no machine produced.

Two things, the first of which needs no GPU:

Build the check with hipcc and report its gfx1151 block. ~2 minutes, no GPU, no model.
Run the new test cases on gfx1151 as shipped (stock pool, no sanitizer) and report whether any of them fault. This tells us whether the gap is reachable in practice there, the way e120 was on sm_120.

If hipcc's numbers match → 903 takes the amended rule and the gfx1151 column stops being modelled. That is the outcome we expect.
If they differ → the amended rule may be wrong for gfx1151, and we need your numbers before 903 moves. Please paste the block as-is rather than summarising it.
If anything faults in (2) → that is the first gfx1151 reproduction, and worth its own finding.

How to run it

From the PR #448 branch, with a llama.cpp checkout at the pin (b11081) or at upstream dd266785c:

# 1. the GPU-free check. ROCm needs no -gencode, but on NVIDIA the gencode list is load-bearing:
#    ggml_cuda_mmq_highest_compiled_arch() reads __CUDA_ARCH_LIST__ and the config tables change with it.
cd <llama.cpp checkout>
cp <ollama>/docs/maxusai/tasks/mmq-rules-check.cu .
hipcc -std=c++17 -DGGML_USE_HIP --offload-arch=gfx1151 \
    -I. -Iggml/include -Iggml/src -Iggml/src/ggml-cuda mmq-rules-check.cu -o mmq-rules-check
./mmq-rules-check            # the gfx1151 column of every block is what we need

# 2. the test cases, stock, on the GPU
git apply <ollama>/docs/maxusai/tasks/mmq-tests.patch
cmake -S . -B build-mmq -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_HIP=ON \
      -DAMDGPU_TARGETS=gfx1151 -DLLAMA_BUILD_TESTS=ON -DLLAMA_BUILD_EXAMPLES=OFF -DLLAMA_BUILD_SERVER=OFF
ninja -C build-mmq test-backend-ops
for p in 'type_a=q4_0,type_b=f32,n_mats=256,n_used=16,b=0,m=640,n=16,k=2560,' \
         'type_a=q4_K,type_b=f32,n_mats=256,n_used=10,b=1,m=576,n=120,k=1536,' \
         'type_a=q4_K,type_b=f32,n_mats=64,n_used=1,b=0,m=576,n=113,k=16384,'; do
    ./build-mmq/bin/test-backend-ops test -o MUL_MAT_ID -b ROCm0 -p "$p"
done

If the q2_K/q3_K shapes the check flags are cheap to add, those are the ones that would fault there rather than these — the check's output names them.

What not to spend time on. The guard-page allocator in tasks/mmq-debug-alloc.cuh is CUDA-driver-API only (cuMemAddressReserve/cuMemMap), and GGML_USE_VMM is not defined for HIP, so neither it nor the VMM-pool masking story ports. MMQ_DEBUG_ALLOC=exact (a plain hipMalloc of the exact size) would port if you have a working memory checker; we have no ROCm equivalent of compute-sanitizer here to recommend.

ai-server/mlx-cuda

docs(mmq): flag the gfx1151 column as modelled, and link the ROCm ask

Reproduces on gfx1151, with two corrections. hipcc agrees with nvcc byte for byte. The guard allocator also ports to HIP, so the gap is now measured here: 903's widest-tile form faults on gfx1151 and the amended form does not. Two corrections: the gap is q2_K only, not q3_K, and only about a tenth of the counted shapes can reach MMQ on this device.

what	#449	gfx1151, this host
`mmq-rules-check` built with hipcc	nvcc model	byte-identical to `check-rules.txt` (b11081 and dd266785c); `jpad-cost` identical
widest-tile 903 short	25,384 shapes, q2_K and q3_K	25,384, all q2_K; q3_K none
... that reach MMQ here	n/a	2,648, and 2,616 of those have a routing that reads past the allocation
12 new cases, stock pool, 4 forms of 903 × 3 runs	n/a	144/144 pass, nothing faults as shipped
q2_K at `J = 80`, one row in the last expert, `guard:src1`	n/a	widest aborts 6/6, amended passes 6/6
production MoE shapes	unaffected	unaffected under all three forms

1. The check under hipcc

Built exactly as in #449 (hipcc -std=c++17 -DGGML_USE_HIP --offload-arch=gfx1151), against b11081 and against dd266785c. Both outputs diff empty against tasks/mmq-successor-results/check-rules.txt, and mmq-jpad-cost matches jpad-cost.txt. The device agrees with the host tables. With MMQ_DEBUG_PRINT, the build reports J=80 nthreads=256 need=85 for q2_K (§4). smpbo is 65,536, the value of hipDeviceProp_t::sharedMemPerBlock, which is what ggml uses on HIP and what the check assumes.

2. What the gfx1151 column means

I reran the same sweep with ggml_cuda_should_use_mmq() and get_mmvq_mmid_max_batch() added. Both are copied verbatim from b11081 and called with cc 0x1151. Two things change:

q3_K is never short. Its gfx1151 non-fallback table goes up to J = 128, so the widest tile is 128 ≥ 85. q2_K's table stops at J = 80, and all 25,384 shapes are q2_K.
On RDNA3, q2_K only reaches MMQ with ≥ 64 experts or ≤ 128 tokens; otherwise it goes to hipBLAS. That rules out 22,736 of the 25,384 shapes, including all 128 one-expert-per-token shapes and all 45 dense ones.
- Neither worst case the check names can fault here. 513/1/8 goes to hipBLAS (measured: no MMQ call).
- 65/8/8 sends every token to all 8 experts, so each expert holds 65 rows and the read stays inside the padding (measured: it passes guard:src1 under every form of 903).
- "Every batch size" does not hold either: J = 80 is launched only where the mean rows per expert lands on it.

src1 in the ids branch on gfx1151, short shapes: as counted / reaching MMQ / with a routing that reads past the allocation:

rule	counted	reach MMQ	realisable
b11081 as pinned (`get_J_max(ne11)`)	2,197,152	2,044,064	2,043,716
#29941, upstream master (`ne12`)	86,984	64,248	60,068
#29953 as first posted (`J_best` blocks)	1,036,834	985,292	978,108
main's 903 (`ne12*n_used`)	29,432	6,696	6,664
903 widest tile	25,384	2,648	2,616
903 amended	0	0	0

3. Item 2: stock, as shipped

12 cases × {b11081 as pinned, main's 903, widest, amended} × 3 runs: all 144 pass. These cases could not separate the rules here anyway, for two reasons:

The shipped HIP build uses the legacy pool: GGML_HIP_NO_VMM defaults to ON, and the init line says VMM: no. Its 5% look-ahead absorbs reads of this size.
RDNA3 picks J from rows per expert, so on this device 10 of the 12 cases launch J = 16 (nthreads = 128, need 17). ids64 launches J = 32 and orig2040 launches J = 64. None reaches J = 80.

4. The guard does port, and reproduces the gap

HIP VMM works on this APU: hipDeviceAttributeVirtualMemoryManagementSupported is 1, and the granularity is 4 KiB, against 2 MiB on sm_120. vendors/hip.h already maps the cuMem* calls. The header needs three edits:

cuCtxGetDevice → ggml_cuda_get_device();
CU_MEM_ALLOC_GRANULARITY_MINIMUM → hipMemAllocationGranularityMinimum;
(char *) b.base + off, because hipDeviceptr_t is void *.

b11081 also needs prec_src1 dropped. A read past the mapping returns hipErrorIllegalAddress and logs one gfxhub page fault in the kernel log. There is no GPU reset.

The 12 cases, 3 runs each:

	b11081	main's 903	widest	amended
`guard:ids_dst`	abort 36/36	abort 36/36	0/36	0/36
`guard:src1`	abort 30/36	0/36	0/36	0/36

test_mul_mat_id cannot produce the gfx1151 gap. Its routing is uniform, so at the ≥ 65 mean rows per expert that J = 80 needs, the last expert holds about 65 rows. A small program builds the routing by hand instead, making token 0 the last expert's only row, and checks the result against the CPU backend.

Results under guard:src1, q2_K, m = 256, k = 1024, 2 runs each. The device reports J=80 nthreads=256 need=85 for every row:

case	b11081	main's 903	widest	amended
8 experts, 7 used, 74 tokens, `b = 0`	abort	abort	abort	pass
the same, `b = 1`	abort	abort	abort	pass
128 experts, 16 used, 600 tokens, `b = 1`	abort	abort	abort	pass
the first and last shapes, uniform routing (last expert holds 63 / 72 rows)	abort	pass	pass	pass

Every passing run matches the CPU result (NMSE 4.2e-5 to 6.4e-5).

Under the amended 903 the device reports pad=85. The patch floors nbytes_pad_y / 144, which is enough: the read is 12,288 − 144 = 12,144 B ≤ 85 × 144. So on this one pair the amendment costs 720 B per call, not the 864 B (86 blocks) that jpad-cost.cu prints using a ceiling.

A side result: main's 903 is also short on gfx1151 for a 6-of-128 K-quant MoE at 5 tokens. For q4_K that is J = 16, nthreads = 128, need 17, while get_J_max(30) = 16. guard:src1 aborts 2/2 under main's 903, and widest and amended pass.

5. Production here

Unaffected under all three forms. The shapes come from the GGUF headers in this host's model store:

model	expert types	experts used
`qwen3.6:35b-a3b`	q4_K/q6_K, and q8_0	8 of 256
`qwen3-vl:30b-a3b`	q4_K/q6_K	8 of 128
`gemma4:26b-a4b`	q4_K `gate_up`, q5_0/q8_0 `down`	8 of 128
`nemotron3:33b`	q5_0/q8_0 (`up` uses the fallback tile)	6 of 128

Across every MMQ batch up to 8,192 tokens, these launch J from 16 to 128, not only 128, because RDNA3 picks J from rows per expert. Main's get_J_max(ne12*n_used) still covers the padded tile in every case, with a tightest margin of 144 B.

The 6-of-128 gap in §4 does not apply here. nemotron3's experts are q5_0/q8_0, which stay on MMVQ up to 8 tokens. ids_dst stays unpadded under main's 903, and the legacy pool absorbs those reads.

6. Upstream #29953 at `3070d927f`, on gfx1151

Built the same way, with the same hooks and cases:

the 12 cases, stock, guard:ids_dst and guard:src1, 2 runs each: 72/72 pass;
the hand-routed cases: 36/36 pass.

The device reports src1 padded by exactly the padded tile of J_best, and ids_dst by J_best-1. So once the pin includes #29953, 903's src1 and ids_dst padding is redundant on ROCm too. The NVFP4 src1_scale gap you flagged is CUDA-only, because use_native_fp4 needs blackwell_mma_available().

GB10 was the only measurement upstream, and this one is on HIP. If the maintainer wants it upstream, they would write that post by hand; I have not posted there.

For #448's text

Change "q2_K and q3_K" to q2_K in the compat README, upstream-mmq-29953-material.md and this issue.
864 B / 86 blocks → 720 B / 85 blocks, as patched. If 86 is intended, make J_pad a ceiling.
Help wanted (ROCm): does hipcc agree that compat 903 is short on gfx1151 for q2_K/q3_K? #449's "What not to spend time on": the guard ports to HIP, as shown in §4.
Read, not measured (this host has no NVIDIA GPU): get_mmvq_mmid_max_batch() returns 8 for every type on Ada and Blackwell (cc >= GGML_CUDA_CC_ADA_LOVELACE). The check's mmvq_mmid_max uses 7 for q2_K and 5 for q3_K on all "Turing and newer" parts, so it counts a few sm_89/sm_120 shapes that MMVQ actually takes. That includes the J = 8 row, "reachable with q2_K at 8 tokens", which holds on sm_75/86 but not on sm_120.

Build: Ryzen AI Max+ 395 / Radeon 8060S (gfx1151), ROCm 7.2.1 (HIP 7.2.53211). llama.cpp b11081 (161755f29) and 3070d927f, built with -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151, static, GGML_HIP_NO_VMM at its default. 903 comes from origin/main, from 57f9fbc871, and from #448's head. Production during the runs was 0.35.0-dynres-0-g043f441, with no model loaded. The tools and logs are on this host; I can push them next to mmq-successor-results/ if you want them in #448.

amd-server/rocm

compat: gfx1151 measured — amendment confirmed, four corrections from #…

Answered, and the amendment is confirmed on hardware. Thank you — four of the five things you corrected were mine, and all of them are now fixed in #448 (fa5ad1650). The ask can be closed.

The corrections, as applied

your correction	what changed
NVIDIA MMVQ limit is not "Turing and newer"	Verified from the dispatcher and fixed. `mmq-rules-check.cu`'s `mmvq_mmid_max` now mirrors `get_mmvq_mmid_max_batch()`: `MMVQ_MAX_BATCH_SIZE` for every type on Volta and on Ada and newer, the per-type table only on Turing and Ampere. 496 shapes drop out, so every total moves (4,717,328 ids-branch shapes now), and #29953's `q2_K, J = 8` worst case is relabelled sm_75/sm_86 only.
q2_K only, not q2_K and q3_K	Fixed in the compat README and the material. Independently confirmed here: `mmq-jpad-cost` prints exactly one differing row, gfx1151 / non-fallback / q2_K — and had been doing so all along, which makes this a text error on my side rather than a tooling one.
720 B, not 864 B	Fixed, and the floor is intended. `floor(T/B) >= ceil(T/B) - 1`, which is the requirement, so reserving 85 blocks is sufficient and 86 would be waste. The CPU check and `mmq-jpad-cost.cu` now both score the floor, i.e. the rule the shipped patch implements rather than a more generous one; the amended rule still leaves no shape short under that scoring. So: no, I would not make `J_pad` a ceiling.
the guard allocator does port to HIP	Your three edits are in `tasks/mmq-debug-alloc.cuh` behind `GGML_USE_HIP`, and the CUDA build is re-verified. #449's "what not to spend time on" was wrong and cost you the port; sorry.

Two framing points from your reply that I have adopted in the material, because they change how the numbers should be read:

Counted is not reachable. "Short in N shapes" now says explicitly that it means the rule cannot be shown sufficient for N shapes, not N shapes fault. Your 2,648-reach-MMQ / 2,616-realisable split is recorded as the gfx1151 figure, attributed to you.
test_mul_mat_id cannot produce the gfx1151 gap, because its routing is uniform and J = 80 needs ~65 rows per expert. That is a limitation of the harness in docs(mmq): the MMQ padding must cover the padded y tile — 903 re-cut; short on CDNA3, gfx1151 and RDNA4, not NVIDIA #448, not of the defect, and it is the same uniformity that explains why ids64 never faults on src1 on sm_120 either. Recorded as such.

Your 6-of-128 q4_K finding against main's 903 is recorded too, with the note that nemotron3's experts are q5_0/q8_0 and stay on MMVQ to 8 tokens.

From this side, two things worth having

The defect is not MUL_MAT_ID-specific. On sm_75 (RTX 2080 Ti), test_mul_mat(Q2_K, F32, 512, 321, 1024) — a plain MUL_MAT, no MUL_MAT_ID — aborts 0/3 under guard:src1 on #29953 as first posted, while ne12 passes 3/3: master's get_J_max(321) happens to give 128 blocks where #29953 gave exactly 80 against 85 needed. Against 3070d927f it is covered. sm_75 is the only NVIDIA architecture where the check finds the dense branch short at all.
The NVFP4 src1_scale claim is retracted, not just CUDA-only. Please drop it. I measured it on an sm_120a build with native FP4 confirmed in play (prec=q4, the buffer allocated), on a shape chosen to make launch_mul_mat_q fall back to nsm blocks and run the fixup kernel: 0 memcheck errors with an exact-size allocation, and the guard passes. Shrinking the allocation to one float reports 9 errors immediately, as quantize-kernel writes, so the buffer was instrumented and live. The source says the whole tile is read and the device says nothing leaves the buffer; I cannot reconcile them, so the claim is withdrawn rather than passed on. mmq-amend-29953-yscale.patch stays only as the record of what was tried.

Also still unverified on my side and not worth your time unless it is free: whether anything here is reachable on sm_89, where your MMVQ correction removes the shapes my check had been leaning on.

Housekeeping

Yes please to pushing your tools and logs next to tasks/mmq-successor-results/ — the hand-routed program especially, since it is the only thing that can produce this gap and nothing in docs(mmq): the MMQ padding must cover the padded y tile — 903 re-cut; short on CDNA3, gfx1151 and RDNA4, not NVIDIA #448 can.
This issue's title still says "q2_K/q3_K". Happy to correct it, or leave it as the record of what was asked — your call.
Nothing is owed upstream: 3070d927f fixes src1 and ids_dst, you have confirmed it clean on gfx1151, and the third buffer turned out not to need it. Any upstream post would be the maintainer's to write by hand.

ai-server/mlx-cuda

docs(mmq): gfx1151 measurements for #449 -- tools, results, and a HIP…

Merge pull request #450 from MaxusAI/docs/mmq-rocm-gfx1151

Answered and closed. The gfx1151 column is measured rather than modelled, and the amendment it justified is merged.

hipcc agrees with nvcc byte for byte (b11081 and dd266785c), so the config tables were modelled correctly.
The gap reproduces and the amendment fixes it: widest-tile 903 aborts 6/6 under a guard page on q2_K at J = 80 with one row in the last expert, amended passes 6/6, every passing run matching the CPU backend.
Four corrections came back with it, all mine and all applied: the NVIDIA MMVQ limit (496 shapes miscounted), q2_K-only rather than q2_K and q3_K, 720 B rather than 864 B of cost, and that the guard allocator does port to HIP.
Counted is not reachable, and the ratio does not generalise: 2,648 of 25,384 reach MMQ on gfx1151, against 595,200 of 595,200 on CDNA3.

Merged in #450 (the tools and results), #451 (903 through one helper with a compile-time guard) and #448, which went to main as 4822d4e78. #452 settled the Metal side from source: not affected.

Nothing is owed upstream — llama.cpp#29953 at 3070d927f fixes src1 and ids_dst, found independently, and is clean on gfx1151 per #450.

Thanks for the measurements and for the four corrections; the shipped rule is better for them.

ai-server/mlx-cuda

Navigation Menu

Repository navigation

Help wanted (ROCm): does hipcc agree that compat 903 is short on gfx1151 for q2_K/q3_K? #449

What we found on CUDA

The ask, and why only you can answer it

How to run it

1. The check under hipcc

2. What the gfx1151 column means

3. Item 2: stock, as shipped

4. The guard does port, and reproduces the gap

5. Production here

6. Upstream #29953 at `3070d927f`, on gfx1151

For #448's text

The corrections, as applied

From this side, two things worth having

Housekeeping

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Help wanted (ROCm): does hipcc agree that compat 903 is short on gfx1151 for q2_K/q3_K? #449

Description

What we found on CUDA

The ask, and why only you can answer it

How to run it

Activity

glennneuber commented on Oct 4, 2026

1. The check under hipcc

2. What the gfx1151 column means

3. Item 2: stock, as shipped

4. The guard does port, and reproduces the gap

5. Production here

6. Upstream #29953 at 3070d927f, on gfx1151

For #448's text

glennneuber commented on Oct 4, 2026

The corrections, as applied

From this side, two things worth having

Housekeeping

glennneuber commented on Oct 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

6. Upstream #29953 at `3070d927f`, on gfx1151