Speculating on a Strix Halo (part 2)

So, part 1 was about learning about speculative decoding and trying it out (training and inference). The PyTorch speedup was nice (2.1×) but the rate was still pretty low because it's PyTorch bf16. The frustrating fact was that this was still way lower than the llama.cpp&Q8 weights which give 46 tok/s! I changed two things at once (engine and weights) for those numbers so I'm not claiming anything causal, but this does motivate my subsequent investigations.

The upshot of that embarrassment here is that maybe I can push up the fast 46 tok/s even higher... speedups are always nice. The brief of it is that: DFlash doesn't work as well for MoE models in these fast engines. This is because the decode step does not stream the same weights as the verify step (since the verify step wakes more experts), so memory bandwidth acts asymmetrically. I get to the measurements below, but roughly: decode (autoregressive generation) speeds up by a whopping 6×! Down from 125 ms to 21 ms. On the other hand, the draft-verify step only receives a 2.3× speedup, from 205 ms to 88 ms. As the verify becomes relatively slower, DFlash is less useful.

I only had loose guesses as to why this was happening. After some experiments (below), I now think this is due to the expert waking in MoE models. Drafts can be wrong, of course, and each wrong token drafted risks unnecessarily woken experts. This costs memory bandwidth.

Not to be doom and gloom though. DFlash still provides a speedup (in some structured corpi) which is very nice. I'll leave that more happy note in part 3. Here I discuss the initial measurements and tangents.

llama curiosities

I'm using an AMD box so the natural shift from part 1 is to DFlash on llama.cpp via Vulkan. It really wasn't too difficult to make this work - I could download the weights for both and hand it to Claude and it worked immediately. I had Claude generate a bunch of artificial prompts for getting a rough sense of timings with and without DFlash via llama-server.

I measured those rates but it was so bad LOL that it'd be dishonest to share them. Specifically, my measurements on greedy decode showed:

1. the outputs changed, which they shouldn't for a lossless speedup;
2. throughput degraded as a session continued for DFlash (worst case 22.8 tok/s down to 13.5 tok/s) but not for the base model;
3. and I missed an edge case (in my token generation count, I assumed that every cycle drafts all 15 tokens but that's not true with token budgets).

Below is my debugging of these oddities.

The verification cog

So DFlash isn't super complex, which I say as a strong compliment. But the above odd observations still cosmically tasked me with debugging this machine, so I will take it apart now... The easiest part to look at is, by-far, the verifier. The idea is that the verifier is just the base model run over the draft. The speedup in that cost is effectively the denominator in part 1's formula for η, so I measured that. I can get verify and decode on the same axes if I consider a span over the number of drafted tokens, k, so I measured that data below.

Good and bad news. Bad news: this slowdown of ~4.6x decode steps is pretty painful. If the drafter were free, this would divide the speedup by this ~4.6. Good news: this is the worst result I've so-far seen, so things get better. I think this slowdown is primarily because the model I'm studying (Qwen3.6-35B-A3B) is an MoE model, so there is the additional complication of expert weights being streamed (I allude a bit to this in part 1): each drafted token must waken a collection of experts to get verified. So increasing the size of a draft costs both compute and memory traffic. These costs should be measured relative to decode (autoregressive generation using the base model): any correct token will always (in the context of this work) require its experts being waken and run over it. When drafting, two things change: one, tokens can be incorrectly drafted which still wakes/wastes bandwidth in streaming experts and their compute; two, woken experts can be shared across the draft. This leaves the question of the relative bandwidth of between autoregressive and drafting unclear - which effect dominates? Well the data above shows that the wasted tokens hurt enough to cause a slowdown (positive slope), not a speedup.

To measure whether this slowdown is primarily due to wasted compute or bandwidth, one can send messages of a single token repeated across the block. This isn't a perfect experiment since the woken experts are not functions of the input tokens, but those of tokens AND their positions, but if we squint our eyes we might expect that more repeated tokens implies fewer experts. There is some evidence for this approximation: on some MATH-500 data, a block of a repeated token only woke ~31 experts per layer, significantly fewer than the ~52 experts that the MATH-500 or random text takes. That the repeated token block takes ~31 experts and not 8 (which a single decode step would need) explains its observed ~3x slowdown: the position-dependence of the woken experts means that ~4x more get woken even with the same token repeated. This tracking carries through on real text: MATH-500 text slows by ~4.7x while waking ~6.5x more experts. This gives a pretty suggestive picture that bandwidth is killing us here.

Comparing to the corresponding numbers in part 1 gives more evidence that this is bandwidth-bound. In part 1, I measured a T_t/L_target ratio of 1.6 decode steps, which implies a much more lenient τ > 1.6 requirement. The increasing ratio can be partially understood in the overall timespan of a step. By moving to llama.cpp, the verify pass sped up from 205 ms to 88 ms (2.3× faster) and the decode pass from 125 ms to 21 ms (6× faster). Take the limit in which compute was free to see the effective bandwidth these times imply: PyTorch decoding moves 5.8 GB over the 125 ms while llama.cpp at Q8 moves 3.1 GB over 21 ms. This gives rise to effective bandwidths of 47 GB/s for the old PyTorch runs and 147 GB/s for the new llama ones. These are both well below my Strix Halo's 256 GB/s rating, but pure weight streaming clearly explains more of the overall time in llama.cpp. The difference in the amount of memory moved is because of the different quantizations and number of experts woken (parameters streamed).

Why drafter performance doesn't really matter

This is a somewhat grim picture, since memory bandwidth is harder to argue with (optimize). I talk about some methods in part 3, but for here I take the approach of stepping up to this increasing demand. As can be seen in the above analysis, it holds even if the drafter is free, so the thing to improve is the number of accepted tokens, tau.

Each drafted token costs ~0.2 decode steps of verify (from the previous plot). So the token has to be accepted with an average probability of >0.2 or so. Well, T_t/L_target is not perfectly linear across the draft size k, so maybe τ isn't either? Of course, k → ∞ does NOT imply τ → ∞: the nth drafted token is only accepted if the prior (n-1) tokens were accepted, too. Over 7000 cycles, 31% accepted nothing, 37% accepted exactly one token, and 3.5% took the whole block. Those runs include the requests that terminated early, so I also checked without them: 22%, 39%, and 5%. These counts come from the runs I said I wouldn't show. The rates from those runs are junk, and the counts are shakier than I'd like, since the stale-K/V bug below perturbs the very logits that decide acceptance. But that perturbation is last-bit noise, and the margin here is 3.5% against 13%, so I use them. Either way the deep end is nowhere near often enough. At the 16th position the block costs about 0.13 decode steps per extra token, so it needs ~13%, and it delivers 3.5%.

Drafting shallower doesn't rescue it either. If acceptance were geometric, the bf16 τ of 4.9 at depth 15 implies an acceptance probability of about 0.80 per position. A depth-4 draft is a 5-position block, which interpolates the verify curve at 3.3 decode steps, and at 0.80 it commits τ ≈ 3.4 tokens per cycle: 3.4/3.3 ≈ 1.04×, against 1.05× at depth 15 — the same place, both sitting at break-even. Or hold the depth and push acceptance toward 0.95 per position, again a probability: τ ≈ 4.5 at depth 4 for 4.5/3.3 ≈ 1.4×, and τ ≈ 11 at depth 15 for 11/4.7 ≈ 2.3×. Both τ figures come from that geometric model, not measurement, and the counts above say the model is wrong in shape: geometric at τ = 4.9 predicts 20% of cycles accepting nothing and 16% exactly one, against the 31% and 37% I measured. Treat them as sketches. Part 3 measures this directly, and finds the same thing: shallower blocks don't beat deeper ones. Both also assume a free drafter: I never measured T_d on this engine, and on a decode this cheap it is unlikely to stay the 0.2 decode steps it was in bf16.

Two bugs upstream

Checking whether speculation was even lossless here turned up two llama.cpp bugs, and I filed both.

The first is a recurrent-state rollback slot that is reported restored but never written¹: a 68-line reproducer, bisected to the commit that added those per-draft-position state snapshots. It was triaged low priority: the snapshots are "only used for speculative decoding, where we never have to remove the anchor token," and the feature "is anyway marked experimental." My proposed fix² cost ~5% of decode to close something the only consumer of those snapshots cannot reach, and was declined: "~5% is ~5% too much for a bug that doesn't exist when using this feature."

The second is a Vulkan flash-attention issue³ where stale K/V left in cache slots that were freed, which speculative rollback produces every cycle, still changes the logits. The root cause is below llama.cpp, in the driver or the hardware. On this machine's RADV stack (Mesa's AMD Vulkan driver) on RDNA 3.5 hardware, the cooperative-matrix multiply-add (the shader path llama.cpp picks on this GPU, cm1 in the table below) returns different results depending on the sign of zero-valued inputs. A standalone 16x16 test reproduces that with no llama.cpp code involved: one column of -1.0, one row of small constants, everything else zero. Flip a row of those zeros to -0.0 and exact answers like -0.25 come back one bit off, while the same binaries on an NVIDIA card are bit-exact either way.

device: Radeon 8060S Graphics (RADV STRIX_HALO) | driver: radv Mesa 26.0.3-1ubuntu1 | accumulator: f32
computing, on the gpu, twice:
D[i,j] = sum over k of A[i,k] * B[k,j]
for 16x16 A, B with f16 elements.
A[i,12] = -1.0 and B[12,j] = (j+1)/16 are the only nonzeros, so every
D[i,j] = -(j+1)/16 exactly; the other 15 terms of each sum are zeros.
input check: the runs' B buffers differ in 16/256 elements, each 0x0000 (+0.0) -> 0x8000 (-0.0);
A is identical in both runs and is +0.0 in every column those elements multiply

  elem         gpu, +0 run  (bits)        gpu, -0 run  (bits)      |ulp|   cpu (double)
  D[ 0, 0]     -0.06250000  0xbd800000    -0.06250001  0xbd800001      1    -0.06250000
  D[ 0, 1]     -0.12500000  0xbe000000    -0.12500001  0xbe000001      1    -0.12500000
  D[ 0, 2]     -0.18750000  0xbe400000    -0.18750001  0xbe400001      1    -0.18750000
  D[ 0, 3]     -0.25000000  0xbe800000    -0.25000003  0xbe800001      1    -0.25000000
                                     [trimmed: it prints 10 of the 256]

differing elements: 256 / 256   (max |gpu(+0) - cpu| = 0.000e+00)
signed-zero sensitivity reproduced

Whose bug is it?

Back to the second bug. Running the original llama.cpp reproducer, real model and real rollback rather than a synthetic matrix, agrees. Same model, same prompt, same rollback, on everything I could run it on:

Forcing an NVIDIA card onto the same cm1 shader with GGML_VK_DISABLE_COOPMAT2=1 gives exact results, so the shader source isn't the problem. The obvious objection is that I bisected this to a llama.cpp commit, so how can it be the driver's fault? That commit rewrote the Vulkan flash-attention shader so that masked-out columns now pass through the matrix instruction. The instruction had presumably always behaved this way; nothing had fed it those inputs before. My standalone reproducer triggers it through the f32-accumulator instruction. I could not isolate the exact f16 configuration the attention shader uses, but sign-of-zero is confirmed as the trigger on the attention op itself.

And RADV is allowed to do this. SPV_KHR_cooperative_matrix⁴ says the order of the adds is implementation-dependent and leaves precision to Vulkan, and Vulkan doesn't specify it either. Vulkan also permits an implementation to ignore the sign of zero here. There is an opt-out flag, but it covers a fixed list of simpler operations and matrix multiply isn't on it. NVIDIA happens to be bit-exact on the same test, so the shader has been depending on something the spec doesn't guarantee. The fix is for the shader to zero its masked columns itself.

This reaches speculation because rollback frees cache slots without clearing them, so they still hold old K/V values, many of them negative. The mask sets those slots' attention weights to +0, and +0 times a negative number is -0, which is the input that triggers the difference. A few last-bit flips per attention op add up to logit shifts of up to about 8e-2 on a 4B model. I didn't measure the 35B model. That is large enough to flip a greedy token at a near-tie, but I never saw it happen.

Neither bug explains the degenerate replies that started all this. On a fresh server the speculative and plain runs agree for 26 tokens. The trace shows token 26 came from a cycle that accepted zero drafted tokens, so it is the target's own choice from an identical prefix, not a drafted token getting through. Prefill is fine. What's left is verify or rollback failing to restore state, the llama.cpp version of what I patched around in transformers. Why token 26 and not earlier, I don't know. This is still open.

References

1. 1. ggml-org/llama.cpp issue #26695, Eval bug: llama_memory_seq_rm returns true but doesn't restore the recurrent state when rolling back entire last decode. Filed by me, August 2026. Maintainer triage, 2026-08-08: "The n_rs_seq stuff is only used for speculative decoding, where we never have to remove the anchor token i.e the first token. I consider this a low priority bug since n_rs_seq is anyway marked experimental." On the proposed fix: "~5% is ~5% too much for a bug that doesn't exist when using this feature. It would likely not be merged."
2. 2. ggml-org/llama.cpp PR #25004, recurrent : support equal splits for recurrent-state rollback. The linked comment carries my one-slot alternative and the timing table: 16.06 ms/token at n_rs_seq=15 on the PR base, 47.41 on the PR, 16.79 with the one-slot fix (Qwen3.5-4B Q4_K_M, Vulkan, Radeon 8060S).
3. 3. ggml-org/llama.cpp issue #26744, Eval bug: vulkan flash attention lets stale K/V in freed cells influence the output. Filed by me, August 2026. "This bug does NOT occur if either GGML_VK_DISABLE_COOPMAT=1 or the computation is done on the CPU."
4. 4. Khronos. SPV_KHR_cooperative_matrix. "The order of the operations is implementation-dependent." "The internal precision of floating-point operations is defined by the client API."