Paging the n-gram table

This post inspired me to play with Qwen3.8-Flash-Next on my Strix Halo. It made the prospect of putting this model on my card seem easy, in contrast to the fighting/debugging that I normally anticipate. After reading that post, my original goal was to see if I could, offline, serve such a powerful model and push its performance (tokens/sec) higher. This exploration instead ended up being a fun lesson in modern memory-optimization methods which, since so much is going on in ML right now, I was previously unaware of.

The TLDR is that there is a ~30GB table (the n-gram table) which gets ~no memory traffic so the vast majority of it can be offloaded with little hit to decode rate. Of course this isn't a novel observation, but it's fun to see how hard you can push these things. I give below the final config (needs a custom fork/quant... see below) that I used for decode rates near the max, while offloading the n-gram table to disk:

llama-server \
  -m Qwen3.8-Flash-Next-UD-IQ4_XS-plesplit.gguf \
  -md Qwen3.8-Flash-Next-MTP-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  -ngl 999 -fa 1 -b 8192 -ub 2048 -c 131072 --jinja \
  -lm mmap --numa distribute -ot "ple_ngram_embd=CPU"

decoding the commands because they still sometimes look foreign to me. First,

-md Qwen3.8-Flash-Next-MTP-Q8_0.gguf \
--spec-type draft-mtp --spec-draft-n-max 4

enables speculative decoding (have a bit of the model predict future tokens so you can check them in parallel... serial -> parallel enables a speedup even if some tokens are wrong...). Second,

-ngl 999 -fa 1 -b 8192 -ub 2048 -c 131072 --jinja

are somewhat standard flags.. the -ngl 999 says to put all layers into VRAM (which we then screw with in the next set of flags), -fa 1 turns on flash attention, -b 8192 and -ub 2048 are batch flags (primarily for prefill), -c 131072 sets the model at a reasonable context of 128k, and --jinja enables the Jinja template so the model can use reasoning. Finally,

-lm mmap --numa distribute -ot "ple_ngram_embd=CPU"

offloads the PLE/n-gram table to the CPU, enabling the big memory reduction. The -lm mmap is what makes that work: the table stays a view into the file on disk rather than being copied into a GPU buffer.

To run the above code (at least as of writing this), you need to use the llama.cpp fork, branch vulkan/qwen4exp-rocmfpx, commit 799bf4dd8 (2026-08-27). Also you need to split off PLE heads from the model weights. That can be done via

cd llama.cpp        # the fork you cloned above
uv run --no-project --with numpy --with pyyaml \
  python gguf-py/gguf/scripts/gguf_split_ple_heads.py \
  Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf \
  Qwen3.8-Flash-Next-UD-IQ4_XS-plesplit.gguf

Let me know if you find better configs!

Setup

I tried to follow the previous reddit post since it seemed nice, so I used their recommended quant UD-IQ4_XS. I think this is pretty good quality while also fitting on my machine... total (without KV) this should be ~90GB or so. With context it should still be under 128GB.

I also used this provided MTP for speculative decoding since that's generally a free speedup (other than a bit of memory, but the n-gram table turns out super-useful here as I discuss below).

Token Rates

I began trying to get lucky (or persistent) in fiddling with config so as to further optimize the token rates. I had the naive ambition that I could find some nice parameters that led to crazy rates. Mostly struck out there (unless I further quantized weights which made me nervous about token quality), but I did recreate many of the nice rates in this post and others: (measured by reading prompt_per_second and predicted_per_second from llama-server, over 5 wikitext prompts with ignore_eos. These are end-to-end server timings, so they read lower than llama-bench on the same box — pp512 is 453 under llama-bench here, against the 390 in the linked post.)

Those rates are already nice, but they weren't really budging any further. With that door closed, I looked to the other: maybe the underlying code is inefficient and profiling would pinpoint it. Good and bad news. Good news: most of the time was in memory traffic and matrix multiplies, so there are no obvious algorithmic inefficiencies slowing down my inference. Bad news: there are no obvious inefficiencies. Well, there was an inefficiency in my memory usage due to how I configured things...

The n-gram Table

Maybe I'm alone in this but, honestly, I had no idea what an n-gram table or PLE was before this. But it quickly caught my attention when I learned it was taking ~30% of the memory associated to this model (~29GB). It's a big fella, 320,001,446 x 160 of data type IQ4_NL. What surprised me about this was that, despite it costing so much memory, we barely read from it. Initial memory traffic measurements showed it near 0% of the traffic... This is because, each decode step, one just reads only 16 rows, and certain rows tend to be much hotter than others (I'm thinking Zipf style analysis).

This became both a thorn and a curiousity. A thorn because 29GB is a ton of memory, but also a curiousity in how to optimize the data access. I tried the (obvious?) next step: since so little of this table needs to be used in practice, maybe we could use a Zipf-inspired truncation of the table. I.e., store in memory only the most heavily-used rows, defaulting to a disk-read if the model requested an uncommon row. This was inspired by the reduced-vocabulary idea I learned about in my speculative decoding experiments. Of course this is not novel, but learning is still fun :). Such an idea works best if the actively-used table is heavily concentrated and if the tail is thin, which ended up being the case-ish. For a given subject, the hot rows indeed are concentrated and the tail is thin:

Code

Docs

Prose

Everything at once

One caveat: I fed each corpus whatever text I had, so they aren't the same size. Rows touched grows with tokens, so the sort partly ranks corpus size rather than how concentrated a subject is.

See the figure below:

More aggressively, we can group the subjects and see how much of the table was needed to run over any of these subjects (so run over all corpi corresponding to code, for example).

But the hot-rows don't transfer too well across subjects... E.g., when running over the prose corpi, only 18.8% of reads landed on rows that were also read by the code corpi... That's pretty bad for a single-reduced table idea (also it suggests that Qwen didn't make an unnecessarily large table!) but maybe that's not needed. An LLM session is typically fairly localized (e.g., I don't shift from code to Moby Dick), so a cache (e.g., LRU) might still allow good reductions. Maybe the table could still be aggressively reduced and the (hopefully rare cache misses) would just cost a semi-negligible slowdown compared to the memory saving.

I tried exactly this tradeoff via:

-lm mmap --numa distribute -ot "ple_ngram_embd=CPU"

This offloads the table to CPU and lets the kernel track it page-by-page (4KB each, though readahead pulls in more like 256KB per fault). Nothing limits how much stays resident other than memory pressure, at which point the kernel drops the pages it hasn't touched recently. To see the cost on performance, I ran those configs over the following corpi (C, C++, Python, shell, Markdown, WikiText, KJV, Shakespeare, Alice, Darwin, Frankenstein, sci-fi, Homer, Kant, Tolstoy, Joyce), same prompts, same order, capped at 192 tokens each (they don't all hit the cap, and I dropped 2 of the 20 prompts that degenerated):

so that's a pretty small hit to decode rate, well within noise, but we reduce the memory footprint by ~30GB! I could probably fit a larger quant because of this.

Sizes of Qwen3.8-Flash-Next in the Limit of No Table

So I got curious about running this on my other computer (which has ~48GB VRAM, ~30GB system RAM). This is a more traditional machine with a CPU, RAM sticks, and discrete GPU. It ended up not working (yet... well I got a trickle of ~12 tok/sec) but I did record below, in the limit of no n-gram table, what I could run: