diff --git a/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md b/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md index f4d50aa..c1849c2 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md +++ b/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md @@ -151,6 +151,39 @@ ctx-independent. At 262,144: 7.25 GiB scaled, 18.75 GiB total, 9.37 GiB/card. At 100,000: 2.77 GiB scaled, 14.3 GiB total, 7.1 GiB/card. `--parallel N` splits the same pool — slots do not multiply KV. +**Reshape (later the same day): four full slots, V q8_0, hot expert +blocks on-GPU.** `CONTEXT_SIZE=1048576` / `PARALLEL_SLOTS=4` — four +conversations, each the FULL native 262,144 window (per-slot window +unchanged, so no client or models.yml implication; slots divide the +pool, never multiply it). `CACHE_TYPE_V=q8_0` with K held at f16: main +KV drops 24,576 → 18,816 B/token (−23.4%), pool law 23,936 B/token. +The shared MTP drafter's KV ignores the V type (measured: draft pool +2,048 MiB = the f16 law at 1,048,576 cells — one layer's worth, the +miss costs ~0.5 GiB on ROCm1). At 4×262,144 the pool is ~23 GiB across +both cards: main 9,408 MiB + indexer 1,536 MiB per card, draft +2,048 MiB on ROCm1. + +Hot expert blocks stay GPU-resident. `-ot` granularity is the whole +per-block expert tensor (512 experts share one tensor), so "critical +experts" = blocks co-located on the card that holds their layers: +blocks 0-3 on ROCm0, 45-47 on ROCm1 (measured 2.42/2.49 GiB per +block). The placement law that makes it work: b11030 wants ONE +comma-joined `--override-tensor` (repeats log DEPRECATED), and the +loader applies the FIRST matching pattern per tensor — the +block-specific patterns MUST precede the broad `exps=CPU` sweep, or +the sweep silently swallows them (verified live: broad-first left +every expert on CPU with no warning anywhere in the log). + +Measured headroom (checkpoint floor: 3-4 GB per card): card0 5.02 GiB +free at boot, ~4.9 GB under load; card1 3.31 GiB, ~3.4 GB under load +(the vision graph costs ~0.43 GiB on card0 at first image). Blocks are +atomic — a 5th on card0 lands at 2.27 GB free, under the floor — so +4/3 is the deployed fill. Live numbers: content decode 13.05 t/s +(acceptance 0.58, mean len 2.16), reasoning phase 7.24 t/s +(acceptance 0.74, mean len 2.48), two concurrent full generations both +delivered, prompt cache reusing 42/64 tokens across requests. All +three Prometheus fleet targets up. + ## Rollback The lucebox DS4.1 material is fully retained on the host: image