frank-primary runbook: record the 4-slot/V-q8/GPU-block reshape

2026-09-26 21:32:09 -07:00
parent de85654b76
commit e330d56a06
1 changed files with 33 additions and 0 deletions
@@ -151,6 +151,39 @@ ctx-independent. At 262,144: 7.25 GiB scaled, 18.75 GiB total, 9.37
 GiB/card. At 100,000: 2.77 GiB scaled, 14.3 GiB total, 7.1 GiB/card.
 `--parallel N` splits the same pool — slots do not multiply KV.

+**Reshape (later the same day): four full slots, V q8_0, hot expert
+blocks on-GPU.** `CONTEXT_SIZE=1048576` / `PARALLEL_SLOTS=4` — four
+conversations, each the FULL native 262,144 window (per-slot window
+unchanged, so no client or models.yml implication; slots divide the
+pool, never multiply it). `CACHE_TYPE_V=q8_0` with K held at f16: main
+KV drops 24,576 → 18,816 B/token (−23.4%), pool law 23,936 B/token.
+The shared MTP drafter's KV ignores the V type (measured: draft pool
+2,048 MiB = the f16 law at 1,048,576 cells — one layer's worth, the
+miss costs ~0.5 GiB on ROCm1). At 4×262,144 the pool is ~23 GiB across
+both cards: main 9,408 MiB + indexer 1,536 MiB per card, draft
+2,048 MiB on ROCm1.
+
+Hot expert blocks stay GPU-resident. `-ot` granularity is the whole
+per-block expert tensor (512 experts share one tensor), so "critical
+experts" = blocks co-located on the card that holds their layers:
+blocks 0-3 on ROCm0, 45-47 on ROCm1 (measured 2.42/2.49 GiB per
+block). The placement law that makes it work: b11030 wants ONE
+comma-joined `--override-tensor` (repeats log DEPRECATED), and the
+loader applies the FIRST matching pattern per tensor — the
+block-specific patterns MUST precede the broad `exps=CPU` sweep, or
+the sweep silently swallows them (verified live: broad-first left
+every expert on CPU with no warning anywhere in the log).
+
+Measured headroom (checkpoint floor: 3-4 GB per card): card0 5.02 GiB
+free at boot, ~4.9 GB under load; card1 3.31 GiB, ~3.4 GB under load
+(the vision graph costs ~0.43 GiB on card0 at first image). Blocks are
+atomic — a 5th on card0 lands at 2.27 GB free, under the floor — so
+4/3 is the deployed fill. Live numbers: content decode 13.05 t/s
+(acceptance 0.58, mean len 2.16), reasoning phase 7.24 t/s
+(acceptance 0.74, mean len 2.48), two concurrent full generations both
+delivered, prompt cache reusing 42/64 tokens across requests. All
+three Prometheus fleet targets up.
+
 ## Rollback

 The lucebox DS4.1 material is fully retained on the host: image