dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary runbook: 5/3 block fill is the deployed shape

Browse Source
This commit is contained in:
Frank
2026-09-26 21:40:25 -07:00
parent d317d6b246
commit d67768e1a5
1 changed files with 12 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md
+12 -7
View File
@@ -166,7 +166,7 @@ both cards: main 9,408 MiB + indexer 1,536 MiB per card, draft
Hot expert blocks stay GPU-resident. `-ot` granularity is the whole
per-block expert tensor (512 experts share one tensor), so "critical
experts" = blocks co-located on the card that holds their layers:
blocks 0-3 on ROCm0, 45-47 on ROCm1 (measured 2.42/2.49 GiB per
blocks 0-4 on ROCm0, 45-47 on ROCm1 (measured 2.42/2.49 GiB per
block). The placement law that makes it work: b11030 wants ONE
comma-joined `--override-tensor` (repeats log DEPRECATED), and the
loader applies the FIRST matching pattern per tensor — the
@@ -174,15 +174,20 @@ block-specific patterns MUST precede the broad `exps=CPU` sweep, or
the sweep silently swallows them (verified live: broad-first left
every expert on CPU with no warning anywhere in the log).
Measured headroom (checkpoint floor: 3-4 GB per card): card0 5.02 GiB
free at boot, ~4.9 GB under load; card1 3.31 GiB, ~3.4 GB under load
(the vision graph costs ~0.43 GiB on card0 at first image). Blocks are
atomic — a 5th on card0 lands at 2.27 GB free, under the floor — so
4/3 is the deployed fill. Live numbers: content decode 13.05 t/s
Measured headroom, deployed fill **5/3** (blocks 0-4 on ROCm0, 45-47
on ROCm1 — the operator directed the fill past the 3-4 GB checkpoint
floor the same day, "shift a few more experts"): card0 2.53 GiB free
at boot, 2.23 GiB under load with the vision graph exercised; card1
3.31 GiB at boot, 3.15 GiB under load. Card1 cannot take another
block — 3.31 − 2.49 = 0.82 GiB, and the vision graph (~0.43 GiB) plus
boot drift alone exceed that — so 5/3 is the physical limit at the
4×262144 / V-q8_0 shape. Earlier 4/3 state for reference: card0 5.02
GiB boot free. Live numbers (4/3 era): content decode 13.05 t/s
(acceptance 0.58, mean len 2.16), reasoning phase 7.24 t/s
(acceptance 0.74, mean len 2.48), two concurrent full generations both
delivered, prompt cache reusing 42/64 tokens across requests. All
three Prometheus fleet targets up.
three Prometheus fleet targets up; the 5/3 deploy re-verified chat,
vision and the target on the same battery class.
## Rollback