docs + guard: single 262k slot state, split-limits analysis, cache incident
- spec matrix + PP doc: production state = 1 slot @ 262144, split 0.53 (34 blocks), KV 8192/8192, 3090 at 23460/24576. New 'Split limits (measured 2026-08-17)' section: 0.50/0.54/0.55 boundary data, the KV-per-device rounding rule (round(f x 66)), and why 35 blocks is unreachable (9-FA flip at f~0.538). - PP doc: cache incident writeup - full Dankserver disk -> 0-byte cache entries -> corrupt weights -> degenerate output; fix + signature. - deployment.md + AGENTS.md + index.md: numbers brought current. - scripts/prune-rpc-cache.sh: cron guard capping the worker cache at 40 GiB (oldest first) to prevent the disk-full recurrence.
This commit is contained in:
Frank committed
1 parent
c85ff6c6fd
commit
aa1941d592
6 files changed
+132
-45
No files matched your search
@@ -5,32 +5,69 @@ Split Qwen3.8-27B UD-Q8_K_XL across the Strix Halo iGPU (head) and the RTX
|
||||
morning-of runbook. Research background:
|
||||
[Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md).
|
||||
|
||||
## Production state (promoted 2026-08-16)
|
||||
## Production state (2026-08-17: single 262k slot, split 0.53)
|
||||
|
||||
This split is the production `frank-default-large` service (renamed from
|
||||
`frank-pp-head`, commit `48142f5`). Current state, verified against the
|
||||
live journal (2026-08-16, 15:03 load):
|
||||
This split is the production `frank-default-large` service. Current
|
||||
state, verified against the live journal and output smoke test
|
||||
(2026-08-17, load 01:21):
|
||||
|
||||
- Split `0.40,0.60` (not the test-day `1,1`): 40% of the layers on the
|
||||
3090 — its maximum under the 2x200700 f16 KV budget — 60% on the iGPU.
|
||||
- Pool `401400`, `PARALLEL=2`, live `n_ctx_slot = 200704`.
|
||||
- Split `0.53,0.47` = 34 of 65 blocks (0-33) on the 3090 — the maximum
|
||||
provably-safe value; see "Split limits" below for the boundary
|
||||
analysis and the failed 0.54/0.55 attempts.
|
||||
- Pool `262144`, `PARALLEL=1`, live `n_ctx_slot = 262144`.
|
||||
- MTP enabled: `--spec-type draft-mtp --spec-draft-n-max 4` + ngram-mod
|
||||
fallback (48/64/24). Journal acceptance: MTP mean len 3.00
|
||||
(per-position 0.730/0.541/0.411/0.323).
|
||||
fallback (48/64/24). Smoke acceptance 0.83 (50/60, 2026-08-17).
|
||||
- Head GPU side: **ROCm0** (no `--device` flag; HIP registers before
|
||||
Vulkan). KV at load: 15680 MiB ROCm0 + 9408 MiB RPC0.
|
||||
Vulkan). KV at load: 8192 MiB ROCm0 + 8192 MiB RPC0 (8 FA each).
|
||||
- Worker steady state: 23460/24576 MiB used on the 3090, ~667 MiB free.
|
||||
- Unit `frank-default-large.service`, launched through
|
||||
`serve-pp-head.sh` + `frank-default-large.env`.
|
||||
|
||||
Open questions from the test day, resolved:
|
||||
## Split limits (measured 2026-08-17)
|
||||
|
||||
1. MTP (in the last layer group, iGPU side) drafts cleanly across the RPC
|
||||
boundary — see the acceptance figures above.
|
||||
2. Two parallel slots hold under production load: 150k-token prompts
|
||||
prefill and generate, slots turn over every few minutes, no RPC
|
||||
faults observed.
|
||||
3. Long prompts (~160k tokens) prefill on the ROCm0 side with no
|
||||
RPC-boundary anomaly in production traffic.
|
||||
The 0.40 split (2026-08-16, 2x200700 pool) was KV-bound at 26 blocks.
|
||||
With the pool shrunk to 262144, the KV per full-attention layer drops to
|
||||
exactly 1024 MiB, freeing headroom. Measurements:
|
||||
|
||||
- 0.50,0.50 -> 32 blocks (0-31), 8 FA on RPC0, KV 8192 MiB, 1477 MiB
|
||||
free. Healthy (2026-08-17 00:29 load).
|
||||
- 0.55,0.45 -> 36 blocks (0-35): the 9th FA block, KV 9216 MiB ->
|
||||
cudaMalloc OOM by ~740 MiB.
|
||||
- 0.54,0.46 -> weights 35 blocks (0-34) but KV still 9216 MiB -> OOM.
|
||||
The KV-per-device count rounds the split fraction over 66 layers
|
||||
(round(f x 66)), independent of the weight boundary; the 9-FA flip
|
||||
sits at f ~= 0.538, just below the weight threshold for block 35, so
|
||||
the 35-block config is unreachable.
|
||||
- 0.53,0.47 -> 34 blocks (0-33), 8 FA, KV 8192 MiB. Safe under every
|
||||
rounding rule (round/floor/ceil of 0.53 x 66 = 34.98). Production.
|
||||
|
||||
GGUF block scan (exact sizes): FA blocks at 3,7,...,63 (409.6 MB each),
|
||||
linear blocks 431.3 MB; total layer weight 27720 MB. The split pool
|
||||
includes the MTP head for weights and token_embd for KV — the two
|
||||
boundaries diverge, which is why 0.54 split them apart.
|
||||
|
||||
## Cache incident (2026-08-17): garbage output from a full worker disk
|
||||
|
||||
Symptom: after a split change, the service loaded "healthy" but every
|
||||
completion degenerated to a repeated junk token (e.g. 48 x "/", empty
|
||||
content, degenerate logits). Root cause chain:
|
||||
|
||||
1. The worker RPC tensor cache keys per session. Each model load writes
|
||||
a fresh full copy (~15 GB); over ~5 sessions of two models it grew to
|
||||
4434 files / 96 GB.
|
||||
2. The Dankserver root disk filled to 100%. Cache writes started
|
||||
failing, leaving 0-byte entries. A subsequent reload hit corrupt
|
||||
cache entries and the worker served garbage weights for the
|
||||
newly-needed layers without validation.
|
||||
3. Fix: move the cache to .AGENTTRASH (then empty it), restart the
|
||||
worker, free the disk, reload the head (full re-stream ~2.5 min).
|
||||
A clean cache and a healthy disk produced correct output immediately
|
||||
(acceptance 0.89 on the first smoke test).
|
||||
|
||||
Guard: `scripts/prune-rpc-cache.sh` on Dankserver (cron, 40 GiB cap).
|
||||
Never let the cache directory grow unchecked; a full disk silently
|
||||
corrupts it. Signature to recognize: healthy load + degenerate repeated-
|
||||
token output.
|
||||
|
||||
## Topology
|
||||
|
||||
|
||||
Reference in new issue
Block a user