docs + guard: single 262k slot state, split-limits analysis, cache incident

- spec matrix + PP doc: production state = 1 slot @ 262144, split 0.53
  (34 blocks), KV 8192/8192, 3090 at 23460/24576. New 'Split limits
  (measured 2026-08-17)' section: 0.50/0.54/0.55 boundary data, the
  KV-per-device rounding rule (round(f x 66)), and why 35 blocks is
  unreachable (9-FA flip at f~0.538).
- PP doc: cache incident writeup - full Dankserver disk -> 0-byte cache
  entries -> corrupt weights -> degenerate output; fix + signature.
- deployment.md + AGENTS.md + index.md: numbers brought current.
- scripts/prune-rpc-cache.sh: cron guard capping the worker cache at
  40 GiB (oldest first) to prevent the disk-full recurrence.
This commit is contained in:
Frank committed 2026-08-17 01:24:06 -07:00
1 parent c85ff6c6fd
commit aa1941d592
6 files changed
+132 -45

No files matched your search

+19 -15
View File
@@ -5,32 +5,36 @@ tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service
(`frank-default-large`), now a pipeline-parallel head with a Dankserver
worker.
## Production state (2026-08-16: PP split + MTP)
## Production state (2026-08-17: single 262k slot, split 0.53)
Current production, verified against the live process and journal
(2026-08-16, service start 15:03). Topology detail:
(2026-08-17, load 01:21, output smoke test). Topology detail:
[[Pipeline Parallelism/Qwen3.8 PP Deployment]].
- Devices: `RPC0` (Dankserver RTX 3090) + `ROCm0` (Strix iGPU). The head
passes no `--device` flag. Default selection puts RPC first and picks
ROCm for the iGPU (HIP registers before Vulkan). The Vulkan0 side stays
an option via `HEAD_BIN=.../build-vulkan-rpc`.
- Split: `--tensor-split 0.40,0.60` — 40% of the layers on the 3090
(its maximum under the KV budget), 60% on the iGPU.
- Context: pool 401400, 2 slots. The journal reports
`n_ctx_slot = 200704`. The pi `models.json` declares 200000
(conservative).
- Spec: **MTP re-enabled** — `--spec-type draft-mtp --spec-draft-n-max 4`
(bundled head) with the `--spec-default` ngram-mod fallback (48/64/24).
Journal acceptance over one production window: MTP mean len 3.00
(per-position 0.730/0.541/0.411/0.323, 58934 drafts); ngram-mod as the
rare fallback (423 drafts in the same window).
- KV: `f16` K / `f16` V on both devices (15680 MiB ROCm0 + 9408 MiB RPC0
at load).
- Split: `--tensor-split 0.53,0.47` — 34 of 65 blocks (0-33) on the
3090, about 52% of the layer weight. This is the maximum provably-safe
value (measured 2026-08-17): 0.54 jumped the KV to 9 full-attention
layers -> cudaMalloc OOM. The KV-per-device count rounds the fraction
over 66 layers, independent of the weight boundary; the 9-FA flip sits
at frac ~= 0.538. Boundary analysis: the PP deployment doc.
- Context: pool 262144, 1 slot. `n_ctx_slot = 262144` exactly. The pi
`models.json` declares 262144 (contextWindow = maxTokens, house
convention).
- Spec: **MTP** — `--spec-type draft-mtp --spec-draft-n-max 4` (bundled
head) with the `--spec-default` ngram-mod fallback (48/64/24). Smoke
acceptance 0.83 (50/60, 2026-08-17); full production-window stats in
the superseded section below.
- KV: `f16` K / `f16` V. At the 262144 pool: 8192 MiB per device
(8 full-attention layers each; FA at blocks 3,7,...,63).
- Worker steady state: 23460/24576 MiB used on the 3090, ~667 MiB free.
- Batch 8192 / ubatch 256, `--direct-io`, `--fit off`,
`--gpu-layers all`.
## Result (superseded 2026-08-16: standalone --spec-default)
## Result (superseded 2026-08-17: 2 slots x 200704, split 0.40)
Superseded for topology on 2026-08-16, when the PP split became
production. The ngram-mod preset survives as the fallback layer under