docs + guard: single 262k slot state, split-limits analysis, cache incident

- spec matrix + PP doc: production state = 1 slot @ 262144, split 0.53
  (34 blocks), KV 8192/8192, 3090 at 23460/24576. New 'Split limits
  (measured 2026-08-17)' section: 0.50/0.54/0.55 boundary data, the
  KV-per-device rounding rule (round(f x 66)), and why 35 blocks is
  unreachable (9-FA flip at f~0.538).
- PP doc: cache incident writeup - full Dankserver disk -> 0-byte cache
  entries -> corrupt weights -> degenerate output; fix + signature.
- deployment.md + AGENTS.md + index.md: numbers brought current.
- scripts/prune-rpc-cache.sh: cron guard capping the worker cache at
  40 GiB (oldest first) to prevent the disk-full recurrence.

AGENTS.md

@@ -5,9 +5,10 @@
 This project manages deployment scripts for the LLM inference fleet.
 Dankclaw runs a llama.cpp pipeline-parallel head for Qwen3.8-27B (unsloth
 UD-Q8_K_XL): MTP draft (bundled head, `--spec-draft-n-max 4`) over the
-`--spec-default` ngram-mod fallback, 2 parallel slots at 200704 context
+`--spec-default` ngram-mod fallback, 1 slot at 262144 context (pool
-(pool 401400), tensors split 0.40/0.60 to the Dankserver RTX 3090
+262144), tensors split 0.53/0.47 (34 of 65 blocks) to the Dankserver RTX
-`ggml-rpc` worker, head GPU side on ROCm0 (no `--device` flag), port 7770.
+3090 `ggml-rpc` worker, head GPU side on ROCm0 (no `--device` flag), port
+7770.
 The Dankclaw NPU endpoint (FastFlowLM, `frank-default-small`, port 52625)
 serves Qwen3.6-35B-A3B as registry tag `qwen3.6-moe:35b-a3b` at 262144 ctx
 (swapped in 2026-08-16; the old llama.cpp 35B service on Dankserver is

docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md

@@ -5,32 +5,69 @@ Split Qwen3.8-27B UD-Q8_K_XL across the Strix Halo iGPU (head) and the RTX
 morning-of runbook. Research background:
 [Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md).
 
-## Production state (promoted 2026-08-16)
+## Production state (2026-08-17: single 262k slot, split 0.53)
 
-This split is the production `frank-default-large` service (renamed from
+This split is the production `frank-default-large` service. Current
-`frank-pp-head`, commit `48142f5`). Current state, verified against the
+state, verified against the live journal and output smoke test
-live journal (2026-08-16, 15:03 load):
+(2026-08-17, load 01:21):
 
-- Split `0.40,0.60` (not the test-day `1,1`): 40% of the layers on the
+- Split `0.53,0.47` = 34 of 65 blocks (0-33) on the 3090 — the maximum
-  3090 — its maximum under the 2x200700 f16 KV budget — 60% on the iGPU.
+  provably-safe value; see "Split limits" below for the boundary
-- Pool `401400`, `PARALLEL=2`, live `n_ctx_slot = 200704`.
+  analysis and the failed 0.54/0.55 attempts.
+- Pool `262144`, `PARALLEL=1`, live `n_ctx_slot = 262144`.
 - MTP enabled: `--spec-type draft-mtp --spec-draft-n-max 4` + ngram-mod
-  fallback (48/64/24). Journal acceptance: MTP mean len 3.00
+  fallback (48/64/24). Smoke acceptance 0.83 (50/60, 2026-08-17).
-  (per-position 0.730/0.541/0.411/0.323).
 - Head GPU side: **ROCm0** (no `--device` flag; HIP registers before
-  Vulkan). KV at load: 15680 MiB ROCm0 + 9408 MiB RPC0.
+  Vulkan). KV at load: 8192 MiB ROCm0 + 8192 MiB RPC0 (8 FA each).
+- Worker steady state: 23460/24576 MiB used on the 3090, ~667 MiB free.
 - Unit `frank-default-large.service`, launched through
   `serve-pp-head.sh` + `frank-default-large.env`.
 
-Open questions from the test day, resolved:
+## Split limits (measured 2026-08-17)
 
-1. MTP (in the last layer group, iGPU side) drafts cleanly across the RPC
+The 0.40 split (2026-08-16, 2x200700 pool) was KV-bound at 26 blocks.
-   boundary — see the acceptance figures above.
+With the pool shrunk to 262144, the KV per full-attention layer drops to
-2. Two parallel slots hold under production load: 150k-token prompts
+exactly 1024 MiB, freeing headroom. Measurements:
-   prefill and generate, slots turn over every few minutes, no RPC
+
-   faults observed.
+- 0.50,0.50 -> 32 blocks (0-31), 8 FA on RPC0, KV 8192 MiB, 1477 MiB
-3. Long prompts (~160k tokens) prefill on the ROCm0 side with no
+  free. Healthy (2026-08-17 00:29 load).
-   RPC-boundary anomaly in production traffic.
+- 0.55,0.45 -> 36 blocks (0-35): the 9th FA block, KV 9216 MiB ->
+  cudaMalloc OOM by ~740 MiB.
+- 0.54,0.46 -> weights 35 blocks (0-34) but KV still 9216 MiB -> OOM.
+  The KV-per-device count rounds the split fraction over 66 layers
+  (round(f x 66)), independent of the weight boundary; the 9-FA flip
+  sits at f ~= 0.538, just below the weight threshold for block 35, so
+  the 35-block config is unreachable.
+- 0.53,0.47 -> 34 blocks (0-33), 8 FA, KV 8192 MiB. Safe under every
+  rounding rule (round/floor/ceil of 0.53 x 66 = 34.98). Production.
+
+GGUF block scan (exact sizes): FA blocks at 3,7,...,63 (409.6 MB each),
+linear blocks 431.3 MB; total layer weight 27720 MB. The split pool
+includes the MTP head for weights and token_embd for KV — the two
+boundaries diverge, which is why 0.54 split them apart.
+
+## Cache incident (2026-08-17): garbage output from a full worker disk
+
+Symptom: after a split change, the service loaded "healthy" but every
+completion degenerated to a repeated junk token (e.g. 48 x "/", empty
+content, degenerate logits). Root cause chain:
+
+1. The worker RPC tensor cache keys per session. Each model load writes
+   a fresh full copy (~15 GB); over ~5 sessions of two models it grew to
+   4434 files / 96 GB.
+2. The Dankserver root disk filled to 100%. Cache writes started
+   failing, leaving 0-byte entries. A subsequent reload hit corrupt
+   cache entries and the worker served garbage weights for the
+   newly-needed layers without validation.
+3. Fix: move the cache to .AGENTTRASH (then empty it), restart the
+   worker, free the disk, reload the head (full re-stream ~2.5 min).
+   A clean cache and a healthy disk produced correct output immediately
+   (acceptance 0.89 on the first smoke test).
+
+Guard: `scripts/prune-rpc-cache.sh` on Dankserver (cron, 40 GiB cap).
+Never let the cache directory grow unchecked; a full disk silently
+corrupts it. Signature to recognize: healthy load + degenerate repeated-
+token output.
 
 ## Topology
 

docs/deployment.md

@@ -143,22 +143,31 @@ curl -sf http://127.0.0.1:7770/health
 The active service on Dankclaw is the llama.cpp **pipeline-parallel
 head**. It serves the unsloth Qwen3.8-27B `UD-Q8_K_XL` model under the
 alias `frank-default-large` on port 7770. The head splits the tensors
-`0.40,0.60` to the `ggml-rpc` worker on Dankserver (RTX 3090, port 50052).
+`0.53,0.47` to the `ggml-rpc` worker on Dankserver (RTX 3090, port 50052):
+34 of 65 blocks (0-33) on the 3090 — the maximum provably-safe split
+(measured 2026-08-17; see the PP doc for the boundary analysis).
 Speculative decoding runs **MTP** (bundled head, `--spec-draft-n-max 4`)
 with the `--spec-default` ngram-mod fallback (48/64/24). The head GPU side
 runs **ROCm0** (no `--device` flag; default device selection puts RPC
-first and picks ROCm for the iGPU). **2 parallel slots** at 200704 context
+first and picks ROCm for the iGPU). **1 slot** at 262144 context (pool
-each (pool 401400), KV cache `f16` K / `f16` V, full layer offload
+262144, `n_ctx_slot = 262144`), KV cache `f16` K / `f16` V (8192 MiB per
-(`--gpu-layers all`, `--fit off`), batch 8192 / ubatch 256, `--direct-io`.
+device), full layer offload (`--gpu-layers all`, `--fit off`), batch 8192
-The unit file is `scripts/frank-default-large.service`. It launches through
+/ ubatch 256, `--direct-io`. The unit file is
+`scripts/frank-default-large.service`. It launches through
 `scripts/serve-pp-head.sh`, which reads tunables from
 `scripts/frank-default-large.env`.
 
+> Note: 2026-08-17 — single-slot config: `PARALLEL=1`, pool 262144
+> (was 2 slots x 200700 at 401400), split 0.53 (was 0.40). The 3090
+> landed at 23460/24576 MiB, ~667 MiB free. Same-day incident: a full
+> Dankserver disk corrupted the worker RPC tensor cache -> degenerate
+> output; fixed by clearing the cache + freeing the disk. Guard:
+> `scripts/prune-rpc-cache.sh` (cron, 40 GiB cap).
+
 > Note: 2026-08-16 — the pipeline-parallel split was promoted to
 > production as `frank-default-large` (renamed from `frank-pp-head`);
 > the standalone single-node service retired. MTP was re-enabled at
-> `n_max 4` over the ngram-mod fallback. The pool shrank from 524288 to
+> `n_max 4` over the ngram-mod fallback. See
-> 401400 (2 slots x 200700) to fit the split KV budget. See
 > [[Pipeline Parallelism/Qwen3.8 PP Deployment]] for the runbook.
 
 > Note: 2026-08-15 — the spec went DSpark (crashed under two concurrent 16k

docs/index.md

@@ -58,7 +58,7 @@ frank-default-small, port 52625]
 - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090
 - [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU
 - [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug)
-- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec history + current production state: PP head, MTP (n_max 4) over the ngram-mod fallback, 2 slots @ 200704 (2026-08-16)
+- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec history + current production state: PP head, MTP (n_max 4) over the ngram-mod fallback, 1 slot @ 262144, split 0.53 (2026-08-17)
 - [[qwen3.8-strix-optimizations]] — Qwen3.8-27B Strix speed research: ranked levers that keep UD-Q8_K_XL (PP to 3090, MTP tuning, same-bit Q8_0 A/B, prefill path, OS), quant landscape, 2026-08-16 test matrix
 - [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state
 - [[dankserver-vram-growth]] — VRAM/RAM growth investigation behind the recurring cuBLAS crash on frank-default-small
@@ -79,7 +79,7 @@ frank-default-small, port 52625]
   - [[Pipeline Parallelism/Mixed-Vendor Experiments]] — NVIDIA + Strix Halo real-world runs
   - [[Pipeline Parallelism/Expected Performance]] — estimated tok/s over 1/2.5/10 GbE
   - [[Pipeline Parallelism/Sample Script Set]] — educational two-node Qwen3.6-27B Q8 split
-  - [[Pipeline Parallelism/Qwen3.8 PP Deployment]] — the production deployment (promoted 2026-08-16) + runbook: Qwen3.8-27B head (Strix) / worker (RTX 3090) over RPC, b10433, scripts in scripts/
+  - [[Pipeline Parallelism/Qwen3.8 PP Deployment]] — the production deployment + runbook: Qwen3.8-27B head (Strix) / worker (RTX 3090) over RPC, b10433, scripts in scripts/; split limits + cache incident measured 2026-08-17
 
 ## Sibling projects
 

docs/qwen3.8-spec-matrix.md

@@ -5,32 +5,36 @@ tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service
 (`frank-default-large`), now a pipeline-parallel head with a Dankserver
 worker.
 
-## Production state (2026-08-16: PP split + MTP)
+## Production state (2026-08-17: single 262k slot, split 0.53)
 
 Current production, verified against the live process and journal
-(2026-08-16, service start 15:03). Topology detail:
+(2026-08-17, load 01:21, output smoke test). Topology detail:
 [[Pipeline Parallelism/Qwen3.8 PP Deployment]].
 
 - Devices: `RPC0` (Dankserver RTX 3090) + `ROCm0` (Strix iGPU). The head
   passes no `--device` flag. Default selection puts RPC first and picks
   ROCm for the iGPU (HIP registers before Vulkan). The Vulkan0 side stays
   an option via `HEAD_BIN=.../build-vulkan-rpc`.
-- Split: `--tensor-split 0.40,0.60` — 40% of the layers on the 3090
+- Split: `--tensor-split 0.53,0.47` — 34 of 65 blocks (0-33) on the
-  (its maximum under the KV budget), 60% on the iGPU.
+  3090, about 52% of the layer weight. This is the maximum provably-safe
-- Context: pool 401400, 2 slots. The journal reports
+  value (measured 2026-08-17): 0.54 jumped the KV to 9 full-attention
-  `n_ctx_slot = 200704`. The pi `models.json` declares 200000
+  layers -> cudaMalloc OOM. The KV-per-device count rounds the fraction
-  (conservative).
+  over 66 layers, independent of the weight boundary; the 9-FA flip sits
-- Spec: **MTP re-enabled** — `--spec-type draft-mtp --spec-draft-n-max 4`
+  at frac ~= 0.538. Boundary analysis: the PP deployment doc.
-  (bundled head) with the `--spec-default` ngram-mod fallback (48/64/24).
+- Context: pool 262144, 1 slot. `n_ctx_slot = 262144` exactly. The pi
-  Journal acceptance over one production window: MTP mean len 3.00
+  `models.json` declares 262144 (contextWindow = maxTokens, house
-  (per-position 0.730/0.541/0.411/0.323, 58934 drafts); ngram-mod as the
+  convention).
-  rare fallback (423 drafts in the same window).
+- Spec: **MTP** — `--spec-type draft-mtp --spec-draft-n-max 4` (bundled
-- KV: `f16` K / `f16` V on both devices (15680 MiB ROCm0 + 9408 MiB RPC0
+  head) with the `--spec-default` ngram-mod fallback (48/64/24). Smoke
-  at load).
+  acceptance 0.83 (50/60, 2026-08-17); full production-window stats in
+  the superseded section below.
+- KV: `f16` K / `f16` V. At the 262144 pool: 8192 MiB per device
+  (8 full-attention layers each; FA at blocks 3,7,...,63).
+- Worker steady state: 23460/24576 MiB used on the 3090, ~667 MiB free.
 - Batch 8192 / ubatch 256, `--direct-io`, `--fit off`,
   `--gpu-layers all`.
 
-## Result (superseded 2026-08-16: standalone --spec-default)
+## Result (superseded 2026-08-17: 2 slots x 200704, split 0.40)
 
 Superseded for topology on 2026-08-16, when the PP split became
 production. The ngram-mod preset survives as the fallback layer under

scripts/prune-rpc-cache.sh

@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+# Bound the ggml-rpc worker tensor cache.
+#
+# The worker (frank-pp-worker on Dankserver) keys cache entries per
+# session, so each model load writes a fresh full copy (~15 GB). Left
+# unbounded, the directory filled the root disk to 100% on 2026-08-17;
+# failed cache writes left 0-byte entries, and a later reload served
+# corrupt weights -> degenerate model output (see the cache incident in
+# docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md).
+#
+# This script prunes the cache to a byte cap, oldest files first. Cache
+# entries are derived data: safe to delete. Run from cron on Dankserver.
+set -euo pipefail
+
+CACHE_DIR=/home/dank/.cache/llama.cpp/rpc
+MAX_BYTES=$((40 * 1024 * 1024 * 1024))   # 40 GiB
+
+if [[ ! -d "$CACHE_DIR" ]]; then
+    exit 0
+fi
+
+total=$(du -sb "$CACHE_DIR" | cut -f1)
+if (( total <= MAX_BYTES )); then
+    exit 0
+fi
+
+while (( total > MAX_BYTES )); do
+    oldest=$(ls -1t "$CACHE_DIR" | tail -1)
+    if [[ -z "$oldest" ]]; then
+        break
+    fi
+    rm -f -- "$CACHE_DIR/$oldest"
+    total=$(du -sb "$CACHE_DIR" | cut -f1)
+done
+
+echo "prune-rpc-cache: pruned to $(du -sh "$CACHE_DIR" | cut -f1)"