frank-primary: four full windows + Restart=always (runtime survivability)

Five 262144-token contexts passed every boot gate (KV 1,350,182 / 5.15x) but
left 0 bytes free at runtime: a real client request (995-token prompt,
max_tokens=261149, sampling params straight from models.yml) killed Worker_TP1
with torch.OutOfMemoryError 'Tried to allocate 50.00 MiB ... 0 bytes free'.
Boot is not proof of runtime survivability.

Four slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93 GiB per
rank for activations/graph capture.

Unit: Restart=always instead of on-failure — a runtime worker fault makes
vLLM's API server exit clean, so ExecStart returned 0 and on-failure never
fired, leaving the endpoint down until a human noticed. Also
StartLimitIntervalSec=0 and OOMScoreAdjust=-500.

serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md

@@ -278,8 +278,8 @@ is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
 ### The deployed shape (2026-10-01: 5 x 262144, fp8, no spec)
 
 ```
-FRANK_PRIMARY_MAXLEN=262144    x SEQS 5 (1,310,720 tokens of contract)
-FRANK_PRIMARY_KV_BYTES=10388766720  PER RANK (fp8, pool scales with windows)
+FRANK_PRIMARY_MAXLEN=262144    x SEQS 4 (1,048,576 tokens of contract)
+FRANK_PRIMARY_KV_BYTES=8311013376   PER RANK (fp8, pool scales with windows)
 FRANK_PRIMARY_KV_DTYPE=fp8    --kv-cache-dtype fp8 (K and V together)
 FRANK_PRIMARY_MNBT=1024       --max-num-batched-tokens 1024
 R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64  (container env)
@@ -297,10 +297,14 @@ memory environment). The night searched two concurrency contracts:
 | 3 × 237568, bf16 | PASS — KV 718,606 tokens |
 | **3 × 241664, bf16** | **PASS — KV 732,359 tokens, 3.03x** (max possible at 3 contexts) |
 | 6 × 241664, fp8 | **OOM** — tried 10.70 GiB against 9.18 GiB free: six full-length contexts do NOT fit |
-| **5 × 262144, fp8** | **PASS on the first rung — KV 1,350,182 tokens, 5.15x; DEPLOYED** (39,462-token margin over the 1,310,720-token contract) |
+| 5 × 262144, fp8 | PASS on the first rung — KV 1,350,182 tokens, 5.15x |
+| **4 × 262144, fp8** | **DEPLOYED.** Five slots booted but left 0 bytes free at runtime: a real client request (995-token prompt, `max_tokens=261149`) killed the worker (`Tried to allocate 50.00 MiB ... 0 bytes free`). Dropping one slot returns ~1.93 GiB per rank for activations and graph capture |
 
-The final shape is Frank's call: **five contexts of the model's full native
-262,144 window** beat six shortened ones. fp8 KV is what buys the capacity —
+Boot is NOT proof of runtime survivability: the five-slot shape passed every
+boot gate and still died on the first long-generation request. After any
+knife-edge shape change, send a request with a large `max_tokens` before
+calling it done. The deployed shape is **four contexts of the model's full
+native 262,144 window**. fp8 KV is what buys the capacity —
 7,926 B/token against bf16's ~13,900-14,047 — at the cost of bf16's ~4 %
 decode edge. Six full-length contexts cannot fit at any KV dtype: the 6-seq
 graph/activation footprint leaves only 9.18 GiB for the pool, and 6 × 262144

serverAppConfigs-docs/LLM/index.md

@@ -4,9 +4,9 @@ Documentation for the LLM inference fleet on the two hosts (2026-10-01):
 **Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
 digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
 orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
-on :7770, TP=2 with automatic tiered-expert offload, **5 seqs x 262144 ctx
-(the model's full native window)**, fp8 KV (6 full-length contexts do not
-fit), no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
+on :7770, TP=2 with automatic tiered-expert offload, **4 seqs x 262144 ctx
+(the model's full native window)**, fp8 KV (5 boot but leave no runtime
+headroom; 6 do not fit), no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
 reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
 measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
 the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
@@ -32,7 +32,7 @@ gfx1201, 32 GiB each]
         B2[tcclaviger RDNA4 vLLM fork, digest-pinned
 podman run, frank-primary, port 7770]
         C[Qwen3.8-Flash-Next FP8 bf16 no-spec
-TP=2, expert offload automatic, 5 x 262144 ctx,
+TP=2, expert offload automatic, 4 x 262144 ctx,
 fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers]
     end
 

serverAppConfigs-docs/log.md

@@ -722,8 +722,22 @@
   <= 262,144) and
   `preserve_thinking` moved from a bare top-level body field to
   `chat_template_kwargs` (the form vLLM actually reads).
-- Production bring-up: the winner (241,664) boots and serves on :7770 —
-  engine log verified (KV 732,359 tokens, 3.03x concurrency, MNBT 1024,
-  gather 32x64). Rollback target for the old fp8+MTP4 shape: git tag
-  `frank-primary-fp8-mtp-20261001`; the 237,568 fallback is documented in
-  the runbook.
+- **Runtime-survivability correction: five slots -> FOUR.** 5 x 262144 fp8
+  passed every boot gate (KV 1,350,182 tokens / 5.15x) but left 0 bytes free
+  at runtime; a real client request (995-token prompt, `max_tokens=261149`,
+  sampling params straight from models.yml) killed Worker_TP1 with
+  `torch.OutOfMemoryError: Tried to allocate 50.00 MiB ... 0 bytes free` and
+  the engine shut down. Boot is not proof of runtime survivability — after a
+  knife-edge shape change, send a large-`max_tokens` request before calling it
+  done. FOUR slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93
+  GiB per rank for activations/graph capture. Frank's call 2026-10-01.
+- `/scripts/frank-primary.service` — **Restart=always, not on-failure** (plus
+  `StartLimitIntervalSec=0` and `OOMScoreAdjust=-500`). A runtime worker fault
+  makes vLLM's API server exit CLEAN, so ExecStart returned 0, systemd logged
+  "Deactivated successfully", and on-failure NEVER fired: the endpoint stayed
+  down until a human noticed. An explicit `systemctl stop` still wins, so
+  bench windows and deploys are unaffected.
+- Production bring-up: the shape boots and serves on :7770 — engine log
+  verified (KV cache size, concurrency, MNBT 1024, gather 32x64) plus live
+  contract probes (reasoning split, tool_calls). Rollback target for the
+  2026-09-29 fp8+MTP4 shape: git tag `frank-primary-fp8-mtp-20261001`.