frank-primary: 5 x full 262144 contexts at fp8 (final shape)

Frank's call after the night ladder: five contexts at the model's full native
262,144 window beat six shortened ones. Six full-length contexts do NOT fit at
any KV dtype (6 x 241664 fp8 asked 10.70 GiB against 9.18 GiB free — the
6-seq graph/activation footprint leaves too little for the pool); 5 x 262144
at fp8 PASSED on the first rung: KV_BYTES 10,388,766,720/rank, measured
'GPU KV cache size: 1,350,182 tokens, Maximum concurrency for 262,144 tokens
per request: 5.15x' (39,462 tokens of margin).

Also documented: the SEQS=3 bf16 max (241,664, KV 732,359 / 3.03x) measured
earlier the same night, and the root-cause read that the 7.2 GiB KV ceiling
law was the MTP4 workspace, not VRAM. Seam and harness (contextWindow 262144)
re-pinned.

serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md

@@ -275,39 +275,46 @@ Depth caveat: every number above is at a near-empty KV. Deep-context
 behavior (the gather win growing with depth, bf16's decode cost at depth)
 is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
 
-### The bf16 no-spec incumbent (production since 2026-10-01)
+### The deployed shape (2026-10-01: 5 x 262144, fp8, no spec)
 
 ```
-FRANK_PRIMARY_MAXLEN=241664    x SEQS 3 (724,992 tokens of contract)
-FRANK_PRIMARY_KV_BYTES=10184000000  PER RANK (bf16, pool scales with window)
-FRANK_PRIMARY_KV_DTYPE=auto   --kv-cache-dtype auto (bf16, unquantized)
+FRANK_PRIMARY_MAXLEN=262144    x SEQS 5 (1,310,720 tokens of contract)
+FRANK_PRIMARY_KV_BYTES=10388766720  PER RANK (fp8, pool scales with windows)
+FRANK_PRIMARY_KV_DTYPE=fp8    --kv-cache-dtype fp8 (K and V together)
 FRANK_PRIMARY_MNBT=1024       --max-num-batched-tokens 1024
 R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64  (container env)
 NO --speculative-config, NO --expert-prefetch
 ```
 
-Boot-probed 2026-10-01 (transient systemd units, full-log capture): 262144
-and 245760 both died at graph capture (`Tried to allocate 76.00 MiB ...
-0 bytes free` — the pool+profile+graph budget exceeds by a hair at util
-0.98); 237568 passed (KV 718,606 tokens) and the +4096 bonus attempt
-**241664 PASSED and is production: KV 732,359 tokens / Maximum concurrency
-for 241,664 tokens per request: 3.03x** — all three full windows fit with
-a 7,367-token margin. Decode measured at the 262k-era shape before the
-ladder: 39.38/38.79/36.42 tok/s (166/1087/4064 ctx).
-The old "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 WORKSPACE
-reservation, not a VRAM wall — no-spec bf16 takes ~9.5-10.2 GB/rank and
-boots clean. The failed 11.3 GB attempt OOMed at the engine's free-memory
-check (`RuntimeError: Worker failed with error 'CUDA out of memory'`,
-captured from the journald json stream — the fork's giant ERROR lines are
-split by journald, so grep the `-o json` output, not `-o cat`).
+Boot-probed 2026-10-01 (transient systemd units at the production uid +
+memlock, full-log capture; the rootless bench probe had masked the production
+memory environment). The night searched two concurrency contracts:
 
-Rollback to the fp8+MTP4 shape: git tag `frank-primary-fp8-mtp-20261001`
-(env trio + seam re-pinned there). The bf16 trade is deliberate: ~14%
-decode cost at 4064 ctx vs the fp8+gather arm (49.25 -> 36.42) bought
-clean cache + the largest window this box boots. The model's native
-262,144 exceeds what graph capture grants at util 0.98 here: the ladder
-measured 262144/245760 OOM and 241,664 as max possible (Frank's
-subtract-16k method, then one +4k bonus step).
+| rung | result |
+|---|---|
+| 3 × 262144, bf16, KV 11,047,010,304 | **OOM at graph capture** — `Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB already allocated by PyTorch |
+| 3 × 245760, bf16 | **OOM identically** |
+| 3 × 237568, bf16 | PASS — KV 718,606 tokens |
+| **3 × 241664, bf16** | **PASS — KV 732,359 tokens, 3.03x** (max possible at 3 contexts) |
+| 6 × 241664, fp8 | **OOM** — tried 10.70 GiB against 9.18 GiB free: six full-length contexts do NOT fit |
+| **5 × 262144, fp8** | **PASS on the first rung — KV 1,350,182 tokens, 5.15x; DEPLOYED** (39,462-token margin over the 1,310,720-token contract) |
+
+The final shape is Frank's call: **five contexts of the model's full native
+262,144 window** beat six shortened ones. fp8 KV is what buys the capacity —
+7,926 B/token against bf16's ~13,900-14,047 — at the cost of bf16's ~4 %
+decode edge. Six full-length contexts cannot fit at any KV dtype: the 6-seq
+graph/activation footprint leaves only 9.18 GiB for the pool, and 6 × 262144
+needs 12.47 GB even at fp8.
+
+The old "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 WORKSPACE
+reservation, not a VRAM wall. Failures are clean and diagnostic: the engine
+reports `torch.OutOfMemoryError: Tried to allocate X GiB. GPU N has ... of
+which Y GiB is free`, and the fork's giant ERROR lines are split by journald —
+read them via `journalctl -o json`, not `-o cat` (and `podman logs`, which is
+only available while the container lives).
+
+Rollback to the 2026-09-29 fp8+MTP4 shape: git tag
+`frank-primary-fp8-mtp-20261001` (env trio + seam re-pinned there).
 
 ## See Also
 

serverAppConfigs-docs/LLM/index.md

@@ -4,9 +4,9 @@ Documentation for the LLM inference fleet on the two hosts (2026-10-01):
 **Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
 digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
 orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
-on :7770, TP=2 with automatic tiered-expert offload, 241664 ctx x 3 seqs
-(max possible — 262144 does not fit at a 3-seq contract on this box),
-bf16 KV, no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
+on :7770, TP=2 with automatic tiered-expert offload, **5 seqs x 262144 ctx
+(the model's full native window)**, fp8 KV (6 full-length contexts do not
+fit), no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
 reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
 measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
 the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
@@ -32,8 +32,8 @@ gfx1201, 32 GiB each]
         B2[tcclaviger RDNA4 vLLM fork, digest-pinned
 podman run, frank-primary, port 7770]
         C[Qwen3.8-Flash-Next FP8 bf16 no-spec
-TP=2, expert offload automatic, 241664 ctx x 3,
-bf16 KV, MNBT 1024, gather 32x64, qwen3 parsers]
+TP=2, expert offload automatic, 5 x 262144 ctx,
+fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers]
     end
 
     subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]

serverAppConfigs-docs/log.md

@@ -709,8 +709,17 @@
   --tool-call-parser qwen3_xml`, `--default-chat-template-kwargs
   '{"enable_thinking": true, "preserve_thinking": true}'`. The seam's
   "NO tool dialect" test is flipped to a PRESENT test.
-- Harness `/home/blank/.omp/agent/models.yml` — contextWindow 262144 ->
-  241664 (compaction gap recomputed: 169,164 + 36,864 <= 241,664) and
+- **Final shape (Frank's call): 5 seqs x the full native 262144 window at
+  fp8 KV.** 6 x 241664 at fp8 died (`Tried to allocate 10.70 GiB`, 9.18 GiB
+  free) — six full-length contexts do not fit at any KV dtype — so the
+  contract moved to five FULL windows: 5 x 262144 fp8 PASSED on the first
+  rung, KV_BYTES 10,388,766,720/rank, measured **1,350,182 tokens /
+  5.15x concurrency** (39,462 tokens of margin). Seam re-pinned; the bf16
+  three-context shape (241,664, KV 732,359 / 3.03x, measured earlier the
+  same night) is the documented max at 3 contexts.
+- Harness `/home/blank/.omp/agent/models.yml` — contextWindow back to 262144
+  with the five-window fp8 shape (compaction gap 183,501 + 36,864 = 220,365
+  <= 262,144) and
   `preserve_thinking` moved from a bare top-level body field to
   `chat_template_kwargs` (the form vLLM actually reads).
 - Production bring-up: the winner (241,664) boots and serves on :7770 —