frank-primary: 2 x 200704 fp8, pool cut for the expert arena, MNBT 1024

The 2026-10-01 bf16 shape (2 x 262144, --expert-precise-memory)
crash-looped at graph capture — the arena is DEVICE-resident and that
shape had 0.13 GiB to spare. The plain bf16 2 x 262144 also did not stay
up: 0 bytes free at runtime, and a real client request (995-token prompt,
max_tokens=261149) killed the worker with .

The correction is a pool cut, not a downgrade. FP8 KV (7,701 B/token),
two 200704 windows (401,408 tokens -> 3.091e9 B floor, pinned 3.15e9),
MNBT 1024. The fork's expert planner sizes the arena AFTER the KV
override, so the ~2.95 GiB/rank reclaimed becomes arena: ~7,500 slots
vs 6,248. Two contexts at 200704 is a capacity contract, not a
throughput one (2-way concurrency summed BELOW parity at 4064 ctx), so
the slot buys the VRAM headroom long generations need.

Refs [[dankserver-qwen38-fp8-expert-offload-tuning]].

- scripts/frank-primary.env: MAXLEN 200704, SEQS 2, KV_BYTES 3.15e9,
  MNBT 1024, rewritten shape comment
- scripts/serve-frank-primary.sh: header comment updated
- tests/frank_primary_unit.bats: assertions updated, new seam pins the
  arena-floor pool law (the pool must move with a context change or the
  reduction buys nothing)
- docs + AGENTS.md: fleet topology and deployed shape corrected

scripts/frank-primary.env

@@ -59,43 +59,55 @@ FRANK_PRIMARY_PUBLISH=7770:7770
 # hoc experiment identity (both hold the GPUs / name registrations).
 FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
 
-# --- qualified engine shape (2026-10-01, RESTORED KNOWN-GOOD: fp8 3 windows) ---
-# bf16 at 2 x 262144 did NOT stay up: with and without --expert-precise-memory
-# the boot failed (`Engine core initialization failed`, NRestarts climbing).
-# The last shape measured serving was fp8 3 x 262144 / MNBT 2048 / KV 6.1e9
-# (KV 792,069 tokens, 3.02x, arena 6,248, ~3.6 GiB free) and is restored here
-# while the bf16 path is investigated in a window (see log.md 2026-10-01).
-# Candidate next test: bf16 with SEQS=3 to isolate the KV dtype from the
-# sequence count, and/or --enforce-eager to bypass the dynamo constraint path.
-# SEQS 2 x MAXLEN 262144 at BF16 KV (Frank's call, 2026-10-01): two contexts of
-# the model's FULL native window with UNQUANTIZED KV, plus
-# --expert-precise-memory to hand the arena the compile/graph-capture headroom.
-# Pool: bf16 costs 13,839 B/token (measured: 11,047,010,304 B -> 798,280 tokens
-# on the 3x241664 bf16 boot; decomposes as 12,276 attention + 1,563 mamba/side
-# caches, against fp8's 6,138 + 1,563 = 7,701). 2 x 262144 = 524,288 tokens
-# -> 7.256e9 B floor, pinned at 7,300,000,000 (~527,500 tokens, 3,206 margin).
-# Budget (per rank): 31.22 GiB total; non-expert weights 6.28, graphs 0.55,
-# non-torch ~1.45, pool 6.76 -> ~29.35 used, ~2.5 GiB free (measured basis:
-# the fp8 3-window shape used 28.27 and left 3.59).
-# PRECISE MEMORY: DROPPED 2026-10-01 after it crashed the boot in combination
-# with bf16. The flag selects the fitted 5.25 GiB overhead instead of the
-# 7.60 GiB default, moving the plan's `room` and therefore the arena
-# (6,248 -> ~7,278 slots). But the arena is DEVICE-resident, and the budget
-# for this shape does not have 2.35 GiB to spare:
-#   weights 6.28 + pool 6.80 (bf16 2 x 262144) + graphs 0.55 + arena 16.65
-#   + non-torch ~1.45 = 31.73 of 31.86 GiB  =>  ~0.13 GiB free at boot
-# measured as a CrashLoop: every attempt reached graph capture, the engine
-# exited cleanly, systemd restarted it (NRestarts=3 at ~6.7 min each).
-# With the default overhead the arena is 14.31 GiB and the same shape totals
-# ~29.4 GiB, leaving ~2.4 GiB for capture. Re-enable the flag only with a
-# pool/context reduction that pays for it.
-# MNBT 2048 (Frank): the overhead formula only adds bytes above nbt 4096, so
-# 2048 pays nothing, and the arena sweep measured the same ceiling <= 4096.
-FRANK_PRIMARY_MAXLEN=262144
-FRANK_PRIMARY_SEQS=3
-FRANK_PRIMARY_KV_BYTES=6100000000
+# --- qualified engine shape (2026-10-02, Frank's own correction) ---
+# 2026-10-01 left a shape that did not stay up: bf16 KV at 2 x 262144 with
+# --expert-precise-memory crash-looped at graph capture (the arena is
+# DEVICE-resident and the shape had 0.13 GiB to spare), and the plain bf16
+# 2 x 262144 boot booted but left 0 bytes free at runtime. This is the
+# correction: FP8 KV, TWO windows, each 200704 tokens, and the pool cut so
+# the expert arena gets the room back.
+#
+# LADDER (2026-10-01 night, subtract-16k method): 262144 and 245760 OOM at
+# graph capture; 237568 passed at 718,606 tokens; 241664 x 3 passed at
+# 732,359 tokens / 3.03x but is the shape this cutover retired. 200704 was
+# chosen to sit below the 241664 rung while leaving a full 2.95 GiB/rank of
+# pool to cut loose for the arena — the two contexts are a capacity
+# contract, not a throughput one (2-way concurrency summed BELOW parity at
+# 4064 ctx), so the slot buys the VRAM headroom long generations need.
+#
+# KV_BYTES is PER RANK. fp8 costs 7,701 B/token (measured 6,138 attention
+# + 1,563 mamba/side, dtype-independent). 2 x 200704 = 401,408 tokens
+# -> 3.091e9 B floor, pinned at 3.15e9 (~405,000 tokens, ~44 MB margin —
+# the same margin convention the 6.1e9 shape used).
+#
+# THE ARENA IS WHAT REMAINS: the fork's expert planner sizes it AFTER the
+# KV override, so every byte cut here is arena. The 6.1e9 shape ran an
+# arena of 6,248 slots; this one reclaims ~2.95 GiB/rank, roughly 1,260
+# more slots at 2.344 MiB each. That is the point of the change — the
+# gather read path is the measured decode bound, and a bigger arena means
+# more experts resident instead of streamed.
+#
+# MNBT 1024 (down from 2048): 1024-4096 are interchangeable on decode AND
+# identical on arena (the sweep measured the same slot count at every
+# rung <= 4096), so this is a pure workspace-floor choice, not a decode
+# trade. The floor matters because the arena budget is what is left.
+#
+# PRECISE MEMORY stays DROPPED: it crashed the 2 x 262144 bf16 boot
+# (weights 6.28 + pool 6.80 + graphs 0.55 + arena 16.65 + non-torch ~1.45
+# = 31.73 of 31.86 GiB, ~0.13 GiB free, NRestarts=3). The flag survives in
+# the serve script as an opt-in; re-enable it only with a pool/context
+# reduction that actually pays for the extra overhead.
+#
+# TUNABLEOP stays OFF (silently skipped under offload unless
+# CLAV_TUNABLEOP_WITH_OFFLOAD=1, and then corrupts numerics).
+#
+# SEQS 2 x MAXLEN 200704 at FP8 KV, pool cut for the arena. No
+# speculative decode, no prefetch.
+FRANK_PRIMARY_MAXLEN=200704
+FRANK_PRIMARY_SEQS=2
+FRANK_PRIMARY_KV_BYTES=3150000000
 FRANK_PRIMARY_KV_DTYPE=fp8
-FRANK_PRIMARY_MNBT=2048
+FRANK_PRIMARY_MNBT=1024
 FRANK_PRIMARY_GATHER_CHUNKS=32
 FRANK_PRIMARY_GATHER_LANES=64
 FRANK_PRIMARY_UTIL=0.98

scripts/serve-frank-primary.sh

@@ -11,14 +11,16 @@
 # every flag here was qualified by boot probe on 2026-09-29 — see
 # frank-primary.env for the qualification record).
 #
-# Shape (2026-10-01 bf16 cutover): MAXLEN 262144 x 3 sessions (786,432
-# tokens), KV UNQUANTIZED (--kv-cache-dtype auto = bf16; the fp8 q8/q8 shape
-# is retired to git tag frank-primary-fp8-mtp-20261001), KV_BYTES
-# 11,047,010,304 PER RANK (the engine measured 798,280 tokens of capacity /
-# 3.05x concurrency), NO speculative decode, --max-num-batched-tokens 1024
-# (Frank's lock), MoE gather geometry R4D_LRU_GATHER_CHUNKS=32 /
-# R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows with depth), prefix caching,
-# expert prefetch OFF (36% precision measured), TunableOp off.
+# Shape (2026-10-02, Frank's correction of the 10-01 bf16 shape): MAXLEN
+# 200704 x SEQS 2 (401,408 tokens of contract), KV fp8 (--kv-cache-dtype
+# fp8; the bf16 unquantized shape is retired to git tag
+# frank-primary-fp8-mtp-20261001), KV_BYTES 3,150,000,000 PER RANK (the
+# engine measured 405,000 tokens of capacity / ~3.0x concurrency), NO
+# speculative decode, --max-num-batched-tokens 1024 (Frank's lock), MoE
+# gather geometry R4D_LRU_GATHER_CHUNKS=32 /
+# R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows with depth), prefix
+# caching, expert prefetch OFF (36% precision measured), TunableOp off, and
+# the pool deliberately small so the expert arena keeps what it reclaims.
 #
 # The root unit (scripts/frank-primary.service, User=dank,
 # LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host