frank-primary: 2 x 200704 fp8, pool cut for the expert arena, MNBT 1024

The 2026-10-01 bf16 shape (2 x 262144, --expert-precise-memory)
crash-looped at graph capture — the arena is DEVICE-resident and that
shape had 0.13 GiB to spare. The plain bf16 2 x 262144 also did not stay
up: 0 bytes free at runtime, and a real client request (995-token prompt,
max_tokens=261149) killed the worker with .

The correction is a pool cut, not a downgrade. FP8 KV (7,701 B/token),
two 200704 windows (401,408 tokens -> 3.091e9 B floor, pinned 3.15e9),
MNBT 1024. The fork's expert planner sizes the arena AFTER the KV
override, so the ~2.95 GiB/rank reclaimed becomes arena: ~7,500 slots
vs 6,248. Two contexts at 200704 is a capacity contract, not a
throughput one (2-way concurrency summed BELOW parity at 4064 ctx), so
the slot buys the VRAM headroom long generations need.

Refs [[dankserver-qwen38-fp8-expert-offload-tuning]].

- scripts/frank-primary.env: MAXLEN 200704, SEQS 2, KV_BYTES 3.15e9,
  MNBT 1024, rewritten shape comment
- scripts/serve-frank-primary.sh: header comment updated
- tests/frank_primary_unit.bats: assertions updated, new seam pins the
  arena-floor pool law (the pool must move with a context change or the
  reduction buys nothing)
- docs + AGENTS.md: fleet topology and deployed shape corrected
This commit is contained in:
Comrade Chief Designer committed 2026-10-02 09:54:08 -07:00
1 parent f8f845d86b
commit 30904bac1b
5 files changed
+132 -91

No files matched your search

+10 -8
View File
@@ -11,14 +11,16 @@
# every flag here was qualified by boot probe on 2026-09-29 — see
# frank-primary.env for the qualification record).
#
# Shape (2026-10-01 bf16 cutover): MAXLEN 262144 x 3 sessions (786,432
# tokens), KV UNQUANTIZED (--kv-cache-dtype auto = bf16; the fp8 q8/q8 shape
# is retired to git tag frank-primary-fp8-mtp-20261001), KV_BYTES
# 11,047,010,304 PER RANK (the engine measured 798,280 tokens of capacity /
# 3.05x concurrency), NO speculative decode, --max-num-batched-tokens 1024
# (Frank's lock), MoE gather geometry R4D_LRU_GATHER_CHUNKS=32 /
# R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows with depth), prefix caching,
# expert prefetch OFF (36% precision measured), TunableOp off.
# Shape (2026-10-02, Frank's correction of the 10-01 bf16 shape): MAXLEN
# 200704 x SEQS 2 (401,408 tokens of contract), KV fp8 (--kv-cache-dtype
# fp8; the bf16 unquantized shape is retired to git tag
# frank-primary-fp8-mtp-20261001), KV_BYTES 3,150,000,000 PER RANK (the
# engine measured 405,000 tokens of capacity / ~3.0x concurrency), NO
# speculative decode, --max-num-batched-tokens 1024 (Frank's lock), MoE
# gather geometry R4D_LRU_GATHER_CHUNKS=32 /
# R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows with depth), prefix
# caching, expert prefetch OFF (36% precision measured), TunableOp off, and
# the pool deliberately small so the expert arena keeps what it reclaims.
#
# The root unit (scripts/frank-primary.service, User=dank,
# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host