frank-primary: 2 x 200704 fp8, pool cut for the expert arena, MNBT 1024
The 2026-10-01 bf16 shape (2 x 262144, --expert-precise-memory) crash-looped at graph capture — the arena is DEVICE-resident and that shape had 0.13 GiB to spare. The plain bf16 2 x 262144 also did not stay up: 0 bytes free at runtime, and a real client request (995-token prompt, max_tokens=261149) killed the worker with . The correction is a pool cut, not a downgrade. FP8 KV (7,701 B/token), two 200704 windows (401,408 tokens -> 3.091e9 B floor, pinned 3.15e9), MNBT 1024. The fork's expert planner sizes the arena AFTER the KV override, so the ~2.95 GiB/rank reclaimed becomes arena: ~7,500 slots vs 6,248. Two contexts at 200704 is a capacity contract, not a throughput one (2-way concurrency summed BELOW parity at 4064 ctx), so the slot buys the VRAM headroom long generations need. Refs [[dankserver-qwen38-fp8-expert-offload-tuning]]. - scripts/frank-primary.env: MAXLEN 200704, SEQS 2, KV_BYTES 3.15e9, MNBT 1024, rewritten shape comment - scripts/serve-frank-primary.sh: header comment updated - tests/frank_primary_unit.bats: assertions updated, new seam pins the arena-floor pool law (the pool must move with a context change or the reduction buys nothing) - docs + AGENTS.md: fleet topology and deployed shape corrected
This commit is contained in:
1 parent
f8f845d86b
commit
30904bac1b
5 files changed
+132
-91
No files matched your search
@@ -11,14 +11,16 @@
|
||||
# every flag here was qualified by boot probe on 2026-09-29 — see
|
||||
# frank-primary.env for the qualification record).
|
||||
#
|
||||
# Shape (2026-10-01 bf16 cutover): MAXLEN 262144 x 3 sessions (786,432
|
||||
# tokens), KV UNQUANTIZED (--kv-cache-dtype auto = bf16; the fp8 q8/q8 shape
|
||||
# is retired to git tag frank-primary-fp8-mtp-20261001), KV_BYTES
|
||||
# 11,047,010,304 PER RANK (the engine measured 798,280 tokens of capacity /
|
||||
# 3.05x concurrency), NO speculative decode, --max-num-batched-tokens 1024
|
||||
# (Frank's lock), MoE gather geometry R4D_LRU_GATHER_CHUNKS=32 /
|
||||
# R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows with depth), prefix caching,
|
||||
# expert prefetch OFF (36% precision measured), TunableOp off.
|
||||
# Shape (2026-10-02, Frank's correction of the 10-01 bf16 shape): MAXLEN
|
||||
# 200704 x SEQS 2 (401,408 tokens of contract), KV fp8 (--kv-cache-dtype
|
||||
# fp8; the bf16 unquantized shape is retired to git tag
|
||||
# frank-primary-fp8-mtp-20261001), KV_BYTES 3,150,000,000 PER RANK (the
|
||||
# engine measured 405,000 tokens of capacity / ~3.0x concurrency), NO
|
||||
# speculative decode, --max-num-batched-tokens 1024 (Frank's lock), MoE
|
||||
# gather geometry R4D_LRU_GATHER_CHUNKS=32 /
|
||||
# R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows with depth), prefix
|
||||
# caching, expert prefetch OFF (36% precision measured), TunableOp off, and
|
||||
# the pool deliberately small so the expert arena keeps what it reclaims.
|
||||
#
|
||||
# The root unit (scripts/frank-primary.service, User=dank,
|
||||
# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
|
||||
|
||||
Reference in new issue
Block a user