frank-primary: revert to known-good fp8 3x262144
The 2 x 200704 fp8 / MNBT 1024 shape (30904ba) crash-loops at boot: the
engine dies in determine_available_memory -> profile_run -> _dummy_run with
the SAME torch.compile dynamo constraint that killed the bf16 2 x 262144
shape on 2026-10-01:
RuntimeError: Worker failed with error 'Constraints violated
(L['query_start_loc'].size()[0])! You marked it as dynamic but your code
specialized it to be a constant (3)'
Not an OOM. Restoring the last shape measured serving (fp8 3 x 262144 /
KV 6.1e9 / MNBT 2048 / arena 6,248) so the endpoint serves again; the
desired edits are re-applied one at a time from this base.
This commit is contained in:
1 parent
30904bac1b
commit
e460bbbab5
3 files changed
+67
-101
No files matched your search
@@ -11,16 +11,14 @@
|
||||
# every flag here was qualified by boot probe on 2026-09-29 — see
|
||||
# frank-primary.env for the qualification record).
|
||||
#
|
||||
# Shape (2026-10-02, Frank's correction of the 10-01 bf16 shape): MAXLEN
|
||||
# 200704 x SEQS 2 (401,408 tokens of contract), KV fp8 (--kv-cache-dtype
|
||||
# fp8; the bf16 unquantized shape is retired to git tag
|
||||
# frank-primary-fp8-mtp-20261001), KV_BYTES 3,150,000,000 PER RANK (the
|
||||
# engine measured 405,000 tokens of capacity / ~3.0x concurrency), NO
|
||||
# speculative decode, --max-num-batched-tokens 1024 (Frank's lock), MoE
|
||||
# gather geometry R4D_LRU_GATHER_CHUNKS=32 /
|
||||
# R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows with depth), prefix
|
||||
# caching, expert prefetch OFF (36% precision measured), TunableOp off, and
|
||||
# the pool deliberately small so the expert arena keeps what it reclaims.
|
||||
# Shape (2026-10-01 bf16 cutover): MAXLEN 262144 x 3 sessions (786,432
|
||||
# tokens), KV UNQUANTIZED (--kv-cache-dtype auto = bf16; the fp8 q8/q8 shape
|
||||
# is retired to git tag frank-primary-fp8-mtp-20261001), KV_BYTES
|
||||
# 11,047,010,304 PER RANK (the engine measured 798,280 tokens of capacity /
|
||||
# 3.05x concurrency), NO speculative decode, --max-num-batched-tokens 1024
|
||||
# (Frank's lock), MoE gather geometry R4D_LRU_GATHER_CHUNKS=32 /
|
||||
# R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows with depth), prefix caching,
|
||||
# expert prefetch OFF (36% precision measured), TunableOp off.
|
||||
#
|
||||
# The root unit (scripts/frank-primary.service, User=dank,
|
||||
# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
|
||||
|
||||
Reference in new issue
Block a user