diff --git a/scripts/frank-archon.env b/scripts/frank-archon.env index 287372c..65ac22c 100644 --- a/scripts/frank-archon.env +++ b/scripts/frank-archon.env @@ -93,14 +93,21 @@ FRANK_ARCHON_SPEC_TOKENS=2 FRANK_ARCHON_MAX_BATCHED_TOKENS=2048 # Step sharing. A prefill chunk that shares a step with decoding sequences holds # the device for the chunk's whole time, and every decode row in that step waits -# that long for its next token (the census: [[LLM Performance Diagnosis]] §9). -# `auto` leaves each decode row behind the chunk a row plus its draft window and -# keeps a share for whatever is waiting to be admitted; a number pins the chunk's -# cap; `off` is the pre-1.4 behaviour this endpoint ran for two eras, where the -# chunk took the step and admission never ran. Interleave inserts N decode-only -# steps after a chunk step taken beside decoders: it protects their cadence and -# holds a waiting request out of those steps, so it stays 0 until the post-migration -# decode-vs-prefill buckets say it is wanted. +# that long for its next token: measured at this shape, decode-only steps 15.5 ms +# against chunk steps 166 ms, which is the harness's ~146 t/s falling to ~11 t/s +# while another session prefills (law and census: [[frank-archon]]). +# `auto` is a PREFILL-SPEED dial, not a decoder dial: it reserves one row plus the +# draft window per decoding sequence (3 tokens) and an eighth of the budget per +# waiting request or waiting prefill, and the source states it will not halve for +# decode rows alone. Live proof: prefill_chunks_capped 5,018 across 62,386 chunk +# steps. A number is the only value that applies whenever the chunk shares a step, +# and it must not go under the prefill stager's smallest pass -- 1025 rows here, +# named on the boot line "prefill staging: ... a step of more than 1024 tokens" -- +# because below it every routed layer reads its pooled experts synchronously, so +# 512 is a trap and the vendor measures the whole job slower there. Interleave +# inserts N decode-only steps after a chunk step taken beside decoders: it protects +# their cadence and holds a waiting request out of those steps. That is the decoder +# dial, and it is 0 until the decode-vs-prefill buckets ask for it. FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto FRANK_ARCHON_PREFILL_INTERLEAVE=0 # Prefix cache: pinned host memory only. The disk tier was removed outright diff --git a/scripts/serve-frank-archon.sh b/scripts/serve-frank-archon.sh index 2cf39f3..d56fbdc 100755 --- a/scripts/serve-frank-archon.sh +++ b/scripts/serve-frank-archon.sh @@ -15,13 +15,16 @@ # log states every budget before anything is allocated (weights vs KV # split, the expert host pool's coverage of the worst case, the PLE pin, # the P2P answer per pair) — read it after every resize. -# Deployed shape (2026-10-19, radiance 1.8.1): THREE decode slots x 200,000 -# tokens, fp8 KV, 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache in -# pinned host RAM, PLE pinned in RAM. The per-card split between the expert slab -# and the KV pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair -# and the budget resolver derives it from headroom, the cache bound and the host -# pool, so the slab size and the pool's token count are LEDGER lines: after every -# boot read them out of the log and check the pool still backs 3 x 200,000. +# Deployed shape (2026-10-19, radiance 1.8.1): TWO decode slots x 200,000 tokens, +# fp8 KV, 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache in pinned host +# RAM, PLE pinned in RAM. The per-card split between the expert slab and the KV +# pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair and the budget +# resolver derives it from headroom, the cache bound and the host pool, so the slab +# size and the pool's token count are LEDGER lines. After every boot read them, plus +# the two lines that size a prefill step — "activation arena: ... lends ... while +# steps are at most N tokens" (the arena levels the expert slab borrows) and +# "prefill staging: ... a step of more than N tokens ..." (the chunk size below which +# pooled experts are read synchronously) — and the cache's session backing. # # The root unit (scripts/frank-archon.service, User=dank, # LimitMEMLOCK=infinity) grants the pin budget: 2 x 44 GiB expert pools +