From 4d41366f4c5cf4c0124b46fe401f9ba4119c9832 Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 11:07:11 -0700 Subject: [PATCH] fix(frank-archon): declare two slots, revert the inert host-pool raise FRANK_ARCHON_MAX_SEQS 3 -> 2. At 1.8.1 the budget resolver sizes the KV cache from the cache bound (--max-model-len x --max-num-seqs), --gpu-headroom-mib and --host-pool-mib, and the shipped CLI has no KV-size flag (the --vram-weights-mib/--vram-kv-mib pair was removed at 1.3.0). The live boot ledger reads that cache at 4.20 GiB = 556,348 tokens = 2.8 sessions at 200,000, so a third full-height conversation is refused while /health answers green and the harness advertises capacity the engine cannot honour. Two slots is what the shape actually serves at the full window; a slot costs KV, never decode, and the fused TP all-reduce drops from 9 rows to 2 x (1 + spec) = 6, still inside wht6's 25-row bit-exact bound. FRANK_ARCHON_HOST_POOL_MIB back to 44928 and FRANK_ARCHON_TTM_MIN_MIB back to 150000, reverting PR #30. That raise bought cache the resolver never hands out: the two boot ledgers differ in exactly one line -- host_pool 44.47 -> 45.09 GiB -- while vram_weights (25.81 GiB) and vram_kv (4.20 GiB / 556,348 tokens, both ranks) are byte-identical, so the extra 640 MiB sat unclaimed in the host pool while the TTM floor moved out from under the preflight's arithmetic. The compute-bound cap on --host-pool-mib is unchanged either way (45,568 -> 22,222 MiB of experts either side of the 22,222 clamp). The bats seam pins the new contract: MAX_SEQS=2, pool 44,928, and the ledger floor that computes to 148,920 MiB at that pool against the 150,000 minimum. --- scripts/frank-archon.env | 27 ++++++++++++++------------- tests/frank_archon_unit.bats | 11 +++++++---- 2 files changed, 21 insertions(+), 17 deletions(-) diff --git a/scripts/frank-archon.env b/scripts/frank-archon.env index 0d4b363..287372c 100644 --- a/scripts/frank-archon.env +++ b/scripts/frank-archon.env @@ -47,10 +47,13 @@ FRANK_ARCHON_TP=2 # MAX_SEQS x (1 + SPEC_TOKENS) below. FRANK_ARCHON_TP_WIRE=wht6 # Decode slots. A slot costs KV, never decode; the pool is sized to this -# contract and nothing more, so a FOURTH full-height conversation is REFUSED as -# too large to fit — never squeezed, paged or truncated. That is Frank's -# explicit trade: the freed MiB went to the expert slab. -FRANK_ARCHON_MAX_SEQS=3 +# contract and nothing more. At 1.8.1 the budget resolver caps the cache at +# 556,348 fp8 tokens — 2.8 sessions at 200,000 — so a THIRD full-height +# conversation answers 400 while /health and every shorter chat work fine +# (3 x 50k streams run at 77.6 t/s aggregate). Frank's trade: declare the two +# slots the engine can hold and let the freed ~1.1 GiB go to the expert plane, +# instead of advertising a third slot that only exists at shorter prompts. +FRANK_ARCHON_MAX_SEQS=2 # Served window. The engine treats `kv group tokens >= max-model-len` as a HARD # limit, and models.yml contextWindow MUST quote this number (over-length is a # refusal, not a truncation). The pool is sized for seqs x maxlen, not one @@ -63,12 +66,11 @@ FRANK_ARCHON_PLACEMENT=expert_tiered # rest, so growing the pool moves experts off the card and shrinking it spills # them onto VRAM the cache wanted. Under-sizing it is a boot failure (36,864 # refused a boot). The ledger's coverage line is the referee, not this file. -# Sized so the cache clears the 3 x MAX_MODEL_LEN contract: the 1.8.1 boot at -# 44,928 resolved the cache to 556,348 tokens — 2.8 sessions — and three -# concurrent full-height requests came back 400 in 0.0 s. Tokens scale with the -# GiB the cache wins, so +640 MiB a rank buys ~44,000 tokens of ceiling with -# ~2.5% off the expert slab. -FRANK_ARCHON_HOST_POOL_MIB=45568 +# Measured, not guessed: 45,568 (PR #30) moved host_pool to 45.09 GiB and left +# vram_kv byte-identical at 4.20 GiB / 556,348 tokens on both ranks — extra +# host RAM above what the expert plane's footprint needs buys no cache. Back at +# 44,928, the size that boots clean. +FRANK_ARCHON_HOST_POOL_MIB=44928 # VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle # floor is built from it. FRANK_ARCHON_GPU_HEADROOM_MIB=96 @@ -117,9 +119,8 @@ FRANK_ARCHON_NGRAM_PLACEMENT=ram # mid-pin with an anonymous allocator error. ttm-preflight.sh raises the knob # itself before refusing by name, so recovery is a boot, not a visit. # Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT -# per the bats ledger test; at the 45,568 pool that computes to 150,200 MiB, so -# the floor sits just above it. -FRANK_ARCHON_TTM_MIN_MIB=152000 +# per the bats ledger test; at the 44,928 pool that computes to 148,920 MiB. +FRANK_ARCHON_TTM_MIN_MIB=150000 FRANK_ARCHON_TTM_TARGET_MIB=204800 # Escape hatch for experiments. Unset -> no extra argv; the serve script expands # this unquoted on purpose, so keep multi-flag values as one string. diff --git a/tests/frank_archon_unit.bats b/tests/frank_archon_unit.bats index 73682ba..de4ed5a 100755 --- a/tests/frank_archon_unit.bats +++ b/tests/frank_archon_unit.bats @@ -135,14 +135,17 @@ nosudo_bin() { grep -q '^FRANK_ARCHON_MAX_MODEL_LEN=200000$' "$ENV_FILE" ml=$(awk -F= '/^FRANK_ARCHON_MAX_MODEL_LEN=/{print $2}' "$ENV_FILE") [ "$ml" -ge 177356 ] - # Concurrency: 3 slots x the served window is the capacity contract, and - # the fused TP all-reduce is declared over seqs x (1 + spec) = 9 rows, + # Concurrency: 2 slots x the served window is the capacity contract at + # 1.8.1. The budget resolver caps the cache at 556,348 fp8 tokens — 2.8 + # sessions at 200,000 — so three concurrent full-height chats answer 400, + # and the shipped CLI has no KV-size flag to raise it (removed at 1.3.0). + # The fused TP all-reduce is declared over seqs x (1 + spec) = 6 rows, # under the 25-row bound at which the wht6 wire stays bit-exact. The # "seqs 2 cannot boot" law belongs to the retired tcclaviger vLLM fork # (torch.compile query_start_loc constraint, 2026-10-02); radiance booted # seqs 2 in the fp8-native era and the 2026-10-06 sweep had seqs 2 and 3 # tied inside the +-3% noise floor — slots buy concurrency, not speed. - grep -q '^FRANK_ARCHON_MAX_SEQS=3$' "$ENV_FILE" + grep -q '^FRANK_ARCHON_MAX_SEQS=2$' "$ENV_FILE" grep -q '^FRANK_ARCHON_TP=2$' "$ENV_FILE" grep -q '^FRANK_ARCHON_TP_WIRE=wht6$' "$ENV_FILE" grep -q '^FRANK_ARCHON_SPEC_TOKENS=2$' "$ENV_FILE" @@ -155,7 +158,7 @@ nosudo_bin() { # so the pool's token count is a boot-ledger line — verified live by the # capacity leg of scripts/probe-frank-archon-kv.py at every cutover. grep -q '^FRANK_ARCHON_PLACEMENT=expert_tiered$' "$ENV_FILE" - grep -q '^FRANK_ARCHON_HOST_POOL_MIB=45568$' "$ENV_FILE" + grep -q '^FRANK_ARCHON_HOST_POOL_MIB=44928$' "$ENV_FILE" grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE" grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE" # Prefix cache: pinned host RAM only (the disk tier was removed), and the -- 2.54.0