fix(frank-archon): size the host pool so the KV cache clears 3 x 200,000
The 1.8.1 boot resolved the budget resolver to 4.20 GiB of cache a card — 556,348 tokens, 2.8 sessions at full height — so three concurrent full-height requests are refused with a 400 at admission: the endpoint's 3-slot contract is broken by the cutover, silently, because /health and every shorter conversation still answer. The resolver gives the expert plane whatever the pinned host pool cannot hold and the cache the rest, so the pool is the lever: +640 MiB a rank moves that much expert demand into host RAM and the cache wins it back. Tokens scale with the GiB won (8.11 KiB/token at fp8), so ~0.63 GiB buys ~44,000 tokens — 600,000 with margin — at ~2.5% off the expert slab. TTM floor moves with the pool (2 x pool + 48,824 + prefix + MNBT = 150,200 MiB computed by the ledger test). Verified by the boot ledger's `kv:` line and the probe's capacity leg, not by this arithmetic.
This commit is contained in:
@@ -63,7 +63,12 @@ FRANK_ARCHON_PLACEMENT=expert_tiered
|
|||||||
# rest, so growing the pool moves experts off the card and shrinking it spills
|
# rest, so growing the pool moves experts off the card and shrinking it spills
|
||||||
# them onto VRAM the cache wanted. Under-sizing it is a boot failure (36,864
|
# them onto VRAM the cache wanted. Under-sizing it is a boot failure (36,864
|
||||||
# refused a boot). The ledger's coverage line is the referee, not this file.
|
# refused a boot). The ledger's coverage line is the referee, not this file.
|
||||||
FRANK_ARCHON_HOST_POOL_MIB=44928
|
# Sized so the cache clears the 3 x MAX_MODEL_LEN contract: the 1.8.1 boot at
|
||||||
|
# 44,928 resolved the cache to 556,348 tokens — 2.8 sessions — and three
|
||||||
|
# concurrent full-height requests came back 400 in 0.0 s. Tokens scale with the
|
||||||
|
# GiB the cache wins, so +640 MiB a rank buys ~44,000 tokens of ceiling with
|
||||||
|
# ~2.5% off the expert slab.
|
||||||
|
FRANK_ARCHON_HOST_POOL_MIB=45568
|
||||||
# VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle
|
# VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle
|
||||||
# floor is built from it.
|
# floor is built from it.
|
||||||
FRANK_ARCHON_GPU_HEADROOM_MIB=96
|
FRANK_ARCHON_GPU_HEADROOM_MIB=96
|
||||||
@@ -107,11 +112,14 @@ FRANK_ARCHON_PREFIX_CACHE_MIB=8192
|
|||||||
# read. `ram` REFUSES the start if memlock is too small — the unit's
|
# read. `ram` REFUSES the start if memlock is too small — the unit's
|
||||||
# LimitMEMLOCK=infinity is the other half of this law.
|
# LimitMEMLOCK=infinity is the other half of this law.
|
||||||
FRANK_ARCHON_NGRAM_PLACEMENT=ram
|
FRANK_ARCHON_NGRAM_PLACEMENT=ram
|
||||||
# TTM preflight. The pinned ledger (~147 GiB) rides TTM's shared pages, and the
|
# TTM preflight. The pinned ledger rides TTM's shared pages, and the kernel
|
||||||
# kernel RESETS pages_limit to half of RAM on every boot; below the plan the
|
# RESETS pages_limit to half of RAM on every boot; below the plan the load dies
|
||||||
# load dies mid-pin with an anonymous allocator error. ttm-preflight.sh raises
|
# mid-pin with an anonymous allocator error. ttm-preflight.sh raises the knob
|
||||||
# the knob itself before refusing by name, so recovery is a boot, not a visit.
|
# itself before refusing by name, so recovery is a boot, not a visit.
|
||||||
FRANK_ARCHON_TTM_MIN_MIB=150000
|
# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT
|
||||||
|
# per the bats ledger test; at the 45,568 pool that computes to 150,200 MiB, so
|
||||||
|
# the floor sits just above it.
|
||||||
|
FRANK_ARCHON_TTM_MIN_MIB=152000
|
||||||
FRANK_ARCHON_TTM_TARGET_MIB=204800
|
FRANK_ARCHON_TTM_TARGET_MIB=204800
|
||||||
# Escape hatch for experiments. Unset -> no extra argv; the serve script expands
|
# Escape hatch for experiments. Unset -> no extra argv; the serve script expands
|
||||||
# this unquoted on purpose, so keep multi-flag values as one string.
|
# this unquoted on purpose, so keep multi-flag values as one string.
|
||||||
|
|||||||
@@ -155,7 +155,7 @@ nosudo_bin() {
|
|||||||
# so the pool's token count is a boot-ledger line — verified live by the
|
# so the pool's token count is a boot-ledger line — verified live by the
|
||||||
# capacity leg of scripts/probe-frank-archon-kv.py at every cutover.
|
# capacity leg of scripts/probe-frank-archon-kv.py at every cutover.
|
||||||
grep -q '^FRANK_ARCHON_PLACEMENT=expert_tiered$' "$ENV_FILE"
|
grep -q '^FRANK_ARCHON_PLACEMENT=expert_tiered$' "$ENV_FILE"
|
||||||
grep -q '^FRANK_ARCHON_HOST_POOL_MIB=44928$' "$ENV_FILE"
|
grep -q '^FRANK_ARCHON_HOST_POOL_MIB=45568$' "$ENV_FILE"
|
||||||
grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE"
|
grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||||
grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE"
|
grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE"
|
||||||
# Prefix cache: pinned host RAM only (the disk tier was removed), and the
|
# Prefix cache: pinned host RAM only (the disk tier was removed), and the
|
||||||
|
|||||||
Reference in New Issue
Block a user