fix(frank-archon): raise the ttm pages cap in the serve path; reshape to 4 x 200,000 fp8 KV
The 2026-10-12 23:17 power failure reset /sys/module/ttm/parameters/pages_limit from the manual 2026-10-05 raise (200 GiB) back to the kernel default (half of RAM = 125,988 MiB), below the pinned-RAM ledger floor of 150,000 MiB. The serve script's preflight named the refusal and exited 1; Restart=always re-ran it 6,919 times over ~20 hours and frank-archon served nobody. scripts/ttm-preflight.sh (sourced by the serve script) now RAISES the knob itself before refusing: the unit is root, the knob is a 0644 root module parameter, and the cap is a runtime knob whose default is half of RAM — so a manual sysfs write is not a repair procedure, it is a countdown to the next reboot. FRANK_ARCHON_TTM_TARGET_MIB=204800 records the host ceiling; the gate re-reads the cap after writing and only then refuses, so a knob the kernel will not move is still a named failure. Shape change (Frank's call): FOUR decode slots x 200,000 tokens with fp8 (quantized) KV. The pool must back seqs x max-model-len, so fp8 is what funds the fourth slot: at bf16 the four slots cost 11,679 MiB of pool and would cut the resident expert slab to 17,441 MiB (-33 %); at fp8 they cost 6,601 MiB, so the pool stays 6,912 MiB (predicts 843,566 tokens), the VRAM pair stays at the measured 29,120 MiB/card ceiling, the weights slab and the 44 GiB host pool are untouched, and the fused TP all-reduce rides 4 x (1+2) = 12 rows, inside the 25-row bound where the wht6 wire is bit-exact.
This commit is contained in:
Frank committed
1 parent
c52ba781e3
commit
1a6a78ef73
4 files changed
+218
-81
No files matched your search
@@ -13,10 +13,11 @@
|
||||
# log states every budget before anything is allocated (weights vs KV
|
||||
# split, the expert host pool's coverage of the worst case, the PLE pin,
|
||||
# the P2P answer per pair) — read it after every resize.
|
||||
# Deployed shape (2026-10-11): 2 decode slots x 200,000 tokens, bf16
|
||||
# (unquantized) KV in a 6,912 MiB pool, the weights slab cut to 22,208 MiB to
|
||||
# pay for it, 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache host
|
||||
# 8 GiB + disk 64 GiB, PLE pinned in RAM.
|
||||
# Deployed shape (2026-10-13): FOUR decode slots x 200,000 tokens, fp8
|
||||
# (quantized) KV in a 6,912 MiB pool — the dtype that funds four slots
|
||||
# without cutting the weights slab below its 22,208 MiB size, 44 GiB/rank
|
||||
# pinned expert pool, MTP depth 2, prefix cache host 8 GiB + disk 64 GiB, PLE
|
||||
# pinned in RAM.
|
||||
#
|
||||
# The root unit (scripts/frank-archon.service, User=dank,
|
||||
# LimitMEMLOCK=infinity) grants the pin budget: 2 x 44 GiB expert pools +
|
||||
@@ -61,6 +62,7 @@ set +a
|
||||
: "${FRANK_ARCHON_PREFIX_DISK_MIB:?FRANK_ARCHON_PREFIX_DISK_MIB must be set}"
|
||||
: "${FRANK_ARCHON_NGRAM_PLACEMENT:?FRANK_ARCHON_NGRAM_PLACEMENT must be set}"
|
||||
: "${FRANK_ARCHON_TTM_MIN_MIB:?FRANK_ARCHON_TTM_MIN_MIB must be set}"
|
||||
:"${FRANK_ARCHON_TTM_TARGET_MIB:?FRANK_ARCHON_TTM_TARGET_MIB must be set}"
|
||||
: "${FRANK_ARCHON_RAD_MIN_MIB:?FRANK_ARCHON_RAD_MIN_MIB must be set}"
|
||||
|
||||
# The image gate is the digest-pinned ref itself: a name@sha256 ref exists
|
||||
@@ -117,34 +119,19 @@ if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# TTM preflight: the pinned ledger (~2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
|
||||
# prefix host = ~132 GiB) rides TTM's shared pages, capped by ttm's
|
||||
# pages_limit (the current kernel exposes NO pages_max; the cap function
|
||||
# takes the min over the knobs that exist). Below FRANK_ARCHON_TTM_MIN_MIB
|
||||
# the load dies mid-pin with an allocator error instead of this named
|
||||
# refusal — raise the knob first (runtime sysfs write; kernel cmdline
|
||||
# fallback + one reboot only if the file is read-only), see the runbook.
|
||||
# RAISED 2026-10-05 pre-cutover to 52,428,800 pages = 200 GiB.
|
||||
ttm_cap_mib() {
|
||||
# min over the knobs THIS kernel exposes — the current kernel has
|
||||
# pages_limit only (no pages_max); a knob that is absent is skipped,
|
||||
# not a zero (a zero would refuse a healthy cap).
|
||||
local v cur=""
|
||||
for p in /sys/module/ttm/parameters/pages_limit /sys/module/ttm/parameters/pages_max; do
|
||||
[ -r "$p" ] || continue
|
||||
v="$(cat "$p" 2>/dev/null)" || continue
|
||||
[ -n "$v" ] || continue
|
||||
v=$(( v * 4 / 1024 ))
|
||||
if [ "$v" -lt "${cur:-999999999}" ]; then cur=$v; fi
|
||||
done
|
||||
echo "${cur:-0}"
|
||||
}
|
||||
TTM_CAP="$(ttm_cap_mib)"
|
||||
if [ "$TTM_CAP" -lt "$FRANK_ARCHON_TTM_MIN_MIB" ]; then
|
||||
echo "ERROR: ttm pages cap is ${TTM_CAP} MiB, the pinned ledger needs >= $FRANK_ARCHON_TTM_MIN_MIB" >&2
|
||||
echo "frank-archon: raise /sys/module/ttm/parameters/pages_limit (and pages_max if the kernel exposes one) to >= $FRANK_ARCHON_TTM_MIN_MIB in 4 KiB pages" >&2
|
||||
exit 1
|
||||
fi
|
||||
# TTM preflight: the pinned ledger (2 x FRANK_ARCHON_HOST_POOL_MIB + the
|
||||
# 47.7 GiB PLE table + the prefix host tier, ~147 GiB) rides TTM's shared
|
||||
# pages, capped by ttm's pages_limit (the current kernel exposes NO
|
||||
# pages_max; the cap function takes the min over the knobs that exist).
|
||||
# Below FRANK_ARCHON_TTM_MIN_MIB the load dies mid-pin with an allocator
|
||||
# error, so the gate refuses first — and RAISES the knob before refusing:
|
||||
# the cap is a runtime knob whose default is half of RAM, so a power failure
|
||||
# resets it (measured 2026-10-12 23:17 on this host: 200 GiB -> 125,988 MiB,
|
||||
# and Restart=always re-ran the refusal 6,919 times over ~20 h).
|
||||
# FRANK_ARCHON_TTM_TARGET_MIB is the recorded host ceiling, not the floor.
|
||||
# shellcheck source=/dev/null
|
||||
source "$SCRIPT_DIR/ttm-preflight.sh"
|
||||
ttm_ensure_cap || exit 1
|
||||
|
||||
# A crashed/killed unit can leave the container name registered (systemd
|
||||
# SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
|
||||
|
||||
Reference in new issue
Block a user