fix(frank-archon): raise the ttm pages cap in the serve path; reshape to 4 x 200,000 fp8 KV

The 2026-10-12 23:17 power failure reset /sys/module/ttm/parameters/pages_limit
from the manual 2026-10-05 raise (200 GiB) back to the kernel default (half of
RAM = 125,988 MiB), below the pinned-RAM ledger floor of 150,000 MiB. The serve
script's preflight named the refusal and exited 1; Restart=always re-ran it
6,919 times over ~20 hours and frank-archon served nobody.

scripts/ttm-preflight.sh (sourced by the serve script) now RAISES the knob
itself before refusing: the unit is root, the knob is a 0644 root module
parameter, and the cap is a runtime knob whose default is half of RAM — so a
manual sysfs write is not a repair procedure, it is a countdown to the next
reboot. FRANK_ARCHON_TTM_TARGET_MIB=204800 records the host ceiling; the gate
re-reads the cap after writing and only then refuses, so a knob the kernel
will not move is still a named failure.

Shape change (Frank's call): FOUR decode slots x 200,000 tokens with fp8
(quantized) KV. The pool must back seqs x max-model-len, so fp8 is what funds
the fourth slot: at bf16 the four slots cost 11,679 MiB of pool and would cut
the resident expert slab to 17,441 MiB (-33 %); at fp8 they cost 6,601 MiB, so
the pool stays 6,912 MiB (predicts 843,566 tokens), the VRAM pair stays at the
measured 29,120 MiB/card ceiling, the weights slab and the 44 GiB host pool are
untouched, and the fused TP all-reduce rides 4 x (1+2) = 12 rows, inside the
25-row bound where the wht6 wire is bit-exact.
This commit is contained in:
Frank
2026-10-13 19:09:58 -07:00
parent c52ba781e3
commit 1a6a78ef73
4 changed files with 218 additions and 81 deletions
+67 -10
View File
@@ -9,8 +9,9 @@
# flags alone do not say: the served window clears the harness worst case,
# the KV pool backs seqs x max-model-len at the declared cache dtype (an
# engine HARD limit, not a ratio), the per-card VRAM pair stays inside the
# measured card ceiling, the pinned-host ledger has a ttm pages cap floor and
# the serve script ENFORCES it, the served window is what models.yml must
# the pinned-host ledger has a ttm pages cap floor, the serve script ENFORCES
# it and RAISES the knob itself when a reboot reset it (the cap's default is
# half of RAM), the served window is what models.yml must
# quote, the host-networking contract (no --publish), the image gate on a
# name@sha256 ref, the .rad payload gate with its size floor, the
# chat-template override mount, the self-healing law (the run returns before
@@ -24,6 +25,7 @@ load helpers
UNIT="$REPO_ROOT/scripts/frank-archon.service"
SERVE="$REPO_ROOT/scripts/serve-frank-archon.sh"
TTM_LIB="$REPO_ROOT/scripts/ttm-preflight.sh"
ENV_FILE="$REPO_ROOT/scripts/frank-archon.env"
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
DOWNLOAD="$REPO_ROOT/scripts/download-frank-archon-swift15.sh"
@@ -34,6 +36,7 @@ TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
@test "frank-archon unit, serve script, env, fetch and build scripts and template exist" {
[ -f "$UNIT" ]
[ -x "$SERVE" ]
[ -f "$TTM_LIB" ]
[ -f "$ENV_FILE" ]
[ -f "$DOWNLOAD" ]
[ -x "$BUILD" ]
@@ -120,37 +123,40 @@ TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
grep -q '^FRANK_ARCHON_MAX_MODEL_LEN=200000$' "$ENV_FILE"
ml=$(awk -F= '/^FRANK_ARCHON_MAX_MODEL_LEN=/{print $2}' "$ENV_FILE")
[ "$ml" -ge 177356 ]
# Concurrency: 2 slots x the served window is the capacity contract, and
# the fused TP all-reduce is declared over seqs x (1 + spec) = 6 rows,
# Concurrency: 4 slots x the served window is the capacity contract, and
# the fused TP all-reduce is declared over seqs x (1 + spec) = 12 rows,
# under the 25-row bound at which the wht6 wire stays bit-exact. The
# "seqs 2 cannot boot" law belongs to the retired tcclaviger vLLM fork
# (torch.compile query_start_loc constraint, 2026-10-02); radiance booted
# seqs 2 in the fp8-native era and the 2026-10-06 sweep had seqs 2 and 3
# tied inside the +-3% noise floor.
grep -q '^FRANK_ARCHON_MAX_SEQS=2$' "$ENV_FILE"
# tied inside the +-3% noise floor — slots buy concurrency, not speed.
grep -q '^FRANK_ARCHON_MAX_SEQS=4$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_TP=2$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_TP_WIRE=wht6$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_SPEC_TOKENS=2$' "$ENV_FILE"
# expert_tiered placement: vram_weights + the pinned host pool serve the
# per-rank resident set TOGETHER, so cutting the VRAM slab to pay for the
# KV pool moves the pool with it. KV is UNQUANTIZED bf16 (the engine takes
# KV pool moves the pool with it. KV is QUANTIZED fp8 (the engine takes
# only bf16|fp8 — no per-tensor K/V split), the PLE n-gram table is pinned
# in host RAM per Frank's law.
grep -q '^FRANK_ARCHON_PLACEMENT=expert_tiered$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_HOST_POOL_MIB=44928$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_VRAM_WEIGHTS_MIB=[1-9]' "$ENV_FILE"
grep -q '^FRANK_ARCHON_VRAM_KV_MIB=[1-9]' "$ENV_FILE"
grep -q '^FRANK_ARCHON_KV_DTYPE=bf16$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE"
# Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
grep -q '^FRANK_ARCHON_PREFIX_HOST_MIB=8192$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_PREFIX_DISK_MIB=65536$' "$ENV_FILE"
# The pinned ledger (2 x pool + 47.7 GiB PLE + prefix host) must be
# under the ttm pages cap, and the env states the floor the serve
# script enforces (>= 140,000 MiB — the current cap ~123 GiB fails it,
# which is the point).
# script enforces (>= 140,000 MiB — the kernel's own default cap, half of
# RAM, fails it, which is the point) plus the value it raises the knob to
# when a reboot reset it (the target must clear the floor with slack).
grep -q '^FRANK_ARCHON_TTM_MIN_MIB=' "$ENV_FILE"
ttmm=$(awk -F= '/^FRANK_ARCHON_TTM_MIN_MIB=/{print $2}' "$ENV_FILE")
ttmt=$(awk -F= '/^FRANK_ARCHON_TTM_TARGET_MIB=/{print $2}' "$ENV_FILE")
[ "$ttmt" -gt "$ttmm" ]
pool=$(awk -F= '/^FRANK_ARCHON_HOST_POOL_MIB=/{print $2}' "$ENV_FILE")
ph=$(awk -F= '/^FRANK_ARCHON_PREFIX_HOST_MIB=/{print $2}' "$ENV_FILE")
# 2 pools + PLE 47.7 GiB (48,824 MiB) + prefix host, rounded up.
@@ -272,6 +278,57 @@ TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
grep -q -- '--device /dev/dri' "$SERVE"
}
@test "the ttm gate raises a reset pages cap and refuses when the write fails" {
# The cap is a RUNTIME knob: the kernel default is half of RAM, so every
# reboot resets it (measured: the 2026-10-12 23:17 power failure on
# dankserver took it 200 GiB -> 125,988 MiB, and the preflight refusal was
# re-run 6,919 times by Restart=always over ~20 h). Behaviour required of
# the gate, in both directions:
# knob below the floor + writable -> cap ends at the target, start passes
# knob below the floor + unwritable -> cap stays low, start REFUSES by name
local dir="$BATS_TEST_TMPDIR/ttm"
mkdir -p "$dir"
export FRANK_ARCHON_TTM_SYS_DIR="$dir"
export FRANK_ARCHON_TTM_MIN_MIB=150000 FRANK_ARCHON_TTM_TARGET_MIB=204800
echo 32253039 > "$dir/pages_limit" # 125,988 MiB: the kernel default here
run bash -c '. "$1"; ttm_ensure_cap' _ "$TTM_LIB"
[ "$status" -eq 0 ]
[ "$(cat "$dir/pages_limit")" = "52428800" ] # 204,800 MiB in 4 KiB pages
echo 32253039 > "$dir/pages_limit"
chmod 444 "$dir/pages_limit" # the write refused: cap cannot move
run bash -c '. "$1"; ttm_ensure_cap' _ "$TTM_LIB"
[ "$status" -ne 0 ]
[[ "$output" == *"ERROR: ttm pages cap is 125988 MiB"* ]]
[ "$(cat "$dir/pages_limit")" = "32253039" ]
}
@test "the ttm cap counts only the knobs the kernel exposes" {
# An absent knob is not a zero: a zero-cap reading would refuse a healthy
# host. The effective cap is the MIN over the knobs that exist; pages_max,
# when a kernel exposes one, is the binding one.
local dir="$BATS_TEST_TMPDIR/ttm2"
mkdir -p "$dir"
export FRANK_ARCHON_TTM_SYS_DIR="$dir"
export FRANK_ARCHON_TTM_MIN_MIB=150000 FRANK_ARCHON_TTM_TARGET_MIB=204800
echo 51200000 > "$dir/pages_limit" # 200,000 MiB, no pages_max knob
run bash -c '. "$1"; echo "$(ttm_cap_mib)"' _ "$TTM_LIB"
[ "$status" -eq 0 ]
[ "$output" = "200000" ]
echo 30000000 > "$dir/pages_max" # 117,187 MiB: now the binding knob
run bash -c '. "$1"; echo "$(ttm_cap_mib)"' _ "$TTM_LIB"
[ "$output" = "117187" ]
chmod 444 "$dir/pages_limit" "$dir/pages_max" # both writes refused
run bash -c '. "$1"; ttm_ensure_cap' _ "$TTM_LIB"
[ "$status" -ne 0 ]
[[ "$output" == *"ERROR: ttm pages cap is 117187 MiB"* ]]
[ "$(cat "$dir/pages_max")" = "30000000" ]
}
@test "the serve script clears stale containers and self-heals via podman wait" {
# A crashed unit can leave the container name registered forever after;
# the legacy identities hold the same GPUs and die here too. Then block