dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(frank-archon): move the endpoint to radiance 1.8.1 and its budget model

Browse Source 
The decode stall is step contention (measured: 18.7 ms decode-only steps
against 317 ms steps that carry a prefill chunk, decode at 13 t/s median
while prefill runs), and the two engine knobs that fix it arrived after this
endpoint's pin. Moving to 1.8.1 (2026-10-18) brings both, and brings two CLI
breaks with them:

  - --vram-weights-mib / --vram-kv-mib were REMOVED at 1.3.0. The per-card
    split between the expert slab and the KV pool is now derived by the
    engine's budget resolver from --gpu-headroom-mib, the cache bound
    (--max-model-len, --max-num-seqs) and --host-pool-mib, and the cache may
    lend idle blocks to the expert slab. The slab size and the pool's token
    count are boot-ledger lines from here on, so the capacity contract
    (3 x 200,000 fp8) is verified live by the probe's capacity leg instead of
    computed from an env number.
  - The prefix cache has no disk tier any more: --prefix-cache-dir and
    --prefix-cache-disk-mib are refusals and --prefix-cache-host-mib is
    --prefix-cache-mib. 64 GiB of parked KV on NVMe becomes re-prefill, so
    prefix_hit_rate is the thing to watch after this boot.

env: image re-pinned to the 1.8.1 digest; the pair and the disk tier dropped;
FRANK_ARCHON_PREFIX_CACHE_MIB replaces the host/disk pair; new
FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto (explicit, because auto vs off is the
difference between decode waiting one chunk and waiting a whole step) and
FRANK_ARCHON_PREFILL_INTERLEAVE=0 (opt-in through a guarded argv array).

serve: argv follows the new CLI, /data mount and the pre-created kvcache path
go with the disk tier, and the retired env keys are no longer required (they
would abort ExecStart). autoresearch-frank-archon.sh takes the same CLI, with
a cold prefill now coming from --prefix-cache-mib 0 instead of a per-run disk
directory. run-swift15-bf16.sh is untouched: it pins its own engines by design.

tests: the two budget-arithmetic seams asserted knobs that no longer exist and
are replaced by one seam that asserts the retired knobs are ABSENT from env and
argv - a stale line is a boot refusal, not a no-op. The container-contract seam
now asserts the tune mount and the absence of /data.
This commit is contained in:
Frank committed 2026-10-19 10:18:24 -07:00
1 parent b9d21ae947
commit 9de58ed6fd
4 files changed
+117 -118
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/frank_archon_unit.bats
+42 -60
View File
@@ -9,9 +9,9 @@
# rollback). It replaces the FreeToken seam of
# 2026-10-05 (git tag frank-archon-freetoken-pr587-20261005) and pins what
# flags alone do not say: the served window clears the harness worst case,
# the KV pool backs seqs x max-model-len at the declared cache dtype (an
# engine HARD limit, not a ratio), the per-card VRAM pair stays inside the
# the pinned-host ledger has a ttm pages cap floor, the serve script ENFORCES
# the retired per-card budget knobs are ABSENT (the pinned engine refuses
# them at ExecStart), the pinned-host ledger has a ttm pages cap floor, the
# serve script ENFORCES
# it and RAISES the knob itself when a reboot reset it (the cap's default is
# half of RAM), the served window is what models.yml must
# quote, the host-networking contract (no --publish), the image gate on a
@@ -146,20 +146,23 @@ nosudo_bin() {
grep -q '^FRANK_ARCHON_TP=2$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_TP_WIRE=wht6$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_SPEC_TOKENS=2$' "$ENV_FILE"
# expert_tiered placement: vram_weights + the pinned host pool serve the
# per-rank resident set TOGETHER, so cutting the VRAM slab to pay for the
# KV pool moves the pool with it. KV is QUANTIZED fp8 (the engine takes
# only bf16|fp8 — no per-tensor K/V split), the PLE n-gram table is pinned
# in host RAM per Frank's law.
# expert_tiered placement: the pinned host pool plus whatever expert VRAM the
# budget resolver leaves serve the per-rank resident set TOGETHER, so the pool
# size is also the lever on which experts stay on the card. KV is QUANTIZED
# fp8 (the engine takes only bf16|fp8 — no per-tensor K/V split), the PLE
# n-gram table is pinned in host RAM per Frank's law. The per-card VRAM pair
# is NOT asserted here: 1.3.0 removed it and the resolver derives the split,
# so the pool's token count is a boot-ledger line — verified live by the
# capacity leg of scripts/probe-frank-archon-kv.py at every cutover.
grep -q '^FRANK_ARCHON_PLACEMENT=expert_tiered$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_HOST_POOL_MIB=44928$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_VRAM_WEIGHTS_MIB=[1-9]' "$ENV_FILE"
grep -q '^FRANK_ARCHON_VRAM_KV_MIB=[1-9]' "$ENV_FILE"
grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE"
# Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
grep -q '^FRANK_ARCHON_PREFIX_HOST_MIB=8192$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_PREFIX_DISK_MIB=65536$' "$ENV_FILE"
# Prefix cache: pinned host RAM only (the disk tier was removed), and the
# step-sharing cap is stated explicitly — auto vs off is the difference
# between decode waiting one chunk and waiting a whole step.
grep -q '^FRANK_ARCHON_PREFIX_CACHE_MIB=8192$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto$' "$ENV_FILE"
# The pinned ledger (2 x pool + 47.7 GiB PLE + prefix host) must be
# under the ttm pages cap, and the env states the floor the serve
# script enforces (>= 140,000 MiB — the kernel's own default cap, half of
@@ -170,7 +173,7 @@ nosudo_bin() {
ttmt=$(awk -F= '/^FRANK_ARCHON_TTM_TARGET_MIB=/{print $2}' "$ENV_FILE")
[ "$ttmt" -gt "$ttmm" ]
pool=$(awk -F= '/^FRANK_ARCHON_HOST_POOL_MIB=/{print $2}' "$ENV_FILE")
ph=$(awk -F= '/^FRANK_ARCHON_PREFIX_HOST_MIB=/{print $2}' "$ENV_FILE")
ph=$(awk -F= '/^FRANK_ARCHON_PREFIX_CACHE_MIB=/{print $2}' "$ENV_FILE")
# 2 pools + PLE 47.7 GiB (48,824 MiB) + prefix host, rounded up.
ledger=$(( 2 * pool + 48824 + ph + 2048 ))
[ "$ttmm" -ge "$ledger" ]
@@ -179,47 +182,21 @@ nosudo_bin() {
[ "$(awk -F= '/^FRANK_ARCHON_RAD_MIN_MIB=/{print $2}' "$ENV_FILE")" -gt 0 ]
}
@test "the KV pool backs seqs x max-model-len at the declared cache dtype" {
# The engine's own hard limit: a request longer than the pool's token
# count is "accepted by the API and then refused as too large to ever fit
# -- this is a hard limit, not a backing ratio". Rates are measured, not
# derived: per rank, kv_attn (13 layers) pages 26.00 KiB per 4 tokens at
# fp8 and 52.00 KiB at bf16, while kv_qsa_bkey (13 layers) pages 3.25 KiB
# at BOTH — the QSA block key is an index, not attention state, and does
# not follow --kv-cache-dtype — so the rate is 7,488 B/token at fp8 and
# 14,144 at bf16 (1.89x, not 2x). The dtype-independent terms also live in
# the pool: ~110 MiB a sequence of recurrent state (kv_gdn_state 54 MiB x 2
# copies + conv/tail/ple) and 448 MiB of prefix checkpoints (8 x 55.99).
# Verified against BOTH live boots: fp8 3543-330-448 = 2765 MiB -> 387,240
# tokens (ledger 386,904); bf16 6912-220-448 = 6244 MiB -> 462,946
# (ledger 462,752). Both within 0.1%.
kv=$(awk -F= '/^FRANK_ARCHON_VRAM_KV_MIB=/{print $2}' "$ENV_FILE")
seqs=$(awk -F= '/^FRANK_ARCHON_MAX_SEQS=/{print $2}' "$ENV_FILE")
maxlen=$(awk -F= '/^FRANK_ARCHON_MAX_MODEL_LEN=/{print $2}' "$ENV_FILE")
case "$(awk -F= '/^FRANK_ARCHON_KV_DTYPE=/{print $2}' "$ENV_FILE")" in
fp8) bpt=7488 ;;
bf16) bpt=14144 ;;
*) fail "unknown FRANK_ARCHON_KV_DTYPE" ;;
esac
tokens=$(( (kv - 110 * seqs - 448) * 1048576 / bpt ))
[ "$tokens" -ge $(( seqs * maxlen )) ]
}
@test "the per-card VRAM pair stays inside the measured card ceiling" {
# The engine's OWN budget table, not a hand-picked reserve. Boot of
# 2026-10-13 at the pair 24000/5120: card 32,624 MiB (31.86 GiB), budgets
# claimed 29,120, `3.42 GiB is neither budgeted nor accounted — the
# driver's context, the activation arena and --gpu-headroom-mib come out of
# it; anything past those is throughput on the floor`, and `card 0 settled
# at 2.01 GiB free against 209.19 MiB (the 96.00 MiB --gpu-headroom-mib and
# 113.19 MiB kept for the static region)`. Claimable reserve above the
# engine's floor: 2,058 - 209 = 1,849 MiB, so the ceiling is 30,969 MiB.
# The 2026-10-11 bound of 29,120 ("keep >= 1 GiB free", card read as 31,856
# MiB) was MY reserve, which the engine never asked for; the shipped pair
# spends the difference and lands ~458 MiB above the floor.
w=$(awk -F= '/^FRANK_ARCHON_VRAM_WEIGHTS_MIB=/{print $2}' "$ENV_FILE")
kv=$(awk -F= '/^FRANK_ARCHON_VRAM_KV_MIB=/{print $2}' "$ENV_FILE")
[ $(( w + kv )) -le 30969 ]
@test "the retired budget knobs are absent from the env and the argv" {
# Not unused leftovers: the pinned engine REFUSES to start on any of them
# (--vram-weights-mib/--vram-kv-mib went away at 1.3.0, the prefix-cache disk
# tier outright), and a unit that refuses at ExecStart sits in
# `activating (auto-restart)` while the port still looks alive from whatever
# held it last. A stale line here is an outage, so it is asserted away.
for key in FRANK_ARCHON_VRAM_WEIGHTS_MIB FRANK_ARCHON_VRAM_KV_MIB \
FRANK_ARCHON_PREFIX_HOST_MIB FRANK_ARCHON_PREFIX_DISK_MIB; do
! grep -q "^${key}=" "$ENV_FILE"
! grep -q "$key" "$SERVE"
done
for flag in --vram-weights-mib --vram-kv-mib --prefix-cache-host-mib \
--prefix-cache-dir --prefix-cache-disk-mib; do
! grep -q -- "$flag" "$SERVE"
done
}
@test "the serve script wires the engine flags from the env" {
@@ -232,11 +209,13 @@ nosudo_bin() {
grep -q -- '--max-model-len "$FRANK_ARCHON_MAX_MODEL_LEN"' "$SERVE"
grep -q -- '--placement "$FRANK_ARCHON_PLACEMENT"' "$SERVE"
grep -q -- '--host-pool-mib "$FRANK_ARCHON_HOST_POOL_MIB"' "$SERVE"
grep -q -- '--vram-weights-mib "$FRANK_ARCHON_VRAM_WEIGHTS_MIB"' "$SERVE"
grep -q -- '--vram-kv-mib "$FRANK_ARCHON_VRAM_KV_MIB"' "$SERVE"
grep -q -- '--kv-cache-dtype "$FRANK_ARCHON_KV_DTYPE"' "$SERVE"
grep -q -- '--prefix-cache-host-mib "$FRANK_ARCHON_PREFIX_HOST_MIB"' "$SERVE"
grep -q -- '--prefix-cache-disk-mib "$FRANK_ARCHON_PREFIX_DISK_MIB"' "$SERVE"
grep -q -- '--prefix-cache-mib "$FRANK_ARCHON_PREFIX_CACHE_MIB"' "$SERVE"
grep -q -- '--busy-prefill-chunk "$FRANK_ARCHON_BUSY_PREFILL_CHUNK"' "$SERVE"
# Interleave is opt-in: it reaches argv only through the guarded array, so a
# 0 in the env leaves the engine's own default in place.
grep -q -- 'PREFILL_ARGS=(--prefill-interleave "$FRANK_ARCHON_PREFILL_INTERLEAVE")' "$SERVE"
grep -qF -- '"${PREFILL_ARGS[@]}"' "$SERVE"
grep -q -- '--ngram-placement "$FRANK_ARCHON_NGRAM_PLACEMENT"' "$SERVE"
grep -q -- '--num-speculative-tokens "$FRANK_ARCHON_SPEC_TOKENS"' "$SERVE"
# The boot-ladder escape hatch is opt-in: unset renders NO extra argv.
@@ -261,7 +240,10 @@ nosudo_bin() {
grep -q -- '--device /dev/kfd' "$SERVE"
grep -q -- '--device /dev/dri' "$SERVE"
grep -q -- '-v "$FRANK_ARCHON_MODEL_DIR":/models:ro' "$SERVE"
grep -q -- '-v "$FRANK_ARCHON_STATE_DIR":/data' "$SERVE"
grep -q -- '-v "$FRANK_ARCHON_STATE_DIR/tune":/opt/radiance/share/radiance/tune' "$SERVE"
# No /data mount: the prefix cache lives in pinned host RAM now, so the only
# host state the container writes is the rad-tune cache above.
! grep -q -- ':/data ' "$SERVE"
grep -q -- '-v "$FRANK_ARCHON_TEMPLATE":/templates/template.jinja:ro' "$SERVE"
}