dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

llm(frank-archon): unquantized bf16 KV at 2 x 200,000

Browse Source 
The KV pool at the fp8 shape backed 386,904 tokens — 1.93 sessions of the
served 200,000 window, so a second full-height conversation preempted to the
host/disk prefix tiers. Moving the cache to bf16 (the engine's only
unquantized rung; `--kv-cache-dtype` takes bf16|fp8 and has no per-tensor
K/V split) doubles what a token costs: 7.3125 -> 14.625 KiB per rank
(kv_attn 26.00 -> 52.00 KiB per 4-token page + kv_qsa_bkey 3.25 -> 6.50).

Pool 3543 -> 6912 MiB (437k tokens, >= 2 x 200,000), and the weights slab
pays for all of it: 26071 -> 22208 MiB, the pair pinned to the measured
card ceiling 29,120 MiB (card 31,856 - 1,710 consumed outside both budgets
- 1,024 free). The resident set that leaves VRAM spills into the pinned host
pool, so 40960 -> 44928 MiB, and the prefix host tier doubles with the entry
size (58.50 -> 117 KiB): 4096 -> 8192 MiB. Pinned ledger ~147 GiB, ttm floor
140000 -> 150000 (cap 200 GiB).

--max-num-seqs 3 -> 2: two slots x the window is the capacity contract, and
the "seqs 2 cannot boot" law is the retired tcclaviger fork's torch.compile
query_start_loc constraint, not radiance's (radiance booted seqs 2 in the
fp8-native era; the 2026-10-06 sweep had 2 and 3 tied inside +-3%). Bonus:
the fused all-reduce is now declared over 2 x 3 = 6 rows, under the 25-row
bound at which the wht6 wire is bit-exact.

Cost, paid twice on this engine: idle KV VRAM is also the expert cache's
spillover (lends 2.69 GiB + 740 MiB of arena), so the pool steals weight
residency from the slab AND the lendable cache. Priors: 2.14 GiB of stolen
weights cost 2.5% decode (MAX_MODEL_LEN 262144), the EVC ladder's other
direction cost 8.9%. Prediction -3..-8% vs the fp8 control (136 t/s at 44.5k,
acceptance 0.7246); the deploy boot ledger and live probes are the referee.

Seams: MAX_SEQS/HOST_POOL/KV_DTYPE/PREFIX_HOST literals follow the shape, and
two new invariants guard it — the KV pool must back seqs x max-model-len at
the declared dtype (an engine hard limit: an over-long request is accepted
then refused as too large to ever fit; the formula reproduces the fp8 boot's
386,904 to within 0.1%), and the VRAM pair must stay inside the ceiling.
This commit is contained in:
Frank
2026-10-11 01:01:58 -07:00
parent 394fd8535b
commit 0bf26b0741
3 changed files with 162 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/frank_archon_unit.bats
+69 -26
View File
@@ -2,21 +2,23 @@
# Seam: frank-archon — the Dankserver LLM endpoint on :7770.
#
# This file asserts the RADIANCE shape (StillDeadcode engine, digest-pinned
# stilldeadcode/radiance image, BOTH R9700s TP=2 on the wht6 wire, the
# OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 at native block-FP8 precision in a
# rad-convert container). It replaces the FreeToken seam of 2026-10-05
# (git tag frank-archon-freetoken-pr587-20261005) and pins what flags
# alone do not say: the served window clears the harness worst case, the
# pinned-host ledger has a ttm pages cap floor and the serve script
# ENFORCES it, the served window is what models.yml must quote, the host-
# networking contract (no --publish), the image gate on a name@sha256 ref,
# the .rad payload gate with its size floor, the chat-template override
# mount, the self-healing law (the run returns before the engine is up, so
# the serve script must wait on the container for Restart=always to mean
# anything), the deployer's false-green MainPID check, the vendored hfget
# fetch path (no token for this ungated repo), and the client contract
# (reasoning auto -> reasoning_content; models.yml maps the field on the
# harness side, not here).
# stilldeadcode/radiance image, BOTH R9700s TP=2 on the wht6 wire, serving
# the Swift1.5 Qwen3.8-Flash-Next GPTQ-w4 .rad built by
# scripts/convert-swift15-w4dn8.sh). It replaces the FreeToken seam of
# 2026-10-05 (git tag frank-archon-freetoken-pr587-20261005) and pins what
# flags alone do not say: the served window clears the harness worst case,
# the KV pool backs seqs x max-model-len at the declared cache dtype (an
# engine HARD limit, not a ratio), the per-card VRAM pair stays inside the
# measured card ceiling, the pinned-host ledger has a ttm pages cap floor and
# the serve script ENFORCES it, the served window is what models.yml must
# quote, the host-networking contract (no --publish), the image gate on a
# name@sha256 ref, the .rad payload gate with its size floor, the
# chat-template override mount, the self-healing law (the run returns before
# the engine is up, so the serve script must wait on the container for
# Restart=always to mean anything), the deployer's false-green MainPID check,
# the vendored hfget fetch path (no token for this ungated repo), and the
# client contract (reasoning auto -> reasoning_content; models.yml maps the
# field on the harness side, not here).
load helpers
@@ -118,25 +120,30 @@ TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
grep -q '^FRANK_ARCHON_MAX_MODEL_LEN=200000$' "$ENV_FILE"
ml=$(awk -F= '/^FRANK_ARCHON_MAX_MODEL_LEN=/{print $2}' "$ENV_FILE")
[ "$ml" -ge 177356 ]
# Decode-sweep shape (2026-10-06): 3 slots, spec depth 2 — the measured
# optimum (33.06 t/s). The fused TP all-reduce is declared over
# seqs x (1 + spec) rows = 9.
grep -q '^FRANK_ARCHON_MAX_SEQS=3$' "$ENV_FILE"
# Concurrency: 2 slots x the served window is the capacity contract, and
# the fused TP all-reduce is declared over seqs x (1 + spec) = 6 rows,
# under the 25-row bound at which the wht6 wire stays bit-exact. The
# "seqs 2 cannot boot" law belongs to the retired tcclaviger vLLM fork
# (torch.compile query_start_loc constraint, 2026-10-02); radiance booted
# seqs 2 in the fp8-native era and the 2026-10-06 sweep had seqs 2 and 3
# tied inside the +-3% noise floor.
grep -q '^FRANK_ARCHON_MAX_SEQS=2$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_TP=2$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_TP_WIRE=wht6$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_SPEC_TOKENS=2$' "$ENV_FILE"
# Native-precision economics: the expert pool is per rank (56.4 GiB
# slice per rank at TP2), the PLE ngram table is pinned in host RAM per
# Frank's law, KV is fp8, the ratio is the publisher's production
# value.
# expert_tiered placement: vram_weights + the pinned host pool serve the
# per-rank resident set TOGETHER, so cutting the VRAM slab to pay for the
# KV pool moves the pool with it. KV is UNQUANTIZED bf16 (the engine takes
# only bf16|fp8 — no per-tensor K/V split), the PLE n-gram table is pinned
# in host RAM per Frank's law.
grep -q '^FRANK_ARCHON_PLACEMENT=expert_tiered$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_HOST_POOL_MIB=40960$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_HOST_POOL_MIB=44928$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_VRAM_WEIGHTS_MIB=[1-9]' "$ENV_FILE"
grep -q '^FRANK_ARCHON_VRAM_KV_MIB=[1-9]' "$ENV_FILE"
grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_KV_DTYPE=bf16$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE"
# Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
grep -q '^FRANK_ARCHON_PREFIX_HOST_MIB=4096$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_PREFIX_HOST_MIB=8192$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_PREFIX_DISK_MIB=65536$' "$ENV_FILE"
# The pinned ledger (2 x pool + 47.7 GiB PLE + prefix host) must be
# under the ttm pages cap, and the env states the floor the serve
@@ -154,6 +161,42 @@ TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
[ "$(awk -F= '/^FRANK_ARCHON_RAD_MIN_MIB=/{print $2}' "$ENV_FILE")" -gt 0 ]
}
@test "the KV pool backs seqs x max-model-len at the declared cache dtype" {
# The engine's own hard limit: a request longer than the pool's token
# count is "accepted by the API and then refused as too large to ever fit
# -- this is a hard limit, not a backing ratio". Rates measured from the
# live boot ledger's group table, per rank: kv_attn (13 layers) is a
# 26.00 KiB page per 4 tokens at fp8 and kv_qsa_bkey (13 layers) 3.25 KiB
# -> 7,488 B/token, and bf16 doubles both -> 14,976 B/token. The terms
# that do NOT follow the dtype also live in the pool: ~110 MiB a sequence
# of recurrent state (kv_gdn_state 54 MiB x 2 copies + conv/tail/ple) and
# 448 MiB of prefix-cache checkpoints (8 x 55.99 MiB). Verified against
# the fp8 boot it replaces: 3543 - 330 - 448 = 2765 MiB -> 387,240 tokens,
# ledger said 386,904.
kv=$(awk -F= '/^FRANK_ARCHON_VRAM_KV_MIB=/{print $2}' "$ENV_FILE")
seqs=$(awk -F= '/^FRANK_ARCHON_MAX_SEQS=/{print $2}' "$ENV_FILE")
maxlen=$(awk -F= '/^FRANK_ARCHON_MAX_MODEL_LEN=/{print $2}' "$ENV_FILE")
case "$(awk -F= '/^FRANK_ARCHON_KV_DTYPE=/{print $2}' "$ENV_FILE")" in
fp8) bpt=7488 ;;
bf16) bpt=14976 ;;
*) fail "unknown FRANK_ARCHON_KV_DTYPE" ;;
esac
tokens=$(( (kv - 110 * seqs - 448) * 1048576 / bpt ))
[ "$tokens" -ge $(( seqs * maxlen )) ]
}
@test "the per-card VRAM pair stays inside the measured card ceiling" {
# Measured on Dankserver 2026-10-11: card 31,856 MiB, a boot consumes
# 1,710 MiB OUTSIDE both budgets (driver context, the 990 MiB activation
# arena, hardware queues, headroom, step staging) and the fp8 pair settled
# at 1,379 MiB free. Claiming more than 29,120 MiB in total is an
# allocation failure at graph capture, not a warning — so the KV pool and
# the weights slab are one budget, and this seam is the guard.
w=$(awk -F= '/^FRANK_ARCHON_VRAM_WEIGHTS_MIB=/{print $2}' "$ENV_FILE")
kv=$(awk -F= '/^FRANK_ARCHON_VRAM_KV_MIB=/{print $2}' "$ENV_FILE")
[ $(( w + kv )) -le 29120 ]
}
@test "the serve script wires the engine flags from the env" {
grep -q -- '--model "/models/$FRANK_ARCHON_MODEL_FILE"' "$SERVE"
grep -q -- '--served-model-name "$FRANK_ARCHON_SERVED_NAME"' "$SERVE"