From 96c5a0625ccf892903bf9cabb6b85b62e2f0496b Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 27 Oct 2026 21:54:19 -0700 Subject: [PATCH 1/2] fix(qwen38): the rig's card split is inherited, not invented MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit run-qwen38-bf16.sh carried its own --host-pool-mib 12288 / --gpu-headroom-mib 768, copied from the w4-era swift15 rig. The engine derives the whole per-card split from those two numbers, and the KV cache's funding is `elastic - expert_floor` where the expert floor is a function of the pinned host pool — so the pool buys cache as well as residency. The first kldref boot got a 270 MiB / 7,436-position pool and the recorder refused all 197 corpus documents ("document 0 is 10554 tokens and the KV pool holds 7436"), exiting 1. The rig now sources FRANK_ARCHON_HOST_POOL_MIB / FRANK_ARCHON_GPU_HEADROOM_MIB from scripts/frank-archon.env, the file that owns them (49152 / 96), with `:?` guards, and the header records the law instead of the claim that the pool never touches logits. tests/qwen38_reference.bats pins the shape verbatim, so its SHAPE contract moves with it; the suite is green (558 tests). --- scripts/run-qwen38-bf16.sh | 45 ++++++++++++++++++++++++------------- tests/qwen38_reference.bats | 2 +- 2 files changed, 30 insertions(+), 17 deletions(-) diff --git a/scripts/run-qwen38-bf16.sh b/scripts/run-qwen38-bf16.sh index 5568ef4..52b3968 100755 --- a/scripts/run-qwen38-bf16.sh +++ b/scripts/run-qwen38-bf16.sh @@ -39,25 +39,33 @@ # speculation, fp8 KV, a 32,768-token window, the PLE n-gram table pinned to RAM, # and the production chat template rather than the one baked into the container — # every scored container is asked the same tokenised questions the reference -# answered. The window is 32,768 because this corpus is long: measured on -# Dankserver 2026-10-27, kld.jsonl is 197 records with a longest of 38,263 -# characters (longest prompt FIELD 36,855, median record 27,228), which is ~9,600 -# tokens at the pessimistic chars/4 and ~7,700 at the ratio this repo actually -# measured on the K3 corpus (380,143 chars = 76,568 tokens, chars/4.97). A 16,384 -# window cleared that by 1.7x; 32,768 clears every record of both corpora by at -# least 2.6x, and the kldref gate below re-checks it against the file on disk. +# answered. The window is 32,768: kld.jsonl is 197 records with a longest of +# 38,263 characters (longest prompt FIELD 36,855, median record 27,228). Its +# token length — 10,554 for that longest record, counted by the engine itself — +# is in the funding paragraph below; the kldref gate re-checks the file against +# the window before it boots anything. # # NO --vram-kv-mib: it was REMOVED at 1.3.0 and naming it is a boot refusal, not # a no-op — verified on the pinned image 2026-10-27: "E --vram-kv-mib was # removed: the card is split between the weights and the cache from what the -# model needs." The budget resolver derives the per-card split from -# --host-pool-mib and --gpu-headroom-mib, and those two are deliberately NOT -# production's 49,152 / 1,536 (frank-archon.env:77): the pool decides WHICH -# experts stay resident, so it buys speed and never logits — the shape the gate -# has to hold identical is the kv dtype, the slot count, the step size, the spec -# depth and the window. It is also the only figure that fits: a 335 GiB bf16 -# container cannot be booted against a 49 GiB resident set on a 192 GiB machine, -# and a prefill-only pass does not care that the disk tier paces at ~3 GiB/s. +# model needs." The budget resolver derives that split from --host-pool-mib and +# --gpu-headroom-mib, and this rig inherits BOTH from frank-archon.env, the file +# that owns them: the cache's funding is `elastic - expert_floor` and the expert +# floor falls one MiB for every MiB of host pool (frank-archon.md, Laws), so the +# pool buys KV as well as residency — a private pair is not a free knob. What +# buying one cost, measured on the first kldref boot the same day with the w4 +# era's 12288 / 768: "I kv: the pool backs 6% of the worst case (4 sequences all +# at 32768 tokens), 0.2 full-length sessions; a larger --host-pool-mib gives the +# cache more" — a 270 MiB, 7,436-position cache, after which the recorder refused +# every one of the 197 corpus documents ("E kld: document 0 is 10554 tokens and +# the KV pool holds 7436, so it can never be scored") and exited 1. Production's +# 49,152 per rank is affordable exactly when the rig runs: the rig stops +# frank-archon first, and that stop frees the same 96 GiB the service was pinning. +# +# The 32,768-token window it pairs with is not up for grabs: the corpus's longest +# prompt is 10,554 tokens by the engine's own count (the header's chars/4.97 +# estimate called that record ~7,700; the true ratio for this file is chars/3.6), +# so the window clears it by 3x and the cache now funds it. # # The step size is ONE value for the recording and every scoring pass: # FRANK_QWEN38_MNBT, default 2,048. The gate's whole claim is that the reference, @@ -148,6 +156,11 @@ KLD_REF_DIR="$CAMPAIGN/kld-ref" CALIB_DIR="$CAMPAIGN/calib" C=qwen38-bf16 SERVICE=frank-archon +# The card split, taken from the file that owns it instead of a private pair: +# --host-pool-mib also funds the KV cache (see the header), so hardcoding it +# here once starved the recorder. The `:?` guards keep an empty env fatal. +HOST_POOL_MIB="${FRANK_ARCHON_HOST_POOL_MIB:?FRANK_ARCHON_HOST_POOL_MIB must be set}" +GPU_HEADROOM_MIB="${FRANK_ARCHON_GPU_HEADROOM_MIB:?FRANK_ARCHON_GPU_HEADROOM_MIB must be set}" # ONE step size for both logits modes. The file tier streams only at <=2,048 and # the gate scores file-tier payloads (the served iq4r, the arms) against the @@ -201,7 +214,7 @@ fi # The measuring shape. Identical for kldref and score by construction — this # array is the only place the campaign's KL numbers are defined. ARGS=(--tp 2 --placement expert_tiered --weights-disk-tier - --host-pool-mib 12288 --gpu-headroom-mib 768 + --host-pool-mib "$HOST_POOL_MIB" --gpu-headroom-mib "$GPU_HEADROOM_MIB" --deterministic --num-speculative-tokens 0 --kv-cache-dtype fp8 --max-model-len "$MAX_MODEL_LEN" --ngram-placement ram diff --git a/tests/qwen38_reference.bats b/tests/qwen38_reference.bats index b999fb0..d60f138 100755 --- a/tests/qwen38_reference.bats +++ b/tests/qwen38_reference.bats @@ -29,7 +29,7 @@ RUNNER="$REPO_ROOT/scripts/run-qwen38-bf16.sh" # The reference shape, spelled exactly as run-qwen38-bf16.sh must pass it: this # is the contract the recorded reference is measured at, so a silent edit here # is a silent edit to every KLD number in the campaign. -SHAPE="--tp 2 --placement expert_tiered --weights-disk-tier --host-pool-mib 12288 --gpu-headroom-mib 768 --deterministic --num-speculative-tokens 0 --kv-cache-dtype fp8 --max-model-len 32768 --ngram-placement ram --override-chat-template /models/tpl.jinja" +SHAPE="--tp 2 --placement expert_tiered --weights-disk-tier --host-pool-mib 49152 --gpu-headroom-mib 96 --deterministic --num-speculative-tokens 0 --kv-cache-dtype fp8 --max-model-len 32768 --ngram-placement ram --override-chat-template /models/tpl.jinja" # oneline — both dry runs print one shell-quoted argument a line; joining them # lets an option and its value be asserted as the pair they are. From 1c56dea67a6559b15c9bdaacfb5712c646634d94 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 27 Oct 2026 22:19:51 -0700 Subject: [PATCH 2/2] =?UTF-8?q?fix(qwen38):=20record=20at=20MNBT=201,024?= =?UTF-8?q?=20=E2=80=94=20the=20step=20size=20the=20cache=20can=20actually?= =?UTF-8?q?=20pay=20for?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The rerun after 96c5a06 refused every corpus document again ("document 196 is 8507 tokens and the KV pool holds 7436"), and the boot table shows why the host-pool change could not have helped: vram_budget.cpp put the container in its `no lender` branch — elastic 25.56 GiB, experts 25.29 GiB (their plan-fixed share), kv cache 271.25 MiB — identical at 12,288 and at 49,152 MiB of host pool. The only supplier left there is the activation arena, and the engine says so at vram_budget.cpp:117: "lower --max-num-batched-tokens (the arena scales with it)". The log prices both rungs directly (a 2,048-token step costs 1.08 GiB, a 1,024-token step 504.39 MiB), so the default moves to 1,024: the cache grows to 866 MiB / 24,300 tokens and scores every one of the 197 documents (longest 11,970). Calibration keeps its wide 65,536; it was never tied to MNBT. This also retracts the mechanism 96c5a06 recorded in the header ("the pool buys KV as well as residency"). Inheriting the env's 49,152 / 96 stays right as single-source-of-truth, but on this payload the pool is not the cache's supplier. --- scripts/run-qwen38-bf16.sh | 68 ++++++++++++++++++++------------------ 1 file changed, 36 insertions(+), 32 deletions(-) diff --git a/scripts/run-qwen38-bf16.sh b/scripts/run-qwen38-bf16.sh index 52b3968..eafd40d 100755 --- a/scripts/run-qwen38-bf16.sh +++ b/scripts/run-qwen38-bf16.sh @@ -48,36 +48,36 @@ # NO --vram-kv-mib: it was REMOVED at 1.3.0 and naming it is a boot refusal, not # a no-op — verified on the pinned image 2026-10-27: "E --vram-kv-mib was # removed: the card is split between the weights and the cache from what the -# model needs." The budget resolver derives that split from --host-pool-mib and -# --gpu-headroom-mib, and this rig inherits BOTH from frank-archon.env, the file -# that owns them: the cache's funding is `elastic - expert_floor` and the expert -# floor falls one MiB for every MiB of host pool (frank-archon.md, Laws), so the -# pool buys KV as well as residency — a private pair is not a free knob. What -# buying one cost, measured on the first kldref boot the same day with the w4 -# era's 12288 / 768: "I kv: the pool backs 6% of the worst case (4 sequences all -# at 32768 tokens), 0.2 full-length sessions; a larger --host-pool-mib gives the -# cache more" — a 270 MiB, 7,436-position cache, after which the recorder refused -# every one of the 197 corpus documents ("E kld: document 0 is 10554 tokens and -# the KV pool holds 7436, so it can never be scored") and exited 1. Production's -# 49,152 per rank is affordable exactly when the rig runs: the rig stops -# frank-archon first, and that stop frees the same 96 GiB the service was pinning. +# model needs." The resolver derives the split from --host-pool-mib, +# --gpu-headroom-mib and the activation arena; the rig inherits the first two from +# frank-archon.env, the file that owns them. On THIS payload the host pool is not +# the cache's supplier: the boot lands in vram_budget.cpp's `no lender` branch, +# where the experts keep their plan-fixed 25.29 GiB and the cache gets the residue. +# Measured twice — 12,288 and 49,152 MiB of host pool both printed the same +# "kv cache 271.25 MiB ... 7,436"; only --gpu-headroom-mib moved anything +# (768 -> 96 grew vram_weights 29.41 -> 30.06 GiB). The supplier the source names +# is the arena: vram_budget.cpp:117, "lower --max-num-batched-tokens (the arena +# scales with it)" — which is what the step size below now does. # -# The 32,768-token window it pairs with is not up for grabs: the corpus's longest -# prompt is 10,554 tokens by the engine's own count (the header's chars/4.97 -# estimate called that record ~7,700; the true ratio for this file is chars/3.6), -# so the window clears it by 3x and the cache now funds it. +# The pool, not the window, decides what the recorder accepts: it refuses any +# document longer than the pool, so the 32,768-token window was decorative while the +# cache held 7,436 — "E kld: document 196 is 8507 tokens and the KV pool holds +# 7436, so it can never be scored" on all 197 records, exit 1. The longest record +# is 10,554 tokens by the engine's own count (chars/3.6 for this file, not the +# chars/4.97 this repo measured on the K3 corpus). # # The step size is ONE value for the recording and every scoring pass: -# FRANK_QWEN38_MNBT, default 2,048. The gate's whole claim is that the reference, +# FRANK_QWEN38_MNBT, default 1,024. The gate's whole claim is that the reference, # the served iq4r and every arm rode one shape, so the ceiling is set by the # TIGHTEST payload the gate boots rather than by the bf16 reference: the quantised # expert FILE tiers (the served iq4r, the arms) stream only at MNBT<=2048 and -# refuse the widened form above it, and 2,048 is what production holds anyway. -# Recording the reference at 32,768 would therefore measure a shape the payloads -# cannot reproduce; at 2,048 the 50,432 positions cost more steps, which is still -# one drained window. Calibration keeps the wide 65,536 — grams are not logits, -# nothing downstream is compared against a step size, and that is where a big -# step pays (a disk-tier step costs the same at 1 token as at 64k). +# refuse the widened form above it. 2,048 was still the wrong default on this +# payload, because the arena the engine charges for a step comes straight out of +# the cache — 1.08 GiB at a 2,048-token step, 504.39 MiB at a 1,024-token one, and +# the experts keep their share either way. At 1,024 the cache is 866 MiB: 24,300 +# tokens, which scores every one of the 197 documents (longest 11,970). +# Calibration keeps the wide 65,536 — grams are not logits, nothing downstream is +# compared against a step size, and that is where a big step pays. # # usage: scripts/run-qwen38-bf16.sh kldref|calib|score RAD OUT [--dry-run] [--image REF] # (RAD and OUT are score's two positionals: RAD is a container under the models @@ -94,8 +94,9 @@ # bf16 container, kldref only), FRANK_QWEN38_CAMPAIGN (holds kld-ref/ and # calib/), FRANK_QWEN38_KLD_CORPUS, FRANK_QWEN38_TPL (the template copy # the container reads through /models), FRANK_QWEN38_LOG_DIR, and -# FRANK_QWEN38_MNBT — the step size shared by kldref and score (2,048 is -# the file tier's ceiling and production's own value, four slots and all), +# FRANK_QWEN38_MNBT — the step size shared by kldref and score (default +# 1,024: the file tier's ceiling is 2,048 and production runs there, but on +# this payload the arena charged for a wider step is taken from the cache), # and FRANK_QWEN38_MAX_MODEL_LEN — the window every mode boots at (32,768; # see the corpus census above, and raise it together with the corpus). # kldref refuses (exit 2) when the longest record of the corpus it is about to @@ -164,9 +165,11 @@ GPU_HEADROOM_MIB="${FRANK_ARCHON_GPU_HEADROOM_MIB:?FRANK_ARCHON_GPU_HEADROOM_MIB # ONE step size for both logits modes. The file tier streams only at <=2,048 and # the gate scores file-tier payloads (the served iq4r, the arms) against the -# reference, so the reference is recorded at the ceiling the payloads can meet — -# never two numbers that a KL between them would have to forgive. -MNBT="${FRANK_QWEN38_MNBT:-2048}" +# reference, so the reference is recorded at a step the payloads can meet — never +# two numbers that a KL between them would have to forgive. 1,024 rather than the +# 2,048 ceiling: on this payload the arena charged for the step is the cache's only +# remaining supplier, and 1,024 is the rung that funds every corpus document. +MNBT="${FRANK_QWEN38_MNBT:-1024}" if ! [[ "$MNBT" =~ ^[0-9]+$ ]]; then echo "REFUSED: FRANK_QWEN38_MNBT must be a token count (got: $MNBT)" >&2 exit 2 @@ -223,9 +226,10 @@ MOUNTS=(-v "$MODELS:/models:ro") TAPENV=() case "$MODE" in kldref) - # Recording is prefill-only over four slots and the shared step, which is - # production's own operating point (frank-archon.env: FRANK_ARCHON_MAX_SEQS=4 - # with FRANK_ARCHON_MAX_BATCHED_TOKENS=2048). Corpus read-only, record dir rw. + # Recording is prefill-only over four slots at the campaign's single step size + # — 1,024, not production's 2,048 (frank-archon.env: FRANK_ARCHON_MAX_SEQS=4, + # FRANK_ARCHON_MAX_BATCHED_TOKENS=2048): the wider step costs the cache the + # room the corpus needs. Corpus read-only, record dir rw. ARGS+=(--model "$MODEL_IN" --kld-record /data/kld-ref --kld-corpus /data/kld.jsonl --max-num-seqs 4 --max-num-batched-tokens "$MNBT")