fix(qwen38): record at MNBT 1,024 — the step size the cache can actually pay for

The rerun after 96c5a06 refused every corpus document again ("document 196 is
8507 tokens and the KV pool holds 7436"), and the boot table shows why the
host-pool change could not have helped: vram_budget.cpp put the container in its
`no lender` branch — elastic 25.56 GiB, experts 25.29 GiB (their plan-fixed
share), kv cache 271.25 MiB — identical at 12,288 and at 49,152 MiB of host
pool. The only supplier left there is the activation arena, and the engine says
so at vram_budget.cpp:117: "lower --max-num-batched-tokens (the arena scales
with it)". The log prices both rungs directly (a 2,048-token step costs 1.08
GiB, a 1,024-token step 504.39 MiB), so the default moves to 1,024: the cache
grows to 866 MiB / 24,300 tokens and scores every one of the 197 documents
(longest 11,970). Calibration keeps its wide 65,536; it was never tied to MNBT.

This also retracts the mechanism 96c5a06 recorded in the header ("the pool buys
KV as well as residency"). Inheriting the env's 49,152 / 96 stays right as
single-source-of-truth, but on this payload the pool is not the cache's supplier.
This commit is contained in:
Frank committed 2026-10-27 22:19:51 -07:00
1 parent 96c5a0625c
commit 1c56dea67a
1 file changed
+36 -32
+36 -32
View File
@@ -48,36 +48,36 @@
# NO --vram-kv-mib: it was REMOVED at 1.3.0 and naming it is a boot refusal, not
# a no-op — verified on the pinned image 2026-10-27: "E --vram-kv-mib was
# removed: the card is split between the weights and the cache from what the
# model needs." The budget resolver derives that split from --host-pool-mib and
# --gpu-headroom-mib, and this rig inherits BOTH from frank-archon.env, the file
# that owns them: the cache's funding is `elastic - expert_floor` and the expert
# floor falls one MiB for every MiB of host pool (frank-archon.md, Laws), so the
# pool buys KV as well as residency — a private pair is not a free knob. What
# buying one cost, measured on the first kldref boot the same day with the w4
# era's 12288 / 768: "I kv: the pool backs 6% of the worst case (4 sequences all
# at 32768 tokens), 0.2 full-length sessions; a larger --host-pool-mib gives the
# cache more" — a 270 MiB, 7,436-position cache, after which the recorder refused
# every one of the 197 corpus documents ("E kld: document 0 is 10554 tokens and
# the KV pool holds 7436, so it can never be scored") and exited 1. Production's
# 49,152 per rank is affordable exactly when the rig runs: the rig stops
# frank-archon first, and that stop frees the same 96 GiB the service was pinning.
# model needs." The resolver derives the split from --host-pool-mib,
# --gpu-headroom-mib and the activation arena; the rig inherits the first two from
# frank-archon.env, the file that owns them. On THIS payload the host pool is not
# the cache's supplier: the boot lands in vram_budget.cpp's `no lender` branch,
# where the experts keep their plan-fixed 25.29 GiB and the cache gets the residue.
# Measured twice — 12,288 and 49,152 MiB of host pool both printed the same
# "kv cache 271.25 MiB ... 7,436"; only --gpu-headroom-mib moved anything
# (768 -> 96 grew vram_weights 29.41 -> 30.06 GiB). The supplier the source names
# is the arena: vram_budget.cpp:117, "lower --max-num-batched-tokens (the arena
# scales with it)" — which is what the step size below now does.
#
# The 32,768-token window it pairs with is not up for grabs: the corpus's longest
# prompt is 10,554 tokens by the engine's own count (the header's chars/4.97
# estimate called that record ~7,700; the true ratio for this file is chars/3.6),
# so the window clears it by 3x and the cache now funds it.
# The pool, not the window, decides what the recorder accepts: it refuses any
# document longer than the pool, so the 32,768-token window was decorative while the
# cache held 7,436 — "E kld: document 196 is 8507 tokens and the KV pool holds
# 7436, so it can never be scored" on all 197 records, exit 1. The longest record
# is 10,554 tokens by the engine's own count (chars/3.6 for this file, not the
# chars/4.97 this repo measured on the K3 corpus).
#
# The step size is ONE value for the recording and every scoring pass:
# FRANK_QWEN38_MNBT, default 2,048. The gate's whole claim is that the reference,
# FRANK_QWEN38_MNBT, default 1,024. The gate's whole claim is that the reference,
# the served iq4r and every arm rode one shape, so the ceiling is set by the
# TIGHTEST payload the gate boots rather than by the bf16 reference: the quantised
# expert FILE tiers (the served iq4r, the arms) stream only at MNBT<=2048 and
# refuse the widened form above it, and 2,048 is what production holds anyway.
# Recording the reference at 32,768 would therefore measure a shape the payloads
# cannot reproduce; at 2,048 the 50,432 positions cost more steps, which is still
# one drained window. Calibration keeps the wide 65,536 — grams are not logits,
# nothing downstream is compared against a step size, and that is where a big
# step pays (a disk-tier step costs the same at 1 token as at 64k).
# refuse the widened form above it. 2,048 was still the wrong default on this
# payload, because the arena the engine charges for a step comes straight out of
# the cache — 1.08 GiB at a 2,048-token step, 504.39 MiB at a 1,024-token one, and
# the experts keep their share either way. At 1,024 the cache is 866 MiB: 24,300
# tokens, which scores every one of the 197 documents (longest 11,970).
# Calibration keeps the wide 65,536 — grams are not logits, nothing downstream is
# compared against a step size, and that is where a big step pays.
#
# usage: scripts/run-qwen38-bf16.sh kldref|calib|score RAD OUT [--dry-run] [--image REF]
# (RAD and OUT are score's two positionals: RAD is a container under the models
@@ -94,8 +94,9 @@
# bf16 container, kldref only), FRANK_QWEN38_CAMPAIGN (holds kld-ref/ and
# calib/), FRANK_QWEN38_KLD_CORPUS, FRANK_QWEN38_TPL (the template copy
# the container reads through /models), FRANK_QWEN38_LOG_DIR, and
# FRANK_QWEN38_MNBT — the step size shared by kldref and score (2,048 is
# the file tier's ceiling and production's own value, four slots and all),
# FRANK_QWEN38_MNBT — the step size shared by kldref and score (default
# 1,024: the file tier's ceiling is 2,048 and production runs there, but on
# this payload the arena charged for a wider step is taken from the cache),
# and FRANK_QWEN38_MAX_MODEL_LEN — the window every mode boots at (32,768;
# see the corpus census above, and raise it together with the corpus).
# kldref refuses (exit 2) when the longest record of the corpus it is about to
@@ -164,9 +165,11 @@ GPU_HEADROOM_MIB="${FRANK_ARCHON_GPU_HEADROOM_MIB:?FRANK_ARCHON_GPU_HEADROOM_MIB
# ONE step size for both logits modes. The file tier streams only at <=2,048 and
# the gate scores file-tier payloads (the served iq4r, the arms) against the
# reference, so the reference is recorded at the ceiling the payloads can meet —
# never two numbers that a KL between them would have to forgive.
MNBT="${FRANK_QWEN38_MNBT:-2048}"
# reference, so the reference is recorded at a step the payloads can meet — never
# two numbers that a KL between them would have to forgive. 1,024 rather than the
# 2,048 ceiling: on this payload the arena charged for the step is the cache's only
# remaining supplier, and 1,024 is the rung that funds every corpus document.
MNBT="${FRANK_QWEN38_MNBT:-1024}"
if ! [[ "$MNBT" =~ ^[0-9]+$ ]]; then
echo "REFUSED: FRANK_QWEN38_MNBT must be a token count (got: $MNBT)" >&2
exit 2
@@ -223,9 +226,10 @@ MOUNTS=(-v "$MODELS:/models:ro")
TAPENV=()
case "$MODE" in
kldref)
# Recording is prefill-only over four slots and the shared step, which is
# production's own operating point (frank-archon.env: FRANK_ARCHON_MAX_SEQS=4
# with FRANK_ARCHON_MAX_BATCHED_TOKENS=2048). Corpus read-only, record dir rw.
# Recording is prefill-only over four slots at the campaign's single step size
# — 1,024, not production's 2,048 (frank-archon.env: FRANK_ARCHON_MAX_SEQS=4,
# FRANK_ARCHON_MAX_BATCHED_TOKENS=2048): the wider step costs the cache the
# room the corpus needs. Corpus read-only, record dir rw.
ARGS+=(--model "$MODEL_IN"
--kld-record /data/kld-ref --kld-corpus /data/kld.jsonl
--max-num-seqs 4 --max-num-batched-tokens "$MNBT")