fix(qwen38): record at MNBT 1,024 — the step size the cache can actually pay for
The rerun after96c5a06refused every corpus document again ("document 196 is 8507 tokens and the KV pool holds 7436"), and the boot table shows why the host-pool change could not have helped: vram_budget.cpp put the container in its `no lender` branch — elastic 25.56 GiB, experts 25.29 GiB (their plan-fixed share), kv cache 271.25 MiB — identical at 12,288 and at 49,152 MiB of host pool. The only supplier left there is the activation arena, and the engine says so at vram_budget.cpp:117: "lower --max-num-batched-tokens (the arena scales with it)". The log prices both rungs directly (a 2,048-token step costs 1.08 GiB, a 1,024-token step 504.39 MiB), so the default moves to 1,024: the cache grows to 866 MiB / 24,300 tokens and scores every one of the 197 documents (longest 11,970). Calibration keeps its wide 65,536; it was never tied to MNBT. This also retracts the mechanism96c5a06recorded in the header ("the pool buys KV as well as residency"). Inheriting the env's 49,152 / 96 stays right as single-source-of-truth, but on this payload the pool is not the cache's supplier.
This commit is contained in:
1 parent
96c5a0625c
commit
1c56dea67a
1 file changed
+36
-32
+36
-32
@@ -48,36 +48,36 @@
|
|||||||
# NO --vram-kv-mib: it was REMOVED at 1.3.0 and naming it is a boot refusal, not
|
# NO --vram-kv-mib: it was REMOVED at 1.3.0 and naming it is a boot refusal, not
|
||||||
# a no-op — verified on the pinned image 2026-10-27: "E --vram-kv-mib was
|
# a no-op — verified on the pinned image 2026-10-27: "E --vram-kv-mib was
|
||||||
# removed: the card is split between the weights and the cache from what the
|
# removed: the card is split between the weights and the cache from what the
|
||||||
# model needs." The budget resolver derives that split from --host-pool-mib and
|
# model needs." The resolver derives the split from --host-pool-mib,
|
||||||
# --gpu-headroom-mib, and this rig inherits BOTH from frank-archon.env, the file
|
# --gpu-headroom-mib and the activation arena; the rig inherits the first two from
|
||||||
# that owns them: the cache's funding is `elastic - expert_floor` and the expert
|
# frank-archon.env, the file that owns them. On THIS payload the host pool is not
|
||||||
# floor falls one MiB for every MiB of host pool (frank-archon.md, Laws), so the
|
# the cache's supplier: the boot lands in vram_budget.cpp's `no lender` branch,
|
||||||
# pool buys KV as well as residency — a private pair is not a free knob. What
|
# where the experts keep their plan-fixed 25.29 GiB and the cache gets the residue.
|
||||||
# buying one cost, measured on the first kldref boot the same day with the w4
|
# Measured twice — 12,288 and 49,152 MiB of host pool both printed the same
|
||||||
# era's 12288 / 768: "I kv: the pool backs 6% of the worst case (4 sequences all
|
# "kv cache 271.25 MiB ... 7,436"; only --gpu-headroom-mib moved anything
|
||||||
# at 32768 tokens), 0.2 full-length sessions; a larger --host-pool-mib gives the
|
# (768 -> 96 grew vram_weights 29.41 -> 30.06 GiB). The supplier the source names
|
||||||
# cache more" — a 270 MiB, 7,436-position cache, after which the recorder refused
|
# is the arena: vram_budget.cpp:117, "lower --max-num-batched-tokens (the arena
|
||||||
# every one of the 197 corpus documents ("E kld: document 0 is 10554 tokens and
|
# scales with it)" — which is what the step size below now does.
|
||||||
# the KV pool holds 7436, so it can never be scored") and exited 1. Production's
|
|
||||||
# 49,152 per rank is affordable exactly when the rig runs: the rig stops
|
|
||||||
# frank-archon first, and that stop frees the same 96 GiB the service was pinning.
|
|
||||||
#
|
#
|
||||||
# The 32,768-token window it pairs with is not up for grabs: the corpus's longest
|
# The pool, not the window, decides what the recorder accepts: it refuses any
|
||||||
# prompt is 10,554 tokens by the engine's own count (the header's chars/4.97
|
# document longer than the pool, so the 32,768-token window was decorative while the
|
||||||
# estimate called that record ~7,700; the true ratio for this file is chars/3.6),
|
# cache held 7,436 — "E kld: document 196 is 8507 tokens and the KV pool holds
|
||||||
# so the window clears it by 3x and the cache now funds it.
|
# 7436, so it can never be scored" on all 197 records, exit 1. The longest record
|
||||||
|
# is 10,554 tokens by the engine's own count (chars/3.6 for this file, not the
|
||||||
|
# chars/4.97 this repo measured on the K3 corpus).
|
||||||
#
|
#
|
||||||
# The step size is ONE value for the recording and every scoring pass:
|
# The step size is ONE value for the recording and every scoring pass:
|
||||||
# FRANK_QWEN38_MNBT, default 2,048. The gate's whole claim is that the reference,
|
# FRANK_QWEN38_MNBT, default 1,024. The gate's whole claim is that the reference,
|
||||||
# the served iq4r and every arm rode one shape, so the ceiling is set by the
|
# the served iq4r and every arm rode one shape, so the ceiling is set by the
|
||||||
# TIGHTEST payload the gate boots rather than by the bf16 reference: the quantised
|
# TIGHTEST payload the gate boots rather than by the bf16 reference: the quantised
|
||||||
# expert FILE tiers (the served iq4r, the arms) stream only at MNBT<=2048 and
|
# expert FILE tiers (the served iq4r, the arms) stream only at MNBT<=2048 and
|
||||||
# refuse the widened form above it, and 2,048 is what production holds anyway.
|
# refuse the widened form above it. 2,048 was still the wrong default on this
|
||||||
# Recording the reference at 32,768 would therefore measure a shape the payloads
|
# payload, because the arena the engine charges for a step comes straight out of
|
||||||
# cannot reproduce; at 2,048 the 50,432 positions cost more steps, which is still
|
# the cache — 1.08 GiB at a 2,048-token step, 504.39 MiB at a 1,024-token one, and
|
||||||
# one drained window. Calibration keeps the wide 65,536 — grams are not logits,
|
# the experts keep their share either way. At 1,024 the cache is 866 MiB: 24,300
|
||||||
# nothing downstream is compared against a step size, and that is where a big
|
# tokens, which scores every one of the 197 documents (longest 11,970).
|
||||||
# step pays (a disk-tier step costs the same at 1 token as at 64k).
|
# Calibration keeps the wide 65,536 — grams are not logits, nothing downstream is
|
||||||
|
# compared against a step size, and that is where a big step pays.
|
||||||
#
|
#
|
||||||
# usage: scripts/run-qwen38-bf16.sh kldref|calib|score RAD OUT [--dry-run] [--image REF]
|
# usage: scripts/run-qwen38-bf16.sh kldref|calib|score RAD OUT [--dry-run] [--image REF]
|
||||||
# (RAD and OUT are score's two positionals: RAD is a container under the models
|
# (RAD and OUT are score's two positionals: RAD is a container under the models
|
||||||
@@ -94,8 +94,9 @@
|
|||||||
# bf16 container, kldref only), FRANK_QWEN38_CAMPAIGN (holds kld-ref/ and
|
# bf16 container, kldref only), FRANK_QWEN38_CAMPAIGN (holds kld-ref/ and
|
||||||
# calib/), FRANK_QWEN38_KLD_CORPUS, FRANK_QWEN38_TPL (the template copy
|
# calib/), FRANK_QWEN38_KLD_CORPUS, FRANK_QWEN38_TPL (the template copy
|
||||||
# the container reads through /models), FRANK_QWEN38_LOG_DIR, and
|
# the container reads through /models), FRANK_QWEN38_LOG_DIR, and
|
||||||
# FRANK_QWEN38_MNBT — the step size shared by kldref and score (2,048 is
|
# FRANK_QWEN38_MNBT — the step size shared by kldref and score (default
|
||||||
# the file tier's ceiling and production's own value, four slots and all),
|
# 1,024: the file tier's ceiling is 2,048 and production runs there, but on
|
||||||
|
# this payload the arena charged for a wider step is taken from the cache),
|
||||||
# and FRANK_QWEN38_MAX_MODEL_LEN — the window every mode boots at (32,768;
|
# and FRANK_QWEN38_MAX_MODEL_LEN — the window every mode boots at (32,768;
|
||||||
# see the corpus census above, and raise it together with the corpus).
|
# see the corpus census above, and raise it together with the corpus).
|
||||||
# kldref refuses (exit 2) when the longest record of the corpus it is about to
|
# kldref refuses (exit 2) when the longest record of the corpus it is about to
|
||||||
@@ -164,9 +165,11 @@ GPU_HEADROOM_MIB="${FRANK_ARCHON_GPU_HEADROOM_MIB:?FRANK_ARCHON_GPU_HEADROOM_MIB
|
|||||||
|
|
||||||
# ONE step size for both logits modes. The file tier streams only at <=2,048 and
|
# ONE step size for both logits modes. The file tier streams only at <=2,048 and
|
||||||
# the gate scores file-tier payloads (the served iq4r, the arms) against the
|
# the gate scores file-tier payloads (the served iq4r, the arms) against the
|
||||||
# reference, so the reference is recorded at the ceiling the payloads can meet —
|
# reference, so the reference is recorded at a step the payloads can meet — never
|
||||||
# never two numbers that a KL between them would have to forgive.
|
# two numbers that a KL between them would have to forgive. 1,024 rather than the
|
||||||
MNBT="${FRANK_QWEN38_MNBT:-2048}"
|
# 2,048 ceiling: on this payload the arena charged for the step is the cache's only
|
||||||
|
# remaining supplier, and 1,024 is the rung that funds every corpus document.
|
||||||
|
MNBT="${FRANK_QWEN38_MNBT:-1024}"
|
||||||
if ! [[ "$MNBT" =~ ^[0-9]+$ ]]; then
|
if ! [[ "$MNBT" =~ ^[0-9]+$ ]]; then
|
||||||
echo "REFUSED: FRANK_QWEN38_MNBT must be a token count (got: $MNBT)" >&2
|
echo "REFUSED: FRANK_QWEN38_MNBT must be a token count (got: $MNBT)" >&2
|
||||||
exit 2
|
exit 2
|
||||||
@@ -223,9 +226,10 @@ MOUNTS=(-v "$MODELS:/models:ro")
|
|||||||
TAPENV=()
|
TAPENV=()
|
||||||
case "$MODE" in
|
case "$MODE" in
|
||||||
kldref)
|
kldref)
|
||||||
# Recording is prefill-only over four slots and the shared step, which is
|
# Recording is prefill-only over four slots at the campaign's single step size
|
||||||
# production's own operating point (frank-archon.env: FRANK_ARCHON_MAX_SEQS=4
|
# — 1,024, not production's 2,048 (frank-archon.env: FRANK_ARCHON_MAX_SEQS=4,
|
||||||
# with FRANK_ARCHON_MAX_BATCHED_TOKENS=2048). Corpus read-only, record dir rw.
|
# FRANK_ARCHON_MAX_BATCHED_TOKENS=2048): the wider step costs the cache the
|
||||||
|
# room the corpus needs. Corpus read-only, record dir rw.
|
||||||
ARGS+=(--model "$MODEL_IN"
|
ARGS+=(--model "$MODEL_IN"
|
||||||
--kld-record /data/kld-ref --kld-corpus /data/kld.jsonl
|
--kld-record /data/kld-ref --kld-corpus /data/kld.jsonl
|
||||||
--max-num-seqs 4 --max-num-batched-tokens "$MNBT")
|
--max-num-seqs 4 --max-num-batched-tokens "$MNBT")
|
||||||
|
|||||||
Reference in new issue
Block a user