Two fixes to the Qwen3.8 campaign's BF16 reference rig, both found by re-running it and reading the engine's own boot ledger.
1c56a06 — fix(qwen38): the rig's card split is inherited, not invented
run-qwen38-bf16.sh carried its own --host-pool-mib 12288 --gpu-headroom-mib 768, copied from the w4-era swift15 rig, instead of the pair scripts/frank-archon.env owns (49,152 / 96). The rig now sources FRANK_ARCHON_HOST_POOL_MIB / FRANK_ARCHON_GPU_HEADROOM_MIB from the env it already sources (:? guards) — one owner per fact. tests/qwen38_reference.bats pins the container argv verbatim, so its SHAPE contract moved with it.
1c56dea — fix(qwen38): record at MNBT 1,024 — the step size the cache can actually pay for
The rerun still refused all 197 documents (E kld: document 196 is 8507 tokens and the KV pool holds 7436, so it can never be scored), and the boot table showed why the first fix could not have moved the cache:
I VRAM budget, per card: card total 31.86 GiB … activation arena 1.08 GiB
claimable 30.33 GiB · static weights 4.77 GiB
elastic 25.56 GiB (no lender: the experts first, the cache at least one full-length sequence)
experts 25.29 GiB 99% of it
kv cache 271.25 MiB 1% of it — 6% backing of the 4 x 32768 worst case
core/mem/vram_budget.cpp puts this payload in its no lender branch, where the experts keep their plan-fixed share and the cache gets the residue — so --host-pool-mib buys nothing here (measured: 271.25 MiB at both 12,288 and 49,152). The supplier the source names is the arena (vram_budget.cpp:117: "lower --max-num-batched-tokens (the arena scales with it)"), and the same log prices both rungs: 1.08 GiB for a 2,048-token step, 504.39 MiB for a 1,024-token one. FRANK_QWEN38_MNBT therefore defaults to 1,024 for kldref/score: the cache grows to ~866 MiB / 24,300 tokens, which scores every corpus document (longest 11,970). calib keeps its wide 65,536 — grams are not logits. The commit also retracts the mechanism the first commit recorded in the header ("the pool buys KV as well as residency"); inheriting the env pair stays correct as single-source-of-truth, it just was not the lever.
Two fixes to the Qwen3.8 campaign's BF16 reference rig, both found by re-running it and reading the engine's own boot ledger.
## 1c56a06 — `fix(qwen38): the rig's card split is inherited, not invented`
`run-qwen38-bf16.sh` carried its own `--host-pool-mib 12288 --gpu-headroom-mib 768`, copied from the w4-era swift15 rig, instead of the pair `scripts/frank-archon.env` owns (49,152 / 96). The rig now sources `FRANK_ARCHON_HOST_POOL_MIB` / `FRANK_ARCHON_GPU_HEADROOM_MIB` from the env it already sources (`:?` guards) — one owner per fact. `tests/qwen38_reference.bats` pins the container argv verbatim, so its `SHAPE` contract moved with it.
## 1c56dea — `fix(qwen38): record at MNBT 1,024 — the step size the cache can actually pay for`
The rerun still refused all 197 documents (`E kld: document 196 is 8507 tokens and the KV pool holds 7436, so it can never be scored`), and the boot table showed why the first fix could not have moved the cache:
```
I VRAM budget, per card: card total 31.86 GiB … activation arena 1.08 GiB
claimable 30.33 GiB · static weights 4.77 GiB
elastic 25.56 GiB (no lender: the experts first, the cache at least one full-length sequence)
experts 25.29 GiB 99% of it
kv cache 271.25 MiB 1% of it — 6% backing of the 4 x 32768 worst case
```
`core/mem/vram_budget.cpp` puts this payload in its `no lender` branch, where the experts keep their plan-fixed share and the cache gets the residue — so `--host-pool-mib` buys nothing here (measured: 271.25 MiB at both 12,288 and 49,152). The supplier the source names is the arena (`vram_budget.cpp:117`: *"lower `--max-num-batched-tokens` (the arena scales with it)"*), and the same log prices both rungs: 1.08 GiB for a 2,048-token step, 504.39 MiB for a 1,024-token one. `FRANK_QWEN38_MNBT` therefore defaults to **1,024** for `kldref`/`score`: the cache grows to ~866 MiB / 24,300 tokens, which scores every corpus document (longest 11,970). `calib` keeps its wide 65,536 — grams are not logits. The commit also retracts the mechanism the first commit recorded in the header ("the pool buys KV as well as residency"); inheriting the env pair stays correct as single-source-of-truth, it just was not the lever.
## Evidence
`./tests/run.sh` → `==> ALL TESTS PASSED` (558 tests, 79 s); `scripts/test-changed.sh` → 4/4. The dry-run argv now resolves to `--tp 2 --placement expert_tiered --weights-disk-tier --host-pool-mib 49152 --gpu-headroom-mib 96 --deterministic --num-speculative-tokens 0 --kv-cache-dtype fp8 --max-model-len 32768 --max-num-seqs 4 --max-num-batched-tokens 1024 --ngram-placement ram --override-chat-template /models/tpl.jinja`.
---
cwd: /home/blank/.omp/wt/wt-20261027-142335-6d1fc1f · session: 01a17f3f-3e2e-7509-aef8-be516c27cf98
run-qwen38-bf16.sh carried its own --host-pool-mib 12288 / --gpu-headroom-mib 768,
copied from the w4-era swift15 rig. The engine derives the whole per-card split from
those two numbers, and the KV cache's funding is `elastic - expert_floor` where the
expert floor is a function of the pinned host pool — so the pool buys cache as well
as residency. The first kldref boot got a 270 MiB / 7,436-position pool and the
recorder refused all 197 corpus documents ("document 0 is 10554 tokens and the KV
pool holds 7436"), exiting 1.
The rig now sources FRANK_ARCHON_HOST_POOL_MIB / FRANK_ARCHON_GPU_HEADROOM_MIB from
scripts/frank-archon.env, the file that owns them (49152 / 96), with `:?` guards, and
the header records the law instead of the claim that the pool never touches logits.
tests/qwen38_reference.bats pins the shape verbatim, so its SHAPE contract moves with
it; the suite is green (558 tests).
The rerun after 96c5a06 refused every corpus document again ("document 196 is
8507 tokens and the KV pool holds 7436"), and the boot table shows why the
host-pool change could not have helped: vram_budget.cpp put the container in its
`no lender` branch — elastic 25.56 GiB, experts 25.29 GiB (their plan-fixed
share), kv cache 271.25 MiB — identical at 12,288 and at 49,152 MiB of host
pool. The only supplier left there is the activation arena, and the engine says
so at vram_budget.cpp:117: "lower --max-num-batched-tokens (the arena scales
with it)". The log prices both rungs directly (a 2,048-token step costs 1.08
GiB, a 1,024-token step 504.39 MiB), so the default moves to 1,024: the cache
grows to 866 MiB / 24,300 tokens and scores every one of the 197 documents
(longest 11,970). Calibration keeps its wide 65,536; it was never tied to MNBT.
This also retracts the mechanism 96c5a06 recorded in the header ("the pool buys
KV as well as residency"). Inheriting the env's 49,152 / 96 stays right as
single-source-of-truth, but on this payload the pool is not the cache's supplier.
dank
merged commit fca47d25d8 into main2026-10-27 23:53:12 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Two fixes to the Qwen3.8 campaign's BF16 reference rig, both found by re-running it and reading the engine's own boot ledger.
1c56a06 —
fix(qwen38): the rig's card split is inherited, not inventedrun-qwen38-bf16.shcarried its own--host-pool-mib 12288 --gpu-headroom-mib 768, copied from the w4-era swift15 rig, instead of the pairscripts/frank-archon.envowns (49,152 / 96). The rig now sourcesFRANK_ARCHON_HOST_POOL_MIB/FRANK_ARCHON_GPU_HEADROOM_MIBfrom the env it already sources (:?guards) — one owner per fact.tests/qwen38_reference.batspins the container argv verbatim, so itsSHAPEcontract moved with it.1c56dea—fix(qwen38): record at MNBT 1,024 — the step size the cache can actually pay forThe rerun still refused all 197 documents (
E kld: document 196 is 8507 tokens and the KV pool holds 7436, so it can never be scored), and the boot table showed why the first fix could not have moved the cache:core/mem/vram_budget.cppputs this payload in itsno lenderbranch, where the experts keep their plan-fixed share and the cache gets the residue — so--host-pool-mibbuys nothing here (measured: 271.25 MiB at both 12,288 and 49,152). The supplier the source names is the arena (vram_budget.cpp:117: "lower--max-num-batched-tokens(the arena scales with it)"), and the same log prices both rungs: 1.08 GiB for a 2,048-token step, 504.39 MiB for a 1,024-token one.FRANK_QWEN38_MNBTtherefore defaults to 1,024 forkldref/score: the cache grows to ~866 MiB / 24,300 tokens, which scores every corpus document (longest 11,970).calibkeeps its wide 65,536 — grams are not logits. The commit also retracts the mechanism the first commit recorded in the header ("the pool buys KV as well as residency"); inheriting the env pair stays correct as single-source-of-truth, it just was not the lever.Evidence
./tests/run.sh→==> ALL TESTS PASSED(558 tests, 79 s);scripts/test-changed.sh→ 4/4. The dry-run argv now resolves to--tp 2 --placement expert_tiered --weights-disk-tier --host-pool-mib 49152 --gpu-headroom-mib 96 --deterministic --num-speculative-tokens 0 --kv-cache-dtype fp8 --max-model-len 32768 --max-num-seqs 4 --max-num-batched-tokens 1024 --ngram-placement ram --override-chat-template /models/tpl.jinja.cwd: /home/blank/.omp/wt/wt-20261027-142335-6d1fc1f · session: 01a17f3f-3e2e-7509-aef8-be516c27cf98
run-qwen38-bf16.sh carried its own --host-pool-mib 12288 / --gpu-headroom-mib 768, copied from the w4-era swift15 rig. The engine derives the whole per-card split from those two numbers, and the KV cache's funding is `elastic - expert_floor` where the expert floor is a function of the pinned host pool — so the pool buys cache as well as residency. The first kldref boot got a 270 MiB / 7,436-position pool and the recorder refused all 197 corpus documents ("document 0 is 10554 tokens and the KV pool holds 7436"), exiting 1. The rig now sources FRANK_ARCHON_HOST_POOL_MIB / FRANK_ARCHON_GPU_HEADROOM_MIB from scripts/frank-archon.env, the file that owns them (49152 / 96), with `:?` guards, and the header records the law instead of the claim that the pool never touches logits. tests/qwen38_reference.bats pins the shape verbatim, so its SHAPE contract moves with it; the suite is green (558 tests).