The calibration mode of the qwen38-w5nl64 rig asked for a step the campaign's own container cannot pay for, and the boot died with startup stopped: out of memory at 04:47 on 2026-10-29. This removes the exemption and makes one geometry serve all three modes.
scripts/run-qwen38-bf16.sh - calib used to ride a private step (--max-num-seqs 8 --max-num-batched-tokens 65536) while kldref and score ride the shared one. Every mode now takes its slot count from FRANK_ARCHON_MAX_SEQS - the same inheritance rule PR #77 established for the card split, and the env file that owns it (scripts/frank-archon.env, sourced at run-qwen38-bf16.sh:117) - and its step from FRANK_QWEN38_MNBT. The two --max-num-seqs 4 literals in kldref and score become "$MAX_SEQS", so no mode can carry its own number any more; MAX_SEQS joins the :?-guard family beside HOST_POOL_MIB / GPU_HEADROOM_MIB, so an empty env is fatal instead of silently blanking the flag.
tests/qwen38_reference.bats - the seam pinned the exemption (calib had to print 65536), so it is re-pinned to the invariant that replaced it: every mode emits the same --max-num-batched-tokensand the same --max-num-seqs, and the test title now says "every mode rides ONE step size and ONE seat count". The retired claim survives as the comment above the new assertion, so the reason is readable where it used to be enforced.
No vault page: this repo has no change diary (destroyed 2026-10-19), git holds the chronology, and the rig header is the point-of-use record for its own argv.
Why it was wrong
The comment justified the wide step as free: "the tap counts pooled positions and nothing downstream is compared against a step size, so the wide step is free throughput (a disk-tier step costs the same at 1 token as at 64k)". The cost is not on the disk - the step's activation arena is charged to the card, and the card pays for it out of the weight tiers and the cache. The header already says so two pages earlier (vram_budget.cpp:117: "lower --max-num-batched-tokens (the arena ...)"); the calib branch just did not read its own file.
Two boots of the same container on the same morning, from the engine's own ledgers on dankserver:
271.25 MiB, 0% backing of the 8 x 32768 worst case: 0.0 full-length sessions
result
served the corpus
E startup stopped: out of memory
A wide step buys throughput with the expert slab, so it is priced like any other VRAM knob, never waved through as free. --max-num-seqs 8 was the second half of the same invention: production runs four slots and the other two modes measure at four.
Campaign state this unblocks
kldref is banked (rc 0, 45 GiB logp.f16 + rows/top/tokens in /home/dank/campaign/qwen38-w5nl64/kld-ref); frank-archon has been active and healthy since 04:47; the 335 GiB reference .rad is still on disk because the chain keeps it until both recordings exist. This fix is the door to re-running calib, and the converts behind it.
Verification
bash tests/run.sh - green, reported in the comment thread. The argv change is also checked directly: scripts/run-qwen38-bf16.sh calib --dry-run and ... kldref --dry-run now emit byte-identical --max-num-seqs 4 --max-num-batched-tokens 1024 --max-model-len 32768 geometry.
## What
The calibration mode of the qwen38-w5nl64 rig asked for a step the campaign's own container cannot pay for, and the boot died with `startup stopped: out of memory` at 04:47 on 2026-10-29. This removes the exemption and makes one geometry serve all three modes.
- `scripts/run-qwen38-bf16.sh` - `calib` used to ride a private step (`--max-num-seqs 8 --max-num-batched-tokens 65536`) while `kldref` and `score` ride the shared one. Every mode now takes its slot count from `FRANK_ARCHON_MAX_SEQS` - the same inheritance rule PR #77 established for the card split, and the env file that owns it (`scripts/frank-archon.env`, sourced at `run-qwen38-bf16.sh:117`) - and its step from `FRANK_QWEN38_MNBT`. The two `--max-num-seqs 4` literals in `kldref` and `score` become `"$MAX_SEQS"`, so no mode can carry its own number any more; `MAX_SEQS` joins the `:?`-guard family beside `HOST_POOL_MIB` / `GPU_HEADROOM_MIB`, so an empty env is fatal instead of silently blanking the flag.
- `tests/qwen38_reference.bats` - the seam pinned the exemption (calib had to print `65536`), so it is re-pinned to the invariant that replaced it: every mode emits the same `--max-num-batched-tokens` *and* the same `--max-num-seqs`, and the test title now says "every mode rides ONE step size and ONE seat count". The retired claim survives as the comment above the new assertion, so the reason is readable where it used to be enforced.
- No vault page: this repo has no change diary (destroyed 2026-10-19), git holds the chronology, and the rig header is the point-of-use record for its own argv.
## Why it was wrong
The comment justified the wide step as free: "the tap counts pooled positions and nothing downstream is compared against a step size, so the wide step is free throughput (a disk-tier step costs the same at 1 token as at 64k)". The cost is not on the disk - the step's activation arena is charged to the card, and the card pays for it out of the weight tiers and the cache. The header already says so two pages earlier (`vram_budget.cpp:117`: "lower --max-num-batched-tokens (the arena ...)"); the calib branch just did not read its own file.
Two boots of the same container on the same morning, from the engine's own ledgers on dankserver:
| | `kldref` - booted, recorded 16 h | `calib` - refused |
|---|---|---|
|`--max-num-seqs` / MNBT | 4 / 1,024 | 8 / 65,536 |
|activation arena | 504.39 MiB (127.70 MiB in use) | **10.65 GiB** (fixed region 1.25 GiB; 16,384-token steps cost 4.02 GiB) |
|claimable VRAM | 30.87 GiB | **16.85 GiB** |
|weights tiered in VRAM | 28.90 GiB | 14.24 GiB - 56.24 GiB left reading from disk |
|KV | 270.00 MiB, accepted | 271.25 MiB, `0% backing of the 8 x 32768 worst case: 0.0 full-length sessions` |
|result | served the corpus | `E startup stopped: out of memory` |
A wide step buys throughput with the expert slab, so it is priced like any other VRAM knob, never waved through as free. `--max-num-seqs 8` was the second half of the same invention: production runs four slots and the other two modes measure at four.
## Campaign state this unblocks
`kldref` is banked (rc 0, 45 GiB `logp.f16` + rows/top/tokens in `/home/dank/campaign/qwen38-w5nl64/kld-ref`); `frank-archon` has been active and healthy since 04:47; the 335 GiB reference `.rad` is still on disk because the chain keeps it until **both** recordings exist. This fix is the door to re-running `calib`, and the converts behind it.
## Verification
`bash tests/run.sh` - green, reported in the comment thread. The argv change is also checked directly: `scripts/run-qwen38-bf16.sh calib --dry-run` and `... kldref --dry-run` now emit byte-identical `--max-num-seqs 4 --max-num-batched-tokens 1024 --max-model-len 32768` geometry.
---
cwd: /home/blank/.omp/wt/wt-20261027-142335-6d1fc1f · session: 01a17f3f-3e2e-7509-aef8-be516c27cf98
The calibration mode kept a private step (8 seats at MNBT 65,536) that it
called "free throughput". The step's activation arena is billed to the card,
so the wide step cost the container its own expert slab and KV: at 65,536
the arena is 10.65 GiB and the claim falls to 16.85 GiB (weights 14.24 GiB
in VRAM, 56.24 GiB stranded on disk, KV 271 MiB for 8 seats), and the boot
dies with "startup stopped: out of memory" (measured 2026-10-29, while the
kldref boot at MNBT 1,024 — arena 504 MiB, claimable 30.87 GiB — had just
recorded for 16 hours without fault).
Every mode now takes its seat count from FRANK_ARCHON_MAX_SEQS, the file that
owns it, exactly as PR #77 established for the card split, and its step from
FRANK_QWEN38_MNBT. The per-mode literals in kldref and score follow, so one
number defines the geometry for all three. The test that pinned the old
exemption is re-pinned to the invariant that replaced it.
tests/run.sh: 576/576 green.
dank
merged commit b4d7f6e7fe into main2026-10-29 11:18:36 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What
The calibration mode of the qwen38-w5nl64 rig asked for a step the campaign's own container cannot pay for, and the boot died with
startup stopped: out of memoryat 04:47 on 2026-10-29. This removes the exemption and makes one geometry serve all three modes.scripts/run-qwen38-bf16.sh-calibused to ride a private step (--max-num-seqs 8 --max-num-batched-tokens 65536) whilekldrefandscoreride the shared one. Every mode now takes its slot count fromFRANK_ARCHON_MAX_SEQS- the same inheritance rule PR #77 established for the card split, and the env file that owns it (scripts/frank-archon.env, sourced atrun-qwen38-bf16.sh:117) - and its step fromFRANK_QWEN38_MNBT. The two--max-num-seqs 4literals inkldrefandscorebecome"$MAX_SEQS", so no mode can carry its own number any more;MAX_SEQSjoins the:?-guard family besideHOST_POOL_MIB/GPU_HEADROOM_MIB, so an empty env is fatal instead of silently blanking the flag.tests/qwen38_reference.bats- the seam pinned the exemption (calib had to print65536), so it is re-pinned to the invariant that replaced it: every mode emits the same--max-num-batched-tokensand the same--max-num-seqs, and the test title now says "every mode rides ONE step size and ONE seat count". The retired claim survives as the comment above the new assertion, so the reason is readable where it used to be enforced.Why it was wrong
The comment justified the wide step as free: "the tap counts pooled positions and nothing downstream is compared against a step size, so the wide step is free throughput (a disk-tier step costs the same at 1 token as at 64k)". The cost is not on the disk - the step's activation arena is charged to the card, and the card pays for it out of the weight tiers and the cache. The header already says so two pages earlier (
vram_budget.cpp:117: "lower --max-num-batched-tokens (the arena ...)"); the calib branch just did not read its own file.Two boots of the same container on the same morning, from the engine's own ledgers on dankserver:
kldref- booted, recorded 16 hcalib- refused--max-num-seqs/ MNBT0% backing of the 8 x 32768 worst case: 0.0 full-length sessionsE startup stopped: out of memoryA wide step buys throughput with the expert slab, so it is priced like any other VRAM knob, never waved through as free.
--max-num-seqs 8was the second half of the same invention: production runs four slots and the other two modes measure at four.Campaign state this unblocks
kldrefis banked (rc 0, 45 GiBlogp.f16+ rows/top/tokens in/home/dank/campaign/qwen38-w5nl64/kld-ref);frank-archonhas been active and healthy since 04:47; the 335 GiB reference.radis still on disk because the chain keeps it until both recordings exist. This fix is the door to re-runningcalib, and the converts behind it.Verification
bash tests/run.sh- green, reported in the comment thread. The argv change is also checked directly:scripts/run-qwen38-bf16.sh calib --dry-runand... kldref --dry-runnow emit byte-identical--max-num-seqs 4 --max-num-batched-tokens 1024 --max-model-len 32768geometry.cwd: /home/blank/.omp/wt/wt-20261027-142335-6d1fc1f · session: 01a17f3f-3e2e-7509-aef8-be516c27cf98