The calibration tap — the layer Hessians rad-convert reads through gptq:calib= — only arms on latest radiance. The production pin carries the code and the plugin strings and never writes a byte.
scripts/run-qwen38-bf16.sh: CALIB_IMAGE -> stilldeadcode/radiance@sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093
(:latest on 2026-10-30 = v1.14.0, revision 4a61d7296bc5), with the law in the header comment;
the tap floor moves 12M -> 1.5M tokens (the vendor's own measurement is that 10M beats 1M by under 1%).
scripts/convert-qwen38-w5nl64.sh: the converter rides the same digest — the tap's file contract
(RADGRAM2, gram.r<rank>.<stem>.bin) is version-stable, so a latest-written tap converts on latest.
tests/qwen38_reference.bats: the calib dry run asserts the 1.5M floor and the latest pin; the
calibration-window dry-run test passes Q38_CALIB_MIN_FREE_MIB=1 because its fixture lands on /tmp (a 16 GB tmpfs) and the 60 GiB seal guard can never pass there — pre-existing and
environment-bound, unrelated to this change.
Evidence
17 hours on the production pin: zero bytes into the tap dir while /health, /stats and the
push all reported healthy.
On latest, the same boot prints I[0] recorded passes: off, the program accumulates calibration Grams (gram_accum) and then calibration wrote layer Hessians after 9129 tokens — 24,768 gram
files, 42.9 GB.
The vendor's calibration shape (--max-num-batched-tokens 65536 --max-num-seqs 64, ~930 tok/s)
cannot boot two 32 GiB R9700s: the activation arena alone is 12.83 GiB of 27.95 GiB claimable.
Wider steps are refused by the arch itself — qwen4exp: the QSA score plane would be 3125 MiB — 16384 rows by --max-model-len 200000 / compress ratio 4 — so the proven 2,048-token step runs.
One seal writes ~40 GiB; the engine exits by itself when the tap seals, so a mid-flight client
gets 429 the server is shutting down (success, not failure).
Testing
./tests/run.sh -> 589/589 pass. The tap run itself is executing on dankserver against :latest;
its artifacts and logs live in /home/dank/campaign/qwen38-w5nl64 (README there is the operating record).
## What
The calibration tap — the layer Hessians `rad-convert` reads through `gptq:calib=` — only arms on
**latest radiance**. The production pin carries the code and the plugin strings and never writes a byte.
- `scripts/run-qwen38-bf16.sh`: `CALIB_IMAGE` -> `stilldeadcode/radiance@sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093`
(`:latest` on 2026-10-30 = v1.14.0, revision `4a61d7296bc5`), with the law in the header comment;
the tap floor moves 12M -> 1.5M tokens (the vendor's own measurement is that 10M beats 1M by under 1%).
- `scripts/convert-qwen38-w5nl64.sh`: the converter rides the same digest — the tap's file contract
(`RADGRAM2`, `gram.r<rank>.<stem>.bin`) is version-stable, so a latest-written tap converts on latest.
- `tests/qwen38_reference.bats`: the calib dry run asserts the 1.5M floor and the latest pin; the
calibration-window dry-run test passes `Q38_CALIB_MIN_FREE_MIB=1` because its fixture lands on
`/tmp` (a 16 GB tmpfs) and the 60 GiB seal guard can never pass there — pre-existing and
environment-bound, unrelated to this change.
## Evidence
- 17 hours on the production pin: **zero bytes** into the tap dir while `/health`, `/stats` and the
push all reported healthy.
- On latest, the same boot prints `I[0] recorded passes: off, the program accumulates calibration
Grams (gram_accum)` and then `calibration wrote layer Hessians after 9129 tokens` — **24,768 gram
files, 42.9 GB**.
- The vendor's calibration shape (`--max-num-batched-tokens 65536 --max-num-seqs 64`, ~930 tok/s)
cannot boot two 32 GiB R9700s: the activation arena alone is 12.83 GiB of 27.95 GiB claimable.
Wider steps are refused by the arch itself — `qwen4exp: the QSA score plane would be 3125 MiB —
16384 rows by --max-model-len 200000 / compress ratio 4` — so the proven 2,048-token step runs.
- One seal writes ~40 GiB; the engine exits by itself when the tap seals, so a mid-flight client
gets `429 the server is shutting down` (success, not failure).
## Testing
`./tests/run.sh` -> **589/589 pass**. The tap run itself is executing on dankserver against `:latest`;
its artifacts and logs live in `/home/dank/campaign/qwen38-w5nl64` (README there is the operating record).
---
cwd: /home/blank/.omp/wt/wt-20261027-142335-6d1fc1f · session: 01a17f3f-3e2e-7509-aef8-be516c27cf98
A 17-hour calibration tap on 2026-10-29/30 wrote ZERO bytes: the runner booted
FRANK_ARCHON_IMAGE (1.8.1) for every mode, and RADIANCE_CALIB_DIR/TOKENS only
arms the expert tap on the 1.2.x line. 1.8.1 still carries the plugin strings and
the "calibration wrote layer Hessians" message, /health answered, /stats showed
1,700 documents and 118 tok/s, and nothing anywhere said the tap was dead.
Four gates, each with the measurement that motivated it:
- CALIB_IMAGE pins the 1.2.0 digest for calib mode (stilldeadcode/radiance
@sha256:665775ed…), overridable only by naming --image deliberately. Proven by
the engagement test: `calibration wrote layer Hessians after 10207 tokens`.
- The tap dir's filesystem must have >= 60 GiB free (Q38_CALIB_MIN_FREE_MIB).
One seal wrote 39.4 GiB in seconds; pointed at a 37 GiB root it filled the
filesystem to zero and every later write came back short.
- restore() kills orphaned `radiance` processes still holding /dev/kfd before it
starts the service. `podman rm -f` does not kill the engine: an orphan made
production's ar_oneshot_2rank_wht6 init fail `device error`, and the unit
restarted 100 times with /health never answering.
- FRANK_QWEN38_CALIB_DIR / Q38_WINDOW_TRASH_ROOT make the tap dir and the bank
target configurable, so the tap lives on /mnt/entropy instead of the root fs.
The tap did not arm on the production pin: a 17-hour calib run on 1.8.1
wrote zero bytes while /health, /stats and the push all read healthy.
On :latest (v1.14.0, revision 4a61d7296bc5) the same boot prints
'recorded passes: off, the program accumulates calibration Grams
(gram_accum)' and a 9,129-token push sealed 24,768 gram files (42.9 GB)
in the fitting shape - 4 slots at a 2,048-token step, production's own.
The vendor's calibration shape (--max-num-batched-tokens 65536,
--max-num-seqs 64, ~930 tok/s) cannot boot two 32 GiB R9700s: the
activation arena alone is 12.83 GiB of the 27.95 GiB claimable, and
every variant stops in the VRAM resolver (short 3.19 GiB .. 86 MiB,
then a starved cache).
Floor moves 12M -> 1.5M tokens: the vendor's own measurement is 'a
pooled Gram from 10M tokens beat one from 1M by under 1%', and 1.5M is
hours at this rig's rate instead of the 29 h a 6M floor cost.
The converter moves to the same digest: rad-convert's gptq:calib=<dir>
input is version-stable, so a latest-written tap converts on latest.
dank
changed title from fix(qwen38): pin calib to 1.2.0, gate the tap's disk, kill orphaned engines to fix(qwen38): the calibration tap runs on latest radiance, not the production pin2026-10-30 15:16:42 -07:00
dank
merged commit 7b2b5b66fe into main2026-10-30 15:30:21 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What
The calibration tap — the layer Hessians
rad-convertreads throughgptq:calib=— only arms onlatest radiance. The production pin carries the code and the plugin strings and never writes a byte.
scripts/run-qwen38-bf16.sh:CALIB_IMAGE->stilldeadcode/radiance@sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093(
:lateston 2026-10-30 = v1.14.0, revision4a61d7296bc5), with the law in the header comment;the tap floor moves 12M -> 1.5M tokens (the vendor's own measurement is that 10M beats 1M by under 1%).
scripts/convert-qwen38-w5nl64.sh: the converter rides the same digest — the tap's file contract(
RADGRAM2,gram.r<rank>.<stem>.bin) is version-stable, so a latest-written tap converts on latest.tests/qwen38_reference.bats: the calib dry run asserts the 1.5M floor and the latest pin; thecalibration-window dry-run test passes
Q38_CALIB_MIN_FREE_MIB=1because its fixture lands on/tmp(a 16 GB tmpfs) and the 60 GiB seal guard can never pass there — pre-existing andenvironment-bound, unrelated to this change.
Evidence
/health,/statsand thepush all reported healthy.
I[0] recorded passes: off, the program accumulates calibration Grams (gram_accum)and thencalibration wrote layer Hessians after 9129 tokens— 24,768 gramfiles, 42.9 GB.
--max-num-batched-tokens 65536 --max-num-seqs 64, ~930 tok/s)cannot boot two 32 GiB R9700s: the activation arena alone is 12.83 GiB of 27.95 GiB claimable.
Wider steps are refused by the arch itself —
qwen4exp: the QSA score plane would be 3125 MiB — 16384 rows by --max-model-len 200000 / compress ratio 4— so the proven 2,048-token step runs.gets
429 the server is shutting down(success, not failure).Testing
./tests/run.sh-> 589/589 pass. The tap run itself is executing on dankserver against:latest;its artifacts and logs live in
/home/dank/campaign/qwen38-w5nl64(README there is the operating record).cwd: /home/blank/.omp/wt/wt-20261027-142335-6d1fc1f · session: 01a17f3f-3e2e-7509-aef8-be516c27cf98
fix(qwen38): pin calib to 1.2.0, gate the tap's disk, kill orphaned enginesto fix(qwen38): the calibration tap runs on latest radiance, not the production pin