--- name: radiance-calib-window-orchestrator description: "Run the fleet's radiance calibration window through the repo orchestrator (scripts/qwen38-calib-window.sh): its gates, the arming proof, the seal contract, and the traps that make a tap look healthy while writing nothing." --- # Run the calibration window through the orchestrator, never a hand-rolled driver `serverAppConfigs/scripts/qwen38-calib-window.sh` is the only sanctioned driver for a radiance calibration tap. Run it from the deploy checkout on the inference host: ```sh cd /home/dank/serverAppConfigs bash scripts/qwen38-calib-window.sh # add --dry-run to print the plan only ``` ## What it owns (do not re-implement any of it) | Gate | Detail | |---|---| | Inputs | reference recording `kld-ref/kld.json`, the BF16 `.rad`, the corpus (`Q38_WINDOW_CALIB_CORPUS`, default `calib-7168.jsonl` — the re-chunked file the rig's pool can hold) | | **Space** | tap dir must have `Q38_CALIB_MIN_FREE_MIB` (default **61440** MiB) free **before a card is touched** | | Rig | boots `scripts/run-qwen38-bf16.sh calib`, waits `/health` within `Q38_WINDOW_HEALTH_TIMEOUT` (3600 s), else kills the rig, restores the service, exits non-zero | | Marker | the runner writes `calib/.rig-boot`; `calib-push.py` refuses without it, refuses if it names another container, refuses if that container is not running | | Bank | the BF16 reference container moves to `~/.AGENTTRASH/-qwen38-bf16-rad/` **only** if both phases exit 0 | A hand-rolled `podman run` + push loop skips the space gate and the corpus choice — that is how a 43 GB seal lands on a volume with 35 GB free. ## Prove the tap ARMS before spending an hour A build can carry every `RADIANCE_CALIB_*` string, seal message and plugin, and still never arm. The arming proof is the boot ledger line: ``` I[0] recorded passes: off, the program accumulates calibration Grams (gram_accum) ``` No such line ⇒ nothing will be written, however healthy `/health`, `/stats` and the push look (measured: 17 h, zero bytes, on a pin that lacked it). Diff the *sources* first if in doubt — the tap code can be identical across versions while one shipped build drops it. ## Reading a live tap run - **A quiet log is not a hang.** The engine logs one line per *completed* request. At 2–4-way concurrency a 7.5 K-token prefill takes 4–8 minutes; `kv_tokens_used` steps in 2,048-token chunks and `W step: nothing has been logged for 21 s` is step-level silence. Count completed requests (`req cmpl`), never log mtime — a log-mtime watchdog kills healthy cycles, and every reboot discards the accumulated tap. - **Throughput is capped by expert streaming** (~60–70 tok/s on a disk-tier payload); concurrency lengthens requests instead of raising throughput. - **A port that answers is not the rig.** A dying production process answers `:7770` for seconds after the stop. Gate on `podman ps` naming the rig container. ## The seal contract - One seal writes **~40 GiB** (24,768 `gram.r.blk..ffn_down_exps..bin` files). - `RADIANCE_CALIB_TOKENS` is a **pooled floor**: the engine sums Grams over prefill tokens and writes at that count (engine default 65,536; the fleet runner defaults 1,500,000). The vendor's own measurement: 10 M tokens beat 1 M by under 1 %. - After writing, **the engine exits by itself**; in-flight clients get `429 rate_limit_error: the server is shutting down` — that is success, not failure. ## Card exclusivity A leftover engine keeps `/dev/kfd`; the next boot dies `E startup stopped: device error` with the reason buried above the startup report. Kill by pid — `for p in $(sudo fuser /dev/kfd); do kill -9 $p; done` — and never `pkill -f ` (the pattern matches your own command line). Verify the device is empty before any boot and after any kill.