command-center/agent/managed-skills/radiance-calib-window-orchestrator/SKILL.md

name, description

name	description
radiance-calib-window-orchestrator	Run the fleet's radiance calibration window through the repo orchestrator (scripts/qwen38-calib-window.sh): its gates, the arming proof, the seal contract, and the traps that make a tap look healthy while writing nothing.

Run the calibration window through the orchestrator, never a hand-rolled driver

serverAppConfigs/scripts/qwen38-calib-window.sh is the only sanctioned driver for a radiance calibration tap. Run it from the deploy checkout on the inference host:

cd /home/dank/serverAppConfigs
bash scripts/qwen38-calib-window.sh          # add --dry-run to print the plan only

What it owns (do not re-implement any of it)

Gate	Detail
Inputs	reference recording kld-ref/kld.json, the BF16 .rad, the corpus (Q38_WINDOW_CALIB_CORPUS, default calib-7168.jsonl — the re-chunked file the rig's pool can hold)
Space	tap dir must have Q38_CALIB_MIN_FREE_MIB (default 61440 MiB) free before a card is touched
Rig	boots scripts/run-qwen38-bf16.sh calib, waits /health within Q38_WINDOW_HEALTH_TIMEOUT (3600 s), else kills the rig, restores the service, exits non-zero
Marker	the runner writes calib/.rig-boot; calib-push.py refuses without it, refuses if it names another container, refuses if that container is not running
Bank	the BF16 reference container moves to ~/.AGENTTRASH/<date>-qwen38-bf16-rad/ only if both phases exit 0

A hand-rolled podman run + push loop skips the space gate and the corpus choice — that is how a 43 GB seal lands on a volume with 35 GB free.

Prove the tap ARMS before spending an hour

A build can carry every RADIANCE_CALIB_* string, seal message and plugin, and still never arm. The arming proof is the boot ledger line:

I[0] recorded passes: off, the program accumulates calibration Grams (gram_accum)

No such line ⇒ nothing will be written, however healthy /health, /stats and the push look (measured: 17 h, zero bytes, on a pin that lacked it). Diff the sources first if in doubt — the tap code can be identical across versions while one shipped build drops it.

Reading a live tap run

• A quiet log is not a hang. The engine logs one line per completed request. At 2–4-way concurrency a 7.5 K-token prefill takes 4–8 minutes; kv_tokens_used steps in 2,048-token chunks and W step: nothing has been logged for 21 s is step-level silence. Count completed requests (req cmpl), never log mtime — a log-mtime watchdog kills healthy cycles, and every reboot discards the accumulated tap.
• Throughput is capped by expert streaming (~60–70 tok/s on a disk-tier payload); concurrency lengthens requests instead of raising throughput.
• A port that answers is not the rig. A dying production process answers :7770 for seconds after the stop. Gate on podman ps naming the rig container.

The seal contract

• One seal writes ~40 GiB (24,768 gram.r<rank>.blk.<L>.ffn_down_exps.<E>.bin files).
• RADIANCE_CALIB_TOKENS is a pooled floor: the engine sums Grams over prefill tokens and writes at that count (engine default 65,536; the fleet runner defaults 1,500,000). The vendor's own measurement: 10 M tokens beat 1 M by under 1 %.
• After writing, the engine exits by itself; in-flight clients get 429 rate_limit_error: the server is shutting down — that is success, not failure.

Card exclusivity

A leftover engine keeps /dev/kfd; the next boot dies E startup stopped: device error with the reason buried above the startup report. Kill by pid — for p in $(sudo fuser /dev/kfd); do kill -9 $p; done — and never pkill -f <name> (the pattern matches your own command line). Verify the device is empty before any boot and after any kill.