69 lines
3.8 KiB
Markdown
69 lines
3.8 KiB
Markdown
---
|
||
name: radiance-calib-window-orchestrator
|
||
description: "Run the fleet's radiance calibration window through the repo orchestrator (scripts/qwen38-calib-window.sh): its gates, the arming proof, the seal contract, and the traps that make a tap look healthy while writing nothing."
|
||
---
|
||
|
||
# Run the calibration window through the orchestrator, never a hand-rolled driver
|
||
|
||
`serverAppConfigs/scripts/qwen38-calib-window.sh` is the only sanctioned driver for a radiance
|
||
calibration tap. Run it from the deploy checkout on the inference host:
|
||
|
||
```sh
|
||
cd /home/dank/serverAppConfigs
|
||
bash scripts/qwen38-calib-window.sh # add --dry-run to print the plan only
|
||
```
|
||
|
||
## What it owns (do not re-implement any of it)
|
||
|
||
| Gate | Detail |
|
||
|---|---|
|
||
| Inputs | reference recording `kld-ref/kld.json`, the BF16 `.rad`, the corpus (`Q38_WINDOW_CALIB_CORPUS`, default `calib-7168.jsonl` — the re-chunked file the rig's pool can hold) |
|
||
| **Space** | tap dir must have `Q38_CALIB_MIN_FREE_MIB` (default **61440** MiB) free **before a card is touched** |
|
||
| Rig | boots `scripts/run-qwen38-bf16.sh calib`, waits `/health` within `Q38_WINDOW_HEALTH_TIMEOUT` (3600 s), else kills the rig, restores the service, exits non-zero |
|
||
| Marker | the runner writes `calib/.rig-boot`; `calib-push.py` refuses without it, refuses if it names another container, refuses if that container is not running |
|
||
| Bank | the BF16 reference container moves to `~/.AGENTTRASH/<date>-qwen38-bf16-rad/` **only** if both phases exit 0 |
|
||
|
||
A hand-rolled `podman run` + push loop skips the space gate and the corpus choice — that is how a
|
||
43 GB seal lands on a volume with 35 GB free.
|
||
|
||
## Prove the tap ARMS before spending an hour
|
||
|
||
A build can carry every `RADIANCE_CALIB_*` string, seal message and plugin, and still never arm.
|
||
The arming proof is the boot ledger line:
|
||
|
||
```
|
||
I[0] recorded passes: off, the program accumulates calibration Grams (gram_accum)
|
||
```
|
||
|
||
No such line ⇒ nothing will be written, however healthy `/health`, `/stats` and the push look
|
||
(measured: 17 h, zero bytes, on a pin that lacked it). Diff the *sources* first if in doubt — the tap
|
||
code can be identical across versions while one shipped build drops it.
|
||
|
||
## Reading a live tap run
|
||
|
||
- **A quiet log is not a hang.** The engine logs one line per *completed* request. At 2–4-way
|
||
concurrency a 7.5 K-token prefill takes 4–8 minutes; `kv_tokens_used` steps in 2,048-token
|
||
chunks and `W step: nothing has been logged for 21 s` is step-level silence. Count completed
|
||
requests (`req cmpl`), never log mtime — a log-mtime watchdog kills healthy cycles, and every
|
||
reboot discards the accumulated tap.
|
||
- **Throughput is capped by expert streaming** (~60–70 tok/s on a disk-tier payload); concurrency
|
||
lengthens requests instead of raising throughput.
|
||
- **A port that answers is not the rig.** A dying production process answers `:7770` for seconds
|
||
after the stop. Gate on `podman ps` naming the rig container.
|
||
|
||
## The seal contract
|
||
|
||
- One seal writes **~40 GiB** (24,768 `gram.r<rank>.blk.<L>.ffn_down_exps.<E>.bin` files).
|
||
- `RADIANCE_CALIB_TOKENS` is a **pooled floor**: the engine sums Grams over prefill tokens and writes
|
||
at that count (engine default 65,536; the fleet runner defaults 1,500,000). The vendor's own
|
||
measurement: 10 M tokens beat 1 M by under 1 %.
|
||
- After writing, **the engine exits by itself**; in-flight clients get
|
||
`429 rate_limit_error: the server is shutting down` — that is success, not failure.
|
||
|
||
## Card exclusivity
|
||
|
||
A leftover engine keeps `/dev/kfd`; the next boot dies `E startup stopped: device error` with the
|
||
reason buried above the startup report. Kill by pid — `for p in $(sudo fuser /dev/kfd); do kill -9 $p; done` —
|
||
and never `pkill -f <name>` (the pattern matches your own command line). Verify the device is empty
|
||
before any boot and after any kill.
|