command-center/agent/managed-skills/radiance-tap-run-observability/SKILL.md

name, description

name	description
radiance-tap-run-observability	Judge a live radiance calibration-tap run: which pin actually arms the tap, why a quiet engine log is not a hang, the watchdog that does not throw away accumulated grams, the seal's side effects, and the container-vs-port gate before pushing.

Judging a live radiance calibration-tap run

Use when a tap (layer Hessians for rad-convert's gptq:calib=) is running for hours on a drained endpoint and you must tell progress from a stall without destroying the accumulation.

Which image actually arms the tap

• Verify :latest by digest, not by tag or by a local label. podman pull then podman inspect … --format '{{.Labels}} {{.Created}}': a working box can hold a stale local :latest whose label reads version=dev/revision=unknown while the registry's latest is a real release (on 2026-10-30 that was v1.14.0, revision 4a61d7296bc5). Compare the two before blaming a version.
• The arming line is the only ground truth that the tap is in the program: I[0] recorded passes: off, the program accumulates calibration Grams (gram_accum). Its absence means no tap, however healthy /health, /stats and the pusher look — that is how a 17-hour run on a production pin wrote zero bytes while everything reported fine.
• Reading source proves nothing about a build: the tap code was byte-identical in the version that arms and the one that never writes. Judge the shipped image.

A quiet log is not a hang

• The engine logs one line per completed request (I req cmpl-… prompt=<n> tok). At 60–70 tok/s of prefill a 7.5 K-token document takes ~2 min alone, ~4 min at 2-way and ~8 min at 4-way concurrency, and nothing is logged in between.
• Concurrently, W step: nothing has been logged for 21 s fires on step-level silence. It is a diagnostic, not a verdict: /health stays 200, /stats keeps reporting running: N, and kv_tokens_used still steps in batch-sized chunks.
• So a watchdog keyed on log mtime kills healthy cycles, and every kill discards the accumulation (grams only materialise at the seal). Key the watchdog on the completed-request count and give it a threshold far above the worst legitimate request latency (45 min for a 1.5 M-token floor).
• Cheap liveness triple: grep -c "req cmpl" log climbing, kv_tokens_used moving, podman ps showing the rig container. Throughput is bounded by expert streaming, so raising concurrency lengthens requests without raising total tokens/s.

The seal and what it does

• One seal is ~43 GB (24,768 gram.r<rank>.blk.<L>.ffn_down_exps.<e>.bin files) — size the tap filesystem before starting, and rotate any previous tap out first.
• The engine exits by itself when the tap seals. A client mid-flight then sees 429 rate_limit_error: the server is shutting down: that is success, not failure.
• The seal line is radiance: calibration wrote layer Hessians after <n> tokens to /data/calib. Since the tap materialises all at once, "grams > 1000 files" is a valid post-seal trigger for a chained imatrix → convert pipeline.

Gate on the container, not the port

After a script stops the production service, the dying process still answers :7770 for seconds, so a health check can pass against the wrong thing. Require podman ps to name the rig container and health to pass. A pusher that trusts the port alone pushes into a dead rig.

Cards must be exclusive

An orphaned engine keeps /dev/kfd, and the next container boot dies with E startup stopped: device error — which looks like a bad argument and is really a busy device. Kill by pid from sudo fuser /dev/kfd, never pkill -f; verify fuser is empty before the next boot. A genuinely wedged driver is a different failure: the production service then fails to serve too, and only a reboot clears it — so prove the card healthy by restarting production and taking one real chat turn before blaming your own shape.