---
name: radiance-tap-run-observability
description: "Judge a live radiance calibration-tap run: which pin actually arms the tap, why a quiet engine log is not a hang, the watchdog that does not throw away accumulated grams, the seal's side effects, and the container-vs-port gate before pushing."
---

# Judging a live radiance calibration-tap run

Use when a tap (layer Hessians for `rad-convert`'s `gptq:calib=`) is running for hours on a drained
endpoint and you must tell progress from a stall without destroying the accumulation.

## Which image actually arms the tap

- **Verify `:latest` by digest, not by tag or by a local label.** `podman pull` then
  `podman inspect … --format '{{.Labels}} {{.Created}}'`: a working box can hold a *stale* local
  `:latest` whose label reads `version=dev/revision=unknown` while the registry's `latest` is a real
  release (on 2026-10-30 that was v1.14.0, revision `4a61d7296bc5`). Compare the two before blaming
  a version.
- The arming line is the only ground truth that the tap is in the program:
  `I[0] recorded passes: off, the program accumulates calibration Grams (gram_accum)`.
  Its absence means no tap, however healthy `/health`, `/stats` and the pusher look — that is how a
  17-hour run on a production pin wrote zero bytes while everything reported fine.
- Reading source proves nothing about a build: the tap code was byte-identical in the version that
  arms and the one that never writes. Judge the shipped image.

## A quiet log is not a hang

- The engine logs **one line per completed request** (`I req cmpl-… prompt=<n> tok`). At 60–70 tok/s
  of prefill a 7.5 K-token document takes ~2 min alone, ~4 min at 2-way and ~8 min at 4-way
  concurrency, and **nothing is logged in between**.
- Concurrently, `W step: nothing has been logged for 21 s` fires on *step*-level silence. It is a
  diagnostic, not a verdict: `/health` stays 200, `/stats` keeps reporting `running: N`, and
  `kv_tokens_used` still steps in batch-sized chunks.
- So a watchdog keyed on log mtime kills healthy cycles, and every kill discards the accumulation
  (grams only materialise at the seal). **Key the watchdog on the completed-request count** and give
  it a threshold far above the worst legitimate request latency (45 min for a 1.5 M-token floor).
- Cheap liveness triple: `grep -c "req cmpl" log` climbing, `kv_tokens_used` moving, `podman ps`
  showing the rig container. Throughput is bounded by expert streaming, so raising concurrency
  lengthens requests without raising total tokens/s.

## The seal and what it does

- One seal is ~43 GB (24,768 `gram.r<rank>.blk.<L>.ffn_down_exps.<e>.bin` files) — size the tap
  filesystem before starting, and rotate any previous tap out first.
- The engine **exits by itself when the tap seals**. A client mid-flight then sees
  `429 rate_limit_error: the server is shutting down`: that is success, not failure.
- The seal line is `radiance: calibration wrote layer Hessians after <n> tokens to /data/calib`.
  Since the tap materialises all at once, "grams > 1000 files" is a valid post-seal trigger for a
  chained imatrix → convert pipeline.

## Gate on the container, not the port

After a script stops the production service, the dying process still answers `:7770` for seconds, so
a health check can pass against the wrong thing. Require `podman ps` to name the rig container
**and** health to pass. A pusher that trusts the port alone pushes into a dead rig.

## Cards must be exclusive

An orphaned engine keeps `/dev/kfd`, and the next container boot dies with
`E startup stopped: device error` — which looks like a bad argument and is really a busy device.
Kill by pid from `sudo fuser /dev/kfd`, never `pkill -f`; verify `fuser` is empty before the next
boot. A genuinely wedged driver is a different failure: the *production* service then fails to serve
too, and only a reboot clears it — so prove the card healthy by restarting production and taking one
real chat turn before blaming your own shape.