64 lines
3.9 KiB
Markdown
64 lines
3.9 KiB
Markdown
---
|
||
name: radiance-tap-run-observability
|
||
description: "Judge a live radiance calibration-tap run: which pin actually arms the tap, why a quiet engine log is not a hang, the watchdog that does not throw away accumulated grams, the seal's side effects, and the container-vs-port gate before pushing."
|
||
---
|
||
|
||
# Judging a live radiance calibration-tap run
|
||
|
||
Use when a tap (layer Hessians for `rad-convert`'s `gptq:calib=`) is running for hours on a drained
|
||
endpoint and you must tell progress from a stall without destroying the accumulation.
|
||
|
||
## Which image actually arms the tap
|
||
|
||
- **Verify `:latest` by digest, not by tag or by a local label.** `podman pull` then
|
||
`podman inspect … --format '{{.Labels}} {{.Created}}'`: a working box can hold a *stale* local
|
||
`:latest` whose label reads `version=dev/revision=unknown` while the registry's `latest` is a real
|
||
release (on 2026-10-30 that was v1.14.0, revision `4a61d7296bc5`). Compare the two before blaming
|
||
a version.
|
||
- The arming line is the only ground truth that the tap is in the program:
|
||
`I[0] recorded passes: off, the program accumulates calibration Grams (gram_accum)`.
|
||
Its absence means no tap, however healthy `/health`, `/stats` and the pusher look — that is how a
|
||
17-hour run on a production pin wrote zero bytes while everything reported fine.
|
||
- Reading source proves nothing about a build: the tap code was byte-identical in the version that
|
||
arms and the one that never writes. Judge the shipped image.
|
||
|
||
## A quiet log is not a hang
|
||
|
||
- The engine logs **one line per completed request** (`I req cmpl-… prompt=<n> tok`). At 60–70 tok/s
|
||
of prefill a 7.5 K-token document takes ~2 min alone, ~4 min at 2-way and ~8 min at 4-way
|
||
concurrency, and **nothing is logged in between**.
|
||
- Concurrently, `W step: nothing has been logged for 21 s` fires on *step*-level silence. It is a
|
||
diagnostic, not a verdict: `/health` stays 200, `/stats` keeps reporting `running: N`, and
|
||
`kv_tokens_used` still steps in batch-sized chunks.
|
||
- So a watchdog keyed on log mtime kills healthy cycles, and every kill discards the accumulation
|
||
(grams only materialise at the seal). **Key the watchdog on the completed-request count** and give
|
||
it a threshold far above the worst legitimate request latency (45 min for a 1.5 M-token floor).
|
||
- Cheap liveness triple: `grep -c "req cmpl" log` climbing, `kv_tokens_used` moving, `podman ps`
|
||
showing the rig container. Throughput is bounded by expert streaming, so raising concurrency
|
||
lengthens requests without raising total tokens/s.
|
||
|
||
## The seal and what it does
|
||
|
||
- One seal is ~43 GB (24,768 `gram.r<rank>.blk.<L>.ffn_down_exps.<e>.bin` files) — size the tap
|
||
filesystem before starting, and rotate any previous tap out first.
|
||
- The engine **exits by itself when the tap seals**. A client mid-flight then sees
|
||
`429 rate_limit_error: the server is shutting down`: that is success, not failure.
|
||
- The seal line is `radiance: calibration wrote layer Hessians after <n> tokens to /data/calib`.
|
||
Since the tap materialises all at once, "grams > 1000 files" is a valid post-seal trigger for a
|
||
chained imatrix → convert pipeline.
|
||
|
||
## Gate on the container, not the port
|
||
|
||
After a script stops the production service, the dying process still answers `:7770` for seconds, so
|
||
a health check can pass against the wrong thing. Require `podman ps` to name the rig container
|
||
**and** health to pass. A pusher that trusts the port alone pushes into a dead rig.
|
||
|
||
## Cards must be exclusive
|
||
|
||
An orphaned engine keeps `/dev/kfd`, and the next container boot dies with
|
||
`E startup stopped: device error` — which looks like a bad argument and is really a busy device.
|
||
Kill by pid from `sudo fuser /dev/kfd`, never `pkill -f`; verify `fuser` is empty before the next
|
||
boot. A genuinely wedged driver is a different failure: the *production* service then fails to serve
|
||
too, and only a reboot clears it — so prove the card healthy by restarting production and taking one
|
||
real chat turn before blaming your own shape.
|