---
name: radiance-tap-arm-and-seal-verify
description: "Prove quickly whether a radiance build writes calibration Grams (arming line, 8k-floor seal, engine exits on seal, 43 GB seal, GPU-exclusivity preflight, pgrep/health gate traps)."
---

# Verify a radiance calibration tap in 15 minutes

Prove whether a radiance build writes calibration Grams (`RADIANCE_CALIB_DIR`) before
committing hours of drained serving to it. The whole answer is two log lines and a
directory size.

## The two lines that decide it

1. **Arming** — on the boot ledger, rank 0:
   `recorded passes: off, the program accumulates calibration Grams (gram_accum)`
   Absent = the tap is not in the program: nothing will ever be written, and
   `/health`, `/stats` and a healthy push prove nothing. A 17-hour run on the
   production pin (radiance 1.8.1) wrote **zero bytes** with no arming line.
2. **Seal** — when the pooled token floor is reached:
   `radiance: calibration wrote layer Hessians after N tokens to /data/calib`
   printed once per rank. Then the directory holds `gram.r<rank>.<block>.<stem>.bin`
   files (~24,768 files, ~43 GB). Check `du -sb`; a plausible-looking empty dir is
   the failure mode.

## The seal ends the server

The engine **shuts itself down** once it seals. In-flight requests then receive
`429 rate_limit_error: the server is shutting down`. That is SUCCESS, not an error:
wait for the seal line, and treat a post-seal 429 as done. Do not retry into it.

## Minimum proof run

- Boot the candidate image with `RADIANCE_CALIB_DIR=/data/calib` and
  `RADIANCE_CALIB_TOKENS=8192` (a small floor, not the campaign floor).
- Push three or four ~7k-token documents concurrently; at ~64 tok/s aggregate the
  seal lands in ~2 minutes.
- One seal costs ~43 GB: point the tap at a filesystem with ≥60 GB free and check
  before booting.
- Consumer side is version-stable: `rad-convert` takes `gptq:calib=<dir>` and reads
  `RADGRAM2` grams written by any pin, so the producer and the converter may be
  different builds.

## Shape: the vendor recipe does not fit two 32 GiB cards

Vendor `docs/MOE-W4.md` prescribes `--max-num-batched-tokens 65536 --max-num-seqs 64`
(~930 tok/s). On two 32 GiB R9700s the activation arena alone is 12.83 GiB of the
27.95 GiB claimable and every variant stops in the VRAM resolver.
The qwen4exp arch prices a **QSA score plane** of `rows × window / compress`:
`the QSA score plane would be 3125 MiB -- 16384 rows by --max-model-len 200000 / 4.
Lower either.` → 16,384 rows is refused outright; 8,192 rows costs ~1.56 GiB.
The shape that fits and seals: production's own — 4 slots, `--max-num-batched-tokens
2048`, fp8 KV, `expert_tiered` + weights disk tier, n-gram table on disk.

## Exclusivity is not optional

A leftover calibration container silently owns the cards and poisons every later
boot with `E startup stopped: device error`. Before any boot:

```
sudo fuser /dev/kfd            # must print nothing
podman ps --format '{{.Names}}'
```

After stopping a rig, `podman rm -f <name>` **and** kill any `radiance` pid still
holding `/dev/kfd` (`kill -9` by the pid from `fuser`, never `pkill -f`). Then verify
`fuser` is empty before booting the next shape — a 21-second `device error` at boot
is a dirty device, not a shape refusal.

Two gate traps, both measured:
- `curl /health` right after stopping the service can be answered by the **dying**
  production process, so "up at 10 s" is a lie. Gate on the container name in
  `podman ps` **and** the marker file the rig writes, not on the port alone.
- `pgrep -f "script.sh"` matches the polling shell's own command line; a
  `while pgrep -f` wait loop then never exits. Gate on a file, a container name, or
  the pid you captured at launch.

## Report

Write the outcome into the page/README that owns the campaign, not a diary: which
build arms the tap, the fitting shape, the floor and its warrant, and the seal size.
Numbers from a shape that is not running now are dead weight — keep the law.