82 lines
3.9 KiB
Markdown
82 lines
3.9 KiB
Markdown
---
|
||
name: radiance-tap-arm-and-seal-verify
|
||
description: "Prove quickly whether a radiance build writes calibration Grams (arming line, 8k-floor seal, engine exits on seal, 43 GB seal, GPU-exclusivity preflight, pgrep/health gate traps)."
|
||
---
|
||
|
||
# Verify a radiance calibration tap in 15 minutes
|
||
|
||
Prove whether a radiance build writes calibration Grams (`RADIANCE_CALIB_DIR`) before
|
||
committing hours of drained serving to it. The whole answer is two log lines and a
|
||
directory size.
|
||
|
||
## The two lines that decide it
|
||
|
||
1. **Arming** — on the boot ledger, rank 0:
|
||
`recorded passes: off, the program accumulates calibration Grams (gram_accum)`
|
||
Absent = the tap is not in the program: nothing will ever be written, and
|
||
`/health`, `/stats` and a healthy push prove nothing. A 17-hour run on the
|
||
production pin (radiance 1.8.1) wrote **zero bytes** with no arming line.
|
||
2. **Seal** — when the pooled token floor is reached:
|
||
`radiance: calibration wrote layer Hessians after N tokens to /data/calib`
|
||
printed once per rank. Then the directory holds `gram.r<rank>.<block>.<stem>.bin`
|
||
files (~24,768 files, ~43 GB). Check `du -sb`; a plausible-looking empty dir is
|
||
the failure mode.
|
||
|
||
## The seal ends the server
|
||
|
||
The engine **shuts itself down** once it seals. In-flight requests then receive
|
||
`429 rate_limit_error: the server is shutting down`. That is SUCCESS, not an error:
|
||
wait for the seal line, and treat a post-seal 429 as done. Do not retry into it.
|
||
|
||
## Minimum proof run
|
||
|
||
- Boot the candidate image with `RADIANCE_CALIB_DIR=/data/calib` and
|
||
`RADIANCE_CALIB_TOKENS=8192` (a small floor, not the campaign floor).
|
||
- Push three or four ~7k-token documents concurrently; at ~64 tok/s aggregate the
|
||
seal lands in ~2 minutes.
|
||
- One seal costs ~43 GB: point the tap at a filesystem with ≥60 GB free and check
|
||
before booting.
|
||
- Consumer side is version-stable: `rad-convert` takes `gptq:calib=<dir>` and reads
|
||
`RADGRAM2` grams written by any pin, so the producer and the converter may be
|
||
different builds.
|
||
|
||
## Shape: the vendor recipe does not fit two 32 GiB cards
|
||
|
||
Vendor `docs/MOE-W4.md` prescribes `--max-num-batched-tokens 65536 --max-num-seqs 64`
|
||
(~930 tok/s). On two 32 GiB R9700s the activation arena alone is 12.83 GiB of the
|
||
27.95 GiB claimable and every variant stops in the VRAM resolver.
|
||
The qwen4exp arch prices a **QSA score plane** of `rows × window / compress`:
|
||
`the QSA score plane would be 3125 MiB -- 16384 rows by --max-model-len 200000 / 4.
|
||
Lower either.` → 16,384 rows is refused outright; 8,192 rows costs ~1.56 GiB.
|
||
The shape that fits and seals: production's own — 4 slots, `--max-num-batched-tokens
|
||
2048`, fp8 KV, `expert_tiered` + weights disk tier, n-gram table on disk.
|
||
|
||
## Exclusivity is not optional
|
||
|
||
A leftover calibration container silently owns the cards and poisons every later
|
||
boot with `E startup stopped: device error`. Before any boot:
|
||
|
||
```
|
||
sudo fuser /dev/kfd # must print nothing
|
||
podman ps --format '{{.Names}}'
|
||
```
|
||
|
||
After stopping a rig, `podman rm -f <name>` **and** kill any `radiance` pid still
|
||
holding `/dev/kfd` (`kill -9` by the pid from `fuser`, never `pkill -f`). Then verify
|
||
`fuser` is empty before booting the next shape — a 21-second `device error` at boot
|
||
is a dirty device, not a shape refusal.
|
||
|
||
Two gate traps, both measured:
|
||
- `curl /health` right after stopping the service can be answered by the **dying**
|
||
production process, so "up at 10 s" is a lie. Gate on the container name in
|
||
`podman ps` **and** the marker file the rig writes, not on the port alone.
|
||
- `pgrep -f "script.sh"` matches the polling shell's own command line; a
|
||
`while pgrep -f` wait loop then never exits. Gate on a file, a container name, or
|
||
the pid you captured at launch.
|
||
|
||
## Report
|
||
|
||
Write the outcome into the page/README that owns the campaign, not a diary: which
|
||
build arms the tap, the fitting shape, the floor and its warrant, and the seal size.
|
||
Numbers from a shape that is not running now are dead weight — keep the law.
|