86 lines
4.4 KiB
Markdown
86 lines
4.4 KiB
Markdown
---
|
||
name: radiance-calib-window-serve-gate
|
||
description: "Diagnose a radiance calibration window that dies at second zero or refuses its push: the health answer coming from the dying production socket, the listener-identity gate, the preflight trio (space, re-chunked corpus, KEEP_RAD), the 12M floor and the wide-step knob."
|
||
---
|
||
|
||
# radiance calibration window — the serve gate and its preflight
|
||
|
||
Friction-earned 2026-10-30 on `dank/serverAppConfigs` (`scripts/qwen38-calib-window.sh`,
|
||
`scripts/run-qwen38-bf16.sh`). A 12 M-token window died at second zero; the cause was the gate,
|
||
not the engine.
|
||
|
||
## The signature, and what it really means
|
||
|
||
```
|
||
phase=calib health ok waited=0s
|
||
the rig container qwen38-bf16 is not running <- pusher refuses
|
||
phase=calib push end rc=2 ... FAIL: serve rc=1 push rc=2
|
||
```
|
||
|
||
`waited=0s` is the tell. The runner stops `frank-archon` **first**; the dying production process keeps
|
||
its listener for a moment, while the freshly created rig container is already `State.Running=true` but
|
||
has not bound the port yet. So `podman inspect` (container running) **and** `/health` 200 (production's
|
||
socket) both hold for the wrong process.
|
||
|
||
**The gate must prove the listener belongs to the rig:**
|
||
|
||
```bash
|
||
lpid=$(ss -H -ltnp 'sport = :7770' | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)
|
||
[ -n "$lpid" ] || return 1
|
||
podman top "$RIG" pid | grep -qw "$lpid"
|
||
```
|
||
|
||
Landed as PR #90 with two red-first tests (foreign listener ⇒ no push, window fails; in-container
|
||
listener ⇒ proceeds and banks). If you see `waited=0s` on a checkout older than that PR, the window
|
||
there is pre-fix: do not run a long window on it.
|
||
|
||
## Preflight trio — the window refuses correctly, so do not "fix" it
|
||
|
||
1. **Tap dir ≥ 60 GiB free.** One seal writes **43 GB in 24,768 files**
|
||
(`gram.r<rank>.blk.<l>.ffn_down_exps.<e>.bin`). Point it at a 35 GiB root and the seal fills the
|
||
filesystem to zero. `Q38_CALIB_MIN_FREE_MIB` overrides — for *tests only* (fixtures on `/tmp` tmpfs).
|
||
2. **The re-chunked corpus.** `Q38_WINDOW_CALIB_CORPUS` defaults to `calib-7168.jsonl`; pushing
|
||
`calib.jsonl` instead runs un-chunked documents through a window sized for chunks.
|
||
3. **`Q38_WINDOW_KEEP_RAD=1` when two windows run back to back.** The window banks the BF16 reference
|
||
container into `~/.AGENTTRASH/` on success; without this, stage 2 has no reference to tap.
|
||
|
||
## Floor and the "add more tokens later" trap
|
||
|
||
`RADIANCE_CALIB_TOKENS` is the campaign's number, not the vendor's: the reference container this arm
|
||
competes with was calibrated on 10 M tokens, so the fleet floor is **12 M**. Engine default is 65,536.
|
||
|
||
**A sealed tap cannot be extended.** Hessians are VRAM accumulators; the seal writes and drops them.
|
||
So "run 1 M to verify, then add 11 M" is two windows — 1 M then a fresh 12 M — not an append. Budget
|
||
accordingly, and rotate the verification tap to `.AGENTTRASH/`.
|
||
|
||
One round of the re-chunked corpus is **9.31 M tokens**, so a 12 M floor seals in round 1 plus part
|
||
of round 2; a flat round is normal (the floor is pooled; `round_verdict` says "continue").
|
||
|
||
## The wide-step knob (the tap's real cost)
|
||
|
||
`Q38_CALIB_MNBT` / `Q38_CALIB_SEQS` set the calibration step and slots; everything else inherits
|
||
production. The tap is **read-bound**: a 7,168-token document cut into four 2,048-token steps
|
||
re-streams each layer's routed expert union four times, so a wider step is worth up to ~4× (the
|
||
runner's own header says this).
|
||
|
||
Wide steps are refused by the arch, not the budget, and the refusal scales with the window:
|
||
|
||
```
|
||
QSA score plane = rows x (--max-model-len / 4) x 4 bytes
|
||
16,384 rows @ 200,000 window -> 3,125 MiB -> REFUSED at boot
|
||
16,384 rows @ 32,768 window -> 512 MiB -> legal
|
||
```
|
||
|
||
The calibration corpus is chunked to 7,168 tokens, so **32,768 is the window to calibrate at** —
|
||
copying production's 200,000 window is what disqualifies the wide step.
|
||
|
||
## Judging a run
|
||
|
||
- **A quiet log is not a hang.** The engine logs one line per *completed* request; at 2-way a 7.5 K
|
||
prefill takes ~4 min. Watch `kv_tokens_used` (steps of the chunk size) and the `req cmpl` count.
|
||
A watchdog on log mtime kills healthy cycles and throws away accumulated tokens.
|
||
- **Sealing is success.** The engine exits by itself; a mid-flight client gets
|
||
`429 rate_limit_error: the server is shutting down`.
|
||
- **Kill by pid** (`sudo fuser /dev/kfd`), never `pkill -f`: an orphaned engine keeps the device and
|
||
the next boot dies in ~20 s with `E startup stopped: device error`, which reads like a bad shape.
|