live: heartbeat sync 2026-10-30T16:20:49-07:00 (Refs #11)

agent/managed-skills/radiance-calibration-tap-latest/SKILL.md

@@ -0,0 +1,85 @@
+---
+name: radiance-calibration-tap-latest
+description: "Capture radiance calibration Grams for GPTQ (RADIANCE_CALIB_DIR) on the R9700 rig: the tap only arms on latest, the fitting shape, the 12M floor rule, and how to read a run that looks hung."
+---
+
+Friction-earned 2026-10-30 on `dank/serverAppConfigs` (qwen38-w5nl64 campaign, 2×R9700 TP=2).
+
+## The tap arms only on the latest radiance image
+
+A 17-hour calib run on the *production* pin (1.8.1) wrote **zero bytes** into `RADIANCE_CALIB_DIR`
+while `/health`, `/stats` and the pusher all reported healthy. The tap's arming code is byte-identical
+in both trees — the shipped 1.8.1 build drops it. The tell is the boot line:
+
+```
+I[0] recorded passes: off, the program accumulates calibration Grams (gram_accum)
+```
+
+Present on latest (v1.14.0 = `:latest` 2026-10-30), absent on 1.8.1. Verify with
+`podman inspect IMG --format '{{json .Labels}}'` → `org.opencontainers.image.version` /
+`revision`, and pin the **index digest**, not the tag: `:latest` moves under you, and a local stale
+`:latest` can be a 3-week-old `dev` build (check `.Created`).
+
+Run the tap from a report container of the *same* model, but the tap needs `--ngram-placement disk`
+for a calibration boot: the n-gram table pinned to RAM (`ram`) OOMs the rig's boot
+(`startup stopped: out of memory`) while kldref/score boots succeed at the same pool.
+
+## The fitting shape (vendor recipe does not boot 2×32 GiB)
+
+The vendor's calibration line (`--max-num-batched-tokens 65536 --max-num-seqs 64`, ~930 tok/s)
+assumes a bigger box: the activation arena alone is 12.83 GiB of 27.95 GiB claimable and every
+variant stops in the VRAM resolver (short 3.19 GiB → 279 MiB → 59 MiB → 86 MiB → cache starved).
+Wider steps are refused by the arch itself, pre-allocation:
+
+```
+qwen4exp: the QSA score plane would be 3125 MiB -- 16384 rows (--max-num-batched-tokens 16384)
+          by --max-model-len 200000 / compress ratio 4. Lower either.
+```
+
+What boots and seals: 4 slots, `--max-num-batched-tokens 2048` (the file-tier ceiling),
+`--kv-cache-dtype fp8`, `--weights-disk-tier`, `--placement expert_tiered`,
+`--ngram-placement disk`, `--max-model-len 200000`, `--num-speculative-tokens 0`, `--deterministic`.
+Ledger at that shape: `elastic 22.8 GiB — experts first, kv cache 1.50 GiB … 1.0 full-length`
+(pool ≈ 199,948 tokens). On 1.8.1 the same payload's pool was 7,436 tokens — the resolver, not the
+window, is what changed.
+
+## Floor
+
+`RADIANCE_CALIB_TOKENS` is a **pooled** token floor. Engine default 65,536. Fleet rule (Frank,
+2026-10-30): **12,000,000**, because the served quant this arm competes with was calibrated on a
+10 M-token set. The vendor's "10 M beats 1 M by under 1 %" measures *their* plate, not ours.
+Consequence: a ~9.3 M-token corpus round seals in round 1 plus part of round 2, and a round that
+seals nothing is NORMAL (`round_verdict` → continue, never failure).
+
+## Reading a run (the trap that cost two cycles)
+
+- Throughput is **streaming-bound**, ~60–70 tok/s whatever the slot count; concurrency only lengthens
+  each request (2-way ≈ 4 min per 7.5 K-token prefill, 4-way ≈ 8).
+- The engine logs **one line per completed request**. A log quiet for 8 minutes is therefore normal;
+  `W step: nothing has been logged for 21 s` is step-level silence. Liveness = **completed-request
+  count**, never log mtime. A watchdog on mtime kills healthy cycles and every false positive throws
+  away accumulated tap tokens.
+- One seal writes **~43 GB** (24,768 `gram.r<rank>.blk.<l>.ffn_down_exps.<e>.bin`) and the engine
+  then **exits by itself**; a mid-flight client gets `429 rate_limit_error: the server is shutting
+  down` — success, not failure.
+- A port that answers is not the rig: after the runner stops production the dying process answers
+  `:7770` briefly. Gate on `podman ps` naming the rig container, and on the `.rig-boot` marker.
+
+## Orchestration and space
+
+Use the repo's orchestrator, never a hand-rolled driver:
+`bash scripts/qwen38-calib-window.sh` (gates: reference present, `.rad` present, tap dir ≥ 60 GiB,
+healthy rig, marker, then push; banks the reference container only on success). Knobs:
+`FRANK_QWEN38_CALIB_TOKENS`, `Q38_WINDOW_CALIB_CORPUS`, `Q38_WINDOW_ROUNDS`,
+`Q38_CALIB_MIN_FREE_MIB`, `Q38_WINDOW_KEEP_RAD`.
+
+A hand-rolled driver skips the space preflight; on this host the tap dir sat on `/` with 35 G free
+while one seal writes 43 GB — the orchestrator refuses that, which is the point. Frank's rule: keep
+artifacts on `/`, never `/mnt/entropy`.
+
+## Then
+
+`gram_files → scripts/grams-to-imatrix.py DIR -o imatrix.gguf` (stdlib-only, skips non-gram names
+like `.rig-boot`) → `scripts/convert-qwen38-w5nl64.sh --plan-only` (space bill, `--plan-only` first)
+→ convert (card-less, CPU BLAS) → `score` each arm + the served payload at the **production shape** →
+`scripts/kld-gate-qwen38.sh` (mean KL ≤ 0.03, p99.9, top-1 ≥ 95 %, every `by_source` slice).