Files
command-center/agent/managed-skills/radiance-step-shape-ledger-audit/SKILL.md
T

2.8 KiB

name, description
name description
radiance-step-shape-ledger-audit Price a radiance serving-shape change (MNBT / max-num-seqs / window) from the engine's own boot ledger instead of a comment, and catch a mode that carries a private step.

Use when a radiance endpoint or campaign rig boots with a step/shape that differs from the one already proven, or refuses with E startup stopped: out of memory while the container image and env are unchanged.

The law

The step's activation arena is charged to the card, not to the disk tier. --max-num-batched-tokens sizes it (see vram_budget.cpp:117, the engine's own fix text: "lower --max-num-batched-tokens (the arena ...)"), so a wider step is taken out of the weight tiers and the KV cache. A comment claiming a wide step is "free throughput" is not evidence. Never let one mode of a multi-mode serve script carry its own --max-num-seqs or step; derive both from the env file that owns them (for frank-archon: scripts/frank-archon.env → FRANK_ARCHON_MAX_SEQS, FRANK_ARCHON_MAX_BATCHED_TOKENS, FRANK_ARCHON_HOST_POOL_MIB, FRANK_ARCHON_GPU_HEADROOM_MIB).

The audit (one ssh call, two logs)

Radiance writes the ledger to the file named beside the serve script (/home/dank/logs/qwen38_bf16_<mode>.log, /home/dank/logs/frank_archon.log) — not the journal. Extract, per boot:

for f in <proven_mode> <failing_mode>; do L=/home/dank/logs/qwen38_bf16_$f.log
  for k in "--max-model-len" "--max-num-seqs" "--max-num-batched-tokens" \
           "--host-pool-mib" "--tp " "--kv-cache-dtype" "--placement"; do
    grep -m1 -E "^I +\\${k%}" "$L" | head -1; done
  grep -m1 "activation arena" "$L"; grep -m1 "claimable" "$L"
  grep -m1 "vram_kv" "$L"; grep -m1 "weights by tier" "$L"
  grep -m1 "full-length sessions" "$L"
done

Compare row by row: activation arena → claimable → vram_kv → weights by tier (VRAM vs container-on-disk). The refusal line names the shape it could not back, e.g. kv cache 271.25 MiB ... 0% backing of the 8 x 32768 worst case: 0.0 full-length sessions.

Reading the rows

  • Arena grows with MNBT and is subtracted before weights/KV are funded (measured: 1,024 → 504 MiB arena, 30.87 GiB claimable; 65,536 → 10.65 GiB arena, 16.85 GiB claimable, experts squeezed to 10.98 GiB, 56 GiB of weights stranded on the disk tier).
  • vram_kv below one full-length sequence for the requested seats ⇒ OOM refusal, even when a narrower shape at the same window booted.
  • expert prefetch 0 B / "no lender" means the engine already gave the expert slab everything the arena left.

Then

Fix by inheriting the proven numbers, delete the per-mode literals (no shim, no FRANK_*_MAX_SEQS alias), update the mode's own comment so the false premise is destroyed rather than re-argued, and prove with <script> <mode> --dry-run that the argv matches the proven mode byte for byte.