38 lines
2.8 KiB
Markdown
38 lines
2.8 KiB
Markdown
---
|
|
name: radiance-step-shape-ledger-audit
|
|
description: "Price a radiance serving-shape change (MNBT / max-num-seqs / window) from the engine's own boot ledger instead of a comment, and catch a mode that carries a private step."
|
|
---
|
|
|
|
Use when a radiance endpoint or campaign rig boots with a step/shape that differs from the one already proven, or refuses with `E startup stopped: out of memory` while the container image and env are unchanged.
|
|
|
|
## The law
|
|
|
|
The step's **activation arena is charged to the card**, not to the disk tier. `--max-num-batched-tokens` sizes it (see `vram_budget.cpp:117`, the engine's own fix text: "lower --max-num-batched-tokens (the arena ...)"), so a wider step is taken out of the weight tiers and the KV cache. A comment claiming a wide step is "free throughput" is not evidence. Never let one mode of a multi-mode serve script carry its own `--max-num-seqs` or step; derive both from the env file that owns them (for frank-archon: `scripts/frank-archon.env` → `FRANK_ARCHON_MAX_SEQS`, `FRANK_ARCHON_MAX_BATCHED_TOKENS`, `FRANK_ARCHON_HOST_POOL_MIB`, `FRANK_ARCHON_GPU_HEADROOM_MIB`).
|
|
|
|
## The audit (one ssh call, two logs)
|
|
|
|
Radiance writes the ledger to the file named beside the serve script (`/home/dank/logs/qwen38_bf16_<mode>.log`, `/home/dank/logs/frank_archon.log`) — **not** the journal. Extract, per boot:
|
|
|
|
```
|
|
for f in <proven_mode> <failing_mode>; do L=/home/dank/logs/qwen38_bf16_$f.log
|
|
for k in "--max-model-len" "--max-num-seqs" "--max-num-batched-tokens" \
|
|
"--host-pool-mib" "--tp " "--kv-cache-dtype" "--placement"; do
|
|
grep -m1 -E "^I +\\${k%}" "$L" | head -1; done
|
|
grep -m1 "activation arena" "$L"; grep -m1 "claimable" "$L"
|
|
grep -m1 "vram_kv" "$L"; grep -m1 "weights by tier" "$L"
|
|
grep -m1 "full-length sessions" "$L"
|
|
done
|
|
```
|
|
|
|
Compare row by row: `activation arena` → `claimable` → `vram_kv` → `weights by tier` (VRAM vs container-on-disk). The refusal line names the shape it could not back, e.g. `kv cache 271.25 MiB ... 0% backing of the 8 x 32768 worst case: 0.0 full-length sessions`.
|
|
|
|
## Reading the rows
|
|
|
|
- Arena grows with MNBT and is subtracted before weights/KV are funded (measured: 1,024 → 504 MiB arena, 30.87 GiB claimable; 65,536 → 10.65 GiB arena, 16.85 GiB claimable, experts squeezed to 10.98 GiB, 56 GiB of weights stranded on the disk tier).
|
|
- `vram_kv` below one full-length sequence for the requested seats ⇒ OOM refusal, even when a narrower shape at the same window booted.
|
|
- `expert prefetch 0 B` / "no lender" means the engine already gave the expert slab everything the arena left.
|
|
|
|
## Then
|
|
|
|
Fix by inheriting the proven numbers, delete the per-mode literals (no shim, no `FRANK_*_MAX_SEQS` alias), update the mode's own comment so the false premise is destroyed rather than re-argued, and prove with `<script> <mode> --dry-run` that the argv matches the proven mode byte for byte.
|