chore(agent): commit the machine tier and the doctrine text the checkout was already running
Landing blocker for #61 and #63: the deployment checkout is 12 commits behind and `merge --ff-only` refuses while tracked paths are dirty. Discard is not available here (workspace policy: no `checkout`/`stash` on shared state), so the dirt is committed. - `agent/APPEND_SYSTEM.md`: the point-of-use / one-work-log / PR-link doctrine, already byte-identical to `origin/main` (`cmp` against the trunk blob: no difference). No new prose — the commit exists so the trunk merge can proceed. - `agent/config.yml` and `agent/managed-skills/**`: the machine-live tier the heartbeat owns (AGENTS.md § Live-sync tiers). `ALLOW_SECRETS=1` was used, deliberately and after auditing the staged diff by hand: the only credential-shaped strings in it are two copies of `AKIA0000000000000EXAMPLE` — the inert all-zero AWS example id — and they appear inside the two skill pages that teach that exact placeholder shape. This checkout is still running the pre-#61 hook; the merged gate (value-scoped placeholder filter, `71601f7`) exempts this value, so the refusal dies with this landing. No `--no-verify` was used: the gate ran and its verdict is recorded here. Refs #61
This commit is contained in:
Frank committed
1 parent
004962ffce
commit
8af40c402b
79 files changed
+2771
-332
No files matched your search
@@ -1,111 +1,39 @@
|
||||
---
|
||||
name: radiance-kv-cache-resize
|
||||
description: "Resize the KV cache / VRAM pair on a radiance (StillDeadcode) endpoint: engine-budget-table ceiling, pool contract arithmetic, slab-slope law, ledger-first verification, free token-count probes"
|
||||
description: "Resize a radiance endpoint's KV cache/VRAM: the resolver's funding formula, which knob actually moves the carve, the pinned-RAM ledger to recompute, and the boot-ledger lines that referee the result."
|
||||
---
|
||||
|
||||
# Radiance KV / VRAM pair resize
|
||||
|
||||
Use when changing `--vram-kv-mib`, `--vram-weights-mib`, `--max-num-seqs`,
|
||||
`--max-model-len` or `--kv-cache-dtype` on a radiance endpoint
|
||||
(frank-archon pattern), or when asked "what can we do with the free VRAM".
|
||||
|
||||
## 1. The ceiling is the engine's, never a hand-picked reserve
|
||||
|
||||
`rocm-smi` free space is NOT spare capacity. The boot prints a `VRAM budget,
|
||||
per card` table; read it:
|
||||
## Method
|
||||
- The endpoint is sized by ONE env file (`scripts/<svc>.env`) plus the serve script; the boot ledger prints the split before allocating anything: `VRAM budget, per card` → `claimable` → `elastic` → `experts / kv cache`, then `kv: the cache can grow to X a card -- N tokens of context, M sessions at W`.
|
||||
- **`--vram-weights-mib` / `--vram-kv-mib` no longer exist (1.8.1).** The engine REFUSES to start on them (`core/config.cpp`), so this skill has no direct KV knob; the budget resolver derives everything from the cache bound (`--max-num-seqs` x `--max-model-len`), `--gpu-headroom-mib` and `--host-pool-mib`.
|
||||
|
||||
## The funding formula (radiance 1.8.1, source-cited — re-read the pinned tree before trusting it against another build)
|
||||
```
|
||||
card total 31.86 GiB
|
||||
vram_weights 25.00 GiB stated by --vram-weights-mib
|
||||
vram_kv 5.00 GiB stated by --vram-kv-mib
|
||||
unclaimed 477.30 MiB the headroom, the unusable above, rounding
|
||||
W VRAM budgets total 30.00 GiB of the card's 31.86 GiB, so 1.86 GiB is neither
|
||||
budgeted nor accounted. The driver's context, the activation arena and
|
||||
--gpu-headroom-mib come out of it; anything past those is throughput on the floor
|
||||
W card 0 settled at 2.01 GiB free against 209.19 MiB (the 96.00 MiB
|
||||
--gpu-headroom-mib and 113.19 MiB kept for the static region)
|
||||
capacity = card_total - already_held - drift - headroom - arena - step_staging - hw_queues - prefetch
|
||||
elastic = capacity - static_weights
|
||||
expert_floor = min(plane+spare, max(plane + 2*unit_max - host_room, 0)) # planner.cpp expert_vram_floor
|
||||
host_room = host_pool_mib * MiB (less the file tier's staging when disk tiering is on)
|
||||
kv_cap = max_num_seqs * max_model_len # DEMAND, in tokens
|
||||
kv = min(kv_cap, elastic - expert_floor) # what it actually gets
|
||||
tokens = min over paged groups of n_blocks*block_size # the printed number
|
||||
```
|
||||
Read consequence: **`--max-num-seqs` raises demand, `--host-pool-mib` raises supply, and both are the same lever** — every MiB added to the host pool lowers the expert floor one-for-one and the cache takes it (up to `kv_cap`); the leftover returns to the experts. `--max-num-seqs` alone does nothing to capacity: it raises demand while the floor is unchanged, so the printed tokens stay put. The page's "funds ~91 % of demand" table rows are that cap, not a policy percentage.
|
||||
- Derive the plane from a live ledger instead of guessing: `plane + 2*unit ≈ printed_experts + host_pool_bytes`.
|
||||
- `--gpu-headroom-mib` lowers go to the cache but the engine's settle floor is built from it — 48 MiB measured decode-neutral, 24 untested. 48 is the first rung.
|
||||
|
||||
`ceiling = (settled_free_MiB - engine_floor_MiB) + current_pair_sum`.
|
||||
Worked (dankserver, 2026-10-13): 29,120 + (2,058 − 209) = **30,969 MiB/card**.
|
||||
A pair at the ceiling settles ~330 MiB free at worst case. If a bound cannot be
|
||||
traced to a ledger line, it is a personal reserve — label it and put it in the
|
||||
bats gate as such.
|
||||
## Procedure
|
||||
1. Read `scripts/<svc>.env` for the current pool: `GPU_HEADROOM_MIB` (96), `HOST_POOL_MIB` (44,928 per rank), `MAX_SEQS` x `MAX_MODEL_LEN` (demand), `HOST_POOL_MIB`, `PLACEMENT`.
|
||||
2. Compute the target as tokens: tokens x bytes/token-per-card, where bytes/token = (cache_given_bytes / tokens) from the latest ledger. If the target exceeds `elastic - expert_floor`, ALSO lower `expert_floor` via `--host-pool-mib` (up = more room = fewer resident experts) — a bigger host pool buys cache only while experts remain on the card; once the host absorbs the whole plane, more host RAM buys nothing (measured: +640 MiB moved `vram_kv` byte-identical).
|
||||
3. Edit ONE knob (`GPU_HEADROOM_MIB`), deploy service-scoped: `./scripts/deploy.sh <host> --service <svc>`. The engine reloads everything (~2-4 min); the deployer's `LLM_WAIT` covers it.
|
||||
4. Re-read the ledger: `vram_kv` must be the new number, byte-exact or the knob lost to another claim; `mover: N slots can be lent` tells what the expert slab re-borrowed from the enlarged pool — that steal is the decode cost of the resize.
|
||||
5. Re-run the probe (`scripts/probe-frank-archon-kv.py`): depth 1.75x tokens, then the 2x refusal. Update `<svc>.md` Laws with the measured pair (tokens, given, backing %).
|
||||
6. Under-funded? Raise `HOST_POOL_MIB` in 1,024-MiB steps and redeploy; never touch demand to fake the number.
|
||||
|
||||
## 2. Pool contract (the engine enforces it at REQUEST time, not boot)
|
||||
## Pinned-RAM ledger to recompute BEFORE booting (this is where a resize kills the host)
|
||||
`total_pin = tp * HOST_POOL_MIB + PLE/n-gram table + PREFIX_CACHE_MIB + (statics + arena ~2,048) + checkpoint_store`, with `checkpoint_store = slots * 111.98 MiB` and `--checkpoint-slots auto` = `max_num_seqs * (1 + what the retention policy keeps)` — **it grows with the slot count**, so a slot bump raises the pin ledger twice (state + store). The repo's bats guard recomputes this and asserts `TTM_MIN_MIB >= ledger`; when you change pool or slots, update `TTM_MIN_MIB` (and that guard) in the same commit, or the seam fails. The serve script's own gate refuses to start below `TTM_MIN_MIB`; `ttm-preflight.sh` raises the knob to `TTM_TARGET_MIB`, so the target must clear the new floor with slack.
|
||||
|
||||
`pool_mib >= seqs*max_model_len*rate + 110*seqs + 448`
|
||||
|
||||
- rate/token/rank: **7,488 B fp8**, **14,144 B bf16** (1.89×, not 2× — the QSA
|
||||
block key is 3.25 KiB at BOTH dtypes; it is an index, not attention state).
|
||||
- +110 MiB a sequence of recurrent state (`kv_gdn_state` 54 MiB × 2 copies +
|
||||
conv/tail/ple) and 448 MiB of prefix checkpoints, dtype-independent.
|
||||
|
||||
A pool that fails this **boots, serves short chats, then 400s every long
|
||||
request** as `does not fit the 200000 token context, which leaves nothing to
|
||||
generate into` — radiance refuses, never truncates. Encode it as a bats gate.
|
||||
|
||||
`--kv-cache-dtype` takes ONLY `bf16|fp8`, one dtype for the whole cache (no
|
||||
llama.cpp-style exact-K/quantized-V pair).
|
||||
|
||||
## 3. Slab and KV are one budget; the slab is a SLOPE
|
||||
|
||||
- Idle KV VRAM is the expert cache's spillover: ledger
|
||||
`mover: N slots can be lent out of the KV cache (X GiB)`. A bigger pool steals
|
||||
residency twice over.
|
||||
- Slab MiB ↔ resident experts is linear (~1,300 slots per 1,792 MiB measured);
|
||||
the pinned host pool hands back the same slot count as the slab grows
|
||||
(16,235→17,548 slab with 8,893→7,580 pool). The pinned RAM path is the slow
|
||||
path, so slab MiB = throughput.
|
||||
- Measured law (fp8, 43.5k single-stream, Qwen3.8-Flash-Next w4):
|
||||
22,208 → 3,397, 24,000 → 3,290, 25,600 → **3,584**, 26,071 → **4,379 t/s**.
|
||||
Two points inside the ±3 % noise band are NOT a plateau — never infer a
|
||||
threshold from two arms.
|
||||
- **Decode is a separate axis.** It follows KV dtype/bandwidth and draft
|
||||
acceptance, not the slab: fp8/slab 26,071 decoded 124.1 t/s cold where
|
||||
bf16/slab 22,208 decoded 94.1 (512 tokens, 51.5k). Slot count moves the
|
||||
batch-shared per-stream floor (3 concurrent ≈ 6 t/s; solo ≈ 90–120).
|
||||
|
||||
## 4. Verify: ledger first, then live probes (no scratch server)
|
||||
|
||||
1. Deploy the service; read `podman logs <svc>`: `backs N sequences all at
|
||||
M tokens at once` is the capacity referee; check slab slots, `unclaimed`,
|
||||
`sampler: ... R rows` (= seqs × (1+spec), must stay inside the 25-row wht6
|
||||
exactness bound), `settled at … free against … floor`.
|
||||
2. Boundary probe: N concurrent prompts sized to `max_model_len - gen`, each
|
||||
with a unique marker at the head; expect N/N marker retrieval.
|
||||
3. Prefill/decode probe: repo `scripts/probe-frank-archon-kv.py` (reads the
|
||||
shape from the deployed env, so it self-resizes).
|
||||
|
||||
### Free token-count calibration (no prefill burned)
|
||||
|
||||
`/v1/completions` refuses when **prompt tokens alone** exceed the window —
|
||||
`prompt: 202,777 prompt tokens does not fit the 200000 token context` — and the
|
||||
refusal names the count BEFORE any prefill. It does NOT fire on
|
||||
prompt + max_tokens. Corpus ratio measured: **2.7617 chars/token** (2.5714 for
|
||||
the fox filler), so a 200,000-token window ≈ 514,286–552,286 chars. Calibrate
|
||||
with a probe ABOVE the window; below it the request is accepted and pays real
|
||||
prefill (~22 s).
|
||||
|
||||
### Probe traps that produced false results
|
||||
|
||||
- Generate **≥128 tokens** (512 preferred) for any decode number; at 8–9 tokens
|
||||
`inter_token` is first-token noise — rates like 92.7 vs 85.8 are noise.
|
||||
- Radiance `timings` can carry `null` rates — `(t.get(k) or 0)`, and
|
||||
`j.get("timings") or {}`.
|
||||
- The disk prefix cache (64 GiB) survives restarts; put a random nonce at the
|
||||
HEAD of a "cold" prompt. A warm 197k prefill returns in ~10 s and looks like
|
||||
a 60,000 t/s miracle.
|
||||
- With thinking enabled, a small `max_tokens` yields empty `content` +
|
||||
`finish_reason: stop` on a healthy server; set
|
||||
`chat_template_kwargs.enable_thinking=false` in probes.
|
||||
- `echo ===` in an ssh'd zsh dies on `=`-expansion; it silently truncates the
|
||||
command chain.
|
||||
|
||||
## 5. Ship
|
||||
|
||||
`scripts/<svc>.env` pair + header, serve-script header, bats gates (pair ≤
|
||||
ceiling; pool ≥ contract), docs (runbook, `AGENTS.md` fleet row, LLM index,
|
||||
log.md), `./tests/run.sh`, `deploy.sh <host> --service <svc>`, then the probes.
|
||||
Record every prediction as falsifiable in the env comment and write the verdict
|
||||
there when the probe lands — including when it kills the prediction.
|
||||
## Traps
|
||||
- The cache is ELASTIC: the ledger's `kv cache` line is the ceiling, and `mover: ... slots can be lent out of the activation arena` (987 MiB) means the slab borrows arena first, then cache blocks. A bigger ceiling only helps if the heat engine stops re-borrowing it — check `flex_bytes`/`flex_capacity` movement after load, not just the budget line.
|
||||
- A resize that shrinks the pool mid-session preempts live sequences (`vllm:num_preemptions_total` jumps); a resize that grows it cannot retroactively un-preempt. Deploy at low traffic.
|
||||
- Do not resize the KV pool AND the expert placement in one deploy: the `experts` and `kv cache` lines move together and the attribution dies.
|
||||
- The `claimable` arithmetic (`capacity`) is computed ONCE before allocation; `card total` differs per card (26.0 vs 26.68 GiB on dankserver), so the SLOWER card's number is the budget — read the `[0]` and `[1]` `pools:` lines separately, not just the shared budget table.
|
||||
Reference in new issue
Block a user