docs(frank-archon): record the bf16 KV shape, its measured cost, and the probe
Runbook section `2026-10-11 — unquantized KV at 2 x 200,000`: the dtype law (only bf16|fp8, no K/V split; the QSA block key is dtype-fixed, which is why bf16 costs 1.89x and not 2x), the pool formula that reproduces both live boots within 0.1%, the seqs-3-to-2 provenance (the tcclaviger fork's torch.compile constraint, not radiance's), the boot ledger, and the A/B. The A/B, fp8 control arm through a transient FRANK_ARCHON_ENV_FILE drop-in: prefill -29% (4,379 -> 3,126 cold), TTFT at the full window +25% (the 2 x 197k pair: 110.5 s -> 138.1 s), decode inside the +-3% band on two of three passes, draft acceptance UP 0.661 -> 0.740. The price is prefill, not decode. Open call for Frank: keep exact KV, or take the cheap fp8 2 x 200k fit (pool 3,648). Also corrects two stale index entries (w4dn8 said STAGED, fp8-native said ACTIVE), adds the worklog entry, and commits the verification tool as scripts/probe-frank-archon-kv.py — it carries the probe law that cost two false negatives: radiance answers in reasoning_content, so a chat probe with a small budget reads as empty content + finish_reason stop on a healthy server. Tests 368/368.
This commit is contained in:
Frank committed
1 parent
0bf26b0741
commit
c52ba781e3
7 files changed
+431
-46
No files matched your search
@@ -97,14 +97,161 @@ policy; 173 GiB, purge when the window closes).
|
||||
|
||||
## The shape (`scripts/frank-archon.env`)
|
||||
|
||||
Unchanged from the FP8-native era: TP=2 + `wht6`, `--max-num-seqs 3`,
|
||||
`--max-model-len 200000`, MNBT 2048 (the MTP resonance law — 2560
|
||||
collapses decode to 2.6–7.7 t/s), fp8 KV, `expert_tiered` host pool
|
||||
40960 (the load-bearing floor), `--ngram-placement ram`, froggeric
|
||||
TP=2 + `wht6`, `--max-model-len 200000`, MNBT 2048 (the MTP resonance law —
|
||||
2560 collapses decode to 2.6–7.7 t/s), `--ngram-placement ram`, froggeric
|
||||
v22.5 template override. `MODEL_FILE` points at
|
||||
`/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416
|
||||
MiB measured**); the serve size floor `FRANK_ARCHON_RAD_MIN_MIB` moves
|
||||
177000 → **110000**.
|
||||
`/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB
|
||||
measured**); the serve size floor `FRANK_ARCHON_RAD_MIN_MIB` moves
|
||||
177000 → **110000**. Concurrency and the KV cache moved on 2026-10-11; that
|
||||
change is documented in its own section below.
|
||||
|
||||
## 2026-10-11 — unquantized KV at 2 × 200,000
|
||||
|
||||
The ask: two concurrent 200,000-token conversations on exact (unquantized)
|
||||
attention state. The fp8 pool backed 3,543 MiB → **386,904 tokens = 1.93
|
||||
sessions of the served window**, so two full-height conversations overflow it
|
||||
and the scheduler preempts the overflow to the host/disk prefix tiers
|
||||
(re-prefilling on return). Measured, that overflow is survivable — the fp8
|
||||
shape passed the same 2 × 197k pair probe — so this is a **headroom** change,
|
||||
not the repair of a hard failure. The hard limit the engine states is per
|
||||
request, not per pair: a request longer than the pool's token count is
|
||||
*"accepted by the API and then refused as too large to ever fit — this is a
|
||||
hard limit, not a backing ratio"*, and at 200,000 ≤ 386,904 no single request
|
||||
ever hit it.
|
||||
|
||||
### The dtype law
|
||||
|
||||
`--kv-cache-dtype` takes **only `bf16 | fp8`**: one `kv_dtype` for the whole
|
||||
cache. llama.cpp's exact-K / quantized-V pair (what frank-zealot runs as
|
||||
`--cache-type-k f16 --cache-type-v q8_0`) has **no radiance equivalent** — the
|
||||
rung under bf16 is fp8 on both halves. Evidence: the flag table in the image
|
||||
itself (`--kv-cache-dtype takes bf16 or fp8, not '%s'`) and the absence of any
|
||||
`cache-type-k`/`-v` symbol in the binary.
|
||||
|
||||
bf16 costs **1.89× fp8, not 2×**, and the reason is in the group table:
|
||||
|
||||
| Group | fp8 page | bf16 page | cost per token, per rank |
|
||||
|---|---|---|---|
|
||||
| `kv_attn` — 13 full-attention layers, 4-token blocks | 26.00 KiB | 52.00 KiB | 6.5 → 13.0 KiB |
|
||||
| `kv_qsa_bkey` — 13 layers, the QSA indexer keys | 3.25 KiB | **3.25 KiB** | 0.8125 KiB, dtype-fixed |
|
||||
| `kv_gdn_state` 54 MiB × 2 copies per sequence (+ conv/tail/ple) | per slot | per slot | ≈ 110 MiB **per sequence** |
|
||||
| prefix checkpoints 8 × 55.99 MiB (inside the pool) | 448 MiB | 448 MiB | fixed |
|
||||
|
||||
→ `pool_mib ≥ tokens × 13.8125 KiB + 110 × seqs + 448`. The formula reproduces
|
||||
both live boots: fp8 `(3543 − 330 − 448) → 387,240` vs the ledger's 386,904;
|
||||
bf16 `(6912 − 220 − 448) → 462,946` vs the ledger's **462,752**. It is encoded
|
||||
as a test seam (`the KV pool backs seqs x max-model-len at the declared cache
|
||||
dtype`) so a future resize cannot silently under-provision the pool.
|
||||
|
||||
### Why the weights slab pays — and why that is not pure loss
|
||||
|
||||
The two VRAM budgets are one budget: card 31,856 MiB, minus the 1,710 MiB a
|
||||
boot consumes outside both (driver context, the 990 MiB activation arena,
|
||||
hardware queues, headroom, step staging), minus 1,024 MiB kept free ⇒
|
||||
**`weights + kv ≤ 29,120 MiB`**. Paying for a 6,912 MiB pool therefore cuts the
|
||||
resident slab 26,071 → 22,208 MiB (−14.8 %), and the evicted experts spill
|
||||
into the pinned host pool (40,960 → 44,928 MiB per rank; ledger: pinned pool
|
||||
slots 7,190 → 10,375). The pinned-RAM ledger rises to ≈147 GiB, so the TTM
|
||||
floor moved 140,000 → 150,000 MiB (the kernel cap is 204,800).
|
||||
|
||||
Half of that loss is refunded by an engine behaviour worth knowing: **idle KV
|
||||
VRAM is the expert cache's spillover**. The fp8 pool lent 2.69 GiB to the expert
|
||||
slab; the bf16 pool lends **6.08 GiB** (`mover: 5146 slots can be lent out of
|
||||
the KV cache`). On a llama.cpp endpoint KV is inert storage; on radiance it is
|
||||
a competing cache, and every KV resize is also an expert-residency resize.
|
||||
|
||||
### seqs 3 → 2
|
||||
|
||||
`--max-num-seqs 3` was never a radiance requirement: it is the tcclaviger vLLM
|
||||
fork's torch.compile constraint (`ConstraintViolationError: Constraints
|
||||
violated (L['query_start_loc'].size()[0])`), retired with that runtime — see
|
||||
[[dankserver-qwen38-fp8-expert-offload-tuning]]. Radiance booted seqs 2 in the
|
||||
fp8-native era, and the 2026-10-06 decode sweep measured seqs 2 and 3 tied
|
||||
inside the ±3 % noise floor. Two slots × the window is the capacity contract;
|
||||
it also saves 110 MiB of per-slot recurrent state and drops the fused TP
|
||||
all-reduce to 2 × 3 = 6 rows, i.e. deeper inside the 25-row bound at which the
|
||||
wht6 wire is bit-exact.
|
||||
|
||||
### Boot ledger (deployed 2026-10-11, commit `0bf26b0`)
|
||||
|
||||
```text
|
||||
kv cache 6.75 GiB 349% of it -- backs 2 sequences all at 200000
|
||||
tokens at once, which is the worst case
|
||||
vram_weights 21.69 GiB stated by --vram-weights-mib
|
||||
vram_kv 6.75 GiB stated by --vram-kv-mib
|
||||
unclaimed 2.03 GiB
|
||||
kv_attn full 13 layers block 4 tok page 52.00 KiB 115688 blocks = 5.74 GiB (462752 tokens)
|
||||
kv_qsa_bkey full 13 layers block 4 tok page 3.25 KiB 115688 blocks = 367.17 MiB (462752 tokens)
|
||||
load: rank 0, mover took 21.69 GiB, weights pool 147.00 KiB free of 21.69 GiB
|
||||
mover: 5146 slots can be lent out of the KV cache (6.08 GiB)
|
||||
card 0 settled at 1.85 GiB free
|
||||
serving on http://:::7770
|
||||
```
|
||||
|
||||
### Verification (live service, no scratch container)
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| `/health`, `/v1/models` | 200 `{"status":"ok","model":"frank-archon"}`; served id `frank-archon` |
|
||||
| 2 × ~197k concurrent, a unique marker buried at the head of each | **both answered their own marker exactly** — 196,919 + 196,918 = 393,837 tokens resident, above the fp8 pool's 386,904 and inside bf16's 462,752; walls 138.1 s / 66.2 s, `finish_reason: stop`, prefill 2,747 / 2,991 t/s |
|
||||
| decode at 51.5k depth, 512 tokens, greedy | 94.1 cold / 102.3 warm / 117.6 on a prefix-cache hit t/s; draft acceptance 0.7284 / 0.7525 (fp8 control at cutover: 0.7246) |
|
||||
| fp8 control arm, same probes (`scripts/probe-frank-archon-kv.py`) | table in the next section |
|
||||
|
||||
Probe law re-learned, worth recording: radiance routes the thinking block to
|
||||
`reasoning_content`, so a chat request whose budget is spent inside `<think>`
|
||||
comes back with **empty `content` and `finish_reason: stop`** — a probe that
|
||||
forbids thinking (`chat_template_kwargs.enable_thinking=false`) and reads both
|
||||
fields is the only honest shape. The first pair probe scored 0/2 on exactly
|
||||
that artifact, not on a KV fault.
|
||||
|
||||
### Measured cost — A/B on the live service
|
||||
|
||||
Method (no scratch container, no mask — the unit file is a relink symlink and
|
||||
`mask` refuses it): write a variant env file on the host
|
||||
(`/home/dank/frank-archon-fp8control.env`), point a transient drop-in at it
|
||||
(`Environment=FRANK_ARCHON_ENV_FILE=…`; drop-ins are read AFTER the unit's own
|
||||
`Environment=` lines, so they win), `daemon-reload`, restart, probe, delete the
|
||||
drop-in, restart. Control = the deployed 2026-10-08 shape, verified from
|
||||
`podman inspect` argv (`--kv-cache-dtype fp8 --vram-weights-mib 26071
|
||||
--vram-kv-mib 3543 --max-num-seqs 3`) and the ledger page size (26.00 KiB).
|
||||
Identical probes, same salts, loopback, greedy, 512 generated tokens.
|
||||
|
||||
| Probe | fp8 control (26071/3543, seqs 3) | bf16 deployed (22208/6912, seqs 2) | Δ |
|
||||
|---|---|---|---|
|
||||
| prefill 51.5k, cold | 4,379 t/s | 3,126 t/s | **−29 %** |
|
||||
| prefill 51.5k, warm | 4,864 t/s | 3,333 t/s | **−31 %** |
|
||||
| decode 51.5k, cold | 124.1 t/s | 94.1 t/s | −24 % |
|
||||
| decode 51.5k, warm | 99.2 t/s | 102.3 t/s | +3 % |
|
||||
| decode on a prefix-cache hit | 127.7 t/s | 117.6 t/s | −8 % |
|
||||
| draft acceptance, mean of passes | 0.661 | 0.740 | **+12 %** |
|
||||
| 2 × 196,919 concurrent pair | both markers HIT, 110.5 s | both markers HIT, 138.1 s | +25 % wall |
|
||||
|
||||
Read it as: **the price of exact KV on this engine is prefill, not decode.**
|
||||
Prefill is expert-bandwidth-bound and the resident slab is 14.8 % smaller, so
|
||||
time-to-first-token at the full window rises ~25 % (the pair: 110 s → 138 s);
|
||||
both prefill passes agree, far outside the ±2 % prefill noise floor. Decode is
|
||||
inside the ±3 % band on two of three passes (warm +3 %, hit −8 %); the cold-pass
|
||||
−24 % is the largest single number, the least reproducible, and it moves the
|
||||
same direction as prefill — the cold arm is paying expert streaming, not
|
||||
attention. Spec-decode acceptance IMPROVED ~8 points (0.661 → 0.740): exact
|
||||
attention state makes the target agree with the MTP drafter more often, which
|
||||
pays on every token of every conversation, and it is the strongest argument for
|
||||
the shape.
|
||||
|
||||
Open call (Frank): keep exact KV at the prefill cost, or take the cheap fp8
|
||||
variant — `--vram-kv-mib 3,648` + `--max-num-seqs 2` fits 2 × 200,000 at fp8
|
||||
for a 105 MiB pool bump and ≈ no decode cost. The tool is committed:
|
||||
`scripts/probe-frank-archon-kv.py` (run it on Dankserver; it rebuilds its
|
||||
prompts from the repo checkout there).
|
||||
|
||||
### Rollback
|
||||
|
||||
Revert `scripts/frank-archon.env` to the fp8 pair (26071/3543, fp8, seqs 3,
|
||||
host pool 40960, prefix host 4096) and redeploy: no payload change, boot ~2 min.
|
||||
The KLD gate scored at the fp8-KV shape, so a bf16-KV production runs *away*
|
||||
from the scored configuration — bf16 can only reduce the attention-state
|
||||
error, and a re-score is a drained-window job if the gate agreement is ever
|
||||
needed in the record.
|
||||
|
||||
## Gates (measured at cutover)
|
||||
|
||||
|
||||
@@ -152,11 +152,17 @@ frank-zealot, port 7771]
|
||||
target; payload on disk; owns the measured boot-pair table)**: the same
|
||||
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
|
||||
104.8 tok/s, 3,571 MiB true free
|
||||
- [[frank-archon-radiance-swift15-w4dn8]] — **STAGED (campaign running)**:
|
||||
- [[frank-archon-radiance-swift15-w4dn8]] — **ACTIVE since 2026-10-08**:
|
||||
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance
|
||||
1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container);
|
||||
replaces the FP8-native payload when the gates pass
|
||||
- [[frank-archon-radiance-qwen38-native-fp8]] — **ACTIVE since 2026-10-05**
|
||||
the FP8-native payload remains the rollback. **2026-10-11: the KV cache is
|
||||
UNQUANTIZED (bf16) at 2 x 200,000** — pool 3543 → 6912 MiB (386,904 →
|
||||
462,752 tokens), weights slab 26071 → 22208, host pool 44928,
|
||||
`--max-num-seqs 2`. bf16 costs 1.89x fp8 per token (the QSA block key is
|
||||
dtype-fixed, so it is not a clean 2x); measured price is prefill (-29%),
|
||||
NOT decode, and spec-decode acceptance rose 0.661 → 0.740.
|
||||
- [[frank-archon-radiance-qwen38-native-fp8]] — **RETIRED 2026-10-08 (the
|
||||
w4dn8 swap replaced it; its `.rad` stays on disk as the rollback)**
|
||||
(root systemd unit, port 7770): the StillDeadcode **radiance** engine
|
||||
(C++/HIP, libr4d gfx1201 kernels, digest-pinned 287 MiB image, engine as
|
||||
the image entrypoint) serving the OFFICIAL `Qwen/Qwen3.8-Flash-Next-FP8`
|
||||
|
||||
Reference in new issue
Block a user