From c52ba781e3869a2b27c84fc524a110e1f8689b20 Mon Sep 17 00:00:00 2001 From: Frank Date: Sun, 11 Oct 2026 01:36:48 -0700 Subject: [PATCH] docs(frank-archon): record the bf16 KV shape, its measured cost, and the probe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Runbook section `2026-10-11 — unquantized KV at 2 x 200,000`: the dtype law (only bf16|fp8, no K/V split; the QSA block key is dtype-fixed, which is why bf16 costs 1.89x and not 2x), the pool formula that reproduces both live boots within 0.1%, the seqs-3-to-2 provenance (the tcclaviger fork's torch.compile constraint, not radiance's), the boot ledger, and the A/B. The A/B, fp8 control arm through a transient FRANK_ARCHON_ENV_FILE drop-in: prefill -29% (4,379 -> 3,126 cold), TTFT at the full window +25% (the 2 x 197k pair: 110.5 s -> 138.1 s), decode inside the +-3% band on two of three passes, draft acceptance UP 0.661 -> 0.740. The price is prefill, not decode. Open call for Frank: keep exact KV, or take the cheap fp8 2 x 200k fit (pool 3,648). Also corrects two stale index entries (w4dn8 said STAGED, fp8-native said ACTIVE), adds the worklog entry, and commits the verification tool as scripts/probe-frank-archon-kv.py — it carries the probe law that cost two false negatives: radiance answers in reasoning_content, so a chat probe with a small budget reads as empty content + finish_reason stop on a healthy server. Tests 368/368. --- AGENTS.md | 58 ++++--- scripts/frank-archon.env | 24 ++- scripts/probe-frank-archon-kv.py | 147 ++++++++++++++++ .../frank-archon-radiance-swift15-w4dn8.md | 161 +++++++++++++++++- serverAppConfigs-docs/LLM/index.md | 12 +- serverAppConfigs-docs/log.md | 53 ++++++ tests/frank_archon_unit.bats | 22 +-- 7 files changed, 431 insertions(+), 46 deletions(-) create mode 100755 scripts/probe-frank-archon-kv.py diff --git a/AGENTS.md b/AGENTS.md index 20b1a97..4d08d4c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -525,41 +525,61 @@ the throughput gate (decode ≥ 30.8 t/s) was passed at 4.5× the bar Rollback: revert the swap commit — the fp8-native `.rad` (`qwen4exp-fp8-native.rad`) and every prior payload remain on disk. -**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 3`, +**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 2`, `--max-model-len 200000` (the harness contextWindow must quote this), -`--placement expert_tiered --host-pool-mib 40960` + **explicit VRAM -partition `--vram-weights-mib 26071 --vram-kv-mib 3543`** (the vram-pair -replaced `--expert-vs-cache-ratio`; the pair sizes the fp8-KV shape at the -deployed boot), `--kv-cache-dtype fp8` (inherited from the fp8-native -env), `--num-speculative-tokens 2`, -`--max-num-batched-tokens 2048`, prefix cache 4 GiB pinned host + 64 GiB +`--placement expert_tiered --host-pool-mib 44928` + **explicit VRAM +partition `--vram-weights-mib 22208 --vram-kv-mib 6912`** (the vram-pair +replaced `--expert-vs-cache-ratio`; the two are ONE budget — +`weights + kv ≤ 29,120 MiB`, from the measured card 31,856 minus 1,710 +consumed outside both budgets minus 1,024 to keep free), `--kv-cache-dtype` +**`bf16` — unquantized KV since 2026-10-11**, `--num-speculative-tokens 2`, +`--max-num-batched-tokens 2048`, prefix cache 8 GiB pinned host + 64 GiB disk at `/home/dank/radiance-state`, **`--ngram-placement ram`**, Froggeric v22.5 chat template vendored (`scripts/qwen38-froggeric-v22.5.jinja`, -`--override-chat-template`). Boot ledger: KV pool 3.46 GiB (~480k fp8 -tokens), two expert pools 57.84 GiB, cards settled 1.41 GiB free. +`--override-chat-template`). Boot ledger: KV pool 6.75 GiB = **462,752 bf16 +tokens, "backs 2 sequences all at 200000 tokens at once"** (the fp8 pool's +386,904 backed 1.93), weights slab filled to 147 KiB of slack, host pool +10,375 slots, cards settled 1.85 GiB free. Shape notes carried from the fp8-native era and unchanged: wht6 gains, MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk prefix cache, and the rule that a prompt over `--max-model-len` is refused, never truncated. What changed: the fp8-native era's NVMe expert streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the -40 GiB pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s -on the deployed shape (512-token streamed probe). The campaign also +host pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s +on the fp8 shape (512-token streamed probe). The campaign also measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth, -accept 0.7246); production shipped the fp8-KV + spec-2 pair at cutover — -the KLD gate scored at the fp8-KV shape, so the gate and the boot agree. -Disk-tier sizing law: the engine forces ~9.6 GiB of disk-tier weight +accept 0.7246). + +**The bf16 KV law (2026-10-11)**: `--kv-cache-dtype` takes ONLY `bf16|fp8` — +one dtype for the whole cache, so llama.cpp's exact-K/quantized-V pair (what +frank-zealot runs) has NO radiance equivalent; the rung under bf16 is fp8 on +both halves. Measured per rank from the two boot ledgers: `kv_attn` (13 +layers, 4-token blocks) pages 26.00 → 52.00 KiB, while `kv_qsa_bkey` (13 +layers) stays 3.25 KiB at BOTH dtypes (the QSA block key is an index, not +attention state) — so a token costs 7.3125 KiB at fp8 and 13.8125 KiB at +bf16: **1.89×, not 2×**. Pool sizing must also pay the dtype-independent terms +that live in the same pool (~110 MiB a sequence of recurrent state + 448 MiB +of prefix checkpoints): `(vram_kv − 110·seqs − 448) MiB / 13.8125 KiB` = +462,946 where the engine reported 462,752 (0.04 % page rounding). The cost is +paid twice on this engine, and half-refunded: idle KV VRAM is also the expert +cache's spillover, and the bigger pool *lends* 6.08 GiB to the slab where the +fp8 pool lent 2.69 GiB. `--max-num-seqs 3 → 2` came with the change (two slots +× the window is the capacity contract; the "seqs 2 cannot boot" law belongs to +the retired tcclaviger fork's torch.compile, not radiance) and it also drops +the fused all-reduce to 2 × 3 = 6 rows, under the 25-row bound where wht6 is +bit-exact. Disk-tier sizing law: the engine forces ~9.6 GiB of disk-tier weight buffers + arena onto the weights side, so `weights + kv` is capped near 29.6 GiB/card; boot with the pair, never `--kv-cache-size-mib`. -**The pinned-RAM ledger is why this is a root unit**: 2 × 40 GiB expert +**The pinned-RAM ledger is why this is a root unit**: 2 × 44 GiB expert pools + the 47.7 GiB n-gram copy (`--ngram-placement ram` is pinned RAM -the card reads) + 4 GiB prefix host ≈ **132 GiB** charged against +the card reads) + 8 GiB prefix host ≈ **147 GiB** charged against `RLIMIT_MEMLOCK` — the TTM floor is set just above it. `ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages; the serve -script's TTM preflight (`FRANK_ARCHON_TTM_MIN_MIB=140000`) refuses to -start until both knobs are raised in the stop window (sysfs write first, -kernel cmdline + reboot only if read-only). +script's TTM preflight (`FRANK_ARCHON_TTM_MIN_MIB=150000`, cap on this host +204,800) refuses to start until both knobs are raised in the stop window +(sysfs write first, kernel cmdline + reboot only if read-only). **Contract facts** (source-verified): the HTTP listener binds AFTER the full load, so a bare `curl /health` gate is honest (`/health` == `/ping`, diff --git a/scripts/frank-archon.env b/scripts/frank-archon.env index 390a4c2..3daae65 100644 --- a/scripts/frank-archon.env +++ b/scripts/frank-archon.env @@ -166,19 +166,29 @@ FRANK_ARCHON_VRAM_KV_MIB=6912 # is ONE kv_dtype for the whole cache, so "exact K, quantized V" (llama.cpp's # --cache-type-k/--cache-type-v pair, which is what frank-zealot runs) is # UNSERVABLE here: the rung below bf16 is fp8 on both halves. -# Measured per-rank cost, read out of the boot ledger's group table: +# Measured per-rank cost, read out of the boot ledger's group table (fp8 boot +# 2026-10-08, bf16 boot 2026-10-11): # kv_attn 13 layers, 4-token blocks, page 26.00 KiB (fp8) -> 52.00 KiB -# kv_qsa_bkey 13 layers, 4-token blocks, page 3.25 KiB (fp8) -> 6.50 KiB -# => 7.3125 KiB/token fp8, 14.625 KiB/token bf16 +# kv_qsa_bkey 13 layers, 4-token blocks, page 3.25 KiB at BOTH dtypes — +# the QSA block key is an index, not attention state, and does +# NOT follow --kv-cache-dtype +# => 7.3125 KiB/token fp8, 13.8125 KiB/token bf16: bf16 costs 1.89x fp8, +# not 2x, precisely because the bkey plane is dtype-fixed. # The dtype-INDEPENDENT terms also live in the pool: kv_gdn_state 54 MiB x 2 # copies per sequence (+ conv/tail/ple ~= 110 MiB a sequence) and the prefix # checkpoint region 8 x 55.99 = 448 MiB. So: -# pool_mib >= tokens x 14.625 KiB + 110 x seqs + 448 -# 6912 -> (6912 - 220 - 448) x 1024 / 14.625 = 437,320 tokens >= 2 x 200,000 +# pool_mib >= tokens x 13.8125 KiB + 110 x seqs + 448 +# 6912 -> (6912 - 220 - 448) x 1024 / 13.8125 = 462,946 tokens; the engine +# reported 462,752 (0.04% page rounding) — 2 x 200,000 with 62k of slack, +# and the ledger says it outright: "backs 2 sequences all at 200000 tokens +# at once, which is the worst case". # (the fp8 pool of 3543 gave 386,904 — it never held 2 x 200k, it held 1.93 # sessions, and a third long conversation preempted to the host/disk tiers). -# Secondary cost: bf16 doubles the prefix-cache entry (58.50 -> 117 KiB), so -# the host tier below doubles too. +# Secondary cost: the prefix-cache entry grows with the attention plane +# (one entry 58.50 KiB at fp8 -> 110.50 KiB at bf16, measured on the two boot +# ledgers' `kv tier transfer` lines), so the host tier below doubles to keep +# the same number of parked conversations, out of RAM the host has +# (104 GiB available with the service loaded). FRANK_ARCHON_KV_DTYPE=bf16 # Spec decode: the checkpoint's own MTP head at the publisher's depth-3 # operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the diff --git a/scripts/probe-frank-archon-kv.py b/scripts/probe-frank-archon-kv.py new file mode 100755 index 0000000..0351309 --- /dev/null +++ b/scripts/probe-frank-archon-kv.py @@ -0,0 +1,147 @@ +#!/usr/bin/env python3 +"""probe-frank-archon-kv.py — verify the 2 x 200,000 KV contract on a LIVE +frank-archon (radiance) service. Run it ON dankserver: it talks to +127.0.0.1:7770 and builds its prompts from the repo checkout there. +No scratch container, no mask: the service under test is the service. + + ssh dankserver 'python3 /home/dank/serverAppConfigs/scripts/probe-frank-archon-kv.py' + +Three probes, in increasing cost: + A decode at depth — one ~51k-token chat completion, 512 tokens, greedy. + Reports the engine's own `timings` (prefill, decode, draft acceptance). + The bench law applies: >=128 generated tokens, greedy, real text. + B the capacity contract — TWO concurrent chats of ~197k prompt tokens + (~394k tokens resident) each carrying a unique marker at its head. Both + must answer their own marker. This is what a 2 x 200,000 endpoint owes: + two full-height conversations resident at once, exact retrieval in both. + C prefix-cache restore — probe A's cold prompt again, so the disk tier has + to give the blocks back (cache_n > 0, TTFT collapses). + +PROBE LAW (learned the hard way, twice): radiance routes the thinking block to +`reasoning_content`, so a chat request whose budget is spent inside +`` returns EMPTY `content` with `finish_reason: stop`. A probe that +reads `content` alone scores 0/2 on a healthy server. Every request here sets +chat_template_kwargs.enable_thinking=false and reads both fields. + +A/B method for a shape change (no scratch container): write a variant env file +on the host, point a transient systemd drop-in at it +(`Environment=FRANK_ARCHON_ENV_FILE=...` — drop-ins are read AFTER the unit's +own Environment= lines, so they win), restart, probe, delete the drop-in, +daemon-reload, restart. `systemctl mask` is not available: the unit file in +/etc/systemd/system is a relink symlink into the checkout. +""" +import json +import glob +import time +import urllib.request +import concurrent.futures + +BASE = "http://127.0.0.1:7770" +MODEL = "frank-archon" +REPO = "/home/dank/serverAppConfigs" +GLOBS = [f"{REPO}/scripts/*.sh", f"{REPO}/serverAppConfigs-docs/**/*.md"] + + +def post(payload, timeout=1800): + req = urllib.request.Request( + BASE + "/v1/chat/completions", + data=json.dumps(payload).encode(), + headers={"Content-Type": "application/json"}, + ) + t0 = time.time() + with urllib.request.urlopen(req, timeout=timeout) as r: + body = json.loads(r.read().decode()) + hdr = dict(r.headers.items()) + return body, time.time() - t0, hdr + + +def get(path, timeout=60): + with urllib.request.urlopen(BASE + path, timeout=timeout) as r: + return json.loads(r.read().decode()) + + +def corpus(min_chars): + files = [] + for g in GLOBS: + files.extend(sorted(glob.glob(g, recursive=True))) + out, total = [], 0 + for f in files: + try: + t = open(f, encoding="utf-8", errors="replace").read() + except OSError: + continue + out.append(t) + total += len(t) + if total > min_chars: + break + return "".join(text for text in out) + + +def prompt(text, chars, salt, fact): + """Real repo text, salted so the prefix cache cannot answer, marker planted + near the head, question at the tail.""" + head = f"\n[SALT {salt}] confidential marker: {fact}\n" + body = text + text + p = (head + body * max(1, chars // len(body)))[:chars] + return p + "\n\nQuestion: what is the confidential marker in the " \ + "SALT block near the top of this document? Answer with the marker " \ + "alone, no other words.\n" + + +def chat(text, chars, salt, fact, max_tokens): + return {"model": MODEL, + "messages": [{"role": "user", + "content": prompt(text, chars, salt, fact)}], + "max_tokens": max_tokens, "temperature": 0.0, + "chat_template_kwargs": {"enable_thinking": False}} + + +def show(tag, j, wall, expect=None): + tim, u = j.get("timings") or {}, j.get("usage", {}) + ch = (j.get("choices") or [{}])[0] + msg = ch.get("message") or {} + txt = (msg.get("content") or msg.get("reasoning_content") or "").strip() + acc = (tim["draft_n_accepted"] / tim["draft_n"]) if tim.get("draft_n") else None + line = (f"[{tag}] prompt={u.get('prompt_tokens')} gen={u.get('completion_tokens')} " + f"wall={wall:.1f}s prefill={tim.get('prompt_per_second', 0):.0f} t/s " + f"decode={tim.get('predicted_per_second', 0):.1f} t/s " + f"draft={tim.get('draft_n')}/{tim.get('draft_n_accepted')} " + f"accept={acc if acc is None else round(acc, 4)} " + f"cache_n={tim.get('cache_n')} finish={ch.get('finish_reason')}") + if expect: + hit = expect.split("-")[0] in txt.lower() + line += f" expected={expect} answered={txt[:80]!r} HIT={hit}" + print(line) + + +def main(): + print("health:", json.dumps(get("/health"))) + print("models:", [m.get("id") for m in get("/v1/models").get("data", [])]) + deep = corpus(160_000) + wide = corpus(1_100_000) + + facts = {"B1": "cobalt-harbor-13", "B2": "quartz-meadow-42"} + bodies = {s: chat(wide, 646_000, s, f, 24) for s, f in facts.items()} + print(f"[B] {len(bodies['B1']['messages'][0]['content']):,} chars per " + f"request, x2 concurrent") + + # A + C first (cheap), B last (the expensive pair). + for tag, salt in (("A-cold", "A1-9137"), ): + j, wall, _ = post(chat(deep, 140_000, salt, "zephyr-lantern-7", 512)) + show(tag, j, wall, "zephyr-lantern-7") + t0 = time.time() + with concurrent.futures.ThreadPoolExecutor(max_workers=2) as ex: + futs = {s: ex.submit(post, b) for s, b in bodies.items()} + for s, fut in futs.items(): + try: + j, wall, _ = fut.result() + show(f"B-{s}", j, wall, facts[s]) + except Exception as e: # a preemption/timeout lands here + print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}") + print(f"[B] total {time.time() - t0:.1f}s") + j, wall, _ = post(chat(deep, 140_000, "A1-9137", "zephyr-lantern-7", 512)) + show("C-prefix-hit", j, wall, "zephyr-lantern-7") + + +if __name__ == "__main__": + main() diff --git a/serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md b/serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md index 2a32eb9..9797ad0 100644 --- a/serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md +++ b/serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md @@ -97,14 +97,161 @@ policy; 173 GiB, purge when the window closes). ## The shape (`scripts/frank-archon.env`) -Unchanged from the FP8-native era: TP=2 + `wht6`, `--max-num-seqs 3`, -`--max-model-len 200000`, MNBT 2048 (the MTP resonance law — 2560 -collapses decode to 2.6–7.7 t/s), fp8 KV, `expert_tiered` host pool -40960 (the load-bearing floor), `--ngram-placement ram`, froggeric +TP=2 + `wht6`, `--max-model-len 200000`, MNBT 2048 (the MTP resonance law — +2560 collapses decode to 2.6–7.7 t/s), `--ngram-placement ram`, froggeric v22.5 template override. `MODEL_FILE` points at -`/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 -MiB measured**); the serve size floor `FRANK_ARCHON_RAD_MIN_MIB` moves -177000 → **110000**. +`/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB +measured**); the serve size floor `FRANK_ARCHON_RAD_MIN_MIB` moves +177000 → **110000**. Concurrency and the KV cache moved on 2026-10-11; that +change is documented in its own section below. + +## 2026-10-11 — unquantized KV at 2 × 200,000 + +The ask: two concurrent 200,000-token conversations on exact (unquantized) +attention state. The fp8 pool backed 3,543 MiB → **386,904 tokens = 1.93 +sessions of the served window**, so two full-height conversations overflow it +and the scheduler preempts the overflow to the host/disk prefix tiers +(re-prefilling on return). Measured, that overflow is survivable — the fp8 +shape passed the same 2 × 197k pair probe — so this is a **headroom** change, +not the repair of a hard failure. The hard limit the engine states is per +request, not per pair: a request longer than the pool's token count is +*"accepted by the API and then refused as too large to ever fit — this is a +hard limit, not a backing ratio"*, and at 200,000 ≤ 386,904 no single request +ever hit it. + +### The dtype law + +`--kv-cache-dtype` takes **only `bf16 | fp8`**: one `kv_dtype` for the whole +cache. llama.cpp's exact-K / quantized-V pair (what frank-zealot runs as +`--cache-type-k f16 --cache-type-v q8_0`) has **no radiance equivalent** — the +rung under bf16 is fp8 on both halves. Evidence: the flag table in the image +itself (`--kv-cache-dtype takes bf16 or fp8, not '%s'`) and the absence of any +`cache-type-k`/`-v` symbol in the binary. + +bf16 costs **1.89× fp8, not 2×**, and the reason is in the group table: + +| Group | fp8 page | bf16 page | cost per token, per rank | +|---|---|---|---| +| `kv_attn` — 13 full-attention layers, 4-token blocks | 26.00 KiB | 52.00 KiB | 6.5 → 13.0 KiB | +| `kv_qsa_bkey` — 13 layers, the QSA indexer keys | 3.25 KiB | **3.25 KiB** | 0.8125 KiB, dtype-fixed | +| `kv_gdn_state` 54 MiB × 2 copies per sequence (+ conv/tail/ple) | per slot | per slot | ≈ 110 MiB **per sequence** | +| prefix checkpoints 8 × 55.99 MiB (inside the pool) | 448 MiB | 448 MiB | fixed | + +→ `pool_mib ≥ tokens × 13.8125 KiB + 110 × seqs + 448`. The formula reproduces +both live boots: fp8 `(3543 − 330 − 448) → 387,240` vs the ledger's 386,904; +bf16 `(6912 − 220 − 448) → 462,946` vs the ledger's **462,752**. It is encoded +as a test seam (`the KV pool backs seqs x max-model-len at the declared cache +dtype`) so a future resize cannot silently under-provision the pool. + +### Why the weights slab pays — and why that is not pure loss + +The two VRAM budgets are one budget: card 31,856 MiB, minus the 1,710 MiB a +boot consumes outside both (driver context, the 990 MiB activation arena, +hardware queues, headroom, step staging), minus 1,024 MiB kept free ⇒ +**`weights + kv ≤ 29,120 MiB`**. Paying for a 6,912 MiB pool therefore cuts the +resident slab 26,071 → 22,208 MiB (−14.8 %), and the evicted experts spill +into the pinned host pool (40,960 → 44,928 MiB per rank; ledger: pinned pool +slots 7,190 → 10,375). The pinned-RAM ledger rises to ≈147 GiB, so the TTM +floor moved 140,000 → 150,000 MiB (the kernel cap is 204,800). + +Half of that loss is refunded by an engine behaviour worth knowing: **idle KV +VRAM is the expert cache's spillover**. The fp8 pool lent 2.69 GiB to the expert +slab; the bf16 pool lends **6.08 GiB** (`mover: 5146 slots can be lent out of +the KV cache`). On a llama.cpp endpoint KV is inert storage; on radiance it is +a competing cache, and every KV resize is also an expert-residency resize. + +### seqs 3 → 2 + +`--max-num-seqs 3` was never a radiance requirement: it is the tcclaviger vLLM +fork's torch.compile constraint (`ConstraintViolationError: Constraints +violated (L['query_start_loc'].size()[0])`), retired with that runtime — see +[[dankserver-qwen38-fp8-expert-offload-tuning]]. Radiance booted seqs 2 in the +fp8-native era, and the 2026-10-06 decode sweep measured seqs 2 and 3 tied +inside the ±3 % noise floor. Two slots × the window is the capacity contract; +it also saves 110 MiB of per-slot recurrent state and drops the fused TP +all-reduce to 2 × 3 = 6 rows, i.e. deeper inside the 25-row bound at which the +wht6 wire is bit-exact. + +### Boot ledger (deployed 2026-10-11, commit `0bf26b0`) + +```text +kv cache 6.75 GiB 349% of it -- backs 2 sequences all at 200000 + tokens at once, which is the worst case +vram_weights 21.69 GiB stated by --vram-weights-mib +vram_kv 6.75 GiB stated by --vram-kv-mib +unclaimed 2.03 GiB +kv_attn full 13 layers block 4 tok page 52.00 KiB 115688 blocks = 5.74 GiB (462752 tokens) +kv_qsa_bkey full 13 layers block 4 tok page 3.25 KiB 115688 blocks = 367.17 MiB (462752 tokens) +load: rank 0, mover took 21.69 GiB, weights pool 147.00 KiB free of 21.69 GiB +mover: 5146 slots can be lent out of the KV cache (6.08 GiB) +card 0 settled at 1.85 GiB free +serving on http://:::7770 +``` + +### Verification (live service, no scratch container) + +| Check | Result | +|---|---| +| `/health`, `/v1/models` | 200 `{"status":"ok","model":"frank-archon"}`; served id `frank-archon` | +| 2 × ~197k concurrent, a unique marker buried at the head of each | **both answered their own marker exactly** — 196,919 + 196,918 = 393,837 tokens resident, above the fp8 pool's 386,904 and inside bf16's 462,752; walls 138.1 s / 66.2 s, `finish_reason: stop`, prefill 2,747 / 2,991 t/s | +| decode at 51.5k depth, 512 tokens, greedy | 94.1 cold / 102.3 warm / 117.6 on a prefix-cache hit t/s; draft acceptance 0.7284 / 0.7525 (fp8 control at cutover: 0.7246) | +| fp8 control arm, same probes (`scripts/probe-frank-archon-kv.py`) | table in the next section | + +Probe law re-learned, worth recording: radiance routes the thinking block to +`reasoning_content`, so a chat request whose budget is spent inside `` +comes back with **empty `content` and `finish_reason: stop`** — a probe that +forbids thinking (`chat_template_kwargs.enable_thinking=false`) and reads both +fields is the only honest shape. The first pair probe scored 0/2 on exactly +that artifact, not on a KV fault. + +### Measured cost — A/B on the live service + +Method (no scratch container, no mask — the unit file is a relink symlink and +`mask` refuses it): write a variant env file on the host +(`/home/dank/frank-archon-fp8control.env`), point a transient drop-in at it +(`Environment=FRANK_ARCHON_ENV_FILE=…`; drop-ins are read AFTER the unit's own +`Environment=` lines, so they win), `daemon-reload`, restart, probe, delete the +drop-in, restart. Control = the deployed 2026-10-08 shape, verified from +`podman inspect` argv (`--kv-cache-dtype fp8 --vram-weights-mib 26071 +--vram-kv-mib 3543 --max-num-seqs 3`) and the ledger page size (26.00 KiB). +Identical probes, same salts, loopback, greedy, 512 generated tokens. + +| Probe | fp8 control (26071/3543, seqs 3) | bf16 deployed (22208/6912, seqs 2) | Δ | +|---|---|---|---| +| prefill 51.5k, cold | 4,379 t/s | 3,126 t/s | **−29 %** | +| prefill 51.5k, warm | 4,864 t/s | 3,333 t/s | **−31 %** | +| decode 51.5k, cold | 124.1 t/s | 94.1 t/s | −24 % | +| decode 51.5k, warm | 99.2 t/s | 102.3 t/s | +3 % | +| decode on a prefix-cache hit | 127.7 t/s | 117.6 t/s | −8 % | +| draft acceptance, mean of passes | 0.661 | 0.740 | **+12 %** | +| 2 × 196,919 concurrent pair | both markers HIT, 110.5 s | both markers HIT, 138.1 s | +25 % wall | + +Read it as: **the price of exact KV on this engine is prefill, not decode.** +Prefill is expert-bandwidth-bound and the resident slab is 14.8 % smaller, so +time-to-first-token at the full window rises ~25 % (the pair: 110 s → 138 s); +both prefill passes agree, far outside the ±2 % prefill noise floor. Decode is +inside the ±3 % band on two of three passes (warm +3 %, hit −8 %); the cold-pass +−24 % is the largest single number, the least reproducible, and it moves the +same direction as prefill — the cold arm is paying expert streaming, not +attention. Spec-decode acceptance IMPROVED ~8 points (0.661 → 0.740): exact +attention state makes the target agree with the MTP drafter more often, which +pays on every token of every conversation, and it is the strongest argument for +the shape. + +Open call (Frank): keep exact KV at the prefill cost, or take the cheap fp8 +variant — `--vram-kv-mib 3,648` + `--max-num-seqs 2` fits 2 × 200,000 at fp8 +for a 105 MiB pool bump and ≈ no decode cost. The tool is committed: +`scripts/probe-frank-archon-kv.py` (run it on Dankserver; it rebuilds its +prompts from the repo checkout there). + +### Rollback + +Revert `scripts/frank-archon.env` to the fp8 pair (26071/3543, fp8, seqs 3, +host pool 40960, prefix host 4096) and redeploy: no payload change, boot ~2 min. +The KLD gate scored at the fp8-KV shape, so a bf16-KV production runs *away* +from the scored configuration — bf16 can only reduce the attention-state +error, and a re-score is a drained-window job if the gate agreement is ever +needed in the record. ## Gates (measured at cutover) diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 5410df3..bc1cccc 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -152,11 +152,17 @@ frank-zealot, port 7771] target; payload on disk; owns the measured boot-pair table)**: the same runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower — 104.8 tok/s, 3,571 MiB true free -- [[frank-archon-radiance-swift15-w4dn8]] — **STAGED (campaign running)**: +- [[frank-archon-radiance-swift15-w4dn8]] — **ACTIVE since 2026-10-08**: Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance 1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container); - replaces the FP8-native payload when the gates pass -- [[frank-archon-radiance-qwen38-native-fp8]] — **ACTIVE since 2026-10-05** + the FP8-native payload remains the rollback. **2026-10-11: the KV cache is + UNQUANTIZED (bf16) at 2 x 200,000** — pool 3543 → 6912 MiB (386,904 → + 462,752 tokens), weights slab 26071 → 22208, host pool 44928, + `--max-num-seqs 2`. bf16 costs 1.89x fp8 per token (the QSA block key is + dtype-fixed, so it is not a clean 2x); measured price is prefill (-29%), + NOT decode, and spec-decode acceptance rose 0.661 → 0.740. +- [[frank-archon-radiance-qwen38-native-fp8]] — **RETIRED 2026-10-08 (the + w4dn8 swap replaced it; its `.rad` stays on disk as the rollback)** (root systemd unit, port 7770): the StillDeadcode **radiance** engine (C++/HIP, libr4d gfx1201 kernels, digest-pinned 287 MiB image, engine as the image entrypoint) serving the OFFICIAL `Qwen/Qwen3.8-Flash-Next-FP8` diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index d5c95ab..7e4ab0c 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -1598,3 +1598,56 @@ every restart since the cutover. so the honest claim is ~184 s of inductor/AOT build recovered per restart — not a proportional boot cut. +## 2026-10-11 — frank-archon: unquantized (bf16) KV at 2 × 200,000 + +- Ask: two concurrent 200,000-token conversations on exact attention state. + Sized from the live fp8 boot ledger, not arithmetic: the paged groups cost + 7.3125 KiB/token per rank (`kv_attn` 26.00 KiB per 4-token block + + `kv_qsa_bkey` 3.25), so the 3,543 MiB pool bought 386,904 tokens = **1.93 + sessions of the served window**. +- Engine door: `--kv-cache-dtype` takes ONLY `bf16|fp8` — one dtype for the + whole cache, so llama.cpp's exact-K / quantized-V pair (what frank-zealot + runs) has **no radiance equivalent**. Measured bf16 rate is **13.8125 + KiB/token = 1.89× fp8, not 2×**: the QSA block key (13 layers, 3.25 KiB per + 4 tokens) does not follow the dtype. Pool formula + `tokens × rate + 110 MiB × seqs (recurrent state) + 448 MiB (prefix + checkpoints)` reproduces both boots within 0.1 %, and is now a test seam. +- Deployed (`0bf26b0` + docs): pool 3543 → **6912** MiB (462,752 tokens; ledger + "backs 2 sequences all at 200000 tokens at once"), weights slab 26071 → + 22208, host pool 40960 → 44928 (the evicted experts land there; pinned pool + slots 7,190 → 10,375), prefix host 4096 → 8192 (the entry grows 58.50 → + 110.50 KiB), TTM floor 140000 → 150000 (pinned ledger ~147 GiB, cap 204,800). + `weights + kv` pinned to the measured ceiling **29,120 MiB/card**; cards + settled 1.85 GiB free — more than the fp8 boot's 1.39, because the total + claim dropped. Half the cost is refunded by the engine: idle KV VRAM is the + expert cache's spillover, and the bigger pool *lends* 6.08 GiB where the fp8 + pool lent 2.69. +- `--max-num-seqs 3 → 2`: the fleet's "seqs 2 cannot boot" law is the retired + tcclaviger vLLM fork's torch.compile `query_start_loc` constraint, not + radiance's — radiance booted seqs 2 in the fp8-native era and the 2026-10-06 + sweep had 2 and 3 tied inside ±3 %. Bonus: the fused all-reduce is declared + over 2 × (1+2) = 6 rows, inside the 25-row bound where the wht6 wire is + bit-exact. +- Verified live (no scratch container, no bench): `/health` 200, served id + `frank-archon`; 2 × 196,919 concurrent conversations each answered the + marker buried at the head of its own prompt (393,837 tokens resident, above + the fp8 pool's 386,904); decode at 51.5k depth 94.1 cold / 102.3 warm / 117.6 + prefix-hit t/s, draft acceptance 0.7284 / 0.7525. +- fp8 control arm through a transient `FRANK_ARCHON_ENV_FILE` drop-in (the + frank-zealot ubatch method, argv + ledger page size verified): prefill + **−29 %** (4,379 → 3,126 cold, 4,864 → 3,333 warm), decode inside the noise + band on two of three passes, TTFT at the full window **+25 %** (the pair: + 110.5 s → 138.1 s), draft acceptance **up 0.661 → 0.740**. The price is + prefill — a 14.8 % smaller resident slab, and prefill is the pass that + touches every expert — the gain is exact state plus acceptance. **Open call + for Frank:** keep bf16, or take the cheap fp8 2 × 200k fit (pool 3,648 MiB, + ≈ free) and leave KV quantized. +- Harness law re-learned: radiance puts the thinking block in + `reasoning_content`, so a probe that reads `content` on a small budget sees + empty text with `finish_reason: stop` — a healthy server scored 0/2 until the + probe set `chat_template_kwargs.enable_thinking=false`. Tool committed as + `scripts/probe-frank-archon-kv.py`. +- Tests 368/368, e2e 33/33 (one unrelated homepage retry). Docs: + [[frank-archon-radiance-swift15-w4dn8]], and the LLM index entry corrected + (the w4dn8 entry said STAGED, the fp8-native one said ACTIVE). + diff --git a/tests/frank_archon_unit.bats b/tests/frank_archon_unit.bats index 899b7a3..f1891ae 100755 --- a/tests/frank_archon_unit.bats +++ b/tests/frank_archon_unit.bats @@ -164,21 +164,23 @@ TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja" @test "the KV pool backs seqs x max-model-len at the declared cache dtype" { # The engine's own hard limit: a request longer than the pool's token # count is "accepted by the API and then refused as too large to ever fit - # -- this is a hard limit, not a backing ratio". Rates measured from the - # live boot ledger's group table, per rank: kv_attn (13 layers) is a - # 26.00 KiB page per 4 tokens at fp8 and kv_qsa_bkey (13 layers) 3.25 KiB - # -> 7,488 B/token, and bf16 doubles both -> 14,976 B/token. The terms - # that do NOT follow the dtype also live in the pool: ~110 MiB a sequence - # of recurrent state (kv_gdn_state 54 MiB x 2 copies + conv/tail/ple) and - # 448 MiB of prefix-cache checkpoints (8 x 55.99 MiB). Verified against - # the fp8 boot it replaces: 3543 - 330 - 448 = 2765 MiB -> 387,240 tokens, - # ledger said 386,904. + # -- this is a hard limit, not a backing ratio". Rates are measured, not + # derived: per rank, kv_attn (13 layers) pages 26.00 KiB per 4 tokens at + # fp8 and 52.00 KiB at bf16, while kv_qsa_bkey (13 layers) pages 3.25 KiB + # at BOTH — the QSA block key is an index, not attention state, and does + # not follow --kv-cache-dtype — so the rate is 7,488 B/token at fp8 and + # 14,144 at bf16 (1.89x, not 2x). The dtype-independent terms also live in + # the pool: ~110 MiB a sequence of recurrent state (kv_gdn_state 54 MiB x 2 + # copies + conv/tail/ple) and 448 MiB of prefix checkpoints (8 x 55.99). + # Verified against BOTH live boots: fp8 3543-330-448 = 2765 MiB -> 387,240 + # tokens (ledger 386,904); bf16 6912-220-448 = 6244 MiB -> 462,946 + # (ledger 462,752). Both within 0.1%. kv=$(awk -F= '/^FRANK_ARCHON_VRAM_KV_MIB=/{print $2}' "$ENV_FILE") seqs=$(awk -F= '/^FRANK_ARCHON_MAX_SEQS=/{print $2}' "$ENV_FILE") maxlen=$(awk -F= '/^FRANK_ARCHON_MAX_MODEL_LEN=/{print $2}' "$ENV_FILE") case "$(awk -F= '/^FRANK_ARCHON_KV_DTYPE=/{print $2}' "$ENV_FILE")" in fp8) bpt=7488 ;; - bf16) bpt=14976 ;; + bf16) bpt=14144 ;; *) fail "unknown FRANK_ARCHON_KV_DTYPE" ;; esac tokens=$(( (kv - 110 * seqs - 448) * 1048576 / bpt ))