docs(frank-archon): record the bf16 KV shape, its measured cost, and the probe
Runbook section `2026-10-11 — unquantized KV at 2 x 200,000`: the dtype law (only bf16|fp8, no K/V split; the QSA block key is dtype-fixed, which is why bf16 costs 1.89x and not 2x), the pool formula that reproduces both live boots within 0.1%, the seqs-3-to-2 provenance (the tcclaviger fork's torch.compile constraint, not radiance's), the boot ledger, and the A/B. The A/B, fp8 control arm through a transient FRANK_ARCHON_ENV_FILE drop-in: prefill -29% (4,379 -> 3,126 cold), TTFT at the full window +25% (the 2 x 197k pair: 110.5 s -> 138.1 s), decode inside the +-3% band on two of three passes, draft acceptance UP 0.661 -> 0.740. The price is prefill, not decode. Open call for Frank: keep exact KV, or take the cheap fp8 2 x 200k fit (pool 3,648). Also corrects two stale index entries (w4dn8 said STAGED, fp8-native said ACTIVE), adds the worklog entry, and commits the verification tool as scripts/probe-frank-archon-kv.py — it carries the probe law that cost two false negatives: radiance answers in reasoning_content, so a chat probe with a small budget reads as empty content + finish_reason stop on a healthy server. Tests 368/368.
This commit is contained in:
Frank
@@ -166,19 +166,29 @@ FRANK_ARCHON_VRAM_KV_MIB=6912
|
||||
# is ONE kv_dtype for the whole cache, so "exact K, quantized V" (llama.cpp's
|
||||
# --cache-type-k/--cache-type-v pair, which is what frank-zealot runs) is
|
||||
# UNSERVABLE here: the rung below bf16 is fp8 on both halves.
|
||||
# Measured per-rank cost, read out of the boot ledger's group table:
|
||||
# Measured per-rank cost, read out of the boot ledger's group table (fp8 boot
|
||||
# 2026-10-08, bf16 boot 2026-10-11):
|
||||
# kv_attn 13 layers, 4-token blocks, page 26.00 KiB (fp8) -> 52.00 KiB
|
||||
# kv_qsa_bkey 13 layers, 4-token blocks, page 3.25 KiB (fp8) -> 6.50 KiB
|
||||
# => 7.3125 KiB/token fp8, 14.625 KiB/token bf16
|
||||
# kv_qsa_bkey 13 layers, 4-token blocks, page 3.25 KiB at BOTH dtypes —
|
||||
# the QSA block key is an index, not attention state, and does
|
||||
# NOT follow --kv-cache-dtype
|
||||
# => 7.3125 KiB/token fp8, 13.8125 KiB/token bf16: bf16 costs 1.89x fp8,
|
||||
# not 2x, precisely because the bkey plane is dtype-fixed.
|
||||
# The dtype-INDEPENDENT terms also live in the pool: kv_gdn_state 54 MiB x 2
|
||||
# copies per sequence (+ conv/tail/ple ~= 110 MiB a sequence) and the prefix
|
||||
# checkpoint region 8 x 55.99 = 448 MiB. So:
|
||||
# pool_mib >= tokens x 14.625 KiB + 110 x seqs + 448
|
||||
# 6912 -> (6912 - 220 - 448) x 1024 / 14.625 = 437,320 tokens >= 2 x 200,000
|
||||
# pool_mib >= tokens x 13.8125 KiB + 110 x seqs + 448
|
||||
# 6912 -> (6912 - 220 - 448) x 1024 / 13.8125 = 462,946 tokens; the engine
|
||||
# reported 462,752 (0.04% page rounding) — 2 x 200,000 with 62k of slack,
|
||||
# and the ledger says it outright: "backs 2 sequences all at 200000 tokens
|
||||
# at once, which is the worst case".
|
||||
# (the fp8 pool of 3543 gave 386,904 — it never held 2 x 200k, it held 1.93
|
||||
# sessions, and a third long conversation preempted to the host/disk tiers).
|
||||
# Secondary cost: bf16 doubles the prefix-cache entry (58.50 -> 117 KiB), so
|
||||
# the host tier below doubles too.
|
||||
# Secondary cost: the prefix-cache entry grows with the attention plane
|
||||
# (one entry 58.50 KiB at fp8 -> 110.50 KiB at bf16, measured on the two boot
|
||||
# ledgers' `kv tier transfer` lines), so the host tier below doubles to keep
|
||||
# the same number of parked conversations, out of RAM the host has
|
||||
# (104 GiB available with the service loaded).
|
||||
FRANK_ARCHON_KV_DTYPE=bf16
|
||||
# Spec decode: the checkpoint's own MTP head at the publisher's depth-3
|
||||
# operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the
|
||||
|
||||
Executable
+147
@@ -0,0 +1,147 @@
|
||||
#!/usr/bin/env python3
|
||||
"""probe-frank-archon-kv.py — verify the 2 x 200,000 KV contract on a LIVE
|
||||
frank-archon (radiance) service. Run it ON dankserver: it talks to
|
||||
127.0.0.1:7770 and builds its prompts from the repo checkout there.
|
||||
No scratch container, no mask: the service under test is the service.
|
||||
|
||||
ssh dankserver 'python3 /home/dank/serverAppConfigs/scripts/probe-frank-archon-kv.py'
|
||||
|
||||
Three probes, in increasing cost:
|
||||
A decode at depth — one ~51k-token chat completion, 512 tokens, greedy.
|
||||
Reports the engine's own `timings` (prefill, decode, draft acceptance).
|
||||
The bench law applies: >=128 generated tokens, greedy, real text.
|
||||
B the capacity contract — TWO concurrent chats of ~197k prompt tokens
|
||||
(~394k tokens resident) each carrying a unique marker at its head. Both
|
||||
must answer their own marker. This is what a 2 x 200,000 endpoint owes:
|
||||
two full-height conversations resident at once, exact retrieval in both.
|
||||
C prefix-cache restore — probe A's cold prompt again, so the disk tier has
|
||||
to give the blocks back (cache_n > 0, TTFT collapses).
|
||||
|
||||
PROBE LAW (learned the hard way, twice): radiance routes the thinking block to
|
||||
`reasoning_content`, so a chat request whose budget is spent inside
|
||||
`<think>` returns EMPTY `content` with `finish_reason: stop`. A probe that
|
||||
reads `content` alone scores 0/2 on a healthy server. Every request here sets
|
||||
chat_template_kwargs.enable_thinking=false and reads both fields.
|
||||
|
||||
A/B method for a shape change (no scratch container): write a variant env file
|
||||
on the host, point a transient systemd drop-in at it
|
||||
(`Environment=FRANK_ARCHON_ENV_FILE=...` — drop-ins are read AFTER the unit's
|
||||
own Environment= lines, so they win), restart, probe, delete the drop-in,
|
||||
daemon-reload, restart. `systemctl mask` is not available: the unit file in
|
||||
/etc/systemd/system is a relink symlink into the checkout.
|
||||
"""
|
||||
import json
|
||||
import glob
|
||||
import time
|
||||
import urllib.request
|
||||
import concurrent.futures
|
||||
|
||||
BASE = "http://127.0.0.1:7770"
|
||||
MODEL = "frank-archon"
|
||||
REPO = "/home/dank/serverAppConfigs"
|
||||
GLOBS = [f"{REPO}/scripts/*.sh", f"{REPO}/serverAppConfigs-docs/**/*.md"]
|
||||
|
||||
|
||||
def post(payload, timeout=1800):
|
||||
req = urllib.request.Request(
|
||||
BASE + "/v1/chat/completions",
|
||||
data=json.dumps(payload).encode(),
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
t0 = time.time()
|
||||
with urllib.request.urlopen(req, timeout=timeout) as r:
|
||||
body = json.loads(r.read().decode())
|
||||
hdr = dict(r.headers.items())
|
||||
return body, time.time() - t0, hdr
|
||||
|
||||
|
||||
def get(path, timeout=60):
|
||||
with urllib.request.urlopen(BASE + path, timeout=timeout) as r:
|
||||
return json.loads(r.read().decode())
|
||||
|
||||
|
||||
def corpus(min_chars):
|
||||
files = []
|
||||
for g in GLOBS:
|
||||
files.extend(sorted(glob.glob(g, recursive=True)))
|
||||
out, total = [], 0
|
||||
for f in files:
|
||||
try:
|
||||
t = open(f, encoding="utf-8", errors="replace").read()
|
||||
except OSError:
|
||||
continue
|
||||
out.append(t)
|
||||
total += len(t)
|
||||
if total > min_chars:
|
||||
break
|
||||
return "".join(text for text in out)
|
||||
|
||||
|
||||
def prompt(text, chars, salt, fact):
|
||||
"""Real repo text, salted so the prefix cache cannot answer, marker planted
|
||||
near the head, question at the tail."""
|
||||
head = f"\n[SALT {salt}] confidential marker: {fact}\n"
|
||||
body = text + text
|
||||
p = (head + body * max(1, chars // len(body)))[:chars]
|
||||
return p + "\n\nQuestion: what is the confidential marker in the " \
|
||||
"SALT block near the top of this document? Answer with the marker " \
|
||||
"alone, no other words.\n"
|
||||
|
||||
|
||||
def chat(text, chars, salt, fact, max_tokens):
|
||||
return {"model": MODEL,
|
||||
"messages": [{"role": "user",
|
||||
"content": prompt(text, chars, salt, fact)}],
|
||||
"max_tokens": max_tokens, "temperature": 0.0,
|
||||
"chat_template_kwargs": {"enable_thinking": False}}
|
||||
|
||||
|
||||
def show(tag, j, wall, expect=None):
|
||||
tim, u = j.get("timings") or {}, j.get("usage", {})
|
||||
ch = (j.get("choices") or [{}])[0]
|
||||
msg = ch.get("message") or {}
|
||||
txt = (msg.get("content") or msg.get("reasoning_content") or "").strip()
|
||||
acc = (tim["draft_n_accepted"] / tim["draft_n"]) if tim.get("draft_n") else None
|
||||
line = (f"[{tag}] prompt={u.get('prompt_tokens')} gen={u.get('completion_tokens')} "
|
||||
f"wall={wall:.1f}s prefill={tim.get('prompt_per_second', 0):.0f} t/s "
|
||||
f"decode={tim.get('predicted_per_second', 0):.1f} t/s "
|
||||
f"draft={tim.get('draft_n')}/{tim.get('draft_n_accepted')} "
|
||||
f"accept={acc if acc is None else round(acc, 4)} "
|
||||
f"cache_n={tim.get('cache_n')} finish={ch.get('finish_reason')}")
|
||||
if expect:
|
||||
hit = expect.split("-")[0] in txt.lower()
|
||||
line += f" expected={expect} answered={txt[:80]!r} HIT={hit}"
|
||||
print(line)
|
||||
|
||||
|
||||
def main():
|
||||
print("health:", json.dumps(get("/health")))
|
||||
print("models:", [m.get("id") for m in get("/v1/models").get("data", [])])
|
||||
deep = corpus(160_000)
|
||||
wide = corpus(1_100_000)
|
||||
|
||||
facts = {"B1": "cobalt-harbor-13", "B2": "quartz-meadow-42"}
|
||||
bodies = {s: chat(wide, 646_000, s, f, 24) for s, f in facts.items()}
|
||||
print(f"[B] {len(bodies['B1']['messages'][0]['content']):,} chars per "
|
||||
f"request, x2 concurrent")
|
||||
|
||||
# A + C first (cheap), B last (the expensive pair).
|
||||
for tag, salt in (("A-cold", "A1-9137"), ):
|
||||
j, wall, _ = post(chat(deep, 140_000, salt, "zephyr-lantern-7", 512))
|
||||
show(tag, j, wall, "zephyr-lantern-7")
|
||||
t0 = time.time()
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as ex:
|
||||
futs = {s: ex.submit(post, b) for s, b in bodies.items()}
|
||||
for s, fut in futs.items():
|
||||
try:
|
||||
j, wall, _ = fut.result()
|
||||
show(f"B-{s}", j, wall, facts[s])
|
||||
except Exception as e: # a preemption/timeout lands here
|
||||
print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}")
|
||||
print(f"[B] total {time.time() - t0:.1f}s")
|
||||
j, wall, _ = post(chat(deep, 140_000, "A1-9137", "zephyr-lantern-7", 512))
|
||||
show("C-prefix-hit", j, wall, "zephyr-lantern-7")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user