frank-archon: the slab is a threshold, and decode is where slots cost

Measured, 512 tokens, greedy, cold, at the deployed 3 x 200,000 fp8 shape:
prefill 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,290 t/s at 43.5k —
the third slab step bought nothing, so the slab is a threshold whose cliff is
the 2026-10-11 arm at 26,071 (4,379), not a slope. Decode ladder: 99.0 t/s
single-stream at 43.5k, 93.3 at 197k (nearly window-free), and 3 concurrent
streams 113.9/43.3/10.9 = 18.4 t/s aggregate — the slot count is paid in
decode concurrency, the axis the shape hunt never measured because probe leg
B generates 24 tokens. probe-frank-archon-kv.py gains leg D (N-stream decode,
512 tokens, per-stream + aggregate) so no resize is priced on prefill alone.
This commit is contained in:
Frank
2026-10-13 20:51:32 -07:00
parent 2581b8d4b6
commit a9d45e7012
4 changed files with 89 additions and 6 deletions
+27 -2
View File
@@ -6,7 +6,7 @@ No scratch container, no mask: the service under test is the service.
ssh dankserver 'python3 /home/dank/serverAppConfigs/scripts/probe-frank-archon-kv.py'
Three probes, in increasing cost:
Four probes, in increasing cost:
A decode at depth — one ~51k-token chat completion, 512 tokens, greedy.
Reports the engine's own `timings` (prefill, decode, draft acceptance).
The bench law applies: >=128 generated tokens, greedy, real text.
@@ -17,7 +17,14 @@ Three probes, in increasing cost:
resident at once, exact retrieval in every one of them.
C prefix-cache restore — probe A's cold prompt again, so the disk tier has
to give the blocks back (cache_n > 0, TTFT collapses).
D concurrency decode — `seqs` CONCURRENT 512-token generations at ~25 % of
the window. This is the cost a slot count imposes on DECODE, which no
prefill number shows: the fused-step schedule is unfair, so every stream is
printed plus the aggregate. Measured 2026-10-13 at 3 x 200,000 fp8:
single-stream 99.0 t/s (43.5k) and 93.3 (197k) — decode is nearly
window-free — against 3-stream 113.9/43.3/10.9 t/s = 18.4 t/s aggregate.
Concurrency, not the slab, is what decode pays for slots. The 2026-10-13
shape hunt missed this for a whole day because leg B generates 24 tokens.
PROBE LAW (learned the hard way, twice): radiance routes the thinking block to
`reasoning_content`, so a chat request whose budget is spent inside
`<think>` returns EMPTY `content` with `finish_reason: stop`. A probe that
@@ -168,6 +175,24 @@ def main():
except Exception as e: # a preemption/timeout lands here
print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}")
print(f"[B] total {time.time() - t0:.1f}s")
# D — concurrency decode: seqs streams, 512 tokens each, ~25 % depth.
dchars = int(0.25 * maxlen * 3.28)
dbodies = [chat(deep, dchars, f"D{i + 1}-9137", f"flint-marker-{i}", 512)
for i in range(seqs)]
print(f"[D] {seqs} concurrent streams x 512 tokens, "
f"{dchars:,} chars each")
t0 = time.time()
with concurrent.futures.ThreadPoolExecutor(max_workers=seqs) as ex:
futs = [ex.submit(post, b) for b in dbodies]
for i, fut in enumerate(futs, 1):
try:
j, wall, _ = fut.result()
show(f"D-s{i}", j, wall)
except Exception as e: # a preemption/timeout lands here
print(f"[D-s{i}] FAILED after {time.time() - t0:.1f}s: {e!r}")
tot = time.time() - t0
print(f"[D] {seqs} x 512 tokens in {tot:.1f}s = "
f"{seqs * 512 / tot:.1f} t/s aggregate decode")
j, wall, _ = post(chat(deep, 140_000, "A1-9137", "zephyr-lantern-7", 512))
show("C-prefix-hit", j, wall, "zephyr-lantern-7")