From a9d45e70120d2fcfb684d7776ffdcd20e1349530 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 13 Oct 2026 20:51:32 -0700 Subject: [PATCH] frank-archon: the slab is a threshold, and decode is where slots cost MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured, 512 tokens, greedy, cold, at the deployed 3 x 200,000 fp8 shape: prefill 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,290 t/s at 43.5k — the third slab step bought nothing, so the slab is a threshold whose cliff is the 2026-10-11 arm at 26,071 (4,379), not a slope. Decode ladder: 99.0 t/s single-stream at 43.5k, 93.3 at 197k (nearly window-free), and 3 concurrent streams 113.9/43.3/10.9 = 18.4 t/s aggregate — the slot count is paid in decode concurrency, the axis the shape hunt never measured because probe leg B generates 24 tokens. probe-frank-archon-kv.py gains leg D (N-stream decode, 512 tokens, per-stream + aggregate) so no resize is priced on prefill alone. --- AGENTS.md | 16 +++++++--- scripts/probe-frank-archon-kv.py | 29 ++++++++++++++++-- .../frank-archon-radiance-swift15-w4dn8.md | 30 +++++++++++++++++++ serverAppConfigs-docs/log.md | 20 +++++++++++++ 4 files changed, 89 insertions(+), 6 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 85e30b3..13b1da8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -550,10 +550,18 @@ all at 200000 tokens at once, which is the worst case` — `kv_attn` page every 1,792 MiB buys ~1,300 resident experts, and the pinned host pool hands the same experts back: 8,893 → 7,580 slots); `unclaimed 477.30 MiB`, cards settled ~336 MiB free against the engine's 209.19 MiB floor, KV lending -4.22 GiB to the expert cache. Prefill tracks this slab: the 24,000 → 25,600 -step measured +8.9 % single-stream (3,290 → 3,584 t/s at 43.5k) and +17.6 % -concurrent (3-way 197k aggregate), so the pinned host pool is the slow path and -every MiB moved from RAM onto the card is throughput. +4.22 GiB to the expert cache. Prefill tracks this slab only up to a +**threshold**: 22,208 → 3,397 and 24,000 → 3,290 t/s at 43.5k are one band +(±3 %), and the 25,600 arm measured **3,290** — the third step bought nothing +(the 3,584 arm did not reproduce at 3 slots); the cliff is at ~26 GiB +(26,071 → 4,379 on 2026-10-11's arm), so below that the slab is a threshold, +not a slope. + +**Decode ladder (2026-10-13, 512 tokens, greedy, this shape)**: 99.0 t/s +single-stream at 43.5k, 93.3 at 197k — decode is nearly window-free (−6 % for +4.5× depth); **3 concurrent streams give 113.9/43.3/10.9 t/s = 18.4 t/s +aggregate**, so the slot count is paid in decode concurrency (a fused-step +lottery), not in prefill; leg D of `probe-frank-archon-kv.py` measures it. Shape notes carried from the fp8-native era and unchanged: wht6 gains, MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk diff --git a/scripts/probe-frank-archon-kv.py b/scripts/probe-frank-archon-kv.py index eac6a76..a08b480 100755 --- a/scripts/probe-frank-archon-kv.py +++ b/scripts/probe-frank-archon-kv.py @@ -6,7 +6,7 @@ No scratch container, no mask: the service under test is the service. ssh dankserver 'python3 /home/dank/serverAppConfigs/scripts/probe-frank-archon-kv.py' -Three probes, in increasing cost: +Four probes, in increasing cost: A decode at depth — one ~51k-token chat completion, 512 tokens, greedy. Reports the engine's own `timings` (prefill, decode, draft acceptance). The bench law applies: >=128 generated tokens, greedy, real text. @@ -17,7 +17,14 @@ Three probes, in increasing cost: resident at once, exact retrieval in every one of them. C prefix-cache restore — probe A's cold prompt again, so the disk tier has to give the blocks back (cache_n > 0, TTFT collapses). - + D concurrency decode — `seqs` CONCURRENT 512-token generations at ~25 % of + the window. This is the cost a slot count imposes on DECODE, which no + prefill number shows: the fused-step schedule is unfair, so every stream is + printed plus the aggregate. Measured 2026-10-13 at 3 x 200,000 fp8: + single-stream 99.0 t/s (43.5k) and 93.3 (197k) — decode is nearly + window-free — against 3-stream 113.9/43.3/10.9 t/s = 18.4 t/s aggregate. + Concurrency, not the slab, is what decode pays for slots. The 2026-10-13 + shape hunt missed this for a whole day because leg B generates 24 tokens. PROBE LAW (learned the hard way, twice): radiance routes the thinking block to `reasoning_content`, so a chat request whose budget is spent inside `` returns EMPTY `content` with `finish_reason: stop`. A probe that @@ -168,6 +175,24 @@ def main(): except Exception as e: # a preemption/timeout lands here print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}") print(f"[B] total {time.time() - t0:.1f}s") + # D — concurrency decode: seqs streams, 512 tokens each, ~25 % depth. + dchars = int(0.25 * maxlen * 3.28) + dbodies = [chat(deep, dchars, f"D{i + 1}-9137", f"flint-marker-{i}", 512) + for i in range(seqs)] + print(f"[D] {seqs} concurrent streams x 512 tokens, " + f"{dchars:,} chars each") + t0 = time.time() + with concurrent.futures.ThreadPoolExecutor(max_workers=seqs) as ex: + futs = [ex.submit(post, b) for b in dbodies] + for i, fut in enumerate(futs, 1): + try: + j, wall, _ = fut.result() + show(f"D-s{i}", j, wall) + except Exception as e: # a preemption/timeout lands here + print(f"[D-s{i}] FAILED after {time.time() - t0:.1f}s: {e!r}") + tot = time.time() - t0 + print(f"[D] {seqs} x 512 tokens in {tot:.1f}s = " + f"{seqs * 512 / tot:.1f} t/s aggregate decode") j, wall, _ = post(chat(deep, 140_000, "A1-9137", "zephyr-lantern-7", 512)) show("C-prefix-hit", j, wall, "zephyr-lantern-7") diff --git a/serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md b/serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md index e476035..7a938ab 100644 --- a/serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md +++ b/serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md @@ -421,6 +421,36 @@ of **199,984 tokens** each (599,952 tokens) were accepted and completed on `/v1/completions`; a fourth full-height request is refused as "too large to ever fit" — radiance never truncates. +### The decode ladder (2026-10-13): slots are paid in concurrency, not prefill + +Frank's challenge — "why are you not tracking decode?" — was right, and the +answer is a number now. 512 generated tokens (the bench law), greedy, cold +prompts, this shape (3 x 200,000, fp8, slab 25,600): + +``` +1-stream 43.5k: decode 99.0 t/s (prefill 4,042, accept 0.952) +1-stream 197k: decode 93.3 t/s (prefill 4,826, accept 0.938) +3-stream 43.5k: decode 113.9 / 43.3 / 10.9 t/s wall 83.4s +3-stream 197k: decode 88.9 / 16.3 / 64.6 t/s wall 122.4s +``` + +Two laws came out of it. **Decode is nearly window-free** — 99.0 at 43.5k, +93.3 at 197k, −6 % for 4.5x the depth — so a long conversation costs +prefill, not tokens-per-second. **Concurrency is where the slot count is +paid**: three simultaneous streams give 18.4 t/s aggregate at 43.5k and 12.5 +at 197k against 99 single-stream, and the split is a lottery +(113.9/43.3/10.9), not a fair share — the fused-step schedule hands the step +to whoever it reached first, so one stream finishes at 113 t/s while its +sibling crawls at 10.9. That is what "3 slots" owes per token, and no prefill +number shows it. + +Corpus caveat, stated at the claim: this probe's filler is a repetitive +ladder, so MTP acceptance is 0.93–0.97 against the 0.740 of the 2026-10-11 +table below; these rows flatter decode relative to that table and are the +baseline for *this* corpus, not a replacement for those rows. +`probe-frank-archon-kv.py` now carries this leg as **D** (leg B generates 24 +tokens, which is why a day of shape work priced only prefill). + ### Recovery evidence ```text diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 920cee7..f5332ca 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -72,6 +72,26 @@ raid6 array itself is healthy, 7/7; fstab has `nofail` so boot continued) — media libraries on it read empty. `frank-job-digest.service` (frankbook job-hunt digest) is in `list-units --state=failed`. +- **The slab is a threshold, not a slope** — measured, same method (probe A + leg: 43,512 tokens, greedy, cold, disk prefix cache clean on every boot): + 22,208 → 3,397 t/s, 24,000 → 3,290, **25,600 → 3,290**. The third step + bought nothing; the +22 % belongs to the 2026-10-11 arm at slab 26,071 + (4,379 t/s), so the cliff is real and sits at ~26 GiB. The 2-slot arm that + would clear it (pool 3,072 + slab 27,897) remains untested — and the + paragraph above is amended: that arm's payoff is +22 % prefill at 2 slots, + not the +8.9 % the deployed pair ever showed (24,000 → 25,600 = 0 %). +- **Frank's decode challenge, answered with numbers**: 512 tokens, greedy, + cold — **decode 99.0 t/s single-stream at 43.5k and 93.3 at 197k** (decode + is nearly window-free: −6 % for 4.5× depth), while **3 concurrent streams + give 113.9/43.3/10.9 t/s = 18.4 t/s aggregate** at 43.5k and 12.5 at 197k. + The slot count is paid in decode concurrency — a fused-step lottery, not a + fair share — which is exactly the axis the shape hunt was blind to (probe + leg B generates 24 tokens). Corpus caveat at the claim: the ladder's filler + is repetitive, acceptance 0.93–0.97 vs the 0.740 of the 2026-10-11 decode + table, so these rows are the baseline for this corpus, not a replacement. +- **Seam:** `probe-frank-archon-kv.py` gains **leg D** (N-stream decode, 512 + tokens, per-stream + aggregate) so no future shape change can be priced on + prefill alone. ## 2026-10-10