From 7daa886dd6ab11a4b8a1790b9be5e8f1f5695bc4 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 13 Oct 2026 19:30:46 -0700 Subject: [PATCH] docs(frank-archon): work-log entry and the 4-way capacity probe Probe on the live service: four concurrent 196,960-token chats, 4/4 exact marker retrieval, 787,842 tokens in 238.4 s (3,305 t/s aggregate prefill = the single-stream ceiling, so the four streams shared one pipeline and the KV pool never ran short); prefix-cache repeat answered in 1.0 s with cache_n=43008. Recorded with the caveat that 8 generated tokens do not make a decode rate. --- .../frank-archon-radiance-swift15-w4dn8.md | 27 +++++++++++++ serverAppConfigs-docs/log.md | 39 +++++++++++++++++++ 2 files changed, 66 insertions(+) diff --git a/serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md b/serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md index 48c830f..179770f 100644 --- a/serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md +++ b/serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md @@ -385,6 +385,33 @@ Oct 13 19:21:08 dankserver systemd[1]: Started frank-archon.service the boot, i.e. the root `ExecStartPre` hook did the write the service process (User=dank) cannot. +### Capacity probe (2026-10-13, live service) + +`scripts/probe-frank-archon-kv.py` now reads the slot count, window and KV dtype +from the deployed env on the host, so the probe cannot drift from the service: +it printed `shape: 4 x 200,000 tokens, kv fp8` and fired **four concurrent chats +of 196,960 prompt tokens each** (646,299 chars, 98.5 % of the window), each +carrying a marker whose first word is unique. + +| Stream | prompt | wall | prefill | retrieval | +|---|---|---|---|---| +| B1 | 196,961 | 238.3 s | 1,423 t/s | exact, HIT | +| B2 | 196,960 | 98.0 s | 2,043 t/s | exact, HIT | +| B3 | 196,961 | 201.1 s | 993 t/s | exact, HIT | +| B4 | 196,960 | 149.9 s | 1,327 t/s | exact, HIT | +| **total** | **787,842** | **238.4 s** | **3,305 t/s aggregate** | 4/4 | + +All four answered their own marker; nothing was preempted, refused, or truncated +(`finish=stop`, `cache_n=0`, so no prefix cache padded the result). Aggregate +prefill equals the single-stream rate measured in the same probe (3,397 t/s at +43.5k), i.e. the four streams share one prefill pipeline at the slab-limited +ceiling and the engine never ran short of KV. Prefix cache intact: the same +43.5k prompt repeated answered in 1.0 s with `cache_n=43008`. + +The per-stream `decode` column is NOT a decode rate here — 8 generated tokens +make it first-token dominated (the bench law: generate ≥ 128 tokens to read a +rate). This probe answers one question only: does the endpoint owe four +full-height conversations at once, and answer each one exactly. It does. ## Gates (measured at cutover) diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 7e4ab0c..bec796c 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -1,5 +1,44 @@ # Bundle Change Log +## 2026-10-13 +- **frank-archon outage: a kernel knob, not the model.** The 2026-10-12 23:17 + power failure on dankserver reset `/sys/module/ttm/parameters/pages_limit` + from the manual 200 GiB raise back to its default (half of RAM = 125,988 MiB), + below the ~147 GiB pinned-RAM ledger the engine needs; the serve script's + preflight named the refusal and exited 1, and `Restart=always` re-ran it + **6,919 times in ~20 h** — `activating (auto-restart)`, exit 1 in 50 ms, no + container, no engine log. Law adopted: **a sysfs raise is a countdown to the + next reboot**, so the gate raises the knob itself. New + `scripts/ttm-preflight.sh` is sourced by the serve script (verify, raise, + re-verify, then refuse by name) and is also the unit's + `ExecStartPre=+…--raise` root hook — `+` because the service is `User=dank` + and the knob is root-owned 0644, which is what the first attempt (a plain + write inside the serve script) got wrong. `FRANK_ARCHON_TTM_TARGET_MIB=204800` + records the host ceiling. +- **Shape: 4 × 200,000 with fp8 KV (Frank's call).** `--max-num-seqs 4`, + `--kv-cache-dtype fp8`, pool and slab untouched (6,912 / 22,208 MiB = the + measured 29,120/card ceiling): the pool must back `seqs × max-model-len`, and + at bf16 four slots cost 11,679 MiB, which would cut the resident expert slab + to 17,441 (−33 %) — fp8 costs 6,601 and leaves it alone. 4 × (1 + spec 2) = 12 + fused-all-reduce rows, inside the 25-row bound where `wht6` is bit-exact. + `models.yml` needed no change: the per-request window is unchanged, only the + concurrency. +- **Deployed at `bae78e7`, gates green.** Boot ledger: `kv cache 6.75 GiB … + backs 4 sequences all at 200000 tokens at once` — 843,276 tokens against the + sizing law's 843,566 (0.03 %), sampler at 12 rows, weights filled to 147 KiB + of slack, cards ~2 GiB free, KV lending 5.87 GiB to the slab. Live probe + (now reading the shape from the deployed env): four concurrent 196,960-token + chats, **4/4 exact marker retrieval**, 787,842 tokens in 238.4 s = 3,305 t/s + aggregate prefill, no preemption; prefix cache verified (repeat answered in + 1.0 s, `cache_n=43008`). Playwright e2e 32 passed + 1 flaky (the known Plex + homepage retry); archon bats seam 21/21 including three new ttm-gate tests. +- **Power-failure casualties, NOT fixed (outside the archon window):** + `mnt-md0.mount` failed at boot, so `/mnt/md0` is an empty mountpoint (the + raid6 array itself is healthy, 7/7; fstab has `nofail` so boot continued) — + media libraries on it read empty. `frank-job-digest.service` (frankbook + job-hunt digest) is in `list-units --state=failed`. + + ## 2026-10-10 - **Fleet service rename: the StarCraft names became the SERVICE names.** The 2026-10-09 wire alias moved onto every repo artifact: units, serve