docs(frank-archon): work-log entry and the 4-way capacity probe

Probe on the live service: four concurrent 196,960-token chats, 4/4 exact
marker retrieval, 787,842 tokens in 238.4 s (3,305 t/s aggregate prefill = the
single-stream ceiling, so the four streams shared one pipeline and the KV pool
never ran short); prefix-cache repeat answered in 1.0 s with cache_n=43008.
Recorded with the caveat that 8 generated tokens do not make a decode rate.
This commit is contained in:
Frank
2026-10-13 19:30:46 -07:00
parent bae78e7dae
commit 7daa886dd6
2 changed files with 66 additions and 0 deletions
@@ -385,6 +385,33 @@ Oct 13 19:21:08 dankserver systemd[1]: Started frank-archon.service
the boot, i.e. the root `ExecStartPre` hook did the write the service process
(User=dank) cannot.
### Capacity probe (2026-10-13, live service)
`scripts/probe-frank-archon-kv.py` now reads the slot count, window and KV dtype
from the deployed env on the host, so the probe cannot drift from the service:
it printed `shape: 4 x 200,000 tokens, kv fp8` and fired **four concurrent chats
of 196,960 prompt tokens each** (646,299 chars, 98.5 % of the window), each
carrying a marker whose first word is unique.
| Stream | prompt | wall | prefill | retrieval |
|---|---|---|---|---|
| B1 | 196,961 | 238.3 s | 1,423 t/s | exact, HIT |
| B2 | 196,960 | 98.0 s | 2,043 t/s | exact, HIT |
| B3 | 196,961 | 201.1 s | 993 t/s | exact, HIT |
| B4 | 196,960 | 149.9 s | 1,327 t/s | exact, HIT |
| **total** | **787,842** | **238.4 s** | **3,305 t/s aggregate** | 4/4 |
All four answered their own marker; nothing was preempted, refused, or truncated
(`finish=stop`, `cache_n=0`, so no prefix cache padded the result). Aggregate
prefill equals the single-stream rate measured in the same probe (3,397 t/s at
43.5k), i.e. the four streams share one prefill pipeline at the slab-limited
ceiling and the engine never ran short of KV. Prefix cache intact: the same
43.5k prompt repeated answered in 1.0 s with `cache_n=43008`.
The per-stream `decode` column is NOT a decode rate here — 8 generated tokens
make it first-token dominated (the bench law: generate ≥ 128 tokens to read a
rate). This probe answers one question only: does the endpoint owe four
full-height conversations at once, and answer each one exactly. It does.
## Gates (measured at cutover)
+39
View File
@@ -1,5 +1,44 @@
# Bundle Change Log
## 2026-10-13
- **frank-archon outage: a kernel knob, not the model.** The 2026-10-12 23:17
power failure on dankserver reset `/sys/module/ttm/parameters/pages_limit`
from the manual 200 GiB raise back to its default (half of RAM = 125,988 MiB),
below the ~147 GiB pinned-RAM ledger the engine needs; the serve script's
preflight named the refusal and exited 1, and `Restart=always` re-ran it
**6,919 times in ~20 h** — `activating (auto-restart)`, exit 1 in 50 ms, no
container, no engine log. Law adopted: **a sysfs raise is a countdown to the
next reboot**, so the gate raises the knob itself. New
`scripts/ttm-preflight.sh` is sourced by the serve script (verify, raise,
re-verify, then refuse by name) and is also the unit's
`ExecStartPre=+…--raise` root hook — `+` because the service is `User=dank`
and the knob is root-owned 0644, which is what the first attempt (a plain
write inside the serve script) got wrong. `FRANK_ARCHON_TTM_TARGET_MIB=204800`
records the host ceiling.
- **Shape: 4 × 200,000 with fp8 KV (Frank's call).** `--max-num-seqs 4`,
`--kv-cache-dtype fp8`, pool and slab untouched (6,912 / 22,208 MiB = the
measured 29,120/card ceiling): the pool must back `seqs × max-model-len`, and
at bf16 four slots cost 11,679 MiB, which would cut the resident expert slab
to 17,441 (−33 %) — fp8 costs 6,601 and leaves it alone. 4 × (1 + spec 2) = 12
fused-all-reduce rows, inside the 25-row bound where `wht6` is bit-exact.
`models.yml` needed no change: the per-request window is unchanged, only the
concurrency.
- **Deployed at `bae78e7`, gates green.** Boot ledger: `kv cache 6.75 GiB …
backs 4 sequences all at 200000 tokens at once` — 843,276 tokens against the
sizing law's 843,566 (0.03 %), sampler at 12 rows, weights filled to 147 KiB
of slack, cards ~2 GiB free, KV lending 5.87 GiB to the slab. Live probe
(now reading the shape from the deployed env): four concurrent 196,960-token
chats, **4/4 exact marker retrieval**, 787,842 tokens in 238.4 s = 3,305 t/s
aggregate prefill, no preemption; prefix cache verified (repeat answered in
1.0 s, `cache_n=43008`). Playwright e2e 32 passed + 1 flaky (the known Plex
homepage retry); archon bats seam 21/21 including three new ttm-gate tests.
- **Power-failure casualties, NOT fixed (outside the archon window):**
`mnt-md0.mount` failed at boot, so `/mnt/md0` is an empty mountpoint (the
raid6 array itself is healthy, 7/7; fstab has `nofail` so boot continued) —
media libraries on it read empty. `frank-job-digest.service` (frankbook
job-hunt digest) is in `list-units --state=failed`.
## 2026-10-10
- **Fleet service rename: the StarCraft names became the SERVICE names.**
The 2026-10-09 wire alias moved onto every repo artifact: units, serve