docs(frank-archon): work-log entry and the 4-way capacity probe
Probe on the live service: four concurrent 196,960-token chats, 4/4 exact marker retrieval, 787,842 tokens in 238.4 s (3,305 t/s aggregate prefill = the single-stream ceiling, so the four streams shared one pipeline and the KV pool never ran short); prefix-cache repeat answered in 1.0 s with cache_n=43008. Recorded with the caveat that 8 generated tokens do not make a decode rate.
This commit is contained in:
1 parent
bae78e7dae
commit
7daa886dd6
2 files changed
+66
No files matched your search
@@ -1,5 +1,44 @@
|
||||
# Bundle Change Log
|
||||
|
||||
## 2026-10-13
|
||||
- **frank-archon outage: a kernel knob, not the model.** The 2026-10-12 23:17
|
||||
power failure on dankserver reset `/sys/module/ttm/parameters/pages_limit`
|
||||
from the manual 200 GiB raise back to its default (half of RAM = 125,988 MiB),
|
||||
below the ~147 GiB pinned-RAM ledger the engine needs; the serve script's
|
||||
preflight named the refusal and exited 1, and `Restart=always` re-ran it
|
||||
**6,919 times in ~20 h** — `activating (auto-restart)`, exit 1 in 50 ms, no
|
||||
container, no engine log. Law adopted: **a sysfs raise is a countdown to the
|
||||
next reboot**, so the gate raises the knob itself. New
|
||||
`scripts/ttm-preflight.sh` is sourced by the serve script (verify, raise,
|
||||
re-verify, then refuse by name) and is also the unit's
|
||||
`ExecStartPre=+…--raise` root hook — `+` because the service is `User=dank`
|
||||
and the knob is root-owned 0644, which is what the first attempt (a plain
|
||||
write inside the serve script) got wrong. `FRANK_ARCHON_TTM_TARGET_MIB=204800`
|
||||
records the host ceiling.
|
||||
- **Shape: 4 × 200,000 with fp8 KV (Frank's call).** `--max-num-seqs 4`,
|
||||
`--kv-cache-dtype fp8`, pool and slab untouched (6,912 / 22,208 MiB = the
|
||||
measured 29,120/card ceiling): the pool must back `seqs × max-model-len`, and
|
||||
at bf16 four slots cost 11,679 MiB, which would cut the resident expert slab
|
||||
to 17,441 (−33 %) — fp8 costs 6,601 and leaves it alone. 4 × (1 + spec 2) = 12
|
||||
fused-all-reduce rows, inside the 25-row bound where `wht6` is bit-exact.
|
||||
`models.yml` needed no change: the per-request window is unchanged, only the
|
||||
concurrency.
|
||||
- **Deployed at `bae78e7`, gates green.** Boot ledger: `kv cache 6.75 GiB …
|
||||
backs 4 sequences all at 200000 tokens at once` — 843,276 tokens against the
|
||||
sizing law's 843,566 (0.03 %), sampler at 12 rows, weights filled to 147 KiB
|
||||
of slack, cards ~2 GiB free, KV lending 5.87 GiB to the slab. Live probe
|
||||
(now reading the shape from the deployed env): four concurrent 196,960-token
|
||||
chats, **4/4 exact marker retrieval**, 787,842 tokens in 238.4 s = 3,305 t/s
|
||||
aggregate prefill, no preemption; prefix cache verified (repeat answered in
|
||||
1.0 s, `cache_n=43008`). Playwright e2e 32 passed + 1 flaky (the known Plex
|
||||
homepage retry); archon bats seam 21/21 including three new ttm-gate tests.
|
||||
- **Power-failure casualties, NOT fixed (outside the archon window):**
|
||||
`mnt-md0.mount` failed at boot, so `/mnt/md0` is an empty mountpoint (the
|
||||
raid6 array itself is healthy, 7/7; fstab has `nofail` so boot continued) —
|
||||
media libraries on it read empty. `frank-job-digest.service` (frankbook
|
||||
job-hunt digest) is in `list-units --state=failed`.
|
||||
|
||||
|
||||
## 2026-10-10
|
||||
- **Fleet service rename: the StarCraft names became the SERVICE names.**
|
||||
The 2026-10-09 wire alias moved onto every repo artifact: units, serve
|
||||
|
||||
Reference in new issue
Block a user