docs: frank-large cache pin verified warm (16x prefill) + depth law

Probe on the live endpoint (2026-10-06): 17.3k cold prefill 27.0 s ->
warm turn resumes at cache_n 16,384, prefill 1.7 s. The boot WARNING
'smaller than ONE full-context entry' sizes against the native 262,144
window, not our 200,700 slot — documented as expected in the checklist.
Tuning notes record the depth law (decode 5.5 t/s @ 46k -> 3.3-3.8 @ 80k
vs 30-50 shallow; KV reads ~8 GB/s effective; no supported flag touches
decode) and the AGENTS.md unit paragraph regains the unit-facts
parenthetical the reflow dropped.
This commit is contained in:
Frank committed 2026-10-06 00:34:59 -07:00
1 parent 3eb4fa2b62
commit d39424ffba
3 files changed
+89 -11

No files matched your search

+8
View File
@@ -81,6 +81,14 @@ frank-small, port 7771]
No server-side sampler — the thinking recipe rides the wire in models.yml.
Rollback: revert the swap commit; the ROCmFPX payload + fork build +
`build-rocmfpx.sh` pin all remain on disk
Tuned 2026-10-06: `HALOGEN_CACHE_MB=16384` pins the prompt cache —
auto was OFF at the 36.9 GB two-entry floor, so every deep request
re-prefilled its 79-81k context (85-180 s each); a 17.3k cold prefill
of 27.0 s now resumes in 1.7 s on the warm turn (cache_n 16,384). The
serve script captures the engine ledger on disk (foreground `podman
run`; the `-d` era wrote only the container ID). Decode-at-depth is
structural: 3.3-3.8 t/s at 80k vs 30-50 shallow; no supported flag
touches it.
- [[frank-small-llama-swift15-iq3s-mtp-131072]] — **ACTIVE since 2026-10-05
(evening); 2 × 131,072 no-spec since 2026-10-06** (root systemd unit,
port 7771): llama.cpp master (CUDA sm_86) serving ukisai/Swift-1.5-