docs: frank-large cache pin verified warm (16x prefill) + depth law
Probe on the live endpoint (2026-10-06): 17.3k cold prefill 27.0 s -> warm turn resumes at cache_n 16,384, prefill 1.7 s. The boot WARNING 'smaller than ONE full-context entry' sizes against the native 262,144 window, not our 200,700 slot — documented as expected in the checklist. Tuning notes record the depth law (decode 5.5 t/s @ 46k -> 3.3-3.8 @ 80k vs 30-50 shallow; KV reads ~8 GB/s effective; no supported flag touches decode) and the AGENTS.md unit paragraph regains the unit-facts parenthetical the reflow dropped.
This commit is contained in:
1 parent
3eb4fa2b62
commit
d39424ffba
3 files changed
+89
-11
No files matched your search
@@ -81,6 +81,14 @@ frank-small, port 7771]
|
||||
No server-side sampler — the thinking recipe rides the wire in models.yml.
|
||||
Rollback: revert the swap commit; the ROCmFPX payload + fork build +
|
||||
`build-rocmfpx.sh` pin all remain on disk
|
||||
Tuned 2026-10-06: `HALOGEN_CACHE_MB=16384` pins the prompt cache —
|
||||
auto was OFF at the 36.9 GB two-entry floor, so every deep request
|
||||
re-prefilled its 79-81k context (85-180 s each); a 17.3k cold prefill
|
||||
of 27.0 s now resumes in 1.7 s on the warm turn (cache_n 16,384). The
|
||||
serve script captures the engine ledger on disk (foreground `podman
|
||||
run`; the `-d` era wrote only the container ID). Decode-at-depth is
|
||||
structural: 3.3-3.8 t/s at 80k vs 30-50 shallow; no supported flag
|
||||
touches it.
|
||||
- [[frank-small-llama-swift15-iq3s-mtp-131072]] — **ACTIVE since 2026-10-05
|
||||
(evening); 2 × 131,072 no-spec since 2026-10-06** (root systemd unit,
|
||||
port 7771): llama.cpp master (CUDA sm_86) serving ukisai/Swift-1.5-
|
||||
|
||||
Reference in new issue
Block a user