docs(frank-zealot): measured exact-KV boot, floor rates, chat-path instrument

Deployed shape verified live: KV 16,384.00 MiB (ledger-exact 32,768 B/token x
524,288), RS 100.50 MiB (2 cells), settled nvidia-smi 21,176 used / 2,949 free,
n_slots 2, n_ctx_slot 262144, DropInPaths empty.

Rates recorded as FLOORS: another consumer drove the endpoint through the whole
window (56 prompts on the other slot in three minutes, 96% GPU-util). Single
stream 134.1-141.5 (145.8/163.2 in a lighter minute) vs the 168.7 q8_0 record;
two-stream aggregate 256.6-265.1 vs ~300 across three; prefill 3,159.5 at
106,282 beats the 2,877 record while contended; decode at 106k depth 71.5 (new
baseline, no prior). Per-stream concurrency improved: ~130 each vs 102.8.

Instrument correction: the PrismML fork build 10777 answers 500 'The model
produced output that does not match the expected Content-only format' for a
greedy ignore_eos run on the raw /completion path, so the per-request timings
must be read off /v1/chat/completions. AGENTS.md Benchmarking Policy corrected.
This commit is contained in:
Frank committed 2026-10-18 21:28:04 -07:00
1 parent 71bc1735d3
commit d457985b8d
3 files changed
+91 -16

No files matched your search

+6 -2
View File
@@ -252,8 +252,12 @@ archive.
real users, and a bench beside a live service starves the GPU and corrupts
results. Use the runtime's own harness: `llama-bench` for the llama.cpp
endpoints; a live llama.cpp server (frank-zealot) also reports honest
per-request numbers in its `/completion` `timings` object (prefill/decode
rates, `draft_n`, `draft_n_accepted`), which is how frank-zealot was measured.
per-request numbers in a `timings` object (prefill/decode rates, `draft_n`,
`draft_n_accepted`) — but on the PrismML fork build 10777 that object is only
reaching you on **`/v1/chat/completions`**: a greedy `ignore_eos` run on the raw
`/completion` path there answers **500 "The model produced output that does not
match the expected Content-only format"**, so measure through the chat path
(measured 2026-10-18 on the exact-KV reslice).
halogen has NO `/completion` path at all — frank-dragoon's rates come from
wall-clock around `/v1/chat/completions` plus the boot ledger. Rolling your own
bench requires asking first.