docs(frank-zealot): measured exact-KV boot, floor rates, chat-path instrument
Deployed shape verified live: KV 16,384.00 MiB (ledger-exact 32,768 B/token x 524,288), RS 100.50 MiB (2 cells), settled nvidia-smi 21,176 used / 2,949 free, n_slots 2, n_ctx_slot 262144, DropInPaths empty. Rates recorded as FLOORS: another consumer drove the endpoint through the whole window (56 prompts on the other slot in three minutes, 96% GPU-util). Single stream 134.1-141.5 (145.8/163.2 in a lighter minute) vs the 168.7 q8_0 record; two-stream aggregate 256.6-265.1 vs ~300 across three; prefill 3,159.5 at 106,282 beats the 2,877 record while contended; decode at 106k depth 71.5 (new baseline, no prior). Per-stream concurrency improved: ~130 each vs 102.8. Instrument correction: the PrismML fork build 10777 answers 500 'The model produced output that does not match the expected Content-only format' for a greedy ignore_eos run on the raw /completion path, so the per-request timings must be read off /v1/chat/completions. AGENTS.md Benchmarking Policy corrected.
This commit is contained in:
1 parent
71bc1735d3
commit
d457985b8d
3 files changed
+91
-16
No files matched your search
@@ -252,8 +252,12 @@ archive.
|
||||
real users, and a bench beside a live service starves the GPU and corrupts
|
||||
results. Use the runtime's own harness: `llama-bench` for the llama.cpp
|
||||
endpoints; a live llama.cpp server (frank-zealot) also reports honest
|
||||
per-request numbers in its `/completion` `timings` object (prefill/decode
|
||||
rates, `draft_n`, `draft_n_accepted`), which is how frank-zealot was measured.
|
||||
per-request numbers in a `timings` object (prefill/decode rates, `draft_n`,
|
||||
`draft_n_accepted`) — but on the PrismML fork build 10777 that object is only
|
||||
reaching you on **`/v1/chat/completions`**: a greedy `ignore_eos` run on the raw
|
||||
`/completion` path there answers **500 "The model produced output that does not
|
||||
match the expected Content-only format"**, so measure through the chat path
|
||||
(measured 2026-10-18 on the exact-KV reslice).
|
||||
halogen has NO `/completion` path at all — frank-dragoon's rates come from
|
||||
wall-clock around `/v1/chat/completions` plus the boot ledger. Rolling your own
|
||||
bench requires asking first.
|
||||
|
||||
Reference in new issue
Block a user