dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(frank-zealot): measured exact-KV boot, floor rates, chat-path instrument

Browse Source 
Deployed shape verified live: KV 16,384.00 MiB (ledger-exact 32,768 B/token x
524,288), RS 100.50 MiB (2 cells), settled nvidia-smi 21,176 used / 2,949 free,
n_slots 2, n_ctx_slot 262144, DropInPaths empty.

Rates recorded as FLOORS: another consumer drove the endpoint through the whole
window (56 prompts on the other slot in three minutes, 96% GPU-util). Single
stream 134.1-141.5 (145.8/163.2 in a lighter minute) vs the 168.7 q8_0 record;
two-stream aggregate 256.6-265.1 vs ~300 across three; prefill 3,159.5 at
106,282 beats the 2,877 record while contended; decode at 106k depth 71.5 (new
baseline, no prior). Per-stream concurrency improved: ~130 each vs 102.8.

Instrument correction: the PrismML fork build 10777 answers 500 'The model
produced output that does not match the expected Content-only format' for a
greedy ignore_eos run on the raw /completion path, so the per-request timings
must be read off /v1/chat/completions. AGENTS.md Benchmarking Policy corrected.
This commit is contained in:
Frank
2026-10-18 21:28:04 -07:00
parent 71bc1735d3
commit d457985b8d
3 changed files with 91 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/log.md
+27
View File
@@ -1,5 +1,32 @@
# Bundle Change Log
## 2026-10-18
- **frank-zealot resliced to two windows of exact KV (Frank's call, PR #21).**
`--ctx-size 786432 --parallel 3` at q8_0 K+V → **`524288 / 2` at f16 K+V**, so
both remaining conversations keep the native 262,144 window and the
compaction hub reads exact K and V. Halving the pool is what funds it: the
f16 pair is 32,768 B/token over the 8 full-attention layers = 16,384.00 MiB,
while the old 786,432-position pool would have needed ~24,576 MiB of KV alone
on a card that leaves 24,576 addressable. Measured boot: KV **16,384.00**,
RS **100.50** (2 cells), compute 1,408.06 + 1,088.07, settled `nvidia-smi`
**21,176 used / 2,949 free**, `n_ctx_slot = 262144`, `DropInPaths` empty. The
doc's old "f16 needs 26,112 MiB and cannot boot" figure was the 27B's
34,816 B/token applied to the 9B — corrected in the runbook.
- **Rates are floors, and that is recorded as such.** The endpoint had another
consumer for the whole measurement window (56 prompts on the other slot in
three minutes, 96 % GPU-util), so single-stream decode came back 134.1–141.5
(145.8/163.2 in a lighter minute) against the 3-slot record of 168.7,
two-stream aggregate 256.6–265.1 against ≈300 across three, and prefill
**3,159.5 at 106,282** beat the record's 2,877 while contended. Per-stream
concurrency improved (≈130 each vs 102.8); a clean re-measure needs the box
quiet.
- **Instrument correction (fleet law).** On the PrismML fork build 10777 the raw
`/completion` path answers **500 "The model produced output that does not
match the expected Content-only format"** for a greedy `ignore_eos` run, so
the honest per-request `timings` for this endpoint must be read off
`/v1/chat/completions`. `AGENTS.md` said `/completion`; it now says the chat
path.
## 2026-10-13
- **frank-archon outage: a kernel knob, not the model.** The 2026-10-12 23:17
power failure on dankserver reset `/sys/module/ttm/parameters/pages_limit`