diff --git a/AGENTS.md b/AGENTS.md index 5a2c8e3..5dc0961 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -252,8 +252,12 @@ archive. real users, and a bench beside a live service starves the GPU and corrupts results. Use the runtime's own harness: `llama-bench` for the llama.cpp endpoints; a live llama.cpp server (frank-zealot) also reports honest -per-request numbers in its `/completion` `timings` object (prefill/decode -rates, `draft_n`, `draft_n_accepted`), which is how frank-zealot was measured. +per-request numbers in a `timings` object (prefill/decode rates, `draft_n`, +`draft_n_accepted`) — but on the PrismML fork build 10777 that object is only +reaching you on **`/v1/chat/completions`**: a greedy `ignore_eos` run on the raw +`/completion` path there answers **500 "The model produced output that does not +match the expected Content-only format"**, so measure through the chat path +(measured 2026-10-18 on the exact-KV reslice). halogen has NO `/completion` path at all — frank-dragoon's rates come from wall-clock around `/v1/chat/completions` plus the boot ledger. Rolling your own bench requires asking first. diff --git a/serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai9-mtp.md b/serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai9-mtp.md index cfc34dd..7706c68 100644 --- a/serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai9-mtp.md +++ b/serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai9-mtp.md @@ -187,14 +187,28 @@ No clamp line, no `cudaMalloc` error. The engine's own projection (21,439-class numbers from the 27B era) excludes the projector — **read settled `nvidia-smi`, never the projection**. -**The exact-KV arm's projection, recorded before its own boot.** Every term of -that ledger survives the reslice except the KV pool (13,056 → 16,384 MiB) and -one recurrent-state cell (−50.25 MiB), so the pre-flight prediction is -`17,922 − 13,056 + 16,384 − 50 ≈ 21,200 MiB used / ~2,920 free`, with ~2 GiB -still standing through the mmproj worst case. **Replace this paragraph with the -MEASURED ledger and settled `nvidia-smi` from the first f16 boot** — a -projection is a pre-flight check, not a record, and the whole reason the shape -changed dtype is that projections drift from ledgers. +**The exact-KV boot, measured** (2026-10-18 evening, PR #21, under the unit, +verbosity 4 — the live shape): + +``` +CUDA0 model buffer size = 2036.71 MiB # + 257.66 MiB CPU_Mapped (mmap) +CUDA0 KV buffer size = 16384.00 MiB # f16 K + f16 V, 524,288 positions +CUDA0 RS buffer size = 100.50 MiB # 2 cells, 32 layers, 2 seqs +CUDA0 compute buffer = 1408.06 MiB # + 1088.07 MiB CUDA_Host +mmproj worst-case (est.) = 868.32 MiB # 248.10 CUDA + 24.93 CPU, AFTER fit +nvidia-smi settled: 21,176 MiB used / 2,949 MiB free of 24,123 +n_slots = 2, n_ctx_slot = 262144, kv_unified = false +``` + +The pre-flight projection was right to the MiB, because the two terms it moved +are exact arithmetic: **16,384.00 MiB of KV is precisely 32,768 B/token × +524,288 positions** — rate derivation and ledger agree with no slop, which is +what buries the old 26,112 MiB figure for good — and the recurrent state is one +50.25 MiB cell lighter (100.50 exactly). Settled card state landed at **21,176 +used / 2,949 free**, within 24 MiB of the 21,200 / 2,920 projection, +`DropInPaths` empty, no clamp line, no `cudaMalloc` error. It is a desktop +margin, though: the mmproj's 868 MiB worst case is still charged AFTER the +engine's own fit check, so ~2 GiB is what an image storm has to live inside. ## Measured — the q8_0 3-slot boot (`/completion` `timings`, greedy, `ignore_eos`) @@ -211,19 +225,49 @@ changed dtype is that projections drift from ledgers. stops on its own: a 24-token countdown measured 72 t/s across three streams while the same shape with `ignore_eos` gave 300. -**These are the numbers the exact-KV arm is judged against.** Exact KV makes -attention read 1.882× the bytes per generated token, so a decode drop is -EXPECTED and has to be measured, not assumed; prefill should hold. Re-run the -same instrument on the first f16 boot — greedy, `ignore_eos`, 512 tokens, the -second request or later, and the two-stream aggregate in place of the -three-stream one — and record it here. +## Measured — the exact-KV boot (floors, not a clean A/B) + +Instrument: `/v1/chat/completions` `timings` (this fork refuses the raw +`/completion` path for greedy runs — see *Contract traps*), greedy, +`ignore_eos`, 512 tokens, first request discarded. + +|Arm|q8_0 3-slot record|f16 2-slot, live co-tenancy| +|---|---|---| +|single-stream decode|168.7|**134.1 – 141.5** over five back-to-back runs; 145.8 and 163.2 in a lighter minute| +|concurrent aggregate|≈300 across 3 (102.8 each)|**256.6 / 262.4 / 265.1** across 2 (≈128–133 each)| +|prefill at ~106k|2,877 at 106,418|**3,159.5 at 106,282**| +|decode at 106k depth|never measured|**71.5**| + +**These are floors, not the shape's rates.** Someone else was driving the +endpoint through the whole window — 56 new prompts on the other slot in three +minutes, `nvidia-smi` at 96 % GPU-util — which is the exact condition the +Benchmarking Policy forbids, and a contended deep arm even came back truncated +(a 512 KB body accepted as `prompt_n = 4`). What the floors already support: +**per-conversation concurrency improved** (two streams hold ~130 t/s each where +three held 102.8), endpoint aggregate sits ~12 % under the 3-slot record, +prefill beat the record (+9.8 %) while contended, and single-stream decode is +between parity and −20 % against it. The 1.882× KV byte bill is real; its true +size on this card is still unmeasured, and quoting 134 t/s or 163 t/s as "the" +number would be a guess in either direction. Re-run the same arms with the box +quiet before the figure is carried anywhere else. ## Contract traps - **`--metrics` is mandatory.** Without it llama.cpp answers **501** on `/metrics`, so the Prometheus scrape reads DOWN while `/health` answers 200 — that is how this endpoint sat red across four payload generations + (`up{job="frank-zealot"}` oldest retained sample, 2026-10-11, is already 0). +- **Bare `/completion` is a dead instrument on this fork build.** A greedy + `ignore_eos` run answers **500 — "The model produced output that does not + match the expected Content-only format"**, because the fork guards the raw + path the moment reasoning markup leaks (older builds only logged `failed to + parse reasoning content` and answered anyway). The honest per-request + instrument on THIS build is `/v1/chat/completions`, whose non-stream response + carries the same `timings` object (`prompt_n`, `prompt_per_second`, + `predicted_n`, `predicted_per_second`). `AGENTS.md` still names the + `/completion` `timings` object as this endpoint's instrument — on build 10777 + read it off the chat path. - **Q8_0 mmproj, not BF16.** The publisher's card: the BF16 tower "can crash llama-server when several image requests run at once; Q8_0 is not affected" — and `--parallel 2` is still precisely that shape. diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 5923d1d..ac39493 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -1,5 +1,32 @@ # Bundle Change Log +## 2026-10-18 +- **frank-zealot resliced to two windows of exact KV (Frank's call, PR #21).** + `--ctx-size 786432 --parallel 3` at q8_0 K+V → **`524288 / 2` at f16 K+V**, so + both remaining conversations keep the native 262,144 window and the + compaction hub reads exact K and V. Halving the pool is what funds it: the + f16 pair is 32,768 B/token over the 8 full-attention layers = 16,384.00 MiB, + while the old 786,432-position pool would have needed ~24,576 MiB of KV alone + on a card that leaves 24,576 addressable. Measured boot: KV **16,384.00**, + RS **100.50** (2 cells), compute 1,408.06 + 1,088.07, settled `nvidia-smi` + **21,176 used / 2,949 free**, `n_ctx_slot = 262144`, `DropInPaths` empty. The + doc's old "f16 needs 26,112 MiB and cannot boot" figure was the 27B's + 34,816 B/token applied to the 9B — corrected in the runbook. +- **Rates are floors, and that is recorded as such.** The endpoint had another + consumer for the whole measurement window (56 prompts on the other slot in + three minutes, 96 % GPU-util), so single-stream decode came back 134.1–141.5 + (145.8/163.2 in a lighter minute) against the 3-slot record of 168.7, + two-stream aggregate 256.6–265.1 against ≈300 across three, and prefill + **3,159.5 at 106,282** beat the record's 2,877 while contended. Per-stream + concurrency improved (≈130 each vs 102.8); a clean re-measure needs the box + quiet. +- **Instrument correction (fleet law).** On the PrismML fork build 10777 the raw + `/completion` path answers **500 "The model produced output that does not + match the expected Content-only format"** for a greedy `ignore_eos` run, so + the honest per-request `timings` for this endpoint must be read off + `/v1/chat/completions`. `AGENTS.md` said `/completion`; it now says the chat + path. + ## 2026-10-13 - **frank-archon outage: a kernel knob, not the model.** The 2026-10-12 23:17 power failure on dankserver reset `/sys/module/ttm/parameters/pages_limit`