docs(frank-zealot): re-key the exact-KV shape across the vault
AGENTS.md: topology row, the frank-zealot runbook section (shape, KV accounting, MTP verdict, compaction-hub concurrency), and two Key Lessons bullets -- the draft-context bill restated with the real knob names, plus a new lesson that a quant's scale overhead belongs in its byte rate (the 6.25 % invented by doubling 13,056.00 MiB). LLM/frank-zealot-llama-prism-bonsai9-mtp.md: retitled to the live shape; the 3-slot q8_0 boot ledger and its measurements are relabelled as the arm this replaced (byte-identical numbers, honest attribution), the corrected f16 rate derivation is added, the MTP census is labelled q8_0-era, the fallback rungs are re-derived for the 524,288 pool, and the boot-ledger section carries an explicit PROJECTION that must be replaced with the measured ledger from the first f16 boot. Pointers re-pointed (live-shape sentences only; every historical ledger left byte-identical): LLM/index.md, vault index.md, Dankclaw/OpenWebUI.md, and the retired-runbook "today" pointers in mimo-q5km, mimo-q6k, qwen35-9b-q6k, swift15, sglang-mimo-v26. log.md gains the dated entry.
This commit is contained in:
@@ -26,9 +26,10 @@ MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots over ONE
|
||||
200704, vision ON)** and
|
||||
**llama.cpp on the PrismML fork (frank-zealot, port 7771: prism-ml
|
||||
Ternary-Bonsai-2-9B at **PQ2_0** — 9B qwen35 hybrid (8 full-attention + 24
|
||||
Gated DeltaNet layers + the blk.32 MTP block), THREE 262,144-token
|
||||
conversations, MTP present in the file but OFF by measurement,
|
||||
q8_0 K + q8_0 V (f16 KV does not fit the RTX 3090 at this window), vision via
|
||||
Gated DeltaNet layers + the blk.32 MTP block), TWO slots over ONE
|
||||
524,288-position pool (2 × the 262,144 window), MTP present in the file but
|
||||
OFF by measurement, **exact f16 K + f16 V** — 32,768 B/token = **16,384 MiB**
|
||||
of KV (three 262,144 windows at exact KV do not fit the RTX 3090), vision via
|
||||
the publisher's own Q8_0 mmproj, on the RTX 3090 — and the custom
|
||||
tensor type 141 + `prism.hadamard.*` metadata mean no other loader can open the
|
||||
file)**.
|
||||
@@ -63,7 +64,7 @@ ONE 401408-position KV pool at ctx 200704 = 2 slots x ctx, MTP always-on, image
|
||||
G[llama.cpp PrismML fork CUDA build
|
||||
frank-zealot, port 7771]
|
||||
H[Ternary-Bonsai-2-9B PQ2_0 GGUF + Q8_0 mmproj
|
||||
3 x 262144 ctx, q8_0 K + q8_0 V, MTP off, mmproj vision]
|
||||
2 slots over ONE 524288-position pool at ctx 262144, exact f16 K + f16 V, MTP off, mmproj vision]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
end
|
||||
@@ -132,19 +133,26 @@ frank-zealot, port 7771]
|
||||
custom PQ2_0 tensor type + `prism.hadamard.*` transform, and since this swap
|
||||
also the minimum build that pairs MTP with a quantized V cache) serving
|
||||
prism-ml/Ternary-Bonsai-2-9B-gguf **PQ2_0 with the MTP head IN the file** on
|
||||
the RTX 3090 — **THREE 262,144-token conversations**
|
||||
(`--ctx-size 786432 --parallel 3`, every slot at the window the model was
|
||||
trained at), **q8_0 K + q8_0 V** (17 KiB/token over the 8 full layers =
|
||||
13,056 MiB ledger-exact; the f16 pair needs 26,112 MiB and dies at
|
||||
`cudaMalloc`), settled **17,876 used / 6,249 free**, vision via the
|
||||
publisher's **Q8_0** mmproj (Q8_0 is the publisher's requirement for
|
||||
concurrent image requests, and this runs three slots), **MTP OFF by
|
||||
measurement** — the draft context bills its own f16 KV over the WHOLE pool
|
||||
(3,072 MiB) and at `UBATCH_SIZE=2048` that squeeze halves decode
|
||||
(167.2 → 109.2 t/s); `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` is the
|
||||
documented 213.5 t/s arm, declined for +17 % prefill and 6.2 GiB of headroom.
|
||||
Measured: 167.2 t/s single-stream decode, 2,877 t/s prefill at 106,418,
|
||||
280.2 t/s aggregate across three streams. `--metrics` rides (`METRICS=1`):
|
||||
the RTX 3090 — **TWO 262,144-token conversations**
|
||||
(`--ctx-size 524288 --parallel 2`, every slot at the window the model was
|
||||
trained at), **unquantized f16 K + f16 V** (32,768 B/token over the 8 full
|
||||
layers = **16,384 MiB** exact; the POOL halved because that rate over the
|
||||
OLD 786,432 pool bills 24,576 MiB against the 24,123 the driver leaves
|
||||
addressable, and the old "26,112 MiB" note was wrong — it doubled the
|
||||
q8_0 figure (17,408 B/token = 13,056 MiB) *including* its 1/32 scale
|
||||
overhead, so f16/q8_0 is **1.882×**, not 2×), the q8_0-era boot settled
|
||||
**17,876 used / 6,249 free**, vision via the publisher's **Q8_0** mmproj
|
||||
(Q8_0 is the publisher's requirement for concurrent image requests, and this
|
||||
runs two slots), **MTP OFF by measurement** — the draft context bills its own
|
||||
f16 KV over the WHOLE pool (4,096 B/position × 524,288 = 2,048 MiB) and at
|
||||
`UBATCH_SIZE=2048` that squeeze halves decode (167.2 → 109.2 t/s);
|
||||
`SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` is the documented 213.5 t/s arm,
|
||||
declined for +17 % prefill and 6.2 GiB of headroom — and exact KV leaves an
|
||||
f16-draft arm no room at all (a quantized draft, `--spec-draft-type-* q8_0`,
|
||||
is the unmeasured door).
|
||||
Measured at the q8_0-era shape: 167.2 t/s single-stream decode, 2,877 t/s
|
||||
prefill at 106,418, 280.2 t/s aggregate across three streams. `--metrics`
|
||||
rides (`METRICS=1`):
|
||||
llama.cpp answers 501 on `/metrics` without it, which is why Prometheus
|
||||
`up{job="frank-zealot"}` read 0 for its whole retained history until
|
||||
2026-10-18. `contextWindow: 262144` is the **widest in the fleet**, so this
|
||||
|
||||
Reference in New Issue
Block a user