docs(frank-zealot): re-key the exact-KV shape across the vault
AGENTS.md: topology row, the frank-zealot runbook section (shape, KV accounting, MTP verdict, compaction-hub concurrency), and two Key Lessons bullets -- the draft-context bill restated with the real knob names, plus a new lesson that a quant's scale overhead belongs in its byte rate (the 6.25 % invented by doubling 13,056.00 MiB). LLM/frank-zealot-llama-prism-bonsai9-mtp.md: retitled to the live shape; the 3-slot q8_0 boot ledger and its measurements are relabelled as the arm this replaced (byte-identical numbers, honest attribution), the corrected f16 rate derivation is added, the MTP census is labelled q8_0-era, the fallback rungs are re-derived for the 524,288 pool, and the boot-ledger section carries an explicit PROJECTION that must be replaced with the measured ledger from the first f16 boot. Pointers re-pointed (live-shape sentences only; every historical ledger left byte-identical): LLM/index.md, vault index.md, Dankclaw/OpenWebUI.md, and the retired-runbook "today" pointers in mimo-q5km, mimo-q6k, qwen35-9b-q6k, swift15, sglang-mimo-v26. log.md gains the dated entry.
This commit is contained in:
1 parent
68afc6dbaf
commit
71bc1735d3
11 files changed
+300
-137
No files matched your search
@@ -2787,3 +2787,37 @@ every restart since the cutover.
|
||||
to the player, temp dir cleaned; the audible pass through the default
|
||||
`pw-play` detection confirmed by Frank. Sessions open before the merge need
|
||||
a restart to pick the command up.
|
||||
|
||||
## 2026-10-18 — frank-zealot: 2 × 262,144 over unquantized f16 KV
|
||||
|
||||
- Frank's instruction: **"shift frank-zealot to 2 parallels of unquantized
|
||||
kv"**. The env delta is four lines: `CTX_SIZE=524288`, `PARALLEL=2`,
|
||||
`CACHE_TYPE_K=f16`, `CACHE_TYPE_V=f16`. Everything else holds — batch
|
||||
**4096** / ubatch **2048**, `--flash-attn on`, `--jinja`, `--metrics`,
|
||||
`--presence-penalty 1.0`, `SPEC_TYPE=none`, the Q8_0 mmproj, port 7771, and
|
||||
the per-slot window **262,144** — `models.yml` `contextWindow` untouched.
|
||||
- **Why the pool had to halve.** Exact KV bills **32,768 B/token** (8
|
||||
full-attention layers × 4 KV heads × 256 dim × 2 B × 2 caches), so the old
|
||||
786,432 pool asks **24,576 MiB** against the **24,123 MiB** the driver leaves
|
||||
addressable — impossible, not tight. 524,288 positions ⇒ **16,384 MiB** and
|
||||
exactly two windows of 262,144.
|
||||
- Arithmetic correction to the repo's long-standing figure: "the f16 pair would
|
||||
need **26,112 MiB**" doubled the q8_0 number *including* its 1/32 scale
|
||||
overhead — q8_0 is 17,408 B/token (1.0625 B/element), so f16/q8_0 is
|
||||
**1.882×**, not 2×, and exact f16 over 786,432 is **24,576 MiB**. The
|
||||
conclusion stands (three f16 windows cannot boot this card); the number was
|
||||
wrong.
|
||||
- The cost: **concurrency 3 → 2 conversations**, and MTP is now **doubly** out.
|
||||
The 13:16 measurement still says off, and exact KV adds a second reason — the
|
||||
draft context bills its own f16 KV over the WHOLE pool (**4,096 B/position**
|
||||
× 524,288 = **2,048 MiB**) on a trunk projecting ~21,200 used.
|
||||
- **Projection, explicitly not a measurement:** ~**21,200 MiB** used /
|
||||
~**2,920 free**, derived from the 13:19 ledger's 17,922 used because only two
|
||||
terms move — the KV pool (**+3,328**) and one recurrent-state cell
|
||||
(**−50.25**, 3 cells → 2). The measured boot ledger and the decode/prefill
|
||||
re-measure land in the follow-up commit, after the first f16 boot.
|
||||
- New gate in `tests/frank_zealot_unit.bats`: dtype × pool plus the measured
|
||||
non-KV terms must fit **24,123 MiB** with **≥ 1 GiB** of slack — and the same
|
||||
gate proves it refuses the impossible arm (**f16 at 786,432**), not merely
|
||||
passes the shipped one. Runbook re-keyed to the new shape:
|
||||
`LLM/frank-zealot-llama-prism-bonsai9-mtp.md`.
|
||||
Reference in new issue
Block a user