Shifts frank-zealot (llama.cpp PrismML fork build 10777, RTX 3090, :7771) to two parallel conversations over an unquantized KV cache, per Frank's instruction: "shift frank-zealot to 2 parallels of unquantized kv".
Shape delta
before
after
--ctx-size (pool)
786,432
524,288
--parallel
3
2
per-slot window
262,144
262,144 (unchanged)
KV dtype
q8_0 K + q8_0 V
f16 K + f16 V
KV pool
13,056.00 MiB
16,384 MiB
concurrency
3 conversations
2
Everything else holds: batch 4096 / ubatch 2048, --flash-attn on, --jinja, --metrics, --presence-penalty 1.0, SPEC_TYPE=none, the Q8_0 mmproj, port 7771. models.ymlcontextWindow: 262144 is untouched — llama.cpp divides --ctx-size across slots, and the per-slot contract is what consumers quote.
Why the pool had to halve
Exact KV bills 32,768 B/token at this geometry (8 full-attention layers × 4 KV heads × 256 dim = 8,192 elements per cache, 2 B, K and V). The old 786,432 pool therefore asks 24,576 MiB against the 24,123 MiB the driver leaves addressable — impossible, not tight. Halving the pool is what funds exact KV; the window could not stretch to compensate because 262,144 is the native wall (no YaRN in this path).
Projection from the live 13:19 boot ledger (only two terms move: KV pool +3,328 MiB, one recurrent-state cell −50.25): ~21,200 MiB used / ~2,900 free, ~2 GiB still standing through the mmproj worst case. This is a projection, not a measurement — the measured ledger and the decode/prefill re-measure land in a follow-up commit after the first f16 boot.
Two repo-wide errors corrected on the way
"f16 KV would bill 26,112 MiB" — that figure doubled the q8_0 ledger number including its 1/32 scale overhead. q8_0 is 1.0625 B/element, f16 exactly 2.0 ⇒ the ratio is 1.882×, not 2×. The wall stands (three f16 windows still cannot boot this card); the number was wrong, and the honest rate is what made 524,288 identifiable as the largest exact-KV pool that fits.
"this build has no flag to quantize the draft cache" — false. llama-server --help on the deployed binary shows --spec-draft-type-k / --spec-draft-type-v (both default f16). So the exact-KV/MTP exclusion is a measurement gap, not a law: the 1,088 MiB quantized-draft arm is legal on b10777 and has never been booted beside exact KV. Recorded as unmeasured, not as impossible.
Verification
tests/frank_zealot_unit.bats: 18/18 green, including a new VRAM fit gate that computes dtype rate × pool + measured non-KV ledger terms against 24,123 MiB with ≥1 GiB of slack, and proves the gate refuses exact KV at the old 786,432 pool — so passing the shipped arm means something. No health gate can see a boot that dies at cudaMalloc; this seam can.
./tests/run.sh: full suite green (447 tests).
--cache-type-k/-v accepted values confirmed against the deployed binary (f16 is the engine default; q8_0 stays legal only with --flash-attn on, which remains pinned so the documented fallback rung stays real).
Cost
One concurrent conversation (3 → 2), and decode reads 1.882× the KV bytes per generated token — a decode drop is expected and must be measured, not assumed. Prefill should hold: this endpoint is the fleet's compaction hub, which is weight-bandwidth bound and does not re-read the cache per generated token. With --parallel 2 a compaction request still runs beside one live generation; with two already busy it waits.
Deploy
Merge here, then ./scripts/deploy.sh dankclaw --service frank-zealot (service-scoped — the fleet-wide form is a drain). The unit Description moved with the shape so systemctl status does not lie.
Shifts `frank-zealot` (llama.cpp PrismML fork build 10777, RTX 3090, :7771) to **two parallel conversations over an unquantized KV cache**, per Frank's instruction: *"shift frank-zealot to 2 parallels of unquantized kv"*.
## Shape delta
| | before | after |
|---|---|---|
|`--ctx-size` (pool)|786,432|**524,288**|
|`--parallel`|3|**2**|
|per-slot window|262,144|**262,144** (unchanged)|
|KV dtype|q8_0 K + q8_0 V|**f16 K + f16 V**|
|KV pool|13,056.00 MiB|**16,384 MiB**|
|concurrency|3 conversations|**2**|
Everything else holds: batch 4096 / ubatch 2048, `--flash-attn on`, `--jinja`, `--metrics`, `--presence-penalty 1.0`, `SPEC_TYPE=none`, the Q8_0 mmproj, port 7771. `models.yml` `contextWindow: 262144` is untouched — llama.cpp divides `--ctx-size` across slots, and the per-slot contract is what consumers quote.
## Why the pool had to halve
Exact KV bills **32,768 B/token** at this geometry (8 full-attention layers × 4 KV heads × 256 dim = 8,192 elements per cache, 2 B, K and V). The old 786,432 pool therefore asks **24,576 MiB** against the **24,123 MiB** the driver leaves addressable — impossible, not tight. Halving the pool is what funds exact KV; the window could not stretch to compensate because 262,144 is the native wall (no YaRN in this path).
Projection from the live 13:19 boot ledger (only two terms move: KV pool +3,328 MiB, one recurrent-state cell −50.25): **~21,200 MiB used / ~2,900 free**, ~2 GiB still standing through the mmproj worst case. This is a **projection, not a measurement** — the measured ledger and the decode/prefill re-measure land in a follow-up commit after the first f16 boot.
## Two repo-wide errors corrected on the way
- **"f16 KV would bill 26,112 MiB"** — that figure doubled the q8_0 ledger number *including* its 1/32 scale overhead. q8_0 is 1.0625 B/element, f16 exactly 2.0 ⇒ the ratio is **1.882×**, not 2×. The wall stands (three f16 windows still cannot boot this card); the number was wrong, and the honest rate is what made 524,288 identifiable as the largest exact-KV pool that fits.
- **"this build has no flag to quantize the draft cache"** — false. `llama-server --help` on the deployed binary shows `--spec-draft-type-k` / `--spec-draft-type-v` (both default f16). So the exact-KV/MTP exclusion is a **measurement gap, not a law**: the 1,088 MiB quantized-draft arm is legal on b10777 and has never been booted beside exact KV. Recorded as unmeasured, not as impossible.
## Verification
- `tests/frank_zealot_unit.bats`: **18/18 green**, including a new VRAM fit gate that computes `dtype rate × pool + measured non-KV ledger terms` against 24,123 MiB with ≥1 GiB of slack, **and proves the gate refuses** exact KV at the old 786,432 pool — so passing the shipped arm means something. No health gate can see a boot that dies at `cudaMalloc`; this seam can.
- `./tests/run.sh`: full suite green (447 tests).
- `--cache-type-k/-v` accepted values confirmed against the deployed binary (`f16` is the engine default; `q8_0` stays legal only with `--flash-attn on`, which remains pinned so the documented fallback rung stays real).
## Cost
One concurrent conversation (3 → 2), and decode reads 1.882× the KV bytes per generated token — a decode drop is expected and must be measured, not assumed. Prefill should hold: this endpoint is the fleet's compaction hub, which is weight-bandwidth bound and does not re-read the cache per generated token. With `--parallel 2` a compaction request still runs beside one live generation; with two already busy it waits.
## Deploy
Merge here, then `./scripts/deploy.sh dankclaw --service frank-zealot` (service-scoped — the fleet-wide form is a drain). The unit Description moved with the shape so `systemctl status` does not lie.
---
cwd: /home/blank/.omp/wt/wt-20261018-204501-6d1fc1f · session: 01a15243-3541-7380-9dae-4985fdc86843
Frank: "shift frank-zealot to 2 parallels of unquantized kv".
Shape: CTX_SIZE 786432 -> 524288, PARALLEL 3 -> 2, CACHE_TYPE_K/V
q8_0 -> f16. The per-slot window is UNCHANGED at 262,144 (the native
trained window, and models.yml contextWindow), so what the exact cache
buys is fidelity and what it pays is one concurrent conversation.
The POOL is the term that had to move: exact KV bills 32,768 B/token at
this geometry (8 full-attention layers x 4 KV heads x 256 dim, 2 B,
K and V), so the old 786,432 pool asks 24,576 MiB against the 24,123 MiB
the driver leaves addressable. 524,288 positions = 16,384 MiB, projected
~21,200 used / ~2,900 free from the 13:19 boot ledger (only the KV pool
and one recurrent-state cell move).
Also corrects a repo-wide arithmetic error: "f16 would bill 26,112 MiB"
doubled the q8_0 ledger figure INCLUDING its 1/32 scale overhead. q8_0 is
1.0625 B/element, f16 exactly 2.0, so the true ratio is 1.882x. The wall
stands; the number was wrong.
And it corrects a false claim about the drafter: this build DOES expose
--spec-draft-type-k / --spec-draft-type-v (verified against
llama-server --help), so the draft cache is quantizable -- the
exact-KV-plus-MTP exclusion is a measurement gap, not a law, and the
1,088 MiB quantized-draft arm is recorded as unmeasured.
Seam: tests/frank_zealot_unit.bats re-pins the shape contract (2 slots,
524,288 pool, 262,144 per slot) and adds a VRAM fit gate that computes
dtype x pool + the measured non-KV ledger terms against 24,123 MiB with
>=1 GiB of slack -- and proves the gate REFUSES exact KV at the old
pool, so passing means something. 18/18 green.
AGENTS.md: topology row, the frank-zealot runbook section (shape, KV
accounting, MTP verdict, compaction-hub concurrency), and two Key
Lessons bullets -- the draft-context bill restated with the real knob
names, plus a new lesson that a quant's scale overhead belongs in its
byte rate (the 6.25 % invented by doubling 13,056.00 MiB).
LLM/frank-zealot-llama-prism-bonsai9-mtp.md: retitled to the live shape;
the 3-slot q8_0 boot ledger and its measurements are relabelled as the
arm this replaced (byte-identical numbers, honest attribution), the
corrected f16 rate derivation is added, the MTP census is labelled
q8_0-era, the fallback rungs are re-derived for the 524,288 pool, and the
boot-ledger section carries an explicit PROJECTION that must be replaced
with the measured ledger from the first f16 boot.
Pointers re-pointed (live-shape sentences only; every historical ledger
left byte-identical): LLM/index.md, vault index.md, Dankclaw/OpenWebUI.md,
and the retired-runbook "today" pointers in mimo-q5km, mimo-q6k,
qwen35-9b-q6k, swift15, sglang-mimo-v26. log.md gains the dated entry.
dank
merged commit 5550d32a2e into main2026-10-18 21:06:08 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Shifts
frank-zealot(llama.cpp PrismML fork build 10777, RTX 3090, :7771) to two parallel conversations over an unquantized KV cache, per Frank's instruction: "shift frank-zealot to 2 parallels of unquantized kv".Shape delta
--ctx-size(pool)--parallelEverything else holds: batch 4096 / ubatch 2048,
--flash-attn on,--jinja,--metrics,--presence-penalty 1.0,SPEC_TYPE=none, the Q8_0 mmproj, port 7771.models.ymlcontextWindow: 262144is untouched — llama.cpp divides--ctx-sizeacross slots, and the per-slot contract is what consumers quote.Why the pool had to halve
Exact KV bills 32,768 B/token at this geometry (8 full-attention layers × 4 KV heads × 256 dim = 8,192 elements per cache, 2 B, K and V). The old 786,432 pool therefore asks 24,576 MiB against the 24,123 MiB the driver leaves addressable — impossible, not tight. Halving the pool is what funds exact KV; the window could not stretch to compensate because 262,144 is the native wall (no YaRN in this path).
Projection from the live 13:19 boot ledger (only two terms move: KV pool +3,328 MiB, one recurrent-state cell −50.25): ~21,200 MiB used / ~2,900 free, ~2 GiB still standing through the mmproj worst case. This is a projection, not a measurement — the measured ledger and the decode/prefill re-measure land in a follow-up commit after the first f16 boot.
Two repo-wide errors corrected on the way
llama-server --helpon the deployed binary shows--spec-draft-type-k/--spec-draft-type-v(both default f16). So the exact-KV/MTP exclusion is a measurement gap, not a law: the 1,088 MiB quantized-draft arm is legal on b10777 and has never been booted beside exact KV. Recorded as unmeasured, not as impossible.Verification
tests/frank_zealot_unit.bats: 18/18 green, including a new VRAM fit gate that computesdtype rate × pool + measured non-KV ledger termsagainst 24,123 MiB with ≥1 GiB of slack, and proves the gate refuses exact KV at the old 786,432 pool — so passing the shipped arm means something. No health gate can see a boot that dies atcudaMalloc; this seam can../tests/run.sh: full suite green (447 tests).--cache-type-k/-vaccepted values confirmed against the deployed binary (f16is the engine default;q8_0stays legal only with--flash-attn on, which remains pinned so the documented fallback rung stays real).Cost
One concurrent conversation (3 → 2), and decode reads 1.882× the KV bytes per generated token — a decode drop is expected and must be measured, not assumed. Prefill should hold: this endpoint is the fleet's compaction hub, which is weight-bandwidth bound and does not re-read the cache per generated token. With
--parallel 2a compaction request still runs beside one live generation; with two already busy it waits.Deploy
Merge here, then
./scripts/deploy.sh dankclaw --service frank-zealot(service-scoped — the fleet-wide form is a drain). The unit Description moved with the shape sosystemctl statusdoes not lie.cwd: /home/blank/.omp/wt/wt-20261018-204501-6d1fc1f · session: 01a15243-3541-7380-9dae-4985fdc86843