dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(frank-zealot): record the ba69ea5 boot ledger and correct the carried credits

Browse Source 
The boot of record for the new shape, from the verbosity-4 journal plus
settled nvidia-smi: target KV 14,406.00 (K f16 9,408.00 + V q8_0 4,998.00),
draft KV 2,352.00 over the same 200,704 cells x 3 seqs, recurrent state
452.25, compute 1,665.00 + 424.88 (draft 480.97 + 424.86), weights 2,283.29
+ 257.66, projector 595.29 + 248.10 + 24.93, settled 22,870 used / 1,255
free with n_slots = 3 / n_ctx_slot = 200704.

Two inherited numbers did not survive it, and both are corrected in the env,
the serve rationale and the page:

  - The ubatch 2048 -> 1024 step was quoted at "-~1,306 MiB". That was the
    786,432 / q8_0-pair arm. At THIS shape the target's compute buffers went
    2,496.13 -> 2,089.88: 406 MiB, with CUDA0 going UP 257 while CUDA_Host
    went down 663. q8_0 V (4,410 MiB, not the 3,840 previously written) is
    what funds the drafter, alone.
  - The fit gate priced 22,428 against a real 22,870: conservative by 442
    MiB, so it stays honest as written.

And one finding stated plainly: this is the first boot on this card that
misses llama.cpp's own free-memory target - the engine printed "cannot meet
free memory target of 1,867 MiB, need to reduce device memory by 102 MiB"
and settled at 1,255 MiB free, under the ~2 GiB band every previous era
held. Cheapest rungs back are CTX_SIZE=401408 or SPEC_TYPE=none, both one
line.

Probes on the same boot go in too: chat turn (448 tokens, 1,755 chars of
thinking - empty content at max_tokens 64 is the documented trap), tools
(finish_reason tool_calls, {"a":24,"b":7}), vision (512 px solid red ->
"Red", 280 prompt tokens), and the drafter's own statistics (acceptance
0.706 / 0.763, mean accepted length 2.41 / 2.53).
This commit is contained in:
Frank
2026-10-19 03:38:03 -07:00
parent ba69ea563e
commit 30e8ed55a0
3 changed files with 60 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-zealot.md
+46 -17
View File
@@ -47,11 +47,12 @@ buys both and costs the keys their precision.
**MTP is ON (Frank, 2026-10-19).** The drafter was always good — 2026-10-18 measured acceptance 0.798 / 0.617 per
position, mean accepted length 2.41, and the publisher's card says depth 2 works best — and arming it is a
budget decision, not a flag: the draft context is built against the *target's* pool, so `blk.32` bills 4,096
B/position (2,352.00 MiB at 602,112) plus ~0.45 GiB of buffers at this ubatch and its own recurrent cells.
`CACHE_TYPE_V=q8_0` (−3,840 MiB against exact V at this pool) and `UBATCH_SIZE=1024` (−~1,306 MiB of compute
buffers, and the rung every measured MTP arm here ran on) are what pay for it; `UBATCH_SIZE=2048` beside an armed
drafter is the shape that halved decode (167.2 → 109.2 t/s). `--spec-draft-type-k`/`--spec-draft-type-v` (default
f16, verified in this build's `--help`) would cut the draft's bill again and stay at their default; there is no
B/position (2,352.00 MiB at 602,112) plus 480.97 MiB of its own CUDA0 buffers, and it triples the recurrent
state (452.25 MiB for three cells against 150.75 with the drafter off). `CACHE_TYPE_V=q8_0` is what pays that:
7,680 B/token cheaper than exact V = **4,410.00 MiB** at this pool. `UBATCH_SIZE=1024` — the rung every measured
MTP arm here ran on — contributes only ~406 MiB at THIS shape, not the ~1,306 the 786,432-pool census showed,
and `UBATCH_SIZE=2048` beside an armed drafter is still the shape that halved decode (167.2 → 109.2 t/s).
`--spec-draft-type-k`/`--spec-draft-type-v` (default f16, verified in this build's `--help`) would cut the draft's bill again and stay at their default; there is no
`--spec-draft-model` — the draft rides the main model's weights, so a draft file would load the trunk twice.
`SPEC_TYPE=none` is the rollback and strips the whole set.
@@ -62,23 +63,47 @@ longer prompt with a 400, never a truncation — plus `maxTokens: 16384`, `input
only `xhigh|medium|low` and raises on anything else), `extraBody` 1.0 / 0.95 / 20 / `min_p 0`, and the
`compactionModel` of [[frank-archon]], [[frank-dragoon]] **and** frank-zealot all pointing here.
**This shape ships unbenchmarked, by law.** Frank ordered the reslice, not a
number, so the evidence a swap needs is the seam's fit gate (it prices cache +
draft cache + recurrent states against the 24,123 MiB the driver leaves
addressable, and refuses the two boots this shape can walk into: exact V with the
drafter armed, and the retired dtype pair at the retired pool), the deployer's
health gate, and the first verbosity-4 ledger of the new dtype pair — which is
recorded on this page because projections drift, and nowhere else. The previous
era's exact-KV boot and its rate floors live in that era's row below; those floors
were measured while another consumer drove the other slot (56 prompts in three
minutes, 96 % GPU-util), the exact condition a bench window exists to exclude.
**No rates, by law.** Frank ordered the reslice, not a number: the evidence a swap needs is
the seam's fit gate, the deployer's health gate, and the boot ledger below. The previous era's
exact-KV boot and its rate floors live in that era's row; those floors were measured while
another consumer drove the other slot (56 prompts in three minutes, 96 % GPU-util), the exact
condition a bench window exists to exclude.
**The 2026-10-19 boot, measured** — verbosity-4 ledger of `ba69ea5`, started 03:21:56 PDT.
Recorded here because projections drift, and nowhere else:
| Term | Ledger |
|---|---|
| Target KV over the 602,112 pool | **14,406.00 MiB** — K (f16) 9,408.00 + V (q8_0) 4,998.00, `200704 cells, 8 layers, 3/3 seqs` |
| Draft KV (`blk.32`, its own f16 context) | **2,352.00 MiB** — 1,176.00 + 1,176.00, `200704 cells, 1 layers, 3/3 seqs`: the whole pool, again |
| Recurrent state | **452.25 MiB** — `3 cells, 32 layers, 3 seqs, 2 rs_seq`, R (f32) 20.25 + S (f32) 432.00 |
| Compute buffers | target 1,665.00 CUDA0 + 424.88 CUDA_Host; draft context 480.97 + 424.86 |
| Weights | 2,283.29 CUDA0 + 257.66 CPU-mapped |
| Projector | 595.29 model + 248.10 compute + 24.93 CPU (its 868.32 worst case is still charged AFTER the engine's fit check) |
| Engine's own verdict | projected 21,639 vs 23,404 free, then **"cannot meet free memory target of 1,867 MiB, need to reduce device memory by 102 MiB"** |
| Settled | **22,870 used / 1,255 free**, `n_slots = 3`, `n_ctx_slot = 200704`, `kv_unified = false` |
This is the first boot on this card that does **not** meet llama.cpp's own free-memory target,
and 1,255 MiB settled sits under the ~2 GiB band every previous era held. It boots, `/health`
answers, and the seam's gate still passes — it priced 22,428 MiB, conservative by 442 against
this ledger. The cheapest rungs back into the band are `CTX_SIZE=401408` (two conversations) or
`SPEC_TYPE=none`; both are one line and neither changes the payload.
**The drafter is engaged, measured on the same boot** (three ordinary requests, not a bench):
acceptance 0.706 and 0.763 accepted-per-generated, mean accepted length 2.41 / 2.53,
per-position rates (0.882, 0.765) and (0.871, 0.699) — depth 2 earning its keep at 0.3–1.5 MiB
of draft state per saved prompt. The three probes that prove a swap: chat → `"zealot alive"`
with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY content with
`finish_reason: length` — the thinking-budget trap, not a fault); tools →
`finish_reason: tool_calls` with `{"a":24,"b":7}`; vision → a 512 px solid-red PNG answers
"Red" at `prompt_tokens` 280.
**Fallback rungs**, cheapest first — one line of `frank-zealot.env` each, and each must keep the seam's fit gate green:
| Knob | Effect from the shipped shape |
|---|---|
| `SPEC_TYPE=none` | the drafter off: hands back 2,352 MiB of draft cache plus its buffers and recurrent cells, at the cost of the ~+28 % decode the 2026-10-18 census measured whenever the card has room |
| `UBATCH_SIZE=2048` | +~1,306 MiB of compute buffers and +4.6 % prefill — and the rung that HALVED decode beside an armed drafter (167.2 → 109.2 t/s) |
| `SPEC_TYPE=none` | the drafter off: hands back its 2,352.00 MiB of cache, 480.97 MiB of CUDA0 buffers and the 301.50 MiB of rollback planes it adds to the recurrent state (~3.1 GiB), at the cost of the ~+28 % decode the 2026-10-18 census measured whenever the card has room |
| `UBATCH_SIZE=2048` | +4.6 % prefill and only ~406 MiB of compute buffers at THIS shape (the ~1,306 MiB the 786,432-pool census showed does not transfer) — and the rung that HALVED decode beside an armed drafter (167.2 → 109.2 t/s) |
| `CTX_SIZE=401408` | two 200,704 conversations: frees 4,802 MiB of cache — `contextWindow` MUST follow the per-slot number |
| `CACHE_TYPE_V=f16` | exact V: +4,410 MiB at this pool with the drafter armed, which is a `cudaMalloc` death the seam refuses — legal only once the drafter or a slot is gone |
| `CACHE_TYPE_K=q8_0` | the fully quantized pair (17,408 B/token): frees 4,410 MiB and moves the quantization error into the keys |
@@ -127,6 +152,10 @@ minutes, 96 % GPU-util), the exact condition a bench window exists to exclude.
`tests/frank_zealot_unit.bats`'s fit gate can see a boot that would die at `cudaMalloc`, so every retune keeps it
green — and the first verbosity-4 ledger plus settled `nvidia-smi` of a new dtype belongs on this page, because
projections drift.
The same trap one knob down: the ubatch 2048 → 1024 step handed back **~1,306 MiB** at the 786,432 / q8_0-pair
arm and **406 MiB** at the 602,112 / f16-K + q8_0-V boot (target compute 2,496.13 → 2,089.88, with CUDA0 going
UP 257 while CUDA_Host went down 663). Changing the dtype or the pool moves the compute buffers too, so a
credit measured at one shape is not a budget line at the next.
The two boots that priced non-KV: 2 × 262,144 **without** the projector — 20,190 used, KV 12,800 ⇒ non-KV
**7,390 MiB**; 3 × 196,608 **with** it — 22,972 used, KV 14,113 ⇒ non-KV **8,859 MiB**. The difference prices the
vision tower at **~1.4 GiB for a 918 MB file**, because the encoder's activation and compute buffers ride with it.