diff --git a/scripts/frank-zealot.env b/scripts/frank-zealot.env index be1037f..97d976b 100644 --- a/scripts/frank-zealot.env +++ b/scripts/frank-zealot.env @@ -32,9 +32,12 @@ PARALLEL=3 # on [[frank-zealot]]); 1024×6 keeps it a clean multiple of the ubatch. BATCH_SIZE=6144 # Ubatch is a prefill knob (decode never moves with it) and it sets the -# compute-buffer size — which is what decides whether a drafter fits: 1024 -# hands back ~1,306 MiB beside the draft context, and it is the rung the -# measured MTP arm ran (213.5 t/s at 1024 against 109.2 at 2048). +# compute-buffer size, which decides whether a drafter fits. 1024 is the rung +# every measured MTP arm here ran on (213.5 t/s against 109.2 at 2048). Its +# VRAM credit is SHAPE-specific: the 786,432-pool arm census showed ~1,306 MiB +# back, this boot measured 2,089.88 MiB of target compute against 2,496.13 at +# ubatch 2048 — 406 MiB back, with the split moving (CUDA0 +257, CUDA_Host +# −663). The drafter is funded by CACHE_TYPE_V below, not by this knob. UBATCH_SIZE=1024 # K exact, V in the 8-bit class: 16,384 + 8,704 = 25,088 B/token (8,192 # elements per token per cache × 2 B for f16, × 1.0625 for q8_0 — there is no diff --git a/scripts/serve-frank-zealot.sh b/scripts/serve-frank-zealot.sh index 29e3348..ee89876 100755 --- a/scripts/serve-frank-zealot.sh +++ b/scripts/serve-frank-zealot.sh @@ -37,10 +37,12 @@ # accepts well (0.798/0.617 per position, mean length 2.41, measured 2026-10-18); # what made it lose that evening was the bill, not the drafts — the draft context # is built against the target's WHOLE pool, so blk.32 adds 4,096 B/position -# (2,352.00 MiB at 602,112) plus ~0.45 GiB of buffers at this ubatch, and the -# recurrent state triples. Dropping V to q8_0 (−3,840 MiB at this pool) and the -# ubatch to 1024 (−~1,306 MiB of compute buffers) is how those two bills got -# paid; the arm census and the rates are on [[frank-zealot]]. SPEC_TYPE=none is +# (2,352.00 MiB at 602,112) plus 480.97 MiB of its own compute buffers on the +# card, and it triples the recurrent state per slot (452.25 MiB for three cells +# against 50.25 a cell with the drafter off). CACHE_TYPE_V=q8_0 is what pays +# that: 7,680 B/token cheaper than exact V, 4,410 MiB at this pool. The ubatch +# drop to 1024 contributes ~406 MiB here, NOT the ~1,306 the 786,432-pool census +# showed — that figure does not transfer between shapes. SPEC_TYPE=none is # the rollback and strips the whole flag set. # # Sampling: the checkpoint carries the publisher's thinking recipe as GGUF @@ -81,8 +83,8 @@ fi # Compute-buffer geometry. Ubatch is a prefill knob (decode never moves). # The logical batch is VRAM- and throughput-neutral at a fixed ubatch, so 6144 # (1024×6) is free; the ubatch is what sets the compute-buffer size, and 1024 is -# the rung that leaves room for the draft context (−~1,306 MiB against 2048, and -# the measured MTP arm ran here: 213.5 t/s at 1024 against 109.2 at 2048). +# the rung every measured MTP arm here ran on (213.5 t/s at 1024 against 109.2 +# at 2048) — worth ~406 MiB at THIS shape, not the ~1,306 the old pool showed. : "${BATCH_SIZE:=6144}" : "${UBATCH_SIZE:=1024}" # Asymmetric KV: f16 K (the engine's own default, spelled out so the contract is diff --git a/serverAppConfigs-docs/LLM/frank-zealot.md b/serverAppConfigs-docs/LLM/frank-zealot.md index 5d2221f..4438195 100644 --- a/serverAppConfigs-docs/LLM/frank-zealot.md +++ b/serverAppConfigs-docs/LLM/frank-zealot.md @@ -47,11 +47,12 @@ buys both and costs the keys their precision. **MTP is ON (Frank, 2026-10-19).** The drafter was always good — 2026-10-18 measured acceptance 0.798 / 0.617 per position, mean accepted length 2.41, and the publisher's card says depth 2 works best — and arming it is a budget decision, not a flag: the draft context is built against the *target's* pool, so `blk.32` bills 4,096 -B/position (2,352.00 MiB at 602,112) plus ~0.45 GiB of buffers at this ubatch and its own recurrent cells. -`CACHE_TYPE_V=q8_0` (−3,840 MiB against exact V at this pool) and `UBATCH_SIZE=1024` (−~1,306 MiB of compute -buffers, and the rung every measured MTP arm here ran on) are what pay for it; `UBATCH_SIZE=2048` beside an armed -drafter is the shape that halved decode (167.2 → 109.2 t/s). `--spec-draft-type-k`/`--spec-draft-type-v` (default -f16, verified in this build's `--help`) would cut the draft's bill again and stay at their default; there is no +B/position (2,352.00 MiB at 602,112) plus 480.97 MiB of its own CUDA0 buffers, and it triples the recurrent +state (452.25 MiB for three cells against 150.75 with the drafter off). `CACHE_TYPE_V=q8_0` is what pays that: +7,680 B/token cheaper than exact V = **4,410.00 MiB** at this pool. `UBATCH_SIZE=1024` — the rung every measured +MTP arm here ran on — contributes only ~406 MiB at THIS shape, not the ~1,306 the 786,432-pool census showed, +and `UBATCH_SIZE=2048` beside an armed drafter is still the shape that halved decode (167.2 → 109.2 t/s). +`--spec-draft-type-k`/`--spec-draft-type-v` (default f16, verified in this build's `--help`) would cut the draft's bill again and stay at their default; there is no `--spec-draft-model` — the draft rides the main model's weights, so a draft file would load the trunk twice. `SPEC_TYPE=none` is the rollback and strips the whole set. @@ -62,23 +63,47 @@ longer prompt with a 400, never a truncation — plus `maxTokens: 16384`, `input only `xhigh|medium|low` and raises on anything else), `extraBody` 1.0 / 0.95 / 20 / `min_p 0`, and the `compactionModel` of [[frank-archon]], [[frank-dragoon]] **and** frank-zealot all pointing here. -**This shape ships unbenchmarked, by law.** Frank ordered the reslice, not a -number, so the evidence a swap needs is the seam's fit gate (it prices cache + -draft cache + recurrent states against the 24,123 MiB the driver leaves -addressable, and refuses the two boots this shape can walk into: exact V with the -drafter armed, and the retired dtype pair at the retired pool), the deployer's -health gate, and the first verbosity-4 ledger of the new dtype pair — which is -recorded on this page because projections drift, and nowhere else. The previous -era's exact-KV boot and its rate floors live in that era's row below; those floors -were measured while another consumer drove the other slot (56 prompts in three -minutes, 96 % GPU-util), the exact condition a bench window exists to exclude. +**No rates, by law.** Frank ordered the reslice, not a number: the evidence a swap needs is +the seam's fit gate, the deployer's health gate, and the boot ledger below. The previous era's +exact-KV boot and its rate floors live in that era's row; those floors were measured while +another consumer drove the other slot (56 prompts in three minutes, 96 % GPU-util), the exact +condition a bench window exists to exclude. + +**The 2026-10-19 boot, measured** — verbosity-4 ledger of `ba69ea5`, started 03:21:56 PDT. +Recorded here because projections drift, and nowhere else: + +| Term | Ledger | +|---|---| +| Target KV over the 602,112 pool | **14,406.00 MiB** — K (f16) 9,408.00 + V (q8_0) 4,998.00, `200704 cells, 8 layers, 3/3 seqs` | +| Draft KV (`blk.32`, its own f16 context) | **2,352.00 MiB** — 1,176.00 + 1,176.00, `200704 cells, 1 layers, 3/3 seqs`: the whole pool, again | +| Recurrent state | **452.25 MiB** — `3 cells, 32 layers, 3 seqs, 2 rs_seq`, R (f32) 20.25 + S (f32) 432.00 | +| Compute buffers | target 1,665.00 CUDA0 + 424.88 CUDA_Host; draft context 480.97 + 424.86 | +| Weights | 2,283.29 CUDA0 + 257.66 CPU-mapped | +| Projector | 595.29 model + 248.10 compute + 24.93 CPU (its 868.32 worst case is still charged AFTER the engine's fit check) | +| Engine's own verdict | projected 21,639 vs 23,404 free, then **"cannot meet free memory target of 1,867 MiB, need to reduce device memory by 102 MiB"** | +| Settled | **22,870 used / 1,255 free**, `n_slots = 3`, `n_ctx_slot = 200704`, `kv_unified = false` | + +This is the first boot on this card that does **not** meet llama.cpp's own free-memory target, +and 1,255 MiB settled sits under the ~2 GiB band every previous era held. It boots, `/health` +answers, and the seam's gate still passes — it priced 22,428 MiB, conservative by 442 against +this ledger. The cheapest rungs back into the band are `CTX_SIZE=401408` (two conversations) or +`SPEC_TYPE=none`; both are one line and neither changes the payload. + +**The drafter is engaged, measured on the same boot** (three ordinary requests, not a bench): +acceptance 0.706 and 0.763 accepted-per-generated, mean accepted length 2.41 / 2.53, +per-position rates (0.882, 0.765) and (0.871, 0.699) — depth 2 earning its keep at 0.3–1.5 MiB +of draft state per saved prompt. The three probes that prove a swap: chat → `"zealot alive"` +with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY content with +`finish_reason: length` — the thinking-budget trap, not a fault); tools → +`finish_reason: tool_calls` with `{"a":24,"b":7}`; vision → a 512 px solid-red PNG answers +"Red" at `prompt_tokens` 280. **Fallback rungs**, cheapest first — one line of `frank-zealot.env` each, and each must keep the seam's fit gate green: | Knob | Effect from the shipped shape | |---|---| -| `SPEC_TYPE=none` | the drafter off: hands back 2,352 MiB of draft cache plus its buffers and recurrent cells, at the cost of the ~+28 % decode the 2026-10-18 census measured whenever the card has room | -| `UBATCH_SIZE=2048` | +~1,306 MiB of compute buffers and +4.6 % prefill — and the rung that HALVED decode beside an armed drafter (167.2 → 109.2 t/s) | +| `SPEC_TYPE=none` | the drafter off: hands back its 2,352.00 MiB of cache, 480.97 MiB of CUDA0 buffers and the 301.50 MiB of rollback planes it adds to the recurrent state (~3.1 GiB), at the cost of the ~+28 % decode the 2026-10-18 census measured whenever the card has room | +| `UBATCH_SIZE=2048` | +4.6 % prefill and only ~406 MiB of compute buffers at THIS shape (the ~1,306 MiB the 786,432-pool census showed does not transfer) — and the rung that HALVED decode beside an armed drafter (167.2 → 109.2 t/s) | | `CTX_SIZE=401408` | two 200,704 conversations: frees 4,802 MiB of cache — `contextWindow` MUST follow the per-slot number | | `CACHE_TYPE_V=f16` | exact V: +4,410 MiB at this pool with the drafter armed, which is a `cudaMalloc` death the seam refuses — legal only once the drafter or a slot is gone | | `CACHE_TYPE_K=q8_0` | the fully quantized pair (17,408 B/token): frees 4,410 MiB and moves the quantization error into the keys | @@ -127,6 +152,10 @@ minutes, 96 % GPU-util), the exact condition a bench window exists to exclude. `tests/frank_zealot_unit.bats`'s fit gate can see a boot that would die at `cudaMalloc`, so every retune keeps it green — and the first verbosity-4 ledger plus settled `nvidia-smi` of a new dtype belongs on this page, because projections drift. + The same trap one knob down: the ubatch 2048 → 1024 step handed back **~1,306 MiB** at the 786,432 / q8_0-pair + arm and **406 MiB** at the 602,112 / f16-K + q8_0-V boot (target compute 2,496.13 → 2,089.88, with CUDA0 going + UP 257 while CUDA_Host went down 663). Changing the dtype or the pool moves the compute buffers too, so a + credit measured at one shape is not a budget line at the next. The two boots that priced non-KV: 2 × 262,144 **without** the projector — 20,190 used, KV 12,800 ⇒ non-KV **7,390 MiB**; 3 × 196,608 **with** it — 22,972 used, KV 14,113 ⇒ non-KV **8,859 MiB**. The difference prices the vision tower at **~1.4 GiB for a 918 MB file**, because the encoder's activation and compute buffers ride with it.