docs(frank-zealot): record the ba69ea5 boot ledger and correct the carried credits

The boot of record for the new shape, from the verbosity-4 journal plus
settled nvidia-smi: target KV 14,406.00 (K f16 9,408.00 + V q8_0 4,998.00),
draft KV 2,352.00 over the same 200,704 cells x 3 seqs, recurrent state
452.25, compute 1,665.00 + 424.88 (draft 480.97 + 424.86), weights 2,283.29
+ 257.66, projector 595.29 + 248.10 + 24.93, settled 22,870 used / 1,255
free with n_slots = 3 / n_ctx_slot = 200704.

Two inherited numbers did not survive it, and both are corrected in the env,
the serve rationale and the page:

  - The ubatch 2048 -> 1024 step was quoted at "-~1,306 MiB". That was the
    786,432 / q8_0-pair arm. At THIS shape the target's compute buffers went
    2,496.13 -> 2,089.88: 406 MiB, with CUDA0 going UP 257 while CUDA_Host
    went down 663. q8_0 V (4,410 MiB, not the 3,840 previously written) is
    what funds the drafter, alone.
  - The fit gate priced 22,428 against a real 22,870: conservative by 442
    MiB, so it stays honest as written.

And one finding stated plainly: this is the first boot on this card that
misses llama.cpp's own free-memory target - the engine printed "cannot meet
free memory target of 1,867 MiB, need to reduce device memory by 102 MiB"
and settled at 1,255 MiB free, under the ~2 GiB band every previous era
held. Cheapest rungs back are CTX_SIZE=401408 or SPEC_TYPE=none, both one
line.

Probes on the same boot go in too: chat turn (448 tokens, 1,755 chars of
thinking - empty content at max_tokens 64 is the documented trap), tools
(finish_reason tool_calls, {"a":24,"b":7}), vision (512 px solid red ->
"Red", 280 prompt tokens), and the drafter's own statistics (acceptance
0.706 / 0.763, mean accepted length 2.41 / 2.53).
This commit is contained in:
Frank
2026-10-19 03:38:03 -07:00
parent ba69ea563e
commit 30e8ed55a0
3 changed files with 60 additions and 26 deletions
+6 -3
View File
@@ -32,9 +32,12 @@ PARALLEL=3
# on [[frank-zealot]]); 1024×6 keeps it a clean multiple of the ubatch.
BATCH_SIZE=6144
# Ubatch is a prefill knob (decode never moves with it) and it sets the
# compute-buffer size — which is what decides whether a drafter fits: 1024
# hands back ~1,306 MiB beside the draft context, and it is the rung the
# measured MTP arm ran (213.5 t/s at 1024 against 109.2 at 2048).
# compute-buffer size, which decides whether a drafter fits. 1024 is the rung
# every measured MTP arm here ran on (213.5 t/s against 109.2 at 2048). Its
# VRAM credit is SHAPE-specific: the 786,432-pool arm census showed ~1,306 MiB
# back, this boot measured 2,089.88 MiB of target compute against 2,496.13 at
# ubatch 2048 — 406 MiB back, with the split moving (CUDA0 +257, CUDA_Host
# −663). The drafter is funded by CACHE_TYPE_V below, not by this knob.
UBATCH_SIZE=1024
# K exact, V in the 8-bit class: 16,384 + 8,704 = 25,088 B/token (8,192
# elements per token per cache × 2 B for f16, × 1.0625 for q8_0 — there is no
+8 -6
View File
@@ -37,10 +37,12 @@
# accepts well (0.798/0.617 per position, mean length 2.41, measured 2026-10-18);
# what made it lose that evening was the bill, not the drafts — the draft context
# is built against the target's WHOLE pool, so blk.32 adds 4,096 B/position
# (2,352.00 MiB at 602,112) plus ~0.45 GiB of buffers at this ubatch, and the
# recurrent state triples. Dropping V to q8_0 (−3,840 MiB at this pool) and the
# ubatch to 1024 (−~1,306 MiB of compute buffers) is how those two bills got
# paid; the arm census and the rates are on [[frank-zealot]]. SPEC_TYPE=none is
# (2,352.00 MiB at 602,112) plus 480.97 MiB of its own compute buffers on the
# card, and it triples the recurrent state per slot (452.25 MiB for three cells
# against 50.25 a cell with the drafter off). CACHE_TYPE_V=q8_0 is what pays
# that: 7,680 B/token cheaper than exact V, 4,410 MiB at this pool. The ubatch
# drop to 1024 contributes ~406 MiB here, NOT the ~1,306 the 786,432-pool census
# showed — that figure does not transfer between shapes. SPEC_TYPE=none is
# the rollback and strips the whole flag set.
#
# Sampling: the checkpoint carries the publisher's thinking recipe as GGUF
@@ -81,8 +83,8 @@ fi
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves).
# The logical batch is VRAM- and throughput-neutral at a fixed ubatch, so 6144
# (1024×6) is free; the ubatch is what sets the compute-buffer size, and 1024 is
# the rung that leaves room for the draft context (−~1,306 MiB against 2048, and
# the measured MTP arm ran here: 213.5 t/s at 1024 against 109.2 at 2048).
# the rung every measured MTP arm here ran on (213.5 t/s at 1024 against 109.2
# at 2048) — worth ~406 MiB at THIS shape, not the ~1,306 the old pool showed.
: "${BATCH_SIZE:=6144}"
: "${UBATCH_SIZE:=1024}"
# Asymmetric KV: f16 K (the engine's own default, spelled out so the contract is