docs(vault): the work log keeps incidents and laws; the numbers move to the page that owns them
log.md drops 466 lines of per-swap ledger prose (1,169 lines out, 703 back) and keeps the causal stories: why the host went down, why a knob resets at boot, why a stale drop-in served a different model for 24 minutes, why the RAM was the fault. Date sections are unchanged (65 before, 65 after) -- this is compression inside sections, not deletion of days. Every number the compression would have orphaned was ported to its owner first, and a census of the deleted lines against the whole vault is what found them: - frank-zealot: the two boots that priced non-KV (20,190/KV 12,800 => 7,390 without the projector; 22,972/KV 14,113 => 8,859 with it, so the vision tower costs ~1.4 GiB for a 918 MB file), the 1,153 MiB-free 3-slot boot that made the seam price non-KV at 9 GiB, and the f16 wall's own figures (fit 33,198 vs 23,558 free on a 25,088.00 MiB KV request). - frank-archon: the pool-cut-is-headroom-not-arena law (2.75 GiB/rank bought zero slots; d(reserve) = d(arena) = +248 slots; 14.87 GiB / 6,496 slots against a 398,658-token pool), the MNBT compile-RANGE vs SEQS+1 law, and the 2 x 196,919 prefix-cache proof (393,837 tokens resident above the pool's 386,904; 94.1/102.3/117.6 t/s, acceptance 0.7284/0.7525). - frank-dragoon: the KL arm of the 2026-10-17 A/B (mean 0.155 / median 0.057, top-1 agreement 85.1 %) joins the PPL and NIAH figures already on the era row. - Payload and Disk Reclaim: the 121 GB layer census lesson -- a huge image layer can be the engine image, not a payload. Also fixes a wikilink the rewrite had hard-wrapped across two lines, which the link checker reads as a target that does not exist. New seam: 'one page per live LLM endpoint'. The live set is derived from scripts/ (an endpoint has both an env and a unit), and any <endpoint>-*.md page in the vault fails it -- the enforcement of the one-page-per-endpoint law, so a re-created era page cannot pass tests.
This commit is contained in:
1 parent
6bdfd2853e
commit
846f81ddb8
6 files changed
+763
-1170
No files matched your search
@@ -118,6 +118,9 @@ re-run the same arms with the box quiet before a figure is carried anywhere else
|
||||
|
||||
- **Addressable ≠ advertised.** The card is 24,576 MiB but leaves **24,123 MiB**: quote `nvidia-smi`'s `free` field, never `total − used` — the driver's reservation is the difference.
|
||||
- **Exact KV and a 786,432 pool cannot coexist here**: 32,768 B/token ⇒ 24,576 MiB before a single weight loads (the retired "26,112 MiB" note doubled q8_0's scale overhead); the 27B analogue died at **25,088 MiB** with `cudaMalloc failed: out of memory`. Halving the pool funds exact KV; the per-slot window stops at the trained 262,144 (no YaRN here), so a third conversation is a KV-dtype decision, never more context.
|
||||
The wall's own figures: `cudaMalloc` failed on a **25,088.00 MiB** KV request when the engine's fit projection
|
||||
asked for **33,198 MiB** against **23,558 MiB** free. It booted f16 at all because the arm env's invented
|
||||
`KV_CACHE_TYPE` was swallowed by the serve script's `:=f16` default — so a failed window became the measurement.
|
||||
- **Exact KV reads 1.882× the bytes per generated token, not 2×** (32,768 vs 17,408 B/token, 2026-10-18): a decode drop of that order is EXPECTED and must be measured, while prefill should hold — and this service can afford the dtype precisely because its load is the compaction hub's ~170k-token prefill, which is weight-bandwidth bound and never re-reads the cache per generated token. The dtype is paid in decode and concurrency.
|
||||
- Never `--fit` with `parallel > 1` — it maximizes the *total* pool and shrinks each slot: set `CTX_SIZE` explicitly and keep it divisible by `PARALLEL` (the seam enforces it).
|
||||
- **KV rate law.** `full_layers × 4 KV heads × 256 dim` = **8,192 elements per token per cache** on this arch (the engine prints `n_embd_k_gqa = n_embd_v_gqa = 1024`), ×2 B (f16), ×1.0625 (q8_0 — 32 codes plus one 2-byte scale
|
||||
@@ -133,6 +136,11 @@ re-run the same arms with the box quiet before a figure is carried anywhere else
|
||||
`tests/frank_zealot_unit.bats`'s fit gate can see a boot that would die at `cudaMalloc`, so every retune keeps it
|
||||
green — and the first verbosity-4 ledger plus settled `nvidia-smi` of a new dtype belongs on this page, because
|
||||
projections drift.
|
||||
The two boots that priced non-KV: 2 × 262,144 **without** the projector — 20,190 used, KV 12,800 ⇒ non-KV
|
||||
**7,390 MiB**; 3 × 196,608 **with** it — 22,972 used, KV 14,113 ⇒ non-KV **8,859 MiB**. The difference prices the
|
||||
vision tower at **~1.4 GiB for a 918 MB file**, because the encoder's activation and compute buffers ride with it.
|
||||
The first 3-slot deploy (3 × 196,608) booted healthy and left only **1,153 MiB** free — that is how the 2–3 GiB
|
||||
band got violated in practice, and why the seam prices non-KV at the measured 9 GiB and refuses that shape.
|
||||
- **Ubatch is a prefill knob.** Decode never moves with it and the logical `--batch-size` is neutral; prefill climbs monotonically to **2048** on this CUDA card (+4.6 % at 8.9k depth; 710 → 1,188 t/s), a prompt shorter than the ubatch cannot see the knob at all (the control row), and a verdict never crosses backends — the 2× R9700 split peaks at 512. Each rung costs ~0.6–0.75 GiB of compute buffers; 2048 is the ceiling for a 27B-on-3090 shape.
|
||||
- **A quant rung is paid in decode**: Q5_K_M → Q6_K is +875 MiB of weights and **−10 %** rate on this bandwidth-bound card, and the byte-bandwidth law predicts a rung's rate within 1 %.
|
||||
- **Speculation pays only with room on the card.** The 2026-10-18 arm census (all q8_0 K+V at the 786,432 pool): 3 × 262,144 spec-OFF ubatch 2048 → 17,876 / 6,249 free, 167.2 t/s; spec-ON ubatch 2048 → 22,706 / 1,419, 109.2 t/s; spec-ON ubatch 1024 → 21,400 / 2,725, 213.5 t/s; 2 × 262,144 spec-ON → 213.2; 1 × 262,144 spec-ON → 214.2. That is **+28 % decode whenever the card has room**, and the doors back are one window per conversation, or q8_0 KV with ubatch 1024. The draft's own bill at that shape was 3,072.00 MiB of KV plus ~1,758 MiB of buffers.
|
||||
|
||||
Reference in new issue
Block a user