frank-primary: shape G measured — arena 17.77 GiB / 7,762 slots, KV 401,408 tokens / 2.00x

Record the live numbers and the UTIL 1.0 verdict (unbootable: worker init
demands free > budget; ValueError on both ranks before a single weight loads).
Driver used 30.97 of 31.86 GiB; the 0.89 GiB/card margin held through a live
4,096-token generation at ~38-39 tok/s.
This commit is contained in:
Frank committed 2026-10-03 03:21:55 -07:00
1 parent c67a2aeb80
commit 47a2c1287b
8 files changed
+185 -41

No files matched your search

@@ -315,32 +315,33 @@ and every term on the right is either fixed or a lie:
| `runtime overhead` | 5.25 GiB (`--expert-precise-memory`) | far lower at idle | no (registry constant) |
Shape F took the graph term (0.55 -> 0.32 GiB) and nudged util to 0.982:
**arena 17.52 GiB / 7,653 slots measured**, +131 slots. That is where the
"honest from flags" line sat — until shape G forced the other two.
**arena 17.52 GiB / 7,653 slots measured**, +131 slots. Shape G takes the
graph term further (0.32 -> 0.07 GiB) at the same util.
**Shape G forces both reachable terms.** The graph base pool is the actual
over-charge: shape F's own boot reported
**Shape G takes the graph term to the measured floor, util to the law.** The
graph base pool is the actual over-charge: shape F's own boot reported
`Graph capturing finished in 4 secs, took -0.06 GiB` — a **negative** capture
delta, so the 256 MiB base was fiction. Charging `0 base + 12 MiB/graph`
(0.07 GiB for the shipped 6-graph plan, and still per-graph so an 18-graph
spec-decode shape pays 0.21 GiB) plus **util 1.0**, the field maximum, moves
the plan to budget 31.86 / graphs 0.07 / **arena 18.35 GiB / ~8,016 slots** —
+363 slots over F, +494 over E, and the ceiling of what a deploy can ask for.
The KV ask went to `3,166,500,000` B/rank, which the engine's own 7,891.8
B/token price lands at ~400,700 tokens (verify the `GPU KV cache size` line;
the price is shape-dependent, 7,701 B/token at 262144).
spec-decode shape pays 0.21 GiB) lands the plan at budget 31.29 / graphs 0.07 /
**arena 17.77 GiB / 7,762 slots measured**. The KV ask went to
`3,166,500,000` B/rank, which the engine's own 7,891.8 B/token price lands at
**401,408 tokens measured / 2.00x concurrency** — the first shape holding two
full-length conversations with real spare. (The ask/landed delta is block
alignment: 196 x 2048. The price is shape-dependent, 7,701 B/token at 262144.)
**The price of the force is the margin, and it is real.** At util 1.0 the
engine's budget equals the whole card, so any under-priced registry term stops
showing up as a plan refusal and starts showing up as a runtime allocation
failure. Measured driver high-water moves 28.28 GiB (E) -> 29.53 GiB (F) ->
~30.4 GiB (G), i.e. **~1.5 GiB of true free VRAM per card** is now the safety
margin — against 0.13 GiB on the bf16 boot that died and 0 bytes on the one
that OOM'd a real 995-token request. Rollback is one number:
`FRANK_PRIMARY_UTIL=0.982` costs 0.57 GiB of arena and buys back 0.57 GiB of
margin. Everything beyond ~8,016 slots needs the registry edited inside the
image (the fork's log line says exactly that: *"Update the registry entry if
this drifts"*), not a deploy.
**UTIL 1.00 is not a knife-edge on this fork, it is UNBOOTABLE — measured
2026-10-03.** One deploy took the field maximum to buy the last 0.57 GiB of
arena; both workers refused init BEFORE loading a single weight, twice:
`ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)` — the worker
demands free > budget and budget == total, so `le=1` on the CLI field is a
trap: the engine dies in a 2-restart crash loop with zero payload progress.
So `0.982` is a LAW, not a preference, and the "why 0.982" answer now has two
proofs: this ValueError, and the historical ceiling deaths (the 2x262144 bf16
boot at 0.13 GiB free; the 5x262144 boot at "0 bytes free" on a real
995-token request). Everything beyond ~7,912 slots needs the registry edited
inside the image (the fork's log line says exactly that: *"Update the
registry entry if this drifts"*), not a deploy.
**MNBT 1024 -> 2048 is free on every axis the plan can see.** The registry
overhead model is `base + 0.22 GiB x max(0, (nbt - 4096)/1024)` anchored at
@@ -506,7 +507,8 @@ memory environment). The night searched two concurrency contracts:
| 3 x 200704, fp8, pool sized for 2 contexts | **DEPLOYED 2026-10-02 (shape D)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
| **3 x 200704, fp8, + `--expert-precise-memory`** | shape E (2026-10-02): same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s |
| 3 x 200704, fp8, MNBT 2048, graph term 0.32 GiB, util 0.982 | shape F (2026-10-03): arena 17.52 GiB / **7,653 slots** measured, budget 31.29, `Estimated CUDA graph memory: 0.32 GiB`, KV unchanged 398,658 |
| **3 x 200704, fp8, graph term 0.07 GiB, util 1.00, KV forced to 400,700** | **DEPLOYED 2026-10-03 (shape G)**: arena 18.35 GiB / ~8,016 slots (+363 over F, +494 over E), budget 31.86 = the whole card, driver high-water ~30.4 GiB -> **~1.5 GiB real free per card is the margin**; rollback is UTIL back to 0.982 |
| 3 x 200704, fp8, graph term 0.07 GiB, util **1.00**, KV 3.1665e9 | **DID NOT BOOT (2026-10-03)**: `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks, twice, zero weights loaded — the worker demands free > budget and budget == total. UTIL 1.0 is unbootable, full stop |
| **3 x 200704, fp8, graph term 0.07 GiB, util 0.982, KV forced to 400,700** | **DEPLOYED 2026-10-03 (shape G)**: arena **17.77 GiB / 7,762 slots** measured (+109 over F, +240 over E), KV **401,408 tokens / 2.00x** concurrency measured, capture 0.31/0.10 GiB vs the 0.07 charge (under-charged, boots and holds), driver used 30.97 GiB -> **0.89 GiB real free per card** held through a live 4,096-token generation (~38-39 tok/s single-request, +13% vs the 33.99 shape-E baseline); rollback is UTIL or KV_BYTES down a notch |
Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation
footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even