frank-primary: shape G measured — arena 17.77 GiB / 7,762 slots, KV 401,408 tokens / 2.00x
Record the live numbers and the UTIL 1.0 verdict (unbootable: worker init demands free > budget; ValueError on both ranks before a single weight loads). Driver used 30.97 of 31.86 GiB; the 0.89 GiB/card margin held through a live 4,096-token generation at ~38-39 tok/s.
This commit is contained in:
1 parent
c67a2aeb80
commit
47a2c1287b
8 files changed
+185
-41
No files matched your search
@@ -315,32 +315,33 @@ and every term on the right is either fixed or a lie:
|
||||
| `runtime overhead` | 5.25 GiB (`--expert-precise-memory`) | far lower at idle | no (registry constant) |
|
||||
|
||||
Shape F took the graph term (0.55 -> 0.32 GiB) and nudged util to 0.982:
|
||||
**arena 17.52 GiB / 7,653 slots measured**, +131 slots. That is where the
|
||||
"honest from flags" line sat — until shape G forced the other two.
|
||||
**arena 17.52 GiB / 7,653 slots measured**, +131 slots. Shape G takes the
|
||||
graph term further (0.32 -> 0.07 GiB) at the same util.
|
||||
|
||||
**Shape G forces both reachable terms.** The graph base pool is the actual
|
||||
over-charge: shape F's own boot reported
|
||||
**Shape G takes the graph term to the measured floor, util to the law.** The
|
||||
graph base pool is the actual over-charge: shape F's own boot reported
|
||||
`Graph capturing finished in 4 secs, took -0.06 GiB` — a **negative** capture
|
||||
delta, so the 256 MiB base was fiction. Charging `0 base + 12 MiB/graph`
|
||||
(0.07 GiB for the shipped 6-graph plan, and still per-graph so an 18-graph
|
||||
spec-decode shape pays 0.21 GiB) plus **util 1.0**, the field maximum, moves
|
||||
the plan to budget 31.86 / graphs 0.07 / **arena 18.35 GiB / ~8,016 slots** —
|
||||
+363 slots over F, +494 over E, and the ceiling of what a deploy can ask for.
|
||||
The KV ask went to `3,166,500,000` B/rank, which the engine's own 7,891.8
|
||||
B/token price lands at ~400,700 tokens (verify the `GPU KV cache size` line;
|
||||
the price is shape-dependent, 7,701 B/token at 262144).
|
||||
spec-decode shape pays 0.21 GiB) lands the plan at budget 31.29 / graphs 0.07 /
|
||||
**arena 17.77 GiB / 7,762 slots measured**. The KV ask went to
|
||||
`3,166,500,000` B/rank, which the engine's own 7,891.8 B/token price lands at
|
||||
**401,408 tokens measured / 2.00x concurrency** — the first shape holding two
|
||||
full-length conversations with real spare. (The ask/landed delta is block
|
||||
alignment: 196 x 2048. The price is shape-dependent, 7,701 B/token at 262144.)
|
||||
|
||||
**The price of the force is the margin, and it is real.** At util 1.0 the
|
||||
engine's budget equals the whole card, so any under-priced registry term stops
|
||||
showing up as a plan refusal and starts showing up as a runtime allocation
|
||||
failure. Measured driver high-water moves 28.28 GiB (E) -> 29.53 GiB (F) ->
|
||||
~30.4 GiB (G), i.e. **~1.5 GiB of true free VRAM per card** is now the safety
|
||||
margin — against 0.13 GiB on the bf16 boot that died and 0 bytes on the one
|
||||
that OOM'd a real 995-token request. Rollback is one number:
|
||||
`FRANK_PRIMARY_UTIL=0.982` costs 0.57 GiB of arena and buys back 0.57 GiB of
|
||||
margin. Everything beyond ~8,016 slots needs the registry edited inside the
|
||||
image (the fork's log line says exactly that: *"Update the registry entry if
|
||||
this drifts"*), not a deploy.
|
||||
**UTIL 1.00 is not a knife-edge on this fork, it is UNBOOTABLE — measured
|
||||
2026-10-03.** One deploy took the field maximum to buy the last 0.57 GiB of
|
||||
arena; both workers refused init BEFORE loading a single weight, twice:
|
||||
`ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)` — the worker
|
||||
demands free > budget and budget == total, so `le=1` on the CLI field is a
|
||||
trap: the engine dies in a 2-restart crash loop with zero payload progress.
|
||||
So `0.982` is a LAW, not a preference, and the "why 0.982" answer now has two
|
||||
proofs: this ValueError, and the historical ceiling deaths (the 2x262144 bf16
|
||||
boot at 0.13 GiB free; the 5x262144 boot at "0 bytes free" on a real
|
||||
995-token request). Everything beyond ~7,912 slots needs the registry edited
|
||||
inside the image (the fork's log line says exactly that: *"Update the
|
||||
registry entry if this drifts"*), not a deploy.
|
||||
|
||||
**MNBT 1024 -> 2048 is free on every axis the plan can see.** The registry
|
||||
overhead model is `base + 0.22 GiB x max(0, (nbt - 4096)/1024)` anchored at
|
||||
@@ -506,7 +507,8 @@ memory environment). The night searched two concurrency contracts:
|
||||
| 3 x 200704, fp8, pool sized for 2 contexts | **DEPLOYED 2026-10-02 (shape D)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
|
||||
| **3 x 200704, fp8, + `--expert-precise-memory`** | shape E (2026-10-02): same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s |
|
||||
| 3 x 200704, fp8, MNBT 2048, graph term 0.32 GiB, util 0.982 | shape F (2026-10-03): arena 17.52 GiB / **7,653 slots** measured, budget 31.29, `Estimated CUDA graph memory: 0.32 GiB`, KV unchanged 398,658 |
|
||||
| **3 x 200704, fp8, graph term 0.07 GiB, util 1.00, KV forced to 400,700** | **DEPLOYED 2026-10-03 (shape G)**: arena 18.35 GiB / ~8,016 slots (+363 over F, +494 over E), budget 31.86 = the whole card, driver high-water ~30.4 GiB -> **~1.5 GiB real free per card is the margin**; rollback is UTIL back to 0.982 |
|
||||
| 3 x 200704, fp8, graph term 0.07 GiB, util **1.00**, KV 3.1665e9 | **DID NOT BOOT (2026-10-03)**: `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks, twice, zero weights loaded — the worker demands free > budget and budget == total. UTIL 1.0 is unbootable, full stop |
|
||||
| **3 x 200704, fp8, graph term 0.07 GiB, util 0.982, KV forced to 400,700** | **DEPLOYED 2026-10-03 (shape G)**: arena **17.77 GiB / 7,762 slots** measured (+109 over F, +240 over E), KV **401,408 tokens / 2.00x** concurrency measured, capture 0.31/0.10 GiB vs the 0.07 charge (under-charged, boots and holds), driver used 30.97 GiB -> **0.89 GiB real free per card** held through a live 4,096-token generation (~38-39 tok/s single-request, +13% vs the 33.99 shape-E baseline); rollback is UTIL or KV_BYTES down a notch |
|
||||
|
||||
Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation
|
||||
footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even
|
||||
|
||||
Reference in new issue
Block a user