frank-primary: shape G measured — arena 17.77 GiB / 7,762 slots, KV 401,408 tokens / 2.00x
Record the live numbers and the UTIL 1.0 verdict (unbootable: worker init demands free > budget; ValueError on both ranks before a single weight loads). Driver used 30.97 of 31.86 GiB; the 0.89 GiB/card margin held through a live 4,096-token generation at ~38-39 tok/s.
This commit is contained in:
Frank committed
1 parent
c67a2aeb80
commit
47a2c1287b
8 files changed
+185
-41
No files matched your search
@@ -315,32 +315,33 @@ and every term on the right is either fixed or a lie:
|
||||
| `runtime overhead` | 5.25 GiB (`--expert-precise-memory`) | far lower at idle | no (registry constant) |
|
||||
|
||||
Shape F took the graph term (0.55 -> 0.32 GiB) and nudged util to 0.982:
|
||||
**arena 17.52 GiB / 7,653 slots measured**, +131 slots. That is where the
|
||||
"honest from flags" line sat — until shape G forced the other two.
|
||||
**arena 17.52 GiB / 7,653 slots measured**, +131 slots. Shape G takes the
|
||||
graph term further (0.32 -> 0.07 GiB) at the same util.
|
||||
|
||||
**Shape G forces both reachable terms.** The graph base pool is the actual
|
||||
over-charge: shape F's own boot reported
|
||||
**Shape G takes the graph term to the measured floor, util to the law.** The
|
||||
graph base pool is the actual over-charge: shape F's own boot reported
|
||||
`Graph capturing finished in 4 secs, took -0.06 GiB` — a **negative** capture
|
||||
delta, so the 256 MiB base was fiction. Charging `0 base + 12 MiB/graph`
|
||||
(0.07 GiB for the shipped 6-graph plan, and still per-graph so an 18-graph
|
||||
spec-decode shape pays 0.21 GiB) plus **util 1.0**, the field maximum, moves
|
||||
the plan to budget 31.86 / graphs 0.07 / **arena 18.35 GiB / ~8,016 slots** —
|
||||
+363 slots over F, +494 over E, and the ceiling of what a deploy can ask for.
|
||||
The KV ask went to `3,166,500,000` B/rank, which the engine's own 7,891.8
|
||||
B/token price lands at ~400,700 tokens (verify the `GPU KV cache size` line;
|
||||
the price is shape-dependent, 7,701 B/token at 262144).
|
||||
spec-decode shape pays 0.21 GiB) lands the plan at budget 31.29 / graphs 0.07 /
|
||||
**arena 17.77 GiB / 7,762 slots measured**. The KV ask went to
|
||||
`3,166,500,000` B/rank, which the engine's own 7,891.8 B/token price lands at
|
||||
**401,408 tokens measured / 2.00x concurrency** — the first shape holding two
|
||||
full-length conversations with real spare. (The ask/landed delta is block
|
||||
alignment: 196 x 2048. The price is shape-dependent, 7,701 B/token at 262144.)
|
||||
|
||||
**The price of the force is the margin, and it is real.** At util 1.0 the
|
||||
engine's budget equals the whole card, so any under-priced registry term stops
|
||||
showing up as a plan refusal and starts showing up as a runtime allocation
|
||||
failure. Measured driver high-water moves 28.28 GiB (E) -> 29.53 GiB (F) ->
|
||||
~30.4 GiB (G), i.e. **~1.5 GiB of true free VRAM per card** is now the safety
|
||||
margin — against 0.13 GiB on the bf16 boot that died and 0 bytes on the one
|
||||
that OOM'd a real 995-token request. Rollback is one number:
|
||||
`FRANK_PRIMARY_UTIL=0.982` costs 0.57 GiB of arena and buys back 0.57 GiB of
|
||||
margin. Everything beyond ~8,016 slots needs the registry edited inside the
|
||||
image (the fork's log line says exactly that: *"Update the registry entry if
|
||||
this drifts"*), not a deploy.
|
||||
**UTIL 1.00 is not a knife-edge on this fork, it is UNBOOTABLE — measured
|
||||
2026-10-03.** One deploy took the field maximum to buy the last 0.57 GiB of
|
||||
arena; both workers refused init BEFORE loading a single weight, twice:
|
||||
`ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)` — the worker
|
||||
demands free > budget and budget == total, so `le=1` on the CLI field is a
|
||||
trap: the engine dies in a 2-restart crash loop with zero payload progress.
|
||||
So `0.982` is a LAW, not a preference, and the "why 0.982" answer now has two
|
||||
proofs: this ValueError, and the historical ceiling deaths (the 2x262144 bf16
|
||||
boot at 0.13 GiB free; the 5x262144 boot at "0 bytes free" on a real
|
||||
995-token request). Everything beyond ~7,912 slots needs the registry edited
|
||||
inside the image (the fork's log line says exactly that: *"Update the
|
||||
registry entry if this drifts"*), not a deploy.
|
||||
|
||||
**MNBT 1024 -> 2048 is free on every axis the plan can see.** The registry
|
||||
overhead model is `base + 0.22 GiB x max(0, (nbt - 4096)/1024)` anchored at
|
||||
@@ -506,7 +507,8 @@ memory environment). The night searched two concurrency contracts:
|
||||
| 3 x 200704, fp8, pool sized for 2 contexts | **DEPLOYED 2026-10-02 (shape D)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
|
||||
| **3 x 200704, fp8, + `--expert-precise-memory`** | shape E (2026-10-02): same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s |
|
||||
| 3 x 200704, fp8, MNBT 2048, graph term 0.32 GiB, util 0.982 | shape F (2026-10-03): arena 17.52 GiB / **7,653 slots** measured, budget 31.29, `Estimated CUDA graph memory: 0.32 GiB`, KV unchanged 398,658 |
|
||||
| **3 x 200704, fp8, graph term 0.07 GiB, util 1.00, KV forced to 400,700** | **DEPLOYED 2026-10-03 (shape G)**: arena 18.35 GiB / ~8,016 slots (+363 over F, +494 over E), budget 31.86 = the whole card, driver high-water ~30.4 GiB -> **~1.5 GiB real free per card is the margin**; rollback is UTIL back to 0.982 |
|
||||
| 3 x 200704, fp8, graph term 0.07 GiB, util **1.00**, KV 3.1665e9 | **DID NOT BOOT (2026-10-03)**: `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks, twice, zero weights loaded — the worker demands free > budget and budget == total. UTIL 1.0 is unbootable, full stop |
|
||||
| **3 x 200704, fp8, graph term 0.07 GiB, util 0.982, KV forced to 400,700** | **DEPLOYED 2026-10-03 (shape G)**: arena **17.77 GiB / 7,762 slots** measured (+109 over F, +240 over E), KV **401,408 tokens / 2.00x** concurrency measured, capture 0.31/0.10 GiB vs the 0.07 charge (under-charged, boots and holds), driver used 30.97 GiB -> **0.89 GiB real free per card** held through a live 4,096-token generation (~38-39 tok/s single-request, +13% vs the 33.99 shape-E baseline); rollback is UTIL or KV_BYTES down a notch |
|
||||
|
||||
Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation
|
||||
footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even
|
||||
|
||||
@@ -1,5 +1,42 @@
|
||||
# Bundle Change Log
|
||||
|
||||
## 2026-10-03
|
||||
- **frank-primary: shape F then shape G — MNBT 2048, the CUDA-graph plan term
|
||||
pinned to the measured capture, KV forced to 400,700 tokens, and UTIL 1.0
|
||||
proven UNBOOTABLE.** Frank asked what the batch was with GPUs showing open
|
||||
space; the audit found the arena is `budget - non-expert - KV reserve -
|
||||
graphs - overhead` where the only deploy-reachable terms are the budget and
|
||||
the graph estimate (registry KV price 9,190 B/token and the 5.25 GiB
|
||||
overhead are image constants; the ~3.5 GiB of driver-free VRAM was
|
||||
estimator slack, not allocatable).
|
||||
- Shape F (c682c63): MNBT 1024 -> 2048 (arena-free — the registry prices
|
||||
overhead flat up to its nbt 4096 anchor and the fitted 5.25 was fitted AT
|
||||
4096), util 0.98 -> 0.982, graph term 0.55 -> 0.32 GiB (256 base +
|
||||
12/graph vs the fork's 256 + 52/graph). Measured: arena 17.52 GiB /
|
||||
7,653 slots, budget 31.29.
|
||||
- Shape G v1 (0a4ff30) tried util 1.0: **both workers refused init before a
|
||||
single weight loaded** — `ValueError: Free memory on device cuda:N
|
||||
(31.29/31.86 GiB ...)`, NRestarts=2 crash loop, zero payload progress.
|
||||
The CLI field is `Field(gt=0, le=1)`, but the worker demands free >
|
||||
budget, so budget == total is a trap, not a knife-edge. Frank: ".982 is
|
||||
the law, never 1.00" — now with a second proof.
|
||||
- Shape G v2 (c67a2ae, deployed green incl. e2e 33/33): util back to
|
||||
0.982, graph base to 0 (the shape F boot reported a NEGATIVE -0.06 GiB
|
||||
capture delta, so the 256 MiB base was fiction; 0 + 12 MiB/graph still
|
||||
prices a future 18-graph spec-decode shape at 0.21 GiB), KV_BYTES
|
||||
3.15e9 -> 3.1665e9 = Frank's 400,700-token ask.
|
||||
- MEASURED live: arena **17.77 GiB / 7,762 slots** (+240 over shape E),
|
||||
KV **401,408 tokens / 2.00x concurrency** — the first shape holding two
|
||||
full-length conversations with real spare; capture measured 0.31/0.10
|
||||
GiB against the 0.07 charge (under-charged, boots and holds); driver
|
||||
used 30.97 of 31.86 GiB -> **0.89 GiB real free per card**, held through
|
||||
a live 4,096-token generation (~38-39 tok/s single-request wall-incl.,
|
||||
+13% vs the 33.99 shape-E baseline; single sample, not a bench).
|
||||
- Verification seam rewritten to assert the shape by its OUTPUTS (util at
|
||||
the 0.982 law, graph charge below the analytic default and per-graph
|
||||
positive, KV pool >= 400,700 tokens at the measured 7,892 B/token).
|
||||
Runbook: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
|
||||
|
||||
## 2026-10-01
|
||||
- **frank-primary rolled BACK to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4;
|
||||
GLM/DFlash2 declared failed and purged.** The DFlash2 drafter booted and
|
||||
|
||||
Reference in new issue
Block a user