dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

frank-primary: 3 slots x 200704, fp8, pool sized for 2 contexts

Browse Source 
Re-applies the desired shape incrementally on the proven-good fp8 3x262144
base, after bisecting why the first attempt (30904ba) crash-looped.

Root cause: `--max-num-seqs 2` cannot compile on this model. AOT
build_guards(strict_error=True) inside profile_run -> _dummy_run specializes
`query_start_loc`'s width to SEQS + 1 (3 at SEQS 2, 4 at SEQS 3) and the
ConstraintViolationError kills engine init ~7 min in. MNBT is orthogonal:
SEQS 2 failed at MNBT 1024 AND 2048; SEQS 3 passed at both. KV dtype was a
red herring. Measured by four single-variable boot probes.

Arena lever measured: the expert arena is sized from `max_model_len` via the
registry-priced KV reserve (9190 B/token x max_model_len), NOT from
--kv-cache-memory-bytes. 262144 -> 14.30 GiB / 6,248 slots; 200704 ->
14.87 GiB / 6,496 slots. The 6.1e9 -> 3.15e9 pool cut bought zero arena and
is a runtime-headroom lever.

Shape: MAXLEN 200704, SEQS 3 (unchanged - the only slot count that compiles),
KV_BYTES 3.15e9/rank = 2.04x full-length concurrency at 7,701 B/token fp8,
MNBT 1024, gather 32x64, no spec decode, no prefetch, no precise-memory.
This commit is contained in:
Frank
2026-10-02 11:33:27 -07:00
parent e460bbbab5
commit c769ff2f63
6 changed files with 218 additions and 151 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md
+67 -19
View File
@@ -275,17 +275,67 @@ Depth caveat: every number above is at a near-empty KV. Deep-context
behavior (the gather win growing with depth, bf16's decode cost at depth)
is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
### The deployed shape (2026-10-02: 2 x 200704, fp8, pool cut for the arena)
### The deployed shape (2026-10-02: 3 slots x 200704, fp8, pool sized for 2 contexts)
```
FRANK_PRIMARY_MAXLEN=200704 x SEQS 2 (401,408 tokens of contract)
FRANK_PRIMARY_KV_BYTES=3150000000 PER RANK (fp8, 7,701 B/token)
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized)
FRANK_PRIMARY_MAXLEN=200704 200704-token window
FRANK_PRIMARY_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
FRANK_PRIMARY_KV_BYTES=3150000000 PER RANK (fp8, 7,701 B/token -> 2.04x)
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized)
FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
NO --speculative-config, NO --expert-prefetch
```
**The pool, not the slot count, sets how many conversations live at once.**
fp8 costs 7,701 B/token (measured 6,138 attention + 1,563 mamba/side,
dtype-independent), so one 200704-token request costs 1.546e9 B and KV_BYTES
3.15e9 admits **2.04 full-length requests**. The endpoint advertises 3 slots: a
third LONG request queues rather than being refused, and shorter requests use
the spare blocks. Two contexts at 200704 are a capacity contract, not a
throughput one (2-way concurrency summed BELOW parity at 4064 ctx).
#### `--max-num-seqs 2` is impossible on this model (measured 2026-10-02)
Four single-variable boot probes against the then-deployed shape, production
stopped for each:
| MAXLEN | SEQS | MNBT | result |
|---|---|---|---|
| 262144 | 2 | 1024 | **Engine core initialization failed** |
| 262144 | 2 | 2048 | **Engine core initialization failed** |
| 262144 | 3 | 1024 | healthy - 792,069 tokens, 3.02x, arena 6,248 |
| 262144 | 3 | 2048 | healthy - the then-deployed incumbent |
The failure is `torch.fx.experimental.symbolic_shapes.ConstraintViolationError:
Constraints violated (L['query_start_loc'].size()[0]) ... specialized it to be
a constant (3)`, raised from AOT `build_guards(strict_error=True)` during
`determine_available_memory -> profile_run -> _dummy_run`. `model_runner.py`
sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets
`num_reqs = min(max_num_batched_tokens, max_num_reqs)`, so the width is
`SEQS + 1` for ANY `MNBT >= SEQS`: 3 at SEQS 2, 4 at SEQS 3. **MNBT is not a
lever here** - it only moves the compile RANGE ((1,1024) vs (1,2048)), which
is why the identical error appears at both rungs. `--enforce-eager` bypasses
the path at the cost of CUDA graphs. The same failure class killed the
2026-10-01 bf16 2 x 262144 shape.
#### The arena lever is `max_model_len`, not `--kv-cache-memory-bytes`
The fork's planner sizes expert cache as `budget - non-expert - KV reserve -
graphs - overhead`, and the KV reserve is registry-priced at **9190 B/token x
max_model_len** - not from the `--kv-cache-memory-bytes` value:
| MAXLEN | KV reserve | arena |
|---|---|---|
| 262144 | 2.48 GiB | 14.30 GiB / **6,248 slots** |
| 200704 | 1.91 GiB | 14.87 GiB / **6,496 slots** |
D(reserve) 0.57 GiB = D(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
pool cut (2.75 GiB) bought **zero** arena slots. It buys runtime HEADROOM,
which is still worth having - the shape it replaced died with `0 bytes free`
on a real 995-token request. Grow the arena by lowering `max_model_len`; the
pool is the headroom lever.
This replaced the 2026-10-01 shape — 2 x 262144 at bf16 KV with
`--expert-precise-memory` — which crash-looped at graph capture: the arena is
DEVICE-resident, the precise-memory flag drops the plan's overhead 7.60 ->
@@ -296,14 +346,12 @@ stay up: it left 0 bytes free at runtime and a real client request (995-token
prompt, max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
... 0 bytes free`.
The correction is a pool cut, not a downgrade. fp8 costs 7,701 B/token
(measured 6,138 attention + 1,563 mamba/side, dtype-independent), so 2 x
200704 = 401,408 tokens needs 3.091e9 B; pinned at 3.15e9, ~44 MB of margin
above the floor — the same convention the 6.1e9 shape used. The arena is what
remains: the planner sizes it AFTER the KV override, so the 6,248 slots of the
6.1e9 shape become roughly 7,500 here. Two contexts at 200704 are a capacity
contract, not a throughput one (2-way concurrency summed BELOW parity at 4064
ctx), so the slot buys the VRAM headroom long generations need.
The correction is a pool cut plus a window trim, not a downgrade. fp8 costs
7,701 B/token, so 3 x 200704 = 602,112 tokens would need 4.64e9 B for three
full-length contexts; pinning the pool at 3.15e9 instead admits 2.04 of them.
That is deliberate: the pool is the runtime-HEADROOM lever (the shape this
replaced died with 0 bytes free), while the arena is sized by `max_model_len`
alone. A third full-length request queues; short requests use the spare blocks.
Boot-probed 2026-10-02 (transient systemd units at the production uid +
memlock, full-log capture; the rootless bench probe had masked the production
@@ -316,11 +364,11 @@ memory environment). The night searched two concurrency contracts:
| 3 x 237568, bf16 | PASS — KV 718,606 tokens |
| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 |
| 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request |
| **2 x 200704, fp8, pool cut for the arena** | **DEPLOYED (2026-10-02, Frank's correction)**: fp8 KV, two 200704 windows, ~7,500 arena slots, ~3.6 GiB free VRAM measured |
7,926 B/token against bf16's ~13,900-14,047 — at the cost of bf16's ~4 %
decode edge. Six full-length contexts cannot fit at any KV dtype: the 6-seq
graph/activation footprint leaves only 9.18 GiB for the pool, and 6 × 262144
needs 12.47 GB even at fp8.
| **3 x 200704, fp8, pool sized for 2 contexts** | **DEPLOYED (2026-10-02)**: 2.04x concurrency, arena 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation
footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even
at fp8.
The old "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 WORKSPACE
reservation, not a VRAM wall. Failures are clean and diagnostic: the engine
@@ -379,8 +427,8 @@ Without the copy the exec dies with `can't open file '/mtp_measure.py': [Errno 2
No such file or directory` — measured 2026-10-01, which silently cost a whole
arm's numbers. A window script must do the copy, not assume the path exists.
### The deployed shape, measured again (2026-10-01 A/B window)
A clean production-environment boot of the deployed config (4 x 262144 fp8, MNBT
### The then-deployed shape, measured again (2026-10-01 A/B window)
A clean production-environment boot of the config deployed at that time (4 x 262144 fp8, MNBT
2048 in that window) reported **`GPU KV cache size: 1,079,582 tokens, Maximum
concurrency for 262,144 tokens per request: 4.12x`** — the same figures as the
deployed MNBT-1024 shape, i.e. MNBT does not move the pool, and the capacity is