dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: 2 x 200704 fp8, pool cut for the expert arena, MNBT 1024

Browse Source 
The 2026-10-01 bf16 shape (2 x 262144, --expert-precise-memory)
crash-looped at graph capture — the arena is DEVICE-resident and that
shape had 0.13 GiB to spare. The plain bf16 2 x 262144 also did not stay
up: 0 bytes free at runtime, and a real client request (995-token prompt,
max_tokens=261149) killed the worker with .

The correction is a pool cut, not a downgrade. FP8 KV (7,701 B/token),
two 200704 windows (401,408 tokens -> 3.091e9 B floor, pinned 3.15e9),
MNBT 1024. The fork's expert planner sizes the arena AFTER the KV
override, so the ~2.95 GiB/rank reclaimed becomes arena: ~7,500 slots
vs 6,248. Two contexts at 200704 is a capacity contract, not a
throughput one (2-way concurrency summed BELOW parity at 4064 ctx), so
the slot buys the VRAM headroom long generations need.

Refs [[dankserver-qwen38-fp8-expert-offload-tuning]].

- scripts/frank-primary.env: MAXLEN 200704, SEQS 2, KV_BYTES 3.15e9,
  MNBT 1024, rewritten shape comment
- scripts/serve-frank-primary.sh: header comment updated
- tests/frank_primary_unit.bats: assertions updated, new seam pins the
  arena-floor pool law (the pool must move with a context change or the
  reduction buys nothing)
- docs + AGENTS.md: fleet topology and deployed shape corrected
This commit is contained in:
Comrade Chief Designer
2026-10-02 09:54:08 -07:00
parent f8f845d86b
commit 30904bac1b
5 changed files with 132 additions and 91 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md
+30 -23
View File
@@ -275,41 +275,48 @@ Depth caveat: every number above is at a near-empty KV. Deep-context
behavior (the gather win growing with depth, bf16's decode cost at depth)
is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
### The deployed shape (2026-10-01: 5 x 262144, fp8, no spec)
### The deployed shape (2026-10-02: 2 x 200704, fp8, pool cut for the arena)
```
FRANK_PRIMARY_MAXLEN=262144 x SEQS 2 (524,288 tokens of contract)
FRANK_PRIMARY_KV_BYTES=7300000000 PER RANK (bf16 = auto; 13,839 B/token)
FRANK_PRIMARY_KV_DTYPE=auto --kv-cache-dtype auto (bf16, unquantized)
FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (fitted overhead
5.25 GiB instead of the 7.60 GiB default,
so ~1,025 more arena slots)
FRANK_PRIMARY_MAXLEN=200704 x SEQS 2 (401,408 tokens of contract)
FRANK_PRIMARY_KV_BYTES=3150000000 PER RANK (fp8, 7,701 B/token)
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized)
FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
NO --speculative-config, NO --expert-prefetch
```
Boot-probed 2026-10-01 (transient systemd units at the production uid +
This replaced the 2026-10-01 shape — 2 x 262144 at bf16 KV with
`--expert-precise-memory` — which crash-looped at graph capture: the arena is
DEVICE-resident, the precise-memory flag drops the plan's overhead 7.60 ->
5.25 GiB, and that shape had 0.13 GiB to spare (weights 6.28 + pool 6.80 +
graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB,
NRestarts=3 at ~6.7 min each). The plain bf16 2 x 262144 boot also did not
stay up: it left 0 bytes free at runtime and a real client request (995-token
prompt, max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
... 0 bytes free`.
The correction is a pool cut, not a downgrade. fp8 costs 7,701 B/token
(measured 6,138 attention + 1,563 mamba/side, dtype-independent), so 2 x
200704 = 401,408 tokens needs 3.091e9 B; pinned at 3.15e9, ~44 MB of margin
above the floor — the same convention the 6.1e9 shape used. The arena is what
remains: the planner sizes it AFTER the KV override, so the 6,248 slots of the
6.1e9 shape become roughly 7,500 here. Two contexts at 200704 are a capacity
contract, not a throughput one (2-way concurrency summed BELOW parity at 4064
ctx), so the slot buys the VRAM headroom long generations need.
Boot-probed 2026-10-02 (transient systemd units at the production uid +
memlock, full-log capture; the rootless bench probe had masked the production
memory environment). The night searched two concurrency contracts:
| rung | result |
|---|---|
| 3 × 262144, bf16, KV 11,047,010,304 | **OOM at graph capture** — `Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB already allocated by PyTorch |
| 3 × 245760, bf16 | **OOM identically** |
| 3 × 237568, bf16 | PASS — KV 718,606 tokens |
| **3 × 241664, bf16** | **PASS — KV 732,359 tokens, 3.03x** (max possible at 3 contexts) |
| 6 × 241664, fp8 | **OOM** — tried 10.70 GiB against 9.18 GiB free: six full-length contexts do NOT fit |
| 5 × 262144, fp8 | PASS on the first rung — KV 1,350,182 tokens, 5.15x |
| 4 × 262144, fp8 | runtime-survivable stopgap — KV 1,079,582 tokens / 4.12x |
| 3 × 262144, fp8, pool at the floor | KV 792,069 tokens / 3.02x; arena 6,248 slots; 3.59 GiB free VRAM measured |
| **2 × 262144, bf16 + `--expert-precise-memory`** | **DEPLOYED** (Frank's call): unquantized KV at the full native window, two contexts; pool 7.3e9 B/rank (bf16 rate 13,839 B/token measured: 12,276 attention + 1,563 mamba/side, vs fp8's 6,138 + 1,563 = 7,701); the precise-memory flag drops the plan's overhead 7.60 -> 5.25 GiB so the arena grows from 6,248 to ~7,278 slots |
Boot is NOT proof of runtime survivability: the five-slot shape passed every
boot gate and still died on the first long-generation request. After any
knife-edge shape change, send a request with a large `max_tokens` before
calling it done. The deployed shape is **four contexts of the model's full
native 262,144 window**. fp8 KV is what buys the capacity —
| 3 x 262144, bf16, KV 11,047,010,304 | **OOM at graph capture** — `Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB already allocated by PyTorch |
| 3 x 245760, bf16 | **OOM identically** |
| 3 x 237568, bf16 | PASS — KV 718,606 tokens |
| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 |
| 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request |
| **2 x 200704, fp8, pool cut for the arena** | **DEPLOYED (2026-10-02, Frank's correction)**: fp8 KV, two 200704 windows, ~7,500 arena slots, ~3.6 GiB free VRAM measured |
7,926 B/token against bf16's ~13,900-14,047 — at the cost of bf16's ~4 %
decode edge. Six full-length contexts cannot fit at any KV dtype: the 6-seq
graph/activation footprint leaves only 9.18 GiB for the pool, and 6 × 262144