dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: shape G measured — arena 17.77 GiB / 7,762 slots, KV 401,408 tokens / 2.00x

Browse Source 
Record the live numbers and the UTIL 1.0 verdict (unbootable: worker init
demands free > budget; ValueError on both ranks before a single weight loads).
Driver used 30.97 of 31.86 GiB; the 0.89 GiB/card margin held through a live
4,096-token generation at ~38-39 tok/s.
This commit is contained in:
Frank
2026-10-03 03:21:55 -07:00
parent c67a2aeb80
commit 47a2c1287b
8 changed files with 185 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+19 -15
View File
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048 (arena-free: the registry prices overhead flat up to its nbt 4096 anchor), MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default → expert arena **17.22 GiB / 7,522 slots**, +16 %), **`VLLM_CUDAGRAPH_MEM_BASE_MB=256` / `_PER_GRAPH_MB=12`** (the plan's CUDA-graph term priced at the measured 0.31 GiB capture instead of the analytic 0.55 GiB → +0.24 GiB of arena), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,150,000,000/rank — the **pool, not the slot count, caps resident conversations at ~2 full-length** (measured live: `GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99×`), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. The pool cut buys runtime **headroom, NOT arena**: the arena is sized by `max_model_len` via the registry KV reserve (262144 → 6,248 slots; 200704 → 6,496) plus the overhead term (`--expert-precise-memory` → 7,522) — util 0.982 (Frank's cap; the field is `le=1`, and the ~3.5 GiB of free VRAM the driver reports is estimator slack, not allocatable arena), and the client contract wired: `--reasoning-parser …
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048 (arena-free: the registry prices overhead flat up to its nbt 4096 anchor), MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default), **`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`** (the plan's CUDA-graph term priced at the measured capture — 0.31 GiB then a NEGATIVE −0.06 GiB delta, so the 256 MiB analytic base was fiction → graph term 0.07 GiB), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,166,500,000/rank (~400,700 KV tokens at the engine's 7,891.8 B/token here) — the **pool, not the slot count, caps resident conversations** (measured live: `GPU KV cache size: 401,408 tokens, Maximum concurrency for 200,704 tokens per request: 2.00×`; the 3.15e9 pool below it measured 398,658/1.99×), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. **util stays 0.982 BY LAW**: at `le=1` the worker init refuses `budget == total` with `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks before a single weight loads — measured crash loop 2026-10-03. Arena at shape G: **17.77 GiB / 7,762 slots measured**, KV 401,408 tokens / 2.00× concurrency, 0.89 GiB of real free VRAM per card held through a live 4,096-token generation (the ~3.5 GiB of driver-reported free VRAM at shape E was estimator slack — the registry's 9190 B/token KV reserve vs the real 7,892, and the 5.25 GiB overhead vs far less — NOT allocatable), and the client contract wired: `--reasoning-parser …
| Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 |
| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **3 × 180,224** (pool 540,672 — NOT the native window; three contexts cost 81,920 tokens per conversation) with **asymmetric KV f16 K + q8_0 V** under flash attention, 21,684 used / **2,441 free** of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. Resized twice 2026-10-02 (Q6_K → Q5_K_M → 3 slots + vision → this pool); rollback `models/MiMo-Q6K/` or `PARALLEL=2 CTX_SIZE=524288` | 7771 |
@@ -433,13 +433,12 @@ The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666
digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`**
(~172.8 GiB, 131 shards) on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**,
**fp8 KV**, **no speculative decode**, **expert prefetch OFF**, and since
2026-10-03 the **shape-F** geometry (shape E + MNBT 2048 + util 0.982 + the
CUDA-graph plan term pinned): `--max-model-len 200704` × 3 slots with
`--kv-cache-memory-bytes 3,150,000,000`/rank, so the pool admits **~2
full-length conversations** (measured live: `398,658 tokens, Maximum
concurrency for 200,704 tokens per request: 1.99×`) while the argv keeps the
only slot count that compiles (below), plus `--expert-precise-memory`,
`VLLM_CUDAGRAPH_MEM_BASE_MB=256` / `_PER_GRAPH_MB=12`,
2026-10-03 the **shape-G** geometry (shape E + MNBT 2048 + the CUDA-graph plan
term at the measured floor): `--max-model-len 200704` × 3 slots with
`--kv-cache-memory-bytes 3,166,500,000`/rank (~400,700 KV tokens — the engine
prices this pool at 7,891.8 B/token here), util **0.982** (the law — see the
unbootable-1.0 lesson below), `--expert-precise-memory`,
`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`,
`GPU_MAX_HW_QUEUES=2` and a mounted `/root/.cache`. `models.yml` carries
`contextWindow: 200704` for this endpoint — the served window, not the model's
native 262144, so the harness compacts before vLLM answers a longer prompt
@@ -482,15 +481,20 @@ the KV reserve is registry-priced at 9190 B/token (`1.91 GiB` at 200704) while
the real pool bills 2.93 GiB at 7,892 B/token, and the precise overhead (5.25
GiB) is far above the idle-measured figure. So the visible free space is the
estimator gap, not allocatable arena. `gpu_memory_utilization` is a real arena
lever (each +0.01 = +0.32 GiB) but the field is `Field(gt=0, le=1)` — **Frank
caps it at 0.982, never 1.00** (this endpoint has twice died at the ceiling).
What IS reachable from flags: `VLLM_CUDAGRAPH_MEM_BASE_MB` /
lever (each +0.01 = +0.32 GiB) but **0.982 is the law, and 1.00 is UNBOOTABLE
on this fork — measured 2026-10-03**: the worker init demands
`free > budget`, and at `le=1` budget == device total, so both ranks raise
`ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)` before a
single weight loads, twice (2-restart crash loop, zero payload progress). The
CLI field accepting `le=1` is the trap; Frank's 0.982 is where it always
ships. What IS reachable from flags: `VLLM_CUDAGRAPH_MEM_BASE_MB` /
`_PER_GRAPH_MB`, read at call time by `estimate_cudagraph_memory()` — the
analytic default (256 MiB + 52 MiB/graph) charged 0.55 GiB where the engine's
own post-capture line reported 0.31 GiB, so pricing it honestly returns
~0.24 GiB to the arena. Everything else needs a registry edit inside the image
(the fork's log line says as much: *"Update the registry entry if this
drifts"*). Verify every deploy by comparing the `Estimated CUDA graph memory`
own post-capture line reported 0.31 GiB, and a second boot (with the plan
already charging 0.32 GiB) reported **-0.06 GiB**, so the 256 MiB base pool is
pure over-charge: shape G charges `0 base + 12 MiB/graph` = 0.07 GiB, which
still prices per graph for a future 18-graph spec-decode shape (0.21 GiB).
Verify every deploy by comparing the `Estimated CUDA graph memory`
line with `Graph capturing finished … took X GiB`; if capture exceeds the
estimate, raise `_PER_GRAPH_MB` back toward 52. MNBT 1024 → 2048 costs ZERO
arena: the registry overhead is `base + 0.22 GiB × max(0,(nbt−4096)/1024)`