dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: shape F — MNBT 2048, util 0.982, CUDA-graph plan term pinned to the measured capture

Browse Source 
The arena is budget - non-expert - KV reserve - graphs - overhead. With util
capped at 0.982 (never 1.00) the graph estimate is the only plan term this repo
can shrink: the fork charges an analytic 256 MiB + 52 MiB/graph = 0.55 GiB where
the engine's own post-capture line reports 0.31 GiB, so pricing it at
256 + 12/graph returns ~0.24 GiB to the expert arena. MNBT 1024 -> 2048 is
arena-free: the registry overhead is base + 0.22 GiB per 1024 ABOVE its nbt 4096
anchor, and the fitted 5.25 GiB was fitted AT 4096.
This commit is contained in:
Frank
2026-10-03 02:22:49 -07:00
parent 6584c059d6
commit c682c6329c
5 changed files with 165 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+29 -5
View File
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default → expert arena **17.22 GiB / 7,522 slots**, +16 %), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,150,000,000/rank — the **pool, not the slot count, caps resident conversations at ~2 full-length** (measured live: `GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99×`), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. The pool cut buys runtime **headroom, NOT arena**: the arena is sized by `max_model_len` via the registry KV reserve (262144 → 6,248 slots; 200704 → 6,496) plus the overhead term (`--expert-precise-memory` → 7,522) — util 0.98, and the client contract wired: `--reasoning-parser …
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048 (arena-free: the registry prices overhead flat up to its nbt 4096 anchor), MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default → expert arena **17.22 GiB / 7,522 slots**, +16 %), **`VLLM_CUDAGRAPH_MEM_BASE_MB=256` / `_PER_GRAPH_MB=12`** (the plan's CUDA-graph term priced at the measured 0.31 GiB capture instead of the analytic 0.55 GiB → +0.24 GiB of arena), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,150,000,000/rank — the **pool, not the slot count, caps resident conversations at ~2 full-length** (measured live: `GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99×`), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. The pool cut buys runtime **headroom, NOT arena**: the arena is sized by `max_model_len` via the registry KV reserve (262144 → 6,248 slots; 200704 → 6,496) plus the overhead term (`--expert-precise-memory` → 7,522) — util 0.982 (Frank's cap; the field is `le=1`, and the ~3.5 GiB of free VRAM the driver reports is estimator slack, not allocatable arena), and the client contract wired: `--reasoning-parser …
| Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 |
| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **3 × 180,224** (pool 540,672 — NOT the native window; three contexts cost 81,920 tokens per conversation) with **asymmetric KV f16 K + q8_0 V** under flash attention, 21,684 used / **2,441 free** of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. Resized twice 2026-10-02 (Q6_K → Q5_K_M → 3 slots + vision → this pool); rollback `models/MiMo-Q6K/` or `PARALLEL=2 CTX_SIZE=524288` | 7771 |
@@ -433,12 +433,13 @@ The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666
digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`**
(~172.8 GiB, 131 shards) on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**,
**fp8 KV**, **no speculative decode**, **expert prefetch OFF**, and since
2026-10-02 the **shape-E** geometry: `--max-model-len 200704` × 3 slots with
2026-10-03 the **shape-F** geometry (shape E + MNBT 2048 + util 0.982 + the
CUDA-graph plan term pinned): `--max-model-len 200704` × 3 slots with
`--kv-cache-memory-bytes 3,150,000,000`/rank, so the pool admits **~2
full-length conversations** (measured live: `398,658 tokens, Maximum
concurrency for 200,704 tokens per request: 1.99×`) while the argv keeps the
only slot count that compiles (below), plus `--expert-precise-memory`
(`runtime overhead 5.25 GiB`, `expert cache space 17.22 GiB / 7,522 slots`),
only slot count that compiles (below), plus `--expert-precise-memory`,
`VLLM_CUDAGRAPH_MEM_BASE_MB=256` / `_PER_GRAPH_MB=12`,
`GPU_MAX_HW_QUEUES=2` and a mounted `/root/.cache`. `models.yml` carries
`contextWindow: 200704` for this endpoint — the served window, not the model's
native 262144, so the harness compacts before vLLM answers a longer prompt
@@ -455,7 +456,8 @@ sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets
((1,1024) vs (1,2048)). `--enforce-eager` bypasses the path but drops CUDA
graphs. The same failure class killed the 2026-10-01 bf16 2×262144 shape.
**The expert arena has TWO levers: `max_model_len` and the overhead term.**
**The expert arena has THREE reachable levers: `max_model_len`, the overhead
term, and the CUDA-graph estimate.**
The fork's planner sizes expert cache as `budget − non-expert − KV reserve −
graphs − overhead`, and the KV reserve is registry-priced at 9190 B/token ×
`max_model_len`: 262144 → 2.48 GiB reserve → arena 14.30 GiB / **6,248 slots**;
@@ -473,6 +475,28 @@ concurrency (a 2.2e9 probe reports 277,686 tokens / **1.38×**) and buy no slots
The arena (2.344 MiB per slot) is the decode-critical region — cutting it for KV
headroom collapses the offload hit rate.
**Free VRAM on this endpoint is NOT arena headroom (2026-10-03 audit).**
`rocm-smi` reads ~3.5 GiB free per card after hours of real traffic, but the
planner charges `budget = total × util` against terms that overstate reality:
the KV reserve is registry-priced at 9190 B/token (`1.91 GiB` at 200704) while
the real pool bills 2.93 GiB at 7,892 B/token, and the precise overhead (5.25
GiB) is far above the idle-measured figure. So the visible free space is the
estimator gap, not allocatable arena. `gpu_memory_utilization` is a real arena
lever (each +0.01 = +0.32 GiB) but the field is `Field(gt=0, le=1)` — **Frank
caps it at 0.982, never 1.00** (this endpoint has twice died at the ceiling).
What IS reachable from flags: `VLLM_CUDAGRAPH_MEM_BASE_MB` /
`_PER_GRAPH_MB`, read at call time by `estimate_cudagraph_memory()` — the
analytic default (256 MiB + 52 MiB/graph) charged 0.55 GiB where the engine's
own post-capture line reported 0.31 GiB, so pricing it honestly returns
~0.24 GiB to the arena. Everything else needs a registry edit inside the image
(the fork's log line says as much: *"Update the registry entry if this
drifts"*). Verify every deploy by comparing the `Estimated CUDA graph memory`
line with `Graph capturing finished … took X GiB`; if capture exceeds the
estimate, raise `_PER_GRAPH_MB` back toward 52. MNBT 1024 → 2048 costs ZERO
arena: the registry overhead is `base + 0.22 GiB × max(0,(nbt−4096)/1024)`
anchored at nbt 4096, so every rung ≤4096 charges the same 5.25 GiB — and the
fitted figure was fitted AT 4096.
**A container's compile cache does NOT survive `podman rm -f`, and on this
endpoint that is 181 s per boot.** Measured 2026-10-02: the image ships **no
`/root/.cache` at all** (fresh container: `No such file or directory`); after a