dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: shape E — --expert-precise-memory (arena 6,496->7,522 slots), GPU_MAX_HW_QUEUES=2, and a mounted /root/.cache

Browse Source 
Measured by boot probe on the shape-D base (production stopped, candidate
booted AS the service), plan numbers read off the engine's own log:

  runtime overhead   7.60 -> 5.25 GiB (precise gfx1201 TP2 registry)
  expert cache      14.87 -> 17.22 GiB
  slots              6,496 -> 7,522 (+1,026, +16%)
  swap region                13.7 GiB / 5,986 slots
  boot                       healthy in 420 s, no OOM

'--expert-precise-memory': True also appears in non-default args, so a silent
argparse drop is visible. The pool is NOT cut (option A) because the pool is
not an arena lever: 3.15e9/rank keeps shape D's measured 1.99x concurrency,
and a 2.2e9 arm probed in the same window (277,686 tokens, 1.38x) is recorded
as the conservative fallback.

Boot-cost bug found while diffing Frank's 3xR9700 recipe: the image ships no
/root/.cache at all, our boot writes 1.1 GB of vLLM AOT/torch_compile
artifacts (4,665 files) plus 229 MB of comgr JIT kernels into the ephemeral
layer, and podman rm -f discards them — so every deploy/restart/reboot re-pays
'torch.compile took 181.20 s in total' (173.19 s of it one graph). Mounting
FRANK_PRIMARY_CACHE_DIR at /root/.cache persists both and hides nothing.

GPU_MAX_HW_QUEUES=2 added at Frank's direction (real HSA knob); not A/B
measured. Retracted my own recipe-diff claim that cudagraph_capture_sizes
needed pinning: capture is 3 PIECEWISE + 2 FULL graphs, 3 s, 0.31 GiB.

Tests: 367/367.
This commit is contained in:
Frank
2026-10-02 14:37:57 -07:00
parent 812193aa63
commit 1ac4d48371
7 changed files with 314 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md
+78 -19
View File
@@ -275,7 +275,7 @@ Depth caveat: every number above is at a near-empty KV. Deep-context
behavior (the gather win growing with depth, bf16's decode cost at depth)
is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
### The deployed shape (2026-10-02: 3 slots x 200704, fp8, pool sized for 2 contexts)
### The deployed shape (2026-10-02 shape E: 3 slots x 200704, fp8, precise-memory arena)
```
FRANK_PRIMARY_MAXLEN=200704 200704-token window
@@ -283,10 +283,66 @@ FRANK_PRIMARY_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
FRANK_PRIMARY_KV_BYTES=3150000000 PER RANK (fp8, 7,892 B/token -> 1.99x)
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized)
FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024
FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2 -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary mounted at /root/.cache
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
NO --speculative-config, NO --expert-prefetch
```
**`--expert-precise-memory` is the arena lever that does not touch the window.**
The planner's overhead term is the only other place the arena can come from, and
the flag swaps the default registry figure for the fitted one. Measured
2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
booted AS the service; numbers are the engine's own plan lines):
| plan line | shape D | **shape E** |
|---|---|---|
| `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) |
| `expert cache space` | 14.87 GiB | **17.22 GiB** |
| slots | 6,496 | **7,522** (+1,026, +16%) |
| `swap region` | - | 13.7 GiB / 5,986 slots, 133-134 per layer |
| boot | - | healthy in 420 s, no OOM |
`'expert_precise_memory': True` also appears in `non-default args`, so a silent
argparse drop is visible in the boot log. The flag lands **in the arena, not the
pool** - `--kv-cache-memory-bytes` is untouched at 3.15e9/rank (option A), because
the pool is not an arena lever and cutting it further would only cost
concurrency: a 2.2e9 pool was probed in the same window and reports 277,686
tokens / **1.38x**, so it is the conservative fallback if free VRAM ever binds,
not the shipped value.
**The compile cache is a boot-cost fix, and it was worth 173 s per boot.**
Measured 2026-10-02: the image ships **no `/root/.cache` at all** (a fresh
container: `No such file or directory`), and after a boot that directory holds
`vllm` at **1,154,714,333 B in 4,665 files** plus `comgr` at 229 MB, all inside
the container's ephemeral layer. Every
`podman rm -f` - i.e. every deploy, restart and reboot - discards it, and the
engine re-pays:
```
Using cache directory: /root/.cache/vllm/torch_compile_cache/b265bd396a/rank_0_0/backbone
Dynamo bytecode transform time: 4.99 s
Compiling a graph for compile range (1, 1024) takes 173.19 s
saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/…
torch.compile took 181.20 s in total
init engine (profile, create kv cache, warmup model) took 217.26 s (compilation: 181.20 s)
```
Mounting a host directory at `/root/.cache` makes every later boot load the AOT
artifact instead. The mount is deliberately the **parent**, not
`/root/.cache/vllm`: `VLLM_CACHE_ROOT` defaults to the `vllm` child, and
`comgr` is the HIP-kernel JIT cache, so one mount persists both — and it hides
nothing, because the image ships no `/root/.cache`. The cache key
includes the compile range, so the MNBT choice costs exactly one compilation.
Entries are version- and config-hashed: a stale entry is a miss, never a wrong
answer.
`GPU_MAX_HW_QUEUES=2` caps the HSA hardware queues per GPU. Verified a real knob
rather than a no-op - the literal is read out of
`/opt/rocm/core-10.0/lib/libhsa-runtime64.so` - and shipped deliberately
alongside the 32x64 gather geometry. It is **not** A/B measured.
**The pool, not the slot count, sets how many conversations live at once.**
Measured live at this shape: `GPU KV cache size: 398,658 tokens, Maximum
concurrency for 200,704 tokens per request: 1.99x` (3,146,129,408 B reserved).
@@ -337,25 +393,27 @@ max_model_len** - not from the `--kv-cache-memory-bytes` value:
D(reserve) 0.57 GiB = D(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
pool cut (2.75 GiB) bought **zero** arena slots. It buys runtime HEADROOM,
which is still worth having - the shape it replaced died with `0 bytes free`
on a real 995-token request. Grow the arena by lowering `max_model_len`; the
pool is the headroom lever.
on a real 995-token request. Grow the arena by lowering `max_model_len`, or by
taking the overhead term with `--expert-precise-memory` (below); the pool is the
headroom lever only.
This replaced the 2026-10-01 shape — 2 x 262144 at bf16 KV with
`--expert-precise-memory` — which crash-looped at graph capture: the arena is
DEVICE-resident, the precise-memory flag drops the plan's overhead 7.60 ->
5.25 GiB, and that shape had 0.13 GiB to spare (weights 6.28 + pool 6.80 +
graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB,
NRestarts=3 at ~6.7 min each). The plain bf16 2 x 262144 boot also did not
stay up: it left 0 bytes free at runtime and a real client request (995-token
prompt, max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
#### The other arena lever: `--expert-precise-memory` (overhead 7.60 -> 5.25 GiB)
The flag is now ON. It was refused on 2026-10-01 for a reason that does not
generalize: the 2 x 262144 bf16 shape was already at the ceiling (weights 6.28 +
pool 6.80 + graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB,
NRestarts=3 at ~6.7 min each), so the 2.35 GiB the flag moves into the arena was
the last stack over the line, not the cause. The plain bf16 2 x 262144 boot also
did not stay up: 0 bytes free at runtime, and a real client request (995-token
prompt, `max_tokens=261149`) killed the worker with `Tried to allocate 50.00 MiB
... 0 bytes free`.
The correction is a pool cut plus a window trim, not a downgrade. Three
full-length 200704-token contexts need ~4.75e9 B; pinning the pool at 3.15e9
instead admits **1.99** of them. That is deliberate: the pool is the
runtime-HEADROOM lever (the shape this replaced died with 0 bytes free), while
the arena is sized by `max_model_len` alone. A third full-length request queues;
short requests use the spare blocks.
At 200704 x 3 the flag fits with room and is worth **+1,026 slots (+16%)**. What
made it affordable is the window trim, not a pool cut: the arena is
DEVICE-resident, the pool is not part of the budget's arithmetic, and the 200704
window's KV reserve (1.91 GiB) plus a smaller attention footprint is what the
plan needed. The pool stays at 3.15e9/rank deliberately (option A) - cutting it
further would cost concurrency (2.2e9 -> 1.38x) and buy no slots.
Boot-probed 2026-10-02 (transient systemd units at the production uid +
memlock, full-log capture; the rootless bench probe had masked the production
@@ -366,9 +424,10 @@ memory environment). The night searched two concurrency contracts:
| 3 x 262144, bf16, KV 11,047,010,304 | **OOM at graph capture** — `Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB already allocated by PyTorch |
| 3 x 245760, bf16 | **OOM identically** |
| 3 x 237568, bf16 | PASS — KV 718,606 tokens |
| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 |
| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 (the ceiling, not the flag) |
| 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request |
| **3 x 200704, fp8, pool sized for 2 contexts** | **DEPLOYED (2026-10-02)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
| 3 x 200704, fp8, pool sized for 2 contexts | **DEPLOYED 2026-10-02 (shape D)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
| **3 x 200704, fp8, + `--expert-precise-memory`** | **DEPLOYED 2026-10-02 (shape E)**: same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s |
Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation
footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even
serverAppConfigs-docs/LLM/index.md
+13 -7
View File
@@ -4,11 +4,14 @@ Documentation for the LLM inference fleet on the two hosts (2026-10-02):
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
on :7770, TP=2 with automatic tiered-expert offload, **shape D since
on :7770, TP=2 with automatic tiered-expert offload, **shape E since
2026-10-02: 3 slots x 200704 ctx** with the KV pool sized for **~2 full-length
conversations** — measured 398,658 tokens = 1.99x — fp8 KV, no speculative
decode, MNBT 1024, MoE gather 32x64, util 0.98, reasoning/tool parsers wired
(qwen3 / qwen3_xml) with preserve_thinking on. `--max-num-seqs` must stay 3:
(qwen3 / qwen3_xml) with preserve_thinking on, plus **`--expert-precise-memory`
(the arena grows to 17.22 GiB / 7,522 slots, +16%), `GPU_MAX_HW_QUEUES=2`, and a
mounted cache dir** (the image ships no `/root/.cache` at all; unmounted, every boot re-pays
`torch.compile took 181.20 s`). `--max-num-seqs` must stay 3:
SEQS 2 cannot compile this model. Rollback frank-primary-fp8-mtp-20261001),
and Dankclaw
runs two endpoints — **ROCmFPX llama.cpp (frank-large, port 7770:
@@ -34,7 +37,8 @@ gfx1201, 32 GiB each]
podman run, frank-primary, port 7770]
C[Qwen3.8-Flash-Next FP8, no spec decode
TP=2, expert offload automatic, 3 x 200704 ctx,
fp8 KV pool ~2 full-length, MNBT 1024, gather 32x64, qwen3 parsers]
fp8 KV pool ~2 full-length, MNBT 1024, gather 32x64, qwen3 parsers,
expert-precise-memory arena 7,522 slots, /root/.cache mounted]
end
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
@@ -81,13 +85,15 @@ frank-small, port 7771]
mmproj for vision, ~100 tok/s decode, ~2.7 GiB of the card left free.
The KV arithmetic, the flash-attn dependency, rollback to Q6_K
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **ACTIVE since
2026-10-02 (shape D)** (root systemd unit, port 7770): the tcclaviger RDNA4
2026-10-02 (shape E)** (root systemd unit, port 7770): the tcclaviger RDNA4
vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
R9700s, TP=2 with automatic expert offload — `--max-model-len 200704` x 3
slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x),
MNBT 1024, gather 32x64, no spec decode, prefetch off. Why `--max-num-seqs`
must stay 3 (SEQS 2 cannot compile this model), why the arena lever is
`max_model_len` rather than the KV pool, and the measured envelope
MNBT 1024, gather 32x64, no spec decode, prefetch off, plus
`--expert-precise-memory` (arena 17.22 GiB / 7,522 slots), GPU_MAX_HW_QUEUES=2
and a mounted compile cache. Why `--max-num-seqs`
must stay 3 (SEQS 2 cannot compile this model), why the arena's levers are
`max_model_len` and the overhead term rather than the KV pool, and the measured envelope
- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
the tcclaviger RDNA4 vLLM fork