frank-primary: shape E — --expert-precise-memory (arena 6,496->7,522 slots), GPU_MAX_HW_QUEUES=2, and a mounted /root/.cache
Measured by boot probe on the shape-D base (production stopped, candidate booted AS the service), plan numbers read off the engine's own log: runtime overhead 7.60 -> 5.25 GiB (precise gfx1201 TP2 registry) expert cache 14.87 -> 17.22 GiB slots 6,496 -> 7,522 (+1,026, +16%) swap region 13.7 GiB / 5,986 slots boot healthy in 420 s, no OOM '--expert-precise-memory': True also appears in non-default args, so a silent argparse drop is visible. The pool is NOT cut (option A) because the pool is not an arena lever: 3.15e9/rank keeps shape D's measured 1.99x concurrency, and a 2.2e9 arm probed in the same window (277,686 tokens, 1.38x) is recorded as the conservative fallback. Boot-cost bug found while diffing Frank's 3xR9700 recipe: the image ships no /root/.cache at all, our boot writes 1.1 GB of vLLM AOT/torch_compile artifacts (4,665 files) plus 229 MB of comgr JIT kernels into the ephemeral layer, and podman rm -f discards them — so every deploy/restart/reboot re-pays 'torch.compile took 181.20 s in total' (173.19 s of it one graph). Mounting FRANK_PRIMARY_CACHE_DIR at /root/.cache persists both and hides nothing. GPU_MAX_HW_QUEUES=2 added at Frank's direction (real HSA knob); not A/B measured. Retracted my own recipe-diff claim that cudagraph_capture_sizes needed pinning: capture is 3 PIECEWISE + 2 FULL graphs, 3 s, 0.31 GiB. Tests: 367/367.
This commit is contained in:
Frank committed
1 parent
812193aa63
commit
1ac4d48371
7 files changed
+314
-51
No files matched your search
@@ -275,7 +275,7 @@ Depth caveat: every number above is at a near-empty KV. Deep-context
|
||||
behavior (the gather win growing with depth, bf16's decode cost at depth)
|
||||
is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
|
||||
|
||||
### The deployed shape (2026-10-02: 3 slots x 200704, fp8, pool sized for 2 contexts)
|
||||
### The deployed shape (2026-10-02 shape E: 3 slots x 200704, fp8, precise-memory arena)
|
||||
|
||||
```
|
||||
FRANK_PRIMARY_MAXLEN=200704 200704-token window
|
||||
@@ -283,10 +283,66 @@ FRANK_PRIMARY_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
|
||||
FRANK_PRIMARY_KV_BYTES=3150000000 PER RANK (fp8, 7,892 B/token -> 1.99x)
|
||||
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized)
|
||||
FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024
|
||||
FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
|
||||
FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2 -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
|
||||
FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary mounted at /root/.cache
|
||||
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
|
||||
NO --speculative-config, NO --expert-prefetch
|
||||
```
|
||||
|
||||
**`--expert-precise-memory` is the arena lever that does not touch the window.**
|
||||
The planner's overhead term is the only other place the arena can come from, and
|
||||
the flag swaps the default registry figure for the fitted one. Measured
|
||||
2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
|
||||
booted AS the service; numbers are the engine's own plan lines):
|
||||
|
||||
| plan line | shape D | **shape E** |
|
||||
|---|---|---|
|
||||
| `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) |
|
||||
| `expert cache space` | 14.87 GiB | **17.22 GiB** |
|
||||
| slots | 6,496 | **7,522** (+1,026, +16%) |
|
||||
| `swap region` | - | 13.7 GiB / 5,986 slots, 133-134 per layer |
|
||||
| boot | - | healthy in 420 s, no OOM |
|
||||
|
||||
`'expert_precise_memory': True` also appears in `non-default args`, so a silent
|
||||
argparse drop is visible in the boot log. The flag lands **in the arena, not the
|
||||
pool** - `--kv-cache-memory-bytes` is untouched at 3.15e9/rank (option A), because
|
||||
the pool is not an arena lever and cutting it further would only cost
|
||||
concurrency: a 2.2e9 pool was probed in the same window and reports 277,686
|
||||
tokens / **1.38x**, so it is the conservative fallback if free VRAM ever binds,
|
||||
not the shipped value.
|
||||
|
||||
**The compile cache is a boot-cost fix, and it was worth 173 s per boot.**
|
||||
Measured 2026-10-02: the image ships **no `/root/.cache` at all** (a fresh
|
||||
container: `No such file or directory`), and after a boot that directory holds
|
||||
`vllm` at **1,154,714,333 B in 4,665 files** plus `comgr` at 229 MB, all inside
|
||||
the container's ephemeral layer. Every
|
||||
`podman rm -f` - i.e. every deploy, restart and reboot - discards it, and the
|
||||
engine re-pays:
|
||||
|
||||
```
|
||||
Using cache directory: /root/.cache/vllm/torch_compile_cache/b265bd396a/rank_0_0/backbone
|
||||
Dynamo bytecode transform time: 4.99 s
|
||||
Compiling a graph for compile range (1, 1024) takes 173.19 s
|
||||
saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/…
|
||||
torch.compile took 181.20 s in total
|
||||
init engine (profile, create kv cache, warmup model) took 217.26 s (compilation: 181.20 s)
|
||||
```
|
||||
|
||||
Mounting a host directory at `/root/.cache` makes every later boot load the AOT
|
||||
artifact instead. The mount is deliberately the **parent**, not
|
||||
`/root/.cache/vllm`: `VLLM_CACHE_ROOT` defaults to the `vllm` child, and
|
||||
`comgr` is the HIP-kernel JIT cache, so one mount persists both — and it hides
|
||||
nothing, because the image ships no `/root/.cache`. The cache key
|
||||
includes the compile range, so the MNBT choice costs exactly one compilation.
|
||||
Entries are version- and config-hashed: a stale entry is a miss, never a wrong
|
||||
answer.
|
||||
|
||||
`GPU_MAX_HW_QUEUES=2` caps the HSA hardware queues per GPU. Verified a real knob
|
||||
rather than a no-op - the literal is read out of
|
||||
`/opt/rocm/core-10.0/lib/libhsa-runtime64.so` - and shipped deliberately
|
||||
alongside the 32x64 gather geometry. It is **not** A/B measured.
|
||||
|
||||
**The pool, not the slot count, sets how many conversations live at once.**
|
||||
Measured live at this shape: `GPU KV cache size: 398,658 tokens, Maximum
|
||||
concurrency for 200,704 tokens per request: 1.99x` (3,146,129,408 B reserved).
|
||||
@@ -337,25 +393,27 @@ max_model_len** - not from the `--kv-cache-memory-bytes` value:
|
||||
D(reserve) 0.57 GiB = D(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
|
||||
pool cut (2.75 GiB) bought **zero** arena slots. It buys runtime HEADROOM,
|
||||
which is still worth having - the shape it replaced died with `0 bytes free`
|
||||
on a real 995-token request. Grow the arena by lowering `max_model_len`; the
|
||||
pool is the headroom lever.
|
||||
on a real 995-token request. Grow the arena by lowering `max_model_len`, or by
|
||||
taking the overhead term with `--expert-precise-memory` (below); the pool is the
|
||||
headroom lever only.
|
||||
|
||||
This replaced the 2026-10-01 shape — 2 x 262144 at bf16 KV with
|
||||
`--expert-precise-memory` — which crash-looped at graph capture: the arena is
|
||||
DEVICE-resident, the precise-memory flag drops the plan's overhead 7.60 ->
|
||||
5.25 GiB, and that shape had 0.13 GiB to spare (weights 6.28 + pool 6.80 +
|
||||
graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB,
|
||||
NRestarts=3 at ~6.7 min each). The plain bf16 2 x 262144 boot also did not
|
||||
stay up: it left 0 bytes free at runtime and a real client request (995-token
|
||||
prompt, max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
|
||||
#### The other arena lever: `--expert-precise-memory` (overhead 7.60 -> 5.25 GiB)
|
||||
|
||||
The flag is now ON. It was refused on 2026-10-01 for a reason that does not
|
||||
generalize: the 2 x 262144 bf16 shape was already at the ceiling (weights 6.28 +
|
||||
pool 6.80 + graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB,
|
||||
NRestarts=3 at ~6.7 min each), so the 2.35 GiB the flag moves into the arena was
|
||||
the last stack over the line, not the cause. The plain bf16 2 x 262144 boot also
|
||||
did not stay up: 0 bytes free at runtime, and a real client request (995-token
|
||||
prompt, `max_tokens=261149`) killed the worker with `Tried to allocate 50.00 MiB
|
||||
... 0 bytes free`.
|
||||
|
||||
The correction is a pool cut plus a window trim, not a downgrade. Three
|
||||
full-length 200704-token contexts need ~4.75e9 B; pinning the pool at 3.15e9
|
||||
instead admits **1.99** of them. That is deliberate: the pool is the
|
||||
runtime-HEADROOM lever (the shape this replaced died with 0 bytes free), while
|
||||
the arena is sized by `max_model_len` alone. A third full-length request queues;
|
||||
short requests use the spare blocks.
|
||||
At 200704 x 3 the flag fits with room and is worth **+1,026 slots (+16%)**. What
|
||||
made it affordable is the window trim, not a pool cut: the arena is
|
||||
DEVICE-resident, the pool is not part of the budget's arithmetic, and the 200704
|
||||
window's KV reserve (1.91 GiB) plus a smaller attention footprint is what the
|
||||
plan needed. The pool stays at 3.15e9/rank deliberately (option A) - cutting it
|
||||
further would cost concurrency (2.2e9 -> 1.38x) and buy no slots.
|
||||
|
||||
Boot-probed 2026-10-02 (transient systemd units at the production uid +
|
||||
memlock, full-log capture; the rootless bench probe had masked the production
|
||||
@@ -366,9 +424,10 @@ memory environment). The night searched two concurrency contracts:
|
||||
| 3 x 262144, bf16, KV 11,047,010,304 | **OOM at graph capture** — `Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB already allocated by PyTorch |
|
||||
| 3 x 245760, bf16 | **OOM identically** |
|
||||
| 3 x 237568, bf16 | PASS — KV 718,606 tokens |
|
||||
| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 |
|
||||
| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 (the ceiling, not the flag) |
|
||||
| 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request |
|
||||
| **3 x 200704, fp8, pool sized for 2 contexts** | **DEPLOYED (2026-10-02)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
|
||||
| 3 x 200704, fp8, pool sized for 2 contexts | **DEPLOYED 2026-10-02 (shape D)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
|
||||
| **3 x 200704, fp8, + `--expert-precise-memory`** | **DEPLOYED 2026-10-02 (shape E)**: same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s |
|
||||
|
||||
Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation
|
||||
footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even
|
||||
|
||||
@@ -4,11 +4,14 @@ Documentation for the LLM inference fleet on the two hosts (2026-10-02):
|
||||
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
|
||||
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
|
||||
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
|
||||
on :7770, TP=2 with automatic tiered-expert offload, **shape D since
|
||||
on :7770, TP=2 with automatic tiered-expert offload, **shape E since
|
||||
2026-10-02: 3 slots x 200704 ctx** with the KV pool sized for **~2 full-length
|
||||
conversations** — measured 398,658 tokens = 1.99x — fp8 KV, no speculative
|
||||
decode, MNBT 1024, MoE gather 32x64, util 0.98, reasoning/tool parsers wired
|
||||
(qwen3 / qwen3_xml) with preserve_thinking on. `--max-num-seqs` must stay 3:
|
||||
(qwen3 / qwen3_xml) with preserve_thinking on, plus **`--expert-precise-memory`
|
||||
(the arena grows to 17.22 GiB / 7,522 slots, +16%), `GPU_MAX_HW_QUEUES=2`, and a
|
||||
mounted cache dir** (the image ships no `/root/.cache` at all; unmounted, every boot re-pays
|
||||
`torch.compile took 181.20 s`). `--max-num-seqs` must stay 3:
|
||||
SEQS 2 cannot compile this model. Rollback frank-primary-fp8-mtp-20261001),
|
||||
and Dankclaw
|
||||
runs two endpoints — **ROCmFPX llama.cpp (frank-large, port 7770:
|
||||
@@ -34,7 +37,8 @@ gfx1201, 32 GiB each]
|
||||
podman run, frank-primary, port 7770]
|
||||
C[Qwen3.8-Flash-Next FP8, no spec decode
|
||||
TP=2, expert offload automatic, 3 x 200704 ctx,
|
||||
fp8 KV pool ~2 full-length, MNBT 1024, gather 32x64, qwen3 parsers]
|
||||
fp8 KV pool ~2 full-length, MNBT 1024, gather 32x64, qwen3 parsers,
|
||||
expert-precise-memory arena 7,522 slots, /root/.cache mounted]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
@@ -81,13 +85,15 @@ frank-small, port 7771]
|
||||
mmproj for vision, ~100 tok/s decode, ~2.7 GiB of the card left free.
|
||||
The KV arithmetic, the flash-attn dependency, rollback to Q6_K
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **ACTIVE since
|
||||
2026-10-02 (shape D)** (root systemd unit, port 7770): the tcclaviger RDNA4
|
||||
2026-10-02 (shape E)** (root systemd unit, port 7770): the tcclaviger RDNA4
|
||||
vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
|
||||
R9700s, TP=2 with automatic expert offload — `--max-model-len 200704` x 3
|
||||
slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x),
|
||||
MNBT 1024, gather 32x64, no spec decode, prefetch off. Why `--max-num-seqs`
|
||||
must stay 3 (SEQS 2 cannot compile this model), why the arena lever is
|
||||
`max_model_len` rather than the KV pool, and the measured envelope
|
||||
MNBT 1024, gather 32x64, no spec decode, prefetch off, plus
|
||||
`--expert-precise-memory` (arena 17.22 GiB / 7,522 slots), GPU_MAX_HW_QUEUES=2
|
||||
and a mounted compile cache. Why `--max-num-seqs`
|
||||
must stay 3 (SEQS 2 cannot compile this model), why the arena's levers are
|
||||
`max_model_len` and the overhead term rather than the KV pool, and the measured envelope
|
||||
- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
|
||||
failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
|
||||
the tcclaviger RDNA4 vLLM fork
|
||||
|
||||
@@ -1106,3 +1106,85 @@ and then answers with nothing, and curl reports "Empty reply from server" (52).
|
||||
- **Operating lesson: `deploy.sh <target>` with no `--service` is a fleet-wide
|
||||
drain.** It restarts every service whose recorded revision is behind HEAD and
|
||||
whose paths changed since. To touch one service, name it.
|
||||
|
||||
## 2026-10-02 — frank-primary shape E: `--expert-precise-memory` (+1,026 arena slots), a mounted compile cache, and the P2P answer
|
||||
|
||||
Shape D left one question open — the plan's overhead term was reserving 2.35 GiB
|
||||
of compile headroom the boot never used — and the answer to Frank's free-VRAM
|
||||
question arrived together with a boot-cost bug that had been costing 181 s on
|
||||
every restart since the cutover.
|
||||
|
||||
- **`--expert-precise-memory` is real and it fits (option A).** Boot-probed on
|
||||
the shape-D base with production stopped, the candidate booted AS the service
|
||||
(`FRANK_PRIMARY_ENV_FILE=/tmp/probe-b.env bash scripts/serve-frank-primary.sh`,
|
||||
trap-guaranteed restore), the numbers read off the engine's own plan lines:
|
||||
|
||||
| plan line | shape D | shape E |
|
||||
|---|---|---|
|
||||
| `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) |
|
||||
| `expert cache space` | 14.87 GiB | **17.22 GiB** |
|
||||
| slots | 6,496 | **7,522** (+1,026, +16 %) |
|
||||
| `swap region` | — | 13.7 GiB / 5,986 slots, 133-134 per layer |
|
||||
| boot | — | **healthy in 420 s**, no OOM |
|
||||
|
||||
The flag also prints `'expert_precise_memory': True` in `non-default args`, so
|
||||
a silent argparse drop cannot hide. It was refused on 2026-10-01 only because
|
||||
the 2 x 262144 bf16 shape was already 0.13 GiB under the ceiling — there the
|
||||
flag was the last stack over the line, not the cause.
|
||||
- **The pool was NOT cut (option A) because the pool is not an arena lever.**
|
||||
A 2.2e9 arm was probed in the same window and reports `277,686 tokens,
|
||||
1.38x` — measured, rejected as the shipped value, kept as the conservative
|
||||
fallback. With 3.15e9/rank the endpoint keeps shape D's 1.99x concurrency and
|
||||
gains the arena, which is the shape D follow-up had actually been looking for.
|
||||
- **THE COMPILE CACHE WAS BEING THROWN AWAY ON EVERY RESTART — 181 s each.**
|
||||
Measured: a fresh container has **no `/root/.cache` at all** (`No such file or
|
||||
directory`); after our boot it holds `/root/.cache/vllm` at
|
||||
**1,154,714,333 B in 4,665 files** plus `/root/.cache/comgr` at 229 MB, inside
|
||||
the ephemeral layer, and `podman rm -f` (every deploy, restart, reboot)
|
||||
discards them. The engine names the loss itself: `Compiling a graph for compile
|
||||
range (1, 1024) takes 173.19 s` ... `torch.compile took 181.20 s in total` ...
|
||||
`init engine (profile, create kv cache, warmup model) took 217.26 s
|
||||
(compilation: 181.20 s)`. The serve script now mounts
|
||||
`FRANK_PRIMARY_CACHE_DIR` at `/root/.cache` — the **parent**, so the vLLM
|
||||
AOT/torch_compile tree (`VLLM_CACHE_ROOT`, default `/root/.cache/vllm`) and
|
||||
the comgr JIT cache both persist, and nothing is hidden because the image has
|
||||
no such directory. Same class of fix as the recipe's
|
||||
`/cache/{triton,vllm,inductor}` volumes; the compile RANGE is part of the
|
||||
cache key, so an MNBT change costs one recompilation, not a permanent tax.
|
||||
- **`GPU_MAX_HW_QUEUES=2`** (HSA hardware queues per GPU) added at Frank's
|
||||
direction. Verified a real knob rather than a no-op — the literal is read from
|
||||
`/opt/rocm/core-10.0/lib/libhsa-runtime64.so` — and shipped deliberately
|
||||
alongside the 32x64 gather geometry. **Not A/B measured.**
|
||||
- **P2P: the hardware supports it, and the engine is already using it.** The
|
||||
live log shows `r4d all-reduce INSTALLED (rank=0 max=49152KB finegrained=T…)`
|
||||
on BOTH ranks and `Using ['R4D_AR', 'PYNCCL'] all-reduce backends` — R4D's
|
||||
P2P path, with RCCL as fallback, measured at **45.274 GB/s bidirectional**.
|
||||
Root ports `00:01.1`/`00:01.3` both report ACS `ReqRedir+`, so P2P hairpins at
|
||||
the root complex; there is no XGMI. The old FAIL predates the 2026-09-20 board
|
||||
swap. Kernel support confirmed: `CONFIG_HSA_AMD_P2P=y`, `CONFIG_PCI_P2PDMA=y`,
|
||||
`CONFIG_HSA_AMD=y`, `CONFIG_ZONE_DEVICE=y`, `iommu=pt` on the cmdline.
|
||||
- **Recipe diff (Frank's 3 x R9700 compose vs this podman run), ranked:**
|
||||
- *Acted on:* persistent compile caches (above — the one real operational
|
||||
gap), and `GPU_MAX_HW_QUEUES`.
|
||||
- *Retracted:* pinned `--compilation-config cudagraph_capture_sizes` was on my
|
||||
list; measured, it is **not** a gap here — capture is 3 PIECEWISE + 2 FULL
|
||||
graphs, `Graph capturing finished in 3 secs, took 0.31 GiB` (0.11 GiB on the
|
||||
lighter rank). No 15-52-size ladder to cut.
|
||||
- *Deliberate divergence, keep ours:* no MTP (the recipe has it; measured
|
||||
fastest here), `qwen3_xml` vs `qwen3_coder` (ours verified live), no
|
||||
`--override-generation-config` (server-side sampling defaults), published
|
||||
port + Caddy rather than `network_mode: host`.
|
||||
- *Non-gaps:* `shm_size 32gb` (ours is `--ipc=host`), `memlock -1` (same),
|
||||
`SYS_PTRACE` (same), `/dev/kfd` + `/dev/dri` (same), `--enable-chunked-prefill`
|
||||
(V1 default), `plecache` volume (the flag is not even passed), and
|
||||
`ROCR_VISIBLE_DEVICES=1,2,3` vs our `HIP_VISIBLE_DEVICES=0,1` — verified
|
||||
equivalent here: `rocm-smi --showid` puts the two R9700s at GPU[0]/GPU[1] and
|
||||
the 7700X iGPU at GPU[2], so our indices are the right two cards.
|
||||
- **Harness note, because it cost two probe attempts:** a probe that reported an
|
||||
instantaneous candidate death was a logging artifact of the inline SSH script,
|
||||
not a crash — the container and its serve script must be polled by recorded id
|
||||
and their logs dumped on the exit path, or a failure looks like a bad flag.
|
||||
The flag was fine both times.
|
||||
- Live: `health=200`, `NRestarts=0`, arena 17.22 GiB / 7,522 slots, KV 398,658
|
||||
tokens / 1.99x. Docs: [[dankserver-qwen38-fp8-expert-offload-tuning]].
|
||||
|
||||
Reference in new issue
Block a user