dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: vault link integrity restored after rogue-session doc WIP landed

Browse Source 
The parallel session's stash pop left GGUF Compatibility, Multi-Device
Splits, Serving Sizing, PCIe Link and Atomics, RDNA4 Container Serving,
vLLM Porting triage and RDNA4 MoE Budget unindexed and carrying 9
unresolved wikilinks (LLM Bench Windows / Model Swap / AGENTS targets
that never existed, a [[nodiscard]] C++ attribute parsed as a wiki link,
and two multi-line-wrapped link tokens). All repointed to real pages;
six rows added to the vault index; seam green.
This commit is contained in:
Frank
2026-10-05 18:40:31 -07:00
parent 380aafa1cc
commit a6ad9af708
10 changed files with 2013 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md
+195 -1
View File
@@ -95,6 +95,51 @@ llama-server --hf-repo <target-GGUF> \
The DFlash drafter proposes, then ngram-mod runs as an additional implementation in the chain. Verified against upstream `common/speculative.cpp` (b10133+).
### Chaining laws (measured on this fleet, 2026-09 → 10)
`--spec-type` takes a comma list, and the gate is `ctx_dft` in
`common/speculative.cpp` (`common_speculative_init_result`): **draftless**
types (the ngram family — simple, map-k, map-k4v, mod, cache) are added
unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`,
`draft-dspark`, `draft-eagle3`) all require a draft CONTEXT. Consequences:
- A chain may hold ONE context-backed drafter plus any number of ngram types.
`draft-dflash,ngram-mod` is legal and measured: 2.6x the drafter alone on
prompt-echo traffic, byte-identical to it on prose at the fleet's 48/64
window.
- Two context-backed types cannot share one process: `draft-mtp,draft-dflash`
dies at load with `context type MTP requested but model doesn't contain MTP
layers` — MTP builds its context against whatever the draft context holds,
so it met the DRAFTER's file, not a missing head. Pin that in a seam so
nobody "fixes the flag" later.
- Internal order is fixed (ngram types first, then drafters); CLI order is
irrelevant.
- **A legal chain can still lose on a recurrent model.** `draft-mtp,ngram-mod`
on the qwen35 hybrid (frank-large era) measured −15 % prose / −7 %
block-copy (2026-10-03): adding the ngram layer logs `recurrent state
rollback is not compatible with 'ngram-mod' - disabling rollback support`,
and MTP verification depends on that rollback, so acceptance drops on ALL
traffic even where the matcher fires. Measure chains per architecture,
never by analogy.
- ngram-mod prints NO startup banner — the load log shows only the drafter's
`adding speculative implementation` line. Never conclude "chain not
engaged" from the load log; identify behaviourally (`draft_n` > 0 only
where text repeats) or by the per-task `draft acceptance = ... mean len`
line. It costs ~64 MiB of host-RAM hash table.
### Drafter KV: read the window policy from the header
The draft context is built with `cparams.n_ctx = llama_n_ctx(ctx_tgt)`
(`common/speculative.cpp`) — there is no flag to cap it — so a drafter that
declares NO sliding window caches the whole target window on whatever device
`--spec-draft-device` names. DFlash declares `sliding_window = 2048` (~40 MiB
of draft KV); DSpark declares none: 5 layers x 2 x 8 kv heads x 128 dim x 2 B
= 20 KiB per position, i.e. **5.0 GiB at 262144** — enough to kill the first
draft graph capture with `ROCm error: out of memory` AFTER the model loaded
and `/health` answered (measured 2026-09-16). Read `sliding_window` out of
the drafter's GGUF header before sizing any spec-decode deployment, and
prefer a skewed `-ts` over a context cut when the full window matters.
DFlash on llama.cpp:
```
@@ -209,6 +254,151 @@ docker run --rm --name vllm-qwen36-dflash \
History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
## A/B protocol on a live llama.cpp endpoint
Verdicts are **traffic-shaped**: a proposer can only draft text it has seen,
so a single prompt shape answers the wrong question. Use fixed shapes:
| Shape | Prompt | Who it can favour |
|---|---|---|
| prose | 1k filler + "reply with one sentence" | model-backed drafters only |
| echo | ~1.5k passage of UNIQUE lines + "repeat verbatim" | ngram (output IS the prompt) |
| structured | repetitive output, no prompt overlap | ngram, if spans really repeat |
| identity | short deterministic question | controls for lossless verification |
Design the echo passage so no long span repeats INSIDE it — otherwise you
measure the matcher against itself. For `structured`, give each line a unique
id: repeated lines make it a negative control, not a repetition test.
Protocol:
1. Stop the unit (`sudo systemctl stop <svc>`), `pkill -x llama-server`; wait
for the port (a retired predecessor's published-port proxy leaves the
endpoint port in TIME_WAIT — the bind fails with address-in-use long after
the unit stopped).
2. Per config: write an env file with the overrides, start the serve script
with `FRANK_*_ENV_FILE`, poll `/health`, measure. Ready-made harness:
`scripts/bench-spec-ab.py <env-file> <serve-script>`.
3. **Restore the unit in a `finally`** and shout if it does not come back
healthy. A crashed harness that leaves production down is worse than no
harness.
4. **Prove losslessness**: one fixed prompt at temp 0 to every config, diff
the greedy text. Speculation only buys speed; differing texts are a
correctness bug. Empty answers (`content: null` when reasoning ate the
budget) make the comparison vacuous — bench clients must fall back to
`reasoning_content`, and echo/structured prompts should disable thinking
per request: `chat_template_kwargs: {"enable_thinking": false}`.
5. **Sweep the depth, not just the mode** — draft depth is model-specific and
single-valued; deeper often loses (measured: at n=7 a third of proposals
survived, at n=3 more than half).
6. **Sweep the ngram window in the same run as the configs**: too short a
`--spec-ngram-mod-n-min/-n-max` proposes on ordinary prose fragments —
rejected, verification still paid (measured 43.9 → 36.0 tok/s).
Read the right counters — per request `timings.draft_n` / `draft_n_accepted` /
`predicted_per_second`; cumulative `/metrics`
`llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`;
per-task `draft acceptance = ... (N accepted / M generated)`. `draft_n == 0`
while tokens flow = the proposer is NOT engaged, whatever the flags say. A
zero-draft config is a useful negative control: it proves the other config's
drafts came from the proposer under test.
Harness hygiene: absolute script paths (a relative path plus `cd` to its own
dir double-counts the prefix); `pkill -x`, never `-f` (the pattern matches
your own command line); nested f-strings with backslash escapes are a
SyntaxError on the fleet's Python (<3.12 semantics) — build log lines with
concatenation. Report `decode tok/s` per shape plus acceptance and mean
accepted length; a config that wins on echo and loses on prose is a traffic
decision, not a winner. Wire the winner into the deployed env file with the
measured table as its justification comment; record losing numbers with "do
not switch without re-measuring".
The A/B vehicle for ANY single-GPU llama.cpp knob (batch/ubatch, KV type,
slot split, quant rung — not just spec flags) is the transient systemd
drop-in procedure in [[frank-large-benchmark]] and [[frank-small-benchmark]].
## Is it worth it? (arithmetic before deployment)
Answer against the INCUMBENT, not against zero.
**Step 0 — check whether the target already speculates.** Many engines ship a
drafter of their own; then your baseline is `A_incumbent / t_s`, not `1 / t_s`.
halogen-flash-server: MTP head ON by default — read its own accounting, e.g.
`serve_api: mtp 63 tok in 1.31s = 47.97 t/s | 33 rounds, commit 1.88/round`.
llama.cpp: `/metrics` counters and per-request `draft acceptance` lines.
**Step 1 — measure the two step times, never guess.** Target step `t_s` = 1 /
(client-visible tok/s ÷ commit-per-round); ask for the rate the operator
actually SEES (engine-internal decode-only numbers run ~1.7–2x optimistic).
Drafter step `d_s` from llama.cpp's per-task `eval time` lines when it is a
plain model on the same box.
**Step 2 — arithmetic:**
```
round_time = K * d_s + 1.1 * t_s # K drafts proposed, K+1 verified in ONE target pass
A = (1 - p^(K+1)) / (1 - p) # expected accepted tokens, p = per-position acceptance
rate = A / round_time # compare to the incumbent
break-even : A > rate_incumbent * (K*d_s + 1.1*t_s)
```
Worked example that decided a real case: `t_s = 67 ms` (25 tok/s, 1.67
commit/round), `d_s = 12 ms` (27B IQ3_S drafter on the 3090) → break-even
`p ≈ 0.70` at K=3, `≈0.68` at K=5; a realistic cross-finetune `p = 0.4–0.6`
gives 17–20 tok/s, i.e. a LOSS. A drafter only ~3x faster per token cannot
pay for K sequential forwards.
**Step 3 — classify the drafter by per-round cost:**
| drafter | cost per round | needs |
|---|---|---|
| plain model (`-md`/`draft-simple`) | **K sequential forwards** | 10–50x faster than the target |
| n-gram / prompt-lookup (`ngram-*`) | ~0, no model | repeated context (agent edit loops) |
| trained head (`draft-mtp`, EAGLE/DFlash) | **~1 pass** | the head must exist FOR THAT CHECKPOINT |
Trained heads are target-specific: EAGLE/DFlash consume the TARGET's hidden
states, so a dense-27B DFlash2 is useless for a 512-expert MoE trunk.
**Step 4 — acceptance reality checks.** Fleet datum: a trained z-lab DFlash2
at depth 7 accepted ~41 % on its matched target (`draft-dflash 7 → 43.5
tok/s`); treat 40–60 % as the honest band for a matched pair. Cheapest proxy
for `p` at temp 0 when both models are greedy and share a tokenizer: greedy
continuation agreement on identical prompts, two API calls, no weights moved.
llama.cpp can also synthesize the projection without any draft artifact:
`--spec-synth-rates P0,P1,...` / `--spec-synth-len L`. Speculation pays IN
PROPORTION TO HOW SLOW THE TARGET IS — the same 41 % gave 2.3x on an
18.8 tok/s layer-split target and a loss on a 25 tok/s one; fast targets are
usually sparse MoE, the worst case for drafting.
**Step 5 — check the attachment surface BEFORE the arithmetic.** Closed
engines refuse external drafters by design: halogen-flash-server's
`DRAFTER_CAP` gates `dflash2` on the loaded checkpoint and an unavailable
name is a deliberate 400 (a silent downgrade would make comparisons lie);
there is no device routing in its loopback token protocol, so a drafter can
never sit on another card. Pure llama.cpp DOES route the draft:
`--spec-draft-device/-devd` (default follows `--device`), `-ngld`,
`-otd`, `-ctkd/-ctvd`, `--spec-draft-n-max` (default 3), `--spec-draft-p-min`.
One process sees only the devices its backends were built with — CUDA+Vulkan
is the combination that shows a CUDA card AND an AMD iGPU (HIP+CUDA cannot
coexist); a CUDA-only build has no `Vulkan0` to draft with.
**Step 6 — cross-model pairs need a byte-identical tokenizer.** Speculation
compares token IDs, so draft and target must map id → same string. Verify
cheaply from the GGUF headers (first ~8 MB over HTTP, no full download):
parse `tokenizer.ggml.tokens`, print vocab size, merges, `pre`, and sha256
over the first 2000 tokens —
```
vocab=248320 merges=247587 model=gpt2 pre=qwen35
sha256(first 2000 tokens)=8619c68fe4466d3a # identical => ids are interchangeable
```
Identity is necessary, NOT sufficient — the drafter still has to be cheap and
accepted. Report in this order: does the target already draft; break-even
`p` at K=1,3,7 versus the incumbent; the realistic band and resulting rate;
whether the drafter fits (VRAM) and the attachment surface exists; the one
measurement that would settle it.
## Notes
- Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS.
@@ -225,4 +415,8 @@ History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 20
- [[Dankclaw System Specs]]
- [[Dankserver System Specs]]
- [[VLLM Overview]]
- [[deployment]]
- [[deployment]]
- [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in
- [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic
- [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands
- [[frank-large-benchmark]] and [[frank-small-benchmark]] — the single-GPU env drop-in A/B vehicle
serverAppConfigs-docs/LLM/GGUF Compatibility.md
+123
View File
@@ -0,0 +1,123 @@
---
type: reference
title: GGUF Compatibility and Runtime Lock-in
description: How to answer "will engine X load this GGUF?" from the file's header alone, why a custom tensor type forces the runtime, and the discipline for pinning a fork build that can actually serve the file.
tags: [gguf, llama.cpp, forks, quantization, speculative]
timestamp: 2026-10-05T00:00:00-07:00
---
# GGUF Compatibility and Runtime Lock-in
**The law: a GGUF's tensor-type histogram and ftype pick the runtime — read the
header before promising a file to any loader.** Stock llama.cpp, Ollama and LM
Studio refuse custom types at load (`unknown tensor type`); a fork-private type
means exactly one engine can serve the file. Measured on this fleet repeatedly:
`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is ftype 115 with 311 tensors of type 103
(`GGML_TYPE_Q8_0_ROCMFPX`) — only charlie12345/ROCmFPX reads it. The full type
tables live in [[Llama Cpp Runtimes]] and [[Llama Cpp Code Diff]]
(ggml_type 100–108, ftype families 100–106 / 110–113 / 118).
## 1. Read the header, do not guess
Parse the GGUF KV section and histogram the tensor types plus
`general.architecture.file_type` — one python read over the first few MiB, no
model load. Counting types out of the KV section settles a runtime argument in
minutes; repo names and model cards do not.
## 2. Compare routing variants by tensor mix, not by name
Two files of the same quant can differ in how many tensors stay at plain Q8_0
(measured: 187 vs 2 output-head tensors in an `_AGENT` variant of the same
size). That difference is what moves spec-decode acceptance (0.953 vs 0.911)
— the "same size" file is not the same file.
## 3. Pin the build by commit SHA
A fork that can read the file today can still crash mid-load: the 2026-08-25
ROCmFPX builds read the ftype-115 file and then SIGSEGV in
`ggml_compute_forward_set_rows`. Record the SHA in the build script
(`scripts/build-rocmfpx.sh` pins `36e9acd`) and check it out explicitly — never
a branch.
## 4. Header diff method (candidate vs known-good)
Answer "will engine X load this GGUF?" by diffing the candidate's header
against a file the engine **already loads** — never by filenames or model cards;
filenames lie about quantization and about MTP/vision heads.
1. **Get the bytes**: both files, from byte 0, at least 64 MiB. Tokenizer KVs
(151k token strings + merges) routinely eat 5–8 MiB, so a small range
desynchronises the parser. True size from `curl -sIL` → `x-linked-size`
(a bare HEAD `Content-Length` can lie):
```sh
curl -sL -r 0-67108863 -o cand.bin \
"https://huggingface.co/<owner>/<repo>/resolve/main/<file>.gguf"
ssh <host> 'head -c 67108864 /path/to/known-good.gguf' > good.bin
```
2. **Parse — and lock the parser against the known-good file first.** Layout
(v2/v3): `magic[4]` `version u32` `n_tensors u64` `n_kv u64`, then KVs, then
tensor infos. Strings are u64 length + bytes. **The value-type enum is NOT
llama.cpp's `gguf.h`:** in the files seen (2026-09-28, unsloth + ajgazin UD
GGUFs) type **8 = STRING, 9 = ARRAY**; llama.cpp's header has STRING=2,
ARRAY=3 and desynchronises immediately. Working table:
```
0 u8 1 i8 2 u16 3 i16 4 u32 5 i32 6 f32
7 bool 8 STR 9 ARRAY(u32 et, u64 n) 10 u64 11 i64 12 f64
```
Validate before trusting: the parse is right iff the tensor-info block ends
somewhere sane and names look real (`blk.0.attn_q.weight`,
`token_embd.weight`, `output_norm.weight`). EOF throw = wrong enum or
offset — do not work around it.
3. **Diff what engines actually bind on**: `general.architecture`; tensor
count; tensor name list in order; per-tensor `ggml_type` and dims; geometry
KVs (`<arch>.block_count`, `.embedding_length`, `.feed_forward_length`,
`.attention.head_count[_kv]`, `.rope.*`, `.ssm.*`, `.context_length`,
`.nextn_predict_layers`); MTP tensors (`nextn` in the name); vision tensors
(`visual`, `v_` prefixes). A same-family fine-tune quantized by the same
tool usually comes out byte-identical in every one of these — that is the
answer.
4. **Then read the engine's source before calling a delta a blocker.** A
metadata difference only matters if the engine reads that key
(`grep -rn "file_type\|recurrent_layers\|full_attention_interval" <src>/`).
Worked example (gufo v0.1.1): `general.file_type` is deliberately not
consulted (UD-*_K_XL ships a stale ftype 15; the tensor table rules), and
`qwen35.attention.recurrent_layers` is never read (layer kinds come from
`full_attention_interval`) — both deltas were non-issues the header diff
alone would have flagged wrongly.
5. **Say what you did not do.** Header analysis is strong evidence, not a
load. State byte sizes, check free disk before proposing a fetch, and do
not download tens of GB unasked — pointing the engine at the file is the
definitive test.
**Report shape:** verdict first (loads / will not / loads-with-caveats), then
the side-by-side diff table, then only the load-bearing caveats: text-only vs
vision tensors, chat-template differences (matters for llama.cpp, not for
engines with compiled-in Jinja), license, disk headroom, and whether the base
model is abliterated/uncensored.
## 5. A/B a fork's spec decode by stripping the whole flag set
`--model-draft` without `--spec-type` segfaults at load (measured 2026-10-02,
core dump, no diagnostic). A baseline "no-draft" arm must remove `--model-draft`
AND every `--spec-draft-*` AND `--spec-type` together — the flags move as a set.
## 6. Before deleting a retired payload, read the rollback tag's env file
A rollback may need a small companion dir: a drafter living in a directory
named after an old quant looks like trash but is load-bearing. Same class as
the gufo payload removal of 2026-10-03, which turned that rollback into a
re-download.
## Related
- [[Llama Cpp Runtimes]] — which fork ships which custom types
- [[Llama Cpp Code Diff]] — the enum tables and code-level differences
- [[frank-large-rocmfpx-q8agent-mtp]] — the ftype-115 lock-in in production
- [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads
serverAppConfigs-docs/LLM/Llama Cpp Multi-Device Splits.md
+274
View File
@@ -0,0 +1,274 @@
---
type: reference
title: Llama Cpp Multi-Device Splits
description: What llama.cpp can and cannot express across devices, the no-collective layer-split path, split balancing by measurement, drafter KV sizing from GGUF headers, and heterogeneous placement triage.
tags: [llama.cpp, multi-gpu, layer-split, pcie, sizing, speculative]
timestamp: 2026-10-05T00:00:00-07:00
---
# Llama Cpp Multi-Device Splits
## 1. Expressibility law — classify the arrangement before running anything
llama.cpp expresses exactly two things across devices, and they are not
interchangeable:
| arrangement | how | expressible? |
|---|---|---|
| **Layer split** — layers 0..n on dev A, rest on dev B | `-sm layer -ts <a>/<b>` | **yes**, no collectives needed; works cross-vendor (CUDA + Vulkan measured 2026-09-16) |
| **Per-tensor placement** — attention+KV on A, experts on B | `-ot <regex>=<dev>` | **NO.** `--override-tensor` sets `has_tensor_overrides()`, a REQUIRED term of the scheduler's `pipeline_parallel` flag (only read site: `llama-context.cpp`, in the `pipeline_parallel = …` expression). That flag sets `n_copies = parallel ? MAX : 1` in `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a non-primary device has no backend and the graph aborts at `ggml-backend.cpp:941` (`pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE)` — `(NONE)` is the leaf's own op name). |
Failure modes observed (mainline `37b53fd`): per-tensor `-ot exps=Vulkan1`
with `-dev CUDA0,Vulkan1` → scheduler abort; layer-scoped `-ot` with a
cross-backend pair → model load fails with only `failed to load model`;
`-ot x=CPU` **works** because the CPU backend is last in the list — **the CPU
is the only legitimate partner in a heterogeneous split.** On a CUDA+AMD host
a card can be the trunk OR the offload host, never the complement of an
AMD-resident peer. Pure Vulkan changes nothing here — the gate is the
override, not the driver mix.
If the plan needs tensors on a specific device: stop, the answer is no for
any vendor mix. Expressible alternatives, in order: (1) two independent
endpoints, one model per device; (2) single-vendor layer split; (3) device +
CPU per-tensor offload (mainline pinning guard: `GGML_CUDA_NO_PINNED`; ik fork
was `GGML_CUDA_NO_PINNED_WEIGHTS` — without it a CPU-overridden tensor can
land in a pinned `CUDA_Host` buffer and page-lock ~108 GiB).
Layer-split arithmetic to check before promising: a whole layer carries its
experts (MoE) — compute GiB/layer, not per param-group; only some layers carry
KV (every 4th on the hybrid), so a share of layers is only a share of KV; big
non-repeating tensors (PLE tables) land on the main device and cannot move —
they cap how small the primary can get.
## 2. No collectives? The atomics wall and the layer-split answer
Decide with one line before designing:
```bash
sudo lspci -vv -s <card2> | grep AtomicOpsCtl # ReqEn- ⇒ no collectives, ever
lspci -tv # a Routing- bridge is a hardware wall
```
`ReqEn-` kills vLLM entirely — TP needs a per-layer allreduce AND PP needs a
device-backend `send`/`recv` of the hidden state
(`vllm/distributed/parallel_state.py`) — i.e. the same RCCL primitive that
fails on an atomics-less card. llama.cpp's `--split-mode layer` needs no
collective: layers distribute in order, only the boundary activation crosses
PCIe once per forward pass. Full probe/repair detail:
[[PCIe Link and Atomics]].
## 3. Build: pin by feature, bundle the runtime at the exact depth
- Pin the ref by FEATURE list (e.g. DFlash2 + qwen35 MTP on 2026-09-16 →
`ggml-org/llama.cpp#27342`); never float. Verify a fork build the same way
(`ls <venv>/…/spec_decode/ # dflash/ vs dflash2/`).
- `-DCMAKE_HIP_ARCHITECTURES=<gfx only>` — a sibling arch links, loads, then
segfaults. `-DGGML_HIP_RCCL=OFF` when there is no atomics path (only tensor
mode degrades to the butterfly fallback; layer split unaffected).
- **TRAP — `cp -r SRC DST/` flattens when `DST` does not exist** and drops one
directory level. rocBLAS/hipBLASLt look for device code at a FIXED depth
(`../lib/rocblas/library`, `../lib/hipblaslt/library/<arch>`); flattened,
the engine loads every GiB, answers `/health`, and dies on the FIRST request
with `Cannot read .../TensileLibrary.dat: No such file or directory for GPU
arch`. `mkdir -p DST && cp -r SRC/. DST/`, then
`patchelf --set-rpath '$ORIGIN/../lib' build/bin/llama-*`.
- Dual-backend build (CUDA + Vulkan is the practical pair; HIP+CUDA do not
coexist): `-DGGML_CUDA=ON -DGGML_VULKAN=ON` in one tree; Vulkan needs
`glslc` + libvulkan headers. **Device names are not what you expect:**
`--list-devices` lists the discrete card under BOTH backends (`CUDA0` AND
`Vulkan0`) and the APU LAST (`Vulkan1`) — a wrong `-ot` name makes
llama.cpp print the valid set and exit (self-diagnosing).
- What the existing build can do: `GPU_BACKEND_*=OFF` lines in its
CMakeCache.
## 4. Serve flags that matter on a split
```bash
HIP_VISIBLE_DEVICES=0,1 llama-server \
--device ROCm0,ROCm1 --split-mode layer --tensor-split 1,1 \
--gpu-layers 999 --fit off --flash-attn on \
--ctx-size <n> --parallel 1 --cache-type-k f16 --cache-type-v f16 \
--jinja --metrics --cache-prompt \
--spec-type draft-dflash --spec-draft-model <dflash2.gguf> \
--spec-draft-device ROCm1 --spec-draft-n-max 7
```
- `--fit off` + explicit `--ctx-size`: auto-fit must not resize a context set
on purpose. **`--parallel N` DIVIDES `--ctx-size`** — one slot at large ctx
is the concurrency policy, concurrent agent loops queue.
- Pin the drafter to the card holding the target's LAST layers
(`--spec-draft-device`); under tensor mode an unpinned draft joins the
meta-device KV allocation and aborts the KV build. `--spec-draft-n-max` must
be ≤ the drafter's `block_size - 1` (DFlash2: 8 ⇒ 7).
- Qwen3.8-family templates RAISE on `reasoning_effort=high` (500) — probe
low/medium/high/xhigh and map harness levels before declaring ready.
## 5. Split balancing: measure, never interpolate
**A `-ts` step is ONE LAYER, and hybrid stacks do not have uniform layers.**
qwen35 is 16 × (3 Gated DeltaNet + 1 Gated Attention): one step ≈ 0.43 GiB on
a DeltaNet boundary, ≈ 1.43 GiB on a full-attention one (weights + that
layer's own f16 KV) — measured on frank-medium. The gap can jump 2.9 GiB in a
single step; the best integer split can sit 0.2 GiB from balance while its
neighbours are 1+ GiB off. An affine fit reproduces the deployed point and
overstates gaps in between (`tests/frank_medium_unit.bats` encodes the fit as
a gate; treat a NEGATIVE fitted constant as a unit bug).
Knobs that move weight and REQUIRE a re-measure: `--ubatch-size` (compute
buffers are **~2x heavier on the output/drafter card** — 512→1536 added
~3.3 GiB there vs ~1.7 GiB elsewhere), a drafter swap (weights + draft KV),
`--ctx-size`/`--parallel` (KV). Total free VRAM is invariant across splits —
only the distribution moves; use that to sanity-check a suspicious row.
**Procedure (one split per load):**
```bash
WORK=/tmp/split.env; cp scripts/serve-<svc>.env $WORK
export FRANK_MEDIUM_ENV_FILE=$WORK
for ts in 39,26 38,27 37,28 36,29 35,30; do
sed -i "s|^TENSOR_SPLIT=.*|TENSOR_SPLIT=$ts|" $WORK
nohup bash scripts/serve-<svc>.sh >/tmp/split.log 2>&1 &
# poll /health (or process exit), then:
rocm-smi --showmeminfo vram | grep 'VRAM Total Used' # sample ~6 s after health
pkill -x llama-server; sleep 6
done
```
Load → sample → kill, NO requests sent (allocation only). Run under `nohup`
and poll from a separate call (a load is 30–90 s). Pick the smallest
inter-card gap that clears the per-card floor with room to spare; one `-ts`
step ≈ 1.5 GiB at ~6k batch / 262k ctx on 30 GiB cards — enough to bracket,
not to solve. Prefer balance: the busy card's headroom bounds the context
(`ctx_extra ≈ (floor − measured_free) / per-position cost on the busier
card`). A faulted split (`SDMA0 UTCL2 page fault` → `illegal memory access` →
`ggml_abort`) is OFF-LIMITS — re-measure, never re-tune; more headroom does
not prevent a boundary-copy fault.
## 6. Drafter KV: sized from the TARGET, bounded by its window policy
`common/speculative.cpp` builds the draft context with
`cparams.n_ctx = llama_n_ctx(ctx_tgt)` — **no flag caps it.** So:
```
draft KV = ctx_tgt × block_count × 2 (K,V) × kv_heads × key_length × dtype
```
…unless the drafter declares a sliding window. Measured at 262,144 target
ctx: DFlash2 (`sliding_window = 2048`) ≈ 40 MiB; a same-shape 5-layer drafter
with NO window = 20 KiB/position = **5.0 GiB** — a four-layer swing in the
balance point, and swapping them invalidated an already-balanced split.
**Failure signature:** the engine loads the whole model, answers `/health`,
and dies on the FIRST draft decode with `ROCm error: out of memory` /
`cudaMalloc failed` inside `common_speculative_impl_draft_dflash::process` —
a sizing failure, not a broken drafter; under a self-healing unit it presents
as a crash-loop. Read `sliding_window`, `block_count`, `head_count_kv`,
`key_length` from the drafter's GGUF header first. A compact reader (the
three format traps: no KV alignment — v3 alignment applies only to the tensor
section; arrays are `type(9) → element_type(u32) → count(u64)`, element type
FIRST; the enum is `7=BOOL 9=ARRAY`, 64-bit = 10/11/12, and 8=STRING/9=ARRAY
is the v2/v3 file order, NOT gguf.h's 2/3):
```python
import struct, sys
def rd(path):
f = open(path, 'rb'); assert f.read(4) == b'GGUF'
struct.unpack('<I', f.read(4)); struct.unpack('<Q', f.read(8))
nk = struct.unpack('<Q', f.read(8))[0]
def s():
n = struct.unpack('<Q', f.read(8))[0]; return f.read(n).decode('utf-8', 'replace')
SCALAR = {0:1,1:1,2:2,3:2,4:4,5:4,6:4,7:1,10:8,11:8,12:8}
FMT = {0:'<B',1:'<b',2:'<H',3:'<h',4:'<I',5:'<i',6:'<f',7:'<?',10:'<Q',11:'<q',12:'<d'}
keys = {}
for _ in range(nk):
k = s(); t = struct.unpack('<I', f.read(4))[0]
if t == 8: keys[k] = s()
elif t in SCALAR: keys[k] = struct.unpack(FMT[t], f.read(SCALAR[t]))[0]
else:
et = struct.unpack('<I', f.read(4))[0]; n = struct.unpack('<Q', f.read(8))[0]
if et == 8: keys[k] = [s() for _ in range(n)]
elif et in SCALAR: keys[k] = ('arr', et, n); f.read(n * SCALAR[et])
else: raise Exception('unsupported arr elem type %d' % et)
return keys
```
## 7. Heterogeneous placement triage (go/no-go with arithmetic)
- **Link:** `LnkCap` vs `LnkSta` at every hop; expect **idle-downshift** (a
Gen4 x4 dock idles at Gen1 x4, ~1 GB/s) — measure under load. Per-hop cost
vs hidden state (5120-dim f16 = 10 KiB/hop ≈ 1.3 µs at Gen4 x4): bytes are
almost never the constraint; **hop count is** — every boundary is a host
sync, and a per-op-split runtime can hit 100 splits/token at ~5 % SM. Read
`graph splits` from the loader log.
- **KV is usually the bottleneck:** `KV/token = kv_bytes_per_position ×
context` (attention reads the whole cache every token). Highest-leverage
placement: **KV on the fastest-memory device together with its attention
compute** — never KV alone over the link (GBs over 7.88 GB/s = hundreds of
ms/token). Check fit: `kv_positions × slots × bytes + attention_trunk ≤
VRAM`.
- **What placement never buys:** prefill is compute-bound and expert weights
are read once per batch — moving them buys nothing and adds a serial stage.
Calibration: Strix Halo + 3090, moving 12.5 % of expert mass to the GPU
gave decode +12 % but prefill 192 → 32 tok/s. The harmonic
`1/(f_big/BW_slow + f_fast/BW_fast)` reproduces the decode gain within ~2
points (assumes sequential stages — true at batch 1).
- **UMA check before assuming additivity:** `mem_info_vram_total` (BIOS
carve-out) vs `mem_info_gtt_total` (aperture onto SYSTEM RAM). A big GTT
window means the "GPU memory" IS the same LPDDR: adding a stage there adds
a compute engine, not capacity/bandwidth. Only discrete VRAM is additive.
- **Speculation as the "cheap" split:** bandwidth-free (token ids per round)
but needs the exact tokenizer for a plain drafter or a head trained for
that target; a plain drafter costs K sequential forwards (break-even needs
p ≈ 0.7+ at r≈0.4–0.5, K=3); compare against the INCUMBENT (a target
already committing ~1.7 tok/round on its own MTP head is a high bar); and
check the runtime accepts external drafters at all (closed engines 400
unknown drafter names by design). Details: [[DFlash DSpark Speculative Decoding]].
- **Hazards:** keep paged/sidecar tables where they are read from; keep mmap
ON for huge-embedding models (anonymous memory → cgroup kill); CUDA graph
capture can abort on unusual graph shapes; make the FIRST measurement a
single-device baseline of the runtime on the new device — a split's
arithmetic is worthless if the baseline loses to the incumbent; a split
consumes the fast device entirely — check what it was serving.
**Verdict ranking:** (1) two independent endpoints — often the real answer;
(2) one process, weights-resident, KV co-located with attention on the fast
pool; (3) hot-expert cache only if `misses × bytes × tok/s` fits the link and
overlaps — no mainstream runtime implements it; (4) name what does NOT help:
TP across unequal devices, KV on the far device, loading over the link,
prefill-motivated placements.
## 8. Harness syntax and verification
`llama-bench` separates a LIST with `/` and SWEEP values with `,`:
`-dev CUDA0,Vulkan1` is TWO single-device runs, never one two-device run;
`-ts 0.85,0.15` is two sweep values each becoming `[0.85, 0, …]` — device 0
takes everything. Write `-dev A/B -ts x/y`. Read the table's own `dev` column;
any placement-dependent number needs an independent memory check
(`nvidia-smi` sampling while loaded). (This artifact produced a wrong fleet
conclusion 2026-09-16 — "CUDA + Vulkan cannot serve one model" — with two
bogus OOMs that were really all-111-GiB-on-the-24-GiB-card.)
Verification battery: `/health`; both cards' memory ≈ half the weights each
(busier card also carries the drafter; compare `model_bytes/2 + drafter +
ctx×KV/2 + buffers`); raise ctx by N and watch BOTH cards rise by
N·bytes/2 — neither card's total may equal the model file (that arithmetic
rules out replication); `spec_decode_num_drafts_total` > 0 and a real
completion; a deep prompt (140k) proves a raised window is REAL, not
declared; repeat prompt → `timings.prompt_n` collapses (prefix cache live);
`/slots` shows expected `(id, n_ctx)` pairs; two concurrent requests with
`journalctl -o cat | grep 'slot print_timing'` proves drafting on BOTH slots
(default journal output truncates these lines).
Ops traps: a server started over ssh outlives stopping the runner (keeps the
port — `pkill -x <binary>` first); `systemctl start` of a root unit needs
sudo (polkit "Access denied" otherwise); old crash `ROCm error` lines live in
the journal forever — check `ActiveEnterTimestamp`/`NRestarts` before
treating a grep count as a live fault.
## Related
- [[PCIe Link and Atomics]] — the atomics wall, link probing, repair
- [[Llama Cpp Serving Sizing]] — slot-pool and KV pricing
- [[DFlash DSpark Speculative Decoding]] — spec types, chaining, A/B
- [[frank-medium-llamacpp-dflash2]] — the layer-split era runbook
- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the measured history
serverAppConfigs-docs/LLM/Llama Cpp Serving Sizing.md
+163
View File
@@ -0,0 +1,163 @@
---
type: reference
title: Llama Cpp Serving Sizing
description: Slot-pool sizing against a headroom band, per-device VRAM accounting, prompt-cache/state sizing and the QSA flash-attn fix — the measurement-first procedures for llama.cpp serving shapes.
tags: [llama.cpp, vram, kv-cache, slots, flash-attn, sizing]
timestamp: 2026-10-05T00:00:00-07:00
---
# Llama Cpp Serving Sizing
## 1. Slot-pool sizing against a headroom band
Goal: choose `CTX_SIZE`/`PARALLEL`/KV types so N conversations fit a GPU while
keeping a requested amount of VRAM free — from measurement, never arithmetic
guesswork.
1. **Fix the invariants.** Read the payload's KV geometry (attention layers
only, KV heads, head_dim, KV dtype bytes) and the per-slot rule
`n_ctx_slot = CTX_SIZE / PARALLEL`. A hybrid model with
`kv_unified = 'false'` gives each slot its own cache AND a fixed per-slot
state buffer — slot count is a memory term, not free.
2. **Two boots on the real card** (production stopped, per the bench law),
each near the target; record `nvidia-smi` used/total plus the boot log's
`n_slots`/`n_ctx_slot`: shape A = current/cheap, shape B = new demand.
`non-KV = used − (tokens × KiB/token)`; the two rows separate
weights/buffers/graphs from the new demand (a 918 MB projector bills
~1.4 GiB).
3. **Solve the band:** `KV_tokens ≤ (card − non-KV − headroom_low) /
KiB/token`, round the pool DOWN to a multiple of `PARALLEL × 4096` so every
slot gets an equal integer window. Boot and measure; if outside the band,
walk one step — never re-derive from a formula.
4. **Propagate the per-slot number — it is the client contract:**
`models.yml` `contextWindow` = `n_ctx_slot` (NOT the model's native window),
or long conversations die with `400 exceed_context_size_error` instead of
compacting; compaction/summary budgets sized to the old window get
re-checked; endpoint docs and the rollback table move too.
5. **Prove behaviour:** boot-log slot count, `nvidia-smi` free in band, N
concurrent requests answered in parallel with N+1 queued, and — if a
projector ships — a solid-color image probe through an **`image_url`
content part** expecting the exact color. The legacy top-level `images`
field is silently ignored, so a probe that uses it measures a
hallucination, not vision.
Gotchas: shorter slots can make decode FASTER (~5 % measured); the KV pool is
allocated whole at boot so post-boot `memory.used` IS the ceiling; record both
measured rows in the service `.env` comment block and mirror the non-KV price
in the bats budget gate so a shape that busts the band fails the suite instead
of the boot.
## 2. Per-device VRAM accounting ("where did the memory go?")
First decide which kind of memory is in question:
| device | kind | read it with |
|---|---|---|
| RTX 3090 (Dankclaw) | discrete GDDR6X | `nvidia-smi --query-gpu=memory.total,memory.used,memory.free` |
| gfx1151 iGPU (Dankclaw) | **UMA** — "VRAM" is host RAM | `/sys/class/drm/card1/device/mem_info_gtt_{total,used}` (114 GiB) vs `mem_info_vram_*` (2 GiB carve-out) |
| R9700s (Dankserver) | discrete | `rocm-smi` figure (64 GiB, SVM-backed) |
Per-process truth on NVIDIA:
`nvidia-smi --query-compute-apps=pid,process_name,used_memory`. Never add the
iGPU's GTT to host RAM as a separate pool; `free -g` is the arbiter for
co-tenants.
- **Engine ledger first.** Halogen prints weights pinned (+ remaining 2 MiB
contiguous blocks — the honest fragmentation signal: 240 blocks/12.3 GiB
after reboot vs 475/14.7 GiB before means the model did not come back into
the same hole), device budget, KV pool/working split, "left for everything
else". llama.cpp prints at `llama-model.cpp:1827` (per-backend model buffer)
and `llama-context.cpp:692` (compute buffer) — but **`llama-bench`
suppresses loader INFO and a systemd unit did not put them in the journal**
(verified `grep -c` = 0). Recipe: one launch through `llama-cli`/`llama-server`
keeping the first ~200 lines.
- **Otherwise compose from tensor sizes.** Group the GGUF tensor table by
role (Flash-Next example: 75.0 GiB experts at 1.5625 GiB/layer, 32.8 GiB PLE
table, 3.2 GiB everything else), KV ≈ 2 KiB/position per f16 attention
layer, × derived context. Vendor delta is real: same layers+KV cost 12.80
GiB under Vulkan vs 14.99 GiB under CUDA (~3.3 vs ~1.1 GiB of
graph/compute pool).
- **Traps:** mmap'd PLE is reclaimable page cache; `-ot` makes it pinned —
"move layers to the card" converts reclaimable cache into device memory and
frees nothing for co-tenants. A served model's footprint ≠ weight size
(frank-small: 22.6 of 24.1 GiB for an 11.29 GiB model). Do the capacity
subtraction before designing: 12 attn layers × 1.62 GiB + 200k f16 KV =
28.6 GiB > 24.1 GiB usable → KV-in-VRAM on that pair only reaches ~64k.
- **Context lengths: derive, never remember.** `llama-bench` computes
`n_ctx = n_prompt + n_gen + n_depth` (defaults 512/128/0, type_k/v F16,
flash_attn AUTO, split_mode LAYER). `-p 512,16384 -n 64` = contexts
576/16,448; a plain `tg` row has ~zero depth — `-d` is the only thing that
puts KV in the picture.
- **Label every number** measured / derived / inference, and state the context
each figure was taken at — 12.8 GiB at 65,568 tokens is not the same claim
as at 200k.
## 3. Prompt cache and state sizing
`--cache-ram` (MiB, default 8192) is HOST RAM for saved prompt states —
unpinned, on demand, moves no VRAM. `--ctx-checkpoints` (default 32) is a
different mechanism: per-slot KV snapshots; `0` disables them.
**State law** (compute before trusting any cap): a state is the serialized KV
of the whole sequence + constant recurrent state + the draft context's KV:
`per_token = 2 × n_full_attn_layers × n_kv_heads × head_dim × bytes (K and V) +
draft_per_token`. Measured on frank-medium (qwen3.5 hybrid, f16, MTP draft):
16×4×256×2×2 = 64 KiB/token + 4 KiB draft = **68 KiB/token**; a full
196,608-token slot state ≈ 12.8 GiB. Cross-check against the journal's
`prompt state size` lines — the two must agree.
**The journal is the instrument:** `exceeds cache size limit … skipping` =
cap binding in production; `making room … removing oldest` / `cache size
limit reached` = eviction, and the printed size proves the active cap. Do NOT
claim a raise is proven by `prompt_tokens_cached_total` rising — on exact
repeats that counter is 0 by construction.
**Hybrid + zero-checkpoint trap:** for a hybrid (DeltaNet/SSM + attention)
model, `seq_pos_min` sits at the END of the slot's own prompt (a recurrent
cell stores only the last position), so with `--ctx-checkpoints 0` there is no
checkpoint to fall back on and the server logs `forcing full prompt
re-processing … (likely due to SWA or hybrid/recurrent memory)`. Consequences:
intra-slot reuse only survives when the request echoes the assistant's
previous reply verbatim (a pure user resend is always a full re-prefill even
at f_sim 1.000); the RAM prompt cache becomes the ONLY carrier of a
conversation state ACROSS slots, so a cap that refuses deep states costs
minutes of re-prefill whenever slots are reassigned; state restoration only
runs via the LRU path (or LCP with f_keep < 0.5) — a plain prefix hit never
calls `prompt_load`.
**Verify a change:** live argv carries the flag (not the deploy log); `/health`
200 + real completion; VRAM unchanged and host RAM mostly free; no new
`exceeds cache size limit` lines, with a `making room` entry larger than the
old cap as direct proof. If the ceiling itself was not exercised live, say so —
crossing it needs a >120k-token conversation and must not run beside a live
service (bench law).
## 4. QSA flash-attn assert fix (qwen4exp)
`--flash-attn on` aborts graph build for qwen4exp models:
`GGML_ASSERT(a->type == GGML_TYPE_F32)` at `ggml.c:5371` through
`build_attn_qsa`. Root cause: `build_attn_qsa` calls `ggml_fill` on the
F16-cast kq mask returned by `get_kq_mask()` when flash attn is on, but
`ggml_fill` requires F32.
Fix (mirrors the MLA sparse path in `llm_graph_context::build_attn`): run the
top-k mask surgery on the F32 base mask (`inp->self_kq_mask`), cast to F16
only at the attention call when `flash_attn` is enabled
(`kq_mask_attn = cparams.flash_attn ? ggml_cast(ctx0, kq_mask_top_k,
GGML_TYPE_F16) : kq_mask_top_k`). The patch lives at
`scripts/patches/qwen4exp-qsa-f32-mask.patch`.
Only the charlie12345/ROCmFPX build-qwen38 (PR #27742 qwen4exp merged) takes
it; CIRU, upstream and stock builds do not. Validate live: load with
`--flash-attn on`, health 200, a real completion finishing `"stop"`, journal
clean of `GGML_ASSERT`/`cudaMalloc`/`missing tensor`, and flash-attn engaged
in the timings. Context cap for full-model residency on that shape stayed
32,768.
## Related
- [[GGUF Compatibility]] — what a payload is before it is priced
- [[frank-small-llama-mimo-q6k-2x200704]] — the current two-boot priced shape
- [[frank-medium-llamacpp-dflash2]] — where the 68 KiB/token datum was measured
- [[DFlash DSpark Speculative Decoding]] — the spec A/B protocol on llama.cpp
- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the multi-device side
serverAppConfigs-docs/LLM/PCIe Link and Atomics.md
+244
View File
@@ -0,0 +1,244 @@
---
type: reference
title: PCIe Link and Atomics
description: Link probing (idle downshift, path ceilings, CUDA/HIP bandwidth probes), the streaming-vs-compute ceiling arithmetic, and the PCIe-atomics wall that decides whether AMD multi-GPU collectives can exist at all.
tags: [pcie, rccl, atomics, multi-gpu, bandwidth, amd, nvidia]
timestamp: 2026-10-05T00:00:00-07:00
---
# PCIe Link and Atomics
## 1. Degraded or just idle?
The trap: an idle GPU reports `LnkSta: Speed 2.5GT/s (downgraded)` — the
driver downshifts to Gen1 for power. That is not a fault and not the slot's
fault. Never conclude from an idle reading.
1. **Path ceiling from the kernel (cheapest, most decisive):**
`sudo journalctl -k -b --no-pager | grep "available PCIe bandwidth"` →
`pci 0000:c4:00.0: 63.012 Gb/s available PCIe bandwidth, limited by 16.0
GT/s PCIe x4 link at 0000:00:03.1 (capable of 252.048 Gb/s with ... x16)`.
The "limited by" device names the **bottleneck** (usually a root port) and
its width — that is the number a swap could move; "capable of x16" is the
endpoint's OWN capability, and the gap between them is the story.
`dmesg` rotates; `journalctl -k -b` keeps this boot.
2. **Both sides separately:** `LnkCap`/`LnkSta`/`LnkCtl2` on endpoint AND root
port. `LnkCtl2: Target Link Speed: 16GT/s` on both ends = the platform
intends Gen4, so a Gen1 current speed is power management, not
misconfiguration. A root port whose `LnkCap` width exceeds the trained
width means the socket/adapter wires fewer lanes — **M.2 connectors are 4
lanes by pinout; an M.2 riser can never deliver more.**
3. **Prove the link under load (the step people skip):** on NVIDIA sample NVML
while the GPU works
(`nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.gen.hostmax,pcie.link.width.current,pcie.link.width.max`):
`gen.current` climbing 1 → `hostmax` = idle downshift, healthy; stuck below
`hostmax` under load = real degradation; `width.current < width.max` under
load = **physical lane limit** — no software touches it. On a busy
production GPU this is a usage smoke test, not a benchmark — never stop a
live service to read its link.
4. **Do not plan hardware from `dmidecode -t slot`** — consumer firmware
labels Ethernet/SD controllers as "M.2 slots" and advertises slots in PCI
domains that do not exist. Trust `lspci -nn | grep "PCI bridge"` and
`lspci -tv` (empty root ports still enumerate; an absent port is absent).
5. **"Would moving the card help?"** Compare the bottleneck device from step 1
with the target port's `LnkCap` width; two M.2 sockets = both x4 = a
no-op. Clean `DevSta`/`CESta` with a low trained speed means "wired
narrow", not "failing". Report ceiling, card capability, what the idle
reading meant, and whether any physical change moves the number.
Gen4 usable payload after 128b/130b: **x4 7.877 GB/s, x8 15.75, x16 31.5**.
## 2. The engine shape decides whether the link matters at all
| engine shape | what crosses PCIe per token | link matters? |
|---|---|---|
| streams weights (expert LRU, paging) | the missed weights (~GB/token) | **yes — decisive** |
| computes offloaded weights in RAM (llama.cpp `-ot exps=CPU`) | activations only (~KB) | no; host RAM bandwidth decides |
| everything resident | nothing | no |
**Ceiling arithmetic:** `link time/token = active expert bytes / measured
H2D rate`; `ceiling = 1 / link_time × resident-fraction saved (LRU hits)`.
Compare against the compute-in-RAM path at host RAM bandwidth (256 GB/s
LPDDR5X on the Strix Halo). **If the link is >10x slower than the memory
behind it (x4 is 31x here), streaming loses by construction and no tuning
rescues it; doubling the link only doubles a losing number.**
Worked example (Dankclaw 2026-09-16): Gen4 x4 measured H2D **6.72 / D2H 6.60
GB/s** (85 % of spec) — pinned H2D ≈ async, pageable within 1 %, i.e.
link-limited not staging-limited. A 3 MiB slab (the expert-streaming
granularity) = 474 µs. A3B MoE at Q4_K_M (~1.70 GiB active/token) → 272
ms/token ⇒ **3.7 tok/s** (4.6 with 20 % resident) vs measured **20.46 tok/s**
with experts computed on the CPU, 25–26.5 on the iGPU. The streaming design
loses 3–5x.
## 3. Measure the link: two probes, same mechanics
Stop **all** production services first (fleet bench law) — the 3090's VRAM
may be nearly full, and the probe contends for the very link it measures.
**CUDA probe** (`pcie-link-test.cu`): sweep {3, 16, 64, 256 MiB, 1, 2 GiB}
with `cudaMalloc` + `cudaHostAlloc`, event-timed; per size measure sync H2D,
async H2D, D2H, one single transfer, and pageable H2D. `3 MiB` is the row
that matters for expert streaming; big sizes give the ceiling.
**HIP probe for RDNA4** (`pcie_bw.hip`) — the ROCm landmines each cost real
time, so encode them:
- `hipGetDeviceProperties` SEGFAULTS on some builds; use `hipGetDeviceCount`
+ `hipSetDevice` only. First line: `setvbuf(stdout, NULL, _IONBF, 0);` — a
segfaulting process loses block-buffered stdout and hides the crash point.
- Build with an explicit arch: `HIP_TARGETS=gfx1201 hipcc -O2
--offload-arch=gfx1201 -o p p.hip`; compile with `| grep -E "error:"`,
never `| tail -N` (truncation hid a missing `#include <cstring>`).
- `hipMemset`/`hipDeviceSynchronize` are `[nodiscard]` — use or cast.
- Mount the whole `/dev/dri` into ROCm containers; individual `renderD*`
nodes enumerate but fail context creation.
- `rocm-smi --showmetrics`: `pcie_link_width` is trustworthy, but
`pcie_link_speed` reads the ASPM idle value and `pcie_bandwidth_acc/inst`
are usually `N/A` on consumer cards — live traffic generally cannot be
sampled.
```cuda
// pcie_bw.hip — sweep sizes, pinned vs pageable
#include <cstdio>
#include <cstring>
#include <cstdlib>
#include <hip/hip_runtime.h>
static double one(size_t bytes, bool pinned, int iters) {
void *dev = NULL, *host = NULL;
if (hipMalloc(&dev, bytes) != hipSuccess) return -1;
if (pinned) { if (hipHostMalloc(&host, bytes, hipHostMallocDefault) != hipSuccess) return -1; }
else { host = malloc(bytes); }
memset(host, 0x5a, bytes);
hipEvent_t a, b; hipEventCreate(&a); hipEventCreate(&b);
for (int i = 0; i < 3; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice);
hipDeviceSynchronize();
hipEventRecord(a, 0);
for (int i = 0; i < iters; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice);
hipEventRecord(b, 0); hipEventSynchronize(b);
float ms = 0; hipEventElapsedTime(&ms, a, b);
double gb = (double)bytes * iters / (ms / 1000.0) / 1e9;
if (!pinned) free(host); else hipHostFree(host);
hipFree(dev); hipEventDestroy(a); hipEventDestroy(b);
return gb;
}
int main(int argc, char **argv) {
setvbuf(stdout, NULL, _IONBF, 0);
if (hipSetDevice((argc > 1) ? atoi(argv[1]) : 0) != hipSuccess) return 1;
size_t sz[] = {2459136UL, 3UL<<20, 16UL<<20, 64UL<<20, 256UL<<20, 1UL<<30};
for (int i = 0; i < 6; i++) {
size_t b = sz[i];
int iters = (b >= (1UL<<30)) ? 10 : (b >= (256UL<<20)) ? 20 : 100;
printf(" %10.3f MiB | pinned %6.2f | pageable %6.2f\n",
b/1048576.0, one(b,true,iters), one(b,false,iters));
}
return 0;
}
```
**Reading it:**
- **Engine-vs-link is the diagnosis, not the raw number:** compare
`engine_effective_GB_s` to `link_GB_s` at the SAME transfer size. Engine at
~40 % of the link at the same granularity = the wire is not the problem and
no hardware change will help.
- Achieved GB/s is itself the proof of the trained link state (6.7 GB/s only
exists at Gen4 x4) — no separate under-load `LnkSta` sample needed.
- **pinned ≈ pageable ⇒ pinning is not a lever.** async ≈ sync ⇒ link-limited
(the usual "pageable costs 2x" rule does not apply at this link).
- Small-transfer penalty (2.345 MiB: ~106 µs vs ~78 µs ideal, ~21 %) is
per-transfer overhead — recoverable only by BATCHING transfers, not by a
faster link. (Dankserver 2026-09-29: bulk 22.7 GB/s at the 2.344 MiB slab =
91 % of Gen5 x8, while the engine's effective read path ran ~9.5 GB/s — the
gap is the fork's gather software path, not the wire.)
- Both GPUs symmetric ⇒ no cross-device rebalance is available; measure both.
- Don't quote a derived line from inside a probe without hand-checking it —
a printf format/argument mismatch in one of these printed a nonsense
ceiling. GiB→GB before dividing.
## 4. The PCIe-atomics wall (AMD multi-GPU collectives)
**Symptom:** multi-GPU AMD host fails TP/PP; the failure surfaces as
`hipLaunchKernel: hipErrorIllegalState` for `ncclDevKernel_*` — nothing in
vLLM/RCCL/PyTorch names atomics. Get the real message:
```bash
AMD_LOG_LEVEL=3 <the failing process>
# → "Pcie atomics not enabled, hostcall not supported" → "AQL dispatch failed!"
```
**Cause chain:** RCCL kernels carry a hidden hostcall-buffer parameter; ROCr
refuses to fill it without PCIe atomics, which it reads **once at driver
probe** (`rocclr/device/rocm/rocdevice.cpp` `checkAtomicSupport()` — no else,
no override). **No env var, module param, or sysfs toggle exists** —
`setpci -s <gpu> CAP_EXP+28.w=0440` (DCTL2 bit 6) changes nothing at runtime.
**Check the CHAIN, not the card:**
```bash
sudo lspci -vv -s <gpu> | grep -E "AtomicOpsCtl|LnkSta|LnkCap" # want ReqEn+
lspci -tv # every hop
sudo lspci -vv -s <bridge> | grep AtomicOpsCap # want Routing+
```
`Routing-` is a read-only hardware capability. Typical AM5 layout: one x16
slot on CPU root ports (atomics fine), the rest behind the chipset (usually
not). Bandwidth for TP is small anyway: ~10 MB per token-step for a 27B at
batch 4 ≈ 1.4 ms at Gen4 x4 — compute-bound, not interconnect-bound.
**Probe matrix (bare 2-rank `torch.distributed`, one process per GPU, 25 s PG
timeout) — one probe answers several design questions:**
| Op | CPU-lane card | chipset card | Decides |
|---|---|---|---|
| `broadcast`/`all_reduce`/`barrier` | OK | FAIL | TP is dead |
| `send`/`recv` (P2P), in ISOLATION | hang | FAIL | **PP is dead too** — vLLM's PP handoff IS this primitive |
Test `send`/`recv` alone (a prior failed op leaves the group broken). Run the
probe from a REAL file (`torch.multiprocessing.spawn` re-executes it; piping
into `python3 -` fails with `FileNotFoundError: '/<stdin>'`). `world_size=1`
always succeeds — **one-card success proves nothing.**
**SAFETY — non-negotiable:** never run a GPU experiment beside a serving
unit: `sudo systemctl stop <svc> && sudo systemctl mask <svc>` first (mask,
never disable — disable deletes a relinked unit file), or use a host that
serves nobody. A hung AQL dispatch can wedge the GPU driver; a probe beside a
live service on 2026-09-16 preceded a hard host down (no ARP, no ICMP from
two subnets, power cycle required). Telling off from wedged afterwards: `ip
neigh show <ip>` from a peer — `FAILED`/`INCOMPLETE` = off/NIC dead; a wedged
kernel usually still answers ARP. Validate your own probe's naming before
reporting an outage (`gitea` vs `gitea-server`).
**Fixes, ranked:**
1. CPU-attached M.2 → PCIe x4 riser (~$30) — puts the card on the CPU root
complex; x4 suffices for TP.
2. Board whose CPU x16 bifurcates x8/x8 (done 2026-09-20 on dankserver — both
R9700s on root ports `00:01.1`/`00:01.3`, every bridge `Routing+`, both
endpoints `ReqEn+`).
3. llama.cpp `--split-mode layer` — no collectives at all ([[Llama Cpp Multi-Device Splits]]); costs the GGUF conversion plus paged attention /
continuous batching / spec decode.
4. Independent TP=1 replicas + routing proxy — cards never share one model.
**Closed software routes (do not re-litigate):** vLLM's IPC custom allreduce
is gated by a CUDA-sm-keyed table (`{9.0, 10.0, 10.3, 10.7}`; ROCm reports
12.0 → off on RDNA4 by code limit) — and patching it is insufficient because
TP init's `broadcast`/`barrier`/logits `all_gather` have no non-RCCL path. A
fork wrapping `.all_reduce` (e.g. radiance `RADIANCE_USE_R4D_AR` one-shot
P2P, gloo for IPC-handle exchange) does not intercept
`PyNcclCommunicator.__init__`'s warmup allreduce. NCCL symmetric memory needs
`ncclDevComm*` (many builds ship the version string without the device API).
P/D disaggregation needs a KV connector AND coordination collectives. gloo is
CPU/TCP — unreachable for per-layer TP.
**Open, unverified:** rebuilding RCCL so kernels stop declaring a hostcall
buffer (`-mprintf-kind=buffered` is the usual claim) — counter-evidence: no
live `printf`/`assert` in RCCL device source. Test path: rebuild, dump a
kernel's declared args (`roc-obj-extract` + `llvm-readelf --notes`), look for
`hidden_hostcall_buffer`; if gone, retry the 2-rank probe (service stopped).
## Related
- [[Llama Cpp Multi-Device Splits]] — the layer-split path around the wall
- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — measured history
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the live gather economics
serverAppConfigs-docs/LLM/RDNA4 Container Serving.md
+250
View File
@@ -0,0 +1,250 @@
---
type: reference
title: RDNA4 Container Serving
description: Bringing a model larger than VRAM onto Dankserver's 2x R9700 (gfx1201) with the tcclaviger vLLM stack — quant-class gating, fetch verification, container GPU-init triage, memory budgeting and the expert-offload plan.
tags: [rdna4, gfx1201, r9700, vllm, expert-offload, podman, rocm]
timestamp: 2026-10-05T00:00:00-07:00
---
# RDNA4 Container Serving
Bring-up path for Dankserver (2x Radeon AI PRO R9700, gfx1201, 31.86 GiB each =
**63.7 GiB real VRAM**, 246 GiB RAM, 7700X) with
`docker.io/tcclaviger/vllm` (vLLM 0.29.0.dev, GLM/Qwen-specialised RDNA4 build
with native MoE quant kernels and a first-class expert offload planner). The
tuning history and current deployed shape live in
[[dankserver-qwen38-fp8-expert-offload-tuning]]; this page is the bring-up law.
## 0. Know the box — and its phantom third GPU
`torch.cuda.device_count()` returns **3**; summing `total_memory` overstates
the box as 186.7 GiB. The third device is the 7700X's integrated Vega
(gfx1036): torch reports ~123 GiB for it, `rocm-smi` reports 0.50 GiB,
allocating on it moves system `MemAvailable` 1:1 (it is GTT over DDR5, not
dedicated VRAM), and it dies with `hipErrorInvalidKernelFile` when the image
ships no code for that arch. **Never sum device `total_memory`; never count
the iGPU as headroom or a workload.**
## 1. Preflight: is the quant class EXECUTABLE, not merely present?
Expert offload is mandatory when weights exceed VRAM, and it requires the
checkpoint's experts class to have BOTH a compiled kernel AND an
`offload_recipe` ("a pinned expert has no host copy, so there is no
read-through fallback" — startup aborts otherwise). Check BEFORE downloading
anything:
```bash
podman run --rm --device /dev/kfd --device /dev/dri --device /dev/mem \
-e HIP_VISIBLE_DEVICES=0,1 --entrypoint bash $IMAGE -c '
nm -D /app/r4dhip/r4d.so | grep -oE "gemm_moe_[a-z0-9_]+" | sort -u
grep -rl offload_recipe /opt/vllm/lib/python*/site-packages/vllm/model_executor/layers/fused_moe/'
```
| quant | kernel in `r4d.so` | experts class | verdict |
|---|---|---|---|
| FP8 | yes | `Fp8HipMoEExperts` | **serves** |
| MXFP4 / DSFP4 | yes (`gemm_moe_mxfp4a8_*` / `dsfp4a8_*`) | `R4dMxfp4MoEExperts` / `R4dDsfp4MoEExperts` | **serves** |
| W4A16 | **no** `gemm_moe_w4a16_*` | `R4dW4a16MoEExperts` exists | silently falls back to generic `TritonWNA16Experts`, no recipe → startup halts |
| RFA | none | no class at all | out |
Also confirm the architecture routes natively with NO overlay
(`ModelRegistry.get_supported_archs()` for `Qwen4Exp`/`Glm5`). The stock image
natively carries Qwen3.8/Qwen4Exp and GLM-5.x including MTP heads. **Do not
apply vendor overlays** against a fast-moving pre-release image — the breakage
misleads (a missing symbol in the overlay's OWN copy of a module masks the
real cause, then moves to another symbol each patch;
`cannot import name 'checkpoint_has_lm_head'` = overlay clobbered
`transformers_utils/config.py`). The image ships quantizers for
`rfa|rfi|mxfp4|mxfp4ocp|paro5` under `/app/tools/quantize.py`
(`--fp8-attn --quant-only --dry-run` first) if a re-quant is needed — note it
can run MXFP4/DSFP4/FP8 but only PRODUCE rfa/rfi/mxfp4/paro.
## 2. Fetch with verification, always
Gated repos: token never in argv (`pgrep -a curl` prints command lines).
Build a mode-600 curl config once:
```bash
{ printf 'header = "Authorization: Bearer '; cat ~/.cache/huggingface/token; printf '"\n'; } \
> ~/.cache/huggingface/curl.conf && chmod 600 ~/.cache/huggingface/curl.conf
curl -K ~/.cache/huggingface/curl.conf -f -sS -L -C - -o "$out" "$url"
```
`--fail` is MANDATORY — `curl -o FILE URL` exits 0 on HTTP 401/403 and writes
the error body as a "successful artifact" (a gated repo once yielded three
153-byte shards with identical sha256s and a cheerful DONE banner). `-C -`
resumes torn shards; HTTP/2 cancels leave silently truncated files, so
re-verify EVERY shard's byte count against the Hub manifest (and the LFS
sha256) before serving. The AGENTS.md Secret Rules carry the token-hygiene
law; [[frank-medium-model-swap]] has the same detail for GGUFs and
`.partNofM` fragments.
## 3. Serve — the known-good invocation and its traps
```bash
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
podman run -d --name <ctr> --ipc=host --ulimit memlock=-1:-1 \
--cap-add SYS_PTRACE \
--device /dev/kfd --device /dev/dri --device /dev/mem \
-e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
-v /home/dank/models/<model>:/models:ro \
$IMAGE /models --served-model-name <name> --port 7770 \
--tensor-parallel-size 2 --max-model-len 65536 \
--kv-cache-memory-bytes 2147483648 \
--max-num-seqs 4 --gpu-memory-utilization 0.95 \
--enable-prefix-caching --trust-remote-code
```
- **Entrypoint already execs `vllm serve "$@"`** — pass the model path as the
first argument and do NOT pass `serve` (podman then tries to pull `serve`
as an image name). `-e VAR=1` must come BEFORE the image name.
- **Mount the whole `/dev/dri`, never individual render nodes** (§4).
- `--ipc=host` and `--shm-size` are mutually exclusive in podman.
- `podman run -d` without `-a` sends container stdout nowhere — read
`podman logs <ctr>`; and vLLM writes `Application startup complete` to
**stderr**, so a `podman logs X 2>/dev/null | grep -q` readiness gate never
fires.
- Start at 65536, not the native ceiling; prove the plan, then scale.
- Load takes ~5 min with `/health` refusing the whole time — do not kill the
process (deploy.sh waits 20 min).
## 4. Container GPU-init triage (ordered)
1. **Enumerate-vs-context trap (most common):** individual
`renderD*` nodes give HIP the render nodes WITHOUT the primary `card*`
nodes — it enumerates agents (`device_count()` returns N) and then refuses
every context: `RuntimeError: No CUDA GPUs are available` on the first
real op. A/B-confirmed 2026-09-19. Rule: `--device /dev/kfd --device
/dev/dri --device /dev/mem`; restrict GPUs with `HIP_VISIBLE_DEVICES`,
never with device nodes.
2. **`device_count()` is not evidence.** Prove usability with a real op
(`(a@a).sum().item()` on a fresh `torch.randn`), then repeat via
`mp.spawn`, `mp.fork`, and `subprocess.run([sys.executable, ...])`, and
cross-check `rocm-smi --showmeminfo vram`.
3. **memlock rlimit → a MISLEADING CUDA OOM.** Host default 8192 KB; a pin
larger than the limit fails as
`torch.AcceleratorError: CUDA error: out of memory` raised from
`pin_memory()` — it names the accelerator but the fault is host pinned
memory. It cannot be raised from inside a container or even from root's
own shell (`Operation not permitted`); only PAM applies it at login, so
fixes only affect NEW sessions:
`/etc/security/limits.d/99-memlock.conf` with
`* - memlock unlimited` + `root - memlock unlimited`; the container needs
`--ulimit memlock=-1:-1` to inherit. Discriminating test after the fix:
the container reports `unlimited` AND small pins succeed; if pins still
fail, memlock was never the cause.
4. **`/proc/meminfo` TAB trap:** the key/value separator is a TAB, so
`grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2` returns
EMPTY and a following awk prints `0.0` — a false "host out of memory" on a
host with hundreds of GiB free, which invites aborting something healthy.
Tab-safe: `awk -v k='MemAvailable:' '$1==k {printf "%.1f", $2/1048576;
exit}' /proc/meminfo`, cross-check `<= MemTotal`, refuse to print a zero.
Use `scripts/dankstatus.sh` instead of hand-rolling.
5. **Silent-parse traps:** `grep -c PATTERN file || echo 0` yields TWO lines
(grep prints 0 AND exits 1) — `[: 0\n0: integer expected`; use
`| head -1` or `|| true`. `systemctl is-enabled` can emit multiple lines.
`rocm-smi --showmeminfo vram --csv` column order is `device, VRAM Total
Memory (B), VRAM Total Used Memory (B)` — reading $3 as total/$4 as used
yields 0.00; filter to `^card[0-9]+$`.
6. **Guard the box:** long loads with co-tenants can drive RAM to 0 — abort
before the OOM killer (`MemAvailable < 4 GiB → podman rm -f`, exit 1).
7. **Drop page cache before a large load** (`echo 3 > /proc/sys/vm/drop_caches`)
and confirm the drop was actually in effect for the run before treating a
failure as a capacity wall — a stale "it doesn't fit" from a contaminated
run is expensive to unwind.
## 5. Memory budgeting: the levers are not where they look
The planner builds `budget = total × util`, subtracts a KV reserve, and gives
the REMAINDER to the expert arena. Consequences, all measured:
- **`--gpu-memory-utilization` grows the ARENA, not the KV pool** (KV stayed
0.64 GiB at util 0.95 and 0.98). The arena is the only elastic term.
- **Lowering `--max-model-len` is the WRONG lever and backfires:** a smaller
context shrinks the reserve, handing the space to the arena — measured 65536
→ 0.64 GiB KV, 32768 → 0.34 GiB KV.
- **Hybrid-attention undercount:** the planner's per-token KV figure (9190
B/token for Qwen4Exp) misses the large fixed per-sequence Mamba state
(~2x), so the reserve is always short and the arena silently covers it.
Fix by pinning the pool: `--kv-cache-memory-bytes <BYTES>` (2 GiB →
113,595 tokens / 3.47x concurrency at 32k). For long context,
`--kv-offloading-size` (GiB) spills KV to host.
- **`--expert-offload-mem` caps the HOST side, not the VRAM arena**
(`cap = int(x * GIB)` against `host_memory_available()`). Do not reach for
it to fix a VRAM problem; to grow the arena, shrink the KV pin or raise
util (see [[dankserver-qwen38-fp8-expert-offload-tuning]] for the
util=1.0 unbootability law).
**Read the plan — it is the real acceptance test:**
```
expert offload: AUTO-ENABLED, weights 62.54 GiB per rank exceed the 30.27 GiB budget (TP2)
budget (total x util) 30.27 GiB
non-expert weights 6.28 GiB
KV reserve 0.36 GiB <- sanity-check against the model's own per-token figure
CUDA graphs 0.66 GiB
runtime overhead 7.60 GiB
expert cache space 15.36 GiB 6712 slots of 2.344 MiB
host-backed experts/rank 52.74 GiB 23040 experts
host: 105.48 GiB across 2 ranks of 160.89 GiB usable
```
Host fits when `host_total < usable`. The PLE n-gram table rides memfd and is
SHARED across TP ranks.
**Loading mechanism** (for reasoning about peak RAM): weights never materialise
in VRAM — expert Parameters start as CPU staging tensors; each layer's
`weight_loader` counts shard events and, on layer completion,
`ExpertTopology.split_layer` moves that layer's experts into the VRAM arena
and compacts the rest into exact-size pinned host tensors behind UVA. Peak
host RAM = `host-backed set + layers still loading`, not the whole model. At
runtime `apply_hot_cold` sends arena-resident experts down a hot GEMM and the
rest straight over PCIe via UVA views — no staging copy in the critical path.
**Emulation check:** a dequantize-to-compute path doubles real host RAM (118
GiB offload request → ~238 GiB) and is invisible in the weight count. Read the
log for `Using 'EMULATION' Mxfp4 MoE backend` before concluding "doesn't fit".
## 6. Performance profile and honest measurement
Decode on this pair is **PCIe-gather-bound, context-independent**: 12.11
tok/s FLAT from 304 to 16,095 context tokens — KV attention is not the cost.
With ~29 % of experts arena-resident, ~0.84 GB crosses PCIe per token at the
engine's effective read rate; cold-vs-warm differing ~10 % confirms the LRU
works; growth is linear in hit rate, and only speculative decode cuts
sequential STEPS rather than making each cheaper.
**Benchmark traps that produced fake numbers here:**
- `max_tokens=1` is not TTFT (the server can satisfy it without doing the
work — prefill looked flat at 1.26 s for 16k AND 30k prompts). A 64-token
non-streamed reply is time-to-64-tokens (decode-bound, flat ~9.6 s at every
length). Correct TTFT: stream and time the FIRST content-bearing chunk.
- A small repeating word loop is prefix-cacheable AND n-gram-predictable — a
43-word cycle reported 22,897 tok/s prefill, almost entirely cache hit. Use
a large vocabulary in shuffled order and print `cached_tokens`.
- ~1.2 s fixed per-request overhead swamps prefill below ~16k tokens; compare
prefill at 30k+ only.
- Size prompts with `/v1/messages/count_tokens` (a chat call for counting
pollutes the metric); guessing words-per-token silently overran the window
(HTTP 400).
- Always run a canary: plant a unique token at the END of a long prompt and
ask the model to quote it — proves prompts reach the model at all.
- Measure warm AND long (`podman exec` + urllib at 127.0.0.1:7770): a 33-token
reply read 14.6 cold / 20.3 warm; a 768-token generation hit 25.7; two
concurrent streams scaled to 47 aggregate.
**Levers (all opt-in):** `--expert-prefetch N` takes an INT depth, not a
boolean (default 0; at depth 1 only ~10 % cold-vs-warm difference — the cache
already works); `CLAV_TUNABLEOP_SWEEP=1` autotunes GEMM shapes (silently
skipped under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1` too —
see [[dankserver-qwen38-fp8-expert-offload-tuning]] for why it was rejected);
`--expert-precise-memory` swaps the fitted overhead figure in.
## Related
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — deployed shape, plan law,
measured campaigns
- [[PCIe Link and Atomics]] — why decode is gather-bound and what cannot fix it
- [[RDNA4 MoE Expert Offload VRAM Budget]] — arena/plan arithmetic deep-dive
- [[frank-large-benchmark]] and [[frank-small-benchmark]] — window discipline on live hosts
serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md
+219
View File
@@ -0,0 +1,219 @@
---
type: reference
title: RDNA4 MoE Expert Offload VRAM Budget
description: The expert-offload plan arithmetic on the tcclaviger fork — which terms a deploy can move, the util law, the CUDA-graph charge, what free VRAM really is, knife-edge survivability, parallelism ceilings and the context ladder.
tags: [vllm, expert-offload, vram, arena, kv-cache, ladder, tcclaviger]
timestamp: 2026-10-05T00:00:00-07:00
---
# RDNA4 MoE Expert Offload VRAM Budget
For VRAM-bound vLLM on the tcclaviger RDNA4 fork (frank-primary: Qwen3.8-
Flash-Next FP8, TP=2, 2x R9700 at 31.86 GiB each, expert offload to host RAM).
Tuning history and the deployed shape live in
[[dankserver-qwen38-fp8-expert-offload-tuning]]; boot-failure capture lives in
[[vLLM Boot Forensics and Config Interrogation]].
## 1. The plan law
```
arena = budget − non_expert − kv_reserve − cuda_graphs − runtime_overhead
budget = device_total × gpu_memory_utilization # expert_plan.py build_plan
slots = arena // expert_bytes # 2.344 MiB/slot ⇒ 437 slots/GiB
```
**The arena is DEVICE-resident, so the plan predicts reality:** the fp8
3-window shape's plan totalled 28.27 GiB and `rocm-smi` measured exactly
28.27 GiB. `weights + pinned_pool + graphs + arena + non-torch (≈1.45 + 0.57
driver reserve)` vs 31.86 GiB predicts a boot failure before it happens.
Which terms a deploy can move:
| term | reachable? | how |
|---|---|---|
| budget (util) | **yes, law-capped** | `--gpu-memory-utilization`; +0.01 = +0.32 GiB on 31.86 GiB cards |
| CUDA graphs | **yes** | container env `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB` (or `_ESTIMATE_MB` for an explicit total) |
| non-expert weights | no | checkpoint census |
| KV reserve | no | registry constant: 9190 B/token × max_model_len × 1.08 + per-seq mamba pages |
| runtime overhead | partly | `--expert-precise-memory` swaps default 7.60 GiB for the registry-fitted 5.25 GiB |
Anything else is an IMAGE EDIT, not a deploy — say so; do not invent a flag
(verify with `podman exec <ctr> grep -n getenv …/offloader/expert_activation_registry.py`).
**UTIL LAW: 0.982, NEVER 1.00 (measured 2026-10-03).** The CLI field is
`Field(gt=0, le=1)`, but the worker init demands `free > budget`: at 1.0
budget == device total and BOTH ranks raise
`ValueError: Free memory on device cuda:N (31.29/31.86 GiB …)` BEFORE a
single weight loads — a crash loop (NRestarts ~2 min apart, zero payload
progress). Not a knife-edge: **unbootable**. Even bootable near-ceiling
shapes die under real load (0.13 GiB free bf16 boot; `0 bytes free` on a real
995-token request).
## 2. The CUDA-graph term: charge the measured capture
Defaults (256 MiB base + 52 MiB/graph) over-charge badly. Measure with the
engine's own post-capture line `Graph capturing finished … took X GiB` (per
rank; heavier rank binds). Measured on the no-spec piecewise plan: 0.31/0.17
GiB, and one boot reported a NEGATIVE −0.06 GiB delta (capture fit inside
pages the profiler had already counted) — the analytic base pool is pure
over-charge. Shipped: `BASE_MB=0`, `PER_GRAPH_MB=12` = 0.07 GiB — per-graph,
so a spec-decode shape (18 graphs) still pays (0.21 GiB). An explicit
`_ESTIMATE_MB` total does NOT scale with ladder changes; base+per-graph does.
**Verify every deploy:** `Estimated CUDA graph memory` line vs the `Graph
capturing finished` line; if capture exceeds the charge, raise `_PER_GRAPH_MB`
back toward 52. Check `VLLM_CUDAGRAPH_*` actually present in
`podman exec … env`.
## 3. The pool is invisible to the plan; free VRAM is the estimator gap
`--kv-cache-memory-bytes` does NOT appear in the arena arithmetic — the plan
reserves the registry price, not the actual pool. Raising the pool consumes
real free VRAM without growing the arena; the registry over-prices the pool
(9190 B/token planned vs 7,891.8 measured at 200704 fp8 — 7,701 at 262144;
**never reuse a B/token ratio across geometries**, and subtract ~3.87 MB
block-alignment loss when converting a token target to bytes). The ask lands
block-aligned: 3,166,500,000 B → 401,408 tokens; verify with
`GPU KV cache size: N tokens, Maximum concurrency … X.Xx`.
**Driver-reported free VRAM after hours of traffic is NOT allocatable
arena.** It is the estimator gap: the registry over-prices KV and charges
5.25 GiB overhead against ~2.5 measured; torch's allocator does not return
slabs, so it reads as the high-water mark. The only reachable levers are util
(within the law) and the graph charge; everything else needs a registry edit
inside the image — the fork's own log line says as much.
**MNBT is arena-free up to the anchor:** registry overhead =
`base + 0.22 GiB × max(0, (nbt−4096)/1024)` anchored at nbt 4096, so rungs
1024/2048/4096 are plan-identical (the fitted 5.25 GiB was fitted AT 4096).
MNBT changes re-key the compile range (one recompile, ~24 s at (1,2048) with
the mounted cache). 512 doubles TTFT at 4064 ctx; 12288 costs ~2.5 % plus
arena. MNBT does not move the KV pool.
## 4. Registry "compile headroom" is not reclaimable slack
`--expert-precise-memory` selects the fitted overhead (5.25 GiB, this arch at
TP=2) over the default (7.60 GiB = fitted + **2.35 GiB of compile and
graph-capture headroom** the registry documents as "transient VRAM the
profiler never sees"). It moves exactly one term — the plan's `room` — so the
arena grows by that 2.35 GiB (~1,025 slots). **That headroom is the budget
for the phase that allocates LAST**: reclaiming it requires paying elsewhere
(smaller pinned pool, fewer sessions, shorter context). Measured 2026-10-01:
bf16 2×262,144 + precise put the total at 31.73 of 31.86 GiB and the boot
crash-looped at graph capture; dropping the flag restored ~2.4 GiB and booted.
## 5. Boot ≠ survival (knife-edge shapes)
A shape can pass EVERY boot gate — weights, profile, KV allocation, graph
capture, `/health` 200 — and die on the first real request. Measured
2026-10-01: `5 × 262144 @ fp8` booted (KV 1,350,182 tokens / 5.15x) and was
killed seconds later by a 995-token prompt with `max_tokens=261149`:
`torch.OutOfMemoryError: Tried to allocate 50.00 MiB … 0 bytes free`. After
any knife-edge change: send a LARGE-`max_tokens` request before calling it
done, and read post-boot free VRAM per rank (`rocm-smi --showmeminfo vram`)
— the survivor had ~3.6 GiB free; the corpse had 0.
**KV accounting — pin the pool BYTES, floor-sized:** set
`--kv-cache-memory-bytes` PER RANK to the contract floor
(`seqs × MAXLEN × measured B/token`), not the native window — extra pool bytes
come out of the arena/headroom and buy nothing. Measured B/token on this
hybrid: **7,701 fp8 / 13,839 bf16** (decomposed: attention 6,138/12,276 +
dtype-independent mamba/side 1,563). Measured rates are shape-dependent —
7,891.8 at the 200704×3 shape. Capacity does NOT scale with `SEQS`: the
planner's KV reserve is for ONE max-model-len request.
**Parallelism ceilings you cannot flag past:**
- `--decode-context-parallel-size` refused for GQA when `TP ≤ num_kv_heads`:
*"requires TP (2) to be greater than the model's total number of KV heads
(2)"* — DCP shards KV along the sequence dimension AFTER KV heads spread
across TP ranks (it reuses TP ranks, not TP×DCP), so a 2-KV-head model
needs TP ≥ 4 = four GPUs.
- EP (`--enable-expert-parallel` + `--all2all-backend`) exists but REPLACES
the fork's local tiered-offload gather with a per-layer PCIe all-to-all.
- DP requires a full model copy per replica — on a model larger than the
card it is impossible, not slow.
## 6. The context ladder (find the max bootable window by measurement)
"Max possible" = the largest value that BOOTS, measured — never the model's
spec maximum, never hand arithmetic.
1. **Scale the pool with the window.** If KV_BYTES is pinned, it must shrink
in lockstep with MAXLEN or the rung tests nothing:
`KV_BYTES = round(KV_BYTES_ref × (seqs×MAXLEN_new) / (seqs×MAXLEN_ref))`
off a booted reference rung. Verify the rung's capacity still covers
`seqs × MAXLEN` from the engine's own line.
2. **Boot under the PRODUCTION environment, not a shell probe** — a rootless
shell lacks `LimitMEMLOCK=infinity` and can pass where production OOMs.
Transient unit:
`sudo systemd-run --uid=<svcuser> --unit=probe-$ML -p LimitMEMLOCK=infinity -p Restart=no env <ENVFILE_OVERRIDE> <serve-script>`
pointing the serve script at a scratch env file via its `*_ENV_FILE` hook.
3. **Judge on health + unit liveness:** poll `/health` (200 up / 503 loading
/ 000 not bound); break early when the unit goes inactive — that is the
failure signal, not a timeout.
4. **Capture the engine log per rung BEFORE teardown** (`podman logs` dies
with `podman rm`; a `podman logs` issued too soon after `podman run -d`
fails with *no container with that name* — record UNKNOWN, never FAIL).
5. **Ladder down on FAIL, one bonus step UP on PASS.** Measured steps on this
box: OOM −16,384 tokens, bonus +4,096. Tiny margins (76 MiB class) mean
the next rung down almost certainly passes: `Tried to allocate 76.00 MiB …
0 bytes free … 30.43 GiB allocated by PyTorch` died in graph CAPTION;
`Tried to allocate 10.28 GiB … 9.46 GiB free` died in KV ALLOCATION.
6. **Clean between rungs:** stop the transient unit, `podman rm -f`, kill
orphans by explicit PID
(`ps aux --sort=-rss | awk '/vllm\/bin\/python3/ && $6 > 50000000 {print $2}'`),
wait for `MemAvailable` to recover.
**The hazard that cost production:** a leftover ladder script whose teardown
`podman rm -f`s the production container fires against a freshly DEPLOYED
engine. Teardown runs ONLY on failure, never after a pass; kill the whole
scratch process tree and verify none remain BEFORE deploying. On PASS: print
the measured `GPU KV cache size / Maximum concurrency` line, LEAVE IT
SERVING, and pin the number into the repo (env + seam + harness
contextWindow) before deploying.
Cutover order: ladder → winner passes and serves → repo env + seam + harness
→ tests → commit+push → verify no scratch scripts remain → stash foreign work
(`git stash -u`) if the tree is dirty → deploy (one restart) → verify health
+ engine flags in the log.
**Measured ladder on this box (200704×3, fp8 KV, precise memory):**
| shape | util | graphs | arena | result |
|---|---|---|---|---|
| E | 0.98 | 0.55 | 17.22 GiB / 7,522 slots | boot ok |
| F | 0.982 | 0.32 | 17.52 GiB / 7,653 slots | boot ok |
| 1.0 | 1.00 | 0.07 | — | UNBOOTABLE (worker-init ValueError, both ranks) |
| G (deployed) | 0.982 | 0.07 | 17.77 GiB / 7,762 slots | KV 401,408 tokens / 2.00x; 0.89 GiB real free/card held through a live 4,096-token generation at ~38–39 tok/s |
bf16 ceilings at util 0.98: 3 seqs → 241,664 passes / 245,760 and 262,144 OOM
at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
~207k per context max.
## 7. Arena audit procedure (when asked for "more experts")
1. Read the live plan before proposing anything — `grep -oE` with a
digit-anchored tail (plain grep/journalctl truncate these columns to
`17.`):
```
ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-primary-qwen -b -o cat --no-pager \
| grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'
```
2. Classify each term (§1 table); take only the two reachable levers.
3. Verify after deploy: plan lines show new budget/graphs/arena;
`GPU KV cache size` matches the ask; graph-estimate vs post-capture;
`rocm-smi` high-water per card = the real margin;
`systemctl show frank-primary -p NRestarts --value` = 0; a real chat
completion from a PEER (health-probe via `192.168.0.235:7770/health`, not
`localhost` inside an ssh session on the host — the latter returns `000`
and looks like an outage).
## Related
- [[RDNA4 Container Serving]] — the bring-up path and container traps
- [[vLLM Boot Forensics and Config Interrogation]] — capture and contracts
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — measured history
- [[Llama Cpp Serving Sizing]] — the llama.cpp equivalent
serverAppConfigs-docs/LLM/vLLM Boot Forensics and Config Interrogation.md
+207
View File
@@ -0,0 +1,207 @@
---
type: reference
title: vLLM Boot Forensics and Config Interrogation
description: Capturing a vLLM container boot failure's real root cause from journald (the container log is already gone), classifying the failure, and interrogating a live endpoint's true config and client contracts.
tags: [vllm, forensics, journalctl, podman, config, contracts]
timestamp: 2026-10-05T00:00:00-07:00
---
# vLLM Boot Forensics and Config Interrogation
For serve-script vLLM endpoints (frank-primary/qwen-fp8 on dankserver, any
`podman rm -f` + restart service).
## 1. Capture the true cause — the journal, not `podman logs`
The serve script runs `podman rm -f <container>` at every start, so a dead
container's log is GONE by the time you look; `no container with name or ID`
means the journal is the only source. The engine's stdout survives in
journald under the container tag:
```bash
journalctl --no-pager --since "20 min ago" | grep -a "<container>\[" | tail -40
journalctl --no-pager --since "8 min ago" | grep -a "<container>\[" \
| grep -aiE "error|exception|assert|raise|invalid|failed" | grep -av APIServer | tail -12
```
Why naive greps fail:
- Default format truncates lines at the display cut, and the fork's giant
single-line ERROR dumps are journald-SPLIT — the decisive sentence never
appears in short format.
- `-o cat` strips the `svc[pid]:` prefix, so grepping the tag against `-o
cat` returns nothing — use the default format for the tag, `-o json` for
the content.
- Time filters: relative (`--since "NN min ago"`) only — absolute timestamps
are local PDT while the engine logs UTC; never mix.
**The JSON decode procedure** (proven 2026-10-01, caught
`torch.OutOfMemoryError` exactly):
1. Bound the window: last traceback lines + the `container died` podman
event.
2. `journalctl --no-pager --since … --until … -o json | python3 -c …` on the
remote: handle `MESSAGE` possibly arriving as a **byte list**
(`bytes(int(x) for x in m).decode("utf-8","replace")`), strip ANSI
(`re.sub(r"\x1b\[[0-9;]*m", "", m)`), dedupe by first ~12 chars (one
logical traceback can arrive as 20+ entries).
3. Reassemble the worker traceback: filter skeleton lines
(`File |…| raise |^+|~+|…<N> lines|…|re-`), keep content lines. The
decisive line is usually the last content line.
4. **The diagnosis is the gap math: `ask − free = shortfall`.**
**The wrapper trap:** `RuntimeError: Engine core initialization failed. See
root cause above` is the WRAPPER — the real cause is the EngineCore traceback
above it, ending in `RuntimeError: Worker failed with error '<cause>'`. Grep
for `Worker failed with error`.
**For probe boots / runner scripts:** capture live BEFORE teardown —
`podman logs -f <ctr> > /home/dank/logs/<arm>.engine.log 2>&1 &` right after
launch. A runner's `podman rm` destroys the only copy; two 2026-10-01
failures had their root cause eaten this way, and an 8-line bootfail capture
is insufficient.
## 2. Failure classes
| signature | class |
|---|---|
| `CUDA out of memory. Tried to allocate X GiB … Y GiB free` | capacity; the gap math names the next rung ([[RDNA4 MoE Expert Offload VRAM Budget]]) |
| `Worker failed with error 'Constraints violated (L['query_start_loc'].size()[0])! … specialized it to a constant (N)'` | torch.compile/dynamo shape specialization (N = num_seqs+1-ish) — NOT memory |
| `Deactivated successfully` with no traceback | container exited CLEAN (vLLM API server exits 0 after an EngineCore fatal); container already removed — re-read the journal, check `NRestarts` |
| `no container with name or ID` | container rm'd; journal only |
Boot-order reality: the plan computes fine at impossible budgets; the
failure lands later in worker init, not at plan time. A plan line printing a
huge arena does NOT mean it will boot.
Bypasses this fork exposes for compile faults: `--enforce-eager` (skips
torch.compile AND CUDA graphs — drops graphs, a real cost),
`VLLM_DISABLE_COMPILE_CACHE=1`, `VLLM_USE_AOT_COMPILE=1`. A stale cache is
usually NOT the cause — verify before "fixing" it (`~/.cache/vllm` may hold
only `modelinfos`; each `podman run` gets a fresh container FS unless the
cache dir is mounted).
**Crash-loop signature (shape does not fit):** `Started … ~6.7 min later …
Deactivated successfully … restart`. Diagnose with `systemctl show -p
NRestarts` + per-attempt duration. `Restart=always` loops visibly;
`Restart=on-failure` stays SILENTLY DEAD (`podman wait` returned 0, systemd
saw a clean exit).
## 3. Observation hygiene during recovery
- A container being STOPPED still answers `podman logs`, so a watcher
grepping its log reports PRE-change values and looks like the fix failed.
Discriminate by container identity:
`podman ps --sort created -a --format '{{.ID}} {{.Status}} {{.CreatedAt}}'`
— require the container to post-date the arm start, then read only that ID.
- A leftover probe script whose tail does `podman rm -f <prod-container>`
fires against a freshly DEPLOYED engine. Signature: a previously-healthy
unit exits 0 (`Started → Deactivated`, code 0) — an external rm, not a
crash. Suspect an external actor before debugging the engine; `pgrep -af`
every script you launched (and kill strays of `deploy-remote.sh` by
explicit PID — killing the dev-side deploy job does NOT kill its host-side
runner).
## 4. Ops laws that bite during recovery
- `Restart=always` does NOT fire after an explicit `systemctl stop` — a
deploy's stop leaves the unit down until something starts it. Check
`systemctl is-active` yourself.
- `ExecStop=podman stop -t 120` requires `TimeoutStopSec` > 120; the 90 s
default kills the stop and logs `Failed with result 'timeout'`.
- A `D`-state worker cannot be reaped by ANY signal (SIGTERM/KILL/`podman rm
-f` all fail). Check `ps -o pid,stat,etime,comm` before declaring a GPU
wedge — measured 2026-10-01: D → R on its own within ~3 min, then the
container stopped normally.
- Never `pgrep -f`/`pkill -f` a pattern that appears in your own command
line — kill by explicit PID list.
- `git stash -u` (not bare `git stash`) for deploy windows — untracked files
still trip the deployer's dirty-tree gate.
- A production engine started by a transient/ladder unit is NOT
boot-persistent; pin the winning numbers in the repo and deploy the real
unit.
## 5. Logging audit (what more can this container say?)
Answer with evidence from the live image, not upstream docs:
1. Enumerate the fork's own env registry:
`podman exec <ctr> python -c "import vllm.envs as e; print(chr(10).join(k for k in dir(e) if k.isupper() and any(s in k for s in ('LOG','DEBUG','TRACE','STATS','PROFILER','VERBOSE'))))"` — then read defaults/clamps in `vllm/envs.py` (e.g. `VLLM_LOG_STATS_INTERVAL` defaults 10.0 s, clamped ≥ 1.0; telemetry emits only while requests are in flight).
2. **Find the REAL HIP/ROCm library paths before grepping for AMD env vars** —
containers often install ROCm into system dirs, not `/opt/rocm`
(`podman exec <ctr> sh -c 'find / -name "libamdhip64.so*"'`). A grep of
`/opt/rocm/lib` that finds nothing is a FALSE NEGATIVE, not an absence.
Then `strings <lib> | grep -E '^AMD_LOG_LEVEL$|^HSA_ENABLE_DEBUG$'`.
3. Scan vendor kernel `.so`s for switches — and separate tuning vars from
logging vars (both share a prefix).
4. **Measure the DEBUG payoff**: count `logger.debug` sites in the module of
interest. 0–10 sites ⇒ a per-module DEBUG bump buys almost nothing; global
`VLLM_LOGGING_LEVEL=DEBUG` additionally makes api_server log full
request/response bodies (incident window only).
5. Check the deployed argv for `--disable-log-stats` / `--log-configs` to
know what is already suppressed.
Report: lever / effect / cost table, split into permanent-safe (stats
interval), incident-window (`AMD_LOG_LEVEL`, DEBUG), and does-not-exist
(absent strings). Wire via env-gated `-e` pass-throughs so default-off
preserves behavior.
## 6. Interrogate a live endpoint's REAL config
Flags and env diverge from intent constantly — rollbacks drop them silently.
Read what the container is actually serving, then prove the client contract
behaviorally:
```bash
# live argv (the truth)
ssh host 'podman exec <ctr> bash -c "cat /proc/1/cmdline | tr \"\\0\" \" \"" | tr " " "\n" \
| grep -A1 -E "max-model-len|max-num-seqs|kv-cache|reasoning-parser|tool-call-parser|template-kwargs|utilization|speculative"'
# container env knobs (fork tuning that never appears in argv)
ssh host 'podman exec <ctr> bash -c "env | grep -E \"R4D_|CLAV_|VLLM_\" | sort"'
# capacity contract as measured at boot
ssh host 'podman logs --tail 3000 <ctr> 2>&1 | grep -aE "GPU KV cache size|Maximum concurrency|non-default args"'
```
`non-default args` lines print what the server overrode — the fast way to
catch a flag that never landed. `GPU KV cache size: N tokens, Maximum
concurrency for M tokens per request: X.XXx` is the capacity contract:
N ≥ requests × M or the last client gets truncated.
**Thinking contract:** a reasoning model on vLLM returns `reasoning` ONLY if
the server runs a reasoning parser; without it the thinking lands INLINE in
`content` with a dangling close-tag and a null `reasoning`. Verify with a
temp-0 completion: pass = `reasoning` non-empty AND no `<\/think>` in
`content`. Diagnose with `ReasoningParserManager.list_registered()`
(import `vllm.reasoning`).
**Tool contract:** `tool_choice: "auto"` on a server WITHOUT
`--enable-auto-tool-choice` returns HTTP 400 — reads like a model fault, is a
config gap. Probe with a real tool schema and check `tool_calls` +
`finish_reason`.
**Restoring a dropped contract from history:**
```bash
git log --all --oneline -S"reasoning-parser"
git show <commit>:scripts/serve-<svc>.sh | grep -B3 -A3 "reasoning-parser"
```
Qwen3-family values that work on the tcclaviger fork: `--reasoning-parser
qwen3`, `--enable-auto-tool-choice --tool-call-parser qwen3_xml`,
`--default-chat-template-kwargs '{"enable_thinking": true,
"preserve_thinking": true}'`. `preserve_thinking`/`enable_thinking` are
chat-template variables, NOT engine flags: they belong in
`--default-chat-template-kwargs` server-side or `chat_template_kwargs` in the
request body; a bare top-level body field of that name is ignored. Check the
payload's `tokenizer_config.json` carries the key before trusting it either
way.
`/health` 200 says the engine is up, not that the contract is on the flags
you think — any client report of odd content (inline thinking, 400 on tools,
clipping at depth) starts with sections 1–3 here before touching anything.
## Related
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic behind OOMs
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — deployed shape and history
- [[frank-large-benchmark]] and [[frank-small-benchmark]] — probe/ladder hygiene on live hosts
serverAppConfigs-docs/LLM/vLLM Porting and Compile Triage.md
+330
View File
@@ -0,0 +1,330 @@
---
type: reference
title: vLLM Porting and Compile Triage
description: Forward-porting a model architecture to a new vLLM major (additive-only discipline, gap enumeration, shims that survive worker spawn) and the torch.compile/dynamo break triage, including the EAGLE3/DFlash2 aux-hidden-state protocol port rules.
tags: [vllm, porting, torch-compile, dynamo, eagle3, dflash, glm]
timestamp: 2026-10-05T00:00:00-07:00
---
# vLLM Porting and Compile Triage
## Part I — Forward-porting a model architecture
### 1. Check whether a port already exists; size the work honestly
Read what the target image already ships before assuming a port is needed: a
backport's upstream PR may be merged in the target version, and the target may
carry a native kernel path the older line lacks (e.g. `rfi_hip` for RDNA4
native quantized MoE). Porting around a native path that was already there is
the expensive mistake.
An upstream backport ships two kinds of file: **additive** (the new model
package, its config, its kernel modules — port these) and **whole-file
replacements of vLLM core** (`model_runner.py`, `scheduler.py`,
`kv_cache_utils.py`, platform files — these are version X's copies of files
version Y restructured; copying them is a DOWNGRADE, not a port). Measure both
(`find … | xargs wc -l`), and say the additive-only ratio out loud before
starting — one real case: a 71,724-line overlay reduced to 10,385 additive,
and the "missing" drafter was already registered in the target.
### 2. Enumerate the gap in ONE pass, not one ImportError at a time
Inventory the target tree first (`model_executor/layers/`,
`v1/attention/backends/`, quantization backends, spec-decode drafters in
`registry.py`, `third_party/flash_linear_attention/ops/`). Then walk the
staged package with `ast`, resolving every `from vllm… import X` against the
target, split into MODULE-missing vs NAME-missing.
- **The double-prefix bug:** the dotted module already starts with `vllm.`,
so joining onto the package dir needs
`rel = mod.split(".", 1)[1].replace(".", os.sep)`.
- **Static scans over-report:** bare annotations, TypeAlias and star
re-exports resolve at runtime. Do a runtime check
(`importlib.import_module` + `getattr`) before quoting numbers — one scan
said 151/163 resolved; runtime said 10/13.
- **Classify:** redirects dominate (version moved a symbol to a submodule) —
a ~60-line rewriter splitting `from X import (a, b, c)` blocks handles
them; **delete a block that empties** (a bare `from X import (` with an
empty body is a SyntaxError) and gate the rewriter on `ast.parse` of every
touched file.
- **Port absent symbols with AST extraction**, not copy-paste: locate the
`ClassDef`, check every base/decorator resolves in the target, append the
exact segment, re-`ast.parse` the target, roll back on failure, skip if
already defined.
### 3. Silent-default traps and transformers gates
- Check every `cfg_int(cfg, key, DEFAULT)` against the new checkpoint: if the
key was renamed, the DEFAULT silently survives and the model computes fluent
nonsense with no error. Diff every config key the old parser reads against
the new `config.json`. When the old port used a fallback, the CHECKPOINT's
value must win (measured: `num_experts_per_token` fallback 7 vs
checkpoint `num_experts_per_tok=8`).
- `model type X but Transformers does not recognize this architecture` =
pinned transformers predates the model. Upgrading transformers inside a
pinned vLLM image breaks things hard to attribute — prefer registering a
genuine `PretrainedConfig` subclass:
`CONFIG_MAPPING.register("glm5_next", Glm5NextConfig)`, gated by a real
round-trip (`AutoConfig.from_pretrained(<ckpt>)` returns the right class).
If the target image's transformers is below vLLM's declared minimum,
upgrading inside the image beats a shim.
- **Hardware-gate flags from the port's own docs**: search the README for the
arch name and "do not" — e.g. a native quant MoE backend that is
CUDA/CDNA-gated falls back to per-call quantize-dequantize emulation at
~5x slower; pin those flags in the serve script with a comment citing the
line.
- **Backend selection:** default to `auto`, then READ the chosen-backend
line. `TRITON_UNFUSED` rejects some activation keys; `TRITON_ATTN` rejects
MLA + sparse attention — both surface as an init rejection. **"Using
'EMULATION' backend" is a FAILED configuration, not a success** (weights
dequantized per call: memory + throughput cost).
- `Loading model from scratch…` for many minutes usually means the port
SUCCEEDED and you are in weight placement — verify all layers constructed,
both TP workers alive, zero errors, before chasing memory. And if it does
not fit, **prove the band is empty**: sweep the knob and show no setting
works; a single OOM is not a conclusion. Keep `MemAvailable` guarded
(abort on low RAM; do not let the OOM killer take co-tenants).
### 4. Shims that must survive worker spawn
vLLM v1 workers are spawned processes — a parent-process monkeypatch is gone
by weight-load time. In order of preference:
1. **Patch the installed library's source file on disk** (reliable). Locate
the real install path at runtime:
`python -c "import transformers,os;print(os.path.dirname(transformers.__file__))"`.
2. `.pth` in site-packages — works but runs at interpreter start and can
import a package mid-import (circular import through `accelerate`);
expect failure.
3. `sitecustomize.py` — often silently DEAD: Debian ships its own at
`/usr/lib/python3.12/sitecustomize.py`. Verify it is actually imported
before debugging why it has no effect. (A `.pth` on `PYTHONPATH` is never
executed by `site.py`; `sitecustomize.py` is.)
Always confirm with a FRESH process — never with the process that set the
patch up (`is_moe`, `num_experts_per_token`, `mhc_num_residual_streams`).
**Misleading OOM names on ROCm:** `torch.AcceleratorError: CUDA error: out of
memory` from `pin_memory()` is usually the HOST `memlock` rlimit, not VRAM.
Cannot be raised from inside a container or root's own shell; only PAM at
login: `/etc/security/limits.d/*.conf` + a NEW session. Confirm by pinning
2/8/16 GiB — if those succeed, the rlimit was a real bug but is not the
failure you are chasing.
### 5. Ship as an idempotent patch-at-start applier
One `apply.sh` at container start, each step guarded by a grep-for-marker
(re-entry safe), ending in a verification block that imports every installed
symbol and prints PASS/FAIL per symbol — ~10 s instead of a multi-minute
load, and it turns "I think it works" into evidence.
---
## Part II — torch.compile / dynamo bring-up triage
### The five laws (measured on torch 2.11+rocm, vLLM 0.29-dev)
1. **`suppress_errors` does NOT cover graph breaks** — fullgraph=True +
suppress_errors=True still raises `Unsupported`. Only `fullgraph=False`
tolerates a break.
2. **A warm `lru_cache` is not trace-safe** — dynamo re-traces the wrapped
body to build its guard even on a hit ("Dynamo ignores the cache wrapper
and directly traces the wrapped function"). Resolve the value ONCE into a
plain module global and read that; a global read folds, a cached call does
not. Verified in-image: warm lru_cache-calling function still fails
fullgraph=True; a plain global read compiles.
3. **AOT compile refuses breaks outright** ("Graph breaks are not supported
with aot compile") — set `VLLM_USE_AOT_COMPILE=0` for a break-tolerant
run.
4. **Fatal vs tolerated breaks:** tolerated breaks get a resume function
(untraceable part runs eagerly, numerics unchanged). Fatal ones cannot:
`Unsupported: Read uninitialized cell` (a traced callee closes over a name
bound after its definition) and `Skip calling torch.compiler.disable()'d
function` under fullgraph=True. `torch.compiler.disable` is the documented
escape hatch for host-side/stateful work (ContextVar reads, lazy
allocation, workspace grab) and is a NO-OP when nothing compiles — safe to
ship in the stock path.
5. **Trust the frame chain, not the line number** — dynamo INLINES user
callees; an inlined failure reports the enclosing function's `def` line.
"KeyError in forward at the def line" is really "a callee raised".
`TORCHDYNAMO_VERBOSE=1` adds nothing for a plain Python exception (only
for dynamo-internal errors).
**The tell** that the trunk lacks compile support at all:
```
Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
provides neither a compiled submodule nor breakable CUDA graphs. Overriding to
FULL_DECODE_ONLY.
```
Diff against the family that DOES compile and copy its
`@support_torch_compile` decorator including `dynamic_arg_dims` and any
`enable_if` hook; env-gate every change so the file can stay mounted for an A/B.
### Procedure
0. **Establish what configuration is ACTUALLY in effect**:
`podman exec <ctr> printenv VLLM_COMPILE_FULLGRAPH VLLM_USE_AOT_COMPILE …`.
A launcher that "sets" an env var in a branch may silently drop it — four
consecutive boots ran `fullgraph=True` while the operator believed
otherwise, which makes the reported failure MOVE as breaks are removed and
invites entirely wrong theories.
1. **Boot with the break list, not the first error:** `TORCH_LOGS=graph_breaks`
+ `VLLM_USE_AOT_COMPILE=0` + `VLLM_COMPILE_FULLGRAPH=0`. Re-verify each
candidate break under the flag-free config you actually intend — a break
that appears only under the logging flag is a diagnostic artifact.
2. **Fix in priority order:** (a) host syncs in the traced region
(`.item()`/`.cpu()`/`.tolist()` — check reachability first, a dead fallback
branch is not your blocker); (b) capability probes doing I/O in `forward`
(`os.path.exists`, `.so`/`ctypes` lookups — resolve once into globals,
warm at import); (c) lazy stateful work (hoist, or `torch.compiler.disable`
a thin wrapper); (d) new kernel specializations — check the vendor ships
SOURCES (`nm -D`/`strings`): a binary-only templated `kernel<128|256|512>`
makes the 288 you need unreachable. Fix at the PATTERN level (one
launcher/decorator family breaking all 39 call sites), not site level.
3. **KeyError on a bare module name** = dynamo's name→object map miss
(`OutputGraph.nn_modules`: registered `output_graph.py:1259`/`:1384`,
read back `:3000`, `_buffers/_parameters/_modules` stripped at `:1225`).
Ship a probe via `sitecustomize.py` that patches
`OutputGraph.example_value_from_input_node` and keeps weakrefs of live
graphs, printing `node.target`, key count, near-miss keys — separates a
cross-graph reference from a name-derivation mismatch.
4. **Instrumenting a startup hook is a FIREARM — arm it once:**
```python
if os.environ.get("PROBE_ARMED") == "1":
pass # an ancestor already armed it -> stay inert
else:
os.environ["PROBE_ARMED"] = "1"
...install probes...
```
A `sitecustomize.py` on `PYTHONPATH` imports in EVERY interpreter and
children inherit `PYTHONPATH`: an unguarded hook that imports the engine
self-replicates (observed: 19+ levels, 33 processes alive 18 min later).
Kill the chain by removing the container whose cgroup owns every
generation, then verify with a precise `/proc/<pid>/environ` count —
NEVER pattern-kill an interpreter name on a host running python in
production containers. Also chain the image's own sitecustomize so its
hook survives.
5. **Cheap in-image classification (no boot):**
`g = torch.compile(fn, backend="eager", fullgraph=True)` inside the
container (with `--device /dev/kfd --device /dev/dri` — torch.compile
queries a device even on the eager backend). ~5 s per question instead of
~7 min boots.
6. **Package as real patch artifacts:** hand-written `@@` headers are not
diffs and `patch(1)` rejects them. Generate
`diff -u --label a/<path> --label b/<path> orig new > name.patch` with
`--- a/` / `+++ b/` paths so `-p1` works; verify by applying to a fresh
extraction of the pinned image and comparing sha256 (`sha256sum` must
equal the `b/` hash recorded in the header). Header carries: STATUS, the
measured evidence, base+patched sha256, how to apply, the NEXT blocker.
### Cost model before investing
**Profile first** — parse the rank0 chrome trace (`POST /start_profile`,
device-activity rows) for the kernel table. Measured case: the long-suspected
expert-gather was **1.05 %** of device time while **44 %** was unfused
pointwise glue (708,500 `vectorized_elementwise_kernel` launches =
1,428/token) — a months-old "structural" conclusion died in one afternoon.
Speculative decoding does NOT help a shape whose per-token cost scales with
the token count, and it hurts prefill. Decide the ceiling up front: if the
fused target is ~13–15 tok/s, do not spend a night of boots chasing 35.
---
## Part III — EAGLE3 / DFlash2 / DSpark aux-hidden-state protocol port
Goal: make a trunk satisfy `SupportsEagle3` so a DFlash2 drafter
(`DFlash2DraftModel`, method `dflash`) boots. Built on the GLM-5.3-Flash port
(tcclaviger fork, 2026-09-30; four failed boots produced the laws below).
### The four laws
1. **Patch the instance the runner calls.**
`set_eagle3_aux_hidden_state_layers(self.model)` (model_runner.py, right
after `model_loader.load_model`) runs on the LOADED instance — for a
multimodal model that is the `…ForConditionalGeneration` WRAPPER (the
APIServer log line `Resolved architecture: …` names it), NOT the text
model it holds as `self.language_model`. Patching only the inner
`…ForCausalLM` reproduces the identical `RuntimeError: Model does not
support EAGLE3 interface` forever. Patch BOTH wrappers.
2. **The check is Protocol isinstance, NOT duck typing.**
`supports_eagle3(model)` = `isinstance(model, SupportsEagle3)`
(interfaces.py ~1728; caller `v1/worker/gpu/spec_decode/eagle/eagle3_utils.py:20`).
`SupportsEagle3` is `@runtime_checkable` over base `SupportsEagleBase`
(`:1486`, `:1659`); `typing._ProtocolMeta.__instancecheck__` does
`getattr_static` over ALL `__protocol_attrs__` — including BASE-protocol
members — and breaks on the FIRST miss. Full member set:
`supports_eagle3`, `has_own_lm_head`, `has_own_embed_tokens`,
`set_aux_hidden_state_layers`,
`get_eagle3_default_aux_hidden_state_layers`. **Class attributes count as
presence; a methods-only implementation FAILS** because the three attrs
are missing. Verified in-image with stock typing, no boot: flag-only →
False; flag + two base members → True.
3. **MRO forbids inheriting the Protocol.** Appending `SupportsEagle3` to a
wrapper already carrying Protocol-derived bases dies at class creation:
`Cannot create a consistent method resolution order (MRO) for bases …` —
MRO simulation (stub Protocols + `type("C", bases, {})`) confirmed no
position works. This failure also PROVES the isinstance-semantics question
is moot: class creation precedes any check.
4. **Write the two methods by hand** — without inheritance the Protocol
defaults do not exist. Text model: unwrap to the trunk
(`self.model.aux_hidden_state_layers = layers`); multimodal wrapper:
delegate to `self.language_model` (the pixtral.py:498 pattern).
`get_eagle3_default_aux_hidden_state_layers` returns `(2, L//2, L-3)`; the
drafter config's `target_layer_ids` wins when present (GLM DFlash2 ships
`[5, 14, 24, 33, 42]` of 45).
The fix shape (on EACH wrapper class):
```python
supports_eagle3: ClassVar[Literal[True]] = True
has_own_lm_head: bool = False # False when embed/lm_head are delegated
has_own_embed_tokens: bool = False
def set_aux_hidden_state_layers(self, layers): ...
def get_eagle3_default_aux_hidden_state_layers(self): ...
```
### Trunk mechanics
The trunk (`Glm5NextModel`) must be `EagleModelMixin`; add
`aux_hidden_state_layers: set[int]` (default empty) + `start_layer` handling.
Capture aux states in the layer loop: the state ENTERING each requested
layer, BEFORE its input norm (GLM's loop carries the single-stream state;
deepseek_v2's equivalent is `hidden_states + residual`).
`use_aux_hidden_state_outputs=True` is set in model_runner.py (~:287) for
methods `dflash`, `dspark`, `extract_hidden_states`; the forward returns
`(hidden_states, aux_hidden_states)` conditionally after `self.norm` (tuple
consumed at `v1/worker/gpu/cudagraph_utils.py:621`).
### Window discipline and verification
- Scratch boots on dankserver MUST stop + runtime-mask production
`frank-primary` first — the expert-offload planner reads `MemAvailable` at
plan time, and beside-prod boots die at plan with
`ExpertPlanError: host holds N of M swappable experts`. After stop+mask,
loop until `MemAvailable ≥ 230 GiB` before launching (a stop against a
mid-load engine SIGKILLs at ~90 s and the freed ~170 GiB takes >1 min to
return — a new window started during restore inherits the false failure).
- Never launch a window while the previous window's RESTORE is in flight
(`systemctl start` returns immediately; the engine loads ~6 min).
- Save full probe logs BEFORE teardown (`podman logs <probe> > full.log`);
decision greps truncate lines at `…` and tracebacks die with the container.
- Cheap verification, no boot: AST-parse the patched file for class bases +
member presence; MRO simulation with stub Protocols; in-image `typing`
replication of the isinstance walk (pure stdlib — importing vllm itself
crashes on GPU-less inspection). Boot check: the worker line `Using Eagle3
auxiliary layers from config: (…)` = gate cleared.
- Honest expectations: a target-side DFlash2/EAGLE3 port on a
pointwise-glue-bound trunk (~44 % of kernel time) buys ~ZERO decode gain —
the verification pass costs ∝ its token count. Port it for capability, not
speed.
## Related
- [[vLLM Boot Forensics and Config Interrogation]] — capture and contracts
- [[RDNA4 Container Serving]] — the serving stack being ported onto
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the live deployment
- [[halogen-flash-server]] — the retired w4b-era endpoint and its profiling notes
serverAppConfigs-docs/index.md
+8
View File
@@ -379,6 +379,14 @@ The LLM inference fleet on the two hosts. Full index:
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
- [[GGUF Compatibility]] — which loaders accept which quant formats; the tensor-type histogram check
- [[Llama Cpp Multi-Device Splits]] — layer vs tensor split across unlike devices; the -ot pipeline gate
- [[Llama Cpp Serving Sizing]] — VRAM arithmetic from two measured boots, never a formula
- [[PCIe Link and Atomics]] — link caps vs loaded rate; RCCL atomics routing on the old board
- [[RDNA4 Container Serving]] — the serving stack recipe: mounts, ulimits, udev, verify gates
- [[RDNA4 MoE Expert Offload VRAM Budget]] — arena sizing law for expert offload: free VRAM is estimator slack, not allocatable headroom
- [[vLLM Boot Forensics and Config Interrogation]] — journal/endpoint forensics for boot failures; argv recovery from a live engine
- [[vLLM Porting and Compile Triage]] — forward-porting vLLM majors; compile failure forensics
- [[ling-3.0-flash-trial]] — the Ling-3.0-Flash trial runbook
- [[ling-3.0-flash-vllm-plan]] — the Ling-3.0-Flash vLLM plan (retired)