serverAppConfigs/serverAppConfigs-docs/LLM/Tcclaviger Fork Ops.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Tcclaviger Fork Ops	Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken cutover) — the ad-hoc sweep harness, measured knob verdicts, sweep hygiene, the large-MoE serving recipe, and compile-cache persistence.	tcclaviger vllm dankserver expert-offload sweeps cache	2026-10-05T00:00:00-07:00

Tcclaviger Fork Ops

The tcclaviger RDNA4 vLLM fork (docker.io/tcclaviger/vllm@sha256:da4029666…, vLLM 0.29.0.dev, GLM-5.3/Qwen-specialised with native MoE quant kernels and a first-class expert-offload planner) served frank-primary at shape G until the 2026-10-05 FreeToken cutover; it remains the rollback target (git tag frank-primary-vllm-shapeG-20261003). Full plan law and VRAM arithmetic: RDNA4 MoE Expert Offload VRAM Budget; boot forensics: vLLM Boot Forensics and Config Interrogation; window discipline: LLM Bench Windows and Measurement.

The ad-hoc sweep harness (dankserver)

Serve script /home/dank/serve_qwen_fp8.sh with env hooks: SPEC (MTP depth), TUNABLE (TunableOp), PREFETCH, UTIL, MAXLEN, KV_BYTES, MNBT, UVA_TOK, GATHER_C/GATHER_L (R4D_LRU gather), TELEM (R4D_LRU_TELEMETRY), PROFDIR (torch profiler bind-mount + --profiler-config JSON). The script re-copies the harness (mtp_measure.py, bench_iter.py) into every fresh container — podman cp files die on container recreate.

• Measure ONLY via usage counters: mtp_measure.py (3 depths) or mtp_measure_short.py (166 ctx). decode = completion_tokens/(wall − ttft); chunk counting is INVALID under spec decode.
• Jobs: bash /home/dank/job.sh start NAME CMD… / wait NAME SECS 40 / out NAME (Remote Job Runner).
• Validity check on ANY speed result: completion_tokens == max_tokens (else repetition-halt or early EOS), finish_reason, and inspect the text — bench_iter prompts are repeated word-lists + ! padding, loop-prone. A loop makes LRU hits ~100% → PCIe traffic vanishes → fake 3–4× decode. Acceptance swings 48–61% run-to-run — noise; decode is the metric.
• Each arm = full reload: ~8.5 min cold, ~12 min with measurement.
• Readiness gate: podman logs <c> 2>&1 (stdout+stderr merged) for Application startup complete — it is on STDERR; an unmerged grep can never match and burns the whole wait timeout.
• GPU util ~0% during load/compile is NORMAL (weights + CUDA-graph capture are CPU-side); VRAM climbing is the load indicator.
• Config edit discipline: the serve script is patched by scp'd python scripts with asserts (anchor must exist, addition must not); bash -n after every patch.

Measured knob verdicts (2026-09-29 campaign)

Incumbent: SPEC=4 PREFETCH=1 UTIL=0.98 MAXLEN=32768 KV_BYTES=2147483648 TUNABLE=0, stock gather 8×16 → 26.80/25.15/26.81 tok/s @ 166/1087/4064 ctx.

Knob	Verdict	Evidence
Gather geometry R4D_LRU_GATHER_{CHUNKS,LANES}	RULED OUT	16×32 worse (~24); 32×64 tie (26.6). The 9.5 vs 22.7 GB/s read-path gap is per-transfer structure (~105 µs/slab vs 78 ideal), not a tunable knob
MTP depth	RULED OUT	SPEC=3 ~9% worse; SPEC=6 collapses (13.5–19 tok/s, acceptance 17.4%) — also proves UVA read-through (crosses at step tokens > R4D_LRU_UVA_TOKENS=5) is SLOWER than copy-in
TunableOp	REJECTED	CLAV_TUNABLEOP_SWEEP=1 silently SKIPPED under expert offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1; adds ~7 min/load, persists nothing (container-local), and CORRUPTS NUMERICS (loops, repetition-halt, fake 79.8 tok/s @ 96.5% acceptance on degenerate completions). Any "3× win" with early-EOS + absurd acceptance = the loop artifact
MNBT	neutral at 8192 vs 2048	small values re-chunk prefill only (hurts TTFT, never decode — decode step = max-num-seqs × (spec+1) tokens). No ubatch concept in vLLM

Decode on this stack is PCIe-gather-bound (~30% GPU duty cycle), so batch-shaped compute knobs move ~nothing.

The R4D_*/CLAV_* knob family (QSA_, MHC, ROUTE, ACT_QUANT, AR_QUANT, SILU_QUANT) are kill-switches defaulting ON, not perf dials; real levers are the R4D_LRU_ family, TunableOp (rejected), R4D_LRU_TELEMETRY, and spec depth.

Telemetry gotchas: --profiler-config takes a JSON blob, not separate flags; torch_profiler_with_stack defaults TRUE — a full measure inside a profile window OOM-kills the engine worker (~40 GB trace buffer; kernel kills python3 at ~95 GB anon-rss). Keep windows to ~15 s / one depth (mtp_measure_short.py). telemetry_report (expert_cache_rank<N>.json) fires ONLY from the worker's profiler-stop path; worker-side stop can hang

5 min on ROCm/kineto. No alternative trigger exists. The LRU hit rate measured 82.45% — NOT the ~27% the arena-fraction arithmetic implies; temporal reuse dominates, so never infer miss rates from resident-slot fractions.

Sweep hygiene (fork-specific preflight)

1. Knob landed: for each -e env knob, confirm in-container podman exec … env (or the engine log's confirmed-values line). A knob the fork silently ignores produces a "parity" arm that is really the default — the v1 gather sweep lost all three arms to exactly this (GATHER_C/GATHER_L silently dropped, all arms ran stock 8×16).
2. Prefetch uniformity: every arm in a sweep MUST carry the same expert-prefetch setting. The 2026-09-29 campaign measured losing arms with prefetch ON and incumbents OFF; untaxed, its verdicts inverted (no-spec 12.11→41.47; gather 32×64 parity→+18.7%). Never re-read a taxed verdict.
3. Spec arms must prove engagement: engine log SpecDecoding metrics: Mean acceptance length … and curl :7770/metrics | grep vllm:spec_decode_. Zero such lines = MTP off.
4. Boot-fail fast-detection: poll podman ps INSIDE the health loop; a refused boot dies in one poll instead of 20 min of health waits. Capture

   8 lines of podman logs on failure — the root cause (e.g. a SEQS init refusal) lives higher up.

5. Report per-depth tables, never one number: g32×64's win grows with depth (54@1k, 39.5@short) — a single 4064 median hides the short-ctx cost. TTFT rows are noisy ±0.15 s and non-monotonic across depths; compare same-ctx rows across arms only.
6. Generate ≥256 tokens per sample (at 8 tokens inter_token is first-token noise).

Boot-log facts survive measure failures: KV pool size (GPU KV cache size), arena slots (expert cache: … -> N slots), and the spec-reservation ladder (MTP3 766,666 / MTP4 729,545 / MTP6 665,263 tokens) print at BOOT — a measure-stage bug costs decode tables but not boot-log findings.

MTP depth-brittleness law (2026-10-02, all arms rc=0): MTP3 on this fork is depth-brittle — 47.01 t/s at 166 tokens (47.88 at temp 0.7) but 23.09 at 4064, vs no-spec baseline 37.02 / 33.99. MTP3 + SEQS 16 + MNBT 4096 + util 0.95 + ladder [4..32] = best overall 37.04 / 38.61 t0.7 and keeps 33.15 at 4064. --expert-precise-memory is load-bearing for MTP (same arm with it OFF = worst arm, 19.81 at 4064). Every spec arm costs KV (398,658 → 313,002 tokens). Acceptance is real: mean length 2.4–2.8, per-position 0.76/0.55/0.33, avg 52–57%. Next lever named by the engine: MNBT 4096 is exactly SEQS×(n+1) (suboptimal warning) → try 8192.

Serving a model larger than VRAM (dankserver recipe)

Dankserver: 2× R9700 (gfx1201), 31.86 GiB each = 63.7 GiB real VRAM, 246 GiB RAM. torch.cuda.device_count() returns 3 and summing total_memory gives a bogus 186.7 GiB (the Raphael iGPU is in the list — see Llama Cpp Serving Sizing for the reading discipline).

Kernel gate — this decides which checkpoint can work at all. /app/r4dhip/r4d.so exports gemm_moe_mxfp4a8_* and gemm_moe_dsfp4a8_* but NO gemm_moe_w4a16_*. Expert offload requires an experts class with BOTH a compiled kernel AND an offload_recipe; the planner aborts startup otherwise. Usable: MXFP4, DSFP4, FP8-HIP (R4dMxfp4MoEExperts, R4dDsfp4MoEExperts, Fp8HipMoEExperts). R4dW4a16MoEExperts has no kernel (the class exists, the r4d.so entry does not); RFA has neither. Check the checkpoint's quant against this list BEFORE downloading anything.

Entrypoint is /app/tools/image_entrypoint.sh, ending exec vllm serve "$@" — pass the model path as the FIRST argument and do NOT pass serve. Known-good invocation:

sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
podman run -d --name <ctr> --ipc=host --ulimit memlock=-1:-1 \
  --cap-add SYS_PTRACE \
  --device /dev/kfd --device /dev/dri --device /dev/mem \
  -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
  -v /home/dank/models/<model>:/models:ro \
  docker.io/tcclaviger/vllm:latest /models \
  --served-model-name <name> --port 7770 \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --kv-cache-memory-bytes 2147483648 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.95 \
  --enable-prefix-caching --trust-remote-code

-p 7770:7770 is not included, so the API is only reachable via podman exec — add -p if host-side curl is needed. Load ~5 min; /health answers connection-refused the whole time; watch for Application startup complete in podman logs <ctr>. Verified working: orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (gated; 172.8 GiB, 131 shards, Qwen4ExpForConditionalGeneration, 48 layers, 512 experts top-10, multimodal, native 262,144).

Traps that each cost a boot:

1. Mount the whole /dev/dri, never individual render nodes. Render nodes without the primary card* nodes → HIP enumerates two agents and refuses a context on either (RuntimeError: No CUDA GPUs are available) while device_count() cheerfully returns 2. Confirmed by direct A/B.
2. memlock must be raised (done on Dankserver via /etc/security/limits.d/99-memlock.conf). Host default 8 MiB makes any large pinned allocation fail as a misleading torch.AcceleratorError: CUDA error: out of memory from pin_memory(); it cannot be raised from inside a container or root's own shell — only PAM applies it at login. Matters because --expert-host-mlock defaults True.
3. --gpu-memory-utilization does NOT buy KV under expert offload — the planner sizes the arena against the budget and leaves KV the remainder; raising util 0.95→0.98 grew the arena and the KV pool stayed at 0.64 GiB.
4. Lowering --max-model-len is the WRONG lever and backfires — the budget subtracts the KV reserve first, so a smaller context shrinks the reserve and hands the space to the arena (measured: 65536 → 0.64 GiB KV; 32768 → 0.34 GiB).
5. The KV reserve undercounts hybrid-attention models ~2× (9190 B/token planned vs ~18,000 real: the Mamba state is a fixed per-sequence cost). Fix by pinning the pool: --kv-cache-memory-bytes.
6. --expert-offload-mem caps the HOST side, not the VRAM arena — structurally orthogonal to --expert-precise-memory (verified in-container 2026-10-05: precise-memory only swaps the GPU overhead term, expert_offload_mem only sets the host bound; they rejoin only inside place_experts, and the plan rewrites expert_offload_mem back to the cap the plan met, so plan and allocation cannot drift).
7. --ipc=host and --shm-size are mutually exclusive in podman; pick one.
8. podman run -d without -a sends no stdout to a redirect — the log file only receives the container ID; use podman logs.

Status reads: use scripts/dankstatus.sh, never hand-rolled /proc/meminfo parsing — /proc/meminfo separates key from value with a TAB, so grep MemAvailable | tr -s " " | cut -d" " -f2 returns empty and awk prints 0.0 — a false "host out of memory" that invites an unnecessary abort. Gated checkpoints: token via a mode-600 curl -K config, never argv (pgrep -a curl prints it); always --fail; compare each file's byte count to the Hub's declared size; -C - resumes truncated shards.

Check whether the stack supports the model natively before porting an overlay. The image already registered Glm5NextForConditionalGeneration, Qwen4ExpForCausalLM and both MTP variants — chasing four missing symbols through two image tags to bring an overlay up was entirely wasted work.

Compile-cache persistence (survives podman rm -f)

Serve scripts that recreate the container discard the writable layer, so every boot re-pays compile/JIT (frank-primary: torch.compile took 181.20 s cold, of which 173 s was one graph).

1. Prove it and locate it: fresh container vs live (podman run --rm --entrypoint bash IMG -c "ls -la /root/.cache" vs podman exec LIVE bash -c "du -sb /root/.cache/*; find /root/.cache -type f | wc -l"). vLLM cache root: VLLM_CACHE_ROOT (default /root/.cache/vllm, torch_compile_cache/); a HIP runtime adds /root/.cache/comgr.
2. Mount the PARENT, not the leaf — one bind on /root/.cache persists both trees, safe whenever the image ships no such directory (step 1 proves it; otherwise you would be hiding baked state). Wire as an opt-in tunable: mkdir -p "$CACHE_DIR" BEFORE the mount, then ${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}. Host path outside the checkout.
3. Verify with two consecutive boots — cold Compiling a graph … takes N s, warm Directly load the compiled graph(s) for compile range (…) from the cache with torch.compile took in single-digit seconds.
4. Traps: the cache key includes the compile RANGE — an MNBT change costs ONE recompile, not a permanent tax; do not claim a proportional boot cut (time to /health is dominated by weight loading: 490 s warm vs 188 s cold compile); stale entries are misses, not wrong answers (version + config hashed); if the serve script is seam-gated, assert the mkdir and the exact mount target.

Plan-line verification checklist (after any fork deploy)

ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-primary-qwen -b -o cat --no-pager \
  | grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'

grep -oE with a digit-anchored tail — plain grep/journalctl truncate these columns to 17.. Verify: new budget/graphs/arena in the plan; GPU KV cache size matches the requested token count; Estimated CUDA graph memory vs the capture line (raise _PER_GRAPH_MB toward 52 if capture exceeds the charge); rocm-smi --showmeminfo vram high-water per card = the real margin; systemctl show frank-primary -p NRestarts --value = 0; VLLM_CUDAGRAPH_* present in podman exec … env; then a real chat completion from a PEER — curl localhost:7770/health inside an ssh session on the host can return 000 and looks like an outage that isn't.

Related

• RDNA4 MoE Expert Offload VRAM Budget — the plan law this harness tunes
• vLLM Boot Forensics and Config Interrogation — journal forensics for these boots
• LLM Bench Windows and Measurement — window discipline and bash laws
• Model Swap and Rollback — where the fork sits in the swap history
• Remote Job Runner — the job.sh harness referenced above