serverAppConfigs/serverAppConfigs-docs/LLM/Tcclaviger Fork Ops.md

---
type: reference
title: Tcclaviger Fork Ops
description: Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken cutover) — the ad-hoc sweep harness, measured knob verdicts, sweep hygiene, the large-MoE serving recipe, and compile-cache persistence.
tags: [tcclaviger, vllm, dankserver, expert-offload, sweeps, cache]
timestamp: 2026-10-05T00:00:00-07:00
---

# Tcclaviger Fork Ops

The tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
vLLM 0.29.0.dev, GLM-5.3/Qwen-specialised with native MoE quant kernels and a
first-class expert-offload planner) served `frank-archon` at shape G until
the 2026-10-05 FreeToken cutover; it remains the rollback target
(`git tag frank-archon-vllm-shapeG-20261003`). Full plan law and VRAM
arithmetic: [[RDNA4 MoE Expert Offload VRAM Budget]]; boot forensics:
[[vLLM Boot Forensics and Config Interrogation]]; window discipline:
[[LLM Bench Windows and Measurement]].

## The ad-hoc sweep harness (dankserver)

Serve script `/home/dank/serve_qwen_fp8.sh` with env hooks: `SPEC` (MTP
depth), `TUNABLE` (TunableOp), `PREFETCH`, `UTIL`, `MAXLEN`, `KV_BYTES`,
`MNBT`, `UVA_TOK`, `GATHER_C/GATHER_L` (R4D_LRU gather), `TELEM`
(R4D_LRU_TELEMETRY), `PROFDIR` (torch profiler bind-mount + --profiler-config
JSON). The script re-copies the harness (`mtp_measure.py`, `bench_iter.py`)
into every fresh container — `podman cp` files die on container recreate.

- Measure ONLY via usage counters: `mtp_measure.py` (3 depths) or
  `mtp_measure_short.py` (166 ctx). `decode = completion_tokens/(wall −
  ttft)`; chunk counting is INVALID under spec decode.
- Jobs: `bash /home/dank/job.sh start NAME CMD…` / `wait NAME SECS 40` /
  `out NAME` ([[Remote Job Runner]]).
- **Validity check on ANY speed result:** `completion_tokens == max_tokens`
  (else repetition-halt or early EOS), finish_reason, and inspect the text —
  `bench_iter` prompts are repeated word-lists + `!` padding, loop-prone. A
  loop makes LRU hits ~100% → PCIe traffic vanishes → fake 3–4× decode.
  Acceptance swings 48–61% run-to-run — noise; decode is the metric.
- Each arm = full reload: ~8.5 min cold, ~12 min with measurement.
- Readiness gate: `podman logs <c> 2>&1` (stdout+stderr merged) for
  `Application startup complete` — it is on STDERR; an unmerged grep can
  never match and burns the whole wait timeout.
- GPU util ~0% during load/compile is NORMAL (weights + CUDA-graph capture
  are CPU-side); VRAM climbing is the load indicator.
- Config edit discipline: the serve script is patched by scp'd python
  scripts with asserts (anchor must exist, addition must not); `bash -n`
  after every patch.

## Measured knob verdicts (2026-09-29 campaign)

Incumbent: `SPEC=4 PREFETCH=1 UTIL=0.98 MAXLEN=32768 KV_BYTES=2147483648
TUNABLE=0`, stock gather 8×16 → 26.80/25.15/26.81 tok/s @ 166/1087/4064 ctx.

| Knob | Verdict | Evidence |
|---|---|---|
| Gather geometry `R4D_LRU_GATHER_{CHUNKS,LANES}` | **RULED OUT** | 16×32 worse (~24); 32×64 tie (26.6). The 9.5 vs 22.7 GB/s read-path gap is per-transfer structure (~105 µs/slab vs 78 ideal), not a tunable knob |
| MTP depth | **RULED OUT** | SPEC=3 ~9% worse; SPEC=6 collapses (13.5–19 tok/s, acceptance 17.4%) — also proves UVA read-through (crosses at step tokens > `R4D_LRU_UVA_TOKENS=5`) is SLOWER than copy-in |
| TunableOp | **REJECTED** | `CLAV_TUNABLEOP_SWEEP=1` silently SKIPPED under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1`; adds ~7 min/load, persists nothing (container-local), and CORRUPTS NUMERICS (loops, repetition-halt, fake 79.8 tok/s @ 96.5% acceptance on degenerate completions). Any "3× win" with early-EOS + absurd acceptance = the loop artifact |
| MNBT | neutral at 8192 vs 2048 | small values re-chunk prefill only (hurts TTFT, never decode — decode step = max-num-seqs × (spec+1) tokens). No ubatch concept in vLLM |

Decode on this stack is PCIe-gather-bound (~30% GPU duty cycle), so
batch-shaped compute knobs move ~nothing.

The `R4D_*/CLAV_*` knob family (QSA_*, MHC, ROUTE, ACT_QUANT, AR_QUANT,
SILU_QUANT) are kill-switches defaulting ON, not perf dials; real levers are
the R4D_LRU_* family, TunableOp (rejected), R4D_LRU_TELEMETRY, and spec
depth.

Telemetry gotchas: `--profiler-config` takes a JSON blob, not separate
flags; `torch_profiler_with_stack` defaults TRUE — a full measure inside a
profile window OOM-kills the engine worker (~40 GB trace buffer; kernel
kills python3 at ~95 GB anon-rss). Keep windows to ~15 s / one depth
(`mtp_measure_short.py`). `telemetry_report` (`expert_cache_rank<N>.json`)
fires ONLY from the worker's profiler-stop path; worker-side stop can hang
>5 min on ROCm/kineto. No alternative trigger exists. The LRU hit rate
measured 82.45% — NOT the ~27% the arena-fraction arithmetic implies;
temporal reuse dominates, so never infer miss rates from resident-slot
fractions.

## Sweep hygiene (fork-specific preflight)

1. **Knob landed:** for each `-e` env knob, confirm in-container
   `podman exec … env` (or the engine log's confirmed-values line). A knob
   the fork silently ignores produces a "parity" arm that is really the
   default — the v1 gather sweep lost all three arms to exactly this
   (GATHER_C/GATHER_L silently dropped, all arms ran stock 8×16).
2. **Prefetch uniformity:** every arm in a sweep MUST carry the same
   expert-prefetch setting. The 2026-09-29 campaign measured losing arms
   with prefetch ON and incumbents OFF; untaxed, its verdicts inverted
   (no-spec 12.11→41.47; gather 32×64 parity→+18.7%). Never re-read a taxed
   verdict.
3. **Spec arms must prove engagement:** engine log
   `SpecDecoding metrics: Mean acceptance length …` and
   `curl :7770/metrics | grep vllm:spec_decode_`. Zero such lines = MTP off.
4. Boot-fail fast-detection: poll `podman ps` INSIDE the health loop; a
   refused boot dies in one poll instead of 20 min of health waits. Capture
   >8 lines of `podman logs` on failure — the root cause (e.g. a SEQS init
   refusal) lives higher up.
5. Report per-depth tables, never one number: g32×64's win grows with depth
   (54@1k, 39.5@short) — a single 4064 median hides the short-ctx cost. TTFT
   rows are noisy ±0.15 s and non-monotonic across depths; compare same-ctx
   rows across arms only.
6. Generate ≥256 tokens per sample (at 8 tokens inter_token is first-token
   noise).

Boot-log facts survive measure failures: KV pool size (`GPU KV cache size`),
arena slots (`expert cache: … -> N slots`), and the spec-reservation ladder
(MTP3 766,666 / MTP4 729,545 / MTP6 665,263 tokens) print at BOOT — a
measure-stage bug costs decode tables but not boot-log findings.

**MTP depth-brittleness law (2026-10-02, all arms rc=0):** MTP3 on this fork
is depth-brittle — 47.01 t/s at 166 tokens (47.88 at temp 0.7) but 23.09 at
4064, vs no-spec baseline 37.02 / 33.99. MTP3 + SEQS 16 + MNBT 4096 + util
0.95 + ladder [4..32] = best overall 37.04 / 38.61 t0.7 and keeps 33.15 at
4064. `--expert-precise-memory` is load-bearing for MTP (same arm with it
OFF = worst arm, 19.81 at 4064). Every spec arm costs KV (398,658 → 313,002
tokens). Acceptance is real: mean length 2.4–2.8, per-position
0.76/0.55/0.33, avg 52–57%. Next lever named by the engine: MNBT 4096 is
exactly SEQS×(n+1) (`suboptimal` warning) → try 8192.

## Serving a model larger than VRAM (dankserver recipe)

Dankserver: 2× R9700 (gfx1201), 31.86 GiB each = 63.7 GiB real VRAM, 246 GiB
RAM. `torch.cuda.device_count()` returns 3 and summing `total_memory` gives
a bogus 186.7 GiB (the Raphael iGPU is in the list — see
[[Llama Cpp Serving Sizing]] for the reading discipline).

**Kernel gate — this decides which checkpoint can work at all.**
`/app/r4dhip/r4d.so` exports `gemm_moe_mxfp4a8_*` and `gemm_moe_dsfp4a8_*`
but NO `gemm_moe_w4a16_*`. Expert offload requires an experts class with
BOTH a compiled kernel AND an `offload_recipe`; the planner aborts startup
otherwise. Usable: MXFP4, DSFP4, FP8-HIP (`R4dMxfp4MoEExperts`,
`R4dDsfp4MoEExperts`, `Fp8HipMoEExperts`). `R4dW4a16MoEExperts` has no
kernel (the class exists, the r4d.so entry does not); RFA has neither. Check
the checkpoint's quant against this list BEFORE downloading anything.

Entrypoint is `/app/tools/image_entrypoint.sh`, ending `exec vllm serve
"$@"` — pass the model path as the FIRST argument and do NOT pass `serve`.
Known-good invocation:

```bash
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
podman run -d --name <ctr> --ipc=host --ulimit memlock=-1:-1 \
  --cap-add SYS_PTRACE \
  --device /dev/kfd --device /dev/dri --device /dev/mem \
  -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
  -v /home/dank/models/<model>:/models:ro \
  docker.io/tcclaviger/vllm:latest /models \
  --served-model-name <name> --port 7770 \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --kv-cache-memory-bytes 2147483648 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.95 \
  --enable-prefix-caching --trust-remote-code
```

`-p 7770:7770` is not included, so the API is only reachable via
`podman exec` — add `-p` if host-side curl is needed. Load ~5 min;
`/health` answers connection-refused the whole time; watch for
`Application startup complete` in `podman logs <ctr>`. Verified working:
`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (gated; 172.8 GiB, 131
shards, `Qwen4ExpForConditionalGeneration`, 48 layers, 512 experts top-10,
multimodal, native 262,144).

Traps that each cost a boot:

1. **Mount the whole `/dev/dri`, never individual render nodes.** Render
   nodes without the primary `card*` nodes → HIP enumerates two agents and
   refuses a context on either (`RuntimeError: No CUDA GPUs are available`)
   while `device_count()` cheerfully returns 2. Confirmed by direct A/B.
2. **memlock must be raised** (done on Dankserver via
   `/etc/security/limits.d/99-memlock.conf`). Host default 8 MiB makes any
   large pinned allocation fail as a misleading
   `torch.AcceleratorError: CUDA error: out of memory` from `pin_memory()`;
   it cannot be raised from inside a container or root's own shell — only
   PAM applies it at login. Matters because `--expert-host-mlock` defaults
   True.
3. **`--gpu-memory-utilization` does NOT buy KV under expert offload** —
   the planner sizes the arena against the budget and leaves KV the
   remainder; raising util 0.95→0.98 grew the arena and the KV pool stayed
   at 0.64 GiB.
4. **Lowering `--max-model-len` is the WRONG lever and backfires** — the
   budget subtracts the KV reserve first, so a smaller context shrinks the
   reserve and hands the space to the arena (measured: 65536 → 0.64 GiB KV;
   32768 → 0.34 GiB).
5. **The KV reserve undercounts hybrid-attention models ~2×** (9190 B/token
   planned vs ~18,000 real: the Mamba state is a fixed per-sequence cost).
   Fix by pinning the pool: `--kv-cache-memory-bytes`.
6. **`--expert-offload-mem` caps the HOST side**, not the VRAM arena —
   structurally orthogonal to `--expert-precise-memory` (verified
   in-container 2026-10-05: precise-memory only swaps the GPU overhead term,
   expert_offload_mem only sets the host bound; they rejoin only inside
   `place_experts`, and the plan rewrites `expert_offload_mem` back to the
   cap the plan met, so plan and allocation cannot drift).
7. `--ipc=host` and `--shm-size` are mutually exclusive in podman; pick one.
8. `podman run -d` without `-a` sends no stdout to a redirect — the log file
   only receives the container ID; use `podman logs`.

Status reads: use `scripts/dankstatus.sh`, never hand-rolled
`/proc/meminfo` parsing — `/proc/meminfo` separates key from value with a
TAB, so `grep MemAvailable | tr -s " " | cut -d" " -f2` returns empty and
awk prints `0.0` — a false "host out of memory" that invites an unnecessary
abort. Gated checkpoints: token via a mode-600 `curl -K` config, never argv
(`pgrep -a curl` prints it); always `--fail`; compare each file's byte count
to the Hub's declared size; `-C -` resumes truncated shards.

**Check whether the stack supports the model natively before porting an
overlay.** The image already registered `Glm5NextForConditionalGeneration`,
`Qwen4ExpForCausalLM` and both MTP variants — chasing four missing symbols
through two image tags to bring an overlay up was entirely wasted work.

## Compile-cache persistence (survives `podman rm -f`)

Serve scripts that recreate the container discard the writable layer, so
every boot re-pays compile/JIT (frank-archon: `torch.compile took 181.20 s`
cold, of which 173 s was one graph).

1. **Prove it and locate it:** fresh container vs live
   (`podman run --rm --entrypoint bash IMG -c "ls -la /root/.cache"` vs
   `podman exec LIVE bash -c "du -sb /root/.cache/*; find /root/.cache -type
   f | wc -l"`). vLLM cache root: `VLLM_CACHE_ROOT` (default
   `/root/.cache/vllm`, `torch_compile_cache/`); a HIP runtime adds
   `/root/.cache/comgr`.
2. **Mount the PARENT, not the leaf** — one bind on `/root/.cache` persists
   both trees, safe whenever the image ships no such directory (step 1
   proves it; otherwise you would be hiding baked state). Wire as an opt-in
   tunable: `mkdir -p "$CACHE_DIR"` BEFORE the mount, then
   `${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}`. Host path outside the
   checkout.
3. **Verify with two consecutive boots** — cold `Compiling a graph … takes
   N s`, warm `Directly load the compiled graph(s) for compile range (…)
   from the cache` with `torch.compile took` in single-digit seconds.
4. Traps: the cache key includes the compile RANGE — an MNBT change costs
   ONE recompile, not a permanent tax; do not claim a proportional boot cut
   (time to `/health` is dominated by weight loading: 490 s warm vs 188 s
   cold compile); stale entries are misses, not wrong answers (version +
   config hashed); if the serve script is seam-gated, assert the mkdir and
   the exact mount target.

## Plan-line verification checklist (after any fork deploy)

```
ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-archon-qwen -b -o cat --no-pager \
  | grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'
```

`grep -oE` with a digit-anchored tail — plain `grep`/journalctl truncate
these columns to `17.`. Verify: new budget/graphs/arena in the plan;
`GPU KV cache size` matches the requested token count; `Estimated CUDA graph
memory` vs the capture line (raise `_PER_GRAPH_MB` toward 52 if capture
exceeds the charge); `rocm-smi --showmeminfo vram` high-water per card = the
real margin; `systemctl show frank-archon -p NRestarts --value` = 0;
`VLLM_CUDAGRAPH_*` present in `podman exec … env`; then a real chat
completion from a PEER — `curl localhost:7770/health` inside an ssh session
on the host can return `000` and looks like an outage that isn't.

## Related

- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law this harness tunes
- [[vLLM Boot Forensics and Config Interrogation]] — journal forensics for these boots
- [[LLM Bench Windows and Measurement]] — window discipline and bash laws
- [[Model Swap and Rollback]] — where the fork sits in the swap history
- [[Remote Job Runner]] — the job.sh harness referenced above