serverAppConfigs/serverAppConfigs-docs/LLM/Tcclaviger Fork Ops.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Tcclaviger Fork Ops	Operating the RDNA4 vLLM fork: the sweep harness, measured knob verdicts, sweep hygiene, compile-cache persistence.	tcclaviger vllm dankserver expert-offload sweeps cache	2026-10-05T00:00:00-07:00

Tcclaviger Fork Ops

The tcclaviger RDNA4 vLLM fork (docker.io/tcclaviger/vllm@sha256:da4029666…, vLLM 0.29.0.dev, GLM-5.3/Qwen-specialised with native MoE quant kernels and a first-class expert-offload planner) served frank-archon at shape G until the 2026-10-05 FreeToken cutover; it remains the rollback target (git tag frank-archon-vllm-shapeG-20261003). Full plan law and VRAM arithmetic: RDNA4 MoE Expert Offload VRAM Budget; boot forensics: vLLM Boot Forensics and Config Interrogation; window discipline: LLM Bench Windows and Measurement.

The ad-hoc sweep harness (dankserver)

Serve script /home/dank/serve_qwen_fp8.sh with env hooks: SPEC (MTP depth), TUNABLE (TunableOp), PREFETCH, UTIL, MAXLEN, KV_BYTES, MNBT, UVA_TOK, GATHER_C/GATHER_L (R4D_LRU gather), TELEM (R4D_LRU_TELEMETRY), PROFDIR (torch profiler bind-mount + --profiler-config JSON). The script re-copies the harness (mtp_measure.py, bench_iter.py) into every fresh container — podman cp files die on container recreate.

• Measure ONLY via usage counters: mtp_measure.py (3 depths) or mtp_measure_short.py (166 ctx). decode = completion_tokens/(wall − ttft); chunk counting is INVALID under spec decode.
• Jobs: bash /home/dank/job.sh start NAME CMD… / wait NAME SECS 40 / out NAME (Remote Job Runner).
• Validity check on ANY speed result: completion_tokens == max_tokens (else repetition-halt or early EOS), finish_reason, and inspect the text — bench_iter prompts are repeated word-lists + ! padding, loop-prone. A loop makes LRU hits ~100% → PCIe traffic vanishes → fake 3–4× decode. Acceptance swings 48–61% run-to-run — noise; decode is the metric.
• Each arm = full reload: ~8.5 min cold, ~12 min with measurement.
• Readiness gate: podman logs <c> 2>&1 (stdout+stderr merged) for Application startup complete — it is on STDERR; an unmerged grep can never match and burns the whole wait timeout.
• GPU util ~0% during load/compile is NORMAL (weights + CUDA-graph capture are CPU-side); VRAM climbing is the load indicator.
• Config edit discipline: the serve script is patched by scp'd python scripts with asserts (anchor must exist, addition must not); bash -n after every patch.

Measured knob verdicts (2026-09-29 campaign)

Incumbent: SPEC=4 PREFETCH=1 UTIL=0.98 MAXLEN=32768 KV_BYTES=2147483648 TUNABLE=0, stock gather 8×16 → 26.80/25.15/26.81 tok/s @ 166/1087/4064 ctx.

Knob	Verdict	Evidence
Gather geometry R4D_LRU_GATHER_{CHUNKS,LANES}	RULED OUT	16×32 worse (~24); 32×64 tie (26.6). The 9.5 vs 22.7 GB/s read-path gap is per-transfer structure (~105 µs/slab vs 78 ideal), not a tunable knob
MTP depth	RULED OUT	SPEC=3 ~9% worse; SPEC=6 collapses (13.5–19 tok/s, acceptance 17.4%) — also proves UVA read-through (crosses at step tokens > R4D_LRU_UVA_TOKENS=5) is SLOWER than copy-in
TunableOp	REJECTED	CLAV_TUNABLEOP_SWEEP=1 silently SKIPPED under expert offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1; adds ~7 min/load, persists nothing (container-local), and CORRUPTS NUMERICS (loops, repetition-halt, fake 79.8 tok/s @ 96.5% acceptance on degenerate completions). Any "3× win" with early-EOS + absurd acceptance = the loop artifact
MNBT	neutral at 8192 vs 2048	small values re-chunk prefill only (hurts TTFT, never decode — decode step = max-num-seqs × (spec+1) tokens). No ubatch concept in vLLM

Decode on this stack is PCIe-gather-bound (~30% GPU duty cycle), so batch-shaped compute knobs move ~nothing. Measured with no spec: 12.11 tok/s FLAT from 304 to 16,095 context tokens — KV attention is not the cost, so a context-slope hunt on this stack is null by construction.

The R4D_*/CLAV_* knob family (QSA_, MHC, ROUTE, ACT_QUANT, AR_QUANT, SILU_QUANT) are kill-switches defaulting ON, not perf dials; real levers are the R4D_LRU_ family, TunableOp (rejected), R4D_LRU_TELEMETRY, and spec depth.

Telemetry gotchas: --profiler-config takes a JSON blob, not separate flags; torch_profiler_with_stack defaults TRUE — a full measure inside a profile window OOM-kills the engine worker (~40 GB trace buffer; kernel kills python3 at ~95 GB anon-rss). Keep windows to ~15 s / one depth (mtp_measure_short.py). telemetry_report (expert_cache_rank<N>.json) fires ONLY from the worker's profiler-stop path; worker-side stop can hang

5 min on ROCm/kineto. No alternative trigger exists. The LRU hit rate measured 82.45% — NOT the ~27% the arena-fraction arithmetic implies; temporal reuse dominates, so never infer miss rates from resident-slot fractions.

Sweep hygiene (fork-specific preflight)

1. Knob landed: for each -e env knob, confirm in-container podman exec … env (or the engine log's confirmed-values line). A knob the fork silently ignores produces a "parity" arm that is really the default — the v1 gather sweep lost all three arms to exactly this (GATHER_C/GATHER_L silently dropped, all arms ran stock 8×16).
2. Prefetch uniformity: every arm in a sweep MUST carry the same expert-prefetch setting. The 2026-09-29 campaign measured losing arms with prefetch ON and incumbents OFF; untaxed, its verdicts inverted (no-spec 12.11→41.47; gather 32×64 parity→+18.7%). Never re-read a taxed verdict.
3. Spec arms must prove engagement: engine log SpecDecoding metrics: Mean acceptance length … and curl :7770/metrics | grep vllm:spec_decode_. Zero such lines = MTP off.
4. Boot-fail fast-detection: poll podman ps INSIDE the health loop; a refused boot dies in one poll instead of 20 min of health waits. Capture

   8 lines of podman logs on failure — the root cause (e.g. a SEQS init refusal) lives higher up.

5. Report per-depth tables, never one number: g32×64's win grows with depth (54@1k, 39.5@short) — a single 4064 median hides the short-ctx cost. TTFT rows are noisy ±0.15 s and non-monotonic across depths; compare same-ctx rows across arms only.
6. Generate ≥256 tokens per sample (at 8 tokens inter_token is first-token noise).

Boot-log facts survive measure failures: KV pool size (GPU KV cache size), arena slots (expert cache: … -> N slots), and the spec-reservation ladder (MTP3 766,666 / MTP4 729,545 / MTP6 665,263 tokens) print at BOOT — a measure-stage bug costs decode tables but not boot-log findings.

MTP depth-brittleness law (2026-10-02, all arms rc=0): MTP3 on this fork is depth-brittle — 47.01 t/s at 166 tokens (47.88 at temp 0.7) but 23.09 at 4064, vs no-spec baseline 37.02 / 33.99. MTP3 + SEQS 16 + MNBT 4096 + util 0.95 + ladder [4..32] = best overall 37.04 / 38.61 t0.7 and keeps 33.15 at 4064. --expert-precise-memory is load-bearing for MTP (same arm with it OFF = worst arm, 19.81 at 4064). Every spec arm costs KV (398,658 → 313,002 tokens). Acceptance is real: mean length 2.4–2.8, per-position 0.76/0.55/0.33, avg 52–57%. Next lever named by the engine: MNBT 4096 is exactly SEQS×(n+1) (suboptimal warning) → try 8192.

Serving a model larger than VRAM (dankserver recipe)

Dankserver's card count, per-card VRAM and host RAM are in Dankserver System Specs; torch.cuda.device_count() returns 3 on this box and summing total_memory overstates it badly (the Raphael iGPU is in the list — reading discipline: Llama Cpp Serving Sizing).

Everything else about running this image — the quant-kernel gate that decides which checkpoints can serve at all, the entrypoint contract, the known-good invocation, the /dev/dri and memlock traps, the util and max-model-len arena traps, the --ipc=host / -d stdout traps and the gated-fetch discipline — is written once on RDNA4 Container Serving, and the plan arithmetic behind the arena/KV traps is RDNA4 MoE Expert Offload VRAM Budget's. Three findings stay here because this harness produced them:

• The KV reserve undercounts hybrid-attention models ~2× (the planner's B/token figure against ~18,000 real: the Mamba state is a fixed per-sequence cost). Fix by pinning the pool: --kv-cache-memory-bytes.
• --expert-offload-mem caps the HOST side, not the VRAM arena — structurally orthogonal to --expert-precise-memory (verified in-container 2026-10-05: precise-memory only swaps the GPU overhead term, expert_offload_mem only sets the host bound; they rejoin only inside place_experts, and the plan rewrites expert_offload_mem back to the cap the plan met, so plan and allocation cannot drift).
• Check whether the stack supports the model natively before porting an overlay. The image already registered Glm5NextForConditionalGeneration, Qwen4ExpForCausalLM and both MTP variants — chasing four missing symbols through two image tags to bring an overlay up was entirely wasted work.

The checkpoint this recipe was verified on is orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (gated; 172.8 GiB, 131 shards, Qwen4ExpForConditionalGeneration, 48 layers, 512 experts top-10, multimodal, native 262,144) — an frank-archon era payload.

Compile-cache persistence (survives podman rm -f)

Serve scripts that recreate the container discard the writable layer, so every boot re-pays compile/JIT (frank-archon: torch.compile took 181.20 s cold, of which 173 s was one graph).

1. Prove it and locate it: fresh container vs live (podman run --rm --entrypoint bash IMG -c "ls -la /root/.cache" vs podman exec LIVE bash -c "du -sb /root/.cache/*; find /root/.cache -type f | wc -l"). vLLM cache root: VLLM_CACHE_ROOT (default /root/.cache/vllm, torch_compile_cache/); a HIP runtime adds /root/.cache/comgr.
2. Mount the PARENT, not the leaf — one bind on /root/.cache persists both trees, safe whenever the image ships no such directory (step 1 proves it; otherwise you would be hiding baked state). Wire as an opt-in tunable: mkdir -p "$CACHE_DIR" BEFORE the mount, then ${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}. Host path outside the checkout.
3. Verify with two consecutive boots — cold Compiling a graph … takes N s, warm Directly load the compiled graph(s) for compile range (…) from the cache with torch.compile took in single-digit seconds.
4. Traps: the cache key includes the compile RANGE — an MNBT change costs ONE recompile, not a permanent tax; do not claim a proportional boot cut (time to /health is dominated by weight loading: 490 s warm vs 188 s cold compile); stale entries are misses, not wrong answers (version + config hashed); if the serve script is seam-gated, assert the mkdir and the exact mount target.

Plan-line verification checklist (after any fork deploy)

The journal one-liner that reads the plan lines (digit-anchored grep -oE, because plain grep truncates those columns to 17.) and the post-deploy verification order — plan vs captured graph memory, GPU KV cache size vs the ask, per-card rocm-smi high-water, NRestarts = 0, VLLM_CUDAGRAPH_* present in the container env, then a real completion from a PEER because curl localhost:7770/health inside an ssh session on the host can return 000 and look like an outage — are step 3 of RDNA4 MoE Expert Offload VRAM Budget §7. Run that; do not re-derive it here.

Related

• RDNA4 MoE Expert Offload VRAM Budget — the plan law this harness tunes
• vLLM Boot Forensics and Config Interrogation — journal forensics for these boots
• LLM Bench Windows and Measurement — window discipline and bash laws
• Model Swap and Rollback — where the fork sits in the swap history
• Remote Job Runner — the job.sh harness referenced above