serverAppConfigs/serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Heterogeneous Split Triage	Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X + LPDDR APU, PCIe x4 link) — link characterisation, the real bottleneck, what placement never buys, UMA pools, drafter splits, and the verdict ranking.	heterogeneous multi-device llama-cpp triage pcie	2026-10-05T00:00:00-07:00

Heterogeneous Split Triage

Produced the 2026-09-16 no-go verdict on splitting Qwen3.8-Flash-Next across the RTX 3090 + Strix Halo pair. Use it to answer "could we split model X across device A and device B" with arithmetic instead of enthusiasm.

1. Characterise the link (bytes vs latency)

lspci -vv -s <bdf> | grep -E "LnkCap:|LnkSta:"          # capability vs actual
nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.gen.current,pcie.link.width.current --format=csv

Expect idle-downshift: a Gen4 x4 dock idles at Gen1 x4 (~1 GB/s) and upshifts under load. Measure while the device is busy, or you size the design against the wrong number. Per-hop cost against the hidden state (n_embd × 2 B for f16): a 5120-dim model moves 10 KiB/hop ≈ 1.3 µs at Gen4 x4.

• Bytes per token ≈ hops/token × hidden state — typically ~1 MiB/token for 2 hops/layer × 64 layers ≈ 0.3 % of a 40 ms token. Bytes are almost never the constraint.
• The real budget line is hop COUNT: every device boundary is a host-side sync. A runtime that splits per op hits ~100 splits/token and pins the GPU at ~5 % SM. Measure graph splits in the loader log; never assume.
• Design rule: partition by where the weights live, never by where the activations flow. Resident weights, activations crossing.

2. Get the workload's real bottleneck (KV is usually it)

KV/token = kv_bytes_per_position × context

Attention reads the whole cached context every token, so KV/token scales with context — compare it against expert/weight traffic per token and total token time. At long context on a hybrid arch (few attention layers, many positions), KV dominates. Consequences:

• The highest-leverage placement is KV cache on the fastest-memory device, together with its attention compute — never KV alone: separating them puts KV/token over the link, which is fatal (GBs across 7.88 GB/s = hundreds of ms/token).
• Check the fit: kv_positions_per_slot × slots × bytes_per_position + attention_trunk ≤ VRAM.

3. What placement never buys

Prefill is compute-bound and expert weights are read once per batch, so moving them to faster memory buys nothing while adding a serial stage. Expect a regression, and size it from calibration: on Strix Halo + RTX 3090 (Qwen3.8-Flash-Next, ik_llama.cpp) moving 12.5 % of the expert mass to the GPU gave decode +12 %, prefill 192 → 32 tok/s. A 1/(f_big/BW_slow + f_fast/BW_fast) harmonic calculation reproduces that decode gain to within ~2 points — usable for decode projections, and it assumes sequential stages (true at batch 1; no pipelining).

4. Is the second device even a separate pool?

On a UMA APU, check before assuming additivity:

cat /sys/class/drm/card*/device/mem_info_vram_total   # BIOS carve-out (small)
cat /sys/class/drm/card*/device/mem_info_gtt_total    # aperture onto SYSTEM RAM
rocm-smi --showmeminfo all | grep -i gtt              # used == what the CPU also sees

Read on Dankclaw 2026-10-18: mem_info_vram_total = 2147483648 (2 GiB of dedicated carve), mem_info_gtt_total = 122402963456 (114 GiB) — the aperture is the whole budget, the carve is noise, and the OS sees 121 GiB of it (Dankclaw System Specs).

A large GTT window means the "GPU memory" is the same LPDDR under one memory controller: adding a stage there adds a compute engine, not capacity or bandwidth. Only the discrete card's VRAM is additive. Fencing more (amdgpu.gttsize, BIOS UMA size) just removes bytes from CPU-visible RAM.

5. Speculative decoding as the "cheap" split

A drafter on the other device is bandwidth-free (token ids per round, tens of bytes), so it is tempting. Before proposing it:

• the drafter needs the target's exact tokenizer for a plain draft model (compare GGUF tokenizer.ggml.tokens, never trust the family name), or it must be a head trained for that target (EAGLE/DFlash/MTP are conditioned on the target's hidden states and are not portable between models);
• a plain drafter costs K sequential forwards per round, so it must be 10–50× faster than the target. Break-even with r = t_draft/t_target: rate_ratio = A/(K·r + 1), A = (1−p^(K+1))/(1−p); at r ≈ 0.4–0.5 (a drafter only ~2× faster) break-even needs p ≈ 0.7+ at K=3;
• compare against the incumbent, not the no-draft baseline: a target already running its own MTP head is a high bar;
• check the runtime accepts an external drafter at all — closed engines may enumerate a fixed drafter set and 400 anything else (capability gates keyed on the loaded checkpoint). That is a design boundary, not a config gap. In llama.cpp the portable mechanism is --spec-draft-device/-devd (draft gets its own device list) plus -ngld/-otd/-ctkd -ctvd.

The llama.cpp half of that bullet is checkable upstream: --spec-draft-device (-devd / --device-draft, line 4259), --spec-draft-ngl (-ngld, 4268), --spec-draft-override-tensor (-otd, 4137) and --spec-draft-type-k/v (-ctkd/-ctvd, 4111/4124) all exist in common/arg.cpp (source, read 2026-10-18).

6. Expressing a split (llama.cpp, the usual vehicle)

-dev CUDA0,Vulkan0 -sm layer -ts F0,F1        # contiguous layer ranges (simplest)
-ot "\.attn_.*=CUDA0" -ot "\.exps\.=Vulkan0"   # per-tensor, by ROLE (KV co-location)

• buffer types match by name at load (ggml_backend_buft_name), so device names work as -ot values;
• -cmoe/-ncmoe are CPU-only (--cpu-moe / --n-cpu-moe, common/arg.cpp:2764,2771); device-side MoE placement is -ot;
• several backends in one binary is fine (CUDA + Vulkan is the practical pair; HIP + CUDA do not coexist (unverified 2026-10-18 — upstream build docs do not settle the combination either way)) — GPU_BACKEND_A=OFF in the existing build's CMakeCache tells you what needs adding;
• contiguous ranges are the WRONG split when KV bandwidth dominates: they drag half the attention layers (and their KV) back to the slow pool.

7. Hazards checklist

• Paged/sidecar tables (huge n-gram/embedding tables) must stay where they are read from; moving them is pure loss.
• Keep mmap on for models with huge embedding tables (anonymous memory gets the process cgroup-killed).
• CUDA graph capture can abort on unusual graph shapes (hyper-connections) — check before assuming optimisations apply.
• A brand-new backend on the second device may not implement the arch's ops at all: make the first measurement a baseline of the runtime on that device alone — a split's arithmetic is worthless if the baseline is worse than the incumbent engine.
• A split consumes the fast device entirely — check what else it was serving before trading it away.
• Benchmark discipline on production hosts: stop and mask first (Fleet Ops Playbook).

8. Verdict shape

Rank and report; prefer the boring answer when it wins:

1. Two independent endpoints (zero shared traffic) — often the real answer to "what's a useful combination", especially over a narrow link.
2. One process, weights-resident, KV co-located with attention on the fast pool — promising and quantifiable (+% decode at long context); needs a dual-backend build and a baseline measurement.
3. Hot-expert cache / per-token weight streaming — affordable only if the miss set per token is small (a few experts; check misses × bytes × tok/s against link bandwidth) and overlapped; injects PCIe latency into the critical path; no mainstream runtime implements a dynamic expert cache.
4. List explicitly what does NOT help: TP/collectives across unequal devices, KV on the far device, model loading over the link, prefill-motivated placements.

Related

• PCIe Link and Atomics — the link and collective walls underneath
• Llama Cpp Multi-Device Splits — the deployed split shapes
• Llama Cpp Serving Sizing — KV and pool arithmetic