command-center/agent/managed-skills/heterogeneous-accelerator-split-triage/SKILL.md

name, description

name	description
heterogeneous-accelerator-split-triage	Decide whether splitting one model across two unequal accelerators on a single host (fast small-VRAM GPU + big slow unified-memory device, or a GPU over a narrow PCIe link) is worth building — with the KV/prefill arithmetic, the probes that settle it, and the doors that are already closed.

Heterogeneous accelerator split — triage before building

Use when someone asks "could we split model X across device A and device B" on one host, when the devices differ in memory size/bandwidth (e.g. 24 GB GDDR6X vs an APU sharing LPDDR), or when a link is narrow (PCIe x4). Produces a go/no-go with arithmetic instead of enthusiasm.

1. Characterise the link (bytes vs latency)

lspci -vv -s <bdf> | grep -E "LnkCap:|LnkSta:"          # capability vs actual
nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.gen.current,pcie.link.width.current --format=csv

Expect idle-downshift: a Gen4 x4 dock can idle at Gen1 x4 (~1 GB/s) and only upshift under load. Measure while the device is busy, or you will size the design against the wrong number.

Compute the per-hop cost against the hidden state (n_embd x 2 B for f16): a 5120-dim model moves 10 KiB/hop = ~1.3 µs at Gen4 x4. Then:

• bytes per token = hops/token x hidden state. Typically ~1 MiB/token for 2 hops/layer x 64 layers -> ~0.3 % of a 40 ms token. Bytes are almost never the constraint.
• the real budget line is hop COUNT: every device boundary is a host-side sync. A runtime that splits per op can hit 100 splits/token and pin the GPU at ~5 % SM. A runtime that batches per-backend graphs is far cheaper — measure graph splits in the loader log, do not assume.

Design rule: partition by where the weights live, never by where the activations flow. Resident weights, activations crossing.

2. Get the workload's real bottleneck (KV is usually it)

KV per position x context = bytes read per token, because attention reads the whole cached context every token:

KV/token = kv_bytes_per_position * context

Compare that against (a) expert/weight traffic per token (read once per token per layer) and (b) total token time. At long context or on a hybrid model with few attention layers but many positions, KV dominates. Then:

• the highest-leverage placement is KV cache on the fastest-memory device, together with its attention compute — never KV alone (separating them puts KV/token over the link, which is fatal: GBs across a 7.88 GB/s link = hundreds of ms/token);
• check the KV + attention weights actually fit: kv_positions_per_slot * slots * bytes_per_position + attention_trunk <= VRAM.

3. What placement never buys

Prefill is compute-bound and expert weights are read once per batch, so moving them to faster memory buys nothing while adding a serial stage. Expect a regression, and size it from calibration: on Strix Halo + RTX 3090 (Qwen3.8-Flash-Next, ik_llama.cpp) moving 12.5 % of the expert mass to the GPU gave decode +12 %, prefill 192 -> 32 tok/s.

A 1/(f_big/BW_slow + f_fast/BW_fast) harmonic calculation reproduces that decode gain to within ~2 points, so it is usable for decode projections — and it assumes the stages are sequential (true at batch 1: no pipelining).

4. Is the second device even a separate pool?

On a UMA APU, check before assuming additivity:

cat /sys/class/drm/card*/device/mem_info_vram_total   # BIOS carve-out (small)
cat /sys/class/drm/card*/device/mem_info_gtt_total    # aperture onto SYSTEM RAM
rocm-smi --showmeminfo all | grep -i gtt              # used == what the CPU also sees

A large GTT window means the "GPU memory" is the same LPDDR under one memory controller: adding a stage there adds a compute engine, not capacity or bandwidth. Only the discrete card's VRAM is additive. Fencing more (amdgpu.gttsize, BIOS UMA size) just removes bytes from CPU-visible RAM.

5. Speculative decoding as the "cheap" split

A drafter on the other device is bandwidth-free (token ids per round, tens of bytes), so it is tempting. Before proposing it:

• the drafter needs the target's exact tokenizer for a plain draft model (verify by comparing GGUF tokenizer.ggml.tokens, not by trusting the family name), or it must be a head trained for that target (EAGLE/DFlash/MTP are conditioned on the target's hidden states and are not portable between models);
• a plain drafter costs K sequential forwards per round, so it must be 10-50x faster than the target. Compute the break-even: with r = t_draft/t_target, rate_ratio = A/(K*r + 1), A = (1-p^(K+1))/(1-p). At r ≈ 0.4-0.5 (a drafter only ~2x faster) break-even needs p ≈ 0.7+ at K=3;
• always compare against the incumbent, not the no-draft baseline: a target already running its own MTP head (e.g. ~1.7 tokens committed/round) is a high bar;
• check the runtime can even accept an external drafter. Closed engines may enumerate a fixed drafter set and 400 anything else (DRAFTER_CAP-style capability gates keyed on the loaded checkpoint) — that is a design boundary, not a config gap. In llama.cpp the portable mechanism is --spec-draft-device/-devd (draft gets its own device list) plus -ngld/-otd/-ctkd-ctvd.

6. Expressing a split (llama.cpp, the usual vehicle)

-dev CUDA0,Vulkan0 -sm layer -ts F0,F1        # contiguous layer ranges (simplest)
-ot "\.attn_.*=CUDA0" -ot "\.exps\.=Vulkan0"   # per-tensor, by ROLE (what KV co-location needs)

• buffer types are matched by name at load (ggml_backend_buft_name), so device names work as -ot values;
• -cmoe / -ncmoe are CPU-only; device-side MoE placement is -ot;
• several backends in one binary is fine (CUDA + Vulkan is the practical pair; HIP + CUDA do not coexist), and GPU_BACKEND_A=OFF in the CMakeCache of the existing build tells you what needs adding;
• contiguous ranges are the wrong split when KV bandwidth dominates, because they drag half the attention layers (and their KV) back to the slow pool.

7. Hazards checklist

• paged/sidecar tables (huge n-gram/embedding tables) must stay where they are read from; moving them is pure loss;
• keep mmap on for models with huge embedding tables (anonymous memory gets the process cgroup-killed);
• CUDA graph capture can abort on unusual graph shapes (hyper-connections etc.) — check before assuming optimisations apply;
• the second device's brand-new backend may not implement the arch's ops at all: make the first measurement a baseline of the runtime on that device alone, because a split's arithmetic is worthless if the baseline is worse than the incumbent engine;
• a split consumes the fast device entirely — check what else it was serving before trading it away;
• benchmark/experiment discipline on production hosts: stop and mask first.

8. Verdict shape

Rank and report, and prefer the boring answer when it wins:

1. Two independent endpoints (zero shared traffic) — often the real answer to "what's a useful combination", especially over a narrow link;
2. One process, weights-resident, KV co-located with attention on the fast pool — promising, quantifiable (+% decode at long context), needs a dual-backend build and a baseline measurement;
3. Hot-expert cache / per-token weight streaming — affordable only if the miss set per token is small (a few experts; check misses x bytes x tok/s against link bandwidth) and overlapped, and it injects PCIe latency into the critical path; no mainstream runtime implements a dynamic expert cache;
4. list explicitly what does not help: TP/collectives across unequal devices, KV on the far device, model loading over the link, prefill-motivated placements.