---
name: heterogeneous-accelerator-split-triage
description: "Decide whether splitting one model across two unequal accelerators on a single host (fast small-VRAM GPU + big slow unified-memory device, or a GPU over a narrow PCIe link) is worth building — with the KV/prefill arithmetic, the probes that settle it, and the doors that are already closed."
---

# Heterogeneous accelerator split — triage before building

Use when someone asks "could we split model X across device A and device B" on one host, when the devices differ in memory size/bandwidth (e.g. 24 GB GDDR6X vs an APU sharing LPDDR), or when a link is narrow (PCIe x4). Produces a go/no-go with arithmetic instead of enthusiasm.

## 1. Characterise the link (bytes vs latency)

```bash
lspci -vv -s <bdf> | grep -E "LnkCap:|LnkSta:"          # capability vs actual
nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.gen.current,pcie.link.width.current --format=csv
```

Expect **idle-downshift**: a Gen4 x4 dock can idle at Gen1 x4 (~1 GB/s) and only upshift under load. Measure while the device is busy, or you will size the design against the wrong number.

Compute the per-hop cost against the hidden state (`n_embd x 2 B` for f16): a 5120-dim model moves 10 KiB/hop = ~1.3 µs at Gen4 x4. Then:

- bytes per token = hops/token x hidden state. Typically ~1 MiB/token for 2 hops/layer x 64 layers -> ~0.3 % of a 40 ms token. **Bytes are almost never the constraint.**
- the real budget line is **hop COUNT**: every device boundary is a host-side sync. A runtime that splits per *op* can hit 100 splits/token and pin the GPU at ~5 % SM. A runtime that batches per-backend graphs is far cheaper — measure `graph splits` in the loader log, do not assume.

**Design rule: partition by where the weights live, never by where the activations flow.** Resident weights, activations crossing.

## 2. Get the workload's real bottleneck (KV is usually it)

KV per position x context = bytes read per token, because attention reads the whole cached context every token:

```
KV/token = kv_bytes_per_position * context
```

Compare that against (a) expert/weight traffic per token (read once per token per layer) and (b) total token time. At long context or on a hybrid model with few attention layers but many positions, **KV dominates**. Then:

- the highest-leverage placement is **KV cache on the fastest-memory device, together with its attention compute** — never KV alone (separating them puts `KV/token` over the link, which is fatal: GBs across a 7.88 GB/s link = hundreds of ms/token);
- check the KV + attention weights actually fit: `kv_positions_per_slot * slots * bytes_per_position + attention_trunk <= VRAM`.

## 3. What placement never buys

Prefill is compute-bound and expert weights are read **once** per batch, so moving them to faster memory buys nothing while adding a serial stage. Expect a regression, and size it from calibration: on Strix Halo + RTX 3090 (Qwen3.8-Flash-Next, ik_llama.cpp) moving 12.5 % of the expert mass to the GPU gave **decode +12 %, prefill 192 -> 32 tok/s**.

A `1/(f_big/BW_slow + f_fast/BW_fast)` harmonic calculation reproduces that decode gain to within ~2 points, so it is usable for decode projections — and it assumes the stages are sequential (true at batch 1: no pipelining).

## 4. Is the second device even a separate pool?

On a UMA APU, check before assuming additivity:

```bash
cat /sys/class/drm/card*/device/mem_info_vram_total   # BIOS carve-out (small)
cat /sys/class/drm/card*/device/mem_info_gtt_total    # aperture onto SYSTEM RAM
rocm-smi --showmeminfo all | grep -i gtt              # used == what the CPU also sees
```

A large GTT window means the "GPU memory" is the *same* LPDDR under one memory controller: adding a stage there adds a compute engine, not capacity or bandwidth. Only the discrete card's VRAM is additive. Fencing more (`amdgpu.gttsize`, BIOS UMA size) just removes bytes from CPU-visible RAM.

## 5. Speculative decoding as the "cheap" split

A drafter on the *other* device is bandwidth-free (token ids per round, tens of bytes), so it is tempting. Before proposing it:

- the drafter needs the target's **exact tokenizer** for a plain draft model (verify by comparing GGUF `tokenizer.ggml.tokens`, not by trusting the family name), or it must be a head **trained for that target** (EAGLE/DFlash/MTP are conditioned on the target's hidden states and are not portable between models);
- a plain drafter costs **K sequential forwards per round**, so it must be 10-50x faster than the target. Compute the break-even: with `r = t_draft/t_target`, `rate_ratio = A/(K*r + 1)`, `A = (1-p^(K+1))/(1-p)`. At `r ≈ 0.4-0.5` (a drafter only ~2x faster) break-even needs `p ≈ 0.7+` at K=3;
- always compare against the **incumbent**, not the no-draft baseline: a target already running its own MTP head (e.g. ~1.7 tokens committed/round) is a high bar;
- check the runtime can even accept an external drafter. Closed engines may enumerate a fixed drafter set and 400 anything else (`DRAFTER_CAP`-style capability gates keyed on the loaded checkpoint) — that is a design boundary, not a config gap. In llama.cpp the portable mechanism is `--spec-draft-device/-devd` (draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd-ctvd`.

## 6. Expressing a split (llama.cpp, the usual vehicle)

```bash
-dev CUDA0,Vulkan0 -sm layer -ts F0,F1        # contiguous layer ranges (simplest)
-ot "\.attn_.*=CUDA0" -ot "\.exps\.=Vulkan0"   # per-tensor, by ROLE (what KV co-location needs)
```

- buffer types are matched by name at load (`ggml_backend_buft_name`), so device names work as `-ot` values;
- `-cmoe` / `-ncmoe` are **CPU-only**; device-side MoE placement is `-ot`;
- several backends in one binary is fine (CUDA + Vulkan is the practical pair; HIP + CUDA do not coexist), and `GPU_BACKEND_A=OFF` in the CMakeCache of the existing build tells you what needs adding;
- contiguous ranges are the *wrong* split when KV bandwidth dominates, because they drag half the attention layers (and their KV) back to the slow pool.

## 7. Hazards checklist

- paged/sidecar tables (huge n-gram/embedding tables) must stay where they are read from; moving them is pure loss;
- keep `mmap` on for models with huge embedding tables (anonymous memory gets the process cgroup-killed);
- CUDA graph capture can abort on unusual graph shapes (hyper-connections etc.) — check before assuming optimisations apply;
- the second device's brand-new backend may not implement the arch's ops at all: make the first measurement a **baseline of the runtime on that device alone**, because a split's arithmetic is worthless if the baseline is worse than the incumbent engine;
- a split consumes the fast device entirely — check what else it was serving before trading it away;
- benchmark/experiment discipline on production hosts: stop and mask first.

## 8. Verdict shape

Rank and report, and prefer the boring answer when it wins:

1. **Two independent endpoints** (zero shared traffic) — often the real answer to "what's a useful combination", especially over a narrow link;
2. **One process, weights-resident, KV co-located with attention on the fast pool** — promising, quantifiable (+% decode at long context), needs a dual-backend build and a baseline measurement;
3. **Hot-expert cache / per-token weight streaming** — affordable only if the miss set per token is small (a few experts; check `misses x bytes x tok/s` against link bandwidth) and overlapped, and it injects PCIe latency into the critical path; no mainstream runtime implements a dynamic expert cache;
4. list explicitly what does *not* help: TP/collectives across unequal devices, KV on the far device, model loading over the link, prefill-motivated placements.