serverAppConfigs/serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md

---
type: reference
title: Heterogeneous Split Triage
description: Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X + LPDDR APU, PCIe x4 link) — link characterisation, the real bottleneck, what placement never buys, UMA pools, drafter splits, and the verdict ranking.
tags: [heterogeneous, multi-device, llama-cpp, triage, pcie]
timestamp: 2026-10-05T00:00:00-07:00
---

# Heterogeneous Split Triage

Produced the 2026-09-16 no-go verdict on splitting Qwen3.8-Flash-Next across
the RTX 3090 + Strix Halo pair. Use it to answer "could we split model X
across device A and device B" with arithmetic instead of enthusiasm.

## 1. Characterise the link (bytes vs latency)

```bash
lspci -vv -s <bdf> | grep -E "LnkCap:|LnkSta:"          # capability vs actual
nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.gen.current,pcie.link.width.current --format=csv
```

Expect **idle-downshift**: a Gen4 x4 dock idles at Gen1 x4 (~1 GB/s) and
upshifts under load. Measure while the device is busy, or you size the design
against the wrong number. Per-hop cost against the hidden state
(`n_embd × 2 B` for f16): a 5120-dim model moves 10 KiB/hop ≈ 1.3 µs at
Gen4 x4.

- **Bytes per token ≈ hops/token × hidden state** — typically ~1 MiB/token
  for 2 hops/layer × 64 layers ≈ 0.3 % of a 40 ms token. **Bytes are almost
  never the constraint.**
- The real budget line is **hop COUNT**: every device boundary is a
  host-side sync. A runtime that splits per *op* hits ~100 splits/token and
  pins the GPU at ~5 % SM. Measure `graph splits` in the loader log; never
  assume.
- **Design rule: partition by where the weights live, never by where the
  activations flow.** Resident weights, activations crossing.

## 2. Get the workload's real bottleneck (KV is usually it)

```
KV/token = kv_bytes_per_position × context
```

Attention reads the whole cached context every token, so KV/token scales
with context — compare it against expert/weight traffic per token and total
token time. At long context on a hybrid arch (few attention layers, many
positions), **KV dominates**. Consequences:

- The highest-leverage placement is **KV cache on the fastest-memory device,
  together with its attention compute** — never KV alone: separating them
  puts KV/token over the link, which is fatal (GBs across 7.88 GB/s =
  hundreds of ms/token).
- Check the fit: `kv_positions_per_slot × slots × bytes_per_position +
  attention_trunk ≤ VRAM`.

## 3. What placement never buys

Prefill is compute-bound and expert weights are read once per batch, so
moving them to faster memory buys nothing while adding a serial stage.
Expect a regression, and size it from calibration: on Strix Halo + RTX 3090
(Qwen3.8-Flash-Next, ik_llama.cpp) moving 12.5 % of the expert mass to the
GPU gave **decode +12 %, prefill 192 → 32 tok/s**. A
`1/(f_big/BW_slow + f_fast/BW_fast)` harmonic calculation reproduces that
decode gain to within ~2 points — usable for decode projections, and it
assumes sequential stages (true at batch 1; no pipelining).

## 4. Is the second device even a separate pool?

On a UMA APU, check before assuming additivity:

```bash
cat /sys/class/drm/card*/device/mem_info_vram_total   # BIOS carve-out (small)
cat /sys/class/drm/card*/device/mem_info_gtt_total    # aperture onto SYSTEM RAM
rocm-smi --showmeminfo all | grep -i gtt              # used == what the CPU also sees
```

A large GTT window means the "GPU memory" is the *same* LPDDR under one
memory controller: adding a stage there adds a compute engine, not capacity
or bandwidth. Only the discrete card's VRAM is additive. Fencing more
(`amdgpu.gttsize`, BIOS UMA size) just removes bytes from CPU-visible RAM.

## 5. Speculative decoding as the "cheap" split

A drafter on the *other* device is bandwidth-free (token ids per round, tens
of bytes), so it is tempting. Before proposing it:

- the drafter needs the target's **exact tokenizer** for a plain draft model
  (compare GGUF `tokenizer.ggml.tokens`, never trust the family name), or it
  must be a head **trained for that target** (EAGLE/DFlash/MTP are
  conditioned on the target's hidden states and are not portable between
  models);
- a plain drafter costs **K sequential forwards per round**, so it must be
  10–50× faster than the target. Break-even with `r = t_draft/t_target`:
  `rate_ratio = A/(K·r + 1)`, `A = (1−p^(K+1))/(1−p)`; at `r ≈ 0.4–0.5`
  (a drafter only ~2× faster) break-even needs `p ≈ 0.7+` at K=3;
- compare against the **incumbent**, not the no-draft baseline: a target
  already running its own MTP head is a high bar;
- check the runtime accepts an external drafter at all — closed engines may
  enumerate a fixed drafter set and 400 anything else (capability gates
  keyed on the loaded checkpoint). That is a design boundary, not a config
  gap. In llama.cpp the portable mechanism is `--spec-draft-device/-devd`
  (draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd -ctvd`.

## 6. Expressing a split (llama.cpp, the usual vehicle)

```bash
-dev CUDA0,Vulkan0 -sm layer -ts F0,F1        # contiguous layer ranges (simplest)
-ot "\.attn_.*=CUDA0" -ot "\.exps\.=Vulkan0"   # per-tensor, by ROLE (KV co-location)
```

- buffer types match by name at load (`ggml_backend_buft_name`), so device
  names work as `-ot` values;
- `-cmoe`/`-ncmoe` are **CPU-only**; device-side MoE placement is `-ot`;
- several backends in one binary is fine (CUDA + Vulkan is the practical
  pair; HIP + CUDA do not coexist) — `GPU_BACKEND_A=OFF` in the existing
  build's CMakeCache tells you what needs adding;
- contiguous ranges are the WRONG split when KV bandwidth dominates: they
  drag half the attention layers (and their KV) back to the slow pool.

## 7. Hazards checklist

- Paged/sidecar tables (huge n-gram/embedding tables) must stay where they
  are read from; moving them is pure loss.
- Keep `mmap` on for models with huge embedding tables (anonymous memory
  gets the process cgroup-killed).
- CUDA graph capture can abort on unusual graph shapes (hyper-connections)
  — check before assuming optimisations apply.
- A brand-new backend on the second device may not implement the arch's ops
  at all: make the first measurement a **baseline of the runtime on that
  device alone** — a split's arithmetic is worthless if the baseline is
  worse than the incumbent engine.
- A split consumes the fast device entirely — check what else it was
  serving before trading it away.
- Benchmark discipline on production hosts: stop and mask first
  ([[Fleet Ops Playbook]]).

## 8. Verdict shape

Rank and report; prefer the boring answer when it wins:

1. **Two independent endpoints** (zero shared traffic) — often the real
   answer to "what's a useful combination", especially over a narrow link.
2. **One process, weights-resident, KV co-located with attention on the
   fast pool** — promising and quantifiable (+% decode at long context);
   needs a dual-backend build and a baseline measurement.
3. **Hot-expert cache / per-token weight streaming** — affordable only if
   the miss set per token is small (a few experts; check
   `misses × bytes × tok/s` against link bandwidth) and overlapped; injects
   PCIe latency into the critical path; no mainstream runtime implements a
   dynamic expert cache.
4. List explicitly what does NOT help: TP/collectives across unequal
   devices, KV on the far device, model loading over the link,
   prefill-motivated placements.

## Related

- [[PCIe Link and Atomics]] — the link and collective walls underneath
- [[Llama Cpp Multi-Device Splits]] — the deployed split shapes
- [[Llama Cpp Serving Sizing]] — KV and pool arithmetic