---
name: llamacpp-cross-backend-device-split
description: "Decide whether two unlike devices (NVIDIA CUDA + AMD Vulkan, or any two backends) can serve one llama.cpp model, and measure the attempt honestly on a serving host — the two scheduler failure modes, what is actually expressible, the dual-backend build recipe, and the measurement traps that make results meaningless."
---

# Cross-backend device splits in llama.cpp

Use when someone proposes serving ONE model across two devices of different
backend families (NVIDIA CUDA + AMD Vulkan/HIP, or a dGPU + an APU), and you
need to know whether it is expressible before spending a build and a service
outage on it.

## Answer first: it is not expressible

A weight tensor in one *device* buffer cannot be consumed by an op the scheduler
placed on another device. Two observed failure modes (mainline llama.cpp
`37b53fd`, 2026-09-16):

| attempt | result |
|---|---|
| per-tensor: `-ot "exps=Vulkan1"` with `-sm none -mg 0 -dev CUDA0,Vulkan1` | `ggml-backend.cpp:941: pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE)` — scheduler abort |
| per-layer: `-sm layer -ts a,b -dev CUDA0,Vulkan1`, or layer-scoped `-ot` | model load fails, message is only `failed to load model` |
| device + **CPU**: `-ot "exps=CPU"` | **works** — the CPU backend is the universal fallback and host buffers can be copied |

The rule: **the CPU is the only legitimate partner in a heterogeneous split.**
Everything else must be single-vendor. On a CUDA+AMD host that means a card can
be the trunk *or* the offload host, never the complement of an AMD-resident
peer — the same wall the R9700 pair hits from the RCCL/atomics direction.

If the answer is "not expressible", stop and consider the expressible
alternatives in this order: (1) two independent endpoints, one model per device;
(2) single-vendor layer split (`-sm layer`, no collective needed); (3)
device + CPU per-tensor offload.

## Build the dual-backend binary

Needs both toolchains in one tree; Vulkan needs `glslc` + `libvulkan` headers
(already present on this fleet's hosts).

```bash
cmake -B build-dual -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
  -DGGML_CUDA=ON -DGGML_VULKAN=ON -DCMAKE_CUDA_ARCHITECTURES=<sm> \
  -DGGML_CCACHE=ON -DLLAMA_CURL=OFF
cmake --build build-dual --target llama-bench llama-cli llama-server -j $(nproc)
```

**Device names are not what you expect.** `--list-devices` on an
NVIDIA-plus-APU host lists the discrete card under BOTH backends (e.g. `CUDA0`
and `Vulkan0`) and the APU last (`Vulkan1`). The iGPU is `Vulkan1`, not
`Vulkan0`. Valid `-ot` buffer types are whatever that list prints; a wrong name
makes llama.cpp print the valid set and exit, so the error is self-diagnosing.

## Measure it without fooling yourself

- **`tg` in llama-bench runs with `n_prompt = 0`.** A plain `-n 64` row is a
  ~zero-depth decode: no KV traffic at all. Anything about context cost or KV
  placement must use `-d/--n-depth`, which fills the cache before timing.
- `-p`/`-n` lists build separate instances (prefill instances with `n_gen = 0`,
  generation instances with `n_prompt = 0`), and `n_ctx = n_prompt + n_gen`.
- Compare at **depth-matched** settings, and remember prefill into a deep cache
  is the slow case (e.g. 456 tok/s fresh vs 232 tok/s into 64k).
- Sample the actual engines during the run (`/sys/class/drm/card*/device/gpu_busy_percent`,
  `nvidia-smi`) — an idle device or a 100 % APU is the difference between a real
  measurement and a silent CPU fallback.
- Watch the **host memory** figure, not just tok/s. A Vulkan-resident model lives
  in device buffers (committed, not reclaimable), whereas an mmap'd CPU-placed
  tensor is page cache — the same model can commit 116 GiB one way and 95 GiB
  the other. On a host with co-tenants, that decides deployability.

## Serving-host discipline

- **Stop and mask every LLM service first** (repo benchmarking policy). For a
  rootless Quadlet, move the `.container` out of
  `~/.config/containers/systemd/` and `daemon-reload` — the generator re-creates
  the unit from that file on every reload, so `mask` is not the lever.
- **`systemctl mask` is refused for a relinked root unit** whose
  `/etc/systemd/system/<svc>.service` is a symlink into the repo: it exits with
  `File ... already exists and is a symlink to ...`. Stop it and ensure nothing
  restarts it, or move the symlink aside.
- Root units need `sudo systemctl start`; without it you get
  `Access denied as the requested operation requires interactive
  authentication` (polkit), which reads like a permissions bug.
- Export the pinning guard: **mainline is `GGML_CUDA_NO_PINNED`** (ik_llama.cpp
  was `GGML_CUDA_NO_PINNED_WEIGHTS`). Without it a CPU-overridden tensor can land
  in a pinned `CUDA_Host` buffer — the failure mode that page-locked 108 GiB and
  starved a host's co-tenants until a power cycle.
- Restore: move the quadlet back, `daemon-reload`, `reset-failed`, start, then
  verify `/health`, `/v1/models` **and the engine's own boot ledger** (e.g.
  halogen's `kv pool:` lines, which say whether the configured pool was clamped).

## Calibration point

On a 121 GiB Strix Halo host with a 3090: a 177B MoE (112 GiB GGUF, 12 of 48
layers carrying KV) ran **32.6 tok/s shallow / 25.8 tok/s at 64k depth / 456 tok/s
prefill** with everything on the iGPU via Vulkan, and **12.1 tok/s** with the
same trunk but experts on the CPU. If a proposed split cannot beat the
iGPU-only number at matched depth, it is not worth the failure modes above.