--- name: llamacpp-device-split-feasibility description: "Decide which multi-device arrangements llama.cpp can actually express on a heterogeneous host (mixed vendors, iGPU+dGPU), and measure them without fooling yourself — covers the expressibility matrix, llama-bench's /-vs-, device/split syntax, independent placement proof, and the stop/restore path." --- # When to use Asked "can we split this model across GPU A and GPU B?" on a host where the devices are mixed (NVIDIA + AMD, iGPU + dGPU) or where the obvious arrangement is "weights on device X, different weights on device Y". Also use before *believing* any earlier "it cannot be done" conclusion about a multi-device split — that verdict is easy to produce by accident. # 1. Classify the arrangement before running anything llama.cpp expresses exactly two things across devices, and they are not interchangeable: | arrangement | how | expressible? | |---|---|---| | **Layer split** — layers 0..n on dev A, the rest on dev B | `-sm layer -ts /` | **yes**, no collectives needed; works cross-vendor (CUDA + Vulkan measured) | | **Per-tensor placement** — e.g. attention+KV on A, experts on B | `-ot =` | **NO.** `--override-tensor` sets `has_tensor_overrides()`, a required term of the scheduler's `pipeline_parallel` flag (the only read site is `llama-context.cpp`, in the `pipeline_parallel = ...` expression). That flag sets `n_copies = parallel ? MAX : 1` in `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a non-primary device has no backend and the graph aborts at `ggml-backend.cpp` (`pre-allocated tensor (...weight) in a buffer (...) that cannot run the operation (NONE)` — `(NONE)` is the leaf's own op name). `-ot x=CPU` works only because the CPU backend is last in the list. | So: if the plan needs *tensors* on a specific device, stop — the answer is no, for any vendor mix, and pure-Vulkan vs CUDA+Vulkan changes nothing. If it needs *layers*, it is expressible and worth measuring. Layer-split arithmetic to check before promising anything: - A whole layer carries its experts, if the model is MoE. Compute GiB/layer, not GiB/param-group. - Only some layers carry KV (e.g. every 4th for hybrid attention). A share of layers is therefore only a share of KV — a contiguous range rarely holds the layers you care about. - Big non-repeating tensors (per-layer-token tables etc.) land on the main device and cannot be moved by `-ts`; they cap how small the primary device's footprint can get. # 2. Get the harness syntax right, or every number is fiction `llama-bench` writes `-dev ` and `-ts `: **`/` separates values within one configuration, `,` separates configurations to sweep.** - `-dev CUDA0,Vulkan1` = **two single-device runs**, never a two-device run. - `-ts 0.85,0.15` = two sweep values, each becoming `[0.85, 0, 0, …]` → device 0 takes everything. Write `-dev A/B -ts x/y`. `llama-cli`/`llama-server` use the classic comma form — which is why a failure seen only under llama-bench is a harness bug, not an engine limit. Cheap pre-flight: run with the intended flags and read the tool's own reported parameters (llama-bench's table has a `dev` column). If a row says one device, your intent was not parsed. # 3. Prove placement — never trust the label Sample the non-primary device *while the run is loaded*: ```bash ( for i in $(seq 1 60); do nvidia-smi --query-gpu=memory.used --format=csv,noheader; sleep 10; done ) > /tmp/vram.txt & ``` Expect roughly (layers × GiB/layer + that device's KV) resident. Tens of MiB means the split did not happen. Cross-check: several "different" configurations that all produce the same tok/s to within 0.2 % are probably all the same single-device run. # 4. Measure at the depth that matters - `llama-bench`'s `tg` runs with `n_prompt = 0`, i.e. ~zero depth. Use `-n 32 -d 65536` (or deeper) or the KV term — the entire reason for the split — is absent. - Compare against the **single-device baseline on the fast-memory device**, and against whatever production runtime already serves the model. A split that wins by a few percent at 64k depth may lose at 200k, because the KV term grows with context and prefill cost grows with the target depth. # 5. Stop / restore on a host that serves - Stop and mask the production units first (benchmarking policy). For rootless quadlets the lever is the `.container` file (move it aside + `daemon-reload`); `Restart=always` otherwise recreates the service mid-run. `systemctl mask` is refused for a unit that is a symlink into a repo checkout. - After restoring, verify `/health`, `/v1/models`, and the boot ledger numbers — and keep a memory-headroom note: the same model can come back with less contiguous host memory than before (a reboot re-fragments), which decides whether the next co-tenant fits. - Landmine seen on a rootless-podman host after a big GPU run: a dead rootless **pause process** takes *all* container management down (every `podman` command: `invalid internal status … could not find any running process`; container starts then fail because the rootless netns holding the bridges is gone: `failed to bind udp listener on 10.89.x.1:53`). `podman system migrate` panics (5.7.0, `teardownNetworkBackend`). Repair is a reboot. Before rebooting, check `loginctl show-user -p Linger` and that units carry `[Install] WantedBy=default.target`, so everything returns by itself. - Do not publish a cause you did not check: `journalctl -k | grep -i oom` decides whether "memory pressure killed it" is a finding or a guess. An invented narrative is worse than an open question. # 6. Report shape that survived review State (a) which arrangement was tested and that the harness parsed it, (b) the placement evidence, (c) the depth-matched numbers next to the single-device baseline, (d) the memory cost on the borrowing device, and (e) whether the split can reach its own premise (KV on the fast device, long context) — not just that it runs.