command-center/agent/managed-skills/llamacpp-device-split-feasibility/SKILL.md

name, description

name	description
llamacpp-device-split-feasibility	Decide which multi-device arrangements llama.cpp can actually express on a heterogeneous host (mixed vendors, iGPU+dGPU), and measure them without fooling yourself — covers the expressibility matrix, llama-bench's /-vs-, device/split syntax, independent placement proof, and the stop/restore path.

When to use

Asked "can we split this model across GPU A and GPU B?" on a host where the devices are mixed (NVIDIA + AMD, iGPU + dGPU) or where the obvious arrangement is "weights on device X, different weights on device Y". Also use before believing any earlier "it cannot be done" conclusion about a multi-device split — that verdict is easy to produce by accident.

1. Classify the arrangement before running anything

llama.cpp expresses exactly two things across devices, and they are not interchangeable:

arrangement	how	expressible?
Layer split — layers 0..n on dev A, the rest on dev B	-sm layer -ts <shareA>/<shareB>	yes, no collectives needed; works cross-vendor (CUDA + Vulkan measured)
Per-tensor placement — e.g. attention+KV on A, experts on B	-ot <regex>=<dev>	NO. --override-tensor sets has_tensor_overrides(), a required term of the scheduler's pipeline_parallel flag (the only read site is llama-context.cpp, in the pipeline_parallel = ... expression). That flag sets n_copies = parallel ? MAX : 1 in ggml_backend_sched_new; with n_copies == 1 a pre-allocated weight on a non-primary device has no backend and the graph aborts at ggml-backend.cpp (pre-allocated tensor (...weight) in a buffer (...) that cannot run the operation (NONE) — (NONE) is the leaf's own op name). -ot x=CPU works only because the CPU backend is last in the list.

So: if the plan needs tensors on a specific device, stop — the answer is no, for any vendor mix, and pure-Vulkan vs CUDA+Vulkan changes nothing. If it needs layers, it is expressible and worth measuring.

Layer-split arithmetic to check before promising anything:

• A whole layer carries its experts, if the model is MoE. Compute GiB/layer, not GiB/param-group.
• Only some layers carry KV (e.g. every 4th for hybrid attention). A share of layers is therefore only a share of KV — a contiguous range rarely holds the layers you care about.
• Big non-repeating tensors (per-layer-token tables etc.) land on the main device and cannot be moved by -ts; they cap how small the primary device's footprint can get.

2. Get the harness syntax right, or every number is fiction

llama-bench writes -dev <dev0/dev1/...> and -ts <ts0/ts1/..>: / separates values within one configuration, , separates configurations to sweep.

• -dev CUDA0,Vulkan1 = two single-device runs, never a two-device run.
• -ts 0.85,0.15 = two sweep values, each becoming [0.85, 0, 0, …] → device 0 takes everything.

Write -dev A/B -ts x/y. llama-cli/llama-server use the classic comma form — which is why a failure seen only under llama-bench is a harness bug, not an engine limit.

Cheap pre-flight: run with the intended flags and read the tool's own reported parameters (llama-bench's table has a dev column). If a row says one device, your intent was not parsed.

3. Prove placement — never trust the label

Sample the non-primary device while the run is loaded:

( for i in $(seq 1 60); do nvidia-smi --query-gpu=memory.used --format=csv,noheader; sleep 10; done ) > /tmp/vram.txt &

Expect roughly (layers × GiB/layer + that device's KV) resident. Tens of MiB means the split did not happen. Cross-check: several "different" configurations that all produce the same tok/s to within 0.2 % are probably all the same single-device run.

4. Measure at the depth that matters

• llama-bench's tg runs with n_prompt = 0, i.e. ~zero depth. Use -n 32 -d 65536 (or deeper) or the KV term — the entire reason for the split — is absent.
• Compare against the single-device baseline on the fast-memory device, and against whatever production runtime already serves the model. A split that wins by a few percent at 64k depth may lose at 200k, because the KV term grows with context and prefill cost grows with the target depth.

5. Stop / restore on a host that serves

• Stop and mask the production units first (benchmarking policy). For rootless quadlets the lever is the .container file (move it aside + daemon-reload); Restart=always otherwise recreates the service mid-run. systemctl mask is refused for a unit that is a symlink into a repo checkout.
• After restoring, verify /health, /v1/models, and the boot ledger numbers — and keep a memory-headroom note: the same model can come back with less contiguous host memory than before (a reboot re-fragments), which decides whether the next co-tenant fits.
• Landmine seen on a rootless-podman host after a big GPU run: a dead rootless pause process takes all container management down (every podman command: invalid internal status … could not find any running process; container starts then fail because the rootless netns holding the bridges is gone: failed to bind udp listener on 10.89.x.1:53). podman system migrate panics (5.7.0, teardownNetworkBackend). Repair is a reboot. Before rebooting, check loginctl show-user <u> -p Linger and that units carry [Install] WantedBy=default.target, so everything returns by itself.
• Do not publish a cause you did not check: journalctl -k | grep -i oom decides whether "memory pressure killed it" is a finding or a guess. An invented narrative is worse than an open question.

6. Report shape that survived review

State (a) which arrangement was tested and that the harness parsed it, (b) the placement evidence, (c) the depth-matched numbers next to the single-device baseline, (d) the memory cost on the borrowing device, and (e) whether the split can reach its own premise (KV on the fast device, long context) — not just that it runs.