command-center/agent/managed-skills/llamacpp-cross-backend-device-split/SKILL.md

name, description

name	description
llamacpp-cross-backend-device-split	Decide whether two unlike devices (NVIDIA CUDA + AMD Vulkan, or any two backends) can serve one llama.cpp model, and measure the attempt honestly on a serving host — the two scheduler failure modes, what is actually expressible, the dual-backend build recipe, and the measurement traps that make results meaningless.

Cross-backend device splits in llama.cpp

Use when someone proposes serving ONE model across two devices of different backend families (NVIDIA CUDA + AMD Vulkan/HIP, or a dGPU + an APU), and you need to know whether it is expressible before spending a build and a service outage on it.

Answer first: it is not expressible

A weight tensor in one device buffer cannot be consumed by an op the scheduler placed on another device. Two observed failure modes (mainline llama.cpp 37b53fd, 2026-09-16):

attempt	result
per-tensor: -ot "exps=Vulkan1" with -sm none -mg 0 -dev CUDA0,Vulkan1	ggml-backend.cpp:941: pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE) — scheduler abort
per-layer: -sm layer -ts a,b -dev CUDA0,Vulkan1, or layer-scoped -ot	model load fails, message is only failed to load model
device + CPU: -ot "exps=CPU"	works — the CPU backend is the universal fallback and host buffers can be copied

The rule: the CPU is the only legitimate partner in a heterogeneous split. Everything else must be single-vendor. On a CUDA+AMD host that means a card can be the trunk or the offload host, never the complement of an AMD-resident peer — the same wall the R9700 pair hits from the RCCL/atomics direction.

If the answer is "not expressible", stop and consider the expressible alternatives in this order: (1) two independent endpoints, one model per device; (2) single-vendor layer split (-sm layer, no collective needed); (3) device + CPU per-tensor offload.

Build the dual-backend binary

Needs both toolchains in one tree; Vulkan needs glslc + libvulkan headers (already present on this fleet's hosts).

cmake -B build-dual -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
  -DGGML_CUDA=ON -DGGML_VULKAN=ON -DCMAKE_CUDA_ARCHITECTURES=<sm> \
  -DGGML_CCACHE=ON -DLLAMA_CURL=OFF
cmake --build build-dual --target llama-bench llama-cli llama-server -j $(nproc)

Device names are not what you expect. --list-devices on an NVIDIA-plus-APU host lists the discrete card under BOTH backends (e.g. CUDA0 and Vulkan0) and the APU last (Vulkan1). The iGPU is Vulkan1, not Vulkan0. Valid -ot buffer types are whatever that list prints; a wrong name makes llama.cpp print the valid set and exit, so the error is self-diagnosing.

Measure it without fooling yourself

• tg in llama-bench runs with n_prompt = 0. A plain -n 64 row is a ~zero-depth decode: no KV traffic at all. Anything about context cost or KV placement must use -d/--n-depth, which fills the cache before timing.
• -p/-n lists build separate instances (prefill instances with n_gen = 0, generation instances with n_prompt = 0), and n_ctx = n_prompt + n_gen.
• Compare at depth-matched settings, and remember prefill into a deep cache is the slow case (e.g. 456 tok/s fresh vs 232 tok/s into 64k).
• Sample the actual engines during the run (/sys/class/drm/card*/device/gpu_busy_percent, nvidia-smi) — an idle device or a 100 % APU is the difference between a real measurement and a silent CPU fallback.
• Watch the host memory figure, not just tok/s. A Vulkan-resident model lives in device buffers (committed, not reclaimable), whereas an mmap'd CPU-placed tensor is page cache — the same model can commit 116 GiB one way and 95 GiB the other. On a host with co-tenants, that decides deployability.

Serving-host discipline

• Stop and mask every LLM service first (repo benchmarking policy). For a rootless Quadlet, move the .container out of ~/.config/containers/systemd/ and daemon-reload — the generator re-creates the unit from that file on every reload, so mask is not the lever.
• systemctl mask is refused for a relinked root unit whose /etc/systemd/system/<svc>.service is a symlink into the repo: it exits with File ... already exists and is a symlink to .... Stop it and ensure nothing restarts it, or move the symlink aside.
• Root units need sudo systemctl start; without it you get Access denied as the requested operation requires interactive authentication (polkit), which reads like a permissions bug.
• Export the pinning guard: mainline is GGML_CUDA_NO_PINNED (ik_llama.cpp was GGML_CUDA_NO_PINNED_WEIGHTS). Without it a CPU-overridden tensor can land in a pinned CUDA_Host buffer — the failure mode that page-locked 108 GiB and starved a host's co-tenants until a power cycle.
• Restore: move the quadlet back, daemon-reload, reset-failed, start, then verify /health, /v1/models and the engine's own boot ledger (e.g. halogen's kv pool: lines, which say whether the configured pool was clamped).

Calibration point

On a 121 GiB Strix Halo host with a 3090: a 177B MoE (112 GiB GGUF, 12 of 48 layers carrying KV) ran 32.6 tok/s shallow / 25.8 tok/s at 64k depth / 456 tok/s prefill with everything on the iGPU via Vulkan, and 12.1 tok/s with the same trunk but experts on the CPU. If a proposed split cannot beat the iGPU-only number at matched depth, it is not worth the failure modes above.