x4 pair round two: pure Vulkan changes nothing, and the first verdict was wrong

Asked whether Vulkan on both devices unlocks the split. It does not: the
per-tensor design aborts byte-identically, because the gate is not the driver
mix but `--override-tensor` — `has_tensor_overrides()` is a required term of
the scheduler's `pipeline_parallel` flag (llama-context.cpp:433, the only place
in the tree that reads it), and that flag sets `n_copies` in
`ggml_backend_sched_new`. With `n_copies == 1` a pre-allocated weight on a
non-primary device has no backend and the graph aborts.

Chasing it exposed a harness artifact that invalidated the earlier conclusion:
llama-bench splits a device LIST with `/` and SWEEP values with `,`, so
`-dev CUDA0,Vulkan1` was two single-device runs all along (and `-ts 0.85,0.15`
two sweep values, i.e. device 0 takes everything — which is where the bogus OOM
came from). With the correct syntax two-device splits work in both flavours:

  pure Vulkan  -dev Vulkan1/Vulkan0 -ts 0.85/0.15 -> 27.55 tok/s @64k (12.8 GiB on the 3090)
  cross-vendor -dev CUDA0/Vulkan1   -ts 0.15/0.85 -> 27.23 tok/s @64k, pp512 268.5 (15.0 GiB)
  iGPU alone                                       25.77 tok/s @64k, pp512 231.8

+7 % decode, +16 % prefill — real but not a regime change, and it costs the
3090 ~15 GiB, which this fleet spends on its only vision endpoint. The design's
premise still fails: whole layers carry their experts (1.62 GiB each), the 12
attention layers are spread every 4th, and the 32.8 GiB PLE must stay on the
main device, so KV cannot be consolidated on the 3090.

Also records the evening's incident (a dead rootless podman pause process took
all container management down; `podman system migrate` panics in 5.7.0; reboot
repaired it) and the netavark host-address lesson from the hermes 502.

AGENTS.md

@@ -379,6 +379,21 @@ linked runbooks.
   18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly
   fallback). That is what `frank-medium` runs — see
   [[frank-medium-llamacpp-dflash2]].
+- **Layer split is also the cross-vendor answer — and `-ot` is what cannot be
+  combined with it.** On Dankclaw (2026-09-16) a CUDA 3090 + Vulkan iGPU layer
+  split ran a real two-device graph: `-dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85`
+  held 14,994 MiB on the 3090 and gave 27.2 tok/s with 268.5 t/s prefill at 64k
+  depth (iGPU alone: 25.8 / 231.8). What does NOT work is per-tensor placement
+  across devices: `--override-tensor <group>=<device>` sets
+  `has_tensor_overrides()`, a REQUIRED term of the scheduler's
+  `pipeline_parallel` flag (llama-context.cpp:433 — the only place in the tree
+  that reads it), and that flag is what sets `n_copies` in
+  `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a
+  non-primary device has no backend and the graph aborts (`ggml-backend.cpp:941`,
+  reported with the leaf's own op name `(NONE)`). `=CPU` is the only override
+  target that works, because the CPU backend is last in the list. Pure Vulkan
+  changes nothing here — the gate is the override, not the driver mix. See
+  [[x4-pair-experiment]].
 
 
 **ROCm / GPU**
@@ -498,6 +513,16 @@ linked runbooks.
 - **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same
   option and the same ~12x prefill collapse described in the ROCm note above.
   f16 KV needs nothing, which is why frank-small runs f16.
+- **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both
+  `-dev` and `-ts` follow this: `-dev CUDA0,Vulkan1` is two *single-device*
+  configurations, never one two-device run, and `-ts 0.85,0.15` is two sweep
+  values each becoming `[0.85, 0, …]` / `[0.15, 0, …]`, i.e. device 0 takes
+  everything. Write `-dev A/B -ts x/y`. This artifact produced a wrong fleet
+  conclusion on 2026-09-16 — "CUDA + Vulkan cannot serve one model", with two
+  bogus OOMs that were really "all 111 GiB on the 24 GiB card" — and the bench
+  table's own `dev` column was printing the truth the whole time. Any
+  placement-dependent number needs an independent check: sample the other
+  device's memory (`nvidia-smi`) while the run is loaded.
 - **A model's own template can REJECT the harness's thinking level.** Qwen3.8's
   template accepts only `xhigh`, `medium` and `low` for `reasoning_effort` and
   raises on anything else — HTTP 500, not a clamp — so the harness's levels
@@ -531,6 +556,26 @@ linked runbooks.
   only the CURRENT instance; history across recreations lives in
   `journalctl --user -u <name>.service`. Read rootless storage WITHOUT sudo
   (per-user storage — `sudo podman logs` finds nothing).
+- **A dead rootless podman pause process breaks ALL container management on the
+  host, and `podman system migrate` panics in 5.7.0.** Symptoms: every `podman`
+  command fails with `invalid internal status … could not find any running
+  process`, every container start fails binding podman's bridge addresses
+  (`Error starting server failed to bind udp listener on 10.89.x.1:53: Cannot
+  assign requested address`), and every `Restart=always` quadlet loops. Container
+  storage is intact; what is gone is `/run/user/<uid>/libpod/tmp/alive` (a UUID
+  token, not a pid) and the rootless netns holding the bridges — deleting the
+  stale token is not enough, and `podman system migrate` segfaults in
+  `teardownNetworkBackend`. Repair: **reboot** (`/run/user` clears and all 28
+  quadlets return on their own — check `Linger=yes` and `WantedBy=default.target`
+  first). Observed trigger 2026-09-16: memory pressure from an iGPU run that
+  committed 116 of the host's 121 GiB.
+- **A container cannot reach a `Network=host` service at the host's LAN IP.**
+  netavark DNATs *published* ports only: from a bridge container
+  `192.168.0.143:<published>` answers while `192.168.0.143:<host bind>` and the
+  gateway `10.89.1.1` are both refused. Use `host.containers.internal`
+  (169.254.1.2). This is what 502'd `hermes.franksdank.cloud` until the Caddyfile
+  upstream was changed on 2026-09-16 (its vhost comment had claimed the LAN IP
+  was open from that bridge — it never was).
 
 ## Runbook Index