diff --git a/AGENTS.md b/AGENTS.md index ff07438..1afa439 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -379,6 +379,21 @@ linked runbooks. 18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly fallback). That is what `frank-medium` runs — see [[frank-medium-llamacpp-dflash2]]. +- **Layer split is also the cross-vendor answer — and `-ot` is what cannot be + combined with it.** On Dankclaw (2026-09-16) a CUDA 3090 + Vulkan iGPU layer + split ran a real two-device graph: `-dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85` + held 14,994 MiB on the 3090 and gave 27.2 tok/s with 268.5 t/s prefill at 64k + depth (iGPU alone: 25.8 / 231.8). What does NOT work is per-tensor placement + across devices: `--override-tensor =` sets + `has_tensor_overrides()`, a REQUIRED term of the scheduler's + `pipeline_parallel` flag (llama-context.cpp:433 — the only place in the tree + that reads it), and that flag is what sets `n_copies` in + `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a + non-primary device has no backend and the graph aborts (`ggml-backend.cpp:941`, + reported with the leaf's own op name `(NONE)`). `=CPU` is the only override + target that works, because the CPU backend is last in the list. Pure Vulkan + changes nothing here — the gate is the override, not the driver mix. See + [[x4-pair-experiment]]. **ROCm / GPU** @@ -498,6 +513,16 @@ linked runbooks. - **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same option and the same ~12x prefill collapse described in the ROCm note above. f16 KV needs nothing, which is why frank-small runs f16. +- **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both + `-dev` and `-ts` follow this: `-dev CUDA0,Vulkan1` is two *single-device* + configurations, never one two-device run, and `-ts 0.85,0.15` is two sweep + values each becoming `[0.85, 0, …]` / `[0.15, 0, …]`, i.e. device 0 takes + everything. Write `-dev A/B -ts x/y`. This artifact produced a wrong fleet + conclusion on 2026-09-16 — "CUDA + Vulkan cannot serve one model", with two + bogus OOMs that were really "all 111 GiB on the 24 GiB card" — and the bench + table's own `dev` column was printing the truth the whole time. Any + placement-dependent number needs an independent check: sample the other + device's memory (`nvidia-smi`) while the run is loaded. - **A model's own template can REJECT the harness's thinking level.** Qwen3.8's template accepts only `xhigh`, `medium` and `low` for `reasoning_effort` and raises on anything else — HTTP 500, not a clamp — so the harness's levels @@ -531,6 +556,26 @@ linked runbooks. only the CURRENT instance; history across recreations lives in `journalctl --user -u .service`. Read rootless storage WITHOUT sudo (per-user storage — `sudo podman logs` finds nothing). +- **A dead rootless podman pause process breaks ALL container management on the + host, and `podman system migrate` panics in 5.7.0.** Symptoms: every `podman` + command fails with `invalid internal status … could not find any running + process`, every container start fails binding podman's bridge addresses + (`Error starting server failed to bind udp listener on 10.89.x.1:53: Cannot + assign requested address`), and every `Restart=always` quadlet loops. Container + storage is intact; what is gone is `/run/user//libpod/tmp/alive` (a UUID + token, not a pid) and the rootless netns holding the bridges — deleting the + stale token is not enough, and `podman system migrate` segfaults in + `teardownNetworkBackend`. Repair: **reboot** (`/run/user` clears and all 28 + quadlets return on their own — check `Linger=yes` and `WantedBy=default.target` + first). Observed trigger 2026-09-16: memory pressure from an iGPU run that + committed 116 of the host's 121 GiB. +- **A container cannot reach a `Network=host` service at the host's LAN IP.** + netavark DNATs *published* ports only: from a bridge container + `192.168.0.143:` answers while `192.168.0.143:` and the + gateway `10.89.1.1` are both refused. Use `host.containers.internal` + (169.254.1.2). This is what 502'd `hermes.franksdank.cloud` until the Caddyfile + upstream was changed on 2026-09-16 (its vhost comment had claimed the LAN IP + was open from that bridge — it never was). ## Runbook Index diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 8eead50..b7dc24d 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -94,10 +94,11 @@ frank-small, port 7771] for the fleet's image path - [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and cannot do with the 3090 + Strix Halo pair, over their only link (the 3090's - Gen4 x4). The tensor inventory that decides any split, the two scheduler - limits that kill a cross-backend one, and the finding that the iGPU alone - matches halogen's deployed decode rate while leaving the 3090 idle — at a - memory cost the co-tenants cannot pay + Gen4 x4). The tensor inventory that decides any split, the single flag (`-ot`) + that kills per-tensor placement across devices, the two-device layer splits + that work (+7 % decode at 64k, cross-vendor included), the harness-parsing trap + that invalidated the first round, and the iGPU-only result that matches + halogen's deployed decode rate — at a memory cost the co-tenants cannot pay - [[frank-large-ik-uncensored]] — RETIRED 2026-09-16 (rollback target): the ik_llama.cpp runbook (root systemd unit, 2026-09-15 → 09-16) serving mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 diff --git a/serverAppConfigs-docs/LLM/x4-pair-experiment.md b/serverAppConfigs-docs/LLM/x4-pair-experiment.md index 7570ae1..2b40797 100644 --- a/serverAppConfigs-docs/LLM/x4-pair-experiment.md +++ b/serverAppConfigs-docs/LLM/x4-pair-experiment.md @@ -1,13 +1,14 @@ --- type: reference title: The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU -description: What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the tensor inventory that decides any split, the two scheduler limits that kill a cross-backend one, and the finding that the iGPU alone (Vulkan) matches halogen's deployed decode rate while leaving the 3090 idle. +description: What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the tensor inventory that decides any split, the single flag that kills per-tensor placement across devices (-ot), the two-device layer splits that DO work (pure-Vulkan and cross-vendor CUDA+Vulkan, +7 % decode at 64k depth), and the harness-parsing trap that made the first round of measurements meaningless. tags: - llm - experiment - dankclaw - llama.cpp - vulkan + - cuda - strix-halo - rtx3090 - flash-next @@ -41,6 +42,9 @@ From the 111 GiB GGUF (`mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, every 4th — `full_attention_interval = 4`) and 36 are GDN/linear. - Only the 12 attention layers carry KV. At the engine's measured 28 KiB per position, a 200k-token conversation reads **5.5 GiB of KV per token**. +- A layer split moves **whole layers**, so it moves ~1.62 GiB of experts with + every layer (75 GiB / 48). Attention layers are spread across the whole depth + (every 4th), so no contiguous layer range can hold them all. That is why the interesting split is *not* "put the experts somewhere else" — experts are re-read once per token but they are only ~1.7 GB of traffic — it is @@ -48,47 +52,88 @@ experts are re-read once per token but they are only ~1.7 GB of traffic — it i ## Results -`llama-bench`, same model, 2 tests per config (`-p 512,16384 -n 64`), plus a -depth-matched pair (`-n 32 -d 65536`). Note llama-bench's `tg` instances run with -`n_prompt = 0`, so a plain `tg` number is **~zero depth**: `-d` is required to -put KV in the picture. +`llama-bench`, same model. Shallow runs are `-p 512,16384 -n 64`; the +depth-matched pair is `-n 32 -d 65536`. Note llama-bench's `tg` instances run +with `n_prompt = 0`, so a plain `tg` number is **~zero depth**: `-d` is required +to put KV in the picture. -| config | placement | pp512 | pp16384 | tg (≈0 depth) | tg32 @64k depth | +| config | placement | pp512 | pp16384 | tg (≈0 depth) | tg @64k depth | |---|---|---|---|---|---| -| **A** | iGPU trunk (Vulkan1) + experts on CPU | 128.7 | 122.1 | 12.10 | — | -| **A\*** | iGPU trunk **and** experts (Vulkan1), PLE on CPU | **456.7** | **403.1** | **32.64** | **25.77** (pp512@64k = 231.8) | -| **B** | trunk + KV on CUDA0, experts on Vulkan1 | — | — | **abort** | — | -| **B2** | attention layers → CUDA0, rest → Vulkan1 (per-layer `-ot`) | — | — | load failure | — | -| **B3** | 25 % of layers → CUDA0, rest → Vulkan1 (`-sm layer -ts 0.25,0.75`) | — | — | load failure | — | +| **A** | iGPU trunk + experts on CPU | 128.7 | 122.1 | 12.10 | — | +| **A\*** | iGPU trunk **and** experts (Vulkan1), PLE on CPU | **456.7** | **403.1** | **32.64** | **25.77** (pp512@64k 231.8) | +| **G2** | **pure Vulkan** layer split, iGPU 85 % / 3090 15 % | — | — | — | **27.55** (pp512@64k 224.3) | +| **H** | **cross-vendor** layer split, CUDA0 15 % / Vulkan1 85 % | — | — | — | 27.23 (pp512@64k **268.5**) | | **B′** | trunk + KV on CUDA0, experts on CPU | 60.1 | 81.4 | 20.46 | — | +| **G1** | the design: trunk + KV on Vulkan0, experts on Vulkan1 (`-ot`) | — | — | **abort** | — | +| **B** | trunk + KV on CUDA0, experts on Vulkan1 (`-ot`) | — | — | **abort** | — | +| **B2/B3** | per-layer `-ot` / `-sm layer -ts` across CUDA+Vulkan | — | — | load failure | — | + +Real placement in the split runs, sampled from `nvidia-smi` during the run: +**12,804–12,832 MiB** resident on the 3090 in G2 and **14,994 MiB** in H — the +split is genuine, not a mislabelled single-device run (see Finding 5 for why +that check is mandatory here). Reference points on the same silicon: halogen (deployed) ≈ **25 tok/s** at real contexts; the retired ik hybrid 3.8–4.6 tok/s; mainline with the trunk on the 3090 and CPU experts 1.01 tok/s (2026-09-15); the CPU expert path's prefill ceiling 192 tok/s. -## Finding 1 — the cross-backend split is not expressible +## Finding 1 — per-tensor placement across devices is killed by one flag: `-ot` -Per-tensor placement between two **device** backends aborts in the scheduler: +The design (trunk + KV on the 3090, experts on the iGPU) needs +`--override-tensor` to place two *different* tensor groups on two devices. That +aborts in the scheduler: ``` ggml-backend.cpp:941: pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE) ``` -A weight tensor living in one device's buffer cannot be consumed by an op the -scheduler placed on another device. The CPU is the exception — `-ot exps=CPU` -works — because the CPU backend is the universal fallback and host buffers can -be copied. Per-**layer** cross-backend splits (`-sm layer -ts`, and layer-scoped -`-ot`) are no better: both failed at model load with no further diagnostic. +The mechanism, read out of the source rather than inferred: -So on this build, **CUDA + Vulkan cannot serve one model**: the 3090 can be the -trunk *or* the expert host, but not the complement of an iGPU-resident peer. -That is the same wall the R9700 pair hit from the other side (collectives), and -it is why every working multi-device llama.cpp deployment in this fleet is -either single-vendor (frank-medium: two AMD cards) or device+CPU. +```c +// llama-context.cpp:429 — the scheduler only gets multi-device copy slots if: +bool pipeline_parallel = + model.n_devices() > 1 && ... && + !model.has_tensor_overrides(); // ← line 433; a required term +// llama-context.cpp:605 — that flag is passed straight to the scheduler: +sched.reset(ggml_backend_sched_new(..., cparams.pipeline_parallel, ...)); +// ggml-backend.cpp:25 — where it decides how many copies each tensor may have: +sched->n_copies = parallel ? GGML_SCHED_MAX_COPIES : 1; +``` -## Finding 2 — the iGPU alone is competitive, and that was not expected +With `n_copies == 1` the scheduler cannot give a pre-allocated (weight) tensor a +second home, so a weight whose buffer belongs to a non-primary device has no +backend and the graph build aborts. `has_tensor_overrides` is consulted in +**exactly one place in the whole tree** (llama-context.cpp:433), so no other +flag can rescue it. + +- **The CPU is the only working target.** `-ot exps=CPU` runs because the CPU + backend is last in the scheduler's list and host buffers are always readable + (configs A and B′). A *device* target is what aborts. +- **Pure Vulkan does not change this.** Config G1 put both Vulkan devices + genuinely in the device list (verified by the correct `-dev` syntax, see + Finding 5) and aborted with the byte-identical message. The blocker is the + override/scheduler interaction, **not** a CUDA-vs-Vulkan driver mismatch. + +## Finding 2 — two-device layer splits DO work, in both flavours + +With no tensor overrides, `pipeline_parallel` stays on and the scheduler happily +runs a graph across two device backends. Two arrangements were measured at 64k +depth: + +- **G2 — pure Vulkan** (`-dev Vulkan1/Vulkan0 -sm layer -ts 0.85/0.15`): iGPU + takes 85 % of layers plus the 32.8 GiB PLE table, the 3090 takes the last + 7 layers (which include attention layers 43 and 47, the only KV-bearing layers + in that range). **27.55 tok/s** vs 25.77 for the iGPU alone: **+6.9 %**. +- **H — cross-vendor** (`-dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85`): the 3090 + runs its layers on its **CUDA** backend, the iGPU on Vulkan. **27.23 tok/s** + and **pp512@64k 268.5** against 231.8 for the iGPU alone: same decode, **+16 % + prefill**. + +So the pair *is* expressible — just not in the shape the design wanted. + +## Finding 3 — the iGPU alone is competitive, and that was not expected Config A\* — mainline llama.cpp, Vulkan/RADV, **everything on the iGPU**, PLE in mmap page cache — delivers 32.6 tok/s shallow and **25.8 tok/s at a 64k-deep @@ -106,32 +151,111 @@ LPDDR's shared 256 GB/s. 5 GiB. Halogen serves the same model holding 95.4 GiB with 14.7 GiB to spare, because it pages the n-gram table and pins only 68 GiB. Throughput parity, memory bankruptcy: the fleet's ~28 co-tenant containers are the reason halogen's shape -exists. +exists. The layer split does relieve some of that pressure (the 3090 absorbs +~12.8–15 GiB of layer+KV) but it does not change the class of the result. -## Finding 3 — the fast pool is too small to help +## Finding 4 — the split cannot reach the design's premise -B′ (trunk + KV on the 3090, experts on CPU) manages only 20.5 tok/s — *worse* -than A\*, which does not use the 3090 at all. And the expressible version of the -KV-on-VRAM design cannot reach its own premise: moving an attention layer to the -3090 costs **1.62 GiB** (its MoE experts travel with it, since only whole layers -move), so 12 attention layers = 19.4 GiB, and adding 200k of KV (9.2 GiB at f16, -4.6 at q8_0) exceeds the card's 24.1 GiB usable. Feasible only to ~64k context — -precisely where the KV term is small enough not to matter. +The premise was "KV on the 936 GB/s pool". A layer split cannot deliver it: + +- Moving an attention layer to the 3090 costs **1.62 GiB** (its MoE experts + travel with it — only whole layers move), so all 12 attention layers = 19.4 GiB. +- Add 200k of KV (9.2 GiB at f16, 4.6 at q8_0) and the card's ~24.1 GiB usable + is exceeded. Feasible only to ~64k context — precisely where the KV term is + small enough not to matter (the +7 % measured above is that residual). +- The 32.8 GiB PLE table must live on the *main* device (the first in the + `-dev` list, or whichever `-ts` share owns the output layer), so the iGPU keeps + a fixed 32.8 GiB no matter how the layers are split. + +`-ot per_layer_token_embd=CPU` would move the PLE — but it is an override, and +Finding 1 applies: it turns multi-device scheduling off. + +## Finding 5 — the harness-parsing trap that invalidated the first round + +The first pass concluded "CUDA + Vulkan cannot serve one model". That conclusion +was an artifact. `llama-bench --help` reads: + +``` +-dev, --device multi values separated by ',' +-ts, --tensor-split multi values separated by ',' +``` + +`parse_devices_arg` splits each `-dev` value on `/` into a device *list*, and the +outer loop splits the argument on `,` into separate *sweep values*. So +`-dev CUDA0,Vulkan1` means **two single-device configurations** (CUDA0 alone, +then Vulkan1 alone) — never a two-device run. Consequences that looked like +findings: + +- "the abort" — a single-device run with `-ot exps=Vulkan1` placing a weight on a + device that is not in the list at all (Finding 1's abort, for the trivial + reason). +- "load failure" for `-sm layer -ts 0.25,0.75` — `-ts 0.25,0.75` is *two sweep + values*, each becoming `[0.25, 0, 0, …]` / `[0.75, 0, 0, …]`, i.e. device 0 + takes everything; the second config put 111 GiB on a 24 GiB card and OOM'd. + +The tell that the runs were single-device: the bench table's own `dev` column +printed `Vulkan1` for every row. Four separate "iGPU-only" runs agree to within +0.2 % (25.71, 25.72, 25.77), which is exactly what an accidentally-single-device +sweep looks like. + +**Rules that follow.** In llama-bench, `/` separates devices *within one +configuration* and `,` separates configurations to sweep; `-dev` and `-ts` both +follow this. And never trust a placement-dependent number without an independent +check of where the memory went — `nvidia-smi` sampled during the run is what +distinguishes a real split from a mislabelled sweep. + +## Other landmines walked into (worth knowing before the next run) + +- **`llama-cli` in conversation mode with EOF on stdin spams `> ` forever** — + 3.2 GB in a few minutes, no error, no exit. Use `llama-server` for scripted + runs, or `-st/--single-turn` with llama-cli. (`-no-cnv` is not a flag in this + build.) +- **A killed rootless podman pause process takes the whole host's container + management down.** After the experiment the pause process was gone (plausible + cause: memory pressure from the 116 GiB iGPU runs), and every `podman` command + for the user failed with `invalid internal status … could not find any running + process`. `podman system migrate` — the remedy podman itself names — **panics** + in 5.7.0 (`teardownNetworkBackend` nil deref). The rootless netns that holds + podman's bridge addresses is gone with it, so `aardvark-dns` cannot bind + `10.89.x.1:53` and *every* container start fails. Repair: reboot (it clears + `/run/user/` and the whole host comes back — 28 quadlets, `WantedBy` + present, linger on). +- **A `Network=host` service is not reachable from a bridge container at the + host's LAN IP.** netavark DNATs *published* ports only; a bare host-netns bind + is refused. Use `host.containers.internal` (169.254.1.2). This was found the + same evening when `hermes.franksdank.cloud` 502'd — fixed in the Caddyfile and + guarded by a strengthened seam (`tests/hermes_webui.bats`). +- **The memory budget includes the co-tenants' patience.** The iGPU runs held + 116 of 121 GiB; the pause-process death above is the visible cost of running a + model that big on a host that also serves. ## Conclusion -On this host the pair is not additive, and the measured best arrangement remains -**two independent endpoints** ([[halogen-flash-server]] on the iGPU via halogen, -[[frank-small-qwen38-gsq-rco]] holding a whole 27B in the 3090's 24 GB). A -single process spanning both is blocked by the scheduler, and every CPU-side -fallback is slower than what the iGPU does alone. +The fleet's answer is unchanged — **two independent endpoints** +([[halogen-flash-server]] on the iGPU via halogen, +[[frank-small-qwen38-gsq-rco]] holding a whole 27B in the 3090's 24 GB) — but two +claims from the first round of this experiment are now corrected: -The one durable surprise worth keeping: **mainline llama.cpp + Vulkan on gfx1151 -is a viable engine for this class of model** (25.8 tok/s at 64k depth), which the -2026-09-15 mainline measurement of 1.01 tok/s did *not* show — that number was -the CPU+3090 hybrid, not the iGPU. Anyone re-opening the "is there a better -engine than halogen?" question should start here, with the memory budget as the -first constraint rather than the last. +1. **Cross-vendor layer split is possible** (measured: CUDA0 + Vulkan1, 27.23 + tok/s and 268.5 t/s prefill at 64k). It is not a regime change (+7 % decode), + and it costs the 3090 ~15 GiB — which this fleet spends on the vision endpoint + (`frank-small` holds 22.4 of the card's 24.6 GiB). An operator who wants the + 7 % must give up the fleet's only image-capable endpoint. +2. **Per-tensor placement across two devices is impossible while `-ot` is used**, + for a scheduler-mechanical reason, regardless of driver or vendor mix. + +The durable surprise worth keeping: **mainline llama.cpp + Vulkan on gfx1151 is a +viable engine for this class of model** (25.8 tok/s at 64k depth, and 27.6 with a +two-device split), which the 2026-09-15 mainline measurement of 1.01 tok/s did +*not* show — that number was the CPU+3090 hybrid, not the iGPU. Anyone re-opening +the "is there a better engine than halogen?" question should start here, with the +memory budget as the first constraint rather than the last. + +[INFERENCE] Extrapolating A\*'s measured KV slope (32.64 → 25.77 from 0 to 64k +depth over 12 attention layers) to 200k gives roughly 20 tok/s, which is where +halogen's ~25 tok/s at real contexts wins outright — the deeper the context, the +worse mainline's position, and depth is filled by prefill that mainline would +have to spend ~200k/232 ≈ 14 minutes on. ## Reproducing @@ -145,9 +269,22 @@ cmake --build build-dual --target llama-bench llama-cli llama-server -j 16 CUDA0 = RTX 3090 Vulkan0 = RTX 3090 (same card) Vulkan1 = Radeon Graphics (RADV STRIX_HALO) <- the iGPU, uma=1, 114 GiB GTT -# config A* (the winner), depth-matched -GGML_CUDA_NO_PINNED=1 llama-bench -m -dev Vulkan1 -ngl 999 \ +export GGML_CUDA_NO_PINNED=1 + +# A* — iGPU only (the single-device baseline), depth-matched +llama-bench -m -dev Vulkan1 -ngl 999 \ -ot per_layer_token_embd=CPU -n 32 -d 65536 -r 1 + +# G2 — pure-Vulkan two-device layer split (SLASH = within one configuration) +llama-bench -m -dev Vulkan1/Vulkan0 -sm layer -ts 0.85/0.15 \ + -ngl 999 -n 32 -d 65536 -r 1 + +# H — cross-vendor split: 3090 on CUDA, iGPU on Vulkan +llama-bench -m -dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85 \ + -ngl 999 -n 32 -d 65536 -r 1 + +# proof of placement — sample the 3090 while the run is loaded +nvidia-smi --query-gpu=memory.used --format=csv,noheader # expect 12–15 GiB ``` Operational notes for the next run: @@ -162,5 +299,6 @@ Operational notes for the next run: of a restart trigger, or move the symlink aside. - Restarting frank-large needs the quadlet source back in `~/.config/containers/systemd/` plus `daemon-reload`; the generator re-creates - the unit from that file only. After the restore the ledger read `400384 - positions, ~27.2 GiB` against a 33.5 GiB budget — no clamp — with 15.2 GiB left. + the unit from that file only. After the restore (and a host reboot, see the + landmines) the ledger read `400384 positions, ~27.2 GiB` against a 33.5 GiB + budget — no clamp — with 15.2 GiB left, and both endpoints answered `/health`. diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 23befa3..1755490 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -196,9 +196,10 @@ The LLM inference fleet on the two hosts. Full index: - [[qwen3.8-strix-optimizations]] — Strix Halo tuning notes - [[x4-pair-experiment]] — what a 3090 + Strix Halo combination can and cannot do in mainline llama.cpp (measured 2026-09-16): the tensor inventory that - decides a split, the scheduler limits that kill a cross-backend one, and the - iGPU-only result that matches halogen's decode rate while leaving the 3090 - idle — for 116 of the host's 121 GiB + decides a split, the one flag (`-ot`) that kills per-tensor placement across + devices, the two-device layer splits that DO work (pure-Vulkan and cross-vendor + CUDA+Vulkan, +7 % decode at 64k), and the harness-parsing trap that made the + first round of measurements say otherwise ### Memory and Hardware diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 5f92580..86a2449 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -659,3 +659,50 @@ timestamp: 2026-09-14 halogen's ledger read `400384 positions, ~27.2 GiB` against a 33.5 GiB budget with 15.2 GiB left). - New page: [[x4-pair-experiment]]. + +## 2026-09-16 (later) — the x4 pair, round two: the first conclusion was a parsing artifact +- Asked whether **pure Vulkan** changes the answer (both devices can run Vulkan — + `Vulkan0 = RTX 3090 (NV_coopmat2)`, `Vulkan1 = RADV STRIX_HALO (KHR_coopmat)`). + It does not: the per-tensor design still aborts, byte-identically. +- **The real gate is `-ot`, not the driver mix.** `has_tensor_overrides()` is a + required term of `pipeline_parallel` in `llama-context.cpp:433` — the ONLY place + in the tree that reads it — and that flag is what sets `n_copies` in + `ggml_backend_sched_new` (`ggml-backend.cpp:25`). With `n_copies == 1` a + pre-allocated weight on a non-primary device has no backend, so the graph + aborts at `ggml-backend.cpp:941` with the leaf's own op name (`(NONE)`). + `-ot exps=CPU` works only because the CPU backend is last in the list. +- **Then the correction.** `llama-bench --help` says `-dev `: `/` + splits a device LIST, `,` splits SWEEP values. So `-dev CUDA0,Vulkan1` was two + *single-device* runs — the earlier "CUDA + Vulkan cannot serve one model", the + "load failures", and the OOMs were artifacts (the OOM was 111 GiB on the 24 GiB + card). The bench table's `dev` column had been printing the truth all along. +- **With the correct syntax two-device splits work, in both flavours** (64k depth, + `-d 65536`; placement proven by `nvidia-smi` sampled during the run): + - pure Vulkan `-dev Vulkan1/Vulkan0 -ts 0.85/0.15` — 12.8 GiB on the 3090, + **27.55 tok/s** (pp512@64k 224.3) + - cross-vendor `-dev CUDA0/Vulkan1 -ts 0.15/0.85` — 15.0 GiB on the 3090, + 27.23 tok/s with **pp512@64k 268.5** + - against the iGPU alone: 25.77 tok/s / 231.8 — so **+7 % decode, +16 % prefill** +- **And the split still cannot reach the design's premise**: whole layers move + their experts with them (1.62 GiB per layer), the 12 attention layers are spread + every 4th across the depth, and the 32.8 GiB PLE must stay on the main device — + so KV cannot be consolidated on the 3090. +- **Incident, same evening:** the rootless podman pause process died (memory + pressure from the 116 GiB iGPU runs is the plausible cause). Every `podman` + command then failed (`invalid internal status … could not find any running + process`), every container start failed on aardvark-dns binding `10.89.x.1:53` + (the rootless netns holding the bridges was gone), and all 28 quadlets looped. + `podman system migrate` — podman's own named remedy — **panics** in 5.7.0 + (`teardownNetworkBackend`). Repair was a **reboot**: all 28 containers returned + on their own (linger + `WantedBy=default.target` present), e2e gate green after. +- **A second, pre-existing defect surfaced by that reboot:** `hermes.franksdank.cloud` + returned 502. hermes-webui is `Network=host`, so 8787 is a bare host-netns bind + and netavark DNATs only *published* ports — from inside the Caddy container + `192.168.0.143:8787` and `10.89.1.1:8787` are refused while + `host.containers.internal:8787` (169.254.1.2) answers 200. Fixed in the + Caddyfile, the stale claim in its comment replaced with the measurement, and the + seam that guarded it **strengthened** (it accepted any numeric upstream, so the + LAN IP had been passing it; reverting now fails it — teeth verified). +- Both endpoints restored and healthy: frank-large `400384 positions, ~27.2 GiB` + against a 33.5 GiB budget (no clamp), frank-small 200 on 7771. Page: + [[x4-pair-experiment]] (rewritten).