agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills

- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh
- models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes
- niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0
- memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills
- managed-skills: 40+ new fleet runbook skills
- agent/skills: import built-in skill set
This commit is contained in:
Frank committed 2026-09-17 12:27:08 -07:00
1 parent 83c9c92b3f
commit 135bb2194d
103 files changed
+5333 -170

No files matched your search

@@ -0,0 +1,62 @@
---
name: llamacpp-device-split-feasibility
description: "Decide which multi-device arrangements llama.cpp can actually express on a heterogeneous host (mixed vendors, iGPU+dGPU), and measure them without fooling yourself — covers the expressibility matrix, llama-bench's /-vs-, device/split syntax, independent placement proof, and the stop/restore path."
---
# When to use
Asked "can we split this model across GPU A and GPU B?" on a host where the devices are mixed (NVIDIA + AMD, iGPU + dGPU) or where the obvious arrangement is "weights on device X, different weights on device Y". Also use before *believing* any earlier "it cannot be done" conclusion about a multi-device split — that verdict is easy to produce by accident.
# 1. Classify the arrangement before running anything
llama.cpp expresses exactly two things across devices, and they are not interchangeable:
| arrangement | how | expressible? |
|---|---|---|
| **Layer split** — layers 0..n on dev A, the rest on dev B | `-sm layer -ts <shareA>/<shareB>` | **yes**, no collectives needed; works cross-vendor (CUDA + Vulkan measured) |
| **Per-tensor placement** — e.g. attention+KV on A, experts on B | `-ot <regex>=<dev>` | **NO.** `--override-tensor` sets `has_tensor_overrides()`, a required term of the scheduler's `pipeline_parallel` flag (the only read site is `llama-context.cpp`, in the `pipeline_parallel = ...` expression). That flag sets `n_copies = parallel ? MAX : 1` in `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a non-primary device has no backend and the graph aborts at `ggml-backend.cpp` (`pre-allocated tensor (...weight) in a buffer (...) that cannot run the operation (NONE)` — `(NONE)` is the leaf's own op name). `-ot x=CPU` works only because the CPU backend is last in the list. |
So: if the plan needs *tensors* on a specific device, stop — the answer is no, for any vendor mix, and pure-Vulkan vs CUDA+Vulkan changes nothing. If it needs *layers*, it is expressible and worth measuring.
Layer-split arithmetic to check before promising anything:
- A whole layer carries its experts, if the model is MoE. Compute GiB/layer, not GiB/param-group.
- Only some layers carry KV (e.g. every 4th for hybrid attention). A share of layers is therefore only a share of KV — a contiguous range rarely holds the layers you care about.
- Big non-repeating tensors (per-layer-token tables etc.) land on the main device and cannot be moved by `-ts`; they cap how small the primary device's footprint can get.
# 2. Get the harness syntax right, or every number is fiction
`llama-bench` writes `-dev <dev0/dev1/...>` and `-ts <ts0/ts1/..>`: **`/` separates values within one configuration, `,` separates configurations to sweep.**
- `-dev CUDA0,Vulkan1` = **two single-device runs**, never a two-device run.
- `-ts 0.85,0.15` = two sweep values, each becoming `[0.85, 0, 0, …]` → device 0 takes everything.
Write `-dev A/B -ts x/y`. `llama-cli`/`llama-server` use the classic comma form — which is why a failure seen only under llama-bench is a harness bug, not an engine limit.
Cheap pre-flight: run with the intended flags and read the tool's own reported parameters (llama-bench's table has a `dev` column). If a row says one device, your intent was not parsed.
# 3. Prove placement — never trust the label
Sample the non-primary device *while the run is loaded*:
```bash
( for i in $(seq 1 60); do nvidia-smi --query-gpu=memory.used --format=csv,noheader; sleep 10; done ) > /tmp/vram.txt &
```
Expect roughly (layers × GiB/layer + that device's KV) resident. Tens of MiB means the split did not happen. Cross-check: several "different" configurations that all produce the same tok/s to within 0.2 % are probably all the same single-device run.
# 4. Measure at the depth that matters
- `llama-bench`'s `tg` runs with `n_prompt = 0`, i.e. ~zero depth. Use `-n 32 -d 65536` (or deeper) or the KV term — the entire reason for the split — is absent.
- Compare against the **single-device baseline on the fast-memory device**, and against whatever production runtime already serves the model. A split that wins by a few percent at 64k depth may lose at 200k, because the KV term grows with context and prefill cost grows with the target depth.
# 5. Stop / restore on a host that serves
- Stop and mask the production units first (benchmarking policy). For rootless quadlets the lever is the `.container` file (move it aside + `daemon-reload`); `Restart=always` otherwise recreates the service mid-run. `systemctl mask` is refused for a unit that is a symlink into a repo checkout.
- After restoring, verify `/health`, `/v1/models`, and the boot ledger numbers — and keep a memory-headroom note: the same model can come back with less contiguous host memory than before (a reboot re-fragments), which decides whether the next co-tenant fits.
- Landmine seen on a rootless-podman host after a big GPU run: a dead rootless **pause process** takes *all* container management down (every `podman` command: `invalid internal status … could not find any running process`; container starts then fail because the rootless netns holding the bridges is gone: `failed to bind udp listener on 10.89.x.1:53`). `podman system migrate` panics (5.7.0, `teardownNetworkBackend`). Repair is a reboot. Before rebooting, check `loginctl show-user <u> -p Linger` and that units carry `[Install] WantedBy=default.target`, so everything returns by itself.
- Do not publish a cause you did not check: `journalctl -k | grep -i oom` decides whether "memory pressure killed it" is a finding or a guess. An invented narrative is worse than an open question.
# 6. Report shape that survived review
State (a) which arrangement was tested and that the harness parsed it, (b) the placement evidence, (c) the depth-matched numbers next to the single-device baseline, (d) the memory cost on the borrowing device, and (e) whether the split can reach its own premise (KV on the fast device, long context) — not just that it runs.