live: heartbeat sync 2026-10-19T14:21:41-07:00 (Refs #11)

agent/managed-skills/amd-apu-vulkan-tenant-onboarding/SKILL.md

@@ -0,0 +1,89 @@
+---
+name: amd-apu-vulkan-tenant-onboarding
+description: "Add a ggml/Vulkan inference tenant to an AMD APU (gfx1151 / Strix Halo iGPU) that already serves another engine: APU memory census, the measured RADV heap gate, render-node resolution on hybrid hosts, build/runtime dependency surface, co-tenancy measurement."
+---
+
+Add a ggml/Vulkan inference tenant to an AMD APU (gfx1151 / Strix Halo iGPU) that already serves another engine: APU memory census, the RADV heap gate, the build/runtime dependency surface, and co-tenancy measurement.
+
+Use when a ggml-based engine (llama.cpp, qwentts.cpp, anything built on ggml) must move from CPU to the **iGPU** of an AMD APU (gfx115x / Strix Halo / Ryzen AI) that **already serves a production tenant**. Answers: what budget actually exists, what gate decides feasibility before a model loads, and exactly which files a Vulkan build needs.
+
+## Why the usual VRAM reasoning is wrong on an APU
+
+`/sys/class/drm/card*/device/mem_info_vram_total` reports a small **carve-out** (2 GiB class), while the real working set lives in **GTT** — and GTT pages are *system pages*. A second GPU tenant therefore consumes host RAM from the same pool the first tenant and the OS draw from; there is no second memory to size against. The aperture is almost never the constraint; available RAM and shader time are.
+
+## Census (read-only; run before proposing anything)
+
+```bash
+for c in /sys/class/drm/card*; do
+  v=$(cat $c/device/mem_info_vram_used 2>/dev/null); g=$(cat $c/device/mem_info_gtt_used 2>/dev/null)
+  [ -n "$v$g" ] && echo "$(basename $c) vram_used=$((v/1048576))MiB gtt_used=$((g/1048576))MiB vram_total=$(($(cat $c/device/mem_info_vram_total)/1048576))MiB"
+done
+cat /sys/class/drm/card*/device/mem_info_gtt_total   # the aperture, in bytes
+free -g                                              # available is the number that bites
+```
+
+**Resolve the render node by driver, never by number** — on a hybrid host (dGPU + APU)
+`renderD128` is often the *other* vendor's node:
+
+```bash
+for r in /sys/class/drm/renderD*; do
+  echo "$(basename $r) $(basename $(readlink -f $r/device/driver))"
+done        # gfx1151 example on a box with an RTX 3090: renderD128 = nvidia, renderD129 = amdgpu
+```
+
+Also check what the *userspace* offers before promising a backend:
+
+```bash
+dpkg -l | grep -iE "mesa-vulkan|vulkan-tools"        # ICD present? probe installed?
+ls /usr/share/vulkan/icd.d/                          # radeon_icd.json = RADV available
+ls <image>/usr/share/vulkan/icd.d/                   # the ICD must exist INSIDE the container
+```
+
+## Gate 1 — the heap, before any model loads
+
+Ask what heap the Vulkan driver hands a tenant on this APU, **in the same container image
+that will host the engine**, with `/dev/dri` passed through:
+
+```bash
+podman run --rm --device /dev/dri -v <dir>:/probe:rw <engine-build-base> bash -lc '
+  apt-get update -qq && apt-get install -y -qq --no-install-recommends mesa-vulkan-drivers vulkan-tools
+  vulkaninfo --text > /probe/full.txt'          # NOT --batch: that flag does not exist and prints help, rc=1
+awk "/memoryHeaps: count/,/memoryTypes: count/" /probe/full.txt   # sizes + budgets + DEVICE_LOCAL flag
+```
+
+Measured on gfx1151 (Strix Halo, 116 GiB aperture) with a production ROCm tenant already
+loaded: `Radeon 8060S Graphics (RADV GFX1151)`, driver `radv`, API 1.4.318, heap 0
+host-visible **38.67 GiB** (budget 32.58) and heap 1 `DEVICE_LOCAL` **77.33 GiB** (budget
+**65.16**). Mesa splits the aperture roughly ⅓/⅔ on APUs, so **`DEVICE_LOCAL` is the
+aperture, not the 2 GiB carve-out** — the "if the heap is the carve-out, give up" branch
+does not trigger on this silicon. ggml asks for `DEVICE_LOCAL`, so a few-GiB engine lands
+inside a tens-of-GiB budget and memory stops being the question. Both heaps are GTT: the
+same host RAM, so a budget is a ceiling on *mappings*, not free silicon memory — `free`
+still decides whether it can be touched.
+
+Two probe traps: `vulkaninfo --batch` is not a flag in Ubuntu 24.04's `vulkan-tools` (it
+prints the help page and exits 1), and a slim container has **no `nvidia_icd.json`**, so
+Vulkan device 0 happens to be the iGPU there — an image that grows an NVIDIA ICD silently
+reorders the list. Select by device *name*, never by index.
+
+This is enumeration: no allocation, no compute — but it opens the render node on a
+production card, so it is still a decision for the operator, not a routine probe.
+
+## Gate 2 — contention, measured on the co-tenant
+
+If the target shape is **co-tenancy** (both resident), the honest measurement is the *existing* tenant's TTFT and decode with the new tenant's work in flight — one request each, no ladder. Stopping the co-tenant is for benchmarks, not for a deployment shape. Expect compute-unit sharing, not memory pressure.
+
+## The dependency surface people miss
+
+1. **Build stage needs more than the flag.** ggml compiles its Vulkan shaders at build time: `-DGGML_VULKAN=ON` alone fails at configure without `libvulkan-dev` **and** `glslang-dev` (or `glslang-tools`). A CPU-only image's build stage typically installs only `cmake g++ git make`.
+2. **The runtime stage needs an ICD, not just a device node.** `mesa-vulkan-drivers` must be installed *in the image*; `/dev/dri` without `radeon_icd.json` in the container buys nothing.
+3. **Rootless/quadlet device access**: `AddDevice=/dev/dri` (or the iGPU's own node — see the census: it is not necessarily `renderD128`) + `GroupAdd=keep-groups`, and the host needs the world-writable udev rule (`KERNEL=="renderD*", MODE="0666"`) — systemd user-unit spawns carry no supplementary groups, so the render group never arrives.
+4. Runtime selection is a device flag on the engine (`--device Vulkan0` for ggml-based tools), not an env var — and confirm which physical device that index resolves to after the image gains its ICD.
+
+## Laws
+
+- Price the tenant from a **booted** footprint, never from `weights + KV + buffers`: on an APU the compute buffers and any projector ride on GTT, which is host RAM — check `free` after the load, not before.
+- Never treat the `vram_total` carve-out as capacity, and never subtract a "dedicated" figure from it to find headroom.
+- Don't reason from `nvidia-smi`-style intuition: on this platform the useful numbers are `mem_info_gtt_used`, `free`, and RADV's heap budgets.
+- Unheld fabric clock (`pp_dpm_fclk` = `auto`) is a hazard only when the NPU joins too; a pure iGPU pair is unaffected — check it before adding a third tenant.
+- Swap on a host with GPU tenants protects co-tenants; paging a KV pool is worse than the OOM it prevents.