live: heartbeat sync 2026-10-19T15:24:41-07:00 (Refs #11)
This commit is contained in:
Frank committed
1 parent
6e3e5594eb
commit
c14c64d6c0
12 files changed
+452
-129
No files matched your search
@@ -1,89 +1,64 @@
|
||||
---
|
||||
name: amd-apu-vulkan-tenant-onboarding
|
||||
description: "Add a ggml/Vulkan inference tenant to an AMD APU (gfx1151 / Strix Halo iGPU) that already serves another engine: APU memory census, the measured RADV heap gate, render-node resolution on hybrid hosts, build/runtime dependency surface, co-tenancy measurement."
|
||||
description: "Add a ggml/Vulkan inference tenant to an AMD APU (gfx1151 / Strix Halo iGPU) that already serves another engine: APU memory census, RADV heap gate, render-node resolution, build-side glslc/ICD deps, co-tenancy measurement."
|
||||
---
|
||||
|
||||
Add a ggml/Vulkan inference tenant to an AMD APU (gfx1151 / Strix Halo iGPU) that already serves another engine: APU memory census, the RADV heap gate, the build/runtime dependency surface, and co-tenancy measurement.
|
||||
Use when a ggml-based engine (llama.cpp, qwentts.cpp, anything built on ggml) must move from CPU to the **iGPU** of an AMD APU (gfx1151 / Strix Halo / Ryzen AI Max) that **already serves another engine on the same card**. Answers two questions only a booted host can: what gate the second tenant must clear, and which files the build needs to clear it.
|
||||
|
||||
Use when a ggml-based engine (llama.cpp, qwentts.cpp, anything built on ggml) must move from CPU to the **iGPU** of an AMD APU (gfx115x / Strix Halo / Ryzen AI) that **already serves a production tenant**. Answers: what budget actually exists, what gate decides feasibility before a model loads, and exactly which files a Vulkan build needs.
|
||||
## 1. The usual VRAM reasoning is wrong on an APU
|
||||
|
||||
## Why the usual VRAM reasoning is wrong on an APU
|
||||
`/sys/class/drm/cardN/device/mem_info_vram_total` reports a small **carve-out** (2 GiB class), and `rocm-smi`'s "dedicated" figure is not the working set either. The real working set lives in **GTT / system pages**: `/proc/meminfo` → `GTT` and `System pages`. A second GPU tenant therefore consumes host RAM from the same pool the first tenant's weights and the OS use — no supplementary groups, no separate aperture to reserve.
|
||||
|
||||
`/sys/class/drm/card*/device/mem_info_vram_total` reports a small **carve-out** (2 GiB class), while the real working set lives in **GTT** — and GTT pages are *system pages*. A second GPU tenant therefore consumes host RAM from the same pool the first tenant and the OS draw from; there is no second memory to size against. The aperture is almost never the constraint; available RAM and shader time are.
|
||||
|
||||
## Census (read-only; run before proposing anything)
|
||||
|
||||
```bash
|
||||
for c in /sys/class/drm/card*; do
|
||||
v=$(cat $c/device/mem_info_vram_used 2>/dev/null); g=$(cat $c/device/mem_info_gtt_used 2>/dev/null)
|
||||
[ -n "$v$g" ] && echo "$(basename $c) vram_used=$((v/1048576))MiB gtt_used=$((g/1048576))MiB vram_total=$(($(cat $c/device/mem_info_vram_total)/1048576))MiB"
|
||||
done
|
||||
cat /sys/class/drm/card*/device/mem_info_gtt_total # the aperture, in bytes
|
||||
free -g # available is the number that bites
|
||||
```
|
||||
|
||||
**Resolve the render node by driver, never by number** — on a hybrid host (dGPU + APU)
|
||||
`renderD128` is often the *other* vendor's node:
|
||||
|
||||
```bash
|
||||
for r in /sys/class/drm/renderD*; do
|
||||
echo "$(basename $r) $(basename $(readlink -f $r/device/driver))"
|
||||
done # gfx1151 example on a box with an RTX 3090: renderD128 = nvidia, renderD129 = amdgpu
|
||||
```
|
||||
|
||||
Also check what the *userspace* offers before promising a backend:
|
||||
|
||||
```bash
|
||||
dpkg -l | grep -iE "mesa-vulkan|vulkan-tools" # ICD present? probe installed?
|
||||
ls /usr/share/vulkan/icd.d/ # radeon_icd.json = RADV available
|
||||
ls <image>/usr/share/vulkan/icd.d/ # the ICD must exist INSIDE the container
|
||||
```
|
||||
|
||||
## Gate 1 — the heap, before any model loads
|
||||
|
||||
Ask what heap the Vulkan driver hands a tenant on this APU, **in the same container image
|
||||
that will host the engine**, with `/dev/dri` passed through:
|
||||
|
||||
```bash
|
||||
podman run --rm --device /dev/dri -v <dir>:/probe:rw <engine-build-base> bash -lc '
|
||||
apt-get update -qq && apt-get install -y -qq --no-install-recommends mesa-vulkan-drivers vulkan-tools
|
||||
vulkaninfo --text > /probe/full.txt' # NOT --batch: that flag does not exist and prints help, rc=1
|
||||
awk "/memoryHeaps: count/,/memoryTypes: count/" /probe/full.txt # sizes + budgets + DEVICE_LOCAL flag
|
||||
```
|
||||
|
||||
Measured on gfx1151 (Strix Halo, 116 GiB aperture) with a production ROCm tenant already
|
||||
loaded: `Radeon 8060S Graphics (RADV GFX1151)`, driver `radv`, API 1.4.318, heap 0
|
||||
host-visible **38.67 GiB** (budget 32.58) and heap 1 `DEVICE_LOCAL` **77.33 GiB** (budget
|
||||
**65.16**). Mesa splits the aperture roughly ⅓/⅔ on APUs, so **`DEVICE_LOCAL` is the
|
||||
aperture, not the 2 GiB carve-out** — the "if the heap is the carve-out, give up" branch
|
||||
does not trigger on this silicon. ggml asks for `DEVICE_LOCAL`, so a few-GiB engine lands
|
||||
inside a tens-of-GiB budget and memory stops being the question. Both heaps are GTT: the
|
||||
same host RAM, so a budget is a ceiling on *mappings*, not free silicon memory — `free`
|
||||
still decides whether it can be touched.
|
||||
|
||||
Two probe traps: `vulkaninfo --batch` is not a flag in Ubuntu 24.04's `vulkan-tools` (it
|
||||
prints the help page and exits 1), and a slim container has **no `nvidia_icd.json`**, so
|
||||
Vulkan device 0 happens to be the iGPU there — an image that grows an NVIDIA ICD silently
|
||||
reorders the list. Select by device *name*, never by index.
|
||||
|
||||
This is enumeration: no allocation, no compute — but it opens the render node on a
|
||||
production card, so it is still a decision for the operator, not a routine probe.
|
||||
|
||||
## Gate 2 — contention, measured on the co-tenant
|
||||
|
||||
If the target shape is **co-tenancy** (both resident), the honest measurement is the *existing* tenant's TTFT and decode with the new tenant's work in flight — one request each, no ladder. Stopping the co-tenant is for benchmarks, not for a deployment shape. Expect compute-unit sharing, not memory pressure.
|
||||
|
||||
## The dependency surface people miss
|
||||
|
||||
1. **Build stage needs more than the flag.** ggml compiles its Vulkan shaders at build time: `-DGGML_VULKAN=ON` alone fails at configure without `libvulkan-dev` **and** `glslang-dev` (or `glslang-tools`). A CPU-only image's build stage typically installs only `cmake g++ git make`.
|
||||
2. **The runtime stage needs an ICD, not just a device node.** `mesa-vulkan-drivers` must be installed *in the image*; `/dev/dri` without `radeon_icd.json` in the container buys nothing.
|
||||
3. **Rootless/quadlet device access**: `AddDevice=/dev/dri` (or the iGPU's own node — see the census: it is not necessarily `renderD128`) + `GroupAdd=keep-groups`, and the host needs the world-writable udev rule (`KERNEL=="renderD*", MODE="0666"`) — systemd user-unit spawns carry no supplementary groups, so the render group never arrives.
|
||||
4. Runtime selection is a device flag on the engine (`--device Vulkan0` for ggml-based tools), not an env var — and confirm which physical device that index resolves to after the image gains its ICD.
|
||||
|
||||
## Laws
|
||||
|
||||
- Price the tenant from a **booted** footprint, never from `weights + KV + buffers`: on an APU the compute buffers and any projector ride on GTT, which is host RAM — check `free` after the load, not before.
|
||||
- Never treat the `vram_total` carve-out as capacity, and never subtract a "dedicated" figure from it to find headroom.
|
||||
- Don't reason from `nvidia-smi`-style intuition: on this platform the useful numbers are `mem_info_gtt_used`, `free`, and RADV's heap budgets.
|
||||
- Unheld fabric clock (`pp_dpm_fclk` = `auto`) is a hazard only when the NPU joins too; a pure iGPU pair is unaffected — check it before adding a third tenant.
|
||||
- Price the tenant from a **booted** footprint, never from `weights + KV + buffers`: on an APU the compute buffers and any projector ride on GTT, which is host RAM; check `free` *after* the load, not before.
|
||||
- Never subtract a "dedicated" figure from it to find headroom.
|
||||
- Unheld fabric clock (`pp_dpm_mech` = `auto`) is a hazard only when the NPU joins too; a pure iGPU pair is unaffected — check it before adding a third tenant.
|
||||
- Swap on a host with GPU tenants protects co-tenants; paging a KV pool is worse than the OOM it prevents.
|
||||
|
||||
## 2. The gate that decides feasibility: the RADV heap a container sees
|
||||
|
||||
`rocm-smi` and `free` cannot see this number; only a container off the **engine's own base image** can:
|
||||
|
||||
```bash
|
||||
# one-off container, no model, no load, no compute — safe beside production
|
||||
podman run --rm --device /dev/dri --group-add keep-groups <engine-base> bash -c \
|
||||
'apt-get update -qq >/dev/null 2>&1; apt-get install -y -qq --no-install-recommends mesa-vulkan-drivers vulkan-tools glslang-tools >/dev/null 2>&1; vulkaninfo --text'
|
||||
```
|
||||
|
||||
Read `VkPhysicalDeviceMemoryProperties` → the `DEVICE_LOCAL` heap size. On a hybrid Strix Halo box Mesa splits the aperture (~1/3 : 2/3), so `DEVICE_LOCAL` is the budget the ggml tool will actually allocate against — measured 77.33 GiB *with the first tenant loaded*, against a 2 GiB DRM carve-out. Two traps, both measured:
|
||||
|
||||
- `vulkaninfo --batch` does not exist in Ubuntu 24.04's `vulkan-tools` (prints help, exits 1). Every `vulkaninfo` parse needs `--text`.
|
||||
- The container may have **no NVIDIA ICD**, so Vulkan index `0` *is* the iGPU — but do not trust ordering by index: `--text` and read `deviceName` + `driverID` (`RADV_GFX1151` vs `NVIDIA`). A hand-picked `/dev/dri/renderDN` is worse than useless on a hybrid host: **`renderD128` there is the NVIDIA card and `renderD129` the amdgpu iGPU**; resolve the node with `readlink -f /sys/class/drm/renderD*/device/driver` and pass `/dev/dri` wholesale, letting the driver pick by name.
|
||||
- A rootless container that drops its supplementary groups enumerates **nothing**. `--group-add keep-groups` (quadlet: `GroupAdd=keep-groups`) plus the host udev rule that makes the render nodes group-writable is the whole requirement; `--device /dev/dri` alone is not enough.
|
||||
|
||||
## 3. Build-side deps: the configure fails before the device ever matters
|
||||
|
||||
ggml's `src/ggml-vulkan/CMakeLists.txt` requires, in order: `find_package(Vulkan COMPONENTS glslc REQUIRED)` then `find_package(SPIRV-Headers CONFIG REQUIRED)`, and it compiles its shaders with **glslc**, never `glslangValidator`. On Ubuntu 24.04 (measured against the digest-pinned `ubuntu:24.04` base, 2026-10-19):
|
||||
|
||||
- `glslang-tools` ships `glslangValidator` and **NOT `glslc`** → `command -v glslc` empty, `find_package(Vulkan … glslc REQUIRED)` fails, and the Vulkan backend silently never exists.
|
||||
- The package is the standalone **`glslc`** (universe, `2023.8-1build1`). Working build-stage set: `libvulkan-dev glslc spirv-headers` — `libvulkan-dev` provides `vulkan/vulkan.h` + the `VulkanHeaders`/`VulkanLoader` CMake configs `find_package(Vulkan)` resolves to, `spirv-headers` ships `/usr/share/cmake/SPIRV-Headers/SPIRV-HeadersConfig.cmake`. A configure probe in the pinned base prints `Found Vulkan: … found components: glslc` then `Including Vulkan backend`.
|
||||
- **The archive's glslc is old.** 2023.8 compiles `GL_KHR_cooperative_matrix` (the path ggml's vectored quantized matmul needs) but reports `cooperative_matrix2`, `integer_dot_product`, `bfloat16`, `float_e2m1/e4m3` as unsupported, so those shader families are compiled out. If a Vulkan prefill reads as inexplicably slow, suspect the **shader compiler before the device**; the fix is a pinned glslang build passed as `-DVulkan_GLSLC_EXECUTABLE=<path>`, not a different runtime.
|
||||
- Runtime stage needs the loader **and an ICD**: `libvulkan1` + `mesa-vulkan-drivers`. Loader-without-ICD enumerates zero devices, and ggml-based servers die in device lookup — often *after* reading the weights, so the journal reads like a model fault.
|
||||
- Ship **no NVIDIA ICD** on purpose: one Vulkan device in the enumeration is what makes the *name* `Vulkan0` safe to write into config. Prefer the name over an index in every flag.
|
||||
|
||||
## 4. Prove enumeration without allocating anything
|
||||
|
||||
Do not boot a scratch second engine beside production to ask which devices exist — a second model load is a real GPU tenant, and fleet law stops that. Instead run the binary with a device name that cannot exist and read its error; it names what it enumerated, with no allocation and no model resident:
|
||||
|
||||
```bash
|
||||
podman run --rm --device /dev/dri --group-add keep-groups --entrypoint /app/tts-server <image> \
|
||||
--model /dev/null --codec /dev/null --device NoSuchDevice 2>&1 | tail -20
|
||||
```
|
||||
|
||||
## 5. The rollback must be a flag, not a rebuild
|
||||
|
||||
Put the device in the env file as a value the entrypoint passes only when set, and make **set-but-empty** mean "pass no flag at all" — the binary's own default (CPU, always built) is the rollback. `: "${VAR:=default}"` is the WRONG form: `:=` fires on an empty value too, so the documented rollback line `VAR=` is silently rewritten back to the GPU and the only way out is a rebuild. Use:
|
||||
|
||||
```bash
|
||||
if [[ ! -v QWENTTS_DEVICE ]]; then QWENTTS_DEVICE=Vulkan0; fi
|
||||
```
|
||||
|
||||
(`podman`'s `--env-file` passes `VAR=` through as set-and-empty, and `source` does the same, so both readers agree.) Pin it in a test: empty device ⇒ the flag is absent from argv.
|
||||
|
||||
## 6. Co-tenancy measurement (only after the tenant is live)
|
||||
|
||||
The gate is the **first tenant's** performance with a real request from the second in flight — not the second tenant's own speed. And price the tenant from a booted footprint, never from weights+KV+buffers: on an APU the compute buffers and any projector ride on GTT, which is host RAM; check `free` after the load, not before.
|
||||
Reference in new issue
Block a user