Frank
065d796d73
The trial's quadlet line said AddDevice=/dev/dri/renderD128. Resolved the nodes by driver instead of assuming: on this hybrid host renderD128 -> nvidia and renderD129 -> amdgpu, so the guess pointed the engine at the wrong card. Also records the gate-1 command that works (vulkaninfo --text; --batch prints help and exits 1) and the probe script path.
181 lines
13 KiB
Markdown
181 lines
13 KiB
Markdown
# TTS on the Strix Halo NPU — spike verdict (2026-10-19)
|
||
|
||
Question: **how can the fleet's TTS engine run on hardware rather than CPU?** Answer in
|
||
one line: not on the NPU — the loader, not the silicon, refuses it; the door that exists is
|
||
the gfx1151 iGPU behind a build flag we already own.
|
||
|
||
Spike evidence lives in `report/npu-tts-runtime-landscape-2026-10-19.md` (which runtimes can
|
||
reach XDNA2 at all) and `report/qwen3-tts-payload-portability-2026-10-19.md` (what the
|
||
payload is and where it can be moved). This page owns only the **measured spike result** and
|
||
the **model-side contract** neither slice has. Throwaway work ran in containers under
|
||
`/home/dank/spikes/npu-tts` on Dankclaw; nothing in the fleet was changed — `frank-tts` and
|
||
`frank-tts-engine` still serve `narrator1|2|3` on CPU.
|
||
|
||
Legend: **[host]** live census 2026-10-19 · **[spike]** measured in the spike containers ·
|
||
**[vendor]** primary vendor source, with the version it holds for.
|
||
|
||
## 1. Verdict
|
||
|
||
**NO-GO on the NPU, and the wall is a loader.** The XDNA2 device is healthy by AMD's own
|
||
instruments; the runtime that owns this model family opens the device and then declines the
|
||
graph, silently. One run loaded successfully; byte-identical runs never reproduced it, and a
|
||
path that cannot be reproduced is not a foundation. **[spike]**
|
||
|
||
**Go on the iGPU, and the lever is in this repo.**
|
||
`dankclaw/voiceservice/Containerfile.engine` builds `qwentts.cpp` (commit `5916bf7`) with
|
||
`-DGGML_VULKAN=OFF` written out deliberately beside CUDA/HIP/METAL/SYCL/RPC — the running
|
||
container carries `libggml-base.so.0.27.0` + `libggml-cpu.so.0.27.0` + `tts-server` and
|
||
nothing else. So: `-DGGML_VULKAN=ON` + `--device Vulkan0`, plus `/dev/dri` passthrough in
|
||
the engine quadlet (the ComfyUI pattern: `AddDevice` + `GroupAdd=keep-groups` + a
|
||
world-writable udev rule). The target shape is **co-tenancy** — the engine resident on the
|
||
gfx1151 *beside* `frank-dragoon`, not a handover. §5 carries the census and both gates:
|
||
the **heap gate is PASSED** — RADV's `DEVICE_LOCAL` heap is 77.33 GiB with 65.16 GiB of
|
||
budget while dragoon is loaded, so memory is not the constraint — which leaves the
|
||
contention gate (dragoon's TTFT with a synthesis in flight) and one unproven piece:
|
||
whether qwentts.cpp's codec ops have Vulkan kernels for every op. **[host]** **[spike]**
|
||
|
||
Why the iGPU is worth doing at all, given CPU already works: the deployed payload needs
|
||
only **12.5 codes/s** (§2), so this is latency headroom and freed cores, not reviving a
|
||
dead service. The independent Strix Halo measurement in the portability slice ("use Vulkan
|
||
exclusively"; NPU decode under 1.5 t/s for an 8B) points the same way.
|
||
|
||
## 2. The two arms do not share a real-time bar
|
||
|
||
|Arm|Talker|Codec|Codes per second of audio|Bar|
|
||
|---|---|---|---|---|
|
||
|**Deployed** (`frank-tts-engine`)|`qwen-talker-1.7b-base-Q4_K_M.gguf`|`qwen-tokenizer-12hz-Q8_0.gguf`|**12.5**|~13 tok/s **[host]**|
|
||
|**Spike** (`amd/neutts-air_rai_1.8.0_npu_4K`)|NuTTS-Air, 24-layer Qwen2|NeuCodec `24000 / hop 480`|**50.00**|~50 tok/s **[spike]**|
|
||
|
||
Measured from the staged NPU artifact — the contract any future arm must satisfy:
|
||
|
||
|Fact|Value|
|
||
|---|---|
|
||
Vocab|`genai_config.json` `vocab_size` **217,652**; base BPE 151,643; specials 151,644–151,670 **[spike]**|
|
||
Speech tokens|**65,536** `<|speech_N|>` entries, ids **151,671–217,206** contiguous; the vendor template maps a codec index straight to its id, `"".join(f"<|speech_{i}|>")`. A 10,001-code assumption silently truncates the space — encoded references really do reach the 60k range. **[spike]**|
|
||
Stop token|`<|SPEECH_GENERATION_END|>` = 151,670, but the export's `model.eos_token_id` is `[151645, 151643]` (the *chat turn* stop) and `search.eos_token_ids` is unset — pass 151,670 explicitly or speech ends after one token **[spike]**|
|
||
Input ceiling|4096 tokens **total**, input + output **[vendor]**|
|
||
Frame rate|50.00 codes/s confirmed across three references (49.96 / 49.99 / 50.00) **[spike]**|
|
||
Reference encode|`librosa.load(sr=16000, mono=True)` → `[1,1,T]` float → `codec.encode_code(audio_or_path=…)` (vendor `NeuTTS.encode_reference`); CPU RTF 0.28–0.38 on 8 threads, so codes are a cacheable one-time cost, never a per-request charge. `neucodec` also needs `torchtune`; install both `--no-deps --target <dir>` so a second torch never shadows the venv's. **[spike]**|
|
||
Pre-encoded voices|`/spike/logs/codes-narrator{1,2,3}.npy` — 994 / 1,249 / 1,250 codes for 19.90 / 24.98 / 25.00 s **[spike]**|
|
||
|
||
**Gap that any arm inherits:** the voice packs carry no transcript. `manifest.json` holds
|
||
provenance and crop geometry (`crop.source_speech_run` 98.739–118.636 s of a 22,569 s
|
||
audiobook) and no text, and Dankclaw has no ASR at all — `whisper`, `whisper.cpp`,
|
||
`faster-whisper` all absent. NuTTS conditioning needs `ref_text`; do not invent it. **[host]**
|
||
|
||
## 3. Exoneration chain — why the blame is not the hardware
|
||
|
||
Every row is an instrument, not an inference.
|
||
|
||
|Probe|Result|
|
||
|---|---|
|
||
`xrt-smi examine`|`[0000:c6:00.1] RyzenAI-npu5 aie2p 6x8`, XRT 2.25.37 (vendor extract), NPU fw 1.1.2.65, `amdxdna 7.0.0-38-generic` **[spike]**|
|
||
`xrt-smi validate`|**PASSED**, including `throughput` **[spike]**|
|
||
`pyxrt.device(0)` in-container|constructs — XRT enumerates the device **[spike]**|
|
||
`strace -f -e trace=openat` on the failing load|`genai_config.json` read → `model.onnx` opened → `libxrt_driver_xdna.so.2` loaded → `/dev/accel/accel0` opened `O_RDWR` → GEM ioctls succeed → **EP declines, before touching `dd_metastate_*` or `cache/txn_bins.zip`** **[spike]**|
|
||
`dmesg`, `fuser /dev/accel/accel0`|silent since boot, no holder — the decline never reaches the kernel **[spike]**|
|
||
Hardlinked mirror dirs of the model tree (`rundir*`)|retired, no change — the "metastate shadowing" theory is dead **[spike]**|
|
||
Every logging lever in that build: `GENAI_ORT_VERBOSE_LOGGING`, `GENAI_LOG_ORT_LIB`, `set_log_options`, `set_log_callback(callable)`, `RYZENAI_EP_TRACING`, `RYZENAI_LLM_PROFILE_INIT`|**zero lines** — the decline path is silent **[spike]**|
|
||
Vendor `quicktest`|`Session successfully initialized` — proves the XRT/driver/device path, **not** NPU residency: that build falls back to CPU silently. Prove residency with `/sys/class/accel/accel0/device/mem_usage` during generation. **[spike] [vendor]**|
|
||
|
||
No NPU token rate was ever measured, so none is quoted here. The single successful load
|
||
(13:17, `[load] 0.84s`, finite logits, XRT banner at *generate* time) has not been
|
||
reproduced by the byte-identical command since, with the model files pristine by inode and
|
||
mtime. **[spike]**
|
||
|
||
## 4. Host gates the spike proved (so they are not re-litigated)
|
||
|
||
- **Rootless containers cannot read `/dev/accel/accel0`** (`root:render` 0660): the host
|
||
gid never enters the userns map. `--group-add keep-groups`, `--userns=keep-id`,
|
||
`--userns=keep-id:gid=990`, `--group-add render`, `--group-add gid:990` — all fail.
|
||
Doors: rootful podman (separate storage), or a world-writable udev rule, which the fleet
|
||
already does for `kfd`/`renderD*`. **[spike]**
|
||
- **Never `dpkg -i` the vendor XRT deb** — it installs a DKMS `amdxdna` over the distro
|
||
driver. `dpkg-deb -x` the debs and bind-mount the tree at `/opt/xilinx/xrt`; the plugin
|
||
deb is what carries `libxrt_driver_xdna.so`. **[spike]**
|
||
- In-tree `amdxdna 0.7.0-38` + `iommu=pt` is enough for context creation; a missing device
|
||
node is *not* the modern failure mode. **[host]**
|
||
- `xrt-smi` inside a container needs `libboost-filesystem1.83.0` +
|
||
`libboost-program-options1.83.0` (**hyphenated** Ubuntu names, after `apt update`);
|
||
boost is needed by `xrt-smi` only, never by the OGA/EP path. **[spike]**
|
||
- Headroom is tight and it is host RAM, not a separate pool — see §5 for tonight's census.
|
||
A multi-GB NPU partition plus an AIE compile workspace does not fit beside
|
||
`frank-dragoon`. Benign-looking `Failed to alloc dev bo memory, ret -28` is that
|
||
pressure. **[host]**
|
||
|
||
## 5. Target shape: the TTS engine shares the gfx1151 with `frank-dragoon`
|
||
|
||
The deployment Frank wants is **both resident at once**, not a handover. Measured 2026-10-19
|
||
(read-only, nothing loaded or unloaded):
|
||
|
||
|Measure|Value|Reading|
|
||
|---|---|---|
|
||
iGPU GTT in use|**18,523 MiB of 116,733 MiB**|≈96 GiB of aperture free; this is the real budget (fleet law) **[host]**|
|
||
DRM "VRAM" carve-out|2,048 MiB total, 174 MiB used|what KFD/DRM calls dedicated — **not** the budget **[host]**|
|
||
Host RAM|121 GiB total, 17 GiB available, 79 GiB buff/cache|**GTT pages are system pages**: an iGPU allocation is host RAM consumed, so the two tenants share one pool **[host]**|
|
||
TTS engine today|3.84 GiB RSS, CPU|its device-resident replacement lands in that same pool **[host]**|
|
||
|
||
**Two gates, in this order.**
|
||
|
||
1. **Heap gate — PASSED.** RADV does **not** hand a container tenant the 2 GiB DRM
|
||
carve-out. `vulkaninfo` inside the engine's own base image (`ubuntu:24.04` plus
|
||
`mesa-vulkan-drivers vulkan-tools`, `--device /dev/dri`, rootless) sees
|
||
`Radeon 8060S Graphics (RADV GFX1151)`, driver `radv`, API 1.4.318, and reports:
|
||
|
||
|Heap|size|budget with dragoon loaded|flags|
|
||
|---|---|---|---|
|
||
|0|38.67 GiB|32.58 GiB|host-visible, not device-local|
|
||
|1|**77.33 GiB**|**65.16 GiB**|`DEVICE_LOCAL`|
|
||
|
||
The heaps sum to the ~116 GiB aperture in Mesa's ⅓/⅔ APU split — the same RAM twice:
|
||
a Vulkan allocation is a GTT mapping, so a budget is a ceiling on mappings, not a pile
|
||
of free silicon memory. ggml's Vulkan backend asks for `DEVICE_LOCAL`, which is heap 1,
|
||
and the TTS engine needs roughly 4 GiB of it. **Memory stops being the question.** **[spike]**
|
||
Probe traps: `vulkaninfo --batch` is not a flag in this `vulkan-tools` (it prints the
|
||
help page and exits 1 — use `vulkaninfo --text`), and the container has **no NVIDIA
|
||
ICD**, so Vulkan device 0 is the iGPU here. Do not trust that ordering — select by
|
||
device *name*, or an image that grows an `nvidia_icd.json` silently points TTS at the
|
||
3090 slot.
|
||
2. **Contention gate.** `frank-dragoon` is the endpoint the operator session runs on, so the
|
||
honest measurement with both loaded is dragoon's own TTFT and decode with a synthesis in
|
||
flight — one request each, no ladder. Expect compute-unit sharing rather than memory
|
||
pressure: the aperture has room, the shaders do not.
|
||
|
||
**Change surface** — the whole edit, so nobody re-derives it:
|
||
|
||
1. `Containerfile.engine` build stage: add `libvulkan-dev glslang-dev`. It installs only
|
||
`ca-certificates cmake g++ git make` today, and ggml's Vulkan backend compiles its
|
||
shaders at build time — the flag alone fails at configure.
|
||
2. `-DGGML_VULKAN=OFF` → `ON` in the same cmake block; the flag list is written out
|
||
explicitly, so this is a one-word diff.
|
||
3. Runtime stage: add `mesa-vulkan-drivers`. Without a Vulkan **ICD inside the image** the
|
||
device node buys nothing — there is no `radeon_icd.json` in the container today.
|
||
4. `frank-tts-engine.container`: `AddDevice=/dev/dri` + `GroupAdd=keep-groups` — **not**
|
||
`renderD128`. On this hybrid box that node is the NVIDIA card; the iGPU is
|
||
`renderD129` (`readlink -f /sys/class/drm/renderD129/device/driver` → `amdgpu`).
|
||
5. Engine argv: `--device Vulkan0` — qwentts.cpp's tools take `--device <name>` at the
|
||
pinned commit.
|
||
|
||
Gate 1 recipe, already run and **PASSED**: `mesa-vulkan-drivers` + `vulkan-tools` in a
|
||
throwaway container off the same base, `--device /dev/dri`, then `vulkaninfo --text` and read
|
||
the `VkPhysicalDeviceMemoryProperties` heap sizes — no model, no load, no compute. The probe
|
||
trap: `vulkaninfo --batch` is not a flag in Ubuntu 24.04's `vulkan-tools` — it prints the help
|
||
page and exits 1, which reads like a driver failure. Every package involved has a candidate on
|
||
the host's apt (`vulkan-tools` 1.4.341, `libvulkan-dev` 1.4.341, `glslang-dev`/`glslang-tools`
|
||
16.2.0); the build stage runs on the digest-pinned Ubuntu 24.04 base, whose versions are older
|
||
but present. Script: `/home/dank/spikes/npu-tts/vk_probe.sh` on Dankclaw.
|
||
|
||
Constraints that do not relax for a co-tenant: `/dev/dri` passthrough needs the ComfyUI
|
||
pattern (`AddDevice` + `GroupAdd=keep-groups` + the world-writable udev rule) because
|
||
systemd user-unit spawns carry no supplementary groups; the 64 GiB swap at `/swap.img`
|
||
exists to protect co-tenants, and paging a KV pool is worse than the OOM it prevents;
|
||
`pp_dpm_fclk` reads `auto` (unheld), which only becomes a hazard if the NPU ever joins.
|
||
|
||
## 6. Re-visit condition
|
||
|
||
Retry the NPU when **a Linux loader claims the graph**: an `onnxruntime-genai-ryzenai`
|
||
build whose RyzenAI adapter takes the NuTTS-Air Full-Fusion graph on `RyzenAI-npu5`, or that
|
||
model appearing in a runtime's own supported catalog with a Linux runner. Then the
|
||
prerequisites are already mapped and priced: the gated NeuCodec encoder for cloned voices,
|
||
the `ref_text` transcripts, the 4096-token prompt budget, and 50 codes/s as the bar.
|