# TTS on the Strix Halo NPU — spike verdict (2026-10-19)

Question: **how can the fleet's TTS engine run on hardware rather than CPU?** Answer in
one line: not on the NPU — the loader, not the silicon, refuses it; the door that exists is
the gfx1151 iGPU behind a build flag we already own.

Spike evidence lives in `report/npu-tts-runtime-landscape-2026-10-19.md` (which runtimes can
reach XDNA2 at all) and `report/qwen3-tts-payload-portability-2026-10-19.md` (what the
payload is and where it can be moved). This page owns only the **measured spike result** and
the **model-side contract** neither slice has. Throwaway work ran in containers under
`/home/dank/spikes/npu-tts` on Dankclaw; nothing in the fleet was changed — `frank-tts` and
`frank-tts-engine` still serve `narrator1|2|3` on CPU.

Legend: **[host]** live census 2026-10-19 · **[spike]** measured in the spike containers ·
**[vendor]** primary vendor source, with the version it holds for.

## 1. Verdict

**NO-GO on the NPU, and the wall is a loader.** The XDNA2 device is healthy by AMD's own
instruments; the runtime that owns this model family opens the device and then declines the
graph, silently. One run loaded successfully; byte-identical runs never reproduced it, and a
path that cannot be reproduced is not a foundation. **[spike]**

**Go on the iGPU, and the lever is in this repo.**
`dankclaw/voiceservice/Containerfile.engine` builds `qwentts.cpp` (commit `5916bf7`) with
`-DGGML_VULKAN=OFF` written out deliberately beside CUDA/HIP/METAL/SYCL/RPC — the running
container carries `libggml-base.so.0.27.0` + `libggml-cpu.so.0.27.0` + `tts-server` and
nothing else. So: `-DGGML_VULKAN=ON` + `--device Vulkan0`, plus `/dev/dri` passthrough in
the engine quadlet (the ComfyUI pattern: `AddDevice` + `GroupAdd=keep-groups` + a
world-writable udev rule). The target shape is **co-tenancy** — the engine resident on the
gfx1151 *beside* `frank-dragoon`, not a handover. §5 carries the census and both gates:
the **heap gate is PASSED** — RADV's `DEVICE_LOCAL` heap is 77.33 GiB with 65.16 GiB of
budget while dragoon is loaded, so memory is not the constraint — which leaves the
contention gate (dragoon's TTFT with a synthesis in flight) and one unproven piece:
whether qwentts.cpp's codec ops have Vulkan kernels for every op. **[host]** **[spike]**

Why the iGPU is worth doing at all, given CPU already works: the deployed payload needs
only **12.5 codes/s** (§2), so this is latency headroom and freed cores, not reviving a
dead service. The independent Strix Halo measurement in the portability slice ("use Vulkan
exclusively"; NPU decode under 1.5 t/s for an 8B) points the same way.

## 2. The two arms do not share a real-time bar

|Arm|Talker|Codec|Codes per second of audio|Bar|
|---|---|---|---|---|
|**Deployed** (`frank-tts-engine`)|`qwen-talker-1.7b-base-Q4_K_M.gguf`|`qwen-tokenizer-12hz-Q8_0.gguf`|**12.5**|~13 tok/s **[host]**|
|**Spike** (`amd/neutts-air_rai_1.8.0_npu_4K`)|NuTTS-Air, 24-layer Qwen2|NeuCodec `24000 / hop 480`|**50.00**|~50 tok/s **[spike]**|

Measured from the staged NPU artifact — the contract any future arm must satisfy:

|Fact|Value|
|---|---|
Vocab|`genai_config.json` `vocab_size` **217,652**; base BPE 151,643; specials 151,644–151,670 **[spike]**|
Speech tokens|**65,536** `<|speech_N|>` entries, ids **151,671–217,206** contiguous; the vendor template maps a codec index straight to its id, `"".join(f"<|speech_{i}|>")`. A 10,001-code assumption silently truncates the space — encoded references really do reach the 60k range. **[spike]**|
Stop token|`<|SPEECH_GENERATION_END|>` = 151,670, but the export's `model.eos_token_id` is `[151645, 151643]` (the *chat turn* stop) and `search.eos_token_ids` is unset — pass 151,670 explicitly or speech ends after one token **[spike]**|
Input ceiling|4096 tokens **total**, input + output **[vendor]**|
Frame rate|50.00 codes/s confirmed across three references (49.96 / 49.99 / 50.00) **[spike]**|
Reference encode|`librosa.load(sr=16000, mono=True)` → `[1,1,T]` float → `codec.encode_code(audio_or_path=…)` (vendor `NeuTTS.encode_reference`); CPU RTF 0.28–0.38 on 8 threads, so codes are a cacheable one-time cost, never a per-request charge. `neucodec` also needs `torchtune`; install both `--no-deps --target <dir>` so a second torch never shadows the venv's. **[spike]**|
Pre-encoded voices|`/spike/logs/codes-narrator{1,2,3}.npy` — 994 / 1,249 / 1,250 codes for 19.90 / 24.98 / 25.00 s **[spike]**|

**Gap that any arm inherits:** the voice packs carry no transcript. `manifest.json` holds
provenance and crop geometry (`crop.source_speech_run` 98.739–118.636 s of a 22,569 s
audiobook) and no text, and Dankclaw has no ASR at all — `whisper`, `whisper.cpp`,
`faster-whisper` all absent. NuTTS conditioning needs `ref_text`; do not invent it. **[host]**

## 3. Exoneration chain — why the blame is not the hardware

Every row is an instrument, not an inference.

|Probe|Result|
|---|---|
`xrt-smi examine`|`[0000:c6:00.1] RyzenAI-npu5 aie2p 6x8`, XRT 2.25.37 (vendor extract), NPU fw 1.1.2.65, `amdxdna 7.0.0-38-generic` **[spike]**|
`xrt-smi validate`|**PASSED**, including `throughput` **[spike]**|
`pyxrt.device(0)` in-container|constructs — XRT enumerates the device **[spike]**|
`strace -f -e trace=openat` on the failing load|`genai_config.json` read → `model.onnx` opened → `libxrt_driver_xdna.so.2` loaded → `/dev/accel/accel0` opened `O_RDWR` → GEM ioctls succeed → **EP declines, before touching `dd_metastate_*` or `cache/txn_bins.zip`** **[spike]**|
`dmesg`, `fuser /dev/accel/accel0`|silent since boot, no holder — the decline never reaches the kernel **[spike]**|
Hardlinked mirror dirs of the model tree (`rundir*`)|retired, no change — the "metastate shadowing" theory is dead **[spike]**|
Every logging lever in that build: `GENAI_ORT_VERBOSE_LOGGING`, `GENAI_LOG_ORT_LIB`, `set_log_options`, `set_log_callback(callable)`, `RYZENAI_EP_TRACING`, `RYZENAI_LLM_PROFILE_INIT`|**zero lines** — the decline path is silent **[spike]**|
Vendor `quicktest`|`Session successfully initialized` — proves the XRT/driver/device path, **not** NPU residency: that build falls back to CPU silently. Prove residency with `/sys/class/accel/accel0/device/mem_usage` during generation. **[spike] [vendor]**|

No NPU token rate was ever measured, so none is quoted here. The single successful load
(13:17, `[load] 0.84s`, finite logits, XRT banner at *generate* time) has not been
reproduced by the byte-identical command since, with the model files pristine by inode and
mtime. **[spike]**

## 4. Host gates the spike proved (so they are not re-litigated)

- **Rootless containers cannot read `/dev/accel/accel0`** (`root:render` 0660): the host
  gid never enters the userns map. `--group-add keep-groups`, `--userns=keep-id`,
  `--userns=keep-id:gid=990`, `--group-add render`, `--group-add gid:990` — all fail.
  Doors: rootful podman (separate storage), or a world-writable udev rule, which the fleet
  already does for `kfd`/`renderD*`. **[spike]**
- **Never `dpkg -i` the vendor XRT deb** — it installs a DKMS `amdxdna` over the distro
  driver. `dpkg-deb -x` the debs and bind-mount the tree at `/opt/xilinx/xrt`; the plugin
  deb is what carries `libxrt_driver_xdna.so`. **[spike]**
- In-tree `amdxdna 0.7.0-38` + `iommu=pt` is enough for context creation; a missing device
  node is *not* the modern failure mode. **[host]**
- `xrt-smi` inside a container needs `libboost-filesystem1.83.0` +
  `libboost-program-options1.83.0` (**hyphenated** Ubuntu names, after `apt update`);
  boost is needed by `xrt-smi` only, never by the OGA/EP path. **[spike]**
- Headroom is tight and it is host RAM, not a separate pool — see §5 for tonight's census.
  A multi-GB NPU partition plus an AIE compile workspace does not fit beside
  `frank-dragoon`. Benign-looking `Failed to alloc dev bo memory, ret -28` is that
  pressure. **[host]**

## 5. Target shape: the TTS engine shares the gfx1151 with `frank-dragoon`

The deployment Frank wants is **both resident at once**, not a handover. Measured 2026-10-19
(read-only, nothing loaded or unloaded):

|Measure|Value|Reading|
|---|---|---|
iGPU GTT in use|**18,523 MiB of 116,733 MiB**|≈96 GiB of aperture free; this is the real budget (fleet law) **[host]**|
DRM "VRAM" carve-out|2,048 MiB total, 174 MiB used|what KFD/DRM calls dedicated — **not** the budget **[host]**|
Host RAM|121 GiB total, 17 GiB available, 79 GiB buff/cache|**GTT pages are system pages**: an iGPU allocation is host RAM consumed, so the two tenants share one pool **[host]**|
TTS engine today|3.84 GiB RSS, CPU|its device-resident replacement lands in that same pool **[host]**|

**Two gates, in this order.**

1. **Heap gate — PASSED.** RADV does **not** hand a container tenant the 2 GiB DRM
   carve-out. `vulkaninfo` inside the engine's own base image (`ubuntu:24.04` plus
   `mesa-vulkan-drivers vulkan-tools`, `--device /dev/dri`, rootless) sees
   `Radeon 8060S Graphics (RADV GFX1151)`, driver `radv`, API 1.4.318, and reports:

   |Heap|size|budget with dragoon loaded|flags|
   |---|---|---|---|
   |0|38.67 GiB|32.58 GiB|host-visible, not device-local|
   |1|**77.33 GiB**|**65.16 GiB**|`DEVICE_LOCAL`|

   The heaps sum to the ~116 GiB aperture in Mesa's ⅓/⅔ APU split — the same RAM twice:
   a Vulkan allocation is a GTT mapping, so a budget is a ceiling on mappings, not a pile
   of free silicon memory. ggml's Vulkan backend asks for `DEVICE_LOCAL`, which is heap 1,
   and the TTS engine needs roughly 4 GiB of it. **Memory stops being the question.** **[spike]**
   Probe traps: `vulkaninfo --batch` is not a flag in this `vulkan-tools` (it prints the
   help page and exits 1 — use `vulkaninfo --text`), and the container has **no NVIDIA
   ICD**, so Vulkan device 0 is the iGPU here. Do not trust that ordering — select by
   device *name*, or an image that grows an `nvidia_icd.json` silently points TTS at the
   3090 slot.
2. **Contention gate — PASSED, measured 2026-10-19 after the cutover.** With a 16.9 s
   synthesis occupying the device, `frank-dragoon` answered a streamed 124-token request at
   **17 ms to first token and ~35 tok/s decode**, against 17 ms and ~33 tok/s for the same
   request with the card idle. Compute-unit sharing showed up as nothing. One request per
   arm, no ladder: these are the gate's evidence, not a throughput record, and the engine's
   own shape lives in [[Frank TTS Engine]].

**Change surface — as landed** (PR #34, `6248f02`; the CPU shape stays reachable as the
rollback):

1. `Containerfile.engine` build stage: add **`libvulkan-dev glslc spirv-headers`**. The
   prediction above (`libvulkan-dev glslang-dev`) is wrong on two counts and the configure
   says so: ggml looks for `glslc`, which Ubuntu's `glslang-tools` does **not** ship (it
   ships `glslangValidator`), and `find_package(SPIRV-Headers CONFIG REQUIRED)` needs the
   distro's `spirv-headers`, which does carry the CMake config. `glslc` is its own source
   package on noble, at 2023.8 — old enough that only `GL_KHR_cooperative_matrix` compiles,
   so the coopmat2 / integer-dot / bf16 / micro-float shader families are built out. Suspect
   that compiler before the device if a Vulkan prefill ever reads as slow.
2. `-DGGML_VULKAN=OFF` → `ON`: one word in the written-out flag list.
3. Runtime stage: `mesa-vulkan-drivers` **and** `libvulkan1` — the ICD and its loader. An
   image with a loader and no ICD enumerates zero devices.
4. `frank-tts-engine.container`: `AddDevice=/dev/dri` + `GroupAdd=keep-groups` — the
   DIRECTORY, never the node: on this hybrid box `renderD128` is the NVIDIA card and
   `renderD129` the amdgpu iGPU (`readlink -f /sys/class/drm/renderD129/device/driver` →
   `amdgpu`). On this host the udev rule already makes the nodes world-writable, so the
   group is belt-and-braces, not the gate.
5. Engine argv: `--device Vulkan0`, reached through `QWENTTS_DEVICE` in
   `scripts/frank-tts-engine.env`. **An empty value passes no flag at all** — that is the
   CPU rollback, and it is why the entrypoint's default is an `-v` test rather than `:=`,
   which also fires on a set-but-empty value and would silently undo the rollback.

Gate 1 recipe, already run and **PASSED**: `mesa-vulkan-drivers` + `vulkan-tools` in a
throwaway container off the same base, `--device /dev/dri`, then `vulkaninfo --text` and read
the `VkPhysicalDeviceMemoryProperties` heap sizes — no model, no load, no compute. The probe
trap: `vulkaninfo --batch` is not a flag in Ubuntu 24.04's `vulkan-tools` — it prints the help
page and exits 1, which reads like a driver failure. Every package involved has a candidate on
the host's apt (`vulkan-tools` 1.4.341, `libvulkan-dev` 1.4.341, `glslang-dev`/`glslang-tools`
16.2.0); the build stage runs on the digest-pinned Ubuntu 24.04 base, whose versions are older
but present. Script: `/home/dank/spikes/npu-tts/vk_probe.sh` on Dankclaw.

Constraints that do not relax for a co-tenant: `/dev/dri` passthrough needs the ComfyUI
pattern (`AddDevice` + `GroupAdd=keep-groups` + the world-writable udev rule) because
systemd user-unit spawns carry no supplementary groups; the 64 GiB swap at `/swap.img`
exists to protect co-tenants, and paging a KV pool is worse than the OOM it prevents;
`pp_dpm_fclk` reads `auto` (unheld), which only becomes a hazard if the NPU ever joins.

## 6. Re-visit condition

Retry the NPU when **a Linux loader claims the graph**: an `onnxruntime-genai-ryzenai`
build whose RyzenAI adapter takes the NuTTS-Air Full-Fusion graph on `RyzenAI-npu5`, or that
model appearing in a runtime's own supported catalog with a Linux runner. Then the
prerequisites are already mapped and priced: the gated NeuCodec encoder for cloned voices,
the `ref_text` transcripts, the 4096-token prompt budget, and 50 codes/s as the bar.