serverAppConfigs/report/tts-npu-spike-verdict-2026-10-19.md

# TTS on the Strix Halo NPU — spike verdict (2026-10-19)

Question: **how can the fleet's TTS engine run on hardware rather than CPU?** Answer in
one line: not on the NPU — the loader, not the silicon, refuses it; the door that exists is
the gfx1151 iGPU behind a build flag we already own.

Spike evidence lives in `report/npu-tts-runtime-landscape-2026-10-19.md` (which runtimes can
reach XDNA2 at all) and `report/qwen3-tts-payload-portability-2026-10-19.md` (what the
payload is and where it can be moved). This page owns only the **measured spike result** and
the **model-side contract** neither slice has. Throwaway work ran in containers under
`/home/dank/spikes/npu-tts` on Dankclaw; nothing in the fleet was changed — `frank-tts` and
`frank-tts-engine` still serve `narrator1|2|3` on CPU.

Legend: **[host]** live census 2026-10-19 · **[spike]** measured in the spike containers ·
**[vendor]** primary vendor source, with the version it holds for.

## 1. Verdict

**NO-GO on the NPU, and the wall is a loader.** The XDNA2 device is healthy by AMD's own
instruments; the runtime that owns this model family opens the device and then declines the
graph, silently. One run loaded successfully; byte-identical runs never reproduced it, and a
path that cannot be reproduced is not a foundation. **[spike]**

**Go on the iGPU, and the lever is in this repo.**
`dankclaw/voiceservice/Containerfile.engine` builds `qwentts.cpp` (commit `5916bf7`) with
`-DGGML_VULKAN=OFF` written out deliberately beside CUDA/HIP/METAL/SYCL/RPC — the running
container carries `libggml-base.so.0.27.0` + `libggml-cpu.so.0.27.0` + `tts-server` and
nothing else. So: `-DGGML_VULKAN=ON` + `--device Vulkan0`, plus `/dev/dri` passthrough in
the engine quadlet (the ComfyUI pattern: `AddDevice` + `GroupAdd=keep-groups` + a
world-writable udev rule). The target shape is **co-tenancy** — the engine resident on the
gfx1151 *beside* `frank-dragoon`, not a handover — so §5 carries the census and the two
gates that decide it: the heap RADV offers an APU tenant, and dragoon's TTFT with a
synthesis in flight. Unproven, stated: whether qwentts.cpp's codec ops have Vulkan kernels
for every op. **[host]** **[spike]**

Why the iGPU is worth doing at all, given CPU already works: the deployed payload needs
only **12.5 codes/s** (§2), so this is latency headroom and freed cores, not reviving a
dead service. The independent Strix Halo measurement in the portability slice ("use Vulkan
exclusively"; NPU decode under 1.5 t/s for an 8B) points the same way.

## 2. The two arms do not share a real-time bar

|Arm|Talker|Codec|Codes per second of audio|Bar|
|---|---|---|---|---|
|**Deployed** (`frank-tts-engine`)|`qwen-talker-1.7b-base-Q4_K_M.gguf`|`qwen-tokenizer-12hz-Q8_0.gguf`|**12.5**|~13 tok/s **[host]**|
|**Spike** (`amd/neutts-air_rai_1.8.0_npu_4K`)|NuTTS-Air, 24-layer Qwen2|NeuCodec `24000 / hop 480`|**50.00**|~50 tok/s **[spike]**|

Measured from the staged NPU artifact — the contract any future arm must satisfy:

|Fact|Value|
|---|---|
Vocab|`genai_config.json` `vocab_size` **217,652**; base BPE 151,643; specials 151,644–151,670 **[spike]**|
Speech tokens|**65,536** `<|speech_N|>` entries, ids **151,671–217,206** contiguous; the vendor template maps a codec index straight to its id, `"".join(f"<|speech_{i}|>")`. A 10,001-code assumption silently truncates the space — encoded references really do reach the 60k range. **[spike]**|
Stop token|`<|SPEECH_GENERATION_END|>` = 151,670, but the export's `model.eos_token_id` is `[151645, 151643]` (the *chat turn* stop) and `search.eos_token_ids` is unset — pass 151,670 explicitly or speech ends after one token **[spike]**|
Input ceiling|4096 tokens **total**, input + output **[vendor]**|
Frame rate|50.00 codes/s confirmed across three references (49.96 / 49.99 / 50.00) **[spike]**|
Reference encode|`librosa.load(sr=16000, mono=True)` → `[1,1,T]` float → `codec.encode_code(audio_or_path=…)` (vendor `NeuTTS.encode_reference`); CPU RTF 0.28–0.38 on 8 threads, so codes are a cacheable one-time cost, never a per-request charge. `neucodec` also needs `torchtune`; install both `--no-deps --target <dir>` so a second torch never shadows the venv's. **[spike]**|
Pre-encoded voices|`/spike/logs/codes-narrator{1,2,3}.npy` — 994 / 1,249 / 1,250 codes for 19.90 / 24.98 / 25.00 s **[spike]**|

**Gap that any arm inherits:** the voice packs carry no transcript. `manifest.json` holds
provenance and crop geometry (`crop.source_speech_run` 98.739–118.636 s of a 22,569 s
audiobook) and no text, and Dankclaw has no ASR at all — `whisper`, `whisper.cpp`,
`faster-whisper` all absent. NuTTS conditioning needs `ref_text`; do not invent it. **[host]**

## 3. Exoneration chain — why the blame is not the hardware

Every row is an instrument, not an inference.

|Probe|Result|
|---|---|
`xrt-smi examine`|`[0000:c6:00.1] RyzenAI-npu5 aie2p 6x8`, XRT 2.25.37 (vendor extract), NPU fw 1.1.2.65, `amdxdna 7.0.0-38-generic` **[spike]**|
`xrt-smi validate`|**PASSED**, including `throughput` **[spike]**|
`pyxrt.device(0)` in-container|constructs — XRT enumerates the device **[spike]**|
`strace -f -e trace=openat` on the failing load|`genai_config.json` read → `model.onnx` opened → `libxrt_driver_xdna.so.2` loaded → `/dev/accel/accel0` opened `O_RDWR` → GEM ioctls succeed → **EP declines, before touching `dd_metastate_*` or `cache/txn_bins.zip`** **[spike]**|
`dmesg`, `fuser /dev/accel/accel0`|silent since boot, no holder — the decline never reaches the kernel **[spike]**|
Hardlinked mirror dirs of the model tree (`rundir*`)|retired, no change — the "metastate shadowing" theory is dead **[spike]**|
Every logging lever in that build: `GENAI_ORT_VERBOSE_LOGGING`, `GENAI_LOG_ORT_LIB`, `set_log_options`, `set_log_callback(callable)`, `RYZENAI_EP_TRACING`, `RYZENAI_LLM_PROFILE_INIT`|**zero lines** — the decline path is silent **[spike]**|
Vendor `quicktest`|`Session successfully initialized` — proves the XRT/driver/device path, **not** NPU residency: that build falls back to CPU silently. Prove residency with `/sys/class/accel/accel0/device/mem_usage` during generation. **[spike] [vendor]**|

No NPU token rate was ever measured, so none is quoted here. The single successful load
(13:17, `[load] 0.84s`, finite logits, XRT banner at *generate* time) has not been
reproduced by the byte-identical command since, with the model files pristine by inode and
mtime. **[spike]**

## 4. Host gates the spike proved (so they are not re-litigated)

- **Rootless containers cannot read `/dev/accel/accel0`** (`root:render` 0660): the host
  gid never enters the userns map. `--group-add keep-groups`, `--userns=keep-id`,
  `--userns=keep-id:gid=990`, `--group-add render`, `--group-add gid:990` — all fail.
  Doors: rootful podman (separate storage), or a world-writable udev rule, which the fleet
  already does for `kfd`/`renderD*`. **[spike]**
- **Never `dpkg -i` the vendor XRT deb** — it installs a DKMS `amdxdna` over the distro
  driver. `dpkg-deb -x` the debs and bind-mount the tree at `/opt/xilinx/xrt`; the plugin
  deb is what carries `libxrt_driver_xdna.so`. **[spike]**
- In-tree `amdxdna 0.7.0-38` + `iommu=pt` is enough for context creation; a missing device
  node is *not* the modern failure mode. **[host]**
- `xrt-smi` inside a container needs `libboost-filesystem1.83.0` +
  `libboost-program-options1.83.0` (**hyphenated** Ubuntu names, after `apt update`);
  boost is needed by `xrt-smi` only, never by the OGA/EP path. **[spike]**
- Headroom is tight and it is host RAM, not a separate pool — see §5 for tonight's census.
  A multi-GB NPU partition plus an AIE compile workspace does not fit beside
  `frank-dragoon`. Benign-looking `Failed to alloc dev bo memory, ret -28` is that
  pressure. **[host]**

## 5. Target shape: the TTS engine shares the gfx1151 with `frank-dragoon`

The deployment Frank wants is **both resident at once**, not a handover. Measured 2026-10-19
(read-only, nothing loaded or unloaded):

|Measure|Value|Reading|
|---|---|---|
iGPU GTT in use|**18,523 MiB of 116,733 MiB**|≈96 GiB of aperture free; this is the real budget (fleet law) **[host]**|
DRM "VRAM" carve-out|2,048 MiB total, 174 MiB used|what KFD/DRM calls dedicated — **not** the budget **[host]**|
Host RAM|121 GiB total, 17 GiB available, 79 GiB buff/cache|**GTT pages are system pages**: an iGPU allocation is host RAM consumed, so the two tenants share one pool **[host]**|
TTS engine today|3.84 GiB RSS, CPU|its device-resident replacement lands in that same pool **[host]**|

Two gates, in this order, before any deploy:

1. **Heap gate (unmeasured — the number that decides everything else).** What heap
   RADV offers a tenant on this APU is not known here: `mesa-vulkan-drivers 26.0.3` and
   `radeon_icd.json` are installed, `vulkan-tools` is not. Enumerate
   `VkPhysicalDeviceMemoryProperties` inside the container that will host the engine —
   **before** loading a model. If the device-local heap is the 2 GiB carve-out rather than
   the aperture, weights + KV + compute buffers fall back allocation-by-allocation and the
   iGPU win is gone. **[inferred]** — the fallback behavior is the risk, and the missing
   number is the gate.
2. **Contention gate.** `frank-dragoon` is the endpoint the operator session runs on, so the
   honest measurement with both loaded is dragoon's own TTFT and decode with a synthesis in
   flight — one request each, no ladder. Expect compute-unit sharing rather than memory
   pressure: the aperture has room, the shaders do not.

**Change surface** — the whole edit, so nobody re-derives it:

1. `Containerfile.engine` build stage: add `libvulkan-dev glslang-dev`. It installs only
   `ca-certificates cmake g++ git make` today, and ggml's Vulkan backend compiles its
   shaders at build time — the flag alone fails at configure.
2. `-DGGML_VULKAN=OFF` → `ON` in the same cmake block; the flag list is written out
   explicitly, so this is a one-word diff.
3. Runtime stage: add `mesa-vulkan-drivers`. Without a Vulkan **ICD inside the image** the
   device node buys nothing — there is no `radeon_icd.json` in the container today.
4. `frank-tts-engine.container`: `AddDevice=/dev/dri/renderD128` + `GroupAdd=keep-groups`.
5. Engine argv: `--device Vulkan0` — qwentts.cpp's tools take `--device <name>` at the
   pinned commit.

Gate 1 is one command: `vulkan-tools` in a throwaway container off the same base with
`/dev/dri` passed through, then `vulkaninfo` and read the `VkPhysicalDeviceMemoryProperties`
heap sizes — no model, no load, no compute. Every package involved has a candidate on the
host's apt (`vulkan-tools` 1.4.341, `libvulkan-dev` 1.4.341, `glslang-dev`/`glslang-tools`
16.2.0); the build stage runs on the digest-pinned Ubuntu 24.04 base, whose versions are
older but present.

Constraints that do not relax for a co-tenant: `/dev/dri` passthrough needs the ComfyUI
pattern (`AddDevice` + `GroupAdd=keep-groups` + the world-writable udev rule) because
systemd user-unit spawns carry no supplementary groups; the 64 GiB swap at `/swap.img`
exists to protect co-tenants, and paging a KV pool is worse than the OOM it prevents;
`pp_dpm_fclk` reads `auto` (unheld), which only becomes a hazard if the NPU ever joins.

## 6. Re-visit condition

Retry the NPU when **a Linux loader claims the graph**: an `onnxruntime-genai-ryzenai`
build whose RyzenAI adapter takes the NuTTS-Air Full-Fusion graph on `RyzenAI-npu5`, or that
model appearing in a runtime's own supported catalog with a Linux runner. Then the
prerequisites are already mapped and priced: the gated NeuCodec encoder for cloned voices,
the `ref_text` transcripts, the 4096-token prompt budget, and 50 codes/s as the bar.