Frank
ea2fc6440b
The cutover PR merged the runtime; this carries the surface that has to be true afterwards. - Engine page: the shape is now `--device Vulkan0` on the gfx1151 as a co-tenant beside frank-dragoon; the registry/thread/streaming laws gain a fourth — the device law — whose numbers stay in the verdict report, and the deploy record gets the cutover steps including the `glslc` configure gate. The CPU-era arm/rollback prose keeps the qualitative argument and drops the dead numbers per the purpose test. - Endpoint page: "Where the engine computes" now says iGPU + co-tenancy, and states the NPU verdict in one paragraph instead of a research tour. - Quadlet comment: it claimed supplementary groups are what grant render access. On this host the udev rule world-writes the render nodes and the container's `id` shows no groups at all, so keep-groups is belt-and-braces; the RSS note now matches the device shape instead of the CPU trial. - Verdict report: gate 2 (contention) recorded as measured after the cutover, and the build-stage prediction corrected — the package is `glslc`, not `libvulkan-dev glslang-dev`. - index.md regenerated (one description changed).
192 lines
14 KiB
Markdown
192 lines
14 KiB
Markdown
# TTS on the Strix Halo NPU — spike verdict (2026-10-19)
|
||
|
||
Question: **how can the fleet's TTS engine run on hardware rather than CPU?** Answer in
|
||
one line: not on the NPU — the loader, not the silicon, refuses it; the door that exists is
|
||
the gfx1151 iGPU behind a build flag we already own.
|
||
|
||
Spike evidence lives in `report/npu-tts-runtime-landscape-2026-10-19.md` (which runtimes can
|
||
reach XDNA2 at all) and `report/qwen3-tts-payload-portability-2026-10-19.md` (what the
|
||
payload is and where it can be moved). This page owns only the **measured spike result** and
|
||
the **model-side contract** neither slice has. Throwaway work ran in containers under
|
||
`/home/dank/spikes/npu-tts` on Dankclaw; nothing in the fleet was changed — `frank-tts` and
|
||
`frank-tts-engine` still serve `narrator1|2|3` on CPU.
|
||
|
||
Legend: **[host]** live census 2026-10-19 · **[spike]** measured in the spike containers ·
|
||
**[vendor]** primary vendor source, with the version it holds for.
|
||
|
||
## 1. Verdict
|
||
|
||
**NO-GO on the NPU, and the wall is a loader.** The XDNA2 device is healthy by AMD's own
|
||
instruments; the runtime that owns this model family opens the device and then declines the
|
||
graph, silently. One run loaded successfully; byte-identical runs never reproduced it, and a
|
||
path that cannot be reproduced is not a foundation. **[spike]**
|
||
|
||
**Go on the iGPU, and the lever is in this repo.**
|
||
`dankclaw/voiceservice/Containerfile.engine` builds `qwentts.cpp` (commit `5916bf7`) with
|
||
`-DGGML_VULKAN=OFF` written out deliberately beside CUDA/HIP/METAL/SYCL/RPC — the running
|
||
container carries `libggml-base.so.0.27.0` + `libggml-cpu.so.0.27.0` + `tts-server` and
|
||
nothing else. So: `-DGGML_VULKAN=ON` + `--device Vulkan0`, plus `/dev/dri` passthrough in
|
||
the engine quadlet (the ComfyUI pattern: `AddDevice` + `GroupAdd=keep-groups` + a
|
||
world-writable udev rule). The target shape is **co-tenancy** — the engine resident on the
|
||
gfx1151 *beside* `frank-dragoon`, not a handover. §5 carries the census and both gates:
|
||
the **heap gate is PASSED** — RADV's `DEVICE_LOCAL` heap is 77.33 GiB with 65.16 GiB of
|
||
budget while dragoon is loaded, so memory is not the constraint — which leaves the
|
||
contention gate (dragoon's TTFT with a synthesis in flight) and one unproven piece:
|
||
whether qwentts.cpp's codec ops have Vulkan kernels for every op. **[host]** **[spike]**
|
||
|
||
Why the iGPU is worth doing at all, given CPU already works: the deployed payload needs
|
||
only **12.5 codes/s** (§2), so this is latency headroom and freed cores, not reviving a
|
||
dead service. The independent Strix Halo measurement in the portability slice ("use Vulkan
|
||
exclusively"; NPU decode under 1.5 t/s for an 8B) points the same way.
|
||
|
||
## 2. The two arms do not share a real-time bar
|
||
|
||
|Arm|Talker|Codec|Codes per second of audio|Bar|
|
||
|---|---|---|---|---|
|
||
|**Deployed** (`frank-tts-engine`)|`qwen-talker-1.7b-base-Q4_K_M.gguf`|`qwen-tokenizer-12hz-Q8_0.gguf`|**12.5**|~13 tok/s **[host]**|
|
||
|**Spike** (`amd/neutts-air_rai_1.8.0_npu_4K`)|NuTTS-Air, 24-layer Qwen2|NeuCodec `24000 / hop 480`|**50.00**|~50 tok/s **[spike]**|
|
||
|
||
Measured from the staged NPU artifact — the contract any future arm must satisfy:
|
||
|
||
|Fact|Value|
|
||
|---|---|
|
||
Vocab|`genai_config.json` `vocab_size` **217,652**; base BPE 151,643; specials 151,644–151,670 **[spike]**|
|
||
Speech tokens|**65,536** `<|speech_N|>` entries, ids **151,671–217,206** contiguous; the vendor template maps a codec index straight to its id, `"".join(f"<|speech_{i}|>")`. A 10,001-code assumption silently truncates the space — encoded references really do reach the 60k range. **[spike]**|
|
||
Stop token|`<|SPEECH_GENERATION_END|>` = 151,670, but the export's `model.eos_token_id` is `[151645, 151643]` (the *chat turn* stop) and `search.eos_token_ids` is unset — pass 151,670 explicitly or speech ends after one token **[spike]**|
|
||
Input ceiling|4096 tokens **total**, input + output **[vendor]**|
|
||
Frame rate|50.00 codes/s confirmed across three references (49.96 / 49.99 / 50.00) **[spike]**|
|
||
Reference encode|`librosa.load(sr=16000, mono=True)` → `[1,1,T]` float → `codec.encode_code(audio_or_path=…)` (vendor `NeuTTS.encode_reference`); CPU RTF 0.28–0.38 on 8 threads, so codes are a cacheable one-time cost, never a per-request charge. `neucodec` also needs `torchtune`; install both `--no-deps --target <dir>` so a second torch never shadows the venv's. **[spike]**|
|
||
Pre-encoded voices|`/spike/logs/codes-narrator{1,2,3}.npy` — 994 / 1,249 / 1,250 codes for 19.90 / 24.98 / 25.00 s **[spike]**|
|
||
|
||
**Gap that any arm inherits:** the voice packs carry no transcript. `manifest.json` holds
|
||
provenance and crop geometry (`crop.source_speech_run` 98.739–118.636 s of a 22,569 s
|
||
audiobook) and no text, and Dankclaw has no ASR at all — `whisper`, `whisper.cpp`,
|
||
`faster-whisper` all absent. NuTTS conditioning needs `ref_text`; do not invent it. **[host]**
|
||
|
||
## 3. Exoneration chain — why the blame is not the hardware
|
||
|
||
Every row is an instrument, not an inference.
|
||
|
||
|Probe|Result|
|
||
|---|---|
|
||
`xrt-smi examine`|`[0000:c6:00.1] RyzenAI-npu5 aie2p 6x8`, XRT 2.25.37 (vendor extract), NPU fw 1.1.2.65, `amdxdna 7.0.0-38-generic` **[spike]**|
|
||
`xrt-smi validate`|**PASSED**, including `throughput` **[spike]**|
|
||
`pyxrt.device(0)` in-container|constructs — XRT enumerates the device **[spike]**|
|
||
`strace -f -e trace=openat` on the failing load|`genai_config.json` read → `model.onnx` opened → `libxrt_driver_xdna.so.2` loaded → `/dev/accel/accel0` opened `O_RDWR` → GEM ioctls succeed → **EP declines, before touching `dd_metastate_*` or `cache/txn_bins.zip`** **[spike]**|
|
||
`dmesg`, `fuser /dev/accel/accel0`|silent since boot, no holder — the decline never reaches the kernel **[spike]**|
|
||
Hardlinked mirror dirs of the model tree (`rundir*`)|retired, no change — the "metastate shadowing" theory is dead **[spike]**|
|
||
Every logging lever in that build: `GENAI_ORT_VERBOSE_LOGGING`, `GENAI_LOG_ORT_LIB`, `set_log_options`, `set_log_callback(callable)`, `RYZENAI_EP_TRACING`, `RYZENAI_LLM_PROFILE_INIT`|**zero lines** — the decline path is silent **[spike]**|
|
||
Vendor `quicktest`|`Session successfully initialized` — proves the XRT/driver/device path, **not** NPU residency: that build falls back to CPU silently. Prove residency with `/sys/class/accel/accel0/device/mem_usage` during generation. **[spike] [vendor]**|
|
||
|
||
No NPU token rate was ever measured, so none is quoted here. The single successful load
|
||
(13:17, `[load] 0.84s`, finite logits, XRT banner at *generate* time) has not been
|
||
reproduced by the byte-identical command since, with the model files pristine by inode and
|
||
mtime. **[spike]**
|
||
|
||
## 4. Host gates the spike proved (so they are not re-litigated)
|
||
|
||
- **Rootless containers cannot read `/dev/accel/accel0`** (`root:render` 0660): the host
|
||
gid never enters the userns map. `--group-add keep-groups`, `--userns=keep-id`,
|
||
`--userns=keep-id:gid=990`, `--group-add render`, `--group-add gid:990` — all fail.
|
||
Doors: rootful podman (separate storage), or a world-writable udev rule, which the fleet
|
||
already does for `kfd`/`renderD*`. **[spike]**
|
||
- **Never `dpkg -i` the vendor XRT deb** — it installs a DKMS `amdxdna` over the distro
|
||
driver. `dpkg-deb -x` the debs and bind-mount the tree at `/opt/xilinx/xrt`; the plugin
|
||
deb is what carries `libxrt_driver_xdna.so`. **[spike]**
|
||
- In-tree `amdxdna 0.7.0-38` + `iommu=pt` is enough for context creation; a missing device
|
||
node is *not* the modern failure mode. **[host]**
|
||
- `xrt-smi` inside a container needs `libboost-filesystem1.83.0` +
|
||
`libboost-program-options1.83.0` (**hyphenated** Ubuntu names, after `apt update`);
|
||
boost is needed by `xrt-smi` only, never by the OGA/EP path. **[spike]**
|
||
- Headroom is tight and it is host RAM, not a separate pool — see §5 for tonight's census.
|
||
A multi-GB NPU partition plus an AIE compile workspace does not fit beside
|
||
`frank-dragoon`. Benign-looking `Failed to alloc dev bo memory, ret -28` is that
|
||
pressure. **[host]**
|
||
|
||
## 5. Target shape: the TTS engine shares the gfx1151 with `frank-dragoon`
|
||
|
||
The deployment Frank wants is **both resident at once**, not a handover. Measured 2026-10-19
|
||
(read-only, nothing loaded or unloaded):
|
||
|
||
|Measure|Value|Reading|
|
||
|---|---|---|
|
||
iGPU GTT in use|**18,523 MiB of 116,733 MiB**|≈96 GiB of aperture free; this is the real budget (fleet law) **[host]**|
|
||
DRM "VRAM" carve-out|2,048 MiB total, 174 MiB used|what KFD/DRM calls dedicated — **not** the budget **[host]**|
|
||
Host RAM|121 GiB total, 17 GiB available, 79 GiB buff/cache|**GTT pages are system pages**: an iGPU allocation is host RAM consumed, so the two tenants share one pool **[host]**|
|
||
TTS engine today|3.84 GiB RSS, CPU|its device-resident replacement lands in that same pool **[host]**|
|
||
|
||
**Two gates, in this order.**
|
||
|
||
1. **Heap gate — PASSED.** RADV does **not** hand a container tenant the 2 GiB DRM
|
||
carve-out. `vulkaninfo` inside the engine's own base image (`ubuntu:24.04` plus
|
||
`mesa-vulkan-drivers vulkan-tools`, `--device /dev/dri`, rootless) sees
|
||
`Radeon 8060S Graphics (RADV GFX1151)`, driver `radv`, API 1.4.318, and reports:
|
||
|
||
|Heap|size|budget with dragoon loaded|flags|
|
||
|---|---|---|---|
|
||
|0|38.67 GiB|32.58 GiB|host-visible, not device-local|
|
||
|1|**77.33 GiB**|**65.16 GiB**|`DEVICE_LOCAL`|
|
||
|
||
The heaps sum to the ~116 GiB aperture in Mesa's ⅓/⅔ APU split — the same RAM twice:
|
||
a Vulkan allocation is a GTT mapping, so a budget is a ceiling on mappings, not a pile
|
||
of free silicon memory. ggml's Vulkan backend asks for `DEVICE_LOCAL`, which is heap 1,
|
||
and the TTS engine needs roughly 4 GiB of it. **Memory stops being the question.** **[spike]**
|
||
Probe traps: `vulkaninfo --batch` is not a flag in this `vulkan-tools` (it prints the
|
||
help page and exits 1 — use `vulkaninfo --text`), and the container has **no NVIDIA
|
||
ICD**, so Vulkan device 0 is the iGPU here. Do not trust that ordering — select by
|
||
device *name*, or an image that grows an `nvidia_icd.json` silently points TTS at the
|
||
3090 slot.
|
||
2. **Contention gate — PASSED, measured 2026-10-19 after the cutover.** With a 16.9 s
|
||
synthesis occupying the device, `frank-dragoon` answered a streamed 124-token request at
|
||
**17 ms to first token and ~35 tok/s decode**, against 17 ms and ~33 tok/s for the same
|
||
request with the card idle. Compute-unit sharing showed up as nothing. One request per
|
||
arm, no ladder: these are the gate's evidence, not a throughput record, and the engine's
|
||
own shape lives in [[Frank TTS Engine]].
|
||
|
||
**Change surface — as landed** (PR #34, `6248f02`; the CPU shape stays reachable as the
|
||
rollback):
|
||
|
||
1. `Containerfile.engine` build stage: add **`libvulkan-dev glslc spirv-headers`**. The
|
||
prediction above (`libvulkan-dev glslang-dev`) is wrong on two counts and the configure
|
||
says so: ggml looks for `glslc`, which Ubuntu's `glslang-tools` does **not** ship (it
|
||
ships `glslangValidator`), and `find_package(SPIRV-Headers CONFIG REQUIRED)` needs the
|
||
distro's `spirv-headers`, which does carry the CMake config. `glslc` is its own source
|
||
package on noble, at 2023.8 — old enough that only `GL_KHR_cooperative_matrix` compiles,
|
||
so the coopmat2 / integer-dot / bf16 / micro-float shader families are built out. Suspect
|
||
that compiler before the device if a Vulkan prefill ever reads as slow.
|
||
2. `-DGGML_VULKAN=OFF` → `ON`: one word in the written-out flag list.
|
||
3. Runtime stage: `mesa-vulkan-drivers` **and** `libvulkan1` — the ICD and its loader. An
|
||
image with a loader and no ICD enumerates zero devices.
|
||
4. `frank-tts-engine.container`: `AddDevice=/dev/dri` + `GroupAdd=keep-groups` — the
|
||
DIRECTORY, never the node: on this hybrid box `renderD128` is the NVIDIA card and
|
||
`renderD129` the amdgpu iGPU (`readlink -f /sys/class/drm/renderD129/device/driver` →
|
||
`amdgpu`). On this host the udev rule already makes the nodes world-writable, so the
|
||
group is belt-and-braces, not the gate.
|
||
5. Engine argv: `--device Vulkan0`, reached through `QWENTTS_DEVICE` in
|
||
`scripts/frank-tts-engine.env`. **An empty value passes no flag at all** — that is the
|
||
CPU rollback, and it is why the entrypoint's default is an `-v` test rather than `:=`,
|
||
which also fires on a set-but-empty value and would silently undo the rollback.
|
||
|
||
Gate 1 recipe, already run and **PASSED**: `mesa-vulkan-drivers` + `vulkan-tools` in a
|
||
throwaway container off the same base, `--device /dev/dri`, then `vulkaninfo --text` and read
|
||
the `VkPhysicalDeviceMemoryProperties` heap sizes — no model, no load, no compute. The probe
|
||
trap: `vulkaninfo --batch` is not a flag in Ubuntu 24.04's `vulkan-tools` — it prints the help
|
||
page and exits 1, which reads like a driver failure. Every package involved has a candidate on
|
||
the host's apt (`vulkan-tools` 1.4.341, `libvulkan-dev` 1.4.341, `glslang-dev`/`glslang-tools`
|
||
16.2.0); the build stage runs on the digest-pinned Ubuntu 24.04 base, whose versions are older
|
||
but present. Script: `/home/dank/spikes/npu-tts/vk_probe.sh` on Dankclaw.
|
||
|
||
Constraints that do not relax for a co-tenant: `/dev/dri` passthrough needs the ComfyUI
|
||
pattern (`AddDevice` + `GroupAdd=keep-groups` + the world-writable udev rule) because
|
||
systemd user-unit spawns carry no supplementary groups; the 64 GiB swap at `/swap.img`
|
||
exists to protect co-tenants, and paging a KV pool is worse than the OOM it prevents;
|
||
`pp_dpm_fclk` reads `auto` (unheld), which only becomes a hazard if the NPU ever joins.
|
||
|
||
## 6. Re-visit condition
|
||
|
||
Retry the NPU when **a Linux loader claims the graph**: an `onnxruntime-genai-ryzenai`
|
||
build whose RyzenAI adapter takes the NuTTS-Air Full-Fusion graph on `RyzenAI-npu5`, or that
|
||
model appearing in a runtime's own supported catalog with a Linux runner. Then the
|
||
prerequisites are already mapped and priced: the gated NeuCodec encoder for cloned voices,
|
||
the `ref_text` transcripts, the 4096-token prompt budget, and 50 codes/s as the bar.
|