From 1e52e45fae6dfd806bbb8fa939918d4b970c87c2 Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 13:56:10 -0700 Subject: [PATCH] docs(frank-tts): record the NPU spike verdict and the iGPU co-tenancy plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The overnight spike settled whether frank-tts can run on the Strix Halo NPU. It cannot, and the wall is the loader, not the silicon: AMD's own Linux runtime opens /dev/accel/accel0, succeeds at the GEM ioctls, then declines the NuTTS-Air graph without emitting a single log line — and the one run that did load has never reproduced. The accelerator that does exist is the gfx1151 iGPU behind -DGGML_VULKAN=ON, a flag our own Containerfile.engine writes OFF, and Frank set the target shape to co-tenancy: the engine resident beside frank-dragoon, not a handover. - report/tts-npu-spike-verdict-2026-10-19.md owns the measured result: the exoneration chain, the two arms' real-time bars (12.5 codes/s deployed vs 50.00 for NuTTS-Air), the 65,536-entry speech-token census (ids 151,671-217,206), the host gates (rootless cannot read accel0; dpkg-deb the vendor XRT, never dpkg -i), the iGPU census (18,523 of 116,733 MiB GTT in use, 17 GiB host RAM available, GTT pages are host pages), the two gates that decide co-tenancy (RADV heap for an APU tenant; dragoon's TTFT with a synthesis in flight), and the five-file change surface for the trial. - report/npu-tts-runtime-landscape-2026-10-19.md and report/qwen3-tts-payload-portability-2026-10-19.md are the research slices; both now point at the verdict page instead of ending in loose ends. - The vault page gains "Where the engine computes", so an operator doing a TTS task learns the NPU is a closed door without re-walking it. No fleet change: frank-tts / frank-tts-engine still serve narrator1-3 on CPU. tests: ./tests/run.sh 450/450 green. --- .../npu-tts-runtime-landscape-2026-10-19.md | 319 +++++++++++++ ...wen3-tts-payload-portability-2026-10-19.md | 438 ++++++++++++++++++ report/tts-npu-spike-verdict-2026-10-19.md | 164 +++++++ serverAppConfigs-docs/Dankclaw/Frank TTS.md | 15 + 4 files changed, 936 insertions(+) create mode 100644 report/npu-tts-runtime-landscape-2026-10-19.md create mode 100644 report/qwen3-tts-payload-portability-2026-10-19.md create mode 100644 report/tts-npu-spike-verdict-2026-10-19.md diff --git a/report/npu-tts-runtime-landscape-2026-10-19.md b/report/npu-tts-runtime-landscape-2026-10-19.md new file mode 100644 index 0000000..80eb84e --- /dev/null +++ b/report/npu-tts-runtime-landscape-2026-10-19.md @@ -0,0 +1,319 @@ +# XDNA2 NPU inference runtimes on Linux (Strix Halo) — can any of them serve TTS? + +Research slice, 2026-10-19. Worker: `NpuRuntimeLandscape`. Research only, no code. +Question: every way to execute model inference on an AMD XDNA2 NPU on Linux as of +2026-10, and whether any of them can serve a speech-synthesis model. + +> **Status, same night — the spike that followed this slice settled it.** NO-GO on the +> NPU (the loader declines the graph, not the silicon); the actionable door is the gfx1151 +> iGPU behind `-DGGML_VULKAN=ON`. Measured contract, exoneration chain and host gates: +> `tts-npu-spike-verdict-2026-10-19.md`. That page supersedes §8–§9 below. + +Convention: vendor claims carry URL + the version/date they hold for. +**[host]** = Dankclaw state verified on the live machine 2026-10-19 (Main), not a +vendor claim. `[INFERENCE]` = my deduction from the cited artifacts. + +## 0. Answers to the three questions Main ranked + +**(1) A runtime that already runs TTS on XDNA2 today: it exists at AMD, not on our +host.** Ryzen AI Software **1.8.0** (docs `latest` == 1.8.0; released 2026-07-23) +lists `neutts-air` and `neutts-nano` as model type **TTS**, variant +*full-optimized* (NPU-only) — ; +release note "New TTS model support: neutts-air, neutts-nano" — +. Pre-compiled artifacts are +published: `amd/neutts-air_rai_1.8.0_npu_4K` (arch `qwen2`, lastModified +2026-07-23, Apache-2.0, "AWQ / Group 128 / Asymmetric / BFP16 activations / UINT4 +Weights") and `amd/neutts-nano_rai_1.8.0_npu_4K` (arch `llama`). +**Neither artifact is a TTS engine.** Both file lists are LM-only (`model.onnx`, +`genai_config.json`, BPE tokenizer, AIE-ML compiler metastate: `dd_metastate_*` +`.ctrlpkt/.super/.state`, `cache/txn_bins.zip`) with **no codec, no vocoder, no +reference-audio encoder**. [INFERENCE from HF file lists] + +**(2) FastFlowLM: no TTS, and we cannot add one** — see §2. Halogen: no TTS entry +**[host]**. So "something installed already serves TTS on the NPU" is FALSE. + +**(3) Our own ONNX graph compiled for the NPU on Linux:** possible for the plain +ONNX flows (they compile on Linux), **impossible for a new LLM/OGA model** — model +generation is Windows-only in 1.8.0 — and gated by an operator table where BF16 +lacks Softmax/LayerNorm/ReLU/Gelu (see §4). + +## 1. Landscape: every way to reach the XDNA2 array on Linux + +| # | Path | License | Strix Halo + Linux, 2026-10 | Speech synthesis | +|---|------|---------|------------------------------|------------------| +| 1 | Ryzen AI SW → ONNX Runtime + Vitis AI / Ryzen AI EP; compile INT8 (`X2`) or BF16 (`VAIML`) ONNX at session creation | AMD proprietary (tarball) | supported: our device ID/rev is in AMD's STX table; compile+run works on Linux (§3, §4) | user-authored ONNX only; no TTS recipe/demo | +| 2 | Same EP driven by **OGA** (`onnxruntime-genai`), "full fusion" NPU-only LLM flow; runner = AMD `model_benchmark` / OGA C++ API | AMD + MS OGA | Linux: **vendor-precompiled models only**, no hybrid flow | **neutts-air / neutts-nano = TTS** (LM half) | +| 3 | **FastFlowLM** `flm` | MIT + free proprietary kernels | Strix Halo "Supported"; Ubuntu 24.04/25.10/**26.04**, Arch | none (ASR input only) | +| 4 | **IRON / mlir-aie** + Peano → hand-written AIE-ML design → XRT/`amdxdna` | Apache-2.0 + LLVM exception | explicitly documented "AMD Ryzen™ AI on Linux"; same driver/XRT stack | anything, hand-written; nothing ships TTS | +| 5 | Vendor-curated AIE-ML graph sets (halogen `HALOGEN_NPU_MODELS`, FLM packs) | vendor | present **[host]** | none **[host]** | +| 6 | Windows-only: DirectML, Ryzen AI Windows EP (`ryzenai-llm`), Studio Effects, `whispercpp` NPU | — | not usable | (per Lemonade matrix: Windows only) | +| 7 | ggml/llama.cpp/**qwentts.cpp**, whisper.cpp, sherpa-onnx, Piper | — | no XDNA backend in ggml/ORT-mainline | CPU / iGPU only | +| 8 | IREE, TVM, ONNX Runtime mainline EPs (OpenVINO = Intel NPU, MIGraphX/ROCm = AMD GPU) | — | no XDNA2 target; AMD's NPU EP is only "Xilinx Vitis-AI (preview)" in the ORT EP list and ships inside AMD's own build | n/a | + +Sources: §2–§6 below. + +## 2. FastFlowLM (`/usr/bin/flm` 1.0.1 **[host]**) — catalog + HTTP surface + +* Repo moved into AMD's ROCm org **2026-08-11** (`github.com/ROCm/FastFlowLM`, + v1.0.0). "Supports all Ryzen™ AI Series chips with XDNA2 NPUs (Strix, Strix Halo, + Kraken, and Gorgon Point)." + (README fetched 2026-10-19). +* Linux support landed 2026-03; distros: Ubuntu 24.04, 25.10, **26.04**, Arch, + generic. Needs NPU firmware ≥ 1.1.0.0, kernel **7.0+** with `amdxdna` (or + `amdxdna-dkms`), XRT stack (`ppa:lemonade-team/stable` → `libxrt-npu2`, + `amdxdna-dkms`); `flm validate` reads the DRM path (`/dev/accel/accel0`), `flm + run` goes through **XRT** and fails with `No such device with index '0'` unless + `xrt-smi examine` lists the NPU; `memlock` must be raised. + (page dated 2026-03-05, fetched + 2026-10-19). Static build bundles **XRT v2.21.75** — the same XRT series as the + distro package on the host **[host]**. +* HTTP surface: `flm serve ` → OpenAI-compatible REST, **port 52625**, no + auth. Maintained endpoints: `v1/models`, `v1/chat/completions`, + `v1/audio/transcriptions`, `v1/embeddings`. **There is no + `/v1/audio/speech`.** + +* Catalog 2026-10-19 (): LLaMA, DeepSeek, + Qwen, Gemma, MedGemma, TranslateGemma, gpt-oss, LiquidAI/LFM, Phi, Nanbeige, + **Whisper — card states "Type: Speech-to-Text (ASR)"** + (), EmbeddingGemma, SmolVLA, + Hy-MT2, MiniCPM. No speech-synthesis entry. +* Engines are **prebuilt, closed, downloaded from Hugging Face per architecture** + ("Internet access to HuggingFace is required to download the optimized model + kernels"). The only in-tree engine build is Gemma-4E + (`cmake --preset linux-default -DFLM_BUILD_GEMMA4E=ON`). + → **A new TTS architecture cannot be added to `flm` by us.** + +## 3. AMD Ryzen AI Software 1.8.0 on Linux — install source, device support + +* **Install channel (important correction to the pypi.amd.com evidence).** The pip + index is Windows-only for the EP runtime (`ryzenai_dynamic_dispatch-1.7.0-… + win_amd64.whl`, `onnxruntime_vitisai-1.23.2-…win_amd64.whl`; `flexmlrt` has a + single **linux_aarch64** wheel) — but AMD's Linux stack is not on pip. It ships + as a tarball that is, as of 2026-10-19, **publicly reachable without an AMD + account**: `https://download.amd.com/opendownload/RyzenAI/1.8.0/ryzen_ai-1.8.0.tgz` + → HTTP 206 on a range probe, `Content-Range: bytes 0-8/**7819506855**` + (**≈7.8 GiB**, magic `1f 8b 08` = gzip). (The docs link it behind the + EULA form `account.amd.com/...ryzenai-eula-public-xef.html`; the CDN path + serves it directly. Verified by me with curl 2026-10-19. I did not finish + downloading 7.8 GiB, so the tarball's *contents* list is unverified — + `[INFERENCE]` only where noted.) +* Driver/XRT package (public, 90,016,137 bytes, `application/zip`): + `https://download.amd.com/opendownload/RyzenAI/Driver/RAI_1.8_Linux_NPU_XRT.zip` + contains `xrt_202620.2.25.37_24.04-amd64-{base,base-dev,npu}.deb` + + `xrt_plugin.2.25.260102.56.release_24.04-amd64-amdxdna.deb` (verified by + download + unpack). **The plugin deb ships firmware for + `/usr/lib/firmware/amdnpu/17f0_11/npu.dev.sbin`** — exactly the directory our + NPU loads **[host]** — plus `1502_00`, `17f0_10`, `17f1_10`, `17f1_13`, + `libvxdna.so`, `libxrt_driver_xdna.so.2.25.260102.56`, and the DKMS driver + source tarball. **No xclbin/PDI and no compiler binary in this package**: it is + driver + XRT shim + firmware only. +* **Our device is officially STX.** AMD's APU table + (): + `0x1022:0x17F0 rev 0x11 → STX` (0x00/0x10 also STX, 0x20 = KRK, `0x1502` rev 0 + = PHX/HPT). Release notes: "Strix and **Strix Halo** as STX", and the STX/KRK + column enables CNN INT8, CNN BF16, NLP BF16, LLM (OGA). +* Vendor driver source (same deb, `share/amdxdna/amdxdna.tar.gz`, source dated + 2026-06-29) maps `0x17f0 rev 0x11 → dev_npu5_info`, `.fw_path = + "amdnpu/17f0_11/"`, `default_vbnv = "RyzenAI-npu5"`, and the revision→name table + has `{ AIE2_DEV_REVISION_HALO, "NPU Strix Halo" }`; `xrt-smi examine` therefore + reports the device as **NPU Strix Halo**. +* Linux install recipe (1.8.0 docs + ): Ubuntu 24.04 packages, + **Python 3.12** (`apt install python3.12 python3.12-venv`), + `libboost-filesystem1.74.0`, dkms; then `install_ryzen_ai.sh -a yes -p `; + `LD_LIBRARY_PATH=…:${RYZEN_AI_INSTALLATION_PATH}/onnxruntime/lib/`; + `quicktest.py` "picks up a simple CNN model, **compiles it** and runs on AMD's + Neural Processing Unit" — i.e. **compilation runs on Linux** for the non-LLM + flows. "Linux currently supports NPU only flow" (no iGPU EP). + Host friction: our box is Ubuntu 26.04 + **Python 3.14** **[host]**, so the venv + must be built with a 3.12 interpreter and the packages are 24.04-built + `[INFERENCE]`. +* LLM flow on Linux + (): prequantized + ready-to-run models from HF, `deployment/lib`, + `RYZENAI_EP_PATH=$PWD/deployment/lib/libonnxruntime_providers_ryzenai.so`, + run via the `model_benchmark` binary; **"Linux does not support Hybrid flow"**; + **"Model generation is not supported in current release"** (repeated in + relnotes 1.8 Known Issues: "Models generated on Windows are compatible with + Linux"); needs `memlock unlimited`. + Sample throughput in that doc: **49.1 tokens/s decode**, 753 tokens/s prefill + (Phi-3.5-mini, 4K model). +* Third-party confirmation of the OS split: Lemonade's backend matrix + (, + fetched 2026-10-19) — `flm`/npu/XDNA2 = **Windows, Linux**; + `ryzenai-llm`/npu = **Windows only**; `whispercpp`/npu (STT) = **Windows only**; + text-to-speech engines `kokoro` and `acestep` = metal / cuda / vulkan / rocm, + **no npu device at all**. Lemonade is the "Server Interface" layer of AMD's own + Ryzen AI stack (docs index), so this matrix is the vendor-sanctioned view. + +## 4. Can we compile our own graph on Linux? (the no-recipe path) + +Mechanics from (1.8.0): +ONNX only, opset 17 recommended; the EP compiles on first `InferenceSession` and +caches (VitisAI EP cache, or ORT **EP context cache** — `ep.context_enable`, and +"precompile the EP context model using Python and deploy it using a C++ program" +is explicitly supported); EP auto-partitions unsupported nodes to CPU; +BF16 = `target VAIML` (`preferred_data_storage: unvectorized` for transformers), +BF16 **STX/KRK-or-newer only**; INT8 = `target=X2` with XINT8/A8W8/A16W8 and +"general asymmetric quantization … to enable third-party quantized models"; +**deployment EP cannot compile BF16 on the fly** — ship a pre-compiled model +(); +cache dirs are not portable across EP or NPU-driver versions. + +Operator coverage is the hard gate +(, "operators supported in +Ryzen AI 1.8.0"; "Y" = broad coverage, some configurations may still fail): + +* Good for TTS front-ends/vocoders: `ConvTranspose` **Y in BF16/A16W8/A8W8** + (absent only from XINT8), `Conv`, `Resize`, `Upsample`, `Pad`, `LSTM` (BF16), + `CumSum` (BF16/XINT8), `STFT` (XINT8 only). +* **Blockers for a decoder-style transformer in the plain flows:** in **BF16** the + table has **no Softmax, no LayerNormalization, no Relu/Gelu**; LayerNorm is + XINT8-only and Softmax is A16W8/A8W8/XINT8-only. So attention either lives in + the **LLM/OGA flow** (whose op set is exactly + `SimplifiedLayerNormalization`, `SkipSimplifiedLayerNormalization`, `MatMulNBits` + (W4ABF16/W4ABFP16), `RotaryEmbedding`, **`GroupQueryAttention`**, `Sigmoid`, + `Mul`, `Add`) or must be decomposed into a quantized INT8 flow. +* Missing entirely: `GRU`, `Trilu`, `OneHot`, `EyeLike`, `Multinomial`, + `NonMaxSuppression`, `CastLike`, `MelWeightMatrix`/`MelFilterGroup`, + `RMSNormalization` → graphs that emit them must be rewritten before they + partition onto the NPU. [INFERENCE from absence in the table.] +* Quantization is a separate tool, not in the driver package: **AMD Quark** + (`quark.torch` / `quarkonnx`), which documents `ConvTranspose2d`, `conv1d`, + softmax/LayerNorm/BFP quantization (relnotes 1.3/1.2/1.1/0.9) — Quark is *not* + on pypi.org under the obvious names **[host probe]** and ships from AMD's + gated/pip-private channel. +* **New LLM-shaped model (our Qwen-Talker analogue): Windows-only today.** + OGA model generation is not available on Linux in 1.8.0 (§3). So a *new* + speech-LM cannot be self-served onto the NPU on Linux without AMD (or a + Windows box) in the loop. +* Practical shape cost: the neutts artifacts carry **one compiled variant per + padded sequence length** (128/256/512/768/1024/1536/2048/2304/3072/4096 for + prefill; 256…4096 for token-phase attention mask) — i.e. per-shape compile + artifacts are the norm for the LLM flow. [INFERENCE from the HF file list.] + +## 5. Does the AMD TTS artifact actually produce speech? (cloned-voice survival) + +NeuTTS is **LM + separate neural codec** ("Simple LM + codec architecture"), +clone from ~3 s of reference audio, and the codec is **NeuCodec (50 Hz, single +codebook)**: `neuphonic/neucodec` (PyTorch, encode+decode), +`neucodec-onnx-decoder{,-int8}` (**decode only**), "Decoder-only (ONNX) codecs +require **pre-encoded references** — these ship with the repo for all bundled +voices." Phoneme input; 2048-token context ≈ 30 s of audio; NeuTTS-Air ~360 M +active / 552 M with embeddings; upstream benchmarks are **CPU-only llama.cpp** +(119 tok/s prefill/decode mix on a Ryzen 9 HX 370) and explicitly exclude the +codec. +(fetched 2026-10-19); `neuphonic/neutts-air` HF metadata (lastModified +2026-08-26, `qwen2`, GGUF + safetensors). + +Consequences for the fleet spike: + +1. AMD's NPU artifact = LM only; **waveform decode and reference encoding are CPU + work either way**. Our `reference.wav` voices survive only if we run the + NeuCodec **encoder** (PyTorch, per the codec table there is no ONNX encoder) + once per voice and then feed pre-encoded codes. That is a one-time offline + step per voice; after that the compiled graph takes codes/prompt prefix as + ordinary inputs, so nothing about the compiled graph forbids a caller-supplied + speaker prefix. `[INFERENCE]` +2. Real-time math: 50 Hz codec means **≥50 tokens/s decode** for 1× RT. AMD's own + Linux sample output for a 4K Phi-3.5-mini is **49.1 tok/s** — so budget for + roughly RT-1× on one 4K "full fusion" graph, before the codec's CPU cost. + `[INFERENCE — different model; must be measured.]` +3. Host RAM: the same doc reports **peak working set 3.54 GB** for that 4K model. + NeuTTS-Air is ~552 M params at W4A-BFP16, so expect a smaller footprint than + 3.5 GB `[INFERENCE]`; still non-trivial against 14 GiB available **[host]**. +4. AMD ships **no TTS demo, runner, or server** on any OS: the examples page has + only CNN/BF16/DistilBERT/super-resolution/OGA-LLM/VLM/RAG + (); the RyzenAI-SW repo + tree (669 paths, `main`, fetched 2026-10-19) contains speech demos **only under + `Demos/ASR/{Whisper,Parakeet-TDT}` and `Transformer-examples/ASR/Whisper-AI`** + — no TTS directory. Serving TTS therefore means writing our own OGA host + (or driving `model_benchmark`-style C++ code) plus phonemizer + NeuCodec glue. + +## 6. Two graphs on one NPU, and the iGPU hazard + +* **Concurrency is a first-class driver feature.** The vendor `amdxdna` source + (in `xrt_plugin.2.25.260102.56`, 2026-06-29) sets `.hwctx_limit = 16` for + npu4/npu5 — i.e. **16 hardware contexts** on our device — and allocates each + context a set of AIE columns with QoS `opc = col_opc (4096) × num_col` + (`aie2_ctx.c`, `aie2_solver.h`), with `frame_boundary_preempt` / + `fill_preempt_slot` in `aie2_pci.h` (firmware preemption at frame boundaries). + So halogen's armed models and a TTS graph can hold separate contexts + simultaneously; they contend for columns and OPC share, they are not mutually + exclusive. +* AMD's user-visible history agrees: 1.4 "Support for up to **16 hardware + contexts** on Ryzen AI 300 series"; 1.5 "Application concurrency: improves the + resource distribution across applications"; 1.3 "Idle detection and suspension + of contexts", "Rebalance feature for AIE hardware resource optimization"; + 1.0 known issue "Support for multiple applications is limited to up to eight" + (all in ). +* **Package conflict, the real risk** [INFERENCE]: the vendor stack wants AMD's + XRT 2.25.37 + `xrt_plugin…amdxdna` deb **[host: distro `libxrt-npu2 + 1:2.21.75+dfsg-4` installed]**, and firmware must match the loaded `amdxdna.ko` + — the vendor README warns that a stale/mismatched `npu.sbin` causes + `ERT_CMD_ABORT`/mailbox timeouts, and that a distro **in-tree** `amdxdna` older + than the XRT SHIM makes `GET_ARRAY`/`QUERY_TELEMETRY` return + `EOPNOTSUPP`/`EINVAL` ( README, + fetched 2026-10-19). Installing AMD's debs over the distro XRT changes the ABI + under FastFlowLM/halogen, so the spike must decide the XRT version **once**, for + the whole host. +* **iGPU / fabric clock:** no vendor document supports a fabric-clock pin, and no + vendor document addresses NPU+iGPU concurrency hangs. AMD's Strix Halo + optimization doc (ROCm docs, dated 2026-02-20, and its successor "AMD RDNA3.5 + system optimization", ROCm 10.1.0) covers GPUVM/GART/GTT, the TTM `pages_limit` + knob, `amd-ttm`, BIOS VRAM sizing, and minimum kernel versions + (, + ). + The `pp_dpm_fclk` level-7 pin and the GPU+NPU hang observation stay + fleet-observed **[host]**, i.e. `[INFERENCE]` territory — no citation exists. + +## 7. Paths I checked and ruled out (so nobody re-walks them) + +* **Windows-only doors**: DirectML; Ryzen AI Windows EP; Lemonade's + `ryzenai-llm` and `whispercpp`-NPU engines (matrix above); AMD's OGA model + generation. Not usable on Dankclaw. +* **Mainline ONNX Runtime** has no AMD-NPU EP: its EP summary lists AMD ROCm and + AMD MIGraphX (GPU) and "Xilinx Vitis-AI (*preview*)" only + (, 2026-10-19); the working + EP is AMD's own build (`libonnxruntime_providers_ryzenai.so` in the tarball's + deployment dir, per llm_linux). +* **ggml / llama.cpp / qwentts.cpp**: no XDNA backend → the current engine + (`tts-server` on GGUF, CPU-only build **[host]**) cannot be "turned on" for the + NPU; migration means a different runtime, not a flag. +* **`flm`**: catalog/engines are closed per-architecture (§2) — the cheapest door + by install cost, and it is not a door for TTS. +* **IRON / mlir-aie** is the only open toolchain that documents running on the + Ryzen AI NPU from Linux ("A close-to-metal Python API for programming AMD Ryzen™ + AI NPUs, built on an open-source MLIR-based compiler toolchain"; "Getting + Started for AMD Ryzen™ AI on Linux", Ubuntu 24.04/24.10, XRT from + `ppa:amd-team/xrt`: `libxrt2 libxrt-npu2 … amdxdna-dkms`, `render` group + required) — , + fetched 2026-10-19. This is how FLM's kernels were made ("Low-level kernels + optimized using IRON + AIE-MLIR", FLM README) — i.e. real, but it is + hand-written per-op tile programming, not a model compiler: writing a TTS model + here is a multi-month engineering project, not a migration. +* **Embedded** Ryzen AI P100 support in the same Linux doc is irrelevant to STX. + +## 8. Recommendation implied by the above (for Main's spike) + +Spike the AMD path as decided, with these constraints planned in: Python 3.12 venv ++ 24.04-built wheels on Ubuntu 26.04; one host-wide XRT decision (AMD 2.25.37 vs +distro 2.21.75) with firmware/driver pairing checked via `xrt-smi examine` → +"NPU Strix Halo"; 7.8 GiB tarball staged off the public CDN URL (§3); pre-encode +each `reference.wav` with the **PyTorch** NeuCodec encoder because the ONNX codec +is decoder-only; keep phonemizer + codec on CPU; measure decode tokens/s against +the 50 Hz budget and peak RSS against the ~14 GiB headroom; expect to write the +serving shell yourself (AMD ships `model_benchmark`, no TTS demo). + +## 9. Loose ends (not verified; I stopped on Main's stand-down) + +* Contents of `ryzen_ai-1.8.0.tgz` (7.8 GiB) — whether the Linux tarball contains + the AIE-ML compiler (`vaiml`, `aiecompiler_client`) for the *BF16/INT8* flows + and which Python wheels it carries. My statement that non-LLM compilation works + on Linux rests on AMD's own Linux doc ("quicktest.py … compiles it and runs on + … NPU"), not on the package contents. Download was killed mid-flight. +* Whether AMD's `neutts` artifacts run correctly under the Linux `model_benchmark` + runner (the TTS rows are listed without OS qualifiers; no Linux TTS run is + documented anywhere in 1.8.0 docs). +* Exact AIE column count/total `ncols` exposed for Strix Halo (driver reads it + from AIE metadata: `ndev->total_col = min(XRS_MAX_COL, ndev->aie.metadata.cols)`). diff --git a/report/qwen3-tts-payload-portability-2026-10-19.md b/report/qwen3-tts-payload-portability-2026-10-19.md new file mode 100644 index 0000000..8e004aa --- /dev/null +++ b/report/qwen3-tts-payload-portability-2026-10-19.md @@ -0,0 +1,438 @@ +# Qwen3-TTS payload + qwentts.cpp runtime: architecture and portability + +Research deliverable (no code). Slice: *what the fleet's TTS payload actually is, and how +portable it is off the CPU ggml path.* Companion slice: `NpuRuntimeLandscape` (what NPU +runtimes exist on Dankclaw). + +Evidence date: 2026-10-19. Everything below is grounded in: + +- `github.com/ServeurpersoCom/qwentts.cpp` @ `5916bf7c79dbb4e66fc059cdc507a79a23a28bb4` + (2026-10-16, upstream HEAD on 2026-10-19) and its ggml submodule + `ServeurpersoCom/ggml` @ `cc5df49a6b878db2b68b9a7b45976e279594627d` (2026-10-16). +- The two GGUF files as they sit on Dankclaw (`/home/dank/models/Qwen3-TTS-GGUF/`), read + byte-for-byte from the live host. +- Upstream Qwen3-TTS model cards / release material. + +Status: COMPLETE for the assigned slice. ANSWER C was confirmed on the live host the same +night — the running `frank-tts-engine` container carries `libggml-base.so.0.27.0` + +`libggml-cpu.so.0.27.0` and nothing else, and our own +`dankclaw/voiceservice/Containerfile.engine` writes `-DGGML_VULKAN=OFF` out explicitly, so +the iGPU change is a flag in a file the fleet owns. Verdict, measured contract and the NPU +exoneration chain: `tts-npu-spike-verdict-2026-10-19.md`. + +--- + +## 1. What the payload is + +Two files, both read byte-for-byte off the live host (`/home/dank/models/Qwen3-TTS-GGUF/`, +owned `dank:dank`, mtimes 2026-10-18 18:31), both from +`https://huggingface.co/Serveurperso/Qwen3-TTS-GGUF` (community converter, not Qwen): + +| file | bytes | GGUF | tensors | arch string | file_type | +|---|---|---|---|---|---| +| `qwen-talker-1.7b-base-Q4_K_M.gguf` | 1,219,245,248 | v3, kv=64 | 480 | `qwen3-tts` | `Q4_K_M` | +| `qwen-tokenizer-12hz-Q8_0.gguf` | 291,150,624 | v3, kv=51 | 398 | `qwen3-tts-tokenizer` | `Q8_0` | + +`general.name` of the talker file is `Qwen3-TTS-12Hz-1.7B-base`; the header sizes add up +to the file sizes (sum of tensor bytes + header + alignment padding), so nothing in these +files is unaccounted for. + +### 1.1 Talker file (480 tensors) + +Three models in one container, per `docs/ARCHITECTURE.md` §GGUF layout +(qwentts.cpp @ pinned commit) and confirmed by the tensor census: + +| prefix | tensors | bytes | what it is | +|---|---|---|---| +| `talker.*` | 316 | 1,061.9 MB | 28-layer Qwen3 decoder (hidden 2048, FFN 6144, 16 Q heads / 8 KV heads, head_dim 128, RoPE θ=1e6, RMS ε=1e-6, ctx 32768), text stream + codec stream summed, `codec_head` → 3072 codebook-0 logits | +| `code_pred.*` | 88 | 114.8 MB | 5-layer Qwen3 MTP head (hidden 1024, FFN 3072, 15 acoustic codebooks × 2048), own KV cache | +| `spk_enc.*` | 76 | 36.6 MB | ECAPA-TDNN speaker encoder (x-vector, base checkpoints only; mel → conv0 → 3 SE-Res2Net → MFA → ASP → 2048-d vector) | + +Dtype split (exact tensor byte sums): `q4_K` 250 tensors / 864.1 MB, `q6_K` 17 / 312.0 MB, +`f32` 176 / 25.8 MB, `f16` 37 / 11.4 MB. + +Which tensors carry which quant is the load-bearing portability fact: + +- `q6_K` (17): `talker.text_embd.weight [2048, 151936]` (255.3 MB), `talker.codec_embd.weight + [2048, 3072]`, and the 15 `code_pred.codec_embd..weight [2048, 2048]` embedding tables. + These are the llama.cpp `Q4_K_M` "embedding/output rows go to Q6_K" rule, applied to + embedding tables rather than to an `lm_head`. +- `q4_K` (250): every attention/FFN projection of the talker and of the code predictor. + Distinct matmul shapes: `[2048,2048]`×73, `[2048,1024]`×62, `[2048,6144]`×56, + `[6144,2048]`×28, `[1024,2048]`×20, `[1024,3072]`×10, `[1024,1024]`×10, `[3072,1024]`×5, + `[2048,3072]`×2, `[2048,151936]`×1. Every one is 2-D and ≥ 1024 on both axes — good + GEMM shapes, nothing exotic. +- `f32` (176) / `f16` (37): all of `spk_enc` (conv weights F16 because K=1/3/5 rows are not + block-alignable, per the converter's `should_quantize` policy in `tools/quantize.cpp` + + `quantize.sh`; every bias/norm/LayerScale vector F32), plus the talker/predictor norms and + the `text_proj` biases. + +Metadata the runtime actually reads (talker file): `qwen3-tts.num_code_groups=16`, +`talker.vocab_size=3072`, `text_vocab_size=151936`, `text_hidden_size=2048`, +`position_id_per_seconds=13`, `rope.mrope_section=[24,20,20]`, `mrope_interleaved=1`, +the 7 codec special ids (pad 2148 / bos 2149 / eos 2150 / think 2154 / nothink 2155 / +think_bos 2156 / think_eos 2157), `language_names` (10: chinese, english, german, italian, +portuguese, spanish, japanese, korean, french, russian) with their codec ids, the text-side +special ids (im_start 151644, im_end 151645, tts_pad 151671, tts_bos 151672, tts_eos 151673), +the whole `generation_config.json` (do_sample, top_k 50, top_p 1.0, temperature 0.9, +repetition_penalty 1.05, `subtalker_*` mirrors, max_new_tokens 8192), and an embedded +**Qwen2 BPE tokenizer** (`tokenizer.ggml.model=gpt2`, 151,676 tokens, 151,291 merges, +eos 151643) — that BPE block is 5.95 MB of the header, i.e. 49 % of the header is vocabulary. + +Two live-file details that contradict the upstream doc text (worth knowing when reading +`docs/ARCHITECTURE.md`: the doc example block is stale against the shipped file): +`mrope_interleaved` is **1** (doc shows `false`), and `spk_enc.sample_rate` is **24000** +(doc says 16000). Doc also says "404 tensors on the 1.7B voice_design build" — the base +build shipped here has 480 because it carries `spk_enc.*`. + +### 1.2 Tokenizer/codec file (398 tensors) + +One neural audio codec, 12.5 frames/s over 24 kHz mono, hop 1920, 16 codebooks × 2048 +entries (1 semantic + 15 acoustic). Dtype split: `f16` 50 / 143.6 MB, `q8_0` 110 / 76.9 MB, +`f32` 238 / 70.7 MB. `tok_dec.*` 253 tensors / 201.3 MB, `tok_enc.*` 145 / 89.9 MB. + +Composition (metadata + tensor census, both files agree): + +- **Encoder** (used only to make voice references, not on the synthesis hot path): SEANet + conv stack (init k=7, 4 stages with ratios 8/6/5/4 → cumulative 960×, 64→512 ch, final + k=3), an 8-layer transformer (hidden 512, 8 heads, head_dim 64, FFN 2048, RoPE 1e4, + LayerNorm+bias, GELU, LayerScale), a downsample conv k=4 s=2 (25 → 12.5 Hz), then the RVQ + argmin loop. Metadata records `encoder.num_quantizers=32` with + `encoder_valid_num_quantizers=16` — 16 codebooks are used, 16 are dead weight carried from + the upstream config. +- **Decoder** (the hot path: every generated frame): 16 codebook lookups + per-split + `output_proj` (256→512), `conv_pre`, an 8-layer transformer (hidden 512, 16/16 heads, + head_dim 64, FFN 1024, **sliding window 72**, RMSNorm, SwiGLU, LayerScale, input_proj + 1024→512 / output_proj 512→1024), a ConvNeXt upampler (`upsampling_ratios=[2,2]`, causal + transposed convs + depthwise k=7), then a DAC-style decoder `dec.0..3` + (`upsample_rates=[8,5,4,3]`, channels 1536→768→384→192→96, `decoder_dim=1536`) whose blocks + are SnakeBeta → causal transposed conv (k = 2·stride) → 3 residual units (dilations 1/3/9) + → snake_post → conv_post k=7 → 96→1. +- All 32 codebook tensors are **F32, 67.1 MB total** (16 encode + 16 decode, `[256, 2048]`), + deliberately unquantized: the converter's `should_quantize` keeps RVQ codebooks and their + wrapping projections at F32 because nearest-neighbour lookup is sensitive to per-row + quantisation noise (`docs/ARCHITECTURE.md` "Quantisation policy"; mirrored in + `quantize.sh`). 49 of the codec's 55 3-D conv tensors are **F16** with non-alignable K + (7, 3, 2, 1, 16, 10, …) — they cannot be block-quantized at all. +- **No flow matching, no diffusion, no vocoder-with-ODE.** It is a pure RVQ codec + (Mimi/SoundStream/Mimi-style: SEANet + transformer + DAC decoder). That matters for the + NPU question: every component is convolution, attention, or embedding lookup. + +### 1.3 Where the payload came from, and what the official weights look like + +Official upstream (`huggingface.co/api/models`, retrieved 2026-10-19): + +| repo | last modified | license | weights | +|---|---|---|---| +| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | 2026-01-23 | apache-2.0 | `model.safetensors` BF16, 1,928,677,440 params, plus an embedded `speech_tokenizer/model.safetensors` | +| `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | 2026-01-29 | apache-2.0 | same layout, named-speaker tables | +| `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | 2026-01-29 | apache-2.0 | 1.7B only | +| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` / `-0.6B-CustomVoice` | 2026-01-29 | apache-2.0 | narrow variant (hidden 1024) | +| `Qwen/Qwen3-TTS-Tokenizer-12Hz` | 2026-01-29 | apache-2.0 | codec alone, F32, 170,557,441 params | + +So the GGUF is **not** the only public form: the safetensors are the canonical form +(Apache-2.0, no separate community license risk), and the GGUF is a lossy re-quantization of +them produced by the same author who wrote the runtime (`Serveurperso`), via +`checkpoints.sh` → `convert.py` (F32 GGUF) → `quantize.sh` (BF16/Q8_0/Q4_K_M). +Sizes to remember for any re-export: the talker container is 1.93 B params in BF16 +(that count includes the embedded codec), the codec alone is 170.6 M params in F32. + +**Portability consequence #1 — the GGUF is a dead end off-ggml.** Q4_K/Q6_K are ggml +block formats (256-element super-blocks, F16 sub-block scales, 144 B / 210 B per block). No +ONNX graph, no Vitis-AI/AIE-ML compiler, and no vendor NPU runtime consumes them; AMD's own +quantizer works from ONNX (`ryzenai-quantize`, QDQ/QOP power-of-two — Ryzen AI Software +1.8.0 release notes, https://ryzenai.docs.amd.com/en/latest/relnotes.html). Moving the +payload to a non-ggml stack therefore means starting from the safetensors again, not +converting the GGUF. + +**Portability consequence #2 — the quantization split is semantic, not cosmetic.** The +converter keeps RVQ codebooks, `input_proj`/`output_proj` around them, and every 1-D and +non-alignable-conv tensor out of the k-quants precisely because quantizing them breaks voice +fidelity. Any re-quantization path (GPTQ/AWQ/QDQ-INT8/INT4 for an NPU compiler) inherits that +constraint: the talker backbone is requantizable, the codec's codebook lookups and SnakeBeta +convs are not, at least not without re-measuring voice quality. + +## 2. What the runtime is + +`qwentts.cpp` @ `5916bf7c` (2026-10-16, MIT; upstream model Apache-2.0) is a C++17 ggml +application — 43 files under `src/` + `tools/`, three binaries (`qwen-tts`, `qwen-codec`, +`tts-server`) over one public plain-C99 ABI (`src/qwen.h`, `qt_*`, `QT_ABI_VERSION` +range-checked; `tests/abi-c.c` is compiled `-std=c99 -Wall -Werror -pedantic` on every +build). It is not a llama.cpp derivative: it links ggml directly and builds its own graphs. + +Five graph families, and their shapes are the whole portability story: + +| graph | batch/time axis | notes | +|---|---|---| +| talker prefill | T = prompt length (hundreds–thousands) | summed text+codec embedding streams, 28-layer GQA decoder, KV cache | +| talker decode | **T = 1 per audio frame** | 12.5 frames/s → 80 ms of audio per step; `codec_head` → 3072 logits | +| code predictor (MTP) | **T = 1, 16 steps per frame** (2 prefill + 14 decode, cache reset every frame) | 5-layer decoder, 15 heads × 2048 | +| speaker encoder | one pass per voice registration | ECAPA-TDNN with mel fused into the graph | +| codec decode | offline: T = 300-frame chunks; streaming: **T = 1 frame** with persistent conv/KV-ring state | the streaming graph is built once with fixed topology so the CUDA graph cache stays in pure replay | + +Per generated frame the host therefore runs ~1 + 16 + 1 tiny graph computes for 80 ms of +audio, and `tts-server` serializes synthesis FIFO (`--max-batch` defaults to 1, and the +fleet runs it at 1). This is the archetypal latency-bound, bandwidth-bound, +batch-1 workload; the only big-M matmuls in the whole system are the prefill and the +chunked (buffered) codec decode. + +Op contract actually issued to ggml (`src/`, counted): `mul_mat` 77, `cont` 70, `add` 57, +`mul` 40, `rms_norm` 22, `clamp` 15, `rope_ext` 12, `get_rows` 12, `set_rows` 8, +`scale_bias` 6, `swiglu_split` 5, `sqr` 5, `mean` 5, `elu` 4, `col2im_1d` 4, `soft_max` 3, +`conv_1d` 3, `im2col` 2, `cumsum` 2, `sin`, `sigmoid`, `tanh`, `div`, `argsort`, `transpose`, +`permute`, `reshape/view`, plus the CPU-side Philox multinomial sampler. + +**No custom ggml op is required today.** The repo doc still says the fork "provides the two +custom ops the codec needs: `GGML_OP_SNAKE` and `GGML_OP_COL2IM_1D`"; that is stale at the +pinned commit. Verified 2026-10-19: + +- the submodule `ServeurpersoCom/ggml` @ `cc5df49` is upstream `ggml-org/ggml` master + (`a05c992`, 2026-10-15) **+2 commits, 0 behind** (GitHub compare API), and + `include/ggml.h` is byte-identical to upstream master (`diff` empty); +- the 2 commits are `193e339` (2026-06-19) "cpu: add fused snake activation + (mul -> sin -> sqr -> mul -> add)" — a **CPU subgraph fusion**, not an op (the runtime + builds snake with `ggml_sin`/`ggml_sqr`/`ggml_mul`/`ggml_add` in `src/dac-decoder-v2.h:222`), + and `cc5df49` "sycl: fix get_rows on reordered quantized weights"; +- `ggml_col2im_1d` is an **upstream ggml API** (same declaration, line 2097, in both trees); + the Vulkan backend fuses the snake chain through ggml's generic subgraph fuser + (`ggml-vulkan.cpp:14548 ggml_vk_can_fuse_snake`, `vulkan-shaders/snake.comp`). + +Consequence: any stock ggml build (upstream, or llama.cpp's vendored ggml) can execute this +payload; only the CPU snake fusion is fork-specific, and it is a performance nicety. + +Backend/device support: every tool takes `--device ` (`CUDA0`, `Vulkan0`, `MTL0`, +`CPU…`, default best available), resolved through the ggml backend registry; the deployed +container turns CUDA/HIP/Vulkan/Metal/SYCL/RPC/BLAS all OFF and compiles the native-CPU +backend with OpenMP, with CPU threads = `hardware_concurrency()/2` (`src/backend.h`). +The registry in this ggml knows CPU, CUDA, Metal, SYCL, Vulkan, WebGPU, ZDNN, OpenCL, +**Hexagon**, BLAS, RPC, VirtGPU, **CANN**, ZenDNN, **OpenVINO**, ET — and nothing AMD-NPU +(no `xdna`/`aie`/`vitis`/`ryzen` string anywhere in the ggml tree). + +Voice handling (what the fleet depends on): the registry is **process RAM only**; a restart +empties it, which is why the unit has a boot warmer. `POST /v1/audio/voices` accepts either +`{name, ref_text, wav_b64}` (server extracts) or `{name, ref_text, spk_b64, rvq_b64}` +(pre-extracted latents verbatim); `ref_text` present ⇒ ICL clone mode, absent ⇒ x-vector-only +mode. `.spk` = the ECAPA x-vector; `.rvq` = 16 codes/frame packed 11 bits LSB-first +(`src/rvq-file.h`). `wav` and `pcm` `response_format` are different audio at the same seed +(buffered chunked decode vs stateful streaming decode) — the fleet already encodes that rule. + +Parity expectations for any port: the upstream harness matches the PyTorch reference per +stage (embed cos 0.999988, all 28 hidden taps ≥ 0.999893, logits 0.999824) but explicitly +reports free-run divergence (`CodesFull exact 4.96 %`, `Audio cos 0.095`) and explains why — +one argmax tie at FP epsilon diverts the AR trajectory. So a migrated endpoint must be +accepted by listening + per-stage cosine, never by waveform diff against today's audio. + +--- + +## 3. The three answers Main asked for + +### ANSWER A — Qwen3-TTS GGUF → ONNX: no, and it does not matter + +There is **no GGUF→ONNX path** and there will not be a useful one: GGUF is a weight +container for ggml, its Q4_K/Q6_K block layouts have no ONNX consumer, and nothing in the +GGUF carries the *graph* (the talker/predictor/codec structure lives in C++ code, only the +op-level structure exists in ggml at runtime). But the GGUF is not the payload's only form — +official Apache-2.0 safetensors are upstream, and **four independent ports of the full +pipeline already exist**: + +| port | form | what it proves | +|---|---|---| +| `xkos/Qwen3-TTS-12Hz-1.7B-ONNX` (HF, updated 2026-03-03, apache-2.0) | FP16+FP32 ONNX, fp16/voice_clone + fp16/voice_design splits: `talker_decode.onnx`, `code_predictor.onnx`, `code_predictor_kv.onnx`, `code_predictor_embed_g0..15.onnx`, `text_embedding.onnx`, `codec_embedding.onnx`, shared `speaker_encoder.onnx`, `speech_tokenizer_encoder.onnx`, `speech_tokenizer_decoder.onnx`, `model_cache.npz` | every component of this architecture exports to ONNX, codec decoder included | +| `zhangluoyang/qwen3-tts-onnxruntime` | export tooling + Python inference for Base/CustomVoice/VoiceDesign | the export is reproducible from the official checkpoints | +| `Echo9Zulu/Qwen3-TTS-12Hz-CustomVoice-1.7B-INT8-OpenVINO`, `aurora2035/Qwen3-TTS-12Hz-0.6B-Base-OpenVINO-INT8` (2026-03) | OpenVINO IR, **INT8**: `talker`, `code_predictor`, `codec_embedding`, `text_model`, `openvino_speaker_encoder_model`, `speech_tokenizer/speech_decoder` | the talker and predictor INT8-quantize for an NPU-targeting stack | +| `litert-community/Qwen3-TTS-12Hz-0.6B-Base` (2026-10-17, apache-2.0) | `qwen3_tts_int8.litertlm` / `_int4.litertlm` (LiteRT-LM `OmniEngine`), legacy split graphs `talker_int4.tflite` (prefill_32/prefill_128/decode signatures, KV 1024, blockwise-32 OCTAV INT4), `mtp_folded_int8.tflite` (16 inner steps × 5 layers folded into one graph, in-graph argmax+gather, GPTQ int8, ~5× faster), `codec_partA` (transformer, fp32) + `codec_partB` (conv upsampler, XNNPACK FORCE_FP16, ~2.5×, "ASR-identical") | an NPU/mobile compiler accepts the whole thing — at the price of graph splitting, static prefill buckets, and folding the per-frame inner loop | + +What the ports had to concede, and what any port inherits: + +- the generation structure does not fit existing engine loops: LiteRT-LM's Engine "does not + support this generation structure yet", so their first shape was **three graphs + a + host-side loop**, and the MTP step was "invoked 17× per frame" until they folded it; +- per-step dispatch cost dominates: even on a Snapdragon 8 Elite Gen 5 they keep the MTP + loop, text projection, embedding gathers and the codec on CPU "avoiding 16 synchronous + CPU↔GPU round-trips per 80 ms frame", putting only the 28-layer talker on the GPU; +- RTF on that hardware is **2.57–2.97 (Galaxy S26) and 5.65–8.48 (Pixel 10 Pro)** for the + 0.6 B model, i.e. 2.6–8× slower than real time. Dankclaw's current CPU engine measures + **warm RTF 0.55–0.67 with TTFA 94–112 ms** on the 1.7 B model (fleet trial, 2026-10-18). + An NPU port is not "faster by default" for this workload; it has to beat that. +- quantize the LM (int8/int4: proven), keep the codec's codebook lookups and SnakeBeta convs + in fp16/fp32 (the LiteRT split does exactly this). + +So: the answer to "can the fleet's payload be exported" is *yes — from the safetensors, not +from the GGUF*, and the work is a pipeline rewrite (multi-graph host loop, per-frame fold, +static prefill buckets), not a conversion. That is the same conclusion NeuTTS-Air's +precompiled-graph route encodes: on AMD's Linux stack the graph is vendor-curated, so the +fleet chooses the model that AMD already compiled rather than compiling Qwen3-TTS. + +### ANSWER B — is there any ggml backend for an AMD NPU? Not upstream; three out-of-tree, none for TTS + +Checked against the trees today (2026-10-19), because this was the crux: + +- **Nothing in-tree.** `ggml-org/llama.cpp` master (4,237 paths, GitHub tree API) contains no + `xdna`/`aie`/`vitis`/`ryzen` file or directory; the ggml backend set is CPU, BLAS, CUDA, + HIP, Metal, Vulkan, SYCL, OpenCL, WebGPU, MUSA, RPC, CANN (Ascend), **Hexagon** (Qualcomm), + OpenVINO (Intel CPU/GPU/NPU), ZenDNN, ZDNN, VirtGPU, ET. The deployed fork's registry + matches. The feature request "Feature Request: XDNA backend" (llama.cpp issue **#21725**, + opened 2026-04-10) is still **open**, 34 comments; the older #1499 was closed stale + 2025-04-07. So Main's premise is confirmed: no ggml XDNA backend exists to flip a flag on. +- **`Cyronius/ggml-xdna`** — a real ggml backend (`XDNA0`) loaded into *stock* llama.cpp via + `GGML_BACKEND_PATH`, no fork. It does bfp16 (8-value blocks sharing an exponent) GEMMs from + GGUF weights, and is **prefill-only**: it takes only matmuls in prompt chunks ≥ 1024 tokens + (`GGML_XDNA_MIN_BATCH` default 1024, `GGML_XDNA_MIN_MFLOP` 256), attention and every decode + token stay on the Vulkan device; pinned to llama.cpp release b10944; tested on Strix Point + (HX 370) only — Strix Halo "should pass, but nobody has run it on one yet"; Linux support + new and untested on hardware by the author; needs `libxrt-npu2`, `/dev/accel/accel0`, + unlimited `memlock`. Best measured prefill gain 1.88× on a 27 B model; models under + 2,048-wide **lose** (0.77× at width 1,536). +- **`BrandedTamarasu-glitch/OllamaAMDNPU`** — a llama.cpp fork with `-DGGML_XDNA=ON`, + explicitly targeting **Ryzen AI MAX / XDNA2 on Linux** (`/dev/accel/accel0`, XRT 2.21.75+, + amdxdna, unlimited memlock). It implements **only `GGML_OP_MUL_MAT`**, dispatched to + mlir-aie-compiled `.xclbin`s built per fixed tile shape (e.g. `M=2048 K=2048 N=64`, + int8 in / int32 out, per-row weight scales cached at first use), with a second slot for a + different K. Its own numbers on a Ryzen AI MAX 385 with Llama-3.1-8B Q4_K_M: NPU prefill + 12–19 t/s vs **Vulkan iGPU 600–930 t/s**, NPU decode **0.65–1.40 t/s** ("Decode does not + use the NPU … M=1 per token, no xclbin covers it"), 0.6 % AIE utilization, and the stated + conclusion "**For LLaMA 8B on Strix Halo: use Vulkan exclusively**". Single-maintainer, + AI-assisted, self-measured — treat as a proof of concept, but it is the closest published + picture of what XDNA2 graph execution costs. +- Also referenced in the same thread: `Atomic-Germ/OpenFlowLM-Next` (whole-model NPU engine, + LLM-only — and slower than the iGPU in the same comparison), `GaTmaNnes/xdna2-`, + `hardWorker254/Npu-Embeddings-XDNA1` (adds Whisper ASR ops, NPU1/Linux). None is a TTS path. + +Why none of them would carry this workload even if installed: the TTS hot loop is **batch-1 +by construction** (one talker step + 16 predictor steps + one codec frame per 80 ms), which +sits under every dispatch gate those backends define (≥1024 tokens / ≥32768 M·N·K / tile M +covering only prefill), and the codec half of the pipeline is conv, col2im, gather and +elementwise ops — not `MUL_MAT` — so it would fall back to CPU op-by-op. The NPU case for +this host is therefore *not* "run Qwen3-TTS on the NPU"; it is "run a vendor-compiled AR TTS +graph on the NPU", which is exactly the NeuTTS-Air spike. + +### ANSWER C — would `qwentts.cpp` build for the gfx1151 iGPU? Yes, via **Vulkan**, not HIP + +- Upstream ships the build scripts and claims the backends: `buildvulkan.sh` ("AMD/Intel GPU + (Vulkan)"), `buildcuda.sh`, `buildsycl.sh`, `buildcpu.sh`, `buildall.sh` (all backends, + runtime dynamic loading), and README line 6 "runs on CPU, CUDA, Metal, Vulkan". CMake knobs + `GGML_VULKAN` / `GGML_HIP` / `GGML_SYCL` are the same ones the fleet's Containerfile turns + OFF, so the engine rebuild is a flag change: `-DGGML_VULKAN=ON` plus + `--device Vulkan0` on the same command line the unit runs today. +- The codec's exotic ops are covered by the Vulkan backend, which is the thing that decides + whether a GPU build is coherent: `vulkan-shaders/col2im_1d.comp`, `snake.comp`, + `swiglu*.comp`, `cumsum*.comp` exist and `supports_op` handles `GGML_OP_COL2IM_1D` / + `CONV_TRANSPOSE_1D` / `CUMSUM`. By contrast `COL2IM_1D` appears in **no** HIP, OpenCL, + WebGPU, Hexagon or CANN source — a **HIP** (ROCm) build would shard the ConvNeXt/DAC + transposed convs back to the CPU per op. On an AMD iGPU the portable path is RADV/Vulkan. +- `--device` also accepts a multi-device split (`-dev`-style naming is ggml's), and the + runtime's documented FP16 guard (`--clamp-fp16`) exists for sub-Ampere CUDA accumulation; + nothing in the tree is NVIDIA-specific. +- Unverified by me: any vendor statement pairing this fork with gfx1151. Supporting evidence + that the hardware/driver combination is viable: this very host already runs a Vulkan + endpoint on the gfx1151 iGPU (fleet ground truth, 2026-10-19), and Mesa's Vulkan + implementation is officially supported on Radeon Software for Linux + (AMD release notes RN-AMDGPU-UNIFIED-LINUX-25.20.3). A gfx1151 Vulkan build of this fork is + a build-and-measure spike, not an unsupported port. +- The real constraint is the one already in the fleet docs: the iGPU belongs to another + endpoint, and GPU+NPU concurrency has hung Strix Halo machines. An iGPU TTS build competes + with the text endpoints for the same silicon; the NPU route's only genuine advantage is + that it does not (the XDNA backend authors make the same argument: "Use NPU when the iGPU + is busy … no GPU contention"). + +--- + +## 4. What any replacement must reproduce, and what the voices cost + +Endpoint contract in production today (from the live unit/env, not the doc): +`tts-server --model qwen-talker-1.7b-base-Q4_K_M.gguf --codec qwen-tokenizer-12hz-Q8_0.gguf +--alias frank-tts-engine --host 0.0.0.0 --port 8880 --lang auto --max-batch 1`, no published +port (the façade on :7772 dials the private bridge name), `GET /health`, `GET /v1/models`, +`GET|POST /v1/audio/voices`, `DELETE /v1/audio/voices/{name}`, `POST /v1/audio/speech` with +`response_format` wav|pcm (different audio at equal seed) and optional `seed`, +`max_new_tokens`, `temperature`, `top_k`, `top_p`, `repetition_penalty`, `subtalker_*`. +Voice registry is RAM-only, so boot warming is mandatory, and `--alias` keeps the quantization +choice out of every consumer's config. + +Live voice census on Dankclaw (2026-10-19, `/home/dank/voices/`): three packs +(`narrator1..3`), each carrying `reference.wav` (24 kHz mono crop), `reference16k.wav`, +`manifest.json` (keys: name, kind, generated_utc, pipeline, source, forbidden_region_guard, +silence_scan, crop, loudnorm, audio, acceptance, acceptance_passed) and ffmpeg analysis +artifacts. **No `.spk` and no `.rvq` exist on the live host** — the repo's +`build-voice-pack.sh` treats them as required-but-they-are-not-there, so today the engine +re-extracts the x-vector (and the ICL codes when a transcript is supplied) from the WAV at +registration time. + +That is the portable part, and it is the same shape AMD's NeuTTS flow wants +(`encode_reference(reference.wav)` + a reference transcript): the fleet's voice identity is +*the WAV plus its transcript*, processed at request/registration time by the model's own +speaker encoder + codec — **never a trained per-voice artifact**. Qwen3-TTS cloning is +inference-time conditioning (`generate_voice_clone(ref_audio=…, ref_text=…, x_vector_only_mode=…)`; +`create_voice_clone_prompt` exists purely to avoid recomputing prompt features). So any +replacement that consumes a WAV (+transcript) keeps all three narrators alive with no +re-derivation, and if it wants precomputed latents instead, the LiteRT precedent shows what +that looks like: a 4 KB x-vector `.npy` per voice, extracted offline +(`extract_speaker_embedding.py`) — for the 1.7 B base checkpoint that vector is 2048-dim +(`qwen3-tts.spk_enc.embedding_length = 2048` in the shipped GGUF; the LiteRT 0.6 B bundle +ships a 1024-dim one). + +One trap for a mixed-runtime future: `.spk`/`.rvq` files, if they are ever written, are +engine-specific encodings (ECAPA x-vector layout; 11-bit-packed RVQ stream) and are **not** +interchangeable between runtimes. Keep `reference.wav` as the pack's source of truth and +regenerate anything derived per engine — which is what the NeuTTS re-clone plan does anyway. + +## 5. Sources + +Runtime/payload (all read locally at pinned versions, 2026-10-19): + +- `github.com/ServeurpersoCom/qwentts.cpp` @ `5916bf7c79dbb4e66fc059cdc507a79a23a28bb4` + (commit date 2026-10-16; also upstream HEAD today): `README.md`, `docs/ARCHITECTURE.md` + (860 lines), `src/*`, `tools/*`, `docs/DOCKER.md`, `buildvulkan.sh`/`buildcpu.sh`/`buildall.sh`. +- submodule `ServeurpersoCom/ggml` @ `cc5df49a6b878db2b68b9a7b45976e279594627d` (2026-10-16); + delta vs `ggml-org/ggml` master `a05c992` (2026-10-15) = ahead 2 / behind 0, per + https://github.com/ServeurpersoCom/ggml/compare/ggml-org:master...HEAD ; + `include/ggml.h` identical to upstream + https://raw.githubusercontent.com/ggml-org/ggml/master/include/ggml.h . +- Live payload bytes: `/home/dank/models/Qwen3-TTS-GGUF/qwen-talker-1.7b-base-Q4_K_M.gguf` + (1,219,245,248 B) and `qwen-tokenizer-12hz-Q8_0.gguf` (291,150,624 B), headers parsed from + the host; sizes/sha256 match `scripts/download-frank-tts-engine-qwentts.sh`. +- Live units/env/packs: `dankclaw/voiceservice/{Containerfile.engine,tts-engine-start.sh,tts-engine-warm.sh,quadlet/*}`, + `scripts/frank-tts-engine.env`, `scripts/frank-tts.env`, `/home/dank/voices/*`. + +Model provenance: + +- `https://huggingface.co/api/models/Qwen/Qwen3-TTS-12Hz-1.7B-Base` (+ CustomVoice, + VoiceDesign, 0.6B variants, `Qwen3-TTS-Tokenizer-12Hz`), apache-2.0, last-modified + 2026-01-23 / 2026-01-29; safetensors param counts 1,928,677,440 (BF16, incl. embedded + `speech_tokenizer/`) and 170,557,441 (F32 codec). +- `https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base/raw/main/README.md` (clone API: + `generate_voice_clone(ref_audio, ref_text, x_vector_only_mode)`, + `create_voice_clone_prompt`; reference stack = PyTorch + `device_map="cuda:0"` + bf16 + + flash-attn-2, Python 3.12 recommended). +- `https://github.com/QwenLM/Qwen3-TTS`. + +Portability precedents: + +- `https://huggingface.co/xkos/Qwen3-TTS-12Hz-1.7B-ONNX` (2026-03-03) · + `https://github.com/zhangluoyang/qwen3-tts-onnxruntime` · + `https://huggingface.co/Echo9Zulu/Qwen3-TTS-12Hz-CustomVoice-1.7B-INT8-OpenVINO` (2026-03-22) · + `https://huggingface.co/aurora2035/Qwen3-TTS-12Hz-0.6B-Base-OpenVINO-INT8` (2026-03-18) · + `https://huggingface.co/litert-community/Qwen3-TTS-12Hz-0.6B-Base` (2026-10-17) incl. + `legacy/README.md` (graph splits, MTP fold, codec partA/partB, RTF table). + +AMD NPU / ggml backend status: + +- `https://api.github.com/repos/ggml-org/llama.cpp/git/trees/master?recursive=1` (no + XDNA/AIE/Vitis backend, 2026-10-19) · + `https://github.com/ggml-org/llama.cpp/issues/21725` (open since 2026-04-10) · + `https://github.com/ggml-org/llama.cpp/issues/1499` (closed stale 2025-04-07). +- `https://github.com/Cyronius/ggml-xdna` (prefill-only, gates, Linux requirements, results + measured 2026-10-05/06 on HX 370) · + `https://github.com/BrandedTamarasu-glitch/OllamaAMDNPU` (XDNA2 ggml backend on Strix Halo, + phases 1–12, "use Vulkan exclusively", measured on Ryzen AI MAX 385). +- `https://github.com/amd/RyzenAI-SW` (main, 669 paths, 2026-08-18): audio content is ASR + only (`Demos/ASR/Whisper`, `Demos/ASR/Parakeet-TDT`, `Transformer-examples/ASR/Whisper-AI`); + **no TTS example anywhere in the official Ryzen AI Software tree**. +- `https://ryzenai.docs.amd.com/en/latest/relnotes.html` (Ryzen AI SW 1.8.0: ONNX-format + models, ONNX Runtime as the loader, power-of-two QDQ/QOP quantization). +- Mesa/RADV on AMD iGPUs: `https://www.amd.com/en/resources/support-articles/release-notes/RN-AMDGPU-UNIFIED-LINUX-25-20.3.html` + (Mesa Vulkan driver officially supported). + +Status: COMPLETE for the assigned slice (payload architecture, runtime architecture, +portability). `[INFERENCE]` marks: none above except the Mesa-version caveat in ANSWER C, +which is flagged there. diff --git a/report/tts-npu-spike-verdict-2026-10-19.md b/report/tts-npu-spike-verdict-2026-10-19.md new file mode 100644 index 0000000..e5e5f7d --- /dev/null +++ b/report/tts-npu-spike-verdict-2026-10-19.md @@ -0,0 +1,164 @@ +# TTS on the Strix Halo NPU — spike verdict (2026-10-19) + +Question: **how can the fleet's TTS engine run on hardware rather than CPU?** Answer in +one line: not on the NPU — the loader, not the silicon, refuses it; the door that exists is +the gfx1151 iGPU behind a build flag we already own. + +Spike evidence lives in `report/npu-tts-runtime-landscape-2026-10-19.md` (which runtimes can +reach XDNA2 at all) and `report/qwen3-tts-payload-portability-2026-10-19.md` (what the +payload is and where it can be moved). This page owns only the **measured spike result** and +the **model-side contract** neither slice has. Throwaway work ran in containers under +`/home/dank/spikes/npu-tts` on Dankclaw; nothing in the fleet was changed — `frank-tts` and +`frank-tts-engine` still serve `narrator1|2|3` on CPU. + +Legend: **[host]** live census 2026-10-19 · **[spike]** measured in the spike containers · +**[vendor]** primary vendor source, with the version it holds for. + +## 1. Verdict + +**NO-GO on the NPU, and the wall is a loader.** The XDNA2 device is healthy by AMD's own +instruments; the runtime that owns this model family opens the device and then declines the +graph, silently. One run loaded successfully; byte-identical runs never reproduced it, and a +path that cannot be reproduced is not a foundation. **[spike]** + +**Go on the iGPU, and the lever is in this repo.** +`dankclaw/voiceservice/Containerfile.engine` builds `qwentts.cpp` (commit `5916bf7`) with +`-DGGML_VULKAN=OFF` written out deliberately beside CUDA/HIP/METAL/SYCL/RPC — the running +container carries `libggml-base.so.0.27.0` + `libggml-cpu.so.0.27.0` + `tts-server` and +nothing else. So: `-DGGML_VULKAN=ON` + `--device Vulkan0`, plus `/dev/dri` passthrough in +the engine quadlet (the ComfyUI pattern: `AddDevice` + `GroupAdd=keep-groups` + a +world-writable udev rule). The target shape is **co-tenancy** — the engine resident on the +gfx1151 *beside* `frank-dragoon`, not a handover — so §5 carries the census and the two +gates that decide it: the heap RADV offers an APU tenant, and dragoon's TTFT with a +synthesis in flight. Unproven, stated: whether qwentts.cpp's codec ops have Vulkan kernels +for every op. **[host]** **[spike]** + +Why the iGPU is worth doing at all, given CPU already works: the deployed payload needs +only **12.5 codes/s** (§2), so this is latency headroom and freed cores, not reviving a +dead service. The independent Strix Halo measurement in the portability slice ("use Vulkan +exclusively"; NPU decode under 1.5 t/s for an 8B) points the same way. + +## 2. The two arms do not share a real-time bar + +|Arm|Talker|Codec|Codes per second of audio|Bar| +|---|---|---|---|---| +|**Deployed** (`frank-tts-engine`)|`qwen-talker-1.7b-base-Q4_K_M.gguf`|`qwen-tokenizer-12hz-Q8_0.gguf`|**12.5**|~13 tok/s **[host]**| +|**Spike** (`amd/neutts-air_rai_1.8.0_npu_4K`)|NuTTS-Air, 24-layer Qwen2|NeuCodec `24000 / hop 480`|**50.00**|~50 tok/s **[spike]**| + +Measured from the staged NPU artifact — the contract any future arm must satisfy: + +|Fact|Value| +|---|---| +Vocab|`genai_config.json` `vocab_size` **217,652**; base BPE 151,643; specials 151,644–151,670 **[spike]**| +Speech tokens|**65,536** `<|speech_N|>` entries, ids **151,671–217,206** contiguous; the vendor template maps a codec index straight to its id, `"".join(f"<|speech_{i}|>")`. A 10,001-code assumption silently truncates the space — encoded references really do reach the 60k range. **[spike]**| +Stop token|`<|SPEECH_GENERATION_END|>` = 151,670, but the export's `model.eos_token_id` is `[151645, 151643]` (the *chat turn* stop) and `search.eos_token_ids` is unset — pass 151,670 explicitly or speech ends after one token **[spike]**| +Input ceiling|4096 tokens **total**, input + output **[vendor]**| +Frame rate|50.00 codes/s confirmed across three references (49.96 / 49.99 / 50.00) **[spike]**| +Reference encode|`librosa.load(sr=16000, mono=True)` → `[1,1,T]` float → `codec.encode_code(audio_or_path=…)` (vendor `NeuTTS.encode_reference`); CPU RTF 0.28–0.38 on 8 threads, so codes are a cacheable one-time cost, never a per-request charge. `neucodec` also needs `torchtune`; install both `--no-deps --target ` so a second torch never shadows the venv's. **[spike]**| +Pre-encoded voices|`/spike/logs/codes-narrator{1,2,3}.npy` — 994 / 1,249 / 1,250 codes for 19.90 / 24.98 / 25.00 s **[spike]**| + +**Gap that any arm inherits:** the voice packs carry no transcript. `manifest.json` holds +provenance and crop geometry (`crop.source_speech_run` 98.739–118.636 s of a 22,569 s +audiobook) and no text, and Dankclaw has no ASR at all — `whisper`, `whisper.cpp`, +`faster-whisper` all absent. NuTTS conditioning needs `ref_text`; do not invent it. **[host]** + +## 3. Exoneration chain — why the blame is not the hardware + +Every row is an instrument, not an inference. + +|Probe|Result| +|---|---| +`xrt-smi examine`|`[0000:c6:00.1] RyzenAI-npu5 aie2p 6x8`, XRT 2.25.37 (vendor extract), NPU fw 1.1.2.65, `amdxdna 7.0.0-38-generic` **[spike]**| +`xrt-smi validate`|**PASSED**, including `throughput` **[spike]**| +`pyxrt.device(0)` in-container|constructs — XRT enumerates the device **[spike]**| +`strace -f -e trace=openat` on the failing load|`genai_config.json` read → `model.onnx` opened → `libxrt_driver_xdna.so.2` loaded → `/dev/accel/accel0` opened `O_RDWR` → GEM ioctls succeed → **EP declines, before touching `dd_metastate_*` or `cache/txn_bins.zip`** **[spike]**| +`dmesg`, `fuser /dev/accel/accel0`|silent since boot, no holder — the decline never reaches the kernel **[spike]**| +Hardlinked mirror dirs of the model tree (`rundir*`)|retired, no change — the "metastate shadowing" theory is dead **[spike]**| +Every logging lever in that build: `GENAI_ORT_VERBOSE_LOGGING`, `GENAI_LOG_ORT_LIB`, `set_log_options`, `set_log_callback(callable)`, `RYZENAI_EP_TRACING`, `RYZENAI_LLM_PROFILE_INIT`|**zero lines** — the decline path is silent **[spike]**| +Vendor `quicktest`|`Session successfully initialized` — proves the XRT/driver/device path, **not** NPU residency: that build falls back to CPU silently. Prove residency with `/sys/class/accel/accel0/device/mem_usage` during generation. **[spike] [vendor]**| + +No NPU token rate was ever measured, so none is quoted here. The single successful load +(13:17, `[load] 0.84s`, finite logits, XRT banner at *generate* time) has not been +reproduced by the byte-identical command since, with the model files pristine by inode and +mtime. **[spike]** + +## 4. Host gates the spike proved (so they are not re-litigated) + +- **Rootless containers cannot read `/dev/accel/accel0`** (`root:render` 0660): the host + gid never enters the userns map. `--group-add keep-groups`, `--userns=keep-id`, + `--userns=keep-id:gid=990`, `--group-add render`, `--group-add gid:990` — all fail. + Doors: rootful podman (separate storage), or a world-writable udev rule, which the fleet + already does for `kfd`/`renderD*`. **[spike]** +- **Never `dpkg -i` the vendor XRT deb** — it installs a DKMS `amdxdna` over the distro + driver. `dpkg-deb -x` the debs and bind-mount the tree at `/opt/xilinx/xrt`; the plugin + deb is what carries `libxrt_driver_xdna.so`. **[spike]** +- In-tree `amdxdna 0.7.0-38` + `iommu=pt` is enough for context creation; a missing device + node is *not* the modern failure mode. **[host]** +- `xrt-smi` inside a container needs `libboost-filesystem1.83.0` + + `libboost-program-options1.83.0` (**hyphenated** Ubuntu names, after `apt update`); + boost is needed by `xrt-smi` only, never by the OGA/EP path. **[spike]** +- Headroom is tight and it is host RAM, not a separate pool — see §5 for tonight's census. + A multi-GB NPU partition plus an AIE compile workspace does not fit beside + `frank-dragoon`. Benign-looking `Failed to alloc dev bo memory, ret -28` is that + pressure. **[host]** + +## 5. Target shape: the TTS engine shares the gfx1151 with `frank-dragoon` + +The deployment Frank wants is **both resident at once**, not a handover. Measured 2026-10-19 +(read-only, nothing loaded or unloaded): + +|Measure|Value|Reading| +|---|---|---| +iGPU GTT in use|**18,523 MiB of 116,733 MiB**|≈96 GiB of aperture free; this is the real budget (fleet law) **[host]**| +DRM "VRAM" carve-out|2,048 MiB total, 174 MiB used|what KFD/DRM calls dedicated — **not** the budget **[host]**| +Host RAM|121 GiB total, 17 GiB available, 79 GiB buff/cache|**GTT pages are system pages**: an iGPU allocation is host RAM consumed, so the two tenants share one pool **[host]**| +TTS engine today|3.84 GiB RSS, CPU|its device-resident replacement lands in that same pool **[host]**| + +Two gates, in this order, before any deploy: + +1. **Heap gate (unmeasured — the number that decides everything else).** What heap + RADV offers a tenant on this APU is not known here: `mesa-vulkan-drivers 26.0.3` and + `radeon_icd.json` are installed, `vulkan-tools` is not. Enumerate + `VkPhysicalDeviceMemoryProperties` inside the container that will host the engine — + **before** loading a model. If the device-local heap is the 2 GiB carve-out rather than + the aperture, weights + KV + compute buffers fall back allocation-by-allocation and the + iGPU win is gone. **[inferred]** — the fallback behavior is the risk, and the missing + number is the gate. +2. **Contention gate.** `frank-dragoon` is the endpoint the operator session runs on, so the + honest measurement with both loaded is dragoon's own TTFT and decode with a synthesis in + flight — one request each, no ladder. Expect compute-unit sharing rather than memory + pressure: the aperture has room, the shaders do not. + +**Change surface** — the whole edit, so nobody re-derives it: + +1. `Containerfile.engine` build stage: add `libvulkan-dev glslang-dev`. It installs only + `ca-certificates cmake g++ git make` today, and ggml's Vulkan backend compiles its + shaders at build time — the flag alone fails at configure. +2. `-DGGML_VULKAN=OFF` → `ON` in the same cmake block; the flag list is written out + explicitly, so this is a one-word diff. +3. Runtime stage: add `mesa-vulkan-drivers`. Without a Vulkan **ICD inside the image** the + device node buys nothing — there is no `radeon_icd.json` in the container today. +4. `frank-tts-engine.container`: `AddDevice=/dev/dri/renderD128` + `GroupAdd=keep-groups`. +5. Engine argv: `--device Vulkan0` — qwentts.cpp's tools take `--device ` at the + pinned commit. + +Gate 1 is one command: `vulkan-tools` in a throwaway container off the same base with +`/dev/dri` passed through, then `vulkaninfo` and read the `VkPhysicalDeviceMemoryProperties` +heap sizes — no model, no load, no compute. Every package involved has a candidate on the +host's apt (`vulkan-tools` 1.4.341, `libvulkan-dev` 1.4.341, `glslang-dev`/`glslang-tools` +16.2.0); the build stage runs on the digest-pinned Ubuntu 24.04 base, whose versions are +older but present. + +Constraints that do not relax for a co-tenant: `/dev/dri` passthrough needs the ComfyUI +pattern (`AddDevice` + `GroupAdd=keep-groups` + the world-writable udev rule) because +systemd user-unit spawns carry no supplementary groups; the 64 GiB swap at `/swap.img` +exists to protect co-tenants, and paging a KV pool is worse than the OOM it prevents; +`pp_dpm_fclk` reads `auto` (unheld), which only becomes a hazard if the NPU ever joins. + +## 6. Re-visit condition + +Retry the NPU when **a Linux loader claims the graph**: an `onnxruntime-genai-ryzenai` +build whose RyzenAI adapter takes the NuTTS-Air Full-Fusion graph on `RyzenAI-npu5`, or that +model appearing in a runtime's own supported catalog with a Linux runner. Then the +prerequisites are already mapped and priced: the gated NeuCodec encoder for cloned voices, +the `ref_text` transcripts, the 4096-token prompt budget, and 50 codes/s as the bar. diff --git a/serverAppConfigs-docs/Dankclaw/Frank TTS.md b/serverAppConfigs-docs/Dankclaw/Frank TTS.md index 4f722fa..face310 100644 --- a/serverAppConfigs-docs/Dankclaw/Frank TTS.md +++ b/serverAppConfigs-docs/Dankclaw/Frank TTS.md @@ -111,6 +111,21 @@ own `/health` answers 200 with the backend name and the voice list it resolved without dialing the engine, so "façade healthy, engine down" is a real state: it shows up as a failed synthesis, never as a down probe. +## Where the engine computes + +CPU, and only CPU is compiled in: `Containerfile.engine` builds `qwentts.cpp` with +`-DGGML_VULKAN=OFF` written out explicitly, so the container carries `libggml-cpu` and no +other backend. The realistic accelerator is the gfx1151 iGPU behind that flag — plus +`/dev/dri` passthrough, the ComfyUI pattern — on the same payload and the same pinned +commit, and the intended shape is **co-tenancy**: the engine resident on the card beside +`frank-dragoon`, not a handover. Memory is not what decides that; shader contention is, and +the census plus both gates are in `report/tts-npu-spike-verdict-2026-10-19.md` §5. +**The Strix Halo NPU is not a door at all**: XDNA2 executes only precompiled AIE +partition programs, and AMD's own Linux runtime opens `/dev/accel/accel0`, succeeds at the +ioctls, then declines this model family's graph without emitting a single log line. The +measured record and the model-side contract live in `report/` at the repo root +(`tts-npu-spike-verdict-2026-10-19.md`). + ## Files | Path | What |