serverAppConfigs/report/npu-tts-runtime-landscape-2026-10-19.md

XDNA2 NPU inference runtimes on Linux (Strix Halo) — can any of them serve TTS?

Research slice, 2026-10-19. Worker: NpuRuntimeLandscape. Research only, no code. Question: every way to execute model inference on an AMD XDNA2 NPU on Linux as of 2026-10, and whether any of them can serve a speech-synthesis model.

Status, same night — the spike that followed this slice settled it. NO-GO on the NPU (the loader declines the graph, not the silicon); the actionable door is the gfx1151 iGPU behind -DGGML_VULKAN=ON. Measured contract, exoneration chain and host gates: tts-npu-spike-verdict-2026-10-19.md. That page supersedes §8–§9 below.

Convention: vendor claims carry URL + the version/date they hold for. [host] = Dankclaw state verified on the live machine 2026-10-19 (Main), not a vendor claim. [INFERENCE] = my deduction from the cited artifacts.

0. Answers to the three questions Main ranked

(1) A runtime that already runs TTS on XDNA2 today: it exists at AMD, not on our host. Ryzen AI Software 1.8.0 (docs latest == 1.8.0; released 2026-07-23) lists neutts-air and neutts-nano as model type TTS, variant full-optimized (NPU-only) — https://ryzenai.docs.amd.com/en/latest/llm_list.html; release note "New TTS model support: neutts-air, neutts-nano" — https://ryzenai.docs.amd.com/en/latest/relnotes.html. Pre-compiled artifacts are published: amd/neutts-air_rai_1.8.0_npu_4K (arch qwen2, lastModified 2026-07-23, Apache-2.0, "AWQ / Group 128 / Asymmetric / BFP16 activations / UINT4 Weights") and amd/neutts-nano_rai_1.8.0_npu_4K (arch llama). Neither artifact is a TTS engine. Both file lists are LM-only (model.onnx, genai_config.json, BPE tokenizer, AIE-ML compiler metastate: dd_metastate_* .ctrlpkt/.super/.state, cache/txn_bins.zip) with no codec, no vocoder, no reference-audio encoder. [INFERENCE from HF file lists]

(2) FastFlowLM: no TTS, and we cannot add one — see §2. Halogen: no TTS entry [host]. So "something installed already serves TTS on the NPU" is FALSE.

(3) Our own ONNX graph compiled for the NPU on Linux: possible for the plain ONNX flows (they compile on Linux), impossible for a new LLM/OGA model — model generation is Windows-only in 1.8.0 — and gated by an operator table where BF16 lacks Softmax/LayerNorm/ReLU/Gelu (see §4).

1. Landscape: every way to reach the XDNA2 array on Linux

#	Path	License	Strix Halo + Linux, 2026-10	Speech synthesis
1	Ryzen AI SW → ONNX Runtime + Vitis AI / Ryzen AI EP; compile INT8 (X2) or BF16 (VAIML) ONNX at session creation	AMD proprietary (tarball)	supported: our device ID/rev is in AMD's STX table; compile+run works on Linux (§3, §4)	user-authored ONNX only; no TTS recipe/demo
2	Same EP driven by OGA (onnxruntime-genai), "full fusion" NPU-only LLM flow; runner = AMD model_benchmark / OGA C++ API	AMD + MS OGA	Linux: vendor-precompiled models only, no hybrid flow	neutts-air / neutts-nano = TTS (LM half)
3	FastFlowLM flm	MIT + free proprietary kernels	Strix Halo "Supported"; Ubuntu 24.04/25.10/26.04, Arch	none (ASR input only)
4	IRON / mlir-aie + Peano → hand-written AIE-ML design → XRT/amdxdna	Apache-2.0 + LLVM exception	explicitly documented "AMD Ryzen™ AI on Linux"; same driver/XRT stack	anything, hand-written; nothing ships TTS
5	Vendor-curated AIE-ML graph sets (halogen HALOGEN_NPU_MODELS, FLM packs)	vendor	present [host]	none [host]
6	Windows-only: DirectML, Ryzen AI Windows EP (ryzenai-llm), Studio Effects, whispercpp NPU	—	not usable	(per Lemonade matrix: Windows only)
7	ggml/llama.cpp/qwentts.cpp, whisper.cpp, sherpa-onnx, Piper	—	no XDNA backend in ggml/ORT-mainline	CPU / iGPU only
8	IREE, TVM, ONNX Runtime mainline EPs (OpenVINO = Intel NPU, MIGraphX/ROCm = AMD GPU)	—	no XDNA2 target; AMD's NPU EP is only "Xilinx Vitis-AI (preview)" in the ORT EP list and ships inside AMD's own build	n/a

Sources: §2–§6 below.

2. FastFlowLM (/usr/bin/flm 1.0.1 [host]) — catalog + HTTP surface

• Repo moved into AMD's ROCm org 2026-08-11 (github.com/ROCm/FastFlowLM, v1.0.0). "Supports all Ryzen™ AI Series chips with XDNA2 NPUs (Strix, Strix Halo, Kraken, and Gorgon Point)." https://github.com/ROCm/FastFlowLM (README fetched 2026-10-19).
• Linux support landed 2026-03; distros: Ubuntu 24.04, 25.10, 26.04, Arch, generic. Needs NPU firmware ≥ 1.1.0.0, kernel 7.0+ with amdxdna (or amdxdna-dkms), XRT stack (ppa:lemonade-team/stable → libxrt-npu2, amdxdna-dkms); flm validate reads the DRM path (/dev/accel/accel0), flm run goes through XRT and fails with No such device with index '0' unless xrt-smi examine lists the NPU; memlock must be raised. https://fastflowlm.com/docs/install_lin/ (page dated 2026-03-05, fetched 2026-10-19). Static build bundles XRT v2.21.75 — the same XRT series as the distro package on the host [host].
• HTTP surface: flm serve <model> → OpenAI-compatible REST, port 52625, no auth. Maintained endpoints: v1/models, v1/chat/completions, v1/audio/transcriptions, v1/embeddings. There is no /v1/audio/speech. https://fastflowlm.com/docs/instructions/server/openapi/
• Catalog 2026-10-19 (https://fastflowlm.com/docs/models/): LLaMA, DeepSeek, Qwen, Gemma, MedGemma, TranslateGemma, gpt-oss, LiquidAI/LFM, Phi, Nanbeige, Whisper — card states "Type: Speech-to-Text (ASR)" (https://fastflowlm.com/docs/models/whisper/), EmbeddingGemma, SmolVLA, Hy-MT2, MiniCPM. No speech-synthesis entry.
• Engines are prebuilt, closed, downloaded from Hugging Face per architecture ("Internet access to HuggingFace is required to download the optimized model kernels"). The only in-tree engine build is Gemma-4E (cmake --preset linux-default -DFLM_BUILD_GEMMA4E=ON). → A new TTS architecture cannot be added to flm by us.

3. AMD Ryzen AI Software 1.8.0 on Linux — install source, device support

• Install channel (important correction to the pypi.amd.com evidence). The pip index is Windows-only for the EP runtime (ryzenai_dynamic_dispatch-1.7.0-… win_amd64.whl, onnxruntime_vitisai-1.23.2-…win_amd64.whl; flexmlrt has a single linux_aarch64 wheel) — but AMD's Linux stack is not on pip. It ships as a tarball that is, as of 2026-10-19, publicly reachable without an AMD account: https://download.amd.com/opendownload/RyzenAI/1.8.0/ryzen_ai-1.8.0.tgz → HTTP 206 on a range probe, Content-Range: bytes 0-8/**7819506855** (≈7.8 GiB, magic 1f 8b 08 = gzip). (The docs link it behind the EULA form account.amd.com/...ryzenai-eula-public-xef.html; the CDN path serves it directly. Verified by me with curl 2026-10-19. I did not finish downloading 7.8 GiB, so the tarball's contents list is unverified — [INFERENCE] only where noted.)
• Driver/XRT package (public, 90,016,137 bytes, application/zip): https://download.amd.com/opendownload/RyzenAI/Driver/RAI_1.8_Linux_NPU_XRT.zip contains xrt_202620.2.25.37_24.04-amd64-{base,base-dev,npu}.deb + xrt_plugin.2.25.260102.56.release_24.04-amd64-amdxdna.deb (verified by download + unpack). The plugin deb ships firmware for /usr/lib/firmware/amdnpu/17f0_11/npu.dev.sbin — exactly the directory our NPU loads [host] — plus 1502_00, 17f0_10, 17f1_10, 17f1_13, libvxdna.so, libxrt_driver_xdna.so.2.25.260102.56, and the DKMS driver source tarball. No xclbin/PDI and no compiler binary in this package: it is driver + XRT shim + firmware only.
• Our device is officially STX. AMD's APU table (https://ryzenai.docs.amd.com/en/latest/app_development.md#apu-types): 0x1022:0x17F0 rev 0x11 → STX (0x00/0x10 also STX, 0x20 = KRK, 0x1502 rev 0 = PHX/HPT). Release notes: "Strix and Strix Halo as STX", and the STX/KRK column enables CNN INT8, CNN BF16, NLP BF16, LLM (OGA).
• Vendor driver source (same deb, share/amdxdna/amdxdna.tar.gz, source dated 2026-06-29) maps 0x17f0 rev 0x11 → dev_npu5_info, .fw_path = "amdnpu/17f0_11/", default_vbnv = "RyzenAI-npu5", and the revision→name table has { AIE2_DEV_REVISION_HALO, "NPU Strix Halo" }; xrt-smi examine therefore reports the device as NPU Strix Halo.
• Linux install recipe (1.8.0 docs https://ryzenai.docs.amd.com/en/latest/linux.html): Ubuntu 24.04 packages, Python 3.12 (apt install python3.12 python3.12-venv), libboost-filesystem1.74.0, dkms; then install_ryzen_ai.sh -a yes -p <venv>; LD_LIBRARY_PATH=…:${RYZEN_AI_INSTALLATION_PATH}/onnxruntime/lib/; quicktest.py "picks up a simple CNN model, compiles it and runs on AMD's Neural Processing Unit" — i.e. compilation runs on Linux for the non-LLM flows. "Linux currently supports NPU only flow" (no iGPU EP). Host friction: our box is Ubuntu 26.04 + Python 3.14 [host], so the venv must be built with a 3.12 interpreter and the packages are 24.04-built [INFERENCE].
• LLM flow on Linux (https://ryzenai.docs.amd.com/en/latest/llm_linux.html): prequantized ready-to-run models from HF, deployment/lib, RYZENAI_EP_PATH=$PWD/deployment/lib/libonnxruntime_providers_ryzenai.so, run via the model_benchmark binary; "Linux does not support Hybrid flow"; "Model generation is not supported in current release" (repeated in relnotes 1.8 Known Issues: "Models generated on Windows are compatible with Linux"); needs memlock unlimited. Sample throughput in that doc: 49.1 tokens/s decode, 753 tokens/s prefill (Phi-3.5-mini, 4K model).
• Third-party confirmation of the OS split: Lemonade's backend matrix (https://raw.githubusercontent.com/lemonade-sdk/lemonade/main/README.md, fetched 2026-10-19) — flm/npu/XDNA2 = Windows, Linux; ryzenai-llm/npu = Windows only; whispercpp/npu (STT) = Windows only; text-to-speech engines kokoro and acestep = metal / cuda / vulkan / rocm, no npu device at all. Lemonade is the "Server Interface" layer of AMD's own Ryzen AI stack (docs index), so this matrix is the vendor-sanctioned view.

4. Can we compile our own graph on Linux? (the no-recipe path)

Mechanics from https://ryzenai.docs.amd.com/en/latest/modelrun.md (1.8.0): ONNX only, opset 17 recommended; the EP compiles on first InferenceSession and caches (VitisAI EP cache, or ORT EP context cache — ep.context_enable, and "precompile the EP context model using Python and deploy it using a C++ program" is explicitly supported); EP auto-partitions unsupported nodes to CPU; BF16 = target VAIML (preferred_data_storage: unvectorized for transformers), BF16 STX/KRK-or-newer only; INT8 = target=X2 with XINT8/A8W8/A16W8 and "general asymmetric quantization … to enable third-party quantized models"; deployment EP cannot compile BF16 on the fly — ship a pre-compiled model (https://ryzenai.docs.amd.com/en/latest/app_development.md); cache dirs are not portable across EP or NPU-driver versions.

Operator coverage is the hard gate (https://ryzenai.docs.amd.com/en/latest/ops_support.md, "operators supported in Ryzen AI 1.8.0"; "Y" = broad coverage, some configurations may still fail):

• Good for TTS front-ends/vocoders: ConvTranspose Y in BF16/A16W8/A8W8 (absent only from XINT8), Conv, Resize, Upsample, Pad, LSTM (BF16), CumSum (BF16/XINT8), STFT (XINT8 only).
• Blockers for a decoder-style transformer in the plain flows: in BF16 the table has no Softmax, no LayerNormalization, no Relu/Gelu; LayerNorm is XINT8-only and Softmax is A16W8/A8W8/XINT8-only. So attention either lives in the LLM/OGA flow (whose op set is exactly SimplifiedLayerNormalization, SkipSimplifiedLayerNormalization, MatMulNBits (W4ABF16/W4ABFP16), RotaryEmbedding, GroupQueryAttention, Sigmoid, Mul, Add) or must be decomposed into a quantized INT8 flow.
• Missing entirely: GRU, Trilu, OneHot, EyeLike, Multinomial, NonMaxSuppression, CastLike, MelWeightMatrix/MelFilterGroup, RMSNormalization → graphs that emit them must be rewritten before they partition onto the NPU. [INFERENCE from absence in the table.]
• Quantization is a separate tool, not in the driver package: AMD Quark (quark.torch / quarkonnx), which documents ConvTranspose2d, conv1d, softmax/LayerNorm/BFP quantization (relnotes 1.3/1.2/1.1/0.9) — Quark is not on pypi.org under the obvious names [host probe] and ships from AMD's gated/pip-private channel.
• New LLM-shaped model (our Qwen-Talker analogue): Windows-only today. OGA model generation is not available on Linux in 1.8.0 (§3). So a new speech-LM cannot be self-served onto the NPU on Linux without AMD (or a Windows box) in the loop.
• Practical shape cost: the neutts artifacts carry one compiled variant per padded sequence length (128/256/512/768/1024/1536/2048/2304/3072/4096 for prefill; 256…4096 for token-phase attention mask) — i.e. per-shape compile artifacts are the norm for the LLM flow. [INFERENCE from the HF file list.]

5. Does the AMD TTS artifact actually produce speech? (cloned-voice survival)

NeuTTS is LM + separate neural codec ("Simple LM + codec architecture"), clone from ~3 s of reference audio, and the codec is NeuCodec (50 Hz, single codebook): neuphonic/neucodec (PyTorch, encode+decode), neucodec-onnx-decoder{,-int8} (decode only), "Decoder-only (ONNX) codecs require pre-encoded references — these ship with the repo for all bundled voices." Phoneme input; 2048-token context ≈ 30 s of audio; NeuTTS-Air ~360 M active / 552 M with embeddings; upstream benchmarks are CPU-only llama.cpp (119 tok/s prefill/decode mix on a Ryzen 9 HX 370) and explicitly exclude the codec. https://raw.githubusercontent.com/neuphonic/neutts/main/README.md (fetched 2026-10-19); neuphonic/neutts-air HF metadata (lastModified 2026-08-26, qwen2, GGUF + safetensors).

Consequences for the fleet spike:

1. AMD's NPU artifact = LM only; waveform decode and reference encoding are CPU work either way. Our reference.wav voices survive only if we run the NeuCodec encoder (PyTorch, per the codec table there is no ONNX encoder) once per voice and then feed pre-encoded codes. That is a one-time offline step per voice; after that the compiled graph takes codes/prompt prefix as ordinary inputs, so nothing about the compiled graph forbids a caller-supplied speaker prefix. [INFERENCE]
2. Real-time math: 50 Hz codec means ≥50 tokens/s decode for 1× RT. AMD's own Linux sample output for a 4K Phi-3.5-mini is 49.1 tok/s — so budget for roughly RT-1× on one 4K "full fusion" graph, before the codec's CPU cost. [INFERENCE — different model; must be measured.]
3. Host RAM: the same doc reports peak working set 3.54 GB for that 4K model. NeuTTS-Air is ~552 M params at W4A-BFP16, so expect a smaller footprint than 3.5 GB [INFERENCE]; still non-trivial against 14 GiB available [host].
4. AMD ships no TTS demo, runner, or server on any OS: the examples page has only CNN/BF16/DistilBERT/super-resolution/OGA-LLM/VLM/RAG (https://ryzenai.docs.amd.com/en/latest/examples.md); the RyzenAI-SW repo tree (669 paths, main, fetched 2026-10-19) contains speech demos only under Demos/ASR/{Whisper,Parakeet-TDT} and Transformer-examples/ASR/Whisper-AI — no TTS directory. Serving TTS therefore means writing our own OGA host (or driving model_benchmark-style C++ code) plus phonemizer + NeuCodec glue.

6. Two graphs on one NPU, and the iGPU hazard

• Concurrency is a first-class driver feature. The vendor amdxdna source (in xrt_plugin.2.25.260102.56, 2026-06-29) sets .hwctx_limit = 16 for npu4/npu5 — i.e. 16 hardware contexts on our device — and allocates each context a set of AIE columns with QoS opc = col_opc (4096) × num_col (aie2_ctx.c, aie2_solver.h), with frame_boundary_preempt / fill_preempt_slot in aie2_pci.h (firmware preemption at frame boundaries). So halogen's armed models and a TTS graph can hold separate contexts simultaneously; they contend for columns and OPC share, they are not mutually exclusive.
• AMD's user-visible history agrees: 1.4 "Support for up to 16 hardware contexts on Ryzen AI 300 series"; 1.5 "Application concurrency: improves the resource distribution across applications"; 1.3 "Idle detection and suspension of contexts", "Rebalance feature for AIE hardware resource optimization"; 1.0 known issue "Support for multiple applications is limited to up to eight" (all in https://ryzenai.docs.amd.com/en/latest/relnotes.html).
• Package conflict, the real risk [INFERENCE]: the vendor stack wants AMD's XRT 2.25.37 + xrt_plugin…amdxdna deb [host: distro libxrt-npu2 1:2.21.75+dfsg-4 installed], and firmware must match the loaded amdxdna.ko — the vendor README warns that a stale/mismatched npu.sbin causes ERT_CMD_ABORT/mailbox timeouts, and that a distro in-tree amdxdna older than the XRT SHIM makes GET_ARRAY/QUERY_TELEMETRY return EOPNOTSUPP/EINVAL (https://github.com/amd/xdna-driver README, fetched 2026-10-19). Installing AMD's debs over the distro XRT changes the ABI under FastFlowLM/halogen, so the spike must decide the XRT version once, for the whole host.
• iGPU / fabric clock: no vendor document supports a fabric-clock pin, and no vendor document addresses NPU+iGPU concurrency hangs. AMD's Strix Halo optimization doc (ROCm docs, dated 2026-02-20, and its successor "AMD RDNA3.5 system optimization", ROCm 10.1.0) covers GPUVM/GART/GTT, the TTM pages_limit knob, amd-ttm, BIOS VRAM sizing, and minimum kernel versions (https://rocm.docs.amd.com/en/docs-7.2.0/how-to/system-optimization/strixhalo.html, https://rocm.docs.amd.com/en/latest/reference/system-optimization/rdna3-5.html). The pp_dpm_fclk level-7 pin and the GPU+NPU hang observation stay fleet-observed [host], i.e. [INFERENCE] territory — no citation exists.

7. Paths I checked and ruled out (so nobody re-walks them)

• Windows-only doors: DirectML; Ryzen AI Windows EP; Lemonade's ryzenai-llm and whispercpp-NPU engines (matrix above); AMD's OGA model generation. Not usable on Dankclaw.
• Mainline ONNX Runtime has no AMD-NPU EP: its EP summary lists AMD ROCm and AMD MIGraphX (GPU) and "Xilinx Vitis-AI (preview)" only (https://onnxruntime.ai/docs/execution-providers/, 2026-10-19); the working EP is AMD's own build (libonnxruntime_providers_ryzenai.so in the tarball's deployment dir, per llm_linux).
• ggml / llama.cpp / qwentts.cpp: no XDNA backend → the current engine (tts-server on GGUF, CPU-only build [host]) cannot be "turned on" for the NPU; migration means a different runtime, not a flag.
• flm: catalog/engines are closed per-architecture (§2) — the cheapest door by install cost, and it is not a door for TTS.
• IRON / mlir-aie is the only open toolchain that documents running on the Ryzen AI NPU from Linux ("A close-to-metal Python API for programming AMD Ryzen™ AI NPUs, built on an open-source MLIR-based compiler toolchain"; "Getting Started for AMD Ryzen™ AI on Linux", Ubuntu 24.04/24.10, XRT from ppa:amd-team/xrt: libxrt2 libxrt-npu2 … amdxdna-dkms, render group required) — https://raw.githubusercontent.com/Xilinx/mlir-aie/main/README.md, fetched 2026-10-19. This is how FLM's kernels were made ("Low-level kernels optimized using IRON + AIE-MLIR", FLM README) — i.e. real, but it is hand-written per-op tile programming, not a model compiler: writing a TTS model here is a multi-month engineering project, not a migration.
• Embedded Ryzen AI P100 support in the same Linux doc is irrelevant to STX.

8. Recommendation implied by the above (for Main's spike)

Spike the AMD path as decided, with these constraints planned in: Python 3.12 venv

• 24.04-built wheels on Ubuntu 26.04; one host-wide XRT decision (AMD 2.25.37 vs distro 2.21.75) with firmware/driver pairing checked via xrt-smi examine → "NPU Strix Halo"; 7.8 GiB tarball staged off the public CDN URL (§3); pre-encode each reference.wav with the PyTorch NeuCodec encoder because the ONNX codec is decoder-only; keep phonemizer + codec on CPU; measure decode tokens/s against the 50 Hz budget and peak RSS against the ~14 GiB headroom; expect to write the serving shell yourself (AMD ships model_benchmark, no TTS demo).

9. Loose ends (not verified; I stopped on Main's stand-down)

• Contents of ryzen_ai-1.8.0.tgz (7.8 GiB) — whether the Linux tarball contains the AIE-ML compiler (vaiml, aiecompiler_client) for the BF16/INT8 flows and which Python wheels it carries. My statement that non-LLM compilation works on Linux rests on AMD's own Linux doc ("quicktest.py … compiles it and runs on … NPU"), not on the package contents. Download was killed mid-flight.
• Whether AMD's neutts artifacts run correctly under the Linux model_benchmark runner (the TTS rows are listed without OS qualifiers; no Linux TTS run is documented anywhere in 1.8.0 docs).
• Exact AIE column count/total ncols exposed for Strix Halo (driver reads it from AIE metadata: ndev->total_col = min(XRS_MAX_COL, ndev->aie.metadata.cols)).