62 lines
9.0 KiB
Markdown
62 lines
9.0 KiB
Markdown
---
|
||
name: ryzenai-npu-oga-linux-spike
|
||
description: "Run an AMD-precompiled AMD Ryzen AI (XDNA2) ONNX-GenAI model on a Linux Strix Halo host from public artifacts: staging, rootless NPU device-access wall, container recipe, and the NuTTS-air speech-token contract (65,536-code vocab, 50 codes/s real-time line)."
|
||
---
|
||
|
||
Run an AMD-precompiled XDNA2 model (OGA / onnxruntime-genai) on a Linux Strix Halo host (Dankclaw-class) without installing the vendor stack on the host.
|
||
|
||
## Where the bits actually live (both public, no AMD account)
|
||
|
||
- Runtime tarball: `https://download.amd.com/opendownload/RyzenAI/1.8.0/ryzen_ai-1.8.0.tgz` (~7.28 GiB). Verify with a ranged GET — first bytes `1f 8b 08`.
|
||
- NPU driver/XRT zip: `https://download.amd.com/opendownload/RyzenAI/Driver/RAI_1.8_Linux_NPU_XRT.zip` (90 MB).
|
||
- Dead ends (404, do not re-probe): `download.amd.com/driver/linux/RAI_1.8_Linux_NPU_XRT.zip`, `.../opendownload/RyzenAI/ryzen_ai-1.8.0.tgz`, `www.xilinx.com/bin/public/openDownload?filename=...` (301 → 404). The login-walled EULA *form* is not the gate; the `opendownload` tree is the real file host.
|
||
|
||
Tarball has 30 top-level entries: cp312/linux_x86_64 wheels (`onnxruntime_genai_ryzenai-0.14.0`, `onnxruntime_vitisai-1.27.0`, `vitis_mllib`, `flexml*`, `device_essentials_strx`, `model_generate`, `voe-1.8.0`), plus `deployment.tar.gz` (contains `lib/libonnxruntime-genai.so`, `lib/libonnxruntime_providers_ryzenai.so`, AIE compiler libs), `onnxruntime.tar.gz`, `quicktest.tar.gz` (VitisAI-EP install validator), `LLM.tar.gz` (text-only OGA examples: `run_model.py`, `tokenizer_factory.py`), `install_ryzen_ai.sh`. `voe-4.0-linux_x86_64.tar.gz` is only `vaip_config.json` — no TTS xclbins. **AMD ships no TTS runner and no TTS example.**
|
||
|
||
## Host rules
|
||
|
||
- **Never `dpkg -i` the vendor XRT.** `dpkg-deb -x <deb> <dir>` and bind-mount `<dir>/opt/xilinx/xrt` at the same path in the container. The plugin deb would install a DKMS `amdxdna` that replaces the distro in-tree driver; the in-tree driver on Ubuntu 26.04 is `amdxdna 0.7.0` and already loads `amdnpu/17f0_11/npu_7.sbin` at boot (`dmesg | grep amdxdna`).
|
||
- **Rootless containers cannot read `/dev/accel/accel0`** (`root:render 0660`): the host gid never enters the userns gid map, so `--device` yields `nobody:nogroup`. Verified failing 2026-10-19: `--group-add keep-groups`, `--userns=keep-id`, `--userns=keep-id:gid=990` (process gets gid 990, node still unreadable), `--group-add render`, `--group-add gid:990` (both error). Options: run rootful (`sudo podman`, separate storage → rebuild the image rootful, ~1 min), or add a world-writable udev rule for `accel0` (the fleet already does exactly this for `kfd`/`renderD*` in `99-comfyui-gpu-access.rules`, whose header explains why: systemd user-unit spawns carry no supplementary groups).
|
||
- **Rootless image build trap:** the vendor installer's `tar -xf` preserves recorded uids (95777 etc.) and aborts under `set -e` with `Cannot change ownership ... Invalid argument`, silently skipping the `deployment`/`onnxruntime` extraction. Patch it in the Containerfile: `sed -i 's/tar -xf/tar --no-same-owner -xf/g' install_ryzen_ai.sh`.
|
||
- Installer usage: `./install_ryzen_ai.sh -a yes -p /opt/rai -c /opt/rai` (python **3.12** required; base `python:3.12-slim`; needs `linux-libc-dev zip pciutils patch libgomp1`). It builds a venv from the local wheels, reinstalls `onnxruntime-vitisai` then `--no-deps onnxruntime-genai-ryzenai`, installs `requirements_extras.txt` (transformers 4.57.6), patches `bin/activate` (sets `RYZEN_AI_INSTALLATION_PATH=$VIRTUAL_ENV` + `LD_LIBRARY_PATH` into site-packages), and untars `LLM/deployment/onnxruntime/voe/quicktest` into the cpp path.
|
||
- `xrt-smi` in a container needs `libboost-filesystem1.83.0` + `libboost-program-options1.83.0` (**hyphenated** Ubuntu names) and `apt update` first; boost is **only** needed by `xrt-smi`, never by the OGA/EP path.
|
||
|
||
## HF artifact staging
|
||
|
||
- Gated (401 on API, 403 on git) for a plain account: `neuphonic/neucodec` (the **only** NeuCodec encoder), `neuphonic/neucodec-onnx-decoder`, `neuphonic/neucodec-onnx-decoder-int8`, `neuphonic/neutts-air`. `amd/neutts-air_rai_1.8.0_npu_4K` is public. Public mirrors of the codec exist (`eugenehp/neucodec` has the full 1.16 GB `pytorch_model.bin`) — treat a mirror of a gated release as a licensing decision for the operator, not a workaround to take silently.
|
||
- Gated-repo loader trap: `HF_HUB_OFFLINE=1` + `HF_HOME` pointed at a cache laid out as `hub/models--<org>--<repo>/snapshots/<rev>/…` works, but `NeuCodec._from_pretrained` **asserts the repo id** is in `["neuphonic/neucodec", "neuphonic/distill-neucodec"]` — a local directory path is refused outright, so pass the repo id and satisfy it from the cache.
|
||
- `github.com/neuphonic/neutts` is public and ships `samples/{dave,emily,greta,jo}.{pt,txt,wav}` = **pre-encoded reference codes + transcripts** → a spike needs zero gated bytes.
|
||
- Auth without leaking the token into `ps`: build a 600-perm credential file with a shell builtin and point git at it —
|
||
`umask 077; printf 'https://oauth2:%s@huggingface.co\n' "$(cat ~/.cache/huggingface/token)" > ~/.git-credentials-hf`
|
||
then `git -c credential.helper="store --file ~/.git-credentials-hf" clone ...`. Never `git clone https://oauth2:$TOKEN@...`.
|
||
|
||
## NuTTS-air on XDNA2 (the model-side contract)
|
||
|
||
- The package is an OGA model dir: `genai_config.json` (`type: qwen2`, 24 layers, hidden 896, 14 heads / 2 KV, ctx 32768, `vocab_size` **217,652**, `provider_options: [{"RyzenAI": {"hybrid_opt_token_backend": "npu"}}]`), plus `dd_metastate_*` AIE artifacts and `model.onnx` + `model.onnx.data`.
|
||
- **Speech tokens live in the text vocab, and the codebook is 65,536 — not 10,001.** `tokenizer.json` `added_tokens` holds exactly 65,536 `<|speech_N|>` entries, ids **151,671–217,206** contiguous, over a base BPE vocab of 151,643; specials are 151,644–151,670 (`<|SPEECH_GENERATION_END|>` = 151,670). The vendor template maps a codec index straight to its id: `"".join(f"<|speech_{i}|>" for i in ref_codes)`. An encoded reference really does contain codes in the 60k range, so a 10k assumption silently truncates the space.
|
||
- The graph's own `eos_token_id` is `[151645, 151643]` (the **chat turn** stop) and `search.eos_token_ids` is unset — pass 151,670 as an extra eos or speech ends after one token.
|
||
- **Real-time line = 50 speech codes/s** — NeuCodec `sample_rate 24000 / hop_length 480`, measured exactly (49.96–50.00 across three 20–25 s references). Judge any NPU/iGPU TTS number against 50. (Qwen3-TTS, the fleet's deployed arm, is 12.5 codes/s — do not conflate the two bars.)
|
||
- Reference encode is cheap and cacheable: `librosa.load(sr=16000, mono=True)` → `[1,1,T]` float tensor → `codec.encode_code(audio_or_path=…)` (mirrors `NeuTTS.encode_reference`); CPU encoder RTF 0.28–0.38 on 8 threads. `neucodec` needs `torchtune` (`RotaryPositionalEmbeddings`) — install both `--no-deps --target <dir>` so a second torch never shadows the venv's.
|
||
- Build the prompt by *importing* the vendor function, never by copying the template:
|
||
`sys.path.insert(0, <neutts repo>); from neutts.neutts import NeuTTS`, then
|
||
`NeuTTS._apply_chat_template(SimpleNamespace(tokenizer=hf_tok, input_format="text"), ref_codes, ref_text, text, None)` — works on a duck-typed `self`, so no backbone/codec load. Template: `user\n<|TEXT_REPLACE|>\nassistant\n<|SPEECH_REPLACE|>`; splice `TEXT_PROMPT_START + encode(f"{ref_text} {text}") + TEXT_PROMPT_END` at TEXT_REPLACE, then truncate at SPEECH_REPLACE and append `[SPEECH_GENERATION_START] + encode("".join(f"<|speech_{i}|>"))`.
|
||
- Voice identity = reference codes + transcript (request-time conditioning). No trained artifact exists to preserve; packs holding only `reference.wav` require the gated encoder once, offline — **and** a transcript, which a voice pack does not carry (`manifest.json` holds provenance/crop geometry only) and a host without ASR cannot produce.
|
||
- Full-Fusion input ceiling: 4096 tokens **total**, input + output.
|
||
|
||
## Status caveat (2026-10-19)
|
||
|
||
Staged artifacts, image build, device-access matrix, prompt contract, reference pre-encode:
|
||
**done and reproducible.** Token generation on the NPU: **NO-GO, and the loader is the
|
||
wall.** The device is healthy (`xrt-smi validate` PASSED incl. `throughput`,
|
||
`RyzenAI-npu5 / aie2p / 6×8`, `pyxrt.device(0)` constructs), and `strace` shows the
|
||
failing load reading `genai_config.json`, opening `model.onnx`, loading
|
||
`libxrt_driver_xdna.so.2` and opening `/dev/accel/accel0` with successful GEM ioctls —
|
||
then the RyzenAI EP **declines the graph before touching any `dd_metastate_*`**, silently:
|
||
`GENAI_ORT_VERBOSE_LOGGING`, `GENAI_LOG_ORT_LIB`, `set_log_options`,
|
||
`set_log_callback(callable)`, `RYZENAI_EP_TRACING`, `RYZENAI_LLM_PROFILE_INIT` all emit
|
||
zero lines. Hardlinked mirror dirs of the model tree were ruled out as shadowing. One run
|
||
loaded and produced finite logits; byte-identical runs never reproduced it. **No token rate
|
||
was ever measured, so quote none**, and remember `quicktest` success ≠ NPU residency (that
|
||
build falls back to CPU silently — prove residency via
|
||
`/sys/class/accel/accel0/device/mem_usage`).
|