49 lines
4.6 KiB
Markdown
49 lines
4.6 KiB
Markdown
---
|
||
name: halogen-npu-feasibility-census
|
||
description: "Census whether halogen-flash-server's NPU small-model path can run on a Strix Halo host: image gate, amdxdna/XRT presence, IOMMU and fabric-clock blockers, enablement deltas."
|
||
---
|
||
|
||
Use when asked whether halogen/halogen-flash-server can use the XDNA2 NPU (embeddings, rerank, moderation/guard, decider, `qwen3.5-2b` generation, `flux2-klein-4b` images, `qwen3-asr-*`) on a Ryzen AI Max / Strix Halo host, or before proposing to enable it.
|
||
|
||
## What the feature is (not a speed knob)
|
||
|
||
The Flash trunk stays on the iGPU. `HALOGEN_NPU_MODELS=<names>` starts a second engine in the SAME container and on the SAME API port; the request's `model` field picks the executor. Unset → server identical to before. Each NPU model is a precompiled AIE-ML program (`devices/devices.hnpm`, per-partition `devices/u<N>.elf`, `devices/consts.hnpb`, `<id>.hnpw` weights, `tokenizer/tokenizer.json`), loaded via the HOST's XRT → kernel `amdxdna` → `/dev/accel/accel0`. One pass at a time; `HALOGEN_NPU_QUEUE` 64, overflow queues server-side, then `503 engine_busy`. Entry `entrypoint.sh` validates every file's bytes + sha256 against `/opt/halogen/npu/models.txt` before anything starts. NPU models run only in single-container "all" mode, NOT the `engine`/`api` split.
|
||
|
||
## Census (all read-only, run from the operator box)
|
||
|
||
```bash
|
||
ssh <host> '
|
||
# 1. engine gate + catalog in the pinned image
|
||
podman exec <ctr> bash -c "grep -rlo HALOGEN_NPU_MODELS /opt/halogen /usr/local/bin /entrypoint.sh 2>/dev/null | head"
|
||
podman exec <ctr> cat /opt/halogen/npu/models.txt | head -40
|
||
# 2. is NPU actually wired today? (expect NO env, NO device, NO /dev/accel inside)
|
||
podman inspect <ctr> --format "{{json .HostConfig.Devices}}"; podman exec <ctr> ls /dev/accel
|
||
# 3. kernel driver + firmware
|
||
lsmod | grep -i amdxdna; sudo dmesg | grep -iE "xdna|npu" | tail -5; ls -l /dev/accel
|
||
# 4. XRT (distro packages are fine; vendor mounts /opt/xilinx/xrt)
|
||
dpkg -l | grep -iE "xrt|xdna"; ls /usr/lib/x86_64-linux-gnu/libxrt* /usr/lib/libxrt* 2>/dev/null
|
||
# 5. BLOCKER 1: IOMMU
|
||
cat /proc/cmdline; ls /sys/kernel/iommu_groups | wc -l
|
||
# 6. BLOCKER 2: fabric clock held?
|
||
for c in /sys/class/drm/card*/device; do echo "== $c"; cat $c/power_dpm_force_performance_level 2>&1; cat $c/pp_dpm_fclk 2>&1 | tr "\n" " "; echo; done'
|
||
```
|
||
|
||
Docs (raw fetch, authoritative): `docs/NPU.md`, `docs/npu-small-models.md`, `deploy/host/halogen-fabric-clock` under `peonist-ai/halogen-flash-server` on GitHub. Tag list: `podman search --list-tags ghcr.io/peonist-ai/halogen-flash-server`.
|
||
|
||
## The two blockers and the enablement delta
|
||
|
||
- **IOMMU must be on.** `amd_iommu=off` turns the NPU off (no PASID/SVA); `iommu=pt` keeps it. Needs a grub edit + reboot. `iommu_groups == 0` ⇒ off.
|
||
- **Fabric clock must be pinned at the top `pp_dpm_fclk` level** (or `power_dpm_force_performance_level=high`); `auto` = unheld. GPU+NPU concurrently HANG Strix Halo machines and corrupt NPU results while that clock moves. Fix: root container with `-v /sys:/host/sys` (server holds it itself), or install `deploy/host/halogen-fabric-clock` + unit (~1 W). If it can't hold, the server FAILS CLOSED and prints the commands.
|
||
- Container delta: `--device /dev/accel/accel0` + `HALOGEN_NPU_MODELS`; `--ulimit memlock=-1:-1` and `--group-add keep-groups` usually already present. Distro XRT: mount each of `libxrt_coreutil.so.2`, `libxrt_core.so.2`, `libxrt_driver_xdna.so.2` TWICE (once at `/opt/xilinx/xrt/lib/`, once at its own path) — never mount an `/opt/xilinx/xrt` holding only symlinks.
|
||
|
||
## Costs to price before proposing
|
||
|
||
- Upstream A/B: IOMMU translation ON costs **13–16 % prefill** (460 → 385 tok/s at 2,048 tokens, Ryzen AI Max+ 395); decode unaffected. Whether `iommu=pt` specifically costs that on a given box is a local measurement, not a published fact — run the image's own `sweep` after the reboot.
|
||
- A few GB host RAM per NPU model (+8 GB FLUX, +1.6/4 GB ASR), and trunk decode drops while the NPU works (shared bus + power).
|
||
|
||
## Laws / traps
|
||
|
||
- Never probe the NPU beside the live service: stop + mask first (fleet benchmarking policy); this host class has a precedent of a concurrent GPU experiment needing a power cycle. Distinguish off vs wedged with `ip neigh show <ip>` from a peer.
|
||
- `/dev/accel/accel0` existing and `amdxdna` probing cleanly does NOT mean the NPU works — with IOMMU off the failure is at context/PASID creation, not at the device node. Do not trust older notes claiming the node itself is missing.
|
||
- `dmesg` lines like `Failed to alloc dev bo memory, ret -28` while large NPU models take turns at NPU memory are benign (vendor-documented); the server recovers.
|