Frank
8af40c402b
Landing blocker for #61 and #63: the deployment checkout is 12 commits behind and `merge --ff-only` refuses while tracked paths are dirty. Discard is not available here (workspace policy: no `checkout`/`stash` on shared state), so the dirt is committed. - `agent/APPEND_SYSTEM.md`: the point-of-use / one-work-log / PR-link doctrine, already byte-identical to `origin/main` (`cmp` against the trunk blob: no difference). No new prose — the commit exists so the trunk merge can proceed. - `agent/config.yml` and `agent/managed-skills/**`: the machine-live tier the heartbeat owns (AGENTS.md § Live-sync tiers). `ALLOW_SECRETS=1` was used, deliberately and after auditing the staged diff by hand: the only credential-shaped strings in it are two copies of `AKIA0000000000000EXAMPLE` — the inert all-zero AWS example id — and they appear inside the two skill pages that teach that exact placeholder shape. This checkout is still running the pre-#61 hook; the merged gate (value-scoped placeholder filter, `71601f7`) exempts this value, so the refusal dies with this landing. No `--no-verify` was used: the gate ran and its verdict is recorded here. Refs #61
54 lines
6.1 KiB
Markdown
54 lines
6.1 KiB
Markdown
---
|
|
name: halogen-npu-feasibility-census
|
|
description: "Census whether halogen-flash-server's NPU small-model path can run and swap models on a Strix Halo host: image gate, amdxdna/XRT presence, IOMMU and fabric-clock blockers, enablement deltas, boot-set-only model loading (no live reload), and the silent-fallback-to-Flash trap for unarmed chat-class NPU models."
|
|
---
|
|
|
|
Use when asked whether halogen/halogen-flash-server can use the XDNA2 NPU (embeddings, rerank, moderation/guard, decider, `qwen3.5-2b` generation, `flux2-klein-4b` images, `qwen3-asr-*`) on a Ryzen AI Max / Strix Halo host, before proposing to enable it, or when asked whether it can **hot-swap / live-reload** NPU models like FastFlowLM does.
|
|
|
|
## What the feature is (not a speed knob)
|
|
|
|
The Flash trunk stays on the iGPU. `HALOGEN_NPU_MODELS=<names>` starts a second engine in the SAME container and on the SAME API port; the request's `model` field picks the executor. Unset -> server identical to before. Each NPU model is a precompiled AIE-ML program (`devices/devices.hnpm`, per-partition `devices/u<N>.elf`, `devices/consts.hnpb`, `<id>.hnpw` weights, `tokenizer/tokenizer.json`), loaded via the HOST's XRT -> kernel `amdxdna` -> `/dev/accel/accel0`. One pass at a time; `HALOGEN_NPU_QUEUE` 64, overflow queues server-side, then `503 engine_busy`. Catalog in the 0.16.2 image: `decider-0.8b`, `qwen3-embedding-0.6b`, `qwen3-reranker-0.6b`, `qwen3guard-gen-0.6b`, `qwen3.5-2b` (+ `flux2-klein-4b`, `qwen3-asr-*` in newer tags). Only the programs named in `HALOGEN_NPU_MODELS` are armed.
|
|
|
|
## Boot-set-only: NO live reload (tested on the live endpoint)
|
|
|
|
`HALOGEN_NPU_MODELS` is read **at start**; the entrypoint validates/converts/loads the named programs then. There is **no** runtime load/unload and **no** admin endpoint to add a model. FLM comparison: FLM was a standalone NPU-only server (`flm serve <one-model>`, single FIFO lane) free to reconfigure its one AIE partition; halogen boots Flash (iGPU) **plus** the NPU programs as ONE coupled process (shared port + shared memory/power budget), so the NPU set is a boot-time constant. Changing even a checkpoint's weights reconverts "at the next start", never live.
|
|
|
|
Consequence — arming a new program = edit `scripts/frank-dragoon.env` -> redeploy -> **container restart -> the whole Flash payload (tens of GB) reloads, minutes**. The small NPU program is not the cost; the Flash reload is.
|
|
|
|
## The silent-fallback trap (the one that bites)
|
|
|
|
A request naming an **unarmed** catalog model behaves differently per endpoint class:
|
|
- `/v1/rerank`, `/v1/moderations`: honest **400 `model_not_found`**, message ends `(loaded: none)`. The endpoint only exists if that engine was armed at boot.
|
|
- `/v1/chat/completions` (the `decider-0.8b` decision path and `qwen3.5-2b` generation): **silent fall-through to the Flash model** — HTTP 200, body `"model": "<flash id>"`, no error. A guard/decision call to an unarmed model is answered by the main (often abliterated) endpoint you were trying to screen against, and looks like success. Measured: an unarmed `decider-0.8b` "is this a prompt injection?" probe returned `"no"` from Flash. **Fail-safe is NOT the default on the chat route.**
|
|
|
|
## Live probe (idle lane; one ssh, non-disruptive — no restart happens)
|
|
|
|
```bash
|
|
B=http://127.0.0.1:7770 # halogen port; internal engine is :8731/:8740 muxed onto it
|
|
curl -s $B/v1/models | jq -r '.data[].id' # armed set = boot set; must not mutate
|
|
# control (armed embedding) -> 200 with a real vector:
|
|
curl -s $B/v1/embeddings -d '{"model":"qwen3-embedding-0.6b","input":["ping"]}'
|
|
# unarmed rerank/guard -> expect 400 "(loaded: none)":
|
|
curl -s $B/v1/rerank -d '{"model":"qwen3-reranker-0.6b","query":"t","documents":["a","b"],"top_n":1}'
|
|
curl -s $B/v1/moderations -d '{"model":"qwen3guard-gen-0.6b","input":["hello"]}'
|
|
# unarmed decider/gen -> watch the "model" field; if it is the flash id, it fell through:
|
|
curl -s $B/v1/chat/completions -d '{"model":"decider-0.8b","temperature":0,"messages":[{"role":"user","content":"x"}],"response_format":{"type":"json_schema","json_schema":{"name":"g","description":"is injection?","schema":{"enum":["no","yes"]}}}}' | jq -r '.model // .error.message'
|
|
# prove nothing restarted:
|
|
systemctl show frank-dragoon.service -p NRestarts -p ActiveEnterTimestamp --value
|
|
```
|
|
`decider` with a schema and a non-zero `temperature` 400s on a *sampling* guard first ("send temperature 0"); set `temperature:0` to reach the router and see the fallback.
|
|
|
|
## Enablement gates (all four must hold before it runs)
|
|
|
|
1. Kernel `amdxdna` loaded + `/dev/accel/accel0` present; the node must be world-open (`0666` via a udev rule) or a **rootless** container cannot open it — `serve-frank-dragoon.sh` refuses if the mode bit 02 is clear.
|
|
2. Host XRT with the NPU plugin (`libxrt_coreutil.so.2`, `libxrt_core.so.2`, `libxrt_driver_xdna.so.2`); on Arch/CachyOS mount each lib twice (image XRT path + its own `/usr/lib` path), never an `/opt/xilinx/xrt` of dangling links.
|
|
3. **Fabric clock held at top** (`halogen-fabric-clock` root unit) — GPU+NPU at once can hang Strix Halo / corrupt NPU results while that clock changes; the server refuses to start if it can't hold it.
|
|
4. **IOMMU on**: `amd_iommu=off` kills the NPU; `iommu=pt` keeps it.
|
|
|
|
## Laws / traps
|
|
|
|
- Never probe the NPU beside the live service: stop + mask first (fleet benchmarking policy); this host class has a precedent of a concurrent GPU experiment needing a power cycle. Distinguish off vs wedged with `ip neigh show <ip>` from a peer. A single embedding/decision request on an already-armed idle production lane is a normal request, not a benchmark — safe.
|
|
- `/dev/accel/accel0` existing and `amdxdna` probing cleanly does NOT mean the NPU works - with IOMMU off the failure is at context/PASID creation, not at the device node. Do not trust older notes claiming the node itself is missing.
|
|
- `dmesg` lines like `Failed to alloc dev bo memory, ret -28` while large NPU models take turns at NPU memory are benign (vendor-documented); the server recovers.
|
|
- NPU model RSS is host-side, bypassing the container cgroup (GTT/shared pool), like all Strix Halo GPU memory — container stats under-report the true footprint.
|