chore(agent): commit the machine tier and the doctrine text the checkout was already running
Landing blocker for #61 and #63: the deployment checkout is 12 commits behind and `merge --ff-only` refuses while tracked paths are dirty. Discard is not available here (workspace policy: no `checkout`/`stash` on shared state), so the dirt is committed. - `agent/APPEND_SYSTEM.md`: the point-of-use / one-work-log / PR-link doctrine, already byte-identical to `origin/main` (`cmp` against the trunk blob: no difference). No new prose — the commit exists so the trunk merge can proceed. - `agent/config.yml` and `agent/managed-skills/**`: the machine-live tier the heartbeat owns (AGENTS.md § Live-sync tiers). `ALLOW_SECRETS=1` was used, deliberately and after auditing the staged diff by hand: the only credential-shaped strings in it are two copies of `AKIA0000000000000EXAMPLE` — the inert all-zero AWS example id — and they appear inside the two skill pages that teach that exact placeholder shape. This checkout is still running the pre-#61 hook; the merged gate (value-scoped placeholder filter, `71601f7`) exempts this value, so the refusal dies with this landing. No `--no-verify` was used: the gate ran and its verdict is recorded here. Refs #61
This commit is contained in:
Frank committed
1 parent
004962ffce
commit
8af40c402b
79 files changed
+2771
-332
No files matched your search
@@ -1,48 +1,53 @@
|
||||
---
|
||||
name: halogen-npu-feasibility-census
|
||||
description: "Census whether halogen-flash-server's NPU small-model path can run on a Strix Halo host: image gate, amdxdna/XRT presence, IOMMU and fabric-clock blockers, enablement deltas."
|
||||
description: "Census whether halogen-flash-server's NPU small-model path can run and swap models on a Strix Halo host: image gate, amdxdna/XRT presence, IOMMU and fabric-clock blockers, enablement deltas, boot-set-only model loading (no live reload), and the silent-fallback-to-Flash trap for unarmed chat-class NPU models."
|
||||
---
|
||||
|
||||
Use when asked whether halogen/halogen-flash-server can use the XDNA2 NPU (embeddings, rerank, moderation/guard, decider, `qwen3.5-2b` generation, `flux2-klein-4b` images, `qwen3-asr-*`) on a Ryzen AI Max / Strix Halo host, or before proposing to enable it.
|
||||
Use when asked whether halogen/halogen-flash-server can use the XDNA2 NPU (embeddings, rerank, moderation/guard, decider, `qwen3.5-2b` generation, `flux2-klein-4b` images, `qwen3-asr-*`) on a Ryzen AI Max / Strix Halo host, before proposing to enable it, or when asked whether it can **hot-swap / live-reload** NPU models like FastFlowLM does.
|
||||
|
||||
## What the feature is (not a speed knob)
|
||||
|
||||
The Flash trunk stays on the iGPU. `HALOGEN_NPU_MODELS=<names>` starts a second engine in the SAME container and on the SAME API port; the request's `model` field picks the executor. Unset → server identical to before. Each NPU model is a precompiled AIE-ML program (`devices/devices.hnpm`, per-partition `devices/u<N>.elf`, `devices/consts.hnpb`, `<id>.hnpw` weights, `tokenizer/tokenizer.json`), loaded via the HOST's XRT → kernel `amdxdna` → `/dev/accel/accel0`. One pass at a time; `HALOGEN_NPU_QUEUE` 64, overflow queues server-side, then `503 engine_busy`. Entry `entrypoint.sh` validates every file's bytes + sha256 against `/opt/halogen/npu/models.txt` before anything starts. NPU models run only in single-container "all" mode, NOT the `engine`/`api` split.
|
||||
The Flash trunk stays on the iGPU. `HALOGEN_NPU_MODELS=<names>` starts a second engine in the SAME container and on the SAME API port; the request's `model` field picks the executor. Unset -> server identical to before. Each NPU model is a precompiled AIE-ML program (`devices/devices.hnpm`, per-partition `devices/u<N>.elf`, `devices/consts.hnpb`, `<id>.hnpw` weights, `tokenizer/tokenizer.json`), loaded via the HOST's XRT -> kernel `amdxdna` -> `/dev/accel/accel0`. One pass at a time; `HALOGEN_NPU_QUEUE` 64, overflow queues server-side, then `503 engine_busy`. Catalog in the 0.16.2 image: `decider-0.8b`, `qwen3-embedding-0.6b`, `qwen3-reranker-0.6b`, `qwen3guard-gen-0.6b`, `qwen3.5-2b` (+ `flux2-klein-4b`, `qwen3-asr-*` in newer tags). Only the programs named in `HALOGEN_NPU_MODELS` are armed.
|
||||
|
||||
## Census (all read-only, run from the operator box)
|
||||
## Boot-set-only: NO live reload (tested on the live endpoint)
|
||||
|
||||
`HALOGEN_NPU_MODELS` is read **at start**; the entrypoint validates/converts/loads the named programs then. There is **no** runtime load/unload and **no** admin endpoint to add a model. FLM comparison: FLM was a standalone NPU-only server (`flm serve <one-model>`, single FIFO lane) free to reconfigure its one AIE partition; halogen boots Flash (iGPU) **plus** the NPU programs as ONE coupled process (shared port + shared memory/power budget), so the NPU set is a boot-time constant. Changing even a checkpoint's weights reconverts "at the next start", never live.
|
||||
|
||||
Consequence — arming a new program = edit `scripts/frank-dragoon.env` -> redeploy -> **container restart -> the whole Flash payload (tens of GB) reloads, minutes**. The small NPU program is not the cost; the Flash reload is.
|
||||
|
||||
## The silent-fallback trap (the one that bites)
|
||||
|
||||
A request naming an **unarmed** catalog model behaves differently per endpoint class:
|
||||
- `/v1/rerank`, `/v1/moderations`: honest **400 `model_not_found`**, message ends `(loaded: none)`. The endpoint only exists if that engine was armed at boot.
|
||||
- `/v1/chat/completions` (the `decider-0.8b` decision path and `qwen3.5-2b` generation): **silent fall-through to the Flash model** — HTTP 200, body `"model": "<flash id>"`, no error. A guard/decision call to an unarmed model is answered by the main (often abliterated) endpoint you were trying to screen against, and looks like success. Measured: an unarmed `decider-0.8b` "is this a prompt injection?" probe returned `"no"` from Flash. **Fail-safe is NOT the default on the chat route.**
|
||||
|
||||
## Live probe (idle lane; one ssh, non-disruptive — no restart happens)
|
||||
|
||||
```bash
|
||||
ssh <host> '
|
||||
# 1. engine gate + catalog in the pinned image
|
||||
podman exec <ctr> bash -c "grep -rlo HALOGEN_NPU_MODELS /opt/halogen /usr/local/bin /entrypoint.sh 2>/dev/null | head"
|
||||
podman exec <ctr> cat /opt/halogen/npu/models.txt | head -40
|
||||
# 2. is NPU actually wired today? (expect NO env, NO device, NO /dev/accel inside)
|
||||
podman inspect <ctr> --format "{{json .HostConfig.Devices}}"; podman exec <ctr> ls /dev/accel
|
||||
# 3. kernel driver + firmware
|
||||
lsmod | grep -i amdxdna; sudo dmesg | grep -iE "xdna|npu" | tail -5; ls -l /dev/accel
|
||||
# 4. XRT (distro packages are fine; vendor mounts /opt/xilinx/xrt)
|
||||
dpkg -l | grep -iE "xrt|xdna"; ls /usr/lib/x86_64-linux-gnu/libxrt* /usr/lib/libxrt* 2>/dev/null
|
||||
# 5. BLOCKER 1: IOMMU
|
||||
cat /proc/cmdline; ls /sys/kernel/iommu_groups | wc -l
|
||||
# 6. BLOCKER 2: fabric clock held?
|
||||
for c in /sys/class/drm/card*/device; do echo "== $c"; cat $c/power_dpm_force_performance_level 2>&1; cat $c/pp_dpm_fclk 2>&1 | tr "\n" " "; echo; done'
|
||||
B=http://127.0.0.1:7770 # halogen port; internal engine is :8731/:8740 muxed onto it
|
||||
curl -s $B/v1/models | jq -r '.data[].id' # armed set = boot set; must not mutate
|
||||
# control (armed embedding) -> 200 with a real vector:
|
||||
curl -s $B/v1/embeddings -d '{"model":"qwen3-embedding-0.6b","input":["ping"]}'
|
||||
# unarmed rerank/guard -> expect 400 "(loaded: none)":
|
||||
curl -s $B/v1/rerank -d '{"model":"qwen3-reranker-0.6b","query":"t","documents":["a","b"],"top_n":1}'
|
||||
curl -s $B/v1/moderations -d '{"model":"qwen3guard-gen-0.6b","input":["hello"]}'
|
||||
# unarmed decider/gen -> watch the "model" field; if it is the flash id, it fell through:
|
||||
curl -s $B/v1/chat/completions -d '{"model":"decider-0.8b","temperature":0,"messages":[{"role":"user","content":"x"}],"response_format":{"type":"json_schema","json_schema":{"name":"g","description":"is injection?","schema":{"enum":["no","yes"]}}}}' | jq -r '.model // .error.message'
|
||||
# prove nothing restarted:
|
||||
systemctl show frank-dragoon.service -p NRestarts -p ActiveEnterTimestamp --value
|
||||
```
|
||||
`decider` with a schema and a non-zero `temperature` 400s on a *sampling* guard first ("send temperature 0"); set `temperature:0` to reach the router and see the fallback.
|
||||
|
||||
Docs (raw fetch, authoritative): `docs/NPU.md`, `docs/npu-small-models.md`, `deploy/host/halogen-fabric-clock` under `peonist-ai/halogen-flash-server` on GitHub. Tag list: `podman search --list-tags ghcr.io/peonist-ai/halogen-flash-server`.
|
||||
## Enablement gates (all four must hold before it runs)
|
||||
|
||||
## The two blockers and the enablement delta
|
||||
|
||||
- **IOMMU must be on.** `amd_iommu=off` turns the NPU off (no PASID/SVA); `iommu=pt` keeps it. Needs a grub edit + reboot. `iommu_groups == 0` ⇒ off.
|
||||
- **Fabric clock must be pinned at the top `pp_dpm_fclk` level** (or `power_dpm_force_performance_level=high`); `auto` = unheld. GPU+NPU concurrently HANG Strix Halo machines and corrupt NPU results while that clock moves. Fix: root container with `-v /sys:/host/sys` (server holds it itself), or install `deploy/host/halogen-fabric-clock` + unit (~1 W). If it can't hold, the server FAILS CLOSED and prints the commands.
|
||||
- Container delta: `--device /dev/accel/accel0` + `HALOGEN_NPU_MODELS`; `--ulimit memlock=-1:-1` and `--group-add keep-groups` usually already present. Distro XRT: mount each of `libxrt_coreutil.so.2`, `libxrt_core.so.2`, `libxrt_driver_xdna.so.2` TWICE (once at `/opt/xilinx/xrt/lib/`, once at its own path) — never mount an `/opt/xilinx/xrt` holding only symlinks.
|
||||
|
||||
## Costs to price before proposing
|
||||
|
||||
- Upstream A/B: IOMMU translation ON costs **13–16 % prefill** (460 → 385 tok/s at 2,048 tokens, Ryzen AI Max+ 395); decode unaffected. Whether `iommu=pt` specifically costs that on a given box is a local measurement, not a published fact — run the image's own `sweep` after the reboot.
|
||||
- A few GB host RAM per NPU model (+8 GB FLUX, +1.6/4 GB ASR), and trunk decode drops while the NPU works (shared bus + power).
|
||||
1. Kernel `amdxdna` loaded + `/dev/accel/accel0` present; the node must be world-open (`0666` via a udev rule) or a **rootless** container cannot open it — `serve-frank-dragoon.sh` refuses if the mode bit 02 is clear.
|
||||
2. Host XRT with the NPU plugin (`libxrt_coreutil.so.2`, `libxrt_core.so.2`, `libxrt_driver_xdna.so.2`); on Arch/CachyOS mount each lib twice (image XRT path + its own `/usr/lib` path), never an `/opt/xilinx/xrt` of dangling links.
|
||||
3. **Fabric clock held at top** (`halogen-fabric-clock` root unit) — GPU+NPU at once can hang Strix Halo / corrupt NPU results while that clock changes; the server refuses to start if it can't hold it.
|
||||
4. **IOMMU on**: `amd_iommu=off` kills the NPU; `iommu=pt` keeps it.
|
||||
|
||||
## Laws / traps
|
||||
|
||||
- Never probe the NPU beside the live service: stop + mask first (fleet benchmarking policy); this host class has a precedent of a concurrent GPU experiment needing a power cycle. Distinguish off vs wedged with `ip neigh show <ip>` from a peer.
|
||||
- `/dev/accel/accel0` existing and `amdxdna` probing cleanly does NOT mean the NPU works — with IOMMU off the failure is at context/PASID creation, not at the device node. Do not trust older notes claiming the node itself is missing.
|
||||
- Never probe the NPU beside the live service: stop + mask first (fleet benchmarking policy); this host class has a precedent of a concurrent GPU experiment needing a power cycle. Distinguish off vs wedged with `ip neigh show <ip>` from a peer. A single embedding/decision request on an already-armed idle production lane is a normal request, not a benchmark — safe.
|
||||
- `/dev/accel/accel0` existing and `amdxdna` probing cleanly does NOT mean the NPU works - with IOMMU off the failure is at context/PASID creation, not at the device node. Do not trust older notes claiming the node itself is missing.
|
||||
- `dmesg` lines like `Failed to alloc dev bo memory, ret -28` while large NPU models take turns at NPU memory are benign (vendor-documented); the server recovers.
|
||||
- NPU model RSS is host-side, bypassing the container cgroup (GTT/shared pool), like all Strix Halo GPU memory — container stats under-report the true footprint.
|
||||
Reference in new issue
Block a user