live: heartbeat sync 2026-10-21T03:35:51-07:00 (Refs #11)

agent/managed-skills/halogen-npu-xrt-third-prefix/SKILL.md

@@ -0,0 +1,44 @@
+---
+name: halogen-npu-xrt-third-prefix
+description: "Triage a halogen frank-dragoon (flash-server) unit crash-loop caused by the NPU lane: signature \"No such library /opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2\", dead-lane-kills-container law, third-prefix mount fix, and the FRANK_DRAGOON_ENV_FILE scratch-boot proof recipe."
+---
+
+# Halogen NPU lane: dead lane kills the container; XRT needs THREE mount prefixes
+
+Friction-earned 2026-10-21 on `frank-dragoon` (Dankclaw): unit boot-looped 88× (`activating`, exit 1 in ~200 ms) with the GPU/serve_api lines looking healthy; chat dead beside a fine-looking ledger.
+
+## Signatures (in order of appearance)
+
+```
+halogen-npu: No such library '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'
+halogen npu: the NPU engine exited before it was ready (rc=1; ...)
+```
+then `systemctl show <svc> -p ExecMainStatus` = 1, `NRestarts` climbing, and `/home/dank/logs/frank_dragoon.log` truncated at ~10 lines each boot (serve script truncates the log per start).
+
+**Laws:** a dead NPU lane is NOT survivable — the engine supervisor takes the WHOLE container down on it. And `Restart=always` makes each restart race its own GPU/NPU debris: the FIRST failure's message is the only real evidence; later loop entries are noise. Stop the unit immediately (`sudo systemctl stop frank-dragoon`) before probing.
+
+## Root cause class
+
+The NPU child `dlopen`s XRT from `/opt/xilinx/lib/<multiarch>/` IN ADDITION to `/opt/xilinx/xrt/lib` and the install path. Host package churn (dpkg.log reconfigure of `libxrt2` the previous day) made the child demand the third prefix; our mounts filled only two. A lane that worked for days with byte-identical mounts can die purely from host package changes — diff the mounts last, check `grep libxrt /var/log/dpkg.log` first.
+
+Fix: mount each REAL file (never a symlink) at all three prefixes; derive the multiarch name from `FRANK_DRAGOON_XRT_DIR`'s basename (`MULTIARCH="${XRT_DIR##*/}"`).
+
+## Proof recipe (production-shaped, no scratch port)
+
+One card holds one server — never bench on a side port; use the sanctioned env-file seam:
+
+1. Scratch env on the host: `source /home/dank/serverAppConfigs/scripts/frank-dragoon.env` + knob overrides (e.g. `HALOGEN_NPU_MODELS=...`).
+2. Copy the serve script with the candidate change applied via `sed` into `/tmp` (or edit + `FRANK_DRAGOON_ENV_FILE`); export `FRANK_DRAGOON_ENV_FILE=<scratch>` and `FRANK_DRAGOON_LOG=/tmp/<name>.log`; run under `timeout 1200`. The seam exists precisely for this.
+3. Judge from `/tmp/<name>.log`: `npu_api: <model> ... buckets u0…u9` = lane alive; `No such library` = still broken. Then `curl :7770/health` → `npu.status`.
+4. Retire the same session: `podman stop -t 60` (weights unpin slowly), `podman rm -f`, kill wrapper procs BY PID (`pgrep -f` the scratch serve script — `pkill -f` a pattern containing your own ssh line kills the session), move scratch files to `~/.AGENTTRASH/<date>-<topic>/`, then prove `pgrep -f serve-<scratch>` is empty. Leftover scratch state silently hijacks the next deploy (same class as leftover bench drop-ins).
+
+## Traps hit en route
+
+- Rootless `podman mount` fails ("cannot run in rootless mode") and leaves `$M` EMPTY — a later `grep -r "$M/..."` then reads the HOST cwd and "finds" the answer in Loki WAL files. Never grep an unverified mount var; `podman unshare` or `--entrypoint` probe instead.
+- Image ships NO `/opt/xilinx` at all: all XRT comes from host mounts; `docs/NPU.md` is not inside the image.
+- `ssh host 'tail <log>'` runs on the DEV box if you forgot `ssh` — a missing-file error is not "the engine wrote nothing".
+- Health `context` field echoes the RUNNING engine's ceiling — use it to prove which shape is live, unlike `/v1/models` name echo.
+
+## Verification that the fix landed
+
+`systemctl show frank-dragoon -p DropInPaths` empty; boot ledger shows `reserving the KV pool … (about 11.0 GiB)`, `… in all`, an `npu_api:` line, no clamp line; deployer prints `NPU lane verified — <model> embedded`.