45 lines
3.9 KiB
Markdown
45 lines
3.9 KiB
Markdown
---
|
||
name: halogen-npu-xrt-third-prefix
|
||
description: "Triage a halogen frank-dragoon (flash-server) unit crash-loop caused by the NPU lane: signature \"No such library /opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2\", dead-lane-kills-container law, third-prefix mount fix, and the FRANK_DRAGOON_ENV_FILE scratch-boot proof recipe."
|
||
---
|
||
|
||
# Halogen NPU lane: dead lane kills the container; XRT needs THREE mount prefixes
|
||
|
||
Friction-earned 2026-10-21 on `frank-dragoon` (Dankclaw): unit boot-looped 88× (`activating`, exit 1 in ~200 ms) with the GPU/serve_api lines looking healthy; chat dead beside a fine-looking ledger.
|
||
|
||
## Signatures (in order of appearance)
|
||
|
||
```
|
||
halogen-npu: No such library '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'
|
||
halogen npu: the NPU engine exited before it was ready (rc=1; ...)
|
||
```
|
||
then `systemctl show <svc> -p ExecMainStatus` = 1, `NRestarts` climbing, and `/home/dank/logs/frank_dragoon.log` truncated at ~10 lines each boot (serve script truncates the log per start).
|
||
|
||
**Laws:** a dead NPU lane is NOT survivable — the engine supervisor takes the WHOLE container down on it. And `Restart=always` makes each restart race its own GPU/NPU debris: the FIRST failure's message is the only real evidence; later loop entries are noise. Stop the unit immediately (`sudo systemctl stop frank-dragoon`) before probing.
|
||
|
||
## Root cause class
|
||
|
||
The NPU child `dlopen`s XRT from `/opt/xilinx/lib/<multiarch>/` IN ADDITION to `/opt/xilinx/xrt/lib` and the install path. Host package churn (dpkg.log reconfigure of `libxrt2` the previous day) made the child demand the third prefix; our mounts filled only two. A lane that worked for days with byte-identical mounts can die purely from host package changes — diff the mounts last, check `grep libxrt /var/log/dpkg.log` first.
|
||
|
||
Fix: mount each REAL file (never a symlink) at all three prefixes; derive the multiarch name from `FRANK_DRAGOON_XRT_DIR`'s basename (`MULTIARCH="${XRT_DIR##*/}"`).
|
||
|
||
## Proof recipe (production-shaped, no scratch port)
|
||
|
||
One card holds one server — never bench on a side port; use the sanctioned env-file seam:
|
||
|
||
1. Scratch env on the host: `source /home/dank/serverAppConfigs/scripts/frank-dragoon.env` + knob overrides (e.g. `HALOGEN_NPU_MODELS=...`).
|
||
2. Copy the serve script with the candidate change applied via `sed` into `/tmp` (or edit + `FRANK_DRAGOON_ENV_FILE`); export `FRANK_DRAGOON_ENV_FILE=<scratch>` and `FRANK_DRAGOON_LOG=/tmp/<name>.log`; run under `timeout 1200`. The seam exists precisely for this.
|
||
3. Judge from `/tmp/<name>.log`: `npu_api: <model> ... buckets u0…u9` = lane alive; `No such library` = still broken. Then `curl :7770/health` → `npu.status`.
|
||
4. Retire the same session: `podman stop -t 60` (weights unpin slowly), `podman rm -f`, kill wrapper procs BY PID (`pgrep -f` the scratch serve script — `pkill -f` a pattern containing your own ssh line kills the session), move scratch files to `~/.AGENTTRASH/<date>-<topic>/`, then prove `pgrep -f serve-<scratch>` is empty. Leftover scratch state silently hijacks the next deploy (same class as leftover bench drop-ins).
|
||
|
||
## Traps hit en route
|
||
|
||
- Rootless `podman mount` fails ("cannot run in rootless mode") and leaves `$M` EMPTY — a later `grep -r "$M/..."` then reads the HOST cwd and "finds" the answer in Loki WAL files. Never grep an unverified mount var; `podman unshare` or `--entrypoint` probe instead.
|
||
- Image ships NO `/opt/xilinx` at all: all XRT comes from host mounts; `docs/NPU.md` is not inside the image.
|
||
- `ssh host 'tail <log>'` runs on the DEV box if you forgot `ssh` — a missing-file error is not "the engine wrote nothing".
|
||
- Health `context` field echoes the RUNNING engine's ceiling — use it to prove which shape is live, unlike `/v1/models` name echo.
|
||
|
||
## Verification that the fix landed
|
||
|
||
`systemctl show frank-dragoon -p DropInPaths` empty; boot ledger shows `reserving the KV pool … (about 11.0 GiB)`, `… in all`, an `npu_api:` line, no clamp line; deployer prints `NPU lane verified — <model> embedded`.
|