67 lines
3.8 KiB
Markdown
67 lines
3.8 KiB
Markdown
---
|
|
name: halogen-npu-mount-prefix-proof
|
|
description: "Prove and fix a halogen frank-dragoon NPU-lane mount failure: crash-loop where a dead lane kills the whole container, dlopen-prefix census, and the sanctioned FRANK_DRAGOON_ENV_FILE proof-boot."
|
|
---
|
|
|
|
# Halogen NPU lane: dead lane kills the container — prove the mount prefix, don't guess
|
|
|
|
Applies when a `frank-dragoon` unit crash-loops in seconds (`exit 1` in ~200 ms,
|
|
`NRestarts` climbing) and the engine log's last lines are:
|
|
|
|
```
|
|
halogen npu: <model>: 15 files checked against this image's record
|
|
halogen-npu: No such library '<some/prefix>/libxrt_core.so.2'
|
|
halogen npu: the NPU engine exited before it was ready (rc=1)
|
|
```
|
|
|
|
## Laws this incident wrote (2026-10-21, frank-dragoon, 88-restart loop)
|
|
- **A dead NPU lane is fatal to the whole container**: the flash-server supervisor
|
|
exits with the lane. Chat dies beside a healthy GPU preflight — do not read the
|
|
GTT/weights lines as "engine fine, side feature broken".
|
|
- The NPU child `dlopen`s distro XRT from **three** container prefixes per library:
|
|
`/opt/xilinx/xrt/lib/`, the library's own install path, **and
|
|
`/opt/xilinx/lib/<multiarch>/`**. The missing one is exactly what the error names.
|
|
- Host state can flip this under a byte-identical container: the lane ran for days
|
|
with two prefixes, then a host `libxrt2`/`libxrt-npu2` reconfigure (check
|
|
`grep libxrt /var/log/dpkg.log`) made the child demand the third. Image digest,
|
|
serve script, payload all unchanged ⇒ suspect HOST package state, not the deploy.
|
|
- Fastest discriminator vs engine faults: exit in <1 s with the engine's preflight
|
|
lines present = serve/lane gate failure, not model load. Model-load failures take
|
|
minutes and print pin/KV lines.
|
|
|
|
## Proof procedure (sanctioned seam, no scratch ports)
|
|
1. Read the error's path verbatim — it names the missing prefix.
|
|
2. `sudo systemctl stop frank-dragoon` (unit must be down: the proof binds :7770
|
|
at container CREATE).
|
|
3. Scratch env file on the host using the bench-sanctioned seam:
|
|
```
|
|
/tmp/dragoon-nputest.env:
|
|
source /home/dank/serverAppConfigs/scripts/frank-dragoon.env
|
|
HALOGEN_NPU_MODELS=qwen3-embedding-0.6b
|
|
```
|
|
4. Scratch serve copy that appends the candidate mount to the `NPU_ARGS+=(--device`
|
|
line (`sed`), REAL files only (`readlink -f`; vendor forbids dangling symlinks
|
|
under /opt/xilinx), and run with:
|
|
`FRANK_DRAGOON_ENV_FILE=/tmp/... FRANK_DRAGOON_LOG=/tmp/... timeout 1200 /tmp/serve-nputest.sh`
|
|
5. Success = boot ledger (`… in all`) **plus** `npu_api: <model> … buckets u0…u9`
|
|
plus `/health` `npu.status ok`. Then codify the mount in the serve script, pin
|
|
it in `tests/frank_dragoon_unit.bats` (the fixture XRT dir's basename IS the
|
|
multiarch name — name it `x86_64-linux-gnu`), update the endpoint page's NPU
|
|
prerequisite to the three-prefix law, and cut over via PR +
|
|
`deploy.sh dankclaw --service frank-dragoon`.
|
|
6. Retire scratch: the `timeout` kills the proof container; `podman rm -f`,
|
|
move /tmp scripts to `~/.AGENTTRASH/<date>-<topic>/`. A proof instance
|
|
serving production while you wait is drift — deploy the merged fix promptly.
|
|
|
|
## Traps
|
|
- Greping image contents rootless: `podman mount` fails (needs `podman unshare`)
|
|
and leaves `$M` EMPTY, so `grep -r $M/opt` silently greps the HOST cwd — that
|
|
is how host `/opt/fastflowlm` masqueraded as image content. Always check `$M`
|
|
non-empty, or use `podman unshare bash -c 'M=$(podman mount C); …'`.
|
|
- The entrypoint swallows `podman run IMG sh -c …` as a mode name; use
|
|
`--entrypoint sh/cat/ls`.
|
|
- The proof container inherits production name/port unless the override is set
|
|
AFTER sourcing the env (the env file re-exports `FRANK_DRAGOON_CONTAINER`).
|
|
- `Restart=always` re-runs the lane into its own GPU/NPU debris; stop the unit
|
|
before any manual boot, don't fight the loop.
|