fix(frank-dragoon): stage the NPU lane inside the payload mount #44

Merged
dank merged 2 commits from feat/frank-dragoon-npu-lane into main 2026-10-19 21:46:25 -07:00
Owner

The problem

The lane's first real boot never reached the engine. frank-dragoon crash-looped 52 times with:

runc create failed: ... create mountpoint for /models/npu mount:
mkdirat .../merged/models/npu: read-only file system

#43 mounted the staged NPU tree as a nested bind (-v $NPU_DIR:/models/npu:ro) inside the payload's read-only /models bind. The image ships no /models at all and the container path is not movable — the shipped entrypoint.sh rejects HALOGEN_NPU_DIR outright ("our NPU models live under /models/npu/<id>/"). So runc had to mkdir the mountpoint through a read-only parent, which cannot succeed. Production chat was down for the window; the endpoint was rolled back with the sanctioned FRANK_DRAGOON_ENV_FILE arm drop-in (lane off, chat healthy, health 200) and that drop-in is retired the same session the fix deploys.

The fix

Mount shape only — nothing about the engine changes:

  • FRANK_DRAGOON_NPU_DIR moves inside the payload dir: /home/dank/models/halogen-v2-abliterated/npu. The existing read-only payload mount then delivers the files at /models/npu/<id>/ by itself; the nested bind line is deleted.
  • The serve script and the deployer both refuse a staging dir outside the payload dir by name, before anything renders — the deployer must not stop the running unit and then hand the fault to runc.
  • download-frank-dragoon-npu.sh reads the staging path from frank-dragoon.env instead of repeating a literal, so the env file owns it.

Measured on the running container, lane still armed off:

$ mv ~/models/halogen-npu ~/models/halogen-v2-abliterated/npu
$ podman exec frank-dragoon-halogen ls /models/npu/qwen3-embedding-0.6b
devices  qwen3-embedding-0.6b.hnpw  tokenizer

Tests

tests/frank_dragoon_unit.bats 33/33, including a new arm:

  • lane-armed render asserts -v $MODEL_DIR:/models:ro and no :/models/npu bind;
  • the lane refuses a staging dir outside the payload dir and says why (nothing renders, no --device);
  • deployer parity grep for the same refusal.

Full ./tests/run.sh run before merge; numbers land in the vault's frank-dragoon page with the enablement, not here.


cwd: /home/blank/.omp/wt/wt-20261019-020309-6d1fc1f · session: 01a15366-7601-7175-bb69-c62186c3f8b5
(build worktree: /home/blank/.omp/wt/wt-dragoon-npu)

# The problem The lane's first real boot never reached the engine. `frank-dragoon` crash-looped 52 times with: ``` runc create failed: ... create mountpoint for /models/npu mount: mkdirat .../merged/models/npu: read-only file system ``` `#43` mounted the staged NPU tree as a **nested** bind (`-v $NPU_DIR:/models/npu:ro`) inside the payload's read-only `/models` bind. The image ships no `/models` at all and the container path is not movable — the shipped `entrypoint.sh` rejects `HALOGEN_NPU_DIR` outright ("our NPU models live under `/models/npu/<id>/`"). So runc had to `mkdir` the mountpoint through a read-only parent, which cannot succeed. Production chat was down for the window; the endpoint was rolled back with the sanctioned `FRANK_DRAGOON_ENV_FILE` arm drop-in (lane off, chat healthy, `health 200`) and that drop-in is retired the same session the fix deploys. # The fix Mount shape only — nothing about the engine changes: - `FRANK_DRAGOON_NPU_DIR` moves **inside** the payload dir: `/home/dank/models/halogen-v2-abliterated/npu`. The existing read-only payload mount then delivers the files at `/models/npu/<id>/` by itself; the nested bind line is deleted. - The serve script and the deployer both **refuse a staging dir outside the payload dir by name**, before anything renders — the deployer must not stop the running unit and then hand the fault to runc. - `download-frank-dragoon-npu.sh` reads the staging path from `frank-dragoon.env` instead of repeating a literal, so the env file owns it. Measured on the **running** container, lane still armed off: ``` $ mv ~/models/halogen-npu ~/models/halogen-v2-abliterated/npu $ podman exec frank-dragoon-halogen ls /models/npu/qwen3-embedding-0.6b devices qwen3-embedding-0.6b.hnpw tokenizer ``` # Tests `tests/frank_dragoon_unit.bats` 33/33, including a new arm: - lane-armed render asserts `-v $MODEL_DIR:/models:ro` and **no** `:/models/npu` bind; - `the lane refuses a staging dir outside the payload dir and says why` (nothing renders, no `--device`); - deployer parity grep for the same refusal. Full `./tests/run.sh` run before merge; numbers land in the vault's `frank-dragoon` page with the enablement, not here. --- cwd: /home/blank/.omp/wt/wt-20261019-020309-6d1fc1f · session: 01a15366-7601-7175-bb69-c62186c3f8b5 (build worktree: /home/blank/.omp/wt/wt-dragoon-npu)
dank added 1 commit 2026-10-19 21:42:49 -07:00
The lane's first boot died at container init, not at the engine:

  runc create failed: ... create mountpoint for /models/npu mount:
  mkdirat .../merged/models/npu: read-only file system

The image ships no /models, and the serve script binds the payload dir
there read-only, so a nested `-v $NPU_DIR:/models/npu:ro` had to create
its own mountpoint underneath a read-only parent — impossible. The
engine's own entrypoint confirms the container path is fixed: it rejects
HALOGEN_NPU_DIR outright ("our NPU models live under /models/npu/<id>/").

Fix is the mount shape, not the engine: FRANK_DRAGOON_NPU_DIR now lives
inside FRANK_DRAGOON_MODEL_DIR, so the payload's single read-only mount
puts the files at /models/npu/<id>/ by itself and the nested bind is
gone. Verified on the running container: after the move, `podman exec
frank-dragoon-halogen ls /models/npu/qwen3-embedding-0.6b` answers with
the staged tree while the lane is still armed off.

Both gates refuse the old shape by name (serve + deployer parity) so a
future payload swap that moves the model dir cannot reintroduce a crash
that lands after the deployer has already stopped the running unit.
dank added 1 commit 2026-10-19 21:42:49 -07:00
The lane's first boot died at container init, not at the engine:

  runc create failed: ... create mountpoint for /models/npu mount:
  mkdirat .../merged/models/npu: read-only file system

The image ships no /models, and the serve script binds the payload dir
there read-only, so a nested `-v $NPU_DIR:/models/npu:ro` had to create
its own mountpoint underneath a read-only parent — impossible. The
engine's own entrypoint confirms the container path is fixed: it rejects
HALOGEN_NPU_DIR outright ("our NPU models live under /models/npu/<id>/").

Fix is the mount shape, not the engine: FRANK_DRAGOON_NPU_DIR now lives
inside FRANK_DRAGOON_MODEL_DIR, so the payload's single read-only mount
puts the files at /models/npu/<id>/ by itself and the nested bind is
gone. Verified on the running container: after the move, `podman exec
frank-dragoon-halogen ls /models/npu/qwen3-embedding-0.6b` answers with
the staged tree while the lane is still armed off.

Both gates refuse the old shape by name (serve + deployer parity) so a
future payload swap that moves the model dir cannot reintroduce a crash
that lands after the deployer has already stopped the running unit.
dank added 1 commit 2026-10-19 21:44:40 -07:00
frank-dragoon's lane section: the staging dir must live inside the payload dir
(the image ships no /models, so a nested :ro bind has no mountpoint to land on),
the entrypoint's rejection of HALOGEN_NPU_DIR, and the fabric-clock paragraph
corrected — the vendor's root host unit is installed and reports `held (manual)`
as of today, so "nothing is held" is retired along with the hand-write it
replaced.

dankclaw-iommu-translation-off: the deferred `pp_dpm_fclk` pin is no longer
deferred; the fact is owned by [[frank-dragoon]] and this page links it.
dank merged commit 823fd52d0e into main 2026-10-19 21:46:25 -07:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dank/serverAppConfigs#44