The armed boot reached the engine and died in its own preflight:
halogen npu: /dev/accel/accel0 is here, but this process cannot open it
(crw-rw---- nobody:nogroup).
The node is root:render 0660 and a rootless container maps exactly one uid/gid,
so `render` is unmapped inside it and `--group-add keep-groups` — the vendor's
advice, and the ComfyUI pattern for /dev/kfd — cannot grant a group the
namespace does not map. Measured 2026-10-19 from a `dank` shell already in
`render`: without the flag denied, with the flag denied, with the udev mode
0666 the open succeeds (OPEN_OK). Same wall 99-comfyui-gpu-access.rules already
solved for the GPU compute nodes, so the accel node gets its counterpart:
dankclaw/udev/98-halogen-npu-access.rules — KERNEL=="accel[0-9]*",
SUBSYSTEM=="accel", MODE="0666"
The deployer installs the rule, reloads rules, re-triggers the node and only
then checks the mode; the serve script refuses any node that is not
other-writable, naming the rule, so this cannot return as a crash loop (189
restarts while chat was down is the failure mode being closed).
Both checks read the permission bit directly: `[ -o FILE ]` is not
"other-writable" — it means "exists" and `-o` doubles as the OR operator inside
test, which is why the first cut of the gate refused everything.
Dragoon seam: 33 -> 35 tests. The fixture node is 0666 like the installed one,
a 0660 node is refused with the rule named and nothing rendered, and the rule
file's shape plus the deployer's install/trigger are asserted.
The containment gate added in #44 parsed scripts/frank-dragoon.env through a
$CHECKOUT variable that does not exist in deploy-remote.sh, so the very next
deploy died at `set -u` before it touched the unit:
./scripts/deploy-remote.sh: line 1445: CHECKOUT: unbound variable
deploy-remote.sh runs from the checkout root and every other path in that block
is a parity-twin default (LLM_DRAGOON_CKPT/_VISION/_TOK_DIR), so the payload dir
gets the same treatment: LLM_DRAGOON_MODEL_DIR, defaulted to the deployed path,
compared directly. The seam now asserts that twin against the env file next to
the NPU twins it already checks, and the containment comparison itself is
verified both ways (inside accepted; the old outside path and a shared-prefix
sibling both refused).
The lane's first boot died at container init, not at the engine:
runc create failed: ... create mountpoint for /models/npu mount:
mkdirat .../merged/models/npu: read-only file system
The image ships no /models, and the serve script binds the payload dir
there read-only, so a nested `-v $NPU_DIR:/models/npu:ro` had to create
its own mountpoint underneath a read-only parent — impossible. The
engine's own entrypoint confirms the container path is fixed: it rejects
HALOGEN_NPU_DIR outright ("our NPU models live under /models/npu/<id>/").
Fix is the mount shape, not the engine: FRANK_DRAGOON_NPU_DIR now lives
inside FRANK_DRAGOON_MODEL_DIR, so the payload's single read-only mount
puts the files at /models/npu/<id>/ by itself and the nested bind is
gone. Verified on the running container: after the move, `podman exec
frank-dragoon-halogen ls /models/npu/qwen3-embedding-0.6b` answers with
the staged tree while the lane is still armed off.
Both gates refuse the old shape by name (serve + deployer parity) so a
future payload swap that moves the model dir cannot reintroduce a crash
that lands after the deployer has already stopped the running unit.
An endpoint on podman's rootless default (pasta) has no address another
container can use, measured on 2026-10-19 from inside open-notebook:
- 192.168.0.143:7770 connection refused (pasta forwards the
published port in the host netns; no netavark
DNAT exists for traffic arriving from a
rootless bridge)
- host.containers.internal:7770 200, but the alias resolves to 169.254.1.2 —
link-local, which open-notebook's URL guard
refuses to store
- frank-dragoon-halogen:8731 name unknown: different netns
- --network pasta --network dankclaw_default
"cannot set multiple networks without bridge
network mode" — pasta cannot be added to
So reaching the NPU endpoints from an app container means moving the container
onto dankclaw_default (FRANK_DRAGOON_NETWORK; empty keeps today's shape). This
is not an exposure change: the engine's unauthenticated token port binds the
container's own loopback (/proc/net/tcp: 127.0.0.1:8730 against 0.0.0.0:8731),
so only the API port becomes name-resolvable — the same trust the published
7770 already carries on the LAN. The deployer refuses a network that does not
exist rather than starting a unit that restart-loops on it.
Co-Authored-By: omp <omp@dankclaw>
halogen-flash-server already ships an NPU path on the pin we run: for each
model named in HALOGEN_NPU_MODELS it loads the vendor's precompiled AIE
programs (XDNA2 executes only AIE-compiled graphs) and answers them on
/v1/embeddings inside the SAME container behind port 7770. That gives the
fleet an in-house embedding endpoint without touching the iGPU trunk.
The lane is opt-in and one line wide: HALOGEN_NPU_MODELS blank is the
pre-lane endpoint byte for byte (no device, no mounts, no engine env).
The three host preconditions the vendor names are gated in the serve script
and mirrored in the deployer (the 2026-10-01 twin law), each refused by name:
- the staged model files. /models is read-only and HALOGEN_DOWNLOAD stays
unset so the container opens no outbound connection at boot, so
scripts/download-frank-dragoon-npu.sh stages them from the pinned
revision with the catalog's own byte counts and sha256s (the engine
re-checks both at boot);
- the distro XRT's libxrt_coreutil/libxrt_core/libxrt_driver_xdna, each
mounted twice — where the image looks for XRT and at its own path —
because those libraries find each other where they were installed and
an /opt/xilinx/xrt of symlinks dangles inside the container;
- the GPU fabric clock HELD. On Strix Halo the iGPU and the NPU co-running
hang the machine (and corrupt NPU results) while FCLK changes speed, and
halogen refuses to start in that state; it cannot read host sysfs from
inside, so the hold is proven host-side. dankclaw/host/ carries the
vendor's halogen-fabric-clock{,.service} verbatim at tag v0.16.2 (sha256
a3a4f7f4e0142e71… / 5f9f1cc4c85a1c02…); the deployer installs, enables
and gates it only while the lane is named, and releases the clock plus
drops the boot hook when it is not.
After health, the deployer's lane gate is a real embedding through
/v1/embeddings — a 200 on /health proves nothing about the NPU programs.
Planner: dankclaw/host/halogen-fabric-clock* re-dispatches frank-dragoon
(the deployer installs it); dankclaw/host/grub and the fetch script plan
nothing.
Seam: 11 new tests — the off arm really renders nothing lane-specific, the
on arm renders the device/mounts/env, every missing precondition refuses by
name, the deployer twins and the host unit shape are pinned.
Co-Authored-By: omp <omp@dankclaw>
frank-dragoon's total in-flight context was HALF frank-zealot's: zealot runs
one 401,408-position pool split into two 200,704 conversations, while dragoon
ran ONE 200,704-position pool behind its two slots, so a single maximum-length
request filled it and two long conversations had to stay under ~100k each.
HALOGEN_KV_POOL_POSITIONS 200704 -> 401408 = slots x ctx, the top of the seam's
legal band (ctx <= pool <= slots * ctx). The PER-REQUEST ceiling does not move:
HALOGEN_CTX stays 200704, because the checkpoint's native window is 262,144 and
the engine's own 400 text refuses to exceed it without HALOGEN_ROPE_YARN. What
this buys is concurrency, not a longer conversation, which is why the harness
contextWindow (155648 = ceiling - 45056) is untouched.
Cost: ~10.8 GiB of KV where 200,704 cost 5.5 (262,144 cost 7.2; the
2026-10-06 shape's 524,288 measured 14.6), taken from the page cache the 51.2 GB
n-gram table reads through — the warm prompt-cache ratio is the number to
re-measure after the boot. Engine-side total goes 74.1 -> ~79.4 GiB of 121.2.
Also carries scripts/download-frank-zealot-bonsai2-ptq1.sh: the PTQ1_0 candidate
payload fetcher (5,946,648,928 B, LFS sha256-verified, mmproj hardlinked), with
the publisher's own throughput table in its header — the packing is smaller but
the card measures it SLOWER on Ampere, so the swap is decided on our card, not
on the card's A100 row.
Frank's call the same evening as the native-payload cutover: HALOGEN_CTX and
HALOGEN_KV_POOL_POSITIONS both 262144 -> 200704 (a multiple of 4096, the rung
frank-zealot already runs). The checkpoint's native window is 262,144 and the
engine serves anything below it, so this is a RAM decision, not a capability
limit: the KV reservation drops 7.2 GiB -> ~5.4 (the 2026-10-06 pool measured
14.6), and by the FLAGS.md law -- RAM the pool takes is RAM that n-gram table
loses -- the freed positions become page cache for the 47.68 GiB table that
long prompts read. Engine-side total ~74 GiB instead of 75.8.
The 400-zone law is invariant to the value, so the harness moves WITH the
ceiling: models.yml contextWindow 217088 -> 155648 = ceiling - 45056, keeping
the gap that absorbs the chars/4 under-count (worst-case request ~156.2k vs
the 200,704 ceiling, ~44k margin). Compaction now fires at 85% = 132,300 est
(~136k measured); the 180k target of the 262,144 era is unreachable because
the window itself is 155,648. maxTokens 45056 and the coupled
cap/timeout 65536/7200 are untouched.
Cost stated plainly: with pool == ctx, ONE maximum-length request fills the
pool, so two long conversations never coexist (two of <=100k do). Acceptable
because frank-dragoon carries only the slow role and the security-reviewer
subagent, not the interactive workhorse.
Seam re-keyed: the fit-law test now proves the invariant at the new pair, and
the pool law (ctx <= pool <= slots * ctx) still holds with pool == ctx.
Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated is now the NATIVE
2026-10-11 v2 conversion of orcarouter's pinned BF16 (66,687,678,432 B,
LFS sha256 b5bd925f...) — 32 bytes smaller than the writer transplant it
used to ship under the same filename. The n-gram table, vision tower and
tokenizer are the stock peonist companions, hardlinked out of the swift15
payload dir: the release publishes only the main file and no source-matched
pair exists, so the engine's own tools are the referee (niah 18/18, ppl
3.968 with mean KL 0.155 against the swift15 logits, 2026-10-17).
Memory saving, per the card: HALOGEN_KV_POOL_POSITIONS 524288 -> 262144,
one pool behind both slots (boot ledger: 7.2 GiB, where the old pool
measured 14.6). HALOGEN_CTX stays 262144 so the per-request ceiling and the
harness contract (contextWindow 217088) do not move; only two MAXIMUM-length
conversations lose the ability to coexist.
Measured against the swift15 boot on identical traffic: cold prefill
+12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking
tokens x1.9 (the finetune's reasoning compression is gone), refusals 0/5 on
both.
The 2026-10-09 wire aliases become the service names everywhere:
systemd units, serve scripts, env files and FRANK_* variables,
deploy-plan/deploy-remote service keys, container names, Prometheus
job names + alert rules, Grafana dashboard queries, bats seams,
runbook filenames, homepage/AGENTS lore, and historical prose.
Rollback tags renamed in lockstep. Residue in the tree: zero.
Retired frank-medium/small/npu names are history and stay.
Known casualty: scripts/patches/*.patch context lines were rewritten;
those retired-era vLLM-fork patches no longer apply byte-exact.
Frank
