feat(llm): swap frank-large to halogen-flash-server, retire frank-npu
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b .hgn 115.55 GiB + quality overlay). Upstream measures this engine at roughly 4x the prefill throughput of the llama.cpp family on the same gfx1151 silicon, and prefill dominates any prompt with real context. Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/), engine on container loopback :8730 (never published), OpenAI front-end :8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB, conservative for a shared 121 GiB host), 4 slots, host reserve 24, vision tower on, weights pinned, all other flags at vendor defaults. Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable /home/dank/models/halogen. The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/ rather than deleted; the STRIX payload and the ROCmFPX build stay on disk as the rollback target. deploy_llm_large becomes a user-quadlet handler: it retires the old ROOT unit (moving its checkout symlink into the home trash, never rm) and gates the first start on the download rather than reporting a failure for a fetch that is working exactly as designed. frank-npu is retired in the same change. The kernel cmdline moves from iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured, and that takes the XDNA2 NPU with it: without PASID/SVA the driver cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are repointed (Hermes vision -> frank-large, Hermes title generation -> frank-small, harness vision -> frank-large, tiny -> frank-small, task -> frank-large:medium) and its provider blocks are removed from both the harness config and Hermes's live config. Audio input leaves the fleet with no replacement; image input survives on halogen's vision tower. Docs: new halogen-flash-server runbook, new IOMMU decision record, and frank-npu rewritten as a retirement record. AGENTS.md topology, runtime sections, benchmarking policy (frank-large is a user unit now) and the harness model-config section are corrected.
This commit is contained in:
Frank
@@ -27,13 +27,13 @@ prefix, reorganized in follow-up commits). The retired target checkouts
|
||||
archived. All services now run from `/home/dank/serverAppConfigs` on each
|
||||
host.
|
||||
|
||||
### LLM fleet topology (2026-08-30)
|
||||
### LLM fleet topology (2026-09-11)
|
||||
|
||||
| Target | Runtime | Service | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | FreeToken v0.1.2 | frank-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 |
|
||||
| Dankclaw | kingjones30/ROCmFPX llama.cpp (card-pinned fork) | frank-large | kingjones777 Qwen3.8-Flash-Next-ROCmFP4-STRIX (113.47 GiB single GGUF, Q4_0_ROCMFP4_STRIX, PLE streams via mmap), 401408 ctx / 2 slots, fa on, no spec decode, reasoning on | 7770 |
|
||||
| Dankclaw | FastFlowLM (FLM) v1.0.1 (`flm serve`, dpkg `fastflowlm`) on the XDNA2 NPU | frank-npu | Gemma4-E4B-IT-NPU2 pack (vision + audio towers); whole local registry served on one port, per-request routing by exact FLM tag; `iommu=pt` boot arg mandatory (SVA bind needs enabled IOMMU) | 7771 |
|
||||
| Dankclaw | halogen-flash-server 0.5.8 (rootless Quadlet container) | frank-large | peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 4 slots, pool 262144, vision tower on, greedy for silent clients) | 7770 |
|
||||
| Dankclaw | *(retired 2026-09-11)* FastFlowLM v1.0.1 on the XDNA2 NPU | frank-npu | Gemma4-E4B-IT-NPU2 pack (vision + audio). Retired with the `amd_iommu=off` decision: the driver's SVA/PASID bind needs an enabled IOMMU | 7771 |
|
||||
|
||||
History: the 2026-08-17 redo retired the pipeline-parallel split and the
|
||||
FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on
|
||||
@@ -52,8 +52,13 @@ Qwen3.6-35B-A3B-FP8. On 2026-08-30 both services swapped again:
|
||||
Dankclaw to the kingjones STRIX model on the card's pinned
|
||||
kingjones30/ROCmFPX runtime, Dankserver to the Huihui abliterated
|
||||
FP8 checkpoint on FreeToken. On 2026-08-30 the NPU endpoint returned as
|
||||
`frank-npu` — FLM v1.0.1 on XDNA2 serving sight+sound (see the
|
||||
Runtime & Troubleshooting section and its runbook). Runbooks:
|
||||
`frank-npu` — FLM v1.0.1 on XDNA2 serving sight+sound. On 2026-09-11
|
||||
Dankclaw's large service swapped runtime *and* model: the kingjones STRIX
|
||||
llama.cpp deployment gave way to halogen-flash-server 0.5.8 serving
|
||||
peonist-ai/halogen-qwen3.8-flash-next, now a rootless Quadlet container, and
|
||||
the kernel cmdline moved from `iommu=pt` to `amd_iommu=off` for 13–16 % of
|
||||
prefill — which retired `frank-npu` for good (see the Runtime &
|
||||
Troubleshooting section and its runbook). Runbooks:
|
||||
[serverAppConfigs-docs/LLM/index.md](serverAppConfigs-docs/LLM/index.md).
|
||||
Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`.
|
||||
|
||||
@@ -128,8 +133,8 @@ selected service and records its successful revision outside the checkout.
|
||||
|
||||
A documentation-only change is a no-op. The LLM services are selected only by
|
||||
their own unit/env/serve files or the shared chat template, so unrelated
|
||||
container changes cannot interrupt inference. `frank-npu` is relink-only
|
||||
when explicitly selected.
|
||||
container changes cannot interrupt inference. `frank-large` is selected by its
|
||||
Quadlet file under `dankclaw/frank-large/quadlet/`.
|
||||
|
||||
Set `FRANK_SUDO_PW` for headless runs on a host without passwordless sudo
|
||||
(remote sudo reads it from stdin via `sudo -S`). The value is never stored
|
||||
@@ -280,92 +285,119 @@ ALWAYS bench using llama-bench for llama.cpp or vllm bench for vllm runtimes.
|
||||
DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE OFFICIAL BENCHES. ASK THE USER BEFORE DEVIATING FROM THIS DIRECTIVE.
|
||||
|
||||
1. Stop the service before any benchmark:
|
||||
`sudo systemctl stop frank-large` (Dankclaw) or
|
||||
`sudo systemctl stop frank-small` (Dankserver).
|
||||
`systemctl --user stop frank-large` (Dankclaw — a **rootless** Quadlet
|
||||
service, so drop the `sudo`) or `sudo systemctl stop frank-small`
|
||||
(Dankserver — still a system unit).
|
||||
Then mask it to prevent the self-healing unit from restarting:
|
||||
`sudo systemctl mask frank-large` (Dankclaw).
|
||||
**CRITICAL WARNING:** Do NOT use `systemctl disable` on these service units.
|
||||
`systemctl --user mask frank-large` (Dankclaw) or
|
||||
`sudo systemctl mask frank-small` (Dankserver).
|
||||
**CRITICAL WARNING:** Do NOT use `systemctl disable` on the **system** units.
|
||||
On these hosts, `systemctl disable` doesn't just remove symlinks — it **deletes
|
||||
the unit file itself** from `/etc/systemd/system/`. You'd have to manually
|
||||
copy the unit file back from this repo, run `daemon-reload`, then enable and
|
||||
start. Mask (`systemctl mask`) creates a symlink to `/dev/null` instead, which
|
||||
is safe and reversible.
|
||||
2. Run the bench (`llama-bench` for llama.cpp / vllm bench for vLLM).
|
||||
Scratch ports are not used for tests or benches on the LLM hosts — a
|
||||
test process cannot share the single GPU with the running service anyway
|
||||
(the model + KV fill the card), so changes are validated directly against
|
||||
the service itself.
|
||||
2. Run the bench with the runtime's own harness: `llama-bench` for llama.cpp,
|
||||
`vllm bench` for vLLM, and for frank-large's halogen-flash-server the
|
||||
image's own `bench`/`sweep` modes (`bench [drafters] [max_tokens] [effort]
|
||||
[reps]`) — `llama-bench` and `vllm bench` do not apply to it. Scratch ports are not used for tests or benches on the LLM
|
||||
hosts — a test process cannot share the single GPU with the running service
|
||||
anyway (the model + KV fill the card), so changes are validated directly
|
||||
against the service itself.
|
||||
3. Restart the service when the bench completes: unmask, then start:
|
||||
`sudo systemctl unmask frank-large && sudo systemctl start frank-large` (Dankclaw).
|
||||
`systemctl --user unmask frank-large && systemctl --user start frank-large`
|
||||
(Dankclaw) or `sudo systemctl unmask frank-small && sudo systemctl start
|
||||
frank-small` (Dankserver).
|
||||
4. Verify health before reporting:
|
||||
`curl http://192.168.0.235:7770/health` or
|
||||
`curl http://192.168.0.143:7770/health`.
|
||||
|
||||
## Pi Model Config (`~/.pi/agent/models.json`)
|
||||
## Model Config (harness)
|
||||
|
||||
The Pi agent reaches the fleet through `/home/dank/.pi/agent/models.json`.
|
||||
This file lives outside the repository. Each provider is one host endpoint.
|
||||
Each model `id` is the served alias. The two aliases are
|
||||
`frank-small` (Dankserver `:7770`) and `frank-large`
|
||||
(Dankclaw `:7770`).
|
||||
The harness reaches the fleet through `/home/dank/.omp/agent/models.yml` —
|
||||
verified live 2026-09-11; the older `/home/dank/.pi/agent/models.json` path
|
||||
this section used to name no longer exists. Both files live outside the
|
||||
repository. Each provider is one host endpoint and each model `id` is the
|
||||
served alias:
|
||||
|
||||
The model `id` values are stable. The file
|
||||
`/home/dank/.pi/agent/settings.json` references them. Two resolution rules
|
||||
apply (verified in `@earendil-works/pi-coding-agent` `model-resolver.js`):
|
||||
| alias | endpoint |
|
||||
|---|---|
|
||||
| `frank-small` | Dankserver `192.168.0.235:7770` |
|
||||
| `frank-large` | Dankclaw `192.168.0.143:7770` |
|
||||
| `gemma4-it:e4b` | *(removed 2026-09-11 with `frank-npu`)* |
|
||||
|
||||
Two resolution rules apply (verified in `@earendil-works/pi-coding-agent`
|
||||
`model-resolver.js`):
|
||||
|
||||
- The default model resolves as a `defaultProvider` + `defaultModel` pair.
|
||||
The `defaultProvider` must name the provider that serves the `defaultModel`.
|
||||
- A subagent override uses a bare model `id`. A bare `id` resolves only when
|
||||
it is unique across all providers.
|
||||
|
||||
When you rename a service or move a model to a new host, update
|
||||
`models.json`. Update the provider key, the model `id`, `contextWindow`, and
|
||||
`input`. If the default model moved hosts, also update `defaultProvider` in
|
||||
`settings.json`. Set `contextWindow` to the per-slot context (both hosts
|
||||
`262144`). The full schema is in the agent knowledge base page
|
||||
`~/.pi/agent/docs/pi/models.md`.
|
||||
Per-role routing lives in `/home/dank/.omp/agent/config.yml` under
|
||||
`modelRoles` (`smol`, `tiny`, `task`, `plan`, `designer`, `advisor`, `slow`,
|
||||
`vision`, …). When you rename a service, retire an endpoint, or move a model
|
||||
to a new host, update the provider block, the model `id`, `contextWindow`,
|
||||
`maxTokens` and `input` in `models.yml`, then fix every `modelRoles` entry
|
||||
that named the old endpoint — a role pointing at a removed provider fails at
|
||||
call time. `frank-large` declares `input: [text, image]` because halogen runs
|
||||
its vision tower.
|
||||
|
||||
## Runtime & Troubleshooting
|
||||
|
||||
### frank-npu — FLM NPU sight+sound endpoint (Dankclaw, ACTIVE)
|
||||
### frank-npu — FLM NPU endpoint (Dankclaw, RETIRED 2026-09-11)
|
||||
|
||||
The third inference endpoint on Dankclaw. The `flm serve` runtime (dpkg
|
||||
package `fastflowlm`, v1.0.1, binary `/usr/bin/flm`) serves the **whole local
|
||||
model registry** (36 tags at first deploy) on a single port — 7771 — and
|
||||
routes chat completions per request by the exact FLM registry tag in the
|
||||
`model` field. Unknown tags do NOT 404: `/v1/chat/completions` answers HTTP 200
|
||||
with `{"error":"Failed to load <tag> model!"}` and no choices (status-only
|
||||
clients silently get nothing), so clients must send exact registry tags, never
|
||||
service aliases. Primary
|
||||
model = Gemma4 E4B IT NPU2 pack (vision + audio towers); the uncensored
|
||||
candidate (`qwythos:9b`) is text-only and fails the modality requirement, so
|
||||
this endpoint trades "uncensored" for "sight + sound". The NPU is a
|
||||
single-lane device — generations serialize FIFO; treat this as a
|
||||
background/triage endpoint, not an interactive multi-user chat.
|
||||
Retired by operator decision together with the kernel cmdline change that
|
||||
bought 13–16 % of prefill for `frank-large`: with `amd_iommu=off` the XDNA2
|
||||
driver cannot bind SVA/PASID, so every open of `/dev/accel/accel0` fails with
|
||||
`err=-19`. The unit is stopped and masked, the service files are staged in
|
||||
`.AGENTTRASH/` (`*.flm-npu`), and the providers that named it were removed
|
||||
from both the harness and Hermes. **Audio input left the fleet with it and
|
||||
has no replacement**; image input survives on `frank-large` through halogen's
|
||||
vision tower. Route map: Hermes `vision` → `frank-large`, Hermes
|
||||
`title_generation` → `frank-small`, harness `vision` → `frank-large`, harness
|
||||
`tiny` → `frank-small`, harness `task` → `frank-large` at medium effort.
|
||||
|
||||
Hard requirements verified at first deploy (2026-08-30):
|
||||
The retired runbook keeps the full record, including the original IOMMU/SVA
|
||||
root cause: [serverAppConfigs-docs/LLM/frank-npu.md](serverAppConfigs-docs/LLM/frank-npu.md).
|
||||
The current kernel posture and its revert path:
|
||||
[serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md](serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md).
|
||||
|
||||
- **IOMMU must be enabled.** With `iommu=off` on the kernel cmdline every
|
||||
open of `/dev/accel/accel0` fails with `err=-19`: the XDNA2 driver's SVA
|
||||
bind needs PASID/SVA, which requires enabled IOMMU hardware. Fix applied:
|
||||
`iommu=pt` in `GRUB_CMDLINE_LINUX_DEFAULT` + reboot (passthrough keeps the
|
||||
iGPU default domain identity-mapped — zero DMA-translation overhead for
|
||||
frank-large — while enabling IOMMU + PASID; PASID domains are
|
||||
paged regardless of the default-domain type). Never set `iommu=off` on
|
||||
this host again; it silently kills the NPU.
|
||||
- Health probe = `GET /v1/version` (FLM has no `/health` route).
|
||||
- Unit: `User=dank`, `LimitMEMLOCK=infinity` (FLM requires unlimited
|
||||
memlock), `Restart=on-failure`, `RestartSec=10`. Deploy via
|
||||
`scripts/deploy.sh dankclaw --service frank-npu`.
|
||||
- Verified live at first deploy: image probe (red PNG as `image_url`
|
||||
data-URI → correct color answer, vision tower consumed the pixels) and an
|
||||
audio control-pair probe (tone WAV via `input_audio` consumed ~22
|
||||
audio-tower tokens vs a text-only control).
|
||||
### halogen-flash-server (Dankclaw, ACTIVE since 2026-09-11)
|
||||
|
||||
Runbook: [serverAppConfigs-docs/LLM/frank-npu.md](serverAppConfigs-docs/LLM/frank-npu.md).
|
||||
### kingjones Qwen3.8-Flash-Next STRIX on kingjones30/ROCmFPX (Dankclaw, ACTIVE)
|
||||
The large service runs `ghcr.io/peonist-ai/halogen-flash-server:0.5.8` as a
|
||||
**rootless Quadlet container**
|
||||
(`dankclaw/frank-large/quadlet/frank-large.container` → generated
|
||||
`frank-large.service` in the user manager), serving
|
||||
`peonist-ai/halogen-qwen3.8-flash-next`: Qwen3.8-Flash-Next as a 115.55 GiB
|
||||
w4b `.hgn` checkpoint plus its quality overlay sidecar. One container runs
|
||||
both halves — the engine on container loopback :8730 (unauthenticated, never
|
||||
published) and the OpenAI front-end on :8731 published as host **7770** —
|
||||
with `HALOGEN_MODEL_ID=frank-large`, so no client wiring changed.
|
||||
|
||||
The large service serves
|
||||
Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=262144`
|
||||
(conservative for a shared host), `HALOGEN_KV_SLOTS=4`,
|
||||
`HALOGEN_HOST_RESERVE_GIB=24`, `HALOGEN_VISION_TOWER=1`, and
|
||||
`HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next` for the first-start
|
||||
fetch; everything else is upstream's default. The weights stay pinned
|
||||
(~68 GiB) and the kernel counts those locked pages as reclaimable page cache,
|
||||
so `free`, `MemAvailable` and anything built on them (Grafana) overstate free
|
||||
memory by the size of the model — believe the server's own startup ledger,
|
||||
not the dashboards.
|
||||
|
||||
Client-visible contract: **no server-side sampler defaults** (a request that
|
||||
sends no `temperature` decodes greedy); the per-request budget defaults to
|
||||
8192 and **includes thinking**, so a budget that runs out mid-thought returns
|
||||
`finish_reason: "length"` with empty content; the cap is 65536; and a request
|
||||
reserves prompt + `max_tokens` positions against the KV pool, so a request
|
||||
larger than the pool is never admitted — it queues and then fails with
|
||||
`503 engine_busy` after the 3600 s queue timeout.
|
||||
|
||||
Runbook: [serverAppConfigs-docs/LLM/halogen-flash-server.md](serverAppConfigs-docs/LLM/halogen-flash-server.md).
|
||||
|
||||
### kingjones Qwen3.8-Flash-Next STRIX on kingjones30/ROCmFPX (Dankclaw, PRIOR — rollback target)
|
||||
|
||||
The large service served
|
||||
`kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` (public, 113.47
|
||||
GiB single file, `Q4_0_ROCMFP4_STRIX` = file_type 105) from
|
||||
`/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/` with
|
||||
@@ -391,8 +423,9 @@ kill, only dmesg evidence; llama.cpp's "use_mmap=1" log line is
|
||||
correct here — mmap pages ARE page cache) and **never pin the PLE
|
||||
tensor with `-ot`** (23 → 13.4 t/s). The 51.2B-param PLE table never
|
||||
enters GPU memory (~20 bytes/token-file per 500 decoded). On-host file
|
||||
sha256 verified == HF LFS oid `c6770d74...49fa`. Env
|
||||
`scripts/frank-large.env`, serve `scripts/serve-frank-large.sh`.
|
||||
sha256 verified == HF LFS oid `c6770d74...49fa`. Its env and serve script
|
||||
(`scripts/frank-large.env`, `scripts/serve-frank-large.sh`) are staged in
|
||||
`.AGENTTRASH/`, suffixed `.llama.cpp`.
|
||||
Runbook:
|
||||
[serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md](serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md).
|
||||
|
||||
@@ -548,7 +581,7 @@ graph TD
|
||||
flags. A different outcome isolates the env file as the cause.
|
||||
3. Run gdb for the backtrace with this command:
|
||||
```
|
||||
gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-large.sh
|
||||
gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-small.sh
|
||||
```
|
||||
`llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost.
|
||||
Trust gdb, not the log.
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
# Quadlet unit: frank-large (rootless, boot-persistent).
|
||||
#
|
||||
# Install path on dankclaw: ~/.config/containers/systemd/frank-large.container
|
||||
# Unit name: frank-large.service (container name: frank-large)
|
||||
#
|
||||
# The large inference service, since 2026-09-11 the halogen-flash-server
|
||||
# runtime serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next,
|
||||
# w4b .hgn checkpoint + quality overlay sidecar). It replaced the llama.cpp /
|
||||
# kingjones30-ROCmFPX STRIX deployment because the engine is ~4x faster on
|
||||
# prefill (the dominant cost of any real prompt) on this exact SKU.
|
||||
#
|
||||
# SHAPE. The image runs both halves in one container: the engine on container
|
||||
# loopback (:8730, UNAUTHENTICATED — never publish it) and the OpenAI
|
||||
# front-end on :8731, published here as host 7770 so the fleet's clients,
|
||||
# services.json probe and docs keep the port they have always used. Same for
|
||||
# HALOGEN_MODEL_ID=frank-large: it is the id at /v1/models, in /health and in
|
||||
# every response, so no client had to be renamed. Halogen never rejects a
|
||||
# request for naming a different model, it just serves it.
|
||||
#
|
||||
# DEVICES. gfx1151 through the amdgpu/KFD stack: /dev/kfd plus the DRM render
|
||||
# node. Both are 0666 on this host, so the rootless user namespace can open
|
||||
# them (id dank is also in render/video). seccomp=unconfined and --ipc=host are
|
||||
# the image's documented requirements; memlock must be unlimited.
|
||||
#
|
||||
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
|
||||
# runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
|
||||
#
|
||||
# MEMORY. Weights are pinned (HALOGEN_FLASH_PIN_TRUNK default 1) — that is the
|
||||
# configuration the published decode numbers were measured with, and `free`,
|
||||
# MemAvailable and every monitor built on them will overstate free memory by
|
||||
# the size of the model (~68 GiB) because the kernel counts the locked weights
|
||||
# as reclaimable page cache. Halogen prints the corrected figure at startup.
|
||||
# The pinned trunk means this container holds most of the host; a co-tenant
|
||||
# that then takes the remainder can stall both for minutes at full CPU with no
|
||||
# disk activity and no output. That is documented behaviour, not a crash, and
|
||||
# the escape hatch is HALOGEN_FLASH_PIN_TRUNK=0 (at several times the decode
|
||||
# cost). Sizing chosen 2026-09-11 for a SHARED 121 GiB host: KV pool 262,144
|
||||
# positions (~7.2 GiB) + the baked 32,768-token prefill arena (16.7 GiB) sits
|
||||
# well inside the device budget once the ~68 GiB of weights and the ~5 GiB of
|
||||
# co-tenant containers are accounted for. HALOGEN_KV_POOL_FIT stays on as the
|
||||
# safety net: if the machine refuses, the engine lowers the pool itself and
|
||||
# prints what it chose.
|
||||
#
|
||||
# WEIGHTS. HALOGEN_DOWNLOAD fetches the whole weights repo on FIRST START ONLY
|
||||
# (the entrypoint checks whether the checkpoint is absent) into the bind mount
|
||||
# below, which must therefore be read-WRITE. huggingface_hub resumes partial
|
||||
# files natively, and the quadlet runs the container with --replace --rm, so a
|
||||
# restart mid-fetch continues rather than starting over. Roughly 127 GB: the
|
||||
# 115.55 GiB checkpoint, the 2.31 GiB quality overlay, the 0.84 GiB vision
|
||||
# tower and the tokenizer. The repo is public and ungated, so no token is
|
||||
# needed; later starts open no outbound connections at all.
|
||||
#
|
||||
# VISION. HALOGEN_VISION_TOWER=1 finds the sidecar beside the checkpoint. This
|
||||
# is the fleet's only image input since the FLM NPU endpoint was retired on
|
||||
# 2026-09-11; audio input has no replacement.
|
||||
#
|
||||
# CLIENT CONTRACT (see serverAppConfigs-docs/LLM/halogen-flash-server.md).
|
||||
# There are no server-side sampler defaults: a request that sends no
|
||||
# temperature decodes GREEDY. The per-request budget defaults to 8,192 tokens
|
||||
# and REASONING COUNTS AGAINST IT, so a budget that runs out mid-thought
|
||||
# returns finish_reason "length" with empty content and the partial reasoning
|
||||
# in reasoning_content. The cap is 65,536; above it a request is a 400, never a
|
||||
# silent truncation.
|
||||
#
|
||||
# IMAGE. The unit runs the localhost/-aliased tag: this host's podman quadlet
|
||||
# path cannot resolve registry-prefixed refs from local storage at create
|
||||
# time. deploy-remote.sh pulls the ghcr.io ref and re-tags it before the
|
||||
# restart. Pull=never keeps boots off the registry.
|
||||
[Unit]
|
||||
Description=frank-large — halogen-flash-server, Qwen3.8-Flash-Next (w4b) on gfx1151
|
||||
After=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=frank-large
|
||||
Image=localhost/peonist-ai/halogen-flash-server:0.5.8
|
||||
# Published port stays 7770: Hermes, open-webui, the harness and the homepage
|
||||
# probe all point here. 8730 (the engine) is deliberately not published.
|
||||
PublishPort=7770:8731
|
||||
AddDevice=/dev/kfd
|
||||
AddDevice=/dev/dri
|
||||
# Read-write: the first start downloads into it. Contains only weights.
|
||||
Volume=/home/dank/models/halogen:/models
|
||||
# Fetch the weights on first start only (absent checkpoint gates it).
|
||||
Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next
|
||||
# Fleet identity: the id at /v1/models, /health and every response body.
|
||||
Environment=HALOGEN_MODEL_ID=frank-large
|
||||
# Native context of the model. Matches what both clients declare.
|
||||
Environment=HALOGEN_CTX=262144
|
||||
# Conservative pool for a shared host: one full-length conversation, or four
|
||||
# at 65k. Raising it to 524288 doubles the resident conversations for ~7 GiB
|
||||
# more device memory — do that only after reading the startup ledger.
|
||||
Environment=HALOGEN_KV_POOL_POSITIONS=262144
|
||||
Environment=HALOGEN_KV_SLOTS=4
|
||||
# RAM left for the n-gram table the engine reads through the page cache.
|
||||
# Above the vendor's 20 because a dozen containers share this host; too low
|
||||
# and long prompts crawl with the disk busy.
|
||||
Environment=HALOGEN_HOST_RESERVE_GIB=24
|
||||
# Image input (the sidecar ships inside the weights repo).
|
||||
Environment=HALOGEN_VISION_TOWER=1
|
||||
# Required by the image: the engine maps the checkpoint and registers it with
|
||||
# the GPU in place, which needs host IPC and an unconfined seccomp profile.
|
||||
PodmanArgs=--ipc=host --security-opt seccomp=unconfined
|
||||
Ulimit=memlock=-1:-1
|
||||
# The image ships its own HEALTHCHECK; this pins it. /health asks the ENGINE
|
||||
# for a PONG before answering, so 200 means "ready to serve", not "process
|
||||
# alive". During the first-start download it stays unhealthy by design.
|
||||
HealthCmd=halogen-healthcheck api
|
||||
HealthInterval=30s
|
||||
HealthTimeout=10s
|
||||
HealthRetries=3
|
||||
HealthStartPeriod=300s
|
||||
Pull=never
|
||||
|
||||
[Service]
|
||||
# The first start reads ~68 GiB off disk and pins it before serving.
|
||||
TimeoutStartSec=900
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -25,23 +25,17 @@ providers:
|
||||
models:
|
||||
frank-small:
|
||||
context_length: 163840
|
||||
dankclaw-npu:
|
||||
api: http://192.168.0.143:7771/v1
|
||||
discover_models: false
|
||||
models:
|
||||
gemma4-it:e4b:
|
||||
context_length: 131072
|
||||
|
||||
# Auxiliary routing from the live configuration. Named providers route via
|
||||
# their `api:` URL; `custom` entries route by their explicit `base_url`.
|
||||
auxiliary:
|
||||
vision: {provider: dankclaw-npu, model: gemma4-it:e4b}
|
||||
vision: {provider: dankclaw-large, model: frank-large}
|
||||
web_extract: {provider: dankserver-small, model: frank-small}
|
||||
compression: {provider: dankserver-small, model: frank-small}
|
||||
skills_hub: {provider: dankserver-small, model: frank-small}
|
||||
approval: {provider: dankserver-small, model: frank-small}
|
||||
mcp: {provider: dankserver-small, model: frank-small}
|
||||
title_generation: {provider: dankclaw-npu, model: gemma4-it:e4b}
|
||||
title_generation: {provider: dankserver-small, model: frank-small}
|
||||
memory_query_rewrite: {provider: custom, model: frank-small, base_url: http://192.168.0.235:7770/v1}
|
||||
tts_audio_tags: {provider: custom, model: frank-small, base_url: http://192.168.0.235:7770/v1}
|
||||
triage_specifier: {provider: dankclaw-large, model: frank-large}
|
||||
|
||||
@@ -59,7 +59,6 @@ SERVICE_ORDER = (
|
||||
"lidarr",
|
||||
"readarr",
|
||||
"frank-large",
|
||||
"frank-npu",
|
||||
"frank-small",
|
||||
)
|
||||
|
||||
@@ -89,7 +88,6 @@ DANKCLAW_SERVICES = {
|
||||
"caddy",
|
||||
"homepage",
|
||||
"frank-large",
|
||||
"frank-npu",
|
||||
}
|
||||
|
||||
DANKSERVER_SERVICES = {
|
||||
@@ -297,6 +295,11 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
if len(parts) >= 2 and parts[1] == "open-webui":
|
||||
_add(services, "open-webui")
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "frank-large":
|
||||
# The service is a rootless Quadlet container since 2026-09-11;
|
||||
# its unit and fleet settings live in one .container file.
|
||||
_add(services, "frank-large")
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "gitea":
|
||||
if len(parts) >= 4 and parts[2] == "quadlet":
|
||||
name = parts[3]
|
||||
@@ -346,15 +349,12 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
|
||||
if parts[0] == "scripts":
|
||||
name = p.name
|
||||
if name in {"frank-large.env", "frank-large.service", "serve-frank-large.sh"}:
|
||||
_add(services, "frank-large")
|
||||
elif name in {"frank-npu.env", "frank-npu.service", "serve-frank-npu.sh"}:
|
||||
_add(services, "frank-npu")
|
||||
elif name in {"frank-small.env", "frank-small.service", "serve-frank-small.sh"}:
|
||||
if name in {"frank-small.env", "frank-small.service", "serve-frank-small.sh"}:
|
||||
_add(services, "frank-small")
|
||||
elif name == "chat-template.jinja":
|
||||
# Only the selected target's LLM consumes this shared template.
|
||||
_add(services, "frank-large" if target == "dankclaw" else "frank-small")
|
||||
# Only frank-small consumes this template now: frank-large runs
|
||||
# halogen-flash-server, which owns its own chat handling.
|
||||
_add(services, "frank-small")
|
||||
elif name in {"homepage-status.py", "homepage-status.service", "homepage-status.timer"}:
|
||||
_add(services, "homepage")
|
||||
# Deployment/test/benchmark scripts and other repo tooling do not
|
||||
|
||||
+76
-30
@@ -90,7 +90,7 @@ service_is_allowed() {
|
||||
dankclaw:authentik-postgres|dankclaw:authentik-server|\
|
||||
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
|
||||
dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|\
|
||||
dankclaw:frank-large|dankclaw:frank-npu|\
|
||||
dankclaw:frank-large|\
|
||||
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:plex|dankserver:tautulli|\
|
||||
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
|
||||
@@ -183,7 +183,6 @@ add_pending_services() {
|
||||
[[ -n "$recorded" && "$recorded" != "$REVISION" ]] || continue
|
||||
if [[ "$TARGET" == dankclaw && "$service" == frank-small ]]; then continue; fi
|
||||
if [[ "$TARGET" == dankserver && "$service" == frank-large ]]; then continue; fi
|
||||
if [[ "$TARGET" == dankserver && "$service" == frank-npu ]]; then continue; fi
|
||||
if ! changed="$(python3 scripts/deploy-plan.py --target "$TARGET" \
|
||||
--repo "$REMOTE_DIR" --base "$recorded" --head "$REVISION")"; then
|
||||
echo "WARN: unable to compare state for $service; retaining it as pending" >&2
|
||||
@@ -826,21 +825,85 @@ deploy_anubis() {
|
||||
wait_container_healthy anubis 60 || return 1
|
||||
}
|
||||
|
||||
# Rename cutover (2026-09-03): frank-default-large -> frank-large,
|
||||
# frank-default-small -> frank-small, frank-default-sightsound -> frank-npu.
|
||||
# The legacy units are stopped and disabled before the renamed unit takes the
|
||||
# port so no legacy process is left holding it; stop/disable of an already
|
||||
# unloaded legacy unit fail harmlessly and are tolerated.
|
||||
deploy_llm_large() {
|
||||
run_privileged systemctl stop frank-default-large || true
|
||||
run_privileged systemctl disable frank-default-large || true
|
||||
relink_unit frank-large
|
||||
# frank-large runs as a rootless Quadlet unit since 2026-09-11 (halogen-flash-
|
||||
# server replaced the llama.cpp/kingjones runtime). Unlike the sibling LLM
|
||||
# services it is therefore NOT a root unit: no relink_unit, no `systemctl
|
||||
# enable` (this host's quadlet generator produces units that cannot be
|
||||
# enabled — boot persistence comes from [Install] WantedBy=default.target in
|
||||
# the .container file plus linger).
|
||||
#
|
||||
# The llama.cpp-era ROOT unit is retired first. Its unit file is a symlink
|
||||
# into this checkout (the old relink_unit did `ln -sfn scripts/frank-large.
|
||||
# service /etc/systemd/system/`), so leaving it behind after the repo file
|
||||
# moves to the trash would leave systemd holding a dangling unit that can
|
||||
# still claim port 7770. Stop it, drop its enable symlinks, move the unit file
|
||||
# into the home trash (never rm), then reload the root manager.
|
||||
#
|
||||
# FIRST START: the container fetches ~127 GB of weights before it can serve,
|
||||
# so /health is legitimately down for the whole fetch. wait_for_health would
|
||||
# burn its budget and report a failure that is not one. The gate below
|
||||
# recognises the bootstrap (checkpoint absent) and waits for the checkpoint
|
||||
# instead, falling through to the normal health gate once it lands.
|
||||
LLM_LARGE_CHECKPOINT="${LLM_LARGE_CHECKPOINT:-/home/dank/models/halogen/qwen38-flash-next-w4b.hgn}"
|
||||
|
||||
retire_root_llm_large() {
|
||||
local trash="$HOME/.AGENTTRASH/20260911-halogen-swap/host-units"
|
||||
run_privileged systemctl stop frank-default-large 2>/dev/null || true
|
||||
run_privileged systemctl disable frank-default-large 2>/dev/null || true
|
||||
run_privileged systemctl stop frank-large.service 2>/dev/null || true
|
||||
# On this host `disable` has been observed to delete the unit file itself
|
||||
# (2026-08-26 incident, see AGENTS.md); tolerate either behaviour and keep
|
||||
# whatever remains by moving it to the trash.
|
||||
run_privileged systemctl disable frank-large.service 2>/dev/null || true
|
||||
if [[ -e /etc/systemd/system/frank-large.service || -L /etc/systemd/system/frank-large.service ]]; then
|
||||
mkdir -p "$trash"
|
||||
run_privileged mv -- /etc/systemd/system/frank-large.service \
|
||||
"$trash/frank-large.service"
|
||||
echo "retired root unit frank-large.service -> $trash"
|
||||
fi
|
||||
run_privileged systemctl daemon-reload
|
||||
run_privileged systemctl enable frank-large
|
||||
run_privileged systemctl restart frank-large
|
||||
}
|
||||
|
||||
deploy_llm_large() {
|
||||
retire_root_llm_large
|
||||
# Pull once, then run the localhost/-aliased tag: this host's quadlet path
|
||||
# cannot resolve registry-prefixed refs from local storage at create time
|
||||
# (same workaround as anubis/hermes).
|
||||
local image="ghcr.io/peonist-ai/halogen-flash-server:0.5.8"
|
||||
local alias="localhost/peonist-ai/halogen-flash-server:0.5.8"
|
||||
podman pull "$image"
|
||||
podman tag "$image" "$alias"
|
||||
# The weights volume: read-write, because the container downloads into it.
|
||||
mkdir -p /home/dank/models/halogen
|
||||
install_quadlet frank-large dankclaw/frank-large/quadlet frank-large
|
||||
systemctl --user restart frank-large.service
|
||||
wait_for_llm_large_ready
|
||||
wait_for_health frank-large http://127.0.0.1:7770/health 1200
|
||||
}
|
||||
|
||||
# wait_for_llm_large_ready — bootstrap-aware gate. If the checkpoint is absent
|
||||
# the container is downloading it; wait for the file (resumable, and the unit
|
||||
# restarts it on failure) rather than gating on a /health that cannot answer
|
||||
# yet. Reports success either way, so the first deploy does not look like a
|
||||
# failure while the engine does exactly what it was told to.
|
||||
wait_for_llm_large_ready() {
|
||||
[[ -f "$LLM_LARGE_CHECKPOINT" ]] && return 0
|
||||
echo "frank-large: checkpoint absent at $LLM_LARGE_CHECKPOINT"
|
||||
echo "frank-large: first start — the container is downloading ~127 GB from"
|
||||
echo "frank-large: peonist-ai/halogen-qwen3.8-flash-next (resumes if interrupted)."
|
||||
local i
|
||||
for ((i = 1; i <= 1200; i++)); do
|
||||
if [[ -f "$LLM_LARGE_CHECKPOINT" ]]; then
|
||||
echo "frank-large: weights landed after approximately $((i * 2))s; gating on /health"
|
||||
return 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
echo "WARN: frank-large download still in progress after 40m — NOT a failure."
|
||||
echo "WARN: the unit keeps fetching; re-run this deploy (or re-check /health) later." >&2
|
||||
return 0
|
||||
}
|
||||
|
||||
deploy_llm_small() {
|
||||
run_privileged systemctl stop frank-default-small || true
|
||||
run_privileged systemctl disable frank-default-small || true
|
||||
@@ -851,22 +914,6 @@ deploy_llm_small() {
|
||||
wait_for_health frank-small http://127.0.0.1:7770/health 1200
|
||||
}
|
||||
|
||||
# frank-npu is a live serving endpoint (FLM on the XDNA2 NPU), not a
|
||||
# relink-only unit: it is enabled at boot and restarted on deploy. The health
|
||||
# gate polls /v1/version, FLM's cheapest always-on route (there is no /health
|
||||
# route in the FLM server).
|
||||
deploy_llm_npu() {
|
||||
run_privileged systemctl stop frank-default-sightsound || true
|
||||
run_privileged systemctl disable frank-default-sightsound || true
|
||||
run_privileged systemctl stop frank-default-npu || true
|
||||
run_privileged systemctl disable frank-default-npu || true
|
||||
relink_unit frank-npu
|
||||
run_privileged systemctl daemon-reload
|
||||
run_privileged systemctl enable frank-npu
|
||||
run_privileged systemctl restart frank-npu
|
||||
wait_for_health frank-npu http://127.0.0.1:7771/v1/version 300
|
||||
}
|
||||
|
||||
wait_for_health() {
|
||||
local service="$1" url="$2" limit="$3"
|
||||
local i
|
||||
@@ -1052,7 +1099,6 @@ deploy_service() {
|
||||
seerr) deploy_seerr ;;
|
||||
frank-large) deploy_llm_large ;;
|
||||
frank-small) deploy_llm_small ;;
|
||||
frank-npu) deploy_llm_npu ;;
|
||||
hermes|hermes-dashboard|hermes-webui) deploy_hermes ;;
|
||||
romm|romm-mariadb|romm-redis|mariadb|redis) deploy_romm_stack "$service" ;;
|
||||
jellyfin) deploy_jellyfin ;;
|
||||
|
||||
@@ -134,6 +134,14 @@ for path in "${CHANGED[@]}"; do
|
||||
add_test tests/homepage_caddy.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
# frank-large moved to a rootless quadlet (halogen-flash-server) on
|
||||
# 2026-09-11; its unit carries the fleet's GPU and podman posture, so
|
||||
# it gets its own seam rather than the generic fallback below.
|
||||
dankclaw/frank-large/*)
|
||||
add_test tests/frank_large_quadlet.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
add_test tests/docs_completeness.bats
|
||||
;;
|
||||
# Quadlets live under dankclaw/<svc>/quadlet/ (reorganized out of the
|
||||
# top-level dankclaw/*.container globs above). Map the whole subtree so
|
||||
# changes to the pihole, open-webui, searxng, and gitea quadlets hit
|
||||
@@ -160,11 +168,6 @@ for path in "${CHANGED[@]}"; do
|
||||
scripts/mount-entropy.sh)
|
||||
add_test tests/mount_entropy.bats
|
||||
;;
|
||||
scripts/frank-large.env|scripts/serve-frank-large.sh)
|
||||
add_test tests/llm_units.bats
|
||||
add_test tests/llm_secrets.bats
|
||||
add_test tests/scripts_syntax.bats
|
||||
;;
|
||||
scripts/frank-small.env|scripts/serve-frank-small.sh)
|
||||
add_test tests/llm_units.bats
|
||||
add_test tests/llm_secrets.bats
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
---
|
||||
type: research
|
||||
title: Dankclaw IOMMU translation off (amd_iommu=off) — decision record
|
||||
description: Why Dankclaw boots with amd_iommu=off since 2026-09-11, what it buys (13-16% prefill for frank-large), what it costs (the XDNA2 NPU endpoint, machine-wide DMA translation), what was deliberately not copied from the reference machine, and how to revert.
|
||||
tags:
|
||||
- dankclaw
|
||||
- iommu
|
||||
- kernel
|
||||
- grub
|
||||
- prefill
|
||||
- frank-large
|
||||
- halogen
|
||||
- frank-npu
|
||||
timestamp: 2026-09-11
|
||||
---
|
||||
|
||||
# Dankclaw IOMMU translation off — decision record
|
||||
|
||||
**Decision (2026-09-11):** Dankclaw boots with **`amd_iommu=off`** instead of
|
||||
`iommu=pt`, to recover the prefill throughput the large-model runtime loses to
|
||||
IOMMU translation. The XDNA2 NPU endpoint was accepted as the cost and is
|
||||
retired ([[frank-npu]]).
|
||||
|
||||
## What changed
|
||||
|
||||
`GRUB_CMDLINE_LINUX_DEFAULT` in `/etc/default/grub` on Dankclaw:
|
||||
|
||||
```
|
||||
# before
|
||||
ro quiet splash iommu=pt crashkernel=…
|
||||
|
||||
# after
|
||||
ro quiet splash amd_iommu=off crashkernel=…
|
||||
```
|
||||
|
||||
Then `sudo update-grub` and a reboot. Nothing else on the command line was
|
||||
touched.
|
||||
|
||||
## Evidence
|
||||
|
||||
The upstream project published the measurement, and it is the only A/B they
|
||||
ran: **460 → 385 tok/s prefill at a 2,048-token prompt, 13 to 16 percent**,
|
||||
taken on a Ryzen AI Max+ 395 with 128 GB — the same SKU as Dankclaw. Their
|
||||
published serving profile boots this way, so every number in their README is a
|
||||
number with translation off. Decode is unaffected; prefill is where the loss is.
|
||||
|
||||
The mechanism offered is DMA translation on the path every GPU allocation
|
||||
takes. The project states plainly that this is *their* configuration and not a
|
||||
tuning guide, that the flag turns off DMA translation machine-wide, and that it
|
||||
takes the NPU with it.
|
||||
|
||||
## What it costs
|
||||
|
||||
1. **The NPU endpoint dies.** With IOMMU off there is no PASID/SVA state and
|
||||
`/dev/accel/accel0` cannot be opened — `err=-19`, the exact failure
|
||||
documented in [[frank-npu]]. Audio input left the fleet with it; image input
|
||||
survives on `frank-large` through halogen's vision tower.
|
||||
2. **Posture change machine-wide.** Dankclaw also serves Caddy (the TLS edge
|
||||
and the Anubis/Authentik guard chain), Authentik, Pi-hole, Gitea, and the
|
||||
observability stack. DMA translation is now off for all of them. This was
|
||||
accepted explicitly as the price of the prefill figure.
|
||||
|
||||
## What was deliberately NOT copied
|
||||
|
||||
The reference machine's command line has six settings. Only one was measured,
|
||||
so only that one was taken:
|
||||
|
||||
| setting | reference value | why not here |
|
||||
|---|---|---|
|
||||
| `amd_iommu=off` | set | **taken** — the measured 13–16 % |
|
||||
| `amdgpu.gttsize` | `126976` MiB | a size, not a constant: it must equal installed RAM (their table: 128 GB → 126976, 96 GB → 95232, 64 GB → 63488). Dankclaw's GTT already reports **114 GiB ≈ 94 % of its 121 GiB** without the flag, so it would buy nothing |
|
||||
| `ttm.pages_limit` | `32505856` | the same number in 4 KiB pages (their 128 GB row is 99.3 % of RAM). Pasting it onto a different machine asks the driver for more GTT than exists |
|
||||
| `amdgpu.vm_update_mode=0` | set | upstream has never run without it, makes no claim about what it buys, and lists it "for completeness rather than recommended" |
|
||||
| `amdgpu.noretry=0` | set | as above |
|
||||
| `amdgpu.sg_display=0` | set | as above |
|
||||
|
||||
## Verification and revert
|
||||
|
||||
```bash
|
||||
cat /proc/cmdline # expect amd_iommu=off, no iommu=pt
|
||||
ls /sys/kernel/iommu_groups/ | wc -l # expect 0 groups (was 30)
|
||||
systemctl is-active frank-npu # expect inactive (masked)
|
||||
curl -s http://127.0.0.1:7770/health # frank-large must still answer 200
|
||||
```
|
||||
|
||||
**To revert:** put `iommu=pt` back in `GRUB_CMDLINE_LINUX_DEFAULT` in place of
|
||||
`amd_iommu=off`, run `sudo update-grub`, reboot, and unmask/start `frank-npu`
|
||||
with the archived unit (`.AGENTTRASH/frank-npu.service.flm-npu`) if the NPU
|
||||
endpoint is wanted back. The IOMMU change is independent of the halogen
|
||||
deployment — reverting one does not require reverting the other.
|
||||
|
||||
## Related
|
||||
|
||||
- [[frank-npu]] — the endpoint this retired, including the err=-19 root cause
|
||||
- [[halogen-flash-server]] — the runtime bought by the prefill figure
|
||||
- [[kingjones-strix-rocmfp4]] — the runtime it replaced (rollback target)
|
||||
|
||||
## Citations
|
||||
|
||||
- [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README "The host settings these numbers were measured on", CHANGELOG 0.5.5–0.5.8
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: configuration
|
||||
title: frank-npu Runbook
|
||||
description: FastFlowLM (FLM) NPU sight+sound endpoint on Dankclaw — XDNA2 NPU serving Qwen3.8 Distilled Heretic vision+text pack on port 7771, including the IOMMU/SVA root cause that took it down at first deploy.
|
||||
type: reference
|
||||
title: frank-npu (retired) — FLM NPU endpoint on Dankclaw
|
||||
description: The retired FastFlowLM endpoint on Dankclaw's XDNA2 NPU — what it served, why it died on 2026-09-11 with the amd_iommu=off decision, and where every dependent route went.
|
||||
tags:
|
||||
- flm
|
||||
- fastflowlm
|
||||
@@ -9,133 +9,125 @@ tags:
|
||||
- npu
|
||||
- dankclaw
|
||||
- frank-npu
|
||||
timestamp: 2026-09-01
|
||||
- retired
|
||||
timestamp: 2026-09-11
|
||||
---
|
||||
|
||||
# frank-npu — FLM NPU sight+sound Runbook
|
||||
# frank-npu — the retired FLM NPU endpoint
|
||||
|
||||
`frank-npu` is the third inference endpoint on Dankclaw: a
|
||||
FastFlowLM (FLM) v1.0.1 server (`/usr/bin/flm`, dpkg package `fastflowlm`)
|
||||
running on the **XDNA2 NPU** (50 TOPS). It serves *sight* (vision) —
|
||||
which the iGPU llama.cpp fleet ([[kingjones-strix-rocmfp4]],
|
||||
`frank-large`) does not do. The model-selection tradeoff: the only
|
||||
NPU2 pack family on this box with BOTH a vision and an audio tower is the
|
||||
Gemma4 E4B IT pack; the uncensored candidate (`qwythos:9b`, 8.94 GB, 262k ctx)
|
||||
is text-only and fails the hard modality requirement, so this endpoint trades
|
||||
"uncensored" for "sight + sound".
|
||||
> **RETIRED 2026-09-11.** The endpoint is gone by operator decision, together
|
||||
> with the IOMMU configuration it required — see
|
||||
> [[dankclaw-iommu-translation-off]]. The service unit, env file and serve
|
||||
> script are staged in `serverAppConfigs/.AGENTTRASH/` (suffixed `.flm-npu`),
|
||||
> and the host unit is stopped and masked.
|
||||
|
||||
## Deployment shape
|
||||
The third inference endpoint on Dankclaw: a FastFlowLM (FLM) v1.0.1 server
|
||||
(`/usr/bin/flm`, dpkg `fastflowlm`) on the **XDNA2 NPU** (50 TOPS), serving the
|
||||
whole local model registry on port 7771 and routing per request by the exact
|
||||
FLM registry tag. Its one capability the iGPU fleet lacked was **sight** — and
|
||||
briefly **sound**: the Gemma4 E4B IT NPU2 pack carries both a vision and an
|
||||
audio tower, which is why the service ran that pack rather than the
|
||||
uncensored-but-text-only alternative.
|
||||
|
||||
This page is kept as the retirement record. The operational detail below was
|
||||
live-verified while the service ran and remains accurate about FLM's shape.
|
||||
|
||||
## Why it is retired
|
||||
|
||||
The replacement large-model runtime is worth **13–16 % of prefill** if the
|
||||
kernel boots with `amd_iommu=off` instead of `iommu=pt`, measured by the
|
||||
upstream project on this exact silicon. That single flag is the whole decision:
|
||||
|
||||
- With IOMMU off there is no PASID/SVA state, and the XDNA2 driver cannot hand
|
||||
out `/dev/accel/accel0` — every open fails with `err=-19` ("No such device").
|
||||
That is precisely the failure the endpoint was rescued from on 2026-08-30,
|
||||
recorded in the next section.
|
||||
- `iommu=pt` is not a middle ground for the NPU *and* the prefill number: the
|
||||
passthrough domain still costs the prefill throughput the vendor measured.
|
||||
- The operator chose prefill, so the NPU endpoint goes.
|
||||
|
||||
Consequences accepted at retirement: **audio input left the fleet with no
|
||||
replacement** (halogen reads images only, no audio or video), and the surviving
|
||||
image path is halogen's vision tower on `frank-large`.
|
||||
|
||||
## Where its routes went
|
||||
|
||||
| consumer | setting | now |
|
||||
|---|---|---|
|
||||
| Hermes | `auxiliary.vision` | `dankclaw-large` / `frank-large` (halogen vision tower) |
|
||||
| Hermes | `auxiliary.title_generation` | `dankserver-small` / `frank-small` |
|
||||
| harness (`~/.omp/agent/models.yml`) | `modelRoles.vision` | `llamacpp (dankclaw)` / `frank-large` |
|
||||
| harness | `modelRoles.tiny` | `llamacpp (dankserver)` / `frank-small` |
|
||||
| harness | `modelRoles.task` | `llamacpp (dankclaw)` / `frank-large:medium` |
|
||||
|
||||
The `flm (dankclaw-npu)` provider block was removed from the harness model
|
||||
config, and the `dankclaw-npu` provider from Hermes's live `config.yaml`.
|
||||
|
||||
## What it was
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
|---|---|
|
||||
| Runtime | FLM v1.0.1 (`/usr/bin/flm`, dpkg `fastflowlm`) on the XDNA2 NPU |
|
||||
| Service | `frank-npu.service` (systemd system unit, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`) |
|
||||
| Files | `scripts/frank-npu.env`, `scripts/serve-frank-npu.sh`, `scripts/frank-npu.service` |
|
||||
| Deploy | `scripts/deploy.sh dankclaw --service frank-npu` (relink + daemon-reload + enable + restart; health gate polls `/v1/version`, 300 s cap — FLM has **no `/health` route**) |
|
||||
| Port | `7771` on `0.0.0.0` (52625/52626 left free for ad-hoc `flm serve` experiments) |
|
||||
| Power mode | `FLM_PMODE=turbo` — no other tenant uses the NPU on this host (the large model runs on the iGPU), so the top power mode costs no neighbour anything; drop to `performance` if iGPU power-budget contention ever shows |
|
||||
| Context | `FLM_CTX_LEN=131072 (128K; verified max on this box; native context length.) |
|
||||
| ASR | `FLM_ASR=0` — the E4B pack answers images natively through its vision tower in `/v1/chat/completions`; setting `FLM_ASR=1` additionally loads Whisper-V3-Turbo-NPU2 (+0.62 GB) and exposes `/v1/audio/transcriptions` |
|
||||
| Service | `frank-npu.service` (systemd **system** unit, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`) |
|
||||
| Files (archived) | `scripts/frank-npu.env`, `scripts/serve-frank-npu.sh`, `scripts/frank-npu.service` |
|
||||
| Port | `7771` on `0.0.0.0` |
|
||||
| Power mode | `FLM_PMODE=turbo` — nothing else used the NPU, so the top power mode cost no neighbour anything |
|
||||
| Context | `FLM_CTX_LEN=131072` (128K, the verified maximum on this box) |
|
||||
| Vision/audio | `FLM_ASR=0`; the E4B pack answers images through its own vision tower, and `FLM_ASR=1` would have added Whisper-V3-Turbo-NPU2 (+0.62 GB) and `/v1/audio/transcriptions` |
|
||||
|
||||
The serve script exports `FLM_CONFIG_PATH=/home/dank/.config/flm/model_list.json`
|
||||
and `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins`; kernels are symlinked
|
||||
from `/opt/fastflowlm/share/flm/xclbins/` into the per-pack directories. A tag
|
||||
swap is therefore an env-file edit, not a script edit.
|
||||
The serve script exported `FLM_CONFIG_PATH=/home/dank/.config/flm/model_list.json`
|
||||
and `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins`, with kernels symlinked
|
||||
from `/opt/fastflowlm/share/flm/xclbins/` into the per-pack directories, so a
|
||||
tag swap was an env-file edit rather than a script edit.
|
||||
|
||||
## Model tags for switching
|
||||
## API contract (as it behaved)
|
||||
|
||||
FLM routes per-request by the exact registry tag in the `model` field. To swap
|
||||
the service model, change `FLM_MODEL` in the env file and restart. To let
|
||||
clients choose, send requests with the desired `model` field.
|
||||
- The endpoint served the **whole local registry** (36 tags at first deploy) on
|
||||
one port; `/v1/models` listed them all. Chat completions routed per request
|
||||
by the exact registry tag in the `model` field — never a service alias.
|
||||
- **Unknown tags did not 404**: `/v1/chat/completions` answered HTTP 200 with a
|
||||
flat error body `{"error":"Failed to load <tag> model!"}` and no `choices`.
|
||||
Status-code-only clients silently got an empty response.
|
||||
- Health was `GET /v1/version`; FLM has **no `/health` route**.
|
||||
- The NPU is a single-lane device — generations serialized FIFO. It was a
|
||||
background/triage endpoint, not interactive multi-user chat.
|
||||
|
||||
| Tag | Model | Modality | Size | Family |
|
||||
|-----|-------|----------|------|--------|
|
||||
| `qwen3.8-distilled-heretic:9b` | Qwen3.8 Distilled Heretic | vision + reasoning | 7.6 GB | qwen3.5 |
|
||||
| `gemma4-it:e4b` | Gemma4 E4B IT | vision + audio | 9.1 GB | gemma4e |
|
||||
| `qwythos:9b` | Qwythos 9B Claude Mythos | vision + reasoning | 8.9 GB | qwen3.5 |
|
||||
| `gpt-oss:20b` | GPT-OSS 20B | text only | 13.8 GB | gpt-oss |
|
||||
## The IOMMU/SVA root cause (first deploy crashed, fixed 2026-08-30)
|
||||
|
||||
**To swap the active model** (changes what the service loads at start):
|
||||
```
|
||||
sed -i 's/^FLM_MODEL=.*/FLM_MODEL=<tag>/' scripts/frank-npu.env
|
||||
./scripts/deploy.sh dankclaw --service frank-npu
|
||||
```
|
||||
First deploy crash-looped every 10 s with
|
||||
`Error: Open /dev/accel/accel0 failed (err=-19): No such device`. The chain,
|
||||
all live-verified on the host:
|
||||
|
||||
**To switch per-request** (service stays loaded, client sends the tag):
|
||||
```
|
||||
curl http://192.168.0.143:7771/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"gemma4-it:e4b","messages":[...],"max_tokens":64}'
|
||||
```
|
||||
|
||||
Known tags are in the user registry at
|
||||
`/home/dank/.config/flm/model_list.json`; unknown tags answer HTTP 200 with
|
||||
`{"error":"Failed to load <tag> model!"}` (no choices).
|
||||
|
||||
## Serving model and API contract
|
||||
|
||||
- The endpoint serves the **whole local registry** (36 tags at first deploy) on
|
||||
one port; `/v1/models` lists them all. Chat completions route per request by
|
||||
the exact FLM registry tag in the `model` field — never a service alias.
|
||||
- **Unknown model tags do NOT 404** (live-probed): `/v1/chat/completions`
|
||||
answers HTTP 200 with a flat error body
|
||||
`{"error":"Failed to load <tag> model!"}` and no `choices`;
|
||||
`/v1/completions` answers an OpenAI-shaped nested error
|
||||
(`type: server_error`, `code: 500`), also under HTTP 200. A valid tag
|
||||
routes that request to that model (response echoes the requested tag); no
|
||||
server restart is needed to switch between registry tags. Status-code-only
|
||||
clients silently get an empty response — check for the `error` key.
|
||||
- Multimodal content parts accepted by `/v1/chat/completions`:
|
||||
`{"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}` and
|
||||
`{"type":"input_audio","input_audio":{"data":"<base64 WAV, no data: prefix>","format":"wav"}}`.
|
||||
- Chat template roles: only `system`/`user`/`assistant`; any other role fails
|
||||
the request with `Unexpected message role.`.
|
||||
- The NPU is a single-lane device: generations serialize in FIFO order. This is
|
||||
a background/triage endpoint, not an interactive multi-user chat.
|
||||
|
||||
## Live-verified behavior (2026-08-30, first deploy)
|
||||
|
||||
- Vision tower: a generated red 8×8 PNG posted as an `image_url` data-URI was
|
||||
answered correctly ("The image is a solid red color."); the request consumed
|
||||
279 prompt tokens — the vision tower consumed the pixels (a text-only model
|
||||
given zero image bytes cannot answer "what color").
|
||||
- Audio tower: a 440 Hz decaying sine WAV posted as `input_audio` consumed
|
||||
37 prompt tokens vs 15 for the identical text-only control → ~22 audio-tower
|
||||
tokens; the model answered truthfully that it heard no speech (it is a tone,
|
||||
not speech — ASR stays off).
|
||||
|
||||
## The IOMMU/SVA root cause (first deploy crashed, fixed same day)
|
||||
|
||||
First deploy crash-looped every 10 s (`RestartSec=10`) with
|
||||
`Error: Open /dev/accel/accel0 failed (err=-19): No such device`. Root-cause
|
||||
chain, all live-verified on the host:
|
||||
|
||||
1. `amdxdna_drm_open` was failing **SVA bind with ret -19** for every client —
|
||||
the XDNA2 driver cannot hand out the accel device without PASID/SVA.
|
||||
2. SVA requires the AMD IOMMU to be enabled; the kernel cmdline carried
|
||||
`iommu=off` (a perf/DMA tuning experiment from 2026-08-20, applied at the
|
||||
2026-08-26 reboot). With IOMMU off there are no IOMMU groups and no PASID
|
||||
state; `PASIDCtl: Enable-` was observed disabled on `0000:c5:00.1`.
|
||||
3. Fix: replace `iommu=off` with **`iommu=pt`** in `GRUB_CMDLINE_LINUX_DEFAULT`
|
||||
on `/etc/default/grub`, `update-grub`, reboot. Passthrough mode keeps the
|
||||
iGPU default domain identity-mapped (zero DMA-translation overhead for
|
||||
`frank-large`) while enabling IOMMU hardware + PASID, which is what
|
||||
the NPU driver's SVA bind requires — PASID domains are paged regardless of
|
||||
the default-domain type.
|
||||
1. `amdxdna_drm_open` failed **SVA bind with ret -19** for every client — the
|
||||
XDNA2 driver cannot hand out the accel device without PASID/SVA.
|
||||
2. SVA requires the AMD IOMMU to be enabled, and the kernel cmdline then
|
||||
carried `iommu=off`. With IOMMU off there are no IOMMU groups and no PASID
|
||||
state; `PASIDCtl: Enable-` was observed on `0000:c5:00.1`.
|
||||
3. The fix was `iommu=pt`: passthrough keeps the iGPU default domain
|
||||
identity-mapped (no DMA-translation overhead for `frank-large`) while
|
||||
enabling the IOMMU hardware and PASID that the driver's SVA bind needs.
|
||||
4. Post-boot verification: `/sys/kernel/iommu_groups/` populated (30 groups),
|
||||
zero `SVA bind device failed` journal entries since boot, accel0 recreated at
|
||||
boot, service active with `NRestarts=0`, and the image/audio probes above
|
||||
served live.
|
||||
zero `SVA bind device failed` entries since boot, `accel0` recreated at
|
||||
boot, service active with `NRestarts=0`.
|
||||
|
||||
The same root cause explains the older `amdgpu 0000:c4:00.0: no vmid pasid
|
||||
mapping supported` kernel log line — a consequence of IOMMU-off, not a GPU
|
||||
fault. Do not set `iommu=off` on this host again; it silently kills the NPU.
|
||||
The same root cause explained the older
|
||||
`amdgpu 0000:c4:00.0: no vmid pasid mapping supported` kernel line — a
|
||||
consequence of IOMMU-off, not a GPU fault.
|
||||
|
||||
## Client wiring
|
||||
**The standing rule this produced — "never set `iommu=off` on this host again" —
|
||||
has been superseded**: the NPU it protected is retired by decision, and
|
||||
`amd_iommu=off` is now intentional. See
|
||||
[[dankclaw-iommu-translation-off]] for the current posture and how to revert it.
|
||||
|
||||
The pi client model config (`/home/dank/.pi/agent/models.json` on Dankware)
|
||||
registers this endpoint under the Dankclaw provider with the **exact** FLM
|
||||
registry tag as the model `id` (unknown tags answer a 200-wrapped error, not
|
||||
a silent 1B fallback), `baseUrl` `http://192.168.0.143:7771/v1`, and a per-request context
|
||||
window equal to the serving ctx-len. Health checks for this service must use
|
||||
`GET /v1/version` (there is no `/health` route).
|
||||
## Client wiring (historical)
|
||||
|
||||
The pi/harness model config registered this endpoint with the **exact** FLM
|
||||
registry tag as the model `id` and `baseUrl` `http://192.168.0.143:7771/v1`.
|
||||
Both that provider block and Hermes's `dankclaw-npu` provider were removed at
|
||||
retirement; the routes are mapped in the table above.
|
||||
|
||||
## Citations
|
||||
|
||||
- [FastFlowLM](https://github.com/FastFlowLM/FastFlowLM) — FLM runtime and NPU model packs
|
||||
- [[dankclaw-iommu-translation-off]] — the decision that retired this endpoint
|
||||
- [[halogen-flash-server]] — the surviving image-input path
|
||||
|
||||
@@ -0,0 +1,240 @@
|
||||
---
|
||||
type: reference
|
||||
title: halogen-flash-server on Dankclaw (frank-large)
|
||||
description: The active frank-large runbook — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing.
|
||||
tags:
|
||||
- llm
|
||||
- halogen
|
||||
- dankclaw
|
||||
- frank-large
|
||||
- strix-halo
|
||||
- gfx1151
|
||||
- qwen4exp
|
||||
- rocm
|
||||
timestamp: 2026-09-11
|
||||
---
|
||||
|
||||
# halogen-flash-server on Dankclaw (frank-large)
|
||||
|
||||
> **ACTIVE since 2026-09-11.** Replaced the llama.cpp / kingjones30-ROCmFPX
|
||||
> STRIX deployment (see [[kingjones-strix-rocmfp4]], now the rollback target).
|
||||
> Deployed as a rootless Quadlet container,
|
||||
> `dankclaw/frank-large/quadlet/frank-large.container`, port 7770.
|
||||
|
||||
The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next
|
||||
architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs.
|
||||
Upstream's own comparison table measures it at **roughly 4x the prefill
|
||||
throughput** of the llama.cpp-family runtimes it ships against (EngramHalo.cpp,
|
||||
ROCmFP4, CIRU-IU4) on a 32K prompt, and prefill is most of the wall clock on any
|
||||
prompt with real context. That, not decode, is the reason for the swap.
|
||||
|
||||
## Shape
|
||||
|
||||
One container runs both halves of the stack:
|
||||
|
||||
| half | port | reachable |
|
||||
|---|---|---|
|
||||
| engine | 8730 | container loopback only — **unauthenticated, never publish it** |
|
||||
| OpenAI front-end | 8731 | published as host **7770** |
|
||||
|
||||
The `all` mode is upstream's default and the shape a single-container host
|
||||
wants: the front-end talks to the engine over loopback inside the container, so
|
||||
there is one process tree, one log, and one published port.
|
||||
|
||||
`HALOGEN_MODEL_ID=frank-large` sets the id reported at `/v1/models`, in
|
||||
`/health`, and in every response body. Requests are **never rejected for naming
|
||||
a different model** — the server serves whatever asks. Keeping the fleet alias
|
||||
is therefore free, and it means Hermes, the harness config, the OpenWebUI
|
||||
discovery, the `services.json` probe and the docs needed no rename.
|
||||
|
||||
## Weights
|
||||
|
||||
`peonist-ai/halogen-qwen3.8-flash-next` (HuggingFace, public and ungated):
|
||||
|
||||
| file | size | role |
|
||||
|---|---|---|
|
||||
| `qwen38-flash-next-w4b.hgn` | 115.55 GiB | the checkpoint |
|
||||
| `qwen38-flash-next-w4b.overlay.hgn` | 2.31 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing |
|
||||
| `qwen38-flash-next-vision.hgn` | 0.84 GiB | vision tower (see below) |
|
||||
| `tokenizer/` | ~21 MB | flat tokenizer dir; found beside the checkpoint, so no extra mount |
|
||||
|
||||
`HALOGEN_DOWNLOAD` fetches the whole repo into the bind mount on **first start
|
||||
only** — the entrypoint acts only when the checkpoint is absent, so a restart
|
||||
never re-downloads, and `huggingface_hub` resumes an interrupted pull. The
|
||||
mount must be read-write for this. Roughly 127 GB; on Dankclaw it lands in
|
||||
`/home/dank/models/halogen`.
|
||||
|
||||
`overlay-speed.hgn` (2.22 GiB) comes along with the repo but is unused: it
|
||||
trades the `o_proj` calibration for about 2 % of decode, and the quality
|
||||
sidecar is the default for a reason.
|
||||
|
||||
## Deployed flags
|
||||
|
||||
Everything else is configured by environment variable — there is no config
|
||||
file, and the flags are read once at startup. These are the departures from
|
||||
upstream's defaults:
|
||||
|
||||
| flag | value | why |
|
||||
|---|---|---|
|
||||
| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
|
||||
| `HALOGEN_CTX` | `262144` | the model's native context, and what both clients declare |
|
||||
| `HALOGEN_KV_POOL_POSITIONS` | `262144` | conservative for a **shared** 121 GiB host: one full-length conversation, or four at 65k, at ~7.2 GiB |
|
||||
| `HALOGEN_KV_SLOTS` | `4` | four concurrent generations; slots share the pool and cost ~115 MiB each |
|
||||
| `HALOGEN_HOST_RESERVE_GIB` | `24` | RAM kept for the n-gram table the engine reads through the page cache — above upstream's 20 because a dozen containers share this host |
|
||||
| `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired |
|
||||
| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
|
||||
|
||||
Everything else is upstream's default, deliberately: prompt cache mode `2`
|
||||
(two entries per conversation — end-of-system-prompt and end-of-history — which
|
||||
already covers the shared-prefix case), `HALOGEN_CACHE_INPLACE=1` (the snapshot
|
||||
stays ~115 MiB at any depth), the speculative drafter `1` (byte-identical at
|
||||
temperature 0, and it stands down whenever a second conversation is
|
||||
generating), the quality sidecar, `HALOGEN_ADMIT_CHUNK=0` (a non-zero value
|
||||
gives up the byte-identity property), the baked `HALOGEN_MAX_TOK=32768`
|
||||
(16.7 GiB prefill arena), and the default `HALOGEN_MAX_TOKENS_CAP=65536`.
|
||||
|
||||
## Memory model
|
||||
|
||||
Three numbers explain the whole box:
|
||||
|
||||
- **~68 GiB pinned weights.** The checkpoint is mapped and registered with the
|
||||
GPU in place, never copied. The kernel counts the locked pages as reclaimable
|
||||
file cache, so `free`, `MemAvailable` and every dashboard built on them
|
||||
**overstate free memory by the size of the model**. The server prints the
|
||||
corrected figure at startup; believe that line, not Grafana.
|
||||
- **~7.2 GiB per 262,144 KV positions**, plus the prefill arena (16.7 GiB at
|
||||
`HALOGEN_MAX_TOK=32768`, 8.4 GiB at 16384). The engine's device-side budget
|
||||
stops at roughly 47 GiB on a 128 GB machine *with the weights pinned* — and
|
||||
lower on a machine carrying more besides this server, which Dankclaw is.
|
||||
- **`HALOGEN_HOST_RESERVE_GIB`** is what is left for the page cache the lookup
|
||||
table is read through. Too little and long prompts crawl with the disk busy
|
||||
and the process in uninterruptible sleep — that is a cache problem, not a
|
||||
memory problem, and the fix is a *smaller* pool, not a bigger one.
|
||||
|
||||
`HALOGEN_KV_POOL_FIT=1` (default) measures that budget at startup and lowers
|
||||
the pool itself if the configured one will not fit, printing what it chose.
|
||||
Read those lines on every restart; they are the ground truth for sizing.
|
||||
|
||||
**Co-tenancy.** `HALOGEN_FLASH_PIN_TRUNK=1` (default) is what the published
|
||||
decode numbers were measured with, and it means this container holds most of
|
||||
the host. If a co-tenant then takes the remainder, both it and the engine can
|
||||
stall for minutes at a time at full CPU with no disk activity and no output.
|
||||
That is documented behaviour, not a crash, and it needs no restart.
|
||||
`HALOGEN_FLASH_PIN_TRUNK=0` gives the memory back and costs several times the
|
||||
decode speed — a last resort, not a tuning option.
|
||||
|
||||
## Client contract
|
||||
|
||||
The differences from llama.cpp that bite are all in the request, not the
|
||||
deployment:
|
||||
|
||||
- **No server-side sampler defaults.** `temperature` absent or `0` is
|
||||
**greedy** decode; above 0 the request samples on the same drafter. Supported
|
||||
fields: `temperature`, `top_p`, `top_k`, `min_p`, `seed`,
|
||||
`presence_penalty`, `frequency_penalty`, `logit_bias`, `logprobs`. Refused
|
||||
with a 400: `top_logprobs`, `logprobs` with `stream: true`, `n > 1`, and any
|
||||
value outside its defined range (values are refused, never clamped).
|
||||
The fleet runs greedy for clients that stay silent — deterministic, and
|
||||
byte-identical with the drafter.
|
||||
- **The token budget covers thinking.** The default is **8192**, and a budget
|
||||
that runs out mid-thought returns `finish_reason: "length"` with an **empty
|
||||
`content`** and the partial reasoning in `reasoning_content` — which most
|
||||
clients do not display. Above the cap (`HALOGEN_MAX_TOKENS_CAP`, 65536) a
|
||||
request is a 400 rather than a silent truncation. `max_tokens`,
|
||||
`max_completion_tokens` and `max_output_tokens` all work and must agree if
|
||||
more than one is sent.
|
||||
- **`reasoning_effort`** is accepted at the top level and inside
|
||||
`chat_template_kwargs`: `minimal`, `low`, `medium`, `high`, `xhigh`. The
|
||||
model's default is `xhigh`. `minimal` aliases the template's `low` and still
|
||||
reasons; `enable_thinking: false` turns thinking off; `preserve_thinking` is
|
||||
honoured. `preserve_reasoning` is a **400** — it is the llama.cpp-era key.
|
||||
- `/v1/models`, `/health` and completions all report the served id, and
|
||||
`reasoning_content` carries the reasoning trace.
|
||||
|
||||
**Pool admission is the one arithmetic trap.** A request reserves its prompt
|
||||
**plus** `max_tokens` positions when it is admitted; if the pool cannot hold it
|
||||
the request waits in arrival order, and after the queue timeout (3600 s) it
|
||||
gets a `503 engine_busy`. A prompt large enough that prompt + budget exceeds
|
||||
the pool can therefore never be admitted at all. The 262,144-position pool is
|
||||
the binding constraint on how large a conversation can get, not `HALOGEN_CTX`
|
||||
and not the client's declared `context_length`.
|
||||
|
||||
## Vision
|
||||
|
||||
With the tower loaded, `/v1/chat/completions` accepts the usual OpenAI image
|
||||
content part carrying a `data:` URL or bare base64 — **`http(s)` URLs are
|
||||
refused deliberately**, because fetching one would make the server issue
|
||||
outbound requests to wherever a client pointed it. One image costs about 2,040
|
||||
tokens of context and 5.5 / 11.8 / 25.3 s at 1280x800 / 1920x1080 / 2560x1440;
|
||||
text at 12 pt and above is read exactly, and a bigger frame past that point
|
||||
adds empty area rather than detail. Audio and video have no path here at all —
|
||||
that capability left with [[frank-npu]].
|
||||
|
||||
## Deploy and operate
|
||||
|
||||
```bash
|
||||
./scripts/deploy.sh dankclaw --service frank-large
|
||||
```
|
||||
|
||||
The handler retires the llama.cpp-era **root** unit first (stops it, drops its
|
||||
enable symlinks, moves `/etc/systemd/system/frank-large.service` into the home
|
||||
trash — never `rm`), then pulls `ghcr.io/peonist-ai/halogen-flash-server:0.5.8`,
|
||||
re-tags it to the `localhost/` alias the unit references, installs the Quadlet,
|
||||
and restarts the generated unit. Boot persistence comes from
|
||||
`[Install] WantedBy=default.target` in the unit file plus linger on the deploy
|
||||
user — generated units **cannot** be `systemctl --user enable`d on this host.
|
||||
|
||||
The health gate is bootstrap-aware: while the checkpoint is absent the
|
||||
container is downloading and `/health` cannot answer, so the handler waits for
|
||||
the file instead of burning its budget and reporting a failure that is not one.
|
||||
|
||||
```bash
|
||||
systemctl --user status frank-large.service # the generated unit
|
||||
podman logs frank-large | head -40 # version banner + memory ledger
|
||||
curl -s http://127.0.0.1:7770/health # 200 only after the engine PONGs
|
||||
curl -s http://127.0.0.1:7770/v1/models
|
||||
```
|
||||
|
||||
`/health` asks the **engine** for a PONG before answering, so a 200 means
|
||||
"ready to serve", not merely "process alive". During the first-start download
|
||||
it stays unhealthy by design.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| symptom | cause |
|
||||
|---|---|
|
||||
| `dmalloc: FAILED requesting … (out of memory)` then `HIP … out of memory` | the KV pool did not fit. `HALOGEN_KV_SLOTS` will not fix it (slots share the pool); lower `HALOGEN_KV_POOL_POSITIONS` to 262144, then `HALOGEN_MAX_TOK` to 16384 (~9 % prefill) |
|
||||
| starts, answers short prompts, collapses to a few tok/s on a long one, disk busy, process in uninterruptible sleep | short of **file cache**, not memory — lower the pool and/or raise `HALOGEN_HOST_RESERVE_GIB` |
|
||||
| long replies that look cut off | read `finish_reason`: `stop` means you have the whole answer, `length` means you ran out of budget (thinking counts) |
|
||||
| a request that never returns, then `503 engine_busy` | the pool could not reserve prompt + `max_tokens`; see the admission trap above |
|
||||
| both the engine and a co-tenant stall for minutes at full CPU | the pinned-trunk trade; see Memory model |
|
||||
| `free`, `MemAvailable`, Grafana all wrong by ~68 GiB | expected — locked weights are counted as reclaimable |
|
||||
|
||||
`HALOGEN_VERBOSE=1` and `HALOGEN_DMALLOC_LOG=1` print the fullest startup
|
||||
account and every allocation over 64 MB; both are the vendor's suggested
|
||||
evidence to attach to a report. The two lines worth grepping first:
|
||||
|
||||
```bash
|
||||
podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):'
|
||||
```
|
||||
|
||||
## Rollback
|
||||
|
||||
The llama.cpp era is intact: the STRIX payload stays at
|
||||
`/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX`, the
|
||||
`kingjones30/ROCmFPX` build stays at
|
||||
`/home/dank/GitPackages/kingjones-ROCmFPX/build`, and the retired unit, env and
|
||||
serve script are in `serverAppConfigs/.AGENTTRASH/` (suffixed `.llama.cpp`).
|
||||
Rolling back is a `git revert` of this change set plus a deploy — **but the
|
||||
Quadlet unit must be removed from `~/.config/containers/systemd/` first**
|
||||
(otherwise the restarted root unit and the still-installed container both
|
||||
claim port 7770). See [[kingjones-strix-rocmfp4]].
|
||||
|
||||
The `amd_iommu=off` kernel argument that came with this change is separate and
|
||||
reverts on its own — see [[dankclaw-iommu-translation-off]].
|
||||
|
||||
## Citations
|
||||
|
||||
- [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README, `docs/FLAGS.md`, `deploy/entrypoint.sh`, `docker-compose.yml`, CHANGELOG (v0.5.8)
|
||||
- [peonist-ai/halogen-qwen3.8-flash-next](https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next) — weights and file inventory
|
||||
- [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) — base model
|
||||
@@ -1,10 +1,9 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on two local machines
|
||||
(2026-08-30): FreeToken v0.1.2 on Dankserver (frank-small,
|
||||
Huihui-Qwen3.6-35B-A3B-abliterated-FP8) and kingjones30/ROCmFPX
|
||||
llama.cpp on Dankclaw (frank-large, kingjones777
|
||||
Qwen3.8-Flash-Next ROCmFP4-STRIX). Both services run the Qwen-Sharp chat template.
|
||||
(2026-09-11): FreeToken v0.1.2 on Dankserver (frank-small,
|
||||
Huihui-Qwen3.6-35B-A3B-abliterated-FP8) and halogen-flash-server 0.5.8 on
|
||||
Dankclaw (frank-large, peonist-ai Qwen3.8-Flash-Next w4b).
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -26,10 +25,10 @@ block-fp8, 20% MoE residency, 453k KV / 4 streams]
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
E2[gfx1151 iGPU 64 GiB unified memory]
|
||||
E[kingjones30/ROCmFPX build
|
||||
E[halogen-flash-server 0.5.8
|
||||
frank-large, port 7770]
|
||||
F[kingjones777 Qwen3.8-Flash-Next
|
||||
ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE]
|
||||
F[peonist-ai Qwen3.8-Flash-Next
|
||||
w4b .hgn + overlay, 262k ctx / 4 slots]
|
||||
E2 --> E --> F
|
||||
end
|
||||
|
||||
@@ -53,14 +52,19 @@ ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE]
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[kingjones-strix-rocmfp4]] — ACTIVE frank-large runbook:
|
||||
kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
|
||||
- [[halogen-flash-server]] — ACTIVE frank-large runbook since 2026-09-11:
|
||||
peonist-ai/halogen-flash-server 0.5.8 serving
|
||||
peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn +
|
||||
quality overlay), rootless Quadlet container on Dankclaw port 7770 —
|
||||
deployed flags, memory model, client contract, troubleshooting
|
||||
- [[dankclaw-iommu-translation-off]] — why Dankclaw boots with
|
||||
`amd_iommu=off` since 2026-09-11 (13-16 % prefill, measured upstream),
|
||||
what it cost (the NPU endpoint), what was deliberately not copied from
|
||||
the reference machine, and how to revert
|
||||
- [[kingjones-strix-rocmfp4]] — PRIOR frank-large runbook and rollback
|
||||
target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
|
||||
(56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned
|
||||
kingjones30/ROCmFPX fork (swapped 2026-08-30)
|
||||
- [[agentionai-fast-imatrix]] — retired frank-large runbook:
|
||||
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB
|
||||
Q4_0_ROCMFP4_FAST imatrix, v2 joined PLE) on build-qwen38
|
||||
(2026-08-29 → 2026-08-30); rollback binary + payload retained
|
||||
kingjones30/ROCmFPX fork (2026-08-30 → 2026-09-11)
|
||||
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
|
||||
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
|
||||
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
|
||||
@@ -78,11 +82,11 @@ ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE]
|
||||
(deployed and reverted 2026-08-22)
|
||||
- [[Dankserver System Specs]] — the RTX 3090 box (frank-small)
|
||||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large,
|
||||
served on the pinned kingjones30/ROCmFPX llama.cpp fork since 2026-08-30)
|
||||
- [[frank-npu]] — ACTIVE FLM NPU sight+sound runbook on
|
||||
Dankclaw: Gemma4 E4B vision+audio pack on the XDNA2 NPU (FLM v1.0.1, port
|
||||
7771, whole-registry serving, per-request model routing), incl. the
|
||||
IOMMU/SVA root cause and the `iommu=pt` fix behind its first-deploy crash
|
||||
on halogen-flash-server since 2026-09-11)
|
||||
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
|
||||
on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
|
||||
the retirement record — the IOMMU/SVA err=-19 root cause and where
|
||||
every dependent route moved
|
||||
|
||||
## CIRU Reference (Dankclaw)
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw
|
||||
description: ACTIVE frank-large runbook. kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the card's pinned kingjones30/ROCmFPX fork. 177B params (56B MoE + 51.2B PLE streamed via mmap), single 113.47 GiB GGUF, 401408 ctx x 2 slots.
|
||||
title: kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw (prior, rollback target)
|
||||
description: PRIOR frank-large runbook and rollback target (2026-08-30 → 2026-09-11). kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the card's pinned kingjones30/ROCmFPX fork. 177B params (56B MoE + 51.2B PLE streamed via mmap), single 113.47 GiB GGUF, 401408 ctx x 2 slots.
|
||||
tags:
|
||||
- llama.cpp
|
||||
- rocm
|
||||
@@ -16,12 +16,22 @@ timestamp: 2026-08-30
|
||||
|
||||
# kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw
|
||||
|
||||
> **Active again since 2026-08-30** on the model card's **pinned runtime**:
|
||||
> the `kingjones30/ROCmFPX` fork (replaced the agentionai FAST deployment
|
||||
> which ran on charlie's `build-qwen38`). Text-only tier; the agentionai
|
||||
> f16 vision tower is paired for mmproj and loads clean. Speculative
|
||||
> decoding is OFF (the card ships no spec flags; the Flash-Next MTP heads
|
||||
> do not load on this loader family — see MTP Draft Head).
|
||||
> **PRIOR / ROLLBACK TARGET since 2026-09-11** — frank-large moved to
|
||||
> [[halogen-flash-server]] (peonist-ai halogen-flash-server 0.5.8 serving
|
||||
> Qwen3.8-Flash-Next w4b), which upstream measures at roughly 4x this
|
||||
> runtime's prefill. Everything below documents the llama.cpp era and stays
|
||||
> accurate for a rollback: the payload is still at
|
||||
> `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX`, the binary
|
||||
> at `/home/dank/GitPackages/kingjones-ROCmFPX/build`, and the unit, env and
|
||||
> serve script are staged in `serverAppConfigs/.AGENTTRASH/` (`.llama.cpp`).
|
||||
> Rolling back also requires removing the halogen Quadlet from
|
||||
> `~/.config/containers/systemd/` so it stops claiming port 7770.
|
||||
>
|
||||
> Ran 2026-08-30 → 2026-09-11 on the model card's **pinned runtime**: the
|
||||
> `kingjones30/ROCmFPX` fork. Text-only tier; the agentionai f16 vision
|
||||
> tower was paired for mmproj and loads clean. Speculative decoding is OFF
|
||||
> (the card ships no spec flags; the Flash-Next MTP heads do not load on
|
||||
> this loader family — see MTP Draft Head).
|
||||
|
||||
`frank-large` (port 7770) serves
|
||||
`Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf` (single file, 113.47 GiB)
|
||||
|
||||
@@ -121,9 +121,17 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
|
||||
Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
|
||||
rollback record)
|
||||
- [[kingjones-strix-rocmfp4]] — the ACTIVE frank-large runbook since
|
||||
2026-08-30: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF (177B params,
|
||||
113.47 GiB) on the card's pinned kingjones30/ROCmFPX build
|
||||
- [[halogen-flash-server]] — the ACTIVE frank-large runbook since
|
||||
2026-09-11: peonist-ai/halogen-flash-server 0.5.8 serving
|
||||
peonist-ai/halogen-qwen3.8-flash-next (w4b) as a rootless Quadlet
|
||||
container on Dankclaw, port 7770 — flags, memory model, client contract
|
||||
- [[dankclaw-iommu-translation-off]] — why Dankclaw boots with
|
||||
`amd_iommu=off` since 2026-09-11 (13-16 % prefill), what it cost
|
||||
(the NPU endpoint), and how to revert
|
||||
- [[kingjones-strix-rocmfp4]] — the PRIOR frank-large runbook
|
||||
(2026-08-30 → 2026-09-11) and the rollback target:
|
||||
kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF on the card's
|
||||
pinned kingjones30/ROCmFPX build
|
||||
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
|
||||
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
@@ -135,10 +143,10 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[Dankserver System Specs]] — the RTX 3090 box hardware
|
||||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large)
|
||||
- [[Dankclaw Improvements]] — host-level changes on the Strix Halo box
|
||||
- [[frank-npu]] — ACTIVE FLM NPU sight+sound runbook on Dankclaw:
|
||||
Gemma4 E4B vision+audio pack on the XDNA2 NPU, port 7771; whole local
|
||||
registry served on one port with per-request model routing; includes the
|
||||
IOMMU/SVA root cause and the `iommu=pt` fix behind its first-deploy crash
|
||||
- [[frank-npu]] — the RETIRED FLM NPU endpoint on Dankclaw (2026-09-11):
|
||||
Gemma4 E4B vision+audio pack on the XDNA2 NPU, port 7771; kept as the
|
||||
retirement record, including the IOMMU/SVA err=-19 root cause and
|
||||
where every dependent route moved
|
||||
|
||||
### Benchmarks and Sizing
|
||||
|
||||
|
||||
@@ -212,7 +212,7 @@
|
||||
{
|
||||
"name": "frank-large",
|
||||
"category": "admin",
|
||||
"description": "The large LLM (27B) on Dankclaw.",
|
||||
"description": "The large LLM (Qwen3.8-Flash-Next, halogen) on Dankclaw.",
|
||||
"url": "",
|
||||
"docs": "",
|
||||
"probe": "127.0.0.1:7770",
|
||||
|
||||
@@ -18,15 +18,23 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
|
||||
@test "deployment planner maps large-model changes to the large model only" {
|
||||
run python3 "$PLANNER" --target dankclaw --paths \
|
||||
scripts/frank-large.env scripts/frank-large.service
|
||||
dankclaw/frank-large/quadlet/frank-large.container
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$output" = "frank-large" ]
|
||||
}
|
||||
|
||||
@test "shared chat-template changes map to the selected host's inference service" {
|
||||
@test "the retired NPU endpoint is absent from the deployment inventory" {
|
||||
! grep -q '"frank-npu"' "$PLANNER"
|
||||
! grep -q 'frank_npu\|frank-npu' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "shared chat-template changes map only to the service that reads it" {
|
||||
# frank-large runs halogen-flash-server, which owns its own chat handling;
|
||||
# frank-small is the only consumer of the tracked template now, so a
|
||||
# template edit is a no-op on Dankclaw.
|
||||
run python3 "$PLANNER" --target dankclaw --paths scripts/chat-template.jinja
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$output" = "frank-large" ]
|
||||
[ -z "$output" ]
|
||||
|
||||
run python3 "$PLANNER" --target dankserver --paths scripts/chat-template.jinja
|
||||
[ "$status" -eq 0 ]
|
||||
@@ -68,10 +76,6 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
run python3 "$PLANNER" --target dankclaw --paths dankclaw/hermes/config.yaml.example
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$output" = "hermes" ]
|
||||
|
||||
run python3 "$PLANNER" --target dankclaw --paths scripts/serve-frank-npu.sh scripts/frank-npu.env
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$output" = "frank-npu" ]
|
||||
}
|
||||
|
||||
@test "homepage is owned by Dankclaw's consolidated proxy" {
|
||||
|
||||
Executable
+95
@@ -0,0 +1,95 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the frank-large quadlet on dankclaw.
|
||||
# frank-large became a rootless Quadlet container on 2026-09-11 when
|
||||
# halogen-flash-server replaced the llama.cpp/kingjones runtime. This seam pins
|
||||
# the properties that make it work and that no reviewer can see by reading the
|
||||
# flags alone: GPU device access and the image's required podman posture (the
|
||||
# engine maps the checkpoint and registers it with the GPU in place), the
|
||||
# single published port that keeps every client pointed where it has always
|
||||
# pointed, the read-write weights mount the first-start download needs, boot
|
||||
# persistence, and a deploy handler that retires the old root unit without
|
||||
# deleting anything.
|
||||
|
||||
load helpers
|
||||
|
||||
UNIT="$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
|
||||
@test "frank-large quadlet exists under dankclaw/frank-large/quadlet/" {
|
||||
[ -f "$UNIT" ]
|
||||
}
|
||||
|
||||
@test "frank-large unit is boot-persistent (WantedBy=default.target)" {
|
||||
# Generated units cannot be enabled on this host's podman 5.x v1 quadlet
|
||||
# generator; [Install] plus linger is the entire boot-persistence story.
|
||||
grep -q '^WantedBy=default.target$' "$UNIT"
|
||||
}
|
||||
|
||||
@test "frank-large unit is self-healing" {
|
||||
grep -q '^Restart=on-failure$' "$UNIT"
|
||||
grep -q '^RestartSec=' "$UNIT"
|
||||
}
|
||||
|
||||
@test "frank-large image is pinned to an exact version tag" {
|
||||
grep -Eq '^Image=localhost/peonist-ai/halogen-flash-server:[0-9]+\.[0-9]+\.[0-9]+$' "$UNIT"
|
||||
}
|
||||
|
||||
@test "frank-large can reach the GPU (kfd + DRM render node)" {
|
||||
# Without both devices the engine cannot map the checkpoint at all; the
|
||||
# failure is a load error, not a degraded mode.
|
||||
grep -q '^AddDevice=/dev/kfd$' "$UNIT"
|
||||
grep -q '^AddDevice=/dev/dri$' "$UNIT"
|
||||
}
|
||||
|
||||
@test "frank-large runs with the podman posture the image requires" {
|
||||
# The image documents --ipc=host, an unconfined seccomp profile and an
|
||||
# unlimited memlock as hard requirements.
|
||||
grep -q -- '--ipc=host' "$UNIT"
|
||||
grep -q -- '--security-opt seccomp=unconfined' "$UNIT"
|
||||
grep -q '^Ulimit=memlock=-1:-1$' "$UNIT"
|
||||
}
|
||||
|
||||
@test "frank-large publishes exactly one port and it is the client port" {
|
||||
# 8731 is the OpenAI front-end; the engine on 8730 is unauthenticated and
|
||||
# must never be published.
|
||||
[ "$(grep -c '^PublishPort=' "$UNIT")" -eq 1 ]
|
||||
grep -q '^PublishPort=7770:8731$' "$UNIT"
|
||||
}
|
||||
|
||||
@test "frank-large keeps the fleet identity every client already points at" {
|
||||
# The id at /v1/models, in /health and in every response body.
|
||||
grep -q '^Environment=HALOGEN_MODEL_ID=frank-large$' "$UNIT"
|
||||
}
|
||||
|
||||
@test "the weights mount is writable, because the first start downloads into it" {
|
||||
grep -q '^Volume=/home/dank/models/halogen:/models$' "$UNIT"
|
||||
! grep -q '^Volume=.*:/models:ro$' "$UNIT"
|
||||
grep -q '^Environment=HALOGEN_DOWNLOAD=' "$UNIT"
|
||||
}
|
||||
|
||||
@test "frank-large unit carries no secret material" {
|
||||
! grep -Eq '^(Environment|EnvironmentFile)=.*(TOKEN|PASSWORD|SECRET|KEY)' "$UNIT"
|
||||
}
|
||||
|
||||
@test "the deploy handler retires the old root unit without deleting it" {
|
||||
grep -q 'retire_root_llm_large' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl stop frank-large.service' "$REMOTE_DEPLOY"
|
||||
# Never `rm`: the retired unit moves into the home trash so the llama.cpp
|
||||
# era stays recoverable.
|
||||
grep -q 'mv -- /etc/systemd/system/frank-large.service' "$REMOTE_DEPLOY"
|
||||
! grep -qE '(^|[^[:alnum:]_])rm (-[a-zA-Z]+ )?/etc/systemd/system/frank-large' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the deploy handler never tries to enable a generated unit" {
|
||||
# `systemctl --user enable frank-large.service` fails on this host
|
||||
# ("transient or generated"); the deploy must install and restart instead.
|
||||
! grep -q 'systemctl --user enable frank-large.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'install_quadlet frank-large dankclaw/frank-large/quadlet frank-large' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl --user restart frank-large.service' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the deploy handler gates the first start on the download, not on a failure" {
|
||||
grep -q 'wait_for_llm_large_ready' "$REMOTE_DEPLOY"
|
||||
grep -q 'qwen38-flash-next-w4b.hgn' "$REMOTE_DEPLOY"
|
||||
grep -q '127.0.0.1:7770/health' "$REMOTE_DEPLOY"
|
||||
}
|
||||
Reference in New Issue
Block a user