feat(llm): swap frank-large to halogen-flash-server, retire frank-npu
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b .hgn 115.55 GiB + quality overlay). Upstream measures this engine at roughly 4x the prefill throughput of the llama.cpp family on the same gfx1151 silicon, and prefill dominates any prompt with real context. Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/), engine on container loopback :8730 (never published), OpenAI front-end :8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB, conservative for a shared 121 GiB host), 4 slots, host reserve 24, vision tower on, weights pinned, all other flags at vendor defaults. Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable /home/dank/models/halogen. The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/ rather than deleted; the STRIX payload and the ROCmFPX build stay on disk as the rollback target. deploy_llm_large becomes a user-quadlet handler: it retires the old ROOT unit (moving its checkout symlink into the home trash, never rm) and gates the first start on the download rather than reporting a failure for a fetch that is working exactly as designed. frank-npu is retired in the same change. The kernel cmdline moves from iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured, and that takes the XDNA2 NPU with it: without PASID/SVA the driver cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are repointed (Hermes vision -> frank-large, Hermes title generation -> frank-small, harness vision -> frank-large, tiny -> frank-small, task -> frank-large:medium) and its provider blocks are removed from both the harness config and Hermes's live config. Audio input leaves the fleet with no replacement; image input survives on halogen's vision tower. Docs: new halogen-flash-server runbook, new IOMMU decision record, and frank-npu rewritten as a retirement record. AGENTS.md topology, runtime sections, benchmarking policy (frank-large is a user unit now) and the harness model-config section are corrected.
This commit is contained in:
Frank
@@ -27,13 +27,13 @@ prefix, reorganized in follow-up commits). The retired target checkouts
|
||||
archived. All services now run from `/home/dank/serverAppConfigs` on each
|
||||
host.
|
||||
|
||||
### LLM fleet topology (2026-08-30)
|
||||
### LLM fleet topology (2026-09-11)
|
||||
|
||||
| Target | Runtime | Service | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | FreeToken v0.1.2 | frank-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 |
|
||||
| Dankclaw | kingjones30/ROCmFPX llama.cpp (card-pinned fork) | frank-large | kingjones777 Qwen3.8-Flash-Next-ROCmFP4-STRIX (113.47 GiB single GGUF, Q4_0_ROCMFP4_STRIX, PLE streams via mmap), 401408 ctx / 2 slots, fa on, no spec decode, reasoning on | 7770 |
|
||||
| Dankclaw | FastFlowLM (FLM) v1.0.1 (`flm serve`, dpkg `fastflowlm`) on the XDNA2 NPU | frank-npu | Gemma4-E4B-IT-NPU2 pack (vision + audio towers); whole local registry served on one port, per-request routing by exact FLM tag; `iommu=pt` boot arg mandatory (SVA bind needs enabled IOMMU) | 7771 |
|
||||
| Dankclaw | halogen-flash-server 0.5.8 (rootless Quadlet container) | frank-large | peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 4 slots, pool 262144, vision tower on, greedy for silent clients) | 7770 |
|
||||
| Dankclaw | *(retired 2026-09-11)* FastFlowLM v1.0.1 on the XDNA2 NPU | frank-npu | Gemma4-E4B-IT-NPU2 pack (vision + audio). Retired with the `amd_iommu=off` decision: the driver's SVA/PASID bind needs an enabled IOMMU | 7771 |
|
||||
|
||||
History: the 2026-08-17 redo retired the pipeline-parallel split and the
|
||||
FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on
|
||||
@@ -52,8 +52,13 @@ Qwen3.6-35B-A3B-FP8. On 2026-08-30 both services swapped again:
|
||||
Dankclaw to the kingjones STRIX model on the card's pinned
|
||||
kingjones30/ROCmFPX runtime, Dankserver to the Huihui abliterated
|
||||
FP8 checkpoint on FreeToken. On 2026-08-30 the NPU endpoint returned as
|
||||
`frank-npu` — FLM v1.0.1 on XDNA2 serving sight+sound (see the
|
||||
Runtime & Troubleshooting section and its runbook). Runbooks:
|
||||
`frank-npu` — FLM v1.0.1 on XDNA2 serving sight+sound. On 2026-09-11
|
||||
Dankclaw's large service swapped runtime *and* model: the kingjones STRIX
|
||||
llama.cpp deployment gave way to halogen-flash-server 0.5.8 serving
|
||||
peonist-ai/halogen-qwen3.8-flash-next, now a rootless Quadlet container, and
|
||||
the kernel cmdline moved from `iommu=pt` to `amd_iommu=off` for 13–16 % of
|
||||
prefill — which retired `frank-npu` for good (see the Runtime &
|
||||
Troubleshooting section and its runbook). Runbooks:
|
||||
[serverAppConfigs-docs/LLM/index.md](serverAppConfigs-docs/LLM/index.md).
|
||||
Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`.
|
||||
|
||||
@@ -128,8 +133,8 @@ selected service and records its successful revision outside the checkout.
|
||||
|
||||
A documentation-only change is a no-op. The LLM services are selected only by
|
||||
their own unit/env/serve files or the shared chat template, so unrelated
|
||||
container changes cannot interrupt inference. `frank-npu` is relink-only
|
||||
when explicitly selected.
|
||||
container changes cannot interrupt inference. `frank-large` is selected by its
|
||||
Quadlet file under `dankclaw/frank-large/quadlet/`.
|
||||
|
||||
Set `FRANK_SUDO_PW` for headless runs on a host without passwordless sudo
|
||||
(remote sudo reads it from stdin via `sudo -S`). The value is never stored
|
||||
@@ -280,92 +285,119 @@ ALWAYS bench using llama-bench for llama.cpp or vllm bench for vllm runtimes.
|
||||
DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE OFFICIAL BENCHES. ASK THE USER BEFORE DEVIATING FROM THIS DIRECTIVE.
|
||||
|
||||
1. Stop the service before any benchmark:
|
||||
`sudo systemctl stop frank-large` (Dankclaw) or
|
||||
`sudo systemctl stop frank-small` (Dankserver).
|
||||
`systemctl --user stop frank-large` (Dankclaw — a **rootless** Quadlet
|
||||
service, so drop the `sudo`) or `sudo systemctl stop frank-small`
|
||||
(Dankserver — still a system unit).
|
||||
Then mask it to prevent the self-healing unit from restarting:
|
||||
`sudo systemctl mask frank-large` (Dankclaw).
|
||||
**CRITICAL WARNING:** Do NOT use `systemctl disable` on these service units.
|
||||
`systemctl --user mask frank-large` (Dankclaw) or
|
||||
`sudo systemctl mask frank-small` (Dankserver).
|
||||
**CRITICAL WARNING:** Do NOT use `systemctl disable` on the **system** units.
|
||||
On these hosts, `systemctl disable` doesn't just remove symlinks — it **deletes
|
||||
the unit file itself** from `/etc/systemd/system/`. You'd have to manually
|
||||
copy the unit file back from this repo, run `daemon-reload`, then enable and
|
||||
start. Mask (`systemctl mask`) creates a symlink to `/dev/null` instead, which
|
||||
is safe and reversible.
|
||||
2. Run the bench (`llama-bench` for llama.cpp / vllm bench for vLLM).
|
||||
Scratch ports are not used for tests or benches on the LLM hosts — a
|
||||
test process cannot share the single GPU with the running service anyway
|
||||
(the model + KV fill the card), so changes are validated directly against
|
||||
the service itself.
|
||||
2. Run the bench with the runtime's own harness: `llama-bench` for llama.cpp,
|
||||
`vllm bench` for vLLM, and for frank-large's halogen-flash-server the
|
||||
image's own `bench`/`sweep` modes (`bench [drafters] [max_tokens] [effort]
|
||||
[reps]`) — `llama-bench` and `vllm bench` do not apply to it. Scratch ports are not used for tests or benches on the LLM
|
||||
hosts — a test process cannot share the single GPU with the running service
|
||||
anyway (the model + KV fill the card), so changes are validated directly
|
||||
against the service itself.
|
||||
3. Restart the service when the bench completes: unmask, then start:
|
||||
`sudo systemctl unmask frank-large && sudo systemctl start frank-large` (Dankclaw).
|
||||
`systemctl --user unmask frank-large && systemctl --user start frank-large`
|
||||
(Dankclaw) or `sudo systemctl unmask frank-small && sudo systemctl start
|
||||
frank-small` (Dankserver).
|
||||
4. Verify health before reporting:
|
||||
`curl http://192.168.0.235:7770/health` or
|
||||
`curl http://192.168.0.143:7770/health`.
|
||||
|
||||
## Pi Model Config (`~/.pi/agent/models.json`)
|
||||
## Model Config (harness)
|
||||
|
||||
The Pi agent reaches the fleet through `/home/dank/.pi/agent/models.json`.
|
||||
This file lives outside the repository. Each provider is one host endpoint.
|
||||
Each model `id` is the served alias. The two aliases are
|
||||
`frank-small` (Dankserver `:7770`) and `frank-large`
|
||||
(Dankclaw `:7770`).
|
||||
The harness reaches the fleet through `/home/dank/.omp/agent/models.yml` —
|
||||
verified live 2026-09-11; the older `/home/dank/.pi/agent/models.json` path
|
||||
this section used to name no longer exists. Both files live outside the
|
||||
repository. Each provider is one host endpoint and each model `id` is the
|
||||
served alias:
|
||||
|
||||
The model `id` values are stable. The file
|
||||
`/home/dank/.pi/agent/settings.json` references them. Two resolution rules
|
||||
apply (verified in `@earendil-works/pi-coding-agent` `model-resolver.js`):
|
||||
| alias | endpoint |
|
||||
|---|---|
|
||||
| `frank-small` | Dankserver `192.168.0.235:7770` |
|
||||
| `frank-large` | Dankclaw `192.168.0.143:7770` |
|
||||
| `gemma4-it:e4b` | *(removed 2026-09-11 with `frank-npu`)* |
|
||||
|
||||
Two resolution rules apply (verified in `@earendil-works/pi-coding-agent`
|
||||
`model-resolver.js`):
|
||||
|
||||
- The default model resolves as a `defaultProvider` + `defaultModel` pair.
|
||||
The `defaultProvider` must name the provider that serves the `defaultModel`.
|
||||
- A subagent override uses a bare model `id`. A bare `id` resolves only when
|
||||
it is unique across all providers.
|
||||
|
||||
When you rename a service or move a model to a new host, update
|
||||
`models.json`. Update the provider key, the model `id`, `contextWindow`, and
|
||||
`input`. If the default model moved hosts, also update `defaultProvider` in
|
||||
`settings.json`. Set `contextWindow` to the per-slot context (both hosts
|
||||
`262144`). The full schema is in the agent knowledge base page
|
||||
`~/.pi/agent/docs/pi/models.md`.
|
||||
Per-role routing lives in `/home/dank/.omp/agent/config.yml` under
|
||||
`modelRoles` (`smol`, `tiny`, `task`, `plan`, `designer`, `advisor`, `slow`,
|
||||
`vision`, …). When you rename a service, retire an endpoint, or move a model
|
||||
to a new host, update the provider block, the model `id`, `contextWindow`,
|
||||
`maxTokens` and `input` in `models.yml`, then fix every `modelRoles` entry
|
||||
that named the old endpoint — a role pointing at a removed provider fails at
|
||||
call time. `frank-large` declares `input: [text, image]` because halogen runs
|
||||
its vision tower.
|
||||
|
||||
## Runtime & Troubleshooting
|
||||
|
||||
### frank-npu — FLM NPU sight+sound endpoint (Dankclaw, ACTIVE)
|
||||
### frank-npu — FLM NPU endpoint (Dankclaw, RETIRED 2026-09-11)
|
||||
|
||||
The third inference endpoint on Dankclaw. The `flm serve` runtime (dpkg
|
||||
package `fastflowlm`, v1.0.1, binary `/usr/bin/flm`) serves the **whole local
|
||||
model registry** (36 tags at first deploy) on a single port — 7771 — and
|
||||
routes chat completions per request by the exact FLM registry tag in the
|
||||
`model` field. Unknown tags do NOT 404: `/v1/chat/completions` answers HTTP 200
|
||||
with `{"error":"Failed to load <tag> model!"}` and no choices (status-only
|
||||
clients silently get nothing), so clients must send exact registry tags, never
|
||||
service aliases. Primary
|
||||
model = Gemma4 E4B IT NPU2 pack (vision + audio towers); the uncensored
|
||||
candidate (`qwythos:9b`) is text-only and fails the modality requirement, so
|
||||
this endpoint trades "uncensored" for "sight + sound". The NPU is a
|
||||
single-lane device — generations serialize FIFO; treat this as a
|
||||
background/triage endpoint, not an interactive multi-user chat.
|
||||
Retired by operator decision together with the kernel cmdline change that
|
||||
bought 13–16 % of prefill for `frank-large`: with `amd_iommu=off` the XDNA2
|
||||
driver cannot bind SVA/PASID, so every open of `/dev/accel/accel0` fails with
|
||||
`err=-19`. The unit is stopped and masked, the service files are staged in
|
||||
`.AGENTTRASH/` (`*.flm-npu`), and the providers that named it were removed
|
||||
from both the harness and Hermes. **Audio input left the fleet with it and
|
||||
has no replacement**; image input survives on `frank-large` through halogen's
|
||||
vision tower. Route map: Hermes `vision` → `frank-large`, Hermes
|
||||
`title_generation` → `frank-small`, harness `vision` → `frank-large`, harness
|
||||
`tiny` → `frank-small`, harness `task` → `frank-large` at medium effort.
|
||||
|
||||
Hard requirements verified at first deploy (2026-08-30):
|
||||
The retired runbook keeps the full record, including the original IOMMU/SVA
|
||||
root cause: [serverAppConfigs-docs/LLM/frank-npu.md](serverAppConfigs-docs/LLM/frank-npu.md).
|
||||
The current kernel posture and its revert path:
|
||||
[serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md](serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md).
|
||||
|
||||
- **IOMMU must be enabled.** With `iommu=off` on the kernel cmdline every
|
||||
open of `/dev/accel/accel0` fails with `err=-19`: the XDNA2 driver's SVA
|
||||
bind needs PASID/SVA, which requires enabled IOMMU hardware. Fix applied:
|
||||
`iommu=pt` in `GRUB_CMDLINE_LINUX_DEFAULT` + reboot (passthrough keeps the
|
||||
iGPU default domain identity-mapped — zero DMA-translation overhead for
|
||||
frank-large — while enabling IOMMU + PASID; PASID domains are
|
||||
paged regardless of the default-domain type). Never set `iommu=off` on
|
||||
this host again; it silently kills the NPU.
|
||||
- Health probe = `GET /v1/version` (FLM has no `/health` route).
|
||||
- Unit: `User=dank`, `LimitMEMLOCK=infinity` (FLM requires unlimited
|
||||
memlock), `Restart=on-failure`, `RestartSec=10`. Deploy via
|
||||
`scripts/deploy.sh dankclaw --service frank-npu`.
|
||||
- Verified live at first deploy: image probe (red PNG as `image_url`
|
||||
data-URI → correct color answer, vision tower consumed the pixels) and an
|
||||
audio control-pair probe (tone WAV via `input_audio` consumed ~22
|
||||
audio-tower tokens vs a text-only control).
|
||||
### halogen-flash-server (Dankclaw, ACTIVE since 2026-09-11)
|
||||
|
||||
Runbook: [serverAppConfigs-docs/LLM/frank-npu.md](serverAppConfigs-docs/LLM/frank-npu.md).
|
||||
### kingjones Qwen3.8-Flash-Next STRIX on kingjones30/ROCmFPX (Dankclaw, ACTIVE)
|
||||
The large service runs `ghcr.io/peonist-ai/halogen-flash-server:0.5.8` as a
|
||||
**rootless Quadlet container**
|
||||
(`dankclaw/frank-large/quadlet/frank-large.container` → generated
|
||||
`frank-large.service` in the user manager), serving
|
||||
`peonist-ai/halogen-qwen3.8-flash-next`: Qwen3.8-Flash-Next as a 115.55 GiB
|
||||
w4b `.hgn` checkpoint plus its quality overlay sidecar. One container runs
|
||||
both halves — the engine on container loopback :8730 (unauthenticated, never
|
||||
published) and the OpenAI front-end on :8731 published as host **7770** —
|
||||
with `HALOGEN_MODEL_ID=frank-large`, so no client wiring changed.
|
||||
|
||||
The large service serves
|
||||
Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=262144`
|
||||
(conservative for a shared host), `HALOGEN_KV_SLOTS=4`,
|
||||
`HALOGEN_HOST_RESERVE_GIB=24`, `HALOGEN_VISION_TOWER=1`, and
|
||||
`HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next` for the first-start
|
||||
fetch; everything else is upstream's default. The weights stay pinned
|
||||
(~68 GiB) and the kernel counts those locked pages as reclaimable page cache,
|
||||
so `free`, `MemAvailable` and anything built on them (Grafana) overstate free
|
||||
memory by the size of the model — believe the server's own startup ledger,
|
||||
not the dashboards.
|
||||
|
||||
Client-visible contract: **no server-side sampler defaults** (a request that
|
||||
sends no `temperature` decodes greedy); the per-request budget defaults to
|
||||
8192 and **includes thinking**, so a budget that runs out mid-thought returns
|
||||
`finish_reason: "length"` with empty content; the cap is 65536; and a request
|
||||
reserves prompt + `max_tokens` positions against the KV pool, so a request
|
||||
larger than the pool is never admitted — it queues and then fails with
|
||||
`503 engine_busy` after the 3600 s queue timeout.
|
||||
|
||||
Runbook: [serverAppConfigs-docs/LLM/halogen-flash-server.md](serverAppConfigs-docs/LLM/halogen-flash-server.md).
|
||||
|
||||
### kingjones Qwen3.8-Flash-Next STRIX on kingjones30/ROCmFPX (Dankclaw, PRIOR — rollback target)
|
||||
|
||||
The large service served
|
||||
`kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` (public, 113.47
|
||||
GiB single file, `Q4_0_ROCMFP4_STRIX` = file_type 105) from
|
||||
`/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/` with
|
||||
@@ -391,8 +423,9 @@ kill, only dmesg evidence; llama.cpp's "use_mmap=1" log line is
|
||||
correct here — mmap pages ARE page cache) and **never pin the PLE
|
||||
tensor with `-ot`** (23 → 13.4 t/s). The 51.2B-param PLE table never
|
||||
enters GPU memory (~20 bytes/token-file per 500 decoded). On-host file
|
||||
sha256 verified == HF LFS oid `c6770d74...49fa`. Env
|
||||
`scripts/frank-large.env`, serve `scripts/serve-frank-large.sh`.
|
||||
sha256 verified == HF LFS oid `c6770d74...49fa`. Its env and serve script
|
||||
(`scripts/frank-large.env`, `scripts/serve-frank-large.sh`) are staged in
|
||||
`.AGENTTRASH/`, suffixed `.llama.cpp`.
|
||||
Runbook:
|
||||
[serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md](serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md).
|
||||
|
||||
@@ -548,7 +581,7 @@ graph TD
|
||||
flags. A different outcome isolates the env file as the cause.
|
||||
3. Run gdb for the backtrace with this command:
|
||||
```
|
||||
gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-large.sh
|
||||
gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-small.sh
|
||||
```
|
||||
`llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost.
|
||||
Trust gdb, not the log.
|
||||
|
||||
Reference in New Issue
Block a user