diff --git a/scripts/frank-large.env b/.AGENTTRASH/frank-large.env.llama.cpp similarity index 100% rename from scripts/frank-large.env rename to .AGENTTRASH/frank-large.env.llama.cpp diff --git a/scripts/frank-large.service b/.AGENTTRASH/frank-large.service.llama.cpp similarity index 100% rename from scripts/frank-large.service rename to .AGENTTRASH/frank-large.service.llama.cpp diff --git a/scripts/frank-npu.env b/.AGENTTRASH/frank-npu.env.flm-npu similarity index 100% rename from scripts/frank-npu.env rename to .AGENTTRASH/frank-npu.env.flm-npu diff --git a/scripts/frank-npu.service b/.AGENTTRASH/frank-npu.service.flm-npu similarity index 100% rename from scripts/frank-npu.service rename to .AGENTTRASH/frank-npu.service.flm-npu diff --git a/scripts/serve-frank-large.sh b/.AGENTTRASH/serve-frank-large.sh.llama.cpp similarity index 100% rename from scripts/serve-frank-large.sh rename to .AGENTTRASH/serve-frank-large.sh.llama.cpp diff --git a/scripts/serve-frank-npu.sh b/.AGENTTRASH/serve-frank-npu.sh.flm-npu similarity index 100% rename from scripts/serve-frank-npu.sh rename to .AGENTTRASH/serve-frank-npu.sh.flm-npu diff --git a/AGENTS.md b/AGENTS.md index ca5ca2b..b7690fe 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,13 +27,13 @@ prefix, reorganized in follow-up commits). The retired target checkouts archived. All services now run from `/home/dank/serverAppConfigs` on each host. -### LLM fleet topology (2026-08-30) +### LLM fleet topology (2026-09-11) | Target | Runtime | Service | Model | Port | |--------|---------|---------|-------|------| | Dankserver | FreeToken v0.1.2 | frank-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 | -| Dankclaw | kingjones30/ROCmFPX llama.cpp (card-pinned fork) | frank-large | kingjones777 Qwen3.8-Flash-Next-ROCmFP4-STRIX (113.47 GiB single GGUF, Q4_0_ROCMFP4_STRIX, PLE streams via mmap), 401408 ctx / 2 slots, fa on, no spec decode, reasoning on | 7770 | -| Dankclaw | FastFlowLM (FLM) v1.0.1 (`flm serve`, dpkg `fastflowlm`) on the XDNA2 NPU | frank-npu | Gemma4-E4B-IT-NPU2 pack (vision + audio towers); whole local registry served on one port, per-request routing by exact FLM tag; `iommu=pt` boot arg mandatory (SVA bind needs enabled IOMMU) | 7771 | +| Dankclaw | halogen-flash-server 0.5.8 (rootless Quadlet container) | frank-large | peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 4 slots, pool 262144, vision tower on, greedy for silent clients) | 7770 | +| Dankclaw | *(retired 2026-09-11)* FastFlowLM v1.0.1 on the XDNA2 NPU | frank-npu | Gemma4-E4B-IT-NPU2 pack (vision + audio). Retired with the `amd_iommu=off` decision: the driver's SVA/PASID bind needs an enabled IOMMU | 7771 | History: the 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on @@ -52,8 +52,13 @@ Qwen3.6-35B-A3B-FP8. On 2026-08-30 both services swapped again: Dankclaw to the kingjones STRIX model on the card's pinned kingjones30/ROCmFPX runtime, Dankserver to the Huihui abliterated FP8 checkpoint on FreeToken. On 2026-08-30 the NPU endpoint returned as -`frank-npu` — FLM v1.0.1 on XDNA2 serving sight+sound (see the -Runtime & Troubleshooting section and its runbook). Runbooks: +`frank-npu` — FLM v1.0.1 on XDNA2 serving sight+sound. On 2026-09-11 +Dankclaw's large service swapped runtime *and* model: the kingjones STRIX +llama.cpp deployment gave way to halogen-flash-server 0.5.8 serving +peonist-ai/halogen-qwen3.8-flash-next, now a rootless Quadlet container, and +the kernel cmdline moved from `iommu=pt` to `amd_iommu=off` for 13–16 % of +prefill — which retired `frank-npu` for good (see the Runtime & +Troubleshooting section and its runbook). Runbooks: [serverAppConfigs-docs/LLM/index.md](serverAppConfigs-docs/LLM/index.md). Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`. @@ -128,8 +133,8 @@ selected service and records its successful revision outside the checkout. A documentation-only change is a no-op. The LLM services are selected only by their own unit/env/serve files or the shared chat template, so unrelated -container changes cannot interrupt inference. `frank-npu` is relink-only -when explicitly selected. +container changes cannot interrupt inference. `frank-large` is selected by its +Quadlet file under `dankclaw/frank-large/quadlet/`. Set `FRANK_SUDO_PW` for headless runs on a host without passwordless sudo (remote sudo reads it from stdin via `sudo -S`). The value is never stored @@ -280,92 +285,119 @@ ALWAYS bench using llama-bench for llama.cpp or vllm bench for vllm runtimes. DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE OFFICIAL BENCHES. ASK THE USER BEFORE DEVIATING FROM THIS DIRECTIVE. 1. Stop the service before any benchmark: - `sudo systemctl stop frank-large` (Dankclaw) or - `sudo systemctl stop frank-small` (Dankserver). + `systemctl --user stop frank-large` (Dankclaw — a **rootless** Quadlet + service, so drop the `sudo`) or `sudo systemctl stop frank-small` + (Dankserver — still a system unit). Then mask it to prevent the self-healing unit from restarting: - `sudo systemctl mask frank-large` (Dankclaw). - **CRITICAL WARNING:** Do NOT use `systemctl disable` on these service units. + `systemctl --user mask frank-large` (Dankclaw) or + `sudo systemctl mask frank-small` (Dankserver). + **CRITICAL WARNING:** Do NOT use `systemctl disable` on the **system** units. On these hosts, `systemctl disable` doesn't just remove symlinks — it **deletes the unit file itself** from `/etc/systemd/system/`. You'd have to manually copy the unit file back from this repo, run `daemon-reload`, then enable and start. Mask (`systemctl mask`) creates a symlink to `/dev/null` instead, which is safe and reversible. -2. Run the bench (`llama-bench` for llama.cpp / vllm bench for vLLM). - Scratch ports are not used for tests or benches on the LLM hosts — a - test process cannot share the single GPU with the running service anyway - (the model + KV fill the card), so changes are validated directly against - the service itself. +2. Run the bench with the runtime's own harness: `llama-bench` for llama.cpp, + `vllm bench` for vLLM, and for frank-large's halogen-flash-server the + image's own `bench`/`sweep` modes (`bench [drafters] [max_tokens] [effort] + [reps]`) — `llama-bench` and `vllm bench` do not apply to it. Scratch ports are not used for tests or benches on the LLM + hosts — a test process cannot share the single GPU with the running service + anyway (the model + KV fill the card), so changes are validated directly + against the service itself. 3. Restart the service when the bench completes: unmask, then start: - `sudo systemctl unmask frank-large && sudo systemctl start frank-large` (Dankclaw). + `systemctl --user unmask frank-large && systemctl --user start frank-large` + (Dankclaw) or `sudo systemctl unmask frank-small && sudo systemctl start + frank-small` (Dankserver). 4. Verify health before reporting: `curl http://192.168.0.235:7770/health` or `curl http://192.168.0.143:7770/health`. -## Pi Model Config (`~/.pi/agent/models.json`) +## Model Config (harness) -The Pi agent reaches the fleet through `/home/dank/.pi/agent/models.json`. -This file lives outside the repository. Each provider is one host endpoint. -Each model `id` is the served alias. The two aliases are -`frank-small` (Dankserver `:7770`) and `frank-large` -(Dankclaw `:7770`). +The harness reaches the fleet through `/home/dank/.omp/agent/models.yml` — +verified live 2026-09-11; the older `/home/dank/.pi/agent/models.json` path +this section used to name no longer exists. Both files live outside the +repository. Each provider is one host endpoint and each model `id` is the +served alias: -The model `id` values are stable. The file -`/home/dank/.pi/agent/settings.json` references them. Two resolution rules -apply (verified in `@earendil-works/pi-coding-agent` `model-resolver.js`): +| alias | endpoint | +|---|---| +| `frank-small` | Dankserver `192.168.0.235:7770` | +| `frank-large` | Dankclaw `192.168.0.143:7770` | +| `gemma4-it:e4b` | *(removed 2026-09-11 with `frank-npu`)* | + +Two resolution rules apply (verified in `@earendil-works/pi-coding-agent` +`model-resolver.js`): - The default model resolves as a `defaultProvider` + `defaultModel` pair. The `defaultProvider` must name the provider that serves the `defaultModel`. - A subagent override uses a bare model `id`. A bare `id` resolves only when it is unique across all providers. -When you rename a service or move a model to a new host, update -`models.json`. Update the provider key, the model `id`, `contextWindow`, and -`input`. If the default model moved hosts, also update `defaultProvider` in -`settings.json`. Set `contextWindow` to the per-slot context (both hosts -`262144`). The full schema is in the agent knowledge base page -`~/.pi/agent/docs/pi/models.md`. +Per-role routing lives in `/home/dank/.omp/agent/config.yml` under +`modelRoles` (`smol`, `tiny`, `task`, `plan`, `designer`, `advisor`, `slow`, +`vision`, …). When you rename a service, retire an endpoint, or move a model +to a new host, update the provider block, the model `id`, `contextWindow`, +`maxTokens` and `input` in `models.yml`, then fix every `modelRoles` entry +that named the old endpoint — a role pointing at a removed provider fails at +call time. `frank-large` declares `input: [text, image]` because halogen runs +its vision tower. ## Runtime & Troubleshooting -### frank-npu — FLM NPU sight+sound endpoint (Dankclaw, ACTIVE) +### frank-npu — FLM NPU endpoint (Dankclaw, RETIRED 2026-09-11) -The third inference endpoint on Dankclaw. The `flm serve` runtime (dpkg -package `fastflowlm`, v1.0.1, binary `/usr/bin/flm`) serves the **whole local -model registry** (36 tags at first deploy) on a single port — 7771 — and -routes chat completions per request by the exact FLM registry tag in the -`model` field. Unknown tags do NOT 404: `/v1/chat/completions` answers HTTP 200 -with `{"error":"Failed to load model!"}` and no choices (status-only -clients silently get nothing), so clients must send exact registry tags, never -service aliases. Primary -model = Gemma4 E4B IT NPU2 pack (vision + audio towers); the uncensored -candidate (`qwythos:9b`) is text-only and fails the modality requirement, so -this endpoint trades "uncensored" for "sight + sound". The NPU is a -single-lane device — generations serialize FIFO; treat this as a -background/triage endpoint, not an interactive multi-user chat. +Retired by operator decision together with the kernel cmdline change that +bought 13–16 % of prefill for `frank-large`: with `amd_iommu=off` the XDNA2 +driver cannot bind SVA/PASID, so every open of `/dev/accel/accel0` fails with +`err=-19`. The unit is stopped and masked, the service files are staged in +`.AGENTTRASH/` (`*.flm-npu`), and the providers that named it were removed +from both the harness and Hermes. **Audio input left the fleet with it and +has no replacement**; image input survives on `frank-large` through halogen's +vision tower. Route map: Hermes `vision` → `frank-large`, Hermes +`title_generation` → `frank-small`, harness `vision` → `frank-large`, harness +`tiny` → `frank-small`, harness `task` → `frank-large` at medium effort. -Hard requirements verified at first deploy (2026-08-30): +The retired runbook keeps the full record, including the original IOMMU/SVA +root cause: [serverAppConfigs-docs/LLM/frank-npu.md](serverAppConfigs-docs/LLM/frank-npu.md). +The current kernel posture and its revert path: +[serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md](serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md). -- **IOMMU must be enabled.** With `iommu=off` on the kernel cmdline every - open of `/dev/accel/accel0` fails with `err=-19`: the XDNA2 driver's SVA - bind needs PASID/SVA, which requires enabled IOMMU hardware. Fix applied: - `iommu=pt` in `GRUB_CMDLINE_LINUX_DEFAULT` + reboot (passthrough keeps the - iGPU default domain identity-mapped — zero DMA-translation overhead for - frank-large — while enabling IOMMU + PASID; PASID domains are - paged regardless of the default-domain type). Never set `iommu=off` on - this host again; it silently kills the NPU. -- Health probe = `GET /v1/version` (FLM has no `/health` route). -- Unit: `User=dank`, `LimitMEMLOCK=infinity` (FLM requires unlimited - memlock), `Restart=on-failure`, `RestartSec=10`. Deploy via - `scripts/deploy.sh dankclaw --service frank-npu`. -- Verified live at first deploy: image probe (red PNG as `image_url` - data-URI → correct color answer, vision tower consumed the pixels) and an - audio control-pair probe (tone WAV via `input_audio` consumed ~22 - audio-tower tokens vs a text-only control). +### halogen-flash-server (Dankclaw, ACTIVE since 2026-09-11) -Runbook: [serverAppConfigs-docs/LLM/frank-npu.md](serverAppConfigs-docs/LLM/frank-npu.md). -### kingjones Qwen3.8-Flash-Next STRIX on kingjones30/ROCmFPX (Dankclaw, ACTIVE) +The large service runs `ghcr.io/peonist-ai/halogen-flash-server:0.5.8` as a +**rootless Quadlet container** +(`dankclaw/frank-large/quadlet/frank-large.container` → generated +`frank-large.service` in the user manager), serving +`peonist-ai/halogen-qwen3.8-flash-next`: Qwen3.8-Flash-Next as a 115.55 GiB +w4b `.hgn` checkpoint plus its quality overlay sidecar. One container runs +both halves — the engine on container loopback :8730 (unauthenticated, never +published) and the OpenAI front-end on :8731 published as host **7770** — +with `HALOGEN_MODEL_ID=frank-large`, so no client wiring changed. -The large service serves +Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=262144` +(conservative for a shared host), `HALOGEN_KV_SLOTS=4`, +`HALOGEN_HOST_RESERVE_GIB=24`, `HALOGEN_VISION_TOWER=1`, and +`HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next` for the first-start +fetch; everything else is upstream's default. The weights stay pinned +(~68 GiB) and the kernel counts those locked pages as reclaimable page cache, +so `free`, `MemAvailable` and anything built on them (Grafana) overstate free +memory by the size of the model — believe the server's own startup ledger, +not the dashboards. + +Client-visible contract: **no server-side sampler defaults** (a request that +sends no `temperature` decodes greedy); the per-request budget defaults to +8192 and **includes thinking**, so a budget that runs out mid-thought returns +`finish_reason: "length"` with empty content; the cap is 65536; and a request +reserves prompt + `max_tokens` positions against the KV pool, so a request +larger than the pool is never admitted — it queues and then fails with +`503 engine_busy` after the 3600 s queue timeout. + +Runbook: [serverAppConfigs-docs/LLM/halogen-flash-server.md](serverAppConfigs-docs/LLM/halogen-flash-server.md). + +### kingjones Qwen3.8-Flash-Next STRIX on kingjones30/ROCmFPX (Dankclaw, PRIOR — rollback target) + +The large service served `kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` (public, 113.47 GiB single file, `Q4_0_ROCMFP4_STRIX` = file_type 105) from `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/` with @@ -391,8 +423,9 @@ kill, only dmesg evidence; llama.cpp's "use_mmap=1" log line is correct here — mmap pages ARE page cache) and **never pin the PLE tensor with `-ot`** (23 → 13.4 t/s). The 51.2B-param PLE table never enters GPU memory (~20 bytes/token-file per 500 decoded). On-host file -sha256 verified == HF LFS oid `c6770d74...49fa`. Env -`scripts/frank-large.env`, serve `scripts/serve-frank-large.sh`. +sha256 verified == HF LFS oid `c6770d74...49fa`. Its env and serve script +(`scripts/frank-large.env`, `scripts/serve-frank-large.sh`) are staged in +`.AGENTTRASH/`, suffixed `.llama.cpp`. Runbook: [serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md](serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md). @@ -548,7 +581,7 @@ graph TD flags. A different outcome isolates the env file as the cause. 3. Run gdb for the backtrace with this command: ``` - gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-large.sh + gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-small.sh ``` `llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost. Trust gdb, not the log. diff --git a/dankclaw/frank-large/quadlet/frank-large.container b/dankclaw/frank-large/quadlet/frank-large.container new file mode 100644 index 0000000..c9d9879 --- /dev/null +++ b/dankclaw/frank-large/quadlet/frank-large.container @@ -0,0 +1,121 @@ +# Quadlet unit: frank-large (rootless, boot-persistent). +# +# Install path on dankclaw: ~/.config/containers/systemd/frank-large.container +# Unit name: frank-large.service (container name: frank-large) +# +# The large inference service, since 2026-09-11 the halogen-flash-server +# runtime serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next, +# w4b .hgn checkpoint + quality overlay sidecar). It replaced the llama.cpp / +# kingjones30-ROCmFPX STRIX deployment because the engine is ~4x faster on +# prefill (the dominant cost of any real prompt) on this exact SKU. +# +# SHAPE. The image runs both halves in one container: the engine on container +# loopback (:8730, UNAUTHENTICATED — never publish it) and the OpenAI +# front-end on :8731, published here as host 7770 so the fleet's clients, +# services.json probe and docs keep the port they have always used. Same for +# HALOGEN_MODEL_ID=frank-large: it is the id at /v1/models, in /health and in +# every response, so no client had to be renamed. Halogen never rejects a +# request for naming a different model, it just serves it. +# +# DEVICES. gfx1151 through the amdgpu/KFD stack: /dev/kfd plus the DRM render +# node. Both are 0666 on this host, so the rootless user namespace can open +# them (id dank is also in render/video). seccomp=unconfined and --ipc=host are +# the image's documented requirements; memlock must be unlimited. +# +# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP +# runtime reads bare TEMP for its JIT cache path (2026-08-13 incident). +# +# MEMORY. Weights are pinned (HALOGEN_FLASH_PIN_TRUNK default 1) — that is the +# configuration the published decode numbers were measured with, and `free`, +# MemAvailable and every monitor built on them will overstate free memory by +# the size of the model (~68 GiB) because the kernel counts the locked weights +# as reclaimable page cache. Halogen prints the corrected figure at startup. +# The pinned trunk means this container holds most of the host; a co-tenant +# that then takes the remainder can stall both for minutes at full CPU with no +# disk activity and no output. That is documented behaviour, not a crash, and +# the escape hatch is HALOGEN_FLASH_PIN_TRUNK=0 (at several times the decode +# cost). Sizing chosen 2026-09-11 for a SHARED 121 GiB host: KV pool 262,144 +# positions (~7.2 GiB) + the baked 32,768-token prefill arena (16.7 GiB) sits +# well inside the device budget once the ~68 GiB of weights and the ~5 GiB of +# co-tenant containers are accounted for. HALOGEN_KV_POOL_FIT stays on as the +# safety net: if the machine refuses, the engine lowers the pool itself and +# prints what it chose. +# +# WEIGHTS. HALOGEN_DOWNLOAD fetches the whole weights repo on FIRST START ONLY +# (the entrypoint checks whether the checkpoint is absent) into the bind mount +# below, which must therefore be read-WRITE. huggingface_hub resumes partial +# files natively, and the quadlet runs the container with --replace --rm, so a +# restart mid-fetch continues rather than starting over. Roughly 127 GB: the +# 115.55 GiB checkpoint, the 2.31 GiB quality overlay, the 0.84 GiB vision +# tower and the tokenizer. The repo is public and ungated, so no token is +# needed; later starts open no outbound connections at all. +# +# VISION. HALOGEN_VISION_TOWER=1 finds the sidecar beside the checkpoint. This +# is the fleet's only image input since the FLM NPU endpoint was retired on +# 2026-09-11; audio input has no replacement. +# +# CLIENT CONTRACT (see serverAppConfigs-docs/LLM/halogen-flash-server.md). +# There are no server-side sampler defaults: a request that sends no +# temperature decodes GREEDY. The per-request budget defaults to 8,192 tokens +# and REASONING COUNTS AGAINST IT, so a budget that runs out mid-thought +# returns finish_reason "length" with empty content and the partial reasoning +# in reasoning_content. The cap is 65,536; above it a request is a 400, never a +# silent truncation. +# +# IMAGE. The unit runs the localhost/-aliased tag: this host's podman quadlet +# path cannot resolve registry-prefixed refs from local storage at create +# time. deploy-remote.sh pulls the ghcr.io ref and re-tags it before the +# restart. Pull=never keeps boots off the registry. +[Unit] +Description=frank-large — halogen-flash-server, Qwen3.8-Flash-Next (w4b) on gfx1151 +After=network-online.target + +[Container] +ContainerName=frank-large +Image=localhost/peonist-ai/halogen-flash-server:0.5.8 +# Published port stays 7770: Hermes, open-webui, the harness and the homepage +# probe all point here. 8730 (the engine) is deliberately not published. +PublishPort=7770:8731 +AddDevice=/dev/kfd +AddDevice=/dev/dri +# Read-write: the first start downloads into it. Contains only weights. +Volume=/home/dank/models/halogen:/models +# Fetch the weights on first start only (absent checkpoint gates it). +Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next +# Fleet identity: the id at /v1/models, /health and every response body. +Environment=HALOGEN_MODEL_ID=frank-large +# Native context of the model. Matches what both clients declare. +Environment=HALOGEN_CTX=262144 +# Conservative pool for a shared host: one full-length conversation, or four +# at 65k. Raising it to 524288 doubles the resident conversations for ~7 GiB +# more device memory — do that only after reading the startup ledger. +Environment=HALOGEN_KV_POOL_POSITIONS=262144 +Environment=HALOGEN_KV_SLOTS=4 +# RAM left for the n-gram table the engine reads through the page cache. +# Above the vendor's 20 because a dozen containers share this host; too low +# and long prompts crawl with the disk busy. +Environment=HALOGEN_HOST_RESERVE_GIB=24 +# Image input (the sidecar ships inside the weights repo). +Environment=HALOGEN_VISION_TOWER=1 +# Required by the image: the engine maps the checkpoint and registers it with +# the GPU in place, which needs host IPC and an unconfined seccomp profile. +PodmanArgs=--ipc=host --security-opt seccomp=unconfined +Ulimit=memlock=-1:-1 +# The image ships its own HEALTHCHECK; this pins it. /health asks the ENGINE +# for a PONG before answering, so 200 means "ready to serve", not "process +# alive". During the first-start download it stays unhealthy by design. +HealthCmd=halogen-healthcheck api +HealthInterval=30s +HealthTimeout=10s +HealthRetries=3 +HealthStartPeriod=300s +Pull=never + +[Service] +# The first start reads ~68 GiB off disk and pins it before serving. +TimeoutStartSec=900 +Restart=on-failure +RestartSec=10 + +[Install] +WantedBy=default.target diff --git a/dankclaw/hermes/config.yaml.example b/dankclaw/hermes/config.yaml.example index 084cb97..098e5d6 100644 --- a/dankclaw/hermes/config.yaml.example +++ b/dankclaw/hermes/config.yaml.example @@ -25,23 +25,17 @@ providers: models: frank-small: context_length: 163840 - dankclaw-npu: - api: http://192.168.0.143:7771/v1 - discover_models: false - models: - gemma4-it:e4b: - context_length: 131072 # Auxiliary routing from the live configuration. Named providers route via # their `api:` URL; `custom` entries route by their explicit `base_url`. auxiliary: - vision: {provider: dankclaw-npu, model: gemma4-it:e4b} + vision: {provider: dankclaw-large, model: frank-large} web_extract: {provider: dankserver-small, model: frank-small} compression: {provider: dankserver-small, model: frank-small} skills_hub: {provider: dankserver-small, model: frank-small} approval: {provider: dankserver-small, model: frank-small} mcp: {provider: dankserver-small, model: frank-small} - title_generation: {provider: dankclaw-npu, model: gemma4-it:e4b} + title_generation: {provider: dankserver-small, model: frank-small} memory_query_rewrite: {provider: custom, model: frank-small, base_url: http://192.168.0.235:7770/v1} tts_audio_tags: {provider: custom, model: frank-small, base_url: http://192.168.0.235:7770/v1} triage_specifier: {provider: dankclaw-large, model: frank-large} diff --git a/scripts/deploy-plan.py b/scripts/deploy-plan.py index 4dc9703..eb50ac8 100755 --- a/scripts/deploy-plan.py +++ b/scripts/deploy-plan.py @@ -59,7 +59,6 @@ SERVICE_ORDER = ( "lidarr", "readarr", "frank-large", - "frank-npu", "frank-small", ) @@ -89,7 +88,6 @@ DANKCLAW_SERVICES = { "caddy", "homepage", "frank-large", - "frank-npu", } DANKSERVER_SERVICES = { @@ -297,6 +295,11 @@ def services_for_path(path: str, target: str) -> set[str]: if len(parts) >= 2 and parts[1] == "open-webui": _add(services, "open-webui") return services + if len(parts) >= 2 and parts[1] == "frank-large": + # The service is a rootless Quadlet container since 2026-09-11; + # its unit and fleet settings live in one .container file. + _add(services, "frank-large") + return services if len(parts) >= 2 and parts[1] == "gitea": if len(parts) >= 4 and parts[2] == "quadlet": name = parts[3] @@ -346,15 +349,12 @@ def services_for_path(path: str, target: str) -> set[str]: if parts[0] == "scripts": name = p.name - if name in {"frank-large.env", "frank-large.service", "serve-frank-large.sh"}: - _add(services, "frank-large") - elif name in {"frank-npu.env", "frank-npu.service", "serve-frank-npu.sh"}: - _add(services, "frank-npu") - elif name in {"frank-small.env", "frank-small.service", "serve-frank-small.sh"}: + if name in {"frank-small.env", "frank-small.service", "serve-frank-small.sh"}: _add(services, "frank-small") elif name == "chat-template.jinja": - # Only the selected target's LLM consumes this shared template. - _add(services, "frank-large" if target == "dankclaw" else "frank-small") + # Only frank-small consumes this template now: frank-large runs + # halogen-flash-server, which owns its own chat handling. + _add(services, "frank-small") elif name in {"homepage-status.py", "homepage-status.service", "homepage-status.timer"}: _add(services, "homepage") # Deployment/test/benchmark scripts and other repo tooling do not diff --git a/scripts/deploy-remote.sh b/scripts/deploy-remote.sh index f730f9a..768493d 100755 --- a/scripts/deploy-remote.sh +++ b/scripts/deploy-remote.sh @@ -90,7 +90,7 @@ service_is_allowed() { dankclaw:authentik-postgres|dankclaw:authentik-server|\ dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\ dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|\ - dankclaw:frank-large|dankclaw:frank-npu|\ + dankclaw:frank-large|\ dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\ dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:plex|dankserver:tautulli|\ dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\ @@ -183,7 +183,6 @@ add_pending_services() { [[ -n "$recorded" && "$recorded" != "$REVISION" ]] || continue if [[ "$TARGET" == dankclaw && "$service" == frank-small ]]; then continue; fi if [[ "$TARGET" == dankserver && "$service" == frank-large ]]; then continue; fi - if [[ "$TARGET" == dankserver && "$service" == frank-npu ]]; then continue; fi if ! changed="$(python3 scripts/deploy-plan.py --target "$TARGET" \ --repo "$REMOTE_DIR" --base "$recorded" --head "$REVISION")"; then echo "WARN: unable to compare state for $service; retaining it as pending" >&2 @@ -826,21 +825,85 @@ deploy_anubis() { wait_container_healthy anubis 60 || return 1 } -# Rename cutover (2026-09-03): frank-default-large -> frank-large, -# frank-default-small -> frank-small, frank-default-sightsound -> frank-npu. -# The legacy units are stopped and disabled before the renamed unit takes the -# port so no legacy process is left holding it; stop/disable of an already -# unloaded legacy unit fail harmlessly and are tolerated. -deploy_llm_large() { - run_privileged systemctl stop frank-default-large || true - run_privileged systemctl disable frank-default-large || true - relink_unit frank-large +# frank-large runs as a rootless Quadlet unit since 2026-09-11 (halogen-flash- +# server replaced the llama.cpp/kingjones runtime). Unlike the sibling LLM +# services it is therefore NOT a root unit: no relink_unit, no `systemctl +# enable` (this host's quadlet generator produces units that cannot be +# enabled — boot persistence comes from [Install] WantedBy=default.target in +# the .container file plus linger). +# +# The llama.cpp-era ROOT unit is retired first. Its unit file is a symlink +# into this checkout (the old relink_unit did `ln -sfn scripts/frank-large. +# service /etc/systemd/system/`), so leaving it behind after the repo file +# moves to the trash would leave systemd holding a dangling unit that can +# still claim port 7770. Stop it, drop its enable symlinks, move the unit file +# into the home trash (never rm), then reload the root manager. +# +# FIRST START: the container fetches ~127 GB of weights before it can serve, +# so /health is legitimately down for the whole fetch. wait_for_health would +# burn its budget and report a failure that is not one. The gate below +# recognises the bootstrap (checkpoint absent) and waits for the checkpoint +# instead, falling through to the normal health gate once it lands. +LLM_LARGE_CHECKPOINT="${LLM_LARGE_CHECKPOINT:-/home/dank/models/halogen/qwen38-flash-next-w4b.hgn}" + +retire_root_llm_large() { + local trash="$HOME/.AGENTTRASH/20260911-halogen-swap/host-units" + run_privileged systemctl stop frank-default-large 2>/dev/null || true + run_privileged systemctl disable frank-default-large 2>/dev/null || true + run_privileged systemctl stop frank-large.service 2>/dev/null || true + # On this host `disable` has been observed to delete the unit file itself + # (2026-08-26 incident, see AGENTS.md); tolerate either behaviour and keep + # whatever remains by moving it to the trash. + run_privileged systemctl disable frank-large.service 2>/dev/null || true + if [[ -e /etc/systemd/system/frank-large.service || -L /etc/systemd/system/frank-large.service ]]; then + mkdir -p "$trash" + run_privileged mv -- /etc/systemd/system/frank-large.service \ + "$trash/frank-large.service" + echo "retired root unit frank-large.service -> $trash" + fi run_privileged systemctl daemon-reload - run_privileged systemctl enable frank-large - run_privileged systemctl restart frank-large +} + +deploy_llm_large() { + retire_root_llm_large + # Pull once, then run the localhost/-aliased tag: this host's quadlet path + # cannot resolve registry-prefixed refs from local storage at create time + # (same workaround as anubis/hermes). + local image="ghcr.io/peonist-ai/halogen-flash-server:0.5.8" + local alias="localhost/peonist-ai/halogen-flash-server:0.5.8" + podman pull "$image" + podman tag "$image" "$alias" + # The weights volume: read-write, because the container downloads into it. + mkdir -p /home/dank/models/halogen + install_quadlet frank-large dankclaw/frank-large/quadlet frank-large + systemctl --user restart frank-large.service + wait_for_llm_large_ready wait_for_health frank-large http://127.0.0.1:7770/health 1200 } +# wait_for_llm_large_ready — bootstrap-aware gate. If the checkpoint is absent +# the container is downloading it; wait for the file (resumable, and the unit +# restarts it on failure) rather than gating on a /health that cannot answer +# yet. Reports success either way, so the first deploy does not look like a +# failure while the engine does exactly what it was told to. +wait_for_llm_large_ready() { + [[ -f "$LLM_LARGE_CHECKPOINT" ]] && return 0 + echo "frank-large: checkpoint absent at $LLM_LARGE_CHECKPOINT" + echo "frank-large: first start — the container is downloading ~127 GB from" + echo "frank-large: peonist-ai/halogen-qwen3.8-flash-next (resumes if interrupted)." + local i + for ((i = 1; i <= 1200; i++)); do + if [[ -f "$LLM_LARGE_CHECKPOINT" ]]; then + echo "frank-large: weights landed after approximately $((i * 2))s; gating on /health" + return 0 + fi + sleep 2 + done + echo "WARN: frank-large download still in progress after 40m — NOT a failure." + echo "WARN: the unit keeps fetching; re-run this deploy (or re-check /health) later." >&2 + return 0 +} + deploy_llm_small() { run_privileged systemctl stop frank-default-small || true run_privileged systemctl disable frank-default-small || true @@ -851,22 +914,6 @@ deploy_llm_small() { wait_for_health frank-small http://127.0.0.1:7770/health 1200 } -# frank-npu is a live serving endpoint (FLM on the XDNA2 NPU), not a -# relink-only unit: it is enabled at boot and restarted on deploy. The health -# gate polls /v1/version, FLM's cheapest always-on route (there is no /health -# route in the FLM server). -deploy_llm_npu() { - run_privileged systemctl stop frank-default-sightsound || true - run_privileged systemctl disable frank-default-sightsound || true - run_privileged systemctl stop frank-default-npu || true - run_privileged systemctl disable frank-default-npu || true - relink_unit frank-npu - run_privileged systemctl daemon-reload - run_privileged systemctl enable frank-npu - run_privileged systemctl restart frank-npu - wait_for_health frank-npu http://127.0.0.1:7771/v1/version 300 -} - wait_for_health() { local service="$1" url="$2" limit="$3" local i @@ -1052,7 +1099,6 @@ deploy_service() { seerr) deploy_seerr ;; frank-large) deploy_llm_large ;; frank-small) deploy_llm_small ;; - frank-npu) deploy_llm_npu ;; hermes|hermes-dashboard|hermes-webui) deploy_hermes ;; romm|romm-mariadb|romm-redis|mariadb|redis) deploy_romm_stack "$service" ;; jellyfin) deploy_jellyfin ;; diff --git a/scripts/test-changed.sh b/scripts/test-changed.sh index c03e83d..ece9421 100755 --- a/scripts/test-changed.sh +++ b/scripts/test-changed.sh @@ -134,6 +134,14 @@ for path in "${CHANGED[@]}"; do add_test tests/homepage_caddy.bats add_test tests/deployment_scoping.bats ;; + # frank-large moved to a rootless quadlet (halogen-flash-server) on + # 2026-09-11; its unit carries the fleet's GPU and podman posture, so + # it gets its own seam rather than the generic fallback below. + dankclaw/frank-large/*) + add_test tests/frank_large_quadlet.bats + add_test tests/deployment_scoping.bats + add_test tests/docs_completeness.bats + ;; # Quadlets live under dankclaw//quadlet/ (reorganized out of the # top-level dankclaw/*.container globs above). Map the whole subtree so # changes to the pihole, open-webui, searxng, and gitea quadlets hit @@ -160,11 +168,6 @@ for path in "${CHANGED[@]}"; do scripts/mount-entropy.sh) add_test tests/mount_entropy.bats ;; - scripts/frank-large.env|scripts/serve-frank-large.sh) - add_test tests/llm_units.bats - add_test tests/llm_secrets.bats - add_test tests/scripts_syntax.bats - ;; scripts/frank-small.env|scripts/serve-frank-small.sh) add_test tests/llm_units.bats add_test tests/llm_secrets.bats diff --git a/serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md b/serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md new file mode 100644 index 0000000..4955502 --- /dev/null +++ b/serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md @@ -0,0 +1,100 @@ +--- +type: research +title: Dankclaw IOMMU translation off (amd_iommu=off) — decision record +description: Why Dankclaw boots with amd_iommu=off since 2026-09-11, what it buys (13-16% prefill for frank-large), what it costs (the XDNA2 NPU endpoint, machine-wide DMA translation), what was deliberately not copied from the reference machine, and how to revert. +tags: + - dankclaw + - iommu + - kernel + - grub + - prefill + - frank-large + - halogen + - frank-npu +timestamp: 2026-09-11 +--- + +# Dankclaw IOMMU translation off — decision record + +**Decision (2026-09-11):** Dankclaw boots with **`amd_iommu=off`** instead of +`iommu=pt`, to recover the prefill throughput the large-model runtime loses to +IOMMU translation. The XDNA2 NPU endpoint was accepted as the cost and is +retired ([[frank-npu]]). + +## What changed + +`GRUB_CMDLINE_LINUX_DEFAULT` in `/etc/default/grub` on Dankclaw: + +``` +# before +ro quiet splash iommu=pt crashkernel=… + +# after +ro quiet splash amd_iommu=off crashkernel=… +``` + +Then `sudo update-grub` and a reboot. Nothing else on the command line was +touched. + +## Evidence + +The upstream project published the measurement, and it is the only A/B they +ran: **460 → 385 tok/s prefill at a 2,048-token prompt, 13 to 16 percent**, +taken on a Ryzen AI Max+ 395 with 128 GB — the same SKU as Dankclaw. Their +published serving profile boots this way, so every number in their README is a +number with translation off. Decode is unaffected; prefill is where the loss is. + +The mechanism offered is DMA translation on the path every GPU allocation +takes. The project states plainly that this is *their* configuration and not a +tuning guide, that the flag turns off DMA translation machine-wide, and that it +takes the NPU with it. + +## What it costs + +1. **The NPU endpoint dies.** With IOMMU off there is no PASID/SVA state and + `/dev/accel/accel0` cannot be opened — `err=-19`, the exact failure + documented in [[frank-npu]]. Audio input left the fleet with it; image input + survives on `frank-large` through halogen's vision tower. +2. **Posture change machine-wide.** Dankclaw also serves Caddy (the TLS edge + and the Anubis/Authentik guard chain), Authentik, Pi-hole, Gitea, and the + observability stack. DMA translation is now off for all of them. This was + accepted explicitly as the price of the prefill figure. + +## What was deliberately NOT copied + +The reference machine's command line has six settings. Only one was measured, +so only that one was taken: + +| setting | reference value | why not here | +|---|---|---| +| `amd_iommu=off` | set | **taken** — the measured 13–16 % | +| `amdgpu.gttsize` | `126976` MiB | a size, not a constant: it must equal installed RAM (their table: 128 GB → 126976, 96 GB → 95232, 64 GB → 63488). Dankclaw's GTT already reports **114 GiB ≈ 94 % of its 121 GiB** without the flag, so it would buy nothing | +| `ttm.pages_limit` | `32505856` | the same number in 4 KiB pages (their 128 GB row is 99.3 % of RAM). Pasting it onto a different machine asks the driver for more GTT than exists | +| `amdgpu.vm_update_mode=0` | set | upstream has never run without it, makes no claim about what it buys, and lists it "for completeness rather than recommended" | +| `amdgpu.noretry=0` | set | as above | +| `amdgpu.sg_display=0` | set | as above | + +## Verification and revert + +```bash +cat /proc/cmdline # expect amd_iommu=off, no iommu=pt +ls /sys/kernel/iommu_groups/ | wc -l # expect 0 groups (was 30) +systemctl is-active frank-npu # expect inactive (masked) +curl -s http://127.0.0.1:7770/health # frank-large must still answer 200 +``` + +**To revert:** put `iommu=pt` back in `GRUB_CMDLINE_LINUX_DEFAULT` in place of +`amd_iommu=off`, run `sudo update-grub`, reboot, and unmask/start `frank-npu` +with the archived unit (`.AGENTTRASH/frank-npu.service.flm-npu`) if the NPU +endpoint is wanted back. The IOMMU change is independent of the halogen +deployment — reverting one does not require reverting the other. + +## Related + +- [[frank-npu]] — the endpoint this retired, including the err=-19 root cause +- [[halogen-flash-server]] — the runtime bought by the prefill figure +- [[kingjones-strix-rocmfp4]] — the runtime it replaced (rollback target) + +## Citations + +- [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README "The host settings these numbers were measured on", CHANGELOG 0.5.5–0.5.8 diff --git a/serverAppConfigs-docs/LLM/frank-npu.md b/serverAppConfigs-docs/LLM/frank-npu.md index da7792f..b7a5761 100644 --- a/serverAppConfigs-docs/LLM/frank-npu.md +++ b/serverAppConfigs-docs/LLM/frank-npu.md @@ -1,7 +1,7 @@ --- -type: configuration -title: frank-npu Runbook -description: FastFlowLM (FLM) NPU sight+sound endpoint on Dankclaw — XDNA2 NPU serving Qwen3.8 Distilled Heretic vision+text pack on port 7771, including the IOMMU/SVA root cause that took it down at first deploy. +type: reference +title: frank-npu (retired) — FLM NPU endpoint on Dankclaw +description: The retired FastFlowLM endpoint on Dankclaw's XDNA2 NPU — what it served, why it died on 2026-09-11 with the amd_iommu=off decision, and where every dependent route went. tags: - flm - fastflowlm @@ -9,133 +9,125 @@ tags: - npu - dankclaw - frank-npu -timestamp: 2026-09-01 + - retired +timestamp: 2026-09-11 --- -# frank-npu — FLM NPU sight+sound Runbook +# frank-npu — the retired FLM NPU endpoint -`frank-npu` is the third inference endpoint on Dankclaw: a -FastFlowLM (FLM) v1.0.1 server (`/usr/bin/flm`, dpkg package `fastflowlm`) -running on the **XDNA2 NPU** (50 TOPS). It serves *sight* (vision) — -which the iGPU llama.cpp fleet ([[kingjones-strix-rocmfp4]], -`frank-large`) does not do. The model-selection tradeoff: the only -NPU2 pack family on this box with BOTH a vision and an audio tower is the -Gemma4 E4B IT pack; the uncensored candidate (`qwythos:9b`, 8.94 GB, 262k ctx) -is text-only and fails the hard modality requirement, so this endpoint trades -"uncensored" for "sight + sound". +> **RETIRED 2026-09-11.** The endpoint is gone by operator decision, together +> with the IOMMU configuration it required — see +> [[dankclaw-iommu-translation-off]]. The service unit, env file and serve +> script are staged in `serverAppConfigs/.AGENTTRASH/` (suffixed `.flm-npu`), +> and the host unit is stopped and masked. -## Deployment shape +The third inference endpoint on Dankclaw: a FastFlowLM (FLM) v1.0.1 server +(`/usr/bin/flm`, dpkg `fastflowlm`) on the **XDNA2 NPU** (50 TOPS), serving the +whole local model registry on port 7771 and routing per request by the exact +FLM registry tag. Its one capability the iGPU fleet lacked was **sight** — and +briefly **sound**: the Gemma4 E4B IT NPU2 pack carries both a vision and an +audio tower, which is why the service ran that pack rather than the +uncensored-but-text-only alternative. + +This page is kept as the retirement record. The operational detail below was +live-verified while the service ran and remains accurate about FLM's shape. + +## Why it is retired + +The replacement large-model runtime is worth **13–16 % of prefill** if the +kernel boots with `amd_iommu=off` instead of `iommu=pt`, measured by the +upstream project on this exact silicon. That single flag is the whole decision: + +- With IOMMU off there is no PASID/SVA state, and the XDNA2 driver cannot hand + out `/dev/accel/accel0` — every open fails with `err=-19` ("No such device"). + That is precisely the failure the endpoint was rescued from on 2026-08-30, + recorded in the next section. +- `iommu=pt` is not a middle ground for the NPU *and* the prefill number: the + passthrough domain still costs the prefill throughput the vendor measured. +- The operator chose prefill, so the NPU endpoint goes. + +Consequences accepted at retirement: **audio input left the fleet with no +replacement** (halogen reads images only, no audio or video), and the surviving +image path is halogen's vision tower on `frank-large`. + +## Where its routes went + +| consumer | setting | now | +|---|---|---| +| Hermes | `auxiliary.vision` | `dankclaw-large` / `frank-large` (halogen vision tower) | +| Hermes | `auxiliary.title_generation` | `dankserver-small` / `frank-small` | +| harness (`~/.omp/agent/models.yml`) | `modelRoles.vision` | `llamacpp (dankclaw)` / `frank-large` | +| harness | `modelRoles.tiny` | `llamacpp (dankserver)` / `frank-small` | +| harness | `modelRoles.task` | `llamacpp (dankclaw)` / `frank-large:medium` | + +The `flm (dankclaw-npu)` provider block was removed from the harness model +config, and the `dankclaw-npu` provider from Hermes's live `config.yaml`. + +## What it was | Field | Value | -|-------|-------| +|---|---| | Runtime | FLM v1.0.1 (`/usr/bin/flm`, dpkg `fastflowlm`) on the XDNA2 NPU | -| Service | `frank-npu.service` (systemd system unit, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`) | -| Files | `scripts/frank-npu.env`, `scripts/serve-frank-npu.sh`, `scripts/frank-npu.service` | -| Deploy | `scripts/deploy.sh dankclaw --service frank-npu` (relink + daemon-reload + enable + restart; health gate polls `/v1/version`, 300 s cap — FLM has **no `/health` route**) | -| Port | `7771` on `0.0.0.0` (52625/52626 left free for ad-hoc `flm serve` experiments) | -| Power mode | `FLM_PMODE=turbo` — no other tenant uses the NPU on this host (the large model runs on the iGPU), so the top power mode costs no neighbour anything; drop to `performance` if iGPU power-budget contention ever shows | -| Context | `FLM_CTX_LEN=131072 (128K; verified max on this box; native context length.) | -| ASR | `FLM_ASR=0` — the E4B pack answers images natively through its vision tower in `/v1/chat/completions`; setting `FLM_ASR=1` additionally loads Whisper-V3-Turbo-NPU2 (+0.62 GB) and exposes `/v1/audio/transcriptions` | +| Service | `frank-npu.service` (systemd **system** unit, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`) | +| Files (archived) | `scripts/frank-npu.env`, `scripts/serve-frank-npu.sh`, `scripts/frank-npu.service` | +| Port | `7771` on `0.0.0.0` | +| Power mode | `FLM_PMODE=turbo` — nothing else used the NPU, so the top power mode cost no neighbour anything | +| Context | `FLM_CTX_LEN=131072` (128K, the verified maximum on this box) | +| Vision/audio | `FLM_ASR=0`; the E4B pack answers images through its own vision tower, and `FLM_ASR=1` would have added Whisper-V3-Turbo-NPU2 (+0.62 GB) and `/v1/audio/transcriptions` | -The serve script exports `FLM_CONFIG_PATH=/home/dank/.config/flm/model_list.json` -and `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins`; kernels are symlinked -from `/opt/fastflowlm/share/flm/xclbins/` into the per-pack directories. A tag -swap is therefore an env-file edit, not a script edit. +The serve script exported `FLM_CONFIG_PATH=/home/dank/.config/flm/model_list.json` +and `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins`, with kernels symlinked +from `/opt/fastflowlm/share/flm/xclbins/` into the per-pack directories, so a +tag swap was an env-file edit rather than a script edit. -## Model tags for switching +## API contract (as it behaved) -FLM routes per-request by the exact registry tag in the `model` field. To swap -the service model, change `FLM_MODEL` in the env file and restart. To let -clients choose, send requests with the desired `model` field. +- The endpoint served the **whole local registry** (36 tags at first deploy) on + one port; `/v1/models` listed them all. Chat completions routed per request + by the exact registry tag in the `model` field — never a service alias. +- **Unknown tags did not 404**: `/v1/chat/completions` answered HTTP 200 with a + flat error body `{"error":"Failed to load model!"}` and no `choices`. + Status-code-only clients silently got an empty response. +- Health was `GET /v1/version`; FLM has **no `/health` route**. +- The NPU is a single-lane device — generations serialized FIFO. It was a + background/triage endpoint, not interactive multi-user chat. -| Tag | Model | Modality | Size | Family | -|-----|-------|----------|------|--------| -| `qwen3.8-distilled-heretic:9b` | Qwen3.8 Distilled Heretic | vision + reasoning | 7.6 GB | qwen3.5 | -| `gemma4-it:e4b` | Gemma4 E4B IT | vision + audio | 9.1 GB | gemma4e | -| `qwythos:9b` | Qwythos 9B Claude Mythos | vision + reasoning | 8.9 GB | qwen3.5 | -| `gpt-oss:20b` | GPT-OSS 20B | text only | 13.8 GB | gpt-oss | +## The IOMMU/SVA root cause (first deploy crashed, fixed 2026-08-30) -**To swap the active model** (changes what the service loads at start): -``` -sed -i 's/^FLM_MODEL=.*/FLM_MODEL=/' scripts/frank-npu.env -./scripts/deploy.sh dankclaw --service frank-npu -``` +First deploy crash-looped every 10 s with +`Error: Open /dev/accel/accel0 failed (err=-19): No such device`. The chain, +all live-verified on the host: -**To switch per-request** (service stays loaded, client sends the tag): -``` -curl http://192.168.0.143:7771/v1/chat/completions \ - -H "Content-Type: application/json" \ - -d '{"model":"gemma4-it:e4b","messages":[...],"max_tokens":64}' -``` - -Known tags are in the user registry at -`/home/dank/.config/flm/model_list.json`; unknown tags answer HTTP 200 with -`{"error":"Failed to load model!"}` (no choices). - -## Serving model and API contract - -- The endpoint serves the **whole local registry** (36 tags at first deploy) on - one port; `/v1/models` lists them all. Chat completions route per request by - the exact FLM registry tag in the `model` field — never a service alias. -- **Unknown model tags do NOT 404** (live-probed): `/v1/chat/completions` - answers HTTP 200 with a flat error body - `{"error":"Failed to load model!"}` and no `choices`; - `/v1/completions` answers an OpenAI-shaped nested error - (`type: server_error`, `code: 500`), also under HTTP 200. A valid tag - routes that request to that model (response echoes the requested tag); no - server restart is needed to switch between registry tags. Status-code-only - clients silently get an empty response — check for the `error` key. -- Multimodal content parts accepted by `/v1/chat/completions`: - `{"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}` and - `{"type":"input_audio","input_audio":{"data":"","format":"wav"}}`. -- Chat template roles: only `system`/`user`/`assistant`; any other role fails - the request with `Unexpected message role.`. -- The NPU is a single-lane device: generations serialize in FIFO order. This is - a background/triage endpoint, not an interactive multi-user chat. - -## Live-verified behavior (2026-08-30, first deploy) - -- Vision tower: a generated red 8×8 PNG posted as an `image_url` data-URI was - answered correctly ("The image is a solid red color."); the request consumed - 279 prompt tokens — the vision tower consumed the pixels (a text-only model - given zero image bytes cannot answer "what color"). -- Audio tower: a 440 Hz decaying sine WAV posted as `input_audio` consumed - 37 prompt tokens vs 15 for the identical text-only control → ~22 audio-tower - tokens; the model answered truthfully that it heard no speech (it is a tone, - not speech — ASR stays off). - -## The IOMMU/SVA root cause (first deploy crashed, fixed same day) - -First deploy crash-looped every 10 s (`RestartSec=10`) with -`Error: Open /dev/accel/accel0 failed (err=-19): No such device`. Root-cause -chain, all live-verified on the host: - -1. `amdxdna_drm_open` was failing **SVA bind with ret -19** for every client — - the XDNA2 driver cannot hand out the accel device without PASID/SVA. -2. SVA requires the AMD IOMMU to be enabled; the kernel cmdline carried - `iommu=off` (a perf/DMA tuning experiment from 2026-08-20, applied at the - 2026-08-26 reboot). With IOMMU off there are no IOMMU groups and no PASID - state; `PASIDCtl: Enable-` was observed disabled on `0000:c5:00.1`. -3. Fix: replace `iommu=off` with **`iommu=pt`** in `GRUB_CMDLINE_LINUX_DEFAULT` - on `/etc/default/grub`, `update-grub`, reboot. Passthrough mode keeps the - iGPU default domain identity-mapped (zero DMA-translation overhead for - `frank-large`) while enabling IOMMU hardware + PASID, which is what - the NPU driver's SVA bind requires — PASID domains are paged regardless of - the default-domain type. +1. `amdxdna_drm_open` failed **SVA bind with ret -19** for every client — the + XDNA2 driver cannot hand out the accel device without PASID/SVA. +2. SVA requires the AMD IOMMU to be enabled, and the kernel cmdline then + carried `iommu=off`. With IOMMU off there are no IOMMU groups and no PASID + state; `PASIDCtl: Enable-` was observed on `0000:c5:00.1`. +3. The fix was `iommu=pt`: passthrough keeps the iGPU default domain + identity-mapped (no DMA-translation overhead for `frank-large`) while + enabling the IOMMU hardware and PASID that the driver's SVA bind needs. 4. Post-boot verification: `/sys/kernel/iommu_groups/` populated (30 groups), - zero `SVA bind device failed` journal entries since boot, accel0 recreated at - boot, service active with `NRestarts=0`, and the image/audio probes above - served live. + zero `SVA bind device failed` entries since boot, `accel0` recreated at + boot, service active with `NRestarts=0`. -The same root cause explains the older `amdgpu 0000:c4:00.0: no vmid pasid -mapping supported` kernel log line — a consequence of IOMMU-off, not a GPU -fault. Do not set `iommu=off` on this host again; it silently kills the NPU. +The same root cause explained the older +`amdgpu 0000:c4:00.0: no vmid pasid mapping supported` kernel line — a +consequence of IOMMU-off, not a GPU fault. -## Client wiring +**The standing rule this produced — "never set `iommu=off` on this host again" — +has been superseded**: the NPU it protected is retired by decision, and +`amd_iommu=off` is now intentional. See +[[dankclaw-iommu-translation-off]] for the current posture and how to revert it. -The pi client model config (`/home/dank/.pi/agent/models.json` on Dankware) -registers this endpoint under the Dankclaw provider with the **exact** FLM -registry tag as the model `id` (unknown tags answer a 200-wrapped error, not -a silent 1B fallback), `baseUrl` `http://192.168.0.143:7771/v1`, and a per-request context -window equal to the serving ctx-len. Health checks for this service must use -`GET /v1/version` (there is no `/health` route). +## Client wiring (historical) + +The pi/harness model config registered this endpoint with the **exact** FLM +registry tag as the model `id` and `baseUrl` `http://192.168.0.143:7771/v1`. +Both that provider block and Hermes's `dankclaw-npu` provider were removed at +retirement; the routes are mapped in the table above. + +## Citations + +- [FastFlowLM](https://github.com/FastFlowLM/FastFlowLM) — FLM runtime and NPU model packs +- [[dankclaw-iommu-translation-off]] — the decision that retired this endpoint +- [[halogen-flash-server]] — the surviving image-input path diff --git a/serverAppConfigs-docs/LLM/halogen-flash-server.md b/serverAppConfigs-docs/LLM/halogen-flash-server.md new file mode 100644 index 0000000..68f40b3 --- /dev/null +++ b/serverAppConfigs-docs/LLM/halogen-flash-server.md @@ -0,0 +1,240 @@ +--- +type: reference +title: halogen-flash-server on Dankclaw (frank-large) +description: The active frank-large runbook — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing. +tags: + - llm + - halogen + - dankclaw + - frank-large + - strix-halo + - gfx1151 + - qwen4exp + - rocm +timestamp: 2026-09-11 +--- + +# halogen-flash-server on Dankclaw (frank-large) + +> **ACTIVE since 2026-09-11.** Replaced the llama.cpp / kingjones30-ROCmFPX +> STRIX deployment (see [[kingjones-strix-rocmfp4]], now the rollback target). +> Deployed as a rootless Quadlet container, +> `dankclaw/frank-large/quadlet/frank-large.container`, port 7770. + +The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next +architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs. +Upstream's own comparison table measures it at **roughly 4x the prefill +throughput** of the llama.cpp-family runtimes it ships against (EngramHalo.cpp, +ROCmFP4, CIRU-IU4) on a 32K prompt, and prefill is most of the wall clock on any +prompt with real context. That, not decode, is the reason for the swap. + +## Shape + +One container runs both halves of the stack: + +| half | port | reachable | +|---|---|---| +| engine | 8730 | container loopback only — **unauthenticated, never publish it** | +| OpenAI front-end | 8731 | published as host **7770** | + +The `all` mode is upstream's default and the shape a single-container host +wants: the front-end talks to the engine over loopback inside the container, so +there is one process tree, one log, and one published port. + +`HALOGEN_MODEL_ID=frank-large` sets the id reported at `/v1/models`, in +`/health`, and in every response body. Requests are **never rejected for naming +a different model** — the server serves whatever asks. Keeping the fleet alias +is therefore free, and it means Hermes, the harness config, the OpenWebUI +discovery, the `services.json` probe and the docs needed no rename. + +## Weights + +`peonist-ai/halogen-qwen3.8-flash-next` (HuggingFace, public and ungated): + +| file | size | role | +|---|---|---| +| `qwen38-flash-next-w4b.hgn` | 115.55 GiB | the checkpoint | +| `qwen38-flash-next-w4b.overlay.hgn` | 2.31 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing | +| `qwen38-flash-next-vision.hgn` | 0.84 GiB | vision tower (see below) | +| `tokenizer/` | ~21 MB | flat tokenizer dir; found beside the checkpoint, so no extra mount | + +`HALOGEN_DOWNLOAD` fetches the whole repo into the bind mount on **first start +only** — the entrypoint acts only when the checkpoint is absent, so a restart +never re-downloads, and `huggingface_hub` resumes an interrupted pull. The +mount must be read-write for this. Roughly 127 GB; on Dankclaw it lands in +`/home/dank/models/halogen`. + +`overlay-speed.hgn` (2.22 GiB) comes along with the repo but is unused: it +trades the `o_proj` calibration for about 2 % of decode, and the quality +sidecar is the default for a reason. + +## Deployed flags + +Everything else is configured by environment variable — there is no config +file, and the flags are read once at startup. These are the departures from +upstream's defaults: + +| flag | value | why | +|---|---|---| +| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename | +| `HALOGEN_CTX` | `262144` | the model's native context, and what both clients declare | +| `HALOGEN_KV_POOL_POSITIONS` | `262144` | conservative for a **shared** 121 GiB host: one full-length conversation, or four at 65k, at ~7.2 GiB | +| `HALOGEN_KV_SLOTS` | `4` | four concurrent generations; slots share the pool and cost ~115 MiB each | +| `HALOGEN_HOST_RESERVE_GIB` | `24` | RAM kept for the n-gram table the engine reads through the page cache — above upstream's 20 because a dozen containers share this host | +| `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired | +| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch | + +Everything else is upstream's default, deliberately: prompt cache mode `2` +(two entries per conversation — end-of-system-prompt and end-of-history — which +already covers the shared-prefix case), `HALOGEN_CACHE_INPLACE=1` (the snapshot +stays ~115 MiB at any depth), the speculative drafter `1` (byte-identical at +temperature 0, and it stands down whenever a second conversation is +generating), the quality sidecar, `HALOGEN_ADMIT_CHUNK=0` (a non-zero value +gives up the byte-identity property), the baked `HALOGEN_MAX_TOK=32768` +(16.7 GiB prefill arena), and the default `HALOGEN_MAX_TOKENS_CAP=65536`. + +## Memory model + +Three numbers explain the whole box: + +- **~68 GiB pinned weights.** The checkpoint is mapped and registered with the + GPU in place, never copied. The kernel counts the locked pages as reclaimable + file cache, so `free`, `MemAvailable` and every dashboard built on them + **overstate free memory by the size of the model**. The server prints the + corrected figure at startup; believe that line, not Grafana. +- **~7.2 GiB per 262,144 KV positions**, plus the prefill arena (16.7 GiB at + `HALOGEN_MAX_TOK=32768`, 8.4 GiB at 16384). The engine's device-side budget + stops at roughly 47 GiB on a 128 GB machine *with the weights pinned* — and + lower on a machine carrying more besides this server, which Dankclaw is. +- **`HALOGEN_HOST_RESERVE_GIB`** is what is left for the page cache the lookup + table is read through. Too little and long prompts crawl with the disk busy + and the process in uninterruptible sleep — that is a cache problem, not a + memory problem, and the fix is a *smaller* pool, not a bigger one. + +`HALOGEN_KV_POOL_FIT=1` (default) measures that budget at startup and lowers +the pool itself if the configured one will not fit, printing what it chose. +Read those lines on every restart; they are the ground truth for sizing. + +**Co-tenancy.** `HALOGEN_FLASH_PIN_TRUNK=1` (default) is what the published +decode numbers were measured with, and it means this container holds most of +the host. If a co-tenant then takes the remainder, both it and the engine can +stall for minutes at a time at full CPU with no disk activity and no output. +That is documented behaviour, not a crash, and it needs no restart. +`HALOGEN_FLASH_PIN_TRUNK=0` gives the memory back and costs several times the +decode speed — a last resort, not a tuning option. + +## Client contract + +The differences from llama.cpp that bite are all in the request, not the +deployment: + +- **No server-side sampler defaults.** `temperature` absent or `0` is + **greedy** decode; above 0 the request samples on the same drafter. Supported + fields: `temperature`, `top_p`, `top_k`, `min_p`, `seed`, + `presence_penalty`, `frequency_penalty`, `logit_bias`, `logprobs`. Refused + with a 400: `top_logprobs`, `logprobs` with `stream: true`, `n > 1`, and any + value outside its defined range (values are refused, never clamped). + The fleet runs greedy for clients that stay silent — deterministic, and + byte-identical with the drafter. +- **The token budget covers thinking.** The default is **8192**, and a budget + that runs out mid-thought returns `finish_reason: "length"` with an **empty + `content`** and the partial reasoning in `reasoning_content` — which most + clients do not display. Above the cap (`HALOGEN_MAX_TOKENS_CAP`, 65536) a + request is a 400 rather than a silent truncation. `max_tokens`, + `max_completion_tokens` and `max_output_tokens` all work and must agree if + more than one is sent. +- **`reasoning_effort`** is accepted at the top level and inside + `chat_template_kwargs`: `minimal`, `low`, `medium`, `high`, `xhigh`. The + model's default is `xhigh`. `minimal` aliases the template's `low` and still + reasons; `enable_thinking: false` turns thinking off; `preserve_thinking` is + honoured. `preserve_reasoning` is a **400** — it is the llama.cpp-era key. +- `/v1/models`, `/health` and completions all report the served id, and + `reasoning_content` carries the reasoning trace. + +**Pool admission is the one arithmetic trap.** A request reserves its prompt +**plus** `max_tokens` positions when it is admitted; if the pool cannot hold it +the request waits in arrival order, and after the queue timeout (3600 s) it +gets a `503 engine_busy`. A prompt large enough that prompt + budget exceeds +the pool can therefore never be admitted at all. The 262,144-position pool is +the binding constraint on how large a conversation can get, not `HALOGEN_CTX` +and not the client's declared `context_length`. + +## Vision + +With the tower loaded, `/v1/chat/completions` accepts the usual OpenAI image +content part carrying a `data:` URL or bare base64 — **`http(s)` URLs are +refused deliberately**, because fetching one would make the server issue +outbound requests to wherever a client pointed it. One image costs about 2,040 +tokens of context and 5.5 / 11.8 / 25.3 s at 1280x800 / 1920x1080 / 2560x1440; +text at 12 pt and above is read exactly, and a bigger frame past that point +adds empty area rather than detail. Audio and video have no path here at all — +that capability left with [[frank-npu]]. + +## Deploy and operate + +```bash +./scripts/deploy.sh dankclaw --service frank-large +``` + +The handler retires the llama.cpp-era **root** unit first (stops it, drops its +enable symlinks, moves `/etc/systemd/system/frank-large.service` into the home +trash — never `rm`), then pulls `ghcr.io/peonist-ai/halogen-flash-server:0.5.8`, +re-tags it to the `localhost/` alias the unit references, installs the Quadlet, +and restarts the generated unit. Boot persistence comes from +`[Install] WantedBy=default.target` in the unit file plus linger on the deploy +user — generated units **cannot** be `systemctl --user enable`d on this host. + +The health gate is bootstrap-aware: while the checkpoint is absent the +container is downloading and `/health` cannot answer, so the handler waits for +the file instead of burning its budget and reporting a failure that is not one. + +```bash +systemctl --user status frank-large.service # the generated unit +podman logs frank-large | head -40 # version banner + memory ledger +curl -s http://127.0.0.1:7770/health # 200 only after the engine PONGs +curl -s http://127.0.0.1:7770/v1/models +``` + +`/health` asks the **engine** for a PONG before answering, so a 200 means +"ready to serve", not merely "process alive". During the first-start download +it stays unhealthy by design. + +## Troubleshooting + +| symptom | cause | +|---|---| +| `dmalloc: FAILED requesting … (out of memory)` then `HIP … out of memory` | the KV pool did not fit. `HALOGEN_KV_SLOTS` will not fix it (slots share the pool); lower `HALOGEN_KV_POOL_POSITIONS` to 262144, then `HALOGEN_MAX_TOK` to 16384 (~9 % prefill) | +| starts, answers short prompts, collapses to a few tok/s on a long one, disk busy, process in uninterruptible sleep | short of **file cache**, not memory — lower the pool and/or raise `HALOGEN_HOST_RESERVE_GIB` | +| long replies that look cut off | read `finish_reason`: `stop` means you have the whole answer, `length` means you ran out of budget (thinking counts) | +| a request that never returns, then `503 engine_busy` | the pool could not reserve prompt + `max_tokens`; see the admission trap above | +| both the engine and a co-tenant stall for minutes at full CPU | the pinned-trunk trade; see Memory model | +| `free`, `MemAvailable`, Grafana all wrong by ~68 GiB | expected — locked weights are counted as reclaimable | + +`HALOGEN_VERBOSE=1` and `HALOGEN_DMALLOC_LOG=1` print the fullest startup +account and every allocation over 64 MB; both are the vendor's suggested +evidence to attach to a report. The two lines worth grepping first: + +```bash +podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):' +``` + +## Rollback + +The llama.cpp era is intact: the STRIX payload stays at +`/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX`, the +`kingjones30/ROCmFPX` build stays at +`/home/dank/GitPackages/kingjones-ROCmFPX/build`, and the retired unit, env and +serve script are in `serverAppConfigs/.AGENTTRASH/` (suffixed `.llama.cpp`). +Rolling back is a `git revert` of this change set plus a deploy — **but the +Quadlet unit must be removed from `~/.config/containers/systemd/` first** +(otherwise the restarted root unit and the still-installed container both +claim port 7770). See [[kingjones-strix-rocmfp4]]. + +The `amd_iommu=off` kernel argument that came with this change is separate and +reverts on its own — see [[dankclaw-iommu-translation-off]]. + +## Citations + +- [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README, `docs/FLAGS.md`, `deploy/entrypoint.sh`, `docker-compose.yml`, CHANGELOG (v0.5.8) +- [peonist-ai/halogen-qwen3.8-flash-next](https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next) — weights and file inventory +- [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) — base model diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 692b9cb..6045a18 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -1,10 +1,9 @@ # LLM Fleet — Knowledge Base Documentation for the LLM inference fleet on two local machines -(2026-08-30): FreeToken v0.1.2 on Dankserver (frank-small, -Huihui-Qwen3.6-35B-A3B-abliterated-FP8) and kingjones30/ROCmFPX -llama.cpp on Dankclaw (frank-large, kingjones777 -Qwen3.8-Flash-Next ROCmFP4-STRIX). Both services run the Qwen-Sharp chat template. +(2026-09-11): FreeToken v0.1.2 on Dankserver (frank-small, +Huihui-Qwen3.6-35B-A3B-abliterated-FP8) and halogen-flash-server 0.5.8 on +Dankclaw (frank-large, peonist-ai Qwen3.8-Flash-Next w4b). > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -26,10 +25,10 @@ block-fp8, 20% MoE residency, 453k KV / 4 streams] subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] E2[gfx1151 iGPU 64 GiB unified memory] - E[kingjones30/ROCmFPX build + E[halogen-flash-server 0.5.8 frank-large, port 7770] - F[kingjones777 Qwen3.8-Flash-Next -ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE] + F[peonist-ai Qwen3.8-Flash-Next +w4b .hgn + overlay, 262k ctx / 4 slots] E2 --> E --> F end @@ -53,14 +52,19 @@ ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE] (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090 - [[deployment]] — deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[kingjones-strix-rocmfp4]] — ACTIVE frank-large runbook: - kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params +- [[halogen-flash-server]] — ACTIVE frank-large runbook since 2026-09-11: + peonist-ai/halogen-flash-server 0.5.8 serving + peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn + + quality overlay), rootless Quadlet container on Dankclaw port 7770 — + deployed flags, memory model, client contract, troubleshooting +- [[dankclaw-iommu-translation-off]] — why Dankclaw boots with + `amd_iommu=off` since 2026-09-11 (13-16 % prefill, measured upstream), + what it cost (the NPU endpoint), what was deliberately not copied from + the reference machine, and how to revert +- [[kingjones-strix-rocmfp4]] — PRIOR frank-large runbook and rollback + target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params (56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned - kingjones30/ROCmFPX fork (swapped 2026-08-30) -- [[agentionai-fast-imatrix]] — retired frank-large runbook: - agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB - Q4_0_ROCMFP4_FAST imatrix, v2 joined PLE) on build-qwen38 - (2026-08-29 → 2026-08-30); rollback binary + payload retained + kingjones30/ROCmFPX fork (2026-08-30 → 2026-09-11) - [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results - [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure - [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model @@ -78,11 +82,11 @@ ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE] (deployed and reverted 2026-08-22) - [[Dankserver System Specs]] — the RTX 3090 box (frank-small) - [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large, - served on the pinned kingjones30/ROCmFPX llama.cpp fork since 2026-08-30) -- [[frank-npu]] — ACTIVE FLM NPU sight+sound runbook on - Dankclaw: Gemma4 E4B vision+audio pack on the XDNA2 NPU (FLM v1.0.1, port - 7771, whole-registry serving, per-request model routing), incl. the - IOMMU/SVA root cause and the `iommu=pt` fix behind its first-deploy crash + on halogen-flash-server since 2026-09-11) +- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint + on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as + the retirement record — the IOMMU/SVA err=-19 root cause and where + every dependent route moved ## CIRU Reference (Dankclaw) diff --git a/serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md b/serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md index 9a1d853..4051584 100644 --- a/serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md +++ b/serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md @@ -1,7 +1,7 @@ --- type: reference -title: kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw -description: ACTIVE frank-large runbook. kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the card's pinned kingjones30/ROCmFPX fork. 177B params (56B MoE + 51.2B PLE streamed via mmap), single 113.47 GiB GGUF, 401408 ctx x 2 slots. +title: kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw (prior, rollback target) +description: PRIOR frank-large runbook and rollback target (2026-08-30 → 2026-09-11). kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the card's pinned kingjones30/ROCmFPX fork. 177B params (56B MoE + 51.2B PLE streamed via mmap), single 113.47 GiB GGUF, 401408 ctx x 2 slots. tags: - llama.cpp - rocm @@ -16,12 +16,22 @@ timestamp: 2026-08-30 # kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw -> **Active again since 2026-08-30** on the model card's **pinned runtime**: -> the `kingjones30/ROCmFPX` fork (replaced the agentionai FAST deployment -> which ran on charlie's `build-qwen38`). Text-only tier; the agentionai -> f16 vision tower is paired for mmproj and loads clean. Speculative -> decoding is OFF (the card ships no spec flags; the Flash-Next MTP heads -> do not load on this loader family — see MTP Draft Head). +> **PRIOR / ROLLBACK TARGET since 2026-09-11** — frank-large moved to +> [[halogen-flash-server]] (peonist-ai halogen-flash-server 0.5.8 serving +> Qwen3.8-Flash-Next w4b), which upstream measures at roughly 4x this +> runtime's prefill. Everything below documents the llama.cpp era and stays +> accurate for a rollback: the payload is still at +> `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX`, the binary +> at `/home/dank/GitPackages/kingjones-ROCmFPX/build`, and the unit, env and +> serve script are staged in `serverAppConfigs/.AGENTTRASH/` (`.llama.cpp`). +> Rolling back also requires removing the halogen Quadlet from +> `~/.config/containers/systemd/` so it stops claiming port 7770. +> +> Ran 2026-08-30 → 2026-09-11 on the model card's **pinned runtime**: the +> `kingjones30/ROCmFPX` fork. Text-only tier; the agentionai f16 vision +> tower was paired for mmproj and loads clean. Speculative decoding is OFF +> (the card ships no spec flags; the Flash-Next MTP heads do not load on +> this loader family — see MTP Draft Head). `frank-large` (port 7770) serves `Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf` (single file, 113.47 GiB) diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index c91f4d9..e72b69c 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -121,9 +121,17 @@ The LLM inference fleet on the two hosts. Full index: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB, Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30; rollback record) -- [[kingjones-strix-rocmfp4]] — the ACTIVE frank-large runbook since - 2026-08-30: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF (177B params, - 113.47 GiB) on the card's pinned kingjones30/ROCmFPX build +- [[halogen-flash-server]] — the ACTIVE frank-large runbook since + 2026-09-11: peonist-ai/halogen-flash-server 0.5.8 serving + peonist-ai/halogen-qwen3.8-flash-next (w4b) as a rootless Quadlet + container on Dankclaw, port 7770 — flags, memory model, client contract +- [[dankclaw-iommu-translation-off]] — why Dankclaw boots with + `amd_iommu=off` since 2026-09-11 (13-16 % prefill), what it cost + (the NPU endpoint), and how to revert +- [[kingjones-strix-rocmfp4]] — the PRIOR frank-large runbook + (2026-08-30 → 2026-09-11) and the rollback target: + kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF on the card's + pinned kingjones30/ROCmFPX build - [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure - [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) @@ -135,10 +143,10 @@ The LLM inference fleet on the two hosts. Full index: - [[Dankserver System Specs]] — the RTX 3090 box hardware - [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large) - [[Dankclaw Improvements]] — host-level changes on the Strix Halo box -- [[frank-npu]] — ACTIVE FLM NPU sight+sound runbook on Dankclaw: - Gemma4 E4B vision+audio pack on the XDNA2 NPU, port 7771; whole local - registry served on one port with per-request model routing; includes the - IOMMU/SVA root cause and the `iommu=pt` fix behind its first-deploy crash +- [[frank-npu]] — the RETIRED FLM NPU endpoint on Dankclaw (2026-09-11): + Gemma4 E4B vision+audio pack on the XDNA2 NPU, port 7771; kept as the + retirement record, including the IOMMU/SVA err=-19 root cause and + where every dependent route moved ### Benchmarks and Sizing diff --git a/shared/homepage/services.json b/shared/homepage/services.json index 0b4648f..69b78c3 100644 --- a/shared/homepage/services.json +++ b/shared/homepage/services.json @@ -212,7 +212,7 @@ { "name": "frank-large", "category": "admin", - "description": "The large LLM (27B) on Dankclaw.", + "description": "The large LLM (Qwen3.8-Flash-Next, halogen) on Dankclaw.", "url": "", "docs": "", "probe": "127.0.0.1:7770", diff --git a/tests/deployment_scoping.bats b/tests/deployment_scoping.bats index adbc857..c921e8a 100644 --- a/tests/deployment_scoping.bats +++ b/tests/deployment_scoping.bats @@ -18,15 +18,23 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" @test "deployment planner maps large-model changes to the large model only" { run python3 "$PLANNER" --target dankclaw --paths \ - scripts/frank-large.env scripts/frank-large.service + dankclaw/frank-large/quadlet/frank-large.container [ "$status" -eq 0 ] [ "$output" = "frank-large" ] } -@test "shared chat-template changes map to the selected host's inference service" { +@test "the retired NPU endpoint is absent from the deployment inventory" { + ! grep -q '"frank-npu"' "$PLANNER" + ! grep -q 'frank_npu\|frank-npu' "$REMOTE_DEPLOY" +} + +@test "shared chat-template changes map only to the service that reads it" { + # frank-large runs halogen-flash-server, which owns its own chat handling; + # frank-small is the only consumer of the tracked template now, so a + # template edit is a no-op on Dankclaw. run python3 "$PLANNER" --target dankclaw --paths scripts/chat-template.jinja [ "$status" -eq 0 ] - [ "$output" = "frank-large" ] + [ -z "$output" ] run python3 "$PLANNER" --target dankserver --paths scripts/chat-template.jinja [ "$status" -eq 0 ] @@ -68,10 +76,6 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" run python3 "$PLANNER" --target dankclaw --paths dankclaw/hermes/config.yaml.example [ "$status" -eq 0 ] [ "$output" = "hermes" ] - - run python3 "$PLANNER" --target dankclaw --paths scripts/serve-frank-npu.sh scripts/frank-npu.env - [ "$status" -eq 0 ] - [ "$output" = "frank-npu" ] } @test "homepage is owned by Dankclaw's consolidated proxy" { diff --git a/tests/frank_large_quadlet.bats b/tests/frank_large_quadlet.bats new file mode 100755 index 0000000..0220b16 --- /dev/null +++ b/tests/frank_large_quadlet.bats @@ -0,0 +1,95 @@ +#!/usr/bin/env bats +# Seam: the frank-large quadlet on dankclaw. +# frank-large became a rootless Quadlet container on 2026-09-11 when +# halogen-flash-server replaced the llama.cpp/kingjones runtime. This seam pins +# the properties that make it work and that no reviewer can see by reading the +# flags alone: GPU device access and the image's required podman posture (the +# engine maps the checkpoint and registers it with the GPU in place), the +# single published port that keeps every client pointed where it has always +# pointed, the read-write weights mount the first-start download needs, boot +# persistence, and a deploy handler that retires the old root unit without +# deleting anything. + +load helpers + +UNIT="$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container" +REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" + +@test "frank-large quadlet exists under dankclaw/frank-large/quadlet/" { + [ -f "$UNIT" ] +} + +@test "frank-large unit is boot-persistent (WantedBy=default.target)" { + # Generated units cannot be enabled on this host's podman 5.x v1 quadlet + # generator; [Install] plus linger is the entire boot-persistence story. + grep -q '^WantedBy=default.target$' "$UNIT" +} + +@test "frank-large unit is self-healing" { + grep -q '^Restart=on-failure$' "$UNIT" + grep -q '^RestartSec=' "$UNIT" +} + +@test "frank-large image is pinned to an exact version tag" { + grep -Eq '^Image=localhost/peonist-ai/halogen-flash-server:[0-9]+\.[0-9]+\.[0-9]+$' "$UNIT" +} + +@test "frank-large can reach the GPU (kfd + DRM render node)" { + # Without both devices the engine cannot map the checkpoint at all; the + # failure is a load error, not a degraded mode. + grep -q '^AddDevice=/dev/kfd$' "$UNIT" + grep -q '^AddDevice=/dev/dri$' "$UNIT" +} + +@test "frank-large runs with the podman posture the image requires" { + # The image documents --ipc=host, an unconfined seccomp profile and an + # unlimited memlock as hard requirements. + grep -q -- '--ipc=host' "$UNIT" + grep -q -- '--security-opt seccomp=unconfined' "$UNIT" + grep -q '^Ulimit=memlock=-1:-1$' "$UNIT" +} + +@test "frank-large publishes exactly one port and it is the client port" { + # 8731 is the OpenAI front-end; the engine on 8730 is unauthenticated and + # must never be published. + [ "$(grep -c '^PublishPort=' "$UNIT")" -eq 1 ] + grep -q '^PublishPort=7770:8731$' "$UNIT" +} + +@test "frank-large keeps the fleet identity every client already points at" { + # The id at /v1/models, in /health and in every response body. + grep -q '^Environment=HALOGEN_MODEL_ID=frank-large$' "$UNIT" +} + +@test "the weights mount is writable, because the first start downloads into it" { + grep -q '^Volume=/home/dank/models/halogen:/models$' "$UNIT" + ! grep -q '^Volume=.*:/models:ro$' "$UNIT" + grep -q '^Environment=HALOGEN_DOWNLOAD=' "$UNIT" +} + +@test "frank-large unit carries no secret material" { + ! grep -Eq '^(Environment|EnvironmentFile)=.*(TOKEN|PASSWORD|SECRET|KEY)' "$UNIT" +} + +@test "the deploy handler retires the old root unit without deleting it" { + grep -q 'retire_root_llm_large' "$REMOTE_DEPLOY" + grep -q 'systemctl stop frank-large.service' "$REMOTE_DEPLOY" + # Never `rm`: the retired unit moves into the home trash so the llama.cpp + # era stays recoverable. + grep -q 'mv -- /etc/systemd/system/frank-large.service' "$REMOTE_DEPLOY" + ! grep -qE '(^|[^[:alnum:]_])rm (-[a-zA-Z]+ )?/etc/systemd/system/frank-large' "$REMOTE_DEPLOY" +} + +@test "the deploy handler never tries to enable a generated unit" { + # `systemctl --user enable frank-large.service` fails on this host + # ("transient or generated"); the deploy must install and restart instead. + ! grep -q 'systemctl --user enable frank-large.service' "$REMOTE_DEPLOY" + grep -q 'install_quadlet frank-large dankclaw/frank-large/quadlet frank-large' "$REMOTE_DEPLOY" + grep -q 'systemctl --user restart frank-large.service' "$REMOTE_DEPLOY" +} + +@test "the deploy handler gates the first start on the download, not on a failure" { + grep -q 'wait_for_llm_large_ready' "$REMOTE_DEPLOY" + grep -q 'qwen38-flash-next-w4b.hgn' "$REMOTE_DEPLOY" + grep -q '127.0.0.1:7770/health' "$REMOTE_DEPLOY" +}