diff --git a/AGENTS.md b/AGENTS.md index 170b615..73f8556 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -28,6 +28,7 @@ host. |--------|---------|---------|-------|------| | Dankserver | FreeToken v0.1.2 | frank-default-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 | | Dankclaw | kingjones30/ROCmFPX llama.cpp (card-pinned fork) | frank-default-large | kingjones777 Qwen3.8-Flash-Next-ROCmFP4-STRIX (113.47 GiB single GGUF, Q4_0_ROCMFP4_STRIX, PLE streams via mmap), 401408 ctx / 2 slots, fa on, no spec decode, reasoning on | 7770 | +| Dankclaw | FastFlowLM (FLM) v1.0.1 (`flm serve`, dpkg `fastflowlm`) on the XDNA2 NPU | frank-default-sightsound | Gemma4-E4B-IT-NPU2 pack (vision + audio towers); whole local registry served on one port, per-request routing by exact FLM tag; `iommu=pt` boot arg mandatory (SVA bind needs enabled IOMMU) | 52626 | History: the 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on @@ -45,7 +46,9 @@ Dankserver moved to the FreeToken runtime serving the official Qwen3.6-35B-A3B-FP8. On 2026-08-30 both services swapped again: Dankclaw to the kingjones STRIX model on the card's pinned kingjones30/ROCmFPX runtime, Dankserver to the Huihui abliterated -FP8 checkpoint on FreeToken. Runbooks: +FP8 checkpoint on FreeToken. On 2026-08-30 the NPU endpoint returned as +`frank-default-sightsound` — FLM v1.0.1 on XDNA2 serving sight+sound (see the +Runtime & Troubleshooting section and its runbook). Runbooks: [serverAppConfigs-docs/LLM/index.md](serverAppConfigs-docs/LLM/index.md). Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`. @@ -279,6 +282,40 @@ When you rename a service or move a model to a new host, update ## Runtime & Troubleshooting +### frank-default-sightsound — FLM NPU sight+sound endpoint (Dankclaw, ACTIVE) + +The third inference endpoint on Dankclaw. The `flm serve` runtime (dpkg +package `fastflowlm`, v1.0.1, binary `/usr/bin/flm`) serves the **whole local +model registry** (36 tags at first deploy) on a single port — 52626 — and +routes chat completions per request by the exact FLM registry tag in the +`model` field. Unknown tags do NOT 404: they silently serve `llama3.2:1b`, +so clients must send exact registry tags, never service aliases. Primary +model = Gemma4 E4B IT NPU2 pack (vision + audio towers); the uncensored +candidate (`qwythos:9b`) is text-only and fails the modality requirement, so +this endpoint trades "uncensored" for "sight + sound". The NPU is a +single-lane device — generations serialize FIFO; treat this as a +background/triage endpoint, not an interactive multi-user chat. + +Hard requirements verified at first deploy (2026-08-30): + +- **IOMMU must be enabled.** With `iommu=off` on the kernel cmdline every + open of `/dev/accel/accel0` fails with `err=-19`: the XDNA2 driver's SVA + bind needs PASID/SVA, which requires enabled IOMMU hardware. Fix applied: + `iommu=pt` in `GRUB_CMDLINE_LINUX_DEFAULT` + reboot (passthrough keeps the + iGPU default domain identity-mapped — zero DMA-translation overhead for + frank-default-large — while enabling IOMMU + PASID; PASID domains are + paged regardless of the default-domain type). Never set `iommu=off` on + this host again; it silently kills the NPU. +- Health probe = `GET /v1/version` (FLM has no `/health` route). +- Unit: `User=dank`, `LimitMEMLOCK=infinity` (FLM requires unlimited + memlock), `Restart=on-failure`, `RestartSec=10`. Deploy via + `scripts/deploy.sh dankclaw --service frank-default-sightsound`. +- Verified live at first deploy: image probe (red PNG as `image_url` + data-URI → correct color answer, vision tower consumed the pixels) and an + audio control-pair probe (tone WAV via `input_audio` consumed ~22 + audio-tower tokens vs a text-only control). + +Runbook: [serverAppConfigs-docs/LLM/frank-default-sightsound.md](serverAppConfigs-docs/LLM/frank-default-sightsound.md). ### kingjones Qwen3.8-Flash-Next STRIX on kingjones30/ROCmFPX (Dankclaw, ACTIVE) The large service serves diff --git a/serverAppConfigs-docs/LLM/frank-default-sightsound.md b/serverAppConfigs-docs/LLM/frank-default-sightsound.md new file mode 100644 index 0000000..ed06ece --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-default-sightsound.md @@ -0,0 +1,106 @@ +--- +type: configuration +title: frank-default-sightsound Runbook +description: FastFlowLM (FLM) NPU sight+sound endpoint on Dankclaw — XDNA2 NPU serving the Gemma4 E4B vision+audio pack on port 52626, including the IOMMU/SVA root cause that took it down at first deploy. +tags: + - flm + - fastflowlm + - xdna2 + - npu + - dankclaw + - frank-default-sightsound +timestamp: 2026-08-30 +--- + +# frank-default-sightsound — FLM NPU sight+sound Runbook + +`frank-default-sightsound` is the third inference endpoint on Dankclaw: a +FastFlowLM (FLM) v1.0.1 server (`/usr/bin/flm`, dpkg package `fastflowlm`) +running on the **XDNA2 NPU** (50 TOPS). It exists to serve *sight and sound* — +image and audio input — which the iGPU llama.cpp fleet ([[kingjones-strix-rocmfp4]], +`frank-default-large`) does not do. The model-selection tradeoff: the only +NPU2 pack family on this box with BOTH a vision and an audio tower is the +Gemma4 E4B IT pack; the uncensored candidate (`qwythos:9b`, 8.94 GB, 262k ctx) +is text-only and fails the hard modality requirement, so this endpoint trades +"uncensored" for "sight + sound". + +## Deployment shape + +| Field | Value | +|-------|-------| +| Runtime | FLM v1.0.1 (`/usr/bin/flm`, dpkg `fastflowlm`) on the XDNA2 NPU | +| Service | `frank-default-sightsound.service` (systemd system unit, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`) | +| Files | `scripts/frank-default-sightsound.env`, `scripts/serve-frank-default-sightsound.sh`, `scripts/frank-default-sightsound.service` | +| Deploy | `scripts/deploy.sh dankclaw --service frank-default-sightsound` (relink + daemon-reload + enable + restart; health gate polls `/v1/version`, 300 s cap — FLM has **no `/health` route**) | +| Port | `52626` on `0.0.0.0` (52625 left free for ad-hoc `flm serve` experiments) | +| Power mode | `FLM_PMODE=turbo` — no other tenant uses the NPU on this host (the large model runs on the iGPU), so the top power mode costs no neighbour anything; drop to `performance` if iGPU power-budget contention ever shows | +| Context | `FLM_CTX_LEN` = verified max for the E4B pack on this box (native 128K context; the registry's default context length is only the fallback). FLM rounds non-power-of-2 values up. | +| ASR | `FLM_ASR=0` — the E4B pack answers audio natively through its own audio tower in `/v1/chat/completions`; setting `FLM_ASR=1` additionally loads Whisper-V3-Turbo-NPU2 (+0.62 GB) and exposes `/v1/audio/transcriptions` | + +The serve script exports `FLM_CONFIG_PATH=/home/dank/.config/flm/model_list.json` +and `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins`; kernels are symlinked +from `/opt/fastflowlm/share/flm/xclbins/` into the per-pack directories. A tag +swap is therefore an env-file edit, not a script edit. + +## Serving model and API contract + +- The endpoint serves the **whole local registry** (36 tags at first deploy) on + one port; `/v1/models` lists them all. Chat completions route per request by + the exact FLM registry tag in the `model` field — never a service alias. +- **Unknown model tags do NOT 404**: FLM logs an error and silently serves + `llama3.2:1b` instead. Client configs must carry the exact registry tag or + they will silently get the 1B fallback. +- Multimodal content parts accepted by `/v1/chat/completions`: + `{"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}` and + `{"type":"input_audio","input_audio":{"data":"","format":"wav"}}`. +- Chat template roles: only `system`/`user`/`assistant`; any other role fails + the request with `Unexpected message role.`. +- The NPU is a single-lane device: generations serialize in FIFO order. This is + a background/triage endpoint, not an interactive multi-user chat. + +## Live-verified behavior (2026-08-30, first deploy) + +- Vision tower: a generated red 8×8 PNG posted as an `image_url` data-URI was + answered correctly ("The image is a solid red color."); the request consumed + 279 prompt tokens — the vision tower consumed the pixels (a text-only model + given zero image bytes cannot answer "what color"). +- Audio tower: a 440 Hz decaying sine WAV posted as `input_audio` consumed + 37 prompt tokens vs 15 for the identical text-only control → ~22 audio-tower + tokens; the model answered truthfully that it heard no speech (it is a tone, + not speech — ASR stays off). + +## The IOMMU/SVA root cause (first deploy crashed, fixed same day) + +First deploy crash-looped every 10 s (`RestartSec=10`) with +`Error: Open /dev/accel/accel0 failed (err=-19): No such device`. Root-cause +chain, all live-verified on the host: + +1. `amdxdna_drm_open` was failing **SVA bind with ret -19** for every client — + the XDNA2 driver cannot hand out the accel device without PASID/SVA. +2. SVA requires the AMD IOMMU to be enabled; the kernel cmdline carried + `iommu=off` (a perf/DMA tuning experiment from 2026-08-20, applied at the + 2026-08-26 reboot). With IOMMU off there are no IOMMU groups and no PASID + state; `PASIDCtl: Enable-` was observed disabled on `0000:c5:00.1`. +3. Fix: replace `iommu=off` with **`iommu=pt`** in `GRUB_CMDLINE_LINUX_DEFAULT` + on `/etc/default/grub`, `update-grub`, reboot. Passthrough mode keeps the + iGPU default domain identity-mapped (zero DMA-translation overhead for + `frank-default-large`) while enabling IOMMU hardware + PASID, which is what + the NPU driver's SVA bind requires — PASID domains are paged regardless of + the default-domain type. +4. Post-boot verification: `/sys/kernel/iommu_groups/` populated (30 groups), + zero `SVA bind device failed` journal entries since boot, accel0 recreated at + boot, service active with `NRestarts=0`, and the image/audio probes above + served live. + +The same root cause explains the older `amdgpu 0000:c4:00.0: no vmid pasid +mapping supported` kernel log line — a consequence of IOMMU-off, not a GPU +fault. Do not set `iommu=off` on this host again; it silently kills the NPU. + +## Client wiring + +The pi client model config (`/home/dank/.pi/agent/models.json` on Dankware) +registers this endpoint under the Dankclaw provider with the **exact** FLM +registry tag as the model `id` (see above: unknown tags silently fall back to +the 1B), `baseUrl` `http://192.168.0.143:52626/v1`, and a per-request context +window equal to the serving ctx-len. Health checks for this service must use +`GET /v1/version` (there is no `/health` route). diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 546ba41..fbec4a9 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -78,6 +78,10 @@ ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE] (deployed and reverted 2026-08-22) - [[Dankserver System Specs]] — the RTX 3090 box (frank-default-small) - [[Dankclaw System Specs]] — the CIRU ROCmFPX box (frank-default-large) +- [[frank-default-sightsound]] — ACTIVE FLM NPU sight+sound runbook on + Dankclaw: Gemma4 E4B vision+audio pack on the XDNA2 NPU (FLM v1.0.1, port + 52626, whole-registry serving, per-request model routing), incl. the + IOMMU/SVA root cause and the `iommu=pt` fix behind its first-deploy crash ## CIRU Reference (Dankclaw) diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 716b15c..ba72348 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -127,6 +127,10 @@ The LLM inference fleet on the two hosts. Full index: - [[Dankserver System Specs]] — the RTX 3090 box hardware - [[Dankclaw System Specs]] — the CIRU ROCmFPX box hardware - [[Dankclaw Improvements]] — host-level changes on the CIRU box +- [[frank-default-sightsound]] — ACTIVE FLM NPU sight+sound runbook on Dankclaw: + Gemma4 E4B vision+audio pack on the XDNA2 NPU, port 52626; whole local + registry served on one port with per-request model routing; includes the + IOMMU/SVA root cause and the `iommu=pt` fix behind its first-deploy crash ### Benchmarks and Sizing diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 806efde..140b3b6 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -11,6 +11,38 @@ timestamp: 2026-09-14 # Bundle Change Log +## 2026-08-30 (night II) — frank-default-sightsound deployed; NPU SVA/IOMMU root cause +- **Deployed** `frank-default-sightsound` on Dankclaw: FLM v1.0.1 (`/usr/bin/flm`, + dpkg `fastflowlm`) serving the whole local registry (36 tags) on port 52626; + primary model Gemma4 E4B IT NPU2 (vision + audio towers). First deploy + crash-looped every 10 s: `Open /dev/accel/accel0 failed (err=-19)`. +- **Root cause** (live-verified): the XDNA2 driver's SVA bind fails with ret -19 + whenever the AMD IOMMU is off; the kernel cmdline carried `iommu=off` (a + perf/DMA tuning experiment from 08-20, applied at the 08-26 reboot). No IOMMU + groups ⇒ no PASID state ⇒ accel0 open returns ENODEV for every client. The + same root cause explains the `amdgpu ... no vmid pasid mapping supported` + kernel line — a consequence, not a GPU fault. +- **Fix**: `iommu=pt` in `GRUB_CMDLINE_LINUX_DEFAULT` + reboot (passthrough keeps + the iGPU default domain identity-mapped — zero DMA-translation overhead for + frank-default-large — while enabling IOMMU hardware + PASID, which SVA needs; + PASID domains are paged regardless of the default-domain type). Post-boot: + 30 IOMMU groups, zero SVA-bind errors since boot, unit `enabled`, + `NRestarts=0` since boot. +- **Live-verified behavior**: whole-registry serving with per-request routing by + exact FLM tag (unknown tags silently serve `llama3.2:1b`); vision tower proved + end-to-end (generated red PNG as `image_url` data-URI → correct color answer, + 279 prompt tokens consumed); audio tower proven with a control pair (440 Hz + tone via `input_audio` = 37 prompt tokens vs 15 for the identical text-only + control). Health probe is `/v1/version` — FLM has no `/health` route. +- `/scripts/frank-default-sightsound.env`, `/scripts/serve-frank-default-sightsound.sh`, + `/scripts/frank-default-sightsound.service` + deployer allowlist entries + (`c3ed3a6`, `cf0c541`) — service files and deploy wiring; unit is boot-persistent + (`WantedBy=multi-user.target`, enabled, `Restart=on-failure`, `RestartSec=10`, + `LimitMEMLOCK=infinity` — FLM requires unlimited memlock). +- **Docs**: new runbook [[frank-default-sightsound]] (linked from both indexes), + AGENTS.md fleet table + runtime section updated with the IOMMU/SVA hard + requirement. Client wiring: pi `models.json` registers the endpoint under the + Dankclaw provider with the exact FLM registry tag as model id. ## 2026-08-30 (night) — frank fleet double swap: STRIX on its pinned fork, Huihui on FreeToken - **Dankclaw / frank-default-large**: swapped to