diff --git a/serverAppConfigs-docs/LLM/frank-default-sightsound.md b/serverAppConfigs-docs/LLM/frank-default-sightsound.md index 9791dea..b449652 100644 --- a/serverAppConfigs-docs/LLM/frank-default-sightsound.md +++ b/serverAppConfigs-docs/LLM/frank-default-sightsound.md @@ -1,7 +1,7 @@ --- type: configuration title: frank-default-sightsound Runbook -description: FastFlowLM (FLM) NPU sight+sound endpoint on Dankclaw — XDNA2 NPU serving the Gemma4 E4B vision+audio pack on port 7771, including the IOMMU/SVA root cause that took it down at first deploy. +description: FastFlowLM (FLM) NPU sight+sound endpoint on Dankclaw — XDNA2 NPU serving Qwen3.8 Distilled Heretic vision+text pack on port 7771, including the IOMMU/SVA root cause that took it down at first deploy. tags: - flm - fastflowlm @@ -9,15 +9,15 @@ tags: - npu - dankclaw - frank-default-sightsound -timestamp: 2026-08-30 +timestamp: 2026-09-01 --- # frank-default-sightsound — FLM NPU sight+sound Runbook `frank-default-sightsound` is the third inference endpoint on Dankclaw: a FastFlowLM (FLM) v1.0.1 server (`/usr/bin/flm`, dpkg package `fastflowlm`) -running on the **XDNA2 NPU** (50 TOPS). It exists to serve *sight and sound* — -image and audio input — which the iGPU llama.cpp fleet ([[kingjones-strix-rocmfp4]], +running on the **XDNA2 NPU** (50 TOPS). It serves *sight* (vision) — +which the iGPU llama.cpp fleet ([[kingjones-strix-rocmfp4]], `frank-default-large`) does not do. The model-selection tradeoff: the only NPU2 pack family on this box with BOTH a vision and an audio tower is the Gemma4 E4B IT pack; the uncensored candidate (`qwythos:9b`, 8.94 GB, 262k ctx) @@ -34,14 +34,44 @@ is text-only and fails the hard modality requirement, so this endpoint trades | Deploy | `scripts/deploy.sh dankclaw --service frank-default-sightsound` (relink + daemon-reload + enable + restart; health gate polls `/v1/version`, 300 s cap — FLM has **no `/health` route**) | | Port | `7771` on `0.0.0.0` (52625/52626 left free for ad-hoc `flm serve` experiments) | | Power mode | `FLM_PMODE=turbo` — no other tenant uses the NPU on this host (the large model runs on the iGPU), so the top power mode costs no neighbour anything; drop to `performance` if iGPU power-budget contention ever shows | -| Context | `FLM_CTX_LEN` = verified max for the E4B pack on this box (native 128K context; the registry's default context length is only the fallback). FLM rounds non-power-of-2 values up. | -| ASR | `FLM_ASR=0` — the E4B pack answers audio natively through its own audio tower in `/v1/chat/completions`; setting `FLM_ASR=1` additionally loads Whisper-V3-Turbo-NPU2 (+0.62 GB) and exposes `/v1/audio/transcriptions` | +| Context | `FLM_CTX_LEN=131072 (128K; verified max on this box; native context length.) | +| ASR | `FLM_ASR=0` — the E4B pack answers images natively through its vision tower in `/v1/chat/completions`; setting `FLM_ASR=1` additionally loads Whisper-V3-Turbo-NPU2 (+0.62 GB) and exposes `/v1/audio/transcriptions` | The serve script exports `FLM_CONFIG_PATH=/home/dank/.config/flm/model_list.json` and `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins`; kernels are symlinked from `/opt/fastflowlm/share/flm/xclbins/` into the per-pack directories. A tag swap is therefore an env-file edit, not a script edit. +## Model tags for switching + +FLM routes per-request by the exact registry tag in the `model` field. To swap +the service model, change `FLM_MODEL` in the env file and restart. To let +clients choose, send requests with the desired `model` field. + +| Tag | Model | Modality | Size | Family | +|-----|-------|----------|------|--------| +| `qwen3.8-distilled-heretic:9b` | Qwen3.8 Distilled Heretic | vision + reasoning | 7.6 GB | qwen3.5 | +| `gemma4-it:e4b` | Gemma4 E4B IT | vision + audio | 9.1 GB | gemma4e | +| `qwythos:9b` | Qwythos 9B Claude Mythos | vision + reasoning | 8.9 GB | qwen3.5 | +| `gpt-oss:20b` | GPT-OSS 20B | text only | 13.8 GB | gpt-oss | + +**To swap the active model** (changes what the service loads at start): +``` +sed -i 's/^FLM_MODEL=.*/FLM_MODEL=/' scripts/frank-default-sightsound.env +./scripts/deploy.sh dankclaw --service frank-default-sightsound +``` + +**To switch per-request** (service stays loaded, client sends the tag): +``` +curl http://192.168.0.143:7771/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{"model":"gemma4-it:e4b","messages":[...],"max_tokens":64}' +``` + +Known tags are in the user registry at +`/home/dank/.config/flm/model_list.json`; unknown tags answer HTTP 200 with +`{"error":"Failed to load model!"}` (no choices). + ## Serving model and API contract - The endpoint serves the **whole local registry** (36 tags at first deploy) on