From 9b9f0f9051ba237772d330b4a7e0636ecaaf8305 Mon Sep 17 00:00:00 2001 From: Frank Date: Sun, 30 Aug 2026 11:41:15 -0700 Subject: [PATCH] docs(sightsound): correct unknown-model-tag contract to live-probed behavior Prior docs claimed unknown tags silently serve llama3.2:1b. Live probes on the live endpoint disprove it: /v1/chat/completions answers HTTP 200 with a flat {"error":"Failed to load model!"} body and no choices; /v1/completions answers a nested OpenAI-shaped server_error (code 500) under HTTP 200. Valid tags route per-request to the named model (response echoes the requested tag); no restart needed to switch between registry tags. Corrected in the runbook, the AGENTS.md runtime section, the serve-script header comment, and the historical FastFlowLM incident note. --- AGENTS.md | 6 ++++-- scripts/serve-frank-default-sightsound.sh | 5 +++-- .../LLM/FastFlowLM NPU Inference.md | 5 +++++ .../LLM/frank-default-sightsound.md | 15 ++++++++++----- 4 files changed, 22 insertions(+), 9 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 73f8556..f7171a0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -288,8 +288,10 @@ The third inference endpoint on Dankclaw. The `flm serve` runtime (dpkg package `fastflowlm`, v1.0.1, binary `/usr/bin/flm`) serves the **whole local model registry** (36 tags at first deploy) on a single port — 52626 — and routes chat completions per request by the exact FLM registry tag in the -`model` field. Unknown tags do NOT 404: they silently serve `llama3.2:1b`, -so clients must send exact registry tags, never service aliases. Primary +`model` field. Unknown tags do NOT 404: `/v1/chat/completions` answers HTTP 200 +with `{"error":"Failed to load model!"}` and no choices (status-only +clients silently get nothing), so clients must send exact registry tags, never +service aliases. Primary model = Gemma4 E4B IT NPU2 pack (vision + audio towers); the uncensored candidate (`qwythos:9b`) is text-only and fails the modality requirement, so this endpoint trades "uncensored" for "sight + sound". The NPU is a diff --git a/scripts/serve-frank-default-sightsound.sh b/scripts/serve-frank-default-sightsound.sh index 3720e8d..6db59e5 100755 --- a/scripts/serve-frank-default-sightsound.sh +++ b/scripts/serve-frank-default-sightsound.sh @@ -9,8 +9,9 @@ # for the memory budget and the model-selection tradeoff. # # NOTE: FLM requires unlimited memlock. The unit sets LimitMEMLOCK=infinity. -# NOTE: an unknown model tag does NOT 404 — FLM logs an error and silently -# serves llama3.2:1b instead. Client model IDs must be the exact FLM registry +# NOTE: an unknown model tag does NOT 404 — /v1/chat/completions answers HTTP 200 with a +# flat {"error":"Failed to load model!"} body and no choices; /v1/completions returns a +# nested OpenAI-shaped server_error (code 500) also under HTTP 200. Client model IDs must be the exact FLM registry # tag (e.g. gemma4-it:e4b), never a service alias. Custom tags only resolve # when FLM_CONFIG_PATH points at the user registry that carries them and their # kernels are symlinked into /home/dank/.config/flm/xclbins/. diff --git a/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md b/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md index 60f24e7..21fe384 100644 --- a/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md +++ b/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md @@ -290,6 +290,11 @@ When a request names a tag that is not in the FLM registry, FLM does not return - FLM then loads the 1B model in place of the requested model. - The files now exist in `~/.config/flm/models/Llama-3.2-1B-NPU2` (1.3 GB) on dankclaw. A future unknown-tag request loads the 1B model silently, with no download and no error. +**Correction (live-probed 2026-08-30):** the fallback no longer fires on the +current serve path — an unknown tag now answers HTTP 200 with a flat +`{"error":"Failed to load model!"}` body and nothing is loaded or served; +`/v1/completions` answers a nested `server_error` (code 500) under HTTP 200. + Keep client model IDs as FLM registry tags. Never send a service alias. Optional cleanup: move `~/.config/flm/models/Llama-3.2-1B-NPU2` (an incident artifact) to `.AGENTTRASH` so a stray tag re-downloads loudly instead of serving the wrong model. ### Request compatibility: the `developer` role (found 2026-08-16) diff --git a/serverAppConfigs-docs/LLM/frank-default-sightsound.md b/serverAppConfigs-docs/LLM/frank-default-sightsound.md index ed06ece..72f53b9 100644 --- a/serverAppConfigs-docs/LLM/frank-default-sightsound.md +++ b/serverAppConfigs-docs/LLM/frank-default-sightsound.md @@ -47,9 +47,14 @@ swap is therefore an env-file edit, not a script edit. - The endpoint serves the **whole local registry** (36 tags at first deploy) on one port; `/v1/models` lists them all. Chat completions route per request by the exact FLM registry tag in the `model` field — never a service alias. -- **Unknown model tags do NOT 404**: FLM logs an error and silently serves - `llama3.2:1b` instead. Client configs must carry the exact registry tag or - they will silently get the 1B fallback. +- **Unknown model tags do NOT 404** (live-probed): `/v1/chat/completions` + answers HTTP 200 with a flat error body + `{"error":"Failed to load model!"}` and no `choices`; + `/v1/completions` answers an OpenAI-shaped nested error + (`type: server_error`, `code: 500`), also under HTTP 200. A valid tag + routes that request to that model (response echoes the requested tag); no + server restart is needed to switch between registry tags. Status-code-only + clients silently get an empty response — check for the `error` key. - Multimodal content parts accepted by `/v1/chat/completions`: `{"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}` and `{"type":"input_audio","input_audio":{"data":"","format":"wav"}}`. @@ -100,7 +105,7 @@ fault. Do not set `iommu=off` on this host again; it silently kills the NPU. The pi client model config (`/home/dank/.pi/agent/models.json` on Dankware) registers this endpoint under the Dankclaw provider with the **exact** FLM -registry tag as the model `id` (see above: unknown tags silently fall back to -the 1B), `baseUrl` `http://192.168.0.143:52626/v1`, and a per-request context +registry tag as the model `id` (unknown tags answer a 200-wrapped error, not +a silent 1B fallback), `baseUrl` `http://192.168.0.143:52626/v1`, and a per-request context window equal to the serving ctx-len. Health checks for this service must use `GET /v1/version` (there is no `/health` route).