docs(vault): halogen has an NPU lane on the pin we already run, and it is greedy-only

- frank-dragoon: measured the response_format contract. json_schema and json_object answer schema-valid JSON at temperature 0 and draw a 400 at any sampled temperature, and because the sampler default is server-side an absent temperature is sampled. The page claimed "response_format is unsupported", which sends the next agent past a working feature. - frank-dragoon: new "The NPU lane" section. The pinned 0.16.2 image already ships the five-program AIE catalog including qwen3-embedding-0.6b, so /v1/embeddings is one env var plus the accel device, the distro-XRT mount and the fabric-clock hold; embeddinggemma-2 needs 0.18.3+, the stock-kernel-driver memory-budget fix 0.18.8+. Host census recorded: no /opt/xilinx/xrt, distro XRT 1:2.21.75 with libxrt_driver_xdna, clocks unheld. - Open Notebook: scriptwriter probe verdict — greedy-only, and open-notebook's model record is {credential, name, provider, type} with no per-row config, so it cannot ask for greedy; frank-archon stays the writer. The embedding gap now has a named door instead of "build a new llama.cpp slot". - Frank TTS: "the NPU is not a door at all" was too broad. It is not a door for the TTS family; halogen's precompiled catalog is a door on the same device. - System Specs: NPU runtime row corrected to the distro XRT actually installed, and the stale "frank-zealot serves 27B" line retired in favour of [[frank-zealot]] as sole owner. Tests: ./tests/run.sh, 455 passed.
2026-10-19 16:40:48 -07:00
parent 69739237c8
commit 6215f1a6c8
4 changed files with 67 additions and 10 deletions
@@ -121,10 +121,12 @@ touching either tenant, because a change to one is a change to both. Co-tenancy
 `frank-dragoon` live on the card (its first token and decode rates unchanged with a synthesis
 in flight); both gate recipes are §5 of `report/tts-npu-spike-verdict-2026-10-19.md`.

-**The Strix Halo NPU is not a door at all**: XDNA2 executes only precompiled AIE partition
-programs, and AMD's own Linux runtime opens `/dev/accel/accel0`, succeeds at the ioctls, then
-declines this model family's graph without emitting a single log line. The measured record and
-the model-side contract live in `report/` at the repo root
+**The NPU is a door only for a model the vendor has already compiled for it.** XDNA2 runs nothing but
+precompiled AIE partition programs, and AMD's own Linux runtime opens `/dev/accel/accel0`, succeeds at the
+ioctls, then declines this model family's graph without emitting a single log line. That is why TTS stays on
+the GPU — and it is not a general veto on the device: halogen ships its own precompiled catalog (embeddings,
+decisions, rerankers) that does run there, which is what [[frank-dragoon]]'s NPU-lane section is about. The
+measured record and the model-side contract live in `report/` at the repo root
 (`tts-npu-spike-verdict-2026-10-19.md`).

 ## Files
@@ -78,9 +78,11 @@ Order that works over REST: credentials → model rows (`POST /api/models` with
 - `192.168.0.143:7771` (frank-zealot, a host-netns listener) and `192.168.0.143:7770` (frank-dragoon, a **root**-published port) both refuse the connection; `host.containers.internal:7771` answers. Full census: [[Container Network Namespaces]].
 - The app refuses to **store** that one working address: its URL guard rejects any base URL that resolves inside link-local `169.254.0.0/16` (`open_notebook/utils/url_validation.py`, "Link-local addresses are not allowed (metadata endpoints)"), unconditionally — no env var touches it, and `AddHost=` cannot dodge it because the check runs on the resolved address.

-`http://192.168.0.235:7770/v1` — another machine, plain RFC1918 — is the only fleet LLM address that passes both tests. Cost: frank-archon runs `--max-num-seqs 2` and the operator session itself lives there, so a render is a slow-down for whoever else is talking to that endpoint, not a stall; `ESPERANTO_LLM_TIMEOUT=180` covers an outline or transcript call comfortably. The ways out, if the contention ever matters: move frank-zealot's listener into a rootless container (container DNS or a rootless-published port then become dialable), or measure halogen's structured-output behaviour on frank-dragoon and use it ([[frank-dragoon]] — currently UNVERIFIED for that call shape, so never the fallback by assumption).
+`http://192.168.0.235:7770/v1` — another machine, plain RFC1918 — is the only fleet LLM address that passes both tests. Cost: frank-archon runs `--max-num-seqs 2` and the operator session itself lives there, so a render is a slow-down for whoever else is talking to that endpoint, not a stall; `ESPERANTO_LLM_TIMEOUT=180` covers an outline or transcript call comfortably.

-**No embedding model exists anywhere in the fleet, and saving a note is where that shows up.** Every note save submits an `embed_note` command that fails in the worker with `No embedding model configured. Please configure one in the Models section.` The note itself is stored and readable; semantic search, source ingestion and notebook-context questions stay dead until something serves `/v1/embeddings`. Podcast generation from `content` does not need embeddings, which is why renders work today. Closing the gap needs a new endpoint (a llama.cpp slot with `--embeddings` and a small embedding model), not a change here.
+**frank-dragoon can write transcripts, but only greedily — and open-notebook cannot ask for greedy.** Measured 2026-10-19 against the live endpoint: `response_format` as `json_schema` or `json_object` returns schema-valid JSON at `temperature: 0` and a **400** at any sampled temperature — halogen decodes greedy under a schema, and because its sampler default is server-side, omitting `temperature` is sampled, not greedy. That 400 is the trap: esperanto's `is_json_schema_unsupported_error` matches the words *not implemented* in it, so `_SchemaFallbackModel` in `podcast_creator/nodes.py` reads it as "this endpoint has no schema support", downgrades to `json_object`, gets the same 400, and **fails the run** — esperanto's own rescue (drop `response_format` entirely) fires only on the different message `'response_format.type' must be 'json_schema'`. The full law is on [[frank-dragoon]]. And the fix is not reachable from here: a model record is `{credential, name, provider, type}` and a model response adds only timestamps (`GET /openapi.json`, `ModelCreate`/`ModelResponse`) — **there is no per-row config field**, no way to attach `temperature: 0` to one row from the UI or the API. Making `frank-dragoon` the writer would therefore need something in the path that rewrites request bodies, which is a heavier tenant than the contention it removes. **frank-archon stays the scriptwriter.** Reachability would be a separate problem anyway: `frank-dragoon` is not on `dankclaw_default` (the serve script passes no `--network`), so nothing on this network has a name for it, and the one address that does answer — `host.containers.internal:7770` — is the link-local the guard refuses to store ([[Container Network Namespaces]]).
+
+**No embedding model is deployed, and saving a note is where that shows up.** Every note save submits an `embed_note` command that fails in the worker with `No embedding model configured. Please configure one in the Models section.` The note itself is stored and readable; semantic search, source ingestion and notebook-context questions stay dead until something serves `/v1/embeddings`. Podcast generation from `content` does not need embeddings, which is why renders work today. The cheapest door is **halogen's NPU lane, not a new endpoint**: `qwen3-embedding-0.6b` is already in the catalog of the image `frank-dragoon` runs and answers `/v1/embeddings` on the same port (1024 unit-length floats; the instruct prefix goes in the text, and `check_embedding_ctx_length=False` for LangChain). What enabling it costs — the accel device, the host XRT mount, and the fabric-clock hold that GPU+NPU co-run requires — is on [[frank-dragoon]]; after that, the embedding row still has to clear the reachability rule above.

 **Secrets never go in a command line.** The API's bearer token *is* `OPEN_NOTEBOOK_PASSWORD`, and a `-H "Authorization: Bearer …"` argument is world-readable in `/proc/<pid>/cmdline`; put it in a mode-600 curl config and pass `-K`:

@@ -43,10 +43,12 @@ Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp
 | Device | /dev/accel/accel0 |
 | Driver | amdxdna 0.7.0, in-tree — bound since 2026-10-18 (`lspci -k -s c6:00.1` → `Kernel driver in use: amdxdna`; journal 2026-10-18 01:25:14 `Initialized amdxdna_accel_driver 0.7.0 for 0000:c6:00.1 on minor 0`) |
 | Firmware | 1.1.2.65 (`/usr/lib/firmware/amdnpu/17f0_11/npu_7.sbin` → `npu.sbin.1.1.2.65.zst`, loaded at boot 2026-10-18 01:25:13) |
-| Runtime | None installed. FastFlowLM 0.9.45 + XRT (installed 2026-08-16) was **retired 2026-09-11** with the `amd_iommu=off` decision — see [[dankclaw-iommu-translation-off]]; the package remains at `/home/dank/GitPackages/fastflowlm_0.9.45_ubuntu26.04_amd64.deb` |
-| Status | Not in service — no NPU endpoint is deployed. The device itself is usable since the 2026-10-18 `iommu=pt` boot (`/dev/accel/accel0` present, `crw-rw---- root:render 261,0`); Gitea issue #3 tracks enablement. See  |
+| Runtime | Distro XRT only: `libxrt2` + `libxrt-npu2` 1:2.21.75 supply `libxrt_coreutil.so.2`, `libxrt_core.so.2` and the NPU plugin `libxrt_driver_xdna.so.2` under `/usr/lib/x86_64-linux-gnu`; **no `/opt/xilinx/xrt`** (census 2026-10-19), no Ryzen AI Python stack. FastFlowLM 0.9.45 + AMD's XRT (installed 2026-08-16) was **retired 2026-09-11** with the `amd_iommu=off` decision — see [[dankclaw-iommu-translation-off]]; the package remains at `/home/dank/GitPackages/fastflowlm_0.9.45_ubuntu26.04_amd64.deb` |
+| Status | Not in service — no NPU consumer is deployed. The device itself is usable since the 2026-10-18 `iommu=pt` boot (`/dev/accel/accel0` present, `crw-rw---- root:render 261,0`); Gitea issue #3 tracks enablement. The only runtime here that ships precompiled programs this device can execute is halogen's — see [[frank-dragoon]] §The NPU lane |

-The NPU endpoint is retired (2026-09-11; ). The `frank-zealot` name now serves **Ternary-Bonsai-2-27B-PQ2_0** on the RTX 3090 via the llama.cpp PrismML fork (see ); the Gemma4-12B QAT rung it once pointed at is retired history (, 2026-09-17 → 10-01).
+This section is device inventory, not a payload record. The name `frank-zealot` served the FLM NPU endpoint until
+2026-09-11 and has served Bonsai-2 on the RTX 3090 through the llama.cpp PrismML fork since; the checkpoint, the
+shape and every era are on [[frank-zealot]], and any figure restated here is the next thing to go stale.

 ### ROCm

@@ -197,7 +197,16 @@ created it — earlier topologies in .
  `chat_template_kwargs.preserve_thinking`, which decided whether `<think>` blocks stayed in recorded
  assistant messages. Sampler defaults 1.0/0.95/20 are server-side, so an absent temperature is NOT
  greedy (halogen 0.1.4 and w4b decoded greedily); unknown sampling fields are a 400, out-of-range
-  values are refused not clamped, and `response_format` is unsupported.
+  values are refused not clamped.
+- **`response_format` works, but only greedy** (measured 2026-10-19 on this pin): `json_schema` and `json_object`
+  both answer schema-valid JSON at `temperature: 0`; at any sampled temperature both draw `400 "response_format
+  json_schema / json_object decodes greedy in this release: send temperature 0 (or omit it). Sampling under a schema
+  is not implemented yet"`. **Omitting `temperature` does not save you** — the sampler default is server-side 1.0, so
+  "absent" means sampled. Clients that pattern-match that string for "endpoint cannot do schemas" (esperanto's
+  `is_json_schema_unsupported_error` fires on the words *not implemented*) downgrade to `json_object`, draw the same
+  400, and fail outright; a structured-output consumer MUST send `temperature: 0`. Escape hatch that keeps sampling:
+  ask for JSON in the prompt and send no `response_format` — the model returns parseable JSON anyway, and the
+  tolerant client-side extractor handles a prose wrapper.
 - Over-ceiling prompt → 400 naming the exact overage, rejected by preflight in <1 s, so probing the
  ceiling is free; over-cap `max_tokens` → 400 naming the cap; `max_tokens`, `max_completion_tokens` and
  `max_output_tokens` all work and must agree.
@@ -210,6 +219,48 @@ created it — earlier topologies in .
 - Deploy waits for :7770 to be free: a rootless container binds it at CREATE and exits 126, and a
  leftover Quadlet beside the root unit guarantees the race.

+## The NPU lane (present in the image, not enabled)
+
+XDNA2 executes only vendor-precompiled AIE partition programs — that is the wall the TTS attempt died at
+([[Frank TTS]]) — and flash-server ships those programs. `HALOGEN_NPU_MODELS=<names>` starts a second engine **in the
+same container behind the same port**, and a request picks one by its `model`; leave the variable out and the server
+is exactly what runs today. The pinned 0.16.2 image already carries the five-program catalog
+(`/opt/halogen/npu/models.txt`: `decider-0.8b`, `qwen3-embedding-0.6b`, `qwen3-reranker-0.6b`, `qwen3guard-gen-0.6b`,
+`qwen3.5-2b`) and `docs/NPU.md` at tag `v0.16.2` documents it, so **`/v1/embeddings` is one env var and three mounts
+away — no runtime bump needed.** Wider catalog: `qwen3-embedding-4b` and `decider-4b` from 0.17.4, `embeddinggemma-2`
+(text ≤ 8,192 tokens, pictures and audio in one vector space) and `wemm-embedding-4b` from 0.18.3.
+
+Embedding contract on that lane: `POST /v1/embeddings`, `model: "qwen3-embedding-0.6b"`, `input` a list of **texts**
+(not token ids), 1024 unit-length floats back, the Qwen instruct prefix goes inside the string
+(`Instruct: …\nQuery:…`), `base64` accepted, and LangChain's `OpenAIEmbeddings` needs
+`check_embedding_ctx_length=False`.
+
+Four prerequisites the trio does not satisfy today, from the vendor's own host list — and this box is one of the two
+configurations they test (Ubuntu 26.04's XRT 2.21 `libxrt2`+`libxrt-npu2`, the kernel's `amdxdna`, `iommu=pt`):
+
+1. `--device /dev/accel/accel0` plus `--group-add keep-groups` (the pattern ComfyUI uses for `/dev/kfd`; `dank` is
+   already in `render`).
+2. The host's XRT reachable inside: there is **no `/opt/xilinx/xrt`** here — distro packages put the three files the
+   container needs (`libxrt_coreutil.so.2`, `libxrt_core.so.2`, `libxrt_driver_xdna.so.2`) in
+   `/usr/lib/x86_64-linux-gnu`, where all three exist.
+3. **The GPU fabric clock held at its top level, or the engine refuses to start.** GPU and NPU work share the die:
+   while `pp_dpm_fclk` is changing speed, concurrent GPU work can hang the machine and corrupt NPU results. The
+   engine holds the clock itself only when it runs as root with `/sys` mounted; `frank-dragoon` is a **rootless**
+   `podman run` under `User=dank`, so the path is the vendor's root host unit (`deploy/host/halogen-fabric-clock`) —
+   which is exactly the `pp_dpm_fclk` level-7 pin deferred in [[dankclaw-iommu-translation-off]] with the words "it
+   goes in when an NPU endpoint actually exists". Held: about a watt. Unheld: the hang. Measured 2026-10-19 nothing
+   is held (fabric DPM lists 400–2000 MHz with no forced level, graphics clock sitting at 857 of 2900 MHz).
+4. `HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next` on a boot with a **writable** `/models/npu` — the payload
+   mount is `:ro` by law, so the programs land via a throwaway writable run or a dedicated read-write bind for that
+   one subdirectory; the vendor notes a read-only volume works but re-runs the conversion at every start.
+
+Price: the programs and their weights sit in host RAM beside the 62.9 GiB of locked weights ("a few GB in all" for
+the default set; ~1.1 GiB for `embeddinggemma-2`), and **the Flash model decodes slower while the NPU works** — same
+memory bus, same power budget. Under this page's RAM law the question is pool versus n-gram table, so read the boot
+ledger after the first NPU boot instead of modelling it. If the lane is opened, prefer **0.18.8 or newer**: that
+release budgets NPU memory before asking for it, fixing kernel errors from several NPU models under a concurrent mix
+**on the stock kernel driver** — the driver this box runs.
+
 ## Files

 - `scripts/frank-dragoon.env` · `scripts/serve-frank-dragoon.sh` (gates, RAM pin-gate wait, foreground