dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
8775e64ed06a4ccd62a45b10b458d993ee46ddad
serverAppConfigs/serverAppConfigs-docs/LLM/frank-npu.md
T
Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts 
Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
2026-10-19 05:00:43 -07:00

9.1 KiB
Raw Blame History

type, title, description, tags, timestamp
type title description tags timestamp
reference frank-npu (retired) — FLM NPU endpoint on Dankclaw The retired NPU endpoint: what it served, why it died with the IOMMU decision, and where every dependent route went.
flm
fastflowlm
xdna2
npu
dankclaw
frank-npu
retired
retired-endpoint
2026-09-11

frank-npu — the retired FLM NPU endpoint

RETIRED 2026-09-11 by operator decision — and still retired: there is no NPU endpoint service on Dankclaw today. The IOMMU configuration it required has since been reversed twice — see dankclaw-iommu-translation-off: the live 2026-10-18 posture is cat /proc/cmdline → … ro quiet splash iommu=pt … (no amd_iommu token), amdxdna is bound and /dev/accel/accel0 exists, so the hardware is reachable again — revival is tracked as Gitea issue #3. The service unit, env file and serve script are staged in serverAppConfigs/.AGENTTRASH/ (suffixed .flm-npu), and the host unit is still masked (systemctl is-enabled frank-npu → masked, read 2026-10-18). Port 7771 belongs to frank-zealot today.

The third inference endpoint on Dankclaw: a FastFlowLM (FLM) v1.0.1 server (/usr/bin/flm, dpkg fastflowlm) on the XDNA2 NPU (50 TOPS), serving the whole local model registry on port 7771 and routing per request by the exact FLM registry tag. Its one capability the iGPU fleet lacked was sight — and briefly sound: the Gemma4 E4B IT NPU2 pack carries both a vision and an audio tower, which is why the service ran that pack rather than the uncensored-but-text-only alternative.

This page is kept as the retirement record. The operational detail below was live-verified while the service ran and remains accurate about FLM's shape.

Why it is retired

The replacement large-model runtime is worth 13–16 % of prefill if the kernel boots with amd_iommu=off instead of iommu=pt, measured by the upstream project on this exact silicon. That single flag is the whole decision:

  • With IOMMU off there is no PASID/SVA state, and the XDNA2 driver cannot hand out /dev/accel/accel0 — every open fails with err=-19 ("No such device"). That is precisely the failure the endpoint was rescued from on 2026-08-30, recorded in the next section.
  • iommu=pt was recorded here as "not a middle ground", on the theory that the passthrough domain still costs the vendor's prefill number. That claim is dead: the host flipped to iommu=pt on 2026-10-18 and the local A/B on byte-identical traffic measured the cost at nothing (+2.0 % cold prefill, 1,616 t/s at 59k tokens). Identity mode does keep a 1:1 page walk plus ATS/PRI/PASID/GCR3 armed for the iGPU — worth 2–4 % on this silicon class, and AMD PerfOpt (Linux 7.4, merged 2026-09-24) removes it outright by enforcing only the IR/IW permission bits for the privileged iGPU — but nothing waits on that now.
  • The precondition is satisfied; the endpoint is still not deployed. amdxdna binds 0000:c6:00.1 (driver 0.7.0, firmware amdnpu/17f0_11/npu_7.sbin), the NPU sits in IOMMU group 26 identity with PASID width 10, /dev/accel/accel0 is crw-rw---- root:render 261,0 and dank is in render — so a revival quadlet needs AddDevice=/dev/accel/accel0 + GroupAdd=keep-groups, the ComfyUI /dev/kfd pattern. Re-read live 2026-10-18: lspci -k -s c6:00.1 → Kernel driver in use: amdxdna; journalctl -k -b → Initialized amdxdna_accel_driver 0.7.0 for 0000:c6:00.1 and Load firmware amdnpu/17f0_11/npu_7.sbin; /sys/kernel/iommu_groups/26/type → identity; ls -l /dev/accel/accel0 → crw-rw---- 1 root render 261, 0 (created 2026-10-18 01:25); id -nG dank contains render. Kernel versions to avoid: 6.18–6.18.7 disables SVA outright (unverified 2026-10-18 — upstream reports place the amdxdna/Strix-Halo breakage at 6.18.4+ and fixed by 6.19.2, not at a clean 6.18–6.18.7 window). Full census in dankclaw-iommu-translation-off § 2026-10-18 execution.
  • The operator took the vendor's prefill figure on the September evidence, so the NPU endpoint went — and it stays retired because nobody has asked for a model on it, not because the hardware is unreachable.

Consequences accepted at retirement: audio input left the fleet with no replacement (halogen reads images only, no audio or video), and the surviving image path is halogen's vision tower on frank-dragoon.

Where its routes went

consumer setting now
Hermes auxiliary.vision dankclaw-large / frank-dragoon (halogen vision tower) (unverified 2026-10-18: /opt/hermes is gone from Dankclaw, so the live config.yaml could not be read)
Hermes auxiliary.title_generation dankserver-small / frank-zealot (unverified 2026-10-18: same)
harness (~/.omp/agent/config.yml) modelRoles.vision moved again: llamacpp (dankclaw-3090)/frank-zealot:xhigh (config.yml:6)
harness modelRoles.tiny llamacpp (dankclaw-3090)/frank-zealot:off (config.yml:10)
harness modelRoles.task llamacpp (dankclaw-3090)/frank-zealot:off (config.yml:11)

The flm (dankclaw-npu) provider block was removed from the harness model config, and the dankclaw-npu provider from Hermes's live config.yaml.

What it was

Field Value
Runtime FLM v1.0.1 (/usr/bin/flm, dpkg fastflowlm; dpkg -l fastflowlm on 2026-10-18 still reports fastflowlm 1.0.1) on the XDNA2 NPU
Service frank-npu.service (systemd system unit, User=dank, Restart=on-failure, RestartSec=10, LimitMEMLOCK=infinity)
Files (archived) scripts/frank-npu.env, scripts/serve-frank-npu.sh, scripts/frank-npu.service
Port 7771 on 0.0.0.0 (that port is frank-zealot's today)
Power mode FLM_PMODE=turbo — nothing else used the NPU, so the top power mode cost no neighbour anything
Context FLM_CTX_LEN=131072 (128K, the verified maximum on this box)
Vision/audio FLM_ASR=0; the E4B pack answers images through its own vision tower, and FLM_ASR=1 would have added Whisper-V3-Turbo-NPU2 (+0.62 GB) and /v1/audio/transcriptions

The serve script exported FLM_CONFIG_PATH=/home/dank/.config/flm/model_list.json and FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins, with kernels symlinked from /opt/fastflowlm/share/flm/xclbins/ into the per-pack directories, so a tag swap was an env-file edit rather than a script edit.

API contract (as it behaved)

  • The endpoint served the whole local registry (36 tags at first deploy) on one port; /v1/models listed them all. Chat completions routed per request by the exact registry tag in the model field — never a service alias.
  • Unknown tags did not 404: /v1/chat/completions answered HTTP 200 with a flat error body {"error":"Failed to load <tag> model!"} and no choices. Status-code-only clients silently got an empty response.
  • Health was GET /v1/version; FLM has no /health route.
  • The NPU is a single-lane device — generations serialized FIFO. It was a background/triage endpoint, not interactive multi-user chat.

The IOMMU/SVA root cause (first deploy crashed, fixed 2026-08-30)

First deploy crash-looped every 10 s with Error: Open /dev/accel/accel0 failed (err=-19): No such device. The chain, all live-verified on the host:

  1. amdxdna_drm_open failed SVA bind with ret -19 for every client — the XDNA2 driver cannot hand out the accel device without PASID/SVA.
  2. SVA requires the AMD IOMMU to be enabled, and the kernel cmdline then carried iommu=off. With IOMMU off there are no IOMMU groups and no PASID state; PASIDCtl: Enable- was observed on 0000:c5:00.1.
  3. The fix was iommu=pt: passthrough keeps the iGPU default domain identity-mapped (no DMA-translation overhead for frank-dragoon) while enabling the IOMMU hardware and PASID that the driver's SVA bind needs.
  4. Post-boot verification: /sys/kernel/iommu_groups/ populated (30 groups), zero SVA bind device failed entries since boot, accel0 recreated at boot, service active with NRestarts=0.

The same root cause explained the older amdgpu 0000:c4:00.0: no vmid pasid mapping supported kernel line — a consequence of IOMMU-off, not a GPU fault.

The standing rule this produced — "never set iommu=off on this host again" — was superseded twice: amd_iommu=off was the intentional posture from 2026-10-16 to 2026-10-18, and the host has since flipped back to iommu=pt (live cat /proc/cmdline, 2026-10-18: … ro quiet splash iommu=pt …). See dankclaw-iommu-translation-off for the current posture and how to revert it.

Client wiring (historical)

The pi/harness model config registered this endpoint with the exact FLM registry tag as the model id and baseUrl http://192.168.0.143:7771/v1. Both that provider block and Hermes's dankclaw-npu provider were removed at retirement; the routes are mapped in the table above.

Citations

Reference in New Issue View Git Blame Copy Permalink