sightsound: move FLM NPU endpoint port 52626 -> 7771
Env file is the source of truth (unit ExecStart runs the serve script which sources it); serve-script default and the deployer health-gate URL flipped in lockstep so the post-restart health gate polls the new port. Docs, fleet table, and both index bullets updated to match.
This commit is contained in:
1 parent
9b9f0f9051
commit
9ef034e5e6
7 files changed
+10
-10
No files matched your search
@@ -28,7 +28,7 @@ host.
|
|||||||
|--------|---------|---------|-------|------|
|
|--------|---------|---------|-------|------|
|
||||||
| Dankserver | FreeToken v0.1.2 | frank-default-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 |
|
| Dankserver | FreeToken v0.1.2 | frank-default-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 |
|
||||||
| Dankclaw | kingjones30/ROCmFPX llama.cpp (card-pinned fork) | frank-default-large | kingjones777 Qwen3.8-Flash-Next-ROCmFP4-STRIX (113.47 GiB single GGUF, Q4_0_ROCMFP4_STRIX, PLE streams via mmap), 401408 ctx / 2 slots, fa on, no spec decode, reasoning on | 7770 |
|
| Dankclaw | kingjones30/ROCmFPX llama.cpp (card-pinned fork) | frank-default-large | kingjones777 Qwen3.8-Flash-Next-ROCmFP4-STRIX (113.47 GiB single GGUF, Q4_0_ROCMFP4_STRIX, PLE streams via mmap), 401408 ctx / 2 slots, fa on, no spec decode, reasoning on | 7770 |
|
||||||
| Dankclaw | FastFlowLM (FLM) v1.0.1 (`flm serve`, dpkg `fastflowlm`) on the XDNA2 NPU | frank-default-sightsound | Gemma4-E4B-IT-NPU2 pack (vision + audio towers); whole local registry served on one port, per-request routing by exact FLM tag; `iommu=pt` boot arg mandatory (SVA bind needs enabled IOMMU) | 52626 |
|
| Dankclaw | FastFlowLM (FLM) v1.0.1 (`flm serve`, dpkg `fastflowlm`) on the XDNA2 NPU | frank-default-sightsound | Gemma4-E4B-IT-NPU2 pack (vision + audio towers); whole local registry served on one port, per-request routing by exact FLM tag; `iommu=pt` boot arg mandatory (SVA bind needs enabled IOMMU) | 7771 |
|
||||||
|
|
||||||
History: the 2026-08-17 redo retired the pipeline-parallel split and the
|
History: the 2026-08-17 redo retired the pipeline-parallel split and the
|
||||||
FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on
|
FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on
|
||||||
@@ -286,7 +286,7 @@ When you rename a service or move a model to a new host, update
|
|||||||
|
|
||||||
The third inference endpoint on Dankclaw. The `flm serve` runtime (dpkg
|
The third inference endpoint on Dankclaw. The `flm serve` runtime (dpkg
|
||||||
package `fastflowlm`, v1.0.1, binary `/usr/bin/flm`) serves the **whole local
|
package `fastflowlm`, v1.0.1, binary `/usr/bin/flm`) serves the **whole local
|
||||||
model registry** (36 tags at first deploy) on a single port — 52626 — and
|
model registry** (36 tags at first deploy) on a single port — 7771 — and
|
||||||
routes chat completions per request by the exact FLM registry tag in the
|
routes chat completions per request by the exact FLM registry tag in the
|
||||||
`model` field. Unknown tags do NOT 404: `/v1/chat/completions` answers HTTP 200
|
`model` field. Unknown tags do NOT 404: `/v1/chat/completions` answers HTTP 200
|
||||||
with `{"error":"Failed to load <tag> model!"}` and no choices (status-only
|
with `{"error":"Failed to load <tag> model!"}` and no choices (status-only
|
||||||
|
|||||||
@@ -647,7 +647,7 @@ deploy_llm_sightsound() {
|
|||||||
run_privileged systemctl daemon-reload
|
run_privileged systemctl daemon-reload
|
||||||
run_privileged systemctl enable frank-default-sightsound
|
run_privileged systemctl enable frank-default-sightsound
|
||||||
run_privileged systemctl restart frank-default-sightsound
|
run_privileged systemctl restart frank-default-sightsound
|
||||||
wait_for_health frank-default-sightsound http://127.0.0.1:52626/v1/version 300
|
wait_for_health frank-default-sightsound http://127.0.0.1:7771/v1/version 300
|
||||||
}
|
}
|
||||||
|
|
||||||
wait_for_health() {
|
wait_for_health() {
|
||||||
|
|||||||
@@ -9,7 +9,7 @@
|
|||||||
# available there at idle, so a 9.1 GB pack fits with headroom.
|
# available there at idle, so a 9.1 GB pack fits with headroom.
|
||||||
FLM_MODEL=gemma4-it:e4b
|
FLM_MODEL=gemma4-it:e4b
|
||||||
FLM_HOST=0.0.0.0
|
FLM_HOST=0.0.0.0
|
||||||
FLM_PORT=52626
|
FLM_PORT=7771
|
||||||
# turbo = max NPU power mode (xrt-smi configure --pmode turbo at serve start).
|
# turbo = max NPU power mode (xrt-smi configure --pmode turbo at serve start).
|
||||||
# No other tenant uses the NPU on this host, so the top mode costs nothing to
|
# No other tenant uses the NPU on this host, so the top mode costs nothing to
|
||||||
# neighbours; drop to `performance` if iGPU power-budget contention ever shows.
|
# neighbours; drop to `performance` if iGPU power-budget contention ever shows.
|
||||||
|
|||||||
@@ -30,7 +30,7 @@ fi
|
|||||||
: "${FLM_BIN:=/usr/bin/flm}"
|
: "${FLM_BIN:=/usr/bin/flm}"
|
||||||
: "${FLM_MODEL:=gemma4-it:e4b}"
|
: "${FLM_MODEL:=gemma4-it:e4b}"
|
||||||
: "${FLM_HOST:=0.0.0.0}"
|
: "${FLM_HOST:=0.0.0.0}"
|
||||||
: "${FLM_PORT:=52626}"
|
: "${FLM_PORT:=7771}"
|
||||||
: "${FLM_PMODE:=turbo}"
|
: "${FLM_PMODE:=turbo}"
|
||||||
: "${FLM_ASR:=0}"
|
: "${FLM_ASR:=0}"
|
||||||
: "${FLM_CTX_LEN:=131072}"
|
: "${FLM_CTX_LEN:=131072}"
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
---
|
---
|
||||||
type: configuration
|
type: configuration
|
||||||
title: frank-default-sightsound Runbook
|
title: frank-default-sightsound Runbook
|
||||||
description: FastFlowLM (FLM) NPU sight+sound endpoint on Dankclaw — XDNA2 NPU serving the Gemma4 E4B vision+audio pack on port 52626, including the IOMMU/SVA root cause that took it down at first deploy.
|
description: FastFlowLM (FLM) NPU sight+sound endpoint on Dankclaw — XDNA2 NPU serving the Gemma4 E4B vision+audio pack on port 7771, including the IOMMU/SVA root cause that took it down at first deploy.
|
||||||
tags:
|
tags:
|
||||||
- flm
|
- flm
|
||||||
- fastflowlm
|
- fastflowlm
|
||||||
@@ -32,7 +32,7 @@ is text-only and fails the hard modality requirement, so this endpoint trades
|
|||||||
| Service | `frank-default-sightsound.service` (systemd system unit, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`) |
|
| Service | `frank-default-sightsound.service` (systemd system unit, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`) |
|
||||||
| Files | `scripts/frank-default-sightsound.env`, `scripts/serve-frank-default-sightsound.sh`, `scripts/frank-default-sightsound.service` |
|
| Files | `scripts/frank-default-sightsound.env`, `scripts/serve-frank-default-sightsound.sh`, `scripts/frank-default-sightsound.service` |
|
||||||
| Deploy | `scripts/deploy.sh dankclaw --service frank-default-sightsound` (relink + daemon-reload + enable + restart; health gate polls `/v1/version`, 300 s cap — FLM has **no `/health` route**) |
|
| Deploy | `scripts/deploy.sh dankclaw --service frank-default-sightsound` (relink + daemon-reload + enable + restart; health gate polls `/v1/version`, 300 s cap — FLM has **no `/health` route**) |
|
||||||
| Port | `52626` on `0.0.0.0` (52625 left free for ad-hoc `flm serve` experiments) |
|
| Port | `7771` on `0.0.0.0` (52625/52626 left free for ad-hoc `flm serve` experiments) |
|
||||||
| Power mode | `FLM_PMODE=turbo` — no other tenant uses the NPU on this host (the large model runs on the iGPU), so the top power mode costs no neighbour anything; drop to `performance` if iGPU power-budget contention ever shows |
|
| Power mode | `FLM_PMODE=turbo` — no other tenant uses the NPU on this host (the large model runs on the iGPU), so the top power mode costs no neighbour anything; drop to `performance` if iGPU power-budget contention ever shows |
|
||||||
| Context | `FLM_CTX_LEN` = verified max for the E4B pack on this box (native 128K context; the registry's default context length is only the fallback). FLM rounds non-power-of-2 values up. |
|
| Context | `FLM_CTX_LEN` = verified max for the E4B pack on this box (native 128K context; the registry's default context length is only the fallback). FLM rounds non-power-of-2 values up. |
|
||||||
| ASR | `FLM_ASR=0` — the E4B pack answers audio natively through its own audio tower in `/v1/chat/completions`; setting `FLM_ASR=1` additionally loads Whisper-V3-Turbo-NPU2 (+0.62 GB) and exposes `/v1/audio/transcriptions` |
|
| ASR | `FLM_ASR=0` — the E4B pack answers audio natively through its own audio tower in `/v1/chat/completions`; setting `FLM_ASR=1` additionally loads Whisper-V3-Turbo-NPU2 (+0.62 GB) and exposes `/v1/audio/transcriptions` |
|
||||||
@@ -106,6 +106,6 @@ fault. Do not set `iommu=off` on this host again; it silently kills the NPU.
|
|||||||
The pi client model config (`/home/dank/.pi/agent/models.json` on Dankware)
|
The pi client model config (`/home/dank/.pi/agent/models.json` on Dankware)
|
||||||
registers this endpoint under the Dankclaw provider with the **exact** FLM
|
registers this endpoint under the Dankclaw provider with the **exact** FLM
|
||||||
registry tag as the model `id` (unknown tags answer a 200-wrapped error, not
|
registry tag as the model `id` (unknown tags answer a 200-wrapped error, not
|
||||||
a silent 1B fallback), `baseUrl` `http://192.168.0.143:52626/v1`, and a per-request context
|
a silent 1B fallback), `baseUrl` `http://192.168.0.143:7771/v1`, and a per-request context
|
||||||
window equal to the serving ctx-len. Health checks for this service must use
|
window equal to the serving ctx-len. Health checks for this service must use
|
||||||
`GET /v1/version` (there is no `/health` route).
|
`GET /v1/version` (there is no `/health` route).
|
||||||
@@ -80,7 +80,7 @@ ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE]
|
|||||||
- [[Dankclaw System Specs]] — the CIRU ROCmFPX box (frank-default-large)
|
- [[Dankclaw System Specs]] — the CIRU ROCmFPX box (frank-default-large)
|
||||||
- [[frank-default-sightsound]] — ACTIVE FLM NPU sight+sound runbook on
|
- [[frank-default-sightsound]] — ACTIVE FLM NPU sight+sound runbook on
|
||||||
Dankclaw: Gemma4 E4B vision+audio pack on the XDNA2 NPU (FLM v1.0.1, port
|
Dankclaw: Gemma4 E4B vision+audio pack on the XDNA2 NPU (FLM v1.0.1, port
|
||||||
52626, whole-registry serving, per-request model routing), incl. the
|
7771, whole-registry serving, per-request model routing), incl. the
|
||||||
IOMMU/SVA root cause and the `iommu=pt` fix behind its first-deploy crash
|
IOMMU/SVA root cause and the `iommu=pt` fix behind its first-deploy crash
|
||||||
|
|
||||||
## CIRU Reference (Dankclaw)
|
## CIRU Reference (Dankclaw)
|
||||||
|
|||||||
@@ -128,7 +128,7 @@ The LLM inference fleet on the two hosts. Full index:
|
|||||||
- [[Dankclaw System Specs]] — the CIRU ROCmFPX box hardware
|
- [[Dankclaw System Specs]] — the CIRU ROCmFPX box hardware
|
||||||
- [[Dankclaw Improvements]] — host-level changes on the CIRU box
|
- [[Dankclaw Improvements]] — host-level changes on the CIRU box
|
||||||
- [[frank-default-sightsound]] — ACTIVE FLM NPU sight+sound runbook on Dankclaw:
|
- [[frank-default-sightsound]] — ACTIVE FLM NPU sight+sound runbook on Dankclaw:
|
||||||
Gemma4 E4B vision+audio pack on the XDNA2 NPU, port 52626; whole local
|
Gemma4 E4B vision+audio pack on the XDNA2 NPU, port 7771; whole local
|
||||||
registry served on one port with per-request model routing; includes the
|
registry served on one port with per-request model routing; includes the
|
||||||
IOMMU/SVA root cause and the `iommu=pt` fix behind its first-deploy crash
|
IOMMU/SVA root cause and the `iommu=pt` fix behind its first-deploy crash
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user