Frank
301914b1c0
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b .hgn 115.55 GiB + quality overlay). Upstream measures this engine at roughly 4x the prefill throughput of the llama.cpp family on the same gfx1151 silicon, and prefill dominates any prompt with real context. Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/), engine on container loopback :8730 (never published), OpenAI front-end :8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB, conservative for a shared 121 GiB host), 4 slots, host reserve 24, vision tower on, weights pinned, all other flags at vendor defaults. Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable /home/dank/models/halogen. The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/ rather than deleted; the STRIX payload and the ROCmFPX build stay on disk as the rollback target. deploy_llm_large becomes a user-quadlet handler: it retires the old ROOT unit (moving its checkout symlink into the home trash, never rm) and gates the first start on the download rather than reporting a failure for a fetch that is working exactly as designed. frank-npu is retired in the same change. The kernel cmdline moves from iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured, and that takes the XDNA2 NPU with it: without PASID/SVA the driver cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are repointed (Hermes vision -> frank-large, Hermes title generation -> frank-small, harness vision -> frank-large, tiny -> frank-small, task -> frank-large:medium) and its provider blocks are removed from both the harness config and Hermes's live config. Audio input leaves the fleet with no replacement; image input survives on halogen's vision tower. Docs: new halogen-flash-server runbook, new IOMMU decision record, and frank-npu rewritten as a retirement record. AGENTS.md topology, runtime sections, benchmarking policy (frank-large is a user unit now) and the harness model-config section are corrected.
24 lines
1.3 KiB
Bash
24 lines
1.3 KiB
Bash
# Tunables for frank-npu (FastFlowLM NPU endpoint, Dankclaw)
|
|
# Runtime: FLM 1.0.1 (/usr/bin/flm) on the XDNA2 NPU (50 TOPS). Boot tag is
|
|
# nuextract3:4b (NuExtract3-4B-NPU2, Qwen3.5-4B skeleton, structured-extraction
|
|
# model); serve routes every request by the exact FLM registry tag in the
|
|
# request's model field, so any registered tag (gemma4-it:e4b, qwythos:9b, ...)
|
|
# is reachable on this port regardless of the boot tag. The NPU is a
|
|
# single-lane device: generations serialize FIFO, and the first request after a
|
|
# model swap pays the ~6-11 s load cost. The NPU draws from the unified RAM
|
|
# pool shared with frank-large (iGPU).
|
|
FLM_MODEL=nuextract3:4b
|
|
FLM_HOST=0.0.0.0
|
|
FLM_PORT=7771
|
|
# turbo = max NPU power mode (xrt-smi configure --pmode turbo at serve start).
|
|
# No other tenant uses the NPU on this host, so the top mode costs nothing to
|
|
# neighbours; drop to `performance` if iGPU power-budget contention ever shows.
|
|
FLM_PMODE=turbo
|
|
# FLM_ASR=1 additionally loads Whisper-V3-Turbo-NPU2 (+0.62 GB) and exposes
|
|
# /v1/audio/transcriptions. NuExtract3 answers text-only; audio stays off.
|
|
FLM_ASR=0
|
|
# -1 = no --ctx-len flag: every pack loads with its own trained context
|
|
# (nuextract3:4b -> 32768, gemma4-it:e4b -> 131072). A global override would
|
|
# over-allocate KV for the small packs.
|
|
FLM_CTX_LEN=-1
|