Frank
df0c669815
Boot tag flips from qwen3.8-distilled-heretic:9b to nuextract3:4b (verified live: per-request routing already serves every registered tag; the boot tag only decides which model is resident at start, so the first request after a swap pays the load cost). FLM_CTX_LEN=-1 drops the global --ctx-len override so each pack loads with its own trained context (nuextract3 32768, gemma4 131072).
24 lines
1.3 KiB
Bash
24 lines
1.3 KiB
Bash
# Tunables for frank-npu (FastFlowLM NPU endpoint, Dankclaw)
|
|
# Runtime: FLM 1.0.1 (/usr/bin/flm) on the XDNA2 NPU (50 TOPS). Boot tag is
|
|
# nuextract3:4b (NuExtract3-4B-NPU2, Qwen3.5-4B skeleton, structured-extraction
|
|
# model); serve routes every request by the exact FLM registry tag in the
|
|
# request's model field, so any registered tag (gemma4-it:e4b, qwythos:9b, ...)
|
|
# is reachable on this port regardless of the boot tag. The NPU is a
|
|
# single-lane device: generations serialize FIFO, and the first request after a
|
|
# model swap pays the ~6-11 s load cost. The NPU draws from the unified RAM
|
|
# pool shared with frank-large (iGPU).
|
|
FLM_MODEL=nuextract3:4b
|
|
FLM_HOST=0.0.0.0
|
|
FLM_PORT=7771
|
|
# turbo = max NPU power mode (xrt-smi configure --pmode turbo at serve start).
|
|
# No other tenant uses the NPU on this host, so the top mode costs nothing to
|
|
# neighbours; drop to `performance` if iGPU power-budget contention ever shows.
|
|
FLM_PMODE=turbo
|
|
# FLM_ASR=1 additionally loads Whisper-V3-Turbo-NPU2 (+0.62 GB) and exposes
|
|
# /v1/audio/transcriptions. NuExtract3 answers text-only; audio stays off.
|
|
FLM_ASR=0
|
|
# -1 = no --ctx-len flag: every pack loads with its own trained context
|
|
# (nuextract3:4b -> 32768, gemma4-it:e4b -> 131072). A global override would
|
|
# over-allocate KV for the small packs.
|
|
FLM_CTX_LEN=-1
|