dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-small: NPU model swap qwen3.6-moe:35b-a3b -> qwythos:9b (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated)

Browse Source 
- FLM_MODEL=qwythos:9b, CTX 262144, turbo+asr kept
- Add FLM_CONFIG_PATH/FLM_XCLBIN_PATH: the custom tag lives in the user
  registry (~/.config/flm/model_list.json) + user xclbins; without them
  flm hits the unknown-tag fallback and serves llama3.2:1b silently
- Note FLM 0.9.45 -> 1.0.1 runtime upgrade (pack declares flm_version 1.0.0)
This commit is contained in:
Frank
2026-08-17 01:22:32 -07:00
parent 2b96043740
commit c85ff6c6fd
1 changed files with 23 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-small.env
+23 -17
View File
@@ -1,27 +1,33 @@
# Tunables for frank-default-small (FastFlowLM NPU endpoint, dankclaw)
# Runtime: FLM 0.9.45 on the XDNA2 NPU (50 TOPS). Model choice, measured
# results, and the ~15 GiB NPU budget are in docs/FastFlowLM NPU Inference.md.
# The NPU allocates from the unified 30 GiB pool shared with
# frank-default-large (the PP head). Keep the resident footprint small
# enough that the head keeps its headroom.
# Runtime: FLM 1.0.1 on the XDNA2 NPU (50 TOPS). Model choice, measured
# results, and the NPU memory budget are in docs/FastFlowLM NPU Inference.md.
# The NPU allocates from the unified pool shared with frank-default-large
# (the PP head). Keep the resident footprint small enough that the head
# keeps its headroom.
#
# 2026-08-16: qwen3.6-moe:35b-a3b chosen after the 4-model bakeoff and the
# user's model swap (was gemma4-it:e4b). Swap FLM_MODEL back to gemma4-it:e4b
# to restore the lighter footprint. gpt-oss:20b is capped at 8k context and
# has no vision.
# 2026-08-17: qwythos:9b (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated,
# Qwen3.5-9B hybrid linear-attention base, Q4NX, 7.76 GB) chosen after the
# user's model swap (was qwen3.6-moe:35b-a3b, which stays installed for
# rollback). FLM upgraded 0.9.45 -> 1.0.1: the pack declares flm_version
# 1.0.0 and 0.9.45 refused to serve it.
FLM_MODEL=qwen3.6-moe:35b-a3b
FLM_MODEL=qwythos:9b
FLM_HOST=0.0.0.0
FLM_PORT=52625
# Power mode: turbo. Measured decode ~13.1 t/s on the 35B-A3B at 262k ctx.
# Power mode: turbo. Smoke decode ~5-7 t/s on the 9B dense at 262k ctx
# (not benchmarked; see docs/FastFlowLM NPU Inference.md).
FLM_PMODE=turbo
# Load the ASR model (Whisper-V3-Turbo-NPU2) alongside the LLM. Exposes
# /v1/audio/transcriptions. Adds ~1 GiB to the resident footprint.
FLM_ASR=1
# Maximum context length. Qwen3.6-35B-A3B native max is 262144 (extensible
# to ~1M via RoPE/YaRN, which FLM does not serve). Verified 2026-08-16:
# loads at -c 262144 with turbo+asr; KV occupancy 0.0687% at ~180 tokens;
# decode 13.1 t/s; PP head unaffected on 7770. The box runs at ~70 MiB free
# RAM while this loads (footprint 24.3 GiB + Whisper) — the user plans a
# BIOS RAM/VRAM reallocation for headroom.
# Maximum context length. Qwythos-9B native max is 262144 (Qwen3.5-9B
# hybrid: 8 of 32 layers are full attention, so the KV cache stays small).
# Verified 2026-08-17: loads at -c 262144 with turbo+asr on FLM 1.0.1;
# KV occupancy 0.007% at 19 tokens.
FLM_CTX_LEN=262144
# Custom-model resolution: the qwythos:9b tag and its xclbin symlinks live
# in the user config, not the system registry. Without these, flm treats
# the tag as unknown and silently serves llama3.2:1b (see the unknown-tag
# fallback in docs/FastFlowLM NPU Inference.md).
FLM_CONFIG_PATH=/home/dank/.config/flm/model_list.json
FLM_XCLBIN_PATH=/home/dank/.config/flm