dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-small: swap to qwen3.6-moe:35b-a3b at 262144 ctx (native max), keep turbo+asr

Browse Source
This commit is contained in:
Frank committed 2026-08-16 14:48:33 -07:00
1 parent 2f1999b2e9
commit 50c55ee220
1 file changed
+14 -14
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-small.env
+14 -14
View File
@@ -5,23 +5,23 @@
# frank-default-large (the PP head). Keep the resident footprint small # frank-default-large (the PP head). Keep the resident footprint small
# enough that the head keeps its headroom. # enough that the head keeps its headroom.
# #
# 2026-08-16: gemma4-it:e4b chosen as the default after the 4-model bakeoff # 2026-08-16: qwen3.6-moe:35b-a3b chosen after the 4-model bakeoff and the
# (12.2 t/s decode, 582 t/s 8k prefill, 4/4 vision, 32k ctx, small # user's model swap (was gemma4-it:e4b). Swap FLM_MODEL back to gemma4-it:e4b
# footprint). Swap FLM_MODEL for qwen3.5:9b, gpt-oss:20b, or # to restore the lighter footprint. gpt-oss:20b is capped at 8k context and
# qwen3.6-moe:35b-a3b to change the served model. gpt-oss:20b is capped at # has no vision.
# 8k context and has no vision; 35b-a3b needs ~29 GiB resident (do not run
# alongside the PP head at full load).
FLM_MODEL=gemma4-it:e4b FLM_MODEL=qwen3.6-moe:35b-a3b
FLM_HOST=0.0.0.0 FLM_HOST=0.0.0.0
FLM_PORT=52625 FLM_PORT=52625
# Power mode: turbo. Measured decode is bandwidth-bound (turbo ~= performance # Power mode: turbo. Measured decode ~13.1 t/s on the 35B-A3B at 262k ctx.
# for this model), but the user requested turbo; it may help prefill.
FLM_PMODE=turbo FLM_PMODE=turbo
# Load the ASR model (Whisper-V3-Turbo-NPU2) alongside Gemma4. Exposes # Load the ASR model (Whisper-V3-Turbo-NPU2) alongside the LLM. Exposes
# /v1/audio/transcriptions. Adds ~1 GiB to the resident footprint. # /v1/audio/transcriptions. Adds ~1 GiB to the resident footprint.
FLM_ASR=1 FLM_ASR=1
# Maximum context length. Gemma4 natively supports 128K; the NPU budget # Maximum context length. Qwen3.6-35B-A3B native max is 262144 (extensible
# (~15 GiB minus the 9.1 GiB footprint) fits 131072. Verified 2026-08-16: # to ~1M via RoPE/YaRN, which FLM does not serve). Verified 2026-08-16:
# loads, KV cache allocates to ~115k, decode ~12 t/s, 13 GiB headroom. # loads at -c 262144 with turbo+asr; KV occupancy 0.0687% at ~180 tokens;
FLM_CTX_LEN=131072 # decode 13.1 t/s; PP head unaffected on 7770. The box runs at ~70 MiB free
# RAM while this loads (footprint 24.3 GiB + Whisper) — the user plans a
# BIOS RAM/VRAM reallocation for headroom.
FLM_CTX_LEN=262144