frank-large: swap halogen Quadlet for ik_llama.cpp (3090 + Strix Halo CPU path)

scripts/frank-large.env

@@ -0,0 +1,95 @@
+# Tunables for frank-large (mradermacher i1-Q4_K_M of
+# orcarouter/Qwen3.8-Flash-Next-Uncensored on ik_llama.cpp, Dankclaw,
+# RTX 3090 CUDA + Strix Halo CPU path, since 2026-09-15).
+#
+# Runtime: ik_llama.cpp master d85a1d7 (2026-09-15) with full qwen4exp
+# support (upstream PRs #2365/#2369/#2373 merged). Build recipe (CUDA for
+# the 3090 at sm_86, Zen5 AVX-512 IQK kernels for the CPU expert path):
+#   cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
+#     -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 \
+#     -DGGML_AVX512=ON -DGGML_AVX512_VBMI=ON -DGGML_AVX512_VNNI=ON \
+#     -DGGML_AVX512_BF16=ON -DLLAMA_CURL=OFF
+# The startup banner must print HAVE_FANCY_SIMD is defined — without those
+# macros the quantized GEMM silently falls back to AVX2 (no warning).
+#
+# Model: mradermacher imatrix quant of the uncensored Flash-Next (qwen4exp
+# MoE: 512 experts / 10 used, 48 layers = 12 attention + 36 GDN/ssm, QSA
+# indexer, hyper-connections, PLE n-gram table ~320M rows). 111 GiB shipped
+# as 3 parts, concatenated on disk; each part's sha256 was verified against
+# its HF LFS oid at download time.
+#
+# Placement (ikawrakow's hybrid recipe, upstream issue #2402):
+#   - --cpu-moe: ALL expert tensors (121B elems, ~71 GB) live in the Strix
+#     Halo's LPDDR5X via the CPU path — ik's fastest known configuration
+#     for this shape (the author's 2x3090 + Threadripper reference box
+#     measures ~50 t/s on the same model family).
+#   - -ngl 999: dense/attention/output weights and the KV cache on the
+#     3090 (~3 GB of weights + ~6 GB KV at 262k f16; 24 GB card).
+#   - The ~35 GB PLE table (per_layer_token_embd) is intentionally
+#     RAM/mmap-resident by design — only a tiny slice is touched per token.
+#     NEVER add --no-mmap (the STRIX-era lesson transfers: the cgroup
+#     killer takes the process with nothing in the server log) and do not
+#     try to pin PLE to VRAM (it cannot fit a 24 GB card).
+#
+# Sampling: greedy-by-default to preserve the halogen-era client contract
+# (a request without temperature decodes greedy; temp 0.0 = argmax).
+# Clients that send their own temperature get their own values.
+#
+# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (see
+# the 2026-08-13 HIP incident in AGENTS.md). llama.cpp reads --host and
+# --temp as CLI flags, never as environment variables.
+
+# ik_llama.cpp runtime binary (master d85a1d7 build).
+LLAMA_BIN=/home/dank/GitPackages/ik_llama.cpp/build/bin/llama-server
+
+# Model package from
+# https://huggingface.co/mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF
+# (parts concatenated). The mmproj is the matching vision tower from the
+# static repo mradermacher/Qwen3.8-Flash-Next-Uncensored-GGUF — ik_llama.cpp
+# links examples/mtmd, so image input survives the runtime swap.
+MODEL_DIR=/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M
+MODEL_FILE=Qwen3.8-Flash-Next-Uncensored.i1-Q4_K_M.gguf
+MMPROJ_PATH=/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M/Qwen3.8-Flash-Next-Uncensored.mmproj-f16.gguf
+
+# Qwen-Sharp chat template (tracked in the repo, staged on the host). The
+# server emits reasoning in DeepSeek format so clients receive
+# reasoning_content separately instead of literal thinking tags in content.
+CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja
+
+# Alias + network.
+ALIAS=frank-large
+BIND_ADDR=0.0.0.0
+LISTEN_PORT=7770
+
+# Serving: single 262144-token slot (the model's full context window),
+# batch 4096 / ubatch 512, 16+16 threads (the Halo CPU has 16 cores, no
+# SMT), f16 KV. The KV cache scales with the 12 full-attention layers only
+# (~6 GB f16 at 262k); the GDN/ssm layers keep constant-size state. If
+# VRAM headroom is needed, --cache-type-k/v q8_0 halves the KV footprint.
+CTX_SIZE=262144
+PARALLEL=1
+BATCH_SIZE=4096
+UBATCH_SIZE=512
+NUM_THREADS=16
+THREAD_BATCH=16
+CACHE_TYPE_K=f16
+CACHE_TYPE_V=f16
+
+# GPU/MoE placement: everything on the 3090 except the experts, which all
+# live in host RAM (--cpu-moe). See the placement notes above.
+NGL=999
+CPU_MOE=on
+FLASH_ATTN=on
+
+# Prompt cache (RAM budget in MiB) and slot state dir.
+CACHE_RAM=8192
+SLOT_SAVE_PATH=/home/dank/serverAppConfigs/scripts/slot-state
+
+# Greedy-by-default (halogen contract). See sampling note above.
+TEMPERATURE=0.0
+
+# Reasoning on: DeepSeek format projects thinking into the API's
+# reasoning_content field instead of literal think tags in content;
+# unlimited reasoning budget (the model decides).
+REASONING_FORMAT=deepseek
+REASONING_BUDGET=-1