llm: retire the trio service names — fleet services renamed to frank-archon/frank-dragoon/frank-zealot
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
This commit is contained in:
@@ -0,0 +1,105 @@
|
||||
# Tunables for frank-dragoon (mradermacher i1-Q4_K_M of
|
||||
# orcarouter/Qwen3.8-Flash-Next-Uncensored on ik_llama.cpp, Dankclaw,
|
||||
# RTX 3090 CUDA + Strix Halo CPU path, since 2026-09-15).
|
||||
#
|
||||
# Runtime: ik_llama.cpp master d85a1d7 (2026-09-15) with full qwen4exp
|
||||
# support (upstream PRs #2365/#2369/#2373 merged). Build recipe (CUDA for
|
||||
# the 3090 at sm_86, Zen5 AVX-512 IQK kernels for the CPU expert path):
|
||||
# cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
|
||||
# -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 \
|
||||
# -DGGML_AVX512=ON -DGGML_AVX512_VBMI=ON -DGGML_AVX512_VNNI=ON \
|
||||
# -DGGML_AVX512_BF16=ON -DLLAMA_CURL=OFF
|
||||
# The startup banner must print HAVE_FANCY_SIMD is defined — without those
|
||||
# macros the quantized GEMM silently falls back to AVX2 (no warning).
|
||||
#
|
||||
# Model: mradermacher imatrix quant of the uncensored Flash-Next (qwen4exp
|
||||
# MoE: 512 experts / 10 used, 48 layers = 12 attention + 36 GDN/ssm, QSA
|
||||
# indexer, hyper-connections, PLE n-gram table ~320M rows). 111 GiB shipped
|
||||
# as 3 parts, concatenated on disk; each part's sha256 was verified against
|
||||
# its HF LFS oid at download time.
|
||||
#
|
||||
# Placement (ikawrakow's hybrid recipe; upstream issue #2402 for the PLE):
|
||||
# - -ot "exps=CPU": ALL 512-expert tensors (121B elems, ~84 GB) live in
|
||||
# the Strix Halo's LPDDR5X via the plain CPU backend buffer — mmap-backed
|
||||
# page cache, ik's fastest known configuration for this shape (the
|
||||
# author's 2x3090 + Threadripper reference box measures ~50 t/s on the
|
||||
# same model family). The shared experts (shexp) keep their GPU-side
|
||||
# default: they run on every token.
|
||||
# - -ot "per_layer_token_embd=CPU": the ~32 GB PLE n-gram table follows —
|
||||
# only a tiny slice is touched per token, so it streams from page cache.
|
||||
# - NEVER use --cpu-moe on this host: it routes the CPU-overridden
|
||||
# weights through the CUDA_Host PINNED buffer (measured 2026-09-15:
|
||||
# 108.12 GiB page-locked, evicted every co-tenant into swap and starved
|
||||
# sshd into unreachability until a power cycle). -ot ...=CPU is the
|
||||
# mmap-backed path with identical DDR5 bandwidth for resident pages,
|
||||
# and the kernel degrades gracefully under pressure (page cache evict
|
||||
# + re-read) instead of stranding anyone.
|
||||
# - GGML_CUDA_NO_PINNED_WEIGHTS=1 (src/llama-load-tensors.cpp) keeps the
|
||||
# CPU-overridden weights mmapped and leaves only the small staging
|
||||
# buffers pinned. (GGML_CUDA_NO_PINNED would also unpin staging —
|
||||
# unnecessary here and slower for prefill transfers.)
|
||||
#
|
||||
# Sampling: greedy-by-default to preserve the halogen-era client contract
|
||||
# (a request without temperature decodes greedy; temp 0.0 = argmax).
|
||||
# Clients that send their own temperature get their own values.
|
||||
#
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (see
|
||||
# the 2026-08-13 HIP incident in AGENTS.md). llama.cpp reads --host and
|
||||
# --temp as CLI flags, never as environment variables.
|
||||
|
||||
# ik_llama.cpp runtime binary (master d85a1d7 build).
|
||||
LLAMA_BIN=/home/dank/GitPackages/ik_llama.cpp/build/bin/llama-server
|
||||
|
||||
# Model package from
|
||||
# https://huggingface.co/mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF
|
||||
# (parts concatenated). The mmproj is the matching vision tower from the
|
||||
# static repo mradermacher/Qwen3.8-Flash-Next-Uncensored-GGUF — ik_llama.cpp
|
||||
# links examples/mtmd, so image input survives the runtime swap.
|
||||
MODEL_DIR=/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M
|
||||
MODEL_FILE=Qwen3.8-Flash-Next-Uncensored.i1-Q4_K_M.gguf
|
||||
MMPROJ_PATH=/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M/Qwen3.8-Flash-Next-Uncensored.mmproj-f16.gguf
|
||||
|
||||
# Qwen-Sharp chat template (tracked in the repo, staged on the host). The
|
||||
# server emits reasoning in DeepSeek format so clients receive
|
||||
# reasoning_content separately instead of literal think tags in content.
|
||||
CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja
|
||||
|
||||
# Alias + network.
|
||||
ALIAS=frank-dragoon
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7770
|
||||
|
||||
# Serving: single 262144-token slot (the model's full context window),
|
||||
# batch 4096 / ubatch 512, 16+16 threads (the Halo CPU has 16 cores, no
|
||||
# SMT), f16 KV. The KV cache scales with the 12 full-attention layers only
|
||||
# (~6 GB f16 at 262k); the GDN/ssm layers keep constant-size state. If
|
||||
# VRAM headroom is needed, --cache-type-k/v q8_0 halves the KV footprint.
|
||||
CTX_SIZE=262144
|
||||
PARALLEL=1
|
||||
BATCH_SIZE=4096
|
||||
UBATCH_SIZE=512
|
||||
NUM_THREADS=16
|
||||
THREAD_BATCH=16
|
||||
CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=f16
|
||||
|
||||
# GPU/MoE placement: everything on the 3090 except the experts and the PLE
|
||||
# table, which both live in host RAM via the CPU backend buffer (see the
|
||||
# placement notes above).
|
||||
NGL=999
|
||||
# ik_llama.cpp CUDA host-memory policy: CPU-overridden weights stay mmapped;
|
||||
# only the small staging buffers are pinned (see the placement notes above).
|
||||
GGML_CUDA_NO_PINNED_WEIGHTS=1
|
||||
|
||||
# Prompt cache (RAM budget in MiB) and slot state dir.
|
||||
CACHE_RAM=8192
|
||||
SLOT_SAVE_PATH=/home/dank/serverAppConfigs/scripts/slot-state
|
||||
|
||||
# Greedy-by-default (halogen contract). See sampling note above.
|
||||
TEMPERATURE=0.0
|
||||
|
||||
# Reasoning on: DeepSeek format projects thinking into the API's
|
||||
# reasoning_content field instead of literal think tags in content;
|
||||
# unlimited reasoning budget (the model decides).
|
||||
REASONING_FORMAT=deepseek
|
||||
REASONING_BUDGET=-1
|
||||
Reference in New Issue
Block a user