Swap LLM fleet to Ornith CRACK (small) and kingjones ROCmFP4 STRIX MTP (large); apply Qwen-Sharp chat template
- frank-default-small (Dankserver): Ornith-1.5-35B-A3B-CRACK Q8_0, bundled native draft-mtp (n_max 4), MoE expert FFN tensors pinned to CPU via an explicit --override-tensor \.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU (replaces --n-cpu-moe), Qwen-Sharp chat template. - frank-default-large (Dankclaw): kingjones Qwen3.8-27B-Uncensored-Q6_0 ROCmFP4 AGENT dense GGUF, standalone mtp-Qwen3.8-27B-Q4_0 draft head (--model-draft, draft-mtp n_max 4), Qwen-Sharp chat template. Drop the retired PromptForge .pfs sidecar lanes. - Track scripts/chat-template.jinja (v22.4.0 Sharp) and apply via --chat-template-file on both services. - Docs: new kingjones-strix-mtp and dealign-ornith-crack runbook pages; update index, deployment, and both system specs; models.json names. Smoke-tested both models on the real hardware on scratch ports before commit: model loads, MTP draft attaches, Sharp template applies, chat completion returns.
This commit is contained in:
1 parent
189934af15
commit
4462aa992c
14 files changed
+827
-156
No files matched your search
@@ -1,38 +1,49 @@
|
||||
# Tunables for frank-default-small (HauhauCS Qwen3.6-35B-A3B Q6_K_P,
|
||||
# ngram-mod spec, Dankserver, RTX 3090).
|
||||
# Tunables for frank-default-small (Ornith-1.5-35B-A3B CRACK Q8_0, MTP MoE,
|
||||
# Dankserver, RTX 3090).
|
||||
#
|
||||
# Runtime: upstream llama.cpp llama-server (b10121). RTX 3090 24 GB;
|
||||
# Ryzen 7 7700X; 62 GB RAM. MoE experts offloaded to CPU (--n-cpu-moe 36).
|
||||
# Restored 2026-08-18 (was the pre-2026-08-17-redo frank-default-small).
|
||||
# Runtime: upstream llama.cpp llama-server (b10121). RTX 3090 24 GB; Ryzen
|
||||
# 7 7700X; 62 GB RAM. MoE expert weights pinned to CPU with an explicit
|
||||
# --override-tensor regex (not --n-cpu-moe). Replaced the HauhauCS
|
||||
# Qwen3.6-35B-A3B model on 2026-08-24.
|
||||
#
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
|
||||
# ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
|
||||
# incident). This box is NVIDIA/CUDA, but the rule keeps the env portable.
|
||||
|
||||
# Model files (already on disk; Q6_K_P GGUF + f16 mmproj vision encoder).
|
||||
MODEL_DIR=/home/dank/models
|
||||
MODEL_FILE=Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q6_K_P.gguf
|
||||
MMPROJ_FILE=mmproj-Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-f16.gguf
|
||||
# Model files (Q8_0 CRACK GGUF + f16 mmproj vision encoder + MTP head bundled
|
||||
# in the GGUF). Dealign2026-08-24.
|
||||
MODEL_DIR=/home/dank/models/Ornith-1.5-35B-A3B-CRACK
|
||||
MODEL_FILE=Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf
|
||||
MMPROJ_FILE=mmproj-Ornith-1.5-35B-A3B-f16.gguf
|
||||
# Upstream llama.cpp binary (b10121).
|
||||
LLAMA_SERVER=/home/dank/llama.cpp/build/bin/llama-server
|
||||
|
||||
# Qwen-Sharp chat template (tracked in the repo, staged on the host).
|
||||
CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja
|
||||
|
||||
ALIAS=frank-default-small
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7770
|
||||
|
||||
# 3 parallel slots x 262144 ctx each = ~786432 total context pool (max per-slot).
|
||||
# Dual-f16 KV at 786432 cells leaves ~15 GiB free on the 3090.
|
||||
# 3 parallel slots x 262144 total context pool (max per-slot), unchanged
|
||||
# from the prior Qwen3.6 service. Lower PARALLEL if VRAM is tight with the
|
||||
# larger Q8_0 payload.
|
||||
CTX_SIZE=786432
|
||||
PARALLEL=3
|
||||
|
||||
# Spec: ngram-mod (n_min=48, n_max=64, n_match=24).
|
||||
SPEC_TYPE=ngram-mod
|
||||
NGRAM_MOD_N_MIN=48
|
||||
NGRAM_MOD_N_MAX=64
|
||||
NGRAM_MOD_N_MATCH=24
|
||||
# Spec: MoE model with a bundled MTP head -> native draft-mtp, n_max 4
|
||||
# (the knee). ngram under MTP hurts; MTP alone is the proven best for MoE.
|
||||
SPEC_TYPE=draft-mtp
|
||||
SPEC_DRAFT_N_MAX=4
|
||||
SPEC_DRAFT_N_MIN=0
|
||||
SPEC_DRAFT_P_MIN=0.0
|
||||
|
||||
# Expert offload: 38 of 40 layers keep MoE weights on CPU; 2 on GPU.
|
||||
N_CPU_MOE=38
|
||||
# Expert offload via explicit tensor override (replaces --n-cpu-moe). Only
|
||||
# the MoE FFN exps tensors move to CPU; everything else up to --gpu-layers
|
||||
# stays on the 24 GB card. The regex is the same one the built-in --cpu-moe
|
||||
# expands. Buffer type "CPU" is the CPU buft name on this host.
|
||||
OVERRIDE_EXPERT='\.ffn_(up|down|gate|gate_up)_(ch|)exps'
|
||||
OVERRIDE_BUFT=CPU
|
||||
N_GPU_LAYERS=all
|
||||
|
||||
# KV cache on GPU, both K and V at f16.
|
||||
@@ -40,22 +51,22 @@ CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=f16
|
||||
|
||||
VERBOSITY=4
|
||||
# Store SWA/context checkpoints in RAM to hold peak GPU memory down.
|
||||
# Store context checkpoints in RAM to hold peak GPU memory down.
|
||||
CTX_CHECKPOINTS=16
|
||||
|
||||
# Big prefill batches (ubatch 2048, batch 4096).
|
||||
# Big prefill batches.
|
||||
BATCH_SIZE=4096
|
||||
UBATCH_SIZE=2048
|
||||
|
||||
# Eager RAM load (MoE experts on CPU; ~31 GB RAM).
|
||||
# Eager RAM load (expert weights on CPU; ~31 GB RAM).
|
||||
NO_MMAP=1
|
||||
|
||||
# Sampling (HauhauCS Qwen3.6-35B-A3B model card, thinking-general).
|
||||
# Sampling (Dealign Ornith model card, thinking-general).
|
||||
TEMPERATURE=1.0
|
||||
TOP_K=20
|
||||
TOP_P=0.95
|
||||
MIN_P=0.0
|
||||
|
||||
# Compute threads lowered to 7 so other processes keep a core.
|
||||
# Compute threads lowered to 8 so other processes keep a core.
|
||||
THREADS=7
|
||||
THREADS_HTTP=8
|
||||
THREADS_HTTP=8
|
||||
Reference in new issue
Block a user