# frank-zealot — llama.cpp (master, CUDA sm_86) serving
# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090,
# :7771.
#
# 2026-10-07 (Frank): swap from Gemma4-26B-A4B QAT Q4_K_M (one 200,704 slot)
# to Qwen3.5-9B Q6_K at TWO 200,704-token conversations, EXACT (f16) KV,
# MTP disabled (SPEC_TYPE=none), log verbosity 4.
#
# Architecture — read from the GGUF's own header (HTTP range read, not the
# model card): arch qwen35, 32 layers, dense 9B, full_attention_interval 4
# ⇒ 8 full-attention layers (3,7,11,...,31) + 24 Gated DeltaNet linear layers,
# 16 attention heads / 4 KV heads × 256 dim, ssm conv_kernel 4 / state_size
# 128 / group_count 16 / time_step_rank 32 / inner_size 4096, native 262,144,
# vocab 248,320, file_type 18 (Q6_K). Tensor-type histogram: Q6_K×250 +
# F32×177 — standard types only, so the stock master build loads it (no
# custom-quant lock-in like the ROCmFPX era). NO nextn/mtp tensors exist, so
# in-file MTP is impossible here; SPEC_TYPE=none is forced by the payload.
#
# KV rate law: 8 full layers × 4 KV heads × 256 dim = 8,192 elements per
# token per cache. f16 K + f16 V = 32,768 B/token = 32 KiB/token, so the
# 401,408-token pool is 12.25 GiB. The 24 DeltaNet layers cost a fixed
# recurrent state per SLOT (not per token) — that is why two 200k slots fit
# where the 26B MoE fit only one.
#
# Feasibility comes from the predecessor's MEASURED boots, not a formula:
# MiMo-V2.6-Distill-Qwen-9B (identical qwen35 geometry, 7,793,710,624 B) at
# 2×200,704 f16 K + f16 V measured 21,276 used / 2,849 free at ubatch 1024
# and 21,846 / 2,279 at ubatch 2048. This file is 415 MiB lighter in weights,
# its BF16 mmproj is 3 MiB heavier ⇒ projected ~21,430 used / ~2,700 free.
# The verbosity-4 boot ledger is the referee.
#
# Ubatch 2048 is the measured CUDA optimum (measured on the MiMo predecessor
# at this exact arch + shape: +4.6 % prefill at 8.9k depth, decode flat, cost
# 616 MiB). 1024 is the fallback rung if the boot lands under the 2 GiB floor.
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
MODEL_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf
MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf
# Served model id — the harness/family name (Protoss rename completed to the
# wire 2026-10-09: harness ids + homepage cards 2026-10-08, /v1/models aliases
# this day). The SERVICE, unit, scripts and this env file keep the
# frank-zealot name; only the alias on the wire changed. Consumers that
# key on the served id: OpenWebUI (model row, access_grant, api_configs
# model_ids) and the harness models.yml.
ALIAS=frank-zealot
BIND_ADDR=0.0.0.0
LISTEN_PORT=7771
# CTX_SIZE is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704
# per conversation. MUST stay divisible by PARALLEL (the seam enforces it) and
# the harness contextWindow quotes the PER-SLOT number — a longer prompt is
# answered with a 400, never truncated.
CTX_SIZE=401408
PARALLEL=2
BATCH_SIZE=4096
UBATCH_SIZE=2048
CACHE_TYPE_K=f16
CACHE_TYPE_V=f16
SPEC_TYPE=none
NGL=99
# Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes
# and graph capture land in the journal, which is what the VRAM-fit forensics
# read. Levels above 4 flood the journal.
VERBOSITY=4