# Tunables for frank-small (ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, the IQ3_S
# build with the MTP head, plus the BF16 vision projector) on mainline
# llama.cpp, Dankclaw's RTX 3090, since 2026-09-16.
#
# Runtime: llama.cpp master 37b53fd (2026-09-16), CUDA build for sm_86:
#   cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
#     -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
#   cmake --build build-cuda --target llama-server -j
# GGML_CUDA_FA_ALL_QUANTS is deliberately OFF in that build: it is only needed
# for QUANTIZED KV with flash attention, and this service runs f16 KV (below).
#
# Model: ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, the `-mtp` build of IQ3_S.
# GSQ-RCO assigns a separate quantization type to every tensor from a
# gradient-based search under a total size budget, rather than applying one
# type throughout; IQ3_S at 3.50 bpw is the release's task-lossless operating
# point (AIME25 100.00 and LiveCodeBench v6 85.71 match BF16 exactly,
# GPQA-Diamond 0.51 behind, at 11.8 GB against 53.8).
#
# WHY THAT FILE: the `-mtp` build is ~0.35 GB larger because it carries the
# model's Multi-Token Prediction head as GGUF block `blk.64.nextn.*`
# (qwen35.nextn_predict_layers = 1). llama.cpp loads it from the SAME file --
# `--spec-type draft-mtp` sets load_mtp, and the draft context runs on the main
# model's weights (common/common.cpp passes shares_model when there is no
# --model-draft; speculative.cpp gives it LLAMA_CONTEXT_TYPE_MTP), so there is
# no second 12 GB copy in VRAM. MTP is therefore a property of the FILE, not of
# the flags: the plain IQ3_S build has no MTP tensors to load.
#
# Architecture (GGUF arch qwen35): 64 layers laid out as 16 x (3 x (Gated
# DeltaNet -> FFN) + 1 x (Gated Attention -> FFN)) -- 48 linear-attention
# layers with CONSTANT-SIZE per-sequence state and 16 full-attention layers,
# 4 KV heads x 256 head_dim. Only the attention layers take KV, so a token
# costs 2 (K,V) x 4 heads x 256 x 2 B = 64 KiB at f16 (34 KiB at q8_0, 18 KiB
# at q4_0), and the DeltaNet layers add ~0.16 GiB per sequence in f32 at any
# depth.
#
# SIZING (RTX 3090, 24576 MiB) -- modelled, and the loader's own ledger
# ("KV self size", "compute buffer size") outranks it:
#   weights                            11.29 GiB (12,120,016,960 B)
#   mmproj                              0.87 GiB (931,146,528 B, 27 blocks)
#   MTP draft KV                        0.50 GiB (its context is n_ctx long
#                                       over one nextn layer: 4 KiB/token)
#   DeltaNet state                      0.16 GiB (one sequence)
#   CUDA context + prefill/decode bufs  ~1.5 GiB
#   KV at CTX_SIZE                      8.00 GiB (131,072 x 64 KiB)
#                                      --------
#                                      ~22.3 GiB of 24, ~2 GiB of slack
# The binding number is the KV: ~9.5 GiB is left after the fixed terms, which
# at 64 KiB/token is ~155k tokens across all slots. The deployed shape is one
# slot at 131,072, full f16 precision, no KV quantization risk, no rebuild.
# Alternatives inside the same budget: 2 x 65536 at f16; 4 x 65536 or
# 1 x 262144 at q8_0, both of which need a rebuild with
# -DGGML_CUDA_FA_ALL_QUANTS=ON (without it quantized-KV prefill collapses
# ~12x on CUDA -- see the CUDA lesson in AGENTS.md).
# --parallel N DIVIDES --ctx-size between slots; it does not multiply. One slot
# means a second concurrent request queues behind the first.
#
# Sampling: greedy by default, which is the fleet contract (a request that
# sends no temperature decodes greedy; a client that sends one gets its own).
# Qwen's own agentic evaluations used temperature 1.0 / top_p 0.95, and the
# harness applies exactly that per model in models.yml (extraBody), the same
# way it did for the earlier frank-large runtime.
#
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
# those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.

# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86.
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server

# Model package. The -mtp file is required for speculative decoding.
MODEL_DIR=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp
MODEL_FILE=Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf
MMPROJ_PATH=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp/mmproj-Qwen3.8-27B-BF16.gguf

# Fleet identity and network. 7771 is this service's own endpoint on Dankclaw;
# 7770 stays frank-large's. The alias is the id at /v1/models and what clients
# ask for -- the harness entry in models.yml names it.
ALIAS=frank-small
BIND_ADDR=0.0.0.0
LISTEN_PORT=7771

# One slot at 131,072 tokens with f16 KV (see the sizing note above).
CTX_SIZE=131072
PARALLEL=1
CACHE_TYPE_K=f16
CACHE_TYPE_V=f16

# Batch geometry: llama.cpp's own defaults (2048 logical / 512 physical). The
# prefill wall clock at this model size is set by the card, not by ubatch, and
# the small ubatch keeps the DeltaNet chunk buffers small. Raise UBATCH_SIZE
# only after reading the compute buffer size the loader reports.
BATCH_SIZE=2048
UBATCH_SIZE=512

# CPU threads: this service runs entirely on the GPU (NGL=999), so threads
# cover tokenization, sampling and the HTTP layer only. Deliberately modest --
# this host's CPU is shared with halogen's engine, which is CPU-hungry.
NUM_THREADS=8
THREAD_BATCH=8

# Every layer on the 3090, with flash attention.
NGL=999
FLASH_ATTN=on

# Speculative decoding from the model's own MTP head (see the model note).
# Draft depth stays llama.cpp's default of 3 tokens.
SPEC_TYPE=draft-mtp

# Greedy default (fleet contract); reasoning is extracted into
# reasoning_content with an unlimited budget (the model decides).
TEMPERATURE=0.0
REASONING_FORMAT=deepseek
REASONING_BUDGET=-1