Frank
43ec77fee6
Dankclaw's endpoints are re-split. frank-large is the halogen-flash-server quadlet again -- the 2026-09-13 unit file was never touched by the two-day ik_llama.cpp interlude, so the runtime was still in the repo and its image is still in local storage -- retuned to the shape asked for: HALOGEN_CTX 200192 with a 400,384-position pool and two slots, exactly two 200k conversations. frank-small is new: mainline llama.cpp on the RTX 3090 serving ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF at IQ3_S, using the -mtp build (it carries blk.64.nextn.*, which is what lets --spec-type draft-mtp run on the main model's own weights instead of a second 12 GiB copy) plus the BF16 vision projector, as one 131,072-token slot with f16 KV on port 7771. The host is shared now, and that is what the halogen retune pays for: HALOGEN_MAX_TOK 24576 (arena 20.6 -> 16.4 GiB, ~4-5% prefill) and the vision tower dropped (1.97 GiB) hand ~6 GiB back to the host, and the reserve rises to 20 to keep the fit check honest. Vision moved wholesale to frank-small. Retired to scripts/.AGENTTRASH/: the ik trio (env, service, serve script), the Qwen-Sharp template only that runtime read, and four benchmark harnesses bound to retired runtimes. Because retiring a file makes any test that greps it pass by absence, the seams assert the archive exists AND the live path does not.
114 lines
5.7 KiB
Bash
114 lines
5.7 KiB
Bash
# Tunables for frank-small (ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, the IQ3_S
|
|
# build with the MTP head, plus the BF16 vision projector) on mainline
|
|
# llama.cpp, Dankclaw's RTX 3090, since 2026-09-16.
|
|
#
|
|
# Runtime: llama.cpp master 37b53fd (2026-09-16), CUDA build for sm_86:
|
|
# cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
|
|
# -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
|
|
# cmake --build build-cuda --target llama-server -j
|
|
# GGML_CUDA_FA_ALL_QUANTS is deliberately OFF in that build: it is only needed
|
|
# for QUANTIZED KV with flash attention, and this service runs f16 KV (below).
|
|
#
|
|
# Model: ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, the `-mtp` build of IQ3_S.
|
|
# GSQ-RCO assigns a separate quantization type to every tensor from a
|
|
# gradient-based search under a total size budget, rather than applying one
|
|
# type throughout; IQ3_S at 3.50 bpw is the release's task-lossless operating
|
|
# point (AIME25 100.00 and LiveCodeBench v6 85.71 match BF16 exactly,
|
|
# GPQA-Diamond 0.51 behind, at 11.8 GB against 53.8).
|
|
#
|
|
# WHY THAT FILE: the `-mtp` build is ~0.35 GB larger because it carries the
|
|
# model's Multi-Token Prediction head as GGUF block `blk.64.nextn.*`
|
|
# (qwen35.nextn_predict_layers = 1). llama.cpp loads it from the SAME file --
|
|
# `--spec-type draft-mtp` sets load_mtp, and the draft context runs on the main
|
|
# model's weights (common/common.cpp passes shares_model when there is no
|
|
# --model-draft; speculative.cpp gives it LLAMA_CONTEXT_TYPE_MTP), so there is
|
|
# no second 12 GB copy in VRAM. MTP is therefore a property of the FILE, not of
|
|
# the flags: the plain IQ3_S build has no MTP tensors to load.
|
|
#
|
|
# Architecture (GGUF arch qwen35): 64 layers laid out as 16 x (3 x (Gated
|
|
# DeltaNet -> FFN) + 1 x (Gated Attention -> FFN)) -- 48 linear-attention
|
|
# layers with CONSTANT-SIZE per-sequence state and 16 full-attention layers,
|
|
# 4 KV heads x 256 head_dim. Only the attention layers take KV, so a token
|
|
# costs 2 (K,V) x 4 heads x 256 x 2 B = 64 KiB at f16 (34 KiB at q8_0, 18 KiB
|
|
# at q4_0), and the DeltaNet layers add ~0.16 GiB per sequence in f32 at any
|
|
# depth.
|
|
#
|
|
# SIZING (RTX 3090, 24576 MiB) -- modelled, and the loader's own ledger
|
|
# ("KV self size", "compute buffer size") outranks it:
|
|
# weights 11.29 GiB (12,120,016,960 B)
|
|
# mmproj 0.87 GiB (931,146,528 B, 27 blocks)
|
|
# MTP draft KV 0.50 GiB (its context is n_ctx long
|
|
# over one nextn layer: 4 KiB/token)
|
|
# DeltaNet state 0.16 GiB (one sequence)
|
|
# CUDA context + prefill/decode bufs ~1.5 GiB
|
|
# KV at CTX_SIZE 8.00 GiB (131,072 x 64 KiB)
|
|
# --------
|
|
# ~22.3 GiB of 24, ~2 GiB of slack
|
|
# The binding number is the KV: ~9.5 GiB is left after the fixed terms, which
|
|
# at 64 KiB/token is ~155k tokens across all slots. The deployed shape is one
|
|
# slot at 131,072, full f16 precision, no KV quantization risk, no rebuild.
|
|
# Alternatives inside the same budget: 2 x 65536 at f16; 4 x 65536 or
|
|
# 1 x 262144 at q8_0, both of which need a rebuild with
|
|
# -DGGML_CUDA_FA_ALL_QUANTS=ON (without it quantized-KV prefill collapses
|
|
# ~12x on CUDA -- see the CUDA lesson in AGENTS.md).
|
|
# --parallel N DIVIDES --ctx-size between slots; it does not multiply. One slot
|
|
# means a second concurrent request queues behind the first.
|
|
#
|
|
# Sampling: greedy by default, which is the fleet contract (a request that
|
|
# sends no temperature decodes greedy; a client that sends one gets its own).
|
|
# Qwen's own agentic evaluations used temperature 1.0 / top_p 0.95, and the
|
|
# harness applies exactly that per model in models.yml (extraBody), the same
|
|
# way it did for the earlier frank-large runtime.
|
|
#
|
|
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
|
|
# those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
|
|
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
|
|
|
|
# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
|
|
|
|
# Model package. The -mtp file is required for speculative decoding.
|
|
MODEL_DIR=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp
|
|
MODEL_FILE=Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf
|
|
MMPROJ_PATH=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp/mmproj-Qwen3.8-27B-BF16.gguf
|
|
|
|
# Fleet identity and network. 7771 is this service's own endpoint on Dankclaw;
|
|
# 7770 stays frank-large's. The alias is the id at /v1/models and what clients
|
|
# ask for -- the harness entry in models.yml names it.
|
|
ALIAS=frank-small
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
|
|
# One slot at 131,072 tokens with f16 KV (see the sizing note above).
|
|
CTX_SIZE=131072
|
|
PARALLEL=1
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=f16
|
|
|
|
# Batch geometry: llama.cpp's own defaults (2048 logical / 512 physical). The
|
|
# prefill wall clock at this model size is set by the card, not by ubatch, and
|
|
# the small ubatch keeps the DeltaNet chunk buffers small. Raise UBATCH_SIZE
|
|
# only after reading the compute buffer size the loader reports.
|
|
BATCH_SIZE=2048
|
|
UBATCH_SIZE=512
|
|
|
|
# CPU threads: this service runs entirely on the GPU (NGL=999), so threads
|
|
# cover tokenization, sampling and the HTTP layer only. Deliberately modest --
|
|
# this host's CPU is shared with halogen's engine, which is CPU-hungry.
|
|
NUM_THREADS=8
|
|
THREAD_BATCH=8
|
|
|
|
# Every layer on the 3090, with flash attention.
|
|
NGL=999
|
|
FLASH_ATTN=on
|
|
|
|
# Speculative decoding from the model's own MTP head (see the model note).
|
|
# Draft depth stays llama.cpp's default of 3 tokens.
|
|
SPEC_TYPE=draft-mtp
|
|
|
|
# Greedy default (fleet contract); reasoning is extracted into
|
|
# reasoning_content with an unlimited budget (the model decides).
|
|
TEMPERATURE=0.0
|
|
REASONING_FORMAT=deepseek
|
|
REASONING_BUDGET=-1
|