Frank
55f87fcdce
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
63 lines
3.3 KiB
Bash
63 lines
3.3 KiB
Bash
# frank-zealot — llama.cpp (master, CUDA sm_86) serving
|
||
# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090,
|
||
# :7771.
|
||
#
|
||
# 2026-10-07 (Frank): swap from Gemma4-26B-A4B QAT Q4_K_M (one 200,704 slot)
|
||
# to Qwen3.5-9B Q6_K at TWO 200,704-token conversations, EXACT (f16) KV,
|
||
# MTP disabled (SPEC_TYPE=none), log verbosity 4.
|
||
#
|
||
# Architecture — read from the GGUF's own header (HTTP range read, not the
|
||
# model card): arch qwen35, 32 layers, dense 9B, full_attention_interval 4
|
||
# ⇒ 8 full-attention layers (3,7,11,...,31) + 24 Gated DeltaNet linear layers,
|
||
# 16 attention heads / 4 KV heads × 256 dim, ssm conv_kernel 4 / state_size
|
||
# 128 / group_count 16 / time_step_rank 32 / inner_size 4096, native 262,144,
|
||
# vocab 248,320, file_type 18 (Q6_K). Tensor-type histogram: Q6_K×250 +
|
||
# F32×177 — standard types only, so the stock master build loads it (no
|
||
# custom-quant lock-in like the ROCmFPX era). NO nextn/mtp tensors exist, so
|
||
# in-file MTP is impossible here; SPEC_TYPE=none is forced by the payload.
|
||
#
|
||
# KV rate law: 8 full layers × 4 KV heads × 256 dim = 8,192 elements per
|
||
# token per cache. f16 K + f16 V = 32,768 B/token = 32 KiB/token, so the
|
||
# 401,408-token pool is 12.25 GiB. The 24 DeltaNet layers cost a fixed
|
||
# recurrent state per SLOT (not per token) — that is why two 200k slots fit
|
||
# where the 26B MoE fit only one.
|
||
#
|
||
# Feasibility comes from the predecessor's MEASURED boots, not a formula:
|
||
# MiMo-V2.6-Distill-Qwen-9B (identical qwen35 geometry, 7,793,710,624 B) at
|
||
# 2×200,704 f16 K + f16 V measured 21,276 used / 2,849 free at ubatch 1024
|
||
# and 21,846 / 2,279 at ubatch 2048. This file is 415 MiB lighter in weights,
|
||
# its BF16 mmproj is 3 MiB heavier ⇒ projected ~21,430 used / ~2,700 free.
|
||
# The verbosity-4 boot ledger is the referee.
|
||
#
|
||
# Ubatch 2048 is the measured CUDA optimum (measured on the MiMo predecessor
|
||
# at this exact arch + shape: +4.6 % prefill at 8.9k depth, decode flat, cost
|
||
# 616 MiB). 1024 is the fallback rung if the boot lands under the 2 GiB floor.
|
||
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
||
MODEL_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf
|
||
MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf
|
||
# Served model id — the harness/family name (Protoss rename completed to the
|
||
# wire 2026-10-09: harness ids + homepage cards 2026-10-08, /v1/models aliases
|
||
# this day). The SERVICE, unit, scripts and this env file keep the
|
||
# frank-zealot name; only the alias on the wire changed. Consumers that
|
||
# key on the served id: OpenWebUI (model row, access_grant, api_configs
|
||
# model_ids) and the harness models.yml.
|
||
ALIAS=frank-zealot
|
||
BIND_ADDR=0.0.0.0
|
||
LISTEN_PORT=7771
|
||
# CTX_SIZE is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704
|
||
# per conversation. MUST stay divisible by PARALLEL (the seam enforces it) and
|
||
# the harness contextWindow quotes the PER-SLOT number — a longer prompt is
|
||
# answered with a 400, never truncated.
|
||
CTX_SIZE=401408
|
||
PARALLEL=2
|
||
BATCH_SIZE=4096
|
||
UBATCH_SIZE=2048
|
||
CACHE_TYPE_K=f16
|
||
CACHE_TYPE_V=f16
|
||
SPEC_TYPE=none
|
||
NGL=99
|
||
# Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes
|
||
# and graph capture land in the journal, which is what the VRAM-fit forensics
|
||
# read. Levels above 4 flood the journal.
|
||
VERBOSITY=4
|