Frank
d4ce76b68b
Every comment that invoked a 'law' (50 word-boundary sites across 28 tracked files) now states what is going on and why, in terms that transfer to any project: gate twins drift until one side greenlights what the other refuses; a tag re-points while a digest pins; a restrictive engine behavior belongs to the runtime that imposed it and must be re-tested, not inherited; a speculative head is tensors in the file, not a flag; snapshot links dangle where the container resolves them; exit 0 is a clean stop to systemd, so a self-exiting daemon on on-failure goes dark reporting success. Event narratives and dates are gone from these comments; the mechanics carry the argument. Behavior-identical: bash -n, py_compile, yaml parse, full bats suite 408/408 green. One symbol rename rides along (stack script local scale_law -> scale_contract, with its log text).
146 lines
6.8 KiB
Bash
Executable File
146 lines
6.8 KiB
Bash
Executable File
#!/usr/bin/env bash
|
||
# Serve HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive (Q6_K) as
|
||
# frank-zealot on Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
|
||
#
|
||
# 2026-10-07 (Frank): Qwen3.5-9B swap at TWO 200,704-token conversations
|
||
# (--ctx-size 401408 --parallel 2) with EXACT f16 KV — the 9B rung frees the
|
||
# room the 26B MoE Gemma4 shape lacked, so quantized V retires. MTP stays off:
|
||
# this GGUF carries no MTP tensors at all (verified from its tensor-name
|
||
# list), so there is no in-file head to draft with. Verbosity 4 ships so the
|
||
# VRAM-fit forensics (KV layout, buffer sizes, graph capture) land in the journal.
|
||
#
|
||
# Architecture: qwen35 — dense 9B, 32 layers with full_attention_interval 4
|
||
# (8 full-attention layers at 3,7,...,31 + 24 Gated DeltaNet linear layers),
|
||
# 16 heads / 4 KV heads × 256 dim, native 262,144, vocab 248,320. The KV pool
|
||
# scales with the 8 full layers: 32 KiB/token at f16 K + f16 V (12.25 GiB at
|
||
# the 401,408 pool); the DeltaNet layers cost a fixed recurrent state per
|
||
# slot. Feasibility is priced from the MiMo-V2.6-Distill-Qwen-9B boots (same
|
||
# geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279
|
||
# at ubatch 2048 — the boot ledger, not arithmetic, is the referee.
|
||
# Vision comes from the publisher's own BF16 projector (the model is natively
|
||
# multimodal), so --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is
|
||
# the documented text-only opt-out.
|
||
#
|
||
# Sampling: llama.cpp takes it per request (the fleet harness sends the
|
||
# publisher recipe from models.yml: thinking temp 0.6 / top_p 0.95 / top_k 20).
|
||
#
|
||
# Tunables come from frank-zealot.env unless FRANK_ZEALOT_ENV_FILE overrides.
|
||
set -euo pipefail
|
||
|
||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||
|
||
# The artifact gates are the shared library (scripts/lib/llm-gates.sh): the
|
||
# serve script and the deployer gate the same payload, and two hand-written
|
||
# copies of the same checks drift apart until one side passes what the
|
||
# other refuses.
|
||
export GATE_PREFIX=frank-zealot
|
||
# shellcheck disable=SC1091
|
||
source "$SCRIPT_DIR/lib/llm-gates.sh"
|
||
ENV_FILE="${FRANK_ZEALOT_ENV_FILE:-$SCRIPT_DIR/frank-zealot.env}"
|
||
if [[ -f "$ENV_FILE" ]]; then
|
||
set -a
|
||
# shellcheck disable=SC1090
|
||
source "$ENV_FILE"
|
||
set +a
|
||
fi
|
||
|
||
# --- defaults (used when the env file does not set them) -------------------
|
||
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
|
||
: "${MODEL_FILE:=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}"
|
||
: "${ALIAS:=frank-zealot}"
|
||
: "${BIND_ADDR:=0.0.0.0}"
|
||
: "${LISTEN_PORT:=7771}"
|
||
# ctx-size is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704
|
||
# per conversation, inside the model's trained 262,144. The harness
|
||
# contextWindow MUST quote the PER-SLOT number — a longer prompt is answered
|
||
# with a 400, never truncated.
|
||
: "${CTX_SIZE:=401408}"
|
||
: "${PARALLEL:=2}"
|
||
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves).
|
||
# 2048 is the measured CUDA optimum (MiMo era, same arch/shape: +4.6 % prefill
|
||
# at 8.9k depth, decode flat, 616 MiB). The logical batch is VRAM- and
|
||
# throughput-neutral at a fixed ubatch.
|
||
: "${BATCH_SIZE:=4096}"
|
||
: "${UBATCH_SIZE:=2048}"
|
||
# Exact (unquantized) KV: f16 K + f16 V, restoring the MiMo-era contract that
|
||
# the 26B Gemma4 shape could not afford. 32 KiB/token at this geometry
|
||
# (8 full-attention layers × 4 KV heads × 256 dim); 12.25 GiB at the 401,408
|
||
# pool. Explicit either way so the contract is visible in the argv, and
|
||
# --flash-attn stays required for any quantized-V fallback (q8_0 = llama.cpp's
|
||
# FP8 class; no true fp8 KV dtype exists in this build).
|
||
: "${CACHE_TYPE_K:=f16}"
|
||
: "${CACHE_TYPE_V:=f16}"
|
||
# Speculative decoding is OFF: this Q6_K file carries NO nextn/mtp tensors,
|
||
# so there is no in-file head — a speculative head is a property of the
|
||
# model file, not of the flags; no flag creates tensors that are absent.
|
||
# naming a drafter requires SPEC_DRAFT_FILE alongside SPEC_TYPE — the Gemma4
|
||
# assistant on disk is arch gemma4-assistant and cannot drafter-match qwen35.
|
||
: "${SPEC_TYPE:=none}"
|
||
: "${SPEC_DRAFT_N_MAX:=3}"
|
||
: "${SPEC_DRAFT_FILE:=}"
|
||
: "${NGL:=99}"
|
||
# Log verbosity threshold (llama.cpp -lv). 3 is the engine default (info);
|
||
# 4 adds debug detail (KV layout, buffer sizes, graph capture) which is what
|
||
# the VRAM-fit forensics need. Levels above 4 flood the journal.
|
||
: "${VERBOSITY:=3}"
|
||
# Vision projector. No colon on purpose: an explicitly empty value means
|
||
# "text-only, on purpose" and must NOT be replaced by the default path.
|
||
: "${MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}"
|
||
|
||
# Fail loudly rather than half-loading: a missing file is a download that
|
||
# has not finished, and Restart=always would turn it into a crash loop that
|
||
# reads like a runtime fault. The refusal text is the shared gate library's,
|
||
# so the deployer's twin refuses identically.
|
||
require_file "$LLAMA_BIN" runtime \
|
||
"build it: scripts/build-llama-master-cuda.sh" || exit 1
|
||
require_file "$MODEL_FILE" gguf \
|
||
"fetch it: bash scripts/download-frank-zealot.sh" || exit 1
|
||
if [[ -n "$MMPROJ_FILE" ]]; then
|
||
require_file "$MMPROJ_FILE" mmproj \
|
||
"fetch it: bash scripts/download-frank-zealot.sh" || exit 1
|
||
fi
|
||
EXTRA_ARGS=()
|
||
if [[ -n "$MMPROJ_FILE" ]]; then
|
||
EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
|
||
fi
|
||
|
||
# Speculative flags move as a set: they reach the argv ONLY when a drafter
|
||
# is named. SPEC_TYPE=none (the shipped contract) strips both — a no-spec
|
||
# baseline must not carry half a spec pair.
|
||
if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
|
||
EXTRA_ARGS+=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
|
||
if [[ -n "$SPEC_DRAFT_FILE" ]]; then
|
||
[[ -e "$SPEC_DRAFT_FILE" ]] || { echo "frank-zealot: missing $SPEC_DRAFT_FILE" >&2; exit 1; }
|
||
EXTRA_ARGS+=(--spec-draft-model "$SPEC_DRAFT_FILE")
|
||
fi
|
||
fi
|
||
|
||
# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
|
||
# open after boot; make sure it exists before the server needs it.
|
||
nvidia-modprobe -u -c=0 2>/dev/null || true
|
||
|
||
# Flash attention stays pinned on: it is REQUIRED for quantized V (q8_0),
|
||
# and the build carries GGML_CUDA_FA_ALL_QUANTS=ON (since 2026-09-17), so
|
||
# the q8_0 KV contract in frank-zealot.env is a real path — without this
|
||
# flag or that build option the engine silently falls back and the
|
||
# contract becomes a lie.
|
||
ARGS=(
|
||
--model "$MODEL_FILE"
|
||
--alias "$ALIAS"
|
||
--host "$BIND_ADDR"
|
||
--port "$LISTEN_PORT"
|
||
--ctx-size "$CTX_SIZE"
|
||
--parallel "$PARALLEL"
|
||
--batch-size "$BATCH_SIZE"
|
||
--ubatch-size "$UBATCH_SIZE"
|
||
--n-gpu-layers "$NGL"
|
||
--flash-attn on
|
||
--cache-type-k "$CACHE_TYPE_K"
|
||
--cache-type-v "$CACHE_TYPE_V"
|
||
--log-verbosity "$VERBOSITY"
|
||
--jinja
|
||
--reasoning auto
|
||
--reasoning-format deepseek
|
||
)
|
||
exec "$LLAMA_BIN" "${ARGS[@]}" ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}
|