frank-secondary: swap Gemma4-26B-A4B Q4_K_M -> Qwen3.5-9B-Uncensored Q6_K 2x200,704
Capacity trade: the 9B dense-hybrid (qwen35: 32 layers = 8 full-attention + 24 Gated DeltaNet, 4 KV heads x 256 dim, native 262,144, MTP-less, natively multimodal) costs ~7 GiB of weights where the 26B MoE cost ~16.6 GiB. The freed room buys back the second conversation slot and exact KV: --ctx-size 401408 --parallel 2 (two 200,704 windows), f16 K + f16 V, MTP off (the file carries no nextn/mtp tensors - flag-only enable is impossible), vision via the publisher's BF16 mmproj, ubatch 1024 (the CUDA decode-optimal value), verbosity 4, publisher sampler recipe (0.6/0.95/20/0) in models.yml. Hosts with the MiMo-V2.6-Distill Q6_K boots at this geometry: 21,276 MiB used / 2,849 free at ubatch 1024, 40,960 tokens of fp16 KV at ubatch 2048. Compaction stays routed here; window math is unchanged (maxTokens 16,384 keeps the worst-case harness request inside the slot). Gemma4 payload (Q4_K_M + mmproj + MTP assistant) stays on disk as the rollback; its runbook is retired in place, keeping the attention_k_eq_v pair-billing law. Co-Authored-By: omp <omp@franksdank.cloud>
This commit is contained in:
Frank
omp
@@ -1225,13 +1225,14 @@ deploy_llm_large() {
|
||||
#
|
||||
# It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
|
||||
# not shared: frank-large's engine runs on the gfx1151 iGPU.
|
||||
LLM_SECONDARY_MODEL_FILE="${LLM_SECONDARY_MODEL_FILE:-/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
|
||||
LLM_SECONDARY_MODEL_FILE="${LLM_SECONDARY_MODEL_FILE:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}"
|
||||
LLM_SECONDARY_RUNTIME="${LLM_SECONDARY_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
|
||||
# The projector IS the vision capability here: the GGUF carries no vision
|
||||
# tensors, and the publisher ships its own BF16 projector in the same repo, so
|
||||
# a deploy that starts without this file silently ships a text-only
|
||||
# endpoint. Checked unconditionally, like frank-large's.
|
||||
LLM_SECONDARY_MMPROJ="${LLM_SECONDARY_MMPROJ:-/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf}"
|
||||
# The projector IS the vision capability here: the model is natively
|
||||
# multimodal and the Q6_K file carries no vision tensors — the publisher ships
|
||||
# its own BF16 projector in the same repo, so a deploy that starts without
|
||||
# this file silently ships a text-only endpoint. Checked unconditionally, like
|
||||
# frank-large's.
|
||||
LLM_SECONDARY_MMPROJ="${LLM_SECONDARY_MMPROJ:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}"
|
||||
|
||||
deploy_llm_secondary() {
|
||||
# Refuse rather than restart-loop: a unit whose runtime or payload is
|
||||
|
||||
@@ -1,27 +1,24 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download the frank-secondary payload for llama.cpp on the RTX 3090:
|
||||
# ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF:RCO-IQ3_XXS
|
||||
# **with the MTP head in the file** (the -mtp variant), plus the
|
||||
# PUBLISHER's OWN vision projector (mmproj-BF16.gguf, same repo). Public
|
||||
# repo — no auth header, and never a token on the command line (it is
|
||||
# world-readable in /proc/<pid>/cmdline).
|
||||
# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at **Q6_K** plus the
|
||||
# PUBLISHER's OWN vision projector (the BF16 mmproj — the model is natively
|
||||
# multimodal, so this file is the endpoint's eyes). Public repo — no auth
|
||||
# header, and never a token on the command line (it is world-readable in
|
||||
# /proc/<pid>/cmdline).
|
||||
#
|
||||
# Fresh downloads, no resume (`curl -C -` once produced a 213 MB oversized
|
||||
# file on a 31 GiB payload); -f/--fail-with-body so an HTTP error is not a
|
||||
# success, then byte count and sha256. This repo is on Xet storage, so
|
||||
# the Hub tree/blob APIs expose no LFS oid: the sha256 values below were
|
||||
# computed locally after the first verified fetch (byte counts match the
|
||||
# Hub's declared sizes). Idempotent: complete files are skipped. The
|
||||
# projector is stored under a self-documenting dest name; the URL name
|
||||
# and sha stay pinned in the table below.
|
||||
# success, then byte count and sha256. The Hub exposes no LFS oid for this
|
||||
# repo (Xet storage), so the sha256 values below were computed locally after
|
||||
# the first verified fetch — byte counts match the Hub's declared sizes.
|
||||
# A sha of "-" means "first fetch: size-verified, sha printed for pinning".
|
||||
# Idempotent: complete files are skipped.
|
||||
#
|
||||
# 2026-10-06 (Frank, evening): the payload moved to the UNCELERSED
|
||||
# Swift-1.5-Qwen3.8-27B at RCO-IQ3_XXS (1.56 GiB lighter than the
|
||||
# IQ3_S-mtp it replaced; the lighter quant funds the MTP draft ctx plus
|
||||
# the second 131,072 slot). Rollback payload stays on disk:
|
||||
# /home/dank/models/Swift15-IQ3S/ (the IQ3_S-mtp that was live until this
|
||||
# swap); earlier payloads Ornith-Q8/ and MiMo-Q6K/ as well. Their
|
||||
# re-fetch recipes are in git history (predecessors of this script).
|
||||
# 2026-10-07 (Frank): the payload moved to Qwen3.5-9B-Uncensored Q6_K at
|
||||
# 2 x 200,704 (qwen35 hybrid: 32 layers, 8 full-attention + 24 Gated
|
||||
# DeltaNet, no MTP tensors in this file). Rollback payloads stay on disk:
|
||||
# /home/dank/models/Gemma4-26B-A4B/ (Q4_K_M, the 2026-10-07 era being
|
||||
# replaced), Swift15-UncIQ3XXS/, MiMo-Q6K/, MiMo-Q5KM/, Ornith-Q8/.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/download-frank-secondary.sh [DEST_DIR] [--verify-only]
|
||||
@@ -35,15 +32,14 @@ for arg in "$@"; do
|
||||
*) DEST="$arg" ;;
|
||||
esac
|
||||
done
|
||||
DEST="${DEST:-/home/dank/models/Swift15-UncIQ3XXS}"
|
||||
DEST="${DEST:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K}"
|
||||
|
||||
# repo | hub-name | dest-name | bytes | sha256 (computed locally; Xet repo)
|
||||
FILES=(
|
||||
"ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF|RCO/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf|Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf|10442827392|e13d3e7400b6c7f980ebd30c0edd6be43eb1865b50be8562a9db042706c7d803"
|
||||
"ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF|mmproj-BF16.gguf|mmproj-BF16.gguf|931145888|19acd7fcf4ee504328566eddf0f8e603ef853a205b4830209d2a4908213abdea"
|
||||
"HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive|Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf|Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf|7359259008|c0ba7beb68fd3fe47891bd549486d38dcf62d00817296ea314ad37017f5a4986"
|
||||
"HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive|mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf|mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf|921704448|05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934"
|
||||
)
|
||||
|
||||
mkdir -p "$DEST"
|
||||
status=0
|
||||
for entry in "${FILES[@]}"; do
|
||||
IFS='|' read -r repo hubname destname size sha <<<"$entry"
|
||||
@@ -51,7 +47,9 @@ for entry in "${FILES[@]}"; do
|
||||
f="$DEST/$destname"
|
||||
have=0
|
||||
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
|
||||
if sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||
if [[ "$sha" == "-" ]]; then
|
||||
have=1
|
||||
elif sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||
have=1
|
||||
fi
|
||||
fi
|
||||
@@ -73,6 +71,10 @@ for entry in "${FILES[@]}"; do
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
if [[ "$sha" == "-" ]]; then
|
||||
echo "VERIFIED $destname (size only) sha256=$(sha256sum "$f" | cut -d' ' -f1)"
|
||||
continue
|
||||
fi
|
||||
if ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||
echo "SHA FAIL $destname"
|
||||
status=1
|
||||
|
||||
+42
-32
@@ -1,46 +1,56 @@
|
||||
# frank-secondary — llama.cpp (master, CUDA sm_86) serving
|
||||
# HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP
|
||||
# (Q4_K_M) on the RTX 3090, :7771.
|
||||
# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090,
|
||||
# :7771.
|
||||
#
|
||||
# 2026-10-07 (Frank): swap from Swift-1.5-Qwen3.8-27B RCO-IQ3_XXS to
|
||||
# Gemma4-26B-A4B Q4_K_M, ONE 200,704 context, f16 K + q8_0 V,
|
||||
# MTP DISABLED (SPEC_TYPE=none), log verbosity 4.
|
||||
# 2026-10-07 (Frank): swap from Gemma4-26B-A4B QAT Q4_K_M (one 200,704 slot)
|
||||
# to Qwen3.5-9B Q6_K at TWO 200,704-token conversations, EXACT (f16) KV,
|
||||
# MTP disabled (SPEC_TYPE=none), log verbosity 4.
|
||||
#
|
||||
# Architecture (read from the file's own header at load): gemma4 MoE — 30
|
||||
# layers, 128 experts (8 active), sliding_window_pattern = 25 sliding
|
||||
# (window 1024, n_head_kv 8) + 5 full-attention (layers 5/11/17/23/29,
|
||||
# n_head_kv 2), head_dim 256, attention_k_eq_v, native 262,144 window.
|
||||
# Architecture — read from the GGUF's own header (HTTP range read, not the
|
||||
# model card): arch qwen35, 32 layers, dense 9B, full_attention_interval 4
|
||||
# ⇒ 8 full-attention layers (3,7,11,...,31) + 24 Gated DeltaNet linear layers,
|
||||
# 16 attention heads / 4 KV heads × 256 dim, ssm conv_kernel 4 / state_size
|
||||
# 128 / group_count 16 / time_step_rank 32 / inner_size 4096, native 262,144,
|
||||
# vocab 248,320, file_type 18 (Q6_K). Tensor-type histogram: Q6_K×250 +
|
||||
# F32×177 — standard types only, so the stock master build loads it (no
|
||||
# custom-quant lock-in like the ROCmFPX era). NO nextn/mtp tensors exist, so
|
||||
# in-file MTP is impossible here; SPEC_TYPE=none is forced by the payload.
|
||||
#
|
||||
# KV rate law (measured, exact): the cache bills the K+V pair TWICE under
|
||||
# attention_k_eq_v — 6.125 B/elem at f16 K + q8_0 V, 2.125 B/elem at
|
||||
# q8_0/q8_0. Full cache = 5 x 2 x 256 x rate x ctx; SWA cache = 25 x 1024 x
|
||||
# 8 x 256 x rate. There is no true fp8 KV dtype in this build; q8_0 is the
|
||||
# FP8 class.
|
||||
# KV rate law: 8 full layers × 4 KV heads × 256 dim = 8,192 elements per
|
||||
# token per cache. f16 K + f16 V = 32,768 B/token = 32 KiB/token, so the
|
||||
# 401,408-token pool is 12.25 GiB. The 24 DeltaNet layers cost a fixed
|
||||
# recurrent state per SLOT (not per token) — that is why two 200k slots fit
|
||||
# where the 26B MoE fit only one.
|
||||
#
|
||||
# MTP is OFF: SPEC_TYPE=none strips --spec-type AND --spec-draft-model
|
||||
# (spec flags move as a set). The assistant file
|
||||
# (mtp-gemma-4-26B-A4B-it.gguf, 0.252 GiB) stays on disk as the drafter
|
||||
# payload if MTP is ever re-enabled. Measured VRAM: MTP ON booted
|
||||
# 23,806 used / 319 free; MTP OFF boots 21,890 used / 2,235 free.
|
||||
# Feasibility comes from the predecessor's MEASURED boots, not a formula:
|
||||
# MiMo-V2.6-Distill-Qwen-9B (identical qwen35 geometry, 7,793,710,624 B) at
|
||||
# 2×200,704 f16 K + f16 V measured 21,276 used / 2,849 free at ubatch 1024
|
||||
# and 21,846 / 2,279 at ubatch 2048. This file is 415 MiB lighter in weights,
|
||||
# its BF16 mmproj is 3 MiB heavier ⇒ projected ~21,430 used / ~2,700 free.
|
||||
# The verbosity-4 boot ledger is the referee.
|
||||
#
|
||||
# Ubatch 2048 is the measured CUDA optimum (measured on the MiMo predecessor
|
||||
# at this exact arch + shape: +4.6 % prefill at 8.9k depth, decode flat, cost
|
||||
# 616 MiB). 1024 is the fallback rung if the boot lands under the 2 GiB floor.
|
||||
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
||||
MODEL_FILE=/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
|
||||
MMPROJ_FILE=/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
|
||||
MODEL_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf
|
||||
MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf
|
||||
ALIAS=frank-secondary
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7771
|
||||
CTX_SIZE=200704
|
||||
PARALLEL=1
|
||||
# CTX_SIZE is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704
|
||||
# per conversation. MUST stay divisible by PARALLEL (the seam enforces it) and
|
||||
# the harness contextWindow quotes the PER-SLOT number — a longer prompt is
|
||||
# answered with a 400, never truncated.
|
||||
CTX_SIZE=401408
|
||||
PARALLEL=2
|
||||
BATCH_SIZE=4096
|
||||
UBATCH_SIZE=1024
|
||||
UBATCH_SIZE=2048
|
||||
CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=q8_0
|
||||
CACHE_TYPE_V=f16
|
||||
SPEC_TYPE=none
|
||||
NGL=99
|
||||
|
||||
# 2026-10-07 (Frank): verbosity 4 (debug — KV layout, buffers, graph
|
||||
# capture), ubatch 1024, K raised to f16 (V stays q8_0). MEASURED BOOT of
|
||||
# this exact shape: 21,868 MiB used / 2,257 MiB free — KV 3,001.25 (full) +
|
||||
# 306.25 (SWA) MiB (+2,161 over q8_0 K), compute buffers 926.39 + 150.63
|
||||
# (the 2048 rung cost ~616 MiB more, which is what funded the raise). The
|
||||
# q8_0-K/ubatch-2048 predecessor booted 21,890 / 2,235.
|
||||
# Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes
|
||||
# and graph capture land in the journal, which is what the VRAM-fit forensics
|
||||
# read. Levels above 4 flood the journal.
|
||||
VERBOSITY=4
|
||||
|
||||
@@ -1,27 +1,27 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve HauhauCS/Gemma4-26B-A4B-QAT-Uncensored (QAT **Q4_K_M**) as
|
||||
# Serve HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive (Q6_K) as
|
||||
# frank-secondary on Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
|
||||
#
|
||||
# 2026-10-07 (Frank): Gemma4-26B-A4B swap with MTP OFF and ONE 200,704-token
|
||||
# conversation (--parallel 1) — the 24 GiB card cannot fund a second slot at
|
||||
# this weight, and the MTP assistant's KV (built against the target's full
|
||||
# window, ~1.9 GiB measured) plus the f16-K raise do not coexist on it.
|
||||
# Verbosity 4 ships so the VRAM-fit forensics land in the journal.
|
||||
# 2026-10-07 (Frank): Qwen3.5-9B swap at TWO 200,704-token conversations
|
||||
# (--ctx-size 401408 --parallel 2) with EXACT f16 KV — the 9B rung frees the
|
||||
# room the 26B MoE Gemma4 shape lacked, so quantized V retires. MTP stays off:
|
||||
# this GGUF carries no MTP tensors at all (verified from its tensor-name
|
||||
# list), so there is no in-file head to draft with. Verbosity 4 ships so the
|
||||
# VRAM-fit forensics (KV layout, buffer sizes, graph capture) land in the journal.
|
||||
#
|
||||
# Architecture: gemma4_26b_a4b — 30-layer MoE (active ~4B), hybrid attention:
|
||||
# 25 sliding-window layers (fixed 1024-token ring each) + 5 full-attention
|
||||
# layers (5, 11, 17, 23, 29), 8 KV heads x 256 dim, n_ctx_train 262,144. KV
|
||||
# therefore scales ONLY with the 5 full layers: ~30.7 KiB/token at f16 K +
|
||||
# q8_0 V (~6.2 GiB at 200,704). attention_k_eq_v is declared by the file and
|
||||
# honoured by the build (bill K-side accounting; the measured boot is the
|
||||
# referee: 21,890 used / 2,235 free at q8_0/q8_0, MTP off).
|
||||
# Vision comes from the projector file (the GGUF carries no vision
|
||||
# tensors; the publisher ships its own BF16 mmproj in the same repo), so
|
||||
# --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is the
|
||||
# documented text-only opt-out.
|
||||
# Architecture: qwen35 — dense 9B, 32 layers with full_attention_interval 4
|
||||
# (8 full-attention layers at 3,7,...,31 + 24 Gated DeltaNet linear layers),
|
||||
# 16 heads / 4 KV heads × 256 dim, native 262,144, vocab 248,320. The KV pool
|
||||
# scales with the 8 full layers: 32 KiB/token at f16 K + f16 V (12.25 GiB at
|
||||
# the 401,408 pool); the DeltaNet layers cost a fixed recurrent state per
|
||||
# slot. Feasibility is priced from the MiMo-V2.6-Distill-Qwen-9B boots (same
|
||||
# geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279
|
||||
# at ubatch 2048 — the boot ledger, not arithmetic, is the referee.
|
||||
# Vision comes from the publisher's own BF16 projector (the model is natively
|
||||
# multimodal), so --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is
|
||||
# the documented text-only opt-out.
|
||||
#
|
||||
# Sampling: llama.cpp takes it per request (the fleet harness sends its
|
||||
# recipe from models.yml); greedy is the harness default.
|
||||
# Sampling: llama.cpp takes it per request (the fleet harness sends the
|
||||
# publisher recipe from models.yml: thinking temp 0.6 / top_p 0.95 / top_k 20).
|
||||
#
|
||||
# Tunables come from frank-secondary.env unless FRANK_SECONDARY_ENV_FILE overrides.
|
||||
set -euo pipefail
|
||||
@@ -37,32 +37,35 @@ fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -------------------
|
||||
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
|
||||
: "${MODEL_FILE:=/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
|
||||
: "${MODEL_FILE:=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}"
|
||||
: "${ALIAS:=frank-secondary}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7771}"
|
||||
# ctx-size is the POOL llama.cpp divides across slots; with --parallel 1 the
|
||||
# pool IS the conversation window. 200,704 is inside the model's trained
|
||||
# 262,144; the harness contextWindow MUST quote this number — a longer prompt
|
||||
# is answered with a 400, never truncated.
|
||||
: "${CTX_SIZE:=200704}"
|
||||
: "${PARALLEL:=1}"
|
||||
# ctx-size is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704
|
||||
# per conversation, inside the model's trained 262,144. The harness
|
||||
# contextWindow MUST quote the PER-SLOT number — a longer prompt is answered
|
||||
# with a 400, never truncated.
|
||||
: "${CTX_SIZE:=401408}"
|
||||
: "${PARALLEL:=2}"
|
||||
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves).
|
||||
# 2026-10-07 (Frank): 1024 — cut from 2048 to fund the f16-K raise. The
|
||||
# logical batch is VRAM- and throughput-neutral at a fixed ubatch.
|
||||
# 2048 is the measured CUDA optimum (MiMo era, same arch/shape: +4.6 % prefill
|
||||
# at 8.9k depth, decode flat, 616 MiB). The logical batch is VRAM- and
|
||||
# throughput-neutral at a fixed ubatch.
|
||||
: "${BATCH_SIZE:=4096}"
|
||||
: "${UBATCH_SIZE:=1024}"
|
||||
# KV at FP8 class = q8_0 on llama.cpp (1.0625 B/elem => 8.5 effective bits;
|
||||
# there is no true fp8 KV dtype). Explicit either way so the contract is
|
||||
# visible in the argv.
|
||||
: "${CACHE_TYPE_K:=q8_0}"
|
||||
: "${CACHE_TYPE_V:=q8_0}"
|
||||
# Speculative decoding is OFF by default since the Gemma4 swap (Frank,
|
||||
# 2026-10-07): Gemma4's MTP is a SEPARATE assistant file (arch
|
||||
# gemma4-assistant), not an in-file head, and its KV is built against the
|
||||
# target's full window — ~1.9 GiB measured, which the f16-K raise cannot
|
||||
# afford. SPEC_TYPE=none strips BOTH speculative flags (flags move as a
|
||||
# set); naming a drafter requires SPEC_DRAFT_FILE alongside SPEC_TYPE.
|
||||
: "${UBATCH_SIZE:=2048}"
|
||||
# Exact (unquantized) KV: f16 K + f16 V, restoring the MiMo-era contract that
|
||||
# the 26B Gemma4 shape could not afford. 32 KiB/token at this geometry
|
||||
# (8 full-attention layers × 4 KV heads × 256 dim); 12.25 GiB at the 401,408
|
||||
# pool. Explicit either way so the contract is visible in the argv, and
|
||||
# --flash-attn stays required for any quantized-V fallback (q8_0 = llama.cpp's
|
||||
# FP8 class; no true fp8 KV dtype exists in this build).
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
# Speculative decoding is OFF: this Q6_K file carries NO nextn/mtp tensors, so
|
||||
# there is no in-file head (the law: MTP is a property of the file, not of the
|
||||
# flags). SPEC_TYPE=none strips BOTH speculative flags (flags move as a set);
|
||||
# naming a drafter requires SPEC_DRAFT_FILE alongside SPEC_TYPE — the Gemma4
|
||||
# assistant on disk is arch gemma4-assistant and cannot drafter-match qwen35.
|
||||
: "${SPEC_TYPE:=none}"
|
||||
: "${SPEC_DRAFT_N_MAX:=3}"
|
||||
: "${SPEC_DRAFT_FILE:=}"
|
||||
@@ -73,7 +76,7 @@ fi
|
||||
: "${VERBOSITY:=3}"
|
||||
# Vision projector. No colon on purpose: an explicitly empty value means
|
||||
# "text-only, on purpose" and must NOT be replaced by the default path.
|
||||
: "${MMPROJ_FILE=/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf}"
|
||||
: "${MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}"
|
||||
|
||||
# Fail loudly rather than half-loading: a missing file is a download that
|
||||
# has not finished, and Restart=always would turn it into a crash loop that
|
||||
|
||||
Reference in New Issue
Block a user