Frank
68a525740d
Payload: HauhauCS/Gemma4-26B-A4B-QAT-Uncensored Q4_K_M (16,796,015,520 B) + the publisher's BF16 mmproj, /home/dank/models/Gemma4-26B-A4B/. Shape: ONE 200,704-token slot (--parallel 1; 2x200,704 does not fit the 24 GiB card), f16 K + q8_0 V under flash attention, batch 4096 / ubatch 1024, --log-verbosity 4, MTP OFF (the gemma4-assistant file is on disk, not loaded: its draft KV is built against the target's full window, measured +1.9 GiB -> 23,806/319 free, no margin beside the f16-K raise). Measured boots (nvidia-smi): q8_0 K + q8_0 V / ubatch 2048 / MTP off = 21,890 used / 2,235 free; with the MTP assistant = 23,806 / 319. Engine facts recorded: gemma4 hybrid attention (25 sliding-window + 5 full-attention layers) so KV scales with 5 layers only, and attention_k_eq_v is honoured by build 5fc4f3c (pool bills K-only; the measured q8_0 pool = 2.05 GiB = the K-only figure). Serve script gained VERBOSITY (+ --log-verbosity in the argv), the SPEC_DRAFT_FILE drafter path, and defaults/lore retargeted to Gemma4. deploy-remote.sh payload gates follow the env. The frank-secondary bats seam is rewritten for the new contract (argv-based) and fixes a pre-existing vacuity: serve_argv regenerates the test env, so mutations made before it were silently undone - tests that mutate now exec through run_stage. frank-secondary is the fleet compaction hub (200,704 is the only window that fits a worst-case compaction input).
44 lines
1.8 KiB
Bash
44 lines
1.8 KiB
Bash
# frank-secondary — llama.cpp (master, CUDA sm_86) serving
|
|
# HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP
|
|
# (Q4_K_M) on the RTX 3090, :7771.
|
|
#
|
|
# 2026-10-07 (Frank): swap from Swift-1.5-Qwen3.8-27B RCO-IQ3_XXS to
|
|
# Gemma4-26B-A4B Q4_K_M, ONE 200,704 context, q8_0 KV (FP8 class),
|
|
# MTP DISABLED (SPEC_TYPE=none).
|
|
#
|
|
# Architecture: gemma4 MoE — 30 layers, 128 experts (8 active),
|
|
# hybrid attention: 25 sliding_window + 5 full_attention
|
|
# (full at layers [5, 11, 17, 23, 29]), sliding_window 1024,
|
|
# 8 KV heads, head_dim 256, attention_k_eq_v (V = K),
|
|
# native 262,144 window.
|
|
#
|
|
# KV: q8_0 K + q8_0 V (FP8 class, 1.0625 B/elem). The build has no
|
|
# true fp8 KV dtype; q8_0 is the FP8 class.
|
|
#
|
|
# MTP is OFF: SPEC_TYPE=none strips --spec-type AND --spec-draft-model
|
|
# (spec flags move as a set). The assistant file
|
|
# (mtp-gemma-4-26B-A4B-it.gguf, 0.252 GiB) stays on disk as the drafter
|
|
# payload if MTP is ever re-enabled. Measured VRAM: MTP ON booted
|
|
# 23,806 used / 319 free; MTP OFF boots 21,890 used / 2,235 free.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
|
MODEL_FILE=/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
|
|
MMPROJ_FILE=/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
|
|
ALIAS=frank-secondary
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
CTX_SIZE=200704
|
|
PARALLEL=1
|
|
BATCH_SIZE=4096
|
|
UBATCH_SIZE=1024
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=q8_0
|
|
SPEC_TYPE=none
|
|
NGL=99
|
|
|
|
# 2026-10-07 (Frank): verbosity 4 (debug — KV layout, buffers, graph capture),
|
|
# ubatch 1024, K raised to f16 (V stays q8_0). f16 K costs +2.05 GiB over q8_0
|
|
# (5 full-attn layers x 200,704 tok x 8 heads x 256 dim x 2 B); the MTP-off
|
|
# boot had 2,235 MiB free, so this is a deliberate tight fit — read the boot
|
|
# ledger, not the arithmetic.
|
|
VERBOSITY=4
|