Frank
5b7b3ccb90
- MODEL_FILE: Swift-1.5-Qwen3.8-27B RCO-IQ3_XXS -> Gemma4-26B-A4B Q4_K_M - CTX_SIZE: 229376 (2x114,688) -> 200704 (1x200,704) - PARALLEL: 2 -> 1 - CACHE_TYPE_K/V: q8_0 (unchanged, FP8 class) - SPEC_TYPE: draft-mtp (unchanged) - SPEC_DRAFT_FILE: new — separate MTP assistant file (mtp-gemma-4-26B-A4B-it.gguf) - serve script: add --spec-draft-model support for separate draft files
36 lines
1.4 KiB
Bash
36 lines
1.4 KiB
Bash
# frank-secondary — llama.cpp (master, CUDA sm_86) serving
|
|
# HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP
|
|
# (Q4_K_M) on the RTX 3090, :7771.
|
|
#
|
|
# 2026-10-07 (Frank): swap from Swift-1.5-Qwen3.8-27B RCO-IQ3_XXS to
|
|
# Gemma4-26B-A4B Q4_K_M, ONE 200,704 context, q8_0 KV (FP8 class),
|
|
# MTP via separate assistant file (--spec-draft-model).
|
|
#
|
|
# Architecture: gemma4 MoE — 30 layers, 128 experts (8 active),
|
|
# hybrid attention: 25 sliding_window + 5 full_attention
|
|
# (full at layers [5, 11, 17, 23, 29]), sliding_window 1024,
|
|
# 8 KV heads, head_dim 256, attention_k_eq_v (V = K),
|
|
# native 262,144 window.
|
|
#
|
|
# KV: q8_0 K + q8_0 V (FP8 class, 1.0625 B/elem). The build has no
|
|
# true fp8 KV dtype; q8_0 is the FP8 class.
|
|
#
|
|
# MTP: separate assistant file (mtp-gemma-4-26B-A4B-it.gguf, 0.252 GiB),
|
|
# loaded via --spec-draft-model + --spec-type draft-mtp.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
|
MODEL_FILE=/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
|
|
MMPROJ_FILE=/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
|
|
ALIAS=frank-secondary
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
CTX_SIZE=200704
|
|
PARALLEL=1
|
|
BATCH_SIZE=4096
|
|
UBATCH_SIZE=2048
|
|
CACHE_TYPE_K=q8_0
|
|
CACHE_TYPE_V=q8_0
|
|
SPEC_TYPE=draft-mtp
|
|
SPEC_DRAFT_N_MAX=3
|
|
SPEC_DRAFT_FILE=/home/dank/models/Gemma4-26B-A4B/mtp-gemma-4-26B-A4B-it.gguf
|
|
NGL=99
|