frank-small: Gemma4-12B Q4_K_M, MTP sidecar + ngram-mod, q8 KV, --fit ctx
The Qwen3.8-27B IQ3_S file carried its own MTP head; Gemma4's ships as a separate assistant file (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers) that loads via --model-draft and shares the target's last layers' KV. The draftless ngram-mod matcher chains behind it in the same --spec-type value — two context-backed drafters cannot share one process, so this chain is the only shape that carries both. KV is q8_0 (16 KiB/token/layer vs 32 at f16) and CTX_SIZE is unset so --fit sizes the pool against what the 7.4 GiB of payloads leave on the 24 GiB card. build-cuda was rebuilt with -DGGML_CUDA_FA_ALL_QUANTS=ON, which quantized KV on CUDA requires. The three seams that pinned the Qwen contract (MTP-in-file, the Qwen mmproj name, the f16 KV arithmetic) now pin the Gemma4 contract instead, and the deploy handler's presence check points at the new payload.
This commit is contained in:
1 parent
b6c083f522
commit
4b7de59dc8
4 files changed
+128
-122
No files matched your search
@@ -1084,12 +1084,12 @@ wait_for_llm_large_ready() {
|
||||
#
|
||||
# It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
|
||||
# not shared: frank-large's halogen engine runs on the gfx1151 iGPU.
|
||||
LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf}"
|
||||
LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/Gemma4-12B-QAT/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
|
||||
|
||||
deploy_llm_small() {
|
||||
if [[ ! -f "$LLM_SMALL_MODEL" ]]; then
|
||||
echo "ERROR: model absent at $LLM_SMALL_MODEL" >&2
|
||||
echo "frank-small: fetch ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF (the -mtp build plus the mmproj) first;" >&2
|
||||
echo "frank-small: fetch HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M, the mmproj and the MTP sidecar) first;" >&2
|
||||
echo "frank-small: refusing to start a unit that cannot load." >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
Reference in new issue
Block a user