frank-default-small: use q8_0 (not q8_KV) KV cache on ik_llama

The ik_llama fast q8_KV cache type crashes at model init on this
QWEN3MOE/MTP + multimodal arch at commit b166e269 with a ggml_view size
assert (ggml.c:5427) right after the output buffer is created, looping
the service. Empirically isolated by toggling cache type: q8_0 and f16
load and serve (smoke-tested correct output), q8_KV asserts. Switch to
q8_0 + Hadamard (K/V), which still gives the 8-bit cache + Hadamard
quality win. KB updated with the crash lesson.
This commit is contained in:
Frank committed 2026-08-26 03:23:18 -07:00
1 parent e2085f1f7d
commit 0e5c37f2f8
5 files changed
+32 -17

No files matched your search

+4 -1
View File
@@ -7,11 +7,14 @@
# 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF;
# this service uses ngram-mod instead of MTP speculation.
#
# KV cache: q8_KV with Hadamard transforms on K and V (requires a build with
# KV cache: q8_0 with Hadamard transforms on K and V (requires a build with
# GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env file).
# The ik_llama fast q8_KV type crashes at init on this arch (ggml.c:5427);
# q8_0 is used instead.
#
# Split mode is none: Dankserver has a single RTX 3090 and ik_llama rejects
# split-mode graph with fewer than 2 devices (hard load error).
#
# Expert offload: the expert (FFN exps) tensors selected by the layer-bounded
# --override-tensor pattern are pinned to CPU. Selected full-attention and