frank-default-small: use q8_0 (not q8_KV) KV cache on ik_llama
The ik_llama fast q8_KV cache type crashes at model init on this QWEN3MOE/MTP + multimodal arch at commit b166e269 with a ggml_view size assert (ggml.c:5427) right after the output buffer is created, looping the service. Empirically isolated by toggling cache type: q8_0 and f16 load and serve (smoke-tested correct output), q8_KV asserts. Switch to q8_0 + Hadamard (K/V), which still gives the 8-bit cache + Hadamard quality win. KB updated with the crash lesson.
This commit is contained in:
1 parent
e2085f1f7d
commit
0e5c37f2f8
5 files changed
+32
-17
No files matched your search
@@ -7,11 +7,14 @@
|
||||
# 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF;
|
||||
# this service uses ngram-mod instead of MTP speculation.
|
||||
#
|
||||
# KV cache: q8_KV with Hadamard transforms on K and V (requires a build with
|
||||
# KV cache: q8_0 with Hadamard transforms on K and V (requires a build with
|
||||
# GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env file).
|
||||
# The ik_llama fast q8_KV type crashes at init on this arch (ggml.c:5427);
|
||||
# q8_0 is used instead.
|
||||
#
|
||||
# Split mode is none: Dankserver has a single RTX 3090 and ik_llama rejects
|
||||
# split-mode graph with fewer than 2 devices (hard load error).
|
||||
|
||||
#
|
||||
# Expert offload: the expert (FFN exps) tensors selected by the layer-bounded
|
||||
# --override-tensor pattern are pinned to CPU. Selected full-attention and
|
||||
|
||||
Reference in new issue
Block a user