--ctx-size is the total pool divided among --parallel slots, so with
PARALLEL=3 the 160k target must be a 491520 pool (163840/slot) to
actually deliver 160k context per slot. Env, serve default, bats updated.
Reconfigure to 3 parallel 160000-ctx slots (was 1 slot). KV cache is
now asymmetric: K at q8_0, V at q5_0, both with Hadamard. The 3 slots
triple the KV, so expert blocks 27-34 are pinned back to CPU (freed
~8.7 GiB) leaving output-adjacent 35-40 on CUDA. Q8_0 model,
split-mode none, spec disabled (ik limits spec to 1 slot). AGENTS.md:
drop the scratch-port rule (test/bench processes cannot share the
single GPU with the running service anyway). Env, serve script, bats,
docs updated.
Switch to CRACK Q8_0 model (37.8 GB, mmproj f16). Servt as a single
160000-ctx slot (PARALLEL=1); speculative decoding stays disabled. KV
cache is q6_0 + Hadamard. iq5_k is not a valid KV-cache type in ik_llama
(cache whitelist excludes iq5_*); q8_KV reproduces the ggml.c:5427 view
assert even at PARALLEL=1/no-spec, so q6_0 is the verified working 6-bit
choice. Env, serve script, tests, and KB updated. Deployment was left
down after the isolation test; this commit redeploys it with the
verified Q8_0/160k/q6_0 config.
Fused MoE (fused_moe / fused_up_gate / fused_mmad) is on by default in
ik_llama - no flag needed; only -no-fmoe disables it. Slide two more
expert blocks (28,29) from CPU onto the RTX 3090 (~1.26 GiB) for the
fused-MoE run, leaving ~1.6 GiB headroom. Update override regex, test,
and KB (including a note that the headroom is now tighter than the
2-3 GiB ideal).
--jinja: ik_llama only renders a custom --chat-template-file as real Jinja
when --jinja is passed (inputs.use_jinja ? apply_jinja : non-jinja
fallback). Without it the Qwen-Sharp template's thinking/image/tool
formatting is silently dropped. Add USE_JINJA=1 -> --jinja.
Experts: the RTX 3090 had ~5.4 GiB headroom; pull expert blocks 30,32,33,34
back to CUDA (each block ~630 MiB) from CPU for lower latency. Update the
OVERRIDE_EXPERT regex, its test, and the KB with the headroom-check note.
ik_llama limits speculative decoding to a single server slot (-np 1) and
hard-exits at load if a spec stage is configured with --parallel > 1:
"Speculative decoding is currently limited to a single server slot
(-np 1)." This service keeps PARALLEL=3 (one slot per user), so set
SPEC_TYPE=none to disable speculation. It is a low-parameter MoE (3B
active of 35B), so throughput is fine without it. MTP stays disabled.
KB and tests updated.
The ik_llama fast q8_KV cache type crashes at model init on this
QWEN3MOE/MTP + multimodal arch at commit b166e269 with a ggml_view size
assert (ggml.c:5427) right after the output buffer is created, looping
the service. Empirically isolated by toggling cache type: q8_0 and f16
load and serve (smoke-tested correct output), q8_KV asserts. Switch to
q8_0 + Hadamard (K/V), which still gives the 8-bit cache + Hadamard
quality win. KB updated with the crash lesson.
--split-mode graph requires >= 2 devices in ik_llama.cpp and errors hard
at model load on a single RTX 3090 ("It is not possible to use split mode
'graph' with less than 2 devices"), looping the service. Set SPLIT_MODE
=none (the default), which works with the --override-tensor CPU expert
offload. KB updated with the two-device requirement so we do not assume
graph degrades gracefully on one GPU.
Switch frank-default-small on Dankserver from upstream llama.cpp (b10121)
to ikawrakow/ik_llama.cpp (b166e269, built with the staged pip cu13
toolkit). The model (Ornith-1.5-35B-A3B CRACK Q6_K) is unchanged and the
old binary is kept for rollback.
Runtime changes:
- KV cache quantized to q8_KV with --k-cache-hadamard/--v-cache-hadamard
- --split-mode graph (single-GPU no-op, retained per operator pref)
- migrate ngram speculation to canonical
--spec-type ngram-mod:n_min=48,n_max=64,ngram_size_n=24
(legacy --spec-ngram-mod-* are rejected in ik main)
- N_GPU_LAYERS 999 (ik -ngl rejects the literal "all"); experts stay
pinned to CPU via --override-tensor
Build: GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON mandatory
for the quantized KV cache; recipe + reasoning baked into
docs/LLM/ikllama-ornith.md. Tests and both indices updated.
The v6 restore commits dropped --chat-template-file and
--chat-template-kwargs while the artifact check still validated the
template file, so the Sharp template was checked but never applied.
Re-add both flags (with --jinja) so reasoning is emitted in DeepSeek
format and preserve_thinking/preserve_reasoning are honoured.
Reformat the script one-flag-per-line with long flag aliases
(--gpu-layers, --ctx-size, --ubatch-size, --cache-type-k/v) like
serve-frank-small, and update the 2 frank_large_config assertions
that grep the old short-flag forms (-ctk/-ctv, -ub).
- scripts/serve-frank-large.sh
- tests/frank_large_config.bats
The pre-commit hook runs scripts/test-changed.sh, which requires a
focused-test mapping for every staged path. The quadlets were moved into
dankclaw/<svc>/quadlet/ but the mapping still only matched top-level
dankclaw/*.container, so any nested quadlet or the pihole addn-hosts/env
data files fell through to the catch-all 'die' and blocked commit.
Add the nested-quadlet mappings (pihole-specific first, then the
dankclaw/*/quadlet/* fallback) plus the pihole data-file paths.
Fan out subagents to compare mainline llama.cpp, charlie12345/ROCmFPX,
and ik_llama.cpp, then add their findings to the project knowledge base:
- docs/Research/Llama Cpp Runtimes.md (positioning + quantization + server fit)
- docs/Research/Llama Cpp Code Diff.md (code-level quant diff, table, recommendations)
- docs/Research/SGLang.md (what SGLang is, comparison, our-boxes fit)
- docs/Research/index.md (Research progressive-disclosure index)
Cross-linked from docs/index.md (root) and docs/LLM/index.md.
AGENTS.md:
- Add Documentation Authoring section: docs/ is an OKF bundle; point doc authors
at the canonical spec in the system-config knowledge base
(~/.pi/agent/docs/references/okf-spec.md), not the orphaned
docs/references/okf-spec.md; consolidate read-index/one-concept-per-file/
progressive-disclosure/citation rules.
- Map AGENTS.md to docs_completeness.bats in the focused pre-commit hook (it was
last committed before the hook existed and had no mapping).
frank-default-large now serves the kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT
model with a standalone MTP draft head (Q4_0), f16 K + TurboQuant V KV cache,
and the Sharp chat template. Context is maxed to 262144 (one slot); reasoning
is on with preserve_thinking + preserve_reasoning in the chat kwargs.
- frank-default-large.env: Q8_0 AGENT model paths, CTX_SIZE=262144, REASONING=on,
REASONING_FORMAT=none, CHAT_TEMPLATE_KWARGS {enable_thinking, preserve_thinking,
preserve_reasoning}
- serve-frank-large.sh: standalone draft-mtp profile (already switched off the
Kairic/PromptForge path)
- download-q8agent.sh: reproducible HF fetch of the three payload files
- docs/LLM/kingjones-q8agent-rocmfpx.md: Q8_0 AGENT runbook
- AGENTS.md: topology + ACTIVE runtime section + TTM/GTT load note
- qwen3.8-strix-optimizations, Dankclaw Improvements: TTM pages_limit now 114 GiB
(applied 2026-08-25), full 262k KV + Q8 payload fit in one slot
- frank_large_config.bats: assert 262144 context, reasoning + preserve_* enabled
The TTM/GTT was raised to 114 GiB (29,883,536 pages) to hold the full 262k KV.
Frank