--ctx-size is the total pool divided among --parallel slots, so with
PARALLEL=3 the 160k target must be a 491520 pool (163840/slot) to
actually deliver 160k context per slot. Env, serve default, bats updated.
Reconfigure to 3 parallel 160000-ctx slots (was 1 slot). KV cache is
now asymmetric: K at q8_0, V at q5_0, both with Hadamard. The 3 slots
triple the KV, so expert blocks 27-34 are pinned back to CPU (freed
~8.7 GiB) leaving output-adjacent 35-40 on CUDA. Q8_0 model,
split-mode none, spec disabled (ik limits spec to 1 slot). AGENTS.md:
drop the scratch-port rule (test/bench processes cannot share the
single GPU with the running service anyway). Env, serve script, bats,
docs updated.
Switch to CRACK Q8_0 model (37.8 GB, mmproj f16). Servt as a single
160000-ctx slot (PARALLEL=1); speculative decoding stays disabled. KV
cache is q6_0 + Hadamard. iq5_k is not a valid KV-cache type in ik_llama
(cache whitelist excludes iq5_*); q8_KV reproduces the ggml.c:5427 view
assert even at PARALLEL=1/no-spec, so q6_0 is the verified working 6-bit
choice. Env, serve script, tests, and KB updated. Deployment was left
down after the isolation test; this commit redeploys it with the
verified Q8_0/160k/q6_0 config.
Fused MoE (fused_moe / fused_up_gate / fused_mmad) is on by default in
ik_llama - no flag needed; only -no-fmoe disables it. Slide two more
expert blocks (28,29) from CPU onto the RTX 3090 (~1.26 GiB) for the
fused-MoE run, leaving ~1.6 GiB headroom. Update override regex, test,
and KB (including a note that the headroom is now tighter than the
2-3 GiB ideal).
--jinja: ik_llama only renders a custom --chat-template-file as real Jinja
when --jinja is passed (inputs.use_jinja ? apply_jinja : non-jinja
fallback). Without it the Qwen-Sharp template's thinking/image/tool
formatting is silently dropped. Add USE_JINJA=1 -> --jinja.
Experts: the RTX 3090 had ~5.4 GiB headroom; pull expert blocks 30,32,33,34
back to CUDA (each block ~630 MiB) from CPU for lower latency. Update the
OVERRIDE_EXPERT regex, its test, and the KB with the headroom-check note.
ik_llama limits speculative decoding to a single server slot (-np 1) and
hard-exits at load if a spec stage is configured with --parallel > 1:
"Speculative decoding is currently limited to a single server slot
(-np 1)." This service keeps PARALLEL=3 (one slot per user), so set
SPEC_TYPE=none to disable speculation. It is a low-parameter MoE (3B
active of 35B), so throughput is fine without it. MTP stays disabled.
KB and tests updated.
Switch frank-default-small on Dankserver from upstream llama.cpp (b10121)
to ikawrakow/ik_llama.cpp (b166e269, built with the staged pip cu13
toolkit). The model (Ornith-1.5-35B-A3B CRACK Q6_K) is unchanged and the
old binary is kept for rollback.
Runtime changes:
- KV cache quantized to q8_KV with --k-cache-hadamard/--v-cache-hadamard
- --split-mode graph (single-GPU no-op, retained per operator pref)
- migrate ngram speculation to canonical
--spec-type ngram-mod:n_min=48,n_max=64,ngram_size_n=24
(legacy --spec-ngram-mod-* are rejected in ik main)
- N_GPU_LAYERS 999 (ik -ngl rejects the literal "all"); experts stay
pinned to CPU via --override-tensor
Build: GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON mandatory
for the quantized KV cache; recipe + reasoning baked into
docs/LLM/ikllama-ornith.md. Tests and both indices updated.
Caddy v2.11.4's caddyfile adapter rejects 'tls { insecure_skip_verify }'
inside transport http (unrecognized subdirective '{') and also rejects
the handler-level form. The working form, verified live against PMS with
a throwaway caddy (200 OK), is the bare token:
reverse_proxy 192.168.0.235:32400 {
transport http {
tls_insecure_skip_verify
}
}
The v6 restore commits dropped --chat-template-file and
--chat-template-kwargs while the artifact check still validated the
template file, so the Sharp template was checked but never applied.
Re-add both flags (with --jinja) so reasoning is emitted in DeepSeek
format and preserve_thinking/preserve_reasoning are honoured.
Reformat the script one-flag-per-line with long flag aliases
(--gpu-layers, --ctx-size, --ubatch-size, --cache-type-k/v) like
serve-frank-small, and update the 2 frank_large_config assertions
that grep the old short-flag forms (-ctk/-ctv, -ub).
- scripts/serve-frank-large.sh
- tests/frank_large_config.bats
frank-default-large now serves the kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT
model with a standalone MTP draft head (Q4_0), f16 K + TurboQuant V KV cache,
and the Sharp chat template. Context is maxed to 262144 (one slot); reasoning
is on with preserve_thinking + preserve_reasoning in the chat kwargs.
- frank-default-large.env: Q8_0 AGENT model paths, CTX_SIZE=262144, REASONING=on,
REASONING_FORMAT=none, CHAT_TEMPLATE_KWARGS {enable_thinking, preserve_thinking,
preserve_reasoning}
- serve-frank-large.sh: standalone draft-mtp profile (already switched off the
Kairic/PromptForge path)
- download-q8agent.sh: reproducible HF fetch of the three payload files
- docs/LLM/kingjones-q8agent-rocmfpx.md: Q8_0 AGENT runbook
- AGENTS.md: topology + ACTIVE runtime section + TTM/GTT load note
- qwen3.8-strix-optimizations, Dankclaw Improvements: TTM pages_limit now 114 GiB
(applied 2026-08-25), full 262k KV + Q8 payload fit in one slot
- frank_large_config.bats: assert 262144 context, reasoning + preserve_* enabled
The TTM/GTT was raised to 114 GiB (29,883,536 pages) to hold the full 262k KV.
Replace the retired kingjones ROCmFP4 STRIX MTP payload on Dankclaw with
cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF (eemin namespace, public), served by the
charlie12345/ROCmFPX llama.cpp build (ROCmI4 W4A4, c49ebdbd) on the Strix Halo
iGPU. Q4_0_ROCMI4 with an embedded NextN/MTP head (no separate draft file);
strict-Qwen MTP n16/p0.60/backend sampling on ROCm0. Binary
/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server; model dir
/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP. Updated serve script + env,
ubatch bench, large-config bats test, docs, AGENTS.md.
FTL's generated dnsmasq.conf has no conf-dir: /etc/dnsmasq.d/ is never
read (the 2026-08-24 scratch-name test proved the file was inert; names
only resolved from the query cache). The only file-based local-record
mechanism is hostsdir=/etc/pihole/hosts. Mount the record file there.
Doc: record the pitfall in Pi-hole.md.
The other vhosts (gitea, grafana, search, ...) now use username 'dankuser'
with CADDY_BASICAUTH_HASH, which is set to the same hash as the homepage
(dankpassword.123). So the whole protected surface logs in as
dankuser / dankpassword.123; the operator /admin/ page still uses frank.
Updated the two tests that asserted the old 'dank' username.
The homepage user page + shared data now use a dedicated CADDY_HOMEPAGE_HASH
(username dankuser), separate from the shared family password (CADDY_BASICAUTH_HASH,
username dank, still used by the other vhosts) and the operator password
(CADDY_ADMIN_HASH, username frank, the /admin/ page). This lets the homepage be
re-logged independently without touching the other sites. deploy.sh renders the
new placeholder; the .env.example documents all three hashes.
Caddy SORTS top-level directives, so a 'rewrite' runs before 'basicauth'.
The rewrite changed /admin/ to /admin.html, and the path-based auth matchers
then saw /admin.html (which matched the family 'not path /admin/*' block, not
the operator '/admin/*' block) — so the operator password was checked against
the family hash and 401'd.
Serve each page from its own 'handle' block (which preserves order, so the
auth is checked on the original path before any file rewrite):
- handle /admin* : operator (frank) only, try_files /admin.html
- handle @data : shared data + assets, family (dank) OR operator (frank), so
the admin page's fetches of services.json/status.json reuse the operator
credential
- handle : everything else (the user page), family (dank)
'@admin rewrite ...' is parsed as a one-liner matcher DEFINITION, so it
re-defined @admin and Caddy rejected the config ('matcher is defined more
than once'), crash-looping the container. Per the Caddyfile matcher docs the
matcher token is the first ARGUMENT after the directive: 'rewrite @admin
/admin.html'. The '*' disambiguator is only needed pre-2.8.0. Validated the
rendered config against the porkbun image: 'Valid configuration'.
Frank