A bare compose container dies silently at reboot (rootless Podman has no
boot-time auto-start; 'restart: unless-stopped' does not survive shutdown's
explicit stop). Rule established by the 2026-08-26 Dankclaw reboot, which
left Caddy down and Pi-hole crash-looping on a sysctl conflict.
Reconfigure to 3 parallel 160000-ctx slots (was 1 slot). KV cache is
now asymmetric: K at q8_0, V at q5_0, both with Hadamard. The 3 slots
triple the KV, so expert blocks 27-34 are pinned back to CPU (freed
~8.7 GiB) leaving output-adjacent 35-40 on CUDA. Q8_0 model,
split-mode none, spec disabled (ik limits spec to 1 slot). AGENTS.md:
drop the scratch-port rule (test/bench processes cannot share the
single GPU with the running service anyway). Env, serve script, bats,
docs updated.
- Place ik's Trellis (IQ*_KT), IQK (IQ*_K), MXFP4, and dedicated fast KV type
Q8_KV in a single high->low bits-per-value table with ROCmFPX TurboQuant KV.
- Mark mainline-provided types (*), KV-native vs weight-family-reused-for-KV.
- Hadamard annotated as a transform on top of a base quant, not a row.
- bpw read from exact ggml block structs (QK_K=256, fp16 scale): type_size x 8 / QK,
verified against the struct comment "6.5625 bits per weight" and the trait table.
Notes where legacy nominal figures (4.0/4.25) diverge.
- AGENTS.md: add benchmark directive (use llama-bench / vllm bench; ask before
rolling your own).
Fan out subagents to compare mainline llama.cpp, charlie12345/ROCmFPX,
and ik_llama.cpp, then add their findings to the project knowledge base:
- docs/Research/Llama Cpp Runtimes.md (positioning + quantization + server fit)
- docs/Research/Llama Cpp Code Diff.md (code-level quant diff, table, recommendations)
- docs/Research/SGLang.md (what SGLang is, comparison, our-boxes fit)
- docs/Research/index.md (Research progressive-disclosure index)
Cross-linked from docs/index.md (root) and docs/LLM/index.md.
AGENTS.md:
- Add Documentation Authoring section: docs/ is an OKF bundle; point doc authors
at the canonical spec in the system-config knowledge base
(~/.pi/agent/docs/references/okf-spec.md), not the orphaned
docs/references/okf-spec.md; consolidate read-index/one-concept-per-file/
progressive-disclosure/citation rules.
- Map AGENTS.md to docs_completeness.bats in the focused pre-commit hook (it was
last committed before the hook existed and had no mapping).
frank-default-large now serves the kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT
model with a standalone MTP draft head (Q4_0), f16 K + TurboQuant V KV cache,
and the Sharp chat template. Context is maxed to 262144 (one slot); reasoning
is on with preserve_thinking + preserve_reasoning in the chat kwargs.
- frank-default-large.env: Q8_0 AGENT model paths, CTX_SIZE=262144, REASONING=on,
REASONING_FORMAT=none, CHAT_TEMPLATE_KWARGS {enable_thinking, preserve_thinking,
preserve_reasoning}
- serve-frank-large.sh: standalone draft-mtp profile (already switched off the
Kairic/PromptForge path)
- download-q8agent.sh: reproducible HF fetch of the three payload files
- docs/LLM/kingjones-q8agent-rocmfpx.md: Q8_0 AGENT runbook
- AGENTS.md: topology + ACTIVE runtime section + TTM/GTT load note
- qwen3.8-strix-optimizations, Dankclaw Improvements: TTM pages_limit now 114 GiB
(applied 2026-08-25), full 262k KV + Q8 payload fit in one slot
- frank_large_config.bats: assert 262144 context, reasoning + preserve_* enabled
The TTM/GTT was raised to 114 GiB (29,883,536 pages) to hold the full 262k KV.
Replace the retired kingjones ROCmFP4 STRIX MTP payload on Dankclaw with
cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF (eemin namespace, public), served by the
charlie12345/ROCmFPX llama.cpp build (ROCmI4 W4A4, c49ebdbd) on the Strix Halo
iGPU. Q4_0_ROCMI4 with an embedded NextN/MTP head (no separate draft file);
strict-Qwen MTP n16/p0.60/backend sampling on ROCm0. Binary
/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server; model dir
/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP. Updated serve script + env,
ubatch bench, large-config bats test, docs, AGENTS.md.
Operator request (2026-08-22): the primary inference server
(frank-default-large, the 27B CIRU model on Dankclaw) moves to the
standard llama.cpp port 7770. Both LLM services now listen on :7770,
distinguished by host (dankserver vs dankclaw).
Functional changes:
- scripts/frank-default-large.env: LISTEN_PORT 52625 -> 7770
- scripts/serve-frank-large.sh: default LISTEN_PORT 52625 -> 7770
- scripts/deploy.sh: LLM_HEALTH[dankclaw] -> :7770
- open-webui quadlet: OPENAI_API_BASE_URL -> 192.168.0.143:7770
(re-pointed so chat keeps working; data dir preserved)
- scripts/bench-fast-greedy.sh: PORT 52625 -> 7770
Tests updated to the new topology (both LLM endpoints on :7770 by host;
open-webui -> :7770). Docs + pi KB updated; dated historical runbooks
keep their point-in-time ports. models.json (pi client) re-pointed to
:7770 separately, effective on the next client restart.
One repo, one git protocol, one deploy script. Merged from the two
former AGENTS.md files:
- Overview: both workloads, both hosts, the 2026-08-22 merge note,
current LLM fleet topology (frank-default-small :7770 Dankserver,
frank-default-large :52625 Dankclaw)
- Git workflow + testing (serverAppConfigs protocol, test step kept)
- Deployment: the unified deploy.sh (stack + LLM per target)
- Secret rules: no passwords in the repo; FRANK_SUDO_PW covers
headless sudo (the live sudo password is NOT carried forward)
- Deletion & Trash Policy (from the vllmScripts 2026-08-22 commit)
- Benchmarking policy (stop the service before any bench)
- Pi model config, runtime & troubleshooting (v3 CIRU active,
Kairic/DFlash2/FastMTP retired), HIP env naming rule, crash-loop
procedure, operational quirks (passwordless sudo verified
2026-08-22), Dankclaw memory model
- README: fix broken 'Also See' placeholder URL -> live Gitea LAN URL
- index.md: expand vllmScripts See Also with the endpoints it serves
- OpenWebUI.md: :7770 backend is the llama.cpp PP head (split to the
3090), managed by vllmScripts; note the FLM NPU :52625 alternative;
fix the broken [[llama.cpp Server]] wikilink
- Gitea.md: hosted repositories table (both projects)
- Dankclaw Stack.md: host-level LLM endpoints in the service inventory
- AGENTS.md: runtime note — LLM endpoints live in vllmScripts, not here
Pre-existing failures (untracked ComfyUI WIP): tests 6, 8, 9, 10.
Docs seams green: ok 19 (compose service docs), ok 20 (index links).
Frank