Replace the retired kingjones ROCmFP4 STRIX MTP payload on Dankclaw with
cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF (eemin namespace, public), served by the
charlie12345/ROCmFPX llama.cpp build (ROCmI4 W4A4, c49ebdbd) on the Strix Halo
iGPU. Q4_0_ROCMI4 with an embedded NextN/MTP head (no separate draft file);
strict-Qwen MTP n16/p0.60/backend sampling on ROCm0. Binary
/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server; model dir
/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP. Updated serve script + env,
ubatch bench, large-config bats test, docs, AGENTS.md.
3 slots x 229376 (262144 - 32768) = 688128 total context. Updates the
.env, models.json (contextWindow/maxTokens 229376 per slot), and the
index/deployment/ornith runbook references.
- frank-default-small (Dankserver): Ornith-1.5-35B-A3B-CRACK Q8_0, bundled
native draft-mtp (n_max 4), MoE expert FFN tensors pinned to CPU via an
explicit --override-tensor \.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU
(replaces --n-cpu-moe), Qwen-Sharp chat template.
- frank-default-large (Dankclaw): kingjones Qwen3.8-27B-Uncensored-Q6_0
ROCmFP4 AGENT dense GGUF, standalone mtp-Qwen3.8-27B-Q4_0 draft head
(--model-draft, draft-mtp n_max 4), Qwen-Sharp chat template. Drop the
retired PromptForge .pfs sidecar lanes.
- Track scripts/chat-template.jinja (v22.4.0 Sharp) and apply via
--chat-template-file on both services.
- Docs: new kingjones-strix-mtp and dealign-ornith-crack runbook pages;
update index, deployment, and both system specs; models.json names.
Smoke-tested both models on the real hardware on scratch ports before
commit: model loads, MTP draft attaches, Sharp template applies, chat
completion returns.
FTL's generated dnsmasq.conf has no conf-dir: /etc/dnsmasq.d/ is never
read (the 2026-08-24 scratch-name test proved the file was inert; names
only resolved from the query cache). The only file-based local-record
mechanism is hostsdir=/etc/pihole/hosts. Mount the record file there.
Doc: record the pitfall in Pi-hole.md.
Fanned out 5 read-only doc-auditor subagents (one per service group) that
fact-checked the knowledge base against the live system + repo config,
researched official setup docs online, and verified progress. Applied:
- All 34 broken links fixed (real pages / external URLs / correct relative
paths); 0 broken links remain. No .AGENTTRASH links (the 4 mentions are
accurate descriptive text, not links).
- Fact-check fixes: Readarr retired (May 2026), Radarr v4->v6, Calibre
9.2.1 live, Calibre-Web 0.6.25 live, Copyparty v1.20.21, LLM index
ncmoe 38 + 786432 ctx, frank-default-small active (not frank-pp-worker),
Hermes/SearXNG/Grafana hostnames, Loki/Alloy local images, Observability
diagram (Promtail->Alloy), VLLM retired note, DFlash status, benchmark
live-drift notes, Homepage test count 116.
The homepage is live on Dankclaw. Update the Status (deployed) and the Auth
section (the handle-block design, the Caddyfile directive-sorting pitfall, the
host /srv/homepage mount, and the container-recreate note). Cross-reference the
agent KB Caddyfile syntax page.
- Homepage.md: status is now 'built, pending deploy' (all assets + checker +
deploy are in the repo, 112 tests green). To go live: add CADDY_ADMIN_HASH
to the target .env, then deploy.sh.
- Homepage Design Grilling.md: the build is complete (all 6 slices, full TDD);
the only remaining step is the operator adding the admin hash + deploying.
- pi log.md: build-complete entry.
- Homepage Design Grilling.md: final decision table (Q1-Q43) + the deltas where
the operator changed the recommendation (per-service probe method, 5s timeout,
TUI panels, toggleable styling, no cursor alteration, no auto-refresh, /docs/
path, user-facing docs, git-driven deploy, full TDD, last-deploy footer).
- Homepage.md: updated to the settled design (per-service probe, 5s, TUI panels,
styling toggle, no auto-refresh, /docs/ workstream, git-driven deploy, TDD).
- pi log.md: 2026-08-23 entry recording the settled design and build order.
Build order is unblocked. No homepage code built yet.
- Homepage.md: rewritten to the settled design (single site, user + /admin/
pages, server-side status.json via a 3-min systemd-timer checker, terminal
TUI theme). Added mermaid diagrams: site structure, status states, checker
flow. Recorded the classification (user vs admin; *arr excluded; LLM admin;
backends status-only) and the documented future shift to client-side probing.
- Homepage Design Grilling.md (new): the decision log — settled Q1-Q5 plus 38
open questions (Q6-Q43) across 10 categories, each with a recommendation.
- docs/index.md: link the grilling page (satisfies the index-links-every-page test).
- report/homepage-grilling-2026-08-22.html: self-contained terminal-TUI progress
report (settled decisions, verified stack baseline, overnight processing, work
threads, the open questions, and how to continue).
Baseline recorded: Dankclaw fully up (10/10); the Dankserver media stack is down
(only Calibre-Web + the LLM are up; no .env on the target) — a separate thread.
No homepage code built yet; the design is pending the operator's answers to Q6-Q43.
Operator request (2026-08-22): the primary inference server
(frank-default-large, the 27B CIRU model on Dankclaw) moves to the
standard llama.cpp port 7770. Both LLM services now listen on :7770,
distinguished by host (dankserver vs dankclaw).
Functional changes:
- scripts/frank-default-large.env: LISTEN_PORT 52625 -> 7770
- scripts/serve-frank-large.sh: default LISTEN_PORT 52625 -> 7770
- scripts/deploy.sh: LLM_HEALTH[dankclaw] -> :7770
- open-webui quadlet: OPENAI_API_BASE_URL -> 192.168.0.143:7770
(re-pointed so chat keeps working; data dir preserved)
- scripts/bench-fast-greedy.sh: PORT 52625 -> 7770
Tests updated to the new topology (both LLM endpoints on :7770 by host;
open-webui -> :7770). Docs + pi KB updated; dated historical runbooks
keep their point-in-time ports. models.json (pi client) re-pointed to
:7770 separately, effective on the next client restart.
- docs/index.md: LLM Fleet section (all 34 pages linked), merge note,
See Also now internal
- docs/LLM/index.md: retitled, dangling [[Dankserver]]/[[Dankclaw]]
links fixed, sibling section replaced with internal links
- docs/LLM/deployment.md: rewritten for the unified deploy flow,
manual-update paths point at the merged checkout, incident history
kept intact
- Deployment Flow / Dankclaw Stack / Gitea: stale vllmScripts
references and retired-topology notes updated to the 2026-08-22
state
- Historical docs (frank-redo, PP deployment, FastFlowLM) keep their
dated runbook paths as-is (records of retired topologies)
The LLM fleet documentation (deployment runbook, model runbooks,
benchmarks, research archive, retired-topology history) joins the
project knowledge base as docs/LLM/. Wikilinks resolve by filename,
so no link updates are needed for the move.
The bridge gateway (10.89.3.1) lives in the rootless netns; llama-server
runs in the host netns, so the gateway refuses. 192.168.0.143:52625
reaches it through the rootless NAT (verified from the container).
Same LAN-IP pattern the doc already prescribes for the Dankserver
backend.
The Caddyfile had 'admin off', which disables Caddy's config file
watcher. Rendered Caddyfile changes never loaded until the container
was recreated, so the double-hash basicauth fix sat on disk unapplied
across deploys (2026-08-22). Admin binds to 127.0.0.1:2019 inside the
container and the port is unpublished, so nothing outside reaches it.
deploy.sh now issues an explicit 'caddy reload' (retry loop) after
'up -d', with the file watcher as backstop.
basicauth (2026-08-22 incident):
- the Caddyfile template carried its own $2a$14$ prefix and deploy.sh
substituted the full hash (prefix included), rendering an invalid
double-prefixed hash. Every basicauth vhost rejected every password
with 401 and Caddy logged nothing.
- template is now a bare REPLACE_WITH_HASH token; tests pin the render
contract (no prefix in template; rendered line is exactly 'dank <hash>').
open-webui (2026-08-22 incident):
- its slirp4netns helper process died, leaving the container running
with only 'lo' in its netns: port 3005 accepted TCP and reset every
request, so chat served empty responses for ~2 days, unlogged.
- move to a pinned bridge network (systemd-open-webui, 10.89.3.0/24,
gw 10.89.3.1) — no per-container helper to die.
- LLM endpoint reached via the gateway (host.containers.internal is
slirp-only): OPENAI_API_BASE_URL=http://10.89.3.1:52625.
- add a quadlet healthcheck: /health (app) + non-loopback interface
(network liveness; the dead-tap state leaves only 'lo').
- deploy script now copies the .network file too.
- docs updated (OpenWebUI History section, Caddy basic-auth contract).
Add a hermes-dashboard compose service that runs 'hermes dashboard' on
0.0.0.0:9119 with HTTP basic auth. The username + scrypt password hash
live in the git-ignored /home/dank/hermes-data config (plaintext never
stored). A separate container from the Signal gateway, since the gateway
entrypoint does not start the dashboard. Both share the data volume and
are boot-persistent. Docs page + index link added (docs-first contract).
ComfyUI-Manager (V3.41) is installed in the mounted custom_nodes dir and
self-installs its pip deps on start. Document the install command, the
bind-mount-shadows-image gotcha, and the /manager/version health check.
Rewrite ComfyUI.md to match reality: comfyui-rocm:gfx1151 built from
Containerfile.comfyui on AMD's gfx1151 base, device passthrough, video group
only (render is unmappable in rootless), data under /home/dank/comfyui-data,
boot-persistent via a user systemd unit. Mark research 'way 1' as applied and
update the repo-consequence section (the CPU config is no longer the default).
The JSON rate cap (API_MAX=4/hour) is hardcoded in Python, not settable in
limiter.toml; pass_ip is the only bypass. Rootless podman strips the client IP
on published-port hits (the container sees a slirp source), so local subagents
must present a pass_ip source. Document the slirp IP-strip, the pass_ip
expansion, and the root-owned config file (one-off sudo cp to update).
Replace the dead dankserver open-webui (bare container, :3000, pointed at a
dead :9090 backend) with a boot-persistent quadlet on dankclaw:
- CPU image (:main); dankclaw is CPU-only, so the :cuda image is wasted.
- Isolated on slirp4netns; publishes 3005 (3000 is Gitea's HTTP port).
- Default LLM backend is the local vLLM on 52625 (frank-default-large).
- Data persists under /home/dank/Containers/open-webui/data.
- Caddy route now proxies to the published LAN port (the container is not on
Caddy's network).
Also corrects the model/port mapping in the docs: the live state (verified
2026-08-19) is frank-default-large on vLLM :52625 (dankclaw) and
frank-default-small on llama.cpp :7770 (dankserver) — the reverse of the
2026-08-17 note.
Update SearXNG, Gluetun, and Valkey to describe the quadlet units instead of
compose: searxng sets Network=gluetun.container, the units install to
~/.config/containers/systemd/, and scripts/deploy-searxng-quadlet.sh deploys
them. Add a Boot Persistence section to the Stack doc. Correct the Gluetun
tunnel check (the :8000 server 404s; verify tun0 instead) and note the pinned
10.89.2.0/24 subnet.
A direct hit on the published port has no X-Forwarded-For header, so the
bot-detection cannot resolve a client and returns 429. SearXNG must be
reached through Caddy (search.dankclaw.local), which sets X-Forwarded-For.
Add the podman bridge subnets to trusted_proxies so Caddy's source is
trusted for XFF extraction; note the CIDR is dynamic and should be pinned.
Verified agentic JSON search returns 26-31 relevant results via this path.
The trusted group lives in limiter.toml, not settings.yml. Add
shared/searxng/limiter.toml trusting 192.168.0.0/24 in pass_ip and
trusted_proxies, copy it into /opt/searxng/config on deploy, and correct
the SearXNG doc. New test seam: tests/searxng_limiter.bats.
- README: fix broken 'Also See' placeholder URL -> live Gitea LAN URL
- index.md: expand vllmScripts See Also with the endpoints it serves
- OpenWebUI.md: :7770 backend is the llama.cpp PP head (split to the
3090), managed by vllmScripts; note the FLM NPU :52625 alternative;
fix the broken [[llama.cpp Server]] wikilink
- Gitea.md: hosted repositories table (both projects)
- Dankclaw Stack.md: host-level LLM endpoints in the service inventory
- AGENTS.md: runtime note — LLM endpoints live in vllmScripts, not here
Pre-existing failures (untracked ComfyUI WIP): tests 6, 8, 9, 10.
Docs seams green: ok 19 (compose service docs), ok 20 (index links).
unattended-upgrades rebooted dankclaw at 15:01; the legacy
podman-compose gitea stack had no systemd units, so rootless podman
did not auto-start the containers. Gitea was down ~1 hour.
- add quadlet units (dankclaw/gitea/quadlet/): gitea-postgres,
gitea-server, gitea-caddy + gitea.network (DNS enabled; the default
rootless podman network has dns_enabled=false)
- add gitea stack Caddyfile (upstream gitea-server:3000) and
.env.example; live password stays in /opt/gitea/.env (0600)
- fix latent name bugs in main stack config: DATABASE_HOST
gitea-postgres, Caddyfile upstream gitea-server
- docs: rewrite Gitea page for the quadlet stack (management, boot
persistence, secrets, gotchas, incident log); fix port 3000 in
stack table; Postgres connection notes
- legacy compose retired on target to /opt/gitea/.deprecated/
Live-verified on Dankclaw 2026-08-14:
- official rocm/comfyui image is built PYTORCH_ROCM_ARCH=gfx942;gfx950
(datacenter only) -> torch.cuda.is_available() False on gfx1151
- AMD gfx1151-tagged PyTorch build (rocm/vllm:...gfx1151...pytorch_2.9.1)
-> avail True, Radeon 8060S, cap (11,5); needs /dev/kfd + /dev/dri
- torch.backends.vulkan is compiled out of standard wheels; no ComfyUI
--vulkan flag exists; Vulkan is a dead end for ComfyUI
- Lemonade Server does image gen via wrapped stable-diffusion.cpp, whose
HIP kernels include gfx1151 -> works where ComfyUI's torch cannot
New pages: ComfyUI ROCm and Vulkan, Lemonade and sd.cpp; ComfyUI.md
qualified; compose stays --cpu until a gfx1151-enabled torch image is built
Frank