frank-default-large now serves the kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT
model with a standalone MTP draft head (Q4_0), f16 K + TurboQuant V KV cache,
and the Sharp chat template. Context is maxed to 262144 (one slot); reasoning
is on with preserve_thinking + preserve_reasoning in the chat kwargs.
- frank-default-large.env: Q8_0 AGENT model paths, CTX_SIZE=262144, REASONING=on,
REASONING_FORMAT=none, CHAT_TEMPLATE_KWARGS {enable_thinking, preserve_thinking,
preserve_reasoning}
- serve-frank-large.sh: standalone draft-mtp profile (already switched off the
Kairic/PromptForge path)
- download-q8agent.sh: reproducible HF fetch of the three payload files
- docs/LLM/kingjones-q8agent-rocmfpx.md: Q8_0 AGENT runbook
- AGENTS.md: topology + ACTIVE runtime section + TTM/GTT load note
- qwen3.8-strix-optimizations, Dankclaw Improvements: TTM pages_limit now 114 GiB
(applied 2026-08-25), full 262k KV + Q8 payload fit in one slot
- frank_large_config.bats: assert 262144 context, reasoning + preserve_* enabled
The TTM/GTT was raised to 114 GiB (29,883,536 pages) to hold the full 262k KV.
Replace the retired kingjones ROCmFP4 STRIX MTP payload on Dankclaw with
cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF (eemin namespace, public), served by the
charlie12345/ROCmFPX llama.cpp build (ROCmI4 W4A4, c49ebdbd) on the Strix Halo
iGPU. Q4_0_ROCMI4 with an embedded NextN/MTP head (no separate draft file);
strict-Qwen MTP n16/p0.60/backend sampling on ROCm0. Binary
/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server; model dir
/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP. Updated serve script + env,
ubatch bench, large-config bats test, docs, AGENTS.md.
FTL's generated dnsmasq.conf has no conf-dir: /etc/dnsmasq.d/ is never
read (the 2026-08-24 scratch-name test proved the file was inert; names
only resolved from the query cache). The only file-based local-record
mechanism is hostsdir=/etc/pihole/hosts. Mount the record file there.
Doc: record the pitfall in Pi-hole.md.
The other vhosts (gitea, grafana, search, ...) now use username 'dankuser'
with CADDY_BASICAUTH_HASH, which is set to the same hash as the homepage
(dankpassword.123). So the whole protected surface logs in as
dankuser / dankpassword.123; the operator /admin/ page still uses frank.
Updated the two tests that asserted the old 'dank' username.
The homepage user page + shared data now use a dedicated CADDY_HOMEPAGE_HASH
(username dankuser), separate from the shared family password (CADDY_BASICAUTH_HASH,
username dank, still used by the other vhosts) and the operator password
(CADDY_ADMIN_HASH, username frank, the /admin/ page). This lets the homepage be
re-logged independently without touching the other sites. deploy.sh renders the
new placeholder; the .env.example documents all three hashes.
Caddy SORTS top-level directives, so a 'rewrite' runs before 'basicauth'.
The rewrite changed /admin/ to /admin.html, and the path-based auth matchers
then saw /admin.html (which matched the family 'not path /admin/*' block, not
the operator '/admin/*' block) — so the operator password was checked against
the family hash and 401'd.
Serve each page from its own 'handle' block (which preserves order, so the
auth is checked on the original path before any file rewrite):
- handle /admin* : operator (frank) only, try_files /admin.html
- handle @data : shared data + assets, family (dank) OR operator (frank), so
the admin page's fetches of services.json/status.json reuse the operator
credential
- handle : everything else (the user page), family (dank)
'@admin rewrite ...' is parsed as a one-liner matcher DEFINITION, so it
re-defined @admin and Caddy rejected the config ('matcher is defined more
than once'), crash-looping the container. Per the Caddyfile matcher docs the
matcher token is the first ARGUMENT after the directive: 'rewrite @admin
/admin.html'. The '*' disambiguator is only needed pre-2.8.0. Validated the
rendered config against the porkbun image: 'Valid configuration'.
Two real bugs found on the first production deploy:
- /admin/ 404'd: file_server looked for /srv/homepage/admin/ (a dir), but the
page is admin.html. Add '@admin rewrite * /admin.html' so the /admin/ path
serves admin.html.
- The pages used relative refs (style.css, app.js, services.json, status.json),
which from /admin/ resolved to /admin/... (404). Add <base href="/"> so all
refs resolve to the site root on both pages.
- status.json was written 0600 (root); the checker now chmods it 0644 so Caddy
can serve it regardless of its user.
Red tests first.
- tests/homepage_deploy.bats: the deploy ships the homepage (assets to
/srv/homepage, the checker timer, both Caddy hashes, deployed.js) and is
git-driven (pulls main); the checker runs as a systemd timer.
- scripts/homepage-status.service / .timer: the checker as a oneshot service
+ a 3-minute timer (Q2/Q35).
- scripts/deploy.sh: the Caddyfile render now substitutes BOTH hashes
(REPLACE_WITH_HASH <- CADDY_BASICAUTH_HASH, REPLACE_WITH_ADMIN_HASH <-
CADDY_ADMIN_HASH). A new homepage step copies the assets to /srv/homepage,
generates deployed.js from the deployed commit's datetime (Q43), installs
the checker timer, and runs the checker once so status.json exists at once.
- tests/llm_units.bats: scoped the three LLM-unit tests to
frank-default-*.service (the homepage unit is a different pattern: it
installs to /usr/local/bin and runs from /srv at deploy time).
Red tests first.
- tests/homepage_caddy.bats: the homepage site block carries a root hash and
a distinct /admin/-scoped admin hash (Q18/Q38), and serves /srv/homepage.
- dankclaw/Caddyfile: the homepage block now uses named matchers -
@user (not /admin/*) with the shared hash, @admin (/admin/*) with the
operator-only hash (REPLACE_WITH_ADMIN_HASH, rendered from CADDY_ADMIN_HASH
at deploy). Both placeholders are distinct.
Red tests first, then the pages.
- tests/homepage_pages.bats: the pages reference the data + theme; no
auto-refresh (Q41); no cursor alteration (Q28); the styling toggle strips
to bare HTML (Q27); the footer shows the last master deployment datetime
(Q43).
- index.html / admin.html: the user and admin pages. A grid of TUI panels,
a summary line, the bare toggle, the deployed-at footer.
- app.js: reads status.json + services.json once (no auto-refresh), renders
one panel per service for the page's category, the toggle, the footer.
- style.css: the terminal TUI theme (green-on-black, phosphor palette,
scanlines) + a ?bare=1 mode that strips to unstyled HTML. No custom cursor.
Functionally verified with a Node DOM shim: 9 user panels, correct summary,
correct footer datetime.
Red tests first (the old services.json was an object, not the new list schema).
- tests/homepage_config.bats: the config is a JSON list; every entry has
name/category/probe; category is user|admin; method is tcp|http; a
showLink entry has a url and a status-only entry does not.
- shared/homepage/services.json: the real fleet. User: SearXNG, ComfyUI,
OpenWebUI, Gitea, Calibre-Web, RomM, Copyparty, Plex, Jellyfin. Admin:
Hermes, Grafana, Prometheus, Loki, node-exporter, alloy, Caddy, dnsmasq,
Gluetun (wip - a tunnel has no simple local port), the two LLMs, mariadb,
valkey, Postgres. Ports confirmed from the repo config (compose/quadlet).
docs fields are empty for now (the /docs/ workstream fills them as each
service goes active).
Red test first: a raw TCP listener that never speaks HTTP must read down
under an http probe (up under tcp), while a real HTTP server reads up. This
test caught a real hang — the backgrounded listener inherited the test
runner's stdout pipe and held it open; fixed by redirecting its output and
killing both listeners in teardown.
scripts/homepage-status.py: probe() now dispatches on method. http does an
urlopen (any HTTP status = up; no connection = down); tcp connects.
The checker is now complete: tcp + http + wip + atomic write + generated_at.
Remaining slices: services.json schema, the pages + theme + toggle, the
Caddyfile (validate + two hashes), the git-driven deploy.
Full TDD (Q36): red test first, then the minimal implementation.
- tests/homepage_checker.bats: seam = homepage-status.py <services.json>
<out.json>. Asserts an open port is up, a closed port is down, a wip
service is wip (not probed), and the output carries generated_at.
- scripts/homepage-status.py: reads the config, probes each backend (tcp,
5 s timeout), skips wip, writes status.json atomically (temp + rename).
Remaining slices: http probe method, services.json schema, the pages +
theme + toggle, the Caddyfile (validate + two hashes), the git-driven deploy.
Operator request (2026-08-22): the primary inference server
(frank-default-large, the 27B CIRU model on Dankclaw) moves to the
standard llama.cpp port 7770. Both LLM services now listen on :7770,
distinguished by host (dankserver vs dankclaw).
Functional changes:
- scripts/frank-default-large.env: LISTEN_PORT 52625 -> 7770
- scripts/serve-frank-large.sh: default LISTEN_PORT 52625 -> 7770
- scripts/deploy.sh: LLM_HEALTH[dankclaw] -> :7770
- open-webui quadlet: OPENAI_API_BASE_URL -> 192.168.0.143:7770
(re-pointed so chat keeps working; data dir preserved)
- scripts/bench-fast-greedy.sh: PORT 52625 -> 7770
Tests updated to the new topology (both LLM endpoints on :7770 by host;
open-webui -> :7770). Docs + pi KB updated; dated historical runbooks
keep their point-in-time ports. models.json (pi client) re-pointed to
:7770 separately, effective on the next client restart.
New seams (written first, red against the pre-merge tree):
- llm_units.bats: every scripts/*.service unit points at files that
exist in this repo (the merged checkout path), env files included
- llm_secrets.bats: the live sudo password is never tracked (the
vllmScripts AGENTS.md once documented it)
- deployment_flow.bats: deploy.sh must restart the LLM services on
both targets and health-check both endpoints (7770, 52625)
- helpers.bash: has_no_leaked_sudo_pw (two-part assembly, same
pattern as the gitea check)
The bridge gateway (10.89.3.1) lives in the rootless netns; llama-server
runs in the host netns, so the gateway refuses. 192.168.0.143:52625
reaches it through the rootless NAT (verified from the container).
Same LAN-IP pattern the doc already prescribes for the Dankserver
backend.
A 'caddy reload' delivered over the admin API cannot apply a config
that disables the admin API -- the fix cannot ride the API it is meant
to enable (2026-08-22: admin off sat in the running config across
deploys). Track the reload result; on failure, restart the container
so it reads the rendered file on start.
The Caddyfile had 'admin off', which disables Caddy's config file
watcher. Rendered Caddyfile changes never loaded until the container
was recreated, so the double-hash basicauth fix sat on disk unapplied
across deploys (2026-08-22). Admin binds to 127.0.0.1:2019 inside the
container and the port is unpublished, so nothing outside reaches it.
deploy.sh now issues an explicit 'caddy reload' (retry loop) after
'up -d', with the file watcher as backstop.
basicauth (2026-08-22 incident):
- the Caddyfile template carried its own $2a$14$ prefix and deploy.sh
substituted the full hash (prefix included), rendering an invalid
double-prefixed hash. Every basicauth vhost rejected every password
with 401 and Caddy logged nothing.
- template is now a bare REPLACE_WITH_HASH token; tests pin the render
contract (no prefix in template; rendered line is exactly 'dank <hash>').
open-webui (2026-08-22 incident):
- its slirp4netns helper process died, leaving the container running
with only 'lo' in its netns: port 3005 accepted TCP and reset every
request, so chat served empty responses for ~2 days, unlogged.
- move to a pinned bridge network (systemd-open-webui, 10.89.3.0/24,
gw 10.89.3.1) — no per-container helper to die.
- LLM endpoint reached via the gateway (host.containers.internal is
slirp-only): OPENAI_API_BASE_URL=http://10.89.3.1:52625.
- add a quadlet healthcheck: /health (app) + non-loopback interface
(network liveness; the dead-tap state leaves only 'lo').
- deploy script now copies the .network file too.
- docs updated (OpenWebUI History section, Caddy basic-auth contract).
- alpine's /bin/busybox is musl-dynamic and cannot execute in the
distroless loki image (missing loader); busybox-static links in
- alloy 1.x refuses to configure the ui service from a config block;
the default UI already listens on 127.0.0.1:12345 where the
healthcheck probes
- alloy 1.x renamed the server component to ui; the old block failed the
initial load and the container crash-looped
- grafana/loki is distroless and grafana/alloy lacks wget, so their
healthchecks could never run; local images add a static busybox from
alpine for that one command
- deploy.sh builds all three local images if missing
- tests: 3 new seams
Measured the rootless mapping: container root IS the invoking user
(dank, owner of the /opt data dirs); the images' default users (loki
10001, prometheus nobody, grafana 472) map to subuids that own nothing.
1000:1000 was the wrong direction.
Container root maps to a subuid that owns nothing under /opt, so the
data-dir writes fail. Container uid 1000 is the invoking user in
rootless podman. Fixes the awk service-block seam to use regex match.
- loki: recent releases require compactor.delete_request_store when
retention is enabled (was crash-looping on start)
- prometheus: scrape targets carried legacy dankclaw.local names with
URL schemes (invalid); point at published LAN ports and the gitea
quadlet's published port instead
- grafana: GF_SERVER_ROOT_URL to the canonical HTTPS name
- tests: 3 new seams for the deployed observability configs
- Caddyfile: drop http:// site prefixes, global acme_dns porkbun block,
HTTPS redirects on the bare-IP and legacy vhosts
- new dankclaw/caddy/Containerfile: xcaddy build with caddy-dns/porkbun
- compose: caddy uses franksdank/caddy-porkbun:latest + Porkbun env vars
- deploy.sh: build the image on first deploy if missing
- .env.example: rename PORKBUN_API_SECRET -> PORKBUN_API_SECRET_KEY
(the name the caddy-dns/porkbun module expects)
- tests: 7 new caddyfile/compose seams for the TLS config
Per user direction, Caddy runs only on dankclaw. It fronts every
service on both hosts: local services by container name, quadlet
services by published LAN port, dankserver services by published LAN
port (192.168.0.235:PORT).
- dankclaw/Caddyfile: merged vhost list (was: two Caddyfiles, one per
host). gitea's single vhost moves in from the retired gitea-caddy
quadlet (option B). Upstreams fixed to live reality: gitea :3000,
search :8081 (gluetun), chat :3005 (open-webui quadlet), media
services via 192.168.0.235. Retired: llms (open-webui on dankserver
is removed; chat is its successor), the dankserver homepage (302 to
the shared one).
- dankserver/Caddyfile, dankclaw/gitea/Caddyfile: deleted (git rm).
- dnsmasq addn-hosts: every name -> 192.168.0.143 (the one Caddy
host); no .235 records, no llms.
- compose.yaml: removed the openwebui/postgres/gitea services (they
are quadlets now); caddy service mounts the rendered Caddyfile from
/home/dank/Containers/caddy, healthcheck probes the bare-IP
catch-all (302).
- deploy.sh: per-target logic (dankserver = pull-only, health via the
shared Caddy at 192.168.0.143); renders CADDY_BASICAUTH_HASH from
the host .env into the mounted Caddyfile.
- .env.example: CADDY_BASICAUTH_HASH on dankclaw; Porkbun block
dropped from dankserver (its Caddy is gone); SEARXNG_BASE_URL to
the flat name; stale OPENAI_* block dropped (quadlet is
self-contained).
- tests: caddyfile.bats now asserts the single-Caddyfile invariant;
dnsmasq.bats cross-checks vhosts -> .143 and the retired names.
Phase 1 of the local-DNS-name work. Users type a name, not a port.
- dnsmasq quadlet on dankclaw (rootless, slirp4netns, port 53 published):
serves the flat name scheme from addn-hosts, upstream 1.1.1.1/1.0.0.1.
Image built from the in-repo Containerfile (localhost/dnsmasq:latest).
- Caddyfiles (both hosts): all *.dankclaw.local / *.dankserver.local vhosts
renamed to flat <service>.franksdank.cloud. Explicit http:// site
addresses for now; TLS (wildcard LE via DNS-01/Porkbun) lands in phase 2.
- Drift reconciliation (live /etc/caddy/Caddyfile on dankserver differs
from the repo): added the live-only llms.franksdank.cloud vhost
(OpenWebUI on :3000). Dropped the legacy search vhost from dankserver
(the name belongs to dankclaw's gluetun-routed SearXNG; the legacy
instance stays reachable at 192.168.0.235:8081).
- Homepage service links updated to the new names.
- New test seams (TDD): tests/dnsmasq.bats (quadlet boot-persistence,
port 53, conf/record validity, Caddyfile<->addn-hosts cross-check on
the right host IPs); caddyfile.bats naming-convention tests updated.
LAN clients get this DNS from the SURFboard G36 (DNS Relay -> statically
configured -> 192.168.0.143); the router change is a pending user step.
Rootless podman cannot map the 'render' gid (990) into the container
namespace (dank's subgid range is 100000+), so --group-add render fails to
start the container. GPU access actually comes from the /dev/kfd + /dev/dri
passthrough, checked host-side (the run user is already in 'render' on the
host). So add only the 'video' group (which resolves) and drop 'render'.
Update the group test to assert video present and render ABSENT (with the
rootless rationale), per the approved rootless config.
Frank