systemd-sysctl applies /etc/sysctl.conf AFTER all of sysctl.d/, so the
stale ip_unprivileged_port_start=80 line there beat every drop-in and kept
the runtime value at 80 after the first deploy pass. deploy_pihole now
comments out an active line in /etc/sysctl.conf (idempotent) and generalizes
the sysctl.d conflict scan to any file setting a value above 53.
Caddy left compose.yaml for a quadlet unit (caddy.service): joins
dankclaw_default for container-name upstreams, Pull=never for the local-only
image, Porkbun credentials via /home/dank/Containers/caddy/env written by
deploy from .env, bare-IP-catch-all health check. deploy_caddy installs,
enables, and starts the unit after retiring the bare container (state is in
host volumes).
Pi-hole: the repo now owns the sysctl drop-in
(dankclaw/pihole/sysctl/99-dnsmasq.conf, ip_unprivileged_port_start=53);
deploy_pihole installs it to /etc/sysctl.d/, stages conflicting later-sorted
drop-ins in the home trash, and applies 'sysctl --system' (the boot code
path). This fixes the 2026-08-26 crash loop (port-53 bind denied, restart
counter 1645) caused by a second drop-in resetting the value to 80.
Tests: new caddy_quadlet.bats seam; caddyfile/deployment_flow assertions
repointed from the removed compose service to the quadlet; pihole.bats pins
the sysctl drop-in and its deployment; test-changed.sh routes dankclaw/caddy/
and the pihole subtree to their seams. Docs: Caddy, Pi-hole, Dankclaw Stack,
work log.
Caddy v2.11.4's caddyfile adapter rejects 'tls { insecure_skip_verify }'
inside transport http (unrecognized subdirective '{') and also rejects
the handler-level form. The working form, verified live against PMS with
a throwaway caddy (200 OK), is the bare token:
reverse_proxy 192.168.0.235:32400 {
transport http {
tls_insecure_skip_verify
}
}
PMS 1.43.x on :32400 accepts, reads, and silently closes every non-TLS
connection from a non-loopback source (loopback exempt). The plain-HTTP
reverse_proxy therefore got EOF and Caddy answered an empty-body 502.
Both Plex vhosts now dial the upstream over TLS with
insecure_skip_verify (PMS self-signed cert, pinned LAN hop only;
client-facing Let's Encrypt TLS unchanged).
dankware carries two +DefaultRoute links (enp47s0 eth, wlan0 wifi) that
both receive the same DHCP DNS list. systemd-resolved picked a different
'current' resolver per link, so flat-name resolution flapped between the
LAN IP (192.168.0.143 via wlan0/Pi-hole) and the WAN IP (Comcast v6 via
enp47s0).
Pin 192.168.0.143 as the static IPv4 resolver on both NetworkManager
connections (ipv4.method=auto keeps the DHCP address/gateway/route,
ignore-auto-dns=yes uses only Pi-hole, ignore-auto-routes=no keeps the
DHCP routes), then restart systemd-resolved. resolved.conf's own DNS= is
a per-link fallback, so the link must be pinned.
Flat names now resolve to 192.168.0.143 on every query; apex/unknown
subdomains still fall through to Porkbun (71.227.227.172).
The pre-commit hook runs scripts/test-changed.sh, which requires a
focused-test mapping for every staged path. The quadlets were moved into
dankclaw/<svc>/quadlet/ but the mapping still only matched top-level
dankclaw/*.container, so any nested quadlet or the pihole addn-hosts/env
data files fell through to the catch-all 'die' and blocked commit.
Add the nested-quadlet mappings (pihole-specific first, then the
dankclaw/*/quadlet/* fallback) plus the pihole data-file paths.
FTL's generated dnsmasq.conf has no conf-dir: /etc/dnsmasq.d/ is never
read (the 2026-08-24 scratch-name test proved the file was inert; names
only resolved from the query cache). The only file-based local-record
mechanism is hostsdir=/etc/pihole/hosts. Mount the record file there.
Doc: record the pitfall in Pi-hole.md.
Fanned out 5 read-only doc-auditor subagents (one per service group) that
fact-checked the knowledge base against the live system + repo config,
researched official setup docs online, and verified progress. Applied:
- All 34 broken links fixed (real pages / external URLs / correct relative
paths); 0 broken links remain. No .AGENTTRASH links (the 4 mentions are
accurate descriptive text, not links).
- Fact-check fixes: Readarr retired (May 2026), Radarr v4->v6, Calibre
9.2.1 live, Calibre-Web 0.6.25 live, Copyparty v1.20.21, LLM index
ncmoe 38 + 786432 ctx, frank-default-small active (not frank-pp-worker),
Hermes/SearXNG/Grafana hostnames, Loki/Alloy local images, Observability
diagram (Promtail->Alloy), VLLM retired note, DFlash status, benchmark
live-drift notes, Homepage test count 116.
The homepage is live on Dankclaw. Update the Status (deployed) and the Auth
section (the handle-block design, the Caddyfile directive-sorting pitfall, the
host /srv/homepage mount, and the container-recreate note). Cross-reference the
agent KB Caddyfile syntax page.
- Homepage.md: status is now 'built, pending deploy' (all assets + checker +
deploy are in the repo, 112 tests green). To go live: add CADDY_ADMIN_HASH
to the target .env, then deploy.sh.
- Homepage Design Grilling.md: the build is complete (all 6 slices, full TDD);
the only remaining step is the operator adding the admin hash + deploying.
- pi log.md: build-complete entry.
- Homepage Design Grilling.md: final decision table (Q1-Q43) + the deltas where
the operator changed the recommendation (per-service probe method, 5s timeout,
TUI panels, toggleable styling, no cursor alteration, no auto-refresh, /docs/
path, user-facing docs, git-driven deploy, full TDD, last-deploy footer).
- Homepage.md: updated to the settled design (per-service probe, 5s, TUI panels,
styling toggle, no auto-refresh, /docs/ workstream, git-driven deploy, TDD).
- pi log.md: 2026-08-23 entry recording the settled design and build order.
Build order is unblocked. No homepage code built yet.
- Homepage.md: rewritten to the settled design (single site, user + /admin/
pages, server-side status.json via a 3-min systemd-timer checker, terminal
TUI theme). Added mermaid diagrams: site structure, status states, checker
flow. Recorded the classification (user vs admin; *arr excluded; LLM admin;
backends status-only) and the documented future shift to client-side probing.
- Homepage Design Grilling.md (new): the decision log — settled Q1-Q5 plus 38
open questions (Q6-Q43) across 10 categories, each with a recommendation.
- docs/index.md: link the grilling page (satisfies the index-links-every-page test).
- report/homepage-grilling-2026-08-22.html: self-contained terminal-TUI progress
report (settled decisions, verified stack baseline, overnight processing, work
threads, the open questions, and how to continue).
Baseline recorded: Dankclaw fully up (10/10); the Dankserver media stack is down
(only Calibre-Web + the LLM are up; no .env on the target) — a separate thread.
No homepage code built yet; the design is pending the operator's answers to Q6-Q43.
The Caddyfile had 'admin off', which disables Caddy's config file
watcher. Rendered Caddyfile changes never loaded until the container
was recreated, so the double-hash basicauth fix sat on disk unapplied
across deploys (2026-08-22). Admin binds to 127.0.0.1:2019 inside the
container and the port is unpublished, so nothing outside reaches it.
deploy.sh now issues an explicit 'caddy reload' (retry loop) after
'up -d', with the file watcher as backstop.
basicauth (2026-08-22 incident):
- the Caddyfile template carried its own $2a$14$ prefix and deploy.sh
substituted the full hash (prefix included), rendering an invalid
double-prefixed hash. Every basicauth vhost rejected every password
with 401 and Caddy logged nothing.
- template is now a bare REPLACE_WITH_HASH token; tests pin the render
contract (no prefix in template; rendered line is exactly 'dank <hash>').
open-webui (2026-08-22 incident):
- its slirp4netns helper process died, leaving the container running
with only 'lo' in its netns: port 3005 accepted TCP and reset every
request, so chat served empty responses for ~2 days, unlogged.
- move to a pinned bridge network (systemd-open-webui, 10.89.3.0/24,
gw 10.89.3.1) — no per-container helper to die.
- LLM endpoint reached via the gateway (host.containers.internal is
slirp-only): OPENAI_API_BASE_URL=http://10.89.3.1:52625.
- add a quadlet healthcheck: /health (app) + non-loopback interface
(network liveness; the dead-tap state leaves only 'lo').
- deploy script now copies the .network file too.
- docs updated (OpenWebUI History section, Caddy basic-auth contract).
Frank