Loki's public vhost had zero auth — any host that can resolve DNS could
query all logs via LogQL. The vhost now terminates at anubis:8923 and
authenticated traffic returns at the internal :8017 listener, where a new
@loki host handler routes to loki:3100. The outpost proxy provider
'loki-proxy' (external_host https://loki.franksdank.cloud/, internal_host
http://caddy:8017) was created before this deploy — provider-first
ordering so authenticated traffic never hits the outpost error page.
Ingestion is unaffected: Alloy pushes to loki:3100 on the compose network.
The flip hunk's first comment line was written with a literal ':' prefix
(edit-tool auto-prefix artifact). Caddyfile parses ':# ...' as a matcher token,
which made the following site block an unrecognized directive and failed
config validation — that is why the earlier deploy validated, failed, and
skipped the reload, leaving the live config pre-flip (comfyui still direct).
The first revision guarded the kfd line with SUBSYSTEM=="misc", which never
matches (misc-class devices carry no subsystem for udev matching), so the rule
was read but never fired and /dev/kfd stayed 0660 root:render. Name-only match
applies 0666 on creation and on trigger.
Also fixes test-changed.sh to exit 0 when changed paths map to zero tests
(udev rules are mapped-but-untestable) instead of invoking bats with an empty
selection.
The comfyui container is spawned by a systemd user unit whose spawn context
carries no supplementary groups, so render-group access to /dev/kfd and
renderD128 was structurally unreachable (keep-groups preserved an empty set;
named group_add 'render' fails name lookup inside the userns). World-mode on
the compute nodes is the only spawn-context-independent grant; display cards
keep their modes. Apply: install to /etc/udev/rules.d, udevadm control
--reload-rules, then trigger by subsystem to re-apply modes to live nodes.
Also maps dankclaw/udev/* in the focused-test selector (not bats-testable).
Rootless podman cannot resolve the literal group name 'render' inside the
container user namespace and fails the start with 'unable to find group
render'. keep-groups preserves the launching user's real supplementary
groups (dank is in render on this host), which is the canonical rootless
ROCm device-access recipe.
/dev/kfd and /dev/dri/renderD128 are root:render 0660 on this host. The
container passed the device nodes but only joined group video(44), so torch's
HIP init failed with 'No HIP GPUs are available' and main.py never bound
:8188 — every proxy hop to the app returned 502. Group membership is a
create-time property, so this deploy recreates the container.
comfyui.franksdank.cloud was exempt (direct reverse_proxy to comfyui:8188).
The comfyui-proxy proxy provider was created on the embedded outpost first
(internal_host http://caddy:8017, flows copied from hermes-proxy), then the
vhost flips onto the guard chain and authenticated traffic routes back via a
@comfy host match. ComfyUI has no auth of its own; this closes the last
unguarded admin surface on the internal hosts.
The quadlet generator space-splits HealthCmd into the healthcheck argv, so the
quoted /bin/sh -c form arrived mangled and failed with exit 1 forever (FailingStreak
569) while the DB itself accepted connections. Bare argv survives both exec forms.
The v1 quadlet generator skips the whole unit when [Container] holds an
unsupported key; the stale key was a leftover of a draft edit. Network=host is
the only host-networking key this generator version accepts.
The v1 generator on this host skips a unit whose [Container] section holds an
unknown key, so the deploy-time regeneration dropped hermes.service entirely
while the container kept running unmanaged. Network=host generates
--network host verbatim (v1 passes extension-less values through).
The gateway unit ran on dankclaw_default and could not reach the host
daemon: from that network only DNAT'd published ports accept connections to
the LAN address. The compose era ran the gateway with network_mode: host, so
the daemon was reachable; the unit silently changed topology at takeover and
broke the adapter. Host networking + a loopback dial removes the dependency
entirely — the gateway is an outbound-only client and publishes nothing.
The trio ran as bare compose containers and died silently at the Aug 26
reboot — nothing restarts compose containers at boot. Each now has a
boot-persistent unit (WantedBy=default.target; the v1 quadlet generator on
this host cannot enable generated units). The deploy handler retires the
compose-era containers only while the units are not yet active, then
(re)starts each unit.
Units run the image under a localhost/hermes-agent:latest alias tag (the
quadlet create path on this host cannot resolve registry-prefixed refs from
local storage); the deploy handler tags first. The gateway's :8090 publish
is dropped — nothing consumes it; the public vhost now routes to the
dashboard's login-gated UI on 9119.
The gateway's SIGNAL_HTTP_URL pointed at the host LAN IP. From container
space on netavark networks only DNAT'd published ports accept connections
to that address; :8080 has no forward, so every connection was refused
and the Signal adapter stayed disconnected since the Aug 26 reboot.
host.containers.internal is the verified working path (probed live from a
probe container on dankclaw_default).
The public hostname also routed to hermes:8090, which serves nothing —
the gateway has no HTTP listener. It now routes to the dashboard's login-
gated admin UI on :9119, behind the existing Anubis+Authentik guard.
Login/PoW/POST traffic to the outpost URL space cannot solve a fresh PoW
challenge per request; exempt the transport prefix (authentik's own session
gate still protects it) so the guarded-host sign-in journey completes.
The redirect-to-flow design with an absolute cross-host next= was rejected by
the flow's open-redirect guard at completion ("Request has been denied:
Invalid next URL"), and meta-header routing does not exist in this build
(providers route on the ORIGINAL Host). :8016 now proxies every request whole
to authentik-server:9000 preserving Host; the embedded outpost serves login
in place and forwards authenticated traffic back to :8017. Requires one proxy
provider per guarded host (provisioned live: search/hermes/prometheus/romm/
files + www, flows backfilled). Tests flipped from the meta-header claim to
the upheld contract: Host preserved, no absolute next= redirects.
This 2026.x build does not mount /outpost.goauthentik.io/* for proxy-provider
meta-header routing (probed live: ping answers 204, app paths with full meta
headers 404). Guarded vhosts now redirect to /if/flow/default-authentication-flow/
with next= back to the app — unauthenticated sees the login page in place,
authenticated sessions complete instantly and bounce straight to the app.
The proxy provider matches external_host exactly, so Host=www matched zero
providers and Authentik answered its error page while the naked domain
served the homepage. Flip the topology: www.franksdank.cloud is the one
guarded vhost carrying the homepage through Anubis -> Authentik -> :8017;
franksdank.cloud (plus legacy names and the bare-IP catch-all) 308-redirects
onto www before any challenge. Tests flipped to assert the new contract.
Jellyfin's Kestrel backend rejects the h2c upgrade preface, so Caddy turned
every request into a 502 while the same port answers 200 over plain HTTP/1.1
from Caddy's network. Default transport (HTTP/1.1) restores the vhost.
The pinned binary's policy parser rejects unknown actions and exits 1 on
startup (verified against the image binary: the valid action literals are
ALLOW, CHALLENGE, DENY). The first cut of the own-endpoints accept rule
used ACCEPT and crash-looped the unit on deploy.
The catch-all path_regex "." policy challenged everything, including
Anubis's own API namespace /.within.website/x/cmd/anubis/*
(make-challenge, pass-challenge, static assets). In whole-request proxy
mode a JS-capable browser POSTs its challenge solution to
pass-challenge and received another challenge page (HTML) instead of the
verification verdict (JSON): the challenge app's fetch failed and the
browser rendered its own error — 'the request filters did not return an
alternative response'. Non-JS clients never POST the solution, so every
curl probe looked green while every human hit a wall.
Accept rule ordered above the catch-all; policy is volume-mounted ro from
this checkout, so the deploy updates it in place.
The forward_auth target /outpost.goauthentik.io/auth/caddy does not exist
in authentik 2026.8 (verified live: 404 for every header combination; the
URL tree has no outpost auth prefix at all — the outpost worker dials back
to core over ws/outpost/<uuid>/ instead). Unauthenticated subrequests got
404, Caddy treated >=400 as 'not authenticated' and redirected browsers to
the dead path — users saw authentik's own Not-Found page.
Fix: guarded vhosts now proxy the whole request through authentik-server:9000
with X-authentik-meta-outpost/-provider/-app headers (the exact header set
this build's Traefik controller emits, adapted for Caddy). Unauthenticated
clients get the login flow served in place; once a session exists,
authentik proxies on to the provider's internal_host — now
http://caddy:8017, a new post-auth internal listener that routes straight to
the app. The chain terminates; nothing dials back into authentik.
Live DB changes applied alongside (postgres is the source of truth):
provider pk=1 authentication_flow=default-authentication-flow (was NULL —
the actual cause of the 404s) and internal_host=http://caddy:8017 (was
self-referential http://authentik-server:9000).
In authentik 2026.8.x the proxy-provider URL space (/outpost.goauthentik.io/*)
is served by the Rust worker binary, not the server — verified live: with a
tenant-matching Host header, /if/flow/... answers 200 on the server while
/auth/caddy still 404s. The outpost instance existed in the DB but no process
ever served it, so Caddy's forward_auth subrequests hit a 404 and browsers got
authentik's own Not-Found page instead of the authentication flow.
The new unit runs the same image as server/worker with Exec=worker and flips
into outpost mode via AUTHENTIK_HOST/AUTHENTIK_INSECURE/AUTHENTIK_TOKEN (the
token is read from the DB into the host .env at deploy time; it never enters
the repo). It dials back to core over websocket and serves forward-auth on its
own 9000. Deploy handler restarts only this unit — the rest of the stack is
untouched.
The image has no shell (ko-build distroless; entrypoint /ko-app/anubis),
and this host's quadlet generator joins a bare HealthCmd into one
\x20-escaped argv element that podman then runs shell-form — exec of a
missing shell exits 1 and the container sits unhealthy. A JSON array in
HealthCmd passes through the generator verbatim and executes the binary
directly.
The v1.13.0 image is distroless: no shell, no HEALTHCHECK in the image
config (verified live: Healthcheck config is null on this host's image),
so quadlet inherits nothing. Declare the probe explicitly in exec form;
'anubis --healthcheck' exits 0 when the challenge server answers.
On this host's podman 5.7 quadlet path, a registry-prefixed Image ref
fails to resolve from local storage at container create time even when
the image is present ('image not known'); the identical argv works from
a shell or transient unit. Verified live: an identical unit with
Image=localhost/techarohq/anubis:v1.13.0 starts immediately. The deploy
handler now tags the alias after the guarded pull; upgrades re-tag
after pulling the new tag.
podman's reference parser rejects mixed-case repository refs before any
network I/O ('repository name must be uppercase'); upstream's own install
docs pull lowercase (ghcr resolves case-insensitively). The first deploy
of this chain exit-2'd on that parse error — the image was not yet
pulled, so the guarded vhosts 502 until this deploy lands.
Bots and AI scrapers hit the SSO-fronted vhosts and eat Authentik auth
work for 403s. Anubis verifies clients first: DENY for known AI scraper
UAs, JS challenge for everyone else; verified traffic resumes the normal
Authentik forward_auth chain on Caddy's internal listener (:8016).
Chain: client -> Caddy (TLS) -> Anubis :8923 (verify) -> Caddy :8016
(auth + routing resumes; Host survives both hops) -> app. Reverse-proxy
through Anubis, not forward_auth: at the pinned tag a CHALLENGE answers
HTTP 200 + HTML, and forward_auth treats 200 as "authenticated" — the
page would pass through unchallenged. Exempt by construction (never
routed through Anubis): plex (plain HTTP legacy), auth.* (IdP login
must be reachable ungated), gitea (git smart-HTTP clients cannot solve
JS challenges), and every app with its own login.
Pinned ghcr.io/TecharoHQ/anubis:v1.13.0 — the tag was source-verified
before pinning: at this tag the policy parser is JSON-only with a
minimal bots[] schema (no imports, no CEL expressions), flags are the
config surface via flagenv (TARGET/POLICY_FNAME map to --target/
--policy-fname), and the image's exec-form HEALTHCHECK is inherited by
the quadlet unit (dodges the v1 generator's HealthCmd space-split quirk).
Rollback nuance: stopping anubis 502s everything behind the chain (the
public listeners proxy through it); a true rollback is reverting the
Caddyfile and redeploying, not just systemctl --user stop anubis.
Verified live on dankclaw: the podman 5.x v1 quadlet generator splits
HealthCmd values on spaces into the container healthcheck argv array, so
quoted probes (sh -c "pg_isready ...") arrive mangled and every probe
exits 127 — postgres read 'unhealthy' with the database actually up.
Probe forms now use only space-free argv tokens: pg_isready passes its
args directly; the server probe runs manage.py check --database default
(verified exit-0 standalone in the image).
Probed the live generator on dankclaw: podman 5.x v1 hard-fails unit
generation on dash-prefixed options (--shm-size killed authentik-server's
unit) and rejects Command= entirely; Exec= is the accepted role selector.
Drop --shm-size (uwsgi uses unix sockets under /run, not /dev/shm; default
64m shm is fine) and drop systemctl --user enable — generated units are
transient and cannot be enabled; WantedBy=default.target in each unit file
is what makes them boot-persistent on this generator.
The compose-era authentik trio died at every boot (rootless podman does not
auto-start bare containers). Convert the trio to Quadlet units installed by
deploy-remote.sh into ~/.config/containers/systemd/, enabled + WantedBy=
default.target:
- 3 unit files under dankclaw/authentik/quadlet/ (postgres -> server -> worker
ordering via Wants=/After=; secrets stay in the git-ignored host .env,
expanded by systemd at start time exactly like the gitea units)
- deploy_authentik() handler: install units, daemon-reload, retire stale
compose-era containers only while units are inactive, enable once, then
start in dependency order with podman-health gates between stages
- compose.yaml: authentik service definitions retired to a comment pointing at
the quadlet dir (same pattern as searxng/open-webui/gitea/caddy)
- tests/authentik.bats rewritten from compose assertions to unit-file
contract assertions (images, Exec roles, ordering, loopback-only publish,
pg_isready probe, secret hygiene, dispatch routing)
All in-repo path references (AGENTS.md, README, old scripts, report artifacts,
test code paths) rewritten in the same commit; focused-test map extended for
.gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's
own convention ('add one before committing'). Obsidian wikilinks are
title-based and unaffected.
Caddy left compose.yaml for a quadlet unit (caddy.service): joins
dankclaw_default for container-name upstreams, Pull=never for the local-only
image, Porkbun credentials via /home/dank/Containers/caddy/env written by
deploy from .env, bare-IP-catch-all health check. deploy_caddy installs,
enables, and starts the unit after retiring the bare container (state is in
host volumes).
Pi-hole: the repo now owns the sysctl drop-in
(dankclaw/pihole/sysctl/99-dnsmasq.conf, ip_unprivileged_port_start=53);
deploy_pihole installs it to /etc/sysctl.d/, stages conflicting later-sorted
drop-ins in the home trash, and applies 'sysctl --system' (the boot code
path). This fixes the 2026-08-26 crash loop (port-53 bind denied, restart
counter 1645) caused by a second drop-in resetting the value to 80.
Tests: new caddy_quadlet.bats seam; caddyfile/deployment_flow assertions
repointed from the removed compose service to the quadlet; pihole.bats pins
the sysctl drop-in and its deployment; test-changed.sh routes dankclaw/caddy/
and the pihole subtree to their seams. Docs: Caddy, Pi-hole, Dankclaw Stack,
work log.
Caddy v2.11.4's caddyfile adapter rejects 'tls { insecure_skip_verify }'
inside transport http (unrecognized subdirective '{') and also rejects
the handler-level form. The working form, verified live against PMS with
a throwaway caddy (200 OK), is the bare token:
reverse_proxy 192.168.0.235:32400 {
transport http {
tls_insecure_skip_verify
}
}
PMS 1.43.x on :32400 accepts, reads, and silently closes every non-TLS
connection from a non-loopback source (loopback exempt). The plain-HTTP
reverse_proxy therefore got EOF and Caddy answered an empty-body 502.
Both Plex vhosts now dial the upstream over TLS with
insecure_skip_verify (PMS self-signed cert, pinned LAN hop only;
client-facing Let's Encrypt TLS unchanged).
The pre-commit hook runs scripts/test-changed.sh, which requires a
focused-test mapping for every staged path. The quadlets were moved into
dankclaw/<svc>/quadlet/ but the mapping still only matched top-level
dankclaw/*.container, so any nested quadlet or the pihole addn-hosts/env
data files fell through to the catch-all 'die' and blocked commit.
Add the nested-quadlet mappings (pihole-specific first, then the
dankclaw/*/quadlet/* fallback) plus the pihole data-file paths.
FTL's generated dnsmasq.conf has no conf-dir: /etc/dnsmasq.d/ is never
read (the 2026-08-24 scratch-name test proved the file was inert; names
only resolved from the query cache). The only file-based local-record
mechanism is hostsdir=/etc/pihole/hosts. Mount the record file there.
Doc: record the pitfall in Pi-hole.md.
The other vhosts (gitea, grafana, search, ...) now use username 'dankuser'
with CADDY_BASICAUTH_HASH, which is set to the same hash as the homepage
(dankpassword.123). So the whole protected surface logs in as
dankuser / dankpassword.123; the operator /admin/ page still uses frank.
Updated the two tests that asserted the old 'dank' username.
The homepage user page + shared data now use a dedicated CADDY_HOMEPAGE_HASH
(username dankuser), separate from the shared family password (CADDY_BASICAUTH_HASH,
username dank, still used by the other vhosts) and the operator password
(CADDY_ADMIN_HASH, username frank, the /admin/ page). This lets the homepage be
re-logged independently without touching the other sites. deploy.sh renders the
new placeholder; the .env.example documents all three hashes.
The caddy container mounted the repo's shared/homepage (the static source) to
/srv/homepage. But the deploy writes the generated files (status.json from the
status checker, deployed.js from the deploy) to the host's /srv/homepage. So
the container served the static files (200) but 404'd the generated ones.
Mount the host's /srv/homepage instead, which holds both the static files (the
deploy copies them) and the generated ones.
Caddy SORTS top-level directives, so a 'rewrite' runs before 'basicauth'.
The rewrite changed /admin/ to /admin.html, and the path-based auth matchers
then saw /admin.html (which matched the family 'not path /admin/*' block, not
the operator '/admin/*' block) — so the operator password was checked against
the family hash and 401'd.
Serve each page from its own 'handle' block (which preserves order, so the
auth is checked on the original path before any file rewrite):
- handle /admin* : operator (frank) only, try_files /admin.html
- handle @data : shared data + assets, family (dank) OR operator (frank), so
the admin page's fetches of services.json/status.json reuse the operator
credential
- handle : everything else (the user page), family (dank)
Frank