HALOGEN_KV_POOL_POSITIONS 262144 -> 524288, which is upstream's own
default (2 x HALOGEN_CTX). The old pool was the binding constraint on
concurrency, not context: a request reserves prompt + max_tokens against
the pool, so with pool == CTX four concurrent 65k chats were contending
for one full-length conversation's worth of positions. Two full-length
conversations, or four at 131k, now stay resident.
Cost is ~7.2 GiB more device RAM (measured 28 KiB/position), taken from
the page cache the 47.7 GiB n-gram lookup table reads through. Live
startup ledger for the prior config: 68.0 weights + 7.2 pool + 21.1
working = 96.3 GiB held, 12.1 GiB left on the host. The doubled pool
puts the remainder near 5 GiB against a ~4.3 GiB co-tenant working set,
and cache at ~69 GiB -- still above the lookup table, so the table can
stay hot. Compensating lever if long prompts start crawling:
HALOGEN_MAX_TOK=16384, which frees 8.3 GiB of prefill arena for ~8%
of prefill throughput.
Docs updated to match, including the admission-trap paragraph, which was
only true while pool == CTX.
Verbose engine logging is the vendor's first-line evidence for load,
allocation and admission questions; shipping it on means an incident
report already carries the full startup account instead of needing a
restart-into-debug. HALOGEN_DMALLOC_LOG stays off (every alloc >64MB is
steady-state noise on a shared host). Doc updated with the rootless
podman-logs-vs-journalctl retrieval note.
The healthcheck added to the loki image was discarded by the build:
'HEALTHCHECK is not supported for OCI image format and will be ignored'.
podman builds OCI by default, and OCI has no HEALTHCHECK field, so the
container came back with no health at all — worse than the failing check it
replaced, because nothing reports the loss.
The build now passes --format docker, and the seam pins it so a future edit
cannot quietly reintroduce an image with no health field.
The loki unit came up 'unhealthy' after the quadlet cutover. Cause: a unit's
HealthCmd reaches podman as CMD-SHELL, i.e. /bin/sh -c, and the loki image is
distroless — no shell exists in it. The busybox command itself works (probed
by hand: /ready answers 'ready'); it was the shell wrapper that could not run.
podman does not accept an exec-form command through --health-cmd either: it
stores a JSON array as one literal argument.
The command therefore moves into dankclaw/loki/Containerfile as a
HEALTHCHECK ... CMD [...] exec form, which is what that image exists for — it
already ships the static busybox solely so a healthcheck has something to run.
podman reads the image's healthcheck when the unit declares none.
The first deploy of the grafana unit failed: 'stat /root/.config: permission
denied', exit 1, restart loop. Cause — the repo-root .env that the compose
service used to pick up is a gluetun env file, and it carries HOME=/root (and
the WireGuard keys). Pointing the unit's [Service] EnvironmentFile at it sets
HOME for the service, so podman looks for its configuration under /root.
grafana now reads /home/dank/Containers/grafana/env, the per-service file the
deployer seeds from dankclaw/.env with a single-variable filter — the same
shape caddy, pihole, copyparty, romm and seerr use. The seed is validated, so
a missing secret fails the deploy loudly instead of starting a container whose
OAuth login cannot complete. The value was copied to dankclaw/.env, which is
where dankclaw/.env.example already documented it.
prometheus, grafana, loki, alloy and node-exporter ran only from
dankclaw/compose.yaml. Rootless podman does not restart bare containers, so
they did not survive the 2026-09-11 reboot for the amd_iommu=off change and
had to be restored by hand — the same gap that had already cost the fleet
Caddy and Pi-hole, which is why the boot-persistence rule exists.
Each is now a rootless Quadlet unit under dankclaw/observability/quadlet/,
installed and dispatched by deploy_observability in deploy-remote.sh, with
the compose definitions retired so only one owner exists per container name.
Images, mounts, arguments and container names are unchanged, so Caddy's
proxies and Prometheus' own scrape config keep resolving.
Two posture changes, both about the unauthenticated surface:
* The three HTTP UIs published on 0.0.0.0 with no authentication of any kind
(Prometheus even with the lifecycle endpoint enabled). They now publish on
127.0.0.1: the browser path is the guarded franksdank.cloud vhost, Caddy
reaches them by name over dankclaw_default, and the host-side homepage
status timer probes loopback. Nothing legitimately used the LAN port.
* node-exporter bound :9100 on every interface; it now binds exactly its two
consumers, loopback (homepage timer) and the LAN address (Prometheus, which
runs in a container and cannot reach the host's loopback).
alloy gains a reachable health endpoint: its UI listens on the container's
own loopback and alloy 1.x refuses a config block for it, so the compose
service had nothing to probe — the homepage status card for alloy read 'down'
continuously. The unit moves the UI with a command-line flag and publishes
127.0.0.1:12345.
grafana keeps its Authentik OAuth client secret without receiving the whole
.env: systemd expands the value from a [Service] EnvironmentFile, which never
reaches the container.
Hermes hardcodes response_format=json_schema on its title call. No local
back-end implements constrained decoding — frank-small and frank-large both
answer 400 'response_format json_object/json_schema is not supported' — so
every session logged a title failure and fell back to a message-derived
title. The NPU endpoint that used to absorb this call was retired on
2026-09-11.
The call now goes out over the Responses transport (the only one that does
not forward response_format) with thinking off. A 64-token cap cannot hold a
thinking pass plus a title: with the default reasoning effort the model spent
all 64 tokens thinking and returned empty content, which is why disabling
reasoning is part of the fix and not decoration.
Verified against the live endpoint with the real prompt and the real
extraction path: raw {"title": "Friendly greeting"} -> title
'Friendly greeting' in 3.4s, where the previous configuration returned 400.
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.
Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.
The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.
frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.
Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.
The unit comment always claimed the webui drives the same agent state as
the gateway, but the image bakes HERMES_HOME=/opt/data and no unit
overrode it. Podman auto-creates a fresh anonymous volume at every
image-VOLUME point on container recreation, so each restart reset the
webui's config home to the image-default template (provider: auto ->
openrouter.ai with no API key), while the gateway read the corrected
/data/config.yaml. Pin HERMES_HOME=/data so both processes share one
config/state root and config edits propagate without a recreate.
Live config was still routing on pre-rename aliases (frank-default-large,
frank-default-small) after the 2026-09-03 frank-large/frank-small/frank-npu
rename cutover; hermes-webui resolved those ids to nonexistent models and
requests failed. Live config fixed in place (backup kept on-host as
config.yaml.bak-20260905); this backup now mirrors the live shape: named
providers for aux routes, title_generation/triage/kanban routed to their
actual models, moa + mcp sections, signal disabled.
Flip the homepage Sonarr card from the user category to admin and repoint
its link at the LAN-direct port (192.168.0.112:8989). Retire the unguarded
public vhost sonarr.franksdank.cloud (direct reverse_proxy, no auth) and its
post-auth @sonarr handler from the Caddyfile; drop the Pi-hole addn-hosts A
record with it. No public Porkbun record ever existed (NXDOMAIN at the zone),
so after this there is no WAN path to Sonarr: users request TV via Seerr
(guarded), admins reach Sonarr's own UI from the LAN only.
Fixes the seerr/sonarr mix-up from 19b2589/e0bade6.
FrankBook-Pro becomes the only *arr/qBittorrent deployment. Every fleet
container (qB + five arrs) now shares gluetun's netns, so all egress —
torrent, indexer APIs, metadata lookups — exits the WireGuard tunnel by
construction (AllowedIPs=0.0.0.0/0); the arrs' own ports publish on the
gluetun unit, none on the arr units themselves.
- dankserver/compose.yaml: remove the six arr services (the Aug 20 compose
copy — never healthy, superseded; config dirs trashed on the host)
- dankclaw/Caddyfile + shared/homepage/services.json: arr vhosts and
probes repoint 192.168.0.235 -> 192.168.0.112 (FrankBook-Pro)
- frankbook-pro units: five arrs join gluetun netns (Wants/After/Network);
gluetun publishes 9696/8989/7878/8686/8787 alongside 8080/6881
- tests/frankbook_quadlet.bats: new VPN egress seam — every arr+qB unit
must share gluetun's netns and publish no port; gluetun must publish the
whole fleet's ports
- deploy-plan.py: gluetun changes carry the whole netns family (a gluetun
recreation kills all six joiners); dankserver compose set loses the arrs
- AGENTS.md: deployment is complete only when unit tests AND the Playwright
E2E suite both pass
The arr stack (prowlarr, radarr, sonarr, lidarr, readarr, qbittorrent)
had A records in addn-hosts but no Caddyfile blocks. This adds:
- Six public vhost blocks in the dankserver section, each proxying to
192.168.0.235:<port> (exempt by construction: own auth, no Anubis)
- Six @arr named handlers inside :8017 for authenticated traffic routing
All 233 bats tests pass.
Mechanical identity rename across the knowledge base, AGENTS.md, README, the
hermes config example, and homepage services.json. No behavior change; served
aliases on the LLM endpoints change with the deploy of commit 69f87c8.
Post-change doc reconciliation for the hermes vhosts + DNS deletion batch:
- Hermes-Dashboard.md: access is now the hermesadmin.franksdank.cloud vhost
(direct Caddy proxy to 192.168.0.143:9119, no guard chain — exempt because the
dashboard enforces its own basic-auth login), plus the raw LAN port. The old
text claimed an Authentik-guarded public vhost, which was decommissioned.
- Hermes-WebUI.md: description updated — the webui is now also reachable via the
unguarded hermes.franksdank.cloud vhost (its own password auth is the gate).
- LAN-DNS.md: "add a name in two places" corrected to a two-or-three-step
decision; internal-only names (prometheus/loki/grafana/seerr) deliberately get
no public A record and resolve on the LAN only, and DNS-01 issuance is
unaffected because the challenge is a TXT record in the same zone.
- Seerr.md: DNS row corrected — addn-hosts only since 2026-09-02; NXDOMAIN on WAN.
- hermes-dashboard.container: unit comment no longer claims Authentik termination.
No behavior change in this commit; tests 233/233 green (docs-completeness seam included).
Both Hermes services get direct (unguarded) vhosts: each authenticates itself
(webui password; dashboard basic_auth in /data/config.yaml), so no Anubis
challenge and no Authentik proxy provider is required. Upstreams are the host
LAN IP because Caddy itself sits on dankclaw_default, where the gateway
address 10.89.1.1 has nothing serving 8787. Internal-only by DNS absence:
hermes.franksdank.cloud already resolves via Pi-hole addn-hosts; hermesadmin
is added in this same commit so the caddyfile/addn-hosts cross-check (test 178)
stays green.
podman 5.7.0 refuses ghcr.io/goauthentik/ldap:2026.8.0's nested manifest
list ('Unexpectedly received a manifest list instead of a manifest for a
single image'); pin the linux/amd64 single-platform digest instead.
Repoint Volume=/EnvironmentFile= in the 8 affected quadlet units to the
caddy-established Containers/<svc> layout: gitea (data, postgres, env),
authentik (data, postgres), hermes (data, webui checkout). Data-root mv on the
target happens at deploy time before each group's unit restart; same-fs rename,
instantly reversible. Running containers keep their inode-bound binds until the
unit restart flips them to the new path.
Adds a digest-pinned headless-shell Chrome DevTools Protocol daemon as a
boot-persistent quadlet unit (chrome-devtools-protocol.container). The
entrypoint override drops the image's socat front so the CDP endpoint binds
loopback-only on 127.0.0.1:9222 in the host network namespace; only the two
host-network consumers (Signal gateway, webui server) can drive it.
hermes-webui flips Network=dankclaw_default -> Network=host so its agent
process shares the daemon's namespace (0.0.0.0:8787 bind is LAN-facing as
before; no exposure change). deploy_hermes installs the new unit and starts
it if inactive without restarting in-flight browser sessions.
DNS at Porkbun was renamed (chat->ai, jellyfin->jelly) and the qbittorrent,
*arr-stack, comfyui, hermes, dankclaw, and dankserver public records were
deleted. This brings the Caddy config in lockstep: ai/jelly vhosts now match
the new names; comfyui (guarded), hermes (guarded), and the *arr/qb vhosts
are removed as their DNS is gone. The legacy dankclaw/dankserver redirect
vhosts stay: Pi-hole still answers those names on the LAN.
LAN clients now connect directly to 192.168.0.143:8787 (password-gated via
HERMES_WEBUI_PASSWORD); the SSH tunnel via port 22 remains the sole
off-network ingress. Recreation also re-reads the rotated password from
dankclaw/.env via the [Container] EnvironmentFile.
The embedded outpost carries no provider token (token=None), so the
auth_proxy identity header was never set and no user could log in. RP
mode is topology-independent: challenge gate stays in front, login runs
inside Grafana via the OAuth2 code flow against auth.franksdank.cloud.
Provider 'grafana' (pk 9, confidential) created via ak shell; secret
written straight into the live .env (never in git). Auth URL uses the
global /application/o/authorize/ path — this fork resolves the provider
by client_id query param; per-slug authorize does not exist.
Caddyfile comment corrected: no identity header is forwarded to Grafana.
Grafana leaves the exempt-by-construction set: grafana.franksdank.cloud
now terminates at the Anubis guard (anubis:8923) and authenticated
traffic returns at the internal :8017 listener, where an @grafana Host
handler proxies through to grafana:3000 with the outpost-proxy trust
token (GRAFANA_PROXY_TOKEN = client_id_client_secret of the new
'grafana-proxy' ProxyProvider on the embedded outpost; created via
ak shell before this route flip — provider-first ordering per the
External Exposure Policy).
The Grafana admin account is intentionally kept: auth.proxy in
auto-upsert mode only creates non-admin users, so the local admin
remains the only way to administer Grafana. The old edge-basicauth
wrapper is retired (shared family password, no per-user audit trail).
- Caddyfile: grafana vhost flipped to the guarded chain; @grafana
handler added at the :8017 return listener.
- compose.yaml: GF_AUTH_PROXY_ENABLED/HEADERS/TOKEN on the grafana
service (auth_proxy trusted-proxy CIDRs left at the image default —
the proxying Caddy container is not statically addressable on
10.89.0.0/24, so Grafana trusts any in-network source and relies on
the token as the capability).
- .env.example: GRAFANA_PROXY_TOKEN placeholder so test env_consistency
keeps guarding the compose<->env contract.
Loki's public vhost had zero auth — any host that can resolve DNS could
query all logs via LogQL. The vhost now terminates at anubis:8923 and
authenticated traffic returns at the internal :8017 listener, where a new
@loki host handler routes to loki:3100. The outpost proxy provider
'loki-proxy' (external_host https://loki.franksdank.cloud/, internal_host
http://caddy:8017) was created before this deploy — provider-first
ordering so authenticated traffic never hits the outpost error page.
Ingestion is unaffected: Alloy pushes to loki:3100 on the compose network.
The flip hunk's first comment line was written with a literal ':' prefix
(edit-tool auto-prefix artifact). Caddyfile parses ':# ...' as a matcher token,
which made the following site block an unrecognized directive and failed
config validation — that is why the earlier deploy validated, failed, and
skipped the reload, leaving the live config pre-flip (comfyui still direct).
The first revision guarded the kfd line with SUBSYSTEM=="misc", which never
matches (misc-class devices carry no subsystem for udev matching), so the rule
was read but never fired and /dev/kfd stayed 0660 root:render. Name-only match
applies 0666 on creation and on trigger.
Also fixes test-changed.sh to exit 0 when changed paths map to zero tests
(udev rules are mapped-but-untestable) instead of invoking bats with an empty
selection.
The comfyui container is spawned by a systemd user unit whose spawn context
carries no supplementary groups, so render-group access to /dev/kfd and
renderD128 was structurally unreachable (keep-groups preserved an empty set;
named group_add 'render' fails name lookup inside the userns). World-mode on
the compute nodes is the only spawn-context-independent grant; display cards
keep their modes. Apply: install to /etc/udev/rules.d, udevadm control
--reload-rules, then trigger by subsystem to re-apply modes to live nodes.
Also maps dankclaw/udev/* in the focused-test selector (not bats-testable).
Rootless podman cannot resolve the literal group name 'render' inside the
container user namespace and fails the start with 'unable to find group
render'. keep-groups preserves the launching user's real supplementary
groups (dank is in render on this host), which is the canonical rootless
ROCm device-access recipe.
/dev/kfd and /dev/dri/renderD128 are root:render 0660 on this host. The
container passed the device nodes but only joined group video(44), so torch's
HIP init failed with 'No HIP GPUs are available' and main.py never bound
:8188 — every proxy hop to the app returned 502. Group membership is a
create-time property, so this deploy recreates the container.
comfyui.franksdank.cloud was exempt (direct reverse_proxy to comfyui:8188).
The comfyui-proxy proxy provider was created on the embedded outpost first
(internal_host http://caddy:8017, flows copied from hermes-proxy), then the
vhost flips onto the guard chain and authenticated traffic routes back via a
@comfy host match. ComfyUI has no auth of its own; this closes the last
unguarded admin surface on the internal hosts.
The quadlet generator space-splits HealthCmd into the healthcheck argv, so the
quoted /bin/sh -c form arrived mangled and failed with exit 1 forever (FailingStreak
569) while the DB itself accepted connections. Bare argv survives both exec forms.
The v1 quadlet generator skips the whole unit when [Container] holds an
unsupported key; the stale key was a leftover of a draft edit. Network=host is
the only host-networking key this generator version accepts.
The v1 generator on this host skips a unit whose [Container] section holds an
unknown key, so the deploy-time regeneration dropped hermes.service entirely
while the container kept running unmanaged. Network=host generates
--network host verbatim (v1 passes extension-less values through).
The gateway unit ran on dankclaw_default and could not reach the host
daemon: from that network only DNAT'd published ports accept connections to
the LAN address. The compose era ran the gateway with network_mode: host, so
the daemon was reachable; the unit silently changed topology at takeover and
broke the adapter. Host networking + a loopback dial removes the dependency
entirely — the gateway is an outbound-only client and publishes nothing.
The trio ran as bare compose containers and died silently at the Aug 26
reboot — nothing restarts compose containers at boot. Each now has a
boot-persistent unit (WantedBy=default.target; the v1 quadlet generator on
this host cannot enable generated units). The deploy handler retires the
compose-era containers only while the units are not yet active, then
(re)starts each unit.
Units run the image under a localhost/hermes-agent:latest alias tag (the
quadlet create path on this host cannot resolve registry-prefixed refs from
local storage); the deploy handler tags first. The gateway's :8090 publish
is dropped — nothing consumes it; the public vhost now routes to the
dashboard's login-gated UI on 9119.
The gateway's SIGNAL_HTTP_URL pointed at the host LAN IP. From container
space on netavark networks only DNAT'd published ports accept connections
to that address; :8080 has no forward, so every connection was refused
and the Signal adapter stayed disconnected since the Aug 26 reboot.
host.containers.internal is the verified working path (probed live from a
probe container on dankclaw_default).
The public hostname also routed to hermes:8090, which serves nothing —
the gateway has no HTTP listener. It now routes to the dashboard's login-
gated admin UI on :9119, behind the existing Anubis+Authentik guard.
Login/PoW/POST traffic to the outpost URL space cannot solve a fresh PoW
challenge per request; exempt the transport prefix (authentik's own session
gate still protects it) so the guarded-host sign-in journey completes.
The redirect-to-flow design with an absolute cross-host next= was rejected by
the flow's open-redirect guard at completion ("Request has been denied:
Invalid next URL"), and meta-header routing does not exist in this build
(providers route on the ORIGINAL Host). :8016 now proxies every request whole
to authentik-server:9000 preserving Host; the embedded outpost serves login
in place and forwards authenticated traffic back to :8017. Requires one proxy
provider per guarded host (provisioned live: search/hermes/prometheus/romm/
files + www, flows backfilled). Tests flipped from the meta-header claim to
the upheld contract: Host preserved, no absolute next= redirects.
This 2026.x build does not mount /outpost.goauthentik.io/* for proxy-provider
meta-header routing (probed live: ping answers 204, app paths with full meta
headers 404). Guarded vhosts now redirect to /if/flow/default-authentication-flow/
with next= back to the app — unauthenticated sees the login page in place,
authenticated sessions complete instantly and bounce straight to the app.
The proxy provider matches external_host exactly, so Host=www matched zero
providers and Authentik answered its error page while the naked domain
served the homepage. Flip the topology: www.franksdank.cloud is the one
guarded vhost carrying the homepage through Anubis -> Authentik -> :8017;
franksdank.cloud (plus legacy names and the bare-IP catch-all) 308-redirects
onto www before any challenge. Tests flipped to assert the new contract.
Jellyfin's Kestrel backend rejects the h2c upgrade preface, so Caddy turned
every request into a 502 while the same port answers 200 over plain HTTP/1.1
from Caddy's network. Default transport (HTTP/1.1) restores the vhost.
The pinned binary's policy parser rejects unknown actions and exits 1 on
startup (verified against the image binary: the valid action literals are
ALLOW, CHALLENGE, DENY). The first cut of the own-endpoints accept rule
used ACCEPT and crash-looped the unit on deploy.
The catch-all path_regex "." policy challenged everything, including
Anubis's own API namespace /.within.website/x/cmd/anubis/*
(make-challenge, pass-challenge, static assets). In whole-request proxy
mode a JS-capable browser POSTs its challenge solution to
pass-challenge and received another challenge page (HTML) instead of the
verification verdict (JSON): the challenge app's fetch failed and the
browser rendered its own error — 'the request filters did not return an
alternative response'. Non-JS clients never POST the solution, so every
curl probe looked green while every human hit a wall.
Accept rule ordered above the catch-all; policy is volume-mounted ro from
this checkout, so the deploy updates it in place.
The forward_auth target /outpost.goauthentik.io/auth/caddy does not exist
in authentik 2026.8 (verified live: 404 for every header combination; the
URL tree has no outpost auth prefix at all — the outpost worker dials back
to core over ws/outpost/<uuid>/ instead). Unauthenticated subrequests got
404, Caddy treated >=400 as 'not authenticated' and redirected browsers to
the dead path — users saw authentik's own Not-Found page.
Fix: guarded vhosts now proxy the whole request through authentik-server:9000
with X-authentik-meta-outpost/-provider/-app headers (the exact header set
this build's Traefik controller emits, adapted for Caddy). Unauthenticated
clients get the login flow served in place; once a session exists,
authentik proxies on to the provider's internal_host — now
http://caddy:8017, a new post-auth internal listener that routes straight to
the app. The chain terminates; nothing dials back into authentik.
Live DB changes applied alongside (postgres is the source of truth):
provider pk=1 authentication_flow=default-authentication-flow (was NULL —
the actual cause of the 404s) and internal_host=http://caddy:8017 (was
self-referential http://authentik-server:9000).
Frank