Retiring dankclaw/compose.yaml and adding alert rules invalidated claims across
the repo, including in the rule file agents read first. Clean cutover includes
the prose.
**Tests that had gone hollow.** Four assertions grepped dankclaw/compose.yaml
for a service name. With the file gone they passed by *absence* — a gate that
can no longer fail. Each now asserts the real invariant ("no dankclaw compose
file competes with the quadlet units") and keeps a live check for a re-added
entry, so it still fails when one appears: verified by recreating a compose file
with a comfyui service and watching three tests go red, then removing it.
**AGENTS.md, which was wrong about ComfyUI.** It said ComfyUI runs CPU-only
("no ROCm wheels for gfx1151, CUDA images wrong"). That was true in the
compose era and is false now: the restored unit runs with /dev/kfd + /dev/dri and
must NOT pass --cpu. An agent trusting the rule file would have re-broken it.
Also corrected: the observability paragraph ("RUNNING BUT UNCONFIGURED, no
datasource, no rules" — all three have been configured since 2026-09-16), the
Active runbook row that still named halogen for frank-large, halogen's absence
from the Retired table, the benchmark policy naming halogen's harness, and the
seam list.
**Unit and page claims.** prometheus.container said "no alerting rules anywhere
in this stack"; node-exporter.container said Dankserver runs from its compose
stack; Prometheus.md documented three jobs as working (they scrape HTML: they
can never report up) and said no dashboard existed; index.md and the three
authentik service pages pointed at the retired file; Authentik.md's upgrade
instruction told you to bump an image tag in it.
One bad edit of my own on the way (a stale line number clobbered a See Also
entry in Prometheus.md) — caught by reading back, fixed, and the vault link
checker now proves the cross-references hold.
The vault is the project's source of truth, so a link to a page that does not
exist is a promise the repo cannot keep. Three were dangling:
- Dankclaw/Gitea.md pointed twice at dankclaw/compose.yaml — **which this
session retired**. The clean cutover obligation from my own change: the page
now describes the unit that publishes 3000 and the canonical Caddy unit, and
records why the compose file is gone.
- Dankclaw/Hermes-Dashboard.md linked [[LAN DNS]]; the page is [[LAN-DNS]].
- LLM/frank-medium-radiance-mxfp4.md linked [[Observability on Dankclaw]]; the
page is [[Observability]].
scripts/check-vault-links.py + a seam keep it that way. The checker models
Obsidian's rules, which took three corrections to get right — and each was a
false positive that would have sent me editing healthy docs:
1. Wikilinks are case-insensitive. A case-sensitive matcher reported 45
breaks; 39 were `[[qBittorrent]]` vs qbittorrent.md.
2. `[[folder/Note]]` resolves by path suffix, not by stem.
3. `%20` in a markdown link is URL-encoding, not a filename.
Validating the linter against the resolver it models is the prerequisite for
trusting any of its output — the same discipline that caught a bad assertion of
my own earlier tonight.
Two gaps behind the same near-miss.
1. **No spec covered hermes.franksdank.cloud** — the one guarded vhost whose
upstream is a host address (`reverse_proxy 192.168.0.143:8787`) rather than a
container name, because its service is host-networked. That is how a
loopback-only bind could take the vhost down with a 502 and leave every gate
green. tests/e2e/specs/17-hermes-vhost.spec.ts asserts what a dead upstream
violates: no 5xx through the guard chain, no Caddy error page, and no app
bytes to an unauthenticated raw client. It does not assert SSO-silence, since
this app has its own password login.
2. **Specs are matched by number prefix, not globbed.** A spec file whose prefix
is in no project silently never runs — which is exactly what happened to the
new one until it was registered. The prefix is registered now, the comment in
playwright.config.ts says why, and tests/e2e_scaffold.bats fails if any
spec file matches no project.
The hermes-webui seam assertion is corrected too: it asserted the loopback bind
I wrongly imposed, and now asserts the real contract — the unit's bind must serve
the address the Caddyfile fronts it on.
Two things, found while checking whether the hermes trio still needed migrating.
1. **The phone-facing webui was exposed to the LAN.** Its unit runs with
Network=host, so there is no Podman port forwarder and the app's own bind is
the exposure. The compose service it replaced published
`127.0.0.1:8787:8787` (loopback); the unit bound `HERMES_WEBUI_HOST=0.0.0.0`,
and the LAN address answered 200 — password-gated, but on every interface,
which is not the documented "SSH tunnel is the sole ingress" posture. It now
binds 127.0.0.1, which serves the tunnel just as well (sshd dials
localhost:8787 from the host). The unit comment claiming "LAN-facing as
before" was wrong about "before", and is corrected.
2. **Dankclaw's compose.yaml is retired.** Auditing ownership showed every
running container on that host reports a PODMAN_SYSTEMD_UNIT label — the
hermes trio's units have owned their containers since 2026-09-12 — so the
three service blocks left in compose.yaml were dead weight that could only
create a duplicate container fighting a unit for a name. deploy_hermes was
already fully unit-based. The file is staged in .AGENTTRASH/; the planner
plans nothing for it, an unknown service on that target now errors instead of
silently no-oping through `podman-compose`, and DANKCLAW_COMPOSE_SERVICES is
empty.
The hermes-webui seam now reads the unit — it read compose, which is exactly how
the bind drifted while the test kept passing against a file nobody ran.
ComfyUI had failed 9,146 times and had been down ~7 hours when this was found:
its container was created 2026-09-15 14:12, BEFORE the GPU declarations were
added to the compose service, and `restart: unless-stopped` kept restarting
that stale container — no /dev/kfd, no /dev/dri, no keep-groups — so it died
every time on `RuntimeError: No HIP GPUs are available`. Its "unit" was a
hand-written ~/.config/systemd/user/comfyui.service that shelled out to
`podman compose up -d`, which is not repo-tracked state and collides by name
with the Quadlet the generator produces.
- dankclaw/comfyui/quadlet/comfyui.container: image comfyui-rocm:gfx1151
(built from Containerfile.comfyui), /dev/kfd + /dev/dri, keep-groups,
the four comfyui-data volumes, loopback-only publish (an unguarded service
is internal-only here; the homepage card probes 127.0.0.1:8188), Restart=
always, WantedBy=default.target.
- the compose service is deleted — one owner per container name.
- deploy_comfyui builds the image, installs the unit, retires the compose-era
unit into ~/.AGENTTRASH, removes the stale container so the unit's own can
take the name, and waits on :8188.
- the healthcheck is a file, because the podman 5.x v1 generator splits
HealthCmd on spaces and an inline `python3 -c "..."` becomes a quoting
puzzle that fails silently.
The comfyui seam now reads the unit instead of compose for devices, command and
groups (and resolves the Containerfile from the deployer's declaration, not
from a directory listing).
Dankserver's node-exporter has been dead since a reboot: `podman ps -a` showed
it `Exited (143)` and Prometheus' `node-dankserver` job has been scraping
nothing. It ran only from `dankserver/compose.yaml`, which is the failure mode
this repository's boot-persistence rule exists for (rootless Podman does not
restart bare containers after the stop that shutdown performs), and the compose
form bound the wildcard address — the unauthenticated-metrics-API-on-every-
interface posture already removed on Dankclaw.
- dankserver/observability/quadlet/node-exporter.container — mirrors the
Dankclaw unit, listening on exactly 127.0.0.1:9100 (host-local probes) and
192.168.0.235:9100 (Prometheus on Dankclaw), with Restart=always and
WantedBy=default.target.
- the compose service is deleted: one owner per container name.
- deploy_observability now installs from "$TARGET/observability/quadlet"; it
hardcoded the Dankclaw path, so a Dankserver unit could not be installed by
the shared dispatcher at all.
- planner: "node-exporter" joins the Dankserver service set, with a path rule
for dankserver/observability/**; the dispatcher's allowlist gains
dankserver:node-exporter.
Seams cover the silent halves: both listen addresses present and the wildcard
absent, the compose service gone, the allowlist entry present, and the planner
mapping the path to the service.
Every measurement behind the frank-medium runbook was taken with scripts that
lived in /tmp on two hosts — reproducible only until the next reboot, and
invisible to anyone reading the repository. This moves the useful four into
scripts/, where this repo already keeps its bench harnesses:
bench-vllm-corpus.py code-shaped corpus from the repo's own sources, because
synthetic random prompts misreport draft acceptance
badly (2.9-3.6 tok/update against ~5.8 on code)
bench-vllm-serve.sh concurrency sweep; encodes the two traps that produced
silent garbage — the tokenizer argument, without which
the client dies resolving "frank-medium" on HF and
reports no numbers at all, and the --ignore-eos
semantics that must not be compared across
bench-turn-latency.py per-turn TTFT for an agent-loop shape; the instrument
that decided the routing policy, and the one that shows
cold-vs-warm prefill directly
bench-llm-soak.py sustained multi-family load; what found the capacity
wall (three 30k-token families, 75% cold re-prefills,
zero failures)
Each header carries its traps and the interpretation; the runbook's Benchmarks
section now points at them instead of describing methodology with no tooling.
Also adds the missing half of the script seam: scripts/*.py are compiled, not
just scripts/*.sh syntax-checked — a harness with a syntax error could otherwise
ship unnoticed until someone needed it during an incident.
Both python harnesses were smoke-tested against the live fleet before commit:
the corpus builder wrote 15 prompts, and the turn harness measured 12.5 s cold
against 1.4 s warm on the same prompt — the cache behaviour the runbook claims.
This test failed on a clean tree for many sessions, and the working belief
became "test 73 is a known failure, ignore it" — which is exactly what a
permanently red suite teaches. The cause was a selector bug: `find
dankclaw -name 'Containerfile*' | head -1` returned whichever Containerfile the
directory listing happened to put first (dankclaw/alloy/Containerfile), so the
assertion read a file that never claimed to be ComfyUI's. It would equally have
passed on a wrong file.
The test now resolves the Containerfile the way the build does — build.context
joined with build.dockerfile, both read from dankclaw/compose.yaml — so it
checks the file the deploy actually builds (dankclaw/Containerfile.comfyui, the
one compose, deploy-remote.sh and the ComfyUI doc all name) and follows a future
rename instead of guessing.
Suite: ALL TESTS PASSED (310), no exceptions to explain away.
The card probing 192.168.0.235:7770 — frank-medium's endpoint since the
radiance cutover — was still called "frank-small", a service retired on
2026-09-15. The probe target was right; only the identity was wrong, which is
the worse half: a green card named after a service that no longer exists teaches
the wrong thing about the fleet. It is now frank-medium, described as what it
actually is (Qwen3.8-27B Uncensored MXFP4, vllm-radiance, two R9700s behind one
endpoint), and frank-large's description stops crediting the retired halogen
runtime for work ik_llama.cpp does.
The seam added to homepage_config.bats pins the cross-file contract that let
this drift: a card's probe target and its name must agree with the fleet, and a
retired service must not keep a card.
The scraped metrics have no rules over them, so every graph was read after the
fact by a human. This adds shared/prometheus/rules/llm-fleet.yml, loaded via
rule_files and installed by the same copier that ships prometheus.yml (a rule
file nothing loads evaluates nothing, and Prometheus reports no error for it):
LlmKvPoolSaturated kv pool > 90% for 10m — the 122,636-token wall
LlmRequestsQueueing waiting > 0 for 10m — MAXSEQS=4 reached
LlmPrefixCacheCollapse reuse < 20% for 15m with traffic — the thrash signature
LlmEndpointDown scrape target down 2m — both replicas gone
LlmLatencyHigh p95 end-to-end > 60s for 10m — what an operator feels
job:llm_prefix_cache_reuse:rate5m — recording rule shared by the alert and the
dashboard, so the two cannot drift
Every threshold is tied to something measured on this fleet, not to a guess,
and the file says plainly at the top that **nothing is notified**: there is no
Alertmanager here, so a firing alert reaches Prometheus' /alerts page, the
ALERTS metric and the dashboard panel added alongside it — and nowhere else.
Choosing how Frank actually gets paged is his call, not a missing config line.
Seam covers the silent failure mode: rules exist, the config references the
directory, the deployer installs it, and the dashboard graphs ALERTS.
Grafana has been running with zero datasources and zero dashboards, so the
metrics scraped last commit were only reachable through the Prometheus API.
This wires the plumbing and one dashboard, both declared in
shared/grafana/provisioning/ and installed to /opt/grafana/provisioning by the
deployer (the unit already mounted that directory read-only, so nothing about
the container changes):
- datasources/prometheus.yml — Prometheus at http://prometheus:9090, the
container name, not the loopback publish, with a fixed uid so dashboards
reference it without a lookup.
- dashboards/llm-fleet.json — KV pool fill, queue depth, prefix-cache reuse
rate, e2e latency p95, and decode/prefill throughput, with an endpoint
variable covering both frank-medium and frank-large. Every series name was
checked against the live /metrics endpoint: this fleet already shipped one
comment naming a metric that does not exist, and a dashboard full of those
would be worse.
Dashboards stay editable in the UI (allowUiUpdates) because the panels are a
starting point, but the file is the source of truth and the deployer
overwrites Grafana's copy on every deploy.
Planner and seams follow the existing pattern: shared/grafana/** selects the
grafana service, and the tests pin the container-name address plus the
provisioning contract so a rename cannot silently empty the dashboard.
Neither LLM endpoint was scraped, and frank-medium exports 418 vllm: series —
including the ones this fleet argues about under load: kv_cache_usage_perc (how
full the per-card pool is), prefix_cache_hit_rate (whether pinned conversations
are being reused), num_requests_waiting (queue depth) and the request-latency
histograms. Everything diagnosed tonight by hand — 122,636 KV tokens per card,
thrash when three large contexts overlap, cold-vs-warm prefix reuse — is a
graph now, on the LAN addresses rather than the guarded vhosts.
frank-large comes along in the same change: llama.cpp exposes throughput
counters under llamacpp:, and watching one LLM endpoint without the other was
arbitrary.
The seam added to observability_quadlet.bats pins the cross-file contract that
breaks silently: if a scrape target drifts from the :7770 the units publish,
the graphs vanish with no error anywhere.
The two replicas behind :7770 were on plain round-robin, which throws away
prefix-cache hits: an agent loop re-sends its system prompt and file context
every turn, and alternating replicas means every other turn pays a cold
prefill. Measured with 90k-token contexts, warm-turn TTFT was 2676 ms.
Pinning everything to one replica is not the answer either — a cold batch
(the shape a benchmark or a burst of parallel subagents produces) then loses a
card's worth of prefill: 16 requests took 30.1 s pinned against 18.6 s spread.
So the policy routes by request shape. A continuation (more than one message —
the agent-loop signature) is pinned to the replica that served its prefix;
anything else spreads across the cards by counter. Measured: 19.97 s for the
burst and 1095 ms warm-turn TTFT — the best of both, without a regression on
either workload.
The policy lives in proxy/routing.py, separate from the HTTP plumbing in
proxy/affinity-proxy.py, so it can be tested without aiohttp, a network or a
GPU. tests/frank_medium_proxy.bats covers it, and it earned its keep: the
first version hashed fresh prefixes to pick a card, which sends every member of
a same-prefix burst to one card (29.8 s, measured), and a stale index in the
pin path returned 500 on every continuation. Both are now assertions.
ROUTING=roundrobin and ROUTING=affinity stay available to re-measure the
choice on this host; the default is adaptive.
The radiance MXFP4 stack cannot run TP=2 on this host. RCCL's collective
kernels carry a hidden hostcall-buffer parameter, and ROCr refuses to
dispatch them on a device without PCIe atomics: the AQL dispatch fails with
hipErrorIllegalState ("the operation cannot be performed in the present
state"), which is what every two-rank collective hit. The second R9700 sits
behind the chipset bridges 05:00.0/06:09.0, which advertise
`AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
atomic routing on that path and the driver's
pci_enable_atomic_ops_to_root() fails for that card. The first card, on CPU
root port 00:01.1, has atomics enabled.
Ruled out: kernel version (7.0.0-28/-30/-31 identical, after a full upgrade),
iommu=pt, P2P hardware (45.3 GB/s between the cards), the shipped librccl (a
from-source rebuild reproduces the failure at the same call path), vLLM and
pynccl (a bare two-rank torch.distributed allreduce fails the same way), all
NCCL transport knobs, and single-rank tests (NCCL short-circuits at
world_size=1, so one-card success proves nothing).
Deployment: three rootless Quadlets — frank-medium-a (:7771, card 1),
frank-medium-b (:7772, card 2), both TP=1 at 114688 context (262144 and
131072 do not fit one card's KV) with vLLM-profiled KV — and
frank-medium-proxy (:7770), a round-robin that preserves the fleet contract
of one endpoint and one served model name. frank-medium.container stays as
the TP=2 target for a host whose second card is atomic-capable; the deploy
handler does not install it and trashes any stale copy.
Measured through the proxy (vllm bench serve, random dataset, --ignore-eos):
conc 1 42.8 tok/s, conc 4 147.6 tok/s, conc 8 230.8 tok/s, 32k prefill at
~2.8k tok/s per card, TPOT p50 23.6-28.8 ms.
Also: healthcheck reads PORT so one probe serves all three units, the hermes
example and the harness models.yml drop to the real 114688 context, and the
runbook records the full evidence chain.
Rootless Quadlet on dankserver serving
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (the radiance stack's
designated alternate target; ships its fp8 MTP head, no rewrite step) with
the DFlash2-FP8 drafter at SPEC=7 on the 2x R9700s (TP=2, host network,
:7770, alias frank-medium unchanged).
- dankserver/frank-medium/: quadlet unit, container entrypoint (fork of
serve-mxfp4.sh's patch block at pinned clone commit 97353293), healthcheck
- dispatcher retires the root llama.cpp unit, re-tags the registry image to
the localhost alias, installs the quadlet, waits on /health
- planner maps dankserver/frank-medium/* to frank-medium; scripts-era
mappings removed
- ROCmFPX-era env/serve/unit/download scripts staged in scripts/.AGENTTRASH
with .rocmfpx suffix; seams updated (frank_medium_quadlet.bats new)
The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48,
gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there at all,
so the `frank-small` service is retired and `frank-medium` takes the endpoint.
New service: charlie12345/ROCmFPX llama.cpp (built for gfx1201 against the
fork's self-contained ROCm 7.14 toolchain, so the host's stale /opt/rocm-6.3.3
never participates) serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF --
a Q8_0_ROCMFPX_AGENT (ftype 115) checkpoint that stock llama.cpp cannot load.
Placeholder config: --split-mode tensor across both cards, 131072 context,
1 slot, f16 KV, MTP draft head on ROCm0.
deploy-remote.sh grows retire_root_llm_small(): the CUDA-era unit is stopped,
disabled and moved to ~/.AGENTTRASH/20260915-frank-small-retire/host-units
before frank-medium starts, because `systemctl disable` has been observed to
delete the unit file itself on this host (2026-08-26 incident).
The retired frank-small repo files (env, unit, serve script) move to
scripts/.AGENTTRASH/, and the planner, dispatcher, service allow-list and the
deployment test seams now name frank-medium.
16-marreta.spec.ts (standard project) asserts the user homepage's Marreta
open link actually reaches the Marreta app through the full guard chain
(Anubis PoW -> Authentik SSO -> app), and explicitly NOT the authentik
login flow. Guards the 2026-09-12 incident where marreta-proxy was created
without a linked Application, so the embedded outpost never loaded it and
the homepage link parked on a login wall.
Wires spec 16 into the standard project testMatch. Verified passing live.
The healthcheck added to the loki image was discarded by the build:
'HEALTHCHECK is not supported for OCI image format and will be ignored'.
podman builds OCI by default, and OCI has no HEALTHCHECK field, so the
container came back with no health at all — worse than the failing check it
replaced, because nothing reports the loss.
The build now passes --format docker, and the seam pins it so a future edit
cannot quietly reintroduce an image with no health field.
The loki unit came up 'unhealthy' after the quadlet cutover. Cause: a unit's
HealthCmd reaches podman as CMD-SHELL, i.e. /bin/sh -c, and the loki image is
distroless — no shell exists in it. The busybox command itself works (probed
by hand: /ready answers 'ready'); it was the shell wrapper that could not run.
podman does not accept an exec-form command through --health-cmd either: it
stores a JSON array as one literal argument.
The command therefore moves into dankclaw/loki/Containerfile as a
HEALTHCHECK ... CMD [...] exec form, which is what that image exists for — it
already ships the static busybox solely so a healthcheck has something to run.
podman reads the image's healthcheck when the unit declares none.
The first deploy of the grafana unit failed: 'stat /root/.config: permission
denied', exit 1, restart loop. Cause — the repo-root .env that the compose
service used to pick up is a gluetun env file, and it carries HOME=/root (and
the WireGuard keys). Pointing the unit's [Service] EnvironmentFile at it sets
HOME for the service, so podman looks for its configuration under /root.
grafana now reads /home/dank/Containers/grafana/env, the per-service file the
deployer seeds from dankclaw/.env with a single-variable filter — the same
shape caddy, pihole, copyparty, romm and seerr use. The seed is validated, so
a missing secret fails the deploy loudly instead of starting a container whose
OAuth login cannot complete. The value was copied to dankclaw/.env, which is
where dankclaw/.env.example already documented it.
prometheus, grafana, loki, alloy and node-exporter ran only from
dankclaw/compose.yaml. Rootless podman does not restart bare containers, so
they did not survive the 2026-09-11 reboot for the amd_iommu=off change and
had to be restored by hand — the same gap that had already cost the fleet
Caddy and Pi-hole, which is why the boot-persistence rule exists.
Each is now a rootless Quadlet unit under dankclaw/observability/quadlet/,
installed and dispatched by deploy_observability in deploy-remote.sh, with
the compose definitions retired so only one owner exists per container name.
Images, mounts, arguments and container names are unchanged, so Caddy's
proxies and Prometheus' own scrape config keep resolving.
Two posture changes, both about the unauthenticated surface:
* The three HTTP UIs published on 0.0.0.0 with no authentication of any kind
(Prometheus even with the lifecycle endpoint enabled). They now publish on
127.0.0.1: the browser path is the guarded franksdank.cloud vhost, Caddy
reaches them by name over dankclaw_default, and the host-side homepage
status timer probes loopback. Nothing legitimately used the LAN port.
* node-exporter bound :9100 on every interface; it now binds exactly its two
consumers, loopback (homepage timer) and the LAN address (Prometheus, which
runs in a container and cannot reach the host's loopback).
alloy gains a reachable health endpoint: its UI listens on the container's
own loopback and alloy 1.x refuses a config block for it, so the compose
service had nothing to probe — the homepage status card for alloy read 'down'
continuously. The unit moves the UI with a command-line flag and publishes
127.0.0.1:12345.
grafana keeps its Authentik OAuth client secret without receiving the whole
.env: systemd expands the value from a [Service] EnvironmentFile, which never
reaches the container.
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.
Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.
The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.
frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.
Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.
Wget-style (no-JS) clients must never receive application bytes from guarded
services: copyparty serves authenticated traffic with anonymous read perms, so
the guard chain is the only boundary between an unauthenticated client and the
file bytes. Assertions target leakage markers in app code (Copyparty UI title,
homepage panel grid, services.json JSON shape), not gate internals, so any
guard-route regression fails the suite while any gate implementation passes.
Also route 15-*.spec.ts into the guest project (empty storage state).
FrankBook-Pro becomes the only *arr/qBittorrent deployment. Every fleet
container (qB + five arrs) now shares gluetun's netns, so all egress —
torrent, indexer APIs, metadata lookups — exits the WireGuard tunnel by
construction (AllowedIPs=0.0.0.0/0); the arrs' own ports publish on the
gluetun unit, none on the arr units themselves.
- dankserver/compose.yaml: remove the six arr services (the Aug 20 compose
copy — never healthy, superseded; config dirs trashed on the host)
- dankclaw/Caddyfile + shared/homepage/services.json: arr vhosts and
probes repoint 192.168.0.235 -> 192.168.0.112 (FrankBook-Pro)
- frankbook-pro units: five arrs join gluetun netns (Wants/After/Network);
gluetun publishes 9696/8989/7878/8686/8787 alongside 8080/6881
- tests/frankbook_quadlet.bats: new VPN egress seam — every arr+qB unit
must share gluetun's netns and publish no port; gluetun must publish the
whole fleet's ports
- deploy-plan.py: gluetun changes carry the whole netns family (a gluetun
recreation kills all six joiners); dankserver compose set loses the arrs
- AGENTS.md: deployment is complete only when unit tests AND the Playwright
E2E suite both pass
Live run against the fleet showed the outpost redirects direct navigation to
auth.franksdank.cloud with the guarded host preserved as redirect_uri, rather
than rendering the flow UI in place. The security property under test is that
an unauthenticated visitor never sees app content; assert that shape (uidField
visible + no leak markers + gate is either in-place or a redirect carrying the
return path) instead of hostname preservation.
14-guard.spec.ts: every authentik-guarded vhost (www, search, romm, files,
seerr, grafana, prometheus, loki) must challenge an unauthenticated direct
navigation; plex is asserted exempt (PMS answers directly, no IdP form).
11-admin.spec.ts: admin direct-nav to grafana/prometheus/loki passes through
with host preserved and no uidField challenge.
frank-default-large -> frank-large (Dankclaw :7770), frank-default-small ->
frank-small (Dankserver :7770), frank-default-sightsound -> frank-npu
(Dankclaw :7771). Units, env files, serve scripts, deploy-plan mappings,
deploy-remote dispatch (stop/disable old units + relink + enable new), and the
test seams move together. Retired frank-default-npu.* artifacts moved to
scripts/.AGENTTRASH/ (tracked-in-history; ignored on disk).
Served aliases become frank-large / frank-small via --alias; the NPU endpoint
serves the whole FLM registry, so its rename changes no served tags.
Evidence from the failed first cutover: podman top shows container root
maps to dank (1000) under UserNS=host rootless (subuid range 100000:65536
cannot express 997), so PUID=997 fell back to root and PMS crash-looped on
write access to its 997:984-owned data root. Fleet model instead: PUID=0/
PGID=0 (container root = dank, same as jellyfin/copyparty data roots) and
deploy_plex chowns /var/lib/plexmediaserver to 1000:1000 inside the
first-cutover block. Zero bytes moved; rollback regains one chown -R
997:984 step, documented in Plex.md. Seam test pins PUID=0/PGID=0 and the
absence of any userns key.
podman 5.7 podman-user-generator on dankserver hard-fails the whole unit on
the unknown key ('unsupported key UsernsMode in group Container'), so the
first cutover attempt masked the native unit and then found no plex.service
at all. Swap both units to UserNS=host (same --userns=host flag), update the
ownership-contract seam test to pin the exact key, and correct both docs.
Pin the seam: the v1 generator rejects unknown Container keys silently at
deploy distance — the test now pins UserNS=, not the misspelled alias.
- deploy_plex(): install unit, stop+mask native plexmediaserver.service on
first cutover (mask, never disable), gate on /identity for 180s
- deploy_tautulli(): install unit, stop+disable snap on first cutover,
seed config.ini with PMS_Web_Host=plex only if absent
- allow-lists (deploy.sh valid_service, deploy-remote.sh service_is_allowed),
planner routing (SERVICE_ORDER, DANKSERVER_SERVICES, quadlet_services),
test-changed cutover case, quadlet seam enumeration + new
userns/PUID=997/PGID=984 ownership-contract test
The loki/grafana/seerr proxy providers were script-provisioned with no
flows and no Application: null flows made the embedded outpost's Rust
deserializer reject the whole config (all guarded vhosts down after any
outpost reload), and the missing apps bounced post-login to authentik's
user UI instead of the app. Backfilled both live; documented the two
provisioning rules and updated the provider inventory. SSO e2e now
asserts the app landing (pathname leaves /if/, no IdP markers in body)
instead of only the hostname.
Seven units installed INSIDE the Mac VM (/home/core/.config/containers/
systemd) by a new deploy_frankbook_stack handler: gluetun (ProtonVPN
WireGuard, env-file credentials, publishes qbit's 8080/6881), qbittorrent
(Network=gluetun.container, verbatim /Volumes data roots, first-boot
config seed with fresh machine-local WebUI password and the Finished
Torrents category), and prowlarr/sonarr/radarr/lidarr/readarr (grab-only,
config migrated from dankserver at provisioning, readarr pinned to
amd64-1.0.980). deploy.sh routes the new target through the ssh-config
alias with REMOTE_SHELL=/usr/local/bin/bash (macOS bash 3.2 lacks
associative arrays); the planner learns FRANKBOOK_SERVICES and the
gluetun->qbittorrent netns pairing; new seam test pins the contract.
The cutover unit joined dankserver_appnet but dropped the compose-era
0.0.0.0:3923 mapping (live inspect: bridge, 3923 published), so
copyparty listened inside the netns while the host refused. New seam
asserts every host-facing port matches the pre-cutover PortBindings:
romm 8085->8080, jellyfin 8096, calibre-web 8083, copyparty 3923.
The podman 5.x v1 quadlet generator rejected the whole jellyfin unit on
the unsupported Device= key (generator log: unsupported key 'Device' in
group 'Container'), so no jellyfin.service was generated and the cutover
restart failed with 'Unit not found' after the data root had already
migrated cleanly. gluetun proves AddDevice= is the house key. New seam
test denies Device= across all six dankserver units and pins the
AddDevice=/dev/dri replacement.
Cutover of the six surviving dankserver services from compose-era bare
containers to boot-persistent Quadlet units (WantedBy=default.target; podman
5.x v1 generator honors it at user-manager start).
Units are faithful to live inspect ground truth (image pins = running tags,
not compose inventory):
- romm trio shares one data root: /home/dank/Containers/romm mounted whole at
/romm; mariadb datadir and redis data dir live inside it. Aliases mariadb/
redis preserved on dankserver_appnet via Network=...:alias=. Secrets stay in
the host-local env file (EnvironmentFile + ${VAR} expansion in the generated
ExecStart, gitea precedent). copyparty passes its admin account via
Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD} (Exec= not Command=: the v1
generator hard-fails on Command=); entropy stays rw only there.
- jellyfin keeps every creation-time env as Environment= lines (zero drift);
Device=/dev/dri + GroupAdd=keep-groups preserve hardware transcoding.
- calibre-web pins the exact running image tag (id 986277563e76 =
0.6.27-ls397), pasta network preserved.
Deployer: first-cutover handlers retire the old container, rescue unshadowed
anonymous-volume content (podman volume export + tar --skip-old-files), mv
data roots on the same fs, then start units in dependency order with health
gates. Re-deploys restart only the selected unit. deploy-plan.py maps compose
keys mariadb/redis to the new unit names; allowlists updated. New seam test
tests/dankserver_quadlet.bats pins boot-persistence + v1-generator contract;
test-changed.sh maps the new dirs to it. Full suite green (209 tests).
Every Caddy-proxied service is now either behind the auth chain or
explicitly internal-only, with the provider-first flip procedure codified.
The comfyui GPU-access seam test now asserts the deployed contract
(group_add keep-groups + udev MODE 0666 on kfd/renderD*) instead of the
pre-fix 'video' group assumption.
The redirect-to-flow design with an absolute cross-host next= was rejected by
the flow's open-redirect guard at completion ("Request has been denied:
Invalid next URL"), and meta-header routing does not exist in this build
(providers route on the ORIGINAL Host). :8016 now proxies every request whole
to authentik-server:9000 preserving Host; the embedded outpost serves login
in place and forwards authenticated traffic back to :8017. Requires one proxy
provider per guarded host (provisioned live: search/hermes/prometheus/romm/
files + www, flows backfilled). Tests flipped from the meta-header claim to
the upheld contract: Host preserved, no absolute next= redirects.
The proxy provider matches external_host exactly, so Host=www matched zero
providers and Authentik answered its error page while the naked domain
served the homepage. Flip the topology: www.franksdank.cloud is the one
guarded vhost carrying the homepage through Anubis -> Authentik -> :8017;
franksdank.cloud (plus legacy names and the bare-IP catch-all) 308-redirects
onto www before any challenge. Tests flipped to assert the new contract.
The forward_auth target /outpost.goauthentik.io/auth/caddy does not exist
in authentik 2026.8 (verified live: 404 for every header combination; the
URL tree has no outpost auth prefix at all — the outpost worker dials back
to core over ws/outpost/<uuid>/ instead). Unauthenticated subrequests got
404, Caddy treated >=400 as 'not authenticated' and redirected browsers to
the dead path — users saw authentik's own Not-Found page.
Fix: guarded vhosts now proxy the whole request through authentik-server:9000
with X-authentik-meta-outpost/-provider/-app headers (the exact header set
this build's Traefik controller emits, adapted for Caddy). Unauthenticated
clients get the login flow served in place; once a session exists,
authentik proxies on to the provider's internal_host — now
http://caddy:8017, a new post-auth internal listener that routes straight to
the app. The chain terminates; nothing dials back into authentik.
Live DB changes applied alongside (postgres is the source of truth):
provider pk=1 authentication_flow=default-authentication-flow (was NULL —
the actual cause of the 404s) and internal_host=http://caddy:8017 (was
self-referential http://authentik-server:9000).
Three compounding deploy-handler bugs found by live deployment:
- podman image exists/pull used a mixed-case ghcr.io ref that podman's
reference parser rejects before any network I/O (repository name must be
lowercase); all refs now lowercase — ghcr resolves case-insensitively.
- the unit runs the image under a localhost/ alias tag: this host's podman
5.7 quadlet path fails container create with 'image not known' for
registry-prefixed refs even when present in storage (verified live).
- the health probe was shell-form and the distroless ko-build image has no
shell; HealthCmd is now a JSON exec-form array, which passes through the
quadlet generator's argv assembly verbatim.
Tests updated to assert the alias-tag reality plus the pull guard.
podman's reference parser rejects mixed-case repository refs before any
network I/O ('repository name must be uppercase'); upstream's own install
docs pull lowercase (ghcr resolves case-insensitively). The first deploy
of this chain exit-2'd on that parse error — the image was not yet
pulled, so the guarded vhosts 502 until this deploy lands.
Frank