The 2026-09-12 homepage outage slipped through because the live-fleet
Playwright suite was a manual 'MUST invoke' that deploys skipped. Wire it
into deploy.sh as the [3/3] gate: after a behavior-affecting deploy,
run tests/e2e/run.sh against the live fleet; a failure exits DEPLOY
INCOMPLETE. Add --skip-e2e as an emergency escape hatch. Update the
AGENTS.md deployment-completeness contract and flow diagram to match.
Verified: bash -n clean, --skip-e2e parsed, unknown flags rejected,
seam suite green (only the pre-existing comfyui test 73 fails).
Deployed live at 91e9410 (gluetun+marreta+caddy), authentik provider pk 13,
guard chain verified end-to-end. Documented that Laravel APP_KEY must carry the
base64: prefix — a bare base64 key 500s every app route with 'Unsupported
cipher or incorrect key length' while /up still passes.
The scoped runner (deploy-remote.sh) already knew marreta; the top-level
deploy.sh valid_service() gate did not, so --service marreta was rejected
before reaching the planner.
Verbose engine logging is the vendor's first-line evidence for load,
allocation and admission questions; shipping it on means an incident
report already carries the full startup account instead of needing a
restart-into-debug. HALOGEN_DMALLOC_LOG stays off (every alloc >64MB is
steady-state noise on a shared host). Doc updated with the rootless
podman-logs-vs-journalctl retrieval note.
Rewrites the [[Observability]] umbrella page and the five component pages for
what is actually deployed: rootless Quadlet units instead of compose services,
the loopback-only publish posture (node-exporter's LAN binding explained), the
loki healthcheck living in its image because the image has no shell, and the
Home=/root trap that killed the first grafana start.
Adds the note that matters most to a reader: the stack is installed, healthy
and UNCONFIGURED. Grafana reports zero datasources and zero dashboards with an
empty provisioning directory; Prometheus has no rule files and no Alertmanager;
Loki is configured but has no reader; node-exporter runs its default
collectors. Alloy is the only configured piece. Metrics and logs are collected
and retained, and nothing is visualized or alerted on.
AGENTS.md's boot-persistence gap note is replaced by the resolution.
The healthcheck added to the loki image was discarded by the build:
'HEALTHCHECK is not supported for OCI image format and will be ignored'.
podman builds OCI by default, and OCI has no HEALTHCHECK field, so the
container came back with no health at all — worse than the failing check it
replaced, because nothing reports the loss.
The build now passes --format docker, and the seam pins it so a future edit
cannot quietly reintroduce an image with no health field.
The loki unit came up 'unhealthy' after the quadlet cutover. Cause: a unit's
HealthCmd reaches podman as CMD-SHELL, i.e. /bin/sh -c, and the loki image is
distroless — no shell exists in it. The busybox command itself works (probed
by hand: /ready answers 'ready'); it was the shell wrapper that could not run.
podman does not accept an exec-form command through --health-cmd either: it
stores a JSON array as one literal argument.
The command therefore moves into dankclaw/loki/Containerfile as a
HEALTHCHECK ... CMD [...] exec form, which is what that image exists for — it
already ships the static busybox solely so a healthcheck has something to run.
podman reads the image's healthcheck when the unit declares none.
The first deploy of the grafana unit failed: 'stat /root/.config: permission
denied', exit 1, restart loop. Cause — the repo-root .env that the compose
service used to pick up is a gluetun env file, and it carries HOME=/root (and
the WireGuard keys). Pointing the unit's [Service] EnvironmentFile at it sets
HOME for the service, so podman looks for its configuration under /root.
grafana now reads /home/dank/Containers/grafana/env, the per-service file the
deployer seeds from dankclaw/.env with a single-variable filter — the same
shape caddy, pihole, copyparty, romm and seerr use. The seed is validated, so
a missing secret fails the deploy loudly instead of starting a container whose
OAuth login cannot complete. The value was copied to dankclaw/.env, which is
where dankclaw/.env.example already documented it.
prometheus, grafana, loki, alloy and node-exporter ran only from
dankclaw/compose.yaml. Rootless podman does not restart bare containers, so
they did not survive the 2026-09-11 reboot for the amd_iommu=off change and
had to be restored by hand — the same gap that had already cost the fleet
Caddy and Pi-hole, which is why the boot-persistence rule exists.
Each is now a rootless Quadlet unit under dankclaw/observability/quadlet/,
installed and dispatched by deploy_observability in deploy-remote.sh, with
the compose definitions retired so only one owner exists per container name.
Images, mounts, arguments and container names are unchanged, so Caddy's
proxies and Prometheus' own scrape config keep resolving.
Two posture changes, both about the unauthenticated surface:
* The three HTTP UIs published on 0.0.0.0 with no authentication of any kind
(Prometheus even with the lifecycle endpoint enabled). They now publish on
127.0.0.1: the browser path is the guarded franksdank.cloud vhost, Caddy
reaches them by name over dankclaw_default, and the host-side homepage
status timer probes loopback. Nothing legitimately used the LAN port.
* node-exporter bound :9100 on every interface; it now binds exactly its two
consumers, loopback (homepage timer) and the LAN address (Prometheus, which
runs in a container and cannot reach the host's loopback).
alloy gains a reachable health endpoint: its UI listens on the container's
own loopback and alloy 1.x refuses a config block for it, so the compose
service had nothing to probe — the homepage status card for alloy read 'down'
continuously. The unit moves the UI with a command-line flag and publishes
127.0.0.1:12345.
grafana keeps its Authentik OAuth client secret without receiving the whole
.env: systemd expands the value from a [Service] EnvironmentFile, which never
reaches the container.
Hermes hardcodes response_format=json_schema on its title call. No local
back-end implements constrained decoding — frank-small and frank-large both
answer 400 'response_format json_object/json_schema is not supported' — so
every session logged a title failure and fell back to a message-derived
title. The NPU endpoint that used to absorb this call was retired on
2026-09-11.
The call now goes out over the Responses transport (the only one that does
not forward response_format) with thinking off. A 64-token cap cannot hold a
thinking pass plus a title: with the default reasoning effort the model spent
all 64 tokens thinking and returned empty content, which is why disabling
reasoning is part of the fix and not decoration.
Verified against the live endpoint with the real prompt and the real
extraction path: raw {"title": "Friendly greeting"} -> title
'Friendly greeting' in 3.4s, where the previous configuration returned 400.
Rebooting Dankclaw for the amd_iommu=off change exposed five services that
exist only as compose.yaml services with no Quadlet unit (prometheus, grafana,
loki, alloy, node-exporter): they did not come back and had to be restored by
hand. The boot-persistence rule already requires units for them; this records
the concrete gap so the next reboot is not a surprise.
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.
Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.
The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.
frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.
Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.
The unit comment always claimed the webui drives the same agent state as
the gateway, but the image bakes HERMES_HOME=/opt/data and no unit
overrode it. Podman auto-creates a fresh anonymous volume at every
image-VOLUME point on container recreation, so each restart reset the
webui's config home to the image-default template (provider: auto ->
openrouter.ai with no API key), while the gateway read the corrected
/data/config.yaml. Pin HERMES_HOME=/data so both processes share one
config/state root and config edits propagate without a recreate.
Live config was still routing on pre-rename aliases (frank-default-large,
frank-default-small) after the 2026-09-03 frank-large/frank-small/frank-npu
rename cutover; hermes-webui resolved those ids to nonexistent models and
requests failed. Live config fixed in place (backup kept on-host as
config.yaml.bak-20260905); this backup now mirrors the live shape: named
providers for aux routes, title_generation/triage/kanban routed to their
actual models, moa + mcp sections, signal disabled.
Wget-style (no-JS) clients must never receive application bytes from guarded
services: copyparty serves authenticated traffic with anonymous read perms, so
the guard chain is the only boundary between an unauthenticated client and the
file bytes. Assertions target leakage markers in app code (Copyparty UI title,
homepage panel grid, services.json JSON shape), not gate internals, so any
guard-route regression fails the suite while any gate implementation passes.
Also route 15-*.spec.ts into the guest project (empty storage state).
Flip the homepage Sonarr card from the user category to admin and repoint
its link at the LAN-direct port (192.168.0.112:8989). Retire the unguarded
public vhost sonarr.franksdank.cloud (direct reverse_proxy, no auth) and its
post-auth @sonarr handler from the Caddyfile; drop the Pi-hole addn-hosts A
record with it. No public Porkbun record ever existed (NXDOMAIN at the zone),
so after this there is no WAN path to Sonarr: users request TV via Seerr
(guarded), admins reach Sonarr's own UI from the LAN only.
Fixes the seerr/sonarr mix-up from 19b2589/e0bade6.
FrankBook-Pro becomes the only *arr/qBittorrent deployment. Every fleet
container (qB + five arrs) now shares gluetun's netns, so all egress —
torrent, indexer APIs, metadata lookups — exits the WireGuard tunnel by
construction (AllowedIPs=0.0.0.0/0); the arrs' own ports publish on the
gluetun unit, none on the arr units themselves.
- dankserver/compose.yaml: remove the six arr services (the Aug 20 compose
copy — never healthy, superseded; config dirs trashed on the host)
- dankclaw/Caddyfile + shared/homepage/services.json: arr vhosts and
probes repoint 192.168.0.235 -> 192.168.0.112 (FrankBook-Pro)
- frankbook-pro units: five arrs join gluetun netns (Wants/After/Network);
gluetun publishes 9696/8989/7878/8686/8787 alongside 8080/6881
- tests/frankbook_quadlet.bats: new VPN egress seam — every arr+qB unit
must share gluetun's netns and publish no port; gluetun must publish the
whole fleet's ports
- deploy-plan.py: gluetun changes carry the whole netns family (a gluetun
recreation kills all six joiners); dankserver compose set loses the arrs
- AGENTS.md: deployment is complete only when unit tests AND the Playwright
E2E suite both pass
Rootless podman on dankserver already maps container root (PUID=0) to
the dank user (uid 1000) by default — /proc/<pid>/uid_map reads
0->1000, 1..->100000+. Verified across plex, tautulli, jellyfin,
calibre-web, copyparty, romm. userns_mode: host is unnecessary, and
podman-compose 1.5.0 cannot apply it anyway (compose services run in
pods; --userns and --pod cannot be set together). The real fix for the
never-booted arr stack was chowning the root-owned /opt/<svc>/config
dirs to 1000:1000.
The arr services were committed (b9de886) with PUID=0 but no userns_mode.
Under rootless podman's default subuid mapping, container-root cannot write
host-root-owned bind mounts, so every app died in logger init
(UnauthorizedAccessException on /config/Sentry) before binding its port —
the stack 502'd since it was first brought up (2026-09-04 00:33).
Per the fleet convention (AGENTS.md, plex/tautulli migration): PUID=0 +
UserNS=host maps container-root to dank (1000), which owns /opt data roots
and /mnt/entropy. Config dirs chowned 1000:1000 at bring-up.
Also drops a duplicate 'networks: [appnet]' key on prowlarr.
The arr stack (prowlarr, radarr, sonarr, lidarr, readarr, qbittorrent)
had A records in addn-hosts but no Caddyfile blocks. This adds:
- Six public vhost blocks in the dankserver section, each proxying to
192.168.0.235:<port> (exempt by construction: own auth, no Anubis)
- Six @arr named handlers inside :8017 for authenticated traffic routing
All 233 bats tests pass.
Boot tag flips from qwen3.8-distilled-heretic:9b to nuextract3:4b (verified
live: per-request routing already serves every registered tag; the boot tag
only decides which model is resident at start, so the first request after a
swap pays the load cost). FLM_CTX_LEN=-1 drops the global --ctx-len override
so each pack loads with its own trained context (nuextract3 32768, gemma4
131072).
Live run against the fleet showed the outpost redirects direct navigation to
auth.franksdank.cloud with the guarded host preserved as redirect_uri, rather
than rendering the flow UI in place. The security property under test is that
an unauthenticated visitor never sees app content; assert that shape (uidField
visible + no leak markers + gate is either in-place or a redirect carrying the
return path) instead of hostname preservation.
Mechanical identity rename across the knowledge base, AGENTS.md, README, the
hermes config example, and homepage services.json. No behavior change; served
aliases on the LLM endpoints change with the deploy of commit 69f87c8.
14-guard.spec.ts: every authentik-guarded vhost (www, search, romm, files,
seerr, grafana, prometheus, loki) must challenge an unauthenticated direct
navigation; plex is asserted exempt (PMS answers directly, no IdP form).
11-admin.spec.ts: admin direct-nav to grafana/prometheus/loki passes through
with host preserved and no uidField challenge.
frank-default-large -> frank-large (Dankclaw :7770), frank-default-small ->
frank-small (Dankserver :7770), frank-default-sightsound -> frank-npu
(Dankclaw :7771). Units, env files, serve scripts, deploy-plan mappings,
deploy-remote dispatch (stop/disable old units + relink + enable new), and the
test seams move together. Retired frank-default-npu.* artifacts moved to
scripts/.AGENTTRASH/ (tracked-in-history; ignored on disk).
Served aliases become frank-large / frank-small via --alias; the NPU endpoint
serves the whole FLM registry, so its rename changes no served tags.
Post-change doc reconciliation for the hermes vhosts + DNS deletion batch:
- Hermes-Dashboard.md: access is now the hermesadmin.franksdank.cloud vhost
(direct Caddy proxy to 192.168.0.143:9119, no guard chain — exempt because the
dashboard enforces its own basic-auth login), plus the raw LAN port. The old
text claimed an Authentik-guarded public vhost, which was decommissioned.
- Hermes-WebUI.md: description updated — the webui is now also reachable via the
unguarded hermes.franksdank.cloud vhost (its own password auth is the gate).
- LAN-DNS.md: "add a name in two places" corrected to a two-or-three-step
decision; internal-only names (prometheus/loki/grafana/seerr) deliberately get
no public A record and resolve on the LAN only, and DNS-01 issuance is
unaffected because the challenge is a TXT record in the same zone.
- Seerr.md: DNS row corrected — addn-hosts only since 2026-09-02; NXDOMAIN on WAN.
- hermes-dashboard.container: unit comment no longer claims Authentik termination.
No behavior change in this commit; tests 233/233 green (docs-completeness seam included).
Both Hermes services get direct (unguarded) vhosts: each authenticates itself
(webui password; dashboard basic_auth in /data/config.yaml), so no Anubis
challenge and no Authentik proxy provider is required. Upstreams are the host
LAN IP because Caddy itself sits on dankclaw_default, where the gateway
address 10.89.1.1 has nothing serving 8787. Internal-only by DNS absence:
hermes.franksdank.cloud already resolves via Pi-hole addn-hosts; hermesadmin
is added in this same commit so the caddyfile/addn-hosts cross-check (test 178)
stays green.
New page Architecture/Container Network Namespaces.md: which local services are
addressable from which Podman network namespace on Dankclaw (rootless podman +
netavark). Verified live 2026-09-02: gateway/webui/CDP run in the host netns and
dial loopback; hermes-dashboard is the only Hermes container on a bridge network
(dankclaw_default, 10.89.1.0/24 gw 10.89.1.1) and can dial published ports only —
unpublished host-netns listeners (signal-cli :8080, webui :8787) are refused from
that namespace even though both bind 0.0.0.0.
Corrected while verifying the above:
- Hermes.md: data root is /home/dank/Containers/hermes/data (not the retired
/home/dank/hermes-data); gateway dials the Signal daemon on loopback; the
daemon itself is a host user unit (signal-cli-daemon.service) bound to
0.0.0.0:8080, so the Signal HTTP API is LAN-reachable.
- Hermes-Dashboard.md: it is a Quadlet unit (Network=dankclaw_default), not a
compose service; data root + /opt/data anonymous volume corrected.
- Hermes-WebUI.md: binds 0.0.0.0:8787 in the host netns (verified: unauthenticated
GET /health from another LAN machine returned 200) — the SSH tunnel is an access
path, not a containment boundary; HealthCmd and data-path corrections.
- Network Architecture.md: no hermes vhost exists in either the tracked or the live
Caddyfile (hermes.franksdank.cloud resolves but routes nowhere); SearXNG exits the
tunnel via the shared named network searxng.network, not service:gluetun.
Docs only — deploy is a no-op for this change.
PMS 1.43 serves TLS-only on 32400: a plain-HTTP GET gets a TLS alert and
curl reports "empty reply from server" (exit 52, code 000), so the card read
down while Plex answered 200 at /web/index.html over HTTPS. TCP connect is the
honest liveness signal for this port; the public URL stays HTTP because Caddy
terminates TLS and re-keys the path.
The Hermes card probed 127.0.0.1:8090, a port retired when the hermes
stack moved from compose (8090 web dashboard) to quadlets: the dashboard
now publishes 9119 and the webui listens on 0.0.0.0:8787 in the host netns.
Nothing has listened on 8090 since, so the card reported "down" while the
gateway, dashboard, and webui were all running.
Probe 127.0.0.1:8787 instead — the webui runs HERMES_HOME=/data, the same
agent state as the gateway, and /health answers only when that server is
alive (uptime, sessions, active runs). The prober runs on Dankclaw, so the
loopback probe hits the host netns where the webui binds.
The C5 probe failed for the wrong reason: plex.container lost its Network=
key in the same edit churn that ate tautulli's ContainerName. Plex ran on
the default podman network — published 32400 still worked (clients fine)
but tautulli could not resolve the plex alias, breaking the seeded
PMS_Web_Host=plex. Restore appnet membership; structure now parallel to
tautulli.container. Lesson recorded: after multi-hunk line edits on these
unit files, diff the whole file — key-order churn has eaten three keys
already.
Without ContainerName= the v1 generator names the container
systemd-tautulli; the fleet references containers by their plain names
(podman inspect/exec, deployer health checks). Container was healthy but
renamed — caught by post-cutover verification, not by the seam; the seam
only asserts file keys, so pin the name here and keep plex/tautulli
structures parallel.
- tautulli.container: HealthCmd switches to the repo's proven curl pattern
(jellyfin unit) — quadlet's systemd-style value parsing mangles embedded
double quotes; PublishPort=8181:8181 restored (a mis-aimed line edit
replaced it; the PublishPort seam test caught the regression)
- deploy_plex readiness gate: 36x5s -> 120x5s; the 2026-09 cutover showed
the first container boot of a years-old library runs ~10 min while PMS
pre-binds 32400 and empty-replies; 180s failed the gate twice with a
healthy, still-booting service
Evidence from the failed first cutover: podman top shows container root
maps to dank (1000) under UserNS=host rootless (subuid range 100000:65536
cannot express 997), so PUID=997 fell back to root and PMS crash-looped on
write access to its 997:984-owned data root. Fleet model instead: PUID=0/
PGID=0 (container root = dank, same as jellyfin/copyparty data roots) and
deploy_plex chowns /var/lib/plexmediaserver to 1000:1000 inside the
first-cutover block. Zero bytes moved; rollback regains one chown -R
997:984 step, documented in Plex.md. Seam test pins PUID=0/PGID=0 and the
absence of any userns key.
podman 5.7 podman-user-generator on dankserver hard-fails the whole unit on
the unknown key ('unsupported key UsernsMode in group Container'), so the
first cutover attempt masked the native unit and then found no plex.service
at all. Swap both units to UserNS=host (same --userns=host flag), update the
ownership-contract seam test to pin the exact key, and correct both docs.
Pin the seam: the v1 generator rejects unknown Container keys silently at
deploy distance — the test now pins UserNS=, not the misspelled alias.
- runtime rows: podman quadlet under the systemd user generator, pinned
image digests, in-place bind mounts, identity model, appnet aliases
- cutover/rollback section on Plex.md (mask-native-unit lever, decommission
gate); drop the false 'updates are apt-only' and 'snap refresh' gotchas
- update path is now deploy.sh dankserver --service plex|tautulli
- plex: lscr.io/linuxserver/plex:1.43.3.10896-cb3ebc72d-ls322 (verified ==
ghcr latest digest; plan's ls9 pin superseded by the re-verify clause),
/var/lib/plexmediaserver bind-mounted in place as /config, /mnt/entropy
mounted ro at the same path, PUID=997/PGID=984 + UsernsMode=host so the
container's file identity is byte-compatible with the native unit
- tautulli: lscr.io/linuxserver/tautulli:v2.18.1-ls242 (verified == ghcr
latest digest), fresh /home/dank/Containers/tautulli config root,
PUID=1000/PGID=1000, python3 /home healthcheck
- both on dankserver_appnet with aliases; 32400/8181 published as before
- deploy_plex(): install unit, stop+mask native plexmediaserver.service on
first cutover (mask, never disable), gate on /identity for 180s
- deploy_tautulli(): install unit, stop+disable snap on first cutover,
seed config.ini with PMS_Web_Host=plex only if absent
- allow-lists (deploy.sh valid_service, deploy-remote.sh service_is_allowed),
planner routing (SERVICE_ORDER, DANKSERVER_SERVICES, quadlet_services),
test-changed cutover case, quadlet seam enumeration + new
userns/PUID=997/PGID=984 ownership-contract test
The loki/grafana/seerr proxy providers were script-provisioned with no
flows and no Application: null flows made the embedded outpost's Rust
deserializer reject the whole config (all guarded vhosts down after any
outpost reload), and the missing apps bounced post-login to authentik's
user UI instead of the app. Backfilled both live; documented the two
provisioning rules and updated the provider inventory. SSO e2e now
asserts the app landing (pathname leaves /if/, no IdP markers in body)
instead of only the hostname.
- qB 5.x's 204 login breaks Readarr's retired client at runtime; 4.6.7 answers
legacy 200+Ok. Pin recorded in the unit and documented.
- Prowlarr download client created via schema template; all five clients test
200 through the podman gateway.
- PBKDF2 gotcha: qB 5.x conf rewrite left a hash 4.6.7 rejects; fresh rewrite
of the same line restored login.
- E2E: torrent through the Panama tunnel at ~17 MB/s, completed into Finished
Torrents, deleted with data.
podman 5.7.0 refuses ghcr.io/goauthentik/ldap:2026.8.0's nested manifest
list ('Unexpectedly received a manifest list instead of a manifest for a
single image'); pin the linux/amd64 single-platform digest instead.
Document the Mac *arr/qBittorrent stack (VM placement, gluetun credential
gate, socat LaunchAgents, PBKDF2 qB seed) and log the day: dankserver
recovery + full quadlet cutover of the six media services, the frankbook
skip-propagation/PBKDF2 handler fix, the 7-service frankbook deploy, and
the jellyfin GPU-parity finding.
The frankbook handler ran port verification even for services the
credential gate skipped, so a parked qbittorrent failed the deploy and
aborted the five arrs. Gate now lives Mac-side (test -s gluetun/env
inside the VM) before restart+verify.
The qB first-boot seed wrote a plaintext WebUI\Password, which qB >= 4.2
ignores (PBKDF2 @ByteArray only), and omitted HostHeaderValidation=false,
which would 401 the LAN forwarder path. Seed now generates the PBKDF2
hash in the VM (python3 hashlib) and sets ServerDomains=* +
HostHeaderValidation=false. Arr categories (lidarr/radarr/tv-sonarr)
pre-created with save_path=/Volumes/COMPLETE/Finished Torrents so any
grab lands in the review dir regardless of wiring details.
Seven units installed INSIDE the Mac VM (/home/core/.config/containers/
systemd) by a new deploy_frankbook_stack handler: gluetun (ProtonVPN
WireGuard, env-file credentials, publishes qbit's 8080/6881), qbittorrent
(Network=gluetun.container, verbatim /Volumes data roots, first-boot
config seed with fresh machine-local WebUI password and the Finished
Torrents category), and prowlarr/sonarr/radarr/lidarr/readarr (grab-only,
config migrated from dankserver at provisioning, readarr pinned to
amd64-1.0.980). deploy.sh routes the new target through the ssh-config
alias with REMOTE_SHELL=/usr/local/bin/bash (macOS bash 3.2 lacks
associative arrays); the planner learns FRANKBOOK_SERVICES and the
gluetun->qbittorrent netns pairing; new seam test pins the contract.
Two live-verified loader failures on Dankserver (2026-09-01), each a
clean exit at load (no swap-thrash, 37 GB fits RAM):
1. ornith-ai/Ornith-1.5-35B-A3B-FP8 (compressed-tensors PER-CHANNEL fp8):
ValueError: Missing MoE expert source layers: {'gate_up': [0, 1, ...]}
— nonstandard expert tensor naming, provider finds no expert sources.
2. lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8
(compressed-tensors BLOCK-fp8 128x128, standard per-expert naming):
RuntimeError: The size of tensor a (2048) must match the size of
tensor b (...) in _dequant_fp8_weight — the dequant path multiplies
weight * weight_scale directly and cannot expand compressed-tensors
block scales.
Empirical matrix for FreeToken v0.1.2 on qwen3_5_moe (RTX 3090):
modelopt block-fp8 (quant_method 'fp8' + weight_scale_inv) -> LOADS
compressed-tensors per-channel FP8 -> fails
compressed-tensors block-fp8 128x128 -> fails
batsclamp/Huihui-...-abliterated-FP8 is the proven uncensored checkpoint;
restoring it until a modelopt/NVFP4 requant of a wanted model exists or
FreeToken learns compressed-tensors.
Phase 1/2 of the podman standardization plan: service pages updated to the
quadlet reality (units, Containers data roots, env-file secret convention,
the v1-generator findings each caught live), Storage Policy documents the
two rootless ownership regimes and the merge-move migration, Deployment
Flow's service-manager table no longer claims dankserver media is bare
inventory-only. Bundle change log entry added.
Ornith-1.5-35B-A3B-FP8 failed to load on FreeToken v0.1.2: its
compressed-tensors checkpoint uses a nonstandard expert tensor layout
(FreeToken: 'Missing MoE expert source layers: gate_up'). Retarget to
the HauhauCS Aggressive FP8 finetune of the SAME Qwen3.6-35B-A3B arch
Huihui used, which carries compressed-tensors BLOCK-fp8 (128x128) with
the standard per-expert naming (mlp.experts.N.gate_proj/up_proj/down_proj
+ .weight_scale) that FreeToken's MoE loader reads.
- MODEL_DIR -> /home/dank/models/hauhaucs-qwen36-35b-a3b-aggressive-fp8
- use the checkpoint's own chat_template.jinja (drop the Sharp override;
forcing a template the finetune wasn't trained on risks format drift)
- pool plan unchanged (same ~35 GB FP8 mass / qwen3_5_moe arch)
The cutover unit joined dankserver_appnet but dropped the compose-era
0.0.0.0:3923 mapping (live inspect: bridge, 3923 published), so
copyparty listened inside the netns while the host refused. New seam
asserts every host-facing port matches the pre-cutover PortBindings:
romm 8085->8080, jellyfin 8096, calibre-web 8083, copyparty 3923.
The podman 5.x v1 quadlet generator rejected the whole jellyfin unit on
the unsupported Device= key (generator log: unsupported key 'Device' in
group 'Container'), so no jellyfin.service was generated and the cutover
restart failed with 'Unit not found' after the data root had already
migrated cleanly. gluetun proves AddDevice= is the house key. New seam
test denies Device= across all six dankserver units and pins the
AddDevice=/dev/dri replacement.
rename() of a directory requires write on the directory itself; mariadb/
redis data roots are owned by the containers' mapped subuids (100998),
so the deploy user cannot move them. Use run_privileged mv (numeric
ownership preserved = rootless mapping contract) and skip the chown pass
for romm via the new keep mode — chowning mapped subuid data to the
deploy uid breaks the in-container mysql/redis users. Collisions with
rescued anonymous-volume stubs now merge one level deep instead of
stranding real data under /opt. Reverse the podman rm order so the
dependent romm container is removed before its backends.
dankserver's ~/.config/containers/systemd did not exist (no quadlet units
had ever been installed there); the dankserver media handlers cp'd into it
without mkdir -p, failing before touching any container.
- MODEL_DIR -> /home/dank/models/ornith-1.5-35b-a3b-fp8 (37 GB,
compressed-tensors per-channel FP8, same qwen3_5_moe arch/mass as
Huihui so the pool plan carries unchanged)
- CHAT_TEMPLATE_FILE -> scripts/chat-template.jinja (Qwen-Sharp, shared
with frank-default-large); the checkpoint's stock template
mis-serializes reasoning per its own model card
- env/serve/unit headers updated; loader-risk + BF16-does-not-fit note
mv of a data root onto a non-empty directory fails (Directory not empty).
The romm/copyparty cutover creates Containers/<svc>/ before migrating:
the seed env file lands there first, and the romm anonymous-volume rescue
extracts into it before the /opt move. Merge the old root's entries in
without overwriting existing state (mv -n), warn and leave residue if a
collision kept the new side, and keep the plain rename when the
destination is absent or empty.
The first deploy after a unit-file edit restarted with the stale generated
unit (generator had not seen the new Volume= paths), crash-looping hermes on
statfs ENOENT for the just-renamed data root. Reload between cp and restart.
Repoint Volume=/EnvironmentFile= in the 8 affected quadlet units to the
caddy-established Containers/<svc> layout: gitea (data, postgres, env),
authentik (data, postgres), hermes (data, webui checkout). Data-root mv on the
target happens at deploy time before each group's unit restart; same-fs rename,
instantly reversible. Running containers keep their inode-bound binds until the
unit restart flips them to the new path.
Cutover of the six surviving dankserver services from compose-era bare
containers to boot-persistent Quadlet units (WantedBy=default.target; podman
5.x v1 generator honors it at user-manager start).
Units are faithful to live inspect ground truth (image pins = running tags,
not compose inventory):
- romm trio shares one data root: /home/dank/Containers/romm mounted whole at
/romm; mariadb datadir and redis data dir live inside it. Aliases mariadb/
redis preserved on dankserver_appnet via Network=...:alias=. Secrets stay in
the host-local env file (EnvironmentFile + ${VAR} expansion in the generated
ExecStart, gitea precedent). copyparty passes its admin account via
Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD} (Exec= not Command=: the v1
generator hard-fails on Command=); entropy stays rw only there.
- jellyfin keeps every creation-time env as Environment= lines (zero drift);
Device=/dev/dri + GroupAdd=keep-groups preserve hardware transcoding.
- calibre-web pins the exact running image tag (id 986277563e76 =
0.6.27-ls397), pasta network preserved.
Deployer: first-cutover handlers retire the old container, rescue unshadowed
anonymous-volume content (podman volume export + tar --skip-old-files), mv
data roots on the same fs, then start units in dependency order with health
gates. Re-deploys restart only the selected unit. deploy-plan.py maps compose
keys mariadb/redis to the new unit names; allowlists updated. New seam test
tests/dankserver_quadlet.bats pins boot-persistence + v1-generator contract;
test-changed.sh maps the new dirs to it. Full suite green (209 tests).
-v /w::A granted anonymous full rwmda on the LAN (verified: anon PUT -> 201,
PROPFIND -> 207). New perms ::r:ra,dank: browse/download for anonymous
(required by the guarded files.franksdank.cloud chain), admin for dank,
no w/m/d for anyone. Runbook corrected with the --help-accounts semantics.
Adds a digest-pinned headless-shell Chrome DevTools Protocol daemon as a
boot-persistent quadlet unit (chrome-devtools-protocol.container). The
entrypoint override drops the image's socat front so the CDP endpoint binds
loopback-only on 127.0.0.1:9222 in the host network namespace; only the two
host-network consumers (Signal gateway, webui server) can drive it.
hermes-webui flips Network=dankclaw_default -> Network=host so its agent
process shares the daemon's namespace (0.0.0.0:8787 bind is LAN-facing as
before; no exposure change). deploy_hermes installs the new unit and starts
it if inactive without restarting in-flight browser sessions.
DNS at Porkbun was renamed (chat->ai, jellyfin->jelly) and the qbittorrent,
*arr-stack, comfyui, hermes, dankclaw, and dankserver public records were
deleted. This brings the Caddy config in lockstep: ai/jelly vhosts now match
the new names; comfyui (guarded), hermes (guarded), and the *arr/qb vhosts
are removed as their DNS is gone. The legacy dankclaw/dankserver redirect
vhosts stay: Pi-hole still answers those names on the LAN.
LAN clients now connect directly to 192.168.0.143:8787 (password-gated via
HERMES_WEBUI_PASSWORD); the SSH tunnel via port 22 remains the sole
off-network ingress. Recreation also re-reads the rotated password from
dankclaw/.env via the [Container] EnvironmentFile.
Verified live 2026-08-31: guarded path serves the Anubis JS challenge at
HTTP 200 to non-browser API clients (status-only consumers silently get
HTML); direct published port 8081 from pass_ip sources returns real JSON.
Decision: no second agent-only instance — same engines and same
rate-sensitive VPN exit IP would see double request volume for zero gain.
Server-side implementation landed on Dankclaw: dedicated 'tunnel' account
(nologin shell, password-locked, key-only auth), sshd drop-in
99-tunnel.conf scoping AllowTcpForwarding to that user with PermitTTY off,
and permitopen="localhost:8787" key-option guidance so a leaked tunnel key
can only forward to the webui port. WebUI stays loopback-only published.
Token exchange chain verified end-to-end in-process on the live provider
(pk 9): authorize -> ORM-minted code -> form-encoded token exchange with
Basic client auth -> userinfo. Key findings documented:
- Application row + grant_types must exist before authorize can complete
- token endpoint consumes form-encoded bodies only (JSON body => no
grant_type parsed -> unsupported_grant_type); client auth is Basic-only
- PropertyMappingEvaluator.evaluate(*args,**kwargs) forwards kwargs to
BaseEvaluator.evaluate(expression_source), so context kwargs become the
expression source and every evaluation raises a swallowed exception;
userinfo therefore answers sub-only claims for every provider in this
fork. Scope granting filter (provider= M2M descriptor) is always empty.
- Grafana identity = sub hash until the evaluator bug is fixed upstream.
The embedded outpost carries no provider token (token=None), so the
auth_proxy identity header was never set and no user could log in. RP
mode is topology-independent: challenge gate stays in front, login runs
inside Grafana via the OAuth2 code flow against auth.franksdank.cloud.
Provider 'grafana' (pk 9, confidential) created via ak shell; secret
written straight into the live .env (never in git). Auth URL uses the
global /application/o/authorize/ path — this fork resolves the provider
by client_id query param; per-slug authorize does not exist.
Caddyfile comment corrected: no identity header is forwarded to Grafana.
Grafana leaves the exempt-by-construction set: grafana.franksdank.cloud
now terminates at the Anubis guard (anubis:8923) and authenticated
traffic returns at the internal :8017 listener, where an @grafana Host
handler proxies through to grafana:3000 with the outpost-proxy trust
token (GRAFANA_PROXY_TOKEN = client_id_client_secret of the new
'grafana-proxy' ProxyProvider on the embedded outpost; created via
ak shell before this route flip — provider-first ordering per the
External Exposure Policy).
The Grafana admin account is intentionally kept: auth.proxy in
auto-upsert mode only creates non-admin users, so the local admin
remains the only way to administer Grafana. The old edge-basicauth
wrapper is retired (shared family password, no per-user audit trail).
- Caddyfile: grafana vhost flipped to the guarded chain; @grafana
handler added at the :8017 return listener.
- compose.yaml: GF_AUTH_PROXY_ENABLED/HEADERS/TOKEN on the grafana
service (auth_proxy trusted-proxy CIDRs left at the image default —
the proxying Caddy container is not statically addressable on
10.89.0.0/24, so Grafana trusts any in-network source and relies on
the token as the capability).
- .env.example: GRAFANA_PROXY_TOKEN placeholder so test env_consistency
keeps guarding the compose<->env contract.
Frank