podman 5.7.0 refuses ghcr.io/goauthentik/ldap:2026.8.0's nested manifest
list ('Unexpectedly received a manifest list instead of a manifest for a
single image'); pin the linux/amd64 single-platform digest instead.
Document the Mac *arr/qBittorrent stack (VM placement, gluetun credential
gate, socat LaunchAgents, PBKDF2 qB seed) and log the day: dankserver
recovery + full quadlet cutover of the six media services, the frankbook
skip-propagation/PBKDF2 handler fix, the 7-service frankbook deploy, and
the jellyfin GPU-parity finding.
The frankbook handler ran port verification even for services the
credential gate skipped, so a parked qbittorrent failed the deploy and
aborted the five arrs. Gate now lives Mac-side (test -s gluetun/env
inside the VM) before restart+verify.
The qB first-boot seed wrote a plaintext WebUI\Password, which qB >= 4.2
ignores (PBKDF2 @ByteArray only), and omitted HostHeaderValidation=false,
which would 401 the LAN forwarder path. Seed now generates the PBKDF2
hash in the VM (python3 hashlib) and sets ServerDomains=* +
HostHeaderValidation=false. Arr categories (lidarr/radarr/tv-sonarr)
pre-created with save_path=/Volumes/COMPLETE/Finished Torrents so any
grab lands in the review dir regardless of wiring details.
Seven units installed INSIDE the Mac VM (/home/core/.config/containers/
systemd) by a new deploy_frankbook_stack handler: gluetun (ProtonVPN
WireGuard, env-file credentials, publishes qbit's 8080/6881), qbittorrent
(Network=gluetun.container, verbatim /Volumes data roots, first-boot
config seed with fresh machine-local WebUI password and the Finished
Torrents category), and prowlarr/sonarr/radarr/lidarr/readarr (grab-only,
config migrated from dankserver at provisioning, readarr pinned to
amd64-1.0.980). deploy.sh routes the new target through the ssh-config
alias with REMOTE_SHELL=/usr/local/bin/bash (macOS bash 3.2 lacks
associative arrays); the planner learns FRANKBOOK_SERVICES and the
gluetun->qbittorrent netns pairing; new seam test pins the contract.
Two live-verified loader failures on Dankserver (2026-09-01), each a
clean exit at load (no swap-thrash, 37 GB fits RAM):
1. ornith-ai/Ornith-1.5-35B-A3B-FP8 (compressed-tensors PER-CHANNEL fp8):
ValueError: Missing MoE expert source layers: {'gate_up': [0, 1, ...]}
— nonstandard expert tensor naming, provider finds no expert sources.
2. lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8
(compressed-tensors BLOCK-fp8 128x128, standard per-expert naming):
RuntimeError: The size of tensor a (2048) must match the size of
tensor b (...) in _dequant_fp8_weight — the dequant path multiplies
weight * weight_scale directly and cannot expand compressed-tensors
block scales.
Empirical matrix for FreeToken v0.1.2 on qwen3_5_moe (RTX 3090):
modelopt block-fp8 (quant_method 'fp8' + weight_scale_inv) -> LOADS
compressed-tensors per-channel FP8 -> fails
compressed-tensors block-fp8 128x128 -> fails
batsclamp/Huihui-...-abliterated-FP8 is the proven uncensored checkpoint;
restoring it until a modelopt/NVFP4 requant of a wanted model exists or
FreeToken learns compressed-tensors.
Phase 1/2 of the podman standardization plan: service pages updated to the
quadlet reality (units, Containers data roots, env-file secret convention,
the v1-generator findings each caught live), Storage Policy documents the
two rootless ownership regimes and the merge-move migration, Deployment
Flow's service-manager table no longer claims dankserver media is bare
inventory-only. Bundle change log entry added.
Ornith-1.5-35B-A3B-FP8 failed to load on FreeToken v0.1.2: its
compressed-tensors checkpoint uses a nonstandard expert tensor layout
(FreeToken: 'Missing MoE expert source layers: gate_up'). Retarget to
the HauhauCS Aggressive FP8 finetune of the SAME Qwen3.6-35B-A3B arch
Huihui used, which carries compressed-tensors BLOCK-fp8 (128x128) with
the standard per-expert naming (mlp.experts.N.gate_proj/up_proj/down_proj
+ .weight_scale) that FreeToken's MoE loader reads.
- MODEL_DIR -> /home/dank/models/hauhaucs-qwen36-35b-a3b-aggressive-fp8
- use the checkpoint's own chat_template.jinja (drop the Sharp override;
forcing a template the finetune wasn't trained on risks format drift)
- pool plan unchanged (same ~35 GB FP8 mass / qwen3_5_moe arch)
The cutover unit joined dankserver_appnet but dropped the compose-era
0.0.0.0:3923 mapping (live inspect: bridge, 3923 published), so
copyparty listened inside the netns while the host refused. New seam
asserts every host-facing port matches the pre-cutover PortBindings:
romm 8085->8080, jellyfin 8096, calibre-web 8083, copyparty 3923.
The podman 5.x v1 quadlet generator rejected the whole jellyfin unit on
the unsupported Device= key (generator log: unsupported key 'Device' in
group 'Container'), so no jellyfin.service was generated and the cutover
restart failed with 'Unit not found' after the data root had already
migrated cleanly. gluetun proves AddDevice= is the house key. New seam
test denies Device= across all six dankserver units and pins the
AddDevice=/dev/dri replacement.
rename() of a directory requires write on the directory itself; mariadb/
redis data roots are owned by the containers' mapped subuids (100998),
so the deploy user cannot move them. Use run_privileged mv (numeric
ownership preserved = rootless mapping contract) and skip the chown pass
for romm via the new keep mode — chowning mapped subuid data to the
deploy uid breaks the in-container mysql/redis users. Collisions with
rescued anonymous-volume stubs now merge one level deep instead of
stranding real data under /opt. Reverse the podman rm order so the
dependent romm container is removed before its backends.
dankserver's ~/.config/containers/systemd did not exist (no quadlet units
had ever been installed there); the dankserver media handlers cp'd into it
without mkdir -p, failing before touching any container.
- MODEL_DIR -> /home/dank/models/ornith-1.5-35b-a3b-fp8 (37 GB,
compressed-tensors per-channel FP8, same qwen3_5_moe arch/mass as
Huihui so the pool plan carries unchanged)
- CHAT_TEMPLATE_FILE -> scripts/chat-template.jinja (Qwen-Sharp, shared
with frank-default-large); the checkpoint's stock template
mis-serializes reasoning per its own model card
- env/serve/unit headers updated; loader-risk + BF16-does-not-fit note
mv of a data root onto a non-empty directory fails (Directory not empty).
The romm/copyparty cutover creates Containers/<svc>/ before migrating:
the seed env file lands there first, and the romm anonymous-volume rescue
extracts into it before the /opt move. Merge the old root's entries in
without overwriting existing state (mv -n), warn and leave residue if a
collision kept the new side, and keep the plain rename when the
destination is absent or empty.
The first deploy after a unit-file edit restarted with the stale generated
unit (generator had not seen the new Volume= paths), crash-looping hermes on
statfs ENOENT for the just-renamed data root. Reload between cp and restart.
Repoint Volume=/EnvironmentFile= in the 8 affected quadlet units to the
caddy-established Containers/<svc> layout: gitea (data, postgres, env),
authentik (data, postgres), hermes (data, webui checkout). Data-root mv on the
target happens at deploy time before each group's unit restart; same-fs rename,
instantly reversible. Running containers keep their inode-bound binds until the
unit restart flips them to the new path.
Cutover of the six surviving dankserver services from compose-era bare
containers to boot-persistent Quadlet units (WantedBy=default.target; podman
5.x v1 generator honors it at user-manager start).
Units are faithful to live inspect ground truth (image pins = running tags,
not compose inventory):
- romm trio shares one data root: /home/dank/Containers/romm mounted whole at
/romm; mariadb datadir and redis data dir live inside it. Aliases mariadb/
redis preserved on dankserver_appnet via Network=...:alias=. Secrets stay in
the host-local env file (EnvironmentFile + ${VAR} expansion in the generated
ExecStart, gitea precedent). copyparty passes its admin account via
Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD} (Exec= not Command=: the v1
generator hard-fails on Command=); entropy stays rw only there.
- jellyfin keeps every creation-time env as Environment= lines (zero drift);
Device=/dev/dri + GroupAdd=keep-groups preserve hardware transcoding.
- calibre-web pins the exact running image tag (id 986277563e76 =
0.6.27-ls397), pasta network preserved.
Deployer: first-cutover handlers retire the old container, rescue unshadowed
anonymous-volume content (podman volume export + tar --skip-old-files), mv
data roots on the same fs, then start units in dependency order with health
gates. Re-deploys restart only the selected unit. deploy-plan.py maps compose
keys mariadb/redis to the new unit names; allowlists updated. New seam test
tests/dankserver_quadlet.bats pins boot-persistence + v1-generator contract;
test-changed.sh maps the new dirs to it. Full suite green (209 tests).
-v /w::A granted anonymous full rwmda on the LAN (verified: anon PUT -> 201,
PROPFIND -> 207). New perms ::r:ra,dank: browse/download for anonymous
(required by the guarded files.franksdank.cloud chain), admin for dank,
no w/m/d for anyone. Runbook corrected with the --help-accounts semantics.
Adds a digest-pinned headless-shell Chrome DevTools Protocol daemon as a
boot-persistent quadlet unit (chrome-devtools-protocol.container). The
entrypoint override drops the image's socat front so the CDP endpoint binds
loopback-only on 127.0.0.1:9222 in the host network namespace; only the two
host-network consumers (Signal gateway, webui server) can drive it.
hermes-webui flips Network=dankclaw_default -> Network=host so its agent
process shares the daemon's namespace (0.0.0.0:8787 bind is LAN-facing as
before; no exposure change). deploy_hermes installs the new unit and starts
it if inactive without restarting in-flight browser sessions.
DNS at Porkbun was renamed (chat->ai, jellyfin->jelly) and the qbittorrent,
*arr-stack, comfyui, hermes, dankclaw, and dankserver public records were
deleted. This brings the Caddy config in lockstep: ai/jelly vhosts now match
the new names; comfyui (guarded), hermes (guarded), and the *arr/qb vhosts
are removed as their DNS is gone. The legacy dankclaw/dankserver redirect
vhosts stay: Pi-hole still answers those names on the LAN.
LAN clients now connect directly to 192.168.0.143:8787 (password-gated via
HERMES_WEBUI_PASSWORD); the SSH tunnel via port 22 remains the sole
off-network ingress. Recreation also re-reads the rotated password from
dankclaw/.env via the [Container] EnvironmentFile.
Verified live 2026-08-31: guarded path serves the Anubis JS challenge at
HTTP 200 to non-browser API clients (status-only consumers silently get
HTML); direct published port 8081 from pass_ip sources returns real JSON.
Decision: no second agent-only instance — same engines and same
rate-sensitive VPN exit IP would see double request volume for zero gain.
Server-side implementation landed on Dankclaw: dedicated 'tunnel' account
(nologin shell, password-locked, key-only auth), sshd drop-in
99-tunnel.conf scoping AllowTcpForwarding to that user with PermitTTY off,
and permitopen="localhost:8787" key-option guidance so a leaked tunnel key
can only forward to the webui port. WebUI stays loopback-only published.
Token exchange chain verified end-to-end in-process on the live provider
(pk 9): authorize -> ORM-minted code -> form-encoded token exchange with
Basic client auth -> userinfo. Key findings documented:
- Application row + grant_types must exist before authorize can complete
- token endpoint consumes form-encoded bodies only (JSON body => no
grant_type parsed -> unsupported_grant_type); client auth is Basic-only
- PropertyMappingEvaluator.evaluate(*args,**kwargs) forwards kwargs to
BaseEvaluator.evaluate(expression_source), so context kwargs become the
expression source and every evaluation raises a swallowed exception;
userinfo therefore answers sub-only claims for every provider in this
fork. Scope granting filter (provider= M2M descriptor) is always empty.
- Grafana identity = sub hash until the evaluator bug is fixed upstream.
The embedded outpost carries no provider token (token=None), so the
auth_proxy identity header was never set and no user could log in. RP
mode is topology-independent: challenge gate stays in front, login runs
inside Grafana via the OAuth2 code flow against auth.franksdank.cloud.
Provider 'grafana' (pk 9, confidential) created via ak shell; secret
written straight into the live .env (never in git). Auth URL uses the
global /application/o/authorize/ path — this fork resolves the provider
by client_id query param; per-slug authorize does not exist.
Caddyfile comment corrected: no identity header is forwarded to Grafana.
Grafana leaves the exempt-by-construction set: grafana.franksdank.cloud
now terminates at the Anubis guard (anubis:8923) and authenticated
traffic returns at the internal :8017 listener, where an @grafana Host
handler proxies through to grafana:3000 with the outpost-proxy trust
token (GRAFANA_PROXY_TOKEN = client_id_client_secret of the new
'grafana-proxy' ProxyProvider on the embedded outpost; created via
ak shell before this route flip — provider-first ordering per the
External Exposure Policy).
The Grafana admin account is intentionally kept: auth.proxy in
auto-upsert mode only creates non-admin users, so the local admin
remains the only way to administer Grafana. The old edge-basicauth
wrapper is retired (shared family password, no per-user audit trail).
- Caddyfile: grafana vhost flipped to the guarded chain; @grafana
handler added at the :8017 return listener.
- compose.yaml: GF_AUTH_PROXY_ENABLED/HEADERS/TOKEN on the grafana
service (auth_proxy trusted-proxy CIDRs left at the image default —
the proxying Caddy container is not statically addressable on
10.89.0.0/24, so Grafana trusts any in-network source and relies on
the token as the capability).
- .env.example: GRAFANA_PROXY_TOKEN placeholder so test env_consistency
keeps guarding the compose<->env contract.
Every Caddy-proxied service is now either behind the auth chain or
explicitly internal-only, with the provider-first flip procedure codified.
The comfyui GPU-access seam test now asserts the deployed contract
(group_add keep-groups + udev MODE 0666 on kfd/renderD*) instead of the
pre-fix 'video' group assumption.
Loki's public vhost had zero auth — any host that can resolve DNS could
query all logs via LogQL. The vhost now terminates at anubis:8923 and
authenticated traffic returns at the internal :8017 listener, where a new
@loki host handler routes to loki:3100. The outpost proxy provider
'loki-proxy' (external_host https://loki.franksdank.cloud/, internal_host
http://caddy:8017) was created before this deploy — provider-first
ordering so authenticated traffic never hits the outpost error page.
Ingestion is unaffected: Alloy pushes to loki:3100 on the compose network.
The flip hunk's first comment line was written with a literal ':' prefix
(edit-tool auto-prefix artifact). Caddyfile parses ':# ...' as a matcher token,
which made the following site block an unrecognized directive and failed
config validation — that is why the earlier deploy validated, failed, and
skipped the reload, leaving the live config pre-flip (comfyui still direct).
The first revision guarded the kfd line with SUBSYSTEM=="misc", which never
matches (misc-class devices carry no subsystem for udev matching), so the rule
was read but never fired and /dev/kfd stayed 0660 root:render. Name-only match
applies 0666 on creation and on trigger.
Also fixes test-changed.sh to exit 0 when changed paths map to zero tests
(udev rules are mapped-but-untestable) instead of invoking bats with an empty
selection.
The comfyui container is spawned by a systemd user unit whose spawn context
carries no supplementary groups, so render-group access to /dev/kfd and
renderD128 was structurally unreachable (keep-groups preserved an empty set;
named group_add 'render' fails name lookup inside the userns). World-mode on
the compute nodes is the only spawn-context-independent grant; display cards
keep their modes. Apply: install to /etc/udev/rules.d, udevadm control
--reload-rules, then trigger by subsystem to re-apply modes to live nodes.
Also maps dankclaw/udev/* in the focused-test selector (not bats-testable).
Rootless podman cannot resolve the literal group name 'render' inside the
container user namespace and fails the start with 'unable to find group
render'. keep-groups preserves the launching user's real supplementary
groups (dank is in render on this host), which is the canonical rootless
ROCm device-access recipe.
/dev/kfd and /dev/dri/renderD128 are root:render 0660 on this host. The
container passed the device nodes but only joined group video(44), so torch's
HIP init failed with 'No HIP GPUs are available' and main.py never bound
:8188 — every proxy hop to the app returned 502. Group membership is a
create-time property, so this deploy recreates the container.
comfyui.franksdank.cloud was exempt (direct reverse_proxy to comfyui:8188).
The comfyui-proxy proxy provider was created on the embedded outpost first
(internal_host http://caddy:8017, flows copied from hermes-proxy), then the
vhost flips onto the guard chain and authenticated traffic routes back via a
@comfy host match. ComfyUI has no auth of its own; this closes the last
unguarded admin surface on the internal hosts.
The quadlet generator space-splits HealthCmd into the healthcheck argv, so the
quoted /bin/sh -c form arrived mangled and failed with exit 1 forever (FailingStreak
569) while the DB itself accepted connections. Bare argv survives both exec forms.
Env file is the source of truth (unit ExecStart runs the serve script which
sources it); serve-script default and the deployer health-gate URL flipped in
lockstep so the post-restart health gate polls the new port. Docs, fleet table,
and both index bullets updated to match.
Frank