A Plex section scan walked 50k files at ~850 r/s and nothing bounded it.
`IOWeight=` in a quadlet looks like the fix and does nothing: systemd delegates
`cpu memory pids` to user@.service and NOT io, and it drops a resource line it
cannot honour without a word — the container boots healthy and its cgroup has no
io.weight file at all. Same family as RequiresMountsFor= (inert in the user
manager) and ConditionPathIsMountPoint= (fail-dead).
Place the two system drop-ins that make the controller reachable (user.slice
IOAccounting=yes puts io in user.slice/cgroup.subtree_control, measured
immediately; user@1000.service DelegateControllers gains io and lands at the
next reboot, because restarting the user manager would kill every rootless
container on the host), then ask for the share: IOWeight=10 on jellyfin and plex.
10 is a convention, not a measured optimum — proportional share, only bites
under contention, only requirement is far below the default 100.
ensure_user_io_delegate places both from git and prints NOTE: io is not
delegated … inert until the next reboot, so the quadlet line never reads as live
config. Verified on the host: subtree_control flipped to `cpu io memory pids`,
the drop-in is attached and parsed (systemctl cat), and the podman 5.7 generator
emits IOWeight=10 for both units (bare key; UnitConfigItem= not needed).
Two limits recorded instead of hidden: a container's weight covers its whole
cgroup (scan and playback share it), and the comparison happens inside
user.slice, so it cannot push a rootless unit below a root unit such as
frank-archon — that pairing is user.slice's own weight, a separate call.
Docs: Quadlet Lifecycle 4b (the three-command read of the chain), Pi-hole (all
rootless containers share one 10.0.2.100, so one chatty container throttles DNS
for the host — 8 events today), E2E Suite gotcha 7 (the gate depends on the
runner's resolver: 21 reds on a lost wired link, 37/37 minutes later).
Seam: an in-repo IOWeight= may only exist when both drop-ins exist AND the
deployer places them. ./tests/run.sh: 789 ok, exit 0.
Dankserver rebooted with the array's mount and the media containers racing. A
rootless quadlet cannot order itself against the system's mnt-entropy.mount, so
`RequiresMountsFor=/mnt/entropy` is inert in the user manager: podman resolved
the bind against the pre-mount directory, Jellyfin and Plex came up blind, and
Jellyfin read an empty library and began `Removing item, Type: "Season"` by the
thousand. Copyparty had no guard at all.
`scripts/wait-for-mount.sh` is the gate: it waits for every named mountpoint,
then refuses by name. `ConditionPathIsMountPoint=` is NOT the fix — measured on
the host, an unmet condition logs `skipped, unmet condition check` with
NRestarts=0 and no start attempt ever follows, which is fail-dead, not
fail-closed. Jellyfin and Plex swap the condition for the gate; Copyparty gets
one; the law is recorded in AGENTS.md and the view page.
Verified on the host: the 5.7.0 quadlet generator accepts ExecStartPre in
[Service] (exit 0, all three units generated, gate present), the gate waits for
a mount that appears mid-window and proceeds, and it refuses a plain directory
after its window naming the path.
tests/media_overlay.bats pinned the old mechanism by name; its intent — both
media servers refuse to start without the view — now rides the gate.
The merged door was /mnt/media, which read like a third volume and split the host
state across two trees. It is now /var/lib/media-meta/view, so the array layer,
the SSD upper layer, the kernel workdir, the reap trash and the merged tree are
one directory — and the mount table gains no new /mnt entry beside the array.
systemd names a mount unit after its ESCAPED path, so the repo copy keeps the
readable name media-meta-view.mount while `install` writes the host file as
var-lib-media\x2dmeta-view.mount; `plan` prints both and the page records them, so
nobody hunts for a unit that "does not exist". install also creates the mount
point itself, because systemd will not.
The seam now additionally pins that the unit's Where= is exactly the path both
containers bind — guard and bind cannot drift apart. The Quadlet Lifecycle
phantom-path example is corrected too: it named an invented /mnt/media/*, while
the real phantom is the legacy /opt/jellyfin/* that dankserver/compose.yaml still
carries for a service whose roots moved to /home/dank/Containers/jellyfin.
Both media servers read a Kodi-format sidecar (movie.nfo / tvshow.nfo /
episode.nfo) only when it sits next to the media file, and the library lives on
the RAID6 array, deliberately mounted read-only for every container and slow to
seek for a thousand tiny XML reads. So neither server binds the array any more:
both bind /mnt/media, an OverlayFS merge of the array (lowerdir, never written)
with /var/lib/media-meta on the NVMe (upperdir, where every write lands).
Jellyfin binds the view rw at /data and is the fleet's only metadata writer;
Plex binds it ro at /mnt/entropy, the container path its own database already
recorded, so only the host side of that bind moved and its library paths keep
resolving.
scripts/media-view.sh plans, installs, verifies, reports and reaps the view. It
refuses to mount over an unmounted array, refuses an upper layer on rotational
storage, and refuses a non-root install. deploy-remote.sh raises the view before
it restarts either media server, both units carry
ConditionPathIsMountPoint=/mnt/media so a missing view skips the start instead of
letting a scan clear the library database, and the planner routes the installer
and dankserver/host/** to the jellyfin+plex pair.
tests/media_overlay.bats (11 seams) pins the resolved shape, both refusals, the
two binds and their guard, the deployer's ordering, the planner mapping, and the
reap behaviour against a fixture (ghosts listed, then moved to the SSD trash --
files that exist only above are sidecars, not ghosts, so only directories are
candidates).
Container images re-resolved against their registries today; each pin is the
manifest-list digest the tag answers now, so no container is served from a
build that upstream has already replaced.
Version-tag moves: authentik 2026.8.3 -> 2026.8.5 (server/worker/outpost and
the ldap image, which now carries the tag instead of a bare digest so the pair
can never drift apart), seerr v3.4.1 -> v3.5.1, calibre 9.15.0 -> 9.17.0,
calibre-web 0.6.27-ls402 -> ls407, plex ls329 -> ls330, tautulli
v2.18.1-ls245 -> v2.18.2-ls250.
Digest-only moves inside the same stream: postgres 16-alpine, mariadb 11,
redis 7-alpine, valkey 9-alpine, grafana, alertmanager, gluetun, searxng,
open-webui, jellyfin, romm, chromedp/headless-shell, lightpanda/browser.
Rollback is the previous digest on the same Image= line.
Closing the command-center security queue #70/#71/#72/#73/#74: rolling tags
let a rebuild move under a running service with nobody approving it, and
make a CVE audit ungradeable. 36 quadlet units across dankclaw, dankserver
and frankbook-pro now carry tag@sha256; localhost/* builds stay exempt
(their build is the pin). The pihole and copyparty pins pull the fixed
builds; valkey 9.1.2 (already running) and redis 7.4.11 are pinned in
place, vendor fixes TBD.
New gate tests/quadlet_image_pins.bats: no unpinned registry image, no
generator-rejected key anywhere. Three seams that pinned the old exact-tag
law (frankbook readarr, open_webui, seerr) now require tag AND digest -
strictly stronger. Quadlet Lifecycle docs: batch-abort law, entrypoint
flags-only law, pin law.
ls329 is the newest build on the linuxserver channel and Plex 1.43.5.11029
is their newest release; the live pin (ls325 / 1.43.4.10903) is two Plex
releases behind. Digest-verified at pin time: the tag's manifest digest
equals the registry's current 'latest' (sha256:8f655bad...e9e2334), the
same law the page records. Quadlet restart runs PMS's usual one-minor
schema migration on first boot; the deployer's /identity HTTPS gate
tolerates the first-boot migration up to 600 s.
Rollback: revert this commit (one line).
The C5 probe failed for the wrong reason: plex.container lost its Network=
key in the same edit churn that ate tautulli's ContainerName. Plex ran on
the default podman network — published 32400 still worked (clients fine)
but tautulli could not resolve the plex alias, breaking the seeded
PMS_Web_Host=plex. Restore appnet membership; structure now parallel to
tautulli.container. Lesson recorded: after multi-hunk line edits on these
unit files, diff the whole file — key-order churn has eaten three keys
already.
Evidence from the failed first cutover: podman top shows container root
maps to dank (1000) under UserNS=host rootless (subuid range 100000:65536
cannot express 997), so PUID=997 fell back to root and PMS crash-looped on
write access to its 997:984-owned data root. Fleet model instead: PUID=0/
PGID=0 (container root = dank, same as jellyfin/copyparty data roots) and
deploy_plex chowns /var/lib/plexmediaserver to 1000:1000 inside the
first-cutover block. Zero bytes moved; rollback regains one chown -R
997:984 step, documented in Plex.md. Seam test pins PUID=0/PGID=0 and the
absence of any userns key.
podman 5.7 podman-user-generator on dankserver hard-fails the whole unit on
the unknown key ('unsupported key UsernsMode in group Container'), so the
first cutover attempt masked the native unit and then found no plex.service
at all. Swap both units to UserNS=host (same --userns=host flag), update the
ownership-contract seam test to pin the exact key, and correct both docs.
Pin the seam: the v1 generator rejects unknown Container keys silently at
deploy distance — the test now pins UserNS=, not the misspelled alias.
- plex: lscr.io/linuxserver/plex:1.43.3.10896-cb3ebc72d-ls322 (verified ==
ghcr latest digest; plan's ls9 pin superseded by the re-verify clause),
/var/lib/plexmediaserver bind-mounted in place as /config, /mnt/entropy
mounted ro at the same path, PUID=997/PGID=984 + UsernsMode=host so the
container's file identity is byte-compatible with the native unit
- tautulli: lscr.io/linuxserver/tautulli:v2.18.1-ls242 (verified == ghcr
latest digest), fresh /home/dank/Containers/tautulli config root,
PUID=1000/PGID=1000, python3 /home healthcheck
- both on dankserver_appnet with aliases; 32400/8181 published as before
dank
Frank