A Plex section scan walked 50k files at ~850 r/s and nothing bounded it.
`IOWeight=` in a quadlet looks like the fix and does nothing: systemd delegates
`cpu memory pids` to user@.service and NOT io, and it drops a resource line it
cannot honour without a word — the container boots healthy and its cgroup has no
io.weight file at all. Same family as RequiresMountsFor= (inert in the user
manager) and ConditionPathIsMountPoint= (fail-dead).
Place the two system drop-ins that make the controller reachable (user.slice
IOAccounting=yes puts io in user.slice/cgroup.subtree_control, measured
immediately; user@1000.service DelegateControllers gains io and lands at the
next reboot, because restarting the user manager would kill every rootless
container on the host), then ask for the share: IOWeight=10 on jellyfin and plex.
10 is a convention, not a measured optimum — proportional share, only bites
under contention, only requirement is far below the default 100.
ensure_user_io_delegate places both from git and prints NOTE: io is not
delegated … inert until the next reboot, so the quadlet line never reads as live
config. Verified on the host: subtree_control flipped to `cpu io memory pids`,
the drop-in is attached and parsed (systemctl cat), and the podman 5.7 generator
emits IOWeight=10 for both units (bare key; UnitConfigItem= not needed).
Two limits recorded instead of hidden: a container's weight covers its whole
cgroup (scan and playback share it), and the comparison happens inside
user.slice, so it cannot push a rootless unit below a root unit such as
frank-archon — that pairing is user.slice's own weight, a separate call.
Docs: Quadlet Lifecycle 4b (the three-command read of the chain), Pi-hole (all
rootless containers share one 10.0.2.100, so one chatty container throttles DNS
for the host — 8 events today), E2E Suite gotcha 7 (the gate depends on the
runner's resolver: 21 reds on a lost wired link, 37/37 minutes later).
Seam: an in-repo IOWeight= may only exist when both drop-ins exist AND the
deployer places them. ./tests/run.sh: 789 ok, exit 0.
lscr.io/linuxserver/calibre:9.15.0 (desktop build) running calibre-server
as the s6 main service on internal 8090 -> host 8084, auth-gated with a
plaintext userdb (mode 600) bootstrapped once by the deploy handler.
The desktop GUI is kept on 8080/8182, basic-authed with the same
credential pair via the machine-local env file.
Evidence from the failed first cutover: podman top shows container root
maps to dank (1000) under UserNS=host rootless (subuid range 100000:65536
cannot express 997), so PUID=997 fell back to root and PMS crash-looped on
write access to its 997:984-owned data root. Fleet model instead: PUID=0/
PGID=0 (container root = dank, same as jellyfin/copyparty data roots) and
deploy_plex chowns /var/lib/plexmediaserver to 1000:1000 inside the
first-cutover block. Zero bytes moved; rollback regains one chown -R
997:984 step, documented in Plex.md. Seam test pins PUID=0/PGID=0 and the
absence of any userns key.
podman 5.7 podman-user-generator on dankserver hard-fails the whole unit on
the unknown key ('unsupported key UsernsMode in group Container'), so the
first cutover attempt masked the native unit and then found no plex.service
at all. Swap both units to UserNS=host (same --userns=host flag), update the
ownership-contract seam test to pin the exact key, and correct both docs.
Pin the seam: the v1 generator rejects unknown Container keys silently at
deploy distance — the test now pins UserNS=, not the misspelled alias.
- deploy_plex(): install unit, stop+mask native plexmediaserver.service on
first cutover (mask, never disable), gate on /identity for 180s
- deploy_tautulli(): install unit, stop+disable snap on first cutover,
seed config.ini with PMS_Web_Host=plex only if absent
- allow-lists (deploy.sh valid_service, deploy-remote.sh service_is_allowed),
planner routing (SERVICE_ORDER, DANKSERVER_SERVICES, quadlet_services),
test-changed cutover case, quadlet seam enumeration + new
userns/PUID=997/PGID=984 ownership-contract test
The cutover unit joined dankserver_appnet but dropped the compose-era
0.0.0.0:3923 mapping (live inspect: bridge, 3923 published), so
copyparty listened inside the netns while the host refused. New seam
asserts every host-facing port matches the pre-cutover PortBindings:
romm 8085->8080, jellyfin 8096, calibre-web 8083, copyparty 3923.
The podman 5.x v1 quadlet generator rejected the whole jellyfin unit on
the unsupported Device= key (generator log: unsupported key 'Device' in
group 'Container'), so no jellyfin.service was generated and the cutover
restart failed with 'Unit not found' after the data root had already
migrated cleanly. gluetun proves AddDevice= is the house key. New seam
test denies Device= across all six dankserver units and pins the
AddDevice=/dev/dri replacement.
Cutover of the six surviving dankserver services from compose-era bare
containers to boot-persistent Quadlet units (WantedBy=default.target; podman
5.x v1 generator honors it at user-manager start).
Units are faithful to live inspect ground truth (image pins = running tags,
not compose inventory):
- romm trio shares one data root: /home/dank/Containers/romm mounted whole at
/romm; mariadb datadir and redis data dir live inside it. Aliases mariadb/
redis preserved on dankserver_appnet via Network=...:alias=. Secrets stay in
the host-local env file (EnvironmentFile + ${VAR} expansion in the generated
ExecStart, gitea precedent). copyparty passes its admin account via
Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD} (Exec= not Command=: the v1
generator hard-fails on Command=); entropy stays rw only there.
- jellyfin keeps every creation-time env as Environment= lines (zero drift);
Device=/dev/dri + GroupAdd=keep-groups preserve hardware transcoding.
- calibre-web pins the exact running image tag (id 986277563e76 =
0.6.27-ls397), pasta network preserved.
Deployer: first-cutover handlers retire the old container, rescue unshadowed
anonymous-volume content (podman volume export + tar --skip-old-files), mv
data roots on the same fs, then start units in dependency order with health
gates. Re-deploys restart only the selected unit. deploy-plan.py maps compose
keys mariadb/redis to the new unit names; allowlists updated. New seam test
tests/dankserver_quadlet.bats pins boot-persistence + v1-generator contract;
test-changed.sh maps the new dirs to it. Full suite green (209 tests).
dank
Frank