feat(fleet): give the media servers an I/O share that actually exists #110

Merged
dank merged 1 commits from feat/media-io-weight into main 2026-10-31 18:43:49 -07:00
Owner

Why

A Plex section scan walked 50k files at ~850 r/s while Jellyfin was rescanning,
and nothing bounded it. Adding IOWeight= to a quadlet looks like the fix and
does nothing: systemd hands user@.service cpu memory pids and drops a
resource line it cannot honour without a word, so the container boots healthy with
no io.weight file in its cgroup at all. Same family as RequiresMountsFor=
(inert in the user manager) and ConditionPathIsMountPoint= (fail-dead) — and
this one I only caught by reading the generated unit and the cgroup.

Change

Host permission first (both files under dankserver/host/systemd/, placed by the
new ensure_user_io_delegate in scripts/deploy-remote.sh, run by the jellyfin
and plex handlers):

File Effect Lands
user.slice.d/io-accounting.conf io enters user.slice/cgroup.subtree_control immediately (measured: cpu memory pids → cpu io memory pids with nothing restarted)
user@1000.service.d/io-delegate.conf the user manager is delegated the io controller next reboot — restarting user@1000.service would kill every rootless container here

Then the share: IOWeight=10 in jellyfin.container and plex.container
(bare key — verified the podman 5.7 generator emits it verbatim). 10 is a
convention, not a measured optimum: it is a proportional share that only bites
under contention, and the only requirement is that it sit far below the default
100. A measured optimum would need a bench window with the media stack stopped,
which fleet law forbids unless ordered. The deployer prints
NOTE: io is not delegated … inert until the next reboot so nobody reads the
line as live.

Two honest limits, written into the page: a container's weight covers everything
in its cgroup (a scan and a playback stream share one weight), and the comparison
happens inside user.slice — so this cannot push a rootless unit below a root
unit like frank-archon. That pairing is user.slice's own weight, a separate
decision.

Verification (on Dankserver, live)

  • user.slice/cgroup.subtree_control: cpu memory pids → cpu io memory pids.
  • systemctl cat user@1000.service shows the drop-in attached and parsed
    (DelegateControllers=cpu memory pids io); the running instance still reports
    the old set, which is exactly what the NOTE says — the check is the reboot's.
  • The quadlet generator over the real repo files emits IOWeight=10 for both
    media units (exit 0).
  • Both drop-ins installed on the host, so the next reboot activates the share with
    no manual step. Probe artifacts retired to ~/.AGENTTRASH/.

Docs

  • Quadlet Lifecycle §4b — the trap, the two drop-ins, and the three-command
    chain that reads the whole story (subtree_control → DelegateControllers →
    the payload cgroup's io.weight).
  • Pi-hole — one shared 10.0.2.100 means one chatty container throttles DNS
    for every rootless container on the host (8 throttle events today, all that one
    address).
  • E2E Suite gotcha 7 — the gate runs from the runner's resolver: a lost wired
    link produced 21 reds across SSO/RBAC/crawl on a deploy that touched one quadlet;
    the same suite passed 37/37 minutes later.

New seam in tests/dankserver_quadlet.bats: an in-repo IOWeight= is only
allowed to exist when both drop-ins exist and the deployer places them — so
inert config cannot come back quietly.


cwd: /home/blank/.omp/wt/io-weight · session: 2026-10-30T23-38-45-297Z_01a18f2e-18b1-7573-bcd5-ac99636f8bcb

## Why A Plex section scan walked 50k files at ~850 r/s while Jellyfin was rescanning, and nothing bounded it. Adding `IOWeight=` to a quadlet looks like the fix and **does nothing**: systemd hands `user@.service` `cpu memory pids` and drops a resource line it cannot honour without a word, so the container boots healthy with no `io.weight` file in its cgroup at all. Same family as `RequiresMountsFor=` (inert in the user manager) and `ConditionPathIsMountPoint=` (fail-dead) — and this one I only caught by reading the generated unit and the cgroup. ## Change Host permission first (both files under `dankserver/host/systemd/`, placed by the new `ensure_user_io_delegate` in `scripts/deploy-remote.sh`, run by the jellyfin and plex handlers): | File | Effect | Lands | |---|---|---| | `user.slice.d/io-accounting.conf` | `io` enters `user.slice/cgroup.subtree_control` | immediately (measured: `cpu memory pids` → `cpu io memory pids` with nothing restarted) | | `user@1000.service.d/io-delegate.conf` | the user manager is delegated the io controller | **next reboot** — restarting `user@1000.service` would kill every rootless container here | Then the share: `IOWeight=10` in `jellyfin.container` and `plex.container` (bare key — verified the podman 5.7 generator emits it verbatim). `10` is a **convention, not a measured optimum**: it is a proportional share that only bites under contention, and the only requirement is that it sit far below the default 100. A measured optimum would need a bench window with the media stack stopped, which fleet law forbids unless ordered. The deployer prints `NOTE: io is not delegated … inert until the next reboot` so nobody reads the line as live. Two honest limits, written into the page: a container's weight covers everything in its cgroup (a scan and a playback stream share one weight), and the comparison happens inside `user.slice` — so this cannot push a rootless unit below a root unit like `frank-archon`. That pairing is `user.slice`'s own weight, a separate decision. ## Verification (on Dankserver, live) - `user.slice/cgroup.subtree_control`: `cpu memory pids` → **`cpu io memory pids`**. - `systemctl cat user@1000.service` shows the drop-in attached and parsed (`DelegateControllers=cpu memory pids io`); the running instance still reports the old set, which is exactly what the NOTE says — the check is the reboot's. - The quadlet generator over the real repo files emits `IOWeight=10` for both media units (exit 0). - Both drop-ins installed on the host, so the next reboot activates the share with no manual step. Probe artifacts retired to `~/.AGENTTRASH/`. ## Docs - [[Quadlet Lifecycle]] §4b — the trap, the two drop-ins, and the three-command chain that reads the whole story (`subtree_control` → `DelegateControllers` → the payload cgroup's `io.weight`). - [[Pi-hole]] — one shared `10.0.2.100` means one chatty container throttles DNS for every rootless container on the host (8 throttle events today, all that one address). - [[E2E Suite]] gotcha 7 — the gate runs from the runner's resolver: a lost wired link produced 21 reds across SSO/RBAC/crawl on a deploy that touched one quadlet; the same suite passed 37/37 minutes later. New seam in `tests/dankserver_quadlet.bats`: an in-repo `IOWeight=` is only allowed to exist when both drop-ins exist **and** the deployer places them — so inert config cannot come back quietly. --- cwd: /home/blank/.omp/wt/io-weight · session: 2026-10-30T23-38-45-297Z_01a18f2e-18b1-7573-bcd5-ac99636f8bcb
dank added 1 commit 2026-10-31 18:42:41 -07:00
A Plex section scan walked 50k files at ~850 r/s and nothing bounded it.
`IOWeight=` in a quadlet looks like the fix and does nothing: systemd delegates
`cpu memory pids` to user@.service and NOT io, and it drops a resource line it
cannot honour without a word — the container boots healthy and its cgroup has no
io.weight file at all. Same family as RequiresMountsFor= (inert in the user
manager) and ConditionPathIsMountPoint= (fail-dead).

Place the two system drop-ins that make the controller reachable (user.slice
IOAccounting=yes puts io in user.slice/cgroup.subtree_control, measured
immediately; user@1000.service DelegateControllers gains io and lands at the
next reboot, because restarting the user manager would kill every rootless
container on the host), then ask for the share: IOWeight=10 on jellyfin and plex.
10 is a convention, not a measured optimum — proportional share, only bites
under contention, only requirement is far below the default 100.

ensure_user_io_delegate places both from git and prints NOTE: io is not
delegated … inert until the next reboot, so the quadlet line never reads as live
config. Verified on the host: subtree_control flipped to `cpu io memory pids`,
the drop-in is attached and parsed (systemctl cat), and the podman 5.7 generator
emits IOWeight=10 for both units (bare key; UnitConfigItem= not needed).

Two limits recorded instead of hidden: a container's weight covers its whole
cgroup (scan and playback share it), and the comparison happens inside
user.slice, so it cannot push a rootless unit below a root unit such as
frank-archon — that pairing is user.slice's own weight, a separate call.

Docs: Quadlet Lifecycle 4b (the three-command read of the chain), Pi-hole (all
rootless containers share one 10.0.2.100, so one chatty container throttles DNS
for the host — 8 events today), E2E Suite gotcha 7 (the gate depends on the
runner's resolver: 21 reds on a lost wired link, 37/37 minutes later).

Seam: an in-repo IOWeight= may only exist when both drop-ins exist AND the
deployer places them. ./tests/run.sh: 789 ok, exit 0.
dank merged commit 6a4291f224 into main 2026-10-31 18:43:49 -07:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dank/serverAppConfigs#110