feat(fleet): give the media servers an I/O share that actually exists

A Plex section scan walked 50k files at ~850 r/s and nothing bounded it.
`IOWeight=` in a quadlet looks like the fix and does nothing: systemd delegates
`cpu memory pids` to user@.service and NOT io, and it drops a resource line it
cannot honour without a word — the container boots healthy and its cgroup has no
io.weight file at all. Same family as RequiresMountsFor= (inert in the user
manager) and ConditionPathIsMountPoint= (fail-dead).

Place the two system drop-ins that make the controller reachable (user.slice
IOAccounting=yes puts io in user.slice/cgroup.subtree_control, measured
immediately; user@1000.service DelegateControllers gains io and lands at the
next reboot, because restarting the user manager would kill every rootless
container on the host), then ask for the share: IOWeight=10 on jellyfin and plex.
10 is a convention, not a measured optimum — proportional share, only bites
under contention, only requirement is far below the default 100.

ensure_user_io_delegate places both from git and prints NOTE: io is not
delegated … inert until the next reboot, so the quadlet line never reads as live
config. Verified on the host: subtree_control flipped to `cpu io memory pids`,
the drop-in is attached and parsed (systemctl cat), and the podman 5.7 generator
emits IOWeight=10 for both units (bare key; UnitConfigItem= not needed).

Two limits recorded instead of hidden: a container's weight covers its whole
cgroup (scan and playback share it), and the comparison happens inside
user.slice, so it cannot push a rootless unit below a root unit such as
frank-archon — that pairing is user.slice's own weight, a separate call.

Docs: Quadlet Lifecycle 4b (the three-command read of the chain), Pi-hole (all
rootless containers share one 10.0.2.100, so one chatty container throttles DNS
for the host — 8 events today), E2E Suite gotcha 7 (the gate depends on the
runner's resolver: 21 reds on a lost wired link, 37/37 minutes later).

Seam: an in-repo IOWeight= may only exist when both drop-ins exist AND the
deployer places them. ./tests/run.sh: 789 ok, exit 0.
This commit is contained in:
dank committed 2026-10-31 18:42:27 -07:00
1 parent 6d274acc47
commit 183a6d5442
9 files changed
+146 -4

No files matched your search

@@ -167,6 +167,43 @@ Gate: `tests/quadlet_image_pins.bats`. Where upstream publishes no version
tags (copyparty-ac stops at 1.19.15), pin `:latest@sha256:` and name the
version the config label carries in the comment beside it.
## 4b. Resource control needs host permission (measured 2026-10-31, Dankserver)
`IOWeight=` in a rootless quadlet does nothing on its own, and nothing says so.
The system manager delegates `cpu memory pids` to `user@.service` and **not** `io`;
systemd drops a resource line it cannot honour **without a word** — the generator
still emits `IOWeight=10`, the container still starts and answers its health check,
and the cgroup simply has no `io.weight` file. Same family as
`RequiresMountsFor=` (inert in the user manager) and `ConditionPathIsMountPoint=`
(fail-dead): the line is accepted, the unit is happy, the setting is dead.
Two system drop-ins make it real, both placed by `ensure_user_io_delegate` in
`scripts/deploy-remote.sh` (repo copies under `dankserver/host/systemd/`):
| File | Effect |
|---|---|
| `user.slice.d/io-accounting.conf` → `[Slice] IOAccounting=yes` | puts `io` into `user.slice/cgroup.subtree_control`; lands immediately, no restart |
| `user@1000.service.d/io-delegate.conf` → `[Service] DelegateControllers=cpu memory pids io` | the user manager can only request controllers it was delegated; read when it starts, so it lands on **the next reboot** (restarting it kills every rootless container on the host) |
Read the chain in this order — each line is the failure mode of the one after it:
```bash
cat /sys/fs/cgroup/user.slice/cgroup.subtree_control # wants: io
systemctl show user@1000.service -p DelegateControllers # wants: io
cat /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/\<unit\>.service/libpod-payload-*/io.weight
```
`io` missing from the first → the accounting drop-in is absent. Missing from the
second → the host has not rebooted since the delegation landed (the deployer prints
`NOTE: io is not delegated …` for exactly this). No file at the third → that unit
carries no weight.
Two limits that survive the fix: a container's weight covers **everything** in its
cgroup, so a library scan and a playback stream share one weight; and the comparison
is made *inside* `user.slice`, so a rootless unit's weight cannot push it below a
root unit (`frank-archon`). Shaping that pairing is the weight of `user.slice`
itself — a fleet decision, not a unit line.
## 5. Rootless Quadlet log retrieval
Step 0 — establish ownership: `systemctl --user status <name>.service`