feat(fleet): give the media servers an I/O share that actually exists
A Plex section scan walked 50k files at ~850 r/s and nothing bounded it. `IOWeight=` in a quadlet looks like the fix and does nothing: systemd delegates `cpu memory pids` to user@.service and NOT io, and it drops a resource line it cannot honour without a word — the container boots healthy and its cgroup has no io.weight file at all. Same family as RequiresMountsFor= (inert in the user manager) and ConditionPathIsMountPoint= (fail-dead). Place the two system drop-ins that make the controller reachable (user.slice IOAccounting=yes puts io in user.slice/cgroup.subtree_control, measured immediately; user@1000.service DelegateControllers gains io and lands at the next reboot, because restarting the user manager would kill every rootless container on the host), then ask for the share: IOWeight=10 on jellyfin and plex. 10 is a convention, not a measured optimum — proportional share, only bites under contention, only requirement is far below the default 100. ensure_user_io_delegate places both from git and prints NOTE: io is not delegated … inert until the next reboot, so the quadlet line never reads as live config. Verified on the host: subtree_control flipped to `cpu io memory pids`, the drop-in is attached and parsed (systemctl cat), and the podman 5.7 generator emits IOWeight=10 for both units (bare key; UnitConfigItem= not needed). Two limits recorded instead of hidden: a container's weight covers its whole cgroup (scan and playback share it), and the comparison happens inside user.slice, so it cannot push a rootless unit below a root unit such as frank-archon — that pairing is user.slice's own weight, a separate call. Docs: Quadlet Lifecycle 4b (the three-command read of the chain), Pi-hole (all rootless containers share one 10.0.2.100, so one chatty container throttles DNS for the host — 8 events today), E2E Suite gotcha 7 (the gate depends on the runner's resolver: 21 reds on a lost wired link, 37/37 minutes later). Seam: an in-repo IOWeight= may only exist when both drop-ins exist AND the deployer places them. ./tests/run.sh: 789 ok, exit 0.
This commit is contained in:
1 parent
6d274acc47
commit
183a6d5442
9 files changed
+146
-4
No files matched your search
@@ -100,3 +100,12 @@ aborts with `DEPLOY INCOMPLETE: e2e gate FAILED` when it does not pass
|
||||
hand-off list (`HANDOFFS`) instead of a single host. The guard contract is
|
||||
untouched: challenge cleared, no authentik `uidField`, status < 500, and every
|
||||
host NOT in the list still has to land on its own vhost.
|
||||
7. **The runner's resolver is part of the suite.** A 2026-10-31 deploy of one
|
||||
Dankserver quadlet failed the gate with 21 reds spread over SSO, homepage RBAC
|
||||
and every vhost crawl — precisely the specs that navigate to an absolute
|
||||
`*.franksdank.cloud` name, while every spec on the relative `baseURL` stayed
|
||||
green. The error was `net::ERR_NAME_NOT_RESOLVED`: the workstation had lost its
|
||||
wired link, and on the fallback its resolver was no longer pinned to Pi-hole.
|
||||
Nothing on the fleet was wrong — the same suite re-run minutes later passed
|
||||
37/37. Before a mass red gets blamed on the guard chain, run
|
||||
`resolvectl query grafana.franksdank.cloud` on the box that runs the gate.
|
||||
@@ -167,6 +167,43 @@ Gate: `tests/quadlet_image_pins.bats`. Where upstream publishes no version
|
||||
tags (copyparty-ac stops at 1.19.15), pin `:latest@sha256:` and name the
|
||||
version the config label carries in the comment beside it.
|
||||
|
||||
## 4b. Resource control needs host permission (measured 2026-10-31, Dankserver)
|
||||
|
||||
`IOWeight=` in a rootless quadlet does nothing on its own, and nothing says so.
|
||||
The system manager delegates `cpu memory pids` to `user@.service` and **not** `io`;
|
||||
systemd drops a resource line it cannot honour **without a word** — the generator
|
||||
still emits `IOWeight=10`, the container still starts and answers its health check,
|
||||
and the cgroup simply has no `io.weight` file. Same family as
|
||||
`RequiresMountsFor=` (inert in the user manager) and `ConditionPathIsMountPoint=`
|
||||
(fail-dead): the line is accepted, the unit is happy, the setting is dead.
|
||||
|
||||
Two system drop-ins make it real, both placed by `ensure_user_io_delegate` in
|
||||
`scripts/deploy-remote.sh` (repo copies under `dankserver/host/systemd/`):
|
||||
|
||||
| File | Effect |
|
||||
|---|---|
|
||||
| `user.slice.d/io-accounting.conf` → `[Slice] IOAccounting=yes` | puts `io` into `user.slice/cgroup.subtree_control`; lands immediately, no restart |
|
||||
| `user@1000.service.d/io-delegate.conf` → `[Service] DelegateControllers=cpu memory pids io` | the user manager can only request controllers it was delegated; read when it starts, so it lands on **the next reboot** (restarting it kills every rootless container on the host) |
|
||||
|
||||
Read the chain in this order — each line is the failure mode of the one after it:
|
||||
|
||||
```bash
|
||||
cat /sys/fs/cgroup/user.slice/cgroup.subtree_control # wants: io
|
||||
systemctl show user@1000.service -p DelegateControllers # wants: io
|
||||
cat /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/\<unit\>.service/libpod-payload-*/io.weight
|
||||
```
|
||||
|
||||
`io` missing from the first → the accounting drop-in is absent. Missing from the
|
||||
second → the host has not rebooted since the delegation landed (the deployer prints
|
||||
`NOTE: io is not delegated …` for exactly this). No file at the third → that unit
|
||||
carries no weight.
|
||||
|
||||
Two limits that survive the fix: a container's weight covers **everything** in its
|
||||
cgroup, so a library scan and a playback stream share one weight; and the comparison
|
||||
is made *inside* `user.slice`, so a rootless unit's weight cannot push it below a
|
||||
root unit (`frank-archon`). Shaping that pairing is the weight of `user.slice`
|
||||
itself — a fleet decision, not a unit line.
|
||||
|
||||
## 5. Rootless Quadlet log retrieval
|
||||
|
||||
Step 0 — establish ownership: `systemctl --user status <name>.service`
|
||||
|
||||
Reference in new issue
Block a user