Merge pull request 'fix(fleet): refuse a container start until the mount it binds has answered' (#109) from fix/array-mount-gate into main

This commit was merged in pull request #109.
This commit is contained in:
dank committed 2026-10-31 18:20:40 -07:00
commit 6d274acc47
8 files changed
+217 -16

No files matched your search

@@ -46,10 +46,11 @@ view really is a mountpoint.
| Installer | `scripts/media-view.sh` — `plan` / `install` / `verify` / `status` / `reap` |
| Written by | Jellyfin only (Plex binds read-only); nothing else in the fleet touches it |
Both consumers guard the view with
`UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view`, and
`scripts/deploy.sh dankserver --service jellyfin|plex` calls `media-view.sh
install` + `verify` before it restarts either container.
Both consumers gate their start on the view with
`ExecStartPre=/home/dank/serverAppConfigs/scripts/wait-for-mount.sh` (it waits
for the mount, then refuses), and `scripts/deploy.sh dankserver --service
jellyfin|plex` calls `media-view.sh install` + `verify` before it restarts
either container.
| Consumer | Bind | Mode | Why that path |
|---|---|---|---|
@@ -80,11 +81,20 @@ the defaults are the host values above.
refuse via `lsblk -dno ROTA` if the upper base lives on the array: an overlay
whose upper is the spinners is the original problem with extra steps, and it
silently turns every metadata write back into a seek.
- **Never mount the view over an unmounted lower.** `RequiresMountsFor=/mnt/entropy`
plus the container `ConditionPathIsMountPoint` guards: an empty `lowerdir`
presents an *empty library* as truth, and a media scan would then clear the
database of everything. systemd does not create a mount point, so `install`
creates `view` itself; without it the mount simply fails.
- **A container that binds the array must refuse to start until the mount
answers.** Live 2026-10-31: the array's mount and the media containers raced at
boot, podman resolved the bind against the pre-mount directory, and both
servers came up blind — Jellyfin read an empty library and began
`Removing item, Type: "Season"` by the thousand. Two dependencies people reach
for do not work here, both measured on that host: `RequiresMountsFor=` in a
**rootless** unit is inert (the user manager cannot order against the system's
`mnt-entropy.mount`), and `ConditionPathIsMountPoint=` is fail-*dead* — an
unmet condition logs `skipped, unmet condition check` with `NRestarts=0` and no
start attempt ever follows, so the unit waits for a human. The working guard is
`scripts/wait-for-mount.sh` as `ExecStartPre`: it waits out the race, refuses by
name after its window, and `Restart=always` carries the unit from there.
systemd does not create a mount point, so `install` creates `view` itself;
without it the mount simply fails.
- **Never delete through the view.** A delete writes an overlayfs whiteout
(`.wh.<name>`) in the upper layer, which hides the array's copy without
removing it — the file reappears the moment the view is unmounted, and the
@@ -107,7 +117,8 @@ the defaults are the host values above.
| Symptom | Meaning |
|---|---|
| `jellyfin.service` / `plex.service` show `ConditionPathIsMountPoint=... was not met` and stay inactive | The view is not mounted. Do not start them anyway: `scripts/media-view.sh install`, then `verify`. |
| `jellyfin.service` / `plex.service` fail with `refused: still not mounted after 90s: …` | The view or the array is not mounted. Do not start them anyway: `scripts/media-view.sh install`, then `verify`. |
| Servers answer, library looks empty, Jellyfin logs `Removing item, Type: "Season"` by the thousand | Blind bind — the container started before the mount and bound the pre-mount directory. `podman exec jellyfin ls /data` against `ls /mnt/entropy` is the whole diagnosis; restarting the container re-resolves the bind. |
| Jellyfin log: `Read-only file system : '/data/Movies/…/movie.nfo'` | The container is binding the raw array, not the view — the quadlet drifted back or the view is gone. `media-view.sh status`. |
| A folder appears in a media server with artwork/`.nfo` but no video | Ghost metadata from a relayout. `media-view.sh reap`. |
| `verify` prints `refused: …/view is not a mountpoint` | A plain directory is sitting where the overlay belongs. That is the fail-closed guard working; `install` on the box. |