0 Fork 0

Files

T

dank e82c8b2d06 fix(fleet): refuse a container start until the mount it binds has answered

Dankserver rebooted with the array's mount and the media containers racing. A
rootless quadlet cannot order itself against the system's mnt-entropy.mount, so
`RequiresMountsFor=/mnt/entropy` is inert in the user manager: podman resolved
the bind against the pre-mount directory, Jellyfin and Plex came up blind, and
Jellyfin read an empty library and began `Removing item, Type: "Season"` by the
thousand. Copyparty had no guard at all.

`scripts/wait-for-mount.sh` is the gate: it waits for every named mountpoint,
then refuses by name. `ConditionPathIsMountPoint=` is NOT the fix — measured on
the host, an unmet condition logs `skipped, unmet condition check` with
NRestarts=0 and no start attempt ever follows, which is fail-dead, not
fail-closed. Jellyfin and Plex swap the condition for the gate; Copyparty gets
one; the law is recorded in AGENTS.md and the view page.

Verified on the host: the 5.7.0 quadlet generator accepts ExecStartPre in
[Service] (exit 0, all three units generated, gate present), the gate waits for
a mount that appears mid-window and proceeds, and it refuses a plain directory
after its window naming the path.

tests/media_overlay.bats pinned the old mechanism by name; its intent — both
media servers refuse to start without the view — now rides the gate.

2026-10-31 18:17:02 -07:00

8.6 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

Media Metadata View

/var/lib/media-meta/view is an OverlayFS merge of the media library: the RAID6 array below, a directory on the NVMe above. It exists so Jellyfin can write Kodi-format sidecars (movie.nfo, tvshow.nfo, episode.nfo, artwork) that Plex can read. Sidecar metadata is only useful sitting next to the media file, and the array is read-only and slow — so the media bytes stay there and the writes land on SSD. One tree to both servers, one copy of the media.

The mount point is not a volume and holds nothing while unmounted. It sits inside /var/lib/media-meta so that layers, kernel bookkeeping, the reap trash and the merged door are one directory — and so the mount table grows no new /mnt entry next to the array.

Not deployed on Dankserver — verified 2026-10-31. scripts/media-view.sh status answers view: /var/lib/media-meta/view NOT MOUNTED, /var/lib/media-meta does not exist, and the installed quadlets under ~/.config/containers/systemd/ still carry the pre-view binds — /mnt/entropy:/data:ro for Jellyfin, /mnt/entropy:/mnt/entropy:ro for Plex. Everything below is the shape git describes, not the shape the servers run. The cutover is an ordinary deploy: deploy.sh dankserver --service jellyfin|plex runs ensure_media_view (install then verify) before it restarts the container, so it must not be fired through the middle of a library scan. Until that deploy lands there is no upper layer, so no relayout can leave ghost metadata and reap has nothing to do — media-apply.py prints that follow-up only when the view really is a mountpoint.

Shape

Item	Value
Lower layer	`/mnt/entropy` — the RAID6 array, source of truth, never written
Upper layer	`/var/lib/media-meta/upper` — NVMe; every write lands here
Work dir	`/var/lib/media-meta/work` — overlayfs internals, same filesystem
Mount point	`/var/lib/media-meta/view` — the merged tree both servers bind
Unit (repo)	`dankserver/host/systemd/media-meta-view.mount` — the source of truth
Unit (host)	`var-lib-media\x2dmeta-view.mount` — systemd names a mount unit after its escaped path (`-` becomes `\x2d`); `media-view.sh install` writes the repo file under that name
Installer	`scripts/media-view.sh` — `plan` / `install` / `verify` / `status` / `reap`
Written by	Jellyfin only (Plex binds read-only); nothing else in the fleet touches it

Both consumers gate their start on the view with ExecStartPre=/home/dank/serverAppConfigs/scripts/wait-for-mount.sh (it waits for the mount, then refuses), and scripts/deploy.sh dankserver --service jellyfin|plex calls media-view.sh install + verify before it restarts either container.

Consumer	Bind	Mode	Why that path
Jellyfin	`/var/lib/media-meta/view:/data`	rw	It is the metadata writer; its sidecars become SSD bytes
Plex	`/var/lib/media-meta/view:/mnt/entropy`	ro	Every path in the PMS database was recorded as `/mnt/entropy/...`; only the HOST side of the bind moved, so its database still resolves

Putting the sidecars inside Plex's reach is not the same as making Plex use them: a library parses .nfo only with its Local media information agent enabled, and this change touches no library setting.

Commands

scripts/media-view.sh plan              # print the resolved shape, touch nothing
sudo scripts/media-view.sh install      # create dirs, install the unit, mount, enable
scripts/media-view.sh verify            # is the view really the overlay? (deploy gate)
scripts/media-view.sh status            # mount options + upper-layer size/file count
scripts/media-view.sh reap              # list ghost metadata (media moved, sidecars left)
sudo scripts/media-view.sh reap --apply # move ghosts to the SSD trash, never delete

--upper-base DIR and --lower DIR relocate the layers for plan and reap; the defaults are the host values above.

Laws

The upper layer must sit on non-rotational storage. install and verify refuse via lsblk -dno ROTA if the upper base lives on the array: an overlay whose upper is the spinners is the original problem with extra steps, and it silently turns every metadata write back into a seek.
A container that binds the array must refuse to start until the mount answers. Live 2026-10-31: the array's mount and the media containers raced at boot, podman resolved the bind against the pre-mount directory, and both servers came up blind — Jellyfin read an empty library and began Removing item, Type: "Season" by the thousand. Two dependencies people reach for do not work here, both measured on that host: RequiresMountsFor= in a rootless unit is inert (the user manager cannot order against the system's mnt-entropy.mount), and ConditionPathIsMountPoint= is fail-dead — an unmet condition logs skipped, unmet condition check with NRestarts=0 and no start attempt ever follows, so the unit waits for a human. The working guard is scripts/wait-for-mount.sh as ExecStartPre: it waits out the race, refuses by name after its window, and Restart=always carries the unit from there. systemd does not create a mount point, so install creates view itself; without it the mount simply fails.
Never delete through the view. A delete writes an overlayfs whiteout (.wh.<name>) in the upper layer, which hides the array's copy without removing it — the file reappears the moment the view is unmounted, and the upper layer now carries a landmine. Deletions belong on the raw path (/mnt/entropy), which the relayout tool already treats as the only writer. reap refuses to remove whiteouts and only warns about them.
A relayout must be followed by reap. Renames happen on the raw path, so sidecars the view already wrote stay at the old coordinates: a ghost folder the servers browse and find nothing in. reap lists upper-layer directories with no counterpart below and moves them (with their subtree) into a dated .AGENTTRASH/ under the SSD base. It is recursive and file-safe: a sidecar that exists only above is normal, so only directories are candidates.
File-watcher events do not cross the overlay. Plex's NFO agent and Jellyfin's real-time monitor see inotify events for writes that land in the upper layer, not for changes on the array underneath. After any relayout or externally-written metadata, trigger a library refresh — see Media Relayout for the refresh procedure.

Failure signatures

Symptom	Meaning
`jellyfin.service` / `plex.service` fail with `refused: still not mounted after 90s: …`	The view or the array is not mounted. Do not start them anyway: `scripts/media-view.sh install`, then `verify`.
Servers answer, library looks empty, Jellyfin logs `Removing item, Type: "Season"` by the thousand	Blind bind — the container started before the mount and bound the pre-mount directory. `podman exec jellyfin ls /data` against `ls /mnt/entropy` is the whole diagnosis; restarting the container re-resolves the bind.
Jellyfin log: `Read-only file system : '/data/Movies/…/movie.nfo'`	The container is binding the raw array, not the view — the quadlet drifted back or the view is gone. `media-view.sh status`.
A folder appears in a media server with artwork/`.nfo` but no video	Ghost metadata from a relayout. `media-view.sh reap`.
`verify` prints `refused: …/view is not a mountpoint`	A plain directory is sitting where the overlay belongs. That is the fail-closed guard working; `install` on the box.
`systemctl status` finds no such unit	Look under the escaped name, `var-lib-media\x2dmeta-view.mount`, not the repo file name.

Rollback

Revert the commit that pointed the two quadlets at the view and re-deploy both services — they bind /mnt/entropy read-only again, which is the pre-view shape. The NVMe upper layer can stay: nothing reads it once the binds are back, and it holds no state either server needs. Unmounting alone (systemctl stop 'var-lib-media\x2dmeta-view.mount') is safe while no container binds it, and the array is never modified by any of this.

8.6 KiB Raw Blame History