dank
6d79709cc0
`ensure_media_view` ran `install` on a host where the documented base directory
did not exist, and `install` refuses by design ("put it on the SSD first") — so
tonight's cutover would have died at the restart step with nobody beside the
box. `media-view.sh ensure-base` now makes the documented directory, and the
deployer calls it first; two seams cover it (the subcommand's behaviour, and the
ordering invariant inside `ensure_media_view`).
Docs flip with the deploy: the view is live on Dankserver (both media servers
verified resolving real media through it), so every "never been deployed" claim
in the view, Plex, Jellyfin and AGENTS pages is corrected, and three laws join
them — a container restart discards the in-flight library scan and nothing
resumes it; Jellyfin 12.4 authenticates only the `MediaBrowser Token` form, so a
401 says nothing about the key until the `ApiKeys` table is read; and Plex
answers 32400 over TLS only, which makes a plaintext health probe read as dead.
149 lines
9.9 KiB
Markdown
149 lines
9.9 KiB
Markdown
---
|
|
type: reference
|
|
title: Media Metadata View
|
|
description: "The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback."
|
|
tags: [dankserver, media-server, jellyfin, plex, storage]
|
|
timestamp: 2026-10-30T00:00:00Z
|
|
---
|
|
|
|
# Media Metadata View
|
|
|
|
`/var/lib/media-meta/view` is an OverlayFS merge of the media library: the RAID6
|
|
array below, a directory on the NVMe above. It exists so [[Jellyfin]] can write
|
|
Kodi-format sidecars (`movie.nfo`, `tvshow.nfo`, `episode.nfo`, artwork) that
|
|
[[Plex]] can read. Sidecar metadata is only useful sitting next to the media
|
|
file, and the array is read-only and slow — so the media bytes stay there and the
|
|
*writes* land on SSD. One tree to both servers, one copy of the media.
|
|
|
|
The mount point is not a volume and holds nothing while unmounted. It sits inside
|
|
`/var/lib/media-meta` so that layers, kernel bookkeeping, the reap trash and the
|
|
merged door are one directory — and so the mount table grows no new `/mnt` entry
|
|
next to the array.
|
|
|
|
**Deployed on Dankserver — 2026-10-31, both servers.** `plex` binds
|
|
`/var/lib/media-meta/view:/mnt/entropy:ro` and `jellyfin` binds
|
|
`/var/lib/media-meta/view:/data:rw`; `findmnt` reports the overlay with
|
|
`lowerdir=/mnt/entropy upperdir=/var/lib/media-meta/upper`, and each container
|
|
was proven to resolve a real media file through it — `podman exec <svc> ls -la
|
|
<its data path>` is the whole proof, because both servers keep the paths their
|
|
databases recorded. Two service-scoped deploys, each gated by
|
|
`media-view.sh verify` and the Playwright suite.
|
|
|
|
Moving a bind means restarting the server, and **a restart discards that
|
|
server's in-flight library scan** — Jellyfin's 7 h 25 m walk ended
|
|
`"Status": "Aborted"` in `data/ScheduledTasks/<task-id>.js` and nothing resumes
|
|
it. Renames made on the array while the view was unmounted are invisible to both
|
|
watchers too, so the additive index has to be asked for by hand; the door is an
|
|
API key, and the row in the failure table below says what to check when it
|
|
refuses.
|
|
|
|
## Shape
|
|
|
|
| Item | Value |
|
|
|---|---|
|
|
| Lower layer | `/mnt/entropy` — the RAID6 array, source of truth, never written |
|
|
| Upper layer | `/var/lib/media-meta/upper` — NVMe; every write lands here |
|
|
| Work dir | `/var/lib/media-meta/work` — overlayfs internals, same filesystem |
|
|
| Mount point | `/var/lib/media-meta/view` — the merged tree both servers bind |
|
|
| Unit (repo) | `dankserver/host/systemd/media-meta-view.mount` — the source of truth |
|
|
| Unit (host) | `var-lib-media\x2dmeta-view.mount` — systemd names a mount unit after its escaped path (`-` becomes `\x2d`); `media-view.sh install` writes the repo file under that name |
|
|
| Installer | `scripts/media-view.sh` — `plan` / `install` / `verify` / `status` / `reap` |
|
|
| Written by | Jellyfin only (Plex binds read-only); nothing else in the fleet touches it |
|
|
|
|
Both consumers gate their start on the view with
|
|
`ExecStartPre=/home/dank/serverAppConfigs/scripts/wait-for-mount.sh` (it waits
|
|
for the mount, then refuses), and `scripts/deploy.sh dankserver --service
|
|
jellyfin|plex` calls `media-view.sh install` + `verify` before it restarts
|
|
either container.
|
|
|
|
| Consumer | Bind | Mode | Why that path |
|
|
|---|---|---|---|
|
|
| [[Jellyfin]] | `/var/lib/media-meta/view:/data` | rw | It is the metadata writer; its sidecars become SSD bytes |
|
|
| [[Plex]] | `/var/lib/media-meta/view:/mnt/entropy` | ro | Every path in the PMS database was recorded as `/mnt/entropy/...`; only the HOST side of the bind moved, so its database still resolves |
|
|
|
|
Putting the sidecars inside Plex's reach is not the same as making Plex use them:
|
|
a library parses `.nfo` only with its *Local media information* agent enabled, and
|
|
this change touches no library setting.
|
|
|
|
## Commands
|
|
|
|
```bash
|
|
scripts/media-view.sh plan # print the resolved shape, touch nothing
|
|
sudo scripts/media-view.sh install # create dirs, install the unit, mount, enable
|
|
scripts/media-view.sh verify # is the view really the overlay? (deploy gate)
|
|
scripts/media-view.sh status # mount options + upper-layer size/file count
|
|
scripts/media-view.sh reap # list ghost metadata (media moved, sidecars left)
|
|
sudo scripts/media-view.sh reap --apply # move ghosts to the SSD trash, never delete
|
|
```
|
|
|
|
`--upper-base DIR` and `--lower DIR` relocate the layers for `plan` and `reap`;
|
|
the defaults are the host values above.
|
|
|
|
## Laws
|
|
|
|
- **The upper layer must sit on non-rotational storage.** `install` and `verify`
|
|
refuse via `lsblk -dno ROTA` if the upper base lives on the array: an overlay
|
|
whose upper is the spinners is the original problem with extra steps, and it
|
|
silently turns every metadata write back into a seek.
|
|
- **A container that binds the array must refuse to start until the mount
|
|
answers.** Live 2026-10-31: the array's mount and the media containers raced at
|
|
boot, podman resolved the bind against the pre-mount directory, and both
|
|
servers came up blind — Jellyfin read an empty library and began
|
|
`Removing item, Type: "Season"` by the thousand. Two dependencies people reach
|
|
for do not work here, both measured on that host: `RequiresMountsFor=` in a
|
|
**rootless** unit is inert (the user manager cannot order against the system's
|
|
`mnt-entropy.mount`), and `ConditionPathIsMountPoint=` is fail-*dead* — an
|
|
unmet condition logs `skipped, unmet condition check` with `NRestarts=0` and no
|
|
start attempt ever follows, so the unit waits for a human. The working guard is
|
|
`scripts/wait-for-mount.sh` as `ExecStartPre`: it waits out the race, refuses by
|
|
name after its window, and `Restart=always` carries the unit from there.
|
|
systemd does not create a mount point, so `install` creates `view` itself;
|
|
without it the mount simply fails.
|
|
- **Never delete through the view.** A delete writes an overlayfs whiteout
|
|
(`.wh.<name>`) in the upper layer, which hides the array's copy without
|
|
removing it — the file reappears the moment the view is unmounted, and the
|
|
upper layer now carries a landmine. Deletions belong on the raw path
|
|
(`/mnt/entropy`), which the relayout tool already treats as the only writer.
|
|
`reap` refuses to remove whiteouts and only warns about them.
|
|
- **A relayout must be followed by `reap`.** Renames happen on the raw path, so
|
|
sidecars the view already wrote stay at the old coordinates: a ghost folder
|
|
the servers browse and find nothing in. `reap` lists upper-layer directories
|
|
with no counterpart below and moves them (with their subtree) into a dated
|
|
`.AGENTTRASH/` under the SSD base. It is recursive and file-safe: a sidecar
|
|
that exists only above is normal, so only *directories* are candidates.
|
|
- **File-watcher events do not cross the overlay.** Plex's NFO agent and
|
|
Jellyfin's real-time monitor see inotify events for writes that land *in the
|
|
upper layer*, not for changes on the array underneath. After any relayout or
|
|
externally-written metadata, trigger a library refresh — see
|
|
[[Media Relayout]] for the refresh procedure.
|
|
|
|
## Failure signatures
|
|
|
|
| Symptom | Meaning |
|
|
|---|---|
|
|
| `jellyfin.service` / `plex.service` fail with `refused: still not mounted after 90s: …` | The view or the array is not mounted. Do not start them anyway: `scripts/media-view.sh install`, then `verify`. |
|
|
| Servers answer, library looks empty, Jellyfin logs `Removing item, Type: "Season"` by the thousand | Blind bind — the container started before the mount and bound the pre-mount directory. `podman exec jellyfin ls /data` against `ls /mnt/entropy` is the whole diagnosis; restarting the container re-resolves the bind. |
|
|
| Jellyfin log: `Read-only file system : '/data/Movies/…/movie.nfo'` | The container is binding the raw array, not the view — the quadlet drifted back or the view is gone. `media-view.sh status`. |
|
|
| A folder appears in a media server with artwork/`.nfo` but no video | Ghost metadata from a relayout. `media-view.sh reap`. |
|
|
| `verify` prints `refused: …/view is not a mountpoint` | A plain directory is sitting where the overlay belongs. That is the fail-closed guard working; `install` on the box. |
|
|
| `install` prints `refused: upper base /var/lib/media-meta does not exist (put it on the SSD first)` | A fresh host. `install` deliberately never creates the base — a mistyped `--upper-base` must not build a metadata tree on the wrong filesystem — so `scripts/media-view.sh ensure-base` makes the documented one, which the deployer calls first. |
|
|
| A media server's scan record reads `"Status": "Aborted"` | Its container restarted (a cutover, an image bump, an OOM) and the walk was discarded; systemd's `Restart=always` restarts the *server*, never the scan. Trigger a refresh — see [[Media Relayout]]. |
|
|
| The Jellyfin API answers 401 to every token header | Almost always the header **form**, not the key: 12.4 authenticates only `Authorization: MediaBrowser Token="…"` and refuses a bare `X-Emby-Token` for every key, valid or not — and one collapsed backslash in the curl config produces the same 401. `[[Jellyfin]]` owns the block that works. Before concluding anything, prove the key against the server's own table: `select count(*) from ApiKeys where AccessToken = ?` on `jellyfin.db` opened `mode=ro&immutable=1`. |
|
|
| `systemctl status` finds no such unit | Look under the escaped name, `var-lib-media\x2dmeta-view.mount`, not the repo file name. |
|
|
|
|
## Rollback
|
|
|
|
Revert the commit that pointed the two quadlets at the view and re-deploy both
|
|
services — they bind `/mnt/entropy` read-only again, which is the pre-view shape.
|
|
The NVMe upper layer can stay: nothing reads it once the binds are back, and it
|
|
holds no state either server needs. Unmounting alone
|
|
(`systemctl stop 'var-lib-media\x2dmeta-view.mount'`) is safe while no container
|
|
binds it, and the array is never modified by any of this.
|
|
|
|
## Links
|
|
|
|
- [[Jellyfin]] — the metadata writer; its data layout section.
|
|
- [[Plex]] — the read-only consumer; in-place bind-mount history.
|
|
- [[Media Relayout]] — the tool that moves media on the raw path and the reason `reap` exists.
|
|
- [[Entropy Volume]] — the array underneath.
|