Merge remote-tracking branch 'origin/main' into feat/jellyfin-provider-keys

# Conflicts: # AGENTS.md # serverAppConfigs-docs/Dankserver/Media Metadata View.md # serverAppConfigs-docs/index.md
2026-11-01 01:36:35 -07:00
commit 14f340370c
@@ -20,7 +20,16 @@ The mount point is not a volume and holds nothing while unmounted. It sits insid
 merged door are one directory — and so the mount table grows no new `/mnt` entry
 next to the array.

-**Deployed on Dankserver — verified 2026-11-01.** `mount` shows the overlay on
+**Deployed on Dankserver — 2026-10-31, both servers.** `plex` binds
+`/var/lib/media-meta/view:/mnt/entropy:ro` and `jellyfin` binds
+`/var/lib/media-meta/view:/data:rw`; `findmnt` reports the overlay with
+`lowerdir=/mnt/entropy upperdir=/var/lib/media-meta/upper`, and each container
+was proven to resolve a real media file through it — `podman exec <svc> ls -la
+<its data path>` is the whole proof, because both servers keep the paths their
+databases recorded. Two service-scoped deploys, each gated by
+`media-view.sh verify` and the Playwright suite.
+
+Re-verified 2026-11-01: `mount` shows the overlay on
 `/var/lib/media-meta/view` (`lowerdir=/mnt/entropy`,
 `upperdir=/var/lib/media-meta/upper`), and the installed quadlets under
 `~/.config/containers/systemd/` carry the view binds:
@@ -29,6 +38,14 @@ next to the array.
 order: `scripts/media-view.sh verify` (the deploy gate), then
 `grep -n Volume ~/.config/containers/systemd/{jellyfin,plex}.container`.

+Moving a bind means restarting the server, and **a restart discards that
+server's in-flight library scan** — Jellyfin's 7 h 25 m walk ended
+`"Status": "Aborted"` in `data/ScheduledTasks/<task-id>.js` and nothing resumes
+it. Renames made on the array while the view was unmounted are invisible to both
+watchers too, so the additive index has to be asked for by hand; the door is an
+API key, and the row in the failure table below says what to check when it
+refuses.
+
 The cutover itself is an ordinary deploy: `deploy.sh dankserver --service
 jellyfin|plex` runs `ensure_media_view` (`install` then `verify`) before it
 restarts the container, so it must not be fired through the middle of a library
@@ -51,10 +68,11 @@ leaves the upper layer untouched, and `reap` correctly finds nothing.
 | Installer | `scripts/media-view.sh` — `plan` / `install` / `verify` / `status` / `reap` |
 | Written by | Jellyfin only (Plex binds read-only); nothing else in the fleet touches it |

-Both consumers guard the view with
-`UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view`, and
-`scripts/deploy.sh dankserver --service jellyfin|plex` calls `media-view.sh
-install` + `verify` before it restarts either container.
+Both consumers gate their start on the view with
+`ExecStartPre=/home/dank/serverAppConfigs/scripts/wait-for-mount.sh` (it waits
+for the mount, then refuses), and `scripts/deploy.sh dankserver --service
+jellyfin|plex` calls `media-view.sh install` + `verify` before it restarts
+either container.

 | Consumer | Bind | Mode | Why that path |
 |---|---|---|---|
@@ -85,11 +103,20 @@ the defaults are the host values above.
  refuse via `lsblk -dno ROTA` if the upper base lives on the array: an overlay
  whose upper is the spinners is the original problem with extra steps, and it
  silently turns every metadata write back into a seek.
- **Never mount the view over an unmounted lower.** `RequiresMountsFor=/mnt/entropy`
-  plus the container `ConditionPathIsMountPoint` guards: an empty `lowerdir`
-  presents an *empty library* as truth, and a media scan would then clear the
-  database of everything. systemd does not create a mount point, so `install`
-  creates `view` itself; without it the mount simply fails.
+- **A container that binds the array must refuse to start until the mount
+  answers.** Live 2026-10-31: the array's mount and the media containers raced at
+  boot, podman resolved the bind against the pre-mount directory, and both
+  servers came up blind — Jellyfin read an empty library and began
+  `Removing item, Type: "Season"` by the thousand. Two dependencies people reach
+  for do not work here, both measured on that host: `RequiresMountsFor=` in a
+  **rootless** unit is inert (the user manager cannot order against the system's
+  `mnt-entropy.mount`), and `ConditionPathIsMountPoint=` is fail-*dead* — an
+  unmet condition logs `skipped, unmet condition check` with `NRestarts=0` and no
+  start attempt ever follows, so the unit waits for a human. The working guard is
+  `scripts/wait-for-mount.sh` as `ExecStartPre`: it waits out the race, refuses by
+  name after its window, and `Restart=always` carries the unit from there.
+  systemd does not create a mount point, so `install` creates `view` itself;
+  without it the mount simply fails.
 - **Never delete through the view.** A delete writes an overlayfs whiteout
  (`.wh.<name>`) in the upper layer, which hides the array's copy without
  removing it — the file reappears the moment the view is unmounted, and the
@@ -112,10 +139,14 @@ the defaults are the host values above.

 | Symptom | Meaning |
 |---|---|
-| `jellyfin.service` / `plex.service` show `ConditionPathIsMountPoint=... was not met` and stay inactive | The view is not mounted. Do not start them anyway: `scripts/media-view.sh install`, then `verify`. |
+| `jellyfin.service` / `plex.service` fail with `refused: still not mounted after 90s: …` | The view or the array is not mounted. Do not start them anyway: `scripts/media-view.sh install`, then `verify`. |
+| Servers answer, library looks empty, Jellyfin logs `Removing item, Type: "Season"` by the thousand | Blind bind — the container started before the mount and bound the pre-mount directory. `podman exec jellyfin ls /data` against `ls /mnt/entropy` is the whole diagnosis; restarting the container re-resolves the bind. |
 | Jellyfin log: `Read-only file system : '/data/Movies/…/movie.nfo'` | The container is binding the raw array, not the view — the quadlet drifted back or the view is gone. `media-view.sh status`. |
 | A folder appears in a media server with artwork/`.nfo` but no video | Ghost metadata from a relayout. `media-view.sh reap`. |
 | `verify` prints `refused: …/view is not a mountpoint` | A plain directory is sitting where the overlay belongs. That is the fail-closed guard working; `install` on the box. |
+| `install` prints `refused: upper base /var/lib/media-meta does not exist (put it on the SSD first)` | A fresh host. `install` deliberately never creates the base — a mistyped `--upper-base` must not build a metadata tree on the wrong filesystem — so `scripts/media-view.sh ensure-base` makes the documented one, which the deployer calls first. |
+| A media server's scan record reads `"Status": "Aborted"` | Its container restarted (a cutover, an image bump, an OOM) and the walk was discarded; systemd's `Restart=always` restarts the *server*, never the scan. Trigger a refresh — see [[Media Relayout]]. |
+| The Jellyfin API answers 401 to every token header | Almost always the header **form**, not the key: 12.4 authenticates only `Authorization: MediaBrowser Token="…"` and refuses a bare `X-Emby-Token` for every key, valid or not — and one collapsed backslash in the curl config produces the same 401. `[[Jellyfin]]` owns the block that works. Before concluding anything, prove the key against the server's own table: `select count(*) from ApiKeys where AccessToken = ?` on `jellyfin.db` opened `mode=ro&immutable=1`. |
 | `systemctl status` finds no such unit | Look under the escaped name, `var-lib-media\x2dmeta-view.mount`, not the repo file name. |

 ## Rollback