Merge pull request 'media-verify reads the WAL; the metadata view is live; duplicates get a page' (#115) from fix/media-verify-wal-and-prune-record into main
This commit was merged in pull request #115.
This commit is contained in:
dank committed
commit
010b9affc7
6 files changed
+180
-5
No files matched your search
@@ -818,6 +818,7 @@ per-attempt page and no per-swap ledger anywhere else.
|
||||
| Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` |
|
||||
| Movie-library relayout (Jellyfin format, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Relayout.md` |
|
||||
| SSD-backed metadata view (Jellyfin writes, Plex reads, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Metadata View.md` |
|
||||
| Duplicate films and the purge hold (Dankserver) | `serverAppConfigs-docs/Dankserver/Media Duplicate Prune.md` |
|
||||
|
||||
There is no page for a retired endpoint and no change diary. A payload swap
|
||||
appends one row to the live endpoint's **Eras** table; when the endpoint itself
|
||||
@@ -860,6 +861,6 @@ history.
|
||||
| Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. |
|
||||
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
|
||||
| Relayout the movie library | **Census → plan → read the exceptions → `scripts/media-apply.py`** — renames only, journal before every move; never hand-rename, never copy. |
|
||||
| Making media metadata visible to both media servers | **Bind `/var/lib/media-meta/view`** — the overlay `scripts/media-view.sh` installs (array below, NVMe above). Never write the array; never delete through the view. **Deployed on Dankserver 2026-10-31 — both media servers bind it; a restart for any reason discards the in-flight library scan, so re-trigger the refresh — see [[Media Metadata View]].** |
|
||||
| Making media metadata visible to both media servers | **Bind `/var/lib/media-meta/view`** — the overlay `scripts/media-view.sh` installs (array below, NVMe above). Never write the array; never delete through the view. **Deployed on Dankserver 2026-10-31, re-verified 2026-11-01: both quadlets bind the view, not the raw array. A restart for any reason discards that server's in-flight library scan, so re-trigger the refresh — see [[Media Metadata View]].** |
|
||||
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
|
||||
| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |
|
||||
@@ -0,0 +1,143 @@
|
||||
---
|
||||
type: reference
|
||||
title: Media Duplicate Prune
|
||||
description: "Find and hold duplicate films on the entropy array: the hash ladder, the keep rules, the guards, the purge."
|
||||
tags: [dankserver, media-server, storage, procedure]
|
||||
timestamp: 2026-11-01T01:00:00Z
|
||||
---
|
||||
|
||||
# Media Duplicate Prune
|
||||
|
||||
The movie library holds the same film twice in two different ways — the same file
|
||||
downloaded again, or the same film grabbed a second time at another quality — and
|
||||
[[Entropy Volume]] is permanently full, so both are worth finding. They need
|
||||
different rules, and neither is decided by name similarity alone.
|
||||
|
||||
`/mnt/entropy/media-park` was a third kind: a parking lot that held whole shows
|
||||
and films nothing served. It is gone; its contents went to `NeedsProcessing`.
|
||||
|
||||
## What counts
|
||||
|
||||
| Class | Test | Action |
|
||||
|---|---|---|
|
||||
| Byte-identical | equal size, equal content hash | keep one copy, hold the rest |
|
||||
| Same work, worse encode | same title, same year, every runtime within 5 % | keep the largest — Frank's rule |
|
||||
| Different cut | same title and year, runtimes more than 5 % apart | **never touched**: an extended and a theatrical version are different products |
|
||||
| Same words, different film | title matches, year or duration does not | never grouped — this is the false-merge class that would destroy a unique file |
|
||||
|
||||
## The screen
|
||||
|
||||
The array cannot afford a full-content read, so identity is established in steps
|
||||
and each step only narrows what the next one touches:
|
||||
|
||||
1. `scripts/media-census.py enumerate` — size is free, and identical files share it.
|
||||
2. Head+tail `blake2b`, 4 MiB at each end, on the size collisions only.
|
||||
3. **Before anything moves**, confirm with a sparse-window hash: 32 MiB at eight
|
||||
evenly spaced offsets. Re-hashing a remux-heavy set in full read 150 GiB for
|
||||
the better part of an hour; the sparse window cost 50 s, a file that merely
|
||||
shares its head and tail cannot survive it, and the destination is a
|
||||
recoverable hold rather than a deletion.
|
||||
4. Durations come from `scripts/media-census.py probe` — the relayout's own
|
||||
census. Do not hand-roll a second probe: it re-seeks the array for an answer
|
||||
the repo tool already produces, and a probe is the one census step that costs
|
||||
hours instead of seconds.
|
||||
|
||||
## Which copy survives
|
||||
|
||||
In order: not one inside an `extras`, `featurettes`, `samples` or `trailers`
|
||||
folder (an extra is not a copy of the film, and extra names collide across films);
|
||||
then the shallowest path; then the file whose name equals its `Title (year)`
|
||||
folder; then the largest. For the same-work class the largest is the answer, and
|
||||
the canonical-name rule only breaks a tie — without it a `Film - 1080p.mkv` can
|
||||
outlive `Film.mkv` for no reason but a space sorting before a dot.
|
||||
|
||||
## Guards
|
||||
|
||||
Each of these is a file that would have been destroyed without it:
|
||||
|
||||
- **`.!qB` is qBittorrent state, not a duplicate.** It can be byte-identical to
|
||||
the finished file while qB still counts the download incomplete; moving it
|
||||
breaks the re-check and re-downloads the film. Report it, leave it. Both
|
||||
leftovers in `Movies` are 13 months stale, so they are Frank's purge decision,
|
||||
not a prune decision.
|
||||
- **Require a year, and keep digits in the title.** Stripping digits merged
|
||||
*Akira — Teaser 1* with *Teaser 2*; without a year, two different extras that
|
||||
happened to share a name merged.
|
||||
- Both members must be feature length, and a name carrying `disc`, `cd` or `part`
|
||||
is excluded: half a multi-disc release is not a copy of the other half.
|
||||
- Media containers only. Grouping every extension buries the report in subtitle
|
||||
and cover pairs — and a duration guard that appears to fire on 720 groups is
|
||||
counting subtitles, not protecting films.
|
||||
|
||||
## The hold
|
||||
|
||||
Nothing is deleted. Every doomed copy moves into a dated hold with the same
|
||||
executor the relayout uses, so any file comes back:
|
||||
|
||||
```bash
|
||||
scripts/media-apply.py apply --plan /mnt/entropy/dup-prune-similar-plan.tsv \
|
||||
--root /mnt/entropy --journal /mnt/entropy/dup-prune-similar.jsonl # dry
|
||||
scripts/media-apply.py apply --plan … --root /mnt/entropy --journal … --apply
|
||||
scripts/media-apply.py verify --plan … --journal /mnt/entropy/dup-prune-similar.jsonl
|
||||
scripts/media-apply.py sweep --journal /mnt/entropy/dup-prune-similar.jsonl --root /mnt/entropy --apply
|
||||
scripts/media-apply.py rollback --journal /mnt/entropy/dup-prune-similar.jsonl --root /mnt/entropy --apply
|
||||
```
|
||||
|
||||
The plan rows are the tool's own format (`action src dst kind`, `mkdir` rows for
|
||||
every hold parent), and the planner re-checks the two things that go stale between
|
||||
analysis and apply: a file that has moved since the census, and a group whose
|
||||
keeper is missing — either one drops the whole group rather than half-applying it.
|
||||
|
||||
- **A hold on the same volume frees nothing.** The purge is a separate, approved
|
||||
act and the only thing that moves the free-space number; `df` does not budge
|
||||
when 1 TiB moves sideways.
|
||||
- `sweep` removes the release folders the pass emptied. Empty folders dated a year
|
||||
ago are torrent detritus (`Subtitle,info`, `Screens`, `Sample`) from a previous
|
||||
era, not this pass — read the mtimes before blaming the prune.
|
||||
- The 2026-11-01 pass held 226 files (~1.1 TiB) in
|
||||
`/mnt/entropy/.AGENTTRASH/20261101-duplicate-prune`, journals
|
||||
`dup-prune-exact.jsonl` and `dup-prune-similar.jsonl`. Purge pending Frank's word.
|
||||
|
||||
## After the pass
|
||||
|
||||
Both servers still hold rows for files that are now in the hold, and neither
|
||||
notices: the media bind is the [[Media Metadata View]], and array changes do not
|
||||
cross it as inotify events. Scan in the order the rescan law sets — Jellyfin to
|
||||
`Idle`, then Plex's Movies section alone, one accepted refresh each; the requests
|
||||
are owned by [[Jellyfin]] and [[Plex]].
|
||||
|
||||
As of 2026-11-01 the Plex leg is deliberately deferred: Plex is stopped and
|
||||
**masked** (`systemctl --user stop plex && systemctl --user mask plex`) so its
|
||||
metadata fetches cannot stack on Jellyfin's live scan, and a masked unit stays
|
||||
stopped through a reboot — `unmask` before starting it. Its Movies section (key 1)
|
||||
holds ~187 rows naming held paths until that one refresh is accepted.
|
||||
|
||||
The same evening's detritus pass is a second, separate hold: 423 files (12 MiB) in
|
||||
`/mnt/entropy/.AGENTTRASH/20261101-detritus-sweep`, plan `detritus-sweep-plan.tsv`,
|
||||
journal `detritus-sweep.jsonl`, 542 husks removed by `sweep`. It moved files only
|
||||
from folders whose whole subtree held no media, so it creates no stale rows and
|
||||
needs no scan. The guards that hold a folder back are the finding: external
|
||||
subtitle folders (428 of them, 6 GiB) and ripped Blu-ray streams — `.sup` PGS,
|
||||
`.thd`/`.eac3`/`.dts` audio, `.cci` captions, 25 GiB — are not trash, and an
|
||||
extension the census does not recognise is a reason to stop, not to sweep.
|
||||
|
||||
`scripts/media-verify.py` reads differently for a prune than for a relayout:
|
||||
`landed: 0` is the **correct** end state (no server may index the hold), so the
|
||||
verdict is `still at src: 0` with `REAL: 0`, and the rows that simply vanished
|
||||
from the database are the pass succeeding.
|
||||
|
||||
## Failure signatures
|
||||
|
||||
| Symptom | Meaning |
|
||||
|---|---|
|
||||
| A size group whose members split under the sparse hash | The head+tail screen collided. Those files stay exactly where they are and get reported. |
|
||||
| The planner holds far fewer files than the analyzer counted | Two implementations of "largest wins". The analyzer must emit `keep`/`hold` marks and the planner only transcribes them — a planner that re-derives the winner from the hold list alone drops the keeper out of the group and prunes almost nothing. |
|
||||
| `df` unchanged after a TiB of moves | Expected: the hold is on the same filesystem. Only the purge frees space. |
|
||||
| A film that "disappeared" after a prune | It is in the hold, named by the journal: `media-apply.py rollback --journal … --apply`. |
|
||||
|
||||
## Links
|
||||
|
||||
- [[Media Relayout]] — the census tools, the executor, the journal format.
|
||||
- [[Media Metadata View]] — why neither server sees the change without a scan.
|
||||
- [[Jellyfin]] · [[Plex]] — the scan requests, one server at a time.
|
||||
- [[Entropy Volume]] — the array that is always full.
|
||||
@@ -3,7 +3,7 @@ type: reference
|
||||
title: Media Metadata View
|
||||
description: "The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback."
|
||||
tags: [dankserver, media-server, jellyfin, plex, storage]
|
||||
timestamp: 2026-10-30T00:00:00Z
|
||||
timestamp: 2026-11-01T01:00:00Z
|
||||
---
|
||||
|
||||
# Media Metadata View
|
||||
@@ -29,6 +29,15 @@ was proven to resolve a real media file through it — `podman exec <svc> ls -la
|
||||
databases recorded. Two service-scoped deploys, each gated by
|
||||
`media-view.sh verify` and the Playwright suite.
|
||||
|
||||
Re-verified 2026-11-01: `mount` shows the overlay on
|
||||
`/var/lib/media-meta/view` (`lowerdir=/mnt/entropy`,
|
||||
`upperdir=/var/lib/media-meta/upper`), and the installed quadlets under
|
||||
`~/.config/containers/systemd/` carry the view binds:
|
||||
`Volume=/var/lib/media-meta/view:/data` for [[Jellyfin]],
|
||||
`Volume=/var/lib/media-meta/view:/mnt/entropy:ro` for [[Plex]]. Prove it in that
|
||||
order: `scripts/media-view.sh verify` (the deploy gate), then
|
||||
`grep -n Volume ~/.config/containers/systemd/{jellyfin,plex}.container`.
|
||||
|
||||
Moving a bind means restarting the server, and **a restart discards that
|
||||
server's in-flight library scan** — Jellyfin's 7 h 25 m walk ended
|
||||
`"Status": "Aborted"` in `data/ScheduledTasks/<task-id>.js` and nothing resumes
|
||||
@@ -37,6 +46,15 @@ watchers too, so the additive index has to be asked for by hand; the door is an
|
||||
API key, and the row in the failure table below says what to check when it
|
||||
refuses.
|
||||
|
||||
The cutover itself is an ordinary deploy: `deploy.sh dankserver --service
|
||||
jellyfin|plex` runs `ensure_media_view` (`install` then `verify`) before it
|
||||
restarts the container, so it must not be fired through the middle of a library
|
||||
scan. Two consequences are now live rather than theoretical: inotify events for
|
||||
array writes do not cross the merge, so any relayout or prune needs a manual
|
||||
library refresh; and a move can leave ghost sidecars above, so `reap` runs after
|
||||
a relayout — a pass that moves only media files with no sidecars next to them
|
||||
leaves the upper layer untouched, and `reap` correctly finds nothing.
|
||||
|
||||
## Shape
|
||||
|
||||
| Item | Value |
|
||||
|
||||
@@ -3,7 +3,7 @@ type: reference
|
||||
title: Media Relayout
|
||||
description: "Census, plan and apply the movie, TV, music and audiobook libraries: the gates, the refusals, the rollback."
|
||||
tags: [dankserver, media-server, jellyfin, plex, procedure]
|
||||
timestamp: 2026-10-30T00:00:00Z
|
||||
timestamp: 2026-11-01T01:00:00Z
|
||||
---
|
||||
|
||||
# Media Relayout
|
||||
@@ -138,6 +138,8 @@ HTTPD close without a single byte; and the same HTTPD drops requests that carry
|
||||
no `Accept` and client-identifier header (see the scan block above for the form
|
||||
that works).
|
||||
|
||||
A duplicate prune is the same census and the same proof, with its own keep rules: [[Media Duplicate Prune]].
|
||||
|
||||
## TV in the movie library
|
||||
|
||||
A box set that arrives in the movie share stays there until someone moves it:
|
||||
|
||||
@@ -27,7 +27,7 @@ public traffic can be fronted by Caddy like every other vhost.
|
||||
| Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
|
||||
| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
|
||||
| Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
|
||||
| Media | git: `/var/lib/media-meta/view` (the [[Media Metadata View]] overlay) bound at the same absolute path `/mnt/entropy` **read-only**; live today: `/mnt/entropy` itself, read-only — the view has never been installed. Either way every library path in the PMS database stays valid, zero re-scan |
|
||||
| Media | `/var/lib/media-meta/view` (the [[Media Metadata View]] overlay) bound at the same absolute path `/mnt/entropy` **read-only** — live since the 2026-10-31 cutover, so the PMS database's recorded paths keep resolving with zero re-scan |
|
||||
| Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
|
||||
| Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration |
|
||||
| Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` |
|
||||
@@ -40,7 +40,7 @@ public traffic can be fronted by Caddy like every other vhost.
|
||||
The migration moved zero bytes: the container bind-mounts the native data
|
||||
root where it already lived, and the media archive re-mounts at
|
||||
`/mnt/entropy` inside the container — only the HOST side of that bind has moved
|
||||
(the array today; the overlay view in git has never been deployed), so every
|
||||
(the array until the 2026-10-31 view cutover, the overlay since), so every
|
||||
absolute library path in the
|
||||
PMS database keeps resolving. The one metadata change: rootless podman
|
||||
cannot map the real host uid 997, so the fleet model applies — container
|
||||
@@ -87,6 +87,16 @@ root runs as the `dank` user, and the deployer adopts the data root with
|
||||
while the HTTP layer finishes loading; the first boot of a years-old
|
||||
library took ~10 minutes (2026-09 cutover). The deployer gate now allows
|
||||
600 s before failing; later boots are much faster (warm page cache).
|
||||
5. **`32400` answers TLS only to remote clients.** PMS requires secure
|
||||
connections, so `curl http://192.168.0.235:32400/identity` from any other
|
||||
machine closes without a single byte (curl exit 52) — that reads as a dead
|
||||
server beside a healthy one, and it is how a post-cutover health check gets
|
||||
misread. Probe with `curl -k https://192.168.0.235:32400/identity`. Two
|
||||
consequences: `plex.franksdank.cloud` works **only** because its `transport
|
||||
http` block carries `tls_insecure_skip_verify` (that option makes Caddy speak
|
||||
TLS to PMS's self-signed cert — delete it and the vhost 502s), and the scan
|
||||
block below rides `http://127.0.0.1` **inside** the container, where insecure
|
||||
connections are allowed.
|
||||
|
||||
## Triggering a scan
|
||||
|
||||
|
||||
@@ -52,6 +52,7 @@ is a locator: one row, one clause, no numbers.
|
||||
| [[Jellyfin]] | Jellyfin on Dankserver: quadlet unit, overlay metadata view, provider keys, ports, access, transcode status. |
|
||||
| [[mariadb]] | The MariaDB backend for RomM: quadlet unit, network alias, env-file secret handling, and the backup path. |
|
||||
| [[Media Audiobook Relayout]] | Re-filing /mnt/entropy/Audiobooks by Plex's chapter rows: narrator shelves, no renumbering, the refusals. |
|
||||
| [[Media Duplicate Prune]] | Find and hold duplicate films on the entropy array: the hash ladder, the keep rules, the guards, the purge. |
|
||||
| [[Media Metadata View]] | The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback. |
|
||||
| [[Media Music Relayout]] | Re-filing /mnt/entropy/Music by Jellyfin's tag rows: the census, the album key, the refusals. |
|
||||
| [[Media Relayout]] | Census, plan and apply the movie, TV, music and audiobook libraries: the gates, the refusals, the rollback. |
|
||||
|
||||
Reference in new issue
Block a user