From 71e7e94b94b3dc8aaefc7276627195cae10833a4 Mon Sep 17 00:00:00 2001 From: dank Date: Wed, 4 Nov 2026 11:58:21 -0800 Subject: [PATCH] docs(media): the 2026-11-04 purge, and the gate that proves a hold is safe to delete MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The duplicate-prune, TV-duplicate, Ghibli and promo-clip holds went out of the array trash on Frank's word: ~1.2 TB back. This replaces the three "purge pending Frank's word" rows with what happened. Two procedures the page did not carry: - The purge gate. `REAL: 0` in both databases only says no row points at a moved path; it says nothing about whether the film survived. Every held file needs a same-work carrier in the live library at >= its size, re-derived at purge time, because the relayout renames folders AND keeper files, accents defeat ASCII matching (Nausicaä), and a carrier is usually another container at another size. That supersedes the 2026-11-01 "re-verify by size" law, which is exactly the test that reports survivors as lost. - The purge mechanics. The repo purger acts on a root's own .AGENTTRASH and nothing else, so a partial purge is staged into a scratch root while the unapproved buckets sit parked at the array root as .HOLD--*. ./tests/run.sh: 824 assertions, all pass. --- .../Dankserver/Media Duplicate Prune.md | 60 +++++++++++++++---- 1 file changed, 48 insertions(+), 12 deletions(-) diff --git a/serverAppConfigs-docs/Dankserver/Media Duplicate Prune.md b/serverAppConfigs-docs/Dankserver/Media Duplicate Prune.md index 336ceeb..7d5da43 100644 --- a/serverAppConfigs-docs/Dankserver/Media Duplicate Prune.md +++ b/serverAppConfigs-docs/Dankserver/Media Duplicate Prune.md @@ -179,21 +179,57 @@ keeper is missing — either one drops the whole group rather than half-applying era, not this pass — read the mtimes before blaming the prune. - The 2026-11-01 pass held 226 files (~1.1 TiB) in `/mnt/entropy/.AGENTTRASH/20261101-duplicate-prune`, journals - `dup-prune-exact.jsonl` and `dup-prune-similar.jsonl`. Purge pending Frank's word. -- The 2026-11-02 TV pass holds 220 files (54 GiB) in + `dup-prune-exact.jsonl` and `dup-prune-similar.jsonl`. +- The 2026-11-02 TV pass held 220 files (54 GiB) in `/mnt/entropy/.AGENTTRASH/20261102-tv-duplicates` — plan `tv-dupes-plan.tsv`, journal `tv-dupes.jsonl`, 220 moves / 16 hold folders / 7 husks swept — and the - same day's TVShows detritus pass holds 109 files (552 KiB) in + same day's TVShows detritus pass staged 109 files (552 KiB) in `/mnt/entropy/.AGENTTRASH/20261102-tvshows-detritus` with plan - `tvshows-detritus-plan.tsv` and journal `tvshows-detritus.jsonl`. Both purges - pending Frank's word, beside the 2026-11-01 hold above. All four journals and - plans live at the `entropy` root, not `/tmp`. -- **Re-verify each survivor by size, not by the recorded path, before purging.** - Detection artifacts and plans key on paths, and a folder renamed by an - interactive session ([[Entropy Volume]], *Write doors*) makes its keeper look - deleted while it sits untouched under the new name. The 2026-11-01 audit hit - three such groups: every keeper was present, same bytes, different folder. A - purge decided from the plan's paths alone would have reported survivors gone. + `tvshows-detritus-plan.tsv` and journal `tvshows-detritus.jsonl`. All four + journals and plans live at the `entropy` root, not `/tmp`. +- **Purged 2026-11-04, on Frank's word, after the gate below:** the duplicate + prune, the TV duplicate pass, the Ghibli duplicates (6 files — both `Nausicaä` + copies among them) and the 2026-11-04 promo-clip hold left the array trash, + ~1.2 TB freed (the array went 218 GB → 1.4 TB available). The journals survive as + the only record of where a purged file came from; rollback after a purge is + impossible, which is why the gate runs first. + +### The purge gate: a carrier for every held file + +`REAL: 0` in both databases says only that no row points at a moved path; it says +nothing about whether the film survived. So a purge waits on a stronger proof: +**every held file has a same-work carrier in the live library at >= its size, +re-derived at purge time**, never copied from the plan. The 2026-11-04 purge re-found +all 452 held files this way; the first pass of the gate could not place 142 of them, +and every one of those was a matcher gap, not a lost film. What the re-derivation has +to absorb: + +- **The layout moves under the plan.** The relayout renames folders to + `Title (YYYY)` and `[tvdbid-N]/Season N` and renames the keeper files too, so the + paths *and* the sizes a plan recorded no longer describe the library. Keys are work + identity — normalized title+year for a film, `(S,E)` for a TV pack — never a path. +- **Fold accents, then compare.** The library holds `Nausicaä`; a plan, a + `find -ipath "*Nausicaa*"`, and an ASCII substring search all miss it. NFKD-fold and + drop combining marks, or a real keeper reads absent for two rounds and the file + beside it starts to look like an orphan. +- **The carrier is often another container, and usually larger.** A held `.mp4` can be + kept as the sibling `.mkv`; keep-larger means the keeper is frequently far bigger (a + 377 MB held episode matched a 501 MB keeper; a 1.6 GiB film stands beside a 34 GiB + BDRemux of itself in another folder). "Same folder, same bytes" reports survivors as + lost — it is the wrong test. +- **An exact twin can sit in a sibling folder**, so "no same-size file in the folder" + needs an array-wide name search before it means anything. + +### Running the purge + +`scripts/purge-agenttrash.sh` is the only door — never `rm`. It acts on exactly one +object: a root's own `.AGENTTRASH`. A partial purge is therefore **staged, not +filtered**: move the approved buckets into a scratch root +(`.PURGE-/.AGENTTRASH/`) and point the purger at that root, and park anything +not approved at the array root as `.HOLD--*`, out of its reach. Dry-run first — +the plan must print exactly one `plan empty` row, and it is the same row `--execute` +acts on — and send `--log` into `.media-campaign/`, because `/tmp` is not a record. +Then read `df`: a purge is the only media act that moves the free-space number. ## After the pass