docs(audiobookshelf): the grouping rule, not "shallowest folder with audio"

Corrects the law pushed earlier today. The mechanism is
`groupFileItemsIntoLibraryItemDirs` (utils/scandir.js), called on a live
`recurseFiles` walk of the tree EVERY scan: each file attaches to the FIRST
directory along its path that already has a group, so one media file at a
collection root owns everything beneath it. Measured: one 949 MB `.cbr` hid
652 book folders behind a single 5,693-track item.

- "Media file" is audio OR ebook (`audiobooksOnly=false` -> SupportedAudioTypes
  ∪ SupportedEbookTypes, 17 audio + 6 ebook extensions). An audio-only census
  cannot see the file that hides an Audiobooks library.
- Deleting the covering item does not fix a shadow: grouping is recomputed from
  disk and the next scan rebuilds the item (measured: `1 Added`, same path).
- Un-shadow by MOVE: one file per folder is ABS's own single-book layout.
- `?force=1` is not a discovery arm (both arms walk the disk; force re-scans
  every existing item, 1:33 against 0:04).
- Paging corrected: `offset` is ignored outright, `page` is the parameter.
- `DELETE /api/items/:id` is a DB row; `?hard=1` is `fs.remove()` of the whole
  swallowed subtree. Never pass `hard`.

Shape table gains the layout row and the move journal; the stale "1,103
shelves" scan count is destroyed.
This commit is contained in:
Frank committed 2026-11-02 03:43:34 -08:00
1 parent f541566714
commit f71c7ccce4
1 file changed
+38 -21
@@ -36,6 +36,7 @@ manager) — **Internal-only:** LAN address, no vhost, no DNS name.
| Self-healing | `Restart=always`, `RestartSec=5`, `WantedBy=default.target` | | Self-healing | `Restart=always`, `RestartSec=5`, `WantedBy=default.target` |
| Deploy | `./scripts/deploy.sh dankserver --service audiobookshelf` | | Deploy | `./scripts/deploy.sh dankserver --service audiobookshelf` |
| Handler | `deploy_audiobookshelf` in `scripts/deploy-remote.sh` | | Handler | `deploy_audiobookshelf` in `scripts/deploy-remote.sh` |
| Library layout | one book per folder — ABS groups by the first directory holding a media file (see Laws); every move is journalled at `/mnt/entropy/media-relayout-audiobooks.jsonl` |
## Bootstrap ## Bootstrap
@@ -59,7 +60,7 @@ Dankserver. Change it in **Settings → Users → frank → Password**; the secr
then holds a stale value and only matters again if the `config` bind is ever then holds a stale value and only matters again if the `config` bind is ever
destroyed (a fresh DB makes the script create a new owner from it). destroyed (a fresh DB makes the script create a new owner from it).
The scan of 1,103 shelves is slow and background; watch `items:` climb from 0, A full scan is slow and runs in the background; watch the `items:` total climb,
never `/healthcheck`. It also reads files that are genuinely broken and logs never `/healthcheck`. It also reads files that are genuinely broken and logs
`[AudioFileScanner] Invalid data found when processing input` per file — that is `[AudioFileScanner] Invalid data found when processing input` per file — that is
the file, not the bind: the same line count grows while the total climbs. the file, not the bind: the same line count grows while the total climbs.
@@ -120,31 +121,47 @@ the file, not the bind: the same line count grows while the total climbs.
- **The item count is not a field on the library DTO, and paging never ends - **The item count is not a field on the library DTO, and paging never ends
itself.** The door is `GET /api/libraries/<id>/items?limit=1` → `total`; a poller itself.** The door is `GET /api/libraries/<id>/items?limit=1` → `total`; a poller
that guesses a plausible name (`numItems`) prints `0` forever beside a scan that that guesses a plausible name (`numItems`) prints `0` forever beside a scan that
is working. An out-of-range `offset` returns the last item rather than an empty is working. Paging is on `page` (0-based) plus `limit`, and **`offset` is
page, so a census that loops "until empty" invents its own total: `limit=500` ignored**: `limit=500&offset=500` returns the same 500 ids as `offset=0`, while
against a `total` of 4,628 hands back exactly 5,000 rows — the collected count `page=1` shares none of them. A census that pages on `offset` therefore reads
is `ceil(total / limit) × limit`, which is why two runs reported "exactly 1,500" `ceil(total / limit) × limit` rows containing only `limit` distinct items — the
and "exactly 4,000" for one library. Stop on `total`, count *unique ids*, and exact origin of the "exactly 1,500" and "exactly 4,000" counts two runs
take the number after the pass settles (`total` itself moves during a scan). reported for one library. Page on `page`, count *unique ids*, stop on `total`,
and take the number after the pass settles (`total` moves during a scan).
- **A fresh scan is the only thing that notices new audio.** `inotify` does not - **A fresh scan is the only thing that notices new audio.** `inotify` does not
cross a rootless podman bind — measured for [[Jellyfin]], whose page owns that cross a rootless podman bind — measured for [[Jellyfin]], whose page owns that
law — so the in-container watcher sees nothing a host process files onto the law — so the in-container watcher sees nothing a host process files onto the
array, and the deployer's re-run of the bootstrapper is the cheap scan trigger. array, and the deployer's re-run of the bootstrapper is the cheap scan trigger.
- **A book is the shallowest folder that holds audio — and that item owns the - **One stray media file at a collection root turns the whole collection into one
subtree beneath it.** A folder with audio in its own file list becomes ONE book book.** Every scan re-walks the tree from disk (`scanFolder` → `recurseFiles`),
and the scanner folds everything below it into that book's tracks instead of and `utils/scandir.js:groupFileItemsIntoLibraryItemDirs` assigns each file to the
making items: 6 stray files at the root of `_unsorted` turned the whole FIRST directory along its path that already has a group: the first file seen at a
collection dump into a single item of 5,693 tracks, titled after whichever file root owns every deeper file, and the folders below it never become items. One
supplied metadata, with 542 real books inside it. So "hundreds of books are 949 MB `.cbr` sitting at `_unsorted`'s root collapsed 652 book folders into a
missing" is a layout fault, not a scan fault — and no scan, forced or not, single item of 5,693 tracks, titled after whichever file supplied metadata.
subdivides an existing item. Census the layout the server's way: a directory is "Hundreds of books are missing" is a layout fault, not a scan fault.
a book folder iff its *own* files include an audio extension. - **"Media file" means audio OR ebook, so an audio-only census cannot see the file
- **`DELETE /api/items/:id` is a DB row; `?hard=1` is `fs.remove()`.** Read that is hiding the library.** With `audiobooksOnly=false` (the deployed shape)
`LibraryItemController.delete`: `hardDelete = req.query.hard == 1` and then the predicate is `SupportedAudioTypes` ∪ `SupportedEbookTypes`: 20 audio
extensions (including `mp4`, `mpg`, `webm`) plus `epub pdf mobi azw3 cbr cbz`. A
census reporting "that root holds no loose audio" can be correct and still
useless — the file doing the shadowing here was a comic archive.
- **Deleting the covering item does not un-shadow anything.** Grouping is recomputed
from disk, so the next scan rebuilds the item from the same stray file (measured:
`DELETE`, then scan → `1 Added`, the item back at the same path). The fix is the
MOVE: give every stray root file a folder of its own (one file per folder is
ABS's own single-book layout; formats of one book share a folder), through the
journal-guarded mover, then delete the stale item so the old path is not kept,
then scan.
- **`?force=1` is not a discovery arm.** Both arms of `POST /libraries/:id/scan`
walk the disk; force additionally re-scans every item that already exists
(measured 1:33 against 0:04 on ~4,600 items). Finding new folders needs neither,
so the deployer's cheap trigger stays on the plain arm.
- **`DELETE /api/items/:id` is a DB row; `?hard=1` is `fs.remove()`.** In
`LibraryItemController.delete`, `hardDelete = req.query.hard == 1` and then
`fs.remove(libraryItemPath)`. Pointed at a covering item, that path is the whole `fs.remove(libraryItemPath)`. Pointed at a covering item, that path is the whole
swallowed subtree — `hard=1` there deletes every audiobook under it off the swallowed subtree — `hard=1` there deletes every book beneath it off the array.
array. Un-swallowing is: file the loose root files with the journal-guarded Assert the item's `path` first, and never pass `hard`.
mover, then delete the stale covering item with NO query string.
- **`Invalid data found when processing input` names the file, not the bind.** The - **`Invalid data found when processing input` names the file, not the bind.** The
scanner logs it per file and skips the file — the book never appears — so a scanner logs it per file and skips the file — the book never appears — so a
missing title beside those lines is library debris, not a server fault. Census of missing title beside those lines is library debris, not a server fault. Census of