Compare commits

..
Author SHA1 Message Date
Frank 0951d909c9 docs(agents): the bracket trick for a pkill pattern that must be named
The existing rule says don't `pkill -f` a pattern your own command line carries,
and to prefer `pkill -x` — but a remote one-shot often has to name the pattern,
and hitting it again today cost a silent `ssh` exit 255 with zero output, which
reads like a network fault. Record the escape (`"serve[-]script.sh"`), the zsh
glob-expansion trap that makes an unquoted `[s]` a no-op, and the exit-code
signature.
2026-11-02 04:47:16 -08:00
Frank 69effb542d docs(jellyfin): Shows is on providers, and the per-library door stays dead anyway
The `Shows` flip landed today (one field, proven in the server's own
`LibraryOptions` answer), so the posture table no longer calls it a silent file
index and `Music`/`Collections` are named as the remaining ones.

Two corrections the same day's measurements force:

- The per-library `POST /Items/{id}/Refresh` answered `204` and queued nothing
  both before the flip and after it — zero journal lines for ten minutes, and
  `Scan Media Library` kept `LastTaskResult=None, EndTimeUtc=None`, the never-run
  signature. The provider switch was never the reason that door is dead; only
  `POST /Library/Refresh` runs (204, then `Running` inside 15 s at 33 % CPU), and
  it walks every library, so a library-scoped pass has no door on this build.
  That paragraph also records the null-pair read that separates "never ran" from
  "just finished", and that query+body on the same call answers `400` with an
  empty body whatever the values.
- The key law: a plugin owns its own config file under its own
  `<PluginConfiguration>` root and omits null settings, so TVDB's `<ApiKey>`
  element never existed and `install` refused — which is what `MISSING key` had
  been reporting since October. PR #121 fixes the writer.

The `Shows` art census replaces speculation with the split that matters: 2,908 of
3,558 series+season carry a primary, 624 have no provider id, 26 have an id and no
poster. A `Download` campaign over them took 20 s and downloaded nothing, which is
the measurement that says identification is the only lever left.
2026-11-02 04:47:16 -08:00
Frank 8889a4ceff docs(jellyfin): episode stills are out of scope, and the cost that says so
A per-episode RemoteImages query is never cached, so the same campaign that
dressed 620 series in an afternoon costs ~11 s per episode for one hit in five.
Across 42k episodes that is days of provider traffic to dress a fifth of them, and
clients show series and season art regardless. Recorded so nobody runs the pass on
a hunch; the door and the resumable log are the same if Frank orders it.
2026-11-02 03:00:42 -08:00
Frank c21c48e61e docs(jellyfin): the door that scans is /Library/Refresh, and the art door names its field
The page sent the next agent to POST /Items/{id}/Refresh?Recursive=true per
library to make a new folder visible. Measured against a Shows library that had
never been identified: that route answers 204 and queues nothing — item census
byte-identical, ScheduledTasks idle, journal silent, and still a silent 204 after
a service restart. The door that works is POST /Library/Refresh, and the progress
read is ScheduledTasks -> Scan Media Library -> CurrentProgressPercentage.
RefreshStatus on the VirtualFolders record stays Idle through a real scan, so it
is not a progress read at all; a 204 cannot distinguish accepted-and-discarded
from accepted-and-queued, so the acceptance read is the journal naming folders.

Also recorded the art door's wire shape, because guessing it reads as a server
bug: the list response carries the address in Url (there is no ImageUrl), the
Download POST takes camelCase type/imageUrl and answers 204, and PascalCase or an
empty imageUrl answers 500. RemoteImages is empty for items with no ProviderIds,
so art coverage is capped by identification, not by the image door.
2026-11-02 02:56:12 -08:00
9 changed files with 94 additions and 528 deletions

No files matched your search

+6 -1
View File
@@ -735,7 +735,12 @@ linked runbooks.
**Ops** **Ops**
- Never `pkill -f` a pattern that appears in your own command line — it kills your ssh session. - Never `pkill -f` a pattern that appears in your own command line — it kills your ssh session.
Use `pkill -x <exact-name>`. Use `pkill -x <exact-name>`, or the bracket trick — `pkill -f "serve[-]script.sh"` —
when the pattern has to be named: the bracket makes the pattern stop matching the
command line that carries it. Quote it: a zsh remote glob-expands an unquoted `[s]`
and the kill silently never runs. Re-hit 2026-11-02 —
`ssh host 'pkill -f jfshowswatch.py; …'` killed the remote shell mid-command, and
ssh exits **255 with zero output**, which reads exactly like a network fault.
- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in committed scripts - Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in committed scripts
(no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts without it. (no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts without it.
- **ComfyUI on Dankclaw runs with GPU access** (`/dev/kfd` + `/dev/dri` via `AddDevice`, - **ComfyUI on Dankclaw runs with GPU access** (`/dev/kfd` + `/dev/dri` via `AddDevice`,
-188
View File
@@ -1,188 +0,0 @@
#!/usr/bin/env bash
# audiobookshelf-init.sh — make an Audiobookshelf server usable, from nothing.
#
# Why this is a script and not a browser session (live 2026-11-02, Dankserver):
# a freshly deployed Audiobookshelf answers /healthcheck with 200 while it has no
# user, no library and no scan — "healthy" says nothing about usefulness, and a
# service that only works after someone remembers to click through a first-run
# wizard is not deployed, it is installed. Everything below is the API the React
# client itself calls, so the same procedure runs unattended and lands in git.
#
# The API facts this script depends on, read off the pinned image rather than
# guessed (dist-server/server/Server.js inside the digest this repo pins):
# * the creation route is POST /init with {"newRoot":{"username","password"}}.
# There is NO /setup route in 2.37.x — POST /setup answers "Unauthorized",
# which reads exactly like an auth fault and is instead a missing route.
# * /status (no /api prefix) reports {"isInit": <has a root user>}; every route
# under /api needs the bearer token that POST /login returns.
# * the library count is NOT a field on the library DTO; the door is
# GET /api/libraries/{id}/items -> {"total": N}.
#
# The credential is generated once into a mode-600 host file and never enters
# the repo, the URL, or a process argument list. Same pattern as calibre-server.
#
# usage: audiobookshelf-init.sh # create what is missing, then scan
# ABS_SCAN=0 audiobookshelf-init.sh # bootstrap without a scan
#
# environment (defaults are the deployed shape):
# ABS_URL server base URL (http://127.0.0.1:8000)
# ABS_SECRET credential file (/home/dank/.config/secrets/audiobookshelf)
# ABS_LIBRARY library name to ensure (Audiobooks)
# ABS_FOLDER folder to bind inside the server (/audiobooks)
# ABS_SCAN 1 triggers a scan (default), 0 skips it
#
# exit 0 every step is satisfied (created or already present)
# exit 1 a step was refused or the server misbehaved (named on stderr)
# exit 2 usage / unreadable credential file
set -euo pipefail
ABS_URL=${ABS_URL:-http://127.0.0.1:8000}
ABS_SECRET=${ABS_SECRET:-/home/dank/.config/secrets/audiobookshelf}
ABS_LIBRARY=${ABS_LIBRARY:-Audiobooks}
ABS_FOLDER=${ABS_FOLDER:-/audiobooks}
ABS_SCAN=${ABS_SCAN:-1}
die() { printf 'refused: %s\n' "$*" >&2; exit 1; }
case "$ABS_SCAN" in
0 | 1) ;;
*) printf 'refused: ABS_SCAN wants 0 or 1, got %s\n' "$ABS_SCAN" >&2; exit 2 ;;
esac
# ---- credential -------------------------------------------------------------
umask 077
if [ ! -f "$ABS_SECRET" ]; then
mkdir -p "$(dirname "$ABS_SECRET")"
user=${ABS_ADMIN_USER:-frank}
# `tr … | head -c 24` is the obvious line and it is a trap under
# `set -o pipefail`: head exits at 24 bytes, tr takes SIGPIPE, the pipeline
# status is 141 and set -e kills the script before the credential is ever
# written (measured 2026-11-02). od reads a FIXED byte count and exits on its
# own, so no consumer closes a pipe early; bash slices the length.
pass=$(od -An -tx1 -N 32 /dev/urandom | tr -d ' \n')
pass=${pass:0:24}
[ "${#pass}" -eq 24 ] || die "could not generate a password (no /dev/urandom)"
printf 'ABS_ADMIN_USER=%s\nABS_ADMIN_PASSWORD=%s\n' "$user" "$pass" > "$ABS_SECRET"
printf 'credential: created %s\n' "$ABS_SECRET"
fi
[ -r "$ABS_SECRET" ] || die "$ABS_SECRET exists but is not readable by $(id -un)"
chmod 600 "$ABS_SECRET"
# ---- everything else, over HTTP ---------------------------------------------
# python3 rather than curl so the password and the bearer token never appear in
# any argv (they would be world-readable in /proc/<pid>/cmdline otherwise).
ABS_URL="$ABS_URL" ABS_SECRET="$ABS_SECRET" ABS_LIBRARY="$ABS_LIBRARY" \
ABS_FOLDER="$ABS_FOLDER" ABS_SCAN="$ABS_SCAN" python3 - <<'PY'
import json, os, sys, urllib.error, urllib.request
BASE = os.environ["ABS_URL"].rstrip("/")
LIB = os.environ["ABS_LIBRARY"]
FOLDER = os.environ["ABS_FOLDER"]
SCAN = os.environ["ABS_SCAN"] == "1"
cred = {}
for line in open(os.environ["ABS_SECRET"]):
if "=" in line:
k, _, v = line.strip().partition("=")
cred[k] = v
user = cred.get("ABS_ADMIN_USER") or "frank"
password = cred.get("ABS_ADMIN_PASSWORD") or ""
if not password:
sys.exit("refused: %s has no ABS_ADMIN_PASSWORD" % os.environ["ABS_SECRET"])
def call(method, path, body=None, token=None, want_json=True):
data = None
headers = {"X-Requested-With": "audiobookshelf-init"}
if body is not None:
data = json.dumps(body).encode()
headers["Content-Type"] = "application/json"
if token:
headers["Authorization"] = "Bearer " + token
req = urllib.request.Request(BASE + path, data=data, headers=headers, method=method)
try:
with urllib.request.urlopen(req, timeout=60) as r:
raw = r.read()
if not raw or not want_json:
return r.status, raw
try:
return r.status, json.loads(raw)
except ValueError:
# Express answers sendStatus() with the STATUS TEXT as the body:
# POST /init and the scan route both reply "OK" (measured on the
# pinned 2.37.2 image). A successful call must never be judged by
# whether its body happens to parse as JSON.
return r.status, raw.decode("utf-8", "replace")[:200]
except urllib.error.HTTPError as e:
raw = e.read()
try:
payload = json.loads(raw)
except Exception:
payload = raw.decode("utf-8", "replace")[:200]
return e.code, payload
except urllib.error.URLError as e:
# Not an HTTP answer at all: nothing listening, DNS dead, connection
# refused. HTTPError is a URLError subclass, so this arm must come after.
return 0, "unreachable (%s)" % e.reason
status, st = call("GET", "/status")
if status != 200 or not isinstance(st, dict):
sys.exit("refused: %s/status answered %s (%s) — is the service up?" % (BASE, status, st))
if st.get("isInit"):
print("account: already registered")
else:
code, body = call("POST", "/init", {"newRoot": {"username": user, "password": password}})
if code != 200:
sys.exit("refused: POST /init answered %s (%s)" % (code, body))
print("account: created (%s)" % user)
code, login = call("POST", "/login", {"username": user, "password": password})
if code != 200 or not isinstance(login, dict) or not login.get("user", {}).get("token"):
sys.exit("refused: login answered %s (%s) — the credential file does not match the server"
% (code, login))
token = login["user"]["token"]
print("token: obtained (%s)" % login["user"].get("type", "user"))
code, libs = call("GET", "/api/libraries", token=token)
if code != 200 or not isinstance(libs, dict) or "libraries" not in libs:
sys.exit("refused: GET /api/libraries answered %s (%s)" % (code, libs))
lib = next((l for l in libs["libraries"] if l.get("name") == LIB), None)
if lib:
print("library: exists %s (%s)" % (LIB, lib["id"]))
else:
body = {
"name": LIB,
"provider": "audiobook",
"icon": "book",
"folders": [{"fullPath": FOLDER}],
"settings": {
"coverEncoding": "jpg",
"skipMatchingMediaWithAsin": False,
"skipMatchingMediaWithIsbn": False,
"autoDetectCrossoverAudiobooks": False,
"ignoreAudiobookMetadata": False,
"useAudiobookFileMetadata": False,
},
"providerMetadata": {},
}
code, made = call("POST", "/api/libraries", body, token=token)
if code != 200:
sys.exit("refused: POST /api/libraries answered %s (%s)" % (code, made))
lib = made
print("library: created %s (%s) folder %s" % (LIB, lib["id"], FOLDER))
if SCAN:
code, body = call("POST", "/api/libraries/%s/scan" % lib["id"], {}, token=token,
want_json=False)
if code != 200:
sys.exit("refused: scan answered %s (%s)" % (code, body))
print("scan: accepted")
else:
print("scan: skipped (ABS_SCAN=0)")
code, items = call("GET", "/api/libraries/%s/items?limit=1" % lib["id"], token=token)
if code == 200 and isinstance(items, dict):
print("items: %s (a fresh scan raises this from 0; watch it, not /healthcheck)"
% items.get("total"))
PY
exit 0
-6
View File
@@ -1318,12 +1318,6 @@ deploy_audiobookshelf() {
# /healthcheck answers once the HTTP server is bound; the library scan that # /healthcheck answers once the HTTP server is bound; the library scan that
# follows runs in the background and must not gate the deploy. # follows runs in the background and must not gate the deploy.
wait_for_health audiobookshelf http://127.0.0.1:8000/healthcheck 90 wait_for_health audiobookshelf http://127.0.0.1:8000/healthcheck 90
# A server with no root user and no library is "healthy" and useless, so the
# bootstrapper runs after the health gate and its failure fails the deploy:
# it creates the account (credential in the mode-600 host secret file), the
# Audiobooks library over the /audiobooks bind, and fires the scan. Every
# step is idempotent, so a re-deploy costs one incremental scan.
scripts/audiobookshelf-init.sh
} }
# The content server's single admin account and the desktop's basic-auth # The content server's single admin account and the desktop's basic-auth
-6
View File
@@ -104,12 +104,6 @@ for path in "${CHANGED[@]}"; do
dankserver/jellyfin/quadlet/*|dankserver/plex/quadlet/*) dankserver/jellyfin/quadlet/*|dankserver/plex/quadlet/*)
add_test tests/media_overlay.bats add_test tests/media_overlay.bats
;; ;;
# The Audiobookshelf bootstrapper (account + library + scan over the API)
# and the deployer hook that runs it after the health gate.
scripts/audiobookshelf-init.sh|tests/audiobookshelf_init.bats)
add_test tests/audiobookshelf_init.bats
add_test tests/deployment_flow.bats
;;
shared/homepage/*) shared/homepage/*)
add_test tests/homepage_caddy.bats add_test tests/homepage_caddy.bats
add_test tests/homepage_config.bats add_test tests/homepage_config.bats
@@ -23,7 +23,7 @@ This repository tests its configuration at defined seams. The tests run with `ba
| Deployment flow | `tests/deployment_flow.bats`, `tests/deployment_scoping.bats` | scoped dispatcher exists, targets are known, service plans isolate inference, Caddy ownership is guarded, and SSH reachability is checked (skips offline) | | Deployment flow | `tests/deployment_flow.bats`, `tests/deployment_scoping.bats` | scoped dispatcher exists, targets are known, service plans isolate inference, Caddy ownership is guarded, and SSH reachability is checked (skips offline) |
| Quadlet units | `tests/*quadlet*.bats` | every unit is boot-persistent, self-healing, repo-provenanced, secret-free | | Quadlet units | `tests/*quadlet*.bats` | every unit is boot-persistent, self-healing, repo-provenanced, secret-free |
| E2E (browser) | `tests/e2e/` + `tests/e2e_scaffold.bats` | Playwright drives the live guard chain; the bats seam asserts structure and credential hygiene. See [[E2E Suite]] | | E2E (browser) | `tests/e2e/` + `tests/e2e_scaffold.bats` | Playwright drives the live guard chain; the bats seam asserts structure and credential hygiene. See [[E2E Suite]] |
| Per-service / unit seams | `tests/llm_units.bats`, `tests/frank_{archon,dragoon,zealot}_unit.bats`, `tests/llm_secrets.bats`, `tests/homepage_{caddy,checker,config,deploy,pages}.bats`, `tests/{audiobookshelf_init,anubis,authentik,hermes_webui,pihole,mount_entropy,searxng_limiter,autoresearch_measure,changed_tests}.bats`, `tests/{caddy,dankserver,frankbook,marreta,observability,open_webui,searxng}_quadlet.bats`, `tests/comfyui_rocm.bats` | the ~30 named seams the summary rows above do not list: LLM systemd-unit shape and env hygiene, homepage config/pages/deploy behaviour, and one detail-seam per deployed service | | Per-service / unit seams | `tests/llm_units.bats`, `tests/frank_{archon,dragoon,zealot}_unit.bats`, `tests/llm_secrets.bats`, `tests/homepage_{caddy,checker,config,deploy,pages}.bats`, `tests/{anubis,authentik,hermes_webui,pihole,mount_entropy,searxng_limiter,autoresearch_measure,changed_tests}.bats`, `tests/{caddy,dankserver,frankbook,marreta,observability,open_webui,searxng}_quadlet.bats`, `tests/comfyui_rocm.bats` | the ~30 named seams the summary rows above do not list: LLM systemd-unit shape and env hygiene, homepage config/pages/deploy behaviour, and one detail-seam per deployed service |
## Focused pre-commit testing ## Focused pre-commit testing
@@ -1,6 +1,6 @@
--- ---
type: reference type: reference
description: "Audiobookshelf on Dankserver: quadlet shape, the read-only array bind, state on the SSD, the API bootstrap, laws." description: "Audiobookshelf on Dankserver: quadlet shape, the read-only array bind, state on the SSD, first-run account, laws."
tags: tags:
- dankserver - dankserver
- audiobooks - audiobooks
@@ -36,34 +36,20 @@ manager) — **Internal-only:** LAN address, no vhost, no DNS name.
| Self-healing | `Restart=always`, `RestartSec=5`, `WantedBy=default.target` | | Self-healing | `Restart=always`, `RestartSec=5`, `WantedBy=default.target` |
| Deploy | `./scripts/deploy.sh dankserver --service audiobookshelf` | | Deploy | `./scripts/deploy.sh dankserver --service audiobookshelf` |
| Handler | `deploy_audiobookshelf` in `scripts/deploy-remote.sh` | | Handler | `deploy_audiobookshelf` in `scripts/deploy-remote.sh` |
| Library layout | one book per folder — ABS groups by the first directory holding a media file (see Laws); every move is journalled at `/mnt/entropy/media-relayout-audiobooks.jsonl` |
## Bootstrap ## First run
`scripts/audiobookshelf-init.sh` — run by the deployer after the health gate, so a 1. Open `http://192.168.0.235:8000` **from the LAN**. ABS accepts the initial
deployed ABS is a *usable* ABS: `/healthcheck` answers 200 on a server with no account creation only from a local-network address; a remote address gets a
user, no library and no scan, and nothing else in the deploy notices the refusal, not a form.
difference. The script is idempotent; re-running it costs one incremental scan. 2. Create the owner account, then add one **Audi Book** library with folder
`/audiobooks`. The path is the in-container name of the read-only bind.
3. The first scan reads the whole tree once. It runs in the background — the
deploy gate is already green by then, and `IOWeight=10` keeps it off the
critical path.
| Step | Door | Printed | Nothing in this repo holds an ABS credential: the account is created in the
|------|------|---------| browser and lives in the `config` bind on the SSD.
| credential | `~/.config/secrets/audiobookshelf` (mode 600, generated once, host-side) | `credential: created …` |
| account | `POST /init` with `{"newRoot":{"username","password"}}` | `account: created (frank)` / `already registered` |
| session | `POST /login` → bearer token | `token: obtained (root)` |
| library | `POST /api/libraries`, folder `/audiobooks`, provider `audiobook` | `library: created Audiobooks (<id>)` / `exists` |
| scan | `POST /api/libraries/<id>/scan` | `scan: accepted` |
| proof | `GET /api/libraries/<id>/items?limit=1` | `items: <total>` |
Sign in at `http://192.168.0.235:8000` as `frank`; the password is
`awk -F= '/^ABS_ADMIN_PASSWORD=/{print $2}' ~/.config/secrets/audiobookshelf` on
Dankserver. Change it in **Settings → Users → frank → Password**; the secret file
then holds a stale value and only matters again if the `config` bind is ever
destroyed (a fresh DB makes the script create a new owner from it).
A full scan is slow and runs in the background; watch the `items:` total climb,
never `/healthcheck`. It also reads files that are genuinely broken and logs
`[AudioFileScanner] Invalid data found when processing input` per file — that is
the file, not the bind: the same line count grows while the total climbs.
## Laws ## Laws
@@ -109,95 +95,10 @@ the file, not the bind: the same line count grows while the total climbs.
in front. Any public vhost requires the Anubis + Authentik chain first, and the in front. Any public vhost requires the Anubis + Authentik chain first, and the
outpost proxy provider has to exist before the vhost flips (external exposure outpost proxy provider has to exist before the vhost flips (external exposure
policy in `AGENTS.md`). policy in `AGENTS.md`).
- **There is no `/setup` route; the creation door is `POST /init`.** `/status`
(no `/api` prefix) reports `isInit` = "a root user exists", `POST /init` with
`{"newRoot":{"username","password"}}` creates it, `POST /login` returns the
bearer every `/api/*` route wants. `POST /setup` answers `Unauthorized` — that
is a missing route wearing an auth message. Read off `dist-server/server/Server.js`
inside the pinned digest rather than from a forum.
- **A 200 may carry plain text.** `res.sendStatus(200)` answers the body `OK`, and
`/init` and the scan route use it; a client that parses every success as JSON
crashes on the one call that matters (`tests/audiobookshelf_init.bats` pins it).
- **The item count is not a field on the library DTO, and paging never ends
itself.** The door is `GET /api/libraries/<id>/items?limit=1` → `total`; a poller
that guesses a plausible name (`numItems`) prints `0` forever beside a scan that
is working. Paging is on `page` (0-based) plus `limit`, and **`offset` is
ignored**: `limit=500&offset=500` returns the same 500 ids as `offset=0`, while
`page=1` shares none of them. A census that pages on `offset` therefore reads
`ceil(total / limit) × limit` rows containing only `limit` distinct items — the
exact origin of the "exactly 1,500" and "exactly 4,000" counts two runs
reported for one library. Page on `page`, count *unique ids*, stop on `total`,
and take the number after the pass settles (`total` moves during a scan).
- **A fresh scan is the only thing that notices new audio.** `inotify` does not
cross a rootless podman bind — measured for [[Jellyfin]], whose page owns that
law — so the in-container watcher sees nothing a host process files onto the
array, and the deployer's re-run of the bootstrapper is the cheap scan trigger.
- **One stray media file at a collection root turns the whole collection into one
book.** Every scan re-walks the tree from disk (`scanFolder` → `recurseFiles`),
and `utils/scandir.js:groupFileItemsIntoLibraryItemDirs` assigns each file to the
FIRST directory along its path that already has a group: the first file seen at a
root owns every deeper file, and the folders below it never become items. One
949 MB `.cbr` sitting at `_unsorted`'s root collapsed 652 book folders into a
single item of 5,693 tracks, titled after whichever file supplied metadata.
"Hundreds of books are missing" is a layout fault, not a scan fault.
- **"Media file" means audio OR ebook, so an audio-only census cannot see the file
that is hiding the library.** With `audiobooksOnly=false` (the deployed shape)
the predicate is `SupportedAudioTypes` ∪ `SupportedEbookTypes`: 20 audio
extensions (including `mp4`, `mpg`, `webm`) plus `epub pdf mobi azw3 cbr cbz`. A
census reporting "that root holds no loose audio" can be correct and still
useless — the file doing the shadowing here was a comic archive.
- **Deleting the covering item does not un-shadow anything.** Grouping is recomputed
from disk, so the next scan rebuilds the item from the same stray file (measured:
`DELETE`, then scan → `1 Added`, the item back at the same path). The fix is the
MOVE: give every stray root file a folder of its own (one file per folder is
ABS's own single-book layout; formats of one book share a folder), through the
journal-guarded mover, then delete the stale item so the old path is not kept,
then scan.
- **`?force=1` is neither a discovery arm nor a repair arm.** Both arms of
`POST /libraries/:id/scan` walk the disk, so finding new folders needs neither
and the deployer's cheap trigger stays on the plain arm. Force additionally
re-scans every item that already exists (measured 1:33 against 0:04 on ~4,600
items) and repaired nothing on a library whose residuals were file-level: item
count, path set and media-less count identical before and after.
- **`DELETE /api/items/:id` is a DB row; `?hard=1` is `fs.remove()`.** In
`LibraryItemController.delete`, `hardDelete = req.query.hard == 1` and then
`fs.remove(libraryItemPath)`. Pointed at a covering item, that path is the whole
swallowed subtree — `hard=1` there deletes every book beneath it off the array.
Assert the item's `path` first, and never pass `hard`.
- **`Invalid data found when processing input` names the file, not the bind.** The
scanner logs it per file and skips that file; a book whose files ALL fail never
becomes an item at all, so the disk-vs-DB folder census shows it as a book with
no item. That is corrupt media, not a lost book. Never clear a suspect file with
a lenient probe: `ffprobe -show_entries format=duration` happily returns three
hours for the WAV bytes wearing a `.mp3` name that the scanner refuses. Read the
log's file paths instead, and `stat -c %s` first — 0-byte files are the common
case, and a truncated download reports `moov atom not found`.
- **A `Disc 01` / `CD 1` subfolder is part of a book, not a book.** The extension
census counts those folders; ABS deliberately does not — its grouping attaches
their files to the ancestor item (measured: 110 such folders under one item
carrying all 192 tracks). Read the basename before believing a disk-vs-DB gap.
- **`.cbr`/`.cbz` group like media and never play.** They sit in the grouping
predicate, so they hide books, but the comic parser throws `ERAR_BAD_ARCHIVE`
(`parseComicMetadata` → `CbrComicBookExtractor`) and the item lands with no
media. A media-less row beside those lines is the file, not the scanner.
- **A scan is finished when a NEW scan uuid completes.** Match
`Library scan "<uuid>" completed in ...`. `podman logs --since 30m` is a sliding
window: the set of visible completion lines changes on its own as old lines age
out, so a poller keyed on that hash reads a mid-scan number and calls it done.
## Verification ## Verification
`./tests/run.sh` covers this unit through `tests/dankserver_quadlet.bats` `./tests/run.sh` covers this unit through `tests/dankserver_quadlet.bats`
(boot persistence, the SSD state roots, the `:ro` array bind, the mount gate, the (boot persistence, the SSD state roots, the `:ro` array bind, the mount gate, the
`wget` health check), `tests/audiobookshelf_init.bats` (the bootstrapper against a `wget` health check) and `tests/media_overlay.bats` (only [[Plex]] and
stub of the API above: create-once, idempotent re-run, no secret in a URL, refusal [[Jellyfin]] may reference the metadata view).
by URL when the server does not answer, and the deployer calling it only after the
health gate) and `tests/media_overlay.bats` (only [[Plex]] and [[Jellyfin]] may
reference the metadata view).
Five book folders hold media no scanner can read, so they stay out of the library:
three 2024 Nebula nominees in the `_unsorted` box (two `.m4b` with no `moov` atom,
one 36-file `.mp3` set that is not MPEG audio at all), `Anthony Heald/Bounty
Hunter Wars - Book 1` and `Star Wars Full Cast Audio Drama/Dark Empire III` — the
latter two also exist in readable form under their author folders. Fix by
re-acquiring the files; the scanner names every one of them in its log.
+72 -12
View File
@@ -89,13 +89,16 @@ looks exactly like an unconfigured library.
| Library | `CollectionType` | `EnableInternetProviders` | `EnableRealtimeMonitor` | | Library | `CollectionType` | `EnableInternetProviders` | `EnableRealtimeMonitor` |
|---|---|---|---| |---|---|---|---|
| `Movies` | movies | true (fixed 2026-11-02) | **false** | | `Movies` | movies | true (fixed 2026-11-02) | **false** |
| `Shows` | tvshows | **false** | true | | `Shows` | tvshows | true (fixed 2026-11-02) | true |
| `Music` | music | **false** | true | | `Music` | music | **false** | true |
| `Collections` | boxsets | **false** | false | | `Collections` | boxsets | **false** | false |
`Shows` and `Music` are silent file indexes in the sense above: a refresh answers `Music` and `Collections` are silent file indexes in the sense above: a refresh answers
`204`, stores nothing, logs nothing. Any metadata or artwork work on either one `204`, stores nothing, logs nothing. Any metadata or artwork work on either one
starts by flipping the switch, never by firing a scan. starts by flipping the switch, never by firing a scan. `Shows` sat in that set
until 2026-11-02; the door is `POST /Library/VirtualFolders/LibraryOptions` with
the library's own `ItemId` (below), and `scripts/jellyfin-providers.sh verify` is
the one command that says which libraries are still in it.
**The per-library option door on 12.x is `POST /Library/VirtualFolders/LibraryOptions`** **The per-library option door on 12.x is `POST /Library/VirtualFolders/LibraryOptions`**
— `GET` there answers `405`, which is how the route announces itself. The 10.x doors — `GET` there answers `405`, which is how the route announces itself. The 10.x doors
@@ -119,6 +122,33 @@ store on a build where the automatic image pass will not run at all — it is ho
the Movies library got its art: 2,515 of 2,564 films carry a primary now, and the the Movies library got its art: 2,515 of 2,564 films carry a primary now, and the
rest have none in TMDb (shorts like `Astartes`, files whose names match nothing). rest have none in TMDb (shorts like `Astartes`, files whose names match nothing).
**The Download door's exact wire shape:** the list response carries the address in
`Url` — there is no `ImageUrl` field — and the POST takes camelCase query params,
`?type=Primary&imageUrl=<Url>`, answering `204`; the stored tag then appears in
`ImageTags`. PascalCase params, or an `imageUrl` left empty, answer `500`, which
reads as a server fault and is a field-name fault.
**`RemoteImages` answers empty for an item with no `ProviderIds`** — no provider id,
nothing to look up. Art coverage is capped by identification, not by the image door:
the items a campaign returns nothing for are the unidentified ones, and no image
retry will change that.
**A `Download` campaign over a library that is already dressed does nothing in
seconds, and that is the measurement proving identification is the only lever
left.** Run over all 753 series and 2,805 seasons of `Shows` on 2026-11-02 it
classified 3,558 items in 20 s: 2,908 already carried a primary, 624 have no
provider id at all, 26 have an id and no poster in TMDb — zero downloads. The
artless remainder is not an image-door problem: it is the unnamed long tail (clip
collections, `(Warhammer TV)`-shaped directories), and the lever is folder names
or an identify pass, never another retry loop.
**Series and season art is fetched; episode stills are not, deliberately.** Every
episode query is a fresh provider round-trip — nothing caches per episode — measured
at ~11 s per item beside ~2 s per series on the same host, with one hit in five in a
sample. Across 42k episodes that is days of provider traffic to dress a fifth of them,
and clients display the series and season art anyway. If episode stills are ever
ordered, the loop is the same door and its log makes the pass resumable.
**API shape on 12.x:** `/Items` and `/Items/{id}` need `UserId` when **API shape on 12.x:** `/Items` and `/Items/{id}` need `UserId` when
authenticated by API key — without it the answer is `400 Error processing authenticated by API key — without it the answer is `400 Error processing
request`, which reads exactly like an authentication failure. `MetadataRefreshMode` request`, which reads exactly like an authentication failure. `MetadataRefreshMode`
@@ -232,9 +262,17 @@ scripts/jellyfin-providers.sh verify # plugin + key LENGTHS, never key bytes
- **The homelab TVDB key is a licensed key** (its JWT claims say - **The homelab TVDB key is a licensed key** (its JWT claims say
`community_supported: false`), so it needs **no PIN** — the optional `community_supported: false`), so it needs **no PIN** — the optional
`TVDB_PIN=` line exists only for a subscriber-shaped key. `TVDB_PIN=` line exists only for a subscriber-shaped key.
- **TVDB is installed and keyed but is not a metadata agent**: TMDb still - **TVDB is installed but is not a metadata agent**: TMDb still identifies
identifies everything. Enabling TVDB as a provider would re-fetch show everything. Enabling TVDB as a provider would re-fetch show metadata across
metadata across the library, so it is a deliberate, separate change. the library, so it is a deliberate, separate change.
- **A plugin owns its own config file, under its own root, and omits null
settings** — so the element an installer wants to fill may simply not exist,
and its root is `<PluginConfiguration>`, not `<Configuration>`. TVDB's file had
no `<ApiKey>` element at all, which is why `install` refused with
`no place to write <ApiKey>` from October until the anchor fix: the secret sat
valid in `provider-keys` the whole time, and `verify`'s `MISSING key` was this
fault, not a missing key. The writer now anchors on the file's own root and
still refuses by name when a file has none.
- **The deployer places the keys before it restarts jellyfin** - **The deployer places the keys before it restarts jellyfin**
(`ensure_provider_keys` in `scripts/deploy-remote.sh`) and warns instead of (`ensure_provider_keys` in `scripts/deploy-remote.sh`) and warns instead of
failing when `provider-keys` is absent, so a host without it still deploys. failing when `provider-keys` is absent, so a host without it still deploys.
@@ -284,12 +322,34 @@ anything. The fix is `systemctl --user restart jellyfin.service`, one POST after
health answers, and then believe `ScheduledTasks` State at 130 % CPU, not the health answers, and then believe `ScheduledTasks` State at 130 % CPU, not the
response code. response code.
**A scan that must see a new folder has to be recursive.** `POST /Library/Refresh` **The door that scans is `POST /Library/Refresh` (204), and `ScheduledTasks` is
is the incremental pass, and inotify does not cross a rootless podman bind, so the progress read.** A task that has never completed answers
folders created on the host stay invisible to it. The move is `LastTaskResult=None, EndTimeUtc=None`, and that pair is what separates "never
`POST /Items/{id}/Refresh?Recursive=true` (204) per library, the id from ran" from "finished a moment ago" — the difference between a door that is dead
`GET /Library/VirtualFolders` — its field is `ItemId`, not `Id` — and that same and one that merely answered `204`.
record's `RefreshStatus` is the progress read.
**The per-library `POST /Items/{id}/Refresh` queues nothing, and the provider
switch is not the reason.** Measured twice on 2026-11-02 against `Shows`: before
the flip and again after it, the call answered `204`, the journal wrote zero lines
over the following ten minutes, and `Scan Media Library` kept its never-run
signature; a service restart changes nothing. `POST /Library/Refresh` does run —
`204`, then `Running` inside 15 s at 33 % CPU — but it walks every library, so on
this build a library-scoped metadata pass has no working door and the global one
is the only lever.
**That endpoint takes its parameters from one source.** The documented query form
(empty body) and the JSON body form (no query) each parse; supplying both at once
answers `400` with an empty body whatever the values, and `ValidationFailure` is
rejected in the body while the query form accepts it.
**`RefreshStatus` on the `GET /Library/VirtualFolders` record is not a progress
read.** It stayed `Idle` through a scan that was verifiably running at 75 %. The
record's useful field is `ItemId`, and the live library set is still the entries
whose `Locations` are non-empty.
`inotify` does not cross a rootless podman bind, so folders created on the host
stay invisible until a pass like this runs; every pass that writes into a library
ends by firing one.
**`Movies` has no watcher at all.** `EnableRealtimeMonitor` is per library (table in **`Movies` has no watcher at all.** `EnableRealtimeMonitor` is per library (table in
*Data layout*) and it is false for `Movies` and `Collections`. Nothing notices a *Data layout*) and it is false for `Movies` and `Collections`. Nothing notices a
+1 -1
View File
@@ -44,7 +44,7 @@ is a locator: one row, one clause, no numbers.
| Page | Owns | | Page | Owns |
|---|---| |---|---|
| [[Audiobookshelf]] | Audiobookshelf on Dankserver: quadlet shape, the read-only array bind, state on the SSD, the API bootstrap, laws. | | [[Audiobookshelf]] | Audiobookshelf on Dankserver: quadlet shape, the read-only array bind, state on the SSD, first-run account, laws. |
| [[Calibre]] | Calibre is the e-book library manager that backs Calibre-Web. | | [[Calibre]] | Calibre is the e-book library manager that backs Calibre-Web. |
| [[Calibre-Web]] | Calibre-Web is a web application that browses, reads, and downloads eBooks from an existing Calibre library database. | | [[Calibre-Web]] | Calibre-Web is a web application that browses, reads, and downloads eBooks from an existing Calibre library database. |
| [[Copyparty]] | Copyparty is a lightweight file server and media streamer that runs as a single Python binary. | | [[Copyparty]] | Copyparty is a lightweight file server and media streamer that runs as a single Python binary. |
-200
View File
@@ -1,200 +0,0 @@
#!/usr/bin/env bats
# Seam: scripts/audiobookshelf-init.sh — the Audiobookshelf bootstrapper.
#
# A freshly deployed Audiobookshelf answers /healthcheck 200 while it has no
# user, no library and no scan, so "healthy" proves nothing about usefulness.
# This seam pins what the bootstrapper must make true, against a stub that
# speaks the same API as the pinned image (POST /init + POST /login at the
# root, bearer-authenticated /api/*, library count only on /items):
# 1. an unregistered server gains the account, the library and a scan, and the
# credential lands mode 600 with both keys;
# 2. a second run creates nothing new (exactly one /init, one library) and
# still fires the incremental scan;
# 3. neither the bearer token nor the password ever rides a URL — a secret in
# a URL or an argv is world-readable in /proc/<pid>/cmdline, which is how
# an HF token leaked into a transcript on 2026-09-16;
# 4. a server that cannot be reached is refused by URL with exit 1, never
# reported as success (a green deploy over a dead API is a lie);
# 5. ABS_SCAN outside 0/1 is a usage error raised before anything is touched;
# 6. the deployer runs the bootstrapper only after the health gate answered.
load helpers
setup() {
INIT="$REPO_ROOT/scripts/audiobookshelf-init.sh"
TMP="$(mktemp -d)"
PORT=18937
cat > "$TMP/stub.py" <<'PY'
import json, sys
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
port, logf = int(sys.argv[1]), sys.argv[2]
state = {"init": False, "user": None, "pw": None, "libs": []}
def log(line):
with open(logf, "a") as f:
f.write(line + "\n")
class H(BaseHTTPRequestHandler):
def log_message(self, *a):
pass
def send(self, code, body, as_json=False):
data = json.dumps(body).encode() if as_json else str(body).encode()
self.send_response(code)
self.send_header("Content-Type", "application/json" if as_json else "text/plain")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
def body(self):
n = int(self.headers.get("Content-Length") or 0)
return json.loads(self.rfile.read(n) or b"{}")
def authed(self):
return self.headers.get("Authorization") == "Bearer stub-token"
def do_GET(self):
path = self.path.split("?")[0]
log("GET %s auth=%s" % (self.path, "yes" if self.headers.get("Authorization") else "no"))
if path == "/status":
return self.send(200, {"app": "audiobookshelf", "isInit": state["init"]}, True)
if not self.authed():
return self.send(401, "Unauthorized")
if path == "/api/libraries":
return self.send(200, {"libraries": state["libs"]}, True)
if path.endswith("/items"):
return self.send(200, {"total": 42, "results": []}, True)
return self.send(404, "nope")
def do_POST(self):
path = self.path.split("?")[0]
payload = self.body()
log("POST %s auth=%s" % (self.path, "yes" if self.headers.get("Authorization") else "no"))
if path == "/init":
if state["init"]:
return self.send(500, "already initialized")
root = payload.get("newRoot", {})
state["init"] = True
state["user"] = root.get("username")
state["pw"] = root.get("password")
log("INIT %s" % root.get("username"))
return self.send(200, "OK")
if path == "/login":
if not state["init"]:
return self.send(401, "Unauthorized")
if (payload.get("username"), payload.get("password")) != (state["user"], state["pw"]):
return self.send(401, "Unauthorized")
return self.send(200, {"user": {"username": state["user"], "type": "root",
"id": "u1", "token": "stub-token"}}, True)
if not self.authed():
return self.send(401, "Unauthorized")
if path == "/api/libraries":
lib = {"id": "lib-1", "name": payload.get("name"), "folders": payload.get("folders", [])}
state["libs"].append(lib)
log("LIBRARY %s %s" % (lib["name"], lib["folders"]))
return self.send(200, lib, True)
if path.endswith("/scan"):
log("SCAN")
return self.send(200, "OK")
return self.send(404, "nope")
ThreadingHTTPServer(("127.0.0.1", port), H).serve_forever()
PY
python3 "$TMP/stub.py" "$PORT" "$TMP/requests.log" > "$TMP/stub.out" 2>&1 &
STUB_PID=$!
for _ in $(seq 1 50); do
curl -s -o /dev/null "http://127.0.0.1:$PORT/status" && break
sleep 0.1
done
}
teardown() {
[ -n "${STUB_PID:-}" ] && kill "$STUB_PID" 2>/dev/null || true
rm -rf "$TMP"
}
boot() {
env ABS_URL="http://127.0.0.1:$PORT" ABS_SECRET="$TMP/cred" \
ABS_ADMIN_USER=frank ABS_LIBRARY=Audiobooks ABS_FOLDER=/audiobooks \
ABS_SCAN="${SCAN:-1}" bash "$INIT"
}
@test "an unregistered server gains the account, the library and the scan" {
run boot
[ "$status" -eq 0 ]
printf '%s\n' "${lines[@]}" | grep -qx "credential: created $TMP/cred"
printf '%s\n' "${lines[@]}" | grep -q "^account: created (frank)"
printf '%s\n' "${lines[@]}" | grep -q "^token: obtained (root)"
printf '%s\n' "${lines[@]}" | grep -q "^library: created Audiobooks (lib-1)"
printf '%s\n' "${lines[@]}" | grep -q "^scan: accepted"
printf '%s\n' "${lines[@]}" | grep -q "^items: 42"
# The state roots of this service hold an account credential, so the file
# that carries it must never be group- or world-readable.
[ "$(stat -c %a "$TMP/cred")" = "600" ]
grep -qx 'ABS_ADMIN_USER=frank' "$TMP/cred"
grep -Eq '^ABS_ADMIN_PASSWORD=[0-9a-f]{24}$' "$TMP/cred"
# The folder the library indexes is the bind inside the container, not the
# host path — a library pointed at /mnt/entropy would answer 0 items and
# look like an empty audiobook collection.
grep -q "LIBRARY Audiobooks \[{'fullPath': '/audiobooks'}\]" "$TMP/requests.log"
}
@test "a second run creates nothing new and still fires the incremental scan" {
boot > /dev/null
[ -e "$TMP/requests.log" ]
run boot
[ "$status" -eq 0 ]
printf '%s\n' "${lines[@]}" | grep -q "^account: already registered"
printf '%s\n' "${lines[@]}" | grep -q "^library: exists Audiobooks (lib-1)"
printf '%s\n' "${lines[@]}" | grep -q "^scan: accepted"
# No second credential, no second account, no duplicate library: the deployer
# may run this on every re-deploy of the service.
! printf '%s\n' "${lines[@]}" | grep -q "credential: created"
[ "$(grep -c '^INIT ' "$TMP/requests.log")" -eq 1 ]
[ "$(grep -c '^LIBRARY ' "$TMP/requests.log")" -eq 1 ]
[ "$(grep -c '^SCAN$' "$TMP/requests.log")" -eq 2 ]
}
@test "the bearer token and the password never ride a URL" {
boot
# The stub logs the request line (method + path). A secret showing up there
# means it travelled as a query parameter — visible in access logs, shell
# history and /proc/<pid>/cmdline.
! awk '{print $2}' "$TMP/requests.log" | grep -q 'stub-token'
pw=$(awk -F= '/^ABS_ADMIN_PASSWORD=/{print $2}' "$TMP/cred")
[ -n "$pw" ]
! awk '{print $2}' "$TMP/requests.log" | grep -q -- "$pw"
# and the authenticated routes only answered once the header was present
grep -q '^GET /api/libraries auth=yes$' "$TMP/requests.log"
}
@test "a server that does not answer is refused by URL, not reported as success" {
run env ABS_URL="http://127.0.0.1:1" ABS_SECRET="$TMP/cred" bash "$INIT"
[ "$status" -eq 1 ]
[[ "$output" == *refused:* ]]
[[ "$output" == *"127.0.0.1:1"* ]]
}
@test "ABS_SCAN outside 0/1 is a usage error raised before anything is touched" {
run env ABS_URL="http://127.0.0.1:$PORT" ABS_SECRET="$TMP/cred" ABS_SCAN=2 \
bash "$INIT"
[ "$status" -eq 2 ]
[[ "$output" == *ABS_SCAN* ]]
[ ! -e "$TMP/cred" ]
}
@test "the deployer bootstraps only after the health gate has answered" {
body=$(awk '/^deploy_audiobookshelf\(\)/,/^}/' "$REPO_ROOT/scripts/deploy-remote.sh")
[ -n "$body" ]
printf '%s\n' "$body" | grep -q \
'wait_for_health audiobookshelf http://127\.0\.0\.1:8000/healthcheck'
printf '%s\n' "$body" | grep -q 'scripts/audiobookshelf-init\.sh'
printf '%s\n' "$body" | awk '
/wait_for_health audiobookshelf/ { health = NR }
/audiobookshelf-init\.sh/ { init = NR }
END { exit !(health && init && init > health) }'
}