Compare commits

..
Author SHA1 Message Date
dank 183a6d5442 feat(fleet): give the media servers an I/O share that actually exists
A Plex section scan walked 50k files at ~850 r/s and nothing bounded it.
`IOWeight=` in a quadlet looks like the fix and does nothing: systemd delegates
`cpu memory pids` to user@.service and NOT io, and it drops a resource line it
cannot honour without a word — the container boots healthy and its cgroup has no
io.weight file at all. Same family as RequiresMountsFor= (inert in the user
manager) and ConditionPathIsMountPoint= (fail-dead).

Place the two system drop-ins that make the controller reachable (user.slice
IOAccounting=yes puts io in user.slice/cgroup.subtree_control, measured
immediately; user@1000.service DelegateControllers gains io and lands at the
next reboot, because restarting the user manager would kill every rootless
container on the host), then ask for the share: IOWeight=10 on jellyfin and plex.
10 is a convention, not a measured optimum — proportional share, only bites
under contention, only requirement is far below the default 100.

ensure_user_io_delegate places both from git and prints NOTE: io is not
delegated … inert until the next reboot, so the quadlet line never reads as live
config. Verified on the host: subtree_control flipped to `cpu io memory pids`,
the drop-in is attached and parsed (systemctl cat), and the podman 5.7 generator
emits IOWeight=10 for both units (bare key; UnitConfigItem= not needed).

Two limits recorded instead of hidden: a container's weight covers its whole
cgroup (scan and playback share it), and the comparison happens inside
user.slice, so it cannot push a rootless unit below a root unit such as
frank-archon — that pairing is user.slice's own weight, a separate call.

Docs: Quadlet Lifecycle 4b (the three-command read of the chain), Pi-hole (all
rootless containers share one 10.0.2.100, so one chatty container throttles DNS
for the host — 8 events today), E2E Suite gotcha 7 (the gate depends on the
runner's resolver: 21 reds on a lost wired link, 37/37 minutes later).

Seam: an in-repo IOWeight= may only exist when both drop-ins exist AND the
deployer places them. ./tests/run.sh: 789 ok, exit 0.
2026-10-31 18:42:27 -07:00
dank 6d274acc47 Merge pull request 'fix(fleet): refuse a container start until the mount it binds has answered' (#109) from fix/array-mount-gate into main 2026-10-31 18:20:40 -07:00
dank 3344b51fbd Merge pull request 'perf(media): the apply fsyncs the journal once per batch, not per move' (#108) from perf/media-apply-batch into main 2026-10-31 18:20:17 -07:00
dank e82c8b2d06 fix(fleet): refuse a container start until the mount it binds has answered
Dankserver rebooted with the array's mount and the media containers racing. A
rootless quadlet cannot order itself against the system's mnt-entropy.mount, so
`RequiresMountsFor=/mnt/entropy` is inert in the user manager: podman resolved
the bind against the pre-mount directory, Jellyfin and Plex came up blind, and
Jellyfin read an empty library and began `Removing item, Type: "Season"` by the
thousand. Copyparty had no guard at all.

`scripts/wait-for-mount.sh` is the gate: it waits for every named mountpoint,
then refuses by name. `ConditionPathIsMountPoint=` is NOT the fix — measured on
the host, an unmet condition logs `skipped, unmet condition check` with
NRestarts=0 and no start attempt ever follows, which is fail-dead, not
fail-closed. Jellyfin and Plex swap the condition for the gate; Copyparty gets
one; the law is recorded in AGENTS.md and the view page.

Verified on the host: the 5.7.0 quadlet generator accepts ExecStartPre in
[Service] (exit 0, all three units generated, gate present), the gate waits for
a mount that appears mid-window and proceeds, and it refuses a plain directory
after its window naming the path.

tests/media_overlay.bats pinned the old mechanism by name; its intent — both
media servers refuse to start without the view — now rides the gate.
2026-10-31 18:17:02 -07:00
15 changed files with 363 additions and 20 deletions

No files matched your search

+9
View File
@@ -156,6 +156,15 @@ Unit requirements:
(`<host>/<svc>/quadlet/`), is installed by the deployer, and is covered by (`<host>/<svc>/quadlet/`), is installed by the deployer, and is covered by
the quadlet test seam. The host-side unit is generated state; the repo file the quadlet test seam. The host-side unit is generated state; the repo file
is the source of truth. is the source of truth.
- **Bind-source gate** — a unit whose `Volume=` source is a mount this host owns
(the entropy array, the metadata view) carries
`ExecStartPre=/home/dank/serverAppConfigs/scripts/wait-for-mount.sh <source>`.
Measured 2026-10-31: in a **rootless** unit `RequiresMountsFor=` is inert (the
user manager cannot order against the system's mount unit), and
`ConditionPathIsMountPoint=` is fail-*dead* — an unmet condition logs `skipped,
unmet condition check` with `NRestarts=0` and no start attempt ever follows.
Bind the array before it is mounted and the container sees the empty pre-mount
directory as truth; a media scan then deletes the library it cannot see.
Before declaring a service deployed: `systemctl --user is-enabled <name>` → Before declaring a service deployed: `systemctl --user is-enabled <name>` →
`enabled`, `is-active` → `active`, and the container answers its health check. `enabled`, `is-active` → `active`, and the container answers its health check.
@@ -36,6 +36,9 @@ Volume=/home/dank/Containers/copyparty/cfg:/cfg
PublishPort=3923:3923 PublishPort=3923:3923
[Service] [Service]
# /w is the array: serving the pre-mount directory instead of the volume shows an
# empty share and nothing complains, so the mount has to answer first.
ExecStartPre=/home/dank/serverAppConfigs/scripts/wait-for-mount.sh /mnt/entropy
EnvironmentFile=/home/dank/Containers/copyparty/env EnvironmentFile=/home/dank/Containers/copyparty/env
Restart=always Restart=always
RestartSec=5 RestartSec=5
@@ -0,0 +1,14 @@
# The io controller is what makes IOWeight= mean anything, and systemd does not
# enable it below `user.slice` by default: with io accounting off, `io` never
# appears in `user.slice/cgroup.subtree_control`, so no rootless container can
# have an `io.weight` file and `IOWeight=` in a quadlet is silently inert.
# Measured on Dankserver 2026-10-31: setting this property flipped
# `user.slice/cgroup.subtree_control` from `cpu memory pids` to `cpu io memory
# pids` immediately, with nothing restarted.
#
# This is half the fix; the other half is delegating io to user@1000.service
# (user@1000.service.d/io-delegate.conf). Both must be in place before a reboot,
# because the delegation is read when the user manager starts. Law and the
# verification command: [[Quadlet Lifecycle]].
[Slice]
IOAccounting=yes
@@ -0,0 +1,12 @@
# The system manager gives the user manager cpu/memory/pids by default (systemd
# 259: `systemctl show user@1000.service -p DelegateControllers` → `cpu memory
# pids`). Without `io` in that set no `io.weight` file can exist anywhere below
# `user.slice`, so `IOWeight=` in a rootless quadlet is silently inert — the same
# class of trap as `RequiresMountsFor=`. Granting the controller here is what
# makes the media servers' I/O share real.
#
# Inert until `user@1000.service` starts again, i.e. the next reboot (restarting
# it would kill every rootless container on the host). Verify after the reboot
# with the check in [[Quadlet Lifecycle]].
[Service]
DelegateControllers=cpu memory pids io
+14 -4
View File
@@ -43,11 +43,10 @@ PublishPort=1900:1900/udp
Volume=/home/dank/Containers/jellyfin/config:/config Volume=/home/dank/Containers/jellyfin/config:/config
Volume=/home/dank/Containers/jellyfin/cache:/cache Volume=/home/dank/Containers/jellyfin/cache:/cache
# The view, writable: Jellyfin is the metadata writer (movie.nfo / tvshow.nfo / # The view, writable: Jellyfin is the metadata writer (movie.nfo / tvshow.nfo /
# episode.nfo + artwork) and those bytes belong on the SSD. The condition is the # episode.nfo + artwork) and those bytes belong on the SSD. The gate in [Service]
# fail-closed guard — starting against a plain, unmounted view directory would let # is the guard: started against a plain, unmounted view directory a scan decides
# a scan conclude the library is empty. # the library is empty and deletes what it cannot see.
Volume=/var/lib/media-meta/view:/data Volume=/var/lib/media-meta/view:/data
UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view
AddDevice=/dev/dri AddDevice=/dev/dri
GroupAdd=keep-groups GroupAdd=keep-groups
HealthCmd=curl -f http://localhost:8096/health HealthCmd=curl -f http://localhost:8096/health
@@ -56,8 +55,19 @@ HealthTimeout=10s
HealthRetries=3 HealthRetries=3
[Service] [Service]
# Refuse to start until the bind source is a real mount — see the header of
# scripts/wait-for-mount.sh for the blind-boot incident this prevents.
ExecStartPre=/home/dank/serverAppConfigs/scripts/wait-for-mount.sh /var/lib/media-meta/view /mnt/entropy
Restart=always Restart=always
RestartSec=5 RestartSec=5
# A library scan reads the array for minutes on end. Weighted LOW (default is
# 100) so that when the array is contended a scan loses to playback and to every
# other container this user runs. Proportional share, not a cap: it changes
# nothing while the disk is idle. 10 is a convention — the requirement is only
# that it sit far below 100 — and it is inert until user@1000.service is
# delegated the io controller ([[Quadlet Lifecycle]]).
IOWeight=10
[Install] [Install]
WantedBy=default.target WantedBy=default.target
+10 -1
View File
@@ -16,12 +16,21 @@ Volume=/var/lib/plexmediaserver:/config
# to the overlay view re-points Plex at the sidecars without touching its DB. # to the overlay view re-points Plex at the sidecars without touching its DB.
# Read-only: Plex consumes the NFO files Jellyfin writes and writes no metadata. # Read-only: Plex consumes the NFO files Jellyfin writes and writes no metadata.
Volume=/var/lib/media-meta/view:/mnt/entropy:ro Volume=/var/lib/media-meta/view:/mnt/entropy:ro
UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view
PublishPort=32400:32400 PublishPort=32400:32400
[Service] [Service]
# Refuse to start until the bind source is a real mount: Plex blind is how paths
# get marked missing in its own database. Both the view and the array under it
# must answer — scripts/wait-for-mount.sh names the 2026-10-31 incident.
ExecStartPre=/home/dank/serverAppConfigs/scripts/wait-for-mount.sh /var/lib/media-meta/view /mnt/entropy
Restart=always Restart=always
RestartSec=5 RestartSec=5
# See the jellyfin unit for the reasoning: a section scan gets a LOW weight
# (default 100) so a walk of 50k files loses the array to playback and to every
# other container under this user. Inert until user@1000.service holds the io
# controller ([[Quadlet Lifecycle]]).
IOWeight=10
[Install] [Install]
WantedBy=default.target WantedBy=default.target
+38 -3
View File
@@ -1208,14 +1208,47 @@ ensure_media_view() {
# array below stays read-only and every sidecar Jellyfin writes lands on the # array below stays read-only and every sidecar Jellyfin writes lands on the
# NVMe. # NVMe.
# Raise it BEFORE either container starts — a start against a bare view # Raise it BEFORE either container starts — a start against a bare view
# directory would let a scan clear the library database. The units also carry # directory would let a scan clear the library database. Each unit gates on
# ConditionPathIsMountPoint, so a missing view skips the start rather than # scripts/wait-for-mount.sh, which waits and then refuses the start. A
# damaging anything. A relayout must be followed by `media-view.sh reap` # relayout must be followed by `media-view.sh reap`
# (see the Media Relayout page), which this is not. # (see the Media Relayout page), which this is not.
run_privileged bash scripts/media-view.sh install run_privileged bash scripts/media-view.sh install
bash scripts/media-view.sh verify bash scripts/media-view.sh verify
} }
ensure_user_io_delegate() {
# The media servers ask for an I/O share (IOWeight= in their quadlets) so a
# library scan cannot starve playback or the array's other reader. Two system
# drop-ins are what make that request real, and both are host state, so the
# deployer places them:
# user.slice.d/io-accounting.conf io enters user.slice's subtree
# user@1000.service.d/io-delegate.conf the user manager gets the controller
# Without the first, `io` never appears in user.slice/cgroup.subtree_control,
# so no container under it can have an io.weight file at all; without the
# second the user manager never asks for the controller. systemd hands
# user@.service cpu/memory/pids by default and drops a line it cannot honour
# without a word — IOWeight= on its own is inert config.
local src dst
for src in \
dankserver/host/systemd/user.slice.d/io-accounting.conf \
dankserver/host/systemd/user@1000.service.d/io-delegate.conf; do
dst=/etc/systemd/system/${src#dankserver/host/systemd/}
run_privileged mkdir -p "$(dirname "$dst")"
if ! run_privileged cmp -s "$src" "$dst"; then
run_privileged install -m 644 "$src" "$dst"
run_privileged systemctl daemon-reload
fi
done
# Accounting lands immediately; the delegation is read when user@1000.service
# starts, and restarting that unit would kill every rootless container on this
# host. So the share becomes real at the next reboot — say so out loud rather
# than let a quadlet line read as live config.
if ! systemctl show user@1000.service -p DelegateControllers | grep -qw io; then
echo "NOTE: io is not delegated to user@1000.service yet - IOWeight= is"
echo "NOTE: inert until the next reboot (see [[Quadlet Lifecycle]])"
fi
}
ensure_provider_keys() { ensure_provider_keys() {
# Jellyfin's metadata-provider keys live in the container-owned plugin # Jellyfin's metadata-provider keys live in the container-owned plugin
# configuration under the data root, never in git; scripts/jellyfin-providers.sh # configuration under the data root, never in git; scripts/jellyfin-providers.sh
@@ -1237,6 +1270,7 @@ ensure_provider_keys() {
deploy_jellyfin() { deploy_jellyfin() {
ensure_media_view ensure_media_view
ensure_user_io_delegate
ensure_provider_keys ensure_provider_keys
cp "dankserver/jellyfin/quadlet/jellyfin.container" "$quadlet_unit_dir/" cp "dankserver/jellyfin/quadlet/jellyfin.container" "$quadlet_unit_dir/"
systemctl --user daemon-reload systemctl --user daemon-reload
@@ -1326,6 +1360,7 @@ deploy_copyparty() {
deploy_plex() { deploy_plex() {
ensure_media_view ensure_media_view
ensure_user_io_delegate
cp "dankserver/plex/quadlet/plex.container" "$quadlet_unit_dir/" cp "dankserver/plex/quadlet/plex.container" "$quadlet_unit_dir/"
systemctl --user daemon-reload systemctl --user daemon-reload
if ! systemctl --user is-active -q plex.service; then if ! systemctl --user is-active -q plex.service; then
+74
View File
@@ -0,0 +1,74 @@
#!/usr/bin/env bash
# wait-for-mount.sh — refuse a container start until what it binds is mounted.
#
# Why this exists (live 2026-10-31, Dankserver): a rootless quadlet cannot order
# itself against a SYSTEM mount unit. `RequiresMountsFor=/mnt/entropy` in a user
# unit resolves to nothing and is inert, so when the array's mount finished a
# second after podman resolved the bind, Jellyfin and Plex started against the
# pre-mount directory. Their library root looked EMPTY and Jellyfin began
# deleting metadata by the thousand (`Removing item, Type: "Season"`). A blind
# bind is not a cosmetic fault; it is a metadata-deleting machine.
#
# A `ConditionPathIsMountPoint=` is NOT the fix, measured on the same host: an
# unmet condition logs "skipped, unmet condition check", NRestarts=0, and no
# start attempt ever follows — the unit stays dead until a human starts it.
# That is fail-dead. This script WAITS (the mount usually lands within seconds)
# and only then refuses, so `Restart=always` carries the unit across the race.
#
# usage: wait-for-mount.sh [-t SECONDS] MOUNTPOINT [MOUNTPOINT...]
# exit 0 every path is a mountpoint
# exit 1 a path is still not a mountpoint after the window (named on stderr)
# exit 2 usage error
set -euo pipefail
usage() { printf 'usage: %s [-t SECONDS] MOUNTPOINT [MOUNTPOINT...]\n' "$0"; }
window=90
while getopts ":t:h" opt; do
case "$opt" in
t) window=$OPTARG ;;
h) usage; exit 0 ;;
*) usage >&2; exit 2 ;;
esac
done
shift $((OPTIND - 1))
[ "$#" -ge 1 ] || { usage >&2; exit 2; }
case "$window" in
'' | *[!0-9]*) printf 'refused: -t wants whole seconds, got %s\n' "$window" >&2; exit 2 ;;
esac
is_mount() { findmnt -M "$1" -rno TARGET >/dev/null 2>&1; }
report() {
local why=$1; shift
printf '%s:' "$why"
for p in "$@"; do printf ' %s' "$p"; done
printf '\n'
}
deadline=$(( $(date +%s) + window ))
warned=0
while :; do
missing=()
for path in "$@"; do
is_mount "$path" || missing+=("$path")
done
[ "${#missing[@]}" -eq 0 ] && break
if [ "$(date +%s)" -ge "$deadline" ]; then
report "refused: still not mounted after ${window}s" "${missing[@]}" >&2
printf 'refusing to start: a container bind-mounted here would see the empty\n' >&2
printf 'directory underneath and conclude the library is empty — which is how a\n' >&2
printf 'scan begins deleting metadata it cannot find.\n' >&2
exit 1
fi
if [ "$warned" -eq 0 ]; then
report "waiting for mount(s) this unit depends on" "${missing[@]}"
warned=1
fi
sleep 1
done
[ "$warned" -eq 0 ] || printf 'mounted; proceeding.\n'
exit 0
@@ -100,3 +100,12 @@ aborts with `DEPLOY INCOMPLETE: e2e gate FAILED` when it does not pass
hand-off list (`HANDOFFS`) instead of a single host. The guard contract is hand-off list (`HANDOFFS`) instead of a single host. The guard contract is
untouched: challenge cleared, no authentik `uidField`, status < 500, and every untouched: challenge cleared, no authentik `uidField`, status < 500, and every
host NOT in the list still has to land on its own vhost. host NOT in the list still has to land on its own vhost.
7. **The runner's resolver is part of the suite.** A 2026-10-31 deploy of one
Dankserver quadlet failed the gate with 21 reds spread over SSO, homepage RBAC
and every vhost crawl — precisely the specs that navigate to an absolute
`*.franksdank.cloud` name, while every spec on the relative `baseURL` stayed
green. The error was `net::ERR_NAME_NOT_RESOLVED`: the workstation had lost its
wired link, and on the fallback its resolver was no longer pinned to Pi-hole.
Nothing on the fleet was wrong — the same suite re-run minutes later passed
37/37. Before a mass red gets blamed on the guard chain, run
`resolvectl query grafana.franksdank.cloud` on the box that runs the gate.
@@ -167,6 +167,43 @@ Gate: `tests/quadlet_image_pins.bats`. Where upstream publishes no version
tags (copyparty-ac stops at 1.19.15), pin `:latest@sha256:` and name the tags (copyparty-ac stops at 1.19.15), pin `:latest@sha256:` and name the
version the config label carries in the comment beside it. version the config label carries in the comment beside it.
## 4b. Resource control needs host permission (measured 2026-10-31, Dankserver)
`IOWeight=` in a rootless quadlet does nothing on its own, and nothing says so.
The system manager delegates `cpu memory pids` to `user@.service` and **not** `io`;
systemd drops a resource line it cannot honour **without a word** — the generator
still emits `IOWeight=10`, the container still starts and answers its health check,
and the cgroup simply has no `io.weight` file. Same family as
`RequiresMountsFor=` (inert in the user manager) and `ConditionPathIsMountPoint=`
(fail-dead): the line is accepted, the unit is happy, the setting is dead.
Two system drop-ins make it real, both placed by `ensure_user_io_delegate` in
`scripts/deploy-remote.sh` (repo copies under `dankserver/host/systemd/`):
| File | Effect |
|---|---|
| `user.slice.d/io-accounting.conf` → `[Slice] IOAccounting=yes` | puts `io` into `user.slice/cgroup.subtree_control`; lands immediately, no restart |
| `user@1000.service.d/io-delegate.conf` → `[Service] DelegateControllers=cpu memory pids io` | the user manager can only request controllers it was delegated; read when it starts, so it lands on **the next reboot** (restarting it kills every rootless container on the host) |
Read the chain in this order — each line is the failure mode of the one after it:
```bash
cat /sys/fs/cgroup/user.slice/cgroup.subtree_control # wants: io
systemctl show user@1000.service -p DelegateControllers # wants: io
cat /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/\<unit\>.service/libpod-payload-*/io.weight
```
`io` missing from the first → the accounting drop-in is absent. Missing from the
second → the host has not rebooted since the delegation landed (the deployer prints
`NOTE: io is not delegated …` for exactly this). No file at the third → that unit
carries no weight.
Two limits that survive the fix: a container's weight covers **everything** in its
cgroup, so a library scan and a playback stream share one weight; and the comparison
is made *inside* `user.slice`, so a rootless unit's weight cannot push it below a
root unit (`frank-archon`). Shaping that pairing is the weight of `user.slice`
itself — a fleet decision, not a unit line.
## 5. Rootless Quadlet log retrieval ## 5. Rootless Quadlet log retrieval
Step 0 — establish ownership: `systemctl --user status <name>.service` Step 0 — establish ownership: `systemctl --user status <name>.service`
@@ -46,10 +46,11 @@ view really is a mountpoint.
| Installer | `scripts/media-view.sh` — `plan` / `install` / `verify` / `status` / `reap` | | Installer | `scripts/media-view.sh` — `plan` / `install` / `verify` / `status` / `reap` |
| Written by | Jellyfin only (Plex binds read-only); nothing else in the fleet touches it | | Written by | Jellyfin only (Plex binds read-only); nothing else in the fleet touches it |
Both consumers guard the view with Both consumers gate their start on the view with
`UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view`, and `ExecStartPre=/home/dank/serverAppConfigs/scripts/wait-for-mount.sh` (it waits
`scripts/deploy.sh dankserver --service jellyfin|plex` calls `media-view.sh for the mount, then refuses), and `scripts/deploy.sh dankserver --service
install` + `verify` before it restarts either container. jellyfin|plex` calls `media-view.sh install` + `verify` before it restarts
either container.
| Consumer | Bind | Mode | Why that path | | Consumer | Bind | Mode | Why that path |
|---|---|---|---| |---|---|---|---|
@@ -80,11 +81,20 @@ the defaults are the host values above.
refuse via `lsblk -dno ROTA` if the upper base lives on the array: an overlay refuse via `lsblk -dno ROTA` if the upper base lives on the array: an overlay
whose upper is the spinners is the original problem with extra steps, and it whose upper is the spinners is the original problem with extra steps, and it
silently turns every metadata write back into a seek. silently turns every metadata write back into a seek.
- **Never mount the view over an unmounted lower.** `RequiresMountsFor=/mnt/entropy` - **A container that binds the array must refuse to start until the mount
plus the container `ConditionPathIsMountPoint` guards: an empty `lowerdir` answers.** Live 2026-10-31: the array's mount and the media containers raced at
presents an *empty library* as truth, and a media scan would then clear the boot, podman resolved the bind against the pre-mount directory, and both
database of everything. systemd does not create a mount point, so `install` servers came up blind — Jellyfin read an empty library and began
creates `view` itself; without it the mount simply fails. `Removing item, Type: "Season"` by the thousand. Two dependencies people reach
for do not work here, both measured on that host: `RequiresMountsFor=` in a
**rootless** unit is inert (the user manager cannot order against the system's
`mnt-entropy.mount`), and `ConditionPathIsMountPoint=` is fail-*dead* — an
unmet condition logs `skipped, unmet condition check` with `NRestarts=0` and no
start attempt ever follows, so the unit waits for a human. The working guard is
`scripts/wait-for-mount.sh` as `ExecStartPre`: it waits out the race, refuses by
name after its window, and `Restart=always` carries the unit from there.
systemd does not create a mount point, so `install` creates `view` itself;
without it the mount simply fails.
- **Never delete through the view.** A delete writes an overlayfs whiteout - **Never delete through the view.** A delete writes an overlayfs whiteout
(`.wh.<name>`) in the upper layer, which hides the array's copy without (`.wh.<name>`) in the upper layer, which hides the array's copy without
removing it — the file reappears the moment the view is unmounted, and the removing it — the file reappears the moment the view is unmounted, and the
@@ -107,7 +117,8 @@ the defaults are the host values above.
| Symptom | Meaning | | Symptom | Meaning |
|---|---| |---|---|
| `jellyfin.service` / `plex.service` show `ConditionPathIsMountPoint=... was not met` and stay inactive | The view is not mounted. Do not start them anyway: `scripts/media-view.sh install`, then `verify`. | | `jellyfin.service` / `plex.service` fail with `refused: still not mounted after 90s: …` | The view or the array is not mounted. Do not start them anyway: `scripts/media-view.sh install`, then `verify`. |
| Servers answer, library looks empty, Jellyfin logs `Removing item, Type: "Season"` by the thousand | Blind bind — the container started before the mount and bound the pre-mount directory. `podman exec jellyfin ls /data` against `ls /mnt/entropy` is the whole diagnosis; restarting the container re-resolves the bind. |
| Jellyfin log: `Read-only file system : '/data/Movies/…/movie.nfo'` | The container is binding the raw array, not the view — the quadlet drifted back or the view is gone. `media-view.sh status`. | | Jellyfin log: `Read-only file system : '/data/Movies/…/movie.nfo'` | The container is binding the raw array, not the view — the quadlet drifted back or the view is gone. `media-view.sh status`. |
| A folder appears in a media server with artwork/`.nfo` but no video | Ghost metadata from a relayout. `media-view.sh reap`. | | A folder appears in a media server with artwork/`.nfo` but no video | Ghost metadata from a relayout. `media-view.sh reap`. |
| `verify` prints `refused: …/view is not a mountpoint` | A plain directory is sitting where the overlay belongs. That is the fail-closed guard working; `install` on the box. | | `verify` prints `refused: …/view is not a mountpoint` | A plain directory is sitting where the overlay belongs. That is the fail-closed guard working; `install` on the box. |
@@ -77,7 +77,7 @@ Both prerequisites survive the dnsmasq → Pi-hole swap. Do not remove them.
| Port 53 `bind: permission denied` crash loop after a reboot | sysctl conflict: the value must be 53 at boot, and remember `/etc/sysctl.conf` is applied **after** all of `sysctl.d/`. Redeploy pihole — `deploy_pihole` reinstalls the drop-in, neutralizes conflicting drop-ins, and comments out a stale `/etc/sysctl.conf` line | | Port 53 `bind: permission denied` crash loop after a reboot | sysctl conflict: the value must be 53 at boot, and remember `/etc/sysctl.conf` is applied **after** all of `sysctl.d/`. Redeploy pihole — `deploy_pihole` reinstalls the drop-in, neutralizes conflicting drop-ins, and comments out a stale `/etc/sysctl.conf` line |
| Single-label name (for example `dankclaw`) answers `127.0.1.1` | The rootless container `/etc/hosts` carries the host entry (`127.0.1.1 <host>`), and FTL serves it. Use the FQDN, or add an `/etc/hosts` entry on the client | | Single-label name (for example `dankclaw`) answers `127.0.1.1` | The rootless container `/etc/hosts` carries the host entry (`127.0.1.1 <host>`), and FTL serves it. Use the FQDN, or add an `/etc/hosts` entry on the client |
| Every `franksdank.cloud` name is NXDOMAIN for LAN clients while external names still answer | FTL holds no local records: `getent hosts <name>` on Dankclaw comes back empty → `podman exec pihole pihole restartdns reload` (the 2026-10-19 pitfall under **Records**) | | Every `franksdank.cloud` name is NXDOMAIN for LAN clients while external names still answer | FTL holds no local records: `getent hosts <name>` on Dankclaw comes back empty → `podman exec pihole pihole restartdns reload` (the 2026-10-19 pitfall under **Records**) |
| One client gets nothing while every other client is fine | FTL rate-limiting that client: `podman exec pihole grep Rate-limiting /var/log/pihole/FTL.log`. A `10.0.2.x` client address is a pasta container's DNS forwarder, not a machine | | One client gets nothing while every other client is fine | FTL rate-limiting that client: `podman exec pihole grep Rate-limiting /var/log/pihole/FTL.log`. A `10.0.2.x` client address is a pasta container's DNS forwarder, not a machine — and **every** rootless container on that host shares the one address, so a single chatty container throttles DNS for all of them (measured 2026-10-31: 8 throttle events in a day, all `10.0.2.100`) |
## See Also ## See Also
+96
View File
@@ -0,0 +1,96 @@
#!/usr/bin/env bash
# Seam: a container that binds the entropy array must refuse to start until the
# thing it binds is actually mounted.
#
# Live 2026-10-31, Dankserver: the array's systemd mount STARTED eight seconds
# before the media containers were created, but a rootless quadlet cannot order
# itself against a SYSTEM mount unit — `RequiresMountsFor=/mnt/entropy` resolves
# to nothing in the user manager and is simply inert. So podman resolved the bind
# against the pre-mount directory (which held one stale subdirectory) and both
# Jellyfin and Plex came up blind: their library root looked empty, Jellyfin
# logged thousands of `Removing item, Type: "Season"`, and the array sat at 80 %
# util doing nothing useful until the host went down. A blind bind is not a
# cosmetic fault — it is a metadata-deleting machine.
#
# The measured fix is `ExecStartPre=<repo>/scripts/wait-for-mount.sh <source>`:
# it WAITS for the mount and then refuses, so `Restart=always` carries the unit
# across a slow mount. A `ConditionPathIsMountPoint=` was measured on the same
# host and is the wrong tool: an unmet condition logs `skipped, unmet condition
# check` with NRestarts=0 and never a start attempt — the unit stays dead until
# a human starts it. That is fail-dead, not fail-closed.
#
# Tests assert only observable behavior of the gate (exit status and the name it
# refuses by) and the structural law over the quadlets; never the internals.
load helpers
GATE="$REPO_ROOT/scripts/wait-for-mount.sh"
@test "the gate accepts a path that really is a mountpoint" {
run "$GATE" -t 2 /proc
[ "$status" -eq 0 ] || { printf '%s\n' "$output" >&2; return 1; }
}
@test "a plain directory is refused, and refused BY NAME" {
dir="$BATS_TEST_TMPDIR/not-a-mount"
mkdir "$dir"
run "$GATE" -t 1 "$dir"
[ "$status" -ne 0 ] || { printf 'accepted a plain directory\n' >&2; return 1; }
[[ "$output" == *"$dir"* ]] || { printf 'refusal did not name the path: %s\n' "$output" >&2; return 1; }
}
@test "a path that does not exist is refused by name" {
run "$GATE" -t 1 "$BATS_TEST_TMPDIR/does/not/exist"
[ "$status" -ne 0 ]
[[ "$output" == *"$BATS_TEST_TMPDIR/does/not/exist"* ]]
}
@test "the gate holds the deadline instead of deciding on one look" {
# A single check that failed at the wrong instant is the boot race itself:
# the mount lands a second later and the unit is dead anyway. So the window
# must be spent waiting, and that is observable from outside as elapsed time.
dir="$BATS_TEST_TMPDIR/never"
mkdir "$dir"
start=$SECONDS
run "$GATE" -t 4 "$dir"
elapsed=$((SECONDS - start))
[ "$status" -ne 0 ]
[ "$elapsed" -ge 4 ] || { printf 'gave up after %ss, window was 4s\n' "$elapsed" >&2; return 1; }
}
@test "the gate accepts a mount that arrives during its window" {
dir="$BATS_TEST_TMPDIR/later"
mkdir "$dir"
if ! mount --bind /proc "$dir" 2>/dev/null; then
skip "creating a mount needs privileges this box does not grant"
fi
run "$GATE" -t 4 "$dir"
umount "$dir" 2>/dev/null || sudo -n umount "$dir" 2>/dev/null || true
[ "$status" -eq 0 ] || { printf 'gate did not wait for a mount that arrived: %s\n' "$output" >&2; return 1; }
}
@test "no unit without the gate binds the array or the metadata view" {
bad=""
while IFS= read -r unit; do
sources=$(sed -n 's/^Volume=\([^:]*\):.*/\1/p' "$unit" \
| grep -E '^(/mnt/entropy|/var/lib/media-meta/view)$' | sort -u)
[ -n "$sources" ] || continue
for src in $sources; do
grep -q "ExecStartPre=.*wait-for-mount.sh.* $src\b" "$unit" \
|| bad="$bad$unit binds $src with no start gate"$'\n'
done
done < <(find "$REPO_ROOT"/dankserver "$REPO_ROOT"/dankclaw "$REPO_ROOT"/frankbook-pro \
-path '*/quadlet/*.container' -not -path '*.AGENTTRASH*')
[ -z "$bad" ] || { printf 'blind binds:\n%s' "$bad" >&2; return 1; }
}
@test "no gated unit keeps the fail-dead ConditionPathIsMountPoint" {
bad=""
while IFS= read -r unit; do
grep -q "wait-for-mount.sh" "$unit" || continue
grep -q "ConditionPathIsMountPoint" "$unit" \
&& bad="$bad$unit: a gate and an unmet-condition skip cannot coexist"$'\n'
done < <(find "$REPO_ROOT"/dankserver "$REPO_ROOT"/dankclaw "$REPO_ROOT"/frankbook-pro \
-path '*/quadlet/*.container' -not -path '*.AGENTTRASH*')
[ -z "$bad" ] || { printf 'fail-dead conditions:\n%s' "$bad" >&2; return 1; }
}
+21
View File
@@ -106,3 +106,24 @@ load helpers
# env file, not an Environment= line. # env file, not an Environment= line.
! grep -qE '^Environment=CUSTOM_(USER|PASSWORD)=' "$unit" || return 1 ! grep -qE '^Environment=CUSTOM_(USER|PASSWORD)=' "$unit" || return 1
} }
@test "a quadlet that asks for an I/O share ships the host permission that makes it real" {
# IOWeight= is silently inert unless the system manager both enables io for
# user.slice and delegates it to the user manager: systemd drops a resource
# line it cannot honour without a word and the container still starts healthy.
# So an in-repo weight is only honest together with both drop-ins AND with a
# deployer that places them. Measured 2026-10-31; verification chain in
# Architecture/Quadlet Lifecycle 4b.
if grep -lq '^IOWeight=' "$REPO_ROOT"/dankserver/*/quadlet/*.container; then
asked=1
else
skip "no dankserver quadlet asks for an I/O share"
fi
grep -q '^IOAccounting=yes$' \
"$REPO_ROOT/dankserver/host/systemd/user.slice.d/io-accounting.conf" || return 1
grep -q '^DelegateControllers=.* io' \
"$REPO_ROOT/dankserver/host/systemd/user@1000.service.d/io-delegate.conf" || return 1
for f in user.slice.d/io-accounting.conf 'user@1000.service.d/io-delegate.conf'; do
grep -q "dankserver/host/systemd/$f" "$REPO_ROOT/scripts/deploy-remote.sh" || return 1
done
}
+4 -1
View File
@@ -61,7 +61,10 @@ UNIT="$REPO_ROOT/dankserver/host/systemd/media-meta-view.mount"
@test "both media servers bind the view and refuse to start without it" { @test "both media servers bind the view and refuse to start without it" {
for unit in "$JELLY" "$PLEX"; do for unit in "$JELLY" "$PLEX"; do
grep -q "^Volume=$MOUNT:" "$unit" || return 1 grep -q "^Volume=$MOUNT:" "$unit" || return 1
grep -q "^UnitConfigItem=ConditionPathIsMountPoint=$MOUNT$" "$unit" || return 1 # The refusal is a start GATE, not a Condition: measured on the host, a
# unit whose Condition goes unmet is skipped forever (NRestarts=0, no
# start attempt) — fail-dead. The gate waits, then refuses.
grep -q "^ExecStartPre=.*wait-for-mount.sh .*$MOUNT" "$unit" || return 1
# The raw array must no longer be a container source: that bind is what # The raw array must no longer be a container source: that bind is what
# made sidecar sharing impossible (read-only) and metadata slow (seeks). # made sidecar sharing impossible (read-only) and metadata slow (seeks).
! grep -q '^Volume=/mnt/entropy' "$unit" ! grep -q '^Volume=/mnt/entropy' "$unit"