Compare commits

..
Author SHA1 Message Date
dank 38737ab84e Merge pull request 'Media metadata view: Jellyfin writes, Plex reads, the array stays read-only' (#97) from feat/media-metadata-view into main 2026-10-30 22:18:27 -07:00
Frank 4437519430 chore(media): retire the legacy Jellyfin block from dankserver/compose.yaml
The container has had one owner since the Quadlet cutover
(dankserver/jellyfin/quadlet/jellyfin.container). The block was a second owner,
and wrong on every path: jellyfin/jellyfin:latest instead of the pinned ghcr
image, /opt/jellyfin/* data roots that no longer exist, and a read-only
/mnt/entropy:/data bind that the metadata view has since replaced.

The planner no longer counts jellyfin as a compose service, so editing the
inventory cannot restart a media server; tests/deployment_scoping.bats pins that
contract for both jellyfin and plex.
2026-10-30 22:17:04 -07:00
Frank 3b29e173c2 refactor(media): mount the view inside /var/lib/media-meta instead of under /mnt
The merged door was /mnt/media, which read like a third volume and split the host
state across two trees. It is now /var/lib/media-meta/view, so the array layer,
the SSD upper layer, the kernel workdir, the reap trash and the merged tree are
one directory — and the mount table gains no new /mnt entry beside the array.

systemd names a mount unit after its ESCAPED path, so the repo copy keeps the
readable name media-meta-view.mount while `install` writes the host file as
var-lib-media\x2dmeta-view.mount; `plan` prints both and the page records them, so
nobody hunts for a unit that "does not exist". install also creates the mount
point itself, because systemd will not.

The seam now additionally pins that the unit's Where= is exactly the path both
containers bind — guard and bind cannot drift apart. The Quadlet Lifecycle
phantom-path example is corrected too: it named an invented /mnt/media/*, while
the real phantom is the legacy /opt/jellyfin/* that dankserver/compose.yaml still
carries for a service whose roots moved to /home/dank/Containers/jellyfin.
2026-10-30 22:05:31 -07:00
Frank 9706f5d2de docs(media): the view moves sidecars into Plex's reach, it does not enable its NFO agent 2026-10-30 21:42:30 -07:00
Frank 206dc3baf1 docs(media): one page for the metadata view, and the three pages it re-points
New owner page Dankserver/Media Metadata View.md: the shape, the two binds and
why Plex keeps the /mnt/entropy container path, the commands, the laws (SSD-only
upper layer, never mount over an unmounted array, never delete through the view
because a whiteout only hides the array's copy, reap after any relayout, inotify
does not cross the merge), the failure signatures, and the rollback.

Jellyfin.md: /data is the view, so Local metadata works; the
"Read-only file system : '/data/…/movie.nfo'" line that used to be expected noise
is now the signature of a missing view. Plex.md: same bind, one fact changed.
Media Relayout.md: two new laws for the metadata that does not follow a rename.
AGENTS.md: runbook-index row plus a summary row. Root index regenerated.
2026-10-30 21:41:56 -07:00
Frank f5a3badb56 feat(media): the relayout executor names the reap its moves leave behind
A relayout renames on the raw array; the sidecars the media servers wrote live in
the NVMe upper layer of the view and stay at the old coordinates, which is how a
ghost folder (artwork and an .nfo, no video) is born. apply and rollback now
print the follow-up — scripts/media-view.sh reap — at the moment the operator
needs it, and the existing apply seam asserts the hint is in the output.
2026-10-30 21:41:56 -07:00
Frank 3e4b4f24f7 feat(media): share one metadata set between Jellyfin and Plex over an SSD overlay
Both media servers read a Kodi-format sidecar (movie.nfo / tvshow.nfo /
episode.nfo) only when it sits next to the media file, and the library lives on
the RAID6 array, deliberately mounted read-only for every container and slow to
seek for a thousand tiny XML reads. So neither server binds the array any more:
both bind /mnt/media, an OverlayFS merge of the array (lowerdir, never written)
with /var/lib/media-meta on the NVMe (upperdir, where every write lands).

Jellyfin binds the view rw at /data and is the fleet's only metadata writer;
Plex binds it ro at /mnt/entropy, the container path its own database already
recorded, so only the host side of that bind moved and its library paths keep
resolving.

scripts/media-view.sh plans, installs, verifies, reports and reaps the view. It
refuses to mount over an unmounted array, refuses an upper layer on rotational
storage, and refuses a non-root install. deploy-remote.sh raises the view before
it restarts either media server, both units carry
ConditionPathIsMountPoint=/mnt/media so a missing view skips the start instead of
letting a scan clear the library database, and the planner routes the installer
and dankserver/host/** to the jellyfin+plex pair.

tests/media_overlay.bats (11 seams) pins the resolved shape, both refusals, the
two binds and their guard, the deployer's ordering, the planner mapping, and the
reap behaviour against a fixture (ghosts listed, then moved to the SSD trash --
files that exist only above are sidecars, not ghosts, so only directories are
candidates).
2026-10-30 21:41:51 -07:00
Frank 86f5903e74 docs(media): how to file a below-threshold episode by hand
Six single-episode files stayed in the movie library when the threshold refused
them; the hand path is the same executor on a hand-written plan and the same
journal. The trap recorded here is the series year -- never inherit it from the
title whose bonus folder an episode happens to be parked in.
2026-10-30 20:45:24 -07:00
Frank bd4ee78811 docs(media): clear the two OKF violations the last push shipped
The seam ran through `grep -c`, which returned success on a non-zero count and
let a red suite through: description over the 120-char cap, and a timestamp
that check-okf reads against the local date, so a UTC stamp a day ahead of
Seattle is 'in the future'. index.md regenerated for the new description.
docs_completeness 6/6, exit status checked this time.
2026-10-30 20:40:27 -07:00
Frank b7b39cdc92 docs(media): Jellyfin 12.4 needs the MediaBrowser header; the scan key is on the box
X-Emby-Token is refused for every key on 12.4 -- which is why this page
carried the false claim that no credential on the box could trigger a scan and
that the two Seerr keys were dead. The documented snippet now authenticates
with Authorization: MediaBrowser Token=..., reads 204 from /Library/Refresh,
and points at ~/.config/secrets/jellyfin (mirrored as JELLYFIN_API_KEY in the
ignored dankserver/.env). Law: prove a 401 against the ApiKeys table before
blaming the key.
2026-10-30 20:39:51 -07:00
Frank 09532e5216 docs(media): a deletion is staged on the item's own filesystem
Trashing 31 GiB of duplicate Monk to ~/.AGENTTRASH would have been a
cross-device copy of the thing being deleted, into 187 GiB of free space, on a
volume at 100% -- and rename(2) refuses it regardless. The stage belongs beside
the item.
2026-10-30 20:37:27 -07:00
Frank e1786e7cf2 docs(media): the Monk duplicate is verified; compare libraries with the series engine
The destination-hold refusal's evidence, stated exactly: 124 episodes both
sides, same episode list, every byte count identical, 31.5 GiB each way,
head+tail blake2b identical on a 6-episode sample.

Plus the law this came out of: a path-substring filter for a series name also
returns other shows (monk matches Monkey Dust), and whichever file a walk
reaches first standing in for an episode turns two libraries into a fake
duplicate — the same sloppiness aimed the other way deletes a unique file.
ep_of on both sides, and a hash before anything is trashed.
2026-10-30 20:33:41 -07:00
dank 409a8d8b0c Merge pull request 'feat(media): a TV mover that will not file a show the library already has' (#96) from feat/media-tv-mover into main 2026-10-30 20:31:38 -07:00
Frank e83162a2ac feat(media): a TV mover that will not file a show the library already has
Two fixes found by one live sweep, shipped together because the second is
what makes the first finishable.

1. scripts/media-tv.py — the other half of media-plan.py's refusal to file
   TV. Groups episode-shaped files by the series name before the marker
   (media-titles.ep_of, imported not restated), writes a plan in the
   executor's own format: <library>/<Series> (<year>)/Season <NN>/<own
   name>.<ext>. Refuses a group below --min, an episode naming no season,
   and an episode the destination library already holds at the same byte
   count — the movie share carries a second copy of all 124 Monk episodes
   the TV library has had for years; a duplicate is Frank's call, never a
   tool's copy. Applied by media-apply.py with its own journal and
   --root /mnt/entropy.

2. scripts/media-apply.py — gone meant "rmdir intended", so a folder whose
   rmdir failed was blacklisted forever and no later sweep could finish it
   (live movie sweep: 4517 targeted, 146 permanently stranded). gone now
   means "actually gone": a failed intent is retried by the next sweep.

tests: 9 new tests in media_relayout.bats, each red against the previous
revision — 8 TV + 1 sweep retry behind a chmod 500 parent. ./tests/run.sh
633 green. No live move run yet.
2026-10-30 20:30:24 -07:00
dank 5cd9a0c4c2 Merge pull request 'fix(media): the series is the name before the marker; a sweep carries its own empty subfolders' (#95) from fix/media-series-key into main 2026-10-30 20:20:24 -07:00
Frank a803aa046a fix(media): a sweep carries the emptied folder's own empty subfolders
The live sweep removed 2271 husks and left 146 standing, every one a release
folder blocked by a junk subfolder (Screens/, Sample/, a BDMV leaf) that no
file ever moved out of. Eligibility was per-folder, so the blocker was never a
target and its parent could never be removed.

Proof now carries downward: inside a folder the journal shows emptied, every
empty directory qualifies. Nothing under such a folder can hold data, and every
removal is still journalled as rmdir and undone by rollback.

The new test is red against the previous revision, green now. The live sweep
touched no file: 8193 before, 8193 after.
2026-10-30 20:18:54 -07:00
Frank 73f77a13c3 fix(media): a series is the name before the episode marker
Keyed on the whole episode title, every episode of a box set became a group of
one and never reached the five-episode size that calls something a series, so
the live run filed 124 Monk episodes as 124 movie folders, each named after its
own episode. ep_of now returns the series — the name before SxxEyy / 1x04 /
Part N — and the planner groups and names the quarantine pile by that series,
so one show is one pile: 'Monk - S05E03 - Mr. Monk and the Big Game' is series
monk. The threshold keeps its job: a lone film named Part 2 stays a film.

Re-planning the live census with this moves nothing differently except the 124
Monk files, which now join the TV pile: 168 episode files across 3 series.
2026-10-30 20:12:05 -07:00
dank dd76fcf5d4 Merge pull request 'feat(media): sweep removes the folders a completed run emptied' (#94) from feat/media-sweep into main 2026-10-30 20:04:24 -07:00
Frank 1cb28095a7 feat(media): sweep removes the folders a completed run emptied
An apply leaves every release folder it emptied standing where it was: the
live library run left 1675 husks at the root. sweep takes the same journal
and deletes only folders it can prove this run emptied - a journalled move
out of the folder or something under it, nothing but empty directories left,
no journalled destination beneath it, not a folder the run created. A folder
nobody moved cannot be selected at all, so a hand-made placeholder survives,
and a symlink anywhere inside disqualifies the subtree. Every rmdir is
journalled, so rollback recreates what a sweep removed.
2026-10-30 20:03:16 -07:00
dank e1760e3316 Merge pull request 'feat(media): the movie-library relayout toolchain — census, plan, executor' (#93) from feat/media-relayout into main 2026-10-30 19:38:03 -07:00
dank 272e06daf6 feat(media): the movie-library relayout toolchain — census, plan, executor
The movie library on the entropy volume is release-folded: the video inside
`Some.Movie.2019.1080p…-FOO/` is named after the release, and Jellyfin matches
a movie by folder and file name, so the noise cost posters, extras and versions.
Three scripts own the fix, and nothing else in the repo moves a media file.

  * media-census.py  the read-only inputs: a stat pass, an ffprobe header pass
                     (nice'd, so it is safe beside a live endpoint), and the
                     Plex metadata cache read that outranks any filename guess
  * media-plan.py    the Jellyfin-format planner: one folder per movie, the
                     video named exactly like the folder, extras in the
                     canonical subfolders, copies of one film as versions. It
                     opens no media file, so a whole-library plan costs seconds,
                     and it ends in gates — bad-names, occupied, double-claim,
                     left-unmoved, unplanned — that must all read zero before a
                     human points the executor at 24 TB
  * media-apply.py   the executor: rename(2) only, the intent fsynced to a
                     write-ahead journal BEFORE the move it authorises, so an
                     interrupted run is finishable and every run is undoable.
                     Refusals are all-or-nothing and it never overwrites a file

Laws the fixture pins (tests/media_relayout.bats, 16 tests, each proven red
against a reverted copy of the rule it guards):

  * a Plex year of 0 means "indexed, never matched" — trusting it produced
    folders "Title (0)": items with no year, no poster, no identity
  * a Plex title carrying "/" is sanitised, or "V/H/S/2" became a five-level
    tree neither matcher reads as a movie
  * a folder inside a movie's folder is bonus territory even when its name is
    misspelled ("Exrtas"), unless the file's own name carries another year
  * matching words split on everything but a letter or digit — the earlier
    tokenizer left "(1960)/extras" glued, so a film could not be found by the
    folder holding it and one box set's featurettes drifted into a neighbour
  * a name with no words in it is bonus material, never a movie folder named 1

Docs: the runbook page owns the procedure, the gates and the rollback; AGENTS.md
links it from the runbook index and the summary table.
2026-10-30 19:33:44 -07:00
dank 4db985b90d Merge pull request 'fix(qwen38): the preflight asks whether the pin is ONE of an image's digests' (#92) from fix/quant-pipeline-hardening into main 2026-10-30 16:30:07 -07:00
Frank 44d72b9e78 fix(qwen38): the preflight asks whether the pin is ONE of an image's digests
A by-digest pull leaves the image carrying every digest it was pulled under, so
reading RepoDigests[0] reported a healthy pinned image as wrong and refused a
window that was ready to run. Membership, not position. Also records the measured
step optimum: 106 tok/s at an 8,192-token step against 76 at 16,384 on a 32,768
calibration window, so the wide step has a ceiling, not a plateau.
2026-10-30 16:30:03 -07:00
dank 8073d0aa9d Merge pull request 'fix(qwen38): one identity preflight owns every fact a calibration window needs' (#91) from fix/quant-pipeline-hardening into main
Merge pull request #91
2026-10-30 16:26:41 -07:00
Frank 5c678b804c fix(qwen38): one identity preflight owns every fact a calibration window needs
Each gate the window used was a proxy, and each lied at least once:
- /health answered 200 from the dying production process (window dead at second zero)
- podman inspect named a rig container that had bound nothing
- the space check asked a directory that merely looked related
- the identity check compared podman's CONTAINER-internal pids (1, 7) to an ss host pid

scripts/qwen38-preflight.sh now checks cwd, the digest a tag actually resolves to,
free space on the tap's own filesystem, stale grams, every /dev/kfd holder's host pid
(podman top <ctr> hpid) and the :7770 listener's host pid against the service's, the
corpus and the reference recording; it prints PASS/FAIL per fact and exits 2. The
window calls it before a card is touched, and its duplicated checks are gone.
2026-10-30 16:26:07 -07:00
dank 81e0995f7d Merge pull request 'fix(qwen38): the calibration window must prove the rig owns the socket' (#90) from fix/quant-pipeline-hardening into main 2026-10-30 16:08:29 -07:00
Frank 167e6d6de3 fix(qwen38): the calibration window must prove the rig owns the socket
The runner stops frank-archon first, and the dying production process keeps its
listener for a moment while the freshly created rig container is still loading.
'podman inspect says running' plus '/health answers' therefore BOTH hold for the
wrong process: measured 2026-10-30, the window logged 'health ok waited=0s',
the pusher refused on the container name, and a 12M window died at second zero.

The gate now asks the kernel who owns :7770 (ss -ltnp) and requires that pid to
be one of the rig container's processes (podman top). Two tests, written red
first against the old rig_up: a foreign listener must fail the window without a
push, and an in-container listener must proceed and bank.

Also renames the new test fixture to tap_window_fixture: it had shadowed the
existing window_fixture helper, which broke the window dry-run test when the
whole file ran.
2026-10-30 15:54:57 -07:00
dank 608b0058e3 Merge pull request 'campaign(qwen38): the calibration floor is 12M tokens' (#89) from fix/quant-pipeline-hardening into main 2026-10-30 15:47:25 -07:00
Frank 4043798053 Merge remote-tracking branch 'origin/main' into fix/quant-pipeline-hardening 2026-10-30 15:47:15 -07:00
Frank 521537a34c Merge branch 'chore/fleet-image-refresh' into main 2026-10-30 15:34:11 -07:00
Frank 19fdffe8bd docs(agents): a start-only quadlet handler no-ops an image bump
The law today's authentik bump paid for: start on an active unit is a no-op, so
the new digest installs and the old container keeps serving behind three green
gates. Restart applies a pin.
2026-10-30 15:34:10 -07:00
Frank 54a4a4180a Merge branch 'chore/fleet-image-refresh' into main 2026-10-30 15:34:03 -07:00
Frank f7aec7f154 fix(authentik): restart the units so an image bump actually applies
The handler called `systemctl --user start` on all three units. Start is a
no-op for an already-active unit, so installing a new digest under an active
authentik left the old container serving: today's 2026.8.3 -> 2026.8.5 bump
moved only the OUTPOST (whose handler restarts) and left server + worker on
2026.8.3 while the deploy, the health gate and the e2e suite were all green.
The generator re-renders from the .container, so a restart is the step that
applies a pin.
2026-10-30 15:34:02 -07:00
Frank 299078eaba campaign(qwen38): the calibration floor is 12M tokens
Frank's rule: the served iq4r this arm competes with was calibrated on a
10M-token set, so the arm is calibrated at least as deeply. The floor was
lowered to 1.5M on the vendor's '10M beats 1M by under 1%' note; that
measures the vendor's own plate, not ours — the arm is judged against a
10M-calibrated reference, so cutting the calibration 6x to save wall clock
buys a worse position on the only comparison that matters.

One round of the re-chunked corpus is ~9.31M tokens, so a 12M floor seals
in round 1 plus part of round 2. A round that seals nothing is normal (the
floor is pooled) and round_verdict treats it as continue, never failure.

Also fixes tests/qwen38_campaign.bats' window dry run: its fixture lands on
/tmp (a 16 GiB tmpfs) where the 60 GiB seal preflight can never pass, so the
test now uses the documented Q38_CALIB_MIN_FREE_MIB override instead of
asserting environment luck.
2026-10-30 15:33:52 -07:00
dank 7b2b5b66fe Merge pull request 'fix(qwen38): the calibration tap runs on latest radiance, not the production pin' (#87) from fix/quant-pipeline-hardening into main 2026-10-30 15:30:21 -07:00
Frank e1d4d59773 docs(dragoon,zealot): record both runtime pins as they now run
frank-dragoon: engine 0.18.20 (image e35133b2…), the 18-model NPU catalog, the
re-cut qwen3-embedding-0.6b device programs, and the wire surface the new engine
actually answers (GET /tokenize is 405, /props is 200, /completion and /ping
stay 404). Eras gains the 2026-10-30 row.

frank-zealot: fork build 12118 (0.7.0-dev, 024474448) with the payload-load
evidence (202 pq2_0 tensors, 202 Hadamard-folded weights), plus the law a
rebuild pays for: cmake --install strips the rpath and installs bin/+lib/, so
the configure needs CMAKE_INSTALL_RPATH and LLAMA_BIN names …/bin/llama-server.
2026-10-30 15:27:21 -07:00
Frank 5787c86954 Merge branch 'chore/fleet-image-refresh' into main 2026-10-30 15:27:21 -07:00
Frank acbb539b97 feat(frank-zealot): PrismML fork b10777 -> build 12118 (0.7.0-dev, 024474448)
Rebuilt the fork from origin/prism at 024474448 with the same configure the
10777 cutover used (Release, GGML_CUDA=ON, sm_86, gcc-13 host compiler, Ninja),
installed to /home/dank/GitPackages/llama-prism-0244744. The payload gate holds:
libllama carries 42 prism.hadamard strings (b10777 had 29) and libggml-base
still registers PQ2_0, so type 142 and the hadamard metadata still load.

LLAMA_BIN inherits a bin/lib install layout now, so the install needs an RPATH
($ORIGIN;$ORIGIN/../lib) — cmake's install step strips the build rpath and the
binary otherwise dies on libllama-server-impl.so.

The unit test that pinned the 10777 path now asserts the invariant its own
comment described (LLAMA_BIN names a llama-prism-* build) instead of a
directory one rebuild happens to install; the suite runs on a box with no
/home/dank, so existence is not checkable there.

Rollback: repoint LLAMA_BIN at llama-prism-b10777/llama-server (still on disk).
2026-10-30 15:25:24 -07:00
Frank 49a83ef1d5 Merge branch 'chore/fleet-image-refresh' into main 2026-10-30 15:25:24 -07:00
Frank d012654fc9 Merge remote-tracking branch 'origin/main' into fix/quant-pipeline-hardening 2026-10-30 15:23:06 -07:00
dank b45771e87e Merge pull request 'docs(radiance): reference the engine — options, plugins, conversion, tools, operations' (#88) from docs/radiance-engine-reference into main 2026-10-30 15:17:58 -07:00
Frank 011ceeb344 fix(npu fetch): never resume into a file from another revision
A version bump re-cuts the device programs, so a staged file of a different
size is stale rather than partial. `curl -C -` against it appended the new
tail to the old bytes and produced the right BYTE COUNT with the wrong
content — the sha256 gate caught it (consts.hnpb, 2026-10-30), and the
failure reads as a corrupt program rather than a fetch fault. Resume only
when the staged file is smaller than the manifest expects.
2026-10-30 15:13:42 -07:00
Frank 88a9bc02be Merge branch 'chore/fleet-image-refresh' into main 2026-10-30 15:13:42 -07:00
Frank 6ac544a7b6 Merge branch 'chore/fleet-image-refresh' into main 2026-10-30 15:13:23 -07:00
Frank 33396d2daa feat(frank-dragoon): halogen-flash-server 0.16.2 -> 0.18.20, NPU lane re-pinned
The engine pin moves to the newest vendor release (0.18.20, image built
2026-10-30). The payload lock is the gate this deploy has to pass: the v2
.hgn loads only in the flash-server line, so /health plus the boot ledger
decide, and the rollback is the 0c61bf84 digest on the same env line.

0.18.20's NPU catalog is NOT the 0.16.2 one: it names 14 device programs
(u0..u13) and an 8,495,512 B consts.hnpb where 0.16.2 had 10 and 5,086,496 B,
and the entrypoint verifies size+sha256 for every staged file before it
starts. HALOGEN_NPU_MODELS names qwen3-embedding-0.6b, so the staging set is
re-pinned to revision 8fd44c9f with the new block — bumping the engine alone
would refuse the boot on the NPU lane.
2026-10-30 15:13:19 -07:00
Frank 3e32cb9cd1 campaign(qwen38): the tap producer is latest radiance, at a vendor-backed floor
The tap did not arm on the production pin: a 17-hour calib run on 1.8.1
wrote zero bytes while /health, /stats and the push all read healthy.
On :latest (v1.14.0, revision 4a61d7296bc5) the same boot prints
'recorded passes: off, the program accumulates calibration Grams
(gram_accum)' and a 9,129-token push sealed 24,768 gram files (42.9 GB)
in the fitting shape - 4 slots at a 2,048-token step, production's own.

The vendor's calibration shape (--max-num-batched-tokens 65536,
--max-num-seqs 64, ~930 tok/s) cannot boot two 32 GiB R9700s: the
activation arena alone is 12.83 GiB of the 27.95 GiB claimable, and
every variant stops in the VRAM resolver (short 3.19 GiB .. 86 MiB,
then a starved cache).

Floor moves 12M -> 1.5M tokens: the vendor's own measurement is 'a
pooled Gram from 10M tokens beat one from 1M by under 1%', and 1.5M is
hours at this rig's rate instead of the 29 h a 6M floor cost.

The converter moves to the same digest: rad-convert's gptq:calib=<dir>
input is version-stable, so a latest-written tap converts on latest.
2026-10-30 15:02:58 -07:00
Frank b4a1e6069e fix(qwen38): pin calib to 1.2.0, gate the tap's disk, kill orphaned engines
A 17-hour calibration tap on 2026-10-29/30 wrote ZERO bytes: the runner booted
FRANK_ARCHON_IMAGE (1.8.1) for every mode, and RADIANCE_CALIB_DIR/TOKENS only
arms the expert tap on the 1.2.x line. 1.8.1 still carries the plugin strings and
the "calibration wrote layer Hessians" message, /health answered, /stats showed
1,700 documents and 118 tok/s, and nothing anywhere said the tap was dead.

Four gates, each with the measurement that motivated it:

- CALIB_IMAGE pins the 1.2.0 digest for calib mode (stilldeadcode/radiance
  @sha256:665775ed…), overridable only by naming --image deliberately. Proven by
  the engagement test: `calibration wrote layer Hessians after 10207 tokens`.
- The tap dir's filesystem must have >= 60 GiB free (Q38_CALIB_MIN_FREE_MIB).
  One seal wrote 39.4 GiB in seconds; pointed at a 37 GiB root it filled the
  filesystem to zero and every later write came back short.
- restore() kills orphaned `radiance` processes still holding /dev/kfd before it
  starts the service. `podman rm -f` does not kill the engine: an orphan made
  production's ar_oneshot_2rank_wht6 init fail `device error`, and the unit
  restarted 100 times with /health never answering.
- FRANK_QWEN38_CALIB_DIR / Q38_WINDOW_TRASH_ROOT make the tap dir and the bank
  target configurable, so the tap lives on /mnt/entropy instead of the root fs.
2026-10-30 14:18:03 -07:00
41 changed files with 4518 additions and 137 deletions

No files matched your search

+19 -8
View File
@@ -26,8 +26,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Payload | Port | | Target | Service | Runtime | Payload | Port |
|--------|---------|---------|---------|------| |--------|---------|---------|---------|------|
| Dankserver | `frank-archon` | radiance, digest-pinned, root unit, TP=2 | publisher `qwen3.8-next-flash-fp8-iq4r-moe.rad` | 7770 | | Dankserver | `frank-archon` | radiance, digest-pinned, root unit, TP=2 | publisher `qwen3.8-next-flash-fp8-iq4r-moe.rad` | 7770 |
| Dankclaw | `frank-dragoon` | halogen-flash-server 0.16.2, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 | | Dankclaw | `frank-dragoon` | halogen-flash-server 0.18.20, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 |
| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 10777, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj + rank-1 refusal adapter | 7771 | | Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 12118, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj + rank-1 refusal adapter | 7771 |
Payload identity, shape, eras, laws and rollback live in ONE page each: Payload identity, shape, eras, laws and rollback live in ONE page each:
[[frank-archon]], [[frank-dragoon]], [[frank-zealot]]. `scripts/<service>.env` [[frank-archon]], [[frank-dragoon]], [[frank-zealot]]. `scripts/<service>.env`
@@ -293,8 +293,8 @@ means off or NIC dead; a merely wedged kernel usually still answers ARP.
2. Bench against the service itself. No scratch ports on an LLM host — one 2. Bench against the service itself. No scratch ports on an LLM host — one
process owns the card. Use the runtime's own harness (`llama-bench`; a process owns the card. Use the runtime's own harness (`llama-bench`; a
live llama.cpp server also reports honest `timings` per request, including live llama.cpp server also reports honest `timings` per request, including
`draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork build `draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork that object
10777 that object only reaches you on **`/v1/chat/completions`**: a greedy only reaches you on **`/v1/chat/completions`** (measured on build 10777): a greedy
`ignore_eos` run on the raw `/completion` path there answers **500 "The `ignore_eos` run on the raw `/completion` path there answers **500 "The
model produced output that does not match the expected Content-only model produced output that does not match the expected Content-only
format"** (measured 2026-10-18). halogen has no `/completion` path at all, format"** (measured 2026-10-18). halogen has no `/completion` path at all,
@@ -439,16 +439,16 @@ disagrees with it is the bug.
### frank-zealot — llama.cpp (PrismML fork), RTX 3090, :7771 ### frank-zealot — llama.cpp (PrismML fork), RTX 3090, :7771
- Owns: `scripts/frank-zealot.env`, `scripts/serve-frank-zealot.sh`, - Owns: `scripts/frank-zealot.env`, `scripts/serve-frank-zealot.sh`,
`scripts/frank-zealot.service`; fork build `llama-prism-b10777` is the ONLY `scripts/frank-zealot.service`; fork build `llama-prism-0244744` (build 12118)
loader (the payload carries the fork's custom PQ2_0 tensor type plus is the ONLY loader — `llama-prism-b10777` stays on disk as its rollback (the payload carries the fork's custom PQ2_0 tensor type plus
`prism.hadamard.*` dequant metadata — stock llama.cpp refuses it, and a `prism.hadamard.*` dequant metadata — stock llama.cpp refuses it, and a
loader ignoring that metadata dequantizes wrong); payload loader ignoring that metadata dequantizes wrong); payload
`Bonsai2-9B-MTP/…PQ2_0.gguf` + the publisher's **Q8_0** mmproj (the BF16 `Bonsai2-9B-MTP/…PQ2_0.gguf` + the publisher's **Q8_0** mmproj (the BF16
tower can crash the server with concurrent image requests) + the rank-1 tower can crash the server with concurrent image requests) + the rank-1
refusal-direction adapter applied at load by `--lora-scaled …:1`. The refusal-direction adapter applied at load by `--lora-scaled …:1`. The
adapter's rollback is `LORA_FILE=` (empty) in the env — one line, no rebuild adapter's rollback is `LORA_FILE=` (empty) in the env — one line, no rebuild
— and build 10777 is also the gate that lets it load: older forks refuse it — and the fork BUILD is also the gate that lets it load: anything older than
with `unsupported prism.hadamard.version: 2`. b10777 refuses it with `unsupported prism.hadamard.version: 2`.
- Shape: `--ctx-size 602112 --parallel 3` = **three conversations at - Shape: `--ctx-size 602112 --parallel 3` = **three conversations at
200,704 each**, `CACHE_TYPE_K=f16` + `CACHE_TYPE_V=q8_0` (asymmetric: 25,088 200,704 each**, `CACHE_TYPE_K=f16` + `CACHE_TYPE_V=q8_0` (asymmetric: 25,088
B/token, 14,406 MiB of pool), batch 6144 / ubatch 1024, B/token, 14,406 MiB of pool), batch 6144 / ubatch 1024,
@@ -741,6 +741,13 @@ linked runbooks.
`/etc/default/grub`. It replaced `amd_iommu=off`, and the local A/B on byte-identical traffic `/etc/default/grub`. It replaced `amd_iommu=off`, and the local A/B on byte-identical traffic
says identity mode costs nothing, so the XDNA2 NPU returned for free. says identity mode costs nothing, so the XDNA2 NPU returned for free.
[[dankclaw-iommu-translation-off]] [[dankclaw-iommu-translation-off]]
- **A deploy handler that only `start`s a quadlet no-ops an image bump.**
`systemctl --user start` on an already-active unit does nothing, so installing
a new digest under a running service keeps the OLD container serving while the
deploy, its health gate and the Playwright suite all pass — the authentik
handler did exactly that on 2026-10-30 (2026.8.3 → 2026.8.5 moved only the
outpost, the one handler that restarted). A pin change is applied by
`restart`, never `start`.
- **Rootless Quadlet containers run with `--replace --rm`**: `podman logs` covers only the CURRENT - **Rootless Quadlet containers run with `--replace --rm`**: `podman logs` covers only the CURRENT
instance; history across recreations lives in `journalctl --user -u <name>.service`. Read instance; history across recreations lives in `journalctl --user -u <name>.service`. Read
rootless storage WITHOUT sudo — it is per-user storage, so `sudo podman logs` finds nothing. rootless storage WITHOUT sudo — it is per-user storage, so `sudo podman logs` finds nothing.
@@ -800,6 +807,8 @@ per-attempt page and no per-swap ledger anywhere else.
| `frank-tts` — cloned-voice speech, Dankclaw | `serverAppConfigs-docs/Dankclaw/Frank TTS.md` | | `frank-tts` — cloned-voice speech, Dankclaw | `serverAppConfigs-docs/Dankclaw/Frank TTS.md` |
| Dankclaw kernel posture (`iommu=pt` since 2026-10-18) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Dankclaw kernel posture (`iommu=pt` since 2026-10-18) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` | | Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` |
| Movie-library relayout (Jellyfin format, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Relayout.md` |
| SSD-backed metadata view (Jellyfin writes, Plex reads, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Metadata View.md` |
There is no page for a retired endpoint and no change diary. A payload swap There is no page for a retired endpoint and no change diary. A payload swap
appends one row to the live endpoint's **Eras** table; when the endpoint itself appends one row to the live endpoint's **Eras** table; when the endpoint itself
@@ -841,5 +850,7 @@ history.
| Writing a doc | **One owner per fact** — edit the endpoint/service page; everywhere else links. Never restate a number. | | Writing a doc | **One owner per fact** — edit the endpoint/service page; everywhere else links. Never restate a number. |
| Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. | | Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. |
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. | | Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
| Relayout the movie library | **Census → plan → read the exceptions → `scripts/media-apply.py`** — renames only, journal before every move; never hand-rename, never copy. |
| Making media metadata visible to both media servers | **Bind `/var/lib/media-meta/view`** — the overlay `scripts/media-view.sh` installs (array below, NVMe above). Never write the array; never delete through the view. |
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. | | Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. | | Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |
+6 -28
View File
@@ -94,34 +94,12 @@ services:
timeout: 10s timeout: 10s
retries: 3 retries: 3
# ═══════════════════════════════════════════════ # Jellyfin left this file when it became a Quadlet: the container is owned by
# Jellyfin — media server (movies, TV, music) # dankserver/jellyfin/quadlet/jellyfin.container, its data roots are under
# ═══════════════════════════════════════════════ # /home/dank/Containers/jellyfin, and its library arrives through the SSD
jellyfin: # overlay at /var/lib/media-meta/view (scripts/media-view.sh). The block that
image: jellyfin/jellyfin:latest # was here named /opt/jellyfin/* and a read-only array bind — a second owner for
container_name: jellyfin # the container name `jellyfin`, wrong on every path. One owner per container.
restart: unless-stopped
ports:
- "8096:8096/tcp"
- "8920:8920/tcp"
- "7359:7359/udp"
- "1900:1900/udp"
environment:
- JELLYFIN_PublishedServerUrl=http://192.168.0.235:8096
- DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true
volumes:
- /opt/jellyfin/config:/config
- /opt/jellyfin/cache:/cache
- /mnt/entropy:/data:ro
devices:
- /dev/dri:/dev/dri
group_add:
- "992"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8096/health"]
interval: 30s
timeout: 10s
retries: 3
# ═══════════════════════════════════════════════ # ═══════════════════════════════════════════════
# Calibre-Web — e-book web frontend # Calibre-Web — e-book web frontend
@@ -0,0 +1,30 @@
# /etc/systemd/system/var-lib-media\x2dmeta-view.mount
# (repo copy: dankserver/host/systemd/media-meta-view.mount — systemd names a
# mount unit after its ESCAPED path, so the host file carries the \x2d spelling)
#
# Installed by `scripts/media-view.sh install` — the repo copy is the source of
# truth (fleet law: never edit host state by hand).
#
# An OverlayFS merge that shows the media library as one tree while splitting
# where its bytes live: lowerdir is the RAID6 array (the source of truth, never
# written), upperdir is a directory on the NVMe that receives every write. The
# two media servers bind the result at /var/lib/media-meta/view, so Jellyfin's
# sidecars and artwork are SSD bytes — which is what makes them fast to browse
# and what keeps Plex's NFO agent off the spinners — while the media itself stays
# on the array untouched.
#
# RequiresMountsFor is load-bearing: without the array mounted, lowerdir is an
# empty directory and the view would present an empty library as truth.
[Unit]
Description=SSD-backed overlay view of the media library (array + NVMe metadata layer)
RequiresMountsFor=/mnt/entropy
[Mount]
What=overlay
Where=/var/lib/media-meta/view
Type=overlay
Options=lowerdir=/mnt/entropy,upperdir=/var/lib/media-meta/upper,workdir=/var/lib/media-meta/work
[Install]
WantedBy=local-fs.target
@@ -4,8 +4,9 @@
# Unit name: jellyfin.service (podman 5.x drops the container- prefix) # Unit name: jellyfin.service (podman 5.x drops the container- prefix)
# #
# Data roots migrated from /opt/jellyfin to /home/dank/Containers/jellyfin at # Data roots migrated from /opt/jellyfin to /home/dank/Containers/jellyfin at
# cutover. All of /mnt/entropy is mounted read-only at /data: nothing automated # cutover. The library is seen through /var/lib/media-meta/view, the SSD-backed
# anywhere writes to the entropy volume. Every # overlay view (scripts/media-view.sh): the array stays read-only and untouched,
# while the sidecars this server writes land on the NVMe. Every
# creation-time environment variable of the compose-era container is kept as # creation-time environment variable of the compose-era container is kept as
# an Environment= line (zero drift); none of them are secrets, so this unit # an Environment= line (zero drift); none of them are secrets, so this unit
# needs no EnvironmentFile. # needs no EnvironmentFile.
@@ -41,7 +42,12 @@ PublishPort=7359:7359/udp
PublishPort=1900:1900/udp PublishPort=1900:1900/udp
Volume=/home/dank/Containers/jellyfin/config:/config Volume=/home/dank/Containers/jellyfin/config:/config
Volume=/home/dank/Containers/jellyfin/cache:/cache Volume=/home/dank/Containers/jellyfin/cache:/cache
Volume=/mnt/entropy:/data:ro # The view, writable: Jellyfin is the metadata writer (movie.nfo / tvshow.nfo /
# episode.nfo + artwork) and those bytes belong on the SSD. The condition is the
# fail-closed guard — starting against a plain, unmounted view directory would let
# a scan conclude the library is empty.
Volume=/var/lib/media-meta/view:/data
UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view
AddDevice=/dev/dri AddDevice=/dev/dri
GroupAdd=keep-groups GroupAdd=keep-groups
HealthCmd=curl -f http://localhost:8096/health HealthCmd=curl -f http://localhost:8096/health
+6 -1
View File
@@ -11,7 +11,12 @@ Environment=PUID=0
Environment=PGID=0 Environment=PGID=0
Environment=TZ=America/Los_Angeles Environment=TZ=America/Los_Angeles
Volume=/var/lib/plexmediaserver:/config Volume=/var/lib/plexmediaserver:/config
Volume=/mnt/entropy:/mnt/entropy:ro # The container path stays /mnt/entropy on purpose: every path Plex recorded in
# its own database is written against it, so swapping the HOST source of this bind
# to the overlay view re-points Plex at the sidecars without touching its DB.
# Read-only: Plex consumes the NFO files Jellyfin writes and writes no metadata.
Volume=/var/lib/media-meta/view:/mnt/entropy:ro
UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view
PublishPort=32400:32400 PublishPort=32400:32400
[Service] [Service]
+1 -1
View File
@@ -21,7 +21,7 @@
# deployed knob has since dropped to 262144. # deployed knob has since dropped to 262144.
set -euo pipefail set -euo pipefail
IMG=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a IMG=ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0
CUR=/home/dank/models/halogen-flashnext-v2 CUR=/home/dank/models/halogen-flashnext-v2
NEW=/home/dank/models/halogen-v2-abliterated NEW=/home/dank/models/halogen-v2-abliterated
CUR_CK=qwen38-flash-next-v2-swift15-abliterated.hgn CUR_CK=qwen38-flash-next-v2-swift15-abliterated.hgn
+6 -1
View File
@@ -47,7 +47,12 @@
set -u -o pipefail set -u -o pipefail
SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
IMAGE=${IMAGE:-stilldeadcode/radiance@sha256:d590e300234bf6d2f942a0b8d456d183153fd808f28820e6c2fa69c9389178ed} # THE TOOLCHAIN PIN. `:latest` on 2026-10-30 is v1.14.0 (revision 4a61d7296bc5),
# the same image the tap producer boots: a tap must be converted by the line that
# is current, and rad-convert's `gptq:calib=<dir>` input contract is version-stable
# (RADGRAM2 + gram.r<rank>.<stem>.bin), so nothing else about the recipe moves with
# it. Production stays on its own pin — this is the campaign's toolchain.
IMAGE=${IMAGE:-stilldeadcode/radiance@sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}
CKPT=${CKPT:-/home/dank/models/Qwen3.8-Flash-Next} CKPT=${CKPT:-/home/dank/models/Qwen3.8-Flash-Next}
CAMPAIGN=${CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64} CAMPAIGN=${CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}
RECIPE=${RECIPE:-$SELF/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe} RECIPE=${RECIPE:-$SELF/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
+11 -1
View File
@@ -155,13 +155,14 @@ DANKCLAW_COMPOSE_SERVICES = set()
# Quadlet unit now — the last compose services were the hermes trio, whose units # Quadlet unit now — the last compose services were the hermes trio, whose units
# have owned their containers since 2026-09-12 and whose compose blocks were # have owned their containers since 2026-09-12 and whose compose blocks were
# dead weight that could only create a duplicate container. # dead weight that could only create a duplicate container.
# jellyfin left the Dankserver file on 2026-10-30 for the same reason: its Quadlet
# has owned the container since the cutover, so a compose edit must not restart it.
DANKSERVER_COMPOSE_SERVICES = { DANKSERVER_COMPOSE_SERVICES = {
"mariadb", "mariadb",
"redis", "redis",
"romm", "romm",
"copyparty", "copyparty",
"jellyfin",
"calibre-web", "calibre-web",
} }
COMPOSE_NAME_MAP = {"mariadb": "romm-mariadb", "redis": "romm-redis"} COMPOSE_NAME_MAP = {"mariadb": "romm-mariadb", "redis": "romm-redis"}
@@ -421,6 +422,11 @@ def services_for_path(path: str, target: str) -> set[str]:
"tautulli": {"tautulli"}, "tautulli": {"tautulli"},
} }
_add(services, *quadlet_services.get(parts[1], set())) _add(services, *quadlet_services.get(parts[1], set()))
if len(parts) >= 2 and parts[1] == "host":
# dankserver/host/systemd/media-meta-view.mount is the overlay view the
# media servers bind (scripts/media-view.sh installs it on the host),
# so editing it restarts jellyfin and plex and nothing else.
_add(services, "jellyfin", "plex")
if len(parts) >= 2 and parts[1] == "observability": if len(parts) >= 2 and parts[1] == "observability":
# Dankserver's node-exporter became a Quadlet unit on 2026-09-16 # Dankserver's node-exporter became a Quadlet unit on 2026-09-16
# (serverAppConfigs-docs/Infrastructure/Observability.md): its # (serverAppConfigs-docs/Infrastructure/Observability.md): its
@@ -466,6 +472,10 @@ def services_for_path(path: str, target: str) -> set[str]:
# that service (target_services then keeps it only on the host being # that service (target_services then keeps it only on the host being
# planned). Editing the collector restarts the running exporter. # planned). Editing the collector restarts the running exporter.
_add(services, "gpu-exporter") _add(services, "gpu-exporter")
if name == "media-view.sh":
# The installer is what creates, mounts and reaps the view both media
# servers read, so a change to it is a restart of that pair.
_add(services, "jellyfin", "plex")
if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}: if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}:
# llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve # llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve
# files here; each one selects frank-zealot only. The ik_llama.cpp # files here; each one selects frank-zealot only. The ik_llama.cpp
+26 -4
View File
@@ -1202,7 +1202,22 @@ deploy_romm_stack() {
systemctl --user restart "$service.service" systemctl --user restart "$service.service"
} }
ensure_media_view() {
# Both media servers read their library through the SSD-backed overlay view
# (scripts/media-view.sh, dankserver/host/systemd/media-meta-view.mount): the
# array below stays read-only and every sidecar Jellyfin writes lands on the
# NVMe.
# Raise it BEFORE either container starts — a start against a bare view
# directory would let a scan clear the library database. The units also carry
# ConditionPathIsMountPoint, so a missing view skips the start rather than
# damaging anything. A relayout must be followed by `media-view.sh reap`
# (see the Media Relayout page), which this is not.
run_privileged bash scripts/media-view.sh install
bash scripts/media-view.sh verify
}
deploy_jellyfin() { deploy_jellyfin() {
ensure_media_view
cp "dankserver/jellyfin/quadlet/jellyfin.container" "$quadlet_unit_dir/" cp "dankserver/jellyfin/quadlet/jellyfin.container" "$quadlet_unit_dir/"
systemctl --user daemon-reload systemctl --user daemon-reload
if ! systemctl --user is-active -q jellyfin.service; then if ! systemctl --user is-active -q jellyfin.service; then
@@ -1290,6 +1305,7 @@ deploy_copyparty() {
} }
deploy_plex() { deploy_plex() {
ensure_media_view
cp "dankserver/plex/quadlet/plex.container" "$quadlet_unit_dir/" cp "dankserver/plex/quadlet/plex.container" "$quadlet_unit_dir/"
systemctl --user daemon-reload systemctl --user daemon-reload
if ! systemctl --user is-active -q plex.service; then if ! systemctl --user is-active -q plex.service; then
@@ -1362,11 +1378,17 @@ deploy_authentik() {
# generated". Boot persistence comes from [Install] WantedBy=default.target # generated". Boot persistence comes from [Install] WantedBy=default.target
# inside each .container file, which the generator honors at user-manager # inside each .container file, which the generator honors at user-manager
# start (linger must stay on for the deploy user). # start (linger must stay on for the deploy user).
systemctl --user start authentik-postgres.service # RESTART, not start: `start` on an already-active unit is a no-op, so an
# image bump would install the new digest and keep serving the old
# container while every gate stayed green (measured 2026-10-30: server and
# worker stayed on the previous release beside an outpost that moved,
# because only the outpost handler restarts). The generator has already
# re-rendered the unit from the .container, so a restart is what applies it.
systemctl --user restart authentik-postgres.service
wait_container_healthy authentik-postgres 120 || return 1 wait_container_healthy authentik-postgres 120 || return 1
systemctl --user start authentik-server.service systemctl --user restart authentik-server.service
wait_container_healthy authentik-server 300 || return 1 wait_container_healthy authentik-server 300 || return 1
systemctl --user start authentik-worker.service systemctl --user restart authentik-worker.service
} }
deploy_authentik_outpost() { deploy_authentik_outpost() {
@@ -1482,7 +1504,7 @@ deploy_anubis() {
# artifact would leave a restart-looping unit and a dark endpoint. The # artifact would leave a restart-looping unit and a dark endpoint. The
# port gate is the same as every cutover since 2026-09-16: nothing # port gate is the same as every cutover since 2026-09-16: nothing
# starts until 7770 is released. # starts until 7770 is released.
LLM_DRAGOON_IMAGE="${LLM_DRAGOON_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a}" LLM_DRAGOON_IMAGE="${LLM_DRAGOON_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0}"
# The payload dir is its own twin because the NPU lane's containment gate below # The payload dir is its own twin because the NPU lane's containment gate below
# reads it; every artifact path under it stays a full literal as usual. # reads it; every artifact path under it stays a full literal as usual.
LLM_DRAGOON_MODEL_DIR="${LLM_DRAGOON_MODEL_DIR:-/home/dank/models/halogen-v2-abliterated}" LLM_DRAGOON_MODEL_DIR="${LLM_DRAGOON_MODEL_DIR:-/home/dank/models/halogen-v2-abliterated}"
+33 -16
View File
@@ -8,8 +8,10 @@
# (/opt/halogen/npu/models.txt inside the image) and refuses to load a model # (/opt/halogen/npu/models.txt inside the image) and refuses to load a model
# whose files do not match. # whose files do not match.
# #
# The manifest below IS that catalog block (dumped from the pinned 0.16.2 # The manifest below IS that catalog block (dumped from the pinned 0.18.20
# image on 2026-10-19), so the files land in the layout the entrypoint # image on 2026-10-30; the 0.16.2 block this replaced named different device
# programs — 10 `u*.elf` and a 5,086,496 B consts.hnpb — so a version bump
# re-fetches the set or the entrypoint refuses the boot), so the files land in the layout the entrypoint
# expects — <dir>/<model-id>/devices/*, <dir>/<model-id>/<model-id>.hnpw and # expects — <dir>/<model-id>/devices/*, <dir>/<model-id>/<model-id>.hnpw and
# <dir>/<model-id>/tokenizer/tokenizer.json — and the endpoint can keep # <dir>/<model-id>/tokenizer/tokenizer.json — and the endpoint can keep
# /models read-only and open no outbound connection at boot (HALOGEN_DOWNLOAD # /models read-only and open no outbound connection at boot (HALOGEN_DOWNLOAD
@@ -57,21 +59,25 @@ manifest() {
case "$1" in case "$1" in
qwen3-embedding-0.6b) qwen3-embedding-0.6b)
NPU_REPO="peonist-ai/halogen-npu-qwen3-embedding-0.6b" NPU_REPO="peonist-ai/halogen-npu-qwen3-embedding-0.6b"
NPU_REV="4960fd9d4423dd15373ad21c83db48dbdb3b2798" NPU_REV="8fd44c9f18861618a509bea75c51ff8a37e3bd5d"
FILES=( FILES=(
"devices/NOTICE.md 3248 0f267b3d2523ddf14997ee58660a492a9eaf10ccf6b08bf19dfdc85b9a478b91" "devices/NOTICE.md 3248 0f267b3d2523ddf14997ee58660a492a9eaf10ccf6b08bf19dfdc85b9a478b91"
"devices/consts.hnpb 5086496 ae430e41545a0b1012938cda373b0c450364d3be631111cca3a8c43679f93336" "devices/consts.hnpb 8495512 5af7c1f2f24d45a9054872bba882461cf7f6ffda832e785b700643d6467ad88b"
"devices/devices.hnpm 24047 4e9bca9d95d084fb89ce5b99f579a00666c8ca1efae2613c53e3a4878cc9f174" "devices/devices.hnpm 35465 85d870f6b900de1829659955f0dc701d57984580f5e5ec251add5d6969f03e3d"
"devices/u0.elf 4931968 b38f165886cb037b7d3951d9cb878473eaba58a432acd45416ff85fd88aa4e46" "devices/u0.elf 2975008 f74df12eb8c79e4e90cbedcc1dd8f142219772d400e3a142fde8a95378b105eb"
"devices/u1.elf 5456000 b1811feee851b8af833b37721da92099d89772110815bbbe2c3fa1c276f0ff42" "devices/u1.elf 3266848 ac5a3f89f934951b315b62ab53e1d0a74dbe6396134192641c405d08da332754"
"devices/u2.elf 6505552 406ea9a95b470628a7382e376f4a72cdf4d7aa4c59b4ddf0a4bdd139d63d6ff2" "devices/u2.elf 4015328 5e4f0d3342f262816289866d7e8c966f56a9f938c2e12e7341718cc7312aa480"
"devices/u3.elf 6817216 bc205d51b523d770151a0c61968f7c70bf624412827580bb6fe0aaf4650dc2ee" "devices/u3.elf 4326976 94cf5d269fa286d91c26bf4654cedf101c047d23c4495b74c9c129cdc33b210f"
"devices/u4.elf 7444032 a5a91338a2603acdcd5c11df532033fda2d3f5479ea44ae87f98fafbdff9a62d" "devices/u4.elf 4953792 cbd2796050d4c675f5cb3da117111087d510173e26f262f1b36aab66dd49dc91"
"devices/u5.elf 7432416 6c2bf41835569e0eba25ded7c05c7264ae69c6700bdeb4791057d884f2a4c126" "devices/u5.elf 4940112 7bf6a4470546bb888da02af395c07d92a24fb6d43aef27eb2151b3a9eda19e4d"
"devices/u6.elf 4941232 919d0038ca088eafddf195f3466ff38db27d3635be1cffa5d4d4b5140902a68e" "devices/u6.elf 6174544 10b9d50b635993e1f9c23775c0b94901d922c1e433cd10910e9f97174235f5ac"
"devices/u7.elf 5488304 d904cf8afae9428dd167960549e18cce5441745f18c62f8c50975dffc077d3c2" "devices/u7.elf 2983776 e0b06063b7379d05c85b3f435b819d96fda1223d8025d6d15958854b5711ace9"
"devices/u8.elf 4941232 db3aef69b17825854ee0dd2df475125a3f5c872f95c686793d6af1ee83596fa7" "devices/u8.elf 3298656 3bad204f782a68b97e27c1172530b13d7fbb0b495fa95f9504530a1048560644"
"devices/u9.elf 5488304 d182727e551084db8fb9600441d8aa373c7f09a3d267ae0a5e921ef92326db1c" "devices/u9.elf 2983776 875e313280358ce116e2a51a29a2c42794b5a9fadb3d48db702c90d18d838c2c"
"devices/u10.elf 3298656 ef2e0f399f149a5476955bcf233c6fa2e896fcef1f86ecb0d8113c952c41a8b2"
"devices/u11.elf 3298656 68a2519eee2d6a61ccc4ca2007e3186d44c9eb4e81162d742fe2f1756ec5a528"
"devices/u12.elf 3298656 2bdc6abfa58cac1d5795a3ef52fe8a9ab4e7c2a4792822d615499a913401f728"
"devices/u13.elf 3298656 4a3ce14f7ba15987b9bcbb4a010083e8aef68cd87c20cbe43403a98263179fe2"
"qwen3-embedding-0.6b.hnpw 813032248 b9f9901429df7b46ec17153aa3899f738adbea8e4356080da0c6e6586fb750d1" "qwen3-embedding-0.6b.hnpw 813032248 b9f9901429df7b46ec17153aa3899f738adbea8e4356080da0c6e6586fb750d1"
"tokenizer/tokenizer.json 11423705 def76fb086971c7867b829c23a26261e38d9d74e02139253b38aeb9df8b4b50a" "tokenizer/tokenizer.json 11423705 def76fb086971c7867b829c23a26261e38d9d74e02139253b38aeb9df8b4b50a"
) ;; ) ;;
@@ -99,7 +105,18 @@ for MODEL in ${MODELS//,/ }; do
# 62 GiB fetch hit this on 2026-10-18). # 62 GiB fetch hit this on 2026-10-18).
url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$HF/$NPU_REPO/resolve/$NPU_REV/$f") \ url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$HF/$NPU_REPO/resolve/$NPU_REV/$f") \
|| { echo "cannot resolve $NPU_REPO/$f" >&2; exit 1; } || { echo "cannot resolve $NPU_REPO/$f" >&2; exit 1; }
curl -fL --retry 5 -C - -o "$path" "$url" # Resume ONLY into a genuinely partial file. A staged file of a
# DIFFERENT size is an artifact of another revision (a version
# bump re-cuts the device programs), and `-C -` against it
# appends the tail to the old bytes: the result is the right
# BYTE COUNT with the wrong content, which reads as a corrupt
# program, not as a fetch fault (measured 2026-10-30 on
# consts.hnpb). Fetch those from zero.
resume=()
if [ -f "$path" ] && [ "$(stat -c %s "$path")" -lt "$want" ]; then
resume=(-C -)
fi
curl -fL --retry 5 "${resume[@]+"${resume[@]}"}" -o "$path" "$url"
fi fi
fi fi
[ -f "$path" ] || { echo "MISSING $path (run without --verify-only)" >&2; exit 1; } [ -f "$path" ] || { echo "MISSING $path (run without --verify-only)" >&2; exit 1; }
+5 -2
View File
@@ -10,8 +10,11 @@
# The .hgn format is the lock: it loads ONLY in the flash-server line, so the # The .hgn format is the lock: it loads ONLY in the flash-server line, so the
# engine moves with the payload, never ahead of it. # engine moves with the payload, never ahead of it.
# Image digest-pinned (a tag pull silently re-points; the digest is what was # Image digest-pinned (a tag pull silently re-points; the digest is what was
# qualified — 0.16.2 is the version the model card names as its quickstart pin). # qualified). 0.18.20 since 2026-10-30 (was 0.16.2, the model card's quickstart
FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a # pin) — the vendor has moved two minor versions and the payload lock is what
# the deploy's health gate plus the boot ledger decide; rollback is the
# previous digest, 0c61bf84…, on this same line.
FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0
# Container identity kept from the 27B era: the unit's ExecStop names it. # Container identity kept from the 27B era: the unit's ExecStop names it.
FRANK_DRAGOON_CONTAINER=frank-dragoon-halogen FRANK_DRAGOON_CONTAINER=frank-dragoon-halogen
# Predecessor identity the serve script clears if it is still holding the GPUs. # Predecessor identity the serve script clears if it is still holding the GPUs.
+5 -4
View File
@@ -1,12 +1,12 @@
# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) on the RTX 3090, # frank-zealot — llama.cpp (PrismML fork build 12118 = 0.7.0-dev, commit
# :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file, plus # 024474448, CUDA) on the RTX 3090, :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file, plus
# the refusal-direction adapter applied at load. # the refusal-direction adapter applied at load.
# #
# ONE OWNER: shape, payload identity, eras, laws and every measured number # ONE OWNER: shape, payload identity, eras, laws and every measured number
# live in serverAppConfigs-docs/LLM/frank-zealot.md. This file carries the # live in serverAppConfigs-docs/LLM/frank-zealot.md. This file carries the
# knobs and only the reason a reader editing THIS line cannot recover # knobs and only the reason a reader editing THIS line cannot recover
# elsewhere. Where the two disagree, this file wins on values. # elsewhere. Where the two disagree, this file wins on values.
LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server LLAMA_BIN=/home/dank/GitPackages/llama-prism-0244744/bin/llama-server
# The fork is a LOCK, not a preference: custom tensor type 142 + prism.hadamard.* # The fork is a LOCK, not a preference: custom tensor type 142 + prism.hadamard.*
# metadata. Stock llama.cpp refuses the file; a loader ignoring the Hadamard # metadata. Stock llama.cpp refuses the file; a loader ignoring the Hadamard
# metadata dequantizes wrong, silently. # metadata dequantizes wrong, silently.
@@ -64,7 +64,8 @@ CACHE_TYPE_V=q8_0
# weights, so there is no --spec-draft-model. Its bill is pool × 4,096 B of # weights, so there is no --spec-draft-model. Its bill is pool × 4,096 B of
# draft KV plus ~0.45 GiB of buffers and a tripled recurrent state — paid for # draft KV plus ~0.45 GiB of buffers and a tripled recurrent state — paid for
# by CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024 above. Build gate: quantized V # by CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024 above. Build gate: quantized V
# beside MTP needs fork build >= b10776; LLAMA_BIN is 10777. # beside MTP needs fork build >= b10776; LLAMA_BIN is 12118 (b10777 is kept on
# disk as the rollback path).
# SPEC_TYPE=none is the rollback and strips the WHOLE set (--spec-type and # SPEC_TYPE=none is the rollback and strips the WHOLE set (--spec-type and
# every --spec-draft-*); half a set has segfaulted at load. # every --spec-draft-*); half a set has segfaulted at load.
SPEC_TYPE=draft-mtp SPEC_TYPE=draft-mtp
+434
View File
@@ -0,0 +1,434 @@
#!/usr/bin/env python3
"""media-apply.py — the only thing that moves movie-library files.
Reads the plan written by scripts/media-plan.py (rows: `action src dst kind`,
plus `mkdir` rows) and performs it with rename(2) — bytes are never copied, so
an interrupted run has destroyed nothing. Every run is resumable and every run
is undoable: the journal is a write-ahead log, one JSON line per intent, fsynced
BEFORE the rename it authorises. That ordering is the whole design: a crash
leaves either an intent whose move happened or an intent whose move never ran,
and both are recognised on the next run. A file that moved with no journal line
cannot have been moved by this tool, and saying so is better than guessing.
Four subcommands:
apply --plan PLAN [--root ROOT] [--journal J] [--apply]
no --apply => report only and write NOTHING, not even folders
rollback --journal J [--root ROOT] [--apply]
back to the paths the census saw; folders this run created are
removed only when empty, never anything else
verify --plan PLAN --journal J [--root ROOT]
every planned file is at its target with the journalled byte count
sweep --journal J [--root ROOT] [--apply]
removes the folders THIS RUN emptied — the journal has to show a
file that moved out of one, so a folder someone made by hand can
never be removed by this tool, and rollback puts them back
Refusals (exit 2, before the first rename, with nothing touched): a target
already occupied by a foreign file, a source that has gone, a duplicate target,
a path outside --root, a journal that was already rolled back, a move that would
cross a filesystem (rename cannot do it and a silent copy at 24 TB would not).
--force relaxes only the guards that are opinion (journal already rolled back,
paths outside --root); it never allows overwriting a file.
The journal defaults to `media-relayout.jsonl` BESIDE the library root, never
inside it: a log inside the library is a file Jellyfin will index.
"""
import argparse
import datetime
import json
import os
import sys
DEFAULT_ROOT = os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies")
class Refuse(Exception):
"""A gate the operator has to see; every refusal leaves the library alone."""
def journal_default(root):
return os.path.join(os.path.dirname(os.path.abspath(root)), "media-relayout.jsonl")
class Journal:
"""Append-only JSONL of intents. Writes are fsynced before the tool acts."""
def __init__(self, path, writing):
self.path, self.writing = path, writing
self.records = []
if os.path.exists(path):
for line in open(path):
line = line.strip()
if line:
self.records.append(json.loads(line))
self._fh = None
def already(self, dst, src):
"""An intent for this exact (src, dst) was written by an earlier run."""
return any(r.get("op") == "move" and r.get("dst") == dst
and r.get("src") == src for r in self.records)
@property
def rolled_back(self):
return any(r.get("op") == "rollback" for r in self.records)
def record(self, **fields):
self.records.append(fields)
if not self.writing:
return
if self._fh is None:
self._fh = open(self.path, "a")
self._fh.write(json.dumps(fields) + "\n")
self._fh.flush()
os.fsync(self._fh.fileno())
def close(self):
if self._fh:
self._fh.close()
def read_plan(path, root, force):
"""Plan rows: (mkdirs, keeps, moves), header dropped, all paths under root."""
mkdirs, keeps, moves = [], [], []
with open(path) as fh:
header = fh.readline()
if not header.startswith("action\t"):
raise Refuse(f"{path} is not a plan (first column is "
f"{header.split(chr(9))[0]!r}, expected 'action') —"
" regenerate it with scripts/media-plan.py")
for line in fh:
f = line.rstrip("\n").split("\t")
if f[0] == "mkdir":
mkdirs.append(f[2])
elif f[0] == "keep":
keeps.append(f[1])
elif f[0] == "move":
moves.append((f[1], f[2], f[3] if len(f) > 3 else ""))
if not force:
for p in mkdirs + keeps + [p for m in moves for p in m[:2]]:
if p and not (p == root or p.startswith(root + os.sep)):
raise Refuse(f"{p} is outside --root {root}"
" (a plan built for another library?)")
return mkdirs, keeps, moves
# A relayout moves bytes on the raw array; the sidecars the media servers wrote
# live in the SSD overlay and stay at the old coordinates. Naming the follow-up
# here is the point of use for that fact — see the Media Metadata View page.
METADATA_HINT = ("note: sidecars written through the metadata view did not follow"
" these moves — next: scripts/media-view.sh reap")
def do_apply(args, root, journal, mkdirs, keeps, moves):
"""Preflight everything, then folders, then renames. Never a partial start."""
problems = []
for p in keeps:
if not os.path.exists(p):
problems.append(f"missing: keep row source is gone: {p}")
seen = {}
for src, dst, _kind in moves:
if src == dst:
problems.append(f"no-op: move row whose target is its source: {src}")
if dst in seen:
problems.append(f"double-claim: {dst} claimed by {seen[dst]} and {src}")
seen[dst] = src
if journal.already(dst, src) and os.path.exists(dst) and not os.path.exists(src):
continue # this intent already ran
if not os.path.exists(src):
if os.path.exists(dst):
problems.append(f"missing: {src} is gone and {dst} carries no journal"
" line — it moved outside this tool, refusing to guess")
else:
problems.append(f"missing: {src} is gone and {dst} is not there either")
continue
if os.path.exists(dst):
problems.append(f"occupied: target already holds another file: {dst}")
if problems:
print(f"refusing: {len(problems)} problem(s), nothing moved")
for p in sorted(problems)[:20]:
print(" " + p)
if len(problems) > 20:
print(f" ... and {len(problems) - 20} more")
return 2
if not args.apply:
print(f"dry run: {len(moves)} moves, {len(mkdirs)} folders, {len(keeps)} files stay"
" — nothing written (pass --apply to run)")
for src, dst, _k in moves[:5]:
print(f" would move {src} -> {dst}")
return 0
made = {r.get("dst") for r in journal.records if r.get("op") == "mkdir"}
dirs_new = 0
for d in sorted(mkdirs, key=lambda q: q.count(os.sep)):
if d in made or os.path.isdir(d):
continue
journal.record(op="mkdir", dst=d)
os.makedirs(d, exist_ok=True)
dirs_new += 1
# same-filesystem gate: rename cannot cross a device, and a silent copy of a
# library this size is not an option anyone would choose.
cross = []
for src, dst, _k in moves:
if journal.already(dst, src) and os.path.exists(dst) and not os.path.exists(src):
continue
parent = os.path.dirname(dst)
if os.path.exists(parent) and os.stat(parent).st_dev != os.stat(src).st_dev:
cross.append(dst)
if cross:
print(f"refusing: {len(cross)} moves would cross a filesystem"
f" (first: {cross[0]}) — this tool only renames")
return 2
moved = done = 0
for src, dst, _kind in moves:
if journal.already(dst, src):
if os.path.exists(dst) and not os.path.exists(src):
done += 1
continue
if os.path.exists(src) and not os.path.exists(dst):
os.rename(src, dst) # the intent ran, the move did not
moved += 1
continue
st = os.stat(src)
journal.record(op="move", src=src, dst=dst, bytes=st.st_size, mode=st.st_mode,
mtime=st.st_mtime, ino=st.st_ino, dev=st.st_dev)
os.rename(src, dst)
moved += 1
print(f"applied: {moved} moved, {done} already done in an earlier run,"
f" {dirs_new} folders created, {len(keeps)} files stayed put")
print(f"journal: {journal.path} — undo with: media-apply.py rollback"
f" --journal {journal.path} --root {root} --apply")
print(METADATA_HINT)
return 0
def do_verify(root, journal, moves, keeps):
"""Every planned file sits at its target carrying the journalled bytes."""
journalled = {(r["src"], r["dst"]): r for r in journal.records
if r.get("op") == "move"}
todo = moves or [(r["src"], r["dst"], "") for r in journal.records
if r.get("op") == "move"]
bad, missing, bytes_total = [], [], 0
for src, dst, _kind in todo:
rec = journalled.get((src, dst))
if not os.path.exists(dst):
missing.append(dst)
continue
size = os.path.getsize(dst)
bytes_total += size
if rec and rec.get("bytes") not in (None, size):
bad.append(f"{dst}: {size} bytes on disk, {rec['bytes']} at move time")
if os.path.exists(src) and src != dst:
bad.append(f"{src}: still in place as well as at {dst}")
missing += [p for p in keeps if not os.path.exists(p)]
if bad or missing:
print(f"verify FAILED: {len(missing)} absent, {len(bad)} wrong")
for line in (missing + bad)[:20]:
print(" " + line)
return 2
empty = [r["dst"] for r in journal.records
if r.get("op") == "mkdir" and os.path.isdir(r["dst"])
and not os.listdir(r["dst"])]
print(f"verified: {len(todo)} files at their targets"
f" ({bytes_total / 2**40:.2f} TiB), {len(keeps)} kept")
if empty:
print(f"note: {len(empty)} folders are empty (left in place — see the trash law)")
return 0
def sweep_targets(root, journal):
"""The folders this run emptied, deepest first — never one it did not touch.
Eligibility is evidence, not appearance. A folder is a target only when the
journal shows a file moved out of it or out of something beneath it, it holds
nothing but empty directories now, it is not a folder this run created, and no
journalled destination lives under it. A symlink anywhere inside disqualifies
the whole subtree: this tool never follows a link, and deleting the folder
that holds one would delete someone's mount point.
Proof carries downward: once a folder qualifies, every empty directory inside
it qualifies too. A release's junk (``Screens/``, ``Sample/``) is never itself
a move source, so without that rule it holds its emptied parent on disk
forever — the parent can never be removed while the child stands.
"""
moves = [r for r in journal.records if r.get("op") == "move"]
made = {r["dst"] for r in journal.records if r.get("op") == "mkdir"}
# An rmdir INTENT that failed left its folder standing, and the next sweep
# must be able to finish it: only a folder that is really gone is out of
# scope. Counting every intent as done made 146 live survivors permanent.
gone = {r["path"] for r in journal.records if r.get("op") == "rmdir"
and not os.path.exists(r["path"])}
dsts = [r["dst"] for r in moves]
marked = set()
for r in moves:
d = os.path.dirname(r["src"])
while d.startswith(root + os.sep):
marked.add(d)
d = os.path.dirname(d)
def removable(d):
if d in made or d in gone or os.path.islink(d):
return False
if any(p == d or p.startswith(d + os.sep) for p in dsts):
return False
try:
entries = os.listdir(d)
except OSError:
return False
for e in entries:
c = os.path.join(d, e)
if os.path.islink(c) or not os.path.isdir(c) or not removable(c):
return False
return True
roots = [d for d in marked if removable(d)]
targets = set(roots)
for d in roots:
for dirpath, dirnames, _ in os.walk(d):
for c in dirnames:
p = os.path.join(dirpath, c)
if removable(p):
targets.add(p)
return sorted(targets, key=lambda q: (-q.count(os.sep), q))
def do_sweep(args, root, journal):
"""Remove the empty husks an apply left standing at the library root."""
moves = [r for r in journal.records if r.get("op") == "move"]
if not moves:
raise Refuse("the journal holds no moves — a sweep follows an apply, it"
" does not guess at emptiness")
outside = sorted({p for r in moves for p in (r["src"], r["dst"])
if not (p == root or p.startswith(root + os.sep))})
if outside and not args.force:
raise Refuse(f"{len(outside)} journalled paths are outside --root {root}"
f" (first: {outside[0]}) — this journal belongs to another"
" library")
targets = sweep_targets(root, journal)
if not targets:
print("swept: 0 folders — nothing this run emptied is still standing empty")
return 0
if not args.apply:
print(f"dry run: {len(targets)} folders this run emptied would be removed"
" (pass --apply to run)")
for d in targets[:5]:
print(f" would remove {d}")
return 0
removed = 0
for d in targets:
journal.record(op="rmdir", path=d)
try:
os.rmdir(d)
except OSError as e:
print(f"left in place: {d} ({e})")
continue
removed += 1
print(f"swept: {removed} folders removed, {len(targets) - removed} left in place")
print(f"journal: {journal.path} — rollback recreates them with their files")
return 0
def do_rollback(args, root, journal):
"""Undo every journalled move, newest first, after a full preflight."""
moves = [r for r in journal.records if r.get("op") == "move"]
made = [r["dst"] for r in journal.records if r.get("op") == "mkdir"]
swept = [r["path"] for r in journal.records if r.get("op") == "rmdir"]
problems = []
for r in reversed(moves):
src, dst = r["src"], r["dst"]
if not args.force and not (src.startswith(root + os.sep)
or dst.startswith(root + os.sep)):
problems.append(f"outside --root: {dst}")
continue
if os.path.exists(dst) and os.path.exists(src):
problems.append(f"occupied: both paths exist, refusing to choose:"
f" {src} | {dst}")
elif os.path.exists(dst):
pass
elif os.path.exists(src):
continue # already undone
else:
problems.append(f"missing: {dst} is gone and {src} never came back")
if problems:
print(f"refusing: {len(problems)} problem(s), nothing moved back")
for p in problems[:20]:
print(" " + p)
return 2
if not args.apply:
print(f"dry run: would move {len(moves)} files back to their census paths"
f" and recreate {len(set(swept))} swept folders (pass --apply)")
return 0
undone = 0
recreated = 0
for d in sorted(set(swept), key=lambda q: q.count(os.sep)):
if not os.path.isdir(d):
os.makedirs(d, exist_ok=True)
recreated += 1
for r in reversed(moves):
if os.path.exists(r["dst"]) and not os.path.exists(r["src"]):
os.makedirs(os.path.dirname(r["src"]), exist_ok=True)
os.rename(r["dst"], r["src"])
undone += 1
dirs_gone = 0
for d in sorted(set(made), key=lambda q: -q.count(os.sep)):
if d == root or not os.path.isdir(d) or os.listdir(d):
continue
try:
os.rmdir(d)
dirs_gone += 1
except OSError as e:
print(f"left in place: {d} ({e})")
journal.record(op="rollback", undone=undone, dirs_removed=dirs_gone,
t=datetime.datetime.now(datetime.timezone.utc).isoformat())
print(f"rolled back: {undone} files back where the census found them,"
f" {dirs_gone} empty folders removed, {recreated} swept folders recreated")
print(METADATA_HINT)
return 0
def main():
ap = argparse.ArgumentParser(description="apply or undo a media relayout plan")
ap.add_argument("cmd", choices=["apply", "rollback", "verify", "sweep"])
ap.add_argument("--plan")
ap.add_argument("--root", default=DEFAULT_ROOT)
ap.add_argument("--journal")
ap.add_argument("--apply", action="store_true", help="actually change the library")
ap.add_argument("--force", action="store_true", help="relax the opinion guards")
args = ap.parse_args()
root = os.path.abspath(args.root)
if not os.path.isdir(root):
raise Refuse(f"no library at {root}")
journal_path = args.journal or journal_default(root)
journal = Journal(journal_path, args.apply)
try:
if args.cmd in ("rollback", "verify", "sweep") and not os.path.exists(journal_path):
raise Refuse(f"no journal at {journal_path} — nothing to {args.cmd}")
if args.cmd in ("rollback", "sweep") and not args.force and journal.rolled_back:
raise Refuse("this journal has already been rolled back"
" (--force to replay it)")
if args.cmd in ("apply", "verify") and not args.plan:
raise Refuse(f"{args.cmd} needs --plan")
mkdirs, keeps, moves = (read_plan(args.plan, root, args.force)
if args.plan else ([], [], []))
if args.cmd == "apply":
return do_apply(args, root, journal, mkdirs, keeps, moves)
if args.cmd == "verify":
return do_verify(root, journal, moves, keeps)
if args.cmd == "sweep":
return do_sweep(args, root, journal)
return do_rollback(args, root, journal)
finally:
journal.close()
if __name__ == "__main__":
try:
sys.exit(main())
except Refuse as r:
print(f"refusing: {r}")
sys.exit(2)
+203
View File
@@ -0,0 +1,203 @@
#!/usr/bin/env python3
r"""Read-only census of a movie library — the inputs a relayout plan is built from.
media-census.py enumerate [--root DIR] [--out TSV]
media-census.py probe [--enum TSV] [--out TSV] [--jobs N]
media-census.py plex [--root DIR] [--db PATH] [--out TSV]
Run it on the machine that mounts the library. Nothing here writes inside the
library and nothing is deleted; the only outputs are the three TSVs below, and
scripts/media-plan.py reads them instead of touching media.
enum bytes <TAB> ext <TAB> path every file under the root
probed bytes <TAB> path <TAB> dur <TAB> vcodec <TAB> WxH <TAB> fps <TAB> vbit
<TAB> pix <TAB> audio <TAB> subs <TAB> spare <TAB> format
one row per video-container file; `audio` is "codec:lang+codec:lang",
`subs` the subtitle-stream count, `spare` the audio streams carrying
neither a default nor a forced tag
plex file <TAB> leaf_title <TAB> … (see PLEX_COLS)
the titles Plex already matched — the authority over any filename guess
`probe` is the slow step (every container is opened for its header, ~10 min for
4,800 files on a spindle array at 6 workers) so it is `nice`d: a census beside a
live service must not starve it. `enumerate` is a stat pass, so re-censusing
before a plan is cheap — and a plan is only ever as good as these three files.
"""
import argparse
import csv
import json
import os
import re
import sqlite3
import subprocess
import sys
from concurrent.futures import ThreadPoolExecutor
# Containers ffprobe can read. .iso/.img are deliberately NOT here: a disc image is
# a mounted volume, not a stream, and the planner treats it as a movie file instead.
PROBE_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".bdmv", ".vob"}
# Trash and thumbnail caches are not library content, and this repo's trash policy
# means .AGENTTRASH can sit inside a movie folder at any time.
SKIP_DIR = re.compile(r"(^|/)(\.AGENTTRASH|\.thumbnails|\.Trash-\d+|@eaDir|lost\+found)(/|$)")
PLEX_DB = ("/var/lib/plexmediaserver/Library/Application Support/Plex Media Server"
"/Plug-in Support/Databases/com.plexapp.plugins.library.db")
PLEX_COLS = ["file", "leaf_title", "leaf_orig", "leaf_edition", "leaf_year",
"parent_title", "parent_orig", "parent_year", "parent_edition",
"container", "vcodec", "acodec", "width", "height"]
PLEX_SQL = """
select p.file, coalesce(mi.title,''), coalesce(mi.original_title,''),
coalesce(mi.edition_title,''), coalesce(mi.year,0),
coalesce(par.title,''), coalesce(par.original_title,''),
coalesce(par.year,0), coalesce(par.edition_title,''),
m.container, m.video_codec, m.audio_codec, m.width, m.height
from media_parts p
join media_items m on m.id = p.media_item_id
join metadata_items mi on mi.id = m.metadata_item_id
left join metadata_items par on par.id = mi.parent_id
where p.file like ? and p.deleted_at is null and mi.deleted_at is null
"""
def enumerate_lib(root, out):
"""One row per real file under root: size, extension, path."""
rows = []
for dirpath, dirnames, filenames in os.walk(root):
dirnames[:] = [d for d in dirnames
if not SKIP_DIR.search(os.path.join(dirpath, d))]
for fn in filenames:
p = os.path.join(dirpath, fn)
if SKIP_DIR.search(p):
continue
try:
st = os.lstat(p)
except OSError:
continue
if not os.path.isfile(p) or os.path.islink(p):
continue # links and sockets are not library content
rows.append((st.st_size, os.path.splitext(fn)[1].lower(), p))
with open(out, "w") as fh:
for size, ext, p in rows:
fh.write(f"{size}\t{ext}\t{p}\n")
vid = [r for r in rows if r[1] in PROBE_EXT]
print(f"files={len(rows)} bytes={sum(r[0] for r in rows)} "
f"probes={len(vid)} probe_bytes={sum(r[0] for r in vid)} -> {out}")
def probe_one(path):
"""ffprobe header read -> the probed TSV row (never the whole file)."""
try:
os.nice(19) # a census must never outrank the media server
except OSError:
pass
try:
size = os.stat(path).st_size
except OSError:
return None
cmd = ["ffprobe", "-v", "error", "-print_format", "json",
"-show_format", "-show_streams", path]
try:
done = subprocess.run(cmd, capture_output=True, text=True, timeout=180)
data = json.loads(done.stdout or "{}")
except (subprocess.TimeoutExpired, subprocess.SubprocessError,
json.JSONDecodeError, OSError):
data = {}
st = data.get("streams", [])
fmt = data.get("format", {})
v = next((s for s in st if s.get("codec_type") == "video"), None)
aud = [s for s in st if s.get("codec_type") == "audio"]
subs = [s for s in st if s.get("codec_type") == "subtitle"]
fps = "-"
if v and v.get("r_frame_rate"):
try:
n, d = (int(x) for x in v["r_frame_rate"].split("/", 1))
fps = f"{round(n / d * 100) / 100:g}" if d else "0"
except ValueError:
pass
def num(x):
try:
return round(float(x))
except (TypeError, ValueError):
return 0
return [size, path, num(fmt.get("duration")),
(v or {}).get("codec_name", "-"),
f"{v['width']}x{v['height']}" if v else "-",
fps, num(fmt.get("bit_rate")), (v or {}).get("pix_fmt", "-"),
"+".join(f"{a.get('codec_name', '?')}:{a.get('language', '?')}" for a in aud),
len(subs),
sum(1 for a in aud if not re.search(
r"default|forced", json.dumps(a.get("tags", {})), re.I)),
fmt.get("format_name", "-") or "-"]
def probe(enum, out, jobs):
"""Probe every video-container row of the enum TSV, in parallel."""
paths = []
with open(enum) as fh:
for line in fh:
f = line.rstrip("\n").split("\t", 2)
if len(f) == 3 and f[1] in PROBE_EXT:
paths.append(f[2])
print(f"probing {len(paths)} files with {jobs} workers", flush=True)
done = unreadable = 0
with open(out, "w") as fh, ThreadPoolExecutor(max_workers=jobs) as pool:
for row in pool.map(probe_one, paths):
if row is None:
continue
if row[3] == "-":
unreadable += 1 # header unreadable: the row still gets written
done += 1
fh.write("\t".join(str(x) for x in row) + "\n")
if done % 250 == 0:
print(f" {done}/{len(paths)}", flush=True)
print(f"probed={done} unreadable={unreadable} -> {out}")
def plex(root, db, out):
"""Plex's matched titles for files under root, from its DB, read-only."""
uri = f"file:{db}?mode=ro&immutable=1"
try:
con = sqlite3.connect(uri, uri=True)
except sqlite3.Error as e:
sys.exit(f"cannot open the Plex database read-only ({db}): {e}\n"
"Plex runs rootless here: run this as the container's owner, never "
"with sudo — rootless storage is per-user, so root sees nothing.")
with open(out, "w", newline="") as fh:
w = csv.writer(fh, delimiter="\t")
w.writerow(PLEX_COLS)
n = 0
for r in con.execute(PLEX_SQL, (root + "/%",)):
w.writerow([str(x).replace("\t", " ").replace("\n", " ") for x in r])
n += 1
print(f"plex-matched={n} -> {out}")
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
sub = ap.add_subparsers(dest="cmd", required=True)
root = {"default": os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
"help": "library root"}
e = sub.add_parser("enumerate", help="stat every file under the root")
e.add_argument("--root", **root)
e.add_argument("--out", default="/tmp/media-enum.tsv")
p = sub.add_parser("probe", help="ffprobe every video container")
p.add_argument("--enum", default="/tmp/media-enum.tsv")
p.add_argument("--out", default="/tmp/media-probed.tsv")
p.add_argument("--jobs", type=int, default=6)
x = sub.add_parser("plex", help="export Plex's matched titles for the root")
x.add_argument("--root", **root)
x.add_argument("--db", default=PLEX_DB)
x.add_argument("--out", default="/tmp/plex-movies.tsv")
a = ap.parse_args()
if a.cmd == "enumerate":
enumerate_lib(a.root, a.out)
elif a.cmd == "probe":
probe(a.enum, a.out, a.jobs)
else:
plex(a.root, a.db, a.out)
if __name__ == "__main__":
main()
+881
View File
@@ -0,0 +1,881 @@
#!/usr/bin/env python3
r"""Plan a Jellyfin-format relayout of a movie library. DRY RUN ONLY: nothing moves.
media-plan.py [--root DIR] [--enum TSV] [--probed TSV] [--plex TSV]
[--out TSV] [--exceptions TXT]
Inputs are the TSVs scripts/media-census.py exports, plus the optional Plex
metadata-cache export whose clean titles outrank the filename parser; the title
authority for everything Plex never matched is scripts/media-titles.py beside this
file. No media file is opened, so planning the whole library costs seconds.
PLAN is a TSV the executor (scripts/media-apply.py) replays verbatim —
`action<TAB>src<TAB>dst<TAB>kind` with action in mkdir | move | keep, ordered so
parents exist before children. EXCEPTIONS is the human review list: every file the
plan could not name a film for, every collision it refused, the READMEs the
executor must write into the piles it created.
Target contract (jellyfin.org/docs/general/server/media/movies):
* one folder per movie, named Title (year)
* the video inside is named exactly like the folder
* extra copies of one film are versions: "<base> - <label>.ext"; the base must
match the folder name character-for-character or Jellyfin splits them apart
* extras live in canonical subfolders (trailers, featurettes, deleted scenes,
interviews, samples, shorts, clips, other, extras, backdrops, theme-music)
* sidecars <base>.<lang>.<flag>.<ext>; images <base>-<poster|backdrop|...>
* <>:"/\|?* break the matcher, so they never survive here
* VIDEO_TS/BDMV folders support neither multiple versions nor parts
"""
import argparse
import collections
import csv
import importlib.util
import os
import re
# Inputs are the census TSVs scripts/media-census.py writes; the engine beside
# this file is the title authority. Nothing here opens a media file.
def parse_args():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--root", default=os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
help="library root the plan moves files inside")
ap.add_argument("--enum", default="/tmp/media-enum.tsv", help="file census TSV")
ap.add_argument("--probed", default="/tmp/media-probed.tsv", help="ffprobe census TSV")
ap.add_argument("--plex", default="/tmp/plex-movies.tsv",
help="Plex metadata export (clean titles); absent = parser only")
ap.add_argument("--out", default="/tmp/jf-plan.tsv")
ap.add_argument("--exceptions", default="/tmp/jf-exceptions.txt")
return ap.parse_args()
ARGS = parse_args()
ROOT = ARGS.root
ENUM, PROBE, PLEX = ARGS.enum, ARGS.probed, ARGS.plex
PLAN, EXCEPTS = ARGS.out, ARGS.exceptions
ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
spec = importlib.util.spec_from_file_location("eng", ENGINE)
eng = importlib.util.module_from_spec(spec)
spec.loader.exec_module(eng)
VIDEO_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".flv", ".vob", ".iso",
".img", ".mpv", ".3gp", ".rm", ".rmvb", ".m2v", ".mpv"}
DISC_DIRS = {"BDMV", "VIDEO_TS", "CERTIFICATE", "AACS", "MLPI", "CLIPINF"}
SUB_EXT = {".srt", ".ass", ".ssa", ".sub", ".vtt"}
IMG_EXT = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".gif", ".tbn"}
AUDIO_DIR_RE = re.compile(r"soundtrack|score|\bost\b|music|song|album", re.I)
SUB_KEEP_RE = re.compile(r"^(?!.*\d)(?!.*[._])(?!.*\[)(?!.*\()([A-Za-z][A-Za-z -]{1,24})$")
MUSIC_TOP_RE = re.compile(r"(?i)soundtrack|score|\bost\b|music|album|song")
SUB_DIR_OK = re.compile(r"(?i)^$|subs?|subtitles?|srt|custom ?subs?(\[.*\])?"
r"|(additional|extra) languages|languages|resources")
AUDIO_DIR_OK = re.compile(r"(?i)^$|audio(s)?|extra audio|additional audio"
r"|(additional|extra) languages|commentar(y|ies)"
r"|audio commentary|audio tracks?")
AUDIO_EXT = {".mp3", ".flac", ".m4a", ".aac", ".dts", ".wav", ".ac3", ".mka",
".opus", ".ogg", ".wma"}
NFO_EXT = {".nfo"}
IMG_PRIMARY = {"poster", "folder", "cover", "default", "movie", "show"}
IMG_OTHER = {"backdrop": "backdrop", "fanart": "backdrop", "background": "backdrop",
"banner": "banner", "logo": "logo", "clearlogo": "logo",
"thumb": "thumb", "landscape": "thumb", "art": "backdrop",
"clearart": "backdrop", "jacket": "poster", "cdart": "poster",
"discart": "poster", "disc": "poster"}
RES_RE = re.compile(r"\b(2160|4320|4k|1080|720|576|480|360|240)(i|p)\b", re.I)
# The characters Jellyfin's matcher cannot carry in a file or folder name. "/" is
# one of them and it is the dangerous one: Plex titles KEEP it ("V/H/S/2", and this
# library holds six of those films), so leaving it in a name turns ONE movie folder
# into a nested tree — Movies/V/H/S/2 (2013)/… — and Jellyfin then sees four
# folders, none of them a movie.
BAD_CHARS = re.compile(r'[<>:"/|?*]')
PART_RE = re.compile(r"(?:^|[\s._-])(cd|dvd|part|pt|disc|disk|disco)"
r"[\s._-]?([1-9abcd])(?:$|[\s._-])", re.I)
LANGS = {"en", "eng", "it", "ita", "ja", "jpn", "de", "deu", "fr", "fra", "es",
"spa", "pt", "por", "ru", "rus", "zh", "chi", "ko", "kor", "sv", "no",
"da", "nl", "pl", "cs", "hu", "fi", "tr", "el", "he", "ar", "uk", "ro",
"bg", "hr", "sr", "sk", "et", "lv", "lt", "id", "th", "vi", "fa", "la"}
FLAGS = {"default", "forced", "foreign", "sdh", "cc"}
NOISE = {"sub", "subs", "subtitle", "subtitles", "srt", "ass", "ext", "external",
"the", "a", "an", "of", "and", "in", "on", "to", "for", "with", "ita",
"eng", "english", "italian", "dual", "audio", "forced", "default"}
# Canonical title where the library only ever carries a localised name.
OVERRIDES = {"28 giorni dopo": "28 Days Later",
"28 settimane dopo": "28 Weeks Later"}
rows = []
for line in open(ENUM):
size, ext, path = line.rstrip("\n").split("\t")
rows.append({"bytes": int(size), "ext": ext.lower(), "path": path})
ext_by = {r["path"]: r["ext"] for r in rows}
size_by = {r["path"]: r["bytes"] for r in rows}
def rel(p):
return p[len(ROOT) + 1:] if p.startswith(ROOT + "/") else p
def parts_of(p):
return [q for q in rel(p).split("/") if q]
def is_qb(p):
return ".!q" in p.lower()
def disc_root(p, fold=True):
"""Absolute root of the disc tree containing p (the MOVIE folder that holds
BDMV/ or VIDEO_TS/), else None. `fold` also folds bookkeeping dirs that sit
beside the tree (CERTIFICATE/, AACS/) into the same movie folder."""
ps = parts_of(p)
kinds = DISC_DIRS if fold else {"BDMV", "VIDEO_TS"}
for i, q in enumerate(ps):
if q.upper() in kinds:
return ROOT + "/" + "/".join(ps[:i])
return None
def row_name(r):
"""The name a disc-tree entry should be identified by: the engine stores the
disc's own folder for it, because "BDMV/index.bdmv" is not a title."""
return r.get("name_src") or r["path"]
def sanitize(name):
name = BAD_CHARS.sub("-", name).replace("\\", "-")
name = re.sub(r"\s+", " ", name).strip().strip(". ").strip()
return (name[:180] or "untitled")
def ok_year(y):
"""True for a year a real film could carry. Plex stores 0 for a file it indexed
but never MATCHED, and "0" is a digit string: trusting it named 67 movie
folders "Title (0)" — Jellyfin items with no identity, no year and no poster."""
return bool(re.fullmatch(r"(18|19|20)\d\d", y or "")) and 1870 <= int(y) <= 2035
WORD_SPLIT = re.compile(r"[^0-9a-z]+")
def words_of(text):
"""The matching words of a name — or of a whole path, which is how callers
pass a file together with the folders it sits in.
Split on everything that is not a letter or a digit: eng.norm does not touch
"/", so feeding it "…/Psycho (1960)/Extras" left glued tokens like
"(1960)/extras" and "(1954)" that no other name could ever agree with, and a
film could not be found by the folder that holds it — which is how one box
set's featurettes ended up in another film's folder. Both sides of every
comparison below go through here. A bare number counts only when it could be
a release year, so release junk ("1080", "2160", "2", "7") never poses as
agreement between two names.
"""
out = set()
for w in WORD_SPLIT.split(eng.norm(text)):
if not w or w in eng.STOP:
continue
if w.isdigit() and not 1870 <= int(w) <= 2100:
continue
out.add(w)
return out
# ------------------------------------------------------------------ Plex titles
# Plex has matched a large part of the library's media files to real titles
# ("Alien", not "Alien.Directors.Cut.1979.1080p.BRrip.x265…"). Where the cache
# holds a MATCH, its title wins; the filename parser carries everything else
# (extras, disc trees, TV, and whatever Plex never matched).
def plex_titles():
if not os.path.exists(PLEX):
return {}
out = {}
with open(PLEX, newline="") as fh:
for r in csv.DictReader(fh, delimiter="\t"):
t, y = r["leaf_title"].strip(), r["leaf_year"].strip()
# Only a match is an authority, and a match carries a year: an
# unmatched row echoes the filename, which the parser reads better
# than a copied release name. The junk filter stays for the matches
# whose title still carries the release.
if t and ok_year(y) and not re.search(
r"2160p|1080p|x26[45]|hevc|bluray|web-?dl", t.lower()):
out[r["file"]] = (t, y)
return out
PLEX_T = plex_titles()
# ------------------------------------------------------------- classification
erows = eng.load(PROBE)
for r in erows:
for k, v in OVERRIDES.items():
if k in r["title"]:
r["title"] = v
r["key"] = eng.key_of(r["title"])
r["base"] = eng.base_words(r["title"], r["key"])
marks = {r["path"] for r in erows if eng.is_watermark(r["path"], r["bytes"])}
extras = {r["path"] for r in erows
if r["path"] not in marks
and eng.is_extra(r["path"], r["title"], r["dur"], ROOT,
matched=r["path"] in PLEX_T)}
eps = collections.defaultdict(list)
for r in erows:
if r["path"] in marks | extras:
continue
ep = eng.ep_of(r["path"])
if ep:
eps[ep[0]].append(r) # keyed by the SERIES, never the episode title
tv_keys = {k for k, v in eps.items() if len(v) >= 5}
tv_set = {r["path"] for k, v in eps.items() if k in tv_keys for r in v}
# A folder INSIDE a movie's folder is bonus territory even when its own name is
# not bonus vocabulary: "Saboteur (1942)/Exrtas/Saboteur - A Closer Look" is a
# misspelled Extras folder, so the engine's folder rule never fired and the
# featurette became a movie folder of its own — whose words then out-scored the
# real Saboteur and drew that box set's other extras into it. The rule needs no
# metadata: the folder holding the longest film owns the shorter video filed
# under it, provided that shorter file's own name speaks the film's name — a
# whole other film filed there stays a film, exactly as the engine's folder rule
# exempts it. A different year in the file's own name IS that other film
# speaking, so it vetoes the rule.
def longest_of(rows):
"""The longest video in each folder: that folder's feature."""
out = {}
for r in rows:
if r["dur"] > 0:
d = os.path.dirname(r["path"])
if d not in out or r["dur"] > out[d]["dur"]:
out[d] = r
return out
def not_disc(r):
"""A disc tree's innards belong to the tree, never to the folder's film."""
return not any(c in DISC_DIRS for c in r["path"].split("/"))
FEATURE_OF = longest_of([r for r in erows if r["path"] not in marks and not_disc(r)])
def under_a_feature(path, dur):
"""True when a folder above this file holds a longer film that this file's
own name names."""
d = os.path.dirname(os.path.dirname(path))
while len(d) > len(ROOT):
f = FEATURE_OF.get(d)
if f:
if f["dur"] < dur:
return False
theirs = (words_of(os.path.basename(f["path"]))
| words_of(os.path.basename(d)))
if not words_of(os.path.basename(path)) & theirs:
return False
y1 = re.search(r"(?:18|19|20)\d\d", os.path.basename(path))
y2 = re.search(r"(?:18|19|20)\d\d", os.path.basename(f["path"]))
return not (y1 and y2 and y1.group() != y2.group())
d = os.path.dirname(d)
return False
extras |= {r["path"] for r in erows
if r["dur"] > 0 and r["path"] not in marks | extras | tv_set
and not_disc(r) and under_a_feature(r["path"], r["dur"])}
film_rows = [r for r in erows if r["path"] not in marks | extras | tv_set]
buckets = collections.defaultdict(list)
for r in film_rows:
buckets[eng.key_of(r["title"])].append(r)
def res_label(r):
m = RES_RE.search(r["res"] or "")
if m:
n, kind = m.group(1).lower(), m.group(2).lower()
return "2160p" if n in ("2160", "4k") else f"{n}{kind}"
try:
w = int((r["res"] or "0x0").split("x")[0])
except (ValueError, IndexError):
w = 0
return "2160p" if w >= 3000 else ("1080p" if w >= 1400 else "SD")
def version_label(r):
bits = [res_label(r)]
vc = (r["vcodec"] or "").lower()
if vc in ("av1", "libaom", "libsvtav1"):
bits.append("AV1")
elif vc in ("vc1", "wmv3"):
bits.append("VC1")
if r["bytes"] > 40 * 2**30:
bits.append("REMUX")
if "imax" in r["path"].lower():
bits.append("IMAX")
return sanitize(" ".join(bits))
def group_name(items):
titles = collections.Counter(r["title"] for r in items)
ascii_t = [t for t in titles if t.isascii()]
title = (collections.Counter(ascii_t) if ascii_t else titles).most_common(1)[0][0]
years = [r["year"] for r in items if r["year"] != "?"] or \
[r["soft"] for r in items if r["soft"]]
year = collections.Counter(years).most_common(1)[0][0] if years else None
return sanitize(f"{title} ({year})" if year else title)
groups = []
for key, items in buckets.items():
for c in eng.dur_clusters(items):
g = {"rows": c, "base": group_name(c)}
known = [PLEX_T[r["path"]] for r in c if r["path"] in PLEX_T]
if known:
# one film, one name: a Plex match for any copy names every copy
t, yr = collections.Counter(known).most_common(1)[0][0]
g["plex"] = t
g["base"] = sanitize(f"{t} ({yr})" if ok_year(yr) else t)
groups.append(g)
clash = {b for b, n in collections.Counter(g["base"] for g in groups).items() if n > 1}
for g in groups:
if g["base"] in clash:
# two genuinely different films share title+year: separate folders, and
# say so in the name, because Jellyfin can only match one item per folder
g["base"] = sanitize(g["base"] + " - " + version_label(g["rows"][0]))
g["flag"] = "same title and year as another film"
moves, mkdirs, problems, readmes = [], set(), [], []
# --------------------------------------------------------------- unnamed groups
# A movie folder has to READ as a title. The engine's runtime veto says any file
# over 45 minutes that is not named like an extra is a film whatever it is called
# — true for Jellyfin, but a 118-minute documentary named "Extra 1" then produced
# a movie folder named "1", and folders named "1", "2", "3" are Jellyfin items
# with no identity whose numeric keys went on to capture every subtitle named
# "3_Eng.srt" in the library. A year proves a dated feature ("300 (2006)" is a
# real film), so the gate is: a year, or a word that is not release noise. With
# neither, the rows are bonus material until a human names them.
def named(g):
if re.search(r"\(\d{4}\)$", g["base"]):
return True
return any(w.isalpha() and len(w) >= 2 for w in eng.name_words(g["base"]))
unnamed = [g for g in groups if not named(g)]
groups = [g for g in groups if named(g)]
for g in unnamed:
for r in g["rows"]:
extras.add(r["path"])
problems.append(
f"UNNAMED film {rel(r['path'])} parses to {g['base']!r} with no year: "
f"bonus pile, not a folder — give the file a real name and re-plan")
group_of = {}
for g in groups:
g["dir"] = f"{ROOT}/{g['base']}"
for r in g["rows"]:
group_of[r["path"]] = g
dr = disc_root(r["path"])
if dr:
group_of[dr] = g
by_path = {r["path"]: r for r in erows}
# -------------------------------------------------------------- bookkeeping
claimed = {}
existing = set(rows_map := {r["path"] for r in rows})
at_target, booked_src = set(), set()
def book(src, dst, kind):
if src == dst:
at_target.add(src)
booked_src.add(src)
return
if os.environ.get("JF_TRACE") and (src in booked_src or dst.lower() in claimed):
why = "rebook" if src in booked_src else "collision"
problems.append(f"TRACE {why} kind={kind} src={rel(src)} dst={rel(dst)}")
if src in booked_src: # one source may only ever move once
problems.append(f"RE-BOOK {rel(src)} -> {rel(dst)} (already planned)")
return
if dst.lower() in claimed:
problems.append(f"COLLISION {rel(dst)}\n from {rel(src)}\n and {rel(claimed[dst.lower()])}")
return
claimed[dst.lower()] = src
booked_src.add(src)
comp = dst.split("/")
for i in range(1, len(comp)):
mkdirs.add("/".join(comp[:i]))
moves.append((src, dst, kind))
def uniq_dir(d, stem, ext, hint=""):
"""A destination that no other source already took."""
cand = f"{d}/{stem}{ext}"
if cand.lower() not in claimed:
return cand
n = 2
while True:
cand = f"{d}/{stem} {n}{ext}"
if cand.lower() not in claimed:
return cand
n += 1
def media_dir(p):
dr = disc_root(p)
return os.path.dirname(dr) if dr else os.path.dirname(p)
# ------------------------------------------------------------------ films
# film / disc moves: the largest copy carries the folder's own name, the rest
# become Jellyfin versions; a disc tree moves as a whole into the movie folder
def vext(p):
return ext_by.get(p) or os.path.splitext(p)[1].lower()
disc_children = collections.defaultdict(list)
for q in rows:
dr = disc_root(q["path"])
if dr:
disc_children[dr].append(q["path"])
for g in groups:
primary_taken = False
disc_kinds = set()
for r in sorted(g["rows"], key=lambda r: -r["bytes"]):
dr = disc_root(r["path"])
if dr:
inner = os.path.basename(dr).upper()
if inner in disc_kinds:
problems.append(
f"DISC-VERSION {g['base']}: extra disc tree {rel(dr)} cannot "
f"share the folder — discs have no multiple versions")
continue
disc_kinds.add(inner)
for src in disc_children[dr]:
tail = os.path.relpath(src, dr)
fold = disc_root(src, fold=False)
if fold != dr and os.path.dirname(src) != dr:
tail = os.path.basename(dr) + "/" + tail
book(src, f"{g['dir']}/{tail}", "disc-file")
continue
label = "" if not primary_taken else f" - {version_label(r)}"
dst = f"{g['dir']}/{g['base']}{label}{vext(r['path'])}"
n = 2
while dst.lower() in claimed:
dst = f"{g['dir']}/{g['base']}{label} {n}{vext(r['path'])}"
n += 1
r["dst"] = dst
primary_taken = True
book(r["path"], dst, "film")
g["primary"] = primary_taken
# ------------------------------------------------------------------ extras
EXTRA_KIND = [("trailers", r"trailer|teaser|preview|promo|tv spot|advert"),
("deleted scenes", r"deleted|alt scene|gag reel|blooper|outtake"),
("interviews", r"interview|cast and crew|session piece|commentary"),
("featurettes", r"featurette|behind the scenes|making of|documentary"
r"|heritage|archival|video archive|music video"),
("samples", r"sample"),
("shorts", r"\bshort\b|short film"),
("clips", r"\bclip\b|scene selection|montage|epilogue|isolated score"),
("other", r"menu|logo|gallery|still|trivia|storyboard|screen test"
r"|audition|test footage|watermark|workprint|poster"
r"|picture in picture|singalong|karaoke|original score")]
THEME_RE = re.compile(r"theme|opening song|ending song|soundtrack|score"
r"|\bost\b|music|song", re.I)
BACKDROP_RE = re.compile(r"backdrop|intro", re.I)
def extras_kind(path):
low = path.lower()
if THEME_RE.search(low):
return "theme-music"
if BACKDROP_RE.search(low):
return "backdrops"
for name, rx in EXTRA_KIND:
if re.search(rx, low):
return name
return "extras"
def attachable_title(path):
t, _y = eng.title_of(path)
return eng.key_of(t) if t and not eng.is_watermark(path, size_by.get(path, 0)) else None
for g in groups:
g["tops"] = {parts_of(r["path"])[0] for r in g["rows"]}
top_groups = collections.defaultdict(list)
for g in groups:
for t in g["tops"]:
top_groups[t].append(g)
group_by_key = collections.defaultdict(list)
for g in groups:
for r in g["rows"]:
group_by_key[eng.key_of(r["title"])].append(g)
group_by_dir = {}
for g in groups:
for r in g["rows"]:
group_by_dir.setdefault(media_dir(r["path"]), g)
dr = disc_root(r["path"])
if dr:
group_by_dir.setdefault(os.path.dirname(dr), g)
def group_words(g):
"""The words that identify a film group: its titles and the names of the
folders its files live in."""
if "words" not in g:
w = set()
if g.get("plex"):
w |= words_of(g["plex"])
for r in g["rows"]:
w |= words_of(r["title"])
w |= words_of(os.path.basename(media_dir(r["path"])))
g["words"] = w
return g["words"]
def score_words(text, groups):
"""Ranks film groups by how many of their words a path carries."""
words = words_of(text)
out = [(len(words & group_words(g)), g) for g in groups]
out.sort(key=lambda x: -x[0])
return out
def group_for_orphan(path):
"""Which movie folder an extra/sidecar belongs to. A collection folder holding
several films must be resolved by WORDS, never by position: taking the first
film of the folder attached every loose subtitle in a box set to whichever
movie happened to sort first."""
stem = os.path.splitext(os.path.basename(path))[0]
key = attachable_title(path)
if key and group_by_key.get(key):
cands = group_by_key[key]
top = parts_of(path)[0] if len(parts_of(path)) > 1 else ""
same_top = [g for g in cands if any(parts_of(r["path"])[0] == top
for r in g["rows"] if len(parts_of(r["path"])) > 1)]
pool = same_top or cands
return pool[0] if len(pool) == 1 else score_words(stem, pool)[0][1]
gtop = parts_of(path)[0] if len(parts_of(path)) > 1 else ""
pool = top_groups.get(gtop, [])
if len(pool) > 1: # a loose collection folder (box set, bonus
n, g = score_words(stem + " " + os.path.dirname(path), pool)[0]
return g if n >= 2 else None # album) is resolved by words, not position
if len(pool) == 1:
return pool[0]
d = os.path.dirname(path)
while d and d != ROOT:
if d in group_by_dir:
return group_by_dir[d]
d = os.path.dirname(d)
return None
tv_tops = {parts_of(p)[0] for p in tv_set}
def film_folder_for(path):
"""A movie folder for a lone feature that has no group: derived from its own
title when a year proves it, else None (goes to the review pile)."""
t, y = eng.title_of(path)
if not t or y == "?":
return None
d = uniq_dir(ROOT, sanitize(f"{t} ({y})"), "", "")
mkdirs.add(d)
return d
for p in sorted(extras):
if disc_root(p):
continue
if len(parts_of(p)) > 1 and parts_of(p)[0] in tv_tops:
continue # a special inside a TV set: staged with it
g = group_for_orphan(p)
ext = ext_by.get(p, os.path.splitext(p)[1].lower())
stem = os.path.splitext(os.path.basename(p))[0]
if g is None:
solo = film_folder_for(p)
if solo:
dst = f"{solo}/{sanitize(stem)}{ext}"
problems.append(f"REVIEW lone feature {rel(p)}: gets its own folder "
f"{os.path.basename(solo)}")
else:
dst = f"{ROOT}/_unsorted bonus/{extras_kind(p)}/{sanitize(stem)}{ext}"
problems.append(f"NO-FILM extra {rel(p)}")
else:
dst = f"{g['dir']}/{extras_kind(p)}/{sanitize(stem)}{ext}"
book(p, uniq_dir(os.path.dirname(dst), os.path.basename(dst)[:-(len(ext) or 1)],
ext, os.path.dirname(p)), "extra")
# Disc trees the engine never saw (a MakeMKV scratch folder, an AACS remnant):
# they belong to the film of the folder they sit in, kept as a named subtree.
for q in rows:
pth = q["path"]
dr = disc_root(pth)
if not dr or pth in by_path or pth in booked_src:
continue
g = group_by_dir.get(dr) or (top_groups[parts_of(pth)[0]][0]
if len(top_groups.get(parts_of(pth)[0], [])) == 1 else None)
if g is None:
continue
inner = os.path.basename(dr)
problems.append(f"REVIEW disc remnant {rel(dr)} moves inside {g['base']} "
f"(a second master: compare before keeping both)")
for src in disc_children[dr]:
if src not in booked_src:
book(src, f"{g['dir']}/{inner}/{os.path.relpath(src, dr)}", "disc-file")
# ----------------------------------------------------------------- sidecars
media_by_dir = collections.defaultdict(list)
for r in erows:
if r["path"] in marks:
continue
dst = r.get("dst")
if dst:
media_by_dir[media_dir(r["path"])].append(r)
def sidecar_target(path, e):
d = os.path.dirname(path)
stem = os.path.splitext(os.path.basename(path))[0]
if d == ROOT:
# A file loose in the library ROOT names no film. media_by_dir[ROOT] is the
# set of root-level MOVIE files, and scoring a stray name against it
# attached every root-level leftover to whichever root-level film was
# largest — the plan sent ".DS_Store" into "Titanic (1997)/other/" that
# way. A root-level stray is nobody's sidecar: it goes to the review pile.
return None, None
cands = media_by_dir.get(d, [])
if not cands:
nd = os.path.dirname(d) # a subs/ or Extras/ folder beside the media
while nd and nd != ROOT and not media_by_dir.get(nd):
nd = os.path.dirname(nd)
if media_by_dir.get(nd):
if nd != d: # a NAMED subfolder (Subs/, Extras/, the
g = group_for_orphan(path) # box set's own root) carries evidence
dirs = {os.path.dirname(r.get("dst", "")) for r in media_by_dir[nd]}
if g is None or g["dir"] not in dirs:
# ...but only a STRONG word match outranks the folder the file
# sits in. A weak one is an accident: "3_Eng.srt" scored 1
# against a junk group named "3" and escaped its own movie's
# Subs/ folder, while "03 xXx 3 Return Of Xander Cage" scores 4
# and genuinely belongs to the other film of the set.
n = score_words(stem, [g])[0][0] if g else 0
if n >= 2:
return (g, None)
cands = media_by_dir[nd]
if not cands:
g = group_for_orphan(path)
return (g, None) if g else (None, None)
if len(cands) > 1:
w = [t for t in eng.norm(stem).split() if t not in NOISE]
cands = sorted(cands, key=lambda r: (
-len([t for t in eng.norm(os.path.basename(r["path"])).split() if t in w]),
-r["bytes"]))
return (None, cands[0])
for q in rows:
p, e = q["path"], q["ext"]
if p in by_path or p in extras or p in marks or is_qb(p) or disc_root(p):
continue
if os.path.basename(p).startswith("._"):
# macOS AppleDouble resource forks: a few hundred bytes carrying a name
# COPIED FROM ANOTHER FILM. Word-matching them to "their" film scattered
# dead files through four movie folders (and they are video extensions in
# the AppleDouble case, so they also skipped every other filter). They
# claim nothing: one pile, and they stay in it — nothing is deleted here.
book(p, uniq_dir(f"{ROOT}/_macos sidecars", sanitize(os.path.basename(p))[:120],
"", os.path.dirname(p)), "junk")
problems.append(f"JUNK AppleDouble sidecar {rel(p)} (macOS resource fork)")
continue
if e == ".iso": # a DVD image: Jellyfin plays it as a movie file
g = group_for_orphan(p)
stem = os.path.splitext(os.path.basename(p))[0]
if g is not None:
book(p, uniq_dir(g["dir"], g["base"], e, stem) if g.get("primary") is not None
else f"{g['dir']}/{g['base']}{e}", "iso")
else:
solo = film_folder_for(p)
if solo:
book(p, f"{solo}/{sanitize(stem)}.iso", "iso")
problems.append(f"REVIEW disc image {rel(p)}: DVD rip, alone in its "
f"folder -> {os.path.basename(solo)}")
else:
book(p, f"{ROOT}/_unsorted files/{sanitize(os.path.basename(p))}", "orphan")
problems.append(f"NO-FILM disc image {rel(p)}")
continue
if e in VIDEO_EXT:
# A video the engine never probed (unreadable header) and not an AppleDouble
# remnant: the plan does not guess at it, so it stays where it is.
continue
stem = os.path.splitext(os.path.basename(p))[0]
g, media = sidecar_target(p, e)
if media is not None:
base = os.path.splitext(os.path.basename(media["dst"]))[0]
d = os.path.dirname(media["dst"])
elif g is not None:
base, d = g["base"], g["dir"]
else:
# Nothing names this file's film. NEVER scatter it: an archive set or an
# album tree must move as the unit it is, so the whole loose folder goes
# to the review pile and a film folder's own leftovers stay with it.
ps = parts_of(p)
if len(ps) < 2:
dst = f"{ROOT}/_unsorted files/{sanitize(stem)}{e}"
elif len(top_groups.get(ps[0], [])):
dst = f"{top_groups[ps[0]][0]['dir']}/other/{'/'.join(sanitize(c) for c in ps[1:])}"
else:
dst = f"{ROOT}/_unsorted files/{'/'.join(sanitize(c) for c in ps)}"
book(dst=dst, src=p, kind="orphan")
problems.append(f"NO-OWNER sidecar {rel(p)}")
continue
if e in IMG_EXT:
low = eng.norm(stem)
word = next((w for w in IMG_PRIMARY if re.search(rf"\b{w}\b", low)), None)
kind = "poster" if word else next(
(IMG_OTHER[w] for w in IMG_OTHER if re.search(rf"\b{w}\b", low)), None)
if re.search(r"\btheme\b", low):
dst = f"{d}/theme{e}"
else:
dst = f"{d}/{base}-{kind or 'art'}{e}" if kind else f"{d}/{sanitize(stem)}{e}"
book(p, uniq_dir(os.path.dirname(dst), os.path.basename(dst)[:-(len(e) or 1)],
e, os.path.dirname(p)), "image")
elif e in NFO_EXT:
book(p, uniq_dir(d, base, ".nfo", os.path.dirname(p)), "nfo")
elif e in SUB_EXT | AUDIO_EXT:
toks = [t for t in re.split(r"[._\-\s]+", stem) if t]
langs = [t.lower() for t in toks if t.lower() in LANGS]
flags = [t.lower() for t in toks if t.lower() in FLAGS]
keep = [sanitize(t) for t in toks
if t.lower() not in langs + flags + list(NOISE)
and not re.fullmatch(r"\d{3,4}p?|cd\d+|\d{4}|\d+ch|\d+.\d", t.lower())]
bits = [base] + ([sanitize(" ".join(keep[:4]))] if keep else []) + langs + flags
if e in SUB_EXT:
# subtitles: <movie>/srt[/language]
d = f"{d}/srt"
pd = os.path.basename(os.path.dirname(p))
if SUB_KEEP_RE.match(pd) and not SUB_DIR_OK.match(pd):
d = f"{d}/{sanitize(pd)}" # a real language folder survives
elif e in AUDIO_EXT and AUDIO_DIR_RE.search(os.path.dirname(p)):
d = f"{d}/theme-music" # bonus audio: one canonical folder
pd = os.path.basename(os.path.dirname(p))
if not MUSIC_TOP_RE.search(pd): # an album folder keeps its name
d = f"{d}/{sanitize(pd)}"
else:
pd = os.path.basename(os.path.dirname(p))
if not AUDIO_DIR_OK.match(pd):
d = f"{d}/{sanitize(pd)}"
stem2 = ".".join(bits) or sanitize(os.path.basename(p))
book(p, uniq_dir(d, stem2, e, os.path.basename(p)), "track")
else:
book(p, uniq_dir(f"{d}/other", sanitize(os.path.basename(p))[:120], e,
os.path.dirname(p)), "misc")
readmes.append(f"README in {ROOT}: this library was relaid out into Jellyfin's "
"one-folder-per-movie layout; `_release-group watermarks/`, "
"`_unsorted files/` and `_unsorted bonus/` hold files no film claims.")
for key in tv_keys:
readmes.append(f"README in {ROOT}/{sanitize('_TV needs a TV library - ' + key)}: "
"these are TV episodes found in the MOVIES library; move the folder "
"to a TV library source, else Jellyfin groups them wrongly as one "
"movie.")
# --------------------------------------------------------------------- TV
for key in tv_keys:
items = eps[key]
year = collections.Counter(r["year"] for r in items if r["year"] != "?")
yr = year.most_common(1)[0][0] if year else None
base = f"_TV needs a TV library - {sanitize(key)}" + (f" ({yr})" if yr else "")
tops = {parts_of(r["path"])[0] for r in items}
members = sorted([r["path"] for r in items]
+ [q for q in extras | marks if len(parts_of(q)) > 1
and parts_of(q)[0] in tops],
key=lambda q: q)
for src in members:
rel_parts = rel(src).split("/")
tail = "/".join(sanitize(c) for c in rel_parts[:-1] + [rel_parts[-1]])
book(src, f"{ROOT}/{base}/{tail}", "tv")
for p in sorted(marks):
stem, ext = os.path.splitext(os.path.basename(p))
book(p, uniq_dir(f"{ROOT}/_release-group watermarks", sanitize(stem)[:120], ext,
os.path.dirname(p)), "watermark")
for q in rows:
if is_qb(q["path"]):
problems.append(f"SKIPPED qBittorrent partial {rel(q['path'])}")
book(q["path"], q["path"], "qb-partial") # stays: resume data
# --------------------------------------------------------------- debug hook
if os.environ.get("JF_ISO"):
x = ROOT + "/High Anxiety (Mel Brooks) [1977] nordic subs DVDrip/High Anxiety [1977].iso"
print("attachable", attachable_title(x), "| dir", os.path.dirname(x) in group_by_dir,
"| g", group_for_orphan(x) and group_for_orphan(x)["base"])
raise SystemExit
if os.environ.get("JF_DEBUG"):
import json
unp = sorted({r["path"] for r in rows} - {m[0] for m in moves} - at_target)
print("at_target", len(at_target))
print("kinds-unplanned", collections.Counter(os.path.splitext(u)[1] for u in unp).most_common(8))
for up in unp[:20]:
d = os.path.dirname(up)
print("UNP", up[:90], "| mediadir?", d in media_by_dir, "| by_path", up in by_path,
"| extras", up in extras, "| discroot", disc_root(up), "| ext", os.path.splitext(up)[1])
# -------------------------------------------------------------- validation
srcs = [m[0] for m in moves]
dup_src = {s for s, n in collections.Counter(srcs).items() if n > 1}
for s in dup_src:
problems.append(f"DOUBLE-CLAIM source {rel(s)}")
all_src = set(srcs)
untouched = {r["path"] for r in rows} - all_src
bad_names, into_alive, lost = [], [], []
for src, dst, kind in moves:
for comp in dst[len(ROOT) + 1:].split("/"):
if (BAD_CHARS.search(comp) or comp != comp.strip(". ") or not comp
or len(comp) > 200):
bad_names.append(f"{kind}: {comp!r} in {rel(dst)}")
if dst in untouched: # something staying put is there
into_alive.append(f"{rel(src)} -> {rel(dst)} (occupied by a file that stays)")
if dst.lower() in {s.lower() for s in untouched}:
into_alive.append(f"{rel(src)} -> {rel(dst)} (case-collides with a survivor)")
if dst.startswith(src + "/") and src in all_src:
lost.append(f"{rel(src)} moves into its own subtree")
# every accounted-for source must exist exactly once in the plan
covered = {r["path"] for r in rows}
not_planned = sorted(covered - all_src - at_target)
with open(PLAN, "w") as out:
out.write("action\tsrc\tdst\tkind\n")
for d in sorted(mkdirs, key=lambda q: (q.count("/"), q)):
if d.startswith(ROOT + "/") and d != ROOT:
out.write(f"mkdir\t\t{d}\tdir\n")
for p in sorted(booked_src - {m[0] for m in moves}):
out.write(f"keep\t{p}\t{p}\n")
for src, dst, kind in sorted(moves, key=lambda m: m[0]):
out.write(f"move\t{src}\t{dst}\t{kind}\n")
kinds = collections.Counter(k for _s, _d, k in moves)
films_n = sum(1 for _s, _d, k in moves if k == "film")
folders = len({os.path.dirname(d) for _s, d, k in moves if k in ("film", "disc-file")})
with open(EXCEPTS, "w") as out:
out.write(f"groups={len(groups)} moves={len(moves)} folders={folders}\n")
out.write(f"kinds={dict(kinds)}\n")
out.write(f"at-target-already={len(at_target)} unplanned={len(not_planned)}\n")
out.write("\n=== WHAT THE EXECUTOR MUST ALSO WRITE ===\n")
out.write("\n".join(sorted(set(readmes))) + "\n")
out.write("\n=== PROBLEMS ===\n")
out.write("\n".join(sorted(set(problems))) + "\n")
out.write("\n=== BAD NAMES ===\n" + "\n".join(sorted(set(bad_names))) + "\n")
out.write("\n=== TARGETS OCCUPIED BY SURVIVORS ===\n"
+ "\n".join(sorted(set(into_alive))) + "\n")
out.write("\n=== LEFTOVER SOURCES (first 60) ===\n"
+ "\n".join(rel(p) for p in not_planned[:60]) + "\n")
print(f"groups={len(groups)} folders={folders} moves={len(moves)} "
f"mkdirs={len(mkdirs)} film-files={films_n}")
print(f"kinds={dict(kinds)}")
print(f"problems={len(set(problems))} bad-names={len(set(bad_names))} "
f"occupied={len(set(into_alive))} double-claim={len(dup_src)}")
print(f"left-unmoved={len(not_planned)}")
print(f"tv-keys={len(tv_keys)} extras={len(extras)} marks={len(marks)} "
f"disc-trees={len(disc_children)}")
+572
View File
@@ -0,0 +1,572 @@
#!/usr/bin/env python3
r"""The fleet's release-filename title parser: the grouping authority the relayout
plan uses for every file the media server never matched.
report: media-titles.py report [--probed TSV] [--out TSV]
the similar-copies census (DUP / REVIEW / EP-DUP groups) over the
ffprobe census scripts/media-census.py writes.
parse: media-titles.py parse NAME [NAME…]
what title_of() makes of a name — the way to check a naming accident
before it merges two films into one folder.
scripts/media-plan.py imports this file; over there the paths are arguments.
Method — three independent signals, so a naming accident cannot merge two films:
* franchise KEY : first two significant title tokens. Stable across parts of a
collection ("resident evil", "jump street") but NOT film-specific — it only
selects candidates.
* DURATION : the identity check. Same master encodes differ by at most a
couple of seconds; different films / cuts do not collide.
* SIZE SPREAD + YEAR: decide the tier. Two files same size AND same duration
are byte-twins (the hash pass confirms); same duration but different sizes are
two different releases of one film (deletable redundancy).
TV episodes (SxxEyy in the name) are grouped separately: their uniform runtime
would otherwise collapse a whole series into one "duplicate" group.
"""
import argparse
import collections, os, re, sys, unicodedata
# Library root: the folder directly under this one names the film, and every
# folder below it is structure (Extras/, Featurettes/, Disc 1/, BDMV/). Tests pass
# their own; nothing here reads media, only the census TSVs about it.
ROOT = os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies")
# Release-tag vocabulary: never part of a title, so the title stops here.
JUNK = {"5.1", "7.1", "2.0", "1.0", "6.1", "4.0", "extras"}
JUNK_RE = re.compile(
r"^("
r"\d{3,5}p|\d{1,2}k|uhd|fhd|imax|x ?26[45]|hevc|h ?26[45]|avc|av1|x ?av1|vp[89]"
r"|divx|xvid|mpeg-?2?|\d{1,2}-?bit|hdr ?10\+?|hdr|dolby ?vision|dvh?|sdr|hybrid"
r"|re-?mux|bd-?re-?mux|blu-?ray|bdrip|brrip|bdmux|bd|web-?(rip|dl)?|hdtv|hdrip"
r"|dvd-?(rip|r|scr)?|hdcam|telesync|screener|ppv|re-?encode|extras?"
r"|ddp? ?\d(\.\d)?|dd\d(\.\d)?|dts-?hd(\.ma)?|dts|truehd|atmos|e-?ac-?3?|ac-?3"
r"|aac\d?(\.\d)?|lpcm|flac|opus|mp3|pcm|\d\.\d(\.\d)?|5 ?1|7 ?1|2 ?0|1 ?0|6 ?1|4 ?0"
r"|subs?-?(it|en|eng|ita)?|subbed|dual(-?audio)?|multi|dub(bed)?"
r"|ita|eng|en|it|de|fr|es|pl|jp|jap|lat|esp|ger|fre|rus|zho|hin|tam|tel"
r"|repack|proper|internal|limited|unrated|extended|director'?s|cut|remaster(ed)?|uncut"
r"|special|edition|version|crc32|sha1|sfv|par2|nfo|nzb|mux|complete|unpack"
r"|certified|scene|p2p|part\d*|cd\d+|disc\d+|disk\d+|vob|idx|srt|ass|ssa|sup"
r"|m3u|cue|log|txt|mkv|mp4|avi|imdb\d[\d.]*|yts(\.\w+)?|rarbg|collection"
r")$", re.I)
RES_RE = re.compile(r"^\d{3,5}p$|^\d{1,2}k$|^\d+x\d+$", re.I)
YEAR_RE = re.compile(r"^(19|20)\d{2}$")
SEP_RE = re.compile(r"[._\s\-\[\](){}+!~,;:&@#/*'\"`´|]+")
IDX_RE = re.compile(r"^\d{1,3}[a-z]?$")
WORD_RE = re.compile(r"[A-Za-z0-9']+$")
EP_RE = re.compile(r"(?i)(?<![a-z0-9])(s(?:eason[\s._-]*)?\d{1,2}[\s._-]*e(?:p(?:isode)?[\s._-]*)?"
r"\d{1,3}|s\d{1,2}\s?e\d{1,3}|\d{1,2}x\d{2,3}|part[\s._-]*\d{1,2})")
STOP = {"a", "an", "the", "il", "la", "le", "i", "un", "une", "el", "los", "las", "de",
"del", "du", "di", "e", "ed", "y", "and", "et", "for", "of", "to", "in", "on",
"vs", "avec", "con"}
def norm(s):
s = unicodedata.normalize("NFKD", s)
return "".join(c for c in s if not unicodedata.combining(c)).lower().strip()
def fold_sep(s):
"""norm(), plus: every character that is left over is not a letter, digit or
apostrophe, so it is a SEPARATOR whatever codepoint it hides behind — not part
of the word. Release groups file titles with lookalikes for punctuation the
filesystem forbids: "V ∕ H ∕ S ∕ 2" carries U+2215 DIVISION SLASH, "S꞉ Viral"
U+A789, and en dashes hide in "Star Wars – A New Hope". NFKD does NOT fold
those (no decomposition mapping), and the scrubber inside title_of DELETES
what it does not recognise, so five different V/H/S films squashed into the one
title "vhs2" and would have shared one folder. Deleting is the wrong operation:
they separate. Accents are gone by then — norm() removed the combining marks —
so "Amélie" stays "amelie" and never splits into "ame lie"."""
return " ".join("".join(c if c.isalnum() or c == "'" else " "
for c in norm(s)).split())
def title_of(path):
"""(title, year) from the basename.
Two passes. (1) Tokenize; every 4-digit year becomes a sentinel in place.
(2) Find the boundary: the first release-noise token (resolution / codec /
source / audio / language / group vocabulary) ends the title, so trailing
tags can never leak in. (3) Collect up to that boundary: leading index
numbers ("03."), leading year sentinels ("1940 - Fantasia") and leading noise
are skipped while nothing is collected; once a real word appears a sentinel
CLOSES the title. A title sitting BEFORE the year sentinel ("300.2006.…" or
"12. Monkeys. 1995") is a numeric title and is prepended to what follows.
"""
# fold_sep BEFORE tokenizing, so a lookalike separator separates (see its
# docstring): the tokens below are then plain alnum/apostrophe words.
name = fold_sep(os.path.splitext(os.path.basename(path))[0])
raw = [x for x in SEP_RE.split(name) if x]
years = [t for t in raw if YEAR_RE.match(t)]
two_year = len(years) > 1 # "2001…1968": one of them is the title
toks, year = [], None
for t in raw:
if YEAR_RE.match(t):
year = t # last year wins
toks.append("\x00")
else:
toks.append(t)
stop, seen_word = len(toks), False
for i, t in enumerate(toks):
if t == "\x00":
continue
c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
if (not c or RES_RE.match(c) or JUNK_RE.match(c) or c in JUNK
or not WORD_RE.match(c)) and not (IDX_RE.match(t) and not seen_word):
stop = i
break
if c.isalpha():
seen_word = True
# A leading number is a track index ("01. Bye Bye Baby", "05. Casablanca") unless
# it is three digits or more ("007", "000", "1408") or the name carries no year
# at all ("12 Monkeys", "831"). A 1-2 digit prefix beside a year is an index;
# the rare real title that looks like one ("0 Things I Hate About You (1999)")
# loses its zero to the parser — Plex's matched title overrides it anyway.
lead = re.split(r"[\s._-]", name, maxsplit=1)[0]
track_idx = bool(re.fullmatch(r"\d{1,2}[a-z]?", lead)) and bool(year)
keep, num_title = [], None
for t in toks[:stop]:
if t == "\x00":
if keep:
break # year closes the title
if two_year and t is not None and years[0] != year:
# two DIFFERENT years ("2001…1968"): the leading one is the title;
# "1940 - Fantasia (1940)" repeats one year, so it is not a title
num_title = num_title or years[0]
continue # leading year: not the title
if track_idx and IDX_RE.match(t) and not keep and t == lead:
continue # "03." / "1c." ordering
c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
if not keep and not c.isalpha():
num_title = num_title or (c or None)
continue # numeric title candidate
keep.append(c)
if len(keep) >= 8:
break
keep = [k for k in keep if not re.fullmatch(r"(disc|disk|cd|dvd|part)\d*", k)]
title = " ".join(keep)
if not keep:
# nothing collected: a pure-numeric leading token is the title even when it
# sits where the release year sits ("1917.2019.2160p…"), and "300.2006…" too
for t in raw:
c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
if re.fullmatch(r"(?:[12]\d{3}|\d{1,3})", c) and (c != year or two_year):
return c, year or "?" # never "2160p" / "1080": they are not pure digits
if (num_title and title and not keep[0].isdigit()
and (two_year or not track_idx)):
# a leading number that only echoes the release year ("1917.2019.2160p…") is
# not a title; one that stands alone ("300.2006…", "1900") is
title = f"{num_title} {title}"
title = title or num_title or norm(name).replace("\x00", " ").strip()[:40]
return title, year or "?"
# Bonus-material vocabulary: names that say nothing about the film, so an extras
# file otherwise pairs with an extras file of another movie. "Short", "Spot" and
# "Clip" are deliberately absent — they are film titles (Short Circuit, Spot,
# Clips) — and the running-time guard in is_extra() carries the rest.
SEP_CLS = r"[\s._-]+"
def _bonus_rx(word):
return r"(?<![a-z0-9])" + re.sub(r"[ \-]+", lambda _: SEP_CLS, word.lower()) + r"(?![a-z0-9])"
EXTRA_WORDS = [
"trailer", "teaser", "featurette", "featurettes", "promo", "promotional",
"tv spot", "advert", "commercial", "behind the scenes", "making of",
"deleted scene", "deleted scenes", "deleted footage", "gag reel", "blooper",
"blooper reel", "bloopers", "outtake", "outtakes", "scene selection", "menu",
"menus", "gallery", "still gallery", "stills", "poster", "posters",
"montage", "preview", "previews", "epilogue", "visual effects",
"music video", "singalong", "karaoke", "trivia", "storyboard", "workprint",
"test footage", "screen test", "audition", "cast and crew", "isolated score",
"picture in picture", "original score", "session piece", "sample",
"watermark",
]
# Words that describe bonus material which can run LONG (a feature-length
# commentary-track cut, a documentary about the film). Trusted only below
# LONG_BONUS_MAX, so a 40-minute "commentary" release stays a film.
EXTRA_LONG_WORDS = ["interview", "interviews", "commentary", "video archive",
"archival", "documentary"]
EXTRA_NAME_RE = re.compile(
"|".join([_bonus_rx(w) for w in EXTRA_WORDS] + [r"scene[\s._-]?\d{1,3}"]), re.I)
EXTRA_LONG_NAME_RE = re.compile(
"|".join(_bonus_rx(w) for w in EXTRA_LONG_WORDS), re.I)
# Bonus-folder vocabulary. fullmatch = the folder IS an extras subtree; a bare
# search = it only mentions extras ("... + Extras" box sets hold feature films).
EXTRA_FOLDER_RE = re.compile(
r"(?:extras?|bonus(?:[\s._-]material)?|specials?|special[\s._-]featurettes?|"
r"featurettes?|deleted(?:[\s._-](?:scenes?|footage))?|trailers?|"
r"making[\s._-]of|behind[\s._-]the[\s._-]scenes|shorts?|interviews?|"
r"archiv(?:al|es)|video[\s._-]archive)", re.I)
VIDEO_SUFFIX = (".mkv", ".mp4", ".avi", ".m4v", ".mov", ".m2ts", ".mpg", ".webm")
# Anything longer than this is a feature film whatever it is called: a 118-minute
# file named "The Menu" or "Trailer" is a movie, not bonus material.
SHORT_MAX = 45 * 60
# Long-shaped bonus vocabulary is only believed below this: a 40-minute
# commentary cut is an alternate version of the film, not a featurette.
LONG_BONUS_MAX = 25 * 60
def name_words(x):
"""Bare words of a file or folder name: no release metadata, no year."""
x = os.path.splitext(x)[0] if x.lower().endswith(VIDEO_SUFFIX) else x
return [w for w in SEP_RE.split(fold_sep(x))
if w and not YEAR_RE.match(w) and not RES_RE.match(w)
and not JUNK_RE.match(w) and w not in JUNK]
def mentions(short, long_):
"""True when one word list appears in the other, anywhere: a release names
the film at the front ("Alien (1979) - Trailer") or in the middle
("Movies/Alien (1979)/Extra - Logo.mkv"), so a start-of-string rule fails."""
a, b = sorted([short, long_], key=len)
if not a:
return False
return any(b[i:i + len(a)] == a for i in range(len(b) - len(a) + 1))
def folder_is_bonus(folder):
"""True when a folder name carries bonus vocabulary as WHOLE words: the
vocabulary is matched against a padded word string, so "Extras" decides and
a word that merely CONTAINS one ("Extraordinary", "The Extras") does not."""
return bool(EXTRA_FOLDER_RE.search(" " + " ".join(SEP_RE.split(norm(folder))) + " "))
def is_extra(path, title="", dur=0.0, root=ROOT, matched=False):
"""True for bonus material, which must never be counted as a film: a file
called "Trailer", "Menu", "Sample" or "Gallery" carries no film identity, so
extras otherwise pair with each other across the whole library.
Evidence, strongest first:
* RUNTIME VETOES EVERYTHING. The film "The Menu", the film "Trailer", the
franchise "Trailer Park Boys" and "ET — The Extra-Terrestrial" all carry
bonus vocabulary and run 100+ minutes: anything over SHORT_MAX is a film —
unless it sits in an explicit bonus folder and metadata never matched it.
Bonus words that describe long-form material (commentary, documentary,
interview) are believed only
under LONG_BONUS_MAX, so a feature-length commentary cut stays a film.
* The file's OWN name carries bonus vocabulary -> bonus.
* ANY folder of the path below the film's own root folder carries bonus
vocabulary ("Alien (1979)/Extras/", ".../Featurettes/Deleted Scenes/",
"Angel's Egg (1985)/04b. Angel's Egg Extras/") -> bonus. The ROOT folder
is exempt because it names the film, and a film may be titled with a
bonus word; and a file whose own name IS that folder's film name is the
feature, not the bonus ("Ghost in the Shell Movies/Extras/
Ghost in the Shell - CM3.mp4" is a bonus, but a whole other film filed
there stays a film).
Bonus vocabulary is matched as whole words only, so the film
"Angel's Egg - Extra - Logo" inside "04b. Angel's Egg Extras" is reached by
the folder rule, and "…Extras" as a fragment of a word is never evidence.
"""
parts = [q for q in path.split("/") if q]
if not parts:
return False
name = (os.path.splitext(parts[-1])[0]
if parts[-1].lower().endswith(VIDEO_SUFFIX) else "")
# A bonus folder anywhere BELOW the film's own folder decides for its
# contents; the film's own folder is the film's name, whatever words it
# contains, so it is exempt. It is the first component AFTER ROOT's own
# components — those are the mount path, not folders of the library.
head = len([q for q in root.split("/") if q]) + 1
bonus_folder = False
for folder in parts[head:-1]:
if folder_is_bonus(folder):
bonus_folder = not mentions(name_words(name), name_words(folder))
break
if dur and dur > SHORT_MAX:
# Runtime vetoes vocabulary, with ONE exception: an explicit bonus folder
# and no metadata match. A 90-minute "Feature Length Rough Cut" sitting in
# a movie's Featurettes/ is a disc extra, and nothing in the library claims
# it as a film. Long-form bonus words still describe a film, so a
# feature-length commentary cut or documentary keeps its runtime veto.
if bonus_folder and not matched and not EXTRA_LONG_NAME_RE.search(name):
return True
return False
if EXTRA_NAME_RE.search(name):
return True
if dur and dur <= LONG_BONUS_MAX and EXTRA_LONG_NAME_RE.search(name):
return True
return bonus_folder
# Release-group watermark clips: a group drops a tiny ad ("ETRG.mp4",
# "RARBG.com.mp4") into many different films, so its name carries no film
# identity at all and it would otherwise pair with itself across the library.
WATERMARK_MAX = 20 * 2**20
WATERMARK_RE = re.compile(r"(?i)^(rarbg(\.com)?|etrg|tgx|fbt|cpbg|yify|yts|"
r"rarbg\.com|playnomore|imprs|gauntlet|dfx|macmote|"
r"psi-axiz|acm|tlf|sa8|ebp|gtv|w4g|don|aAF|enjoy|hd4u)$")
def is_watermark(path, size):
"""True for a release group's own watermark video, not a film or an extra."""
return (size < WATERMARK_MAX
and bool(WATERMARK_RE.match(os.path.splitext(os.path.basename(path))[0])))
def key_of(title):
"""Franchise key: first two alphabetic tokens (numeric-only title = itself).
Deliberately NOT film-specific — it only selects candidates for the duration
check, so a two-token collision (two films starting "star trek") cannot by
itself create a duplicate group."""
toks = [t for t in title.split() if t]
words = [t for t in toks if not t.isdigit()]
if not words:
return " ".join(toks)
sig = [t for t in words if t not in STOP] or words
return " ".join(sig[:2])
def ep_of(path):
"""(series, episode-token) when the basename carries SxxEyy / 1x04 / Part N.
The series is the name BEFORE the marker. The box-set convention writes
"Monk - S05E03 - Mr. Monk and the Big Game", and the episode title that
follows the marker is not the show's name: keying on the whole title gave
every episode of one series its own group, so a 124-episode box set never
reached the size that calls something a series and each episode was filed
as its own film. A file named only by its marker has nothing before it, so
its whole title stands in."""
name = os.path.splitext(os.path.basename(path))[0]
m = EP_RE.search(name)
if not m:
return None
head = name[:m.start()].strip(" ._\u2013-")
series = title_of(head)[0] if head else None
return series or title_of(path)[0], norm(m.group(1)).replace(" ", "")
def base_words(title, key):
"""Title tokens beyond the franchise key — the FILM identity part.
"evil dead rise" under key "evil dead" -> {rise}; "28 days later" -> {}."""
kt = set(key.split())
return frozenset(t for t in title.split() if t not in kt and t not in STOP
and not t.isdigit() and not YEAR_RE.match(t))
def folder_year(path):
"""Soft year evidence: a 19xx/20xx token in any parent directory name
(collection folders are written both "(2002)" and "[2009]"; a range such as
"(1998-2000)" deliberately yields nothing)."""
for d in reversed(os.path.dirname(path).split("/")):
ys = [y for y in re.split(r"[^0-9]+", d) if YEAR_RE.match(y)]
if len(ys) == 1:
return ys[0]
return None
def same_film(a, b):
"""Two files are copies of ONE film only when the film-identity tokens agree
(one side may carry none) AND no year evidence contradicts them. The key says
the franchise matches; this says the actual film matches — it is what keeps
'Evil Dead' (2013) apart from 'Evil Dead Rise' (2023) and the four Austin
Powers apart when their runtimes happen to agree."""
ba, bb = a["base"], b["base"]
if ba and bb and ba != bb:
return False
ya = a["year"] if a["year"] != "?" else a["soft"]
yb = b["year"] if b["year"] != "?" else b["soft"]
if ya and yb and ya != yb:
return False
return True
def dur_close(a, b):
if a <= 0 or b <= 0:
return True # unknown duration: let titles decide
return abs(a - b) <= max(2.0, 0.003 * max(a, b))
# Blu-ray folder trees: ffprobe only sees the clip it is pointed at, so each tree
# is folded into ONE entry, described by its longest main-feature clip (measured
# with ffprobe on the largest BDMV/STREAM file) and sized by the whole tree.
BDMV_MAIN = {
"Avengers.Age.of.Ultron.2015.2160p.BluRay.HEVC.TrueHD.7.1.Atmos-TERMiNAL":
dict(dur=8478.2, vcodec="hevc", res="3840x2160", pix="yuv420p10le", fps=23.976),
"Full.Metal.Jacket.1987.2160p.BluRay.HEVC.DTS-HD.MA.5.1-EATDIK":
dict(dur=6993.3, vcodec="hevc", res="3840x2160", pix="yuv420p10le", fps=23.976),
}
def load(probed):
"""Probe rows -> film entries. AppleDouble (._*) sidecars are dropped; each
BDMV tree collapses to one entry carrying the tree's total size."""
trees = {}
rows = []
for line in open(probed):
f = line.rstrip("\n").split("\t")
if len(f) < 12:
continue
b, p, dur, vc, res, fps, vbit, pix, aud, subs, _d, fmt = f[:12]
base = os.path.basename(p)
if base.startswith("._") or base == ".DS_Store":
continue # macOS sidecar, not media
m = re.search(r"/(BDMV|VIDEO_TS)/", p)
root = p[:m.start()] if m else None
if root or re.search(r"/(CERTIFICATE|AACS|MLPI|CLIPINF|PLAYLIST|BACKUP|REGISTRL)/", p):
if not root:
continue # disc bookkeeping: folds into its tree, never a film
if root:
t = trees.setdefault(root, dict(bytes=0, audio=set(), subs=0))
t["bytes"] += int(b)
t["audio"].update(a for a in aud.split("+") if a)
t["subs"] += int(subs or 0)
continue
try:
dur = float(dur)
except ValueError:
dur = 0.0
rows.append(dict(bytes=int(b), path=p, dur=dur, vcodec=vc, res=res, fps=fps,
vbit=int(float(vbit or 0)), pix=pix, audio=aud, subs=subs, fmt=fmt))
for root, t in trees.items():
folder = os.path.basename(root)
m = BDMV_MAIN.get(folder, {})
rows.append(dict(bytes=t["bytes"], path=root + "/BDMV/index.bdmv",
name_src=folder, dur=m.get("dur", 0.0),
vcodec=m.get("vcodec", "hevc"), res=m.get("res", "-"),
fps=m.get("fps", 0), vbit=0, pix=m.get("pix", "-"),
audio="+".join(sorted(t["audio"])), subs=str(t["subs"]),
fmt="BDMV tree"))
for r in rows:
r["title"], r["year"] = title_of(r.get("name_src") or r["path"])
r["key"] = key_of(r["title"])
r["base"] = base_words(r["title"], r["key"])
r["soft"] = folder_year(r["path"])
r["ep"] = ep_of(r["path"])
return rows
def dur_clusters(items):
"""Group files that are the SAME FILM and the SAME MASTER duration. Candidate
buckets come from the franchise key; same_film then rejects the different
installments whose runtimes happen to agree."""
clusters = []
for r in sorted(items, key=lambda r: -r["dur"]):
for c in clusters:
if dur_close(c[0]["dur"], r["dur"]) and same_film(c[0], r):
c.append(r)
break
else:
clusters.append([r])
return clusters
def tier(groups):
"""Split same-film/same-duration clusters: DUP = different releases of one
film (real, deletable redundancy); REVIEW = identical size AND duration
(byte-twins — the hash pass is the authority) or clashing year evidence."""
dup, review = [], []
for g in groups:
g.sort(key=lambda r: -r["bytes"])
years = {(r["year"] if r["year"] != "?" else r["soft"]) for r in g}
years.discard(None)
spread = (g[0]["bytes"] - g[-1]["bytes"]) / g[0]["bytes"] if g[0]["bytes"] else 0
if len(years) > 1 or spread < 0.005:
review.append(g)
else:
dup.append(g)
return dup, review
def redun(g):
return sum(r["bytes"] for r in sorted(g, key=lambda r: -r["bytes"])[1:])
def line(r):
return (f"{r['bytes']/2**30:8.2f} GiB {r['res']:>9} {r['vcodec']:<7} "
f"{r['pix']:<11} {r['dur']/60:5.0f}m {r['audio'][:34]:<34}")
def dump(tag, groups, out, note=""):
for gi, g in enumerate(groups, 1):
g.sort(key=lambda r: -r["bytes"])
print(f"\n### {tag}-{gi}. {g[0]['title']} ({g[0]['year']}) — {len(g)} files, "
f"{redun(g)/2**30:.1f} GiB redundant {note}".rstrip())
for rank, r in enumerate(g):
print(f" {line(r)} [{r['year']}] "
f"{r['path'].replace(ROOT + '/', '')}")
out.write(f"{tag}\t{gi}\t{rank}\t{r['bytes']}\t{r['res']}\t{r['vcodec']}\t"
f"{r['pix']}\t{int(r['dur'])}\t{r['audio']}\t{r['title']}|{r['year']}\t"
f"{r['path']}\n")
def report(probed, out_path):
rows = load(probed)
for r in rows:
r["mark"] = is_watermark(r["path"], r["bytes"])
r["extra"] = not r["mark"] and is_extra(r["path"], r["title"], r["dur"])
marks = [r for r in rows if r["mark"]]
movies = [r for r in rows if not r["ep"] and not r["extra"] and not r["mark"]]
episodes = [r for r in rows if r["ep"] and not r["extra"] and not r["mark"]]
extras = [r for r in rows if r["extra"]]
print(f"census: {len(rows)} video entries = {len(movies)} films + {len(episodes)} "
f"TV episodes + {len(extras)} bonus/trailers "
f"({sum(r['bytes'] for r in extras)/2**30:.1f} GiB) "
f"+ {len(marks)} release-group watermarks")
out = open(out_path, "w")
buckets = collections.defaultdict(list)
for r in movies:
buckets[r["key"]].append(r)
cand = {k: v for k, v in buckets.items() if len(v) > 1}
clusters = [c for v in cand.values() for c in dur_clusters(v) if len(c) > 1]
dup, review = tier(clusters)
dup.sort(key=lambda g: -redun(g))
review.sort(key=lambda g: -redun(g))
dump("DUP", dup, out)
dump("REVIEW", review, out, "— same key+duration, sizes near-identical OR years clash: RECHECK")
ebuf = collections.defaultdict(list)
for r in episodes:
ebuf[(r["ep"][0], r["ep"][1])].append(r)
edup, ereview = tier([c for v in ebuf.values() if len(v) > 1
for c in dur_clusters(v) if len(c) > 1])
edup.sort(key=lambda g: -redun(g))
dump("EP-DUP", edup, out)
shows = collections.Counter()
for r in episodes:
shows[r["ep"][0]] += 1
multi = {s: n for s, n in shows.items() if n > 4}
print("\n=== TV-series content inside Movies/ (>4 episodes) ===")
for s, n in sorted(multi.items(), key=lambda kv: -kv[1]):
print(f" {n:4d} episodes {s}")
out.close()
print(f"\nDUP groups={len(dup)} files={sum(len(g) for g in dup)} "
f"redundant={sum(redun(g) for g in dup)/2**40:.2f} TiB", file=sys.stderr)
print(f"REVIEW groups={len(review)} "
f"redundant={sum(redun(g) for g in review)/2**40:.2f} TiB", file=sys.stderr)
print(f"EP-DUP groups={len(edup)} "
f"ep-redundant={sum(redun(g) for g in edup)/2**30:.1f} GiB", file=sys.stderr)
def parse(names):
for n in names:
t, y = title_of(n)
print(f"{t!r} ({y})\tkey={key_of(t)!r}\t{n[:70]}")
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
sub = ap.add_subparsers(dest="cmd", required=True)
rep = sub.add_parser("report", help="similar-copy census over a probe TSV")
rep.add_argument("--probed", default="/tmp/media-probed.tsv")
rep.add_argument("--out", default="/tmp/media-similar.tsv")
par = sub.add_parser("parse", help="show the parsed title/year of a name")
par.add_argument("names", nargs="+")
a = ap.parse_args()
if a.cmd == "report":
report(a.probed, a.out)
else:
parse(a.names)
if __name__ == "__main__":
main()
+281
View File
@@ -0,0 +1,281 @@
#!/usr/bin/env python3
"""media-tv.py — move TV episodes out of the movie library into a TV library.
`scripts/media-plan.py` refuses to file TV: it parks episode files in a
`_TV needs a TV library - <series>` pile for a human. This is the other half of
that refusal. It walks the movie library, groups the files whose own name
carries an episode marker by SERIES, and writes a plan that places each series
in the shape both Jellyfin and Plex read without help:
<library>/<Series> (<year>)/Season <NN>/<the file's own name>.<ext>
It writes a plan and nothing else. Every rename belongs to
`scripts/media-apply.py` — the write-ahead journal, the refusal to overwrite,
the all-or-nothing preflight, the rollback. Apply it with `--root` set to the
share that holds BOTH libraries (`/mnt/entropy`, not `/mnt/entropy/Movies`) so
the executor's own path guard still has teeth, and give it its own journal —
one journal is one undo, and a TV move must not be welded to a movie relayout:
python3 scripts/media-tv.py --root /mnt/entropy/Movies \
--library /mnt/entropy/TVShows --out /mnt/entropy/media-tv-plan.tsv
python3 scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv \
--root /mnt/entropy --journal /mnt/entropy/media-relayout-tv.jsonl
python3 scripts/media-apply.py verify --plan /mnt/entropy/media-tv-plan.tsv \
--root /mnt/entropy --journal /mnt/entropy/media-relayout-tv.jsonl
python3 scripts/media-apply.py sweep --root /mnt/entropy \
--journal /mnt/entropy/media-relayout-tv.jsonl --apply
Guards, and why each one is here:
* The same five-file threshold as the planner. An episode-shaped name on one
file is not a series — `The Godfather Part 2` is a film — so a smaller group
is reported and left exactly where it is.
* An episode whose name carries no season number is reported and left, never
filed into an invented `Season 01`.
* A sidecar that shares an episode's own name (`.srt`, `.nfo`, …) follows the
episode; an episode without its subtitles is a half-move.
* A group that would claim one destination twice is not resolved here. The
executor refuses the whole plan before its first rename, which is the right
outcome: nobody but Frank decides which copy of an episode is the keeper.
* An episode the destination library already holds at the same byte count is
left where it is. The TV library has held a complete Monk box set for years
while the movie library carried a second copy of all 124 episodes; a mover
that did not look where it was filing would have put both inside one library.
* A symlink is never followed and never moved.
"""
import argparse
import collections
import importlib.util
import os
import re
def parse_args():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--root", default=os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
help="the movie library to empty of TV")
ap.add_argument("--library", default="/mnt/entropy/TVShows",
help="the TV library to fill (a different share or folder)")
ap.add_argument("--out", default=None,
help="plan TSV for scripts/media-apply.py"
" (default: media-tv-plan.tsv beside --root)")
ap.add_argument("--min", type=int, default=5,
help="episode files that make something a series")
a = ap.parse_args()
if not a.out:
a.out = os.path.join(os.path.dirname(os.path.abspath(a.root)),
"media-tv-plan.tsv")
return a
ARGS = parse_args()
ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
spec = importlib.util.spec_from_file_location("eng", ENGINE)
eng = importlib.util.module_from_spec(spec)
spec.loader.exec_module(eng)
BAD_CHARS = re.compile(r'[<>:"/|?*]')
# The season and episode, from the token media-titles.ep_of() hands back: s05e03, 1x04.
SEASON_RE = re.compile(r"^(?:s(\d{1,2})e(\d{1,4})|(\d{1,2})x(\d{1,4}))$")
# Sidecars that follow an episode; macOS forks and resume data do not.
SKIP_SIDECAR = re.compile(r"^(\._|\.DS_|dl_state)")
# A box set that stamps the show's year into every file name; the folder's year
# must not be added on top of it.
YEAR_TAIL = re.compile(r"[(\[]\s*(18|19|20)\d\d\s*[)\]]\s*$")
SMALL = {"a", "an", "the", "and", "of", "for", "in", "on", "to", "with", "at",
"by", "de", "la", "vs"}
def clean(name):
"""A name the filesystem and both matchers can read (same law as the plan)."""
name = BAD_CHARS.sub("-", name).replace("\\", "-")
name = re.sub(r"\s+", " ", name).strip().strip(". ").strip()
return name[:180] or "untitled"
def episode_name(path):
"""The file's own name, kept: the publisher's spelling beats a re-derivation.
A dot-separated release name (`monk.s05e03.mkv`) becomes words; a name that
already carries spaces is left exactly as it was written."""
stem, ext = os.path.splitext(os.path.basename(path))
if " " not in stem and stem.count(".") >= 2:
stem = stem.replace(".", " ")
return clean(stem) + ext.lower()
def head_of(path):
"""The publisher's series name: the text BEFORE the episode marker.
`Monk - S05E03 - Mr. Monk and the Big Game` is `Monk`; the episode title
after the marker is not the show's name, and case is kept because it is the
library's own spelling, not a normalised key."""
name = os.path.splitext(os.path.basename(path))[0]
m = eng.EP_RE.search(name)
head = name[:m.start()] if m else name
return re.sub(r"\s+", " ", re.sub(r"[._]+", " ", head)).strip(" -\u2013").strip()
def title_series(head):
"""The series folder in the case a TV library is written in.
Only a name with no capital at all is retitled: a box set that was
lower-cased on the way in would otherwise file `monk (2002)` among
`Dexter's Laboratory (1996)`. A name the publisher wrote with case is left
exactly as it was written."""
if any(c.isupper() for c in head):
return head # the publisher wrote it with case; keep it
words = head.split()
return " ".join(w if i and w.lower() in SMALL else w.capitalize()
for i, w in enumerate(words))
def season_episode(token):
"""(season, episode) an episode token states, or None when it states none."""
m = SEASON_RE.match(token)
if not m:
return None
return int(m.group(1) or m.group(3)), int(m.group(2) or m.group(4))
def sidecars(path):
"""The files beside PATH that carry its exact stem — its subtitles, its nfo."""
d, stem = os.path.split(path)
stem = os.path.splitext(stem)[0]
out = []
try:
entries = sorted(os.listdir(d))
except OSError:
return out
for e in entries:
p = os.path.join(d, e)
if p == path or os.path.islink(p) or not os.path.isfile(p):
continue
s, _ext = os.path.splitext(e)
if s == stem and not SKIP_SIDECAR.match(e):
out.append(p)
return out
def episodes_under(root):
"""(path, series, token) for every episode-shaped video file under root."""
for dirpath, dirnames, filenames in os.walk(root):
dirnames[:] = sorted(d for d in dirnames
if not d.startswith(".") and d != "@eaDir")
for f in filenames:
p = os.path.join(dirpath, f)
if os.path.islink(p) or not f.lower().endswith(eng.VIDEO_SUFFIX):
continue
e = eng.ep_of(p)
if e:
yield p, e[0], e[1]
def walk(root, library):
"""Episode files under ROOT keyed by series, ignoring the destination."""
groups = collections.defaultdict(list)
lib = os.path.abspath(library)
for p, series, token in episodes_under(root):
if os.path.abspath(p).startswith(lib + os.sep):
continue
groups[series].append((p, token))
return groups
def holdings(library):
"""What the destination already carries: (series, season, episode) -> sizes.
The one expensive read here, and the one that matters. The movie library
holds a complete Monk box set that the TV library has held for years: 124
episodes, the same episode list, identical byte counts. Moving it would
have filed a second copy of a whole show into the library that already had
it. Byte count is the test on purpose — two encodes of one episode are two
files and both are wanted; the same bytes are the same file."""
out = collections.defaultdict(set)
for p, series, token in episodes_under(library):
se = season_episode(token)
if se is None:
continue
try:
out[(series, *se)].add(os.path.getsize(p))
except OSError:
continue
return out
def main():
root, library = os.path.abspath(ARGS.root), os.path.abspath(ARGS.library)
groups = walk(root, library)
held_here = holdings(library)
mkdirs = {library}
rows, lines, skipped = set(), [], []
for series, items in sorted(groups.items()):
if len(items) < ARGS.min:
skipped.append(f"skipped: {head_of(items[0][0]) or series} — {len(items)} "
f"episode-shaped file{'s' if len(items) != 1 else ''},"
f" below {ARGS.min}: not a series")
continue
years = collections.Counter(y for y in (eng.folder_year(p) for p, _t in items)
if y)
year = years.most_common(1)[0][0] if years else None
heads = collections.Counter(head_of(p) for p, _t in items)
head = heads.most_common(1)[0][0]
if year and YEAR_TAIL.search(head):
year = None
name = clean(f"{title_series(head)}{f' ({year})' if year else ''}")
series_dir = os.path.join(library, name)
moved = held = 0
for path, token in items:
se = season_episode(token)
if se is None:
skipped.append(f"skipped: {path} — no season number in its name,"
" nothing here invents one")
continue
try:
size = os.path.getsize(path)
except OSError:
continue
if size in held_here.get((series, *se), ()):
held += 1
continue
dst_dir = os.path.join(series_dir, f"Season {se[0]:02d}")
mkdirs.update([series_dir, dst_dir])
base = episode_name(path)
rows.add((path, os.path.join(dst_dir, base)))
moved += 1
for side in sidecars(path):
rows.add((side, os.path.join(
dst_dir, clean(os.path.splitext(base)[0]
+ os.path.splitext(side)[1].lower()))))
if moved:
lines.append(f"{name}: {moved} episodes"
f"{'' if year else ' (no year in any folder name)'}"
f" -> {series_dir}")
if held:
lines.append(f"{name}: {held}"
f" {'episode' if held == 1 else 'episodes'} already in"
f" {library} at the same byte count — left there")
# An empty plan creates nothing, not even the library folder: a run that
# found no series must not be able to leave a stray directory behind.
dirs = sorted(mkdirs, key=lambda p: (p.count(os.sep), p)) if rows else []
with open(ARGS.out, "w") as out:
out.write("action\tsrc\tdst\tkind\n")
for d in dirs:
out.write(f"mkdir\t\t{d}\tdir\n")
for src, dst in sorted(rows):
out.write(f"move\t{src}\t{dst}\ttv\n")
for line in lines + skipped:
print(line)
print(f"plan: {len(rows)} moves, {len(dirs)} folders -> {ARGS.out}")
if rows:
print(f"next: python3 scripts/media-apply.py apply --plan {ARGS.out}"
f" --root {os.path.dirname(root)}"
f" --journal {os.path.join(os.path.dirname(root),'media-relayout-tv.jsonl')}"
" (add --apply to run it)")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+241
View File
@@ -0,0 +1,241 @@
#!/usr/bin/env bash
# The SSD-backed metadata view that lets Jellyfin and Plex share one metadata set.
#
# Both servers read a Kodi-format sidecar (movie.nfo / tvshow.nfo / episode.nfo)
# ONLY when it sits next to the media file. The media lives on the RAID6 array at
# /mnt/entropy — five spinning disks, deliberately mounted read-only for every
# container, and slow to seek for a thousand tiny XML reads. So neither media
# server binds the array any more: they bind an OverlayFS merge of it with a
# writable layer on the NVMe.
#
# lowerdir /mnt/entropy RAID6 (ROTA=1) — source of truth, never written
# upperdir /var/lib/media-meta/upper NVMe — every sidecar and image byte
# workdir /var/lib/media-meta/work NVMe — kernel bookkeeping
# Where /var/lib/media-meta/view the merged tree jellyfin and plex bind
#
# Everything the view needs — layers, bookkeeping, trash, the merged door — lives
# under the one directory /var/lib/media-meta, so nothing new appears under /mnt.
#
# Consequences worth knowing:
# * A sidecar write is a few KB onto the SSD; the media bytes never move and the
# array is still never written by anything automated.
# * Deleting through the view writes a whiteout into the upper layer instead of
# deleting from the array. Nothing should delete through the view: deletions
# go to .AGENTTRASH on the RAW path, and `reap` clears the resulting ghosts.
# * inotify events on the lower layer do not cross the merge, so a file that
# arrives on the raw array is invisible to a file-system watcher in the view:
# trigger a library refresh (see the Media Relayout page).
#
# Subcommands
# plan [--upper-base D] [--lower D] print the resolved shape, touch nothing
# verify prove the live mount IS that shape
# install [--upper-base D] create the layers, install+enable+start the unit
# reap [--apply] [--upper-base D] [--lower D]
# list upper-layer entries whose media is gone
# status one line per fact a human wants
#
# install must run as root (it writes /etc/systemd/system and mounts); the others
# are read-only and safe unprivileged.
set -euo pipefail
LOWER=/mnt/entropy
UPPER_BASE=/var/lib/media-meta
VIEW=$UPPER_BASE/view
# systemd names a mount unit after its ESCAPED path ('/' -> '-', a literal '-'
# -> '\x2d'), so the unit on the host is var-lib-media\x2dmeta-view.mount. The
# repo copy keeps a readable name; install writes it under the escaped one.
UNIT='var-lib-media\x2dmeta-view.mount'
UNIT_DIR=/etc/systemd/system
# The rootless containers write as this host uid (the deploy user). The upper
# layer has to accept those writes, so it is owned by them.
MEDIA_UID="${MEDIA_VIEW_UID:-1000}"
MEDIA_GID="${MEDIA_VIEW_GID:-1000}"
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
UNIT_SRC="$REPO_ROOT/dankserver/host/systemd/media-meta-view.mount"
die() {
printf 'media-view: %s\n' "$*" >&2
exit 1
}
usage() {
printf 'usage: %s plan|verify|install|reap|status [options]\n' "$0" >&2
exit 2
}
cmd="${1:-}"
[ -n "$cmd" ] || usage
shift || true
upper_base="$UPPER_BASE"
lower="$LOWER"
apply=0
while [ $# -gt 0 ]; do
case "$1" in
--upper-base) [ $# -ge 2 ] || usage; upper_base="$2"; shift 2 ;;
--lower) [ $# -ge 2 ] || usage; lower="$2"; shift 2 ;;
--apply) apply=1; shift ;;
*) usage ;;
esac
done
upper="$upper_base/upper"
work="$upper_base/work"
# The device backing a path, and whether the kernel calls it rotational.
rotational_of() {
local src rota
src="$(findmnt -no SOURCE --target "$1" 2>/dev/null || true)"
[ -n "$src" ] || die "cannot find a block device behind $1"
rota="$(lsblk -no ROTA "$src" 2>/dev/null | head -1 || true)"
[ -n "$rota" ] || die "cannot read the rotational flag of $src"
printf '%s %s\n' "$src" "$rota"
}
require_upper_base() {
[ -d "$upper_base" ] \
|| die "refused: upper base $upper_base does not exist (put it on the SSD first)"
}
do_plan() {
printf 'lowerdir=%s\nupperdir=%s\nworkdir=%s\nmountpoint=%s\nunit=%s/%s\n' \
"$lower" "$upper" "$work" "$VIEW" "$UNIT_DIR" "$UNIT"
}
do_verify() {
local fail=0 fst opts src rota
if ! mountpoint -q "$VIEW"; then
printf 'refused: %s is not a mountpoint — a plain directory here is a\n' "$VIEW" >&2
printf ' library that scans itself empty, not a metadata view.\n' >&2
return 1
fi
fst="$(findmnt -no FSTYPE --mountpoint "$VIEW" 2>/dev/null || true)"
if [ "$fst" != overlay ]; then
printf 'refused: %s is mounted %s, not overlay\n' "$VIEW" "${fst:-nothing}" >&2
fail=1
fi
opts="$(findmnt -no OPTIONS --mountpoint "$VIEW" 2>/dev/null || true)"
local want
for want in "lowerdir=$LOWER" "upperdir=$UPPER_BASE/upper" "workdir=$UPPER_BASE/work"; do
case ",$opts," in
*",${want},"*) ;;
*) printf 'refused: %s options lack %s (got: %s)\n' "$VIEW" "$want" "$opts" >&2
fail=1 ;;
esac
done
read -r src rota <<<"$(rotational_of "$UPPER_BASE")"
if [ "$rota" != 0 ]; then
printf 'refused: the upper layer sits on %s which is rotational — metadata\n' "$src" >&2
printf ' belongs on the SSD, not on the array.\n' >&2
fail=1
fi
if [ ! -w "$UPPER_BASE/upper" ]; then
printf 'refused: %s is not writable\n' "$upper" >&2
fail=1
fi
[ "$fail" -eq 0 ] || return 1
printf 'view ok: %s <- overlay(%s,%s)\n' "$VIEW" "$LOWER" "$UPPER_BASE/upper"
}
do_install() {
[ "$(id -u)" -eq 0 ] || die "install must run as root (writes $UNIT_DIR and mounts)"
require_upper_base
mountpoint -q "$LOWER" \
|| die "refused: $LOWER is not mounted; overlaying an empty directory would present an empty library"
local src rota
read -r src rota <<<"$(rotational_of "$upper_base")"
[ "$rota" = 0 ] || die "refused: $upper_base is on $src, a rotational device — the metadata layer must be on the SSD"
[ -f "$UNIT_SRC" ] || die "refused: $UNIT_SRC is missing from $REPO_ROOT"
# systemd will not create a mount point; a missing directory fails the start.
install -d -m 0755 -o "$MEDIA_UID" -g "$MEDIA_GID" "$upper" "$work" "$VIEW"
install -d -m 0700 "$upper_base/.AGENTTRASH"
install -m 0644 "$UNIT_SRC" "$UNIT_DIR/$UNIT"
systemctl daemon-reload
systemctl enable "$UNIT" >/dev/null 2>&1 || true
systemctl start "$UNIT"
do_verify
}
# An upper-layer DIRECTORY with no counterpart below is metadata for a media
# folder that has been moved or renamed on the RAW path — a ghost the media
# servers would still browse. Files are never candidates on their own: a sidecar
# (movie.nfo, an .srt) exists ONLY above, by design, and is exactly what the view
# is for. A copy-up whose lower file was deleted by hand is the one case this
# leaves in place; the raw path is where deletions belong, so it stays visible.
do_reap() {
[ -d "$upper" ] || die "refused: $upper does not exist (no upper layer at $upper_base)"
stamp="$upper_base/.AGENTTRASH/$(date -u +%Y%m%d-%H%M%S)"
count=0
ghosts=0
reap_walk ""
if [ "$count" -eq 0 ] && [ "$ghosts" -eq 0 ]; then
printf 'reap: no ghosts under %s\n' "$upper"
return 0
fi
if [ "$ghosts" -gt 0 ]; then
printf 'reap: %d whiteout%s above — someone deleted THROUGH the view.\n' \
"$ghosts" "$([ "$ghosts" -eq 1 ] || printf s)" >&2
printf ' Leaving them: removing a whiteout un-hides the file it hides.\n' >&2
printf ' Deletions belong in .AGENTTRASH on the raw path.\n' >&2
fi
if [ "$count" -eq 0 ]; then return 0; fi
if [ "$apply" -eq 1 ]; then
printf 'reap: moved %d entr%s to %s\n' "$count" \
"$([ "$count" -eq 1 ] && printf y || printf ies)" "$stamp"
else
printf 'reap: %d ghost entr%s above; re-run with --apply to move them to %s\n' \
"$count" "$([ "$count" -eq 1 ] && printf y || printf ies)" "$stamp"
fi
}
# Descend only where the lower layer still has the directory: a directory that is
# missing below is itself the ghost, and its whole subtree goes with it.
reap_walk() {
local rel="$1" entries abs r name
entries="$(find "$upper${rel:+/$rel}" -mindepth 1 -maxdepth 1 -print 2>/dev/null || true)"
[ -n "$entries" ] || return 0
while IFS= read -r abs; do
r="${abs#"$upper"/}"
name="${abs##*/}"
if [ -c "$abs" ] || printf '%s' "$name" | grep -q '^\.wh\.'; then
ghosts=$((ghosts + 1))
continue
fi
if [ -e "$lower/$r" ]; then
[ -d "$abs" ] && reap_walk "$r"
continue
fi
[ -d "$abs" ] || continue
count=$((count + 1))
printf '%s\n' "$r"
if [ "$apply" -eq 1 ]; then
mkdir -p "$stamp/$(dirname "$r")"
mv "$abs" "$stamp/$r"
fi
done <<<"$entries"
}
do_status() {
if mountpoint -q "$VIEW"; then
printf 'view: %s (%s)\n' "$VIEW" "$(findmnt -no OPTIONS --mountpoint "$VIEW")"
else
printf 'view: %s NOT MOUNTED\n' "$VIEW"
fi
if [ -d "$upper" ]; then
printf 'upper: %s %s files %s\n' "$upper" \
"$(find "$upper" -type f 2>/dev/null | wc -l)" "$(du -sh "$upper" 2>/dev/null | cut -f1)"
else
printf 'upper: %s missing\n' "$upper"
fi
systemctl is-active "$UNIT" 2>/dev/null | sed 's/^/unit: /' || printf 'unit: %s unknown\n' "$UNIT"
}
case "$cmd" in
plan) do_plan ;;
verify) do_verify ;;
install) do_install ;;
reap) do_reap ;;
status) do_status ;;
*) usage ;;
esac
+33 -12
View File
@@ -12,8 +12,13 @@
# (2026-10-29) the rig's boot died of an OOM, the health loop quietly fell through, and the # (2026-10-29) the rig's boot died of an OOM, the health loop quietly fell through, and the
# pusher fed four rounds of 10M tokens into RESTORED PRODUCTION with nothing recorded: # pusher fed four rounds of 10M tokens into RESTORED PRODUCTION with nothing recorded:
# #
# 1. the runner must still be alive and its container must answer /health, or this script # 1. the runner must still be alive and its container must answer /health AND own the
# kills it, restores the service and exits non-zero — no push, ever; # listening socket, or this script kills it, restores the service and exits non-zero —
# no push, ever. The socket clause is not decoration: the runner stops frank-archon
# first, the dying process keeps its listener for a moment, and a rig container that
# is still loading has not bound the port yet, so `podman inspect` + `/health` both
# report success for the WRONG process (measured 2026-10-30: `health ok waited=0s`,
# then the pusher refused on the container name and the window died at second zero);
# 2. the pusher refuses on its own unless the rig wrote its marker into the tap dir, the # 2. the pusher refuses on its own unless the rig wrote its marker into the tap dir, the
# marker names this container, and the container is running (scripts/calib-push.py # marker names this container, and the container is running (scripts/calib-push.py
# --tap-marker), so a second lock survives a mistake in this file; # --tap-marker), so a second lock survives a mistake in this file;
@@ -35,20 +40,26 @@ MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}" CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}" RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}" CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}"
KLDREF="$CAMPAIGN/kld-ref/kld.json" CALIB="${FRANK_QWEN38_CALIB_DIR:-$CAMPAIGN/calib}"
CALIB="$CAMPAIGN/calib"
HEALTH_TIMEOUT="${Q38_WINDOW_HEALTH_TIMEOUT:-3600}" HEALTH_TIMEOUT="${Q38_WINDOW_HEALTH_TIMEOUT:-3600}"
ROUNDS="${Q38_WINDOW_ROUNDS:-6}" ROUNDS="${Q38_WINDOW_ROUNDS:-6}"
RIG=qwen38-bf16 RIG=qwen38-bf16
SERVICE=frank-archon SERVICE=frank-archon
PY="${PY:-python3}" PY="${PY:-python3}"
# ---- preflight: everything this window needs, before a card is touched ------- # ---- preflight: every fact by identity, one owner (scripts/qwen38-preflight.sh) ----
[ -f "$KLDREF" ] || { echo "REFUSED: no reference recording at $KLDREF — record it first (scripts/run-qwen38-bf16.sh kldref); this window only fills the calibration tap" >&2; exit 2; } # The checks that used to live here were proxies, and each lied at least once: the space check
# asked a directory that merely LOOKED related, `/health` answered for a dying production
# process, `podman inspect` named a container that existed but had bound nothing. The shared
# script binds each fact to its object — cwd, the digest the tag actually resolves to, free
# space on the tap's own filesystem, every /dev/kfd holder's HOST pid against the service's,
# the :7770 listener's host pid, the corpus and the reference recording. A resumed window
# keeps an existing tap, so the tap-empty fact is waived.
[ -f "$RAD" ] || { echo "REFUSED: no BF16 reference container at $RAD — nothing to serve; scripts/build-qwen38-bf16-rad.sh" >&2; exit 2; } [ -f "$RAD" ] || { echo "REFUSED: no BF16 reference container at $RAD — nothing to serve; scripts/build-qwen38-bf16-rad.sh" >&2; exit 2; }
[ -f "$CORPUS" ] || { echo "REFUSED: no calibration corpus at $CORPUS" >&2; exit 2; } FRANK_QWEN38_CAMPAIGN="$CAMPAIGN" \
[ -d "$CALIB" ] || { echo "REFUSED: no tap dir at $CALIB" >&2; exit 2; } Q38_WINDOW_CALIB_CORPUS="$CORPUS" \
[ -w "$CALIB" ] || { echo "REFUSED: tap dir $CALIB is not writable by $(id -un)" >&2; exit 2; } Q38_CALIB_MIN_FREE_MIB="${Q38_CALIB_MIN_FREE_MIB:-61440}" \
bash scripts/qwen38-preflight.sh --allow-existing-tap || exit 2
! [[ "$HEALTH_TIMEOUT" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_HEALTH_TIMEOUT must be seconds (got: $HEALTH_TIMEOUT)" >&2; exit 2; } ! [[ "$HEALTH_TIMEOUT" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_HEALTH_TIMEOUT must be seconds (got: $HEALTH_TIMEOUT)" >&2; exit 2; }
! [[ "$ROUNDS" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_ROUNDS must be a round count (got: $ROUNDS)" >&2; exit 2; } ! [[ "$ROUNDS" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_ROUNDS must be a round count (got: $ROUNDS)" >&2; exit 2; }
@@ -69,8 +80,18 @@ say "phase=calib serve begin"
bash scripts/run-qwen38-bf16.sh calib & spid=$! bash scripts/run-qwen38-bf16.sh calib & spid=$!
rig_up() { rig_up() {
podman inspect -f '{{.State.Running}}' "$RIG" 2>/dev/null | grep -qx true \ podman inspect -f '{{.State.Running}}' "$RIG" 2>/dev/null | grep -qx true || return 1
&& curl -sf --max-time 10 -o /dev/null http://127.0.0.1:7770/health curl -sf --max-time 10 -o /dev/null http://127.0.0.1:7770/health || return 1
# The answer must come from the CONTAINER. The runner stops frank-archon first, and the
# dying production process keeps its listener for a moment while the freshly created
# rig container is still loading — so `container running` + `/health` 200 both hold
# before the rig has bound anything. Measured 2026-10-30: `health ok waited=0s`, then the
# pusher refused on the container name and the window died at second zero. Ask the kernel
# who owns the listening socket and require it to be a rig process.
local lpid
lpid=$(ss -H -ltnp 'sport = :7770' 2>/dev/null | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)
[ -n "${lpid:-}" ] || return 1
podman top "$RIG" pid 2>/dev/null | grep -qw "$lpid"
} }
waited=0 waited=0
@@ -115,7 +136,7 @@ say "TAP BANKED: $(ls -1 "$CALIB" | wc -l) file(s) in $CALIB"
if [ "${Q38_WINDOW_KEEP_RAD:-0}" = "1" ]; then if [ "${Q38_WINDOW_KEEP_RAD:-0}" = "1" ]; then
say "reference .rad kept (Q38_WINDOW_KEEP_RAD=1)" say "reference .rad kept (Q38_WINDOW_KEEP_RAD=1)"
else else
T="$HOME/.AGENTTRASH/$(date +%Y%m%d)-qwen38-bf16-rad" T="${Q38_WINDOW_TRASH_ROOT:-$HOME/.AGENTTRASH}/$(date +%Y%m%d)-qwen38-bf16-rad"
mkdir -p "$T" || { echo "FAIL: cannot create trash dir $T" >&2; exit 1; } mkdir -p "$T" || { echo "FAIL: cannot create trash dir $T" >&2; exit 1; }
mv "$RAD" "$T/" && say "reference .rad banked in $T (the checkpoint that rebuilds it stays)" mv "$RAD" "$T/" && say "reference .rad banked in $T (the checkpoint that rebuilds it stays)"
fi fi
+136
View File
@@ -0,0 +1,136 @@
#!/usr/bin/env bash
# qwen38-preflight.sh — the facts a calibration window must have BEFORE it takes the cards,
# each checked by IDENTITY and not by proxy. Every line is PASS or FAIL with the reason.
#
# Why this file exists: on 2026-10-30 a window died at second zero because `/health` answered
# 200 from the *production* process that was still draining its listener, while the freshly
# created rig container had not bound anything. Other runs died from a port that answered for
# the wrong process, a /dev/kfd held by an orphaned engine, a tap dir on a 35 GiB filesystem,
# and a launcher whose cwd silently lacked the repo. None of those were math problems; all of
# them were a check that measured something adjacent to the thing it meant.
#
# Usage: bash scripts/qwen38-preflight.sh [--allow-existing-tap]
# Exit: 0 all PASS · 2 any FAIL · 3 a fact could not be evaluated (missing tool)
set -uo pipefail
SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SELF_DIR/.." && pwd)"
CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
TAP="$CAMPAIGN/calib"
CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}"
NEED_MIB="${Q38_CALIB_MIN_FREE_MIB:-61440}"
ALLOW_EXISTING=0
[ "${1:-}" = "--allow-existing-tap" ] && ALLOW_EXISTING=1
# The producer pin is the calib image; the window fails closed if the tag on this host is not it.
WANT_DIGEST="${Q38_PREFLIGHT_IMAGE_DIGEST:-sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}"
IMAGE_REF="${Q38_CALIB_IMAGE:-stilldeadcode/radiance@$WANT_DIGEST}"
SERVICE="frank-archon"
PORT=7770
fails=0
ok() { printf 'PASS %-14s %s\n' "$1" "$2"; }
bad() { printf 'FAIL %-14s %s\n' "$1" "$2"; fails=$((fails + 1)); }
skip() { printf 'WARN %-14s %s\n' "$1" "$2"; }
# 1. cwd — a detached launcher that lost its cwd fails here in a second, not in the log later.
if [ "$(cd "$REPO_ROOT" && pwd)" = "$(pwd)" ] && [ -x "$REPO_ROOT/scripts/run-qwen38-bf16.sh" ]; then
ok cwd "running from the repo root ($REPO_ROOT)"
else
bad cwd "cwd is $(pwd), not the repo root ($REPO_ROOT) — every relative script path would break"
fi
# 2. image — resolve the REF to a digest. A stale local `:latest` is a different model of the world.
if command -v podman >/dev/null 2>&1; then
# An image carries EVERY digest it was pulled under (a tag plus the by-digest pull), so this
# asks whether the pin is one of them — reading RepoDigests[0] reports a healthy image as wrong.
got="$(podman image inspect -f '{{range .RepoDigests}}{{println .}}{{end}}' "$IMAGE_REF" 2>/dev/null \
| sed 's/.*@//' | grep -c "^${WANT_DIGEST}$")"
if [ "${got:-0}" -ge 1 ]; then
ok image "$IMAGE_REF carries ${WANT_DIGEST:0:19}…"
elif podman image exists "$IMAGE_REF" 2>/dev/null; then
bad image "$IMAGE_REF is present but does not carry $WANT_DIGEST (a retagged or older pull)"
else
bad image "$IMAGE_REF is not present on this host"
fi
else
skip image "podman not on PATH — cannot resolve $IMAGE_REF"
fi
# 3. space — df the FILE PATH the seal writes to, never a directory that merely looks related.
if [ -d "$TAP" ] || mkdir -p "$TAP" 2>/dev/null; then
avail="$(df --output=avail -B 1M "$TAP" 2>/dev/null | tail -1 | tr -d ' ')"
if [[ "${avail:-}" =~ ^[0-9]+$ ]] && [ "$avail" -ge "$NEED_MIB" ]; then
ok space "$avail MiB free on $(df --output=target "$TAP" | tail -1) (need $NEED_MIB)"
else
bad space "${avail:-unknown} MiB free on $TAP, need $NEED_MIB — one seal writes ~43 GB"
fi
# 4. tap cleanliness — stale grams from another run must never be mixed in.
n=$(ls -A "$TAP" 2>/dev/null | grep -c . || true)
if [ "$n" -eq 0 ] || [ "$ALLOW_EXISTING" = 1 ]; then
ok tap-empty "$n entries in $TAP"
else
bad tap-empty "$n entries in $TAP — rotate them to .AGENTTRASH first"
fi
else
bad space "cannot create $TAP"
fi
# 5. device — every /dev/kfd holder must be a process of the service we are about to stop. An
# orphaned engine from a previous run makes the next boot die with `startup stopped: device
# error`. Identity comes from podman's own host-pid list: `podman top <ctr> pid` prints
# CONTAINER-internal pids (1, 7, …), which compare against nothing, and a rootless cgroup
# carries the container ID, never the service name — both are proxies that lie.
PROD_CTR="${Q38_PREFLIGHT_PROD_CONTAINER:-$SERVICE}"
service_pids() { podman top "$PROD_CTR" hpid 2>/dev/null | tail -n +2; }
if [ -e /dev/kfd ]; then
mine="$(service_pids | tr '\n' ' ')"
foreign=""
for p in $(fuser /dev/kfd 2>/dev/null); do
case " $mine " in
*" $p "*) : ;; # the service we will stop
*) foreign="$foreign $p:$(cat /proc/$p/comm 2>/dev/null)" ;;
esac
done
if [ -z "$foreign" ]; then
ok device "no foreign /dev/kfd holders"
else
bad device "foreign holders:$foreign — kill by pid, never by pattern"
fi
else
skip device "/dev/kfd absent — not an AMD host?"
fi
# 6. port — a listener on :7770 must be a process of the service we stop, never an unknown one.
lpid="$(ss -H -ltnp "sport = :$PORT" 2>/dev/null | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)"
if [ -z "${lpid:-}" ]; then
ok port ":$PORT is free"
else
mine="$(service_pids | tr '\n' ' ')"
case " $mine " in
*" $lpid "*) ok port ":$PORT is held by $SERVICE (host pid $lpid) — the window stops it" ;;
*) bad port ":$PORT is held by host pid $lpid ($(cat /proc/$lpid/comm 2>/dev/null)), not $SERVICE" ;;
esac
fi
# 7. corpus — the push source must exist and be the re-chunked file the window documents.
if [ -s "$CORPUS" ]; then
ok corpus "$(wc -l < "$CORPUS") prompts in $CORPUS"
else
bad corpus "$CORPUS is missing or empty"
fi
# 8. reference — the KL gate has nothing to compare against without it.
if [ -s "$CAMPAIGN/kld-ref/kld.json" ]; then
ok reference "$(du -h "$CAMPAIGN/kld-ref/kld.json" | cut -f1) kld.json present"
else
bad reference "$CAMPAIGN/kld-ref/kld.json is missing"
fi
echo
if [ "$fails" -gt 0 ]; then
echo "PREFLIGHT FAILED: $fails fact(s). The window must not take the cards."
exit 2
fi
echo "PREFLIGHT PASSED: every fact checked by identity."
+54 -6
View File
@@ -28,10 +28,18 @@
# (RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS=12000000). A # (RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS=12000000). A
# separate driver — scripts/calib-push.py --port 7770 # separate driver — scripts/calib-push.py --port 7770
# --container qwen38-bf16 --corpus # --container qwen38-bf16 --corpus
# /home/dank/corpora/out/calib.jsonl — pushes the corpus; at # /home/dank/corpora/out/calib-7168.jsonl — pushes the corpus;
# 12M pooled positions the engine writes the Grams and the # at the floor the engine writes the Grams and the tap switches
# tap switches itself off (calib-push prints TAP_DONE). Stop # itself off (calib-push prints TAP_DONE). Stop this script
# this script after that; the trap brings frank-archon back. # after that; the trap brings frank-archon back. THE FLOOR IS
# THE CAMPAIGN'S RULE (12 M tokens, Frank 2026-10-30): the
# served iq4r this arm competes with was calibrated on a 10 M-
# token set, so the arm is calibrated at least as deeply. One
# round of the re-chunked corpus is ~9.31 M tokens, so sealing
# takes round 1 plus part of round 2 — a round that seals
# nothing is NORMAL (the floor is pooled), and the pusher's
# round_verdict treats a flat round as "continue", never as a
# failure. One seal writes ~40 GiB.
# #
# Shape: the vendor's bf16 calibration lines — expert_tiered placement with the # Shape: the vendor's bf16 calibration lines — expert_tiered placement with the
# weights disk tier, deterministic (the resident set pinned to the loader's plan, # weights disk tier, deterministic (the resident set pinned to the loader's plan,
@@ -113,6 +121,18 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091 # shellcheck disable=SC1091
source "$SCRIPT_DIR/frank-archon.env" source "$SCRIPT_DIR/frank-archon.env"
IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}" IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
# THE CALIBRATION PIN: the LATEST radiance, pinned by digest because a tag is a
# moving target and a tap must name the producer that wrote it. `:latest` on
# 2026-10-30 resolves to v1.14.0 (revision 4a61d7296bc5), and on that image a
# calib boot prints the arming line — "recorded passes: off, the program
# accumulates calibration Grams (gram_accum)" — where the production pin
# (1.8.1) prints nothing of the kind: a 17-hour run on 1.8.1 wrote ZERO bytes
# into the tap dir while /health, /stats and the push all looked healthy
# (measured 2026-10-30). The tap code is identical in both trees' sources, so
# the shipped 1.8.1 build is what drops it; do not spend another window on it.
# Any other image must be named deliberately with --image.
CALIB_IMAGE="${Q38_CALIB_IMAGE:-stilldeadcode/radiance@sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}"
IMAGE_EXPLICIT=0
MODE="" MODE=""
POS=() POS=()
@@ -122,7 +142,7 @@ while [ $# -gt 0 ]; do
kldref|calib|score) MODE="$1" ;; kldref|calib|score) MODE="$1" ;;
--dry-run) DRY=1 ;; --dry-run) DRY=1 ;;
--image) [ "$#" -ge 2 ] || { echo "--image needs a reference" >&2; exit 2; } --image) [ "$#" -ge 2 ] || { echo "--image needs a reference" >&2; exit 2; }
IMAGE="$2"; shift ;; IMAGE="$2"; IMAGE_EXPLICIT=1; shift ;;
*) *)
# score's two positional arguments, and nothing else may be bare. # score's two positional arguments, and nothing else may be bare.
if [ "$MODE" = score ] && [ "${#POS[@]}" -lt 2 ]; then if [ "$MODE" = score ] && [ "${#POS[@]}" -lt 2 ]; then
@@ -152,7 +172,7 @@ TEMPLATE="$SCRIPT_DIR/qwen38-froggeric-v22.5.jinja"
LOG_DIR="${FRANK_QWEN38_LOG_DIR:-/home/dank/logs}" LOG_DIR="${FRANK_QWEN38_LOG_DIR:-/home/dank/logs}"
LOG="$LOG_DIR/qwen38_bf16_${MODE}.log" LOG="$LOG_DIR/qwen38_bf16_${MODE}.log"
KLD_REF_DIR="$CAMPAIGN/kld-ref" KLD_REF_DIR="$CAMPAIGN/kld-ref"
CALIB_DIR="$CAMPAIGN/calib" CALIB_DIR="${FRANK_QWEN38_CALIB_DIR:-$CAMPAIGN/calib}"
C=qwen38-bf16 C=qwen38-bf16
SERVICE=frank-archon SERVICE=frank-archon
# The card split, taken from the file that owns it instead of a private pair: # The card split, taken from the file that owns it instead of a private pair:
@@ -314,6 +334,21 @@ case "$MODE" in
# buys throughput with the expert slab, so price it like any other # buys throughput with the expert slab, so price it like any other
# VRAM knob. The slots follow frank-archon.env's FRANK_ARCHON_MAX_SEQS # VRAM knob. The slots follow frank-archon.env's FRANK_ARCHON_MAX_SEQS
# through MAX_SEQS, exactly as kldref and score do. # through MAX_SEQS, exactly as kldref and score do.
[ "$IMAGE_EXPLICIT" = 1 ] || IMAGE="$CALIB_IMAGE"
# ONE TAP SEAL WRITES ~40 GiB. Measured 2026-10-30: a 8,192-token floor wrote
# 39.4 GiB into the tap dir within seconds; pointed at a 37 GiB root it filled
# the filesystem to zero, after which every write came back short ("calibration
# cannot write", "write was short") and the system itself began refusing
# writes. The room is demanded up front, on the tap's own filesystem.
if [ "$DRY" != 1 ]; then
avail_mib=$(df --output=avail -B 1M "$CALIB_DIR" 2>/dev/null | tail -1 | tr -d ' ')
[[ "$avail_mib" =~ ^[0-9]+$ ]] || avail_mib=0
need_mib="${Q38_CALIB_MIN_FREE_MIB:-61440}"
if [ "$avail_mib" -lt "$need_mib" ]; then
echo "REFUSED: tap dir $CALIB_DIR has ${avail_mib} MiB free; one seal writes ~40 GiB and this run needs ${need_mib} MiB (Q38_CALIB_MIN_FREE_MIB overrides)" >&2
exit 2
fi
fi
ARGS+=(--model "$MODEL_IN" --port 7770 ARGS+=(--model "$MODEL_IN" --port 7770
--max-num-seqs "$CALIB_SEQS" --max-num-batched-tokens "$CALIB_MNBT") --max-num-seqs "$CALIB_SEQS" --max-num-batched-tokens "$CALIB_MNBT")
# The tap is only real if this directory exists and the engine can write it: # The tap is only real if this directory exists and the engine can write it:
@@ -417,6 +452,19 @@ cp -f "$TEMPLATE" "$TPL" || {
# where the service reported `active` while its engine starved on the rig's VRAM. # where the service reported `active` while its engine starved on the rig's VRAM.
restore() { restore() {
podman rm -f "$C" >/dev/null 2>&1 || true podman rm -f "$C" >/dev/null 2>&1 || true
# Removing the container does NOT kill its engine. Measured 2026-10-30: a rig
# radiance kept /dev/kfd after `podman rm -f`, production's
# ar_oneshot_2rank_wht6 init then failed `device error`, the engine exited 0,
# and Restart=always looped it 100 times with /health never answering. The
# service is stopped for the whole run, so any radiance still holding the
# device here is an orphan — kill it BY PID (pkill -f matches the caller's
# own command line, which is how a session gets killed instead of the orphan).
for p in $(fuser /dev/kfd 2>/dev/null); do
if [ "$(cat "/proc/$p/comm" 2>/dev/null)" = radiance ]; then
echo "restore: killing orphaned radiance pid $p still holding /dev/kfd" >&2
kill -9 "$p" 2>/dev/null || true
fi
done
sudo systemctl start "$SERVICE" >/dev/null 2>&1 || true sudo systemctl start "$SERVICE" >/dev/null 2>&1 || true
} }
trap restore EXIT trap restore EXIT
+1 -1
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# serve-frank-dragoon.sh — frank-dragoon endpoint: halogen-flash-server 0.16.2 # serve-frank-dragoon.sh — frank-dragoon endpoint: halogen-flash-server 0.18.20
# (closed-source engine, image ghcr.io/peonist-ai/halogen-flash-server, # (closed-source engine, image ghcr.io/peonist-ai/halogen-flash-server,
# digest-pinned in frank-dragoon.env) serving Quat3rnion's NATIVE v2 # digest-pinned in frank-dragoon.env) serving Quat3rnion's NATIVE v2
# conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next on Dankclaw's # conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next on Dankclaw's
+3 -3
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Serve prism-ml/Ternary-Bonsai-2-9B-gguf (PQ2_0 ternary quant WITH the MTP head # Serve prism-ml/Ternary-Bonsai-2-9B-gguf (PQ2_0 ternary quant WITH the MTP head
# in the file) as frank-zealot on Dankclaw's RTX 3090 via llama.cpp (PrismML # in the file) as frank-zealot on Dankclaw's RTX 3090 via llama.cpp (PrismML
# vendor build 10777, CUDA sm_86). # vendor build 12118 (0.7.0-dev, commit 024474448), CUDA sm_86).
# #
# 2026-10-18 (Frank): Bonsai-2 27B PQ2_0 → the 9B MTP export, then the same # 2026-10-18 (Frank): Bonsai-2 27B PQ2_0 → the 9B MTP export, then the same
# evening TWO 262,144-token conversations (--ctx-size 524288 --parallel 2) over # evening TWO 262,144-token conversations (--ctx-size 524288 --parallel 2) over
@@ -78,7 +78,7 @@ if [[ -f "$ENV_FILE" ]]; then
fi fi
# --- defaults (used when the env file does not set them) ------------------- # --- defaults (used when the env file does not set them) -------------------
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism-b10777/llama-server}" : "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism-0244744/bin/llama-server}"
: "${MODEL_FILE:=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf}" : "${MODEL_FILE:=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf}"
: "${ALIAS:=frank-zealot}" : "${ALIAS:=frank-zealot}"
: "${BIND_ADDR:=0.0.0.0}" : "${BIND_ADDR:=0.0.0.0}"
@@ -119,7 +119,7 @@ fi
# above are what pay for it. The draft cache dtype IS a further knob # above are what pay for it. The draft cache dtype IS a further knob
# (--spec-draft-type-k / --spec-draft-type-v, both default f16 — verified in this # (--spec-draft-type-k / --spec-draft-type-v, both default f16 — verified in this
# build's --help) and stays at its default here. Card gate: quantized V alongside # build's --help) and stays at its default here. Card gate: quantized V alongside
# MTP needs fork build >= prism-b10776, which is why LLAMA_BIN is the 10777 # MTP needs fork build >= prism-b10776; LLAMA_BIN is 12118 (b10777 kept as rollback)
# prebuilt; the type-142 loader lock keeps it pinned regardless. # prebuilt; the type-142 loader lock keeps it pinned regardless.
# SPEC_TYPE=none is the rollback and strips the whole set (flags move together). # SPEC_TYPE=none is the rollback and strips the whole set (flags move together).
: "${SPEC_TYPE:=draft-mtp}" : "${SPEC_TYPE:=draft-mtp}"
+10
View File
@@ -94,6 +94,16 @@ for path in "${CHANGED[@]}"; do
dankserver/.env.example|dankclaw/.env.example) dankserver/.env.example|dankclaw/.env.example)
add_test tests/env_consistency.bats add_test tests/env_consistency.bats
;; ;;
# The SSD-backed metadata view: the installer, the host mount unit, and the
# two media units that bind the view rather than the raw array.
scripts/media-view.sh|dankserver/host/*)
add_test tests/media_overlay.bats
add_test tests/dankserver_quadlet.bats
add_test tests/deployment_scoping.bats
;;
dankserver/jellyfin/quadlet/*|dankserver/plex/quadlet/*)
add_test tests/media_overlay.bats
;;
shared/homepage/*) shared/homepage/*)
add_test tests/homepage_caddy.bats add_test tests/homepage_caddy.bats
add_test tests/homepage_config.bats add_test tests/homepage_config.bats
@@ -195,8 +195,9 @@ deploy.sh, never a live env poke.
check every namespace (`sudo podman ps -a`) and native units check every namespace (`sudo podman ps -a`) and native units
(`systemctl list-units`) before declaring a service absent. Compare (`systemctl list-units`) before declaring a service absent. Compare
compose mount sources against the real filesystem — compose files name compose mount sources against the real filesystem — compose files name
phantom paths (e.g. `/mnt/media/*` vs the real LUKS array at phantom paths (the legacy `dankserver/compose.yaml` still names `/opt/jellyfin/*`
`/mnt/entropy`). Fix compose through git first, deploy second. for a service whose roots moved to `/home/dank/Containers/jellyfin`). Fix
compose through git first, deploy second.
2. **Pre-create bind-mount sources** (`sudo mkdir -p` + `chown 1000:1000`) 2. **Pre-create bind-mount sources** (`sudo mkdir -p` + `chown 1000:1000`)
or creation fails `statfs …: no such file or directory`. or creation fails `statfs …: no such file or directory`.
3. **Staged bring-up:** infra first (`up -d mariadb redis`), poll 3. **Staged bring-up:** infra first (`up -d mariadb redis`), poll
+18 -6
View File
@@ -1,9 +1,9 @@
--- ---
type: reference type: reference
title: Jellyfin title: Jellyfin
description: "Jellyfin on Dankserver: quadlet unit, entropy-volume data layout, ports, client access, hardware-transcode status." description: "Jellyfin on Dankserver: quadlet unit, overlay metadata view layout, ports, access, hardware-transcode status."
tags: [jellyfin, dankserver, media-server] tags: [jellyfin, dankserver, media-server]
timestamp: 2026-09-03T00:00:00Z timestamp: 2026-10-30T00:00:00Z
--- ---
# Jellyfin # Jellyfin
@@ -26,7 +26,7 @@ public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
| Image | `ghcr.io/jellyfin/jellyfin:latest` | | Image | `ghcr.io/jellyfin/jellyfin:latest` |
| Network | `dankserver_appnet`; publishes 8096/tcp, 8920/tcp, 7359/udp, 1900/udp | | Network | `dankserver_appnet`; publishes 8096/tcp, 8920/tcp, 7359/udp, 1900/udp |
| Data roots | `/home/dank/Containers/jellyfin/config:/config`, `/home/dank/Containers/jellyfin/cache:/cache` | | Data roots | `/home/dank/Containers/jellyfin/config:/config`, `/home/dank/Containers/jellyfin/cache:/cache` |
| Media | `Volume=/mnt/entropy:/data:ro` — the whole [[Entropy Volume]] mounted read-only; nothing automated writes to it | | Media | `Volume=/var/lib/media-meta/view:/data` — the [[Media Metadata View]] overlay, **writable**; sidecars land on the NVMe, the [[Entropy Volume]] array underneath stays read-only and untouched |
| GPU | none — no `AddDevice`; CPU decode only (jellyfin-ffmpeg path) | | GPU | none — no `AddDevice`; CPU decode only (jellyfin-ffmpeg path) |
| Env quirks | `DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true`, `LD_PRELOAD=libjemalloc.so.2`, `MALLOC_TRIM_THRESHOLD_=131072` kept verbatim from the compose-era container (zero-drift cutover) | | Env quirks | `DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true`, `LD_PRELOAD=libjemalloc.so.2`, `MALLOC_TRIM_THRESHOLD_=131072` kept verbatim from the compose-era container (zero-drift cutover) |
| Health | `curl -f http://localhost:8096/health` every 30 s | | Health | `curl -f http://localhost:8096/health` every 30 s |
@@ -36,11 +36,23 @@ public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
``` ```
/home/dank/Containers/jellyfin/config -> /config (JELLYFIN_DATA_DIR) /home/dank/Containers/jellyfin/config -> /config (JELLYFIN_DATA_DIR)
/home/dank/Containers/jellyfin/cache -> /cache (JELLYFIN_CACHE_DIR, XDG_CACHE_HOME) /home/dank/Containers/jellyfin/cache -> /cache (JELLYFIN_CACHE_DIR, XDG_CACHE_HOME)
/mnt/entropy -> /data (read-only media volume) /var/lib/media-meta/view -> /data (the SSD-backed overlay view)
``` ```
Library folders point at `/data` paths; because the mount uses the same Library folders point at `/data` paths, which are the view's paths: stable
absolute path as on the host, library paths survive container recreation. across container recreation, and identical to the raw array layout below it.
`Local metadata` — save metadata to media folders — is enabled, and it works
because `/data` is the [[Media Metadata View]] rather than the read-only array:
`movie.nfo`, `tvshow.nfo`, `episode.nfo` and artwork are written once, on the
NVMe upper layer, and [[Plex]] reads the same files. Jellyfin is the fleet's
only metadata writer.
**Failure signature:** `System.IO.IOException: Read-only file system :
'/data/Movies/<film>/movie.nfo'` means the container is binding the raw array
instead of the view — the view is unmounted or the quadlet drifted. That used to
be the expected state (pre-view, every item update logged it); today it is a
fault. Run `scripts/media-view.sh status`.
## Ports & access ## Ports & access
@@ -0,0 +1,117 @@
---
type: reference
title: Media Metadata View
description: "The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback."
tags: [dankserver, media-server, jellyfin, plex, storage]
timestamp: 2026-10-30T00:00:00Z
---
# Media Metadata View
`/var/lib/media-meta/view` is an OverlayFS merge of the media library: the RAID6
array below, a directory on the NVMe above. It exists so [[Jellyfin]] can write
Kodi-format sidecars (`movie.nfo`, `tvshow.nfo`, `episode.nfo`, artwork) that
[[Plex]] can read. Sidecar metadata is only useful sitting next to the media
file, and the array is read-only and slow — so the media bytes stay there and the
*writes* land on SSD. One tree to both servers, one copy of the media.
The mount point is not a volume and holds nothing while unmounted. It sits inside
`/var/lib/media-meta` so that layers, kernel bookkeeping, the reap trash and the
merged door are one directory — and so the mount table grows no new `/mnt` entry
next to the array.
## Shape
| Item | Value |
|---|---|
| Lower layer | `/mnt/entropy` — the RAID6 array, source of truth, never written |
| Upper layer | `/var/lib/media-meta/upper` — NVMe; every write lands here |
| Work dir | `/var/lib/media-meta/work` — overlayfs internals, same filesystem |
| Mount point | `/var/lib/media-meta/view` — the merged tree both servers bind |
| Unit (repo) | `dankserver/host/systemd/media-meta-view.mount` — the source of truth |
| Unit (host) | `var-lib-media\x2dmeta-view.mount` — systemd names a mount unit after its escaped path (`-` becomes `\x2d`); `media-view.sh install` writes the repo file under that name |
| Installer | `scripts/media-view.sh` — `plan` / `install` / `verify` / `status` / `reap` |
| Written by | Jellyfin only (Plex binds read-only); nothing else in the fleet touches it |
Both consumers guard the view with
`UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view`, and
`scripts/deploy.sh dankserver --service jellyfin|plex` calls `media-view.sh
install` + `verify` before it restarts either container.
| Consumer | Bind | Mode | Why that path |
|---|---|---|---|
| [[Jellyfin]] | `/var/lib/media-meta/view:/data` | rw | It is the metadata writer; its sidecars become SSD bytes |
| [[Plex]] | `/var/lib/media-meta/view:/mnt/entropy` | ro | Every path in the PMS database was recorded as `/mnt/entropy/...`; only the HOST side of the bind moved, so its database still resolves |
Putting the sidecars inside Plex's reach is not the same as making Plex use them:
a library parses `.nfo` only with its *Local media information* agent enabled, and
this change touches no library setting.
## Commands
```bash
scripts/media-view.sh plan # print the resolved shape, touch nothing
sudo scripts/media-view.sh install # create dirs, install the unit, mount, enable
scripts/media-view.sh verify # is the view really the overlay? (deploy gate)
scripts/media-view.sh status # mount options + upper-layer size/file count
scripts/media-view.sh reap # list ghost metadata (media moved, sidecars left)
sudo scripts/media-view.sh reap --apply # move ghosts to the SSD trash, never delete
```
`--upper-base DIR` and `--lower DIR` relocate the layers for `plan` and `reap`;
the defaults are the host values above.
## Laws
- **The upper layer must sit on non-rotational storage.** `install` and `verify`
refuse via `lsblk -dno ROTA` if the upper base lives on the array: an overlay
whose upper is the spinners is the original problem with extra steps, and it
silently turns every metadata write back into a seek.
- **Never mount the view over an unmounted lower.** `RequiresMountsFor=/mnt/entropy`
plus the container `ConditionPathIsMountPoint` guards: an empty `lowerdir`
presents an *empty library* as truth, and a media scan would then clear the
database of everything. systemd does not create a mount point, so `install`
creates `view` itself; without it the mount simply fails.
- **Never delete through the view.** A delete writes an overlayfs whiteout
(`.wh.<name>`) in the upper layer, which hides the array's copy without
removing it — the file reappears the moment the view is unmounted, and the
upper layer now carries a landmine. Deletions belong on the raw path
(`/mnt/entropy`), which the relayout tool already treats as the only writer.
`reap` refuses to remove whiteouts and only warns about them.
- **A relayout must be followed by `reap`.** Renames happen on the raw path, so
sidecars the view already wrote stay at the old coordinates: a ghost folder
the servers browse and find nothing in. `reap` lists upper-layer directories
with no counterpart below and moves them (with their subtree) into a dated
`.AGENTTRASH/` under the SSD base. It is recursive and file-safe: a sidecar
that exists only above is normal, so only *directories* are candidates.
- **File-watcher events do not cross the overlay.** Plex's NFO agent and
Jellyfin's real-time monitor see inotify events for writes that land *in the
upper layer*, not for changes on the array underneath. After any relayout or
externally-written metadata, trigger a library refresh — see
[[Media Relayout]] for the refresh procedure.
## Failure signatures
| Symptom | Meaning |
|---|---|
| `jellyfin.service` / `plex.service` show `ConditionPathIsMountPoint=... was not met` and stay inactive | The view is not mounted. Do not start them anyway: `scripts/media-view.sh install`, then `verify`. |
| Jellyfin log: `Read-only file system : '/data/Movies/…/movie.nfo'` | The container is binding the raw array, not the view — the quadlet drifted back or the view is gone. `media-view.sh status`. |
| A folder appears in a media server with artwork/`.nfo` but no video | Ghost metadata from a relayout. `media-view.sh reap`. |
| `verify` prints `refused: …/view is not a mountpoint` | A plain directory is sitting where the overlay belongs. That is the fail-closed guard working; `install` on the box. |
| `systemctl status` finds no such unit | Look under the escaped name, `var-lib-media\x2dmeta-view.mount`, not the repo file name. |
## Rollback
Revert the commit that pointed the two quadlets at the view and re-deploy both
services — they bind `/mnt/entropy` read-only again, which is the pre-view shape.
The NVMe upper layer can stay: nothing reads it once the binds are back, and it
holds no state either server needs. Unmounting alone
(`systemctl stop 'var-lib-media\x2dmeta-view.mount'`) is safe while no container
binds it, and the array is never modified by any of this.
## Links
- [[Jellyfin]] — the metadata writer; its data layout section.
- [[Plex]] — the read-only consumer; in-place bind-mount history.
- [[Media Relayout]] — the tool that moves media on the raw path and the reason `reap` exists.
- [[Entropy Volume]] — the array underneath.
@@ -0,0 +1,281 @@
---
type: reference
title: Media Relayout
description: "Census, plan and apply the movie library into Jellyfin format, and move TV out of it: the gates, refusals, rollback."
tags: [dankserver, media-server, jellyfin, plex, procedure]
timestamp: 2026-10-30T00:00:00Z
---
# Media Relayout
The movie library on [[Entropy Volume]] arrived as release folders —
`Some.Movie.2019.1080p.WiKiTaLx.x265-FOO/` with the video inside named after
the release. [[Jellyfin]] matches a movie by folder and file name, so the
release noise cost posters, extras and versions. These scripts turn that
library into Jellyfin's own shape — and move TV back out of it — and nothing
else in the repo touches a media file.
| Script | Touches media? | Output |
|---|---|---|
| `scripts/media-census.py` | reads headers only | `enum`, `probed`, `plex` TSVs |
| `scripts/media-plan.py` | no — TSVs in, TSVs out | the plan + the exception report |
| `scripts/media-apply.py` | **renames, never copies**; sweeps only what a run emptied | the moves, plus a journal |
| `scripts/media-tv.py` | no — TSV out | a plan that moves TV into the TV library |
The seam is `tests/media_relayout.bats`; the target contract is quoted from
[jellyfin.org/docs/general/server/media/movies](https://jellyfin.org/docs/general/server/media/movies)
in `media-plan.py`'s docstring.
## Procedure
Run the census on the machine that mounts the library. `probe` is the slow step
(a header read per container) and is `nice`d — it is safe beside a live service
because it never competes for the GPU and never writes inside the library.
```bash
C=scripts/media-census.py
$C enumerate --root /mnt/entropy/Movies --out /tmp/media-enum.tsv
$C probe --enum /tmp/media-enum.tsv --out /tmp/media-probed.tsv --jobs 6
$C plex --root /mnt/entropy/Movies --out /tmp/plex-movies.tsv
```
`plex` reads the Plex metadata cache (`--db` defaults to the path the
[[Plex]] container bind-mounts at `/var/lib/plexmediaserver`) with
`mode=ro&immutable=1`: a live server is never locked or written.
Then plan — this reads only the three TSVs, so it costs seconds and can run
anywhere, including a laptop:
```bash
scripts/media-plan.py --root /mnt/entropy/Movies \
--enum /tmp/media-enum.tsv --probed /tmp/media-probed.tsv \
--plex /tmp/plex-movies.tsv --out /tmp/jf-plan.tsv --exceptions /tmp/jf-exceptions.txt
```
**Read `/tmp/jf-exceptions.txt` before applying.** It is not a log: every line
is a file the plan could not name a film for, a pile that needs a README, or a
collision it refused. The printed gate lines are the go/no-go:
`problems`, `bad-names`, `occupied`, `double-claim`, `left-unmoved` must all be
0 and `unplanned=0` must appear in the report. Any non-zero number means a file
that would be filed wrong, and the plan is not run until it is understood.
```bash
scripts/media-apply.py apply --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies # dry, writes nothing
scripts/media-apply.py apply --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies --apply
scripts/media-apply.py verify --plan /tmp/jf-plan.tsv --journal /mnt/entropy/media-relayout.jsonl
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies # dry
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies --apply
scripts/media-apply.py rollback --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies --apply
```
An apply also leaves the release folders it emptied standing where they were —
hundreds of them at the library root — which is what `sweep` removes, dry-run
first, from the same journal.
After it lands, both servers need a scan or the change is invisible.
```bash
# Run this from the host shell through `podman exec`, never from a neighbour
# container: Plex's HTTPD closes without a single byte on any request that
# arrives from another container's netns (10.89.x.x reads as "empty reply", not
# as a 401). The token rides stdin, never argv, where any process on the host
# could read it out of /proc/<pid>/cmdline.
umask 077
podman exec plex sh -c "sed -n 's/.*PlexOnlineToken=\"\\([^\"]*\\)\".*/\\1/p' \
'/config/Library/Application Support/Plex Media Server/Preferences.xml'" >/tmp/px.tok
podman exec -i plex sh -c 'read -r T
for k in $(curl -s -H "X-Plex-Token: $T" -H "Accept: application/json" \
http://127.0.0.1:32400/library/sections | grep -o "\"key\":\"[0-9]*\"" | cut -d\" -f4); do
curl -s -o /dev/null -w "plex section $k: %{http_code}\n" -X POST \
-H "X-Plex-Token: $T" "http://127.0.0.1:32400/library/sections/$k/refresh"
done' </tmp/px.tok
```
[[Jellyfin]]'s scan key lives in `~/.config/secrets/jellyfin` (mode 600; the
same value is `JELLYFIN_API_KEY` in the ignored `dankserver/.env`). It rides a
mode-600 curl config, never a header argument, and it must use the
`MediaBrowser` form — Jellyfin 12.4 refuses a bare `X-Emby-Token` for any key:
```bash
umask 077
{ printf 'header = "Authorization: MediaBrowser Token=\\"%s\\""\n' \
"$(cat ~/.config/secrets/jellyfin)"
echo 'url = "http://127.0.0.1:8096/Library/Refresh"'
echo 'request = "POST"'; } >/tmp/jf.hdr
curl -s -K /tmp/jf.hdr -o /dev/null -w "jellyfin scan: %{http_code}\n"
shred -u /tmp/jf.hdr
```
`204` is acceptance; the scan then runs on the server's own schedule.
## TV in the movie library
A box set that arrives in the movie share stays there until someone moves it:
`media-plan.py` refuses to file TV and parks each series in
`_TV needs a TV library - <series>`. `scripts/media-tv.py` is the mover. It
walks the movie library itself (no census needed), groups episode-shaped names
by series, and writes a plan in the executor's own format:
```bash
scripts/media-tv.py --root /mnt/entropy/Movies --library /mnt/entropy/TVShows \
--out /mnt/entropy/media-tv-plan.tsv
scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv --root /mnt/entropy
scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv --root /mnt/entropy \
--journal /mnt/entropy/media-relayout-tv.jsonl --apply
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout-tv.jsonl \
--root /mnt/entropy --apply
```
The shape it writes is the one both matchers read — `Monk (2002)/Season 05/
Monk - S05E03 - Mr. Monk and the Big Game.mkv` — keeping the file's own name,
because the publisher's spelling beats a re-derivation, with the year taken
from the box-set folder. What it refuses it prints and leaves alone:
- fewer than `--min` (5) files under one series name: one file named `Part 2`
is a film, not a series;
- an episode whose name states no season number — no invented `Season 01`;
- a destination two files both claim: the executor refuses the whole plan
before its first rename, and only Frank decides which copy is the keeper.
- an episode the destination library already holds at the same byte count. The
movie share carried a second copy of the whole Monk box set — 124 episodes
both sides, the same episode list, every byte count identical, 31.5 GiB each
way, and head+tail blake2b identical on a 6-episode sample. The mover leaves
such a series in place and says so; a duplicate is deleted or kept by Frank,
never copied into one library by a tool.
A sidecar that shares an episode's own name (`.srt`, `.nfo`) moves with it.
Below the threshold the mover prints and leaves, so a lone downloaded episode of
a real show stays in the movie library on purpose — six did on 2026-10-30.
Filing one by hand is the same executor on a hand-written plan and the same
journal; the trap is the series year. Read it from the file's own name or the
folder it owns, never from the title whose bonus folder it happens to be parked
in — `The Cleveland Show S03E07` inside `Die Hard (1988)/extras/` is not a 1988
show. A name whose marker is `Part 2` is a film and stays put.
## Laws
- **A moved file is a 404 in [[Plex]] until its section is rescanned.** The
database keeps the release-folder path, so playback fails `Couldn't find the
file to stream` while the scanner works through the section. Never read such a
404 as a lost file: the file is where the journal says it is.
- **Plex's matched title outranks the filename**, and a Plex year of `0` means
*indexed but never matched* — trusting it named folders `Title (0)`, which is
a Jellyfin item with no year, no poster and no identity. A year that is not a
plausible year is no information.
- **`<>:"/\|?*` never survive a name.** Plex's own title for *V/H/S/2* is
`V/H/S/2`; kept verbatim it became `Movies/V/H/S/2 (2013)/` — a tree five
levels deep that neither matcher reads as a movie.
- **A folder inside a movie's folder is bonus territory even when its name is
misspelled.** `Saboteur (1942)/Exrtas/Saboteur - A Closer Look` is a typo for
`Extras`; the vocabulary rule never fired, the featurette became a movie
folder, and its words then out-scored the real film and drew that box set's
other extras into it. The rule that replaced it needs no metadata: the folder
holding the longest film owns the shorter video filed under it, unless that
file's own name carries a different year — which is another film speaking.
- **Matching words are split on everything that is not a letter or a digit.**
Splitting on separators but not on `/` left tokens like `(1960)/extras` that
no other name could ever agree with, so a film could not be found by the
folder holding it, and extras drifted into a neighbour. A bare number counts
only when it could be a release year, so `1080` and `2160` never pose as
agreement between two names.
- **Compare two media libraries with the series engine, never a path
substring.** Filtering paths for `monk` also returns *Monkey Dust*, and
letting whichever file a walk happens to reach first stand for an episode
paired two libraries into a fake duplicate — the same sloppiness pointed the
other way deletes a file that is unique. The comparison is `media-titles.ep_of`
on both sides: series, season, episode, bytes, and a hash before anything is
trashed.
- **A deletion is staged on the item's own filesystem.** The trash rule names
`~/.AGENTTRASH`, but `/home` and `/mnt/entropy` are different devices: a 31 GiB
stage there is a copy rather than a rename, it has to fit in the 187 GiB of
free space, and `rename(2)` refuses it anyway. Trash goes beside the item —
`/mnt/entropy/.AGENTTRASH/<date>-<topic>/` — so it costs one inode change and
rolls back exactly like the move that made it.
- **A Jellyfin 401 is the header, not the key.** 12.4 authenticates only
`Authorization: MediaBrowser Token="…"` and refuses a bare `X-Emby-Token` for
every key, valid or not. This page carried "no credential on this box can
trigger a scan, the two Seerr keys answer 401" for weeks because of that
header — the keys were innocent. Before believing a 401, prove the key against
the server's own table: `select count(*) from ApiKeys where AccessToken = ?`
on `jellyfin.db` opened `mode=ro&immutable=1`.
- **A scan that must see a new folder has to be recursive.** `POST
/Library/Refresh` is the incremental pass, and inotify does not cross a
rootless podman bind, so folders created on the host stay invisible to it. The
move is `POST /Items/{id}/Refresh?Recursive=true` (204) per library, the id
from `GET /Library/VirtualFolders` — its field is `ItemId`, not `Id` — and
that same record's `RefreshStatus` is the progress read.
- **A file whose name carries no words is bonus material, not a folder.**
`.Extra 1 ( 1H 27min multisub ).mkv` parses to `1`; a movie folder named `1`
is not a movie. It becomes a review line telling a human to name the file.
- **The executor renames and nothing else.** `rename(2)` cannot cross a
filesystem, so a plan that would is refused rather than silently copying tens
of terabytes.
- **The journal is a write-ahead log: the intent is fsynced before the rename.**
An interrupted run therefore leaves only two states — an intent whose move
happened, and an intent whose move never ran — and the next run finishes the
job without moving anything twice. A file that moved with no journal line did
not move through this tool, and the tool refuses to guess on its behalf.
- **The journal lives BESIDE the library root, never inside it** (default:
`media-relayout.jsonl` next to the root): a log inside the library is a file
Jellyfin indexes.
- **Refusals are all-or-nothing.** An occupied target, a vanished source or a
duplicate destination stops the run *before the first rename*, so the operator
fixes one thing and re-runs instead of untangling a half-applied plan.
`--force` relaxes only the opinion guards; it never permits overwriting.
- **A sweep deletes only what the journal proves the run emptied.** A folder is
eligible when a journalled file moved out of it or out of something under it,
it now holds nothing but empty directories, no journalled destination lives
beneath it, and it is not a folder the run created. A folder nobody moved —
someone's placeholder — cannot be selected at all, and a symlink anywhere
inside disqualifies the subtree, because the thing behind a link is not this
tool's to delete. Every removal is journalled as `rmdir` and `rollback`
recreates the folder, so the sweep is undone by the same command as the moves.
Proof carries downward: inside a folder the journal proves emptied, every
empty directory goes with it. A release's `Screens/` or `Sample/` is never a
move source itself, so 146 of them held their emptied parents on disk after
the first sweep.
A failed `rmdir` is not a removal: only a folder that is really gone leaves
scope, so the next sweep finishes what a permission problem stopped.
- **An episode-shaped name marks a SERIES, not a film — and the series is the
name BEFORE the marker.** `Monk - S05E03 - Mr. Monk and the Big Game` is series
`Monk`. Keyed on the whole title instead, every episode became a group of one
and never reached the five-episode size that calls something a series, so a
box set was filed as 124 movies in 124 folders. The threshold still earns its
keep: a lone film named `Part 2` stays a film. Episodes of a series are parked
in `_TV needs a TV library - <series>` for a human to move to a TV source —
the movie tool never files TV.
- **One journal is one undo.** The TV move runs on its own journal, and with
`--root` set to the share that holds BOTH libraries: the executor refuses any
path outside `--root`, and a journal shared with the movie relayout would
undo both runs when only one of them is wanted.
- **Sidecar metadata does not follow a rename, because it lives on another
filesystem.** The media servers write `.nfo` and artwork through the
[[Media Metadata View]] (NVMe upper layer); this tool moves bytes on the raw
array path. After any sweep or TV move, run `scripts/media-view.sh reap`, then
the same command with `--apply`: the sidecars left at the old coordinates become
ghost folders — artwork and an `.nfo` with no video — that both servers browse
and report as content.
- **Nothing deletes through the metadata view.** A delete there writes an overlayfs
whiteout in the upper layer, which hides the array's copy without removing it;
the file returns when the view is unmounted. Deletions in this pipeline happen
on the raw path only, which is what the journal can undo.
## Rollback
One command, from the journal of the run:
`media-apply.py rollback --journal …/media-relayout.jsonl --apply`. It recreates
any folder a sweep removed, restores every source path, removes only the folders
that run created *and that are now empty*, and refuses everything if any moved
file cannot be accounted for. A rolled-back journal refuses to roll back twice.
## See Also
[[Jellyfin]], [[Plex]], [[Media Metadata View]], [[Entropy Volume]], [[Testing Policy]], [[Storage Policy]]
## Citations
- Jellyfin Movie documentation — folder/file naming, extras subfolders,
`Movie (2013) - 216p.mkv` version and `- 4K.mkv` handling:
<https://jellyfin.org/docs/general/server/media/movies>
+6 -4
View File
@@ -4,7 +4,7 @@ description: "Plex Media Server is a **containerized media server on Dankserver*
tags: tags:
- dankserver - dankserver
- plex - plex
timestamp: 2026-10-21 timestamp: 2026-10-30
--- ---
# Plex # Plex
@@ -13,7 +13,8 @@ Plex Media Server is a **containerized media server on Dankserver**: a podman
quadlet (`plex.service`) under the systemd **user** generator, generated from quadlet (`plex.service`) under the systemd **user** generator, generated from
`dankserver/plex/quadlet/plex.container`. It shares the host with a second live `dankserver/plex/quadlet/plex.container`. It shares the host with a second live
media server, [[Jellyfin]] (`jellyfin.service`, listening on 8096); both serve media server, [[Jellyfin]] (`jellyfin.service`, listening on 8096); both serve
from the same read-only Entropy Volume mount. from the same tree — the [[Media Metadata View]] overlay — so the Kodi-format
sidecars Jellyfin writes are the files Plex reads, with no copy and no sync.
**Host:** Dankserver — **Ports:** 32400/tcp (PMS web + API), 32400 is the **Host:** Dankserver — **Ports:** 32400/tcp (PMS web + API), 32400 is the
only port PMS needs for client streaming; LAN clients connect directly, only port PMS needs for client streaming; LAN clients connect directly,
@@ -26,7 +27,7 @@ public traffic can be fronted by Caddy like every other vhost.
| Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) | | Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) | | Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
| Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes | | Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
| Media | `/mnt/entropy` mounted at the same absolute path **read-only** — every library path in the PMS database stayed valid, zero re-scan | | Media | `/var/lib/media-meta/view` (the [[Media Metadata View]] overlay) bound at the same absolute path `/mnt/entropy` **read-only** — every library path in the PMS database stayed valid, zero re-scan |
| Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover | | Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
| Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration | | Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration |
| Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` | | Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` |
@@ -38,7 +39,8 @@ public traffic can be fronted by Caddy like every other vhost.
The migration moved zero bytes: the container bind-mounts the native data The migration moved zero bytes: the container bind-mounts the native data
root where it already lived, and the media archive re-mounts at root where it already lived, and the media archive re-mounts at
`/mnt/entropy` inside the container, so every absolute library path in the `/mnt/entropy` inside the container — only the HOST side of that bind has moved
(array, then the overlay view), so every absolute library path in the
PMS database keeps resolving. The one metadata change: rootless podman PMS database keeps resolving. The one metadata change: rootless podman
cannot map the real host uid 997, so the fleet model applies — container cannot map the real host uid 997, so the fleet model applies — container
root runs as the `dank` user, and the deployer adopts the data root with root runs as the `dank` user, and the deployer adopts the data root with
@@ -10,7 +10,7 @@ timestamp: 2026-09-03
## Server Overview ## Server Overview
Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[frank-dragoon]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon]] (halogen-flash-server 0.16.2 on the iGPU, port 7770) and [[frank-zealot]] (llama.cpp PrismML fork on the RTX 3090, port 7771). Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[frank-dragoon]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon]] (halogen-flash-server 0.18.20 on the iGPU, port 7770) and [[frank-zealot]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
## Hardware Specifications ## Hardware Specifications
### CPU ### CPU
@@ -105,7 +105,7 @@ cable between claw and server is required for 10G.
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM. - Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
- The GPU budget is the GTT aperture: ~114 GiB (`mem_info_gtt_total` in `/sys/class/drm/card1/device/` = 122,402,963,456 B; `card1` is the amdgpu iGPU, `PCI_ID=1002:1586`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget. - The GPU budget is the GTT aperture: ~114 GiB (`mem_info_gtt_total` in `/sys/class/drm/card1/device/` = 122,402,963,456 B; `card1` is the amdgpu iGPU, `PCI_ID=1002:1586`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.16.2** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon]]) — and `frank-zealot` on 7771 ([[frank-zealot]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`. - The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.18.20** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon]]) — and `frank-zealot` on 7771 ([[frank-zealot]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
- The gufo-era note that the active runtime links against the host ROCm (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH belongs to that retired source build. The serving runtimes do not depend on the host ROCm that way: halogen runs inside its digest-pinned image, and `frank-zealot` is a CUDA build on the 3090. The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) was never in the serving path either. - The gufo-era note that the active runtime links against the host ROCm (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH belongs to that retired source build. The serving runtimes do not depend on the host ROCm that way: halogen runs inside its digest-pinned image, and `frank-zealot` is a CUDA build on the 3090. The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) was never in the serving path either.
- The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18. - The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
- 424 GB free on `/` (`df -h`, 2026-10-18) is ample for model storage. - 424 GB free on `/` (`df -h`, 2026-10-18) is ample for model storage.
+23
View File
@@ -571,6 +571,29 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
of falling through its health wait. All three exist because on 2026-10-29 the unguarded chain's of falling through its health wait. All three exist because on 2026-10-29 the unguarded chain's
rig boot died of an OOM, the health loop expired without failing, and the pusher fed four rounds rig boot died of an OOM, the health loop expired without failing, and the pusher fed four rounds
of 10M tokens into **restored production** — 86 minutes of live prefill that recorded nothing. of 10M tokens into **restored production** — 86 minutes of live prefill that recorded nothing.
- **A calibration step has an OPTIMUM, and it is not the widest step that boots.** The tap's cost is
expert re-reads per document (a 7,168-token document cut into four 2,048-token steps re-streams
each layer's routed expert union four times), which argues for the widest step — but the measured
curve turns over: at the 32,768 calibration window one 2-way prefill of the same document ran
**106 tok/s at an 8,192-token step and 76 tok/s at 16,384**, with the computed activation arena
growing 3.39 → 4.50 GiB, so the wider step pays more compute than it saves in reads. The
200,000-window `QSA score plane` refusal that made the wide step look impossible is an artifact of
serving the production window: the same 16,384-token step needs a 512 MiB plane at 32,768.
`Q38_CALIB_MNBT` is the only knob — the KL modes keep production's step, because their numbers are
only comparable at one shape.
- **A container that exists, a port that answers, and a `/health` 200 are three different claims —
bind identity before believing any of them.** Measured 2026-10-30: `podman inspect` named the rig
while the port still belonged to the *dying* production process (`health ok waited=0s`, window dead
at second zero), and the identity check written to catch that compared two different number spaces:
`podman top <ctr> pid` prints **container-internal** pids (`1`, `7`), never host pids, and a rootless
cgroup carries the container ID rather than the service name — so both "matches" can only ever fail.
The binding is `podman top <ctr> hpid` against the listener from
`ss -H -ltnp 'sport = :7770'`, and `fuser /dev/kfd` likewise resolved through podman.
`scripts/qwen38-preflight.sh` owns those facts now (cwd, the digest a tag actually resolves to,
free space on the tap's **own** filesystem, stale grams, every GPU holder and the listener by host
pid), prints `PASS`/`FAIL` per fact, and `qwen38-calib-window.sh` calls it before a card is touched.
Corollary for every ad-hoc driver: a detached launcher that loses its cwd fails with
`No such file or directory` in a log nobody reads — assert cwd, and never bypass the orchestrator.
- **The n-gram table's pin is what starves a rig boot, and the tier is a knob.** On the BF16 payload - **The n-gram table's pin is what starves a rig boot, and the tier is a knob.** On the BF16 payload
the table is 95.37 GiB, `--ngram-placement ram` pins it, and the calibration boot died at the table is 95.37 GiB, `--ngram-placement ram` pins it, and the calibration boot died at
`startup stopped: out of memory` with 230.22 GiB pinned of 246.07 GiB and 8.06 GiB available — `startup stopped: out of memory` with 230.22 GiB pinned of 246.07 GiB and 8.06 GiB available —
+21 -14
View File
@@ -1,6 +1,6 @@
--- ---
type: reference type: reference
title: "frank-dragoon on Dankclaw — halogen-flash-server 0.16.2, native v2 abliterated Qwen3.8-Flash-Next (canonical)" title: "frank-dragoon on Dankclaw — halogen-flash-server 0.18.20, native v2 abliterated Qwen3.8-Flash-Next (canonical)"
description: "The iGPU endpoint: live shape, every era with its rollback path, the laws it paid for, and the files that own it." description: "The iGPU endpoint: live shape, every era with its rollback path, the laws it paid for, and the files that own it."
tags: tags:
- llm - llm
@@ -10,10 +10,10 @@ tags:
- gfx1151 - gfx1151
- halogen - halogen
- endpoint - endpoint
timestamp: 2026-10-21 timestamp: 2026-10-30
--- ---
# frank-dragoon — halogen-flash-server 0.16.2, native v2 abliterated (Dankclaw iGPU) # frank-dragoon — halogen-flash-server 0.18.20, native v2 abliterated (Dankclaw iGPU)
Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container
under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18 under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18
@@ -33,8 +33,8 @@ flags and nothing else. Payload `/home/dank/models/halogen-v2-abliterated/`, ver
| `qwen38-flash-next-vision.hgn` | 897,916,416 | `d62e0ae5…` | peonist `@e053f488`, hardlinked — vision ON | | `qwen38-flash-next-vision.hgn` | 897,916,416 | `d62e0ae5…` | peonist `@e053f488`, hardlinked — vision ON |
| `tokenizer/` (6 flat files) | ~22 MB | `tokenizer.json` LFS | peonist `@e053f488`, hardlinked | | `tokenizer/` (6 flat files) | ~22 MB | `tokenizer.json` LFS | peonist `@e053f488`, hardlinked |
Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a` Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0`
(0.16.2, closed source), container `frank-dragoon-halogen`, published `7770:8731`; the engine's (0.18.20, closed source), container `frank-dragoon-halogen`, published `7770:8731`; the engine's
unauthenticated token port 8730 stays on container loopback. Engine stdout → unauthenticated token port 8730 stays on container loopback. Engine stdout →
`/home/dank/logs/frank_dragoon.log`, **not** the journal. `/home/dank/logs/frank_dragoon.log`, **not** the journal.
@@ -74,6 +74,7 @@ rollback plan — recoverable only until the trash is purged.
| Window | Payload | Runtime | Shape | Replaced because | Rollback | | Window | Payload | Runtime | Shape | Replaced because | Rollback |
|---|---|---|---|---|---| |---|---|---|---|---|---|
| 2026-10-30 → | same v2 abliterated `.hgn` trio (payload untouched) | flash-server **0.18.20** `e35133b2…` | unchanged (2 slots, 262,144 ceiling, pool 401,408, MTP always-on) | vendor moved two minor versions past the 0.16.2 quickstart pin; the payload lock **held** — the same boot served chat, a `tool_calls` turn, vision ("Red", 132 prompt tokens) and `/v1/embeddings`, with the NPU lane's 19-file set verified against the new catalog | env line back to `0c61bf84…` + redeploy |
| 2026-10-06 → 10-17 | Swift1.5 abliterated v2 `.hgn` r2 66,687,678,464 B `9c880b8b…` + same trio | flash-server 0.16.2, same digest | 2 slots, ctx 262,144, pool 524,288 (14.6 GiB), MTP always-on | — | `/home/dank/models/halogen-flashnext-v2/` + `scripts/download-frank-dragoon-flashnext.sh` | | 2026-10-06 → 10-17 | Swift1.5 abliterated v2 `.hgn` r2 66,687,678,464 B `9c880b8b…` + same trio | flash-server 0.16.2, same digest | 2 slots, ctx 262,144, pool 524,288 (14.6 GiB), MTP always-on | — | `/home/dank/models/halogen-flashnext-v2/` + `scripts/download-frank-dragoon-flashnext.sh` |
| 2026-10-05 → 10-06 | peonist `halogen-qwen3.8-27b` `qwen3.8-27b-p1w4d-d2.hgn` 35,865,565,184 B `274c3fc7…` | halogen 0.1.4 `sha256:dc0a39a0…` | 1 slot × 200,700, DFlash2 drafter, `HALOGEN_CACHE_MB=16384`, text-only, censored base | flash-server v2 restored vision, abliteration and MTP | `/home/dank/models/halogen-qwen3.8-27b/` + `scripts/download-frank-dragoon-halogen.sh`; 0.1.4 digest still in podman storage | | 2026-10-05 → 10-06 | peonist `halogen-qwen3.8-27b` `qwen3.8-27b-p1w4d-d2.hgn` 35,865,565,184 B `274c3fc7…` | halogen 0.1.4 `sha256:dc0a39a0…` | 1 slot × 200,700, DFlash2 drafter, `HALOGEN_CACHE_MB=16384`, text-only, censored base | flash-server v2 restored vision, abliteration and MTP | `/home/dank/models/halogen-qwen3.8-27b/` + `scripts/download-frank-dragoon-halogen.sh`; 0.1.4 digest still in podman storage |
| 2026-10-03 → 10-05 | kingjones777 Uncensored `Q6_0_ROCMFPX_AGENT.gguf` 24,856,607,008 B `e966deb3…` + `mtp-Q4_0` `051a1764…` + BF16 mmproj | `kingjones30/ROCmFPX` @ `36e9acd` (`scripts/build-rocmfpx.sh`) | 1 × 262,144 f16 KV, `draft-mtp` n-max 4, vision on | — | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/` + `/home/dank/GitPackages/kingjones-ROCmFPX/build` + `download-frank-dragoon-kingjones-uncensored.sh` | | 2026-10-03 → 10-05 | kingjones777 Uncensored `Q6_0_ROCMFPX_AGENT.gguf` 24,856,607,008 B `e966deb3…` + `mtp-Q4_0` `051a1764…` + BF16 mmproj | `kingjones30/ROCmFPX` @ `36e9acd` (`scripts/build-rocmfpx.sh`) | 1 × 262,144 f16 KV, `draft-mtp` n-max 4, vision on | — | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/` + `/home/dank/GitPackages/kingjones-ROCmFPX/build` + `download-frank-dragoon-kingjones-uncensored.sh` |
@@ -190,9 +191,11 @@ created it — earlier topologies in .
## Contract (wire) ## Contract (wire)
- Live: `/v1/models`, `/health`, `/metrics`, POST `/v1/chat/completions`; `404`: `/ping`, `/props` - Live: `/v1/models`, `/health`, `/metrics`, `/props` (model_alias, total_slots, chat_template),
`/completion`, `/tokenize`. **No `Server-Timing` header and no `/completion` timing object** — measure POST `/v1/chat/completions`, POST `/v1/embeddings` when the NPU lane is armed; `404`: `/ping`,
wall-clock plus `usage.prompt_tokens`. `/completion`; `/tokenize` answers **405 to a GET** (the route exists and wants POST). **No
`Server-Timing` header and no `/completion` timing object** — measure wall-clock plus
`usage.prompt_tokens`.
- Thinking lands in `reasoning_content` and **counts against `max_tokens`**: a budget that dies - Thinking lands in `reasoning_content` and **counts against `max_tokens`**: a budget that dies
mid-thought returns `finish_reason: "length"` with empty `content`, so clients read both fields; size mid-thought returns `finish_reason: "length"` with empty `content`, so clients read both fields; size
vision probes at ≥512 tokens or the model looks blind. Budget for the base reasoning long: 1,476 vision probes at ≥512 tokens or the model looks blind. Budget for the base reasoning long: 1,476
@@ -232,11 +235,15 @@ created it — earlier topologies in .
XDNA2 executes only vendor-precompiled AIE partition programs — that is the wall the TTS attempt died at XDNA2 executes only vendor-precompiled AIE partition programs — that is the wall the TTS attempt died at
([[Frank TTS]]) — and flash-server ships those programs. `HALOGEN_NPU_MODELS=<names>` starts a second engine **in the ([[Frank TTS]]) — and flash-server ships those programs. `HALOGEN_NPU_MODELS=<names>` starts a second engine **in the
same container behind the same port**, and a request picks one by its `model`; leave the variable out and the server same container behind the same port**, and a request picks one by its `model`; leave the variable out and the server
is exactly what runs today. The pinned 0.16.2 image already carries the five-program catalog is exactly what runs today. The pinned 0.18.20 image carries an 18-model catalog
(`/opt/halogen/npu/models.txt`: `decider-0.8b`, `qwen3-embedding-0.6b`, `qwen3-reranker-0.6b`, `qwen3guard-gen-0.6b`, (`/opt/halogen/npu/models.txt`: `decider-0.8b`, `decider-4b`, `qwen3-embedding-0.6b`, `qwen3-embedding-4b`,
`qwen3.5-2b`) and `docs/NPU.md` at tag `v0.16.2` documents it, so **`/v1/embeddings` is one env var and three mounts `qwen3-reranker-0.6b`, `qwen3-reranker-4b`, `qwen3guard-gen-0.6b`, `qwen3guard-stream-0.6b`,
away — no runtime bump needed.** Wider catalog: `qwen3-embedding-4b` and `decider-4b` from 0.17.4, `embeddinggemma-2` `embeddinggemma-2`, `wemm-embedding-4b`, `qwen3.5-2b`, `flux2-klein-4b`, `qwen3-asr-0.6b`, `qwen3-asr-1.7b`,
(text ≤ 8,192 tokens, pictures and audio in one vector space) and `wemm-embedding-4b` from 0.18.3. `d3-lite`, `d3-nano`, `qwen3-tts-0.6b-customvoice`, `qwen3-tts-1.7b-customvoice`), so **`/v1/embeddings` is one
env var and three mounts away — no runtime bump needed.** The catalog is **not** version-stable: 0.18.20
re-cut `qwen3-embedding-0.6b`'s device programs (14 `u*.elf` + an 8,495,512 B `consts.hnpb`, where 0.16.2
had 10 and 5,086,496 B) and the entrypoint verifies size+sha256 per file before it starts, so **an engine
bump re-fetches the staged set** — `scripts/download-frank-dragoon-npu.sh` carries the catalog block to do it.
Embedding contract on that lane: `POST /v1/embeddings`, `model: "qwen3-embedding-0.6b"`, `input` a list of **texts** Embedding contract on that lane: `POST /v1/embeddings`, `model: "qwen3-embedding-0.6b"`, `input` a list of **texts**
(not token ids), 1024 unit-length floats back, the Qwen instruct prefix goes inside the string (not token ids), 1024 unit-length floats back, the Qwen instruct prefix goes inside the string
@@ -282,7 +289,7 @@ lane is **armed and serving** since 2026-10-19 (`HALOGEN_NPU_MODELS=qwen3-embedd
unset, so the container opens no outbound connection; `scripts/download-frank-dragoon-npu.sh` pins every file by unset, so the container opens no outbound connection; `scripts/download-frank-dragoon-npu.sh` pins every file by
size and sha256 against the catalog and the payload mount carries the tree in. size and sha256 against the catalog and the payload mount carries the tree in.
Armed, the boot ledger reads (`/home/dank/logs/frank_dragoon.log`, 2026-10-19): the GPU fabric clock is held at its top level (2000 MHz) **before** the NPU starts; `/dev/accel/accel0` is `crw-rw-rw- nobody:nogroup` under driver `amdxdna` with the distro XRT and its NPU plugin; the program arrives as `15 files checked against this image's record`; then `npu_api: qwen3-embedding-0.6b (embeddings) on the NPU engine at 127.0.0.1:8740 (buckets u0…u9; inputs up to 4096 tokens)` and `up to 64 passes queued … up to 7200 s`. Callers never see `:8740` — the lane answers on the same `:8731` as the Flash model and a request picks the engine by its `model` field, so `/v1/models` lists `frank-dragoon` **and** `qwen3-embedding-0.6b`. The fleet's consumer is open-notebook's embedding default ([[Open Notebook]]); that page owns the app-side shapes (the URL guard, defaults-by-id, the `/api/embed` 500 that still writes the vector). Armed, the boot ledger reads (`/home/dank/logs/frank_dragoon.log`, 2026-10-19): the GPU fabric clock is held at its top level (2000 MHz) **before** the NPU starts; `/dev/accel/accel0` is `crw-rw-rw- nobody:nogroup` under driver `amdxdna` with the distro XRT and its NPU plugin; the program arrives as `19 files checked against this image's record`; then `npu_api: qwen3-embedding-0.6b (embeddings) on the NPU engine at 127.0.0.1:8740 (buckets u0, u1, u10, u11, u12, u13, u2, u3, u4, u5, u6, u7, u8, u9; inputs up to 4096 tokens)` and `up to 64 passes queued … up to 7200 s`. Callers never see `:8740` — the lane answers on the same `:8731` as the Flash model and a request picks the engine by its `model` field, so `/v1/models` lists `frank-dragoon` **and** `qwen3-embedding-0.6b`. The fleet's consumer is open-notebook's embedding default ([[Open Notebook]]); that page owns the app-side shapes (the URL guard, defaults-by-id, the `/api/embed` 500 that still writes the vector).
Price: the programs and their weights sit in host RAM beside the locked weights ("a few GB in all" for the default Price: the programs and their weights sit in host RAM beside the locked weights ("a few GB in all" for the default
set; ~1.1 GiB for `embeddinggemma-2`), and **the Flash model decodes slower while the NPU works** — same memory bus, set; ~1.1 GiB for `embeddinggemma-2`), and **the Flash model decodes slower while the NPU works** — same memory bus,
+11 -3
View File
@@ -1,6 +1,6 @@
--- ---
type: reference type: reference
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Ternary-Bonsai-2-9B PQ2_0-MTP + refusal adapter, 3 × 200704, f16 K + q8_0 V, MTP armed" title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 12118), Ternary-Bonsai-2-9B PQ2_0-MTP + refusal adapter, 3 × 200704, f16 K + q8_0 V, MTP armed"
description: "The RTX 3090 endpoint: live shape exactly as the env ships it, every era with its rollback path, and its card laws." description: "The RTX 3090 endpoint: live shape exactly as the env ships it, every era with its rollback path, and its card laws."
tags: tags:
- llm - llm
@@ -31,7 +31,7 @@ is the bug.
| Projector | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf` — 624,229,664 B, sha256 `e0cb6f6988cea323…` — **Q8_0 is mandatory**, not a preference (see Contract) | | Projector | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf` — 624,229,664 B, sha256 `e0cb6f6988cea323…` — **Q8_0 is mandatory**, not a preference (see Contract) |
| Adapter | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf` — **3,155,424 B**, sha256 `861b9c099fb5cd81…` — a rank-1 refusal-direction LoRA (`adapter.type lora`, 128 tensors, every transformer layer, embedding untouched), applied at load by `--lora-scaled <file>:1`. Its metadata names `Ternary-Bonsai-2-9B-PTQ1_0.gguf` (`3561df7a…`) as the pack it was fitted against — **not this packing, and not on this host**; see Laws | | Adapter | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf` — **3,155,424 B**, sha256 `861b9c099fb5cd81…` — a rank-1 refusal-direction LoRA (`adapter.type lora`, 128 tensors, every transformer layer, embedding untouched), applied at load by `--lora-scaled <file>:1`. Its metadata names `Ternary-Bonsai-2-9B-PTQ1_0.gguf` (`3561df7a…`) as the pack it was fitted against — **not this packing, and not on this host**; see Laws |
| Fetch | `scripts/download-frank-zealot-bonsai9.sh [DEST]` (payload + projector) · `scripts/download-frank-zealot-uncensored-lora.sh [DEST]` (the adapter) — both `--verify-only`-able, `-f`, byte count and sha256 per file | | Fetch | `scripts/download-frank-zealot-bonsai9.sh [DEST]` (payload + projector) · `scripts/download-frank-zealot-uncensored-lora.sh [DEST]` (the adapter) — both `--verify-only`-able, `-f`, byte count and sha256 per file |
| Runtime | `/home/dank/GitPackages/llama-prism-b10777/llama-server` — PrismML **fork**, `0.2.0-dev (build 10777, commit e3de71522)`; the only loader for this payload | | Runtime | `/home/dank/GitPackages/llama-prism-0244744/bin/llama-server` — PrismML **fork**, `0.7.0-dev (build 12118, commit 024474448)`; the only loader for this payload |
| Trio | `scripts/frank-zealot.service` (root, relinked, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`, `TimeoutStopSec=180`) · `scripts/frank-zealot.env` · `scripts/serve-frank-zealot.sh` | | Trio | `scripts/frank-zealot.service` (root, relinked, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`, `TimeoutStopSec=180`) · `scripts/frank-zealot.env` · `scripts/serve-frank-zealot.sh` |
| Address / alias | `BIND_ADDR=0.0.0.0`, `LISTEN_PORT=7771`, `ALIAS=frank-zealot` (a flag, not a fact) | | Address / alias | `BIND_ADDR=0.0.0.0`, `LISTEN_PORT=7771`, `ALIAS=frank-zealot` (a flag, not a fact) |
| Flags | `--ctx-size 602112 --parallel 3` ⇒ 200,704 per slot · `--cache-type-k f16 --cache-type-v q8_0` (asymmetric, **14,406.00 MiB**) · `--flash-attn on` · `--batch-size 6144 --ubatch-size 1024` · `--n-gpu-layers 99` · `--jinja --reasoning auto --reasoning-format deepseek` · `--metrics` · `--mmproj` · **`--lora-scaled Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf:1`** · `--log-verbosity 4` · `--presence-penalty 1.0` · **`--spec-type draft-mtp`** with `--spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0` | | Flags | `--ctx-size 602112 --parallel 3` ⇒ 200,704 per slot · `--cache-type-k f16 --cache-type-v q8_0` (asymmetric, **14,406.00 MiB**) · `--flash-attn on` · `--batch-size 6144 --ubatch-size 1024` · `--n-gpu-layers 99` · `--jinja --reasoning auto --reasoning-format deepseek` · `--metrics` · `--mmproj` · **`--lora-scaled Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf:1`** · `--log-verbosity 4` · `--presence-penalty 1.0` · **`--spec-type draft-mtp`** with `--spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0` |
@@ -136,6 +136,7 @@ with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY c
| Window | Payload | Runtime | Shape | Replaced because | Rollback | | Window | Payload | Runtime | Shape | Replaced because | Rollback |
|---|---|---|---|---|---| |---|---|---|---|---|---|
| 2026-10-30 → | same 9B MTP file, Q8_0 mmproj and refusal adapter (payload untouched) | PrismML fork **12118** (`0.7.0-dev`, `024474448`) | unchanged (3 × 200,704; f16 K + q8_0 V; MTP depth 2) | fork rebuilt from `origin/prism`; the payload gate held — 202 `pq2_0` tensors and 202 Hadamard-folded weights loaded, MTP drafting engaged, chat/tools/vision answered | `LLAMA_BIN` back to `llama-prism-b10777/llama-server` (kept on disk) + redeploy |
| 2026-10-19 → | the same 9B MTP file and projector, **plus** `Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf` (3,155,424 B `861b9c09…`) applied by `--lora-scaled …:1` | PrismML fork 10777 (its verified build) | — (shape unchanged; the adapter is behaviour) | Frank ordered the refusal direction removed on this endpoint | one env line — `LORA_FILE=` (or `LORA_SCALE=0`) — and redeploy; the plain pack is the same file it already loads | | 2026-10-19 → | the same 9B MTP file and projector, **plus** `Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf` (3,155,424 B `861b9c09…`) applied by `--lora-scaled …:1` | PrismML fork 10777 (its verified build) | — (shape unchanged; the adapter is behaviour) | Frank ordered the refusal direction removed on this endpoint | one env line — `LORA_FILE=` (or `LORA_SCALE=0`) — and redeploy; the plain pack is the same file it already loads |
| 2026-10-18 21:06 → 2026-10-19 | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered the third conversation back AND the drafter armed, and exact V funds neither: it bills 18,816 MiB at the pool that pays for the third slot, leaving nothing for the draft's 2,352 | one line each — `CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=f16 BATCH_SIZE=4096 UBATCH_SIZE=2048 SPEC_TYPE=none` — and redeploy; no payload move | | 2026-10-18 21:06 → 2026-10-19 | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered the third conversation back AND the drafter armed, and exact V funds neither: it bills 18,816 MiB at the pool that pays for the third slot, leaving nothing for the draft's 2,352 | one line each — `CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=f16 BATCH_SIZE=4096 UBATCH_SIZE=2048 SPEC_TYPE=none` — and redeploy; no payload move |
| 2026-10-18 13:19 → same evening (the boot `68afc6d` replaced) | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered unquantized KV, and that pool cannot survive it: 24,576 MiB of f16 cache > 24,123 addressable | one env line — `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` — and redeploy; no payload move | | 2026-10-18 13:19 → same evening (the boot `68afc6d` replaced) | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered unquantized KV, and that pool cannot survive it: 24,576 MiB of f16 cache > 24,123 addressable | one env line — `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` — and redeploy; no payload move |
@@ -201,7 +202,14 @@ with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY c
- `serve-frank-zealot.sh` is a direct `ExecStart`, so a missing shebang is a `203/EXEC` boot loop at `Mem peak: 1.9M` with the GPU untouched: `head -1` is a deploy check. - `serve-frank-zealot.sh` is a direct `ExecStart`, so a missing shebang is a `203/EXEC` boot loop at `Mem peak: 1.9M` with the GPU untouched: `head -1` is a deploy check.
- `No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; KV spilled into unified memory halves decode silently, so re-read `nvidia-smi` after any shape change. - `No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; KV spilled into unified memory halves decode silently, so re-read `nvidia-smi` after any shape change.
- **An adapter's provenance is metadata, not a gate.** Its GGUF carries `prism.abliterate.base_gguf = Ternary-Bonsai-2-9B-PTQ1_0.gguf` and that pack's sha256; the endpoint applies the file to the PQ2_0+MTP packing anyway, because the loader matches **tensor names, not hashes**. So an adapter's effect is only ever proven behaviourally (a refusal probe against the live endpoint), never by the card, and an adapter whose rank landed on `token_embd` would be a crash rather than a degradation — the publisher leaves the embedding untouched for exactly that reason. - **An adapter's provenance is metadata, not a gate.** Its GGUF carries `prism.abliterate.base_gguf = Ternary-Bonsai-2-9B-PTQ1_0.gguf` and that pack's sha256; the endpoint applies the file to the PQ2_0+MTP packing anyway, because the loader matches **tensor names, not hashes**. So an adapter's effect is only ever proven behaviourally (a refusal probe against the live endpoint), never by the card, and an adapter whose rank landed on `token_embd` would be a crash rather than a degradation — the publisher leaves the embedding untouched for exactly that reason.
- **The fork pin now has a third reason.** The payload's custom type 142, the b10776 floor that pairs MTP with a quantized V, and now this: older PrismML builds (e.g. `prism-b10709`) refuse the adapter outright with `unsupported prism.hadamard.version: 2`, so a runtime that can load the pack can still be unable to carry the adapter. - **A fork rebuild needs an install RPATH and a bin/lib-aware `LLAMA_BIN`.** `cmake --install` strips the
build-tree rpath, so the installed `llama-server` dies on `libllama-server-impl.so` unless the configure
carries `-DCMAKE_INSTALL_RPATH='$ORIGIN;$ORIGIN/../lib'`; the installed layout is `bin/` + `lib/`, not the
flat directory b10777 shipped, so the env's `LLAMA_BIN` has to name `…/bin/llama-server`. Build 12118 was
made from `origin/prism` with the 10777 configure (Release, `GGML_CUDA=ON`, `sm_86`,
`CMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13`, Ninja) — gcc 15 is the box default and nvcc 12.4 rejects it.
- **The fork pin now has a third reason.** The payload's custom type 142, the b10776 floor that pairs MTP with a quantized V, and now this: older PrismML builds (e.g. `prism-b10709`) refuse the adapter outright with `unsupported prism.hadamard.version: 2`, so a runtime that can load the pack can still be unable to carry the adapter. Build 12118 clears all
three.
## Contract ## Contract
+3 -1
View File
@@ -49,8 +49,10 @@ is a locator: one row, one clause, no numbers.
| [[Copyparty]] | Copyparty is a lightweight file server and media streamer that runs as a single Python binary. | | [[Copyparty]] | Copyparty is a lightweight file server and media streamer that runs as a single Python binary. |
| [[Dankserver Stack]] | Service inventory and routing overview for the media and library host. | | [[Dankserver Stack]] | Service inventory and routing overview for the media and library host. |
| [[Entropy Volume]] | The Dankserver entropy volume is a RAID-backed LUKS filesystem mounted at | | [[Entropy Volume]] | The Dankserver entropy volume is a RAID-backed LUKS filesystem mounted at |
| [[Jellyfin]] | Jellyfin on Dankserver: quadlet unit, entropy-volume data layout, ports, client access, hardware-transcode status. | | [[Jellyfin]] | Jellyfin on Dankserver: quadlet unit, overlay metadata view layout, ports, access, hardware-transcode status. |
| [[mariadb]] | The MariaDB backend for RomM: quadlet unit, network alias, env-file secret handling, and the backup path. | | [[mariadb]] | The MariaDB backend for RomM: quadlet unit, network alias, env-file secret handling, and the backup path. |
| [[Media Metadata View]] | The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback. |
| [[Media Relayout]] | Census, plan and apply the movie library into Jellyfin format, and move TV out of it: the gates, refusals, rollback. |
| [[Plex]] | Plex Media Server is a **containerized media server on Dankserver**: a podman | | [[Plex]] | Plex Media Server is a **containerized media server on Dankserver**: a podman |
| [[redis]] | The Redis cache and task queue for RomM: quadlet unit, network alias topology, and the backup path. | | [[redis]] | The Redis cache and task queue for RomM: quadlet unit, network alias topology, and the backup path. |
| [[RomM]] | RomM (ROM Manager) is a self-hosted ROM library manager and player. | | [[RomM]] | RomM (ROM Manager) is a self-hosted ROM library manager and player. |
+10
View File
@@ -133,6 +133,16 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
[ -z "$output" ] [ -z "$output" ]
} }
@test "a compose edit never restarts a Quadlet-owned media server" {
# dankserver/compose.yaml is an inventory; the media servers' Quadlets own
# their containers. A compose edit that restarted them would cut real viewers
# for a change that cannot affect them.
run python3 "$PLANNER" --target dankserver --paths dankserver/compose.yaml
[ "$status" -eq 0 ]
! printf '%s\n' "$output" | grep -qx jellyfin
! printf '%s\n' "$output" | grep -qx plex
}
@test "compose planner narrows a compose edit to the changed service" { @test "compose planner narrows a compose edit to the changed service" {
local base head local base head
base=$(git -C "$REPO_ROOT" rev-parse HEAD) base=$(git -C "$REPO_ROOT" rev-parse HEAD)
+1 -1
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env bats #!/usr/bin/env bats
# Seam: frank-dragoon serves Quat3rnion's NATIVE v2 conversion (2026-10-11) # Seam: frank-dragoon serves Quat3rnion's NATIVE v2 conversion (2026-10-11)
# of the abliterated Qwen3.8-Flash-Next (a .hgn flash-server payload) on the # of the abliterated Qwen3.8-Flash-Next (a .hgn flash-server payload) on the
# Strix Halo iGPU through the halogen-flash-server CONTAINER (0.16.2, # Strix Halo iGPU through the halogen-flash-server CONTAINER (0.18.20,
# digest-pinned), driven by the root unit's serve script — 2 slots over ONE # digest-pinned), driven by the root unit's serve script — 2 slots over ONE
# 401,408-position KV pool at the native 262,144 ceiling, MTP head always on (HALOGEN_SPEC_ADAPT=0) # 401,408-position KV pool at the native 262,144 ceiling, MTP head always on (HALOGEN_SPEC_ADAPT=0)
# and the vision tower mounted. 2026-10-17 cutover from the Swift1.5-finetune # and the vision tower mounted. 2026-10-17 cutover from the Swift1.5-finetune
+15 -4
View File
@@ -126,11 +126,22 @@ run_stage() {
@test "the shipped env points at the fork runtime, the MTP pair and the refusal adapter" { @test "the shipped env points at the fork runtime, the MTP pair and the refusal adapter" {
# The payload path IS the endpoint's identity, and the runtime is part of # The payload path IS the endpoint's identity, and the runtime is part of
# it here: the file's custom tensor type 142 + prism.hadamard.* metadata # it here: the file's custom tensor type 142 + prism.hadamard.* metadata
# make the PrismML build the only loader, and this build must be >= b10776 # make the PrismML build the only loader, and that build must be >= b10776
# because MTP beside q8_0 V is illegal below it. A retarget that moves the # because MTP beside q8_0 V is illegal below it. A retarget that moves the
# model but leaves the old build behind ships an endpoint that cannot load # model but leaves a stock llama.cpp behind ships an endpoint that cannot
# (type 142) or cannot run the documented MTP arm (the build gate). # load (type 142) or cannot run the documented MTP arm (the build gate).
grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server$' "$ENV_FILE" #
# Assert the fork FAMILY, not the directory one rebuild happens to install:
# a literal path turns every runtime move into a test edit, and the suite
# runs on a box that has no /home/dank at all, so existence cannot be
# checked here either.
local bin
bin="$(sed -n 's/^LLAMA_BIN=//p' "$ENV_FILE")"
if [[ ! "$bin" =~ ^/home/dank/GitPackages/llama-prism-[^/]+/llama-server$ ]] &&
[[ ! "$bin" =~ ^/home/dank/GitPackages/llama-prism-[^/]+/bin/llama-server$ ]]; then
echo "LLAMA_BIN does not name a PrismML fork build: '$bin'" >&2
return 1
fi
grep -qE '^MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0\.gguf$' "$ENV_FILE" grep -qE '^MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0\.gguf$' "$ENV_FILE"
grep -qE '^MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0\.gguf$' "$ENV_FILE" grep -qE '^MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0\.gguf$' "$ENV_FILE"
grep -qE '^LORA_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA\.gguf$' "$ENV_FILE" grep -qE '^LORA_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA\.gguf$' "$ENV_FILE"
+161
View File
@@ -0,0 +1,161 @@
#!/usr/bin/env bats
# Seam: the SSD-backed metadata view (OverlayFS) that lets Jellyfin write
# Kodi-format sidecars for Plex to read, without ever writing to the spindle
# array and without letting either media server start against a missing view.
#
# The contracts, in the order the seam tests them:
# 1. the plan puts the writable layer on the SSD and the media layer below it;
# 2. bad input is refused by name instead of mounting something surprising;
# 3. verify refuses a view that is not mounted (so nothing trusts a directory);
# 4. both media units bind the VIEW and refuse to start without it;
# 5. only those two units touch the view — the LLM and library tenants keep the
# raw volume, so the blast radius is two containers;
# 6. the deployer brings the view up BEFORE it restarts either server;
# 7. the planner routes a change to these files to both media services.
load helpers
VIEW="$REPO_ROOT/scripts/media-view.sh"
JELLY="$REPO_ROOT/dankserver/jellyfin/quadlet/jellyfin.container"
PLEX="$REPO_ROOT/dankserver/plex/quadlet/plex.container"
MOUNT=/var/lib/media-meta/view
UNIT="$REPO_ROOT/dankserver/host/systemd/media-meta-view.mount"
@test "the plan names an overlay with the spindle below and the SSD above" {
run bash "$VIEW" plan
[ "$status" -eq 0 ]
# Lower = the array; upper/work = the NVMe. A user asked for metadata to be
# SSD-resident, so a spindle upper layer would defeat the whole mechanism.
[ "${lines[0]}" = "lowerdir=/mnt/entropy" ]
printf '%s\n' "${lines[@]}" | grep -qx 'upperdir=/var/lib/media-meta/upper'
printf '%s\n' "${lines[@]}" | grep -qx 'workdir=/var/lib/media-meta/work'
printf '%s\n' "${lines[@]}" | grep -Fqx "mountpoint=$MOUNT"
# systemd names a mount unit after its escaped path, backslash and all.
printf '%s\n' "${lines[@]}" | grep -Fqx \
'unit=/etc/systemd/system/var-lib-media\x2dmeta-view.mount'
}
@test "install refuses a caller who cannot write the unit directory or mount" {
# Half an install is worse than none: the unit would exist without the
# directories behind it. Refuse by name before anything is created.
if [ "$(id -u)" -eq 0 ]; then
skip "this seam is running as root"
fi
run bash "$VIEW" install
[ "$status" -ne 0 ]
[[ "$output" == *"root"* ]]
}
@test "verify refuses a view that is not mounted" {
# Every consumer trusts the mount point to BE the library. On a host where it
# is a plain empty directory, verify must fail loudly: that is the difference
# between a scan that finds nothing and a scan that empties a library DB.
if mountpoint -q "$MOUNT"; then
skip "$MOUNT is a real mountpoint on this host"
fi
run bash "$VIEW" verify
[ "$status" -ne 0 ]
[[ "$output" == *"$MOUNT"* ]]
}
@test "both media servers bind the view and refuse to start without it" {
for unit in "$JELLY" "$PLEX"; do
grep -q "^Volume=$MOUNT:" "$unit" || return 1
grep -q "^UnitConfigItem=ConditionPathIsMountPoint=$MOUNT$" "$unit" || return 1
# The raw array must no longer be a container source: that bind is what
# made sidecar sharing impossible (read-only) and metadata slow (seeks).
! grep -q '^Volume=/mnt/entropy' "$unit"
done
# Jellyfin is the writer, so it needs the view writable; Plex only reads the
# sidecars, and keeping its bind read-only costs nothing.
grep -q "^Volume=$MOUNT:/data$" "$JELLY"
grep -q "^Volume=$MOUNT:/mnt/entropy:ro$" "$PLEX"
}
@test "only the two media units reference the view" {
run grep -rl --include='*.container' "$MOUNT" "$REPO_ROOT"/dankserver \
"$REPO_ROOT"/dankclaw "$REPO_ROOT"/frankbook-pro 2>/dev/null
[ "$status" -eq 0 ]
[ "$(printf '%s\n' "$output" | wc -l)" -eq 2 ]
printf '%s\n' "$output" | grep -q 'jellyfin/quadlet/jellyfin.container'
printf '%s\n' "$output" | grep -q 'plex/quadlet/plex.container'
}
@test "the mount unit is an overlay whose upper layer lives on the SSD" {
[ -f "$UNIT" ]
run grep -E '^Options=lowerdir=/mnt/entropy,upperdir=/var/lib/media-meta/upper,workdir=/var/lib/media-meta/work' "$UNIT"
[ "$status" -eq 0 ]
# The unit must mount exactly where the two containers bind, or the guard and
# the bind disagree and a media server browses an empty directory.
run grep -Fqx "Where=$MOUNT" "$UNIT"
[ "$status" -eq 0 ]
# It must wait for the array to be unlocked: mounting an overlay whose lower
# dir is an empty /mnt/entropy would present an empty library, not an error.
run grep -E '^RequiresMountsFor=/mnt/entropy$' "$UNIT"
[ "$status" -eq 0 ]
}
@test "the deployer raises the view before it restarts either media server" {
for svc in jellyfin plex; do
body="$(sed -n "/^deploy_${svc}()/,/^\}/p" "$REPO_ROOT/scripts/deploy-remote.sh")"
[ -n "$body" ]
up="$(printf '%s\n' "$body" | grep -n 'ensure_media_view' | head -1 | cut -d: -f1)"
up_line="${up:-999}"
restart="$(printf '%s\n' "$body" | grep -n "systemctl --user restart $svc.service" | head -1 | cut -d: -f1)"
restart_line="${restart:-0}"
[ "$up_line" -lt "$restart_line" ]
done
}
@test "the planner routes the view files to both media services" {
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankserver --paths \
scripts/media-view.sh dankserver/host/systemd/media-meta-view.mount
[ "$status" -eq 0 ]
[ "$output" = "jellyfin
plex" ]
}
@test "a docs-only change still plans nothing for the view" {
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankserver --paths \
"serverAppConfigs-docs/Dankserver/Media Metadata View.md"
[ "$status" -eq 0 ]
[ -z "$output" ]
}
@test "reap names the metadata whose media has moved and touches nothing" {
# A relayout moves media on the RAW path; the sidecars the view already wrote
# stay behind at the old coordinates. Left alone they become folders a media
# server browses and finds nothing in.
tmp="$(mktemp -d)"
mkdir -p "$tmp/meta/upper/Movies/Ghost (2020)" "$tmp/meta/upper/Movies/Keeper (2020)" \
"$tmp/meta/upper/TVShows/Big Little Lies/Season 01" "$tmp/meta/work" \
"$tmp/array/Movies/Keeper (2020)" "$tmp/array/TVShows"
: > "$tmp/meta/upper/Movies/Ghost (2020)/movie.nfo"
: > "$tmp/meta/upper/Movies/Keeper (2020)/movie.nfo"
: > "$tmp/meta/upper/TVShows/Big Little Lies/Season 01/show.nfo"
: > "$tmp/array/Movies/Keeper (2020)/Keeper (2020).mkv"
run bash "$VIEW" reap --upper-base "$tmp/meta" --lower "$tmp/array"
[ "$status" -eq 0 ]
printf '%s\n' "$output" | grep -qxF 'Movies/Ghost (2020)'
# The series folder is still below, so the ghost is the season inside it — and
# it is reported at the level that is actually missing, not deeper.
printf '%s\n' "$output" | grep -qxF 'TVShows/Big Little Lies'
! printf '%s\n' "$output" | grep -q 'Keeper'
[ -f "$tmp/meta/upper/Movies/Ghost (2020)/movie.nfo" ]
}
@test "reap --apply moves ghosts to the SSD trash and keeps live metadata" {
tmp="$(mktemp -d)"
mkdir -p "$tmp/meta/upper/Movies/Ghost (2020)" "$tmp/meta/upper/Movies/Keeper (2020)" \
"$tmp/array/Movies/Keeper (2020)"
: > "$tmp/meta/upper/Movies/Ghost (2020)/movie.nfo"
: > "$tmp/meta/upper/Movies/Keeper (2020)/movie.nfo"
run bash "$VIEW" reap --apply --upper-base "$tmp/meta" --lower "$tmp/array"
[ "$status" -eq 0 ]
[ ! -e "$tmp/meta/upper/Movies/Ghost (2020)" ]
[ -f "$tmp/meta/upper/Movies/Keeper (2020)/movie.nfo" ]
# Recoverable, not deleted: the item lands in a dated trash under the SSD base.
[ "$(find "$tmp/meta/.AGENTTRASH" -name movie.nfo | wc -l)" -eq 1 ]
}
+623
View File
@@ -0,0 +1,623 @@
#!/usr/bin/env bats
# Seam: the movie-library relayout — scripts/media-census.py (the read-only
# census), scripts/media-plan.py (the plan) and scripts/media-apply.py (the only
# thing that moves files).
#
# The library is 8k files on a 100 %-full array, so the two things this seam
# exists to catch are the ones that cost a re-run of a 20-minute census: a file
# that ends up in the WRONG movie folder, and a move that destroys something.
# Everything below runs on a synthetic library in the test's own tmpdir: no
# ffprobe, no network, no server. The probe step is the one part a fixture
# cannot run (it opens real containers), so the tests feed the planner the
# probed TSV in the column order scripts/media-census.py exports — which pins
# that contract as a side effect.
#
# The plan's own gate lines (unplanned, left-unmoved, double-claim, occupied)
# are asserted as output, because they are how the operator learns the plan is
# trustworthy before pointing the executor at 24 TB.
load helpers
PLAN="$REPO_ROOT/scripts/media-plan.py"
CENSUS="$REPO_ROOT/scripts/media-census.py"
APPLY="$REPO_ROOT/scripts/media-apply.py"
TV="$REPO_ROOT/scripts/media-tv.py"
# probe_row FILE DUR [RES] [AUDIO] [SUBS] — one probed-TSV row: the twelve
# columns media-census.py probe writes (bytes, path, dur, vcodec, res, fps,
# vbit, pix, audio, subs, spare, fmt).
probe_row() {
local p="$1" dur="$2" res="${3:-1920x1080}" aud="${4:-aac+aac}" subs="${5:-0}"
printf '%s\t%s\t%s\thevc\t%s\t23.976\t5000000\t8bit\t%s\t%s\t0\tmatroska,webm\n' \
"$(stat -c %s "$p")" "$p" "$dur" "$res" "$aud" "$subs" >> "$PROBED"
}
# plex_row FILE TITLE YEAR — one Plex metadata row; the planner reads these
# three columns and treats a row without a plausible year as no match at all.
plex_row() {
printf '%s\t%s\t%s\n' "$1" "$2" "$3" >> "$PLEX"
}
# at PATH SIZE — a file with deterministic content, big enough to matter.
at() { mkdir -p "$(dirname "$1")"; head -c "$2" /dev/zero > "$1"; }
setup() {
LIB="$BATS_TEST_TMPDIR/Movies"
ENUM="$BATS_TEST_TMPDIR/enum.tsv"
PROBED="$BATS_TEST_TMPDIR/probed.tsv"
PLEX="$BATS_TEST_TMPDIR/plex.tsv"
OUT="$BATS_TEST_TMPDIR/plan.tsv"
EXC="$BATS_TEST_TMPDIR/exceptions.txt"
TVLIB="$BATS_TEST_TMPDIR/TVShows"
TVPLAN="$BATS_TEST_TMPDIR/tv.tsv"
TVJ="$BATS_TEST_TMPDIR/tv.jsonl"
: > "$PROBED"
printf 'file\tleaf_title\tleaf_year\n' > "$PLEX"
# Two films that share a box set and a word, plus their bonus material:
# the shape that made one film's featurettes land in another's folder.
at "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" 4000
at "$LIB/Alien (1979)/Featurettes/A Bug And A Bite - Making Alien.mkv" 3300000
at "$LIB/Alien (1979)/Extras/Extra - Logo.mkv" 900000
at "$LIB/Alien (1979)/subs/3_Eng.srt" 40000
at "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" 5000000
at "$LIB/Marnie (1964)/Featurettes/The Trouble With Marnie.mkv" 2400000
# A long extra: over 45 minutes, so only the folder may call it bonus.
at "$LIB/Marnie (1964)/Featurettes/Commentary And Archives.mkv" 2600000
# A misspelled bonus folder: no bonus vocabulary, so only the folder that
# holds the feature can decide that this is not a film.
at "$LIB/Marnie (1964)/Exrtas/Marnie - A Closer Look.mkv" 2500000
# A Plex title that carries "/" and a Plex row that was never matched.
at "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv" 4200000
at "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv" 3100000
# Junk the layout has to park somewhere other than a movie folder.
at "$LIB/.DS_Store" 6148
at "$LIB/Alien (1979)/._Alien.1979.1080p.BluRay.x265.10bit.mkv" 220
# A bonus file whose name is a number: never a movie folder named "1".
at "$LIB/Bruce Lee Collection 1971-1981/.Extra 1.mkv" 2700000
# A TV box set sitting in the movie library. This one names each file
# "Show - SxxEyy - Episode Title", so the episode title comes AFTER the
# marker; that convention is what made 124 episodes of one series look like
# 124 different films, each under its own threshold.
for e in 01 02 03 04 05; do
at "$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E$e - Mr Monk and the Rival.mkv" 3000000
done
# The same trouble named the other way round, with no episode title at all.
for e in 01 02 03 04 05; do
at "$LIB/Big Little Lies Season 1/Big Little Lies S01E$e.mp4" 3000000
done
# A film whose own name carries a bare "Part 2": one episode-shaped token is
# not a series, and it must not be dragged into the TV pile.
at "$LIB/The Godfather Part 2 (1974)/The.Godfather.Part.2.1974.1080p.BluRay.mkv" 9000000
probe_row "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" 4700
probe_row "$LIB/Alien (1979)/Featurettes/A Bug And A Bite - Making Alien.mkv" 2700
probe_row "$LIB/Alien (1979)/Extras/Extra - Logo.mkv" 30
probe_row "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" 4800
probe_row "$LIB/Marnie (1964)/Featurettes/The Trouble With Marnie.mkv" 1200
probe_row "$LIB/Marnie (1964)/Featurettes/Commentary And Archives.mkv" 3000
probe_row "$LIB/Marnie (1964)/Exrtas/Marnie - A Closer Look.mkv" 2600
probe_row "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv" 5700
probe_row "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv" 6300
probe_row "$LIB/Bruce Lee Collection 1971-1981/.Extra 1.mkv" 5220
for e in 01 02 03 04 05; do
probe_row "$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E$e - Mr Monk and the Rival.mkv" 2600
probe_row "$LIB/Big Little Lies Season 1/Big Little Lies S01E$e.mp4" 3300
done
probe_row "$LIB/The Godfather Part 2 (1974)/The.Godfather.Part.2.1974.1080p.BluRay.mkv" 6200
plex_row "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" "Alien" "1979"
plex_row "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" "Marnie" "1964"
plex_row "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv" "V/H/S/2" "2013"
# An unmatched Plex row: the leaf title echoes the file and the year is 0.
plex_row "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv" "AMERICAN POP" "0"
plex_row "$LIB/The Godfather Part 2 (1974)/The.Godfather.Part.2.1974.1080p.BluRay.mkv" \
"The Godfather Part 2" "1974"
export MEDIA_ROOT="$LIB"
run python3 "$CENSUS" enumerate --root "$LIB" --out "$ENUM"
[ "$status" -eq 0 ]
}
# plan — run the planner over the fixture, keeping its plan TSV, its exception
# report and the gate lines it prints (that stdout is the operator's evidence
# that the plan accounts for every file, so the tests read it too).
plan() {
PLANOUT="$BATS_TEST_TMPDIR/plan.out"
run python3 "$PLAN" --root "$LIB" --enum "$ENUM" --probed "$PROBED" \
--plex "$PLEX" --out "$OUT" --exceptions "$EXC"
[ "$status" -eq 0 ]
printf '%s\n' "$output" > "$PLANOUT"
}
# dst_of SRC — the destination the plan gives one file.
dst_of() {
python3 -c '
import sys
for line in open(sys.argv[1]):
f = line.rstrip("\n").split("\t")
if len(f) > 2 and f[1] == sys.argv[2]:
print(f[2]); break
' "$OUT" "$1"
}
# tv_plan — the TV mover over the fixture library; its stdout is the operator's
# census of what it called a series and what it refused to touch.
tv_plan() {
run python3 "$TV" --root "$LIB" --library "$TVLIB" --out "$TVPLAN"
[ "$status" -eq 0 ]
printf '%s\n' "$output" > "$BATS_TEST_TMPDIR/tv.out"
}
# tv_apply — that plan through the same executor. The destination is a sibling
# share, so the plan is applied at the root that holds both libraries.
tv_apply() {
run python3 "$APPLY" apply --plan "$TVPLAN" --root "$BATS_TEST_TMPDIR" \
--journal "$TVJ" --apply
[ "$status" -eq 0 ]
}
# tv_dst_of SRC — the destination the TV plan gives one file, empty if none.
tv_dst_of() {
python3 -c '
import sys
for line in open(sys.argv[1]):
f = line.rstrip("\n").split("\t")
if len(f) > 2 and f[1] == sys.argv[2]:
print(f[2]); break
' "$TVPLAN" "$1"
}
@test "a film lands in Title (year)/Title (year).mkv, named by Plex" {
plan
[ "$(dst_of "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv")" \
= "$LIB/Alien (1979)/Alien (1979).mkv" ]
}
@test "a Plex title that carries a slash becomes one folder, not a nested tree" {
plan
local d; d="$(dst_of "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv")"
[ "$(dirname "$d")" = "$LIB/V-H-S-2 (2013)" ]
[ ! -e "$LIB/V" ]
# the folder must be ONE mkdir: an un-sanitized name would make the
# executor build Movies/V/H/S/2 (2013) instead
run grep -F "mkdir" "$OUT"
[[ "$output" == *"$(printf '%s' "$LIB/V-H-S-2 (2013)")"* ]]
run awk -F'\t' '$1=="mkdir"{print $3}' "$OUT"
[[ "$output" != *"$LIB/V/"* ]]
}
@test "an unmatched Plex row never names a folder with a zero year" {
plan
run awk -F'\t' '{print $3}' "$OUT"
[[ "$output" != *"(0)"* ]]
# the filename's own year carries the folder, the folder and the file share
# the name (Jellyfin's rule), and the film sits at the library root rather
# than in an unsorted pile
local d; d="$(dst_of "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv")"
[ "$(dirname "$(dirname "$d")")" = "$LIB" ]
[[ "$(basename "$(dirname "$d")")" == *" (2023)" ]]
[ "$(basename "$d")" = "$(basename "$(dirname "$d")").mkv" ]
}
@test "a bonus folder keeps its featurettes inside its own film, not a neighbour's" {
plan
local d
d="$(dst_of "$LIB/Alien (1979)/Featurettes/A Bug And A Bite - Making Alien.mkv")"
[ "$(dirname "$(dirname "$d")")" = "$LIB/Alien (1979)" ]
d="$(dst_of "$LIB/Marnie (1964)/Featurettes/The Trouble With Marnie.mkv")"
[ "$(dirname "$(dirname "$d")")" = "$LIB/Marnie (1964)" ]
d="$(dst_of "$LIB/Alien (1979)/Extras/Extra - Logo.mkv")"
[ "$(dirname "$(dirname "$d")")" = "$LIB/Alien (1979)" ]
# the feature-length extra is bonus too: it sits in a bonus folder and no
# metadata matched it, so it may not become a movie of its own
d="$(dst_of "$LIB/Marnie (1964)/Featurettes/Commentary And Archives.mkv")"
[ "$(dirname "$(dirname "$d")")" = "$LIB/Marnie (1964)" ]
}
@test "a folder inside a movie folder is bonus even when its name is misspelled" {
plan
local d; d="$(dst_of "$LIB/Marnie (1964)/Exrtas/Marnie - A Closer Look.mkv")"
[ "$(dirname "$(dirname "$d")")" = "$LIB/Marnie (1964)" ]
}
@test "a subtitle follows the film of the folder it sits in" {
plan
local d; d="$(dst_of "$LIB/Alien (1979)/subs/3_Eng.srt")"
[ "$(dirname "$(dirname "$d")")" = "$LIB/Alien (1979)" ]
}
@test "macOS sidecars are parked in one pile, never inside a movie folder" {
plan
local d; d="$(dst_of "$LIB/Alien (1979)/._Alien.1979.1080p.BluRay.x265.10bit.mkv")"
[[ "$(dirname "$d")" == "$LIB"/_* ]]
d="$(dst_of "$LIB/.DS_Store")"
[[ "$(dirname "$d")" == "$LIB"/_* ]]
grep -q "AppleDouble" "$EXC"
}
@test "a file whose name is a number becomes a review line, not a folder named 1" {
plan
run awk -F'\t' '$1=="mkdir"{print $3}' "$OUT"
[[ "$output" != "$LIB/1"$'\n'* ]] && [[ "$output" != "$LIB/1" ]]
[[ "$output" != *"$LIB/1/"* ]]
grep -q "UNNAMED" "$EXC"
}
@test "a box set in the movie library is quarantined, not filed as movies" {
plan
local d
d="$(dst_of "$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E01 - Mr Monk and the Rival.mkv")"
[[ "$d" == "$LIB"/_TV\ needs\ a\ TV\ library\ -\ monk* ]]
d="$(dst_of "$LIB/Big Little Lies Season 1/Big Little Lies S01E01.mp4")"
[[ "$d" == "$LIB"/_TV\ needs\ a\ TV\ library\ -\ big\ little\ lies* ]]
# one series is one pile, whatever its episodes are called
run python3 -c '
import re, sys
pile = {}
for line in open(sys.argv[1]):
f = line.rstrip("\n").split("\t")
if len(f) > 2 and f[0] == "move":
m = re.search(r"/_TV needs a TV library - ([^/]*)/", f[2] + "/")
if m:
pile[m.group(1)] = pile.get(m.group(1), 0) + 1
for k in sorted(pile):
print(f"{k}={pile[k]}")
' "$OUT"
[ "${#lines[@]}" -eq 2 ]
[[ "${lines[0]}" == big\ little\ lies*=5 ]]
[[ "${lines[1]}" == monk*=5 ]]
}
@test "a film named Part 2 is not called a series" {
plan
[ "$(dst_of "$LIB/The Godfather Part 2 (1974)/The.Godfather.Part.2.1974.1080p.BluRay.mkv")" \
= "$LIB/The Godfather Part 2 (1974)/The Godfather Part 2 (1974).mkv" ]
}
@test "the TV mover files a box set into the TV library in Jellyfin shape" {
tv_plan
tv_apply
[ -e "$TVLIB/Monk (2002)/Season 01/Monk - S01E01 - Mr Monk and the Rival.mkv" ]
[ -e "$TVLIB/Big Little Lies/Season 01/Big Little Lies S01E01.mp4" ]
[ ! -e "$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E01 - Mr Monk and the Rival.mkv" ]
# films are not the TV mover's business
[ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
}
@test "the TV mover leaves a film whose name only looks like an episode" {
tv_plan
tv_apply
local gf="$LIB/The Godfather Part 2 (1974)/The.Godfather.Part.2.1974.1080p.BluRay.mkv"
[ "$(tv_dst_of "$gf")" = "" ]
[ -e "$gf" ]
grep -q "Godfather.*not a series" "$BATS_TEST_TMPDIR/tv.out"
}
@test "the TV mover keeps an episode's own subtitle with the episode" {
at "$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E01 - Mr Monk and the Rival.srt" 40000
tv_plan
tv_apply
[ -e "$TVLIB/Monk (2002)/Season 01/Monk - S01E01 - Mr Monk and the Rival.srt" ]
}
@test "a second TV move finds nothing left to move" {
tv_plan
tv_apply
tv_plan
grep -q "0 moves" "$BATS_TEST_TMPDIR/tv.out"
}
@test "a show whose own name carries the year is not given it twice" {
local d="$LIB/Leah Remini Scientology and the Aftermath (2016)/Season 1"
for e in 01 02 03 04 05; do
at "$d/Leah Remini Scientology and the Aftermath (2016) - S01E$e - Episode.mkv" 3000000
done
tv_plan
tv_apply
[ -e "$TVLIB/Leah Remini Scientology and the Aftermath (2016)/Season 01/Leah Remini Scientology and the Aftermath (2016) - S01E01 - Episode.mkv" ]
[ ! -e "$TVLIB/Leah Remini Scientology and the Aftermath (2016) (2016)" ]
}
@test "an all-lowercase series name is written the way the library writes names" {
for e in 01 02 03 04 05; do
at "$LIB/downton abbey 2010/Season 1/downton abbey - s01e$e - episode title.mkv" 3000000
done
tv_plan
tv_apply
# the FOLDER is titled; the file keeps the name the library already holds
[ -e "$TVLIB/Downton Abbey (2010)/Season 01/downton abbey - s01e01 - episode title.mkv" ]
}
@test "the mover leaves an episode the TV library already holds" {
local src="$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E01 - Mr Monk and the Rival.mkv"
for e in 01 02 03 04 05; do
at "$TVLIB/Monk (2002)/Season 01/Monk - S01E$e - Mr Monk and the Rival.mkv" 3000000
done
tv_plan
[ -z "$(tv_dst_of "$src")" ]
[ -e "$src" ]
grep -q "Monk.*already in" "$BATS_TEST_TMPDIR/tv.out"
}
@test "an episode the TV library holds in another encode is still moved" {
for e in 01 02 03 04 05; do
at "$TVLIB/Big Little Lies/Season 01/Big Little Lies S01E$e 1080p.mp4" 4000000
done
tv_plan
tv_apply
[ -e "$TVLIB/Big Little Lies/Season 01/Big Little Lies S01E01.mp4" ]
[ -e "$TVLIB/Big Little Lies/Season 01/Big Little Lies S01E01 1080p.mp4" ]
}
@test "every file in the census is placed or kept — nothing is dropped" {
plan
python3 - "$ENUM" "$OUT" <<'PY'
import sys
enum = {l.rstrip("\n").split("\t")[2] for l in open(sys.argv[1])}
placed = set()
for line in open(sys.argv[2]):
f = line.rstrip("\n").split("\t")
if f[0] in ("move", "keep"):
placed.add(f[1])
missing = enum - placed
assert not missing, f"{len(missing)} census files with no plan row: {sorted(missing)[:3]}"
PY
grep -q "left-unmoved=0" "$PLANOUT"
grep -q "double-claim=0" "$PLANOUT"
grep -q "occupied=0" "$PLANOUT"
grep -q "bad-names=0" "$PLANOUT"
grep -q "unplanned=0" "$EXC"
}
# ---------------------------------------------------------------- executor
# apply_plan — plan, then apply for real; the journal goes in the tmpdir.
apply_plan() {
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
}
@test "a run without --apply says what it would do and touches nothing" {
plan
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL"
[ "$status" -eq 0 ]
[[ "$output" == *"--apply"* ]]
[ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
[ ! -e "$LIB/Alien (1979)/Alien (1979).mkv" ]
# a dry run may not even build the folders: an operator must be able to
# run it against the live library with no --apply and change nothing
[ ! -d "$LIB/V-H-S-2 (2013)" ]
[ ! -e "$JOURNAL" ]
}
@test "apply moves every planned file, keeps its bytes, and journals each move" {
plan
apply_plan
[ "$status" -eq 0 ]
[ -e "$LIB/Alien (1979)/Alien (1979).mkv" ]
[ ! -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
local planned moved
planned=$(awk -F'\t' '$1=="move"' "$OUT" | wc -l)
moved=$(python3 -c 'import json,sys
print(sum(1 for l in open(sys.argv[1]) if json.loads(l).get("op") == "move"))' "$JOURNAL")
[ "$moved" -eq "$planned" ]
# The media servers' sidecars live in the metadata overlay and did not follow
# these moves, so the run names that cleanup at the moment it is needed.
[[ "$output" == *"media-view.sh reap"* ]]
run python3 "$APPLY" verify --plan "$OUT" --root "$LIB" --journal "$JOURNAL"
[ "$status" -eq 0 ]
}
@test "a target already holding a different file stops the whole run" {
plan
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
at "$LIB/Alien (1979)/Alien (1979).mkv" 7
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -ne 0 ]
[[ "$output" == *"occupied"* ]]
# nothing moved at all: the other films are still where the census saw them
[ -e "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" ]
[ ! -e "$JOURNAL" ]
}
@test "a plan row whose source has gone stops the run instead of inventing a move" {
plan
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
rm "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv"
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -ne 0 ]
[[ "$output" == *"missing"* ]]
[ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
}
@test "a run interrupted between the journal write and the move finishes the job" {
plan
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
# The journal is written before each move, so an interruption leaves two
# states and only two: an intent whose move happened, and an intent whose
# move never ran. Both are simulated here by hand.
python3 - "$OUT" "$JOURNAL" <<'PY'
import json, os, sys
moves = [r for r in (l.rstrip("\n").split("\t") for l in open(sys.argv[1]))
if r[0] == "move"][:3]
with open(sys.argv[2], "w") as fh:
for i, r in enumerate(moves):
fh.write(json.dumps({"op": "move", "src": r[1], "dst": r[2],
"bytes": os.path.getsize(r[1])}) + "\n")
if i < 2: # the first two moves actually ran
os.makedirs(os.path.dirname(r[2]), exist_ok=True)
os.rename(r[1], r[2])
PY
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
run python3 "$APPLY" verify --plan "$OUT" --root "$LIB" --journal "$JOURNAL"
[ "$status" -eq 0 ]
python3 - "$OUT" "$JOURNAL" <<'PY'
import json, os, sys
moves = [r for r in (l.rstrip("\n").split("\t") for l in open(sys.argv[1]))
if r[0] == "move"]
journal = [json.loads(l) for l in open(sys.argv[2])]
seen = [j["dst"] for j in journal]
assert len(seen) == len(set(seen)), "a destination was journalled twice"
for r in moves:
assert os.path.exists(r[2]) and not os.path.exists(r[1]), r[2]
PY
}
@test "rollback puts every file back where the census found it" {
plan
apply_plan
[ "$status" -eq 0 ]
run python3 "$APPLY" rollback --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
[ -e "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" ]
[ -e "$LIB/Alien (1979)/subs/3_Eng.srt" ]
[ ! -e "$LIB/Alien (1979)/Alien (1979).mkv" ]
# the folders the run created are empty afterwards and must not litter the
# library: this one never existed before the plan ran
[ -d "$LIB/VHS 2 (2013)" ]
[ ! -e "$LIB/V-H-S-2 (2013)" ]
}
@test "rollback refuses when a moved file has gone missing" {
plan
apply_plan
[ "$status" -eq 0 ]
rm "$LIB/Alien (1979)/Alien (1979).mkv"
run python3 "$APPLY" rollback --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -ne 0 ]
[[ "$output" == *"missing"* ]]
# the refusal is all-or-nothing: another film did not move back
[ -e "$LIB/Marnie (1964)/Marnie (1964).mkv" ]
}
# ---------------------------------------------------------------- sweep
#
# A relayout leaves the release folders it emptied standing at the library root:
# 1675 of them after the live run, each holding nothing but empty subfolders. The
# sweep pass removes only folders THIS RUN emptied — proven from the journal,
# never from "it looks empty" — so a folder someone made by hand is untouchable,
# and every removal rides the same rollback as the moves.
@test "sweep removes only the folders this run emptied" {
plan
apply_plan
mkdir -p "$LIB/A Empty Placeholder" # someone's folder, never a move source
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[[ "$output" == *"swept"* ]]
[ ! -e "$LIB/AMERICAN POP" ]
[ ! -e "$LIB/VHS 2 (2013)" ]
[ ! -e "$LIB/Alien (1979)/Featurettes" ]
[ ! -e "$LIB/Alien (1979)/subs" ]
[ ! -e "$LIB/Marnie (1964)/Exrtas" ]
# a folder the run never moved a file out of survives, empty or not
[ -d "$LIB/A Empty Placeholder" ]
}
@test "a swept folder carries the empty junk folders inside it" {
mkdir -p "$LIB/AMERICAN POP/Screens" # screenshots, never a move source
plan
apply_plan
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[[ "$output" == *"0 left in place"* ]]
[ ! -e "$LIB/AMERICAN POP/Screens" ]
[ ! -e "$LIB/AMERICAN POP" ]
}
@test "a sweep whose rmdir failed tries those folders again" {
plan
apply_plan
mkdir -p "$LIB/AMERICAN POP/Screens"
chmod 500 "$LIB/AMERICAN POP" # readable, but nothing can be removed
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[[ "$output" == *"left in place"* ]]
[ -e "$LIB/AMERICAN POP/Screens" ]
chmod 700 "$LIB/AMERICAN POP" # the operator cleared the blocker
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[[ "$output" != *"left in place: "* ]]
[ ! -e "$LIB/AMERICAN POP" ]
}
@test "sweep without --apply reports the husks and changes nothing" {
plan
apply_plan
local before jlines
before=$(find "$LIB" -type d | wc -l)
jlines=$(wc -l < "$JOURNAL")
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL"
[ "$status" -eq 0 ]
[[ "$output" == *"--apply"* ]]
[ "$(find "$LIB" -type d | wc -l)" -eq "$before" ]
[ "$(wc -l < "$JOURNAL")" -eq "$jlines" ]
[ -e "$LIB/AMERICAN POP" ]
}
@test "a folder that still holds content is never swept" {
plan
apply_plan
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
# the film folder, the folder the run created, and the piles all stay
[ -d "$LIB/Marnie (1964)" ]
[ -e "$LIB/Marnie (1964)/Marnie (1964).mkv" ]
[ -d "$LIB/V-H-S-2 (2013)" ]
[ -d "$LIB/_unsorted bonus/extras" ]
}
@test "rollback after a sweep brings the folders and the files back" {
plan
apply_plan
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[ ! -e "$LIB/AMERICAN POP" ]
run python3 "$APPLY" rollback --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[ -e "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv" ]
[ -e "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv" ]
[ -e "$LIB/Alien (1979)/Featurettes/A Bug And A Bite - Making Alien.mkv" ]
[ -e "$LIB/Alien (1979)/subs/3_Eng.srt" ]
}
@test "a second sweep has nothing left to do and says so" {
plan
apply_plan
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
local before
before=$(find "$LIB" -type d | wc -l)
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[[ "$output" == *"swept: 0"* ]]
[ "$(find "$LIB" -type d | wc -l)" -eq "$before" ]
}
@test "sweep never follows a symlink and never removes the folder it guards" {
plan
apply_plan
mkdir -p "$LIB/Elsewhere"
at "$LIB/Elsewhere/keep.txt" 10
ln -s "$LIB/Elsewhere" "$LIB/AMERICAN POP/pointer"
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
# a link is content, not nothing: the husk it sits in survives untouched
[ -d "$LIB/AMERICAN POP" ]
[ -L "$LIB/AMERICAN POP/pointer" ]
[ -e "$LIB/Elsewhere/keep.txt" ]
}
@test "sweep refuses a journal that belongs to another library" {
plan
apply_plan
mkdir -p "$BATS_TEST_TMPDIR/Other"
run python3 "$APPLY" sweep --root "$BATS_TEST_TMPDIR/Other" \
--journal "$JOURNAL" --apply
[ "$status" -ne 0 ]
[[ "$output" == *"outside"* ]]
[ -d "$LIB/AMERICAN POP" ]
}
+213 -8
View File
@@ -673,26 +673,44 @@ PY
window_fixture() { window_fixture() {
D="$BATS_TEST_TMPDIR/win" D="$BATS_TEST_TMPDIR/win"
mkdir -p "$D/campaign/kld-ref" "$D/campaign/calib" "$D/rad" "$D/corpora" mkdir -p "$D/campaign/kld-ref" "$D/campaign/calib" "$D/rad" "$D/corpora" "$D/bin"
echo '{}' > "$D/campaign/kld-ref/kld.json" echo '{}' > "$D/campaign/kld-ref/kld.json"
: > "$D/rad/qwen38-bf16.rad" : > "$D/rad/qwen38-bf16.rad"
: > "$D/corpora/calib.jsonl" echo "{\"p\":1}" > "$D/corpora/calib.jsonl"
# the window's preflight resolves the image digest through podman and asks about the
# port/device by host pid; these stubs answer for a quiet host
cat > "$D/bin/podman" <<'EOS'
#!/usr/bin/env bash
case "${1:-}" in
image) echo "sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093" ;;
esac
EOS
printf '#!/usr/bin/env bash\nexit 0\n' > "$D/bin/ss"
printf '#!/usr/bin/env bash\nexit 0\n' > "$D/bin/fuser"
chmod +x "$D/bin/podman" "$D/bin/ss" "$D/bin/fuser"
echo "$D" echo "$D"
} }
@test "the calibration window refuses before touching a card without a reference recording" { @test "the calibration window refuses before touching a card without a reference recording" {
D="$BATS_TEST_TMPDIR/win-less" D="$(window_fixture)"
mkdir -p "$D/campaign/calib" mv "$D/campaign/kld-ref/kld.json" "$D/campaign/kld-ref/kld.json.absent"
run env FRANK_QWEN38_CAMPAIGN="$D/campaign" bash "$WINDOW" run env PATH="$D/bin:$PATH" \
FRANK_QWEN38_CAMPAIGN="$D/campaign" \
FRANK_QWEN38_RAD="$D/rad/qwen38-bf16.rad" \
Q38_WINDOW_CALIB_CORPUS="$D/corpora/calib.jsonl" \
Q38_CALIB_MIN_FREE_MIB=1 bash "$WINDOW"
[ "$status" -eq 2 ] [ "$status" -eq 2 ]
[[ "$output" == *"no reference recording"* ]] [[ "$output" == *"FAIL reference"* ]]
[[ "$output" == *"must not take the cards"* ]]
} }
@test "the calibration window's dry run prints the gated plan, including what it would bank" { @test "the calibration window's dry run prints the gated plan, including what it would bank" {
D="$(window_fixture)" D="$(window_fixture)"
run env FRANK_QWEN38_CAMPAIGN="$D/campaign" \ run env PATH="$D/bin:$PATH" \
FRANK_QWEN38_CAMPAIGN="$D/campaign" \
FRANK_QWEN38_RAD="$D/rad/qwen38-bf16.rad" \ FRANK_QWEN38_RAD="$D/rad/qwen38-bf16.rad" \
Q38_WINDOW_CALIB_CORPUS="$D/corpora/calib.jsonl" bash "$WINDOW" --dry-run Q38_WINDOW_CALIB_CORPUS="$D/corpora/calib.jsonl" \
Q38_CALIB_MIN_FREE_MIB=1 bash "$WINDOW" --dry-run
[ "$status" -eq 0 ] [ "$status" -eq 0 ]
[[ "$output" == *"run-qwen38-bf16.sh calib"* ]] [[ "$output" == *"run-qwen38-bf16.sh calib"* ]]
[[ "$output" == *"else kill the rig"* ]] [[ "$output" == *"else kill the rig"* ]]
@@ -731,3 +749,190 @@ PY
[ "$status" -eq 2 ] [ "$status" -eq 2 ]
[[ "$output" == *"Q38_CALIB_MNBT must be a number"* ]] [[ "$output" == *"Q38_CALIB_MNBT must be a number"* ]]
} }
# ---- the window must not mistake a dying production socket for the rig -------
# 2026-10-30: the runner stopped frank-archon, the rig container was still loading,
# and /health answered 200 from the production process draining its listener. The
# window logged `health ok waited=0s`, the pusher refused on the container name, and
# a 12M window died at second zero. The gate must prove the LISTENER is in the rig.
tap_window_fixture() {
export W="$BATS_TEST_TMPDIR/window"
mkdir -p "$W/scripts" "$W/campaign/calib" "$W/campaign/kld-ref" "$W/models/rad" "$W/corpora" "$W/bin"
cp "$BATS_TEST_DIRNAME/../scripts/qwen38-calib-window.sh" "$W/scripts/"
cp "$BATS_TEST_DIRNAME/../scripts/qwen38-preflight.sh" "$W/scripts/"
echo "{}" > "$W/campaign/kld-ref/kld.json"
: > "$W/models/rad/qwen38-bf16.rad"
echo "{\"p\":1}" > "$W/corpora/calib.jsonl"
# a gram so the empty-tap check passes on the positive path
: > "$W/campaign/calib/gram.r0.blk.0.ffn_down_exps.0.bin"
cat > "$W/scripts/run-qwen38-bf16.sh" <<'EOS'
#!/usr/bin/env bash
[ -n "${STUB_RIG_MARKER:-}" ] && : > "$STUB_RIG_MARKER"
sleep "${STUB_RIG_SLEEP:-3}"
exit 0
EOS
cat > "$W/bin/py" <<'EOS'
#!/usr/bin/env bash
[ -n "${STUB_PUSH_SENTINEL:-}" ] && : > "$STUB_PUSH_SENTINEL"
exit 0
EOS
cat > "$W/bin/podman" <<'EOS'
#!/usr/bin/env bash
case "${1:-}" in
inspect) echo true ;;
# the PREFLIGHT asks who owns :7770 (production, per LISTEN_PID); the WINDOW asks which
# pids belong to the rig (RIG_PID). Both stubs are the same binary, so keep them apart.
image) echo "sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093" ;;
top) echo PID
# the WINDOW asks "which pids are the rig's", the PREFLIGHT asks "which are
# production's". One stub, two answers: that separation IS the identity lesson.
case "${2:-}" in
qwen38-bf16) echo "${RIG_PID:-424242}" ;;
*) echo "${LISTEN_PID:-424242}" ;;
esac ;;
*) : ;;
esac
EOS
printf '#!/usr/bin/env bash\nexit 0\n' > "$W/bin/fuser"
cat > "$W/bin/ss" <<'EOS'
#!/usr/bin/env bash
echo "LISTEN 0 4096 *:7770 *:* users:((\"radiance\",pid=${LISTEN_PID:-424242},fd=9))"
EOS
cat > "$W/bin/curl" <<'EOS'
#!/usr/bin/env bash
exit 0
EOS
chmod +x "$W/scripts/run-qwen38-bf16.sh" "$W/bin/py" "$W/bin/podman" "$W/bin/ss" "$W/bin/curl" "$W/bin/fuser"
}
tap_window_env() {
env PATH="$W/bin:$PATH" \
FRANK_QWEN38_CAMPAIGN="$W/campaign" \
FRANK_QWEN38_RAD="$W/models/rad/qwen38-bf16.rad" \
Q38_WINDOW_CALIB_CORPUS="$W/corpora/calib.jsonl" \
Q38_CALIB_MIN_FREE_MIB=1 \
Q38_WINDOW_HEALTH_TIMEOUT=1 \
PY="$W/bin/py" \
STUB_RIG_MARKER="$W/rig.marker" \
STUB_RIG_SLEEP=0 \
STUB_PUSH_SENTINEL="$W/push.ran" \
bash "$W/scripts/qwen38-calib-window.sh"
}
@test "a health answer from a listener outside the rig is not the rig: no push, window fails" {
tap_window_fixture
# health 200, container "running", but the listener pid belongs to production
export LISTEN_PID=999999 RIG_PID=424242
run tap_window_env
[ "$status" -ne 0 ]
[ -z "$(ls "$W/push.ran" 2>/dev/null)" ]
[[ "$output" == *"before it ever answered /health"* ]]
}
@test "when the listener is inside the rig the window proceeds and banks the tap" {
tap_window_fixture
export LISTEN_PID=424242 RIG_PID=424242
run tap_window_env
[ "$status" -eq 0 ]
[ -e "$W/push.ran" ]
[[ "$output" == *"TAP BANKED"* ]]
}
# ---- the preflight: every fact by IDENTITY, and it must fail loudly -------------------------
# Each case here is a lie that cost a real window on 2026-10-30.
preflight_fixture() {
export PF="$BATS_TEST_TMPDIR/pf"
mkdir -p "$PF/campaign/calib" "$PF/campaign/kld-ref" "$PF/bin"
echo '{}' > "$PF/campaign/kld-ref/kld.json"
echo '{"p":1}' > "$PF/corpus.jsonl"
cat > "$PF/bin/podman" <<'EOS'
#!/usr/bin/env bash
case "$1" in
image) # an image carries every digest it was pulled under; the pin is one of them
printf '%s\n' "docker.io/stilldeadcode/radiance@sha256:4beb81eea874e3e308fbffbe1a9402a2f845c19e68ea612ba949b43bd5cb7cb0"
printf '%s\n' "docker.io/stilldeadcode/radiance@${STUB_IMAGE_DIGEST:-sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}" ;;
top) echo HPID; echo "${STUB_PROD_PIDS:-4242}" ;;
esac
EOS
cat > "$PF/bin/ss" <<'EOS'
#!/usr/bin/env bash
[ -n "${STUB_LISTENER_PID:-}" ] && echo "LISTEN 0 4096 *:7770 *:* users:((\"x\",pid=${STUB_LISTENER_PID},fd=9))"
exit 0
EOS
cat > "$PF/bin/fuser" <<'EOS'
#!/usr/bin/env bash
[ -n "${STUB_KFD_PID:-}" ] && echo "${STUB_KFD_PID}"
exit 0
EOS
chmod +x "$PF/bin/podman" "$PF/bin/ss" "$PF/bin/fuser"
}
run_preflight() {
( cd "$BATS_TEST_DIRNAME/.." && env PATH="$PF/bin:$PATH" \
FRANK_QWEN38_CAMPAIGN="$PF/campaign" \
Q38_WINDOW_CALIB_CORPUS="$PF/corpus.jsonl" \
Q38_CALIB_MIN_FREE_MIB=1 \
bash scripts/qwen38-preflight.sh "$@" )
}
@test "preflight passes when every fact is true, and says so" {
preflight_fixture
run run_preflight
[ "$status" -eq 0 ]
[[ "$output" == *"PREFLIGHT PASSED"* ]]
[[ "$output" == *"carries sha256:69add58f"* ]]
}
@test "preflight accepts the service's own process on the port, not just any answer" {
preflight_fixture
export STUB_PROD_PIDS=4242 STUB_LISTENER_PID=4242 STUB_KFD_PID=4242
run run_preflight
[ "$status" -eq 0 ]
[[ "$output" == *"held by frank-archon (host pid 4242)"* ]]
}
@test "preflight fails when the tag resolves to a different digest" {
preflight_fixture
export STUB_IMAGE_DIGEST="sha256:deadbeefdeadbeef"
run run_preflight
[ "$status" -eq 2 ]
[[ "$output" == *"does not carry sha256:69add58f"* ]]
}
@test "preflight fails when :7770 belongs to a host pid the service does not own" {
preflight_fixture
export STUB_LISTENER_PID=999999
run run_preflight
[ "$status" -eq 2 ]
[[ "$output" == *":7770 is held by host pid 999999"* ]]
}
@test "preflight fails when another process holds the GPU" {
preflight_fixture
export STUB_KFD_PID=999998
run run_preflight
[ "$status" -eq 2 ]
[[ "$output" == *"foreign holders:"* ]]
}
@test "preflight fails when the tap dir already holds grams from another run" {
preflight_fixture
: > "$PF/campaign/calib/gram.r0.blk.0.ffn_down_exps.0.bin"
run run_preflight
[ "$status" -eq 2 ]
[[ "$output" == *"rotate them to .AGENTTRASH"* ]]
run run_preflight --allow-existing-tap
[ "$status" -eq 0 ]
}
@test "preflight fails when the launcher is not in the repo root" {
preflight_fixture
run env PATH="$PF/bin:$PATH" FRANK_QWEN38_CAMPAIGN="$PF/campaign" \
Q38_WINDOW_CALIB_CORPUS="$PF/corpus.jsonl" Q38_CALIB_MIN_FREE_MIB=1 \
bash -c "cd /tmp && bash '$BATS_TEST_DIRNAME/../scripts/qwen38-preflight.sh'"
[ "$status" -eq 2 ]
[[ "$output" == *"not the repo root"* ]]
}
+4 -1
View File
@@ -157,12 +157,15 @@ flag_value() {
[[ "$output" != *"--vram-kv-mib"* ]] [[ "$output" != *"--vram-kv-mib"* ]]
} }
@test "the calib dry run opens the tap at 12M tokens and serves on 7770" { @test "the calib dry run opens the tap at the 12M floor, on the latest pin, and serves on 7770" {
run bash "$RUNNER" calib --dry-run run bash "$RUNNER" calib --dry-run
[ "$status" -eq 0 ] [ "$status" -eq 0 ]
flat="$(oneline)" flat="$(oneline)"
[[ "$flat" == *"-e RADIANCE_CALIB_TOKENS=12000000"* ]] [[ "$flat" == *"-e RADIANCE_CALIB_TOKENS=12000000"* ]]
[[ "$flat" == *"-e RADIANCE_CALIB_DIR=/data/calib"* ]] [[ "$flat" == *"-e RADIANCE_CALIB_DIR=/data/calib"* ]]
# the tap producer is the campaign's own pin (latest radiance), not production's
[[ "$flat" == *"radiance@sha256:69add58f"* ]]
[[ "$flat" != *"radiance@sha256:d590e300"* ]]
[[ "$flat" == *"--port 7770"* ]] [[ "$flat" == *"--port 7770"* ]]
[[ "$flat" == *":/data/calib "* ]] [[ "$flat" == *":/data/calib "* ]]
# calib serves; it does not record # calib serves; it does not record