Compare commits
141
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7aa32d1b64 | ||
|
|
576c3421b8 | ||
|
|
e4656cfd7a | ||
|
|
acab4cef22 | ||
|
|
04802b3e48 | ||
|
|
44e55ad172 | ||
|
|
132805a25a | ||
|
|
2406641bc6 | ||
|
|
dbcf8a2c35 | ||
|
|
a5a9c04da2 | ||
|
|
c95ae20bfb | ||
|
|
a0dfa61623 | ||
|
|
4b67c3c4d8 | ||
|
|
7ee831adee | ||
|
|
afad21dda5 | ||
|
|
f430e2264d | ||
|
|
82a2183d40 | ||
|
|
c70f9b3f6d | ||
|
|
32b8901bff | ||
|
|
6d03bbc8e7 | ||
|
|
32b58dd787 | ||
|
|
0a365274b6 | ||
|
|
e531e90278 | ||
|
|
c3ab565d9b | ||
|
|
5dbbfcb8a0 | ||
|
|
02a568fb97 | ||
|
|
1e1527d101 | ||
|
|
9eb85ea1f9 | ||
|
|
96944f0b17 | ||
|
|
33f97cd3b3 | ||
|
|
254fb41fd0 | ||
|
|
3613f69593 | ||
|
|
6bf70c6a64 | ||
|
|
f1b8b60817 | ||
|
|
4ae2bde9ab | ||
|
|
ceb5617a42 | ||
|
|
a7ceda95fd | ||
|
|
a32361ee5e | ||
|
|
ce6b3b1d3b | ||
|
|
382e3eb0b8 | ||
|
|
54a393edab | ||
|
|
a5459b0b1e | ||
|
|
b03931f7ce | ||
|
|
82f6ec55c2 | ||
|
|
536949c18d | ||
|
|
337afac91e | ||
|
|
c0c77590b1 | ||
|
|
441b908937 | ||
|
|
1a1dbac268 | ||
|
|
949d3cb6f4 | ||
|
|
3ce8d62e70 | ||
|
|
38737ab84e | ||
|
|
4437519430 | ||
|
|
3b29e173c2 | ||
|
|
9706f5d2de | ||
|
|
206dc3baf1 | ||
|
|
f5a3badb56 | ||
|
|
3e4b4f24f7 | ||
|
|
86f5903e74 | ||
|
|
bd4ee78811 | ||
|
|
b7b39cdc92 | ||
|
|
09532e5216 | ||
|
|
e1786e7cf2 | ||
|
|
409a8d8b0c | ||
|
|
e83162a2ac | ||
|
|
5cd9a0c4c2 | ||
|
|
a803aa046a | ||
|
|
73f77a13c3 | ||
|
|
dd76fcf5d4 | ||
|
|
1cb28095a7 | ||
|
|
e1760e3316 | ||
|
|
272e06daf6 | ||
|
|
4db985b90d | ||
|
|
44d72b9e78 | ||
|
|
8073d0aa9d | ||
|
|
5c678b804c | ||
|
|
81e0995f7d | ||
|
|
167e6d6de3 | ||
|
|
608b0058e3 | ||
|
|
4043798053 | ||
|
|
521537a34c | ||
|
|
19fdffe8bd | ||
|
|
54a4a4180a | ||
|
|
f7aec7f154 | ||
|
|
299078eaba | ||
|
|
7b2b5b66fe | ||
|
|
e1d4d59773 | ||
|
|
5787c86954 | ||
|
|
acbb539b97 | ||
|
|
49a83ef1d5 | ||
|
|
d012654fc9 | ||
|
|
b45771e87e | ||
|
|
011ceeb344 | ||
|
|
88a9bc02be | ||
|
|
6ac544a7b6 | ||
|
|
33396d2daa | ||
|
|
bdc0613739 | ||
|
|
3e32cb9cd1 | ||
|
|
6e80cd44f0 | ||
|
|
3a04e34414 | ||
|
|
9085a80bd1 | ||
|
|
58184008dd | ||
|
|
b4a1e6069e | ||
|
|
9f997f0841 | ||
|
|
f6298e3501 | ||
|
|
f14efdce1a | ||
|
|
a6fb0209e8 | ||
|
|
92a412e99d | ||
|
|
c238e32d6f | ||
|
|
da5eb968f9 | ||
|
|
b3052e9243 | ||
|
|
3e5a37a21f | ||
|
|
1e4fa14772 | ||
|
|
e6fba5d97e | ||
|
|
ca1475543d | ||
|
|
f83c241668 | ||
|
|
255796b3dc | ||
|
|
b4d7f6e7fe | ||
|
|
a3ff55efd8 | ||
|
|
fad87aa522 | ||
|
|
a3c2465638 | ||
|
|
58dc7a64c1 | ||
|
|
9c386d4f89 | ||
|
|
fca47d25d8 | ||
|
|
1c56dea67a | ||
|
|
96c5a0625c | ||
|
|
8270524edd | ||
|
|
a7329d190f | ||
|
|
5fe04e569f | ||
|
|
bb027265db | ||
|
|
4e551fafd7 | ||
|
|
3d8d18a45c | ||
|
|
c13a2568d4 | ||
|
|
e7a835e41f | ||
|
|
0ecfe830f7 | ||
|
|
ce370a5b28 | ||
|
|
bfe4d185bd | ||
|
|
94a9f3aa31 | ||
|
|
d2810ab610 | ||
|
|
7a8cc83385 | ||
|
|
c4623f1629 |
No files matched your search
@@ -26,8 +26,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
| Target | Service | Runtime | Payload | Port |
|
||||
|--------|---------|---------|---------|------|
|
||||
| Dankserver | `frank-archon` | radiance, digest-pinned, root unit, TP=2 | publisher `qwen3.8-next-flash-fp8-iq4r-moe.rad` | 7770 |
|
||||
| Dankclaw | `frank-dragoon` | halogen-flash-server 0.16.2, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 |
|
||||
| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 10777, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj + rank-1 refusal adapter | 7771 |
|
||||
| Dankclaw | `frank-dragoon` | halogen-flash-server 0.18.20, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 |
|
||||
| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 12118, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj + rank-1 refusal adapter | 7771 |
|
||||
|
||||
Payload identity, shape, eras, laws and rollback live in ONE page each:
|
||||
[[frank-archon]], [[frank-dragoon]], [[frank-zealot]]. `scripts/<service>.env`
|
||||
@@ -293,8 +293,8 @@ means off or NIC dead; a merely wedged kernel usually still answers ARP.
|
||||
2. Bench against the service itself. No scratch ports on an LLM host — one
|
||||
process owns the card. Use the runtime's own harness (`llama-bench`; a
|
||||
live llama.cpp server also reports honest `timings` per request, including
|
||||
`draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork build
|
||||
10777 that object only reaches you on **`/v1/chat/completions`**: a greedy
|
||||
`draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork that object
|
||||
only reaches you on **`/v1/chat/completions`** (measured on build 10777): a greedy
|
||||
`ignore_eos` run on the raw `/completion` path there answers **500 "The
|
||||
model produced output that does not match the expected Content-only
|
||||
format"** (measured 2026-10-18). halogen has no `/completion` path at all,
|
||||
@@ -439,16 +439,16 @@ disagrees with it is the bug.
|
||||
### frank-zealot — llama.cpp (PrismML fork), RTX 3090, :7771
|
||||
|
||||
- Owns: `scripts/frank-zealot.env`, `scripts/serve-frank-zealot.sh`,
|
||||
`scripts/frank-zealot.service`; fork build `llama-prism-b10777` is the ONLY
|
||||
loader (the payload carries the fork's custom PQ2_0 tensor type plus
|
||||
`scripts/frank-zealot.service`; fork build `llama-prism-0244744` (build 12118)
|
||||
is the ONLY loader — `llama-prism-b10777` stays on disk as its rollback (the payload carries the fork's custom PQ2_0 tensor type plus
|
||||
`prism.hadamard.*` dequant metadata — stock llama.cpp refuses it, and a
|
||||
loader ignoring that metadata dequantizes wrong); payload
|
||||
`Bonsai2-9B-MTP/…PQ2_0.gguf` + the publisher's **Q8_0** mmproj (the BF16
|
||||
tower can crash the server with concurrent image requests) + the rank-1
|
||||
refusal-direction adapter applied at load by `--lora-scaled …:1`. The
|
||||
adapter's rollback is `LORA_FILE=` (empty) in the env — one line, no rebuild
|
||||
— and build 10777 is also the gate that lets it load: older forks refuse it
|
||||
with `unsupported prism.hadamard.version: 2`.
|
||||
— and the fork BUILD is also the gate that lets it load: anything older than
|
||||
b10777 refuses it with `unsupported prism.hadamard.version: 2`.
|
||||
- Shape: `--ctx-size 602112 --parallel 3` = **three conversations at
|
||||
200,704 each**, `CACHE_TYPE_K=f16` + `CACHE_TYPE_V=q8_0` (asymmetric: 25,088
|
||||
B/token, 14,406 MiB of pool), batch 6144 / ubatch 1024,
|
||||
@@ -741,6 +741,13 @@ linked runbooks.
|
||||
`/etc/default/grub`. It replaced `amd_iommu=off`, and the local A/B on byte-identical traffic
|
||||
says identity mode costs nothing, so the XDNA2 NPU returned for free.
|
||||
[[dankclaw-iommu-translation-off]]
|
||||
- **A deploy handler that only `start`s a quadlet no-ops an image bump.**
|
||||
`systemctl --user start` on an already-active unit does nothing, so installing
|
||||
a new digest under a running service keeps the OLD container serving while the
|
||||
deploy, its health gate and the Playwright suite all pass — the authentik
|
||||
handler did exactly that on 2026-10-30 (2026.8.3 → 2026.8.5 moved only the
|
||||
outpost, the one handler that restarted). A pin change is applied by
|
||||
`restart`, never `start`.
|
||||
- **Rootless Quadlet containers run with `--replace --rm`**: `podman logs` covers only the CURRENT
|
||||
instance; history across recreations lives in `journalctl --user -u <name>.service`. Read
|
||||
rootless storage WITHOUT sudo — it is per-user storage, so `sudo podman logs` finds nothing.
|
||||
@@ -800,6 +807,8 @@ per-attempt page and no per-swap ledger anywhere else.
|
||||
| `frank-tts` — cloned-voice speech, Dankclaw | `serverAppConfigs-docs/Dankclaw/Frank TTS.md` |
|
||||
| Dankclaw kernel posture (`iommu=pt` since 2026-10-18) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` |
|
||||
| Movie-library relayout (Jellyfin format, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Relayout.md` |
|
||||
| SSD-backed metadata view (Jellyfin writes, Plex reads, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Metadata View.md` |
|
||||
|
||||
There is no page for a retired endpoint and no change diary. A payload swap
|
||||
appends one row to the live endpoint's **Eras** table; when the endpoint itself
|
||||
@@ -841,5 +850,7 @@ history.
|
||||
| Writing a doc | **One owner per fact** — edit the endpoint/service page; everywhere else links. Never restate a number. |
|
||||
| Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. |
|
||||
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
|
||||
| Relayout the movie library | **Census → plan → read the exceptions → `scripts/media-apply.py`** — renames only, journal before every move; never hand-rename, never copy. |
|
||||
| Making media metadata visible to both media servers | **Bind `/var/lib/media-meta/view`** — the overlay `scripts/media-view.sh` installs (array below, NVMe above). Never write the array; never delete through the view. **Not yet deployed on Dankserver: the live quadlets still bind the array read-only — see [[Media Metadata View]].** |
|
||||
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
|
||||
| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |
|
||||
@@ -23,7 +23,7 @@ Network=dankclaw_default
|
||||
# Image=ghcr.io/goauthentik/ldap:2026.8.3 (amd64 digest; podman 5.7.0 refuses
|
||||
# the tag's nested manifest list, so pin the single-platform digest).
|
||||
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
|
||||
Image=ghcr.io/goauthentik/ldap@sha256:2e7b259eb8c37faac916946ddf48ec681ee9d3a9f724fe8521bf161973f13cf9
|
||||
Image=ghcr.io/goauthentik/ldap:2026.8.5@sha256:4c75bd27918418d2d6fbbfe4581bbd5a199fe38942d24559c4483edd64a1ea87
|
||||
Environment=AUTHENTIK_HOST=https://auth.franksdank.cloud
|
||||
Environment=AUTHENTIK_INSECURE=false
|
||||
Environment=AUTHENTIK_TOKEN=${AUTHENTIK_LDAP_OUTPOST_TOKEN}
|
||||
|
||||
@@ -20,7 +20,7 @@ Wants=authentik-server.service
|
||||
|
||||
[Container]
|
||||
ContainerName=authentik-outpost
|
||||
Image=ghcr.io/goauthentik/server:2026.8.3
|
||||
Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
|
||||
Exec=worker
|
||||
Network=dankclaw_default
|
||||
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
|
||||
|
||||
@@ -14,7 +14,7 @@ After=network.target
|
||||
|
||||
[Container]
|
||||
ContainerName=authentik-postgres
|
||||
Image=docker.io/library/postgres:16-alpine
|
||||
Image=docker.io/library/postgres:16-alpine@sha256:0cefaabd882e42bcb2a2f3dc56d3ce4770560ba62fd7f197f1a2e008fd5731ab
|
||||
Network=dankclaw_default
|
||||
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
|
||||
Environment=POSTGRES_DB=authentik
|
||||
|
||||
@@ -19,7 +19,7 @@ Wants=authentik-postgres.service
|
||||
|
||||
[Container]
|
||||
ContainerName=authentik-server
|
||||
Image=ghcr.io/goauthentik/server:2026.8.3
|
||||
Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
|
||||
Exec=server
|
||||
Network=dankclaw_default
|
||||
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
|
||||
|
||||
@@ -16,7 +16,7 @@ Wants=authentik-server.service
|
||||
|
||||
[Container]
|
||||
ContainerName=authentik-worker
|
||||
Image=ghcr.io/goauthentik/server:2026.8.3
|
||||
Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
|
||||
Exec=worker
|
||||
Network=dankclaw_default
|
||||
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
|
||||
|
||||
@@ -15,7 +15,7 @@ After=network.target
|
||||
[Container]
|
||||
ContainerName=gitea-postgres
|
||||
Network=gitea.network
|
||||
Image=docker.io/library/postgres:16-alpine
|
||||
Image=docker.io/library/postgres:16-alpine@sha256:0cefaabd882e42bcb2a2f3dc56d3ce4770560ba62fd7f197f1a2e008fd5731ab
|
||||
EnvironmentFile=/home/dank/Containers/gitea/env
|
||||
Environment=POSTGRES_USER=gitea
|
||||
Environment=POSTGRES_PASSWORD=${GITEA_DB_PASSWORD}
|
||||
|
||||
@@ -19,7 +19,7 @@ After=gitea-postgres.service network.target
|
||||
[Container]
|
||||
ContainerName=gitea-server
|
||||
Network=gitea.network
|
||||
Image=docker.io/gitea/gitea:latest
|
||||
Image=docker.io/gitea/gitea:latest@sha256:30777c91280d973fa02be48b1ade735bd6859e8f6f7af3b4444d519437852504
|
||||
EnvironmentFile=/home/dank/Containers/gitea/env
|
||||
Environment=DATABASE_TYPE=postgres
|
||||
Environment=DATABASE_HOST=gitea-postgres
|
||||
|
||||
@@ -25,7 +25,7 @@ Description=Chrome DevTools Protocol daemon (headless-shell) for hermes
|
||||
[Container]
|
||||
ContainerName=chrome-devtools-protocol
|
||||
Network=host
|
||||
Image=docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5b5fd7c0fe99215ff662339c57407ee2e8c0a11af93516b04
|
||||
Image=docker.io/chromedp/headless-shell@sha256:7bee201478b2f6266aae01243740ee85c16281e32f42d3aed9fdf04f4eace7a1
|
||||
Entrypoint=/headless-shell/headless-shell
|
||||
Exec=--no-sandbox --disable-gpu --use-gl=angle --use-angle=swiftshader --remote-debugging-address=127.0.0.1 --remote-debugging-port=9222
|
||||
ShmSize=1073741824
|
||||
|
||||
@@ -23,7 +23,7 @@ After=gluetun.service network.target
|
||||
[Container]
|
||||
ContainerName=marreta_browser
|
||||
Network=gluetun.container
|
||||
Image=docker.io/lightpanda/browser:nightly
|
||||
Image=docker.io/lightpanda/browser:nightly@sha256:e72730e364d1fbf352c1149f29ffa4b278f3b1a55ff0dec3262298ebe624ec7a
|
||||
|
||||
[Service]
|
||||
TimeoutStartSec=300
|
||||
|
||||
@@ -32,7 +32,7 @@ After=gluetun.service marreta-browser.service network.target
|
||||
[Container]
|
||||
ContainerName=marreta
|
||||
Network=gluetun.container
|
||||
Image=ghcr.io/manualdousuario/marreta:latest
|
||||
Image=ghcr.io/manualdousuario/marreta:latest@sha256:30f2a34567b79b66da794b4b9423751bbbcd07ceff600d9f6a2d219e7e439362
|
||||
# Relocate off SearXNG's 8080 inside the shared netns.
|
||||
Environment=NGINX_HTTP_PORT=8085
|
||||
Environment=APP_NAME=Marreta
|
||||
|
||||
@@ -33,7 +33,7 @@ Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=alertmanager
|
||||
Image=docker.io/prom/alertmanager:latest
|
||||
Image=docker.io/prom/alertmanager:latest@sha256:39d8bb3966c694a124428a3a88fbcffd40bd069129ecd47c059f8a3f17c55f3b
|
||||
Network=dankclaw_default
|
||||
User=0:0
|
||||
Exec=--config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager
|
||||
|
||||
@@ -35,7 +35,7 @@ Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=cadvisor
|
||||
Image=gcr.io/cadvisor/cadvisor:v0.49.1
|
||||
Image=gcr.io/cadvisor/cadvisor:v0.49.1@sha256:3cde6faf0791ebf7b41d6f8ae7145466fed712ea6f252c935294d2608b1af388
|
||||
PodmanArgs=--pid=host --cgroupns=host
|
||||
PublishPort=127.0.0.1:8080:8080
|
||||
PublishPort=192.168.0.143:8080:8080
|
||||
|
||||
@@ -34,7 +34,7 @@ Wants=network-online.target prometheus.service
|
||||
|
||||
[Container]
|
||||
ContainerName=grafana
|
||||
Image=docker.io/grafana/grafana:latest
|
||||
Image=docker.io/grafana/grafana:latest@sha256:488984b716c0862639f91a698adbe2862677a5a07bad42da5f03f3030883b205
|
||||
Network=dankclaw_default
|
||||
User=0:0
|
||||
PublishPort=127.0.0.1:3002:3000
|
||||
|
||||
@@ -33,7 +33,7 @@ Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=node-exporter
|
||||
Image=docker.io/prom/node-exporter:latest
|
||||
Image=docker.io/prom/node-exporter:latest@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
|
||||
Network=host
|
||||
PodmanArgs=--pid=host
|
||||
Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.143:9100
|
||||
|
||||
@@ -43,14 +43,16 @@ Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=prometheus
|
||||
Image=docker.io/prom/prometheus:latest
|
||||
Image=docker.io/prom/prometheus:latest@sha256:fa3302279710be2f21a8301ec17a8b4f2301fd829bd113839b3b6b02b6f191a2
|
||||
Network=dankclaw_default
|
||||
# Name the podman gateway so scrape jobs can reach same-host targets (the
|
||||
# netavark hairpin gap bars this container from dialing its own host's LAN
|
||||
# IP; the cadvisor/gpu jobs and the blackbox scrape all ride this name).
|
||||
# Same mechanism the hermes Caddy upstream learned the hard way (Container
|
||||
# Host Reachability in the vault).
|
||||
HostAddr=host.containers.internal=169.254.1.2
|
||||
# Same-host scrape targets are dialed at host.containers.internal (the podman
|
||||
# gateway): a bridge container cannot dial its own host's LAN IP (netavark
|
||||
# hairpin gap, measured 2026-10-21), and traffic to the gateway lands on host
|
||||
# loopback / published ports (cadvisor/gpu jobs and the blackbox scrape all
|
||||
# ride it). The name resolves natively — NO HostAddr key: quadlet 5.7.0 calls
|
||||
# it unsupported and the WHOLE user quadlet batch then fails to generate,
|
||||
# silently orphaning every other unit (live 2026-10-21: prometheus.service
|
||||
# vanished while its container kept running as an orphan).
|
||||
User=0:0
|
||||
Exec=--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=30d --web.enable-lifecycle
|
||||
PublishPort=127.0.0.1:9090:9090
|
||||
|
||||
@@ -27,7 +27,7 @@ After=network.target
|
||||
[Container]
|
||||
ContainerName=open-webui
|
||||
Network=open-webui.network
|
||||
Image=ghcr.io/open-webui/open-webui:main
|
||||
Image=ghcr.io/open-webui/open-webui:main@sha256:0b399a8ba34558e181d316750455f2a95376a38cd60a61b9c942e138898944bd
|
||||
PublishPort=3005:8080
|
||||
Environment=OPENAI_API_BASE_URL=http://192.168.0.143:7770
|
||||
Environment=ANONYMIZED_TELEMETRY=False
|
||||
|
||||
@@ -24,7 +24,7 @@ Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=pihole
|
||||
Image=docker.io/pihole/pihole:latest
|
||||
Image=docker.io/pihole/pihole:latest@sha256:d34134716fc32038b052a8674c7c6879e0ff21f0e6d8af345166fffa2c27c113
|
||||
Network=slirp4netns
|
||||
PublishPort=0.0.0.0:53:53/udp
|
||||
PublishPort=0.0.0.0:53:53/tcp
|
||||
|
||||
@@ -16,7 +16,7 @@ After=searxng-network.service network.target
|
||||
[Container]
|
||||
ContainerName=gluetun
|
||||
Network=searxng.network
|
||||
Image=docker.io/qmcgaw/gluetun:latest
|
||||
Image=docker.io/qmcgaw/gluetun:latest@sha256:4af0c804003481e100180e1f8d4d5e9c48163fedc0f88d061881b615aef9d183
|
||||
AddCapability=NET_ADMIN
|
||||
AddDevice=/dev/net/tun:/dev/net/tun
|
||||
Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}
|
||||
|
||||
@@ -16,7 +16,7 @@ After=searxng-network.service network.target
|
||||
ContainerName=searxng-valkey
|
||||
Network=searxng.network
|
||||
NetworkAlias=valkey
|
||||
Image=docker.io/valkey/valkey:9-alpine
|
||||
Image=docker.io/valkey/valkey:9-alpine@sha256:bf09d58e96d6b4524a15396b65b242bbe18beebbcc15a04d8ee75ddac62fae0b
|
||||
Volume=/opt/searxng/valkey:/data
|
||||
|
||||
[Service]
|
||||
|
||||
@@ -16,7 +16,7 @@ After=gluetun.service searxng-valkey.service network.target
|
||||
[Container]
|
||||
ContainerName=searxng
|
||||
Network=gluetun.container
|
||||
Image=ghcr.io/searxng/searxng:latest
|
||||
Image=ghcr.io/searxng/searxng:latest@sha256:f4aa660c54d062dedb268be84e821ea24e39f48c7a496af8dd02cc9eedb93847
|
||||
Environment=SEARXNG_SECRET=${SEARXNG_SECRET}
|
||||
Environment=SEARXNG_BASE_URL=${SEARXNG_BASE_URL}
|
||||
Volume=/opt/searxng/config:/etc/searxng
|
||||
|
||||
@@ -29,7 +29,7 @@ Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=seerr
|
||||
Image=ghcr.io/seerr-team/seerr:v3.4.1
|
||||
Image=ghcr.io/seerr-team/seerr:v3.5.1@sha256:d8fab675b55ad293ded98fd71efcfd46054cbad9746534ae1cc4794a88c0b3be
|
||||
Network=dankclaw_default
|
||||
PublishPort=127.0.0.1:5055:5055
|
||||
Volume=/home/dank/Containers/seerr/data:/app/config
|
||||
|
||||
@@ -43,7 +43,7 @@ After=network.target
|
||||
|
||||
[Container]
|
||||
ContainerName=calibre-server
|
||||
Image=lscr.io/linuxserver/calibre:9.15.0
|
||||
Image=lscr.io/linuxserver/calibre:9.17.0@sha256:b198fc03a6abc6be7e251a9958e50e74634b00e447c7b25e92c191c8077ba378
|
||||
Network=pasta
|
||||
EnvironmentFile=/home/dank/Containers/calibre-server/env
|
||||
Environment=PUID=1000
|
||||
|
||||
@@ -16,7 +16,7 @@ After=network.target
|
||||
|
||||
[Container]
|
||||
ContainerName=calibre-web
|
||||
Image=lscr.io/linuxserver/calibre-web:0.6.27-ls402
|
||||
Image=lscr.io/linuxserver/calibre-web:0.6.27-ls407@sha256:470b4ed9a7d6ea1475fef17dbcf957cb60880910296d226bd0dca652c05f68ca
|
||||
Network=pasta
|
||||
Environment=DOCKER_MODS=linuxserver/mods:universal-calibre
|
||||
Environment=LANG=en_US.UTF-8
|
||||
|
||||
+6
-28
@@ -94,34 +94,12 @@ services:
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
|
||||
# ═══════════════════════════════════════════════
|
||||
# Jellyfin — media server (movies, TV, music)
|
||||
# ═══════════════════════════════════════════════
|
||||
jellyfin:
|
||||
image: jellyfin/jellyfin:latest
|
||||
container_name: jellyfin
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8096:8096/tcp"
|
||||
- "8920:8920/tcp"
|
||||
- "7359:7359/udp"
|
||||
- "1900:1900/udp"
|
||||
environment:
|
||||
- JELLYFIN_PublishedServerUrl=http://192.168.0.235:8096
|
||||
- DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true
|
||||
volumes:
|
||||
- /opt/jellyfin/config:/config
|
||||
- /opt/jellyfin/cache:/cache
|
||||
- /mnt/entropy:/data:ro
|
||||
devices:
|
||||
- /dev/dri:/dev/dri
|
||||
group_add:
|
||||
- "992"
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8096/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
# Jellyfin left this file when it became a Quadlet: the container is owned by
|
||||
# dankserver/jellyfin/quadlet/jellyfin.container, its data roots are under
|
||||
# /home/dank/Containers/jellyfin, and its library arrives through the SSD
|
||||
# overlay at /var/lib/media-meta/view (scripts/media-view.sh). The block that
|
||||
# was here named /opt/jellyfin/* and a read-only array bind — a second owner for
|
||||
# the container name `jellyfin`, wrong on every path. One owner per container.
|
||||
|
||||
# ═══════════════════════════════════════════════
|
||||
# Calibre-Web — e-book web frontend
|
||||
|
||||
@@ -21,7 +21,13 @@ After=network.target
|
||||
|
||||
[Container]
|
||||
ContainerName=copyparty
|
||||
Image=ghcr.io/9001/copyparty-ac:latest
|
||||
# Digest-pinned: upstream stopped publishing version tags after 1.19.15, so
|
||||
# the pin rides :latest@digest (the tag in the reference is cosmetic; the
|
||||
# digest is the pull). Pinned 2026-10-21 at the build whose label is
|
||||
# 1.20.26 - closes GHSA-3c88-p299-7hvr (1.20.25) and GHSA-hg99-jqfp-rf76
|
||||
# (1.20.26); the previously running build was the rolling :latest at
|
||||
# 1.20.24 (issue #73 on command-center).
|
||||
Image=ghcr.io/9001/copyparty-ac:latest@sha256:7f77459160a4bf34fdca79f2965dff4f0c1d7069a3d9cdfb3632acb041dd2f6a
|
||||
Network=dankserver_appnet
|
||||
EnvironmentFile=/home/dank/Containers/copyparty/env
|
||||
Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD} -v /w::r:ra,dank
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
entropy UUID=cd86c728-459e-4f3a-bcc2-41ecf84685ec /etc/entropy nofail,luks
|
||||
@@ -0,0 +1,18 @@
|
||||
# /etc/fstab: static file system information.
|
||||
#
|
||||
# Use 'blkid' to print the universally unique identifier for a
|
||||
# device; this may be used with UUID= as a more robust way to name devices
|
||||
# that works even if disks are added and removed. See fstab(5).
|
||||
#
|
||||
# <file system> <mount point> <type> <options> <dump> <pass>
|
||||
# / was on /dev/nvme0n1p2 during curtin installation
|
||||
/dev/disk/by-uuid/4f3731d2-fb7b-48e7-a79f-0e33cb8a4162 / ext4 defaults 0 1
|
||||
# /boot/efi was on /dev/nvme0n1p1 during curtin installation
|
||||
/dev/disk/by-uuid/183C-442D /boot/efi vfat defaults 0 1
|
||||
#/dev/disk/by-label/archive-one /mnt/archive-one auto nosuid,nodev,nofail,x-gvfs-show 0 0
|
||||
#/dev/disk/by-label/archive-two /mnt/archive-two auto nosuid,nodev,nofail,x-gvfs-show 0 0
|
||||
# /dev/md0 is a LUKS container, not a filesystem; its ext4 is mounted via
|
||||
# /dev/mapper/entropy on the next line. Mounting /dev/md0 directly fails.
|
||||
#/dev/md0 /mnt/md0 ext4 auto,nofail,nosuid,nodev 0 0
|
||||
/dev/mapper/entropy /mnt/entropy ext4 auto,nosuid,nofail,nodev,x-gvfs-show 0 0
|
||||
/swapfile none swap sw 0 0
|
||||
@@ -0,0 +1,30 @@
|
||||
# /etc/systemd/system/var-lib-media\x2dmeta-view.mount
|
||||
# (repo copy: dankserver/host/systemd/media-meta-view.mount — systemd names a
|
||||
# mount unit after its ESCAPED path, so the host file carries the \x2d spelling)
|
||||
#
|
||||
# Installed by `scripts/media-view.sh install` — the repo copy is the source of
|
||||
# truth (fleet law: never edit host state by hand).
|
||||
#
|
||||
# An OverlayFS merge that shows the media library as one tree while splitting
|
||||
# where its bytes live: lowerdir is the RAID6 array (the source of truth, never
|
||||
# written), upperdir is a directory on the NVMe that receives every write. The
|
||||
# two media servers bind the result at /var/lib/media-meta/view, so Jellyfin's
|
||||
# sidecars and artwork are SSD bytes — which is what makes them fast to browse
|
||||
# and what keeps Plex's NFO agent off the spinners — while the media itself stays
|
||||
# on the array untouched.
|
||||
#
|
||||
# RequiresMountsFor is load-bearing: without the array mounted, lowerdir is an
|
||||
# empty directory and the view would present an empty library as truth.
|
||||
|
||||
[Unit]
|
||||
Description=SSD-backed overlay view of the media library (array + NVMe metadata layer)
|
||||
RequiresMountsFor=/mnt/entropy
|
||||
|
||||
[Mount]
|
||||
What=overlay
|
||||
Where=/var/lib/media-meta/view
|
||||
Type=overlay
|
||||
Options=lowerdir=/mnt/entropy,upperdir=/var/lib/media-meta/upper,workdir=/var/lib/media-meta/work
|
||||
|
||||
[Install]
|
||||
WantedBy=local-fs.target
|
||||
@@ -4,8 +4,9 @@
|
||||
# Unit name: jellyfin.service (podman 5.x drops the container- prefix)
|
||||
#
|
||||
# Data roots migrated from /opt/jellyfin to /home/dank/Containers/jellyfin at
|
||||
# cutover. All of /mnt/entropy is mounted read-only at /data: nothing automated
|
||||
# anywhere writes to the entropy volume. Every
|
||||
# cutover. The library is seen through /var/lib/media-meta/view, the SSD-backed
|
||||
# overlay view (scripts/media-view.sh): the array stays read-only and untouched,
|
||||
# while the sidecars this server writes land on the NVMe. Every
|
||||
# creation-time environment variable of the compose-era container is kept as
|
||||
# an Environment= line (zero drift); none of them are secrets, so this unit
|
||||
# needs no EnvironmentFile.
|
||||
@@ -16,7 +17,7 @@ After=network.target
|
||||
|
||||
[Container]
|
||||
ContainerName=jellyfin
|
||||
Image=ghcr.io/jellyfin/jellyfin:latest
|
||||
Image=ghcr.io/jellyfin/jellyfin:latest@sha256:fea863d0cd9ec445985d9c75fc6ed64e38a1d2766cf0c250a33d553f4fc1b771
|
||||
Network=dankserver_appnet
|
||||
Environment=DEBIAN_FRONTEND=noninteractive
|
||||
Environment=DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true
|
||||
@@ -41,7 +42,12 @@ PublishPort=7359:7359/udp
|
||||
PublishPort=1900:1900/udp
|
||||
Volume=/home/dank/Containers/jellyfin/config:/config
|
||||
Volume=/home/dank/Containers/jellyfin/cache:/cache
|
||||
Volume=/mnt/entropy:/data:ro
|
||||
# The view, writable: Jellyfin is the metadata writer (movie.nfo / tvshow.nfo /
|
||||
# episode.nfo + artwork) and those bytes belong on the SSD. The condition is the
|
||||
# fail-closed guard — starting against a plain, unmounted view directory would let
|
||||
# a scan conclude the library is empty.
|
||||
Volume=/var/lib/media-meta/view:/data
|
||||
UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view
|
||||
AddDevice=/dev/dri
|
||||
GroupAdd=keep-groups
|
||||
HealthCmd=curl -f http://localhost:8096/health
|
||||
|
||||
@@ -18,7 +18,7 @@ Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=cadvisor
|
||||
Image=gcr.io/cadvisor/cadvisor:v0.49.1
|
||||
Image=gcr.io/cadvisor/cadvisor:v0.49.1@sha256:3cde6faf0791ebf7b41d6f8ae7145466fed712ea6f252c935294d2608b1af388
|
||||
PodmanArgs=--pid=host --cgroupns=host
|
||||
PublishPort=127.0.0.1:8080:8080
|
||||
PublishPort=192.168.0.235:8080:8080
|
||||
|
||||
@@ -37,7 +37,7 @@ Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=node-exporter
|
||||
Image=docker.io/prom/node-exporter:latest
|
||||
Image=docker.io/prom/node-exporter:latest@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
|
||||
Network=host
|
||||
PodmanArgs=--pid=host
|
||||
Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.235:9100
|
||||
|
||||
@@ -4,14 +4,19 @@ After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
Image=lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329
|
||||
Image=lscr.io/linuxserver/plex:1.43.5.11079-b3b13e506-ls330@sha256:01a53d9c7adcbb3701fbb0a4230c0a37c85453759b6b05a67356c3c7a09de08d
|
||||
ContainerName=plex
|
||||
Network=dankserver_appnet:alias=plex
|
||||
Environment=PUID=0
|
||||
Environment=PGID=0
|
||||
Environment=TZ=America/Los_Angeles
|
||||
Volume=/var/lib/plexmediaserver:/config
|
||||
Volume=/mnt/entropy:/mnt/entropy:ro
|
||||
# The container path stays /mnt/entropy on purpose: every path Plex recorded in
|
||||
# its own database is written against it, so swapping the HOST source of this bind
|
||||
# to the overlay view re-points Plex at the sidecars without touching its DB.
|
||||
# Read-only: Plex consumes the NFO files Jellyfin writes and writes no metadata.
|
||||
Volume=/var/lib/media-meta/view:/mnt/entropy:ro
|
||||
UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view
|
||||
PublishPort=32400:32400
|
||||
|
||||
[Service]
|
||||
|
||||
@@ -14,7 +14,7 @@ After=network.target
|
||||
|
||||
[Container]
|
||||
ContainerName=romm-mariadb
|
||||
Image=docker.io/library/mariadb:11
|
||||
Image=docker.io/library/mariadb:11@sha256:c4124d1afa875e47f2892edf26b482fa54f796e2f342d3717a04ca091a22a9bd
|
||||
Network=dankserver_appnet:alias=mariadb
|
||||
EnvironmentFile=/home/dank/Containers/romm/env
|
||||
Environment=MARIADB_DATABASE=romm
|
||||
|
||||
@@ -12,7 +12,7 @@ After=network.target
|
||||
|
||||
[Container]
|
||||
ContainerName=romm-redis
|
||||
Image=docker.io/library/redis:7-alpine
|
||||
Image=docker.io/library/redis:7-alpine@sha256:8da19c7399dca900c6822a1cf30f9d2285585e2c4b457ff4095b0f4500054d3f
|
||||
Network=dankserver_appnet:alias=redis
|
||||
Volume=/home/dank/Containers/romm/redis:/data
|
||||
HealthCmd=redis-cli ping
|
||||
|
||||
@@ -18,7 +18,7 @@ After=romm-mariadb.service romm-redis.service network.target
|
||||
|
||||
[Container]
|
||||
ContainerName=romm
|
||||
Image=ghcr.io/rommapp/romm:latest
|
||||
Image=ghcr.io/rommapp/romm:latest@sha256:8677fb7b81b1d3db20b849e989a1dadc2a11159e51eb34a2acdcb01102274edf
|
||||
Network=dankserver_appnet
|
||||
EnvironmentFile=/home/dank/Containers/romm/env
|
||||
Environment=DB_HOST=mariadb
|
||||
|
||||
@@ -4,7 +4,7 @@ After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
Image=lscr.io/linuxserver/tautulli:v2.18.1-ls245
|
||||
Image=lscr.io/linuxserver/tautulli:v2.18.2-ls250@sha256:529c7eabcc40554c1dc48f33f1f3cc3b1438462d73e0ec2dac8ac0b66aeeb8d4
|
||||
ContainerName=tautulli
|
||||
Network=dankserver_appnet:alias=tautulli
|
||||
Environment=PUID=0
|
||||
|
||||
@@ -19,7 +19,7 @@ Description=Gluetun VPN tunnel (FrankBook-Pro podman machine)
|
||||
|
||||
[Container]
|
||||
ContainerName=gluetun
|
||||
Image=docker.io/qmcgaw/gluetun:latest
|
||||
Image=docker.io/qmcgaw/gluetun:latest@sha256:c2ee4191762fca6b6a3c3359a731198a95d9c9af4db32d466e223bf3a84e779b
|
||||
AddCapability=NET_ADMIN
|
||||
AddDevice=/dev/net/tun:/dev/net/tun
|
||||
Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}
|
||||
|
||||
@@ -15,7 +15,7 @@ After=gluetun.service
|
||||
|
||||
[Container]
|
||||
ContainerName=lidarr
|
||||
Image=lscr.io/linuxserver/lidarr:latest
|
||||
Image=lscr.io/linuxserver/lidarr:latest@sha256:d8549ff9ea29b4015c32257360652c1000c31ea471a2b3f0835ae77292faf35b
|
||||
Environment=PUID=0
|
||||
Environment=PGID=0
|
||||
Environment=TZ=America/Los_Angeles
|
||||
|
||||
@@ -24,7 +24,7 @@ After=gluetun.service
|
||||
|
||||
[Container]
|
||||
ContainerName=prowlarr
|
||||
Image=lscr.io/linuxserver/prowlarr:latest
|
||||
Image=lscr.io/linuxserver/prowlarr:latest@sha256:b3b00047ed20f6ce2b16780145cb6a0acb51c9e28ea6aa6f5dcd33c7034f1797
|
||||
Environment=PUID=0
|
||||
Environment=PGID=0
|
||||
Environment=TZ=America/Los_Angeles
|
||||
|
||||
@@ -29,7 +29,7 @@ Network=gluetun.container
|
||||
# Pinned to the last 4.x line: Readarr's retired qB client requires the
|
||||
# qB 4.x login response (HTTP 200, body "Ok."); qB 5.x answers 204 with an
|
||||
# empty body, which fails Readarr's auth check. All other clients accept 4.x.
|
||||
Image=lscr.io/linuxserver/qbittorrent:4.6.7
|
||||
Image=lscr.io/linuxserver/qbittorrent:4.6.7@sha256:55f15d44396315551f87294a176efae733b16e283f38980308e46073950257c6
|
||||
Environment=PUID=0
|
||||
Environment=PGID=0
|
||||
Environment=TZ=America/Los_Angeles
|
||||
|
||||
@@ -15,7 +15,7 @@ After=gluetun.service
|
||||
|
||||
[Container]
|
||||
ContainerName=radarr
|
||||
Image=lscr.io/linuxserver/radarr:latest
|
||||
Image=lscr.io/linuxserver/radarr:latest@sha256:fc0ae3c50f8128b97e84381903ec8d6845c3a569641cb7117c124376cccb57c1
|
||||
Environment=PUID=0
|
||||
Environment=PGID=0
|
||||
Environment=TZ=America/Los_Angeles
|
||||
|
||||
@@ -19,7 +19,7 @@ After=gluetun.service
|
||||
|
||||
[Container]
|
||||
ContainerName=readarr
|
||||
Image=lscr.io/linuxserver/readarr:amd64-1.0.980
|
||||
Image=lscr.io/linuxserver/readarr:amd64-1.0.980@sha256:a95e82dbb0eb2bb43de4c150f1dab9dd821306c67ea0b418983a957e731f2b61
|
||||
Environment=PUID=0
|
||||
Environment=PGID=0
|
||||
Environment=TZ=America/Los_Angeles
|
||||
|
||||
@@ -16,7 +16,7 @@ After=gluetun.service
|
||||
|
||||
[Container]
|
||||
ContainerName=sonarr
|
||||
Image=lscr.io/linuxserver/sonarr:latest
|
||||
Image=lscr.io/linuxserver/sonarr:latest@sha256:889dcc58c6aea859033985894b519b2e6b1e55d92527ed123f9f52e42ae7c97b
|
||||
Environment=PUID=0
|
||||
Environment=PGID=0
|
||||
Environment=TZ=America/Los_Angeles
|
||||
|
||||
@@ -1,27 +1,45 @@
|
||||
#!/usr/bin/env bash
|
||||
# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite the recipe's
|
||||
# PROTECTED-BEGIN/END section from Swift's OWN calibration grams: run
|
||||
# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite a recipe's
|
||||
# PROTECTED-BEGIN/END section from the campaign's OWN calibration grams: run
|
||||
# derive-protected-experts.py over the calib dir, splice its cast lines into
|
||||
# the marked section of the recipe in place, keep the pre-edit recipe in the
|
||||
# campaign trash. Refuses to run twice (marker text says derived).
|
||||
# campaign trash. Refuses to run twice (marker text says derived), and refuses
|
||||
# when a requested target count was not reached. Run it on a COPY of the
|
||||
# template so the committed recipe stays the clean, reviewable baseline.
|
||||
set -euo pipefail
|
||||
CALIB=${1:-/home/dank/corpora/calib}
|
||||
RECIPE=${2:-/home/dank/serverAppConfigs-swift15/scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe}
|
||||
PY=/home/dank/venvs/radcorpus/bin/python
|
||||
DERIVE=/home/dank/serverAppConfigs-swift15/scripts/derive-protected-experts.py
|
||||
SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
CALIB=${1:-/home/dank/campaign/qwen38-w5nl64/calib}
|
||||
RECIPE=${2:-$SELF_DIR/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
|
||||
TARGET_COUNT=${3:-0}
|
||||
PY=${PY:-/home/dank/venvs/radcorpus/bin/python}
|
||||
DERIVE=${DERIVE:-$SELF_DIR/derive-protected-experts.py}
|
||||
TS=$(date +%Y%m%d-%H%M%S)
|
||||
# OUT and TRASH_DIR are overridable so the seam can drive the whole script in a
|
||||
# scratch directory; on the box they are the campaign's usual places.
|
||||
OUT=${OUT:-/tmp/protected-cast-$TS.txt}
|
||||
TRASH_DIR=${TRASH_DIR:-/home/dank/.AGENTTRASH}
|
||||
|
||||
grep -q "derive-protected-experts.py rewrote" "$RECIPE" && {
|
||||
echo "recipe already carries derived cast lines; refusing to re-derive" >&2; exit 1; }
|
||||
grep -q "PROTECTED-BEGIN" "$RECIPE" || { echo "no PROTECTED markers in $RECIPE" >&2; exit 1; }
|
||||
|
||||
"$PY" "$DERIVE" "$CALIB" > "/tmp/protected-cast-$TS.txt"
|
||||
N=$(grep -c 'cast dtype=bf16' "/tmp/protected-cast-$TS.txt")
|
||||
[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing" >&2; exit 1; }
|
||||
echo "derived $N protected experts -> /tmp/protected-cast-$TS.txt"
|
||||
ARGS=("$DERIVE" "$CALIB")
|
||||
if [ "$TARGET_COUNT" -gt 0 ]; then
|
||||
ARGS+=(--target-count "$TARGET_COUNT")
|
||||
fi
|
||||
"$PY" "${ARGS[@]}" > "$OUT"
|
||||
N=$(grep -c 'cast dtype=bf16' "$OUT")
|
||||
[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing (see $OUT)" >&2; exit 1; }
|
||||
if [ "$TARGET_COUNT" -gt 0 ] && [ "$N" -lt "$TARGET_COUNT" ]; then
|
||||
echo "derive reached $N protected experts, target was $TARGET_COUNT; refusing to convert" >&2
|
||||
echo " (the bar sweep bottomed out: the recording is too thin or the target too high)" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "derived $N protected experts -> $OUT"
|
||||
|
||||
cp -f "$RECIPE" "/home/dank/.AGENTTRASH/$TS-qwen4exp-swift15-w4dn8-i8-hc8m.recipe"
|
||||
"$PY" - "$RECIPE" "/tmp/protected-cast-$TS.txt" <<'PYEOF'
|
||||
mkdir -p "$TRASH_DIR"
|
||||
cp -f "$RECIPE" "$TRASH_DIR/$TS-$(basename "$RECIPE")"
|
||||
"$PY" - "$RECIPE" "$OUT" <<'PYEOF'
|
||||
import sys
|
||||
recipe, castfile = sys.argv[1], sys.argv[2]
|
||||
lines = open(recipe).read().splitlines()
|
||||
@@ -35,7 +53,7 @@ for l in lines:
|
||||
elif 'PROTECTED-END' in l:
|
||||
insec = False
|
||||
out.append(l)
|
||||
out.append('# (derive-protected-experts.py rewrote this section from Swift\'s own grams)')
|
||||
out.append('# (derive-protected-experts.py rewrote this section from the campaign grams)')
|
||||
elif not insec:
|
||||
out.append(l)
|
||||
open(recipe, 'w').write('\n'.join(out) + '\n')
|
||||
|
||||
Executable
+220
@@ -0,0 +1,220 @@
|
||||
#!/usr/bin/env bash
|
||||
# build-qwen38-bf16-rad.sh — RUNS ON DANKSERVER. rad-convert the vendor BF16
|
||||
# checkpoint /home/dank/models/Qwen3.8-Flash-Next into
|
||||
# /home/dank/models/rad/qwen38-bf16.rad with NO recipe and NO --reuse: a weight
|
||||
# no rule matches keeps the checkpoint's encoding, so every tensor stays the
|
||||
# bf16 it is in the safetensors and the container is the checkpoint's own
|
||||
# numbers behind a .rad header.
|
||||
#
|
||||
# This file is the campaign's measuring stick, twice over: scripts/
|
||||
# run-qwen38-bf16.sh kldref records the KLD reference logits from it (every
|
||||
# arm's --kld-ref is scored against that record) and the same container serves
|
||||
# with the calibration tap to write the Grams that derive the protected-expert
|
||||
# list and feed GPTQ. A reference that is not the vendor checkpoint, byte for
|
||||
# byte in bf16, silently moves every gate in the campaign — hence no recipe, no
|
||||
# --reuse, and the four refusal gates below.
|
||||
#
|
||||
# Why a container at all: a checkpoint dir has no mapped/file tiers, and a
|
||||
# weight only a host kernel reads (the 47.7 GiB PLE n-gram table) cannot be
|
||||
# served from one; rad-convert also bakes the tokenizer. The convert runs on
|
||||
# the HOST backend — no --device /dev/kfd, no /dev/dri — so declare resolves
|
||||
# kernels on CPU and production serving is untouched for the whole run (the
|
||||
# precedent that proved it: scripts/convert-swift15-w4dn8.sh).
|
||||
#
|
||||
# Gates, every one a refusal with exit 2 and a named reason, BEFORE anything is
|
||||
# written, cheapest first — the verifier runs last of them because its Hub gate
|
||||
# hashes 343 GiB and must not be paid for by a run that was going to refuse on
|
||||
# the output file anyway.
|
||||
# checkpoint the tree must be on disk; an absent dir names the command that
|
||||
# makes one (scripts/download-qwen38-bf16.sh --fetch) and stops.
|
||||
# free space the convert writes ~343,323 MiB next to the ~343,323 MiB
|
||||
# checkpoint; QWEN38_MIN_FREE_MIB (default 350000 = that figure
|
||||
# plus 6.5 GiB of slack) is the floor. The volume that cannot
|
||||
# hold both is discovered at minute 2, not hour 5.
|
||||
# output the campaign converts the reference ONCE. An existing
|
||||
# qwen38-bf16.rad is refused; --force retires it to
|
||||
# ~/.AGENTTRASH/<utc>-qwen38-bf16.rad (trash law: nothing is
|
||||
# deleted, and the move is a rename on the same volume).
|
||||
# verified scripts/download-qwen38-bf16.sh's verify mode must pass. A
|
||||
# failed verify exits 1, not 2: the tree being the wrong payload
|
||||
# is a result, a missing environment is a refusal.
|
||||
# plan rad-convert --plan-only -v first — the oracle for "does this
|
||||
# shape boot". A refused plan costs seconds and saves a convert.
|
||||
#
|
||||
# usage: scripts/build-qwen38-bf16-rad.sh [--dry-run] [--plan-only] [--force]
|
||||
# --dry-run print the argv it would exec (podman wrapping rad-convert),
|
||||
# one shell-quoted argument a line, and exit 0. No gate runs, no
|
||||
# container starts, nothing is written — the offline test seam.
|
||||
# It does resolve the checkpoint and output paths through the
|
||||
# /models mount, because an argv that carries host paths would
|
||||
# not run either.
|
||||
# --plan-only run the plan and every gate except the disk and output ones —
|
||||
# those two protect bytes the plan never writes, and this is the
|
||||
# probe to run when the disk gate itself is what refuses
|
||||
# --force retire an existing output container to the trash dir first
|
||||
#
|
||||
# knobs: FRANK_QWEN38_MODELS the host dir mounted at /models (default
|
||||
# /home/dank/models); FRANK_QWEN38_CKPT (default $MODELS/
|
||||
# Qwen3.8-Flash-Next, the same knob the verifier reads);
|
||||
# FRANK_QWEN38_BF16_OUT (default $MODELS/rad/qwen38-bf16.rad);
|
||||
# QWEN38_MIN_FREE_MIB; QWEN38_TRASH (default $HOME/.AGENTTRASH). The
|
||||
# verifier's own census knobs (FRANK_QWEN38_SHARDS / _BYTES / _REPO)
|
||||
# reach it through the environment untouched.
|
||||
# exit: 0 converted (or planned), 1 the checkpoint failed verification,
|
||||
# 2 refused before anything was written
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# ONE owner of the engine pin: the digest frank-archon serves on. The reference
|
||||
# must be built by the SAME engine build that scores it, or a KLD delta is
|
||||
# partly a version delta.
|
||||
# shellcheck disable=SC1091
|
||||
source "$SCRIPT_DIR/frank-archon.env"
|
||||
IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
|
||||
|
||||
MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
|
||||
CKPT="${FRANK_QWEN38_CKPT:-$MODELS/Qwen3.8-Flash-Next}"
|
||||
OUT="${FRANK_QWEN38_BF16_OUT:-$MODELS/rad/qwen38-bf16.rad}"
|
||||
TRASH="${QWEN38_TRASH:-$HOME/.AGENTTRASH}"
|
||||
MIN_FREE_MIB="${QWEN38_MIN_FREE_MIB:-350000}"
|
||||
LOG="/home/dank/logs/qwen38_bf16_convert.log"
|
||||
|
||||
DRY=0
|
||||
PLAN_ONLY=0
|
||||
FORCE=0
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--dry-run) DRY=1 ;;
|
||||
--plan-only) PLAN_ONLY=1 ;;
|
||||
--force) FORCE=1 ;;
|
||||
*) echo "usage: ${0##*/} [--dry-run] [--plan-only] [--force]" >&2
|
||||
exit 2 ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
# The convert sees the tree only through the /models mount, so both paths must
|
||||
# be under it; the container-side spelling is what goes into the argv.
|
||||
case "$CKPT" in
|
||||
"$MODELS"/*) CKPT_IN="/models/${CKPT#"$MODELS"/}" ;;
|
||||
*) echo "REFUSED: checkpoint $CKPT is not under the mounted models dir $MODELS — the convert cannot see it" >&2
|
||||
exit 2 ;;
|
||||
esac
|
||||
case "$OUT" in
|
||||
"$MODELS"/*) OUT_IN="/models/${OUT#"$MODELS"/}" ;;
|
||||
*) echo "REFUSED: output $OUT is not under the mounted models dir $MODELS — rad-convert could not write it" >&2
|
||||
exit 2 ;;
|
||||
esac
|
||||
|
||||
# Declare with NO recipe line — that is what keeps every tensor bf16 — and at the
|
||||
# bound this container will actually be served at. Both flags come off the pinned
|
||||
# image's own help: --kv-cache-dtype at convert "only picks the kernels it
|
||||
# checks" and the container is identical either way, so naming fp8 is free
|
||||
# insurance — if the fp8 KV kernels the fleet serves would not resolve, the
|
||||
# convert refuses here instead of the boot failing on a drained window.
|
||||
# --max-ctx is the bound declare is run at, whose default is the checkpoint's
|
||||
# training context (262,144 here, whose fp8 cache is 258.8 GiB against a 192 GiB
|
||||
# machine, so the default is not an option). 32,768 is set by the corpus, not by
|
||||
# taste: kld.jsonl's longest record is 38,263 characters — ~9,600 tokens at the
|
||||
# pessimistic chars/4, ~7,700 at the chars/4.97 this repo measured on the K3
|
||||
# corpus — so the window the rig boots at clears every record 2.6x over and the
|
||||
# reference boot cannot die on a sequence that will not fit. The rig's window is
|
||||
# one knob (FRANK_QWEN38_MAX_MODEL_LEN in scripts/run-qwen38-bf16.sh, default
|
||||
# 32,768) and that script refuses a corpus its window cannot hold, so raising one
|
||||
# without the other is caught before the drained window, not inside it. The
|
||||
# payload containers the gate scores declare at the training context because
|
||||
# production must hold the 200k shape; a declared bound is a declare-time check,
|
||||
# not a weight change, so the reference and the payloads carry the same numbers
|
||||
# whatever each declared at — which is why one can be measured against the other.
|
||||
CONVERT_ARGS=("$CKPT_IN" -o "$OUT_IN" --max-ctx 32768 --kv-cache-dtype fp8)
|
||||
PODMAN=(podman run --rm --entrypoint rad-convert -v "$MODELS:/models")
|
||||
PLAN_CMD=("${PODMAN[@]}" --name qwen38-bf16-plan "$IMAGE" "${CONVERT_ARGS[@]}" --plan-only -v)
|
||||
CONV_CMD=("${PODMAN[@]}" --name qwen38-bf16-convert "$IMAGE" "${CONVERT_ARGS[@]}")
|
||||
|
||||
if [ "$DRY" = "1" ]; then
|
||||
echo "dry run: no gate checked, no container started, nothing written"
|
||||
echo "# $CKPT -> $OUT (the convert sees them as $CKPT_IN and $OUT_IN)"
|
||||
echo "# plan argv:"
|
||||
printf '%q\n' "${PLAN_CMD[@]}"
|
||||
echo "# convert argv:"
|
||||
printf '%q\n' "${CONV_CMD[@]}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ---- gate: the checkpoint is on disk ---------------------------------------
|
||||
if [ ! -d "$CKPT" ]; then
|
||||
echo "REFUSED: no checkpoint at $CKPT — fetch and verify it first: scripts/download-qwen38-bf16.sh --fetch" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# ---- gate: peak disk --------------------------------------------------------
|
||||
# The deepest existing ancestor is the right thing to ask: a fresh
|
||||
# /home/dank/models/rad has no free space of its own to report.
|
||||
OUT_DIR="$(dirname "$OUT")"
|
||||
PROBE="$OUT_DIR"
|
||||
while [ ! -d "$PROBE" ] && [ "$PROBE" != "/" ]; do PROBE="$(dirname "$PROBE")"; done
|
||||
FREE_MIB="$(df --output=avail -m "$PROBE" | tail -1 | tr -d ' ')"
|
||||
if ! [[ "$FREE_MIB" =~ ^[0-9]+$ ]]; then
|
||||
echo "REFUSED: cannot read the free space under $PROBE (df said: $FREE_MIB)" >&2
|
||||
exit 2
|
||||
fi
|
||||
if [ "$PLAN_ONLY" != "1" ] && [ "$FREE_MIB" -lt "$MIN_FREE_MIB" ]; then
|
||||
echo "REFUSED peak-disk: $FREE_MIB MiB free under $PROBE, $MIN_FREE_MIB MiB wanted (QWEN38_MIN_FREE_MIB) — the bf16 container rewrites the checkpoint's own 343,323 MiB of tensors beside it" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# ---- gate: one conversion ---------------------------------------------------
|
||||
if [ -e "$OUT" ]; then
|
||||
if [ "$PLAN_ONLY" = "1" ]; then
|
||||
echo "note: $OUT already exists; --plan-only writes nothing, so it stays"
|
||||
elif [ "$FORCE" = "1" ]; then
|
||||
mkdir -p "$TRASH"
|
||||
DEST="$TRASH/$(date -u +%Y%m%dT%H%M%SZ)-$(basename "$OUT")"
|
||||
mv "$OUT" "$DEST"
|
||||
echo "TRASHED: retired the existing reference container $OUT -> $DEST"
|
||||
else
|
||||
echo "REFUSED: $OUT already exists — the campaign converts the reference once; re-run with --force to retire it to $TRASH first" >&2
|
||||
exit 2
|
||||
fi
|
||||
fi
|
||||
|
||||
# ---- gate: the tree really is the checkpoint (the full verify) -------------
|
||||
echo "== verifying $CKPT (scripts/download-qwen38-bf16.sh, full gates)"
|
||||
rc=0
|
||||
bash "$SCRIPT_DIR/download-qwen38-bf16.sh" --dir "$CKPT" || rc=$?
|
||||
if [ "$rc" -ne 0 ]; then
|
||||
if [ "$rc" -eq 2 ]; then
|
||||
echo "REFUSED: the checkpoint verifier refused the environment (rc=2); nothing was converted" >&2
|
||||
exit 2
|
||||
fi
|
||||
echo "REFUSED: $CKPT failed verification (rc=$rc) — a half tree converts to a half reference" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ---- plan, then convert ------------------------------------------------------
|
||||
mkdir -p "$(dirname "$LOG")"
|
||||
# rad-convert opens -o during declare as well, so the output directory exists
|
||||
# before the plan; that empty directory is all this creates ahead of it.
|
||||
mkdir -p "$OUT_DIR"
|
||||
echo "== rad-convert --plan-only -v (declare + plan; writes nothing)"
|
||||
"${PLAN_CMD[@]}" 2>&1 | tee -a "$LOG"
|
||||
rc=${PIPESTATUS[0]}
|
||||
if [ "$rc" -ne 0 ]; then
|
||||
echo "REFUSED plan: rad-convert --plan-only exited rc=$rc — the engine refuses this encoding/shape, no container was written (log: $LOG)" >&2
|
||||
exit 2
|
||||
fi
|
||||
if [ "$PLAN_ONLY" = "1" ]; then
|
||||
echo "PLAN_OK: $CKPT_IN declares for $OUT_IN; no container written (log: $LOG)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "== convert: $CKPT -> $OUT ($(date -Is))"
|
||||
"${CONV_CMD[@]}" 2>&1 | tee -a "$LOG"
|
||||
rc=${PIPESTATUS[0]}
|
||||
if [ "$rc" -ne 0 ]; then
|
||||
echo "rad-convert rc=$rc; log: $LOG" >&2
|
||||
exit "$rc"
|
||||
fi
|
||||
MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
|
||||
echo "BUILD_DONE: $OUT is ${MIB} MiB (log: $LOG)"
|
||||
echo "NEXT: scripts/run-qwen38-bf16.sh kldref records the reference logits from it"
|
||||
+149
-18
@@ -22,6 +22,17 @@ Mechanics notes:
|
||||
prompts do not bias the Hessians (the corpus is drawn round-robin over
|
||||
source classes, so loss is class-neutral as long as failures are rare;
|
||||
a systematic 4xx is loud in the log and must be fixed, not ignored).
|
||||
- a round that seals no tap bytes is NORMAL and continues: RADIANCE_CALIB_TOKENS
|
||||
is a pooled floor and this campaign's corpus is 10.03M tokens against a 12M
|
||||
floor, so round 1 seals nothing by construction. A tap that LOSES bytes fails
|
||||
the run, and a run that never sees the engine's completion line exits non-zero.
|
||||
The first version failed on a flat round, which aborted exactly one round too
|
||||
early (2026-10-29).
|
||||
- before any prompt, the tap must carry the rig's own marker
|
||||
(`--tap-marker`, written by run-qwen38-bf16.sh calib) naming THIS
|
||||
container, and that container must be running. Measured 2026-10-29:
|
||||
with neither check, four rounds of 10M tokens went into restored
|
||||
production after the rig's boot had died, and recorded nothing.
|
||||
|
||||
Usage: calib-push.py [--port 7770] [--parallel 64] [--container arbench]
|
||||
[--corpus /home/dank/corpora/out/calib.jsonl] [--max-rounds 4]
|
||||
@@ -29,11 +40,18 @@ Usage: calib-push.py [--port 7770] [--parallel 64] [--container arbench]
|
||||
import argparse
|
||||
import concurrent.futures
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
_HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
|
||||
if _HERE not in sys.path:
|
||||
sys.path.insert(0, _HERE)
|
||||
|
||||
import qwen38_common as qc # noqa: E402 - sibling module, same directory
|
||||
|
||||
DONE_MARK = "calibration wrote"
|
||||
|
||||
|
||||
@@ -49,9 +67,9 @@ def tap_done(container):
|
||||
return DONE_MARK in (out.stdout + out.stderr)
|
||||
|
||||
|
||||
def push_one(url, prompt):
|
||||
def push_one(url, prompt, model):
|
||||
body = json.dumps({
|
||||
"model": "frank-archon",
|
||||
"model": model,
|
||||
"prompt": prompt,
|
||||
"max_tokens": 1,
|
||||
"temperature": 0,
|
||||
@@ -68,38 +86,151 @@ def push_one(url, prompt):
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
def push_round(url, prompts, model, parallel):
|
||||
"""Push the whole corpus once; returns how many requests were accepted."""
|
||||
ok = 0
|
||||
with concurrent.futures.ThreadPoolExecutor(parallel) as ex:
|
||||
for good in ex.map(lambda p: push_one(url, p, model), prompts):
|
||||
ok += 1 if good else 0
|
||||
return ok
|
||||
|
||||
|
||||
def tap_bytes(path):
|
||||
"""Bytes the tap has sealed so far; 0 when the directory is absent.
|
||||
|
||||
A tap whose directory is wrong, full or read-only stops writing while the
|
||||
engine keeps answering 200 — the driver then re-pushes the whole corpus for
|
||||
every round it is allowed and the loss is invisible until the hours are gone.
|
||||
"""
|
||||
total = 0
|
||||
try:
|
||||
for root, _, files in os.walk(path):
|
||||
for f in files:
|
||||
try:
|
||||
total += os.path.getsize(os.path.join(root, f))
|
||||
except OSError:
|
||||
pass
|
||||
except OSError:
|
||||
return 0
|
||||
return total
|
||||
|
||||
|
||||
def round_verdict(before, after, done):
|
||||
"""What one round's tap bytes mean.
|
||||
|
||||
'done' the engine logged that it sealed the tap — the run is over
|
||||
'shrink' the tap LOST bytes: reset, pruned or replaced under us
|
||||
'grew' sealed bytes this round
|
||||
'flat' sealed nothing — which is legitimate: RADIANCE_CALIB_TOKENS is a
|
||||
pooled floor, and a round that does not reach it has nothing to
|
||||
write yet (the campaign's corpus is 10.03M tokens against a 12M
|
||||
floor, so round 1 is always flat). Failing on it aborted the run
|
||||
exactly one round too early.
|
||||
"""
|
||||
if done:
|
||||
return "done"
|
||||
if after < before:
|
||||
return "shrink"
|
||||
if after == before:
|
||||
return "flat"
|
||||
return "grew"
|
||||
|
||||
|
||||
def require_rig(a):
|
||||
"""Refuse unless the rig that opened this tap is up and serving right now.
|
||||
|
||||
Three facts, in order: the runner wrote its marker into the tap dir, the
|
||||
marker names the container this pusher targets, and that container is
|
||||
running. A marker without a live container is a boot that died; a live
|
||||
container without a marker is some other engine on the port — on 2026-10-29
|
||||
it was restored production, which took four rounds of 10M tokens before the
|
||||
old driver admitted it had recorded nothing.
|
||||
"""
|
||||
marker = os.path.join(a.tap_dir, a.tap_marker)
|
||||
if not os.path.isfile(marker):
|
||||
qc.refuse(f"no rig marker at {marker}: this tap was not opened by "
|
||||
"run-qwen38-bf16.sh calib, so a push here would feed whatever "
|
||||
"answers the port and record nothing")
|
||||
try:
|
||||
lines = open(marker, encoding="utf-8", errors="replace").read().splitlines()
|
||||
except OSError as exc:
|
||||
qc.refuse(f"cannot read the rig marker {marker}: {exc}")
|
||||
meta = dict(line.split("=", 1) for line in lines if "=" in line)
|
||||
who = meta.get("container", "")
|
||||
if who != a.container:
|
||||
qc.refuse(f"the rig marker names container {who!r} and this pusher targets "
|
||||
f"{a.container!r} — wrong tap, or a stale marker")
|
||||
try:
|
||||
p = subprocess.run(["podman", "inspect", "-f", "{{.State.Running}}", a.container],
|
||||
capture_output=True, text=True, timeout=30)
|
||||
except Exception as exc:
|
||||
qc.refuse(f"cannot inspect the rig container {a.container}: {exc}")
|
||||
if p.returncode != 0 or p.stdout.strip() != "true":
|
||||
qc.refuse(f"the rig container {a.container} is not running — whatever is "
|
||||
"answering the port, it is not the calibration rig")
|
||||
|
||||
|
||||
def parse_args():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--port", type=int, default=7770)
|
||||
ap.add_argument("--parallel", type=int, default=64)
|
||||
ap.add_argument("--container", default="arbench")
|
||||
ap.add_argument("--model", default="frank-archon")
|
||||
ap.add_argument("--tap-dir", default="/home/dank/campaign/qwen38-w5nl64/calib")
|
||||
ap.add_argument("--tap-marker", default=".rig-boot",
|
||||
help="the file run-qwen38-bf16.sh calib writes into the tap dir; without it the pusher refuses, because an unmarked tap is not a rig")
|
||||
ap.add_argument("--corpus", default="/home/dank/corpora/out/calib.jsonl")
|
||||
ap.add_argument("--max-rounds", type=int, default=4)
|
||||
a = ap.parse_args()
|
||||
url = f"http://127.0.0.1:{a.port}/v1/completions"
|
||||
return ap.parse_args()
|
||||
|
||||
|
||||
def load_prompts(a) -> list[str]:
|
||||
if not os.path.isfile(a.corpus):
|
||||
qc.refuse(f"no corpus at {a.corpus}")
|
||||
prompts = []
|
||||
with open(a.corpus) as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if line:
|
||||
prompts.append(json.loads(line)["prompt"])
|
||||
print(f"corpus: {len(prompts)} prompts", flush=True)
|
||||
for lineno, rec in qc.iter_jsonl(a.corpus):
|
||||
if not isinstance(rec.get("prompt"), str):
|
||||
qc.refuse(f"{a.corpus}:{lineno} has no string 'prompt' field")
|
||||
prompts.append(rec["prompt"])
|
||||
if not prompts:
|
||||
qc.refuse(f"the corpus at {a.corpus} holds no prompts")
|
||||
return prompts
|
||||
|
||||
|
||||
def main():
|
||||
a = parse_args()
|
||||
url = f"http://127.0.0.1:{a.port}/v1/completions"
|
||||
prompts = load_prompts(a)
|
||||
require_rig(a)
|
||||
chars = sum(len(p) for p in prompts)
|
||||
print(f"corpus: {len(prompts)} prompts, ~{chars // 4} tokens (chars/4)",
|
||||
flush=True)
|
||||
|
||||
for rnd in range(1, a.max_rounds + 1):
|
||||
if tap_done(a.container):
|
||||
break
|
||||
before = tap_bytes(a.tap_dir)
|
||||
t0 = time.time()
|
||||
ok = 0
|
||||
with concurrent.futures.ThreadPoolExecutor(a.parallel) as ex:
|
||||
for good in ex.map(lambda p: push_one(url, p), prompts):
|
||||
ok += 1 if good else 0
|
||||
dt = time.time() - t0
|
||||
ok = push_round(url, prompts, a.model, a.parallel)
|
||||
dt = max(time.time() - t0, 1e-9)
|
||||
print(f"round {rnd}: {ok}/{len(prompts)} ok in {dt:.0f}s "
|
||||
f"({sum(len(p) for p in prompts)/4/dt:.0f} chars/s est)",
|
||||
flush=True)
|
||||
f"({chars / 4 / dt:.0f} tokens/s est)", flush=True)
|
||||
if ok == 0:
|
||||
print("FAIL: not one prompt was accepted; fix the engine or the argv "
|
||||
"before spending another round", flush=True)
|
||||
sys.exit(1)
|
||||
if tap_done(a.container):
|
||||
break
|
||||
after = tap_bytes(a.tap_dir)
|
||||
verdict = round_verdict(before, after, False)
|
||||
if verdict == "shrink":
|
||||
print(f"FAIL: the tap at {a.tap_dir} LOST bytes this round "
|
||||
f"({before} B before, {after} B after) — it was reset or replaced "
|
||||
"under us", flush=True)
|
||||
sys.exit(1)
|
||||
if verdict == "flat":
|
||||
print(f"round {rnd}: the tap sealed nothing yet (byte floor not reached; "
|
||||
f"{before} B) — continuing", flush=True)
|
||||
else:
|
||||
print("max rounds exhausted, tap not done", flush=True)
|
||||
sys.exit(1)
|
||||
|
||||
@@ -21,7 +21,7 @@
|
||||
# deployed knob has since dropped to 262144.
|
||||
set -euo pipefail
|
||||
|
||||
IMG=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
|
||||
IMG=ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0
|
||||
CUR=/home/dank/models/halogen-flashnext-v2
|
||||
NEW=/home/dank/models/halogen-v2-abliterated
|
||||
CUR_CK=qwen38-flash-next-v2-swift15-abliterated.hgn
|
||||
|
||||
Executable
+236
@@ -0,0 +1,236 @@
|
||||
#!/usr/bin/env bash
|
||||
# convert-qwen38-w5nl64.sh — RUNS ON DANKSERVER. The CPU half of the Qwen3.8-Flash-Next five-bit
|
||||
# campaign: convert the vendor BF16 checkpoint into the campaign's arms, each one the whole
|
||||
# `w5nl64a8h` rung on BOTH expert projections with the protected experts kept bf16.
|
||||
#
|
||||
# Nothing here touches a GPU: the container is started without /dev/kfd or /dev/dri, so the
|
||||
# converter runs on the host backend and the fleet can keep serving through the whole script.
|
||||
# The engine pin is the digest that serves frank-archon, so a winning arm crosses no version
|
||||
# boundary between measurement and serving.
|
||||
#
|
||||
# Three arms, because the campaign's question is which search knob pays for its hours
|
||||
# (recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe):
|
||||
#
|
||||
# cd3 the recipe as written — coordinate descent three sweeps. The control.
|
||||
# cd6 six sweeps: the same search, twice the passes.
|
||||
# ao1 act_order=1 instead of cd: GPTQ sweeps columns in activation-importance order and
|
||||
# stores them permuted. This is a SEPARATE arm, not a fourth sweep — libquant refuses
|
||||
# cd= together with act_order=1 ("cd= sweeps the columns in their stored order, and
|
||||
# act_order=1 stores them permuted; one or the other", libquant/lq_registry.cpp), so
|
||||
# the two knobs cannot be combined into a best-of-both container.
|
||||
#
|
||||
# The arms are generated from ONE template with `--quant` overrides, not three near-duplicate
|
||||
# recipes: --quant rules are inserted BEFORE the recipe file and the first rule whose glob matches
|
||||
# a weight decides it (core/format/recipe.cpp: Recipe::match), so an override cleanly replaces the
|
||||
# expert line and the other 60 lines cannot drift between arms.
|
||||
#
|
||||
# Protected experts: apply-protected-cast.sh derives the bf16 set from THIS campaign's grams
|
||||
# (--target-count 48, two per layer so each rank's 256-expert half stays even) and splices it into
|
||||
# a per-run COPY of the template; the committed recipe keeps its markers and stays reviewable.
|
||||
#
|
||||
# Gates, in the order they bite — each one is a lesson the swift15 campaign paid for:
|
||||
# inputs checkpoint (dir + index), recipe, grams, imatrix, tokenizer. A missing grams
|
||||
# directory converts UNCALIBRATED with one stderr note, which is silently worse
|
||||
# than failing here.
|
||||
# reference the 335.3 GiB BF16 reference container must be GONE before the converts start.
|
||||
# It and the checkpoint together eat the volume; its job (the KLD recording and
|
||||
# the calibration grams) is finished by then, and both survive on disk without the
|
||||
# container beside it. The FP8-native anchor is FETCHED after the converts (disk
|
||||
# sequencing), so nothing here waits on it.
|
||||
# plan --plan-only runs FIRST, per arm, and prints "about X would be written". The
|
||||
# number is the bill: free space must cover it plus the 128k-prompt overshoot
|
||||
# margin, and the finished artifact is checked against it.
|
||||
# outputs no arm overwrites anything; move the old one to trash.
|
||||
#
|
||||
# Usage: convert-qwen38-w5nl64.sh [--arm cd3|cd6|ao1]... [--dry-run] [--plan-only]
|
||||
# [--target-count N] [--keep-recipe]
|
||||
set -u -o pipefail
|
||||
|
||||
SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# THE TOOLCHAIN PIN. `:latest` on 2026-10-30 is v1.14.0 (revision 4a61d7296bc5),
|
||||
# the same image the tap producer boots: a tap must be converted by the line that
|
||||
# is current, and rad-convert's `gptq:calib=<dir>` input contract is version-stable
|
||||
# (RADGRAM2 + gram.r<rank>.<stem>.bin), so nothing else about the recipe moves with
|
||||
# it. Production stays on its own pin — this is the campaign's toolchain.
|
||||
IMAGE=${IMAGE:-stilldeadcode/radiance@sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}
|
||||
CKPT=${CKPT:-/home/dank/models/Qwen3.8-Flash-Next}
|
||||
CAMPAIGN=${CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}
|
||||
RECIPE=${RECIPE:-$SELF/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
|
||||
CALIB=${CALIB:-$CAMPAIGN/calib}
|
||||
IMATRIX=${IMATRIX:-$CAMPAIGN/imatrix.gguf}
|
||||
OUTDIR=${OUTDIR:-/home/dank/models/rad}
|
||||
BF16_RAD=${BF16_RAD:-$OUTDIR/qwen38-bf16.rad} # the reference container, not an arm output
|
||||
STATE=${STATE:-$CAMPAIGN} # the dir the container mounts read-only: the
|
||||
# spliced recipe and the imatrix live here
|
||||
TARGET_COUNT=${TARGET_COUNT:-48}
|
||||
MIN_GRAMS=${MIN_GRAMS:-24576} # per-expert DOWN grams: 48 recorded layers x 512
|
||||
# experts. The checkpoint has 49 MoE layers, but the
|
||||
# MTP layer is never recorded, so its 512 experts are
|
||||
# absent from the tap's output by design
|
||||
MARGIN_MIB=${MARGIN_MIB:-40960} # the long-prompt overshoot, off the plan number
|
||||
TS=$(date +%Y%m%d-%H%M%S)
|
||||
|
||||
ARMS=()
|
||||
DRY=0
|
||||
PLAN_ONLY=0
|
||||
KEEP_RECIPE=0
|
||||
while [ "$#" -gt 0 ]; do
|
||||
case "$1" in
|
||||
--arm) [ "$#" -ge 2 ] || { echo "--arm needs a value (cd3|cd6|ao1)" >&2; exit 2; }
|
||||
ARMS+=("$2"); shift 2 ;;
|
||||
--dry-run) DRY=1; shift ;;
|
||||
--plan-only) PLAN_ONLY=1; shift ;;
|
||||
--target-count) [ "$#" -ge 2 ] || { echo "--target-count needs a value" >&2; exit 2; }
|
||||
TARGET_COUNT="$2"; shift 2 ;;
|
||||
--keep-recipe) KEEP_RECIPE=1; shift ;;
|
||||
*) echo "unknown argument: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
[ "${#ARMS[@]}" -gt 0 ] || ARMS=(cd3 cd6 ao1)
|
||||
for a in "${ARMS[@]}"; do
|
||||
case "$a" in cd3|cd6|ao1) ;; *) echo "unknown arm: $a (cd3|cd6|ao1)" >&2; exit 2 ;; esac
|
||||
done
|
||||
|
||||
# The expert rule's option set, spelled once. --quant replaces the template's line wholesale, so
|
||||
# every option has to be repeated here; `calib=$CALIB` is expanded by the recipe reader from the
|
||||
# container environment (core/format/recipe.cpp: expand refuses an unset name rather than taking
|
||||
# it for empty).
|
||||
expert_opts='table=w5nl,group=64,scale=fp8_e4m3,scale2=f32,block2=*x*,scale2_value=0.0001220703125,transform=fwht128,rule=search,calib=$CALIB'
|
||||
arm_quant() {
|
||||
case "$1" in
|
||||
# The control is the template itself: no override, so the recipe's own line decides.
|
||||
cd3) return 0 ;;
|
||||
cd6) echo "blk.*.ffn_*_exps.*.weight=gptq:${expert_opts},cd=6" ;;
|
||||
# act_order=1 WITHOUT cd: libquant refuses the pair (lq_registry.cpp:105, "cd= sweeps the
|
||||
# columns in their stored order, and act_order=1 stores them permuted; one or the other"), so
|
||||
# this arm is the interleave knob alone, not cd=3 plus it.
|
||||
ao1) echo "blk.*.ffn_*_exps.*.weight=gptq:${expert_opts},act_order=1" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# --- gates ------------------------------------------------------------------------------------
|
||||
[ -d "$CKPT" ] || { echo "no checkpoint at $CKPT; run scripts/download-qwen38-bf16.sh --fetch" >&2; exit 2; }
|
||||
[ -f "$CKPT/model.safetensors.index.json" ] || { echo "checkpoint at $CKPT has no safetensors index — a torn fetch" >&2; exit 2; }
|
||||
[ -f "$CKPT/tokenizer.json" ] || { echo "no tokenizer.json in $CKPT; the .rad cannot carry one without it" >&2; exit 2; }
|
||||
[ -f "$RECIPE" ] || { echo "no recipe at $RECIPE" >&2; exit 2; }
|
||||
GRAMS=$(find "$CALIB" -maxdepth 1 -name 'gram.r*.bin' 2>/dev/null | wc -l)
|
||||
[ "$GRAMS" -ge "$MIN_GRAMS" ] || {
|
||||
echo "calib dir $CALIB holds $GRAMS gram files (<$MIN_GRAMS); run $0 after run-qwen38-bf16.sh calib" >&2; exit 2; }
|
||||
[ -f "$IMATRIX" ] || { echo "no imatrix at $IMATRIX; run scripts/grams-to-imatrix.py over $CALIB" >&2; exit 2; }
|
||||
# The container sees the campaign dir at one fixed mount point, so an imatrix kept elsewhere has
|
||||
# no path inside it: copy it in (or point IMATRIX at the campaign dir) rather than let the
|
||||
# converter fail halfway through a planned arm.
|
||||
[ "$(dirname "$IMATRIX")" = "$STATE" ] || {
|
||||
echo "imatrix $IMATRIX is not in the campaign dir $STATE the container mounts; move it there" >&2; exit 2; }
|
||||
if [ -e "$BF16_RAD" ]; then
|
||||
echo "the 335.3 GiB BF16 reference container $BF16_RAD is still on disk; its recording and" \
|
||||
"the calibration grams are already taken, so move it to trash — the converts need its" \
|
||||
"space (peak-disk law)" >&2
|
||||
exit 2
|
||||
fi
|
||||
for a in "${ARMS[@]}"; do
|
||||
[ -e "$OUTDIR/qwen38-w5nl64-$a.rad" ] && {
|
||||
echo "$OUTDIR/qwen38-w5nl64-$a.rad already exists; move it to trash first" >&2; exit 2; }
|
||||
# A killed run leaves its container behind (--rm only fires when podman itself
|
||||
# exits), and the next arm then dies on "name already in use" AFTER the plan
|
||||
# phase; refuse here, where the fix is one `podman rm -f`.
|
||||
podman container exists "qwen38-convert-$a" && {
|
||||
echo "container qwen38-convert-$a still exists (a killed run?);" \
|
||||
"'podman rm -f qwen38-convert-$a' and re-run" >&2
|
||||
exit 2; }
|
||||
done
|
||||
|
||||
# --- the protected plane, spliced into a per-run copy of the template --------------------------
|
||||
mkdir -p "$STATE"
|
||||
WORK="$STATE/recipe-$TS.recipe"
|
||||
cp "$RECIPE" "$WORK"
|
||||
if [ "$DRY" = 1 ]; then
|
||||
echo "would run: $SELF/apply-protected-cast.sh $CALIB $WORK $TARGET_COUNT"
|
||||
else
|
||||
# Positional, because the tool reads its recipe and calib dir by position: passing them as
|
||||
# environment variables would be silently ignored and it would rewrite the COMMITTED
|
||||
# template in place instead of this run's copy.
|
||||
"$SELF/apply-protected-cast.sh" "$CALIB" "$WORK" "$TARGET_COUNT" \
|
||||
|| { echo "the protected-expert derivation refused; the arms would convert unprotected" >&2; exit 1; }
|
||||
fi
|
||||
|
||||
run_convert() { # run_convert ARM [extra flags...]
|
||||
local arm="$1"; shift
|
||||
local out="$OUTDIR/qwen38-w5nl64-$arm.rad" log="$STATE/convert-$arm-$TS.log"
|
||||
local argv=(--rm --name "qwen38-convert-$arm"
|
||||
-e "CALIB=/data/calib" -e "OMP_NUM_THREADS=$(nproc)"
|
||||
-v "$(dirname "$CKPT"):/models:ro"
|
||||
-v "$CALIB:/data/calib:ro"
|
||||
-v "$STATE:/campaign:ro"
|
||||
-v "$OUTDIR:/out")
|
||||
local conv=(/models/"${CKPT##*/}" --recipe "/campaign/$(basename "$WORK")"
|
||||
--tokenizer "/models/${CKPT##*/}/tokenizer.json"
|
||||
--imatrix /campaign/"${IMATRIX##*/}"
|
||||
# No --max-ctx: the context declare then runs at the checkpoint's training context
|
||||
# (262144), which is what the shipped container did — rad-info --meta on it carries
|
||||
# no override. A container declared at a smaller bound cannot be expected to serve
|
||||
# the 200k window D2 holds, and the convert is not where to discover that.
|
||||
# --kv-cache-dtype changes nothing in the artifact; it picks the kernels the declare
|
||||
# checks, and the fleet serves fp8 KV, so an unresolvable fp8 kernel must be said no
|
||||
# to by the convert, not by a boot on a drained window.
|
||||
--kv-cache-dtype fp8 -o "/out/$(basename "$out")")
|
||||
local q; q=$(arm_quant "$arm")
|
||||
[ -n "$q" ] && conv+=(--quant "$q")
|
||||
if [ "$DRY" = 1 ]; then
|
||||
echo "podman run ${argv[*]} \$IMAGE --entrypoint /opt/radiance/bin/rad-convert ${conv[*]} $*"
|
||||
return 0
|
||||
fi
|
||||
podman run "${argv[@]}" --entrypoint /opt/radiance/bin/rad-convert "$IMAGE" "${conv[@]}" "$@" \
|
||||
2>&1 | tee "$log"
|
||||
return "${PIPESTATUS[0]}"
|
||||
}
|
||||
|
||||
# --- plan, then bill, then convert -------------------------------------------------------------
|
||||
overall=0
|
||||
for arm in "${ARMS[@]}"; do
|
||||
echo "== arm $arm =="
|
||||
if [ "$DRY" = 1 ]; then
|
||||
# Both lines, in order: the plan runs before the convert, and the convert is what the
|
||||
# operator is about to pay an afternoon for.
|
||||
run_convert "$arm" --plan-only
|
||||
run_convert "$arm"
|
||||
continue
|
||||
fi
|
||||
rc=0; plan_log=$(run_convert "$arm" --plan-only) || rc=$?
|
||||
[ "$rc" -eq 0 ] || { echo "arm $arm: --plan-only failed (rc=$rc); nothing was written" >&2; overall=1; continue; }
|
||||
echo "$plan_log"
|
||||
# "about 141.2 GiB would be written" is the converter's own bill; parse it rather than
|
||||
# hardcoding a size, so the gate still works after any recipe edit.
|
||||
want=$(printf '%s' "$plan_log" | sed -n 's/.*about \([0-9.]*\) \([KMGT]iB\) would be written.*/\1 \2/p' | tail -1)
|
||||
[ -n "$want" ] || { echo "arm $arm: the plan printed no size; refusing to convert blind" >&2; overall=1; continue; }
|
||||
need_mib=$(awk -v n="${want% *}" -v u="${want#* }" \
|
||||
'BEGIN{ f = (u=="KiB") ? 2^-10 : (u=="MiB") ? 1 : (u=="GiB") ? 1024 : (u=="TiB") ? 1048576 : 0
|
||||
if (f == 0) { exit 1 }; printf "%d", n*f + 0.5 }') || {
|
||||
echo "arm $arm: the plan printed an unknown unit in '$want'" >&2; overall=1; continue; }
|
||||
free_mib=$(df --output=avail -m "$OUTDIR" | tail -1 | tr -d ' ')
|
||||
echo "plan: $want ($need_mib MiB); free ${free_mib} MiB on $OUTDIR (margin $MARGIN_MIB MiB)"
|
||||
if [ "$free_mib" -lt $((need_mib + MARGIN_MIB)) ]; then
|
||||
echo "arm $arm: needs ${need_mib} MiB + ${MARGIN_MIB} margin, ${free_mib} free; free space and re-run" >&2
|
||||
overall=1; continue
|
||||
fi
|
||||
if [ "$PLAN_ONLY" = 1 ]; then echo "planned, not converted (--plan-only)"; continue; fi
|
||||
rc=0; run_convert "$arm" || rc=$?
|
||||
[ "$rc" -eq 0 ] || { echo "arm $arm: rad-convert rc=$rc; log: $STATE/convert-$arm-$TS.log" >&2; overall=1; continue; }
|
||||
out="$OUTDIR/qwen38-w5nl64-$arm.rad"
|
||||
[ -f "$out" ] || { echo "arm $arm: rad-convert exited 0 but wrote no $out; log: $STATE/convert-$arm-$TS.log" >&2; overall=1; continue; }
|
||||
got_mib=$(( $(stat -c %s "$out") / 1048576 ))
|
||||
# The receipt: the artifact is what the plan predicted, within the metadata the plan does not
|
||||
# count. An arm that came out materially smaller lost a plane somewhere.
|
||||
lo=$(( need_mib * 98 / 100 )); hi=$(( need_mib * 103 / 100 ))
|
||||
if [ "$got_mib" -lt "$lo" ] || [ "$got_mib" -gt "$hi" ]; then
|
||||
echo "arm $arm: $out is ${got_mib} MiB, the plan said ${need_mib} MiB (allowed ${lo}-${hi})" >&2
|
||||
overall=1; continue
|
||||
fi
|
||||
echo "CONVERT_DONE $arm ${got_mib} MiB -> $out"
|
||||
done
|
||||
|
||||
if [ "$KEEP_RECIPE" = 0 ] && [ "$DRY" = 0 ]; then
|
||||
mkdir -p "$CAMPAIGN/.AGENTTRASH"
|
||||
mv "$WORK" "$CAMPAIGN/.AGENTTRASH/$(basename "$WORK")" 2>/dev/null || true
|
||||
fi
|
||||
exit "$overall"
|
||||
+16
-1
@@ -155,13 +155,14 @@ DANKCLAW_COMPOSE_SERVICES = set()
|
||||
# Quadlet unit now — the last compose services were the hermes trio, whose units
|
||||
# have owned their containers since 2026-09-12 and whose compose blocks were
|
||||
# dead weight that could only create a duplicate container.
|
||||
# jellyfin left the Dankserver file on 2026-10-30 for the same reason: its Quadlet
|
||||
# has owned the container since the cutover, so a compose edit must not restart it.
|
||||
|
||||
DANKSERVER_COMPOSE_SERVICES = {
|
||||
"mariadb",
|
||||
"redis",
|
||||
"romm",
|
||||
"copyparty",
|
||||
"jellyfin",
|
||||
"calibre-web",
|
||||
}
|
||||
COMPOSE_NAME_MAP = {"mariadb": "romm-mariadb", "redis": "romm-redis"}
|
||||
@@ -421,6 +422,11 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
"tautulli": {"tautulli"},
|
||||
}
|
||||
_add(services, *quadlet_services.get(parts[1], set()))
|
||||
if len(parts) >= 2 and parts[1] == "host":
|
||||
# dankserver/host/systemd/media-meta-view.mount is the overlay view the
|
||||
# media servers bind (scripts/media-view.sh installs it on the host),
|
||||
# so editing it restarts jellyfin and plex and nothing else.
|
||||
_add(services, "jellyfin", "plex")
|
||||
if len(parts) >= 2 and parts[1] == "observability":
|
||||
# Dankserver's node-exporter became a Quadlet unit on 2026-09-16
|
||||
# (serverAppConfigs-docs/Infrastructure/Observability.md): its
|
||||
@@ -466,6 +472,15 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
# that service (target_services then keeps it only on the host being
|
||||
# planned). Editing the collector restarts the running exporter.
|
||||
_add(services, "gpu-exporter")
|
||||
if name == "media-view.sh":
|
||||
# The installer is what creates, mounts and reaps the view both media
|
||||
# servers read, so a change to it is a restart of that pair.
|
||||
_add(services, "jellyfin", "plex")
|
||||
if name == "jellyfin-providers.sh":
|
||||
# The deployer runs this before jellyfin restarts, so a change to
|
||||
# the installer is a change to the provider state Jellyfin loads at
|
||||
# start-up: it selects jellyfin and nothing else.
|
||||
_add(services, "jellyfin")
|
||||
if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}:
|
||||
# llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve
|
||||
# files here; each one selects frank-zealot only. The ik_llama.cpp
|
||||
|
||||
@@ -252,16 +252,24 @@ install_signal_relay() {
|
||||
# The relay binds the host's LAN IP (the one address a bridge container
|
||||
# can reach as host.containers.internal), so keep the unauthenticated
|
||||
# endpoint off the rest of the LAN: admit loopback, the host itself
|
||||
# (its own health probe) and the podman bridge subnet only. Idempotent:
|
||||
# -C before -I.
|
||||
# (its own health probe) and the podman bridge subnet only.
|
||||
# ORDER IS THE GATE: `iptables -I` inserts at position 1, so the old
|
||||
# shape (-C/-I x3 ACCEPT, then -C/-I the DROP) landed the blanket DROP
|
||||
# ABOVE the ACCEPTs and every probe timed out beside a LISTENing socket
|
||||
# (live 2026-10-21: 159 packets on the DROP, 0 on each ACCEPT; the whole
|
||||
# dankclaw drain aborted at the alertmanager gate twice before the cause
|
||||
# was read off `nft list ruleset`). Rebuild the tail instead of inserting
|
||||
# it: delete any blanket DROP wherever it sits, keep the three ACCEPTs
|
||||
# at the head (-C before -I, unchanged), then APPEND the DROP so it is
|
||||
# last and the ACCEPTs decide first. Converges on poisoned hosts.
|
||||
run_privileged iptables -D INPUT -p tcp --dport 18093 -j DROP 2>/dev/null || true
|
||||
run_privileged iptables -C INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT 2>/dev/null \
|
||||
|| run_privileged iptables -I INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT
|
||||
run_privileged iptables -C INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT 2>/dev/null \
|
||||
|| run_privileged iptables -I INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT
|
||||
run_privileged iptables -C INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT 2>/dev/null \
|
||||
|| run_privileged iptables -I INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT
|
||||
run_privileged iptables -C INPUT -p tcp --dport 18093 -j DROP 2>/dev/null \
|
||||
|| run_privileged iptables -I INPUT -p tcp --dport 18093 -j DROP
|
||||
run_privileged iptables -A INPUT -p tcp --dport 18093 -j DROP
|
||||
# The interpreter needs a moment to bind after restart; a one-shot curl
|
||||
# raced the socket on 2026-09-17 and failed a healthy deploy.
|
||||
local i
|
||||
@@ -1194,7 +1202,42 @@ deploy_romm_stack() {
|
||||
systemctl --user restart "$service.service"
|
||||
}
|
||||
|
||||
ensure_media_view() {
|
||||
# Both media servers read their library through the SSD-backed overlay view
|
||||
# (scripts/media-view.sh, dankserver/host/systemd/media-meta-view.mount): the
|
||||
# array below stays read-only and every sidecar Jellyfin writes lands on the
|
||||
# NVMe.
|
||||
# Raise it BEFORE either container starts — a start against a bare view
|
||||
# directory would let a scan clear the library database. The units also carry
|
||||
# ConditionPathIsMountPoint, so a missing view skips the start rather than
|
||||
# damaging anything. A relayout must be followed by `media-view.sh reap`
|
||||
# (see the Media Relayout page), which this is not.
|
||||
run_privileged bash scripts/media-view.sh install
|
||||
bash scripts/media-view.sh verify
|
||||
}
|
||||
|
||||
ensure_provider_keys() {
|
||||
# Jellyfin's metadata-provider keys live in the container-owned plugin
|
||||
# configuration under the data root, never in git; scripts/jellyfin-providers.sh
|
||||
# is what places them, reading the per-service host env file (the fleet
|
||||
# convention: grafana, caddy, pihole, copyparty, romm, seerr) that holds
|
||||
# TMDB_API_KEY= and TVDB_API_KEY=, mode 600.
|
||||
# Missing keys mean a keyless provider — a degraded library, not a broken
|
||||
# deploy — so this warns rather than failing the restart. But when the keys
|
||||
# ARE placed it must happen before the restart: the plugin and its keys are
|
||||
# read at start-up, so a restart first would boot the old state.
|
||||
local keys=/home/dank/Containers/jellyfin/provider-keys
|
||||
if [ -f "$keys" ]; then
|
||||
bash scripts/jellyfin-providers.sh install
|
||||
else
|
||||
echo "WARN: $keys is absent - provider keys NOT placed"
|
||||
echo "WARN: create it (TMDB_API_KEY=, TVDB_API_KEY=, chmod 600) and re-deploy jellyfin"
|
||||
fi
|
||||
}
|
||||
|
||||
deploy_jellyfin() {
|
||||
ensure_media_view
|
||||
ensure_provider_keys
|
||||
cp "dankserver/jellyfin/quadlet/jellyfin.container" "$quadlet_unit_dir/"
|
||||
systemctl --user daemon-reload
|
||||
if ! systemctl --user is-active -q jellyfin.service; then
|
||||
@@ -1282,6 +1325,7 @@ deploy_copyparty() {
|
||||
}
|
||||
|
||||
deploy_plex() {
|
||||
ensure_media_view
|
||||
cp "dankserver/plex/quadlet/plex.container" "$quadlet_unit_dir/"
|
||||
systemctl --user daemon-reload
|
||||
if ! systemctl --user is-active -q plex.service; then
|
||||
@@ -1354,11 +1398,17 @@ deploy_authentik() {
|
||||
# generated". Boot persistence comes from [Install] WantedBy=default.target
|
||||
# inside each .container file, which the generator honors at user-manager
|
||||
# start (linger must stay on for the deploy user).
|
||||
systemctl --user start authentik-postgres.service
|
||||
# RESTART, not start: `start` on an already-active unit is a no-op, so an
|
||||
# image bump would install the new digest and keep serving the old
|
||||
# container while every gate stayed green (measured 2026-10-30: server and
|
||||
# worker stayed on the previous release beside an outpost that moved,
|
||||
# because only the outpost handler restarts). The generator has already
|
||||
# re-rendered the unit from the .container, so a restart is what applies it.
|
||||
systemctl --user restart authentik-postgres.service
|
||||
wait_container_healthy authentik-postgres 120 || return 1
|
||||
systemctl --user start authentik-server.service
|
||||
systemctl --user restart authentik-server.service
|
||||
wait_container_healthy authentik-server 300 || return 1
|
||||
systemctl --user start authentik-worker.service
|
||||
systemctl --user restart authentik-worker.service
|
||||
}
|
||||
|
||||
deploy_authentik_outpost() {
|
||||
@@ -1474,7 +1524,7 @@ deploy_anubis() {
|
||||
# artifact would leave a restart-looping unit and a dark endpoint. The
|
||||
# port gate is the same as every cutover since 2026-09-16: nothing
|
||||
# starts until 7770 is released.
|
||||
LLM_DRAGOON_IMAGE="${LLM_DRAGOON_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a}"
|
||||
LLM_DRAGOON_IMAGE="${LLM_DRAGOON_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0}"
|
||||
# The payload dir is its own twin because the NPU lane's containment gate below
|
||||
# reads it; every artifact path under it stays a full literal as usual.
|
||||
LLM_DRAGOON_MODEL_DIR="${LLM_DRAGOON_MODEL_DIR:-/home/dank/models/halogen-v2-abliterated}"
|
||||
|
||||
@@ -20,9 +20,16 @@ of the Gram is its down-input energy. Rank halves: 0-255, 256-511
|
||||
Output: the cast-rule lines (recipe format) on stdout, a report on stderr.
|
||||
|
||||
Usage: derive-protected-experts.py <calib dir> [--min-share 0.03]
|
||||
[--min-route-mult 3.0]
|
||||
[--min-route-mult 3.0] [--target-count 0]
|
||||
--target-count N fills the byte bill: the bars pick the base set exactly as
|
||||
the bar-only pass does, then the energy ranking admits experts one at a time
|
||||
(each with its parity partner) until the count lands. The energy bar NEVER
|
||||
relaxes, so asking for 48 bf16 experts never quietly buys 80; the ratio bar
|
||||
walks down only when the ranking cannot produce that many candidates.
|
||||
0 takes the bars as given.
|
||||
"""
|
||||
import argparse
|
||||
import collections
|
||||
import glob
|
||||
import os
|
||||
import re
|
||||
@@ -32,6 +39,7 @@ import sys
|
||||
import numpy as np
|
||||
|
||||
NAME_RE = re.compile(r"^gram\.r(\d)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
|
||||
CAST_RE = re.compile(r"^blk\.(\d+)\.ffn_\*_exps\.(\d+)\.weight")
|
||||
|
||||
|
||||
def read_gram(path):
|
||||
@@ -49,27 +57,8 @@ def read_gram(path):
|
||||
return k, rank, int(rows), tr
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("calib_dir")
|
||||
ap.add_argument("--min-share", type=float, default=0.03)
|
||||
ap.add_argument("--min-route-mult", type=float, default=3.0)
|
||||
a = ap.parse_args()
|
||||
|
||||
# layer -> global expert -> (rank, rows, energy)
|
||||
data = {}
|
||||
for path in sorted(glob.glob(os.path.join(
|
||||
a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
|
||||
m = NAME_RE.match(os.path.basename(path))
|
||||
if not m:
|
||||
continue
|
||||
rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
|
||||
k, _, rows, tr = read_gram(path)
|
||||
data.setdefault(layer, {})[expert] = (rank, rows, tr)
|
||||
|
||||
if not data:
|
||||
raise SystemExit("no per-expert down grams found")
|
||||
|
||||
def select(data, min_share, min_route_mult):
|
||||
"""(cast lines, report lines, count) for one pair of bars."""
|
||||
lines, report = [], []
|
||||
total = 0
|
||||
for layer in sorted(data):
|
||||
@@ -83,7 +72,7 @@ def main():
|
||||
for e, (rank, rows, tr) in experts.items():
|
||||
es, rs = tr / tot_e, rows / max(tot_r, 1)
|
||||
rec = (e, rank, es, rs)
|
||||
if es >= a.min_share and es >= a.min_route_mult * rs:
|
||||
if es >= min_share and es >= min_route_mult * rs:
|
||||
qual.append(rec)
|
||||
elif es > 0:
|
||||
pool.append(rec)
|
||||
@@ -107,10 +96,148 @@ def main():
|
||||
f"energy {es*100:5.2f}%, routing {rs*100:5.3f}%, "
|
||||
f"ratio {es/max(rs,1e-12):6.1f}")
|
||||
total += len(chosen)
|
||||
return lines, report, total
|
||||
|
||||
|
||||
def parity_repair(chosen, by_half, report, only=None):
|
||||
"""Add the next-best candidate to every half — or to `only` — holding an odd count.
|
||||
|
||||
The four-bit GEMM takes each half's remainder as two equal tables, so an odd count
|
||||
of bf16 experts on a half is not a style problem: the two tables would not match.
|
||||
`by_half` is energy-descending, so the first unchosen candidate is the next most
|
||||
energetic, which is the expert worth keeping. Returns how many were added.
|
||||
"""
|
||||
added = 0
|
||||
for key in ([only] if only is not None else sorted(by_half)):
|
||||
layer, half = key
|
||||
n = sum(1 for (l, x) in chosen if l == layer and x // 256 == half)
|
||||
if n % 2 == 0:
|
||||
continue
|
||||
for e, es, rs in by_half.get(key, []):
|
||||
if (layer, e) not in chosen:
|
||||
chosen[(layer, e)] = (es, rs)
|
||||
added += 1
|
||||
report.append(f"blk.{layer}: half {half} odd -> added e{e} "
|
||||
f"(energy {es*100:.2f}%)")
|
||||
break
|
||||
else:
|
||||
report.append(f"blk.{layer}: half {half} holds {n} and no candidate is "
|
||||
f"left to make it even")
|
||||
return added
|
||||
|
||||
|
||||
def select_to_count(data, min_share, min_route_mult, target):
|
||||
"""(lines, report, total, cutoff) for the set that LANDS the count.
|
||||
|
||||
The bars decide the base set; the count is then filled from the energy ranking,
|
||||
one expert at a time, each addition taking its half's next-best candidate with it
|
||||
so every rank's half stays even. Only the RATIO bar walks down to find candidates
|
||||
— it is the law that says an expert is disproportionate and not merely busy, and
|
||||
the caller's byte bill has to be filled — while the ENERGY bar holds, because that
|
||||
bar is what the extra GiB of bf16 is supposed to buy. `cutoff` is the energy share
|
||||
the set bottoms out at: the bar this campaign actually converted with.
|
||||
"""
|
||||
report = []
|
||||
shares, ranked = {}, []
|
||||
for layer in sorted(data):
|
||||
experts = data[layer]
|
||||
tot_e = sum(v[2] for v in experts.values())
|
||||
tot_r = sum(v[1] for v in experts.values())
|
||||
if tot_e <= 0 or tot_r <= 0:
|
||||
report.append(f"blk.{layer}: empty recording, skipped")
|
||||
continue
|
||||
for e, (_rank, rows, tr) in experts.items():
|
||||
es, rs = tr / tot_e, rows / max(tot_r, 1)
|
||||
if es <= 0:
|
||||
continue
|
||||
shares[(layer, e)] = (es, rs)
|
||||
ranked.append((es, layer, e, rs))
|
||||
ranked.sort(key=lambda r: (-r[0], r[1], r[2]))
|
||||
by_half = {}
|
||||
for es, layer, e, rs in ranked:
|
||||
by_half.setdefault((layer, e // 256), []).append((e, es, rs))
|
||||
|
||||
chosen = {k: v for k, v in shares.items()
|
||||
if v[0] >= min_share and v[0] >= min_route_mult * v[1]}
|
||||
total = len(chosen)
|
||||
total += parity_repair(chosen, by_half, report)
|
||||
|
||||
mult = min_route_mult
|
||||
while True:
|
||||
pool = [(es, l, e, rs) for es, l, e, rs in ranked
|
||||
if es >= mult * rs and (l, e) not in chosen]
|
||||
if len(pool) + total >= target or mult <= 0.0:
|
||||
break
|
||||
mult -= 0.5
|
||||
report.append(f"ratio bar {mult + 0.5:g} leaves {len(pool)} fill candidates "
|
||||
f"for {target - total} slots; relaxing toward 0")
|
||||
for es, layer, e, rs in pool:
|
||||
if total >= target:
|
||||
break
|
||||
if (layer, e) in chosen:
|
||||
continue
|
||||
chosen[(layer, e)] = (es, rs)
|
||||
total += 1
|
||||
report.append(f"blk.{layer}: count fill -> added e{e} (energy {es*100:.2f}%)")
|
||||
total += parity_repair(chosen, by_half, report, only=(layer, e // 256))
|
||||
lines, cutoff = [], None
|
||||
for (layer, e) in sorted(chosen):
|
||||
es, rs = chosen[(layer, e)]
|
||||
cutoff = es if cutoff is None else min(cutoff, es)
|
||||
lines.append(f"blk.{layer}.ffn_*_exps.{e}.weight cast dtype=bf16")
|
||||
return lines, report, total, cutoff
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("calib_dir")
|
||||
ap.add_argument("--min-share", type=float, default=0.03)
|
||||
ap.add_argument("--min-route-mult", type=float, default=3.0)
|
||||
ap.add_argument("--target-count", type=int, default=0)
|
||||
a = ap.parse_args()
|
||||
|
||||
# layer -> global expert -> (rank, rows, energy)
|
||||
data = {}
|
||||
for path in sorted(glob.glob(os.path.join(
|
||||
a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
|
||||
m = NAME_RE.match(os.path.basename(path))
|
||||
if not m:
|
||||
continue
|
||||
rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
|
||||
k, _, rows, tr = read_gram(path)
|
||||
data.setdefault(layer, {})[expert] = (rank, rows, tr)
|
||||
|
||||
if not data:
|
||||
raise SystemExit("no per-expert down grams found")
|
||||
|
||||
if a.target_count > 0:
|
||||
lines, report, total, cutoff = select_to_count(
|
||||
data, a.min_share, a.min_route_mult, a.target_count)
|
||||
bar = f"energy cutoff {cutoff:.3g}" if cutoff is not None else "no candidates"
|
||||
else:
|
||||
lines, report, total = select(data, a.min_share, a.min_route_mult)
|
||||
bar = f"min-share {a.min_share:.3g}"
|
||||
|
||||
# The emitted lines ARE the contract with the four-bit GEMM: every rank's half of
|
||||
# every layer must hold an even number of bf16 experts. Judged on the output and
|
||||
# not on the internal sets, so a slip in either pass is caught here rather than at
|
||||
# engine load time.
|
||||
halves = collections.Counter()
|
||||
for rule in lines:
|
||||
m = CAST_RE.match(rule)
|
||||
if m:
|
||||
halves[(int(m.group(1)), int(m.group(2)) // 256)] += 1
|
||||
odd = {key: n for key, n in halves.items() if n % 2}
|
||||
if odd:
|
||||
print("PARITY LAW BROKEN: " + ", ".join(
|
||||
f"blk.{lay} half {half} holds {n} protected experts"
|
||||
for (lay, half), n in sorted(odd.items())), file=sys.stderr)
|
||||
raise SystemExit(2)
|
||||
|
||||
print("\n".join(lines))
|
||||
print("\n".join(report), file=sys.stderr)
|
||||
print(f"total protected: {total}", file=sys.stderr)
|
||||
print(f"total protected: {total} (target {a.target_count or 'none'}, {bar}, "
|
||||
f"min-route-mult {a.min_route_mult:g})", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -8,8 +8,10 @@
|
||||
# (/opt/halogen/npu/models.txt inside the image) and refuses to load a model
|
||||
# whose files do not match.
|
||||
#
|
||||
# The manifest below IS that catalog block (dumped from the pinned 0.16.2
|
||||
# image on 2026-10-19), so the files land in the layout the entrypoint
|
||||
# The manifest below IS that catalog block (dumped from the pinned 0.18.20
|
||||
# image on 2026-10-30; the 0.16.2 block this replaced named different device
|
||||
# programs — 10 `u*.elf` and a 5,086,496 B consts.hnpb — so a version bump
|
||||
# re-fetches the set or the entrypoint refuses the boot), so the files land in the layout the entrypoint
|
||||
# expects — <dir>/<model-id>/devices/*, <dir>/<model-id>/<model-id>.hnpw and
|
||||
# <dir>/<model-id>/tokenizer/tokenizer.json — and the endpoint can keep
|
||||
# /models read-only and open no outbound connection at boot (HALOGEN_DOWNLOAD
|
||||
@@ -57,21 +59,25 @@ manifest() {
|
||||
case "$1" in
|
||||
qwen3-embedding-0.6b)
|
||||
NPU_REPO="peonist-ai/halogen-npu-qwen3-embedding-0.6b"
|
||||
NPU_REV="4960fd9d4423dd15373ad21c83db48dbdb3b2798"
|
||||
NPU_REV="8fd44c9f18861618a509bea75c51ff8a37e3bd5d"
|
||||
FILES=(
|
||||
"devices/NOTICE.md 3248 0f267b3d2523ddf14997ee58660a492a9eaf10ccf6b08bf19dfdc85b9a478b91"
|
||||
"devices/consts.hnpb 5086496 ae430e41545a0b1012938cda373b0c450364d3be631111cca3a8c43679f93336"
|
||||
"devices/devices.hnpm 24047 4e9bca9d95d084fb89ce5b99f579a00666c8ca1efae2613c53e3a4878cc9f174"
|
||||
"devices/u0.elf 4931968 b38f165886cb037b7d3951d9cb878473eaba58a432acd45416ff85fd88aa4e46"
|
||||
"devices/u1.elf 5456000 b1811feee851b8af833b37721da92099d89772110815bbbe2c3fa1c276f0ff42"
|
||||
"devices/u2.elf 6505552 406ea9a95b470628a7382e376f4a72cdf4d7aa4c59b4ddf0a4bdd139d63d6ff2"
|
||||
"devices/u3.elf 6817216 bc205d51b523d770151a0c61968f7c70bf624412827580bb6fe0aaf4650dc2ee"
|
||||
"devices/u4.elf 7444032 a5a91338a2603acdcd5c11df532033fda2d3f5479ea44ae87f98fafbdff9a62d"
|
||||
"devices/u5.elf 7432416 6c2bf41835569e0eba25ded7c05c7264ae69c6700bdeb4791057d884f2a4c126"
|
||||
"devices/u6.elf 4941232 919d0038ca088eafddf195f3466ff38db27d3635be1cffa5d4d4b5140902a68e"
|
||||
"devices/u7.elf 5488304 d904cf8afae9428dd167960549e18cce5441745f18c62f8c50975dffc077d3c2"
|
||||
"devices/u8.elf 4941232 db3aef69b17825854ee0dd2df475125a3f5c872f95c686793d6af1ee83596fa7"
|
||||
"devices/u9.elf 5488304 d182727e551084db8fb9600441d8aa373c7f09a3d267ae0a5e921ef92326db1c"
|
||||
"devices/consts.hnpb 8495512 5af7c1f2f24d45a9054872bba882461cf7f6ffda832e785b700643d6467ad88b"
|
||||
"devices/devices.hnpm 35465 85d870f6b900de1829659955f0dc701d57984580f5e5ec251add5d6969f03e3d"
|
||||
"devices/u0.elf 2975008 f74df12eb8c79e4e90cbedcc1dd8f142219772d400e3a142fde8a95378b105eb"
|
||||
"devices/u1.elf 3266848 ac5a3f89f934951b315b62ab53e1d0a74dbe6396134192641c405d08da332754"
|
||||
"devices/u2.elf 4015328 5e4f0d3342f262816289866d7e8c966f56a9f938c2e12e7341718cc7312aa480"
|
||||
"devices/u3.elf 4326976 94cf5d269fa286d91c26bf4654cedf101c047d23c4495b74c9c129cdc33b210f"
|
||||
"devices/u4.elf 4953792 cbd2796050d4c675f5cb3da117111087d510173e26f262f1b36aab66dd49dc91"
|
||||
"devices/u5.elf 4940112 7bf6a4470546bb888da02af395c07d92a24fb6d43aef27eb2151b3a9eda19e4d"
|
||||
"devices/u6.elf 6174544 10b9d50b635993e1f9c23775c0b94901d922c1e433cd10910e9f97174235f5ac"
|
||||
"devices/u7.elf 2983776 e0b06063b7379d05c85b3f435b819d96fda1223d8025d6d15958854b5711ace9"
|
||||
"devices/u8.elf 3298656 3bad204f782a68b97e27c1172530b13d7fbb0b495fa95f9504530a1048560644"
|
||||
"devices/u9.elf 2983776 875e313280358ce116e2a51a29a2c42794b5a9fadb3d48db702c90d18d838c2c"
|
||||
"devices/u10.elf 3298656 ef2e0f399f149a5476955bcf233c6fa2e896fcef1f86ecb0d8113c952c41a8b2"
|
||||
"devices/u11.elf 3298656 68a2519eee2d6a61ccc4ca2007e3186d44c9eb4e81162d742fe2f1756ec5a528"
|
||||
"devices/u12.elf 3298656 2bdc6abfa58cac1d5795a3ef52fe8a9ab4e7c2a4792822d615499a913401f728"
|
||||
"devices/u13.elf 3298656 4a3ce14f7ba15987b9bcbb4a010083e8aef68cd87c20cbe43403a98263179fe2"
|
||||
"qwen3-embedding-0.6b.hnpw 813032248 b9f9901429df7b46ec17153aa3899f738adbea8e4356080da0c6e6586fb750d1"
|
||||
"tokenizer/tokenizer.json 11423705 def76fb086971c7867b829c23a26261e38d9d74e02139253b38aeb9df8b4b50a"
|
||||
) ;;
|
||||
@@ -99,7 +105,18 @@ for MODEL in ${MODELS//,/ }; do
|
||||
# 62 GiB fetch hit this on 2026-10-18).
|
||||
url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$HF/$NPU_REPO/resolve/$NPU_REV/$f") \
|
||||
|| { echo "cannot resolve $NPU_REPO/$f" >&2; exit 1; }
|
||||
curl -fL --retry 5 -C - -o "$path" "$url"
|
||||
# Resume ONLY into a genuinely partial file. A staged file of a
|
||||
# DIFFERENT size is an artifact of another revision (a version
|
||||
# bump re-cuts the device programs), and `-C -` against it
|
||||
# appends the tail to the old bytes: the result is the right
|
||||
# BYTE COUNT with the wrong content, which reads as a corrupt
|
||||
# program, not as a fetch fault (measured 2026-10-30 on
|
||||
# consts.hnpb). Fetch those from zero.
|
||||
resume=()
|
||||
if [ -f "$path" ] && [ "$(stat -c %s "$path")" -lt "$want" ]; then
|
||||
resume=(-C -)
|
||||
fi
|
||||
curl -fL --retry 5 "${resume[@]+"${resume[@]}"}" -o "$path" "$url"
|
||||
fi
|
||||
fi
|
||||
[ -f "$path" ] || { echo "MISSING $path (run without --verify-only)" >&2; exit 1; }
|
||||
|
||||
Executable
+249
@@ -0,0 +1,249 @@
|
||||
#!/usr/bin/env bash
|
||||
# download-qwen38-bf16.sh — RUNS ON DANKSERVER. Fetch and verify the vendor
|
||||
# BF16 checkpoint Qwen/Qwen3.8-Flash-Next: the source the five-bit campaign
|
||||
# converts, and the source the KLD reference is recorded from.
|
||||
#
|
||||
# Why the base and not a finetune: the campaign converts to compare against the
|
||||
# publisher's own iq4r container, and that container was built from this
|
||||
# checkpoint. Base-in, base-out; anything else confounds the gate. (The
|
||||
# predecessor campaign ran ukisai/Swift-1.5-Qwen3.8-Flash-Next-BF16, whose key
|
||||
# set is identical to this repo's but whose weights are not — none of its
|
||||
# calibration grams, KLD reference or protected list carries over.)
|
||||
#
|
||||
# Why re-fetch instead of reusing: every BF16 copy of this family was destroyed
|
||||
# in the 2026-10-15 storage purge — only the 113.6 GiB served .rad survived.
|
||||
# 335.3 GiB of safetensors, 131 shards, ungated (the predecessor campaign
|
||||
# verified the key set against this repo, so no gate letter is needed).
|
||||
#
|
||||
# Verification is four gates deep, because the failure this repo actually lived
|
||||
# through is a fetch that reported success: `curl -o FILE URL` exits 0 on an
|
||||
# HTTP 401/403 and writes the error body as the artifact — three shards of
|
||||
# identical tiny size and a cheerful DONE in 20 seconds for 104 GiB.
|
||||
#
|
||||
# 1. shard census: the file count must equal the expected shard count, so a
|
||||
# partially fetched fragment tree (the .partNofM trap) cannot pass.
|
||||
# 2. per-shard completeness, from the shard's OWN safetensors header: the
|
||||
# header declares where the tensor data ends, so every shard's true size is
|
||||
# known offline. A torn tail is named here, by file, instead of surfacing
|
||||
# as a load error five hours later.
|
||||
# 3. index coverage: every shard model.safetensors.index.json references
|
||||
# exists and is named.
|
||||
# 4. total bytes against the number the Hub declared, then, unless --no-hub,
|
||||
# every file's sha256 against the Hub's LFS record — including the shards
|
||||
# already on disk, so a previous partial download is never trusted.
|
||||
#
|
||||
# The token never enters a command line (the 2026-09-16 leak law): it is read
|
||||
# into HF_TOKEN in the environment, where a local process listing is the only
|
||||
# reader. This repo is ungated, so the fetch simply runs without one when no
|
||||
# token file exists. scripts/hfget.sh is curl+jq only — no Python, no venv, no
|
||||
# hf_transfer — and it drops the pickle/ONNX weight duplicates by itself.
|
||||
#
|
||||
# usage: download-qwen38-bf16.sh [--fetch] [--no-hub] [--dir DIR]
|
||||
# [--shards N] [--bytes B] [--repo ID]
|
||||
# --fetch download first (resumable: re-run to continue), then verify
|
||||
# --no-hub skip the Hub-side sha256 sweep; the other gates always run
|
||||
# exit: 0 verified, 1 verification failed, 2 environment / usage error
|
||||
set -euo pipefail
|
||||
|
||||
REPO="${FRANK_QWEN38_HF_REPO:-Qwen/Qwen3.8-Flash-Next}"
|
||||
LOCAL_DIR="${FRANK_QWEN38_CKPT:-/home/dank/models/Qwen3.8-Flash-Next}"
|
||||
# Hub census of 2026-10-27: 144 files, 131 of them .safetensors, together
|
||||
# 360,023,351,514 bytes; the shards alone are 360,000,192,888. This gate counts
|
||||
# shards only, so a config file edited on the Hub cannot fail it.
|
||||
EXPECTED_SHARDS="${FRANK_QWEN38_SHARDS:-131}"
|
||||
EXPECTED_BYTES="${FRANK_QWEN38_BYTES:-360000192888}"
|
||||
MIN_FREE_MIB="${FRANK_QWEN38_MIN_FREE_MIB:-350000}"
|
||||
|
||||
FETCH=0
|
||||
HUB=1
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--fetch) FETCH=1 ;;
|
||||
--no-hub) HUB=0 ;;
|
||||
--dir) LOCAL_DIR="$2"; shift ;;
|
||||
--shards) EXPECTED_SHARDS="$2"; shift ;;
|
||||
--bytes) EXPECTED_BYTES="$2"; shift ;;
|
||||
--repo) REPO="$2"; shift ;;
|
||||
*) echo "usage: ${0##*/} [--fetch] [--no-hub] [--dir DIR] [--shards N] [--bytes B] [--repo ID]" >&2
|
||||
exit 2 ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
FAILS=0
|
||||
note() { echo "$@"; FAILS=$((FAILS + 1)); }
|
||||
|
||||
if [ "$FETCH" = "1" ]; then
|
||||
# Every fetch gate is a hard stop: skipping the download silently and then
|
||||
# verifying a half tree is how a campaign ships a stale payload.
|
||||
HFGET="$SELF_DIR/hfget.sh"
|
||||
if [ ! -f "$HFGET" ]; then
|
||||
echo "REFUSED: the fetch helper is not here: $HFGET" >&2
|
||||
exit 2
|
||||
fi
|
||||
if ! command -v jq >/dev/null 2>&1; then
|
||||
echo "REFUSED: hfget.sh needs jq on PATH" >&2
|
||||
exit 2
|
||||
fi
|
||||
mkdir -p "$LOCAL_DIR"
|
||||
# The campaign's peak-disk law starts at the fetch: the checkpoint and the
|
||||
# reference container built from it are ~335 GiB each, and a tree that does
|
||||
# not fit is discovered hours into a transfer, when it costs the most.
|
||||
free_mib=$(df -m --output=avail "$LOCAL_DIR" | tail -1 | tr -dc '0-9')
|
||||
if [ "${free_mib:-0}" -lt "$MIN_FREE_MIB" ]; then
|
||||
echo "REFUSED: $free_mib MiB free at $LOCAL_DIR; the fetch wants $MIN_FREE_MIB MiB free" >&2
|
||||
echo " (retire the peak-disk predecessors first — nothing is downloaded, so" >&2
|
||||
echo " nothing is left half-written)" >&2
|
||||
exit 2
|
||||
fi
|
||||
echo "fetch: $REPO -> $LOCAL_DIR ($free_mib MiB free)"
|
||||
if [ -z "${HF_TOKEN:-}" ]; then
|
||||
CREDSTORE="$HOME/.config/huggingface/credentials/https://huggingface.co"
|
||||
if [ -f "$CREDSTORE" ]; then
|
||||
HF_TOKEN="$(awk '/^ OauthToken: / {print $2; exit}' "$CREDSTORE")"
|
||||
elif [ -f "$HOME/.cache/huggingface/token" ]; then
|
||||
# The legacy single-line token file the hub library writes.
|
||||
HF_TOKEN="$(tr -d '[:space:]' < "$HOME/.cache/huggingface/token")"
|
||||
elif [ -f "$HOME/.hf_token" ]; then
|
||||
HF_TOKEN="$(tr -d '[:space:]' < "$HOME/.hf_token")"
|
||||
fi
|
||||
fi
|
||||
if [ -n "${HF_TOKEN:-}" ]; then
|
||||
export HF_TOKEN
|
||||
fi
|
||||
bash "$HFGET" "$REPO" "$LOCAL_DIR" main
|
||||
unset HF_TOKEN
|
||||
fi
|
||||
|
||||
if [ ! -d "$LOCAL_DIR" ]; then
|
||||
echo "REFUSED: $LOCAL_DIR does not exist (run with --fetch, or pass --dir)" >&2
|
||||
exit 1
|
||||
fi
|
||||
INDEX="$LOCAL_DIR/model.safetensors.index.json"
|
||||
if [ ! -f "$INDEX" ]; then
|
||||
echo "REFUSED: $INDEX is missing — the checkpoint tree is not addressable without it" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "== gate 1: shard census ($LOCAL_DIR)"
|
||||
SHARDS=$(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' | wc -l)
|
||||
if [ "$SHARDS" != "$EXPECTED_SHARDS" ]; then
|
||||
note "FAIL SHARD_COUNT: $SHARDS safetensors files on disk, expected $EXPECTED_SHARDS"
|
||||
else
|
||||
echo " ok: $SHARDS shards"
|
||||
fi
|
||||
|
||||
echo "== gate 2: every shard is whole, judged by its own header"
|
||||
# A safetensors file is [u64 header length][JSON header][tensor data]. The JSON
|
||||
# declares where the data ends, so the file's true size is computable offline:
|
||||
# 8 + header + the largest end offset. Truncation and error bodies both fall
|
||||
# here, and each is reported with the file that caused it.
|
||||
while IFS= read -r f; do
|
||||
state=$(python3 - "$f" <<'PY'
|
||||
import json, os, struct, sys
|
||||
path = sys.argv[1]
|
||||
size = os.path.getsize(path)
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
(hlen,) = struct.unpack("<Q", fh.read(8))
|
||||
if hlen == 0 or hlen > size:
|
||||
print("NOT_SAFETENSORS")
|
||||
raise SystemExit
|
||||
header = json.loads(fh.read(hlen))
|
||||
except Exception:
|
||||
print("NOT_SAFETENSORS")
|
||||
raise SystemExit
|
||||
if not header:
|
||||
print("NOT_SAFETENSORS")
|
||||
raise SystemExit
|
||||
end = max((v["data_offsets"][1] for v in header.values()
|
||||
if isinstance(v, dict) and "data_offsets" in v), default=0)
|
||||
want = 8 + hlen + end
|
||||
print("OK" if want == size else f"TRUNCATED want={want} have={size}")
|
||||
PY
|
||||
)
|
||||
case "$state" in
|
||||
OK) ;;
|
||||
NOT_SAFETENSORS*) note "FAIL NOT_SAFETENSORS: $(basename "$f") is not a safetensors file (an error body written as an artifact?)" ;;
|
||||
TRUNCATED*) note "FAIL TRUNCATED: $(basename "$f") $state" ;;
|
||||
esac
|
||||
done < <(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' | sort)
|
||||
if [ "$FAILS" -eq 0 ]; then
|
||||
echo " ok: every shard's data extent matches its size on disk"
|
||||
fi
|
||||
|
||||
echo "== gate 3: the index's shards are all present"
|
||||
INDEX_SHARDS=$(python3 - "$INDEX" <<'PY'
|
||||
import json, sys
|
||||
map_ = json.load(open(sys.argv[1]))["weight_map"]
|
||||
print("\n".join(sorted(set(map_.values()))))
|
||||
PY
|
||||
)
|
||||
while IFS= read -r name; do
|
||||
[ -n "$name" ] || continue
|
||||
if [ ! -f "$LOCAL_DIR/$name" ]; then
|
||||
note "FAIL MISSING: $name is named by the index and is not on disk"
|
||||
fi
|
||||
done <<< "$INDEX_SHARDS"
|
||||
if [ "$FAILS" -eq 0 ]; then
|
||||
echo " ok: $(printf '%s\n' "$INDEX_SHARDS" | grep -c .) referenced shards on disk"
|
||||
fi
|
||||
|
||||
echo "== gate 4: total bytes vs the Hub's declaration"
|
||||
TOTAL=$(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' -printf '%s\n' | awk '{s+=$1} END{print s+0}')
|
||||
if [ "$TOTAL" != "$EXPECTED_BYTES" ]; then
|
||||
note "FAIL TOTAL_BYTES: shards total $TOTAL bytes, expected $EXPECTED_BYTES"
|
||||
else
|
||||
echo " ok: $TOTAL bytes"
|
||||
fi
|
||||
|
||||
if [ "$HUB" = "1" ]; then
|
||||
echo "== gate 5: every file's sha256 vs the Hub's LFS record"
|
||||
# size = file size; lfs.sha256 = sha256 of the file itself (LFS or not).
|
||||
TMPD="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMPD"' EXIT
|
||||
MANIFEST="$TMPD/tree.json"
|
||||
PLAN="$TMPD/plan"
|
||||
curl -sf "https://huggingface.co/api/models/$REPO/tree/main?recursive=true&expand=true" \
|
||||
-o "$MANIFEST" || { echo "FAIL: the Hub API call failed — refusing to report success" >&2; exit 1; }
|
||||
python3 - "$MANIFEST" > "$PLAN" <<'PY'
|
||||
import json, sys
|
||||
for e in json.load(open(sys.argv[1])):
|
||||
if e.get("type") != "file":
|
||||
continue
|
||||
sha = (e.get("lfs") or {}).get("sha256") or e.get("sha") or ""
|
||||
if sha:
|
||||
print(e["path"], e["size"], sha)
|
||||
PY
|
||||
BAD=0
|
||||
while read -r rel size sha; do
|
||||
[ -n "${rel:-}" ] || continue
|
||||
if [ ! -f "$LOCAL_DIR/$rel" ]; then
|
||||
note "FAIL MISSING: $rel is on the Hub and not on disk"
|
||||
continue
|
||||
fi
|
||||
have=$(stat -c %s "$LOCAL_DIR/$rel")
|
||||
if [ "$have" != "$size" ]; then
|
||||
note "FAIL SIZE: $rel is $have bytes, Hub says $size"
|
||||
continue
|
||||
fi
|
||||
got=$(sha256sum "$LOCAL_DIR/$rel" | awk '{print $1}')
|
||||
if [ "$got" != "$sha" ]; then
|
||||
note "FAIL SHA256: $rel got $got want $sha"
|
||||
BAD=$((BAD + 1))
|
||||
fi
|
||||
done < "$PLAN"
|
||||
if [ "$BAD" -eq 0 ] && [ "$FAILS" -eq 0 ]; then
|
||||
echo " ok: sha256 matches for every file"
|
||||
fi
|
||||
rm -rf "$TMPD"
|
||||
trap - EXIT
|
||||
fi
|
||||
|
||||
if [ "$FAILS" -gt 0 ]; then
|
||||
echo "VERIFY_FAILED: $FAILS problem(s) in $LOCAL_DIR" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "VERIFY_OK: $SHARDS shards, $TOTAL bytes of safetensors in $LOCAL_DIR"
|
||||
echo "NEXT: build the BF16 reference container with scripts/build-qwen38-bf16-rad.sh"
|
||||
@@ -74,6 +74,13 @@ FRANK_ARCHON_PLACEMENT=expert_tiered
|
||||
# residency -- ~3.2 GiB a card of experts now lives in host RAM and is read over the
|
||||
# x8 link when the router picks it. The ledger's coverage line is the referee, not
|
||||
# this file: if vram_kv comes back under 700,000 tokens, raise this 1024 and redeploy.
|
||||
# 2026-10-29 A/B, measured on both payloads and folded back the same day: at 66,560 the
|
||||
# endpoint printed the identical `elastic 26.73 GiB / experts 20.72 / kv cache 5.64 /
|
||||
# vram_weights 24.30` it had printed at 49,152, opening the bigger pool with `0 B used`;
|
||||
# the campaign rig (bf16, same step) moved `vram_kv 269.99 -> 271.25 MiB`, a rounding
|
||||
# step — its window is fixed by its own n-gram table (95.37 GiB) and expert plane, not
|
||||
# by this pool, so the refused calibration documents were fixed by cutting them to the
|
||||
# window (scripts/rechunk-kld-corpus.py), not by widening the pool.
|
||||
FRANK_ARCHON_HOST_POOL_MIB=49152
|
||||
# VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle
|
||||
# floor is built from it.
|
||||
@@ -137,7 +144,8 @@ FRANK_ARCHON_NGRAM_PLACEMENT=ram
|
||||
# itself before refusing by name, so recovery is a boot, not a visit.
|
||||
# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT + the
|
||||
# auto checkpoint store (MAX_SEQS x 10 slots of 111.98 MiB), per the bats ledger test;
|
||||
# at the 49,152 pool with 4 slots that computes to 161,848 MiB.
|
||||
# at the 49,152 pool with 4 slots that computes to 161,848 MiB (196,664 at the
|
||||
# 66,560 pool the 2026-10-29 A/B rejected).
|
||||
FRANK_ARCHON_TTM_MIN_MIB=162000
|
||||
FRANK_ARCHON_TTM_TARGET_MIB=204800
|
||||
# Escape hatch for experiments. Unset -> no extra argv; the serve script expands
|
||||
|
||||
@@ -10,8 +10,11 @@
|
||||
# The .hgn format is the lock: it loads ONLY in the flash-server line, so the
|
||||
# engine moves with the payload, never ahead of it.
|
||||
# Image digest-pinned (a tag pull silently re-points; the digest is what was
|
||||
# qualified — 0.16.2 is the version the model card names as its quickstart pin).
|
||||
FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
|
||||
# qualified). 0.18.20 since 2026-10-30 (was 0.16.2, the model card's quickstart
|
||||
# pin) — the vendor has moved two minor versions and the payload lock is what
|
||||
# the deploy's health gate plus the boot ledger decide; rollback is the
|
||||
# previous digest, 0c61bf84…, on this same line.
|
||||
FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0
|
||||
# Container identity kept from the 27B era: the unit's ExecStop names it.
|
||||
FRANK_DRAGOON_CONTAINER=frank-dragoon-halogen
|
||||
# Predecessor identity the serve script clears if it is still holding the GPUs.
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) on the RTX 3090,
|
||||
# :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file, plus
|
||||
# frank-zealot — llama.cpp (PrismML fork build 12118 = 0.7.0-dev, commit
|
||||
# 024474448, CUDA) on the RTX 3090, :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file, plus
|
||||
# the refusal-direction adapter applied at load.
|
||||
#
|
||||
# ONE OWNER: shape, payload identity, eras, laws and every measured number
|
||||
# live in serverAppConfigs-docs/LLM/frank-zealot.md. This file carries the
|
||||
# knobs and only the reason a reader editing THIS line cannot recover
|
||||
# elsewhere. Where the two disagree, this file wins on values.
|
||||
LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server
|
||||
LLAMA_BIN=/home/dank/GitPackages/llama-prism-0244744/bin/llama-server
|
||||
# The fork is a LOCK, not a preference: custom tensor type 142 + prism.hadamard.*
|
||||
# metadata. Stock llama.cpp refuses the file; a loader ignoring the Hadamard
|
||||
# metadata dequantizes wrong, silently.
|
||||
@@ -64,7 +64,8 @@ CACHE_TYPE_V=q8_0
|
||||
# weights, so there is no --spec-draft-model. Its bill is pool × 4,096 B of
|
||||
# draft KV plus ~0.45 GiB of buffers and a tripled recurrent state — paid for
|
||||
# by CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024 above. Build gate: quantized V
|
||||
# beside MTP needs fork build >= b10776; LLAMA_BIN is 10777.
|
||||
# beside MTP needs fork build >= b10776; LLAMA_BIN is 12118 (b10777 is kept on
|
||||
# disk as the rollback path).
|
||||
# SPEC_TYPE=none is the rollback and strips the WHOLE set (--spec-type and
|
||||
# every --spec-draft-*); half a set has segfaulted at load.
|
||||
SPEC_TYPE=draft-mtp
|
||||
|
||||
Executable
+418
@@ -0,0 +1,418 @@
|
||||
#!/usr/bin/env python3
|
||||
"""grams-to-imatrix.py — the engine's own calibration recording, re-voiced as a
|
||||
llama.cpp importance matrix for `rad-convert --imatrix F`. RUNS ON DANKSERVER
|
||||
beside the recording. Python 3 stdlib only: numpy must NOT be required, because
|
||||
this runs in the plain container next to a 335 GiB checkpoint, not in a venv.
|
||||
|
||||
WHY. `rad-convert` takes its per-column activation second moments and its
|
||||
expert-popularity profile from ONE input, an imatrix (core/format/imatrix.h).
|
||||
Only the GGUF form carries a per-expert dimension — with the legacy flat form,
|
||||
or with none, every expert in a layer is weighted by the layer's aggregate,
|
||||
which is exactly the wrong thing on a 512-expert MoE. The calibration tap has
|
||||
already measured everything; this tool only re-voices it, so the imatrix and the
|
||||
protected-expert census (derive-protected-experts.py) are the SAME recording:
|
||||
one measurement, two consumers, neither able to disagree about which experts are
|
||||
hot.
|
||||
|
||||
INPUT. RADGRAM2 files in a RADIANCE_CALIB_DIR output directory
|
||||
(arch/common/rad_arch.h, calib_write). Header law, same parse as
|
||||
scripts/derive-protected-experts.py — 32 bytes, `<8sIIQI`: magic `RADGRAM2`,
|
||||
u32 k, u32 rank, u64 rows, u32 domain, a pad word, then k*k f32 row-major. Two
|
||||
kinds of name, both GGUF-style, both with the checkpoint's GLOBAL expert
|
||||
numbering (rad_block_moe_fp8.h: ge = expert_base + e):
|
||||
|
||||
gram.r<rank>.blk.L.ffn_down_exps.bin the layer's pooled gram
|
||||
gram.r<rank>.blk.L.ffn_down_exps.<e>.bin expert e's own gram
|
||||
|
||||
`rows` is that file's routing/token count and the matrix is Σ x xᵀ over those
|
||||
rows, so **the diagonal is exactly the per-column activation second moment an
|
||||
imatrix wants**, and `rows` is the count the reader hands back from `count()`.
|
||||
Every off-diagonal term is dropped on purpose: the GGUF format has nowhere to
|
||||
put a full matrix, and the quantiser asks for columns.
|
||||
|
||||
OUTPUT, per layer L (core/format/imatrix.h: names WITHOUT `.weight`, a MoE
|
||||
layer's experts folded into ONE entry, so expert 9 is slice 9 of
|
||||
`blk.L.ffn_down_exps`):
|
||||
|
||||
blk.L.ffn_down_exps in_sum2 [k, n_expert] expert-major, counts [1, n_expert]
|
||||
blk.L.ffn_gate_up_exps the SAME matrices at the SAME offsets. A routed expert's
|
||||
gate_up input is the same hidden-state rows as its down
|
||||
input, so the second moment over them is identical and
|
||||
so is the routing count. Two tensor infos over one
|
||||
payload: the reader maps each half independently
|
||||
(imatrix.cpp joins `<base>.in_sum2` with `<base>.counts`
|
||||
by base name), and copying ~200 MB to state one fact
|
||||
twice buys no correctness.
|
||||
blk.L.ssm_inz [k] and [1], from the pooled gram: that row's input is
|
||||
the same hidden state, so its columns take the pooled
|
||||
diagonal and its count is the layer's pooled rows.
|
||||
|
||||
Then STOP. Nothing whose input width is not the gram's k — no ssm_out, no
|
||||
attn_output, no mtp.*, no ple_ngram. importance_for() answers false when the
|
||||
entry's cols do not equal the weight's width and the convert proceeds without
|
||||
it: an entry the reader silently ignores is a lie in the artifact, worse than
|
||||
an absent one. The corollary to check before trusting the two hidden-width
|
||||
entries: the tap writes the gate_up gram at g.n_embd and the down gram at the
|
||||
expert width, so the gram's k must equal the checkpoint's hidden width (its
|
||||
config.json). Where they differ, gate_up/ssm_inz are inert, not wrong.
|
||||
|
||||
INHERITANCE. An expert with no own gram carries the layer's POOLED diagonal, and
|
||||
its count stays its own routing rows — the rule the engine itself applies when it
|
||||
picks a Hessian (libquant/lq_hess.h, calib_resolve: an expert's own gram only
|
||||
once its rank files hold kOwnRowsPerK * K rows; below that a fit to so few rows
|
||||
is worse than round-to-nearest, and those experts carry a fraction of a percent
|
||||
of the routing). The count never inherits: popularity is `count`'s job. The count
|
||||
of inherited experts is reported on stderr.
|
||||
|
||||
RANK FOLDING. TP=2 writes gram.r0.* and gram.r1.* with GLOBAL expert numbering,
|
||||
so the ranks describe DISJOINT experts and the fold is a UNION into the one
|
||||
[k, n_expert] entry. A pooled gram is the opposite case: every rank summed the
|
||||
same stem over its own rows, so the fold is a SUM, rows included (calib_sum,
|
||||
calib_rows). The same global expert in two per-expert files is legal only when
|
||||
the numbers agree exactly; otherwise the recording is corrupt and the entry would
|
||||
be whichever file sorted last, so refuse.
|
||||
|
||||
GGUF v3 (the only version imatrix.cpp accepts — version != 3 is a refusal):
|
||||
`general.alignment` u32, `imatrix.chunk_size` u32, `imatrix.chunk_count` u32,
|
||||
`imatrix.datasets` array of strings; tensors f32 (ggml type 0) with offsets
|
||||
relative to the aligned data section. Every tensor offset is padded up to
|
||||
`general.alignment` and the data section itself starts aligned. That is the
|
||||
classic hand-rolled-writer bug: an unaligned offset makes the reader's extent
|
||||
arithmetic point into the middle of the previous tensor's floats, and it reads
|
||||
them without complaint.
|
||||
|
||||
chunk_count DERIVATION. The corpus row count is the LARGEST single-layer pooled
|
||||
row total, not a sum over layers: every layer's tap accumulated over the whole
|
||||
corpus, so adding the layers counts each token once per layer and inflates the
|
||||
figure by the layer count. chunk_count = that row total // chunk_size, the
|
||||
"chunks of chunk_size" line imatrix.cpp prints for itself.
|
||||
|
||||
REFUSALS, each exit 2 with a message naming what is missing:
|
||||
NO_GRAMS the directory holds no RADGRAM2 down-projection gram
|
||||
NOT_RADGRAM2 a gram whose magic is not RADGRAM2
|
||||
SHORT a gram whose length is not 32 + k*k*4
|
||||
NO_POOLED a layer with expert grams and no pooled gram to inherit from
|
||||
K_MISMATCH a layer whose files disagree about k (rank pooled grams, or an
|
||||
expert gram against its layer's pooled gram)
|
||||
EXPERT_CONFLICT a global expert recorded twice with different numbers
|
||||
EXISTS the -o path already holds a file (the trash law: move it)
|
||||
CHUNK_COUNT the derived chunk count does not fit the u32 metadata field
|
||||
|
||||
USAGE. grams-to-imatrix.py <calib-dir> -o <out.gguf>
|
||||
[--datasets NAME,...] [--chunk-size N] [--min-rows N] [-v]
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import struct
|
||||
import sys
|
||||
|
||||
MAGIC = b"RADGRAM2"
|
||||
HDR = 32 # 8s I I Q I + pad, rad_arch.h
|
||||
KOWN_ROWS_PER_K = 4 # libquant/lq_hess.h kOwnRowsPerK
|
||||
ALIGNMENT = 32 # general.alignment, and the pad unit
|
||||
|
||||
POOLED_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.bin$")
|
||||
EXPERT_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
|
||||
|
||||
GT_U32, GT_STR, GT_ARR = 4, 8, 9
|
||||
GGML_F32 = 0
|
||||
|
||||
|
||||
def refuse(msg):
|
||||
sys.stderr.write("REFUSED %s\n" % msg)
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
def read_gram(path):
|
||||
"""(k, rank, rows, domain, diagonal) from one RADGRAM2 file."""
|
||||
try:
|
||||
with open(path, "rb") as f:
|
||||
raw = f.read()
|
||||
except OSError as e:
|
||||
refuse("UNREADABLE %s: %s" % (path, e))
|
||||
if len(raw) < HDR:
|
||||
refuse("SHORT %s: %d bytes, under the %d-byte RADGRAM2 header"
|
||||
% (path, len(raw), HDR))
|
||||
magic, k, rank, rows, domain = struct.unpack_from("<8sIIQI", raw, 0)
|
||||
if magic != MAGIC:
|
||||
refuse("NOT_RADGRAM2 %s: magic %r, expected b'RADGRAM2'" % (path, magic))
|
||||
if k <= 0:
|
||||
refuse("SHORT %s: header claims k=%d, which describes no matrix" % (path, k))
|
||||
need = HDR + k * k * 4
|
||||
if len(raw) != need:
|
||||
refuse("SHORT %s: %d bytes, expected %d (32 + k*k*4 for k=%d)"
|
||||
% (path, len(raw), need, k))
|
||||
# The diagonal only, read by stride out of the row-major matrix.
|
||||
diag = [struct.unpack_from("<f", raw, HDR + (i * k + i) * 4)[0]
|
||||
for i in range(k)]
|
||||
return k, rank, rows, domain, diag
|
||||
|
||||
|
||||
def load_dir(d, verbose):
|
||||
"""{layer: {"pooled": gram|None, "experts": {global e: gram}}} + rotated count.
|
||||
|
||||
Files outside the ffn_down_exps stem are not ours to fold: the tap names a
|
||||
pooled gram after every projection it taps, and those carry other widths."""
|
||||
if not os.path.isdir(d):
|
||||
refuse("NO_GRAMS %s: not a directory" % d)
|
||||
layers, rotated = {}, 0
|
||||
for name in sorted(os.listdir(d)):
|
||||
m = POOLED_RE.match(name) or EXPERT_RE.match(name)
|
||||
if not m:
|
||||
continue
|
||||
path = os.path.join(d, name)
|
||||
if not os.path.isfile(path):
|
||||
continue
|
||||
pooled = bool(POOLED_RE.match(name))
|
||||
layer = int(m.group(2))
|
||||
k, rank, rows, domain, diag = read_gram(path)
|
||||
if domain != 0:
|
||||
rotated += 1
|
||||
L = layers.setdefault(layer, {"pooled": None, "experts": {}})
|
||||
if pooled:
|
||||
P = L["pooled"]
|
||||
if P is None:
|
||||
L["pooled"] = {"k": k, "rows": rows, "diag": diag, "paths": [path]}
|
||||
elif P["k"] != k:
|
||||
refuse("K_MISMATCH layer %d: pooled grams %s (k=%d) and %s (k=%d)"
|
||||
" disagree about the width"
|
||||
% (layer, P["paths"][0], P["k"], path, k))
|
||||
else:
|
||||
# calib_sum: each rank summed the same stem over its own rows.
|
||||
P["rows"] += rows
|
||||
P["paths"].append(path)
|
||||
for i in range(k):
|
||||
P["diag"][i] += diag[i]
|
||||
if verbose:
|
||||
sys.stderr.write("pooled layer %d: %s k=%d rows=%d\n"
|
||||
% (layer, name, k, rows))
|
||||
continue
|
||||
e = int(m.group(3))
|
||||
own = L["experts"].get(e)
|
||||
if own is None:
|
||||
L["experts"][e] = {"k": k, "rows": rows, "diag": diag,
|
||||
"path": path, "ranks": [rank]}
|
||||
if verbose:
|
||||
sys.stderr.write("expert layer %d r%d e%d: rows=%d\n"
|
||||
% (layer, rank, e, rows))
|
||||
continue
|
||||
if own["rows"] != rows or own["diag"] != diag:
|
||||
refuse("EXPERT_CONFLICT layer %d expert %d: %s (rows=%d) and %s"
|
||||
" (rows=%d) record the same global expert with different"
|
||||
" numbers" % (layer, e, own["path"], own["rows"], path, rows))
|
||||
own["ranks"].append(rank)
|
||||
sys.stderr.write("note: layer %d expert %d recorded identically by rank %s"
|
||||
" and %s; folded once\n"
|
||||
% (layer, e, own["path"], name))
|
||||
if not layers:
|
||||
refuse("NO_GRAMS %s: no RADGRAM2 gram.r<rank>.blk.<L>.ffn_down_exps"
|
||||
"[.<expert>].bin files in the directory" % d)
|
||||
return layers, rotated
|
||||
|
||||
|
||||
def build_entries(layers, min_rows, verbose):
|
||||
"""([ (name, dims, payload|None, key) ], census, layers emitted)."""
|
||||
entries = []
|
||||
census = {"inherited": 0, "experts": 0, "routing": 0, "rows": 0, "layers": 0}
|
||||
for layer in sorted(layers):
|
||||
P, E = layers[layer]["pooled"], layers[layer]["experts"]
|
||||
if E and P is None:
|
||||
refuse("NO_POOLED layer %d: %d expert gram(s) but no pooled gram (an"
|
||||
" expected gram.r<rank>.blk.%d.ffn_down_exps.bin), so an expert"
|
||||
" under the row threshold would have nothing to inherit"
|
||||
% (layer, len(E), layer))
|
||||
if P is None:
|
||||
continue
|
||||
k = P["k"]
|
||||
census["layers"] += 1
|
||||
census["rows"] = max(census["rows"], P["rows"])
|
||||
dn = "blk.%d.ffn_down_exps" % layer
|
||||
# A staging key names ONE payload. A gate_up entry reuses the down
|
||||
# layer's bytes by carrying the same key with no payload of its own;
|
||||
# in_sum2 and counts are different bytes and must never share a key.
|
||||
ksum, kcnt = (layer, "sum2"), (layer, "counts")
|
||||
kpgm, kpcnt = (layer, "pooled_sum2"), (layer, "pooled_counts")
|
||||
# One entry holds one width: an expert gram recorded at another k would
|
||||
# inherit (or be inherited by) a diagonal of the wrong length, and the
|
||||
# reader would then answer `cols` from whichever file set dims[0].
|
||||
for e in sorted(E):
|
||||
if E[e]["k"] != k:
|
||||
refuse("K_MISMATCH layer %d expert %d: %s holds k=%d and this"
|
||||
" layer's pooled gram is k=%d" % (layer, e, E[e]["path"],
|
||||
E[e]["k"], k))
|
||||
|
||||
if not E:
|
||||
# No per-expert recording for this layer: emit the dense form
|
||||
# ([k], [1]) rather than 512 identical slices, which would claim a
|
||||
# per-expert measurement that was never taken.
|
||||
entries.append((dn + ".in_sum2", [k],
|
||||
struct.pack("<%df" % k, *P["diag"]), ksum))
|
||||
entries.append((dn + ".counts", [1],
|
||||
struct.pack("<f", float(P["rows"])), kcnt))
|
||||
sys.stderr.write("note: layer %d has no per-expert grams; %s is"
|
||||
" dense over the pooled gram\n" % (layer, dn))
|
||||
else:
|
||||
floor = min_rows if min_rows > 0 else KOWN_ROWS_PER_K * k
|
||||
n_expert = max(E) + 1
|
||||
sum2, counts = bytearray(), bytearray()
|
||||
for e in range(n_expert):
|
||||
own = E.get(e)
|
||||
if own is not None and own["rows"] >= floor:
|
||||
src = own["diag"]
|
||||
rows = own["rows"]
|
||||
else:
|
||||
src = P["diag"]
|
||||
rows = own["rows"] if own is not None else 0
|
||||
census["inherited"] += 1
|
||||
sum2 += struct.pack("<%df" % k, *src)
|
||||
counts += struct.pack("<f", float(rows))
|
||||
census["experts"] += 1
|
||||
census["routing"] += rows
|
||||
if min(E) != 0:
|
||||
sys.stderr.write("note: layer %d has no gram for experts 0..%d;"
|
||||
" those slices carry the pooled gram at count 0\n"
|
||||
% (layer, min(E) - 1))
|
||||
entries.append((dn + ".in_sum2", [k, n_expert], bytes(sum2), ksum))
|
||||
entries.append((dn + ".counts", [1, n_expert], bytes(counts), kcnt))
|
||||
gu = "blk.%d.ffn_gate_up_exps" % layer
|
||||
entries.append((gu + ".in_sum2", [k, n_expert], None, ksum))
|
||||
entries.append((gu + ".counts", [1, n_expert], None, kcnt))
|
||||
if verbose:
|
||||
sys.stderr.write("layer %d: k=%d n_expert=%d pooled_rows=%d"
|
||||
" own-gram floor=%d\n"
|
||||
% (layer, k, n_expert, P["rows"], floor))
|
||||
|
||||
inz = "blk.%d.ssm_inz" % layer
|
||||
entries.append((inz + ".in_sum2", [k],
|
||||
struct.pack("<%df" % k, *P["diag"]), kpgm))
|
||||
entries.append((inz + ".counts", [1],
|
||||
struct.pack("<f", float(P["rows"])), kpcnt))
|
||||
return entries, census
|
||||
|
||||
|
||||
def enc_str(v):
|
||||
b = v.encode("utf-8")
|
||||
return struct.pack("<Q", len(b)) + b
|
||||
|
||||
|
||||
def write_gguf(path, entries, datasets, chunk_size, chunk_count):
|
||||
"""Write GGUF v3 and return its size in bytes.
|
||||
|
||||
One payload per key: entries that share a key share the bytes and the
|
||||
offset, so a second tensor info states the same fact without copying it."""
|
||||
kv = bytearray()
|
||||
kv += enc_str("general.alignment") + struct.pack("<I", GT_U32) \
|
||||
+ struct.pack("<I", ALIGNMENT)
|
||||
kv += enc_str("imatrix.chunk_size") + struct.pack("<I", GT_U32) \
|
||||
+ struct.pack("<I", chunk_size)
|
||||
kv += enc_str("imatrix.chunk_count") + struct.pack("<I", GT_U32) \
|
||||
+ struct.pack("<I", chunk_count)
|
||||
kv += enc_str("imatrix.datasets") + struct.pack("<I", GT_ARR) \
|
||||
+ struct.pack("<I", GT_STR) + struct.pack("<Q", len(datasets))
|
||||
for d in datasets:
|
||||
kv += enc_str(d)
|
||||
|
||||
head = bytearray()
|
||||
head += b"GGUF" + struct.pack("<I", 3)
|
||||
head += struct.pack("<Q", len(entries)) + struct.pack("<Q", 4)
|
||||
head += kv
|
||||
|
||||
payloads = {}
|
||||
for name, _dims, payload, key in entries:
|
||||
if payload is None:
|
||||
continue
|
||||
prev = payloads.get(key)
|
||||
if prev is not None and prev != payload:
|
||||
# Two different payloads on one key would ship whichever was
|
||||
# appended last while BOTH tensor infos pointed at it: the counts
|
||||
# payload landing over the in_sum2 payload is exactly how a
|
||||
# hand-rolled GGUF writer loses half its data without an error.
|
||||
refuse("INTERNAL %s: two different payloads share one staging key"
|
||||
% name)
|
||||
payloads.setdefault(key, payload)
|
||||
infos = bytearray()
|
||||
offsets, order, size = {}, [], 0
|
||||
for name, dims, payload, key in entries:
|
||||
if key not in payloads:
|
||||
refuse("INTERNAL %s: no payload staged for its key" % name)
|
||||
if key not in offsets:
|
||||
offsets[key] = (size + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
|
||||
size = offsets[key] + len(payloads[key])
|
||||
order.append(key)
|
||||
infos += enc_str(name) + struct.pack("<I", len(dims))
|
||||
for d in dims:
|
||||
infos += struct.pack("<Q", d)
|
||||
infos += struct.pack("<I", GGML_F32) + struct.pack("<Q", offsets[key])
|
||||
|
||||
body = bytes(head) + bytes(infos)
|
||||
data_start = (len(body) + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
|
||||
with open(path, "wb") as f:
|
||||
f.write(body)
|
||||
f.write(b"\0" * (data_start - len(body)))
|
||||
# The offsets in the tensor infos are the file's law, not a hint: each
|
||||
# payload must land at its own padded offset. Writing them back to back
|
||||
# instead leaves every reader that trusts the info — imatrix.cpp does —
|
||||
# reading its neighbour's floats: plausible values, no error, wrong
|
||||
# quantisation.
|
||||
here = data_start
|
||||
for key in order:
|
||||
f.write(b"\0" * (data_start + offsets[key] - here))
|
||||
f.write(payloads[key])
|
||||
here = data_start + offsets[key] + len(payloads[key])
|
||||
return f.tell()
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(
|
||||
description="Fold a radiance calibration recording into a GGUF v3 imatrix.")
|
||||
ap.add_argument("calib_dir")
|
||||
ap.add_argument("-o", "--out", required=True)
|
||||
ap.add_argument("--datasets", default="",
|
||||
help="comma-separated names for imatrix.datasets")
|
||||
ap.add_argument("--chunk-size", type=int, default=4096,
|
||||
help="rows per chunk for imatrix.chunk_size (default 4096)")
|
||||
ap.add_argument("--min-rows", type=int, default=0,
|
||||
help="own-gram row threshold (default 4*k, the engine's"
|
||||
" kOwnRowsPerK law)")
|
||||
ap.add_argument("-v", "--verbose", action="store_true")
|
||||
a = ap.parse_args()
|
||||
|
||||
if os.path.exists(a.out):
|
||||
# The repo's deletion law: an artifact that exists is moved to trash by
|
||||
# a human, never silently overwritten by a re-run of the tool that made
|
||||
# it, so a second convert cannot destroy the imatrix the first one fed.
|
||||
refuse("EXISTS %s: an imatrix is already there; move it to"
|
||||
" .AGENTTRASH/ or name another -o" % a.out)
|
||||
if a.chunk_size <= 0:
|
||||
refuse("BAD_CHUNK_SIZE --chunk-size %d is not positive" % a.chunk_size)
|
||||
layers, rotated = load_dir(a.calib_dir, a.verbose)
|
||||
entries, census = build_entries(layers, a.min_rows, a.verbose)
|
||||
if not entries:
|
||||
refuse("NO_GRAMS %s: no layer holds a pooled gram to fold" % a.calib_dir)
|
||||
if rotated:
|
||||
sys.stderr.write("note: %d gram(s) recorded in a rotated domain: their"
|
||||
" diagonal is the rotated activation's second moment,"
|
||||
" not the model's own\n" % rotated)
|
||||
|
||||
chunk_count = census["rows"] // a.chunk_size
|
||||
if chunk_count > 0xFFFFFFFF:
|
||||
# imatrix.chunk_count is a u32 in the reader's grammar, so there is no
|
||||
# legal way to record a bigger one. Reaching this means either a chunk
|
||||
# size picked for a far larger run or a row total read out of the wrong
|
||||
# header offset — rows is a u64 and a u32 parse is off by 2**32 — and
|
||||
# the reader would trust whatever number landed in the field.
|
||||
refuse("CHUNK_COUNT %d = %d rows // %d does not fit the u32"
|
||||
" imatrix.chunk_count field; raise --chunk-size"
|
||||
% (chunk_count, census["rows"], a.chunk_size))
|
||||
nbytes = write_gguf(a.out, entries, [d for d in a.datasets.split(",") if d],
|
||||
a.chunk_size, chunk_count)
|
||||
sys.stderr.write("IMATRIX_OK entries=%d layers=%d experts=%d inherited=%d"
|
||||
" rows=%d routing=%d chunks=%d/%d bytes=%d out=%s\n"
|
||||
% (len(entries), census["layers"], census["experts"],
|
||||
census["inherited"], census["rows"], census["routing"],
|
||||
chunk_count, a.chunk_size, nbytes, a.out))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+391
@@ -0,0 +1,391 @@
|
||||
#!/usr/bin/env bash
|
||||
# Metadata-provider keys for Jellyfin on Dankserver, placed from the service's
|
||||
# host env file with the plugin artifact pinned by content hash.
|
||||
#
|
||||
# Jellyfin 12 ships TMDb, OMDb, StudioImages and MusicBrainz as CORE plugins:
|
||||
# they have no DLL, only a configuration file under the data root, and that
|
||||
# file is created by the running container. So the fleet's provider state lives
|
||||
# in two places the repo does not track —
|
||||
#
|
||||
# plugins/configurations/Jellyfin.Plugin.Tmdb.xml <TmdbApiKey /> (empty = keyless)
|
||||
# plugins/configurations/Jellyfin.Plugin.Tvdb.xml <ApiKey /> (needs the plugin)
|
||||
#
|
||||
# and both are owned by the container user. This tool is the only thing that
|
||||
# writes them: it is declarative, idempotent, and it never prints a key, so it
|
||||
# is safe to run while someone watches.
|
||||
#
|
||||
# Keys never enter git and never enter argv (a value in argv is world-readable
|
||||
# through /proc/<pid>/cmdline). They live in the per-service host env file that
|
||||
# is the fleet's convention for container credentials (grafana, caddy, pihole,
|
||||
# copyparty, romm, seerr), mode 600:
|
||||
#
|
||||
# /home/dank/Containers/jellyfin/provider-keys
|
||||
# TMDB_API_KEY=<key>
|
||||
# TVDB_API_KEY=<key>
|
||||
# TVDB_PIN=<pin> # optional: only a subscriber-shaped TVDB key
|
||||
# # needs one; a licensed key does not
|
||||
#
|
||||
# TVDB itself is not core in 12 (the metadata provider was cut in 10.9), so it
|
||||
# is installed from the official catalog, pinned three ways: sha256 of the
|
||||
# artifact, the plugin's own guid, and its version. A matching sha proves only
|
||||
# that the bytes are the bytes fetched — the guid check is what says it is the
|
||||
# official plugin, the version check is what says it is the one this file was
|
||||
# reviewed against.
|
||||
#
|
||||
# Subcommands
|
||||
# plan print the resolved shape and every pending change, touch nothing
|
||||
# install fetch+verify the plugin, then write the keys (backing up first)
|
||||
# verify prove the plugin and both keys are actually in place
|
||||
#
|
||||
# Nothing here restarts Jellyfin: the deployer owns that
|
||||
# (`scripts/deploy.sh dankserver --service jellyfin`), and the plugin is read at
|
||||
# start-up, so a key written while the server runs is not live until then.
|
||||
set -euo pipefail
|
||||
|
||||
# Jellyfin's data root on Dankserver (the rootless quadlet's config volume).
|
||||
JF_ROOT="${JF_ROOT:-/home/dank/Containers/jellyfin/config}"
|
||||
PROV_KEYS="${PROV_KEYS:-/home/dank/Containers/jellyfin/provider-keys}"
|
||||
TRASH_DIR="${TRASH_DIR:-$HOME/.AGENTTRASH/$(date +%Y%m%d)-jellyfin-providers}"
|
||||
|
||||
PLUGINS="$JF_ROOT/plugins"
|
||||
CONF="$PLUGINS/configurations"
|
||||
TMDB_XML="$CONF/Jellyfin.Plugin.Tmdb.xml"
|
||||
TVDB_XML="$CONF/Jellyfin.Plugin.Tvdb.xml"
|
||||
|
||||
# repo.jellyfin.org catalog entry for TheTVDB (owner: jellyfin), version 24.0.0.0,
|
||||
# targetAbi 12.0.0.0 — published 2026-09-09, verified against Jellyfin 12.4.0.
|
||||
TVDB_PLUGIN_URL="${TVDB_PLUGIN_URL:-https://repo.jellyfin.org/files/plugin/thetvdb/thetvdb_24.0.0.0.zip}"
|
||||
TVDB_PLUGIN_SHA256="${TVDB_PLUGIN_SHA256:-6302fba3857bdde8d186dd099fac09ca43247041f531a528a2c788add70a5319}"
|
||||
TVDB_PLUGIN_GUID="${TVDB_PLUGIN_GUID:-a677c0da-fac5-4cde-941a-7134223f14c8}"
|
||||
TVDB_PLUGIN_VERSION="${TVDB_PLUGIN_VERSION:-24.0.0.0}"
|
||||
PLUGIN_DIR="thetvdb_${TVDB_PLUGIN_VERSION}"
|
||||
TVDB_DLL="Jellyfin.Plugin.Tvdb.dll"
|
||||
|
||||
die() {
|
||||
printf 'jellyfin-providers: %s\n' "$*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
usage() {
|
||||
printf 'usage: %s plan|install|verify\n' "$0" >&2
|
||||
exit 2
|
||||
}
|
||||
|
||||
# check_keys — refuse anything that is not a readable private file. An open
|
||||
# mode is an error, not a warning: a provider key readable by every user on the
|
||||
# box is the failure this check exists to catch.
|
||||
check_keys() {
|
||||
[ -f "$PROV_KEYS" ] ||
|
||||
die "$PROV_KEYS not found (create it with TMDB_API_KEY= and TVDB_API_KEY=, then chmod 600)"
|
||||
local mode
|
||||
mode="$(stat -c '%a' "$PROV_KEYS")"
|
||||
case "$mode" in
|
||||
600 | 400) ;;
|
||||
*) die "$PROV_KEYS is mode $mode; provider keys must be 600" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# keys_value <VAR> — the value of one variable in the env file, or nothing.
|
||||
keys_value() {
|
||||
awk -v k="$1" -F= '$1 == k { sub(/^[^=]*=/, ""); print; exit }' "$PROV_KEYS"
|
||||
}
|
||||
|
||||
# keys_len <VAR> — the length, and a refusal naming the variable when absent.
|
||||
# The value itself is never printed.
|
||||
keys_len() {
|
||||
local v
|
||||
v="$(keys_value "$1")"
|
||||
[ -n "$v" ] || die "$PROV_KEYS has no $1= line"
|
||||
printf '%s' "${#v}"
|
||||
}
|
||||
|
||||
# xml_value <file> <element> — the element's text, empty when it is absent or
|
||||
# self-closing.
|
||||
xml_value() {
|
||||
python3 - "$1" "$2" <<'PYEOF'
|
||||
import re, sys
|
||||
src = open(sys.argv[1], encoding="utf-8").read()
|
||||
m = re.search(r"<%s(?:\s*/>|>(.*?)</%s>)" % (sys.argv[2], sys.argv[2]), src, re.S)
|
||||
print(m.group(1).strip() if (m and m.group(1)) else "")
|
||||
PYEOF
|
||||
}
|
||||
|
||||
# xml_secret <file> <element> <var> — set <element> to that variable from the
|
||||
# env file. The python process reads the env file itself, so the value never
|
||||
# appears in argv.
|
||||
xml_secret() {
|
||||
python3 - "$1" "$2" "$3" "$PROV_KEYS" <<'PYEOF'
|
||||
import os, re, sys, tempfile
|
||||
from xml.sax.saxutils import escape
|
||||
|
||||
xml_path, elem, var, keys_path = sys.argv[1:5]
|
||||
val = ""
|
||||
for line in open(keys_path, encoding="utf-8"):
|
||||
key, sep, rest = line.partition("=")
|
||||
if sep and key.strip() == var:
|
||||
val = rest.strip()
|
||||
break
|
||||
if not val:
|
||||
raise SystemExit(f"{keys_path} has no {var}= line")
|
||||
|
||||
src = open(xml_path, encoding="utf-8").read()
|
||||
pat = re.compile(r"([ \t]*)<%s(?:\s*/>|>.*?</%s>)" % (elem, elem), re.S)
|
||||
text = escape(val)
|
||||
if pat.search(src):
|
||||
out = pat.sub(lambda m: f"{m.group(1)}<{elem}>{text}</{elem}>", src, count=1)
|
||||
else:
|
||||
out = src.replace("</Configuration>", f" <{elem}>{text}</{elem}>\n</Configuration>", 1)
|
||||
if out == src:
|
||||
raise SystemExit(f"{xml_path}: no place to write <{elem}>")
|
||||
|
||||
d, base = os.path.split(xml_path)
|
||||
fd, tmp = tempfile.mkstemp(prefix=base + ".", dir=d)
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as fh:
|
||||
fh.write(out)
|
||||
os.chmod(tmp, os.stat(xml_path).st_mode & 0o777)
|
||||
os.replace(tmp, xml_path)
|
||||
PYEOF
|
||||
}
|
||||
|
||||
# plugin_meta <zip> — "guid version" out of the artifact's own meta.json.
|
||||
plugin_meta() {
|
||||
python3 - "$1" <<'PYEOF'
|
||||
import json, sys, zipfile
|
||||
meta = json.loads(zipfile.ZipFile(sys.argv[1]).read("meta.json").decode("utf-8"))
|
||||
print(meta.get("guid", ""), meta.get("version", ""))
|
||||
PYEOF
|
||||
}
|
||||
|
||||
check_root() {
|
||||
# Never fabricate a data root: creating plugins/ beside a typo'd path would
|
||||
# install a plugin Jellyfin never loads and hide the operator's mistake.
|
||||
[ -d "$CONF" ] || die "no Jellyfin configuration directory at $CONF (is JF_ROOT right?)"
|
||||
}
|
||||
|
||||
check_keys_report() {
|
||||
check_keys
|
||||
# A refusal inside a command substitution used as a printf argument would
|
||||
# be swallowed, so the guard runs as its own statement.
|
||||
keys_len TMDB_API_KEY >/dev/null
|
||||
keys_len TVDB_API_KEY >/dev/null
|
||||
printf '= keys read from %s (mode %s)\n' "$PROV_KEYS" "$(stat -c '%a' "$PROV_KEYS")"
|
||||
}
|
||||
|
||||
# The plugin's state, as one line in the shared vocabulary.
|
||||
plugin_state() {
|
||||
if [ -d "$PLUGINS/$PLUGIN_DIR" ]; then
|
||||
printf '= %s already installed\n' "$PLUGIN_DIR"
|
||||
return 0
|
||||
fi
|
||||
local other
|
||||
other="$(find "$PLUGINS" -maxdepth 1 -type d -name 'thetvdb_*' ! -name "$PLUGIN_DIR" -printf '%f\n' 2>/dev/null | sort | tr '\n' ' ')"
|
||||
if [ -n "$other" ]; then
|
||||
printf '+ install %s (a different version is present: %s)\n' "$PLUGIN_DIR" "$other"
|
||||
return 0
|
||||
fi
|
||||
printf '+ install %s from %s\n' "$PLUGIN_DIR" "${TVDB_PLUGIN_URL#*://}"
|
||||
}
|
||||
|
||||
tmdb_state() {
|
||||
if [ ! -f "$TMDB_XML" ]; then
|
||||
printf '! %s is missing — save the TMDb plugin settings once in the UI, then re-run\n' "$TMDB_XML" >&2
|
||||
return 1
|
||||
fi
|
||||
local now want
|
||||
now="$(xml_value "$TMDB_XML" TmdbApiKey)"
|
||||
want="$(keys_value TMDB_API_KEY)"
|
||||
if [ -n "$now" ] && [ "$now" = "$want" ]; then
|
||||
printf '= TmdbApiKey already set (%s chars)\n' "${#now}"
|
||||
elif [ -n "$now" ]; then
|
||||
printf '+ replace the TmdbApiKey in Jellyfin.Plugin.Tmdb.xml\n'
|
||||
else
|
||||
printf '+ set TmdbApiKey in Jellyfin.Plugin.Tmdb.xml\n'
|
||||
fi
|
||||
}
|
||||
|
||||
tvdb_state() {
|
||||
local now
|
||||
if [ -f "$TVDB_XML" ]; then
|
||||
now="$(xml_value "$TVDB_XML" ApiKey)"
|
||||
if [ -n "$now" ]; then
|
||||
printf '= Jellyfin.Plugin.Tvdb.xml already has an ApiKey (kept)\n'
|
||||
else
|
||||
printf '+ fill the empty ApiKey in Jellyfin.Plugin.Tvdb.xml\n'
|
||||
fi
|
||||
else
|
||||
printf '+ create Jellyfin.Plugin.Tvdb.xml with ApiKey\n'
|
||||
fi
|
||||
}
|
||||
|
||||
write_plugin() {
|
||||
local tmp guid version other
|
||||
other="$(find "$PLUGINS" -maxdepth 1 -type d -name 'thetvdb_*' ! -name "$PLUGIN_DIR" -printf '%f\n' 2>/dev/null | sort | tr '\n' ' ')"
|
||||
[ -z "$other" ] ||
|
||||
die "another TheTVDB version is already installed ($other); move it to the trash first — two copies of one plugin means two loads"
|
||||
|
||||
tmp="$(mktemp "${TMPDIR:-/tmp}/thetvdb.XXXXXX.zip")"
|
||||
# The pins are checked before anything is unpacked, so a bad artifact never
|
||||
# reaches the plugins directory.
|
||||
curl -fsSL --max-time 300 -o "$tmp" "$TVDB_PLUGIN_URL" ||
|
||||
{ rm -f "$tmp"; die "could not fetch $TVDB_PLUGIN_URL"; }
|
||||
local got
|
||||
got="$(sha256sum "$tmp" | cut -d' ' -f1)"
|
||||
if [ "$got" != "$TVDB_PLUGIN_SHA256" ]; then
|
||||
rm -f "$tmp"
|
||||
die "artifact sha256 is $got, pinned $TVDB_PLUGIN_SHA256 — refusing to install"
|
||||
fi
|
||||
read -r guid version <<<"$(plugin_meta "$tmp")"
|
||||
if [ "$guid" != "$TVDB_PLUGIN_GUID" ]; then
|
||||
rm -f "$tmp"
|
||||
die "artifact guid is $guid, expected the official $TVDB_PLUGIN_GUID — refusing to install"
|
||||
fi
|
||||
if [ "$version" != "$TVDB_PLUGIN_VERSION" ]; then
|
||||
rm -f "$tmp"
|
||||
die "artifact version is $version, pinned $TVDB_PLUGIN_VERSION — refusing to install"
|
||||
fi
|
||||
mkdir -p "$PLUGINS/$PLUGIN_DIR"
|
||||
python3 -m zipfile -e "$tmp" "$PLUGINS/$PLUGIN_DIR/"
|
||||
rm -f "$tmp"
|
||||
[ -f "$PLUGINS/$PLUGIN_DIR/$TVDB_DLL" ] ||
|
||||
die "$PLUGIN_DIR has no $TVDB_DLL — the artifact does not match this plugin"
|
||||
printf '+ installed %s\n' "$PLUGIN_DIR"
|
||||
}
|
||||
|
||||
backup_to_trash() {
|
||||
local f="$1"
|
||||
mkdir -p "$TRASH_DIR"
|
||||
local dest="$TRASH_DIR/$(basename "$f").$(date +%H%M%S)"
|
||||
cp -p "$f" "$dest"
|
||||
printf '+ previous copy kept at %s\n' "$dest"
|
||||
}
|
||||
|
||||
cmd_write_keys() {
|
||||
local now want
|
||||
KEYS_CHANGED=0
|
||||
# TMDb: the running container owns this file and every setting in it. Only
|
||||
# the key element is touched, so country, language and adult settings
|
||||
# survive the edit.
|
||||
[ -f "$TMDB_XML" ] ||
|
||||
die "$TMDB_XML is missing — save the TMDb plugin settings once in the UI, then re-run"
|
||||
now="$(xml_value "$TMDB_XML" TmdbApiKey)"
|
||||
keys_len TMDB_API_KEY >/dev/null
|
||||
want="$(keys_value TMDB_API_KEY)"
|
||||
if [ "$now" = "$want" ]; then
|
||||
printf '= TmdbApiKey already set (%s chars)\n' "${#now}"
|
||||
else
|
||||
backup_to_trash "$TMDB_XML"
|
||||
xml_secret "$TMDB_XML" TmdbApiKey TMDB_API_KEY
|
||||
printf '+ set TmdbApiKey (%s chars)\n' "${#want}"
|
||||
KEYS_CHANGED=1
|
||||
fi
|
||||
|
||||
# TVDB: never overwrite a config the operator saved through the UI.
|
||||
if [ -f "$TVDB_XML" ]; then
|
||||
now="$(xml_value "$TVDB_XML" ApiKey)"
|
||||
if [ -n "$now" ]; then
|
||||
printf '= Jellyfin.Plugin.Tvdb.xml already has an ApiKey (kept)\n'
|
||||
else
|
||||
keys_len TVDB_API_KEY >/dev/null
|
||||
backup_to_trash "$TVDB_XML"
|
||||
xml_secret "$TVDB_XML" ApiKey TVDB_API_KEY
|
||||
if [ -n "$(keys_value TVDB_PIN)" ] && [ -z "$(xml_value "$TVDB_XML" Pin)" ]; then
|
||||
xml_secret "$TVDB_XML" Pin TVDB_PIN
|
||||
fi
|
||||
printf '+ filled the ApiKey in Jellyfin.Plugin.Tvdb.xml\n'
|
||||
KEYS_CHANGED=1
|
||||
fi
|
||||
else
|
||||
keys_len TVDB_API_KEY >/dev/null
|
||||
cat >"$TVDB_XML" <<XML
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<Configuration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
|
||||
<ApiKey />
|
||||
</Configuration>
|
||||
XML
|
||||
xml_secret "$TVDB_XML" ApiKey TVDB_API_KEY
|
||||
if [ -n "$(keys_value TVDB_PIN)" ]; then
|
||||
xml_secret "$TVDB_XML" Pin TVDB_PIN
|
||||
fi
|
||||
printf '+ created Jellyfin.Plugin.Tvdb.xml with ApiKey\n'
|
||||
KEYS_CHANGED=1
|
||||
fi
|
||||
}
|
||||
|
||||
cmd_plan() {
|
||||
check_root
|
||||
check_keys_report
|
||||
printf 'root %s\n' "$JF_ROOT"
|
||||
plugin_state
|
||||
tmdb_state
|
||||
tvdb_state
|
||||
}
|
||||
|
||||
cmd_install() {
|
||||
check_root
|
||||
check_keys_report
|
||||
KEYS_CHANGED=0
|
||||
if [ ! -d "$PLUGINS/$PLUGIN_DIR" ]; then
|
||||
write_plugin
|
||||
KEYS_CHANGED=1
|
||||
else
|
||||
printf '= %s already installed\n' "$PLUGIN_DIR"
|
||||
fi
|
||||
cmd_write_keys
|
||||
if [ "$KEYS_CHANGED" -eq 1 ]; then
|
||||
# Nothing is restarted here on purpose: this host may be playing media
|
||||
# right now, and both the plugin and its keys are read at start-up.
|
||||
printf 'next: scripts/deploy.sh dankserver --service jellyfin (the plugin and keys load at start-up)\n'
|
||||
else
|
||||
printf '= no change\n'
|
||||
fi
|
||||
}
|
||||
|
||||
cmd_verify() {
|
||||
check_root
|
||||
local bad=0
|
||||
if [ ! -d "$PLUGINS/$PLUGIN_DIR" ]; then
|
||||
printf 'MISSING plugin directory %s (thetvdb not installed)\n' "$PLUGIN_DIR" >&2
|
||||
bad=1
|
||||
else
|
||||
local dlls abi
|
||||
dlls="$(find "$PLUGINS/$PLUGIN_DIR" -maxdepth 1 -name '*.dll' | wc -l)"
|
||||
[ "$dlls" -ge 1 ] || {
|
||||
printf 'MISSING %s inside %s\n' "$TVDB_DLL" "$PLUGIN_DIR" >&2
|
||||
bad=1
|
||||
}
|
||||
abi="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1])).get("targetAbi","?"))' \
|
||||
"$PLUGINS/$PLUGIN_DIR/meta.json" 2>/dev/null || echo '?')"
|
||||
printf 'plugin %s: %s dll(s), targetAbi %s\n' "$PLUGIN_DIR" "$dlls" "$abi"
|
||||
fi
|
||||
local pair elem n
|
||||
for pair in "$TMDB_XML|TmdbApiKey" "$TVDB_XML|ApiKey"; do
|
||||
elem="${pair##*|}"
|
||||
if [ ! -f "${pair%%|*}" ]; then
|
||||
printf 'MISSING %s\n' "${pair%%|*}" >&2
|
||||
bad=1
|
||||
continue
|
||||
fi
|
||||
if ! python3 -c 'import sys, xml.dom.minidom; xml.dom.minidom.parse(sys.argv[1])' "${pair%%|*}"; then
|
||||
printf 'UNPARSEABLE %s\n' "${pair%%|*}" >&2
|
||||
bad=1
|
||||
continue
|
||||
fi
|
||||
n="$(xml_value "${pair%%|*}" "$elem")"
|
||||
if [ -z "$n" ]; then
|
||||
printf 'MISSING key: %s has no %s value\n' "$(basename "${pair%%|*}")" "$elem" >&2
|
||||
bad=1
|
||||
else
|
||||
printf '%s %s chars\n' "$elem" "${#n}"
|
||||
fi
|
||||
done
|
||||
[ "$bad" -eq 0 ] || return 1
|
||||
printf 'providers configured\n'
|
||||
}
|
||||
|
||||
cmd="${1:-}"
|
||||
case "$cmd" in
|
||||
plan) cmd_plan ;;
|
||||
install) cmd_install ;;
|
||||
verify) cmd_verify ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
Executable
+186
@@ -0,0 +1,186 @@
|
||||
#!/usr/bin/env bash
|
||||
# kld-gate-qwen38.sh — RUNS ON DANKSERVER (or anywhere the reports are). The campaign's
|
||||
# acceptance gate: one verdict from the KLD reports of the containers that ride one reference
|
||||
# recording.
|
||||
#
|
||||
# The reference is the BF16 container built from the vendor checkpoint (run-qwen38-bf16.sh
|
||||
# kldref, in the same drained window that records the calibration grams). Every scored
|
||||
# container is measured against that one recording, so the KL scale runs from 0 = the
|
||||
# checkpoint itself downward in faithfulness.
|
||||
#
|
||||
# TWO MODES, because the disk decides how many anchors can exist at once:
|
||||
#
|
||||
# gate IQ4R.json CAND.json candidate vs the served payload
|
||||
# gate FP8.json IQ4R.json CAND.json the same, inside the absolute interval
|
||||
#
|
||||
# iq4r.json is the payload frank-archon serves — the bar to beat. When the FP8-native release
|
||||
# (172.8 GiB, fetched last because of the peak-disk law) is on disk, pass its report first:
|
||||
# "half the gap" is then half of the distance between the 4-bit payload and the near-lossless
|
||||
# release, not half of an invented distance to the checkpoint. Without it the gate still
|
||||
# answers the campaign's question — does the candidate beat what is served — and says so.
|
||||
#
|
||||
# The law, on the summary of one slice (--slice all | a corpus source):
|
||||
#
|
||||
# candidate vs served (2 reports)
|
||||
# 1. mean KL(cand) < mean KL(iq4r) the served payload is beaten on the corpus
|
||||
# 2. p99.9(cand) <= p99.9(iq4r) and the tail is no worse — a mean can hide a few
|
||||
# positions taking the whole hit
|
||||
# 3. top-1(cand) >= top-1(iq4r) the agreement the harness keys on does not fall
|
||||
# 4. every source slice passes 1-3 a win bought by regressing one distribution
|
||||
# (code, prose, tools) is not a win
|
||||
#
|
||||
# inside the interval (3 reports), laws 1-4 above (1-3 renumbered to the interval and the
|
||||
# recovery fraction, as agreed 2026-10-27) plus:
|
||||
# 5. p99.9(cand) <= p99.9(iq4r) the tail law
|
||||
# 6. every source slice passes the slice law
|
||||
#
|
||||
# Booting the containers is NOT this script's job: `run-qwen38-bf16.sh score RAD OUT.json`
|
||||
# writes the reports and owns the single engine argv (same pin, --tp 2, --kv-cache-dtype fp8,
|
||||
# same step size, --deterministic, no speculation) that makes the numbers comparable.
|
||||
# This script is arithmetic, so it re-runs against old reports and is testable without a GPU.
|
||||
#
|
||||
# Usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]
|
||||
# kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]
|
||||
set -uo pipefail
|
||||
|
||||
if [ "$#" -lt 2 ]; then
|
||||
echo "usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]" >&2
|
||||
echo " kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]" >&2
|
||||
exit 2
|
||||
fi
|
||||
python3 - "$@" <<'PY'
|
||||
import json, sys
|
||||
|
||||
a = sys.argv[1:]
|
||||
opts, files = [], []
|
||||
i = 0
|
||||
while i < len(a):
|
||||
if a[i].startswith("--"):
|
||||
opts += [a[i], a[i + 1]]
|
||||
i += 2
|
||||
else:
|
||||
files.append(a[i])
|
||||
i += 1
|
||||
if len(files) not in (2, 3):
|
||||
sys.exit("usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]\n"
|
||||
" kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]")
|
||||
o = dict(zip(opts[0::2], opts[1::2]))
|
||||
slice_ = o.get("--slice", "all")
|
||||
recover = float(o.get("--recover", 0.5))
|
||||
if not 0.0 < recover <= 1.0:
|
||||
sys.exit(f"--recover must be in (0,1], got {recover}")
|
||||
interval = len(files) == 3
|
||||
|
||||
|
||||
def die(msg):
|
||||
print(msg, file=sys.stderr)
|
||||
raise SystemExit(2)
|
||||
|
||||
|
||||
def load(path):
|
||||
try:
|
||||
return json.load(open(path))
|
||||
except Exception as e:
|
||||
die(f"cannot read the KLD report {path}: {e}")
|
||||
|
||||
|
||||
def summary(path, d):
|
||||
"""The slice this gate grades: 'all', or one corpus source."""
|
||||
if slice_ == "all":
|
||||
g = d.get("all")
|
||||
else:
|
||||
g = (d.get("by_source") or {}).get(slice_)
|
||||
if g is None:
|
||||
die(f"{path} has no slice {slice_!r}; it has: "
|
||||
f"{', '.join(sorted(d.get('by_source') or {})) or '(none)'}")
|
||||
if g is None or "kld" not in g or "top1_agreement" not in g:
|
||||
die(f"{path} has no {slice_!r} summary — wrong report version?")
|
||||
return g
|
||||
|
||||
|
||||
docs = {label: load(path) for label, path in
|
||||
zip(("fp8", "iq4r", "cand") if interval else ("iq4r", "cand"), files)}
|
||||
rows = []
|
||||
for label, path in zip(("fp8", "iq4r", "cand") if interval else ("iq4r", "cand"), files):
|
||||
g = summary(path, docs[label])
|
||||
rows.append((label, path, docs[label].get("model", "?"), g["kld"]["mean"],
|
||||
g["kld"]["p99_9"], g["top1_agreement"], g["positions"]))
|
||||
|
||||
print(f"reference: BF16 container; slice: {slice_}; "
|
||||
+ ("inside the FP8 interval" if interval else "candidate vs served (no FP8 anchor)"))
|
||||
print(f"{'':6} {'model':38} {'KL mean':>10} {'KL p99.9':>10} {'top-1':>8} positions")
|
||||
for label, path, model, kl, p999, top1, n in rows:
|
||||
print(f"{label:6} {str(model)[:38]:38} {kl:10.6f} {p999:10.4f} {100*top1:7.2f}% {n}")
|
||||
r = {label: (kl, p999, top1) for label, _p, _m, kl, p999, top1, _n in rows}
|
||||
kl_cand, p_cand, t_cand = r["cand"]
|
||||
kl_iq4r, p_iq4r, t_iq4r = r["iq4r"]
|
||||
ok = True
|
||||
|
||||
|
||||
def law(n, text, passed):
|
||||
global ok
|
||||
ok = ok and passed
|
||||
print(f" [{'PASS' if passed else 'FAIL'}] {n}. {text}")
|
||||
|
||||
|
||||
def tail_law(n):
|
||||
law(n, f"candidate KL p99.9 {p_cand:.4f} <= served iq4r {p_iq4r:.4f}",
|
||||
p_cand <= p_iq4r)
|
||||
|
||||
|
||||
# --- the slice law: every corpus source must pass the metric laws -------------
|
||||
def slice_law(n):
|
||||
"""A win that regresses one source class is not a win: check them all."""
|
||||
mine = docs["cand"].get("by_source") or {}
|
||||
theirs = docs["iq4r"].get("by_source") or {}
|
||||
if not mine or not theirs:
|
||||
law(n, "no per-source summaries to check", True)
|
||||
return
|
||||
missing = sorted(set(theirs) - set(mine))
|
||||
extra = sorted(set(mine) - set(theirs))
|
||||
if missing or extra:
|
||||
law(n, "same source slices in both reports"
|
||||
+ (f" (missing from cand: {', '.join(missing)})" if missing else "")
|
||||
+ (f" (only in cand: {', '.join(extra)})" if extra else ""), False)
|
||||
return
|
||||
bad = []
|
||||
for src in sorted(theirs):
|
||||
c, q = mine[src], theirs[src]
|
||||
worse_mean = c["kld"]["mean"] >= q["kld"]["mean"]
|
||||
worse_tail = c["kld"]["p99_9"] > q["kld"]["p99_9"]
|
||||
worse_top1 = c["top1_agreement"] < q["top1_agreement"]
|
||||
if worse_mean or worse_tail or worse_top1:
|
||||
why = ",".join(m for m, w in (("mean", worse_mean), ("p99.9", worse_tail),
|
||||
("top-1", worse_top1)) if w)
|
||||
bad.append(f"{src} ({why})")
|
||||
law(n, "every source slice passes (mean, p99.9, top-1)"
|
||||
+ (f" — regressed: {'; '.join(bad)}" if bad else ""), not bad)
|
||||
|
||||
|
||||
if interval:
|
||||
kl_fp8 = r["fp8"][0]
|
||||
gap = kl_iq4r - kl_fp8
|
||||
law(1, f"anchors ordered: fp8 {kl_fp8:.6f} < iq4r {kl_iq4r:.6f} (gap {gap:.6f})",
|
||||
gap > 0)
|
||||
law(2, f"candidate inside the interval: {kl_fp8:.6f} < {kl_cand:.6f} < {kl_iq4r:.6f}",
|
||||
kl_fp8 < kl_cand < kl_iq4r)
|
||||
need = kl_iq4r - recover * gap
|
||||
law(3, f"candidate closes {100*max((kl_iq4r - kl_cand)/gap, 0):.1f}% of the "
|
||||
f"{gap:.6f} gap (needs {100*recover:.0f}%, i.e. KL <= {need:.6f})"
|
||||
if gap > 0 else "gap is not positive; cannot score recovery",
|
||||
gap > 0 and kl_cand <= need)
|
||||
law(4, f"candidate top-1 {100*t_cand:.2f}% >= served iq4r top-1 {100*t_iq4r:.2f}%",
|
||||
t_cand >= t_iq4r)
|
||||
tail_law(5)
|
||||
slice_law(6)
|
||||
else:
|
||||
law(1, f"candidate beats the served payload: KL {kl_cand:.6f} < iq4r {kl_iq4r:.6f}",
|
||||
kl_cand < kl_iq4r)
|
||||
tail_law(2)
|
||||
law(3, f"candidate top-1 {100*t_cand:.2f}% >= served iq4r top-1 {100*t_iq4r:.2f}%",
|
||||
t_cand >= t_iq4r)
|
||||
slice_law(4)
|
||||
|
||||
print("KLD GATE " + ("PASS" if ok else "FAIL"))
|
||||
sys.exit(0 if ok else 1)
|
||||
PY
|
||||
+140
-77
@@ -34,14 +34,25 @@ import json
|
||||
import os
|
||||
import random
|
||||
import re
|
||||
import sys
|
||||
|
||||
from jinja2 import Environment, FileSystemLoader
|
||||
from tokenizers import Tokenizer
|
||||
|
||||
_HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
|
||||
if _HERE not in sys.path:
|
||||
sys.path.insert(0, _HERE)
|
||||
|
||||
import qwen38_common as qc # noqa: E402 - sibling module, same directory
|
||||
|
||||
CODE_EXT = (".sh", ".py", ".jinja", ".recipe", ".env", ".example",
|
||||
".yaml", ".yml", ".bats", ".service", ".container", ".toml")
|
||||
JSONL_EXTS = (".jsonl", ".json")
|
||||
DOC_DIR = "serverAppConfigs-docs"
|
||||
FRAGMENT_TOKENS = 2400 # what one fragment may contribute
|
||||
CLASS_MIX = {"code": .45, "prose": .30, "tools": .25}
|
||||
HF_CAP = {"code": 40000, "prose": 60000, "tools": 0}
|
||||
POOL_CYCLES = 40 # how many times a class pool may repeat before giving up
|
||||
KLD_GUARD = 5000 # rendered conversations tried before declaring the pool short
|
||||
|
||||
FLEET_CMDS = [
|
||||
"systemctl status frank-archon --no-pager",
|
||||
@@ -142,7 +153,7 @@ def load_parquet_texts(path: str, kind: str, rng: random.Random,
|
||||
return out[:cap]
|
||||
|
||||
|
||||
def main() -> None:
|
||||
def parse_args() -> argparse.Namespace:
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("--repo", required=True)
|
||||
ap.add_argument("--tokenizer-json", required=True)
|
||||
@@ -156,22 +167,26 @@ def main() -> None:
|
||||
ap.add_argument("--hf-code-parquet", default="")
|
||||
ap.add_argument("--hf-prose-parquet", default="")
|
||||
ap.add_argument("--seed", type=int, default=20261006)
|
||||
a = ap.parse_args()
|
||||
return ap.parse_args()
|
||||
|
||||
rng = random.Random(a.seed)
|
||||
tokenizer = Tokenizer.from_file(a.tokenizer_json)
|
||||
env = Environment(loader=FileSystemLoader(os.path.dirname(a.template)),
|
||||
|
||||
def load_template(template: str) -> tuple[Environment, str]:
|
||||
"""The chat template as the engine will render it, plus its short name."""
|
||||
env = Environment(loader=FileSystemLoader(os.path.dirname(template)),
|
||||
trim_blocks=True, lstrip_blocks=True)
|
||||
env.filters["tojson"] = lambda v: json.dumps(v, ensure_ascii=False)
|
||||
env.filters["raise_exception"] = lambda m: (_ for _ in ()).throw(
|
||||
ValueError(m))
|
||||
tname = os.path.basename(a.template)
|
||||
return env, os.path.basename(template)
|
||||
|
||||
|
||||
def classify(repo: str) -> dict[str, list[str]]:
|
||||
"""Walk the checkout into the three source classes the mix draws from."""
|
||||
code, prose, jsonl = [], [], []
|
||||
for root, dirs, files in os.walk(a.repo):
|
||||
for root, dirs, files in os.walk(repo):
|
||||
dirs[:] = [d for d in dirs if not d.startswith(".")]
|
||||
for f in sorted(files):
|
||||
p = os.path.relpath(os.path.join(root, f), a.repo)
|
||||
p = os.path.relpath(os.path.join(root, f), repo)
|
||||
if DOC_DIR in p and f.endswith(".md"):
|
||||
prose.append(p)
|
||||
elif p.startswith(DOC_DIR):
|
||||
@@ -180,64 +195,92 @@ def main() -> None:
|
||||
code.append(p)
|
||||
elif f.endswith(JSONL_EXTS) and f != "package-lock.json":
|
||||
jsonl.append(p)
|
||||
# file-level split: bucket 0 -> KLD pool, 1..4 -> calibration pool
|
||||
pools: dict[str, tuple[list, list]] = {}
|
||||
for cls, lst in (("code", code), ("prose", prose), ("tools", jsonl)):
|
||||
pools[cls] = ([p for p in lst if fnv(p) % 5 == 0],
|
||||
return {"code": code, "prose": prose, "tools": jsonl}
|
||||
|
||||
|
||||
def split_pools(classes: dict[str, list[str]]) -> dict[str, tuple[list, list]]:
|
||||
"""File-level split: bucket 0 -> KLD pool, everything else -> calibration."""
|
||||
return {cls: ([p for p in lst if fnv(p) % 5 == 0],
|
||||
[p for p in lst if fnv(p) % 5 != 0])
|
||||
for cls, lst in classes.items()}
|
||||
|
||||
def text_of(p: str) -> str:
|
||||
path = os.path.join(a.repo, p)
|
||||
raw = open(path, encoding="utf-8", errors="replace").read()
|
||||
if p.endswith(JSONL_EXTS):
|
||||
lines = [json.loads(l)["content"] for l in raw.splitlines()
|
||||
if l.strip() and "content" in l]
|
||||
raw = "\n".join(lines)
|
||||
toks = tokenizer.encode(raw, add_special_tokens=False).ids
|
||||
return tokenizer.decode(toks[:2400], skip_special_tokens=True)
|
||||
|
||||
def jsonl_content(raw: str) -> str:
|
||||
"""The content fields of a content-keyed JSONL, or "" for anything else.
|
||||
|
||||
A .json in this repo is not necessarily a content-keyed JSONL: the substring
|
||||
test passes on any line that merely mentions "content", and a list-shaped
|
||||
document raises TypeError — both used to crash the whole build. Skip what
|
||||
does not fit instead.
|
||||
"""
|
||||
lines = []
|
||||
for line in raw.splitlines():
|
||||
if not line.strip() or "content" not in line:
|
||||
continue
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if isinstance(rec, dict) and isinstance(rec.get("content"), str):
|
||||
lines.append(rec["content"])
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def fragment_text(repo: str, rel: str, tok) -> str:
|
||||
"""One file as a fragment: its first FRAGMENT_TOKENS tokens, decoded."""
|
||||
raw = open(os.path.join(repo, rel), encoding="utf-8",
|
||||
errors="replace").read()
|
||||
if rel.endswith(JSONL_EXTS):
|
||||
raw = jsonl_content(raw)
|
||||
ids = tok.encode(raw, add_special_tokens=False).ids
|
||||
return tok.decode(ids[:FRAGMENT_TOKENS], skip_special_tokens=True)
|
||||
|
||||
|
||||
def build_frags(a: argparse.Namespace, tok,
|
||||
rng: random.Random) -> dict[str, dict[str, list[str]]]:
|
||||
"""Both fragment pools per class, seeded so the same seed rebuilds the corpus."""
|
||||
pools = split_pools(classify(a.repo))
|
||||
cmds = harvest_cmds(a.repo, rng)
|
||||
# fragment pools per class, corpus-split first
|
||||
frags: dict[str, dict[str, list[str]]] = {}
|
||||
for cls in pools:
|
||||
kld_files, cal_files = pools[cls]
|
||||
fr = {}
|
||||
src = {"kld": kld_files, "cal": cal_files}
|
||||
for cls, (kld_files, cal_files) in pools.items():
|
||||
if cls == "tools":
|
||||
fr["kld"] = [tool_call(rng, cmds) for _ in range(600)]
|
||||
fr["cal"] = [tool_call(rng, cmds) for _ in range(40000)]
|
||||
frags[cls] = {"kld": [tool_call(rng, cmds) for _ in range(600)],
|
||||
"cal": [tool_call(rng, cmds) for _ in range(40000)]}
|
||||
else:
|
||||
fr["kld"] = [text_of(p) for p in src["kld"]]
|
||||
fr["cal"] = [text_of(p) for p in src["cal"]]
|
||||
frags[cls] = fr
|
||||
frags[cls] = {"kld": [fragment_text(a.repo, p, tok) for p in kld_files],
|
||||
"cal": [fragment_text(a.repo, p, tok) for p in cal_files]}
|
||||
# per-class fragment caps: prose rows are short (~66 tok) so need many;
|
||||
# codeparrot files are long (~2.6k tok) so a small cap suffices
|
||||
hfcap = {"code": 40000, "prose": 60000, "tools": 0}
|
||||
for cls, spec in (("code", a.hf_code_parquet),
|
||||
("prose", a.hf_prose_parquet)):
|
||||
for path in [p for p in spec.split(",") if p]:
|
||||
frags[cls]["cal"].extend(
|
||||
load_parquet_texts(path, cls, rng, cap=hfcap[cls]))
|
||||
load_parquet_texts(path, cls, rng, cap=HF_CAP[cls]))
|
||||
for cls in frags:
|
||||
rng.shuffle(frags[cls]["cal"])
|
||||
return frags
|
||||
|
||||
def toks(s: str) -> int:
|
||||
return len(tokenizer.encode(s, add_special_tokens=False).ids)
|
||||
|
||||
def emit_calib() -> tuple[int, list[dict]]:
|
||||
targets = {"code": .45, "prose": .30, "tools": .25}
|
||||
rows: list[dict] = []
|
||||
for cls, frac in targets.items():
|
||||
want = int(a.calib_tokens * frac)
|
||||
pool = frags[cls]["cal"]
|
||||
def pack_class(cls: str, pool: list[str], want: int, tok,
|
||||
chunk_tokens: int) -> list[dict]:
|
||||
"""Greedily pack `pool` into chunk_tokens-sized calibration prompts."""
|
||||
if not pool:
|
||||
continue
|
||||
got, i, buf, nbuf = 0, 0, [], 0
|
||||
return []
|
||||
rows: list[dict] = []
|
||||
# Count every fragment once: the loop cycles the pool up to POOL_CYCLES
|
||||
# times, and re-tokenizing a 2.4k-token fragment on each pass dominated the
|
||||
# build's wall time.
|
||||
counts = [qc.n_tokens(tok, f) for f in pool]
|
||||
got = 0
|
||||
i = 0
|
||||
buf: list[str] = []
|
||||
nbuf = 0
|
||||
while got < want:
|
||||
frag = pool[i % len(pool)]
|
||||
j = i % len(pool)
|
||||
frag = pool[j]
|
||||
i += 1
|
||||
n = toks(frag)
|
||||
if nbuf + n > a.chunk_tokens and buf:
|
||||
n = counts[j]
|
||||
if nbuf + n > chunk_tokens and buf:
|
||||
rows.append({"prompt": "\n".join(buf), "cls": cls})
|
||||
got += nbuf
|
||||
buf, nbuf = [], 0
|
||||
@@ -245,22 +288,36 @@ def main() -> None:
|
||||
break
|
||||
buf.append(frag)
|
||||
nbuf += n
|
||||
if i > len(pool) * 40:
|
||||
if i > len(pool) * POOL_CYCLES:
|
||||
break # pool exhausted with repeats; stop
|
||||
if buf and got < want:
|
||||
rows.append({"prompt": "\n".join(buf), "cls": cls})
|
||||
got += nbuf
|
||||
print(f"calib {cls}: {got} tokens, {i} fragments")
|
||||
rng.shuffle(rows)
|
||||
return len(rows), [{k: v for k, v in r.items() if k == "prompt"}
|
||||
for r in rows]
|
||||
return rows
|
||||
|
||||
def emit_kld() -> tuple[int, list[dict]]:
|
||||
rows, total = [], 0
|
||||
|
||||
def emit_calib(a: argparse.Namespace, frags, tok,
|
||||
rng: random.Random) -> tuple[int, list[dict]]:
|
||||
"""calib.jsonl rows: the class mix, shuffled, prompts only."""
|
||||
rows: list[dict] = []
|
||||
for cls, frac in CLASS_MIX.items():
|
||||
rows.extend(pack_class(cls, frags[cls]["cal"], int(a.calib_tokens * frac),
|
||||
tok, a.chunk_tokens))
|
||||
rng.shuffle(rows)
|
||||
return len(rows), [{"prompt": r["prompt"]} for r in rows]
|
||||
|
||||
|
||||
def emit_kld(a: argparse.Namespace, frags, tok, rng: random.Random,
|
||||
env: Environment, tname: str) -> tuple[int, list[dict], int]:
|
||||
"""kld.jsonl rows: multi-turn conversations, each with a scored tail."""
|
||||
template = env.get_template(tname)
|
||||
rows: list[dict] = []
|
||||
total = scored_total = 0
|
||||
classes = ["code", "prose", "tools"]
|
||||
i = {c: 0 for c in classes}
|
||||
cursor = {c: 0 for c in classes}
|
||||
guard = 0
|
||||
while total < a.kld_tokens and guard < 5000:
|
||||
while total < a.kld_tokens and guard < KLD_GUARD:
|
||||
guard += 1
|
||||
msgs, srcs = [], []
|
||||
for t in range(rng.choice([2, 3, 4])):
|
||||
@@ -268,49 +325,55 @@ def main() -> None:
|
||||
pool = frags[cls]["kld"]
|
||||
if not pool:
|
||||
continue
|
||||
frag = pool[i[cls] % len(pool)]
|
||||
i[cls] += 2
|
||||
frag = pool[cursor[cls] % len(pool)]
|
||||
cursor[cls] += 1
|
||||
msgs.append({"role": "user",
|
||||
"content": "Summarize and explain this "
|
||||
f"engineering material:\n{frag}"})
|
||||
msgs.append({"role": "assistant",
|
||||
"content": frag[:4000]})
|
||||
srcs.append(cls)
|
||||
rendered = env.get_template(tname).render(
|
||||
messages=msgs, add_generation_prompt=False)
|
||||
n = toks(rendered)
|
||||
rendered = template.render(messages=msgs, add_generation_prompt=False)
|
||||
n = qc.n_tokens(tok, rendered)
|
||||
if n < a.kld_min_prompt:
|
||||
continue
|
||||
rows.append({"prompt": rendered,
|
||||
"score_from": n - a.score_tail,
|
||||
"source": "+".join(sorted(set(srcs)))})
|
||||
total += n
|
||||
return len(rows), rows
|
||||
scored_total += a.score_tail
|
||||
return len(rows), rows, scored_total
|
||||
|
||||
os.makedirs(a.out_dir, exist_ok=True)
|
||||
nc, rows = emit_calib()
|
||||
with open(os.path.join(a.out_dir, "calib.jsonl"), "w") as f:
|
||||
for r in rows:
|
||||
f.write(json.dumps(r) + "\n")
|
||||
nk, krows = emit_kld()
|
||||
with open(os.path.join(a.out_dir, "kld.jsonl"), "w") as f:
|
||||
for r in krows:
|
||||
f.write(json.dumps(r) + "\n")
|
||||
scored = sum(len(tokenizer.encode(r["prompt"], add_special_tokens=False).ids)
|
||||
- r["score_from"] for r in krows)
|
||||
manifest = {
|
||||
|
||||
def manifest(a: argparse.Namespace, tname: str, nc: int, nk: int,
|
||||
scored: int) -> dict:
|
||||
return {
|
||||
"seed": a.seed, "template": tname,
|
||||
"tokenizer": os.path.basename(a.tokenizer_json),
|
||||
"calib_prompts": nc, "kld_prompts": nk,
|
||||
"kld_scored_positions": scored,
|
||||
"kld_score_tail": a.score_tail,
|
||||
"disjoint_file_pools": "local KLD material only; HF code feeds calib",
|
||||
"calib_mix": {"code": .45, "prose": .30, "tools": .25},
|
||||
"calib_mix": dict(CLASS_MIX),
|
||||
"hf_code": a.hf_code_parquet, "hf_prose": a.hf_prose_parquet,
|
||||
}
|
||||
with open(os.path.join(a.out_dir, "manifest.json"), "w") as f:
|
||||
json.dump(manifest, f, indent=1)
|
||||
print(json.dumps(manifest, indent=1))
|
||||
|
||||
|
||||
def main() -> None:
|
||||
a = parse_args()
|
||||
rng = random.Random(a.seed)
|
||||
tok = qc.load_tokenizer(a.tokenizer_json)
|
||||
env, tname = load_template(a.template)
|
||||
|
||||
frags = build_frags(a, tok, rng)
|
||||
os.makedirs(a.out_dir, exist_ok=True)
|
||||
nc, calib_rows = emit_calib(a, frags, tok, rng)
|
||||
qc.write_jsonl(os.path.join(a.out_dir, "calib.jsonl"), calib_rows)
|
||||
nk, kld_rows, scored = emit_kld(a, frags, tok, rng, env, tname)
|
||||
qc.write_jsonl(os.path.join(a.out_dir, "kld.jsonl"), kld_rows)
|
||||
man = manifest(a, tname, nc, nk, scored)
|
||||
qc.write_json(os.path.join(a.out_dir, "manifest.json"), man)
|
||||
print(json.dumps(man, indent=1))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
Executable
+454
@@ -0,0 +1,454 @@
|
||||
#!/usr/bin/env python3
|
||||
"""media-apply.py — the only thing that moves movie-library files.
|
||||
|
||||
Reads the plan written by scripts/media-plan.py (rows: `action src dst kind`,
|
||||
plus `mkdir` rows) and performs it with rename(2) — bytes are never copied, so
|
||||
an interrupted run has destroyed nothing. Every run is resumable and every run
|
||||
is undoable: the journal is a write-ahead log, one JSON line per intent, fsynced
|
||||
BEFORE the rename it authorises. That ordering is the whole design: a crash
|
||||
leaves either an intent whose move happened or an intent whose move never ran,
|
||||
and both are recognised on the next run. A file that moved with no journal line
|
||||
cannot have been moved by this tool, and saying so is better than guessing.
|
||||
|
||||
Four subcommands:
|
||||
|
||||
apply --plan PLAN [--root ROOT] [--journal J] [--apply]
|
||||
no --apply => report only and write NOTHING, not even folders
|
||||
rollback --journal J [--root ROOT] [--apply]
|
||||
back to the paths the census saw; folders this run created are
|
||||
removed only when empty, never anything else
|
||||
verify --plan PLAN --journal J [--root ROOT]
|
||||
every planned file is at its target with the journalled byte count
|
||||
sweep --journal J [--root ROOT] [--apply]
|
||||
removes the folders THIS RUN emptied — the journal has to show a
|
||||
file that moved out of one, so a folder someone made by hand can
|
||||
never be removed by this tool, and rollback puts them back
|
||||
|
||||
Refusals (exit 2, before the first rename, with nothing touched): a target
|
||||
already occupied by a foreign file, a source that has gone, a duplicate target,
|
||||
a path outside --root, a journal that was already rolled back, a move that would
|
||||
cross a filesystem (rename cannot do it and a silent copy at 24 TB would not).
|
||||
|
||||
--force relaxes only the guards that are opinion (journal already rolled back,
|
||||
paths outside --root); it never allows overwriting a file.
|
||||
|
||||
--metadata-view names the overlay the media servers read their library through
|
||||
(default /var/lib/media-meta/view). The sidecar follow-up printed after an apply
|
||||
or rollback is conditioned on that path being a mountpoint: where no view is
|
||||
mounted the servers bind the array itself, the sidecars moved with the media,
|
||||
and a reap order would point at a directory that is not there.
|
||||
|
||||
The journal defaults to `media-relayout.jsonl` BESIDE the library root, never
|
||||
inside it: a log inside the library is a file Jellyfin will index.
|
||||
"""
|
||||
import argparse
|
||||
import datetime
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
DEFAULT_ROOT = os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies")
|
||||
|
||||
|
||||
class Refuse(Exception):
|
||||
"""A gate the operator has to see; every refusal leaves the library alone."""
|
||||
|
||||
|
||||
def journal_default(root):
|
||||
return os.path.join(os.path.dirname(os.path.abspath(root)), "media-relayout.jsonl")
|
||||
|
||||
|
||||
class Journal:
|
||||
"""Append-only JSONL of intents. Writes are fsynced before the tool acts."""
|
||||
|
||||
def __init__(self, path, writing):
|
||||
self.path, self.writing = path, writing
|
||||
self.records = []
|
||||
if os.path.exists(path):
|
||||
for line in open(path):
|
||||
line = line.strip()
|
||||
if line:
|
||||
self.records.append(json.loads(line))
|
||||
self._fh = None
|
||||
|
||||
def already(self, dst, src):
|
||||
"""An intent for this exact (src, dst) was written by an earlier run."""
|
||||
return any(r.get("op") == "move" and r.get("dst") == dst
|
||||
and r.get("src") == src for r in self.records)
|
||||
|
||||
@property
|
||||
def rolled_back(self):
|
||||
return any(r.get("op") == "rollback" for r in self.records)
|
||||
|
||||
def record(self, **fields):
|
||||
self.records.append(fields)
|
||||
if not self.writing:
|
||||
return
|
||||
if self._fh is None:
|
||||
self._fh = open(self.path, "a")
|
||||
self._fh.write(json.dumps(fields) + "\n")
|
||||
self._fh.flush()
|
||||
os.fsync(self._fh.fileno())
|
||||
|
||||
def close(self):
|
||||
if self._fh:
|
||||
self._fh.close()
|
||||
|
||||
|
||||
def read_plan(path, root, force):
|
||||
"""Plan rows: (mkdirs, keeps, moves), header dropped, all paths under root."""
|
||||
mkdirs, keeps, moves = [], [], []
|
||||
with open(path) as fh:
|
||||
header = fh.readline()
|
||||
if not header.startswith("action\t"):
|
||||
raise Refuse(f"{path} is not a plan (first column is "
|
||||
f"{header.split(chr(9))[0]!r}, expected 'action') —"
|
||||
" regenerate it with scripts/media-plan.py")
|
||||
for line in fh:
|
||||
f = line.rstrip("\n").split("\t")
|
||||
if f[0] == "mkdir":
|
||||
mkdirs.append(f[2])
|
||||
elif f[0] == "keep":
|
||||
keeps.append(f[1])
|
||||
elif f[0] == "move":
|
||||
moves.append((f[1], f[2], f[3] if len(f) > 3 else ""))
|
||||
if not force:
|
||||
for p in mkdirs + keeps + [p for m in moves for p in m[:2]]:
|
||||
if p and not (p == root or p.startswith(root + os.sep)):
|
||||
raise Refuse(f"{p} is outside --root {root}"
|
||||
" (a plan built for another library?)")
|
||||
return mkdirs, keeps, moves
|
||||
|
||||
|
||||
# The media servers may read the library through the SSD overlay view
|
||||
# (scripts/media-view.sh). A relayout moves bytes on the raw array, so sidecars
|
||||
# the servers wrote through that view stay at the old coordinates — naming the
|
||||
# follow-up here is the point of use for that fact, but only where the view
|
||||
# exists: where it is not mounted the servers bind the array itself, the
|
||||
# sidecars moved with the media, and a reap order points at a directory that
|
||||
# is not there. See the Media Metadata View page for the host's state.
|
||||
METADATA_VIEW = "/var/lib/media-meta/view"
|
||||
METADATA_HINT = ("note: sidecars written through the metadata view did not follow"
|
||||
" these moves — next: scripts/media-view.sh reap")
|
||||
|
||||
|
||||
def metadata_note(args):
|
||||
"""Print the sidecar follow-up only when the overlay is really mounted."""
|
||||
if os.path.ismount(args.metadata_view):
|
||||
print(METADATA_HINT)
|
||||
|
||||
|
||||
def do_apply(args, root, journal, mkdirs, keeps, moves):
|
||||
"""Preflight everything, then folders, then renames. Never a partial start."""
|
||||
problems = []
|
||||
for p in keeps:
|
||||
if not os.path.exists(p):
|
||||
problems.append(f"missing: keep row source is gone: {p}")
|
||||
seen = {}
|
||||
for src, dst, _kind in moves:
|
||||
if src == dst:
|
||||
problems.append(f"no-op: move row whose target is its source: {src}")
|
||||
if dst in seen:
|
||||
problems.append(f"double-claim: {dst} claimed by {seen[dst]} and {src}")
|
||||
seen[dst] = src
|
||||
if journal.already(dst, src) and os.path.exists(dst) and not os.path.exists(src):
|
||||
continue # this intent already ran
|
||||
if not os.path.exists(src):
|
||||
if os.path.exists(dst):
|
||||
problems.append(f"missing: {src} is gone and {dst} carries no journal"
|
||||
" line — it moved outside this tool, refusing to guess")
|
||||
else:
|
||||
problems.append(f"missing: {src} is gone and {dst} is not there either")
|
||||
continue
|
||||
if os.path.exists(dst):
|
||||
problems.append(f"occupied: target already holds another file: {dst}")
|
||||
if problems:
|
||||
print(f"refusing: {len(problems)} problem(s), nothing moved")
|
||||
for p in sorted(problems)[:20]:
|
||||
print(" " + p)
|
||||
if len(problems) > 20:
|
||||
print(f" ... and {len(problems) - 20} more")
|
||||
return 2
|
||||
|
||||
if not args.apply:
|
||||
print(f"dry run: {len(moves)} moves, {len(mkdirs)} folders, {len(keeps)} files stay"
|
||||
" — nothing written (pass --apply to run)")
|
||||
for src, dst, _k in moves[:5]:
|
||||
print(f" would move {src} -> {dst}")
|
||||
return 0
|
||||
|
||||
made = {r.get("dst") for r in journal.records if r.get("op") == "mkdir"}
|
||||
dirs_new = 0
|
||||
for d in sorted(mkdirs, key=lambda q: q.count(os.sep)):
|
||||
if d in made or os.path.isdir(d):
|
||||
continue
|
||||
journal.record(op="mkdir", dst=d)
|
||||
os.makedirs(d, exist_ok=True)
|
||||
dirs_new += 1
|
||||
|
||||
# same-filesystem gate: rename cannot cross a device, and a silent copy of a
|
||||
# library this size is not an option anyone would choose.
|
||||
cross = []
|
||||
for src, dst, _k in moves:
|
||||
if journal.already(dst, src) and os.path.exists(dst) and not os.path.exists(src):
|
||||
continue
|
||||
parent = os.path.dirname(dst)
|
||||
if os.path.exists(parent) and os.stat(parent).st_dev != os.stat(src).st_dev:
|
||||
cross.append(dst)
|
||||
if cross:
|
||||
print(f"refusing: {len(cross)} moves would cross a filesystem"
|
||||
f" (first: {cross[0]}) — this tool only renames")
|
||||
return 2
|
||||
|
||||
moved = done = 0
|
||||
for src, dst, _kind in moves:
|
||||
if journal.already(dst, src):
|
||||
if os.path.exists(dst) and not os.path.exists(src):
|
||||
done += 1
|
||||
continue
|
||||
if os.path.exists(src) and not os.path.exists(dst):
|
||||
os.rename(src, dst) # the intent ran, the move did not
|
||||
moved += 1
|
||||
continue
|
||||
st = os.stat(src)
|
||||
journal.record(op="move", src=src, dst=dst, bytes=st.st_size, mode=st.st_mode,
|
||||
mtime=st.st_mtime, ino=st.st_ino, dev=st.st_dev)
|
||||
os.rename(src, dst)
|
||||
moved += 1
|
||||
print(f"applied: {moved} moved, {done} already done in an earlier run,"
|
||||
f" {dirs_new} folders created, {len(keeps)} files stayed put")
|
||||
print(f"journal: {journal.path} — undo with: media-apply.py rollback"
|
||||
f" --journal {journal.path} --root {root} --apply")
|
||||
metadata_note(args)
|
||||
return 0
|
||||
|
||||
|
||||
def do_verify(root, journal, moves, keeps):
|
||||
"""Every planned file sits at its target carrying the journalled bytes."""
|
||||
journalled = {(r["src"], r["dst"]): r for r in journal.records
|
||||
if r.get("op") == "move"}
|
||||
todo = moves or [(r["src"], r["dst"], "") for r in journal.records
|
||||
if r.get("op") == "move"]
|
||||
bad, missing, bytes_total = [], [], 0
|
||||
for src, dst, _kind in todo:
|
||||
rec = journalled.get((src, dst))
|
||||
if not os.path.exists(dst):
|
||||
missing.append(dst)
|
||||
continue
|
||||
size = os.path.getsize(dst)
|
||||
bytes_total += size
|
||||
if rec and rec.get("bytes") not in (None, size):
|
||||
bad.append(f"{dst}: {size} bytes on disk, {rec['bytes']} at move time")
|
||||
if os.path.exists(src) and src != dst:
|
||||
bad.append(f"{src}: still in place as well as at {dst}")
|
||||
missing += [p for p in keeps if not os.path.exists(p)]
|
||||
if bad or missing:
|
||||
print(f"verify FAILED: {len(missing)} absent, {len(bad)} wrong")
|
||||
for line in (missing + bad)[:20]:
|
||||
print(" " + line)
|
||||
return 2
|
||||
empty = [r["dst"] for r in journal.records
|
||||
if r.get("op") == "mkdir" and os.path.isdir(r["dst"])
|
||||
and not os.listdir(r["dst"])]
|
||||
print(f"verified: {len(todo)} files at their targets"
|
||||
f" ({bytes_total / 2**40:.2f} TiB), {len(keeps)} kept")
|
||||
if empty:
|
||||
print(f"note: {len(empty)} folders are empty (left in place — see the trash law)")
|
||||
return 0
|
||||
|
||||
|
||||
def sweep_targets(root, journal):
|
||||
"""The folders this run emptied, deepest first — never one it did not touch.
|
||||
|
||||
Eligibility is evidence, not appearance. A folder is a target only when the
|
||||
journal shows a file moved out of it or out of something beneath it, it holds
|
||||
nothing but empty directories now, it is not a folder this run created, and no
|
||||
journalled destination lives under it. A symlink anywhere inside disqualifies
|
||||
the whole subtree: this tool never follows a link, and deleting the folder
|
||||
that holds one would delete someone's mount point.
|
||||
|
||||
Proof carries downward: once a folder qualifies, every empty directory inside
|
||||
it qualifies too. A release's junk (``Screens/``, ``Sample/``) is never itself
|
||||
a move source, so without that rule it holds its emptied parent on disk
|
||||
forever — the parent can never be removed while the child stands.
|
||||
"""
|
||||
moves = [r for r in journal.records if r.get("op") == "move"]
|
||||
made = {r["dst"] for r in journal.records if r.get("op") == "mkdir"}
|
||||
# An rmdir INTENT that failed left its folder standing, and the next sweep
|
||||
# must be able to finish it: only a folder that is really gone is out of
|
||||
# scope. Counting every intent as done made 146 live survivors permanent.
|
||||
gone = {r["path"] for r in journal.records if r.get("op") == "rmdir"
|
||||
and not os.path.exists(r["path"])}
|
||||
dsts = [r["dst"] for r in moves]
|
||||
marked = set()
|
||||
for r in moves:
|
||||
d = os.path.dirname(r["src"])
|
||||
while d.startswith(root + os.sep):
|
||||
marked.add(d)
|
||||
d = os.path.dirname(d)
|
||||
|
||||
def removable(d):
|
||||
if d in made or d in gone or os.path.islink(d):
|
||||
return False
|
||||
if any(p == d or p.startswith(d + os.sep) for p in dsts):
|
||||
return False
|
||||
try:
|
||||
entries = os.listdir(d)
|
||||
except OSError:
|
||||
return False
|
||||
for e in entries:
|
||||
c = os.path.join(d, e)
|
||||
if os.path.islink(c) or not os.path.isdir(c) or not removable(c):
|
||||
return False
|
||||
return True
|
||||
|
||||
roots = [d for d in marked if removable(d)]
|
||||
targets = set(roots)
|
||||
for d in roots:
|
||||
for dirpath, dirnames, _ in os.walk(d):
|
||||
for c in dirnames:
|
||||
p = os.path.join(dirpath, c)
|
||||
if removable(p):
|
||||
targets.add(p)
|
||||
return sorted(targets, key=lambda q: (-q.count(os.sep), q))
|
||||
|
||||
|
||||
def do_sweep(args, root, journal):
|
||||
"""Remove the empty husks an apply left standing at the library root."""
|
||||
moves = [r for r in journal.records if r.get("op") == "move"]
|
||||
if not moves:
|
||||
raise Refuse("the journal holds no moves — a sweep follows an apply, it"
|
||||
" does not guess at emptiness")
|
||||
outside = sorted({p for r in moves for p in (r["src"], r["dst"])
|
||||
if not (p == root or p.startswith(root + os.sep))})
|
||||
if outside and not args.force:
|
||||
raise Refuse(f"{len(outside)} journalled paths are outside --root {root}"
|
||||
f" (first: {outside[0]}) — this journal belongs to another"
|
||||
" library")
|
||||
targets = sweep_targets(root, journal)
|
||||
if not targets:
|
||||
print("swept: 0 folders — nothing this run emptied is still standing empty")
|
||||
return 0
|
||||
if not args.apply:
|
||||
print(f"dry run: {len(targets)} folders this run emptied would be removed"
|
||||
" (pass --apply to run)")
|
||||
for d in targets[:5]:
|
||||
print(f" would remove {d}")
|
||||
return 0
|
||||
removed = 0
|
||||
for d in targets:
|
||||
journal.record(op="rmdir", path=d)
|
||||
try:
|
||||
os.rmdir(d)
|
||||
except OSError as e:
|
||||
print(f"left in place: {d} ({e})")
|
||||
continue
|
||||
removed += 1
|
||||
print(f"swept: {removed} folders removed, {len(targets) - removed} left in place")
|
||||
print(f"journal: {journal.path} — rollback recreates them with their files")
|
||||
return 0
|
||||
|
||||
|
||||
def do_rollback(args, root, journal):
|
||||
"""Undo every journalled move, newest first, after a full preflight."""
|
||||
moves = [r for r in journal.records if r.get("op") == "move"]
|
||||
made = [r["dst"] for r in journal.records if r.get("op") == "mkdir"]
|
||||
swept = [r["path"] for r in journal.records if r.get("op") == "rmdir"]
|
||||
problems = []
|
||||
for r in reversed(moves):
|
||||
src, dst = r["src"], r["dst"]
|
||||
if not args.force and not (src.startswith(root + os.sep)
|
||||
or dst.startswith(root + os.sep)):
|
||||
problems.append(f"outside --root: {dst}")
|
||||
continue
|
||||
if os.path.exists(dst) and os.path.exists(src):
|
||||
problems.append(f"occupied: both paths exist, refusing to choose:"
|
||||
f" {src} | {dst}")
|
||||
elif os.path.exists(dst):
|
||||
pass
|
||||
elif os.path.exists(src):
|
||||
continue # already undone
|
||||
else:
|
||||
problems.append(f"missing: {dst} is gone and {src} never came back")
|
||||
if problems:
|
||||
print(f"refusing: {len(problems)} problem(s), nothing moved back")
|
||||
for p in problems[:20]:
|
||||
print(" " + p)
|
||||
return 2
|
||||
if not args.apply:
|
||||
print(f"dry run: would move {len(moves)} files back to their census paths"
|
||||
f" and recreate {len(set(swept))} swept folders (pass --apply)")
|
||||
return 0
|
||||
undone = 0
|
||||
recreated = 0
|
||||
for d in sorted(set(swept), key=lambda q: q.count(os.sep)):
|
||||
if not os.path.isdir(d):
|
||||
os.makedirs(d, exist_ok=True)
|
||||
recreated += 1
|
||||
for r in reversed(moves):
|
||||
if os.path.exists(r["dst"]) and not os.path.exists(r["src"]):
|
||||
os.makedirs(os.path.dirname(r["src"]), exist_ok=True)
|
||||
os.rename(r["dst"], r["src"])
|
||||
undone += 1
|
||||
dirs_gone = 0
|
||||
for d in sorted(set(made), key=lambda q: -q.count(os.sep)):
|
||||
if d == root or not os.path.isdir(d) or os.listdir(d):
|
||||
continue
|
||||
try:
|
||||
os.rmdir(d)
|
||||
dirs_gone += 1
|
||||
except OSError as e:
|
||||
print(f"left in place: {d} ({e})")
|
||||
journal.record(op="rollback", undone=undone, dirs_removed=dirs_gone,
|
||||
t=datetime.datetime.now(datetime.timezone.utc).isoformat())
|
||||
print(f"rolled back: {undone} files back where the census found them,"
|
||||
f" {dirs_gone} empty folders removed, {recreated} swept folders recreated")
|
||||
metadata_note(args)
|
||||
return 0
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description="apply or undo a media relayout plan")
|
||||
ap.add_argument("cmd", choices=["apply", "rollback", "verify", "sweep"])
|
||||
ap.add_argument("--plan")
|
||||
ap.add_argument("--root", default=DEFAULT_ROOT)
|
||||
ap.add_argument("--journal")
|
||||
ap.add_argument("--apply", action="store_true", help="actually change the library")
|
||||
ap.add_argument("--force", action="store_true", help="relax the opinion guards")
|
||||
ap.add_argument("--metadata-view", default=METADATA_VIEW,
|
||||
help="overlay the media servers read through; the sidecar"
|
||||
" note prints only when this path is a mountpoint")
|
||||
args = ap.parse_args()
|
||||
|
||||
root = os.path.abspath(args.root)
|
||||
if not os.path.isdir(root):
|
||||
raise Refuse(f"no library at {root}")
|
||||
journal_path = args.journal or journal_default(root)
|
||||
journal = Journal(journal_path, args.apply)
|
||||
try:
|
||||
if args.cmd in ("rollback", "verify", "sweep") and not os.path.exists(journal_path):
|
||||
raise Refuse(f"no journal at {journal_path} — nothing to {args.cmd}")
|
||||
if args.cmd in ("rollback", "sweep") and not args.force and journal.rolled_back:
|
||||
raise Refuse("this journal has already been rolled back"
|
||||
" (--force to replay it)")
|
||||
if args.cmd in ("apply", "verify") and not args.plan:
|
||||
raise Refuse(f"{args.cmd} needs --plan")
|
||||
mkdirs, keeps, moves = (read_plan(args.plan, root, args.force)
|
||||
if args.plan else ([], [], []))
|
||||
if args.cmd == "apply":
|
||||
return do_apply(args, root, journal, mkdirs, keeps, moves)
|
||||
if args.cmd == "verify":
|
||||
return do_verify(root, journal, moves, keeps)
|
||||
if args.cmd == "sweep":
|
||||
return do_sweep(args, root, journal)
|
||||
return do_rollback(args, root, journal)
|
||||
finally:
|
||||
journal.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except Refuse as r:
|
||||
print(f"refusing: {r}")
|
||||
sys.exit(2)
|
||||
Executable
+425
@@ -0,0 +1,425 @@
|
||||
#!/usr/bin/env python3
|
||||
r"""media-audiobooks.py — a plan that re-files the audiobook library into one folder per book.
|
||||
|
||||
media-audiobooks.py --root DIR --enum TSV --rows TSV [--out TSV] [--exceptions FILE]
|
||||
|
||||
Plex reads this library and Jellyfin does not: its music scanner has no audiobook
|
||||
concept, and the tree is indexed entirely by Plex, which matched 58,975 of the
|
||||
62,879 files here into its Artist/Album/Track shape. So the authority is
|
||||
`scripts/media-census.py audiobooks` — the BOOK (album row), its NARRATOR (that
|
||||
book's single artist row) and its CHAPTERS (track rows) — and never the folder a
|
||||
release pack made. The plan is:
|
||||
|
||||
Audiobooks/<Narrator>/<Book> (<year>)/<the file's own name>.<ext>
|
||||
... the same folder, " (2)" on the name, when two chapters
|
||||
of one book carry the same name
|
||||
|
||||
What that buys is the one property both media servers' book scanners assume: a
|
||||
folder holds exactly one book. Today 212 top-level folders are a mix of
|
||||
author-named shelves and collection packs, and the books inside the packs are
|
||||
split across `… [Part 1]`, `… [Part 2]` folders three or four deep.
|
||||
|
||||
Three laws here are the opposite of what a music shelf does, and each is forced
|
||||
by what the rows actually hold:
|
||||
|
||||
* A chapter is NEVER renumbered. 7,556 of the 10,485 books contain at least one
|
||||
chapter whose index Plex does not know, so a number written into a name would
|
||||
be invented — and it would buy nothing, because both servers order playback by
|
||||
the embedded track tag, not by the file name. A wrong number in a name is worse
|
||||
than none: it is the number a human later trusts. A file already carrying its
|
||||
own number keeps it, untouched.
|
||||
* The shelf is the NARRATOR, not the author. Every book has exactly one artist row
|
||||
and it is the narrator; the author survives only inside 10% of the album titles
|
||||
(`Kryptos by Graham McNeill`) and in no column Plex stores, so deriving the
|
||||
author would mean parsing titles for one book in ten and inventing it for the
|
||||
rest. One rule that always resolves beats a shelf that is half author and half
|
||||
narrator; the author stays visible in the book folder's own name, which is the
|
||||
title Plex shows.
|
||||
* Nothing is guessed at. A chapter row with no book or no narrator parks — a
|
||||
tag-scan placeholder (`[Unknown Album]`) names neither, since unrelated books
|
||||
share it and filing on one merges them, and so does a chapter number the ripper
|
||||
mistagged as the artist (`0.4`). An audio file Plex never matched parks, a
|
||||
video container parks, and a loose file in a folder holding two books parks.
|
||||
|
||||
Nothing is deleted and nothing is overwritten. The output is a list of rename(2)
|
||||
intents for `scripts/media-apply.py`, whose preflight refuses any target that is
|
||||
already claimed and whose journal is the rollback.
|
||||
"""
|
||||
import argparse
|
||||
import csv
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from collections import Counter, defaultdict
|
||||
|
||||
# What a chapter file can be. .m4b is the audiobook container; the rest are the
|
||||
# audio a pack ships when it was ripped as music.
|
||||
AUDIO_EXT = {".mp3", ".m4a", ".m4b", ".ogg", ".oga", ".opus", ".wav", ".flac",
|
||||
".wma", ".aiff", ".aif", ".m4v", ".aac", ".mka"}
|
||||
# Never an audiobook, whatever the library it sits in says: Plex reads these as
|
||||
# video, and renaming one into an audio extension would claim it is a chapter.
|
||||
VIDEO_EXT = {".mkv", ".mp4", ".webm", ".avi", ".mov", ".mpg", ".mpeg", ".ts",
|
||||
".m2ts", ".vob", ".ifo", ".wmv", ".iso", ".img", ".bdmv", ".ogm",
|
||||
".divx"}
|
||||
|
||||
# Tag-scan placeholders. These strings are not identities: 1,106 chapters from
|
||||
# 14 unrelated top-level folders carry the album `[Unknown Album]`, so filing on
|
||||
# them would merge books that have nothing to do with each other — the exact
|
||||
# shape this relayout exists to make impossible. Such a chapter parks with the
|
||||
# path it has today and the fix belongs in its tags. `Various Artists` is NOT a
|
||||
# placeholder: that is what a multi-cast read carries, and the one book here
|
||||
# with 1,189 chapters under it is genuinely one book.
|
||||
PLACEHOLDER_BOOK = {"[unknown album]", "unknown album", "no title", "untitled",
|
||||
"[untitled]", "unknown title", "n/a", "-"}
|
||||
PLACEHOLDER_NARRATOR = {"[unknown artist]", "unknown artist", "unknown author",
|
||||
"<unknown>", "[unknown]", "unknown"}
|
||||
|
||||
|
||||
def identity(value, placeholders, needs_letter=False):
|
||||
"""The tag value Plex matched, when it names something; '' when it does not.
|
||||
|
||||
A trailing parenthesized group is stripped for the test only, because rippers
|
||||
glue timestamps to the placeholder (`Unknown album (03/03/2015 21:39:55)`).
|
||||
|
||||
`needs_letter` is for a NARRATOR alone: a shelf is a person's name, and 272
|
||||
chapters here carry a chapter number in the artist tag (`0.4`, `31`). The same
|
||||
test must never judge a BOOK title, which may honestly be `1984`.
|
||||
"""
|
||||
v = (value or "").strip()
|
||||
bare = re.sub(r"\s*\([^()]*\)\s*$", "", v).strip().casefold()
|
||||
if bare in placeholders or (needs_letter and not any(c.isalpha() for c in v)):
|
||||
return ""
|
||||
return v
|
||||
|
||||
|
||||
# The characters Plex documents as breaking playback. They are dropped, not
|
||||
# substituted: a replacement character is a guess at what the title meant.
|
||||
BAD_CHARS = set('<>:"/\\|?*')
|
||||
TRASH_DIR = re.compile(r"(^|/)(\.AGENTTRASH|\.thumbnails|\.Trash-\d+|@eaDir"
|
||||
r"|lost\+found)(/|$)")
|
||||
|
||||
ARGS = None
|
||||
HELD = {} # path -> (bytes, ext)
|
||||
ROWS = {} # path -> the chapter row Plex matched
|
||||
PLACED = {} # chapter path -> the destination the plan gave it
|
||||
CLAIMED = set() # every destination path the plan has handed out
|
||||
MOVES, KEEPS = [], []
|
||||
PARKED = defaultdict(list) # reason -> [path]
|
||||
NOTES = defaultdict(list) # section -> [line]
|
||||
AUDIO_BY_DIR = defaultdict(set) # source dir -> chapter files a row names
|
||||
CHILD_OF = defaultdict(set) # dir -> the dirs directly below it holding chapters
|
||||
FOLDERS_SEEN = {} # (dir, depth) -> book folders placed below it
|
||||
SUBTREE_DEPTH = 2 # how far below a folder its chapters may sit
|
||||
BOOKS = {} # (narrator, book, year) -> the book group
|
||||
|
||||
AMBIGUOUS = object()
|
||||
|
||||
|
||||
def dominant(values):
|
||||
"""The spelling the rows mostly agree on: the most frequent value, ties broken
|
||||
by sort so the answer never depends on the order rows arrived in."""
|
||||
counts = Counter(values)
|
||||
return min(counts.items(), key=lambda kv: (-kv[1], kv[0]))[0]
|
||||
|
||||
|
||||
def rel(path):
|
||||
"""The path as seen from the library root."""
|
||||
return os.path.relpath(path, ARGS.root)
|
||||
|
||||
|
||||
def safe(name):
|
||||
"""A name every client can carry: no reserved character, nothing unprintable,
|
||||
no trailing dot or space, and short enough to rename."""
|
||||
out = "".join(c for c in (name or "")
|
||||
if c not in BAD_CHARS and (c.isprintable() or c == " "))
|
||||
out = re.sub(r"\s+", " ", out).strip().rstrip(". ")
|
||||
while out and len(out.encode("utf-8", "ignore")) > 200:
|
||||
out = out[:-4].rstrip()
|
||||
return out
|
||||
|
||||
|
||||
def assign(src, folder, name, ext, kind, note):
|
||||
"""Book one file into `folder` under `name`, renamed if that exact destination
|
||||
is already taken — by another file in the plan, or by a file standing there
|
||||
today. Two files claiming one path is the fault that costs a re-run of a 60k
|
||||
plan, and so is a plan that only works if the renames happen in the right
|
||||
order: the executor refuses any target that is occupied, so an occupied path
|
||||
is never a destination, even when its occupant is scheduled to leave.
|
||||
A file already standing where the contract puts it is a keep, not a move."""
|
||||
stem = safe(name) or "file"
|
||||
dst = os.path.join(folder, stem + ext)
|
||||
if dst == src:
|
||||
CLAIMED.add(dst.lower()) # it is standing on its own target
|
||||
KEEPS.append((src, dst, kind))
|
||||
return dst
|
||||
# A file already in this folder under this name — or under the numbered
|
||||
# variant the collision rule gives it — keeps the name it has. Renumbering it
|
||||
# every run would turn the plan into a shuffle instead of a fixed point.
|
||||
base = os.path.splitext(os.path.basename(src))[0]
|
||||
if os.path.dirname(src) == folder and (base == stem
|
||||
or base.startswith(stem + " (")):
|
||||
CLAIMED.add(src.lower())
|
||||
KEEPS.append((src, src, kind))
|
||||
return src
|
||||
n = 1
|
||||
while dst.lower() in CLAIMED:
|
||||
n += 1
|
||||
dst = os.path.join(folder, f"{stem} ({n}){ext}")
|
||||
if n > 1:
|
||||
NOTES[note].append(rel(dst))
|
||||
CLAIMED.add(dst.lower())
|
||||
MOVES.append((src, dst, kind))
|
||||
return dst
|
||||
|
||||
|
||||
def park(path, reason):
|
||||
"""Hold a file where it is, under the park folder, keeping its relative path."""
|
||||
PARKED[reason].append(path)
|
||||
dst = os.path.join(ARGS.park_dir, rel(path))
|
||||
if dst != path:
|
||||
CLAIMED.add(dst.lower())
|
||||
MOVES.append((path, dst, "park"))
|
||||
|
||||
|
||||
def under_park(path):
|
||||
"""True for a file already parked: that pile is for a human to review, and
|
||||
re-parking it every run would only push it deeper."""
|
||||
return path.startswith(ARGS.park_dir + os.sep)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------ books
|
||||
def group_books():
|
||||
"""Every matched chapter into one group per (narrator, book, year).
|
||||
|
||||
The key is the identity Plex asserts in its own database, not the folder the
|
||||
file sits in: one book arrives as four part folders inside a collection pack,
|
||||
and two copies of one book in one pack are one book to Plex, whose album item
|
||||
IS that narrator, title and year. The year is part of the key so a 1965
|
||||
recording and a 2020 recording that share a title stay two books.
|
||||
"""
|
||||
for path, r in sorted(ROWS.items()):
|
||||
if path not in HELD or under_park(path) or TRASH_DIR.search(path):
|
||||
continue
|
||||
d = os.path.dirname(path)
|
||||
AUDIO_BY_DIR[d].add(path)
|
||||
CHILD_OF[os.path.dirname(d)].add(d) # so a cover above Part 1/Part 2 sees both
|
||||
narrator = identity(r["narrator"], PLACEHOLDER_NARRATOR, needs_letter=True)
|
||||
book = identity(r["book"], PLACEHOLDER_BOOK)
|
||||
if not book:
|
||||
park(path, "a chapter row that names no book — Plex's album title is a "
|
||||
"placeholder like [Unknown Album], so filing on it would "
|
||||
"merge unrelated books onto one shelf")
|
||||
continue
|
||||
if not narrator:
|
||||
park(path, "a chapter row that names no narrator — the artist row is a "
|
||||
"placeholder like [Unknown Artist] or a chapter number a "
|
||||
"ripper mistagged as the artist, so no shelf owns it")
|
||||
continue
|
||||
key = (narrator.casefold(), book.casefold(), (r["year"] or "").strip())
|
||||
g = BOOKS.setdefault(key, {"narrators": [], "books": [], "year": key[2],
|
||||
"members": [], "indexed": 0})
|
||||
g["members"].append(path)
|
||||
g["narrators"].append(narrator)
|
||||
g["books"].append(book)
|
||||
if str(r["track"] or "").isdigit():
|
||||
g["indexed"] += 1
|
||||
|
||||
|
||||
def book_folder(g):
|
||||
"""The shelf Plex's own rows name: <narrator>/<book> (<year>). None when a
|
||||
name is made of characters no path can hold — inventing a shelf is filing it
|
||||
wrong, so those chapters park instead."""
|
||||
narrator = safe(dominant(g["narrators"]))
|
||||
book = safe(dominant(g["books"]))
|
||||
if not narrator or not book:
|
||||
return None
|
||||
return os.path.join(ARGS.root, narrator,
|
||||
f"{book} ({g['year']})" if g["year"] else book)
|
||||
|
||||
|
||||
def place_books():
|
||||
"""One folder per book; every chapter keeps the name it has today."""
|
||||
for key in sorted(BOOKS):
|
||||
g = BOOKS[key]
|
||||
folder = book_folder(g)
|
||||
if folder is None:
|
||||
for path in sorted(g["members"]):
|
||||
park(path, "a book or narrator Plex spells with characters no "
|
||||
"folder name can hold")
|
||||
continue
|
||||
for path in sorted(g["members"]):
|
||||
ext = HELD[path][1]
|
||||
base = os.path.splitext(os.path.basename(path))[0]
|
||||
# The name the ripper wrote, unchanged: the track tag inside the file
|
||||
# is what orders playback, and 72% of the books here have chapters no
|
||||
# row numbers, so any number added here would be invented.
|
||||
PLACED[path] = assign(path, folder, base, ext, "chapter",
|
||||
"two chapters of one book carried the same name")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------- sidecars
|
||||
def folders_below(d, depth):
|
||||
"""The book folders the chapters in `d`, or within `depth` directories below
|
||||
it, were placed in. Memoised, because every loose file in a big collection
|
||||
folder asks the same question about it."""
|
||||
if (d, depth) not in FOLDERS_SEEN:
|
||||
seen = {os.path.dirname(PLACED[a]) for a in AUDIO_BY_DIR.get(d, ())
|
||||
if a in PLACED}
|
||||
if depth:
|
||||
for child in CHILD_OF.get(d, ()):
|
||||
seen |= folders_below(child, depth - 1)
|
||||
FOLDERS_SEEN[(d, depth)] = seen
|
||||
return FOLDERS_SEEN[(d, depth)]
|
||||
|
||||
|
||||
def owner_folder(d):
|
||||
"""The book folder that owns this directory — the first place up the tree
|
||||
where chapters sit, in the directory itself or within SUBTREE_DEPTH levels
|
||||
below it, which is how a pack whose parts are `… [Part 1]`/`… [Part 2]` owns
|
||||
the cover art lying above them. None when no chapter is near enough to say,
|
||||
AMBIGUOUS when the chapters there belong to more than one book."""
|
||||
while d.startswith(ARGS.root + os.sep) and d != ARGS.root:
|
||||
folders = folders_below(d, SUBTREE_DEPTH)
|
||||
if folders:
|
||||
return next(iter(folders)) if len(folders) == 1 else AMBIGUOUS
|
||||
d = os.path.dirname(d)
|
||||
return None
|
||||
|
||||
|
||||
def place_other():
|
||||
"""Everything that is not a chapter Plex matched: covers, booklets, EPUBs,
|
||||
logs, checksums, playlists — and the clutter that is none of those."""
|
||||
for path, (_size, ext) in sorted(HELD.items()):
|
||||
if path in ROWS or under_park(path) or TRASH_DIR.search(path):
|
||||
continue
|
||||
d = os.path.dirname(path)
|
||||
base = os.path.splitext(os.path.basename(path))[0]
|
||||
if ext in VIDEO_EXT:
|
||||
park(path, "a video container inside the audiobook library: Plex reads "
|
||||
"it as video, and renaming it would claim it is a chapter")
|
||||
elif ext in AUDIO_EXT:
|
||||
park(path, "an audio file no chapter row names — it is not in the "
|
||||
"library Plex matched, so nothing says which book it is")
|
||||
else:
|
||||
owner = owner_folder(d)
|
||||
if owner is AMBIGUOUS:
|
||||
park(path, "a file whose folder holds more than one book, so "
|
||||
"nothing says which one it belongs to")
|
||||
elif owner:
|
||||
assign(path, owner, base, ext,
|
||||
"playlist" if ext in (".m3u", ".m3u8") else "sidecar",
|
||||
"one book folder would have carried two files of the same"
|
||||
" sidecar name")
|
||||
else:
|
||||
park(path, "not an audiobook file, and no book's folder lies with"
|
||||
" it to carry it")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------- outputs
|
||||
def write_plan():
|
||||
"""mkdir rows for every destination folder and its whole chain, then the moves.
|
||||
Parents first: the executor creates directories in the order they are listed."""
|
||||
chain = set()
|
||||
for _s, dst, _k in MOVES:
|
||||
d = os.path.dirname(dst)
|
||||
while d and d != ARGS.root and d != os.path.dirname(d):
|
||||
chain.add(d)
|
||||
d = os.path.dirname(d)
|
||||
dirs = sorted(chain, key=lambda p: (p.count("/"), p))
|
||||
with open(ARGS.out, "w") as fh:
|
||||
fh.write("action\tsrc\tdst\tkind\n")
|
||||
for d in dirs:
|
||||
fh.write(f"mkdir\t\t{d}\tdir\n")
|
||||
for src, dst, kind in sorted(MOVES, key=lambda m: (m[1], m[0])):
|
||||
fh.write(f"move\t{src}\t{dst}\t{kind}\n")
|
||||
for src, dst, kind in sorted(KEEPS, key=lambda m: (m[1], m[0])):
|
||||
fh.write(f"keep\t{src}\t{dst}\t{kind}\n")
|
||||
parked = [m for m in MOVES if m[2] == "park"]
|
||||
sized = sum(HELD[s][0] for s, _d, _k in MOVES if s in HELD)
|
||||
print(f"plan: {len(MOVES) - len(parked)} moves, {len(parked)} parked"
|
||||
f" ({sized / 2**40:.2f} TiB touched), {len(KEEPS)} already placed,"
|
||||
f" {len(dirs)} folders -> {ARGS.out}")
|
||||
|
||||
|
||||
def write_exceptions():
|
||||
with open(ARGS.exceptions, "w") as fh:
|
||||
fh.write(f"Audiobook relayout exceptions for {ARGS.root}\n\n")
|
||||
fh.write(f"Everything below is parked under {ARGS.park_dir}, keeping the\n"
|
||||
"relative path it has today. Nothing is deleted and nothing is\n"
|
||||
"overwritten: a parked file is one rename from wherever the plan is\n"
|
||||
"pointed next.\n\n")
|
||||
for reason in sorted(PARKED, key=lambda r: -len(PARKED[r])):
|
||||
rs = PARKED[reason]
|
||||
fh.write(f"{len(rs):>6} files {reason}\n")
|
||||
for p in sorted(rs, key=rel)[:400]:
|
||||
fh.write(f" {rel(p)}\n")
|
||||
if len(rs) > 400:
|
||||
fh.write(f" … and {len(rs) - 400} more\n")
|
||||
fh.write("\n")
|
||||
for section in sorted(NOTES, key=lambda s: -len(NOTES[s])):
|
||||
lines = sorted(set(NOTES[section]))
|
||||
fh.write(f"{len(lines):>6} {section}\n")
|
||||
for line in lines[:200]:
|
||||
fh.write(f" {line}\n")
|
||||
if len(lines) > 200:
|
||||
fh.write(f" … and {len(lines) - 200} more\n")
|
||||
fh.write("\n")
|
||||
unindexed = sorted(f"{safe(dominant(g['narrators']))}/{safe(dominant(g['books']))}"
|
||||
for g in BOOKS.values()
|
||||
if 0 < g["indexed"] < len(g["members"]))
|
||||
if unindexed:
|
||||
fh.write(f"{len(unindexed):>6} books have chapters Plex does not number. Their\n")
|
||||
fh.write(" files keep the names they carry: playback order comes from the\n"
|
||||
" track tag inside each file, and a number written here would be\n"
|
||||
" invented for the chapters with no index. Fix it in the tags, not\n"
|
||||
" in the names — Plex re-reads tags on a scan and never names files.\n\n")
|
||||
fh.write(" Playlists move with their book and their contents are never\n"
|
||||
" rewritten: the paths inside an .m3u go stale.\n")
|
||||
print(f"exceptions -> {ARGS.exceptions}")
|
||||
|
||||
|
||||
def main():
|
||||
global ARGS
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--root", default=os.environ.get("MEDIA_AUDIOBOOKS_ROOT",
|
||||
"/mnt/entropy/Audiobooks"))
|
||||
ap.add_argument("--enum", default="/tmp/audiobooks-enum.tsv",
|
||||
help="the enumerate TSV (bytes, ext, path)")
|
||||
ap.add_argument("--rows", default="/tmp/plex-audiobooks.tsv",
|
||||
help="the chapter rows scripts/media-census.py audiobooks wrote")
|
||||
ap.add_argument("--out", default="/tmp/audiobooks-plan.tsv")
|
||||
ap.add_argument("--exceptions", default="/tmp/audiobooks-exceptions.txt")
|
||||
ap.add_argument("--park", default="_unsorted",
|
||||
help="the folder inside the root that holds what cannot be filed")
|
||||
ARGS = ap.parse_args()
|
||||
ARGS.root = os.path.normpath(ARGS.root)
|
||||
ARGS.park_dir = os.path.join(ARGS.root, ARGS.park)
|
||||
read_inputs()
|
||||
# Every path a file occupies today is claimed before a single destination is
|
||||
# handed out: the executor refuses an occupied target, so a plan that aimed at
|
||||
# one would only fail after the renames that were supposed to clear it.
|
||||
CLAIMED.update(p.lower() for p in HELD)
|
||||
group_books()
|
||||
place_books()
|
||||
place_other()
|
||||
write_plan()
|
||||
write_exceptions()
|
||||
|
||||
|
||||
def read_inputs():
|
||||
with open(ARGS.enum) as fh:
|
||||
for line in fh:
|
||||
parts = line.rstrip("\n").split("\t")
|
||||
if len(parts) >= 3:
|
||||
HELD[parts[2]] = (int(parts[0]), parts[1].lower())
|
||||
with open(ARGS.rows, newline="") as fh:
|
||||
for row in csv.DictReader(fh, delimiter="\t"):
|
||||
if row.get("file"):
|
||||
ROWS[row["file"]] = row
|
||||
gone = [p for p in ROWS if p not in HELD]
|
||||
if gone:
|
||||
NOTES["a chapter row whose file the census no longer sees"].extend(gone)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+526
@@ -0,0 +1,526 @@
|
||||
#!/usr/bin/env python3
|
||||
r"""Read-only census of a media library — the inputs a relayout plan is built from.
|
||||
|
||||
media-census.py enumerate [--root DIR] [--out TSV]
|
||||
media-census.py probe [--enum TSV] [--out TSV] [--jobs N]
|
||||
media-census.py plex [--root DIR] [--db PATH] [--out TSV]
|
||||
media-census.py jellyfin [--root DIR] [--db PATH] [--container PREFIX] [--out TSV]
|
||||
media-census.py audiobooks [--root DIR] [--db PATH] [--out TSV]
|
||||
|
||||
Run it on the machine that mounts the library. Nothing here writes inside the
|
||||
library and nothing is deleted; the only outputs are the TSVs below, and the
|
||||
scripts/media-plan.py and scripts/media-shows.py read them instead of touching media.
|
||||
|
||||
enum bytes <TAB> ext <TAB> path every file under the root
|
||||
probed bytes <TAB> path <TAB> dur <TAB> vcodec <TAB> WxH <TAB> fps <TAB> vbit
|
||||
<TAB> pix <TAB> audio <TAB> subs <TAB> spare <TAB> format
|
||||
one row per video-container file; `audio` is "codec:lang+codec:lang",
|
||||
`subs` the subtitle-stream count, `spare` the audio streams carrying
|
||||
neither a default nor a forced tag
|
||||
plex file <TAB> leaf_title <TAB> … (see PLEX_COLS)
|
||||
the titles Plex already matched — the authority over any filename guess
|
||||
|
||||
jellyfin file <TAB> series <TAB> year <TAB> season <TAB> episode <TAB> extra
|
||||
<TAB> tmdb <TAB> title <TAB> tvdb (see JF_COLS)
|
||||
one row per episode Jellyfin has matched and numbered, its path mapped
|
||||
from the container's view back to the host; `year`, `tmdb` and `tvdb`
|
||||
are the SERIES' values, `title` is the episode title the row carries,
|
||||
`extra` is Jellyfin's own extra type when the row is bonus material
|
||||
rather than an episode
|
||||
|
||||
music file <TAB> artist <TAB> album <TAB> year <TAB> disc <TAB> track <TAB> title
|
||||
<TAB> mb <TAB> performer (see MUSIC_COLS)
|
||||
one row per track Jellyfin has tagged: the tag read is what the row IS,
|
||||
so `artist` is the album artist (the track artist when that is empty) and
|
||||
`album`/`year` are the album item's provider values when the folder is an
|
||||
album item and the tags' otherwise; `mb` is the album's MusicBrainz
|
||||
release group, `performer` the track artist the tags carry
|
||||
|
||||
audiobooks file <TAB> narrator <TAB> book <TAB> year <TAB> track <TAB> chapter
|
||||
(see AB_COLS)
|
||||
one row per chapter Plex matched under the audiobook root: `book` is the
|
||||
album row Plex built from the files' own tags, `narrator` that book's
|
||||
artist row — Plex keeps exactly one artist per book, which is why it, and
|
||||
not an author parsed out of a title, names the shelf. `track` is empty
|
||||
wherever Plex has no chapter number, which is most of this library.
|
||||
|
||||
`probe` is the slow step (every container is opened for its header, ~10 min for
|
||||
4,800 files on a spindle array at 6 workers) so it is `nice`d: a census beside a
|
||||
live service must not starve it. `enumerate` is a stat pass, so re-censusing
|
||||
before a plan is cheap — and a plan is only ever as good as these files.
|
||||
"""
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import sys
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
# Containers ffprobe can read. .iso/.img are deliberately NOT here: a disc image is
|
||||
# a mounted volume, not a stream, and the planner treats it as a movie file instead.
|
||||
PROBE_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
|
||||
".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".bdmv", ".vob"}
|
||||
# Trash and thumbnail caches are not library content, and this repo's trash policy
|
||||
# means .AGENTTRASH can sit inside a movie folder at any time.
|
||||
SKIP_DIR = re.compile(r"(^|/)(\.AGENTTRASH|\.thumbnails|\.Trash-\d+|@eaDir|lost\+found)(/|$)")
|
||||
PLEX_DB = ("/var/lib/plexmediaserver/Library/Application Support/Plex Media Server"
|
||||
"/Plug-in Support/Databases/com.plexapp.plugins.library.db")
|
||||
PLEX_COLS = ["file", "leaf_title", "leaf_orig", "leaf_edition", "leaf_year",
|
||||
"parent_title", "parent_orig", "parent_year", "parent_edition",
|
||||
"container", "vcodec", "acodec", "width", "height"]
|
||||
PLEX_SQL = """
|
||||
select p.file, coalesce(mi.title,''), coalesce(mi.original_title,''),
|
||||
coalesce(mi.edition_title,''), coalesce(mi.year,0),
|
||||
coalesce(par.title,''), coalesce(par.original_title,''),
|
||||
coalesce(par.year,0), coalesce(par.edition_title,''),
|
||||
m.container, m.video_codec, m.audio_codec, m.width, m.height
|
||||
from media_parts p
|
||||
join media_items m on m.id = p.media_item_id
|
||||
join metadata_items mi on mi.id = m.metadata_item_id
|
||||
left join metadata_items par on par.id = mi.parent_id
|
||||
where p.file like ? and p.deleted_at is null and mi.deleted_at is null
|
||||
"""
|
||||
JF_DB = "/home/dank/Containers/jellyfin/config/data/jellyfin.db"
|
||||
JF_COLS = ["file", "series", "year", "season", "episode", "extra",
|
||||
"tmdb", "title", "tvdb"]
|
||||
# Jellyfin stores the entity's .NET class as its type; the TV tree is these three.
|
||||
JF_TV = ("MediaBrowser.Controller.Entities.TV.Episode",
|
||||
"MediaBrowser.Controller.Entities.TV.Season",
|
||||
"MediaBrowser.Controller.Entities.TV.Series")
|
||||
JF_SQL = ("""select Id, Path, IndexNumber, ParentIndexNumber, ExtraType,
|
||||
IsVirtualItem, Name from BaseItems where Type = ? and Path like ?""")
|
||||
|
||||
# The music tree is these two entity classes. The track row carries the tags
|
||||
# Jellyfin read out of the file; the album row carries what the provider matched,
|
||||
# which is the better folder name of the two.
|
||||
JF_AUDIO = "MediaBrowser.Controller.Entities.Audio.Audio"
|
||||
JF_ALBUM = "MediaBrowser.Controller.Entities.Audio.MusicAlbum"
|
||||
MUSIC_COLS = ["file", "artist", "album", "year", "disc", "track", "title", "mb",
|
||||
"performer"]
|
||||
MUSIC_SQL = ("""select b.Path, b.AlbumArtists, b.Artists, b.Album, b.ProductionYear,
|
||||
b.ParentIndexNumber, b.IndexNumber, b.Name, b.Id,
|
||||
a.Name, a.ProductionYear, a.Id
|
||||
from BaseItems b
|
||||
left join BaseItems a on a.Id = b.ParentId and a.Type = ?
|
||||
where b.Type = ? and b.Path like ?""")
|
||||
# The album's identity, in preference order. A release group is the album; a bare
|
||||
# release id is one reissue of it.
|
||||
MUSIC_PROVIDERS = ("MusicBrainzReleaseGroup", "MusicBrainzAlbum")
|
||||
|
||||
# Plex's audiobook tree is the same Artist/Album/Track shape as its music tree:
|
||||
# the leaf (10) is a CHAPTER, its parent (9) is the BOOK, and that book's artist
|
||||
# (8) is the NARRATOR. The track's own `index` is exported because the exceptions
|
||||
# report counts how many books have chapters Plex does not number — the reason the
|
||||
# relayout refuses to invent numbers into file names.
|
||||
PLEX_ARTIST, PLEX_ALBUM, PLEX_TRACK = 8, 9, 10
|
||||
AB_COLS = ["file", "narrator", "book", "year", "track", "chapter"]
|
||||
AB_SQL = f"""
|
||||
select p.file, coalesce(ar.title,''), coalesce(al.title,''), coalesce(al.year,0),
|
||||
coalesce(mi."index",-1), coalesce(mi.title,'')
|
||||
from media_parts p
|
||||
join media_items m on m.id = p.media_item_id
|
||||
join metadata_items mi on mi.id = m.metadata_item_id
|
||||
join metadata_items al on al.id = mi.parent_id and al.metadata_type = {PLEX_ALBUM}
|
||||
left join metadata_items ar on ar.id = al.parent_id
|
||||
and ar.metadata_type = {PLEX_ARTIST}
|
||||
where p.file like ? and p.deleted_at is null and mi.deleted_at is null
|
||||
and al.deleted_at is null
|
||||
"""
|
||||
|
||||
def open_plex_ro(db):
|
||||
"""Plex's library, read the way Plex writes it.
|
||||
|
||||
`immutable=1` tells sqlite the file never changes, so it skips the
|
||||
write-ahead log: every row a scan has committed but not yet checkpointed
|
||||
is invisible, and reading a live database that way can fail outright as
|
||||
"database disk image is malformed". Plain ro reads the WAL through its
|
||||
-shm; immutable is only the fallback for a database whose -shm cannot be
|
||||
read. The same law scripts/media-verify.py states.
|
||||
"""
|
||||
last = None
|
||||
for opt in ("mode=ro", "mode=ro&immutable=1"):
|
||||
try:
|
||||
return sqlite3.connect(f"file:{db}?{opt}", uri=True)
|
||||
except sqlite3.Error as e:
|
||||
last = e
|
||||
sys.exit(f"cannot open the Plex database read-only ({db}): {last}\n"
|
||||
"Plex runs rootless here: run this as the container's owner, never "
|
||||
"with sudo — rootless storage is per-user, so root sees nothing.")
|
||||
|
||||
|
||||
def enumerate_lib(root, out):
|
||||
"""One row per real file under root: size, extension, path."""
|
||||
rows = []
|
||||
for dirpath, dirnames, filenames in os.walk(root):
|
||||
dirnames[:] = [d for d in dirnames
|
||||
if not SKIP_DIR.search(os.path.join(dirpath, d))]
|
||||
for fn in filenames:
|
||||
p = os.path.join(dirpath, fn)
|
||||
if SKIP_DIR.search(p):
|
||||
continue
|
||||
try:
|
||||
st = os.lstat(p)
|
||||
except OSError:
|
||||
continue
|
||||
if not os.path.isfile(p) or os.path.islink(p):
|
||||
continue # links and sockets are not library content
|
||||
rows.append((st.st_size, os.path.splitext(fn)[1].lower(), p))
|
||||
with open(out, "w") as fh:
|
||||
for size, ext, p in rows:
|
||||
fh.write(f"{size}\t{ext}\t{p}\n")
|
||||
vid = [r for r in rows if r[1] in PROBE_EXT]
|
||||
print(f"files={len(rows)} bytes={sum(r[0] for r in rows)} "
|
||||
f"probes={len(vid)} probe_bytes={sum(r[0] for r in vid)} -> {out}")
|
||||
|
||||
|
||||
def probe_one(path):
|
||||
"""ffprobe header read -> the probed TSV row (never the whole file)."""
|
||||
try:
|
||||
os.nice(19) # a census must never outrank the media server
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
size = os.stat(path).st_size
|
||||
except OSError:
|
||||
return None
|
||||
cmd = ["ffprobe", "-v", "error", "-print_format", "json",
|
||||
"-show_format", "-show_streams", path]
|
||||
try:
|
||||
done = subprocess.run(cmd, capture_output=True, text=True, timeout=180)
|
||||
data = json.loads(done.stdout or "{}")
|
||||
except (subprocess.TimeoutExpired, subprocess.SubprocessError,
|
||||
json.JSONDecodeError, OSError):
|
||||
data = {}
|
||||
st = data.get("streams", [])
|
||||
fmt = data.get("format", {})
|
||||
v = next((s for s in st if s.get("codec_type") == "video"), None)
|
||||
aud = [s for s in st if s.get("codec_type") == "audio"]
|
||||
subs = [s for s in st if s.get("codec_type") == "subtitle"]
|
||||
fps = "-"
|
||||
if v and v.get("r_frame_rate"):
|
||||
try:
|
||||
n, d = (int(x) for x in v["r_frame_rate"].split("/", 1))
|
||||
fps = f"{round(n / d * 100) / 100:g}" if d else "0"
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
def num(x):
|
||||
try:
|
||||
return round(float(x))
|
||||
except (TypeError, ValueError):
|
||||
return 0
|
||||
|
||||
return [size, path, num(fmt.get("duration")),
|
||||
(v or {}).get("codec_name", "-"),
|
||||
f"{v['width']}x{v['height']}" if v else "-",
|
||||
fps, num(fmt.get("bit_rate")), (v or {}).get("pix_fmt", "-"),
|
||||
"+".join(f"{a.get('codec_name', '?')}:{a.get('language', '?')}" for a in aud),
|
||||
len(subs),
|
||||
sum(1 for a in aud if not re.search(
|
||||
r"default|forced", json.dumps(a.get("tags", {})), re.I)),
|
||||
fmt.get("format_name", "-") or "-"]
|
||||
|
||||
|
||||
def probe(enum, out, jobs):
|
||||
"""Probe every video-container row of the enum TSV, in parallel."""
|
||||
paths = []
|
||||
with open(enum) as fh:
|
||||
for line in fh:
|
||||
f = line.rstrip("\n").split("\t", 2)
|
||||
if len(f) == 3 and f[1] in PROBE_EXT:
|
||||
paths.append(f[2])
|
||||
print(f"probing {len(paths)} files with {jobs} workers", flush=True)
|
||||
done = unreadable = 0
|
||||
with open(out, "w") as fh, ThreadPoolExecutor(max_workers=jobs) as pool:
|
||||
for row in pool.map(probe_one, paths):
|
||||
if row is None:
|
||||
continue
|
||||
if row[3] == "-":
|
||||
unreadable += 1 # header unreadable: the row still gets written
|
||||
done += 1
|
||||
fh.write("\t".join(str(x) for x in row) + "\n")
|
||||
if done % 250 == 0:
|
||||
print(f" {done}/{len(paths)}", flush=True)
|
||||
print(f"probed={done} unreadable={unreadable} -> {out}")
|
||||
|
||||
|
||||
def plex(root, db, out):
|
||||
"""Plex's matched titles for files under root, from its DB, read-only."""
|
||||
con = open_plex_ro(db)
|
||||
with open(out, "w", newline="") as fh:
|
||||
w = csv.writer(fh, delimiter="\t", lineterminator="\n")
|
||||
w.writerow(PLEX_COLS)
|
||||
n = 0
|
||||
for r in con.execute(PLEX_SQL, (root + "/%",)):
|
||||
w.writerow([str(x).replace("\t", " ").replace("\n", " ") for x in r])
|
||||
n += 1
|
||||
print(f"plex-matched={n} -> {out}")
|
||||
|
||||
|
||||
def jellyfin(root, db, container, out):
|
||||
"""The episodes Jellyfin has already numbered, for the library at root.
|
||||
|
||||
Jellyfin stores each path as the CONTAINER sees it (its quadlet binds the
|
||||
share at /data), so `container` is the in-container prefix that stands for
|
||||
root and every exported path is mapped back to the host before it is written
|
||||
— the planner reads host paths, because it plans moves on the array.
|
||||
|
||||
The numbering is what matters, not the name: a release group's file name says
|
||||
nothing about which episode a file is, and Jellyfin has already answered that
|
||||
question against a provider. Season/episode and the episode's own title come
|
||||
from the episode row; the series name, its year and its TMDb and Tvdb ids come
|
||||
from the Series row the episode climbs to, because those are the only place
|
||||
Jellyfin keeps them.
|
||||
"""
|
||||
try:
|
||||
con = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
|
||||
except sqlite3.Error as e:
|
||||
sys.exit(f"cannot open the Jellyfin database read-only ({db}): {e}\n"
|
||||
"It is a WAL database: copy it with its -wal and -shm beside it and"
|
||||
" point --db at the copy. Never sudo it — the data dir belongs to"
|
||||
" the container's owner, and rootless storage is per-user.")
|
||||
have = {r[1] for r in con.execute("pragma table_info(BaseItems)")}
|
||||
need = {"Id", "ParentId", "Type", "Path", "IndexNumber", "ParentIndexNumber",
|
||||
"ExtraType", "IsVirtualItem", "ProductionYear", "Data"}
|
||||
if not need <= have:
|
||||
sys.exit(f"{db} is not a Jellyfin library database (BaseItems has no"
|
||||
f" {','.join(sorted(need - have))})")
|
||||
node = {}
|
||||
for i, pid, typ, name, year, data in con.execute(
|
||||
"select Id, ParentId, Type, Name, ProductionYear, Data from BaseItems"
|
||||
" where Type in (?,?,?)", JF_TV):
|
||||
node[i] = (pid, typ, name, year, data)
|
||||
|
||||
# Jellyfin 12 keeps provider ids in their own table; older builds kept them in
|
||||
# the row's Data blob. Both shapes are read, because the tvdb id the series
|
||||
# naming depends on is wherever this build happened to write it.
|
||||
prov = {}
|
||||
if "BaseItemProviders" in {r[0] for r in con.execute(
|
||||
"select name from sqlite_master where type='table'")}:
|
||||
for iid, pid, val in con.execute(
|
||||
"select ItemId, ProviderId, ProviderValue from BaseItemProviders"
|
||||
" where ProviderId in ('Tmdb', 'Tvdb')"):
|
||||
prov[(iid, pid)] = val
|
||||
|
||||
def provider_of(item_id, want):
|
||||
got = prov.get((item_id, want))
|
||||
if got:
|
||||
return str(got)
|
||||
try:
|
||||
blob = json.loads(node[item_id][4] or "{}").get("ProviderIds") or {}
|
||||
except (KeyError, ValueError, TypeError):
|
||||
return ""
|
||||
return str(blob.get(want, "") or "")
|
||||
|
||||
def series_of(item_id):
|
||||
"""Climb episode -> season -> series; empty when the chain is broken."""
|
||||
for _ in range(4):
|
||||
if item_id not in node:
|
||||
return "", "", "", ""
|
||||
pid, typ = node[item_id][0], node[item_id][1]
|
||||
if typ.endswith(".Series"):
|
||||
return (node[item_id][2] or "", node[item_id][3] or "",
|
||||
provider_of(item_id, "Tmdb"), provider_of(item_id, "Tvdb"))
|
||||
item_id = pid
|
||||
return "", "", "", ""
|
||||
|
||||
# The fleet's quadlets bind the share at /data, so the container's view of
|
||||
# this library is /data/<the library's own folder name> unless told otherwise.
|
||||
if not container:
|
||||
container = "/data/" + os.path.basename(os.path.normpath(root))
|
||||
prefix = container.rstrip("/") + "/"
|
||||
written = skipped = 0
|
||||
with open(out, "w", newline="") as fh:
|
||||
w = csv.writer(fh, delimiter="\t", lineterminator="\n")
|
||||
w.writerow(JF_COLS)
|
||||
for iid, path, idx, pidx, extra, virt, epname in con.execute(
|
||||
JF_SQL, (JF_TV[0], prefix + "%")):
|
||||
if virt or not path or not path.startswith(prefix):
|
||||
skipped += 1 # a virtual item is a variant, not a file
|
||||
continue
|
||||
sname, syear, tmdb, tvdb = series_of(iid)
|
||||
w.writerow([root + "/" + path[len(prefix):],
|
||||
sname.replace("\t", " "), syear,
|
||||
"" if pidx is None else pidx,
|
||||
"" if idx is None else idx,
|
||||
(extra or "").replace("\t", " "), tmdb,
|
||||
(epname or "").replace("\t", " "), tvdb])
|
||||
written += 1
|
||||
print(f"episodes={written} virtual-or-blank={skipped} -> {out}")
|
||||
|
||||
|
||||
def music(root, db, container, out):
|
||||
"""The tracks Jellyfin has already tagged, for the library at root.
|
||||
|
||||
Jellyfin reads music metadata out of the file itself, so an Audio row IS the
|
||||
tag read, done once, by the vendor's own parser: its album-artist, album,
|
||||
year, disc and track columns are what the embedded tags say and its Name is
|
||||
the track title. That makes this export the whole authority for a music
|
||||
relayout — the path a ripper chose is not evidence of anything, and the
|
||||
folder Jellyfin put a track under is often a release pack, not an album.
|
||||
|
||||
Two gaps this export answers from the row rather than the tree. Many tracks
|
||||
sit under a folder that is not an album item at all (a disc folder, or a
|
||||
mixed one), so the album name and year fall back to the tags when the album
|
||||
row is absent. And an album's MusicBrainz ids are not in the row's own Data
|
||||
— the current schema leaves that empty for music — they are rows of
|
||||
BaseItemProviders keyed by the item's Id.
|
||||
"""
|
||||
try:
|
||||
con = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
|
||||
except sqlite3.Error as e:
|
||||
sys.exit(f"cannot open the Jellyfin database read-only ({db}): {e}\n"
|
||||
"It is a WAL database: copy it with its -wal and -shm beside it and"
|
||||
" point --db at the copy. Never sudo it — the data dir belongs to"
|
||||
" the container's owner, and rootless storage is per-user.")
|
||||
have = {r[1] for r in con.execute("pragma table_info(BaseItems)")}
|
||||
need = {"Id", "ParentId", "Type", "Path", "Name", "IndexNumber",
|
||||
"ParentIndexNumber", "ProductionYear", "Album", "AlbumArtists",
|
||||
"Artists"}
|
||||
if not need <= have:
|
||||
sys.exit(f"{db} is not a Jellyfin library database with music columns"
|
||||
f" (BaseItems has no {','.join(sorted(need - have))})")
|
||||
# Provider ids, best identity first: the release group is the album, a bare
|
||||
# release id is one reissue of it and never overrides a group.
|
||||
mb = {}
|
||||
if "BaseItemProviders" in {r[0] for r in con.execute(
|
||||
"select name from sqlite_master where type='table'")}:
|
||||
rank = {p: i for i, p in enumerate(MUSIC_PROVIDERS)}
|
||||
best = {}
|
||||
for iid, pid, val in con.execute(
|
||||
"select ItemId, ProviderId, ProviderValue from BaseItemProviders"
|
||||
" where ProviderId in (?, ?)", MUSIC_PROVIDERS):
|
||||
if iid not in best or rank[pid] < best[iid][0]:
|
||||
best[iid] = (rank[pid], val)
|
||||
mb = {k: v[1] for k, v in best.items()}
|
||||
|
||||
# The fleet's quadlets bind the share at /data, so the container's view of
|
||||
# this library is /data/<the library's own folder name> unless told otherwise.
|
||||
if not container:
|
||||
container = "/data/" + os.path.basename(os.path.normpath(root))
|
||||
prefix = container.rstrip("/") + "/"
|
||||
written = skipped = 0
|
||||
with open(out, "w", newline="") as fh:
|
||||
w = csv.writer(fh, delimiter="\t", lineterminator="\n")
|
||||
w.writerow(MUSIC_COLS)
|
||||
for (path, aal, arts, tag_album, tag_year, disc, track, name, iid,
|
||||
alb_name, alb_year, alb_id) in con.execute(
|
||||
MUSIC_SQL, (JF_ALBUM, JF_AUDIO, prefix + "%")):
|
||||
if not path or not path.startswith(prefix):
|
||||
skipped += 1 # a row for a file outside this library
|
||||
continue
|
||||
# The album artist owns the folder; the track artist is the fallback
|
||||
# for the compilations whose album-artist tag was left empty.
|
||||
w.writerow([root + "/" + path[len(prefix):],
|
||||
((aal or "").strip() or (arts or "").strip()),
|
||||
(alb_name or tag_album or "").replace("\t", " "),
|
||||
alb_year or tag_year or "",
|
||||
"" if disc is None else disc,
|
||||
"" if track is None else track,
|
||||
(name or "").replace("\t", " "),
|
||||
mb.get(alb_id or "") or mb.get(iid or "") or "",
|
||||
(arts or "").replace("\t", " ")])
|
||||
written += 1
|
||||
print(f"tracks={written} outside-root-or-blank={skipped} -> {out}")
|
||||
|
||||
|
||||
def audiobooks(root, db, out):
|
||||
"""The chapters Plex matched inside the audiobook library, read-only.
|
||||
|
||||
Plex stores HOST paths here (unlike Jellyfin, which stores the container's
|
||||
view), so no prefix mapping is needed: the part's `file` is the path the
|
||||
operator has today. Rows for files outside `root` are skipped, because one
|
||||
Plex library DB carries every library on the box.
|
||||
|
||||
The chapter's `index` is exported as an empty field when Plex has none. That
|
||||
is not a gap to fill in here: three quarters of the books on this box contain
|
||||
at least one un-indexed chapter, so the relayout that reads this file must
|
||||
not renumber anything from it.
|
||||
"""
|
||||
con = open_plex_ro(db)
|
||||
tables = {r[0] for r in con.execute(
|
||||
"select name from sqlite_master where type='table'")}
|
||||
missing = {"media_parts", "media_items", "metadata_items"} - tables
|
||||
if missing:
|
||||
sys.exit(f"{db} is not a Plex library database (no "
|
||||
f"{', '.join(sorted(missing))})")
|
||||
written = skipped = 0
|
||||
with open(out, "w", newline="") as fh:
|
||||
w = csv.writer(fh, delimiter="\t", lineterminator="\n")
|
||||
w.writerow(AB_COLS)
|
||||
for file, narrator, book, year, track, chapter in con.execute(
|
||||
AB_SQL, (root.rstrip("/") + "/%",)):
|
||||
if not file or not file.startswith(root.rstrip("/") + "/"):
|
||||
skipped += 1 # a row for a file outside this library
|
||||
continue
|
||||
w.writerow([file.replace("\t", " "), narrator.replace("\t", " "),
|
||||
book.replace("\t", " "), "" if not year else year,
|
||||
"" if track is None or int(track) < 0 else track,
|
||||
chapter.replace("\t", " ")])
|
||||
written += 1
|
||||
print(f"chapters={written} outside-root={skipped} -> {out}")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
sub = ap.add_subparsers(dest="cmd", required=True)
|
||||
root = {"default": os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
|
||||
"help": "library root"}
|
||||
e = sub.add_parser("enumerate", help="stat every file under the root")
|
||||
e.add_argument("--root", **root)
|
||||
e.add_argument("--out", default="/tmp/media-enum.tsv")
|
||||
p = sub.add_parser("probe", help="ffprobe every video container")
|
||||
p.add_argument("--enum", default="/tmp/media-enum.tsv")
|
||||
p.add_argument("--out", default="/tmp/media-probed.tsv")
|
||||
p.add_argument("--jobs", type=int, default=6)
|
||||
x = sub.add_parser("plex", help="export Plex's matched titles for the root")
|
||||
x.add_argument("--root", **root)
|
||||
x.add_argument("--db", default=PLEX_DB)
|
||||
x.add_argument("--out", default="/tmp/plex-movies.tsv")
|
||||
j = sub.add_parser("jellyfin",
|
||||
help="export the episodes Jellyfin has numbered for the root")
|
||||
j.add_argument("--root", default=os.environ.get("MEDIA_ROOT",
|
||||
"/mnt/entropy/TVShows"),
|
||||
help="TV library root on the host")
|
||||
j.add_argument("--db", default=JF_DB)
|
||||
j.add_argument("--container", default=None,
|
||||
help="the in-container prefix that stands for --root"
|
||||
" (default /data/<root's folder name>)")
|
||||
j.add_argument("--out", default="/tmp/jellyfin-tv.tsv")
|
||||
m = sub.add_parser("music",
|
||||
help="export the tracks Jellyfin has tagged for the root")
|
||||
m.add_argument("--root", default=os.environ.get("MEDIA_MUSIC_ROOT",
|
||||
"/mnt/entropy/Music"),
|
||||
help="music library root on the host")
|
||||
m.add_argument("--db", default=JF_DB)
|
||||
m.add_argument("--container", default=None,
|
||||
help="the in-container prefix that stands for --root"
|
||||
" (default /data/<root's folder name>)")
|
||||
m.add_argument("--out", default="/tmp/jellyfin-music.tsv")
|
||||
b = sub.add_parser("audiobooks",
|
||||
help="export the chapters Plex matched for the audiobook root")
|
||||
b.add_argument("--root", default=os.environ.get("MEDIA_AUDIOBOOKS_ROOT",
|
||||
"/mnt/entropy/Audiobooks"),
|
||||
help="audiobook library root on the host")
|
||||
b.add_argument("--db", default=PLEX_DB)
|
||||
b.add_argument("--out", default="/tmp/plex-audiobooks.tsv")
|
||||
a = ap.parse_args()
|
||||
if a.cmd == "enumerate":
|
||||
enumerate_lib(a.root, a.out)
|
||||
elif a.cmd == "probe":
|
||||
probe(a.enum, a.out, a.jobs)
|
||||
elif a.cmd == "plex":
|
||||
plex(a.root, a.db, a.out)
|
||||
elif a.cmd == "music":
|
||||
music(a.root, a.db, a.container, a.out)
|
||||
elif a.cmd == "audiobooks":
|
||||
audiobooks(a.root, a.db, a.out)
|
||||
else:
|
||||
jellyfin(a.root, a.db, a.container, a.out)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+445
@@ -0,0 +1,445 @@
|
||||
#!/usr/bin/env python3
|
||||
r"""media-music.py — a plan that re-files the music library into Jellyfin's own shape.
|
||||
|
||||
media-music.py --root DIR --enum TSV --rows TSV [--out TSV] [--exceptions FILE]
|
||||
|
||||
Jellyfin's music contract (https://jellyfin.org/docs/general/server/media/music/)
|
||||
is not the movie one. Jellyfin reads the metadata out of the file, so a folder is
|
||||
an ALBUM and nothing else, every disc of an album belongs in that one folder, the
|
||||
tags decide the track order, and a file name only matters when it carries a
|
||||
character known to break playback (`< > : " / \ | ? *`) or when a lyric file has
|
||||
to match the basename of the track it belongs to. The plan is therefore:
|
||||
|
||||
Music/<album artist>/<Album> (<year>)/NN - <Title>.<ext>
|
||||
... the same folder, D.NN - <Title> when the album has discs
|
||||
Music/Various Artists/... when its tracks name
|
||||
more than one artist
|
||||
|
||||
The authority is the tag row Jellyfin already read — `scripts/media-census.py
|
||||
music` — never the release-pack folder a ripper made. A file with no row is not
|
||||
guessed at: it parks under `<root>/_unsorted` with the relative path it has today.
|
||||
|
||||
The rules below exist because of what a downloaded discography pack actually
|
||||
contains:
|
||||
|
||||
* A cuesheet beside exactly one audio file means the album is one image with a
|
||||
track list inside it. That file keeps its own name: numbering it `01` would
|
||||
claim it is one track, and Jellyfin shows one track either way.
|
||||
* Sidecars — images, cuesheets, EAC logs, checksums, booklets, playlists —
|
||||
follow the album the audio in their own folder belongs to. A folder holding
|
||||
two albums gives a sidecar no single owner, so it parks instead of being
|
||||
guessed into one.
|
||||
* A video container (.mkv, .vob, .ifo, …) inside a music library is not music.
|
||||
Renaming it to .mka/.m4a would tell Jellyfin to index video as audio, so it
|
||||
parks for a human decision.
|
||||
* Playlists move with their album and their CONTENTS are never rewritten: the
|
||||
paths inside an .m3u go stale. Editing a file the operator owns to keep a
|
||||
convenience file tidy is not this tool's job.
|
||||
|
||||
Nothing is deleted and nothing is overwritten. The output is a list of rename(2)
|
||||
intents for `scripts/media-apply.py`, whose preflight refuses any target that is
|
||||
already claimed and whose journal is the rollback.
|
||||
"""
|
||||
import argparse
|
||||
import csv
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from collections import Counter, defaultdict
|
||||
|
||||
# What Jellyfin will index as music. .mp4/.mkv/.webm are deliberately NOT here:
|
||||
# the vendor doc says an audio-only stream in those containers is never read as
|
||||
# music, and this tool renames for a reason it can prove, not to guess at one.
|
||||
AUDIO_EXT = {".flac", ".mp3", ".m4a", ".m4b", ".ogg", ".oga", ".opus", ".wav",
|
||||
".wma", ".aiff", ".aif", ".ape", ".dsf", ".dff", ".wv", ".mka",
|
||||
".mpc", ".aac"}
|
||||
VIDEO_EXT = {".mkv", ".vob", ".ifo", ".bup", ".mpg", ".mpeg", ".avi", ".mov",
|
||||
".ts", ".m2ts", ".wmv", ".webm", ".mp4", ".m4v", ".iso", ".img",
|
||||
".bdmv", ".divx", ".ogm"}
|
||||
# A lyric has to carry the basename of its track, so it is named after the audio
|
||||
# file and not after itself. .txt counts only when its stem matches an audio stem:
|
||||
# a loose .txt in an album folder is a note.
|
||||
LYRIC_EXT = {".lrc", ".elrc", ".txt"}
|
||||
IMAGE_EXT = {".jpg", ".jpeg", ".png", ".webp", ".gif", ".bmp", ".tiff"}
|
||||
# The album-artist strings that mean "compilation", so such an album is not filed
|
||||
# under one artist who does not own it.
|
||||
VARIOUS = {"various", "various artists", "unknown", "unknown artist", "unknow"}
|
||||
# The characters Jellyfin documents as breaking playback. They are dropped, not
|
||||
# substituted: a replacement character is a guess at what the title meant.
|
||||
BAD_CHARS = set('<>:"/\\|?*')
|
||||
TRASH_DIR = re.compile(r"(^|/)(\.AGENTTRASH|\.thumbnails|\.Trash-\d+|@eaDir"
|
||||
r"|lost\+found)(/|$)")
|
||||
|
||||
ARGS = None
|
||||
HELD = {} # path -> (bytes, ext)
|
||||
ROWS = {} # path -> the tag row
|
||||
PLACED = {} # audio path -> the destination the plan gave it
|
||||
CLAIMED = set() # every destination path the plan has handed out
|
||||
MOVES, KEEPS = [], []
|
||||
PARKED = defaultdict(list) # reason -> [path]
|
||||
NOTES = defaultdict(list) # section -> [line]
|
||||
CUE_DIRS = set() # source dirs holding a .cue
|
||||
AUDIO_BY_DIR = defaultdict(set) # source dir -> audio files a row names
|
||||
CHILD_OF = defaultdict(set) # dir -> the dirs directly below it holding audio
|
||||
FOLDERS_SEEN = {} # (dir, depth) -> album folders placed below it
|
||||
SUBTREE_DEPTH = 2 # how far below a folder its audio may sit
|
||||
ALBUMS = {} # (artist, album, year) -> the album group
|
||||
|
||||
AMBIGUOUS = object()
|
||||
|
||||
|
||||
def dominant(values):
|
||||
"""The spelling the tags mostly agree on: the most frequent value, ties broken
|
||||
by sort so the answer never depends on the order rows arrived in."""
|
||||
counts = Counter(values)
|
||||
return min(counts.items(), key=lambda kv: (-kv[1], kv[0]))[0]
|
||||
|
||||
|
||||
def rel(path):
|
||||
"""The path as seen from the library root."""
|
||||
return os.path.relpath(path, ARGS.root)
|
||||
|
||||
|
||||
def safe(name):
|
||||
"""A name every client can carry: no reserved character, nothing
|
||||
unprintable, no trailing dot or space, and short enough to rename."""
|
||||
out = "".join(c for c in (name or "")
|
||||
if c not in BAD_CHARS and (c.isprintable() or c == " "))
|
||||
out = re.sub(r"\s+", " ", out).strip().rstrip(". ")
|
||||
while out and len(out.encode("utf-8", "ignore")) > 200:
|
||||
out = out[:-4].rstrip()
|
||||
return out
|
||||
|
||||
|
||||
def assign(src, folder, name, ext, kind, note):
|
||||
"""Book one file into `folder` under `name`, renamed if that exact destination
|
||||
is already taken — by another file in the plan, or by a file standing there
|
||||
today. Two files claiming one path is the fault that costs a re-run of a 90k
|
||||
plan, and so is a plan that only works if the renames happen in the right
|
||||
order: the executor refuses any target that is occupied, so an occupied path
|
||||
is never a destination, even when its occupant is scheduled to leave.
|
||||
A file already standing where the contract puts it is a keep, not a move."""
|
||||
stem = safe(name) or "file"
|
||||
dst = os.path.join(folder, stem + ext)
|
||||
if dst == src:
|
||||
CLAIMED.add(dst.lower()) # it is standing on its own target
|
||||
KEEPS.append((src, dst, kind))
|
||||
return dst
|
||||
# A file already in this folder under this name — or under the numbered
|
||||
# variant the collision rule gives it — keeps the name it has. Renumbering it
|
||||
# every run would turn the plan into a shuffle instead of a fixed point.
|
||||
base = os.path.splitext(os.path.basename(src))[0]
|
||||
if os.path.dirname(src) == folder and (base == stem
|
||||
or base.startswith(stem + " (")):
|
||||
CLAIMED.add(src.lower())
|
||||
KEEPS.append((src, src, kind))
|
||||
return src
|
||||
n = 1
|
||||
while dst.lower() in CLAIMED:
|
||||
n += 1
|
||||
dst = os.path.join(folder, f"{stem} ({n}){ext}")
|
||||
if n > 1:
|
||||
NOTES[note].append(rel(dst))
|
||||
CLAIMED.add(dst.lower())
|
||||
MOVES.append((src, dst, kind))
|
||||
return dst
|
||||
|
||||
|
||||
def park(path, reason):
|
||||
"""Hold a file where it is, under the park folder, keeping its relative path."""
|
||||
PARKED[reason].append(path)
|
||||
dst = os.path.join(ARGS.park_dir, rel(path))
|
||||
if dst != path:
|
||||
CLAIMED.add(dst.lower())
|
||||
MOVES.append((path, dst, "park"))
|
||||
|
||||
|
||||
def under_park(path):
|
||||
"""True for a file already parked: that pile is for a human to review, and
|
||||
re-parking it every run would only push it deeper."""
|
||||
return path.startswith(ARGS.park_dir + os.sep)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------- albums
|
||||
def group_albums():
|
||||
"""Every tagged track into one group per (album-artist tag, album, year).
|
||||
|
||||
The key is the identity the tags assert, not the folder the file sits in: one
|
||||
album arrives as three discs in three folders, and two copies of an album in
|
||||
one pack are one album to Jellyfin, whose album item is that name, artist and
|
||||
year. The artist in the key is the ALBUM-artist tag and never the performer,
|
||||
because a compilation is one album whose tracks name many performers — keying
|
||||
on the performer would file one album once per artist.
|
||||
"""
|
||||
for path, r in sorted(ROWS.items()):
|
||||
if path not in HELD or under_park(path):
|
||||
continue
|
||||
d = os.path.dirname(path)
|
||||
AUDIO_BY_DIR[d].add(path)
|
||||
CHILD_OF[os.path.dirname(d)].add(d) # so a cover above CD1/CD2 sees both
|
||||
tag = (r["artist"] or "").strip()
|
||||
album = (r["album"] or "").strip()
|
||||
performer = (r["performer"] or "").strip()
|
||||
if not album:
|
||||
park(path, "a track no album names")
|
||||
continue
|
||||
if not tag and not performer:
|
||||
park(path, "a track no artist names")
|
||||
continue
|
||||
key = (tag.casefold(), album.casefold(), (r["year"] or "").strip())
|
||||
g = ALBUMS.setdefault(key, {"tags": [], "albums": [], "year": key[2],
|
||||
"members": [], "performers": set(),
|
||||
"mb": set()})
|
||||
g["members"].append(path)
|
||||
g["tags"].append(tag)
|
||||
g["albums"].append(album)
|
||||
g["performers"].add(performer or tag)
|
||||
if r.get("mb"):
|
||||
g["mb"].add(r["mb"])
|
||||
|
||||
|
||||
def folder_artist(g):
|
||||
"""Whose shelf the album sits on. The album-artist tag owns it when it names
|
||||
one; when the tags name only performers, a single performer names the album
|
||||
and several make it a compilation, which belongs to no one of them. The key is
|
||||
case-folded, so the spelling that reaches the folder is the one the tracks
|
||||
mostly carry — see dominant()."""
|
||||
tags = [t for t in g["tags"] if t]
|
||||
if tags:
|
||||
name = dominant(tags)
|
||||
return "Various Artists" if name.casefold() in VARIOUS else name
|
||||
if len({p.casefold() for p in g["performers"]}) == 1:
|
||||
return dominant(g["performers"])
|
||||
return "Various Artists"
|
||||
|
||||
|
||||
def album_folder(g):
|
||||
name = safe(dominant(g["albums"])) or "Untitled"
|
||||
return os.path.join(ARGS.root, safe(folder_artist(g)) or "Unknown Artist",
|
||||
f"{name} ({g['year']})" if g["year"] else name)
|
||||
|
||||
|
||||
def is_cue_image(path):
|
||||
"""A folder holding this one audio file and a cuesheet is an album image."""
|
||||
d = os.path.dirname(path)
|
||||
return len(AUDIO_BY_DIR.get(d, ())) == 1 and d in CUE_DIRS
|
||||
|
||||
|
||||
def number_of(value, width):
|
||||
return f"{int(value):0{width}d}" if str(value or "").isdigit() else ""
|
||||
|
||||
|
||||
def place_albums():
|
||||
for key in sorted(ALBUMS):
|
||||
g = ALBUMS[key]
|
||||
folder = album_folder(g)
|
||||
discs = {g0["disc"] for g0 in (ROWS[p] for p in g["members"]) if g0["disc"]}
|
||||
tracks = [int(ROWS[p]["track"]) for p in g["members"]
|
||||
if str(ROWS[p]["track"] or "").isdigit()]
|
||||
# More than one disc, or a disc numbered past the first: the track number
|
||||
# alone cannot name a file uniquely, so the disc goes in front of it.
|
||||
multi = len(discs) > 1 or any(str(d).isdigit() and int(d) > 1 for d in discs)
|
||||
width = 3 if multi or (max(tracks, default=0) > 99) else 2
|
||||
for path in sorted(g["members"]):
|
||||
r = ROWS[path]
|
||||
ext = HELD[path][1]
|
||||
base = os.path.splitext(os.path.basename(path))[0]
|
||||
if is_cue_image(path):
|
||||
name = safe(base) or "image"
|
||||
else:
|
||||
title = safe(r["title"] or base) or "untitled"
|
||||
num = number_of(r["track"], 2 if multi else width)
|
||||
if multi and num:
|
||||
num = f"{r['disc'] or '1'}.{num}"
|
||||
name = f"{num} - {title}" if num else title
|
||||
PLACED[path] = assign(path, folder, name, ext, "track",
|
||||
"two tracks claimed one name inside one album")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------- sidecars
|
||||
def folders_below(d, depth):
|
||||
"""The album folders the audio in `d`, or within `depth` directories below it,
|
||||
was placed in. Memoised, because every loose file in a big discography folder
|
||||
asks the same question about it."""
|
||||
if (d, depth) not in FOLDERS_SEEN:
|
||||
seen = {os.path.dirname(PLACED[a]) for a in AUDIO_BY_DIR.get(d, ())
|
||||
if a in PLACED}
|
||||
if depth:
|
||||
for child in CHILD_OF.get(d, ()):
|
||||
seen |= folders_below(child, depth - 1)
|
||||
FOLDERS_SEEN[(d, depth)] = seen
|
||||
return FOLDERS_SEEN[(d, depth)]
|
||||
|
||||
|
||||
def owner_folder(d):
|
||||
"""The album folder that owns this directory — the first place up the tree
|
||||
where audio sits, in the directory itself or within SUBTREE_DEPTH levels below
|
||||
it, which is how a box with its discs in CD1/CD2 owns the cover lying above
|
||||
them. None when no audio is near enough to say, AMBIGUOUS when the audio there
|
||||
belongs to more than one album."""
|
||||
while d.startswith(ARGS.root + os.sep) and d != ARGS.root:
|
||||
folders = folders_below(d, SUBTREE_DEPTH)
|
||||
if folders:
|
||||
return next(iter(folders)) if len(folders) == 1 else AMBIGUOUS
|
||||
d = os.path.dirname(d)
|
||||
return None
|
||||
|
||||
|
||||
def place_sidecars():
|
||||
"""Everything that is not a tagged track: lyrics, images, cuesheets, logs,
|
||||
booklets, playlists — and the clutter that is none of those."""
|
||||
# The lyric rule needs the audio's stem as it is named TODAY, before the plan
|
||||
# renames it, because that is what the sidecar's own name was copied from.
|
||||
stem = {(d, os.path.splitext(os.path.basename(a))[0]): a
|
||||
for d, audios in AUDIO_BY_DIR.items() for a in audios}
|
||||
for path, (_size, ext) in sorted(HELD.items()):
|
||||
if path in ROWS or under_park(path) or TRASH_DIR.search(path):
|
||||
continue
|
||||
d, base = os.path.dirname(path), os.path.splitext(os.path.basename(path))[0]
|
||||
if ext in VIDEO_EXT:
|
||||
park(path, "a video container inside the music library: Jellyfin will "
|
||||
"not read it as music, and renaming it would claim it is")
|
||||
elif ext in AUDIO_EXT:
|
||||
park(path, "an audio file no provider row names — its tags are not in "
|
||||
"the library Jellyfin has matched")
|
||||
elif ext in (LYRIC_EXT | IMAGE_EXT) and (d, base) in stem:
|
||||
# A lyric, or a picture named after its track: it follows that track
|
||||
# and takes the name the track now has, or Jellyfin cannot find it.
|
||||
audio = stem[(d, base)]
|
||||
dst = PLACED.get(audio)
|
||||
if dst:
|
||||
folder, name = os.path.dirname(dst), os.path.splitext(
|
||||
os.path.basename(dst))[0]
|
||||
assign(path, folder, name, ext, "lyric",
|
||||
"a sidecar whose name collided with another in its album")
|
||||
else:
|
||||
park(path, "a sidecar of a track the plan could not place")
|
||||
else:
|
||||
owner = owner_folder(d)
|
||||
if owner is AMBIGUOUS:
|
||||
park(path, "a sidecar whose folder holds more than one album, so "
|
||||
"nothing says which one it belongs to")
|
||||
elif owner:
|
||||
assign(path, owner, base, ext,
|
||||
"playlist" if ext in (".m3u", ".m3u8") else "sidecar",
|
||||
"one album folder would have carried two files of the same "
|
||||
"sidecar name")
|
||||
else:
|
||||
park(path, "not music, or music with nothing beside it to belong to")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------- outputs
|
||||
def write_plan():
|
||||
"""mkdir rows for every destination folder and its whole chain, then the moves.
|
||||
Parents first: the executor creates directories in the order they are listed."""
|
||||
chain = set()
|
||||
for _s, dst, _k in MOVES:
|
||||
d = os.path.dirname(dst)
|
||||
while d and d != ARGS.root and d != os.path.dirname(d):
|
||||
chain.add(d)
|
||||
d = os.path.dirname(d)
|
||||
dirs = sorted(chain, key=lambda p: (p.count("/"), p))
|
||||
with open(ARGS.out, "w") as fh:
|
||||
fh.write("action\tsrc\tdst\tkind\n")
|
||||
for d in dirs:
|
||||
fh.write(f"mkdir\t\t{d}\tdir\n")
|
||||
for src, dst, kind in sorted(MOVES, key=lambda m: (m[1], m[0])):
|
||||
fh.write(f"move\t{src}\t{dst}\t{kind}\n")
|
||||
for src, dst, kind in sorted(KEEPS, key=lambda m: (m[1], m[0])):
|
||||
fh.write(f"keep\t{src}\t{dst}\t{kind}\n")
|
||||
parked = [m for m in MOVES if m[2] == "park"]
|
||||
sized = sum(HELD[s][0] for s, _d, _k in MOVES if s in HELD)
|
||||
print(f"plan: {len(MOVES) - len(parked)} moves, {len(parked)} parked"
|
||||
f" ({sized / 2**40:.2f} TiB touched), {len(KEEPS)} already placed,"
|
||||
f" {len(dirs)} folders -> {ARGS.out}")
|
||||
|
||||
|
||||
def write_exceptions():
|
||||
with open(ARGS.exceptions, "w") as fh:
|
||||
fh.write(f"Music relayout exceptions for {ARGS.root}\n\n")
|
||||
fh.write(f"Everything below is parked under {ARGS.park_dir}, keeping the\n"
|
||||
"relative path it has today. Nothing is deleted and nothing is\n"
|
||||
"overwritten: a parked file is one rename from wherever the plan is\n"
|
||||
"pointed next.\n\n")
|
||||
for reason in sorted(PARKED, key=lambda r: -len(PARKED[r])):
|
||||
rs = PARKED[reason]
|
||||
fh.write(f"{len(rs):>6} files {reason}\n")
|
||||
for p in sorted(rs, key=rel)[:400]:
|
||||
fh.write(f" {rel(p)}\n")
|
||||
if len(rs) > 400:
|
||||
fh.write(f" … and {len(rs) - 400} more\n")
|
||||
fh.write("\n")
|
||||
for section in sorted(NOTES, key=lambda s: -len(NOTES[s])):
|
||||
lines = sorted(set(NOTES[section]))
|
||||
fh.write(f"{len(lines):>6} {section}\n")
|
||||
for line in lines[:200]:
|
||||
fh.write(f" {line}\n")
|
||||
if len(lines) > 200:
|
||||
fh.write(f" … and {len(lines) - 200} more\n")
|
||||
fh.write("\n")
|
||||
no_year = sorted(f"{folder_artist(g)}/{dominant(g['albums'])}"
|
||||
for g in ALBUMS.values() if not g["year"])
|
||||
if no_year:
|
||||
fh.write(f"{len(no_year):>6} albums carry no year in any tag, so their folders\n"
|
||||
" are named without one; Jellyfin still matches them.\n")
|
||||
fh.write(" " + ", ".join(no_year[:300]) + "\n\n")
|
||||
clash = sorted(f"{folder_artist(g)}/{dominant(g['albums'])}"
|
||||
for g in ALBUMS.values() if len(g["mb"]) > 1)
|
||||
if clash:
|
||||
fh.write(f"{len(clash):>6} albums put one folder name on more than one\n"
|
||||
" MusicBrainz release group — different releases, one name.\n"
|
||||
" They share the folder Jellyfin's own album item names.\n")
|
||||
fh.write(" " + ", ".join(clash[:200]) + "\n\n")
|
||||
fh.write(" Playlists move with their album and their contents are never\n"
|
||||
" rewritten: the paths inside an .m3u go stale.\n")
|
||||
print(f"exceptions -> {ARGS.exceptions}")
|
||||
|
||||
|
||||
def main():
|
||||
global ARGS
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--root", default=os.environ.get("MEDIA_MUSIC_ROOT",
|
||||
"/mnt/entropy/Music"))
|
||||
ap.add_argument("--enum", default="/tmp/music-enum.tsv",
|
||||
help="the enumerate TSV (bytes, ext, path)")
|
||||
ap.add_argument("--rows", default="/tmp/jellyfin-music.tsv",
|
||||
help="the tag rows scripts/media-census.py music wrote")
|
||||
ap.add_argument("--out", default="/tmp/music-plan.tsv")
|
||||
ap.add_argument("--exceptions", default="/tmp/music-exceptions.txt")
|
||||
ap.add_argument("--park", default="_unsorted",
|
||||
help="the folder inside the root that holds what cannot be filed")
|
||||
ARGS = ap.parse_args()
|
||||
ARGS.root = os.path.normpath(ARGS.root)
|
||||
ARGS.park_dir = os.path.join(ARGS.root, ARGS.park)
|
||||
read_inputs()
|
||||
# Every path a file occupies today is claimed before a single destination is
|
||||
# handed out: the executor refuses an occupied target, so a plan that aimed at
|
||||
# one would only fail after the renames that were supposed to clear it.
|
||||
CLAIMED.update(p.lower() for p in HELD)
|
||||
for path, (_size, ext) in HELD.items():
|
||||
if ext == ".cue":
|
||||
CUE_DIRS.add(os.path.dirname(path))
|
||||
group_albums()
|
||||
place_albums()
|
||||
place_sidecars()
|
||||
write_plan()
|
||||
write_exceptions()
|
||||
|
||||
|
||||
def read_inputs():
|
||||
with open(ARGS.enum) as fh:
|
||||
for line in fh:
|
||||
parts = line.rstrip("\n").split("\t")
|
||||
if len(parts) >= 3:
|
||||
HELD[parts[2]] = (int(parts[0]), parts[1].lower())
|
||||
with open(ARGS.rows, newline="") as fh:
|
||||
for row in csv.DictReader(fh, delimiter="\t"):
|
||||
if row.get("file"):
|
||||
ROWS[row["file"]] = row
|
||||
gone = [p for p in ROWS if p not in HELD]
|
||||
if gone:
|
||||
NOTES["a tag row whose file the census no longer sees"].extend(gone)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+881
@@ -0,0 +1,881 @@
|
||||
#!/usr/bin/env python3
|
||||
r"""Plan a Jellyfin-format relayout of a movie library. DRY RUN ONLY: nothing moves.
|
||||
|
||||
media-plan.py [--root DIR] [--enum TSV] [--probed TSV] [--plex TSV]
|
||||
[--out TSV] [--exceptions TXT]
|
||||
|
||||
Inputs are the TSVs scripts/media-census.py exports, plus the optional Plex
|
||||
metadata-cache export whose clean titles outrank the filename parser; the title
|
||||
authority for everything Plex never matched is scripts/media-titles.py beside this
|
||||
file. No media file is opened, so planning the whole library costs seconds.
|
||||
|
||||
PLAN is a TSV the executor (scripts/media-apply.py) replays verbatim —
|
||||
`action<TAB>src<TAB>dst<TAB>kind` with action in mkdir | move | keep, ordered so
|
||||
parents exist before children. EXCEPTIONS is the human review list: every file the
|
||||
plan could not name a film for, every collision it refused, the READMEs the
|
||||
executor must write into the piles it created.
|
||||
|
||||
Target contract (jellyfin.org/docs/general/server/media/movies):
|
||||
* one folder per movie, named Title (year)
|
||||
* the video inside is named exactly like the folder
|
||||
* extra copies of one film are versions: "<base> - <label>.ext"; the base must
|
||||
match the folder name character-for-character or Jellyfin splits them apart
|
||||
* extras live in canonical subfolders (trailers, featurettes, deleted scenes,
|
||||
interviews, samples, shorts, clips, other, extras, backdrops, theme-music)
|
||||
* sidecars <base>.<lang>.<flag>.<ext>; images <base>-<poster|backdrop|...>
|
||||
* <>:"/\|?* break the matcher, so they never survive here
|
||||
* VIDEO_TS/BDMV folders support neither multiple versions nor parts
|
||||
"""
|
||||
import argparse
|
||||
import collections
|
||||
import csv
|
||||
import importlib.util
|
||||
import os
|
||||
import re
|
||||
|
||||
# Inputs are the census TSVs scripts/media-census.py writes; the engine beside
|
||||
# this file is the title authority. Nothing here opens a media file.
|
||||
def parse_args():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--root", default=os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
|
||||
help="library root the plan moves files inside")
|
||||
ap.add_argument("--enum", default="/tmp/media-enum.tsv", help="file census TSV")
|
||||
ap.add_argument("--probed", default="/tmp/media-probed.tsv", help="ffprobe census TSV")
|
||||
ap.add_argument("--plex", default="/tmp/plex-movies.tsv",
|
||||
help="Plex metadata export (clean titles); absent = parser only")
|
||||
ap.add_argument("--out", default="/tmp/jf-plan.tsv")
|
||||
ap.add_argument("--exceptions", default="/tmp/jf-exceptions.txt")
|
||||
return ap.parse_args()
|
||||
|
||||
|
||||
ARGS = parse_args()
|
||||
ROOT = ARGS.root
|
||||
ENUM, PROBE, PLEX = ARGS.enum, ARGS.probed, ARGS.plex
|
||||
PLAN, EXCEPTS = ARGS.out, ARGS.exceptions
|
||||
ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
|
||||
|
||||
spec = importlib.util.spec_from_file_location("eng", ENGINE)
|
||||
eng = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(eng)
|
||||
|
||||
VIDEO_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
|
||||
".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".flv", ".vob", ".iso",
|
||||
".img", ".mpv", ".3gp", ".rm", ".rmvb", ".m2v", ".mpv"}
|
||||
DISC_DIRS = {"BDMV", "VIDEO_TS", "CERTIFICATE", "AACS", "MLPI", "CLIPINF"}
|
||||
SUB_EXT = {".srt", ".ass", ".ssa", ".sub", ".vtt"}
|
||||
IMG_EXT = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".gif", ".tbn"}
|
||||
AUDIO_DIR_RE = re.compile(r"soundtrack|score|\bost\b|music|song|album", re.I)
|
||||
SUB_KEEP_RE = re.compile(r"^(?!.*\d)(?!.*[._])(?!.*\[)(?!.*\()([A-Za-z][A-Za-z -]{1,24})$")
|
||||
MUSIC_TOP_RE = re.compile(r"(?i)soundtrack|score|\bost\b|music|album|song")
|
||||
SUB_DIR_OK = re.compile(r"(?i)^$|subs?|subtitles?|srt|custom ?subs?(\[.*\])?"
|
||||
r"|(additional|extra) languages|languages|resources")
|
||||
AUDIO_DIR_OK = re.compile(r"(?i)^$|audio(s)?|extra audio|additional audio"
|
||||
r"|(additional|extra) languages|commentar(y|ies)"
|
||||
r"|audio commentary|audio tracks?")
|
||||
|
||||
|
||||
AUDIO_EXT = {".mp3", ".flac", ".m4a", ".aac", ".dts", ".wav", ".ac3", ".mka",
|
||||
".opus", ".ogg", ".wma"}
|
||||
NFO_EXT = {".nfo"}
|
||||
IMG_PRIMARY = {"poster", "folder", "cover", "default", "movie", "show"}
|
||||
IMG_OTHER = {"backdrop": "backdrop", "fanart": "backdrop", "background": "backdrop",
|
||||
"banner": "banner", "logo": "logo", "clearlogo": "logo",
|
||||
"thumb": "thumb", "landscape": "thumb", "art": "backdrop",
|
||||
"clearart": "backdrop", "jacket": "poster", "cdart": "poster",
|
||||
"discart": "poster", "disc": "poster"}
|
||||
RES_RE = re.compile(r"\b(2160|4320|4k|1080|720|576|480|360|240)(i|p)\b", re.I)
|
||||
# The characters Jellyfin's matcher cannot carry in a file or folder name. "/" is
|
||||
# one of them and it is the dangerous one: Plex titles KEEP it ("V/H/S/2", and this
|
||||
# library holds six of those films), so leaving it in a name turns ONE movie folder
|
||||
# into a nested tree — Movies/V/H/S/2 (2013)/… — and Jellyfin then sees four
|
||||
# folders, none of them a movie.
|
||||
BAD_CHARS = re.compile(r'[<>:"/|?*]')
|
||||
PART_RE = re.compile(r"(?:^|[\s._-])(cd|dvd|part|pt|disc|disk|disco)"
|
||||
r"[\s._-]?([1-9abcd])(?:$|[\s._-])", re.I)
|
||||
LANGS = {"en", "eng", "it", "ita", "ja", "jpn", "de", "deu", "fr", "fra", "es",
|
||||
"spa", "pt", "por", "ru", "rus", "zh", "chi", "ko", "kor", "sv", "no",
|
||||
"da", "nl", "pl", "cs", "hu", "fi", "tr", "el", "he", "ar", "uk", "ro",
|
||||
"bg", "hr", "sr", "sk", "et", "lv", "lt", "id", "th", "vi", "fa", "la"}
|
||||
FLAGS = {"default", "forced", "foreign", "sdh", "cc"}
|
||||
NOISE = {"sub", "subs", "subtitle", "subtitles", "srt", "ass", "ext", "external",
|
||||
"the", "a", "an", "of", "and", "in", "on", "to", "for", "with", "ita",
|
||||
"eng", "english", "italian", "dual", "audio", "forced", "default"}
|
||||
# Canonical title where the library only ever carries a localised name.
|
||||
OVERRIDES = {"28 giorni dopo": "28 Days Later",
|
||||
"28 settimane dopo": "28 Weeks Later"}
|
||||
|
||||
rows = []
|
||||
for line in open(ENUM):
|
||||
size, ext, path = line.rstrip("\n").split("\t")
|
||||
rows.append({"bytes": int(size), "ext": ext.lower(), "path": path})
|
||||
ext_by = {r["path"]: r["ext"] for r in rows}
|
||||
size_by = {r["path"]: r["bytes"] for r in rows}
|
||||
|
||||
|
||||
def rel(p):
|
||||
return p[len(ROOT) + 1:] if p.startswith(ROOT + "/") else p
|
||||
|
||||
|
||||
def parts_of(p):
|
||||
return [q for q in rel(p).split("/") if q]
|
||||
|
||||
|
||||
def is_qb(p):
|
||||
return ".!q" in p.lower()
|
||||
|
||||
|
||||
def disc_root(p, fold=True):
|
||||
"""Absolute root of the disc tree containing p (the MOVIE folder that holds
|
||||
BDMV/ or VIDEO_TS/), else None. `fold` also folds bookkeeping dirs that sit
|
||||
beside the tree (CERTIFICATE/, AACS/) into the same movie folder."""
|
||||
ps = parts_of(p)
|
||||
kinds = DISC_DIRS if fold else {"BDMV", "VIDEO_TS"}
|
||||
for i, q in enumerate(ps):
|
||||
if q.upper() in kinds:
|
||||
return ROOT + "/" + "/".join(ps[:i])
|
||||
return None
|
||||
|
||||
|
||||
def row_name(r):
|
||||
"""The name a disc-tree entry should be identified by: the engine stores the
|
||||
disc's own folder for it, because "BDMV/index.bdmv" is not a title."""
|
||||
return r.get("name_src") or r["path"]
|
||||
|
||||
|
||||
def sanitize(name):
|
||||
name = BAD_CHARS.sub("-", name).replace("\\", "-")
|
||||
name = re.sub(r"\s+", " ", name).strip().strip(". ").strip()
|
||||
return (name[:180] or "untitled")
|
||||
|
||||
|
||||
def ok_year(y):
|
||||
"""True for a year a real film could carry. Plex stores 0 for a file it indexed
|
||||
but never MATCHED, and "0" is a digit string: trusting it named 67 movie
|
||||
folders "Title (0)" — Jellyfin items with no identity, no year and no poster."""
|
||||
return bool(re.fullmatch(r"(18|19|20)\d\d", y or "")) and 1870 <= int(y) <= 2035
|
||||
|
||||
|
||||
WORD_SPLIT = re.compile(r"[^0-9a-z]+")
|
||||
|
||||
|
||||
def words_of(text):
|
||||
"""The matching words of a name — or of a whole path, which is how callers
|
||||
pass a file together with the folders it sits in.
|
||||
|
||||
Split on everything that is not a letter or a digit: eng.norm does not touch
|
||||
"/", so feeding it "…/Psycho (1960)/Extras" left glued tokens like
|
||||
"(1960)/extras" and "(1954)" that no other name could ever agree with, and a
|
||||
film could not be found by the folder that holds it — which is how one box
|
||||
set's featurettes ended up in another film's folder. Both sides of every
|
||||
comparison below go through here. A bare number counts only when it could be
|
||||
a release year, so release junk ("1080", "2160", "2", "7") never poses as
|
||||
agreement between two names.
|
||||
"""
|
||||
out = set()
|
||||
for w in WORD_SPLIT.split(eng.norm(text)):
|
||||
if not w or w in eng.STOP:
|
||||
continue
|
||||
if w.isdigit() and not 1870 <= int(w) <= 2100:
|
||||
continue
|
||||
out.add(w)
|
||||
return out
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ Plex titles
|
||||
# Plex has matched a large part of the library's media files to real titles
|
||||
# ("Alien", not "Alien.Directors.Cut.1979.1080p.BRrip.x265…"). Where the cache
|
||||
# holds a MATCH, its title wins; the filename parser carries everything else
|
||||
# (extras, disc trees, TV, and whatever Plex never matched).
|
||||
def plex_titles():
|
||||
if not os.path.exists(PLEX):
|
||||
return {}
|
||||
out = {}
|
||||
with open(PLEX, newline="") as fh:
|
||||
for r in csv.DictReader(fh, delimiter="\t"):
|
||||
t, y = r["leaf_title"].strip(), r["leaf_year"].strip()
|
||||
# Only a match is an authority, and a match carries a year: an
|
||||
# unmatched row echoes the filename, which the parser reads better
|
||||
# than a copied release name. The junk filter stays for the matches
|
||||
# whose title still carries the release.
|
||||
if t and ok_year(y) and not re.search(
|
||||
r"2160p|1080p|x26[45]|hevc|bluray|web-?dl", t.lower()):
|
||||
out[r["file"]] = (t, y)
|
||||
return out
|
||||
|
||||
|
||||
PLEX_T = plex_titles()
|
||||
|
||||
# ------------------------------------------------------------- classification
|
||||
erows = eng.load(PROBE)
|
||||
for r in erows:
|
||||
for k, v in OVERRIDES.items():
|
||||
if k in r["title"]:
|
||||
r["title"] = v
|
||||
r["key"] = eng.key_of(r["title"])
|
||||
r["base"] = eng.base_words(r["title"], r["key"])
|
||||
marks = {r["path"] for r in erows if eng.is_watermark(r["path"], r["bytes"])}
|
||||
extras = {r["path"] for r in erows
|
||||
if r["path"] not in marks
|
||||
and eng.is_extra(r["path"], r["title"], r["dur"], ROOT,
|
||||
matched=r["path"] in PLEX_T)}
|
||||
eps = collections.defaultdict(list)
|
||||
for r in erows:
|
||||
if r["path"] in marks | extras:
|
||||
continue
|
||||
ep = eng.ep_of(r["path"])
|
||||
if ep:
|
||||
eps[ep[0]].append(r) # keyed by the SERIES, never the episode title
|
||||
tv_keys = {k for k, v in eps.items() if len(v) >= 5}
|
||||
tv_set = {r["path"] for k, v in eps.items() if k in tv_keys for r in v}
|
||||
# A folder INSIDE a movie's folder is bonus territory even when its own name is
|
||||
# not bonus vocabulary: "Saboteur (1942)/Exrtas/Saboteur - A Closer Look" is a
|
||||
# misspelled Extras folder, so the engine's folder rule never fired and the
|
||||
# featurette became a movie folder of its own — whose words then out-scored the
|
||||
# real Saboteur and drew that box set's other extras into it. The rule needs no
|
||||
# metadata: the folder holding the longest film owns the shorter video filed
|
||||
# under it, provided that shorter file's own name speaks the film's name — a
|
||||
# whole other film filed there stays a film, exactly as the engine's folder rule
|
||||
# exempts it. A different year in the file's own name IS that other film
|
||||
# speaking, so it vetoes the rule.
|
||||
def longest_of(rows):
|
||||
"""The longest video in each folder: that folder's feature."""
|
||||
out = {}
|
||||
for r in rows:
|
||||
if r["dur"] > 0:
|
||||
d = os.path.dirname(r["path"])
|
||||
if d not in out or r["dur"] > out[d]["dur"]:
|
||||
out[d] = r
|
||||
return out
|
||||
|
||||
|
||||
def not_disc(r):
|
||||
"""A disc tree's innards belong to the tree, never to the folder's film."""
|
||||
return not any(c in DISC_DIRS for c in r["path"].split("/"))
|
||||
|
||||
|
||||
FEATURE_OF = longest_of([r for r in erows if r["path"] not in marks and not_disc(r)])
|
||||
|
||||
|
||||
def under_a_feature(path, dur):
|
||||
"""True when a folder above this file holds a longer film that this file's
|
||||
own name names."""
|
||||
d = os.path.dirname(os.path.dirname(path))
|
||||
while len(d) > len(ROOT):
|
||||
f = FEATURE_OF.get(d)
|
||||
if f:
|
||||
if f["dur"] < dur:
|
||||
return False
|
||||
theirs = (words_of(os.path.basename(f["path"]))
|
||||
| words_of(os.path.basename(d)))
|
||||
if not words_of(os.path.basename(path)) & theirs:
|
||||
return False
|
||||
y1 = re.search(r"(?:18|19|20)\d\d", os.path.basename(path))
|
||||
y2 = re.search(r"(?:18|19|20)\d\d", os.path.basename(f["path"]))
|
||||
return not (y1 and y2 and y1.group() != y2.group())
|
||||
d = os.path.dirname(d)
|
||||
return False
|
||||
|
||||
|
||||
extras |= {r["path"] for r in erows
|
||||
if r["dur"] > 0 and r["path"] not in marks | extras | tv_set
|
||||
and not_disc(r) and under_a_feature(r["path"], r["dur"])}
|
||||
film_rows = [r for r in erows if r["path"] not in marks | extras | tv_set]
|
||||
|
||||
buckets = collections.defaultdict(list)
|
||||
for r in film_rows:
|
||||
buckets[eng.key_of(r["title"])].append(r)
|
||||
|
||||
|
||||
def res_label(r):
|
||||
m = RES_RE.search(r["res"] or "")
|
||||
if m:
|
||||
n, kind = m.group(1).lower(), m.group(2).lower()
|
||||
return "2160p" if n in ("2160", "4k") else f"{n}{kind}"
|
||||
try:
|
||||
w = int((r["res"] or "0x0").split("x")[0])
|
||||
except (ValueError, IndexError):
|
||||
w = 0
|
||||
return "2160p" if w >= 3000 else ("1080p" if w >= 1400 else "SD")
|
||||
|
||||
|
||||
def version_label(r):
|
||||
bits = [res_label(r)]
|
||||
vc = (r["vcodec"] or "").lower()
|
||||
if vc in ("av1", "libaom", "libsvtav1"):
|
||||
bits.append("AV1")
|
||||
elif vc in ("vc1", "wmv3"):
|
||||
bits.append("VC1")
|
||||
if r["bytes"] > 40 * 2**30:
|
||||
bits.append("REMUX")
|
||||
if "imax" in r["path"].lower():
|
||||
bits.append("IMAX")
|
||||
return sanitize(" ".join(bits))
|
||||
|
||||
|
||||
def group_name(items):
|
||||
titles = collections.Counter(r["title"] for r in items)
|
||||
ascii_t = [t for t in titles if t.isascii()]
|
||||
title = (collections.Counter(ascii_t) if ascii_t else titles).most_common(1)[0][0]
|
||||
years = [r["year"] for r in items if r["year"] != "?"] or \
|
||||
[r["soft"] for r in items if r["soft"]]
|
||||
year = collections.Counter(years).most_common(1)[0][0] if years else None
|
||||
return sanitize(f"{title} ({year})" if year else title)
|
||||
|
||||
|
||||
groups = []
|
||||
for key, items in buckets.items():
|
||||
for c in eng.dur_clusters(items):
|
||||
g = {"rows": c, "base": group_name(c)}
|
||||
known = [PLEX_T[r["path"]] for r in c if r["path"] in PLEX_T]
|
||||
if known:
|
||||
# one film, one name: a Plex match for any copy names every copy
|
||||
t, yr = collections.Counter(known).most_common(1)[0][0]
|
||||
g["plex"] = t
|
||||
g["base"] = sanitize(f"{t} ({yr})" if ok_year(yr) else t)
|
||||
groups.append(g)
|
||||
clash = {b for b, n in collections.Counter(g["base"] for g in groups).items() if n > 1}
|
||||
for g in groups:
|
||||
if g["base"] in clash:
|
||||
# two genuinely different films share title+year: separate folders, and
|
||||
# say so in the name, because Jellyfin can only match one item per folder
|
||||
g["base"] = sanitize(g["base"] + " - " + version_label(g["rows"][0]))
|
||||
g["flag"] = "same title and year as another film"
|
||||
|
||||
moves, mkdirs, problems, readmes = [], set(), [], []
|
||||
# --------------------------------------------------------------- unnamed groups
|
||||
# A movie folder has to READ as a title. The engine's runtime veto says any file
|
||||
# over 45 minutes that is not named like an extra is a film whatever it is called
|
||||
# — true for Jellyfin, but a 118-minute documentary named "Extra 1" then produced
|
||||
# a movie folder named "1", and folders named "1", "2", "3" are Jellyfin items
|
||||
# with no identity whose numeric keys went on to capture every subtitle named
|
||||
# "3_Eng.srt" in the library. A year proves a dated feature ("300 (2006)" is a
|
||||
# real film), so the gate is: a year, or a word that is not release noise. With
|
||||
# neither, the rows are bonus material until a human names them.
|
||||
def named(g):
|
||||
if re.search(r"\(\d{4}\)$", g["base"]):
|
||||
return True
|
||||
return any(w.isalpha() and len(w) >= 2 for w in eng.name_words(g["base"]))
|
||||
|
||||
|
||||
unnamed = [g for g in groups if not named(g)]
|
||||
groups = [g for g in groups if named(g)]
|
||||
for g in unnamed:
|
||||
for r in g["rows"]:
|
||||
extras.add(r["path"])
|
||||
problems.append(
|
||||
f"UNNAMED film {rel(r['path'])} parses to {g['base']!r} with no year: "
|
||||
f"bonus pile, not a folder — give the file a real name and re-plan")
|
||||
|
||||
|
||||
group_of = {}
|
||||
for g in groups:
|
||||
g["dir"] = f"{ROOT}/{g['base']}"
|
||||
for r in g["rows"]:
|
||||
group_of[r["path"]] = g
|
||||
dr = disc_root(r["path"])
|
||||
if dr:
|
||||
group_of[dr] = g
|
||||
by_path = {r["path"]: r for r in erows}
|
||||
|
||||
# -------------------------------------------------------------- bookkeeping
|
||||
claimed = {}
|
||||
existing = set(rows_map := {r["path"] for r in rows})
|
||||
|
||||
|
||||
at_target, booked_src = set(), set()
|
||||
|
||||
|
||||
def book(src, dst, kind):
|
||||
if src == dst:
|
||||
at_target.add(src)
|
||||
booked_src.add(src)
|
||||
return
|
||||
if os.environ.get("JF_TRACE") and (src in booked_src or dst.lower() in claimed):
|
||||
why = "rebook" if src in booked_src else "collision"
|
||||
problems.append(f"TRACE {why} kind={kind} src={rel(src)} dst={rel(dst)}")
|
||||
if src in booked_src: # one source may only ever move once
|
||||
problems.append(f"RE-BOOK {rel(src)} -> {rel(dst)} (already planned)")
|
||||
return
|
||||
if dst.lower() in claimed:
|
||||
problems.append(f"COLLISION {rel(dst)}\n from {rel(src)}\n and {rel(claimed[dst.lower()])}")
|
||||
return
|
||||
claimed[dst.lower()] = src
|
||||
booked_src.add(src)
|
||||
comp = dst.split("/")
|
||||
for i in range(1, len(comp)):
|
||||
mkdirs.add("/".join(comp[:i]))
|
||||
moves.append((src, dst, kind))
|
||||
|
||||
|
||||
def uniq_dir(d, stem, ext, hint=""):
|
||||
"""A destination that no other source already took."""
|
||||
cand = f"{d}/{stem}{ext}"
|
||||
if cand.lower() not in claimed:
|
||||
return cand
|
||||
n = 2
|
||||
while True:
|
||||
cand = f"{d}/{stem} {n}{ext}"
|
||||
if cand.lower() not in claimed:
|
||||
return cand
|
||||
n += 1
|
||||
|
||||
|
||||
def media_dir(p):
|
||||
dr = disc_root(p)
|
||||
return os.path.dirname(dr) if dr else os.path.dirname(p)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ films
|
||||
# film / disc moves: the largest copy carries the folder's own name, the rest
|
||||
# become Jellyfin versions; a disc tree moves as a whole into the movie folder
|
||||
|
||||
|
||||
def vext(p):
|
||||
return ext_by.get(p) or os.path.splitext(p)[1].lower()
|
||||
disc_children = collections.defaultdict(list)
|
||||
for q in rows:
|
||||
dr = disc_root(q["path"])
|
||||
if dr:
|
||||
disc_children[dr].append(q["path"])
|
||||
for g in groups:
|
||||
primary_taken = False
|
||||
disc_kinds = set()
|
||||
for r in sorted(g["rows"], key=lambda r: -r["bytes"]):
|
||||
dr = disc_root(r["path"])
|
||||
if dr:
|
||||
inner = os.path.basename(dr).upper()
|
||||
if inner in disc_kinds:
|
||||
problems.append(
|
||||
f"DISC-VERSION {g['base']}: extra disc tree {rel(dr)} cannot "
|
||||
f"share the folder — discs have no multiple versions")
|
||||
continue
|
||||
disc_kinds.add(inner)
|
||||
for src in disc_children[dr]:
|
||||
tail = os.path.relpath(src, dr)
|
||||
fold = disc_root(src, fold=False)
|
||||
if fold != dr and os.path.dirname(src) != dr:
|
||||
tail = os.path.basename(dr) + "/" + tail
|
||||
book(src, f"{g['dir']}/{tail}", "disc-file")
|
||||
continue
|
||||
label = "" if not primary_taken else f" - {version_label(r)}"
|
||||
dst = f"{g['dir']}/{g['base']}{label}{vext(r['path'])}"
|
||||
n = 2
|
||||
while dst.lower() in claimed:
|
||||
dst = f"{g['dir']}/{g['base']}{label} {n}{vext(r['path'])}"
|
||||
n += 1
|
||||
r["dst"] = dst
|
||||
primary_taken = True
|
||||
book(r["path"], dst, "film")
|
||||
g["primary"] = primary_taken
|
||||
|
||||
# ------------------------------------------------------------------ extras
|
||||
EXTRA_KIND = [("trailers", r"trailer|teaser|preview|promo|tv spot|advert"),
|
||||
("deleted scenes", r"deleted|alt scene|gag reel|blooper|outtake"),
|
||||
("interviews", r"interview|cast and crew|session piece|commentary"),
|
||||
("featurettes", r"featurette|behind the scenes|making of|documentary"
|
||||
r"|heritage|archival|video archive|music video"),
|
||||
("samples", r"sample"),
|
||||
("shorts", r"\bshort\b|short film"),
|
||||
("clips", r"\bclip\b|scene selection|montage|epilogue|isolated score"),
|
||||
("other", r"menu|logo|gallery|still|trivia|storyboard|screen test"
|
||||
r"|audition|test footage|watermark|workprint|poster"
|
||||
r"|picture in picture|singalong|karaoke|original score")]
|
||||
THEME_RE = re.compile(r"theme|opening song|ending song|soundtrack|score"
|
||||
r"|\bost\b|music|song", re.I)
|
||||
BACKDROP_RE = re.compile(r"backdrop|intro", re.I)
|
||||
|
||||
|
||||
def extras_kind(path):
|
||||
low = path.lower()
|
||||
if THEME_RE.search(low):
|
||||
return "theme-music"
|
||||
if BACKDROP_RE.search(low):
|
||||
return "backdrops"
|
||||
for name, rx in EXTRA_KIND:
|
||||
if re.search(rx, low):
|
||||
return name
|
||||
return "extras"
|
||||
|
||||
|
||||
def attachable_title(path):
|
||||
t, _y = eng.title_of(path)
|
||||
return eng.key_of(t) if t and not eng.is_watermark(path, size_by.get(path, 0)) else None
|
||||
|
||||
|
||||
for g in groups:
|
||||
g["tops"] = {parts_of(r["path"])[0] for r in g["rows"]}
|
||||
top_groups = collections.defaultdict(list)
|
||||
for g in groups:
|
||||
for t in g["tops"]:
|
||||
top_groups[t].append(g)
|
||||
group_by_key = collections.defaultdict(list)
|
||||
for g in groups:
|
||||
for r in g["rows"]:
|
||||
group_by_key[eng.key_of(r["title"])].append(g)
|
||||
group_by_dir = {}
|
||||
for g in groups:
|
||||
for r in g["rows"]:
|
||||
group_by_dir.setdefault(media_dir(r["path"]), g)
|
||||
dr = disc_root(r["path"])
|
||||
if dr:
|
||||
group_by_dir.setdefault(os.path.dirname(dr), g)
|
||||
|
||||
|
||||
def group_words(g):
|
||||
"""The words that identify a film group: its titles and the names of the
|
||||
folders its files live in."""
|
||||
if "words" not in g:
|
||||
w = set()
|
||||
if g.get("plex"):
|
||||
w |= words_of(g["plex"])
|
||||
for r in g["rows"]:
|
||||
w |= words_of(r["title"])
|
||||
w |= words_of(os.path.basename(media_dir(r["path"])))
|
||||
g["words"] = w
|
||||
return g["words"]
|
||||
|
||||
|
||||
def score_words(text, groups):
|
||||
"""Ranks film groups by how many of their words a path carries."""
|
||||
words = words_of(text)
|
||||
out = [(len(words & group_words(g)), g) for g in groups]
|
||||
out.sort(key=lambda x: -x[0])
|
||||
return out
|
||||
|
||||
|
||||
def group_for_orphan(path):
|
||||
"""Which movie folder an extra/sidecar belongs to. A collection folder holding
|
||||
several films must be resolved by WORDS, never by position: taking the first
|
||||
film of the folder attached every loose subtitle in a box set to whichever
|
||||
movie happened to sort first."""
|
||||
stem = os.path.splitext(os.path.basename(path))[0]
|
||||
key = attachable_title(path)
|
||||
if key and group_by_key.get(key):
|
||||
cands = group_by_key[key]
|
||||
top = parts_of(path)[0] if len(parts_of(path)) > 1 else ""
|
||||
same_top = [g for g in cands if any(parts_of(r["path"])[0] == top
|
||||
for r in g["rows"] if len(parts_of(r["path"])) > 1)]
|
||||
pool = same_top or cands
|
||||
return pool[0] if len(pool) == 1 else score_words(stem, pool)[0][1]
|
||||
gtop = parts_of(path)[0] if len(parts_of(path)) > 1 else ""
|
||||
pool = top_groups.get(gtop, [])
|
||||
if len(pool) > 1: # a loose collection folder (box set, bonus
|
||||
n, g = score_words(stem + " " + os.path.dirname(path), pool)[0]
|
||||
return g if n >= 2 else None # album) is resolved by words, not position
|
||||
if len(pool) == 1:
|
||||
return pool[0]
|
||||
d = os.path.dirname(path)
|
||||
while d and d != ROOT:
|
||||
if d in group_by_dir:
|
||||
return group_by_dir[d]
|
||||
d = os.path.dirname(d)
|
||||
return None
|
||||
|
||||
|
||||
tv_tops = {parts_of(p)[0] for p in tv_set}
|
||||
def film_folder_for(path):
|
||||
"""A movie folder for a lone feature that has no group: derived from its own
|
||||
title when a year proves it, else None (goes to the review pile)."""
|
||||
t, y = eng.title_of(path)
|
||||
if not t or y == "?":
|
||||
return None
|
||||
d = uniq_dir(ROOT, sanitize(f"{t} ({y})"), "", "")
|
||||
mkdirs.add(d)
|
||||
return d
|
||||
|
||||
|
||||
for p in sorted(extras):
|
||||
if disc_root(p):
|
||||
continue
|
||||
if len(parts_of(p)) > 1 and parts_of(p)[0] in tv_tops:
|
||||
continue # a special inside a TV set: staged with it
|
||||
g = group_for_orphan(p)
|
||||
ext = ext_by.get(p, os.path.splitext(p)[1].lower())
|
||||
stem = os.path.splitext(os.path.basename(p))[0]
|
||||
if g is None:
|
||||
solo = film_folder_for(p)
|
||||
if solo:
|
||||
dst = f"{solo}/{sanitize(stem)}{ext}"
|
||||
problems.append(f"REVIEW lone feature {rel(p)}: gets its own folder "
|
||||
f"{os.path.basename(solo)}")
|
||||
else:
|
||||
dst = f"{ROOT}/_unsorted bonus/{extras_kind(p)}/{sanitize(stem)}{ext}"
|
||||
problems.append(f"NO-FILM extra {rel(p)}")
|
||||
else:
|
||||
dst = f"{g['dir']}/{extras_kind(p)}/{sanitize(stem)}{ext}"
|
||||
book(p, uniq_dir(os.path.dirname(dst), os.path.basename(dst)[:-(len(ext) or 1)],
|
||||
ext, os.path.dirname(p)), "extra")
|
||||
|
||||
# Disc trees the engine never saw (a MakeMKV scratch folder, an AACS remnant):
|
||||
# they belong to the film of the folder they sit in, kept as a named subtree.
|
||||
for q in rows:
|
||||
pth = q["path"]
|
||||
dr = disc_root(pth)
|
||||
if not dr or pth in by_path or pth in booked_src:
|
||||
continue
|
||||
g = group_by_dir.get(dr) or (top_groups[parts_of(pth)[0]][0]
|
||||
if len(top_groups.get(parts_of(pth)[0], [])) == 1 else None)
|
||||
if g is None:
|
||||
continue
|
||||
inner = os.path.basename(dr)
|
||||
problems.append(f"REVIEW disc remnant {rel(dr)} moves inside {g['base']} "
|
||||
f"(a second master: compare before keeping both)")
|
||||
for src in disc_children[dr]:
|
||||
if src not in booked_src:
|
||||
book(src, f"{g['dir']}/{inner}/{os.path.relpath(src, dr)}", "disc-file")
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- sidecars
|
||||
media_by_dir = collections.defaultdict(list)
|
||||
for r in erows:
|
||||
if r["path"] in marks:
|
||||
continue
|
||||
dst = r.get("dst")
|
||||
if dst:
|
||||
media_by_dir[media_dir(r["path"])].append(r)
|
||||
|
||||
|
||||
|
||||
def sidecar_target(path, e):
|
||||
d = os.path.dirname(path)
|
||||
stem = os.path.splitext(os.path.basename(path))[0]
|
||||
if d == ROOT:
|
||||
# A file loose in the library ROOT names no film. media_by_dir[ROOT] is the
|
||||
# set of root-level MOVIE files, and scoring a stray name against it
|
||||
# attached every root-level leftover to whichever root-level film was
|
||||
# largest — the plan sent ".DS_Store" into "Titanic (1997)/other/" that
|
||||
# way. A root-level stray is nobody's sidecar: it goes to the review pile.
|
||||
return None, None
|
||||
cands = media_by_dir.get(d, [])
|
||||
if not cands:
|
||||
nd = os.path.dirname(d) # a subs/ or Extras/ folder beside the media
|
||||
while nd and nd != ROOT and not media_by_dir.get(nd):
|
||||
nd = os.path.dirname(nd)
|
||||
if media_by_dir.get(nd):
|
||||
if nd != d: # a NAMED subfolder (Subs/, Extras/, the
|
||||
g = group_for_orphan(path) # box set's own root) carries evidence
|
||||
dirs = {os.path.dirname(r.get("dst", "")) for r in media_by_dir[nd]}
|
||||
if g is None or g["dir"] not in dirs:
|
||||
# ...but only a STRONG word match outranks the folder the file
|
||||
# sits in. A weak one is an accident: "3_Eng.srt" scored 1
|
||||
# against a junk group named "3" and escaped its own movie's
|
||||
# Subs/ folder, while "03 xXx 3 Return Of Xander Cage" scores 4
|
||||
# and genuinely belongs to the other film of the set.
|
||||
n = score_words(stem, [g])[0][0] if g else 0
|
||||
if n >= 2:
|
||||
return (g, None)
|
||||
cands = media_by_dir[nd]
|
||||
if not cands:
|
||||
g = group_for_orphan(path)
|
||||
return (g, None) if g else (None, None)
|
||||
if len(cands) > 1:
|
||||
w = [t for t in eng.norm(stem).split() if t not in NOISE]
|
||||
cands = sorted(cands, key=lambda r: (
|
||||
-len([t for t in eng.norm(os.path.basename(r["path"])).split() if t in w]),
|
||||
-r["bytes"]))
|
||||
return (None, cands[0])
|
||||
|
||||
|
||||
for q in rows:
|
||||
p, e = q["path"], q["ext"]
|
||||
if p in by_path or p in extras or p in marks or is_qb(p) or disc_root(p):
|
||||
continue
|
||||
if os.path.basename(p).startswith("._"):
|
||||
# macOS AppleDouble resource forks: a few hundred bytes carrying a name
|
||||
# COPIED FROM ANOTHER FILM. Word-matching them to "their" film scattered
|
||||
# dead files through four movie folders (and they are video extensions in
|
||||
# the AppleDouble case, so they also skipped every other filter). They
|
||||
# claim nothing: one pile, and they stay in it — nothing is deleted here.
|
||||
book(p, uniq_dir(f"{ROOT}/_macos sidecars", sanitize(os.path.basename(p))[:120],
|
||||
"", os.path.dirname(p)), "junk")
|
||||
problems.append(f"JUNK AppleDouble sidecar {rel(p)} (macOS resource fork)")
|
||||
continue
|
||||
if e == ".iso": # a DVD image: Jellyfin plays it as a movie file
|
||||
g = group_for_orphan(p)
|
||||
stem = os.path.splitext(os.path.basename(p))[0]
|
||||
if g is not None:
|
||||
book(p, uniq_dir(g["dir"], g["base"], e, stem) if g.get("primary") is not None
|
||||
else f"{g['dir']}/{g['base']}{e}", "iso")
|
||||
else:
|
||||
solo = film_folder_for(p)
|
||||
if solo:
|
||||
book(p, f"{solo}/{sanitize(stem)}.iso", "iso")
|
||||
problems.append(f"REVIEW disc image {rel(p)}: DVD rip, alone in its "
|
||||
f"folder -> {os.path.basename(solo)}")
|
||||
else:
|
||||
book(p, f"{ROOT}/_unsorted files/{sanitize(os.path.basename(p))}", "orphan")
|
||||
problems.append(f"NO-FILM disc image {rel(p)}")
|
||||
continue
|
||||
if e in VIDEO_EXT:
|
||||
# A video the engine never probed (unreadable header) and not an AppleDouble
|
||||
# remnant: the plan does not guess at it, so it stays where it is.
|
||||
continue
|
||||
stem = os.path.splitext(os.path.basename(p))[0]
|
||||
g, media = sidecar_target(p, e)
|
||||
if media is not None:
|
||||
base = os.path.splitext(os.path.basename(media["dst"]))[0]
|
||||
d = os.path.dirname(media["dst"])
|
||||
elif g is not None:
|
||||
base, d = g["base"], g["dir"]
|
||||
else:
|
||||
# Nothing names this file's film. NEVER scatter it: an archive set or an
|
||||
# album tree must move as the unit it is, so the whole loose folder goes
|
||||
# to the review pile and a film folder's own leftovers stay with it.
|
||||
ps = parts_of(p)
|
||||
if len(ps) < 2:
|
||||
dst = f"{ROOT}/_unsorted files/{sanitize(stem)}{e}"
|
||||
elif len(top_groups.get(ps[0], [])):
|
||||
dst = f"{top_groups[ps[0]][0]['dir']}/other/{'/'.join(sanitize(c) for c in ps[1:])}"
|
||||
else:
|
||||
dst = f"{ROOT}/_unsorted files/{'/'.join(sanitize(c) for c in ps)}"
|
||||
book(dst=dst, src=p, kind="orphan")
|
||||
problems.append(f"NO-OWNER sidecar {rel(p)}")
|
||||
continue
|
||||
if e in IMG_EXT:
|
||||
low = eng.norm(stem)
|
||||
word = next((w for w in IMG_PRIMARY if re.search(rf"\b{w}\b", low)), None)
|
||||
kind = "poster" if word else next(
|
||||
(IMG_OTHER[w] for w in IMG_OTHER if re.search(rf"\b{w}\b", low)), None)
|
||||
if re.search(r"\btheme\b", low):
|
||||
dst = f"{d}/theme{e}"
|
||||
else:
|
||||
dst = f"{d}/{base}-{kind or 'art'}{e}" if kind else f"{d}/{sanitize(stem)}{e}"
|
||||
book(p, uniq_dir(os.path.dirname(dst), os.path.basename(dst)[:-(len(e) or 1)],
|
||||
e, os.path.dirname(p)), "image")
|
||||
elif e in NFO_EXT:
|
||||
book(p, uniq_dir(d, base, ".nfo", os.path.dirname(p)), "nfo")
|
||||
elif e in SUB_EXT | AUDIO_EXT:
|
||||
toks = [t for t in re.split(r"[._\-\s]+", stem) if t]
|
||||
langs = [t.lower() for t in toks if t.lower() in LANGS]
|
||||
flags = [t.lower() for t in toks if t.lower() in FLAGS]
|
||||
keep = [sanitize(t) for t in toks
|
||||
if t.lower() not in langs + flags + list(NOISE)
|
||||
and not re.fullmatch(r"\d{3,4}p?|cd\d+|\d{4}|\d+ch|\d+.\d", t.lower())]
|
||||
bits = [base] + ([sanitize(" ".join(keep[:4]))] if keep else []) + langs + flags
|
||||
if e in SUB_EXT:
|
||||
# subtitles: <movie>/srt[/language]
|
||||
d = f"{d}/srt"
|
||||
pd = os.path.basename(os.path.dirname(p))
|
||||
if SUB_KEEP_RE.match(pd) and not SUB_DIR_OK.match(pd):
|
||||
d = f"{d}/{sanitize(pd)}" # a real language folder survives
|
||||
elif e in AUDIO_EXT and AUDIO_DIR_RE.search(os.path.dirname(p)):
|
||||
d = f"{d}/theme-music" # bonus audio: one canonical folder
|
||||
pd = os.path.basename(os.path.dirname(p))
|
||||
if not MUSIC_TOP_RE.search(pd): # an album folder keeps its name
|
||||
d = f"{d}/{sanitize(pd)}"
|
||||
else:
|
||||
pd = os.path.basename(os.path.dirname(p))
|
||||
if not AUDIO_DIR_OK.match(pd):
|
||||
d = f"{d}/{sanitize(pd)}"
|
||||
stem2 = ".".join(bits) or sanitize(os.path.basename(p))
|
||||
book(p, uniq_dir(d, stem2, e, os.path.basename(p)), "track")
|
||||
else:
|
||||
book(p, uniq_dir(f"{d}/other", sanitize(os.path.basename(p))[:120], e,
|
||||
os.path.dirname(p)), "misc")
|
||||
|
||||
readmes.append(f"README in {ROOT}: this library was relaid out into Jellyfin's "
|
||||
"one-folder-per-movie layout; `_release-group watermarks/`, "
|
||||
"`_unsorted files/` and `_unsorted bonus/` hold files no film claims.")
|
||||
for key in tv_keys:
|
||||
readmes.append(f"README in {ROOT}/{sanitize('_TV needs a TV library - ' + key)}: "
|
||||
"these are TV episodes found in the MOVIES library; move the folder "
|
||||
"to a TV library source, else Jellyfin groups them wrongly as one "
|
||||
"movie.")
|
||||
|
||||
# --------------------------------------------------------------------- TV
|
||||
for key in tv_keys:
|
||||
items = eps[key]
|
||||
year = collections.Counter(r["year"] for r in items if r["year"] != "?")
|
||||
yr = year.most_common(1)[0][0] if year else None
|
||||
base = f"_TV needs a TV library - {sanitize(key)}" + (f" ({yr})" if yr else "")
|
||||
tops = {parts_of(r["path"])[0] for r in items}
|
||||
members = sorted([r["path"] for r in items]
|
||||
+ [q for q in extras | marks if len(parts_of(q)) > 1
|
||||
and parts_of(q)[0] in tops],
|
||||
key=lambda q: q)
|
||||
for src in members:
|
||||
rel_parts = rel(src).split("/")
|
||||
tail = "/".join(sanitize(c) for c in rel_parts[:-1] + [rel_parts[-1]])
|
||||
book(src, f"{ROOT}/{base}/{tail}", "tv")
|
||||
for p in sorted(marks):
|
||||
stem, ext = os.path.splitext(os.path.basename(p))
|
||||
book(p, uniq_dir(f"{ROOT}/_release-group watermarks", sanitize(stem)[:120], ext,
|
||||
os.path.dirname(p)), "watermark")
|
||||
for q in rows:
|
||||
if is_qb(q["path"]):
|
||||
problems.append(f"SKIPPED qBittorrent partial {rel(q['path'])}")
|
||||
book(q["path"], q["path"], "qb-partial") # stays: resume data
|
||||
|
||||
# --------------------------------------------------------------- debug hook
|
||||
if os.environ.get("JF_ISO"):
|
||||
x = ROOT + "/High Anxiety (Mel Brooks) [1977] nordic subs DVDrip/High Anxiety [1977].iso"
|
||||
print("attachable", attachable_title(x), "| dir", os.path.dirname(x) in group_by_dir,
|
||||
"| g", group_for_orphan(x) and group_for_orphan(x)["base"])
|
||||
raise SystemExit
|
||||
if os.environ.get("JF_DEBUG"):
|
||||
import json
|
||||
unp = sorted({r["path"] for r in rows} - {m[0] for m in moves} - at_target)
|
||||
print("at_target", len(at_target))
|
||||
print("kinds-unplanned", collections.Counter(os.path.splitext(u)[1] for u in unp).most_common(8))
|
||||
for up in unp[:20]:
|
||||
d = os.path.dirname(up)
|
||||
print("UNP", up[:90], "| mediadir?", d in media_by_dir, "| by_path", up in by_path,
|
||||
"| extras", up in extras, "| discroot", disc_root(up), "| ext", os.path.splitext(up)[1])
|
||||
|
||||
# -------------------------------------------------------------- validation
|
||||
srcs = [m[0] for m in moves]
|
||||
dup_src = {s for s, n in collections.Counter(srcs).items() if n > 1}
|
||||
for s in dup_src:
|
||||
problems.append(f"DOUBLE-CLAIM source {rel(s)}")
|
||||
all_src = set(srcs)
|
||||
untouched = {r["path"] for r in rows} - all_src
|
||||
bad_names, into_alive, lost = [], [], []
|
||||
for src, dst, kind in moves:
|
||||
for comp in dst[len(ROOT) + 1:].split("/"):
|
||||
if (BAD_CHARS.search(comp) or comp != comp.strip(". ") or not comp
|
||||
or len(comp) > 200):
|
||||
bad_names.append(f"{kind}: {comp!r} in {rel(dst)}")
|
||||
if dst in untouched: # something staying put is there
|
||||
into_alive.append(f"{rel(src)} -> {rel(dst)} (occupied by a file that stays)")
|
||||
if dst.lower() in {s.lower() for s in untouched}:
|
||||
into_alive.append(f"{rel(src)} -> {rel(dst)} (case-collides with a survivor)")
|
||||
if dst.startswith(src + "/") and src in all_src:
|
||||
lost.append(f"{rel(src)} moves into its own subtree")
|
||||
# every accounted-for source must exist exactly once in the plan
|
||||
covered = {r["path"] for r in rows}
|
||||
not_planned = sorted(covered - all_src - at_target)
|
||||
|
||||
with open(PLAN, "w") as out:
|
||||
out.write("action\tsrc\tdst\tkind\n")
|
||||
for d in sorted(mkdirs, key=lambda q: (q.count("/"), q)):
|
||||
if d.startswith(ROOT + "/") and d != ROOT:
|
||||
out.write(f"mkdir\t\t{d}\tdir\n")
|
||||
for p in sorted(booked_src - {m[0] for m in moves}):
|
||||
out.write(f"keep\t{p}\t{p}\n")
|
||||
for src, dst, kind in sorted(moves, key=lambda m: m[0]):
|
||||
out.write(f"move\t{src}\t{dst}\t{kind}\n")
|
||||
|
||||
kinds = collections.Counter(k for _s, _d, k in moves)
|
||||
films_n = sum(1 for _s, _d, k in moves if k == "film")
|
||||
folders = len({os.path.dirname(d) for _s, d, k in moves if k in ("film", "disc-file")})
|
||||
with open(EXCEPTS, "w") as out:
|
||||
out.write(f"groups={len(groups)} moves={len(moves)} folders={folders}\n")
|
||||
out.write(f"kinds={dict(kinds)}\n")
|
||||
out.write(f"at-target-already={len(at_target)} unplanned={len(not_planned)}\n")
|
||||
out.write("\n=== WHAT THE EXECUTOR MUST ALSO WRITE ===\n")
|
||||
out.write("\n".join(sorted(set(readmes))) + "\n")
|
||||
out.write("\n=== PROBLEMS ===\n")
|
||||
out.write("\n".join(sorted(set(problems))) + "\n")
|
||||
out.write("\n=== BAD NAMES ===\n" + "\n".join(sorted(set(bad_names))) + "\n")
|
||||
out.write("\n=== TARGETS OCCUPIED BY SURVIVORS ===\n"
|
||||
+ "\n".join(sorted(set(into_alive))) + "\n")
|
||||
out.write("\n=== LEFTOVER SOURCES (first 60) ===\n"
|
||||
+ "\n".join(rel(p) for p in not_planned[:60]) + "\n")
|
||||
print(f"groups={len(groups)} folders={folders} moves={len(moves)} "
|
||||
f"mkdirs={len(mkdirs)} film-files={films_n}")
|
||||
print(f"kinds={dict(kinds)}")
|
||||
print(f"problems={len(set(problems))} bad-names={len(set(bad_names))} "
|
||||
f"occupied={len(set(into_alive))} double-claim={len(dup_src)}")
|
||||
print(f"left-unmoved={len(not_planned)}")
|
||||
print(f"tv-keys={len(tv_keys)} extras={len(extras)} marks={len(marks)} "
|
||||
f"disc-trees={len(disc_children)}")
|
||||
@@ -0,0 +1,35 @@
|
||||
# The operator's tvdb id map: `folder <TAB> tvdb id <TAB> note`.
|
||||
#
|
||||
# These are the pack folders whose show tvdb's NAME SEARCH cannot reach: the
|
||||
# record exists, but tvdb titles it in the show's own language and the English
|
||||
# name rides in aliases the search never reads. Without a row here the folder
|
||||
# keeps the release folder's words and no episode title; with one, both matchers
|
||||
# read `[tvdbid-NNNN]` and the files rename to tvdb's own name and titles.
|
||||
#
|
||||
# Every id was read from tvdb's own record (`/series/<id>/extended`) on
|
||||
# 2026-10-31, and the note states what that record says plus the neighbouring
|
||||
# record a name search would have picked instead — the row is checkable without
|
||||
# re-searching, because one wrong id renames a whole pack to the wrong show. The
|
||||
# plan refuses a line whose id is not a number and lists every folder the map
|
||||
# touched in the exceptions file.
|
||||
#
|
||||
# Folders the name search DOES reach (the EXACT and PUNCT classes of the proposal
|
||||
# table) stay off this file on purpose: a map row would hide a live lookup behind
|
||||
# a number nobody re-checks.
|
||||
[Anime Time] Pokemon Complete Series (Season 01-25+Movies+Specials+Shorts+OST) [DVD+BD][1080p][HEVC 10bit x265][AAC] 76703 tvdb: ポケットモンスター (1997), the umbrella record for every era this pack spans; 1268 files
|
||||
[Anime Time] One Piece (0001-1071+Movies+Specials) [BD+CR] [Dual Audio] [1080p][HEVC 10bit x265][AAC][Multi Sub] 81797 tvdb: ワンピース (1999), alias ONE PIECE; the bare alias "One Piece" belongs to 392276, the 2023 live action; 1072 files
|
||||
[Judas] Naruto (Complete Series) [BD 1080p][HEVC x265 10bit][Multi-Audio][Multi-Subs] 78857 tvdb: NARUTO-ナルト- (2002), alias "Naruto" carries the full-width dashes; 220 files
|
||||
Star Trek Deep Space 9 Deluxe DVD Boxset + Extras DVDRip TSV 72073 tvdb: Star Trek: Deep Space Nine (1993), alias "Star Trek Deep Space 9"; the box set's own words match nothing; 279 files
|
||||
Top Gear UK 1-17 74608 tvdb: Top Gear (2002), no "UK" alias — the US version is 199051; 131 files
|
||||
Black Clover 001 331753 tvdb: ブラッククローバー (2017), which holds NO alias at all; 378762 is Mugyutto! Black Clover (2019), a different show; 108 files
|
||||
Shin Chan - Complete Eng Dub Series 2160p Ai Upscaled -Mesc 437356 tvdb: Shin Chan (2006) — the English dub record this pack is; 79654 is the 1992 Japanese original; 78 files
|
||||
Fairy Tail (2009-2014) (Seasons 1-8 + OVAs) (BD 1080p)(HEVC x265 10bit)(Dual-Audio)(Eng-Subs)-Judas[TGx] 114801 tvdb: フェアリーテイル (2009), alias FAIRY TAIL; 410031 is FAIRY TAIL 100 NEN QUEST (2024), the sequel; 59 files
|
||||
(CBB) Tokyo Ghoul - The Complete Series (BD 1080p)(Dual-Audio)(HEVC-10bit-x265) 281249 tvdb: 東京喰種トーキョーグール (2014) — :re and √A are seasons of THIS record, confirmed by search; 58 files
|
||||
[Anime Time] Demon Slayer - Kimetsu no Yaiba (S01+S02+S03+S04+Mugen Train Arc+Movie) [BD] [Dual Audio][1080p][HEVC 10bit x265][AAC][Eng Sub] 348545 tvdb: 鬼滅の刃 (2019), alias "Demon Slayer: Kimetsu no Yaiba" — the folder's hyphen splits it differently; 56 files
|
||||
(CBB) Tengen Toppa Gurren Lagann - The Complete Series (BD 1080p)(Dual-Audio)(HEVC-x265-10bit) 80096 tvdb: 天元突破グレンラガン (2007), alias Tengen Toppa Gurren Lagann; 51 files
|
||||
Fullmetal Alchemist - Brotherhood 85249 tvdb: 鋼の錬金術師 FULLMETAL ALCHEMIST (2009), alias "Full Metal Alchemist Brotherhood" — spaced, unlike this folder; 64 files
|
||||
[NewbSubs] Attack on Titan~Shingeki no Kyojin Series (1080p Blu-ray x265 Dual Audio) 267440 tvdb: 進撃の巨人 (2013), aliases carry "Attack on Titan" and "Shingeki no Kyojin", joined here by a tilde; 27 files
|
||||
Kimetsu no Yaiba 348545 the same record as the Anime Time pack above: two release folders, ONE show; 12 files
|
||||
Rick and Morty The Anime 420279 tvdb: リック・アンド・モーティ:ジ・アニメ (2024), no English alias on the record; 10 files
|
||||
[Anime Time] Dorohedoro (Season 1+OVA+Extra) [Dual Audio][BD][1080p][HEVC 10bit x265][AAC][Eng Sub] 370761 tvdb: ドロヘドロ (2020), alias Dorohedoro; 9 files
|
||||
Ken Il Guerriero HOKUTO NO KEN 79156 tvdb: 北斗の拳 (1984), alias "Hokuto no Ken"; the Italian title matches nothing and 465216 is the 2026 remake; 5 files
|
||||
|
Can't render this file because it contains an unexpected character in line 20 and column 182.
|
Executable
+306
@@ -0,0 +1,306 @@
|
||||
#!/usr/bin/env python3
|
||||
r"""Ask tvdb what it calls the shows a plan renamed WITHOUT a tvdb id.
|
||||
|
||||
media-shows-tvdb.py [--plan TSV] [--root DIR] [--park DIR] [--map TSV]
|
||||
[--out TSV] [--credentials FILE]
|
||||
|
||||
`media-shows.py` renames a series to tvdb's own words only when an id backs the
|
||||
name; everything else keeps the metadata provider's words and no episode title,
|
||||
and the plan's mkdir rows are where those folders show up. This script turns that
|
||||
list into a decision sheet: one row per destination series folder that carries no
|
||||
`[tvdbid-NNNN]`, with what tvdb's search answers for it and a verdict that says
|
||||
HOW MUCH was guessed to get there.
|
||||
|
||||
It changes nothing. The sheet is for the operator; the file that changes
|
||||
behaviour is `scripts/media-shows-tvdb-ids.tsv`, whose rows he approves. That is
|
||||
the loop: plan -> this sheet -> approved map rows -> re-plan.
|
||||
|
||||
Verdicts, weakest last:
|
||||
|
||||
EXACT tvdb's own name equals the folder's words as they stand.
|
||||
PUNCT it equals them only after release junk is stripped (pack numbers,
|
||||
codecs, group names, `Complete Series`); the difference is spelling,
|
||||
not identity.
|
||||
NEAR tvdb answered something whose name is NOT the query. The best hit is
|
||||
printed so a human can judge it, and it is never applied by anything.
|
||||
NONE nothing left to ask with, or no answer at all.
|
||||
|
||||
Acceptance is deliberately unforgiving — a hit counts only when tvdb's name
|
||||
equals the query with every non-alphanumeric dropped from both sides, and the
|
||||
series' year still has to fall inside any year span the folder names. The
|
||||
cleaner is the only thing that got smarter over the passes this tool replaces;
|
||||
the acceptance test never moved, because a wrong id renames a whole pack to the
|
||||
wrong show. A query that needs the cleaner is reported as PUNCT for exactly that
|
||||
reason: the machine stripped words, so a human reads the row.
|
||||
|
||||
Read-only against tvdb. The credential file is the planner's own
|
||||
(`api_key=`, optional `pin=`), read here and never printed.
|
||||
"""
|
||||
import argparse
|
||||
import collections
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
ARGS = argparse.ArgumentParser(description=__doc__.splitlines()[0],
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ARGS.add_argument("--plan", default="/tmp/tv-plan.tsv",
|
||||
help="the plan media-shows.py just wrote (its mkdir rows are "
|
||||
"the folders this sheet asks about)")
|
||||
ARGS.add_argument("--root", default=os.environ.get("TV_ROOT", "/mnt/entropy/TVShows"),
|
||||
help="the TV library the plan renames")
|
||||
ARGS.add_argument("--park", default="",
|
||||
help="the review pile inside it (its folders are not shows)")
|
||||
ARGS.add_argument("--map",
|
||||
default=os.path.join(os.path.dirname(os.path.abspath(__file__)),
|
||||
"media-shows-tvdb-ids.tsv"),
|
||||
help="the operator's approved map; its folders are reported as "
|
||||
"MAPPED and never asked again")
|
||||
ARGS.add_argument("--credentials",
|
||||
default=os.path.expanduser("~/.config/secrets/tvdb"),
|
||||
help="file with an api_key= line (and a pin= for a user-issued "
|
||||
"key); mode 600, and never printed")
|
||||
ARGS.add_argument("--out", default="/tmp/tvdb-proposal.tsv",
|
||||
help="the decision sheet")
|
||||
ARGS = ARGS.parse_args()
|
||||
|
||||
ROOT = os.path.abspath(ARGS.root)
|
||||
PARK = os.path.abspath(ARGS.park or os.path.join(ROOT, "_unsorted"))
|
||||
|
||||
_spec = importlib.util.spec_from_file_location(
|
||||
"eng", os.path.join(os.path.dirname(os.path.abspath(__file__)),
|
||||
"media-titles.py"))
|
||||
eng = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(eng)
|
||||
|
||||
# Release metadata, not the show: everything a scene folder name says about the
|
||||
# ENCODING, the packaging or the languages, which tvdb has never heard of.
|
||||
JUNK = re.compile(
|
||||
r"complete|series|dual|multi|audio|sub|subs|multisub|dualaudio|"
|
||||
r"\d{3,4}p|1080p|720p|2160p|480p|4k|hd|hdtv|cam|hdcam|telesync|x264|x265|hevc|avc|"
|
||||
r"\d+bit|bit|bluray|blu-?ray|bdrip|web-?rip|web-?dl|webrip|hdtvrip|"
|
||||
r"dvdrip|dvdr|dvdscr|tsr|tsv|mkv|avi|mp4|aac|ac3|dts|eac3|flac|ddp?5[.1]1|"
|
||||
r"bd|dvd|rip|boxset|box|season|seasons|episodes?|specials?|shorts|ost|"
|
||||
r"ova|ovas|oad|repack|proper|internal|mi?xed|budgetbits|deluxe|remux|"
|
||||
r"remastered|uncut|collector|edition|extras?|volume|volumes|part|"
|
||||
r"disc|disk|cd|ntsc|pal|region|fixed|patched|proof|english|esp|"
|
||||
r"espagnol|french|vostfr|subtitul\w*|spanish|german|italian|ita|eng|fre|spa|lat|"
|
||||
r"upscaled?|ai|dd|movie|movies",
|
||||
re.I)
|
||||
GROUP = re.compile(r"KONTRAST|TGx|Judas|Silence|FreetheFilms|NTb|DEADPOOL|Xploit|"
|
||||
r"SM737|TSA|3OISL3R|PLAYWEB|EPiC|FLUX|HONEY|Zaxton|Cca3rW|"
|
||||
r"monikeres|frank|hdh|psi|rtfpemaker|cbm|web|asser|astro|"
|
||||
r"yfn|uindex|www|org|com|net|mesc|ataraxiaprime|seph\d*|"
|
||||
r"[a-z]+subs?|edge", re.I)
|
||||
BRACKET = re.compile(r"[\[({]([^\])}]*)[\])}]")
|
||||
NUM_RANGE = re.compile(r"(?<!\w)\d{1,4}\s*(?:-|–|to|through)\s*\d{1,4}(?!\w)", re.I)
|
||||
INSIDE = re.compile(r"all|complete|full|dual|multi|audio|sub|subs|ova|ovas|"
|
||||
r"movies?|episodes?|extras?|bonus|pack|remux|region\s*\d*|"
|
||||
r"bd|dvd|cd|ntsc|pal|hr-?dvd", re.I)
|
||||
|
||||
|
||||
def drop_bracket(m):
|
||||
"""A bracket says nothing when its words are junk; keep it when they aren't."""
|
||||
toks = [t for t in re.split(r"[\s_.,+~&]+", m.group(1)) if t]
|
||||
if not toks or all(INSIDE.fullmatch(t) or JUNK.fullmatch(t)
|
||||
or GROUP.fullmatch(t)
|
||||
or re.fullmatch(r"\d+[a-z]?", t, re.I)
|
||||
or re.fullmatch(r"(19|20)\d\d", t) for t in toks):
|
||||
return " "
|
||||
return " " + m.group(1) + " "
|
||||
|
||||
|
||||
def clean(name):
|
||||
"""The folder's words with everything about the RELEASE taken out, so what is
|
||||
left is what a person would type into tvdb. `Star Trek Deep Space 9 Deluxe
|
||||
DVD Boxset + Extras DVDRip TSV` becomes `Star Trek Deep Space 9`; `SNL 16-20`
|
||||
becomes `SNL`."""
|
||||
name = re.sub(r"^\s*www\b.*?\b(org|com|net)\b\s*[-–]\s*", " ", name, flags=re.I)
|
||||
name = re.sub(r"\bS\d{1,2}([ .Ee]\d+)?[-–Ee\d.]*", " ", name)
|
||||
name = re.sub(r"\b\d{1,2}[xX]\d{1,3}\b", " ", name)
|
||||
name = NUM_RANGE.sub(" ", name)
|
||||
name = re.sub(r"(19|20)\d\d(?:\s*[-/–]\s*(?:19|20)\d\d)?", " ", name)
|
||||
name = re.sub(r"\bby\s+[A-Za-z0-9_.-]+$", " ", name)
|
||||
name = BRACKET.sub(drop_bracket, name)
|
||||
name = re.sub(r"[\[()\]:,;!?*]", " ", name)
|
||||
toks = re.split(r"[\s_.\-+~&]+", name)
|
||||
out = []
|
||||
for i, t in enumerate(toks):
|
||||
if not t or JUNK.fullmatch(t) or GROUP.fullmatch(t):
|
||||
continue
|
||||
if re.fullmatch(r"\d+", t):
|
||||
after = sum(1 for x in toks[i + 1:] if x
|
||||
and not JUNK.fullmatch(x) and not GROUP.fullmatch(x)
|
||||
and not re.fullmatch(r"\d+", x))
|
||||
# A pack index (`001`) and a lead-in number over a real title
|
||||
# (`01 Charles Barkley`) are noise; `30 Rock`, `24`, `Deep Space 9`
|
||||
# are the title itself.
|
||||
if len(t) >= 3 or (not out and after >= 2):
|
||||
continue
|
||||
out.append(t)
|
||||
while out and out[-1].lower() in ("and", "with", "the", "a", "an", "to", "of"):
|
||||
out.pop()
|
||||
return " ".join(out).strip()
|
||||
|
||||
|
||||
def years(name):
|
||||
return [int(y) for y in re.findall(r"\b(19\d\d|20\d\d)\b", name)]
|
||||
|
||||
|
||||
def bare(s):
|
||||
return re.sub(r"[^a-z0-9]", "", s.lower())
|
||||
|
||||
|
||||
CRED = {}
|
||||
try:
|
||||
for _line in open(ARGS.credentials):
|
||||
if "=" in _line:
|
||||
k, v = _line.strip().split("=", 1)
|
||||
CRED[k.strip()] = v.strip()
|
||||
except OSError:
|
||||
print(f"tvdb: no credentials at {ARGS.credentials} — the sheet would be all "
|
||||
"NEAR and NONE, which is not a decision sheet", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
if not CRED.get("api_key"):
|
||||
print(f"tvdb: no api_key in {ARGS.credentials}", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
TOKEN = {"jwt": ""}
|
||||
|
||||
|
||||
def call(path):
|
||||
"""One tvdb v4 GET, logging in again once when the JWT expires mid-run."""
|
||||
for attempt in (1, 2):
|
||||
if not TOKEN["jwt"]:
|
||||
body = {"apiKey": CRED["api_key"]}
|
||||
if CRED.get("pin"):
|
||||
body["token"] = CRED["pin"]
|
||||
req = urllib.request.Request("https://api4.thetvdb.com/v4/login",
|
||||
data=json.dumps(body).encode(),
|
||||
headers={"Content-Type":
|
||||
"application/json"})
|
||||
TOKEN["jwt"] = json.load(
|
||||
urllib.request.urlopen(req, timeout=30))["data"]["token"]
|
||||
try:
|
||||
req = urllib.request.Request("https://api4.thetvdb.com/v4" + path,
|
||||
headers={"Authorization":
|
||||
"Bearer " + TOKEN["jwt"]})
|
||||
return json.load(urllib.request.urlopen(req, timeout=30))["data"]
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 401 and attempt == 1:
|
||||
TOKEN["jwt"] = ""
|
||||
continue
|
||||
raise
|
||||
|
||||
|
||||
MAPPED = {}
|
||||
if ARGS.map and os.path.exists(ARGS.map):
|
||||
for _line in open(ARGS.map):
|
||||
_line = _line.rstrip("\n")
|
||||
if not _line.strip() or _line.startswith("#"):
|
||||
continue
|
||||
cols = _line.split("\t")
|
||||
if len(cols) > 1 and cols[1].strip().isdigit():
|
||||
MAPPED[cols[0].strip()] = cols[1].strip()
|
||||
|
||||
# The folders the plan created inside the library, with the file count each one
|
||||
# will hold. The review pile is not a show: its files belong to no identity, and
|
||||
# naming the pile would put one junk series on the sheet for all of them.
|
||||
shows = collections.Counter()
|
||||
for line in open(ARGS.plan).read().splitlines()[1:]:
|
||||
parts = line.split("\t")
|
||||
if len(parts) < 4 or parts[0] != "mkdir":
|
||||
continue
|
||||
dst = os.path.abspath(parts[2])
|
||||
if not dst.startswith(ROOT + os.sep) or dst.startswith(PARK + os.sep):
|
||||
continue
|
||||
top = os.path.relpath(dst, ROOT).split(os.sep)[0]
|
||||
if top.startswith((".", "_")):
|
||||
continue
|
||||
if "[tvdbid-" in top:
|
||||
continue # an id already backs it
|
||||
shows.setdefault(top, 0)
|
||||
for line in open(ARGS.plan).read().splitlines()[1:]:
|
||||
parts = line.split("\t")
|
||||
if len(parts) < 4 or parts[0] != "move":
|
||||
continue
|
||||
dst = os.path.abspath(parts[2])
|
||||
if not dst.startswith(ROOT + os.sep) or dst.startswith(PARK + os.sep):
|
||||
continue
|
||||
top = os.path.relpath(dst, ROOT).split(os.sep)[0]
|
||||
if "[tvdbid-" in top:
|
||||
continue
|
||||
shows[top] += 1
|
||||
|
||||
tally = collections.Counter()
|
||||
rows = []
|
||||
for folder in sorted(shows, key=lambda f: (-shows[f], f)):
|
||||
files, tag, note = shows[folder], "NONE", ""
|
||||
if folder in MAPPED:
|
||||
rows.append([files, "MAPPED", folder, "", "", "", MAPPED[folder],
|
||||
"the operator's map already states this id"])
|
||||
tally["MAPPED"] += 1
|
||||
continue
|
||||
query, untouched = clean(folder), clean(folder) == folder
|
||||
ys = years(folder)
|
||||
if not query:
|
||||
rows.append([files, "NONE", folder, "", "", "", "",
|
||||
"nothing is left of the folder's name once release junk"
|
||||
" is gone"])
|
||||
else:
|
||||
try:
|
||||
hits = call("/search?query=" + urllib.parse.quote(query)
|
||||
+ "&type=series") or []
|
||||
except Exception as e: # noqa: BLE001
|
||||
hits = []
|
||||
note = f"tvdb did not answer: {e}"
|
||||
want = bare(query)
|
||||
pick = None
|
||||
for h in hits:
|
||||
if bare(h.get("name") or "") != want:
|
||||
continue
|
||||
hy = str(h.get("year") or "")
|
||||
if ys and hy.isdigit() and not (min(ys) <= int(hy) <= max(ys)):
|
||||
continue
|
||||
pick = h
|
||||
break
|
||||
if pick:
|
||||
tag = "EXACT" if untouched else "PUNCT"
|
||||
rows.append([files, tag, folder, query, pick.get("name", ""),
|
||||
str(pick.get("year", "") or ""),
|
||||
str(pick.get("tvdb_id", "") or ""), ""])
|
||||
elif hits:
|
||||
best = hits[0]
|
||||
rows.append([files, "NEAR", folder, query, best.get("name", ""),
|
||||
str(best.get("year", "") or ""),
|
||||
str(best.get("tvdb_id", "") or ""),
|
||||
f"{len(hits)} answer(s); tvdb's own name is not the "
|
||||
"query, so nothing would take it"])
|
||||
else:
|
||||
rows.append([files, "NONE", folder, query, "", "", "",
|
||||
note or "tvdb has no series by that name"])
|
||||
tally[rows[-1][1]] += 1
|
||||
time.sleep(0.12)
|
||||
|
||||
# Two folders mapping to one record is not two shows; it is one show the release
|
||||
# folders split. Say it on every row involved, or the operator approves the same
|
||||
# id twice and reads the second as a mistake.
|
||||
by_id = collections.Counter(r[6] for r in rows if r[6] and r[1] != "MAPPED")
|
||||
for r in rows:
|
||||
if r[6] and r[1] != "MAPPED" and by_id[r[6]] > 1:
|
||||
r[7] = (f"MERGE: {by_id[r[6]]} folders map to tvdb {r[6]} — one show, "
|
||||
"the releases split it; " + r[7]).strip("; ")
|
||||
|
||||
with open(ARGS.out, "w") as fh:
|
||||
fh.write("files\tverdict\tfolder\tquery\ttvdb_name\ttvdb_year\ttvdb_id\tnote\n")
|
||||
for r in rows:
|
||||
fh.write("\t".join(str(c) for c in r) + "\n")
|
||||
print(f"sheet: {len(rows)} folders"
|
||||
+ "".join(f" {t}={n}" for t, n in sorted(tally.items()))
|
||||
+ f" -> {ARGS.out}")
|
||||
Executable
+1235
File diff suppressed because it is too large.
Load diff
Executable
+579
@@ -0,0 +1,579 @@
|
||||
#!/usr/bin/env python3
|
||||
r"""The fleet's release-filename title parser: the grouping authority the relayout
|
||||
plan uses for every file the media server never matched.
|
||||
|
||||
report: media-titles.py report [--probed TSV] [--out TSV]
|
||||
the similar-copies census (DUP / REVIEW / EP-DUP groups) over the
|
||||
ffprobe census scripts/media-census.py writes.
|
||||
parse: media-titles.py parse NAME [NAME…]
|
||||
what title_of() makes of a name — the way to check a naming accident
|
||||
before it merges two films into one folder.
|
||||
|
||||
scripts/media-plan.py imports this file; over there the paths are arguments.
|
||||
|
||||
Method — three independent signals, so a naming accident cannot merge two films:
|
||||
* franchise KEY : first two significant title tokens. Stable across parts of a
|
||||
collection ("resident evil", "jump street") but NOT film-specific — it only
|
||||
selects candidates.
|
||||
* DURATION : the identity check. Same master encodes differ by at most a
|
||||
couple of seconds; different films / cuts do not collide.
|
||||
* SIZE SPREAD + YEAR: decide the tier. Two files same size AND same duration
|
||||
are byte-twins (the hash pass confirms); same duration but different sizes are
|
||||
two different releases of one film (deletable redundancy).
|
||||
|
||||
TV episodes (SxxEyy in the name) are grouped separately: their uniform runtime
|
||||
would otherwise collapse a whole series into one "duplicate" group.
|
||||
"""
|
||||
import argparse
|
||||
import collections, os, re, sys, unicodedata
|
||||
|
||||
# Library root: the folder directly under this one names the film, and every
|
||||
# folder below it is structure (Extras/, Featurettes/, Disc 1/, BDMV/). Tests pass
|
||||
# their own; nothing here reads media, only the census TSVs about it.
|
||||
ROOT = os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies")
|
||||
|
||||
# Release-tag vocabulary: never part of a title, so the title stops here.
|
||||
JUNK = {"5.1", "7.1", "2.0", "1.0", "6.1", "4.0", "extras"}
|
||||
JUNK_RE = re.compile(
|
||||
r"^("
|
||||
r"\d{3,5}p|\d{1,2}k|uhd|fhd|imax|x ?26[45]|hevc|h ?26[45]|avc|av1|x ?av1|vp[89]"
|
||||
r"|divx|xvid|mpeg-?2?|\d{1,2}-?bit|hdr ?10\+?|hdr|dolby ?vision|dvh?|sdr|hybrid"
|
||||
r"|re-?mux|bd-?re-?mux|blu-?ray|bdrip|brrip|bdmux|bd|web-?(rip|dl)?|hdtv|hdrip"
|
||||
r"|dvd-?(rip|r|scr)?|hdcam|telesync|screener|ppv|re-?encode|extras?"
|
||||
r"|ddp? ?\d(\.\d)?|dd\d(\.\d)?|dts-?hd(\.ma)?|dts|truehd|atmos|e-?ac-?3?|ac-?3"
|
||||
r"|aac\d?(\.\d)?|lpcm|flac|opus|mp3|pcm|\d\.\d(\.\d)?|5 ?1|7 ?1|2 ?0|1 ?0|6 ?1|4 ?0"
|
||||
r"|subs?-?(it|en|eng|ita)?|subbed|dual(-?audio)?|multi|dub(bed)?"
|
||||
r"|ita|eng|en|it|de|fr|es|pl|jp|jap|lat|esp|ger|fre|rus|zho|hin|tam|tel"
|
||||
r"|repack|proper|internal|limited|unrated|extended|director'?s|cut|remaster(ed)?|uncut"
|
||||
r"|special|edition|version|crc32|sha1|sfv|par2|nfo|nzb|mux|complete|unpack"
|
||||
r"|certified|scene|p2p|part\d*|cd\d+|disc\d+|disk\d+|vob|idx|srt|ass|ssa|sup"
|
||||
r"|m3u|cue|log|txt|mkv|mp4|avi|imdb\d[\d.]*|yts(\.\w+)?|rarbg|collection"
|
||||
r")$", re.I)
|
||||
RES_RE = re.compile(r"^\d{3,5}p$|^\d{1,2}k$|^\d+x\d+$", re.I)
|
||||
YEAR_RE = re.compile(r"^(19|20)\d{2}$")
|
||||
SEP_RE = re.compile(r"[._\s\-\[\](){}+!~,;:&@#/*'\"`´|]+")
|
||||
IDX_RE = re.compile(r"^\d{1,3}[a-z]?$")
|
||||
WORD_RE = re.compile(r"[A-Za-z0-9']+$")
|
||||
EP_RE = re.compile(r"(?i)(?<![a-z0-9])(s(?:eason[\s._-]*)?\d{1,2}[\s._-]*e(?:p(?:isode)?[\s._-]*)?"
|
||||
r"\d{1,3}|s\d{1,2}\s?e\d{1,3}|\d{1,2}x\d{2,3}|part[\s._-]*\d{1,2})")
|
||||
STOP = {"a", "an", "the", "il", "la", "le", "i", "un", "une", "el", "los", "las", "de",
|
||||
"del", "du", "di", "e", "ed", "y", "and", "et", "for", "of", "to", "in", "on",
|
||||
"vs", "avec", "con"}
|
||||
|
||||
|
||||
def norm(s):
|
||||
s = unicodedata.normalize("NFKD", s)
|
||||
return "".join(c for c in s if not unicodedata.combining(c)).lower().strip()
|
||||
|
||||
|
||||
def fold_sep(s):
|
||||
"""norm(), plus: every character that is left over is not a letter, digit or
|
||||
apostrophe, so it is a SEPARATOR whatever codepoint it hides behind — not part
|
||||
of the word. Release groups file titles with lookalikes for punctuation the
|
||||
filesystem forbids: "V ∕ H ∕ S ∕ 2" carries U+2215 DIVISION SLASH, "S꞉ Viral"
|
||||
U+A789, and en dashes hide in "Star Wars – A New Hope". NFKD does NOT fold
|
||||
those (no decomposition mapping), and the scrubber inside title_of DELETES
|
||||
what it does not recognise, so five different V/H/S films squashed into the one
|
||||
title "vhs2" and would have shared one folder. Deleting is the wrong operation:
|
||||
they separate. Accents are gone by then — norm() removed the combining marks —
|
||||
so "Amélie" stays "amelie" and never splits into "ame lie"."""
|
||||
return " ".join("".join(c if c.isalnum() or c == "'" else " "
|
||||
for c in norm(s)).split())
|
||||
|
||||
def title_of(path):
|
||||
"""(title, year) from the basename.
|
||||
|
||||
Two passes. (1) Tokenize; every 4-digit year becomes a sentinel in place.
|
||||
(2) Find the boundary: the first release-noise token (resolution / codec /
|
||||
source / audio / language / group vocabulary) ends the title, so trailing
|
||||
tags can never leak in. (3) Collect up to that boundary: leading index
|
||||
numbers ("03."), leading year sentinels ("1940 - Fantasia") and leading noise
|
||||
are skipped while nothing is collected; once a real word appears a sentinel
|
||||
CLOSES the title. A title sitting BEFORE the year sentinel ("300.2006.…" or
|
||||
"12. Monkeys. 1995") is a numeric title and is prepended to what follows.
|
||||
"""
|
||||
# fold_sep BEFORE tokenizing, so a lookalike separator separates (see its
|
||||
# docstring): the tokens below are then plain alnum/apostrophe words.
|
||||
name = fold_sep(os.path.splitext(os.path.basename(path))[0])
|
||||
raw = [x for x in SEP_RE.split(name) if x]
|
||||
years = [t for t in raw if YEAR_RE.match(t)]
|
||||
two_year = len(years) > 1 # "2001…1968": one of them is the title
|
||||
toks, year = [], None
|
||||
for t in raw:
|
||||
if YEAR_RE.match(t):
|
||||
year = t # last year wins
|
||||
toks.append("\x00")
|
||||
else:
|
||||
toks.append(t)
|
||||
stop, seen_word = len(toks), False
|
||||
for i, t in enumerate(toks):
|
||||
if t == "\x00":
|
||||
continue
|
||||
c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
|
||||
if (not c or RES_RE.match(c) or JUNK_RE.match(c) or c in JUNK
|
||||
or not WORD_RE.match(c)) and not (IDX_RE.match(t) and not seen_word):
|
||||
stop = i
|
||||
break
|
||||
if c.isalpha():
|
||||
seen_word = True
|
||||
# A leading number is a track index ("01. Bye Bye Baby", "05. Casablanca") unless
|
||||
# it is three digits or more ("007", "000", "1408") or the name carries no year
|
||||
# at all ("12 Monkeys", "831"). A 1-2 digit prefix beside a year is an index;
|
||||
# the rare real title that looks like one ("0 Things I Hate About You (1999)")
|
||||
# loses its zero to the parser — Plex's matched title overrides it anyway.
|
||||
lead = re.split(r"[\s._-]", name, maxsplit=1)[0]
|
||||
track_idx = bool(re.fullmatch(r"\d{1,2}[a-z]?", lead)) and bool(year)
|
||||
keep, num_title = [], None
|
||||
for t in toks[:stop]:
|
||||
if t == "\x00":
|
||||
if keep:
|
||||
break # year closes the title
|
||||
if two_year and t is not None and years[0] != year:
|
||||
# two DIFFERENT years ("2001…1968"): the leading one is the title;
|
||||
# "1940 - Fantasia (1940)" repeats one year, so it is not a title
|
||||
num_title = num_title or years[0]
|
||||
continue # leading year: not the title
|
||||
if track_idx and IDX_RE.match(t) and not keep and t == lead:
|
||||
continue # "03." / "1c." ordering
|
||||
c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
|
||||
if not keep and not c.isalpha():
|
||||
num_title = num_title or (c or None)
|
||||
continue # numeric title candidate
|
||||
keep.append(c)
|
||||
if len(keep) >= 8:
|
||||
break
|
||||
keep = [k for k in keep if not re.fullmatch(r"(disc|disk|cd|dvd|part)\d*", k)]
|
||||
title = " ".join(keep)
|
||||
if not keep:
|
||||
# nothing collected: a pure-numeric leading token is the title even when it
|
||||
# sits where the release year sits ("1917.2019.2160p…"), and "300.2006…" too
|
||||
for t in raw:
|
||||
c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
|
||||
if re.fullmatch(r"(?:[12]\d{3}|\d{1,3})", c) and (c != year or two_year):
|
||||
return c, year or "?" # never "2160p" / "1080": they are not pure digits
|
||||
if (num_title and title and not keep[0].isdigit()
|
||||
and (two_year or not track_idx)):
|
||||
# a leading number that only echoes the release year ("1917.2019.2160p…") is
|
||||
# not a title; one that stands alone ("300.2006…", "1900") is
|
||||
title = f"{num_title} {title}"
|
||||
title = title or num_title or norm(name).replace("\x00", " ").strip()[:40]
|
||||
return title, year or "?"
|
||||
|
||||
|
||||
# Bonus-material vocabulary: names that say nothing about the film, so an extras
|
||||
# file otherwise pairs with an extras file of another movie. "Short", "Spot" and
|
||||
# "Clip" are deliberately absent — they are film titles (Short Circuit, Spot,
|
||||
# Clips) — and the running-time guard in is_extra() carries the rest.
|
||||
SEP_CLS = r"[\s._-]+"
|
||||
|
||||
|
||||
def _bonus_rx(word):
|
||||
return r"(?<![a-z0-9])" + re.sub(r"[ \-]+", lambda _: SEP_CLS, word.lower()) + r"(?![a-z0-9])"
|
||||
|
||||
|
||||
EXTRA_WORDS = [
|
||||
"trailer", "teaser", "featurette", "featurettes", "promo", "promotional",
|
||||
"tv spot", "advert", "commercial", "behind the scenes", "making of",
|
||||
"deleted scene", "deleted scenes", "deleted footage", "gag reel", "blooper",
|
||||
"blooper reel", "bloopers", "outtake", "outtakes", "scene selection", "menu",
|
||||
"menus", "gallery", "still gallery", "stills", "poster", "posters",
|
||||
"montage", "preview", "previews", "epilogue", "visual effects",
|
||||
"music video", "singalong", "karaoke", "trivia", "storyboard", "workprint",
|
||||
"test footage", "screen test", "audition", "cast and crew", "isolated score",
|
||||
"picture in picture", "original score", "session piece", "sample",
|
||||
"watermark",
|
||||
]
|
||||
# Words that describe bonus material which can run LONG (a feature-length
|
||||
# commentary-track cut, a documentary about the film). Trusted only below
|
||||
# LONG_BONUS_MAX, so a 40-minute "commentary" release stays a film.
|
||||
EXTRA_LONG_WORDS = ["interview", "interviews", "commentary", "video archive",
|
||||
"archival", "documentary"]
|
||||
EXTRA_NAME_RE = re.compile(
|
||||
"|".join([_bonus_rx(w) for w in EXTRA_WORDS] + [r"scene[\s._-]?\d{1,3}"]), re.I)
|
||||
EXTRA_LONG_NAME_RE = re.compile(
|
||||
"|".join(_bonus_rx(w) for w in EXTRA_LONG_WORDS), re.I)
|
||||
# Bonus-folder vocabulary. fullmatch = the folder IS an extras subtree; a bare
|
||||
# search = it only mentions extras ("... + Extras" box sets hold feature films).
|
||||
EXTRA_FOLDER_RE = re.compile(
|
||||
r"(?:extras?|bonus(?:[\s._-]material)?|specials?|special[\s._-]featurettes?|"
|
||||
r"featurettes?|deleted(?:[\s._-](?:scenes?|footage))?|trailers?|"
|
||||
r"making[\s._-]of|behind[\s._-]the[\s._-]scenes|shorts?|interviews?|"
|
||||
r"archiv(?:al|es)|video[\s._-]archive)", re.I)
|
||||
VIDEO_SUFFIX = (".mkv", ".mp4", ".avi", ".m4v", ".mov", ".m2ts", ".mpg", ".webm")
|
||||
|
||||
# Anything longer than this is a feature film whatever it is called: a 118-minute
|
||||
# file named "The Menu" or "Trailer" is a movie, not bonus material.
|
||||
SHORT_MAX = 45 * 60
|
||||
|
||||
# Long-shaped bonus vocabulary is only believed below this: a 40-minute
|
||||
# commentary cut is an alternate version of the film, not a featurette.
|
||||
LONG_BONUS_MAX = 25 * 60
|
||||
|
||||
|
||||
def name_words(x):
|
||||
"""Bare words of a file or folder name: no release metadata, no year."""
|
||||
x = os.path.splitext(x)[0] if x.lower().endswith(VIDEO_SUFFIX) else x
|
||||
return [w for w in SEP_RE.split(fold_sep(x))
|
||||
if w and not YEAR_RE.match(w) and not RES_RE.match(w)
|
||||
and not JUNK_RE.match(w) and w not in JUNK]
|
||||
|
||||
|
||||
def mentions(short, long_):
|
||||
"""True when one word list appears in the other, anywhere: a release names
|
||||
the film at the front ("Alien (1979) - Trailer") or in the middle
|
||||
("Movies/Alien (1979)/Extra - Logo.mkv"), so a start-of-string rule fails."""
|
||||
a, b = sorted([short, long_], key=len)
|
||||
if not a:
|
||||
return False
|
||||
return any(b[i:i + len(a)] == a for i in range(len(b) - len(a) + 1))
|
||||
|
||||
|
||||
|
||||
def folder_is_bonus(folder):
|
||||
"""True when a folder name carries bonus vocabulary as WHOLE words: the
|
||||
vocabulary is matched against a padded word string, so "Extras" decides and
|
||||
a word that merely CONTAINS one ("Extraordinary", "The Extras") does not."""
|
||||
return bool(EXTRA_FOLDER_RE.search(" " + " ".join(SEP_RE.split(norm(folder))) + " "))
|
||||
|
||||
|
||||
def is_extra(path, title="", dur=0.0, root=ROOT, matched=False):
|
||||
"""True for bonus material, which must never be counted as a film: a file
|
||||
called "Trailer", "Menu", "Sample" or "Gallery" carries no film identity, so
|
||||
extras otherwise pair with each other across the whole library.
|
||||
|
||||
Evidence, strongest first:
|
||||
* RUNTIME VETOES EVERYTHING. The film "The Menu", the film "Trailer", the
|
||||
franchise "Trailer Park Boys" and "ET — The Extra-Terrestrial" all carry
|
||||
bonus vocabulary and run 100+ minutes: anything over SHORT_MAX is a film —
|
||||
unless it sits in an explicit bonus folder and metadata never matched it.
|
||||
Bonus words that describe long-form material (commentary, documentary,
|
||||
interview) are believed only
|
||||
under LONG_BONUS_MAX, so a feature-length commentary cut stays a film.
|
||||
* The file's OWN name carries bonus vocabulary -> bonus.
|
||||
* ANY folder of the path below the film's own root folder carries bonus
|
||||
vocabulary ("Alien (1979)/Extras/", ".../Featurettes/Deleted Scenes/",
|
||||
"Angel's Egg (1985)/04b. Angel's Egg Extras/") -> bonus. The ROOT folder
|
||||
is exempt because it names the film, and a film may be titled with a
|
||||
bonus word; and a file whose own name IS that folder's film name is the
|
||||
feature, not the bonus ("Ghost in the Shell Movies/Extras/
|
||||
Ghost in the Shell - CM3.mp4" is a bonus, but a whole other film filed
|
||||
there stays a film).
|
||||
Bonus vocabulary is matched as whole words only, so the film
|
||||
"Angel's Egg - Extra - Logo" inside "04b. Angel's Egg Extras" is reached by
|
||||
the folder rule, and "…Extras" as a fragment of a word is never evidence.
|
||||
"""
|
||||
parts = [q for q in path.split("/") if q]
|
||||
if not parts:
|
||||
return False
|
||||
name = (os.path.splitext(parts[-1])[0]
|
||||
if parts[-1].lower().endswith(VIDEO_SUFFIX) else "")
|
||||
# A bonus folder anywhere BELOW the film's own folder decides for its
|
||||
# contents; the film's own folder is the film's name, whatever words it
|
||||
# contains, so it is exempt. It is the first component AFTER ROOT's own
|
||||
# components — those are the mount path, not folders of the library.
|
||||
head = len([q for q in root.split("/") if q]) + 1
|
||||
bonus_folder = False
|
||||
for folder in parts[head:-1]:
|
||||
if folder_is_bonus(folder):
|
||||
# The folder's words are often the film's own name ("Angel's Egg
|
||||
# Extras/"), and then they are not bonus evidence. Only WORDS can name
|
||||
# a film: "extras" is release junk that name_words drops, so a folder
|
||||
# like "01. Extras" reduces to its index — and an index matching an
|
||||
# index ("NCOP 01") is a coincidence, not a title.
|
||||
nw = [w for w in name_words(name) if not w.isdigit()]
|
||||
fw = [w for w in name_words(folder) if not w.isdigit()]
|
||||
bonus_folder = not fw or not mentions(nw, fw)
|
||||
break
|
||||
if dur and dur > SHORT_MAX:
|
||||
# Runtime vetoes vocabulary, with ONE exception: an explicit bonus folder
|
||||
# and no metadata match. A 90-minute "Feature Length Rough Cut" sitting in
|
||||
# a movie's Featurettes/ is a disc extra, and nothing in the library claims
|
||||
# it as a film. Long-form bonus words still describe a film, so a
|
||||
# feature-length commentary cut or documentary keeps its runtime veto.
|
||||
if bonus_folder and not matched and not EXTRA_LONG_NAME_RE.search(name):
|
||||
return True
|
||||
return False
|
||||
if EXTRA_NAME_RE.search(name):
|
||||
return True
|
||||
if dur and dur <= LONG_BONUS_MAX and EXTRA_LONG_NAME_RE.search(name):
|
||||
return True
|
||||
return bonus_folder
|
||||
|
||||
# Release-group watermark clips: a group drops a tiny ad ("ETRG.mp4",
|
||||
# "RARBG.com.mp4") into many different films, so its name carries no film
|
||||
# identity at all and it would otherwise pair with itself across the library.
|
||||
WATERMARK_MAX = 20 * 2**20
|
||||
WATERMARK_RE = re.compile(r"(?i)^(rarbg(\.com)?|etrg|tgx|fbt|cpbg|yify|yts|"
|
||||
r"rarbg\.com|playnomore|imprs|gauntlet|dfx|macmote|"
|
||||
r"psi-axiz|acm|tlf|sa8|ebp|gtv|w4g|don|aAF|enjoy|hd4u)$")
|
||||
|
||||
|
||||
def is_watermark(path, size):
|
||||
"""True for a release group's own watermark video, not a film or an extra."""
|
||||
return (size < WATERMARK_MAX
|
||||
and bool(WATERMARK_RE.match(os.path.splitext(os.path.basename(path))[0])))
|
||||
|
||||
|
||||
|
||||
def key_of(title):
|
||||
"""Franchise key: first two alphabetic tokens (numeric-only title = itself).
|
||||
Deliberately NOT film-specific — it only selects candidates for the duration
|
||||
check, so a two-token collision (two films starting "star trek") cannot by
|
||||
itself create a duplicate group."""
|
||||
toks = [t for t in title.split() if t]
|
||||
words = [t for t in toks if not t.isdigit()]
|
||||
if not words:
|
||||
return " ".join(toks)
|
||||
sig = [t for t in words if t not in STOP] or words
|
||||
return " ".join(sig[:2])
|
||||
|
||||
|
||||
def ep_of(path):
|
||||
"""(series, episode-token) when the basename carries SxxEyy / 1x04 / Part N.
|
||||
|
||||
The series is the name BEFORE the marker. The box-set convention writes
|
||||
"Monk - S05E03 - Mr. Monk and the Big Game", and the episode title that
|
||||
follows the marker is not the show's name: keying on the whole title gave
|
||||
every episode of one series its own group, so a 124-episode box set never
|
||||
reached the size that calls something a series and each episode was filed
|
||||
as its own film. A file named only by its marker has nothing before it, so
|
||||
its whole title stands in."""
|
||||
name = os.path.splitext(os.path.basename(path))[0]
|
||||
m = EP_RE.search(name)
|
||||
if not m:
|
||||
return None
|
||||
head = name[:m.start()].strip(" ._\u2013-")
|
||||
series = title_of(head)[0] if head else None
|
||||
return series or title_of(path)[0], norm(m.group(1)).replace(" ", "")
|
||||
|
||||
|
||||
def base_words(title, key):
|
||||
"""Title tokens beyond the franchise key — the FILM identity part.
|
||||
"evil dead rise" under key "evil dead" -> {rise}; "28 days later" -> {}."""
|
||||
kt = set(key.split())
|
||||
return frozenset(t for t in title.split() if t not in kt and t not in STOP
|
||||
and not t.isdigit() and not YEAR_RE.match(t))
|
||||
|
||||
|
||||
def folder_year(path):
|
||||
"""Soft year evidence: a 19xx/20xx token in any parent directory name
|
||||
(collection folders are written both "(2002)" and "[2009]"; a range such as
|
||||
"(1998-2000)" deliberately yields nothing)."""
|
||||
for d in reversed(os.path.dirname(path).split("/")):
|
||||
ys = [y for y in re.split(r"[^0-9]+", d) if YEAR_RE.match(y)]
|
||||
if len(ys) == 1:
|
||||
return ys[0]
|
||||
return None
|
||||
|
||||
|
||||
def same_film(a, b):
|
||||
"""Two files are copies of ONE film only when the film-identity tokens agree
|
||||
(one side may carry none) AND no year evidence contradicts them. The key says
|
||||
the franchise matches; this says the actual film matches — it is what keeps
|
||||
'Evil Dead' (2013) apart from 'Evil Dead Rise' (2023) and the four Austin
|
||||
Powers apart when their runtimes happen to agree."""
|
||||
ba, bb = a["base"], b["base"]
|
||||
if ba and bb and ba != bb:
|
||||
return False
|
||||
ya = a["year"] if a["year"] != "?" else a["soft"]
|
||||
yb = b["year"] if b["year"] != "?" else b["soft"]
|
||||
if ya and yb and ya != yb:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def dur_close(a, b):
|
||||
if a <= 0 or b <= 0:
|
||||
return True # unknown duration: let titles decide
|
||||
return abs(a - b) <= max(2.0, 0.003 * max(a, b))
|
||||
|
||||
|
||||
# Blu-ray folder trees: ffprobe only sees the clip it is pointed at, so each tree
|
||||
# is folded into ONE entry, described by its longest main-feature clip (measured
|
||||
# with ffprobe on the largest BDMV/STREAM file) and sized by the whole tree.
|
||||
BDMV_MAIN = {
|
||||
"Avengers.Age.of.Ultron.2015.2160p.BluRay.HEVC.TrueHD.7.1.Atmos-TERMiNAL":
|
||||
dict(dur=8478.2, vcodec="hevc", res="3840x2160", pix="yuv420p10le", fps=23.976),
|
||||
"Full.Metal.Jacket.1987.2160p.BluRay.HEVC.DTS-HD.MA.5.1-EATDIK":
|
||||
dict(dur=6993.3, vcodec="hevc", res="3840x2160", pix="yuv420p10le", fps=23.976),
|
||||
}
|
||||
|
||||
|
||||
def load(probed):
|
||||
"""Probe rows -> film entries. AppleDouble (._*) sidecars are dropped; each
|
||||
BDMV tree collapses to one entry carrying the tree's total size."""
|
||||
trees = {}
|
||||
rows = []
|
||||
for line in open(probed):
|
||||
f = line.rstrip("\n").split("\t")
|
||||
if len(f) < 12:
|
||||
continue
|
||||
b, p, dur, vc, res, fps, vbit, pix, aud, subs, _d, fmt = f[:12]
|
||||
base = os.path.basename(p)
|
||||
if base.startswith("._") or base == ".DS_Store":
|
||||
continue # macOS sidecar, not media
|
||||
m = re.search(r"/(BDMV|VIDEO_TS)/", p)
|
||||
root = p[:m.start()] if m else None
|
||||
if root or re.search(r"/(CERTIFICATE|AACS|MLPI|CLIPINF|PLAYLIST|BACKUP|REGISTRL)/", p):
|
||||
if not root:
|
||||
continue # disc bookkeeping: folds into its tree, never a film
|
||||
if root:
|
||||
t = trees.setdefault(root, dict(bytes=0, audio=set(), subs=0))
|
||||
t["bytes"] += int(b)
|
||||
t["audio"].update(a for a in aud.split("+") if a)
|
||||
t["subs"] += int(subs or 0)
|
||||
continue
|
||||
try:
|
||||
dur = float(dur)
|
||||
except ValueError:
|
||||
dur = 0.0
|
||||
rows.append(dict(bytes=int(b), path=p, dur=dur, vcodec=vc, res=res, fps=fps,
|
||||
vbit=int(float(vbit or 0)), pix=pix, audio=aud, subs=subs, fmt=fmt))
|
||||
for root, t in trees.items():
|
||||
folder = os.path.basename(root)
|
||||
m = BDMV_MAIN.get(folder, {})
|
||||
rows.append(dict(bytes=t["bytes"], path=root + "/BDMV/index.bdmv",
|
||||
name_src=folder, dur=m.get("dur", 0.0),
|
||||
vcodec=m.get("vcodec", "hevc"), res=m.get("res", "-"),
|
||||
fps=m.get("fps", 0), vbit=0, pix=m.get("pix", "-"),
|
||||
audio="+".join(sorted(t["audio"])), subs=str(t["subs"]),
|
||||
fmt="BDMV tree"))
|
||||
for r in rows:
|
||||
r["title"], r["year"] = title_of(r.get("name_src") or r["path"])
|
||||
r["key"] = key_of(r["title"])
|
||||
r["base"] = base_words(r["title"], r["key"])
|
||||
r["soft"] = folder_year(r["path"])
|
||||
r["ep"] = ep_of(r["path"])
|
||||
return rows
|
||||
|
||||
|
||||
def dur_clusters(items):
|
||||
"""Group files that are the SAME FILM and the SAME MASTER duration. Candidate
|
||||
buckets come from the franchise key; same_film then rejects the different
|
||||
installments whose runtimes happen to agree."""
|
||||
clusters = []
|
||||
for r in sorted(items, key=lambda r: -r["dur"]):
|
||||
for c in clusters:
|
||||
if dur_close(c[0]["dur"], r["dur"]) and same_film(c[0], r):
|
||||
c.append(r)
|
||||
break
|
||||
else:
|
||||
clusters.append([r])
|
||||
return clusters
|
||||
|
||||
|
||||
def tier(groups):
|
||||
"""Split same-film/same-duration clusters: DUP = different releases of one
|
||||
film (real, deletable redundancy); REVIEW = identical size AND duration
|
||||
(byte-twins — the hash pass is the authority) or clashing year evidence."""
|
||||
dup, review = [], []
|
||||
for g in groups:
|
||||
g.sort(key=lambda r: -r["bytes"])
|
||||
years = {(r["year"] if r["year"] != "?" else r["soft"]) for r in g}
|
||||
years.discard(None)
|
||||
spread = (g[0]["bytes"] - g[-1]["bytes"]) / g[0]["bytes"] if g[0]["bytes"] else 0
|
||||
if len(years) > 1 or spread < 0.005:
|
||||
review.append(g)
|
||||
else:
|
||||
dup.append(g)
|
||||
return dup, review
|
||||
|
||||
|
||||
def redun(g):
|
||||
return sum(r["bytes"] for r in sorted(g, key=lambda r: -r["bytes"])[1:])
|
||||
|
||||
|
||||
def line(r):
|
||||
return (f"{r['bytes']/2**30:8.2f} GiB {r['res']:>9} {r['vcodec']:<7} "
|
||||
f"{r['pix']:<11} {r['dur']/60:5.0f}m {r['audio'][:34]:<34}")
|
||||
|
||||
|
||||
def dump(tag, groups, out, note=""):
|
||||
for gi, g in enumerate(groups, 1):
|
||||
g.sort(key=lambda r: -r["bytes"])
|
||||
print(f"\n### {tag}-{gi}. {g[0]['title']} ({g[0]['year']}) — {len(g)} files, "
|
||||
f"{redun(g)/2**30:.1f} GiB redundant {note}".rstrip())
|
||||
for rank, r in enumerate(g):
|
||||
print(f" {line(r)} [{r['year']}] "
|
||||
f"{r['path'].replace(ROOT + '/', '')}")
|
||||
out.write(f"{tag}\t{gi}\t{rank}\t{r['bytes']}\t{r['res']}\t{r['vcodec']}\t"
|
||||
f"{r['pix']}\t{int(r['dur'])}\t{r['audio']}\t{r['title']}|{r['year']}\t"
|
||||
f"{r['path']}\n")
|
||||
|
||||
|
||||
def report(probed, out_path):
|
||||
rows = load(probed)
|
||||
for r in rows:
|
||||
r["mark"] = is_watermark(r["path"], r["bytes"])
|
||||
r["extra"] = not r["mark"] and is_extra(r["path"], r["title"], r["dur"])
|
||||
marks = [r for r in rows if r["mark"]]
|
||||
movies = [r for r in rows if not r["ep"] and not r["extra"] and not r["mark"]]
|
||||
episodes = [r for r in rows if r["ep"] and not r["extra"] and not r["mark"]]
|
||||
extras = [r for r in rows if r["extra"]]
|
||||
print(f"census: {len(rows)} video entries = {len(movies)} films + {len(episodes)} "
|
||||
f"TV episodes + {len(extras)} bonus/trailers "
|
||||
f"({sum(r['bytes'] for r in extras)/2**30:.1f} GiB) "
|
||||
f"+ {len(marks)} release-group watermarks")
|
||||
|
||||
out = open(out_path, "w")
|
||||
|
||||
buckets = collections.defaultdict(list)
|
||||
for r in movies:
|
||||
buckets[r["key"]].append(r)
|
||||
cand = {k: v for k, v in buckets.items() if len(v) > 1}
|
||||
clusters = [c for v in cand.values() for c in dur_clusters(v) if len(c) > 1]
|
||||
dup, review = tier(clusters)
|
||||
dup.sort(key=lambda g: -redun(g))
|
||||
review.sort(key=lambda g: -redun(g))
|
||||
dump("DUP", dup, out)
|
||||
dump("REVIEW", review, out, "— same key+duration, sizes near-identical OR years clash: RECHECK")
|
||||
|
||||
|
||||
ebuf = collections.defaultdict(list)
|
||||
for r in episodes:
|
||||
ebuf[(r["ep"][0], r["ep"][1])].append(r)
|
||||
edup, ereview = tier([c for v in ebuf.values() if len(v) > 1
|
||||
for c in dur_clusters(v) if len(c) > 1])
|
||||
edup.sort(key=lambda g: -redun(g))
|
||||
dump("EP-DUP", edup, out)
|
||||
|
||||
shows = collections.Counter()
|
||||
for r in episodes:
|
||||
shows[r["ep"][0]] += 1
|
||||
multi = {s: n for s, n in shows.items() if n > 4}
|
||||
print("\n=== TV-series content inside Movies/ (>4 episodes) ===")
|
||||
for s, n in sorted(multi.items(), key=lambda kv: -kv[1]):
|
||||
print(f" {n:4d} episodes {s}")
|
||||
out.close()
|
||||
|
||||
print(f"\nDUP groups={len(dup)} files={sum(len(g) for g in dup)} "
|
||||
f"redundant={sum(redun(g) for g in dup)/2**40:.2f} TiB", file=sys.stderr)
|
||||
print(f"REVIEW groups={len(review)} "
|
||||
f"redundant={sum(redun(g) for g in review)/2**40:.2f} TiB", file=sys.stderr)
|
||||
print(f"EP-DUP groups={len(edup)} "
|
||||
f"ep-redundant={sum(redun(g) for g in edup)/2**30:.1f} GiB", file=sys.stderr)
|
||||
|
||||
|
||||
def parse(names):
|
||||
for n in names:
|
||||
t, y = title_of(n)
|
||||
print(f"{t!r} ({y})\tkey={key_of(t)!r}\t{n[:70]}")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
sub = ap.add_subparsers(dest="cmd", required=True)
|
||||
rep = sub.add_parser("report", help="similar-copy census over a probe TSV")
|
||||
rep.add_argument("--probed", default="/tmp/media-probed.tsv")
|
||||
rep.add_argument("--out", default="/tmp/media-similar.tsv")
|
||||
par = sub.add_parser("parse", help="show the parsed title/year of a name")
|
||||
par.add_argument("names", nargs="+")
|
||||
a = ap.parse_args()
|
||||
if a.cmd == "report":
|
||||
report(a.probed, a.out)
|
||||
else:
|
||||
parse(a.names)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+281
@@ -0,0 +1,281 @@
|
||||
#!/usr/bin/env python3
|
||||
"""media-tv.py — move TV episodes out of the movie library into a TV library.
|
||||
|
||||
`scripts/media-plan.py` refuses to file TV: it parks episode files in a
|
||||
`_TV needs a TV library - <series>` pile for a human. This is the other half of
|
||||
that refusal. It walks the movie library, groups the files whose own name
|
||||
carries an episode marker by SERIES, and writes a plan that places each series
|
||||
in the shape both Jellyfin and Plex read without help:
|
||||
|
||||
<library>/<Series> (<year>)/Season <NN>/<the file's own name>.<ext>
|
||||
|
||||
It writes a plan and nothing else. Every rename belongs to
|
||||
`scripts/media-apply.py` — the write-ahead journal, the refusal to overwrite,
|
||||
the all-or-nothing preflight, the rollback. Apply it with `--root` set to the
|
||||
share that holds BOTH libraries (`/mnt/entropy`, not `/mnt/entropy/Movies`) so
|
||||
the executor's own path guard still has teeth, and give it its own journal —
|
||||
one journal is one undo, and a TV move must not be welded to a movie relayout:
|
||||
|
||||
python3 scripts/media-tv.py --root /mnt/entropy/Movies \
|
||||
--library /mnt/entropy/TVShows --out /mnt/entropy/media-tv-plan.tsv
|
||||
python3 scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv \
|
||||
--root /mnt/entropy --journal /mnt/entropy/media-relayout-tv.jsonl
|
||||
python3 scripts/media-apply.py verify --plan /mnt/entropy/media-tv-plan.tsv \
|
||||
--root /mnt/entropy --journal /mnt/entropy/media-relayout-tv.jsonl
|
||||
python3 scripts/media-apply.py sweep --root /mnt/entropy \
|
||||
--journal /mnt/entropy/media-relayout-tv.jsonl --apply
|
||||
|
||||
Guards, and why each one is here:
|
||||
|
||||
* The same five-file threshold as the planner. An episode-shaped name on one
|
||||
file is not a series — `The Godfather Part 2` is a film — so a smaller group
|
||||
is reported and left exactly where it is.
|
||||
* An episode whose name carries no season number is reported and left, never
|
||||
filed into an invented `Season 01`.
|
||||
* A sidecar that shares an episode's own name (`.srt`, `.nfo`, …) follows the
|
||||
episode; an episode without its subtitles is a half-move.
|
||||
* A group that would claim one destination twice is not resolved here. The
|
||||
executor refuses the whole plan before its first rename, which is the right
|
||||
outcome: nobody but Frank decides which copy of an episode is the keeper.
|
||||
* An episode the destination library already holds at the same byte count is
|
||||
left where it is. The TV library has held a complete Monk box set for years
|
||||
while the movie library carried a second copy of all 124 episodes; a mover
|
||||
that did not look where it was filing would have put both inside one library.
|
||||
* A symlink is never followed and never moved.
|
||||
"""
|
||||
import argparse
|
||||
import collections
|
||||
import importlib.util
|
||||
import os
|
||||
import re
|
||||
|
||||
|
||||
def parse_args():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--root", default=os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
|
||||
help="the movie library to empty of TV")
|
||||
ap.add_argument("--library", default="/mnt/entropy/TVShows",
|
||||
help="the TV library to fill (a different share or folder)")
|
||||
ap.add_argument("--out", default=None,
|
||||
help="plan TSV for scripts/media-apply.py"
|
||||
" (default: media-tv-plan.tsv beside --root)")
|
||||
ap.add_argument("--min", type=int, default=5,
|
||||
help="episode files that make something a series")
|
||||
a = ap.parse_args()
|
||||
if not a.out:
|
||||
a.out = os.path.join(os.path.dirname(os.path.abspath(a.root)),
|
||||
"media-tv-plan.tsv")
|
||||
return a
|
||||
|
||||
|
||||
ARGS = parse_args()
|
||||
ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
|
||||
spec = importlib.util.spec_from_file_location("eng", ENGINE)
|
||||
eng = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(eng)
|
||||
|
||||
BAD_CHARS = re.compile(r'[<>:"/|?*]')
|
||||
# The season and episode, from the token media-titles.ep_of() hands back: s05e03, 1x04.
|
||||
SEASON_RE = re.compile(r"^(?:s(\d{1,2})e(\d{1,4})|(\d{1,2})x(\d{1,4}))$")
|
||||
# Sidecars that follow an episode; macOS forks and resume data do not.
|
||||
SKIP_SIDECAR = re.compile(r"^(\._|\.DS_|dl_state)")
|
||||
# A box set that stamps the show's year into every file name; the folder's year
|
||||
# must not be added on top of it.
|
||||
YEAR_TAIL = re.compile(r"[(\[]\s*(18|19|20)\d\d\s*[)\]]\s*$")
|
||||
SMALL = {"a", "an", "the", "and", "of", "for", "in", "on", "to", "with", "at",
|
||||
"by", "de", "la", "vs"}
|
||||
|
||||
|
||||
def clean(name):
|
||||
"""A name the filesystem and both matchers can read (same law as the plan)."""
|
||||
name = BAD_CHARS.sub("-", name).replace("\\", "-")
|
||||
name = re.sub(r"\s+", " ", name).strip().strip(". ").strip()
|
||||
return name[:180] or "untitled"
|
||||
|
||||
|
||||
def episode_name(path):
|
||||
"""The file's own name, kept: the publisher's spelling beats a re-derivation.
|
||||
|
||||
A dot-separated release name (`monk.s05e03.mkv`) becomes words; a name that
|
||||
already carries spaces is left exactly as it was written."""
|
||||
stem, ext = os.path.splitext(os.path.basename(path))
|
||||
if " " not in stem and stem.count(".") >= 2:
|
||||
stem = stem.replace(".", " ")
|
||||
return clean(stem) + ext.lower()
|
||||
|
||||
|
||||
def head_of(path):
|
||||
"""The publisher's series name: the text BEFORE the episode marker.
|
||||
|
||||
`Monk - S05E03 - Mr. Monk and the Big Game` is `Monk`; the episode title
|
||||
after the marker is not the show's name, and case is kept because it is the
|
||||
library's own spelling, not a normalised key."""
|
||||
name = os.path.splitext(os.path.basename(path))[0]
|
||||
m = eng.EP_RE.search(name)
|
||||
head = name[:m.start()] if m else name
|
||||
return re.sub(r"\s+", " ", re.sub(r"[._]+", " ", head)).strip(" -\u2013").strip()
|
||||
|
||||
|
||||
def title_series(head):
|
||||
"""The series folder in the case a TV library is written in.
|
||||
|
||||
Only a name with no capital at all is retitled: a box set that was
|
||||
lower-cased on the way in would otherwise file `monk (2002)` among
|
||||
`Dexter's Laboratory (1996)`. A name the publisher wrote with case is left
|
||||
exactly as it was written."""
|
||||
if any(c.isupper() for c in head):
|
||||
return head # the publisher wrote it with case; keep it
|
||||
words = head.split()
|
||||
return " ".join(w if i and w.lower() in SMALL else w.capitalize()
|
||||
for i, w in enumerate(words))
|
||||
|
||||
|
||||
def season_episode(token):
|
||||
"""(season, episode) an episode token states, or None when it states none."""
|
||||
m = SEASON_RE.match(token)
|
||||
if not m:
|
||||
return None
|
||||
return int(m.group(1) or m.group(3)), int(m.group(2) or m.group(4))
|
||||
|
||||
|
||||
def sidecars(path):
|
||||
"""The files beside PATH that carry its exact stem — its subtitles, its nfo."""
|
||||
d, stem = os.path.split(path)
|
||||
stem = os.path.splitext(stem)[0]
|
||||
out = []
|
||||
try:
|
||||
entries = sorted(os.listdir(d))
|
||||
except OSError:
|
||||
return out
|
||||
for e in entries:
|
||||
p = os.path.join(d, e)
|
||||
if p == path or os.path.islink(p) or not os.path.isfile(p):
|
||||
continue
|
||||
s, _ext = os.path.splitext(e)
|
||||
if s == stem and not SKIP_SIDECAR.match(e):
|
||||
out.append(p)
|
||||
return out
|
||||
|
||||
|
||||
def episodes_under(root):
|
||||
"""(path, series, token) for every episode-shaped video file under root."""
|
||||
for dirpath, dirnames, filenames in os.walk(root):
|
||||
dirnames[:] = sorted(d for d in dirnames
|
||||
if not d.startswith(".") and d != "@eaDir")
|
||||
for f in filenames:
|
||||
p = os.path.join(dirpath, f)
|
||||
if os.path.islink(p) or not f.lower().endswith(eng.VIDEO_SUFFIX):
|
||||
continue
|
||||
e = eng.ep_of(p)
|
||||
if e:
|
||||
yield p, e[0], e[1]
|
||||
|
||||
|
||||
def walk(root, library):
|
||||
"""Episode files under ROOT keyed by series, ignoring the destination."""
|
||||
groups = collections.defaultdict(list)
|
||||
lib = os.path.abspath(library)
|
||||
for p, series, token in episodes_under(root):
|
||||
if os.path.abspath(p).startswith(lib + os.sep):
|
||||
continue
|
||||
groups[series].append((p, token))
|
||||
return groups
|
||||
|
||||
|
||||
def holdings(library):
|
||||
"""What the destination already carries: (series, season, episode) -> sizes.
|
||||
|
||||
The one expensive read here, and the one that matters. The movie library
|
||||
holds a complete Monk box set that the TV library has held for years: 124
|
||||
episodes, the same episode list, identical byte counts. Moving it would
|
||||
have filed a second copy of a whole show into the library that already had
|
||||
it. Byte count is the test on purpose — two encodes of one episode are two
|
||||
files and both are wanted; the same bytes are the same file."""
|
||||
out = collections.defaultdict(set)
|
||||
for p, series, token in episodes_under(library):
|
||||
se = season_episode(token)
|
||||
if se is None:
|
||||
continue
|
||||
try:
|
||||
out[(series, *se)].add(os.path.getsize(p))
|
||||
except OSError:
|
||||
continue
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
root, library = os.path.abspath(ARGS.root), os.path.abspath(ARGS.library)
|
||||
groups = walk(root, library)
|
||||
held_here = holdings(library)
|
||||
mkdirs = {library}
|
||||
rows, lines, skipped = set(), [], []
|
||||
for series, items in sorted(groups.items()):
|
||||
if len(items) < ARGS.min:
|
||||
skipped.append(f"skipped: {head_of(items[0][0]) or series} — {len(items)} "
|
||||
f"episode-shaped file{'s' if len(items) != 1 else ''},"
|
||||
f" below {ARGS.min}: not a series")
|
||||
continue
|
||||
years = collections.Counter(y for y in (eng.folder_year(p) for p, _t in items)
|
||||
if y)
|
||||
year = years.most_common(1)[0][0] if years else None
|
||||
heads = collections.Counter(head_of(p) for p, _t in items)
|
||||
head = heads.most_common(1)[0][0]
|
||||
if year and YEAR_TAIL.search(head):
|
||||
year = None
|
||||
name = clean(f"{title_series(head)}{f' ({year})' if year else ''}")
|
||||
series_dir = os.path.join(library, name)
|
||||
moved = held = 0
|
||||
for path, token in items:
|
||||
se = season_episode(token)
|
||||
if se is None:
|
||||
skipped.append(f"skipped: {path} — no season number in its name,"
|
||||
" nothing here invents one")
|
||||
continue
|
||||
try:
|
||||
size = os.path.getsize(path)
|
||||
except OSError:
|
||||
continue
|
||||
if size in held_here.get((series, *se), ()):
|
||||
held += 1
|
||||
continue
|
||||
dst_dir = os.path.join(series_dir, f"Season {se[0]:02d}")
|
||||
mkdirs.update([series_dir, dst_dir])
|
||||
base = episode_name(path)
|
||||
rows.add((path, os.path.join(dst_dir, base)))
|
||||
moved += 1
|
||||
for side in sidecars(path):
|
||||
rows.add((side, os.path.join(
|
||||
dst_dir, clean(os.path.splitext(base)[0]
|
||||
+ os.path.splitext(side)[1].lower()))))
|
||||
if moved:
|
||||
lines.append(f"{name}: {moved} episodes"
|
||||
f"{'' if year else ' (no year in any folder name)'}"
|
||||
f" -> {series_dir}")
|
||||
if held:
|
||||
lines.append(f"{name}: {held}"
|
||||
f" {'episode' if held == 1 else 'episodes'} already in"
|
||||
f" {library} at the same byte count — left there")
|
||||
|
||||
# An empty plan creates nothing, not even the library folder: a run that
|
||||
# found no series must not be able to leave a stray directory behind.
|
||||
dirs = sorted(mkdirs, key=lambda p: (p.count(os.sep), p)) if rows else []
|
||||
with open(ARGS.out, "w") as out:
|
||||
out.write("action\tsrc\tdst\tkind\n")
|
||||
for d in dirs:
|
||||
out.write(f"mkdir\t\t{d}\tdir\n")
|
||||
for src, dst in sorted(rows):
|
||||
out.write(f"move\t{src}\t{dst}\ttv\n")
|
||||
|
||||
for line in lines + skipped:
|
||||
print(line)
|
||||
print(f"plan: {len(rows)} moves, {len(dirs)} folders -> {ARGS.out}")
|
||||
if rows:
|
||||
print(f"next: python3 scripts/media-apply.py apply --plan {ARGS.out}"
|
||||
f" --root {os.path.dirname(root)}"
|
||||
f" --journal {os.path.join(os.path.dirname(root),'media-relayout-tv.jsonl')}"
|
||||
" (add --apply to run it)")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Executable
+221
@@ -0,0 +1,221 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Prove a relayout landed, using the media server's own tables.
|
||||
|
||||
A scan answers `204` whether or not it found anything, and a rename that the
|
||||
scanner has not followed looks exactly like a rename that never happened from
|
||||
the outside. The only honest proof is the run's journal replayed against the
|
||||
database the server actually serves from: every `move` row must have a row at
|
||||
its destination, and NO row may still name its source — a source row is a scan
|
||||
that has not caught up, and it is the one signature of a library pointing at
|
||||
files that no longer exist.
|
||||
|
||||
Two doors, both read-only, both without a lock while the server runs (there is
|
||||
no `sqlite3` binary on these hosts). Open `mode=ro` PLAIN: both servers keep
|
||||
their libraries in WAL mode, and `immutable=1` — which claims the file never
|
||||
changes — makes sqlite skip the write-ahead log entirely, so every row the
|
||||
server has committed but not yet checkpointed reads as MISSING, and a read
|
||||
beside a running scanner can even fail as "database disk image is malformed".
|
||||
`immutable=1` is only the fallback, for a database whose `-shm` cannot be read.
|
||||
|
||||
jellyfin `BaseItems.Path`. The container binds the share at `/data`, so the
|
||||
database holds `/data/<library>/…` where the journal holds
|
||||
`/mnt/entropy/<library>/…`. Translate, or every row reads as missing.
|
||||
plex `media_parts.file`, joined up to `metadata_items`. Plex is bound at
|
||||
the host path, so its rows match the journal verbatim. Filter
|
||||
`deleted_at is null` on BOTH tables — a deleted scan leaves its parts
|
||||
behind, and one of those rows is not proof anything landed.
|
||||
|
||||
Neither server holds a row for everything the journal renames: a `.log`, `.cue`
|
||||
or cover image has no row to land in, so its absence is not a fault. The report
|
||||
therefore breaks the missing rows down by extension, and `--ignore-ext` names the
|
||||
extensions this server does not index, which are then counted separately and kept
|
||||
out of the verdict. What is left is split by what the filesystem says, because
|
||||
the database alone cannot: a file still at its `src` is `REAL` (this pass's own
|
||||
damage), a file at neither path is `gone` (a later pass or the server renamed it),
|
||||
and a file sitting at its `dst` with no row naming it is simply something the
|
||||
server does not index. A row that still names `src` is none of these: it is the
|
||||
server's own stale pointer — a phantom — and it costs a re-scan, never a restore.
|
||||
|
||||
Exit codes: 0 every move landed; 1 something is stale or missing (the details are
|
||||
printed); 2 the journal or the database is not usable at all.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
from collections import Counter
|
||||
|
||||
JF_DB = "/home/dank/Containers/jellyfin/config/data/jellyfin.db"
|
||||
PLEX_DB = ("/var/lib/plexmediaserver/Library/Application Support/Plex Media Server"
|
||||
"/Plug-in Support/Databases/com.plexapp.plugins.library.db")
|
||||
|
||||
# Per kind: the table whose columns are required, those columns, and the query
|
||||
# that returns every path the server believes it has.
|
||||
KINDS = {
|
||||
"jellyfin": {
|
||||
"table": "BaseItems",
|
||||
"columns": {"Path"},
|
||||
"sql": "select Path from BaseItems",
|
||||
"default_db": JF_DB,
|
||||
},
|
||||
"plex": {
|
||||
"table": "media_parts",
|
||||
"columns": {"file", "media_item_id", "deleted_at"},
|
||||
"sql": ("select p.file from media_parts p"
|
||||
" join media_items m on m.id = p.media_item_id"
|
||||
" join metadata_items mi on mi.id = m.metadata_item_id"
|
||||
" where p.deleted_at is null and mi.deleted_at is null"),
|
||||
"default_db": PLEX_DB,
|
||||
},
|
||||
}
|
||||
|
||||
def die(msg):
|
||||
print(f"media-verify: {msg}", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
def read_moves(path):
|
||||
"""The (src, dst) pairs of a journal, in the order the run performed them."""
|
||||
moves = []
|
||||
try:
|
||||
fh = open(path, encoding="utf-8")
|
||||
except OSError as e:
|
||||
die(f"cannot read the journal {path}: {e.strerror}")
|
||||
with fh:
|
||||
for n, line in enumerate(fh, 1):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
die(f"{path} line {n} is not a journal record")
|
||||
if rec.get("op") == "move":
|
||||
if not rec.get("src") or not rec.get("dst"):
|
||||
die(f"{path} line {n} is a move without both src and dst")
|
||||
moves.append((rec["src"], rec["dst"]))
|
||||
return moves
|
||||
|
||||
|
||||
def server_paths(db, kind):
|
||||
"""Every path the server holds, or a refusal that names what is missing."""
|
||||
spec = KINDS[kind]
|
||||
con = None
|
||||
err = None
|
||||
# Plain ro reads the WAL the server commits through; immutable is the last
|
||||
# resort, because it cannot see a row the server has not checkpointed.
|
||||
for opt in ("mode=ro", "mode=ro&immutable=1"):
|
||||
try:
|
||||
con = sqlite3.connect(f"file:{db}?{opt}", uri=True)
|
||||
break
|
||||
except sqlite3.Error as e:
|
||||
err = e
|
||||
if con is None:
|
||||
die(f"cannot open {db}: {err}")
|
||||
with con:
|
||||
have = {r[1] for r in con.execute(f"pragma table_info({spec['table']})")}
|
||||
if not spec["columns"] <= have:
|
||||
die(f"{db} is not a {kind} library database ({spec['table']} has no "
|
||||
f"{','.join(sorted(spec['columns'] - have))})")
|
||||
try:
|
||||
return {r[0] for r in con.execute(spec["sql"]) if r[0]}
|
||||
except sqlite3.Error as e:
|
||||
die(f"{db} refused the {kind} query: {e}")
|
||||
|
||||
|
||||
def to_host(path, container, share):
|
||||
"""The database's path in the journal's terms; identity when it is not a
|
||||
container-side path."""
|
||||
if not container:
|
||||
return path
|
||||
prefix = container.rstrip("/") + "/"
|
||||
if path.startswith(prefix):
|
||||
return share.rstrip("/") + "/" + path[len(prefix):]
|
||||
return path
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--journal", required=True,
|
||||
help="the run's journal (scripts/media-apply.py writes it)")
|
||||
ap.add_argument("--db-kind", choices=sorted(KINDS), required=True)
|
||||
ap.add_argument("--db", default=None, help="the server's database on this host")
|
||||
ap.add_argument("--share", default="/mnt/entropy",
|
||||
help="the share every journal path lives under")
|
||||
ap.add_argument("--container", default=None,
|
||||
help="what the share is called inside the container; empty"
|
||||
" compares verbatim (default /data for jellyfin, nothing"
|
||||
" for plex)")
|
||||
ap.add_argument("--show", type=int, default=20,
|
||||
help="how many offenders to name before saying 'and N more'")
|
||||
ap.add_argument("--ignore-ext", default="",
|
||||
help="comma-separated extensions this server does not index,"
|
||||
" kept out of the verdict (e.g. log,cue,nfo,srt)")
|
||||
args = ap.parse_args()
|
||||
|
||||
db = args.db or KINDS[args.db_kind]["default_db"]
|
||||
container = ("/data" if args.db_kind == "jellyfin" else "") \
|
||||
if args.container is None else args.container
|
||||
ignore = set()
|
||||
for ext in args.ignore_ext.split(","):
|
||||
ext = ext.strip().lower()
|
||||
if ext:
|
||||
ignore.add(ext if ext.startswith(".") else "." + ext)
|
||||
|
||||
moves = read_moves(args.journal)
|
||||
stored = {to_host(p, container, args.share) for p in server_paths(db, args.db_kind)}
|
||||
|
||||
landed, stale, missing, ignored = 0, [], [], 0
|
||||
for src, dst in moves:
|
||||
if os.path.splitext(dst)[1].lower() in ignore:
|
||||
ignored += 1
|
||||
continue
|
||||
at_dst, at_src = dst in stored, src in stored
|
||||
if at_dst and not at_src:
|
||||
landed += 1
|
||||
continue
|
||||
if at_src:
|
||||
stale.append(src)
|
||||
if not at_dst:
|
||||
missing.append((src, dst))
|
||||
|
||||
# The database cannot tell a lost file from one it never indexes: both read
|
||||
# as "no row at the destination". Stat both paths, so the report separates
|
||||
# the classes the runbook names — `REAL` is the only one that is this pass's
|
||||
# own damage, and `gone` is a later pass or the server's own rename.
|
||||
on_disk, real, gone = [], [], []
|
||||
for src, dst in missing:
|
||||
if os.path.exists(dst):
|
||||
on_disk.append(dst)
|
||||
elif os.path.exists(src):
|
||||
real.append(src)
|
||||
else:
|
||||
gone.append(dst)
|
||||
|
||||
print(f"{args.db_kind}: {len(moves)} moves checked")
|
||||
print(f"landed: {landed}")
|
||||
print(f"still at src: {len(stale)}")
|
||||
print(f"missing from the DB: {len(missing)}")
|
||||
print(f" on disk at dst, no row names it: {len(on_disk)}")
|
||||
print(f" REAL — file still at src: {len(real)}")
|
||||
print(f" gone — at neither path: {len(gone)}")
|
||||
if ignored:
|
||||
print(f"not indexed (extension ignored): {ignored}")
|
||||
if missing:
|
||||
exts = Counter(os.path.splitext(d)[1].lower() or "(no extension)"
|
||||
for _, d in missing)
|
||||
print("missing by extension: "
|
||||
+ ", ".join(f"{e} {n}" for e, n in sorted(exts.items())))
|
||||
for label, items in (("real", real), ("gone", gone), ("missing", on_disk),
|
||||
("stale", stale)):
|
||||
for path in items[:args.show]:
|
||||
print(f" {label}: {path}")
|
||||
if len(items) > args.show:
|
||||
print(f" {label}: … and {len(items) - args.show} more")
|
||||
return 1 if (stale or missing) else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Executable
+241
@@ -0,0 +1,241 @@
|
||||
#!/usr/bin/env bash
|
||||
# The SSD-backed metadata view that lets Jellyfin and Plex share one metadata set.
|
||||
#
|
||||
# Both servers read a Kodi-format sidecar (movie.nfo / tvshow.nfo / episode.nfo)
|
||||
# ONLY when it sits next to the media file. The media lives on the RAID6 array at
|
||||
# /mnt/entropy — five spinning disks, deliberately mounted read-only for every
|
||||
# container, and slow to seek for a thousand tiny XML reads. So neither media
|
||||
# server binds the array any more: they bind an OverlayFS merge of it with a
|
||||
# writable layer on the NVMe.
|
||||
#
|
||||
# lowerdir /mnt/entropy RAID6 (ROTA=1) — source of truth, never written
|
||||
# upperdir /var/lib/media-meta/upper NVMe — every sidecar and image byte
|
||||
# workdir /var/lib/media-meta/work NVMe — kernel bookkeeping
|
||||
# Where /var/lib/media-meta/view the merged tree jellyfin and plex bind
|
||||
#
|
||||
# Everything the view needs — layers, bookkeeping, trash, the merged door — lives
|
||||
# under the one directory /var/lib/media-meta, so nothing new appears under /mnt.
|
||||
#
|
||||
# Consequences worth knowing:
|
||||
# * A sidecar write is a few KB onto the SSD; the media bytes never move and the
|
||||
# array is still never written by anything automated.
|
||||
# * Deleting through the view writes a whiteout into the upper layer instead of
|
||||
# deleting from the array. Nothing should delete through the view: deletions
|
||||
# go to .AGENTTRASH on the RAW path, and `reap` clears the resulting ghosts.
|
||||
# * inotify events on the lower layer do not cross the merge, so a file that
|
||||
# arrives on the raw array is invisible to a file-system watcher in the view:
|
||||
# trigger a library refresh (see the Media Relayout page).
|
||||
#
|
||||
# Subcommands
|
||||
# plan [--upper-base D] [--lower D] print the resolved shape, touch nothing
|
||||
# verify prove the live mount IS that shape
|
||||
# install [--upper-base D] create the layers, install+enable+start the unit
|
||||
# reap [--apply] [--upper-base D] [--lower D]
|
||||
# list upper-layer entries whose media is gone
|
||||
# status one line per fact a human wants
|
||||
#
|
||||
# install must run as root (it writes /etc/systemd/system and mounts); the others
|
||||
# are read-only and safe unprivileged.
|
||||
set -euo pipefail
|
||||
|
||||
LOWER=/mnt/entropy
|
||||
UPPER_BASE=/var/lib/media-meta
|
||||
VIEW=$UPPER_BASE/view
|
||||
# systemd names a mount unit after its ESCAPED path ('/' -> '-', a literal '-'
|
||||
# -> '\x2d'), so the unit on the host is var-lib-media\x2dmeta-view.mount. The
|
||||
# repo copy keeps a readable name; install writes it under the escaped one.
|
||||
UNIT='var-lib-media\x2dmeta-view.mount'
|
||||
UNIT_DIR=/etc/systemd/system
|
||||
# The rootless containers write as this host uid (the deploy user). The upper
|
||||
# layer has to accept those writes, so it is owned by them.
|
||||
MEDIA_UID="${MEDIA_VIEW_UID:-1000}"
|
||||
MEDIA_GID="${MEDIA_VIEW_GID:-1000}"
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
UNIT_SRC="$REPO_ROOT/dankserver/host/systemd/media-meta-view.mount"
|
||||
|
||||
die() {
|
||||
printf 'media-view: %s\n' "$*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
usage() {
|
||||
printf 'usage: %s plan|verify|install|reap|status [options]\n' "$0" >&2
|
||||
exit 2
|
||||
}
|
||||
|
||||
cmd="${1:-}"
|
||||
[ -n "$cmd" ] || usage
|
||||
shift || true
|
||||
|
||||
upper_base="$UPPER_BASE"
|
||||
lower="$LOWER"
|
||||
apply=0
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--upper-base) [ $# -ge 2 ] || usage; upper_base="$2"; shift 2 ;;
|
||||
--lower) [ $# -ge 2 ] || usage; lower="$2"; shift 2 ;;
|
||||
--apply) apply=1; shift ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
done
|
||||
|
||||
upper="$upper_base/upper"
|
||||
work="$upper_base/work"
|
||||
|
||||
# The device backing a path, and whether the kernel calls it rotational.
|
||||
rotational_of() {
|
||||
local src rota
|
||||
src="$(findmnt -no SOURCE --target "$1" 2>/dev/null || true)"
|
||||
[ -n "$src" ] || die "cannot find a block device behind $1"
|
||||
rota="$(lsblk -no ROTA "$src" 2>/dev/null | head -1 || true)"
|
||||
[ -n "$rota" ] || die "cannot read the rotational flag of $src"
|
||||
printf '%s %s\n' "$src" "$rota"
|
||||
}
|
||||
|
||||
require_upper_base() {
|
||||
[ -d "$upper_base" ] \
|
||||
|| die "refused: upper base $upper_base does not exist (put it on the SSD first)"
|
||||
}
|
||||
|
||||
do_plan() {
|
||||
printf 'lowerdir=%s\nupperdir=%s\nworkdir=%s\nmountpoint=%s\nunit=%s/%s\n' \
|
||||
"$lower" "$upper" "$work" "$VIEW" "$UNIT_DIR" "$UNIT"
|
||||
}
|
||||
|
||||
do_verify() {
|
||||
local fail=0 fst opts src rota
|
||||
if ! mountpoint -q "$VIEW"; then
|
||||
printf 'refused: %s is not a mountpoint — a plain directory here is a\n' "$VIEW" >&2
|
||||
printf ' library that scans itself empty, not a metadata view.\n' >&2
|
||||
return 1
|
||||
fi
|
||||
fst="$(findmnt -no FSTYPE --mountpoint "$VIEW" 2>/dev/null || true)"
|
||||
if [ "$fst" != overlay ]; then
|
||||
printf 'refused: %s is mounted %s, not overlay\n' "$VIEW" "${fst:-nothing}" >&2
|
||||
fail=1
|
||||
fi
|
||||
opts="$(findmnt -no OPTIONS --mountpoint "$VIEW" 2>/dev/null || true)"
|
||||
local want
|
||||
for want in "lowerdir=$LOWER" "upperdir=$UPPER_BASE/upper" "workdir=$UPPER_BASE/work"; do
|
||||
case ",$opts," in
|
||||
*",${want},"*) ;;
|
||||
*) printf 'refused: %s options lack %s (got: %s)\n' "$VIEW" "$want" "$opts" >&2
|
||||
fail=1 ;;
|
||||
esac
|
||||
done
|
||||
read -r src rota <<<"$(rotational_of "$UPPER_BASE")"
|
||||
if [ "$rota" != 0 ]; then
|
||||
printf 'refused: the upper layer sits on %s which is rotational — metadata\n' "$src" >&2
|
||||
printf ' belongs on the SSD, not on the array.\n' >&2
|
||||
fail=1
|
||||
fi
|
||||
if [ ! -w "$UPPER_BASE/upper" ]; then
|
||||
printf 'refused: %s is not writable\n' "$upper" >&2
|
||||
fail=1
|
||||
fi
|
||||
[ "$fail" -eq 0 ] || return 1
|
||||
printf 'view ok: %s <- overlay(%s,%s)\n' "$VIEW" "$LOWER" "$UPPER_BASE/upper"
|
||||
}
|
||||
|
||||
do_install() {
|
||||
[ "$(id -u)" -eq 0 ] || die "install must run as root (writes $UNIT_DIR and mounts)"
|
||||
require_upper_base
|
||||
mountpoint -q "$LOWER" \
|
||||
|| die "refused: $LOWER is not mounted; overlaying an empty directory would present an empty library"
|
||||
local src rota
|
||||
read -r src rota <<<"$(rotational_of "$upper_base")"
|
||||
[ "$rota" = 0 ] || die "refused: $upper_base is on $src, a rotational device — the metadata layer must be on the SSD"
|
||||
[ -f "$UNIT_SRC" ] || die "refused: $UNIT_SRC is missing from $REPO_ROOT"
|
||||
# systemd will not create a mount point; a missing directory fails the start.
|
||||
install -d -m 0755 -o "$MEDIA_UID" -g "$MEDIA_GID" "$upper" "$work" "$VIEW"
|
||||
install -d -m 0700 "$upper_base/.AGENTTRASH"
|
||||
install -m 0644 "$UNIT_SRC" "$UNIT_DIR/$UNIT"
|
||||
systemctl daemon-reload
|
||||
systemctl enable "$UNIT" >/dev/null 2>&1 || true
|
||||
systemctl start "$UNIT"
|
||||
do_verify
|
||||
}
|
||||
|
||||
# An upper-layer DIRECTORY with no counterpart below is metadata for a media
|
||||
# folder that has been moved or renamed on the RAW path — a ghost the media
|
||||
# servers would still browse. Files are never candidates on their own: a sidecar
|
||||
# (movie.nfo, an .srt) exists ONLY above, by design, and is exactly what the view
|
||||
# is for. A copy-up whose lower file was deleted by hand is the one case this
|
||||
# leaves in place; the raw path is where deletions belong, so it stays visible.
|
||||
do_reap() {
|
||||
[ -d "$upper" ] || die "refused: $upper does not exist (no upper layer at $upper_base)"
|
||||
stamp="$upper_base/.AGENTTRASH/$(date -u +%Y%m%d-%H%M%S)"
|
||||
count=0
|
||||
ghosts=0
|
||||
reap_walk ""
|
||||
if [ "$count" -eq 0 ] && [ "$ghosts" -eq 0 ]; then
|
||||
printf 'reap: no ghosts under %s\n' "$upper"
|
||||
return 0
|
||||
fi
|
||||
if [ "$ghosts" -gt 0 ]; then
|
||||
printf 'reap: %d whiteout%s above — someone deleted THROUGH the view.\n' \
|
||||
"$ghosts" "$([ "$ghosts" -eq 1 ] || printf s)" >&2
|
||||
printf ' Leaving them: removing a whiteout un-hides the file it hides.\n' >&2
|
||||
printf ' Deletions belong in .AGENTTRASH on the raw path.\n' >&2
|
||||
fi
|
||||
if [ "$count" -eq 0 ]; then return 0; fi
|
||||
if [ "$apply" -eq 1 ]; then
|
||||
printf 'reap: moved %d entr%s to %s\n' "$count" \
|
||||
"$([ "$count" -eq 1 ] && printf y || printf ies)" "$stamp"
|
||||
else
|
||||
printf 'reap: %d ghost entr%s above; re-run with --apply to move them to %s\n' \
|
||||
"$count" "$([ "$count" -eq 1 ] && printf y || printf ies)" "$stamp"
|
||||
fi
|
||||
}
|
||||
|
||||
# Descend only where the lower layer still has the directory: a directory that is
|
||||
# missing below is itself the ghost, and its whole subtree goes with it.
|
||||
reap_walk() {
|
||||
local rel="$1" entries abs r name
|
||||
entries="$(find "$upper${rel:+/$rel}" -mindepth 1 -maxdepth 1 -print 2>/dev/null || true)"
|
||||
[ -n "$entries" ] || return 0
|
||||
while IFS= read -r abs; do
|
||||
r="${abs#"$upper"/}"
|
||||
name="${abs##*/}"
|
||||
if [ -c "$abs" ] || printf '%s' "$name" | grep -q '^\.wh\.'; then
|
||||
ghosts=$((ghosts + 1))
|
||||
continue
|
||||
fi
|
||||
if [ -e "$lower/$r" ]; then
|
||||
[ -d "$abs" ] && reap_walk "$r"
|
||||
continue
|
||||
fi
|
||||
[ -d "$abs" ] || continue
|
||||
count=$((count + 1))
|
||||
printf '%s\n' "$r"
|
||||
if [ "$apply" -eq 1 ]; then
|
||||
mkdir -p "$stamp/$(dirname "$r")"
|
||||
mv "$abs" "$stamp/$r"
|
||||
fi
|
||||
done <<<"$entries"
|
||||
}
|
||||
|
||||
do_status() {
|
||||
if mountpoint -q "$VIEW"; then
|
||||
printf 'view: %s (%s)\n' "$VIEW" "$(findmnt -no OPTIONS --mountpoint "$VIEW")"
|
||||
else
|
||||
printf 'view: %s NOT MOUNTED\n' "$VIEW"
|
||||
fi
|
||||
if [ -d "$upper" ]; then
|
||||
printf 'upper: %s %s files %s\n' "$upper" \
|
||||
"$(find "$upper" -type f 2>/dev/null | wc -l)" "$(du -sh "$upper" 2>/dev/null | cut -f1)"
|
||||
else
|
||||
printf 'upper: %s missing\n' "$upper"
|
||||
fi
|
||||
systemctl is-active "$UNIT" 2>/dev/null | sed 's/^/unit: /' || printf 'unit: %s unknown\n' "$UNIT"
|
||||
}
|
||||
|
||||
case "$cmd" in
|
||||
plan) do_plan ;;
|
||||
verify) do_verify ;;
|
||||
install) do_install ;;
|
||||
reap) do_reap ;;
|
||||
status) do_status ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
Executable
+335
@@ -0,0 +1,335 @@
|
||||
#!/bin/sh
|
||||
# Deterministically empty every .AGENTTRASH directory under the given roots.
|
||||
#
|
||||
# PURPOSE
|
||||
# The fleet's deletion policy parks deleted files in .AGENTTRASH/ — in-repo
|
||||
# beside the item, elsewhere in $HOME as .AGENTTRASH/YYYYMMDD-topic — and
|
||||
# waits for approval to purge. This script is that purge: given a machine,
|
||||
# it clears .AGENTTRASH directories and nothing else.
|
||||
#
|
||||
# CONTRACT
|
||||
# Default dry-run: the plan prints and nothing is deleted; --execute deletes.
|
||||
# Every line the script can ever print for a candidate is decided at plan time
|
||||
# and stored in the plan table, so a dry run prints exactly what --execute
|
||||
# will print. A .AGENTTRASH that is a root's own trash (a repo's
|
||||
# .AGENTTRASH, a machine's ~/.AGENTTRASH) prints `plan empty <path>`: the
|
||||
# directory is the convention and survives, its staged contents are cleared.
|
||||
# A .AGENTTRASH deeper than that is content and prints `plan remove <path>`
|
||||
# plus, once executed, `removed <path>`; its own nested .AGENTTRASH entries
|
||||
# belong to that tree and are never planned again. Candidates are found with
|
||||
# `find -xdev` per root, so a mounted tree can never be walked by accident;
|
||||
# ordering is lexicographic (LC_ALL=C). `/`, symlinked, and foreign-owned
|
||||
# candidates are refused and reported (exit 1), never touched.
|
||||
# A summary line `removed=<candidates> deleted=<removed> refused=<n>` closes
|
||||
# every run and one dated line is appended to the log.
|
||||
# Exit: 0 success, 1 operational failure, 2 usage error. A second run over
|
||||
# the same tree plans nothing further and exits 0.
|
||||
#
|
||||
# PORTABILITY
|
||||
# POSIX sh, clean under `shellcheck -S style`. Externals: find (must support
|
||||
# -delete), stat, rm, date, mkdir. No bashisms, no GNU-only flags, no arrays,
|
||||
# no `local`. Paths containing whitespace are rejected at parse time — the
|
||||
# same rule applied to roots and candidates — so the plan can be walked with
|
||||
# plain word splitting and nothing can be silently mis-split.
|
||||
#
|
||||
# USAGE
|
||||
# purge-agenttrash [--execute] [--quiet] [--log PATH] [PATH...]
|
||||
# --execute delete; without it the run is a plan and deletes nothing
|
||||
# --quiet suppress per-line output; errors and the summary still print
|
||||
# --log PATH log destination (default: $HOME/.AGENTTRASH-cleanup.log)
|
||||
# PATH... roots to search (default: $HOME)
|
||||
#
|
||||
# Everything else is fixed by design: the root default, the ordering, the
|
||||
# guards. There is no configuration file and no environment override.
|
||||
|
||||
set -eu
|
||||
|
||||
usage() {
|
||||
sed -n '3,44p' "$0" | sed 's/^# \{0,1\}//'
|
||||
}
|
||||
|
||||
die() {
|
||||
printf '%s: %s\n' 'purge-agenttrash' "$1" >&2
|
||||
exit "${2:-1}"
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------ arg parser
|
||||
# Roots accumulate as one space-separated string: a root containing whitespace is
|
||||
# rejected at parse time rather than silently split later. That leaves the plan
|
||||
# iteratable with plain `$@`-free word splitting, the only portable construct
|
||||
# that cannot corrupt a path it has already vetted.
|
||||
MODE=plan
|
||||
QUIET=0
|
||||
LOGPATH=
|
||||
ROOTS=
|
||||
while [ "$#" -gt 0 ]; do
|
||||
case "$1" in
|
||||
--help|-h)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
--execute)
|
||||
MODE=execute
|
||||
;;
|
||||
--quiet)
|
||||
QUIET=1
|
||||
;;
|
||||
--log)
|
||||
[ "$#" -ge 2 ] || die '--log needs a value' 2
|
||||
LOGPATH=$2
|
||||
shift
|
||||
;;
|
||||
--log=*)
|
||||
LOGPATH=${1#--log=}
|
||||
;;
|
||||
-*)
|
||||
die "unknown option: $1" 2
|
||||
;;
|
||||
*)
|
||||
case $1 in
|
||||
*' '*) die "root contains whitespace: $1" 2 ;;
|
||||
esac
|
||||
if [ -z "$ROOTS" ]; then
|
||||
ROOTS=$1
|
||||
else
|
||||
ROOTS="$ROOTS $1"
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
[ -n "${HOME:-}" ] || die 'HOME is not set; pass roots explicitly or set HOME' 2
|
||||
[ -n "$LOGPATH" ] || LOGPATH=$HOME/.AGENTTRASH-cleanup.log
|
||||
|
||||
# ------------------------------------------------------------ tool availability
|
||||
require() {
|
||||
command -v "$1" >/dev/null 2>&1 || die "required tool not found: $1" 1
|
||||
}
|
||||
require find
|
||||
require stat
|
||||
require rm
|
||||
require date
|
||||
require mkdir
|
||||
|
||||
# `find -delete` backs the in-place emptying. Probe it rather than trusting
|
||||
# `find --version`: a shell function named `find` must not be able to fake it.
|
||||
if ! find . -maxdepth 0 -name '.purge-probe-does-not-exist' -delete >/dev/null 2>&1; then
|
||||
die 'find does not support -delete; refusing to guess at deletion' 1
|
||||
fi
|
||||
|
||||
UID_SELF=$(id -u)
|
||||
|
||||
# ------------------------------------------------------------------- locking
|
||||
# The lock dir also hosts this run's plan table, so both live and die together:
|
||||
# the exit trap removes the whole directory and only the log survives the run.
|
||||
LOCKDIR=$HOME/.agenttrash-purge.lock
|
||||
PLANFILE=$LOCKDIR/plan
|
||||
|
||||
acquire_lock() {
|
||||
if ! mkdir "$LOCKDIR" 2>/dev/null; then
|
||||
held_pid=$(cat "$LOCKDIR/pid" 2>/dev/null || printf '%s\n' '')
|
||||
if [ -n "$held_pid" ] && kill -0 "$held_pid" 2>/dev/null; then
|
||||
die "another purge holds $LOCKDIR (pid $held_pid)" 1
|
||||
fi
|
||||
# Owner died mid-run: drop its pid, then retry exactly once. An
|
||||
# unremovable/nonempty lock dir is not ours to reclaim.
|
||||
rm -f "$LOCKDIR/pid" 2>/dev/null || true
|
||||
rmdir "$LOCKDIR" 2>/dev/null || die "$LOCKDIR exists and is not ours to reclaim" 1
|
||||
mkdir "$LOCKDIR" 2>/dev/null || die "cannot reacquire $LOCKDIR" 1
|
||||
fi
|
||||
printf '%s\n' "$$" >"$LOCKDIR/pid" || die "cannot stamp $LOCKDIR/pid" 1
|
||||
: >"$PLANFILE" || die "cannot write $PLANFILE" 1
|
||||
trap 'rm -rf "$LOCKDIR"' EXIT
|
||||
trap 'exit 130' INT
|
||||
trap 'exit 143' TERM
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------ predicates
|
||||
is_dir() { # true only for a real directory, never a symlink to one
|
||||
[ -d "$1" ] && [ ! -L "$1" ]
|
||||
}
|
||||
|
||||
is_ours() { # true only when the invoking user owns the path
|
||||
owner=$(stat -c '%u' "$1" 2>/dev/null) || return 1
|
||||
[ "$owner" = "$UID_SELF" ]
|
||||
}
|
||||
|
||||
refuse() { # refuse <path> <reason> — reported, never deleted
|
||||
printf '%s: refused %s (%s)\n' 'purge-agenttrash' "$1" "$2" >&2
|
||||
REFUSED=$((REFUSED + 1))
|
||||
}
|
||||
|
||||
emit() { # emit <line> — silenced by --quiet
|
||||
[ "$QUIET" -eq 1 ] || printf '%s\n' "$1"
|
||||
}
|
||||
|
||||
# --------------------------------------------------------------- the plan table
|
||||
# One row per candidate: `<action> <path> <line>`. The first two fields tell
|
||||
# `act_on` what to do; the rest of the row is EXACTLY what the run prints for
|
||||
# that candidate — in plan mode verbatim, in execute mode as the first line of
|
||||
# its report. The table is a file, so no shell quoting rule (quoted-for,
|
||||
# heredoc, IFS games) can corrupt an approved plan, and the two modes can
|
||||
# never drift apart. A removal's second line (`removed <path>`) is derived from
|
||||
# the row's action, so it is also fixed at plan time.
|
||||
CANDIDATES=0
|
||||
REFUSED=0
|
||||
|
||||
record() { # record <action> <path> <line> — append one row to the plan table
|
||||
printf '%s %s %s\n' "$1" "$2" "$3" >>"$PLANFILE"
|
||||
CANDIDATES=$((CANDIDATES + 1))
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------- candidate rule
|
||||
# The whole contract in one place:
|
||||
# `/` is refused outright (never a candidate, never walked);
|
||||
# a root that is a symlink or a plain file is refused, never followed;
|
||||
# a root (or any path) that IS a real .AGENTTRASH prints `plan empty <path>`:
|
||||
# the convention persists, its staged contents are what gets cleared;
|
||||
# otherwise the root's DIRECT .AGENTTRASH child is also `empty` — it is that
|
||||
# level's own trash ($HOME/.AGENTTRASH, x1/.AGENTTRASH, repo-a/.AGENTTRASH);
|
||||
# every .AGENTTRASH deeper than that is content: `remove`, printed as
|
||||
# `plan remove <path>` (+ `removed <path>` when executed), unless it lives
|
||||
# inside a tree already claimed (then it dies with that tree, never planned
|
||||
# twice) or is foreign-owned/symlinked (then refused).
|
||||
collect_root() { # collect_root <path> -> 0 ok, 1 refused
|
||||
target=$1
|
||||
if [ "$target" = / ]; then
|
||||
refuse / 'root refusal'
|
||||
return 1
|
||||
fi
|
||||
if [ -L "$target" ] || ! [ -d "$target" ]; then
|
||||
refuse "$target" 'not a real directory'
|
||||
return 1
|
||||
fi
|
||||
if ! is_ours "$target"; then
|
||||
refuse "$target" "not owned by uid $UID_SELF"
|
||||
return 1
|
||||
fi
|
||||
if [ "${target##*/}" = .AGENTTRASH ]; then
|
||||
record empty "$target" "plan empty $target"
|
||||
return 0
|
||||
fi
|
||||
# `own` is the root's own .AGENTTRASH once handled; hits below it belong to
|
||||
# that tree and must never be planned a second time.
|
||||
own=
|
||||
if [ -e "$target/.AGENTTRASH" ]; then
|
||||
if ! is_ours "$target/.AGENTTRASH"; then
|
||||
refuse "$target/.AGENTTRASH" "not owned by uid $UID_SELF"
|
||||
elif ! is_dir "$target/.AGENTTRASH"; then
|
||||
refuse "$target/.AGENTTRASH" 'not a real directory'
|
||||
else
|
||||
record empty "$target/.AGENTTRASH" "plan empty $target/.AGENTTRASH"
|
||||
own=$target/.AGENTTRASH
|
||||
fi
|
||||
fi
|
||||
# No `-type d`: implementations disagree on whether that matches a
|
||||
# symlink-to-directory (GNU find says no, others yes), so every entry named
|
||||
# .AGENTTRASH is collected and classified here instead of by `find`.
|
||||
if ! found=$(find "$target" -xdev -name .AGENTTRASH 2>/dev/null); then
|
||||
refuse "$target" 'find failed'
|
||||
return 1
|
||||
fi
|
||||
previous=
|
||||
for entry in $found; do
|
||||
[ -n "$entry" ] || continue
|
||||
if [ -n "$own" ]; then
|
||||
case $entry in
|
||||
"$own"|"$own"/*) continue ;; # already emptied in place
|
||||
esac
|
||||
fi
|
||||
if [ -n "$previous" ]; then
|
||||
case $entry in
|
||||
"$previous"/*) continue ;; # nested: dies with its parent
|
||||
esac
|
||||
fi
|
||||
previous=$entry
|
||||
if ! is_ours "$entry"; then
|
||||
refuse "$entry" "not owned by uid $UID_SELF"
|
||||
continue
|
||||
fi
|
||||
if ! is_dir "$entry"; then
|
||||
refuse "$entry" 'not a real directory'
|
||||
continue
|
||||
fi
|
||||
record remove "$entry" "plan remove $entry"
|
||||
done
|
||||
return 0
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------ execution
|
||||
remove_tree() { # rm -rf a real directory; symlinks inside die as entries
|
||||
is_dir "$1" || return 1
|
||||
rm -rf -- "$1"
|
||||
}
|
||||
|
||||
empty_dir() { # empty a real directory in place, never following symlinks
|
||||
is_dir "$1" || return 1
|
||||
find "$1" -mindepth 1 -xdev -delete
|
||||
}
|
||||
|
||||
# act_on <action> <path> <line> -> 0 when the action succeeded.
|
||||
# remove: prints the stored line (`plan remove <path>`) then `removed <path>`.
|
||||
# empty: prints the stored line (`plan empty <path>`) and nothing else — the
|
||||
# directory survives, so the plan line IS the outcome.
|
||||
act_on() { # act_on <action> <path> <line> -> 0 when the action succeeded
|
||||
if [ "$1" = remove ]; then
|
||||
if remove_tree "$2" 2>/dev/null; then
|
||||
emit "$3"
|
||||
emit "removed $2"
|
||||
return 0
|
||||
fi
|
||||
printf '%s: failed to remove %s\n' 'purge-agenttrash' "$2" >&2
|
||||
return 1
|
||||
fi
|
||||
if empty_dir "$2" 2>/dev/null; then
|
||||
emit "$3"
|
||||
return 0
|
||||
fi
|
||||
printf '%s: failed to empty %s\n' 'purge-agenttrash' "$2" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
# Walk the plan table once: plan mode prints every row's line verbatim, execute
|
||||
# mode acts on each row and counts failures separately from plan-time refusals.
|
||||
walk_plan() {
|
||||
while read -r action path line; do
|
||||
[ -n "$action" ] || continue
|
||||
if [ "$MODE" = plan ]; then
|
||||
emit "$line"
|
||||
elif act_on "$action" "$path" "$line"; then
|
||||
REMOVED=$((REMOVED + 1))
|
||||
else
|
||||
EXEC_REFUSED=$((EXEC_REFUSED + 1))
|
||||
fi
|
||||
done <"$PLANFILE"
|
||||
}
|
||||
|
||||
# ----------------------------------------------------------------------- main
|
||||
[ -n "$ROOTS" ] || ROOTS=$HOME
|
||||
|
||||
acquire_lock
|
||||
|
||||
for root in $ROOTS; do
|
||||
collect_root "$root" || true
|
||||
done
|
||||
|
||||
REMOVED=0
|
||||
EXEC_REFUSED=0
|
||||
walk_plan
|
||||
|
||||
STATUS=0
|
||||
[ "$REFUSED" -eq 0 ] || STATUS=1
|
||||
[ "$EXEC_REFUSED" -eq 0 ] || STATUS=1
|
||||
|
||||
printf 'removed=%s deleted=%s refused=%s\n' \
|
||||
"$CANDIDATES" "$REMOVED" "$((REFUSED + EXEC_REFUSED))"
|
||||
|
||||
if ! printf '%s\tplan\tcandidates=%s\tremoved=%s\trefused=%s\n' \
|
||||
"$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$CANDIDATES" "$REMOVED" \
|
||||
"$((REFUSED + EXEC_REFUSED))" >>"$LOGPATH" 2>/dev/null; then
|
||||
printf '%s: cannot write log %s\n' 'purge-agenttrash' "$LOGPATH" >&2
|
||||
STATUS=1
|
||||
fi
|
||||
|
||||
exit "$STATUS"
|
||||
Executable
+143
@@ -0,0 +1,143 @@
|
||||
#!/usr/bin/env bash
|
||||
# qwen38-calib-window.sh — RUNS ON DANKSERVER. The calibration window, and nothing else:
|
||||
# boot the BF16 rig, wait until it is actually serving, push the corpus until the engine says
|
||||
# the tap is full, stop the rig, restore frank-archon, and bank the reference container.
|
||||
#
|
||||
# The reference recording is already taken (kld-ref/kld.json exists), so this window is the
|
||||
# calibration tap alone — the one job that needs both cards after it. The kldref phase is NOT
|
||||
# re-run here: the chained launcher that used to do that moved the banked kld-ref into trash
|
||||
# first, which is how a 16-hour phase gets paid for twice.
|
||||
#
|
||||
# Every phase is gated on the previous one, because the last time this ran unguarded
|
||||
# (2026-10-29) the rig's boot died of an OOM, the health loop quietly fell through, and the
|
||||
# pusher fed four rounds of 10M tokens into RESTORED PRODUCTION with nothing recorded:
|
||||
#
|
||||
# 1. the runner must still be alive and its container must answer /health AND own the
|
||||
# listening socket, or this script kills it, restores the service and exits non-zero —
|
||||
# no push, ever. The socket clause is not decoration: the runner stops frank-archon
|
||||
# first, the dying process keeps its listener for a moment, and a rig container that
|
||||
# is still loading has not bound the port yet, so `podman inspect` + `/health` both
|
||||
# report success for the WRONG process (measured 2026-10-30: `health ok waited=0s`,
|
||||
# then the pusher refused on the container name and the window died at second zero);
|
||||
# 2. the pusher refuses on its own unless the rig wrote its marker into the tap dir, the
|
||||
# marker names this container, and the container is running (scripts/calib-push.py
|
||||
# --tap-marker), so a second lock survives a mistake in this file;
|
||||
# 3. the reference container is banked into trash only when BOTH the serve and the push
|
||||
# exited 0 and the tap dir holds grams. The checkpoint that reproduces it stays.
|
||||
#
|
||||
# knobs: Q38_WINDOW_CALIB_CORPUS (calib-7168.jsonl — the re-chunked file the rig's pool can
|
||||
# hold), Q38_WINDOW_HEALTH_TIMEOUT (seconds to wait for /health), Q38_WINDOW_ROUNDS
|
||||
# (pusher rounds; one is 10M tokens), Q38_WINDOW_KEEP_RAD=1 to leave the .rad in place.
|
||||
# Usage: scripts/qwen38-calib-window.sh [--dry-run]
|
||||
set -uo pipefail
|
||||
cd "$(dirname "${BASH_SOURCE[0]}")/.." || exit 9
|
||||
|
||||
DRY=0
|
||||
[ "${1:-}" = "--dry-run" ] && DRY=1
|
||||
say() { printf '%s %s\n' "$(date -Is)" "$*"; }
|
||||
|
||||
MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
|
||||
CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
|
||||
RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
|
||||
CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}"
|
||||
CALIB="${FRANK_QWEN38_CALIB_DIR:-$CAMPAIGN/calib}"
|
||||
HEALTH_TIMEOUT="${Q38_WINDOW_HEALTH_TIMEOUT:-3600}"
|
||||
ROUNDS="${Q38_WINDOW_ROUNDS:-6}"
|
||||
RIG=qwen38-bf16
|
||||
SERVICE=frank-archon
|
||||
PY="${PY:-python3}"
|
||||
|
||||
# ---- preflight: every fact by identity, one owner (scripts/qwen38-preflight.sh) ----
|
||||
# The checks that used to live here were proxies, and each lied at least once: the space check
|
||||
# asked a directory that merely LOOKED related, `/health` answered for a dying production
|
||||
# process, `podman inspect` named a container that existed but had bound nothing. The shared
|
||||
# script binds each fact to its object — cwd, the digest the tag actually resolves to, free
|
||||
# space on the tap's own filesystem, every /dev/kfd holder's HOST pid against the service's,
|
||||
# the :7770 listener's host pid, the corpus and the reference recording. A resumed window
|
||||
# keeps an existing tap, so the tap-empty fact is waived.
|
||||
[ -f "$RAD" ] || { echo "REFUSED: no BF16 reference container at $RAD — nothing to serve; scripts/build-qwen38-bf16-rad.sh" >&2; exit 2; }
|
||||
FRANK_QWEN38_CAMPAIGN="$CAMPAIGN" \
|
||||
Q38_WINDOW_CALIB_CORPUS="$CORPUS" \
|
||||
Q38_CALIB_MIN_FREE_MIB="${Q38_CALIB_MIN_FREE_MIB:-61440}" \
|
||||
bash scripts/qwen38-preflight.sh --allow-existing-tap || exit 2
|
||||
! [[ "$HEALTH_TIMEOUT" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_HEALTH_TIMEOUT must be seconds (got: $HEALTH_TIMEOUT)" >&2; exit 2; }
|
||||
! [[ "$ROUNDS" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_ROUNDS must be a round count (got: $ROUNDS)" >&2; exit 2; }
|
||||
|
||||
if [ "$DRY" = 1 ]; then
|
||||
echo "dry run: calib window — the service is left running, nothing is mounted or written"
|
||||
echo "would run: bash scripts/run-qwen38-bf16.sh calib (rig boots, writes $CALIB/.rig-boot)"
|
||||
echo "would wait: /health up within ${HEALTH_TIMEOUT}s, else kill the rig, restore $SERVICE, exit 1"
|
||||
echo "would run: $PY scripts/calib-push.py --container $RIG --corpus $CORPUS --max-rounds $ROUNDS"
|
||||
echo "would bank: $RAD -> ~/.AGENTTRASH/$(date +%Y%m%d)-qwen38-bf16-rad/ (only if both phases exit 0)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
say "calib window start pid=$$ corpus=$CORPUS rounds=$ROUNDS"
|
||||
[ -n "$(ls -A "$CALIB" 2>/dev/null)" ] && say "tap dir $CALIB is not empty; the engine's token floor is pooled, so an earlier partial tap still counts"
|
||||
|
||||
# ---- phase 1: the rig -------------------------------------------------------
|
||||
say "phase=calib serve begin"
|
||||
bash scripts/run-qwen38-bf16.sh calib & spid=$!
|
||||
|
||||
rig_up() {
|
||||
podman inspect -f '{{.State.Running}}' "$RIG" 2>/dev/null | grep -qx true || return 1
|
||||
curl -sf --max-time 10 -o /dev/null http://127.0.0.1:7770/health || return 1
|
||||
# The answer must come from the CONTAINER. The runner stops frank-archon first, and the
|
||||
# dying production process keeps its listener for a moment while the freshly created
|
||||
# rig container is still loading — so `container running` + `/health` 200 both hold
|
||||
# before the rig has bound anything. Measured 2026-10-30: `health ok waited=0s`, then the
|
||||
# pusher refused on the container name and the window died at second zero. Ask the kernel
|
||||
# who owns the listening socket and require it to be a rig process.
|
||||
local lpid
|
||||
lpid=$(ss -H -ltnp 'sport = :7770' 2>/dev/null | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)
|
||||
[ -n "${lpid:-}" ] || return 1
|
||||
podman top "$RIG" pid 2>/dev/null | grep -qw "$lpid"
|
||||
}
|
||||
|
||||
waited=0
|
||||
until rig_up; do
|
||||
if ! kill -0 "$spid" 2>/dev/null; then
|
||||
wait "$spid"; crc=$?
|
||||
say "phase=calib serve end rc=$crc (the rig exited before it ever answered /health)"
|
||||
echo "FAIL: the rig boot died (rc=$crc) — nothing was pushed; read /home/dank/logs/qwen38_bf16_calib.log" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ "$waited" -ge "$HEALTH_TIMEOUT" ]; then
|
||||
say "phase=calib health never came after ${HEALTH_TIMEOUT}s; killing the rig"
|
||||
podman stop "$RIG" >/dev/null 2>&1 || true
|
||||
wait "$spid"
|
||||
echo "FAIL: the rig never answered /health within ${HEALTH_TIMEOUT}s — nothing was pushed" >&2
|
||||
exit 1
|
||||
fi
|
||||
sleep 15
|
||||
waited=$((waited + 15))
|
||||
done
|
||||
say "phase=calib health ok waited=${waited}s"
|
||||
|
||||
# ---- phase 2: the tap -------------------------------------------------------
|
||||
say "phase=calib push begin"
|
||||
"$PY" scripts/calib-push.py --port 7770 --parallel 8 --container "$RIG" \
|
||||
--tap-dir "$CALIB" --corpus "$CORPUS" --max-rounds "$ROUNDS"
|
||||
prc=$?
|
||||
say "phase=calib push end rc=$prc"
|
||||
|
||||
podman stop "$RIG" >/dev/null 2>&1 || true
|
||||
wait "$spid"; crc=$?
|
||||
say "phase=calib serve end rc=$crc"
|
||||
|
||||
if [ "$prc" -ne 0 ] || [ "$crc" -ne 0 ]; then
|
||||
echo "FAIL: serve rc=$crc push rc=$prc — the tap is NOT banked and $RAD stays: $(basename "$RAD")" >&2
|
||||
exit 1
|
||||
fi
|
||||
[ -n "$(ls -A "$CALIB" 2>/dev/null)" ] || { echo "FAIL: both phases exited 0 but the tap dir $CALIB is empty" >&2; exit 1; }
|
||||
say "TAP BANKED: $(ls -1 "$CALIB" | wc -l) file(s) in $CALIB"
|
||||
|
||||
# ---- phase 3: bank the reference container ----------------------------------
|
||||
if [ "${Q38_WINDOW_KEEP_RAD:-0}" = "1" ]; then
|
||||
say "reference .rad kept (Q38_WINDOW_KEEP_RAD=1)"
|
||||
else
|
||||
T="${Q38_WINDOW_TRASH_ROOT:-$HOME/.AGENTTRASH}/$(date +%Y%m%d)-qwen38-bf16-rad"
|
||||
mkdir -p "$T" || { echo "FAIL: cannot create trash dir $T" >&2; exit 1; }
|
||||
mv "$RAD" "$T/" && say "reference .rad banked in $T (the checkpoint that rebuilds it stays)"
|
||||
fi
|
||||
say "calib window end"
|
||||
Executable
+136
@@ -0,0 +1,136 @@
|
||||
#!/usr/bin/env bash
|
||||
# qwen38-preflight.sh — the facts a calibration window must have BEFORE it takes the cards,
|
||||
# each checked by IDENTITY and not by proxy. Every line is PASS or FAIL with the reason.
|
||||
#
|
||||
# Why this file exists: on 2026-10-30 a window died at second zero because `/health` answered
|
||||
# 200 from the *production* process that was still draining its listener, while the freshly
|
||||
# created rig container had not bound anything. Other runs died from a port that answered for
|
||||
# the wrong process, a /dev/kfd held by an orphaned engine, a tap dir on a 35 GiB filesystem,
|
||||
# and a launcher whose cwd silently lacked the repo. None of those were math problems; all of
|
||||
# them were a check that measured something adjacent to the thing it meant.
|
||||
#
|
||||
# Usage: bash scripts/qwen38-preflight.sh [--allow-existing-tap]
|
||||
# Exit: 0 all PASS · 2 any FAIL · 3 a fact could not be evaluated (missing tool)
|
||||
set -uo pipefail
|
||||
|
||||
SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "$SELF_DIR/.." && pwd)"
|
||||
|
||||
CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
|
||||
TAP="$CAMPAIGN/calib"
|
||||
CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}"
|
||||
NEED_MIB="${Q38_CALIB_MIN_FREE_MIB:-61440}"
|
||||
ALLOW_EXISTING=0
|
||||
[ "${1:-}" = "--allow-existing-tap" ] && ALLOW_EXISTING=1
|
||||
|
||||
# The producer pin is the calib image; the window fails closed if the tag on this host is not it.
|
||||
WANT_DIGEST="${Q38_PREFLIGHT_IMAGE_DIGEST:-sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}"
|
||||
IMAGE_REF="${Q38_CALIB_IMAGE:-stilldeadcode/radiance@$WANT_DIGEST}"
|
||||
SERVICE="frank-archon"
|
||||
PORT=7770
|
||||
|
||||
fails=0
|
||||
ok() { printf 'PASS %-14s %s\n' "$1" "$2"; }
|
||||
bad() { printf 'FAIL %-14s %s\n' "$1" "$2"; fails=$((fails + 1)); }
|
||||
skip() { printf 'WARN %-14s %s\n' "$1" "$2"; }
|
||||
|
||||
# 1. cwd — a detached launcher that lost its cwd fails here in a second, not in the log later.
|
||||
if [ "$(cd "$REPO_ROOT" && pwd)" = "$(pwd)" ] && [ -x "$REPO_ROOT/scripts/run-qwen38-bf16.sh" ]; then
|
||||
ok cwd "running from the repo root ($REPO_ROOT)"
|
||||
else
|
||||
bad cwd "cwd is $(pwd), not the repo root ($REPO_ROOT) — every relative script path would break"
|
||||
fi
|
||||
|
||||
# 2. image — resolve the REF to a digest. A stale local `:latest` is a different model of the world.
|
||||
if command -v podman >/dev/null 2>&1; then
|
||||
# An image carries EVERY digest it was pulled under (a tag plus the by-digest pull), so this
|
||||
# asks whether the pin is one of them — reading RepoDigests[0] reports a healthy image as wrong.
|
||||
got="$(podman image inspect -f '{{range .RepoDigests}}{{println .}}{{end}}' "$IMAGE_REF" 2>/dev/null \
|
||||
| sed 's/.*@//' | grep -c "^${WANT_DIGEST}$")"
|
||||
if [ "${got:-0}" -ge 1 ]; then
|
||||
ok image "$IMAGE_REF carries ${WANT_DIGEST:0:19}…"
|
||||
elif podman image exists "$IMAGE_REF" 2>/dev/null; then
|
||||
bad image "$IMAGE_REF is present but does not carry $WANT_DIGEST (a retagged or older pull)"
|
||||
else
|
||||
bad image "$IMAGE_REF is not present on this host"
|
||||
fi
|
||||
else
|
||||
skip image "podman not on PATH — cannot resolve $IMAGE_REF"
|
||||
fi
|
||||
|
||||
# 3. space — df the FILE PATH the seal writes to, never a directory that merely looks related.
|
||||
if [ -d "$TAP" ] || mkdir -p "$TAP" 2>/dev/null; then
|
||||
avail="$(df --output=avail -B 1M "$TAP" 2>/dev/null | tail -1 | tr -d ' ')"
|
||||
if [[ "${avail:-}" =~ ^[0-9]+$ ]] && [ "$avail" -ge "$NEED_MIB" ]; then
|
||||
ok space "$avail MiB free on $(df --output=target "$TAP" | tail -1) (need $NEED_MIB)"
|
||||
else
|
||||
bad space "${avail:-unknown} MiB free on $TAP, need $NEED_MIB — one seal writes ~43 GB"
|
||||
fi
|
||||
# 4. tap cleanliness — stale grams from another run must never be mixed in.
|
||||
n=$(ls -A "$TAP" 2>/dev/null | grep -c . || true)
|
||||
if [ "$n" -eq 0 ] || [ "$ALLOW_EXISTING" = 1 ]; then
|
||||
ok tap-empty "$n entries in $TAP"
|
||||
else
|
||||
bad tap-empty "$n entries in $TAP — rotate them to .AGENTTRASH first"
|
||||
fi
|
||||
else
|
||||
bad space "cannot create $TAP"
|
||||
fi
|
||||
|
||||
# 5. device — every /dev/kfd holder must be a process of the service we are about to stop. An
|
||||
# orphaned engine from a previous run makes the next boot die with `startup stopped: device
|
||||
# error`. Identity comes from podman's own host-pid list: `podman top <ctr> pid` prints
|
||||
# CONTAINER-internal pids (1, 7, …), which compare against nothing, and a rootless cgroup
|
||||
# carries the container ID, never the service name — both are proxies that lie.
|
||||
PROD_CTR="${Q38_PREFLIGHT_PROD_CONTAINER:-$SERVICE}"
|
||||
service_pids() { podman top "$PROD_CTR" hpid 2>/dev/null | tail -n +2; }
|
||||
if [ -e /dev/kfd ]; then
|
||||
mine="$(service_pids | tr '\n' ' ')"
|
||||
foreign=""
|
||||
for p in $(fuser /dev/kfd 2>/dev/null); do
|
||||
case " $mine " in
|
||||
*" $p "*) : ;; # the service we will stop
|
||||
*) foreign="$foreign $p:$(cat /proc/$p/comm 2>/dev/null)" ;;
|
||||
esac
|
||||
done
|
||||
if [ -z "$foreign" ]; then
|
||||
ok device "no foreign /dev/kfd holders"
|
||||
else
|
||||
bad device "foreign holders:$foreign — kill by pid, never by pattern"
|
||||
fi
|
||||
else
|
||||
skip device "/dev/kfd absent — not an AMD host?"
|
||||
fi
|
||||
|
||||
# 6. port — a listener on :7770 must be a process of the service we stop, never an unknown one.
|
||||
lpid="$(ss -H -ltnp "sport = :$PORT" 2>/dev/null | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)"
|
||||
if [ -z "${lpid:-}" ]; then
|
||||
ok port ":$PORT is free"
|
||||
else
|
||||
mine="$(service_pids | tr '\n' ' ')"
|
||||
case " $mine " in
|
||||
*" $lpid "*) ok port ":$PORT is held by $SERVICE (host pid $lpid) — the window stops it" ;;
|
||||
*) bad port ":$PORT is held by host pid $lpid ($(cat /proc/$lpid/comm 2>/dev/null)), not $SERVICE" ;;
|
||||
esac
|
||||
fi
|
||||
|
||||
# 7. corpus — the push source must exist and be the re-chunked file the window documents.
|
||||
if [ -s "$CORPUS" ]; then
|
||||
ok corpus "$(wc -l < "$CORPUS") prompts in $CORPUS"
|
||||
else
|
||||
bad corpus "$CORPUS is missing or empty"
|
||||
fi
|
||||
|
||||
# 8. reference — the KL gate has nothing to compare against without it.
|
||||
if [ -s "$CAMPAIGN/kld-ref/kld.json" ]; then
|
||||
ok reference "$(du -h "$CAMPAIGN/kld-ref/kld.json" | cut -f1) kld.json present"
|
||||
else
|
||||
bad reference "$CAMPAIGN/kld-ref/kld.json is missing"
|
||||
fi
|
||||
|
||||
echo
|
||||
if [ "$fails" -gt 0 ]; then
|
||||
echo "PREFLIGHT FAILED: $fails fact(s). The window must not take the cards."
|
||||
exit 2
|
||||
fi
|
||||
echo "PREFLIGHT PASSED: every fact checked by identity."
|
||||
Executable
+94
@@ -0,0 +1,94 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Shared plumbing for the qwen38 campaign corpus tools.
|
||||
|
||||
The three tools move the same two shapes — a JSONL of prompt records and a
|
||||
one-artifact JSON manifest — and they all tokenize with the SAME tokenizer. That
|
||||
last part is not a detail: a KLD reference is only meaningful against the exact
|
||||
token sequence the corpus recorded, so `n_tokens` exists once, here, and every
|
||||
tool counts the same way. A window that decode/encode does not round-trip is a
|
||||
refusal, never a warning.
|
||||
|
||||
Loaded as a sibling module. Each tool puts its own directory on `sys.path`
|
||||
first, so `python3 scripts/<tool>.py` and the driver's container mount
|
||||
(`runpy.run_path('/work/<tool>.py')`, `-v scripts:/work:ro`) both resolve it —
|
||||
run the tools by path, not piped through stdin.
|
||||
"""
|
||||
import contextlib
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
|
||||
def refuse(msg: str) -> None:
|
||||
"""Exit 2 with a REFUSED line — the code argparse uses for unusable input."""
|
||||
print(f"REFUSED: {msg}", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
def load_tokenizer(path: str):
|
||||
try:
|
||||
from tokenizers import Tokenizer
|
||||
except ImportError as exc: # the corpus tools carry their own venv
|
||||
refuse(f"tokenizers is not importable by {sys.executable} — run the "
|
||||
f"corpus venv's python (on dankserver: "
|
||||
f"/home/dank/venvs/radcorpus/bin/python) or set "
|
||||
f"FRANK_QWEN38_TOKENIZER: {exc}")
|
||||
if not os.path.isfile(path):
|
||||
refuse(f"no tokenizer at {path}")
|
||||
return Tokenizer.from_file(path)
|
||||
|
||||
|
||||
def n_tokens(tok, text: str) -> int:
|
||||
"""The corpus's own count: no special tokens added, prompt side only."""
|
||||
return len(tok.encode(text, add_special_tokens=False).ids)
|
||||
|
||||
|
||||
def iter_jsonl(path: str):
|
||||
"""Yield `(lineno, record)` for every non-blank line; refuse on a bad line.
|
||||
|
||||
Field validation stays with the caller: the three corpora carry different
|
||||
fields (`prompt` alone, `prompt`+`cls`, `prompt`+`score_from`+`source`), and
|
||||
a record missing the one the tool needs is that tool's refusal to make.
|
||||
"""
|
||||
with open(path, encoding="utf-8", errors="replace") as fh:
|
||||
for lineno, line in enumerate(fh, 1):
|
||||
if not line.strip():
|
||||
continue
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
except json.JSONDecodeError as exc:
|
||||
refuse(f"{path}:{lineno} is not JSON: {exc}")
|
||||
yield lineno, rec
|
||||
|
||||
|
||||
def write_jsonl(path: str, rows) -> None:
|
||||
with open(path, "w", encoding="utf-8") as fh:
|
||||
for row in rows:
|
||||
fh.write(json.dumps(row, ensure_ascii=False) + "\n")
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def atomic_writer(path: str):
|
||||
"""Write `path` through a temporary sibling and rename on success.
|
||||
|
||||
An interrupted run then leaves no half corpus behind, and the temporary name
|
||||
keeps the one-artifact-one-name law the driver applies to a score report.
|
||||
"""
|
||||
d = os.path.dirname(os.path.abspath(path)) or "."
|
||||
fd, tmp = tempfile.mkstemp(dir=d, prefix="." + os.path.basename(path) + "-")
|
||||
try:
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as fh:
|
||||
yield fh
|
||||
os.replace(tmp, path)
|
||||
except BaseException:
|
||||
if os.path.exists(tmp):
|
||||
os.unlink(tmp)
|
||||
raise
|
||||
os.chmod(path, 0o644)
|
||||
|
||||
|
||||
def write_json(path: str, obj) -> None:
|
||||
with atomic_writer(path) as fh:
|
||||
json.dump(obj, fh, indent=1)
|
||||
fh.write("\n")
|
||||
Executable
+192
@@ -0,0 +1,192 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Re-chunk a KLD corpus so every document fits the container that scores it.
|
||||
|
||||
A KLD reference is recorded by booting one container and prefilling every corpus
|
||||
document through it. radiance refuses a document the KV pool cannot hold on its
|
||||
own (core/mem/kv.cpp `can_ever_fit`: a paged group holds `n_blocks *
|
||||
block_size` tokens), and the number that matters is printed by the boot ledger,
|
||||
not by a flag:
|
||||
|
||||
W kv group 'kv_attn' holds 7436 tokens but the context limit is 32768, so
|
||||
a prompt longer than 7436 tokens is refused (400) ... or lower
|
||||
--max-model-len to 7436
|
||||
|
||||
The BF16 qwen4exp container at the campaign's shape gets 270.00 MiB of cache --
|
||||
the expert plane owns both cards -- so its real window is 7,436 tokens however
|
||||
`--max-model-len` is spelled. The corpus the campaign inherited from the swift15
|
||||
era is 197 documents of 8,193-11,970 tokens, because make-calib-corpus.py's
|
||||
`--kld-min-prompt 8192` sets a floor and no ceiling: every one of them is
|
||||
unscoreable there, and three drained windows have proved it.
|
||||
|
||||
This script moves the document boundaries and nothing else. Each record's token
|
||||
ids -- counted with the same tokenizer the generator used -- are cut into
|
||||
windows of at most `--max-doc-tokens`, so a record's windows concatenate back to
|
||||
that record's own token sequence, unchanged: no token dropped, rewritten or
|
||||
reworded. Each window gets its own `score_from = len - --score-tail`, the same
|
||||
relation the generator wrote for the parent, so the reference scores 256
|
||||
positions per window instead of per document: 100,864 measured positions over
|
||||
the same text against the 50,432 the inherited corpus scored, at a shorter
|
||||
context. The calibration corpus is untouched, so the quantization itself does
|
||||
not move.
|
||||
|
||||
Usage:
|
||||
rechunk-kld-corpus.py --corpus kld.jsonl --out kld.jsonl.new \
|
||||
[--tokenizer-json PATH] [--max-doc-tokens 7168] [--score-tail 256] [--force]
|
||||
|
||||
--out must not exist and be non-empty without --force: the same
|
||||
one-artifact-one-name law run-qwen38-bf16.sh applies to a score report. The file
|
||||
is written through a temporary name and renamed, so an interrupted run leaves no
|
||||
half corpus behind. A JSON summary of what was written goes to stdout.
|
||||
"""
|
||||
import argparse
|
||||
import dataclasses
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
_HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
|
||||
if _HERE not in sys.path:
|
||||
sys.path.insert(0, _HERE)
|
||||
|
||||
import qwen38_common as qc
|
||||
|
||||
MISSING = object() # "the parent record carried no source field"
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--corpus", required=True,
|
||||
help="the kld.jsonl to re-chunk (records: prompt, score_from, source)")
|
||||
ap.add_argument("--out", required=True,
|
||||
help="file to write; refuse if it exists and is non-empty without --force")
|
||||
ap.add_argument("--tokenizer-json",
|
||||
default=os.environ.get("FRANK_QWEN38_TOKENIZER",
|
||||
"/home/dank/models/Qwen3.8-Flash-Next/tokenizer.json"),
|
||||
help="the tokenizer the corpus was authored with (the same one the "
|
||||
"manifest in the corpus output dir names)")
|
||||
ap.add_argument("--max-doc-tokens", type=int, default=7168,
|
||||
help="the widest document the scoring container can hold; 7,168 is "
|
||||
"7x1024, the largest whole kilopage under the 7,436 the bf16 "
|
||||
"container's ledger reports")
|
||||
ap.add_argument("--score-tail", type=int, default=256,
|
||||
help="positions scored at the end of every document, as the generator "
|
||||
"recorded it")
|
||||
ap.add_argument("--force", action="store_true",
|
||||
help="overwrite a non-empty --out (after moving the old file to trash)")
|
||||
return ap.parse_args()
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Window:
|
||||
"""One scoring document cut out of a parent record."""
|
||||
prompt: str
|
||||
score_from: int
|
||||
n_tokens: int
|
||||
clamped: bool
|
||||
source: object = MISSING
|
||||
|
||||
def record(self) -> dict:
|
||||
rec = {"prompt": self.prompt, "score_from": self.score_from}
|
||||
if self.source is not MISSING:
|
||||
rec["source"] = self.source
|
||||
return rec
|
||||
|
||||
|
||||
def validate(a: argparse.Namespace) -> None:
|
||||
"""Everything that makes this run unusable, named before a token is cut."""
|
||||
if a.max_doc_tokens <= 0:
|
||||
qc.refuse(f"--max-doc-tokens must be positive (got {a.max_doc_tokens})")
|
||||
if a.score_tail <= 0:
|
||||
qc.refuse(f"--score-tail must be positive (got {a.score_tail})")
|
||||
if a.max_doc_tokens <= a.score_tail:
|
||||
qc.refuse(f"--max-doc-tokens {a.max_doc_tokens} cannot hold a "
|
||||
f"{a.score_tail}-position score tail: every window would be all "
|
||||
"tail and no context")
|
||||
if not os.path.isfile(a.corpus):
|
||||
qc.refuse(f"no corpus at {a.corpus}")
|
||||
if os.path.realpath(a.out) == os.path.realpath(a.corpus):
|
||||
qc.refuse(f"--out {a.out} is the corpus itself; move it to trash first — the "
|
||||
"trash law means the old segmentation stays recoverable")
|
||||
if os.path.exists(a.out) and os.path.getsize(a.out) > 0 and not a.force:
|
||||
qc.refuse(f"{a.out} exists and is not empty; pass --force after moving it to trash")
|
||||
|
||||
|
||||
def cut(rec: dict, tok, a: argparse.Namespace,
|
||||
where: str) -> tuple[list[Window], int]:
|
||||
"""Cut one record's token ids into windows, proving each one round-trips.
|
||||
|
||||
The round-trip encode is the whole reason this tool may claim the reference
|
||||
scores the corpus's own token sequence: decoding a window to text and
|
||||
encoding it back must return the identical ids, or the score is of a
|
||||
different document than the one on disk.
|
||||
"""
|
||||
ids = tok.encode(rec["prompt"], add_special_tokens=False).ids
|
||||
windows: list[Window] = []
|
||||
for start in range(0, max(len(ids), 1), a.max_doc_tokens):
|
||||
window = ids[start:start + a.max_doc_tokens]
|
||||
if not window:
|
||||
continue
|
||||
text = tok.decode(window)
|
||||
if tok.encode(text, add_special_tokens=False).ids != window:
|
||||
qc.refuse(f"{where} window at token {start} does not round-trip through "
|
||||
"text — the reference would score a different token sequence "
|
||||
"than the corpus holds")
|
||||
windows.append(Window(prompt=text,
|
||||
score_from=max(len(window) - a.score_tail, 0),
|
||||
n_tokens=len(window),
|
||||
clamped=len(window) < a.score_tail,
|
||||
source=rec["source"] if "source" in rec else MISSING))
|
||||
return windows, len(ids)
|
||||
|
||||
|
||||
def rechunk(a: argparse.Namespace, tok) -> dict:
|
||||
"""Write the re-chunked corpus and return the summary it printed."""
|
||||
docs_in = docs_out = tokens_in = tokens_out = 0
|
||||
longest_child = clamped = scored = 0
|
||||
sources: dict[str, int] = {}
|
||||
with qc.atomic_writer(a.out) as out:
|
||||
for lineno, rec in qc.iter_jsonl(a.corpus):
|
||||
where = f"{a.corpus}:{lineno}"
|
||||
if not isinstance(rec.get("prompt"), str):
|
||||
qc.refuse(f"{where} has no string 'prompt' field")
|
||||
windows, n_parent = cut(rec, tok, a, where)
|
||||
docs_in += 1
|
||||
tokens_in += n_parent
|
||||
for w in windows:
|
||||
out.write(json.dumps(w.record(), ensure_ascii=False) + "\n")
|
||||
docs_out += 1
|
||||
tokens_out += w.n_tokens
|
||||
longest_child = max(longest_child, w.n_tokens)
|
||||
clamped += 1 if w.clamped else 0
|
||||
scored += w.n_tokens - w.score_from
|
||||
if "source" in rec:
|
||||
for cls in str(rec["source"]).split("+"):
|
||||
sources[cls] = sources.get(cls, 0) + 1
|
||||
if tokens_out != tokens_in:
|
||||
raise SystemExit(f"REFUSED: the windows hold {tokens_out} tokens and the corpus "
|
||||
f"has {tokens_in} — re-chunking lost or invented tokens")
|
||||
return {
|
||||
"corpus": a.corpus,
|
||||
"out": a.out,
|
||||
"tokenizer": os.path.basename(a.tokenizer_json),
|
||||
"documents_in": docs_in,
|
||||
"documents_out": docs_out,
|
||||
"tokens_in": tokens_in,
|
||||
"tokens_out": tokens_out,
|
||||
"longest_document": longest_child,
|
||||
"max_doc_tokens": a.max_doc_tokens,
|
||||
"score_tail": a.score_tail,
|
||||
"scored_positions": scored,
|
||||
"clamped_score_from": clamped,
|
||||
"source_documents": sources,
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
a = parse_args()
|
||||
validate(a)
|
||||
print(json.dumps(rechunk(a, qc.load_tokenizer(a.tokenizer_json)), indent=1))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,80 @@
|
||||
# qwen4exp-qwen38-w5nl64-bf16trunk.recipe — every routed expert, BOTH projections, at five bits:
|
||||
# u5 codes into libquant's 32-level w5nl table, an E4M3 scale a group of 64 under the fixed 2^-13
|
||||
# second level, rotated by the 128-wide Hadamard, chosen by GPTQ against $CALIB and swept three
|
||||
# times by coordinate descent. This is `w5nl64a8h` in the engine's served-encoding table — the top of
|
||||
# the four-bit class — applied wider than any published radiance container of this checkpoint: the
|
||||
# author's own w5 rung (`qwen4exp-w5dn64-i8-hc8m.recipe`) moves only `down`, because that rung was
|
||||
# priced as a byte-for-error trade against prefill; this campaign buys the whole rung and pays for
|
||||
# it out of the pinned host pool (+15.3 GiB of expert plane, `--host-pool-mib 61440` per rank), not
|
||||
# out of the KV pool.
|
||||
#
|
||||
# Every other line is the served container's recipe (dump it with
|
||||
# `rad-info --recipe /home/dank/models/rad/qwen3.8-next-flash-fp8-iq4r-moe.rad`), minus the rows this
|
||||
# campaign leaves at the checkpoint:
|
||||
#
|
||||
# * the trunk's block linears, the shared expert and the hyper-connection mixers are NOT listed —
|
||||
# dropping their i8/fp8 lines leaves them bf16, which is what the checkpoint holds. Measured bill
|
||||
# +1.978 GiB of always-resident weight = ~131k tokens of KV off the pool.
|
||||
# * `ssm_inz`/`ssm_out` STAY int8: the delta-net gates are the largest trunk rows (+1.9 GiB, 126k
|
||||
# tokens) and the least sensitive per byte.
|
||||
# * `mtp.draft_head` STAYS 2-bit and the PLE n-gram table STAYS one fixed E4M3 scale: the draft
|
||||
# head's rung is cheap and its acceptance cost is a decode question this campaign does not touch,
|
||||
# and bf16 for the n-gram table would pin +47.69 GiB of host RAM for a lookup.
|
||||
#
|
||||
# Protected experts: none in this file. `scripts/apply-protected-cast.sh` appends the derived set
|
||||
# (48, two per layer so each rank's 256-expert half stays even — the four-bit GEMM takes the rest of
|
||||
# a half as two equal tables by parity) from the fresh 12M-token energy/routing census, and they must
|
||||
# sit ABOVE the expert row because the first rule that matches a weight decides it.
|
||||
#
|
||||
# CALIB=/home/dank/campaign/qwen38-w5nl64/calib \
|
||||
# rad-convert /home/dank/models/Qwen3.8-Flash-Next \
|
||||
# --recipe this.file --imatrix .../imatrix.gguf -o out.rad
|
||||
#
|
||||
# The arm driver (`scripts/convert-qwen38-w5nl64.sh`) generates the per-arm variants of the expert
|
||||
# row from this template — cd=3 (control), cd=6, and `act_order=1` with NO cd= (libquant refuses the
|
||||
# pair, lq_registry.cpp:105) — rather than keeping three near-duplicate recipes that can drift apart.
|
||||
#
|
||||
# The whole campaign, in the order that fits on one disk (every step runs on dankserver):
|
||||
#
|
||||
# 1. scripts/download-qwen38-bf16.sh --fetch the vendor BF16 checkpoint, 335.3 GiB
|
||||
# 2. scripts/build-qwen38-bf16-rad.sh the BF16 reference container, another 335.3 GiB
|
||||
# 3. one drained window, two boots:
|
||||
# scripts/run-qwen38-bf16.sh kldref the KLD reference recording (the zero of scale)
|
||||
# scripts/run-qwen38-bf16.sh calib the 12M-position calibration tap -> grams
|
||||
# 4. trash qwen38-bf16.rad the peak-disk law: the converts need its room
|
||||
# 5. scripts/grams-to-imatrix.py grams -> imatrix.gguf in the campaign dir
|
||||
# 6. scripts/convert-qwen38-w5nl64.sh the arms, planned and billed before each one runs
|
||||
# 7. fetch the published FP8-native release, then per container:
|
||||
# scripts/run-qwen38-bf16.sh score RAD OUT fp8 anchor, served iq4r, each arm
|
||||
# then scripts/kld-gate-qwen38.sh fp8.json iq4r.json cand.json
|
||||
# 8. a winner is a trio edit plus `scripts/deploy.sh dankserver --service frank-archon`; the
|
||||
# rollback is the commit before it. If the win needs the bigger pinned plane, raise
|
||||
# FRANK_ARCHON_HOST_POOL_MIB and FRANK_ARCHON_TTM_TARGET_MIB in the SAME commit: the preflight
|
||||
# floor is 2xHOST_POOL + 48,824 + prefix cache + MNBT + the store, and a pool raised without
|
||||
# the TTM target is a boot refusal that Restart=always repeats forever, invisible to deploy.sh.
|
||||
|
||||
# PROTECTED-BEGIN — scripts/apply-protected-cast.sh splices the derived cast lines in here.
|
||||
# They MUST sit above the expert rule: the first rule whose glob matches a weight decides it, so a
|
||||
# cast line below the five-bit rule would never win. The set is derived from the campaign's own
|
||||
# 12M-token recording (`derive-protected-experts.py --target-count 48` over the fresh grams), and the
|
||||
# driver splices it into a per-arm COPY so this template stays the clean, reviewable baseline.
|
||||
# PROTECTED-END
|
||||
|
||||
# Every routed expert, trunk and MTP layer alike, both projections, at five bits. The experts' scales
|
||||
# sit between 2^-15 and 2^-10.6 under w4nl's top level of 22 and half a binade lower under w5nl's 30
|
||||
# — inside E4M3's normal range under 2^-13 either way. The MTP layer has no recording and is rounded
|
||||
# to nearest, which libquant says on stderr; it is one layer of 49 and only ever changes drafts.
|
||||
blk.*.ffn_*_exps.*.weight gptq table=w5nl group=64 scale=fp8_e4m3 scale2=f32 block2=*x* scale2_value=0.0001220703125 transform=fwht128 rule=search cd=3 calib=$CALIB
|
||||
|
||||
# The delta-net's in/out gates, int8 with a bf16 scale a group of 128 searched for least error — the
|
||||
# served recipe's lines, unchanged.
|
||||
blk.*.ssm_inz.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
|
||||
blk.*.ssm_out.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
|
||||
|
||||
# The MTP head's two mixing rows, E4M3 as served. They only ever change the drafts.
|
||||
mtp.fc_hidden.weight rtn codes=fp8_e4m3 group=128 scale=f32
|
||||
mtp.fc_embedding.weight rtn codes=fp8_e4m3 group=128 scale=f32
|
||||
|
||||
# The n-gram table at one fixed E4M3 scale, and the 2-bit draft head — both the served lines.
|
||||
blk.*.ple_ngram.weight rtn codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
|
||||
mtp.draft_head.weight rtn codes=u2 zero=u8 group=128 scale=f16
|
||||
Executable
+485
@@ -0,0 +1,485 @@
|
||||
#!/usr/bin/env bash
|
||||
# run-qwen38-bf16.sh — RUNS ON DANKSERVER. Boot a container over the campaign's
|
||||
# ONE measuring shape for the phases that need both R9700s. The engine owns the
|
||||
# cards, so frank-archon is stopped first and restored on every exit path — the
|
||||
# autoresearch pattern: stop, never mask (the unit file in /etc/systemd/system
|
||||
# is a symlink into the checkout and mask fails on a relinked unit; the deploy
|
||||
# restores it).
|
||||
#
|
||||
# kldref --kld-record the reference log-probabilities: the engine
|
||||
# runs /home/dank/corpora/out/kld.jsonl (197 prompts, 50,432
|
||||
# scored positions, score tail 256) through prefill, writes
|
||||
# /home/dank/campaign/qwen38-w5nl64/kld-ref, and EXITS. The
|
||||
# model behind it is the BF16 container from
|
||||
# scripts/build-qwen38-bf16-rad.sh — the vendor checkpoint's
|
||||
# own numbers, which is what makes it the zero point.
|
||||
# score RAD OUT --kld-ref score ONE container against that recording and
|
||||
# EXIT: the FP8-native anchor, the served iq4r, and every
|
||||
# arm. Deliberately the SAME engine pin, the same TP, the
|
||||
# same step size, the same fp8 KV, --deterministic and no
|
||||
# speculation as kldref — KL and top-1 agreement are only
|
||||
# comparable across boots when everything except the weights
|
||||
# is identical, so this mode is one argv with --model and
|
||||
# the two --kld-* flags changed, never a second shape.
|
||||
# RAD must sit under the models dir (the container reads it
|
||||
# through the ro /models mount); OUT lands on a writable
|
||||
# mount because --kld-out F also writes F.rows beside it.
|
||||
# calib serve on :7770 with the calibration tap open
|
||||
# (RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS=12000000). A
|
||||
# separate driver — scripts/calib-push.py --port 7770
|
||||
# --container qwen38-bf16 --corpus
|
||||
# /home/dank/corpora/out/calib-7168.jsonl — pushes the corpus;
|
||||
# at the floor the engine writes the Grams and the tap switches
|
||||
# itself off (calib-push prints TAP_DONE). Stop this script
|
||||
# after that; the trap brings frank-archon back. THE FLOOR IS
|
||||
# THE CAMPAIGN'S RULE (12 M tokens, Frank 2026-10-30): the
|
||||
# served iq4r this arm competes with was calibrated on a 10 M-
|
||||
# token set, so the arm is calibrated at least as deeply. One
|
||||
# round of the re-chunked corpus is ~9.31 M tokens, so sealing
|
||||
# takes round 1 plus part of round 2 — a round that seals
|
||||
# nothing is NORMAL (the floor is pooled), and the pusher's
|
||||
# round_verdict treats a flat round as "continue", never as a
|
||||
# failure. One seal writes ~40 GiB.
|
||||
#
|
||||
# Shape: the vendor's bf16 calibration lines — expert_tiered placement with the
|
||||
# weights disk tier, deterministic (the resident set pinned to the loader's plan,
|
||||
# so a recorded reference and a scored candidate see the same one), no
|
||||
# speculation, fp8 KV, a 32,768-token window, the PLE n-gram table pinned to RAM,
|
||||
# and the production chat template rather than the one baked into the container —
|
||||
# every scored container is asked the same tokenised questions the reference
|
||||
# answered. The window the flag spells is 32,768; the pool this payload's boot
|
||||
# actually gets is 7,436 tokens, so kld.jsonl is re-chunked by
|
||||
# scripts/rechunk-kld-corpus.py into documents of at most 7,168 tokens — the
|
||||
# widest one the reference can hold. The kldref gate below re-checks the file
|
||||
# against the declared window, and the re-chunker is what keeps that honest.
|
||||
#
|
||||
# NO --vram-kv-mib: it was REMOVED at 1.3.0 and naming it is a boot refusal, not
|
||||
# a no-op — verified on the pinned image 2026-10-27: "E --vram-kv-mib was
|
||||
# removed: the card is split between the weights and the cache from what the
|
||||
# model needs." The resolver derives the split from --host-pool-mib,
|
||||
# --gpu-headroom-mib and the activation arena; the rig inherits the first two from
|
||||
# frank-archon.env, the file that owns them. On THIS payload the host pool is not
|
||||
# the cache's supplier: the boot lands in vram_budget.cpp's `no lender` branch,
|
||||
# where the experts keep their plan-fixed 25.29 GiB and the cache gets the residue.
|
||||
# Measured twice — 12,288 and 49,152 MiB of host pool both printed the same
|
||||
# "kv cache 271.25 MiB ... 7,436"; only --gpu-headroom-mib moved anything
|
||||
# (768 -> 96 grew vram_weights 29.41 -> 30.06 GiB). The supplier the source names
|
||||
# is the arena: vram_budget.cpp:117, "lower --max-num-batched-tokens (the arena
|
||||
# scales with it)" — which is what the step size below now does.
|
||||
#
|
||||
# The pool, not the window, decides what the recorder accepts: it refuses any
|
||||
# document longer than the pool, so the 32,768-token window was decorative while the
|
||||
# cache held 7,436 — "E kld: document 196 is 8507 tokens and the KV pool holds
|
||||
# 7436, so it can never be scored" on all 197 records, exit 1. The longest record
|
||||
# is 10,554 tokens by the engine's own count (chars/3.6 for this file, not the
|
||||
# chars/4.97 this repo measured on the K3 corpus).
|
||||
#
|
||||
# The step size is ONE value for the recording and every scoring pass, and it is
|
||||
# the value frank-archon.env owns (FRANK_ARCHON_MAX_BATCHED_TOKENS, 2,048): the
|
||||
# quantised expert FILE tiers (the served iq4r, the arms) stream only at
|
||||
# MNBT<=2048 and refuse the widened form above it, and the arena a step costs the
|
||||
# card comes straight out of the cache — 1.08 GiB at a 2,048-token step against
|
||||
# 504.39 MiB at 1,024. The corpora are cut to the window the boot ledger reports,
|
||||
# which is what the pool raise is for (frank-archon.env: HOST_POOL_MIB); a narrow
|
||||
# step was never the fix, it was the symptom's disguise.
|
||||
#
|
||||
# usage: scripts/run-qwen38-bf16.sh kldref|calib|score RAD OUT [--dry-run] [--image REF]
|
||||
# (RAD and OUT are score's two positionals: RAD is a container under the models
|
||||
# dir, OUT the report path whose directory is created if missing)
|
||||
# --dry-run print the exact `podman run` argv, one shell-quoted argument a
|
||||
# line, and exit 0: the service is NOT stopped, nothing is
|
||||
# mounted, nothing is written — the offline test seam. It still
|
||||
# validates the argv's own shape (paths through the /models mount,
|
||||
# a step size that is a number), because an argv that could not
|
||||
# boot is not worth printing.
|
||||
# --image REF override the digest pin (a probe boot, or the test seam)
|
||||
#
|
||||
# knobs: FRANK_QWEN38_MODELS (the dir mounted at /models), FRANK_QWEN38_RAD (the
|
||||
# bf16 container, kldref only), FRANK_QWEN38_CAMPAIGN (holds kld-ref/ and
|
||||
# calib/), FRANK_QWEN38_KLD_CORPUS, FRANK_QWEN38_TPL (the template copy
|
||||
# the container reads through /models), FRANK_QWEN38_LOG_DIR, and the step
|
||||
# and slot counts inherited as FRANK_ARCHON_MAX_BATCHED_TOKENS /
|
||||
# FRANK_ARCHON_MAX_SEQS — no per-mode literals (the audit law: a mode that
|
||||
# carries its own step measures a shape nothing serves; calib is exempt, see
|
||||
# Q38_CALIB_MNBT / Q38_CALIB_SEQS below — its output is never compared) —
|
||||
# and FRANK_QWEN38_MAX_MODEL_LEN — the window every mode boots at (32,768;
|
||||
# see the corpus census above, and raise it together with the corpus).
|
||||
# kldref refuses (exit 2) when the longest record of the corpus it is about to
|
||||
# read is longer than MAX_MODEL_LEN x 3 characters — chars/3 being the
|
||||
# pessimistic token:char ratio, so the script says no before the engine finds out
|
||||
# inside a drained window, after the checkpoint work is already done. score
|
||||
# refuses when the reference dir is absent, when RAD is not on disk or not under
|
||||
# the models mount, or when OUT already exists: the trash law means a second
|
||||
# score over the same name is a deliberate act, and two numbers in one file is
|
||||
# how a campaign loses its own history.
|
||||
# Engine stdout goes to $LOG_DIR/qwen38_bf16_<mode>.log — the boot ledger is the
|
||||
# record of what the resolver actually gave the cards; the journal never sees it.
|
||||
# exit: the engine's code; 2 = refused before the service was touched
|
||||
set -uo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "$SCRIPT_DIR/frank-archon.env"
|
||||
IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
|
||||
# THE CALIBRATION PIN: the LATEST radiance, pinned by digest because a tag is a
|
||||
# moving target and a tap must name the producer that wrote it. `:latest` on
|
||||
# 2026-10-30 resolves to v1.14.0 (revision 4a61d7296bc5), and on that image a
|
||||
# calib boot prints the arming line — "recorded passes: off, the program
|
||||
# accumulates calibration Grams (gram_accum)" — where the production pin
|
||||
# (1.8.1) prints nothing of the kind: a 17-hour run on 1.8.1 wrote ZERO bytes
|
||||
# into the tap dir while /health, /stats and the push all looked healthy
|
||||
# (measured 2026-10-30). The tap code is identical in both trees' sources, so
|
||||
# the shipped 1.8.1 build is what drops it; do not spend another window on it.
|
||||
# Any other image must be named deliberately with --image.
|
||||
CALIB_IMAGE="${Q38_CALIB_IMAGE:-stilldeadcode/radiance@sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}"
|
||||
IMAGE_EXPLICIT=0
|
||||
|
||||
MODE=""
|
||||
POS=()
|
||||
DRY=0
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
kldref|calib|score) MODE="$1" ;;
|
||||
--dry-run) DRY=1 ;;
|
||||
--image) [ "$#" -ge 2 ] || { echo "--image needs a reference" >&2; exit 2; }
|
||||
IMAGE="$2"; IMAGE_EXPLICIT=1; shift ;;
|
||||
*)
|
||||
# score's two positional arguments, and nothing else may be bare.
|
||||
if [ "$MODE" = score ] && [ "${#POS[@]}" -lt 2 ]; then
|
||||
POS+=("$1")
|
||||
else
|
||||
echo "usage: ${0##*/} kldref|calib|score RAD OUT [--dry-run] [--image REF] (unknown argument: $1)" >&2
|
||||
exit 2
|
||||
fi ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
if [ -z "$MODE" ]; then
|
||||
echo "usage: ${0##*/} kldref|calib|score RAD OUT [--dry-run] [--image REF] (no mode given)" >&2
|
||||
exit 2
|
||||
fi
|
||||
if [ "$MODE" = score ] && [ "${#POS[@]}" -ne 2 ]; then
|
||||
echo "usage: ${0##*/} score RAD OUT — RAD is the container to score, OUT the report to write (got ${#POS[@]} argument(s))" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
|
||||
CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
|
||||
KLD_CORPUS="${FRANK_QWEN38_KLD_CORPUS:-/home/dank/corpora/out/kld.jsonl}"
|
||||
RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
|
||||
TPL="${FRANK_QWEN38_TPL:-$MODELS/tpl.jinja}"
|
||||
TEMPLATE="$SCRIPT_DIR/qwen38-froggeric-v22.5.jinja"
|
||||
LOG_DIR="${FRANK_QWEN38_LOG_DIR:-/home/dank/logs}"
|
||||
LOG="$LOG_DIR/qwen38_bf16_${MODE}.log"
|
||||
KLD_REF_DIR="$CAMPAIGN/kld-ref"
|
||||
CALIB_DIR="${FRANK_QWEN38_CALIB_DIR:-$CAMPAIGN/calib}"
|
||||
C=qwen38-bf16
|
||||
SERVICE=frank-archon
|
||||
# The card split, taken from the file that owns it instead of a private pair:
|
||||
# --host-pool-mib also funds the KV cache (see the header), so hardcoding it
|
||||
# here once starved the recorder. The `:?` guards keep an empty env fatal.
|
||||
HOST_POOL_MIB="${FRANK_ARCHON_HOST_POOL_MIB:?FRANK_ARCHON_HOST_POOL_MIB must be set}"
|
||||
GPU_HEADROOM_MIB="${FRANK_ARCHON_GPU_HEADROOM_MIB:?FRANK_ARCHON_GPU_HEADROOM_MIB must be set}"
|
||||
# The slot count, from the same file that owns the card split: production runs
|
||||
# four slots and every mode measures at production's geometry. A per-mode
|
||||
# literal here is what let calib ask for eight slots and then starve its own
|
||||
# cache. The `:?` guard keeps an empty env fatal, as for the card split.
|
||||
MAX_SEQS="${FRANK_ARCHON_MAX_SEQS:?FRANK_ARCHON_MAX_SEQS must be set}"
|
||||
if ! [[ "$MAX_SEQS" =~ ^[0-9]+$ ]]; then
|
||||
echo "REFUSED: FRANK_ARCHON_MAX_SEQS must be a slot count (got: $MAX_SEQS)" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# ONE step size for every mode, quoted from the file that owns it — the audit
|
||||
# law for a radiance rig is to inherit the proven number, not to carry a
|
||||
# campaign-private value: production's FRANK_ARCHON_MAX_BATCHED_TOKENS. The
|
||||
# 1,024 default this script carried was PR #80's answer to a starved cache;
|
||||
# production moved to its widest legal step (2,048) on 2026-10-29 and the rig
|
||||
# follows. The pool raise that shipped with it (66,560 a rank) bought this
|
||||
# container 1.26 MiB of cache (`vram_kv 269.99 -> 271.25 MiB`, window still
|
||||
# 7,436 tokens — the 95.37 GiB n-gram table's claim decides the rig's cache,
|
||||
# not the pool) and was folded back the same day. The corpora stay cut to the
|
||||
# window the boot ledger reports, never the other way round.
|
||||
MNBT="${FRANK_ARCHON_MAX_BATCHED_TOKENS:?FRANK_ARCHON_MAX_BATCHED_TOKENS must be set}"
|
||||
if ! [[ "$MNBT" =~ ^[0-9]+$ ]]; then
|
||||
echo "REFUSED: FRANK_ARCHON_MAX_BATCHED_TOKENS must be a token count (got: $MNBT)" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Calib's shape is NOT a measuring shape. kldref and score produce KL numbers that are only
|
||||
# comparable across boots at an identical shape, which is why they inherit production's step and
|
||||
# slots to the letter. The tap's output is layer Hessians, never a number any report compares, so
|
||||
# its step and slot count are its own — and they are the tap's dominant cost: a 7,168-token
|
||||
# document cut into four 2,048-token steps re-streams each layer's routed expert union four
|
||||
# times from the container, so one wide step is worth up to ~4x the read volume. Measured
|
||||
# 2026-10-29 at the inherited 2,048/4: prefill 71 tok/s and a 47-hour tap; the probe exists to
|
||||
# find out what the wide step is worth before the campaign pays for two days of drained serving.
|
||||
# Both default to the inherited value, so nothing moves unless a probe asks it to.
|
||||
CALIB_MNBT="${Q38_CALIB_MNBT:-$MNBT}"
|
||||
CALIB_SEQS="${Q38_CALIB_SEQS:-$MAX_SEQS}"
|
||||
for pair in "Q38_CALIB_MNBT=$CALIB_MNBT" "Q38_CALIB_SEQS=$CALIB_SEQS"; do
|
||||
if ! [[ "${pair#*=}" =~ ^[0-9]+$ ]]; then
|
||||
echo "REFUSED: ${pair%%=*} must be a number (got: ${pair#*=})" >&2
|
||||
exit 2
|
||||
fi
|
||||
done
|
||||
|
||||
# The window every mode boots at. Over-length is a REFUSAL, not a truncation (the
|
||||
# env file says so for production), so this is the number the corpus must fit in.
|
||||
MAX_MODEL_LEN="${FRANK_QWEN38_MAX_MODEL_LEN:-32768}"
|
||||
if ! [[ "$MAX_MODEL_LEN" =~ ^[0-9]+$ ]]; then
|
||||
echo "REFUSED: FRANK_QWEN38_MAX_MODEL_LEN must be a token count (got: $MAX_MODEL_LEN)" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Where the n-gram table lives. On the bf16 payload it is 95.37 GiB and `ram`
|
||||
# PINS it: the calibration boot died at "startup stopped: out of memory" with
|
||||
# 230.22 GiB pinned of 246.07 GiB and 8.06 GiB available (measured 2026-10-29),
|
||||
# while the reference recording and every score boot had already succeeded with
|
||||
# the same pool and slots. The difference that matters is the job: calib only
|
||||
# needs the routed weights prefilled, so it reads the table from the container's
|
||||
# own file tier and the pin never happens; kldref and score keep production's
|
||||
# placement, because their numbers are only comparable when every other term of
|
||||
# the shape is identical. FRANK_QWEN38_NGRAM_PLACEMENT overrides either default.
|
||||
case "$MODE" in
|
||||
calib) NGRAM_PLACEMENT="${FRANK_QWEN38_NGRAM_PLACEMENT:-disk}" ;;
|
||||
*) NGRAM_PLACEMENT="${FRANK_QWEN38_NGRAM_PLACEMENT:-ram}" ;;
|
||||
esac
|
||||
case "$NGRAM_PLACEMENT" in
|
||||
auto|vram|ram|disk) ;;
|
||||
*) echo "REFUSED: FRANK_QWEN38_NGRAM_PLACEMENT must be auto|vram|ram|disk (got: $NGRAM_PLACEMENT)" >&2
|
||||
exit 2 ;;
|
||||
esac
|
||||
|
||||
# Every container this campaign boots is read through the ro /models mount, so
|
||||
# the host path must live under it; the container-side spelling is what goes
|
||||
# into the argv. Same rule for the container a score pass opens.
|
||||
under_models() {
|
||||
case "$1" in
|
||||
"$MODELS"/*) printf '/models/%s' "${1#"$MODELS"/}" ;;
|
||||
*) return 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# same_dir A B — the two name one directory, judged only when BOTH exist: a dry
|
||||
# run has no fixtures on disk, and two absent paths are not the same place.
|
||||
same_dir() {
|
||||
[ -d "$1" ] && [ -d "$2" ] && [ "$(cd "$1" && pwd)" = "$(cd "$2" && pwd)" ]
|
||||
}
|
||||
|
||||
if [ "$MODE" = score ]; then
|
||||
SCORE_RAD="${POS[0]}"
|
||||
SCORE_OUT="${POS[1]}"
|
||||
OUT_DIR="$(dirname "$SCORE_OUT")"
|
||||
MODEL_IN="$(under_models "$SCORE_RAD")" || {
|
||||
echo "REFUSED: container to score $SCORE_RAD is not under the mounted models dir $MODELS — the engine cannot read it" >&2
|
||||
exit 2
|
||||
}
|
||||
else
|
||||
MODEL_IN="$(under_models "$RAD")" || {
|
||||
echo "REFUSED: reference container $RAD is not under the mounted models dir $MODELS — the engine cannot read it" >&2
|
||||
exit 2
|
||||
}
|
||||
fi
|
||||
|
||||
# The measuring shape. Identical for kldref and score by construction — this
|
||||
# array is the only place the campaign's KL numbers are defined.
|
||||
ARGS=(--tp 2 --placement expert_tiered --weights-disk-tier
|
||||
--host-pool-mib "$HOST_POOL_MIB" --gpu-headroom-mib "$GPU_HEADROOM_MIB"
|
||||
--deterministic --num-speculative-tokens 0
|
||||
--kv-cache-dtype fp8 --max-model-len "$MAX_MODEL_LEN"
|
||||
--ngram-placement "$NGRAM_PLACEMENT"
|
||||
--override-chat-template /models/tpl.jinja)
|
||||
MOUNTS=(-v "$MODELS:/models:ro")
|
||||
TAPENV=()
|
||||
case "$MODE" in
|
||||
kldref)
|
||||
# Recording is prefill-only over four slots at the campaign's single step
|
||||
# size, which is frank-archon.env's FRANK_ARCHON_MAX_BATCHED_TOKENS (2,048) —
|
||||
# see the header: the file tier's ceiling and the slot count both come from
|
||||
# the file that owns them. Corpus read-only, record dir rw.
|
||||
ARGS+=(--model "$MODEL_IN"
|
||||
--kld-record /data/kld-ref --kld-corpus /data/kld.jsonl
|
||||
--max-num-seqs "$MAX_SEQS" --max-num-batched-tokens "$MNBT")
|
||||
MOUNTS+=(-v "$KLD_CORPUS:/data/kld.jsonl:ro"
|
||||
-v "$KLD_REF_DIR:/data/kld-ref")
|
||||
;;
|
||||
score)
|
||||
# The same argv with three words changed: which container, and that this
|
||||
# time the engine reads a recording and writes a report instead of the
|
||||
# reverse. The reference dir mounts READ-ONLY — a scoring pass must not be
|
||||
# able to touch the zero point — and the report gets its own writable bind.
|
||||
if [ "$OUT_DIR" = "$KLD_REF_DIR" ] || same_dir "$OUT_DIR" "$KLD_REF_DIR" \
|
||||
|| [ "$(realpath -m "$OUT_DIR")" = "$(realpath -m "$KLD_REF_DIR")" ]; then
|
||||
echo "REFUSED: score output $SCORE_OUT is inside the reference dir $KLD_REF_DIR, which this mode mounts read-only" >&2
|
||||
exit 2
|
||||
fi
|
||||
ARGS+=(--model "$MODEL_IN"
|
||||
--kld-ref /data/kld-ref --kld-out "/data/out/$(basename "$SCORE_OUT")"
|
||||
--max-num-seqs "$MAX_SEQS" --max-num-batched-tokens "$MNBT")
|
||||
MOUNTS+=(-v "$KLD_REF_DIR:/data/kld-ref:ro"
|
||||
-v "$OUT_DIR:/data/out")
|
||||
;;
|
||||
calib)
|
||||
# Calibration rides the shared step like every other mode. The old
|
||||
# exemption — eight slots at MNBT 65,536 for "free throughput", because
|
||||
# "a disk-tier step costs the same at 1 token as at 64k" — is false: the
|
||||
# step's activation arena is billed on the card, not on the disk. The
|
||||
# engine's own ledger for this container: at MNBT 1,024 the arena is
|
||||
# 504 MiB and the card claims 30.87 GiB (weights 28.90 GiB of it); at
|
||||
# 65,536 the arena is 10.65 GiB (fixed region 1.25 GiB), the claim falls
|
||||
# to 16.85 GiB, the experts shrink to 10.98 GiB and the KV to 271 MiB —
|
||||
# one slot's worth of cache for eight slots, and the boot dies at
|
||||
# "startup stopped: out of memory" (measured 2026-10-29). A wide step
|
||||
# buys throughput with the expert slab, so price it like any other
|
||||
# VRAM knob. The slots follow frank-archon.env's FRANK_ARCHON_MAX_SEQS
|
||||
# through MAX_SEQS, exactly as kldref and score do.
|
||||
[ "$IMAGE_EXPLICIT" = 1 ] || IMAGE="$CALIB_IMAGE"
|
||||
# ONE TAP SEAL WRITES ~40 GiB. Measured 2026-10-30: a 8,192-token floor wrote
|
||||
# 39.4 GiB into the tap dir within seconds; pointed at a 37 GiB root it filled
|
||||
# the filesystem to zero, after which every write came back short ("calibration
|
||||
# cannot write", "write was short") and the system itself began refusing
|
||||
# writes. The room is demanded up front, on the tap's own filesystem.
|
||||
if [ "$DRY" != 1 ]; then
|
||||
avail_mib=$(df --output=avail -B 1M "$CALIB_DIR" 2>/dev/null | tail -1 | tr -d ' ')
|
||||
[[ "$avail_mib" =~ ^[0-9]+$ ]] || avail_mib=0
|
||||
need_mib="${Q38_CALIB_MIN_FREE_MIB:-61440}"
|
||||
if [ "$avail_mib" -lt "$need_mib" ]; then
|
||||
echo "REFUSED: tap dir $CALIB_DIR has ${avail_mib} MiB free; one seal writes ~40 GiB and this run needs ${need_mib} MiB (Q38_CALIB_MIN_FREE_MIB overrides)" >&2
|
||||
exit 2
|
||||
fi
|
||||
fi
|
||||
ARGS+=(--model "$MODEL_IN" --port 7770
|
||||
--max-num-seqs "$CALIB_SEQS" --max-num-batched-tokens "$CALIB_MNBT")
|
||||
# The tap is only real if this directory exists and the engine can write it:
|
||||
# without the check a typo'd CALIB_DIR auto-creates a root-owned empty dir in
|
||||
# whatever path was given, and the run burns hours serving prompts into a tap
|
||||
# nobody reads. RADIANCE_CALIB_TOKENS is the whole corpus' floor, not a round's.
|
||||
[ "$DRY" = 1 ] || [ -w "$CALIB_DIR" ] || {
|
||||
echo "REFUSED: calib tap dir $CALIB_DIR is not writable (mkdir it, chown it, then re-run)" >&2
|
||||
exit 2
|
||||
}
|
||||
MOUNTS+=(-v "$CALIB_DIR:/data/calib")
|
||||
TAPENV=(-e RADIANCE_CALIB_DIR=/data/calib
|
||||
-e "RADIANCE_CALIB_TOKENS=${FRANK_QWEN38_CALIB_TOKENS:-12000000}")
|
||||
# The tap's own authorization. A pusher must see this marker before it sends
|
||||
# a prompt (calib-push.py --tap-marker), it names the container so a push
|
||||
# aimed anywhere else refuses, and a boot that fails removes it again below.
|
||||
# Measured 2026-10-29: with no marker, the chain pushed 4 rounds x 10M tokens
|
||||
# into RESTORED PRODUCTION after the rig's boot died, and nothing recorded.
|
||||
if [ "$DRY" = 1 ]; then
|
||||
:
|
||||
else
|
||||
printf 'container=%s\nimage=%s\nmode=calib\nstarted=%s\npid=%s\n' \
|
||||
"$C" "$IMAGE" "$(date -Is)" "$$" > "$CALIB_DIR/.rig-boot"
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
|
||||
CMD=(podman run --rm --name "$C"
|
||||
--network host --init
|
||||
--security-opt seccomp=unconfined
|
||||
--ulimit memlock=-1:-1
|
||||
--device /dev/kfd --device /dev/dri
|
||||
"${MOUNTS[@]}" "${TAPENV[@]}"
|
||||
"$IMAGE" "${ARGS[@]}")
|
||||
|
||||
# ---- gate: the window must hold the corpus (kldref alone can meet it) --------
|
||||
# score reads a recording instead of a corpus, and every arm declares at the
|
||||
# checkpoint's training context, so neither can meet a sequence too long to boot.
|
||||
# Measured on the raw JSON line — the pessimistic direction, and the same figure
|
||||
# the census in the header quotes. It runs on a dry run too, because a refusal
|
||||
# this cheap should never have to wait for the drained window to be discovered.
|
||||
if [ "$MODE" = kldref ] && [ -f "$KLD_CORPUS" ]; then
|
||||
read -r REC_NO REC_CHARS < <(awk '{ if (length($0) > best) { best = length($0); where = NR } } END { print where + 0, best + 0 }' "$KLD_CORPUS")
|
||||
LIMIT_CHARS=$(( MAX_MODEL_LEN * 3 ))
|
||||
if [ "$REC_CHARS" -gt "$LIMIT_CHARS" ]; then
|
||||
echo "REFUSED: record $REC_NO of $KLD_CORPUS is $REC_CHARS chars, past the $LIMIT_CHARS a $MAX_MODEL_LEN-token window promises at the pessimistic chars/3 — raise FRANK_QWEN38_MAX_MODEL_LEN, and the container's declared --max-ctx with it, before booting the reference" >&2
|
||||
exit 2
|
||||
fi
|
||||
echo "corpus fits: longest record $REC_CHARS chars (number $REC_NO) inside the $MAX_MODEL_LEN-token window (limit $LIMIT_CHARS chars)"
|
||||
fi
|
||||
|
||||
if [ "$DRY" = 1 ]; then
|
||||
echo "dry run: $MODE — the production unit is left running, nothing is mounted, nothing is written"
|
||||
printf '%q\n' "${CMD[@]}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ---- gates that only a real run needs ---------------------------------------
|
||||
if [ "$MODE" = score ]; then
|
||||
if [ ! -d "$KLD_REF_DIR" ]; then
|
||||
echo "REFUSED: no KLD reference at $KLD_REF_DIR — record it first in a drained window: scripts/run-qwen38-bf16.sh kldref" >&2
|
||||
exit 2
|
||||
fi
|
||||
if [ ! -f "$SCORE_RAD" ]; then
|
||||
echo "REFUSED: no container to score at $SCORE_RAD" >&2
|
||||
exit 2
|
||||
fi
|
||||
if [ -e "$SCORE_OUT" ]; then
|
||||
echo "REFUSED: $SCORE_OUT already exists — one report per name; move the old one to trash before re-scoring" >&2
|
||||
exit 2
|
||||
fi
|
||||
mkdir -p "$OUT_DIR"
|
||||
else
|
||||
if [ ! -f "$RAD" ]; then
|
||||
echo "REFUSED: no reference container at $RAD — build it first: scripts/build-qwen38-bf16-rad.sh" >&2
|
||||
exit 2
|
||||
fi
|
||||
[ "$MODE" = kldref ] && [ ! -f "$KLD_CORPUS" ] && {
|
||||
echo "REFUSED: no KLD corpus at $KLD_CORPUS — the record would be empty" >&2
|
||||
exit 2
|
||||
}
|
||||
fi
|
||||
if [ ! -f "$TEMPLATE" ]; then
|
||||
echo "REFUSED: no chat template at $TEMPLATE" >&2
|
||||
exit 2
|
||||
fi
|
||||
mkdir -p "$LOG_DIR"
|
||||
# The prod template, staged where the ro /models mount reaches it: a reference
|
||||
# recorded with the container's baked template scores a different tokenisation
|
||||
# than the arms served with froggeric's, and the KL would be measuring that.
|
||||
cp -f "$TEMPLATE" "$TPL" || {
|
||||
echo "REFUSED: cannot stage the chat template at $TPL (the container reads it through /models)" >&2
|
||||
exit 2
|
||||
}
|
||||
[ "$MODE" = kldref ] && mkdir -p "$KLD_REF_DIR"
|
||||
[ "$MODE" = calib ] && mkdir -p "$CALIB_DIR"
|
||||
|
||||
# Stop the campaign container BEFORE handing the cards back: a killed run leaves
|
||||
# its conmon alive (--rm only fires when podman itself exits), and production
|
||||
# then boots beside a container still holding the GPUs — measured 2026-10-29,
|
||||
# where the service reported `active` while its engine starved on the rig's VRAM.
|
||||
restore() {
|
||||
podman rm -f "$C" >/dev/null 2>&1 || true
|
||||
# Removing the container does NOT kill its engine. Measured 2026-10-30: a rig
|
||||
# radiance kept /dev/kfd after `podman rm -f`, production's
|
||||
# ar_oneshot_2rank_wht6 init then failed `device error`, the engine exited 0,
|
||||
# and Restart=always looped it 100 times with /health never answering. The
|
||||
# service is stopped for the whole run, so any radiance still holding the
|
||||
# device here is an orphan — kill it BY PID (pkill -f matches the caller's
|
||||
# own command line, which is how a session gets killed instead of the orphan).
|
||||
for p in $(fuser /dev/kfd 2>/dev/null); do
|
||||
if [ "$(cat "/proc/$p/comm" 2>/dev/null)" = radiance ]; then
|
||||
echo "restore: killing orphaned radiance pid $p still holding /dev/kfd" >&2
|
||||
kill -9 "$p" 2>/dev/null || true
|
||||
fi
|
||||
done
|
||||
sudo systemctl start "$SERVICE" >/dev/null 2>&1 || true
|
||||
}
|
||||
trap restore EXIT
|
||||
echo "run-qwen38-bf16 $MODE: stopping $SERVICE ($(date -Is)); log: $LOG"
|
||||
sudo systemctl stop "$SERVICE" >/dev/null 2>&1 || true
|
||||
podman rm -f "$C" >/dev/null 2>&1 || true
|
||||
|
||||
"${CMD[@]}" 2>&1 | tee "$LOG"
|
||||
rc=${PIPESTATUS[0]}
|
||||
# A boot that failed must not leave the tap marked as rig-authorized: the marker
|
||||
# is what calib-push.py trusts, and the point of it is that a dead rig can never
|
||||
# be pushed into again.
|
||||
if [ "$MODE" = calib ] && [ "$rc" -ne 0 ] && [ -f "$CALIB_DIR/.rig-boot" ]; then
|
||||
mv -f "$CALIB_DIR/.rig-boot" "$CALIB_DIR/.rig-boot.failed-$(date +%Y%m%d-%H%M%S)" 2>/dev/null || true
|
||||
echo "the rig boot failed (rc=$rc): the tap marker is retired, so no pusher can take this tap for a live rig"
|
||||
fi
|
||||
echo "run-qwen38-bf16 $MODE exited rc=$rc; $SERVICE restored; log: $LOG"
|
||||
exit "$rc"
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
# serve-frank-dragoon.sh — frank-dragoon endpoint: halogen-flash-server 0.16.2
|
||||
# serve-frank-dragoon.sh — frank-dragoon endpoint: halogen-flash-server 0.18.20
|
||||
# (closed-source engine, image ghcr.io/peonist-ai/halogen-flash-server,
|
||||
# digest-pinned in frank-dragoon.env) serving Quat3rnion's NATIVE v2
|
||||
# conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next on Dankclaw's
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve prism-ml/Ternary-Bonsai-2-9B-gguf (PQ2_0 ternary quant WITH the MTP head
|
||||
# in the file) as frank-zealot on Dankclaw's RTX 3090 via llama.cpp (PrismML
|
||||
# vendor build 10777, CUDA sm_86).
|
||||
# vendor build 12118 (0.7.0-dev, commit 024474448), CUDA sm_86).
|
||||
#
|
||||
# 2026-10-18 (Frank): Bonsai-2 27B PQ2_0 → the 9B MTP export, then the same
|
||||
# evening TWO 262,144-token conversations (--ctx-size 524288 --parallel 2) over
|
||||
@@ -78,7 +78,7 @@ if [[ -f "$ENV_FILE" ]]; then
|
||||
fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -------------------
|
||||
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism-b10777/llama-server}"
|
||||
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism-0244744/bin/llama-server}"
|
||||
: "${MODEL_FILE:=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf}"
|
||||
: "${ALIAS:=frank-zealot}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
@@ -119,7 +119,7 @@ fi
|
||||
# above are what pay for it. The draft cache dtype IS a further knob
|
||||
# (--spec-draft-type-k / --spec-draft-type-v, both default f16 — verified in this
|
||||
# build's --help) and stays at its default here. Card gate: quantized V alongside
|
||||
# MTP needs fork build >= prism-b10776, which is why LLAMA_BIN is the 10777
|
||||
# MTP needs fork build >= prism-b10776; LLAMA_BIN is 12118 (b10777 kept as rollback)
|
||||
# prebuilt; the type-142 loader lock keeps it pinned regardless.
|
||||
# SPEC_TYPE=none is the rollback and strips the whole set (flags move together).
|
||||
: "${SPEC_TYPE:=draft-mtp}"
|
||||
|
||||
@@ -337,8 +337,12 @@ def main():
|
||||
if any(meta[shard_ids[i]][4][1] != hd for i in range(n_shards)):
|
||||
log("ERROR: PLE shards disagree on HD")
|
||||
return 2
|
||||
if hd % 128 != 0 and hd != 160:
|
||||
log(f"WARNING: PLE HD={hd} is neither 160 nor a multiple of 128")
|
||||
if hd != 160 and hd % 128 != 0:
|
||||
# The pack below writes lanes of 128 codes; any other HD would produce a
|
||||
# table that decodes to the wrong rows. The old warning let it continue.
|
||||
log(f"ERROR: PLE HD={hd} is neither 160 nor a multiple of 128, and the "
|
||||
"128-lane pack cannot express it")
|
||||
return 2
|
||||
|
||||
# ---- idempotence gate -------------------------------------------------
|
||||
stacked_names = [f"{p}.mlp.experts.gate_up_proj" for p in prefixes]
|
||||
@@ -521,12 +525,16 @@ def verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
|
||||
}
|
||||
for name, (dt, shp) in want.items():
|
||||
e = hdr[name]
|
||||
assert e["dtype"] == dt and list(e["shape"]) == shp, name
|
||||
assert e["data_offsets"][1] - e["data_offsets"][0] == \
|
||||
int(np.prod(shp)) * (1 if dt == F8 else 2), name
|
||||
if e["dtype"] != dt or list(e["shape"]) != shp:
|
||||
raise RuntimeError(f"{name}: header is {e['dtype']}{e['shape']}, expected {dt}{shp}")
|
||||
if e["data_offsets"][1] - e["data_offsets"][0] != int(np.prod(shp)) * (1 if dt == F8 else 2):
|
||||
raise RuntimeError(f"{name}: data span is {e['data_offsets']}, shape {shp} needs "
|
||||
f"{int(np.prod(shp)) * (1 if dt == F8 else 2)} B")
|
||||
_, phdr = parse_header(out_path)
|
||||
e = phdr[next(iter(phdr))]
|
||||
assert e["dtype"] == BF16 and list(e["shape"]) == [sum(rows), hd]
|
||||
if e["dtype"] != BF16 or list(e["shape"]) != [sum(rows), hd]:
|
||||
raise RuntimeError(f"PLE table header is {e['dtype']}{e['shape']}, "
|
||||
f"expected {BF16}[{sum(rows)}, {hd}]")
|
||||
|
||||
# sampled PLE rows: recompute and compare bytes exactly
|
||||
lut = e4m3_lut()
|
||||
@@ -546,7 +554,8 @@ def verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
|
||||
with open(out_path, "rb") as ofh:
|
||||
ofh.seek(out_off + got_row * hd * 2)
|
||||
got = ofh.read(hd * 2)
|
||||
assert got == want, f"PLE row {got_row} mismatch"
|
||||
if got != want:
|
||||
raise RuntimeError(f"PLE row {got_row} does not match the decoded source")
|
||||
mm.close()
|
||||
log("verify: new-file headers, sizes and sampled PLE rows all exact; "
|
||||
"expert slabs were byte-checked in-stack")
|
||||
|
||||
@@ -94,6 +94,16 @@ for path in "${CHANGED[@]}"; do
|
||||
dankserver/.env.example|dankclaw/.env.example)
|
||||
add_test tests/env_consistency.bats
|
||||
;;
|
||||
# The SSD-backed metadata view: the installer, the host mount unit, and the
|
||||
# two media units that bind the view rather than the raw array.
|
||||
scripts/media-view.sh|dankserver/host/*)
|
||||
add_test tests/media_overlay.bats
|
||||
add_test tests/dankserver_quadlet.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
dankserver/jellyfin/quadlet/*|dankserver/plex/quadlet/*)
|
||||
add_test tests/media_overlay.bats
|
||||
;;
|
||||
shared/homepage/*)
|
||||
add_test tests/homepage_caddy.bats
|
||||
add_test tests/homepage_config.bats
|
||||
@@ -265,6 +275,13 @@ for path in "${CHANGED[@]}"; do
|
||||
add_test tests/voice_pack_unit.bats
|
||||
add_test tests/scripts_syntax.bats
|
||||
;;
|
||||
# The KLD corpus re-chunker: its seam is its own (the campaign's
|
||||
# one-token-series invariant is asserted there), so a change to it does
|
||||
# not land on the generic scripts/* fallback below.
|
||||
scripts/rechunk-kld-corpus.py)
|
||||
add_test tests/qwen38_rechunk.bats
|
||||
add_test tests/scripts_syntax.bats
|
||||
;;
|
||||
scripts/*)
|
||||
add_test tests/scripts_syntax.bats
|
||||
;;
|
||||
|
||||
@@ -48,6 +48,14 @@ Use the `Authorization: token` header for everything.
|
||||
- **Read an issue**: `curl -s -K "$cfg" $API/issues/<number>` (fields: `number`, `title`, `body`, `labels[].name`, `state`)
|
||||
- **List issues**: `curl -s -K "$cfg" "$API/issues?state=open&limit=50"` — add `&labels=<id>` to filter
|
||||
- **Comment**: `curl -s -K "$cfg" -X POST -d "$(jq -n --rawfile b "$BODY" '{body:$b}')" $API/issues/<number>/comments`
|
||||
- **Comment on a PR**: the conversation thread is `POST $API/issues/<PR-number>/comments`
|
||||
(201, and the answer's `html_url` points at `pulls/<n>`; Gitea's issue index is
|
||||
unified, so the PR's number is the same number). `$API/pulls/<n>/comments` is the
|
||||
**review**-comment store and answers `405` to a plain body.
|
||||
- **A body built by `jq --rawfile b /dev/stdin` inside `$(…)` is empty**: the command
|
||||
substitution does not inherit the heredoc, so Gitea receives `{}` and answers 422 —
|
||||
and `jq -r '.id'` on the error prints nothing, which reads as success. Write the
|
||||
body to a file and pass `--rawfile` that file.
|
||||
- **Apply labels**: `POST $API/issues/<number>/labels` with `{"labels":[<label-ID numbers>]}` — Gitea wants numeric label **IDs**, not names; resolve them from `GET $API/labels` (see [[Triage Labels]] for which labels exist)
|
||||
- **Remove a label**: `DELETE $API/issues/<number>/labels/<label-id>`
|
||||
- **Close**: `PATCH $API/issues/<number>` with `{"state":"closed"}`
|
||||
|
||||
@@ -130,10 +130,19 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
|
||||
|
||||
## 4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)
|
||||
|
||||
- **Hard-fail keys** — the whole unit silently fails to generate
|
||||
(`systemctl --user cat <unit>` shows "could not be found"): any
|
||||
dash-prefixed `[Container]` option (`--shm-size=…` → `unsupported key …
|
||||
in group 'Container'`); `Command=` is unsupported, use `Exec=`.
|
||||
- **Hard-fail keys abort the WHOLE BATCH, not one unit** (podman 5.7.0,
|
||||
live 2026-10-21): one unsupported key — e.g. `HostAddr=` — makes the
|
||||
generator log `unsupported key … in group 'Container'` plus
|
||||
`processing encountered some errors` and generate NOTHING for that
|
||||
manager. The running containers keep going as orphans, so the failure
|
||||
first appears on the next deploy as `Unit <x>.service not found` for a
|
||||
service that is visibly up. Other offenders: any dash-prefixed
|
||||
`[Container]` option (`--shm-size=…`), `Command=` (use `Exec=`).
|
||||
The seam `tests/quadlet_image_pins.bats` fails the build on any
|
||||
generator-rejected key in any host's units.
|
||||
Also: an image whose ENTRYPOINT is the binary takes FLAGS ONLY in
|
||||
`Exec=` — repeating the binary makes it a positional arg
|
||||
(`unexpected /bin/blackbox_exporter`, live 2026-10-21).
|
||||
- **HealthCmd space-splitting:** the v1 generator splits `HealthCmd` on
|
||||
spaces into the healthcheck argv; a quoted probe
|
||||
`/bin/sh -c "pg_isready …"` lands as argv token `"pg_isready` →
|
||||
@@ -148,6 +157,16 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
|
||||
"unsupported key"`, `systemctl --user cat probe.service` (shows the
|
||||
generated ExecStart verbatim), then remove + reload.
|
||||
|
||||
## 4a. Image pins (fleet law, 2026-10-21)
|
||||
|
||||
Every registry image in a quadlet is digest-pinned (`tag@sha256:…`);
|
||||
locally built images (`localhost/*`, `franksdank/*`) are exempt — their
|
||||
build is the pin. A rolling tag lets a rebuild move under a running
|
||||
service with nobody approving it, and it makes a CVE audit ungradeable.
|
||||
Gate: `tests/quadlet_image_pins.bats`. Where upstream publishes no version
|
||||
tags (copyparty-ac stops at 1.19.15), pin `:latest@sha256:` and name the
|
||||
version the config label carries in the comment beside it.
|
||||
|
||||
## 5. Rootless Quadlet log retrieval
|
||||
|
||||
Step 0 — establish ownership: `systemctl --user status <name>.service`
|
||||
@@ -176,8 +195,9 @@ deploy.sh, never a live env poke.
|
||||
check every namespace (`sudo podman ps -a`) and native units
|
||||
(`systemctl list-units`) before declaring a service absent. Compare
|
||||
compose mount sources against the real filesystem — compose files name
|
||||
phantom paths (e.g. `/mnt/media/*` vs the real LUKS array at
|
||||
`/mnt/entropy`). Fix compose through git first, deploy second.
|
||||
phantom paths (the legacy `dankserver/compose.yaml` still names `/opt/jellyfin/*`
|
||||
for a service whose roots moved to `/home/dank/Containers/jellyfin`). Fix
|
||||
compose through git first, deploy second.
|
||||
2. **Pre-create bind-mount sources** (`sudo mkdir -p` + `chown 1000:1000`)
|
||||
or creation fails `statfs …: no such file or directory`.
|
||||
3. **Staged bring-up:** infra first (`up -d mariadb redis`), poll
|
||||
|
||||
@@ -34,15 +34,16 @@ https://github.com/open-webui/open-webui
|
||||
The picker is built from the `model` table in `webui.db` (one row per model)
|
||||
plus the `model_ids` of each **enabled** OpenAI connection, NOT from live
|
||||
connection discovery — so the picker shows exactly what is seeded there.
|
||||
As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`** *(unverified
|
||||
2026-10-18: `webui.db` is the authority; the fleet itself serves all three ids
|
||||
`frank-dragoon`/`frank-zealot`/`frank-archon`, per the cards in
|
||||
`shared/homepage/services.json`)*.
|
||||
Since 2026-10-21 the picker exposes **all three fleet models to every
|
||||
account**: `frank-dragoon`, `frank-archon`, `frank-zealot` (per Frank: open
|
||||
`frank-archon` to everyone; the other two were already public). Proven with
|
||||
a `role=user` token: `/api/models` lists all three ids and a chat turn to
|
||||
`frank-archon` answers end-to-end.
|
||||
|
||||
| Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Endpoint page |
|
||||
| Connection (`openai.api_configs` idx) | `openai.api_base_urls` (live 2026-10-21) | Enabled | `model_ids` | Endpoint page |
|
||||
|---|---|---|---|---|
|
||||
| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | [[frank-dragoon]] |
|
||||
| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | [[frank-archon]] |
|
||||
| 0 | `http://host.containers.internal:7770/v1` | yes | `frank-dragoon` | [[frank-dragoon]] |
|
||||
| 1 | `http://192.168.0.235:7770/v1` | yes | `frank-archon` | [[frank-archon]] |
|
||||
| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | [[frank-zealot]] |
|
||||
|
||||
The URLs in the first column are the DB's recorded values and predate the 2026-08-22
|
||||
@@ -50,9 +51,10 @@ bridge move; on this network they are wrong — see the reachability gotcha belo
|
||||
is the corrected law. The picker row for `frank-zealot` was re-keyed on 2026-10-18 after
|
||||
that day's payload swap (see [[frank-zealot]] for what was deployed and when).
|
||||
|
||||
The `model` table carries exactly two decoration rows, `frank-zealot` and
|
||||
`frank-dragoon`. Both the context length and the description live **inside
|
||||
JSON columns** — `params` holds `{"context": {"length": …}}` and `meta` holds
|
||||
The `model` table carries three decoration rows — `frank-zealot`,
|
||||
`frank-dragoon`, `frank-archon` (200,000 context, added 2026-10-21).
|
||||
Both the context length and the description live **inside JSON columns** —
|
||||
`params` holds `{"context": {"length": …}}` and `meta` holds
|
||||
`{"description": …}`; there is no `context_length` or `description` column, so
|
||||
`sqlite3` one-liners against those names fail. `params.context.length` is a copy
|
||||
of the endpoint's per-slot window, so the authoritative number lives on the
|
||||
@@ -380,6 +382,20 @@ neither.
|
||||
name: /selected model/i })`) and the welcome text "How can I help you
|
||||
today?"; that snapshot is also the cheapest proof the picker is populated.
|
||||
|
||||
- **2026-10-21 — picker opened to all three models, for everyone.** Per
|
||||
Frank: `frank-archon` accessible to everyone. Three-leg repair on
|
||||
`webui.db` (snapshot `webui.db.bak-20261021` first): connection idx 1
|
||||
`enable: true`, a `model` row `frank-archon` (params
|
||||
`{"context":{"length":200000}}`, meta description from the endpoint card),
|
||||
and an `access_grant` row `(resource_type='model', resource_id='frank-archon',
|
||||
principal_id='*', 'read')` — `principal_id='*'` is the public grant, not a
|
||||
role row. Verified as a plain `role=user` token: `/api/models` lists all
|
||||
three ids, chat to `frank-archon` answers. Token-mint gotchas: the backend
|
||||
package imports only with cwd `/app/backend`, and `WEBUI_SECRET_KEY` must
|
||||
come from **PID 1's environ** (`podman exec` sessions do not inherit the
|
||||
app process env; the `.webui_secret_key` file happens to equal it — use it
|
||||
explicitly, `podman exec -e`).
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankclaw Stack]]
|
||||
|
||||
@@ -1,51 +1,68 @@
|
||||
---
|
||||
type: reference
|
||||
description: "The Dankserver entropy volume is a RAID-backed LUKS filesystem mounted at"
|
||||
description: "Dankserver's /mnt/entropy volume: the LUKS/array chain, the crypttab + fstab boot path, and the containers bound to it"
|
||||
tags:
|
||||
- dankserver
|
||||
timestamp: 2026-08-28
|
||||
timestamp: 2026-10-31
|
||||
---
|
||||
|
||||
# Entropy Volume
|
||||
|
||||
The Dankserver entropy volume is a RAID-backed LUKS filesystem mounted at
|
||||
`/mnt/entropy`. The command-line helper reuses the host's desktop-integrated
|
||||
`udisks2`, `crypttab`, and `fstab` configuration; it does not store or handle
|
||||
the passphrase itself.
|
||||
`/mnt/entropy` is the ext4 filesystem on the LUKS container holding `/dev/md0`
|
||||
(RAID6, seven disks). The media stack's data lives there, so the mount is a boot
|
||||
dependency for those containers rather than a convenience.
|
||||
|
||||
## Command-line mount
|
||||
|
||||
```bash
|
||||
/home/dank/serverAppConfigs/scripts/mount-entropy.sh
|
||||
```
|
||||
|
||||
Run it as the normal desktop user (`dank`), not from a root shell. `udisksctl`
|
||||
will securely prompt for the LUKS passphrase. `sudo mount /mnt/entropy` then
|
||||
uses the existing `/etc/fstab` entry, preserving its options:
|
||||
## Chain
|
||||
|
||||
```text
|
||||
nosuid,nodev
|
||||
/dev/md0 <- mdadm (its mdadm.conf ARRAY is what makes md0 exist at boot)
|
||||
-> LUKS UUID=cd86c728-459e-4f3a-bcc2-41ecf84685ec keyfile: /etc/entropy
|
||||
-> /dev/mapper/entropy <- systemd-cryptsetup@entropy.service
|
||||
-> ext4 at /mnt/entropy <- mnt-entropy.mount, from /etc/fstab
|
||||
```
|
||||
|
||||
The existing mountpoint is owned by `dank:dankserver` with mode `0755`; the
|
||||
helper deliberately does not run `chmod` or `chown`, so it cannot silently
|
||||
change those permissions.
|
||||
## Files that own it
|
||||
|
||||
## Device path
|
||||
| Repo file | Installs to | Produces at boot |
|
||||
|---|---|---|
|
||||
| `dankserver/host/crypttab` | `/etc/crypttab` | `systemd-cryptsetup@entropy.service` → `/dev/mapper/entropy` |
|
||||
| `dankserver/host/fstab` | `/etc/fstab` | `mnt-entropy.mount` (mountpoint `dank:dankserver`, `0755`) |
|
||||
|
||||
The current storage chain is:
|
||||
Deploy a change by copying the file onto the host, then
|
||||
`systemctl daemon-reload && update-initramfs -u`. The initramfs step is not
|
||||
optional: the cryptsetup hook bakes `/etc/crypttab` into the initrd, and the
|
||||
array is unlocked there, before the real root mounts. `scripts/mount-entropy.sh`
|
||||
(`udisksctl unlock` + `mount`) stays as the manual path for a volume that was
|
||||
never unlocked; on a correct boot it is redundant.
|
||||
|
||||
```text
|
||||
/dev/md0 -> LUKS UUID cd86c728-459e-4f3a-bcc2-41ecf84685ec
|
||||
-> ext4 label entropy
|
||||
-> /mnt/entropy
|
||||
```
|
||||
## Law
|
||||
|
||||
The helper checks that `/dev/md0` exists and exits successfully if the volume
|
||||
is already mounted. It is intentionally specific to this host and volume:
|
||||
that avoids accidentally unlocking or mounting a different disk.
|
||||
**`crypttab`'s device field takes the LUKS UUID, never the md array UUID.**
|
||||
`blkid` prints both on adjacent lines for a RAID-backed container, and the file
|
||||
shipped since 2025 carried the array UUID (`61bd6eaf-…`) where the LUKS container
|
||||
is `cd86c728-…`. The generator produced a perfectly valid unit pointing at a
|
||||
device that never appears, so the mapper came into existence only through a
|
||||
desktop `udisks2` session — the volume looked mounted on every login while every
|
||||
boot failed to mount it. The matching `fstab` line named the same wrong UUID
|
||||
directly instead of the mapper.
|
||||
|
||||
## Related
|
||||
**A LUKS keyfile is hashed whole, so a trailing newline is a different
|
||||
passphrase.** The shipped `/etc/entropy` was 27 bytes — 26 characters plus `\n` —
|
||||
and `cryptsetup` rejected it even after the UUID was corrected. Keyfiles stay
|
||||
`0600` with no trailing newline; prove it with
|
||||
`cryptsetup open --test-passphrase --key-file /etc/entropy UUID=…` *before*
|
||||
relying on it for a boot.
|
||||
|
||||
- [[Storage Policy]]
|
||||
- [[Dankserver Stack]]
|
||||
**Prove a mount fix with the artifacts systemd generates, not with a live
|
||||
mount.** The regenerated `systemd-cryptsetup@entropy.service` naming the right
|
||||
device, `findmnt /mnt/entropy` sourcing `/dev/mapper/entropy`, and a matching
|
||||
checksum between the repo copy and the host copy are the evidence. A mount that
|
||||
exists because someone logged in proves nothing about the boot path — that
|
||||
exactly describes the year this bug survived.
|
||||
|
||||
## Consumers
|
||||
|
||||
`jellyfin`, `copyparty`, and `plex` bind paths under `/mnt/entropy`; with the
|
||||
volume absent they start `active (healthy)` against empty directories, which is
|
||||
how a missing 91 TiB volume hides. See [[Dankserver Stack]] for the stack and
|
||||
[[Storage Policy]] for what may be written here.
|
||||
@@ -1,9 +1,9 @@
|
||||
---
|
||||
type: reference
|
||||
title: Jellyfin
|
||||
description: "Jellyfin on Dankserver: quadlet unit, entropy-volume data layout, ports, client access, hardware-transcode status."
|
||||
description: "Jellyfin on Dankserver: quadlet unit, overlay metadata view, provider keys, ports, access, transcode status."
|
||||
tags: [jellyfin, dankserver, media-server]
|
||||
timestamp: 2026-09-03T00:00:00Z
|
||||
timestamp: 2026-10-30T00:00:00Z
|
||||
---
|
||||
|
||||
# Jellyfin
|
||||
@@ -18,7 +18,7 @@ clients.
|
||||
public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
|
||||
[[Authentik]] outpost (post-auth route proxies `localhost:8096`).
|
||||
|
||||
## Facts (verified from the live unit, 2026-09-03)
|
||||
## Facts (the repo quadlet; the live unit is still the pre-view generation — see Data layout)
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
@@ -26,7 +26,7 @@ public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
|
||||
| Image | `ghcr.io/jellyfin/jellyfin:latest` |
|
||||
| Network | `dankserver_appnet`; publishes 8096/tcp, 8920/tcp, 7359/udp, 1900/udp |
|
||||
| Data roots | `/home/dank/Containers/jellyfin/config:/config`, `/home/dank/Containers/jellyfin/cache:/cache` |
|
||||
| Media | `Volume=/mnt/entropy:/data:ro` — the whole [[Entropy Volume]] mounted read-only; nothing automated writes to it |
|
||||
| Media | `Volume=/var/lib/media-meta/view:/data` in git — the [[Media Metadata View]] overlay, **writable**, sidecars land on the NVMe and the [[Entropy Volume]] array underneath stays untouched. **The live unit is still pre-view: `/mnt/entropy:/data:ro`** — nothing is written into the library until the view is deployed |
|
||||
| GPU | none — no `AddDevice`; CPU decode only (jellyfin-ffmpeg path) |
|
||||
| Env quirks | `DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true`, `LD_PRELOAD=libjemalloc.so.2`, `MALLOC_TRIM_THRESHOLD_=131072` kept verbatim from the compose-era container (zero-drift cutover) |
|
||||
| Health | `curl -f http://localhost:8096/health` every 30 s |
|
||||
@@ -36,11 +36,27 @@ public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
|
||||
```
|
||||
/home/dank/Containers/jellyfin/config -> /config (JELLYFIN_DATA_DIR)
|
||||
/home/dank/Containers/jellyfin/cache -> /cache (JELLYFIN_CACHE_DIR, XDG_CACHE_HOME)
|
||||
/mnt/entropy -> /data (read-only media volume)
|
||||
/var/lib/media-meta/view -> /data (git: the overlay view)
|
||||
/mnt/entropy -> /data (live today: the array, read-only)
|
||||
```
|
||||
|
||||
Library folders point at `/data` paths; because the mount uses the same
|
||||
absolute path as on the host, library paths survive container recreation.
|
||||
Library folders point at `/data` paths: stable across container recreation and
|
||||
identical to the raw array layout below. Which side of that path the container
|
||||
is on is the open question on this host — the installed unit binds the read-only
|
||||
array, and `scripts/media-view.sh status` reports `NOT MOUNTED` (2026-10-31).
|
||||
|
||||
`Local metadata` — save metadata to media folders — is enabled, and it is the
|
||||
reason the view exists: `movie.nfo`, `tvshow.nfo`, `episode.nfo` and artwork
|
||||
written once on the NVMe upper layer are files [[Plex]] can read too. While
|
||||
`/data` is bound read-only every item update logs the error below and no sidecar
|
||||
is written; Jellyfin becomes the fleet's only metadata writer with the deploy
|
||||
that raises the view.
|
||||
|
||||
**Failure signature:** `System.IO.IOException: Read-only file system :
|
||||
'/data/Movies/<film>/movie.nfo'` means the container is binding the raw array
|
||||
instead of the view. On Dankserver that is today's expected state — the view has
|
||||
never been installed (`scripts/media-view.sh status`). Once it is deployed, the
|
||||
same line is a fault: the quadlet drifted or the mount died.
|
||||
|
||||
## Ports & access
|
||||
|
||||
@@ -105,12 +121,109 @@ where the LDAP plugin runs. Verified 2026-09-25: `alex` authenticates over
|
||||
`AuthenticateByName` (HTTP 200; full plugin flow logged — service bind,
|
||||
search, `ldapUid`, user bind, token).
|
||||
|
||||
## Metadata providers and their keys (2026-10-30)
|
||||
|
||||
TMDb, OMDb, StudioImages and MusicBrainz are **core plugins** in Jellyfin 12 —
|
||||
no DLL, just a configuration file the running container owns under the data
|
||||
root. TVDB is not core (its metadata provider was removed in 10.9), so it is an
|
||||
official-catalog plugin.
|
||||
|
||||
| Provider | Shape on disk | Key lives in |
|
||||
|---|---|---|
|
||||
| TMDb | `config/plugins/configurations/Jellyfin.Plugin.Tmdb.xml` → `<TmdbApiKey>` | `/home/dank/Containers/jellyfin/provider-keys` → `TMDB_API_KEY=` (mode 600) |
|
||||
| TVDB | `config/plugins/thetvdb_24.0.0.0/` + `…Jellyfin.Plugin.Tvdb.xml` → `<ApiKey>` | the same file → `TVDB_API_KEY=` |
|
||||
| OMDb | `…Jellyfin.Plugin.Omdb.xml` | **not configured** — its key is issued by email, 1 000 req/day, and its ToS forbids storing the returned data; ratings stay Jellyfin's own lookup |
|
||||
|
||||
**None of it is tracked in git.** Those XML files belong to the container and
|
||||
hold credentials, and `tests/jellyfin_providers.bats` fails the suite if one is
|
||||
ever committed. The keys themselves sit in the per-service host env file that is
|
||||
the fleet's convention for container credentials (grafana, caddy, pihole,
|
||||
copyparty, romm, seerr): `provider-keys`, mode 600, beside the data root.
|
||||
`scripts/jellyfin-providers.sh` is the only thing that writes the XML:
|
||||
|
||||
```
|
||||
scripts/jellyfin-providers.sh plan # what would change; prints no key
|
||||
scripts/jellyfin-providers.sh install # plugin + keys, previous copy to trash
|
||||
scripts/jellyfin-providers.sh verify # plugin + key LENGTHS, never key bytes
|
||||
```
|
||||
|
||||
- **A key never enters git, argv, or the tool's output.** The installer reads
|
||||
that 600-perm file itself (a value in argv is world-readable through
|
||||
`/proc/<pid>/cmdline`) and refuses it when it is group- or world-readable —
|
||||
an exposed key is an error, not a warning. A key that shows up in a transcript
|
||||
or `ps` output has leaked: rotate it with the provider, rewrite
|
||||
`provider-keys`, re-run `install`.
|
||||
- **The TVDB artifact is pinned three ways**: sha256
|
||||
`6302fba3857bdde8d186dd099fac09ca43247041f531a528a2c788add70a5319`, the
|
||||
official plugin guid `a677c0da-fac5-4cde-941a-7134223f14c8`, and version
|
||||
`24.0.0.0` (`targetAbi 12.0.0.0`, which fits the deployed 12.4.0). A sha
|
||||
match alone only proves the bytes are the bytes fetched — the guid check is
|
||||
what says it is the official plugin. Rollback: move `thetvdb_24.0.0.0/` into
|
||||
the trash and revert the pin; a second `thetvdb_*` directory is a refusal,
|
||||
never a quiet second copy.
|
||||
- **The homelab TVDB key is a licensed key** (its JWT claims say
|
||||
`community_supported: false`), so it needs **no PIN** — the optional
|
||||
`TVDB_PIN=` line exists only for a subscriber-shaped key.
|
||||
- **TVDB is installed and keyed but is not a metadata agent**: TMDb still
|
||||
identifies everything. Enabling TVDB as a provider would re-fetch show
|
||||
metadata across the library, so it is a deliberate, separate change.
|
||||
- **The deployer places the keys before it restarts jellyfin**
|
||||
(`ensure_provider_keys` in `scripts/deploy-remote.sh`) and warns instead of
|
||||
failing when `provider-keys` is absent, so a host without it still deploys.
|
||||
- **Keys and plugins are read at start-up**: after a manual `install` the
|
||||
change is inert until `./scripts/deploy.sh dankserver --service jellyfin`.
|
||||
- TMDb's terms are the reason this stays narrow: attribution required, no
|
||||
caching beyond 6 months, no derivative works. Fine for a personal homelab,
|
||||
and the reason no bulk enrichment pass runs against it.
|
||||
|
||||
## Configuration workflow
|
||||
|
||||
1. Create libraries pointing at `/data` folders (movies, shows, audio).
|
||||
2. Set metadata/language preferences (`LANG=en_US.UTF-8` pinned in the unit).
|
||||
3. Configure clients to use `http://192.168.0.235:8096` or the guarded vhost.
|
||||
|
||||
## Triggering a scan
|
||||
|
||||
The scan key lives in `~/.config/secrets/jellyfin` (mode 600; the same value is
|
||||
`JELLYFIN_API_KEY` in the ignored `dankserver/.env`). It rides a mode-600 curl
|
||||
config, never a header argument, in the `MediaBrowser` form: 12.4 authenticates
|
||||
only `Authorization: MediaBrowser Token="…"` and refuses a bare `X-Emby-Token`
|
||||
for every key, valid or not. Before believing a 401, prove the key against the
|
||||
server's own table — `select count(*) from ApiKeys where AccessToken = ?` on
|
||||
`jellyfin.db` opened `mode=ro&immutable=1`. The backslashes below matter:
|
||||
|
||||
```bash
|
||||
umask 077
|
||||
{ printf 'header = "Authorization: MediaBrowser Token=\\"%s\\""\n' \
|
||||
"$(cat ~/.config/secrets/jellyfin)"
|
||||
echo 'url = "http://127.0.0.1:8096/Library/Refresh"'
|
||||
echo 'request = "POST"'; } >/tmp/jf.hdr
|
||||
curl -s -K /tmp/jf.hdr -o /dev/null -w "jellyfin scan: %{http_code}\n"
|
||||
shred -u /tmp/jf.hdr
|
||||
```
|
||||
|
||||
`204` is acceptance; anything else queued no scan, so re-issue the identical
|
||||
request instead of diagnosing. A `401` from a key that answers `200` on a
|
||||
hand-typed request is this block's quoting, not the server: bash `printf`
|
||||
collapses `\"` to `"`, curl ends the quoted value at that quote, and Jellyfin
|
||||
receives `Authorization: MediaBrowser Token=` with nothing after it (`\\"` answers
|
||||
204, `\"` answers 401 — both run on this host).
|
||||
|
||||
One accepted refresh is the whole procedure. A second one while a scan runs moves
|
||||
`Scan Media Library` to `Cancelling`, and once the runner holds `Cancelling` every
|
||||
later `204` is accepted and then discarded — the status code stops meaning
|
||||
anything. The fix is `systemctl --user restart jellyfin.service` (never
|
||||
`deploy.sh`: that runs the never-tested metadata-view cutover), one POST after
|
||||
health answers, and then believe `ScheduledTasks` State at 130 % CPU, not the
|
||||
response code.
|
||||
|
||||
**A scan that must see a new folder has to be recursive.** `POST /Library/Refresh`
|
||||
is the incremental pass, and inotify does not cross a rootless podman bind, so
|
||||
folders created on the host stay invisible to it. The move is
|
||||
`POST /Items/{id}/Refresh?Recursive=true` (204) per library, the id from
|
||||
`GET /Library/VirtualFolders` — its field is `ItemId`, not `Id` — and that same
|
||||
record's `RefreshStatus` is the progress read.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Plex]] — co-resident media server on the same volume
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
---
|
||||
type: reference
|
||||
title: Media Audiobook Relayout
|
||||
description: "Re-filing /mnt/entropy/Audiobooks by Plex's chapter rows: narrator shelves, no renumbering, the refusals."
|
||||
tags: [dankserver, media-server, plex, media-relayout]
|
||||
timestamp: 2026-10-31T00:00:00Z
|
||||
---
|
||||
|
||||
# Media Audiobook Relayout
|
||||
|
||||
`/mnt/entropy/Audiobooks` is the one library here that [[Jellyfin]] does not read
|
||||
at all — its music scanner has no audiobook concept — while [[Plex]] indexes all
|
||||
of it as Artist → Album → Track: the album is the BOOK, that book's single artist
|
||||
is the NARRATOR, both matched out of the files' own tags. Today one book is split
|
||||
across `… [Part 1]`, `… [Part 2]` folders three or four deep inside a collection
|
||||
pack, so a scan serves three quarters of a novel; `scripts/media-audiobooks.py`
|
||||
re-files the tree so the property both servers' book scanners assume is true —
|
||||
|
||||
Audiobooks/<Narrator>/<Book> (<year>)/<the file keeps its own name>
|
||||
|
||||
The shared gates, the executor, the journal law and the rollback live in
|
||||
[[Media Relayout]]; this page is the audiobook arm only.
|
||||
|
||||
```bash
|
||||
# 1. The authority: Plex's chapter rows for this root (the DB opens read-only).
|
||||
python3 scripts/media-census.py audiobooks --root /mnt/entropy/Audiobooks \
|
||||
--out /tmp/ab_rows.tsv
|
||||
# 2. What is on disk (stat only).
|
||||
python3 scripts/media-census.py enumerate --root /mnt/entropy/Audiobooks \
|
||||
--out /tmp/ab_enum.tsv
|
||||
# 3. The plan.
|
||||
python3 scripts/media-audiobooks.py --root /mnt/entropy/Audiobooks \
|
||||
--enum /tmp/ab_enum.tsv --rows /tmp/ab_rows.tsv --out /tmp/ab_plan.tsv \
|
||||
--exceptions /tmp/ab_exceptions.txt
|
||||
# 4. Read the exceptions file FIRST. Then the same executor, journal and rollback
|
||||
# as [[Media Relayout]] shows, on its own journal, and refresh the Plex section
|
||||
# as in step 5 of the movie Procedure — from inside the container, never over
|
||||
# the host port.
|
||||
scripts/media-apply.py apply --plan /tmp/ab_plan.tsv --root /mnt/entropy \
|
||||
--journal /mnt/entropy/media-relayout-audiobooks.jsonl --apply
|
||||
```
|
||||
|
||||
The group is the identity Plex asserts in its own database — **(narrator, book,
|
||||
year)** — so one book arriving in four part folders becomes one folder, and a
|
||||
1965 and a 2020 recording that share a title stay two books.
|
||||
|
||||
Two laws are forced by what the rows actually hold:
|
||||
|
||||
+- **A chapter is never renumbered.** Most books contain chapters whose index Plex
|
||||
does not know, so a number written into a name would be invented — and it would
|
||||
buy nothing, because both servers order playback by the embedded track tag, not
|
||||
by the file name. A wrong number in a name is the number a human later trusts.
|
||||
Fix order in the tags: a Plex scan re-reads tags and never renames a file.
|
||||
+- **The shelf is the narrator, not the author.** Every book has exactly one artist
|
||||
row and it is the narrator; the author survives only inside some album titles
|
||||
(`Kryptos by Graham McNeill`) and in no column Plex stores, so deriving it would
|
||||
mean parsing titles where a ripper wrote one and inventing it everywhere else.
|
||||
The author stays visible in the folder name Plex itself shows.
|
||||
|
||||
What it refuses, the music arm's list with one addition: a video container, audio
|
||||
no chapter row names, and a loose file in a folder holding two books all park under
|
||||
`_unsorted` with the path they have today — and so does a chapter whose row names
|
||||
**no book or no narrator**: a placeholder (`[Unknown Album]`, `[Unknown Artist]`)
|
||||
that unrelated books share, or a chapter number mistagged as the artist (`0.4`).
|
||||
`Various Artists` is not a placeholder — that is a multi-cast read.
|
||||
|
||||
The reviewed plan lives at `/mnt/entropy/media-relayout-audiobooks-plan.tsv`; it
|
||||
applies on its own journal, behind the music writer, and then refreshes the Plex
|
||||
section. Verify it with `--db-kind plex` — Jellyfin holds no rows here, so a
|
||||
Jellyfin pass proves nothing either way.
|
||||
|
||||
## See Also
|
||||
|
||||
[[Media Relayout]], [[Media Music Relayout]], [[Plex]]
|
||||
@@ -0,0 +1,130 @@
|
||||
---
|
||||
type: reference
|
||||
title: Media Metadata View
|
||||
description: "The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback."
|
||||
tags: [dankserver, media-server, jellyfin, plex, storage]
|
||||
timestamp: 2026-10-30T00:00:00Z
|
||||
---
|
||||
|
||||
# Media Metadata View
|
||||
|
||||
`/var/lib/media-meta/view` is an OverlayFS merge of the media library: the RAID6
|
||||
array below, a directory on the NVMe above. It exists so [[Jellyfin]] can write
|
||||
Kodi-format sidecars (`movie.nfo`, `tvshow.nfo`, `episode.nfo`, artwork) that
|
||||
[[Plex]] can read. Sidecar metadata is only useful sitting next to the media
|
||||
file, and the array is read-only and slow — so the media bytes stay there and the
|
||||
*writes* land on SSD. One tree to both servers, one copy of the media.
|
||||
|
||||
The mount point is not a volume and holds nothing while unmounted. It sits inside
|
||||
`/var/lib/media-meta` so that layers, kernel bookkeeping, the reap trash and the
|
||||
merged door are one directory — and so the mount table grows no new `/mnt` entry
|
||||
next to the array.
|
||||
|
||||
**Not deployed on Dankserver — verified 2026-10-31.** `scripts/media-view.sh
|
||||
status` answers `view: /var/lib/media-meta/view NOT MOUNTED`,
|
||||
`/var/lib/media-meta` does not exist, and the installed quadlets under
|
||||
`~/.config/containers/systemd/` still carry the pre-view binds — `/mnt/entropy:/data:ro`
|
||||
for [[Jellyfin]], `/mnt/entropy:/mnt/entropy:ro` for [[Plex]]. Everything below
|
||||
is the shape git describes, not the shape the servers run. The cutover is an
|
||||
ordinary deploy: `deploy.sh dankserver --service jellyfin|plex` runs
|
||||
`ensure_media_view` (`install` then `verify`) before it restarts the container,
|
||||
so it must not be fired through the middle of a library scan. Until that deploy
|
||||
lands there is no upper layer, so no relayout can leave ghost metadata and
|
||||
`reap` has nothing to do — `media-apply.py` prints that follow-up only when the
|
||||
view really is a mountpoint.
|
||||
|
||||
## Shape
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Lower layer | `/mnt/entropy` — the RAID6 array, source of truth, never written |
|
||||
| Upper layer | `/var/lib/media-meta/upper` — NVMe; every write lands here |
|
||||
| Work dir | `/var/lib/media-meta/work` — overlayfs internals, same filesystem |
|
||||
| Mount point | `/var/lib/media-meta/view` — the merged tree both servers bind |
|
||||
| Unit (repo) | `dankserver/host/systemd/media-meta-view.mount` — the source of truth |
|
||||
| Unit (host) | `var-lib-media\x2dmeta-view.mount` — systemd names a mount unit after its escaped path (`-` becomes `\x2d`); `media-view.sh install` writes the repo file under that name |
|
||||
| Installer | `scripts/media-view.sh` — `plan` / `install` / `verify` / `status` / `reap` |
|
||||
| Written by | Jellyfin only (Plex binds read-only); nothing else in the fleet touches it |
|
||||
|
||||
Both consumers guard the view with
|
||||
`UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view`, and
|
||||
`scripts/deploy.sh dankserver --service jellyfin|plex` calls `media-view.sh
|
||||
install` + `verify` before it restarts either container.
|
||||
|
||||
| Consumer | Bind | Mode | Why that path |
|
||||
|---|---|---|---|
|
||||
| [[Jellyfin]] | `/var/lib/media-meta/view:/data` | rw | It is the metadata writer; its sidecars become SSD bytes |
|
||||
| [[Plex]] | `/var/lib/media-meta/view:/mnt/entropy` | ro | Every path in the PMS database was recorded as `/mnt/entropy/...`; only the HOST side of the bind moved, so its database still resolves |
|
||||
|
||||
Putting the sidecars inside Plex's reach is not the same as making Plex use them:
|
||||
a library parses `.nfo` only with its *Local media information* agent enabled, and
|
||||
this change touches no library setting.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
scripts/media-view.sh plan # print the resolved shape, touch nothing
|
||||
sudo scripts/media-view.sh install # create dirs, install the unit, mount, enable
|
||||
scripts/media-view.sh verify # is the view really the overlay? (deploy gate)
|
||||
scripts/media-view.sh status # mount options + upper-layer size/file count
|
||||
scripts/media-view.sh reap # list ghost metadata (media moved, sidecars left)
|
||||
sudo scripts/media-view.sh reap --apply # move ghosts to the SSD trash, never delete
|
||||
```
|
||||
|
||||
`--upper-base DIR` and `--lower DIR` relocate the layers for `plan` and `reap`;
|
||||
the defaults are the host values above.
|
||||
|
||||
## Laws
|
||||
|
||||
- **The upper layer must sit on non-rotational storage.** `install` and `verify`
|
||||
refuse via `lsblk -dno ROTA` if the upper base lives on the array: an overlay
|
||||
whose upper is the spinners is the original problem with extra steps, and it
|
||||
silently turns every metadata write back into a seek.
|
||||
- **Never mount the view over an unmounted lower.** `RequiresMountsFor=/mnt/entropy`
|
||||
plus the container `ConditionPathIsMountPoint` guards: an empty `lowerdir`
|
||||
presents an *empty library* as truth, and a media scan would then clear the
|
||||
database of everything. systemd does not create a mount point, so `install`
|
||||
creates `view` itself; without it the mount simply fails.
|
||||
- **Never delete through the view.** A delete writes an overlayfs whiteout
|
||||
(`.wh.<name>`) in the upper layer, which hides the array's copy without
|
||||
removing it — the file reappears the moment the view is unmounted, and the
|
||||
upper layer now carries a landmine. Deletions belong on the raw path
|
||||
(`/mnt/entropy`), which the relayout tool already treats as the only writer.
|
||||
`reap` refuses to remove whiteouts and only warns about them.
|
||||
- **A relayout must be followed by `reap`.** Renames happen on the raw path, so
|
||||
sidecars the view already wrote stay at the old coordinates: a ghost folder
|
||||
the servers browse and find nothing in. `reap` lists upper-layer directories
|
||||
with no counterpart below and moves them (with their subtree) into a dated
|
||||
`.AGENTTRASH/` under the SSD base. It is recursive and file-safe: a sidecar
|
||||
that exists only above is normal, so only *directories* are candidates.
|
||||
- **File-watcher events do not cross the overlay.** Plex's NFO agent and
|
||||
Jellyfin's real-time monitor see inotify events for writes that land *in the
|
||||
upper layer*, not for changes on the array underneath. After any relayout or
|
||||
externally-written metadata, trigger a library refresh — see
|
||||
[[Media Relayout]] for the refresh procedure.
|
||||
|
||||
## Failure signatures
|
||||
|
||||
| Symptom | Meaning |
|
||||
|---|---|
|
||||
| `jellyfin.service` / `plex.service` show `ConditionPathIsMountPoint=... was not met` and stay inactive | The view is not mounted. Do not start them anyway: `scripts/media-view.sh install`, then `verify`. |
|
||||
| Jellyfin log: `Read-only file system : '/data/Movies/…/movie.nfo'` | The container is binding the raw array, not the view — the quadlet drifted back or the view is gone. `media-view.sh status`. |
|
||||
| A folder appears in a media server with artwork/`.nfo` but no video | Ghost metadata from a relayout. `media-view.sh reap`. |
|
||||
| `verify` prints `refused: …/view is not a mountpoint` | A plain directory is sitting where the overlay belongs. That is the fail-closed guard working; `install` on the box. |
|
||||
| `systemctl status` finds no such unit | Look under the escaped name, `var-lib-media\x2dmeta-view.mount`, not the repo file name. |
|
||||
|
||||
## Rollback
|
||||
|
||||
Revert the commit that pointed the two quadlets at the view and re-deploy both
|
||||
services — they bind `/mnt/entropy` read-only again, which is the pre-view shape.
|
||||
The NVMe upper layer can stay: nothing reads it once the binds are back, and it
|
||||
holds no state either server needs. Unmounting alone
|
||||
(`systemctl stop 'var-lib-media\x2dmeta-view.mount'`) is safe while no container
|
||||
binds it, and the array is never modified by any of this.
|
||||
|
||||
## Links
|
||||
|
||||
- [[Jellyfin]] — the metadata writer; its data layout section.
|
||||
- [[Plex]] — the read-only consumer; in-place bind-mount history.
|
||||
- [[Media Relayout]] — the tool that moves media on the raw path and the reason `reap` exists.
|
||||
- [[Entropy Volume]] — the array underneath.
|
||||
@@ -0,0 +1,68 @@
|
||||
---
|
||||
type: reference
|
||||
title: Media Music Relayout
|
||||
description: "Re-filing /mnt/entropy/Music by Jellyfin's tag rows: the census, the album key, the refusals."
|
||||
tags: [dankserver, media-server, jellyfin, media-relayout]
|
||||
timestamp: 2026-10-31T00:00:00Z
|
||||
---
|
||||
|
||||
# Media Music Relayout
|
||||
|
||||
`/mnt/entropy/Music` is rips and discography packs, and Jellyfin has matched and
|
||||
tagged almost all of it. `scripts/media-music.py` re-files it into the shape
|
||||
Jellyfin's music scanner reads —
|
||||
|
||||
Music/<album artist>/<Album> (<year>)/NN - <Title>.ext
|
||||
|
||||
— and the authority is the tag row of the track, never the name the ripper gave
|
||||
the file. The shared gates, the executor, the journal law and the rollback live in
|
||||
[[Media Relayout]]; this page is the music arm only.
|
||||
|
||||
```bash
|
||||
# 1. The authority: Jellyfin's audio rows, one per track, with the album and
|
||||
# artist ids its own database carries (the DB opens read-only).
|
||||
python3 scripts/media-census.py music --root /mnt/entropy/Music \
|
||||
--out /tmp/music_rows.tsv
|
||||
# 2. What is on disk (stat only).
|
||||
python3 scripts/media-census.py enumerate --root /mnt/entropy/Music \
|
||||
--out /tmp/music_enum.tsv
|
||||
# 3. The plan.
|
||||
python3 scripts/media-music.py --root /mnt/entropy/Music --enum /tmp/music_enum.tsv \
|
||||
--rows /tmp/music_rows.tsv --out /tmp/music_plan.tsv \
|
||||
--exceptions /tmp/music_exceptions.txt
|
||||
# 4. Read the exceptions file FIRST — it is the review, not a log. Then the same
|
||||
# executor, journal and rollback as [[Media Relayout]] shows, on its own journal.
|
||||
scripts/media-apply.py apply --plan /tmp/music_plan.tsv --root /mnt/entropy \
|
||||
--journal /mnt/entropy/media-relayout-music.jsonl --apply
|
||||
```
|
||||
|
||||
The group is what the tags assert, not where the files sit: **(album-artist tag,
|
||||
album, year)**. One album arrives as three disc folders and as two copies of one
|
||||
rip, and one MusicBrainz release group is one album. A compilation is one album
|
||||
whose tracks name many performers, so it files once under `Various Artists`:
|
||||
keying on the performer would file one album once per artist. Two different
|
||||
album-artist tags on one album name are two albums, which is what Jellyfin's two
|
||||
album items say.
|
||||
|
||||
What it refuses:
|
||||
|
||||
+- **A sidecar is placed by the audio around it, never by its own name.** A
|
||||
`.lrc`, `.txt` or picture that shares a track's stem follows that track and
|
||||
takes the name the track now has, or Jellyfin cannot find it; anything else
|
||||
goes to the album its own folder was placed in, or to the folder whose discs
|
||||
sit one level below it. A folder whose audio belongs to two albums parks its
|
||||
sidecar rather than guessing.
|
||||
+- **An image album keeps its name.** A folder holding one audio file and a
|
||||
`.cue` is a whole album in one file; numbering it `01 - …` tells Jellyfin to
|
||||
index it as one track.
|
||||
+- **A video container parks**, because Jellyfin will not index it as music and a
|
||||
rename to `.m4a` would claim it is; **audio no provider row names parks** too,
|
||||
because unmatched audio is a tagging job, not a renaming job.
|
||||
|
||||
ONE writer at a time on this array: two rename passes race, and neither journal
|
||||
tells. Verify the run against Jellyfin's own tables — see *Proving the move
|
||||
landed* in [[Media Relayout]].
|
||||
|
||||
## See Also
|
||||
|
||||
[[Media Relayout]], [[Jellyfin]], [[Media Audiobook Relayout]]
|
||||
@@ -0,0 +1,475 @@
|
||||
---
|
||||
type: reference
|
||||
title: Media Relayout
|
||||
description: "Census, plan and apply the movie, TV, music and audiobook libraries: the gates, the refusals, the rollback."
|
||||
tags: [dankserver, media-server, jellyfin, plex, procedure]
|
||||
timestamp: 2026-10-30T00:00:00Z
|
||||
---
|
||||
|
||||
# Media Relayout
|
||||
|
||||
The movie library on [[Entropy Volume]] arrived as release folders —
|
||||
`Some.Movie.2019.1080p.WiKiTaLx.x265-FOO/` with the video inside named after
|
||||
the release. [[Jellyfin]] matches a movie by folder and file name, so the
|
||||
release noise cost posters, extras and versions. These scripts turn that
|
||||
library into the shape its media server reads — the movies, the TV packs, the TV
|
||||
that turns up in the movie share, the music library, and the audiobooks, whose
|
||||
only reader is Plex — and nothing else in the repo touches a media file.
|
||||
|
||||
| Script | Touches media? | Output |
|
||||
|---|---|---|
|
||||
| `scripts/media-census.py` | reads headers and the servers' own DBs, never writes | `enum`, `probed`, and one authority TSV per library |
|
||||
| `scripts/media-plan.py` | no — TSVs in, TSVs out | the plan + the exception report |
|
||||
| `scripts/media-apply.py` | **renames, never copies**; sweeps only what a run emptied | the moves, plus a journal |
|
||||
| `scripts/media-tv.py` | no — TSV out | a plan that moves TV into the TV library |
|
||||
| `scripts/media-shows.py` | no — TSV out | a plan that re-files the TV library itself |
|
||||
| `scripts/media-music.py` | no — TSV out | a plan that re-files the music library by its tags |
|
||||
| `scripts/media-audiobooks.py` | no — TSV out | a plan that re-files the audiobooks by the book Plex matched |
|
||||
| `scripts/media-verify.py` | no — reads the servers' DBs read-only | the verdict that every journaled move landed |
|
||||
|
||||
The seams are `tests/media_relayout.bats`, `tests/media_shows.bats`,
|
||||
`tests/media_music.bats`, `tests/media_audiobooks.bats` and `tests/media_verify.bats`;
|
||||
the movie and TV target contracts are quoted from
|
||||
[jellyfin.org/docs/general/server/media/movies](https://jellyfin.org/docs/general/server/media/movies)
|
||||
in `media-plan.py`'s docstring.
|
||||
|
||||
## Procedure
|
||||
|
||||
Run the census on the machine that mounts the library. `probe` is the slow step
|
||||
(a header read per container) and is `nice`d — it is safe beside a live service
|
||||
because it never competes for the GPU and never writes inside the library.
|
||||
|
||||
```bash
|
||||
C=scripts/media-census.py
|
||||
$C enumerate --root /mnt/entropy/Movies --out /tmp/media-enum.tsv
|
||||
$C probe --enum /tmp/media-enum.tsv --out /tmp/media-probed.tsv --jobs 6
|
||||
$C plex --root /mnt/entropy/Movies --out /tmp/plex-movies.tsv
|
||||
```
|
||||
|
||||
`plex` reads the Plex metadata cache (`--db` defaults to the path the
|
||||
[[Plex]] container bind-mounts at `/var/lib/plexmediaserver`) with plain
|
||||
`mode=ro`: a live server is never locked or written, and plain ro is also what
|
||||
makes a scan's uncheckpointed rows visible. The same rule as
|
||||
`scripts/media-verify.py` — see *Proving the move landed*.
|
||||
|
||||
Then plan — this reads only the three TSVs, so it costs seconds and can run
|
||||
anywhere, including a laptop:
|
||||
|
||||
```bash
|
||||
scripts/media-plan.py --root /mnt/entropy/Movies \
|
||||
--enum /tmp/media-enum.tsv --probed /tmp/media-probed.tsv \
|
||||
--plex /tmp/plex-movies.tsv --out /tmp/jf-plan.tsv --exceptions /tmp/jf-exceptions.txt
|
||||
```
|
||||
|
||||
**Read `/tmp/jf-exceptions.txt` before applying.** It is not a log: every line
|
||||
is a file the plan could not name a film for, a pile that needs a README, or a
|
||||
collision it refused. The printed gate lines are the go/no-go:
|
||||
`problems`, `bad-names`, `occupied`, `double-claim`, `left-unmoved` must all be
|
||||
0 and `unplanned=0` must appear in the report. Any non-zero number means a file
|
||||
that would be filed wrong, and the plan is not run until it is understood.
|
||||
|
||||
```bash
|
||||
scripts/media-apply.py apply --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies # dry, writes nothing
|
||||
scripts/media-apply.py apply --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies --apply
|
||||
scripts/media-apply.py verify --plan /tmp/jf-plan.tsv --journal /mnt/entropy/media-relayout.jsonl
|
||||
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies # dry
|
||||
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies --apply
|
||||
scripts/media-apply.py rollback --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies --apply
|
||||
```
|
||||
|
||||
An apply also leaves the release folders it emptied standing where they were —
|
||||
hundreds of them at the library root — which is what `sweep` removes, dry-run
|
||||
first, from the same journal.
|
||||
|
||||
After it lands, both servers need a scan or the change is invisible. [[Plex]] and
|
||||
[[Jellyfin]] own the exact requests; two laws hold at this step: `204` is
|
||||
acceptance and anything else queued nothing — re-issue the identical request
|
||||
rather than diagnosing — and ONE accepted refresh is the whole procedure, because
|
||||
a second while a scan runs moves `Scan Media Library` to `Cancelling` and every
|
||||
later `204` is accepted and then discarded.
|
||||
|
||||
### Proving the move landed
|
||||
|
||||
A scan answers `204` whether or not it finds anything, so proof is the journal
|
||||
replayed against both servers' own tables: every `move` row has a row at its
|
||||
`dst`, and **zero** rows still name its `src`. `scripts/media-verify.py` is that
|
||||
replay: it opens each database read-only (there is no `sqlite3` binary on these
|
||||
hosts or inside either container) and exits non-zero while a row is stale.
|
||||
|
||||
Both libraries live in WAL mode, so the tool opens them plain `mode=ro`:
|
||||
`immutable=1` promises sqlite the file never changes, which skips the
|
||||
write-ahead log — a scanner's own uncheckpointed rows then read as missing, and
|
||||
a read beside a running scan can fail as "database disk image is malformed".
|
||||
`immutable` stays only the fallback for a database whose `-shm` cannot be read.
|
||||
|
||||
```bash
|
||||
scripts/media-verify.py --journal /mnt/entropy/media-relayout-music.jsonl \
|
||||
--db-kind jellyfin --ignore-ext jpg,jpeg,png,log,cue,toc,nfo,txt,m3u,accurip
|
||||
```
|
||||
|
||||
"No row at the destination" is not one verdict but four. **never indexed**: the
|
||||
server indexes no file of that extension (its own extension census says so) — a
|
||||
sidecar, not a lost film. **gone**: the file sits at neither src nor dst, because
|
||||
a later pass or the server's own extension sweep renamed it. **phantom**: a row
|
||||
still naming src while the file has left src. **REAL**: no row at dst *and* the
|
||||
file still at src — the only class that is this relayout's own damage. Phantoms
|
||||
are normal for both servers (Plex carries thousands that predate any relayout);
|
||||
re-scan to clear them and never read them as loss.
|
||||
|
||||
`media-verify.py` prints this split itself — `on disk at dst, no row names it`,
|
||||
`REAL — file still at src`, `gone — at neither path` — so nobody hand-stats a
|
||||
thousand paths to find out whether a film is lost or merely unindexed.
|
||||
|
||||
The owning server decides whether a pass can convict anything: Plex's roots are
|
||||
its `sections` rows joined to their location rows (`Movies`, `TVShows`, `Music`,
|
||||
`Audiobooks`, `LiveRecording`) and Jellyfin holds no audiobook library at all, so
|
||||
the audiobook journal verifies with `--db-kind plex` or it proves nothing.
|
||||
|
||||
Three doors that look right and are not: `GET /search?query=` matches metadata
|
||||
titles, never file paths; `?path=` on `/library/sections/N/all` makes Plex's
|
||||
HTTPD close without a single byte; and the same HTTPD drops requests that carry
|
||||
no `Accept` and client-identifier header (see the scan block above for the form
|
||||
that works).
|
||||
|
||||
## TV in the movie library
|
||||
|
||||
A box set that arrives in the movie share stays there until someone moves it:
|
||||
`media-plan.py` refuses to file TV and parks each series in
|
||||
`_TV needs a TV library - <series>`. `scripts/media-tv.py` is the mover. It
|
||||
walks the movie library itself (no census needed), groups episode-shaped names
|
||||
by series, and writes a plan in the executor's own format:
|
||||
|
||||
```bash
|
||||
scripts/media-tv.py --root /mnt/entropy/Movies --library /mnt/entropy/TVShows \
|
||||
--out /mnt/entropy/media-tv-plan.tsv
|
||||
scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv --root /mnt/entropy
|
||||
scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv --root /mnt/entropy \
|
||||
--journal /mnt/entropy/media-relayout-tv.jsonl --apply
|
||||
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout-tv.jsonl \
|
||||
--root /mnt/entropy --apply
|
||||
```
|
||||
|
||||
The shape it writes is the one both matchers read — `Monk (2002)/Season 05/
|
||||
Monk - S05E03 - Mr. Monk and the Big Game.mkv` — keeping the file's own name,
|
||||
because the publisher's spelling beats a re-derivation, with the year taken
|
||||
from the box-set folder. What it refuses it prints and leaves alone:
|
||||
|
||||
- fewer than `--min` (5) files under one series name: one file named `Part 2`
|
||||
is a film, not a series;
|
||||
- an episode whose name states no season number — no invented `Season 01`;
|
||||
- a destination two files both claim: the executor refuses the whole plan
|
||||
before its first rename, and only Frank decides which copy is the keeper.
|
||||
- an episode the destination library already holds at the same byte count. The
|
||||
movie share once carried a second copy of a whole box set — same episode list,
|
||||
every byte count identical. The mover leaves such a series in place and says
|
||||
so; a duplicate is deleted or kept by Frank, never copied into one library by
|
||||
a tool.
|
||||
|
||||
A sidecar that shares an episode's own name (`.srt`, `.nfo`) moves with it.
|
||||
|
||||
Below the threshold the mover prints and leaves, so a lone downloaded episode of
|
||||
a real show stays in the movie library on purpose.
|
||||
Filing one by hand is the same executor on a hand-written plan and the same
|
||||
journal; the trap is the series year. Read it from the file's own name or the
|
||||
folder it owns, never from the title whose bonus folder it happens to be parked
|
||||
in — `The Cleveland Show S03E07` inside `Die Hard (1988)/extras/` is not a 1988
|
||||
show. A name whose marker is `Part 2` is a film and stays put.
|
||||
|
||||
## The TV library's own relayout
|
||||
|
||||
`/mnt/entropy/TVShows` is scene packs rather than a library, and Jellyfin has
|
||||
already matched and numbered most of it. `scripts/media-shows.py` re-files that
|
||||
library into the shape Jellyfin reads, with **tvdb as the name authority**:
|
||||
|
||||
<Series> (<year>) [tvdbid-NNNN]/Season NN/
|
||||
<Series> (<year>) - SxxEyy - <Episode Title>.ext
|
||||
|
||||
It never guesses: every series name, episode number and title it writes traces to
|
||||
an authority, and what has none is parked for a human to read.
|
||||
|
||||
```bash
|
||||
# 1. The authority: Jellyfin's own matched rows — each carries the series' TMDb
|
||||
# and Tvdb ids and the title the row holds (the DB opens read-only).
|
||||
python3 scripts/media-census.py jellyfin --root /mnt/entropy/TVShows \
|
||||
--out /tmp/tv_titles.tsv
|
||||
# 2. What is on disk (stat only), and the runtimes the film arm needs.
|
||||
python3 scripts/media-census.py enumerate --root /mnt/entropy/TVShows \
|
||||
--out /tmp/tv_enum.tsv
|
||||
python3 scripts/media-census.py probe --root /mnt/entropy/TVShows \
|
||||
--enum /tmp/tv_enum.tsv --jobs 6 --out /tmp/tv_probed.tsv
|
||||
# The probe is the slow step: ffprobe reads both ends of every file. So re-use
|
||||
# the last one — a duration belongs to the file, not to its name: re-key the
|
||||
# previous census through that pass's own journal (`src` -> `dst`) instead of
|
||||
# seeking the array again. It is also the only column the film arm reads, and
|
||||
# only for a file with no marker and no provider row (4,794 of 58,551 today,
|
||||
# all of them covered by a re-key that costs seconds, not an hour).
|
||||
# 3. The plan. --tmdb needs TMDB_API_KEY — the v3 key in ~/.config/secrets/tmdb,
|
||||
# which travels as the `api_key` query parameter. Only a v4 request token is
|
||||
# accepted as a Bearer header, and the wrong shape answers 401 on EVERY call,
|
||||
# which the summary line flags as NOT provider-complete. --tvdb reads its
|
||||
# `api_key=` line from ~/.config/secrets/tvdb, never argv. Both write the SAME
|
||||
# cache file, so the plan is reproducible offline afterwards: run it once to
|
||||
# fill the cache, and quote the second run.
|
||||
# Paths are only strings to a dry run, so this step can run on the workstation
|
||||
# against scp'd TSVs — which is where the two provider secrets live.
|
||||
python3 scripts/media-shows.py --root /mnt/entropy/TVShows --enum /tmp/tv_enum.tsv \
|
||||
--titles /tmp/tv_titles.tsv --probed /tmp/tv_probed.tsv \
|
||||
--tmdb-cache /mnt/entropy/media-tv-names.json --tmdb --tvdb \
|
||||
--park /mnt/entropy/media-park/TVShows \
|
||||
--out /mnt/entropy/media-relayout-shows2-plan.tsv \
|
||||
--exceptions /mnt/entropy/media-relayout-shows2-exceptions.txt
|
||||
# 4. Read the exceptions file FIRST — it is the review, not a log. Same executor
|
||||
# as above, but on this arm's OWN journal, NAMED PER PASS: the journal of an
|
||||
# applied pass is its unwind record, so a second pass that reuses the name
|
||||
# destroys the only map back.
|
||||
scripts/media-apply.py apply --plan /mnt/entropy/media-relayout-shows2-plan.tsv \
|
||||
--root /mnt/entropy --journal /mnt/entropy/media-relayout-shows2.jsonl --apply
|
||||
```
|
||||
|
||||
Evidence, strongest first — the order the planner believes things:
|
||||
|
||||
- the **episode marker the file's own name states** (`S01E02-E03`, `1x04`);
|
||||
- the **`Season NN` folder it sits in and the number its name leads with inside
|
||||
it**: `Season 19/01 Charles Barkley, Nirvana.avi` is S19E01;
|
||||
- the **provider row**: which episode the file IS, and the numbering of a file
|
||||
whose own name says nothing;
|
||||
- the **row's tvdb id, else its TMDb id carried through TMDb's external ids, else
|
||||
an exact tvdb name search**: that id names the SERIES FOLDER and supplies the
|
||||
episode title, because a name the server read off a folder is not a show name.
|
||||
The folder CARRIES the id: `[tvdbid-NNNN]` is what Jellyfin's Tvdb agent and
|
||||
Plex's scanner both read, so a show whose title is ambiguous resolves by number
|
||||
instead of by a human guessing which folder is which. The id rides the folder
|
||||
only — a file name gets 180 characters and the episode title needs the room.
|
||||
- a **bonus folder** anywhere above the file, or Jellyfin's `ExtraType` → the
|
||||
series' canonical extra folder;
|
||||
- a runtime over `SHORT_MAX` makes it film-shaped, and it leaves for `--movies` only
|
||||
when the movie provider's title is a run of words the FILE itself states and its
|
||||
year agrees with the file's: the query is only what `title_of` salvaged from a
|
||||
release name, and the first hit for a short one is a near miss — a 1963 Bond film
|
||||
for the word "russia", *Special Edition I* for a file saying *Edition III*. Else it
|
||||
parks.
|
||||
|
||||
Four refusals carry the design: what the planner will NOT do is the point of it.
|
||||
|
||||
- **Folders resolve a series; they never invent one.** A folder's words may
|
||||
resolve a file to a series a provider has already named — `(CBB) Tengen Toppa
|
||||
Gurren Lagann - The Complete Series (BD 1080p)` names the registered *Tengen
|
||||
Toppa Gurren Lagann* — but a bonus subfolder is not a show. An early run made
|
||||
`…/03. Extra/01. NCOP/NCOP 01.mkv` into a root series called `01 NCOP`.
|
||||
- **A scraped row does not outrank the filesystem.** Jellyfin numbers
|
||||
multi-season packs badly: every file under `SNL 16-20/Season 16..19` carries
|
||||
season 16 episode 20, and one Gundam row says season 2004. The folder, then
|
||||
the file's own leading number, win.
|
||||
- **A season that is a YEAR is the date matcher, not a season.** Jellyfin files
|
||||
specials by airdate, so a row can arrive saying `season 2002`; that is not a
|
||||
season and it never becomes a folder. The one authority on such a row is tvdb's
|
||||
episode list read backwards: the title the row carries names the (season,
|
||||
episode) tvdb gives it. A title tvdb gives to two episodes names nothing, and
|
||||
the file parks with the reason it always carried. `Season 00` IS a legal
|
||||
season — specials live there — so a row that really says 0 files there instead
|
||||
of parking.
|
||||
- **A season tvdb's own list contradicts gets no tvdb title.** An anime pack
|
||||
numbered by its release group books `S05E01` in a season tvdb does not hold, or
|
||||
books more episodes there than tvdb lists: the slot its number points at is a
|
||||
different episode, and dressing it in that episode's title is a confident lie.
|
||||
The file still renames to the standard shape; the title does not ride, and the
|
||||
series is listed under *seasons whose numbering tvdb contradicts* in the
|
||||
exceptions. A season holding FEWER files than tvdb lists stays plausible — a
|
||||
partial pack is the normal shape. Neither reading holds for a list tvdb
|
||||
cut off at its own page limit: the default episode query answers at most
|
||||
500 rows and states no total, and One Piece, Crayon Shin-chan and
|
||||
Detective Conan each answer exactly 500 while holding over a thousand —
|
||||
a page bounds nothing above itself, so it contradicts nothing.
|
||||
|
||||
**A provider name is evidence only when an id backs it.** Jellyfin names a Series
|
||||
after the folder it scanned, so a library filed by release folders ends up with
|
||||
one Series item per box set — `24 (2001) Season 1`, `Modern Family S05` — and
|
||||
copying those names verbatim launders folder junk into the contract. The planner
|
||||
resolves each series to its tvdb id and writes tvdb's name and year; the merged
|
||||
identity is also what turns a box set split across five release folders into ONE
|
||||
folder, and what makes two files claiming one episode MEET in the duplicate rule.
|
||||
A series no id resolves keeps the provider's words and is listed under *no tvdb
|
||||
identity* in the exceptions — that list, not a rename, is the review.
|
||||
|
||||
**The review pile lives outside the library.** Files parked inside
|
||||
`/mnt/entropy/TVShows` were indexed as one show named `_unsorted`, and that
|
||||
show's rows then looked like evidence. So a row whose series is the hidden
|
||||
top-level folder the file itself sits in is dropped — and the folder is matched
|
||||
with its leading underscore stripped, because the server *displays* `_unsorted` as
|
||||
`unsorted`: compare the spelled names and the pile survives the relayout as a
|
||||
visible series folder that real files get renamed into. `--park` outside the root
|
||||
makes the next plan re-file the pile instead of skipping it. Inside a hidden
|
||||
top-level folder a file's own words may MATCH a registered show, never invent one.
|
||||
|
||||
**The operator's id map is the door for a name tvdb cannot be searched by.**
|
||||
`scripts/media-shows.py` reads `scripts/media-shows-tvdb-ids.tsv` by default: one
|
||||
line per pack folder, `folder <TAB> tvdb id <TAB> note`. The file exists because
|
||||
tvdb titles a record in the show's own language and the English name rides in
|
||||
*aliases* the name search never reads: `ブラッククローバー` (331753) answers to no
|
||||
spelling of "Black Clover", and the bare alias `One Piece` belongs to 392276, the
|
||||
2023 live action, not to the 1999 record (81797) the pack here is. The map
|
||||
outranks a provider id — a box set's provider id is the server's own guess at a
|
||||
folder name — and it merges the release folders that are one show. Everything it
|
||||
touched is written to the exceptions: every folder it named, every row that
|
||||
matched no folder, every disagreement with a provider id, every id tvdb never
|
||||
answered. A line whose id is not a number stops the plan, because a typo renames
|
||||
a whole pack to the wrong show. Only folders the search CANNOT reach belong in
|
||||
it; a row for a folder the search would resolve hides a live lookup behind a
|
||||
number nobody re-checks.
|
||||
|
||||
`scripts/media-shows-tvdb.py` makes the sheet the map is built from. It reads a
|
||||
plan's `mkdir` rows, asks tvdb about every destination folder carrying no
|
||||
`[tvdbid-`, and writes one row per folder with the verdict that says how much was
|
||||
guessed to get there: `EXACT` (tvdb's own spelling), `PUNCT` (equal only after
|
||||
release junk is stripped from the folder's words), `NEAR` (tvdb answered
|
||||
something else — the best hit is printed for a human and nothing applies it),
|
||||
`NONE`. Acceptance never moved across the passes that built this: a hit counts
|
||||
only when tvdb's name equals the query with every non-alphanumeric dropped from
|
||||
both sides, and the record's year still has to fall inside any year span the
|
||||
folder names. That gate is what caught three confident picks — Fairy Tail's 2024
|
||||
sequel (410031), the 2026 Hokuto no Ken remake (465216) and `Mugyutto! Black
|
||||
Clover` (378762) offered for Black Clover. It is why the map is approved by hand
|
||||
and the sheet is never fed to the planner.
|
||||
|
||||
## The music and audiobook arms
|
||||
|
||||
Each is a page of its own, because each reads a different authority and refuses
|
||||
for a different reason: [[Media Music Relayout]] files by Jellyfin's track tags,
|
||||
[[Media Audiobook Relayout]] by Plex's chapter rows. Both share this page's
|
||||
executor, the one-journal-per-arm law, the sweep and the rollback.
|
||||
|
||||
## Laws
|
||||
|
||||
- **A moved file is a 404 in [[Plex]] until its section is rescanned.** The
|
||||
database keeps the release-folder path, so playback fails `Couldn't find the
|
||||
file to stream` while the scanner works through the section. Never read such a
|
||||
404 as a lost file: the file is where the journal says it is.
|
||||
- **Plex's matched title outranks the filename**, and a Plex year of `0` means
|
||||
*indexed but never matched* — trusting it named folders `Title (0)`, which is
|
||||
a Jellyfin item with no year, no poster and no identity. A year that is not a
|
||||
plausible year is no information.
|
||||
- **`<>:"/\|?*` never survive a name.** Plex's own title for *V/H/S/2* is
|
||||
`V/H/S/2`; kept verbatim it became `Movies/V/H/S/2 (2013)/` — a tree five
|
||||
levels deep that neither matcher reads as a movie.
|
||||
- **A folder inside a movie's folder is bonus territory even when its name is
|
||||
misspelled.** `Saboteur (1942)/Exrtas/Saboteur - A Closer Look` is a typo for
|
||||
`Extras`; the vocabulary rule never fired, the featurette became a movie
|
||||
folder, and its words then out-scored the real film and drew that box set's
|
||||
other extras into it. The rule that replaced it needs no metadata: the folder
|
||||
holding the longest film owns the shorter video filed under it, unless that
|
||||
file's own name carries a different year — which is another film speaking.
|
||||
- **Matching words are split on everything that is not a letter or a digit.**
|
||||
Splitting on separators but not on `/` left tokens like `(1960)/extras` that
|
||||
no other name could ever agree with, so a film could not be found by the
|
||||
folder holding it, and extras drifted into a neighbour. A bare number counts
|
||||
only when it could be a release year, so `1080` and `2160` never pose as
|
||||
agreement between two names.
|
||||
- **Compare two media libraries with the series engine, never a path
|
||||
substring.** Filtering paths for `monk` also returns *Monkey Dust*, and
|
||||
letting whichever file a walk happens to reach first stand for an episode
|
||||
paired two libraries into a fake duplicate — the same sloppiness pointed the
|
||||
other way deletes a file that is unique. The comparison is `media-titles.ep_of`
|
||||
on both sides: series, season, episode, bytes, and a hash before anything is
|
||||
trashed.
|
||||
- **A deletion is staged on the item's own filesystem.** The trash rule names
|
||||
`~/.AGENTTRASH`, but `/home` and `/mnt/entropy` are different devices: a 31 GiB
|
||||
stage there is a copy rather than a rename, it has to fit in the 187 GiB of
|
||||
free space, and `rename(2)` refuses it anyway. Trash goes beside the item —
|
||||
`/mnt/entropy/.AGENTTRASH/<date>-<topic>/` — so it costs one inode change and
|
||||
rolls back exactly like the move that made it.
|
||||
- **A file whose name carries no words is bonus material, not a folder.**
|
||||
`.Extra 1 ( 1H 27min multisub ).mkv` parses to `1`; a movie folder named `1`
|
||||
is not a movie. It becomes a review line telling a human to name the file.
|
||||
- **The executor renames and nothing else.** `rename(2)` cannot cross a
|
||||
filesystem, so a plan that would is refused rather than silently copying tens
|
||||
of terabytes.
|
||||
- **The journal is a write-ahead log: the intent is fsynced before the rename.**
|
||||
An interrupted run therefore leaves only two states — an intent whose move
|
||||
happened, and an intent whose move never ran — and the next run finishes the
|
||||
job without moving anything twice. A file that moved with no journal line did
|
||||
not move through this tool, and the tool refuses to guess on its behalf.
|
||||
- **The journal lives BESIDE the library root, never inside it** (default:
|
||||
`media-relayout.jsonl` next to the root): a log inside the library is a file
|
||||
Jellyfin indexes.
|
||||
- **Refusals are all-or-nothing.** An occupied target, a vanished source or a
|
||||
duplicate destination stops the run *before the first rename*, so the operator
|
||||
fixes one thing and re-runs instead of untangling a half-applied plan.
|
||||
`--force` relaxes only the opinion guards; it never permits overwriting.
|
||||
- **A sweep deletes only what the journal proves the run emptied.** A folder is
|
||||
eligible when a journalled file moved out of it or out of something under it,
|
||||
it now holds nothing but empty directories, no journalled destination lives
|
||||
beneath it, and it is not a folder the run created. A folder nobody moved —
|
||||
someone's placeholder — cannot be selected at all, and a symlink anywhere
|
||||
inside disqualifies the subtree, because the thing behind a link is not this
|
||||
tool's to delete. Every removal is journalled as `rmdir` and `rollback`
|
||||
recreates the folder, so the sweep is undone by the same command as the moves.
|
||||
Proof carries downward: inside a folder the journal proves emptied, every
|
||||
empty directory goes with it. A release's `Screens/` or `Sample/` is never a
|
||||
move source itself, so it can hold an emptied parent until the next sweep.
|
||||
A failed `rmdir` is not a removal: only a folder that is really gone leaves
|
||||
scope, so the next sweep finishes what a permission problem stopped.
|
||||
- **An episode-shaped name marks a SERIES, not a film — and the series is the
|
||||
name BEFORE the marker.** `Monk - S05E03 - Mr. Monk and the Big Game` is series
|
||||
`Monk`. Keyed on the whole title instead, every episode became a group of one,
|
||||
never reaching the five-episode size that calls something a series — a whole
|
||||
box set filed as one folder per episode. The threshold still earns its keep:
|
||||
a lone film named `Part 2` stays a film. Episodes of a series are parked
|
||||
in `_TV needs a TV library - <series>` for a human to move to a TV source —
|
||||
the movie tool never files TV.
|
||||
- **One journal is one undo.** The TV move runs on its own journal, and with
|
||||
`--root` set to the share that holds BOTH libraries: the executor refuses any
|
||||
path outside `--root`, and a journal shared with the movie relayout would
|
||||
undo both runs when only one of them is wanted.
|
||||
|
||||
- **Sidecar metadata does not follow a rename, because it lives on another
|
||||
filesystem.** The media servers write `.nfo` and artwork through the
|
||||
[[Media Metadata View]] (NVMe upper layer); this tool moves bytes on the raw
|
||||
array path, so where that view is mounted a sweep or a move must be followed by
|
||||
`scripts/media-view.sh reap --apply` — stranded sidecars become ghost folders
|
||||
that both servers browse and report as content. Whether this host mounts the
|
||||
view at all is stated on that page; `media-apply.py` stays silent when it does not.
|
||||
- **Nothing deletes through the metadata view.** A delete there writes an overlayfs
|
||||
whiteout in the upper layer, which hides the array's copy without removing it;
|
||||
the file returns when the view is unmounted. Deletions in this pipeline happen
|
||||
on the raw path only, which is what the journal can undo.
|
||||
- **The music album key is case-folded; the folder name is not.** `Master Of
|
||||
Puppets` and `Master of Puppets` are one album to the matcher, so the spelling
|
||||
that reaches the folder is the one most tracks carry. Taking it from whichever
|
||||
row sorted first made the folder name depend on row order, and re-planning an
|
||||
already-filed library renamed albums that were already in place.
|
||||
- **An occupied path is never a destination**, not even when its occupant is
|
||||
scheduled to leave: the executor refuses an occupied target, so a plan that
|
||||
aims at one fails after the renames that were supposed to clear it. A file
|
||||
standing where the contract puts it is a keep, and a file already carrying a
|
||||
collision suffix keeps that suffix — otherwise the plan is a shuffle, not a
|
||||
fixed point, and a re-run moves albums that are already right.
|
||||
- **`media-census.py music` reads the `BaseItemProviders` table, not the item's
|
||||
JSON.** This Jellyfin build carries an empty `ProviderIds` inside
|
||||
`BaseItems.Data` for albums, so a census that parses the JSON finds no
|
||||
MusicBrainz or Spotify id on any album and reads as "the library is untagged".
|
||||
|
||||
## Rollback
|
||||
|
||||
One command, from the journal of the run:
|
||||
`media-apply.py rollback --journal …/media-relayout.jsonl --apply`. It recreates
|
||||
any folder a sweep removed, restores every source path, removes only the folders
|
||||
that run created *and that are now empty*, and refuses everything if any moved
|
||||
file cannot be accounted for. A rolled-back journal refuses to roll back twice.
|
||||
|
||||
**Roll a chain back newest journal first, or the tool refuses.** Each pass
|
||||
re-files what an earlier one moved, so a journal is a clean inverse only until
|
||||
something downstream touches its files; `ls -l /mnt/entropy/media-relayout*.jsonl`
|
||||
is the order. Refusing is the tool working: on 2026-10-31 the movie journal could
|
||||
not restore files the later TV passes had re-filed, and the stray journal could
|
||||
not restore files Jellyfin's extension sweep had renamed. Roll back what still
|
||||
inverts and leave the rest as the current state — never force it.
|
||||
|
||||
## See Also
|
||||
|
||||
[[Jellyfin]], [[Plex]], [[Media Metadata View]], [[Entropy Volume]], [[Testing Policy]], [[Storage Policy]]
|
||||
|
||||
## Citations
|
||||
|
||||
- Jellyfin Movie documentation — folder/file naming, extras subfolders,
|
||||
`Movie (2013) - 216p.mkv` version and `- 4K.mkv` handling:
|
||||
<https://jellyfin.org/docs/general/server/media/movies>
|
||||
@@ -4,7 +4,7 @@ description: "Plex Media Server is a **containerized media server on Dankserver*
|
||||
tags:
|
||||
- dankserver
|
||||
- plex
|
||||
timestamp: 2026-10-21
|
||||
timestamp: 2026-10-30
|
||||
---
|
||||
|
||||
# Plex
|
||||
@@ -13,7 +13,8 @@ Plex Media Server is a **containerized media server on Dankserver**: a podman
|
||||
quadlet (`plex.service`) under the systemd **user** generator, generated from
|
||||
`dankserver/plex/quadlet/plex.container`. It shares the host with a second live
|
||||
media server, [[Jellyfin]] (`jellyfin.service`, listening on 8096); both serve
|
||||
from the same read-only Entropy Volume mount.
|
||||
from the same tree — the [[Media Metadata View]] overlay — so the Kodi-format
|
||||
sidecars Jellyfin writes are the files Plex reads, with no copy and no sync.
|
||||
|
||||
**Host:** Dankserver — **Ports:** 32400/tcp (PMS web + API), 32400 is the
|
||||
only port PMS needs for client streaming; LAN clients connect directly,
|
||||
@@ -26,7 +27,7 @@ public traffic can be fronted by Caddy like every other vhost.
|
||||
| Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
|
||||
| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
|
||||
| Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
|
||||
| Media | `/mnt/entropy` mounted at the same absolute path **read-only** — every library path in the PMS database stayed valid, zero re-scan |
|
||||
| Media | git: `/var/lib/media-meta/view` (the [[Media Metadata View]] overlay) bound at the same absolute path `/mnt/entropy` **read-only**; live today: `/mnt/entropy` itself, read-only — the view has never been installed. Either way every library path in the PMS database stays valid, zero re-scan |
|
||||
| Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
|
||||
| Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration |
|
||||
| Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` |
|
||||
@@ -38,7 +39,9 @@ public traffic can be fronted by Caddy like every other vhost.
|
||||
|
||||
The migration moved zero bytes: the container bind-mounts the native data
|
||||
root where it already lived, and the media archive re-mounts at
|
||||
`/mnt/entropy` inside the container, so every absolute library path in the
|
||||
`/mnt/entropy` inside the container — only the HOST side of that bind has moved
|
||||
(the array today; the overlay view in git has never been deployed), so every
|
||||
absolute library path in the
|
||||
PMS database keeps resolving. The one metadata change: rootless podman
|
||||
cannot map the real host uid 997, so the fleet model applies — container
|
||||
root runs as the `dank` user, and the deployer adopts the data root with
|
||||
@@ -84,3 +87,45 @@ root runs as the `dank` user, and the deployer adopts the data root with
|
||||
while the HTTP layer finishes loading; the first boot of a years-old
|
||||
library took ~10 minutes (2026-09 cutover). The deployer gate now allows
|
||||
600 s before failing; later boots are much faster (warm page cache).
|
||||
|
||||
## Triggering a scan
|
||||
|
||||
A rename on disk is invisible until the section is rescanned, and a moved file is
|
||||
a `Couldn't find the file to stream` 404 until that scan passes it. Run this from
|
||||
the host shell through `podman exec`, never from a neighbour container: PMS's
|
||||
HTTPD closes without a single byte on any request that arrives from another
|
||||
container's netns (10.89.x.x reads as "empty reply", not as a 401). The token
|
||||
rides stdin, never argv, where any process on the host could read it out of
|
||||
`/proc/<pid>/cmdline`; it lives in `Preferences.xml` inside the RW data bind.
|
||||
|
||||
```bash
|
||||
umask 077
|
||||
podman exec plex sh -c "sed -n 's/.*PlexOnlineToken=\"\\([^\"]*\\)\".*/\\1/p' \
|
||||
'/config/Library/Application Support/Plex Media Server/Preferences.xml'" >/tmp/px.tok
|
||||
cat >/tmp/pxscan.sh <<'EOS'
|
||||
read -r T
|
||||
cfg=$(mktemp)
|
||||
mk() { { printf 'header = "X-Plex-Token: %s"\nheader = "Accept: application/json"\n' "$T"
|
||||
printf 'url = "%s"\n' "$1"; } > "$cfg"; }
|
||||
mk "http://127.0.0.1:32400/library/sections"
|
||||
for k in $(curl -s -K "$cfg" | grep -o '"key":"[0-9]*"' | cut -d\" -f4); do
|
||||
mk "http://127.0.0.1:32400/library/sections/$k/refresh"
|
||||
printf 'request = "POST"\n' >> "$cfg"
|
||||
echo "plex section $k: $(curl -s -o /dev/null -w '%{http_code}' -K "$cfg")"
|
||||
done
|
||||
shred -u "$cfg"
|
||||
EOS
|
||||
podman cp /tmp/pxscan.sh plex:/tmp/pxscan.sh
|
||||
podman exec -i plex sh /tmp/pxscan.sh </tmp/px.tok
|
||||
shred -u /tmp/px.tok
|
||||
```
|
||||
|
||||
Every section answers `204` — or `200` with a Directory body when
|
||||
`Accept: application/json` is sent. Inside the quoted `printf` format a
|
||||
backslash-newline is LITERAL, not a continuation, so `mk` writes two `printf`s:
|
||||
one glued line and one stray backslash is a config file curl rejects, and the
|
||||
failure prints as `option -K: is unknown`, never as a bad config line.
|
||||
|
||||
Proof that a rename landed is not a scan response: `scripts/media-verify.py`
|
||||
replays the run's journal against the database — the only read that
|
||||
distinguishes "not scanned yet" from "gone"; see [[Media Relayout]].
|
||||
@@ -74,11 +74,19 @@ published ports — node `:9100`, cadvisor `:8080`, gpu `:9401` all answer there
|
||||
— while the host's own LAN IP is refused (`192.168.0.143:8080`, `:9401`), so
|
||||
every same-host Prometheus target uses the gateway and only cross-host targets
|
||||
(dankserver) use plain LAN. Whether LAN works at all is network-specific: the
|
||||
relay at `192.168.0.143:18093` IS reached from alertmanager's network, with
|
||||
iptables INPUT admitting loopback, the host itself and the podman bridge subnet
|
||||
`10.89.0.0/16` before the DROP (`scripts/deploy-remote.sh:255-262`). Never
|
||||
point a scrape target at a host LAN IP from its own host on the strength of
|
||||
one working example; dial the gateway and probe.
|
||||
relay at `192.168.0.143:18093` is reached from alertmanager's network only
|
||||
while `install_signal_relay`'s INPUT rules are in force: loopback, the host
|
||||
itself and the podman bridge subnet `10.89.0.0/16` accepted **above** a
|
||||
trailing blanket DROP. The law the poisoned host taught (2026-10-21):
|
||||
`iptables -I` prepends, so inserting the DROP last lands it first — the old
|
||||
install order dropped every packet (159 on the DROP, 0 on the ACCEPTs), the
|
||||
deployer's health gate failed beside a LISTENing socket, and the drain
|
||||
aborted twice before `nft list ruleset` named the cause. The install now
|
||||
rebuilds the tail (delete the DROP wherever it sits, ACCEPTs at the head,
|
||||
DROP appended); verify any change to it with the sandboxed bats case, not
|
||||
with `-C` greps — `-C` answers existence, never order. Never point a scrape
|
||||
target at a host LAN IP from its own host on the strength of one working
|
||||
example; dial the gateway and probe.
|
||||
|
||||
## Deployment
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ timestamp: 2026-09-03
|
||||
|
||||
## Server Overview
|
||||
|
||||
Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[frank-dragoon]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon]] (halogen-flash-server 0.16.2 on the iGPU, port 7770) and [[frank-zealot]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
|
||||
Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[frank-dragoon]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon]] (halogen-flash-server 0.18.20 on the iGPU, port 7770) and [[frank-zealot]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
|
||||
## Hardware Specifications
|
||||
|
||||
### CPU
|
||||
@@ -105,7 +105,7 @@ cable between claw and server is required for 10G.
|
||||
|
||||
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
|
||||
- The GPU budget is the GTT aperture: ~114 GiB (`mem_info_gtt_total` in `/sys/class/drm/card1/device/` = 122,402,963,456 B; `card1` is the amdgpu iGPU, `PCI_ID=1002:1586`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
|
||||
- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.16.2** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon]]) — and `frank-zealot` on 7771 ([[frank-zealot]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
|
||||
- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.18.20** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon]]) — and `frank-zealot` on 7771 ([[frank-zealot]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
|
||||
- The gufo-era note that the active runtime links against the host ROCm (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH belongs to that retired source build. The serving runtimes do not depend on the host ROCm that way: halogen runs inside its digest-pinned image, and `frank-zealot` is a CUDA build on the 3090. The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) was never in the serving path either.
|
||||
- The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
|
||||
- 424 GB free on `/` (`df -h`, 2026-10-18) is ample for model storage.
|
||||
|
||||
@@ -0,0 +1,287 @@
|
||||
---
|
||||
type: "howto"
|
||||
title: "Radiance Architecture Plugins — declaring a model"
|
||||
description: "Writing a radiance architecture plugin: declaring weights, buffers, KV groups and ops, then issuing the step."
|
||||
tags:
|
||||
- llm
|
||||
- radiance
|
||||
- plugins
|
||||
- howto
|
||||
timestamp: 2026-10-30
|
||||
---
|
||||
|
||||
# Radiance architecture plugins
|
||||
|
||||
An architecture plugin declares a model in two phases: `rad_arch_declare` names every weight,
|
||||
buffer, KV group and op **once**, against the model's metadata; `rad_arch_step` issues the ops for
|
||||
one step. The engine picks the plugin by the exact `(rad_arch_id, rad_arch_quant)` pair — no
|
||||
fallback — so one family at two weight formats is two `.so` files, and branching on a quant
|
||||
descriptor inside one plugin is the anti-pattern the split exists to prevent.
|
||||
|
||||
The in-tree plugins are the templates: `arch/llama_fp8/llama_fp8.cpp` (the smallest complete
|
||||
decoder), `arch/gemma3`, `arch/qwen35_bf16`, and the header-only blocks in `arch/common/`
|
||||
(`rad_block_attn*.h`, `rad_block_mlp*.h`, `rad_block_moe_fp8.h`, `rad_block_gdn*.h`,
|
||||
`rad_block_dflash2.h`, `rad_block_mtp*.h`, `rad_block_vit.h`, `rad_block_ple.h`,
|
||||
`rad_block_glm_*.h`, `rad_qsa.h`, …). Include those by `<arch/rad_arch.h>` — they are installed
|
||||
beside the ABI and the in-tree plugins include them by that spelling.
|
||||
|
||||
## Exports
|
||||
|
||||
```c
|
||||
const char* rad_arch_id(void); /* "llama", "qwen35", ... */
|
||||
const char* rad_arch_quant(void); /* "" unquantised, else the descriptor served */
|
||||
int rad_arch_serves_imports(void); /* optional: 0 declines an imported quantisation */
|
||||
int rad_arch_probe(const RadModelMeta* meta, RadArchProbe* out); /* optional */
|
||||
int rad_arch_declare(RadBuilder* b, const RadModelMeta* meta, const RadBuildCtx* ctx);
|
||||
void rad_arch_step(struct RadCtx* c, const struct RadBatch* batch);
|
||||
const RadChatFormat* rad_arch_chat_format(const RadModelMeta* meta); /* optional */
|
||||
const RadAudioFormat* rad_arch_audio_format(const RadModelMeta* meta); /* optional */
|
||||
```
|
||||
|
||||
Everything but declare/step is optional. `RAD_ARCH_PLUGIN(NS, ID, QUANT, VERSION, DESC)` in
|
||||
`arch/common/rad_arch.h` emits the whole export block, including `rad_plugin_info` (kind
|
||||
`RAD_PLUGIN_ARCH`). The plugin's **name must be unique**; the architecture id may repeat.
|
||||
|
||||
`rad_arch_serves_imports` exists because a checkpoint whose quantisation was imported weight by
|
||||
weight (Quark, ParoQuant, GPTQ, NVFP4) names its producing tool, which no plugin claims: the family's
|
||||
one quantised plugin serves it, and two are refused by name. A second quantised plugin that reads
|
||||
only its own containers returns 0.
|
||||
|
||||
## Declare: the metadata
|
||||
|
||||
`RadModelMeta` is what the loader read from the container's header before any plugin ran: `arch_id`,
|
||||
`name`, `quant`, `n_layers`, `n_embd`, `n_head`, `n_head_kv`, **`head_dim`**, `n_ff`, `n_vocab`,
|
||||
`n_ctx_train`, expert counts, `rms_eps`, `rope_theta`, and a free-form `kv_key`/`kv_val` table for
|
||||
everything else (`rad_meta_geti/getf/gets`). Read dimensions from here — one plugin covers every size
|
||||
in a family.
|
||||
|
||||
**`head_dim` is read, never derived.** On the Qwen hybrids it is 256 while `hidden / heads` is 213;
|
||||
a derived value silently breaks projection widths, KV page size and the attention kernel's band.
|
||||
|
||||
`RadBuildCtx` is the deployment the plugin must declare against:
|
||||
|
||||
| field | meaning |
|
||||
|---|---|
|
||||
| `rank`, `world_size` | declare runs once per rank; **dimensions are already divided** — the core inserts no collectives |
|
||||
| `max_tok`, `max_seqs`, `max_ctx`, `max_spec` | the step bounds: buffer sizing, kernel bands, the scheduler's window |
|
||||
| `max_mixed_tok` | the most a multi-sequence step may carry; a wider step is one sequence (prefill span) |
|
||||
| `tp_wire_lossy`, `tp_wire_min_bytes` | invert into `all_reduce`'s `exact`; below the floor a message must go exact |
|
||||
| `kv_dtype`, `kv_quant` | the attention cache width and, under fp8, its narrow layout (`RAD_KVQ_V4`, `RAD_KVQ_K6V4`) |
|
||||
| `attn_prefill_fp8` | `RAD_ATTN_FP8_*` bits to pass as `attn_paged`'s `prefill_fp8` |
|
||||
| `kv_resident_blocks`, `kv_stream_ok` | KV streaming: the ring depth, and permission to declare the streamed-only ops |
|
||||
| `state_dtype` | `RAD_F16` when `--state-cache-dtype f16` is served by this plugin |
|
||||
| `is_draft`, `scope` | a drafter/encoder declare; `scope` prefixes declared names (`draft`, `vision`, `audio`) |
|
||||
| `shape_probe` | a **sizing** declare at a smaller `max_tok`; must not touch state the real declare wrote |
|
||||
| `max_enc_patches`, `max_audio_frames` | the encoder pass sizes |
|
||||
| `max_out_rows` | logits rows the KL mode needs (0 = the plugin's own rule) |
|
||||
| `cvec` | the loaded control vector, or null |
|
||||
| `tp_attn_rows` | row-split prefill attention requested |
|
||||
| `draft_coupling`, `draft_tree` | declare the coupled/tree forms |
|
||||
| `weight_prefetch_off` | declare no weight prefetch beside collectives |
|
||||
|
||||
## Declare: weights
|
||||
|
||||
```c
|
||||
static const RadWeightDecl d = {
|
||||
.dtype = RAD_BF16, .rank = 2, .shape = {n_embd, n_ff},
|
||||
.access = RAD_ACCESS_PER_TOKEN, .shard = RAD_SHARD_ROW,
|
||||
.group = grp_layer(l), .optional = 0,
|
||||
.n_row_parts = 0, .source = nullptr, .planes = nullptr,
|
||||
};
|
||||
rad_weight w = rad_decl_weight(b, "blk.0.mlp.gate.weight", &d);
|
||||
```
|
||||
|
||||
- **`access`** tells the placement planner how the weight is read: `PER_TOKEN` (dense),
|
||||
`PER_REQUEST` (an encoder tower, read once per request), `CONDITIONAL` (routed experts),
|
||||
`RARE`, `VOCAB` (lm_head/embedding: read every token but huge, and one is a gather).
|
||||
- **`shard`**: `NONE` (replicated), `ROW` (split dim 0), `COL` (split dim 1, needs a following
|
||||
all-reduce). The plugin says; the core does not infer. Uneven splits use
|
||||
`rad_weight_shard_span`/`_parts` (a width that does not divide into whole 128-column blocks; the
|
||||
delta net's `[q|k|v|z]` of unequal part widths).
|
||||
- **`group`** is the unit of movement (`layer`, `expert`, `slot`); an expert's gate/up/down share one,
|
||||
because the unit of transfer is an expert, not a tensor.
|
||||
- **`row_parts`** describes what a `SHARD_ROW` weight stacks along dim 0 (`[gate|up]`,
|
||||
`[q|k|v|z]`). **A tensor-parallel rank owns a slice of each stacked projection**, never a
|
||||
contiguous slice of the stack; the parts cannot be inferred from a fused container tensor, and a
|
||||
contiguous slice produces fluent wrong text.
|
||||
- **`source`/`planes`** view an existing stored weight: a block-fp8 linear's `codes,scale` are one
|
||||
logical weight taken by a GEMM as two operands. Null `source` is the weight of its own name; null
|
||||
`planes` takes every plane.
|
||||
- A dtype other than the plane's is legal only as an exact widening (a bf16 norm read as f32).
|
||||
|
||||
The **name map** (`rad_decl_name_map`) is the plugin's map from checkpoint tensor names to declared
|
||||
weight names, and it is the only place convert-time fusions live: `COPY`, `CONCAT`, `WNORM` (a
|
||||
weight-normalised pair folded to the plain weight), `ROWS`, `TRANSPOSE`, `EMA` (an EnCodec codebook),
|
||||
`TAP`, `COLS`. `rad-convert` reads this map from the plugin, so it needs no knowledge of the model.
|
||||
`RAD_MAP_COPY` with several sources means "first present wins" (a tied lm_head); `CONCAT` requires
|
||||
all. Ask for a weight's encoding with `rad_weight_encoding` **after** its map entry when the graph
|
||||
depends on the format (fp8 GEMM vs plain, rotated expert kernels).
|
||||
|
||||
## Declare: buffers and KV groups
|
||||
|
||||
`rad_decl_buffer` names every activation: `RAD_BUF_TRANSIENT` (liveness-analysed, may share bytes),
|
||||
`RAD_BUF_PERSIST` (across steps: a calibration accumulator, a routing histogram), or
|
||||
`RAD_BUF_DERIVED` (computed once per step by the core from the batch — `derive` names the
|
||||
derivation, e.g. `conv_state_index`). Buffers live in one arena sized once, not an allocator per
|
||||
step.
|
||||
|
||||
`rad_decl_kv_group` declares each state kind: `RAD_KV_FULL` (paged per-token, growing),
|
||||
`RAD_KV_WINDOW` (bounded by a window), `RAD_KV_LINEAR` (per-sequence fixed state, GDN),
|
||||
`RAD_KV_CONV` (a rolling window of `width-1+num_spec`), `RAD_KV_RING` (window floor set at runtime).
|
||||
The **block size is not declared** — it comes from the resolved attention kernel's constraints; read
|
||||
it back with `rad_kv_block_size`. `rad_bind_layer_kv` binds each layer so the block manager knows how
|
||||
many state instances a sequence owns; `rad_kv_group_zero` (a rank with no heads),
|
||||
`rad_kv_group_replicated` (every rank holds the same bytes), and `rad_kv_group_dense_read` (read over
|
||||
the whole history every step, so it cannot stream) are the three refinements.
|
||||
|
||||
## Declare: ops
|
||||
|
||||
```c
|
||||
rad_op h = RAD_OP(b, "gemm_w4a8_nt",
|
||||
RAD_PARAMS(RAD_RANGE("M", 1, max_tok), RAD_INT("N", 8192), RAD_INT("K", 5120)),
|
||||
RAD_WEIGHTS(w));
|
||||
rad_op_reads (b, h, RAD_BUFS(b_x), n);
|
||||
rad_op_writes(b, h, RAD_BUFS(b_y), n);
|
||||
```
|
||||
|
||||
- Parameter names and operand order come from `docs/OPS.md`; a new name is allowed only when the
|
||||
vocabulary has none, and the first plugin to declare it fixes its schema.
|
||||
- `RAD_RANGE` on exactly one parameter per op gives the selector its bucket table. The core carries
|
||||
every parameter opaquely except `RAD_PROLE_SEQ_CHUNK` (cut steps on it) and `RAD_PROLE_CAPACITY`.
|
||||
- `rad_op_reads`/`rad_op_writes` are the buffer planner's liveness. An undeclared use is assumed live
|
||||
for the whole program: sound but wasteful.
|
||||
- **Fusion is selection.** Ask for the fused op; if `rad_op_resolved` is false, emit the chain. A
|
||||
fusion's `replaces` hook is a checkable equivalence, not a rewrite.
|
||||
- `rad_op_declared(b, "op")` asks whether any library declares an optional op; use it before
|
||||
emitting a fused form only an optional library defines.
|
||||
- Buffers touched from the **second lane** go through `rad_buf_concurrent`, and an architecture whose
|
||||
lane-1 work lies in one stretch declares `rad_lane1_window_begin/end` around it. The runtime
|
||||
refuses a lane-1 op outside the window; the window lets buffers used only outside it share bytes.
|
||||
|
||||
Step output: `rad_declare_logits` names the buffer the sampler reads (its dim 0 bounds `n_out`; a
|
||||
[n_tok, vocab] logits buffer at a large vocabulary is gigabytes — declare the gather +
|
||||
`logits_gemm` shape instead). An embedding program declares `rad_declare_pooled` instead (one row a
|
||||
sequence, no sampler, no generation). `rad_declare_scores` opts into host-side scored steps.
|
||||
|
||||
## Declare: drafters, encoders, generators
|
||||
|
||||
- `rad_declare_drafter(RadDrafterDecl)` — depth, block, proposal shape. `rad_drafter_shared_kv` (a
|
||||
drafter with no cache of its own), `rad_drafter_tree` (a tree over the proposals),
|
||||
`rad_drafter_short_block` (a second, shorter pass) are refinements. Declare the drafter **last**.
|
||||
- `rad_declare_encoder(RadEncoderDecl)` — a vision/audio tower, media preprocessed by the core.
|
||||
- `rad_declare_generator` / `rad_declare_audio_generator` / `rad_declare_video_generator` — image,
|
||||
audio and video generation programs; the engine sizes the step from the probe's `gen_*` fields
|
||||
before declare.
|
||||
- Optional format exports let a model state what it knows and the server learn nothing new:
|
||||
`rad_arch_chat_format` (a data description of reasoning/tool-call markup, replacing template
|
||||
analysis), `rad_arch_audio_format` (how a speech model hears: clip tokens, prompt, language,
|
||||
transcript start), `rad_arch_text_prepare` (rewrite request text into training form),
|
||||
`rad_arch_video_positions`/`rad_arch_video_pack` (video pass geometry and packing),
|
||||
`rad_arch_encode_geom` (encoder pass geometry). Each struct grows only at its end behind
|
||||
`struct_size`, so adding a field does not move the ABI.
|
||||
|
||||
**Media preprocessing is the core's**, not the plugin's: resizing, patching, placeholder expansion
|
||||
and positions in six processor families (Qwen-VL, Gemma 4, LFM2-VL, Unlimited-OCR, GLM-5.3, Pixtral
|
||||
for Nemotron), chosen by `vision_config.model_type` plus owning-model checks; audio through Gemma 4's
|
||||
extractor or Qwen3-ASR's Whisper one, chosen by `audio_config.model_type`. A vision model with another
|
||||
pipeline needs a core change.
|
||||
|
||||
## Step: issuing the graph
|
||||
|
||||
`RadBatch` is built once per step and shared by every block: `phase` (`PREFILL`/`DECODE`/`MIXED`),
|
||||
`n_tok`, `n_seq`, `token_ids`, `positions`, `cu_seqlens`, `seq_ids`, `lora_slot`, `n_out`/`out_ids`,
|
||||
`q_lens`/`ctx_lens`/`max_q_len`/`max_ctx_len`, the KV-group array, the speculative fields
|
||||
(`n_spec`, `draft_pass`, …).
|
||||
|
||||
```c
|
||||
static void step(RadCtx* c, const RadBatch* batch) {
|
||||
if (batch->draft_pass > 0) { m.ds.step(c, batch); return; } /* query pass */
|
||||
if (batch->draft_pass < 0) { m.ds.context(c, batch); return; } /* context pass */
|
||||
RAD_ISSUE(c, m.op_embed, praw(batch->token_ids, RAD_I32, batch->n_tok),
|
||||
RAD_W(m.w_tok), brows(m.b_x, batch->n_tok));
|
||||
for (auto& layer : m.layers) { layer.attn.step(c, batch); layer.mlp.step(c, batch); }
|
||||
/* ...heads; the lm_head only over batch->n_out rows... */
|
||||
}
|
||||
```
|
||||
|
||||
- `rad_issue(c, op, operands…, n)` / `RAD_ISSUE` (n = the batch's ranged extent) /
|
||||
`RAD_ISSUE_N` (an explicit row count). `rad_step_fail(c, what)` sets the status helpers return.
|
||||
- **A mixed step is the trap.** Continuous batching produces steps with a prefill chunk beside decode
|
||||
rows. A block that issues one path for the whole step sends its decode rows down the chunked path,
|
||||
which commits recurrent state directly and takes no `num_accepted`; a rejected draft cannot be
|
||||
rolled back, and the model stays fluent while drifting. Issue the recurrent path over
|
||||
`RadBatch::n_seq_decode` sequences and the chunked path over the rest (the core orders steps
|
||||
decode-first, so each half is contiguous; `cu_seqlens` holds absolute offsets, so pass
|
||||
`cu + n_seq_decode` with the token operands at their base).
|
||||
- A step issued on lane 1 outside the declared window fails by name. An op the core appends behind a
|
||||
declared one (runtime LoRA) counts at that op's position.
|
||||
- **Zero rows is not a call**: issuing an op over zero rows (a zero ranged extent, or all-empty
|
||||
sequences) returns `RAD_E_SHAPE` and writes nothing. The caller skips the issue, as the
|
||||
architectures' decode and prefill halves do.
|
||||
- `n_out` is 0 for a chunk that finishes no sequence — do not run the lm_head.
|
||||
|
||||
## Probe: what the plugin measured
|
||||
|
||||
`rad_arch_probe(meta, out)` runs before declare. Its fields are the engine's permissions:
|
||||
|
||||
| field | effect |
|
||||
|---|---|
|
||||
| `draft_depth`, `draft_depth_fixed` | the container's spec depth; a block drafter's trained block is the only depth it serves |
|
||||
| `shape_probe_ok` | the plugin tolerates a sizing declare, so the activation arena can be lent |
|
||||
| `enc_merge`, `audio_frames` | encoder pass granularity; longest audio clip |
|
||||
| `gen_rows`, `gen_patches`, `gen_audio`, `gen_ctx`, `gen_video` | generator sizing before declare |
|
||||
| `kv_quant_ok` | bits of the narrow KV layouts this model's kernels serve |
|
||||
| `chain_pays`, `tree_pays` | the plugin measured chained launches / draft trees faster |
|
||||
| `kv_stream_ok` | the architecture can stream a KV group (sparse, block-addressed, with `kv_resolve`) |
|
||||
| `control_vector_ok` | the plugin applies `RadBuildCtx::cvec` |
|
||||
| `tp_attn_rows_ok` | the plugin serves row-split prefill attention |
|
||||
| `state_f16_ok` | the plugin declares recurrent state at f16 and its kernels read it |
|
||||
|
||||
A flag whose probe bit is 0 is **refused at startup with the plugin named** (`--kv-cache-dtype
|
||||
fp8_v4`, `--control-vector`, `--tp-attn-split rows`, `--state-cache-dtype f16`, `--kv-resident`).
|
||||
That is the design: a deployment switch the plugin cannot honour is an error, never a silent
|
||||
substitution.
|
||||
|
||||
## Worked pattern: the smallest complete plugin
|
||||
|
||||
`arch/llama_fp8/llama_fp8.cpp` builds twice from one source — `llama_fp8` (quant `fp8_e4m3`) and
|
||||
`llama_dense_fp8` (quant `""`), differing only in the descriptor and the graph-dump text. It shows
|
||||
the whole shape:
|
||||
|
||||
1. `struct Layer { AttnFP8 attn; MlpFP8 mlp; }`, `struct Model` holding `Names`, `Geom`, buffer and
|
||||
op handles.
|
||||
2. `declare()`: name map entries first, then weights (with `rad_weight_encoding` where the head's
|
||||
format decides the kernel), buffers, KV group, the embedding and its all-reduce, then each layer's
|
||||
`declare()` from the shared blocks, the final norm, gather and lm_head, **the drafter last**, and
|
||||
`rad_note` lines explaining the shape.
|
||||
3. `step()`: draft passes first (sign selects query/context), then embedding (+ exact all-reduce
|
||||
when needed), each layer's `step()`, the final norm, and the head only over `n_out`.
|
||||
4. `probe()`: reads `draft.block_size` from the metadata and refuses any other depth for a block
|
||||
drafter.
|
||||
5. `RAD_ARCH_PLUGIN(...)` at the bottom.
|
||||
|
||||
Add a new family by copying a neighbour, replacing the name map and geometry, and composing the
|
||||
blocks you need; add a new *format* by cloning the plugin with another `(id, quant)` pair and the
|
||||
ops that format needs.
|
||||
|
||||
## Verifying a new plugin
|
||||
|
||||
```sh
|
||||
radiance --model m.rad --debug-graph # every op, its resolved kernel, and why
|
||||
radiance --model m.rad --debug-placement # the plan
|
||||
rad-convert $SRC -o m.rad --plan-only -v # a convert that plans will boot
|
||||
rad-kbench -m m.rad --kernels mylib,libr4d,libref --report k.md --bench
|
||||
rad-tune -m m.rad --tp 2 # only at the shapes you serve
|
||||
```
|
||||
|
||||
`tests/arch_test.cpp` drives a plugin's builder the way the core does, including lane-window
|
||||
liveness, so the pinned graphs and steps are the ones the core plans. A plugin that declares a weight
|
||||
form no kernel `layout` accepts is a declare-time error naming the weight, encoding and kernel — the
|
||||
same check `rad-convert` runs before writing.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Radiance Plugins]] — kernel libraries, encodings, the build path, ABI law.
|
||||
- [[Radiance Conversion]] — how the name map drives a convert, and `--set`/descriptor selection.
|
||||
- [[Radiance Model Families]] — the existing `(id, quant)` pairs and their recipes.
|
||||
- [[Radiance Tools]] — `rad-kbench`, `rad-tune`, `rad-info --plugins`.
|
||||
@@ -0,0 +1,354 @@
|
||||
---
|
||||
type: "howto"
|
||||
title: "Radiance Conversion — checkpoints into .rad containers"
|
||||
description: "Turning a checkpoint into a .rad container: rad-convert, the recipe language, quantisers, reuse and export."
|
||||
tags:
|
||||
- llm
|
||||
- radiance
|
||||
- quantization
|
||||
- howto
|
||||
timestamp: 2026-10-30
|
||||
---
|
||||
|
||||
# Radiance conversion
|
||||
|
||||
`rad-convert` turns a Hugging Face checkpoint into a `.rad` container, choosing each weight's
|
||||
encoding from a **recipe**. `rad-export` wraps it for finetunes (a merged checkpoint or a PEFT
|
||||
adapter) with the stock model's recipe. Neither opens a GPU: conversion is CPU-only and safe beside a
|
||||
running server.
|
||||
|
||||
Fetch checkpoints with `scripts/hfget.sh <org/repo> <dir>` (resumable; reads `.safetensors`,
|
||||
skips `.bin`/ONNX/GGUF when safetensors exist; `HF_TOKEN` for gated repos; writes a
|
||||
`.hfget-manifest` of content keys). `radiance --model <owner/name>` runs the same fetch itself.
|
||||
|
||||
## What a conversion does
|
||||
|
||||
1. Read `config.json` (or GGUF metadata), and the shard index if present.
|
||||
2. Load the architecture, kernel and quantiser plugins and run **declare**, as the engine does.
|
||||
3. Resolve every op and ask each resolved kernel's `layout` hook about the planned encoding. A
|
||||
recipe no kernel can serve is refused **here**, before any I/O.
|
||||
4. Write each logical weight once: the checkpoint's bytes where no rule matches, otherwise the
|
||||
planes the matching quantiser writes, a block of rows at a time.
|
||||
5. Measure every quantised weight against its source through the core's decoder and print the worst.
|
||||
6. Bake the tokeniser, chat template, metadata, recipe and any draft head.
|
||||
|
||||
```sh
|
||||
build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-27B -o qwen3.8-27b-bf16.rad \
|
||||
--recipe data/recipes/q38-27b-bf16.recipe
|
||||
build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-Flash-Next -o flashnext-bf16.rad --max-ctx 4096
|
||||
```
|
||||
|
||||
## Every `rad-convert` option
|
||||
|
||||
| option | meaning |
|
||||
|---|---|
|
||||
| `<input>` | a `.gguf`, a `.safetensors`, an `index.json`, or a checkpoint directory |
|
||||
| `-o, --out FILE` | the container to write |
|
||||
| `--recipe FILE` | one rule a line: `PATTERN QUANTISER key=value …`; repeatable; first match decides |
|
||||
| `--quant RULE` | `PATTERN=QUANTISER[:key=value,…]`, placed **ahead of** the recipe's rules; repeatable |
|
||||
| `--list-quantizers` | every loaded quantiser and the options it takes, then exit |
|
||||
| `--arch ID` | override the architecture id read from the checkpoint |
|
||||
| `--home DIR` | `$RADIANCE_HOME` (kernels, architectures, quantizers) |
|
||||
| `--kernels A:B` | kernel libraries to load, first wins; declare checks the recipe against them |
|
||||
| `--tokenizer F` | `tokenizer.json`; its declared chain is baked instead of reconstructed from a pre-tokenizer name. Required for a GGUF, without it the chain is a guess |
|
||||
| `--tokenizer2 F` | a second `tokenizer.json` baked as the second vocab (a generator reading its prompt through two tokenisers); keep `tokenizer_config.json` beside it |
|
||||
| `--add FILE` | merge one more `.safetensors`, tensor names kept; repeatable; a name in two files is refused |
|
||||
| `--config DIR` | read `config.json`, `generation_config.json`, `tokenizer.json` from DIR for a single-file checkpoint |
|
||||
| `--chat-template F` | a Jinja template baked in, overriding the checkpoint's (or its absence); trailing newlines stripped; without one `/v1/chat/completions` answers 501 |
|
||||
| `--imatrix F` | llama.cpp imatrix: expert-popularity profile, per-column importances, error weighting; required by `ggml` types `iq2_xxs`, `iq2_xs`, `iq1_s` |
|
||||
| `--set K=V` | a metadata key, seen by declare **and** written into the container; repeatable |
|
||||
| `--max-tok N` | the `max_tok` declare runs at (default 8192, or a generator's largest pass) |
|
||||
| `--max-ctx N` | the context declare runs at (default: the checkpoint's training context) |
|
||||
| `--max-spec N` | the `max_spec` declare runs at (default 4, or the block a drafter states); `0` leaves a draft head out |
|
||||
| `--draft-model DIR` | a drafter shipped as its own repository, merged under `draft.` |
|
||||
| `--kv-cache-dtype T` | `bf16` or `fp8`: the cache width declare runs at; the container is identical at either |
|
||||
| `--reuse FILE` | copy every weight an earlier container holds with the same encoding, quantiser, options and shape |
|
||||
| `--changed FILE` | with `--reuse`: checkpoint tensors (one name a line) that differ; every weight reading one is converted |
|
||||
| `--in-place` | with `--reuse`: extend that container; new weights go past its end, the header last; keeps `<file>.pre-append` |
|
||||
| `--lora FILE\|DIR` | merge a PEFT adapter into the checkpoint tensors before converting (`W + strength · scale · B·A`); repeatable, merged in order; each merged tensor joins `--changed` |
|
||||
| `--lora-strength F` | strength for every `--lora` (default 1) |
|
||||
| `--lora-map FILE` | JSON map for adapters whose module names differ: `{"strip":[prefix…],"modules":{"<pat>":[target…]}}`; one `*` matches a block index; several targets mean a fused module (rows split in order) |
|
||||
| `--plan-only` | declare, plan every encoding, print the plan, stop before writing |
|
||||
| `--report N` | print the N worst quantisation errors (default 20) |
|
||||
| `-v, --verbose` | per-weight detail |
|
||||
|
||||
`--reuse` matches a weight on its **encoding, quantiser, options and shape — not on its source
|
||||
bytes** — and carries whatever calibration made the old planes. Use it only when source and
|
||||
calibration are unchanged, or name the changed tensors with `--changed`. `--in-place` refuses to
|
||||
change a weight the container holds or to add an expert.
|
||||
|
||||
## The recipe language
|
||||
|
||||
```
|
||||
# PATTERN QUANTISER OPTIONS
|
||||
blk.*.ffn_*_exps.*.weight gptq codes=i4 group=128 scale=bf16 transform=fwht128 calib=$CALIB
|
||||
blk.*.ple_ngram.weight rtn codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
|
||||
output.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
|
||||
blk.*.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
|
||||
```
|
||||
|
||||
- One rule a line: `PATTERN QUANTISER [key=value …]`. `#` starts a comment.
|
||||
- `PATTERN` is a glob over the **whole logical weight name**: `*` crosses dots, `?` is one character.
|
||||
- **The first matching rule decides.** No rule → the checkpoint's own encoding, which must be one
|
||||
the engine serves as is: `bf16`, `f16`, `f32`, or block-scaled FP8 (`X.weight` with
|
||||
`X.weight_scale_inv`). Anything else is named and refused.
|
||||
- `$NAME` / `${NAME}` in a value comes from the environment; an **unset variable is refused**, never
|
||||
expanded empty (so an empty `$CALIB` cannot silently produce an uncalibrated convert).
|
||||
- A quantiser may **decline** a weight (`RAD_E_UNSUPPORTED`), which keeps the checkpoint's encoding
|
||||
— `rtn` declines a rank-1 tensor, so a pattern that also matches a norm leaves the norm alone.
|
||||
- An option the quantiser does not declare is refused, by name, before anything is quantised.
|
||||
- Two plugins naming the same quantiser are refused, both named.
|
||||
|
||||
## Quantisers
|
||||
|
||||
| quantiser | what it does |
|
||||
|---|---|
|
||||
| `rtn` | round to nearest on a grid; each block's scale chosen by `rule` |
|
||||
| `gptq` | error feedback against the layer's input Hessian (`calib=DIR`, damped by `damp` = 0.01 of the mean diagonal by default); `act_order=1` quantises columns by descending input energy and stores the permutation; `cd=N` adds coordinate-descent sweeps |
|
||||
| `awq` | per-input-column scaling to a searched power, then a per-block clip search, both against the Hessian; column scales are the last scale level |
|
||||
| `autoround` | sign-gradient descent on a rounding offset per value and a range factor per block; `iters` steps; slow (a 27B dense model takes most of a working day at defaults; `iters=50` a quarter) |
|
||||
| `paroquant` | channel scaling plus `stages` Givens rotations learned against the Hessian, then AutoRound on the rotated weight; the `givens` transform |
|
||||
| `m2xfp4` | M2XFP's weight-side Sg-EM: MXFP4 plus one metadata byte per group of 32; its own `m2xfp4` scheme; `adaptive=0` pins the bias, `wide=1` widens it |
|
||||
| `ggml` | a llama.cpp format (`q4_0`, `q4_1`, `q5_0`, `q5_1`, `q8_0`, `q2_K`…`q6_K`, `iq4_nl`, `iq4_xs`, `iq2_xxs`, `iq2_xs`, `iq2_s`, `iq3_xxs`, `iq3_s`, `iq1_s`, `iq1_m`), by llama.cpp's own search, imatrix-weighted, codes byte for byte |
|
||||
| `cast` | a float weight in another float dtype (`dtype=` f32, bf16, f16, fp8_e4m3, fp8_e5m2), no scale |
|
||||
| `import` | rad-convert's own, not a plugin: write the numbers a quantised release ships, in the spelling the kernels read, instead of quantising again |
|
||||
|
||||
Every quantiser except `m2xfp4`, `ggml`, `cast` and `import` takes the grid options:
|
||||
|
||||
| option | values |
|
||||
|---|---|
|
||||
| `codes` | `i2`…`i8`, `i16`, `i32` symmetric; `u1`…`u8`, `u16` with `zero=`; `fp8_e4m3`, `fp8_e5m2`, `fp6_e2m3`, `fp6_e3m2`, `fp4_e2m1` |
|
||||
| `table` | a codebook the codes index: `nf4`, `iq4nl`, `w4nl`, `w5nl`, `binary`, `ternary`, or `;`-separated values |
|
||||
| `group=N` / `block=RxC` | where the scales sit; `*` is a whole extent |
|
||||
| `scale` | f32, bf16, f16, fp8_e4m3, e8m0; under `scale2` also an integer sub-scale `u4`…`u8`, `i6`, `i8` |
|
||||
| `scale2`, `group2` / `block2` | a second scale level the first is relative to (default the whole tensor) |
|
||||
| `scale2_value` | the second level fixed at this value, exact in `scale2`'s dtype |
|
||||
| `zero` | an integer zero point per block — an asymmetric grid |
|
||||
| `transform=fwhtN` | rotate each row by the Hadamard first |
|
||||
| `rule` | `absmax`, `mx` (OCP shared exponent), `fixed` (`scale_value=`), `search` (least squared error over candidates and refits, imatrix-weighted) |
|
||||
| `clamp` | a signed code's range: `full` or `sym` |
|
||||
|
||||
Named grids: MXFP4 = `codes=fp4_e2m1 group=32 scale=e8m0`; NVFP4 =
|
||||
`codes=fp4_e2m1 group=16 scale=fp8_e4m3 scale2=f32`; NF4 = `table=nf4 group=64`; a K-quant-like grid
|
||||
= `codes=i4 group=32 scale=u6 scale2=f16 group2=256`; a ternary BitNet weight = `table=ternary`.
|
||||
|
||||
**A container can hold only what some kernel reads.** rad-convert checks every planned encoding
|
||||
against the `layout` hooks of the kernels declare resolved and refuses one no kernel accepts. libr4d
|
||||
reads the plain floats; block-scaled fp8 and fp8 rows; int8 block rows plain or rotated; routed
|
||||
experts as fp8, int4 (`i4*bf16[1x128]/fwht128`) or the `w4nl`/`w5nl` codebooks under E4M3 scales;
|
||||
the dense w4, w8 and mxfp4 GEMM rows; and the 2-bit draft head. See [[Radiance Plugins]] for the
|
||||
encoding model.
|
||||
|
||||
## Importing a release someone else quantised
|
||||
|
||||
These forms are read weight by weight and written in the spelling the kernels read; a recipe rule
|
||||
re-quantises from the exact dequantisation where the family's kernels want another form.
|
||||
|
||||
| `quantization_config` | read as | notes |
|
||||
|---|---|---|
|
||||
| `quark` MXFP4 (fp4, group 32, e8m0) | `fp4_e2m1*e8m0[1x32]` | byte for byte; also servable as it stands |
|
||||
| `paroquant_mxfp4` | `fp4_e2m1*e8m0[1x32]/givens` | codes/exponents byte for byte plus learned rotations as `givens` tables |
|
||||
| `paroquant`, `int5-bitplane`, plain `paroquant` | `u5*f16[1x128]-u8[1x128]/givens`, `u4*…` | AWQ-packed nibbles re-laid as a code bit stream; f16 scales, integer zeros, rotations |
|
||||
| `compressed-tensors` `pack-quantized`, 4-bit `int`, group, act-order none/static/weight | `i4*<scale>[1xG]` | `weight_packed` made signed beside `weight_scale`; group/dynamic reorder is refused |
|
||||
| same with `symmetric:false` (AWQ) | `u4*<scale>[1xG]-u4[1xG]` | nibbles plus `weight_zero_point`, bit for bit |
|
||||
| `auto-round` `auto_round:auto_gptq`, `gptq` v1 | `i4*bf16[1x128]` or `i8*bf16[1x128]` | scales rounded to bf16; convert-time; `import` keeps shipped f16 |
|
||||
| `nvfp4-pack-quantized` | `fp4_e2m1*f32[1x16]` | `weight_scale / weight_global_scale`; **no kernel serves NVFP4**, so it always takes a rule |
|
||||
| `modelopt` NVFP4 / FP8 / MIXED_PRECISION | NVFP4 `fp4_e2m1*f32[1x16]`; FP8 `fp8_e4m3*f32[1x*]` | ModelOpt's second scale **multiplies**; `input_scale` is not read |
|
||||
| compressed-tensors all-FP8 groups | `fp8_e4m3*<scale>[1x*]` or `[RxC]` | nothing imported: stock FP8 with its scale; a per-tensor scale is written once a row |
|
||||
| GGUF `Q8_0`, `Q4_K`, `Q6_K` | `i8*f16[1x32]`, affine u4/u6…, `i6*i8…` | blocks split bit for bit; **not servable as they stand** |
|
||||
|
||||
The width is read **per tensor**, so a mixed release imports int4 experts and int8 projections
|
||||
together. What an import cannot read (an asymmetric GPTQ zero point, a group-reordering act-order
|
||||
index, a GPTQ group under 128, a missing/zero global scale, an unlisted format) is refused by name.
|
||||
The release's `quantization_config` names the tool, not a served form, so it is kept as
|
||||
`imported_quantization_config.*`.
|
||||
|
||||
`import` options: `codes=uN` makes a symmetric grid unsigned under a zero point of `2^(N-1)`;
|
||||
`zero=` the zero plane's dtype (`u4`, `u8`); `scale=` f16/bf16/f32, each shipped scale rounded to
|
||||
nearest; `block=RxC` repeats an FP8 scale shipped per row/tensor over that grid. It reads an imported
|
||||
integer grid or FP8 with a per-row/per-tensor scale and keeps a plain float weight as it is.
|
||||
|
||||
**NVFP4 to MXFP4 rounds twice** (e4m3 per 16 → power of two per 32), so it loses more than MXFP4 made
|
||||
from bf16. Prefer a calibrated MXFP4 container when one exists; the quality verdict is a KL gate, not
|
||||
a file size.
|
||||
|
||||
## The descriptor law (`--set`)
|
||||
|
||||
The architecture plugin is chosen by the exact `(arch id, quant descriptor)` pair. A convert-time
|
||||
choice that must also hold at serve time is stated once with `--set`, which hands it to declare and
|
||||
writes it into the container. Storage format is not such a choice (the encoding is in the container
|
||||
and the plugin reads it there), but the **serving plugin** is: a `quantization_config.*` key
|
||||
re-derives the descriptor.
|
||||
|
||||
```sh
|
||||
rad-convert <bf16 Qwen3.8-27B> -o q38-27b-fp8.rad \
|
||||
--recipe data/recipes/q38-27b-fp8block-df2.recipe \
|
||||
--set quantization_config.quant_method=fp8 --set quantization_config.fmt=e4m3 \
|
||||
--set "quantization_config.weight_block_size=128 128" \
|
||||
--set quantization_config.activation_scheme=dynamic
|
||||
```
|
||||
|
||||
A bf16 checkpoint whose recipe writes Qwen's block FP8 must therefore be declared *as that format*;
|
||||
otherwise the selection pair does not match any plugin and the convert (or the boot) is refused. The
|
||||
same law bites the other way when a release's own `quant_method` names a pair no plugin claims.
|
||||
|
||||
## Inspect and validate the result
|
||||
|
||||
```sh
|
||||
rad-info out.rad # architecture, source quant, recipe, sections
|
||||
rad-info --recipe out.rad # the expanded rules — did the intended one win?
|
||||
rad-info -v out.rad | head -40 # every tensor, its encoding, and what made it
|
||||
rad-kbench -m out.rad --home $RADIANCE_HOME --report kbench.md
|
||||
radiance --model out.rad --debug-graph # every op resolves
|
||||
```
|
||||
|
||||
The converter's own error report (the worst `--report N` weights) measures each quantised weight
|
||||
against its source through the core decoder. `rad-kbench` adds the artifact test: weight operands
|
||||
read from the container and passed through each kernel's `layout`/`relayout` hooks. Both are on
|
||||
[[Radiance Tools]].
|
||||
|
||||
## Calibration (GPTQ, AWQ, AutoRound, ParoQuant)
|
||||
|
||||
`calib=DIR` is a directory an engine run wrote with `RADIANCE_CALIB_DIR` set: each layer's **Gram
|
||||
matrices of its input**, in the domain the kernel multiplies it in. A block that characterises an
|
||||
activation declares an f32 `[K,K]` PERSIST buffer and a `gram_accum` op; at `RADIANCE_CALIB_TOKENS`
|
||||
(default 65536) the run writes the files.
|
||||
|
||||
Three-step bootstrap: convert with plain `rtn` → serve that container with `RADIANCE_CALIB_DIR` →
|
||||
push real prompts through prefill → convert again with the Hessians. The second moment is robust to
|
||||
the first container's small quantisation error. A weight with no file is quantised uncalibrated and
|
||||
the converter says so; `awq`, `autoround` and `paroquant` fall back to the imatrix diagonal when one
|
||||
is given.
|
||||
|
||||
- The accumulator must be **PERSIST**: a transient buffer may share bytes and be overwritten between
|
||||
the launches it sums.
|
||||
- Files are per rank; the consumer sums them. Under expert parallelism a rank's `down` input covers
|
||||
only its experts' tokens; `gate_up` input is the full token set on every rank, so the sum scales
|
||||
`H` by world size — GPTQ is invariant to a positive scale.
|
||||
- Under tensor parallelism a column-split linear sees only its rank's columns, so a tap may write
|
||||
`gram.r<rank>.<stem>.part.bin`; libquant places the parts on the diagonal in rank order, leaving
|
||||
cross-rank products zero. They must cover K and be in the model's own domain.
|
||||
- `--imatrix` supplies the expert-popularity profile the placement planner ranks units by, the
|
||||
per-column importances a quantiser uses, and the error report's weighting.
|
||||
|
||||
## Reuse, drafters, merging
|
||||
|
||||
```sh
|
||||
# re-quantise only what changed
|
||||
rad-convert $SRC -o new.rad --recipe new.recipe --reuse old.rad --changed changed.txt
|
||||
|
||||
# merge a drafter shipped as its own repo (DFlash2, DSpark, MTP assistant)
|
||||
rad-convert $SRC -o out.rad --draft-model /srv/models/z-lab/Qwen3.8-27B-DFlash2 \
|
||||
--recipe data/recipes/q38-27b-mxfp4-df2.recipe
|
||||
|
||||
# merge a PEFT adapter at convert time
|
||||
rad-convert $SRC -o tuned.rad --lora ~/run/adapter --lora-strength 1.0 \
|
||||
--recipe data/recipes/q38-27b-mxfp4-df2.recipe
|
||||
```
|
||||
|
||||
- A drafter's tensors go in under `draft.`, its config under `draft.` keys. It has no embedding and
|
||||
no `lm_head` of its own; it uses the target's.
|
||||
- `--lora-map` exists for adapters trained on names the checkpoint does not use; a fused target
|
||||
(`qkv`) takes several targets and splits the adapter's rows in order.
|
||||
- `--add` + `--config` + `--tokenizer2` assemble a ComfyUI-style release of separate single files
|
||||
(Anima: diffusion model, text encoder, VAE). A name in two files is refused.
|
||||
|
||||
## `rad-export` — finetunes without hand-writing the command
|
||||
|
||||
```sh
|
||||
rad-export ~/outputs/my-run/checkpoint-400 --dry-run # what it would run
|
||||
rad-export ~/outputs/my-run/checkpoint-400 # adapter -> <name>.rad
|
||||
rad-export ~/exports/merged -o ~/models/mine.rad --recipe q38-27b-fp8block-df2
|
||||
```
|
||||
|
||||
| flag | default | meaning |
|
||||
|---|---|---|
|
||||
| `SRC` | required | a merged 16-bit checkpoint (`config.json` + `*.safetensors`) or a PEFT adapter (`adapter_config.json` + `adapter_model.safetensors`) |
|
||||
| `-o, --out FILE` | SRC's folder name + `.rad`, beside SRC or in `out_dir` | the container |
|
||||
| `--recipe NAME\|FILE\|auto` | `auto` | a recipe name, a file, or the one recipe whose header names the base model, takes a bf16 checkpoint and needs no calibration |
|
||||
| `--base DIR` | the adapter's `base_model_name_or_path`, from disk or the HF cache, else downloaded | the base a LoRA merges into |
|
||||
| `--draft DIR\|ORG/NAME` | the config, else the Hub repo the recipe's usage line names | the drafter when the recipe merges one; a usage line with a placeholder asks for `--draft` |
|
||||
| `--image IMAGE` | `$RADIANCE_IMAGE`, the config's, then `stilldeadcode/radiance` | run `rad-convert` in this image (docker, CPU-only, no network, **not pulled**) |
|
||||
| `--rad-convert PATH` | — | run this binary instead of docker; recipes come from its `share/radiance` |
|
||||
| `--memory M` | — | docker memory limit for the convert |
|
||||
| `--dry-run` | — | resolve recipe, drafter and paths; print the `rad-convert` line; write nothing |
|
||||
| `--keep-merged` | — | keep the merged checkpoint beside OUT |
|
||||
| `--config FILE` | `~/.config/rad-export/config.json` | the config below |
|
||||
| `--list-recipes [--json]` | — | list the recipes the image/installed tool carries |
|
||||
|
||||
- Input: a merged 16-bit checkpoint (`config.json` + `*.safetensors`) or a PEFT LoRA adapter
|
||||
(`adapter_config.json` + `adapter_model.safetensors`).
|
||||
- An adapter is merged **exactly as PEFT's `merge_and_unload`** computes it on CPU: `scale` =
|
||||
alpha/r (alpha/√r with rsLoRA), `rank_pattern`/`alpha_pattern` honoured, delta in fp32 rounded to
|
||||
the adapter's dtype. It streams tensor by tensor into 1 GiB shards, so a 27B merge needs a few GiB
|
||||
of RAM, not the model twice. Plain linear LoRA only: embedding LoRA, trained biases, DoRA, saved
|
||||
modules, convolution LoRA, a rank other than the config's, and a missing target are refused before
|
||||
anything is written. The adapter's tokeniser/template replace the base's where shipped.
|
||||
- The recipe comes from the image (`/opt/radiance/share/radiance/recipes`) or the installed
|
||||
`rad-convert`'s share; `auto` matches the base model's name in recipe headers and asks when several
|
||||
fit. A recipe needing calibration is listed but never picked by `auto` — a finetune's Hessians are
|
||||
not the stock model's.
|
||||
- It runs `rad-convert` on the CPU, under Docker as the caller with no network, and writes the log to
|
||||
`OUT.log`.
|
||||
- `~/.config/rad-export/config.json` pins what auto cannot know: `image`, `out_dir`,
|
||||
`recipes{model:recipe}`, `drafts{recipe:dir}`, `bases{repo:dir}`.
|
||||
- It is Python 3 with no dependencies; merging needs `torch`/`safetensors`, fetching a missing base or
|
||||
drafter needs `huggingface_hub`. The image carries no Python; extract the script with
|
||||
`docker run --rm --entrypoint busybox stilldeadcode/radiance cat /opt/radiance/bin/rad-export`.
|
||||
|
||||
## Scoring a container (the KL mode)
|
||||
|
||||
```sh
|
||||
# once, on the bf16 container
|
||||
radiance --model bf16.rad --kld-record /srv/kld/q38 --kld-corpus corpus.jsonl
|
||||
# each candidate, same flags otherwise
|
||||
radiance --model quant.rad --kld-ref /srv/kld/q38 --kld-out report.json
|
||||
```
|
||||
|
||||
- The corpus is JSONL, one `{"prompt", "score_from"?, "source"?}` a line; only `prompt` is required.
|
||||
Positions `[score_from, n-1)` are scored; `source` groups the report. Keep the scoring corpus
|
||||
disjoint from the calibration corpus.
|
||||
- `--kld-record` writes the reference's log-probabilities (f16 over the vocabulary, the 16 most
|
||||
likely tokens exactly, the token ids). DIR is never overwritten and is read only once `kld.json`
|
||||
exists, which is written last.
|
||||
- `--kld-ref` scores this model's f32 logits against stored rows: mean, median, 99/99.9/99.99th
|
||||
percentile KL, top-1 agreement, and both models' perplexity. `--kld-out` writes the report; `FILE.rows`
|
||||
is `[positions, 4]` f32 (KL, both NLLs, argmax agreement) for position-by-position comparison.
|
||||
- The mode turns speculation and the prefix cache off, turns the step pipeline off, and refuses an
|
||||
explicit spec depth above 0. The logits buffer holds a row per token of a step, so a large
|
||||
`--max-num-batched-tokens` costs pinned host memory as well as VRAM.
|
||||
- `--kld-lora NAME` scores every document through a served adapter, against a reference recorded
|
||||
from the merged checkpoint; `--kld-lora-sources PREFIX` runs only matching documents through it,
|
||||
the rest as the base, in the same steps — each document's rows should be byte-identical to the
|
||||
all-adapter and all-base runs.
|
||||
|
||||
The gate the fleet uses (mean KL ≤ 0.03, top-1 ≥ 95 %) is on [[frank-archon]] and
|
||||
[[Quantization Production]]; the mode here is only the instrument.
|
||||
|
||||
## Convert laws paid for in full
|
||||
|
||||
- **`--plan-only` first.** A plan that resolves every op will boot; it costs seconds against an hour
|
||||
of I/O on a large checkpoint.
|
||||
- **Declare runs at the step and context bounds, and some graphs depend on them.** Qwen4-Exp sizes
|
||||
its QSA buffers by `--max-ctx` and refuses a `--max-tok`/context pair whose score plane would pass
|
||||
2 GiB; the defaults (8192 tokens, the checkpoint's training context) fit most models, and a
|
||||
generator's largest pass may override `--max-tok`.
|
||||
- **Scale names follow the tensor.** `scale_name()` appends `_scale_inv` to the *tensor* name, so a
|
||||
stacked `…experts.gate_up_proj` keeps its scales at `…gate_up_proj_scale_inv`.
|
||||
- **File shadowing: first wins.** The reader opens index-named files sorted and emplaces every tensor
|
||||
each file physically holds, first-wins. A merged PLE shard must sort first
|
||||
(`00-ple.merged.safetensors`) or a raw shard wins the name.
|
||||
- **Never write through a staging symlink.** A checkpoint may symlink its raw index; `open(path,"w")`
|
||||
writes through it and corrupts the checkpoint. Write a temp file and `os.replace`.
|
||||
- **One encoding per expert table.** A whole expert kept bf16 is the only thing legal beside a
|
||||
table; a split tier (down=fp8, gate_up=w4) is unservable.
|
||||
- **The descriptor pair has no fallback** (above), and **convert is card-less**: GPTQ solves on CPU
|
||||
BLAS, so the GPU is irrelevant to production and the card only decides what can be decoded.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Radiance Model Families]] — the (arch id, quant descriptor) table and the command per model.
|
||||
- [[Radiance Plugins]] — quantiser and kernel-plugin contracts behind these options.
|
||||
- [[Radiance Tools]] — `rad-info`, `rad-kbench`, `rad-tune`.
|
||||
- [[Quantization Production]] — the fleet's calibration/KLD campaign and its gates.
|
||||
- [[frank-archon]] — the deployment whose recipes and convert laws these are.
|
||||
@@ -0,0 +1,140 @@
|
||||
---
|
||||
type: "reference"
|
||||
title: "Radiance Engine — the inference server behind frank-archon"
|
||||
description: "The radiance inference engine: its shape, plugin model, container format, and how the reference pages fit together."
|
||||
tags:
|
||||
- llm
|
||||
- radiance
|
||||
- reference
|
||||
timestamp: 2026-10-30
|
||||
---
|
||||
|
||||
# Radiance Engine — shape, plugin model, container
|
||||
|
||||
Radiance is StillDeadcode's inference server for AMD GPUs, written in C++20 and HIP: language
|
||||
models, plus image, video, speech and music generation and embeddings. One HTTP front end speaks
|
||||
OpenAI's chat, completions, Responses, images, audio and video APIs, Anthropic's Messages API and
|
||||
llama-server's native one. Nothing in the serving path is Python.
|
||||
|
||||
The fleet's only radiance endpoint is [[frank-archon]] (2× R9700, TP=2, Dankserver). **This page
|
||||
and its siblings own the engine reference; the live deployment shape, its Eras and its measured
|
||||
laws stay on the endpoint page.** The numbers here describe upstream `main` as read 2026-10-30
|
||||
(release 1.14.0, plugin ABI 31). The fleet pin is older (1.8.1): a flag below that post-dates the pin
|
||||
is a boot refusal, never a silent no-op, and the endpoint page records the CLI families that pin
|
||||
itself retired relative to its own rollback target.
|
||||
|
||||
Source: `https://codeberg.org/StillDeadcode/radiance` at `8da043c`. Engine docs live in `docs/`
|
||||
(`GUIDE.md` for serving, `TOOLS.md` for tools, `PLUGIN.md` for plugins, `ARCHITECTURES.md` per
|
||||
family, `OPS.md` for the op vocabulary, `spec.md` for the design).
|
||||
|
||||
## The four moving parts
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[Checkpoint<br/>safetensors / GGUF] -->|rad-convert + recipe| B[.rad container]
|
||||
B --> C[radiance engine]
|
||||
C --> D[OpenAI / Anthropic /<br/>llama-server wire APIs]
|
||||
P1[kernel library .so<br/>libr4d, libavx, libref] --> C
|
||||
P2[architecture .so<br/>one per arch id × quant descriptor] --> C
|
||||
P3[quantiser .so<br/>libquant] -->|rad-convert| A
|
||||
```
|
||||
|
||||
- **Engine** — `core/`: declare phase, placement/memory planner, scheduler, KV manager, sampler,
|
||||
tokeniser, HTTP server, media preprocessing.
|
||||
- **Kernel libraries** — implement ops for hardware. `libr4d` targets RDNA4 (`gfx1200`,
|
||||
`gfx1201`); `libavx` serves the host backend (AVX-512/AVX2) with no GPU at all; `libref` is the
|
||||
slow reference implementation every other library is checked against.
|
||||
- **Architecture plugins** — declare one model family at one weight format: its weights, buffers,
|
||||
KV groups and the ops it issues each step. The engine picks one by an **exact (architecture id,
|
||||
quantisation descriptor) pair, with no fallback**.
|
||||
- **Quantiser plugins** — `libquant`'s `rtn`, `gptq`, `awq`, `autoround`, `paroquant`, `m2xfp4`,
|
||||
`ggml`, `cast`, `import`; they write a weight's numbers into an encoding's planes.
|
||||
- **Tools** — `rad-info`, `rad-convert`, `rad-export`, `rad-kbench`, `rad-tune`, `rad-schemas`,
|
||||
`rad-tokcheck`, plus `rad-chatparse`, `rad-avx-check`, `r4d_selftest` in the build tree.
|
||||
|
||||
## Two phases, one static graph
|
||||
|
||||
`rad_arch_declare` runs once per rank and names every weight, buffer, KV group and op, in order.
|
||||
The core resolves every op against the kernel hierarchy per band, per domain, and completes the
|
||||
whole plan before a byte of weights is read. `rad_arch_step` then only *issues* handles obtained at
|
||||
declare; it may not allocate and may not synchronise. A model whose declare succeeds is fully known
|
||||
statically, which is what makes `--debug-graph`, the buffer liveness plan and `rad-convert`'s
|
||||
plan-only check possible.
|
||||
|
||||
Consequences a plugin author must respect:
|
||||
|
||||
- An op that never resolves is `RAD_NULL_HANDLE` and only fails if issued. A fusion is a *selection*:
|
||||
ask for the fused op, emit the chain when `rad_op_resolved` is false.
|
||||
- Declaration order is the buffer planner's liveness. A drafter must be declared after the trunk
|
||||
it runs behind, or trunk and drafter buffers silently share bytes.
|
||||
- Every op in the graph must have a row in a loaded kernel library. `libref` implements the
|
||||
conventional vocabulary in `docs/OPS.md`; device-only ops (most fusions) exist only in `libr4d`.
|
||||
|
||||
## The `.rad` container (RAD2)
|
||||
|
||||
A container is one file holding everything the server needs: weights as **canonical encoding
|
||||
planes**, the tokeniser (and optionally a second vocab), the chat template, metadata, the recipe it
|
||||
was made with, and any draft head. It names no kernel library.
|
||||
|
||||
- Planes are kernel-independent: codes, scales, zero points, codebooks — row-major, bit-packed
|
||||
sub-byte where needed. A kernel rearranges them into its own layout at load (`layout`/`relayout`
|
||||
hooks), losslessly, so a kernel library can change layouts without reconverting the model.
|
||||
- Encodings are data, not an enum: `RadEncoding` names the scheme (`plain`, `affine`, or a
|
||||
quantiser's own), the planes by role, the scale grouping, and a transform (`fwhtN`, `perm`,
|
||||
`givens`). That is how one file can hold block-FP8, int4, MXFP4, NVFP4-imported, ParoQuant
|
||||
rotations and llama.cpp grids.
|
||||
- `rad-info` is the reader: sections (strings, metadata, directory, planes, encodings, vocab,
|
||||
profile, data), `--meta`, `--vocab`, `--recipe`, `--plugins`, and `-v` for every tensor with its
|
||||
encoding and what made it.
|
||||
|
||||
See [[Radiance Conversion]] for producing one and [[Radiance Tools]] for inspecting it.
|
||||
|
||||
## Selection key and version law
|
||||
|
||||
- A checkpoint's architecture id is its `config.json` `model_type` without underscores; `rad-convert
|
||||
--arch ID` overrides it. `rad_arch_quant` supplies the quantisation descriptor (empty for an
|
||||
unquantised checkpoint).
|
||||
- **No fallback.** A raw official FP8 checkpoint whose `quantization_config.quant_method` is `fp8`
|
||||
is not the pair the `qwen4exp_fp8` plugin claims (`(qwen4exp, "")`), so both serve and convert
|
||||
refuse it until the descriptor is changed with `--set` or a staged `config.json`. The deployment
|
||||
lesson is on [[frank-archon]]; the mechanism is on [[Radiance Conversion]].
|
||||
- `RAD_ABI_VERSION` covers every header a plugin compiles against. A plugin reporting a different
|
||||
number is refused at load, with both numbers printed. It moves on any shape change, not on a new
|
||||
optional export; rebuild when it changes. Release version and ABI are independent: `/version`
|
||||
reports the release, `rad-info --plugins` reports what loaded.
|
||||
|
||||
## Environment
|
||||
|
||||
| variable | meaning |
|
||||
|---|---|
|
||||
| `RADIANCE_HOME` | plugin search path: `:`-separated homes, each holding `kernels/`, `architectures/`, `quantizers/`; first wins, same-name files shadow later homes. Compiled-in default is `<prefix>/share/radiance` |
|
||||
| `RADIANCE_KERNELS` | default for `--kernels`: ordered kernel libraries, first wins; unnamed libraries are not loaded |
|
||||
| `RADIANCE_TUNE` | pin a kernel's tuned axes for a measurement (a measurement aid, not a deployment setting) |
|
||||
| `RADIANCE_CALIB_DIR` | a serve run records per-layer input Gram matrices here (the GPTQ calibration tap) |
|
||||
| `RADIANCE_CALIB_TOKENS` | tokens after which the tap stops and writes (default 65536) |
|
||||
| `RADIANCE_LOG`, `RADIANCE_LOG_STEPS` | log level; per-step token counts |
|
||||
| `RADIANCE_DEBUG_RESID` | residual-stream probe per (layer, site) |
|
||||
| `RADIANCE_DFLASH_Q_TEMP` | temperature of a DFlash2 coupled walk's candidate softmax |
|
||||
|
||||
The full list of debug knobs (`RADIANCE_DEBUG_*`, `RADIANCE_DUMP_*`, `RADIANCE_OP_ORACLE`, …) is in
|
||||
the source's `getenv` calls; none is a deployment setting.
|
||||
|
||||
## Where the pages are
|
||||
|
||||
| Page | Owns |
|
||||
|---|---|
|
||||
| [[Radiance Serve Options]] | every serve flag, grouped, with defaults and the law behind each knob |
|
||||
| [[Radiance HTTP API]] | endpoints, request fields, streaming, tool calls, refusal rules |
|
||||
| [[Radiance Conversion]] | `rad-convert`, the recipe language, quantisers, imports, reuse, `rad-export`, KL scoring |
|
||||
| [[Radiance Model Families]] | which plugin claims which model, and the conversion command per family |
|
||||
| [[Radiance Plugins]] | kernel and quantiser plugins: ABI, rows, hooks, encodings, build and checks |
|
||||
| [[Radiance Architecture Plugins]] | declaring a model: weights, buffers, KV groups, ops, step, probe |
|
||||
| [[Radiance Tools]] | `rad-info`, `rad-kbench`, `rad-tune`, `rad-schemas`, `rad-tokcheck`, harnesses |
|
||||
| [[Radiance Operations]] | the container contract, host prerequisites, readiness, ledgers, restarts |
|
||||
| [[frank-archon]] | the fleet's live radiance endpoint: shape, Eras, rollback, measured laws |
|
||||
|
||||
## Sources
|
||||
|
||||
- `README.md`, `docs/GUIDE.md` §1–§4, `docs/TOOLS.md`, `docs/PLUGIN.md`, `docs/DOCKER.md`,
|
||||
`docs/ARCHITECTURES.md`, `docs/OPS.md`, `spec.md` §2–§4, `abi/*.h`, `core/config.cpp`,
|
||||
`core/rad_core.h` at commit `8da043c` (codeberg.org/StillDeadcode/radiance).
|
||||
@@ -0,0 +1,198 @@
|
||||
---
|
||||
type: "reference"
|
||||
title: "Radiance HTTP API — endpoints, fields, refusals"
|
||||
description: "The HTTP surface radiance serves: endpoints, request fields, streaming, tools, and refusal rules."
|
||||
tags:
|
||||
- llm
|
||||
- radiance
|
||||
- api
|
||||
- reference
|
||||
timestamp: 2026-10-30
|
||||
---
|
||||
|
||||
# Radiance HTTP API
|
||||
|
||||
One server serves one model. It answers OpenAI's chat/completions/Responses/images/audio/video APIs,
|
||||
Anthropic's Messages API and llama-server's native one, so their clients and SDKs work unchanged.
|
||||
`model` in a request is accepted and **not enforced**.
|
||||
|
||||
## Endpoints
|
||||
|
||||
| route | what it serves |
|
||||
|---|---|
|
||||
| `POST /v1/chat/completions` | chat: tools, reasoning, streaming, grammars |
|
||||
| `POST /v1/completions` | raw text completion |
|
||||
| `POST /v1/embeddings` | an embedding head, else 501 |
|
||||
| `POST /v1/images/generations`, `/v1/images/edits` | OpenAI images API; needs an image model, else 501 |
|
||||
| `POST /v1/systemone` | LiquidAI d1 decisions: named yes/no, choice and score questions as probabilities; needs a d1 container, else 501 |
|
||||
| `POST /v1/audio/transcriptions` | OpenAI speech-to-text; needs a transcribing plugin, else 501 |
|
||||
| `GET /v1/realtime` (WebSocket) | streaming transcription session |
|
||||
| `POST /v1/audio/speech` | speech (Breeze-TTS-2, drama-3) or a song (MiniMax-Music3), whole or streamed |
|
||||
| `GET /v1/audio/speech/stream` (WebSocket) | text streamed in, each sentence's audio out |
|
||||
| `POST /v1/videos`; `GET /v1/videos[/:id[/content]]`; `DELETE /v1/videos/:id` | video jobs made in the background, downloaded as MP4 |
|
||||
| `POST /v1/messages`, `/v1/messages/count_tokens` | Anthropic Messages, for Claude Code and the Anthropic SDKs |
|
||||
| `POST /v1/responses`, `/v1/responses/input_tokens` | OpenAI Responses, for Codex and SDK clients |
|
||||
| `POST /completion`, `/completions`, `/infill` | llama-server native completion and fill-in-the-middle |
|
||||
| `GET /v1/models`, `/v1/models/:id` | what is loaded, with `max_model_len`, `root`, `capabilities` |
|
||||
| `GET /health`, `/v1/health`, `/ping`, `POST /ping` | liveness |
|
||||
| `GET /load`, `/version`, `/props`, `/server_info`, `/diagnostics` | load; release version; llama-server props; full configuration; the startup report as JSON (`/diagnostics` is same-machine-only without an API key) |
|
||||
| `POST /apply-template`, `/v1/chat/completions/input_tokens` | the rendered prompt; its token count |
|
||||
| `GET /metrics` | Prometheus |
|
||||
| `POST /tokenize`, `/detokenize`; `GET /get_tokenizer_info` | vLLM's shape and llama-server's at once |
|
||||
| `POST /reset_prefix_cache`; `DELETE /sessions/<label>`, `/sessions?id=<id>` | drop every cached prefix; forget a stored conversation |
|
||||
| `GET /`, `/dashboard`; `GET /stats`, `/graph`, `/sessions` | the dashboard and the JSON it is drawn from |
|
||||
|
||||
Unversioned shorthands exist for chat, chat input tokens, embeddings, images, audio transcriptions,
|
||||
audio speech, models, responses and completions, for a client given the bare host as base URL.
|
||||
`/completions` **without** `/v1` is llama-server's native completion. A path nothing serves is a JSON
|
||||
404.
|
||||
|
||||
A text prompt to `/v1/completions`, `/v1/embeddings` or `/tokenize` reads control tokens as tokens
|
||||
(as vLLM and llama.cpp do), so a client rendering its own template can send `<|im_start|>user\n…`.
|
||||
The model's declared BOS is prepended unless the prompt already opens with it.
|
||||
|
||||
## Chat
|
||||
|
||||
```sh
|
||||
curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
|
||||
-d '{"model":"m","messages":[{"role":"user","content":"hi"}],"max_tokens":200,"temperature":0.7}'
|
||||
```
|
||||
|
||||
- **No chat template → 501** on this route (the container carries one; `--override-chat-template`
|
||||
replaces it). Use `/v1/completions` instead.
|
||||
- Thinking off has three spellings: `chat_template_kwargs.enable_thinking=false`,
|
||||
top-level `enable_thinking=false`, or `reasoning_effort:"none"`.
|
||||
- Reasoning returns in `choices[].message.reasoning_content`, separate from `content`.
|
||||
**Reasoning counts against `max_tokens`**; a reply that runs out inside it has empty `content` and
|
||||
`finish_reason:"length"`. `usage.completion_tokens_details.reasoning_tokens` tells that case from a
|
||||
model that answered nothing.
|
||||
- `thinking_token_budget` (aliases `thinking_budget`, `thinking_budget_tokens`,
|
||||
`reasoning_budget_tokens`) closes the reasoning block at N tokens and forces the close, so it holds
|
||||
under speculation and at any temperature. It needs a one-token end marker; otherwise the response
|
||||
has no `completion_tokens_details` and a budget is a 400.
|
||||
- `continue_final_message` continues a conversation ending in an assistant message instead of
|
||||
answering it; `add_generation_prompt` opens a new turn.
|
||||
- Tool calls: `tools`, `tool_choice` (`auto`, `none`, `required`, named function),
|
||||
`parallel_tool_calls`. `required`/named hold the whole reply to calls after the reasoning block and
|
||||
cannot combine with `response_format` or `grammar`.
|
||||
- Structured output: `response_format` (`json_object` or `json_schema`), llama-server's `json_schema`
|
||||
shorthand, or a raw GBNF `grammar`. A chat template that drops the grammar makes the request a
|
||||
refusal.
|
||||
|
||||
## Streaming
|
||||
|
||||
```sh
|
||||
curl -N localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
|
||||
-d '{"model":"m","messages":[{"role":"user","content":"Count to ten."}],
|
||||
"max_tokens":100,"stream":true,"stream_options":{"include_usage":true}}'
|
||||
```
|
||||
|
||||
Server-sent events in OpenAI's delta format, terminated by `data: [DONE]`. **Nothing is sent while
|
||||
the prompt is read**: the first event (the `role: assistant` chunk) goes out with the first generated
|
||||
token, so a client timing its first event measures time-to-first-token. `return_progress` adds
|
||||
llama-server's `prompt_progress` chunks during prefill; `timings_per_token` adds `timings` to every
|
||||
chunk.
|
||||
|
||||
## Completions, embeddings, images, audio, video, tokens
|
||||
|
||||
- **Completions** — `/v1/completions` takes `prompt` (string or token ids), `echo`, `logprobs`,
|
||||
`preserved_tokens`. `/completion` is llama-server's native route; `/infill` is fill-in-the-middle.
|
||||
- **Embeddings** — `input` is text, a batch, or token ids; `encoding_format` `float`/`base64`;
|
||||
`dimensions` truncates to the first N and renormalises; `prompt_name` prepends the model's named
|
||||
prompt. Pooling is the model's; the engine does not renormalise.
|
||||
- **Images** — `/v1/images/generations` returns `b64_json`; `size` defaults to 2048×2048. Edits take
|
||||
one picture or up to ten (`image[]`), multipart as OpenAI's SDKs send them, with an optional mask
|
||||
(repaint only where transparent). `partial_images` streams previews as SSE.
|
||||
- **Speech to text** — `/v1/audio/transcriptions`; `/v1/realtime` streams audio in and words back as
|
||||
they are spoken. The model's prompt shape comes from its plugin's audio format, not from the server.
|
||||
- **Speech and music** — `/v1/audio/speech` (whole or streamed) with `instructions` describing a voice
|
||||
(Breeze-TTS-2) or speakers written in the input (`<|speaker:1|>`, `[laughs]` for drama-3); a music
|
||||
model takes the lyrics as `input` and the description as `instructions`. `/v1/audio/speech/stream`
|
||||
streams text in and audio out as each sentence completes.
|
||||
- **Video** — `/v1/videos` is a job: a video with its stereo soundtrack from a prompt, 5–14 s, up to
|
||||
1344×768, made in the background; `GET /v1/videos/:id/content` downloads the MP4.
|
||||
- **Tokenising** — `POST /tokenize` accepts vLLM's shape (`prompt` or `messages` in, `prompt` out) and
|
||||
llama-server's (`content` in/out, `with_pieces`). `POST /detokenize` reverses it.
|
||||
`GET /get_tokenizer_info` reports the tokeniser configuration.
|
||||
|
||||
## Request fields
|
||||
|
||||
Accepted fields, in OpenAI's, vLLM's and llama-server's spellings:
|
||||
|
||||
| field | notes |
|
||||
|---|---|
|
||||
| `model` | accepted, not enforced |
|
||||
| `messages` / `prompt` / `input` | chat / completion / embedding |
|
||||
| `max_tokens`, `max_completion_tokens`, `n_predict` | default is what the context leaves; `-1` is llama-server's "no limit"; clamped to context and to `--max-tokens-cap` |
|
||||
| `temperature`, `top_p`, `top_k`, `min_p`, `typical_p` | sampler |
|
||||
| `presence_penalty`, `frequency_penalty`, `repetition_penalty`/`repeat_penalty`, `repeat_last_n` | penalties |
|
||||
| `xtc_probability`, `xtc_threshold`, `dry_*` | XTC and DRY |
|
||||
| `seed` | absent means "pick one", not 0 |
|
||||
| `n` | 1..`--max-n`; prefix cache makes the fan-out nearly free after the first |
|
||||
| `stop` | string or array, bounded by `--max-stop-strings`/`--max-stop-bytes` |
|
||||
| `priority` | scheduler hint |
|
||||
| `ignore_eos` | generate to `max_tokens` whatever the model emits |
|
||||
| `stream`, `stream_options.include_usage`, `timings_per_token`, `return_progress` | streaming controls |
|
||||
| `samplers` | llama-server's order; this engine's chain is penalties, DRY, top-k, top-p, min-p, typical, XTC, with temperature first or last; any other list is refused by name |
|
||||
| `session_id` | labels the conversation on `/sessions`, dashboard and `DELETE` |
|
||||
| `control_vector` | `true` applies the deployment's vector to this request; absent takes the deployment default; a deployment with no vector refuses `true` |
|
||||
| `logprobs`, `top_logprobs`, `n_probs` | up to 20 alternatives; not with `echo` |
|
||||
| `tools`, `tool_choice`, `parallel_tool_calls`, legacy `functions`/`function_call` | chat only |
|
||||
| `add_generation_prompt`, `chat_template_kwargs`, `continue_final_message` | chat only |
|
||||
| `reasoning_format`, `reasoning_effort`, `enable_thinking`, `thinking_token_budget`, `preserve_thinking` | chat only; levels are the template's (`capabilities.reasoning_efforts`); `none`/`minimal` turn thinking off |
|
||||
| `response_format`, `json_schema`, `grammar` | structured output; `grammar` and `response_format` are mutually exclusive |
|
||||
| `echo`, `preserved_tokens` | `/v1/completions` only |
|
||||
| `encoding_format`, `dimensions`, `prompt_name`, `image_tokens` | `/v1/embeddings` only |
|
||||
| `images_config` | image models only: a named processor mode or an object restating served sizes |
|
||||
|
||||
## What is refused, ignored, and clamped
|
||||
|
||||
The contract: **a 200 means every field that could change the answer was honoured.** Fields fall
|
||||
into four classes.
|
||||
|
||||
- **Inert** — accepted and ignored because they cannot change the answer: OpenAI/OpenRouter
|
||||
bookkeeping (`user`, `metadata`, `store`, `service_tier`, `prompt_cache_key`, …) and llama-server's
|
||||
own speed/slot knobs (`cache_prompt`, `id_slot`, `n_cache_reuse`, `backend_sampling`,
|
||||
`speculative.*`, …). The prefix cache and speculation here are the server's and never change output.
|
||||
- **Guarded** — known, unimplemented, and able to change the answer: accepted at their off value,
|
||||
refused by name otherwise. `logit_bias` (empty), `mirostat` (0), `top_n_sigma` (≤0),
|
||||
`dynatemp_range` (≤0), `min_keep` (0), `post_sampling_probs` (false), `best_of` (≤1), `suffix`
|
||||
(empty), `lora` (empty — an adapter is chosen by `model`), `modalities`/`audio` (`["text"]`/null),
|
||||
`n_indent` (0), `t_max_predict_ms` (≤0), `return_tokens` (false), `grammar_lazy`,
|
||||
`grammar_triggers`, `preserved_tokens` (empty on chat).
|
||||
- **Ignored** — anything else unknown, logged at debug. An unknown field is far more often a client's
|
||||
extension than a misunderstood request.
|
||||
- **`max_tokens` is the exception**: if prompt + `max_tokens` exceeds the context, it is **clamped**
|
||||
and served, ending `finish_reason:"length"`, because refusing would produce no tokens instead of
|
||||
fewer. A prompt that does not fit on its own is a 400 `context_length_exceeded` naming
|
||||
`n_prompt_tokens`, `n_ctx`, and how many tokens must go.
|
||||
|
||||
## Anthropic, Responses, llama-server
|
||||
|
||||
- `/v1/messages` is translated into a chat request, so chat defaults, refusals and limits apply.
|
||||
Streamed (`message_start`, content blocks, `message_delta`, `message_stop`) or not:
|
||||
`ANTHROPIC_BASE_URL=http://host:8000 ANTHROPIC_API_KEY=$KEY claude` works. `/v1/messages/count_tokens`
|
||||
returns the prompt length.
|
||||
- `/v1/responses` maps `instructions`/`input` (strings, messages, `function_call`,
|
||||
`function_call_output`, `reasoning` items) into the chat request; `max_output_tokens`,
|
||||
`reasoning.effort`, `text.format` map to chat fields; undefined fields pass through. The event stream
|
||||
carries `output_index`, `content_index`, `sequence_number`, and opens with `response.created` and
|
||||
`response.in_progress`. A reply cut at `max_output_tokens` is `incomplete`. Function tools a
|
||||
`namespace` groups are named `<namespace>__<name>`; built-in tools and freeform custom tools are
|
||||
left out.
|
||||
- `/completion`, `/completions`, `/infill` are llama-server's own shapes, for llama.vim/llama.vscode.
|
||||
|
||||
## Auth, CORS, errors
|
||||
|
||||
- `--api-key KEY` requires `Authorization: Bearer KEY` or `X-Api-Key: KEY` on every request except
|
||||
`/health`, `/v1/health`, `/ping` and the dashboard page. An empty key is refused at startup.
|
||||
- CORS headers are sent unless `--no-cors`.
|
||||
- Errors are JSON. Over-context is 400 `context_length_exceeded`; a full queue is 429 with
|
||||
`Retry-After`; a capability the model lacks is 501; a guarded field is 400 naming the field and its
|
||||
accepted off value.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Radiance Serve Options]] — the defaults every omitted field takes.
|
||||
- [[Radiance Operations]] — `/health` semantics, metrics, dashboard.
|
||||
- [[frank-archon]] — the harness contract built on this surface.
|
||||
Loaded 100 of 131 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user