docs: the bare-deploy drain law + the plex gate fix in the work log

`deploy.sh <target>` with no `--service` also walks the per-service revision
markers and restarts everything behind HEAD whose paths changed — a drain, not
a sync. That is how the latent plex gate bug surfaced: the backlog fired all
at once and plex's http:// readiness probe, which can never succeed against a
PMS that drops plain HTTP off-loopback, burned its full 600 s and aborted the
run with every later service left pending.

Both facts recorded: AGENTS.md gains the drain law under Deployment, log.md
gains the plex entry with the evidence chain.
This commit is contained in:
Frank
2026-10-02 12:26:55 -07:00
parent 39f0e0d8d4
commit 812193aa63
2 changed files with 44 additions and 0 deletions
+36
View File
@@ -1070,3 +1070,39 @@ top of the proven-good base, root cause first.
- Rollback for the pre-shape-D fp8+MTP4 era stays git tag
`frank-primary-fp8-mtp-20261001`. Runbook
[[dankserver-qwen38-fp8-expert-offload-tuning]].
## 2026-10-02 - plex: the readiness gate probed HTTP against a TLS-only service
Found by accident, and it had been latent since the secure-connections posture
was set: `deploy_plex` gated on `curl -fsS http://127.0.0.1:32400/identity`,
which **cannot** succeed. PMS drops plain HTTP off-loopback - the probe's
connection arrives on the container's eth0 - so Plex accepts the TCP connection
and then answers with nothing, and curl reports "Empty reply from server" (52).
- Trigger: a bare `./scripts/deploy.sh dankserver` (no `--service`) drains the
whole pending backlog, not just the service you were thinking about. plex's
per-service revision had been behind for long enough that this gate had never
run against the current posture. It burned its full 600 s and failed the run;
every service after plex in the order was left undeployed.
- **Plex was healthy the entire time.** `https://127.0.0.1:32400/identity`
returned 200 with the MediaContainer XML in 44 ms while the gate was still
timing out. The in-container `http://127.0.0.1:32400/identity` also answers
200 - loopback is the one path PMS serves in the clear, which is exactly why
the host-side probe could not see the difference.
- Evidence chain that settled it: `podman exec plex curl http://127.0.0.1:32400`
= 200; `curl http://127.0.0.1:32400` from the host = empty reply;
`/proc/net/tcp` inside the container showed `00000000:7E90` LISTEN plus stuck
ESTABLISHED pairs on `1703590A:7E90` (10.89.3.23, the container's own eth0
address) that never produced a response.
- **The repo already knew this law, in a different file:**
`tests/caddyfile.bats` pins the plex vhosts to a TLS upstream with the comment
"PMS drops plain HTTP off-loopback". The deployer's gate never got the same
treatment. Fixed to `curl -fsSk --max-time 3 https://127.0.0.1:32400/identity`
(`-k`: PMS serves its own self-signed cert on 32400).
- Re-deployed `39f0e0d`: plex recorded deployed, the *arr services recorded
(compose-recorded on this host, no restart attempted), tautulli deployed,
e2e 33/33. Verified after: `plex-https=200`, plex container up, tautulli
healthy, frank-primary untouched (`health=200`, NRestarts=0).
- **Operating lesson: `deploy.sh <target>` with no `--service` is a fleet-wide
drain.** It restarts every service whose recorded revision is behind HEAD and
whose paths changed since. To touch one service, name it.