docs: the bare-deploy drain law + the plex gate fix in the work log
`deploy.sh <target>` with no `--service` also walks the per-service revision markers and restarts everything behind HEAD whose paths changed — a drain, not a sync. That is how the latent plex gate bug surfaced: the backlog fired all at once and plex's http:// readiness probe, which can never succeed against a PMS that drops plain HTTP off-loopback, burned its full 600 s and aborted the run with every later service left pending. Both facts recorded: AGENTS.md gains the drain law under Deployment, log.md gains the plex entry with the evidence chain.
This commit is contained in:
@@ -1070,3 +1070,39 @@ top of the proven-good base, root cause first.
|
||||
- Rollback for the pre-shape-D fp8+MTP4 era stays git tag
|
||||
`frank-primary-fp8-mtp-20261001`. Runbook
|
||||
[[dankserver-qwen38-fp8-expert-offload-tuning]].
|
||||
|
||||
## 2026-10-02 - plex: the readiness gate probed HTTP against a TLS-only service
|
||||
|
||||
Found by accident, and it had been latent since the secure-connections posture
|
||||
was set: `deploy_plex` gated on `curl -fsS http://127.0.0.1:32400/identity`,
|
||||
which **cannot** succeed. PMS drops plain HTTP off-loopback - the probe's
|
||||
connection arrives on the container's eth0 - so Plex accepts the TCP connection
|
||||
and then answers with nothing, and curl reports "Empty reply from server" (52).
|
||||
|
||||
- Trigger: a bare `./scripts/deploy.sh dankserver` (no `--service`) drains the
|
||||
whole pending backlog, not just the service you were thinking about. plex's
|
||||
per-service revision had been behind for long enough that this gate had never
|
||||
run against the current posture. It burned its full 600 s and failed the run;
|
||||
every service after plex in the order was left undeployed.
|
||||
- **Plex was healthy the entire time.** `https://127.0.0.1:32400/identity`
|
||||
returned 200 with the MediaContainer XML in 44 ms while the gate was still
|
||||
timing out. The in-container `http://127.0.0.1:32400/identity` also answers
|
||||
200 - loopback is the one path PMS serves in the clear, which is exactly why
|
||||
the host-side probe could not see the difference.
|
||||
- Evidence chain that settled it: `podman exec plex curl http://127.0.0.1:32400`
|
||||
= 200; `curl http://127.0.0.1:32400` from the host = empty reply;
|
||||
`/proc/net/tcp` inside the container showed `00000000:7E90` LISTEN plus stuck
|
||||
ESTABLISHED pairs on `1703590A:7E90` (10.89.3.23, the container's own eth0
|
||||
address) that never produced a response.
|
||||
- **The repo already knew this law, in a different file:**
|
||||
`tests/caddyfile.bats` pins the plex vhosts to a TLS upstream with the comment
|
||||
"PMS drops plain HTTP off-loopback". The deployer's gate never got the same
|
||||
treatment. Fixed to `curl -fsSk --max-time 3 https://127.0.0.1:32400/identity`
|
||||
(`-k`: PMS serves its own self-signed cert on 32400).
|
||||
- Re-deployed `39f0e0d`: plex recorded deployed, the *arr services recorded
|
||||
(compose-recorded on this host, no restart attempted), tautulli deployed,
|
||||
e2e 33/33. Verified after: `plex-https=200`, plex container up, tautulli
|
||||
healthy, frank-primary untouched (`health=200`, NRestarts=0).
|
||||
- **Operating lesson: `deploy.sh <target>` with no `--service` is a fleet-wide
|
||||
drain.** It restarts every service whose recorded revision is behind HEAD and
|
||||
whose paths changed since. To touch one service, name it.
|
||||
|
||||
Reference in New Issue
Block a user