diff --git a/AGENTS.md b/AGENTS.md index 6a8da83..519f114 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -74,6 +74,14 @@ unrelated container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless runs on a host without passwordless sudo; the value never enters the repo. +**`deploy.sh ` with no `--service` is a fleet-wide DRAIN, not a sync.** +It also walks every per-service revision marker under the deploy-state dir and +restarts each service whose marker is behind HEAD *and* whose paths changed +since that marker — so a backlog accumulated by service-scoped deploys fires +all at once, and one failing health gate aborts the run with every later +service left pending. Name the service unless a drain is what you want +(`./scripts/deploy.sh dankserver --service frank-primary`). + ### Deployment completeness (MANDATORY) A deploy is **not complete** when `deploy.sh` exits green. It requires: diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 95addee..0452f0a 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -1070,3 +1070,39 @@ top of the proven-good base, root cause first. - Rollback for the pre-shape-D fp8+MTP4 era stays git tag `frank-primary-fp8-mtp-20261001`. Runbook [[dankserver-qwen38-fp8-expert-offload-tuning]]. + +## 2026-10-02 - plex: the readiness gate probed HTTP against a TLS-only service + +Found by accident, and it had been latent since the secure-connections posture +was set: `deploy_plex` gated on `curl -fsS http://127.0.0.1:32400/identity`, +which **cannot** succeed. PMS drops plain HTTP off-loopback - the probe's +connection arrives on the container's eth0 - so Plex accepts the TCP connection +and then answers with nothing, and curl reports "Empty reply from server" (52). + +- Trigger: a bare `./scripts/deploy.sh dankserver` (no `--service`) drains the + whole pending backlog, not just the service you were thinking about. plex's + per-service revision had been behind for long enough that this gate had never + run against the current posture. It burned its full 600 s and failed the run; + every service after plex in the order was left undeployed. +- **Plex was healthy the entire time.** `https://127.0.0.1:32400/identity` + returned 200 with the MediaContainer XML in 44 ms while the gate was still + timing out. The in-container `http://127.0.0.1:32400/identity` also answers + 200 - loopback is the one path PMS serves in the clear, which is exactly why + the host-side probe could not see the difference. +- Evidence chain that settled it: `podman exec plex curl http://127.0.0.1:32400` + = 200; `curl http://127.0.0.1:32400` from the host = empty reply; + `/proc/net/tcp` inside the container showed `00000000:7E90` LISTEN plus stuck + ESTABLISHED pairs on `1703590A:7E90` (10.89.3.23, the container's own eth0 + address) that never produced a response. +- **The repo already knew this law, in a different file:** + `tests/caddyfile.bats` pins the plex vhosts to a TLS upstream with the comment + "PMS drops plain HTTP off-loopback". The deployer's gate never got the same + treatment. Fixed to `curl -fsSk --max-time 3 https://127.0.0.1:32400/identity` + (`-k`: PMS serves its own self-signed cert on 32400). +- Re-deployed `39f0e0d`: plex recorded deployed, the *arr services recorded + (compose-recorded on this host, no restart attempted), tautulli deployed, + e2e 33/33. Verified after: `plex-https=200`, plex container up, tautulli + healthy, frank-primary untouched (`health=200`, NRestarts=0). +- **Operating lesson: `deploy.sh ` with no `--service` is a fleet-wide + drain.** It restarts every service whose recorded revision is behind HEAD and + whose paths changed since. To touch one service, name it.