docs: the bare-deploy drain law + the plex gate fix in the work log
`deploy.sh <target>` with no `--service` also walks the per-service revision markers and restarts everything behind HEAD whose paths changed — a drain, not a sync. That is how the latent plex gate bug surfaced: the backlog fired all at once and plex's http:// readiness probe, which can never succeed against a PMS that drops plain HTTP off-loopback, burned its full 600 s and aborted the run with every later service left pending. Both facts recorded: AGENTS.md gains the drain law under Deployment, log.md gains the plex entry with the evidence chain.
This commit is contained in:
1 parent
39f0e0d8d4
commit
812193aa63
2 files changed
+44
No files matched your search
@@ -74,6 +74,14 @@ unrelated
|
|||||||
container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
|
container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
|
||||||
runs on a host without passwordless sudo; the value never enters the repo.
|
runs on a host without passwordless sudo; the value never enters the repo.
|
||||||
|
|
||||||
|
**`deploy.sh <target>` with no `--service` is a fleet-wide DRAIN, not a sync.**
|
||||||
|
It also walks every per-service revision marker under the deploy-state dir and
|
||||||
|
restarts each service whose marker is behind HEAD *and* whose paths changed
|
||||||
|
since that marker — so a backlog accumulated by service-scoped deploys fires
|
||||||
|
all at once, and one failing health gate aborts the run with every later
|
||||||
|
service left pending. Name the service unless a drain is what you want
|
||||||
|
(`./scripts/deploy.sh dankserver --service frank-primary`).
|
||||||
|
|
||||||
### Deployment completeness (MANDATORY)
|
### Deployment completeness (MANDATORY)
|
||||||
|
|
||||||
A deploy is **not complete** when `deploy.sh` exits green. It requires:
|
A deploy is **not complete** when `deploy.sh` exits green. It requires:
|
||||||
|
|||||||
@@ -1070,3 +1070,39 @@ top of the proven-good base, root cause first.
|
|||||||
- Rollback for the pre-shape-D fp8+MTP4 era stays git tag
|
- Rollback for the pre-shape-D fp8+MTP4 era stays git tag
|
||||||
`frank-primary-fp8-mtp-20261001`. Runbook
|
`frank-primary-fp8-mtp-20261001`. Runbook
|
||||||
[[dankserver-qwen38-fp8-expert-offload-tuning]].
|
[[dankserver-qwen38-fp8-expert-offload-tuning]].
|
||||||
|
|
||||||
|
## 2026-10-02 - plex: the readiness gate probed HTTP against a TLS-only service
|
||||||
|
|
||||||
|
Found by accident, and it had been latent since the secure-connections posture
|
||||||
|
was set: `deploy_plex` gated on `curl -fsS http://127.0.0.1:32400/identity`,
|
||||||
|
which **cannot** succeed. PMS drops plain HTTP off-loopback - the probe's
|
||||||
|
connection arrives on the container's eth0 - so Plex accepts the TCP connection
|
||||||
|
and then answers with nothing, and curl reports "Empty reply from server" (52).
|
||||||
|
|
||||||
|
- Trigger: a bare `./scripts/deploy.sh dankserver` (no `--service`) drains the
|
||||||
|
whole pending backlog, not just the service you were thinking about. plex's
|
||||||
|
per-service revision had been behind for long enough that this gate had never
|
||||||
|
run against the current posture. It burned its full 600 s and failed the run;
|
||||||
|
every service after plex in the order was left undeployed.
|
||||||
|
- **Plex was healthy the entire time.** `https://127.0.0.1:32400/identity`
|
||||||
|
returned 200 with the MediaContainer XML in 44 ms while the gate was still
|
||||||
|
timing out. The in-container `http://127.0.0.1:32400/identity` also answers
|
||||||
|
200 - loopback is the one path PMS serves in the clear, which is exactly why
|
||||||
|
the host-side probe could not see the difference.
|
||||||
|
- Evidence chain that settled it: `podman exec plex curl http://127.0.0.1:32400`
|
||||||
|
= 200; `curl http://127.0.0.1:32400` from the host = empty reply;
|
||||||
|
`/proc/net/tcp` inside the container showed `00000000:7E90` LISTEN plus stuck
|
||||||
|
ESTABLISHED pairs on `1703590A:7E90` (10.89.3.23, the container's own eth0
|
||||||
|
address) that never produced a response.
|
||||||
|
- **The repo already knew this law, in a different file:**
|
||||||
|
`tests/caddyfile.bats` pins the plex vhosts to a TLS upstream with the comment
|
||||||
|
"PMS drops plain HTTP off-loopback". The deployer's gate never got the same
|
||||||
|
treatment. Fixed to `curl -fsSk --max-time 3 https://127.0.0.1:32400/identity`
|
||||||
|
(`-k`: PMS serves its own self-signed cert on 32400).
|
||||||
|
- Re-deployed `39f0e0d`: plex recorded deployed, the *arr services recorded
|
||||||
|
(compose-recorded on this host, no restart attempted), tautulli deployed,
|
||||||
|
e2e 33/33. Verified after: `plex-https=200`, plex container up, tautulli
|
||||||
|
healthy, frank-primary untouched (`health=200`, NRestarts=0).
|
||||||
|
- **Operating lesson: `deploy.sh <target>` with no `--service` is a fleet-wide
|
||||||
|
drain.** It restarts every service whose recorded revision is behind HEAD and
|
||||||
|
whose paths changed since. To touch one service, name it.
|
||||||
Reference in new issue
Block a user