deploy: make Playwright e2e a mandatory post-deploy gate

The 2026-09-12 homepage outage slipped through because the live-fleet
Playwright suite was a manual 'MUST invoke' that deploys skipped. Wire it
into deploy.sh as the [3/3] gate: after a behavior-affecting deploy,
run tests/e2e/run.sh against the live fleet; a failure exits DEPLOY
INCOMPLETE. Add --skip-e2e as an emergency escape hatch. Update the
AGENTS.md deployment-completeness contract and flow diagram to match.

Verified: bash -n clean, --skip-e2e parsed, unknown flags rejected,
seam suite green (only the pre-existing comfyui test 73 fails).
This commit is contained in:
Frank
2026-09-12 11:50:40 -07:00
parent 476b21b85d
commit 09c54113b1
2 changed files with 36 additions and 3 deletions
+10 -3
View File
@@ -148,9 +148,12 @@ only when BOTH of the following pass:
1. **Unit/config tests** — `./tests/run.sh` (all seams green), and
2. **Playwright end-to-end tests** — `tests/e2e/run.sh` drives the LIVE fleet
(homepage, Anubis/Authentik guard chain, guarded vhosts) in a real
browser. The E2E suite is deliberately excluded from `./tests/run.sh`
because it needs live hosts and credentials, so it MUST be invoked
explicitly after every behavior-affecting deploy.
browser. As of 2026-09-12 this runs **automatically** as the `[3/3]` gate
at the end of `deploy.sh`: a behavior-affecting deploy that fails the e2e
suite exits `DEPLOY INCOMPLETE` and must not be treated as good. The suite
stays out of `./tests/run.sh` (it needs live hosts plus the `qa-e2e`
credentials in `tests/e2e/.env.local`). Bypass only with
`deploy.sh --skip-e2e` in an emergency.
```mermaid
graph LR
@@ -161,6 +164,10 @@ graph LR
E --> F[Target pulls with ff-only]
F --> G[deploy-remote.sh dispatches selected service]
G --> H[Per-service health check]
H --> I[Playwright e2e gate on live fleet]
I --> J{e2e green?}
J -->|yes| K[Deploy complete]
J -->|no| L[DEPLOY INCOMPLETE]
```
## Boot Persistence & Self-Healing (MANDATORY)
+26
View File
@@ -29,6 +29,7 @@ Usage: deploy.sh <dankclaw|dankserver|frankbook-pro> [options]
Options:
--service NAME deploy only NAME (repeatable; bypasses automatic planning)
--dry-run show the plan without pushing or connecting to a target
--skip-e2e skip the mandatory Playwright live-fleet gate (emergency only)
-h, --help show this help
Examples:
@@ -74,6 +75,7 @@ case "$TARGET" in
esac
DRY_RUN=0
SKIP_E2E=0
EXPLICIT_SERVICES=()
while [[ "$#" -gt 0 ]]; do
case "$1" in
@@ -81,6 +83,10 @@ while [[ "$#" -gt 0 ]]; do
DRY_RUN=1
shift
;;
--skip-e2e)
SKIP_E2E=1
shift
;;
--service)
[[ "$#" -ge 2 ]] || { echo "--service requires a name" >&2; exit 2; }
EXPLICIT_SERVICES+=("$2")
@@ -233,4 +239,24 @@ exec "$REMOTE_SHELL" ./scripts/deploy-remote.sh "$TARGET" "$NEW_REVISION" "$@" \
--from-revision "$OLD_REVISION"
REMOTE_EOF
# --- Playwright e2e gate (MANDATORY after behavior-affecting deploys) ---
# The unit seams in tests/run.sh cannot see the live guard chain, LAN DNS, or
# the rendered homepage — exactly the layers that broke on 2026-09-12. This gate
# drives the LIVE fleet (Anubis -> Authentik outpost -> apps) and fails the
# deploy on any regression. It needs tests/e2e/.env.local with the qa-e2e IdP
# credentials (see .env.local.example). Use --skip-e2e for an emergency only.
if [[ "$SKIP_E2E" -eq 1 ]]; then
echo "[3/3] e2e gate SKIPPED (--skip-e2e): live fleet NOT verified." >&2
else
echo "[3/3] Running Playwright e2e gate against the live fleet..."
if bash "$REPO_ROOT/tests/e2e/run.sh"; then
echo "e2e gate PASSED."
else
echo "=== DEPLOY INCOMPLETE: e2e gate FAILED on $TARGET ===" >&2
echo "The live-fleet Playwright suite did not pass. Do NOT treat this" >&2
echo "deploy as good. Re-run manually: tests/e2e/run.sh" >&2
exit 1
fi
fi
echo "=== Deploy complete: $TARGET ==="