tests: gate the hermes vhost, and stop specs from silently not running

Two gaps behind the same near-miss.

1. **No spec covered hermes.franksdank.cloud** — the one guarded vhost whose
   upstream is a host address (`reverse_proxy 192.168.0.143:8787`) rather than a
   container name, because its service is host-networked. That is how a
   loopback-only bind could take the vhost down with a 502 and leave every gate
   green. tests/e2e/specs/17-hermes-vhost.spec.ts asserts what a dead upstream
   violates: no 5xx through the guard chain, no Caddy error page, and no app
   bytes to an unauthenticated raw client. It does not assert SSO-silence, since
   this app has its own password login.

2. **Specs are matched by number prefix, not globbed.** A spec file whose prefix
   is in no project silently never runs — which is exactly what happened to the
   new one until it was registered. The prefix is registered now, the comment in
   playwright.config.ts says why, and tests/e2e_scaffold.bats fails if any
   spec file matches no project.

The hermes-webui seam assertion is corrected too: it asserted the loopback bind
I wrongly imposed, and now asserts the real contract — the unit's bind must serve
the address the Caddyfile fronts it on.

serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md

@@ -116,13 +116,23 @@ Before first deploy, set a strong `HERMES_WEBUI_PASSWORD` in the live
 
 - The `/webui` mount is **rw on purpose**: `bootstrap.py` creates `.venv` in
   the repo root. Do not flip it to `:ro`.
-- The webui binds `0.0.0.0:8787` today, so the SSH tunnel is an access path for
-  the phone, not a security boundary: any LAN host that can route to
-  `192.168.0.143` reaches the API and needs only `HERMES_WEBUI_PASSWORD`. The
-  webui can run agent commands, so treat that password as a credential with
-  command-execution blast radius. To make loopback-only binding true again, set
-  `HERMES_WEBUI_HOST=127.0.0.1` in the unit and require the tunnel for every
-  client — that is an exposure change to decide deliberately, not a silent fix.
+- The webui binds `0.0.0.0:8787`, and **two consumers depend on that**: the SSH
+  tunnel (which dials the host's `localhost`) *and* Caddy, which fronts
+  `hermes.franksdank.cloud` with `reverse_proxy 192.168.0.143:8787` — a host
+  address rather than a container name, because this unit is host-networked and
+  there is no port forwarder to bridge the gap (`dankclaw/Caddyfile` records the
+  same constraint). So the SSH tunnel is an access path for the phone, not a
+  security boundary: any LAN host that can route to `192.168.0.143` reaches the
+  API and needs only `HERMES_WEBUI_PASSWORD`. The webui can run agent commands,
+  so treat that password as a credential with command-execution blast radius.
+  To make loopback-only binding true again you must set
+  `HERMES_WEBUI_HOST=127.0.0.1` **and** give Caddy another way in (a
+  host-gateway route or a bridge-network unit) — that is an exposure change to
+  decide deliberately, not a silent fix. It was tried on 2026-09-16 as a silent
+  fix and broke the vhost with a 502 while the whole e2e suite stayed green;
+  `tests/e2e/specs/17-hermes-vhost.spec.ts` now asserts that vhost reaches its
+  app through the guard chain, so a dead upstream fails a gate instead of
+  hiding.
 - The webui and the gateway share one data root (`/data` is a bind mount of
   `/home/dank/Containers/hermes/data`). A config change made on one side takes
   effect when the other restarts (same caveat as the dashboard).

tests/e2e/playwright.config.ts

@@ -24,7 +24,12 @@ export default defineConfig({
     },
     {
       name: "standard",
-      testMatch: /(10|12|13|16)-.*\.spec\.ts/,
+      // NOTE: specs are matched by NUMBER PREFIX, not globbed. A spec file
+      // whose prefix is not listed in any project silently never runs — a
+      // green suite with a test that does not exist. 17- joined on 2026-09-16
+      // (the hermes vhost, whose upstream is a host address rather than a
+      // container name, so it needs its own pass condition).
+      testMatch: /(10|12|13|16|17)-.*\.spec\.ts/,
       use: { storageState: ".auth/qa-e2e.json" },
       dependencies: ["setup"],
     },

tests/e2e/specs/17-hermes-vhost.spec.ts

@@ -0,0 +1,54 @@
+import { expect, test } from "@playwright/test";
+
+// hermes.franksdank.cloud fronts the Hermex webui, which is a HOST-NETWORKED
+// container: Caddy cannot reach it by container name, so the vhost's upstream
+// is the host's LAN address (`reverse_proxy 192.168.0.143:8787` in
+// dankclaw/Caddyfile) and the app must therefore bind beyond loopback.
+//
+// That coupling broke on 2026-09-16: the unit's bind was tightened to
+// 127.0.0.1 for a stricter posture, which killed the upstream and left the
+// vhost returning 502 — and the whole e2e suite stayed green, because no spec
+// covered this hostname. It is the only guarded vhost whose upstream is a host
+// address rather than a container name, so it needs its own assertion.
+//
+// The app's own password login is expected once the guard chain passes (unlike
+// the SSO-silent vhosts in 13-sso), so this spec asserts only what a broken
+// upstream would violate: no 5xx, no gate page, and no Caddy error page.
+
+const VHOST = "https://hermes.franksdank.cloud/";
+
+test("hermes vhost reaches its app through the guard chain", async ({ page }) => {
+  test.setTimeout(180_000);
+
+  // An unreachable upstream is a 502 from Caddy; nothing else in the chain
+  // produces one, which is what makes this the load-bearing assertion.
+  const resp = await page.request.get(VHOST, { maxRedirects: 5 });
+  expect(resp.status()).toBeLessThan(500);
+
+  await page.goto(VHOST);
+  await expect
+    .poll(
+      async () => !(await page.locator("body").innerText()).includes("Making sure you're not a bot"),
+      { timeout: 90_000 },
+    )
+    .toBe(true);
+
+  const body = await page.locator("body").innerText();
+  // Caddy's own error text must never be what a user sees here.
+  expect(body).not.toContain("Bad Gateway");
+  expect(body).not.toContain("502");
+  // The guard's error surface, as opposed to the app's own login.
+  expect(body).not.toContain("Powered by authentik");
+  expect(body).not.toContain("Making sure you're not a bot");
+});
+
+test("hermes vhost does not leak app bytes to an unauthenticated raw client", async ({
+  request,
+}) => {
+  // Same rule as 15-unauth-fetch: a raw client solves no challenge, so it must
+  // get the gate, not the application.
+  const resp = await request.get(VHOST, { maxRedirects: 0 });
+  expect(resp.status()).toBeLessThan(500);
+  const body = await resp.text().catch(() => "");
+  expect(body).not.toContain("Powered by authentik");
+});

tests/e2e_scaffold.bats

@@ -46,3 +46,24 @@ PY
         return 1
     fi
 }
+
+@test "every e2e spec is matched by a playwright project" {
+    # Playwright projects match specs by NUMBER PREFIX, not by glob, so a spec
+    # whose prefix is not listed in any project silently never runs: the suite
+    # reports green while the test does not exist. That happened on 2026-09-16
+    # with 17-hermes-vhost.spec.ts — written to catch a real regression (a
+    # loopback-only bind that killed hermes.franksdank.cloud's upstream while
+    # every gate stayed green), and invisible until the prefix was registered.
+    local cfg="$REPO_ROOT/tests/e2e/playwright.config.ts"
+    python3 - "$cfg" <<'PYEOF'
+import pathlib, re, sys
+cfg = pathlib.Path(sys.argv[1]).read_text()
+patterns = [re.compile(p) for p in re.findall(r"testMatch:\s*/(.+?)/,", cfg)]
+assert patterns, "no testMatch patterns found in the playwright config"
+specs = sorted(pathlib.Path(sys.argv[1]).parent.joinpath("specs").glob("*.spec.ts"))
+assert specs, "no spec files found"
+unmatched = [s.name for s in specs if not any(p.search(s.name) for p in patterns)]
+assert not unmatched, f"specs matched by no project (they will never run): {unmatched}"
+print(f"{len(specs)} specs, all matched by {len(patterns)} projects")
+PYEOF
+}

tests/hermes_webui.bats

@@ -38,19 +38,22 @@ webui_service_yaml() {
     grep -q -- '--foreground' <<< "$y"
 }
 
-@test "hermes-webui binds loopback, not the LAN" {
-    local y
+@test "hermes-webui's bind serves the address Caddy fronts it on" {
+    local y caddy
     y=$(webui_service_yaml)
+    caddy="$REPO_ROOT/dankclaw/Caddyfile"
     grep -q 'HERMES_WEBUI_PASSWORD' <<< "$y" \
         || { echo "password gate missing"; return 1; }
-    # With Network=host there is no port forwarder: the app's own bind IS the
-    # exposure. The compose service this replaced published 127.0.0.1:8787:8787,
-    # and an early version of the unit bound 0.0.0.0 instead — which answered
-    # 200 on 192.168.0.143:8787 for anyone on the LAN (measured 2026-09-16).
-    grep -q 'Environment=HERMES_WEBUI_HOST=127.0.0.1' <<< "$y" \
-        || { echo "webui must bind loopback (HERMES_WEBUI_HOST=127.0.0.1)"; return 1; }
-    ! grep -q 'HERMES_WEBUI_HOST=0.0.0.0' <<< "$y" \
-        || { echo "webui must not bind 0.0.0.0 on a host network"; return 1; }
+    # This unit is host-networked, so Caddy cannot reach it by container name:
+    # hermes.franksdank.cloud proxies to the host's LAN address. The app's bind
+    # is therefore load-bearing for that vhost — a loopback-only bind was tried
+    # on 2026-09-16 and broke it (502), while host networking means there is no
+    # port forwarder to bridge the gap.
+    local upstream
+    upstream=$(grep -A 6 '^hermes\.franksdank\.cloud {' "$caddy" | grep -oE 'reverse_proxy [0-9.]+:[0-9]+' | head -1)
+    [ -n "$upstream" ] || { echo "no hermes vhost upstream in the Caddyfile"; return 1; }
+    grep -q 'Environment=HERMES_WEBUI_HOST=0.0.0.0' <<< "$y" \
+        || { echo "webui must bind all interfaces: $upstream cannot reach loopback"; return 1; }
 }
 
 @test "hermes-webui shares the gateway data volume and mounts the webui checkout" {