Frank
ab06e4c221
Audit of all 12 e2e specs for the 'green-but-vacuous' class that broke
Grafana's SSO. The suite is otherwise sound (incident-driven, leakage-based);
two real coverage gaps surfaced:
- notebook.franksdank.cloud (open-notebook, deployed 2026-10-19) was a guarded
vhost in NEITHER the 13-/14- GUARDED lists NOR any dedicated spec. A broken
upstream would stay invisible - the exact 17-hermes class. Added to the 14-
guard audit (unauth -> authentik gate, 'Open Notebook' leak marker) plus a
new 21- that proves authenticated reachability to open-notebook's own login
through the guard chain, with a positive app marker.
- prometheus.franksdank.cloud was only ever asserted '<500 and not a login
wall' (13-/11-), which a 404 also passes. New 22- adds the app-is-really-
there proof 20- gives Grafana: SSO handoff + its own /api/v1/status/buildinfo
answers {status:success}.
Both specs wired into the standard project (playwright testMatch), which the
e2e_scaffold bats gate enforces. Verified live against the fleet; full bats
suite green.
67 lines
2.4 KiB
TypeScript
67 lines
2.4 KiB
TypeScript
import { defineConfig } from "@playwright/test";
|
|
import { BASE_URL, REAL_UA } from "./lib/env";
|
|
|
|
export default defineConfig({
|
|
testDir: "./specs",
|
|
timeout: 90_000,
|
|
expect: { timeout: 20_000 },
|
|
retries: 1,
|
|
workers: 2,
|
|
fullyParallel: false,
|
|
reporter: [["list"]],
|
|
outputDir: "./test-results",
|
|
use: {
|
|
baseURL: BASE_URL,
|
|
userAgent: REAL_UA,
|
|
ignoreHTTPSErrors: true,
|
|
viewport: { width: 1440, height: 900 },
|
|
trace: "retain-on-failure",
|
|
},
|
|
projects: [
|
|
{
|
|
name: "setup",
|
|
testMatch: /00-auth\.setup\.ts/,
|
|
},
|
|
{
|
|
name: "standard",
|
|
// NOTE: specs are matched by NUMBER PREFIX, not globbed. A spec file
|
|
// whose prefix is not listed in any project silently never runs — a
|
|
// green suite with a test that does not exist. 17- joined on 2026-09-16
|
|
// (the hermes vhost, whose upstream is a host address rather than a
|
|
// container name, so it needs its own pass condition). 18- joined on
|
|
// 2026-09-24 (OpenWebUI SSO through authentik). 20- joined on 2026-10-21
|
|
// (Grafana's OAuth2 relying-party login, which the edge-only 13- never
|
|
// follows). 21- joined 2026-10-21 (the open-notebook vhost, deployed
|
|
// 2026-10-19 - the newest guarded app, in neither GUARDED list nor any
|
|
// spec until now: the 17-hermes "uncovered vhost broke silently" class).
|
|
// 22- joins Prometheus's app-is-really-there proof (its own API), closing
|
|
// the "not a login wall but maybe a 404" hole 20- closed for Grafana.
|
|
testMatch: /(10|12|13|16|17|18|20|21|22)-.*\.spec\.ts/,
|
|
use: { storageState: ".auth/qa-e2e.json" },
|
|
dependencies: ["setup"],
|
|
},
|
|
{
|
|
name: "admin",
|
|
testMatch: /11-.*\.spec\.ts/,
|
|
use: { storageState: ".auth/qa-e2e-admin.json" },
|
|
dependencies: ["setup"],
|
|
},
|
|
{
|
|
name: "guest",
|
|
testMatch: /(14|15)-.*\.spec\.ts/,
|
|
use: { storageState: { cookies: [], origins: [] } },
|
|
},
|
|
{
|
|
name: "frankbook",
|
|
// The *arr/qBittorrent walk (19-). Excluded from default runs and from
|
|
// the deploy e2e gate unless a frankbook trigger path changed — the
|
|
// links terminate on FrankBook-Pro, which sleeps (see tests/e2e/run.sh
|
|
// and scripts/deploy.sh). Matched by number prefix like every project
|
|
// here; joined 2026-10-05.
|
|
testMatch: /19-.*\.spec\.ts/,
|
|
use: { storageState: ".auth/qa-e2e-admin.json" },
|
|
dependencies: ["setup"],
|
|
},
|
|
],
|
|
});
|