Reviewed all 12 e2e specs for the 'green-but-vacuous' class that let Grafana's SSO stay broken. The suite is otherwise strong: every spec has incident-driven rationale and anti-vacuous assertions (registry-driven exact-set compares in 10-/11-, leakage markers not status codes in 15-, positive app content in 16-marreta). 20-grafana was the one shallow spec (fixed separately in #54).
Two real gaps found and closed
notebook.franksdank.cloud (open-notebook, deployed 2026-10-19) — a guarded vhost in neither the GUARDED list of 13-/14- nor any dedicated spec. The exact 17-hermes 'uncovered vhost broke silently' class.
Added notebook to the 14- guard audit (unauth -> authentik gate; Open Notebook leak marker). Verified live: unauth hits default-authentication-flow with the notebook redirect_uri.
New 21-notebook.spec.ts: authenticated reachability through the guard to open-notebook's own login, with a positive app marker (title Open Notebook + 'Enter your password'), not just 'not a login wall'.
prometheus.franksdank.cloud — only ever asserted <500 + not-a-login-wall (13-/11-), which a 404 also passes. New 22-prometheus-sso.spec.ts: the app-is-really-there proof #54 gave Grafana — silent SSO handoff + its own /api/v1/status/buildinfo returns {status:success} (verified: 200, Prometheus 3.15.0).
Deliberately NOT hardened
search/romm/files/seerr keep their 'reached app edge, no IdP shell' scope (13-/12-): after SSO they show their own login (copyparty, romm, seerr), so a positive logged-in marker would be wrong to assert — same reasoning as hermes. Only the apps that SSO into a full session (Grafana, Prometheus) get the API proof.
Evidence
playwright test --list: 21-/22- matched into standard; notebook gate into guest (Total 38).
## Audit
Reviewed all 12 e2e specs for the **'green-but-vacuous'** class that let Grafana's SSO stay broken. The suite is otherwise strong: every spec has incident-driven rationale and anti-vacuous assertions (registry-driven exact-set compares in 10-/11-, leakage markers not status codes in 15-, positive app content in 16-marreta). **20-grafana was the one shallow spec** (fixed separately in #54).
## Two real gaps found and closed
1. **`notebook.franksdank.cloud`** (open-notebook, deployed 2026-10-19) — a guarded vhost in **neither** the `GUARDED` list of 13-/14- **nor** any dedicated spec. The exact **17-hermes 'uncovered vhost broke silently'** class.
- Added `notebook` to the 14- guard audit (unauth -> authentik gate; `Open Notebook` leak marker). Verified live: unauth hits `default-authentication-flow` with the notebook redirect_uri.
- New **21-notebook.spec.ts**: authenticated reachability through the guard to open-notebook's own login, with a positive app marker (title `Open Notebook` + 'Enter your password'), not just 'not a login wall'.
2. **`prometheus.franksdank.cloud`** — only ever asserted `<500 + not-a-login-wall` (13-/11-), which a **404 also passes**. New **22-prometheus-sso.spec.ts**: the app-is-really-there proof #54 gave Grafana — silent SSO handoff + its own `/api/v1/status/buildinfo` returns `{status:success}` (verified: 200, Prometheus 3.15.0).
## Deliberately NOT hardened
search/romm/files/seerr keep their 'reached app edge, no IdP shell' scope (13-/12-): after SSO they show their **own** login (copyparty, romm, seerr), so a positive logged-in marker would be wrong to assert — same reasoning as hermes. Only the apps that SSO into a full session (Grafana, Prometheus) get the API proof.
## Evidence
- `playwright test --list`: 21-/22- matched into **standard**; notebook gate into **guest** (Total 38).
- Live run: **21 ✓ 1.4s, 22 ✓ 1.5s, notebook gate ✓ 1.9s**.
- `e2e_scaffold` bats test 6 ('every e2e spec is matched by a playwright project') passes — the gate that would catch an unwired spec.
- Full `./tests/run.sh`: **472 passed**.
---
cwd: /tmp/wt-e2e-audit · session: wt-20261020-125416-6d1fc1f
Audit of all 12 e2e specs for the 'green-but-vacuous' class that broke
Grafana's SSO. The suite is otherwise sound (incident-driven, leakage-based);
two real coverage gaps surfaced:
- notebook.franksdank.cloud (open-notebook, deployed 2026-10-19) was a guarded
vhost in NEITHER the 13-/14- GUARDED lists NOR any dedicated spec. A broken
upstream would stay invisible - the exact 17-hermes class. Added to the 14-
guard audit (unauth -> authentik gate, 'Open Notebook' leak marker) plus a
new 21- that proves authenticated reachability to open-notebook's own login
through the guard chain, with a positive app marker.
- prometheus.franksdank.cloud was only ever asserted '<500 and not a login
wall' (13-/11-), which a 404 also passes. New 22- adds the app-is-really-
there proof 20- gives Grafana: SSO handoff + its own /api/v1/status/buildinfo
answers {status:success}.
Both specs wired into the standard project (playwright testMatch), which the
e2e_scaffold bats gate enforces. Verified live against the fleet; full bats
suite green.
dank
merged commit 961713e92d into main2026-10-20 21:21:44 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Audit
Reviewed all 12 e2e specs for the 'green-but-vacuous' class that let Grafana's SSO stay broken. The suite is otherwise strong: every spec has incident-driven rationale and anti-vacuous assertions (registry-driven exact-set compares in 10-/11-, leakage markers not status codes in 15-, positive app content in 16-marreta). 20-grafana was the one shallow spec (fixed separately in #54).
Two real gaps found and closed
notebook.franksdank.cloud(open-notebook, deployed 2026-10-19) — a guarded vhost in neither theGUARDEDlist of 13-/14- nor any dedicated spec. The exact 17-hermes 'uncovered vhost broke silently' class.notebookto the 14- guard audit (unauth -> authentik gate;Open Notebookleak marker). Verified live: unauth hitsdefault-authentication-flowwith the notebook redirect_uri.Open Notebook+ 'Enter your password'), not just 'not a login wall'.prometheus.franksdank.cloud— only ever asserted<500 + not-a-login-wall(13-/11-), which a 404 also passes. New 22-prometheus-sso.spec.ts: the app-is-really-there proof #54 gave Grafana — silent SSO handoff + its own/api/v1/status/buildinforeturns{status:success}(verified: 200, Prometheus 3.15.0).Deliberately NOT hardened
search/romm/files/seerr keep their 'reached app edge, no IdP shell' scope (13-/12-): after SSO they show their own login (copyparty, romm, seerr), so a positive logged-in marker would be wrong to assert — same reasoning as hermes. Only the apps that SSO into a full session (Grafana, Prometheus) get the API proof.
Evidence
playwright test --list: 21-/22- matched into standard; notebook gate into guest (Total 38).e2e_scaffoldbats test 6 ('every e2e spec is matched by a playwright project') passes — the gate that would catch an unwired spec../tests/run.sh: 472 passed.cwd: /tmp/wt-e2e-audit · session: wt-20261020-125416-6d1fc1f
Audit of all 12 e2e specs for the 'green-but-vacuous' class that broke Grafana's SSO. The suite is otherwise sound (incident-driven, leakage-based); two real coverage gaps surfaced: - notebook.franksdank.cloud (open-notebook, deployed 2026-10-19) was a guarded vhost in NEITHER the 13-/14- GUARDED lists NOR any dedicated spec. A broken upstream would stay invisible - the exact 17-hermes class. Added to the 14- guard audit (unauth -> authentik gate, 'Open Notebook' leak marker) plus a new 21- that proves authenticated reachability to open-notebook's own login through the guard chain, with a positive app marker. - prometheus.franksdank.cloud was only ever asserted '<500 and not a login wall' (13-/11-), which a 404 also passes. New 22- adds the app-is-really- there proof 20- gives Grafana: SSO handoff + its own /api/v1/status/buildinfo answers {status:success}. Both specs wired into the standard project (playwright testMatch), which the e2e_scaffold bats gate enforces. Verified live against the fleet; full bats suite green.