docs(vault): destroy the dead weight — 53 pages and the change diary
The vault was 155 pages / 31,210 lines, most of it measurement ledgers for payloads that no longer run. A throughput figure for a runtime nobody operates is not a record: AI moves faster than the vault, so the number is wrong before anyone reads it, and a wrong number in a runbook gets believed. Per the new purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed. Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest): - the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM, the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two "Running Qwen3.6 …" pages; - the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag) and the multi-device/x4-pair/hybrid/offload experiment pages; - every vLLM page — no vLLM endpoint exists on this fleet; - `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff, self-hosted app census, the TTS survey) that were obsolete by construction; - `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the page that owns the thing it governs, never to a diary. Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras tables keep the window, payload, runtime, reason and rollback pointer; the rate columns are blanked. The live shapes stay verbatim — those numbers are what the deployer and `models.yml` consume.
This commit is contained in:
Frank
@@ -93,7 +93,7 @@ and `hermesadmin.` (app-enforced passwords, no public DNS for the first,
|
||||
(`dankclaw/Caddyfile:241-251`), and the FrankBook-Pro *arr/qBittorrent vhosts
|
||||
(`dankclaw/Caddyfile:260-352`). There is also no vhost at all for ComfyUI, so
|
||||
it is exempt by absence — and therefore unauthenticated on the LAN at
|
||||
`comfyui.franksdank.cloud:8188`. The guarded set is exactly `www.`, `search.`,
|
||||
`comfyui.franksdank.cloud:8188`. The guarded set is exactly `www.`, `search.`
|
||||
`grafana.`, `prometheus.`, `loki.`, `seerr.`, `marreta.`, `romm.`, `files.`
|
||||
(`dankclaw/Caddyfile:283-423`).
|
||||
|
||||
@@ -140,7 +140,7 @@ it is exempt by absence — and therefore unauthenticated on the LAN at
|
||||
instead of allowed, the browser's fetch throws and it renders its own
|
||||
error page (iOS/macOS browsers phrase it as "the request failed and the
|
||||
interceptors did not return an alternative response") instead of the
|
||||
login flow. Verify with a JS-capable client end-to-end — mint via
|
||||
login flow. Verify with a JS-capable client end-to-end — mint
|
||||
`make-challenge`, solve, submit with browser-exact query params — before
|
||||
declaring a challenge chain healthy.
|
||||
7. **Single-file bind mount + git pull = stale policy.** The policy is
|
||||
@@ -158,7 +158,7 @@ it is exempt by absence — and therefore unauthenticated on the LAN at
|
||||
answers any request lacking `X-Real-Ip` with a 500 whose log line reads
|
||||
`[misconfiguration] X-Real-Ip header is not set`. The real chain always
|
||||
sets it (every guarded Caddy listener carries `header_up X-Real-Ip
|
||||
{remote_host}` — the bats seam enforces the line), so users never see
|
||||
{remote_host}` — the bats seam enforces the line), so users never
|
||||
this. But any direct probe of `:8923` (scratch container, curl, a new
|
||||
reverse-proxy hop) MUST set the header or it looks like a policy failure.
|
||||
Scratch-verify with `-H "X-Real-Ip: 192.168.0.20"`.
|
||||
|
||||
@@ -23,8 +23,8 @@ read-only `SELECT`s, 2026-10-18). Nine proxy providers, all attached to outpost
|
||||
`7e39655b-58d3-4fe6-a80a-513f31051217` ("authentik Embedded Outpost", 9
|
||||
members in `authentik_outposts_outpost_providers`), all with
|
||||
`internal_host = http://caddy:8017`, all carrying the **same three flow
|
||||
UUIDs** (`default-authentication-flow`,
|
||||
`default-provider-authorization-implicit-consent`,
|
||||
UUIDs** (`default-authentication-flow`
|
||||
`default-provider-authorization-implicit-consent`
|
||||
`default-provider-invalidation-flow`), and each backed by an Application
|
||||
(the `slug` column below):
|
||||
|
||||
@@ -67,7 +67,7 @@ Two provisioning rules, each verified live by an outage:
|
||||
reload** — including hosts whose providers were fine (verified 2026-09-02
|
||||
when three providers provisioned without any flows took down every guarded
|
||||
vhost after a server restart). Backfill all three flows (copy from pk 1:
|
||||
`default-authentication-flow`,
|
||||
`default-authentication-flow`
|
||||
`default-provider-authorization-implicit-consent` (the implicit-consent
|
||||
authorization flow, `default-invalidation-flow`) and the retry loop
|
||||
recovers on its next 3-second cycle. The silent killer: while the outpost
|
||||
@@ -94,7 +94,7 @@ implementation diverges from upstream in ways that bit us; all verified live:
|
||||
contains `"authorization_code"` (enum value with underscore), the token
|
||||
endpoint rejects every exchange with `invalid_request`.
|
||||
- **Token endpoint is form-encoded + Basic-auth only.** A JSON body makes
|
||||
`parse_token_request` see no `grant_type` → `unsupported_grant_type`;
|
||||
`parse_token_request` see no `grant_type` → `unsupported_grant_type`
|
||||
POST-body client secrets are not accepted — authenticate the confidential
|
||||
client with `Authorization: Basic base64(client_id:client_secret)`.
|
||||
- **Provider resolution is by `client_id` query param** on the global
|
||||
@@ -107,7 +107,7 @@ implementation diverges from upstream in ways that bit us; all verified live:
|
||||
becomes the *expression source* instead of the context; every call raises a
|
||||
swallowed `PropertyMappingExpressionException` and the claim is skipped.
|
||||
Scope-grant filtering has the same class of bug: `ScopeMapping` has no
|
||||
provider FK (linkage lives in the `ProviderPropertyMapping` through model),
|
||||
provider FK (linkage lives in the `ProviderPropertyMapping` through model)
|
||||
so `filter(provider=...)` always yields an empty queryset and tokens asked
|
||||
to carry scopes get zero. Net effect: userinfo/introspection answers with
|
||||
`{sub}` only — email/groups/entitlements claims cannot be delivered by this
|
||||
@@ -156,7 +156,7 @@ Hard-won rules (the first is now contested by the live database):
|
||||
and retest.
|
||||
- `ak shell` import path for the provider model in 2026.8:
|
||||
`from authentik.providers.ldap.models import LDAPProvider`.
|
||||
*(unverified 2026-10-18: importing it outside a configured Django shell fails
|
||||
- (unverified 2026-10-18: importing it outside a configured Django shell fails
|
||||
with `ImproperlyConfigured`, and no `ak shell` was driven against the live
|
||||
auth service for this check)*
|
||||
|
||||
@@ -188,13 +188,13 @@ is a `password` field on the `default-user-settings` prompt
|
||||
|
||||
Two ORM gotchas: `Flow.stages.add()` fails on this through model
|
||||
(`Can't bulk create a multi-table inherited model` — `FlowStageBinding` is
|
||||
MTI), so create the `FlowStageBinding` row directly with an explicit `order`;
|
||||
MTI), so create the `FlowStageBinding` row directly with an explicit `order`
|
||||
and a prompt field is a `Prompt` row linked through `PromptStageField`
|
||||
(`prompt_stage` + `prompt`; the order comes from `Prompt.order`).
|
||||
|
||||
**A blank password must not set an empty password.** `set_password("")` would
|
||||
make the account log in with no password, so a validation policy
|
||||
(`default-user-settings-no-empty-password`, bound to the prompt via
|
||||
(`default-user-settings-no-empty-password`, bound to the prompt
|
||||
`PromptStageValidationPolicy`) denies the form when `prompt_data["password"]`
|
||||
is `""` and allows it when set. Verified with the `PolicyEvaluator`
|
||||
(empty → deny, set → allow, absent → allow) and end-to-end by driving
|
||||
@@ -213,7 +213,7 @@ three `*_flow_id` columns; the proxy table is
|
||||
`authentik_providers_proxy_proxyprovider` (doubled `proxy`) and its PK column
|
||||
really is **`oauth2provider_ptr_id`** (proxy providers inherit the OAuth2
|
||||
table in 2026.8), the LDAP table is
|
||||
`authentik_providers_ldap_ldapprovider` with PK `provider_ptr_id`;
|
||||
`authentik_providers_ldap_ldapprovider` with PK `provider_ptr_id`
|
||||
`authentik_core_application` attaches to a provider through `provider_id`
|
||||
(its PK is `policybindingmodel_ptr_id`, not `id`), and `authentik_outposts_outpost`
|
||||
is keyed by `uuid`. So `Provider.objects.filter(type=…)` raising `FieldError`
|
||||
|
||||
@@ -15,20 +15,20 @@ application-level authentication with its proxy outpost. Not wrapped: Gitea,
|
||||
OpenWebUI, Jellyfin (LDAP outpost), the *arr stack, Pi-hole (basicauth edge) —
|
||||
they handle their own session. Grafana is the exception that is both: it is a
|
||||
guarded vhost (`grafana-proxy`, pk 8, locked to Authentik SSO) *and* an OAuth2
|
||||
relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132`,
|
||||
relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132`
|
||||
`dankclaw/Caddyfile:374-377`; live provider table in [[Authentik Outpost Reference]]).
|
||||
|
||||
- **Version:** `2026.8.3` (pinned in `dankclaw/authentik/quadlet/*.container`;
|
||||
the outpost's boot log reads `"Booting authentik" … "version": "2026.8.3"`,
|
||||
- **Version:** `2026.8.3` (pinned in `dankclaw/authentik/quadlet/*.container`
|
||||
the outpost's boot log reads `"Booting authentik" … "version": "2026.8.3"`
|
||||
2026-10-18)
|
||||
- **Host:** dankclaw (`192.168.0.143`)
|
||||
- **Name:** `auth.franksdank.cloud`
|
||||
- **Stack: five quadlet units, no compose.** `authentik-postgres`,
|
||||
`authentik-server`, `authentik-worker` (`deploy_authentik`,
|
||||
- **Stack: five quadlet units, no compose.** `authentik-postgres`
|
||||
`authentik-server`, `authentik-worker` (`deploy_authentik`
|
||||
`scripts/deploy-remote.sh:980-1004`) plus two outposts — `authentik-outpost`
|
||||
(the Rust **proxy** outpost that actually serves `/outpost.goauthentik.io/*`,
|
||||
(the Rust **proxy** outpost that actually serves `/outpost.goauthentik.io/*`
|
||||
`deploy_authentik_outpost`, `scripts/deploy-remote.sh:1006-1018`) and
|
||||
`authentik-outpost-ldap` (`deploy_authentik_outpost_ldap`,
|
||||
`authentik-outpost-ldap` (`deploy_authentik_outpost_ldap`
|
||||
`scripts/deploy-remote.sh:1020-1032`). All five report `generated` +
|
||||
`active`, `Restart=always`, `WantedBy=default.target`, with linger on
|
||||
(`systemctl --user`, 2026-10-18). The "3 compose services" wording is the
|
||||
@@ -36,7 +36,7 @@ relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132`,
|
||||
- **LDAP outpost:** serves LDAP for Jellyfin — see the section below.
|
||||
|
||||
**Split off from this page:** [[Authentik Processes]] owns the per-container
|
||||
facts for `authentik-postgres`, `authentik-server` and `authentik-worker`;
|
||||
facts for `authentik-postgres`, `authentik-server` and `authentik-worker`
|
||||
[[Authentik Outpost Reference]] owns the provider/flow/token lookup material —
|
||||
the live proxy-provider inventory, the proxy-provider provisioning rules, the
|
||||
OAuth2 relying-party quirks, the LDAP provider objects, the user-settings flow
|
||||
@@ -136,7 +136,7 @@ routing table, and the Authentik proxy-provider list must be the same set of
|
||||
hosts. A guarded host with no provider 404s at login time; a provider with no
|
||||
DNS record breaks off-LAN clients.
|
||||
|
||||
The live proxy-provider inventory (the per-vhost `external_host` → app mapping),
|
||||
The live proxy-provider inventory (the per-vhost `external_host` → app mapping)
|
||||
the two provisioning rules the outpost enforces, and this fork's OAuth2
|
||||
relying-party quirks are reference lookup: [[Authentik Outpost Reference]].
|
||||
|
||||
@@ -144,7 +144,7 @@ relying-party quirks are reference lookup: [[Authentik Outpost Reference]].
|
||||
|
||||
The second outpost — `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`
|
||||
→ `authentik-outpost-ldap.service`, one of the five units in this stack —
|
||||
serves LDAP for Jellyfin's LDAP-Auth plugin (`enabled`=generated, `active`,
|
||||
serves LDAP for Jellyfin's LDAP-Auth plugin (`enabled`=generated, `active`
|
||||
`Restart=always`, container up, 2026-10-18). The image is pinned by its **amd64
|
||||
single-platform digest**
|
||||
`ghcr.io/goauthentik/ldap@sha256:2e7b259eb8c37faac916946ddf48ec681ee9d3a9f724fe8521bf161973f13cf9`
|
||||
@@ -159,7 +159,7 @@ rule at the current `2026.8.3` pin. Resolve once with `podman manifest inspect
|
||||
.digest'` and pin that digest.
|
||||
|
||||
- Ports **389** (ldap) and **636** (ldaps) publish on the LAN
|
||||
(`PublishPort=389:3389`, `636:6636`,
|
||||
(`PublishPort=389:3389`, `636:6636`
|
||||
`dankclaw/authentik/quadlet/authentik-outpost-ldap.container:30-31`; `ss -ltnp`
|
||||
shows `0.0.0.0:389` and `0.0.0.0:636`, 2026-10-18) and Dankserver can reach
|
||||
them — a bare TCP connect from Dankserver to `192.168.0.143:389` succeeds
|
||||
@@ -196,7 +196,7 @@ every outpost session (see the ORM playbook).
|
||||
core quadlets, starts postgres health-gated (`wait_container_healthy
|
||||
authentik-postgres 120`), then the server (300 s health wait) and the worker.
|
||||
The outposts are separate handlers: `--service authentik-outpost` and
|
||||
`--service authentik-outpost-ldap` (`scripts/deploy-remote.sh:1006-1032`,
|
||||
`--service authentik-outpost-ldap` (`scripts/deploy-remote.sh:1006-1032`
|
||||
dispatch at `:1579-1588`).
|
||||
2. Open `https://auth.franksdank.cloud` — the initial setup wizard runs. It
|
||||
creates the superuser (email + password) and generates the DB schema.
|
||||
@@ -210,7 +210,7 @@ every outpost session (see the ORM playbook).
|
||||
4. For apps with native OIDC: add an **OAuth2/OpenID Connect** provider, create
|
||||
the app with the correct redirect URI, and paste the `client_id` /
|
||||
`client_secret` into the app's SSO settings. The relying parties that exist
|
||||
today are `grafana` (pk 9), `Seerr OIDC` (pk 12) and `Open WebUI` (pk 14);
|
||||
today are `grafana` (pk 9), `Seerr OIDC` (pk 12) and `Open WebUI` (pk 14)
|
||||
**Gitea has no provider row at all** (live provider table, 2026-10-18) — it
|
||||
authenticates itself and stays off the chain by design
|
||||
(`dankclaw/Caddyfile:110-120`).
|
||||
@@ -243,7 +243,7 @@ every outpost session (see the ORM playbook).
|
||||
The embedded outpost does not re-evaluate group membership per request. Each
|
||||
completed authorize writes one row to `authentik_providers_proxy_proxysession`
|
||||
whose `session_data` JSON holds the claims (`groups` included) that get served
|
||||
for that browser session until the row expires (observed TTLs 1–24 h per
|
||||
for that browser session until the row expires (observed TTLs 1–24 h
|
||||
row). A session authorized **before** a user was added to a group keeps
|
||||
serving the old group list — Caddy-side RBAC header gates then 403 despite
|
||||
correct membership, and no authentik event is generated (the 403 happens at
|
||||
@@ -309,7 +309,7 @@ client → TLS edge → Anubis :8923 → outpost decision → internal
|
||||
000 (connection refused) — the post-auth listener lives in the caddy
|
||||
container's netns; probe inside it or grep the config. Non-browser clients
|
||||
cannot solve the JS challenge — before flipping, audit service-to-service
|
||||
traffic: ingestion/push paths must target internal addresses (`loki:3100`),
|
||||
traffic: ingestion/push paths must target internal addresses (`loki:3100`)
|
||||
never a public hostname of a guarded service. Exempt-by-construction
|
||||
categories stay unguarded: the IdP itself (`auth.`), git smart-HTTP
|
||||
endpoints, services with their own complete auth — document the exemption
|
||||
@@ -336,14 +336,14 @@ EOF
|
||||
`sed -n '/objects imported/,$p'` to drop the noise *(unverified 2026-10-18:
|
||||
the exact banner text and the 229-object count — no `ak shell` was driven
|
||||
against the live auth service for this audit)*.
|
||||
- Import paths (each wrong guess costs a failed exec): `User`, `Group`,
|
||||
- Import paths (each wrong guess costs a failed exec): `User`, `Group`
|
||||
`Application`, `ApplicationEntitlement`, `PropertyMapping` →
|
||||
`authentik.core.models`; `ProxyProvider` →
|
||||
`authentik.providers.proxy.models` (NOT `authentik.outposts.models`);
|
||||
`LDAPProvider` → `authentik.providers.ldap.models`;
|
||||
`authentik.providers.proxy.models` (NOT `authentik.outposts.models`)
|
||||
`LDAPProvider` → `authentik.providers.ldap.models`
|
||||
`authentik.core.models.application` does NOT exist as an import path; there
|
||||
is no `authentik.policies.group` module in this build.
|
||||
*(unverified 2026-10-18: these module paths were not re-executed — importing
|
||||
- (unverified 2026-10-18: these module paths were not re-executed — importing
|
||||
outside a configured Django shell fails with `ImproperlyConfigured`)*
|
||||
- The live database's table and column names (the 2026.x renames, the MTI parent
|
||||
keys, the read-only `psql` audit recipe) are reference lookup:
|
||||
@@ -376,7 +376,7 @@ downstream —
|
||||
`category: user` panels from `services.json`.
|
||||
- `openwebui_users` / `openwebui_admins` → ApplicationEntitlements
|
||||
**"Open WebUI Users"** and **"Open WebUI Admins"**, both rows attached to the
|
||||
`Open WebUI` application (`authentik_core_applicationentitlement`,
|
||||
`Open WebUI` application (`authentik_core_applicationentitlement`
|
||||
2026-10-18), consumed by the "Open WebUI roles" property mapping → `roles`
|
||||
claim *(unverified 2026-10-18: the group→entitlement binding and the mapping
|
||||
expression were not read; the mapping itself is not visible under that name
|
||||
@@ -392,7 +392,7 @@ downstream —
|
||||
artem 16, kirby 17, ry 18, nic 19, debo 20, freya 21, mitch 22, eric 23 —
|
||||
and every one carries the `<user>@franksdank.cloud` address the template
|
||||
prescribes. Creation dates (2026-10-07 → 2026-10-15) were not re-checked
|
||||
*(unverified 2026-10-18)*. Template: `username` +
|
||||
- (unverified 2026-10-18)*. Template: `username` +
|
||||
`<user>@franksdank.cloud` + display name + `is_active=True` + the password
|
||||
set via `set_password`, membership in `Non-Admin Users` + `jellyfin_users` +
|
||||
`openwebui_users`. `alex` was recreated from scratch on 2026-10-08 into this
|
||||
@@ -400,7 +400,7 @@ downstream —
|
||||
the row carried no MFA device, token, entitlement, or proxy session, and
|
||||
events survive a user delete because `Event.user` is a denormalized JSON
|
||||
field, not a FK).
|
||||
- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`,
|
||||
- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`
|
||||
`openwebui_admins`, `authentik Admins`, `authentik Read-only`.
|
||||
- Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty,
|
||||
Plex, Seerr, Marreta) need no per-user group: any authenticated user
|
||||
@@ -429,8 +429,8 @@ More field gotchas (each cost a failed exec): `ApplicationEntitlement.applicatio
|
||||
can be null (guard with `getattr`); `LDAPProvider` has
|
||||
`search_mode`/`bind_mode`/`base_dn` and lacks `search_base`/`of_groups`/
|
||||
`organization` — the backing table
|
||||
(`authentik_providers_ldap_ldapprovider`) has exactly `provider_ptr_id`,
|
||||
`base_dn`, `certificate_id`, `tls_server_name`, `gid_start_number`,
|
||||
(`authentik_providers_ldap_ldapprovider`) has exactly `provider_ptr_id`
|
||||
`base_dn`, `certificate_id`, `tls_server_name`, `gid_start_number`
|
||||
`uid_start_number`, `search_mode`, `bind_mode`, `mfa_support`
|
||||
(`information_schema.columns`, 2026-10-18); `OAuth2Provider` has no `.scopes`
|
||||
(scopes are linked through `ProviderPropertyMapping`, which is why
|
||||
@@ -484,5 +484,5 @@ podman run --rm --network container:<container> --entrypoint /bin/busybox \
|
||||
- [Jellyfin](../Dankserver/Jellyfin.md) — consumes the LDAP outpost via its LDAP-Auth plugin
|
||||
- [Network Architecture](../Architecture/Network%20Architecture.md) — the flat naming scheme
|
||||
- Research: `.docaudit/research/auth.md` — the full provider comparison
|
||||
*(the file is gone from the repo and from the host checkout as of
|
||||
- (the file is gone from the repo and from the host checkout as of
|
||||
2026-10-18)*
|
||||
|
||||
@@ -47,7 +47,7 @@ torrent.
|
||||
WIREGUARD_PERSISTENT_KEEPALIVE_INTERVAL=25s
|
||||
|
||||
The private key is hashed into the VM env file and never enters the repo.
|
||||
NAT-PMP inbound forwarding is available on Proton (gluetun supports it via
|
||||
NAT-PMP inbound forwarding is available on Proton (gluetun supports it
|
||||
`FIREWALL_INPUT_PORTS` + `NATPMP=true`) — deliberately deferred: the e2e
|
||||
gate passes outbound-only, and swarm peers needing an inbound port is a
|
||||
seeding-era concern, not a grab-only one.
|
||||
@@ -86,7 +86,7 @@ are unaffected.
|
||||
got `Unauthenticated`); the live conf was patched and
|
||||
`scripts/deploy-remote.sh` fixed to write the `_PBKDF2` key. Gotcha
|
||||
(live, 2026-09-01): after qB 5.x ran, its conf rewrite left a hash line
|
||||
4.6.7 rejects (`invalid credentials` for every login, including the arrs);
|
||||
4.6.7 rejects (`invalid credentials` for every login, including the arrs)
|
||||
rewriting the same `@ByteArray` line fresh (same password, 100k
|
||||
iterations) restored login. Any future version flip: regenerate the line.
|
||||
- `HostHeaderValidation=false` + `ServerDomains=*`: the socat forwarder
|
||||
@@ -98,7 +98,7 @@ are unaffected.
|
||||
path (no auto-TMM), so the default itself points at `Finished Torrents`:
|
||||
every grab lands there regardless of client behavior; human review moves
|
||||
files onward; nothing writes to the entropy volume.
|
||||
- Categories seeded: `Finished Torrents`, `lidarr`, `radarr`, `readarr`,
|
||||
- Categories seeded: `Finished Torrents`, `lidarr`, `radarr`, `readarr`
|
||||
`tv-sonarr` (all → `Finished Torrents`).
|
||||
- **Download clients** (gateway `10.88.0.1:8080`, user `admin`): all five test
|
||||
green against live qB — sonarr (`tv-sonarr`), radarr, lidarr, readarr
|
||||
@@ -109,12 +109,12 @@ are unaffected.
|
||||
as "Authentication failure" — that kills runtime auth too, not just API
|
||||
creates; the fix was pinning qB itself (below), after which readarr tests
|
||||
200 like the rest.
|
||||
- **qB pinned to 4.6.7** (`Image=lscr.io/linuxserver/qbittorrent:4.6.7`,
|
||||
- **qB pinned to 4.6.7** (`Image=lscr.io/linuxserver/qbittorrent:4.6.7`
|
||||
not `latest`): `latest` resolved to 5.x, whose `204` login breaks
|
||||
Readarr's frozen client; 4.6.7 answers the legacy `200` + `"Ok."` that
|
||||
every client accepts. Pinned in
|
||||
`frankbook-pro/qbittorrent/quadlet/qbittorrent.container`.
|
||||
- **Prowlarr Applications** push to sonarr/radarr/lidarr (`fullSync`,
|
||||
- **Prowlarr Applications** push to sonarr/radarr/lidarr (`fullSync`
|
||||
gateway ports). Readarr deliberately omitted: prowlarr's indexer-push
|
||||
test to readarr's `/api/v1/indexer/test` 500s (EOL schema drift).
|
||||
Legacy `Prowlarr-*` indexer rows carried in the migrated dankserver DBs
|
||||
|
||||
@@ -46,7 +46,7 @@ The Prometheus datasource is declared in git, not clicked into `grafana.db`:
|
||||
address Caddy proxies to, never the loopback-published port — and carries
|
||||
`timeInterval: 15s` so Grafana's rate windows cannot be shorter than the
|
||||
scrape interval (`prometheus.yml:5`). The `llm-fleet` dashboard is provisioned
|
||||
alongside it by `dashboards/provider.yml` (provider `fleet`,
|
||||
alongside it by `dashboards/provider.yml` (provider `fleet`
|
||||
`updateIntervalSeconds: 30`, `allowUiUpdates: true`).
|
||||
|
||||
A Loki datasource is still **not** provisioned — the tree contains exactly one
|
||||
@@ -70,7 +70,7 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-
|
||||
Basic client auth, form-encoded token body, and the fork's userinfo-claims
|
||||
bug that leaves identity at the `sub` hash only) is in
|
||||
[[Authentik Outpost Reference]] → "OAuth2 relying-party quirks". Consequences for Grafana:
|
||||
The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true`,
|
||||
The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true`
|
||||
`ORG_ID=1` and **no** role-attribute mapping (`grafana.container:52-55`), so
|
||||
users land as Viewer with `login = <sub hex>`; no email/groups/role claims
|
||||
reach Grafana until the evaluator bug is fixed — do not write role mappings
|
||||
@@ -79,7 +79,7 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-
|
||||
the host without the database or an admin password. *(unverified 2026-10-18)*)
|
||||
- **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That
|
||||
file is a gluetun env file and it carries `HOME=/root`; systemd imports it
|
||||
for the service, podman then looks for its configuration under `/root`,
|
||||
for the service, podman then looks for its configuration under `/root`
|
||||
cannot read it, and the unit dies at start with
|
||||
`stat /root/.config: permission denied`. The unit reads
|
||||
`/home/dank/Containers/grafana/env`, which the deployer seeds with exactly
|
||||
|
||||
@@ -51,7 +51,7 @@ See [[Container Host Reachability]] for the netns rules.
|
||||
published nothing → structurally unprobeable from the host).
|
||||
3. Probe from the **correct vantage**: from a bridge-net container the host
|
||||
is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on
|
||||
`dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`),
|
||||
`dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`)
|
||||
which belongs to the default `podman` net (`10.88.0.0/16`). Probe with
|
||||
bash `/dev/tcp` (no `python3`/`ip` in these images); escape loop vars so
|
||||
they expand **inside** `podman exec bash -c`.
|
||||
@@ -83,7 +83,7 @@ python3 -m http.server 8899 --bind 127.0.0.1
|
||||
```
|
||||
|
||||
Then in the browser: load `http://127.0.0.1:8899/index.html`, **wait ~1.5 s**
|
||||
(panels render after the async fetch; checking immediately returns 0 panels),
|
||||
(panels render after the async fetch; checking immediately returns 0 panels)
|
||||
extract via `tab.evaluate` — LLM cards render `.desc-role`/`.desc-spec`/
|
||||
`.desc-inputs`/`.desc-use` from the card's `details` object, other cards a
|
||||
plain `.desc`; `getComputedStyle(el).color` confirms tones. Screenshot for
|
||||
|
||||
@@ -22,7 +22,7 @@ The homepage is live on Dankclaw. `./scripts/deploy.sh dankclaw` copies the stat
|
||||
**Docs page shipped (2026-09-17):** `/docs` serves a static user documentation
|
||||
page (`shared/homepage/docs.html`) — access walkthrough, the running-model
|
||||
guide, media how-tos, and troubleshooting. Per-service `/docs/<svc>/` pages
|
||||
remain a separate thread. The three LLM cards (`frank-dragoon`, `frank-medium`,
|
||||
remain a separate thread. The three LLM cards (`frank-dragoon`, `frank-medium`
|
||||
`frank-zealot`) moved from admin-only to the **user** page with live model
|
||||
descriptions and a `docs` link to `/docs#models`.
|
||||
|
||||
@@ -59,7 +59,7 @@ without which the picker showed "No results found" for every family account
|
||||
"equivalent to Claude Sonnet 5 / OpenAI-Luna (Summer 2026 frontier
|
||||
intelligence)", frank-archon "equivalent to Claude Opus 4.8 / OpenAI-Terra
|
||||
(same era)", frank-zealot "equivalent to last year's frontier level
|
||||
(2025)". Rendered by `descHTML()` as `.desc-equiv` (head color, italic);
|
||||
(2025)". Rendered by `descHTML()` as `.desc-equiv` (head color, italic)
|
||||
pinned in `homepage_config.bats`.
|
||||
|
||||
**Protoss rename + Model Information component (2026-10-08):** the model
|
||||
@@ -91,7 +91,7 @@ same renames, api-id mapping, supervision and compaction notes.
|
||||
app card now carries hex-clipped art beside its name (`services.json` `art`
|
||||
field, files under `img/apps/`): 19 brand marks tinted amber from the
|
||||
Simple Icons CDN plus 19 hand-improvised monogram SVGs (amber hex + VT323
|
||||
initials) for apps with no mark — Copyparty, Marreta, Hermes (+dashboard),
|
||||
initials) for apps with no mark — Copyparty, Marreta, Hermes (+dashboard)
|
||||
node-exporter, Gluetun, OpenWebUI, Calibre-Web, RomM, Seerr, Loki, Alloy,
|
||||
Alertmanager, Valkey, Prowlarr, Readarr, Lidarr, Tautulli, ComfyUI. Model
|
||||
cards render in the order zealot → dragoon → archon (registry order is the
|
||||
@@ -143,8 +143,8 @@ per-app colors: Copyparty gold, Marreta brick, Grafana orange, Loki
|
||||
indigo, Neo4j network, Alloy violet, Promtail pink, Alertmanager red,
|
||||
Prometheus orange, Node Exporter teal, cAdvisor sky, Guake yellow,
|
||||
Authentik blue, Seer green radar, RomM a purple GameBoy. New per-service
|
||||
fields: `art_bg` (art backdrop — OpenWebUI's dark logo needs `#f5f5f5`),
|
||||
`warn` on OpenWebUI (searches ride the Seerr→SearXNG path),
|
||||
fields: `art_bg` (art backdrop — OpenWebUI's dark logo needs `#f5f5f5`)
|
||||
`warn` on OpenWebUI (searches ride the Seerr→SearXNG path)
|
||||
`model.compaction` (the amber "set this as compaction model" callout
|
||||
frank-zealot carries). frank-dragoon's card portrait is the SC2 unit
|
||||
portrait (fandom wiki `Dragoon_portrait.jpg`, webp→jpeg). The whole
|
||||
@@ -175,7 +175,7 @@ titles are chamfered plates with the diamond glyph inside the plate.
|
||||
Fonts: body is **Share Tech Mono** (legible retro terminal), headings
|
||||
are **Rajdhani 600** (the StarCraft UI face) — VT323 is retired; all
|
||||
three pages load the pair from Google Fonts. Four tiles that had
|
||||
reverted to initials got pictograms: Calibre-Web an open book (orange),
|
||||
reverted to initials got pictograms: Calibre-Web an open book (orange)
|
||||
RomM a purple GameBoy, Copyparty a blue folder with upload arrow,
|
||||
Marreta a crumbling brick wall with a red crack (`img/apps/app-*.svg`).
|
||||
frank-dragoon's card accent is phase-crystal teal (`#5fe0d0`), matching
|
||||
@@ -197,7 +197,7 @@ state — the only animation left is the speaking-voice pulse. The result
|
||||
is flat navy panels on crisp 1px borders: same Protoss direction, half
|
||||
the pixels.
|
||||
|
||||
**Model ids official (2026-10-08):** the card ids `frank-zealot`,
|
||||
**Model ids official (2026-10-08):** the card ids `frank-zealot`
|
||||
`frank-dragoon`, `frank-archon` are now the harness's official model ids
|
||||
in `models.yml`/`config.yml` (see [[OMP Harness]]); the
|
||||
systemd services keep the `frank-archon`/`frank-dragoon`/
|
||||
@@ -205,7 +205,7 @@ systemd services keep the `frank-archon`/`frank-dragoon`/
|
||||
one name per model.
|
||||
|
||||
**Api ids renamed on the wire (2026-10-09):** the promise in
|
||||
`docs.html` is kept — the served model ids are now `frank-zealot`,
|
||||
`docs.html` is kept — the served model ids are now `frank-zealot`
|
||||
`frank-dragoon`, `frank-archon`, so the docs page no longer carries
|
||||
`api id:` annotations, and the curl/opencode/Pi examples send
|
||||
`frank-zealot`. The systemd services keep the old names. `docs.html`
|
||||
@@ -216,7 +216,7 @@ ids, which did not change.
|
||||
clip-paths replaced with `border-radius` (the clipped corners read as
|
||||
gaps in the 1px border); section-title diamonds moved from absolute
|
||||
positioning to a fixed-width inline glyph (aligned, wrap-safe); closed
|
||||
WIP items grey out (dim type, desaturated art, muted border via
|
||||
WIP items grey out (dim type, desaturated art, muted border
|
||||
`:has(.badge.wip-closed)`); the Dragoon tagline is one spliced track
|
||||
(`sfx/dragoon-tagline.ogg` — loudnorm-matched, silence-trimmed, 350 ms
|
||||
beat; the two SC2 source oggs are trashed); privacy notes became
|
||||
|
||||
@@ -188,7 +188,7 @@ sudo systemctl restart systemd-resolved && resolvectl flush-caches
|
||||
```
|
||||
|
||||
The `~` (route-only) prefix is load-bearing: it routes only those domains to
|
||||
Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143`;
|
||||
Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143`
|
||||
`resolvectl status` shows `DNS Domain: ~franksdank.cloud ~franksdank.casa`.
|
||||
|
||||
**Layer 2 — browser DoH (the bypass: "curl works, browser times out").** If
|
||||
@@ -236,7 +236,7 @@ Endpoints (all POST, JSON body):
|
||||
`.content.records[]` as some docs claim — parse with
|
||||
`jq -r '.records[] | [.id,.name,.type,.content,(.ttl|tostring)] | @tsv'`.
|
||||
- Delete: `/dns/delete/{domain}/{id}` (body carries apikey+secretapikey).
|
||||
- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`,
|
||||
- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`
|
||||
`type`, `content`, `ttl`, `prio`).
|
||||
|
||||
Quirks that bite (verified live 2026-08-31):
|
||||
|
||||
@@ -55,7 +55,7 @@ The [[Alloy]] agent on Dankclaw is wired to ship container and journal logs to `
|
||||
The service answers and the API works; there is simply no data in it. Measured on 2026-10-18:
|
||||
|
||||
- `curl -s http://127.0.0.1:3100/loki/api/v1/labels` → `{"status":"success"}` with no `data` key (no label names exist), and a 7-day `query_range` on `{job="systemd-journal"}` returns `"result":[]`.
|
||||
- `du -sh /opt/loki/data` → a few hundred KB of directory scaffolding (`wal`,
|
||||
- `du -sh /opt/loki/data` → a few hundred KB of directory scaffolding (`wal`
|
||||
`chunks`, `compactor`, `tsdb-shipper-active`, `tsdb-shipper-cache`) —
|
||||
`find /opt/loki/data -type f` counted **8 files**, all WAL segments,
|
||||
cluster-seed and delete-request bookkeeping, **no chunk objects**
|
||||
|
||||
@@ -21,7 +21,7 @@ committed.
|
||||
| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
|
||||
| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
|
||||
|
||||
Targets and datasources reference **container names** (`grafana:3000`,
|
||||
Targets and datasources reference **container names** (`grafana:3000`
|
||||
`loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
|
||||
loopback-published ports; `alloy` is **not scraped at all** (no job in the
|
||||
config — its `127.0.0.1:12345` UI is only probed by the homepage timer,
|
||||
@@ -47,7 +47,7 @@ from `deploy_observability`:
|
||||
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` (`:286-300`)
|
||||
|
||||
`scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
|
||||
`alertmanager` when the changed file is `alertmanager.yml`, `:251-256`),
|
||||
`alertmanager` when the changed file is `alertmanager.yml`, `:251-256`)
|
||||
`shared/loki/**` to `loki` and `shared/alloy/**` to `alloy` (`:257-260`), and
|
||||
`shared/grafana/**` to `grafana` (`:261-264`); changes under
|
||||
`dankclaw/observability/` select the touched unit and its dependency consumer
|
||||
@@ -71,9 +71,9 @@ HTTP probe, not a metrics endpoint.
|
||||
|
||||
**Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`.
|
||||
Current delivered set: **4 alert rules + 2 recording rules** —
|
||||
`LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`,
|
||||
`LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`
|
||||
`LlmEndpointDown`, plus `job:llm_prefix_cache_reuse:rate5m` and
|
||||
`job:llm_spec_accept_length:rate5m` (`shared/prometheus/rules/llm-fleet.yml:32-110`),
|
||||
`job:llm_spec_accept_length:rate5m` (`shared/prometheus/rules/llm-fleet.yml:32-110`)
|
||||
confirmed live: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18)
|
||||
reports 4 alerting + 2 recording. Rules reload with the prometheus service.
|
||||
|
||||
@@ -101,7 +101,7 @@ curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.to
|
||||
|
||||
The two Prometheus queries were run on 2026-10-18 and are what every number on
|
||||
this page comes from: 4 alerting + 2 recording rules, and `up` for
|
||||
`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`,
|
||||
`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`
|
||||
`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
|
||||
listed in [[Prometheus]]. The Grafana query needs the local admin password and
|
||||
was not run here; the filesystem side (`ls /opt/grafana/provisioning/*`) is
|
||||
@@ -114,12 +114,12 @@ table (refused = outage, 404/HTML = config defect).
|
||||
|
||||
## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18)
|
||||
|
||||
Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`,
|
||||
Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`
|
||||
loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
|
||||
unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
|
||||
via `host.containers.internal`) → signal-cli, addressed by number
|
||||
`+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked
|
||||
2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`,
|
||||
2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`
|
||||
`curl -s http://192.168.0.143:18093/health` → `ok`, and
|
||||
`journalctl --user -u signal-relay` ends with successful `sent to +12062476336:`
|
||||
lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and
|
||||
@@ -148,7 +148,7 @@ state lives in Prometheus' process, not on disk — a restart re-stamps
|
||||
`activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not
|
||||
evidence of a fresh fault. (Alertmanager by contrast **does** persist:
|
||||
`--storage.path=/alertmanager` on a host volume keeps silences and the
|
||||
notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`,
|
||||
notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`
|
||||
`/opt/alertmanager/data`.)
|
||||
|
||||
Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
|
||||
|
||||
@@ -51,13 +51,13 @@ Dankserver runs one member of this family, node-exporter, as its own unit
|
||||
## Exposure
|
||||
|
||||
The five HTTP surfaces publish on `127.0.0.1` only — `ss -ltnp` (2026-10-18)
|
||||
shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100`,
|
||||
shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100`
|
||||
`127.0.0.1:12345` and no wildcard — because each of them is unauthenticated
|
||||
inside, and several carry lifecycle or query APIs. Browser access goes through
|
||||
Caddy: each vhost (`dankclaw/Caddyfile:128-145`) proxies to the guard
|
||||
`anubis:8923`, and the post-auth handlers route by container name on
|
||||
`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`),
|
||||
`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`),
|
||||
`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`)
|
||||
`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`)
|
||||
`prometheus.franksdank.cloud` → `prometheus:9090` (`:379-381`).
|
||||
Alertmanager has no vhost at all. node-exporter is the exception that must live
|
||||
in the host network namespace, and it binds exactly two addresses rather than
|
||||
@@ -70,13 +70,13 @@ and the podman bridge subnet `10.89.0.0/16` before the DROP
|
||||
|
||||
## Deployment
|
||||
|
||||
`./scripts/deploy.sh dankclaw --service <name>` runs `deploy_observability`,
|
||||
`./scripts/deploy.sh dankclaw --service <name>` runs `deploy_observability`
|
||||
which installs the unit, copies that service's config
|
||||
(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config`,
|
||||
(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config`
|
||||
`copy_alloy_config`, `copy_grafana_provisioning`, plus
|
||||
`install_signal_relay` with the alertmanager deploy), then waits on the loopback
|
||||
health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`,
|
||||
`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy`,
|
||||
health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
|
||||
`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy`
|
||||
`/node-exporter`: `/metrics`). Startup order follows the data dependencies —
|
||||
`grafana` after `prometheus`, `alloy` after `loki`
|
||||
(`grafana.container:32-33`, `alloy.container:38-39`). See
|
||||
|
||||
Reference in New Issue
Block a user