diff --git a/agent/managed-skills/anubis-guard-chain-flip/SKILL.md b/agent/managed-skills/anubis-guard-chain-flip/SKILL.md new file mode 100644 index 0000000..7d84f67 --- /dev/null +++ b/agent/managed-skills/anubis-guard-chain-flip/SKILL.md @@ -0,0 +1,25 @@ +--- +name: anubis-guard-chain-flip +description: "Runbook for moving a Caddy-fronted service behind the Anubis + Authentik embedded-outpost guard chain on Dankclaw (provider-first ordering, validate gate, verification probes)." +--- + +# Anubis + Authentik guard-chain flip (Dankclaw) + +Move a public vhost from `reverse_proxy :` onto: client → TLS edge → Anubis challenge (:8923) → Authentik embedded outpost decision → return to internal `http://:8017` listener → Host-matched handler → real upstream. + +## Order (never reorder) +1. **Provider first.** Create the outpost ProxyProvider before touching routes, or authed traffic hits the outpost error page instead of the app: + - Write a Python snippet to a local /tmp file that imports `authentik.providers.proxy.models.ProxyProvider`, constructs it with `name`, `external_host="https:///"` (exact, trailing slash), `internal_host="http://caddy:8017"`, sets `p.pk = None`, then `p.save()`. + - Pipe raw: `ssh -o BatchMode=yes dankclaw 'podman exec -i authentik-server ak shell' < /tmp/x.py`. Heredoc quoting through ssh SyntaxErrors; the raw-stdin pattern is the proven one (ak shell does `exec(sys.stdin.read())`). +2. **Route flip in ONE commit/deploy**: vhost becomes `reverse_proxy anubis:8923 { header_up X-Real-Ip {remote_host} }`; inside the `http://:8017` listener add `@ host ` + `handle @ { reverse_proxy }`. +3. Deploy via the repo deploy flow (commit first; deploy pushes main itself). + +## Verification (all three) +- Unauthenticated probe from client: `curl -s https:///` → HTTP 200 challenge page, grep body for challenge markers (`challenge|not a bot|anubis`). A 502 here instead of the challenge = chain mis-wired. +- Post-auth routing proof: grep the LIVE config inside the container — `podman exec caddy sh -c "grep -n -A3 /etc/caddy/Caddyfile"` — after the deploy prints its reload line; confirm the reload actually ran, don't trust exit 0. +- Outpost reachability: `podman exec authentik-server getent hosts caddy` resolves the compose DNS alias (10.89.1.x) that internal_host dials. + +## Pitfalls +- Probing `http://localhost:8017/` from the host returns HTTP 000 (connection refused): the post-auth listener lives in the caddy container's netns and is not bound on host loopback. Probe inside the netns or grep the config instead. +- Non-browser clients cannot solve the JS challenge. Before flipping, audit service-to-service traffic: ingestion/push paths (e.g. Alloy → `loki:3100`) must target internal addresses; any backend datasource URL pointing at a public hostname of a guarded service breaks the moment you flip. +- Exempt-by-construction categories that must stay unguarded: the IdP itself (`auth.` — its login flow can't sit behind a challenge), git smart-HTTP endpoints (Gitea git clients), services with their own complete auth. Document the exemption in the vhost comment. diff --git a/agent/managed-skills/arr-auth-posture-probe/SKILL.md b/agent/managed-skills/arr-auth-posture-probe/SKILL.md new file mode 100644 index 0000000..4896429 --- /dev/null +++ b/agent/managed-skills/arr-auth-posture-probe/SKILL.md @@ -0,0 +1,20 @@ +--- +name: arr-auth-posture-probe +description: "Verify a self-hosted *arr-style service's real auth posture by behavioral API probes instead of config-flag inference; includes the sed-masks-a-miss pitfall and wrong-host probe pitfall" +--- + +# Arr-auth posture probe + +Ground truth for "is this arr/gate open?" comes from behavior, not config flags. `AuthenticationMethod=None` + `AuthenticationRequired=Enabled` in `config.xml` is ambiguous on paper; on the wire it means a login screen with **zero users** — first-account-wins. + +## Recipe +1. Confirm which host runs the service before probing (`podman ps` on the target, not the box you ssh'd into first — arrs live on Dankserver, chat/searxng on Dankclaw). +2. Probe an API endpoint that bypasses the SPA shell, locally and through the vhost: + `curl -s -o /dev/null -w "%{http_code}" http://:/api/v3/system/status` + - SPA roots (`/`) always serve a 200 shell — never conclude "open" or "gated" from `/`. + - 401 = gated; 200 + JSON = API fully open (anyone on the net can drive it with curl). +3. For config truth, dump the whole file (`podman exec cat /config/config.xml`), not a piped `grep | sed -n 2p`: **`sed -n Np` exits 0 on zero matches**, so the pipeline hides a miss and you conclude "empty field" when the grep itself failed. +4. API keys from config.xml are local-only secrets; they stop being secret the moment a vhost serves the API unauthenticated. + +## Decision framing for the user +When posture = gated-with-zero-users, present it as a decision menu (None behind the front-door wall vs External/authentikarr), not a silent change — first-account-wins means whoever logs in first owns the instance. diff --git a/agent/managed-skills/arr-stack-api-config/SKILL.md b/agent/managed-skills/arr-stack-api-config/SKILL.md new file mode 100644 index 0000000..a7475e6 --- /dev/null +++ b/agent/managed-skills/arr-stack-api-config/SKILL.md @@ -0,0 +1,30 @@ +--- +name: arr-stack-api-config +description: "Configure/download-client and indexer wiring for the *arr stack on Dankserver via live API calls — schema-first, test-before-enable; covers Sonarr/Radarr v3 vs Lidarr/Readarr v1 API split and Prowlarr v2 Applications/sync model." +--- + +# Arr-stack API config (Dankserver) + +## API version split +- Sonarr/Radarr: `/api/v3/...`; Lidarr/Readarr/Prowlarr: `/api/v1/...`. +- `PUT /downloadclient` and `PUT /indexer/{id}` take a SINGLE resource object (GET it, patch `settings`, PUT whole object back). An array body fails with `Path: $ | BytePositionInLine: 1`. +- API keys live in each container's config: `podman exec grep -o "[^<]*" /config/config.xml | cut -d">" -f2`. + +## Schema-first probing (validators are free documentation) +Send the FULL schema template object with fields filled; v1 validators reject partials: +- `'Fields' must not be empty` -> send full schema copy, not `fields:[]`. +- `'Base Url' must not be empty` when keys were camelCase -> PascalCase required (`BaseUrl`, `ApiKey`). +- `targetType null` -> configContract not registered in that build (this Prowlarr build has no Prowlarr-plugin indexer; integration is Torznab-family only). +- `testingPackageId` in a PUT body downgrades validation failures to warnings. + +## Prowlarr v2 model (inverted from v1) +Indexers live ON THE ARR SIDE; Prowlarr syncs indexer configs via `ApplicationSync` commands (`POST /api/v1/command {"name":"ApplicationSync","body":{"applicationId":N}}`). Applications settings keys must be PascalCase or sync throws NullConfig. Torznab feed paths can answer with the SPA HTML shell when absent — verify by byte count (HTML shell ~1946 bytes, 1 ms) vs XML, not just status code. + +## DB as arbiter +When endpoint shapes are opaque, stop the container and edit `/opt/prowlarr/config/prowlarr.db` with `sudo python3` sqlite directly, then start. Indexers table columns are Id/Name/Implementation/Settings/ConfigContract/Enable/Priority/Added — the enable flag column is `Enable` (not EnableRss). + +## Verification rules +- `POST /downloadclient` returns the COLLECTION, not the created object — verify via GET. +- Test endpoints return verdicts in the BODY at HTTP 200 (`{}` = pass, list = fail) — judge from bodies, not status codes. +- `syncLevel` is enum ApplicationSyncLevel — omit or use int 2; string rejected. +- Command POSTs (MoviesSearch etc.) run server-side and respond only when the work finishes: curl `-m` timeouts (HTTP 000) during search/grab are NOT failures; poll queue/history afterward instead of re-firing. Warm small commands (RefreshMonitoredDownloads) answer in ~20 ms — a hang means live work in progress, not a dead API. diff --git a/agent/managed-skills/arr-v1-api-validator-peeling/SKILL.md b/agent/managed-skills/arr-v1-api-validator-peeling/SKILL.md new file mode 100644 index 0000000..59bc197 --- /dev/null +++ b/agent/managed-skills/arr-v1-api-validator-peeling/SKILL.md @@ -0,0 +1,29 @@ +--- +name: arr-v1-api-validator-peeling +description: "When driving Sonarr/Radarr (v3) vs Lidarr/Readarr/Prowlarr (v1 .NET) APIs over HTTP: probe endpoints by status code, peel validators one live call at a time, and treat config.xml on disk as truth — v1 rejects literal enum strings that break AuthorizationMiddleware." +--- + +# *arr v1 API validator peeling + +Use when creating root folders, profiles, or settings via the *arr HTTP APIs, or when a v1 (.NET) arr 500-sprawls after a config edit/restart. + +## API version split +- Sonarr/Radarr: `/api/v3/...`. Lidarr/Readarr/Prowlarr: `/api/v1/...` (Prowlarr port 9696, NOT 8090 — that's Jackett). +- Probe endpoints by status code first: **empty body = endpoint absent**, not a failure. v1 has no `/api/v1/profile` endpoint at all. + +## Peel validators one live call at a time +POST a minimal body and let the validator error name the next required field, instead of guessing payload shapes: +- `DefaultMetadataProfileId must be >0` → that field is required; fetch real ids from `GET /api/v1/metadataprofile` (returns rows like `(1,'Standard')`). +- `'Name' must not be empty` → include `name`. +- `Quality Profile does not exist` → the id you sent doesn't exist; list real ones first (`GET /api/v1/qualitydefinition` returns the quality catalog, ids ≠ profiles). +Then POST the full body with **existing** ids: `{"name":"music","path":"/music","defaultQualityProfileId":1,"defaultMetadataProfileId":1}`. + +## Auth-config pitfall (v1 .NET apps) +- `None` + `Enabled` = self-contradictory: UI Save permanently disabled, and any save throws `DryIoc.ScopedItemException` in `AuthorizationMiddleware` → 500 on every page after restart. +- The valid no-auth representation is **absence of the `` key**, not the literal string `Disabled`. Fix = delete the key from `/config/config.xml` (`sed -i "//d"`), restart container. +- Tell-tale: one v1 app boots clean while siblings 500-sprawl → diff their config.xml auth blocks; the healthy one shows the correct shape. + +## General rules +- Container `State=running` ≠ app serving; read `podman logs --tail 12 ` before theorizing. +- When a UI gate blocks saving, go under it: these apps keep plaintext XML config on disk (`/config/config.xml`) — read/edit it directly via `podman exec`. +- API keys live in each container's `/config/config.xml` ``; grep them live rather than trusting docs. diff --git a/agent/managed-skills/authentik-claim-cache-purge/SKILL.md b/agent/managed-skills/authentik-claim-cache-purge/SKILL.md new file mode 100644 index 0000000..3a6b254 --- /dev/null +++ b/agent/managed-skills/authentik-claim-cache-purge/SKILL.md @@ -0,0 +1,38 @@ +--- +name: authentik-claim-cache-purge +description: "Fix persistent 403s at Caddy RBAC header gates after authentik group changes: purge the embedded outpost's proxysession claim cache and verify silent re-auth" +--- + +# Authentik outpost claim-cache purge + +Use when a user gets persistent 403s at a Caddy `header_regexp` RBAC gate (e.g. `/admin`) even though their authentik group membership is correct, and re-login did not help. + +## Mechanism (authentik 2026.8, this fleet) +- The embedded outpost does NOT re-evaluate groups per request. Each completed authorize writes one row to `authentik_providers_proxy_proxysession`; `session_data` JSON holds the claims (incl. `groups`) served for that browser session until the row expires (observed TTLs 1–24 h per row). +- A session minted before the group add keeps serving the old group list → gate 403s forever until expiry. +- The 403 happens at Caddy and generates ZERO authentik events — silence in `authentik_events_event` is expected. +- Re-login does not fix it: the browser's outpost cookie re-maps to the same stale row. +- `proxysession.user_id` is null; map rows to users via `session_data->'claims'->>'preferred_username'`. + +## Diagnosis (run on dankclaw) +```bash +podman exec authentik-postgres psql -U authentik -d authentik -At -c "SELECT session_data->'claims'->>'preferred_username' AS user, session_data->'claims'->>'groups' AS cached_groups, expires FROM authentik_providers_proxy_proxysession ORDER BY expires;" +``` +Stale row = `cached_groups` missing the new group while repo/Caddy gate expects it. + +## Fix +```bash +podman exec authentik-postgres psql -U authentik -d authentik -At -c "WITH d AS (DELETE FROM authentik_providers_proxy_proxysession WHERE session_data->'claims'->>'preferred_username' IN ('') RETURNING 1) SELECT 'deleted='||count(*) FROM d;" +``` +- The user's authentik django login session is untouched → next visit re-authorizes SILENTLY (no password prompt) with fresh claims. +- NEVER rotate `AUTHENTIK_SECRET_KEY` for this — it invalidates every outpost session fleet-wide. + +## Verification without the user's password +Simulate their browser: delete a TEST user's proxysession rows (e.g. `qa-e2e-admin`), keep its e2e storage state on disk, then run the admin spec bypassing setup (setup ALWAYS fresh-logins, which would mask the path): +```bash +cd tests/e2e && npx playwright test --project=admin --no-deps +``` +Passing with no login prompt = silent re-auth proven. `--no-deps` is required because project dependencies would rerun setup. + +## Doc +Mechanism + SQL live in `serverAppConfigs-docs/Infrastructure/Authentik.md`, section "Outpost claim cache (proxysession)" (commit ea0bba3). diff --git a/agent/managed-skills/authentik-flow-ui-automation/SKILL.md b/agent/managed-skills/authentik-flow-ui-automation/SKILL.md new file mode 100644 index 0000000..704ba9d --- /dev/null +++ b/agent/managed-skills/authentik-flow-ui-automation/SKILL.md @@ -0,0 +1,46 @@ +--- +name: authentik-flow-ui-automation +description: Drive the authentik two-stage login through real browser UI for E2E tests on Anubis+outpost-guarded routes; hardened against the Lit stage re-render race and Anubis PoW budgets +--- + +Drive authentik's two-stage login (identification → password) through a real browser UI for E2E tests on routes guarded by the Anubis + Authentik chain (Caddy → Anubis :8923 → outpost :9000). Proven in `tests/e2e/lib/login.ts` (`akLogin`) against authentik 2026.x. + +## Hard rules + +1. **Fresh context per login.** Never reuse a context or page across logins, and never retry a failed login inside the same context — a half-finished flow poisons authentik's `pending_user` stage and the next login hangs on a "Continue as " interstitial that never completes. Each login = new `browser.newContext()`. +2. **Real Chrome UA.** Anubis denies obvious headless UAs; set `userAgent` to a real Chrome UA in `use:` (config-level) or `launchOptions`. Headless chromium + REAL_UA solves the PoW fine. +3. **Shadow-DOM locators.** authentik stages are Lit web components; locate inputs with `page.locator('input[name=username]')` etc. (Playwright pierces shadow roots) — never `page.$`. + +## The Lit re-render race (the big one) + +The flow executor re-renders a stage right after mount. A programmatic `fill()` can succeed and then be WIPED by the re-render; the submit then POSTs an empty form ("This field is required") and `waitForURL` times out while the page still shows the password stage with an empty field. + +Hardened sequence per stage: + +```ts +for (let attempt = 1; attempt <= 3; attempt++) { + await page.fill('input[name=password]', pass); + // VERIFY the value survived the re-render before submitting + if ((await page.inputValue('input[name=password]')) !== pass) continue; + await page.keyboard.press('Enter'); // native form path; avoids button re-render races + try { + await page.waitForURL(u => !u.pathname.includes('/if/flow/'), { timeout: 30_000 }); + return; // past this stage + } catch { /* maybe wiped; retry */ } +} +throw new Error(`akLogin: stage never advanced after 3 attempts`); +``` + +- Submit with **Enter**, not a button click — the button can be re-created between snapshot and click. +- Verify-after-fill is the load-bearing step; without it you silently submit empty forms. +- The identification stage has the same race; apply the same fill-verify-Enter there. + +## Anubis PoW budgets + +The first guarded-vhost hit in a fresh context runs an Anubis proof-of-work; on a cold/loaded box it can outlive default assertion budgets. Give title/URL asserts after `goto` 60 s budgets. With `COOKIE_DOMAIN` set fleet-wide (Anubis ≥ v1.27), every later guarded vhost in the same context skips the challenge (~0.5 s per vhost in the crawl specs). + +## See also + +- `serverAppConfigs-docs/Architecture/E2E Suite.md` — project layout, credentials model (`tests/e2e/.env.local`, git-ignored), run commands, gotcha list. +- `serverAppConfigs-docs/Infrastructure/Anubis.md` — chain behavior, v1.27 gotchas (X-Real-Ip mandatory, DENY = HTTP 200 body). +- `arr-auth-posture-probe` for non-browser API-posture checks of guarded routes. diff --git a/agent/managed-skills/authentik-outpost-verification/SKILL.md b/agent/managed-skills/authentik-outpost-verification/SKILL.md new file mode 100644 index 0000000..7f57756 --- /dev/null +++ b/agent/managed-skills/authentik-outpost-verification/SKILL.md @@ -0,0 +1,34 @@ +--- +name: authentik-outpost-verification +description: "Verify authentik embedded-outpost and proxy-provider state on a live host by probing from inside the container network namespace; covers meta-header routing, rootless podman observation artifacts, and 2026.x schema quirks (renamed columns, ak shell swallowing piped stdin)." +--- + +# Authentik outpost/provider verification (live host) + +## When to use +Guarded apps behind Anubis+authentik return an error page after authentication, or you need ground truth on outpost/provider state on a host running authentik 2026.x with podman quadlet units. + +## Core insight: probe from inside the network namespace +- Rootless podman gives each container its own netns; **host `ss -ltn` cannot see listeners inside other containers' namespaces**. A missing listener on the host is an observation artifact, not a fault — verify by dialing *from inside* the namespace. +- Working probe (distroless images have no shell, so `podman exec sh` fails): + ``` + podman run --rm --network container: --entrypoint /bin/busybox docker.io/library/busybox:latest \ + wget -q -T 4 -t 1 -O /dev/null -S "" 2>&1 | head -8 + ``` +- Quote URLs over ssh: the remote shell is zsh and bare `?` in query strings triggers globbing (`no matches found`). + +## Meta-header routing truth +- The outpost routes by **meta-header middleware**, not URL-path lookup. A request to `/outpost.goauthentik.io/login` **without** `X-authentik-meta-outpost/-provider/-app` headers is a path that genuinely does not exist → a 404 there is *correct behavior*, not a fault. +- Probe the auth hop exactly as Caddy sends it: full meta-header set (`X-authentik-meta-outpost: `, `X-authentik-meta-provider: `, `X-authentik-meta-app: name=`). If a full-meta-header request still 404s, the outpost's URL space is not mounted in the answering process — that is the real failure point, and it means the transport is aimed at a URL space that doesn't exist there (embedded outposts mount their URL space at `authentik_host`; check the outpost row's `_config.authentik_host` before theorizing). +- Boot transience: immediately after a restart the same probe can return 503 (configure task pending); poll until the answer changes before concluding. + +## Schema quirks in 2026.x +- `ak shell` is interactive-only; piped stdin is swallowed (prints import banner, discards output). For live DB truth use: + ``` + podman exec authentik-postgres psql -U authentik -d authentik -x -c "select * from authentik_outposts_outpost" + ``` +- Columns are renamed vs older versions: `Provider.objects.filter(type=...)` raises FieldError (type moved to subclass tables); the outpost `enabled` column may not exist under that name; provider pk is `oauth2provider_ptr_id`. **Query with `select *` and stop guessing column names.** +- `podman inspect --format "{{.Config.NetMode}}"` errors on this podman version — the field is `.HostConfig.NetworkMode`. + +## Decision fork when the auth hop 404s +A) point the guarded apps' auth hop at where auth actually lives (the vhost matching `authentik_host`), or B) flip the outpost to standalone (own container, own URL space, new quadlet unit + `AUTHENTIK_TOKEN`). This is an architecture decision for the user — present both with blast radius, don't pick silently. diff --git a/agent/managed-skills/compose-stack-bringup-groundtruth/SKILL.md b/agent/managed-skills/compose-stack-bringup-groundtruth/SKILL.md new file mode 100644 index 0000000..bb96372 --- /dev/null +++ b/agent/managed-skills/compose-stack-bringup-groundtruth/SKILL.md @@ -0,0 +1,64 @@ +--- +name: compose-stack-bringup-groundtruth +description: "Bring up a compose-defined container stack on a rootless podman host by verifying reality first (mounts, images, in-image interfaces) instead of trusting compose env/flag guesses; includes the copyparty ac-edition argv-interface recipe and boot-persistence audit." +--- + +# Compose-stack bring-up with ground truth (rootless podman hosts) + +Use when standing up (or debugging) a compose-defined service stack on a rootless +podman host where the stack has never run, or answers with surprise statuses +(SIGSEGV at boot, HTTP 500 on `/`, "manifest unknown"). + +## Procedure + +1. **Audit reality before pressing buttons.** `podman ps -a` runs per-user — + check every namespace (`sudo podman ps -a`) and native units + (`systemctl list-units`) before declaring a service absent; a service may run + as a native systemd unit or snap, not a container. Compare compose mount source + paths against the real filesystem (`ls /mnt/...`): compose files often name + phantom paths that never existed (e.g. `/mnt/media/*` vs the real LUKS array at + `/mnt/entropy`). Fix compose through git first, deploy second. + +2. **Pre-create bind-mount sources.** Rootless podman fails container creation + with `statfs /opt//...: no such file or directory` when a bind source is + missing. `sudo mkdir -p` + `chown 1000:1000` every compose volume source + before the first `podman-compose up -d`. + +3. **Staged bring-up.** Infra first (`up -d mariadb redis`), poll healthchecks via + `podman inspect --format "{{.State.Health.Status}}"`, then `up -d` the rest. + +4. **Never guess image tags — query ground truth.** ghcr.io answers **403 on the + bearer-token request for nonexistent/private repos** ("manifest unknown" / token + len 0 = repo or tag does not exist, not an auth problem). Query the registry + API (`ghcr.io/token?scope=repository:/:pull` then + `/v2///tags/list`) before pinning a tag. + +5. **Read the interface from inside the image** before guessing flags/envs: + `podman exec cat /z/cpp.sh`, `cat /z/initcfg`, + `podman exec python3 -m copyparty --help-accounts`. A service answering + HTTP 500 with its own styled error page is *running* — it is misconfigured, + not down. Copyparty ac edition ground truth: entrypoint + `/bin/ash /z/cpp.sh -c /z/initcfg` then `exec python3 -m copyparty "$@"`; + accounts/volumes are **argv flags** (`-a USER:PASS`, `-v SRC:DST:PERMS`), not + `COPAY_*` envs (zero `COPAY_` references in the package). With no accounts and + no volumes configured it serves cwd read/write for everyone — an + unauthenticated world-writable share of the whole mount. + +6. **Beware remote-shell quoting traps.** Remote zsh eats bare `===`/`?` tokens + (glob qualifiers/globs); grep patterns with nested quotes over ssh can be + quote-eaten into silent zero-match. A zero-match from a mangled pattern is not + absence — re-verify the target exists before concluding it. + +7. **Boot-persistence audit last.** On rootless podman hosts with + `podman-user-generator`, containers started via podman-compose get generated + `container-.service` units at first boot (evidence: an existing unit for + one container proves the generator works). Verify after first reboot: + `systemctl --user is-enabled .service`. + +## Gotchas +- jellyfinetype SIGSEGV at boot with `user: "1000:1000"` = unmapped uid inside the + rootless userns (host uid 1000 == container uid 0); drop the user line and let it + run as container-root (= host deploy user). +- `up -d --force-recreate` applies compose changes that change command/env; probe with + a poll loop (`for i in $(seq...)`) because fresh containers answer `000`/refused for + the first seconds while indexing. diff --git a/agent/managed-skills/flm-npu-pack-registration/SKILL.md b/agent/managed-skills/flm-npu-pack-registration/SKILL.md new file mode 100644 index 0000000..d9bc4cf --- /dev/null +++ b/agent/managed-skills/flm-npu-pack-registration/SKILL.md @@ -0,0 +1,27 @@ +--- +name: flm-npu-pack-registration +description: "Runbook for registering a custom FastFlowLM NPU2 model pack on a tag that isn't in FLM's built-in catalog (Dankclaw :7771), including the LFS-pointer and vision-weight traps and the NPU hwctx contention rule." +--- + +# FLM custom-tag pack registration (Dankclaw NPU endpoint) + +Use when adding a non-catalog NPU2 pack (e.g. an extraction or fine-tuned model) to the FLM registry served by `frank-npu` on port 7771. Companion diagnosis skill: `flm-tag-load-failure-diagnosis`. + +## Registry facts +- User registry: `/home/dank/.config/flm/model_list.json`, shape `{"model_path": "models", "models": {"": {"": {name, url, file_url, size, flm_min_version, files[], vlm, default_context_length, max_prefill_len, details{format:"NPU2", family, ...}, label[]}}}}}`. +- Serve resolves tags through `FLM_CONFIG_PATH` at startup; `flm list` with the env var set lists custom tags (`⏬` = not downloaded). + +## Procedure +1. Back up the registry JSON first (`cp -a`). +2. Insert the entry via a python json edit + atomic `os.replace`. Set `files` to the FULL HF file list — **including `vision_weight.q4nx`** even for text-only packs: the qwen3.5-family loader opens it unconditionally and a missing file makes the first request HANG silently (no error log). +3. **Do not use `flm pull ` for custom entries** — it throws `json.exception out_of_range key ':' not found` while building the download list. Place files manually into `/home/dank/.config/flm/models//` with: + `curl -sfL "https://huggingface.co//resolve/main/?download=true" -o ` + Plain `raw/main/` returns 135-byte LFS pointer stubs, not weights — verify byte counts against the HF tree API before trusting a download. +4. Kernel packs: copy from the same-shape family pack (shape-match `config.json` on hidden_size/num_hidden_layers/head_dim first) into `/opt/fastflowlm/share/flm/xclbins//` via `sudo cp -a` (root-owned dir). +5. Deploy/restart the serve unit (deploy path, not ad-hoc restarts) so the new process loads with all files present. +6. Verify with a REAL completion request on 7771 (`model: ":"`) — `/v1/models` alone proves registration, not loadability. + +## Hard constraints +- Only ONE `flm serve` process can hold the NPU hardware context. A second concurrent serve process fails every load with `DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22` (aie2 mgmt msg opcode 0x2 status 0x2000003). Never run scratch/verify instances while production holds the NPU — verify via the deployed service instead. +- Tag switching happens INSIDE one serve process; first request after a model switch pays ~6–11 s cold-load cost (visible in `usage.load_duration`). +- Over ssh, never `pkill -f ""` with a pattern present in your own command line — it kills your own session; use bracket-guarded patterns (`[q]wen3.5:4b`) or exact PIDs. diff --git a/agent/managed-skills/flm-tag-load-failure-diagnosis/SKILL.md b/agent/managed-skills/flm-tag-load-failure-diagnosis/SKILL.md new file mode 100644 index 0000000..cf66834 --- /dev/null +++ b/agent/managed-skills/flm-tag-load-failure-diagnosis/SKILL.md @@ -0,0 +1,34 @@ +--- +name: flm-tag-load-failure-diagnosis +description: "Diagnose FLM (FastFlowLM) NPU serve failures where a tag listed in /v1/models answers \"Failed to load model: No such file ...xclbin\" — missing kernel-pack symlink under FLM_XCLBIN_PATH is the root cause; includes per-request switch-cost behavior." +--- + +# FLM tag load-failure diagnosis (Dankclaw sight+sound endpoint) + +## Symptom +`POST /v1/chat/completions` on the FLM serve endpoint answers **HTTP 200** with a flat error body `{"error":"Failed to load model!"}` and no `choices`. `/v1/completions` instead returns a nested OpenAI-shaped `server_error` (code 500) also under HTTP 200. A status-code-only client reads this as a successful empty completion — always check for the `error` key. + +Two distinct causes, distinguish via `journalctl -u frank-default-sightsound.service`: + +1. **Unknown tag** (not in the registry): immediate error, nothing loaded. Correct behavior; not a bug. +2. **Registered tag whose kernel pack is missing**: FLM resolves the tag to its pack dir, downloads missing weight files on the fly (first request hangs ~90–110 s downloading), then fails with `Failed to load model: No such file '/home/dank/.config/flm/xclbins//.xclbin'`. Subsequent attempts fail fast (~2 s). + +## Root cause of case 2 +The serve script exports `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins` (user dir), so **every** kernel lookup resolves under the user dir — but that dir only contains symlinks for the packs someone linked. A tag listed in `/v1/models` is NOT proof it is loadable on this host. + +## Fix +One symlink per missing pack, run on Dankclaw: +``` +ln -s /opt/fastflowlm/share/flm/xclbins/ /home/dank/.config/flm/xclbins/ +``` +Verified 2026-08-30: `Qwen3.5-4B-NPU2` — after the symlink, first request loaded and answered in 7.4 s wall (cold load + generate). + +## Switch-cost behavior (measured same day) +- The `model` field routes per-request; no restart needed to switch between registry tags. +- A request whose model differs from the resident one triggers a full reload ("Loading model:" in journal before prefill): measured 7–11 s wall for an 8-token completion on ~4B-class packs. Same-tag requests are fast; alternating tags pays the swap each time. +- Single-lane NPU: concurrent requests queue FIFO behind the current lock holder (`NPU Locked!` → `NPU Lock Released!`). + +## Probe recipe (repeatable) +1. `curl -s http://192.168.0.143:52626/v1/models` — registry listing is authoritative for *registered* tags, not loadability. +2. On a failed load, grep the journal for `Failed to load model` (use `--full`) — the quoted xclbin path names the missing pack dir exactly. +3. Compare `ls /home/dank/.config/flm/xclbins/` (symlinks = installed) against `/opt/fastflowlm/share/flm/xclbins/` (all available packs); diff = fixable with symlinks. diff --git a/agent/managed-skills/frankbook-qbittorrent-e2e-verify/SKILL.md b/agent/managed-skills/frankbook-qbittorrent-e2e-verify/SKILL.md new file mode 100644 index 0000000..7af8e3e --- /dev/null +++ b/agent/managed-skills/frankbook-qbittorrent-e2e-verify/SKILL.md @@ -0,0 +1,57 @@ +--- +name: frankbook-qbittorrent-e2e-verify +description: "Verify the FrankBook-Pro gluetun→qBittorrent grab pipeline end-to-end (egress gate inside gluetun, generic SID login, legal test torrent, category path check, data-safe cleanup) after any tunnel/qB/category change" +--- + +# FrankBook-Pro qBittorrent E2E verification + +Use after any change to the FrankBook-Pro torrent pipeline — WireGuard credential rotation, qB version change, gluetun reconfig, category re-seeding. Proves the full grab path (tunnel egress → qB download → category save path → clean removal). All commands run from Dankware via `ssh frankbook-pro`; inside the Mac shell export `PATH="/usr/local/bin:/opt/homebrew/bin:$PATH"` for `podman machine ssh`. + +## 1. Egress gate — MUST run inside gluetun + +```bash +podman machine ssh 'podman exec gluetun wget -qO- https://ipinfo.io/json' +``` + +- Require `org` = AS9009 (M247 — ProtonVPN Panama PA-12 config). Judge by org, never country alone. +- NEVER run this one ssh-context out (e.g. bare `wget` on the Mac host): it silently returns the residential Comcast IP and reads as a hard-gate leak that does not exist. If a check shows a residential IP, first re-run inside the container before diagnosing. + +## 2. Login — generic cookie parse + +The QPW fetch is flaky (~1 in N returns empty); assert length before use. An empty-password run once produced a Forbidden loop that cost hours. + +```bash +QPW=$(podman machine ssh 'cat /home/core/Containers/qbittorrent/env' | grep QBIT_WEBUI_PASSWORD | cut -d= -f2) +[ ${#QPW} -eq 20 ] || exit 1 +SID=$(curl -si --data-urlencode "username=admin" --data-urlencode "password=$QPW" \ + http://127.0.0.1:8080/api/v2/auth/login \ + | grep -oE '[A-Za-z_]*SID[A-Za-z0-9_]*=[^;[:space:]]*' | head -1) +curl -b "$SID" ... # pass the WHOLE match +``` + +- qB 4.x cookie name is plain `SID`; qB 5.x uses `QBT_SID_`. Hard-coding either name breaks the other version's parse (sidlen=0 → 403 cascades). +- Parse-bug 403s never reach qB as failed logins — no ban accrues. Check server-side truth in the log: `/Volumes/COMPLETE/qbittorrent-data/qBittorrent/config/qBittorrent/logs/qbittorrent.log` (`WebAPI login success/failure` + source IP). Repeated REAL failures accrue toward qB's 5-failures/3600s IP ban. + +## 3. Add a legal test torrent + +Blender open-movie torrents (~300 MB, e.g. Big Buck Bunny) via `POST /torrents/add` with the raw `.torrent` file (`-F "file=@..."`) and `category=Finished Torrents`. Never leave a real-looking grab behind. + +## 4. Poll and verify + +Poll `GET /torrents/info` (~20 s): state `downloading` → `moving` (INCOMPLETE→COMPLETE move) → `stalledUP`/`uploading` at progress=1. `save_path` must be `/Volumes/COMPLETE/Finished Torrents` (qB 4.x auto-creates unknown categories at DefaultSavePath; 5.x did NOT inherit category paths — one more reason for the 4.6.7 pin). Confirm on disk: `ls "/Volumes/COMPLETE/Finished Torrents/"`. Throughput ~17 MB/s is normal for the tunnel. + +## 5. Cleanup — delete only your artifact + +```bash +curl -s -o /dev/null -w "%{http_code}" -b "$SID" \ + --data-urlencode "hashes=" \ + --data-urlencode "deleteFiles=true" \ + http://127.0.0.1:8080/api/v2/torrents/delete # expect 200 +``` + +- Target the known hash literally — the user's real grabs share the directory and MUST be untouched. Verify: hash absent from `/torrents/info`, your dir gone, all other entries intact. + +## Version-coupling constraints + +- qB is pinned to 4.6.7 (readarr's EOL client rejects qB ≥ 5's `204` login at runtime). Do not unpin while readarr exists. +- After ANY qB version flip, regenerate the `WebUI\Password_PBKDF2` line fresh (same password, 100k iterations) — a 5.x-written hash line makes 4.6.7 reject every login. Rewrite while qB is stopped. diff --git a/agent/managed-skills/freetoken-build-dankclaw/SKILL.md b/agent/managed-skills/freetoken-build-dankclaw/SKILL.md new file mode 100644 index 0000000..7d1852c --- /dev/null +++ b/agent/managed-skills/freetoken-build-dankclaw/SKILL.md @@ -0,0 +1,55 @@ +--- +name: freetoken-build-dankclaw +description: "Build FreeToken on Dankclaw (Ubuntu 24.04) with CUDA — installs nvidia-cuda-toolkit, sets CUDA_HOME=/usr, overrides nvcc/CUDA version mismatch" +--- + +# FreeToken Build on Dankclaw + +Dankclaw is Ubuntu 24.04 with ROCm already installed. FreeToken builds for **CUDA** (not ROCm) because the kernel package (`_pinned_tensor`) links against `cudart`. + +## Prerequisites (install if missing) + +```bash +export PATH="/home/dank/.local/bin:$PATH" +export CUDA_HOME=/usr +export FREETOKEN_ALLOW_CUDA_MISMATCH=1 + +# CUDA toolkit (if not present): +sudo apt-get install -y nvidia-cuda-toolkit +# UV (if not present): +curl -LsSf https://astral.sh/uv/install.sh | sh +``` + +## Key environment variables + +|Variable|Value|Why| +|---|---|---| +|`CUDA_HOME`|`/usr`|Ubuntu CUDA toolkit package places nvcc at `/usr/bin/nvcc` and libcudart at `/usr/lib/x86_64-linux-gnu/`| +|`FREETOKEN_ALLOW_CUDA_MISMATCH=1`|Set|nvcc is 12.4 but torch 2.11.0+cu130 ships CUDA 13.0 — safe to override| +|`PATH`|`/home/dank/.local/bin:$PATH`|uv installs here| + +## Build procedure + +```bash +cd /home/dank +git clone https://github.com/FlashML-org/FreeToken.git +cd FreeToken +uv venv && source .venv/bin/activate +uv pip install -e ".[accel]" +``` + +## Verification + +```bash +ft --version # → freetoken version 0.1.2 +python3 -c "import torch; print(torch.__version__)" # → 2.11.0+cu130 +python3 -c "from freetoken.kernel import _pinned_tensor; print(_pinned_tensor.__file__)" +python3 -c "import flashinfer; print(flashinfer.__version__)" # → 0.6.18 +python3 -c "import sgl_kernel; print(sgl_kernel.__version__)" # → 0.4.5 +``` + +## Notes + +- `torch.cuda.is_available()` may return `False` with "Can't initialize NVML" warning — this is the management library, not CUDA runtime. The kernels are built and linked correctly. +- The NVIDIA driver modules may or may not be loaded; they don't affect the build. +- ROCm is present on Dankclaw but FreeToken's kernel build specifically requires CUDA (links `cudart`). diff --git a/agent/managed-skills/freetoken-dankclaw-build/SKILL.md b/agent/managed-skills/freetoken-dankclaw-build/SKILL.md new file mode 100644 index 0000000..3445a9f --- /dev/null +++ b/agent/managed-skills/freetoken-dankclaw-build/SKILL.md @@ -0,0 +1,39 @@ +--- +name: freetoken-dankclaw-build +description: "Build FreeToken on Dankclaw (Dankware) — installs CUDA toolkit, sets CUDA_HOME, handles nvcc/CUDA version mismatch" +--- + +## Build FreeToken on Dankclaw + +### Prerequisites +- CUDA toolkit: `sudo apt install nvidia-cuda-toolkit` (gives nvcc at /usr/bin, libcudart at /usr/lib/x86_64-linux-gnu/) +- uv: installed at /home/dank/.local/bin/uv (or install via `curl -LsSf https://astral.sh/uv/install.sh | sh`) +- CMake, git, Python >= 3.10 — all present + +### Build steps +```bash +cd /home/dank +git clone https://github.com/FlashML-org/FreeToken.git # or cd into existing +cd FreeToken +uv venv && source .venv/bin/activate +export CUDA_HOME=/usr +export FREETOKEN_ALLOW_CUDA_MISMATCH=1 # nvcc 12.4 + torch cu130 is safe +uv pip install -e ".[accel]" +``` + +### Verification +```bash +ft --version +python3 -c " +from freetoken.checkpoint import ftw +import torch +t = torch.randn(3,4,dtype=torch.float16) +print(ftw._dtype_str(t.dtype), ftw._elsize(t.dtype)) +" +``` + +### Key notes +- FTW format is dtype-agnostic: FP16, BF16 (default), FP32, FP8 all work +- GGUF support: only Gemma4GGUFForCausalLM +- GGML types: F32/F16/BF16 (native views), Q4_0/Q6_K (dequant), Q8_0 (listed, no dequant) +- The NVIDIA driver modules may not be loaded at build time (NVML warning) — not a problem for the build itself diff --git a/agent/managed-skills/freetoken-hf-model-swap/SKILL.md b/agent/managed-skills/freetoken-hf-model-swap/SKILL.md new file mode 100644 index 0000000..2e0dfab --- /dev/null +++ b/agent/managed-skills/freetoken-hf-model-swap/SKILL.md @@ -0,0 +1,36 @@ +--- +name: freetoken-hf-model-swap +description: "Swap the FreeToken model behind frank-default-small on Dankserver: preflight the HF quant format against the loader matrix BEFORE downloading, then download, repoint MODEL_DIR, deploy, and verify with a REAL completion (health 200 is not enough)." +--- + +# FreeToken HF Model Swap (frank-default-small, Dankserver) + +Swap the model behind `frank-default-small` (FreeToken v0.1.2, Dankserver, RTX 3090) by downloading a HF safetensors checkpoint into a model dir, repointing `MODEL_DIR` in `scripts/frank-default-small.env`, and deploying. Verify with a REAL completion — health 200 is NOT enough. + +## Preflight — verify quant format BEFORE downloading + +FreeToken v0.1.2 loads ONLY modelopt-style quant configs for `qwen3_5_moe`. Every compressed-tensors FP8 checkpoint fails (live-verified 2026-09-01: two candidates, 72 GB downloaded, both unloadable). Check before any multi-GB download: + +```bash +curl -sL https://huggingface.co///raw/main/config.json | \ + python3 -c "import json,sys; c=json.load(sys.stdin); q=c.get('quantization_config',{}); print(q.get('quant_method'), q.get('weight_block_size'), list((q.get('weight_scale') or ['?','?'])) if isinstance(q.get('weight_scale'),list) else q.get('weight_scale_inv') is not None)" +``` + +Decision matrix: + +- `quant_method: "fp8"` + `weight_block_size: [128,128]` + `weight_scale_inv` → ✅ modelopt block-fp8 (the Huihui/official-Qwen format; only proven FP8 path) +- `quant_method: "compressed-tensors"` → ❌ FAILS. Per-channel aborts at expert-source resolution (`Missing MoE expert source layers: {'gate_up'}` when expert tensor naming is nonstandard); block-fp8 128×128 aborts in `_dequant_fp8_weight` (multiplies `weight × weight_scale` directly; block scales don't broadcast → shape RuntimeError) +- No quant config (BF16) → loads but 35B = ~70 GB weights >> 62 GB system RAM → sustained swap-thrash that locks out SSH (2026-09-01 outage). Never deploy BF16 for a 35B-class model on this box; a clean preflight compares disk size × precision against free RAM + 24 GB VRAM headroom. +- Also check: architecture `qwen3_5_moe`, expert tensor naming `mlp.experts.{N}.gate_proj/up_proj/down_proj` (read `model.safetensors.index.json` weight_map on HF if unsure — nonstandard prefixes defeat the MoE expert-source provider), config.json ships the chat template you want (FreeToken reads it from the checkpoint), disk space ≥ checkpoint size. + +If the checkpoint fails the matrix, STOP — do not download, do not touch the running service. Find a compliant quant or use a different runtime. + +## Swap steps + +1. Download into `/home/dank/models//` via `snapshot_download` with the freetoken venv python (background the SSH call; poll until file count matches the index). +2. Edit `scripts/frank-default-small.env`: `MODEL_DIR`, description line (name, quant, arch, disk size, host, date), header comment. Keep F84 pool numbers unless the weight mass or KV dtype changed (FP8 35B ≈ 35 GB: `MODEL_CACHE_SIZE=2048`, `NUM_TOKENS=453005`, reasoning `on`; BF16 66 GB fits nothing — see preflight). +3. Run `./tests/run.sh`, commit, push to Gitea origin (git-only flow; never edit on the host). +4. `./scripts/deploy.sh dankserver --service frank-default-small`. The LLM_WAIT window is up to 20 min; load of ~35 GB FP8 is minutes warm. +5. **Verify for real** — the deploy health gate has reported "healthy" while a doomed worker was still starting (twice). Check journal for Uvicorn startup complete, then `curl /v1/models` (root = your model dir, ctx 262144) and a REAL completion: reasoning must parse into `reasoning_content` and content must answer correctly with a natural `finish_reason: stop`. Give reasoning models enough `max_tokens` to finish thinking (a tight budget yields empty content + `finish: length` — that is correct behavior, not a failure). +6. If load fails: journal shows the exact loader error, service exits cleanly (no swap-thrash), `Restart=on-failure` re-loop is harmless. Revert env via `git checkout -- `, redeploy, verify. +7. Update the runbook (`serverAppConfigs-docs/LLM/freetoken-frank-default-small.md` quant matrix + history), `log.md`, and AGENTS.md fleet table; stage superseded model dirs to `~/.AGENTTRASH/-topic/` on the host and list sizes for the user — never `rm`. diff --git a/agent/managed-skills/freetoken-rocm-build/SKILL.md b/agent/managed-skills/freetoken-rocm-build/SKILL.md new file mode 100644 index 0000000..5886f4b --- /dev/null +++ b/agent/managed-skills/freetoken-rocm-build/SKILL.md @@ -0,0 +1,49 @@ +--- +name: freetoken-rocm-build +description: "Verify and execute FreeToken build on ROCm hosts — check gfx1151, ROCm 6.3, hipcc, and dependency wheel availability (torch 2.11, sgl-kernel 0.4.5, flashinfer)" +--- + +# FreeToken ROCm Build + +## When to use +Verify FreeToken can build on an ROCm host (Dankclaw, similar systems) before committing to the build path. + +## Steps + +1. **Check hardware** + ``` + rocminfo | grep 'Name.*gfx' # verify GPU architecture + lsmod | grep amdgpu # driver loaded + ls /opt/rocm/bin/ | grep hipcc # compiler available + ``` + +2. **Check build deps** + ``` + cmake --version # needs >=3.20 + python3 --version # needs >=3.10 + gcc --version # needs >=9 + ninja --version # optional but recommended + ``` + +3. **Verify wheel availability** + ``` + pip index versions torch # needs 2.11-2.12 range + pip index versions sgl-kernel # needs 0.4.5 + pip index versions flashinfer-python # needs >=0.6 + ``` + Check PyPI for `rocm` in wheel filenames, or use PyTorch ROCm index. + +4. **Build** + ``` + git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken + python3 -m venv .venv && source .venv/bin/activate + pip install -e ".[accel]" # installs fi + sgl extras + ``` + +## Known issues +- nvcc missing on Dankclaw — not needed for ROCm build, only for CUDA JIT +- sgl-kernel at versioned endpoint may return 404 — use unversioned `pip index versions` +- triton 3.6.0 pins to Linux only + +## CUDA alternative +If building for CUDA: install `sudo pacman -S cuda` for nvcc, or use pre-built CUDA wheels that ship pre-compiled kernels. diff --git a/agent/managed-skills/gui-hang-remote-diagnosis/SKILL.md b/agent/managed-skills/gui-hang-remote-diagnosis/SKILL.md new file mode 100644 index 0000000..d1a2d5f --- /dev/null +++ b/agent/managed-skills/gui-hang-remote-diagnosis/SKILL.md @@ -0,0 +1,46 @@ +--- +name: gui-hang-remote-diagnosis +description: "Diagnose a frozen GUI/VT-switch failure over SSH by reading kernel hung-task evidence, decoding the lock-holder stack, and deciding recoverability" +--- + +# GUI Hang Remote Diagnosis + +Use when the desktop is frozen, VT switches do nothing, but SSH still works. + +## Sequence + +1. **Find the smoking gun** — kernel hung-task reporter already logged it: + ```bash + journalctl -b -p err --since "-2 hours" --no-pager | tail -80 + ``` + Look for `INFO: task X blocked for more than N seconds` and `blocked on a semaphore likely last held by task Y:PID` (modern kernels name the lock holder outright). +2. **Get full stacks** for victim and holder: + ```bash + journalctl -b -k --since "" --no-pager | tail -120 + cat /proc//status | head -5 # State: R (spinning) vs D (blocked) + cat /proc//status | head -5 + ``` + - Victim `D` + holder `R` = livelock/deadlock with lock held. + - `voluntary_ctxt_switches` in the millions on an `R` kthread = spinning, not progressing. +3. **Prove whether hardware or software** — GPU/CPU still alive? No Xid errors? + ```bash + nvidia-smi --query-gpu=driver_version,pstate,temperature.gpu,clocks_event_reasons.active --format=csv + journalctl -b -k --no-pager | grep -i xid + ``` + If those answer instantly with no Xid, it's a driver lock issue, not hardware. +4. **Find the trigger window**: block start = (first hung message timestamp − reported seconds). Then inspect user-session events: + ```bash + journalctl -b --since "" --until "" --no-pager | grep -v audit + ``` +5. **Enumerate collateral**: any other `D` tasks (`ps -eo pid,stat,comm | awk '$2 ~ /D/'`), USB workers, etc. + +## Recoverability verdict + +- Compositor in `D` state → unkillable; spinning kernel thread holding the lock → cannot be poked. + **Only exit is reboot**: `sync && systemctl reboot --force --force` (double-force skips stop-jobs that wait forever on D-state PIDs). +- If holder is `D` (sleeping on something that can complete, e.g. storage), recovery may be possible by clearing that dependency first. + +## After diagnosing + +- Map stack symbols to driver internals to name the exact subsystem (e.g. `DifrPrefetchEventDeferredWork` → NVKMS DIFR prefetch). +- Check upstream: search the exact symbol names + driver version; propose targeted off-switches (`NVreg_RegistryDwords`, module params from `/proc/driver/nvidia/params`) rather than blanket driver swaps. diff --git a/agent/managed-skills/hermes-config-home-drift/SKILL.md b/agent/managed-skills/hermes-config-home-drift/SKILL.md new file mode 100644 index 0000000..0f1bcf8 --- /dev/null +++ b/agent/managed-skills/hermes-config-home-drift/SKILL.md @@ -0,0 +1,18 @@ +--- +name: hermes-config-home-drift +description: "Diagnose Hermes webui/agent model-connection failures caused by HERMES_HOME config-home drift (image VOLUME anonymous volumes resetting on recreation) — verify which home each runtime reads, resolve provider→endpoint with the app's own resolver code, then probe the endpoint." +--- + +# Hermes config-home drift diagnosis + +Symptom: hermes-webui/agent "cannot connect" to models even though the model endpoints answer curl. + +1. **Find the real config home per runtime.** `podman exec printenv HERMES_HOME` for each hermes container. If webui reads `/opt/data` (image default) while the gateway reads `/data`, the webui runs the image-default template config (`provider: auto` → openrouter.ai with no key = connect failure). +2. **Know the recreation trap.** Image `VOLUME` points get a fresh anonymous volume on every container recreation (`podman run --replace --rm`) — edits inside such a mount are lost on restart/redeploy. Compare volume ids before/after a restart to confirm. +3. **Fix:** pin `Environment=HERMES_HOME=/data` in the quadlet unit (shared state root), deploy via service-scoped deployer, restart. +4. **Verify with the app's own code**, not guesses: + - `from api import config as C; C.resolve_model_provider("")` → returns `(model_id, provider, base_url)`; bare ids map to owning provider by scanning the `providers:` dict models allow-list; `@provider:model` syntax forces an explicit provider. + - `hermes_cli.providers.resolve_user_provider(name, cfg["providers"])` → ProviderDef with real `base_url`; accepts `api`/`url`/`base_url` keys on named provider entries. +5. **Probe the endpoint with the exact served model id** (FLM/NPU endpoints route only by exact registry tag and answer HTTP 200 with `{"error":"Failed to load model!"}` for unknown tags — status-code checks miss this). + +Key gotcha: named providers (`providers:` dict) are a catalog + connection source; `custom:` connections resolve only via `custom_providers[]` entries. If neither holds an entry for the provider name, the request has no endpoint and fails to connect. diff --git a/agent/managed-skills/hermes-live-config-edit/SKILL.md b/agent/managed-skills/hermes-live-config-edit/SKILL.md new file mode 100644 index 0000000..d7600ce --- /dev/null +++ b/agent/managed-skills/hermes-live-config-edit/SKILL.md @@ -0,0 +1,21 @@ +--- +name: hermes-live-config-edit +description: "Runbook for changing Hermes (Dankclaw) live config.yaml — model routing, providers, aux routes — including backup, apply, restart, and verification steps" +--- + +# Editing live Hermes config on Dankclaw + +Hermes-webui/gateway read `/home/dank/Containers/hermes/data/config.yaml` (dir is `drwx------` uid 109999; all reads/edits need `sudo`). The repo file `dankclaw/hermes/config.yaml.example` is a sanitized backup and can lag live — always diff the live file first; never treat the example as truth. + +## Procedure +1. **Backup first:** `sudo cp -a .../config.yaml .../config.yaml.bak-YYYYMMDD` (in-dir trash `.AGENTTRASH/` also exists for removals). +2. **Apply change** with surgical `sudo sed -i` or full-file write; YAML comments in the live file are tool-generated — preserve them. +3. **Sweep stale references** across the whole data dir, not just config.yaml: `sudo grep -rln "" /home/dank/Containers/hermes/data --include="*.yaml" --include="*.json"`. Session files/MEMORY.md are historical — leave them. +4. **Derived caches:** `webui/models_cache.json` is keyed by a config fingerprint (`_source_fingerprint.config_yaml` mtime/size); after edits it's stale. Either let it regenerate lazily or trash it (mv to data dir `.AGENTTRASH/`) — never hand-edit. +5. **Restart the model-consuming units** (they parse config at startup): `systemctl --user restart hermes-webui.service hermes.service`. `hermes-dashboard.service` is unaffected by routing changes. +6. **Verify behaviorally**, not by status codes: one tiny completion per endpoint with the exact served id — POST `/v1/chat/completions` on 7770 (Dankclaw, llama.cpp serves alias `frank-large`), on Dankserver 7770 (FreeToken REQUIRES an explicit `model` field), and on 7771/v1 (FLM routes by exact registry tag, e.g. `gemma4-it:e4b`; unknown tags answer HTTP 200 with `{"error":"Failed to load model!"}` — status-code checks silently pass). +7. **Sync the repo backup** (`dankclaw/hermes/config.yaml.example`) to the live shape, redacting secrets (password_hash, phone account stay live-only), then run `./tests/run.sh` before commit. + +## Gotchas +- `provider: custom` + `base_url` entries route by base_url and send the model id verbatim; named providers (`dankclaw-large`, `dankserver-small`, `dankclaw-npu`) resolve via their `api:` URL. +- Renaming a served alias anywhere (serve scripts, env ALIAS) MUST propagate to this config in the same change or Hermes requests 404 on nonexistent models — this exact gap caused the 2026-09-05 outage. diff --git a/agent/managed-skills/hermes-webui-secret-rotation/SKILL.md b/agent/managed-skills/hermes-webui-secret-rotation/SKILL.md new file mode 100644 index 0000000..8502592 --- /dev/null +++ b/agent/managed-skills/hermes-webui-secret-rotation/SKILL.md @@ -0,0 +1,20 @@ +--- +name: hermes-webui-secret-rotation +description: Rotate the Hermex (hermes-webui) password on Dankclaw and verify auth behaviorally — covers the dual EnvironmentFile gotcha where only dankclaw/.env is effective at container create. +--- + +# Hermex webui secret rotation (Dankclaw) + +Auth contract (verified live 2026-08-31): `POST /api/auth/login` with JSON `{"password":"..."}` → 200 + signed HttpOnly `hermes_session` cookie (24 h TTL) on match, 401 on miss. No username field. `/health` answers 200 unauthenticated and `/` answers 302 — never verify auth via those; always probe login with a deliberately wrong password (expect 401) and the real one (expect 200). + +## Rotation procedure +1. `ssh -o BatchMode=yes dankclaw 'bash -s'` with a stdin script (zsh on the host mangles inline `${...}` even inside single-quoted ssh args — never interpolate curl `-w %{http_code}` inline; pipe the whole probe as a heredoc'd bash script). +2. Update BOTH env files with `sed -i "s/^HERMES_WEBUI_PASSWORD=.*/HERMES_WEBUI_PASSWORD=/"`: + - `/home/dank/serverAppConfigs/.env` ([Service] EnvironmentFile) + - `/home/dank/serverAppConfigs/dankclaw/.env` — **the effective one**: the quadlet unit's `[Container] EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env` is what the container reads at create; rotating only the root file leaves the running container serving the old secret. +3. `systemctl --user restart hermes-webui.service` (unit recreation re-reads env; no deploy needed if the unit file itself didn't change). +4. Verify: wrong-pw login → 401, right-pw → 200, from both host loopback and a LAN client at `http://192.168.0.143:8787` (webui publishes on the LAN interface since commit 4443897; off-network ingress is SSH tunnel only — user `tunnel`, port **22**, key-only auth, forward remote end localhost:8787). + +## Gotchas +- `sudo podman exec` fails exit 125 here: root's namespace does not see rootless containers — run `podman exec hermes-webui printenv HERMES_WEBUI_PASSWORD` unprivileged (ssh user owns the env files, mode 600). +- If rotating because a secret leaked into transcripts/logs, rotate in place via a script that fetches the live value from the running container (`printenv`) instead of transcribing it. diff --git a/agent/managed-skills/homepage-status-probe-namespace-triage/SKILL.md b/agent/managed-skills/homepage-status-probe-namespace-triage/SKILL.md new file mode 100644 index 0000000..360d0e2 --- /dev/null +++ b/agent/managed-skills/homepage-status-probe-namespace-triage/SKILL.md @@ -0,0 +1,32 @@ +--- +name: homepage-status-probe-namespace-triage +description: "Triage a homepage status card (or any host-side TCP probe) reporting \"down\" for a service that is actually up: decide by network namespace whether the probe target is reachable at all before calling it an outage." +--- + +# Goal + +Decide, in one pass, whether a `down` card on the homepage status page (`scripts/homepage-status.py`, run by `homepage-status.timer` on Dankclaw, probes TCP/HTTP from the **host** netns) is a real outage or a structurally unreachable probe target. The checker only has three states — `wip` | `up` | `down`; there is no "unknown", so an unprobeable service can never render honestly and must be dropped or re-pointed, not "fixed". + +# Why it goes wrong + +A host-vantage TCP probe can only reach a listener that is **published to the host** (rootless podman publishes via rootlessport/pasta; unpublished ports are invisible from the host netns). Three recurring false-down shapes on this fleet: + +1. **TLS-only app probed with `method:"http"`.** Plex Media Server 1.43 serves TLS only on `32400`; a plain-HTTP GET gets a TLS alert → curl exit 52 "empty reply from server", code `000` → card says down while PMS answers `200` over HTTPS. Fix = `"method":"tcp"` (connect is the honest liveness signal; the public URL stays HTTP because Caddy re-keys). +2. **Retired port.** The Hermes card probed `127.0.0.1:8090` after the compose→quadlet move moved the dashboard to `9119` and the webui to `8787`; nothing listened on 8090, so a fully healthy stack read `down`. +3. **Unpublished port / wrong netns.** `gitea-postgres` (`db:5432`, net `systemd-gitea`), `searxng-valkey` (`valkey:6379`, net `systemd-searxng`), and `alloy` (OTLP 12345) publish **no** ports. Container DNS aliases are netns-scoped, so from the host these are unreachable *by construction* — no probe value makes them honest. + +# Procedure + +1. Read the card's probe target in `shared/homepage/services.json`; note method (`tcp` vs `http`). +2. Ask: does that port have a **host-visible listener**? `podman ps -a --format '{{.Names}}|{{.Status}}'`, then per container `podman inspect --format '{{json .NetworkSettings.Networks}} {{.HostConfig.NetworkMode}}'` and `podman port ` (empty output = published nothing → structurally unprobeable from the host). +3. Probe from the **correct vantage**. From a bridge-net container, the host is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on `dankclaw_default`), *not* `host.containers.internal` (`169.254.1.2`) — that pasta alias belongs to the default `podman` net (`10.88.0.0/16`, gw `10.88.0.1`) and is not routed from other nets. Probe with bash `/dev/tcp` (no `python3`/`ip` in these images), and escape the loop var so it expands **inside** `podman exec bash -c`. +4. Classify: real outage (container exited — read `State.ExitCode`: 137 = SIGKILL, 143 = SIGTERM/manual stop, `OOMKilled=false` rules out the OOM killer) vs false down (running + unreachable target). +5. Fix accordingly: re-point or re-method the probe for a real HTTP/TLS mismatch; **drop** cards whose only exposure is an internal netns (or accept permanent false-downs and say so); never invent a probe target for them. +6. Verify by re-running the checker path, not by eyeballing: after deploy, re-read `/srv/homepage/status.json` and diff the `down` list before/after. + +# Watch out + +- Deploying this file is service-scoped: `./scripts/run.sh` (233 tests) → commit → `./scripts/deploy.sh dankclaw --service homepage`; the deployer *pushes* `main` first, so any already-committed local work rides along — say so before pushing. +- The checker writes `/srv/homepage/status.json` world-readable (0644) so Caddy can serve it; keep that mode or Caddy 403s the page. +- Removing a card's `probe` key without code change crashes the checker (`svc["probe"]` KeyError) — dropping the whole entry is the only zero-code fix. +- Bare compose services with `restart=unless-stopped` (node-exporter on both hosts) do **not** survive reboot: rootless podman does not autostart bare containers, and `deploy-remote.sh` refuses to `compose up` Dankserver's compose services by design → they need a quadlet unit, not a restart. diff --git a/agent/managed-skills/html-change-report/SKILL.md b/agent/managed-skills/html-change-report/SKILL.md new file mode 100644 index 0000000..caaddad --- /dev/null +++ b/agent/managed-skills/html-change-report/SKILL.md @@ -0,0 +1,33 @@ +--- +name: html-change-report +description: Generate and open a self-contained dark-theme HTML change report (Keep a Changelog categories + code samples) for a git repo time window; use when Frank asks for a rendered report of recent work. +--- + +# HTML change report (Keep a Changelog + review aids) + +Produce a self-contained HTML report of repo work in a time window and open it in Firefox. + +## Data gathering +- Window commits: `git log --since="24 hours ago" --pretty='%h|%ad|%s' --date=format:%H:%M` (split on `|`). +- Metrics: `git diff --shortstat ^..HEAD`, per-area file counts via `git diff --name-only … | awk -F/ '{print $1"/"$2}' | sort | uniq -c | sort -rn`. +- Bucket commits into phase/theme rows by time-of-day for the timeline table. + +## Report structure (standard: Keep a Changelog 1.1.0) +1. Executive summary (metrics + 3-5 bullets). +2. Full commit timeline table, phase-color-tagged (border-left per phase). +3. `git show ` command list per theme — Frank uses this for his review pass. +4. Changes by category: Added/Changed/Fixed/Removed/Security; omit empty categories; cite commit hashes per item. Fixed = old behavior was a bug; Changed = intentional. +5. 4-6 notable code excerpts: read real files, pick load-bearing/instructive code, note file:lines + commits + why it matters. HTML-escape everything (`html.escape`). +6. Verification evidence table (what / how / result). +7. Incidents & open risks. +8. Actions required from Frank (numbered, effort hints). + +## HTML requirements +- Self-contained: inline CSS only, NO CDNs (must work from file:// offline). Dark theme CSS vars (#1a1b26 bg family works). +- Sticky top nav, section anchors, `
` blocks, commit hashes as `.tag` chips.
+- Build via python (eval kernel): subprocess for git data, string concat for CSS (avoid f-string brace pain), f-strings for rows.
+
+## Delivery + verification
+- Write to `/home/dank/Reports/--.html` (outside repo, tree stays clean).
+- Validate: python HTMLParser feed (parse errors) + open in headless browser (`write` JSON to xd://browser: open → run with `document.querySelectorAll` counts + fullPage screenshot) → close tab.
+- Open for Frank: `nohup firefox --new-tab file:/// >/dev/null 2>&1 & disown`.
diff --git a/agent/managed-skills/imagegen-openwebui/SKILL.md b/agent/managed-skills/imagegen-openwebui/SKILL.md
new file mode 100644
index 0000000..460409f
--- /dev/null
+++ b/agent/managed-skills/imagegen-openwebui/SKILL.md
@@ -0,0 +1,43 @@
+---
+name: imagegen-openwebui
+description: Runbook for running high-quality local image generation via Open WebUI imagegen API endpoint.
+---
+
+# Open WebUI Image Generation Runbook
+
+## Overview
+Use this procedure to generate images locally using the Open WebUI image generation endpoint.
+This approach leverages local ComfyUI integration configured in Open WebUI.
+
+## Endpoint
+POST to the imagegen endpoint of your Open WebUI instance.
+
+### Example Request
+```python
+import requests
+
+url = "http://:/api/images/generations"
+headers = {"Authorization": "Bearer "}
+data = {
+    "model": "imagegen",
+    "prompt": "A desired image description.",
+    "n": 1,
+    "size": "1024x1024"
+}
+
+response = requests.post(url, headers=headers, json=data)
+```
+
+### Example Response
+Returns an object with base64-encoded image data or a URL.
+
+## Best Practices
+- Keep prompts descriptive for best results.
+- Use the default model unless configured otherwise.
+- Verify that ComfyUI is running and connected before invoking.
+- Check Open WebUI logs for troubleshooting if generation fails.
+
+## Common Issues
+- **Connection refused**: Ensure Open WebUI and ComfyUI are running.
+- **Timeout errors**: Large image sizes may take longer; check ComfyUI resources.
+- **Authentication errors**: Verify API token validity in the request headers.
diff --git a/agent/managed-skills/llama-flash-attn-qsa-fix/SKILL.md b/agent/managed-skills/llama-flash-attn-qsa-fix/SKILL.md
new file mode 100644
index 0000000..4072843
--- /dev/null
+++ b/agent/managed-skills/llama-flash-attn-qsa-fix/SKILL.md
@@ -0,0 +1,73 @@
+---
+name: llama-flash-attn-qsa-fix
+description: Restore --flash-attn on for qwen4exp models when QSA indexer path aborts with GGML_ASSERT F32 error
+---
+
+## Problem
+`--flash-attn on` aborts graph build for qwen4exp models:
+```
+GGML_ASSERT(a->type == GGML_TYPE_F32)
+at ggml/src/ggml.c:5371, backtrace through build_attn_qsa
+```
+Root cause: `build_attn_qsa` calls `ggml_fill` on the F16-cast kq mask returned by `get_kq_mask()` when flash attn is enabled, but `ggml_fill` requires F32.
+
+## Solution
+Apply the QSA F32-mask patch that mirrors the MLA sparse path (`llm_graph_context::build_attn`):
+1. Use the F32 base mask (`inp->self_kq_mask`) instead of the F16-cast version for the top-k surgery
+2. Cast to F16 only at the attention call when `flash_attn` is enabled
+
+### Patch hunks (unified diff format)
+```diff
+@@ -560,9 +565,13 @@ ggml_tensor * llama_model_qwen4exp::graph::build_attn_qsa(
+         ggml_build_forward_expand(gf, mctx_cur->cpy_v(ctx0, v_cur, v_idxs, il));
+     }
+-    ggml_tensor * kq_mask = inp->get_kq_mask();
++    // get_kq_mask() returns the F16-converted mask when flash attention is
++    // on, but ggml_fill/set_rows require F32. Mirror the MLA sparse path
++    // (llm_graph_context::build_attn): run the top-k mask surgery on the F32
++    // base mask and cast to the attention type only at the end.
++    ggml_tensor * kq_mask = inp->self_kq_mask;
+     // prepare new kq mask - starts filled with -INFINITY
+     ggml_tensor * kq_mask_all = ggml_fill(ctx0, kq_mask, -INFINITY);
+ 
+@@ -592,9 +601,12 @@ ggml_tensor * llama_model_qwen4exp::graph::build_attn_qsa(
+     ggml_tensor * q = q_cur;
+     ggml_tensor * k = mctx_cur->get_k(ctx0, il);
+     ggml_tensor * v = mctx_cur->get_v(ctx0, il);
+-    ggml_tensor * cur = build_attn_mha(q, k, v, nullptr, kq_mask_top_k, nullptr, nullptr, kq_scale, il);
++    // flash attention consumes an F16 mask; softmax consumes F32 as-is
++    ggml_tensor * kq_mask_attn = inp->cparams.flash_attn ? ggml_cast(ctx0, kq_mask_top_k, GGML_TYPE_F16) : kq_mask_top_k;
++
++    ggml_tensor * cur = build_attn_mha(q, k, v, nullptr, kq_mask_attn, nullptr, nullptr, kq_scale, il);
+     cb(cur, "kqv_out", il);
+     return cur;
+ }
+```
+
+## Procedure
+1. **Apply patch to fork:**
+   ```bash
+   cd /home/dank/GitPackages/ROCmFPX
+   git apply /path/to/qwen4exp-qsa-f32-mask.patch
+   make -j16 llama-server
+   ```
+2. **Validate live (on test port):**
+   ```bash
+   sudo systemctl stop frank-default-large
+   nohup ./build-qwen38/bin/llama-server \n    --model ... --flash-attn on --ctx-size 32768 --fit off -t 16 -tb 16 \n    --port 7999 > /tmp/fa-test.log 2>&1 &
+   sleep 90 && grep 'model loaded' /tmp/fa-test.log && curl http://127.0.0.1:7999/health
+   ```
+3. **Verify completion:** Send a real `/v1/chat/completions` request — if it completes without asserts, the fix works.
+4. **Deploy production:** Commit config (FLASH_ATTN=on), push, pull on target, restart service.
+
+## Verification
+- Model loads without asserts
+- Health endpoint returns 200
+- Real completion succeeds (finish: "stop")
+- Journal clean: no `GGML_ASSERT`, no `cudaMalloc`, no `missing tensor`
+- Flash attention engaged (check server logs for "flash attention" or timing improvement vs off)
+
+## Known Constraints
+- Works only with charlie12345/ROCmFPX build-qwen38 (PR #27742 qwen4exp merged)
+- Does NOT work with CIRU runtime, upstream llama.cpp, or stock binaries
+- ctx-size cap remains 32768 for full-model GPU residency (87 GiB model + KV buffer OOMs beyond)
diff --git a/agent/managed-skills/llm-fleet-model-swap/SKILL.md b/agent/managed-skills/llm-fleet-model-swap/SKILL.md
new file mode 100644
index 0000000..5c1d632
--- /dev/null
+++ b/agent/managed-skills/llm-fleet-model-swap/SKILL.md
@@ -0,0 +1,41 @@
+---
+name: llm-fleet-model-swap
+description: "Swap the model/runtime behind frank-default-large/small on Dankclaw/Dankserver via the git flow, with pre-deploy live validation and the port-holder health-check trap check."
+---
+
+# LLM fleet model swap (frank-default-large / frank-default-small)
+
+Use when switching model, quant, or llama.cpp/FreeToken runtime for a `frank-default-*` service in `~/Projects/serverAppConfigs`.
+
+## 0. Ground truth first
+- Read the HF model card (runtime pin, hard constraints, recommended args) + repo files `scripts/frank-default-.env`, `scripts/serve-frank-.sh`, `scripts/frank-default-.service`, and the runbook under `serverAppConfigs-docs/LLM/`.
+- Check the host before downloading anything: `ls /home/dank/models/`, `df -h /home`, `du -sh` — bench-era or prior-deployment files often already exist.
+
+## 1. Artifact integrity
+- List repo files: `curl -s https://huggingface.co/api/models//tree/main?recursive=true` → `.lfs.oid` holds the sha256.
+- On-host file: compare size AND `sha256sum` against the LFS oid before trusting any existing copy (hash runs ~few min for >100 GiB; run it in background).
+- New download on Dankserver: `/home/dank/freetoken/.venv/bin/hf download  --local-dir ...`.
+
+## 2. Runtime (if the card pins a fork/build)
+- Clone + build with the card's exact cmake line, **plus `-DLLAMA_BUILD_WEBUI=OFF`** — WebUI asset provisioning (npm / HF llama-ui bucket) aborts llama.cpp link at 98% on these hosts; API server unaffected.
+- Verify every flag our serve script passes exists in the new binary before touching the service:
+  `llama-server --help | grep -q -- ""` for each; list MISSes.
+
+## 3. Config via git only
+- Edit `scripts/frank-default-.env` (LLAMA_BIN, MODEL_DIR, MODEL_FILE, MMPROJ_PATH) + serve script header/defaults; `bash -n` the script; `./tests/run.sh` must pass; commit; push; `git pull --ff-only` on the target.
+- Do NOT edit configs on the remote. Renaming a model dir (`mv`) is host state, allowed.
+
+## 4. Live validation (no scratch ports — GPU/RAM can't host two big models)
+- Ensure the old unit is truly gone: `systemctl is-active` may hang in `stop-sigterm` because llama.cpp never preempts an in-flight generation; poll until the PID exits before launching.
+- Stop service, run the real serve script manually (`setsid nohup bash scripts/serve-frank-.sh > /tmp/x.log 2>&1 &` — plain `ssh 'nohup ... &'` dies with its ssh when foregrounded; keep ssh under timeout or it gets killed mid-stop).
+- Verify: health 200; a real `/v1/chat/completions` (FreeToken REQUIRES explicit `"model"` field; llama.cpp servers don't care); reasoning flows (`reasoning_content` non-empty; content may be empty with finish=length if the thinking eats the budget).
+
+## 5. systemd handoff + THE TRAP
+- `./scripts/deploy.sh  --service ` — but a leftover manual server holding :7770 makes the health check pass while the UNIT crash-loops on bind.
+- Always confirm ownership: `systemctl show  -p MainPID` equals the single `pgrep -x llama-server` (or the ft serve PID); only then trust "healthy after Ns".
+- If a real long-context task is queued/running, do the handoff anyway (client aborts are recoverable; drift is not) — or defer cosmetic-only redeploys.
+
+## 6. Close-out (mandatory)
+- Docs: active runbook updated (build recipe, profile, validation numbers incl. real-workload prefill/decode), superseded runbook gets RETIRED banner kept as rollback record, `LLM/index.md` mermaid + active list, AGENTS.md fleet table + Runtime section + history chain, work log entry in `serverAppConfigs-docs/log.md`.
+- `~/.pi/agent/models.json`: contextWindow must equal the REAL per-slot context — llama.cpp logs `n_ctx_slot` per prompt; parallel N clamps `n_ctx_train` (e.g. 262144/2 = 131072/slot). Overselling causes silent client-side overflow.
+- Rollback = one env commit (paths + LLAMA_BIN); keep prior binary + payload on disk.
diff --git a/agent/managed-skills/llm-model-swap-fit-preflight/SKILL.md b/agent/managed-skills/llm-model-swap-fit-preflight/SKILL.md
new file mode 100644
index 0000000..01afe49
--- /dev/null
+++ b/agent/managed-skills/llm-model-swap-fit-preflight/SKILL.md
@@ -0,0 +1,50 @@
+---
+name: llm-model-swap-fit-preflight
+description: "Preflight an LLM model swap for hardware fit before deploying; catch the silent swap-thrash outage mode (large RAM OOM, self-restarting + boot-enabled units) and recover."
+---
+
+# LLM Model-Swap Fit Preflight
+
+Never deploy a model swap without confirming the checkpoint FITS, and know the
+failure mode when it doesn't: it is a silent, self-sustaining thrash, not a crash.
+
+## Preflight (before deploy)
+
+1. Get the checkpoint's on-disk/weight size AND precision from the HF card
+   (`Precision: BF16` = 2 bytes/param; FP8/Q8 ≈ 1 byte; Q4 ≈ 0.5 byte).
+   35B params: BF16 ≈ 66–70 GB, FP8/Q8 ≈ 33–35 GB, Q4 ≈ 17–20 GB.
+2. Compare against the HOST'S USABLE RAM (not just VRAM) for runtimes that
+   stream weights from RAM (FreeToken host-RAM experts, llama.cpp CPU offload,
+   PP/worker splits).
+3. Budget RAM for weights + KV pool + OS + page cache. If weights alone exceed
+   RAM, ABORT — pool tuning shrinks KV, it does not shrink weights.
+
+## The self-sustaining thrash (why 'it'll just OOM and recover' is wrong)
+
+- Large swap (tens of GB) → kernel PAGES instead of OOM-killing → sshd/user
+  space starved indefinitely; no recovery window opens by itself.
+- systemd `Restart=on-failure|always` → crash-reload-rethrash loop.
+- Unit `enabled` + `WantedBy=multi-user.target` → a reboot re-thrashes on boot
+  until the bad config is masked/reverted. A plain power-cycle does NOT fix it.
+
+## Recovery (needs a shell; may need console / SysRq-to-GRUB)
+
+```
+sudo systemctl mask --now         # stop + break the reboot trap FIRST
+cd  && git fetch origin && git reset --hard origin/main   # pick up revert
+grep MODEL_DIR                    # confirm fixed target
+sudo systemctl unmask  && sudo systemctl restart 
+curl -s                         # then verify with a REAL completion
+```
+
+## Deploy-script caveat
+
+`deploy.sh`/health-gate 'healthy after Ns' = systemd ACTIVE, not model LOADED.
+Always confirm load with a real completion (curl /v1/chat/completions), not
+status alone.
+
+## GPU quant note
+
+RTX 3090 (Ampere) has no FP8/FP4 tensor cores: FP8 weights run via upcast but
+still must fit in RAM/VRAM. Choose quant by SIZE fit, then accept the compute
+path the card supports.
diff --git a/agent/managed-skills/pico8-asset-standards/SKILL.md b/agent/managed-skills/pico8-asset-standards/SKILL.md
new file mode 100644
index 0000000..a161cdf
--- /dev/null
+++ b/agent/managed-skills/pico8-asset-standards/SKILL.md
@@ -0,0 +1,46 @@
+---
+name: pico8-asset-standards
+description: "Standards for building cohesive, production-grade PICO-8 asset sheets, tilesets, and spritesheets with consistent palette, perspective, and detail flow."
+---
+
+# PICO-8 Asset Standards
+
+## Purpose
+Ensure every PICO-8 asset (tileset, spritesheet, character sheet, or asset sheet) is cohesive, production-grade, and ready to drop into a game without rework.
+
+## Core Requirements
+
+### Palette Discipline
+- Use the PICO-8 16-color palette exclusively.
+- Establish a dominant color family (2–3 related hues) for the main subject.
+- Reserve accent colors for highlights, danger zones, and focal points only.
+- Maintain consistent color roles across all assets in a sheet (e.g., if shadow is dark blue on one tile, it is dark blue everywhere).
+
+### Perspective & Lighting
+- Choose ONE perspective per sheet (top-down, side-view, or isometric) and never mix.
+- Apply light from a single consistent direction across all tiles and sprites.
+- Shadows follow the same shape logic on every asset (e.g., all cast shadows point bottom-right).
+
+### Detail Flow & Hierarchy
+- Start tiles/props from the palette's dominant colors, then add edge highlights and shading.
+- Keep tile density uniform — avoid one hyper-detailed tile next to empty flat tiles.
+- Props and decorations should read clearly at 8x8 or 16x16 without zooming.
+
+### Sheet Organization
+- Group related assets together (terrain together, props together, characters together).
+- Leave consistent spacing between assets for clean cropping.
+- Label sections visually through grouping, not text.
+
+### Cohesion Checks
+Before finishing an asset sheet, verify:
+- All colors appear in the PICO-8 palette.
+- Perspective does not flip mid-sheet.
+- Lighting direction is consistent.
+- Edge treatment (dither, outline, anti-aliasing style) matches across assets.
+- No orphan tiles that don't connect to any neighbor.
+
+## Common Pitfalls
+- Mixing 8-bit and 16-bit detail levels.
+- Using outline style on some tiles but not others.
+- Forgetting palette constraint — colors outside the 16 PICO-8 colors.
+- Inconsistent tile edge connections (a grass tile that only connects to one dirt variant).
diff --git a/agent/managed-skills/podman-netavark-host-port-probe/SKILL.md b/agent/managed-skills/podman-netavark-host-port-probe/SKILL.md
new file mode 100644
index 0000000..0b87463
--- /dev/null
+++ b/agent/managed-skills/podman-netavark-host-port-probe/SKILL.md
@@ -0,0 +1,29 @@
+---
+name: podman-netavark-host-port-probe
+description: "Diagnose \"connection refused\" when a rootless-podman container tries to reach a host service port on this netavark host; proves the published-port-only quirk and picks the working path (host.containers.internal)."
+---
+
+# Podman/netavark: container → host port connectivity probe
+
+Use when a containerized service on a rootless-podman + netavark host cannot reach a HOST process by LAN IP (connection refused immediately, not a timeout) even though `ss -ltnp` proves the listener is up in the host netns.
+
+## Why this happens
+On netavark-backed networks, container→host-LAN-IP traffic is only delivered for ports that have a live port-forward/DNAT (published ports). Un-published host ports refuse instantly — there is no firewall rule and no stale DNAT to find; `nft list tables`/`iptables -t nat` will be empty. Do not hunt for REJECT rules; probe instead.
+
+## Probe recipe (proves the quirk in one command)
+```bash
+ssh  "podman run --rm --network  --entrypoint sh docker.io/library/busybox:latest -c 'for p in  ; do nc -w2 -z -v  \$p >/dev/null 2>&1 && echo port-\$p-OK || echo port-\$p-BAD; done'"
+```
+- Published port OK + target port BAD ⇒ published-port-only quirk confirmed.
+- Also probe the alias path from inside the actual service container:
+```bash
+podman exec  bash -c 'exec 3<>/dev/tcp/host.containers.internal/ && echo ALIAS-OK || echo ALIAS-FAIL'
+```
+Note: busybox `nc -z` reports any failure as failure — use it for bulk port sweeps; use the `/dev/tcp` probe (or `nc -v`) when the refusal-vs-timeout distinction matters.
+
+## Fix decision
+- If the alias path connects and the LAN-IP path refuses → set the service's host-facing URL to `http://host.containers.internal:` in its compose/quadlet env, then redeploy via the normal git flow (`deploy.sh  --service ` recreates the container with fresh env).
+- Do NOT publish the daemon port just to make the LAN IP work — publishing is exactly what makes other ports refuse.
+
+## Post-fix verification
+`podman exec  printenv ` then tail logs for the connect confirmation (e.g. `Signal: connected`, `SSE: connected`) within ~40 s of recreation.
diff --git a/agent/managed-skills/podman-netns-addressability-probe/SKILL.md b/agent/managed-skills/podman-netns-addressability-probe/SKILL.md
new file mode 100644
index 0000000..02ff4eb
--- /dev/null
+++ b/agent/managed-skills/podman-netns-addressability-probe/SKILL.md
@@ -0,0 +1,51 @@
+---
+name: podman-netns-addressability-probe
+description: "Runbook to answer \"can container X actually reach service Y\" on a rootless-podman/netavark host, instead of assuming localhost works everywhere; includes the probe commands and the false-positive traps."
+---
+
+# Podman netns addressability probe
+
+Answer **"can this container actually dial that service?"** per *network namespace*,
+not per host. On rootless podman + netavark, a bridge-net container cannot reach
+host-netns loopback listeners at all, so `localhost:PORT` reasoning is wrong for it.
+
+## Decision rule
+- `NetworkMode=host` → the container **is** the host netns: dials everything on
+  `127.0.0.1`, binds any free port.
+- bridge/named network → own netns. Only **published** ports are reachable, via the
+  network's **gateway IP**. Unpublished host listeners are **refused instantly (RST)**,
+  not routed — even when the listener is bound `0.0.0.0`.
+
+## Probe sequence (all read-only, run on the host)
+1. Namespace: `podman inspect  --format '{{.HostConfig.NetworkMode}}'`
+2. Owner of a listener: `sudo ss -ltnp 'sport = :'` (host netns only — this is
+   why a `0.0.0.0` listener can still be unreachable from a bridge net).
+3. From inside the suspect container:
+   `podman exec  cat /proc/net/route` → default route = gateway IP to dial.
+4. Probe the candidate address **inside that netns**:
+   `podman exec  sh -c 'timeout 2 bash -c "exec 3<>/dev/tcp//"; echo rc=$?'`
+
+## Interpretation (do not blur these)
+- **instant refusal** = nothing listening in a reachable namespace (or rootlessport
+  rejected an unpublished port map).
+- **timeout** = no route to that address at all. `host.containers.internal`
+  (`169.254.1.2`) is an alias on the **default `podman` net only**; containers on a
+  named network time out against it rather than being refused.
+
+## Probe-tool gotchas verified on this host (podman 5.x, netavark)
+- `podman network inspect  --format '{{json .containers}}'` **errors**
+  (`can't evaluate field containers in type interface {}`) — membership is not exposed
+  there. List from the container side instead:
+  `podman ps -a --format '{{.Names}} {{.Networks}}'`
+- An **empty** `Networks` field in that output = host network or a shared namespace
+  (`--network=container:`), not "no network".
+- Quadlet `Network=.network` yields runtime network name **`systemd-`**;
+  probe with `podman network ls`, never by the `.network` filename.
+
+## Fix options when a bridge-net container must reach a host-netns listener
+(a) move it to `Network=host`, or (b) publish the port — (b) makes that port reachable
+from every namespace **and the LAN**, so it is an exposure change needing explicit
+operator approval, never a silent fix.
+
+Repo source of truth for this fleet: `serverAppConfigs-docs/Architecture/Container
+Network Namespaces.md` (commit `2e179b0`).
diff --git a/agent/managed-skills/podman-recreate-anon-volume-wipe/SKILL.md b/agent/managed-skills/podman-recreate-anon-volume-wipe/SKILL.md
new file mode 100644
index 0000000..aeb3b15
--- /dev/null
+++ b/agent/managed-skills/podman-recreate-anon-volume-wipe/SKILL.md
@@ -0,0 +1,28 @@
+---
+name: podman-recreate-anon-volume-wipe
+description: "Diagnose \"container data wiped after restart\" on this rootless-podman fleet: quadlet restarts recreate containers via podman run --replace --rm, and anonymous volumes reset to empty on every recreation. Includes the live audit procedure and the known affected services (hermes trio /opt/data, romm /romm DB, searxng cache)."
+---
+
+# Podman recreation & anonymous-volume wipe audit
+
+## Mechanism (verified live 2026-08-31 on Dankclaw)
+A quadlet unit restart is NOT a start of the same container: the generated unit's ExecStart is
+`podman run --name systemd-%N --replace --rm ...`. Every `systemctl --user restart .service` (and every deploy that does one, e.g. `deploy_hermes`) **destroys and recreates** the container. Consequences at recreation time:
+
+- Writable-layer writes: gone by definition.
+- Anonymous volumes (image-declared `VOLUME` with no explicit `-v`/`Volume=` for that path): NOT carried across recreation — the new container gets a FRESH EMPTY one seeded only from image content. Orphans stay on disk until pruned (`podman volume prune` only removes unmounted ones).
+- Named volumes and bind mounts: persist (remounted by name/path).
+
+## Audit procedure
+1. `ssh  'podman ps -a'` — map which host runs which stack (aliases `dankclaw`/`dankserver` match their hostnames).
+2. Per container: `podman inspect  --format "{{json .Mounts}}"` → classify each mount `"Type":"bind"` vs `"Type":"volume"` (hex-name volume = anonymous).
+3. Quirks: `podman ps --format` template field is `.Names`, not `.Name` (psReporter has no `Name` field — using `.Name` errors). Volume `_data` dirs under `~/.local/share/containers/storage/volumes//_data` are mode 0700 — unreadable by the plain ssh user; use `podman unshare`/sudo if content inspection is required.
+
+## Fleet state at audit time
+- **hermes trio** (`systemd-hermes`, `hermes-dashboard`, `hermes-webui`; gateway container auto-names `systemd-` because its unit lacks `ContainerName=`): `/data` = bind of `/home/dank/hermes-data` (durable state: config.yaml, sessions/, skills/, state.db) survives recreations; `/opt/data` (image default VOLUME + `HERMES_LAZY_INSTALL_TARGET`) is an anonymous volume → resets to image defaults on every deploy. So "Hermes lost data after restart" = anything written outside `/data`/`/webui` was reset, not the durable state.
+- **romm** (dankserver): runs with anonymous volumes at `/romm` and `/redis-data` → its DB lives in an anon volume; recreating romm (`podman-compose up -d` after image bump) boots it with a fresh empty DB. Fix = named volume or bind before next recreation.
+- **searxng**: `/var/cache/searxng` is an anonymous volume — cache only, benign reset on recreation.
+- Everything else on both hosts was verified as bind mounts (or stateless by design: authentik-server has no mounts at all; anubis is a memory store with client-side JWTs).
+
+## Fixes when data must survive recreation
+Pin a NAMED volume (`Volume=hermes-data:/opt/data` in the quadlet unit) or point the env var at a bind-mounted path. Never rely on anonymous volumes surviving recreation.
diff --git a/agent/managed-skills/porkbun-dns-ops/SKILL.md b/agent/managed-skills/porkbun-dns-ops/SKILL.md
new file mode 100644
index 0000000..a2b8478
--- /dev/null
+++ b/agent/managed-skills/porkbun-dns-ops/SKILL.md
@@ -0,0 +1,26 @@
+---
+name: porkbun-dns-ops
+description: "Runbook for mutating Porkbun DNS records for franksdank.cloud via API from the target host, covering the relative-name edit quirk, rate-limit failure shape, and verify steps."
+---
+
+# Porkbun DNS ops (franksdank.cloud)
+
+Operate the public DNS zone for `franksdank.cloud` via the Porkbun API v3, run **on Dankclaw** so the API key/secret sourced from `/home/dank/serverAppConfigs/.env` (`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) never leaves the host. Pipe scripts to the target with `ssh -o BatchMode=yes dankclaw 'bash -s' < /tmp/script.sh`.
+
+## Endpoints (all POST, JSON body)
+- List: `https://api.porkbun.com/api/json/v3/dns/retrieve/franksdank.cloud` — auth via headers `X-API-Key` + `X-Secret-API-Key` (Bearer is wrong here). Response shape is **top-level `.records[]`**, NOT `.content.records[]` as some docs claim — parse with `jq -r '.records[] | [.id,.name,.type,.content,(.ttl|tostring)] | @tsv'`.
+- Delete: `/dns/delete/{domain}/{id}` (body carries apikey+secretapikey).
+- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`, `type`, `content`, `ttl`, `prio`).
+
+## Quirks that bite (verified live 2026-08-31)
+1. **`name` is a RELATIVE label on edit.** Passing `ai.franksdank.cloud` yields the record `ai.franksdank.cloud.franksdank.cloud`. Pass the bare label (`ai`) and the API appends the zone itself. Verify after every rename with a retrieve before declaring done.
+2. **Rate limit answers as an HTML 404-style page, not JSON.** A batch of rapid mutations makes some calls return `...` where jq then errors (`Cannot iterate over null` / `Invalid numeric literal`). Do not assume the call succeeded — sleep ~8 s and retry the failed ids; verify by re-listing the zone at the end.
+3. Deletes/renames are per-record-id: always retrieve first, map name→id, then mutate by id.
+
+## Rename procedure (vhost rename = DNS + Caddy together)
+1. `retrieve` → collect ids for the renamed names.
+2. `edit` each id with the new **relative** label, preserving type/content/ttl.
+3. Re-retrieve and confirm `.franksdank.cloud A `.
+4. In-repo lockstep before deploy: rename the Caddyfile vhost block AND the Pi-hole `dankclaw/pihole/addn-hosts` line in the same commit — test 178 (`tests/pihole.bats`) fails the suite if any Caddyfile vhost lacks an addn-hosts A record on 192.168.0.143.
+5. Deploy via `scripts/deploy.sh dankclaw` (plans pihole+caddy from changed paths). Capture the suite's true exit with `out=$(./tests/run.sh 2>&1); rc=$?` — piping to `tail` masks the exit code behind `tail`'s own status and has shipped untested commits here.
+6. Verify live: `dig +short  @192.168.0.143` then `curl -o /dev/null -w %{http_code} https://` (expect 200/302; first hit triggers dns-01 issuance, which works without inbound port 80).
diff --git a/agent/managed-skills/vhost-proxy-health-probes/SKILL.md b/agent/managed-skills/vhost-proxy-health-probes/SKILL.md
new file mode 100644
index 0000000..b595c9e
--- /dev/null
+++ b/agent/managed-skills/vhost-proxy-health-probes/SKILL.md
@@ -0,0 +1,30 @@
+---
+name: vhost-proxy-health-probes
+description: Verify reverse-proxied homelab services from the client side with status-code probe batteries; decode 404/502/login-page answers instead of trusting 200s
+---
+
+# vhost proxy health probes
+
+After deploying or auditing a Caddy/Anubis-style front door, verify services **from a client**, not from the backend's localhost — that is where transport mismatches, auth walls, and dead upstreams show up.
+
+## Probe battery (per vhost, from a LAN client)
+
+```
+curl -s -o /dev/null -m 6 -w "%{http_code}\n" https:///ping
+curl -s -o /dev/null -m 6 -w "%{http_code}\n" https:///
+```
+
+## Decoding the answer — a status code is a diagnosis, not a failure
+
+- `200` on `/ping` from *arrs = alive; **auth ON/OFF unknown** (`/ping` bypasses auth by design) — probe `/` to decide.
+- `404` = backend **alive**, route missing (qBittorrent has no `/ping`; 404 there = healthy).
+- `502` = Caddy could not reach/negotiate with the upstream: dead backend **or transport mismatch**. Probe the backend port directly from the proxy's network (`ssh  curl http://:/...`). Direct 200 + vhost 502 ⇒ config bug, not an outage.
+- Login page to anonymous (grep body for `password`) = auth ON; instant redirect to `/transferlist/`-style routes = auth OFF = open door.
+
+## Transport-mismatch mechanism
+
+`transport http {versions h2c 1.1}` makes Caddy send an h2c upgrade preface; .NET/Kestrel backends (Jellyfin) reject it → 502 per request while the same port answers 200 on plain HTTP/1.1. Fix = drop the transport override (default HTTP/1.1). Opposite twin: Plex PMS drops *plain* HTTP off-loopback and needs a TLS transport. Match the transport to what the backend speaks.
+
+## Shell trap when gating commits on a test suite
+
+`grep -c "^not ok"` exits **1 when zero mismatches** — inside an `&&` chain it silently kills everything after it while still printing `0`. Wrap: `(./tests/run.sh | grep -cE '^not ok' || true) && git commit ...`.
diff --git a/agent/memories/--home-dank-.omp--/skills/niri_compositor_ops/SKILL.md b/agent/memories/--home-dank-.omp--/skills/niri_compositor_ops/SKILL.md
new file mode 100644
index 0000000..a02e73b
--- /dev/null
+++ b/agent/memories/--home-dank-.omp--/skills/niri_compositor_ops/SKILL.md
@@ -0,0 +1 @@
+Provides operational playbook for niri Wayland compositor. Covers cursor/Browser XDG defaults, SIGUSR1 hot-reload workflow, and nvidia-modeset semaphore deadlock recovery via modprobe config and force-reboot.
diff --git a/agent/memories/--home-dank-.omp--/skills/niri_compositor_ops/scripts/scripts/reload.sh b/agent/memories/--home-dank-.omp--/skills/niri_compositor_ops/scripts/scripts/reload.sh
new file mode 100644
index 0000000..be2c74f
--- /dev/null
+++ b/agent/memories/--home-dank-.omp--/skills/niri_compositor_ops/scripts/scripts/reload.sh
@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+# Hot-reload niri without killing session
+if pgrep -x niri > /dev/null; then
+  niri-msg reload
+  # Fallback signal method
+  kill -SIGUSR1 $(pgrep niri)
+fi
diff --git a/agent/memories/--home-dank-.omp--/skills/niri_compositor_ops/templates/templates/nvidia_workarounds.conf b/agent/memories/--home-dank-.omp--/skills/niri_compositor_ops/templates/templates/nvidia_workarounds.conf
new file mode 100644
index 0000000..90e199e
--- /dev/null
+++ b/agent/memories/--home-dank-.omp--/skills/niri_compositor_ops/templates/templates/nvidia_workarounds.conf
@@ -0,0 +1,3 @@
+# /etc/modprobe.d/nvidia-workarounds.conf
+# Resolves nvidia-modeset semaphore deadlock freezing niri	options nvidia-modeset disable_vga=1
+options nvidia-modeset modeset=1
diff --git a/agent/memories/--home-dank-.omp--/skills/omp_git_workflow/SKILL.md b/agent/memories/--home-dank-.omp--/skills/omp_git_workflow/SKILL.md
new file mode 100644
index 0000000..aeccca1
--- /dev/null
+++ b/agent/memories/--home-dank-.omp--/skills/omp_git_workflow/SKILL.md
@@ -0,0 +1 @@
+Git operations tailored for OMP environment. Addresses dirty `agent/memories/` tracking pitfalls, implements stash-before-checkout protocol, and standardizes PR creation in dank/command-center.
diff --git a/agent/memories/--home-dank-.omp--/skills/omp_git_workflow/scripts/scripts/prepare_checkout.sh b/agent/memories/--home-dank-.omp--/skills/omp_git_workflow/scripts/scripts/prepare_checkout.sh
new file mode 100644
index 0000000..1464081
--- /dev/null
+++ b/agent/memories/--home-dank-.omp--/skills/omp_git_workflow/scripts/scripts/prepare_checkout.sh
@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+# Safely checkout main in OMP
+if [ -n "$(git status -s agent/memories/)" ]; then
+  git stash
+  echo "Stashed dirty agent/memories/ files."
+fi
+git checkout main
+git pull origin main
diff --git a/agent/memories/--home-dank-.omp--/skills/omp_git_workflow/templates/templates/pr_branch.md b/agent/memories/--home-dank-.omp--/skills/omp_git_workflow/templates/templates/pr_branch.md
new file mode 100644
index 0000000..dd94464
--- /dev/null
+++ b/agent/memories/--home-dank-.omp--/skills/omp_git_workflow/templates/templates/pr_branch.md
@@ -0,0 +1,5 @@
+# PR Branch Convention
+- Branch: pr/fix-models
+- Target: main
+- Strategy: Single commit, clean history
+- Workflow: Feature -> PR -> Squash/Merge
diff --git a/agent/memories/--home-dank-.omp--/skills/omp_harness_config/SKILL.md b/agent/memories/--home-dank-.omp--/skills/omp_harness_config/SKILL.md
deleted file mode 100644
index 6877781..0000000
--- a/agent/memories/--home-dank-.omp--/skills/omp_harness_config/SKILL.md
+++ /dev/null
@@ -1,20 +0,0 @@
-# OMP Harness & Migration Playbook
-
-## Environment Setup
-- Home: /home/dank/
-- Config: ~/.omp/agent/APPEND_SYSTEM.md (replaces Pi's SYSTEM.md)
-- Prompt Behavior: APPEND renders after default harness prompt. Use `omp config set personality none` to clear built-in personality while preserving custom APPEND.
-- Skills: Auto-discovered from ~/.claude/skills. Trigger via /skill:.
-
-## Tool Migration (Pi → OMP)
-| Pi | OMP | Notes |
-|---|---|---|
-| subagent | task | Direct replacement |
-| concurrency: N | Dropped | Harness auto-queues excess |
-| context: "fresh"/"fork" | Dropped | Subagents start blank/self-contained |
-| async: true | Dropped | Use local:// unique paths if needed |
-
-## Prompt Maintenance
-- Remove hardcoded dates (OMP injects real time).
-- Keep persona constraints in APPEND_SYSTEM.md.
-- Reference omp:// docs for extensions/hooks/mcp-config.
diff --git a/agent/memories/--home-dank-.omp--/skills/omp_model_config/SKILL.md b/agent/memories/--home-dank-.omp--/skills/omp_model_config/SKILL.md
new file mode 100644
index 0000000..c4b7c5d
--- /dev/null
+++ b/agent/memories/--home-dank-.omp--/skills/omp_model_config/SKILL.md
@@ -0,0 +1 @@
+Handles dual config synchronization, Ki-style context sizing, and validation for Frank's OMP model definitions. Covers lock-step editing of models.json and models.yml, contextWindow/maxTokens mirroring, and FLM WebP constraints.
diff --git a/agent/memories/--home-dank-.omp--/skills/omp_model_config/scripts/scripts/validate.sh b/agent/memories/--home-dank-.omp--/skills/omp_model_config/scripts/scripts/validate.sh
new file mode 100644
index 0000000..abcbd7a
--- /dev/null
+++ b/agent/memories/--home-dank-.omp--/skills/omp_model_config/scripts/scripts/validate.sh
@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+# Validates model config sync
+jq -e '.providers."llamacpp (dankserver)".models[0] | .contextWindow, .maxTokens' ~/.pi/agent/models.json
+python3 -c "import yaml; d=yaml.safe_load(open('/home/dank/.omp/agent/models.yml')); m=d['providers']['llamacpp (dankserver)']['models'][0]; print(m['id'], m['contextWindow'], m['maxTokens'])"
diff --git a/agent/memories/--home-dank-.omp--/skills/omp_model_config/templates/templates/context_update.json b/agent/memories/--home-dank-.omp--/skills/omp_model_config/templates/templates/context_update.json
new file mode 100644
index 0000000..e1d5c6a
--- /dev/null
+++ b/agent/memories/--home-dank-.omp--/skills/omp_model_config/templates/templates/context_update.json
@@ -0,0 +1,4 @@
+{
+  "contextWindow": 204800,
+  "maxTokens": 204800
+}
diff --git a/agent/memories/--home-dank-.omp--/skills/peon_ping_integration/SKILL.md b/agent/memories/--home-dank-.omp--/skills/peon_ping_integration/SKILL.md
deleted file mode 100644
index d5fb1b6..0000000
--- a/agent/memories/--home-dank-.omp--/skills/peon_ping_integration/SKILL.md
+++ /dev/null
@@ -1,21 +0,0 @@
-# Peon-Ping Skill Integration
-
-## Available Skills
-- peon-ping-config
-- peon-ping-log
-- peon-ping-toggle
-- peon-ping-use
-
-## Usage Playbook
-1. Locate skills in ~/.claude/skills (shared across agents).
-2. Trigger via slash commands: /skill:grilling, /skill:tdd.
-3. skills.enableSkillCommands defaults to true.
-4. Use appropriate sub-skill based on operation:
-   - Config: peon-ping-config to set thresholds/paths.
-   - Monitor: peon-ping-log for status/output.
-   - Control: peon-ping-toggle to enable/disable.
-   - Execute: peon-ping-use to run targeted checks.
-
-## Notes
-- Skills persist across agent sessions.
-- Integrate with OMP task tool for asynchronous monitoring if needed.
diff --git a/agent/memories/--home-dank-Projects-serverAppConfigs--/skills/caddy-auth-dns-playbook/SKILL.md b/agent/memories/--home-dank-Projects-serverAppConfigs--/skills/caddy-auth-dns-playbook/SKILL.md
new file mode 100644
index 0000000..e668652
--- /dev/null
+++ b/agent/memories/--home-dank-Projects-serverAppConfigs--/skills/caddy-auth-dns-playbook/SKILL.md
@@ -0,0 +1,29 @@
+# Caddy, Authentik, Anubis, and split-DNS playbook
+
+Use this when modifying franksdank.cloud routes, DNS records, or auth wrapping.
+
+Architecture:
+- Single Caddy container on Dankclaw fronts services on Dankclaw and Dankserver.
+- LAN DNS via Pi-hole resolves service names to 192.168.0.143.
+- Public DNS via Porkbun points exposed names to WAN IP 71.227.227.172.
+- TLS uses Porkbun DNS-01 wildcard certificates; ordinary A-record deletion does not break wildcard renewal.
+- Porkbun credentials are machine-local secrets, never tracked.
+
+Caddy rules:
+- Keep Caddy admin enabled on 127.0.0.1:2019 inside the container so file watcher hot-reload works.
+- Caddy routes by Host header; it does not care whether the name resolved via Pi-hole or Porkbun.
+
+DNS changes:
+1. If a service should be LAN-only, it is usually safe to remove its ordinary Porkbun A record while keeping Pi-hole local DNS.
+2. Removing public DNS does not remove router port forwards or block direct IP access.
+3. Do not casually remove public DNS for auth.franksdank.cloud; external login flows may need it.
+4. Be cautious with gitea.franksdank.cloud because external git smart-HTTP use may depend on public DNS.
+
+Auth wrapping:
+- Protected via Authentik forward_auth: homepage, SearXNG, Hermes, Prometheus, RomM, Copyparty.
+- Usually bypass forward_auth because they have native auth: Grafana, Gitea, OpenWebUI, Jellyfin, *arr, Pi-hole.
+
+Verification pitfalls:
+- Curl or bot-like clients may get Anubis JS challenge page with HTTP 200. That does not prove the app or outpost succeeded.
+- Authentik outpost provider matching depends on exact Host header. Missing/wrong Host in CLI probes can produce misleading 404s.
+- Verify end-to-end login in a real browser after solving Anubis. Use logs to debug outpost 404/provider host-match failures.
diff --git a/agent/memories/--home-dank-Projects-serverAppConfigs--/skills/frank-default-large-ops/SKILL.md b/agent/memories/--home-dank-Projects-serverAppConfigs--/skills/frank-default-large-ops/SKILL.md
new file mode 100644
index 0000000..0763c2f
--- /dev/null
+++ b/agent/memories/--home-dank-Projects-serverAppConfigs--/skills/frank-default-large-ops/SKILL.md
@@ -0,0 +1,36 @@
+# frank-default-large operations
+
+Use this when changing or troubleshooting the large model service on Dankclaw.
+
+Current durable model/runtime:
+- Host: Dankclaw, Strix Halo gfx1151 unified memory.
+- Alias: frank-default-large.
+- Endpoint: http://192.168.0.143:7770/v1, health on port 7770.
+- Repo files: scripts/serve-frank-large.sh and scripts/frank-default-large.env under /home/dank/Projects/serverAppConfigs.
+- Runtime: /home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server.
+- Model: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF.
+- Local GGUF: /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf.
+
+Hard constraints:
+- Keep --fit off unless the qwen4exp hyper-connection auto-fit segfault is proven fixed.
+- Keep PLE table mmap/page-cache streamed. Do not add --no-mmap and do not force the PLE table into GPU memory.
+- Avoid bare env vars TEMP, HOST, and GPU_MAX_HEAP_SIZE. HIP/ROCm can read them unexpectedly.
+- Do not enable MTP/speculative decoding casually. Prior MTP heads failed because blk.0.hc_attn_norm.weight was missing. Latest durable state has SPEC_TYPE=none.
+- Stock llama.cpp cannot load this model; qwen4exp architecture support and ROCmFP4 tensor support must coexist in the same runtime tree.
+- If flash-attn graph build fails on QSA mask, port scripts/patches/qwen4exp-qsa-f32-mask.patch into the active runtime tree and rebuild.
+
+Before edits:
+1. cd /home/dank/Projects/serverAppConfigs
+2. git status --short
+3. Inspect current scripts/frank-default-large.env and scripts/serve-frank-large.sh; do not rely on old memory for exact tunables.
+
+After edits:
+1. Run repo tests appropriate to the change.
+2. Deploy/restart using the current repo workflow.
+3. Verify health: curl -s http://192.168.0.143:7770/health or via localhost on Dankclaw.
+4. Watch logs during model load; 113+ GiB mmap/model load can take significant time.
+
+Benchmarking notes:
+- Separate prefill/TTFT from decode tok/s.
+- For decode sweeps, an 8K prompt and 256 generated tokens is sufficient unless testing long-run thermal/stability.
+- Prior decode sweep found THREAD_BATCH=16 best, but current repo env always wins.
diff --git a/agent/memories/--home-dank-Projects-serverAppConfigs--/skills/hermes-dankclaw-ops/SKILL.md b/agent/memories/--home-dank-Projects-serverAppConfigs--/skills/hermes-dankclaw-ops/SKILL.md
new file mode 100644
index 0000000..a91dae5
--- /dev/null
+++ b/agent/memories/--home-dank-Projects-serverAppConfigs--/skills/hermes-dankclaw-ops/SKILL.md
@@ -0,0 +1,31 @@
+# Hermes on Dankclaw operations
+
+Use this when Hermes, Hermes WebUI, dashboard, Signal, or Gemma/FLM auxiliary routing is involved.
+
+Facts:
+- Host: Dankclaw 192.168.0.143.
+- Live data: /home/dank/hermes-data mounted as /data.
+- HERMES_DATA_DIR and HERMES_HOME must point to /data.
+- Repo config.yaml.example is sanitized only; preserve live secrets and generated state.
+- WebUI has listened on port 8787. Dashboard uses 9119. Chrome DevTools Protocol binds to 127.0.0.1:9222.
+- Signal HTTP adapter expects signal-cli-daemon on 127.0.0.1:8080.
+
+Health/status check:
+- ssh dankclaw 'for u in hermes hermes-webui hermes-dashboard signal-cli-daemon; do printf "%-22s %s %s\n" "$u" "$(systemctl --user is-active $u.service 2>/dev/null)" "$(systemctl --user is-enabled $u.service 2>/dev/null)"; done; podman ps -a --format "{{.Names}} {{.Status}} nets=[{{.Networks}}]" | grep -Ei "hermes|signal|chrome"; ss -lntp 2>/dev/null | grep -wE "8080|8787|9119|9222" | tr -s " "'
+
+If WebUI is up but Signal/Hermes transport errors appear:
+1. Check signal-cli-daemon first:
+   - ssh dankclaw 'systemctl --user status signal-cli-daemon.service --no-pager; journalctl --user -u signal-cli-daemon.service -n 120 --no-pager'
+2. Only if appropriate, start/enable:
+   - ssh dankclaw 'systemctl --user enable --now signal-cli-daemon.service'
+3. Verify:
+   - ssh dankclaw 'systemctl --user is-active signal-cli-daemon.service; ss -lntp | grep -w 8080'
+
+Config edit caution:
+- Do not replace /home/dank/hermes-data/config.yaml wholesale with repo example.
+- Live /data/config.yaml ownership may require sudo-safe edit flow.
+- After live config edits, restart the appropriate Hermes unit/container and verify health.
+
+Model routing memory:
+- Main model: frank-default-large via dankclaw-large at http://192.168.0.143:7770/v1.
+- Vision auxiliary: gemma4-it:e4b at http://192.168.0.143:7771/v1 via provider/custom endpoint.
diff --git a/agent/memories/--home-dank-Projects-serverAppConfigs--/skills/homelab-service-triage/SKILL.md b/agent/memories/--home-dank-Projects-serverAppConfigs--/skills/homelab-service-triage/SKILL.md
new file mode 100644
index 0000000..56e9c7e
--- /dev/null
+++ b/agent/memories/--home-dank-Projects-serverAppConfigs--/skills/homelab-service-triage/SKILL.md
@@ -0,0 +1,30 @@
+# Homelab service triage
+
+Use this when checking service state across Dankclaw/Dankserver from Dankware or another controller.
+
+1. Start from the repo and trust current repo state over memory:
+   - cd /home/dank/Projects/serverAppConfigs
+   - git status --short
+   - find . -path ./.git -prune -o -name '*.container' -print | sort
+
+2. Identify host and unit style:
+   - Dankclaw: 192.168.0.143
+   - Dankserver: 192.168.0.235
+   - Most services are rootless Quadlet user units: systemctl --user  .service
+   - Some model services may be system units or custom scripts; inspect repo before acting.
+
+3. Inventory a host:
+   - ssh HOST 'systemctl --user list-units --type=service; podman ps -a --format "{{.Names}} {{.Status}} nets=[{{.Networks}}]"'
+   - For a suspected unit: ssh HOST 'systemctl --user status UNIT.service --no-pager; journalctl --user -u UNIT.service -n 120 --no-pager'
+
+4. For port/listener issues:
+   - ssh HOST 'ss -lntp 2>/dev/null | tr -s " "'
+   - Match published ports to Quadlet PublishPort and podman ps networks.
+
+5. For deploy issues:
+   - Review scripts/deploy.sh behavior in current repo.
+   - Confirm target checkout is clean or intentionally stashed by deploy workflow.
+   - Do not hand-edit generated systemd output; fix Quadlet/source files.
+   - After Quadlet changes run daemon reload via deployer or systemctl --user daemon-reload, then inspect generated unit with systemctl --user cat.
+
+6. Do not commit secrets. Machine-local env files are the source for Caddy/Porkbun, app passwords, API keys, OAuth creds, phone numbers, sessions, and live generated state.
diff --git a/agent/memories/--home-dank-Projects-serverAppConfigs--/skills/quadlet-media-migration/SKILL.md b/agent/memories/--home-dank-Projects-serverAppConfigs--/skills/quadlet-media-migration/SKILL.md
new file mode 100644
index 0000000..7f5650d
--- /dev/null
+++ b/agent/memories/--home-dank-Projects-serverAppConfigs--/skills/quadlet-media-migration/SKILL.md
@@ -0,0 +1,31 @@
+# Quadlet media migration notes
+
+Use this when migrating or repairing media services on Dankserver or standardizing remaining compose services.
+
+General rules:
+- Prefer Quadlet .container units for boot persistence.
+- Use bind mounts only for durable state.
+- Linux data root standard: /home/dank/Containers/, unless a service intentionally uses an in-place legacy root.
+- Use UserNS=host in Quadlet when host UID/GID behavior matters; do not use compose-style UsernsMode=.
+- Stop/mask old native units or disable old snaps only after verifying rollback path.
+
+Plex/Tautulli known-good design:
+- Plex image: lscr.io/linuxserver/plex:1.43.3.10896-cb3ebc72d-ls322.
+- Tautulli image: lscr.io/linuxserver/tautulli:v2.18.1-ls242.
+- Network: dankserver_appnet, aliases plex and tautulli.
+- Ports: Plex 32400:32400, Tautulli 8181:8181.
+- Plex volumes: /var/lib/plexmediaserver:/config rw and /mnt/entropy:/mnt/entropy:ro.
+- Tautulli volume: /home/dank/Containers/tautulli:/config.
+- Plex rootless mapping: UserNS=host, PUID=0, PGID=0, data root chowned to 1000:1000. Rollback chown is 997:984.
+- Tautulli: PUID=1000, PGID=1000.
+- Tautulli seed: PMS_Web_Host=plex, PMS_Web_Port=32400, PMS_Secure=0.
+- Plex health gate should be generous, around 600 seconds, because large DB migrations can cold-load slowly.
+
+Copyparty/Jellyfin caution:
+- User directive after verification: stop touching Jellyfin/Copyparty unless explicitly asked.
+- Jellyfin mount verified: /mnt/entropy:/data:ro. DB table is BaseItems.
+- Copyparty must be read-only for broad web/LAN access. Bare A in copyparty means everyone full access. Read-only permission shape is r for anonymous and ra,dank for admin browsing, without write/move/delete perms.
+
+Before decommission:
+- Do not apt purge plexmediaserver or snap remove tautulli without explicit sign-off.
+- Keep rollback steps documented before deleting old service data or packages.
diff --git a/agent/memories/--home-dank-Projects-serverAppConfigs--/skills/rocmfp4-strix-benchmark-orchestration/SKILL.md b/agent/memories/--home-dank-Projects-serverAppConfigs--/skills/rocmfp4-strix-benchmark-orchestration/SKILL.md
deleted file mode 100644
index 9e6dea2..0000000
--- a/agent/memories/--home-dank-Projects-serverAppConfigs--/skills/rocmfp4-strix-benchmark-orchestration/SKILL.md
+++ /dev/null
@@ -1 +0,0 @@
-Orchestrate llama.cpp benchmarking on Dankclaw (gfx1151) using frank-default-large. Follow strict env constraints, health-check polling patterns, and resource isolation. SSH from controller, use frank-default-small as driver, poll health endpoints with 5+ checks at ~5s intervals, and apply env file tunings before starting throughput tests.