chore(skills): promote runbook skills to managed-skills; drop superseded SKILL.md
- 31 new managed skills (anubis/arr/authentik/hermes/freetoken/podman/llm-swap runbooks etc.) now tracked. - Removed superseded skills: omp_harness_config, peon_ping_integration (replaced by user-authored peon-ping-* managed skills), rocmfp4-strix-benchmark-orchestration. - New memory-root skills: niri_compositor_ops, omp_git_workflow, omp_model_config, quadlet-media-migration, homelab-service-triage, hermes-dankclaw-ops, frank-default-large-ops, caddy-auth-dns-playbook.
This commit is contained in:
Frank
@@ -0,0 +1,25 @@
|
||||
---
|
||||
name: anubis-guard-chain-flip
|
||||
description: "Runbook for moving a Caddy-fronted service behind the Anubis + Authentik embedded-outpost guard chain on Dankclaw (provider-first ordering, validate gate, verification probes)."
|
||||
---
|
||||
|
||||
# Anubis + Authentik guard-chain flip (Dankclaw)
|
||||
|
||||
Move a public vhost from `reverse_proxy <app>:<port>` onto: client → TLS edge → Anubis challenge (:8923) → Authentik embedded outpost decision → return to internal `http://:8017` listener → Host-matched handler → real upstream.
|
||||
|
||||
## Order (never reorder)
|
||||
1. **Provider first.** Create the outpost ProxyProvider before touching routes, or authed traffic hits the outpost error page instead of the app:
|
||||
- Write a Python snippet to a local /tmp file that imports `authentik.providers.proxy.models.ProxyProvider`, constructs it with `name`, `external_host="https://<vhost>/"` (exact, trailing slash), `internal_host="http://caddy:8017"`, sets `p.pk = None`, then `p.save()`.
|
||||
- Pipe raw: `ssh -o BatchMode=yes dankclaw 'podman exec -i authentik-server ak shell' < /tmp/x.py`. Heredoc quoting through ssh SyntaxErrors; the raw-stdin pattern is the proven one (ak shell does `exec(sys.stdin.read())`).
|
||||
2. **Route flip in ONE commit/deploy**: vhost becomes `reverse_proxy anubis:8923 { header_up X-Real-Ip {remote_host} }`; inside the `http://:8017` listener add `@<svc> host <vhost>` + `handle @<svc> { reverse_proxy <upstream> }`.
|
||||
3. Deploy via the repo deploy flow (commit first; deploy pushes main itself).
|
||||
|
||||
## Verification (all three)
|
||||
- Unauthenticated probe from client: `curl -s https://<vhost>/` → HTTP 200 challenge page, grep body for challenge markers (`challenge|not a bot|anubis`). A 502 here instead of the challenge = chain mis-wired.
|
||||
- Post-auth routing proof: grep the LIVE config inside the container — `podman exec caddy sh -c "grep -n -A3 <handler> /etc/caddy/Caddyfile"` — after the deploy prints its reload line; confirm the reload actually ran, don't trust exit 0.
|
||||
- Outpost reachability: `podman exec authentik-server getent hosts caddy` resolves the compose DNS alias (10.89.1.x) that internal_host dials.
|
||||
|
||||
## Pitfalls
|
||||
- Probing `http://localhost:8017/` from the host returns HTTP 000 (connection refused): the post-auth listener lives in the caddy container's netns and is not bound on host loopback. Probe inside the netns or grep the config instead.
|
||||
- Non-browser clients cannot solve the JS challenge. Before flipping, audit service-to-service traffic: ingestion/push paths (e.g. Alloy → `loki:3100`) must target internal addresses; any backend datasource URL pointing at a public hostname of a guarded service breaks the moment you flip.
|
||||
- Exempt-by-construction categories that must stay unguarded: the IdP itself (`auth.` — its login flow can't sit behind a challenge), git smart-HTTP endpoints (Gitea git clients), services with their own complete auth. Document the exemption in the vhost comment.
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
name: arr-auth-posture-probe
|
||||
description: "Verify a self-hosted *arr-style service's real auth posture by behavioral API probes instead of config-flag inference; includes the sed-masks-a-miss pitfall and wrong-host probe pitfall"
|
||||
---
|
||||
|
||||
# Arr-auth posture probe
|
||||
|
||||
Ground truth for "is this arr/gate open?" comes from behavior, not config flags. `AuthenticationMethod=None` + `AuthenticationRequired=Enabled` in `config.xml` is ambiguous on paper; on the wire it means a login screen with **zero users** — first-account-wins.
|
||||
|
||||
## Recipe
|
||||
1. Confirm which host runs the service before probing (`podman ps` on the target, not the box you ssh'd into first — arrs live on Dankserver, chat/searxng on Dankclaw).
|
||||
2. Probe an API endpoint that bypasses the SPA shell, locally and through the vhost:
|
||||
`curl -s -o /dev/null -w "%{http_code}" http://<host>:<port>/api/v3/system/status`
|
||||
- SPA roots (`/`) always serve a 200 shell — never conclude "open" or "gated" from `/`.
|
||||
- 401 = gated; 200 + JSON = API fully open (anyone on the net can drive it with curl).
|
||||
3. For config truth, dump the whole file (`podman exec <svc> cat /config/config.xml`), not a piped `grep | sed -n 2p`: **`sed -n Np` exits 0 on zero matches**, so the pipeline hides a miss and you conclude "empty field" when the grep itself failed.
|
||||
4. API keys from config.xml are local-only secrets; they stop being secret the moment a vhost serves the API unauthenticated.
|
||||
|
||||
## Decision framing for the user
|
||||
When posture = gated-with-zero-users, present it as a decision menu (None behind the front-door wall vs External/authentikarr), not a silent change — first-account-wins means whoever logs in first owns the instance.
|
||||
@@ -0,0 +1,30 @@
|
||||
---
|
||||
name: arr-stack-api-config
|
||||
description: "Configure/download-client and indexer wiring for the *arr stack on Dankserver via live API calls — schema-first, test-before-enable; covers Sonarr/Radarr v3 vs Lidarr/Readarr v1 API split and Prowlarr v2 Applications/sync model."
|
||||
---
|
||||
|
||||
# Arr-stack API config (Dankserver)
|
||||
|
||||
## API version split
|
||||
- Sonarr/Radarr: `/api/v3/...`; Lidarr/Readarr/Prowlarr: `/api/v1/...`.
|
||||
- `PUT /downloadclient` and `PUT /indexer/{id}` take a SINGLE resource object (GET it, patch `settings`, PUT whole object back). An array body fails with `Path: $ | BytePositionInLine: 1`.
|
||||
- API keys live in each container's config: `podman exec <svc> grep -o "<ApiKey>[^<]*" /config/config.xml | cut -d">" -f2`.
|
||||
|
||||
## Schema-first probing (validators are free documentation)
|
||||
Send the FULL schema template object with fields filled; v1 validators reject partials:
|
||||
- `'Fields' must not be empty` -> send full schema copy, not `fields:[]`.
|
||||
- `'Base Url' must not be empty` when keys were camelCase -> PascalCase required (`BaseUrl`, `ApiKey`).
|
||||
- `targetType null` -> configContract not registered in that build (this Prowlarr build has no Prowlarr-plugin indexer; integration is Torznab-family only).
|
||||
- `testingPackageId` in a PUT body downgrades validation failures to warnings.
|
||||
|
||||
## Prowlarr v2 model (inverted from v1)
|
||||
Indexers live ON THE ARR SIDE; Prowlarr syncs indexer configs via `ApplicationSync` commands (`POST /api/v1/command {"name":"ApplicationSync","body":{"applicationId":N}}`). Applications settings keys must be PascalCase or sync throws NullConfig. Torznab feed paths can answer with the SPA HTML shell when absent — verify by byte count (HTML shell ~1946 bytes, 1 ms) vs XML, not just status code.
|
||||
|
||||
## DB as arbiter
|
||||
When endpoint shapes are opaque, stop the container and edit `/opt/prowlarr/config/prowlarr.db` with `sudo python3` sqlite directly, then start. Indexers table columns are Id/Name/Implementation/Settings/ConfigContract/Enable/Priority/Added — the enable flag column is `Enable` (not EnableRss).
|
||||
|
||||
## Verification rules
|
||||
- `POST /downloadclient` returns the COLLECTION, not the created object — verify via GET.
|
||||
- Test endpoints return verdicts in the BODY at HTTP 200 (`{}` = pass, list = fail) — judge from bodies, not status codes.
|
||||
- `syncLevel` is enum ApplicationSyncLevel — omit or use int 2; string rejected.
|
||||
- Command POSTs (MoviesSearch etc.) run server-side and respond only when the work finishes: curl `-m` timeouts (HTTP 000) during search/grab are NOT failures; poll queue/history afterward instead of re-firing. Warm small commands (RefreshMonitoredDownloads) answer in ~20 ms — a hang means live work in progress, not a dead API.
|
||||
@@ -0,0 +1,29 @@
|
||||
---
|
||||
name: arr-v1-api-validator-peeling
|
||||
description: "When driving Sonarr/Radarr (v3) vs Lidarr/Readarr/Prowlarr (v1 .NET) APIs over HTTP: probe endpoints by status code, peel validators one live call at a time, and treat config.xml on disk as truth — v1 rejects literal enum strings that break AuthorizationMiddleware."
|
||||
---
|
||||
|
||||
# *arr v1 API validator peeling
|
||||
|
||||
Use when creating root folders, profiles, or settings via the *arr HTTP APIs, or when a v1 (.NET) arr 500-sprawls after a config edit/restart.
|
||||
|
||||
## API version split
|
||||
- Sonarr/Radarr: `/api/v3/...`. Lidarr/Readarr/Prowlarr: `/api/v1/...` (Prowlarr port 9696, NOT 8090 — that's Jackett).
|
||||
- Probe endpoints by status code first: **empty body = endpoint absent**, not a failure. v1 has no `/api/v1/profile` endpoint at all.
|
||||
|
||||
## Peel validators one live call at a time
|
||||
POST a minimal body and let the validator error name the next required field, instead of guessing payload shapes:
|
||||
- `DefaultMetadataProfileId must be >0` → that field is required; fetch real ids from `GET /api/v1/metadataprofile` (returns rows like `(1,'Standard')`).
|
||||
- `'Name' must not be empty` → include `name`.
|
||||
- `Quality Profile does not exist` → the id you sent doesn't exist; list real ones first (`GET /api/v1/qualitydefinition` returns the quality catalog, ids ≠ profiles).
|
||||
Then POST the full body with **existing** ids: `{"name":"music","path":"/music","defaultQualityProfileId":1,"defaultMetadataProfileId":1}`.
|
||||
|
||||
## Auth-config pitfall (v1 .NET apps)
|
||||
- `<AuthenticationMethod>None</AuthenticationMethod>` + `<AuthenticationRequired>Enabled</AuthenticationRequired>` = self-contradictory: UI Save permanently disabled, and any save throws `DryIoc.ScopedItemException` in `AuthorizationMiddleware` → 500 on every page after restart.
|
||||
- The valid no-auth representation is **absence of the `<AuthenticationRequired>` key**, not the literal string `Disabled`. Fix = delete the key from `/config/config.xml` (`sed -i "/<AuthenticationRequired>/d"`), restart container.
|
||||
- Tell-tale: one v1 app boots clean while siblings 500-sprawl → diff their config.xml auth blocks; the healthy one shows the correct shape.
|
||||
|
||||
## General rules
|
||||
- Container `State=running` ≠ app serving; read `podman logs --tail 12 <c>` before theorizing.
|
||||
- When a UI gate blocks saving, go under it: these apps keep plaintext XML config on disk (`/config/config.xml`) — read/edit it directly via `podman exec`.
|
||||
- API keys live in each container's `/config/config.xml` `<ApiKey>`; grep them live rather than trusting docs.
|
||||
@@ -0,0 +1,38 @@
|
||||
---
|
||||
name: authentik-claim-cache-purge
|
||||
description: "Fix persistent 403s at Caddy RBAC header gates after authentik group changes: purge the embedded outpost's proxysession claim cache and verify silent re-auth"
|
||||
---
|
||||
|
||||
# Authentik outpost claim-cache purge
|
||||
|
||||
Use when a user gets persistent 403s at a Caddy `header_regexp` RBAC gate (e.g. `/admin`) even though their authentik group membership is correct, and re-login did not help.
|
||||
|
||||
## Mechanism (authentik 2026.8, this fleet)
|
||||
- The embedded outpost does NOT re-evaluate groups per request. Each completed authorize writes one row to `authentik_providers_proxy_proxysession`; `session_data` JSON holds the claims (incl. `groups`) served for that browser session until the row expires (observed TTLs 1–24 h per row).
|
||||
- A session minted before the group add keeps serving the old group list → gate 403s forever until expiry.
|
||||
- The 403 happens at Caddy and generates ZERO authentik events — silence in `authentik_events_event` is expected.
|
||||
- Re-login does not fix it: the browser's outpost cookie re-maps to the same stale row.
|
||||
- `proxysession.user_id` is null; map rows to users via `session_data->'claims'->>'preferred_username'`.
|
||||
|
||||
## Diagnosis (run on dankclaw)
|
||||
```bash
|
||||
podman exec authentik-postgres psql -U authentik -d authentik -At -c "SELECT session_data->'claims'->>'preferred_username' AS user, session_data->'claims'->>'groups' AS cached_groups, expires FROM authentik_providers_proxy_proxysession ORDER BY expires;"
|
||||
```
|
||||
Stale row = `cached_groups` missing the new group while repo/Caddy gate expects it.
|
||||
|
||||
## Fix
|
||||
```bash
|
||||
podman exec authentik-postgres psql -U authentik -d authentik -At -c "WITH d AS (DELETE FROM authentik_providers_proxy_proxysession WHERE session_data->'claims'->>'preferred_username' IN ('<user>') RETURNING 1) SELECT 'deleted='||count(*) FROM d;"
|
||||
```
|
||||
- The user's authentik django login session is untouched → next visit re-authorizes SILENTLY (no password prompt) with fresh claims.
|
||||
- NEVER rotate `AUTHENTIK_SECRET_KEY` for this — it invalidates every outpost session fleet-wide.
|
||||
|
||||
## Verification without the user's password
|
||||
Simulate their browser: delete a TEST user's proxysession rows (e.g. `qa-e2e-admin`), keep its e2e storage state on disk, then run the admin spec bypassing setup (setup ALWAYS fresh-logins, which would mask the path):
|
||||
```bash
|
||||
cd tests/e2e && npx playwright test --project=admin --no-deps
|
||||
```
|
||||
Passing with no login prompt = silent re-auth proven. `--no-deps` is required because project dependencies would rerun setup.
|
||||
|
||||
## Doc
|
||||
Mechanism + SQL live in `serverAppConfigs-docs/Infrastructure/Authentik.md`, section "Outpost claim cache (proxysession)" (commit ea0bba3).
|
||||
@@ -0,0 +1,46 @@
|
||||
---
|
||||
name: authentik-flow-ui-automation
|
||||
description: Drive the authentik two-stage login through real browser UI for E2E tests on Anubis+outpost-guarded routes; hardened against the Lit stage re-render race and Anubis PoW budgets
|
||||
---
|
||||
|
||||
Drive authentik's two-stage login (identification → password) through a real browser UI for E2E tests on routes guarded by the Anubis + Authentik chain (Caddy → Anubis :8923 → outpost :9000). Proven in `tests/e2e/lib/login.ts` (`akLogin`) against authentik 2026.x.
|
||||
|
||||
## Hard rules
|
||||
|
||||
1. **Fresh context per login.** Never reuse a context or page across logins, and never retry a failed login inside the same context — a half-finished flow poisons authentik's `pending_user` stage and the next login hangs on a "Continue as <user>" interstitial that never completes. Each login = new `browser.newContext()`.
|
||||
2. **Real Chrome UA.** Anubis denies obvious headless UAs; set `userAgent` to a real Chrome UA in `use:` (config-level) or `launchOptions`. Headless chromium + REAL_UA solves the PoW fine.
|
||||
3. **Shadow-DOM locators.** authentik stages are Lit web components; locate inputs with `page.locator('input[name=username]')` etc. (Playwright pierces shadow roots) — never `page.$`.
|
||||
|
||||
## The Lit re-render race (the big one)
|
||||
|
||||
The flow executor re-renders a stage right after mount. A programmatic `fill()` can succeed and then be WIPED by the re-render; the submit then POSTs an empty form ("This field is required") and `waitForURL` times out while the page still shows the password stage with an empty field.
|
||||
|
||||
Hardened sequence per stage:
|
||||
|
||||
```ts
|
||||
for (let attempt = 1; attempt <= 3; attempt++) {
|
||||
await page.fill('input[name=password]', pass);
|
||||
// VERIFY the value survived the re-render before submitting
|
||||
if ((await page.inputValue('input[name=password]')) !== pass) continue;
|
||||
await page.keyboard.press('Enter'); // native form path; avoids button re-render races
|
||||
try {
|
||||
await page.waitForURL(u => !u.pathname.includes('/if/flow/'), { timeout: 30_000 });
|
||||
return; // past this stage
|
||||
} catch { /* maybe wiped; retry */ }
|
||||
}
|
||||
throw new Error(`akLogin: stage never advanced after 3 attempts`);
|
||||
```
|
||||
|
||||
- Submit with **Enter**, not a button click — the button can be re-created between snapshot and click.
|
||||
- Verify-after-fill is the load-bearing step; without it you silently submit empty forms.
|
||||
- The identification stage has the same race; apply the same fill-verify-Enter there.
|
||||
|
||||
## Anubis PoW budgets
|
||||
|
||||
The first guarded-vhost hit in a fresh context runs an Anubis proof-of-work; on a cold/loaded box it can outlive default assertion budgets. Give title/URL asserts after `goto` 60 s budgets. With `COOKIE_DOMAIN` set fleet-wide (Anubis ≥ v1.27), every later guarded vhost in the same context skips the challenge (~0.5 s per vhost in the crawl specs).
|
||||
|
||||
## See also
|
||||
|
||||
- `serverAppConfigs-docs/Architecture/E2E Suite.md` — project layout, credentials model (`tests/e2e/.env.local`, git-ignored), run commands, gotcha list.
|
||||
- `serverAppConfigs-docs/Infrastructure/Anubis.md` — chain behavior, v1.27 gotchas (X-Real-Ip mandatory, DENY = HTTP 200 body).
|
||||
- `arr-auth-posture-probe` for non-browser API-posture checks of guarded routes.
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
name: authentik-outpost-verification
|
||||
description: "Verify authentik embedded-outpost and proxy-provider state on a live host by probing from inside the container network namespace; covers meta-header routing, rootless podman observation artifacts, and 2026.x schema quirks (renamed columns, ak shell swallowing piped stdin)."
|
||||
---
|
||||
|
||||
# Authentik outpost/provider verification (live host)
|
||||
|
||||
## When to use
|
||||
Guarded apps behind Anubis+authentik return an error page after authentication, or you need ground truth on outpost/provider state on a host running authentik 2026.x with podman quadlet units.
|
||||
|
||||
## Core insight: probe from inside the network namespace
|
||||
- Rootless podman gives each container its own netns; **host `ss -ltn` cannot see listeners inside other containers' namespaces**. A missing listener on the host is an observation artifact, not a fault — verify by dialing *from inside* the namespace.
|
||||
- Working probe (distroless images have no shell, so `podman exec sh` fails):
|
||||
```
|
||||
podman run --rm --network container:<container> --entrypoint /bin/busybox docker.io/library/busybox:latest \
|
||||
wget -q -T 4 -t 1 -O /dev/null -S "<url>" 2>&1 | head -8
|
||||
```
|
||||
- Quote URLs over ssh: the remote shell is zsh and bare `?` in query strings triggers globbing (`no matches found`).
|
||||
|
||||
## Meta-header routing truth
|
||||
- The outpost routes by **meta-header middleware**, not URL-path lookup. A request to `/outpost.goauthentik.io/login` **without** `X-authentik-meta-outpost/-provider/-app` headers is a path that genuinely does not exist → a 404 there is *correct behavior*, not a fault.
|
||||
- Probe the auth hop exactly as Caddy sends it: full meta-header set (`X-authentik-meta-outpost: <uuid>`, `X-authentik-meta-provider: <pk>`, `X-authentik-meta-app: name=<app>`). If a full-meta-header request still 404s, the outpost's URL space is not mounted in the answering process — that is the real failure point, and it means the transport is aimed at a URL space that doesn't exist there (embedded outposts mount their URL space at `authentik_host`; check the outpost row's `_config.authentik_host` before theorizing).
|
||||
- Boot transience: immediately after a restart the same probe can return 503 (configure task pending); poll until the answer changes before concluding.
|
||||
|
||||
## Schema quirks in 2026.x
|
||||
- `ak shell` is interactive-only; piped stdin is swallowed (prints import banner, discards output). For live DB truth use:
|
||||
```
|
||||
podman exec authentik-postgres psql -U authentik -d authentik -x -c "select * from authentik_outposts_outpost"
|
||||
```
|
||||
- Columns are renamed vs older versions: `Provider.objects.filter(type=...)` raises FieldError (type moved to subclass tables); the outpost `enabled` column may not exist under that name; provider pk is `oauth2provider_ptr_id`. **Query with `select *` and stop guessing column names.**
|
||||
- `podman inspect --format "{{.Config.NetMode}}"` errors on this podman version — the field is `.HostConfig.NetworkMode`.
|
||||
|
||||
## Decision fork when the auth hop 404s
|
||||
A) point the guarded apps' auth hop at where auth actually lives (the vhost matching `authentik_host`), or B) flip the outpost to standalone (own container, own URL space, new quadlet unit + `AUTHENTIK_TOKEN`). This is an architecture decision for the user — present both with blast radius, don't pick silently.
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
name: compose-stack-bringup-groundtruth
|
||||
description: "Bring up a compose-defined container stack on a rootless podman host by verifying reality first (mounts, images, in-image interfaces) instead of trusting compose env/flag guesses; includes the copyparty ac-edition argv-interface recipe and boot-persistence audit."
|
||||
---
|
||||
|
||||
# Compose-stack bring-up with ground truth (rootless podman hosts)
|
||||
|
||||
Use when standing up (or debugging) a compose-defined service stack on a rootless
|
||||
podman host where the stack has never run, or answers with surprise statuses
|
||||
(SIGSEGV at boot, HTTP 500 on `/`, "manifest unknown").
|
||||
|
||||
## Procedure
|
||||
|
||||
1. **Audit reality before pressing buttons.** `podman ps -a` runs per-user —
|
||||
check every namespace (`sudo podman ps -a`) and native units
|
||||
(`systemctl list-units`) before declaring a service absent; a service may run
|
||||
as a native systemd unit or snap, not a container. Compare compose mount source
|
||||
paths against the real filesystem (`ls /mnt/...`): compose files often name
|
||||
phantom paths that never existed (e.g. `/mnt/media/*` vs the real LUKS array at
|
||||
`/mnt/entropy`). Fix compose through git first, deploy second.
|
||||
|
||||
2. **Pre-create bind-mount sources.** Rootless podman fails container creation
|
||||
with `statfs /opt/<svc>/...: no such file or directory` when a bind source is
|
||||
missing. `sudo mkdir -p` + `chown 1000:1000` every compose volume source
|
||||
before the first `podman-compose up -d`.
|
||||
|
||||
3. **Staged bring-up.** Infra first (`up -d mariadb redis`), poll healthchecks via
|
||||
`podman inspect --format "{{.State.Health.Status}}"`, then `up -d` the rest.
|
||||
|
||||
4. **Never guess image tags — query ground truth.** ghcr.io answers **403 on the
|
||||
bearer-token request for nonexistent/private repos** ("manifest unknown" / token
|
||||
len 0 = repo or tag does not exist, not an auth problem). Query the registry
|
||||
API (`ghcr.io/token?scope=repository:<ns>/<repo>:pull` then
|
||||
`/v2/<ns>/<repo>/tags/list`) before pinning a tag.
|
||||
|
||||
5. **Read the interface from inside the image** before guessing flags/envs:
|
||||
`podman exec <ctr> cat /z/cpp.sh`, `cat /z/initcfg`,
|
||||
`podman exec <ctr> python3 -m copyparty --help-accounts`. A service answering
|
||||
HTTP 500 with its own styled error page is *running* — it is misconfigured,
|
||||
not down. Copyparty ac edition ground truth: entrypoint
|
||||
`/bin/ash /z/cpp.sh -c /z/initcfg` then `exec python3 -m copyparty "$@"`;
|
||||
accounts/volumes are **argv flags** (`-a USER:PASS`, `-v SRC:DST:PERMS`), not
|
||||
`COPAY_*` envs (zero `COPAY_` references in the package). With no accounts and
|
||||
no volumes configured it serves cwd read/write for everyone — an
|
||||
unauthenticated world-writable share of the whole mount.
|
||||
|
||||
6. **Beware remote-shell quoting traps.** Remote zsh eats bare `===`/`?` tokens
|
||||
(glob qualifiers/globs); grep patterns with nested quotes over ssh can be
|
||||
quote-eaten into silent zero-match. A zero-match from a mangled pattern is not
|
||||
absence — re-verify the target exists before concluding it.
|
||||
|
||||
7. **Boot-persistence audit last.** On rootless podman hosts with
|
||||
`podman-user-generator`, containers started via podman-compose get generated
|
||||
`container-<name>.service` units at first boot (evidence: an existing unit for
|
||||
one container proves the generator works). Verify after first reboot:
|
||||
`systemctl --user is-enabled <name>.service`.
|
||||
|
||||
## Gotchas
|
||||
- jellyfinetype SIGSEGV at boot with `user: "1000:1000"` = unmapped uid inside the
|
||||
rootless userns (host uid 1000 == container uid 0); drop the user line and let it
|
||||
run as container-root (= host deploy user).
|
||||
- `up -d --force-recreate` applies compose changes that change command/env; probe with
|
||||
a poll loop (`for i in $(seq...)`) because fresh containers answer `000`/refused for
|
||||
the first seconds while indexing.
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
name: flm-npu-pack-registration
|
||||
description: "Runbook for registering a custom FastFlowLM NPU2 model pack on a tag that isn't in FLM's built-in catalog (Dankclaw :7771), including the LFS-pointer and vision-weight traps and the NPU hwctx contention rule."
|
||||
---
|
||||
|
||||
# FLM custom-tag pack registration (Dankclaw NPU endpoint)
|
||||
|
||||
Use when adding a non-catalog NPU2 pack (e.g. an extraction or fine-tuned model) to the FLM registry served by `frank-npu` on port 7771. Companion diagnosis skill: `flm-tag-load-failure-diagnosis`.
|
||||
|
||||
## Registry facts
|
||||
- User registry: `/home/dank/.config/flm/model_list.json`, shape `{"model_path": "models", "models": {"<tag>": {"<size>": {name, url, file_url, size, flm_min_version, files[], vlm, default_context_length, max_prefill_len, details{format:"NPU2", family, ...}, label[]}}}}}`.
|
||||
- Serve resolves tags through `FLM_CONFIG_PATH` at startup; `flm list` with the env var set lists custom tags (`⏬` = not downloaded).
|
||||
|
||||
## Procedure
|
||||
1. Back up the registry JSON first (`cp -a`).
|
||||
2. Insert the entry via a python json edit + atomic `os.replace`. Set `files` to the FULL HF file list — **including `vision_weight.q4nx`** even for text-only packs: the qwen3.5-family loader opens it unconditionally and a missing file makes the first request HANG silently (no error log).
|
||||
3. **Do not use `flm pull <tag>` for custom entries** — it throws `json.exception out_of_range key '<tag>:<size>' not found` while building the download list. Place files manually into `/home/dank/.config/flm/models/<PackName>/` with:
|
||||
`curl -sfL "https://huggingface.co/<repo>/resolve/main/<file>?download=true" -o <file>`
|
||||
Plain `raw/main/<file>` returns 135-byte LFS pointer stubs, not weights — verify byte counts against the HF tree API before trusting a download.
|
||||
4. Kernel packs: copy from the same-shape family pack (shape-match `config.json` on hidden_size/num_hidden_layers/head_dim first) into `/opt/fastflowlm/share/flm/xclbins/<PackName>/` via `sudo cp -a` (root-owned dir).
|
||||
5. Deploy/restart the serve unit (deploy path, not ad-hoc restarts) so the new process loads with all files present.
|
||||
6. Verify with a REAL completion request on 7771 (`model: "<tag>:<size>"`) — `/v1/models` alone proves registration, not loadability.
|
||||
|
||||
## Hard constraints
|
||||
- Only ONE `flm serve` process can hold the NPU hardware context. A second concurrent serve process fails every load with `DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22` (aie2 mgmt msg opcode 0x2 status 0x2000003). Never run scratch/verify instances while production holds the NPU — verify via the deployed service instead.
|
||||
- Tag switching happens INSIDE one serve process; first request after a model switch pays ~6–11 s cold-load cost (visible in `usage.load_duration`).
|
||||
- Over ssh, never `pkill -f "<pattern>"` with a pattern present in your own command line — it kills your own session; use bracket-guarded patterns (`[q]wen3.5:4b`) or exact PIDs.
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
name: flm-tag-load-failure-diagnosis
|
||||
description: "Diagnose FLM (FastFlowLM) NPU serve failures where a tag listed in /v1/models answers \"Failed to load model: No such file ...xclbin\" — missing kernel-pack symlink under FLM_XCLBIN_PATH is the root cause; includes per-request switch-cost behavior."
|
||||
---
|
||||
|
||||
# FLM tag load-failure diagnosis (Dankclaw sight+sound endpoint)
|
||||
|
||||
## Symptom
|
||||
`POST /v1/chat/completions` on the FLM serve endpoint answers **HTTP 200** with a flat error body `{"error":"Failed to load <tag> model!"}` and no `choices`. `/v1/completions` instead returns a nested OpenAI-shaped `server_error` (code 500) also under HTTP 200. A status-code-only client reads this as a successful empty completion — always check for the `error` key.
|
||||
|
||||
Two distinct causes, distinguish via `journalctl -u frank-default-sightsound.service`:
|
||||
|
||||
1. **Unknown tag** (not in the registry): immediate error, nothing loaded. Correct behavior; not a bug.
|
||||
2. **Registered tag whose kernel pack is missing**: FLM resolves the tag to its pack dir, downloads missing weight files on the fly (first request hangs ~90–110 s downloading), then fails with `Failed to load model: No such file '/home/dank/.config/flm/xclbins/<PackName>/<kernel>.xclbin'`. Subsequent attempts fail fast (~2 s).
|
||||
|
||||
## Root cause of case 2
|
||||
The serve script exports `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins` (user dir), so **every** kernel lookup resolves under the user dir — but that dir only contains symlinks for the packs someone linked. A tag listed in `/v1/models` is NOT proof it is loadable on this host.
|
||||
|
||||
## Fix
|
||||
One symlink per missing pack, run on Dankclaw:
|
||||
```
|
||||
ln -s /opt/fastflowlm/share/flm/xclbins/<PackName> /home/dank/.config/flm/xclbins/<PackName>
|
||||
```
|
||||
Verified 2026-08-30: `Qwen3.5-4B-NPU2` — after the symlink, first request loaded and answered in 7.4 s wall (cold load + generate).
|
||||
|
||||
## Switch-cost behavior (measured same day)
|
||||
- The `model` field routes per-request; no restart needed to switch between registry tags.
|
||||
- A request whose model differs from the resident one triggers a full reload ("Loading model:" in journal before prefill): measured 7–11 s wall for an 8-token completion on ~4B-class packs. Same-tag requests are fast; alternating tags pays the swap each time.
|
||||
- Single-lane NPU: concurrent requests queue FIFO behind the current lock holder (`NPU Locked!` → `NPU Lock Released!`).
|
||||
|
||||
## Probe recipe (repeatable)
|
||||
1. `curl -s http://192.168.0.143:52626/v1/models` — registry listing is authoritative for *registered* tags, not loadability.
|
||||
2. On a failed load, grep the journal for `Failed to load model` (use `--full`) — the quoted xclbin path names the missing pack dir exactly.
|
||||
3. Compare `ls /home/dank/.config/flm/xclbins/` (symlinks = installed) against `/opt/fastflowlm/share/flm/xclbins/` (all available packs); diff = fixable with symlinks.
|
||||
@@ -0,0 +1,57 @@
|
||||
---
|
||||
name: frankbook-qbittorrent-e2e-verify
|
||||
description: "Verify the FrankBook-Pro gluetun→qBittorrent grab pipeline end-to-end (egress gate inside gluetun, generic SID login, legal test torrent, category path check, data-safe cleanup) after any tunnel/qB/category change"
|
||||
---
|
||||
|
||||
# FrankBook-Pro qBittorrent E2E verification
|
||||
|
||||
Use after any change to the FrankBook-Pro torrent pipeline — WireGuard credential rotation, qB version change, gluetun reconfig, category re-seeding. Proves the full grab path (tunnel egress → qB download → category save path → clean removal). All commands run from Dankware via `ssh frankbook-pro`; inside the Mac shell export `PATH="/usr/local/bin:/opt/homebrew/bin:$PATH"` for `podman machine ssh`.
|
||||
|
||||
## 1. Egress gate — MUST run inside gluetun
|
||||
|
||||
```bash
|
||||
podman machine ssh 'podman exec gluetun wget -qO- https://ipinfo.io/json'
|
||||
```
|
||||
|
||||
- Require `org` = AS9009 (M247 — ProtonVPN Panama PA-12 config). Judge by org, never country alone.
|
||||
- NEVER run this one ssh-context out (e.g. bare `wget` on the Mac host): it silently returns the residential Comcast IP and reads as a hard-gate leak that does not exist. If a check shows a residential IP, first re-run inside the container before diagnosing.
|
||||
|
||||
## 2. Login — generic cookie parse
|
||||
|
||||
The QPW fetch is flaky (~1 in N returns empty); assert length before use. An empty-password run once produced a Forbidden loop that cost hours.
|
||||
|
||||
```bash
|
||||
QPW=$(podman machine ssh 'cat /home/core/Containers/qbittorrent/env' | grep QBIT_WEBUI_PASSWORD | cut -d= -f2)
|
||||
[ ${#QPW} -eq 20 ] || exit 1
|
||||
SID=$(curl -si --data-urlencode "username=admin" --data-urlencode "password=$QPW" \
|
||||
http://127.0.0.1:8080/api/v2/auth/login \
|
||||
| grep -oE '[A-Za-z_]*SID[A-Za-z0-9_]*=[^;[:space:]]*' | head -1)
|
||||
curl -b "$SID" ... # pass the WHOLE match
|
||||
```
|
||||
|
||||
- qB 4.x cookie name is plain `SID`; qB 5.x uses `QBT_SID_<port>`. Hard-coding either name breaks the other version's parse (sidlen=0 → 403 cascades).
|
||||
- Parse-bug 403s never reach qB as failed logins — no ban accrues. Check server-side truth in the log: `/Volumes/COMPLETE/qbittorrent-data/qBittorrent/config/qBittorrent/logs/qbittorrent.log` (`WebAPI login success/failure` + source IP). Repeated REAL failures accrue toward qB's 5-failures/3600s IP ban.
|
||||
|
||||
## 3. Add a legal test torrent
|
||||
|
||||
Blender open-movie torrents (~300 MB, e.g. Big Buck Bunny) via `POST /torrents/add` with the raw `.torrent` file (`-F "file=@..."`) and `category=Finished Torrents`. Never leave a real-looking grab behind.
|
||||
|
||||
## 4. Poll and verify
|
||||
|
||||
Poll `GET /torrents/info` (~20 s): state `downloading` → `moving` (INCOMPLETE→COMPLETE move) → `stalledUP`/`uploading` at progress=1. `save_path` must be `/Volumes/COMPLETE/Finished Torrents` (qB 4.x auto-creates unknown categories at DefaultSavePath; 5.x did NOT inherit category paths — one more reason for the 4.6.7 pin). Confirm on disk: `ls "/Volumes/COMPLETE/Finished Torrents/"`. Throughput ~17 MB/s is normal for the tunnel.
|
||||
|
||||
## 5. Cleanup — delete only your artifact
|
||||
|
||||
```bash
|
||||
curl -s -o /dev/null -w "%{http_code}" -b "$SID" \
|
||||
--data-urlencode "hashes=<literal-hash>" \
|
||||
--data-urlencode "deleteFiles=true" \
|
||||
http://127.0.0.1:8080/api/v2/torrents/delete # expect 200
|
||||
```
|
||||
|
||||
- Target the known hash literally — the user's real grabs share the directory and MUST be untouched. Verify: hash absent from `/torrents/info`, your dir gone, all other entries intact.
|
||||
|
||||
## Version-coupling constraints
|
||||
|
||||
- qB is pinned to 4.6.7 (readarr's EOL client rejects qB ≥ 5's `204` login at runtime). Do not unpin while readarr exists.
|
||||
- After ANY qB version flip, regenerate the `WebUI\Password_PBKDF2` line fresh (same password, 100k iterations) — a 5.x-written hash line makes 4.6.7 reject every login. Rewrite while qB is stopped.
|
||||
@@ -0,0 +1,55 @@
|
||||
---
|
||||
name: freetoken-build-dankclaw
|
||||
description: "Build FreeToken on Dankclaw (Ubuntu 24.04) with CUDA — installs nvidia-cuda-toolkit, sets CUDA_HOME=/usr, overrides nvcc/CUDA version mismatch"
|
||||
---
|
||||
|
||||
# FreeToken Build on Dankclaw
|
||||
|
||||
Dankclaw is Ubuntu 24.04 with ROCm already installed. FreeToken builds for **CUDA** (not ROCm) because the kernel package (`_pinned_tensor`) links against `cudart`.
|
||||
|
||||
## Prerequisites (install if missing)
|
||||
|
||||
```bash
|
||||
export PATH="/home/dank/.local/bin:$PATH"
|
||||
export CUDA_HOME=/usr
|
||||
export FREETOKEN_ALLOW_CUDA_MISMATCH=1
|
||||
|
||||
# CUDA toolkit (if not present):
|
||||
sudo apt-get install -y nvidia-cuda-toolkit
|
||||
# UV (if not present):
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
```
|
||||
|
||||
## Key environment variables
|
||||
|
||||
|Variable|Value|Why|
|
||||
|---|---|---|
|
||||
|`CUDA_HOME`|`/usr`|Ubuntu CUDA toolkit package places nvcc at `/usr/bin/nvcc` and libcudart at `/usr/lib/x86_64-linux-gnu/`|
|
||||
|`FREETOKEN_ALLOW_CUDA_MISMATCH=1`|Set|nvcc is 12.4 but torch 2.11.0+cu130 ships CUDA 13.0 — safe to override|
|
||||
|`PATH`|`/home/dank/.local/bin:$PATH`|uv installs here|
|
||||
|
||||
## Build procedure
|
||||
|
||||
```bash
|
||||
cd /home/dank
|
||||
git clone https://github.com/FlashML-org/FreeToken.git
|
||||
cd FreeToken
|
||||
uv venv && source .venv/bin/activate
|
||||
uv pip install -e ".[accel]"
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
ft --version # → freetoken version 0.1.2
|
||||
python3 -c "import torch; print(torch.__version__)" # → 2.11.0+cu130
|
||||
python3 -c "from freetoken.kernel import _pinned_tensor; print(_pinned_tensor.__file__)"
|
||||
python3 -c "import flashinfer; print(flashinfer.__version__)" # → 0.6.18
|
||||
python3 -c "import sgl_kernel; print(sgl_kernel.__version__)" # → 0.4.5
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- `torch.cuda.is_available()` may return `False` with "Can't initialize NVML" warning — this is the management library, not CUDA runtime. The kernels are built and linked correctly.
|
||||
- The NVIDIA driver modules may or may not be loaded; they don't affect the build.
|
||||
- ROCm is present on Dankclaw but FreeToken's kernel build specifically requires CUDA (links `cudart`).
|
||||
@@ -0,0 +1,39 @@
|
||||
---
|
||||
name: freetoken-dankclaw-build
|
||||
description: "Build FreeToken on Dankclaw (Dankware) — installs CUDA toolkit, sets CUDA_HOME, handles nvcc/CUDA version mismatch"
|
||||
---
|
||||
|
||||
## Build FreeToken on Dankclaw
|
||||
|
||||
### Prerequisites
|
||||
- CUDA toolkit: `sudo apt install nvidia-cuda-toolkit` (gives nvcc at /usr/bin, libcudart at /usr/lib/x86_64-linux-gnu/)
|
||||
- uv: installed at /home/dank/.local/bin/uv (or install via `curl -LsSf https://astral.sh/uv/install.sh | sh`)
|
||||
- CMake, git, Python >= 3.10 — all present
|
||||
|
||||
### Build steps
|
||||
```bash
|
||||
cd /home/dank
|
||||
git clone https://github.com/FlashML-org/FreeToken.git # or cd into existing
|
||||
cd FreeToken
|
||||
uv venv && source .venv/bin/activate
|
||||
export CUDA_HOME=/usr
|
||||
export FREETOKEN_ALLOW_CUDA_MISMATCH=1 # nvcc 12.4 + torch cu130 is safe
|
||||
uv pip install -e ".[accel]"
|
||||
```
|
||||
|
||||
### Verification
|
||||
```bash
|
||||
ft --version
|
||||
python3 -c "
|
||||
from freetoken.checkpoint import ftw
|
||||
import torch
|
||||
t = torch.randn(3,4,dtype=torch.float16)
|
||||
print(ftw._dtype_str(t.dtype), ftw._elsize(t.dtype))
|
||||
"
|
||||
```
|
||||
|
||||
### Key notes
|
||||
- FTW format is dtype-agnostic: FP16, BF16 (default), FP32, FP8 all work
|
||||
- GGUF support: only Gemma4GGUFForCausalLM
|
||||
- GGML types: F32/F16/BF16 (native views), Q4_0/Q6_K (dequant), Q8_0 (listed, no dequant)
|
||||
- The NVIDIA driver modules may not be loaded at build time (NVML warning) — not a problem for the build itself
|
||||
@@ -0,0 +1,36 @@
|
||||
---
|
||||
name: freetoken-hf-model-swap
|
||||
description: "Swap the FreeToken model behind frank-default-small on Dankserver: preflight the HF quant format against the loader matrix BEFORE downloading, then download, repoint MODEL_DIR, deploy, and verify with a REAL completion (health 200 is not enough)."
|
||||
---
|
||||
|
||||
# FreeToken HF Model Swap (frank-default-small, Dankserver)
|
||||
|
||||
Swap the model behind `frank-default-small` (FreeToken v0.1.2, Dankserver, RTX 3090) by downloading a HF safetensors checkpoint into a model dir, repointing `MODEL_DIR` in `scripts/frank-default-small.env`, and deploying. Verify with a REAL completion — health 200 is NOT enough.
|
||||
|
||||
## Preflight — verify quant format BEFORE downloading
|
||||
|
||||
FreeToken v0.1.2 loads ONLY modelopt-style quant configs for `qwen3_5_moe`. Every compressed-tensors FP8 checkpoint fails (live-verified 2026-09-01: two candidates, 72 GB downloaded, both unloadable). Check before any multi-GB download:
|
||||
|
||||
```bash
|
||||
curl -sL https://huggingface.co/<org>/<repo>/raw/main/config.json | \
|
||||
python3 -c "import json,sys; c=json.load(sys.stdin); q=c.get('quantization_config',{}); print(q.get('quant_method'), q.get('weight_block_size'), list((q.get('weight_scale') or ['?','?'])) if isinstance(q.get('weight_scale'),list) else q.get('weight_scale_inv') is not None)"
|
||||
```
|
||||
|
||||
Decision matrix:
|
||||
|
||||
- `quant_method: "fp8"` + `weight_block_size: [128,128]` + `weight_scale_inv` → ✅ modelopt block-fp8 (the Huihui/official-Qwen format; only proven FP8 path)
|
||||
- `quant_method: "compressed-tensors"` → ❌ FAILS. Per-channel aborts at expert-source resolution (`Missing MoE expert source layers: {'gate_up'}` when expert tensor naming is nonstandard); block-fp8 128×128 aborts in `_dequant_fp8_weight` (multiplies `weight × weight_scale` directly; block scales don't broadcast → shape RuntimeError)
|
||||
- No quant config (BF16) → loads but 35B = ~70 GB weights >> 62 GB system RAM → sustained swap-thrash that locks out SSH (2026-09-01 outage). Never deploy BF16 for a 35B-class model on this box; a clean preflight compares disk size × precision against free RAM + 24 GB VRAM headroom.
|
||||
- Also check: architecture `qwen3_5_moe`, expert tensor naming `mlp.experts.{N}.gate_proj/up_proj/down_proj` (read `model.safetensors.index.json` weight_map on HF if unsure — nonstandard prefixes defeat the MoE expert-source provider), config.json ships the chat template you want (FreeToken reads it from the checkpoint), disk space ≥ checkpoint size.
|
||||
|
||||
If the checkpoint fails the matrix, STOP — do not download, do not touch the running service. Find a compliant quant or use a different runtime.
|
||||
|
||||
## Swap steps
|
||||
|
||||
1. Download into `/home/dank/models/<name>/` via `snapshot_download` with the freetoken venv python (background the SSH call; poll until file count matches the index).
|
||||
2. Edit `scripts/frank-default-small.env`: `MODEL_DIR`, description line (name, quant, arch, disk size, host, date), header comment. Keep F84 pool numbers unless the weight mass or KV dtype changed (FP8 35B ≈ 35 GB: `MODEL_CACHE_SIZE=2048`, `NUM_TOKENS=453005`, reasoning `on`; BF16 66 GB fits nothing — see preflight).
|
||||
3. Run `./tests/run.sh`, commit, push to Gitea origin (git-only flow; never edit on the host).
|
||||
4. `./scripts/deploy.sh dankserver --service frank-default-small`. The LLM_WAIT window is up to 20 min; load of ~35 GB FP8 is minutes warm.
|
||||
5. **Verify for real** — the deploy health gate has reported "healthy" while a doomed worker was still starting (twice). Check journal for Uvicorn startup complete, then `curl /v1/models` (root = your model dir, ctx 262144) and a REAL completion: reasoning must parse into `reasoning_content` and content must answer correctly with a natural `finish_reason: stop`. Give reasoning models enough `max_tokens` to finish thinking (a tight budget yields empty content + `finish: length` — that is correct behavior, not a failure).
|
||||
6. If load fails: journal shows the exact loader error, service exits cleanly (no swap-thrash), `Restart=on-failure` re-loop is harmless. Revert env via `git checkout <last-good-commit> -- <three files>`, redeploy, verify.
|
||||
7. Update the runbook (`serverAppConfigs-docs/LLM/freetoken-frank-default-small.md` quant matrix + history), `log.md`, and AGENTS.md fleet table; stage superseded model dirs to `~/.AGENTTRASH/<date>-topic/` on the host and list sizes for the user — never `rm`.
|
||||
@@ -0,0 +1,49 @@
|
||||
---
|
||||
name: freetoken-rocm-build
|
||||
description: "Verify and execute FreeToken build on ROCm hosts — check gfx1151, ROCm 6.3, hipcc, and dependency wheel availability (torch 2.11, sgl-kernel 0.4.5, flashinfer)"
|
||||
---
|
||||
|
||||
# FreeToken ROCm Build
|
||||
|
||||
## When to use
|
||||
Verify FreeToken can build on an ROCm host (Dankclaw, similar systems) before committing to the build path.
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Check hardware**
|
||||
```
|
||||
rocminfo | grep 'Name.*gfx' # verify GPU architecture
|
||||
lsmod | grep amdgpu # driver loaded
|
||||
ls /opt/rocm/bin/ | grep hipcc # compiler available
|
||||
```
|
||||
|
||||
2. **Check build deps**
|
||||
```
|
||||
cmake --version # needs >=3.20
|
||||
python3 --version # needs >=3.10
|
||||
gcc --version # needs >=9
|
||||
ninja --version # optional but recommended
|
||||
```
|
||||
|
||||
3. **Verify wheel availability**
|
||||
```
|
||||
pip index versions torch # needs 2.11-2.12 range
|
||||
pip index versions sgl-kernel # needs 0.4.5
|
||||
pip index versions flashinfer-python # needs >=0.6
|
||||
```
|
||||
Check PyPI for `rocm` in wheel filenames, or use PyTorch ROCm index.
|
||||
|
||||
4. **Build**
|
||||
```
|
||||
git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken
|
||||
python3 -m venv .venv && source .venv/bin/activate
|
||||
pip install -e ".[accel]" # installs fi + sgl extras
|
||||
```
|
||||
|
||||
## Known issues
|
||||
- nvcc missing on Dankclaw — not needed for ROCm build, only for CUDA JIT
|
||||
- sgl-kernel at versioned endpoint may return 404 — use unversioned `pip index versions`
|
||||
- triton 3.6.0 pins to Linux only
|
||||
|
||||
## CUDA alternative
|
||||
If building for CUDA: install `sudo pacman -S cuda` for nvcc, or use pre-built CUDA wheels that ship pre-compiled kernels.
|
||||
@@ -0,0 +1,46 @@
|
||||
---
|
||||
name: gui-hang-remote-diagnosis
|
||||
description: "Diagnose a frozen GUI/VT-switch failure over SSH by reading kernel hung-task evidence, decoding the lock-holder stack, and deciding recoverability"
|
||||
---
|
||||
|
||||
# GUI Hang Remote Diagnosis
|
||||
|
||||
Use when the desktop is frozen, VT switches do nothing, but SSH still works.
|
||||
|
||||
## Sequence
|
||||
|
||||
1. **Find the smoking gun** — kernel hung-task reporter already logged it:
|
||||
```bash
|
||||
journalctl -b -p err --since "-2 hours" --no-pager | tail -80
|
||||
```
|
||||
Look for `INFO: task X blocked for more than N seconds` and `blocked on a semaphore likely last held by task Y:PID` (modern kernels name the lock holder outright).
|
||||
2. **Get full stacks** for victim and holder:
|
||||
```bash
|
||||
journalctl -b -k --since "<hang minute -5>" --no-pager | tail -120
|
||||
cat /proc/<holder_pid>/status | head -5 # State: R (spinning) vs D (blocked)
|
||||
cat /proc/<victim_pid>/status | head -5
|
||||
```
|
||||
- Victim `D` + holder `R` = livelock/deadlock with lock held.
|
||||
- `voluntary_ctxt_switches` in the millions on an `R` kthread = spinning, not progressing.
|
||||
3. **Prove whether hardware or software** — GPU/CPU still alive? No Xid errors?
|
||||
```bash
|
||||
nvidia-smi --query-gpu=driver_version,pstate,temperature.gpu,clocks_event_reasons.active --format=csv
|
||||
journalctl -b -k --no-pager | grep -i xid
|
||||
```
|
||||
If those answer instantly with no Xid, it's a driver lock issue, not hardware.
|
||||
4. **Find the trigger window**: block start = (first hung message timestamp − reported seconds). Then inspect user-session events:
|
||||
```bash
|
||||
journalctl -b --since "<start-2m>" --until "<start+1m>" --no-pager | grep -v audit
|
||||
```
|
||||
5. **Enumerate collateral**: any other `D` tasks (`ps -eo pid,stat,comm | awk '$2 ~ /D/'`), USB workers, etc.
|
||||
|
||||
## Recoverability verdict
|
||||
|
||||
- Compositor in `D` state → unkillable; spinning kernel thread holding the lock → cannot be poked.
|
||||
**Only exit is reboot**: `sync && systemctl reboot --force --force` (double-force skips stop-jobs that wait forever on D-state PIDs).
|
||||
- If holder is `D` (sleeping on something that can complete, e.g. storage), recovery may be possible by clearing that dependency first.
|
||||
|
||||
## After diagnosing
|
||||
|
||||
- Map stack symbols to driver internals to name the exact subsystem (e.g. `DifrPrefetchEventDeferredWork` → NVKMS DIFR prefetch).
|
||||
- Check upstream: search the exact symbol names + driver version; propose targeted off-switches (`NVreg_RegistryDwords`, module params from `/proc/driver/nvidia/params`) rather than blanket driver swaps.
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
name: hermes-config-home-drift
|
||||
description: "Diagnose Hermes webui/agent model-connection failures caused by HERMES_HOME config-home drift (image VOLUME anonymous volumes resetting on recreation) — verify which home each runtime reads, resolve provider→endpoint with the app's own resolver code, then probe the endpoint."
|
||||
---
|
||||
|
||||
# Hermes config-home drift diagnosis
|
||||
|
||||
Symptom: hermes-webui/agent "cannot connect" to models even though the model endpoints answer curl.
|
||||
|
||||
1. **Find the real config home per runtime.** `podman exec <container> printenv HERMES_HOME` for each hermes container. If webui reads `/opt/data` (image default) while the gateway reads `/data`, the webui runs the image-default template config (`provider: auto` → openrouter.ai with no key = connect failure).
|
||||
2. **Know the recreation trap.** Image `VOLUME` points get a fresh anonymous volume on every container recreation (`podman run --replace --rm`) — edits inside such a mount are lost on restart/redeploy. Compare volume ids before/after a restart to confirm.
|
||||
3. **Fix:** pin `Environment=HERMES_HOME=/data` in the quadlet unit (shared state root), deploy via service-scoped deployer, restart.
|
||||
4. **Verify with the app's own code**, not guesses:
|
||||
- `from api import config as C; C.resolve_model_provider("<model-id>")` → returns `(model_id, provider, base_url)`; bare ids map to owning provider by scanning the `providers:` dict models allow-list; `@provider:model` syntax forces an explicit provider.
|
||||
- `hermes_cli.providers.resolve_user_provider(name, cfg["providers"])` → ProviderDef with real `base_url`; accepts `api`/`url`/`base_url` keys on named provider entries.
|
||||
5. **Probe the endpoint with the exact served model id** (FLM/NPU endpoints route only by exact registry tag and answer HTTP 200 with `{"error":"Failed to load <tag> model!"}` for unknown tags — status-code checks miss this).
|
||||
|
||||
Key gotcha: named providers (`providers:` dict) are a catalog + connection source; `custom:<slug>` connections resolve only via `custom_providers[]` entries. If neither holds an entry for the provider name, the request has no endpoint and fails to connect.
|
||||
@@ -0,0 +1,21 @@
|
||||
---
|
||||
name: hermes-live-config-edit
|
||||
description: "Runbook for changing Hermes (Dankclaw) live config.yaml — model routing, providers, aux routes — including backup, apply, restart, and verification steps"
|
||||
---
|
||||
|
||||
# Editing live Hermes config on Dankclaw
|
||||
|
||||
Hermes-webui/gateway read `/home/dank/Containers/hermes/data/config.yaml` (dir is `drwx------` uid 109999; all reads/edits need `sudo`). The repo file `dankclaw/hermes/config.yaml.example` is a sanitized backup and can lag live — always diff the live file first; never treat the example as truth.
|
||||
|
||||
## Procedure
|
||||
1. **Backup first:** `sudo cp -a .../config.yaml .../config.yaml.bak-YYYYMMDD` (in-dir trash `.AGENTTRASH/` also exists for removals).
|
||||
2. **Apply change** with surgical `sudo sed -i` or full-file write; YAML comments in the live file are tool-generated — preserve them.
|
||||
3. **Sweep stale references** across the whole data dir, not just config.yaml: `sudo grep -rln "<old-id>" /home/dank/Containers/hermes/data --include="*.yaml" --include="*.json"`. Session files/MEMORY.md are historical — leave them.
|
||||
4. **Derived caches:** `webui/models_cache.json` is keyed by a config fingerprint (`_source_fingerprint.config_yaml` mtime/size); after edits it's stale. Either let it regenerate lazily or trash it (mv to data dir `.AGENTTRASH/`) — never hand-edit.
|
||||
5. **Restart the model-consuming units** (they parse config at startup): `systemctl --user restart hermes-webui.service hermes.service`. `hermes-dashboard.service` is unaffected by routing changes.
|
||||
6. **Verify behaviorally**, not by status codes: one tiny completion per endpoint with the exact served id — POST `/v1/chat/completions` on 7770 (Dankclaw, llama.cpp serves alias `frank-large`), on Dankserver 7770 (FreeToken REQUIRES an explicit `model` field), and on 7771/v1 (FLM routes by exact registry tag, e.g. `gemma4-it:e4b`; unknown tags answer HTTP 200 with `{"error":"Failed to load <tag> model!"}` — status-code checks silently pass).
|
||||
7. **Sync the repo backup** (`dankclaw/hermes/config.yaml.example`) to the live shape, redacting secrets (password_hash, phone account stay live-only), then run `./tests/run.sh` before commit.
|
||||
|
||||
## Gotchas
|
||||
- `provider: custom` + `base_url` entries route by base_url and send the model id verbatim; named providers (`dankclaw-large`, `dankserver-small`, `dankclaw-npu`) resolve via their `api:` URL.
|
||||
- Renaming a served alias anywhere (serve scripts, env ALIAS) MUST propagate to this config in the same change or Hermes requests 404 on nonexistent models — this exact gap caused the 2026-09-05 outage.
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
name: hermes-webui-secret-rotation
|
||||
description: Rotate the Hermex (hermes-webui) password on Dankclaw and verify auth behaviorally — covers the dual EnvironmentFile gotcha where only dankclaw/.env is effective at container create.
|
||||
---
|
||||
|
||||
# Hermex webui secret rotation (Dankclaw)
|
||||
|
||||
Auth contract (verified live 2026-08-31): `POST /api/auth/login` with JSON `{"password":"..."}` → 200 + signed HttpOnly `hermes_session` cookie (24 h TTL) on match, 401 on miss. No username field. `/health` answers 200 unauthenticated and `/` answers 302 — never verify auth via those; always probe login with a deliberately wrong password (expect 401) and the real one (expect 200).
|
||||
|
||||
## Rotation procedure
|
||||
1. `ssh -o BatchMode=yes dankclaw 'bash -s'` with a stdin script (zsh on the host mangles inline `${...}` even inside single-quoted ssh args — never interpolate curl `-w %{http_code}` inline; pipe the whole probe as a heredoc'd bash script).
|
||||
2. Update BOTH env files with `sed -i "s/^HERMES_WEBUI_PASSWORD=.*/HERMES_WEBUI_PASSWORD=<new>/"`:
|
||||
- `/home/dank/serverAppConfigs/.env` ([Service] EnvironmentFile)
|
||||
- `/home/dank/serverAppConfigs/dankclaw/.env` — **the effective one**: the quadlet unit's `[Container] EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env` is what the container reads at create; rotating only the root file leaves the running container serving the old secret.
|
||||
3. `systemctl --user restart hermes-webui.service` (unit recreation re-reads env; no deploy needed if the unit file itself didn't change).
|
||||
4. Verify: wrong-pw login → 401, right-pw → 200, from both host loopback and a LAN client at `http://192.168.0.143:8787` (webui publishes on the LAN interface since commit 4443897; off-network ingress is SSH tunnel only — user `tunnel`, port **22**, key-only auth, forward remote end localhost:8787).
|
||||
|
||||
## Gotchas
|
||||
- `sudo podman exec` fails exit 125 here: root's namespace does not see rootless containers — run `podman exec hermes-webui printenv HERMES_WEBUI_PASSWORD` unprivileged (ssh user owns the env files, mode 600).
|
||||
- If rotating because a secret leaked into transcripts/logs, rotate in place via a script that fetches the live value from the running container (`printenv`) instead of transcribing it.
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
name: homepage-status-probe-namespace-triage
|
||||
description: "Triage a homepage status card (or any host-side TCP probe) reporting \"down\" for a service that is actually up: decide by network namespace whether the probe target is reachable at all before calling it an outage."
|
||||
---
|
||||
|
||||
# Goal
|
||||
|
||||
Decide, in one pass, whether a `down` card on the homepage status page (`scripts/homepage-status.py`, run by `homepage-status.timer` on Dankclaw, probes TCP/HTTP from the **host** netns) is a real outage or a structurally unreachable probe target. The checker only has three states — `wip` | `up` | `down`; there is no "unknown", so an unprobeable service can never render honestly and must be dropped or re-pointed, not "fixed".
|
||||
|
||||
# Why it goes wrong
|
||||
|
||||
A host-vantage TCP probe can only reach a listener that is **published to the host** (rootless podman publishes via rootlessport/pasta; unpublished ports are invisible from the host netns). Three recurring false-down shapes on this fleet:
|
||||
|
||||
1. **TLS-only app probed with `method:"http"`.** Plex Media Server 1.43 serves TLS only on `32400`; a plain-HTTP GET gets a TLS alert → curl exit 52 "empty reply from server", code `000` → card says down while PMS answers `200` over HTTPS. Fix = `"method":"tcp"` (connect is the honest liveness signal; the public URL stays HTTP because Caddy re-keys).
|
||||
2. **Retired port.** The Hermes card probed `127.0.0.1:8090` after the compose→quadlet move moved the dashboard to `9119` and the webui to `8787`; nothing listened on 8090, so a fully healthy stack read `down`.
|
||||
3. **Unpublished port / wrong netns.** `gitea-postgres` (`db:5432`, net `systemd-gitea`), `searxng-valkey` (`valkey:6379`, net `systemd-searxng`), and `alloy` (OTLP 12345) publish **no** ports. Container DNS aliases are netns-scoped, so from the host these are unreachable *by construction* — no probe value makes them honest.
|
||||
|
||||
# Procedure
|
||||
|
||||
1. Read the card's probe target in `shared/homepage/services.json`; note method (`tcp` vs `http`).
|
||||
2. Ask: does that port have a **host-visible listener**? `podman ps -a --format '{{.Names}}|{{.Status}}'`, then per container `podman inspect <c> --format '{{json .NetworkSettings.Networks}} {{.HostConfig.NetworkMode}}'` and `podman port <c>` (empty output = published nothing → structurally unprobeable from the host).
|
||||
3. Probe from the **correct vantage**. From a bridge-net container, the host is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on `dankclaw_default`), *not* `host.containers.internal` (`169.254.1.2`) — that pasta alias belongs to the default `podman` net (`10.88.0.0/16`, gw `10.88.0.1`) and is not routed from other nets. Probe with bash `/dev/tcp` (no `python3`/`ip` in these images), and escape the loop var so it expands **inside** `podman exec bash -c`.
|
||||
4. Classify: real outage (container exited — read `State.ExitCode`: 137 = SIGKILL, 143 = SIGTERM/manual stop, `OOMKilled=false` rules out the OOM killer) vs false down (running + unreachable target).
|
||||
5. Fix accordingly: re-point or re-method the probe for a real HTTP/TLS mismatch; **drop** cards whose only exposure is an internal netns (or accept permanent false-downs and say so); never invent a probe target for them.
|
||||
6. Verify by re-running the checker path, not by eyeballing: after deploy, re-read `/srv/homepage/status.json` and diff the `down` list before/after.
|
||||
|
||||
# Watch out
|
||||
|
||||
- Deploying this file is service-scoped: `./scripts/run.sh` (233 tests) → commit → `./scripts/deploy.sh dankclaw --service homepage`; the deployer *pushes* `main` first, so any already-committed local work rides along — say so before pushing.
|
||||
- The checker writes `/srv/homepage/status.json` world-readable (0644) so Caddy can serve it; keep that mode or Caddy 403s the page.
|
||||
- Removing a card's `probe` key without code change crashes the checker (`svc["probe"]` KeyError) — dropping the whole entry is the only zero-code fix.
|
||||
- Bare compose services with `restart=unless-stopped` (node-exporter on both hosts) do **not** survive reboot: rootless podman does not autostart bare containers, and `deploy-remote.sh` refuses to `compose up` Dankserver's compose services by design → they need a quadlet unit, not a restart.
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
name: html-change-report
|
||||
description: Generate and open a self-contained dark-theme HTML change report (Keep a Changelog categories + code samples) for a git repo time window; use when Frank asks for a rendered report of recent work.
|
||||
---
|
||||
|
||||
# HTML change report (Keep a Changelog + review aids)
|
||||
|
||||
Produce a self-contained HTML report of repo work in a time window and open it in Firefox.
|
||||
|
||||
## Data gathering
|
||||
- Window commits: `git log --since="24 hours ago" --pretty='%h|%ad|%s' --date=format:%H:%M` (split on `|`).
|
||||
- Metrics: `git diff --shortstat <first>^..HEAD`, per-area file counts via `git diff --name-only … | awk -F/ '{print $1"/"$2}' | sort | uniq -c | sort -rn`.
|
||||
- Bucket commits into phase/theme rows by time-of-day for the timeline table.
|
||||
|
||||
## Report structure (standard: Keep a Changelog 1.1.0)
|
||||
1. Executive summary (metrics + 3-5 bullets).
|
||||
2. Full commit timeline table, phase-color-tagged (border-left per phase).
|
||||
3. `git show <hash>` command list per theme — Frank uses this for his review pass.
|
||||
4. Changes by category: Added/Changed/Fixed/Removed/Security; omit empty categories; cite commit hashes per item. Fixed = old behavior was a bug; Changed = intentional.
|
||||
5. 4-6 notable code excerpts: read real files, pick load-bearing/instructive code, note file:lines + commits + why it matters. HTML-escape everything (`html.escape`).
|
||||
6. Verification evidence table (what / how / result).
|
||||
7. Incidents & open risks.
|
||||
8. Actions required from Frank (numbered, effort hints).
|
||||
|
||||
## HTML requirements
|
||||
- Self-contained: inline CSS only, NO CDNs (must work from file:// offline). Dark theme CSS vars (#1a1b26 bg family works).
|
||||
- Sticky top nav, section anchors, `<pre class=code><code>` blocks, commit hashes as `.tag` chips.
|
||||
- Build via python (eval kernel): subprocess for git data, string concat for CSS (avoid f-string brace pain), f-strings for rows.
|
||||
|
||||
## Delivery + verification
|
||||
- Write to `/home/dank/Reports/<repo>-<window>-<date>.html` (outside repo, tree stays clean).
|
||||
- Validate: python HTMLParser feed (parse errors) + open in headless browser (`write` JSON to xd://browser: open → run with `document.querySelectorAll` counts + fullPage screenshot) → close tab.
|
||||
- Open for Frank: `nohup firefox --new-tab file:///<path> >/dev/null 2>&1 & disown`.
|
||||
@@ -0,0 +1,43 @@
|
||||
---
|
||||
name: imagegen-openwebui
|
||||
description: Runbook for running high-quality local image generation via Open WebUI imagegen API endpoint.
|
||||
---
|
||||
|
||||
# Open WebUI Image Generation Runbook
|
||||
|
||||
## Overview
|
||||
Use this procedure to generate images locally using the Open WebUI image generation endpoint.
|
||||
This approach leverages local ComfyUI integration configured in Open WebUI.
|
||||
|
||||
## Endpoint
|
||||
POST to the imagegen endpoint of your Open WebUI instance.
|
||||
|
||||
### Example Request
|
||||
```python
|
||||
import requests
|
||||
|
||||
url = "http://<host>:<port>/api/images/generations"
|
||||
headers = {"Authorization": "Bearer <your_token>"}
|
||||
data = {
|
||||
"model": "imagegen",
|
||||
"prompt": "A desired image description.",
|
||||
"n": 1,
|
||||
"size": "1024x1024"
|
||||
}
|
||||
|
||||
response = requests.post(url, headers=headers, json=data)
|
||||
```
|
||||
|
||||
### Example Response
|
||||
Returns an object with base64-encoded image data or a URL.
|
||||
|
||||
## Best Practices
|
||||
- Keep prompts descriptive for best results.
|
||||
- Use the default model unless configured otherwise.
|
||||
- Verify that ComfyUI is running and connected before invoking.
|
||||
- Check Open WebUI logs for troubleshooting if generation fails.
|
||||
|
||||
## Common Issues
|
||||
- **Connection refused**: Ensure Open WebUI and ComfyUI are running.
|
||||
- **Timeout errors**: Large image sizes may take longer; check ComfyUI resources.
|
||||
- **Authentication errors**: Verify API token validity in the request headers.
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
name: llama-flash-attn-qsa-fix
|
||||
description: Restore --flash-attn on for qwen4exp models when QSA indexer path aborts with GGML_ASSERT F32 error
|
||||
---
|
||||
|
||||
## Problem
|
||||
`--flash-attn on` aborts graph build for qwen4exp models:
|
||||
```
|
||||
GGML_ASSERT(a->type == GGML_TYPE_F32)
|
||||
at ggml/src/ggml.c:5371, backtrace through build_attn_qsa
|
||||
```
|
||||
Root cause: `build_attn_qsa` calls `ggml_fill` on the F16-cast kq mask returned by `get_kq_mask()` when flash attn is enabled, but `ggml_fill` requires F32.
|
||||
|
||||
## Solution
|
||||
Apply the QSA F32-mask patch that mirrors the MLA sparse path (`llm_graph_context::build_attn`):
|
||||
1. Use the F32 base mask (`inp->self_kq_mask`) instead of the F16-cast version for the top-k surgery
|
||||
2. Cast to F16 only at the attention call when `flash_attn` is enabled
|
||||
|
||||
### Patch hunks (unified diff format)
|
||||
```diff
|
||||
@@ -560,9 +565,13 @@ ggml_tensor * llama_model_qwen4exp::graph::build_attn_qsa(
|
||||
ggml_build_forward_expand(gf, mctx_cur->cpy_v(ctx0, v_cur, v_idxs, il));
|
||||
}
|
||||
- ggml_tensor * kq_mask = inp->get_kq_mask();
|
||||
+ // get_kq_mask() returns the F16-converted mask when flash attention is
|
||||
+ // on, but ggml_fill/set_rows require F32. Mirror the MLA sparse path
|
||||
+ // (llm_graph_context::build_attn): run the top-k mask surgery on the F32
|
||||
+ // base mask and cast to the attention type only at the end.
|
||||
+ ggml_tensor * kq_mask = inp->self_kq_mask;
|
||||
// prepare new kq mask - starts filled with -INFINITY
|
||||
ggml_tensor * kq_mask_all = ggml_fill(ctx0, kq_mask, -INFINITY);
|
||||
|
||||
@@ -592,9 +601,12 @@ ggml_tensor * llama_model_qwen4exp::graph::build_attn_qsa(
|
||||
ggml_tensor * q = q_cur;
|
||||
ggml_tensor * k = mctx_cur->get_k(ctx0, il);
|
||||
ggml_tensor * v = mctx_cur->get_v(ctx0, il);
|
||||
- ggml_tensor * cur = build_attn_mha(q, k, v, nullptr, kq_mask_top_k, nullptr, nullptr, kq_scale, il);
|
||||
+ // flash attention consumes an F16 mask; softmax consumes F32 as-is
|
||||
+ ggml_tensor * kq_mask_attn = inp->cparams.flash_attn ? ggml_cast(ctx0, kq_mask_top_k, GGML_TYPE_F16) : kq_mask_top_k;
|
||||
+
|
||||
+ ggml_tensor * cur = build_attn_mha(q, k, v, nullptr, kq_mask_attn, nullptr, nullptr, kq_scale, il);
|
||||
cb(cur, "kqv_out", il);
|
||||
return cur;
|
||||
}
|
||||
```
|
||||
|
||||
## Procedure
|
||||
1. **Apply patch to fork:**
|
||||
```bash
|
||||
cd /home/dank/GitPackages/ROCmFPX
|
||||
git apply /path/to/qwen4exp-qsa-f32-mask.patch
|
||||
make -j16 llama-server
|
||||
```
|
||||
2. **Validate live (on test port):**
|
||||
```bash
|
||||
sudo systemctl stop frank-default-large
|
||||
nohup ./build-qwen38/bin/llama-server \n --model ... --flash-attn on --ctx-size 32768 --fit off -t 16 -tb 16 \n --port 7999 > /tmp/fa-test.log 2>&1 &
|
||||
sleep 90 && grep 'model loaded' /tmp/fa-test.log && curl http://127.0.0.1:7999/health
|
||||
```
|
||||
3. **Verify completion:** Send a real `/v1/chat/completions` request — if it completes without asserts, the fix works.
|
||||
4. **Deploy production:** Commit config (FLASH_ATTN=on), push, pull on target, restart service.
|
||||
|
||||
## Verification
|
||||
- Model loads without asserts
|
||||
- Health endpoint returns 200
|
||||
- Real completion succeeds (finish: "stop")
|
||||
- Journal clean: no `GGML_ASSERT`, no `cudaMalloc`, no `missing tensor`
|
||||
- Flash attention engaged (check server logs for "flash attention" or timing improvement vs off)
|
||||
|
||||
## Known Constraints
|
||||
- Works only with charlie12345/ROCmFPX build-qwen38 (PR #27742 qwen4exp merged)
|
||||
- Does NOT work with CIRU runtime, upstream llama.cpp, or stock binaries
|
||||
- ctx-size cap remains 32768 for full-model GPU residency (87 GiB model + KV buffer OOMs beyond)
|
||||
@@ -0,0 +1,41 @@
|
||||
---
|
||||
name: llm-fleet-model-swap
|
||||
description: "Swap the model/runtime behind frank-default-large/small on Dankclaw/Dankserver via the git flow, with pre-deploy live validation and the port-holder health-check trap check."
|
||||
---
|
||||
|
||||
# LLM fleet model swap (frank-default-large / frank-default-small)
|
||||
|
||||
Use when switching model, quant, or llama.cpp/FreeToken runtime for a `frank-default-*` service in `~/Projects/serverAppConfigs`.
|
||||
|
||||
## 0. Ground truth first
|
||||
- Read the HF model card (runtime pin, hard constraints, recommended args) + repo files `scripts/frank-default-<size>.env`, `scripts/serve-frank-<size>.sh`, `scripts/frank-default-<size>.service`, and the runbook under `serverAppConfigs-docs/LLM/`.
|
||||
- Check the host before downloading anything: `ls /home/dank/models/`, `df -h /home`, `du -sh` — bench-era or prior-deployment files often already exist.
|
||||
|
||||
## 1. Artifact integrity
|
||||
- List repo files: `curl -s https://huggingface.co/api/models/<repo>/tree/main?recursive=true` → `.lfs.oid` holds the sha256.
|
||||
- On-host file: compare size AND `sha256sum` against the LFS oid before trusting any existing copy (hash runs ~few min for >100 GiB; run it in background).
|
||||
- New download on Dankserver: `/home/dank/freetoken/.venv/bin/hf download <repo> --local-dir ...`.
|
||||
|
||||
## 2. Runtime (if the card pins a fork/build)
|
||||
- Clone + build with the card's exact cmake line, **plus `-DLLAMA_BUILD_WEBUI=OFF`** — WebUI asset provisioning (npm / HF llama-ui bucket) aborts llama.cpp link at 98% on these hosts; API server unaffected.
|
||||
- Verify every flag our serve script passes exists in the new binary before touching the service:
|
||||
`llama-server --help | grep -q -- "<flag>"` for each; list MISSes.
|
||||
|
||||
## 3. Config via git only
|
||||
- Edit `scripts/frank-default-<size>.env` (LLAMA_BIN, MODEL_DIR, MODEL_FILE, MMPROJ_PATH) + serve script header/defaults; `bash -n` the script; `./tests/run.sh` must pass; commit; push; `git pull --ff-only` on the target.
|
||||
- Do NOT edit configs on the remote. Renaming a model dir (`mv`) is host state, allowed.
|
||||
|
||||
## 4. Live validation (no scratch ports — GPU/RAM can't host two big models)
|
||||
- Ensure the old unit is truly gone: `systemctl is-active` may hang in `stop-sigterm` because llama.cpp never preempts an in-flight generation; poll until the PID exits before launching.
|
||||
- Stop service, run the real serve script manually (`setsid nohup bash scripts/serve-frank-<size>.sh > /tmp/x.log 2>&1 &` — plain `ssh 'nohup ... &'` dies with its ssh when foregrounded; keep ssh under timeout or it gets killed mid-stop).
|
||||
- Verify: health 200; a real `/v1/chat/completions` (FreeToken REQUIRES explicit `"model"` field; llama.cpp servers don't care); reasoning flows (`reasoning_content` non-empty; content may be empty with finish=length if the thinking eats the budget).
|
||||
|
||||
## 5. systemd handoff + THE TRAP
|
||||
- `./scripts/deploy.sh <host> --service <svc>` — but a leftover manual server holding :7770 makes the health check pass while the UNIT crash-loops on bind.
|
||||
- Always confirm ownership: `systemctl show <svc> -p MainPID` equals the single `pgrep -x llama-server` (or the ft serve PID); only then trust "healthy after Ns".
|
||||
- If a real long-context task is queued/running, do the handoff anyway (client aborts are recoverable; drift is not) — or defer cosmetic-only redeploys.
|
||||
|
||||
## 6. Close-out (mandatory)
|
||||
- Docs: active runbook updated (build recipe, profile, validation numbers incl. real-workload prefill/decode), superseded runbook gets RETIRED banner kept as rollback record, `LLM/index.md` mermaid + active list, AGENTS.md fleet table + Runtime section + history chain, work log entry in `serverAppConfigs-docs/log.md`.
|
||||
- `~/.pi/agent/models.json`: contextWindow must equal the REAL per-slot context — llama.cpp logs `n_ctx_slot` per prompt; parallel N clamps `n_ctx_train` (e.g. 262144/2 = 131072/slot). Overselling causes silent client-side overflow.
|
||||
- Rollback = one env commit (paths + LLAMA_BIN); keep prior binary + payload on disk.
|
||||
@@ -0,0 +1,50 @@
|
||||
---
|
||||
name: llm-model-swap-fit-preflight
|
||||
description: "Preflight an LLM model swap for hardware fit before deploying; catch the silent swap-thrash outage mode (large RAM OOM, self-restarting + boot-enabled units) and recover."
|
||||
---
|
||||
|
||||
# LLM Model-Swap Fit Preflight
|
||||
|
||||
Never deploy a model swap without confirming the checkpoint FITS, and know the
|
||||
failure mode when it doesn't: it is a silent, self-sustaining thrash, not a crash.
|
||||
|
||||
## Preflight (before deploy)
|
||||
|
||||
1. Get the checkpoint's on-disk/weight size AND precision from the HF card
|
||||
(`Precision: BF16` = 2 bytes/param; FP8/Q8 ≈ 1 byte; Q4 ≈ 0.5 byte).
|
||||
35B params: BF16 ≈ 66–70 GB, FP8/Q8 ≈ 33–35 GB, Q4 ≈ 17–20 GB.
|
||||
2. Compare against the HOST'S USABLE RAM (not just VRAM) for runtimes that
|
||||
stream weights from RAM (FreeToken host-RAM experts, llama.cpp CPU offload,
|
||||
PP/worker splits).
|
||||
3. Budget RAM for weights + KV pool + OS + page cache. If weights alone exceed
|
||||
RAM, ABORT — pool tuning shrinks KV, it does not shrink weights.
|
||||
|
||||
## The self-sustaining thrash (why 'it'll just OOM and recover' is wrong)
|
||||
|
||||
- Large swap (tens of GB) → kernel PAGES instead of OOM-killing → sshd/user
|
||||
space starved indefinitely; no recovery window opens by itself.
|
||||
- systemd `Restart=on-failure|always` → crash-reload-rethrash loop.
|
||||
- Unit `enabled` + `WantedBy=multi-user.target` → a reboot re-thrashes on boot
|
||||
until the bad config is masked/reverted. A plain power-cycle does NOT fix it.
|
||||
|
||||
## Recovery (needs a shell; may need console / SysRq-to-GRUB)
|
||||
|
||||
```
|
||||
sudo systemctl mask --now <svc> # stop + break the reboot trap FIRST
|
||||
cd <checkout> && git fetch origin && git reset --hard origin/main # pick up revert
|
||||
grep MODEL_DIR <env> # confirm fixed target
|
||||
sudo systemctl unmask <svc> && sudo systemctl restart <svc>
|
||||
curl -s <health> # then verify with a REAL completion
|
||||
```
|
||||
|
||||
## Deploy-script caveat
|
||||
|
||||
`deploy.sh`/health-gate 'healthy after Ns' = systemd ACTIVE, not model LOADED.
|
||||
Always confirm load with a real completion (curl /v1/chat/completions), not
|
||||
status alone.
|
||||
|
||||
## GPU quant note
|
||||
|
||||
RTX 3090 (Ampere) has no FP8/FP4 tensor cores: FP8 weights run via upcast but
|
||||
still must fit in RAM/VRAM. Choose quant by SIZE fit, then accept the compute
|
||||
path the card supports.
|
||||
@@ -0,0 +1,46 @@
|
||||
---
|
||||
name: pico8-asset-standards
|
||||
description: "Standards for building cohesive, production-grade PICO-8 asset sheets, tilesets, and spritesheets with consistent palette, perspective, and detail flow."
|
||||
---
|
||||
|
||||
# PICO-8 Asset Standards
|
||||
|
||||
## Purpose
|
||||
Ensure every PICO-8 asset (tileset, spritesheet, character sheet, or asset sheet) is cohesive, production-grade, and ready to drop into a game without rework.
|
||||
|
||||
## Core Requirements
|
||||
|
||||
### Palette Discipline
|
||||
- Use the PICO-8 16-color palette exclusively.
|
||||
- Establish a dominant color family (2–3 related hues) for the main subject.
|
||||
- Reserve accent colors for highlights, danger zones, and focal points only.
|
||||
- Maintain consistent color roles across all assets in a sheet (e.g., if shadow is dark blue on one tile, it is dark blue everywhere).
|
||||
|
||||
### Perspective & Lighting
|
||||
- Choose ONE perspective per sheet (top-down, side-view, or isometric) and never mix.
|
||||
- Apply light from a single consistent direction across all tiles and sprites.
|
||||
- Shadows follow the same shape logic on every asset (e.g., all cast shadows point bottom-right).
|
||||
|
||||
### Detail Flow & Hierarchy
|
||||
- Start tiles/props from the palette's dominant colors, then add edge highlights and shading.
|
||||
- Keep tile density uniform — avoid one hyper-detailed tile next to empty flat tiles.
|
||||
- Props and decorations should read clearly at 8x8 or 16x16 without zooming.
|
||||
|
||||
### Sheet Organization
|
||||
- Group related assets together (terrain together, props together, characters together).
|
||||
- Leave consistent spacing between assets for clean cropping.
|
||||
- Label sections visually through grouping, not text.
|
||||
|
||||
### Cohesion Checks
|
||||
Before finishing an asset sheet, verify:
|
||||
- All colors appear in the PICO-8 palette.
|
||||
- Perspective does not flip mid-sheet.
|
||||
- Lighting direction is consistent.
|
||||
- Edge treatment (dither, outline, anti-aliasing style) matches across assets.
|
||||
- No orphan tiles that don't connect to any neighbor.
|
||||
|
||||
## Common Pitfalls
|
||||
- Mixing 8-bit and 16-bit detail levels.
|
||||
- Using outline style on some tiles but not others.
|
||||
- Forgetting palette constraint — colors outside the 16 PICO-8 colors.
|
||||
- Inconsistent tile edge connections (a grass tile that only connects to one dirt variant).
|
||||
@@ -0,0 +1,29 @@
|
||||
---
|
||||
name: podman-netavark-host-port-probe
|
||||
description: "Diagnose \"connection refused\" when a rootless-podman container tries to reach a host service port on this netavark host; proves the published-port-only quirk and picks the working path (host.containers.internal)."
|
||||
---
|
||||
|
||||
# Podman/netavark: container → host port connectivity probe
|
||||
|
||||
Use when a containerized service on a rootless-podman + netavark host cannot reach a HOST process by LAN IP (connection refused immediately, not a timeout) even though `ss -ltnp` proves the listener is up in the host netns.
|
||||
|
||||
## Why this happens
|
||||
On netavark-backed networks, container→host-LAN-IP traffic is only delivered for ports that have a live port-forward/DNAT (published ports). Un-published host ports refuse instantly — there is no firewall rule and no stale DNAT to find; `nft list tables`/`iptables -t nat` will be empty. Do not hunt for REJECT rules; probe instead.
|
||||
|
||||
## Probe recipe (proves the quirk in one command)
|
||||
```bash
|
||||
ssh <host> "podman run --rm --network <container-network> --entrypoint sh docker.io/library/busybox:latest -c 'for p in <suspicious-port> <published-port>; do nc -w2 -z -v <host-lan-ip> \$p >/dev/null 2>&1 && echo port-\$p-OK || echo port-\$p-BAD; done'"
|
||||
```
|
||||
- Published port OK + target port BAD ⇒ published-port-only quirk confirmed.
|
||||
- Also probe the alias path from inside the actual service container:
|
||||
```bash
|
||||
podman exec <svc> bash -c 'exec 3<>/dev/tcp/host.containers.internal/<port> && echo ALIAS-OK || echo ALIAS-FAIL'
|
||||
```
|
||||
Note: busybox `nc -z` reports any failure as failure — use it for bulk port sweeps; use the `/dev/tcp` probe (or `nc -v`) when the refusal-vs-timeout distinction matters.
|
||||
|
||||
## Fix decision
|
||||
- If the alias path connects and the LAN-IP path refuses → set the service's host-facing URL to `http://host.containers.internal:<port>` in its compose/quadlet env, then redeploy via the normal git flow (`deploy.sh <target> --service <svc>` recreates the container with fresh env).
|
||||
- Do NOT publish the daemon port just to make the LAN IP work — publishing is exactly what makes other ports refuse.
|
||||
|
||||
## Post-fix verification
|
||||
`podman exec <svc> printenv <URL_VAR>` then tail logs for the connect confirmation (e.g. `Signal: connected`, `SSE: connected`) within ~40 s of recreation.
|
||||
@@ -0,0 +1,51 @@
|
||||
---
|
||||
name: podman-netns-addressability-probe
|
||||
description: "Runbook to answer \"can container X actually reach service Y\" on a rootless-podman/netavark host, instead of assuming localhost works everywhere; includes the probe commands and the false-positive traps."
|
||||
---
|
||||
|
||||
# Podman netns addressability probe
|
||||
|
||||
Answer **"can this container actually dial that service?"** per *network namespace*,
|
||||
not per host. On rootless podman + netavark, a bridge-net container cannot reach
|
||||
host-netns loopback listeners at all, so `localhost:PORT` reasoning is wrong for it.
|
||||
|
||||
## Decision rule
|
||||
- `NetworkMode=host` → the container **is** the host netns: dials everything on
|
||||
`127.0.0.1`, binds any free port.
|
||||
- bridge/named network → own netns. Only **published** ports are reachable, via the
|
||||
network's **gateway IP**. Unpublished host listeners are **refused instantly (RST)**,
|
||||
not routed — even when the listener is bound `0.0.0.0`.
|
||||
|
||||
## Probe sequence (all read-only, run on the host)
|
||||
1. Namespace: `podman inspect <name> --format '{{.HostConfig.NetworkMode}}'`
|
||||
2. Owner of a listener: `sudo ss -ltnp 'sport = :<port>'` (host netns only — this is
|
||||
why a `0.0.0.0` listener can still be unreachable from a bridge net).
|
||||
3. From inside the suspect container:
|
||||
`podman exec <name> cat /proc/net/route` → default route = gateway IP to dial.
|
||||
4. Probe the candidate address **inside that netns**:
|
||||
`podman exec <name> sh -c 'timeout 2 bash -c "exec 3<>/dev/tcp/<gw-ip>/<port>"; echo rc=$?'`
|
||||
|
||||
## Interpretation (do not blur these)
|
||||
- **instant refusal** = nothing listening in a reachable namespace (or rootlessport
|
||||
rejected an unpublished port map).
|
||||
- **timeout** = no route to that address at all. `host.containers.internal`
|
||||
(`169.254.1.2`) is an alias on the **default `podman` net only**; containers on a
|
||||
named network time out against it rather than being refused.
|
||||
|
||||
## Probe-tool gotchas verified on this host (podman 5.x, netavark)
|
||||
- `podman network inspect <net> --format '{{json .containers}}'` **errors**
|
||||
(`can't evaluate field containers in type interface {}`) — membership is not exposed
|
||||
there. List from the container side instead:
|
||||
`podman ps -a --format '{{.Names}} {{.Networks}}'`
|
||||
- An **empty** `Networks` field in that output = host network or a shared namespace
|
||||
(`--network=container:<other>`), not "no network".
|
||||
- Quadlet `Network=<name>.network` yields runtime network name **`systemd-<name>`**;
|
||||
probe with `podman network ls`, never by the `.network` filename.
|
||||
|
||||
## Fix options when a bridge-net container must reach a host-netns listener
|
||||
(a) move it to `Network=host`, or (b) publish the port — (b) makes that port reachable
|
||||
from every namespace **and the LAN**, so it is an exposure change needing explicit
|
||||
operator approval, never a silent fix.
|
||||
|
||||
Repo source of truth for this fleet: `serverAppConfigs-docs/Architecture/Container
|
||||
Network Namespaces.md` (commit `2e179b0`).
|
||||
@@ -0,0 +1,28 @@
|
||||
---
|
||||
name: podman-recreate-anon-volume-wipe
|
||||
description: "Diagnose \"container data wiped after restart\" on this rootless-podman fleet: quadlet restarts recreate containers via podman run --replace --rm, and anonymous volumes reset to empty on every recreation. Includes the live audit procedure and the known affected services (hermes trio /opt/data, romm /romm DB, searxng cache)."
|
||||
---
|
||||
|
||||
# Podman recreation & anonymous-volume wipe audit
|
||||
|
||||
## Mechanism (verified live 2026-08-31 on Dankclaw)
|
||||
A quadlet unit restart is NOT a start of the same container: the generated unit's ExecStart is
|
||||
`podman run --name systemd-%N --replace --rm ...`. Every `systemctl --user restart <unit>.service` (and every deploy that does one, e.g. `deploy_hermes`) **destroys and recreates** the container. Consequences at recreation time:
|
||||
|
||||
- Writable-layer writes: gone by definition.
|
||||
- Anonymous volumes (image-declared `VOLUME` with no explicit `-v`/`Volume=` for that path): NOT carried across recreation — the new container gets a FRESH EMPTY one seeded only from image content. Orphans stay on disk until pruned (`podman volume prune` only removes unmounted ones).
|
||||
- Named volumes and bind mounts: persist (remounted by name/path).
|
||||
|
||||
## Audit procedure
|
||||
1. `ssh <host> 'podman ps -a'` — map which host runs which stack (aliases `dankclaw`/`dankserver` match their hostnames).
|
||||
2. Per container: `podman inspect <name> --format "{{json .Mounts}}"` → classify each mount `"Type":"bind"` vs `"Type":"volume"` (hex-name volume = anonymous).
|
||||
3. Quirks: `podman ps --format` template field is `.Names`, not `.Name` (psReporter has no `Name` field — using `.Name` errors). Volume `_data` dirs under `~/.local/share/containers/storage/volumes/<id>/_data` are mode 0700 — unreadable by the plain ssh user; use `podman unshare`/sudo if content inspection is required.
|
||||
|
||||
## Fleet state at audit time
|
||||
- **hermes trio** (`systemd-hermes`, `hermes-dashboard`, `hermes-webui`; gateway container auto-names `systemd-<unit>` because its unit lacks `ContainerName=`): `/data` = bind of `/home/dank/hermes-data` (durable state: config.yaml, sessions/, skills/, state.db) survives recreations; `/opt/data` (image default VOLUME + `HERMES_LAZY_INSTALL_TARGET`) is an anonymous volume → resets to image defaults on every deploy. So "Hermes lost data after restart" = anything written outside `/data`/`/webui` was reset, not the durable state.
|
||||
- **romm** (dankserver): runs with anonymous volumes at `/romm` and `/redis-data` → its DB lives in an anon volume; recreating romm (`podman-compose up -d` after image bump) boots it with a fresh empty DB. Fix = named volume or bind before next recreation.
|
||||
- **searxng**: `/var/cache/searxng` is an anonymous volume — cache only, benign reset on recreation.
|
||||
- Everything else on both hosts was verified as bind mounts (or stateless by design: authentik-server has no mounts at all; anubis is a memory store with client-side JWTs).
|
||||
|
||||
## Fixes when data must survive recreation
|
||||
Pin a NAMED volume (`Volume=hermes-data:/opt/data` in the quadlet unit) or point the env var at a bind-mounted path. Never rely on anonymous volumes surviving recreation.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: porkbun-dns-ops
|
||||
description: "Runbook for mutating Porkbun DNS records for franksdank.cloud via API from the target host, covering the relative-name edit quirk, rate-limit failure shape, and verify steps."
|
||||
---
|
||||
|
||||
# Porkbun DNS ops (franksdank.cloud)
|
||||
|
||||
Operate the public DNS zone for `franksdank.cloud` via the Porkbun API v3, run **on Dankclaw** so the API key/secret sourced from `/home/dank/serverAppConfigs/.env` (`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) never leaves the host. Pipe scripts to the target with `ssh -o BatchMode=yes dankclaw 'bash -s' < /tmp/script.sh`.
|
||||
|
||||
## Endpoints (all POST, JSON body)
|
||||
- List: `https://api.porkbun.com/api/json/v3/dns/retrieve/franksdank.cloud` — auth via headers `X-API-Key` + `X-Secret-API-Key` (Bearer is wrong here). Response shape is **top-level `.records[]`**, NOT `.content.records[]` as some docs claim — parse with `jq -r '.records[] | [.id,.name,.type,.content,(.ttl|tostring)] | @tsv'`.
|
||||
- Delete: `/dns/delete/{domain}/{id}` (body carries apikey+secretapikey).
|
||||
- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`, `type`, `content`, `ttl`, `prio`).
|
||||
|
||||
## Quirks that bite (verified live 2026-08-31)
|
||||
1. **`name` is a RELATIVE label on edit.** Passing `ai.franksdank.cloud` yields the record `ai.franksdank.cloud.franksdank.cloud`. Pass the bare label (`ai`) and the API appends the zone itself. Verify after every rename with a retrieve before declaring done.
|
||||
2. **Rate limit answers as an HTML 404-style page, not JSON.** A batch of rapid mutations makes some calls return `<!DOCTYPE html>...` where jq then errors (`Cannot iterate over null` / `Invalid numeric literal`). Do not assume the call succeeded — sleep ~8 s and retry the failed ids; verify by re-listing the zone at the end.
|
||||
3. Deletes/renames are per-record-id: always retrieve first, map name→id, then mutate by id.
|
||||
|
||||
## Rename procedure (vhost rename = DNS + Caddy together)
|
||||
1. `retrieve` → collect ids for the renamed names.
|
||||
2. `edit` each id with the new **relative** label, preserving type/content/ttl.
|
||||
3. Re-retrieve and confirm `<new>.franksdank.cloud A <ip>`.
|
||||
4. In-repo lockstep before deploy: rename the Caddyfile vhost block AND the Pi-hole `dankclaw/pihole/addn-hosts` line in the same commit — test 178 (`tests/pihole.bats`) fails the suite if any Caddyfile vhost lacks an addn-hosts A record on 192.168.0.143.
|
||||
5. Deploy via `scripts/deploy.sh dankclaw` (plans pihole+caddy from changed paths). Capture the suite's true exit with `out=$(./tests/run.sh 2>&1); rc=$?` — piping to `tail` masks the exit code behind `tail`'s own status and has shipped untested commits here.
|
||||
6. Verify live: `dig +short <name> @192.168.0.143` then `curl -o /dev/null -w %{http_code} https://<new-name>` (expect 200/302; first hit triggers dns-01 issuance, which works without inbound port 80).
|
||||
@@ -0,0 +1,30 @@
|
||||
---
|
||||
name: vhost-proxy-health-probes
|
||||
description: Verify reverse-proxied homelab services from the client side with status-code probe batteries; decode 404/502/login-page answers instead of trusting 200s
|
||||
---
|
||||
|
||||
# vhost proxy health probes
|
||||
|
||||
After deploying or auditing a Caddy/Anubis-style front door, verify services **from a client**, not from the backend's localhost — that is where transport mismatches, auth walls, and dead upstreams show up.
|
||||
|
||||
## Probe battery (per vhost, from a LAN client)
|
||||
|
||||
```
|
||||
curl -s -o /dev/null -m 6 -w "%{http_code}\n" https://<vhost>/ping
|
||||
curl -s -o /dev/null -m 6 -w "%{http_code}\n" https://<vhost>/
|
||||
```
|
||||
|
||||
## Decoding the answer — a status code is a diagnosis, not a failure
|
||||
|
||||
- `200` on `/ping` from *arrs = alive; **auth ON/OFF unknown** (`/ping` bypasses auth by design) — probe `/` to decide.
|
||||
- `404` = backend **alive**, route missing (qBittorrent has no `/ping`; 404 there = healthy).
|
||||
- `502` = Caddy could not reach/negotiate with the upstream: dead backend **or transport mismatch**. Probe the backend port directly from the proxy's network (`ssh <proxy-host> curl http://<backend>:<port>/...`). Direct 200 + vhost 502 ⇒ config bug, not an outage.
|
||||
- Login page to anonymous (grep body for `password`) = auth ON; instant redirect to `/transferlist/`-style routes = auth OFF = open door.
|
||||
|
||||
## Transport-mismatch mechanism
|
||||
|
||||
`transport http {versions h2c 1.1}` makes Caddy send an h2c upgrade preface; .NET/Kestrel backends (Jellyfin) reject it → 502 per request while the same port answers 200 on plain HTTP/1.1. Fix = drop the transport override (default HTTP/1.1). Opposite twin: Plex PMS drops *plain* HTTP off-loopback and needs a TLS transport. Match the transport to what the backend speaks.
|
||||
|
||||
## Shell trap when gating commits on a test suite
|
||||
|
||||
`grep -c "^not ok"` exits **1 when zero mismatches** — inside an `&&` chain it silently kills everything after it while still printing `0`. Wrap: `(./tests/run.sh | grep -cE '^not ok' || true) && git commit ...`.
|
||||
@@ -0,0 +1 @@
|
||||
Provides operational playbook for niri Wayland compositor. Covers cursor/Browser XDG defaults, SIGUSR1 hot-reload workflow, and nvidia-modeset semaphore deadlock recovery via modprobe config and force-reboot.
|
||||
@@ -0,0 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
# Hot-reload niri without killing session
|
||||
if pgrep -x niri > /dev/null; then
|
||||
niri-msg reload
|
||||
# Fallback signal method
|
||||
kill -SIGUSR1 $(pgrep niri)
|
||||
fi
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# /etc/modprobe.d/nvidia-workarounds.conf
|
||||
# Resolves nvidia-modeset semaphore deadlock freezing niri options nvidia-modeset disable_vga=1
|
||||
options nvidia-modeset modeset=1
|
||||
@@ -0,0 +1 @@
|
||||
Git operations tailored for OMP environment. Addresses dirty `agent/memories/` tracking pitfalls, implements stash-before-checkout protocol, and standardizes PR creation in dank/command-center.
|
||||
+8
@@ -0,0 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
# Safely checkout main in OMP
|
||||
if [ -n "$(git status -s agent/memories/)" ]; then
|
||||
git stash
|
||||
echo "Stashed dirty agent/memories/ files."
|
||||
fi
|
||||
git checkout main
|
||||
git pull origin main
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
# PR Branch Convention
|
||||
- Branch: pr/fix-models
|
||||
- Target: main
|
||||
- Strategy: Single commit, clean history
|
||||
- Workflow: Feature -> PR -> Squash/Merge
|
||||
@@ -1,20 +0,0 @@
|
||||
# OMP Harness & Migration Playbook
|
||||
|
||||
## Environment Setup
|
||||
- Home: /home/dank/
|
||||
- Config: ~/.omp/agent/APPEND_SYSTEM.md (replaces Pi's SYSTEM.md)
|
||||
- Prompt Behavior: APPEND renders after default harness prompt. Use `omp config set personality none` to clear built-in personality while preserving custom APPEND.
|
||||
- Skills: Auto-discovered from ~/.claude/skills. Trigger via /skill:<name>.
|
||||
|
||||
## Tool Migration (Pi → OMP)
|
||||
| Pi | OMP | Notes |
|
||||
|---|---|---|
|
||||
| subagent | task | Direct replacement |
|
||||
| concurrency: N | Dropped | Harness auto-queues excess |
|
||||
| context: "fresh"/"fork" | Dropped | Subagents start blank/self-contained |
|
||||
| async: true | Dropped | Use local:// unique paths if needed |
|
||||
|
||||
## Prompt Maintenance
|
||||
- Remove hardcoded dates (OMP injects real time).
|
||||
- Keep persona constraints in APPEND_SYSTEM.md.
|
||||
- Reference omp:// docs for extensions/hooks/mcp-config.
|
||||
@@ -0,0 +1 @@
|
||||
Handles dual config synchronization, Ki-style context sizing, and validation for Frank's OMP model definitions. Covers lock-step editing of models.json and models.yml, contextWindow/maxTokens mirroring, and FLM WebP constraints.
|
||||
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
# Validates model config sync
|
||||
jq -e '.providers."llamacpp (dankserver)".models[0] | .contextWindow, .maxTokens' ~/.pi/agent/models.json
|
||||
python3 -c "import yaml; d=yaml.safe_load(open('/home/dank/.omp/agent/models.yml')); m=d['providers']['llamacpp (dankserver)']['models'][0]; print(m['id'], m['contextWindow'], m['maxTokens'])"
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"contextWindow": 204800,
|
||||
"maxTokens": 204800
|
||||
}
|
||||
@@ -1,21 +0,0 @@
|
||||
# Peon-Ping Skill Integration
|
||||
|
||||
## Available Skills
|
||||
- peon-ping-config
|
||||
- peon-ping-log
|
||||
- peon-ping-toggle
|
||||
- peon-ping-use
|
||||
|
||||
## Usage Playbook
|
||||
1. Locate skills in ~/.claude/skills (shared across agents).
|
||||
2. Trigger via slash commands: /skill:grilling, /skill:tdd.
|
||||
3. skills.enableSkillCommands defaults to true.
|
||||
4. Use appropriate sub-skill based on operation:
|
||||
- Config: peon-ping-config to set thresholds/paths.
|
||||
- Monitor: peon-ping-log for status/output.
|
||||
- Control: peon-ping-toggle to enable/disable.
|
||||
- Execute: peon-ping-use to run targeted checks.
|
||||
|
||||
## Notes
|
||||
- Skills persist across agent sessions.
|
||||
- Integrate with OMP task tool for asynchronous monitoring if needed.
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
# Caddy, Authentik, Anubis, and split-DNS playbook
|
||||
|
||||
Use this when modifying franksdank.cloud routes, DNS records, or auth wrapping.
|
||||
|
||||
Architecture:
|
||||
- Single Caddy container on Dankclaw fronts services on Dankclaw and Dankserver.
|
||||
- LAN DNS via Pi-hole resolves service names to 192.168.0.143.
|
||||
- Public DNS via Porkbun points exposed names to WAN IP 71.227.227.172.
|
||||
- TLS uses Porkbun DNS-01 wildcard certificates; ordinary A-record deletion does not break wildcard renewal.
|
||||
- Porkbun credentials are machine-local secrets, never tracked.
|
||||
|
||||
Caddy rules:
|
||||
- Keep Caddy admin enabled on 127.0.0.1:2019 inside the container so file watcher hot-reload works.
|
||||
- Caddy routes by Host header; it does not care whether the name resolved via Pi-hole or Porkbun.
|
||||
|
||||
DNS changes:
|
||||
1. If a service should be LAN-only, it is usually safe to remove its ordinary Porkbun A record while keeping Pi-hole local DNS.
|
||||
2. Removing public DNS does not remove router port forwards or block direct IP access.
|
||||
3. Do not casually remove public DNS for auth.franksdank.cloud; external login flows may need it.
|
||||
4. Be cautious with gitea.franksdank.cloud because external git smart-HTTP use may depend on public DNS.
|
||||
|
||||
Auth wrapping:
|
||||
- Protected via Authentik forward_auth: homepage, SearXNG, Hermes, Prometheus, RomM, Copyparty.
|
||||
- Usually bypass forward_auth because they have native auth: Grafana, Gitea, OpenWebUI, Jellyfin, *arr, Pi-hole.
|
||||
|
||||
Verification pitfalls:
|
||||
- Curl or bot-like clients may get Anubis JS challenge page with HTTP 200. That does not prove the app or outpost succeeded.
|
||||
- Authentik outpost provider matching depends on exact Host header. Missing/wrong Host in CLI probes can produce misleading 404s.
|
||||
- Verify end-to-end login in a real browser after solving Anubis. Use logs to debug outpost 404/provider host-match failures.
|
||||
+36
@@ -0,0 +1,36 @@
|
||||
# frank-default-large operations
|
||||
|
||||
Use this when changing or troubleshooting the large model service on Dankclaw.
|
||||
|
||||
Current durable model/runtime:
|
||||
- Host: Dankclaw, Strix Halo gfx1151 unified memory.
|
||||
- Alias: frank-default-large.
|
||||
- Endpoint: http://192.168.0.143:7770/v1, health on port 7770.
|
||||
- Repo files: scripts/serve-frank-large.sh and scripts/frank-default-large.env under /home/dank/Projects/serverAppConfigs.
|
||||
- Runtime: /home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server.
|
||||
- Model: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF.
|
||||
- Local GGUF: /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf.
|
||||
|
||||
Hard constraints:
|
||||
- Keep --fit off unless the qwen4exp hyper-connection auto-fit segfault is proven fixed.
|
||||
- Keep PLE table mmap/page-cache streamed. Do not add --no-mmap and do not force the PLE table into GPU memory.
|
||||
- Avoid bare env vars TEMP, HOST, and GPU_MAX_HEAP_SIZE. HIP/ROCm can read them unexpectedly.
|
||||
- Do not enable MTP/speculative decoding casually. Prior MTP heads failed because blk.0.hc_attn_norm.weight was missing. Latest durable state has SPEC_TYPE=none.
|
||||
- Stock llama.cpp cannot load this model; qwen4exp architecture support and ROCmFP4 tensor support must coexist in the same runtime tree.
|
||||
- If flash-attn graph build fails on QSA mask, port scripts/patches/qwen4exp-qsa-f32-mask.patch into the active runtime tree and rebuild.
|
||||
|
||||
Before edits:
|
||||
1. cd /home/dank/Projects/serverAppConfigs
|
||||
2. git status --short
|
||||
3. Inspect current scripts/frank-default-large.env and scripts/serve-frank-large.sh; do not rely on old memory for exact tunables.
|
||||
|
||||
After edits:
|
||||
1. Run repo tests appropriate to the change.
|
||||
2. Deploy/restart using the current repo workflow.
|
||||
3. Verify health: curl -s http://192.168.0.143:7770/health or via localhost on Dankclaw.
|
||||
4. Watch logs during model load; 113+ GiB mmap/model load can take significant time.
|
||||
|
||||
Benchmarking notes:
|
||||
- Separate prefill/TTFT from decode tok/s.
|
||||
- For decode sweeps, an 8K prompt and 256 generated tokens is sufficient unless testing long-run thermal/stability.
|
||||
- Prior decode sweep found THREAD_BATCH=16 best, but current repo env always wins.
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
# Hermes on Dankclaw operations
|
||||
|
||||
Use this when Hermes, Hermes WebUI, dashboard, Signal, or Gemma/FLM auxiliary routing is involved.
|
||||
|
||||
Facts:
|
||||
- Host: Dankclaw 192.168.0.143.
|
||||
- Live data: /home/dank/hermes-data mounted as /data.
|
||||
- HERMES_DATA_DIR and HERMES_HOME must point to /data.
|
||||
- Repo config.yaml.example is sanitized only; preserve live secrets and generated state.
|
||||
- WebUI has listened on port 8787. Dashboard uses 9119. Chrome DevTools Protocol binds to 127.0.0.1:9222.
|
||||
- Signal HTTP adapter expects signal-cli-daemon on 127.0.0.1:8080.
|
||||
|
||||
Health/status check:
|
||||
- ssh dankclaw 'for u in hermes hermes-webui hermes-dashboard signal-cli-daemon; do printf "%-22s %s %s\n" "$u" "$(systemctl --user is-active $u.service 2>/dev/null)" "$(systemctl --user is-enabled $u.service 2>/dev/null)"; done; podman ps -a --format "{{.Names}} {{.Status}} nets=[{{.Networks}}]" | grep -Ei "hermes|signal|chrome"; ss -lntp 2>/dev/null | grep -wE "8080|8787|9119|9222" | tr -s " "'
|
||||
|
||||
If WebUI is up but Signal/Hermes transport errors appear:
|
||||
1. Check signal-cli-daemon first:
|
||||
- ssh dankclaw 'systemctl --user status signal-cli-daemon.service --no-pager; journalctl --user -u signal-cli-daemon.service -n 120 --no-pager'
|
||||
2. Only if appropriate, start/enable:
|
||||
- ssh dankclaw 'systemctl --user enable --now signal-cli-daemon.service'
|
||||
3. Verify:
|
||||
- ssh dankclaw 'systemctl --user is-active signal-cli-daemon.service; ss -lntp | grep -w 8080'
|
||||
|
||||
Config edit caution:
|
||||
- Do not replace /home/dank/hermes-data/config.yaml wholesale with repo example.
|
||||
- Live /data/config.yaml ownership may require sudo-safe edit flow.
|
||||
- After live config edits, restart the appropriate Hermes unit/container and verify health.
|
||||
|
||||
Model routing memory:
|
||||
- Main model: frank-default-large via dankclaw-large at http://192.168.0.143:7770/v1.
|
||||
- Vision auxiliary: gemma4-it:e4b at http://192.168.0.143:7771/v1 via provider/custom endpoint.
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
# Homelab service triage
|
||||
|
||||
Use this when checking service state across Dankclaw/Dankserver from Dankware or another controller.
|
||||
|
||||
1. Start from the repo and trust current repo state over memory:
|
||||
- cd /home/dank/Projects/serverAppConfigs
|
||||
- git status --short
|
||||
- find . -path ./.git -prune -o -name '*.container' -print | sort
|
||||
|
||||
2. Identify host and unit style:
|
||||
- Dankclaw: 192.168.0.143
|
||||
- Dankserver: 192.168.0.235
|
||||
- Most services are rootless Quadlet user units: systemctl --user <action> <name>.service
|
||||
- Some model services may be system units or custom scripts; inspect repo before acting.
|
||||
|
||||
3. Inventory a host:
|
||||
- ssh HOST 'systemctl --user list-units --type=service; podman ps -a --format "{{.Names}} {{.Status}} nets=[{{.Networks}}]"'
|
||||
- For a suspected unit: ssh HOST 'systemctl --user status UNIT.service --no-pager; journalctl --user -u UNIT.service -n 120 --no-pager'
|
||||
|
||||
4. For port/listener issues:
|
||||
- ssh HOST 'ss -lntp 2>/dev/null | tr -s " "'
|
||||
- Match published ports to Quadlet PublishPort and podman ps networks.
|
||||
|
||||
5. For deploy issues:
|
||||
- Review scripts/deploy.sh behavior in current repo.
|
||||
- Confirm target checkout is clean or intentionally stashed by deploy workflow.
|
||||
- Do not hand-edit generated systemd output; fix Quadlet/source files.
|
||||
- After Quadlet changes run daemon reload via deployer or systemctl --user daemon-reload, then inspect generated unit with systemctl --user cat.
|
||||
|
||||
6. Do not commit secrets. Machine-local env files are the source for Caddy/Porkbun, app passwords, API keys, OAuth creds, phone numbers, sessions, and live generated state.
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
# Quadlet media migration notes
|
||||
|
||||
Use this when migrating or repairing media services on Dankserver or standardizing remaining compose services.
|
||||
|
||||
General rules:
|
||||
- Prefer Quadlet .container units for boot persistence.
|
||||
- Use bind mounts only for durable state.
|
||||
- Linux data root standard: /home/dank/Containers/<svc>, unless a service intentionally uses an in-place legacy root.
|
||||
- Use UserNS=host in Quadlet when host UID/GID behavior matters; do not use compose-style UsernsMode=.
|
||||
- Stop/mask old native units or disable old snaps only after verifying rollback path.
|
||||
|
||||
Plex/Tautulli known-good design:
|
||||
- Plex image: lscr.io/linuxserver/plex:1.43.3.10896-cb3ebc72d-ls322.
|
||||
- Tautulli image: lscr.io/linuxserver/tautulli:v2.18.1-ls242.
|
||||
- Network: dankserver_appnet, aliases plex and tautulli.
|
||||
- Ports: Plex 32400:32400, Tautulli 8181:8181.
|
||||
- Plex volumes: /var/lib/plexmediaserver:/config rw and /mnt/entropy:/mnt/entropy:ro.
|
||||
- Tautulli volume: /home/dank/Containers/tautulli:/config.
|
||||
- Plex rootless mapping: UserNS=host, PUID=0, PGID=0, data root chowned to 1000:1000. Rollback chown is 997:984.
|
||||
- Tautulli: PUID=1000, PGID=1000.
|
||||
- Tautulli seed: PMS_Web_Host=plex, PMS_Web_Port=32400, PMS_Secure=0.
|
||||
- Plex health gate should be generous, around 600 seconds, because large DB migrations can cold-load slowly.
|
||||
|
||||
Copyparty/Jellyfin caution:
|
||||
- User directive after verification: stop touching Jellyfin/Copyparty unless explicitly asked.
|
||||
- Jellyfin mount verified: /mnt/entropy:/data:ro. DB table is BaseItems.
|
||||
- Copyparty must be read-only for broad web/LAN access. Bare A in copyparty means everyone full access. Read-only permission shape is r for anonymous and ra,dank for admin browsing, without write/move/delete perms.
|
||||
|
||||
Before decommission:
|
||||
- Do not apt purge plexmediaserver or snap remove tautulli without explicit sign-off.
|
||||
- Keep rollback steps documented before deleting old service data or packages.
|
||||
-1
@@ -1 +0,0 @@
|
||||
Orchestrate llama.cpp benchmarking on Dankclaw (gfx1151) using frank-default-large. Follow strict env constraints, health-check polling patterns, and resource isolation. SSH from controller, use frank-default-small as driver, poll health endpoints with 5+ checks at ~5s intervals, and apply env file tunings before starting throughput tests.
|
||||
Reference in New Issue
Block a user