refactor(skills): consolidate managed skill library 94 -> 25

The managed-skill library was regenerated: 94 narrow per-incident skills
replaced by 25 broader ones (radiance-*, llamacpp-*, homepage-*, fleet/
endpoint ops). Git renders this as mass delete + mass add because whole
directories changed name; old content stays in history at 12c5e88.

Surface check: the 25 on disk are exactly the skills the harness advertises
this session.

agent/managed-skills/amd-multigpu-cooperation-triage/SKILL.md

@@ -1,71 +0,0 @@
----
-name: amd-multigpu-cooperation-triage
-description: "Decide whether two (or more) AMD GPUs on a host can work together on one model, or must run as independent single-GPU replicas, by checking PCIe atomic-op routing on the full path, the runtime's collective needs, and the non-collective alternatives."
----
-
-# AMD Multi-GPU Cooperation Triage
-
-When someone wants N GPUs "working together" on one model (TP/PP), decide *feasibility first* — the answer on consumer AMD platforms is often "no, and here is the $30 fix".
-
-## 1. Check atomic-op routing on the whole path
-
-RCCL collective and P2P kernels carry a hidden **hostcall buffer** parameter. ROCr only supplies it when the device reports PCIe atomics, which is read **once at driver probe** from a *queried hardware property*:
-
-```cpp
-void Device::checkAtomicSupport() {                      // rocclr/device/rocm/rocdevice.cpp
-  link_attrs.push_back({LinkAttribute::kLinkAtomicSupport, 0});
-  if (findLinkInfo(cpu_agent_info_->fine_grain_pool, &link_attrs))
-    if (link_attrs[0].second == 1) info_.pcie_atomics_ = true;   // no else, no override
-}
-```
-
-```bash
-# Endpoint capability + negotiated link, and EVERY upstream bridge
-for d in <gpu-btfs>; do sudo lspci -vv -s $d | grep -E 'LnkCap:|LnkSta:|AtomicOpsCap|AtomicOpsCtl'; done
-lspci -tv            # is the card on a CPU root port or behind the chipset?
-```
-
-- Endpoint must show `AtomicOpsCtl: ReqEn+`.
-- Every hop to the CPU must show `AtomicOpsCap: Routing+`. `Routing-` is a **read-only capability** — no software can enable it, and `setpci ... CAP_EXP+28.w=0440` does not help because the driver already consumed the value at probe.
-
-Failure signature (never names atomics): `AMD_LOG_LEVEL=3` → `Pcie atomics not enabled, hostcall not supported` → `AQL dispatch failed!` → `hipLaunchKernel: hipErrorIllegalState` on `ncclDevKernel_Generic_4`.
-
-## 2. Map the board before recommending anything
-
-Read the **vendor's techspec page**, not a review: which slots are CPU-attached vs chipset, each slot's *electrical* width (a "x16 slot" can be x1 mode), and which M.2 slots the CPU owns. Free CPU-attached M.2 slots are gold: an M.2→PCIe x4 riser puts a card on the CPU root complex for ~$30 instead of a ~$300 x8/x8 board swap. Bandwidth needed for TP allreduce is small: hidden_states ≈ 10 KB per token at batch 1, ~10 MB per token-step for a 27B at batch 4 → ~1.4 ms at Gen4 x4.
-
-## 3. Prove what the runtime actually needs (do it OFF the production host)
-
-Bare 2-rank probe, one device per rank, with a short PG timeout so a walled op raises instead of hanging:
-
-```python
-torch.distributed.init_process_group("nccl", init_method="tcp://127.0.0.1:PORT",
-    world_size=2, rank=r, timeout=timedelta(seconds=25))
-# probe: broadcast, all_reduce, barrier, send/recv, all_gather
-```
-
-- Expect the chipset-attached rank to fail **every** op — including `send/recv`, which is what pipeline parallelism uses, so PP=2 is not an escape from a TP wall.
-- `world_size=1` always succeeds; one-card success proves nothing.
-- Run the probe with a *real file* (`torch.multiprocessing.spawn` re-executes it — piping a script into `python3 -` fails with `FileNotFoundError: '/<stdin>'`).
-- **Stop and mask the serving unit first** (`systemctl stop X && systemctl mask X`; mask, never disable). A hung dispatch can wedge the GPU driver and take the host down — this has happened.
-
-## 4. Don't be fooled by "collective-free" claims
-
-Check the gate, not the feature list. vLLM's IPC custom allreduce (`custom_ar=True` when the ops are compiled) is gated by a **CUDA-sm-keyed table** (`CUSTOM_ALL_REDUCE_MAX_SIZES = {'9.0','10.0','10.3','10.7'}`) while ROCm reports e.g. `DeviceCapability(major=12, minor=0)` — so it is off on RDNA4 by a *code* limit, and `current_platform.is_rocm()` exclusions appear nearby. Even patching it is insufficient: TP init's `broadcast`/`barrier`/`all_gather` have no non-RCCL path and die the same way. **No RCCL kernel involving the atomics-less card can launch** ⇒ anything built on RCCL (TP, PP, P/D KV transfer) is out.
-
-## 5. The option set, ranked
-
-| Option | Works when | Cost | Trade |
-|---|---|---|---|
-| CPU-attached M.2 (x4) riser | a CPU-lane M.2 slot is free | ~$30 | TP=2 with x4 bandwidth — sufficient |
-| CPU x8/x8 board | board supports CPU bifurcation | ~$300+ | best bandwidth, no riser |
-| llama.cpp/ik_llama.cpp `--split-mode layer` | always (no collectives) | $0 | GGUF model; loses paged attention, continuous batching, spec decode |
-| Independent TP=1 replicas + router | always | $0 | cards never share one model; best throughput today |
-| Patch the custom-AR table | never | — | other collectives still die |
-| P/D disaggregation | ROCm KV-transfer connector + collectives | — | out on this class of host |
-
-## Pitfalls
-
-- A one-card collective test proves nothing (NCCL short-circuits at `world_size=1`).
-- Validation must come from the peer/host doing the work: probe reachability with `ip neigh show <ip>` — `FAILED`/`INCOMPLETE` means off or NIC dead, whereas a wedged kernel usually still answers ARP.
-- Validate your own probe's naming before reporting a service outage (e.g. `systemctl --user is-active gitea` vs the real unit `gitea-server`).

agent/managed-skills/anubis-guard-chain-flip/SKILL.md

@@ -1,25 +0,0 @@
----
-name: anubis-guard-chain-flip
-description: "Runbook for moving a Caddy-fronted service behind the Anubis + Authentik embedded-outpost guard chain on Dankclaw (provider-first ordering, validate gate, verification probes)."
----
-
-# Anubis + Authentik guard-chain flip (Dankclaw)
-
-Move a public vhost from `reverse_proxy <app>:<port>` onto: client → TLS edge → Anubis challenge (:8923) → Authentik embedded outpost decision → return to internal `http://:8017` listener → Host-matched handler → real upstream.
-
-## Order (never reorder)
-1. **Provider first.** Create the outpost ProxyProvider before touching routes, or authed traffic hits the outpost error page instead of the app:
-   - Write a Python snippet to a local /tmp file that imports `authentik.providers.proxy.models.ProxyProvider`, constructs it with `name`, `external_host="https://<vhost>/"` (exact, trailing slash), `internal_host="http://caddy:8017"`, sets `p.pk = None`, then `p.save()`.
-   - Pipe raw: `ssh -o BatchMode=yes dankclaw 'podman exec -i authentik-server ak shell' < /tmp/x.py`. Heredoc quoting through ssh SyntaxErrors; the raw-stdin pattern is the proven one (ak shell does `exec(sys.stdin.read())`).
-2. **Route flip in ONE commit/deploy**: vhost becomes `reverse_proxy anubis:8923 { header_up X-Real-Ip {remote_host} }`; inside the `http://:8017` listener add `@<svc> host <vhost>` + `handle @<svc> { reverse_proxy <upstream> }`.
-3. Deploy via the repo deploy flow (commit first; deploy pushes main itself).
-
-## Verification (all three)
-- Unauthenticated probe from client: `curl -s https://<vhost>/` → HTTP 200 challenge page, grep body for challenge markers (`challenge|not a bot|anubis`). A 502 here instead of the challenge = chain mis-wired.
-- Post-auth routing proof: grep the LIVE config inside the container — `podman exec caddy sh -c "grep -n -A3 <handler> /etc/caddy/Caddyfile"` — after the deploy prints its reload line; confirm the reload actually ran, don't trust exit 0.
-- Outpost reachability: `podman exec authentik-server getent hosts caddy` resolves the compose DNS alias (10.89.1.x) that internal_host dials.
-
-## Pitfalls
-- Probing `http://localhost:8017/` from the host returns HTTP 000 (connection refused): the post-auth listener lives in the caddy container's netns and is not bound on host loopback. Probe inside the netns or grep the config instead.
-- Non-browser clients cannot solve the JS challenge. Before flipping, audit service-to-service traffic: ingestion/push paths (e.g. Alloy → `loki:3100`) must target internal addresses; any backend datasource URL pointing at a public hostname of a guarded service breaks the moment you flip.
-- Exempt-by-construction categories that must stay unguarded: the IdP itself (`auth.` — its login flow can't sit behind a challenge), git smart-HTTP endpoints (Gitea git clients), services with their own complete auth. Document the exemption in the vhost comment.

agent/managed-skills/arr-auth-posture-probe/SKILL.md

@@ -1,20 +0,0 @@
----
-name: arr-auth-posture-probe
-description: "Verify a self-hosted *arr-style service's real auth posture by behavioral API probes instead of config-flag inference; includes the sed-masks-a-miss pitfall and wrong-host probe pitfall"
----
-
-# Arr-auth posture probe
-
-Ground truth for "is this arr/gate open?" comes from behavior, not config flags. `AuthenticationMethod=None` + `AuthenticationRequired=Enabled` in `config.xml` is ambiguous on paper; on the wire it means a login screen with **zero users** — first-account-wins.
-
-## Recipe
-1. Confirm which host runs the service before probing (`podman ps` on the target, not the box you ssh'd into first — arrs live on Dankserver, chat/searxng on Dankclaw).
-2. Probe an API endpoint that bypasses the SPA shell, locally and through the vhost:
-   `curl -s -o /dev/null -w "%{http_code}" http://<host>:<port>/api/v3/system/status`
-   - SPA roots (`/`) always serve a 200 shell — never conclude "open" or "gated" from `/`.
-   - 401 = gated; 200 + JSON = API fully open (anyone on the net can drive it with curl).
-3. For config truth, dump the whole file (`podman exec <svc> cat /config/config.xml`), not a piped `grep | sed -n 2p`: **`sed -n Np` exits 0 on zero matches**, so the pipeline hides a miss and you conclude "empty field" when the grep itself failed.
-4. API keys from config.xml are local-only secrets; they stop being secret the moment a vhost serves the API unauthenticated.
-
-## Decision framing for the user
-When posture = gated-with-zero-users, present it as a decision menu (None behind the front-door wall vs External/authentikarr), not a silent change — first-account-wins means whoever logs in first owns the instance.

agent/managed-skills/arr-stack-api-config/SKILL.md

@@ -1,30 +0,0 @@
----
-name: arr-stack-api-config
-description: "Configure/download-client and indexer wiring for the *arr stack on Dankserver via live API calls — schema-first, test-before-enable; covers Sonarr/Radarr v3 vs Lidarr/Readarr v1 API split and Prowlarr v2 Applications/sync model."
----
-
-# Arr-stack API config (Dankserver)
-
-## API version split
-- Sonarr/Radarr: `/api/v3/...`; Lidarr/Readarr/Prowlarr: `/api/v1/...`.
-- `PUT /downloadclient` and `PUT /indexer/{id}` take a SINGLE resource object (GET it, patch `settings`, PUT whole object back). An array body fails with `Path: $ | BytePositionInLine: 1`.
-- API keys live in each container's config: `podman exec <svc> grep -o "<ApiKey>[^<]*" /config/config.xml | cut -d">" -f2`.
-
-## Schema-first probing (validators are free documentation)
-Send the FULL schema template object with fields filled; v1 validators reject partials:
-- `'Fields' must not be empty` -> send full schema copy, not `fields:[]`.
-- `'Base Url' must not be empty` when keys were camelCase -> PascalCase required (`BaseUrl`, `ApiKey`).
-- `targetType null` -> configContract not registered in that build (this Prowlarr build has no Prowlarr-plugin indexer; integration is Torznab-family only).
-- `testingPackageId` in a PUT body downgrades validation failures to warnings.
-
-## Prowlarr v2 model (inverted from v1)
-Indexers live ON THE ARR SIDE; Prowlarr syncs indexer configs via `ApplicationSync` commands (`POST /api/v1/command {"name":"ApplicationSync","body":{"applicationId":N}}`). Applications settings keys must be PascalCase or sync throws NullConfig. Torznab feed paths can answer with the SPA HTML shell when absent — verify by byte count (HTML shell ~1946 bytes, 1 ms) vs XML, not just status code.
-
-## DB as arbiter
-When endpoint shapes are opaque, stop the container and edit `/opt/prowlarr/config/prowlarr.db` with `sudo python3` sqlite directly, then start. Indexers table columns are Id/Name/Implementation/Settings/ConfigContract/Enable/Priority/Added — the enable flag column is `Enable` (not EnableRss).
-
-## Verification rules
-- `POST /downloadclient` returns the COLLECTION, not the created object — verify via GET.
-- Test endpoints return verdicts in the BODY at HTTP 200 (`{}` = pass, list = fail) — judge from bodies, not status codes.
-- `syncLevel` is enum ApplicationSyncLevel — omit or use int 2; string rejected.
-- Command POSTs (MoviesSearch etc.) run server-side and respond only when the work finishes: curl `-m` timeouts (HTTP 000) during search/grab are NOT failures; poll queue/history afterward instead of re-firing. Warm small commands (RefreshMonitoredDownloads) answer in ~20 ms — a hang means live work in progress, not a dead API.

agent/managed-skills/arr-v1-api-validator-peeling/SKILL.md

@@ -1,29 +0,0 @@
----
-name: arr-v1-api-validator-peeling
-description: "When driving Sonarr/Radarr (v3) vs Lidarr/Readarr/Prowlarr (v1 .NET) APIs over HTTP: probe endpoints by status code, peel validators one live call at a time, and treat config.xml on disk as truth — v1 rejects literal enum strings that break AuthorizationMiddleware."
----
-
-# *arr v1 API validator peeling
-
-Use when creating root folders, profiles, or settings via the *arr HTTP APIs, or when a v1 (.NET) arr 500-sprawls after a config edit/restart.
-
-## API version split
-- Sonarr/Radarr: `/api/v3/...`. Lidarr/Readarr/Prowlarr: `/api/v1/...` (Prowlarr port 9696, NOT 8090 — that's Jackett).
-- Probe endpoints by status code first: **empty body = endpoint absent**, not a failure. v1 has no `/api/v1/profile` endpoint at all.
-
-## Peel validators one live call at a time
-POST a minimal body and let the validator error name the next required field, instead of guessing payload shapes:
-- `DefaultMetadataProfileId must be >0` → that field is required; fetch real ids from `GET /api/v1/metadataprofile` (returns rows like `(1,'Standard')`).
-- `'Name' must not be empty` → include `name`.
-- `Quality Profile does not exist` → the id you sent doesn't exist; list real ones first (`GET /api/v1/qualitydefinition` returns the quality catalog, ids ≠ profiles).
-Then POST the full body with **existing** ids: `{"name":"music","path":"/music","defaultQualityProfileId":1,"defaultMetadataProfileId":1}`.
-
-## Auth-config pitfall (v1 .NET apps)
-- `<AuthenticationMethod>None</AuthenticationMethod>` + `<AuthenticationRequired>Enabled</AuthenticationRequired>` = self-contradictory: UI Save permanently disabled, and any save throws `DryIoc.ScopedItemException` in `AuthorizationMiddleware` → 500 on every page after restart.
-- The valid no-auth representation is **absence of the `<AuthenticationRequired>` key**, not the literal string `Disabled`. Fix = delete the key from `/config/config.xml` (`sed -i "/<AuthenticationRequired>/d"`), restart container.
-- Tell-tale: one v1 app boots clean while siblings 500-sprawl → diff their config.xml auth blocks; the healthy one shows the correct shape.
-
-## General rules
-- Container `State=running` ≠ app serving; read `podman logs --tail 12 <c>` before theorizing.
-- When a UI gate blocks saving, go under it: these apps keep plaintext XML config on disk (`/config/config.xml`) — read/edit it directly via `podman exec`.
-- API keys live in each container's `/config/config.xml` `<ApiKey>`; grep them live rather than trusting docs.

agent/managed-skills/authentik-claim-cache-purge/SKILL.md

@@ -1,38 +0,0 @@
----
-name: authentik-claim-cache-purge
-description: "Fix persistent 403s at Caddy RBAC header gates after authentik group changes: purge the embedded outpost's proxysession claim cache and verify silent re-auth"
----
-
-# Authentik outpost claim-cache purge
-
-Use when a user gets persistent 403s at a Caddy `header_regexp` RBAC gate (e.g. `/admin`) even though their authentik group membership is correct, and re-login did not help.
-
-## Mechanism (authentik 2026.8, this fleet)
-- The embedded outpost does NOT re-evaluate groups per request. Each completed authorize writes one row to `authentik_providers_proxy_proxysession`; `session_data` JSON holds the claims (incl. `groups`) served for that browser session until the row expires (observed TTLs 1–24 h per row).
-- A session minted before the group add keeps serving the old group list → gate 403s forever until expiry.
-- The 403 happens at Caddy and generates ZERO authentik events — silence in `authentik_events_event` is expected.
-- Re-login does not fix it: the browser's outpost cookie re-maps to the same stale row.
-- `proxysession.user_id` is null; map rows to users via `session_data->'claims'->>'preferred_username'`.
-
-## Diagnosis (run on dankclaw)
-```bash
-podman exec authentik-postgres psql -U authentik -d authentik -At -c "SELECT session_data->'claims'->>'preferred_username' AS user, session_data->'claims'->>'groups' AS cached_groups, expires FROM authentik_providers_proxy_proxysession ORDER BY expires;"
-```
-Stale row = `cached_groups` missing the new group while repo/Caddy gate expects it.
-
-## Fix
-```bash
-podman exec authentik-postgres psql -U authentik -d authentik -At -c "WITH d AS (DELETE FROM authentik_providers_proxy_proxysession WHERE session_data->'claims'->>'preferred_username' IN ('<user>') RETURNING 1) SELECT 'deleted='||count(*) FROM d;"
-```
-- The user's authentik django login session is untouched → next visit re-authorizes SILENTLY (no password prompt) with fresh claims.
-- NEVER rotate `AUTHENTIK_SECRET_KEY` for this — it invalidates every outpost session fleet-wide.
-
-## Verification without the user's password
-Simulate their browser: delete a TEST user's proxysession rows (e.g. `qa-e2e-admin`), keep its e2e storage state on disk, then run the admin spec bypassing setup (setup ALWAYS fresh-logins, which would mask the path):
-```bash
-cd tests/e2e && npx playwright test --project=admin --no-deps
-```
-Passing with no login prompt = silent re-auth proven. `--no-deps` is required because project dependencies would rerun setup.
-
-## Doc
-Mechanism + SQL live in `serverAppConfigs-docs/Infrastructure/Authentik.md`, section "Outpost claim cache (proxysession)" (commit ea0bba3).

agent/managed-skills/authentik-flow-ui-automation/SKILL.md

@@ -1,46 +0,0 @@
----
-name: authentik-flow-ui-automation
-description: Drive the authentik two-stage login through real browser UI for E2E tests on Anubis+outpost-guarded routes; hardened against the Lit stage re-render race and Anubis PoW budgets
----
-
-Drive authentik's two-stage login (identification → password) through a real browser UI for E2E tests on routes guarded by the Anubis + Authentik chain (Caddy → Anubis :8923 → outpost :9000). Proven in `tests/e2e/lib/login.ts` (`akLogin`) against authentik 2026.x.
-
-## Hard rules
-
-1. **Fresh context per login.** Never reuse a context or page across logins, and never retry a failed login inside the same context — a half-finished flow poisons authentik's `pending_user` stage and the next login hangs on a "Continue as <user>" interstitial that never completes. Each login = new `browser.newContext()`.
-2. **Real Chrome UA.** Anubis denies obvious headless UAs; set `userAgent` to a real Chrome UA in `use:` (config-level) or `launchOptions`. Headless chromium + REAL_UA solves the PoW fine.
-3. **Shadow-DOM locators.** authentik stages are Lit web components; locate inputs with `page.locator('input[name=username]')` etc. (Playwright pierces shadow roots) — never `page.$`.
-
-## The Lit re-render race (the big one)
-
-The flow executor re-renders a stage right after mount. A programmatic `fill()` can succeed and then be WIPED by the re-render; the submit then POSTs an empty form ("This field is required") and `waitForURL` times out while the page still shows the password stage with an empty field.
-
-Hardened sequence per stage:
-
-```ts
-for (let attempt = 1; attempt <= 3; attempt++) {
-  await page.fill('input[name=password]', pass);
-  // VERIFY the value survived the re-render before submitting
-  if ((await page.inputValue('input[name=password]')) !== pass) continue;
-  await page.keyboard.press('Enter');   // native form path; avoids button re-render races
-  try {
-    await page.waitForURL(u => !u.pathname.includes('/if/flow/'), { timeout: 30_000 });
-    return;                              // past this stage
-  } catch { /* maybe wiped; retry */ }
-}
-throw new Error(`akLogin: stage never advanced after 3 attempts`);
-```
-
-- Submit with **Enter**, not a button click — the button can be re-created between snapshot and click.
-- Verify-after-fill is the load-bearing step; without it you silently submit empty forms.
-- The identification stage has the same race; apply the same fill-verify-Enter there.
-
-## Anubis PoW budgets
-
-The first guarded-vhost hit in a fresh context runs an Anubis proof-of-work; on a cold/loaded box it can outlive default assertion budgets. Give title/URL asserts after `goto` 60 s budgets. With `COOKIE_DOMAIN` set fleet-wide (Anubis ≥ v1.27), every later guarded vhost in the same context skips the challenge (~0.5 s per vhost in the crawl specs).
-
-## See also
-
-- `serverAppConfigs-docs/Architecture/E2E Suite.md` — project layout, credentials model (`tests/e2e/.env.local`, git-ignored), run commands, gotcha list.
-- `serverAppConfigs-docs/Infrastructure/Anubis.md` — chain behavior, v1.27 gotchas (X-Real-Ip mandatory, DENY = HTTP 200 body).
-- `arr-auth-posture-probe` for non-browser API-posture checks of guarded routes.

agent/managed-skills/authentik-outpost-verification/SKILL.md

@@ -1,34 +0,0 @@
----
-name: authentik-outpost-verification
-description: "Verify authentik embedded-outpost and proxy-provider state on a live host by probing from inside the container network namespace; covers meta-header routing, rootless podman observation artifacts, and 2026.x schema quirks (renamed columns, ak shell swallowing piped stdin)."
----
-
-# Authentik outpost/provider verification (live host)
-
-## When to use
-Guarded apps behind Anubis+authentik return an error page after authentication, or you need ground truth on outpost/provider state on a host running authentik 2026.x with podman quadlet units.
-
-## Core insight: probe from inside the network namespace
-- Rootless podman gives each container its own netns; **host `ss -ltn` cannot see listeners inside other containers' namespaces**. A missing listener on the host is an observation artifact, not a fault — verify by dialing *from inside* the namespace.
-- Working probe (distroless images have no shell, so `podman exec sh` fails):
-  ```
-  podman run --rm --network container:<container> --entrypoint /bin/busybox docker.io/library/busybox:latest \
-    wget -q -T 4 -t 1 -O /dev/null -S "<url>" 2>&1 | head -8
-  ```
-- Quote URLs over ssh: the remote shell is zsh and bare `?` in query strings triggers globbing (`no matches found`).
-
-## Meta-header routing truth
-- The outpost routes by **meta-header middleware**, not URL-path lookup. A request to `/outpost.goauthentik.io/login` **without** `X-authentik-meta-outpost/-provider/-app` headers is a path that genuinely does not exist → a 404 there is *correct behavior*, not a fault.
-- Probe the auth hop exactly as Caddy sends it: full meta-header set (`X-authentik-meta-outpost: <uuid>`, `X-authentik-meta-provider: <pk>`, `X-authentik-meta-app: name=<app>`). If a full-meta-header request still 404s, the outpost's URL space is not mounted in the answering process — that is the real failure point, and it means the transport is aimed at a URL space that doesn't exist there (embedded outposts mount their URL space at `authentik_host`; check the outpost row's `_config.authentik_host` before theorizing).
-- Boot transience: immediately after a restart the same probe can return 503 (configure task pending); poll until the answer changes before concluding.
-
-## Schema quirks in 2026.x
-- `ak shell` is interactive-only; piped stdin is swallowed (prints import banner, discards output). For live DB truth use:
-  ```
-  podman exec authentik-postgres psql -U authentik -d authentik -x -c "select * from authentik_outposts_outpost"
-  ```
-- Columns are renamed vs older versions: `Provider.objects.filter(type=...)` raises FieldError (type moved to subclass tables); the outpost `enabled` column may not exist under that name; provider pk is `oauth2provider_ptr_id`. **Query with `select *` and stop guessing column names.**
-- `podman inspect --format "{{.Config.NetMode}}"` errors on this podman version — the field is `.HostConfig.NetworkMode`.
-
-## Decision fork when the auth hop 404s
-A) point the guarded apps' auth hop at where auth actually lives (the vhost matching `authentik_host`), or B) flip the outpost to standalone (own container, own URL space, new quadlet unit + `AUTHENTIK_TOKEN`). This is an architecture decision for the user — present both with blast radius, don't pick silently.

agent/managed-skills/authentik-provision-fleet-user/SKILL.md

@@ -0,0 +1,42 @@
+---
+name: authentik-provision-fleet-user
+description: "Provision or audit an Authentik user on Dankclaw (auth.franksdank.cloud): ak shell ORM drive, group set, behavioral login verify, stale-claim caveat."
+---
+
+# Provision an Authentik user on Dankclaw
+
+Source of truth: `serverAppConfigs-docs/Infrastructure/Authentik.md` ("User provisioning playbook"). Authentik objects are NOT repo-managed — no deploy, no file edits.
+
+1. Inspect first, mirror the newest human user's groups:
+```bash
+ssh dankclaw 'podman exec -i authentik-server ak shell' <<'EOF' 2>&1 | sed -n '/objects imported/,$p'
+from authentik.core.models import User, Group
+for u in User.objects.all().order_by("username"):
+    print(u.username, u.email, sorted(u.groups.values_list("name", flat=True)))
+print(sorted(Group.objects.values_list("name", flat=True)))
+EOF
+```
+2. Create (idempotent guard; password goes in the heredoc = stdin, never argv, so it never lands in `ps`):
+```python
+from authentik.core.models import User, Group
+u = User.objects.filter(username="X").first()
+if not u:
+    u = User(username="X", email="X@franksdank.cloud", name="X", is_active=True)
+    u.set_password("...")
+    u.save()
+for g in ("Non-Admin Users", "jellyfin_users", "openwebui_users"):  # human-user precedent: ridley
+    u.groups.add(Group.objects.get(name=g))
+```
+3. Verify through the exact PasswordStage code path:
+```python
+from django.contrib.auth import authenticate
+assert authenticate(request=None, username="X", password="...")
+assert authenticate(request=None, username="X", password="nope") is None
+```
+
+## Rules
+- No `User.create_user`; `is_superuser` is a group-derived property, not a constructor kwarg (raises `TypeError`). `path` defaults to `users`.
+- Guarded-vhost access = any authenticated user (all proxy providers use implicit-consent); groups are consumed downstream: `homepage-admins` (Caddy `/admin` header gate), `openwebui_users`/`_admins` (roles claim), `jellyfin_users`/`_admins` (LDAP outpost). EXCLUDE admin groups from a plain user; ask before granting `homepage-admins`.
+- Group add after a login → Caddy RBAC can 403 on the outpost's cached claims; purge that user's rows: `DELETE FROM authentik_providers_proxy_proxysession WHERE session_data->'claims'->>'preferred_username' = 'X';` (in the `authentik-postgres` container) — next visit re-authorizes silently.
+- Jellyfin visibility rides the LDAP outpost cache; expect a lag.
+- Look up flows by slug, not name. Do not rotate `AUTHENTIK_SECRET_KEY` to fix claim issues.

agent/managed-skills/authentik-proxy-provider-app-link/SKILL.md

@@ -1,52 +0,0 @@
----
-name: authentik-proxy-provider-app-link
-description: "Diagnose and fix an authentik embedded-outpost proxy provider that never authenticates (login wall / raw flow served on the app host): the provider must be linked to an Application, have a non-null invalidation_flow, and an exact external_host match."
----
-
-# authentik proxy provider not authenticating (login wall)
-
-Symptom: a service behind the authentik embedded outpost (Caddy -> Anubis -> outpost -> app) never lets an authenticated user through. The browser is parked on a login page served ON THE APP'S OWN HOST, e.g. `https://app.example.com/if/flow/default-authentication-flow/?next=/` or `https://app.example.com/flows/-/default/authentication/`. The IdP session cookie lives on `auth.example.com` and is never sent to `app.example.com`, so login is demanded forever.
-
-## Root causes (check in this order)
-
-1. **Provider not linked to an Application (most common).** The embedded outpost's config API filters `ProxyProvider.objects.filter(application__isnull=False)`. A provider created with `ProxyProvider.objects.create(...)` and no Application is never returned to the outpost, so the outpost never loads it and the request falls through to raw flow serving.
-   - FIX: create an Application bound to the provider:
-     ```python
-     app = Application.objects.create(name="Marreta", slug="marreta", launch_url="https://marreta.franksdank.cloud")
-     app.provider = proxy_provider; app.save()
-     proxy_provider.application = app; proxy_provider.save()
-     ```
-2. **Null `invalidation_flow`.** The Rust outpost serde requires a formatted UUID; a null value aborts `get_outpost` with `invalid type: null, expected a formatted UUID string at line 1 column N` and takes down EVERY provider on that outpost.
-   - FIX: `proxy_provider.invalidation_flow = InvalidationFlow.objects.get(name="default-provider-invalidation-flow")`.
-3. **`external_host` mismatch.** Must equal the Host header the outpost sees. Trailing-slash mismatch (`https://x` vs `https://x/`) = no match. Match the fleet convention (no trailing slash).
-4. **Missing `authentication_flow` / `authorization_flow`.** Set to the fleet defaults (`default-authentication-flow`, `default-provider-authorization-implicit-consent`).
-
-## Diagnose
-
-```bash
-# 1. Compare broken vs working provider field set (ak shell, as the authentik user):
-sudo -u authentik AUTHENTIK_POSTGRESQL_HOST=127.0.0.1 AUTHENTIK_POSTGRESQL_PORT=5432 \
-  AUTHENTIK_POSTGRESQL_NAME=authentik AUTHENTIK_POSTGRESQL_USER=authentik \
-  AUTHENTIK_POSTGRESQL_PASSWORD='<pw>' ak shell
-```
-```python
-for p in ProxyProvider.objects.all():
-    print(p.name, p.external_host, bool(p.application), p.authentication_flow_id, p.authorization_flow_id, p.invalidation_flow_id)
-```
-```bash
-# 2. The serde error (this is the smoking gun for cause #2):
-journalctl --user -u authentik-outpost.service --since '-10 min' | grep -iE 'error|invalid|panic'
-# 3. Confirm the outpost loaded the provider after the fix:
-journalctl --user -u authentik-outpost.service --since '-2 min' | grep -i 'loaded app'
-```
-
-## Apply + restart
-
-After any provider/application change: `systemctl --user restart authentik-server.service authentik-outpost.service` (rootless, user dank on Dankclaw). Then verify the OAuth dance:
-```bash
-curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' https://app.example.com/
-# expect 302 -> /outpost.goauthentik.io/start?rd=... (NOT a raw /flows/ URL)
-```
-
-## Provenance note
-On the Dank fleet these providers are NOT repo-managed (no blueprint). Fixes applied via `ak shell` are live-only. Capture any new provider+application as a repo blueprint or setup script to avoid drift.

agent/managed-skills/autoresearch-llm-bench-window/SKILL.md

@@ -0,0 +1,76 @@
+---
+name: autoresearch-llm-bench-window
+description: "How to run an iterative config-tuning window on a Dank-fleet LLM endpoint: harness pattern, noise-floor-first discipline, bracket mapping, and the frank-primary laws measured in the 2026-10-06 segment."
+---
+
+# Autoresearch bench window for a fleet LLM endpoint
+
+Use when running (or re-running) an iterative config-tuning segment against a
+Dank-fleet LLM endpoint (frank-primary/frank-large/frank-small): build the
+harness, establish the noise floor, then bracket-map one flag at a time.
+
+## Harness pattern (already built for frank-primary — reuse, don't rebuild)
+
+Committed at `autoresearch.sh` (dev box) + `scripts/autoresearch-frank-primary.sh`
+(dankserver) in serverAppConfigs. The pattern generalizes to any fleet endpoint:
+
+1. Dev-box wrapper scp's the candidate env file + bench script to the target
+   host, then ssh-runs the bench. Never push the candidate config to origin
+   (the scp dirties the host checkout on purpose).
+2. Host script, in order: stop the production unit (relinked units: `systemctl
+   stop`, NEVER mask — mask refuses a symlinked unit file) → boot a scratch
+   container (distinct name, e.g. `arbench`) on the SAME devices with the
+   candidate env — mirror the production podman contract exactly, differing
+   only in container name and a FRESH prefix-cache tier (wiped per run) so cold
+   measurements stay cold → poll the health endpoint until it answers (the
+   listener binds AFTER the full load, so a bare curl is honest; allow ~2-3 min
+   cold boot, 720s timeout) → run ONE self-contained deterministic request →
+   parse the engine's timings object into METRIC lines → tear down the scratch →
+   restore production.
+3. The restore MUST be an EXIT trap (`restore() { podman rm -f scratch;
+   systemctl start <unit>; }`) so a failed bench never leaves the endpoint
+   down. Verify after EVERY run: `systemctl is-active`, health curl until it
+   returns JSON (200 right after start is NOT enough — it can bind 60-90s
+   later), and no scratch leftover.
+4. Bench request: fixed prompt built from a sorted repo-corpus glob with a
+   fixed random seed (deterministic → stable prompt_n), greedy sampling
+   (temperature 0) for a fixed decode length, sent locally (127.0.0.1) to
+   remove LAN jitter. Read prefill/decode/draft-acceptance from the engine's
+   own timings object — never roll a custom bench without asking.
+5. Scratch containers must die by teardown; never leave a script whose tail
+   stops/removes a production unit (the stale-probe-tail lesson).
+
+## Discipline (the laws that make the numbers mean anything)
+
+1. **Noise floor FIRST.** Before iterating, replicate an identical config 2-3x
+   and treat that spread as the floor. Fleet endpoints here: decode ±3%,
+   prefill ±2%. A single-run "win" under the floor is NOT a win — this exact
+   mistake cost a false 33.06 keep in the 2026-10-06 segment.
+2. **One flag change per iteration**; log every run honestly (keep/discard/
+   crash); discards auto-revert the worktree.
+3. **Bracket-map, don't climb.** Test both directions of every lever around
+   its deployed value, INCLUDING boot failures — crashes are envelope data
+   ("0.82/2048/wht6/40960 is the seqs=3-viable quartet" came from four crash
+   walls). Test downward too: 40960 MiB host pool is a floor, not slack.
+4. **Repro before recording a mechanism.** An anomalous result next to an
+   abandoned bench may be contamination (boot race). Re-run clean once; only
+   then name a law. The MNBT 2560 collapse (2.6 t/s) was reproduced clean to
+   rule this out.
+5. **Watch for truncation.** predicted_n < max_tokens means the generation
+   died mid-run — that config is pathological regardless of the rate.
+6. **When the sweep ties, tie-break on real-traffic properties** (e.g. draft
+   acceptance for multi-turn workloads), never on probe artifacts, and prefer
+   the config that matches the documented runbook.
+
+## frank-primary specifics (2026-10-06 segment, commit c80c250)
+
+- MNBT=2048 is a RESONANCE POINT at MTP-on shapes, not an optimum: 2560
+  collapses decode to 2.6-7.7 t/s (acceptance normal, verify stalls ~130ms/step,
+  truncated output), 1536 truncates at 57/256, 4096 won't boot at seqs=3. Do
+  not re-test.
+- HOST_POOL_MIB 40960 is the load-bearing floor (36864 crashes boot).
+- Spec depth 1-3 and seqs 2-3 all tie within noise; 0 and 4 are real
+  regressions (29.57 / 24.57 t/s vs 31.72 baseline).
+- Levers known flat: GPU_HEADROOM, HOST_POOL upward, maxlen above 200k.
+- Final kept config = Frank's original (spec=3, seqs=2, maxlen=200k, wht6,
+  0.82, 2048, 40960, fp8 KV, TP=2), baseline decode 31.72 t/s at 43.6k depth.

agent/managed-skills/bench-llama-server-params/SKILL.md

@@ -1,69 +0,0 @@
----
-name: bench-llama-server-params
-description: "Benchmark llama.cpp server-side parameters (--threads-batch, --ubatch-size, --spec-type) using HTTP API with max_tokens=1 for prefill, max_tokens=16384+ for decode"
----
-
-# Benchmarking llama.cpp Server-Side Parameters
-
-## Problem
-`llama-bench` benchmarks the bench binary itself, not the running server.
-It cannot measure the effect of server-side parameters like `--threads-batch`,
-`--ubatch-size`, `--spec-type`, or `--parallel`.
-
-## Solution: HTTP-Based Benchmarking
-
-### Prefill throughput (prompt processing)
-1. Start the server with the target parameter value
-2. Send a large prompt via the chat API with `max_tokens=1`
-3. Measure `prompt_tokens / wall_time` from the response usage
-
-`max_tokens=1` is correct here — you want only prompt processing time,
-not generation. The server crashes on `max_tokens=0`.
-
-### Decode throughput (generation)
-1. Start the server with the target parameter value
-2. Send a prompt that generates a long response
-3. Use `max_tokens=16384` minimum (16k) — **never `max_tokens=1`** for decode
-   benchmarking. One token tells you nothing about sustained generation rate.
-4. Measure `predicted_per_second` from the response `timings` object, or
-   compute `completion_tokens / wall_time`
-
-### Why at least 16k tokens for decode
-- Decode throughput varies over the generation (warmup, cache growth, MoE
-  routing). A single token is noise.
-- Speculative decoding acceptance rates stabilize over thousands of tokens.
-- Short generations finish before the draft head reaches steady state.
-- 16k tokens gives a stable average at ~22 tok/s (≈12 min per run).
-
-### Key Pattern
-```python
-# Decode benchmark — measure generation throughput
-body = json.dumps({
-    "model": "<alias>",
-    "messages": [{"role": "user", "content": "<prompt that generates 10k+ tokens>"}],
-    "max_tokens": 16384,   # 16k minimum for decode; use max_tokens=1 ONLY for prefill
-    "temperature": 0,     # deterministic for reproducibility
-    "stream": False
-})
-# Measure wall time from request send to response receipt
-# Get predicted_per_second from timings, or completion_tokens / wall_time
-```
-
-### When to Use
-- `--threads-batch` (prefill): use `max_tokens=1`
-- `--ubatch-size` (decode): use `max_tokens=16384+`
-- `--spec-type` / speculative decoding: use `max_tokens=16384+` (decode only)
-- Any server-side parameter that `llama-bench` cannot target
-
-### Benchmarking policy (MANDATORY)
-1. **STOP the production service** before running any benchmark
-2. **Mask** the service (`systemctl mask`, never `disable` — on these hosts
-   `disable` deletes the unit file from `/etc/systemd/system/`)
-3. Run the bench
-4. **Unmask + restart** the service when done
-5. **Verify health** before reporting results
-
-### Example Scripts
-- `scripts/bench-threads-batch-http.py` — prefill sweep (max_tokens=1)
-- `scripts/bench-decode-threads-batch.py` — decode sweep (max_tokens=256+)
-- `scripts/bench-spec-decode.py` — speculative decoding config sweep

agent/managed-skills/closed-engine-runtime-investigation/SKILL.md

@@ -0,0 +1,29 @@
+---
+name: closed-engine-runtime-investigation
+description: "Recipe for diagnosing/tuning a closed-source LLM runtime (halogen-class) on this fleet: docs, env-lever enumeration from the binary, ledger scraping, cache-assisted A/B under the bench law."
+---
+
+# Investigating a closed-source LLM runtime (halogen-class)
+
+Use when a containerized closed-source engine (e.g. halogen on frank-large) underperforms and the repo docs don't explain it. Fleet law: stop the production unit BEFORE any GPU experiment (`sudo systemctl stop <unit>`; mask FAILS on relinked units — stop alone, redeploy after).
+
+## 1. Ground truth before flags
+- Boot ledger + per-request timings live in `podman logs <container>` (container-lifetime only) UNLESS the serve script captures stdout on disk (foreground `podman run`). A `podman run -d ... > LOG` redirect captures ONLY the container ID — a one-line hex file is the tell.
+- `serve_api:`-style log lines carry per-request `prompt/prefill/rounds/commit/detok` — build the depth-vs-rate curve from REAL traffic before any bench. Shallow-healthy + deep-collapsed = a depth-linear term (KV reads), not a broken engine.
+- If the engine's public repo exists, read its FLAGS/docs (e.g. `docs/FLAGS.md`) AND the bench tool source (e.g. `tools/halogen-bench.py`) — bench tools reveal which regimes the published numbers never measured (halogen's tg rows are shallow-only).
+
+## 2. Enumerate the undocumented levers
+The binary self-announces every env var it reads. The image often lacks `strings`:
+```
+podman run --rm --entrypoint /bin/sh <image> -c \
+  'grep -aoE "<PREFIX>_[A-Z0-9_]+" /usr/local/bin/<engine> | sort -u'
+```
+Classify by kernel family (attention vs linear-state vs GEMV) against the suspected term. Only test levers with public precedent (e.g. issue #12 proved HALOGEN_FA_TQ) or an obvious name match; defaults are the publisher's measured winners — an A/B must show >=~15% to adopt.
+
+## 3. Cache-assisted A/B (cheap reps)
+Boot each probe arm as its own container (env is read at startup) with the prompt cache ON (explicit MB >= one full-entry at ~70 KB/token): rep 1 pays the cold prefill, rep 2+ resumes (~2 s) — depth measurements then cost decode-time only. Publish the probe on host LOOPBACK (127.0.0.1:<scratch>), run the driver ON the host (python3 stdlib is enough), read the engine's own `timings` (prompt_ms/cache_n/predicted_ms/draft_n_accepted). Never benchmark beside a live engine on the same GPU.
+
+## 4. Adopt or revert
+- Winner: env line into the serve env + test-seam update + docs, then `deploy.sh <target> --service <svc>` (restart restores everything; no unmask needed after stop-only).
+- Loser: `podman rm -f` probes, start the unit, revert nothing.
+- Either way: record the measured before/after in the runbook; report the depth behavior upstream if the publisher invites benchmark data — their bench gaps (depth, carve-topologies) are exactly what issues like #12 close.

agent/managed-skills/command-center-dotfiles-import/SKILL.md

@@ -1,35 +0,0 @@
----
-name: command-center-dotfiles-import
-description: "Runbook for deploying the command-center repo (dotfiles env layer + OMP harness) to fleet machines via full-replica rsync and deploy.sh, including override-package mechanics and ssh/sshpass transport gotchas; executed end-to-end on blankpad 2026-09-09."
----
-
-# Command-center fleet deployment (repo = deployment)
-
-Repo `~/.omp` is itself the deployment: packages are home-relative trees under `dotfiles/<pkg>/`; `scripts/deploy.sh` symlinks live files into the repo checkout. `git pull` + `deploy.sh` is the whole propagation story.
-
-## Transport decision (agreed with Frank, executed 2026-09-09)
-- Ship via **full-replica rsync of `/home/dank/.omp/` including `.git`** to `<user>@<host>:/home/<user>/.omp/`, NOT clone-from-Gitea — `main` can be ahead of `origin/main`; push is a user-only trigger, so clone would ship stale state.
-- Deploy runs on-target as the target user: `bash /home/<user>/.omp/scripts/deploy.sh` (resolves repo via `git -C $(dirname $(realpath $0)) rev-parse --show-toplevel`, so `.git` must be present at target path — full-replica transport guarantees this).
-
-## Pre-overwrite audit (MANDATORY before first sync on a non-blank host)
-Diff every file that `deploy.sh --check` reports as DRIFT against the repo copy BEFORE letting quarantine-and-replace run. Host-specific divergence belongs in an override package, not in quarantine:
-1. Pull divergent live files from target (`ssh ... 'cat <path>' > local-staged-copy`).
-2. Secret-audit staged copies (grep token/secret/password/key shapes) before anything enters git.
-3. Compose `dotfiles/zz-<hostname>/` on the source machine from those live copies; commit to repo first.
-4. Ship package dir via rsync, then run deploy sync — override deploys LAST and wins over base packages.
-
-## deploy.sh mechanics (verified on blankpad)
-- `--check` = pure report (ADOPT/LINK/DRIFT lines), writes nothing. Run it first, always.
-- Default mode = sync: adopt identicals silently, link new, quarantine-and-replace divergents into `.quarantine/<host>/<ts>/` (gitignored; never rm).
-- **Override-target bug (fixed cc2c652):** `zz-<host>` packages nest base-package dirs internally (`zz-blankpad/niri/.config/...`); target derivation must strip the nested package segment, else overrides deploy to phantom `$HOME/niri/.config/...` while real targets keep pointing at canonical copies (visible symptom: check shows 0 actions but `readlink ~/.zshrc` points at base `zsh/.zshrc`).
-- systemd units auto-enable only when ExecStart binary resolves on that host; dconf package loads via `dconf load /`, never symlinks.
-- `core.hooksPath=.githooks` is armed by deploy.sh per machine — set it once post-transfer even before first sync.
-
-## ssh/sshpass transport gotchas (bit us live)
-- Resolve target via mDNS FQDN (`avahi-resolve -4 -n <host>.local`); bare hostname may not resolve. Drop-not-refuse on port 22 = sshd down or firewall default-drop, not auth failure — needs local/console fix on target.
-- `SSHPASS='...' sshpass -e ssh ...` inline per invocation; env passed via tool `env` param did not propagate to the sshpass child process.
-- Remote login shell may be **fish**: never send bash syntax (`for/do/done`, `$?`) bare through `ssh host 'cmd'`; wrap as `bash /path/script.sh` or `bash -c "..."`.
-- File shipping: `ssh 'cat > /remote/path' < localfile` (stdin redirect). A here-doc attached to the ssh command is consumed by `cat` — the intended script body becomes the file content, silently. Verify shipped bytes with md5sum compare on both ends; do NOT compare `/dev/stdin` after `cat` drained it (false DIVERGED).
-
-## Verification of a completed deploy
-Re-run `deploy.sh --check` → expect exactly zero actions/quarantined/skipped = idempotent steady state. Spot-check `readlink` on override files points into `dotfiles/zz-<host>/`, not base packages.

agent/managed-skills/compose-stack-bringup-groundtruth/SKILL.md

@@ -1,64 +0,0 @@
----
-name: compose-stack-bringup-groundtruth
-description: "Bring up a compose-defined container stack on a rootless podman host by verifying reality first (mounts, images, in-image interfaces) instead of trusting compose env/flag guesses; includes the copyparty ac-edition argv-interface recipe and boot-persistence audit."
----
-
-# Compose-stack bring-up with ground truth (rootless podman hosts)
-
-Use when standing up (or debugging) a compose-defined service stack on a rootless
-podman host where the stack has never run, or answers with surprise statuses
-(SIGSEGV at boot, HTTP 500 on `/`, "manifest unknown").
-
-## Procedure
-
-1. **Audit reality before pressing buttons.** `podman ps -a` runs per-user —
-   check every namespace (`sudo podman ps -a`) and native units
-   (`systemctl list-units`) before declaring a service absent; a service may run
-   as a native systemd unit or snap, not a container. Compare compose mount source
-   paths against the real filesystem (`ls /mnt/...`): compose files often name
-   phantom paths that never existed (e.g. `/mnt/media/*` vs the real LUKS array at
-   `/mnt/entropy`). Fix compose through git first, deploy second.
-
-2. **Pre-create bind-mount sources.** Rootless podman fails container creation
-   with `statfs /opt/<svc>/...: no such file or directory` when a bind source is
-   missing. `sudo mkdir -p` + `chown 1000:1000` every compose volume source
-   before the first `podman-compose up -d`.
-
-3. **Staged bring-up.** Infra first (`up -d mariadb redis`), poll healthchecks via
-   `podman inspect --format "{{.State.Health.Status}}"`, then `up -d` the rest.
-
-4. **Never guess image tags — query ground truth.** ghcr.io answers **403 on the
-   bearer-token request for nonexistent/private repos** ("manifest unknown" / token
-   len 0 = repo or tag does not exist, not an auth problem). Query the registry
-   API (`ghcr.io/token?scope=repository:<ns>/<repo>:pull` then
-   `/v2/<ns>/<repo>/tags/list`) before pinning a tag.
-
-5. **Read the interface from inside the image** before guessing flags/envs:
-   `podman exec <ctr> cat /z/cpp.sh`, `cat /z/initcfg`,
-   `podman exec <ctr> python3 -m copyparty --help-accounts`. A service answering
-   HTTP 500 with its own styled error page is *running* — it is misconfigured,
-   not down. Copyparty ac edition ground truth: entrypoint
-   `/bin/ash /z/cpp.sh -c /z/initcfg` then `exec python3 -m copyparty "$@"`;
-   accounts/volumes are **argv flags** (`-a USER:PASS`, `-v SRC:DST:PERMS`), not
-   `COPAY_*` envs (zero `COPAY_` references in the package). With no accounts and
-   no volumes configured it serves cwd read/write for everyone — an
-   unauthenticated world-writable share of the whole mount.
-
-6. **Beware remote-shell quoting traps.** Remote zsh eats bare `===`/`?` tokens
-   (glob qualifiers/globs); grep patterns with nested quotes over ssh can be
-   quote-eaten into silent zero-match. A zero-match from a mangled pattern is not
-   absence — re-verify the target exists before concluding it.
-
-7. **Boot-persistence audit last.** On rootless podman hosts with
-   `podman-user-generator`, containers started via podman-compose get generated
-   `container-<name>.service` units at first boot (evidence: an existing unit for
-   one container proves the generator works). Verify after first reboot:
-   `systemctl --user is-enabled <name>.service`.
-
-## Gotchas
-- jellyfinetype SIGSEGV at boot with `user: "1000:1000"` = unmapped uid inside the
-  rootless userns (host uid 1000 == container uid 0); drop the user line and let it
-  run as container-root (= host deploy user).
-- `up -d --force-recreate` applies compose changes that change command/env; probe with
-  a poll loop (`for i in $(seq...)`) because fresh containers answer `000`/refused for
-  the first seconds while indexing.

agent/managed-skills/compose-to-quadlet-cutover/SKILL.md

@@ -1,72 +0,0 @@
----
-name: compose-to-quadlet-cutover
-description: "Convert a compose-only service to a boot-persistent rootless Quadlet unit on the Dankclaw/Dankserver fleet — checklist plus the traps (CMD-SHELL healthchecks in distroless images, --format docker, EnvironmentFile HOME=/root, loopback publish posture)"
----
-
-# Compose service → boot-persistent Quadlet unit (this fleet)
-
-Use when a service in `dankclaw/compose.yaml` (or `dankserver/`) must survive a
-reboot. Precedents to copy: `dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`
-and `dankclaw/observability/quadlet/` (five units, 2026-09-11).
-
-## Checklist
-
-1. **Unit file** at `dankclaw/<svc>/quadlet/<name>.container`, mirroring the
-   compose service line for line: `Image=`, `ContainerName=<the compose name>`
-   (Caddy and scrape configs address containers by name), `Network=dankclaw_default`,
-   `User=0:0` when the image's own user cannot write a rootless-owned bind mount,
-   `Volume=` for each mount (`:ro` preserved), `Exec=` for the compose `command:`,
-   `PublishPort=`, `HealthCmd`/`HealthInterval`/`HealthTimeout`/`HealthRetries`/
-   `HealthStartPeriod`, `[Service] Restart=always` + `RestartSec=10`,
-   `[Install] WantedBy=default.target`. Never `systemctl --user enable` the
-   generated unit — `[Install]` plus linger IS the boot persistence.
-2. **Exposure**: publish on `127.0.0.1` unless something off-host genuinely
-   needs it. Caddy reaches containers by name over the shared network, and the
-   homepage status timer probes loopback, so the 0.0.0.0 publish is almost
-   always gratuitous and is a guard-chain bypass.
-3. **Handler** in `scripts/deploy-remote.sh`: `install_quadlet <dir> <name>` →
-   `systemctl --user restart <name>.service` → `wait_for_health` on the
-   loopback endpoint. Rebuild locally built images first
-   (`build_compose_image`), and seed any per-service env file (see traps).
-4. **Planner**: map `dankclaw/<svc>/*` and `shared/<svc>/*` to the service in
-   `scripts/deploy-plan.py`; keep dependencies ordered in `SERVICE_ORDER`.
-   Add a `scripts/test-changed.sh` case *above* the generic
-   `dankclaw/*/quadlet/*.container` fallback.
-5. **Retire the compose entry** in the same change — two owners for one
-   container name is how a stale duplicate comes back.
-6. **Seam test** `tests/<svc>_quadlet.bats`: boot persistence, restart policy,
-   no wildcard publish, container name still matching Caddy/scrape references,
-   health command, deployer installs+restarts+waits, never `rm`, never
-   `enable`. Migrate any older assertion that read the compose file.
-7. **Deploy** per service (`./scripts/deploy.sh dankclaw --service <name>`),
-   then prove: `is-active`/`is-enabled` = active/generated, container health
-   `healthy`, the endpoint answers, `list-dependencies default.target` shows the
-   unit, and off-host probes on the previously published ports now refuse.
-
-## Traps that cost real time
-
-- **`HealthCmd` = CMD-SHELL.** podman runs it as `/bin/sh -c`; a distroless
-  image has no shell, so the check fails forever. `--health-cmd` will not take
-  exec form either (a JSON array is stored as one literal argument). Put
-  `HEALTHCHECK ... CMD ["binary","arg"]` in the image and **build with
-  `--format docker`** — the default OCI format has no HEALTHCHECK field and
-  drops it with only a warning.
-- **Never point a unit's `EnvironmentFile` at the repo root `.env`.** That file
-  is a gluetun env file carrying `HOME=/root`: systemd sets HOME for the
-  service, podman then looks for its config under `/root`, and the unit dies
-  with `stat /root/.config: permission denied`. Use a per-service file under
-  `/home/dank/Containers/<svc>/env`, seeded by the deploy handler with a
-  single-variable filter and validated so a missing secret fails loudly.
-- **`Network=host` exporters** that a containerized Prometheus scrapes must
-  bind the host's LAN address (containers cannot reach the host loopback) plus
-  `127.0.0.1` for the homepage probe — two `--web.listen-address` flags, never
-  the wildcard.
-- **alloy's UI** listens on the container's loopback and 1.x refuses the `ui`
-  config block, so the unit must pass `--server.http.listen-addr=0.0.0.0:12345`
-  before its port can be published; without it the service has no reachable
-  health endpoint and its status card reads down forever.
-- **Locally built images** need `Pull=never` and `Image=localhost/<repo>/<name>:tag`;
-  registry-prefixed refs do not resolve from local storage on these hosts.
-- Compose-managed containers are replaced in place by `podman run --replace
-  --rm`; verify afterwards that the container no longer carries
-  `io.podman.compose.project`.

agent/managed-skills/container-config-drift-crashloop/SKILL.md

@@ -1,82 +0,0 @@
----
-name: container-config-drift-crashloop
-description: "Diagnose a container that crash-loops or stays unhealthy while its compose/unit definition looks correct: config drift kept alive by a restart policy, with the inspect commands, the Devices-vs-CreateCommand false negative, and the compose-to-Quadlet migration order."
----
-
-# A container that restarts forever while its definition is correct
-
-## When to use
-
-A container is crash-looping or perpetually "unhealthy", but the file that
-defines it (compose service, unit, deploy script) looks right. Also when a
-service has been quietly down for days and nobody noticed, because a restart
-policy kept something running.
-
-The cause is usually **config drift**: the running container was created from an
-older definition, and the restart policy restarts it instead of recreating it.
-A compose/unit edit does not touch an existing container.
-
-## Diagnose (5 minutes, in this order)
-
-1. **Who started it?** The label names the owner, not your guesswork:
-
-   ```
-   podman inspect <name> --format 'unit={{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}} created={{.Created}}'
-   ```
-
-   `podman-compose@…` means compose owns it, whatever unit files exist.
-   Compare `created` with when the definition last changed. A container older
-   than the edit that should have replaced it is the bug.
-
-2. **What is it ACTUALLY running?** Read the argv, not the summary fields:
-
-   ```
-   podman inspect <name> --format '{{.Config.CreateCommand}}'
-   ```
-
-   Trap: `{{.HostConfig.Devices}}` prints an empty list even when `--device`
-   flags are present — a false negative that can send you hunting the wrong
-   root cause. `CreateCommand` is the truth; grep it for the flag you care
-   about (`--device`, `--group-add`, `-e`, mounts).
-
-3. **What does the process say?** `podman logs <name> | grep -iE 'error|cannot|Runtime'`.
-   A capability error (`No HIP GPUs are available`) alongside a missing device
-   flag in step 2 confirms drift rather than a code or image fault. Count the
-   damage: `podman inspect --format '{{.RestartCount}}'`.
-
-## Fix
-
-Quickest unblock: recreate from the definition (`podman compose up -d
---force-recreate <svc>`), which is what should have happened automatically.
-
-Durable fix, when the host is governed by a repo that requires boot-persistent
-units (this one does): migrate the service to a Quadlet. Order that worked:
-
-1. build the image and `install` the unit file to `~/.config/containers/systemd/`
-2. stop/disable the legacy unit and **move its file to `~/.AGENTTRASH/`** — a
-   leftover `~/.config/systemd/user/<svc>.service` collides by name with the
-   unit the Quadlet generator produces
-3. `podman rm -f <container>` — the stale container holds the name the unit's
-   container needs, and it carries the old config
-4. `systemctl --user daemon-reload && systemctl --user restart <svc>.service`
-5. delete the compose service so only one owner remains per container name
-
-## Verify the capability, not the status
-
-`Up` and even `healthy` can be true of a half-configured container. Check the
-thing that was broken, from inside:
-
-```
-podman exec <name> python3 -c "import torch; print(torch.cuda.is_available())"
-ss -ltn | grep <port>            # confirm the bind addresses, not the intent
-```
-
-And confirm boot persistence the way the generator exposes it:
-`systemctl --user is-enabled <svc>.service` → `generated` is healthy; an
-explicit `enable` fails on generated units.
-
-## Related reading in this repo
-
-- AGENTS.md "Boot Persistence & Self-Healing" — the rule this enforces
-- `serverAppConfigs-docs/Dankclaw/ComfyUI.md` — the 9,146-restart incident
-- `serverAppConfigs-docs/Infrastructure/Observability.md` — same class, twice

agent/managed-skills/dankclaw-guard-chain-layer-diagnosis/SKILL.md

@@ -1,33 +0,0 @@
----
-name: dankclaw-guard-chain-layer-diagnosis
-description: "When a user reports a guarded dankclaw page (homepage/vhost) broken but the server may be healthy: verify each layer of the Anubis→Authentik→Caddy chain server-side and isolate client-side vs server-side."
----
-
-# Dankclaw Guard-Chain Layer Diagnosis
-
-Use when a user reports a guarded page (e.g. `www.franksdank.cloud` homepage) "broken / links gone / not loading" and you must determine whether the fault is server-side or client-side/session.
-
-## Chain topology
-`Browser → Caddy TLS edge → Anubis :8923 → Caddy :8016 → Authentik outpost → Caddy :8017 (@home / @data handlers) → app/static`
-- Public vhosts `reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}`.
-- Post-auth `http://:8017` listener routes by `@<svc> host <vhost>`.
-- Homepage static: `/srv/homepage/` (index.html, app.js, style.css, services.json, status.json, deployed.js).
-
-## Layer-by-layer checks (run from the host via ssh)
-1. **Caddy unit + guard containers**: `systemctl --user is-active caddy.service`; `podman ps --format '{{.Names}}\t{{.Status}}' | grep -iE 'anubis|authentik|caddy'` — expect `Up ... (healthy)`.
-2. **Anubis challenge page validity**: `curl -sk https://www.franksdank.cloud/ --resolve www.franksdank.cloud:443:192.168.0.143` — a healthy challenge contains `"anubis_challenge"` JSON with `difficulty` and the main JS. If present, the guard serves fine.
-3. **Authentik outpost errors**: `podman logs --tail 40 authentik-outpost 2>&1 | grep -iE 'error|exception|traceback|500|provider'` — routine dramatiq task logs are normal.
-4. **@home / @data from INSIDE the caddy container** (the only way to reach 8016/8017):
-   `podman exec caddy sh -c 'wget -q -S -T5 --header "Host: www.franksdank.cloud" -O- http://127.0.0.1:8017/'`
-   and the same for `/services.json`, `/status.json`. Expect 200 + body.
-5. **Files on disk**: `ls -la /srv/homepage/` and validate `status.json` parses (`python3 -c "import json;json.load(open('/srv/homepage/status.json'))"`).
-
-## Gotchas (cost me calls)
-- **BusyBox wget in the caddy container**: NO `-H` flag. Use `--header "Host: ..."`.
-- **8016/8017 live in the caddy container's netns**: `curl http://127.0.0.1:8017` from the HOST returns `000` (connection refused) — EXPECTED, not a fault. Test from inside the container.
-- **anubis container is distroless**: `podman exec anubis sh` fails (`sh: executable not found`). Inspect via its logs / the served challenge page instead.
-- **x-real-ip masking**: Anubis logs `x-real-ip: 10.89.1.22` (Caddy container podman IP) for ALL clients — rootless-podman published-port SNAT. Pre-existing; clients indistinguishable by IP; clearance is cookie-based. Do NOT mistake this for the bug.
-- **Homepage render**: `app.js` does `Promise.all([fetch services.json, fetch status.json])`; on ANY fetch failure the catch sets `#summary` to `status unavailable: <err>` and the grid stays EMPTY — which a user reads as "links gone / not loading." A 30-service status.json with a 9-service services.json renders fine (stateByName lookup).
-
-## Decision
-If every layer above is healthy (challenge valid, outpost clean, @home/@data 200, files render), the fault is client-side/session. STOP thrashing the live system (repeated failed browser challenges share the masked 10.89.1.22 identity). Get the authenticated view: ask for the exact symptom (challenge spinning? Authentik login loop? empty grid? "status unavailable" text?) or the Authentik login to drive the page in a real browser.

agent/managed-skills/dankclaw-homepage-asset-verify/SKILL.md

@@ -0,0 +1,13 @@
+---
+name: dankclaw-homepage-asset-verify
+description: "Add/verify homepage assets on Dankclaw: Caddy @data allowlist, deploy copy step, e2e panel census, live content-type check."
+---
+
+Every new top-level homepage asset directory (e.g. `shared/homepage/img/`, `shared/homepage/sfx/`) needs FOUR coordinated changes, all in one commit:
+
+1. **Caddy allowlist** — `dankclaw/Caddyfile` `@data path` matcher must include `/<dir>/*`. Without it the post-auth `@home` catch-all answers with `index.html` (`200 text/html` for a PNG/OGG — silent breakage).
+2. **Deploy copy** — `scripts/deploy-remote.sh` `deploy_homepage()` copies `shared/homepage/<dir>/` with the `cp -r ... /srv/homepage/` merge pattern beside the fonts/img/sfx blocks.
+3. **Test pins** — `tests/homepage_caddy.bats` grep-pins the new `@data` path; if the asset changes panel rendering, the e2e census in `tests/e2e/specs/10-homepage.spec.ts` must count the new render style (it derives expected names from services.json: `.panel .name` for normal cards, `.mi-name` for fused Model Information cards — a new fused component breaks the panel census otherwise, which is exactly how the 2026-10-08 Protoss deploy failed its e2e gate).
+4. **Deploy both services** — `deploy.sh dankclaw --service homepage` and `--service caddy` (Caddyfile change), then live-verify from a browser tab: `fetch('/<dir>/<file>')` must return `200` with the correct `content-type` (`image/jpeg`, `audio/ogg`), not `text/html`.
+
+Checker note: `status.json` names follow `services.json` automatically (the deployed checker re-reads it via the homepage deploy's restart of homepage-status.timer), so renaming cards needs no checker change.

agent/managed-skills/defuse-shell-rc-secrets/SKILL.md

@@ -0,0 +1,21 @@
+---
+name: defuse-shell-rc-secrets
+description: "Find literal API keys/tokens in shell rc files and replace them with 600-perm secret files + loader lines, syncing the canonical tool token file."
+---
+
+# Defuse shell-rc secrets
+
+Symptom: `export KEY=...` literal in `.zshrc`/`.bashrc` — world-readable in every process env, one `git add -A` from a dotfiles repo.
+
+## Procedure (repeat per host)
+1. Sweep: `grep -n "export.*KEY\|export.*TOKEN" ~/.zshrc ~/.zprofile ~/.profile ~/.bashrc 2>/dev/null` (run per host; when batching hosts in one ssh compound command, label which output came from which host — misattribution wastes turns).
+2. Extract WITHOUT echoing: `mkdir -p ~/.config/secrets && chmod 700` it, then `sed -n 's/^export NAME="\(.*\)"$/\1/p' ~/.zshrc > ~/.config/secrets/<name>` (quote variant and unquoted variant are different sed patterns — verify byte counts with `wc -c`, a 0-byte file means the pattern missed).
+3. `chmod 600` every secret file; sync any canonical tool file the ecosystem expects (e.g. `~/.cache/huggingface/token`).
+4. Replace the rc literal (hashline `PUT N.=N:` range — never echo the secret in the op): `[ -f ~/.config/secrets/<name> ] && export NAME="$(cat ~/.config/secrets/<name>)"`.
+5. Verify: `zsh -n ~/.zshrc` (or `bash -n`); keep a dated `.bak` of the rc on remote hosts; shred any /tmp transfer copies immediately (`shred -u`).
+
+## Laws
+- A token pasted into chat lands in transcripts and model context — treat chat as a leak channel; give the user the `read -rs` snippet instead.
+- scp of a secret FILE (encrypted channel, no argv) is fine; the token is never a command-line argument.
+- After replacing a legacy HF token, probe scope non-destructively: `HfApi().whoami()` (legacy read tokens show `type: access_token` without canWrite; the decisive probe is `create_repo(exist_ok=True)` on the real target repo).
+- Verify extraction byte-count before deleting/trusting; a successful-looking sed can write an empty file.

agent/managed-skills/detached-fleet-job-launch/SKILL.md

@@ -0,0 +1,46 @@
+---
+name: detached-fleet-job-launch
+description: Launch a long-running remote job on Dank fleet hosts so it survives ssh/logind teardown
+---
+
+# Detached long-running jobs on Dank fleet hosts
+
+## Law
+`ssh host 'setsid nohup CMD & '` is NOT detached. systemd-logind deactivates
+`session-N.scope` when the ssh command returns and SIGKILLs the whole cgroup:
+no journal entry, no OOM trace, python block-buffers stdout so the log stays
+0 bytes. setsid/nohup do not escape the cgroup.
+
+## Canonical launch
+```
+ssh host 'export XDG_RUNTIME_DIR=/run/user/$(id -u); \
+  systemd-run --user --collect --unit=NAME \
+    -p SendSIGKILL=no \
+    -p StandardOutput=append:/home/dank/logs/NAME.log \
+    -p StandardError=append:/home/dank/logs/NAME.log \
+    sh -c "CMD -u; echo \$? > /home/dank/logs/NAME.rc"'
+```
+- transient user unit = outside every session scope; survives ssh teardown
+  (requires `Linger=yes`, true for `dank` on both hosts)
+- `echo $? > …rc` inside the unit = completion sentinel the unit itself writes
+- `-u` / unbuffered output so the log is live
+- verify launch in the same ssh call: `systemctl --user is-active NAME.service`
+
+## Watcher (one async job, never poll)
+```
+ssh host 'while [ ! -f /home/dank/logs/NAME.rc ]; do sleep 60; done; \
+  echo RC=$(cat /home/dank/logs/NAME.rc); tail -25 /home/dank/logs/NAME.log'
+```
+Run it async with timeout 0; the result is injected on completion.
+
+## Forensics when a detached job died
+1. `ps -p PID` — gone.
+2. Log 0 bytes → buffered stdout lost at SIGKILL.
+3. `journalctl --since … | grep -E "session-[0-9]+.*(Deactivated|Removed session)"` —
+   timestamps matching the death window confirm the scope kill (vs OOM, which
+   leaves kernel lines).
+
+## Notes
+- zsh remote shells: `echo ===X===` fails (`=cmd` expansion); quote sentinels.
+- The HF upload of 2026-10-08 (113.7 GiB, hf_xet) died this way ~2 min in;
+  relaunch as above succeeded.

agent/managed-skills/e2e-arr-rawip-reachability-triage/SKILL.md

@@ -1,72 +0,0 @@
----
-name: e2e-arr-rawip-reachability-triage
-description: "Triage an e2e admin-SSO failure on a *arr raw IP (192.168.0.112:8989) by probing from the box the browser runs on, instead of assuming the Mac is asleep and burning retries waking it."
----
-
-# e2e raw-IP *arr failures — probe from the browser's box, not a peer
-
-`tests/e2e/specs/11-admin.spec.ts` navigates the real browser to
-`http://192.168.0.112:8989/` (Sonarr's forwarded IP on FrankBook-Pro). When it
-fails with `ERR_CONNECTION_TIMED_OUT` the instinct is "the Mac is asleep".
-Sometimes true. Not always — and waking it is not the only recovery.
-
-## The decisive question
-
-**Who cannot reach the port?** Playwright runs on the dev box (blankpad), not on
-a fleet host. A healthy probe from dankclaw proves nothing about the browser's
-path.
-
-```bash
-# 1. From the box that RUNS the browser:
-curl -s -o /dev/null -w '%{http_code}\n' --max-time 7 http://192.168.0.112:8989/
-
-# 2. From a LAN peer, same port:
-ssh dankclaw 'for p in 8989 7878 9696; do curl -s -o /dev/null -w "$p %{http_code}\n" \
-  --max-time 6 http://192.168.0.112:$p/; done'
-
-# 3. Liveness of the Mac itself (ICMP, not TCP):
-ping -c 2 -W 3 192.168.0.112
-```
-
-## Reading the matrix
-
-| peer probe | blankpad probe | ping | meaning |
-|---|---|---|---|
-| 200 | 200 | ok | transient — re-run the spec |
-| timeout | timeout | fails | Mac asleep; wake, ports return in seconds |
-| **200** | **rc=28** | **ok** | **path problem, not sleep** — ICMP allowed, TCP to those ports blocked. Waking and re-running cannot help |
-
-## If it is the third row, confirm the Mac is not at fault
-
-Through the dankclaw jump (`ssh dankclaw "ssh dank@192.168.0.112 ..."`):
-
-```bash
-pgrep -fl socat            # forwarders bound to 192.168.0.112:port
-netstat -an | grep -c 8989 # listening
-curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8989/   # 200 = app fine
-/usr/libexec/ApplicationFirewall/socketfilterfw --getglobalstate
-```
-
-Forwarders up + loopback 200 + `Firewall is disabled` ⇒ the Mac is innocent. The
-block is between the dev box and the Mac, so stop optimising the Mac.
-
-## What to do instead
-
-Don't hold a deploy hostage to a known-environmental red gate:
-
-1. Verify the deployed service **directly** — its own ledger/health/real
-   completion — and state that the e2e gate was skipped or partially red, and
-   why (`deploy.sh --skip-e2e` prints exactly that warning, so the skip is
-   recorded, not hidden).
-2. Report the residual gap plainly: *"29/30 e2e pass; 11-admin fails on Sonarr's
-   raw IP from the dev box; peer probe returns 200, so the deploy is verified by
-   other means and the gate gap is a dev-box network path issue."*
-3. Do not weaken or skip the spec to make it green — it is guarding a real
-   SSO-plus-RBAC contract.
-
-## Note on the sleep case
-
-The classic flake is real: with the Mac asleep, ports refuse from **everywhere**
-including peers, ping latency runs ~150 ms, and an SSH/ping wake restores
-200/302 within seconds. That is the cheap first check — just don't stop there
-when the peer probe is already green.

agent/managed-skills/fandom-wiki-media-fetch/SKILL.md

@@ -0,0 +1,19 @@
+---
+name: fandom-wiki-media-fetch
+description: Fetch audio/image media from a Fandom wiki via its MediaWiki API when the wiki HTML pages 403 curl or hang headless browsers.
+---
+
+Goal: get direct media URLs (and files) from a starcraft.fandom.com-style Fandom wiki.
+
+Why this route: Fandom HTML pages return a ~5 KB JS shell to `curl` (403 via `read ...:raw`) and time out headless Chromium (`browser.open`/`goto` >30 s on quotation megapages). The MediaWiki API is plain JSON and answers plain curl.
+
+Steps:
+1. Find the content page's wikitext:
+   `curl -sL -A "Mozilla/5.0" "https://<wiki>.fandom.com/api.php?action=parse&page=<Page_Name>&prop=wikitext&format=json&formatversion=2"`
+   Top-level pages are often tabber/transclusion shells — follow the `{{:Subpage}}` / `{{TemplateName}}` references with the same call (e.g. `StarCraft_unit_quotations` → `/Protoss` → `Template:QuotesSC1Zealot`).
+2. Extract media names from the wikitext. StarCraft quotes use `{{sm2|File.ogg|displayed quote}}` inside `{{UnitQuoteBox|select=...}}` fields — match the quote text to pick the exact file.
+3. Resolve direct CDN URLs (works for images too):
+   `curl -sL -A "Mozilla/5.0" ".../api.php?action=query&prop=imageinfo&iiprop=url|size&format=json&formatversion=2&titles=File:A.ogg|File:B.jpg"`
+   → `query.pages[].imageinfo[0].url` = `https://static.wikia.nocookie.net/...` — downloadable with plain `curl -sfL`.
+4. To discover art on a unit page: `action=query&prop=images&imlimit=100&titles=<Page>`, filter titles (icon/art/portrait/Head/Render), then resolve the keepers via imageinfo. Note splash/head art naming: `File:Zealot SC2 Art1.jpg`, `File:Archon SC2 Head1.jpg`; some obvious guesses (SC1-era units' SC2 splash) do not exist — verify with imageinfo, absent pages return `missing` with no `imageinfo`.
+5. Verify downloads (`file sfx/*.ogg` → `Ogg data, Vorbis audio`) — a 403 body saves as a success with `-o` and no `-f`.

agent/managed-skills/fleet-container-ownership-audit/SKILL.md

@@ -1,91 +0,0 @@
----
-name: fleet-container-ownership-audit
-description: "Audit a rootless-Podman fleet for containers that will not survive a reboot: owner-label detection, the stale-container and inspect-field traps, and the migration shape that fixes them."
----
-
-# Finding the containers that will not survive a reboot
-
-Use when auditing a Podman fleet for boot persistence, when a service is
-"up" but stopped working at some past reboot, or when a compose-managed stack is
-being converted to Quadlet units. This is the *detection* half; the conversion
-checklist is separate.
-
-## The one-line test
-
-Every running container on a boot-persistent host should carry a
-`PODMAN_SYSTEMD_UNIT` label. Anything without one is compose-only or bare, and
-rootless Podman does not restart bare containers after the explicit stop that
-shutdown performs — so it will be missing tomorrow morning.
-
-```bash
-for c in $(podman ps --format '{{.Names}}'); do
-  printf '%-26s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')"
-done
-```
-
-Compare the result against the host's installed units
-(`ls ~/.config/containers/systemd/*.container`) and the repo's
-`<host>/<service>/quadlet/`. Three outcomes, three meanings:
-
-- **label present** → the unit owns it; boot persistence comes from `[Install]
-  WantedBy=default.target` plus `loginctl show-user <user> -p Linger` = yes.
-- **label absent, compose service exists** → the classic gap: it dies at reboot.
-  Also check the publish address while you are there; compose forms often bind
-  the wildcard, which puts an unauthenticated UI/API on every interface.
-- **label present AND a compose service with the same name** → a second owner.
-  The runtime is fine, but a stray `podman compose up` can create a duplicate
-  that fights the unit for the container name and the port. Remove the compose
-  service and leave an explanatory comment; that is the repo's convention.
-
-## Traps that make this hard to see
-
-- **Editing a compose service does not replace a running container.**
-  `restart: unless-stopped` will happily keep an old container — old image, old
-  device list, old mounts — alive forever. One instance ran 9,146 restart
-  attempts on stale config before anyone looked. If the container predates the
-  edit, recreate it explicitly; do not debug the service.
-- **`podman inspect --format '{{.HostConfig.Devices}}'` is not a device check.**
-  It can read empty on a container that demonstrably has the devices. Read the
-  real argv with `{{.Config.CreateCommand}}` and, decisively, probe from inside:
-  `podman exec <c> python3 -c 'import torch; print(torch.cuda.is_available())'`.
-- **A monitoring target's failure string can prove a fix.** A `/metrics` scrape
-  that flips from `connection refused` to `unsupported Content-Type "text/html"`
-  means the service came up and only the scrape is wrong. Read the error, not
-  just up/down — and never "fix" a red probe by deleting the job, because an
-  absent scrape and a healthy one look identical.
-- **A hand-written `~/.config/systemd/user/<svc>.service` that shells out to
-  `podman compose up` collides by name with the Quadlet the generator produces
-  for `<svc>.container`.** Retire it to a dated trash directory before
-  installing the unit.
-- **`is-enabled` saying `generated` is the healthy state** for Quadlet units;
-  `systemctl --user enable` fails on them.
-
-## Migration shape that works
-
-1. Unit file in the repo at `<host>/<service>/quadlet/<service>.container`:
-   `Image=` with the `localhost/` prefix for locally built images, `Pull=never`,
-   explicit `PublishPort` (loopback unless the service is the documented LAN
-   exception), `AddDevice=`/`GroupAdd=keep-groups` where a GPU is needed,
-   `Restart=always`, `[Install] WantedBy=default.target`.
-2. A healthcheck **as a file**, never inline: the podman 5.x v1 generator
-   splits `HealthCmd` on spaces, so `python3 -c "..."` becomes a quoting puzzle
-   that fails silently.
-3. `deploy_<svc>()` in the remote handler: install the unit, retire any
-   compose-era unit into `~/.AGENTTRASH/`, `podman rm -f <container>` so the
-   unit's own can take the name, daemon-reload, restart, wait for health.
-4. Delete the compose service (one owner per container name) and leave a comment
-   naming the unit, so the next reader does not re-add it.
-5. Planner: add a path rule (`<host>/<dir>/**` -> service) **and** a
-   `service_is_allowed` entry, or the shared dispatcher rejects the service
-   before anything runs.
-6. Migrate the seams: tests that read the compose file for devices, command
-   flags or groups must read the unit instead — and resolve file paths from the
-   tooling's own declarations, never from `find ... | head -1`.
-
-## When the last service leaves a compose file
-
-The repo's compose-validity seam requires at least one service, so a file whose
-services have all migrated fails its own test. Either keep the file as
-comments and relax that seam deliberately, or delete the file and fix the
-planner's compose-diff path — decide with the owner, because the deploy tooling
-reads it.

agent/managed-skills/fleet-llm-payload-purge-refresh/SKILL.md

@@ -1,117 +0,0 @@
----
-name: fleet-llm-payload-purge-refresh
-description: "Safely purge and refresh LLM model payloads on the Dank fleet (Dankclaw/Dankserver): protect live service payloads, confirm upstream re-fetchability, fingerprint before deleting, and use each service's own refresh path (halogen's first-start HALOGEN_DOWNLOAD vs frank-small's plain-GGUF fetch) — including the empty-bind-mount-dir trap."
----
-
-# Purge and refresh LLM model payloads (Dank fleet)
-
-Use when asked to clear / reclaim / refresh model caches on a host that is
-*serving* users, or to prove that a payload is up to date.
-
-## 1. Inventory before touching anything
-
-```bash
-du -sh /home/dank/models/* | sort -h        # what is actually there
-```
-
-Then find which of those the live services mount — never delete a payload a unit
-mounts without an explicit, in-this-conversation instruction:
-
-```bash
-grep -aE "^(Volume|Environment=.*(MODEL|CHECKPOINT|MODELS))" \
-  /home/dank/.config/containers/systemd/*.container \
-  /etc/systemd/system/frank-*.service
-```
-
-On Dankclaw the protected set has been: `models/halogen` (frank-large's .hgn
-checkpoint dir), `models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp` (frank-small's GGUF +
-mmproj). ComfyUI uses `/home/dank/comfyui-data/models`, **not**
-`models/checkpoints` — verify, because `models/checkpoints` looks like cache but
-has held Flux weights whose upstream is licence-gated.
-
-## 2. Confirm each payload is re-fetchable BEFORE deleting it
-
-Query the Hub anonymously; a gated repo 403s even with the host token:
-
-```bash
-curl -s "https://huggingface.co/api/models/<repo>/tree/main?recursive=true&blobs=true" \
-  | python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d),'entries'); [print(e.get('size'),e['path']) for e in d[:5]]"
-```
-
-An error/empty result means it is NOT safe to delete. Also filter by **quant
-name**, never by `endswith('.gguf')` — mradermacher ships large quants as
-`.partNofM` fragments which that test hides.
-
-## 3. Fingerprint, then delete
-
-Record byte sizes (and sha256 when the file matters) **before** removal: that is
-the only way to answer "did the refresh actually change anything" afterwards.
-
-Per this repo's policy `rm` needs explicit operator permission; otherwise stage
-into `~/.AGENTTRASH/<date>-<topic>/`. Note that a same-filesystem trash reclaims
-nothing — say so rather than implying space was freed. Root-owned leftovers (a
-trashed swapfile) need `sudo`, and check `swapon --show` first.
-
-## 4. Refresh, per service
-
-**frank-large (halogen)** — the engine fetches for you. Clear the directory but
-leave it **existing and empty**, then start the unit:
-
-```bash
-rm -rf /home/dank/models/halogen/*        # keep the directory itself
-systemctl --user start frank-large.service
-journalctl --user -u frank-large.service -f | grep -a halogen:
-```
-
-The engine sees the checkpoint absent and runs `HALOGEN_DOWNLOAD`
-(`peonist-ai/halogen-qwen3.8-flash-next`) into the mount via `huggingface_hub`,
-resuming if interrupted. **Trap:** podman refuses a bind mount whose source
-directory is missing — `Error: statfs /home/dank/models/halogen: no such file or
-directory`, with the unit looping on `Restart=on-failure`. `mkdir -p` the empty
-directory and restart. ~124 GiB at 130-240 MB/s ≈ 10-20 min, then load.
-
-**frank-small (llama.cpp)** — no self-fetch; the serve script refuses to start
-with weights missing, so pull them yourself and verify:
-
-```bash
-DEST=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp   # repo ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
-# files: Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf + mmproj-Qwen3.8-27B-BF16.gguf
-curl -fL -C - --retry 10 -o "$DEST/<file>" \
-  "https://huggingface.co/<repo>/resolve/main/<file>"   # -f is mandatory
-sha256sum "$DEST"/*; head -c 4 "$DEST"/*.gguf           # compare to the LFS oid + GGUF magic
-```
-
-`curl` exits 0 on HTTP 401/403 and writes the error body as the artifact — always
-`-f`, always compare byte counts to the Hub's declared size, always check the LFS
-sha256. Public repos need no token: omit `-H "Authorization: Bearer ..."` from
-argv entirely (it is world-readable in `/proc/<pid>/cmdline`).
-
-## 5. Verify, then record
-
-```bash
-curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health   # frank-large
-curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7771/health   # frank-small
-curl -s http://127.0.0.1:7770/v1/models; curl -s http://127.0.0.1:7771/v1/models
-# a REAL completion on both, then halogen's boot ledger:
-journalctl --user -u frank-large.service | grep -aE "weights locked|kv pool:|left for everything"
-```
-
-Also sweep the vhosts (`www`, `grafana`, `prometheus`, `hermes`, `marreta`,
-`gitea`, `loki` — `curl -sk --resolve <host>:443:192.168.0.143`) because a purge
-touches state the whole host shares.
-
-## 6. Things that bite
-
-- **Sidecars are versioned in place.** Equal file sizes are not proof the repo is
-  unchanged, and a stale local sidecar is invisible from its name. Compare local
-  sizes against the Hub's declared sizes; that is how a 2.31 GiB → 2.40 GiB
-  overlay update was caught.
-- **Do not run a 100 GB+ re-fetch concurrently with a big iGPU benchmark.** The
-  A* run commits ~116 GiB of host memory and leaves ~10 GiB available; adding
-  cache churn there risks the OOM/state failures this fleet has already seen.
-- **Reboot is the repair of last resort** if rootless podman's pause process dies
-  during the work (`invalid internal status … could not find any running
-  process`); `podman system migrate` panics in 5.7.0.
-- Because a peer session may be committing to `serverAppConfig` concurrently,
-  re-read a doc region immediately before editing it, and check that the edit
-  response's neighbouring lines are the ones you expect.

agent/managed-skills/fleet-model-cache-purge-refresh/SKILL.md

@@ -1,73 +0,0 @@
----
-name: fleet-model-cache-purge-refresh
-description: "Clear and re-fetch model payloads on a Dank-fleet host that serves production LLM endpoints — inventory, protect live payloads, verify every fetch, restore the service, and use the refresh to detect upstream updates."
----
-
-Clearing a model cache on a serving host is a two-sided operation: the operator wants no stale
-payload, and the host must keep serving. Do it in this order, on the target host.
-
-## 1. Inventory before touching anything
-
-```bash
-du -sh /home/dank/models/* | sort -h      # payload dirs, biggest last
-grep -rlE "models/|<model-name>" /home/dank/.config/containers/systemd/ /etc/systemd/system/*.service
-```
-
-Classify every dir: **live service payload** (referenced by a unit/env), **retired runtime
-payload**, **unreferenced model dir**, **tool/script dir**, **logs** (evidence — keep). Check the
-service env files (`scripts/*.env`) for `MODEL_DIR`/`MODEL` paths, not just unit `Volume=` lines:
-a payload can be mounted by path and named by env. `models/logs` and small tool dirs are not cache.
-
-## 2. Record fingerprints first
-
-`ls -l --block-size=1 <dir>` gives sizes; for the LLM payloads compare each to the Hub's declared
-size (`/api/models/<repo>/tree/main?recursive=true&blobs=true`). They can differ — see §6. If a
-file is irreplaceable upstream (delisted revision, gated repo) say so before deleting it.
-
-## 3. Protect live payloads by delete-then-refetch, never delete-only
-
-Deleting a live payload leaves the endpoint unable to start. Know the recovery path *before*
-deleting: some engines fetch their own weights (halogen: `HALOGEN_DOWNLOAD`, triggered when the
-checkpoint is absent), others need a manual fetch (llama.cpp GGUF services: the serve script
-refuses to start without the weights).
-
-**Bind-mount trap:** podman refuses a mount whose source directory is missing —
-`Error: statfs /home/dank/models/<dir>: no such file or directory`, with the unit looping on
-`Restart=on-failure`. Recreate the directory **empty** (`mkdir -p`), and the engine's first-start
-fetch runs.
-
-## 4. Fetch with verification (never trust an exit code)
-
-`curl -o FILE URL` exits 0 on 401/403 and writes the error body — a gated repo produced three
-153-byte shards with identical sha256s and a cheerful DONE line. Always: `-fL -C -`, compare the
-byte count against the Hub's declared size, and compare `sha256sum` against the LFS oid. Public
-repos means **no token in argv** (it is world-readable in `/proc/<pid>/cmdline`).
-
-**Parts trap:** mradermacher-style repos ship large quants as `.part1ofN … .partNofN` — a filter on
-`endswith('.gguf')` makes those quants look absent. Fetch the parts, verify each, then
-`cat part1 part2 part3 > name.gguf` in numeric order; the assembled file's size equals the sum
-(concatenation is deterministic, so per-part hashes verify the whole).
-
-## 5. Restore, then verify behaviour — not health alone
-
-Put the quadlet back (`mv` from trash + `systemctl --user daemon-reload`), `reset-failed`, `start`;
-root units need `sudo`. Watch the engine's own fetch/load lines in the journal, then check
-`/health`, `/v1/models`, and a **real completion** on every restored endpoint. Read the engine's
-memory ledger (kv pool / weights / blocks left) rather than assuming the config still fits.
-
-## 6. A refresh is also an update check
-
-Clearing and re-fetching only guarantees freshness *at fetch time* — if upstream has not changed,
-the bytes come back identical, which is itself the finding. Compare recorded fingerprints against
-the new local sizes and the Hub's declared sizes: sidecars are often versioned **in place**, so a
-stale copy is invisible from its file name (observed: `overlay.hgn` 2,477,677,120 → 2,572,466,560
-while the main checkpoint returned byte-identical).
-
-## 7. Close out
-
-- Clear test payloads once the benchmark using them is done, and keep the payload the *next*
-  experiment needs.
-- Leave the operator's decision points explicit: unreferenced-but-licensed dirs, other services'
-  caches, and whether they want the payloads absent at their next run or fresh as of now.
-- Record sizes/verdicts in the vault (runbook + `log.md`); never `rm` without explicit permission,
-  and remember a same-filesystem trash reclaims no space.

agent/managed-skills/fleet-monitoring-gap-audit/SKILL.md

@@ -1,79 +0,0 @@
----
-name: fleet-monitoring-gap-audit
-description: "Audit a git-declared homelab fleet for monitoring gaps: find services that should exist but silently don't (compose-only drift), classify each down target as outage vs probe defect, and check exporter bind addresses."
----
-
-# Auditing a homelab fleet's monitoring for what *should* be there
-
-Use after any reboot, or whenever Prometheus/Targets looks wrong on a fleet whose
-services are declared in a git repo. The two failure modes this catches are
-different, and only the first announces itself.
-
-## 1. A service that should exist and doesn't is invisible
-
-A missing exporter shows up as **nothing**: no failing target if the job was
-configured for a host that no longer serves, no alert because the series is
-simply absent. So check existence, not health:
-
-```bash
-# Every container the host actually runs, including dead ones
-podman ps -a --format '{{.Names}} {{.Status}}'
-# Every container that is supposed to be boot-persistent
-ls ~/.config/containers/systemd/*.container
-systemctl --user is-enabled <name>.service   # expect: generated
-```
-
-A container in `ps -a` that is `Exited (…)` **and has no unit** is the classic
-case: rootless Podman does not restart bare containers after the stop that
-shutdown performs, so a compose-only service dies at the first reboot and stays
-dead. Check the compose file for the name — if it is there and has no unit, the
-repo's own rule is being violated. Fix by writing the unit (mirror the same
-service on the other host), deleting the compose entry (one owner per container
-name), and making sure the deploy path can reach it:
-
-- planner: service in the target's service set + a path rule for its directory
-- dispatcher allowlist: `<target>:<service>`
-- handler: install from `"$TARGET/..."`, never a hardcoded peer's path
-
-## 2. Classify every down target — do not report a count
-
-`targets?state=active` gives health **and** the last error. Two down targets can
-mean opposite things:
-
-| Symptom | Meaning |
-|---|---|
-| `connection refused` | the service is not listening → real outage (check `podman ps`, logs, exit codes) |
-| `404` on `/metrics` | the exporter path is wrong or metrics are disabled → config defect, service fine |
-| `unsupported Content-Type …` | you are scraping an HTML *page* as if it were metrics → the probe is inert; Prometheus 3 refuses HTML. A "blackbox-style probe" needs a blackbox exporter |
-| target absent entirely | see §1 |
-
-```bash
-curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | python3 -c "
-import json,sys
-for t in json.load(sys.stdin)['data']['activeTargets']:
-    if t['health'] != 'up':
-        print(t['labels']['job'], t['scrapeUrl'], (t['lastError'] or '')[:80])"
-# and probe each by hand before believing the scraper
-curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http://<target>/metrics
-```
-
-Say which are down and why. "7 of 11 up" without the four reasons is how a real
-outage hides behind three config defects — that is exactly what one fleet did:
-its crash-looping GPU service was one of four failures reported as a number.
-
-## 3. Check the exposure posture while you are in there
-
-An exporter that binds the wildcard publishes an unauthenticated API on every
-interface. Expect exactly two listeners per host-exporter — loopback for
-host-local probes, the host's own LAN address for the Prometheus container:
-
-```bash
-ss -ltn | grep <port>    # must show the two explicit addresses, never 0.0.0.0
-```
-
-## 4. Leave the decisions that are not yours
-
-Fix mechanical drift (missing unit, wrong path, wildcard bind). Do **not**:
-disable a probe to make a dashboard green, or "fix" a crash-looping service whose
-correct posture depends on what the owner wants — capture the evidence (restart
-count, exit code, last error line) and hand it over intact.

agent/managed-skills/fleet-second-llm-endpoint/SKILL.md

@@ -1,97 +0,0 @@
----
-name: fleet-second-llm-endpoint
-description: "Add a second LLM endpoint to a host that already serves one on the Dank fleet (new port, unit shape, co-tenancy budget, harness provider/roles, monitoring, cutover, verification) — with the port-bind race, the reasoning_effort template trap, and the tools/--jinja trap that silently break subagents."
----
-
-# Adding a second LLM endpoint to a Dank fleet host
-
-Use when a host must serve two models at once (e.g. an iGPU runtime plus a
-second card), or when one endpoint is moving and a new one takes the freed
-hardware. Rebuilt from the 2026-09-16 Dankclaw split (halogen `frank-large` on
-:7770 + llama.cpp `frank-small` on :7771).
-
-## 0. Decide the shape before touching hardware
-
-- **One provider per HOST:PORT** in `~/.omp/agent/models.yml` (on the box where
-the harness runs, not on the servers). A second endpoint is a new provider
-block, not a second model in the existing one.
-- **Read the budget as an inequality, not a wish.** For a unified-memory runtime
-the engine's own fit check is the arbiter (`budget = host − pinned weights −
-reserve`; `need = KV + arena + vision`); for a discrete card, weight the KV by
-counting only the full-attention layers — and remember `--parallel N` DIVIDES
-`--ctx-size`, it does not multiply.
-- **A co-tenant is a budget line, not a footnote.** Free host RAM by changing
-the POOL or the ARENA (the reserve is a fit-check term, never an allocation),
-and drop optional charges (a vision tower is ~2 GiB) rather than hoping.
-
-## 1. Repo first, and retire in the same commit
-
-1. New service files: `scripts/<svc>.env` (documented tunables), `.service`
-   (root unit for a native binary; a Quadlet only for a container),
-   `serve-<svc>.sh` (defaults + hard guards for missing artifacts).
-2. Planner: add the service to `SERVICE_ORDER` AND the target's service set in
-   `scripts/deploy-plan.py`, map its paths (including "the unit/env/serve trio
-   selects only me"), add it to the inventories in `deploy.sh` and
-   `deploy-remote.sh`, and the dispatch case.
-3. Retire what the cutover obsoletes into `.AGENTTRASH/` with a suffix — and
-   **check what pointed at it**: bench harnesses, a chat template only the old
-   runtime read, and `scripts/test-changed.sh` mappings are the usual victims
-   (a mapping to a test file retired earlier fails the pre-commit wrapper
-   silently for weeks).
-4. Seams: one bats file per service asserting the *invariants* (`pool == 2 ×
-   ctx`, KV bytes ≤ card budget, port separation), not just constant values.
-   A negative grep must strip comments (`grep -vE '^\s*#' file | grep -q ...`)
-   or it will match the prose that explains the very thing you removed.
-
-## 2. The port-bind race (rootless podman)
-
-pasta binds the published port at container CREATE. A predecessor that has just
-stopped still holds it, so a naive handler starts a doomed container:
-
-```
-Deactivated successfully          <- predecessor gone
-pasta: Listen failed for HOST TCP port */PORT: Address already in use
-exit 126                          <- only Restart=on-failure saves you
-```
-
-Order the handler: **retire the old manager's unit → stop THIS service's own
-unit (no-op on a first cutover) → wait for every socket on the port to clear
-(listener, accepted, TIME_WAIT) → install → start.** Without the second step a
-redeploy deadlocks against its own gate.
-
-## 3. Wire the surfaces, and refuse impossible signals
-
-- Harness roles: point the small/fast + image roles at the new endpoint, and
-  move the vision role with the vision tower (only one endpoint should own
-  each capability).
-- `shared/homepage/services.json`: a card per endpoint, probed from the
-  monitor's own host (loopback is legal there; the LAN address otherwise).
-- Prometheus: only add a job if the runtime actually exposes Prometheus text.
-  Halogen publishes `/health`, `/cache` and `/v1/*` and nothing else — scraping
-  it would be red by construction, so the job is REMOVED with the reason written
-  down rather than left as a permanent red.
-
-## 4. Cutover, then verify by driving it
-
-```bash
-./scripts/deploy.sh <target> --service <large> --service <small>   # order matters
-```
-
-Then, per endpoint: `/health`; `/v1/models` reports the alias; a chat completion
-returns content AND `reasoning_content`; **a request carrying `tools` returns
-`tool_calls`**; an image request is answered (or refused with a named reason);
-the runtime's own speculation counters move; and the unit is `enabled` /
-`generated` with linger on. Read the engine's boot ledger for a silent clamp.
-
-Finally spend one real request through the harness per changed role — routing
-labels lie, the servers' own journals do not: `journalctl -u <svc>` shows which
-endpoint actually served it.
-
-## Traps that look like model faults
-
-| symptom | cause |
-|---|---|
-| `500 tools param requires --jinja flag` | the endpoint's runtime lacks the Jinja chat-template path, so every tool-using agent on it is broken. mainline llama.cpp defaults it on — pin it |
-| `500 Jinja Exception: Unexpected reasoning effort <level>` | the model's own template has a narrower accepted set (Qwen3.8: xhigh/medium/low only) than the harness's levels; fold them in `compat.reasoningEffortMap` |
-| the same request works by hand but a subagent dies | subagents run at a different thinking level — check both |
-| the container dies instantly although the predecessor "stopped" | the port race above |

agent/managed-skills/fleet-service-precommit-iteration/SKILL.md

@@ -1,64 +0,0 @@
----
-name: fleet-service-precommit-iteration
-description: "Iterate on a service that lives in the serverAppConfigs git checkout on a fleet host (dankclaw/dankserver) before committing: ship files for live testing, keep the checkout pullable, then commit and deploy — covers the untracked-files pull trap that recurs every session."
----
-
-# Iterating on a fleet service before you commit
-
-This repo's contract is git → `deploy.sh` → host. But live testing needs the new
-file *on the host* before it is committed, and the naive way (copy it into the
-host checkout) blocks the very deploy that would install it. This is the loop
-that works, plus the traps that cost real time.
-
-## The loop
-
-1. Edit in the local repo. Run the seams: `./tests/run.sh`.
-2. Ship the working files to the host **for testing**:
-   `cat local/file | ssh <host> 'cat > /home/dank/serverAppConfigs/<path>'`
-   (then `chmod +x` if it is executed — scp/`cat >` drops the exec bit, and a
-   container entrypoint mounted from that path fails with status 126, "command
-   found but not executable").
-3. Test against the live service; iterate until it works.
-4. **Before committing**, prove the host copy is byte-identical to what you are
-   about to commit: `sha256sum` locally and on the host. If they differ, you are
-   about to lose work in step 5.
-5. Commit and push.
-6. Clean the host checkout so the deploy can pull:
-   - untracked hand copies → move them out of the checkout
-     (`mkdir -p /tmp/handcopy && mv <paths> /tmp/handcopy/`);
-   - modified tracked files → `git checkout -- <paths>` (this *discards* the
-     hand copy, which is safe only because of step 4);
-   - confirm with `git status --short` printing nothing.
-7. `./scripts/deploy.sh <target> --service <svc>` — it pushes, pulls ff-only,
-   installs units/config, restarts, waits for health, then runs the Playwright
-   e2e gate.
-
-## Traps, in the order they bite
-
-- **The pull aborts on hand copies.** `deploy.sh` pulls with ff-only and git
-  refuses: *"The following untracked working tree files would be overwritten by
-  merge"*. Any untracked file in the checkout whose path the pull wants to
-  create stops the deploy — which is exactly the file you just shipped. Always
-  do step 6.
-- **`git checkout --` silently eats uncommitted work.** Without step 4 you can
-  destroy the only copy of an edit that exists nowhere else.
-- **A runtime mounts files *straight out of the checkout*.** Here
-  `dankserver/frank-medium/**` and `shared/grafana/provisioning/**` are mounted
-  read-only into containers, so a hand copy takes effect at the next container
-  restart with no deploy. Test that way, then let the deploy make it official.
-- **Root-owned files from a container write** (e.g. a patched `.so` produced
-  inside a build container) can land in the checkout as `root:root`; `chown` them
-  back to the deploy user before git operations.
-- **A restart is not a verification.** Health endpoints answer from a stale
-  process, a proxied service can be served by a peer, and a provisioned
-  dashboard may live in unified storage rather than the SQLite table you grep.
-  Verify the *specific* claim: the new code path's own output, the file's
-  checksum on the host, the API that serves the resource.
-
-## Why not keep a scratch copy on the host?
-
-Because the deployed artifact must be the committed one. Iterating outside git
-(the `/tmp` loop is fine for *probes*) is for measurement scripts; anything the
-service actually runs has to land in the checkout, and the checkout has to stay
-pullable. Two directories, one rule: probes may live anywhere, runtime state
-lives in git.

agent/managed-skills/flm-npu-pack-registration/SKILL.md

@@ -1,27 +0,0 @@
----
-name: flm-npu-pack-registration
-description: "Runbook for registering a custom FastFlowLM NPU2 model pack on a tag that isn't in FLM's built-in catalog (Dankclaw :7771), including the LFS-pointer and vision-weight traps and the NPU hwctx contention rule."
----
-
-# FLM custom-tag pack registration (Dankclaw NPU endpoint)
-
-Use when adding a non-catalog NPU2 pack (e.g. an extraction or fine-tuned model) to the FLM registry served by `frank-npu` on port 7771. Companion diagnosis skill: `flm-tag-load-failure-diagnosis`.
-
-## Registry facts
-- User registry: `/home/dank/.config/flm/model_list.json`, shape `{"model_path": "models", "models": {"<tag>": {"<size>": {name, url, file_url, size, flm_min_version, files[], vlm, default_context_length, max_prefill_len, details{format:"NPU2", family, ...}, label[]}}}}}`.
-- Serve resolves tags through `FLM_CONFIG_PATH` at startup; `flm list` with the env var set lists custom tags (`⏬` = not downloaded).
-
-## Procedure
-1. Back up the registry JSON first (`cp -a`).
-2. Insert the entry via a python json edit + atomic `os.replace`. Set `files` to the FULL HF file list — **including `vision_weight.q4nx`** even for text-only packs: the qwen3.5-family loader opens it unconditionally and a missing file makes the first request HANG silently (no error log).
-3. **Do not use `flm pull <tag>` for custom entries** — it throws `json.exception out_of_range key '<tag>:<size>' not found` while building the download list. Place files manually into `/home/dank/.config/flm/models/<PackName>/` with:
-   `curl -sfL "https://huggingface.co/<repo>/resolve/main/<file>?download=true" -o <file>`
-   Plain `raw/main/<file>` returns 135-byte LFS pointer stubs, not weights — verify byte counts against the HF tree API before trusting a download.
-4. Kernel packs: copy from the same-shape family pack (shape-match `config.json` on hidden_size/num_hidden_layers/head_dim first) into `/opt/fastflowlm/share/flm/xclbins/<PackName>/` via `sudo cp -a` (root-owned dir).
-5. Deploy/restart the serve unit (deploy path, not ad-hoc restarts) so the new process loads with all files present.
-6. Verify with a REAL completion request on 7771 (`model: "<tag>:<size>"`) — `/v1/models` alone proves registration, not loadability.
-
-## Hard constraints
-- Only ONE `flm serve` process can hold the NPU hardware context. A second concurrent serve process fails every load with `DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22` (aie2 mgmt msg opcode 0x2 status 0x2000003). Never run scratch/verify instances while production holds the NPU — verify via the deployed service instead.
-- Tag switching happens INSIDE one serve process; first request after a model switch pays ~6–11 s cold-load cost (visible in `usage.load_duration`).
-- Over ssh, never `pkill -f "<pattern>"` with a pattern present in your own command line — it kills your own session; use bracket-guarded patterns (`[q]wen3.5:4b`) or exact PIDs.

agent/managed-skills/flm-tag-load-failure-diagnosis/SKILL.md

@@ -1,34 +0,0 @@
----
-name: flm-tag-load-failure-diagnosis
-description: "Diagnose FLM (FastFlowLM) NPU serve failures where a tag listed in /v1/models answers \"Failed to load model: No such file ...xclbin\" — missing kernel-pack symlink under FLM_XCLBIN_PATH is the root cause; includes per-request switch-cost behavior."
----
-
-# FLM tag load-failure diagnosis (Dankclaw sight+sound endpoint)
-
-## Symptom
-`POST /v1/chat/completions` on the FLM serve endpoint answers **HTTP 200** with a flat error body `{"error":"Failed to load <tag> model!"}` and no `choices`. `/v1/completions` instead returns a nested OpenAI-shaped `server_error` (code 500) also under HTTP 200. A status-code-only client reads this as a successful empty completion — always check for the `error` key.
-
-Two distinct causes, distinguish via `journalctl -u frank-default-sightsound.service`:
-
-1. **Unknown tag** (not in the registry): immediate error, nothing loaded. Correct behavior; not a bug.
-2. **Registered tag whose kernel pack is missing**: FLM resolves the tag to its pack dir, downloads missing weight files on the fly (first request hangs ~90–110 s downloading), then fails with `Failed to load model: No such file '/home/dank/.config/flm/xclbins/<PackName>/<kernel>.xclbin'`. Subsequent attempts fail fast (~2 s).
-
-## Root cause of case 2
-The serve script exports `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins` (user dir), so **every** kernel lookup resolves under the user dir — but that dir only contains symlinks for the packs someone linked. A tag listed in `/v1/models` is NOT proof it is loadable on this host.
-
-## Fix
-One symlink per missing pack, run on Dankclaw:
-```
-ln -s /opt/fastflowlm/share/flm/xclbins/<PackName> /home/dank/.config/flm/xclbins/<PackName>
-```
-Verified 2026-08-30: `Qwen3.5-4B-NPU2` — after the symlink, first request loaded and answered in 7.4 s wall (cold load + generate).
-
-## Switch-cost behavior (measured same day)
-- The `model` field routes per-request; no restart needed to switch between registry tags.
-- A request whose model differs from the resident one triggers a full reload ("Loading model:" in journal before prefill): measured 7–11 s wall for an 8-token completion on ~4B-class packs. Same-tag requests are fast; alternating tags pays the swap each time.
-- Single-lane NPU: concurrent requests queue FIFO behind the current lock holder (`NPU Locked!` → `NPU Lock Released!`).
-
-## Probe recipe (repeatable)
-1. `curl -s http://192.168.0.143:52626/v1/models` — registry listing is authoritative for *registered* tags, not loadability.
-2. On a failed load, grep the journal for `Failed to load model` (use `--full`) — the quoted xclbin path names the missing pack dir exactly.
-3. Compare `ls /home/dank/.config/flm/xclbins/` (symlinks = installed) against `/opt/fastflowlm/share/flm/xclbins/` (all available packs); diff = fixable with symlinks.

agent/managed-skills/frank-medium-batch-split-retune/SKILL.md

@@ -1,95 +0,0 @@
----
-name: frank-medium-batch-split-retune
-description: "Retune frank-medium (Dankserver, llama.cpp layer split across 2x R9700) batch/ubatch geometry and the tensor-split balance: sweep with scripts/bench-batch-ubatch.sh, re-measure the split at the deployed shape behind a VRAM settle gate, re-fit the seam, deploy, verify."
----
-
-# frank-medium batch geometry + split rebalance
-
-Use when changing `--batch-size`, `--ubatch-size`, `--ctx-size`, `--parallel` or the
-model on frank-medium, or when someone reports that prefill "does not respond" to
-those knobs. The batch and the tensor split are **coupled**: the compute buffers are
-not symmetric across the two cards, so any batch change moves the balance point.
-
-## Order of work (do not skip the second half)
-
-1. **Sweep the batch geometry with llama-bench**, which is the only tool that can hold
-   a shape constant across dozens of configs:
-   `scripts/bench-batch-ubatch.sh`, defaults `-p 65536 -n 128 -d 65536`, one invocation
-   per cell, `-r 3 --no-warmup`.
-2. **Re-measure the split at the DEPLOYED shape** (262144, two slots, drafter loaded) by
-   loading the real service with a per-config env override and reading `rocm-smi`. A
-   llama-bench number cannot decide the split: it has no `--parallel`, sizes its own
-   context as `n_prompt + n_gen + n_depth`, and cannot load the drafter.
-3. **Re-fit the balance seam** in `tests/frank_medium_unit.bats` on the new deployed
-   point, then **deploy** and verify.
-
-## 1. The sweep
-
-```bash
-# from a transient unit, never a shell: the deploy's LimitMEMLOCK=infinity is
-# load-bearing for ROCm staging buffers and an SSH shell gets 8 MiB here
-sudo systemd-run --unit=fm-bench --collect --pipe --property=Type=oneshot \
-     --property=LimitMEMLOCK=infinity --property=User=dank \
-     --setenv=OUT=/home/dank/models/logs/bench-bub \
-     --setenv=BATCHES=6144 --setenv=UBATCHES="256 384 512 768 1024 1536 2048" \
-     /home/dank/serverAppConfigs/scripts/bench-batch-ubatch.sh
-```
-
-The script stops `frank-medium`, runs one `llama-bench` per cell into `cells.tsv`
-(appended as it goes, so progress is readable), samples VRAM per cell, and restarts
-the service from an EXIT trap. A re-run skips cells that already have a `bench.json`.
-
-Read `n_ubatch` out of each `bench.json` to prove the knob was applied — if it does not
-track the request, nothing else in the table means anything. The VRAM columns scaling
-monotonically with ubatch is the independent confirmation.
-
-Expected shape (2026-09-16, 2x R9700, UD-Q8_K_XL + DFlash2, 64k prefill at 64k depth):
-an optimum at **ubatch 512** (~1193 tok/s) with losses on both sides (256 -> 1046,
-384 -> 1068, 2048 -> 1077). The logical batch is throughput- and VRAM-neutral
-(2048..8192 within 2 %), and decode never moves (15.88 tok/s at depth 65536 on every
-cell). Test **both** ends of the range: sampling only upward makes a shallow basin look
-like a plateau.
-
-Sanity rule: the same ubatch measured under different logical batches should agree to
-under 1 tok/s. If it does not, the table is noise and the first suspect is a stray
-GPU user, not the knob.
-
-## 2. The split, at the deployed shape
-
-`/tmp/measure-ub512b.sh` (pattern; rewrite per run) copies `scripts/frank-medium.env`
-to a temp file with `sed` overrides, launches `scripts/serve-frank-medium.sh` with
-`FRANK_MEDIUM_ENV_FILE=<temp>`, polls `/health` for `"status":"ok"`, reads
-`rocm-smi --showmeminfo vram --csv`, kills, and repeats per split. Run it under
-`systemd-run --pipe --property=LimitMEMLOCK=infinity --property=User=dank`.
-
-**The settle gate is mandatory.** Between configs, wait until `rocm-smi` shows BOTH
-cards under ~1 GiB used before the next load. amdgpu does not return ~26 GiB when the
-process dies — a 180 s sleep fails, and even 600 s has failed while the process is still
-alive inside the unit. Skipping the gate produces
-`error loading model: unable to allocate ROCm0 buffer`, which is indistinguishable from
-"this split does not fit" and will send you chasing a phantom OOM.
-
-Free bytes to expect, and the arithmetic behind them:
-`card = weights_share + KV_share + constant + ubatch_term`, where the ubatch term costs
-the drafter-carrying card roughly **1.85x** what it costs the other (measured 2.41 GiB
-against 1.30 GiB going 1536 -> 512). Balance by measurement only: one `-ts` step is one
-layer, worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on an attention one, so the
-reachable minimum gap is about one DeltaNet layer, never zero.
-
-2026-09-16 reference: 36,29 @1536 -> 4.71 / 4.91; 36,29 @512 -> 6.01 / 7.32;
-**35,30 @512 -> 6.44 / 6.90 (deployed)**.
-
-## 3. Seam, deploy, verify
-
-- Re-fit both ubatch coefficients in `tests/frank_medium_unit.bats` from the measured
-  1536 -> 512 deltas at the deployed shape, and re-anchor the per-card constants so the
-  model reproduces the new point. Check that neighbouring splits still FAIL the balance
-  gate, or the seam has stopped discriminating. Keep the floor at 1.5 GiB/card and the
-  slack at one DeltaNet layer.
-- Do not pin a literal split elsewhere: the unit test asserts the *shape* (two shares
-  summing to the model's 65 blocks, ROCm1 the smaller).
-- `./tests/run.sh`, commit, `./scripts/deploy.sh dankserver --service frank-medium`.
-- Verify live: the flags in `/proc/<pid>/cmdline`, `/slots` = 2 x 131072, VRAM matching
-  the pre-deploy measurement, a tool call, two concurrent requests, `draft acceptance`
-  lines on BOTH slots, and 0 ROCm errors in the journal. The VRAM figure sits ~0.2 GiB
-  lower once the service has served traffic than straight after load.

agent/managed-skills/frank-primary-decode-tuning/SKILL.md

@@ -0,0 +1,22 @@
+---
+name: frank-primary-decode-tuning
+description: "Tune frank-primary (radiance TP=2) decode throughput: benchmark harness usage, flag levers, known optima and boot-failure walls."
+---
+
+# frank-primary decode tuning
+
+Harness: `bash autoresearch.sh` (dev box) scp's `scripts/frank-primary.env` + `scripts/autoresearch-frank-primary.sh` to dankserver, stops prod (`systemctl stop frank-primary`, NEVER mask — relinked unit), boots scratch container `arbench` on :7770 (fresh prefix-cache tier at `$STATE_DIR/kvcache/arbench` for cold prefill), polls health 720s, sends ONE greedy completion (fixed 140k-char repo-corpus prompt -> 43,627 tokens, 256-token decode), parses `timings` into METRIC lines, tears down, restores prod via EXIT trap. ~4 min/run; decode noise ~0.3%.
+
+Metric: `METRIC decode_tps` = `timings.predicted_per_second` at 43.6k depth.
+
+## Optima (measured 2026-10-06, keep these)
+- `FRANK_PRIMARY_SPEC_TOKENS=2` (unimodal: 1->31.89, 2->32.40, 3->31.72, 4->24.57 t/s; acceptance collapses past 3)
+- `FRANK_PRIMARY_MAX_SEQS=3` with `MAX_MODEL_LEN=200000` (600k KV; +4.2% total; 4 slots regresses)
+- `FRANK_PRIMARY_EXPERT_VS_CACHE=0.82` bracket-confirmed (0.78 fails boot, 0.84 -8.9%, 0.87 collapse)
+- `TP_WIRE=wht6`, `MAX_BATCHED_TOKENS=2048` — REQUIRED at seqs=3: bf16 wire and MNBT 4096 both fail boot (VRAM)
+
+## Non-levers (decode-neutral, don't spend runs)
+GPU_HEADROOM_MIB (48 = neutral), HOST_POOL_MIB 49152 (neutral), MAX_MODEL_LEN any change (regresses: bigger KV steals resident weights), prefix-cache tiers (prefill-side).
+
+## Rules
+One flag per iteration; keep only > +1% decode over best; a boot failure is `crash` (trap restores prod). Constraint floor: MAX_SEQS*MAX_MODEL_LEN >= 400000.

agent/managed-skills/frankbook-qbittorrent-e2e-verify/SKILL.md

@@ -1,57 +0,0 @@
----
-name: frankbook-qbittorrent-e2e-verify
-description: "Verify the FrankBook-Pro gluetun→qBittorrent grab pipeline end-to-end (egress gate inside gluetun, generic SID login, legal test torrent, category path check, data-safe cleanup) after any tunnel/qB/category change"
----
-
-# FrankBook-Pro qBittorrent E2E verification
-
-Use after any change to the FrankBook-Pro torrent pipeline — WireGuard credential rotation, qB version change, gluetun reconfig, category re-seeding. Proves the full grab path (tunnel egress → qB download → category save path → clean removal). All commands run from Dankware via `ssh frankbook-pro`; inside the Mac shell export `PATH="/usr/local/bin:/opt/homebrew/bin:$PATH"` for `podman machine ssh`.
-
-## 1. Egress gate — MUST run inside gluetun
-
-```bash
-podman machine ssh 'podman exec gluetun wget -qO- https://ipinfo.io/json'
-```
-
-- Require `org` = AS9009 (M247 — ProtonVPN Panama PA-12 config). Judge by org, never country alone.
-- NEVER run this one ssh-context out (e.g. bare `wget` on the Mac host): it silently returns the residential Comcast IP and reads as a hard-gate leak that does not exist. If a check shows a residential IP, first re-run inside the container before diagnosing.
-
-## 2. Login — generic cookie parse
-
-The QPW fetch is flaky (~1 in N returns empty); assert length before use. An empty-password run once produced a Forbidden loop that cost hours.
-
-```bash
-QPW=$(podman machine ssh 'cat /home/core/Containers/qbittorrent/env' | grep QBIT_WEBUI_PASSWORD | cut -d= -f2)
-[ ${#QPW} -eq 20 ] || exit 1
-SID=$(curl -si --data-urlencode "username=admin" --data-urlencode "password=$QPW" \
-  http://127.0.0.1:8080/api/v2/auth/login \
-  | grep -oE '[A-Za-z_]*SID[A-Za-z0-9_]*=[^;[:space:]]*' | head -1)
-curl -b "$SID" ...   # pass the WHOLE match
-```
-
-- qB 4.x cookie name is plain `SID`; qB 5.x uses `QBT_SID_<port>`. Hard-coding either name breaks the other version's parse (sidlen=0 → 403 cascades).
-- Parse-bug 403s never reach qB as failed logins — no ban accrues. Check server-side truth in the log: `/Volumes/COMPLETE/qbittorrent-data/qBittorrent/config/qBittorrent/logs/qbittorrent.log` (`WebAPI login success/failure` + source IP). Repeated REAL failures accrue toward qB's 5-failures/3600s IP ban.
-
-## 3. Add a legal test torrent
-
-Blender open-movie torrents (~300 MB, e.g. Big Buck Bunny) via `POST /torrents/add` with the raw `.torrent` file (`-F "file=@..."`) and `category=Finished Torrents`. Never leave a real-looking grab behind.
-
-## 4. Poll and verify
-
-Poll `GET /torrents/info` (~20 s): state `downloading` → `moving` (INCOMPLETE→COMPLETE move) → `stalledUP`/`uploading` at progress=1. `save_path` must be `/Volumes/COMPLETE/Finished Torrents` (qB 4.x auto-creates unknown categories at DefaultSavePath; 5.x did NOT inherit category paths — one more reason for the 4.6.7 pin). Confirm on disk: `ls "/Volumes/COMPLETE/Finished Torrents/"`. Throughput ~17 MB/s is normal for the tunnel.
-
-## 5. Cleanup — delete only your artifact
-
-```bash
-curl -s -o /dev/null -w "%{http_code}" -b "$SID" \
-  --data-urlencode "hashes=<literal-hash>" \
-  --data-urlencode "deleteFiles=true" \
-  http://127.0.0.1:8080/api/v2/torrents/delete   # expect 200
-```
-
-- Target the known hash literally — the user's real grabs share the directory and MUST be untouched. Verify: hash absent from `/torrents/info`, your dir gone, all other entries intact.
-
-## Version-coupling constraints
-
-- qB is pinned to 4.6.7 (readarr's EOL client rejects qB ≥ 5's `204` login at runtime). Do not unpin while readarr exists.
-- After ANY qB version flip, regenerate the `WebUI\Password_PBKDF2` line fresh (same password, 100k iterations) — a 5.x-written hash line makes 4.6.7 reject every login. Rewrite while qB is stopped.

agent/managed-skills/freetoken-build-dankclaw/SKILL.md

@@ -1,55 +0,0 @@
----
-name: freetoken-build-dankclaw
-description: "Build FreeToken on Dankclaw (Ubuntu 24.04) with CUDA — installs nvidia-cuda-toolkit, sets CUDA_HOME=/usr, overrides nvcc/CUDA version mismatch"
----
-
-# FreeToken Build on Dankclaw
-
-Dankclaw is Ubuntu 24.04 with ROCm already installed. FreeToken builds for **CUDA** (not ROCm) because the kernel package (`_pinned_tensor`) links against `cudart`.
-
-## Prerequisites (install if missing)
-
-```bash
-export PATH="/home/dank/.local/bin:$PATH"
-export CUDA_HOME=/usr
-export FREETOKEN_ALLOW_CUDA_MISMATCH=1
-
-# CUDA toolkit (if not present):
-sudo apt-get install -y nvidia-cuda-toolkit
-# UV (if not present):
-curl -LsSf https://astral.sh/uv/install.sh | sh
-```
-
-## Key environment variables
-
-|Variable|Value|Why|
-|---|---|---|
-|`CUDA_HOME`|`/usr`|Ubuntu CUDA toolkit package places nvcc at `/usr/bin/nvcc` and libcudart at `/usr/lib/x86_64-linux-gnu/`|
-|`FREETOKEN_ALLOW_CUDA_MISMATCH=1`|Set|nvcc is 12.4 but torch 2.11.0+cu130 ships CUDA 13.0 — safe to override|
-|`PATH`|`/home/dank/.local/bin:$PATH`|uv installs here|
-
-## Build procedure
-
-```bash
-cd /home/dank
-git clone https://github.com/FlashML-org/FreeToken.git
-cd FreeToken
-uv venv && source .venv/bin/activate
-uv pip install -e ".[accel]"
-```
-
-## Verification
-
-```bash
-ft --version                    # → freetoken version 0.1.2
-python3 -c "import torch; print(torch.__version__)"  # → 2.11.0+cu130
-python3 -c "from freetoken.kernel import _pinned_tensor; print(_pinned_tensor.__file__)"
-python3 -c "import flashinfer; print(flashinfer.__version__)"  # → 0.6.18
-python3 -c "import sgl_kernel; print(sgl_kernel.__version__)"  # → 0.4.5
-```
-
-## Notes
-
-- `torch.cuda.is_available()` may return `False` with "Can't initialize NVML" warning — this is the management library, not CUDA runtime. The kernels are built and linked correctly.
-- The NVIDIA driver modules may or may not be loaded; they don't affect the build.
-- ROCm is present on Dankclaw but FreeToken's kernel build specifically requires CUDA (links `cudart`).

agent/managed-skills/freetoken-dankclaw-build/SKILL.md

@@ -1,39 +0,0 @@
----
-name: freetoken-dankclaw-build
-description: "Build FreeToken on Dankclaw — installs CUDA toolkit, sets CUDA_HOME, handles nvcc/CUDA version mismatch"
----
-
-## Build FreeToken on Dankclaw
-
-### Prerequisites
-- CUDA toolkit: `sudo apt install nvidia-cuda-toolkit` (gives nvcc at /usr/bin, libcudart at /usr/lib/x86_64-linux-gnu/)
-- uv: installed at /home/dank/.local/bin/uv (or install via `curl -LsSf https://astral.sh/uv/install.sh | sh`)
-- CMake, git, Python >= 3.10 — all present
-
-### Build steps
-```bash
-cd /home/dank
-git clone https://github.com/FlashML-org/FreeToken.git  # or cd into existing
-cd FreeToken
-uv venv && source .venv/bin/activate
-export CUDA_HOME=/usr
-export FREETOKEN_ALLOW_CUDA_MISMATCH=1  # nvcc 12.4 + torch cu130 is safe
-uv pip install -e ".[accel]"
-```
-
-### Verification
-```bash
-ft --version
-python3 -c "
-from freetoken.checkpoint import ftw
-import torch
-t = torch.randn(3,4,dtype=torch.float16)
-print(ftw._dtype_str(t.dtype), ftw._elsize(t.dtype))
-"
-```
-
-### Key notes
-- FTW format is dtype-agnostic: FP16, BF16 (default), FP32, FP8 all work
-- GGUF support: only Gemma4GGUFForCausalLM
-- GGML types: F32/F16/BF16 (native views), Q4_0/Q6_K (dequant), Q8_0 (listed, no dequant)
-- The NVIDIA driver modules may not be loaded at build time (NVML warning) — not a problem for the build itself

agent/managed-skills/freetoken-hf-model-swap/SKILL.md

@@ -1,36 +0,0 @@
----
-name: freetoken-hf-model-swap
-description: "Swap the FreeToken model behind frank-default-small on Dankserver: preflight the HF quant format against the loader matrix BEFORE downloading, then download, repoint MODEL_DIR, deploy, and verify with a REAL completion (health 200 is not enough)."
----
-
-# FreeToken HF Model Swap (frank-default-small, Dankserver)
-
-Swap the model behind `frank-default-small` (FreeToken v0.1.2, Dankserver, RTX 3090) by downloading a HF safetensors checkpoint into a model dir, repointing `MODEL_DIR` in `scripts/frank-default-small.env`, and deploying. Verify with a REAL completion — health 200 is NOT enough.
-
-## Preflight — verify quant format BEFORE downloading
-
-FreeToken v0.1.2 loads ONLY modelopt-style quant configs for `qwen3_5_moe`. Every compressed-tensors FP8 checkpoint fails (live-verified 2026-09-01: two candidates, 72 GB downloaded, both unloadable). Check before any multi-GB download:
-
-```bash
-curl -sL https://huggingface.co/<org>/<repo>/raw/main/config.json | \
-  python3 -c "import json,sys; c=json.load(sys.stdin); q=c.get('quantization_config',{}); print(q.get('quant_method'), q.get('weight_block_size'), list((q.get('weight_scale') or ['?','?'])) if isinstance(q.get('weight_scale'),list) else q.get('weight_scale_inv') is not None)"
-```
-
-Decision matrix:
-
-- `quant_method: "fp8"` + `weight_block_size: [128,128]` + `weight_scale_inv` → ✅ modelopt block-fp8 (the Huihui/official-Qwen format; only proven FP8 path)
-- `quant_method: "compressed-tensors"` → ❌ FAILS. Per-channel aborts at expert-source resolution (`Missing MoE expert source layers: {'gate_up'}` when expert tensor naming is nonstandard); block-fp8 128×128 aborts in `_dequant_fp8_weight` (multiplies `weight × weight_scale` directly; block scales don't broadcast → shape RuntimeError)
-- No quant config (BF16) → loads but 35B = ~70 GB weights >> 62 GB system RAM → sustained swap-thrash that locks out SSH (2026-09-01 outage). Never deploy BF16 for a 35B-class model on this box; a clean preflight compares disk size × precision against free RAM + 24 GB VRAM headroom.
-- Also check: architecture `qwen3_5_moe`, expert tensor naming `mlp.experts.{N}.gate_proj/up_proj/down_proj` (read `model.safetensors.index.json` weight_map on HF if unsure — nonstandard prefixes defeat the MoE expert-source provider), config.json ships the chat template you want (FreeToken reads it from the checkpoint), disk space ≥ checkpoint size.
-
-If the checkpoint fails the matrix, STOP — do not download, do not touch the running service. Find a compliant quant or use a different runtime.
-
-## Swap steps
-
-1. Download into `/home/dank/models/<name>/` via `snapshot_download` with the freetoken venv python (background the SSH call; poll until file count matches the index).
-2. Edit `scripts/frank-default-small.env`: `MODEL_DIR`, description line (name, quant, arch, disk size, host, date), header comment. Keep F84 pool numbers unless the weight mass or KV dtype changed (FP8 35B ≈ 35 GB: `MODEL_CACHE_SIZE=2048`, `NUM_TOKENS=453005`, reasoning `on`; BF16 66 GB fits nothing — see preflight).
-3. Run `./tests/run.sh`, commit, push to Gitea origin (git-only flow; never edit on the host).
-4. `./scripts/deploy.sh dankserver --service frank-default-small`. The LLM_WAIT window is up to 20 min; load of ~35 GB FP8 is minutes warm.
-5. **Verify for real** — the deploy health gate has reported "healthy" while a doomed worker was still starting (twice). Check journal for Uvicorn startup complete, then `curl /v1/models` (root = your model dir, ctx 262144) and a REAL completion: reasoning must parse into `reasoning_content` and content must answer correctly with a natural `finish_reason: stop`. Give reasoning models enough `max_tokens` to finish thinking (a tight budget yields empty content + `finish: length` — that is correct behavior, not a failure).
-6. If load fails: journal shows the exact loader error, service exits cleanly (no swap-thrash), `Restart=on-failure` re-loop is harmless. Revert env via `git checkout <last-good-commit> -- <three files>`, redeploy, verify.
-7. Update the runbook (`serverAppConfigs-docs/LLM/freetoken-frank-default-small.md` quant matrix + history), `log.md`, and AGENTS.md fleet table; stage superseded model dirs to `~/.AGENTTRASH/<date>-topic/` on the host and list sizes for the user — never `rm`.

agent/managed-skills/freetoken-rocm-build/SKILL.md

@@ -1,49 +0,0 @@
----
-name: freetoken-rocm-build
-description: "Verify and execute FreeToken build on ROCm hosts — check gfx1151, ROCm 6.3, hipcc, and dependency wheel availability (torch 2.11, sgl-kernel 0.4.5, flashinfer)"
----
-
-# FreeToken ROCm Build
-
-## When to use
-Verify FreeToken can build on an ROCm host (Dankclaw, similar systems) before committing to the build path.
-
-## Steps
-
-1. **Check hardware**
-   ```
-   rocminfo | grep 'Name.*gfx'      # verify GPU architecture
-   lsmod | grep amdgpu              # driver loaded
-   ls /opt/rocm/bin/ | grep hipcc   # compiler available
-   ```
-
-2. **Check build deps**
-   ```
-   cmake --version                  # needs >=3.20
-   python3 --version                # needs >=3.10
-   gcc --version                    # needs >=9
-   ninja --version                  # optional but recommended
-   ```
-
-3. **Verify wheel availability**
-   ```
-   pip index versions torch         # needs 2.11-2.12 range
-   pip index versions sgl-kernel    # needs 0.4.5
-   pip index versions flashinfer-python  # needs >=0.6
-   ```
-   Check PyPI for `rocm` in wheel filenames, or use PyTorch ROCm index.
-
-4. **Build**
-   ```
-   git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken
-   python3 -m venv .venv && source .venv/bin/activate
-   pip install -e ".[accel]"        # installs fi + sgl extras
-   ```
-
-## Known issues
-- nvcc missing on Dankclaw — not needed for ROCm build, only for CUDA JIT
-- sgl-kernel at versioned endpoint may return 404 — use unversioned `pip index versions`
-- triton 3.6.0 pins to Linux only
-
-## CUDA alternative
-If building for CUDA: install `sudo pacman -S cuda` for nvcc, or use pre-built CUDA wheels that ship pre-compiled kernels.

agent/managed-skills/grafana-prometheus-provisioning-as-code/SKILL.md

@@ -1,94 +0,0 @@
----
-name: grafana-prometheus-provisioning-as-code
-description: "Add or change Grafana datasources/dashboards and Prometheus scrape jobs/alert rules in this homelab repo as code — including the planner+copier+seam triad, the silent failure modes, and the API-based verification that avoids two false negatives."
----
-
-# Observability as code (Grafana + Prometheus, Dankclaw fleet)
-
-Use when adding a scrape target, an alert rule, a datasource, or a dashboard to
-this homelab. Everything lives in git; Grafana and Prometheus only ever read
-what the deployer installs.
-
-## The triad — every change touches three places
-
-1. **Config in the repo**
-   - `shared/prometheus/prometheus.yml` — `scrape_configs`, and `rule_files:`
-     pointing at `/etc/prometheus/rules/*.yml`
-   - `shared/prometheus/rules/<name>.yml` — alert/recording rules
-   - `shared/grafana/provisioning/datasources/prometheus.yml` — datasource
-   - `shared/grafana/provisioning/dashboards/{provider.yml,<name>.json}`
-2. **The copier in `scripts/deploy-remote.sh`** — `copy_prometheus_config`
-   installs config **and** rules into `/opt/prometheus/config/` (the container
-   mounts that to `/etc/prometheus`); `copy_grafana_provisioning` installs into
-   `/opt/grafana/provisioning/` (the grafana unit already mounts it read-only).
-   Paths are absolute and `run_privileged`.
-3. **The planner in `scripts/deploy-plan.py`** — a `shared/<group>/**` change
-   must select the service that reads it (`shared/prometheus` → prometheus,
-   `shared/grafana` → grafana). Missing mapping = the file deploys nowhere and
-   nothing reports an error.
-
-Add a seam in `tests/observability_quadlet.bats` for each change: the file
-exists, the config references it, the deployer copies it, and — where two files
-must agree (target vs published port) — that they do.
-
-## Reach targets by container name, never the loopback publish
-
-Datasources use `http://prometheus:9090` and scrape jobs for co-located
-services use `prometheus:9090` / `loki:3100`. Published ports are
-`127.0.0.1`-only by design (see the exposure rules), so a browser-facing URL or
-a published port in a datasource is wrong from inside the network. LLM
-endpoints are the exception: they are on other hosts, so use their LAN address.
-
-## Three failure modes that produce no error message
-
-- **A rule file nothing loads evaluates nothing.** Prometheus reports no
-  problem for a `rule_files` entry that matches nothing, or for a rules file
-  nobody references. Ship both together and assert the reference in a seam.
-- **A provisioned dashboard is not in the legacy `dashboard` table** on the
-  current Grafana. Reading `grafana.db` for it returns empty and looks like a
-  failed provision. It lives in unified storage behind
-  `dashboard.grafana.app`; query the API instead.
-- **Lifetime averages are not per-request evidence.** e.g. an engine's
-  "Prefix cache hit rate: 29 %" is every prompt it ever saw, including
-  benchmarks. Measure the behaviour you care about with a targeted probe.
-
-## Verify through the channel the consumer uses
-
-```bash
-# Prometheus: config and rules parse/load
-ssh dankclaw 'podman exec prometheus promtool check rules /etc/prometheus/rules/<name>.yml'
-ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/rules?type=alert" \
-  | python3 -c "import json,sys; print([r[\"name\"] for g in json.load(sys.stdin)[\"data\"][\"groups\"] for r in g[\"rules\"]])"'
-ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | ...'   # every target health=up
-ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/alerts" | ...'                  # 0 firing when healthy
-
-# Grafana: provisioned objects and a live query (admin/admin on this install)
-ssh dankclaw 'curl -s -u admin:admin "http://127.0.0.1:3002/api/search?type=dash-db"'
-ssh dankclaw 'curl -s -u admin:admin "http://127.0.0.1:3002/api/dashboards/uid/<uid>"'
-ssh dankclaw 'curl -s -u admin:admin \
-  "http://127.0.0.1:3002/api/datasources/proxy/uid/prometheus/api/v1/query?query=count(up==1)"'
-```
-
-A target that is `up` proves reachability, not data: also run one query for a
-series you expect and confirm it returns a sample.
-
-## Grafana facts for this install
-
-- No UI-configured state: `editable: false` on the datasource, fixed
-  `uid: prometheus` so dashboards reference it without a lookup.
-- Dashboards re-read from disk every 30 s (`updateIntervalSeconds`), so a JSON
-  edit appears without a restart — but the file must still be *installed*, which
-  means a deploy.
-- There is **no Alertmanager and no contact point**: rules evaluate, alerts show
-  on `/alerts`, in the `ALERTS` metric and on a dashboard panel, and reach
-  nobody. Say that plainly in docs rather than implying observability.
-
-## Naming metrics
-
-Never write a series name from memory into a config, a rule, or a dashboard —
-one shipped comment here named a metric that does not exist. Dump the live
-names first and paste them:
-
-```bash
-ssh dankserver 'curl -s http://127.0.0.1:7770/metrics | grep -E "^vllm:" | awk "{print \$1}" | sed "s/{.*//" | sort -u'
-```

agent/managed-skills/grafana-provisioning-as-code/SKILL.md

@@ -1,92 +0,0 @@
----
-name: grafana-provisioning-as-code
-description: "Provision a Grafana datasource or dashboard as code on the Dankclaw fleet (shared/grafana/provisioning, deployer copier, planner, seams) and verify it through the Grafana API — including the unified-storage trap that makes the grafana.db check a false negative."
----
-
-# Grafana provisioning as code (Dankclaw fleet)
-
-Use when adding or changing a Grafana datasource or dashboard on this fleet, or
-when a dashboard "does not appear" after a deploy.
-
-## Shape of the thing
-
-Grafana on Dankclaw is an **empty install configured entirely from git**. The
-quadlet mounts `/opt/grafana/provisioning` read-only into the container, so a
-provisioned artefact exists in exactly three places:
-
-```
-shared/grafana/provisioning/datasources/<name>.yml    # repo: source of truth
-shared/grafana/provisioning/dashboards/provider.yml   # dashboard loader config
-shared/grafana/provisioning/dashboards/<name>.json    # dashboards
-        |  deploy-remote.sh: copy_grafana_provisioning()
-        v
-/opt/grafana/provisioning/...                          # host: installed state
-        |  read-only mount
-        v
-/etc/grafana/provisioning/...                          # generated: inside container
-```
-
-Nothing is ever clicked into `grafana.db`. A UI-configured datasource would be
-invisible to review, unknown to anyone reading the repo, and lost with the data
-volume.
-
-## Procedure
-
-1. Write the files under `shared/grafana/provisioning/`.
-   - Datasource: `apiVersion: 1`, a **fixed `uid`** (dashboards reference it by
-     uid, so a random one breaks them), `access: proxy`, and the URL by
-     **container name** (`http://prometheus:9090`) — the published port is
-     loopback-only and unreachable from inside the network.
-   - Diagram: `provider.yml` with `allowUiUpdates: true` (three-person homelab;
-     failing an operator who wants a different rate window is the wrong trade)
-     and a JSON dashboard under a stable `uid`.
-2. Add the file(s) to `copy_grafana_provisioning()` in
-   `scripts/deploy-remote.sh` (it `mkdir -p`s the two directories and `cp`s each
-   file — follow `copy_prometheus_config` for style).
-3. Planner: `shared/grafana/**` → `grafana` in `scripts/deploy-plan.py`; verify
-   with `python3 scripts/deploy-plan.py --target dankclaw --paths <file>`.
-4. Add/update a seam in `tests/observability_quadlet.bats`: the JSON must parse,
-   the datasource URL must be the container name, the unit must mount
-   `/opt/grafana/provisioning`, and the deployer must call the copier. Then
-   `./tests/run.sh`.
-5. Apply: `./scripts/deploy.sh dankclaw --service grafana` (pulls, installs the
-   provisioning tree, restarts Grafana, then runs the e2e gate).
-
-## Verify — and the trap that makes the obvious check lie
-
-A **provisioned dashboard does not appear in the legacy `dashboard` table** of
-`grafana.db` on the current Grafana build; dashboards live in unified storage
-behind the `dashboard.grafana.app` API group. The datasource *does* land in
-`data_source`, and that asymmetry is easy to misread as "datasource worked,
-dashboard failed". A `sqlite3`/python query against `dashboard` is a **false
-negative**, not evidence.
-
-Check in this order instead:
-
-```bash
-# 1. provisioning actually ran (these loggers name themselves)
-podman logs --since 10m grafana | grep -iE "inserting datasource|provision dashboards"
-
-# 2. the dashboard exists (uid is what you set in the JSON)
-curl -s -u admin:admin "http://127.0.0.1:3002/api/search?type=dash-db"
-
-# 3. the datasource really reaches Prometheus (proves the whole chain)
-curl -s -u admin:admin \
-  "http://127.0.0.1:3002/api/datasources/proxy/uid/<uid>/api/v1/query?query=up"
-```
-
-Grafana publishes on loopback `:3002` only; the off-host path is the guarded
-vhost. Its admin login is still the default `admin:admin` — which is why every
-assertion above needs credentials, and which is worth changing.
-
-## Before writing any panel or config comment
-
-Grep the metric names out of the live endpoint; naming a series that does not
-exist has already shipped in this repo once (`vllm:prefix_cache_hit_rate` — the
-real pair is `vllm:prefix_cache_hits_total` / `vllm:prefix_cache_queries_total`,
-whose *rates* give the hit rate). A comment naming an unqueryable metric, or a
-panel that silently renders "No data", is worse than no panel:
-
-```bash
-curl -s http://192.168.0.235:7770/metrics | awk '{print $1}' | sed 's/{.*//' | sort -u | grep vllm
-```

agent/managed-skills/gui-hang-remote-diagnosis/SKILL.md

@@ -1,46 +0,0 @@
----
-name: gui-hang-remote-diagnosis
-description: "Diagnose a frozen GUI/VT-switch failure over SSH by reading kernel hung-task evidence, decoding the lock-holder stack, and deciding recoverability"
----
-
-# GUI Hang Remote Diagnosis
-
-Use when the desktop is frozen, VT switches do nothing, but SSH still works.
-
-## Sequence
-
-1. **Find the smoking gun** — kernel hung-task reporter already logged it:
-   ```bash
-   journalctl -b -p err --since "-2 hours" --no-pager | tail -80
-   ```
-   Look for `INFO: task X blocked for more than N seconds` and `blocked on a semaphore likely last held by task Y:PID` (modern kernels name the lock holder outright).
-2. **Get full stacks** for victim and holder:
-   ```bash
-   journalctl -b -k --since "<hang minute -5>" --no-pager | tail -120
-   cat /proc/<holder_pid>/status | head -5   # State: R (spinning) vs D (blocked)
-   cat /proc/<victim_pid>/status | head -5
-   ```
-   - Victim `D` + holder `R` = livelock/deadlock with lock held.
-   - `voluntary_ctxt_switches` in the millions on an `R` kthread = spinning, not progressing.
-3. **Prove whether hardware or software** — GPU/CPU still alive? No Xid errors?
-   ```bash
-   nvidia-smi --query-gpu=driver_version,pstate,temperature.gpu,clocks_event_reasons.active --format=csv
-   journalctl -b -k --no-pager | grep -i xid
-   ```
-   If those answer instantly with no Xid, it's a driver lock issue, not hardware.
-4. **Find the trigger window**: block start = (first hung message timestamp − reported seconds). Then inspect user-session events:
-   ```bash
-   journalctl -b --since "<start-2m>" --until "<start+1m>" --no-pager | grep -v audit
-   ```
-5. **Enumerate collateral**: any other `D` tasks (`ps -eo pid,stat,comm | awk '$2 ~ /D/'`), USB workers, etc.
-
-## Recoverability verdict
-
-- Compositor in `D` state → unkillable; spinning kernel thread holding the lock → cannot be poked.
-  **Only exit is reboot**: `sync && systemctl reboot --force --force` (double-force skips stop-jobs that wait forever on D-state PIDs).
-- If holder is `D` (sleeping on something that can complete, e.g. storage), recovery may be possible by clearing that dependency first.
-
-## After diagnosing
-
-- Map stack symbols to driver internals to name the exact subsystem (e.g. `DifrPrefetchEventDeferredWork` → NVKMS DIFR prefetch).
-- Check upstream: search the exact symbol names + driver version; propose targeted off-switches (`NVreg_RegistryDwords`, module params from `/proc/driver/nvidia/params`) rather than blanket driver swaps.

agent/managed-skills/halogen-flash-server-ops/SKILL.md

@@ -1,67 +0,0 @@
----
-name: halogen-flash-server-ops
-description: "Operate the frank-large halogen-flash-server deployment on Dankclaw (model swaps, restarts, memory sizing, health and client verification, rollback) — use for anything touching frank-large after the 2026-09-11 runtime swap"
----
-
-# halogen-flash-server ops (frank-large, Dankclaw)
-
-frank-large is a **rootless Quadlet container** since 2026-09-11:
-`dankclaw/frank-large/quadlet/frank-large.container` in the serverAppConfigs repo,
-installed to `~/.config/containers/systemd/`, generated unit `frank-large.service`
-in the **user** manager. Full runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`.
-
-## Never use sudo systemctl on it
-
-It is a user unit. `sudo systemctl stop frank-large` does nothing useful.
-
-```bash
-systemctl --user status frank-large.service
-podman logs frank-large | head -40        # version banner + memory ledger
-curl -s http://127.0.0.1:7770/health      # 200 only after the engine PONGs
-curl -s http://127.0.0.1:7770/v1/models   # must list frank-large
-```
-
-## Deploy / change flags
-
-1. Edit the `.container` file in the repo (all tunables are inline `Environment=` lines).
-2. `./tests/run.sh` (the quadlet seam covers GPU devices, memlock, ipc/seccomp, the single published port, writable weights mount).
-3. Commit, push, then `./scripts/deploy.sh dankclaw --service frank-large`.
-
-The handler retires the old root unit (moving its symlink to the home trash), pulls and
-re-tags the image to the `localhost/` alias, installs the unit, restarts it, and gates on
-the download when the checkpoint is absent. Never `systemctl --user enable` a generated unit.
-
-## Sizing a swap
-
-Read the startup ledger before and after any change: it prints the locked weights, the KV
-pool it chose, the working set, the BIOS iGPU carve-out, and the corrected free-memory
-figure. `free`, MemAvailable and every monitor built on them overstate free memory by the
-size of the model (~68 GiB) because the kernel counts the locked weights as reclaimable
-page cache.
-
-- KV is ~7.2 GiB per 262,144 positions; the prefill arena is 16.7 GiB at `HALOGEN_MAX_TOK=32768`.
-- The device budget stops near 47 GiB *with the weights pinned*, and lower on a machine
-  carrying more besides this server. `HALOGEN_KV_POOL_FIT=1` lowers the pool to fit and
-  prints what it chose; a refused start needs a smaller `HALOGEN_KV_POOL_POSITIONS`
-  (`HALOGEN_KV_SLOTS` will not help — slots share one pool).
-- Long prompts crawling with the disk busy = file cache, not memory: lower the pool or
-  raise `HALOGEN_HOST_RESERVE_GIB`.
-
-## Client contract traps
-
-- No server-side sampler defaults: a request without `temperature` decodes GREEDY.
-- The per-request budget defaults to 8192 and **includes thinking**; a reasoning model
-  plus a small budget returns EMPTY content with `finish_reason: "length"`. Cap is 65536
-  (a 400 above it), and `max_tokens`/`max_completion_tokens`/`max_output_tokens` must agree.
-- `response_format` is a 400 ("no constrained decoding") — as it is on FreeToken/frank-small.
-  Any caller that needs structured output or a tiny budget needs a different backend.
-- A request reserves prompt + `max_tokens` positions against the pool; larger requests queue
-  and then fail with `503 engine_busy` after 3600 s.
-
-## Verification after any change
-
-Run a real completion (never just `/health`), a vision probe if the tower is enabled
-(image content part with a `data:` URL; `http(s)` URLs are refused), then one call from the
-client that consumes it. Rollback: the llama.cpp era is in `.AGENTTRASH/` (`.llama.cpp`),
-the STRIX payload and ROCmFPX build stay on disk — but remove the Quadlet from
-`~/.config/containers/systemd/` before restoring the root unit, or both claim port 7770.

agent/managed-skills/halogen-kv-pool-fit/SKILL.md

@@ -1,26 +0,0 @@
----
-name: halogen-kv-pool-fit
-description: "Resize halogen-flash-server's KV pool or prefill arena without getting silently clamped or OOMing the boot: pool, arena and vision share one device budget, HALOGEN_HOST_RESERVE_GIB only scales the fit-check ceiling and is NOT an allocation, HALOGEN_KV_POOL_FIT only ever lowers, and the sizing warning prints only when a clamp happened."
----
-
-# halogen-flash-server KV pool / arena resizing
-
-One device budget covers KV pool + prefill arena + vision tower:
-
-```
-budget = host RAM (121.2) - resident weights (67.7) - HALOGEN_HOST_RESERVE_GIB
-need   = KV (~28 KiB/position, ~11.0 GiB at 401408 as reserved) + arena + vision (1.97)
-arena  = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
-```
-
-Measured ledger (Dankclaw, engine's own startup print): 262144/32768/r24 → 96.3 held, 12.1 left, 2081 blocks; 524288/16384/r24 → 94.6/13.8/3742; 524288/32768/r15 → 103.5/7.1/310 (stall NOTE prints); 524288/32768/r16 → identical (reserve is NOT an allocation); **401408/32768/r15 (deployed 2026-09-13) → 100.0/9.1/1396, NOTE gone**.
-
-Procedure:
-1. Edit `HALOGEN_KV_POOL_POSITIONS` / `HALOGEN_MAX_TOK` in `dankclaw/frank-large/quadlet/frank-large.container` only. Never raise `HALOGEN_HOST_RESERVE_GIB` — it only moves the fit-check ceiling (at 524288 the ceiling was 16; engine needs budget ≥ 36.5 there; 18 clamps to 262144).
-2. Over budget → `HALOGEN_KV_POOL_FIT` silently LOWERS the pool and deploy still reports healthy. Way over → arena alone OOMs the boot while the unit still reads ACTIVE (max_tok 65536 did this). Always `podman logs frank-large | grep 'kv pool:'` and match the reserved count to the requested one; /health proves nothing about pool size.
-3. Under budget → nothing prints; absence of WARNING is not evidence of headroom.
-4. After restart, FIRST request can silence the engine ~45-60 s ("has not answered PING", /health engine_unresponsive) — one-time pool first-touch, self-recovers; HealthStartPeriod=300s covers it. Do not restart the unit. A completion retried after that window succeeds instantly.
-
-To free host memory: cut the pool (262144 hands back ~3.6 from 401408) or the arena (MAX_TOK 16384 = 8.4). Host-left gain comes in below the KV delta (engine keeps freed space as working slack): the 401408 trim modelled +3.4, measured +2.0 GiB but 4.5x contiguous blocks.
-
-Omp client config on blankpad: frank-large `maxTokens` must be ≤ 65536 (server 400s above the cap) and prompt+max_tokens must fit the pool or the request queues 3600 s then 503s.

agent/managed-skills/hermes-config-home-drift/SKILL.md

@@ -1,18 +0,0 @@
----
-name: hermes-config-home-drift
-description: "Diagnose Hermes webui/agent model-connection failures caused by HERMES_HOME config-home drift (image VOLUME anonymous volumes resetting on recreation) — verify which home each runtime reads, resolve provider→endpoint with the app's own resolver code, then probe the endpoint."
----
-
-# Hermes config-home drift diagnosis
-
-Symptom: hermes-webui/agent "cannot connect" to models even though the model endpoints answer curl.
-
-1. **Find the real config home per runtime.** `podman exec <container> printenv HERMES_HOME` for each hermes container. If webui reads `/opt/data` (image default) while the gateway reads `/data`, the webui runs the image-default template config (`provider: auto` → openrouter.ai with no key = connect failure).
-2. **Know the recreation trap.** Image `VOLUME` points get a fresh anonymous volume on every container recreation (`podman run --replace --rm`) — edits inside such a mount are lost on restart/redeploy. Compare volume ids before/after a restart to confirm.
-3. **Fix:** pin `Environment=HERMES_HOME=/data` in the quadlet unit (shared state root), deploy via service-scoped deployer, restart.
-4. **Verify with the app's own code**, not guesses:
-   - `from api import config as C; C.resolve_model_provider("<model-id>")` → returns `(model_id, provider, base_url)`; bare ids map to owning provider by scanning the `providers:` dict models allow-list; `@provider:model` syntax forces an explicit provider.
-   - `hermes_cli.providers.resolve_user_provider(name, cfg["providers"])` → ProviderDef with real `base_url`; accepts `api`/`url`/`base_url` keys on named provider entries.
-5. **Probe the endpoint with the exact served model id** (FLM/NPU endpoints route only by exact registry tag and answer HTTP 200 with `{"error":"Failed to load <tag> model!"}` for unknown tags — status-code checks miss this).
-
-Key gotcha: named providers (`providers:` dict) are a catalog + connection source; `custom:<slug>` connections resolve only via `custom_providers[]` entries. If neither holds an entry for the provider name, the request has no endpoint and fails to connect.

agent/managed-skills/hermes-live-config-edit/SKILL.md

@@ -1,21 +0,0 @@
----
-name: hermes-live-config-edit
-description: "Runbook for changing Hermes (Dankclaw) live config.yaml — model routing, providers, aux routes — including backup, apply, restart, and verification steps"
----
-
-# Editing live Hermes config on Dankclaw
-
-Hermes-webui/gateway read `/home/dank/Containers/hermes/data/config.yaml` (dir is `drwx------` uid 109999; all reads/edits need `sudo`). The repo file `dankclaw/hermes/config.yaml.example` is a sanitized backup and can lag live — always diff the live file first; never treat the example as truth.
-
-## Procedure
-1. **Backup first:** `sudo cp -a .../config.yaml .../config.yaml.bak-YYYYMMDD` (in-dir trash `.AGENTTRASH/` also exists for removals).
-2. **Apply change** with surgical `sudo sed -i` or full-file write; YAML comments in the live file are tool-generated — preserve them.
-3. **Sweep stale references** across the whole data dir, not just config.yaml: `sudo grep -rln "<old-id>" /home/dank/Containers/hermes/data --include="*.yaml" --include="*.json"`. Session files/MEMORY.md are historical — leave them.
-4. **Derived caches:** `webui/models_cache.json` is keyed by a config fingerprint (`_source_fingerprint.config_yaml` mtime/size); after edits it's stale. Either let it regenerate lazily or trash it (mv to data dir `.AGENTTRASH/`) — never hand-edit.
-5. **Restart the model-consuming units** (they parse config at startup): `systemctl --user restart hermes-webui.service hermes.service`. `hermes-dashboard.service` is unaffected by routing changes.
-6. **Verify behaviorally**, not by status codes: one tiny completion per endpoint with the exact served id — POST `/v1/chat/completions` on 7770 (Dankclaw, llama.cpp serves alias `frank-large`), on Dankserver 7770 (FreeToken REQUIRES an explicit `model` field), and on 7771/v1 (FLM routes by exact registry tag, e.g. `gemma4-it:e4b`; unknown tags answer HTTP 200 with `{"error":"Failed to load <tag> model!"}` — status-code checks silently pass).
-7. **Sync the repo backup** (`dankclaw/hermes/config.yaml.example`) to the live shape, redacting secrets (password_hash, phone account stay live-only), then run `./tests/run.sh` before commit.
-
-## Gotchas
-- `provider: custom` + `base_url` entries route by base_url and send the model id verbatim; named providers (`dankclaw-large`, `dankserver-small`, `dankclaw-npu`) resolve via their `api:` URL.
-- Renaming a served alias anywhere (serve scripts, env ALIAS) MUST propagate to this config in the same change or Hermes requests 404 on nonexistent models — this exact gap caused the 2026-09-05 outage.

agent/managed-skills/hermes-local-aux-routing/SKILL.md

@@ -1,87 +0,0 @@
----
-name: hermes-local-aux-routing
-description: "Fix a broken Hermes auxiliary route (title generation, compression, vision, etc.) when the target is a local OpenAI-compatible backend — diagnose by probing the real call path in-container, then repoint the transport"
----
-
-# Hermes auxiliary routing against local backends
-
-Use when a Hermes feature that calls an auxiliary task fails against a local
-endpoint on this fleet (Frank-small on Dankserver :7770, frank-large on Dankclaw
-:7770). The usual cause is a request shape the local runtime does not implement,
-not a routing mistake.
-
-## Diagnose with the real call path
-
-Do not guess from the config. Run the real function inside the container — it
-uses the caller's actual kwargs, the config-driven provider resolution and the
-real response parsing:
-
-```bash
-base64 -w0 /tmp/probe.py | ssh dank@192.168.0.143 \
-  'base64 -d > /tmp/probe.py && systemctl --user restart hermes.service && sleep 20 && \
-   podman cp /tmp/probe.py systemd-hermes:/tmp/probe.py && \
-   podman exec -i systemd-hermes /opt/hermes/.venv/bin/python /tmp/probe.py'
-```
-
-Copy the file AFTER any restart (`--replace --rm` discards the container layer),
-and import the app's own constants so the probe cannot drift from production:
-
-```python
-import sys; sys.path.insert(0, "/opt/hermes")
-from agent.title_generator import _TITLE_PROMPT_TEMPLATE, _TITLE_RESPONSE_FORMAT, _clean_title, _extract_title_text
-from agent.auxiliary_client import call_llm
-resp = call_llm(task="title_generation", messages=[...], max_tokens=64,
-                temperature=0.3, extra_body={"response_format": _TITLE_RESPONSE_FORMAT})
-```
-
-Pass `api_mode=` / extra `extra_body` keys to A/B a fix before touching config.
-(The CLI at `/opt/hermes/hermes` uses the system python and dies on
-`No module named 'yaml'`; use `/opt/hermes/.venv/bin/python`.)
-
-## What actually breaks, and the fix
-
-- **`response_format` (json_object / json_schema) is a hard 400** on every local
-  backend here — neither FreeToken nor halogen implements constrained decoding.
-  Config cannot override it: `call_llm` merges the task's `extra_body` first and
-  then `update()`s the caller's, so the hardcoded one wins. Only temperature and
-  max_tokens have reactive retry paths.
-- **Fix: move that one task to the Responses transport**, which never forwards
-  `response_format`:
-  ```yaml
-  auxiliary:
-    title_generation:
-      provider: dankserver-small
-      model: frank-small
-      api_mode: codex_responses        # real per-task key
-      extra_body:
-        reasoning:
-          effort: none                 # mandatory, see below
-  ```
-- **Disable thinking for any cap-limited task.** The adapter omits reasoning
-  only when the config says `reasoning.enabled: false`; note that the
-  `reasoning_effort: none` shorthand produces exactly that, which leaves the
-  server default (thinking ON) and the 64-token title budget is spent on the
-  think — empty content, `finish_reason: length`. The explicit
-  `extra_body.reasoning.effort: none` instead sends
-  `{"effort":"none","summary":"auto"}`, which the local /v1/responses honors.
-- Both local runtimes also answer `/v1/responses` and `/v1/messages`, so no
-  proxy is needed to use the alternate transport.
-
-## Verify and land it
-
-1. Re-run the probe with **no** `api_mode`/reasoning kwargs — it must pass from
-   config alone; that is the production path.
-2. Keep the live file (`/home/dank/Containers/hermes/data/config.yaml`, owned by
-   uid 109999, needs sudo) and the repo example
-   `dankclaw/hermes/config.yaml.example` in step; the example is a sanitized
-   backup and lags otherwise.
-3. `./tests/run.sh`, commit, push — a Hermes config change is docs+example only,
-   no service deploy needed.
-
-## Gotchas
-
-- `auxiliary.<task>.enabled: false` is checked BEFORE `apply_instant_title`, so
-  disabling a broken title task also removes the derived title — it does not
-  degrade gracefully the way a failure does.
-- A local call can also fail at the model level: a 200 with an error-text body,
-  or a chatty non-title answer. Always read the raw content, not the status.

agent/managed-skills/hermes-webui-secret-rotation/SKILL.md

@@ -1,20 +0,0 @@
----
-name: hermes-webui-secret-rotation
-description: Rotate the Hermex (hermes-webui) password on Dankclaw and verify auth behaviorally — covers the dual EnvironmentFile gotcha where only dankclaw/.env is effective at container create.
----
-
-# Hermex webui secret rotation (Dankclaw)
-
-Auth contract (verified live 2026-08-31): `POST /api/auth/login` with JSON `{"password":"..."}` → 200 + signed HttpOnly `hermes_session` cookie (24 h TTL) on match, 401 on miss. No username field. `/health` answers 200 unauthenticated and `/` answers 302 — never verify auth via those; always probe login with a deliberately wrong password (expect 401) and the real one (expect 200).
-
-## Rotation procedure
-1. `ssh -o BatchMode=yes dankclaw 'bash -s'` with a stdin script (zsh on the host mangles inline `${...}` even inside single-quoted ssh args — never interpolate curl `-w %{http_code}` inline; pipe the whole probe as a heredoc'd bash script).
-2. Update BOTH env files with `sed -i "s/^HERMES_WEBUI_PASSWORD=.*/HERMES_WEBUI_PASSWORD=<new>/"`:
-   - `/home/dank/serverAppConfigs/.env` ([Service] EnvironmentFile)
-   - `/home/dank/serverAppConfigs/dankclaw/.env` — **the effective one**: the quadlet unit's `[Container] EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env` is what the container reads at create; rotating only the root file leaves the running container serving the old secret.
-3. `systemctl --user restart hermes-webui.service` (unit recreation re-reads env; no deploy needed if the unit file itself didn't change).
-4. Verify: wrong-pw login → 401, right-pw → 200, from both host loopback and a LAN client at `http://192.168.0.143:8787` (webui publishes on the LAN interface since commit 4443897; off-network ingress is SSH tunnel only — user `tunnel`, port **22**, key-only auth, forward remote end localhost:8787).
-
-## Gotchas
-- `sudo podman exec` fails exit 125 here: root's namespace does not see rootless containers — run `podman exec hermes-webui printenv HERMES_WEBUI_PASSWORD` unprivileged (ssh user owns the env files, mode 600).
-- If rotating because a secret leaked into transcripts/logs, rotate in place via a script that fetches the live value from the running container (`printenv`) instead of transcribing it.

agent/managed-skills/heterogeneous-accelerator-split-triage/SKILL.md

@@ -1,94 +0,0 @@
----
-name: heterogeneous-accelerator-split-triage
-description: "Decide whether splitting one model across two unequal accelerators on a single host (fast small-VRAM GPU + big slow unified-memory device, or a GPU over a narrow PCIe link) is worth building — with the KV/prefill arithmetic, the probes that settle it, and the doors that are already closed."
----
-
-# Heterogeneous accelerator split — triage before building
-
-Use when someone asks "could we split model X across device A and device B" on one host, when the devices differ in memory size/bandwidth (e.g. 24 GB GDDR6X vs an APU sharing LPDDR), or when a link is narrow (PCIe x4). Produces a go/no-go with arithmetic instead of enthusiasm.
-
-## 1. Characterise the link (bytes vs latency)
-
-```bash
-lspci -vv -s <bdf> | grep -E "LnkCap:|LnkSta:"          # capability vs actual
-nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.gen.current,pcie.link.width.current --format=csv
-```
-
-Expect **idle-downshift**: a Gen4 x4 dock can idle at Gen1 x4 (~1 GB/s) and only upshift under load. Measure while the device is busy, or you will size the design against the wrong number.
-
-Compute the per-hop cost against the hidden state (`n_embd x 2 B` for f16): a 5120-dim model moves 10 KiB/hop = ~1.3 µs at Gen4 x4. Then:
-
-- bytes per token = hops/token x hidden state. Typically ~1 MiB/token for 2 hops/layer x 64 layers -> ~0.3 % of a 40 ms token. **Bytes are almost never the constraint.**
-- the real budget line is **hop COUNT**: every device boundary is a host-side sync. A runtime that splits per *op* can hit 100 splits/token and pin the GPU at ~5 % SM. A runtime that batches per-backend graphs is far cheaper — measure `graph splits` in the loader log, do not assume.
-
-**Design rule: partition by where the weights live, never by where the activations flow.** Resident weights, activations crossing.
-
-## 2. Get the workload's real bottleneck (KV is usually it)
-
-KV per position x context = bytes read per token, because attention reads the whole cached context every token:
-
-```
-KV/token = kv_bytes_per_position * context
-```
-
-Compare that against (a) expert/weight traffic per token (read once per token per layer) and (b) total token time. At long context or on a hybrid model with few attention layers but many positions, **KV dominates**. Then:
-
-- the highest-leverage placement is **KV cache on the fastest-memory device, together with its attention compute** — never KV alone (separating them puts `KV/token` over the link, which is fatal: GBs across a 7.88 GB/s link = hundreds of ms/token);
-- check the KV + attention weights actually fit: `kv_positions_per_slot * slots * bytes_per_position + attention_trunk <= VRAM`.
-
-## 3. What placement never buys
-
-Prefill is compute-bound and expert weights are read **once** per batch, so moving them to faster memory buys nothing while adding a serial stage. Expect a regression, and size it from calibration: on Strix Halo + RTX 3090 (Qwen3.8-Flash-Next, ik_llama.cpp) moving 12.5 % of the expert mass to the GPU gave **decode +12 %, prefill 192 -> 32 tok/s**.
-
-A `1/(f_big/BW_slow + f_fast/BW_fast)` harmonic calculation reproduces that decode gain to within ~2 points, so it is usable for decode projections — and it assumes the stages are sequential (true at batch 1: no pipelining).
-
-## 4. Is the second device even a separate pool?
-
-On a UMA APU, check before assuming additivity:
-
-```bash
-cat /sys/class/drm/card*/device/mem_info_vram_total   # BIOS carve-out (small)
-cat /sys/class/drm/card*/device/mem_info_gtt_total    # aperture onto SYSTEM RAM
-rocm-smi --showmeminfo all | grep -i gtt              # used == what the CPU also sees
-```
-
-A large GTT window means the "GPU memory" is the *same* LPDDR under one memory controller: adding a stage there adds a compute engine, not capacity or bandwidth. Only the discrete card's VRAM is additive. Fencing more (`amdgpu.gttsize`, BIOS UMA size) just removes bytes from CPU-visible RAM.
-
-## 5. Speculative decoding as the "cheap" split
-
-A drafter on the *other* device is bandwidth-free (token ids per round, tens of bytes), so it is tempting. Before proposing it:
-
-- the drafter needs the target's **exact tokenizer** for a plain draft model (verify by comparing GGUF `tokenizer.ggml.tokens`, not by trusting the family name), or it must be a head **trained for that target** (EAGLE/DFlash/MTP are conditioned on the target's hidden states and are not portable between models);
-- a plain drafter costs **K sequential forwards per round**, so it must be 10-50x faster than the target. Compute the break-even: with `r = t_draft/t_target`, `rate_ratio = A/(K*r + 1)`, `A = (1-p^(K+1))/(1-p)`. At `r ≈ 0.4-0.5` (a drafter only ~2x faster) break-even needs `p ≈ 0.7+` at K=3;
-- always compare against the **incumbent**, not the no-draft baseline: a target already running its own MTP head (e.g. ~1.7 tokens committed/round) is a high bar;
-- check the runtime can even accept an external drafter. Closed engines may enumerate a fixed drafter set and 400 anything else (`DRAFTER_CAP`-style capability gates keyed on the loaded checkpoint) — that is a design boundary, not a config gap. In llama.cpp the portable mechanism is `--spec-draft-device/-devd` (draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd-ctvd`.
-
-## 6. Expressing a split (llama.cpp, the usual vehicle)
-
-```bash
--dev CUDA0,Vulkan0 -sm layer -ts F0,F1        # contiguous layer ranges (simplest)
--ot "\.attn_.*=CUDA0" -ot "\.exps\.=Vulkan0"   # per-tensor, by ROLE (what KV co-location needs)
-```
-
-- buffer types are matched by name at load (`ggml_backend_buft_name`), so device names work as `-ot` values;
-- `-cmoe` / `-ncmoe` are **CPU-only**; device-side MoE placement is `-ot`;
-- several backends in one binary is fine (CUDA + Vulkan is the practical pair; HIP + CUDA do not coexist), and `GPU_BACKEND_A=OFF` in the CMakeCache of the existing build tells you what needs adding;
-- contiguous ranges are the *wrong* split when KV bandwidth dominates, because they drag half the attention layers (and their KV) back to the slow pool.
-
-## 7. Hazards checklist
-
-- paged/sidecar tables (huge n-gram/embedding tables) must stay where they are read from; moving them is pure loss;
-- keep `mmap` on for models with huge embedding tables (anonymous memory gets the process cgroup-killed);
-- CUDA graph capture can abort on unusual graph shapes (hyper-connections etc.) — check before assuming optimisations apply;
-- the second device's brand-new backend may not implement the arch's ops at all: make the first measurement a **baseline of the runtime on that device alone**, because a split's arithmetic is worthless if the baseline is worse than the incumbent engine;
-- a split consumes the fast device entirely — check what else it was serving before trading it away;
-- benchmark/experiment discipline on production hosts: stop and mask first.
-
-## 8. Verdict shape
-
-Rank and report, and prefer the boring answer when it wins:
-
-1. **Two independent endpoints** (zero shared traffic) — often the real answer to "what's a useful combination", especially over a narrow link;
-2. **One process, weights-resident, KV co-located with attention on the fast pool** — promising, quantifiable (+% decode at long context), needs a dual-backend build and a baseline measurement;
-3. **Hot-expert cache / per-token weight streaming** — affordable only if the miss set per token is small (a few experts; check `misses x bytes x tok/s` against link bandwidth) and overlapped, and it injects PCIe latency into the critical path; no mainstream runtime implements a dynamic expert cache;
-4. list explicitly what does *not* help: TP/collectives across unequal devices, KV on the far device, model loading over the link, prefill-motivated placements.

agent/managed-skills/hf-checkpoint-profile/SKILL.md

@@ -0,0 +1,26 @@
+---
+name: hf-checkpoint-profile
+description: "Profile a Hugging Face safetensors checkpoint (sizes, dtypes, tensor-name space, gated/license) with compact HTTP-range reads — safe for subagents on small context slots."
+---
+
+# Profile an HF safetensors checkpoint cheaply
+
+Goal: inventory a repo (files/bytes, dtype histogram, tensor-name patterns, per-group GiB, config keys, license/access) without pulling gigabytes and without overflowing a subagent's context slot.
+
+## Endpoints (all public, no token needed even for gated models)
+- `https://huggingface.co/api/models/<id>?blobs=true` → `siblings` (name+size), `gated`, `private`, `downloads`, `safetensors` metadata, `tags`.
+- `https://huggingface.co/api/models/<id>` for tags/base model; `https://huggingface.co/api/models?author=<x>&limit=100` to find sibling repos (canonical id check: cards often link both `X-1.5-Y` and `X-Y`).
+- `https://huggingface.co/api/models/<id>/resolve/main/<file>` for `config.json`, `model.safetensors.index.json`, `chat_template.jinja`, `LICENSE`, `generation_config.json`; `/raw/main/<file>` for text.
+
+## dtype + per-group bytes without downloading shards
+Parse the safetensors header over HTTP Range: first 8 bytes = little-endian u64 header length, then that many bytes of JSON (`{name: {dtype, shape, data_offsets}}`). Fetch every shard header with a thread pool (131 shards ≈ seconds), then bytes = prod(shape) × dtype size. Collapse names with `re.sub(r"\.\d+\.", ".N.", k)` (and `\.shard_\d+` → `.shard_N`) to get the pattern space; group-sum by prefix to see where the mass sits (experts / ngram-PLE / MTP / vision / trunk).
+
+Compare the key set against the base model's index to prove "full-weight finetune, identical names" (a zero-diff key set means per-tensor recipes written for the base apply verbatim, and it reveals fused vs per-expert layout, PLE/ngram shards, MTP head, vision tower, absence of `quantization_config`).
+
+## Guardrails
+- Run python scripts from a clean directory: a stray `/tmp/inspect.py` shadows stdlib `inspect` and kills `concurrent.futures` imports.
+- Never put an HF token on a command line (`/proc/<pid>/cmdline` is world-readable); use `curl -K <mode-600 file>`.
+- `curl -f` / `--fail-with-body` and verify byte counts — a 401/403 writes the error body as the "artifact" and exits 0.
+
+## Subagent context budget
+Subagents on a 131k-token slot die from `exceed_context_size_error` when told to read whole doc pages. Give grep-only instructions (`| head -40`, `-o`, `-c`), demand a line cap ("report in ≤250 lines, no verbatim dumps"), and prefer `git clone --depth 5` + local grep to web-page reads. A peer scout that overflows returns exit 1 with the 400 in the harness log and NO structured output — assume nothing was learned and redo the fact inline.

agent/managed-skills/hf-gguf-payload-fetch-verify/SKILL.md

@@ -1,98 +0,0 @@
----
-name: hf-gguf-payload-fetch-verify
-description: "Inventory, fetch, hash-verify and assemble a Hugging Face GGUF payload (including mradermacher's .partNofM fragments and gated mirrors) before pointing an LLM service at it — on the Dank fleet or anywhere else."
----
-
-# Fetch and verify an HF GGUF payload
-
-Use before any model swap where the weights come from Hugging Face, especially
-large (>32 GiB) quantised models. The failure modes below all look like success.
-
-## 1. Inventory — query by quant name, never by extension
-
-```bash
-curl -s "https://huggingface.co/api/models/<org>/<repo>/tree/main?recursive=true&blobs=true" \
- | python3 -c "
-import json,sys
-for e in json.load(sys.stdin):
-    p=e.get('path','')
-    if 'Q4_K_S' in p:                       # the QUANT, not '.gguf'
-        print(p, e.get('size',0), (e.get('lfs') or {}).get('oid','-'))
-"
-```
-
-**Large quants ship as `.partNofM` fragments.** `mradermacher/*-i1-GGUF` repos
-hold e.g. Q4_K_M as `part1of3/2of3/3of3` (37.00+37.00+36.97 = 110.97 GiB) and
-Q4_K_S the same way, while the smaller rungs (Q2_K, Q3_K_*, IQ3_*, IQ4_XS) are
-single `.gguf` files. Filtering on `endswith('.gguf')` therefore makes Q4_K_S
-look like it does not exist — do not conclude a quant is missing from an
-extension-filtered listing.
-
-Check both mradermacher repos: `…-i1-GGUF` (imatrix) and `…-GGUF` (plain), and
-the model's own repo. Gated mirrors exist (`orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF`
-carries the same ladder) and return **403 even with a valid HF token** until
-that account accepts the gate — prefer the ungated copy rather than chasing access.
-
-## 2. Fetch — `-f` is mandatory, secrets stay out of argv
-
-```bash
-# token out of band: mode-600 config, never -H on the command line
-printf 'header = "Authorization: Bearer %s"\n' "$(cat ~/.cache/huggingface/token)" > /tmp/hf.curlrc
-chmod 600 /tmp/hf.curlrc
-curl -fL -C - --retry 10 --retry-delay 5 --retry-max-time 21600 -K /tmp/hf.curlrc \
-     -o "$DEST/$name" "$BASE/$name"
-```
-
-- `curl -o FILE URL` **exits 0 on HTTP 401/403** and writes the error body as the
-  artifact. A gated repo yields identical small files (153 bytes) that hash the
-  same and a `DOWNLOAD_DONE` in seconds for a 104 GiB payload.
-- A token in `-H "Authorization: Bearer …"` is visible in `/proc/<pid>/cmdline`
-  to any `ps`/`pgrep -a` on the host. Never `pgrep -a` a running fetch.
-
-## 3. Verify, then assemble
-
-```bash
-# per part: byte count must equal the Hub's declared size, sha256 must equal lfs.oid
-sha256sum "$PARTS"/*.part* | sort
-
-# concatenate in NUMERIC order, then verify the total and the container magic
-cat "$PARTS"/X.gguf.part1of3 "$PARTS"/X.gguf.part2of3 "$PARTS"/X.gguf.part3of3 > "$OUT"
-stat -c%s "$OUT"          # must equal the sum of the part sizes
-head -c 4 "$OUT"          # must print GGUF
-```
-
-The assembled file has no published hash (only the parts do), so the per-part
-LFS oids plus an exact total are the verification chain. If a size check fails,
-suspect your own arithmetic before the payload: the sum of the parts is the
-expected value, and it is easy to mis-add by hand (`python3 -c "print(a+b+c)"`).
-
-## 4. Housekeeping
-
-- Failure artifacts (error pages) and stray directories: move to
-  `~/.AGENTTRASH/YYYYMMDD-topic`, never `rm`; list them and wait for approval to
-  purge.
-- Keep the `.partNofM` fragments after assembly when disk allows: they are the
-  only copies that can be re-verified against the Hub. Note the total they cost
-  and ask before purging.
-- Never delete the payload being replaced as a side effect of a swap: upstreams
-  delist quants (both mradermacher and orcarouter have delisted Q4_K_M for
-  Qwen3.8-Flash-Next-Uncensored), and the local file may be the last copy.
-
-## 5. Waiting for a long fetch
-
-Do not `pgrep -f <script>` where the pattern appears in your own command line —
-the loop matches itself and spins its full iteration count while the download is
-fine. Watch the artifact instead:
-
-```bash
-while [ "$(stat -c%s "$DEST/$name" 2>/dev/null || echo 0)" -lt "$EXPECTED" ]; do sleep 30; done
-```
-
-## 6. Before choosing a smaller quant
-
-A smaller quant buys memory and ~3–5 % decode, and costs quality. It does **not**
-change what the runtime can express: on llama.cpp, per-tensor cross-device
-placement is refused whenever `--override-tensor` is set
-(`has_tensor_overrides()` gates `pipeline_parallel`, which sets `n_copies`), and a
-layer split can only move contiguous layer ranges. State which of the two you are
-buying — headroom or a design change — before spending 100 GiB of bandwidth.

agent/managed-skills/hf-large-artifact-publication/SKILL.md

@@ -0,0 +1,40 @@
+---
+name: hf-large-artifact-publication
+description: "Publish a large quantized artifact + model card (and an optional anonymized corpus dataset) to HF Hub from dankserver: staging set, HF-style card spine, upload gates, transient-unit launcher, sentinel watching, card-metadata and scrub laws."
+---
+
+# HF large-artifact publication (dankserver)
+
+Publish a big model artifact (100+ GiB) + card, and optionally an anonymized corpus dataset, from dankserver under `frank8465`.
+
+## Staging
+- Dev box `~/Projects/hf-swift15-upload/` (author card + scripts here), `scp -r` synced to `/home/dank/hf-swift15-upload/` on dankserver. Keep staging + logs + `.rc` sentinels after publication as receipts.
+- Payload set: artifact, `README.md` card, `scripts/` (recipe/drivers/probes), `artifacts/` (gate JSONs, per-position `.rows`, manifests, run logs).
+- Secret-scan the whole payload before arming: credential regexes (hf_/ghp_/pat, private keys, Bearer values, AKIA/sk-), LAN IPs, hostnames, domains, emails. Verdict each hit — variable *references* and upstream test fixtures are not leaks; check real values in context.
+
+## Card law (HF validator)
+- Custom licenses MUST be `license: other` + `license_name:` + `license_link:` (mirror the base model's card trio — fetch `https://huggingface.co/api/models/<base>` and read `cardData`). A raw non-SPDX `license:` value raises `ValueError: Invalid metadata in README.md` inside `upload_file` — AFTER all local gates have run.
+- Card spine: what-the-format-is + engine pin digest, recipe table, quality gates (KL/top-1/PPL + per-position data shipped), proof hardware, exact `podman run` line, license chain, manifest table with the artifact's sha256 receipt, "In plain English:" block per section.
+- Quote the SHIPPED filename (build name), not a prettified one — a hub-side rename of a 100 GiB LFS file costs a full re-upload.
+
+## Upload mechanics
+- venv `/home/dank/venvs/radcorpus` (huggingface_hub 1.28 + hf_xet); `HF_XET_HIGH_PERFORMANCE=1`; token from `~/.cache/huggingface/token` (see defuse-shell-rc-secrets pattern: `~/.config/secrets/<name>` 600 + rc loader, mirrored to the cache file). NEVER token in argv/chat.
+- Script gates, in order: sha256 of live artifact vs recorded digest (`gate 1 : MATCH`), `create_repo(..., private=False, exist_ok=True)`, small files first, artifact last, post-upload census — **compare remote sizes to local and FAIL the RC on any mismatch** (this caught a stale card mid-campaign).
+- hub 1.28 API traps: `upload_file`/`create_commit` kwargs are keyword-only; no `rename_file`, `hf_hub_url`, `HfApi.commit_api` (use `CommitOperationAdd`/`CommitOperationDelete` from `huggingface_hub` top-level if needed); `whoami` hides `canWrite` for legacy tokens — probe write scope by `create_repo(exist_ok=True)` on the real target.
+- zsh-over-ssh: never `echo ===` (triggers `=cmd` expansion); quote it.
+
+## Launcher + watchers
+`setsid nohup` over ssh dies at session teardown. Use a transient user unit:
+```
+systemd-run --user --collect --unit=hf-upload -p SendSIGKILL=no \
+  -p StandardOutput=append:/home/dank/logs/hf-upload.log \
+  -p StandardError=append:/home/dank/logs/hf-upload.log \
+  sh -c "cd <staging> && <venv>/bin/python -u <uploader>.py --go; echo \$? > /home/dank/logs/hf-upload.rc"
+```
+Sentinel: background ssh loop polling the `.rc` file (60 s period, bounded), tails the log on appearance. Progress bars are `\r`-fed — `tr '\r' '\n' | tail` to read them.
+
+## Corpus dataset variant
+- License audit first: read `urls.txt`/manifest, fetch each source card (`license` in cardData). Share-alike sources (CC BY-SA) → publish derived corpus CC BY-SA with attribution; model weights unaffected.
+- Fleet-harvested slices = agent-session transcripts: scrub with deterministic substring-class rules (NOT word-boundary: catches `dankclaw_default`, `FRANKBOOK_SVCS`, `franksdank/alloy` image namespace): LAN IPs→10.x, hostnames→server-a/b/client-c, domain→example.com (catch `\.`-escaped forms + bare brand), `/home/<user>`, emails→user@example.com. Residual probes run on OUTPUT; whitelist self-replacements + audited fixture noise.
+- Publish original hashes, disclose non-byte-identity, cross-link model card ↔ dataset card, verify via `raw/main/README.md` fetches.
+- Dataset repo needs dataset-write scope (fine-grained token; access tokens 403). `repo_type="dataset"` everywhere; 45 MB rides in ~60 s.

agent/managed-skills/homepage-smoke-verify-loop/SKILL.md

@@ -0,0 +1,21 @@
+---
+name: homepage-smoke-verify-loop
+description: "Smoke-verify Dankclaw homepage changes locally before commit/deploy: isolated static tree + fake status + browser census, then deploy gate."
+---
+
+# Homepage smoke-verify loop (serverAppConfig)
+
+Verify `shared/homepage/` changes against a real browser BEFORE commit/deploy.
+
+## Procedure
+1. `node --check shared/homepage/app.js` and `python3 -c "import json; json.load(open('shared/homepage/services.json'))"`.
+2. Isolated tree: `rm -rf /tmp/hp-smokeN && cp -r shared/homepage /tmp/hp-smokeN`.
+3. Fabricate runtime files in the tree:
+   - `status.json`: `{"checked":"x","services":[{"name":<every services.json name>,"state":"up"}]}` (set one model `down` to exercise the red-pulse path).
+   - `deployed.js`: `window.DEPLOYED_AT="smoke"`.
+4. Serve: `(cd /tmp/hp-smokeN && nohup python3 -m http.server 8099 >/dev/null 2>&1 & echo $! > /tmp/.hp-pid)`.
+   - If reusing port 8099: `pkill -f "http.server 8099"` FIRST — a second server on a busy port fails silently and the OLD tree keeps serving.
+5. Browser census via eval tab on `http://127.0.0.1:8099/?v=<epoch>` — ALWAYS cache-bust with `?v=`; python http.server gets heuristically disk-cached by the browser. Assert: broken images (`.app-art,.mi-art` naturalWidth==0), computed styles (`getComputedStyle` for palette/fonts/clipPath), presence of new classes, `document.querySelectorAll('.docs section').map(s=>s.id)` for doc order, regex text census for banned words.
+6. Screenshot full page + `selector` shots; inspect visually.
+7. `./tests/run.sh` (expect only the known frank-primary MAX_SEQS failure), commit, `./scripts/deploy.sh dankclaw --service homepage` (runs Playwright e2e gate), then repeat the census against `https://www.franksdank.cloud/?v=<epoch>`.
+8. Cleanup: kill server, move `/tmp/hp-smokeN` to `~/.AGENTTRASH/YYYYMMDD-homepage-smoke/` (never `rm`).

agent/managed-skills/homepage-status-probe-namespace-triage/SKILL.md

@@ -1,32 +0,0 @@
----
-name: homepage-status-probe-namespace-triage
-description: "Triage a homepage status card (or any host-side TCP probe) reporting \"down\" for a service that is actually up: decide by network namespace whether the probe target is reachable at all before calling it an outage."
----
-
-# Goal
-
-Decide, in one pass, whether a `down` card on the homepage status page (`scripts/homepage-status.py`, run by `homepage-status.timer` on Dankclaw, probes TCP/HTTP from the **host** netns) is a real outage or a structurally unreachable probe target. The checker only has three states — `wip` | `up` | `down`; there is no "unknown", so an unprobeable service can never render honestly and must be dropped or re-pointed, not "fixed".
-
-# Why it goes wrong
-
-A host-vantage TCP probe can only reach a listener that is **published to the host** (rootless podman publishes via rootlessport/pasta; unpublished ports are invisible from the host netns). Three recurring false-down shapes on this fleet:
-
-1. **TLS-only app probed with `method:"http"`.** Plex Media Server 1.43 serves TLS only on `32400`; a plain-HTTP GET gets a TLS alert → curl exit 52 "empty reply from server", code `000` → card says down while PMS answers `200` over HTTPS. Fix = `"method":"tcp"` (connect is the honest liveness signal; the public URL stays HTTP because Caddy re-keys).
-2. **Retired port.** The Hermes card probed `127.0.0.1:8090` after the compose→quadlet move moved the dashboard to `9119` and the webui to `8787`; nothing listened on 8090, so a fully healthy stack read `down`.
-3. **Unpublished port / wrong netns.** `gitea-postgres` (`db:5432`, net `systemd-gitea`), `searxng-valkey` (`valkey:6379`, net `systemd-searxng`), and `alloy` (OTLP 12345) publish **no** ports. Container DNS aliases are netns-scoped, so from the host these are unreachable *by construction* — no probe value makes them honest.
-
-# Procedure
-
-1. Read the card's probe target in `shared/homepage/services.json`; note method (`tcp` vs `http`).
-2. Ask: does that port have a **host-visible listener**? `podman ps -a --format '{{.Names}}|{{.Status}}'`, then per container `podman inspect <c> --format '{{json .NetworkSettings.Networks}} {{.HostConfig.NetworkMode}}'` and `podman port <c>` (empty output = published nothing → structurally unprobeable from the host).
-3. Probe from the **correct vantage**. From a bridge-net container, the host is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on `dankclaw_default`), *not* `host.containers.internal` (`169.254.1.2`) — that pasta alias belongs to the default `podman` net (`10.88.0.0/16`, gw `10.88.0.1`) and is not routed from other nets. Probe with bash `/dev/tcp` (no `python3`/`ip` in these images), and escape the loop var so it expands **inside** `podman exec bash -c`.
-4. Classify: real outage (container exited — read `State.ExitCode`: 137 = SIGKILL, 143 = SIGTERM/manual stop, `OOMKilled=false` rules out the OOM killer) vs false down (running + unreachable target).
-5. Fix accordingly: re-point or re-method the probe for a real HTTP/TLS mismatch; **drop** cards whose only exposure is an internal netns (or accept permanent false-downs and say so); never invent a probe target for them.
-6. Verify by re-running the checker path, not by eyeballing: after deploy, re-read `/srv/homepage/status.json` and diff the `down` list before/after.
-
-# Watch out
-
-- Deploying this file is service-scoped: `./scripts/run.sh` (233 tests) → commit → `./scripts/deploy.sh dankclaw --service homepage`; the deployer *pushes* `main` first, so any already-committed local work rides along — say so before pushing.
-- The checker writes `/srv/homepage/status.json` world-readable (0644) so Caddy can serve it; keep that mode or Caddy 403s the page.
-- Removing a card's `probe` key without code change crashes the checker (`svc["probe"]` KeyError) — dropping the whole entry is the only zero-code fix.
-- Bare compose services with `restart=unless-stopped` (node-exporter on both hosts) do **not** survive reboot: rootless podman does not autostart bare containers, and `deploy-remote.sh` refuses to `compose up` Dankserver's compose services by design → they need a quadlet unit, not a restart.

agent/managed-skills/homepage-theme-smoke/SKILL.md

@@ -0,0 +1,15 @@
+---
+name: homepage-theme-smoke
+description: "Locally smoke-test the Dankclaw homepage theme (style.css/app.js/docs.html) in the managed headless browser, with the stale-server traps that fake the result."
+---
+
+Goal: prove a homepage theme change on a throwaway tree before commit/deploy.
+
+Protocol:
+1. Stage a fresh copy: `cp -r shared/homepage/. /tmp/hp-smoke-<rand>/`, drop `.AGENTTRASH` dirs, and write a fake `status.json` (`{"generated_at":"x","services":[{"name":...,"state":"up"}]}`) — app.js fetches services.json + status.json and renders NOTHING without both.
+2. Serve: `cd <tree> && nohup python3 -m http.server 8099 --bind 127.0.0.1 &`.
+3. STALE-LISTENER TRAP (fakes every census): a survivor server from an older wave keeps answering on 8099 — `ss -ltn | grep 8099` proves a listener exists, not WHICH tree it serves. After every (re)start verify identity: `for p in $(pgrep -f "http.server 8099"); do readlink /proc/$p/cwd; done` must print the NEW tree; then `curl -s localhost:8099/<file> | grep -c <new-marker>`. `kill` the stale pid — never trust a `||`-guarded restart. Also: `pkill -f "http.server 8099"` from a shell whose own command line contains that string kills the shell; kill by pid.
+4. Browser census: navigate with `?v=<epoch>` cache-bust (browser disk-caches http.server responses); `waitForSelector('.mi-card', ...)` before evaluating — rendering is async (fetch). Geometry checks: rect-gap to 1px for bottom-pinned chips; screenshot for visual proof. `screenshot({path})` may not write — use the returned omp-sshots artifact.
+5. In headless pages `new Audio()` is detached: prove voice playback via `tab.requests()` (206 on the /sfx file) + the `mi-speaking` class, not a DOM `<audio>` query.
+6. Live-domain check after deploy: the homepage is at `/`, the docs page ONLY at `/docs/` — `/docs.html` falls through the SPA fallback and serves the homepage shell. Anubis adds a challenge: `waitForSelector` with 30-45 s timeout.
+7. Deploy loop: `./tests/run.sh` → commit/push → `./scripts/deploy.sh dankclaw --service homepage` (runs the Playwright e2e gate) → live census with fresh `?v=`.

agent/managed-skills/html-change-report/SKILL.md

@@ -1,34 +0,0 @@
----
-name: html-change-report
-description: Generate and open a self-contained dark-theme HTML change report (Keep a Changelog categories + code samples) for a git repo time window; use when Frank asks for a rendered report of recent work.
----
-
-# HTML change report (Keep a Changelog + review aids)
-
-Produce a self-contained HTML report of repo work in a time window and open it in Firefox.
-
-## Data gathering
-- Window commits: `git log --since="24 hours ago" --pretty='%h|%ad|%s' --date=format:%H:%M` (split on `|`).
-- Metrics: `git diff --shortstat <first>^..HEAD`, per-area file counts via `git diff --name-only … | awk -F/ '{print $1"/"$2}' | sort | uniq -c | sort -rn`.
-- Bucket commits into phase/theme rows by time-of-day for the timeline table.
-
-## Report structure (standard: Keep a Changelog 1.1.0)
-1. Executive summary (metrics + 3-5 bullets).
-2. Full commit timeline table, phase-color-tagged (border-left per phase).
-3. `git show <hash>` command list per theme — Frank uses this for his review pass.
-4. Changes by category: Added/Changed/Fixed/Removed/Security; omit empty categories; cite commit hashes per item. Fixed = old behavior was a bug; Changed = intentional.
-5. 4-6 notable code excerpts: read real files, pick load-bearing/instructive code, note file:lines + commits + why it matters. HTML-escape everything (`html.escape`).
-6. Verification evidence table (what / how / result).
-7. Incidents & open risks.
-8. Actions required from Frank (numbered, effort hints).
-
-## HTML requirements
-- Self-contained: inline CSS only, NO CDNs (must work from file:// offline). Dark theme CSS vars (#1a1b26 bg family works).
-- Sticky top nav, section anchors, `<pre class=code><code>` blocks, commit hashes as `.tag` chips.
-- Build via python (eval kernel): subprocess for git data, string concat for CSS (avoid f-string brace pain), f-strings for rows.
-- Literal `{}` in HTML text (e.g. `fallbackChains: {}`) inside an f-string is a SyntaxError — build that fragment by concatenation or double the braces.
-
-## Delivery + verification
-- Write to `$HOME/Reports/<repo>-<window>-<date>.html` (outside repo, tree stays clean; create the dir — on a workstation account that is /home/blank/Reports, on dank hosts /home/dank/Reports).
-- Validate: python HTMLParser feed (parse errors) + open in headless browser (`write` JSON to xd://browser: open → run with `document.querySelectorAll` counts) → close tab. Element counts are sufficient proof; a fullPage screenshot of a long report returns a multi-MB raw pixel buffer dump — only screenshot short pages.
-- Open for Frank: `nohup firefox --new-tab file:///<path> >/dev/null 2>&1 & disown`.

agent/managed-skills/ik-llama-hybrid-moe-performance/SKILL.md

@@ -1,29 +0,0 @@
----
-name: ik-llama-hybrid-moe-performance
-description: Diagnose and tune ik_llama.cpp hybrid CPU/GPU MoE decode/prefill on the dankclaw 3090+Strix Halo frank-large deployment; includes the barrier-split root-cause method and the systemd-unit test-interference trap
----
-
-# ik_llama.cpp hybrid MoE performance campaign (dankclaw frank-large)
-
-Use when tuning or debugging decode/prefill speed of the ik_llama.cpp hybrid deployment (RTX 3090 + Strix Halo LPDDR, CPU-resident MoE experts) on dankclaw, or when a config experiment gives surprising numbers.
-
-## Measured envelope (2026-09-16)
-
-- Decode 3.8-4.3 tok/s, flat across thread counts (T1 ≈ T16). Prefill ~192 tok/s at 1.8k-32k prompts = the CPU AVX512-VNNI compute ceiling for CPU experts (batch amortizes expert reads; compute-bound, NOT memory-bound).
-- Decode root cause: ~100 CPU↔GPU backend splits/token; GPU at 5% SM during decode (launch/sync-bound over the PCIe x4 dock); T16 profile shows ~90% of cycles in `pause` (ggml_barrier spin) because threads wait on the serial critical path. Memory placement (mmap/anon/THP/deferred) does NOT matter — all variants land 3.8-4.6.
-- CUDA-graph capture aborts: qwen4exp's hc broadcasts use per-layer `ggml_repeat_4d` whose CUDA path does a capture-illegal op. Until ik gains capture-safe repeat/hc ops (mainline PR #28901 direction), graphs stay dead. Mainline llama.cpp itself runs 1.0 tok/s on this model (CPU MoE kernels 4x slower than ik iqk) — do not switch for speed.
-- MTP/speculative impossible for mradermacher i1 GGUFs: the NextN head is stripped (no blk.48 tensors).
-
-## Tuning levers, ranked
-
-1. `-ncmoe 42` puts 6 layers' experts on the 3090: decode 4.54, prefill (small prompt) 32 — but VRAM 22/24.6 GiB at 262k ctx; needs an OOM-margin decision from the operator.
-2. `--prefetch-experts` is a no-op when the page cache is resident.
-3. `-thp` huge-mmap usually fails (`Cannot allocate memory`) and falls back silently.
-
-## Traps
-
-- The frank-large root systemd unit has Restart=always: `pkill`ing llama-server respawns it in ~4 s. For manual experiments: `sudo systemctl stop frank-large.service` first (root unit — NOT `systemctl --user`, that is stale halogen-era docs); verify `pgrep -xc llama-server` == 1 and 'HTTP server listening' in the current log before trusting any measurement.
-- Any `-ot`/`-ncmoe` override disables mmap for the whole model load in ik (`create_tensors` returns false).
-- `-ncmoe N` without `GGML_CUDA_NO_PINNED_WEIGHTS=1` page-locks ~108 GiB (co-tenant killer — the 2026-09-15 incident).
-- `--defer-experts` defers only PLE; under `-ngl 999` the experts still target CUDA0 → cudaMalloc OOM (~79.7 GiB on a 24 GiB card).
-- Never run `-j16` builds concurrently with decode measurements — both results are garbage.

agent/managed-skills/imagegen-openwebui/SKILL.md

@@ -1,43 +0,0 @@
----
-name: imagegen-openwebui
-description: Runbook for running high-quality local image generation via Open WebUI imagegen API endpoint.
----
-
-# Open WebUI Image Generation Runbook
-
-## Overview
-Use this procedure to generate images locally using the Open WebUI image generation endpoint.
-This approach leverages local ComfyUI integration configured in Open WebUI.
-
-## Endpoint
-POST to the imagegen endpoint of your Open WebUI instance.
-
-### Example Request
-```python
-import requests
-
-url = "http://<host>:<port>/api/images/generations"
-headers = {"Authorization": "Bearer <your_token>"}
-data = {
-    "model": "imagegen",
-    "prompt": "A desired image description.",
-    "n": 1,
-    "size": "1024x1024"
-}
-
-response = requests.post(url, headers=headers, json=data)
-```
-
-### Example Response
-Returns an object with base64-encoded image data or a URL.
-
-## Best Practices
-- Keep prompts descriptive for best results.
-- Use the default model unless configured otherwise.
-- Verify that ComfyUI is running and connected before invoking.
-- Check Open WebUI logs for troubleshooting if generation fails.
-
-## Common Issues
-- **Connection refused**: Ensure Open WebUI and ComfyUI are running.
-- **Timeout errors**: Large image sizes may take longer; check ComfyUI resources.
-- **Authentication errors**: Verify API token validity in the request headers.

agent/managed-skills/incident-window-reconstruction/SKILL.md

@@ -1,92 +0,0 @@
----
-name: incident-window-reconstruction
-description: "Reconstruct which command/config was running at the instant an incident symptom first appeared, from on-disk artifacts (log mtimes + per-run durations + journal first-failure time) when memory and /tmp are gone."
----
-
-# Incident window reconstruction
-
-Use when the question is **"what was running when it broke?"** — an incident,
-outage, resource blow-up, or unexplained service death — and the answer must
-come from artifacts, not recollection. Typical triggers: a host went down during
-an experiment, a service died mid-run, someone asks for the parameters in effect
-at the moment of failure.
-
-## Principle
-
-Three independent clocks, cross-checked — never recollection:
-
-1. **Per-run duration** — every script wraps commands in `time`, so the log
-   holds `real 4m19.129s`.
-2. **Log mtime** — the last write of that log.
-3. **Journal first-failure instant** — the first error line from the affected
-   service.
-
-`start ≈ mtime − real` for the *last* run in a log, and each run nests inside
-the previous one's window. The run whose `[start, end]` contains the journal's
-first-failure instant is the answer.
-
-## Collect
-
-```bash
-# 1. ordering and clocks of every artifact (log mtimes)
-ssh <host> 'ls -l --time-style=+%m-%d_%H:%M:%S <logdir>/*.log | awk "{print \$6, \$7}"'
-
-# 2. durations + per-run section markers + the exact command lines
-ssh <host> 'grep -ah "^real" <logdir>/*.log'
-ssh <host> 'for f in <names>; do echo "## $f"; \
-  grep -aE "^####|^\+ /path/to/binary" <logdir>/$f.log \
-  | sed "s/\(.\{130\}\)/\1\n        /g"; \
-  grep -aE "^real|_EXIT=" <logdir>/$f.log; done'
-
-# 3. first failure of the affected service, and what preceded it
-ssh <host> 'journalctl --user --since "<date> <HH:MM>" | grep -a "invalid internal status" | head -2'
-ssh <host> 'journalctl --user --since "<date> <HH:MM>" | grep -aiE "Started|Created container" | tail -6'
-```
-
-## Traps (all hit for real)
-
-- **`/tmp` does not survive a reboot.** Launch experiment scripts from a
-  persistent path (e.g. `~/models/...`) and log beside them; if the repair path
-  is a reboot (common), `/tmp` scripts *and* any sampler output written there are
-  gone. On the Dank fleet: `/home/dank/models/logs/<tag>.log` survived while
-  `/tmp/x4*.sh` did not.
-- **A log's mtime is not proof the run finished.** A log truncated (or rotated)
-  while a script still holds the fd keeps growing from that point; a 6-line log
-  with an `ALL_DONE` marker means the earlier bulk was removed, not that nothing
-  ran.
-- **Long command lines are display-truncated by agent tools.** Wrap before
-  reading: `sed 's/\(.\{130\}\)/\1\n        /g'`. Otherwise you will report
-  parameters that were never used.
-- **Off-by-one file names silently skip evidence.** `x4b.log` vs
-  `x4b-experiment.log` in a `for f in …; do [ -f $f.log ]` loop made half the
-  artifacts invisible. List the directory and match exactly.
-- **Overlap is a hypothesis, not an observation.** Two resource-heavy
-  invocations in the same minute may or may not have been resident at once;
-  say so explicitly unless a sampler or cgroup counter proves it.
-- **Do not invent the cause.** Reconstruct the *window* (evidence) and keep the
-  *mechanism* separate: grep the proof that would establish it (`journalctl -k |
-  grep -i oom` for a killed process, driver logs for a GPU fault). If it is not
-  there, publish "trigger not established" — an invented cause is worse than an
-  open question because the next agent plans against it.
-
-## Fleet-specific sensitivity
-
-On this fleet, container healthchecks shell out to the **`podman` CLI**, so the
-first `invalid internal status … could not find any running process` in the user
-journal is an *early, per-minute* indicator of rootless-podman state death — and
-it timestamps the incident to within one healthcheck interval. That is how the
-18:58:30 Dankclaw break was pinned to the bench still loading at the time.
-
-## Worked example (2026-09-16, Dankclaw)
-
-| evidence | value |
-|---|---|
-| log mtimes | x4e-run 18:58:29, x4f-run 18:59:56 |
-| durations | E1 `real 4m53.946s` (llama-cli spam), F `real 5m24.785s` (exit 143 = killed) |
-| journal | first `invalid internal status` 18:58:30 |
-| ⇒ in flight | **F**, launched 18:54:31 (inside its 18:54:31–18:59:56 window) |
-
-Answer delivered: the exact flags of F, plus the note that the preceding
-`llama-cli` had written 3.19 GB of spinner lines and that the `/tmp` VRAM sampler
-was gone — so the overlap of the two 111 GiB-class invocations stayed marked as
-[INFERENCE] rather than becoming a claim.

agent/managed-skills/lan-client-split-dns-fix/SKILL.md

@@ -1,36 +0,0 @@
----
-name: lan-client-split-dns-fix
-description: Fix a LAN client timing out on *.franksdank.cloud because it resolves the public WAN IP (no 443 forward) instead of the Pi-hole LAN answer; diagnose by resolver and fix with a systemd-resolved route-only drop-in.
----
-
-# LAN client split-DNS fix (*.franksdank.cloud)
-
-Symptom: a LAN machine gets `connection timeout` (curl rc=28) on `https://*.franksdank.cloud`, or LAN-only names return NXDOMAIN, while the server is actually healthy.
-
-## Root cause
-Public DNS for `franksdank.cloud` = WAN IP `71.227.227.172`, which has **no 443 port-forward** (filtered). Pi-hole `192.168.0.143` returns the LAN answer `192.168.0.143` via `addn-hosts`. If the client's resolver is using an external server (1.1.1.1/quad9), it gets the WAN answer → SYN to a filtered port → timeout.
-
-## Diagnose (fast)
-```bash
-dig +short www.franksdank.cloud @192.168.0.143   # expect 192.168.0.143
-dig +short www.franksdank.cloud @1.1.1.1          # expect 71.227.227.172 (the trap)
-resolvectl status | grep -iE 'Current DNS|DNS Server'  # which server is active?
-# prove the server is fine by pinning the LAN IP:
-curl -sk --resolve www.franksdank.cloud:443:192.168.0.143 https://www.franksdank.cloud/ -o /dev/null -w '%{http_code}\n'   # 200
-```
-
-## Fix (systemd-resolved route-only drop-in)
-```bash
-sudo mkdir -p /etc/systemd/resolved.conf.d
-sudo tee /etc/systemd/resolved.conf.d/franksdank-lan.conf >/dev/null <<'EOF'
-[Resolve]
-DNS=192.168.0.143
-Domains=~franksdank.cloud ~franksdank.casa
-EOF
-sudo systemctl restart systemd-resolved
-resolvectl flush-caches
-```
-The `~` (route-only) prefix sends `*.franksdank.cloud` ONLY to Pi-hole and never to the default-route servers. Verify: `getent hosts www.franksdank.cloud` → 192.168.0.143.
-
-## Note
-This is a per-client fix. Every LAN client that must reach the homelab by name needs Pi-hole as its DNS (or this drop-in). The server side needs no change.

agent/managed-skills/live-llm-benchmark-hygiene/SKILL.md

@@ -1,70 +0,0 @@
----
-name: live-llm-benchmark-hygiene
-description: "Measure and tune a live LLM server without fooling yourself: workload-matched corpora, salted runs against warm prefix caches, turn-level clients, engine ledger for capacity, soak tests, and reboot verification."
----
-
-# Benchmarking a live LLM server honestly
-
-Use when measuring or tuning a served model (vLLM, llama.cpp, halogen, radiance,
-any OpenAI-compatible endpoint). The point is to avoid the four confounds that
-turned a night's numbers into fiction before each one was fixed.
-
-## 1. Match the corpus to the workload
-
-Synthetic token soup (`vllm bench serve --dataset-name random`) under-reports
-speculative decoding badly: measured 2.9-3.6 tok/update against 3.86 on real
-code, and acceptance — not step time — is what moves throughput. For a coding
-fleet, build the corpus from the repo's own sources: file body plus an
-instruction that requires reading it. Then `tok/update` and TPOT mean something.
-
-## 2. Salt every run so the prefix cache cannot help
-
-vLLM runs with `--enable-prefix-caching`, and a repeated corpus silently
-measures a warm cache. Put a unique per-run token in the first bytes (it is both
-the engine's cache key and, if a proxy routes on it, the routing key). Without
-this, the second run of any comparison is not comparable to the first.
-
-## 3. Measure turns, not only batches
-
-A batch of distinct prompts parallelises prefill and hides the real cost of a
-conversation: an agent loop re-sends its context every turn. Two workloads need
-two clients:
-
-- **batch/throughput** — `vllm bench serve --dataset-name custom --dataset-path
-  <jsonl>` with `--custom-output-len`, several concurrency levels.
-- **sequential turns** — a small client that sends turn after turn with growing
-  history and a large shared prefix, timing **TTFT of the first streamed chunk**
-  per turn. This is the only measurement that exposes prefix-cache behaviour.
-
-## 4. Read the engine's own ledger for capacity
-
-Do not infer memory from the card's size. The server prints the truth at boot:
-
-```
-Available KV cache memory: 5.44 GiB
-GPU KV cache size: 122,636 tokens
-Maximum concurrency for 114,688 tokens per request: 1.07x
-```
-
-That is the number to quote for "how many concurrent conversations fit".
-Related trap: the arena/KV budget is **not monotonic in chunk size** — on one
-host, available KV was 5.44 GiB at `CHUNK=8192`, 4.22 GiB at 16384 and 4.16 GiB
-at 4096 (where it would not even boot). Never assume "smaller chunk frees KV";
-measure three points before treating a knob as a lever.
-
-## 5. Soak before believing a tuning change
-
-Run several families concurrently for 15-30 minutes and count failures, not just
-latency: a nightly soak returned 155/155 answers with zero failovers while only
-25% of turns hit the cache — which is how the capacity ceiling was found as a
-*latency* effect rather than an error. Report hit/miss split, medians, and the
-min/max, because the median of a bimodal distribution describes neither mode.
-
-## 6. Verification that is not the benchmark
-
-- Run the client the fleet actually uses (e.g. `omp --model <alias> -p
-  --no-tools`) — a config file that parses is not a config that works, and a
-  non-obvious API mode (raw `/v1/completions`) may still be fine because the
-  client templates its own prompt.
-- A deployed inference unit is not proven until an **unattended reboot** brings
-  it back: verify `is-active`, `/health`, and one real completion afterwards.

agent/managed-skills/live-llm-telemetry-no-bench/SKILL.md

@@ -1,87 +0,0 @@
----
-name: live-llm-telemetry-no-bench
-description: "Measure a LIVE LLM endpoint's real prefill/decode/spec-decode performance and its PCIe cost from the runtime's own journald telemetry plus nvidia-smi dmon, without stopping the service or running a benchmark beside it — for hosts whose rules forbid benchmarking a serving GPU."
----
-
-# Live LLM telemetry without benchmarking
-
-Use when asked "how is the model performing?" on a host that SERVES production
-inference, where benchmarking policy forbids running `llama-bench`/`vllm bench`
-beside the live service. Instead of a synthetic bench, mine the runtime's own
-per-request telemetry — it measures the real workload and costs nothing.
-
-## 1. Where the numbers live
-
-| Runtime | Unit kind | Log command |
-|---|---|---|
-| llama.cpp (mainline, `-lv 3`) | root systemd unit | `sudo journalctl -u <unit> --since "-3h" --no-pager` |
-| halogen-flash-server | rootless Quadlet | `journalctl --user -u <unit> --no-pager` (**never** sudo — per-user storage) |
-
-llama.cpp emits, per completed task:
-
-```
-I slot print_timing: id 0 | task 82 | prompt eval time =  339.51 ms /  57 tokens ( 5.96 ms per token, 167.89 tokens per second)
-I slot print_timing: id 0 | task 82 |        eval time =  428.49 ms /  36 tokens (12.24 ms per token,  81.68 tokens per second)
-I slot print_timing: id 0 | task 82 |       total time =  768.00 ms /  93 tokens
-I slot print_timing: id 0 | task 82 |    graphs reused =         84
-I slot print_timing: id 0 | task 82 | draft acceptance = 0.75758 (   25 accepted /    33 generated), mean len =  3.27
-I slot print_timing: id 0 | task 97 | n_gen = 188, tg = 62.06 t/s, tg_3s = 62.39 t/s
-```
-
-halogen-flash-server emits one line per request:
-
-```
-serve_api: mtp 28 tok in 0.66s = 42.45 t/s | 16 rounds, commit 1.75/round | prompt 58, prefill 0.92s | detok 24us/tok
-```
-
-Grep both with `grep -E "print_timing|release|serve_api: mtp"`. Long lines get
-truncated in a terminal — redirect to a file/artifact and read it whole.
-
-## 2. Read the envelope, not one sample
-
-- `prompt eval` on a SHORT prompt is overhead-dominated (57 tok @ 168 t/s) — use
-  the longest prompt available for the prefill figure (1106 tok @ 1013 t/s).
-- `eval` (decode) and `draft acceptance` are content-dependent: measured
-  acceptance 0.46-0.76 on one unchanged build. Never quote a single task as
-  "the model's speed", and never compare a live sample to a deploy-time
-  baseline without a masked `llama-bench`.
-- `graphs reused` > 0 and `selected slot by LCP similarity` prove prefix-cache
-  reuse is working; `prompt 53 (48 cached), prefill 0.05s` vs `1.58s` cold is
-  prefix caching caught in the act.
-- A runtime's own window metric (`tg_3s`) is a cross-check on your arithmetic.
-
-## 3. Attribute PCIe without touching the model
-
-Run the counter in the background and issue ONE ordinary request:
-
-```bash
-ssh host 'nvidia-smi dmon -s t -c 16 -o T & sleep 0.5; \
-  curl -s -m 60 -o /dev/null -w "http=%{http_code} wall=%{time_total}s\n" \
-    http://127.0.0.1:7771/v1/chat/completions \
-    -H "Content-Type: application/json" \
-    -d "{\"model\":\"<alias>\",\"messages\":[{\"role\":\"user\",\"content\":\"...\"}],\"max_tokens\":200}"; \
-  wait; sudo journalctl -u <unit> --since "-1min" --no-pager | grep print_timing'
-```
-
-`-s t` = `rxpci`/`txpci` in MB/s. Measured on a fully-VRAM-resident 27B on a
-3090: peak ~230 MB/s against a Gen4 x4 link worth ~7,900 MB/s = ~3%, and 0 at
-idle. Decode traffic is essentially the sampled-logits stream, so on a 4-lane
-link the transport is a non-factor — say so with the measured number instead of
-hand-waving.
-
-## 4. Traps
-
-- **`wait` or you lose the telemetry.** If the remote command exits while the
-  curl is backgrounded, ssh closes the session and the client dies: the journal
-  shows `W srv stop: cancel task, id_task = N` and NO `print_timing` line, while
-  curl still reports a plausible status. Always end the compound command with
-  `wait`.
-- **Do not run a GPU workload to "get a reading".** One max_tokens=200 request
-  is production traffic and is fine; anything that pins the card for a while is
-  a bench and needs the stop-and-mask procedure first.
-- **Check which instance you are reading.** Quadlet units run `--replace --rm`
-  and root units get restarted: `systemctl status <unit> | head -12` for the
-  start time; `task N` IDs increment per request and jump across restarts.
-- **The engine's own ledger outranks your model of it** — quote `KV self size`,
-  `n_slots`, `n_ctx_slot` and the VRAM figure from the log/NVML rather than
-  arithmetic.

agent/managed-skills/llama-bench-fleet-sweep/SKILL.md

@@ -1,51 +0,0 @@
----
-name: llama-bench-fleet-sweep
-description: "Run a llama-bench sweep (batch/ubatch, ncmoe, split, or any dimension) on a Dank-fleet LLM host correctly: stop the service, run from a transient unit, one invocation per cell, sample VRAM, restore and verify — including the llama-bench vs llama-server traps that have already produced wrong numbers here."
----
-
-# llama-bench sweeps on a Dank-fleet LLM host
-
-Use when tuning a llama.cpp service's serve-time parameters (batch/ubatch, `-ncmoe`, `-sm`/`-ts`, KV type) offline, i.e. any question of the form "how large may X be / what does X buy".
-
-Reference implementation: `scripts/bench-batch-ubatch.sh` (repo) — read it before writing another harness.
-
-## Preconditions
-
-1. **Stop the service** — Benchmarking Policy in AGENTS.md is mandatory: a bench beside a live service starves the same GPU and corrupts results. Never bench against the live port.
-2. **Run from a transient unit, not a shell:**
-
-   ```bash
-   sudo systemd-run --unit=<name> --collect --no-block --property=Type=oneshot \
-        --property=LimitMEMLOCK=infinity --property=User=dank \
-        /home/dank/serverAppConfigs/scripts/<harness>.sh
-   ```
-
-   The deploy's unit sets `LimitMEMLOCK=infinity` for the ROCm staging buffers; an SSH shell gets 8 MiB (`ulimit -l`), which fails the load for a reason unrelated to what you are measuring. Passwordless sudo is available on both hosts.
-
-## The traps (each one has produced a wrong number)
-
-- **`/` vs `,`.** llama-bench splits `/` = devices *within one configuration*, `,` = *sweep values*. A legitimate llama-server `--tensor-split 36,29` becomes `-ts 36/29`; the comma form silently benches each card alone. Same for `--device ROCm0,ROCm1` → `-dev ROCm0/ROCm1`.
-- **No `-c`.** `n_ctx = n_prompt + n_gen + n_depth`. The context is a consequence of the test shape, so a bench can NEVER prove that a configuration's activation buffers fit beside the *deployed* KV — that requires starting the real service at the real context and slot count.
-- **A test that cannot allocate calls `exit(1)`**, killing the whole process. One invocation per cell, or one infeasible value loses every later cell — and infeasibility is usually what you are looking for.
-- **No `--spec-*` and no `--parallel`.** Decode figures are the *drafterless* baseline, and slots are never exercised. `-d <n>` is what puts KV in front of the measurement: a `tg` instance runs with `n_prompt = 0`, i.e. at ~zero depth otherwise.
-- **A cell that looks fast because its allocation was smaller than intended is indistinguishable from a fast cell** unless you sample VRAM. Sample `rocm-smi --showmeminfo vram --csv` every 2 s for the cell's whole life and report min-free per card.
-- **`local a=$1 b=$x` expands `$x` before the builtin assigns anything** — under `set -u` that is `b: unbound variable`, and it fires *after* your harness has already stopped the production service. Declare in two statements.
-- **Ranges exist:** `-ub 512-2048+256` == `512,768,...,2048`.
-
-## Shape of a good harness
-
-- One `llama-bench` invocation per cell; append one TSV row per cell as it completes, so progress is readable mid-run and an interrupted sweep resumes (skip cells whose `bench.json` is non-empty).
-- Write `bench.json` per cell (`-o json`), plus `bench.log` (`--progress`), plus the VRAM trace.
-- **Stop the service in the harness and restore it from a `trap ... EXIT`** — and still verify the service afterwards. A harness that crashes mid-run leaves production down.
-- Match the deployment's load flags exactly (`-ngl`, `-sm`, `-ts`, `-fa`, `-ctk/-ctv`, `-t`), or you are benching a different service. Note `-ts`/`-dev` take slashes here while the unit takes commas.
-
-## Verify before burning hours
-
-Dry-run the harness against a **stub** `llama-bench` (`BIN=/tmp/stubdir`, plus `SERVICE=<nonexistent-name>` so the real service is untouched) with a tiny grid, including one cell forced to `exit 3`: prove the TSV row, the failure capture, and the resume logic before the real sweep. Then validate the real flag set in ~1 s by pointing `-m` at a nonexistent file — argument errors surface *before* the model load.
-
-## Interpreting results
-
-- A prefill difference smaller than the run-to-run spread is not a result; check each cell's own `stddev_ts`.
-- Single-sequence decode is usually flat across batch/ubatch — that knob is prefill-side.
-- Batch ≥ ubatch always (llama.cpp clamps ubatch to batch, so smaller-batch cells silently duplicate).
-- Report what the harness cannot see (drafter, slots, real context) rather than implying the sweep settled it.

agent/managed-skills/llama-bench-host-sweep/SKILL.md

@@ -1,109 +0,0 @@
----
-name: llama-bench-host-sweep
-description: "Run a llama-bench sweep (batch/ubatch, split, quant, depth) on a live fleet LLM host without fooling yourself: what llama-bench cannot express, the separator trap, per-cell isolation, the memlock/transient-unit requirement, and the dry-run that must precede a multi-hour run."
----
-
-# Running a llama-bench sweep on a fleet host
-
-Use when a bench must answer a serving question on one of the LLM hosts (Dankclaw
-frank-large/frank-small, Dankserver frank-medium) — batch/ubatch sizing, split
-choice, KV type, depth behaviour.
-
-Repo policy applies: **stop the production service first** (AGENTS.md Benchmarking
-Policy). The harness below does that for you and restores it.
-
-## 1. What llama-bench cannot express — decide this BEFORE designing the sweep
-
-| Not available | Consequence |
-|---|---|
-| `-c/--ctx-size` | `n_ctx = n_prompt + n_gen + n_depth` (llama-bench.cpp). The context is a consequence of the test shape, so **the deployment's real allocation is never tested** (e.g. a 16 GiB KV at 262144 over 2 slots). Only starting the real unit answers "does it fit". |
-| `--parallel` | single sequence only; no slot shape, no concurrency effect. |
-| `--spec-*` | no drafter. Any `tg` number is the **autoregressive baseline**, not the served rate. Useful as a control, never as the deployed figure. |
-| warmup per config | a warmup run is itself a full-size run; at 64k prompts `--no-warmup` buys a second sample for the same wall clock. |
-
-`-d N` is a depth PREFIX: it puts N tokens of KV in place before the measured run and
-is cached across reps through a `llama_state_seq_*` save/restore, so the fill is paid
-once per cell and is not part of the measured number. Without `-d`, a `tg` test runs at
-~zero depth and the number is meaningless for long conversations.
-
-## 2. Two traps that silently change the experiment
-
-- **Separators.** `/` separates devices *within one configuration*; `,` separates
-  configurations to sweep. `-ts` and `-dev` both follow this. `-ts 36,29` parses as the
-  single value `36` → everything lands on one card and you have benched a different
-  machine. llama-server takes the comma form for the same pair. List-valued args also
-  accept ranges: `-ub 512-2048+256`.
-- **memlock.** The deploy units set `LimitMEMLOCK=infinity` because the ROCm backend
-  pins staging buffers; a plain SSH shell on these hosts has soft=hard=8 MiB
-  (`ulimit -l`), so a load can fail for a reason unrelated to the model. Run the sweep
-  inside a transient unit:
-
-  ```bash
-  sudo systemd-run --unit=fm-bench --collect --no-block \
-      --property=Type=oneshot --property=LimitMEMLOCK=infinity --property=User=dank \
-      /home/dank/serverAppConfigs/scripts/bench-batch-ubatch.sh
-  ```
-
-  This also survives an ssh disconnect, unlike a foreground remote process.
-
-## 3. One invocation per cell
-
-A test that cannot run calls `exit(1)` immediately (failed depth/prompt/gen run), so a
-single cross-product invocation loses **every later cell** the moment one ubatch fails to
-allocate — and infeasibility is usually one of the questions being asked.
-
-Drive the grid from a shell loop: one llama-bench call per (b, ub), continue on a
-non-zero exit, record why, and skip cells whose output file already exists and is
-non-empty (that makes an interrupted sweep resumable). Write rows to a TSV as the sweep
-proceeds so progress is readable mid-run.
-
-## 4. Sample VRAM per cell, covering the load
-
-Batch/ubatch sizes the activation buffers, so a cell that looks fast because its
-allocation was silently smaller must be distinguishable from one that is genuinely fast.
-Poll `rocm-smi --showmeminfo vram --csv` every 2 s for the cell's whole life and keep the
-peak (minimum free) per card. Correlate by taking the window between the cell's start and
-end.
-
-## 5. Always restore the service from a trap
-
-`trap <restart> EXIT` in the harness, plus a manual check afterwards. A harness that
-stops a production unit and then dies leaves inference down; this fleet has had that
-happen.
-
-## 6. Dry-run before a multi-hour run
-
-Stub the binary so the whole path is exercised without loading a 29 GiB model:
-
-```bash
-# /tmp/benchstub/llama-bench prints the -o json shape and exits non-zero for one ubatch
-SERVICE=frank-medium-dryrun BIN=/tmp/benchstub MODEL=/tmp/benchstub/m.gguf OUT=/tmp/bub-test \
-    N_PROMPT=8 N_GEN=4 N_DEPTH=0 REPS=1 BATCHES="2048 4096" UBATCHES="512 1024" \
-    bash scripts/bench-batch-ubatch.sh
-```
-
-`SERVICE=<nonexistent>` means the trap and the stop are never reached, so the real
-service is untouched. Check that a FAILED cell still produces a row with the right column
-count before trusting the table.
-
-## 7. Read the result sanely
-
-- Bracket the sweep: run the reference cell (the deployed configuration) first and
-  again last, so drift over hours is visible.
-- Sanity-check one number against the runbook's prior figures before believing the grid.
-- Remember what the grid does NOT prove: it did not run the drafter, the slots, or the
-  deployment's context. The winner still has to be applied and verified live.
-
-## Pitfall while writing such a harness
-
-`local b=$1 ub=$2 dir="$OUT/b${b}_ub${ub}"` expands **all** the builtin's arguments
-before the builtin assigns any of them, so under `set -u` this aborts with
-`b: unbound variable`. Declare in two statements:
-
-```bash
-local b=$1 ub=$2
-local dir="$OUT/b${b}_ub${ub}" sampler start rc wall
-```
-
-Ref: `scripts/bench-batch-ubatch.sh` in serverAppConfigs (the fleet's implementation of
-all of the above).

agent/managed-skills/llama-cpp-spec-decode-ab/SKILL.md

@@ -1,58 +0,0 @@
----
-name: llama-cpp-spec-decode-ab
-description: "Measure speculative-decoding configurations (drafter vs MTP vs ngram-mod vs chains) on a live llama.cpp endpoint without fooling yourself: shape taxonomy, negative controls, unit-stopped protocol, window sweeps, and the traps that produce false verdicts."
----
-
-# Spec-decoding A/B on a live llama.cpp endpoint
-
-Use when asked "is X faster than Y?", "can these be chained?", or "should we use MTP/ngram/drafter".
-Verdicts here are **traffic-shaped**: a proposer can only draft text it has seen, so a single prompt shape answers the wrong question.
-
-## Step 0 — settle chaining in the source, before spending GPU time
-
-`common/speculative.cpp` (`common_speculative_init_result`) builds `configs` and gates each type:
-
-- **draftless** (ngram family: simple, map-k, map-k4v, mod, cache) → added unconditionally
-- **context-backed** (`draft-mtp`, `draft-dflash`, `draft-dspark`, `draft-eagle3`) → require `params.draft.ctx_dft != nullptr`
-
-Consequences: one context-backed drafter + any number of ngram types chains fine; **two context-backed types never do**
-(`draft-mtp,draft-dflash` → `context type MTP requested but model doesn't contain MTP layers`).
-Internal order is fixed (ngram first, then drafters), so CLI order is irrelevant.
-
-## Step 1 — shapes (the part that decides the answer)
-
-| Shape | Prompt | Who it can favour |
-|---|---|---|
-| prose | 1k filler + "reply with one sentence" | model-backed drafters only |
-| echo | ~1.5k passage of UNIQUE lines + "repeat verbatim" | ngram (output IS the prompt) |
-| structured | repetitive output, no prompt overlap | ngram, if spans really repeat |
-| identity | short deterministic question | controls for lossless verification |
-
-Design the echo passage so no long span repeats *inside* it — otherwise you measure the matcher against itself.
-For `structured`, use lines that genuinely repeat: unique ids per line make it a negative control, not a repetition test.
-
-## Step 2 — protocol
-
-1. Stop the unit (`sudo systemctl stop <svc>`), `pkill -x llama-server`; wait for the port.
-2. Per config: write an env file with overrides, start the serve script with `FRANK_*_ENV_FILE`, poll `/health` until ok, then measure.
-3. **Restore the unit in a `finally`** and shout if it does not come back healthy. A crashed harness that leaves production down is worse than no harness.
-4. Compare greedy text across configs per shape; warn loudly on any mismatch (speculation must be lossless) and on any EMPTY answer (an empty answer makes the comparison vacuous).
-
-## Step 3 — read the right counter
-
-- Per request: `timings.draft_n` / `timings.draft_n_accepted`, `predicted_per_second`.
-- Cumulative: `llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`.
-- **`draft_n == 0` while tokens flow = the proposer is not engaged**, even when every flag looks right.
-- A config with zero drafting is a useful *negative control*: it proves the other config's drafts came from the proposer under test.
-
-## Traps that produce false verdicts
-
-- **Thinking eats the answer.** Echo/structured prompts with thinking on put the copy in `reasoning_content`; `content` is empty, so identity checks compare `''` to `''`. Disable per request: `chat_template_kwargs: {"enable_thinking": false}`.
-- **No banner.** ngram-mod prints nothing at startup; the journal shows only the drafter's `adding speculative implementation`. Never conclude "chain not engaged" from the load log.
-- **Windows change the verdict.** ngram-mod `--spec-ngram-mod-n-min/--spec-ngram-mod-n-max` decide which traffic it can see: too short → proposals on ordinary prose fragments, rejected, verification still paid (measured 43.9 → 36.0 tok/s). Sweep the window in the same run as the configs.
-- **Unused-tensor warnings are normal**: with `draft-dflash` on an MTP-carrying GGUF, `blk.<n>.nextn.*` is `unused ... ignoring` — correct, not a truncated file.
-- **Harness hygiene**: absolute script paths (a relative path plus `cd` to its own dir double-counts the prefix), `pkill -x` never `-f`.
-
-## Interpreting
-
-Report `decode tok/s` per shape plus acceptance (`accepted/draft_n`) and mean accepted length; a config that wins on echo and loses on prose is a traffic decision, not a winner. Then wire the winner into the deployed env file with the measured table as its justification comment, and assert it in the test seam.

agent/managed-skills/llama-flash-attn-qsa-fix/SKILL.md

@@ -1,73 +0,0 @@
----
-name: llama-flash-attn-qsa-fix
-description: Restore --flash-attn on for qwen4exp models when QSA indexer path aborts with GGML_ASSERT F32 error
----
-
-## Problem
-`--flash-attn on` aborts graph build for qwen4exp models:
-```
-GGML_ASSERT(a->type == GGML_TYPE_F32)
-at ggml/src/ggml.c:5371, backtrace through build_attn_qsa
-```
-Root cause: `build_attn_qsa` calls `ggml_fill` on the F16-cast kq mask returned by `get_kq_mask()` when flash attn is enabled, but `ggml_fill` requires F32.
-
-## Solution
-Apply the QSA F32-mask patch that mirrors the MLA sparse path (`llm_graph_context::build_attn`):
-1. Use the F32 base mask (`inp->self_kq_mask`) instead of the F16-cast version for the top-k surgery
-2. Cast to F16 only at the attention call when `flash_attn` is enabled
-
-### Patch hunks (unified diff format)
-```diff
-@@ -560,9 +565,13 @@ ggml_tensor * llama_model_qwen4exp::graph::build_attn_qsa(
-         ggml_build_forward_expand(gf, mctx_cur->cpy_v(ctx0, v_cur, v_idxs, il));
-     }
--    ggml_tensor * kq_mask = inp->get_kq_mask();
-+    // get_kq_mask() returns the F16-converted mask when flash attention is
-+    // on, but ggml_fill/set_rows require F32. Mirror the MLA sparse path
-+    // (llm_graph_context::build_attn): run the top-k mask surgery on the F32
-+    // base mask and cast to the attention type only at the end.
-+    ggml_tensor * kq_mask = inp->self_kq_mask;
-     // prepare new kq mask - starts filled with -INFINITY
-     ggml_tensor * kq_mask_all = ggml_fill(ctx0, kq_mask, -INFINITY);
- 
-@@ -592,9 +601,12 @@ ggml_tensor * llama_model_qwen4exp::graph::build_attn_qsa(
-     ggml_tensor * q = q_cur;
-     ggml_tensor * k = mctx_cur->get_k(ctx0, il);
-     ggml_tensor * v = mctx_cur->get_v(ctx0, il);
--    ggml_tensor * cur = build_attn_mha(q, k, v, nullptr, kq_mask_top_k, nullptr, nullptr, kq_scale, il);
-+    // flash attention consumes an F16 mask; softmax consumes F32 as-is
-+    ggml_tensor * kq_mask_attn = inp->cparams.flash_attn ? ggml_cast(ctx0, kq_mask_top_k, GGML_TYPE_F16) : kq_mask_top_k;
-+
-+    ggml_tensor * cur = build_attn_mha(q, k, v, nullptr, kq_mask_attn, nullptr, nullptr, kq_scale, il);
-     cb(cur, "kqv_out", il);
-     return cur;
- }
-```
-
-## Procedure
-1. **Apply patch to fork:**
-   ```bash
-   cd /home/dank/GitPackages/ROCmFPX
-   git apply /path/to/qwen4exp-qsa-f32-mask.patch
-   make -j16 llama-server
-   ```
-2. **Validate live (on test port):**
-   ```bash
-   sudo systemctl stop frank-default-large
-   nohup ./build-qwen38/bin/llama-server \n    --model ... --flash-attn on --ctx-size 32768 --fit off -t 16 -tb 16 \n    --port 7999 > /tmp/fa-test.log 2>&1 &
-   sleep 90 && grep 'model loaded' /tmp/fa-test.log && curl http://127.0.0.1:7999/health
-   ```
-3. **Verify completion:** Send a real `/v1/chat/completions` request — if it completes without asserts, the fix works.
-4. **Deploy production:** Commit config (FLASH_ATTN=on), push, pull on target, restart service.
-
-## Verification
-- Model loads without asserts
-- Health endpoint returns 200
-- Real completion succeeds (finish: "stop")
-- Journal clean: no `GGML_ASSERT`, no `cudaMalloc`, no `missing tensor`
-- Flash attention engaged (check server logs for "flash attention" or timing improvement vs off)
-
-## Known Constraints
-- Works only with charlie12345/ROCmFPX build-qwen38 (PR #27742 qwen4exp merged)
-- Does NOT work with CIRU runtime, upstream llama.cpp, or stock binaries
-- ctx-size cap remains 32768 for full-model GPU residency (87 GiB model + KV buffer OOMs beyond)

agent/managed-skills/llama-server-arch-verify/SKILL.md

@@ -0,0 +1,50 @@
+---
+name: llama-server-arch-verify
+description: Verify llama.cpp binary supports a model architecture without relying on binary grep
+---
+
+# llama-server Arch Verification
+
+## Problem
+`grep -ac <arch> llama-server` returns 0 for ALL architecture names (including ones the binary is actively serving). The arch strings are stored in a packed/constructed form that grep can't catch.
+
+## Solution
+Verify arch support via the source registration table, not binary grep:
+
+1. **Check the source registration** (`src/llama-arch.cpp` or `src/llmarch.h`):
+   ```bash
+   grep -n "gemma4\|qwen3_5\|<arch>" src/llama-arch.cpp
+   ```
+   If the arch is registered in the source at the commit the binary was built from, it's supported.
+
+2. **Verify the binary is current**:
+   ```bash
+   stat -c "%y" /path/to/llama-server
+   git log -1 --format=%cd <commit>
+   ```
+   The binary mtime should be after the commit date.
+
+3. **Check for model-specific source files**:
+   ```bash
+   ls src/models/<arch>.cpp src/models/<arch>-*.cpp 2>/dev/null
+   ```
+
+4. **Check for spec-type support** (for MTP/draft models):
+   ```bash
+   /path/to/llama-server --help 2>&1 | grep "spec-type\|model-draft"
+   ```
+
+## KV Cache Types
+The build's allowed `--cache-type-k` / `--cache-type-v` values:
+`f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`
+
+**No true fp8 KV dtype** — `q8_0` is the "FP8 class" (1.0625 B/elem, 8.5 effective bits).
+
+## Separate Draft Model (MTP assistant)
+For models where the MTP head is a SEPARATE file (not in-file like Swift):
+```bash
+--spec-type draft-mtp \
+--spec-draft-n-max 3 \
+--spec-draft-model /path/to/mtp-assistant.gguf
+```
+The `--spec-draft-model` flag (aka `-md`, `--model-draft`) loads the separate draft file.

agent/managed-skills/llamacpp-cross-backend-device-split/SKILL.md

@@ -1,97 +0,0 @@
----
-name: llamacpp-cross-backend-device-split
-description: "Decide whether two unlike devices (NVIDIA CUDA + AMD Vulkan, or any two backends) can serve one llama.cpp model, and measure the attempt honestly on a serving host — the two scheduler failure modes, what is actually expressible, the dual-backend build recipe, and the measurement traps that make results meaningless."
----
-
-# Cross-backend device splits in llama.cpp
-
-Use when someone proposes serving ONE model across two devices of different
-backend families (NVIDIA CUDA + AMD Vulkan/HIP, or a dGPU + an APU), and you
-need to know whether it is expressible before spending a build and a service
-outage on it.
-
-## Answer first: it is not expressible
-
-A weight tensor in one *device* buffer cannot be consumed by an op the scheduler
-placed on another device. Two observed failure modes (mainline llama.cpp
-`37b53fd`, 2026-09-16):
-
-| attempt | result |
-|---|---|
-| per-tensor: `-ot "exps=Vulkan1"` with `-sm none -mg 0 -dev CUDA0,Vulkan1` | `ggml-backend.cpp:941: pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE)` — scheduler abort |
-| per-layer: `-sm layer -ts a,b -dev CUDA0,Vulkan1`, or layer-scoped `-ot` | model load fails, message is only `failed to load model` |
-| device + **CPU**: `-ot "exps=CPU"` | **works** — the CPU backend is the universal fallback and host buffers can be copied |
-
-The rule: **the CPU is the only legitimate partner in a heterogeneous split.**
-Everything else must be single-vendor. On a CUDA+AMD host that means a card can
-be the trunk *or* the offload host, never the complement of an AMD-resident
-peer — the same wall the R9700 pair hits from the RCCL/atomics direction.
-
-If the answer is "not expressible", stop and consider the expressible
-alternatives in this order: (1) two independent endpoints, one model per device;
-(2) single-vendor layer split (`-sm layer`, no collective needed); (3)
-device + CPU per-tensor offload.
-
-## Build the dual-backend binary
-
-Needs both toolchains in one tree; Vulkan needs `glslc` + `libvulkan` headers
-(already present on this fleet's hosts).
-
-```bash
-cmake -B build-dual -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
-  -DGGML_CUDA=ON -DGGML_VULKAN=ON -DCMAKE_CUDA_ARCHITECTURES=<sm> \
-  -DGGML_CCACHE=ON -DLLAMA_CURL=OFF
-cmake --build build-dual --target llama-bench llama-cli llama-server -j $(nproc)
-```
-
-**Device names are not what you expect.** `--list-devices` on an
-NVIDIA-plus-APU host lists the discrete card under BOTH backends (e.g. `CUDA0`
-and `Vulkan0`) and the APU last (`Vulkan1`). The iGPU is `Vulkan1`, not
-`Vulkan0`. Valid `-ot` buffer types are whatever that list prints; a wrong name
-makes llama.cpp print the valid set and exit, so the error is self-diagnosing.
-
-## Measure it without fooling yourself
-
-- **`tg` in llama-bench runs with `n_prompt = 0`.** A plain `-n 64` row is a
-  ~zero-depth decode: no KV traffic at all. Anything about context cost or KV
-  placement must use `-d/--n-depth`, which fills the cache before timing.
-- `-p`/`-n` lists build separate instances (prefill instances with `n_gen = 0`,
-  generation instances with `n_prompt = 0`), and `n_ctx = n_prompt + n_gen`.
-- Compare at **depth-matched** settings, and remember prefill into a deep cache
-  is the slow case (e.g. 456 tok/s fresh vs 232 tok/s into 64k).
-- Sample the actual engines during the run (`/sys/class/drm/card*/device/gpu_busy_percent`,
-  `nvidia-smi`) — an idle device or a 100 % APU is the difference between a real
-  measurement and a silent CPU fallback.
-- Watch the **host memory** figure, not just tok/s. A Vulkan-resident model lives
-  in device buffers (committed, not reclaimable), whereas an mmap'd CPU-placed
-  tensor is page cache — the same model can commit 116 GiB one way and 95 GiB
-  the other. On a host with co-tenants, that decides deployability.
-
-## Serving-host discipline
-
-- **Stop and mask every LLM service first** (repo benchmarking policy). For a
-  rootless Quadlet, move the `.container` out of
-  `~/.config/containers/systemd/` and `daemon-reload` — the generator re-creates
-  the unit from that file on every reload, so `mask` is not the lever.
-- **`systemctl mask` is refused for a relinked root unit** whose
-  `/etc/systemd/system/<svc>.service` is a symlink into the repo: it exits with
-  `File ... already exists and is a symlink to ...`. Stop it and ensure nothing
-  restarts it, or move the symlink aside.
-- Root units need `sudo systemctl start`; without it you get
-  `Access denied as the requested operation requires interactive
-  authentication` (polkit), which reads like a permissions bug.
-- Export the pinning guard: **mainline is `GGML_CUDA_NO_PINNED`** (ik_llama.cpp
-  was `GGML_CUDA_NO_PINNED_WEIGHTS`). Without it a CPU-overridden tensor can land
-  in a pinned `CUDA_Host` buffer — the failure mode that page-locked 108 GiB and
-  starved a host's co-tenants until a power cycle.
-- Restore: move the quadlet back, `daemon-reload`, `reset-failed`, start, then
-  verify `/health`, `/v1/models` **and the engine's own boot ledger** (e.g.
-  halogen's `kv pool:` lines, which say whether the configured pool was clamped).
-
-## Calibration point
-
-On a 121 GiB Strix Halo host with a 3090: a 177B MoE (112 GiB GGUF, 12 of 48
-layers carrying KV) ran **32.6 tok/s shallow / 25.8 tok/s at 64k depth / 456 tok/s
-prefill** with everything on the iGPU via Vulkan, and **12.1 tok/s** with the
-same trunk but experts on the CPU. If a proposed split cannot beat the
-iGPU-only number at matched depth, it is not worth the failure modes above.

agent/managed-skills/llamacpp-device-split-feasibility/SKILL.md

@@ -1,62 +0,0 @@
----
-name: llamacpp-device-split-feasibility
-description: "Decide which multi-device arrangements llama.cpp can actually express on a heterogeneous host (mixed vendors, iGPU+dGPU), and measure them without fooling yourself — covers the expressibility matrix, llama-bench's /-vs-, device/split syntax, independent placement proof, and the stop/restore path."
----
-
-# When to use
-
-Asked "can we split this model across GPU A and GPU B?" on a host where the devices are mixed (NVIDIA + AMD, iGPU + dGPU) or where the obvious arrangement is "weights on device X, different weights on device Y". Also use before *believing* any earlier "it cannot be done" conclusion about a multi-device split — that verdict is easy to produce by accident.
-
-# 1. Classify the arrangement before running anything
-
-llama.cpp expresses exactly two things across devices, and they are not interchangeable:
-
-| arrangement | how | expressible? |
-|---|---|---|
-| **Layer split** — layers 0..n on dev A, the rest on dev B | `-sm layer -ts <shareA>/<shareB>` | **yes**, no collectives needed; works cross-vendor (CUDA + Vulkan measured) |
-| **Per-tensor placement** — e.g. attention+KV on A, experts on B | `-ot <regex>=<dev>` | **NO.** `--override-tensor` sets `has_tensor_overrides()`, a required term of the scheduler's `pipeline_parallel` flag (the only read site is `llama-context.cpp`, in the `pipeline_parallel = ...` expression). That flag sets `n_copies = parallel ? MAX : 1` in `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a non-primary device has no backend and the graph aborts at `ggml-backend.cpp` (`pre-allocated tensor (...weight) in a buffer (...) that cannot run the operation (NONE)` — `(NONE)` is the leaf's own op name). `-ot x=CPU` works only because the CPU backend is last in the list. |
-
-So: if the plan needs *tensors* on a specific device, stop — the answer is no, for any vendor mix, and pure-Vulkan vs CUDA+Vulkan changes nothing. If it needs *layers*, it is expressible and worth measuring.
-
-Layer-split arithmetic to check before promising anything:
-
-- A whole layer carries its experts, if the model is MoE. Compute GiB/layer, not GiB/param-group.
-- Only some layers carry KV (e.g. every 4th for hybrid attention). A share of layers is therefore only a share of KV — a contiguous range rarely holds the layers you care about.
-- Big non-repeating tensors (per-layer-token tables etc.) land on the main device and cannot be moved by `-ts`; they cap how small the primary device's footprint can get.
-
-# 2. Get the harness syntax right, or every number is fiction
-
-`llama-bench` writes `-dev <dev0/dev1/...>` and `-ts <ts0/ts1/..>`: **`/` separates values within one configuration, `,` separates configurations to sweep.**
-
-- `-dev CUDA0,Vulkan1` = **two single-device runs**, never a two-device run.
-- `-ts 0.85,0.15` = two sweep values, each becoming `[0.85, 0, 0, …]` → device 0 takes everything.
-
-Write `-dev A/B -ts x/y`. `llama-cli`/`llama-server` use the classic comma form — which is why a failure seen only under llama-bench is a harness bug, not an engine limit.
-
-Cheap pre-flight: run with the intended flags and read the tool's own reported parameters (llama-bench's table has a `dev` column). If a row says one device, your intent was not parsed.
-
-# 3. Prove placement — never trust the label
-
-Sample the non-primary device *while the run is loaded*:
-
-```bash
-( for i in $(seq 1 60); do nvidia-smi --query-gpu=memory.used --format=csv,noheader; sleep 10; done ) > /tmp/vram.txt &
-```
-
-Expect roughly (layers × GiB/layer + that device's KV) resident. Tens of MiB means the split did not happen. Cross-check: several "different" configurations that all produce the same tok/s to within 0.2 % are probably all the same single-device run.
-
-# 4. Measure at the depth that matters
-
-- `llama-bench`'s `tg` runs with `n_prompt = 0`, i.e. ~zero depth. Use `-n 32 -d 65536` (or deeper) or the KV term — the entire reason for the split — is absent.
-- Compare against the **single-device baseline on the fast-memory device**, and against whatever production runtime already serves the model. A split that wins by a few percent at 64k depth may lose at 200k, because the KV term grows with context and prefill cost grows with the target depth.
-
-# 5. Stop / restore on a host that serves
-
-- Stop and mask the production units first (benchmarking policy). For rootless quadlets the lever is the `.container` file (move it aside + `daemon-reload`); `Restart=always` otherwise recreates the service mid-run. `systemctl mask` is refused for a unit that is a symlink into a repo checkout.
-- After restoring, verify `/health`, `/v1/models`, and the boot ledger numbers — and keep a memory-headroom note: the same model can come back with less contiguous host memory than before (a reboot re-fragments), which decides whether the next co-tenant fits.
-- Landmine seen on a rootless-podman host after a big GPU run: a dead rootless **pause process** takes *all* container management down (every `podman` command: `invalid internal status … could not find any running process`; container starts then fail because the rootless netns holding the bridges is gone: `failed to bind udp listener on 10.89.x.1:53`). `podman system migrate` panics (5.7.0, `teardownNetworkBackend`). Repair is a reboot. Before rebooting, check `loginctl show-user <u> -p Linger` and that units carry `[Install] WantedBy=default.target`, so everything returns by itself.
-- Do not publish a cause you did not check: `journalctl -k | grep -i oom` decides whether "memory pressure killed it" is a finding or a guess. An invented narrative is worse than an open question.
-
-# 6. Report shape that survived review
-
-State (a) which arrangement was tested and that the harness parsed it, (b) the placement evidence, (c) the depth-matched numbers next to the single-device baseline, (d) the memory cost on the borrowing device, and (e) whether the split can reach its own premise (KV on the fast device, long context) — not just that it runs.

agent/managed-skills/llamacpp-drafter-kv-split-sizing/SKILL.md

@@ -1,168 +0,0 @@
----
-name: llamacpp-drafter-kv-split-sizing
-description: "Size a llama.cpp multi-GPU layer-split deployment that carries a speculative drafter: read the drafter's KV law from its GGUF header, then find the split balance by measurement (the batch asymmetry, the ctx/floor trade, and the OOM-vs-page-fault signatures)."
----
-
-# Sizing a llama.cpp layer-split with a speculative drafter
-
-Use when a multi-GPU llama.cpp deployment carries a draft model (`--spec-draft-model`)
-and the question is *where do the layers go and how big can the context be*. The two
-effects below decide it, and neither is visible in the flags.
-
-## 1. Read both GGUF headers first
-
-The drafter's own properties set the memory bill. `llama-server` prints almost none of
-them, and the loader's ledger is not printed at default verbosity.
-
-Minimal GGUF KV reader (no tooling needed) — write `ggufmeta.py`:
-
-```python
-import struct, sys
-
-def rd(path):
-    f = open(path, 'rb')
-    assert f.read(4) == b'GGUF'
-    ver = struct.unpack('<I', f.read(4))[0]
-    struct.unpack('<Q', f.read(8))                     # tensor count
-    nk = struct.unpack('<Q', f.read(8))[0]             # kv count
-    def s():
-        n = struct.unpack('<Q', f.read(8))[0]
-        return f.read(n).decode('utf-8', 'replace')
-    # 0=UINT8 1=INT8 2=UINT16 3=INT16 4=UINT32 5=INT32 6=FLOAT32 7=BOOL
-    # 8=STRING 9=ARRAY 10=UINT64 11=INT64 12=FLOAT64
-    SCALAR = {0:1, 1:1, 2:2, 3:2, 4:4, 5:4, 6:4, 7:1, 10:8, 11:8, 12:8}
-    FMT = {0:'<B',1:'<b',2:'<H',3:'<h',4:'<I',5:'<i',6:'<f',7:'<?',10:'<Q',11:'<q',12:'<d'}
-    keys = {}
-    for _ in range(nk):
-        k = s()
-        t = struct.unpack('<I', f.read(4))[0]
-        if t == 8:
-            v = s()
-        elif t in SCALAR:
-            v = struct.unpack(FMT[t], f.read(SCALAR[t]))[0]
-        else:
-            et = struct.unpack('<I', f.read(4))[0]     # element type FIRST
-            n = struct.unpack('<Q', f.read(8))[0]      # then the count
-            if et == 8:
-                v = ('arr_str', [s() for _ in range(n)])
-            elif et in SCALAR:
-                v = ('arr', et, n); f.read(n * SCALAR[et])
-            else:
-                raise Exception('unsupported arr elem type %d' % et)
-        keys[k] = v
-    return ver, keys
-
-for p in sys.argv[1:]:
-    ver, k = rd(p)
-    arch = k.get('general.architecture')
-    print('=====', p, 'v%d' % ver, 'arch:', arch)
-    for key in sorted(k):
-        if key.startswith('tokenizer.') or isinstance(k[key], tuple):
-            continue
-        if key.startswith(arch + '.') or not key.startswith(('general.', 'quantize.')):
-            print(' ', key, '=', k[key])
-```
-
-The three format traps, each of which cost a debugging session:
-
-1. **The KV stream has NO alignment.** GGUF v3's alignment (from `general.alignment`,
-   default 32) applies only to the *tensor data section* that follows. Do not pad
-   between KV pairs.
-2. **Array layout is `type(9) -> element_type(u32) -> count(u64)`.** Count-then-element
-   is wrong and yields a `MemoryError`/`UnicodeDecodeError` a few pairs later.
-3. **The type enum is not the order you expect**: `7=BOOL`, `9=ARRAY`, and the 64-bit
-   types are `10/11/12`. Getting `3=INT16` (2 bytes) or `5=INT32` (4 bytes) wrong shifts
-   the whole stream.
-
-Fields that matter: `*.block_count`, `*.block_size`, `*.attention.head_count_kv`,
-`*.attention.key_length`/`value_length`, `*.attention.sliding_window`,
-`*.context_length`.
-
-## 2. The drafter's KV is sized from the TARGET
-
-`common/speculative.cpp` builds the draft context with:
-
-```cpp
-// the draft context holds as many tokens per sequence as the target context
-cparams.n_ctx = llama_n_ctx(ctx_tgt);
-```
-
-There is **no flag** to cap it (the `--spec-draft-*` surface has none). Therefore:
-
-```
-draft KV bytes = ctx_tgt x block_count x 2 (K,V) x head_count_kv x key_length x dtype
-```
-
-*unless* the drafter declares a sliding window, in which case the window bounds it.
-
-Measured on the same 262144-token target: a sidecar with `sliding_window = 2048` cost
-~40 MiB of draft KV; a sidecar of the same 5-layer shape with **no** window cost
-**5.0 GiB** (20 KiB/position). Swapping between them invalidated an already-balanced
-layer split.
-
-**Failure signature:** the engine loads the whole model, answers `/health`, and dies on
-the FIRST draft decode with an allocation error (`ROCm error: out of memory` /
-`cudaMalloc failed`) inside `common_speculative_impl_draft_dflash::process`. An
-allocation failure *after* a successful load is a sizing failure, not a broken drafter
--- and under a self-healing unit it looks like a crash-loop instead.
-
-## 3. A bigger batch is asymmetric across the split
-
-`--ubatch-size` is the physical batch the graph is built at, so it sets the compute
-buffers. Those buffers are **~2x heavier on the card that holds the output layers and
-the drafter's graphs** than on the other(s). Measured, 2 GPUs at 262144 ctx:
-
-| ubatch | output/drafter card | other card |
-|---|---|---|
-| 512 -> 1536 | +~3.3 GiB | +~1.7 GiB |
-
-Consequence: **a split balanced at one batch is unbalanced at another.** A split at
-1:1 or a hand-measured skew must be re-measured after any change to `--ubatch-size`,
-`--ctx-size` or `--parallel`.
-
-## 4. Find the balance by measurement, not arithmetic
-
-The per-step cost is not derivable from the parameter count (KV lands only on the
-attention layers, and buffer placement is per-device). One `-ts` step is worth roughly
-1.5 GiB of load at ~6k batch / 262k ctx on 30 GiB-class cards — enough to bracket, not
-to solve. Procedure:
-
-1. Copy the env file to a scratch path and run the serve script against it
-   (`SERVICE_ENV_FILE=/tmp/test.env bash scripts/serve-<svc>.sh`) rather than editing
-   the deployed file. Do it with the unit stopped — one GPU, one process.
-2. Bracket: try a step, read `rocm-smi --showmeminfo vram` for both cards after a real
-   generation, and step again. Three points is usually enough to land within ~0.1 GiB.
-3. Prefer the balanced split, because the busier card's headroom is what bounds the
-   context:
-
-   ```
-   ctx_extra ~= (headroom_target - measured_free) / (per-position cost on the busier card)
-   ```
-
-   where per-position cost = the card's share of the target's KV + the drafter's KV if
-   that card carries it. A balanced split therefore MAXIMISES ctx for a given floor.
-4. Verify with the full battery before committing: `/health`, `/v1/models`, a real
-   completion, a tool call, two concurrent long prompts (one per slot), `slots`
-   reporting the expected `n_ctx` per slot, and the per-task `draft acceptance = ...`
-   line per slot.
-
-## 5. Keep the two failure signatures apart
-
-| Signature | Meaning |
-|---|---|
-| loads, `/health` ok, first decode OOMs | sizing: drafter KV, batch buffers, or KV vs ctx (section 2) |
-| `SDMA0 ... page fault` + `illegal memory access` + `ggml_abort` | a GPU fault in the cross-card boundary copy — the layer split's hot path. **Not** an OOM; headroom does not prevent it |
-
-The second one is why a rebalance should be validated by a request battery and not just
-by `rocm-smi` numbers.
-
-## Ops traps
-
-- A server started over ssh (including under a supervisor/`hub`-style runner) **outlives
-  stopping that runner** — the remote process keeps the port, and the next start fails
-  with `couldn't bind HTTP server socket`. `pkill -x <exact-binary-name>` first.
-- `systemctl start` of a root unit from a non-root ssh session fails with
-  `Access denied ... requires interactive authentication`: use sudo.
-- An old crash's `ROCm error` lines remain in `journalctl -u <svc>` forever. Check the
-  current instance's `ActiveEnterTimestamp`/`NRestarts` before treating a grep count as
-  a live fault.

agent/managed-skills/llamacpp-endpoint-model-swap/SKILL.md

@@ -0,0 +1,28 @@
+---
+name: llamacpp-endpoint-model-swap
+description: "Swap the model on a dank-fleet llama.cpp endpoint (frank-secondary pattern): fetch, env edit, spec flags, deploy through git, rewrite the bats seam, verify VRAM with the engine's own numbers."
+---
+
+# llama.cpp endpoint model swap (dank fleet)
+
+Use when replacing the model/shape on a llama.cpp endpoint (e.g. frank-secondary on the RTX 3090). The engine's boot numbers are the verdict — do NOT commit to a feasibility verdict from head math; boot it and read `nvidia-smi`.
+
+## Sequence
+
+1. **Geometry first, from the loaded model's own log** (`--log-verbosity 4`, or the HF `config.json` under `text_config` for multimodal): layers, `n_head_kv` (an ARRAY — per-layer, e.g. gemma4 has 8 on sliding layers, 2 on full-attention), head_dim, `sliding_window_pattern`, `n_swa`, `attention_k_eq_v`. KV scales only with full-attention layers; sliding layers cost a fixed window-size ring. Single-value HF fields lie about per-layer heterogeneity.
+2. **Fit the KV rate law to the engine, not to arithmetic.** Run once with `--log-verbosity 4` and read the `llama_kv_cache` / `sched_reserve` buffer MiB lines. Known law for master 5fc4f3c: bytes/elem = (K bytes + V bytes) **x2** when the file declares `attention_k_eq_v` (6.125 at f16 K + q8_0 V, 2.125 at q8_0/q8_0) — k_eq_v is a compute-graph mapping and does NOT shrink the cache. Full cache = n_full x n_head_kv x head_dim x rate x ctx; SWA cache = n_swa x window x n_head_kv_swa x head_dim x rate. Both reproduce the printed MiB exactly; if your formula does not, your law is wrong, not the engine.
+3. **Probe the binary's real capabilities** before promising flags: `--help | grep -A2 cache-type-k` (this fleet's build has NO fp8; q8_0 is the "FP8 class"), `--help | grep -A2 verbosity` (`-lv, --log-verbosity N`; ship `VERBOSITY` as an env tunable + `--log-verbosity "$VERBOSITY"` so VRAM forensics land in the journal), `--help | grep model-draft` for separate drafters. NEVER `grep -c <archname>` on the packed binary to check arch support — it returns 0 for every arch; check the source tree at the build's commit.
+4. **Fetch with `-f` + byte-count check** into `/home/dank/models/<name>/`.
+5. **Edit the repo trio** (`scripts/serve-<svc>.sh`, `scripts/<svc>.env`, `~/.omp/agent/models.yml`): spec flags move as a SET (`SPEC_TYPE=none` strips `--spec-type` AND `--spec-draft-model`/`--spec-draft-*`); a drafter with its own arch file (gemma4-assistant) needs `SPEC_DRAFT_FILE`, and its draft KV is built against the TARGET's full window — price it from a boot pair. `models.yml` `contextWindow` = per-slot window (`CTX_SIZE/PARALLEL`); recompute `maxTokens` for the compaction clamp `min(maxTokens, ctx - estimate - 4096)`.
+6. **Rewrite the bats seam in the same commit** (`tests/<svc>_unit.bats`): payload path, slot geometry, KV pair, spec state, batch pair, verbosity, mmproj opt-out, reasoning flags — all argv-asserted against the stub engine; where the contract is a repo value (batch pair, verbosity) assert argv == env value so a retune moves one file.
+7. **Deploy through git** (no direct remote edits). If the remote tree is dirty: `git stash push <files>` on the host, `git pull --ff-only`, land the change as a commit. A pull that aborts leaves the host at the OLD revision while the service runs the NEW edits — verify `git log -1` on the host.
+8. **Verify the engine, not your math**: `/proc/<pid>/cmdline` (argv is the contract), `/v1/models` (`n_ctx`, `ftype`, `n_ctx_train`), `nvidia-smi --query-gpu=memory.used,memory.free`, `/health` 200, a vision probe (solid color >=256 px via `image_url`, expect the exact word), and one generation probe reading `timings.predicted_per_second`. Record the numbers in the env comment + runbook, then push and ff-pull the host.
+9. **Cutover AGENTS.md in the same commit**: fleet-topology table row, the `### <svc>` section, the `Model Config` contextWindow/vision note, the runbook index ACTIVE row, a RETIRED row for the superseded era. The new docs page must be linked from the TOP-LEVEL `serverAppConfigs-docs/index.md`.
+
+## Gotchas
+
+- A systemd restart reporting `inactive` in the first seconds is usually model-load time; check `journalctl -u <svc>` for `model done`/`server is listening` before declaring failure.
+- Rollback = the previous payload stays on disk; one `MODEL_FILE` + shape edit back.
+- A thinking model can consume the whole token budget in `reasoning_content` (empty `content`, `finish_reason: length`) — probe with a budget that clears thinking.
+- **Bats vacuity trap**: `serve_argv` REWRITES `$STAGE/test.env` from the shipped env, so a test that mutates the file (`sed -i`, append) then calls `serve_argv` asserts on regenerated content — a tautology. Tests that mutate must exec via a `run_stage` helper running the file as-is. Symptom: the assertion fails only after you fix the vacuity, and the env dump shows the mutation gone.
+- `env NAME=value cmd` needs valid identifiers; a flag-like token (`nvidia-modprobe=true`) is only legal as an `env` ARGUMENT, not as a shell assignment prefix.

agent/managed-skills/llamacpp-layer-split-rdna4/SKILL.md

@@ -1,113 +0,0 @@
----
-name: llamacpp-layer-split-rdna4
-description: "Serve one model across two (or more) AMD RDNA4/ROCm GPUs on a host where RCCL cannot dispatch collectives — mainline llama.cpp built for gfx1201 with a bundled ROCm runtime, --split-mode layer, DFlash2/MTP speculative decoding, and the verification that proves each layer works"
----
-
-# llama.cpp layer-split serving on RDNA4 (gfx1201)
-
-Use when one model must use two AMD GPUs whose interconnect cannot carry a
-collective (no PCIe atomic routing on one card's path), or when a host has no
-usable RCCL. The shape: ONE llama-server process, `--split-mode layer` across
-both cards, spec decode from a drafter GGUF or the model's own MTP head.
-
-## When to reach for this
-
-- vLLM cannot help: TP needs a per-layer allreduce and **PP's stage handoff is a
-device-backend `send`/`recv`** (`vllm/distributed/parallel_state.py`,
-`send_tensor_dict` → tensor handles "on the device backend"), i.e. the same
-RCCL primitive that fails on an atomics-less card. Verify with
-`sudo lspci -vv -s <gpu> | grep AtomicOpsCtl` (want `ReqEn+`) and
-`lspci -tv` (a `Routing-` bridge is a hardware wall).
-- Confirming the wall: any RCCL kernel launch on that card returns
-`hipErrorIllegalState` with `AMD_LOG_LEVEL=3` printing
-`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch failed!`.
-  A single-card test proves nothing (NCCL short-circuits at world_size=1).
-- **Never run a multi-GPU collective probe beside a live service** on these
-  hosts: stop and mask it first, or use a host that serves nobody.
-
-## Build (pin the ref, bundle the runtime)
-
-```bash
-git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
-git checkout --detach <ref>          # pin; do not float
-# ROCm toolchain: system ROCm or a TheRock nightly tarball, e.g.
-#   https://rocm.nightlies.amd.com/tarball-multi-arch/therock-dist-linux-gfx120X-all-<ver>.tar.gz
-ROCM=$HOME/.local/share/rocm-<ver>
-cmake -S . -B build-rdna4 -G Ninja -DCMAKE_BUILD_TYPE=Release \
-  -DCMAKE_C_COMPILER=$ROCM/llvm/bin/clang -DCMAKE_CXX_COMPILER=$ROCM/llvm/bin/clang++ \
-  -DCMAKE_CXX_FLAGS="-I$ROCM/include" -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
-  -DGGML_HIP=ON -DGGML_HIP_RCCL=OFF -DGGML_CUDA=OFF -DGGML_VULKAN=OFF \
-  -DCMAKE_HIP_ARCHITECTURES=gfx1201 -DGPU_TARGETS=gfx1201 \
-  -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_WEBUI=OFF -DBUILD_SHARED_LIBS=OFF \
-  -DCMAKE_PREFIX_PATH=$ROCM -DROCM_DIR=$ROCM
-cmake --build build-rdna4 -j $(nproc) --target llama-server llama-cli llama-bench
-patchelf --set-rpath '$ORIGIN/../lib' build-rdna4/bin/llama-*
-```
-
-- `-DGGML_HIP_RCCL=OFF` is deliberate: with RCCL off, only `--split-mode tensor`
-  degrades (butterfly fallback); layer split is unaffected.
-- **Bundle device libraries at the exact depth the runtime looks for.** Copy
-  `$ROCM/lib/rocblas/library/` → `build/lib/rocblas/library/` and
-  `$ROCM/lib/hipblaslt/library/` → `build/lib/hipblaslt/library/` with
-  `mkdir -p DEST && cp -r SRC/. DEST/`. `cp -r SRC DEST/` with a missing DEST
-  flattens one level and the failure appears at REQUEST time, not load time
-  (`Cannot read .../lib/rocblas/library/TensileLibrary.dat`).
-- Build only the arch you own. A sibling-arch binary (gfx1200 on gfx1201)
-  links, loads the model, then segfaults.
-- Verify: `build-rdna4/bin/llama-server --version` and
-  `--list-devices` (expect both cards; an iGPU can enumerate as a third).
-
-## Serve
-
-```bash
-HIP_VISIBLE_DEVICES=0,1 llama-server \
-  --model <target.gguf> --alias <fleet-name> --host 0.0.0.0 --port 7770 \
-  --device ROCm0,ROCm1 --split-mode layer --tensor-split 1,1 \
-  --gpu-layers 999 --fit off --flash-attn on \
-  --ctx-size <n> --parallel 1 --cache-type-k f16 --cache-type-v f16 \
-  --jinja --metrics --cache-prompt \
-  --reasoning on --reasoning-format deepseek --reasoning-budget -1 \
-  --spec-type draft-dflash --spec-draft-model <dflash2-q8_0.gguf> --spec-draft-n-max 7
-```
-
-- `--fit off` + an explicit `--ctx-size`: auto-fit must not resize a context
-  you set on purpose. `--parallel N` DIVIDES `--ctx-size`; it does not multiply.
-- Spec decode choices: `--spec-type draft-dflash` with a DFlash2 GGUF (the
-  drafter's own metadata selects the DFlash2 path — the init line prints
-  `block_size=8, n_extract=5, sample_from_anchor=true`); `n_max <= block_size-1`.
-  Or `draft-mtp` for a model whose GGUF carries `blk.<n>.nextn.*` (no draft
-  file; the head loads from the main weights). Types can be chained
-  (`draft-mtp,draft-dflash`). With `draft-dflash`, the target's `nextn`
-  tensors log as "unused tensor ... ignoring" — that is correct, not a broken
-  download (check bytes with the fetch script's `--verify-only`).
-- Layer split puts the drafter on the card holding the target's last layers
-  (`--spec-draft-device ROCm1`); under tensor mode an unpinned draft joins the
-  meta-device KV allocation and aborts the KV build.
-
-## Verify (each layer separately)
-
-1. Load: `n_ctx_slot` in the log matches `--ctx-size`; `/health` answers.
-2. Both cards: `rocm-smi --showmeminfo vram` — each card holds roughly half the
-   weights; the busier one also carries the drafter. Compare against
-   `model_bytes/2 + drafter + ctx*64KiB/2 + ~2 GiB` (qwen35 hybrid: only its
-   full-attention layers take KV, 64 KiB/token at f16).
-3. **Spec decode is live**: `curl /metrics | grep spec_decode_num_drafts_total`
-   and `..._accepted_tokens_total` (their ratio is acceptance); a request's
-   `timings.draft_n` / `draft_n_accepted` gives it per call. Drafts stuck at 0
-   with tokens flowing means the drafter is not running.
-4. Real completion through the fleet alias, then a deep prompt (a 140k-token
-   prompt proves a raised window is real, not just declared).
-5. Prefix cache: repeat the same prompt — `timings.prompt_n` should collapse to
-   a handful of tokens.
-
-## Pitfalls that cost real time
-
-- A healthy-looking build that dies on the first request is almost always the
-  bundled-library depth above.
-- Serve scripts that `source` an env file AFTER defaults cannot be overridden
-  from the command line; override the env-file PATH instead.
-- Qwen3.8 chat templates RAISE on `reasoning_effort=high` (500). Probe
-  low/medium/high/xhigh before declaring an endpoint ready, and map the
-  harness's levels onto the supported set.
-- One slot at a large context is the concurrency limit: concurrent agent loops
-  queue, and first-turn TTFT then includes the wait.

agent/managed-skills/llamacpp-layer-split-vram-rebalance/SKILL.md

@@ -1,70 +0,0 @@
----
-name: llamacpp-layer-split-vram-rebalance
-description: "Re-measure and set the --tensor-split VRAM balance for a multi-GPU llama.cpp server when the batch, context, slots or spec drafter change, instead of interpolating between splits."
----
-
-Re-balancing is needed whenever a knob moves weight between cards: `--ubatch-size` (compute buffers), a spec drafter (weights + draft KV), `--ctx-size`/`--parallel` (KV), or a different drafter with a different window policy. A split balanced at one configuration is unbalanced at another — and the second card's asymmetry (output layers + drafter graphs) is what makes it non-obvious.
-
-## 0. Stop the service first
-
-Serving a user? Stop it, then free the port, before loading test instances. Do not bench beside a live server.
-
-```bash
-sudo systemctl stop <svc>        # root unit; for a rootless quadlet use systemctl --user
-pkill -x llama-server && sleep 3 # a manual/child run holds the port and the VRAM
-```
-
-`pkill -f <pattern>` is forbidden here: it matches your own command line and kills the ssh session. `mask` FAILS on a unit that is a symlink into a repo checkout (`File ... already exists and is a symlink to ...`); a `stop` is enough because nothing else restarts it.
-
-## 1. Measure one split per load
-
-The serve script is env-driven, so point it at a copy of the env file with `TENSOR_SPLIT` patched and read the allocation back from `rocm-smi`. Load -> sample -> kill, with NO requests sent: this touches allocation only, not the compute path.
-
-```bash
-WORK=/tmp/split.env; cp scripts/serve-<svc>.env $WORK
-export FRANK_MEDIUM_ENV_FILE=$WORK          # the script's own override hook
-for ts in 39,26 38,27 37,28 36,29 35,30; do
-  sed -i "s|^TENSOR_SPLIT=.*|TENSOR_SPLIT=$ts|" $WORK
-  nohup bash scripts/serve-<svc>.sh >/tmp/split.log 2>&1 &
-  for _ in $(seq 1 150); do
-    [ "$(curl -s -o /dev/null -m 2 -w '%{http_code}' http://127.0.0.1:PORT/health)" = 200 ] && break
-    pgrep -x llama-server >/dev/null || break
-    sleep 2
-  done
-  python3 -c "
-import subprocess,re
-CAP=34208743424  # one card, as rocm-smi reports it
-o=subprocess.run(['rocm-smi','--showmeminfo','vram'],capture_output=True,text=True).stdout
-u=[int(x) for x in re.findall(r'VRAM Total Used Memory \\(B\\): (\\d+)',o)][:2]
-print('$ts', f'{(CAP-u[0])/2**30:.2f}', f'{(CAP-u[1])/2**30:.2f}')"
-  pkill -x llama-server; sleep 6
-done
-```
-
-Sample ~6 s after `/health` returns 200 — the last allocations land after listening. Run it under `nohup` and poll the log from a separate call: a load is 30-90 s, which exceeds a single command's patience.
-
-## 2. Read the table, do not interpolate
-
-Pick the split with the smallest inter-card gap, then confirm it clears any per-card floor (e.g. 1.5 GiB) with room to spare. **A `-ts` step is ONE LAYER, and hybrid stacks do not have uniform layers.** Measured on qwen35 (16 x (3 Gated DeltaNet + 1 Gated Attention)): 0.43 GiB on a DeltaNet boundary, 1.43 GiB on a full-attention one (weights + that layer's own f16 KV). So the gap can jump 2.9 GiB in a single step, and the best integer split is ~0.2 GiB from balanced while its neighbours are 1+ GiB off. Interpolating between two measured splits produces a split nobody should deploy.
-
-Total free VRAM is invariant across splits (only its distribution moves) — use that to sanity-check a suspicious row.
-
-## 3. Check the drafter's window before blaming the split
-
-A draft context is built with `n_ctx = llama_n_ctx(ctx_tgt)` and there is no flag to cap it, so a drafter declaring NO sliding window caches the WHOLE target window on `--spec-draft-device`. DFlash2 (`sliding_window 2048`) = ~40 MiB; DSpark (no window) = 20 KiB/position = 5.0 GiB at 262144 — a four-layer swing in the balance point. Read `sliding_window` out of the drafter's GGUF header before sizing, and prefer a skewed `-ts` over cutting `--ctx-size`.
-
-## 4. If the repo has a fitted VRAM model, refit it — with the same unit convention
-
-Match the existing expression literally (e.g. `1442 * 1024 * ubatch` means ~1.4 MB per ubatch unit, not 1442 bytes). Fit so the deployed point reproduces `rocm-smi` to ~0.05 GiB, keep the physics (weights + proportional KV + a window-bounded draft KV + ubatch-scaled buffers), and treat a NEGATIVE fitted constant as a unit bug. State in the comment where the model is conservative: a uniform per-layer cost overstates gaps in a hybrid stack, and a fitted ubatch coefficient makes any larger-batch verdict an extrapolation, not a measurement.
-
-## 5. Verify behaviourally after deploy, not just by flags
-
-- `/slots` -> the expected `(id, n_ctx)` pairs.
-- The drafter's init lines from `journalctl -u <svc>`: implementation name, `n_max`, `block_size` — read the NUMBERS, since DFlash2 and DSpark log under the same `draft_dflash` symbol.
-- `rocm-smi` free VRAM at the deployed split (the fitted prediction and the real figure should agree to ~0.05 GiB).
-- One tool call and two CONCURRENT requests, one per slot, then `journalctl -o cat | grep 'slot print_timing' | grep -o 'slot print_timing: id *[01].*draft acceptance.*'` to prove drafting is live on BOTH slots (`journalctl` default output truncates these lines; `-o cat` plus `grep -o` gets them whole).
-- `grep -c 'ROCm error\|illegal memory\|ggml_abort'` on the current instance must be 0.
-
-## Faults to watch, not to tune away
-
-A cross-card boundary copy can fault (`amdgpu SDMA0 UTCL2 page fault` -> `ROCm error: an illegal memory access was encountered` -> `ggml_abort`) on a split that loads and answers `/health` perfectly. It is not an OOM — more headroom does not prevent it — so treat a split that has faulted as off-limits and re-measure rather than re-tuning. Also: a load that succeeds and then dies on the FIRST draft decode with an OOM is a sizing failure (the draft KV), not a broken drafter.

agent/managed-skills/llamacpp-mtp-vision-endpoint-bringup/SKILL.md

@@ -1,132 +0,0 @@
----
-name: llamacpp-mtp-vision-endpoint-bringup
-description: "Bring up a new llama.cpp (mainline) OpenAI endpoint on a single CUDA GPU with MTP speculative decoding and vision — GGUF header preflight by HTTP range, hybrid-architecture KV arithmetic, the -mtp / --spec-type draft-mtp contract, --parallel vs ctx math, and the verification battery that proves MTP, tools and vision actually work."
----
-
-# llama.cpp MTP + vision endpoint bring-up (single CUDA GPU)
-
-Use when standing up or swapping a mainline llama.cpp endpoint for a GGUF that
-may ship an MTP (NextN) head and a vision projector. Verified end to end on
-Dankclaw 2026-09-16 (RTX 3090 24 GiB, Qwen3.8-27B-GSQ-RCO IQ3_S `-mtp` +
-BF16 mmproj, 131072 ctx / 1 slot, port 7771).
-
-## 1. Preflight the GGUF WITHOUT downloading it
-
-GGUF metadata lives in the header, and HuggingFace honours ranged GETs, so a
-few MB answers every sizing question before a 13 GB transfer:
-
-```python
-import urllib.request
-req = urllib.request.Request(url, headers={"Range": "bytes=0-8000000", "User-Agent": "curl/8"})
-buf = urllib.request.urlopen(req, timeout=180).read()
-```
-
-Parse: magic `GGUF`, u32 version, u64 n_tensors, u64 n_kv, then key/value pairs
-(string = u64 len + bytes; ARRAY = u32 elem type + u64 count; BOOL is 1 byte —
-the type hand-written parsers forget). Walk arrays by byte size even when you
-only want a summary: a naive parser desynchronises on `tokenizer.ggml.tokens`
-and lands gigabytes off.
-
-Read these keys: `general.architecture`, `<arch>.block_count`,
-`<arch>.context_length`, `<arch>.nextn_predict_layers`, `<arch>.head_count_kv`,
-`<arch>.key_length` / `value_length`, `<arch>.full_attention_interval`,
-`<arch>.ssm.state_size` / `group_count` / `time_step_rank`, and for the
-projector file `clip.projector_type` (e.g. `qwen3vl_merger`) plus
-`clip.has_vision_encoder`.
-
-## 2. Decide the file from `nextn_predict_layers`
-
-**MTP is a property of the FILE, not of the flags.** If the release ships an
-`-mtp` build, that build carries the head as `blk.<n>.nextn.*` (a few hundred
-MB larger); the plain build has no MTP tensors at all.
-
-With the `-mtp` file, llama.cpp needs only `--spec-type draft-mtp`: it sets
-`load_mtp` and the draft context runs on the MAIN model's weights
-(`shares_model`, `LLAMA_CONTEXT_TYPE_MTP`). Do NOT add `--model-draft` — that
-loads the whole file a second time (12 GiB twice on a 24 GiB card). A separate
-`-md` is only correct for standalone MTP-only sidecars.
-
-Confirm the flag exists in the deployed build before writing the script:
-`grep -rn "common_speculative_all_types_str" common/speculative.cpp` and
-`llama-server --help | grep -A4 spec-type`.
-
-## 3. KV arithmetic (do this before choosing ctx)
-
-Only **full-attention** layers take KV. For a hybrid arch (Qwen3-Next/3.8 style
-`16 x (3 x Gated DeltaNet + 1 x Gated Attention)`):
-
-```
-attn_layers = block_count / full_attention_interval
-bytes/token = attn_layers x n_head_kv x head_dim x 2 (K,V) x bytes_per_elem
-              f16 = 2 B | q8_0 = 1.0625 B | q4_0 = 0.5625 B
-```
-
-The linear-attention layers hold constant-size per-sequence state instead
-(~0.16 GiB per sequence at 48 layers / 128 d_state / 48 v-heads in f32), and
-the MTP draft context costs its own `n_ctx x (1 layer x n_head_kv x head_dim x
-2 x 2 B)` ≈ 4 KiB/token for a 1-layer head.
-
-VRAM budget: weights + mmproj + MTP draft KV + DeltaNet state + ~1.5 GiB of
-CUDA context and compute buffers. On a 24 GiB card that leaves ~9.5 GiB for KV,
-which at 64 KiB/token is ~155k tokens total.
-
-**`--parallel N` DIVIDES `--ctx-size` between slots** — it does not multiply.
-`-np 1 -c 131072` is one 131k conversation; `-np 4 -c 131072` is four 32k ones.
-
-## 4. Flags that matter
-
-- `-ngl 999`, `-fa on`, `-ctk/-ctv f16` for the usual shape.
-- Quantized KV (`q8_0`, `q4_0`) requires a build with
-  `-DGGML_CUDA_FA_ALL_QUANTS=ON`; without it CUDA prefill collapses ~12x. f16
-  needs nothing, so prefer it unless the context demand is real.
-- `--jinja` — pin it explicitly even when the build defaults it on: the GGUF's
-own template supplies `reasoning_effort`, `preserve_thinking` and the
-tool-call grammar, and a future default flip must not silently remove tools.
-- `--metrics` if anything scrapes `/metrics`, `--alias` to set the served id,
-  `--temp 0.0` for a greedy-by-default fleet contract,
-  `--reasoning-format deepseek --reasoning-budget -1` to project thinking into
-  `reasoning_content`.
-
-## 5. Serve-script and unit invariants
-
-Fail loudly before loading VRAM: check the binary, the model file and the
-mmproj exist, and `exit 1` with a named message otherwise — a missing weight
-must not become a `Restart=on-failure` crash loop that reads like a runtime
-fault. For a native CUDA binary the unit is a ROOT systemd unit
-(`[Install] WantedBy=multi-user.target`, `Restart=on-failure`,
-`RestartSec=10`, `LimitMEMLOCK=infinity`); a container would be a Quadlet.
-
-## 6. Verification battery (prove each claim, do not infer)
-
-1. `/health` 200, and `/v1/models` reports the intended alias.
-2. A chat completion returns `content` AND `reasoning_content`.
-3. A request carrying `tools` returns `tool_calls` — a RUNTIME property: the
-   ik_llama.cpp runtime answered `500 tools param requires --jinja flag`.
-4. `GET /metrics` after a generation: `llamacpp:spec_decode_num_draft_tokens_total`
-   and `..._accepted_tokens_total` both > 0, acceptance 0.6-0.95 ⇒ MTP engaged,
-   not merely configured. The slot timing lines in the journal say the same.
-5. One image request (`data:` URL or bare base64) ⇒ an answer that reflects the
-   image; confirm the journal printed a multimodal projector load.
-6. `nvidia-smi` memory used is close to the modelled total, and `n_slots` /
-   `n_ctx_slot` in the load log match the intended shape.
-
-## Traps that cost real time
-
-- **A template can reject the client's thinking level.** Qwen3.8 accepts only
-  `xhigh|medium|low` for `reasoning_effort` and raises otherwise — HTTP 500
-  naming Jinja, not the model. Map every harness level onto the accepted set
-  (e.g. `compat.reasoningEffortMap: {minimal: low, low: low, medium: medium,
-  high: xhigh, max: xhigh}`) and probe each level with curl before routing
-  roles at the endpoint.
-- **Thinking counts against `max_tokens`.** A budget that runs out mid-thought
-  returns `finish_reason: "length"` with empty `content` — not a broken
-  endpoint, not a vision failure. Raise the budget.
-- **Never point `--model-draft` at the `-mtp` file** (step 2).
-- **Any `-ot` / `-ncmoe`-style override can silently disable mmap** for the
-  whole load, turning a mapped model into anonymous memory.
-- On a host that publishes ports through **pasta**, the replacement must not
-  start until the predecessor released the port: poll
-  `ss -tan "sport = :<port>"` (listener, accepted AND TIME_WAIT) or the
-  container dies with `Address already in use`.
-- On a shared host, read the serving engine's own boot ledger for device
-  budget vs need: a pool that silently clamped still answers a green `/health`.

agent/managed-skills/llamacpp-multigpu-layer-split-cutover/SKILL.md

@@ -1,88 +0,0 @@
----
-name: llamacpp-multigpu-layer-split-cutover
-description: "Cut a fleet LLM endpoint over to llama.cpp layer-split across multiple GPUs when cross-GPU collectives are impossible (RCCL/PCIe-atomics wall), including the build recipe, the bundling trap that fails on the first request, DFlash2/MTP wiring, and the measurement battery that proves the split and the drafter are real."
----
-
-# llama.cpp layer-split cutover (no-collective multi-GPU)
-
-Use when one model must serve from TWO+ cards and the box cannot dispatch a
-cross-card collective (second card behind a chipset bridge with
-`AtomicOpsCap: Routing-`, so ROCr refuses RCCL's hostcall-buffer kernels).
-Decide with one line before designing anything:
-
-```bash
-sudo lspci -vv -s <card2> | grep AtomicOpsCtl      # ReqEn- ⇒ no collectives, ever
-```
-
-`ReqEn-` kills vLLM entirely — tensor parallelism needs an allreduce **and
-pipeline parallelism needs an RCCL `send`/`recv` of the hidden state**
-(`vllm/distributed/parallel_state.py`: metadata on gloo, tensors on the device
-backend). llama.cpp's `--split-mode layer` needs no collective: layers are
-distributed in order, only the boundary activation crosses PCIe once per
-forward pass.
-
-## 1. Pin the runtime by FEATURE, not by "latest"
-
-Write down which features the deployment needs, then find the ref that has all
-of them and pin the 40-hex sha in the build script. On 2026-09-16 that was
-DFlash2 (`ggml-org/llama.cpp#27342`, detected from the draft GGUF:
-`selector_top_k > 0`, spec type stays `draft-dflash`) and qwen35 MTP. The
-ROCmFPX/r9700-toolbox images/fork tips predate both — verify by listing the
-built image:
-
-```bash
-ls <venv>/lib/python*/site-packages/vllm/v1/worker/gpu/spec_decode/   # dflash/ vs dflash2/
-```
-
-## 2. Build self-contained for one arch
-
-- Private ROCm toolchain (TheRock nightly) with `ROCM_PATH`/`HIP_PATH` exported,
-  `-DCMAKE_HIP_ARCHITECTURES=<gfx only>` (a sibling arch links, loads and then
-  segfaults), `-DGGML_HIP_RCCL=OFF` when there is no atomics path, `-DGGML_NATIVE=OFF`.
-- Bundle the runtime into `build/lib` and set `--set-rpath '$ORIGIN/../lib'`.
-- **TRAP — `cp -r SRC DST/` flattens when `DST` does not exist.** rocBLAS and
-  hipBLASLt look for device code at a FIXED depth (`../lib/rocblas/library`,
-  `../lib/hipblaslt/library/<arch>`). Flattened, the engine loads every GiB of
-  weights, answers `/health`, and dies on the **first request** with
-  `Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch`.
-  `mkdir -p DST && cp -r SRC/. DST/`.
-
-## 3. Deploy as an env + serve script + unit trio
-
-One env file holds every tunable with the sizing arithmetic in comments; the
-serve script defaults each var, refuses to start when the runtime or payload is
-missing, and builds an args array (spec flags assembled conditionally so the
-same script serves `draft-dflash` with a file and `draft-mtp` in-model). Root
-systemd unit: `User=<deploy user>`, `LimitMEMLOCK=infinity`,
-`Restart=on-failure`, `WantedBy=multi-user.target`, plus an
-`Environment=…_ENV_FILE=<repo path>` so the repo file is the single source.
-
-Deploy handler must: refuse when runtime/payload is missing; retire the previous
-runtime's units (into trash, never `rm`); **wait for the port to be released**
-(a retired reverse proxy or container holds it); then enable/restart and gate on
-`/health`.
-
-DRLaf: `--split-mode layer --device ROCm0,ROCm1 --tensor-split 1,1`,
-`--parallel 1`, f16 KV, `--fit off`. Pin the drafter with `--spec-draft-device`
-to the card holding the target's last layers. `--spec-draft-n-max` must be
-<= the draft's `block_size - 1` (DFlash2: 8 ⇒ 7).
-
-## 4. Prove it, do not assert it
-
-- **The split is real**: raise `--ctx-size` by N tokens and watch `rocm-smi` —
-  a layer split adds ~N·(bytes/token)/2 to EACH card (both cards carry the KV
-  for their own attention layers), and neither card's TOTAL can equal the model
-  file, which is the arithmetic that rules out replication.
-- **The drafter is live**: `timings.draft_n`/`draft_n_accepted` per request, and
-  the cumulative `llamacpp:spec_decode_num_drafts_total` — the flags can be
-  right while the draft model silently failed to load. The loader's init line
-  (`block_size=`, `n_extract=`, `mask_token_id=`) identifies the drafter family.
-- **Watch for a 500 on `reasoning_effort`**: Qwen3.8-family chat templates
-  accept only low/medium/xhigh and RAISE on `high`; map every harness level or
-  subagents 500 on their first request.
-- **Re-point the monitors in the same change**: a runtime swap changes the
-  metric namespace (`vllm:` → `llamacpp:`), and rules/dashboard panels naming
-  series that no longer exist draw nothing forever without erroring. Retire
-  what cannot be expressed (llama.cpp exports no KV-fill gauge and no latency
-  histogram) and name the consequence instead (queue depth, spec-decode-off).
-- One slot is a concurrency policy, not a bug: `--parallel` DIVIDES the context.

agent/managed-skills/llamacpp-serve-unit-boot-loop/SKILL.md

@@ -0,0 +1,27 @@
+---
+name: llamacpp-serve-unit-boot-loop
+description: "Diagnose a Dank-fleet LLM systemd unit that boot-loops: classify 203/EXEC vs model-load OOM vs payload gate, using journal signatures and nvidia-smi/rocm-smi."
+---
+
+# llama.cpp / fleet serve unit boot loop
+
+A fleet LLM unit (`frank-large`, `frank-secondary`, `frank-primary`) restarts every `RestartSec`. Read the journal FIRST, act second:
+
+```
+ssh <host> 'sudo systemctl status <svc> --no-pager | head -12; sudo journalctl -u <svc> --since "-6 min" -o cat --no-pager | tail -40'
+```
+
+## Classify by signature
+
+| Signature | Cause | Fix |
+|---|---|---|
+| `status=203/EXEC` + `Failed at step EXEC spawning` + `Mem peak: ~1.9M` + GPU flat at 1 MiB | the `ExecStart` script never ran: missing shebang, lost exec bit, or path moved by the deploy (git pull mid-restart) | `head -1 <script>` must be `#!/usr/bin/env bash`; `git ls-files -s <script>` must show `100755`; commit the fix, redeploy |
+| `status=203/EXEC` right after a deploy, then self-heals | git replaced the file mid-checkout while the restart loop ran | ignore if the newest attempts succeed; verify `podman ps --format '{{.CreatedAt}}'` identity |
+| Engine prints weights loading, then dies | VRAM/disk budget, engine flags | read the engine's own ledger (verbosity 4) for KV/compute-buffer MiB; price from two boots, never a formula |
+| Health 503, no error, minutes of silence | normal cold load (big PLE/mmap) | do not kill; the listener binds after the full load |
+
+## Rules
+- Judge the newest container/attempt only: `podman ps -a --format '{{.ID}} {{.CreatedAt}}'`, `journalctl --since`; a mid-stop container still serves old logs.
+- `systemctl is-active` is `failed` after every intentional stop on foreground-podman units (exit 143) — cosmetic.
+- The unit's `Description` is the last-deployed text, not the repo's. A stale Description is not an incident; the next `relink_unit` fixes it — never restart a serving endpoint for cosmetics.
+- Fixing the script = commit + `./scripts/deploy.sh <host> --service <svc>`; never edit on the host (a direct `scp` leaves drift that blocks the next deploy's ff-only pull with "local changes would be overwritten"; repair with `git checkout -- <file>`).

agent/managed-skills/llamacpp-vram-accounting/SKILL.md

@@ -1,77 +0,0 @@
----
-name: llamacpp-vram-accounting
-description: "Produce and interpret a per-device VRAM/memory breakdown for a llama.cpp or halogen LLM deployment on the Dank fleet: we/weights vs KV vs compute pool, discrete vs UMA accounting, and the measurement traps that make the numbers lie."
----
-
-# llama.cpp / halogen VRAM accounting (Dank fleet)
-
-Use when asked *where the memory went* for an LLM service on Dankclaw/Dankserver: "what's the VRAM breakdown", "why does it not fit", "how much headroom is left for the co-tenants", or before sizing a card for a new model.
-
-## 0. Decide which kind of memory you are talking about
-
-| device | kind | read it with |
-|---|---|---|
-| RTX 3090 (Dankclaw) | discrete GDDR6X | `nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv,noheader` |
-| gfx1151 iGPU (Dankclaw) | **UMA** — "VRAM" is host RAM | `/sys/class/drm/card1/device/mem_info_gtt_{total,used}` (real, 114 GiB) vs `mem_info_vram_{total,used}` (2 GiB carve-out window) |
-| R9700s (Dankserver) | discrete | rocm-smi figure is authoritative (64 GiB claimed, SVM-backed) |
-
-Per-process truth on NVIDIA: `nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader`.
-
-**Never** add the iGPU's GTT to host RAM as if they were separate pools, and never quote a UMA engine's total as "VRAM": halogens's 95.4 GiB is 68 GiB of **pinned host pages** (mlock + page cache) plus ~33.7 GiB inside the device budget. `free -g` is the arbiter for what the co-tenants have left.
-
-## 1. Get the engine's own ledger first
-
-```bash
-podman logs frank-large 2>&1 | grep -aiE "weights pinned|kv pool|left for everything|overlay|vision|budget"
-```
-
-Halogen prints the numbers that matter: weights pinned (and how many contiguous 2 MiB blocks remain), the device budget (`host RAM − resident weights − reserve`), the KV pool size and pool-vs-working split, and the final "left for everything else" block count. Contiguous-block count is the honest fragmentation signal — 240 blocks / 12.3 GiB after a reboot vs 475 / 14.7 GiB before means the same model did not come back into the same hole.
-
-## 2. For llama.cpp: the print sites exist, but no artifact here has them
-
-- `src/llama-model.cpp:1827` — per-backend `model buffer size` (unconditional `LLAMA_LOG_INFO`)
-- `src/llama-context.cpp:692` — `compute buffer size`
-- plus a KV self-size line at context creation
-
-Traps verified on 2026-09-16:
-
-- **`llama-bench` suppresses the loader's INFO** — its logs contain only the device banner and the results table, so a bench matrix leaves *no* memory accounting behind.
-- **`llama-server` under systemd did not put them in the journal either** — `sudo journalctl -u frank-small | grep -c "model buffer size"` returned **0** for the unit's entire history. Check with `grep -c` before claiming a log has the breakdown.
-- Recipe that works: launch the model **once** through `llama-cli` (or `llama-server`) and keep the first ~200 lines; add `-v` if the level filters them.
-
-## 3. Otherwise compose it — arithmetic on measured tensor sizes
-
-Read the GGUF's tensor table once (names + shapes are enough) and group by role, e.g. Flash-Next (111 GiB):
-
-| group | size | per layer |
-|---|---|---|
-| `ffn_{gate,up,down}_exps` | 75.0 GiB | **1.5625 GiB** (experts travel with their layer) |
-| `per_layer_token_embd` (PLE) | 32.8 GiB | one table |
-| everything else (attn/GDN/etc) | 3.2 GiB | 0.067 GiB |
-
-And KV ≈ 2 KiB per position per f16 attention layer (12 attention layers on Flash-Next => ~24 KiB/position; halogen's ledger says ~28 KiB/position including block scratch). Multiply by the *derived* context (see §5).
-
-Then take the measured per-device total as the constraint and let the remainder be compute/graph scratch. Vendor delta is real and sizeable: same 7 layers + KV on the 3090 came to **12.80 GiB under Vulkan** and **14.99 GiB under CUDA** — ~3.3 GiB vs ~1.1 GiB of graph/compute pool.
-
-## 4. Interpretation traps
-
-- **mmap PLE is reclaimable; `-ot` makes it not.** With everything on the iGPU the 32.8 GiB PLE sits in page cache (reclaimable). In a layer split, where `-ot` is disallowed, it becomes a pinned device buffer in GTT. "Move layers to the card" therefore converts reclaimable cache into pinned device memory and does not free room for co-tenants.
-- **A served model's footprint is not its weight size.** frank-small: 22,582–22,592 MiB of 24,576 for an 11.29 GiB model — weights + f16 KV at 131,072 × 1 slot + BF16 projector + CUDA compute pool.
-- **"Usable" is total minus driver/container overhead** (~24.1 GiB on a 24,576 MiB card).
-- Do the capacity subtraction *before* designing: 12 attention layers × 1.62 GiB = 19.4 GiB, plus 200k f16 KV (9.2 GiB) = 28.6 GiB > 24.1 — which is why a KV-on-VRAM design on this pair is only feasible to ~64k context.
-
-## 5. Context lengths: derive, never remember
-
-`llama-bench` never logs the context; it computes (`tools/llama-bench/llama-bench.cpp:1293`):
-
-```c
-cparams.n_ctx = n_prompt + n_gen + n_depth;
-```
-
-Defaults: `n_prompt 512`, `n_gen 128`, `n_depth 0`, `n_batch 2048`, `n_ubatch 512`, `type_k`/`type_v` **F16**, `flash_attn AUTO`, `split_mode LAYER`, `reps 5`.
-
-So `-p 512,16384 -n 64` = contexts **576 / 16,448 / 64**, and `-n 32 -d 65536` = **66,080 / 65,568**. A `tg` row runs with `n_prompt = 0`, which is why a plain `tg` number is ~zero depth: `-d` is the only thing that puts KV in the picture. Cross-check by reading the table's own test column (`pp512 @ d65536`, `tg32 @ d65536`).
-
-## 6. Reporting shape that survives review
-
-Label every number: **measured** (nvidia-smi / sysfs / engine ledger line), **derived** (arithmetic on tensor sizes + derived context), or **inference** (e.g. depth extrapolation). State the context each memory figure was taken at, and whether KV was resident — 12.8 GiB at 65,568 tokens is not the same claim as 12.8 GiB at 200k.

agent/managed-skills/llm-bench-noise-discipline/SKILL.md

@@ -0,0 +1,27 @@
+---
+name: llm-bench-noise-discipline
+description: "Noise-floor discipline for single-run LLM endpoint A/B benchmarks: the ±3% law, replication-before-keeping, and bracket-confirmation of tuned values."
+---
+
+# LLM Bench Noise Discipline
+
+Single-run A/B benchmarks against an LLM endpoint's own timings API have a much larger run-to-run noise floor than the metric's instantaneous precision suggests. On frank-primary (radiance, 256-token greedy decode at 43.6k-token prompt depth), three runs of the IDENTICAL config measured 33.06 / 32.03 / 30.90 t/s — a ~3% spread.
+
+## The ±3% law
+- Treat any single-run delta under ~3% as noise, regardless of the harness's reported confidence multiplier.
+- Before believing a "keep", run the same config a second time (no edit, pure replication). If the replication drops below the noise band, flag the lucky draw as suspect (`flag_runs`) so best-metric math stops trusting it.
+- Noise sources: TP all-reduce wire loss (wht6), page-cache warm/cold of the 51 GiB n-gram table, container boot ordering, expert-pool hit-rate warm-up.
+
+## Bracket before trusting an optimum
+A "tuned" value is only confirmed when BOTH neighboring steps are measured: one side may crash (boot wall) and the other may regress. Example at seqs=3: EXPERT_VS_CACHE 0.78 crash / 0.82 optimum / 0.84 = -9% (0.87 collapses to 0% per runbook); MNBT 4096 crash at seqs=3 while the seqs=2 A/B showed it bootable.
+
+## Boot failures are data, not waste
+A scratch arm that never reaches /health (720s timeout) maps the viable envelope. Three crashes at seqs=3 (0.78 weights-share, 4096-token chunks, bf16 wire) prove the production trio wht6/2048/0.82 is exactly what makes the 3-slot shape fit.
+
+## Outside-noise signals found at this endpoint
+- SPEC_TOKENS=4: -23% decode, draft acceptance 0.831->0.497 (the 4th draft token is rejected too often to pay its verification).
+- SPEC_TOKENS=0: -6.8%; MTP drafting always pays.
+- KV shape seqs=4/maxlen=100k (same 400k total): -11%; slot overhead is not paid by freed VRAM.
+
+## Harness pattern that worked
+stop prod -> boot scratch container on the SAME two GPUs with candidate argv -> poll /health -> ONE self-contained greedy request built from a fixed-seed repo corpus (deterministic prompt_n) -> parse timings -> tear down -> restore prod via trap (never mask a relinked unit; stop only). Fresh prefix-cache dir per run so the cold prefill is cold.

agent/managed-skills/llm-fleet-model-swap/SKILL.md

@@ -1,41 +0,0 @@
----
-name: llm-fleet-model-swap
-description: "Swap the model/runtime behind frank-default-large/small on Dankclaw/Dankserver via the git flow, with pre-deploy live validation and the port-holder health-check trap check."
----
-
-# LLM fleet model swap (frank-default-large / frank-default-small)
-
-Use when switching model, quant, or llama.cpp/FreeToken runtime for a `frank-default-*` service in `~/Projects/serverAppConfigs`.
-
-## 0. Ground truth first
-- Read the HF model card (runtime pin, hard constraints, recommended args) + repo files `scripts/frank-default-<size>.env`, `scripts/serve-frank-<size>.sh`, `scripts/frank-default-<size>.service`, and the runbook under `serverAppConfigs-docs/LLM/`.
-- Check the host before downloading anything: `ls /home/dank/models/`, `df -h /home`, `du -sh` — bench-era or prior-deployment files often already exist.
-
-## 1. Artifact integrity
-- List repo files: `curl -s https://huggingface.co/api/models/<repo>/tree/main?recursive=true` → `.lfs.oid` holds the sha256.
-- On-host file: compare size AND `sha256sum` against the LFS oid before trusting any existing copy (hash runs ~few min for >100 GiB; run it in background).
-- New download on Dankserver: `/home/dank/freetoken/.venv/bin/hf download <repo> --local-dir ...`.
-
-## 2. Runtime (if the card pins a fork/build)
-- Clone + build with the card's exact cmake line, **plus `-DLLAMA_BUILD_WEBUI=OFF`** — WebUI asset provisioning (npm / HF llama-ui bucket) aborts llama.cpp link at 98% on these hosts; API server unaffected.
-- Verify every flag our serve script passes exists in the new binary before touching the service:
-  `llama-server --help | grep -q -- "<flag>"` for each; list MISSes.
-
-## 3. Config via git only
-- Edit `scripts/frank-default-<size>.env` (LLAMA_BIN, MODEL_DIR, MODEL_FILE, MMPROJ_PATH) + serve script header/defaults; `bash -n` the script; `./tests/run.sh` must pass; commit; push; `git pull --ff-only` on the target.
-- Do NOT edit configs on the remote. Renaming a model dir (`mv`) is host state, allowed.
-
-## 4. Live validation (no scratch ports — GPU/RAM can't host two big models)
-- Ensure the old unit is truly gone: `systemctl is-active` may hang in `stop-sigterm` because llama.cpp never preempts an in-flight generation; poll until the PID exits before launching.
-- Stop service, run the real serve script manually (`setsid nohup bash scripts/serve-frank-<size>.sh > /tmp/x.log 2>&1 &` — plain `ssh 'nohup ... &'` dies with its ssh when foregrounded; keep ssh under timeout or it gets killed mid-stop).
-- Verify: health 200; a real `/v1/chat/completions` (FreeToken REQUIRES explicit `"model"` field; llama.cpp servers don't care); reasoning flows (`reasoning_content` non-empty; content may be empty with finish=length if the thinking eats the budget).
-
-## 5. systemd handoff + THE TRAP
-- `./scripts/deploy.sh <host> --service <svc>` — but a leftover manual server holding :7770 makes the health check pass while the UNIT crash-loops on bind.
-- Always confirm ownership: `systemctl show <svc> -p MainPID` equals the single `pgrep -x llama-server` (or the ft serve PID); only then trust "healthy after Ns".
-- If a real long-context task is queued/running, do the handoff anyway (client aborts are recoverable; drift is not) — or defer cosmetic-only redeploys.
-
-## 6. Close-out (mandatory)
-- Docs: active runbook updated (build recipe, profile, validation numbers incl. real-workload prefill/decode), superseded runbook gets RETIRED banner kept as rollback record, `LLM/index.md` mermaid + active list, AGENTS.md fleet table + Runtime section + history chain, work log entry in `serverAppConfigs-docs/log.md`.
-- `~/.omp/agent/models.yml`: contextWindow must equal the REAL per-slot context — llama.cpp logs `n_ctx_slot` per prompt; parallel N clamps `n_ctx_train` (e.g. 262144/2 = 131072/slot). Overselling causes silent client-side overflow.
-- Rollback = one env commit (paths + LLAMA_BIN); keep prior binary + payload on disk.

agent/managed-skills/llm-model-swap-fit-preflight/SKILL.md

@@ -1,50 +0,0 @@
----
-name: llm-model-swap-fit-preflight
-description: "Preflight an LLM model swap for hardware fit before deploying; catch the silent swap-thrash outage mode (large RAM OOM, self-restarting + boot-enabled units) and recover."
----
-
-# LLM Model-Swap Fit Preflight
-
-Never deploy a model swap without confirming the checkpoint FITS, and know the
-failure mode when it doesn't: it is a silent, self-sustaining thrash, not a crash.
-
-## Preflight (before deploy)
-
-1. Get the checkpoint's on-disk/weight size AND precision from the HF card
-   (`Precision: BF16` = 2 bytes/param; FP8/Q8 ≈ 1 byte; Q4 ≈ 0.5 byte).
-   35B params: BF16 ≈ 66–70 GB, FP8/Q8 ≈ 33–35 GB, Q4 ≈ 17–20 GB.
-2. Compare against the HOST'S USABLE RAM (not just VRAM) for runtimes that
-   stream weights from RAM (FreeToken host-RAM experts, llama.cpp CPU offload,
-   PP/worker splits).
-3. Budget RAM for weights + KV pool + OS + page cache. If weights alone exceed
-   RAM, ABORT — pool tuning shrinks KV, it does not shrink weights.
-
-## The self-sustaining thrash (why 'it'll just OOM and recover' is wrong)
-
-- Large swap (tens of GB) → kernel PAGES instead of OOM-killing → sshd/user
-  space starved indefinitely; no recovery window opens by itself.
-- systemd `Restart=on-failure|always` → crash-reload-rethrash loop.
-- Unit `enabled` + `WantedBy=multi-user.target` → a reboot re-thrashes on boot
-  until the bad config is masked/reverted. A plain power-cycle does NOT fix it.
-
-## Recovery (needs a shell; may need console / SysRq-to-GRUB)
-
-```
-sudo systemctl mask --now <svc>        # stop + break the reboot trap FIRST
-cd <checkout> && git fetch origin && git reset --hard origin/main   # pick up revert
-grep MODEL_DIR <env>                   # confirm fixed target
-sudo systemctl unmask <svc> && sudo systemctl restart <svc>
-curl -s <health>                        # then verify with a REAL completion
-```
-
-## Deploy-script caveat
-
-`deploy.sh`/health-gate 'healthy after Ns' = systemd ACTIVE, not model LOADED.
-Always confirm load with a real completion (curl /v1/chat/completions), not
-status alone.
-
-## GPU quant note
-
-RTX 3090 (Ampere) has no FP8/FP4 tensor cores: FP8 weights run via upcast but
-still must fit in RAM/VRAM. Choose quant by SIZE fit, then accept the compute
-path the card supports.

agent/managed-skills/llm-replica-routing-prefix-cache/SKILL.md

@@ -1,100 +0,0 @@
----
-name: llm-replica-routing-prefix-cache
-description: "Route requests across multiple LLM replicas by workload shape (pin agent-loop continuations for prefix-cache hits, spread batches for parallel prefill), with the measurement traps and the capacity wall that no routing policy fixes."
----
-
-# Prefix-cache-aware routing for multi-replica LLM serving
-
-Use when several single-GPU inference replicas sit behind one endpoint and
-clients send long shared prefixes (agent loops, chat with a big system prompt,
-RAG with a fixed context).
-
-## The two failure modes, both measurable
-
-Measure with a **sequential-turn client**, not only a batch benchmark: a batch
-of distinct prompts parallelises prefill across replicas, so round-robin looks
-great there and hides the problem.
-
-1. **Round-robin loses every other cache hit.** The loop resends the same
-   prefix each turn; alternating replicas means half the turns prefill from
-   scratch. Measured: 90k-token contexts, warm-turn TTFT 2676 ms round-robin
-   vs 1096 ms pinned.
-2. **Pinning everything loses a card.** A cold batch pays one card's prefill
-   throughput: 16 requests in 18.6 s spread vs 30.1 s pinned.
-
-## The rule that gets both
-
-Route by request shape:
-
-- **Continuation** (request carries more than one message — the signature of an
-  agent loop re-sending its context) -> pin to the replica that served that
-  prefix before.
-- **Anything else** (one-shot question, batch member, parallel subagent) ->
-  spread across replicas with a **counter**, not with a hash of the prefix.
-
-Measured on the same host: 19.97 s burst *and* 1095 ms warm turns.
-
-Also add **failover**: a replica that restarts is invisible to a health table
-refreshed every 10 s, so a request routed to it returns 502 even though the
-other replica is up. On connection failure, demote it immediately and retry the
-request on a live replica before writing anything to the client (the OpenAI
-endpoints are stateless, so re-sending a request that never reached an engine
-is safe; never retry after streaming started). Measured: six requests during a
-replica stop, six 200s, one logged failover, no client-visible error.
-
-## The capacity wall that no policy fixes
-
-Per-card KV is finite and **a running sequence and its cached prefix draw on
-the same pool**. With N families on M cards, one card carries two of them, and
-its budget must hold both cached prefixes plus whichever sequence is
-generating. Three concurrent 50-80k-token families against 122,636 tokens per
-card thrash regardless of how requests are placed:
-
-| policy | result |
-|---|---|
-| counter placement | heavy 25.4 s median (23 % cached), mediums 34.7 / 45.2 s |
-| load-balanced placement (new families to the card carrying less context) | no improvement — reverted |
-| one large request admitted per card | no improvement — reverted |
-| smaller/larger prefill chunk | *costs* KV in both directions (a sweet spot exists; find it) |
-
-So: measure placement and admission before shipping them, and when the working
-set exceeds aggregate KV, say so — the levers are capacity (another card),
-fewer concurrent large contexts (client-side context compaction), or a different
-platform. Routing cannot rescue arithmetic.
-
-## Measurement traps
-
-- **A lifetime engine counter is not a per-request verdict.** "Prefix cache hit
-  rate: 21 %" is an average over every prompt the engine ever saw, including
-  benchmark traffic that never repeats. Probe the question directly instead:
-  send the same long prompt to an idle replica twice — 22.3 s cold then 1.4 s
-  warm is what a working cache looks like (and it holds with a varying tail).
-- **Salt your corpora.** Two runs sharing a prefix measure a warm cache, not
-  the thing under test; a unique leading line per run makes the prefix new.
-- **Record the sample size.** A 14-minute arm and an 8-minute arm are not a
-  comparison, and medians from five turns say little.
-- **Check the tool's own requirements**: `vllm bench serve` needs
-  `--tokenizer <local path>` (plus the model directory mounted) or it tries to
-  resolve the served alias on HuggingFace and dies before sending a request —
-  a run that produces no numbers and no obvious error.
-- **Verify metric names against the live endpoint** before writing dashboards
-  or alerts: guessing one that does not exist is invisible until someone needs
-  the graph.
-
-## Test shape that catches it
-
-Drive the routing function directly with synthetic request bodies (keep the
-policy in a stdlib-only module so tests need no HTTP framework):
-
-- a batch of single-turn requests -> assert both replicas are used and
-  consecutive picks alternate;
-- a growing conversation (same leading bytes, longer history) -> assert every
-  turn agrees with the first turn's replica;
-- each mode with a replica marked down -> assert the live one is chosen, and
-  -1 when none is;
-- the fixed comparison modes (`roundrobin`, `affinity`) kept behind an env var
-  so the choice can be re-measured rather than argued.
-
-One more: the identity of a conversation is its **first N KiB** of prompt. If
-the tested prefix is shorter than that window its varying tail falls inside it,
-turns hash differently, and pinning silently never happens.

agent/managed-skills/llm-serving-benchmark-methodology/SKILL.md

@@ -1,67 +0,0 @@
----
-name: llm-serving-benchmark-methodology
-description: "Measure and A/B a self-hosted multi-replica LLM endpoint honestly: corpus construction, per-arm salting, repeated samples, engine-counter semantics, and the traps that make results meaningless."
----
-
-# Bench a multi-replica LLM deployment so the numbers mean something
-
-Use when measuring or tuning a self-hosted LLM endpoint (vLLM, llama.cpp,
-radiance/MXFP4), especially behind a proxy in front of several replicas.
-
-## Set up a client that can actually send requests
-
-`vllm bench serve` needs `--tokenizer <local model path>` with the model
-directory mounted. Given only `--model <served-alias>` it tries to resolve that
-alias on HuggingFace and dies before the first request — a run that looks like
-it worked and produced nothing. Confirm with a single prompt before benching.
-
-## Build the workload from real material
-
-`--dataset-name random` generates token soup whose draft acceptance is nothing
-like production (measured 2.9-3.6 tok/update against 5.8 on code), so it
-mis-ranks speculative-decoding settings. Generate a JSONL corpus from the
-repository's own sources — `{"prompt": "<instruction>\n\n<file body>"}` — and
-fix output length with `--custom-output-len`. Keep two corpus shapes:
-
-- **distinct prompts** (batch behaviour, prefill parallelism)
-- **a shared long prefix with a varying tail** (conversation turns, where
-  prefix caching decides everything)
-
-## Salt every arm
-
-Put a unique marker in the first bytes of every prompt, different per A/B arm.
-Without it the second arm silently measures the first arm's warm cache and both
-arms look identical.
-
-## Measure the same workload before and after
-
-Keep the workload fixed and change one knob. Repeat each arm — run-to-run
-variance of 5-10 % is normal, and a single sample nearly made me ship the wrong
-setting. Report medians per family/class, not one aggregate number.
-
-## Read engine counters correctly
-
-- vLLM's `Prefix cache hit rate:` line is a **lifetime average** across every
-  prompt that replica ever saw. It cannot answer "is this conversation served
-  warm?". Send the same prompt twice to an idle engine and compare TTFTs
-  instead (measured: 22.3 s cold → 1.4 s warm for a 50k-token prefix).
-- `GPU KV cache usage` tells you the pool is not full; it does not tell you a
-  prefix survived. Capacity per card is the boot-time ledger line
-  (`GPU KV cache size: N tokens`), not the card's VRAM.
-- Reserve memory is not allocation: a bigger reserve does not free anything for
-  anyone else, it only removes cache.
-
-## Route-shape traps
-
-A cold batch of distinct prompts prefills in parallel across replicas; the same
-conversation's next turn only pays off if it lands where its prefix is cached.
-These pull in opposite directions, so measure both shapes separately and route
-by request shape (spread single-turn, pin continuations).
-
-## When the answer is "no"
-
-Write down rejected hypotheses with their numbers and the mechanism. Placement
-balance, concurrency admission and chunk-size tweaks all looked obvious and all
-measured flat or worse on this fleet; the wall turned out to be arithmetic
-(cached prefixes plus live sequences share one pool). A runbook that records
-that saves the next operator a night.

agent/managed-skills/niri-output-rotation-pinning/SKILL.md

@@ -1,96 +0,0 @@
----
-name: niri-output-rotation-pinning
-description: "Rotate/re-arrange niri monitors (portrait transforms, positions, scale) persistently and verify — covers the output-block-resets-defaults trap and auto-layout reflow that flings unpinned outputs, plus the counter-clockwise transform convention."
----
-
-# Rotating and pinning outputs on niri
-
-Verified on blankpad, niri 26.04, 2026-09-15 (two PM1601 externals + eDP-1 laptop panel).
-
-## 0. Ground truth first
-
-```bash
-niri msg outputs            # connector names, current mode, scale, transform, logical pos/size
-niri --version
-```
-
-Niri output **names are connector names** (`eDP-1`, `DP-2`, `HDMI-A-1`), not EDID model strings. The EDID strings printed before them in `niri msg outputs` are decoration — use the parenthesized name.
-
-On blankpad the live config is split by `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl`, and `cfg/display.kdl` is a **symlink into the dotfiles repo** (`~/.omp/dotfiles/zz-blankpad/niri/.config/niri/cfg/display.kdl`). Edit the repo path, not the symlink target path — they are the same file, but the repo path is the one that gets committed. Check with `ls -ld`.
-
-## 1. Transform direction is counter-clockwise
-
-```bash
-niri msg output <NAME> transform --help   # "Transform to set, counter-clockwise"
-```
-
-| value | effect |
-|---|---|
-| `90` | content rotated 90° CCW ⇒ the panel's **top bezel sits on the LEFT** |
-| `270` | top bezel on the RIGHT |
-| `180` | upside down |
-| `flipped-90` / `flipped-270` | mirrored variants |
-
-`90` vs `270` is a physical-stand fact no tool can read (portable monitors expose no orientation in EDID/MCCS). Pick one, apply it, and give the user the one-liner to flip — don't stall on the question.
-
-## 2. Two traps that bite when you edit the config
-
-1. **An `output "X" { … }` block resets every property you don't restate to its default.** Pinning position on `eDP-1` without repeating `scale 1.25` silently drops the laptop panel to scale 1.0. Restate `scale` (and `mode` if you care) in every block you touch.
-2. **Niri re-runs auto-layout for outputs with no pinned position.** Pinning only the externals shoved `eDP-1` from `0,0` to `3936,0` (laptop became rightmost). If you want the pre-existing left-to-right order, pin every output's position.
-
-Positions are logical pixels at each output's own scale. Original order preserved example:
-
-```kdl
-output "DP-2" {
-    scale 1
-    transform "90"
-    position x=1536 y=0
-}
-
-output "HDMI-A-1" {
-    scale 1
-    transform "90"
-    position x=2736 y=0
-}
-
-output "eDP-1" {
-    scale 1.25          // MUST restate: dashed-out by the block otherwise
-    position x=0 y=0
-}
-```
-
-x-offsets chain: `eDP-1` logical width 1536 → DP-2 at 1536 → +1200 (portrait width) → HDMI-A-1 at 2736.
-
-## 3. Apply and verify
-
-```bash
-niri validate -c ~/.config/niri/config.kdl          # exit 0, "config is valid"
-niri msg action load-config-file                    # live reload; load-config-file, NOT reload-config
-sleep 1
-niri msg outputs | grep -E '^Output|Transform|Logical|Scale'
-```
-
-Success looks like `Logical size: 1200x1920` (swapped) plus `Transform: 90° counter-clockwise` on each rotated panel, with positions matching the intended chain.
-
-One-off rotation without touching config (temporary, forgotten on next config change):
-
-```bash
-niri msg output DP-2 transform 90
-```
-
-## 4. Commit
-
-Dotfiles repo is `dank/command-center` (`main`); push to the local Gitea is permitted.
-
-```bash
-cd ~/.omp && git pull --ff-only
-git add dotfiles/zz-blankpad/niri/.config/niri/cfg/display.kdl
-git commit -m "blankpad(niri): <what changed>"
-git push
-```
-
-## Pitfalls
-
-- `reload-config` is not a niri subcommand — it's `niri msg action load-config-file`.
-- Never SIGUSR1 a terminal to "reload"; niri config reloads do not touch ghostty themes.
-- No man page for niri on this host; `niri <sub> --help` and `/usr/share/doc/niri/default-config.kdl` are the docs.

agent/managed-skills/noctalia-config-remote-edit/SKILL.md

@@ -1,28 +0,0 @@
----
-name: noctalia-config-remote-edit
-description: "Edit live Noctalia v5 shell config on a fleet host over SSH: which config layer is live, why noctalia msg fails from ssh context, and the systemd-run --pipe workaround that works."
----
-
-# Noctalia config edit over SSH (v5 native, both fleet hosts)
-
-## Current architecture (since 2026-09-10 unification)
-Both the workstation and blankpad run the bare `noctalia` v5 binary as the desktop shell (launch: niri `spawn-at-startup "noctalia"`). The old Quickshell-hosted v4 architecture (`qs -c noctalia-shell`, JSON config) is retired on both hosts. Live config layer = TOML only:
-- Hand-edit declarative layer: `~/.config/noctalia/<name>.toml` (e.g. `config.toml`).
-- Runtime state file `~/.local/state/noctalia/settings.toml` loads LAST and overrides declarative values for the same keys — never hand-edit it; UI/IPC writes land there.
-
-## Why msg fails remotely, and the workaround that works
-`noctalia msg <command>` resolves the running instance via the D-Bus session bus. sshd-spawned non-login shells lack `DBUS_SESSION_BUS_ADDRESS`, so plain ssh exec prints `error: noctalia is not running` even when the shell IS running. Do not conclude "shell dead" from this error.
-Working remote procedure (runs in systemd user-manager context where the bus address exists):
-```
-systemd-run --user --pipe --wait noctalia msg <command> [args]
-```
-Verified: `msg status` returns live JSON state; unknown panel ids return an error listing all valid panel ids (clipboard, control-center, launcher, polkit, session, setup-wizard, test, tray-drawer, wallpaper).
-
-## Verified v5 msg verb grammar (keybinds-safe)
-`panel-toggle <id>`, `session lock|suspend|lock-and-suspend|logout|reboot|shutdown`, `volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute`, `media next|previous|playPause` (graceful no-op with rc 0 when no active player), `brightness-up [target] [step]`, `brightness-down`, `settings-open [context]`, `settings-toggle [context]`, `config-reload`, `wallpaper-get [connector]`, `wallpaper-set <path>`, `wallpaper-next`.
-
-## Wallpaper directory semantics
-Empty/unset `[wallpaper] directory` resolves to XDG Pictures (`$HOME/Pictures`) via user-dirs.dirs; an explicit directory key overrides. Fleet default pattern: point `directory` at the repo-tracked wallpapers dir so git-tracked wallpapers serve the shell without copying files. After writing config.toml over ssh, the file watcher hot-reloads automatically; `config-reload` via systemd-run --pipe forces it immediately. Note: the *current* wallpaper stays the built-in default asset until the next rotation or an explicit `wallpaper-next`; `wallpaper-get` then reports a path inside the configured directory = confirmation the config took effect.
-
-## Lock chain (unified)
-swayidle-lock.service ExecStart on both hosts: `/usr/bin/swayidle -w timeout 300 '/usr/bin/noctalia msg session lock' before-sleep '...' after-resume '...'`. `msg` resolves via session bus, so no WAYLAND_DISPLAY pinning is needed in unit context (niri's own /proc/PID/environ lacks WAYLAND_DISPLAY but spawn-at-startup children still receive a working display connection empirically — verified by qs-hosted autostart working from boot pre-unification).

agent/managed-skills/noctalia-custom-palette-recolor/SKILL.md

@@ -1,67 +0,0 @@
----
-name: noctalia-custom-palette-recolor
-description: "Recolor a live Noctalia desktop (accent, terminal palette, terminal foreground) by cutting over to a custom palette instead of editing the community-palette cache, then re-rendering and reloading templates."
----
-
-# Noctalia custom-palette recolor
-
-Recolor the running Noctalia shell + all themed apps without restarting the session.
-
-## Why custom, never the community cache
-- `~/.local/state/noctalia/community-palettes/*.json` is a **sync cache**. `~/.local/state/noctalia/community-palettes/.catalog/palettes.json` carries an `md5` per palette and noctalia re-fetches from `https://api.noctalia.dev/palettes`. Edits there get reverted.
-- Custom palettes live at `~/.config/noctalia/palettes/<Name>.json` (`NOCTALIA_CONFIG_HOME` then `XDG_CONFIG_HOME` honored) and are never synced. Missing/invalid file → silent fallback to the builtin palette.
-- Palette JSON shape = the community file's shape: `dark`/`light` maps of `mPrimary`, `mOnPrimary`, `mSecondary`, `mTertiary`, `mError`, `mSurface`, `mSurfaceVariant`, `mOutline`, `mShadow`, `mHover` plus a `terminal` block (`foreground`, `background`, `normal.{black..white}`, `bright.{...}`, `cursor`, `cursorText`, `selectionFg`, `selectionBg`). NOTE: `noctalia theme -o` emits M3 snake_case tokens (`on_primary_container`, …) — that is the *generator* format, not this one. Copy an existing community JSON and mutate it with `jq`.
-
-## Procedure
-1. Read the live scheme first — never assume: `noctalia msg color-scheme-get` → e.g. `community Gruber Darker`; `noctalia msg theme-mode-get`.
-2. Back up settings: `cp -a ~/.local/state/noctalia/settings.toml /tmp/noctalia-settings-backup-$(date +%s).toml`.
-3. Build the palette from the current one (jq keeps structure; no hand-editing 2 KB of JSON):
-   ```sh
-   mkdir -p ~/.config/noctalia/palettes
-   jq '.dark.mPrimary = "#f43841"
-       | .dark.terminal.normal.yellow = "#f43841"
-       | .dark.terminal.bright.yellow = "#ff4f58"
-       | .dark.terminal.foreground   = "#f43841"' \
-      "$HOME/.local/state/noctalia/community-palettes/Gruber%20Darker.json" \
-      > ~/.config/noctalia/palettes/"Gruber Red.json"
-   ```
-   Prefer a hex already present in the palette (its own `mError` red / accent) so the theme stays internally consistent. Set BOTH `normal` and `bright` variants, and both `dark` and `light` if the user toggles modes.
-4. Switch (persists `theme.source = custom` + `theme.custom_palette`):
-   `noctalia msg color-scheme-set custom "Gruber Red"` → `ok`; confirm with `noctalia msg color-scheme-get`.
-5. Re-render every configured template: `noctalia msg templates-apply`. Regenerates `~/.config/ghostty/themes/noctalia`, `~/.config/gtk-{3,4}.0/noctalia.css`, kcolorscheme, foot, wezterm, starship, etc. Some of those paths are symlinked into the dotfiles repo → they show up as modified files worth committing.
-6. Reload surfaces:
-   - Compositor (focus border, `noctalia.kdl`): `niri msg action load-config-file`. `reload-config` is NOT a valid niri subcommand.
-   - Ghostty: see the reload trap below.
-
-## Verification
-- `ghostty +show-config | grep -E '^(foreground|palette)'` — proves the theme file resolves into the config (e.g. `foreground = #f43841`, `palette 3/11` red).
-- `noctalia config validate` → `✓ Config is valid`.
-- Visual: `noctalia msg screenshot-fullscreen` saves to `~/Pictures/screenshot_<ts>.png` (it ALSO overwrites the clipboard with the PNG — warn the user). `read` that PNG to confirm accent/toggle/border color.
-- `grep -c <old-hex> ~/.config/noctalia/palettes/*.json` → 0 for no leftovers.
-
-## Track it in the dotfiles repo (command-center `~/.omp`)
-`scripts/deploy.sh` is a stow-equivalent: packages are home-relative trees under
-`dotfiles/<pkg>/`, and deploying replaces live files with symlinks into the repo. Mirror that
-shape for the new palette so `git pull` + `deploy.sh` is the whole propagation story:
-```sh
-mkdir -p ~/.omp/dotfiles/noctalia/.config/noctalia/palettes
-mv ~/.config/noctalia/palettes/"<Name>.json" ~/.omp/dotfiles/noctalia/.config/noctalia/palettes/
-ln -s "$HOME/.omp/dotfiles/noctalia/.config/noctalia/palettes/<Name>.json" \
-      ~/.config/noctalia/palettes/"<Name>.json"
-noctalia msg color-scheme-set custom "<Name>"   # re-reads through the symlink; expect ok
-```
-Do the link by hand for one file instead of running `deploy.sh` — sync mode quarantines and
-replaces every divergent live file, which is broader than a single-palette change.
-
-## Traps
-- **NEVER `kill -USR1` ghostty** to reload. It does not reload config on this fleet — it kills/respawns the terminal (took out Frank's window 2026-09-13; he said "stop killing the terminal"). Correct: the default keybind `ctrl+shift+,` = `reload_config`, per `ghostty +show-config --default`. Existing windows keep old colors until reloaded; new windows inherit immediately.
-- Palette hex values are NOT a proxy for what the user sees: in `Gruber Darker` the *accent* (`mPrimary`) is yellow `#ffdd33`, so "change the yellow" means the accent too, not just terminal index 3.
-- Terminal text turned into a saturated color makes ANSI 1/9 indistinguishable from default text (grep highlights flatten) — flag it and offer a lighter red.
-- Contrast sanity: `#f43841` on `#181818` ≈ 4.6:1 (dark-on-color beats white-on-color here, ~3.9:1, so keep `mOnPrimary` dark).
-- Revert = `noctalia msg color-scheme-set community "<original name>"`; the community cache is untouched by this whole flow.
-- `ghostty +show-config` prints only the delta from Ghostty defaults — a key whose value equals
-  the default (e.g. `font-size = 12`) simply does not appear. Cross-check with
-  `ghostty +show-config --default` before concluding a key is unset.
-- Generated theme files that are symlinked into the dotfiles repo (`gtk-3.0`/`gtk-4.0`
-  `noctalia.css`, ghostty theme) show up as tracked modifications after `templates-apply`;
-  commit them together with the new palette JSON so every machine gets the same recolor.

agent/managed-skills/noctalia-reference/SKILL.md

@@ -1,32 +0,0 @@
----
-name: noctalia-reference
-description: "Quick reference for Noctalia desktop environment: config paths, IPC commands, app templates, palette, and migration checklist for Niri-based setups."
----
-
-# Noctalia Desktop Environment
-
-Noctalia is a Niri-based desktop environment at `/usr/bin/noctalia`.
-
-## Config
-- Main: `~/.config/noctalia/config.toml`
-- Niri config: includes `noctalia.kdl` for display settings
-
-## IPC
-Direct `noctalia msg <command>` replaces `qs -c noctalia-shell ipc call` wrapper.
-Key commands: `panel-toggle`, `session`, `lock`, `launcher`.
-
-## App Templates
-`/usr/share/noctalia/assets/templates/` contains: gtk, ghostty, alacritty, hyprland, sway, wezterm, helix, btop, starship, qt, kde, cava, scroll, mango, umbriel, foot, emacs, kitty.
-
-## Palette
-Generate from image: `noctalia theme [image]`
-
-## Migration Checklist
-- [ ] Niri keybinds → `noctalia msg` commands
-- [ ] Alacritty → import noctalia.toml theme
-- [ ] Ghostty → `theme = noctalia`
-- [ ] GTK 3/4 → update accent palette (#b267e4 purple)
-
-## Text Antialiasing
-- **Ghostty**: `font-subpixel-render = "off"` to disable subpixel AA
-- **Alacritty**: `font.antialias = false` in `alacritty.toml`

agent/managed-skills/noctalia-template-offline-render/SKILL.md

@@ -1,45 +0,0 @@
----
-name: noctalia-template-offline-render
-description: "Recover from missing noctalia-generated theme files (niri noctalia.kdl include, ghostty/alacritty noctalia theme) using the offline noctalia theme --render CLI when msg templates-apply is a no-op"
----
-
-# Noctalia template offline render
-
-## Failure signature
-- niri: `failed to read included config from ~/.config/niri/noctalia.kdl: No such file or directory`; desktop runs but with DEFAULT config (niri 26.04 falls back, does not exit): no custom keybinds/rules, no noctalia autostart, default output placement.
-- ghostty: `theme "noctalia" not found, tried ~/.config/ghostty/themes/noctalia`.
-- alacritty: import of `~/.config/alacritty/themes/noctalia.toml` missing.
-
-## Root cause pattern
-Upstream dotfiles commits theme switches (niri include, ghostty `theme = noctalia`, alacritty import) that depend on noctalia TEMPLATE-GENERATED files. Those files are per-host outputs (rendered from the local palette) and are intentionally NOT in the repo. On any host where `templates-apply` never ran, they don't exist.
-
-## Fix (offline, works while a session runs)
-Built-in templates live at /usr/share/noctalia/assets/templates/ and are declared in builtin.toml. Render each needed one with the live palette:
-
-```bash
-# niri include (plain host-local file; do NOT symlink into repo — regenerated per host)
-noctalia theme --theme-json ~/.config/noctalia/colors.json \
-  --render "/usr/share/noctalia/assets/templates/niri/niri.kdl:$HOME/.config/niri/noctalia.kdl"
-
-# ghostty
-mkdir -p ~/.config/ghostty/themes && \
-noctalia theme --theme-json ~/.config/noctalia/colors.json \
-  --render "/usr/share/noctalia/assets/templates/ghostty/ghostty:$HOME/.config/ghostty/themes/noctalia"
-
-# alacritty
-mkdir -p ~/.config/alacritty/themes && \
-noctalia theme --theme-json ~/.config/noctalia/colors.json \
-  --render "/usr/share/noctalia/assets/templates/alacritty/alacritty.toml:$HOME/.config/alacritty/themes/noctalia.toml"
-```
-
-All template tokens (primary, surface, error, primary_container, …) resolve from the flat 16-key colors.json; container tokens are derived, never blank.
-
-## Verify
-1. `niri validate` → "config is valid" (rc 0)
-2. `ghostty +validate-config` → rc 0
-3. On next login: journal shows `loaded config from config.kdl` with NO parse WARN, custom output placement (compare `niri msg outputs` vs display.kdl), noctalia autostarted (`pgrep noctalia`).
-
-## Traps
-- `noctalia msg templates-apply` on 5.0.1 returns "ok" but applies NOTHING (opt-in config schema this build does not act on). Do not trust its exit code.
-- Niri config errors at boot are NOT fatal — the compositor keeps running on defaults, which looks like "some settings silently stopped working", not a crash.
-- Regenerate after any palette change; the rendered files are static snapshots.

agent/managed-skills/omp-harness-window-fit-audit/SKILL.md

@@ -0,0 +1,24 @@
+---
+name: omp-harness-window-fit-audit
+description: "Audit an omp harness model entry against an engine's prompt+max_tokens fit check; explains and fixes 400 \"max_tokens does not fit\" at 60-85 % of the declared window."
+---
+
+Use when a local endpoint returns 400 `max_tokens N does not fit: prompt is P tokens and the context is C, leaving room for R` on omp, typically at 65-85 % of the declared window.
+
+## The three numbers that must agree
+1. Engine hard ceiling: prompt_tokens + max_tokens ≤ window (halogen `HALOGEN_CTX`/slot, llama.cpp per-slot `ctx_size/parallel`, radiance `--max-model-len`). A 400, never truncation.
+2. Harness clamp: `max_tokens = min(maxTokens, contextWindow − est − 4096)`, est = chars/4 over the conversation+tools. The dump in `~/.omp/logs/http-400-requests/*.json` proves it: est = contextWindow − 4096 − max_completion_tokens.
+3. Compaction fires only at `compaction.thresholdPercent` × contextWindow (config.yml; ignore stale reserveTokens/70 % comments).
+
+## The trap
+chars/4 under-counts the rendered chat template by ~3 % on code/agent traffic. If contextWindow == engine ceiling, requests with est > 4096/0.03 ≈ 137k are guaranteed 400s below the compaction threshold (65-85 % of a 200k window). maxTokens ≤ 16,384 hides this; 45,056 exposes it.
+
+## Fix procedure
+1. Compute required slack: maxTokens + 0.10 × contextWindow (robust to est error).
+2. Raise the engine ceiling to ≥ contextWindow + slack (native window is the natural choice: Qwen3.8-Flash-Next 262,144). Halogen: pool must stay slots × CTX (`HALOGEN_KV_POOL_POSITIONS`, ~28 KiB/position); check memory ledger.
+3. Keep harness contextWindow below the ceiling by ≥ slack (217,088 = 212×1024 for a 262,144 ceiling with maxTokens 45,056).
+4. Verify compaction itself fits its target (remoteEndpoint slot ≥ thresholdPercent × contextWindow − keepRecentTokens(20k) + overhead).
+5. Update env + seam test + models.yml + AGENTS.md together; a single-file change re-creates the drift.
+
+## Verification
+Probe a prompt at ≥ 180k tokens with max_tokens = the clamped value; expect completion, not 400.

agent/managed-skills/omp-per-model-sampler-config/SKILL.md

@@ -1,60 +0,0 @@
----
-name: omp-per-model-sampler-config
-description: "Set per-model sampler params and preserve_thinking in OMP's models.yml via compat.extraBody, and verify the exact wire body with an echo-capture profile."
----
-
-# OMP Per-Model Sampler Config (`compat.extraBody`)
-
-`~/.omp/agent/models.yml` has **no per-model sampler keys**. Sampling is global (`config.yml`: `temperature`, `topP`, `topK`, `minP`, `presencePenalty`, `repetitionPenalty`; default `-1` = **not sent**). The per-model hook is `compat.extraBody`.
-
-## Why it works
-`applyOpenAIExtraBody` runs **last** in the openai-completions builder — after `applyChatCompletionsCompatPolicy` — via `Object.assign(params, extraBody)`. So extraBody overrides global settings and compat-policy output.
-
-## Recipe
-Use **wire names**, not OMP camelCase:
-
-```yaml
-- id: frank-large
-  compat:
-    reasoningEffortMap:
-      max: xhigh
-    extraBody:
-      temperature: 1.0
-      top_p: 0.95
-      top_k: 20
-      min_p: 0.0
-      presence_penalty: 0.0
-      repetition_penalty: 1.0
-      preserve_thinking: true
-```
-
-## Two traps
-
-1. **`preserve_thinking` is not user-settable.** `qwenPreserveThinking` is auto-derived in `buildOpenAICompat` from the *auto-detected* `thinkingFormat` + local baseUrl, **before** `applyCompatOverrides`. Setting `compat.thinkingFormat: qwen` in models.yml does **not** enable it. Custom-named models (`frank-large`) match no Qwen id pattern → use `extraBody: {preserve_thinking: true}`.
-2. **Never put `chat_template_kwargs` in `extraBody`.** `Object.assign` is shallow — it replaces the whole object and silently drops the `reasoning_effort` kwarg the compat policy set. Top-level `preserve_thinking` is honoured by halogen; use that.
-
-## Gotcha: silent greedy decode
-halogen-flash-server has **no server-side sampler defaults** — `temperature` absent or `0` = greedy. If OMP sends nothing, the endpoint decodes greedy while looking configured.
-
-## Verify the exact wire body (echo-capture)
-Don't trust the YAML — capture what OMP actually emits.
-
-1. Echo server on `127.0.0.1:18099` that dumps the JSON body to a file and returns a canned `chat.completion` (include `reasoning_content` so thinking paths exercise).
-2. Copy the **real** models.yml into a throwaway profile with only the baseUrl swapped:
-   ```bash
-   sed 's|http://<host>:7770/v1|http://127.0.0.1:18099/v1|' \
-     ~/.omp/agent/models.yml > ~/.omp/profiles/wiretest/agent/models.yml
-   ```
-3. Run and inspect:
-   ```bash
-   omp --profile wiretest --model "<provider>/<model>" -p "hi" \
-     --no-tools --no-session --no-extensions --no-skills --no-rules
-   jq 'del(.messages, .tools)' last-body.json
-   ```
-   Confirm every intended field is present with the right value.
-
-## Verify a field is *honoured*, not just accepted
-Unknown fields often return 200 anyway. Prove behavior with a **prompt-token delta**: replay an assistant turn carrying a long `reasoning_content`, then compare `usage.prompt_tokens` with the flag true vs false. A large delta means the field changed template rendering; zero delta means it was ignored.
-
-## Cleanup
-Trash the probe profile and echo server (`profiles/.AGENTTRASH`), never `rm`.

agent/managed-skills/openwebui-picker-acl-repair/SKILL.md

@@ -0,0 +1,29 @@
+---
+name: openwebui-picker-acl-repair
+description: "Fix OpenWebUI 0.6 \"No results found\" / Model not found in the model picker for non-admin users (Dankclaw fleet) after a connection rebuild."
+---
+
+# OpenWebUI 0.6 picker ACL repair (Dankclaw)
+
+Symptom: admin sees models in OpenWebUI; regular users get "No results found" in the picker, or chat answers "Model not found". Cause: an OpenAI-connection rebuild (config keys `openai.api_base_urls` / `openai.api_configs` in the `config` table) that dropped the ACL pair for the model.
+
+## Law
+A non-admin user sees an external OpenAI-connection model only if BOTH exist in `webui.db`:
+1. `model` table (singular) row: `id` == upstream model id, `base_model_id` NULL, `is_active` 1, `meta` JSON (description etc.).
+2. `access_grant` row: `(resource_type='model', resource_id=<model id>, principal_type='user', principal_id='*', permission='read')`.
+
+The aggregation (`utils/models.py get_filtered_models`) drops info-less models for `role=user`; admins bypass. `get_all_models` responses are served from `model_ids` in `api_config` when non-empty (no upstream fetch). An alias row (`id` != upstream, `base_model_id` set) renders in the picker but chat fails "Model not found" — id must equal the upstream id.
+
+## Steps
+1. Reproduce as the affected role: in a browser tab, `fetch("/api/models", {headers:{Authorization:"Bearer "+localStorage.token}})` → `data: []` while an admin API key (`api_key` table, column `key`, `sk-…`) lists the model.
+2. DB at `/home/dank/Containers/open-webui/data/webui.db` (host; no sqlite3 CLI on Dankclaw — use host `python3` heredoc over ssh; container path `/app/backend/data`).
+3. Insert/repair the pair (uuid4 id, epoch-ms `created_at`):
+   - `model`: id/name = upstream id, base_model_id NULL, is_active 1, user_id = admin uid (role='admin' in `user` table), meta `{"description": "…"}`.
+   - `access_grant`: shape above.
+4. Cache: `MODELS_CACHE_TTL` defaults to 1 s — no restart needed for lists; a restart (`systemctl --user restart open-webui`, rootless) only if config edits were cached oddly. An OpenWebUI restart invalidates browser JWTs (401) — expect a silent SSO re-round-trip; kill stale tabs first so OAuth `state` cookies don't race (MismatchingStateError → clear cookies/storage, one clean flow).
+5. Verify end-to-end: pick the model in the UI, send a short exact-match prompt, confirm the reply; check `/api/models` as the user role lists the id.
+
+## Related knobs
+- User API keys (User menu → Settings → Account → API keys → Show → Create new secret key) require `user.permissions` JSON `features.api_keys=true` in `config` (default false).
+- New SSO signups land as role `pending` (`ui.default_user_role=pending`) — activate: `UPDATE user SET role='user' WHERE email LIKE …`. Table is `user`, singular; no `active` column.
+- Anubis gates `ai.franksdank.cloud` for headless browsers (webdriver detection): drive direct `:3005` for UI work, or emulate a desktop Chrome UA.

agent/managed-skills/pcie-link-capability-probe/SKILL.md

@@ -1,122 +0,0 @@
----
-name: pcie-link-capability-probe
-description: "Measure a GPU-to-host PCIe link the way a weight-streaming engine experiences it, and convert the result into a tokens/s ceiling (streamed expert weights) versus the compute-in-RAM alternative — before recommending or rejecting an offload design."
----
-
-# PCIe link capability probe
-
-Use when someone asks "is the link fast enough to run model X with engine Y (on one GPU)?"
-and Y keeps some weights in host RAM — MoE expert offload with an LRU GPU cache, layer
-paging, weight streaming. The answer is arithmetic once the link is measured, and the
-*shape* of the engine decides whether the link matters at all:
-
-| engine shape | what crosses PCIe per token | link matters? |
-|---|---|---|
-| streams weights (expert LRU, paging) | the missed weights (~GB/token) | **yes — decisive** |
-| computes offloaded weights in RAM (llama.cpp `-ot exps=CPU`) | activations only (~KB) | no; host RAM bandwidth decides |
-| everything resident on the GPU | nothing | no |
-
-## 1. Establish the hardware ceiling (never trust the card alone)
-
-```bash
-lspci -tv | grep -iA2 vga                       # which bridge the GPU hangs off
-d=$(lspci -D | grep -i nvidia | awk '{print $1}' | head -1)
-sudo lspci -vv -s "$d" | grep -E 'LnkCap:|LnkSta:'    # the CARD's capability + trained state
-sudo lspci -vv -s 00:03.1 | grep -E 'LnkCap:|LnkSta:'  # the PLATFORM PORT (this is the real limit)
-```
-
-- `LnkCap` of the **upstream port** = the electrical ceiling. A card with `Width x16` under
-  an `x8` port cannot exceed x8; an x4 riser/cable caps both.
-- `LnkSta` at idle usually shows a downgraded speed (PCIe ASPM downshift, e.g.
-  `2.5GT/s (downgraded)`) — that is NOT the loaded rate. Trust the measurement in step 2.
-- Gen4 usable payload: x4 **7.877 GB/s**, x8 15.75, x16 31.5 (128b/130b encoding).
-
-## 2. Measure it (the program that did the job)
-
-Stop **all** production services on the host first (fleet policy), then compile and run:
-
-```cuda
-// pcie-link-test.cu : nvcc -O2 -o /tmp/pcie-link-test /tmp/pcie-link-test.cu
-#include <cstdio>
-#include <cstdlib>
-#include <cstring>
-#include <cuda_runtime.h>
-#define CK(x) do{ cudaError_t e=(x); if(e!=cudaSuccess){fprintf(stderr,"CUDA %s (%s:%d)\n",cudaGetErrorString(e),__FILE__,__LINE__); return 1;} }while(0)
-static double ms_between(cudaEvent_t a, cudaEvent_t b){float ms=0; cudaEventElapsedTime(&ms,a,b); return (double)ms;}
-static int bench_size(size_t bytes){
-    const int iters = bytes>=(1UL<<30)?10:(bytes>=(256UL<<20)?20:100);
-    void *dev=NULL,*host=NULL,*pageable=NULL; cudaStream_t s=NULL;
-    CK(cudaMalloc(&dev,bytes)); CK(cudaHostAlloc(&host,bytes,cudaHostAllocDefault));
-    memset(host,0x5a,bytes); CK(cudaStreamCreate(&s));
-    cudaEvent_t a,b; CK(cudaEventCreate(&a)); CK(cudaEventCreate(&b));
-    for(int i=0;i<3;i++) CK(cudaMemcpy(dev,host,bytes,cudaMemcpyHostToDevice));
-    CK(cudaDeviceSynchronize());
-    CK(cudaEventRecord(a)); for(int i=0;i<iters;i++) CK(cudaMemcpy(dev,host,bytes,cudaMemcpyHostToDevice));
-    CK(cudaEventRecord(b)); CK(cudaEventSynchronize(b)); double h2d=(double)bytes*iters/(ms_between(a,b)/1000.0)/1e9;
-    CK(cudaEventRecord(a)); for(int i=0;i<iters;i++) CK(cudaMemcpyAsync(dev,host,bytes,cudaMemcpyHostToDevice,s));
-    CK(cudaEventRecord(b)); CK(cudaEventSynchronize(b)); double h2da=(double)bytes*iters/(ms_between(a,b)/1000.0)/1e9;
-    CK(cudaEventRecord(a)); for(int i=0;i<iters;i++) CK(cudaMemcpy(host,dev,bytes,cudaMemcpyDeviceToHost));
-    CK(cudaEventRecord(b)); CK(cudaEventSynchronize(b)); double d2h=(double)bytes*iters/(ms_between(a,b)/1000.0)/1e9;
-    CK(cudaMemcpy(dev,host,bytes,cudaMemcpyHostToDevice));
-    CK(cudaEventRecord(a)); CK(cudaMemcpy(dev,host,bytes,cudaMemcpyHostToDevice)); CK(cudaEventRecord(b)); CK(cudaEventSynchronize(b));
-    double one_us=ms_between(a,b)*1000.0;
-    pageable=malloc(bytes); memset(pageable,0x5a,bytes);
-    for(int i=0;i<2;i++) CK(cudaMemcpy(dev,pageable,bytes,cudaMemcpyHostToDevice));
-    CK(cudaEventRecord(a)); for(int i=0;i<iters;i++) CK(cudaMemcpy(dev,pageable,bytes,cudaMemcpyHostToDevice));
-    CK(cudaEventRecord(b)); CK(cudaEventSynchronize(b)); double pag=(double)bytes*iters/(ms_between(a,b)/1000.0)/1e9;
-    printf("%8.1f MiB | %6.2f | %6.2f | %6.2f | %8.1f us | %6.2f\n", bytes/1048576.0,h2d,h2da,d2h,one_us,pag);
-    free(pageable); cudaFreeHost(host); cudaFree(dev); cudaEventDestroy(a); cudaEventDestroy(b); cudaStreamDestroy(s); return 0;
-}
-int main(void){ cudaDeviceProp p; CK(cudaGetDeviceProperties(&p,0));
-    printf("device %s | CUDA %d.%d\n", p.name, p.major, p.minor);
-    printf("   MiB  | H2D    | async  | D2H    | 1x xfer    | pageable\n");
-    size_t sz[]={3UL<<20,16UL<<20,64UL<<20,256UL<<20,1UL<<30,2UL<<30};
-    for(int i=0;i<6;i++) if(bench_size(sz[i])) return 1; return 0; }
-```
-
-Size choice matters: **3 MiB** is the granularity an expert-streaming engine moves, so
-its single-transfer time is the number that matters; the big sizes give the ceiling.
-
-## 3. Read it
-
-- **Achieved GB/s is itself the proof of the trained link state** — 6.7 GB/s is only
-  possible at Gen4 x4 (Gen1 x4 ≈ 1, Gen2 ≈ 2, Gen3 ≈ 3.94), so no separate `LnkSta`
-  sample under load is needed.
-- **async ≈ sync and pageable ≈ pinned ⇒ link-limited, not staging-limited.** If async is
-  much faster than sync, the engine's fill pipeline (batching copies on one stream) is
-  where the win is; if pageable is much slower, pinning is a real lever. Measure, don't
-  assume.
-- Per-slab launch overhead = single-xfer time − (slab bytes ÷ rate). Small next to
-  bandwidth cost at 3 MiB.
-
-## 4. Convert to a tokens/s ceiling
-
-```
-active expert bytes/token  = active_params × bytes/param   (A3B @ Q4_K_M ≈ 1.70 GiB)
-link time/token            = active expert bytes / measured H2D rate
-ceiling (tok/s)            = 1 / link time/token      × resident-fraction saved (LRU hits)
-```
-
-Then compare against the alternatives on the same host — and against the
-compute-in-RAM path computed at host RAM bandwidth (256 GB/s LPDDR5X on this fleet's
-Strix Halo). If the link is >10× slower than the memory behind it (x4 is 31× here),
-streaming loses by construction and no tuning rescues it; doubling the link (x4 → x8)
-only doubles a losing number.
-
-## Worked example (Dankclaw 2026-09-16)
-
-Gen4 x4, measured H2D 6.72 / D2H 6.60 GB/s (85 % of spec), 3 MiB slab = 474 µs.
-A3B MoE, Q4_K_M (75 GiB experts, ~1.70 GiB active/token): 272 ms/token ⇒ **3.7 tok/s**
-(4.6 with 20 % resident). Same host, same model, measured: **20.46 tok/s** with the
-experts computed on the CPU, 25 (halogen), 26.5 (mainline on the iGPU). Verdict: the
-streaming design loses 3-5×, and the engine in question could not load the GGUF anyway.
-
-## Pitfalls
-
-- Stop production services before running this (fleet policy) — and remember the 3090's
-  VRAM may be almost fully occupied, so a 2 GiB test buffer needs the service stopped.
-- Don't quote a derived line from inside the program without checking it: a printf
-  format/argument mismatch in this very probe printed a nonsense ceiling. Recompute by hand.
-- GiB vs GB: convert before dividing (1.70 GiB = 1.825 GB).
-- The idle `LnkSta` (2.5 GT/s, downgraded) is not the loaded link — quoting it makes a
-  working x4 look like a broken x1.

agent/managed-skills/pcie-link-degradation-vs-idle-triage/SKILL.md

@@ -1,101 +0,0 @@
----
-name: pcie-link-degradation-vs-idle-triage
-description: "Decide whether a GPU/NVMe PCIe link is genuinely degraded (bad riser, bad slot, wrong connector) or just idle-downshifted, and whether moving the card to another slot could widen it — using kernel boot bandwidth lines, port LnkCap vs LnkSta, and NVML sampled under load."
----
-
-# PCIe link: degraded or just idle?
-
-Answer "is my GPU running at full speed?" without fooling yourself. The trap: an
-idle NVIDIA GPU reports `LnkSta: Speed 2.5GT/s (downgraded)` because the driver
-downshifts the link to Gen1 for power. That is not a fault, and it is not the
-slot's fault either.
-
-## Step 1 — the path ceiling, from the kernel (cheapest, most decisive)
-
-```bash
-sudo journalctl -k -b --no-pager | grep "available PCIe bandwidth"
-```
-
-Reads out per endpoint, e.g.:
-
-```
-pci 0000:c4:00.0: 63.012 Gb/s available PCIe bandwidth, limited by 16.0 GT/s
-PCIe x4 link at 0000:00:03.1 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
-```
-
-- `63.012 Gb/s` = Gen4 **x4** (~7.88 GB/s/direction after 128b/130b encoding).
-- "limited by ... link at <BDF>" names the **bottleneck device** (usually a root
-  port) and its width — this is the number that decides whether a swap can help.
-- "capable of ... x16 link" is the **endpoint's own** capability, not the path's.
-  A big gap between the two is the whole story.
-
-`dmesg` is often rotated; `journalctl -k -b` keeps this boot.
-
-## Step 2 — the two sides separately
-
-```bash
-sudo lspci -vv -s <endpoint> | grep -E "LnkCap:|LnkSta:|LnkCtl2:|LnkCap2:"
-sudo lspci -vv -s <root port> | grep -E "LnkCap:|LnkSta:|LnkCtl2:"
-```
-
-- `LnkSta` on a *bridge* is the trained link; on an endpoint it is that
-  endpoint's view. Widths/`(downgraded)` compare against the **own** `LnkCap`.
-- `LnkCtl2: Target Link Speed: 16GT/s` on both ends = the platform *intends*
-  Gen4, so a Gen1 current speed is power management, not misconfiguration.
-- A root port whose `LnkCap` width exceeds the trained width means the socket or
-  adapter wires fewer lanes than the controller can drive. **M.2 connectors are
-  4 lanes by pinout** — an M.2 riser can never deliver more, regardless of what
-  the bridge advertises.
-
-## Step 3 — prove the link under load (the step people skip)
-
-Never conclude from an idle reading on NVIDIA. Sample NVML while the GPU works:
-
-```bash
-nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.gen.max,\
-pcie.link.gen.hostmax,pcie.link.width.current,pcie.link.width.max --format=csv
-```
-
-Fire a small, legitimate request at whatever already serves on that GPU (e.g. a
-`max_tokens` completion against the local LLM endpoint) and poll the query while
-utilization is high:
-
-- `gen.current` climbing 1 → `hostmax` = idle downshift, link healthy.
-- `gen.current` stuck below `hostmax` under load = real degradation.
-- `width.current < width.max` under load = **physical lane limit** (wiring,
-  riser, or connector) — no software touches this.
-- On a busy production GPU, this is a usage smoke test, not a benchmark: never
-  stop/bench a live service to read its link.
-
-## Step 4 — do not plan hardware from `dmidecode -t slot`
-
-On consumer/mini-PC firmware this table is routinely fiction — it has been seen
-labelling an Ethernet port and an SD-reader controller as "M.2 Socket 3" slots
-and advertising a free slot in a **PCI domain that does not exist** (e.g.
-`0200:00:03.2`). Cross-check with the live bridge list:
-
-```bash
-lspci -nn | grep "PCI bridge"    # every root port, whether populated or empty
-lspci -t                          # topology; which endpoints hang off which port
-```
-
-Empty root ports still enumerate, so a genuinely absent port is genuinely
-absent. Trust live PCI, not SMBIOS.
-
-## Step 5 — verdict on "would moving the card to another slot help?"
-
-- Compare the **bottleneck device named in step 1** with the target slot's port
-  `LnkCap` width. If the bottleneck is the *endpoint side* (riser/adapter) and
-  the target port is wider, a move can help.
-- If both candidate connectors cap at the same width (two M.2 sockets = both
-  x4), moving is a **no-op** — say so plainly instead of endorsing the swap.
-- Error counters are a useful tiebreaker: clean `DevSta`/`CESta` with a low
-  trained speed means "wired narrow", not "failing".
-
-## Reporting
-
-Give the ceiling in Gb/s, the card's own capability, what the idle reading
-meant, and whether any physical change can move the number. If the answer is
-"the link is already at the hardware maximum", state that and quantify whether
-the workload actually cares (resident weights = PCIe is nearly idle per token;
-CPU layer offload = it hurts).

agent/managed-skills/pico8-asset-standards/SKILL.md

@@ -1,46 +0,0 @@
----
-name: pico8-asset-standards
-description: "Standards for building cohesive, production-grade PICO-8 asset sheets, tilesets, and spritesheets with consistent palette, perspective, and detail flow."
----
-
-# PICO-8 Asset Standards
-
-## Purpose
-Ensure every PICO-8 asset (tileset, spritesheet, character sheet, or asset sheet) is cohesive, production-grade, and ready to drop into a game without rework.
-
-## Core Requirements
-
-### Palette Discipline
-- Use the PICO-8 16-color palette exclusively.
-- Establish a dominant color family (2–3 related hues) for the main subject.
-- Reserve accent colors for highlights, danger zones, and focal points only.
-- Maintain consistent color roles across all assets in a sheet (e.g., if shadow is dark blue on one tile, it is dark blue everywhere).
-
-### Perspective & Lighting
-- Choose ONE perspective per sheet (top-down, side-view, or isometric) and never mix.
-- Apply light from a single consistent direction across all tiles and sprites.
-- Shadows follow the same shape logic on every asset (e.g., all cast shadows point bottom-right).
-
-### Detail Flow & Hierarchy
-- Start tiles/props from the palette's dominant colors, then add edge highlights and shading.
-- Keep tile density uniform — avoid one hyper-detailed tile next to empty flat tiles.
-- Props and decorations should read clearly at 8x8 or 16x16 without zooming.
-
-### Sheet Organization
-- Group related assets together (terrain together, props together, characters together).
-- Leave consistent spacing between assets for clean cropping.
-- Label sections visually through grouping, not text.
-
-### Cohesion Checks
-Before finishing an asset sheet, verify:
-- All colors appear in the PICO-8 palette.
-- Perspective does not flip mid-sheet.
-- Lighting direction is consistent.
-- Edge treatment (dither, outline, anti-aliasing style) matches across assets.
-- No orphan tiles that don't connect to any neighbor.
-
-## Common Pitfalls
-- Mixing 8-bit and 16-bit detail levels.
-- Using outline style on some tiles but not others.
-- Forgetting palette constraint — colors outside the 16 PICO-8 colors.
-- Inconsistent tile edge connections (a grass tile that only connects to one dirt variant).

agent/managed-skills/podman-netavark-host-port-probe/SKILL.md

@@ -1,29 +0,0 @@
----
-name: podman-netavark-host-port-probe
-description: "Diagnose \"connection refused\" when a rootless-podman container tries to reach a host service port on this netavark host; proves the published-port-only quirk and picks the working path (host.containers.internal)."
----
-
-# Podman/netavark: container → host port connectivity probe
-
-Use when a containerized service on a rootless-podman + netavark host cannot reach a HOST process by LAN IP (connection refused immediately, not a timeout) even though `ss -ltnp` proves the listener is up in the host netns.
-
-## Why this happens
-On netavark-backed networks, container→host-LAN-IP traffic is only delivered for ports that have a live port-forward/DNAT (published ports). Un-published host ports refuse instantly — there is no firewall rule and no stale DNAT to find; `nft list tables`/`iptables -t nat` will be empty. Do not hunt for REJECT rules; probe instead.
-
-## Probe recipe (proves the quirk in one command)
-```bash
-ssh <host> "podman run --rm --network <container-network> --entrypoint sh docker.io/library/busybox:latest -c 'for p in <suspicious-port> <published-port>; do nc -w2 -z -v <host-lan-ip> \$p >/dev/null 2>&1 && echo port-\$p-OK || echo port-\$p-BAD; done'"
-```
-- Published port OK + target port BAD ⇒ published-port-only quirk confirmed.
-- Also probe the alias path from inside the actual service container:
-```bash
-podman exec <svc> bash -c 'exec 3<>/dev/tcp/host.containers.internal/<port> && echo ALIAS-OK || echo ALIAS-FAIL'
-```
-Note: busybox `nc -z` reports any failure as failure — use it for bulk port sweeps; use the `/dev/tcp` probe (or `nc -v`) when the refusal-vs-timeout distinction matters.
-
-## Fix decision
-- If the alias path connects and the LAN-IP path refuses → set the service's host-facing URL to `http://host.containers.internal:<port>` in its compose/quadlet env, then redeploy via the normal git flow (`deploy.sh <target> --service <svc>` recreates the container with fresh env).
-- Do NOT publish the daemon port just to make the LAN IP work — publishing is exactly what makes other ports refuse.
-
-## Post-fix verification
-`podman exec <svc> printenv <URL_VAR>` then tail logs for the connect confirmation (e.g. `Signal: connected`, `SSE: connected`) within ~40 s of recreation.

agent/managed-skills/podman-netns-addressability-probe/SKILL.md

@@ -1,51 +0,0 @@
----
-name: podman-netns-addressability-probe
-description: "Runbook to answer \"can container X actually reach service Y\" on a rootless-podman/netavark host, instead of assuming localhost works everywhere; includes the probe commands and the false-positive traps."
----
-
-# Podman netns addressability probe
-
-Answer **"can this container actually dial that service?"** per *network namespace*,
-not per host. On rootless podman + netavark, a bridge-net container cannot reach
-host-netns loopback listeners at all, so `localhost:PORT` reasoning is wrong for it.
-
-## Decision rule
-- `NetworkMode=host` → the container **is** the host netns: dials everything on
-  `127.0.0.1`, binds any free port.
-- bridge/named network → own netns. Only **published** ports are reachable, via the
-  network's **gateway IP**. Unpublished host listeners are **refused instantly (RST)**,
-  not routed — even when the listener is bound `0.0.0.0`.
-
-## Probe sequence (all read-only, run on the host)
-1. Namespace: `podman inspect <name> --format '{{.HostConfig.NetworkMode}}'`
-2. Owner of a listener: `sudo ss -ltnp 'sport = :<port>'` (host netns only — this is
-   why a `0.0.0.0` listener can still be unreachable from a bridge net).
-3. From inside the suspect container:
-   `podman exec <name> cat /proc/net/route` → default route = gateway IP to dial.
-4. Probe the candidate address **inside that netns**:
-   `podman exec <name> sh -c 'timeout 2 bash -c "exec 3<>/dev/tcp/<gw-ip>/<port>"; echo rc=$?'`
-
-## Interpretation (do not blur these)
-- **instant refusal** = nothing listening in a reachable namespace (or rootlessport
-  rejected an unpublished port map).
-- **timeout** = no route to that address at all. `host.containers.internal`
-  (`169.254.1.2`) is an alias on the **default `podman` net only**; containers on a
-  named network time out against it rather than being refused.
-
-## Probe-tool gotchas verified on this host (podman 5.x, netavark)
-- `podman network inspect <net> --format '{{json .containers}}'` **errors**
-  (`can't evaluate field containers in type interface {}`) — membership is not exposed
-  there. List from the container side instead:
-  `podman ps -a --format '{{.Names}} {{.Networks}}'`
-- An **empty** `Networks` field in that output = host network or a shared namespace
-  (`--network=container:<other>`), not "no network".
-- Quadlet `Network=<name>.network` yields runtime network name **`systemd-<name>`**;
-  probe with `podman network ls`, never by the `.network` filename.
-
-## Fix options when a bridge-net container must reach a host-netns listener
-(a) move it to `Network=host`, or (b) publish the port — (b) makes that port reachable
-from every namespace **and the LAN**, so it is an exposure change needing explicit
-operator approval, never a silent fix.
-
-Repo source of truth for this fleet: `serverAppConfigs-docs/Architecture/Container
-Network Namespaces.md` (commit `2e179b0`).

agent/managed-skills/podman-recreate-anon-volume-wipe/SKILL.md

@@ -1,28 +0,0 @@
----
-name: podman-recreate-anon-volume-wipe
-description: "Diagnose \"container data wiped after restart\" on this rootless-podman fleet: quadlet restarts recreate containers via podman run --replace --rm, and anonymous volumes reset to empty on every recreation. Includes the live audit procedure and the known affected services (hermes trio /opt/data, romm /romm DB, searxng cache)."
----
-
-# Podman recreation & anonymous-volume wipe audit
-
-## Mechanism (verified live 2026-08-31 on Dankclaw)
-A quadlet unit restart is NOT a start of the same container: the generated unit's ExecStart is
-`podman run --name systemd-%N --replace --rm ...`. Every `systemctl --user restart <unit>.service` (and every deploy that does one, e.g. `deploy_hermes`) **destroys and recreates** the container. Consequences at recreation time:
-
-- Writable-layer writes: gone by definition.
-- Anonymous volumes (image-declared `VOLUME` with no explicit `-v`/`Volume=` for that path): NOT carried across recreation — the new container gets a FRESH EMPTY one seeded only from image content. Orphans stay on disk until pruned (`podman volume prune` only removes unmounted ones).
-- Named volumes and bind mounts: persist (remounted by name/path).
-
-## Audit procedure
-1. `ssh <host> 'podman ps -a'` — map which host runs which stack (aliases `dankclaw`/`dankserver` match their hostnames).
-2. Per container: `podman inspect <name> --format "{{json .Mounts}}"` → classify each mount `"Type":"bind"` vs `"Type":"volume"` (hex-name volume = anonymous).
-3. Quirks: `podman ps --format` template field is `.Names`, not `.Name` (psReporter has no `Name` field — using `.Name` errors). Volume `_data` dirs under `~/.local/share/containers/storage/volumes/<id>/_data` are mode 0700 — unreadable by the plain ssh user; use `podman unshare`/sudo if content inspection is required.
-
-## Fleet state at audit time
-- **hermes trio** (`systemd-hermes`, `hermes-dashboard`, `hermes-webui`; gateway container auto-names `systemd-<unit>` because its unit lacks `ContainerName=`): `/data` = bind of `/home/dank/hermes-data` (durable state: config.yaml, sessions/, skills/, state.db) survives recreations; `/opt/data` (image default VOLUME + `HERMES_LAZY_INSTALL_TARGET`) is an anonymous volume → resets to image defaults on every deploy. So "Hermes lost data after restart" = anything written outside `/data`/`/webui` was reset, not the durable state.
-- **romm** (dankserver): runs with anonymous volumes at `/romm` and `/redis-data` → its DB lives in an anon volume; recreating romm (`podman-compose up -d` after image bump) boots it with a fresh empty DB. Fix = named volume or bind before next recreation.
-- **searxng**: `/var/cache/searxng` is an anonymous volume — cache only, benign reset on recreation.
-- Everything else on both hosts was verified as bind mounts (or stateless by design: authentik-server has no mounts at all; anubis is a memory store with client-side JWTs).
-
-## Fixes when data must survive recreation
-Pin a NAMED volume (`Volume=hermes-data:/opt/data` in the quadlet unit) or point the env var at a bind-mounted path. Never rely on anonymous volumes surviving recreation.

agent/managed-skills/porkbun-dns-ops/SKILL.md

@@ -1,26 +0,0 @@
----
-name: porkbun-dns-ops
-description: "Runbook for mutating Porkbun DNS records for franksdank.cloud via API from the target host, covering the relative-name edit quirk, rate-limit failure shape, and verify steps."
----
-
-# Porkbun DNS ops (franksdank.cloud)
-
-Operate the public DNS zone for `franksdank.cloud` via the Porkbun API v3, run **on Dankclaw** so the API key/secret sourced from `/home/dank/serverAppConfigs/.env` (`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) never leaves the host. Pipe scripts to the target with `ssh -o BatchMode=yes dankclaw 'bash -s' < /tmp/script.sh`.
-
-## Endpoints (all POST, JSON body)
-- List: `https://api.porkbun.com/api/json/v3/dns/retrieve/franksdank.cloud` — auth via headers `X-API-Key` + `X-Secret-API-Key` (Bearer is wrong here). Response shape is **top-level `.records[]`**, NOT `.content.records[]` as some docs claim — parse with `jq -r '.records[] | [.id,.name,.type,.content,(.ttl|tostring)] | @tsv'`.
-- Delete: `/dns/delete/{domain}/{id}` (body carries apikey+secretapikey).
-- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`, `type`, `content`, `ttl`, `prio`).
-
-## Quirks that bite (verified live 2026-08-31)
-1. **`name` is a RELATIVE label on edit.** Passing `ai.franksdank.cloud` yields the record `ai.franksdank.cloud.franksdank.cloud`. Pass the bare label (`ai`) and the API appends the zone itself. Verify after every rename with a retrieve before declaring done.
-2. **Rate limit answers as an HTML 404-style page, not JSON.** A batch of rapid mutations makes some calls return `<!DOCTYPE html>...` where jq then errors (`Cannot iterate over null` / `Invalid numeric literal`). Do not assume the call succeeded — sleep ~8 s and retry the failed ids; verify by re-listing the zone at the end.
-3. Deletes/renames are per-record-id: always retrieve first, map name→id, then mutate by id.
-
-## Rename procedure (vhost rename = DNS + Caddy together)
-1. `retrieve` → collect ids for the renamed names.
-2. `edit` each id with the new **relative** label, preserving type/content/ttl.
-3. Re-retrieve and confirm `<new>.franksdank.cloud A <ip>`.
-4. In-repo lockstep before deploy: rename the Caddyfile vhost block AND the Pi-hole `dankclaw/pihole/addn-hosts` line in the same commit — test 178 (`tests/pihole.bats`) fails the suite if any Caddyfile vhost lacks an addn-hosts A record on 192.168.0.143.
-5. Deploy via `scripts/deploy.sh dankclaw` (plans pihole+caddy from changed paths). Capture the suite's true exit with `out=$(./tests/run.sh 2>&1); rc=$?` — piping to `tail` masks the exit code behind `tail`'s own status and has shipped untested commits here.
-6. Verify live: `dig +short <name> @192.168.0.143` then `curl -o /dev/null -w %{http_code} https://<new-name>` (expect 200/302; first hit triggers dns-01 issuance, which works without inbound port 80).

agent/managed-skills/published-port-cutover-gate/SKILL.md

@@ -1,88 +0,0 @@
----
-name: published-port-cutover-gate
-description: "When replacing or restarting a service that publishes a TCP port on this rootless-podman/pasta fleet, gate on the port actually being free first — otherwise pasta's bind fails at container create and the failure hides behind Restart=on-failure."
----
-
-# Published-port cutover gate (rootless podman / pasta)
-
-On hosts where **rootless podman publishes ports through pasta**, the published
-port is bound in userspace at **container CREATE**, not when the app binds. A
-predecessor process that has just been stopped can still hold that port —
-listener, accepted connection, or TIME_WAIT residue from the connections it
-served — so a replacement started immediately after `systemctl stop` dies with:
-
-```
-pasta: ... bind: Address already in use
-conmon ... exit status 126
-```
-
-and because the unit is usually `Restart=on-failure`/`RestartSec=10`, it
-**self-heals seconds-to-minutes later**. That is the trap: the deploy looks
-like it failed, then looks fine, and the real defect (no port gate) stays
-invisible because the retry masks it.
-
-Observed on Dankclaw 2026-09-16 cutting `frank-large` from a root unit
-(ik_llama.cpp) to a rootless Quadlet (halogen): `systemd` logged
-"Deactivated successfully", and pasta tried to bind 7770 about a second later
-and reported "Address already in use". Only the quadlet's own restart policy
-recovered it.
-
-## Procedure
-
-Before starting the replacement:
-
-1. **Retire the old unit and wait for it to be genuinely inactive.**
-   `systemctl stop <old>` (root) or `systemctl --user stop <old>`.
-2. **Stop the NEW unit too, if this is a redeploy.**
-   `systemctl --user stop <new>.service || true`
-
-   Without this the gate deadlocks against itself: the currently-running
-   replacement is the thing holding the port. This mistake was made and caught
-   in the same session — a gate placed before the stop would have failed every
-   subsequent deploy of an already-running service.
-3. **Gate on the port being empty, covering every socket state:**
-
-   ```bash
-   wait_port_free() {
-       local port="$1" limit="${2:-90}" i
-       for ((i = 1; i <= limit; i++)); do
-           if [[ -z "$(ss -Htan "sport = :$port" 2>/dev/null)" ]]; then
-               return 0
-           fi
-           sleep 1
-       done
-       echo "FAIL: port $port is still held after ${limit}s:" >&2
-       ss -tan "sport = :$port" >&2 || true
-       return 1
-   }
-   ```
-
-   `sport` matches both the listener and the server side of accepted
-   connections, so this also catches TIME_WAIT residue that a plain
-   `ss -ltn` check misses. Fail loudly with the socket table attached; a deploy
-   that starts a doomed container has not deployed anything.
-4. **Then** install/pull and start, and wait on the service's health endpoint.
-
-## Verifying the gate has teeth
-
-- Re-run the same deploy: it must pass with the service already running (that
-  is the idempotency case the self-deadlock broke).
-- Reproduce the failure once on purpose if the service is not user-facing:
-  stop the container and start the replacement with the gate bypassed, and
-  confirm `Address already in use` / exit 126.
-- Assert it in the repo's seam tests: e.g. `grep -q 'wait_port_free <port>'`
-  and `grep -q '^wait_port_free() {'` in the deploy handler, so a later
-  refactor cannot quietly drop the gate.
-
-## Related traps
-
-- `systemctl disable` on some of these hosts **deletes the unit file** rather
-  than just the symlink, so a "move the unit to trash" step must tolerate a
-  file that is already gone — and should say so, or "no message" carries no
-  evidence.
-- Never `rm`; move retired units into trash (`$HOME/.AGENTTRASH/<date>-<topic>/`).
-- A rootless Quadlet's generated unit **cannot** be `systemctl --user enable`d;
-  boot persistence comes from `[Install] WantedBy=default.target` in the
-  `.container`. To keep a masked/failed quadlet from being regenerated, move
-  the `.container` out of `~/.config/containers/systemd/` and `daemon-reload` —
-  the generator's source is the lever, not `mask`.

agent/managed-skills/quadlet-v1-generator-quirks/SKILL.md

@@ -1,29 +0,0 @@
----
-name: quadlet-v1-generator-quirks
-description: "When writing or deploying Quadlet .container units on podman 5.x (v1 generator, e.g. Dankclaw), avoid keys the v1 generator hard-fails on and verify generated ExecStart before starting; generated units cannot be enabled."
----
-
-# Quadlet v1 generator quirks (podman 5.x)
-
-The podman 5.x user generator (`/usr/lib/systemd/user-generators/podman-user-generator`, quadlet **v1**) behaves differently from the v4 docs. Verified live on Dankclaw 2026-08-28 while deploying the authentik trio.
-
-## Hard-fail keys (unit silently fails to generate — `systemctl --user cat <unit>` shows "could not be found")
-- Any dash-prefixed option line in `[Container]`: `--shm-size=512m` etc. → generator logs `unsupported key '--shm-size' in group 'Container'` and skips the whole unit. Omit them; accept podman defaults.
-- `Command=` is unsupported — use `Exec=` (that is what this generator accepts).
-
-## HealthCmd space-splitting
-The v1 generator splits `HealthCmd` on spaces into the healthcheck argv array (`podman inspect --format {{.Config.Healthcheck.Test}} <ctr>` shows `[CMD-SHELL pg_isready -d ... ]`). A quoted probe like `/bin/sh -c "pg_isready ..."` lands as argv token `"pg_isready` → command-not-found → container reads **unhealthy while the DB is actually up**. Rule: HealthCmd must contain only space-free argv tokens, e.g. `HealthCmd=pg_isready -d authentik -U authentik` or `python /manage.py check --database default`.
-
-## Generated units cannot be enabled
-`systemctl --user enable <generated-unit>` fails with "unit is transient or generated". Boot persistence comes from `[Install] WantedBy=default.target` inside the `.container` file, honored by the generator at user-manager start; linger must stay on for the deploy user. Deploy handlers: install unit → `systemctl --user daemon-reload` → `systemctl --user start`; no `enable`.
-
-## Probe a new key before shipping it
-Drop a throwaway `probe.container` into `~/.config/containers/systemd/`, `systemctl --user daemon-reload`, then check the generator verdict:
-
-```bash
-journalctl --user --since "1 min ago" -o cat | grep -i "unsupported key"
-systemctl --user cat probe.service   # shows the generated ExecStart verbatim
-rm ~/.config/containers/systemd/probe.container && systemctl --user daemon-reload
-```
-
-If a healthcheck ever reads unhealthy but logs look fine, suspect this quirk first: compare the baked `Healthcheck.Test` array against the intended command before believing the status.

agent/managed-skills/radiance-frank-primary-benching/SKILL.md

@@ -0,0 +1,32 @@
+---
+name: radiance-frank-primary-benching
+description: "Bench frank-primary (radiance) decode/prefill via the scratch-container harness: method, noise floor, and the MNBT/HOST_POOL/envelope laws to respect before proposing any flag change."
+---
+
+# Benching frank-primary (radiance)
+
+## Method (works as of 2026-10-06)
+- Entry point: `bash autoresearch.sh` (dev box) → scp's `scripts/frank-primary.env` + `scripts/autoresearch-frank-primary.sh` to dankserver → the bench script ON dankserver: stops frank-primary (relinked unit: `stop`, never `mask`), boots scratch container `arbench` on the candidate env, polls /health (binds after full load, ~110 s cold), runs ONE greedy completion (fixed ~140k-char repo-corpus prompt, seed 1234 → 43,627 prompt tokens, 256 decode tokens, temp 0), parses the engine `timings` object, tears down, and restores prod via a trap.
+- Metrics from `timings`: `prompt_per_second` (cold prefill), `predicted_per_second` (decode at depth = (predicted_n-1)/predicted_ms), `prompt_ms` (TTFT), `draft_n`/`draft_n_accepted` (acceptance).
+- Fresh prefix-cache tier per run (`/data/kvcache/arbench`) so cold prefill is cold; prod's tier untouched.
+- Bench locally on dankserver (127.0.0.1) — no LAN jitter. Run-to-run ~4-5 min.
+
+## Noise floor (measurement law)
+- Decode: ~±3% on IDENTICAL config (33.06 / 32.03 / 30.90 observed). Single-run deltas <3% are not evidence; re-run before keeping.
+- Prefill: ~±2% (453/448/457 identical config).
+
+## Laws at the seqs=3 / maxlen=200k / fp8-KV / MTP-on shape (2×R9700, TP=2)
+- **MNBT is a sharp resonance at 2048**: 2560 → decode 2.6-7.7 t/s (acceptance stays normal, verify stalls ~130 ms/step, generation truncates); 1536 → decode 24.7 + 22% prefill loss + truncation; 4096 → boot failure at seqs=3 (boots at seqs=2, -2.6% only). DO NOT move MNBT off 2048.
+- **HOST_POOL_MIB 40960 is the load-bearing floor**: 36864 fails boot; 49152 boots but decode-flat.
+- **EXPERT_VS_CACHE 0.82 is bracket-confirmed**: 0.78 fails boot; 0.84 costs 8.9% (toward the documented 0.87 collapse cliff).
+- **TP_WIRE wht6 is mandatory** at seqs=3: bf16 (lossless) fails boot.
+- **SPEC_TOKENS**: depths 1-3 statistically tied on decode; 0 = -6.8% (real), 4 = -22.5% (acceptance collapses to 0.50). Depth 2 nominal peak.
+- MAX_MODEL_LEN 262144 (native) costs 2.5% — the larger KV pool steals resident-weights VRAM at expert_tiered.
+- GPU_HEADROOM 48 vs 96: no effect.
+
+## Hazard: abandoned bench contamination
+- If a prior bench run is skipped/abandoned mid-flight, the next scratch boot may race the leftover engine and produce garbage numbers. Before trusting any collapse measurement: `podman ps -a` on dankserver — confirm the previous arbench is gone and frank-primary is active — then reproduce the collapse on a clean boot before writing it into the record.
+
+## State
+- HEAD on the autoresearch branch: spec=2, seqs=3, maxlen=200k (df6b5bb6e807). All spec 1-3 × seqs 2-3 configs decode within 30.9-33.1 — statistically tied. Frank's decision pending: keep spec=2/seqs=3 or revert to his spec=3/seqs=2 (83% acceptance, likely better for real multi-turn traffic).
+- Production verified healthy (health 200) after every run via the harness trap.

agent/managed-skills/radiance-precision-selection/SKILL.md

@@ -0,0 +1,31 @@
+---
+name: radiance-precision-selection
+description: "Choose tensor precisions for a radiance (StillDeadcode) rad-convert recipe on RDNA4: what is native, what is supported, what the hardware rewards."
+---
+
+# Radiance precision selection
+
+Use when writing/choosing a `rad-convert` recipe for radiance on RDNA4 (gfx1201), or when arguing why a model was converted a particular way.
+
+## Project map (which repo is authoritative)
+- `codeberg.org/StillDeadcode/radiance` — standalone engine (C++/HIP). Recipes: `data/recipes/*.recipe`; docs: `docs/{GUIDE,MOE-W4,KERNELS,PLUGIN,TOOLS,OPS,ARCHITECTURES,DOCKER}.md`. Prefer the RAW form of Codeberg URLs when reading: `/raw/branch/main/<path>`.
+- `codeberg.org/StillDeadcode/libr4d` — kernel library; entry points encode exact geometry and reject mismatch (`attn_decode_h256_gqa6_fp8kv`, `gdn_chunk_scan_k128_v128_c64_bf16`, `ar_oneshot_2rank_exact`). Kernel names are the ground truth for which dtypes have a fast path.
+- `codeberg.org/StillDeadcode/vllm-radiance` — deprecated vLLM wrapper; README still carries the best prose on the FP8 dispatcher and kernel list.
+
+## Precision ladder
+| Precision | Role |
+|---|---|
+| block-FP8 (E4M3 + bf16 block scales) | native tuned GEMM/attention/KV — "preshuffled block-FP8 dispatcher" |
+| MXFP4 (OCP group-32) / W4A8 | supported weight packing; activations dynamically quantized to FP8 E4M3, compute is FP8 WMMA |
+| bf16 | KV (`auto`), skinny GEMMs, Gated-DeltaNet kernels |
+| int8 / int4 | draft head and integer GEMM paths |
+
+## Decision rules
+- Prefer block-FP8 for anything the engine qualifies: generic upstream AITER linear in place of the FP8 dispatcher measured **9–11 % slower**; all upstream-qualified models are FP8 (Qwen3.8-27B-FP8, Qwen3.6-27B-FP8, Qwen3.6-35B-A3B-FP8, Gemma-4-31B-it-FP8).
+- Weight packing is a *storage* choice, activation dtype is the *compute* choice: MXFP4 buys bytes, not an FP4 compute path.
+- The matrix core is integer-biased: f16 WMMA is half the int8 rate and a quarter of the int4 rate ⇒ int4 > int8 > f16. FP8/INT8 kernels must fuse double-K WMMA to saturate the 128-bit memory interface (GPUOpen WMMA guide part 2) — a "slow" quantized kernel usually means a missing K-fused variant, not an inherent dtype cost.
+- Keep low-bit tricks on the draft head (int4/int8) where verification tolerates them; keep lm_head and hard-selection tensors (QSA indexer top-k) at high precision — the shipped recipes quantify the cost (int8 lm_head beats E4M3 by ~0.6 top-1 points).
+
+## Verification
+- Read the recipe's own header comments: radiance recipes state measured KL / top-1 / prefill-decode deltas for their choice, and name the `--reuse` inputs.
+- Confirm the kernel exists before promising a dtype: grep libr4d entry points for the dtype suffix (`fp8kv`, `bf16`, `exact`).