Add blankpad-host auto-learn skills and memory store
Tracked the harness artifacts accumulated on blankpad (blank@), matching the existing convention for the --home-dank-* memory stores and the 36 already-tracked managed-skills. managed-skills (9): authentik-proxy-provider-app-link, dankclaw-guard-chain-layer-diagnosis, e2e-arr-rawip-reachability-triage, halogen-kv-pool-fit, lan-client-split-dns-fix, noctalia-reference, rootless-quadlet-log-retrieval, ssh-key-bootstrap-no-sshpass, swapfile-resize-safe-ordering. memories (2 stores): --home-blank-.omp-- (dotfiles_git, handy_lifecycle, input_permissions) and --home-blank-Projects-serverAppConfig-- (frank_large_quadlet). Secret sweep before staging: no PEM blocks, no credential literals. Content is fleet topology/IPs only, same class as the tracked dank stores.
This commit is contained in:
Frank committed
1 parent
b60e95158c
commit
4b82f35354
37 files changed
+669
No files matched your search
@@ -0,0 +1,52 @@
|
||||
---
|
||||
name: authentik-proxy-provider-app-link
|
||||
description: "Diagnose and fix an authentik embedded-outpost proxy provider that never authenticates (login wall / raw flow served on the app host): the provider must be linked to an Application, have a non-null invalidation_flow, and an exact external_host match."
|
||||
---
|
||||
|
||||
# authentik proxy provider not authenticating (login wall)
|
||||
|
||||
Symptom: a service behind the authentik embedded outpost (Caddy -> Anubis -> outpost -> app) never lets an authenticated user through. The browser is parked on a login page served ON THE APP'S OWN HOST, e.g. `https://app.example.com/if/flow/default-authentication-flow/?next=/` or `https://app.example.com/flows/-/default/authentication/`. The IdP session cookie lives on `auth.example.com` and is never sent to `app.example.com`, so login is demanded forever.
|
||||
|
||||
## Root causes (check in this order)
|
||||
|
||||
1. **Provider not linked to an Application (most common).** The embedded outpost's config API filters `ProxyProvider.objects.filter(application__isnull=False)`. A provider created with `ProxyProvider.objects.create(...)` and no Application is never returned to the outpost, so the outpost never loads it and the request falls through to raw flow serving.
|
||||
- FIX: create an Application bound to the provider:
|
||||
```python
|
||||
app = Application.objects.create(name="Marreta", slug="marreta", launch_url="https://marreta.franksdank.cloud")
|
||||
app.provider = proxy_provider; app.save()
|
||||
proxy_provider.application = app; proxy_provider.save()
|
||||
```
|
||||
2. **Null `invalidation_flow`.** The Rust outpost serde requires a formatted UUID; a null value aborts `get_outpost` with `invalid type: null, expected a formatted UUID string at line 1 column N` and takes down EVERY provider on that outpost.
|
||||
- FIX: `proxy_provider.invalidation_flow = InvalidationFlow.objects.get(name="default-provider-invalidation-flow")`.
|
||||
3. **`external_host` mismatch.** Must equal the Host header the outpost sees. Trailing-slash mismatch (`https://x` vs `https://x/`) = no match. Match the fleet convention (no trailing slash).
|
||||
4. **Missing `authentication_flow` / `authorization_flow`.** Set to the fleet defaults (`default-authentication-flow`, `default-provider-authorization-implicit-consent`).
|
||||
|
||||
## Diagnose
|
||||
|
||||
```bash
|
||||
# 1. Compare broken vs working provider field set (ak shell, as the authentik user):
|
||||
sudo -u authentik AUTHENTIK_POSTGRESQL_HOST=127.0.0.1 AUTHENTIK_POSTGRESQL_PORT=5432 \
|
||||
AUTHENTIK_POSTGRESQL_NAME=authentik AUTHENTIK_POSTGRESQL_USER=authentik \
|
||||
AUTHENTIK_POSTGRESQL_PASSWORD='<pw>' ak shell
|
||||
```
|
||||
```python
|
||||
for p in ProxyProvider.objects.all():
|
||||
print(p.name, p.external_host, bool(p.application), p.authentication_flow_id, p.authorization_flow_id, p.invalidation_flow_id)
|
||||
```
|
||||
```bash
|
||||
# 2. The serde error (this is the smoking gun for cause #2):
|
||||
journalctl --user -u authentik-outpost.service --since '-10 min' | grep -iE 'error|invalid|panic'
|
||||
# 3. Confirm the outpost loaded the provider after the fix:
|
||||
journalctl --user -u authentik-outpost.service --since '-2 min' | grep -i 'loaded app'
|
||||
```
|
||||
|
||||
## Apply + restart
|
||||
|
||||
After any provider/application change: `systemctl --user restart authentik-server.service authentik-outpost.service` (rootless, user dank on Dankclaw). Then verify the OAuth dance:
|
||||
```bash
|
||||
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' https://app.example.com/
|
||||
# expect 302 -> /outpost.goauthentik.io/start?rd=... (NOT a raw /flows/ URL)
|
||||
```
|
||||
|
||||
## Provenance note
|
||||
On the Dank fleet these providers are NOT repo-managed (no blueprint). Fixes applied via `ak shell` are live-only. Capture any new provider+application as a repo blueprint or setup script to avoid drift.
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
name: dankclaw-guard-chain-layer-diagnosis
|
||||
description: "When a user reports a guarded dankclaw page (homepage/vhost) broken but the server may be healthy: verify each layer of the Anubis→Authentik→Caddy chain server-side and isolate client-side vs server-side."
|
||||
---
|
||||
|
||||
# Dankclaw Guard-Chain Layer Diagnosis
|
||||
|
||||
Use when a user reports a guarded page (e.g. `www.franksdank.cloud` homepage) "broken / links gone / not loading" and you must determine whether the fault is server-side or client-side/session.
|
||||
|
||||
## Chain topology
|
||||
`Browser → Caddy TLS edge → Anubis :8923 → Caddy :8016 → Authentik outpost → Caddy :8017 (@home / @data handlers) → app/static`
|
||||
- Public vhosts `reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}`.
|
||||
- Post-auth `http://:8017` listener routes by `@<svc> host <vhost>`.
|
||||
- Homepage static: `/srv/homepage/` (index.html, app.js, style.css, services.json, status.json, deployed.js).
|
||||
|
||||
## Layer-by-layer checks (run from the host via ssh)
|
||||
1. **Caddy unit + guard containers**: `systemctl --user is-active caddy.service`; `podman ps --format '{{.Names}}\t{{.Status}}' | grep -iE 'anubis|authentik|caddy'` — expect `Up ... (healthy)`.
|
||||
2. **Anubis challenge page validity**: `curl -sk https://www.franksdank.cloud/ --resolve www.franksdank.cloud:443:192.168.0.143` — a healthy challenge contains `"anubis_challenge"` JSON with `difficulty` and the main JS. If present, the guard serves fine.
|
||||
3. **Authentik outpost errors**: `podman logs --tail 40 authentik-outpost 2>&1 | grep -iE 'error|exception|traceback|500|provider'` — routine dramatiq task logs are normal.
|
||||
4. **@home / @data from INSIDE the caddy container** (the only way to reach 8016/8017):
|
||||
`podman exec caddy sh -c 'wget -q -S -T5 --header "Host: www.franksdank.cloud" -O- http://127.0.0.1:8017/'`
|
||||
and the same for `/services.json`, `/status.json`. Expect 200 + body.
|
||||
5. **Files on disk**: `ls -la /srv/homepage/` and validate `status.json` parses (`python3 -c "import json;json.load(open('/srv/homepage/status.json'))"`).
|
||||
|
||||
## Gotchas (cost me calls)
|
||||
- **BusyBox wget in the caddy container**: NO `-H` flag. Use `--header "Host: ..."`.
|
||||
- **8016/8017 live in the caddy container's netns**: `curl http://127.0.0.1:8017` from the HOST returns `000` (connection refused) — EXPECTED, not a fault. Test from inside the container.
|
||||
- **anubis container is distroless**: `podman exec anubis sh` fails (`sh: executable not found`). Inspect via its logs / the served challenge page instead.
|
||||
- **x-real-ip masking**: Anubis logs `x-real-ip: 10.89.1.22` (Caddy container podman IP) for ALL clients — rootless-podman published-port SNAT. Pre-existing; clients indistinguishable by IP; clearance is cookie-based. Do NOT mistake this for the bug.
|
||||
- **Homepage render**: `app.js` does `Promise.all([fetch services.json, fetch status.json])`; on ANY fetch failure the catch sets `#summary` to `status unavailable: <err>` and the grid stays EMPTY — which a user reads as "links gone / not loading." A 30-service status.json with a 9-service services.json renders fine (stateByName lookup).
|
||||
|
||||
## Decision
|
||||
If every layer above is healthy (challenge valid, outpost clean, @home/@data 200, files render), the fault is client-side/session. STOP thrashing the live system (repeated failed browser challenges share the masked 10.89.1.22 identity). Get the authenticated view: ask for the exact symptom (challenge spinning? Authentik login loop? empty grid? "status unavailable" text?) or the Authentik login to drive the page in a real browser.
|
||||
@@ -0,0 +1,72 @@
|
||||
---
|
||||
name: e2e-arr-rawip-reachability-triage
|
||||
description: "Triage an e2e admin-SSO failure on a *arr raw IP (192.168.0.112:8989) by probing from the box the browser runs on, instead of assuming the Mac is asleep and burning retries waking it."
|
||||
---
|
||||
|
||||
# e2e raw-IP *arr failures — probe from the browser's box, not a peer
|
||||
|
||||
`tests/e2e/specs/11-admin.spec.ts` navigates the real browser to
|
||||
`http://192.168.0.112:8989/` (Sonarr's forwarded IP on FrankBook-Pro). When it
|
||||
fails with `ERR_CONNECTION_TIMED_OUT` the instinct is "the Mac is asleep".
|
||||
Sometimes true. Not always — and waking it is not the only recovery.
|
||||
|
||||
## The decisive question
|
||||
|
||||
**Who cannot reach the port?** Playwright runs on the dev box (blankpad), not on
|
||||
a fleet host. A healthy probe from dankclaw proves nothing about the browser's
|
||||
path.
|
||||
|
||||
```bash
|
||||
# 1. From the box that RUNS the browser:
|
||||
curl -s -o /dev/null -w '%{http_code}\n' --max-time 7 http://192.168.0.112:8989/
|
||||
|
||||
# 2. From a LAN peer, same port:
|
||||
ssh dankclaw 'for p in 8989 7878 9696; do curl -s -o /dev/null -w "$p %{http_code}\n" \
|
||||
--max-time 6 http://192.168.0.112:$p/; done'
|
||||
|
||||
# 3. Liveness of the Mac itself (ICMP, not TCP):
|
||||
ping -c 2 -W 3 192.168.0.112
|
||||
```
|
||||
|
||||
## Reading the matrix
|
||||
|
||||
| peer probe | blankpad probe | ping | meaning |
|
||||
|---|---|---|---|
|
||||
| 200 | 200 | ok | transient — re-run the spec |
|
||||
| timeout | timeout | fails | Mac asleep; wake, ports return in seconds |
|
||||
| **200** | **rc=28** | **ok** | **path problem, not sleep** — ICMP allowed, TCP to those ports blocked. Waking and re-running cannot help |
|
||||
|
||||
## If it is the third row, confirm the Mac is not at fault
|
||||
|
||||
Through the dankclaw jump (`ssh dankclaw "ssh dank@192.168.0.112 ..."`):
|
||||
|
||||
```bash
|
||||
pgrep -fl socat # forwarders bound to 192.168.0.112:port
|
||||
netstat -an | grep -c 8989 # listening
|
||||
curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8989/ # 200 = app fine
|
||||
/usr/libexec/ApplicationFirewall/socketfilterfw --getglobalstate
|
||||
```
|
||||
|
||||
Forwarders up + loopback 200 + `Firewall is disabled` ⇒ the Mac is innocent. The
|
||||
block is between the dev box and the Mac, so stop optimising the Mac.
|
||||
|
||||
## What to do instead
|
||||
|
||||
Don't hold a deploy hostage to a known-environmental red gate:
|
||||
|
||||
1. Verify the deployed service **directly** — its own ledger/health/real
|
||||
completion — and state that the e2e gate was skipped or partially red, and
|
||||
why (`deploy.sh --skip-e2e` prints exactly that warning, so the skip is
|
||||
recorded, not hidden).
|
||||
2. Report the residual gap plainly: *"29/30 e2e pass; 11-admin fails on Sonarr's
|
||||
raw IP from the dev box; peer probe returns 200, so the deploy is verified by
|
||||
other means and the gate gap is a dev-box network path issue."*
|
||||
3. Do not weaken or skip the spec to make it green — it is guarding a real
|
||||
SSO-plus-RBAC contract.
|
||||
|
||||
## Note on the sleep case
|
||||
|
||||
The classic flake is real: with the Mac asleep, ports refuse from **everywhere**
|
||||
including peers, ping latency runs ~150 ms, and an SSH/ping wake restores
|
||||
200/302 within seconds. That is the cheap first check — just don't stop there
|
||||
when the peer probe is already green.
|
||||
@@ -0,0 +1,90 @@
|
||||
---
|
||||
name: halogen-kv-pool-fit
|
||||
description: "Resize halogen-flash-server's KV pool or prefill arena without getting silently clamped or OOMing the boot: pool, arena and vision share one device budget, HALOGEN_HOST_RESERVE_GIB only scales the fit-check ceiling and is NOT an allocation, HALOGEN_KV_POOL_FIT only ever lowers, and the sizing warning prints only when a clamp happened."
|
||||
---
|
||||
|
||||
# Sizing a halogen-flash-server KV pool
|
||||
|
||||
`HALOGEN_KV_POOL_FIT=1` (default) computes a device budget at startup and **only ever lowers** the configured `HALOGEN_KV_POOL_POSITIONS`. It never raises it, and never below the context.
|
||||
|
||||
```
|
||||
device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB
|
||||
fit check = needs budget >= (KV + arena + vision) + 1.5 GiB margin
|
||||
KV = 28 KiB/position
|
||||
arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384) # measured, linear
|
||||
vision = 1.97 GiB when HALOGEN_VISION_TOWER=1
|
||||
```
|
||||
|
||||
Prefer the engine's printed requirement over the model: a clamped boot states
|
||||
it directly, e.g. `524288 positions need ~35.0 GiB (plus 1.5 of margin)`.
|
||||
|
||||
## The reserve is NOT an allocation
|
||||
|
||||
`HALOGEN_HOST_RESERVE_GIB` is a term in the fit-check inequality, nothing more. Proved by A/B: reserve 15 and reserve 16 both held **103.5 GiB** with **7.1 GiB left for everything else** — identical footprint. Actual footprint is `weights + KV + arena + vision`, none of which the reserve touches.
|
||||
|
||||
So the knob has only two regimes: below the pool's requirement it changes nothing observable; above it, it silently deletes the pool. **Raising the reserve can never give memory back to co-tenants.** To free host RAM, cut the pool or the arena:
|
||||
|
||||
| change | RAM handed back |
|
||||
|---|---|
|
||||
| `HALOGEN_KV_POOL_POSITIONS` 524288 → 262144 | 7.2 GiB |
|
||||
| `HALOGEN_MAX_TOK` 32768 → 16384 | 8.4 GiB |
|
||||
|
||||
Lowering the reserve DOES work — it buys fit margin for a bigger pool or arena. On a 121.2 GiB host with 67.7 GiB weights, for 524288 + max_tok 32768 (needs 36.5):
|
||||
|
||||
| reserve | budget | vs 36.5 | outcome |
|
||||
|---|---|---|---|
|
||||
| 24 | 29.5 | −7.0 | clamps |
|
||||
| **18** | **35.5** | **−1.0** | **clamps — looks like a healthy deploy** |
|
||||
| 17 | 36.5 | +0.02 | knife edge, inside the noise of the weights term |
|
||||
| 16 | 37.5 | +1.0 | fits, zero benefit over 15 |
|
||||
| **15** | **38.5** | +2.0 | **deployed** |
|
||||
|
||||
Ceiling here is 16; pick the lowest reserve that fits, not the highest.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **The sizing block and its `WARNING` print ONLY after a clamp.** A fitting boot prints no `kv pool:` lines at all, so absence of a warning in old logs is NOT evidence of headroom.
|
||||
- **`deploy.sh` health-passes a clamped pool.** Container `active`, `/health` 200, in ~12 s. Only the ledger proves the pool size.
|
||||
- **Beware the false positive when grepping for a clamp**: the NOTE line contains the sentence "Lowering HALOGEN_KV_POOL_POSITIONS gives some back", which matches `LOWERING` case-insensitively. Grep `LOWERING THE POOL` / `cannot spare it`.
|
||||
- **`free`/`MemAvailable` cannot see this server.** Weights are pinned and the kernel counts locked pages as reclaimable page cache — `MemAvailable` read ~74 GiB while the engine held 103.5 of 121.2 GiB. Read the engine's lines instead.
|
||||
- **Host RAM is rounded DOWN when displayed.** `121` is `MemTotal 127104060 kB` = `121.2159`; subtracting displayed figures gives 29.3 instead of the true 29.5.
|
||||
- **Vendor "on the device" figures are pool + arena totals, not pool.** Reading them as pool-only makes a doubling look affordable.
|
||||
- **A 2.0 GiB firmware iGPU carve-out** is reported by the ledger and is not free memory.
|
||||
- `HALOGEN_MAX_TOK` is the arena / per-request generation budget (thinking included) — do not conflate with `HALOGEN_CTX` or the pool. `HALOGEN_KV_SLOTS` costs only ~111 MiB/slot, irrelevant to sizing.
|
||||
|
||||
## Three failure modes, worst last
|
||||
|
||||
1. **Silent clamp** — need exceeds budget; pool drops to the floor, service healthy. Caught only in the ledger.
|
||||
2. **Boot OOM** — enough `HALOGEN_MAX_TOK` and the arena alone beats the budget: `dmalloc: FAILED requesting 0.250 GiB after 39.970 GiB in 566 allocations (out of memory)`. Container stays `active / Up (starting)` forever; health never passes.
|
||||
3. **Minutes-long stall** — reserve cut so far that the engine holds most of a shared host. It prints its own note: neighbours and this server "can stall for minutes at a time at full CPU with no disk activity and no output, which is not a crash and needs no restart." Looks like a hang, invisible to `free`, hits co-tenants too.
|
||||
|
||||
## Verify
|
||||
|
||||
```
|
||||
podman logs frank-large 2>&1 | grep -aiE 'for the device|reserving the KV|ONE pool of|memory: [0-9.]+ GiB of weights|left for everything|LOWERING THE POOL|cannot spare it|listening on 127'
|
||||
```
|
||||
|
||||
Expect the budget line, `reserving the KV pool, <N> positions`, `ONE pool of <N> positions`, zero clamp matches, a `host memory left for everything else` figure, then `listening on 127.0.0.1:8730`. (A failing 7770 publish on restart is a stale-netns race — self-heals; `podman network reload --all` forces it.)
|
||||
|
||||
Then a real completion — `/health` 200 alone proves nothing:
|
||||
|
||||
```
|
||||
curl -s http://127.0.0.1:7770/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"frank-large","messages":[{"role":"user","content":"Reply with exactly: OK"}],"max_tokens":48,"reasoning_effort":"low"}'
|
||||
```
|
||||
|
||||
Expect `finish_reason: stop`. A silent client decodes **greedy** — this server ships no sampler defaults.
|
||||
|
||||
## Reference points (121.2 GiB host, 67.7 GiB weights, vision on)
|
||||
|
||||
| pool | max_tok | reserve | need | budget | outcome |
|
||||
|---|---|---|---|---|---|
|
||||
| 262144 | 32768 | 24 | 29.8 | 29.5 | boots (0.3 over), no sizing lines |
|
||||
| 524288 | 16384 | 24 | 28.6 | 29.5 | fits clean |
|
||||
| 524288 | 32768 | 24 | 37.0 | 29.5 | clamped, deploy green |
|
||||
| 524288 | 65536 | 24 | 46.5 | 29.5 | OOM, will not boot |
|
||||
| 524288 | 32768 | 15 | 37.0 | 38.5 | **both features, deployed** |
|
||||
| 524288 | 32768 | 16 | 37.0 | 37.5 | fits, footprint identical |
|
||||
|
||||
## Procedure
|
||||
|
||||
Compute `need` vs `budget` with unrounded RAM, then confirm against the engine's own printed requirement. After each deploy read the ledger for actual pool, clamp matches, **and the host-left line** — and A/B it: if two settings report the same `held`, the knob you turned doesn't do what its name says. Keep reserve and pool documented as a matched pair. Roll back with `git revert` (never `--no-verify`).
|
||||
@@ -0,0 +1,36 @@
|
||||
---
|
||||
name: lan-client-split-dns-fix
|
||||
description: Fix a LAN client timing out on *.franksdank.cloud because it resolves the public WAN IP (no 443 forward) instead of the Pi-hole LAN answer; diagnose by resolver and fix with a systemd-resolved route-only drop-in.
|
||||
---
|
||||
|
||||
# LAN client split-DNS fix (*.franksdank.cloud)
|
||||
|
||||
Symptom: a LAN machine gets `connection timeout` (curl rc=28) on `https://*.franksdank.cloud`, or LAN-only names return NXDOMAIN, while the server is actually healthy.
|
||||
|
||||
## Root cause
|
||||
Public DNS for `franksdank.cloud` = WAN IP `71.227.227.172`, which has **no 443 port-forward** (filtered). Pi-hole `192.168.0.143` returns the LAN answer `192.168.0.143` via `addn-hosts`. If the client's resolver is using an external server (1.1.1.1/quad9), it gets the WAN answer → SYN to a filtered port → timeout.
|
||||
|
||||
## Diagnose (fast)
|
||||
```bash
|
||||
dig +short www.franksdank.cloud @192.168.0.143 # expect 192.168.0.143
|
||||
dig +short www.franksdank.cloud @1.1.1.1 # expect 71.227.227.172 (the trap)
|
||||
resolvectl status | grep -iE 'Current DNS|DNS Server' # which server is active?
|
||||
# prove the server is fine by pinning the LAN IP:
|
||||
curl -sk --resolve www.franksdank.cloud:443:192.168.0.143 https://www.franksdank.cloud/ -o /dev/null -w '%{http_code}\n' # 200
|
||||
```
|
||||
|
||||
## Fix (systemd-resolved route-only drop-in)
|
||||
```bash
|
||||
sudo mkdir -p /etc/systemd/resolved.conf.d
|
||||
sudo tee /etc/systemd/resolved.conf.d/franksdank-lan.conf >/dev/null <<'EOF'
|
||||
[Resolve]
|
||||
DNS=192.168.0.143
|
||||
Domains=~franksdank.cloud ~franksdank.casa
|
||||
EOF
|
||||
sudo systemctl restart systemd-resolved
|
||||
resolvectl flush-caches
|
||||
```
|
||||
The `~` (route-only) prefix sends `*.franksdank.cloud` ONLY to Pi-hole and never to the default-route servers. Verify: `getent hosts www.franksdank.cloud` → 192.168.0.143.
|
||||
|
||||
## Note
|
||||
This is a per-client fix. Every LAN client that must reach the homelab by name needs Pi-hole as its DNS (or this drop-in). The server side needs no change.
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
name: noctalia-reference
|
||||
description: "Quick reference for Noctalia desktop environment: config paths, IPC commands, app templates, palette, and migration checklist for Niri-based setups."
|
||||
---
|
||||
|
||||
# Noctalia Desktop Environment
|
||||
|
||||
Noctalia is a Niri-based desktop environment at `/usr/bin/noctalia`.
|
||||
|
||||
## Config
|
||||
- Main: `~/.config/noctalia/config.toml`
|
||||
- Niri config: includes `noctalia.kdl` for display settings
|
||||
|
||||
## IPC
|
||||
Direct `noctalia msg <command>` replaces `qs -c noctalia-shell ipc call` wrapper.
|
||||
Key commands: `panel-toggle`, `session`, `lock`, `launcher`.
|
||||
|
||||
## App Templates
|
||||
`/usr/share/noctalia/assets/templates/` contains: gtk, ghostty, alacritty, hyprland, sway, wezterm, helix, btop, starship, qt, kde, cava, scroll, mango, umbriel, foot, emacs, kitty.
|
||||
|
||||
## Palette
|
||||
Generate from image: `noctalia theme [image]`
|
||||
|
||||
## Migration Checklist
|
||||
- [ ] Niri keybinds → `noctalia msg` commands
|
||||
- [ ] Alacritty → import noctalia.toml theme
|
||||
- [ ] Ghostty → `theme = noctalia`
|
||||
- [ ] GTK 3/4 → update accent palette (#b267e4 purple)
|
||||
|
||||
## Text Antialiasing
|
||||
- **Ghostty**: `font-subpixel-render = "off"` to disable subpixel AA
|
||||
- **Alacritty**: `font.antialias = false` in `alacritty.toml`
|
||||
@@ -0,0 +1,42 @@
|
||||
---
|
||||
name: rootless-quadlet-log-retrieval
|
||||
description: "Retrieve logs for a rootless Podman Quadlet service on the Dank fleet — which command covers which time window, and the sudo/per-user-storage trap"
|
||||
---
|
||||
|
||||
# Rootless Quadlet Log Retrieval (Dank fleet)
|
||||
|
||||
Applies to any service deployed as a **rootless** Quadlet `.container` unit (frank-large on Dankclaw is the reference case). Rootless storage is per-user, so the deploy user matters more than the container name.
|
||||
|
||||
## Step 0 — establish ownership before choosing a command
|
||||
|
||||
```bash
|
||||
# Which manager owns the unit, and which user runs it?
|
||||
systemctl --user status <name>.service # works => rootless user unit
|
||||
loginctl show-user <user> -p Linger # Linger=yes => starts at boot
|
||||
```
|
||||
|
||||
If `systemctl --user` knows the unit, **every** subsequent command runs as that user with NO `sudo`.
|
||||
|
||||
## Step 1 — pick the log source by time window
|
||||
|
||||
| Need | Command |
|
||||
|---|---|
|
||||
| Current instance, tail | `podman logs --tail 200 <name>` |
|
||||
| Current instance, follow | `podman logs -f <name>` |
|
||||
| Across restarts/recreates | `journalctl --user -u <name>.service -n 200 --no-pager` |
|
||||
| Time-bounded | `journalctl --user -u <name>.service --since "14:30"` |
|
||||
| Crash/restart signal only | `journalctl --user -u <name>.service -p warning` |
|
||||
| Kernel-side evidence (OOM, SVM) | `sudo journalctl -k --since ...` |
|
||||
|
||||
Quadlet units commonly run `--replace --rm`. Consequence: **`podman logs` is destroyed when the container is recreated**; the journal captured `podman run`'s stdout and is the only durable record. When the question is "what happened before the last restart", go straight to `journalctl --user`.
|
||||
|
||||
## Step 2 — the traps
|
||||
|
||||
- `sudo podman logs <name>` → `Error: no such container`. Root's store does not contain rootless containers. Same for `sudo podman ps`.
|
||||
- A system-level `journalctl -u <name>` returns nothing for a user unit — the `-u` must be paired with `--user`.
|
||||
- During a first-start weight download, health checks report unhealthy **by design**. Do not read that as a crash; check whether the payload file is still growing.
|
||||
- Vendor verbose switches (e.g. `HALOGEN_VERBOSE=1`, `HALOGEN_DMALLOC_LOG=1`) are a **config change**: they go through git and `deploy.sh`, never a live env poke on the host.
|
||||
|
||||
## Step 3 — if you cannot reach the host
|
||||
|
||||
No SSH transport is configured on the `blank` dev box. Do not probe `dank@`/`frank@` and hope. Ask Frank for a key or alias, and label any answer derived only from repo files as **not live-verified**.
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
name: ssh-key-bootstrap-no-sshpass
|
||||
description: "Bootstrap passwordless SSH from a dev box to fleet hosts when sshpass/expect are unavailable, using OpenSSH SSH_ASKPASS_REQUIRE=force so the password never touches disk."
|
||||
---
|
||||
|
||||
# Passwordless SSH bootstrap without sshpass/expect
|
||||
|
||||
Use when you must install your pubkey on remote hosts but `sshpass` and `expect`
|
||||
are not installed, and you only have a password. Modern OpenSSH (>= 8.4) can
|
||||
drive the password prompt from a helper program, so the password flows through
|
||||
the **process env only** — never a file.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. **Generate a key** (if none):
|
||||
```bash
|
||||
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N "" -C "host-deploy-$(date +%Y%m%d)"
|
||||
```
|
||||
|
||||
2. **Create an askpass helper** that echoes the password from an env var
|
||||
(the helper itself stores NO secret):
|
||||
```bash
|
||||
printf '#!/bin/sh\necho "$DEPLOY_PW"\n' > /tmp/askpass.sh
|
||||
chmod 700 /tmp/askpass.sh
|
||||
```
|
||||
|
||||
3. **Copy the pubkey**, forcing the askpass path and detaching from the TTY
|
||||
with `setsid` (ssh only uses SSH_ASKPASS when there is no controlling TTY,
|
||||
unless `SSH_ASKPASS_REQUIRE=force`):
|
||||
```bash
|
||||
env DEPLOY_PW='<password>' \
|
||||
SSH_ASKPASS=/tmp/askpass.sh \
|
||||
SSH_ASKPASS_REQUIRE=force \
|
||||
setsid ssh-copy-id -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 user@host
|
||||
```
|
||||
Pass the password via the `env` argument (transient), never in a committed
|
||||
file or shell history.
|
||||
|
||||
4. **Add host aliases** in `~/.ssh/config` so deploy scripts resolve names:
|
||||
```
|
||||
Host myhost
|
||||
HostName 10.0.0.5
|
||||
User deployuser
|
||||
IdentityFile ~/.ssh/id_ed25519
|
||||
IdentitiesOnly yes
|
||||
```
|
||||
|
||||
5. **Verify** key auth with no prompt:
|
||||
```bash
|
||||
ssh -o BatchMode=yes myhost 'echo OK; hostname'
|
||||
```
|
||||
`BatchMode=yes` fails fast if a password would be required — a clean
|
||||
success proves the key works.
|
||||
|
||||
## Gotchas
|
||||
- Without `SSH_ASKPASS_REQUIRE=force`, ssh ignores SSH_ASKPASS when a TTY is
|
||||
present and just prompts interactively (hangs in a non-interactive tool).
|
||||
- `setsid` detaches the controlling terminal — required on older OpenSSH even
|
||||
with `force`, harmless otherwise.
|
||||
- The helper must be executable (`chmod 700`) and print ONLY the password.
|
||||
- `IdentitiesOnly yes` stops ssh offering other agent keys that could be
|
||||
rejected before yours, avoiding "Too many authentication failures".
|
||||
- Passwordless **sudo** on the target is a separate concern (NOPASSWD /
|
||||
`FRANK_SUDO_PW`); this skill only covers SSH key auth.
|
||||
@@ -0,0 +1,62 @@
|
||||
---
|
||||
name: swapfile-resize-safe-ordering
|
||||
description: "Resize a swapfile on a host whose MemAvailable is inflated by pinned model weights or huge page cache: put the replacement swap online BEFORE swapoff so live pages migrate to disk instead of into RAM that does not exist."
|
||||
---
|
||||
|
||||
# Resizing a swapfile without triggering the OOM killer
|
||||
|
||||
You cannot grow an active swapfile — it must be swapped off first. On a host that is already tight, **`swapoff` must move every live swapped page somewhere**, and if the replacement is not online yet, that somewhere is RAM.
|
||||
|
||||
The headline number lies. `MemAvailable` counts reclaimable page cache, and pinned/locked model weights plus ordinary cache make it read tens of GiB while only a few GiB is genuinely free. Never size a `swapoff` against `free` or `MemAvailable`.
|
||||
|
||||
## The ordering
|
||||
|
||||
Create and enable the replacement **while the old swap is still live**, so pages migrate to disk:
|
||||
|
||||
```bash
|
||||
# 1. Replacement online first — old swap untouched
|
||||
sudo fallocate -l 64G /swap2.img
|
||||
sudo chmod 600 /swap2.img
|
||||
sudo mkswap /swap2.img # also the hole check; refuses sparse files
|
||||
sudo swapon /swap2.img # total swap = old + new
|
||||
|
||||
# 2. Now drain the old one — pages go into the new swap, not RAM
|
||||
sudo swapoff /swap.img # minutes; watch `free` for used-RAM creep
|
||||
|
||||
# 3. Consolidate onto the conventional path so fstab needs no edit
|
||||
sudo swapoff /swap2.img # only the residual is left
|
||||
sudo mkdir -p ~/.AGENTTRASH/<date>-swapfile-<oldsize>
|
||||
sudo mv /swap.img ~/.AGENTTRASH/<date>-swapfile-<oldsize>/swap.img
|
||||
sudo mv /swap2.img /swap.img
|
||||
sudo swapon /swap.img
|
||||
```
|
||||
|
||||
Chain step 3 as one tight command — it leaves the host with **zero swap** for a few seconds. Do not leave that window open.
|
||||
|
||||
## Why each step
|
||||
|
||||
- **`mkswap` is a free correctness gate.** It rejects swapfiles containing holes, so a sparse `fallocate` fails loudly at format time instead of at 3 a.m. under memory pressure. Verify with `filefrag` if unsure; `unwritten` extents are allocated, not holes, and are fine.
|
||||
|
||||
- **Consolidating to the original path means fstab never changes**, so boot persistence carries over with no edit and no risk of a stale swap entry dropping boot to emergency. Renaming an *active* swapfile works (inode-based) but leaves a stale path in `/proc/swaps` — swap it off first, then rename, then `swapon`.
|
||||
|
||||
- **Migration is observable.** In the reference run, `swapoff` of 4 GiB took 65 s and used-RAM rose only ~3 GiB (pages touched during migration), the rest relocated on disk. If used-RAM climbs toward the whole swapped amount, abort reasoning and let it finish or re-enable.
|
||||
|
||||
## Verify
|
||||
|
||||
```bash
|
||||
swapon --show=NAME,TYPE,SIZE,USED,PRIO
|
||||
systemctl is-active <file>.swap # e.g. swap.img.swap — generated from fstab
|
||||
grep swap /etc/fstab # entry intact -> boot persistence
|
||||
```
|
||||
|
||||
Then prove the workload survived with a **real completion**, not `/health`.
|
||||
|
||||
## Filesystem caveat
|
||||
|
||||
ext4 and xfs take swapfiles directly. **btrfs needs `chattr +C` (NOCOW)**, no compression, and no snapshots in the path — check `df -T` before planning, or the swapfile will be rejected.
|
||||
|
||||
## After the change
|
||||
|
||||
More swap does not relieve pinned memory. Check `vm.swappiness`: at the default 60 the kernel will page anonymous memory under pressure, and if a large service's hot working set is anonymous (an LLM KV pool, for instance), paging it is worse than the OOM kill it prevents. The knob for that is lower swappiness or a cgroup cap on the service — not a bigger swapfile.
|
||||
|
||||
Stage the old file in trash rather than deleting it, and get explicit approval before purging.
|
||||
Reference in new issue
Block a user