dank/command-center
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

Add blankpad-host auto-learn skills and memory store

Browse Source 
Tracked the harness artifacts accumulated on blankpad (blank@), matching
the existing convention for the --home-dank-* memory stores and the 36
already-tracked managed-skills.

managed-skills (9): authentik-proxy-provider-app-link,
dankclaw-guard-chain-layer-diagnosis, e2e-arr-rawip-reachability-triage,
halogen-kv-pool-fit, lan-client-split-dns-fix, noctalia-reference,
rootless-quadlet-log-retrieval, ssh-key-bootstrap-no-sshpass,
swapfile-resize-safe-ordering.

memories (2 stores): --home-blank-.omp-- (dotfiles_git, handy_lifecycle,
input_permissions) and --home-blank-Projects-serverAppConfig--
(frank_large_quadlet).

Secret sweep before staging: no PEM blocks, no credential literals.
Content is fleet topology/IPs only, same class as the tracked dank stores.
This commit is contained in:
Frank
2026-09-13 00:25:29 -07:00
parent b60e95158c
commit 4b82f35354
37 changed files with 669 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
agent/memories/--home-blank-Projects-serverAppConfig--/learned.md
+15
View File
@@ -0,0 +1,15 @@
- Dankclaw host swap (as of 2026-09-12): 64 GiB file at /swap.img, ext4 on /dev/nvme0n1p2, fallocated hole-free, driven by /etc/fstab line 12 (/swap.img none swap sw 0 0) which generates the swap.img.swap systemd unit — fstab is the only control point, there is no hand-written unit and no repo file, so deploy.sh cannot see swap state. vm.swappiness=60 (distro default, UNTESTED against this workload), zswap disabled. Raised from 8 GiB; 748 GiB still free on /. The old 8.1 GiB file is staged at dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/swap.img awaiting Frank's approval to purge. Important framing: swap on this host protects the ~28 co-tenant containers, NOT frank-large — its ~68 GiB of pinned weights are not relieved by swap, and paging the KV pool (which IS anonymous) would be worse than the OOM kill it prevents, so if the box thrashes the knob is lower swappiness or a cgroup cap, not more swap. _(context: Raised swap on Dankclaw 2026-09-12 after the frank-large reserve-15 config left only ~7.1 GiB for co-tenants. Documented in serverAppConfigs-docs/Dankclaw/Swap Configuration.md and AGENTS.md memory-model section.)_
- In the serverAppConfig repo, ./tests/run.sh exits 1 on a **clean tree** because of one pre-existing failure: tests/comfyui_rocm.bats test 73 "comfyui Containerfile is based on the gfx1151 torch image". It is a test-side selector bug, not a product regression — its find ... | head -1 picks dankclaw/alloy/Containerfile instead of the ComfyUI one, so the assertion reads the wrong file. Verify before blaming your own change: a good run reports 285 ok plus this single not ok 73. Confirmed present at HEAD across several sessions (2026-09-12 frank-large work), independent of frank-large/Caddy/quadlet edits. Do NOT treat a green deploy as blocked by it, and do NOT "fix" it by weakening the assertion — the correct fix is to target the actual ComfyUI Containerfile path. Note deploy.sh gates on its own per-service health plus the Playwright e2e suite, not on tests/run.sh, so this failure does not block deploys. _(context: Every ./tests/run.sh run during the 2026-09-12 frank-large KV-pool/reserve work returned exit=1 with exactly not ok 73 comfyui Containerfile is based on the gfx1151 torch image while 285 tests passed, including on commits where nothing ComfyUI-related changed.)_
- Frank resolves contested tuning calls by **running the experiment**, not by reading the argument. On 2026-09-12 he overruled documentation-based pushback twice in a row with "lets just try it". Working with this, rather than against it: when he pushes back on a knob, state the prediction numerically up front, then run it and report measured vs predicted — he engages with that and it twice produced a better model than the docs (the arena linearity, and the vision-tower term). Corollaries: (1) get the rollback staged and named BEFORE the risky deploy, because the experiment may take production down — the 65536 trial OOMed frank-large for ~2 minutes; (2) use deploy.sh --skip-e2e for risky single-service experiments so the 4-minute e2e gate is not sitting on top of an outage; (3) after a failure, git revert sha + redeploy restores in ~10s, because the weights are already page-cached. Also learned the hard way: when a config experiment ends in a state that quietly reverses the user's original request (32768 + KV_POOL_FIT clamp meant the pool stayed 262144, i.e. the doubling never happened), say so plainly — do not let a green health check imply the goal was met. Separately, on log forensics: journalctl --user -u frank-large.service did NOT reach back to the 2026-09-11 boot even when asked with --since 2026-09-11 (rootless user journal retention is short). To compare against a container's ORIGINAL startup ledger, capture podman logs BEFORE the first deploy of the session — that is the only reliable copy of pre-change boot state. _(context: frank-large KV-pool doubling work on 2026-09-12: Frank overruled doc-based pushback twice ("let's just try it") — once raising HALOGEN_MAX_TOK to 65536 (OOMed the production service for ~2 min) and once back to 32768, which quietly un-did the doubling he had originally asked for. Also: journalctl --user retention for frank-large.service did not cover the 2026-09-11 boot even when queried with --si)_
- CORRECTION to the earlier "e2e 11-admin.spec.ts failing on 192.168.0.112:8989 means the Mac is asleep — wake it and re-run" lesson. That diagnosis is incomplete and can send you in circles. On 2026-09-12 the admin SSO spec failed on page.goto http://192.168.0.112:8989/ with ERR_CONNECTION_TIMED_OUT, and waking the Mac did NOT fix it: the Mac was verifiably awake (ping 33-54ms), all six socat forwarders were bound to 192.168.0.112, macOS Application Firewall reported State = 0 (disabled), netstat showed 8989 listening, and from **dankclaw** the ports answered 8989→200, 7878→302, 9696→302. Yet from **blankpad** (the dev box where Playwright actually runs) the same ports returned curl rc=28 / 000, while ICMP ping to .112 succeeded. So the blocker is a blankpad↔192.168.0.112 path problem — ICMP allowed, TCP to the arr ports not — not sleep and not the Mac's own firewall. Distinguish the two causes by probing from the box the browser runs on, not from a fleet peer: curl -s -o /dev/null -w '%{http_code}' --max-time 7 http://192.168.0.112:8989/ on blankpad vs ssh dankclaw 'curl ... 192.168.0.112:8989'. Peer 200 + local timeout = path/firewall issue, and no amount of waking or re-running will make that spec pass from this box. Consequence: the mandated e2e gate currently cannot go fully green from blankpad; 29/30 pass, 11-admin fails on Sonarr's raw IP. It is environmental and unrelated to what is being deployed — verify the deployed service directly (ledger, /health, real completion) rather than treating the red gate as a regression, and say which gate was skipped and why. _(context: 2026-09-12 frank-large KV-pool work: e2e admin SSO spec failed on Sonarr raw IP while the Mac was provably awake and dankclaw reached it fine; blankpad could not open the TCP ports at all.)_
- halogen-flash-server (frank-large, Dankclaw) device-RAM sizing is ONE shared budget, and it is now measured rather than estimated: device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB (121 - 67.7 - 24.0 = 29.5 GiB here, plus a 1.5 GiB vision margin). The KV pool costs ~28 KiB/position (524,288 = 14.4 GiB) and the PREFILL ARENA IS LINEAR IN HALOGEN_MAX_TOK at 3.80 GiB base + 8.40 GiB per 16,384 tokens — verified against three restarts to 0.1 GiB. So the ceiling is max_tok = (budget - margin - pool - 3.8)/8.4 * 16384; with the doubled pool that is max_tok = 19,115, making the deployed 16384 the CEILING, not a conservative setting. Corner check confirms: max_tok 32768 leaves 7.4 GiB = ~269k positions, exactly the old 262144 config. TWO DIFFERENT FAILURE MODES when you violate this: (a) under the ceiling, HALOGEN_KV_POOL_FIT silently LOWERS the pool and deploy.sh still reports healthy in 12s — you think you doubled it, you did not; (b) OVER the ceiling the arena alone outgrows the budget and the server WILL NOT BOOT at all — measured at max_tok=65536: "WARNING even 262144 positions ... needs ~44.5 GiB", then "dmalloc: FAILED requesting 0.250 GiB after 39.970 GiB (out of memory)". Mode (b) is the dangerous one because systemctl stays active, the container reads Up (starting), and deploy.sh waits out the full LLM_WAIT instead of failing fast — an outage that looks like a slow deploy. Always grep podman logs frank-large for kv pool: and confirm the reserved position count equals the requested one; /health 200 proves nothing about pool size.
- The e2e suite's 11-admin.spec.ts navigates to the *arr services at their raw LAN IPs (192.168.0.112:8989 Sonarr, 7878 Radarr, 9696 Prowlarr), and Playwright runs from the DEV BOX (blankpad), not from Dankclaw. There are therefore TWO independent ways that spec fails, and they need different fixes: (1) The known one: FrankBook-Pro asleep - arr ports unreachable from everywhere. Wake it, re-run. (2) NEW, found 2026-09-12: blankpad cannot reach those forwarded ports AT ALL, even with the Mac fully awake. Measured: from blankpad, ping 192.168.0.112 succeeds (33-54 ms, 0% loss) but TCP to 8989/7878/9696 all return curl rc=28 (connection timeout). From dankclaw the same ports return 200/302/302. On the Mac itself: socat forwarders ARE running and bound to bind=192.168.0.112, curl 127.0.0.1:8989 - 200, and socketfilterfw --getglobalstate reports "Firewall is disabled (State = 0)". So the Mac is not the blocker — something on the blankpad-FrankBook-Pro path drops TCP to those ports while passing ICMP. Diagnostic order that separates them, before assuming "the Mac is asleep": a. From the dev box: curl -s -o /dev/null -w '%{http_code}' --max-time 7 http://192.168.0.112:8989/ AND ping -c2 192.168.0.112. b. From dankclaw: same curl. If dankclaw gets 200 but the dev box times out while ping works, it is failure (2), not sleep — re-running the spec will never go green from the dev box. c. On the Mac (via ssh dankclaw "ssh dank@192.168.0.112 ..."): pgrep -fl socat, curl 127.0.0.1:8989, firewall state. Implication: a red 11-admin.spec.ts is NOT automatically evidence of a fleet outage, and it is NOT caused by whatever was just deployed. Confirm which failure mode by comparing the dev-box probe against the dankclaw probe before attributing the failure to your change. _(context: 2026-09-12, after doubling frank-large's KV pool on Dankclaw. The e2e gate's 11-admin.spec.ts failed on http://192.168.0.112:8989/ (ERR_CONNECTION_TIMED_OUT) on both the initial run and retry. Waking the Mac did NOT fix it, which is why this qualifies the older sleep lesson.)_
- halogen-flash-server (frank-large on Dankclaw): raising HALOGEN_KV_POOL_POSITIONS can SILENTLY NOT TAKE EFFECT, while the deploy still reports healthy. HALOGEN_KV_POOL_FIT=1 (default) measures the device budget at startup and lowers the pool itself if the configured value will not fit — and it never raises. The budget is ONE pool covering the KV AND the prefill arena together: host RAM − resident weights − HALOGEN_HOST_RESERVE_GIB. Measured on Dankclaw 2026-09-12: 121 GiB − 67.7 weights − 24.0 reserve = 29.5 GiB device budget. Pool 524288 with HALOGEN_MAX_TOK=32768 needs ~35.0 GiB (+1.5 margin) → refused, silently lowered to 262144, deploy reported "healthy after 12s". Fix: HALOGEN_MAX_TOK=16384 halves the arena (~20.6 → ~12.3 GiB working mem), bringing the 524288 need to ~26.6 GiB, which fits with ~2.9 GiB margin at ~8-9% prefill cost. Net result was CHEAPER than the old config: 94.6 GiB held / 13.8 GiB host-left vs 96.3 / 12.1. RULE: after ANY halogen restart, read podman logs frank-large | grep 'kv pool:' and confirm the pool it reserved matches what the unit asked for — a green /health and a green deploy.sh say nothing about pool size. The engine prints "LOWERING THE POOL TO N" in the refusal; that line is the tell.
- frankbook-pro (FrankBook-Pro.local, macOS 192.168.0.112) is reachable from this dev box ONLY via the dankclaw jump host: ssh dankclaw "ssh dank@192.168.0.112 'bash -l -s'" script.sh (fleet inter-host key works for dank@112; frank@/admin@ are denied; NO passwordless sudo on the Mac — sudo -n fails). The *arr stack (sonarr 8989, radarr 7878, readarr 8686, lidarr 8787, prowlarr 9696, qbittorrent, gluetun) runs inside podman-machine-default (applehv); on the macOS host, socat forwarders bind 192.168.0.112:port → 127.0.0.1:port (gvproxy → VM). macOS Application Firewall is DISABLED and /etc/pf.conf is stock. THE FLAKE: when the Mac sleeps, the arr ports go unreachable from the LAN (curl connect 0.000s refused / ERR_CONNECTION_TIMED_OUT in Playwright), which fails the e2e 11-admin.spec.ts SSO test. Waking the Mac (SSH/ping) restores all ports to 200/302 within seconds. So an admin-SSO e2e failure on a *arr link is usually "the Mac is asleep" — wake it and re-run before treating it as a real outage. Diagnostic order: (1) ping 112 (high latency ~150ms = asleep/waking), (2) from the Mac host curl 127.0.0.1:8989 and curl 192.168.0.112:8989 (both 200 = chain healthy, block is external/sleep), (3) podman machine list + podman ps -a (Up = services fine). _(context: Diagnosing the e2e admin SSO test failing on Sonarr raw-IP (112:8989) during the 2026-09-12 marreta work — turned out to be frankbook-pro sleeping, not a real outage.)_
- authentik embedded-outpost proxy providers: a ProxyProvider is INVISIBLE to the Rust embedded outpost unless it is linked to an authentik Application. The outpost's config API (ProxyOutpostConfigViewSet in authentik/providers/proxy/api.py) filters ProxyProvider.objects.filter(application__isnull=False). So a provider created via ProxyProvider.objects.create(...) with no Application never reaches the outpost, and requests to its external_host fall through to raw flow serving on the app's own host — where the IdP session cookie (scoped to auth.domain) is invisible, so the user is stuck on a login wall (e.g. /if/flow/default-authentication-flow/ or /flows/-/default/authentication/). FIX: create Application(name=..., slug=..., provider=proxy_provider, launch_url=...) and restart authentik-server + the outpost. Also required for the same provider to work: invalidation_flow must be a real UUID (null breaks the Rust outpost's serde with "invalid type: null, expected a formatted UUID string" and takes down ALL providers on that outpost), and external_host must exactly match the Host header (trailing-slash mismatch = no match). Diagnose by: (1) ak shell → list providers with application/invalidation_flow; (2) journalctl --user -u authentik-outpost.service | grep -iE "error|invalid" for the serde error; (3) compare a working provider's full field set against the broken one. _(context: Fixed marreta.franksdank.cloud login-wall on Dankclaw 2026-09-12: the marreta-proxy ProxyProvider was created without a linked Application, so the embedded outpost never loaded it. Root cause chain: null invalidation_flow (serde crash) - missing flows - missing Application link.)_
- NEVER send SIGHUP to PID 1 inside a podman container to reload a service. In podman containers PID 1 is the container supervisor/init, not the app — SIGHUP to it TERMINATES the whole container. On 2026-09-12 I tried podman exec pihole kill -HUP 1 to reload dnsmasq addn-hosts; it killed the pihole container and took LAN DNS down for ~15s until the quadlet unit's Restart=always (plus an explicit systemctl --user restart pihole.service) brought it back. The app process (pihole-FTL) was PID 54, not 1. Correct ways to reload a service in a podman container: (a) systemctl --user restart svc.service (the quadlet unit — clean, intended), or (b) signal the ACTUAL service PID found via podman exec c ps -e -o pid,comm, never PID 1. For Pi-hole specifically, the repo flow is deploy.sh dankclaw --service pihole which does the addn-hosts cp + unit restart. _(context: 2026-09-12, closing the marreta addn-hosts gap on the live Pi-hole. pihole reload is not a valid subcommand (printed usage, exit 0), so the || kill -HUP 1 fallback fired and killed the container.)_
- On the Dankware/Dankclaw homelab, a "connection timeout" to any *.franksdank.cloud service from a LAN client is usually a CLIENT split-DNS problem, not a server outage. The public DNS record for franksdank.cloud points at the WAN IP (71.227.227.172) which has NO 443 port-forward (filtered). Pi-hole (192.168.0.143) returns the correct LAN answer 192.168.0.143 via addn-hosts. If a client's systemd-resolved is using 1.1.1.1 (or any external resolver) as its current server, franksdank.cloud resolves to the WAN IP → TCP SYN to a filtered port → timeout (curl rc=28). LAN-only names (dankclaw/grafana/marreta) return NXDOMAIN instead. Diagnose: dig +short www.franksdank.cloud @192.168.0.143 (→143) vs @1.1.1.1 (→WAN); resolvectl status shows the current server. Fix on the client: a systemd-resolved drop-in routing the domain route-only to Pi-hole. Verify the server itself is fine by pinning the LAN IP: curl -sk --resolve www.franksdank.cloud:443:192.168.0.143 https://www.franksdank.cloud/ → 200. _(context: 2026-09-12 homepage "links gone / not loading / connection timeout" on dankware. Server was fully healthy (Caddy @home 200, Anubis challenge valid, Authentik outpost healthy, files render). Root cause was dankware's resolved using 1.1.1.1. Fixed with /etc/systemd/resolved.conf.d/franksdank-lan.conf [Resolve] DNS=192.168.0.143 Domains=~franksdank.cloud ~franksdank.casa + restart systemd-resolved.)_
- Dankware fleet — "service unreachable / connection timeout" triage for Anubis+Authentik+Caddy guard-chain vhosts. Two durable gotchas: (1) Rootless-podman published-port SNAT makes Anubis log x-real-ip: 10.89.1.22 (the podman gateway) for EVERY client — the real client IP is lost at the published-port layer, so never use Anubis x-real-ip for per-client identity/rate-limit reasoning. (2) The WAN IP 71.227.227.172 has NO 443 port-forward; any client resolving *.franksdank.cloud to the WAN IP (i.e. NOT using Pi-hole 192.168.0.143 as DNS, or off-LAN) gets a TCP connection timeout — the SYN hits the filtered public address. LAN clients must resolve via Pi-hole to 192.168.0.143. To PROVE a guard-chain service is server-side healthy (and thus the fault is the complaining client's DNS/network, not the server): test from a LAN peer — ssh dankserver then dig +short www.franksdank.cloud (expect 192.168.0.143), /dev/tcp/192.168.0.143/443 (expect OPEN), curl -sk https://www.franksdank.cloud/ (expect 200). Also check Pi-hole addn-hosts has the 192.168.0.143 host line and iptables -L INPUT policy is ACCEPT. If the LAN peer gets 200 but one device times out, the server is fine — fix that device's DNS (point at 192.168.0.143) or its LAN connectivity. Note: a curl to the bare IP without matching SNI returns code 35 (SSL name mismatch) — that is NOT a real failure; always use the hostname URL with --resolve or real DNS. _(context: 2026-09-12: homepage reported "links gone" then "connection timeout" from danklivingroom. Exhaustive server-side trace (Anubis challenge page valid, Authentik outpost healthy, Caddy @home/@data 200, files render with visible links, no CSP/redirect-loop, no fail2ban) proved the stack healthy; dankserver LAN peer got https 200 in 7ms. Root cause isolated to the complaining client resolving to the fi)_
- On dankclaw's Anubis→Authentik→Caddy guard chain, the rootless-podman published-port SNAT masks the real client IP: Anubis logs x-real-ip: 10.89.1.22 (the Caddy container's podman-network address) for EVERY client — verified with both host-local curl and a browser from a different machine (blankpad). So Anubis cannot distinguish clients by IP; clearance is cookie-based, not IP-based. This is pre-existing fleet behavior, NOT caused by config edits, and does not break clearance. When a user reports a guarded page "broken" but the server is healthy, the break is client-side/session — you need the authenticated view (Authentik creds) or the exact symptom, not more server probing. _(context: 2026-09-12 homepage "links gone / not loading" investigation; server-side fully verified healthy, root cause not reached (awaiting user symptom).)_
- TRANSPORT UPDATE (2026-09-12, supersedes the old "blankpad has no fleet SSH" lesson): blankpad (~/.ssh on user blank) NOW has passwordless SSH to the whole fleet. An ed25519 key was generated (id_ed25519, comment blankpad-deploy-20260912) and its pubkey installed on dank@192.168.0.143 (dankclaw) and dank@192.168.0.235 (dankserver). ~/.ssh/config defines Host aliases dankclaw and dankserver (User dank, IdentityFile ~/.ssh/id_ed25519, IdentitiesOnly yes). So ssh dankclaw / ssh dankserver and ./scripts/deploy.sh dankclaw ... work from this box now — no more "ask Frank for a transport." Passwordless sudo on the targets is separate (FRANK_SUDO_PW / NOPASSWD). MARRETA/LARAVEL APP_KEY GOTCHA: a Laravel APP_KEY must be the base64:-prefixed form — base64:$(openssl rand -base64 32). A bare base64 string (no prefix) is parsed as a raw key of the wrong length and every app route 500s with Unsupported cipher or incorrect key length at Encrypter.php:61, while the health route /up (which skips the encrypter) still returns 200 — so a green healthcheck masks a fully-broken app. Fix before first boot (no encrypted data persisted yet). _(context: Deployed marreta on dankclaw 2026-09-12; had to set up SSH keys from blankpad first, then fix a bare-base64 APP_KEY that 500'd all app routes.)_
- On the Dankware/Dankclaw fleet, LLM services that are rootless Quadlet units (e.g. frank-large on Dankclaw, user dank) must be read WITHOUT sudo: sudo podman logs name fails with "no such container" because rootless storage is per-user. And because Quadlet runs --replace --rm, podman logs only covers the CURRENT container instance — history across recreations lives only in journalctl --user -u name.service. So: current instance → podman logs; anything before the last recreate → journalctl --user. Separately: the dev box /home/blank/Projects/serverAppConfig (user blank) has NO ssh key or host alias for the fleet hosts — ~/.ssh holds only known_hosts, and dank@192.168.0.143 / frank@192.168.0.143 both return Permission denied (publickey,password). Do not burn calls probing fleet SSH from this box; ask Frank for a working transport before claiming live verification, and label repo-grounded answers as not live-verified. _(context: Answering "how do I retrieve logs for frank-large?" (2026-09-12); grounded in dankclaw/frank-large/quadlet/frank-large.container and serverAppConfigs-docs/LLM/halogen-flash-server.md:191-219. Live verification attempt failed on missing SSH credentials.)_