chore(memory): record faillock sudo-lockout law + mint recovery skill

redact gate pattern literals in the audit prose — the pre-commit gate
self-matches its own documented patterns when quoted verbatim

agent/managed-skills/faillock-sudo-lockout-recovery/SKILL.md

@@ -0,0 +1,27 @@
+---
+name: faillock-sudo-lockout-recovery
+description: "Diagnose and recover from a pam_faillock sudo lockout on Frank's Arch workstations, typically caused by an agent's non-interactive sudo retries"
+---
+
+# Faillock Sudo Lockout Recovery
+
+## Symptom
+User reports "sudo password rejected / changed"; journal shows
+`pam_faillock(sudo:auth): User <u> is temporarily locked out due to 3 consecutive failed login attempts`.
+
+## Diagnose (read-only, one round)
+1. `stat -c '%n mtime=%y ctime=%z' /etc/shadow` — unchanged mtime = password was NEVER changed.
+2. `journalctl --since '<window>' | grep 'sudo\['` — lines with `PWD=` / `COMMAND=` name the spammer (agent harness loops show `PWD=/home/.../.omp`, `COMMAND=...yay/pacman...`, no `TTY=`).
+3. `faillock --user <user>` — shows the valid failure entries.
+
+## Root cause law
+In the omp harness sudo has NO TTY: each `sudo <cmd>` = 3 pam_unix failures. Arch pam_faillock defaults deny=3 with ~600s unlock_time, and each retry re-arms it — the user's real interactive sudo is locked while the loop runs.
+
+## Recover
+1. Kill the loop: `ps -eo pid,ppid,etime,cmd | grep -E 'sudo|yay|makepkg'` → kill offending PIDs; stop/kill the responsible agent session.
+2. `faillock --user "$USER" --reset` — works unprivileged for your own user (dir /run/faillock/<user> is 0700 owner).
+3. User verifies with `sudo -v` in a real terminal.
+
+## Prevent
+- Agents NEVER run sudo/yay/pacman/pkexec; no SUDO_ASKPASS/expect workarounds.
+- Root-requiring steps are handed to the user as exact copy-paste commands in the final report.

agent/memories/--home-blank-.omp--/learned.md

@@ -1,3 +1,7 @@
+- On Frank's workstations (Arch, pam_faillock deny=3), subagents MUST NEVER run sudo/yay/pacman: in the harness sudo has no TTY, so every attempt logs 3 pam_unix failures and trips pam_faillock "temporarily locked out" (~600s), which re-arms on every retry and locks out the USER's real interactive sudo — observed 2026-10-09 on blankpad when a Handy-install subagent looped sudo yay -Syu handy-bin for 15 min; the user saw "incorrect password"/lockout and (understandably) suspected his password was changed. Diagnostics that settle it in one step: stat /etc/shadow (unchanged mtime = no password change), journalctl | grep 'sudo\[' (PWD/COMMAND identify the spammer), faillock --user blank. Recovery: kill the loop, then faillock --user self --reset works unprivileged (per-user dir /run/faillock/user is owner-writable); then user runs sudo -v. Any package install requiring root must be handed to the user as exact copy-paste commands, never attempted by an agent.
+- Tool traps verified on blankpad 2026-10-09 (niri/noctalia/Handy DE session): (1) niri msg action (and other niri msg subcommands that print help/action lists) write to STDERR — niri msg action 2/dev/null | grep ... yields empty output and looks like "feature missing"; always pipe with 2&1. Confirmed live: niri 26.04 has swap-window-left/right (window walks sideways, entering/leaving columns; column order untouched), move-window-to-monitor-*, focus-window-up/down (tabbed columns navigate with up/down; focus-window-left/right does NOT exist — no gap to fill). (2) frank-zealot (llamacpp dankclaw-3090 :7771) as a subagent model crashed mid-run with invalid string: surrogate U+D800..U+DBFF when a tool-call JSON argument contained many emoji — server-side JSON parse, type=server_error; keep subagent prompt/context payloads ASCII (no emoji) when agents run on fleet llama.cpp/radiance models; the crash is infrastructure, retry after stripping emoji. (3) The failed agent HAD already written its files before crashing (omp-docs/tools/handy.md, noctalia-shell.md rewrite survived; only its cleanup grep died) — after a subagent exit-1, always inspect target files for partial/complete output before respawning; its summary claims are unverified, diff against live truth (it invented a Handy settings JSON shape and flipped post_process_enabled true; live value false, real shape has bindings.transcribe/current_binding). (4) edit tool payload format: every change block MUST start with a PUT N.=M: (or DEL N.=M) hunk header followed by +/- lines; bare N:text + +text lines are rejected with "payload line has no preceding hunk header". (5) blankpad repo fact for pending work: live ~/.config/niri/cfg/{misc,rules}.kdl symlinks point at the GENERIC dotfiles/niri copies, which pin render-node /dev/dri/renderD129 "Intel Arc" — blankpad is single AMD Cezanne (renderD128); the newer dotfiles/zz-blankpad copies (no render-node, cursor 36, blur off, noctalia layer rules) are the correct live files, _(context: Grilling session on blankpad niri 26.04 / noctalia 5.2.1 / Handy 0.9.6 DE config; fact-check sweep + doc rewrites in dank/command-center omp-docs.)_
+- Blankpad DE stack, verified 2026-10-09 against live system + upstream (source of truth for future niri/noctalia/handy work; supersedes omp-docs claims): (1) Noctalia is v5 (installed 5.2.1-1.1, native noctalia binary). The v4 Quickshell line is DEAD upstream — repo doc omp-docs/tools/noctalia-shell.md (v4.7.7, qs -c noctalia-shell ipc call …) is stale; correct API is noctalia msg cmd, full list from noctalia msg --help (~110 cmds: screenshot-region/-fullscreen/-annotate, window-switcher, clipboard-*, workspace-alert-*, theme-mode-toggle, session lock-and-suspend, dpms-*, greeter-sync, templates-apply). Noctalia's niri docs (docs.noctalia.dev/noctalia/compositor-settings/niri/): autostart spawn-at-startup "noctalia", lid policy via niri switch-events { lid-close { spawn "noctalia" "msg" "session" "lock-and-suspend"; } }, wallpaper/blur layer-rules, [shell] niri_overview_type_to_launch_enabled. (2) niri 26.04 IS latest (repo moved to niri-wm org). New since docs: mainline blur/xray (background-effect, popups block, layer matcher), include optional=true + ~ expansion, niri msg action load-config-file --path file (live-swap config path — key for dotfiles symlink testing), cast IPC (niri msg casts, stop-cast), screenshot-window show-pointer=true, output options variable-refresh-rate (+on-demand=true), focus-at-startup, backdrop-color, per-output layout{}, mode custom=true/modeline (25.11), Escape cancels drag-and-drop. (3) Handy (cjpais/Handy) 0.9.6 installed via local/handy-bin; latest 0.9.8 (2026-10-03). Upstream LAW: SIGUSR1 handler REMOVED (WebKitGTK owns SIGUSR1; pkill -USR1 handy can crash the app) — toggle is SIGUSR2 or CLI flags handy --toggle-transcription / --toggle-post-process / --cancel; startup flags --start-hidden --no-tray. Wayland: compositor must own global binds (call the CLI flags from niri binds); text input needs wtype/dotool (NOT installed on blankpad; current paste_method=none + copy_to_clipboard, so no tool needed); overlay disabled-by-default up _(context: Fact-check sweep of niri/noctalia/handy docs on blankpad, 2026-10-09, ahead of a pending config-change grilling (Round 1 questions pending user answers))_
+- command-center (~/.omp) secret hygiene, audited 2026-10-09 at 79e780f: repo history is CLEAN — no live value (gitea/hf_token/hf_token_write/kagi_api_key) in any of 77 revs / 765 reachable blobs / 2 dangling blobs / 23 git-lfs payloads; the only sk-shaped string is a 35-byte unreachable fixture whose body is the alphabet + 1234. DURABLE GAP: .githooks/pre-commit (core.hooksPath=.githooks) matches only the OpenAI-style 18+ alnum token, ghp 36, github-pat 22+, glpat 20, the xox[baprs] family, AWS key ids, PEM private-key headers, aes 40-char, eyJ….eyJ (JWT). It does NOT match hf 30, npm 30, pypi, AIza 35, Stripe live/test keys, Telegram bot tokens, or a scheme://user:token@host basic-auth URL — the basic-auth shape is exactly what sat in remote.origin.url on both workstation clones until 2026-10-09, so the gate would never have caught it. The gate DID fire correctly on a synthetic token literal quoted in a commit (2026-10-09) AND on the audit prose itself (the gate self-matches its own documented pattern literals — quote them descriptively in docs/memory), so redact literals in docs/commit messages instead of using ALLOW_SECRETS. Live secrets live in ~/.config/secrets/* (dir 0700, files 0600 — hf_token_write was 0644 and was fixed); ~/.cache/huggingface/token duplicates hf_token at 0600 by HF convention.
 - Gitea client auth on Frank's workstations (2026-10-09): the HTTP token lives ONLY in ~/.config/secrets/gitea (0600) behind a global helper git config --global credential.helper 'store --file ~/.config/secrets/gitea'. All remote URLs are bare (http://192.168.0.143:3000/dank/repo.git) — a token embedded in remote.origin.url leaks via .git/config, git remote -v, and the git-lfs "Locking support detected on remote" warning, which echoes the full URL on every push (silence it per repo with git config lfs.bare-url/info/lfs.locksverify true). TRAP, measured on git 2.56.0: git credential-store does NOT parse the protocol=/host=/username=/password= block shape in the store file — that shape is only what git credential fill accepts on stdin, and the man page's ~/.git-credentials example is misleading. A block-format file yields a SILENT empty get, git falls through to askpass and dies with "could not read Username for …" (SSH_ASKPASS=/usr/bin/false in the omp harness env). The store file must be one URL per line WITH the port: http://dank:token@192.168.0.143:3000. Verify before trusting it, with prompts disabled so failure is loud: printf 'protocol=http\nhost=192.168.0.143:3000\n\n' | git credential-store --file ~/.config/secrets/gitea get must print username+password; then GIT_TERMINAL_PROMPT=0 git ls-remote --heads origin main and git push --dry-run. Rotation = rewrite that one file, nothing else. Documented in omp-docs/tools/gitea/client-access.md (command-center commit cfed6ef).
 - The origin remote of the command-center repo (~/.omp) embeds the Dankclaw Gitea token in plaintext: http://dank:token@192.168.0.143:3000/dank/command-center.git in .git/config. Consequence observed 2026-10-09: git push origin main echoes that URL (the git-lfs "Locking support detected" hint prints the full URL with the token) into stdout, so any transcript/log of a push contains a live credential. Recommended remediation proposed to Frank and NOT yet approved/applied — do not change remote auth unilaterally: git remote set-url origin http://192.168.0.143:3000/dank/command-center.git plus git config credential.helper 'store --file ~/.config/secrets/gitea' (that directory is Frank's existing 600-perm secret convention), and git config lfs.http://192.168.0.143:3000/dank/command-center.git/info/lfs.locksverify true to silence the lock hint. Reuse rule: when a token must be read from this repo, read it from the existing origin URL and reference its location only; never copy it into docs, skills, memory, or commit messages. Repo hooks: core.hooksPath=.githooks → pre-commit secret gate (patterns incl. eyJ… JWT shapes; override ALLOW_SECRETS=1) and pre-push git lfs pre-push, so pushes carry LFS objects.
 - Fleet LLM model ids in ~/.omp/agent/models.yml were renamed to the unit-name scheme on 2026-10-08 (committed 0cb021b, pushed 2026-10-09): frank-medium → frank-archon (llamacpp dankserver, 192.168.0.235:7770, ctx 200000 / maxTokens 16384), frank-large → frank-dragoon (dankclaw, 192.168.0.143:7770, 217088 / 45056, text+image), frank-small → frank-zealot (dankclaw-3090, 192.168.0.143:7771, 200704 / 16384). Service/unit/script names are UNCHANGED — /v1/models still advertises frank-primary, frank-large, frank-secondary. LAW (verified live with chat-completion probes on all three): radiance, halogen-flash-server and llama.cpp all accept an arbitrary request model id, so a harness-side id rename needs no endpoint change; the response model field just echoes the served name. Server-side compaction moved from dankserver:7770 to 192.168.0.143:7771 (frank-zealot), thresholdPercent 70 → 85, and every model's compactionModel now points at llamacpp (dankclaw-3090)/frank-zealot so the wire id stays valid at the compaction endpoint. Provider names in models.yml are labels referenced from config.yml (modelRoles, task.agentModelOverrides) and from compactionModel — renaming one requires updating every reference; verify by parsing both YAML files and cross-checking all llamacpp (host)/id tokens (regex must allow the space+parens in the provider label; a naive whitespace split yields false "unresolved" hits). Also: git renders whole-directory renames as mass delete + mass untracked — the 2026-10-09 tree showed 116 D + 73 ?? for the managed-skills consolidation (94 → 25) and the memory-skill re-bucketing (dotfile-git-workflow → ghostty-text-aa + niri-display-rotation + noctalia-theme; handy-service-mgmt → handy-flm-inference; inference-runtime → llama-cpp-parallelism). Nothing was lost: index matched HEAD, so git restore recovers any of it.

agent/memories/--home-blank-Projects-serverAppConfig--/learned.md

@@ -1,3 +1,4 @@
+- serverAppConfigs secret-scan baseline (2026-10-09, gitleaks 8.24 full history + manual sweeps, 559 files/1167 commits): tree and history CLEAN of real secrets. Known gitleaks false positives to expect on re-runs — (1) Bearer PASTE-YOUR-KEY in shared/homepage/docs.html (placeholder in public API guide); (2) curl -u admin:admin in serverAppConfigs-docs/Operations/Fleet Audit and Posture.md (Grafana default-credential RESIDUAL CHECK, not a leak); (3) FRANK_LARGE_*_SHA256 checksums in scripts/frank-*.env; (4) OAUTH_CLIENT_ID / GF_AUTH_GENERIC_OAUTH_CLIENT_ID in open-webui.container, grafana.container, dankclaw/compose.yaml — OAuth2 client IDs are public identifiers; the client SECRETS live only in git-ignored host .env files. Real-value secret file on disk: tests/e2e/.env.local (gitignored, verified). PENDING POSTURE ITEM reported to Frank, not yet approved for action: Grafana on dankclaw answers admin:admin on loopback 127.0.0.1:3002 (probe: ssh dankclaw curl -u admin:admin http://127.0.0.1:3002/api/org → 200) — rotation recommended, awaiting his go.
 - serverAppConfigs authentik user provisioning (five accounts kirby/ry/nic/debo/freya pk 17-21 created 2026-10-09; law now in vault commit 65dec6a): (1) After ak shell provisioning with the human-user template (groups Non-Admin Users + jellyfin_users + openwebui_users), OpenWebUI mints each account on FIRST SSO login with role pending (ui.default_user_role=pending in webui.db config table) — they log in fine, then cannot chat / picker is empty until UPDATE "user" SET role='user' WHERE email=…; the row does not exist before first login, so the flip is a first-login-sighting follow-up, not a provisioning step. No per-user model grant needed — the picker model carries the global access_grant row (model, frank-secondary, user, *, read). Written up in serverAppConfigs-docs/Dankclaw/OpenWebUI.md (SSO section). (2) The proxysession claim purge is only due when a group change follows an existing login; users created with their groups already in place have zero rows — verify with a per-user census instead of purging blindly. (3) The authentik-postgres container's DB role/db are authentik/authentik (POSTGRES_USER env), NOT postgres: podman exec authentik-postgres psql -U authentik -d authentik -tAc "select session_data-'claims'-'preferred_username', count(*) from authentik_providers_proxy_proxysession group by 1". (4) User.app_entitlements() requires the Application argument (Application.objects.get(slug="open-webui")) — bare call raises TypeError. Verification bar for a provision: authenticate(request=None, …) accepts the password and rejects a wrong one (the exact PasswordStage path) plus the open-webui entitlement resolves. _(context: serverAppConfigs 2026-10-09: provisioned five authentik human accounts on Dankclaw; vault pages Infrastructure/Authentik.md, Dankclaw/OpenWebUI.md, log.md updated (commit 65dec6a), bats suite 366 green.)_
 - radiance arch-capability audit (dankserver, pin 1.0.13, verified 2026-10-08): the container ships 6 arch plugins in /opt/radiance/share/radiance/architectures/ — qwen4exp_fp8, qwen35moe_fp8, qwen35_fp8, qwen35_bf16, llama_fp8, llama_dense_fp8. Capability check per plugin: grep -ac RADIANCE_CALIB / grep -ac gram / grep -ac mtp / grep -ac ngram on the .so (qwen35moe_fp8: calib=2 grams=2 mtp=4 ngram=0; gptq=0 in ALL plugins — GPTQ lives in rad-convert, not the plugin). Introspection pitfalls: the digest-pinned name (stilldeadcode/radiance@sha256:4c3f48…) does NOT resolve locally (podman tries a registry pull and fails); use the local tag docker.io/stilldeadcode/radiance:latest — image lives in ROOTLESS storage as dank (frank-primary runs rootless under the root unit; sudo podman ps is empty). Image has no ls/strings; podman exec frank-primary ls fails; correct form: podman run --rm --entrypoint sh docker.io/stilldeadcode/radiance:latest -c '…'. Verdict recorded: Qwen3.6-35B-A3B (qwen3_5_moe_text, 40 layers, 256 experts top-8, MTP×1) is plugin-supported; w4dn8 recipe maps minus PLE/HC terms; ~20 GiB w4+i8 = fully VRAM-resident on ONE 32 GiB R9700 (no TP=2/host-pool/disk-tier/MNBT-2048 law); convert ~1–1.5 h. Full audit in serverAppConfigs-docs/LLM/Quantization Production.md. _(context: serverAppConfigs 2026-10-08: feasibility audit of porting the frank-archon w4dn8 radiance recipe to Qwen3.6-35B-A3B; KB page Quantization Production.md carries the full write-up.)_
 - Quantization production research (2026-10-08, full detail in repo page serverAppConfigs-docs/LLM/Quantization Production.md, linked from both vault indexes): (1) Law — production is hardware-agnostic (imatrix/GPTQ/ablation/pruning are torch-CPU work); the card only picks what it can DECODE. (2) Frank's framing: the 3090 is the ONLY unlocked fleet device — llama.cpp there is a choice; sm_86 decodes GGUF K/I, GPTQ/AWQ-Marlin INT4, SmoothQuant INT8, weight-only FP8 (FP8 W8A8 compute is Ada/Hopper SM≥89 only; NVFP4 Blackwell; 3090 has NO FP8/FP4 silicon), EXL3 (ExLlamaV3, convert.py runs on-card). The 3090 is the natural quant R&D bench (produce on host, decode+score vs fleet KLD gates). (3) REAP (Cerebras, arXiv 2510.13999, github CerebrasResearch/reap) expert saliency = router gate-values × activation energy — the INVERSE of our derive-protected-experts.py energy-share law; REAPing Flash-Next experts 25–50% re-enters the radiance pipeline at calibration, zero new tooling. (4) HauhauCS K_P "Perfect" = imatrix on abliterated weights + per-model per-tensor mixed-type profile via llama-quantize --tensor-type (plain GGUF, no fork; Q8_K_P census: Q8_K+F32+F16); reproducible with binaries built by scripts/build-llama-rdna4.sh. (5) Heretic (p-e-w/heretic, AGPLv3) = automated abliteration (refusal-direction ablation + layer dropping, refusals-vs-KL tuned), NOT a quantizer; abliteration (Arditi 2406.11717) is rank-1 fp-editing. Order law: surgery → calibrate → quantize; keep refusal tokens out of calibration. (6) .hgn production door is CLOSED — no public quantizer exists in peonist-ai repos; self-quant on Strix runs through GGUF runtimes. _(context: serverAppConfigs research session 2026-10-08; scouts + first-hand verification of REAP/Heretic/abliteration/HauhauCS K_P and per-runtime quant matrices; KB page Quantization Production.md committed to main.)_
@@ -97,4 +98,3 @@
 - serverAppConfig multi-session triage rule (proven 2026-10-05, three events in one day): when ./tests/run.sh goes red and the failing seams point at files you did not touch, FIRST split the breakage by git state: (a) COMMITTED-but-broken states from a parallel session (swap commits that forgot their seam/deployer twins) → repair them yourself in clearly-labeled follow-up commits; that is what kept the suite green twice. (b) UNCOMMITTED working-tree state (M/untracked files owned by a LIVE concurrent session — check git status --porcelain and untracked ?? entries against HEAD, which may not have moved) → do NOT edit those files; the other session is mid-flight and will reconcile env+seam+docs in its own commit; editing its dirty tree risks colliding with an active writer. Verify your own pushed state instead (rev-list HEAD...origin/main, targeted seams on your files) and report the named dirty files as the red-suite owner. Signature of case (b): HEAD identical across two suite runs yet different failures, because bats seams read the working tree, not the commit. _(context: 2026-10-05 goal session: frankbook e2e split (9c50821/2930431/1c725ea/5573ca1) interleaved with a live parallel session committing Ornith-swap breakage and later holding the tree dirty (M frank-small.env, M frank_small_unit.bats, ?? GGUF Compatibility.md).)_
 - Wrapper-script law (serverAppConfigs tests/e2e/run.sh): when converting a pure passthrough exec tool "$@" into "default argument set + passthrough", the default set must be APPENDED to "$@", not substituted for it — exec npx playwright test "${PROJECT_ARGS[@]}" silently DROPS every other arg (--list, --grep, file filters), so run.sh --list ran the entire 34-test suite instead of listing. The regression is only visible by diffing observed behavior against requested semantics; the seam (bash -n) cannot see it. Correct shape: build PROJECT_ARGS only when no user --project is present, then exec npx playwright test "$@" "${PROJECT_ARGS[@]}". Caught 2026-10-05 because the --list proof printed execution checkmarks instead of a listing; verify arg-forwarding wrappers by exercising EVERY documented passthrough arg, not just the happy path. _(context: 2026-10-05 frankbook e2e split; regression introduced by my own default-project-set rewrite and caught during verification (commit 1c725ea).)_
 - frank-small seam congruence rule (serverAppConfigs repo): ANY change to scripts/frank-small.env has TWO consumers that must flip in the same change: (1) scripts/deploy-remote.sh's LLM_SMALL_MODEL_FILE/LLM_SMALL_MMPROJ defaults (the payload-existence gate — test 196 greps for the exact env path inside the deployer default), and (2) tests/frank_small_unit.bats (TARGET_STUB/MM_STUB fixture names, the MODEL_FILE env assertion, and the KV-contract test pinning --cache-type-k/--cache-type-v values from the env). On 2026-10-05 two consecutive commits broke it: the Ornith-1.5-9B Q8_0 payload swap (77decb5) left both consumers on MiMo-Q6K paths, and the KV reprice (1862dc3, f16 V→q8_0 V) left the KV test asserting the old unquantized contract — its commit message even claimed the seam was 'already retargeted in 8f53df7' when 8f53df7 covered only payload paths. Symptom each time: full ./tests/run.sh red in frank_small_unit.bats only ('missing /tmp/.../mmproj-...' or argv-value mismatch). Fix = grep the env for the new values and update all three spots. _(context: 2026-10-05 goal session (frankbook e2e split) blocked twice by frank-small seam staleness from concurrent sessions; fixed via 8f53df7-equivalent edits and 5573ca1.)_
-- Wrapper scripts that inject a default argument set must keep "$@" in the final exec — dropping it silently converts user flags into no-ops. Measured 2026-10-05 in serverAppConfig tests/e2e/run.sh: the first default-project-set rewrite exec'd only the injected --project ... args, so ./run.sh --list silently RAN the full 34-test suite (~79 s) instead of listing — the mislabeled output then nearly passed off a full run as a listing (and ran a test against a host that was assumed asleep). Correct shape: build PROJECT_ARGS only when the user gave no --project, then exec npx playwright test "$@" "${PROJECT_ARGS[@]}" — user args first, defaults appended, explicit --project still takes over. Corollary on verification hygiene: condition-dependent pass claims ("green with the host asleep") require the condition observed at run time (ping/curl), not assumed — the Mac was awake during that run, so only the mechanism (default set contains no frankbook project, proven by --list composition) carried the claim. _(context: serverAppConfigs e2e frankbook split session 2026-10-05: run.sh default project set (frankbook excluded, E2E_INCLUDE_FRANKBOOK=1 opt-in) introduced by commit 1c725ea after the passthrough regression was caught by an E2E_INCLUDE_FRANKBOOK=1 --list invocation that ran instead of listing.)_