Compare commits

..
28 changed files with 652 additions and 368 deletions
@@ -1,61 +1,41 @@
--- ---
name: audit-subagent-deliverables name: audit-subagent-deliverables
description: "Integrate subagent-written code safely: line-by-line audit, mechanical gates (bash test/lint), adversarial-review-report triage, resource-safety pass (fd/memory), and cutover with running-process restarts." description: "Integrate subagent-written code safely: line-by-line audit, mechanical gates (bash test/lint), merge shared-function collisions, fix scope creep, verify env var propagation."
--- ---
# Audit subagent deliverables Integrating code written by subagents (fan-out implementation). The code is usually correct; the failures are structural — collisions between agents, scope creep, and assumptions that never reach the runtime.
A subagent that wrote code is a source of plausible output, not a verified change. Integrate it the way you'd integrate an untrusted PR — with extra suspicion, because the author is an agent that shares your blind spots and its prose is more confident than its evidence. ## Workflow
Order of operations: read everything → gates → tests (as evidence, not proof) → resource-safety pass → fix in-place → cutover. 1. **Read every deliverable line by line** before integrating — not the summary.
2. **Run the mechanical gates immediately**, in parallel: `bash -n` on every script; `bats tests/ 2>&1 | grep "not ok"` for the failure list, never `tail` (tail shows a passing tail and hides the red in the middle).
3. **Merge shared-function collisions.** When two agents add different versions of the same function, pick the superset and re-point the other's callers at it. (Real case: one agent's `fleet_base()` carried `-b`, the other's `resolve_fleet_base()` carried `--base-url`; resolve to the one flag set the tests use.)
4. **Delete scope creep.** Code added for a goal nobody asked for — a new env var, a new CLI flag, a new output file — goes, with its docs, unless the operator asked.
5. **Verify env var propagation.** A var the orchestrator computes must be exported before the `bash -c` / subshell that reads it, or the child silently takes a default. Check the export site, not the doc.
## 1. Read every produced file, line by line ## Audit checks
Full file, not a diff skim. You are the reviewer who must catch the line the author never thought about.
## 2. Mechanical gates before logic review | Check | How |
```bash |---|---|
bash -n <script> # syntax (scripts) | Flag names real | `bash script.sh --help` |
bunx tsc --noEmit # types (TS) | Paths exist | `test -e` each literal in the diff |
bun test # suite | Env reaches the runtime | grep the export/inline-assignment site, then the read |
``` | Tests assert behaviour | not `[[ -n "$X" ]]`, not `test -f <new file>` |
Run the suite for information, not as the stopping point — it only proves the author's own scenarios. | Tests not weakened | `git diff HEAD~ -- tests/` — a deleted assertion is a red flag |
## 3. Hunt the three classic agent defects ## Red flags
- **Quoting bugs:** `set -e` + `"${@/$p/$r}"` = substitution regex containing user input (bash can't escape `/` in a `${var/re/pat}` pattern: `/` reads as the pattern/replacement separator). Fix with a separator absent from inputs (e.g. `%`) and `printf '%q'` for shell-interpolated values.
- **`cd` assumptions:** parallel `runShell` calls share one CWD (pool mode). Any command sequence that assumes CWD (e.g. `git -C .` after a sibling's `cd`) must use explicit absolute paths.
- **Unbounded resources:** every fd opened must be closed on ALL paths (success, throw, validation fail) — `using` for fs.sinks (closes on throw; sinks are `AsyncDisposable`, not `fs.FileHandle`, no `.close()`); bounded `Map` for caches (`MAX` + FIFO eviction); explicit caps (lines read, file sizes).
## 4. Check the test file is real - Test asserting a new file exists rather than behaviour.
Tests that assert internal state (`expect(wcwd).toBe(...)`) over output contracts; tests of mock echo; tautologies. Delete them per house rules, keep behavior contracts. - Doc claiming a flag the script never defines.
- Function added with no caller.
- Doc example that contradicts the code it documents.
- Same helper defined twice with different flag spelling.
- New env var read but never exported.
## 5. Cutover audit (repo-wide) ## Re-verify after the operator's own edits
`grep -r` the symbol namespace the subagent touched — `wcwd`-style globals are the classic blast radius. Grep every `cd`/`process.chdir` site and verify each paired command sequence uses explicit paths. Stale comments describing the old design must go — comments that lie are time bombs for the next reader.
## 6. Cutover restarts running processes Run the gates again — their edit may undo an integration (a re-typed helper, a reverted flag).
A code fix on disk is NOT live in any process that loaded the file at spawn (bun/node load-at-spawn). After shipping fixes: restart every long-running supervisor/daemon/UI-server that carries the affected code path; prefer takeover protocols (stale-pid detection) that make restarts cheap. Symptom of forgetting: "fixed" code producing old-code symptoms in production (observed: 812 poisoned ledger rows).
## 7. Triaging an adversarial REVIEW report (not code) ## Report
When a subagent produced findings, treat each finding as a hypothesis with a base rate ~40% (measured over 3 rounds, 2026-10):
1. **Walk every finding end-to-end against the CURRENT tree** before accepting — cite file:line yourself. Reviewers frequently read a mid-edit snapshot; many findings are stale by construction.
2. **Classification that kills most findings:**
- *Stale read* — code changed during review; verify against live lines.
- *Inverted logic* — "failed rows are skipped" when the filter makes failed rows RETRY. Re-read the filter.
- *Unreachable* — trigger requires breaking a stated design law (single-worker, append-only). Not a defect.
- *By-design* — the reviewer labels a spec law a bug (serial execution, non-stream client). Check the spec, not the reviewer's confidence.
- *Constant-trust* — "XSS via manifest field" where the field is git-pinned config; "path traversal via REMOTE_TMP" where it's a hardcoded const. Ask: what untrusted input reaches this sink?
- *Self-refuting* — the reviewer's own probe output disproves its claim (raw==dedup probe in a "dedup" finding). Quote it back.
3. **Acceptance bar:** concrete trigger with real inputs + full path walked. Real accepts cluster in: resource caps (pids/timeout/awaited-cleanup), honest labels (default signatures), escaping discipline, self-healing probes.
4. **One-line verdicts + evidence per finding**, shipped in the commit message; calibration section ("genuinely solid") accepted as free verification of past fixes.
## Verification (the non-negotiable) Per deliverable: correct / changed / removed, with file:line, plus the gate output that proves it (`bats` failure list before and after).
Build/run the real artifact with its real inputs. Observe behavior at the seams:
- concurrent calls → no CWD races;
- resource ceilings → caps enforced;
- cleanup → no fd/pid/dir leaks;
- security → injection probes inert (quotes, slashes, metachars);
- normal case → output identical to pre-audit.
## Evidence (2026-10-07 + 2026-10-10/11 sessions)
- Shell-engine subagent: 426 LOC, `bun test` 39/39 green. Line-audit found: 5 leaked fds per call (`fd` sink never closed), CWD-race in `git -C .`, `set -e` + `${@/\/g\//}` slash-escaping bug, unbounded 10k-entry cache, 45s timeout on git-status.
- Review-triage census over 3 blind adversarial rounds (same repo, ~41 findings): accepted 9, rejected ~32 — rejects dominated by stale reads (protocol rewritten mid-review), inverted filter logic, unreachable-under-law triggers, base64-alphabet "quote injection" (charset has no quotes), SIGPIPE non-leaks, "no memory cap" quoting the `--memory 4g` line above the claim. Real accepts: fork-bomb pids cap, stalled-ssh upload cap, stale-flag self-heal probe, honest census label, freshness gate on pid reuse.
- Poison-law incident: sandbox fix shipped while supervisor ran old build → 812 rows graded by the dead protocol; takeover made recovery cheap. The restart step (#6) exists because of this.
@@ -1,49 +1,44 @@
--- ---
name: docker-pin-latest-census name: docker-pin-latest-census
description: "Prove a Docker Hub / lscr.io image pin is the latest version before bumping it: registry-API tag census plus digest-equals-latest verification, no docker CLI needed." description: "Prove a Docker Hub / lscr.io image pin is the latest version before bumping it: registry-API tag enumeration, the rolling-latest trap, digest pinning, and rollback proof."
--- ---
# Is my pinned image actually the latest? Use when asked whether a Docker Hub / lshr.io image pin is the latest version before bumping it, or when a manifest claims an image is stale. Registry API is the answer; the `latest` tag is not.
Use when bumping a pinned container image tag (e.g. `lscr.io/linuxserver/*` quadlet `Image=` lines) and "latest" must be PROVEN, not assumed. Works from any box with curl — no podman/docker daemon required. `lscr.io` fronts Docker Hub; its tags are NOT on quay.io (querying quay returns nothing and reads like "no newer tags exist" — false). ## The API
## 1. Full tag census via the registry API
```bash ```bash
tok=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:<owner>/<repo>:pull" \ # hub: token then paginate (the list is 100 per page)
| python3 -c "import json,sys;print(json.load(sys.stdin)['token'])") tok=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" | jq -r .token)
curl -s "https://registry-1.docker.io/v2/<owner>/<repo>/tags/list?n=2000" \ for off in 0 100 200 300 400 500; do
-H "Authorization: Bearer $tok" | python3 - <<'PY' curl -s -H "Authorization: Bearer $tok" \
import json,sys,re "https://hub.docker.com/v2/repositories/${repo}/tags?page_size=100&ordering=last_updated&page=${page}" \
tags=json.load(sys.stdin).get('tags') or [] | jq -r '.results[] | "\(.name)\t\(.last_updated)"'
# linuxserver form: <version>-<sha>-ls<N>; group by version, keep newest ls per version done | sort
v={}
for t in tags:
m=re.match(r'^(\d+(?:\.\d+)+)-([0-9a-f]+)-ls(\d+)$',t)
if m: v.setdefault(tuple(map(int,m.group(1).split('.'))),[]).append((int(m.group(3)),t))
for k in sorted(v)[-5:]:
print('.'.join(map(str,k)),'-> newest:',max(v[k])[1]) # max on (ls#, tag) tuple
PY
``` ```
- Sort versions as **tuples of ints**, never strings (`1.4.10 < 1.4.9` lexicographically).
- Keep the max `ls` build per version, then the newest version — both can lag independently.
- Trap already eaten: joining the tuple key to a string mid-loop then re-looking-it-up → `KeyError`. Keep keys as tuples; format only when printing.
- Some channels have NO `latest` tag in `/tags/list` (or the endpoint truncates) — absence there is not evidence.
## 2. Digest-proof the candidate against `latest`
```bash ```bash
for t in latest <candidate-tag>; do # lscr.io (linuxserver): tags are semver, `latest` is a rolling alias
curl -sI "https://registry-1.docker.io/v2/<owner>/<repo>/manifests/$t" \ curl -s https://images-api.github.com/v2/linuxserver/${repo}/tags/list # wrong host for lscr
-H "Authorization: Bearer $tok" \ curl -s "https://hub.docker.com/v2/repositories/linuxserver/${repo}/tags?page_size=100" | jq -r '.results[].name'
-H "Accept: application/vnd.oci.image.index.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.docker.distribution.manifest.v2+json" \
| grep -i '^docker-content-digest'
done
``` ```
Equal digests ⇒ the pinned tag IS what `latest` serves right now — the strongest "ensure latest" evidence, and recordable in the endpoint page's pin note (the dank-fleet law: *pinned; verified == the registry's `latest` manifest digest at pin time*). A multi-arch index digest matches only if both requests send the same Accept set — send all three types above. Docker Hub's own tag list (`hub.docker.com/v2/repositories/<repo>/tags`, ordered by `last_updated`) is the practical source. The registry API (`/v2/<repo>/tags/list`) needs the bearer token flow above; both return tags, neither sorts by semver.
## 3. Bump through git, not by hand ## The `latest` trap
Edit the quadlet/unit `Image=` line + the vault page's pin row (one owner each), run the repo suite, PR, deploy `--service <svc>` only. Rollback stays "revert one line". - `latest` is a **rolling alias**, not a version. "Pin is older than `latest`" says nothing about whether a *versioned* tag exists.
- `latest` can lag or be rebuilt without a version bump; comparing a digest pin against it proves nothing.
- For linuxserver images: `stable` is the rolling branch and version tags (`2026-09-30`) are what you pin; multi-arch manifest lists mean the same tag has per-arch digests — the pin must be the manifest-list digest (`RepoDigests[0]`) you actually pull, or a re-pull on another arch silently changes the image.
## Verification recipe (before bumping a pin)
1. Read the pinned tag and digest from the manifest/quadlet.
2. Enumerate tags from the registry API (paginate; `last_updated` is not semver order).
3. Compare **versions**, not recency: does a version greater than the pinned one exist?
4. If bumping: pull the new tag and record `docker/podman inspect --format '{{index .RepoDigests 0}}'` as the new digest — pin the digest, not the tag.
5. Prove the service: it starts, version endpoint answers the new version, no crash-loop, and the rollback is the previous digest.
## Report
"pinned: X (digest …), latest available: Y, upstream changelog: …" — or "pin is current: no version tag newer than X exists". Never claim staleness from the `latest` tag alone.
@@ -1,35 +1,60 @@
--- ---
name: docs-dedupe-verify-campaign name: docs-dedupe-verify-campaign
description: "Run an exhaustive docs dedupe + fact-verification campaign in dank/command-center: scout cluster ledgers, override scouts with evidence, gate every commit, and sweep scaffolding before PR." description: "Run an exhaustive docs dedupe + fact-verification campaign (command-center or any OKF vault, e.g. franksign): scout cluster ledgers, override scouts with evidence, gate every commit, and sweep scaffolding before PR."
--- ---
# Exhaustive docs dedupe + verification campaign (dank/command-center) # Exhaustive docs dedupe + verification campaign (dank/command-center; transplants to any OKF vault — proven on franksign 2026-10-21, #64 2026-10-21)
## Frame ## Frame
- Read `omp-docs/references/okf-spec.md` + root `omp-docs/index.md` first; inventory with `find omp-docs -name '*.md' | wc -l`. - Read `omp-docs/references/okf-spec.md` (or the vault's vendored OKF contract, e.g. `franksign-docs/read-only/okf-v0.2-spec.md`) + root `index.md` first; inventory with `find <vault> -name '*.md' | wc -l`.
- Run `scripts/kb-lint.sh omp-docs` baseline; index-coverage gate is `scripts/kb-coverage.py omp-docs` (each dir's `index.md` must link every sibling file/subdir). - Run `scripts/kb-lint.sh omp-docs` baseline; index-coverage gate is `scripts/kb-coverage.py omp-docs` (each dir's `index.md` must link every sibling file/subdir). On franksign the whole mechanical gate is one seam: `bun tests/lib/okf-lint.mjs` (frontmatter, links, index coverage, version-once) — run it before and after.
- Branch off fresh `origin/main` in a NEW worktree (`git -C ~/.omp worktree add ~/.omp/wt/<name> -b docs/<issue>-slug origin/main`) — workspace policy blocks `git checkout -b` in session worktrees. - Branch off fresh `origin/main` in a NEW worktree (`git -C ~/.omp worktree add ~/.omp/wt/<name> -b docs/<issue>-slug origin/main`) — workspace policy blocks `git checkout -b` in session worktrees. In a session worktree, `git checkout -b` is intercepted anyway: commit on the session branch, then named-ref push `HEAD:refs/heads/docs/<n>-slug` (verify `git log origin/main..HEAD` shows ONLY your commit first).
- Open the Gitea issue first (multi-moving-parts law); branch name `docs/<issue#>-<slug>`. - Open the Gitea issue first (multi-moving-parts law); branch name `docs/<issue#>-<slug>`.
## Scout fleet (overlap detection) ## Scout fleet (overlap detection)
- One `task` call, `tasks[]` batch, agent `scout`, read-only; one cluster per task (skills trees, llama trees, pi vs packages, niri, sleep/power, core, gitea, gitea+cross-cutting sweep). Cap 2 concurrent; queue rest. - One `task` call, `tasks[]` batch, agent `scout`, read-only; one cluster per task. **Fan out 2-3 clusters at a time, never seven**: `task.maxConcurrency` is 2 on this fleet, so the excess only queues, and `proc://` reports queued tasks as `running` — identical uptimes with zero deliveries means thrash, not progress (#64: seven scouts, nothing in 17 min; killing the five redundant ones let the two survivors land within minutes).
- Ledger format per cluster → `local://dedupe/<cluster>.md`: inventory table (file | lines | verdict KEEP|MERGE|DELETE|VENDOR|UPDATE), overlaps, facts-to-keep, inbound links, staleness. - Do NOT scout a cluster you have already audited inline. Kill the redundant jobs early with `write proc://<id>/kill` — the endpoint is the scarce resource, and your own session shares it.
- **Scouts lie.** Verified misses in the 2026-10-19 campaign: fabricated ledger entries ("peer-reviewed BrowseComp" claim that exists in no file), a DELETE verdict for a generated artifact that is checked in by design AND under test (`tools/niri/keybindings.md`, see `scripts/gen-keybinds.sh` + `tests/gen-keybinds.bats`), and a KEEP for a leftover the scout misread its own changelog about. NEVER apply a scout verdict without re-checking the file, the tests, and the changelog yourself. - Ledger format per cluster: inventory rows (file | verdict KEEP|MERGE|DELETE|VENDOR|UPDATE | evidence line/command). **Demand the ledger as the yield TEXT, never a `local://` file reference**: #64 produced one 0-byte ledger plus three files whose names contained the scout's own serialized parameter block (`gitea-caddy-inventory.md` + newline + a parameter tag), recoverable only by `ls`-ing `~/.omp/agent/sessions/<session>/local/<dir>/`.
- **Scouts lie.** Verified misses in the 2026-10-19 campaign: fabricated ledger entries ("peer-reviewed BrowseComp" claim that exists in no file), a DELETE verdict for a generated artifact that is checked in by design AND under test (`tools/niri/keybindings.md`, see `scripts/gen-keybinds.sh` + `tests/gen-keybinds.bats`), and a KEEP for a leftover the scout misread its own changelog about. Franksign 2026-10-21: 2 of 3 cluster scouts fabricated — "7 missing component stories" refuted by the pairing-law seam that runs green on every push, invented entry/page counts, DELETE verdicts on live-box census numbers the purpose test KEEPS; a third hit the 30-min runtime cap producing zero output. #64 same day: three pages "missing an H1" that have one on their first heading line; a 215-line `omp-docs/AGENTS.md` that does not exist; a "wikilinks + 700-char wall" defect in `core/settings.md` that a commit from that same morning had already removed; `core/subagent-limits.md` called superseded while `agent/config.yml` carries the cited `maxRuntimeMs` / `softRequestBudget` / `maxRecursionDepth` values verbatim.
- **Four pre-checks before a scout row enters the diff**: (1) the file exists; (2) the named heading/marker is actually absent (`grep -c '^# '`, `grep -c '^# Citations'`); (3) `git log --oneline -3 -- <file>` — a same-day commit may already have fixed it; (4) if the claim is "config supersedes the doc", grep the config for the cited keys.
- Cluster sizing: for a vault under ~20 pages, scout fleets are net-negative — audit inline yourself (reading 5 pages beats babysitting a scout). Reserve scouts for multi-hundred-page trees.
## Mechanical sweeps that pay (run on every vault)
- Empty-text links: `grep -rnE '(^|[ ,])\]\(' <vault>` — a generated "See Also" pass left five `- ](index.md)` links (#64); rendered, an empty bullet reads as style, never as a broken reference.
- Citation convention census BEFORE adding anything: count `grep -rl '^resource:'` against `grep -rl '^# Citations'`. command-center's convention is `resource:` frontmatter (12 vs 3) — "add Citations to six pages" is a convention change, not a fix; only the pages missing `resource:` were real hits.
- Long-line walls: `awk 'length>300 {print FILENAME":"NR"("length")"}'` — reflow prose to ~100 columns, wording intact; leave tables (they cannot wrap) and the text of verbatim blockquotes alone.
- Dated log headings: three `## YYYY-MM-DD` sections for one day is three logs wearing one date. Collapse to one `##` plus one `###` per campaign, and write the shape into the spec page so nobody has to infer it.
## Fact verification ## Fact verification
- Fleet facts: read-only probes over `ssh -o BatchMode=yes dankclaw|dankserver` (podman ps/inspect/top, stat, /etc/os-release, curl localhost APIs, `podman exec <c> <tool> --version`). Live state beats docs even when docs claim recent verification — the 2026-10-18 "audit" still drifted (zealot 27B→9B, build 10754→10777, container census 14→32). - Fleet facts: read-only probes over `ssh -o BatchMode=yes dankclaw|dankserver` (podman ps/inspect/top, stat, /etc/os-release, curl localhost APIs, `podman exec <c> <tool> --version`). Live state beats docs even when docs claim recent verification — the 2026-10-18 "audit" still drifted (zealot 27B→9B, build 10754→10777, container census 14→32; #64 then found `8C/16T` where SMT is off, a "masked" unit that is simply absent, Pi-hole missing from a DNS list, and a weekly-drifting engine count that became a size class with `/config` named as the only authority).
- Local box facts too: `pacman -Qs <pkg>` / `command -v` before trusting a page's "installed tool" claim (#64: `wlr-randr` documented as available, never installed).
- Unit health is not endpoint health: an `:7770` that answers `/v1/models` while its systemd unit sits inactive `Result=exit-code` means a container escaped its unit. Report the drift to the operator; never record transient state as docs, and never touch a production unit without a per-run OK.
- Code-bound facts (franksign-style): every doc claim about routes/columns/shipped status gets grepped against `apps/*/src`, `packages/db/migrations/` — docs narrating features that never shipped ("page ships in the slice" while only the API landed) get truthed with a status pointer to the owning issue.
- External claims: web_search/web read yourself (no subagents per Frank's rule). Kill fabrications outright (invented distro `FreeToken`, nonexistent `ornith-llama-cpp-setup` tree, unattributable "44.5% peer-reviewed CUA") — delete/drop with an explicit note, never keep "plausible" numbers. - External claims: web_search/web read yourself (no subagents per Frank's rule). Kill fabrications outright (invented distro `FreeToken`, nonexistent `ornith-llama-cpp-setup` tree, unattributable "44.5% peer-reviewed CUA") — delete/drop with an explicit note, never keep "plausible" numbers.
- Policy drift (live ≠ tuned docs): flag in the doc + PR body as an operator decision; NEVER silently re-apply policy. - Policy drift (live ≠ tuned docs): flag in the doc + PR body as an operator decision; NEVER silently re-apply policy.
- Deletion proposals from scouts die by default: archived trees carrying an honest archive banner (`pi/**`, banner 2026-10-18) stay, a directory `index.md` cannot be deleted (index-coverage gate), and incident logs are laws, not bookkeeping. Deletion is a pruning session with Frank present.
## OKF §9 / frontmatter audit (cheap wins, check every vault)
- Log entries filed under the wrong ISO-date heading (crossed midnight): cut the entry (`CUT N.=M @reg`), insert the new date heading above the old one (newest-first headings), paste the entry under it.
- Missing trust family (`generated` on machine-written pages) — lint may not catch it; grep every frontmatter block.
- Laws living only in `log.md`: promote each to its owner page (mechanism law → architecture page owning the thing), leave the log entry as the incident record; the HANDOFF/README boundary then points at the vault instead of restating.
- A law restated in two tiers (vault spec page + always-loaded `AGENTS.md`) is not a duplicate to cut: name the co-ownership in the spec page ("change one, change the other") instead of silently dropping one copy.
## Log compaction (log.md) ## Log compaction (log.md)
- Entry law: dated entry = what changed + durable lesson + concept-page pointer; narratives recoverable via `git log --follow`. Record the law at the top of the log. Tag historical-era entries. - Entry law: dated entry = what changed + durable lesson + concept-page pointer; narratives recoverable via `git log --follow`. Record the law at the top of the log. Tag historical-era entries.
- One `## YYYY-MM-DD` per day, `###` per campaign. Put the fan-out and ledger-verification lesson in the entry — that is what stops the next campaign from trusting a scout.
## Gates + landing ## Gates + landing
- Gate every commit: `bash scripts/kb-lint.sh omp-docs && python3 scripts/kb-coverage.py omp-docs && bats tests/`. Pre-commit hook prints `error: command not found: caddy` on Caddyfile-touching commits — warning only, commit still lands. - Gate every commit: `bash scripts/kb-lint.sh omp-docs && python3 scripts/kb-coverage.py omp-docs && bats tests/`. Pre-commit hook prints `error: command not found: caddy` on Caddyfile-touching commits — warning only, commit still lands. On franksign: kill the :3000 demo stack first (a live API hangs the gate ~10 min; law lives in `franksign-docs/architecture/testing.md`), then `bun tests/lib/okf-lint.mjs && tests/run.sh docs`.
- Before opening the PR: `git diff --stat origin/main..HEAD | grep -i 'tmp\|scratch\|\.bak'` — scaffolding merged under a `.tmp-` name in the 2026-10-19 campaign and needed a follow-up PR to promote. - Before opening the PR: `git diff --stat origin/main..HEAD | grep -i 'tmp\|scratch\|\.bak'` — scaffolding merged under a `.tmp-` name in the 2026-10-19 campaign and needed a follow-up PR to promote.
- PR via API (skill `gitea-api-pr-operations` for token-extraction + merge payload); `xdg-open` the PR the moment it opens; comment the campaign ledger on the issue. - PR via API (skills `gitea-api-pr-operations`, `gitea-merge-form-encoded`): token through a 600-perm `-K` curl config, never `-H` (`/proc/<pid>/cmdline` is world-readable); create/PATCH = JSON WITH Content-Type, merge = form body with NO Content-Type, `do=merge&merge_style=no-ff`, success = HTTP 200 empty body verified by landed state, not by re-reading the PR. `shred -u` the cfg and payload afterwards. `xdg-open` the PR the moment it opens; comment the campaign ledger on the issue — ownership-move table, re-read facts with the command that read them, and rejected findings with their evidence.
- After Frank's merge OK: propagate deployment `git -C ~/.omp fetch && git merge --ff-only origin/main`; reap campaign worktree (`git worktree remove --force`) + delete branch local/remote; close issue with landing comment. - After Frank's merge OK: propagate the deployment with `git -C ~/.omp fetch && git merge --ff-only origin/main`, but first prove the FF cannot touch live machine-tier state: `comm -12 <(git -C ~/.omp status --porcelain | grep -v '^??' | awk '{print $2}' | sort) <(git diff --name-only HEAD..origin/main | sort)` must print nothing, and re-check the dirty count after the merge.
- Worktree reaping is the live-sync heartbeat's job (`wt/*` older than 7 days) unless Frank says otherwise — no `rm`, no deleting a session branch you did not create.
## Line-edit op syntax trap (bit three times) ## Line-edit op syntax trap (bit five times)
- Valid ops: `PUT N.=M:` (body `+text` lines), `CUT N.=M` / `CUT N.=M @reg`, `PUT <N @reg`, `PUT >N:` / `PUT <N:`. There is NO `PUT N+=:` form — it lands in the file as a literal garbage line and shifts the insert to the wrong position. For gap-insert use a lone `PUT >N:` or `PUT <N:` per call; pre-compute line numbers against the file map (ops reference original lines within one call, but a failed op poisons positions). - Valid ops: `PUT N.=M:` (body `+text` lines), `CUT N.=M` / `CUT N.=M @reg`, `PUT <N @reg`, `PUT >N:` / `PUT <N:`. There is NO `PUT N+=:` form — it lands in the file as a literal garbage line and shifts the insert to the wrong position. For gap-insert use a lone `PUT >N:` or `PUT <N:` per call; pre-compute line numbers against the file map (ops reference original lines within one call, but a failed op poisons positions).
- **PUT with an empty body DELETES the range** (franksign 2026-10-21: a typo-fix `PUT 7-7:` with empty body deleted the whole entry). Bodyless deletes: use `CUT N.=M`.
- A `CUT`/`PUT` range outside the file errors cleanly (`out of range (file has N lines)`) — that is the cheapest way to learn a body was shorter than assumed; re-read the tail rather than guessing offsets.
- Bare body rows get auto-prefixed with `+` and a warning — harmless, but never rely on it: prefix every body row yourself.
- Editing a PUT body that ends adjacent to the next original line can silently consume the closing frontmatter `---` — after any frontmatter-range PUT, re-read lines 1–10 of the result.
- Recovery recipe when a literal op line lands: read exact bounds, `CUT N.=N` (junk), `CUT A.=B @block` (misplaced section), then `PUT <M @block` at the true anchor, re-read tail, add spacing blank if a `---` now abuts a list line. - Recovery recipe when a literal op line lands: read exact bounds, `CUT N.=N` (junk), `CUT A.=B @block` (misplaced section), then `PUT <M @block` at the true anchor, re-read tail, add spacing blank if a `---` now abuts a list line.
@@ -1,62 +1,29 @@
--- ---
name: fleet-service-runtime-census name: fleet-service-runtime-census
description: "Answer \"what is service running right now?\" for a dank-fleet service: live endpoint probe plus host units, image ids, live-vs-repo env, and drop-ins." description: "Answer \"what is service running right now?\" for a dank-fleet service: unit state, container image/digest, endpoint contract, loaded env — machine over repo, in that order."
--- ---
# Fleet service runtime census Answer "what is service running right now?" for a dank-fleet service without trusting the repo. Repo state is intent; the machine is truth, and they differ by however long since the last deploy.
Answer the question "what is `<service>` running currently?" with evidence, not from docs. Docs say what *should* run; the host says what *does*. Run all five probes, then report one table. ## Order of operations (cheap → expensive)
## 1. Probe the published port (works from any LAN box, no ssh) 1. **Unit state on the host** (rootless quadlet ⇒ `--user`; some units are root):
`ssh -o BatchMode=yes host 'systemctl --user show S -p ActiveState -p SubState -p ExecMainStartTimestamp -p NRestarts'`
Boot-loop tell: `NRestarts` climbing with a short `ExecMainStartTimestamp`.
2. **The container's own view**: `podman ps --filter name=S --format '{{.Image}} {{.Status}} {{.Names}}'`, then `podman inspect` for `Image` (tag **and** `RepoDigests`) and `Env`.
3. **The endpoint contract, from the service itself** — this is the only answer that matters to a consumer:
`curl -s host:port/v1/models` (served model id), `curl -s host:port/props` (`n_ctx`, `n_ctx_train`, caps), `curl -s -o /dev/null -w '%{http_code}' host:port/health`.
4. **Effective flags**: the live process is the truth. `systemctl --user cat S` shows `EnvironmentFile=`; read that file. A repo `scripts/S.env` can be ahead of, or behind, what the running unit loaded — and a unit restart is what re-reads it.
5. **Logs last**: `journalctl --user -u S -n 80 --no-pager` for the boot ledger / refusal messages, after you know whether it is even up.
```bash ## Facts that bite
curl -s -m 10 http://<host-ip>:<port>/health; echo
curl -s -m 10 http://<host-ip>:<port>/v1/audio/voices # per-endpoint extras: /v1/models, /metrics
```
The extra route matters more than `/health` for identity claims: - **A service is up ≠ configured how you think.** llama.cpp's `/v1/models` reports `max_model_len`, not the engine's per-slot ceiling; halogen reports `n_ctx` per slot. Read `/props` before quoting any window number.
- llama.cpp and radiance **echo the requested model name** — a 200 proves nothing; read `/v1/models`. - **`systemctl is-enabled` answers enablement, not activity.** A timerless service reports `linked`/`static`; `deploy.sh` that prints `ENABLE` for it is lying. Read `systemctl is-enabled --quiet` exit status, not the text.
- halogen 404s `/completion`, `/ping`, `/props`, `/tokenize`. - **`NRestarts: 0` with `ActiveState=activating` is a crash-loop only if the timestamps move**; check `ExecMainStartTimestamp` twice, 5 s apart.
- The frank-tts façade answers `/health` **without dialing the engine**, so "façade healthy, engine down" is a real state; the voice list in that same response is the honest census. - **Read-only is always allowed** from the dev box (`ssh -o BatchMode=yes`); state changes on remote boxes are not, without a per-run OK. Production services (`frank-archon`, `frank-dragoon`, `frank-zealot`) are never stopped/restarted to "see what happens".
- **Digest-pinned images**: "when was this image updated?" is answered by `podman inspect` `RepoDigests` + the registry API, not by the unit's start time (a restart re-uses the local digest).
## 2. Units + containers on the host (ssh as the deploy user, no sudo) ## Report
```bash One line per layer with the evidence: unit (`active (running)`, since, restarts), container (image + digest), endpoint (served id, `n_ctx`, health code), loaded env (file path + the knobs that matter), and explicitly which of those came from the machine versus the repo — that gap is the answer to most "why is it behaving like that" questions.
ssh <host> 'systemctl --user status <svc>.service <engine>.service --no-pager \
| grep -E "^(●| Active| Loaded)"; \
podman images --format "{{.Repository}}:{{.Tag}} {{.ID}} {{.Created}}"; \
podman ps -a --format "{{.Names}}\t{{.Status}}\t{{.Image}}"'
```
- `active (running) since …` + `Up N hours (healthy)` is the answer; `activating (auto-restart)` with a ~50 ms exit is a **preflight refusal**, not an engine fault.
- **NEVER prefix these podman calls with `sudo`.** Containers live in the deploy user's rootless storage; `sudo podman inspect frank-tts-engine` returns `no such object` while the plain form succeeds. Treating that as "container missing" is a false finding.
## 3. Drop-ins — the override that no health gate sees
```bash
ssh <host> 'systemctl --user show <svc>.service -p DropInPaths; \
systemctl --user show <engine>.service -p DropInPaths'
```
`DropInPaths=` must be empty. A leftover bench drop-in (`Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env`) makes the next deploy boot a different model at a different shape while `/health` still answers 200 and the e2e gate passes.
## 4. Live env vs repo env — the only truth test
```bash
# repo side (local checkout)
grep -vE '^\s*#|^\s*$' scripts/<svc>.env
# host side
ssh <host> 'sudo cat /home/dank/serverAppConfigs/scripts/<svc>.env' | grep -vE '^\s*#|^\s*$'
```
Diff them. Per repo law the env file beside the serve script **is** the authority; a doc page that disagrees with it is the bug, and host-side drift means an uncommitted edit. Also check `git log --oneline -3 -- scripts/<svc>.env` to confirm the host isn't ahead of git.
## 5. Report
Table of: unit state + since, image repo:tag@short-ID + age, backend/alias actually served (from step 1, not from the name), drop-ins, and whether live env == repo env. State explicitly when a fact came from docs rather than from a probe.
## Traps that produce false findings
- Digest-pinned images: the local image ID's *creation date* answers "is this stale" better than the tag; `latest` + `Pull=never` is a locally built image, not a registry pull.
- `ssh` user context changes what podman sees (root vs rootless) — see step 2.
- Two ports on one host are two providers in `models.yml`; a port mismatch is a rename or drift, not a config typo.
@@ -1,35 +1,37 @@
--- ---
name: frank-job-hunt-maintenance name: frank-job-hunt-maintenance
description: "Maintain frank-job-hunt agents: live-census gate hardening loop, registry locator verification, and the workstation/Dankserver/Dankclaw deploy-sync ritual." description: "Maintaining frank-job-hunt: the census gate's fail-closed rules, ATS locator token derivation, and the two-host deploy-sync verification."
--- ---
# frank-job-hunt maintenance Use when maintaining `frank-job-hunt`: the live-census gate, the registry locator, or Dankserver/Dankclaw deploy-sync. The repo is built and deployed; agents live on Dankserver, ntfy on Dankclaw.
Repo `~/Projects/frank-job-hunt` (Gitea `dank/frank-job-hunt` on Dankclaw 192.168.0.143:3000). Production: Dankserver user timers `frank-job-sweep.timer` 07:30, `frank-job-digest.timer` 08:15 (`Linger=yes`). ntfy relay: Dankclaw quadlet `ntfy.service`, route `ntfy.franksdank.cloud` (no anubis), LAN record in `hosts/custom.list` (Pi-hole v6 file at `/home/dank/Containers/pihole/etc-pihole/hosts/custom.list`; v6 API paths are hostile — edit the file, then `podman exec pihole pihole restartdns`). Secrets: `~/.config/frank-job-hunt.env` (600, workstation + Dankserver). ## The two rules that caused every bug here
## Law (validated 2026-10-11) 1. **A URL is data.** The census gate and the registry locator treat portal URLs as opaque strings; nothing may template a URL from a company name. `jobs.dankclaw.example` is a fixture host (RFC 2606) — never derive a real domain from a company name, and never click a link the pipeline constructed.
- The fleet judge (`frank-zealot`, dankclaw:7771) is a **ranker, never a gate** — it inflates anything reaching it (non-technical roles scored 88-91). All discrimination lives in deterministic gates (`agents/lib/gates.ts`); zealot only ranks survivors. 2. **A stale row is worse than a missing row.** Every gate must fail closed: an unverifiable posting is reported as unverifiable, not kept as live. Deleting is allowed; asserting is not.
- Thinking-model fault: verdict lands in `reasoning_content` with `content:""` → judge sends `chat_template_kwargs:{enable_thinking:false}` (0.7s clean JSON).
- **Family beats topic**: role family (recruiter/sales/finance/PM/designer/people-ops/revops/…) gates `gate:scope` unconditionally — no infra-token override ("DevOps Recruiter" is recruiting). Compound technical titles keep the `INFRA_TITLE` override via `NEG_TITLE` branch.
- **gate:region**: remote is US-only (visa reality). "Remote - Bangalore" = onsite abroad → gate; `US` regex pass wins over `NONUS` match ("U.S. & Canada" passes).
## Gate hardening loop (live-census, TDD) ## Census gate
1. `bun agents/sweep.ts --dry` (full lane ≈ 30–90 s; 852 postings ≈ 62 pass / 790 filed).
2. Classify leaks in the top-10: family gap / region gap / scoring tail. Accept long tail (Director/Founder edge cases) — digest + human curation is the design.
3. Token fix in `gates.ts` regexes (`NEG_FAMILY` = no override; `METRO`/`US`/`NONUS` for arrangement). NEVER dedupe-check by memory — read file first; the file has bitten line-offset edits repeatedly.
4. Pin every leak with a named regression test in `agents/test/gates.test.ts` before/with the fix.
5. `bun test agents/test/ && bun x tsc --noEmit` — hooks re-run these at commit.
6. Ship: `git commit` (pre-commit = tests + tsc + docs-link + secret scan) → `git push` → `ssh dankserver 'cd ~/Projects/frank-job-hunt && git pull'` (timers pull from master; Dankserver drift = tomorrow's run is stale).
## Registry verification (monthly, `agents/verify-registry.ts`) `scripts/census-gate.sh` decides keep / stale / unverifiable per row.
Scouts' `ats`/`ats_identifier` are guesses (14/84 were wrong). Ritual:
1. `bun agents/verify-registry.ts` (dry, ~70 s serial) → REPAIR census.
2. `--write` applies; repairs with live jobs preferred over empty boards (0-job boards 404 later).
3. Unresolved = honest manual lane (vanity Workdays, self-built ATS incl. majors, neogov) — NOT a code bug; per-entry error census in sweep output is the same signal.
4. Update `targets/README.md` census lines + `docs/log.md`; commit/push/pull (same ship step).
## Pitfalls - Live proof = HTTP GET on the posting URL against the ATS API or the page itself; `404`/`410` ⇒ stale, non-200 with no body ⇒ unverifiable. Never infer from age alone.
- `--ats=<x>` smoke filters the census line — "N API-sweepable" reflects the filter, not the registry; don't diagnose a phantom bug. - The gate runs against the *live* registry, so a row that passes here and then fails in production means the registry drifted — re-run before blaming the gate.
- Workday vanity careers URLs (`careers.uw.edu`) are NOT CXS-API-resolvable; tenant-only identifiers are (site path required). - Test seams: fixture ATS server + `…_BASE_URL` env. Tests assert the row's fate (kept / dropped / flagged), never internal state.
- Root `README.md` vs `targets/README.md` name collision in edit anchors: use full path headers.
- Zealot endpoint probe: `curl http://192.168.0.143:7771/v1/models`. ## Registry locator
Resolves which ATS a company uses (Greenhouse `boards-api.greenhouse.io/v1/boards/<token>/jobs`, Lever `api.lever.co/v0/postings/<token>`, Ashby `api.ashbyhq.com/posting-api/job-board/<token>`, Workable, SmartRecruiters).
- Token ≠ company name: it comes from the discovered posting URL, so a locator change must keep the derivation (URL → token), not substitute a slug.
- Unknown ATS ⇒ unverifiable, not "no jobs". "Zero jobs" is a strong claim; only a successful API answer with an empty list may say it.
- Rate limits: 429 ⇒ back off and mark unverifiable for this pass, never stale.
## Deploy-sync (Dankserver agents, Dankclaw ntfy)
- Two hosts, two roles: inference/agents on Dankserver, notification sink on Dankclaw. A deploy that only touches one leaves the pair inconsistent — the agent can post to a sink that no longer accepts its token, silently.
- Deploy is `scripts/deploy.sh` from the correct checkout (dev side, clean tree); it pushes the branch first, so a stale local `main` makes the push fail — `git merge --ff-only origin/main` before deploying after any externally merged PR.
- Verify after deploy on the *served* surface: the agent answers a probe, and a test ntfy message arrives on the phone. "The unit is active" is not the proof.
## Tests
Black-box, fixtures over network: fake ATS servers on localhost, fake ntfy sink, temp registry file. Assert output rows and exit status. Never assert that an env var is set or that a helper was called.
@@ -19,6 +19,8 @@ for p in (home/".config/secrets").iterdir():
if m: vals.add(m.group(1)) if m: vals.add(m.group(1))
known[p.name] = {v for v in vals if len(v) >= 12} known[p.name] = {v for v in vals if len(v) >= 12}
``` ```
Also add every secret the repo itself embeds (e.g. credentials living in a *test helper*)
by concatenating its quoted fragments, then search for the joined value.
Report ONLY `value[:4] + len + sha256[:10]`. Never print a value, and never quote a Report ONLY `value[:4] + len + sha256[:10]`. Never print a value, and never quote a
secret-shaped literal in a doc/commit message — the pre-commit gate will (correctly) block it. secret-shaped literal in a doc/commit message — the pre-commit gate will (correctly) block it.
@@ -41,6 +43,19 @@ git status --porcelain -uall --ignored=matching # !! lines: on-disk ignored fi
git config --list --show-origin # local/global/system credentials git config --list --show-origin # local/global/system credentials
git ls-files | grep -E '(auth\.json|trust\.json|\.env|\.pem|\.key|identity)' # quarantine git ls-files | grep -E '(auth\.json|trust\.json|\.env|\.pem|\.key|identity)' # quarantine
``` ```
**Invocation forms that actually work** (each of these cost a wasted round-trip):
- `git grep --all -F '<fragment>'` — the options MUST precede the pattern; `git grep -l -F '<frag>' --all`
treats `--all` as a pathspec and silently returns 0 lines. Grepping a *fragment* (not the full
value) is what catches split-concatenation disguises like `"Assf.""1234"`.
- `git ls-tree -r --format=... <rev>` works for ONE rev; pass several revs and it emits nothing,
no error. For sha→path across all refs use `git rev-list --objects --all`, which yields
`"<sha> <path>"` lines directly.
- `git cat-file --batch` fed with `rev-list --objects` lines echoes `sha path` headers, so naive
header parsing misaligns and content is lost. Feed bare SHAs in chunks, or use
`git cat-file --batch-all-objects --batch` and parse `"<sha> blob <size>\n<content>\n"` records.
- Check every worktree/checkout: `git -C <other-checkout> rev-parse HEAD` often differs from the
audited worktree, so the leak may exist on `main` while the audited tree looks clean.
LFS: scan `.git/lfs/objects/**` as BYTES (`decode("latin-1")`) — a screenshot is a classic LFS: scan `.git/lfs/objects/**` as BYTES (`decode("latin-1")`) — a screenshot is a classic
secret carrier and the tracked tree holds only pointers. secret carrier and the tracked tree holds only pointers.
@@ -53,6 +68,9 @@ Flag every one whose value is in `known` as a LIVE leak; list the rest for triag
- prose: `Set password: user.set_password(...)`, `authenticate(user, password)` - prose: `Set password: user.set_password(...)`, `authenticate(user, password)`
- 40-hex hits that are git object ids / packfile names (`pack-<40hex>.pack`) - 40-hex hits that are git object ids / packfile names (`pack-<40hex>.pack`)
- memory/KB records that *describe* a secret operation - memory/KB records that *describe* a secret operation
- length-threshold gates that miss short passwords: a check like
`len(val) >= 24 and re.fullmatch(...)` clears a real 12-char password, so a passing
secret-scan test proves nothing about short credentials
A hit is a leak only when the value matches a live secret, decodes to a real credential, or A hit is a leak only when the value matches a live secret, decodes to a real credential, or
sits in a field a program reads. sits in a field a program reads.
@@ -61,8 +79,12 @@ Check file modes of the secret dir (`0600`, dir `0700`) — world-readable secre
most common real finding. Then grep the live values across shell rc files, `~/.gitconfig`, most common real finding. Then grep the live values across shell rc files, `~/.gitconfig`,
`~/.npmrc`, `~/.netrc`, `~/.curlrc`, `~/.docker/config.json`, `~/.cache/huggingface/token`, `~/.npmrc`, `~/.netrc`, `~/.curlrc`, `~/.docker/config.json`, `~/.cache/huggingface/token`,
`~/.aws/credentials`, `~/.ssh/config` and report filenames only. `~/.aws/credentials`, `~/.ssh/config` and report filenames only.
Also check files the deployer *generates* on hosts: a script that writes a credential file
without `chmod 600` (while sibling writes in the same script do chmod) is the same class.
## 6. Report ## 6. Report
Per layer: scope numbers + verdict. Every hit: path:line, redacted value, verdict Per layer: scope numbers + verdict. Every hit: path:line, redacted value, verdict
(leak / benign + why). Name the gate's coverage gap if the project's own hook misses shapes (leak / benign + why). Name the gate's coverage gap if the project's own hook misses shapes
you matched. Fix modes/chmod on the spot; changes to a security hook are the operator's call. you matched. Fix modes/chmod on the spot; changes to a security hook are the operator's call.
A secret in git history is only removable by rotation or history rewrite — say so, don't
imply an edit fixes it.
@@ -0,0 +1,27 @@
---
name: gitea-silent-ignore-patch-trap
description: "Gitea API answers success codes while silently ignoring mutation bodies: labels in PATCH /issues/N, and any JSON body sent without Content-Type: application/json. Verify mutations with a follow-up GET, never the status code."
---
# Gitea silent-ignore family (issue mutations)
Two verified failure modes on the Dankclaw Gitea API (1.27.3, verified live 2026-10-21 on `dank/franksign`); both **return a success code and change nothing**:
1. **`PATCH /repos/{o}/{r}/issues/{n}` with `{"labels":[...]}`** — 200, labels untouched, even *with* a proper JSON Content-Type. Labels move only via the dedicated endpoints:
- add: `POST /issues/{n}/labels` with `{"labels":[<id>]}`
- replace: `PUT /issues/{n}/labels` with `{"labels":[ids]}` (works without Content-Type)
- remove: `DELETE /issues/{n}/labels/{id}`
2. **Any JSON body without `header = "Content-Type: application/json"` in the curl config** — e.g. closing with `PATCH {"state":"closed"}` returns **201 and no-ops**; identical request with the header closes. curl's default `-d` type (`application/x-www-form-urlencoded`) makes this silent.
## Rules
- Mutating JSON calls: always send `Content-Type: application/json` alongside `Authorization: token` in the same `-K` config file.
- **Never trust the status code for mutations — follow up with `GET` and read the field** (`labels`, `state`). Both traps look like success otherwise.
- Token stays out of `-H` args (world-readable `/proc/<pid>/cmdline`): extract from `~/.config/secrets/gitea` into a 600-perm curl config, use `-K`.
## Quick verification idiom
```bash
curl -sK "$cfg" "http://192.168.0.143:3000/api/v1/repos/dank/franksign/issues/22" \
| jq '{state, labels: [.labels[].name]}'
```
Franksign's vault page `franksign-docs/agents/issue-tracker.md` owns the repo-local copy of this law; this skill exists so other repos (serverAppConfigs, command-center) don't re-burn the cycle.
@@ -13,7 +13,10 @@ http://dank:<40-char-token>@192.168.0.143%3a3000
```bash ```bash
tok=$(sed -nE 's#^[a-z]+://[^:]+:([^@]+)@.*#\1#p' ~/.config/secrets/gitea | head -1) tok=$(sed -nE 's#^[a-z]+://[^:]+:([^@]+)@.*#\1#p' ~/.config/secrets/gitea | head -1)
curl -s -K <600-perm cfg file> http://192.168.0.143:3000/api/v1/<path> echo "tok len ${#tok}" # must be 40 — assert before any call
cfg=$(mktemp); chmod 600 "$cfg"; printf 'header = "Authorization: token %s"\n' "$tok" > "$cfg"
curl -s -K "$cfg" http://192.168.0.143:3000/api/v1/<path>
rm -f "$cfg" # in the SAME session: the file holds a live token
``` ```
## `git credential-store --file … fill` answers EMPTY — do not debug it ## `git credential-store --file … fill` answers EMPTY — do not debug it
@@ -24,20 +27,21 @@ Tried 2026-10-21 while opening `dank/command-center` PR #61: the obvious extract
printf 'protocol=http\nhost=192.168.0.143\nport=3000\n\n' | git credential-store --file ~/.config/secrets/gitea fill printf 'protocol=http\nhost=192.168.0.143\nport=3000\n\n' | git credential-store --file ~/.config/secrets/gitea fill
``` ```
prints **nothing**, with no error. It is not a permissions problem and not a wrong path: the stored line encodes the port inside the host field (`@192.168.0.143%3a3000`), so a `host=`+`port=` query can never match it. Add `path=` or drop `port=` and it still answers empty. Read the file with `sed` (recipe above) and move on — the chain `TOKEN=$(…) && [ -n "$TOKEN" ] && …` fails silently at the test, which reads like a curl failure if the `&&` hides it. prints **nothing**, with no error. Not permissions, not a wrong path: the stored line encodes the port inside the host field (`@192.168.0.143%3a3000`), so a `host=`+`port=` query can never match it. Add `path=` or drop `port=` and it still answers empty. Read the file with `sed` (recipe above) and move on — the chain `TOKEN=$(…) && [ -n "$TOKEN" ] && …` fails silently at the test, which reads like a curl failure because the `&&` hides which link broke.
## The silent-empty-token trap (bit 2026-10-21, cost a debug detour) ## The silent-empty-token trap (bit 2026-10-21, cost a debug detour)
An `awk -F= '/^username=/{...}'` parse of this file yields `tok=""` **silently** — and reads still appear to work when the repo is **public**: `dank/serverAppConfigs` and `dank/command-center` return real issue/label data unauthenticated, so every census "passes" until the first **write** (issue comment, label, PR create/merge) fails with `{"message":"token is always required"}`-style errors or 404s (private repos 404 on an empty token — that is an auth problem, NOT a missing repo). An `awk -F= '/^username=/{...}'` parse of this file yields `tok=""` **silently** — and reads still appear to work when the repo is **public**: `dank/serverAppConfigs` and `dank/command-center` return real issue/label data unauthenticated, so every census "passes" until the first **write** (issue comment, label, PR create/merge) fails with `{"message":"token is always required"}`-style errors or 404s (private repos 404 on an empty token — that is an auth problem, NOT a missing repo).
- `echo "tok len ${#tok}"` (must be 40) before believing ANY API result — reads included. - `echo "tok len ${#tok}"` (must be 40) before believing ANY API result — reads included.
- `curl -s -K <600-perm file>` carrying `header = "Authorization: token $tok"` — never `-H` on argv (`/proc/<pid>/cmdline` is world-readable; this leaked an HF token once).
- Prove auth once per session: `GET /api/v1/user` → `"login":"dank"`. - Prove auth once per session: `GET /api/v1/user` → `"login":"dank"`.
- The temp `-K` config holds a live token: `rm -f` it in the same session that made the calls, and never `cat` it. - `curl -s -K <600-perm file>` carrying `header = "Authorization: token $tok"` — never `-H` on argv (`/proc/<pid>/cmdline` is world-readable; this leaked an HF token once).
- `jq -n --arg t … --arg h … --arg b "$(/bin/cat /tmp/pr.md)" '{…}'` — `$(cat file)` gets optimised away in this harness, so use `--rawfile b /tmp/pr.md` instead.
## Other notes ## Other notes
- `git credential fill` fails headless here (`could not read Username … terminal prompts disabled`); read the store file directly instead. - `git credential fill` fails headless here (`could not read Username … terminal prompts disabled`); read the store file directly instead.
- Never print the token: mask when inspecting the file — - Never print the token: mask when inspecting the file —
`sed -E 's#(://[^:]+:)[^@]+@#\1REDACTED@#' ~/.config/secrets/gitea` `sed -E 's#(://[^:]+:)[^@]+@#\1REDACTED@#' ~/.config/secrets/gitea`
- Merge writes need form-encoded fields, not JSON — see `gitea-pr-api-cycle`.
- Issue/PR numbers are **per repo**: `dank/dankvoice` #8–#11 (refactor, hardening, AbortError fix, menu defects) are unrelated to `dank/command-center` #8–#11 (SDLC, trunk-law). Read the body from the repo the work targets before referencing an issue in a commit message. - Issue/PR numbers are **per repo**: `dank/dankvoice` #8–#11 (refactor, hardening, AbortError fix, menu defects) are unrelated to `dank/command-center` #8–#11 (SDLC, trunk-law). Read the body from the repo the work targets before referencing an issue in a commit message.
@@ -1,41 +1,48 @@
--- ---
name: gitignore-trailing-slash-symlink-contamination name: gitignore-trailing-slash-symlink-contamination
description: "Trailing-slash gitignore rules don't match symlinks, so worktree dependency symlinks get committed; detect with git ls-tree mode 120000, fix with git rm --cached." description: "Trailing-slash gitignore rules don't match symlinks, so worktree dependency symlinks get tracked and pushed as 1-line blobs; same-name rules silently match worktree dirs, and .deploy-stale pollutes. Three fixes with tests."
--- ---
A `.gitignore` entry written with a **trailing slash matches directories only**. A symlink of the same name is not a directory, so it slips past the rule, and `git add -A` in a `git worktree` checkout silently **tracks** it. Cost measured 2026-10-19: a machine-specific symlink (`node_modules -> /home/blank/Projects/dankvoice/node_modules`) landed in `main` with a merged PR. Gitignore semantics trap with three real-world consequences (serverAppConfigs, 2026-10-19):
Why worktrees are the trigger: a worktree has no `node_modules` (or other installed-deps dir) of its own, so the cheap way to make build/test work there is a symlink into the primary checkout. With `node_modules/` ignored, nothing complains. ## The rule
## Detect `.gitignore` patterns ending in `/` match directories only. Worktree-relative: `**/node_modules/` is `**/node_modules/` → a symlink named `node_modules` does NOT match, so it is "untracked" no more — it gets added, committed, and pushed as a 1-line blob containing its target path.
## Consequence 1: worktree symlink contamination
- `git add -A` then commit then push in a worktree whose root holds `node_modules -> ...` (the worktree convention symlink) commits it.
- The repo then holds `package/frontend/node_modules` as a text file containing the absolute path on the author's box.
- The receiving worktree gets that 1-line file instead of the symlink → "module not found" and "why is this a 30-byte file?".
- Detection: `git ls-tree -r origin/main | awk '$1=="120000"'` (mode 120000 = symlink) or `git ls-files -s | grep 120000`.
- Fix: ignore rules WITHOUT trailing slash (`**/node_modules`, `**/.venv`) so symlinks match too.
## Consequence 2: same-name worktree rules silently matching worktree dirs
A root rule like `wt-*` (meant to ignore local worktree checkouts) matches every path whose last segment starts with `wt-` — so legitimate content under such a directory goes untracked. `git status --ignored` is the only visible trace; detection: `git check-ignore -v <path>` prints the owning rule + line number. (Repo-specific: `dank/command-center`'s `.gitignore` carries a "per-worktree memory buckets" block ignoring `agent/memories/sharpshooter/wt-*/` and `agent/memories/summaries/wt-*/` — intentional there.)
## Consequence 3: `.deploy-stale`
`deploy.sh` renames a stale tree to `<path>.deploy-stale` before recreating it. Without an ignore rule those directories land in `git status` as untracked and get swept into `git add -A` commits; they accumulate on every server.
## Fixes with tests
```gitignore
**/node_modules
**/.venv
.deploy-stale/
```
Black-box test against the real rules, not a fixture: `git check-ignore -v <symlink-target>` must name the rule. `git check-ignore` takes hypothetical paths — nothing needs to exist on disk:
```bash ```bash
git ls-tree HEAD -- node_modules # mode 120000 == tracked symlink git check-ignore -v natives/x.wasm dotfiles/x/.config/noctalia/Noctalia.json
git ls-files -s | awk '$1==120000' # every tracked symlink in the index
``` ```
## Fix (branch, never on main directly) Two ignore surfaces must be kept in sync (deploy exclusions vs ignore rules): if a deployer excludes `*/Noctalia*` but the ignore list has only the lowercase class, generated files become tracked-but-never-deployed — test both halves with `git check-ignore`.
```bash ## Recovery (if contamination already pushed)
git rm -q --cached node_modules # untrack; leaves the file on disk
```
plus relax the ignore rule to the slash-free form, with the reason inline so nobody re-adds the slash: 1. `git rm --cached <path>` on the symlink blob, commit, push.
2. Fix the ignore rule first (else the next `git add -A` re-adds it).
``` 3. Untracked ≠ deleted from history: the blob stays in every old commit. Only a rewrite removes it; that is the operator's call, not a cleanup run alone.
# Trailing slash would only match directories: a node_modules symlink (the
# shape `git worktree` checkouts end up with) slips past it and gets tracked.
node_modules
```
Then `git check-ignore -v node_modules` must print the rule, and the suite runs after (the symlink still resolves, it is just ignored).
## Prevent
In a fresh worktree, create the deps symlink **after** confirming the ignore rule has no trailing slash:
```bash
git check-ignore -v node_modules || sed -i 's|^node_modules/$|node_modules|' .gitignore
```
Same class for any ignored-by-convention dir that a worktree might provide as a link: `node_modules`, `vendor`, `.venv`, `target`, `dist` when built artifacts are ignored.
@@ -1,44 +1,47 @@
--- ---
name: grafana-provisioned-dashboard-verify name: grafana-provisioned-dashboard-verify
description: "Author and verify repo-provisioned Grafana dashboards on Dankclaw live: staging to the host provider path, the Grafana-13 unified-storage false-negative on grafana.db, the authenticated /api/search check, per-panel PromQL/LogQL validation before shipping, and the Loki $__auto shell-expansion trap in test harnesses." description: "Author and verify repo-provisioned dashboards on Dankclaw live: Grafana 13 has no unauthenticated API and no local admin, so prove through the e2e SSO browser path, and close the copier gap that hides never-provisioned boards."
--- ---
Verify or ship Grafana dashboards that are provisioned from the repo (`shared/grafana/provisioning/dashboards/*.json`, auto-loaded by the `fleet` file provider). Use when adding/staging a dashboard, or when asked to confirm provisioned dashboards actually load. ## When to use
You changed `shared/grafana/provisioning/dashboards/*.json` (or panel PromQL) and must prove the LIVE boards actually receive data — or a panel/whole board reads empty and you must classify: dead expression vs never-provisioned vs legitimately-empty.
## Paths (Dankclaw) ## The two gaps this skill exists for
- Repo source: `shared/grafana/provisioning/dashboards/` (git is the owner). 1. **The copier gap.** A repo dashboard JSON is NOT a live dashboard until the deployer carries it. Until 2026-10-21 `copy_grafana_provisioning` in `scripts/deploy-remote.sh` named `llm-fleet.json` alone: the `containers` board was merged, seam-tested, and documented as "working" yet had NEVER been provisioned; the other boards existed on disk (`/opt/grafana/provisioning/dashboards/`) only via one-time hand-copies, and one had silently gone stale. Fixed: copier globs `dashboards/*.json`; `GRAFANA_PROV_ROOT` is the test seam; a bats case executes the copier for real into a sandbox.
- Host provider dir: `/opt/grafana/provisioning/dashboards` (root-owned), bind-mounted `:ro` into the container at `/etc/grafana/provisioning/dashboards` (from `dankclaw/observability/quadlet/grafana.container`: `Volume=/opt/grafana/provisioning:/etc/grafana/provisioning:ro`). 2. **The API-proof illusion.** The vault used to say "verify through the APIs". On Grafana 13.2.2 that is dead: `/api/health`, `/api/frontend`, `/api/org`, `/login` all 404 **unauthenticated**, and there is NO local admin password (root `.env` sets no `GF_SECURITY_ADMIN_*`; the only account is the Authentik RP login). Do not hunt for credentials; use the browser path below.
- Provider reload interval = 30s, so a staged file appears within ~35s.
## Stage on the host (canonical install is `deploy.sh dankclaw --service grafana` after merge) ## Procedure
### 1. Mechanical expression census (over SSH — Prometheus binds loopback)
Extract every panel `expr` from the repo JSONs with python, substitute template vars (`$host`,`$container`,`$job` → `.*`, `$__auto`/`$interval` → `5m`), and run each as an instant query against `http://127.0.0.1:9090/api/v1/query` via `ssh dankclaw`. `curl` to `127.0.0.1:9090` from the dev box is blocked; keep queries on-host. Watch zsh globbing on `?`/`[` — quote the remote command.
Empty result ≠ defect automatically:
- `ALERTS` panel empty = **healthy** (ALERTS carries only firing/pending series).
- Zero by design: no cgroup limits exist on the fleet → a memory-vs-limit panel is dead by premise (rewrite as % of host total, don't delete).
- Real join defects: cadvisor `host` label vs node-exporter `job` label sets are incompatible — fix with `group_left label_replace(node_memory_MemTotal_bytes{job=~"node-.*"}, "host", "$1", "job", "node-(.*)")`, and `count by (job) (expr) / on(job) count by (job) (...)` needs the `on(job)` matcher when label sets differ. Parens around `on(host)` after `group_left` are **illegal**; `group_left label_replace(...)` without them parses fine.
### 2. Prove the provisioned files (filesystem = installation proof only)
``` ```
scp -q <files>.json dankclaw:/tmp/ ssh dankclaw 'for f in /opt/grafana/provisioning/dashboards/*.json; do \
ssh dankclaw 'for f in <names>; do sudo install -m644 -o root -g root /tmp/$f.json /opt/grafana/provisioning/dashboards/$f.json; done' cmp -s "$f" "/home/dank/serverAppConfigs/shared/grafana/provisioning/dashboards/$(basename "$f")" \
&& echo "$(basename $f) same" || echo "$(basename $f) DIFFERS"; done'
``` ```
Missing file or DIFFERS ⇒ the copier (or an unmerged branch) is the bug, not the JSON.
## Verification — what DOES and DOESN'T work ### 3. Browser proof (the real surface)
Throwaway `.mjs` placed **inside `tests/e2e/`** (`@playwright/test` resolves from `node_modules` there; a script in `/tmp` fails with `ERR_MODULE_NOT_FOUND`). Node on this box is Zig-rewritten v26: `readFileSync(p,"text")` throws `ERR_INVALID_ARG_VALUE` — use `readFileSync(p).toString()`.
Sequence, exactly as `specs/20-grafana-sso.spec.ts` drives it:
1. `newContext({storageState: ".auth/qa-e2e-admin.json", userAgent: REAL_UA})` — state is bound to the homepage origin; on the grafana vhost it is only the Authentik session cookie, so you MUST:
2. `goto https://grafana.franksdank.cloud/` and poll until body lacks `"Making sure you're not a bot"` (Anubis PoW warm-up).
3. `goto .../login/generic_oauth` and poll until `url` host is the vhost and path ≠ `/login` (RP round-trip). Landing on `/login` = auth flow broken, not the dashboard.
4. `goto /d/<uid>?orgId=1&from=now-1h&to=now`, wait ~10 s, then read panels: titles via `.panel-title,[class*='panel-title']`; a panel is `NO-DATA` iff its container text contains `"No data"`, `RENDERED` iff a descendant matches `svg:not(:empty),canvas,img`. Panels lazy-render; scroll (`mouse.wheel`) before judging.
Classify: `PANEL-NOT-FOUND` for a whole board ⇒ board not provisioned (step 2). `NO-DATA` with a proven-live expression ⇒ variable/label mismatch — re-run the expression with template vars expanded.
Throwaway scripts → `~/.AGENTTRASH/<date>-topic/` after use; never `rm`.
**DO NOT trust the sqlite `dashboard` table.** Grafana 13 uses **unified storage**: file-provisioned dashboards do NOT live in the legacy `dashboard` table. Reading `grafana.db` (`sudo python3 - <<PY ... select ... from dashboard`) returns **total dashboards: 0** even when everything is provisioned correctly — a false negative. There is also no `sqlite3` binary in the grafana image or on the host. Grafana logs **no** line on a successful provider load (only errors), so a clean `podman logs grafana` proves only "no error", not "loaded". ## Sandboxed deployment-function tests (bats)
To behaviorally test a deploy-remote.sh copier: extract with `sed -n '/^func()/,/^}/p'`, `eval` it, stub `run_privileged() { "$@"; }`, point the function's `*_ROOT` seam env var at a sandbox, execute, assert files. Traps:
- `VAR=x func` prefix assignments do NOT bind for shell **functions** — `export VAR=...` first.
- `mktemp -d` + `trap 'rm -rf' RETURN` races bats' trap machinery: the sandbox vanishes mid-function and bats blames the `trap` line. Use `$BATS_TEST_TMPDIR` (auto-cleaned).
- Prove red on the OLD body by patching a `mktemp -d` copy of `tests/ shared/ scripts/` and running bats there — no checkout mutation.
- Full suite must run from the repo root (`./tests/run.sh`), not from a subdirectory.
**The authoritative check is `/api/search` through an authenticated browser session.** No admin password is configured and anonymous is off — do NOT reset the Grafana admin to do a read check. Drive the RP SSO login with the Playwright `qa-e2e` storage state, then GET the API: ## Land flow (repo law)
```js Branch off fresh `origin/main`; merge needs Frank's explicit current OK; Gitea merge endpoint wants **JSON** `{"Do":"merge"}` (form-encoded `Do=merge` 422s with `[Do]: Required`); then `deploy.sh dankclaw --service grafana` and repeat steps 2–3 as the post-deploy gate.
const ctx = await b.newContext({ storageState: '.auth/qa-e2e.json', userAgent: "<chrome UA>" });
const p = await ctx.newPage();
await p.goto('https://grafana.franksdank.cloud/', { waitUntil: 'domcontentloaded' });
for (let i=0;i<30;i++){ const t=await p.locator('body').innerText(); if(!t.includes('not a bot')) break; await p.waitForTimeout(3000);} // clear Anubis PoW
await p.goto('https://grafana.franksdank.cloud/login/generic_oauth', { waitUntil: 'domcontentloaded' });
await p.waitForTimeout(6000);
const r = await ctx.request.get('https://grafana.franksdank.cloud/api/search?type=dash-db&limit=50'); // 200 + [{uid,title}...] = provisioned
```
Run it from the checkout that has `tests/e2e/node_modules` + `.auth` + `.env.local` (a fresh worktree lacks them; symlink `node_modules` and copy `.auth` + `.env.local`). Throwaway `_*.cjs`, then move to `.AGENTTRASH/`.
## Validate every panel query BEFORE shipping (catch empty / mis-scaled panels)
Query Prometheus (`http://127.0.0.1:9090/api/v1/query`) and Loki (`http://127.0.0.1:3100/loki/api/v1/query`) on Dankclaw loopback and confirm non-empty. Real traps found here:
- **Unit mislabel**: `sum by(job,mode)(rate(node_cpu_seconds_total{mode!="idle"}[5m]))` returns **cores per mode** (up to #cores) — unit `short`, NOT `percentunit`. For a per-host busy fraction use `1 - avg by(job)(rate(node_cpu_seconds_total{mode="idle"}[5m]))`.
- **Missing metrics**: `scrape_duration_seconds_bucket` and `prometheus_*`-prefixed self metrics may be absent — confirm from `/api/v1/label/__name__/values?match[]=<metric>` before building a panel; `prometheus_target_interval_length_seconds` (a summary with a `quantile` label) exists.
- **Alertmanager** runs on :9093 but is **NOT a Prometheus scrape target** → its metrics are unavailable; don't build panels on it (they read empty).
- **Loki streams**: journal arrives as `{job="systemd-journal"}`; `container_name` is a **parsed JSON field, not a stream label** → group at query time: `sum by (container_name)(count_over_time({job="systemd-journal"} | json [$__auto]))`.
- **Shell trap in test harnesses**: `$__auto` inside a double-quoted ssh/curl string is eaten by the local shell → empty query → Loki returns empty (looks like a broken panel, is a harness bug). For live validation substitute a concrete `[5m]` window, and drive Loki queries with an on-host Python `urllib` client to avoid quoting hell entirely.
## Datasource uids
`prometheus` (uid `prometheus`), `loki` (uid `loki`); reference as `"datasource": {"type":"prometheus","uid":"prometheus"}` in panel JSON.
@@ -0,0 +1,29 @@
---
name: halogen-npu-lane-boot-loop-triage
description: "Triage a frank-dragoon (halogen-flash-server) unit crash-loop caused by the NPU lane: signature classification, restart-debris trap, XRT mount-path census, and the transient env-file proof boot."
---
Triage when `frank-dragoon` (or any halogen-flash-server unit with `HALOGEN_NPU_MODELS` armed) sits in `activating (auto-restart)` with `ExecStart … code=exited, status=1/FAILURE` seconds after start, while `/health` stays dark.
## 1. Classify the exit BEFORE touching the config
- `NRestarts` climbing + exit in <5 s = refusal, not an engine fault. Read `journalctl -u frank-dragoon --since "-30m"` AND the engine log `/home/dank/logs/frank_dragoon.log` (engine stdout never goes to the journal).
- Lane signature: log ends with `halogen npu: … <name>: N files checked against this image's record` then `halogen-npu: No such library '<path>'` + `the NPU engine exited before it was ready (rc=1)` → **the NPU lane exit is fatal to the whole container** (supervisor kills the Flash engine with it). Chat-down is the lane's fault even though chat code never changed.
- KV/pre-flight lines present but no `locked in RAM` line = died before weight pinning.
## 2. The restart-debris trap
`Restart=always` re-runs the failing boot every ~10 s; each attempt leaves GPU/NPU handles being released, and the next boot logs `GTT in use before this start: N GiB … held by 1 process(es)`. That NOTE is debris from your own crash-loop — do not chase it as a foreign co-tenant (real co-tenants: ComfyUI `python` on `/dev/kfd`, find with `sudo fuser -v /dev/dri/renderD128 /dev/kfd`). `sudo systemctl stop <unit>` FIRST, then any diagnosis.
## 3. XRT path census (the `No such library '/opt/xilinx/lib/<multiarch>/libxrt_*.so.2'` case)
- The image ships NO XRT at all (`find / -xdev -name "libxrt*"` inside the image = empty); the NPU child `dlopen`s the host's libs through the serve script's mounts, which land at TWO destinations only: `/opt/xilinx/xrt/lib/<L>` and `$XRT_DIR/<L>`. A child searching `/opt/xilinx/lib/<multiarch>/` names a THIRD prefix nobody mounts.
- The serve loop resolves `REAL="$(readlink -f …)"` and mounts resolved files — if a host package upgrade retargets the symlinks (check `grep libxrt /var/log/dpkg.log` + `ldconfig -p | grep xrt` mtimes vs the last good boot), the file set the child expects can change under byte-identical mounts.
- Rootless gotcha: `podman mount` fails outside `podman unshare` and leaves `$M` EMPTY — `grep -r pat "$M/opt"` then greps your CWD = HOST tree (false hits like `/opt/fastflowlm`). Always `podman unshare bash -c 'M=$(podman mount $C); …'` or `--entrypoint ls|cat` probes instead, and distrust hits that name host-only paths.
## 4. Proof boot without touching production
The serve script honors `FRANK_DRAGOON_ENV_FILE` (sanctioned seam). Throwaway script on the host:
1. scratch env: `source /home/dank/serverAppConfigs/scripts/frank-dragoon.env` + the knob under test.
2. `sed` the serve script to inject the candidate extra mounts into the `NPU_ARGS+=(--device …` line — mount RESOLVED real files (`.so.2.25.00`), never symlinks (vendor: dangling links inside the container).
3. `export FRANK_DRAGOON_ENV_FILE=/tmp/… FRANK_DRAGOON_LOG=/tmp/…; FRANK_DRAGOON_CONTAINER=<scratch-name> timeout 1200 /tmp/serve-test.sh` — the container still publishes 7770, so `systemctl stop` the unit first (port binds at CREATE, exit 126 on race).
4. Retire same session: `podman rm -f <scratch-name>`, trash `/tmp` scripts; the unit stays stopped until the real fix deploys.
## 5. Restore chat while the lane is unfixed
`HALOGEN_NPU_MODELS=` (blank) is the documented one-line lane-off switch — lane absent = endpoint exactly as before the lane existed. Land through git + `deploy.sh dankclaw --service frank-dragoon`; never hand-edit the remote env.
@@ -0,0 +1,66 @@
---
name: halogen-npu-mount-prefix-proof
description: "Prove and fix a halogen frank-dragoon NPU-lane mount failure: crash-loop where a dead lane kills the whole container, dlopen-prefix census, and the sanctioned FRANK_DRAGOON_ENV_FILE proof-boot."
---
# Halogen NPU lane: dead lane kills the container — prove the mount prefix, don't guess
Applies when a `frank-dragoon` unit crash-loops in seconds (`exit 1` in ~200 ms,
`NRestarts` climbing) and the engine log's last lines are:
```
halogen npu: <model>: 15 files checked against this image's record
halogen-npu: No such library '<some/prefix>/libxrt_core.so.2'
halogen npu: the NPU engine exited before it was ready (rc=1)
```
## Laws this incident wrote (2026-10-21, frank-dragoon, 88-restart loop)
- **A dead NPU lane is fatal to the whole container**: the flash-server supervisor
exits with the lane. Chat dies beside a healthy GPU preflight — do not read the
GTT/weights lines as "engine fine, side feature broken".
- The NPU child `dlopen`s distro XRT from **three** container prefixes per library:
`/opt/xilinx/xrt/lib/`, the library's own install path, **and
`/opt/xilinx/lib/<multiarch>/`**. The missing one is exactly what the error names.
- Host state can flip this under a byte-identical container: the lane ran for days
with two prefixes, then a host `libxrt2`/`libxrt-npu2` reconfigure (check
`grep libxrt /var/log/dpkg.log`) made the child demand the third. Image digest,
serve script, payload all unchanged ⇒ suspect HOST package state, not the deploy.
- Fastest discriminator vs engine faults: exit in <1 s with the engine's preflight
lines present = serve/lane gate failure, not model load. Model-load failures take
minutes and print pin/KV lines.
## Proof procedure (sanctioned seam, no scratch ports)
1. Read the error's path verbatim — it names the missing prefix.
2. `sudo systemctl stop frank-dragoon` (unit must be down: the proof binds :7770
at container CREATE).
3. Scratch env file on the host using the bench-sanctioned seam:
```
/tmp/dragoon-nputest.env:
source /home/dank/serverAppConfigs/scripts/frank-dragoon.env
HALOGEN_NPU_MODELS=qwen3-embedding-0.6b
```
4. Scratch serve copy that appends the candidate mount to the `NPU_ARGS+=(--device`
line (`sed`), REAL files only (`readlink -f`; vendor forbids dangling symlinks
under /opt/xilinx), and run with:
`FRANK_DRAGOON_ENV_FILE=/tmp/... FRANK_DRAGOON_LOG=/tmp/... timeout 1200 /tmp/serve-nputest.sh`
5. Success = boot ledger (`… in all`) **plus** `npu_api: <model> … buckets u0…u9`
plus `/health` `npu.status ok`. Then codify the mount in the serve script, pin
it in `tests/frank_dragoon_unit.bats` (the fixture XRT dir's basename IS the
multiarch name — name it `x86_64-linux-gnu`), update the endpoint page's NPU
prerequisite to the three-prefix law, and cut over via PR +
`deploy.sh dankclaw --service frank-dragoon`.
6. Retire scratch: the `timeout` kills the proof container; `podman rm -f`,
move /tmp scripts to `~/.AGENTTRASH/<date>-<topic>/`. A proof instance
serving production while you wait is drift — deploy the merged fix promptly.
## Traps
- Greping image contents rootless: `podman mount` fails (needs `podman unshare`)
and leaves `$M` EMPTY, so `grep -r $M/opt` silently greps the HOST cwd — that
is how host `/opt/fastflowlm` masqueraded as image content. Always check `$M`
non-empty, or use `podman unshare bash -c 'M=$(podman mount C); …'`.
- The entrypoint swallows `podman run IMG sh -c …` as a mode name; use
`--entrypoint sh/cat/ls`.
- The proof container inherits production name/port unless the override is set
AFTER sourcing the env (the env file re-exports `FRANK_DRAGOON_CONTAINER`).
- `Restart=always` re-runs the lane into its own GPU/NPU debris; stop the unit
before any manual boot, don't fight the loop.
@@ -0,0 +1,44 @@
---
name: halogen-npu-xrt-third-prefix
description: "Triage a halogen frank-dragoon (flash-server) unit crash-loop caused by the NPU lane: signature \"No such library /opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2\", dead-lane-kills-container law, third-prefix mount fix, and the FRANK_DRAGOON_ENV_FILE scratch-boot proof recipe."
---
# Halogen NPU lane: dead lane kills the container; XRT needs THREE mount prefixes
Friction-earned 2026-10-21 on `frank-dragoon` (Dankclaw): unit boot-looped 88× (`activating`, exit 1 in ~200 ms) with the GPU/serve_api lines looking healthy; chat dead beside a fine-looking ledger.
## Signatures (in order of appearance)
```
halogen-npu: No such library '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'
halogen npu: the NPU engine exited before it was ready (rc=1; ...)
```
then `systemctl show <svc> -p ExecMainStatus` = 1, `NRestarts` climbing, and `/home/dank/logs/frank_dragoon.log` truncated at ~10 lines each boot (serve script truncates the log per start).
**Laws:** a dead NPU lane is NOT survivable — the engine supervisor takes the WHOLE container down on it. And `Restart=always` makes each restart race its own GPU/NPU debris: the FIRST failure's message is the only real evidence; later loop entries are noise. Stop the unit immediately (`sudo systemctl stop frank-dragoon`) before probing.
## Root cause class
The NPU child `dlopen`s XRT from `/opt/xilinx/lib/<multiarch>/` IN ADDITION to `/opt/xilinx/xrt/lib` and the install path. Host package churn (dpkg.log reconfigure of `libxrt2` the previous day) made the child demand the third prefix; our mounts filled only two. A lane that worked for days with byte-identical mounts can die purely from host package changes — diff the mounts last, check `grep libxrt /var/log/dpkg.log` first.
Fix: mount each REAL file (never a symlink) at all three prefixes; derive the multiarch name from `FRANK_DRAGOON_XRT_DIR`'s basename (`MULTIARCH="${XRT_DIR##*/}"`).
## Proof recipe (production-shaped, no scratch port)
One card holds one server — never bench on a side port; use the sanctioned env-file seam:
1. Scratch env on the host: `source /home/dank/serverAppConfigs/scripts/frank-dragoon.env` + knob overrides (e.g. `HALOGEN_NPU_MODELS=...`).
2. Copy the serve script with the candidate change applied via `sed` into `/tmp` (or edit + `FRANK_DRAGOON_ENV_FILE`); export `FRANK_DRAGOON_ENV_FILE=<scratch>` and `FRANK_DRAGOON_LOG=/tmp/<name>.log`; run under `timeout 1200`. The seam exists precisely for this.
3. Judge from `/tmp/<name>.log`: `npu_api: <model> ... buckets u0…u9` = lane alive; `No such library` = still broken. Then `curl :7770/health` → `npu.status`.
4. Retire the same session: `podman stop -t 60` (weights unpin slowly), `podman rm -f`, kill wrapper procs BY PID (`pgrep -f` the scratch serve script — `pkill -f` a pattern containing your own ssh line kills the session), move scratch files to `~/.AGENTTRASH/<date>-<topic>/`, then prove `pgrep -f serve-<scratch>` is empty. Leftover scratch state silently hijacks the next deploy (same class as leftover bench drop-ins).
## Traps hit en route
- Rootless `podman mount` fails ("cannot run in rootless mode") and leaves `$M` EMPTY — a later `grep -r "$M/..."` then reads the HOST cwd and "finds" the answer in Loki WAL files. Never grep an unverified mount var; `podman unshare` or `--entrypoint` probe instead.
- Image ships NO `/opt/xilinx` at all: all XRT comes from host mounts; `docs/NPU.md` is not inside the image.
- `ssh host 'tail <log>'` runs on the DEV box if you forgot `ssh` — a missing-file error is not "the engine wrote nothing".
- Health `context` field echoes the RUNNING engine's ceiling — use it to prove which shape is live, unlike `/v1/models` name echo.
## Verification that the fix landed
`systemctl show frank-dragoon -p DropInPaths` empty; boot ledger shows `reserving the KV pool … (about 11.0 GiB)`, `… in all`, an `npu_api:` line, no clamp line; deployer prints `NPU lane verified — <model> embedded`.
@@ -1,60 +1,62 @@
--- ---
name: harden-precommit-secret-gate name: harden-precommit-secret-gate
description: "Fix or build a pre-commit secret-scan hook that actually blocks leaks: scope hole, fail-closed, quarantine list, and the self-trip that kills an automated committer." description: "Fix or build a pre-commit secret-scan hook that actually blocks: the -z path hole, line-wide placeholder exemptions, inert doc placeholders, glued report lines, self-test fixture paths — each with a red-first bats case."
--- ---
Fix or build a pre-commit secret-scan hook that actually blocks leaks: scope hole, fail-closed, quarantine list, and the self-trip that kills an automated committer. Use when a secret gate is wrong (misses a leak) or too right (rejects an inert example and bricks the heartbeat). A pre-commit secret gate that fails open is worse than no gate: it teaches the operator that scanning happened. Four defect classes, each with a black-box test that goes red against the broken version (serverAppConfigs PR #70, command-center #59 — 4 holes closed same night, all red first).
## The two failure directions ## 1. Non-ASCII paths fail open
A gate can be **too weak** (misses a real key) or **too strong** (rejects its own documentation and takes down an automated committer). Both are outages; the second is silent. `git diff --cached --name-only` quotes non-ASCII paths (`"docs/\303\251.md"`), so `git diff` on that name returns **empty**, and a loop that `continue`s on an empty diff skips the file silently. Fail closed with NUL delimiting:
## Audit recipe for an existing gate
1. Read the hook and list what it actually inspects. The classic scope hole: `git diff --cached -- '*.env' '*.pem'` — an extension whitelist. A credential in `agent/config.yml`, a quadlet unit, or a SKILL.md is then invisible to the gate. The rule that survives: scan **the entire staged diff**, every path, and let only binary content out.
2. Test fail-closed behaviour: can the hook be made to print nothing and still exit 0? Common escapes — running outside a git repo, unreadable staged diff, a hook that pipes through `grep` and inherits `grep`'s exit code from the wrong stage, or `set -e` swallowing the verdict. Verdict must be: unknown ⇒ **block**.
3. Quarantine list, never tracked: `.env`, `auth.json`, `trust.json`, `*.pem`, `*.key`, `identity*` — and the `.gitignore` policy must quarantine env files **at any depth under any name** (`**/.env*`, not `.env`), or `config/prod.env` slips past.
4. Escape hatch must be explicit, per-commit, and loud: `ALLOW_SECRETS=1 git commit …`, and the hook says so when it honours it.
5. Gate the gate: a test that commits a fixture repo through the hook and asserts it blocks. `tests/secret_gate.bats` here covers: shape in an unexpected extension, bcrypt hash, JWT, placeholder/indirection allowed, prose allowed, binary ignored, unreadable diff refused, deletion of a credential never blocked, `ALLOW_SECRETS` override.
## Self-trip: the gate rejects its own example (live 2026-10-21)
Symptom: a systemd timer that commits machine-written files fails **every hour**, and the journal shows the gate's own refusal:
```
SECRET GATE: staged diff contains 1 secret-shaped string(s):
agent/managed-skills/git-secret-gate-hook/SKILL.md
Fake secrets must be inert (RFC 2606 domains, all-zero ids, `AKIA0000000000000EXAMPLE`) AND must not
```
The offender is the AWS **documentation** example key — entirely inert — and `AKIA[0-9A-Z]{16}` cannot tell it from a leak. So the gate's own advice about writing an inert fake is uncommittable by the gate. Auto-learned skills describe credential shapes for a living, so this recurs by construction, and an automated committer cannot use `ALLOW_SECRETS=1`: that would blind the gate for the *other* 64 staged files.
Diagnose, in this order:
```bash ```bash
journalctl --user -u live-sync.service -b --no-pager | tail -25 # which file, which line git diff --cached -z --name-only | while IFS= read -r -d '' f; do ...
git -C ~/.omp status --short | head # how much is queued behind it
# then scan the WHOLE staged queue, not just the reported file:
git diff --cached -U0 | awk '/^diff --git/{f=$3;sub("^a/","",f)} /^\+/ && !/^\+\+\+/ {
l=substr($0,2)
if (l ~ /sk-[A-Za-z0-9]{16,}|ghp_[A-Za-z0-9]{20,}|github_pat_[A-Za-z0-9_]{20,}|glpat-[A-Za-z0-9_-]{16,}|AKIA[0-9A-Z]{16}|xox[baprs]-[A-Za-z0-9-]{10,}|eyJ[A-Za-z0-9_-]{20,}\.[A-Za-z0-9_-]{10,}\.|-----BEGIN [A-Z ]*PRIVATE KEY-----/)
print f": "substr(l,1,90) }'
``` ```
The second count is the decision input: one hit ⇒ fix the line; many ⇒ the gate needs an exemption channel before anything lands. Test: a tracked file whose name contains a non-ASCII character, holding a fake key on an added line, must be blocked.
Fixes, in order of preference: ## 2. Placeholder filters must be value-scoped
1. **Describe the shape instead of printing a matching literal** — `AKIA` followed by zeroes and `EXAMPLE`, spelled out in prose, cannot match. One line, the gate stays strict, the next timer run drains the queue. `if echo "$line" | grep -q "$placeholders"` exempts the whole line, so a comment word vouches for the credential beside it:
2. **Per-line allow marker** honoured by the hook (e.g. a trailing `secret-gate: allow` on the flagged line) — durable, keeps documentation honest, still fail-closed everywhere else. Needs a test both ways: a marked line passes, an unmarked one still blocks. Note the risk honestly: an attacker-adjacent string can self-authorise, so the marker belongs in review-able prose tiers, not in a diff that also touches secrets config.
3. Never widen the regex to skip markdown, never run the automation with the override env var set.
Inert-fake rule for anything that must *show* a credential shape: RFC 2606 domains, all-zero ids, the word `EXAMPLE` — and if the shape itself trips the gate, describe it in prose rather than print it. ```python
token = "sk-…" # example config for the docs
```
## Prose must not be collaterally blocked Filter only the captured value (the `sk-…` group), never the line. Two directions to test: `key = "example"` stays allowed; `token = "sk-<real>" # example` is blocked.
Markdown explaining credential handling is the bulk of what a knowledge repo commits. `xox` inside a sentence about Slack token prefixes, or a `sk-` prefix discussed abstractly, must pass; the shapes are anchored long-suffix patterns for exactly that reason. When a gate fires on prose, that is a bug in the gate — but if the prose contains a *literal* matching token, the fix is the prose (option 1 above), not the pattern. ## 3. Inert placeholders in documentation
## Verifying a gate change A skill/README that *documents* a secret gate quotes the vendor's placeholder ids (`AKIA0000000000000EXAMPLE`, all-zero JWTs). Exempt them by value shape (all-zero / `EXAMPLE` suffix / RFC 2606 domain), or the machine tier stops committing — and the refusal usually lands on a heartbeat commit, where it looks like infrastructure failure, not policy. Also note: an unlanded fix means the checkout runs the **old** hook, so a landing can be blocked by the very bug the PR corrects; audit the staged diff and use the documented override (`ALLOW_SECRETS=1`, never `--no-verify`), recording the audit in the commit message.
Write the fixture repo in a temp dir, commit through the hook, assert block/allow for each case. Do not test the regex by grepping the hook's source for the pattern — a copy of the pattern in a comment is not a gate that runs. ## 4. Report-line assembly
`report+=(" $file: $line")` inside a loop that keeps only the last hit per file glues the next filename onto the previous line when a second hit exists — the report attributes the finding to the wrong file. Test a two-file staged set where both match, and assert each report line names exactly one existing path.
## 5. Self-test fixtures in the hook itself
The hook's own fixtures contain fake keys by necessity, so the gate would refuse any commit touching the hook or its tests (or a docs page quoting a key as documentation). Pin the bypass to specific paths and to added lines only, and print what was exempted:
```bash
case "$f" in .githooks/pre-commit|tests/secret-gate.bats)
echo "secret-gate: self-test fixtures exempt: $f" >&2; continue;; esac
```
## The scan rule
Scan **added content only** — `git diff --cached -U0 | grep '^+[^+]'` — or every commit touching a file with an old key in context gets blocked, and the operator's response to a gate that blocks everything is to override it always.
## Test harness (fixture repo, never the real checkout)
`git init` a temp repo, `git config core.hooksPath .githooks`, copy the hook in, `git add` a payload, then run it with the env overrides the hook honours (point its cache/notify paths at `$TMP` so tests never touch the user's config). Pin, in order:
| Case | Expect |
|---|---|
| fake key on an added line | exit 1, message names the file |
| `key = "example"` | pass |
| real key + `# example` comment | exit 1 |
| non-ASCII filename with a key | exit 1 |
| two files with hits | report names both |
| self-test fixtures | pass, with the exempt line printed |
`bats -r` after each edit. `--no-verify` is forbidden on Frank's boxes: the hook IS the control.
@@ -1,48 +1,49 @@
--- ---
name: html-report-ground-truth-audit name: html-report-ground-truth-audit
description: "Audit a self-generated HTML report: ground every numeric/code/diagram claim against the repo, fix, and prove structure + render" description: "Audit a self-generated HTML report: ground every numeric/code/diagram claim against the repo, fix wrong values, delete filler that restates git, verify the served artifact byte-identical, and flag unverifiable content to the operator."
--- ---
Use after generating a large HTML report (e.g. via `repo-change-html-report`) whose claims were partly reconstructed from session memory. Fabricated numbers, invented code blocks, and silently-swallowed sections are the failure modes; the audit makes each traceable to the repo, a live command, or a recorded run. Ground-truth audit of an HTML report (generated by an agent, for a human). Three defects hide in a plausible report: **wrong numbers, unverifiable claims, and filler that restates git for a reader who has git.**
## Law
Every numeric claim must be re-measured against the repo, not the narrative that produced it.
## Procedure ## Procedure
### 1. Enumerate every checkable claim 1. **Dump claims with file:line anchors.**
Grep the report for numerics, file paths, symbol names, and code blocks: `sed 's/<[^>]*>//g' report.html | grep -vE '^\s*$' | grep -nE 'commit|file|line|fix|refactor' > /tmp/claims.txt`
- line counts (`wc -l` the real file) Then enumerate the ground truth: `git log --oneline origin/main..HEAD` for commits; `git diff --stat origin/main..HEAD` for the per-file census — **sort it**: `--sort=-deletions` and `--sort=-creations` give the top movers, so the "largest single change" and "largest net code change" are computed facts.
- test counts (`grep -c '@test' tests/<f>.bats` — NOT suite-run totals: conditional skips make raw counts differ; run-observed totals are legitimate run history, keep them labeled as such)
- diff stats (`git diff --shortstat main..HEAD` — the footer's files/insertions/deletions)
- commit hashes (`git log --oneline`)
- "verbatim" code blocks (see 3)
- diagram node lists and captions (see 4)
### 2. Ground numeric claims with one command each 2. **Check code blocks compile.** Write every snippet to a temp file and `bash -n` each. Common breakages from HTML generation: `&quot;`/`&amp;` entities leaking into code, and `<br>` line breaks that split a line mid-pipe.
Fix every mismatch in place. Record the was/now pair — the audit table IS the evidence.
### 3. Verbatim code blocks: prove them 3. **Check the diagrams carry information.** A mermaid `flowchart LR` whose nodes merely restate the bullet list above it is decoration — delete it.
For a block labeled verbatim, the exact string must exist:
- live code: `grep -n '<distinctive line>' <file>`
- historical: `git show <sha> -- <file> | grep -E '^[+-].*(<token1>|<token2>)'`
If the block was fabricated (plausible-looking pseudo-code), replace it with the real excerpt. If condensed, the block header must say "condensed" and every shown line must be real.
### 4. Diagram claims: run the engine 4. **Check counts.** Commit and file counts must match `git rev-list --count` and `--name-only | sort -u | wc -l`. A report whose sections were written by different subagents drifts: each counts its own slice ("4 commits / 14 files" vs the true 11 / 25).
Behavioral diagrams (plan traces, closure graphs) must be re-derived by running the actual CLI:
`python3 scripts/service-map.py plan --host <h> <path>` — then edit node labels, captions, and the prose count ("four services") to match output. Cross-check derived-set diagrams against the source relation (e.g. `grep -rl 'Network=gluetun.container' */**/*.container` for who rides a tunnel).
### 5. Check for swallowed sections 5. **Flag what cannot be verified.** Test pass claims, external tool output, runtime behaviour — if the repo can't confirm it, say so to the operator instead of asserting it.
Large prior-session PUT ranges silently overwrite neighbors. Grep for headings/captions the plan promised; re-insert any that vanished, anchored on a fresh `read` of the region.
### 6. Structural balance ## Filler test (delete on failure)
Python tag-balance pass over the file:
- per-tag open/close counts (`div`, `pre`, `figure`, `svg`, `table`, `span`, `figcaption`)
- running-div-depth scan printing per-section balance to localize stray closers
Common defect: `.ln`/`.nt` pair emitted without its `.cg` wrapper → stray `</div>`; fix by opening `<div class="cg">` before the pair.
### 7. Render proof - A `file → line-count → description` table restating `git diff --stat`? Delete — the reader has git.
`file://` is blocked on the shared browser — serve the dir: `bash` service `python3 -m http.server <port> --bind 127.0.0.1 --directory <dir>`, ready on that port. Then: - A "what this means" / "conclusion" / "key takeaways" section? Delete — it is the same facts in softer prose. (Same rule as vault pages: point of use or gone.)
- `browser.open({name, url: 'http://127.0.0.1:<port>/<file>'})` - Keep only what a reader cannot rebuild from `git log` + `git show`.
- DOM assertions via `tab.evaluate(() => …)` — NOT `tab.run` (that runs in Bun; no `document`). Sections are `<h2 id>` + siblings: scope queries to `document.body`/`document.querySelectorAll`, not `#ch5 …`.
- screenshot the repaired regions (scroll via `tab.evaluate(el.scrollIntoView())`), then `tab.close()` and kill the server proc (`write proc://<name>/kill`).
## Deliver ## Serve and verify the artifact
A was/now table of every fix with its ground command, plus the list of claims verified intact. State honestly which numbers are run-history and not recomputable.
**Verify the file that is served, not the source.** If a `public/` copy is byte-identical, edit both or the served artifact stays wrong. Then re-fetch:
```bash
base=http://host:port/reports/x.html
for i in $(seq 1 30); do curl -fsS -o /tmp/got.html "$base" && break; sleep 1; done # service may still be activating
for i in 1 2; do curl -fsS "$base" | md5sum; done # stable across refetches
diff <(curl -fsS "$base") /path/to/source.html && echo "served == source"
```
## Voice drift
If the operator asked for a tone ("no filler", "terse"), the audit must enforce it too — marketing phrasing ("ships the", "This means:") is a style defect, fix it in the same pass, not a later one.
## Report
Fixes grouped by defect class (wrong number / unverifiable / filler / style), then what stays unverifiable and why, with the measured ground truth quoted.
@@ -0,0 +1,34 @@
---
name: infra-repo-security-review
description: "Full-repo security review of an infra/config repo (serverAppConfigs-shaped: shell deployers, quadlets, systemd, Caddy edge, bats/Playwright) — slice fan-out, mechanical censuses, and subagent-claim verification."
---
# Whole-repo security review (infra/config repo)
For "full security review of this repo" on a deployment-config repo (shell scripts + podman quadlets + systemd units + Caddy + tests). NOT for a diff review — that is `code-review`. NOT for a secrets-only question — that is `git-repo-secret-audit`.
## 1. Slice, then fan out in ONE `task` batch
Five `security-reviewer` slices, each with the repo's own laws pre-injected in `context` so no agent re-derives them (AGENTS.md policy text + exclusions: docs prose, `.AGENTTRASH/**`, binaries, `package-lock.json`):
1. **Shell** — all `*.sh` + `.githooks/` + `tests/run.sh`: injection, quoting, `mktemp` vs `/tmp/fixed`, `set -e`, `curl -f`, secrets in argv, `rm`/`git clean`/`systemctl disable`.
2. **Python** — every `*.py`: `subprocess(shell=True)`, `verify=False`, unbounded `read()`, path traversal on request-supplied filenames, fixed `/tmp` writes, **which HTTP servers exist and which are unauthenticated**.
3. **Container + systemd posture** — every `*.container`/`.network`/`.service`/Containerfile.
4. **Edge + web** — Caddyfile vhost inventory, Anubis/Authentik chain, response headers, static-asset XSS sinks, service configs (`limiter.toml`, `prometheus.yml`, `alertmanager.yml`).
5. **Tests** — vacuous assertions + credentials/secrets in fixtures + harness safety.
**Set a yield deadline in the prompt.** Subagents here get killed at `task.maxRuntimeMs` (30 min) and the dead ones produced **zero** output. When one runs long, write early: `write agent://<id>` "yield now, unconfirmed items prefixed UNVERIFIED" — the resumed agent returns its partial report instead of losing it. Two of four needed this.
## 2. Verify every citation before reporting
`verify-subagent-review-findings` applies in full: re-grep each cited `file:line` and compare text. Expect ~40% of lines to be off (e.g. an agent described a 13-line hook; the file is 6) and several files to be hallucinated entirely (`.githooks/compose/*.container` did not exist). Discard mismatches; publish a **Discarded subagent claims** section. Re-grep your own citations too — two of mine were off by two lines and had to be corrected in an addendum.
## 3. Mechanical censuses (cheap, high yield — run these yourself)
- **Port binding**: every `^PublishPort=` line, classify `127.0.0.1:` / `192.168.0.x:` / **no address prefix** (= implicit `0.0.0.0`). The unprefixed set is the auth-bypass surface: a LAN client hits the container port directly and skips the Caddy→Anubis→Authentik chain that the vhost enforces. Report as one consolidated finding listing all unprefixed lines, and cite the repo's own correct examples (`127.0.0.1:9090`, `192.168.0.143:8080`) as the fix pattern.
- **Image pinning**: count `^Image=` lines with vs without `@sha256:` (this repo: 4 of 54 pinned) — floating tags + `Pull=never` = unreproducible edge.
- **Secret modes**: grep every `chmod 600` in the deployer and diff against every file it writes from an `.env`; a `>` redirect without the `chmod` is the finding (the Caddy/Porkbun env file).
- **Systemd hardening**: `User=`/`ProtectSystem`/`PrivateTmp`/`NoNewPrivileges` per unit, and `ExecStartPre=+` lines — a `+` hook on a root unit running a checkout-writable script is a local privesc.
- **Vhost table**: vhost → upstream → guarded? → exemption documented? Cross-check each upstream against the quadlet `PublishPort` list; unguarded vhost with no exemption comment is a policy violation even when the service self-authenticates.
- **Hook wiring**: `git config core.hooksPath` + `ls "$(git rev-parse --git-path hooks)"`. A repo can ship `.githooks/pre-commit` that git never executes — and its own test can assert the *file* is executable and still miss that it isn't installed.
- **Secret sweep**: `git cat-file --batch-all-objects --batch` (all history + dangling), match known values read from `~/.config/secrets`; report `prefix…(len N, sha 10)` only, never the value. Watch for the **split-concat disguise** (`PW="as""set"`): their own regex can't see it, but `grep -F` on a fragment finds it and history keeps it forever → rotation required.
## 4. Report
Severity-ranked table (Critical→Low, one line each: location + flaw), then prose detail for Critical/High only, then the Discarded-claims section, then remaining coverage gaps, then an ordered fix set and an offer to branch. Do not create a report file unless asked — chat is the deliverable.
@@ -1,50 +1,82 @@
--- ---
name: land-merged-pr-with-blocked-tree-ops name: land-merged-pr-with-blocked-tree-ops
description: "Use when a merged PR must land in a checkout whose working tree is dirty and workspace policy blocks checkout/stash — prove the delta content-free, hand the operator the two-liner, verify via live config meanwhile." description: "Use when a merged PR must land in a checkout whose working tree is dirty and workspace policy blocks git checkout/restore/stash: commit-the-dirt pattern, the secret-gate false positive that blocks the landing commit, conflict resolution proven settings-vs-comments, and heartbeat (live-sync) handoff."
--- ---
Landing a merged PR when the deployment checkout is dirty **and** workspace policy blocks `git checkout`/`git stash`. Use when `git merge --ff-only origin/main` aborts with "Please commit your changes or stash them before you merge" in a SHARED workspace where other agents have work in flight. Earned 2026-10-20 on `dank/command-center` (the checkout IS the deployment). Landing a merged trunk into a checkout that IS the deployment (dank/command-center `~/.omp`), when tracked files are dirty and policy forbids `checkout`/`restore`/`stash` on shared state. Re-earned 2026-10-21 landing PRs #61 + #63 into `~/.omp` (27 commits behind, two conflicts, one gate refusal).
# Land a merged PR with blocked tree ops ## 1. Classify the dirt before touching it
## Why it blocks `git status --porcelain` then compare against what the merge brings in — only overlap blocks the merge:
`--ff-only` refuses when **any** incoming path is dirty in the worktree — it checks dirtiness against the *old* HEAD, not whether your content matches the target. Proven: overwriting the file so it is **byte-identical to the target commit** (`cmp` clean against `git show origin/main:<path>`) still aborts with the same message. So "make the file match main" is not a way in, and neither is `git merge` from another worktree (the branch is checked out in the dirty one).
The blocking set is what you must know, exactly:
```bash ```bash
comm -12 <(git diff --name-only HEAD..origin/main | LC_ALL=C sort -u) \ { git diff --name-only; git diff --cached --name-only; } | sort -u > /tmp/dirty.txt
<(git status --porcelain | awk '{print $NF}' | LC_ALL=C sort -u) git diff --name-only HEAD origin/main | sort -u > /tmp/incoming.txt
comm -12 /tmp/dirty.txt /tmp/incoming.txt # these paths must be clean
``` ```
`LC_ALL=C sort` matters — git's own output order trips `comm: file N is not in sorted order` and you get garbage overlap lists. In the case this was earned on, 12 incoming paths and 57 dirty ones intersected in exactly **one** file. Note the first column of porcelain: `M ` = **staged**, ` M` = unstaged. A staged file shows nothing in `git diff` — look at `--cached` too, or you will "verify" a file is unchanged while it is staged. `comm` needs both inputs sorted; unsorted input prints "file is not in sorted order" and a partial answer.
## Procedure ## 2. Prove a dirty file content-free before resolving
1. **Fetch, don't merge yet.** List incoming commits (`git log --oneline HEAD..origin/main`) so you can show what is waiting. `cmp` it against the trunk blob, never against memory:
2. **Compute the blocking set** with the `comm` above (both inputs `LC_ALL=C sort -u`).
3. **Prove the local delta is content-free**, in the direction that matters — lines the working tree has that `main` does not:
```bash
diff <(git show origin/main:<path>) <path> | grep -c '^>' # local-unique lines
```
Zero means the working tree is a strict subset of what was merged: discarding loses nothing.
4. **Preserve before you touch:** copy the current file to `~/.AGENTTRASH/` (snapshot), and copy anything you had moved into a gitignored `.AGENTTRASH/` inside a build worktree to its permanent home in the deployment — otherwise pruning the worktree destroys the only copy of a quarantined page.
5. **Do NOT work around the policy.** `git checkout -- <path>`, `git stash`, and `git stash push -- <path>` are all blocked in a shared workspace and the block is correct: it protects other agents' in-flight work. Do not reach for `git reset --soft` + `git restore --staged` to slide the pointer past it either — that leaves the index holding staged *reverts* of every incoming path, one careless `git commit -a` away from undoing the merge.
6. **Hand the operator the two-liner** and keep working on what does not depend on it:
```bash
git -C ~/.omp checkout -- <path> && git -C ~/.omp merge --ff-only origin/main
```
Say why it is safe (step 3's number, plus where the snapshots live) and what it unblocks.
## Why waiting is not free ```bash
git show origin/main:$f > /tmp/incoming
cmp -s "$f" /tmp/incoming && echo "IDENTICAL to trunk" # commit/discard is content-free
```
`scripts/live-sync.sh` merges with a plain `git merge --no-edit origin/main` (its line ~88), so the hourly heartbeat hits the identical wall and raises its ntfy alert `merge conflict with origin/main — local commit kept; needs you at the keyboard`. Report that consequence: the dirt is not merely untidy, it alerts. If identical, the dirt is a sync artifact, not user work. Say so in the commit message with the `cmp` result — that is what makes a direct commit on `main` defensible for a prose-tier file.
Also note for later: `git branch -d <merged-branch>` **refuses until the commits reach `main`** — `-d` checks against the current HEAD, not `origin/main` — so local branch deletion goes on the post-landing list, not the pruning list. ## 3. Commit rather than discard (policy-blocked verbs)
## What you CAN do while it waits With `checkout`/`restore`/`stash` unavailable, the only non-destructive unblock is a commit. Two separate commits, and order matters:
- **Verify the merged behaviour through the live config, not the branch pointer.** Merged prompt/config changes take effect from the deployment's files: spawn a fresh session/subagent and have it read and quote the changed file back. That proves the change is live even while `main` is behind. 1. Prose-tier dirt proven byte-identical to trunk — commit alone, message states `cmp` proved no prose change.
- Prune your own build worktrees, but check nothing is running inside them first: `ls -l /proc/*/cwd 2>/dev/null | grep -c "wt/<name>"` (the same guard `live-sync.sh` uses). 2. Machine tiers (`agent/memories/**`, `agent/managed-skills/**`, `agent/config.yml`, `agent/models.yml`, `dotfiles/`) — let the heartbeat do it: `LC_ALL=C scripts/live-sync.sh` stages allowlist only, commits, merges trunk, pushes, and **exits 0 even when it aborts the merge on conflict** (the alert goes to stderr + ntfy). `exit 0` does not mean landed — check `behind=` afterwards.
- Delete the merged remote branches (`git push origin --delete <branch>`, allowed on the local Gitea).
Do NOT `git add -A`: the index may already hold dozens of staged machine-tier files, and `git add <one file>` does not clear them — they ride along into your commit.
## 4. The gate refusal that blocks the landing commit
The pre-commit secret gate scans the **staged** diff, so a single-file commit still trips on previously staged content. It fires on inert placeholders inside the skill docs that document the placeholder (e.g. `AKIA0000000000000EXAMPLE` — the AWS all-zero example id in `git-secret-gate-hook` / `harden-precommit-secret-gate`). If the checkout is behind, it is running the **pre-fix** hook while the fix sits unlanded in trunk: the landing is blocked by the very bug the PR corrects.
Audit first, then override deliberately — never `--no-verify`:
```bash
git diff --cached | grep -nE 'AKIA[A-Z0-9]{12,}|sk-[A-Za-z0-9]{16,}|ghp_[A-Za-z0-9]{20,}|glpat-[A-Za-z0-9]{16,}|BEGIN [A-Z ]*PRIVATE KEY|eyJ[A-Za-z0-9_-]{20,}'
git diff --cached | grep -oE 'AKIA[A-Z0-9]{12,}|sk-[A-Za-z0-9]{16,}' | sort | uniq -c # what exactly, how many
ALLOW_SECRETS=1 git commit -F msg # message records the audit + that the gate ran
```
Record in the message: the hits are inert placeholders, which files they sit in, that this checkout runs the pre-fix hook, and that the refusal dies with this landing.
## 5. Merge conflicts: resolve by proving settings identical
```bash
git merge --no-commit --no-ff origin/main
git diff --name-only --diff-filter=U
```
For each conflicted file, get all three blobs (`HEAD`, merge base, `origin/main`) and ask what the local side uniquely holds:
- **Prose / doc pages**: diff ours vs trunk; if ours only lacks the incoming trim, take trunk.
- **Real settings files (kdl/yaml/env)**: strip comments and diff — `diff <(grep -v '^\s*//' ours) <(grep -v '^\s*//' theirs)`. Empty output ⇒ only comments differ ⇒ taking trunk changes no deployed behaviour, and you can say that in the merge message. (2026-10-21: `desk-1-home.kdl` — modes byte-identical, trunk's comment corrected the false "no 240 Hz mode exists".)
Resolve without the blocked verbs by writing the trunk blob: `git show origin/main:$f > $f && git add $f`. Then validate the live artifact (`niri validate`, the suite) before committing the merge, and push.
## 6. Verify the push actually landed
`git push` output can be swallowed when wrapped in command substitution inside a batch — verify from the remote, not from the push:
```bash
git fetch -q origin && printf 'HEAD=%s origin/main=%s behind=%s\n' \
"$(git rev-parse --short HEAD)" "$(git rev-parse --short origin/main)" "$(git rev-list --count HEAD..origin/main)"
```
## 7. Close and prune
- Close the issues with a comment giving merge SHA + PR link + title (numbers are per-repo; a bare `#N` is not a reference).
- Prune guards, all three before `git worktree remove --force`: `ls -l /proc/*/cwd | grep -c '<wt-name>'` must be 0; `git -C <wt> status --porcelain | wc -l` must be 0; `git rev-list --count main..<branch>` must be 0. Check for `.AGENTTRASH/**` inside the worktree first — removal destroys the only copy.
- `git branch -d` (not `-D`) once the branch is reached by main; `git push origin --delete <branch>` is allowed on the local Gitea.
- Delete the temp `-K` token config file in the same session that created it.
@@ -0,0 +1,25 @@
---
name: okf-small-bundle-docs-audit
description: "Audit + dedupe a small repo's docs bundle for OKF conformance when the kb-lint/kb-coverage toolchain does not exist: inline conformance and link-audit scripts, owner-per-fact cuts, index trio, one log."
---
# OKF docs audit for a small repo bundle (no kb-lint toolchain)
Use for a repo whose KB is one docs directory (e.g. `dankvoice` → `dankvoice-docs/`), where `scripts/kb-lint.sh` / `kb-coverage.py` do not exist. For the big `dank/command-center` campaign see skill `docs-dedupe-verify-campaign`; same frame, no scouts — a corpus under ~1500 lines gets read fully yourself (scouts lie).
## Frame
- OKF spec lives at `~/.omp/omp-docs/references/okf-spec.md` (the command-center deploy checkout). v0.1 conformance = every non-reserved `.md` has parseable frontmatter with non-empty `type`; `index.md` = listing with NO frontmatter; `log.md` reserved, exactly one, bundle root.
- Branch off `origin/main` in a fresh worktree; docs-only + src-comment changes still need the repo's normal gate.
## Procedure
1. Inventory `wc -l` every page; read all fully. Ledger per fact: which page OWNS it (laws page owns laws, decisions own rationale, contributing owns process, README owns user-facing, GLOSSARY owns terms). Everything else becomes a pointer.
2. Hunt the rot classes that repeat: dead test citations (name in docs not in `grep -c tests/` — verify with `git log --all -S`), hand-maintained censuses (test counts, scenario lists) — replace with a pointer to the authoritative source, never refresh the copy; deleted doc lines for live env vars (compare `git show <fixed>:file | grep -c VAR` vs HEAD); sections contradicting the page's own head; open items already resolved in live state (check the live settings file, not the doc).
3. Frontmatter: `type` + one-line `description` + `tags` + `timestamp`. On edit-rejection the tool prints the file's current `#TAG` — reissue with it, no re-read needed.
4. Conformance check (inline python): walk bundle dirs, regex `^---\n(.*?)\n---` then `^type: \S` on each non-reserved `.md`.
5. Link audit (inline python): per md file, `\]\(([^)#]+)`; skip http; `os.path.normpath(join(dirname, target))` must exist. Report `N checked / ALL RESOLVE`.
6. Create `index.md` at bundle root + each subdir (listing, no frontmatter, relative links) and `log.md` at the bundle root seeded with laws/incidents only.
## Gotchas
- Src docstring fixes are artifact-free only if the bundler strips comments — prove with `bun run build; cmp dist/... old` before claiming no rebuild needed.
- README must NOT enumerate design records; point at `decisions/index.md` so new records need no README edit.
- Mark superseded decision records in their frontmatter `description` ("its X design is superseded by the Y record") — never rewrite history bodies.
@@ -1,4 +1,4 @@
{ {
"v": 1, "v": 1,
"lastConsolidatedAt": 1792557324990 "lastConsolidatedAt": 1792574106585
} }
@@ -1,10 +1,10 @@
{ {
"v": 1, "v": 1,
"lastConsolidatedAt": 1792572225299, "lastConsolidatedAt": 1792604806049,
"lastResult": { "lastResult": {
"at": 1792557082020, "at": 1792573818108,
"sessions": 1, "sessions": 1,
"deltas": 1, "deltas": 2,
"model": "frank-zealot" "model": "frank-zealot"
} }
} }
@@ -0,0 +1,7 @@
## Okf Format
- "okf format"
## Human Readability
- "human readability"
@@ -1,10 +1,10 @@
{ {
"v": 1, "v": 1,
"lastConsolidatedAt": 1792572021650, "lastConsolidatedAt": 1792604841754,
"lastResult": { "lastResult": {
"at": 1792571693973, "at": 1792574016725,
"sessions": 1, "sessions": 1,
"deltas": 1, "deltas": 2,
"model": "frank-zealot" "model": "frank-zealot"
} }
} }
@@ -0,0 +1,3 @@
## Formatting
- okf format
- formatted for human readability
@@ -1,6 +1,6 @@
{ {
"v": 1, "v": 1,
"lastConsolidatedAt": 1792572038693, "lastConsolidatedAt": 1792604738996,
"lastResult": { "lastResult": {
"at": 1792556461435, "at": 1792556461435,
"sessions": 1, "sessions": 1,
@@ -1,6 +1,6 @@
{ {
"v": 1, "v": 1,
"lastConsolidatedAt": 1792572210019, "lastConsolidatedAt": 1792604850405,
"lastResult": { "lastResult": {
"at": 1792566095222, "at": 1792566095222,
"sessions": 1, "sessions": 1,
@@ -0,0 +1,4 @@
{
"v": 1,
"lastConsolidatedAt": 1792574109325
}
@@ -0,0 +1,4 @@
{
"v": 1,
"lastConsolidatedAt": 1792604683894
}
+11 -10
View File
@@ -96,19 +96,20 @@ providers:
# halogen flash-server on the Strix Halo iGPU. Payload, shape, eras, # halogen flash-server on the Strix Halo iGPU. Payload, shape, eras,
# laws, rollback: serverAppConfigs-docs/LLM/frank-dragoon.md — and # laws, rollback: serverAppConfigs-docs/LLM/frank-dragoon.md — and
# scripts/frank-dragoon.env is the authority for every number there. # scripts/frank-dragoon.env is the authority for every number there.
# contextWindow sits deliberately BELOW the engine ceiling # contextWindow: 200,000 under the native 262,144 engine ceiling
# (HALOGEN_CTX): the clamp below leaves the gap that absorbs the # (HALOGEN_CTX, 2026-10-21) — the 400-zone law: the window MUST sit
# harness's chars/4 under-count, so a window equal to the ceiling # ~45k under the ceiling, never at it.
# would make every long request a guaranteed 400. contextWindow: 200000
contextWindow: 155648
# The clamp is max_tokens = min(maxTokens, contextWindow − est − # The clamp is max_tokens = min(maxTokens, contextWindow − est −
# 4096); the chars/4 est UNDER-counts the rendered prompt by ~3 % # 4096); the chars/4 est UNDER-counts the rendered prompt by ~3 %
# on code/agent traffic (measured 2026-10-07: est 152,938 -> # on code/agent traffic (measured 2026-10-07: est 152,938 ->
# server 157,388). Worst-case request = window − 4096 + 3 % est # server 157,388). Worst-case request = (200,000 − 4,096) × 1.03
# error ≈ 156.2k, inside the engine ceiling with ~44k of margin # ≈ 201.8k; the compaction trigger fires at 95 % (config.yml
# (robust to a ~25 % under-count). Compaction fires at 85 % of the # thresholdPercent): 190,000 × 1.03 + 45,056 ≈ 240.8k — both inside
# window and rides on frank-zealot: input ≈ threshold − # 262,144 (margin 60k / 21k). Compaction rides frank-zealot:
# keepRecent(20k), comfortably inside its per-slot window. # input ≈ 190,000 − keepRecent(20k) ≈ 175k ≤ its 200,704 slot.
# Recompute all of it after ANY window or threshold change; tests/
# frank_dragoon_unit.bats pins this arithmetic at 95 %.
# HALOGEN_MAX_TOKENS_CAP in frank-dragoon.env must stay >= this. # HALOGEN_MAX_TOKENS_CAP in frank-dragoon.env must stay >= this.
maxTokens: 45056 maxTokens: 45056
reasoning: true reasoning: true