Compare commits
111
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
dcdf84719c | ||
|
|
8c3dc6bfc9 | ||
|
|
27da5cd9ec | ||
|
|
ec9bf77b10 | ||
|
|
6a12ffaef4 | ||
|
|
d33ee26e67 | ||
|
|
b8f66e22b2 | ||
|
|
07768ce8b5 | ||
|
|
d845160e62 | ||
|
|
3ab48bf1f2 | ||
|
|
51f9846ac6 | ||
|
|
4085ee403f | ||
|
|
1dc01bd006 | ||
|
|
de7dd6f897 | ||
|
|
64bb23b329 | ||
|
|
336201e814 | ||
|
|
e0b1466f74 | ||
|
|
d368394656 | ||
|
|
ec19771b8b | ||
|
|
8b87b5ae1a | ||
|
|
047f264fcf | ||
|
|
54bb0202c2 | ||
|
|
fb36a1b0fd | ||
|
|
c37ad6e870 | ||
|
|
c6e2a1daca | ||
|
|
88d486b37b | ||
|
|
49f67b2192 | ||
|
|
e49537c2ac | ||
|
|
62a70caebb | ||
|
|
4735702965 | ||
|
|
22d2f3bcd8 | ||
|
|
205b54adcb | ||
|
|
8af40c402b | ||
|
|
6e89c63e5d | ||
|
|
e690a2ca66 | ||
|
|
d4d6dd0c44 | ||
|
|
66e1129e6c | ||
|
|
a280c29d2c | ||
|
|
8393dc1090 | ||
|
|
d0587a5bc5 | ||
|
|
334e5d0ee0 | ||
|
|
66fc40de65 | ||
|
|
71601f72a8 | ||
|
|
f3ce42eb80 | ||
|
|
812be6a0a6 | ||
|
|
06f45e83c1 | ||
|
|
40b9725433 | ||
|
|
671222b881 | ||
|
|
71a7e5cb9a | ||
|
|
97b65067b1 | ||
|
|
23372e96f5 | ||
|
|
ab94c91bcf | ||
|
|
5cf7e6bd8e | ||
|
|
1cc4b76479 | ||
|
|
59920f849d | ||
|
|
bfd6a6340e | ||
|
|
4849f46eb2 | ||
|
|
dd91ef1d59 | ||
|
|
289f74062a | ||
|
|
d7bd05398d | ||
|
|
004962ffce | ||
|
|
56fc899763 | ||
|
|
35ceae6d7b | ||
|
|
655a39ffb6 | ||
|
|
b4fc881164 | ||
|
|
6df45d709a | ||
|
|
5ab0a49ea4 | ||
|
|
4666d0dce0 | ||
|
|
d013795922 | ||
|
|
2b7b0bc3cd | ||
|
|
3809e1ed22 | ||
|
|
e278e4b863 | ||
|
|
c05f5ad451 | ||
|
|
ecc089c6ef | ||
|
|
7523ca5c19 | ||
|
|
3022277167 | ||
|
|
689ada428f | ||
|
|
9ebf5fe85f | ||
|
|
86280608e5 | ||
|
|
411b32ff59 | ||
|
|
04f93133fc | ||
|
|
7da5e9dfa8 | ||
|
|
0d43eb0948 | ||
|
|
5110060c24 | ||
|
|
6f1f55345a | ||
|
|
6af3714bd0 | ||
|
|
495e2229f5 | ||
|
|
37ef4897ae | ||
|
|
06e549cb27 | ||
|
|
c96741794a | ||
|
|
e14d0906ea | ||
|
|
d5be16127b | ||
|
|
b7aef40266 | ||
|
|
fafdaa2617 | ||
|
|
7213068d91 | ||
|
|
05dbe8c5c1 | ||
|
|
806352f8f6 | ||
|
|
34a7af2847 | ||
|
|
635c64d773 | ||
|
|
64d9c54657 | ||
|
|
aec7f0f068 | ||
|
|
4023b21365 | ||
|
|
e14fc884a3 | ||
|
|
b3a84c5bb8 | ||
|
|
9e6254108a | ||
|
|
ce87958b77 | ||
|
|
802014fea9 | ||
|
|
e22acf4913 | ||
|
|
c14c64d6c0 | ||
|
|
6e3e5594eb | ||
|
|
36ff3742b4 |
@@ -1,3 +1,13 @@
|
||||
# LFS routing for image types. Measured 2026-10-21: this repo tracks ZERO LFS files
|
||||
# (`git lfs ls-files` = 0) — the wallpaper library became live theme state and is gitignored
|
||||
# (omp-docs/tools/dotfiles-deploy.md), so these four rules match nothing today. They are kept
|
||||
# because tracked media is a plausible future and idle filters cost nothing; deleting the file
|
||||
# is a pruning call for a session with Frank, not a cleanup an agent runs alone.
|
||||
#
|
||||
# Do not cite this file as the reason `git-lfs` is installed on a box. The manifest carries its
|
||||
# own justification (packages/common.pkgs), and a tool justified by dead config is a tool nobody
|
||||
# can defend when the config goes.
|
||||
|
||||
*.jpg filter=lfs diff=lfs merge=lfs -text
|
||||
*.jpeg filter=lfs diff=lfs merge=lfs -text
|
||||
*.png filter=lfs diff=lfs merge=lfs -text
|
||||
|
||||
+130
-10
@@ -1,25 +1,145 @@
|
||||
#!/usr/bin/env bash
|
||||
# Secret gate: block any commit whose staged diff contains live-secret shapes.
|
||||
# This repo is pushed to a Gitea remote; anything committed is instantly visible
|
||||
# Secret gate: block any commit whose staged diff contains live-secret shapes or
|
||||
# credential assignments. This repo IS the deployment and every workstation
|
||||
# pulls it, so a committed secret is instantly in every agent's context window
|
||||
# on the fleet. Deliberate override: ALLOW_SECRETS=1 git commit ...
|
||||
#
|
||||
# Two rules, deliberately different in scope:
|
||||
#
|
||||
# 1. Token SHAPES are scanned across the ENTIRE staged diff, whatever the file
|
||||
# type. The first version of this gate whitelisted extensions
|
||||
# (*.md *.yml *.yaml *.json *.toml *.sh *.ts *.js), so a `postgres.env`, a
|
||||
# `.container` quadlet, an `.ini`, or an extension-less dotfile — exactly
|
||||
# the shapes the dotfiles tier is made of — was never read at all.
|
||||
# 2. Credential ASSIGNMENTS (PASSWORD=…, secret_key: …) are scanned only in
|
||||
# non-prose files. The auto-learned memory tier is prose ABOUT credentials,
|
||||
# and a gate that fires there gets overridden until it means nothing; the
|
||||
# 2026-10-20 audit measured 7 such prose hits on the tracked tree. Markdown
|
||||
# keeps shape detection only, which is what it had before.
|
||||
#
|
||||
# Bcrypt hashes and Basic-auth headers joined the shape list on 2026-10-20
|
||||
# (issue #40) after a live Caddy basicauth hash plus its plaintext password —
|
||||
# and the Pi-hole internal password — were found in a tracked memory file.
|
||||
set -uo pipefail
|
||||
|
||||
staged=$(git diff --cached --diff-filter=ACM -U0 -- '*.md' '*.yml' '*.yaml' '*.json' '*.toml' '*.sh' '*.ts' '*.js')
|
||||
[ -z "$staged" ] && exit 0
|
||||
shapes='sk-[A-Za-z0-9]{18,}|ghp_[A-Za-z0-9]{36}|github_pat_[A-Za-z0-9_]{22,}|glpat-[A-Za-z0-9_-]{20}|xox[baprs]-[A-Za-z0-9-]{10,}|AKIA[A-Z0-9]{16}|ASIA[A-Z0-9]{16}|BEGIN [A-Z ]*PRIVATE KEY|aes_[A-Za-z0-9/+]{40}|eyJ[A-Za-z0-9_-]{20,}\.ey|\$2[aby]\$[0-9]{2}\$[./A-Za-z0-9]{20,}|Authorization:[[:space:]]*[Bb]asic[[:space:]]+[A-Za-z0-9+/=]{24,}'
|
||||
|
||||
patterns='sk-[A-Za-z0-9]{18,}|ghp_[A-Za-z0-9]{36}|github_pat_[A-Za-z0-9_]{22,}|glpat-[A-Za-z0-9_-]{20}|xox[baprs]-[A-Za-z0-9-]{10,}|AKIA[A-Z0-9]{16}|ASIA[A-Z0-9]{16}|BEGIN [A-Z ]*PRIVATE KEY|aes_[A-Za-z0-9/+]{40}|eyJ[A-Za-z0-9_-]{20,}\.ey'
|
||||
# `newpass1234`, `secret_key: "rotated-value"` — an assignment with a literal
|
||||
# value. `$VAR`, `${VAULT}`, <placeholders> and empty values cannot match.
|
||||
assign='[A-Za-z0-9_]*(PASSW|SECRET|TOKEN|API_?KEY|ACCESS_KEY|PRIVATE_KEY|CREDENTIAL)[A-Za-z0-9_]*[[:space:]]*[:=][[:space:]]*["'"'"']?[A-Za-z0-9/+._@-]{8,}'
|
||||
placeholders='changeme|change[-_]|placeholder|redacted|example|sample|dummy|nokey|no[-_]key|not[-_]needed|your[-_]|localhost|xxxx|yyyy|\$\{|<[a-z]|\*\*\*|%s|null|false|true'
|
||||
|
||||
# Exemptions: a variable naming a location (`…_FILE=`, `…_PATH=`) or a value
|
||||
# that is a filesystem path is a pointer, not a credential. Flagging those is
|
||||
# how a gate gets permanently overridden.
|
||||
assign_exempt='(_FILE|_PATH|_DIR|_URL|_ENDPOINT|_SOCKET)[[:space:]]*[:=]|[:=][[:space:]]*["'"'"']?(/|\.\./)'
|
||||
# Fail closed: a gate that cannot read the staged diff must not report success.
|
||||
#
|
||||
# `-z`, and the list lives in a file. Two traps decide this: git quotes any path it cannot print
|
||||
# as UTF-8 (core.quotePath is on by default), and a quoted name handed back to
|
||||
# `git diff -- <path>` matches nothing — the file is skipped, and a skipped file used to be a
|
||||
# silent pass. Command substitution cannot hold the list either, because $() deletes NUL bytes,
|
||||
# so the names go to a temp file and the loop reads them with a NUL delimiter.
|
||||
TMPD=$(mktemp -d) || TMPD=""
|
||||
if [ -z "$TMPD" ]; then
|
||||
echo "SECRET GATE: cannot create a temp dir — refusing to commit." >&2
|
||||
[ -n "${ALLOW_SECRETS:-}" ] && exit 0
|
||||
exit 1
|
||||
fi
|
||||
trap 'rm -rf "$TMPD"' EXIT
|
||||
if ! git diff --cached --diff-filter=ACM --name-only -z > "$TMPD/files" 2> "$TMPD/err"; then
|
||||
{
|
||||
echo "SECRET GATE: cannot read the staged diff — refusing to commit."
|
||||
sed 's/^/ /' "$TMPD/err"
|
||||
echo "Override deliberately: ALLOW_SECRETS=1 git commit ..."
|
||||
} >&2
|
||||
[ -n "${ALLOW_SECRETS:-}" ] && exit 0
|
||||
exit 1
|
||||
fi
|
||||
# git's stderr stays out of the name list on purpose: a `warning:` line mistaken for a path
|
||||
# becomes a file that "cannot be read", which is how a gate starts skipping the real ones.
|
||||
[ -s "$TMPD/files" ] || exit 0
|
||||
|
||||
report=""
|
||||
exempt=""
|
||||
count=0
|
||||
while IFS= read -r -d '' f; do
|
||||
[ -z "$f" ] && continue
|
||||
# Self-exemption, deliberately an exact filename list and not `tests/`: the
|
||||
# gate's fixtures must contain credential-shaped strings or there is nothing to
|
||||
# test against, and this hook documents the shapes it looks for. A secret parked
|
||||
# in any other test file is still caught.
|
||||
case "$f" in
|
||||
.githooks/pre-commit|tests/secret-gate.bats) exempt="$exempt $f"; continue ;;
|
||||
esac
|
||||
diff=$(git diff --cached -U0 -- "$f")
|
||||
[ -z "$diff" ] && continue
|
||||
# Only ADDED lines can leak. Matching removals too blocks the very commit that
|
||||
# deletes a credential — which is exactly what happened on 2026-10-20, when the
|
||||
# gate refused the redaction of the password it exists to catch.
|
||||
added=$(printf '%s\n' "$diff" | grep -E '^\+' | grep -vE '^\+\+\+' | sed 's/^+//' || true)
|
||||
[ -z "$added" ] && continue
|
||||
# Shape hits are judged on the MATCHED TOKEN, not on the whole line. Documentation about
|
||||
# credentials quotes vendor placeholders — AWS publishes an all-zeros access-key id that ends
|
||||
# in EXAMPLE for exactly this purpose — and
|
||||
# a gate that blocks prose over an inert example gets worked around until it means nothing:
|
||||
# the live-sync heartbeat failed three ticks on 2026-10-21 because a managed skill page quotes
|
||||
# exactly that id. A line is exempted only when EVERY shape on it is inert, so one live token
|
||||
# sharing the file with a documented fake still blocks.
|
||||
hits=$(printf '%s\n' "$added" | while IFS= read -r line; do
|
||||
live=0
|
||||
while IFS= read -r m; do
|
||||
# The shape regexes carry fixed quantifiers, so the match they return is TRUNCATED:
|
||||
# `AKIA` + exactly 16 characters stops inside the word EXAMPLE and the placeholder
|
||||
# disappears from what is judged. Judge the match plus what follows it instead.
|
||||
ctx=$(printf '%s' "$line" | awk -v m="$m" '{ i = index($0, m); print (i > 0 ? substr($0, i, length(m) + 8) : m) }')
|
||||
printf '%s' "$ctx" | grep -Eiq 'EXAMPLE|X{4,}|Y{4,}|Z{4,}|_{4,}|REDACTED|CHANGEME|PLACEHOLDER' || live=1
|
||||
done < <(printf '%s\n' "$line" | grep -Eio "$shapes" || true)
|
||||
[ "$live" = 1 ] && printf '%s\n' "$line"
|
||||
done)
|
||||
if [ "${f##*.}" != "md" ]; then
|
||||
# The placeholder filter reads the VALUE, never the whole line. A trailing comment that
|
||||
# says `# example` says nothing about the token the same line assigns; scoping the filter
|
||||
# to what follows the separator is what keeps `API_TOKEN=<real> # placeholder` a block.
|
||||
assigns=$(printf '%s\n' "$added" | grep -Ei "$assign" | grep -Ev "$assign_exempt" \
|
||||
| while IFS= read -r line; do
|
||||
# The judged value is the assigned token alone: it ends at whitespace or `#`, so a
|
||||
# trailing comment cannot vouch for it. Truncating a quoted value early only ever errs
|
||||
# toward blocking, which is the safe direction for a gate that can be overridden.
|
||||
value=${line#*[:=]}
|
||||
value=${value#"${value%%[![:space:]]*}"}
|
||||
value=${value%%[[:space:]]*}
|
||||
value=${value%%#*}
|
||||
printf '%s\n' "$value" | grep -Eiq "$placeholders" && continue
|
||||
printf '%s\n' "$line"
|
||||
done)
|
||||
hits=$(printf '%s\n%s\n' "$hits" "$assigns" | sed '/^[[:space:]]*$/d')
|
||||
fi
|
||||
if [ -n "$hits" ]; then
|
||||
n=$(printf '%s\n' "$hits" | wc -l)
|
||||
count=$((count + n))
|
||||
report+=" $f"$'\n'"$(printf '%s\n' "$hits" | cut -c1-100 | sed 's/^/ /')"$'\n'
|
||||
fi
|
||||
done < "$TMPD/files"
|
||||
|
||||
# Say so out loud: an exemption that prints nothing is how a carve-out becomes
|
||||
# invisible, and invisible is the property a secret wants.
|
||||
if [ -n "$exempt" ]; then
|
||||
printf 'secret-gate: self-test fixtures exempt:%s\n' "$exempt" >&2
|
||||
fi
|
||||
|
||||
hits=$(printf '%s\n' "$staged" | grep -Ei "$patterns" || true)
|
||||
if [ -n "${ALLOW_SECRETS:-}" ]; then
|
||||
[ -n "$hits" ] && printf 'secret-gate: ALLOW_SECRETS set, allowing %s secret-shaped string(s) through.\n' "$(printf '%s\n' "$hits" | wc -l)"
|
||||
[ -n "$report" ] && printf 'secret-gate: ALLOW_SECRETS set, allowing %s secret-shaped string(s) through.\n' "$count"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ -n "$hits" ]; then
|
||||
if [ -n "$report" ]; then
|
||||
{
|
||||
echo "SECRET GATE: staged diff contains $(printf '%s\n' "$hits" | wc -l) secret-shaped string(s):"
|
||||
printf '%s\n' "$hits" | sed 's/^/ /'
|
||||
echo "SECRET GATE: staged diff contains $count secret-shaped string(s):"
|
||||
printf '%s\n' "$report"
|
||||
echo "Fix it, or override deliberately: ALLOW_SECRETS=1 git commit ..."
|
||||
echo "A credential that reaches a tracked file is already in history on every"
|
||||
echo "machine that pulls this repo — deleting it later does not un-leak it."
|
||||
} >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
+36
@@ -26,6 +26,10 @@ auth.json
|
||||
**/auth.json
|
||||
.env
|
||||
**/.env
|
||||
*.env
|
||||
# `*.env` is the rule that matters: `.env` alone does not match `postgres.env`
|
||||
# or `frank-job-hunt.env`, and those are the files that actually carry
|
||||
# POSTGRES_PASSWORD / API keys on this fleet (issue #40).
|
||||
trust.json
|
||||
**/trust.json
|
||||
*.pem
|
||||
@@ -74,3 +78,35 @@ wt/
|
||||
# A session run inside a git worktree mints a bucket named after the worktree
|
||||
# path; the worktree is deleted, the bucket is orphan state, not config.
|
||||
agent/memories/--home-blank-.omp-wt-*
|
||||
|
||||
# --- theme media: content the pipeline consumes, not configuration ---------
|
||||
# 23 wallpapers, 80 MiB. Tracked until 2026-10-20: every clone of a config repo was
|
||||
# carrying a library no diff can review. The live library at
|
||||
# ~/.local/share/noctalia/wallpapers is a real directory of real files (measured
|
||||
# 2026-10-20), so this path in the repo was a duplicate of it, not its source — and
|
||||
# note that untracking deletes the copy in every other checkout regardless of
|
||||
# `git rm --cached` on the branch. A fresh box starts with an empty library; every
|
||||
# filename carries its wallhaven ID; `/w/<id>` answers `text/html`, so resolve the CDN URL at
|
||||
# `https://wallhaven.cc/api/v1/w/<id>` and download `data.path` — the recipe, with the reason it
|
||||
# is not a plain `curl -L`, is on omp-docs/tools/dotfiles-deploy.md
|
||||
dotfiles/noctalia/.local/share/noctalia/wallpapers/
|
||||
|
||||
# --- generated theme output: machine-written, never reviewable, never deployed ---
|
||||
# noctalia/matugen rewrite these in place; a tracked copy is a stale snapshot that
|
||||
# deploy.sh would push back over a live palette. These patterns are the ignore half of the
|
||||
# exclusion list in `deploy_file` (scripts/deploy.sh): change one, change the other, or a
|
||||
# generated file becomes tracked-but-never-deployed — the fiction class #41 removed. The
|
||||
# capital-N rule is noctalia's own palette-directory naming, and it was in deploy.sh only.
|
||||
# Law: omp-docs/inventory/home-config-census.md, Refs #42, #60
|
||||
dotfiles/**/themes/noctalia*
|
||||
dotfiles/**/colors/noctalia*
|
||||
dotfiles/**/noctalia.*
|
||||
dotfiles/**/noctalia-*
|
||||
dotfiles/**/Noctalia*
|
||||
|
||||
# --- harness natives: unpacked runtime payloads, never configuration ----------
|
||||
# 2.4 MB of tree-sitter grammars the harness unpacks into the repo root on first use (measured
|
||||
# on blankpad 2026-10-21). Regenerable, unreviewable, not config — every clause of AGENTS.md's
|
||||
# runtime-state doctrine applies, and it sat unnamed in the root, one `git add -A` away from
|
||||
# being committed as if it were configuration.
|
||||
natives/
|
||||
|
||||
@@ -1,28 +0,0 @@
|
||||
---
|
||||
name: command-center-dotfiles-import
|
||||
description: Runbook for importing client env-layer configs into the command-center repo and deploying them via stow on new fleet machines; decisions agreed with Frank 2026-09-09 (pre-execution — verify execution state before assuming done).
|
||||
---
|
||||
|
||||
Design agreed with Frank for importing client application/environment configs into the `~/.omp` command-center repo and deploying them on fresh machines. **Status: plan agreed, not yet executed** — check repo state (does `dotfiles/` or `scripts/deploy.sh` exist?) before re-deriving; if already implemented, this becomes the operating model, not a to-do.
|
||||
|
||||
## Agreed decisions (each was an explicit user choice)
|
||||
1. **Target**: unknown/TBD fresh machine → design stays host-agnostic; deploy derives overrides from `$(hostname)` at run time. Nothing hardcodes a target.
|
||||
2. **Scope**: environment layer only — niri (+cfg includes), gtk-3.0/4.0 settings, ghostty, alacritty, shell rc files (`.zshrc`, `.bashrc`, `.profile`, `.p10k.zsh`), fish, systemd user units, autostart, dconf. App *data* (browser profiles, vaults, DBs) never imported.
|
||||
3. **Layout**: stow-style package dirs `dotfiles/<pkg>/<home-relative path>` inside the repo; repo clones to `~/.omp` on every host that runs the harness (full command-center clone, not env-layer only).
|
||||
4. **Deploy mechanism**: symlink farm — `stow -t ~` from `~/.omp/dotfiles`; relative symlinks within `$HOME` work because repo root lives inside `$HOME`.
|
||||
5. **Dogfood order**: convert the workstation itself (real dirs → symlinks via adopt) BEFORE any target import; iterate where the compositor reload is observable.
|
||||
6. **Secrets policy**: never commit secret values. Split rc files into committed logic + untracked sourced-if-exists stub (`~/.config/environment.d/secrets.env` or `.zshrc.local`); real values provisioned out-of-band only on machines that need them. Pre-commit secret scanner stays as safety net, not gate of first resort.
|
||||
7. **Divergence**: `zz-<hostname>` override package stowed LAST with `--override`; only per-host diffs live there. (Stow descends into conflicting symlinked dirs — per-file overrides only work via this last-stowed `--override` pattern.)
|
||||
8. **Deploy trigger**: explicit `scripts/deploy.sh` invocation, human- or agent-invoked. NO auto-deploy on pull — pull and deploy are separate decisions; `--check` = drift report mode.
|
||||
9. **Ownership**: Frank is sole push authority; exactly one editing agent at a time → no multi-writer race machinery needed. Each host's agent edits only its own `zz-<hostname>` package.
|
||||
10. **Conflict policy**: pre-existing regular file that differs from repo truth → move to timestamped `.quarantine/` dir on the target (never `rm`, same convention as this repo's trash rule), symlink anyway, print every replacement in deploy output and `--check` report. Repo truth always wins; divergence recoverable from quarantine.
|
||||
11. **First-clone auth**: human types credential once console-side (URL-with-token or prompt); the new machine's secrets exist only there afterward.
|
||||
|
||||
## Implementation gotchas baked into the design
|
||||
- `core.hooksPath` is a per-clone local setting — does NOT survive clone; bootstrap/deploy must set it on every fresh clone or the secret-gate pre-commit hook silently doesn't run there.
|
||||
- dconf/gsettings are databases, not files: import as `dconf dump /` text into repo; deploy runs `dconf load` instead of symlinking.
|
||||
- p10k installer and toolchain hooks append to rc files at install time — expect conflicts on `.zshrc`-class files at first adopt; quarantine-and-replace handles it, but diff live-vs-repo before adopting so pre-existing drift is seen, not silently adopted.
|
||||
- Known secret at import time: `GITEA_TOKEN` export lived in `.zshrc` (line ~22) on the workstation — quarantine BEFORE importing rc files; consider token rotation at cutover since it had been exposed in plaintext rc + remote URL.
|
||||
|
||||
## Deploy script spec agreed
|
||||
`scripts/deploy.sh`: sets hooksPath → stows all base packages → stows `zz-$(hostname)` with `--override` → first run uses `--adopt` for pre-existing identical files → divergent files quarantined-and-reported; `--check` mode emits drift report (modified tracked files vs deployed symlinks) without mutating.
|
||||
@@ -1,33 +0,0 @@
|
||||
---
|
||||
name: stash-supersession-audit-before-drop
|
||||
description: "Before dropping a stale git stash, prove it's superseded by HEAD via per-file line-set diff of git show stash@{0}:file vs HEAD; archive as an annotated tag and drop only after auditing unique facts — zero-loss alternative to blind drop."
|
||||
---
|
||||
|
||||
# Stash Supersession Audit Before Drop
|
||||
|
||||
Use when a repo (especially a live-deployment checkout where the working tree IS the deployment, e.g. `~/.omp` command-center) has an old stash whose content may have been superseded by later commits, and dropping blindly risks losing unique facts.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. **Inventory the stash** — see which files it touches:
|
||||
```bash
|
||||
git stash show --stat stash@{0}
|
||||
```
|
||||
2. **Probe for unique content per file.** For each touched file, diff stash version against HEAD and count lines that exist only in the stash version:
|
||||
```bash
|
||||
git show "stash@{0}:<path>" > /tmp/stash_copy
|
||||
# compare normalized line sets; report lines present in stash but absent from HEAD
|
||||
```
|
||||
A file with `unique-vs-HEAD = 0` is fully superseded — safe to drop for that file.
|
||||
3. **Classify each unique-content hit**:
|
||||
- *Stale fact* (superseded by a later refactor, e.g. old model/provider naming) → drop is correct.
|
||||
- *Live durable fact* (e.g. an operational invariant never re-recorded) → extract it into the live memory/docs file or flag to the user before dropping.
|
||||
4. **Archive before dropping** — zero-loss: tag the stash commit, then drop from the stash list:
|
||||
```bash
|
||||
git tag archive/stash-<YYYY-MM-DD>-<base-branch>-wip "stash@{0}"
|
||||
git stash drop "stash@{0}"
|
||||
```
|
||||
The tag pins the full stash object (including its index parent); recoverable later via `git show archive/stash-...` or `git checkout archive/stash-...^3 -- <path>` for the untracked-files third parent if present.
|
||||
|
||||
## Why
|
||||
Stashes are invisible in normal history and expire from reflogs (~90 days). A stash that *looks* superseded can still hold the only copy of a durable fact; tagging converts "drop" into a reversible archive with zero workflow cost.
|
||||
@@ -86,15 +86,44 @@ never committed, only surfaced. Conflicts abort and alert (ntfy); it never
|
||||
force-pushes, auto-resolves, or uses `--no-verify`. Details:
|
||||
[omp-docs/tools/live-sync.md](omp-docs/tools/live-sync.md).
|
||||
|
||||
## Agent skills
|
||||
|
||||
The engineering skills (`triage`, `to-tickets`, `to-spec`, `research`,
|
||||
`implement`, …) read their per-repo configuration from three pages in the
|
||||
knowledge base. Those pages own their facts; this is only the pointer.
|
||||
|
||||
### Issue tracker
|
||||
|
||||
See [`omp-docs/skills/issue-tracker.md`](omp-docs/skills/issue-tracker.md).
|
||||
|
||||
### Triage labels
|
||||
|
||||
See [`omp-docs/skills/triage-labels.md`](omp-docs/skills/triage-labels.md).
|
||||
|
||||
### Domain docs
|
||||
|
||||
ADRs: `omp-docs/decisions/`. `omp-docs/glossary.md` is intentionally not created.
|
||||
See [`omp-docs/skills/domain-docs.md`](omp-docs/skills/domain-docs.md).
|
||||
|
||||
## Secret gate (HARD rule)
|
||||
|
||||
The pre-commit hook (`.githooks/pre-commit`, enabled via
|
||||
`git config core.hooksPath .githooks`) scans every staged diff for live-secret
|
||||
shapes (`sk-…`, `ghp_…`, `github_pat_…`, `glpat-…`, `xox…`, `AKIA…`, PEM headers,
|
||||
JWTs) and blocks the commit. Quarantine list — never tracked, ever: `auth.json`,
|
||||
`.env`, `trust.json`, `*.pem`, `*.key`, `identity*`. Deliberate override:
|
||||
`git config core.hooksPath .githooks`) blocks the commit in two ways. Token
|
||||
**shapes** (`sk-…`, `ghp_…`, `github_pat_…`, `glpat-…`, `xox…`, `AKIA…`, PEM
|
||||
headers, JWTs, bcrypt hashes, `Authorization: Basic` headers) are scanned across
|
||||
the **entire** staged diff — every file type, because the first version of the
|
||||
gate whitelisted extensions and never read a `.env`, `.container`, `.ini`, or
|
||||
extension-less dotfile at all. Credential **assignments** (`PASSWORD=…`,
|
||||
`secret_key: …`) are scanned in non-prose files only; markdown keeps shape
|
||||
detection, because the auto-learned memory tier is prose *about* credentials and a
|
||||
gate that fires there gets overridden until it means nothing. The gate **fails
|
||||
closed**: if it cannot read the staged diff it refuses the commit rather than
|
||||
passing it. Quarantine list — never tracked, ever: `auth.json`, `.env` and
|
||||
`*.env` (`postgres.env`, `frank-job-hunt.env` are the ones that carry real
|
||||
values), `trust.json`, `*.pem`, `*.key`, `identity*`. Deliberate override:
|
||||
`ALLOW_SECRETS=1 git commit …`. Remember why: a secret in a tracked file is a
|
||||
secret in every agent's context window on every machine that pulls this repo.
|
||||
secret in every agent's context window on every machine that pulls this repo, and
|
||||
redacting it later does not un-leak the history.
|
||||
|
||||
## Machine facts (verify before trusting)
|
||||
|
||||
|
||||
+18
-3
@@ -54,13 +54,27 @@ Suggest these tools or languages as possible solutions to problems if they are a
|
||||
|
||||
### 1. Maintain your projects source of truth.
|
||||
|
||||
The knowledge base at `docs/` is the source of truth for your project. Your training is out of date. Default to the documentation and the code before you assume anything. Maintain a work log of actions taken and their result. Refer to the work log to remember the progress of your tasks.
|
||||
Documentation exists for ONE reason: **point of use**. A page earns its bytes only if it helps you perform a task at the moment you perform it — the shape to deploy, the flag that must be passed, the failure mode to recognise, the rollback to run. It is NOT a museum, NOT a lab notebook, and NOT a place to keep track of numbers.
|
||||
|
||||
**THE PURPOSE TEST — apply it before you write, and again before you keep:**
|
||||
- Would an agent or Frank, doing a real task tomorrow, read this line and act differently because of it? If no: **delete it.**
|
||||
- Is this a measured number (t/s, tokens, ms, MiB, %, bench ladder, acceptance rate) for a model, runtime, or hardware configuration that is NOT currently deployed and NOT currently being compared? Then it is **dead weight — destroy it**, do not archive it, do not "preserve it in an Eras row", do not cite it. AI advances make such a figure wrong within weeks, and a wrong number in a runbook is worse than no number: it gets believed.
|
||||
- "It is the only copy of that fact" is NOT a reason to keep it. Sole custody of a useless fact is still useless. Losing information is an acceptable — often correct — outcome; do not hesitate to lose it.
|
||||
- What survives the test: live configuration and its exact numbers, the procedure that must be followed, the law/causal story that prevents a repeat failure, security posture, and the interface contract a consumer depends on. Qualitative laws outlive the measurement that taught them — keep the law, drop the numbers.
|
||||
- Never let a tool's "this is duplicated / this is the only record" census decide what to keep. That is a question of USEFULNESS, which only the operator answers.
|
||||
|
||||
1. Read the relevant `index.md` before you touch a topic.
|
||||
2. Follow `omp-docs/references/okf-spec.md` for every doc you create or edit.
|
||||
3. Read the target file completely before you edit it.
|
||||
4. Always update the documentation when you've completed a task.
|
||||
5. Compact the work log when necessary.
|
||||
4. Always update the documentation when you've completed a task — and when updating means deleting a page or a paragraph, delete it.
|
||||
5. Compact the work log when necessary; a work log holds laws and incidents, never bookkeeping.
|
||||
|
||||
#### THE WORK LOG IS ONE FILE.
|
||||
The work log is a single page in the knowledge base: `omp-docs/log.md` in this repo, or whichever change-log page the repo's docs index names. Write the entry when the change lands — in the same commit as the change — and read that page to recover where things stand before starting. It follows its own entry law: a dated entry is what changed plus the durable lesson, a few lines.
|
||||
|
||||
- NEVER create a second log: not per directory, not per project, feature, branch or run, not `NOTES.md` / `PROGRESS.md` / `STATUS.md` / `worklog/`, not a dated diary. In `dank/command-center` the KB linter reports any `log.md` that is not the bundle root's; the rule and its reasoning are owned by `omp-docs/references/okf-spec.md`.
|
||||
- A log holds laws and incidents, never bookkeeping: no per-file inventories, no run-by-run numbers, no transcripts. A number in a log still has to pass the purpose test above.
|
||||
- If a knowledge base has no log page, add one at its root and list it in the index. That is the fix — never a log somewhere else.
|
||||
|
||||
#### USE PROGRESSIVE DISCLOSURE.
|
||||
Always ensure that all produced documentation is laid out in files following WIKI conventions: start with a top level README.md index file that provides a brief, concise introduction to the item being documented, and then SPLIT OFF individual aspects, features, or arguments into their own interlinked pages so that future agents can read the documentation in a piecewise fashion, only taking in the details they need. Where relevant, crosslink any created documentation with other relevant documentation. This structure may also be nested, if necessary.
|
||||
@@ -90,6 +104,7 @@ Never assume the user reports correct information. He is human, and humans make
|
||||
- Before starting new work, `git pull` from the remote, and start a new branch in a separate worktree using `master` or `main` as the base branch.
|
||||
- Commit atomically: after every change that forms a complete functional unit, commit. Do so early and often.
|
||||
- When the work is complete, ask the user if they would like to open a pull request.
|
||||
- **Report a pull request as link AND title, every time.** A bare `#54` is not a reference: numbers are per-repository, so a number alone does not say which work, in which repo, is being reported. Give the PR's web URL on that forge — on the Dankclaw Gitea that is `https://gitea.franksdank.cloud/<owner>/<repo>/pulls/N` — together with the PR's title. Applies everywhere a PR is mentioned: chat replies, status summaries, issue and commit bodies, alerts. Another forge (GitHub, GitLab, …): the equivalent URL its web UI serves, same pairing.
|
||||
|
||||
### 5. Test Driven Development
|
||||
|
||||
|
||||
+1
-1
@@ -68,7 +68,7 @@ retry:
|
||||
- llamacpp (dankclaw)/frank-dragoon
|
||||
maxRetries: 10
|
||||
searxng:
|
||||
endpoint: search.franksdank.cloud
|
||||
endpoint: http://192.168.0.143:8081
|
||||
exa:
|
||||
enabled: false
|
||||
memory:
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
name: amd-apu-vulkan-tenant-onboarding
|
||||
description: "Add a ggml/Vulkan inference tenant to an AMD APU (gfx1151 / Strix Halo iGPU) that already serves another engine: APU memory census, RADV heap gate, render-node resolution, build-side glslc/ICD deps, co-tenancy measurement."
|
||||
---
|
||||
|
||||
Use when a ggml-based engine (llama.cpp, qwentts.cpp, anything built on ggml) must move from CPU to the **iGPU** of an AMD APU (gfx1151 / Strix Halo / Ryzen AI Max) that **already serves another engine on the same card**. Answers two questions only a booted host can: what gate the second tenant must clear, and which files the build needs to clear it.
|
||||
|
||||
## 1. The usual VRAM reasoning is wrong on an APU
|
||||
|
||||
`/sys/class/drm/cardN/device/mem_info_vram_total` reports a small **carve-out** (2 GiB class), and `rocm-smi`'s "dedicated" figure is not the working set either. The real working set lives in **GTT / system pages**: `/proc/meminfo` → `GTT` and `System pages`. A second GPU tenant therefore consumes host RAM from the same pool the first tenant's weights and the OS use — no supplementary groups, no separate aperture to reserve.
|
||||
|
||||
- Price the tenant from a **booted** footprint, never from `weights + KV + buffers`: on an APU the compute buffers and any projector ride on GTT, which is host RAM; check `free` *after* the load, not before.
|
||||
- Never subtract a "dedicated" figure from it to find headroom.
|
||||
- Unheld fabric clock (`pp_dpm_mech` = `auto`) is a hazard only when the NPU joins too; a pure iGPU pair is unaffected — check it before adding a third tenant.
|
||||
- Swap on a host with GPU tenants protects co-tenants; paging a KV pool is worse than the OOM it prevents.
|
||||
|
||||
## 2. The gate that decides feasibility: the RADV heap a container sees
|
||||
|
||||
`rocm-smi` and `free` cannot see this number; only a container off the **engine's own base image** can:
|
||||
|
||||
```bash
|
||||
# one-off container, no model, no load, no compute — safe beside production
|
||||
podman run --rm --device /dev/dri --group-add keep-groups <engine-base> bash -c \
|
||||
'apt-get update -qq >/dev/null 2>&1; apt-get install -y -qq --no-install-recommends mesa-vulkan-drivers vulkan-tools glslang-tools >/dev/null 2>&1; vulkaninfo --text'
|
||||
```
|
||||
|
||||
Read `VkPhysicalDeviceMemoryProperties` → the `DEVICE_LOCAL` heap size. On a hybrid Strix Halo box Mesa splits the aperture (~1/3 : 2/3), so `DEVICE_LOCAL` is the budget the ggml tool will actually allocate against — measured 77.33 GiB *with the first tenant loaded*, against a 2 GiB DRM carve-out. Two traps, both measured:
|
||||
|
||||
- `vulkaninfo --batch` does not exist in Ubuntu 24.04's `vulkan-tools` (prints help, exits 1). Every `vulkaninfo` parse needs `--text`.
|
||||
- The container may have **no NVIDIA ICD**, so Vulkan index `0` *is* the iGPU — but do not trust ordering by index: `--text` and read `deviceName` + `driverID` (`RADV_GFX1151` vs `NVIDIA`). A hand-picked `/dev/dri/renderDN` is worse than useless on a hybrid host: **`renderD128` there is the NVIDIA card and `renderD129` the amdgpu iGPU**; resolve the node with `readlink -f /sys/class/drm/renderD*/device/driver` and pass `/dev/dri` wholesale, letting the driver pick by name.
|
||||
- A rootless container that drops its supplementary groups enumerates **nothing**. `--group-add keep-groups` (quadlet: `GroupAdd=keep-groups`) plus the host udev rule that makes the render nodes group-writable is the whole requirement; `--device /dev/dri` alone is not enough.
|
||||
|
||||
## 3. Build-side deps: the configure fails before the device ever matters
|
||||
|
||||
ggml's `src/ggml-vulkan/CMakeLists.txt` requires, in order: `find_package(Vulkan COMPONENTS glslc REQUIRED)` then `find_package(SPIRV-Headers CONFIG REQUIRED)`, and it compiles its shaders with **glslc**, never `glslangValidator`. On Ubuntu 24.04 (measured against the digest-pinned `ubuntu:24.04` base, 2026-10-19):
|
||||
|
||||
- `glslang-tools` ships `glslangValidator` and **NOT `glslc`** → `command -v glslc` empty, `find_package(Vulkan … glslc REQUIRED)` fails, and the Vulkan backend silently never exists.
|
||||
- The package is the standalone **`glslc`** (universe, `2023.8-1build1`). Working build-stage set: `libvulkan-dev glslc spirv-headers` — `libvulkan-dev` provides `vulkan/vulkan.h` + the `VulkanHeaders`/`VulkanLoader` CMake configs `find_package(Vulkan)` resolves to, `spirv-headers` ships `/usr/share/cmake/SPIRV-Headers/SPIRV-HeadersConfig.cmake`. A configure probe in the pinned base prints `Found Vulkan: … found components: glslc` then `Including Vulkan backend`.
|
||||
- **The archive's glslc is old.** 2023.8 compiles `GL_KHR_cooperative_matrix` (the path ggml's vectored quantized matmul needs) but reports `cooperative_matrix2`, `integer_dot_product`, `bfloat16`, `float_e2m1/e4m3` as unsupported, so those shader families are compiled out. If a Vulkan prefill reads as inexplicably slow, suspect the **shader compiler before the device**; the fix is a pinned glslang build passed as `-DVulkan_GLSLC_EXECUTABLE=<path>`, not a different runtime.
|
||||
- Runtime stage needs the loader **and an ICD**: `libvulkan1` + `mesa-vulkan-drivers`. Loader-without-ICD enumerates zero devices, and ggml-based servers die in device lookup — often *after* reading the weights, so the journal reads like a model fault.
|
||||
- Ship **no NVIDIA ICD** on purpose: one Vulkan device in the enumeration is what makes the *name* `Vulkan0` safe to write into config. Prefer the name over an index in every flag.
|
||||
|
||||
## 4. Prove enumeration without allocating anything
|
||||
|
||||
Do not boot a scratch second engine beside production to ask which devices exist — a second model load is a real GPU tenant, and fleet law stops that. Instead run the binary with a device name that cannot exist and read its error; it names what it enumerated, with no allocation and no model resident:
|
||||
|
||||
```bash
|
||||
podman run --rm --device /dev/dri --group-add keep-groups --entrypoint /app/tts-server <image> \
|
||||
--model /dev/null --codec /dev/null --device NoSuchDevice 2>&1 | tail -20
|
||||
```
|
||||
|
||||
## 5. The rollback must be a flag, not a rebuild
|
||||
|
||||
Put the device in the env file as a value the entrypoint passes only when set, and make **set-but-empty** mean "pass no flag at all" — the binary's own default (CPU, always built) is the rollback. `: "${VAR:=default}"` is the WRONG form: `:=` fires on an empty value too, so the documented rollback line `VAR=` is silently rewritten back to the GPU and the only way out is a rebuild. Use:
|
||||
|
||||
```bash
|
||||
if [[ ! -v QWENTTS_DEVICE ]]; then QWENTTS_DEVICE=Vulkan0; fi
|
||||
```
|
||||
|
||||
(`podman`'s `--env-file` passes `VAR=` through as set-and-empty, and `source` does the same, so both readers agree.) Pin it in a test: empty device ⇒ the flag is absent from argv.
|
||||
|
||||
## 6. Co-tenancy measurement (only after the tenant is live)
|
||||
|
||||
The gate is the **first tenant's** performance with a real request from the second in flight — not the second tenant's own speed. And price the tenant from a booted footprint, never from weights+KV+buffers: on an APU the compute buffers and any projector ride on GTT, which is host RAM; check `free` after the load, not before.
|
||||
@@ -1,61 +1,41 @@
|
||||
---
|
||||
name: audit-subagent-deliverables
|
||||
description: "Integrate subagent-written code safely: line-by-line audit, mechanical gates (bash test/lint), adversarial-review-report triage, resource-safety pass (fd/memory), and cutover with running-process restarts."
|
||||
description: "Integrate subagent-written code safely: line-by-line audit, mechanical gates (bash test/lint), merge shared-function collisions, fix scope creep, verify env var propagation."
|
||||
---
|
||||
|
||||
# Audit subagent deliverables
|
||||
Integrating code written by subagents (fan-out implementation). The code is usually correct; the failures are structural — collisions between agents, scope creep, and assumptions that never reach the runtime.
|
||||
|
||||
A subagent that wrote code is a source of plausible output, not a verified change. Integrate it the way you'd integrate an untrusted PR — with extra suspicion, because the author is an agent that shares your blind spots and its prose is more confident than its evidence.
|
||||
## Workflow
|
||||
|
||||
Order of operations: read everything → gates → tests (as evidence, not proof) → resource-safety pass → fix in-place → cutover.
|
||||
1. **Read every deliverable line by line** before integrating — not the summary.
|
||||
2. **Run the mechanical gates immediately**, in parallel: `bash -n` on every script; `bats tests/ 2>&1 | grep "not ok"` for the failure list, never `tail` (tail shows a passing tail and hides the red in the middle).
|
||||
3. **Merge shared-function collisions.** When two agents add different versions of the same function, pick the superset and re-point the other's callers at it. (Real case: one agent's `fleet_base()` carried `-b`, the other's `resolve_fleet_base()` carried `--base-url`; resolve to the one flag set the tests use.)
|
||||
4. **Delete scope creep.** Code added for a goal nobody asked for — a new env var, a new CLI flag, a new output file — goes, with its docs, unless the operator asked.
|
||||
5. **Verify env var propagation.** A var the orchestrator computes must be exported before the `bash -c` / subshell that reads it, or the child silently takes a default. Check the export site, not the doc.
|
||||
|
||||
## 1. Read every produced file, line by line
|
||||
Full file, not a diff skim. You are the reviewer who must catch the line the author never thought about.
|
||||
## Audit checks
|
||||
|
||||
## 2. Mechanical gates before logic review
|
||||
```bash
|
||||
bash -n <script> # syntax (scripts)
|
||||
bunx tsc --noEmit # types (TS)
|
||||
bun test # suite
|
||||
```
|
||||
Run the suite for information, not as the stopping point — it only proves the author's own scenarios.
|
||||
| Check | How |
|
||||
|---|---|
|
||||
| Flag names real | `bash script.sh --help` |
|
||||
| Paths exist | `test -e` each literal in the diff |
|
||||
| Env reaches the runtime | grep the export/inline-assignment site, then the read |
|
||||
| Tests assert behaviour | not `[[ -n "$X" ]]`, not `test -f <new file>` |
|
||||
| Tests not weakened | `git diff HEAD~ -- tests/` — a deleted assertion is a red flag |
|
||||
|
||||
## 3. Hunt the three classic agent defects
|
||||
- **Quoting bugs:** `set -e` + `"${@/$p/$r}"` = substitution regex containing user input (bash can't escape `/` in a `${var/re/pat}` pattern: `/` reads as the pattern/replacement separator). Fix with a separator absent from inputs (e.g. `%`) and `printf '%q'` for shell-interpolated values.
|
||||
- **`cd` assumptions:** parallel `runShell` calls share one CWD (pool mode). Any command sequence that assumes CWD (e.g. `git -C .` after a sibling's `cd`) must use explicit absolute paths.
|
||||
- **Unbounded resources:** every fd opened must be closed on ALL paths (success, throw, validation fail) — `using` for fs.sinks (closes on throw; sinks are `AsyncDisposable`, not `fs.FileHandle`, no `.close()`); bounded `Map` for caches (`MAX` + FIFO eviction); explicit caps (lines read, file sizes).
|
||||
## Red flags
|
||||
|
||||
## 4. Check the test file is real
|
||||
Tests that assert internal state (`expect(wcwd).toBe(...)`) over output contracts; tests of mock echo; tautologies. Delete them per house rules, keep behavior contracts.
|
||||
- Test asserting a new file exists rather than behaviour.
|
||||
- Doc claiming a flag the script never defines.
|
||||
- Function added with no caller.
|
||||
- Doc example that contradicts the code it documents.
|
||||
- Same helper defined twice with different flag spelling.
|
||||
- New env var read but never exported.
|
||||
|
||||
## 5. Cutover audit (repo-wide)
|
||||
`grep -r` the symbol namespace the subagent touched — `wcwd`-style globals are the classic blast radius. Grep every `cd`/`process.chdir` site and verify each paired command sequence uses explicit paths. Stale comments describing the old design must go — comments that lie are time bombs for the next reader.
|
||||
## Re-verify after the operator's own edits
|
||||
|
||||
## 6. Cutover restarts running processes
|
||||
A code fix on disk is NOT live in any process that loaded the file at spawn (bun/node load-at-spawn). After shipping fixes: restart every long-running supervisor/daemon/UI-server that carries the affected code path; prefer takeover protocols (stale-pid detection) that make restarts cheap. Symptom of forgetting: "fixed" code producing old-code symptoms in production (observed: 812 poisoned ledger rows).
|
||||
Run the gates again — their edit may undo an integration (a re-typed helper, a reverted flag).
|
||||
|
||||
## 7. Triaging an adversarial REVIEW report (not code)
|
||||
When a subagent produced findings, treat each finding as a hypothesis with a base rate ~40% (measured over 3 rounds, 2026-10):
|
||||
1. **Walk every finding end-to-end against the CURRENT tree** before accepting — cite file:line yourself. Reviewers frequently read a mid-edit snapshot; many findings are stale by construction.
|
||||
2. **Classification that kills most findings:**
|
||||
- *Stale read* — code changed during review; verify against live lines.
|
||||
- *Inverted logic* — "failed rows are skipped" when the filter makes failed rows RETRY. Re-read the filter.
|
||||
- *Unreachable* — trigger requires breaking a stated design law (single-worker, append-only). Not a defect.
|
||||
- *By-design* — the reviewer labels a spec law a bug (serial execution, non-stream client). Check the spec, not the reviewer's confidence.
|
||||
- *Constant-trust* — "XSS via manifest field" where the field is git-pinned config; "path traversal via REMOTE_TMP" where it's a hardcoded const. Ask: what untrusted input reaches this sink?
|
||||
- *Self-refuting* — the reviewer's own probe output disproves its claim (raw==dedup probe in a "dedup" finding). Quote it back.
|
||||
3. **Acceptance bar:** concrete trigger with real inputs + full path walked. Real accepts cluster in: resource caps (pids/timeout/awaited-cleanup), honest labels (default signatures), escaping discipline, self-healing probes.
|
||||
4. **One-line verdicts + evidence per finding**, shipped in the commit message; calibration section ("genuinely solid") accepted as free verification of past fixes.
|
||||
## Report
|
||||
|
||||
## Verification (the non-negotiable)
|
||||
Build/run the real artifact with its real inputs. Observe behavior at the seams:
|
||||
- concurrent calls → no CWD races;
|
||||
- resource ceilings → caps enforced;
|
||||
- cleanup → no fd/pid/dir leaks;
|
||||
- security → injection probes inert (quotes, slashes, metachars);
|
||||
- normal case → output identical to pre-audit.
|
||||
|
||||
## Evidence (2026-10-07 + 2026-10-10/11 sessions)
|
||||
- Shell-engine subagent: 426 LOC, `bun test` 39/39 green. Line-audit found: 5 leaked fds per call (`fd` sink never closed), CWD-race in `git -C .`, `set -e` + `${@/\/g\//}` slash-escaping bug, unbounded 10k-entry cache, 45s timeout on git-status.
|
||||
- Review-triage census over 3 blind adversarial rounds (same repo, ~41 findings): accepted 9, rejected ~32 — rejects dominated by stale reads (protocol rewritten mid-review), inverted filter logic, unreachable-under-law triggers, base64-alphabet "quote injection" (charset has no quotes), SIGPIPE non-leaks, "no memory cap" quoting the `--memory 4g` line above the claim. Real accepts: fork-bomb pids cap, stalled-ssh upload cap, stale-flag self-heal probe, honest census label, freshness gate on pid reuse.
|
||||
- Poison-law incident: sandbox fix shipped while supervisor ran old build → 812 rows graded by the dead protocol; takeover made recovery cheap. The restart step (#6) exists because of this.
|
||||
Per deliverable: correct / changed / removed, with file:line, plus the gate output that proves it (`bats` failure list before and after).
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
---
|
||||
name: authentik-rp-sso-login-diagnosis
|
||||
description: "Diagnose/repair a dead OAuth2 relying-party login against authentik (Grafana-style): authorize 500, missing email claim, Grafana /emails fallback. Live probe harness + ak-shell gotchas."
|
||||
---
|
||||
|
||||
# Authentik OAuth2 relying-party (RP) SSO login diagnosis
|
||||
|
||||
Use when a vhost authenticates through authentik via the OIDC authorization-code
|
||||
flow (Grafana, Seerr OIDC, any "Sign in with <authentik>" button) and the login
|
||||
fails — even though the guard-chain e2e (Anubis + outpost) passes green.
|
||||
|
||||
## Key insight: the edge test lies
|
||||
|
||||
The outpost-guard spec (e.g. `13-sso`) only proves an authenticated session
|
||||
reaches the app's *login page*. It never follows the RP OAuth button. So the RP
|
||||
login can be 100% dead and the suite still passes. **Always drive the real RP
|
||||
round-trip** to get ground truth (harness below).
|
||||
|
||||
## Symptom → cause map (all verified live on 2026-10-20, grafana)
|
||||
|
||||
| Signature | Root cause | Fix |
|
||||
|---|---|---|
|
||||
| Browser hits `…/application/o/authorize/?client_id=…` → "Server Error — Powered by authentik" (500), traceback `AttributeError: 'NoneType' object has no attribute 'slug'` in `flows/planner.py` | provider `authorization_flow` is NULL (raw `ak shell` `create()` leaves it unset) | set `authorization_flow = default-provider-authorization-implicit-consent` |
|
||||
| Grafana log `logger=oauth.generic_oauth level=error msg="Error getting email address" url=<api_url>/emails … 404/403` | the granted scope mapping returns a **bare string**, not a dict → no `email` claim emitted | fix the ScopeMapping expression to `return {"email": user.email, "email_verified": True}` |
|
||||
| authorize 500 on scope-overlap, log `Application requested scopes not configured` for a scope like `user:email` | client requested a scope authentik doesn't define (GitLab-ism) | client must request authentik's real scopes: `openid email profile` |
|
||||
| Grafana `?api=1` on API URL → GET `<…>/userinfo/?api=1/emails` → 403 | Grafana v13 appends `/emails` to a `?api=1` URL | use the standard `…/userinfo/` with **no `?api=1`** |
|
||||
|
||||
Crucial causal note: **Grafana v13 GETs `<api_url>/emails` for the address ONLY
|
||||
when the token carries no `email` claim.** So the `/emails` 404 is a *symptom of
|
||||
the missing claim*, not a Grafana misconfig. Do NOT chase it with `USE_ID_TOKEN`
|
||||
or attribute paths; fix the claim. `USE_ID_TOKEN=true` does NOT stop the
|
||||
`/emails` fetch. Once the claim exists, standard userinfo carries `email` and no
|
||||
`USE_ID_TOKEN` is needed.
|
||||
|
||||
## Live RP-login probe harness (ground truth)
|
||||
|
||||
A `qa-e2e` authentik session storage state exists at
|
||||
`tests/e2e/.auth/qa-e2e.json` after an e2e run; the e2e node_modules live in the
|
||||
main checkout (`tests/e2e/`). Throwaway probe:
|
||||
|
||||
```js
|
||||
// node _p.cjs (run from a checkout with node_modules + .auth)
|
||||
const { chromium } = require('playwright');
|
||||
(async () => {
|
||||
const b = await chromium.launch({ headless: true });
|
||||
const ctx = await b.newContext({ storageState: '.auth/qa-e2e.json',
|
||||
userAgent: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36" });
|
||||
const p = await ctx.newPage();
|
||||
await p.goto('https://<vhost>/', { waitUntil: 'domcontentloaded' });
|
||||
for (let i=0;i<30;i++){ const t=await p.locator('body').innerText().catch(()=>''); if(!t.includes('not a bot')) break; await p.waitForTimeout(3000);}
|
||||
await p.goto('https://<vhost>/login/generic_oauth', { waitUntil: 'domcontentloaded' });
|
||||
await p.waitForTimeout(9000);
|
||||
const r = await ctx.request.get('https://<vhost>/api/user');
|
||||
console.log(p.url(), r.status(), (await r.text()).slice(0,200));
|
||||
await b.close();
|
||||
})();
|
||||
```
|
||||
|
||||
Success = final URL on the app host (not `/login`) AND `/api/user` 200 with an
|
||||
email. Before the fix it is `/login` + 401. Move the throwaway to
|
||||
`~/.AGENTTRASH/…` after (never `rm`).
|
||||
|
||||
## ak-shell census: operational traps (authentik 2026.8)
|
||||
|
||||
- **Run it via `ssh dankclaw bash -s <<'EOF' … EOF`, NEVER direct
|
||||
`ssh host '… "{{…}}"'`.** Direct ssh eats inner quotes locally; `podman inspect
|
||||
--format '{{…}}'` and multi-line `ak shell -c "…"` only survive inside a quoted
|
||||
heredoc. Authentik runs **rootless** (`podman exec authentik-server …`, no sudo).
|
||||
- `OAuth2Provider` has **no `properties` attribute** in 2026.8; scopes are derived
|
||||
from `p.property_mappings`. Accessing `.properties` raises and silently aborts
|
||||
the whole `-c` payload.
|
||||
- `ScopeMapping` is **not importable** from `authentik.core.models`. To read
|
||||
scopes: iterate `p.property_mappings.all()`, then `m.scopemapping.scope_name`
|
||||
(reverse one-to-one; the base `PropertyMapping` row has no `scope_name`).
|
||||
- Introspect before guessing: `[f.name for f in Model._meta.get_fields()]`. Don't
|
||||
trial-and-error attribute names — that wastes turns.
|
||||
- Claims-into-id_token field is `include_claims_in_id_token`.
|
||||
- Provider census that works:
|
||||
```
|
||||
from authentik.providers.oauth2.models import OAuth2Provider
|
||||
for p in OAuth2Provider.objects.all():
|
||||
af=p.authorization_flow
|
||||
print(p.name, af.slug if af else None)
|
||||
# email mapping shape (the real bug):
|
||||
p=OAuth2Provider.objects.filter(name='grafana').first()
|
||||
for m in p.property_mappings.all():
|
||||
print(m.name, getattr(getattr(m,'scopemapping',None),'scope_name',None), repr(getattr(m,'expression','')))
|
||||
```
|
||||
|
||||
## Repair = idempotent script (DB state, not deployable)
|
||||
|
||||
Authentik config lives in its DB, not in any repo file; a blueprint re-import or
|
||||
an `ak shell` re-creation reverts provider fields silently. Ship an idempotent
|
||||
`scripts/authentik-<svc>-sso-fix.sh` that `ssh HOST 'podman exec -i
|
||||
authentik-server ak shell' <<'PYEOF' … PYEOF` and re-applies each fact guarded by
|
||||
"already set?" checks. Match the style of
|
||||
`scripts/authentik-set-login-session-duration.sh`; add a header note "DB state —
|
||||
re-run after a rebuild". Reference it from the service page. Grafana's half
|
||||
(env) lives in `grafana.container` and deploys normally.
|
||||
|
||||
## Repo test-seam gotchas this triggers
|
||||
|
||||
- **literal-secret seam** (`observability_quadlet.bats`, "no literal secret")
|
||||
globs keys `*_SECRET|*_PASSWORD|*_TOKEN|*_API_KEY` and demands a `${VAR}`
|
||||
value. A boolean env ending in `_TOKEN` (`…_USE_ID_TOKEN=true`) is a **false
|
||||
positive**. Prefer not to set such a key (drop `USE_ID_TOKEN`); otherwise the
|
||||
seam must be refined (ask the owner before editing a pre-existing test).
|
||||
- Quadlet `Environment=` **word-splits on spaces** (systemd semantics) — a spaced
|
||||
list must be quoted: `Environment="KEY=a b c"`, else it truncates to `a`. The
|
||||
repo already uses this (`open-webui.container` OAUTH_SCOPES).
|
||||
- **OKF conformance gate** rejects a page frontmatter `timestamp` in the future.
|
||||
The fleet host clock reads UTC (often a day ahead of local "today") — set the
|
||||
timestamp to the repo's today, not the host's UTC date.
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
name: authentik-session-ttl-query
|
||||
description: "Query or change authentik login-session TTL on Dankclaw via ak shell — the string-typed UserLoginStage.session_duration trap, the apply logic, and why SESSION_SAVE_EVERY_REQUEST is unavailable."
|
||||
---
|
||||
|
||||
Change or audit how long an authentik login session lives (Dankclaw `authentik-server`, build 2026.x). The real knob is a DB stage row, not env — and the field format will bite you.
|
||||
|
||||
## Where the TTL actually lives
|
||||
- Run program through raw stdin: `ssh dankclaw 'podman exec -i authentik-server ak shell' <<'EOF' ... EOF 2>&1 | sed -n '/objects imported/,$p'`.
|
||||
- Get models by label, NOT import paths (`from authentik.core.models.setting import ...` fails; `authentik.core.models` is not a package; GlobalSettings is NOT a model in 2026.x):
|
||||
`from django.apps import apps; ULS = apps.get_model("authentik_stages_user_login", "UserLoginStage")`
|
||||
- The password sign-in path is flow `default-authentication-flow` → stage `default-authentication-login` (FlowStageBinding FK is `target`, not `flow`). Other stages `default-source-authentication-login` / `default-source-enrollment-login` govern external-IdP/enrollment logins only.
|
||||
|
||||
## THE FORMAT TRAP (will break login if wrong)
|
||||
`UserLoginStage.session_duration` is **string-typed** in authentik's own idiom: `"days=7"` / `"seconds=2"`.
|
||||
- Assigning a `datetime.timedelta` stores Python's `str(timedelta)` = `'7 days, 0:00:00'`, which authentik's parser `authentik.lib.utils.time.timedelta_from_string` **REJECTS** (`ValueError: not enough values to unpack`). Left like that, login crashes computing the expiry.
|
||||
- Correct write: `s.session_duration = "days=7"; s.save(update_fields=["session_duration"])`
|
||||
- Verify it parses: `from authentik.lib.utils.time import timedelta_from_string as P; P(s2.session_duration)` → `7 days, 0:00:00`. A single `key=value` token only; `"days=7 seconds=0"` also errors.
|
||||
|
||||
## How it applies (stage.py:77-87, verified)
|
||||
`delta = timedelta_from_string(session_duration)`; `if remember: delta += remember_me_offset`; then `set_expiry(0) if delta==0 else set_expiry(delta)`. So any positive duration applies to EVERY login (remember-me only adds on top), and `set_expiry(<positive>)` overrides `SESSION_EXPIRE_AT_BROWSER_CLOSE` → persistent cookie. Existing sessions keep their old expiry; new logins only.
|
||||
|
||||
## SESSION_SAVE_EVERY_REQUEST (sliding window) is NOT available
|
||||
`/authentik/root/settings.py` reads `SESSION_COOKIE_AGE` from `CONFIG.get("sessions.unauthenticated_age","days=1")`, sets `SESSION_EXPIRE_AT_BROWSER_CLOSE=True`, and references **`SESSION_SAVE_EVERY_REQUEST` nowhere** → no `AUTHENTIK_*` env or config.yml key can flip it. Enabling a sliding window means patching the vendor image; instead raise the login-stage duration (absolute, supported).
|
||||
|
||||
## Audit the rest of the expiry surface (live values)
|
||||
- framework: `settings.SESSION_COOKIE_AGE`, `SESSION_SAVE_EVERY_REQUEST`
|
||||
- OAuth2 RPs: `apps.get_model("authentik_providers_oauth2","OAuth2Provider")` → `access_token_validity` / `refresh_token_validity` (this fleet: access 1h [embedded-outpost 24h], refresh 30d)
|
||||
- proxy claim cache: `authentik_providers_proxy.ProxySession.expires` (observed ~1–24h; silent re-auth, not a logout)
|
||||
- tenant: `authentik_tenants.Tenant.default_token_duration` (hand API tokens, not sessions)
|
||||
|
||||
## Notes
|
||||
- These stage/provider rows are DB state, NOT repo-managed → a blueprint re-import or fresh rebuild reverts them; drop a setup script/blueprint in the repo if reproducibility matters.
|
||||
- Don't probe the guarded vhost for the agent API and don't confuse Anubis' 168h challenge cookie with the authentik login session.
|
||||
@@ -0,0 +1,28 @@
|
||||
---
|
||||
name: bash-cwd-drift-multi-worktree
|
||||
description: "Diagnose \"files keep reverting / phantom edits\" in multi-worktree sessions: the persistent bash shell's cwd silently reverted to another checkout, so probes and git commands ran against different files."
|
||||
---
|
||||
|
||||
# Multi-worktree bash cwd drift
|
||||
|
||||
## Symptom (all of these at once, in one session)
|
||||
- A file "keeps reverting" to an older version between edits (esp. test files).
|
||||
- `grep -n` line numbers disagree with the tool/test output (e.g. bats reports `line 637`, grep says the match is at 519).
|
||||
- `wc -l`/`md5sum` stable when sampled back-to-back, but state "flaps" across the session.
|
||||
- `git status` clean in one call, surprising in the next.
|
||||
- Probes against `src/` show code that doesn't exist in the branch you think you're on.
|
||||
|
||||
## Root cause
|
||||
The persistent bash shell resets to the session's ORIGINAL cwd when a call omits `cwd` (e.g. after a shell respawn at a turn boundary). With several git worktrees of the same repo checked out, consecutive calls silently alternate between two full copies of the tree — half the reads land in the old branch's files. Nothing is rewriting anything; two files are being inspected.
|
||||
|
||||
## Decision
|
||||
- In multi-worktree work, pass an **explicit `cwd` on EVERY bash call**. Never rely on persistence.
|
||||
- Use **absolute paths** for `read`/`edit`/`write` (already required) — file tools don't drift; only bash does.
|
||||
|
||||
## Check
|
||||
When state seems to flap: `pwd` in the same call as the probe. If two calls to the same relative path report different content, `git worktree list` and census each checkout separately — you have two files, not a gremlin.
|
||||
|
||||
## Repair pattern (if corruption got committed via the wrong tree)
|
||||
1. `git log --oneline --all -- <file>`; per candidate commit print `wc -l` + counts of a few marker strings to classify clean vs corrupted blobs.
|
||||
2. Restore the good blob with `git show <commit>:<path> > <path>` (no checkout — safe under worktree policies).
|
||||
3. Re-apply the intended additions as surgical edits; verify with targeted tests before the full suite.
|
||||
@@ -0,0 +1,57 @@
|
||||
---
|
||||
name: bats-assertion-trust
|
||||
description: "When writing or reviewing bats shell tests, catch vacuously-passing assertions: grep -vq, sed range boundaries, unanchored section slices, and fixtures too small to trigger pipefail/SIGPIPE bugs."
|
||||
---
|
||||
|
||||
Prove a bats assertion can actually fail before trusting that it passes. These four traps each made a green suite lie in real work.
|
||||
|
||||
## 1. `grep -vq` proves nothing
|
||||
|
||||
`some_list | grep -vq 'x'` succeeds when ANY line lacks `x` — nearly always true. To assert absence, capture the text and test it:
|
||||
|
||||
```bash
|
||||
s="$(report_section)"
|
||||
[[ "$s" != *libfoo* ]] # absence
|
||||
[[ "$s" == *bun* ]] # presence
|
||||
```
|
||||
|
||||
Same trap in reverse: bash `[[ $s == *A*B* ]]` — `*` crosses newlines, so a substring check spanning two lines matches the wrong lines. Slice to one line first, then assert.
|
||||
|
||||
## 2. Section slicing helpers must have explicit boundary semantics
|
||||
|
||||
A helper used to scope assertions to one report section must define, and comment, both ends. The default you want: **start inclusive** (labels carry the count/names), **terminator exclusive** (a `sed -n "/A/,/B/p"` range includes B and leaks the next section's first line):
|
||||
|
||||
```bash
|
||||
# Call sites pass a bare anchored regex; both ends documented.
|
||||
section() { printf '%s\n' "$output" | awk -v s="$1" -v e="$2" '$0 ~ e {on=0} $0 ~ s {on=1} on'; }
|
||||
```
|
||||
|
||||
Two bugs this shape hides:
|
||||
- Helper prepends `^` while call sites already pass `^name` → pattern `/^^name/` → never matches → empty slice.
|
||||
- An **unanchored start address** matches the first line and the range runs to end-of-output, so every absence assertion becomes a vacuous pass. Symptom: tests that "verify X is not in section A" pass while X is printed later in the same output.
|
||||
|
||||
After fixing a helper like this, re-run the whole suite: tests that were passing vacuously suddenly fail, and that is the fix working.
|
||||
|
||||
## 3. Fixtures too small hide pipefail bugs
|
||||
|
||||
`set -euo pipefail` + `sort | head -1`: `head` exits, closes the pipe, `sort` dies of SIGPIPE, the command substitution returns `141`, `set -e` aborts the script. Small fixtures never overflow the 64 KiB pipe buffer, so the suite stays green and the first run against real data fails with no output.
|
||||
|
||||
- Take a minimum in one pass: `awk '{ if (m=="" || $1<m) m=$1 } END { print m }' file`
|
||||
- `awk 'NR==1{print; exit}'` is **not** a fix — exiting early closes the pipe exactly like `head`.
|
||||
- Pin it with a fixture larger than the pipe buffer: `seq 1 20000 | awk '{print "[2026-01-01T10:00:00-0700] [ALPM] installed pkgfiller (1-1)"}' > big.log` (use `seq`, not `yes | head`, or the fixture itself SIGPIPEs).
|
||||
|
||||
## 4. `tr` swallows the final newline and downstream tools preserve that
|
||||
|
||||
`tr '\n' ' ' < file` emits no trailing newline; `fold`, `sed`, and `cat` faithfully preserve the absence, so the *next* output line gets glued on. Breaks any tool that later slices output by line-start labels.
|
||||
|
||||
```bash
|
||||
{ tr '\n' ' ' < file; echo; } | fold -s -w 88 | sed 's/^/ /'
|
||||
```
|
||||
|
||||
Pin with: assert the following section's label does NOT appear inside the preceding section's slice.
|
||||
|
||||
## Debugging order when a bats assertion fails but the script output looks right
|
||||
|
||||
1. Reproduce the exact command outside bats (`bash -x` against the script under test) — if it works there, the bug is in the *test helper*, not the script.
|
||||
2. Dump the constructed argument: `printf '%s' "$sed_script" | od -An -c` inside the helper. That is how `/^^ repo /` (double anchor) got found.
|
||||
3. Keep stubs faithful to the real interface: if the stub answers `-Sqq` but the script calls `-Sl`, the test measures nothing. Make the stub exit `64` on unhandled args so a new real query fails loudly instead of silently returning empty.
|
||||
@@ -0,0 +1,51 @@
|
||||
---
|
||||
name: bats-output-glob-red-proof-traps
|
||||
description: "Write red bats tests over command output: [[ ]] glob adjacency (quoted * is literal), run-capture vs test stdout in failure dumps, and proving red against pre-fix code before trusting green."
|
||||
---
|
||||
|
||||
Three traps hit while writing red-then-green bats tests for `remote-job.sh` (Refs #60, `dank/command-center` PR #61). Each one cost a detour because it makes a test look like a bug report when it is an assertion bug.
|
||||
|
||||
## 1. In `[[ $s == pat ]]` a quoted `*` is LITERAL — and it eats the wildcard beside it
|
||||
|
||||
```bash
|
||||
s=' x* DONE rc=0 1 lines'
|
||||
[[ "$s" == *"x*"DONE*"rc=0"* ]] # NOMATCH — and it looks like the tool under test broke
|
||||
[[ "$s" == " x*"*"DONE rc=0"* ]] # MATCH — quoted literals, UNQUOTED separators
|
||||
```
|
||||
|
||||
`*"x*"DONE*` means: anything, then the literal substring `x*DONE`, then anything. Quoting only makes the `*` inside the quotes literal; it does **not** keep the next unquoted token from butting up against it. When the subject has whitespace between the two facts, the pattern must carry an **unquoted** `*` between the quoted literals:
|
||||
|
||||
```
|
||||
"literal-a"*"literal-b" # a, then anything, then b
|
||||
```
|
||||
|
||||
Existing suites in the repo write `*"a"*DONE*rc=0*` — unquoted separators throughout — precisely for this reason. Copy that shape; do not quote for "clarity".
|
||||
|
||||
Discriminate before believing the test: run the pattern against BOTH the expected line and the line the buggy code produced (`correct-red` / `NOMATCH` on the second is the pass condition). A pattern that matches the old output was never a regression test.
|
||||
|
||||
## 2. A bats failure dump prints the TEST's stdout, not `$output`
|
||||
|
||||
```
|
||||
not ok 1 status treats a job name as a name, not a pattern
|
||||
# `[[ "$output" == *"x*"DONE* ]]` failed
|
||||
# started x*
|
||||
```
|
||||
|
||||
`started x*` is the stdout the test body itself printed (from a `start` call several lines earlier), **not** what `run` captured. Reading it as the captured value sends you to debug the tool. To see the real capture, echo it:
|
||||
|
||||
```bash
|
||||
run some-command
|
||||
echo "CAP:[${output}]"
|
||||
```
|
||||
|
||||
in a scratch copy of the test (`/tmp/dbg.bats`, same `setup`/harness) rather than adding diagnostics to the committed test.
|
||||
|
||||
## 3. Reproduce outside bats when green refuses to arrive
|
||||
|
||||
When a test fails but the same command works by hand, replicate the *exact* bats call shape in `/tmp` with the same env exports (`REMOTE_JOBS_DIR` etc.) and the same `run bash -c '…' _ "$A" "$B" "$C"` argument mapping. If the scratch copy passes and the committed one fails, the difference is the assertion, not the system — go back to trap 1.
|
||||
|
||||
## Red-proof rule
|
||||
|
||||
Run the new test against the pre-fix code and read *why* it failed. Both traps above produced a failure that looked meaningful while proving nothing (one matched the buggy output; one printed the wrong stream). Keep the test only when the failure reason is the defect you set out to pin.
|
||||
|
||||
Related: `bats-assertion-trust` (vacuous assertions), `red-verify-without-stash` (proving red in a shared worktree).
|
||||
@@ -0,0 +1,47 @@
|
||||
---
|
||||
name: browser-screenshot-disk-and-cookies
|
||||
description: Save omp managed-browser screenshots to real files (tab.screenshot path is silently ignored) and reset the shared Chromium cookie jar between persona captures.
|
||||
---
|
||||
|
||||
# Screenshots to disk with the omp `browser` eval helper
|
||||
|
||||
## Traps (both cost a full re-capture to discover)
|
||||
1. `tab.screenshot({ path })` **ignores `path` silently** — the option is not in
|
||||
its schema (`{selector, fullPage, silent, annotate, format, quality, ifChanged, threshold}`).
|
||||
The image attaches inline to the tool result and **nothing is written to disk**;
|
||||
`ls` shows an empty dir. Inline images may also be dropped from transcript
|
||||
archives later, so "attached to chat" is not "saved".
|
||||
2. The shared managed Chromium keeps **cookies across tabs and turns** (profile
|
||||
persists after `tab.close()`). A second capture script that expects the
|
||||
anonymous view lands signed-in and `waitForSelector("testid/…")` fails fast.
|
||||
|
||||
## Procedure
|
||||
Write files via `tab.run`, which exposes the raw Puppeteer `page` — its
|
||||
`screenshot({path})` does write:
|
||||
|
||||
```js
|
||||
const dir = "/tmp/<session-dir>";
|
||||
const tab = await browser.open({ name: "caps", url });
|
||||
const shot = (file) =>
|
||||
tab.run(async ({ page }, f) => { await page.screenshot({ path: f }); return f; },
|
||||
{ args: [`${dir}/${file}.png`] });
|
||||
|
||||
await tab.clearCookies(); // ALWAYS before an anonymous capture
|
||||
await tab.goto(url);
|
||||
await tab.waitForSelector("testid/first-state");
|
||||
await shot("1-anonymous");
|
||||
// ...drive states with direct helpers: click/fill("testid/<id>", ...) ...
|
||||
await shot("2-next-state");
|
||||
await tab.close();
|
||||
```
|
||||
|
||||
Verify with `ls -la <dir> && file <dir>/*.png` — do not trust silent success.
|
||||
|
||||
## Notes
|
||||
- Selectors accept `testid/<id>` handler; prefer it over CSS on `[data-testid]`.
|
||||
- `tab.run(fn, { args })`: functions receive `{tab, page, browser, wait, assert}`
|
||||
first, then each arg; a bare code string runs in an **isolated runtime, not the
|
||||
page** — `document` is undefined there. Use direct helpers or `tab.evaluate`
|
||||
(page-global expression; top-level `return` invalid) for page access.
|
||||
- Byte-identical PNG sizes between two frames = same view; fine as proof of a
|
||||
round-trip (state A → B → A).
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
name: command-center-bundle-lint-gate
|
||||
description: "Before committing docs in dank/command-center, lint the real omp-docs bundle directly — the bats suite only lints fixtures, so a green suite is not proof."
|
||||
---
|
||||
|
||||
# The bundle lint gate (`dank/command-center`)
|
||||
|
||||
Adding or editing pages under `omp-docs/`? `bats tests/` going green proves
|
||||
nothing about the pages you just wrote.
|
||||
|
||||
## Why this exists
|
||||
|
||||
`tests/kb-lint.bats` exercises `scripts/kb-lint.sh` against `mktemp -d` fixtures
|
||||
(`"conformant bundle passes"`, `"missing frontmatter is reported"`, …). For years
|
||||
nothing in the suite pointed the linter at the shipped tree, so a page with no
|
||||
frontmatter, an index that stopped listing a file, or a broken relative link
|
||||
committed under a green suite. (Two tests appended 2026-10-20 now run both
|
||||
checkers against `omp-docs/`; verify they are still present before assuming the
|
||||
suite covers it — `grep -n omp-docs tests/kb-lint.bats`.)
|
||||
|
||||
The general law: **a lint suite that tests only the linter does not lint your
|
||||
content.** Check which one a suite runs before calling green a gate.
|
||||
|
||||
## Gate a docs commit
|
||||
|
||||
```bash
|
||||
cd <worktree>
|
||||
bats tests/ ; grep -E '^1\.\.' /tmp/… # suite (count, failures)
|
||||
./scripts/kb-lint.sh omp-docs # want: kb-lint: 0 violation(s)
|
||||
python3 scripts/kb-coverage.py omp-docs # want: index-coverage: 0 issue(s)
|
||||
```
|
||||
|
||||
`kb-lint.sh <kb-dir>` (default `omp-docs`) = OKF v0.1 conformance **plus link
|
||||
integrity**, so it is the check that absolute bundle links like
|
||||
`/tools/gitea/client-access.md` resolve. `kb-coverage.py` = every page listed by
|
||||
its directory `index.md` (repo doctrine: nothing may appear in a directory its
|
||||
`index.md` does not list).
|
||||
|
||||
OKF shape that passes: every concept page carries frontmatter with a `type`
|
||||
(`type: reference` + `title`/`description`/`tags`/`timestamp`/`verified`);
|
||||
`index.md` files carry **no** frontmatter; new pages get an entry in their
|
||||
directory index in the same commit, plus a dated entry in `omp-docs/log.md`
|
||||
(one entry = what changed + the durable lesson, a few lines, never a narrative).
|
||||
|
||||
## Commit-message trap in this harness
|
||||
|
||||
A multi-line message written as a heredoc inside a chained `bash` command
|
||||
(`git commit -q -F - <<'MSG' … && …`) fails with
|
||||
`pi-natives:command: syntax error at end of input`. Write the message to
|
||||
`/tmp/msg-<topic>.txt` with the write tool and `git commit -q -F /tmp/msg-<topic>.txt`.
|
||||
Keep the `&&` chain so a failed gate stops the commit — hooks (secret gate) stay armed,
|
||||
never `--no-verify`.
|
||||
|
||||
## Land and reap
|
||||
|
||||
Merge is form-encoded `do=merge&merge_style=no-ff` and answers `200` with an
|
||||
**empty body**; never read that as ambiguous — verify by landing. Then
|
||||
`git -C ~/.omp merge --ff-only origin/main` (the checkout *is* the deployment;
|
||||
dirty `agent/memories/`, `agent/config.yml`, `APPEND_SYSTEM.md` are machine-tier
|
||||
live state and do not block an ff whose diff misses them), fast-forward the
|
||||
session's `wt/*` scratch branch the same way, then `git worktree remove <build-wt>`
|
||||
and `git branch -d <type>/<issue>-<slug>` (`-d` succeeding is itself proof of
|
||||
merge).
|
||||
|
||||
Stray files whose names start with `-` (e.g. `-c`, `-es` — vim takes a
|
||||
`--startuptime` flag name as the output filename) show as untracked dirt:
|
||||
`mv ./-c .AGENTTRASH/<date>-<meaning>.log`, never `rm`.
|
||||
@@ -1,49 +1,62 @@
|
||||
---
|
||||
name: dankclaw-pihole-lan-dns-record
|
||||
description: "Add/repair a LAN DNS record for a new *.franksdank.cloud service on Dankclaw's Pi-hole v6 (hosts/custom.list path), plus the public Porkbun A record step and client cache gotchas."
|
||||
description: "Add/repair a LAN DNS record for a new *.franksdank.cloud service on Dankclaw's Pi-hole v6 (hosts/custom.list + repo addn-hosts, plus the public Porkbun A record side) — and triage \"LAN names stopped resolving\" including the FTL boot-time record-load failure."
|
||||
---
|
||||
|
||||
Use when a new `*.franksdank.cloud` service resolves publicly/LAN-needs a local address, on Dankclaw (Pi-hole v6 container, quadlet `pihole.container`).
|
||||
# Dankclaw Pi-hole LAN DNS record
|
||||
|
||||
## Ground truth layout (verified 2026-10-11)
|
||||
## What this fixes
|
||||
|
||||
- Quadlet: `~/.config/containers/systemd/pihole.container`; volumes `/home/dank/Containers/pihole/{etc-pihole,etc-dockers,addn-hosts}`.
|
||||
- **LAN records live in `/home/dank/Containers/pihole/etc-pihole/hosts/custom.list`** — plain hosts format: `192.168.0.143 gitea.franksdank.cloud`. This is the boring, working path.
|
||||
- `pihole.toml` `[[dns.records]]` array is EMPTY on this host — do not expect v6 UI-style records there.
|
||||
- `gravity.db` exists but has NO `domain` table (v6 layout) — do not insert local records into SQLite.
|
||||
A new `*.franksdank.cloud` service resolves publicly (via Porkbun → `franksdank.cloud` A record → Caddy on Dankclaw `192.168.0.143`) but **does not resolve on the LAN**. Pi-hole is the LAN DNS server (`dankclaw_default` bridge, published port 53) and serves local records only from files it imports.
|
||||
|
||||
## Procedure (30 s, proven)
|
||||
## Live truth
|
||||
|
||||
```bash
|
||||
ssh dankclaw 'echo "192.168.0.143 <name>.franksdank.cloud" | sudo tee -a /home/dank/Containers/pihole/etc-pihole/hosts/custom.list >/dev/null \
|
||||
&& podman exec pihole pihole restartdns'
|
||||
ssh dankclaw 'getent hosts <name>.franksdank.cloud' # must resolve
|
||||
dig +short <name>.franksdank.cloud @192.168.0.143 # from any LAN box
|
||||
- **The repo owns the records.** `dankclaw/pihole/addn-hosts` (hosts-style lines: `192.168.0.143 <name>`) is copied by `deploy-remote.sh` to `/home/dank/Containers/pihole/addn-hosts` and bind-mounted **read-only** into the container at `/etc/pihole/hosts/addn-hosts` (`Volume=/home/dank/Containers/pihole/addn-hosts:/etc/pihole/hosts/addn-hosts:ro` in `dankclaw/pihole/quadlet/pihole.container`). Add the name there, in the repo, then deploy — that is the durable path. `tests/pihole.bats` validates every line (IPv4, LAN range, name suffix, Caddyfile cross-check).
|
||||
- `/home/dank/Containers/pihole/etc-pihole/hosts/custom.list` also exists on the host (older mechanism, mode `640` owned by the container uid, root reads it with `sudo`). It is **not** the repo's owner file — do not hand-edit it as the fix.
|
||||
- `pihole.toml` carries a `dns.hosts = [ "192.168.0.143 …", … ]` array too (FTL's own in-config records). Both sources exist; if LAN names fail, neither is being applied.
|
||||
- Pi-hole runs as a rootless quadlet named `pihole` — query it with `podman exec pihole …`, never `sudo podman` (per-user storage law).
|
||||
- `pihole.franksdank.cloud` resolves through Pi-hole itself (loop/edge), so a vhost record pointing at `192.168.0.143` is correct for every Caddy-fronted service.
|
||||
|
||||
## Procedure (repo-first)
|
||||
|
||||
1. Append the line to `dankclaw/pihole/addn-hosts`:
|
||||
```
|
||||
192.168.0.143 <name>.franksdank.cloud
|
||||
```
|
||||
One record per vhost; keep it in sync with the Caddyfile (the bats seam cross-checks the two).
|
||||
2. `./tests/run.sh` (the `pihole` seam validates the file), commit, push.
|
||||
3. Deploy so the copy step runs: `./scripts/deploy.sh dankclaw --service pihole`.
|
||||
4. Verify from the box: `getent hosts <name>.franksdank.cloud` → `192.168.0.143`.
|
||||
5. Public edge record (optional; skip when every subscriber is LAN-only): Porkbun `POST /api/v3/json` with `apikey`/`secretapikey`/`command=add`/`type=A`/`hostname=<name>`/`data=76.49.118.136` (see `dankclaw-porkbun-edge`).
|
||||
|
||||
Client-side proof (avoid the resolver trap):
|
||||
|
||||
```
|
||||
sudo resolvectl flush-caches # systemd-resolved otherwise keeps the negative cache
|
||||
curl -sv --max-time 5 https://<name>.franksdank.cloud 2>&1 | grep 'Connected to'
|
||||
```
|
||||
|
||||
Hosts files are read-only-readable only via sudo (dir owned by uid 100999). `sudo -n` works on dankclaw.
|
||||
Expected: `Connected to <name>.franksdank.cloud (192.168.0.143)` — LAN route, not the public IP.
|
||||
|
||||
## API maze — DON'T re-explore
|
||||
## Triage: every LAN name broke at once (2026-10-19 law)
|
||||
|
||||
- FTL API base is `/api` at the web port (8083) **with `Host: pi.hole` header**; `/admin/api/*` → bad_request hint; bare 127.0.0.1 without the vhost → 404 `not_found`.
|
||||
- Login: `POST /api/login {"password": <WEBPASSWORD from /home/dank/Containers/pihole/env>}` → session in `X-Cors` header/cookie.
|
||||
- `POST /api/apps/domains/custom` → **404 on this build** (FTL version predates it); `/api/customdns/custom` also absent. There is no container sqlite3. Use custom.list.
|
||||
Signature: **all** `*.franksdank.cloud` return NXDOMAIN / Playwright `ERR_NAME_NOT_RESOLVED` while **external names still resolve**. The external-OK/local-fail split is the tell: FTL is answering external names from cache and holds **no local records at all** after a container/host restart — the mounts and files are intact and readable inside the container (`ls -l /etc/pihole/hosts/`, `grep -c . …/addn-hosts`), `dns.hosts` in `pihole.toml` is complete. The mechanism is unestablished; do not go hunting for a lost file.
|
||||
|
||||
## Public record (off-LAN reachability)
|
||||
|
||||
Porkbun API, keys in `/home/dank/Containers/caddy/env` (`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`):
|
||||
|
||||
```bash
|
||||
curl -sL -X POST https://api.porkbun.com/api/json3/domain/api/create/franksdank.cloud \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"apikey\":\"$K\",\"secretapi\":\"$S\",\"name\":\"<name>\",\"type\":\"A\",\"content\":\"71.227.227.172\",\"ttl\":\"600\"}"
|
||||
```
|
||||
ssh dankclaw 'getent hosts grafana.franksdank.cloud' # empty => FTL, not the client
|
||||
ssh dankclaw 'podman exec pihole pihole restartdns reload' # fixes it in seconds
|
||||
```
|
||||
|
||||
- `json3` path 302-redirects → **`-L` required**; no-`-s` to see status.
|
||||
- Porkbun edge (openresty) intermittently 503s — transient; retry later, don't burn turns. Public record is OPTIONAL when LAN-only subscribers exist.
|
||||
- Wildcard `*.franksdank.cloud` TLS is already covered: Caddy + Porkbun DNS-01 (caddy-porkbun image), so a new subdomain needs NO new cert work — just the Caddyfile site block + reload.
|
||||
Distinguish before acting: a **single** client failing while others work is FTL rate-limiting that client — `podman exec pihole grep Rate-limiting /var/log/pihole/FTL.log`; a `10.0.2.x` client address is a pasta container's DNS forwarder (SNAT), not a machine. Full causal page: `serverAppConfigs-docs/Infrastructure/Pi-hole.md` (**Records** pitfall + Failure modes table).
|
||||
|
||||
## Client-side cache gotchas
|
||||
## Gotchas
|
||||
|
||||
- Workstation (systemd-resolved): negative cache survives Pi-hole changes; `sudo resolvectl flush-caches` needs a TTY (faillock-sensitive — no `-S` guessing). Non-interactive proof: `curl --resolve <name>.franksdank.cloud:443:192.168.0.143 https://<name>.franksdank.cloud/...`; cache self-expires (SOA negative TTL).
|
||||
- Dankclaw/Dankserver resolve via Pi-hole directly — verify there first.
|
||||
- Do NOT insert local records into SQLite. v6's `gravity.db` `domain` table has NO `type=2` custom-DNS rows in this build.
|
||||
- `pihole restartdns reload` re-imports the hosts files **and** `dns.hosts`; that is the remedy for the boot-load failure above. Plain `pihole restartdns` also restarts the resolver — brief LAN-wide DNS blip.
|
||||
- `dig @<ip> <name>` from the workstation bypasses systemd-resolved — use it to check FTL's answer directly (`+short @192.168.0.143`).
|
||||
|
||||
## Cross-reference
|
||||
|
||||
- Fleet DNS flow: `dankclaw/Caddyfile` (Caddy binds `:443` on the host, serves `*.franksdank.cloud` vhosts)
|
||||
- Pi-hole service page: `serverAppConfigs-docs/Infrastructure/Pi-hole.md`, flow page `Infrastructure/LAN-DNS.md`
|
||||
- Public edge: managed skill `dankclaw-porkbun-edge`
|
||||
|
||||
@@ -1,44 +1,44 @@
|
||||
---
|
||||
name: dankvoice-bats-serialization
|
||||
description: "Run dankvoice bats suites without hangs: serialize builds on shared dist/, DV_HOLD is seconds, stub needs sleep&wait trap before hold, timing tests assert CALLS/at not tail PLAYS."
|
||||
description: "Run dankvoice bats suites without hangs or false failures: serialize dist rebuilds, never edit src mid-run, and rebuild before committing the tracked bundle."
|
||||
---
|
||||
|
||||
# Dankvoice bats: serialization and hang-proofing
|
||||
Use when running the dankvoice `bats tests/` suite on the shared box, when a suite run hangs or times out, or when the "committed dist is byte-identical to a fresh build" pin test fails after a commit.
|
||||
|
||||
## Serialize builds on the shared artifact
|
||||
`tests/behavior.bats` setup runs `bun run build` into the repo's shared `dist/index.js`
|
||||
before EVERY test (`DV_EXT="$REPO/dist/index.js"`). Any second concurrent bats run,
|
||||
`bun run build`, or `bun run typecheck` racing the same repo contends on that file and
|
||||
can stall single tests past 60 s. NEVER run two bats processes (or bats + build) against
|
||||
`~/Projects/dankvoice` at the same time — including harness background jobs. If a suite
|
||||
gets backgrounded, kill it before launching any other repo command; re-run serially.
|
||||
## The two serialization hazards
|
||||
|
||||
## The stub player's units and traps
|
||||
- `DV_HOLD` is in **seconds** (`sleep "${DV_HOLD:-0.1}"`), not ms. `DV_HOLD=1500`
|
||||
means a 25-minute "hung" test. Use `DV_HOLD=2` for multi-second holds.
|
||||
- The stub must sleep as `sleep "${DV_HOLD:-0.1}" &` + `wait $!`, with
|
||||
`trap 'printf "KILLED\n" >> "$DV_STUB_LOG"; exit 143' TERM` installed **before** the
|
||||
sleep: a foreground `sleep` defers the trap until the sleep finishes, so SIGTERM-kill
|
||||
tests (shutdown/interrupt pins) stay red and every kill test eats the full hold.
|
||||
- Keep the `cp *.wav "$DV_COPY"` loop when editing the stub — the byte-equality test
|
||||
depends on it.
|
||||
### 1. `setup()` rebuilds `dist/` from `src/`
|
||||
Every test runs `cd "$REPO" && bun run build` in `setup()`. Consequences:
|
||||
|
||||
## Timing-sensitive assertions
|
||||
The driver prints `PLAYS` from the stub log at scenario end (~800 ms settle), so with a
|
||||
long `DV_HOLD` in-flight playback is UNDERCOUNTED — assert `CALLS`/arrival timestamps,
|
||||
not tail `PLAYS`. The fixture stamps every request with `at=` (ms since driver start,
|
||||
comparable between CALL lines); prefetch overlap = `t2 - t1 < 1500` with `DV_HOLD=2`
|
||||
(sequential chain cannot answer inside 2000 ms).
|
||||
- **Never edit `src/` while a suite is running.** The next test's `setup()` compiles whatever is on disk at that instant, so an in-flight edit gets attributed to the run and produces a failure with no owner. Finish the gate, then write.
|
||||
- **Two suites on the same checkout race on `dist/`.** Serialize: run one, wait for its report, then start the next. Multiple worktrees of the repo each build their own `dist/` and do not collide.
|
||||
- Suite runtime is ~85 s for 55 tests; a backgrounded job with a 600 s deadline is the right shape (`timeout: 600`, `async: true`). Never poll it — `setup()` output lands in the conversation when the job ends.
|
||||
|
||||
## Diagnosing a hung bats test (deterministic recipe)
|
||||
1. `timeout 45 bats -f "<name>" tests/behavior.bats; echo rc=$?` — rc=124 confirms hang.
|
||||
2. Launch the same test as a background job, then probe live:
|
||||
`pgrep -af "driver.ts|bats"` and `ps afx | grep -E "sleep|stub"` — a `sleep <N>` child
|
||||
with N seconds in the stub log names the culprit (usually DV_HOLD misuse).
|
||||
3. Detached `(cmd &)` inside a temp shell is KILLED when the shell exits — use the
|
||||
harness background jobs for anything you must probe.
|
||||
4. Direct driver repro: extract the heredoc with
|
||||
`sed -n '/cat > "$WORK\/driver.ts" <<.TS./,/^TS$/p' tests/behavior.bats | sed '1d;$d' > $W/driver.ts`,
|
||||
build a fresh ext, then run `bun $W/driver.ts <scenario>` with the env the setup block
|
||||
exports. Caveat: hand-made fixture env can yield different segment counts than bats —
|
||||
trust in-bats numbers for assertions.
|
||||
### 2. `dist/` is tracked, and a pin test enforces it
|
||||
One test rebuilds and compares against the committed bundle. Therefore the commit order is:
|
||||
|
||||
```bash
|
||||
bun run build # BEFORE staging
|
||||
git add -A && git commit -m '…'
|
||||
bats tests/ # gate AFTER the commit
|
||||
```
|
||||
|
||||
Committing `src/` changes without rebuilding first fails the pin test (`not ok … byte-identical to a fresh build`) — the tracked bundle is the *previous* state. If that happens on an unpushed branch, fix it by amend rather than a follow-up commit:
|
||||
|
||||
```bash
|
||||
bun run build && git add dist && git commit --amend --no-edit
|
||||
```
|
||||
|
||||
Verify freshness any time the pin is in doubt:
|
||||
|
||||
```bash
|
||||
bun run build >/dev/null 2>&1 && git status --porcelain dist # empty = committed == fresh build
|
||||
```
|
||||
|
||||
## If a backgrounded gate job vanishes
|
||||
A crash can kill the job before its report; `read proc://bg_N` returning "not found" means it is gone, not that it failed. Reconstruct what landed before resuming — `git log --oneline -3`, `git status --porcelain`, then the freshness check above — and rerun the suite; the commit may well have succeeded while the gate never ran.
|
||||
|
||||
## Gotchas
|
||||
- `DV_HOLD` is seconds (`sleep "${DV_HOLD:-0.1}"`), so `DV_HOLD=3` really holds the stub for three seconds.
|
||||
- Isolation tests must not count shared state: any assertion that sweeps `$TMPDIR` or `~/.omp/cache` observes every other live session on the box (each narrating session creates `dankvoice-*` scratch dirs). Point `TMPDIR` at a private directory instead — see the `cutting off mid-utterance leaves no orphaned audio behind` test.
|
||||
- `bun run build` does not typecheck. Run `bunx tsc --noEmit` separately when moving code between modules; the bundler happily emits a bundle that references a deleted identifier.
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
---
|
||||
name: dankvoice-bundle-swap-red-proof
|
||||
description: "Prove a dankvoice behavior test is red against the pre-change bundle by pointing DV_EXT at git show HEAD:dist/index.js, with the driver extracted from the bats heredoc into a pinned scratch directory. Use before claiming a new bats assertion is red-capable."
|
||||
---
|
||||
|
||||
Prove a new `tests/behavior.bats` assertion is **red-capable** — that it fails against the *previous* build, not just passes against the new one — without stashing, branching, or editing `src/`.
|
||||
|
||||
## Why
|
||||
|
||||
`tests/behavior.bats` is a black-box suite: the plugin bundle is loaded through `DV_EXT`, so a *different bundle* is the red side. Stashing the fix would break the `dist` byte-identity pin test and the shared worktree.
|
||||
|
||||
## Working recipe
|
||||
|
||||
```bash
|
||||
cd ~/Projects/dankvoice # or the feature worktree: ~/Projects/wt-dv-*
|
||||
bun run build # current src = the green side
|
||||
git show HEAD:dist/index.js > /tmp/old-dist.js # the red side (pre-change bundle)
|
||||
|
||||
W=/tmp/dv-redproof # PINNED name; see traps below
|
||||
rm -rf "$W" && mkdir -p "$W/bin"
|
||||
cat > "$W/player.sh" <<'SH'
|
||||
#!/usr/bin/env bash
|
||||
printf '%s\n' "$*" >> "$DV_STUB_LOG"
|
||||
for arg in "$@"; do case "$arg" in *.wav) cp "$arg" "$DV_COPY" ;; esac; done
|
||||
sleep "${DV_HOLD:-0.1}"
|
||||
SH
|
||||
chmod +x "$W/player.sh"
|
||||
ln -s "$W/player.sh" "$W/bin/pw-play"
|
||||
printf '#!/usr/bin/env bash\nprintf "116\\talsa_output.usb-KT.0\\taudio/sink\\t*\\n"\n' > "$W/bin/wpctl"
|
||||
chmod +x "$W/bin/wpctl"
|
||||
# The driver lives inside the bats file as a heredoc: extract it verbatim.
|
||||
end=$(grep -n '^TS$' tests/behavior.bats | cut -d: -f1)
|
||||
sed -n "86,$((end - 1))p" tests/behavior.bats > "$W/driver.ts"
|
||||
|
||||
export DV_EXT="$PWD/dist/index.js" DV_STUB="$W/player.sh" DV_STUB_LOG="$W/player.log" \
|
||||
DV_COPY="$W/played.wav" DV_OMPCONFIG=/dev/null DANKVOICE_CONFIG="$W/settings.json" \
|
||||
DV_MODELS="$W/models.yml" DV_WPCTL=x DV_SELECT=""
|
||||
for ext in /tmp/old-dist.js "$PWD/dist/index.js"; do
|
||||
rm -f "$W/player.log" "$DANKVOICE_CONFIG"
|
||||
DV_EXT="$ext" bun "$W/driver.ts" <scenario> 2>&1 | grep -E '^PLAYS|^CALLS|^NOTIFY|^SETTINGS'
|
||||
done
|
||||
```
|
||||
|
||||
The scenario name is the driver's third argv (`speak`, `auto`, `repeat`, `humanizecycle`, `badurl`, …); the bats `-f` test body's env (`DV_PLAYER_BIN`, `DV_SETTINGS_JSON`, `DANKVOICE_VERBOSITY`, …) is passed the same way. `DV_SELECT` holds the menu answers, `|`-separated, consumed in order.
|
||||
|
||||
## Traps, each one cost a turn
|
||||
|
||||
- **The scratch directory must be pinned by name.** `ls -dt /tmp/tmp.*` picks up *other sessions'* temp dirs, and then `player.sh` and `driver.ts` live in different directories — the run silently does the wrong thing.
|
||||
- **The stub must append to `$DV_STUB_LOG`.** The driver counts player invocations from that file; a stub that only `sleep`s reports `PLAYS 0` for *both* bundles and the whole proof reads as "fix does nothing".
|
||||
- **`DV_EXT` is required.** Missing it yields `Cannot find package 'undefined'` from the driver, which looks like a broken driver, not a missing env var.
|
||||
- **`DV_SETTINGS_JSON` supports `__PORT__`** (the driver substitutes the fixture port), so a preseed can name a prefixed route — needed whenever the test is about *which* endpoint got resolved.
|
||||
- **Never run this while a `bats` run is in flight** in the same repo: `setup()` runs `bun run build`, so two runs clobber `dist/` and the pin test.
|
||||
|
||||
## Reading the result
|
||||
|
||||
Red side must show the *defect* (`PLAYS 0`, `644`, a chat `CALL` with the wrong prefix, `Invalid URL`), green side the fix — assert on printed markers only (`PLAYS`, `CALLS`, `CALL <url>`, `NOTIFY`, `SETTINGS`). `CALLS` counts `/audio/speech` calls only; the rewrite call appears as its own `CALL /…/chat/completions … sys=<full system prompt>` line, which is how prompt-level knobs (verbosity, heckle) are observed.
|
||||
|
||||
Prefer this over a synthetic leak: if the check already caught a real leftover/failure once, that event *is* the sensitivity proof — say so in the commit message instead of faking one.
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
name: dankvoice-epithet-pool-audit
|
||||
description: "Grow or audit the dankvoice 1000-name epithet pool: gender classes, individual eviction, typo sweep, safe regeneration into framing.ts."
|
||||
---
|
||||
|
||||
# Dankvoice epithet pool audit
|
||||
|
||||
Use when adding names to, reclassifying, or cleaning `src/framing.ts` `EPITHETS_*` in `dank/dankvoice`. The pool is law: exactly 1000 distinct names, curated at build time, never user-editable, "the commons, never the characters".
|
||||
|
||||
## Classification rules (proven traps in parentheses)
|
||||
- **man**: gendered lexicon (king, lord, priest, uncle...) + tokens ending `man|men|master` — but never `human|inhuman|nonhuman`. Compounds like `quartermaster`, `hangwoman` are single tokens: token-equality checks miss them, use suffix rules.
|
||||
- **woman**: lexicon (queen, sister, daughter...) + tokens ending `woman` + an EXPLICIT `-ess` title list (baroness, countess, prioress, canoness, chartress, heiress, priestess...). Never a bare `-ess` suffix match: `endless`, `harmless`, `fortress` all end in "ess".
|
||||
- **Possessives follow the head noun**: `lieutenant of the king's hull` is a lieutenant → neutral commons; `count's armsman` is an armsman → man. Do not gender by words inside `'s` phrases.
|
||||
- Modern-neutral offices stay commons: hero, governor, wizard. `witch/lass/bride` are female-coded; `warlock` male-coded.
|
||||
- Regex trap for absence checks: `he\b` matches inside `the`; token-split before matching.
|
||||
|
||||
## Individual eviction law
|
||||
Evict: named fictional/historical individuals (duke caladan, uncle toby, pyrrhus, usul, endymion, skynet, moya-born, medusa), named protagonist ships (rocinante), and real authors in universe clothing (strugatsky, niven).
|
||||
Keep: species, orders, houses, orgs/companies (sardaukar, jedi, cyberdyne), places (kessel; `falconer of the caladan eyries` is fine — the person problem was only `duke caladan`), institutions (zekma, karass).
|
||||
Replace each eviction with the generic office beside it, class-stable, and unique against the pool.
|
||||
|
||||
## Invariants tests pin (breaking any of these goes red)
|
||||
- 1000 names, all distinct.
|
||||
- Sentinel literals must survive: `hangman of the holiday` (man), `high priestess of the ear moon` (woman), `comrade operator` (other).
|
||||
- Pool firsts feed pool[0] probe tests — union order is MAN-first / WOMAN-first / OTHER; keep source order within classes so firsts stay: `chairman of the presidium of processors` / `hangwoman of the public convenience` / `comrade operator`.
|
||||
- Class counts (86/15/899 as of 2026-10) may churn but must be reported; tests never pin them.
|
||||
|
||||
## Regeneration
|
||||
Pool lives as three generated arrays between markers `/** Male-coded` and `/** The unions` in `src/framing.ts`. Splice new arrays between those markers (python splice of a generated `const NAME = [ "x", ... ] as const;` block is fine; do not hand-edit 1000 lines). Then `bun run build` (dist pin test byte-compares), `bun run typecheck`, and the full `bats tests/` — the class sweeps are exhaustive `Math.random`-cycled walks (each ~200 s; suite ~17 min; never two bats processes at once).
|
||||
|
||||
## Fleet cutover after merge
|
||||
main checkout ff-only → `omp plugin upgrade omp-dankvoice` → `cmp` `~/.omp/plugins/node_modules/omp-dankvoice/dist/index.js` against merged `dist/index.js`.
|
||||
@@ -0,0 +1,38 @@
|
||||
---
|
||||
name: dankvoice-menu-knob-workflow
|
||||
description: "Add a persisted /say knob to dankvoice and test its menu row through the bats driver (DV_SELECT picks, INPUT stub, dist-pin rebuild gate)."
|
||||
---
|
||||
|
||||
Adding a setting to `dankvoice` (`~/Projects/dankvoice`, omp speech extension) means five touchpoints plus one behavioral test. Skip one and TypeScript or a silent drop catches it; skip the test and the row ships dead.
|
||||
|
||||
## The five touchpoints (order matters)
|
||||
|
||||
1. `src/settings.ts` — field on `Settings` (optional) **and** a type-guarded line in `readSettings`. A field without a guard line is written to disk and never read back; tsc stays clean, so nothing else catches it.
|
||||
2. `src/config.ts` — `Config` field (required, defaulted), resolution line, entry in the returned object, and the knob documented in the file header comment (that header is the README's pointer for "full knob list").
|
||||
- Convention: explicit env wins over settings (`env.X === "1" ? true : env.X === "0" ? false : (settings.x ?? false)`); booleans use `1`/`0`, enumerations use the literal value.
|
||||
3. `src/index.ts` — `persistSettings()` key list (a knob missing there never survives the session) **and** the getter in the `new Speaker({...})` deps object.
|
||||
4. `src/utterance.ts` — getter on `SpeakerSettings` if the Speaker reads it at speak time.
|
||||
5. `src/menu.ts` — the row. Rows carry their own `run`; never dispatch on `choice === options[N]`.
|
||||
|
||||
Gate at the call site, not the prompt, when a mode must not apply to operator-typed text: `Speaker.speak(ctx, text, thinking, auto)` — `auto=true` only from `message_end`/`agent_end`; `/say <text>` passes nothing. Prompt-only gating leaks.
|
||||
|
||||
## Test the row behaviorally, never as a config assertion
|
||||
|
||||
`tests/behavior.bats` drives the built `dist/index.js` against a mock harness. Selects and typed text both come from one pipe-separated queue: `DV_SELECT="row label|sub choice|typed text"`, consumed in call order; a consumed-exhausted `select` returns `options[0]`, so always name the row by its **rendered label** (`"Verbosity: normal (change)"`, `"Intro: none (change)"`).
|
||||
|
||||
- `ui.input(title, placeholder)` in the driver mock logs `INPUT <title> <placeholder>` and takes the next queue entry. If the mock lacks `input`, no typed value was ever testable — add the stub before writing the test.
|
||||
- Scenario pattern that proves a pick works (`case "knobs"`): `say.handler("menu")` → `say.handler("on")` → two `agentEnd(answer(...))` with `settle()` between. Then assert on observable output, not on the settings file:
|
||||
- prompt-shaping knob: `[[ "$output" == *"ONE spoken sentence"* ]]` (directive reached the rewrite request)
|
||||
- framing: `[[ "$output" == *"input=Update, chief!"* ]]`
|
||||
- throttle/framing counting: `[[ "$output" == *"CALLS 1"* ]]`
|
||||
- A guard test whose negative case would pass trivially must also assert its premise, e.g. truncation: assert `CALLS 1` so "no outro" cannot be a green caused by nothing ever being cut.
|
||||
|
||||
Red first, in a batch: write all the new `@test` blocks, run `bats tests/behavior.bats -f "<pattern>"`, confirm the new ones fail, then implement.
|
||||
|
||||
## Gates and gotchas
|
||||
|
||||
- **Two `bats` runs at once corrupt each other**: `setup()` runs `bun run build`, so a concurrent run rebuilds `dist/` mid-suite and the byte-identical pin test fails spuriously. Serialize.
|
||||
- **Edit `src/` → rebuild before commit**: the pin test compares committed `dist/` to a fresh build. Sequence is `bun run build && bunx tsc --noEmit && bats -f <pattern>`, then `git add -A` (dist included). A failure there is normally a stale bundle, not a bad assertion.
|
||||
- `VAR=value some_function args` in bats does reach the child process, so knob tests need no `export`.
|
||||
- `edit` tool: inserting after a line whose content you know but that the current tag never displayed gets refused — the refusal echoes the real line; if it matches intent, re-issue the identical edit instead of re-reading.
|
||||
- Ship: branch `feat/<issue#>-<slug>` off fresh `origin/main`, one commit per knob, PR targets `main`, merge only on Frank's explicit OK (`do=merge`, `merge_style=no-ff`), then `omp plugin upgrade dankvoice` and `cmp` installed `dist/index.js` against the merged one.
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
name: dankvoice-silence-triage
|
||||
description: "Triage silent or late dankvoice auto-speak on the fleet: arming key, /say grammar, endpoint/sink, event granularity, humanizer latency, plus the black-box bats driver."
|
||||
---
|
||||
|
||||
# Triage "dankvoice is silent" on the fleet
|
||||
|
||||
Symptom that recurs: auto-speak only speaks when `/say <text>` is typed
|
||||
explicitly. Work the gates in this order; each has an objective check.
|
||||
|
||||
1. **Armed?** `cat ~/.omp/cache/dankvoice.json | jq .auto` — must be `true`
|
||||
(or `DANKVOICE_AUTO=1`). Arming is read at **startup only** and `/say on`
|
||||
never persists, so a session started before the key was added is silent even
|
||||
though fresh sessions work. `/say start` is *not* a subcommand: unknown
|
||||
arguments are spoken verbatim (it read out the word "start"). `/say on` is
|
||||
the verb; `/say status` must print `auto-speak on`.
|
||||
2. **Endpoint + sink?** `/say status` must show a real endpoint (not
|
||||
`NO ENDPOINT`) and the saved sink must exist:
|
||||
`pw-dump | grep -o 'alsa_output\.[^"]*' | sort -u`.
|
||||
3. **Did it synthesize?** `pgrep -a pw-play` during a turn: one process per
|
||||
segment, `--target <sink>`, wav under `/tmp/dankvoice-*/say-<epochms>.wav`.
|
||||
One temp **directory per utterance chain** — count dirs to count utterances,
|
||||
pids to count segments.
|
||||
4. **Event granularity (the structural cause).** `agent_end` fires once per
|
||||
*prompt* when the loop settles and is skipped when the harness maps a
|
||||
non-terminal settle (`willContinue`) — long runs with background jobs or
|
||||
subagents therefore narrate nothing. Mid-run narration needs `turn_end`
|
||||
(fires per assistant message + tool results). Verify delivery, don't infer
|
||||
it: skill `omp-extension-event-probe`.
|
||||
5. **Latency is not silence.** Narration passes through the humanizer
|
||||
(frank-zealot) before synthesis, so an utterance lands several seconds after
|
||||
its turn; under endpoint load that stretches to tens of seconds.
|
||||
`/say humanize off` = mechanical voice, near-instant. Judge cadence from
|
||||
probe-log timestamps, never from when audio starts.
|
||||
|
||||
## Design rules the plugin now encodes (dank/dankvoice#2→#3)
|
||||
|
||||
- Narrate `turn_end` per turn **plus** `agent_end` for the settled answer;
|
||||
dedupe identical text per run (reset on `agent_start`) so the answer speaks once.
|
||||
- A turn landing while the player is alive is **dropped** — no queue, no
|
||||
interrupting; a tool loop would cut every sentence off after a word.
|
||||
Busy test: `state.child !== undefined && state.child.exitCode === null`.
|
||||
- Tool-only turns (no prose, no narration-flagged thinking) stay silent.
|
||||
- `print`/`json` modes stay silent, so benches never chatter.
|
||||
- Settings writes must **round-trip the file** (`{...onDisk, ...owned keys}`).
|
||||
Rebuilding from owned keys deleted `"auto"` on any menu choice and silently
|
||||
un-armed every later session — the classic cause of "it worked yesterday".
|
||||
|
||||
## Black-box test harness (repo `dank/dankvoice`)
|
||||
|
||||
`tests/behavior.bats` drives the built artifact through an in-process driver:
|
||||
`const say = DV_EXT(ctx, cfg)[0]`, handlers via `const events = new Map();
|
||||
const pi = { on: (n,h)=>events.set(n,h) }`, then `events.get("turn_end")?.({
|
||||
turnIndex:0, message:{role:"assistant",content:[{type:"text",text:"…"}]},
|
||||
toolResults:[]}, ctx)`. Assert on `CALL`/`PLAYS`/`NOTIFY` lines and
|
||||
`SETTINGS <file contents>`; stubs are `DV_PLAYER_CMD` (logs voice+input, honours
|
||||
`DV_HOLD`) and `DV_REWRITE_URL` (404 → falls back to mechanical text). `setup()`
|
||||
runs `bun run build` per test → serialize suites on the shared `dist/`.
|
||||
`tests/packaging.bats` requires `dist/` **committed** (compares to HEAD), so
|
||||
build → commit → the drift pin goes green.
|
||||
@@ -0,0 +1,44 @@
|
||||
---
|
||||
name: docker-pin-latest-census
|
||||
description: "Prove a Docker Hub / lscr.io image pin is the latest version before bumping it: registry-API tag enumeration, the rolling-latest trap, digest pinning, and rollback proof."
|
||||
---
|
||||
|
||||
Use when asked whether a Docker Hub / lshr.io image pin is the latest version before bumping it, or when a manifest claims an image is stale. Registry API is the answer; the `latest` tag is not.
|
||||
|
||||
## The API
|
||||
|
||||
```bash
|
||||
# hub: token then paginate (the list is 100 per page)
|
||||
tok=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" | jq -r .token)
|
||||
for off in 0 100 200 300 400 500; do
|
||||
curl -s -H "Authorization: Bearer $tok" \
|
||||
"https://hub.docker.com/v2/repositories/${repo}/tags?page_size=100&ordering=last_updated&page=${page}" \
|
||||
| jq -r '.results[] | "\(.name)\t\(.last_updated)"'
|
||||
done | sort
|
||||
```
|
||||
|
||||
```bash
|
||||
# lscr.io (linuxserver): tags are semver, `latest` is a rolling alias
|
||||
curl -s https://images-api.github.com/v2/linuxserver/${repo}/tags/list # wrong host for lscr
|
||||
curl -s "https://hub.docker.com/v2/repositories/linuxserver/${repo}/tags?page_size=100" | jq -r '.results[].name'
|
||||
```
|
||||
|
||||
Docker Hub's own tag list (`hub.docker.com/v2/repositories/<repo>/tags`, ordered by `last_updated`) is the practical source. The registry API (`/v2/<repo>/tags/list`) needs the bearer token flow above; both return tags, neither sorts by semver.
|
||||
|
||||
## The `latest` trap
|
||||
|
||||
- `latest` is a **rolling alias**, not a version. "Pin is older than `latest`" says nothing about whether a *versioned* tag exists.
|
||||
- `latest` can lag or be rebuilt without a version bump; comparing a digest pin against it proves nothing.
|
||||
- For linuxserver images: `stable` is the rolling branch and version tags (`2026-09-30`) are what you pin; multi-arch manifest lists mean the same tag has per-arch digests — the pin must be the manifest-list digest (`RepoDigests[0]`) you actually pull, or a re-pull on another arch silently changes the image.
|
||||
|
||||
## Verification recipe (before bumping a pin)
|
||||
|
||||
1. Read the pinned tag and digest from the manifest/quadlet.
|
||||
2. Enumerate tags from the registry API (paginate; `last_updated` is not semver order).
|
||||
3. Compare **versions**, not recency: does a version greater than the pinned one exist?
|
||||
4. If bumping: pull the new tag and record `docker/podman inspect --format '{{index .RepoDigests 0}}'` as the new digest — pin the digest, not the tag.
|
||||
5. Prove the service: it starts, version endpoint answers the new version, no crash-loop, and the rollback is the previous digest.
|
||||
|
||||
## Report
|
||||
|
||||
"pinned: X (digest …), latest available: Y, upstream changelog: …" — or "pin is current: no version tag newer than X exists". Never claim staleness from the `latest` tag alone.
|
||||
@@ -1,35 +1,60 @@
|
||||
---
|
||||
name: docs-dedupe-verify-campaign
|
||||
description: "Run an exhaustive docs dedupe + fact-verification campaign in dank/command-center: scout cluster ledgers, override scouts with evidence, gate every commit, and sweep scaffolding before PR."
|
||||
description: "Run an exhaustive docs dedupe + fact-verification campaign (command-center or any OKF vault, e.g. franksign): scout cluster ledgers, override scouts with evidence, gate every commit, and sweep scaffolding before PR."
|
||||
---
|
||||
|
||||
# Exhaustive docs dedupe + verification campaign (dank/command-center)
|
||||
# Exhaustive docs dedupe + verification campaign (dank/command-center; transplants to any OKF vault — proven on franksign 2026-10-21, #64 2026-10-21)
|
||||
|
||||
## Frame
|
||||
- Read `omp-docs/references/okf-spec.md` + root `omp-docs/index.md` first; inventory with `find omp-docs -name '*.md' | wc -l`.
|
||||
- Run `scripts/kb-lint.sh omp-docs` baseline; index-coverage gate is `scripts/kb-coverage.py omp-docs` (each dir's `index.md` must link every sibling file/subdir).
|
||||
- Branch off fresh `origin/main` in a NEW worktree (`git -C ~/.omp worktree add ~/.omp/wt/<name> -b docs/<issue>-slug origin/main`) — workspace policy blocks `git checkout -b` in session worktrees.
|
||||
- Read `omp-docs/references/okf-spec.md` (or the vault's vendored OKF contract, e.g. `franksign-docs/read-only/okf-v0.2-spec.md`) + root `index.md` first; inventory with `find <vault> -name '*.md' | wc -l`.
|
||||
- Run `scripts/kb-lint.sh omp-docs` baseline; index-coverage gate is `scripts/kb-coverage.py omp-docs` (each dir's `index.md` must link every sibling file/subdir). On franksign the whole mechanical gate is one seam: `bun tests/lib/okf-lint.mjs` (frontmatter, links, index coverage, version-once) — run it before and after.
|
||||
- Branch off fresh `origin/main` in a NEW worktree (`git -C ~/.omp worktree add ~/.omp/wt/<name> -b docs/<issue>-slug origin/main`) — workspace policy blocks `git checkout -b` in session worktrees. In a session worktree, `git checkout -b` is intercepted anyway: commit on the session branch, then named-ref push `HEAD:refs/heads/docs/<n>-slug` (verify `git log origin/main..HEAD` shows ONLY your commit first).
|
||||
- Open the Gitea issue first (multi-moving-parts law); branch name `docs/<issue#>-<slug>`.
|
||||
|
||||
## Scout fleet (overlap detection)
|
||||
- One `task` call, `tasks[]` batch, agent `scout`, read-only; one cluster per task (skills trees, llama trees, pi vs packages, niri, sleep/power, core, gitea, gitea+cross-cutting sweep). Cap 2 concurrent; queue rest.
|
||||
- Ledger format per cluster → `local://dedupe/<cluster>.md`: inventory table (file | lines | verdict KEEP|MERGE|DELETE|VENDOR|UPDATE), overlaps, facts-to-keep, inbound links, staleness.
|
||||
- **Scouts lie.** Verified misses in the 2026-10-19 campaign: fabricated ledger entries ("peer-reviewed BrowseComp" claim that exists in no file), a DELETE verdict for a generated artifact that is checked in by design AND under test (`tools/niri/keybindings.md`, see `scripts/gen-keybinds.sh` + `tests/gen-keybinds.bats`), and a KEEP for a leftover the scout misread its own changelog about. NEVER apply a scout verdict without re-checking the file, the tests, and the changelog yourself.
|
||||
- One `task` call, `tasks[]` batch, agent `scout`, read-only; one cluster per task. **Fan out 2-3 clusters at a time, never seven**: `task.maxConcurrency` is 2 on this fleet, so the excess only queues, and `proc://` reports queued tasks as `running` — identical uptimes with zero deliveries means thrash, not progress (#64: seven scouts, nothing in 17 min; killing the five redundant ones let the two survivors land within minutes).
|
||||
- Do NOT scout a cluster you have already audited inline. Kill the redundant jobs early with `write proc://<id>/kill` — the endpoint is the scarce resource, and your own session shares it.
|
||||
- Ledger format per cluster: inventory rows (file | verdict KEEP|MERGE|DELETE|VENDOR|UPDATE | evidence line/command). **Demand the ledger as the yield TEXT, never a `local://` file reference**: #64 produced one 0-byte ledger plus three files whose names contained the scout's own serialized parameter block (`gitea-caddy-inventory.md` + newline + a parameter tag), recoverable only by `ls`-ing `~/.omp/agent/sessions/<session>/local/<dir>/`.
|
||||
- **Scouts lie.** Verified misses in the 2026-10-19 campaign: fabricated ledger entries ("peer-reviewed BrowseComp" claim that exists in no file), a DELETE verdict for a generated artifact that is checked in by design AND under test (`tools/niri/keybindings.md`, see `scripts/gen-keybinds.sh` + `tests/gen-keybinds.bats`), and a KEEP for a leftover the scout misread its own changelog about. Franksign 2026-10-21: 2 of 3 cluster scouts fabricated — "7 missing component stories" refuted by the pairing-law seam that runs green on every push, invented entry/page counts, DELETE verdicts on live-box census numbers the purpose test KEEPS; a third hit the 30-min runtime cap producing zero output. #64 same day: three pages "missing an H1" that have one on their first heading line; a 215-line `omp-docs/AGENTS.md` that does not exist; a "wikilinks + 700-char wall" defect in `core/settings.md` that a commit from that same morning had already removed; `core/subagent-limits.md` called superseded while `agent/config.yml` carries the cited `maxRuntimeMs` / `softRequestBudget` / `maxRecursionDepth` values verbatim.
|
||||
- **Four pre-checks before a scout row enters the diff**: (1) the file exists; (2) the named heading/marker is actually absent (`grep -c '^# '`, `grep -c '^# Citations'`); (3) `git log --oneline -3 -- <file>` — a same-day commit may already have fixed it; (4) if the claim is "config supersedes the doc", grep the config for the cited keys.
|
||||
- Cluster sizing: for a vault under ~20 pages, scout fleets are net-negative — audit inline yourself (reading 5 pages beats babysitting a scout). Reserve scouts for multi-hundred-page trees.
|
||||
|
||||
## Mechanical sweeps that pay (run on every vault)
|
||||
- Empty-text links: `grep -rnE '(^|[ ,])\]\(' <vault>` — a generated "See Also" pass left five `- ](index.md)` links (#64); rendered, an empty bullet reads as style, never as a broken reference.
|
||||
- Citation convention census BEFORE adding anything: count `grep -rl '^resource:'` against `grep -rl '^# Citations'`. command-center's convention is `resource:` frontmatter (12 vs 3) — "add Citations to six pages" is a convention change, not a fix; only the pages missing `resource:` were real hits.
|
||||
- Long-line walls: `awk 'length>300 {print FILENAME":"NR"("length")"}'` — reflow prose to ~100 columns, wording intact; leave tables (they cannot wrap) and the text of verbatim blockquotes alone.
|
||||
- Dated log headings: three `## YYYY-MM-DD` sections for one day is three logs wearing one date. Collapse to one `##` plus one `###` per campaign, and write the shape into the spec page so nobody has to infer it.
|
||||
|
||||
## Fact verification
|
||||
- Fleet facts: read-only probes over `ssh -o BatchMode=yes dankclaw|dankserver` (podman ps/inspect/top, stat, /etc/os-release, curl localhost APIs, `podman exec <c> <tool> --version`). Live state beats docs even when docs claim recent verification — the 2026-10-18 "audit" still drifted (zealot 27B→9B, build 10754→10777, container census 14→32).
|
||||
- Fleet facts: read-only probes over `ssh -o BatchMode=yes dankclaw|dankserver` (podman ps/inspect/top, stat, /etc/os-release, curl localhost APIs, `podman exec <c> <tool> --version`). Live state beats docs even when docs claim recent verification — the 2026-10-18 "audit" still drifted (zealot 27B→9B, build 10754→10777, container census 14→32; #64 then found `8C/16T` where SMT is off, a "masked" unit that is simply absent, Pi-hole missing from a DNS list, and a weekly-drifting engine count that became a size class with `/config` named as the only authority).
|
||||
- Local box facts too: `pacman -Qs <pkg>` / `command -v` before trusting a page's "installed tool" claim (#64: `wlr-randr` documented as available, never installed).
|
||||
- Unit health is not endpoint health: an `:7770` that answers `/v1/models` while its systemd unit sits inactive `Result=exit-code` means a container escaped its unit. Report the drift to the operator; never record transient state as docs, and never touch a production unit without a per-run OK.
|
||||
- Code-bound facts (franksign-style): every doc claim about routes/columns/shipped status gets grepped against `apps/*/src`, `packages/db/migrations/` — docs narrating features that never shipped ("page ships in the slice" while only the API landed) get truthed with a status pointer to the owning issue.
|
||||
- External claims: web_search/web read yourself (no subagents per Frank's rule). Kill fabrications outright (invented distro `FreeToken`, nonexistent `ornith-llama-cpp-setup` tree, unattributable "44.5% peer-reviewed CUA") — delete/drop with an explicit note, never keep "plausible" numbers.
|
||||
- Policy drift (live ≠ tuned docs): flag in the doc + PR body as an operator decision; NEVER silently re-apply policy.
|
||||
- Deletion proposals from scouts die by default: archived trees carrying an honest archive banner (`pi/**`, banner 2026-10-18) stay, a directory `index.md` cannot be deleted (index-coverage gate), and incident logs are laws, not bookkeeping. Deletion is a pruning session with Frank present.
|
||||
|
||||
## OKF §9 / frontmatter audit (cheap wins, check every vault)
|
||||
- Log entries filed under the wrong ISO-date heading (crossed midnight): cut the entry (`CUT N.=M @reg`), insert the new date heading above the old one (newest-first headings), paste the entry under it.
|
||||
- Missing trust family (`generated` on machine-written pages) — lint may not catch it; grep every frontmatter block.
|
||||
- Laws living only in `log.md`: promote each to its owner page (mechanism law → architecture page owning the thing), leave the log entry as the incident record; the HANDOFF/README boundary then points at the vault instead of restating.
|
||||
- A law restated in two tiers (vault spec page + always-loaded `AGENTS.md`) is not a duplicate to cut: name the co-ownership in the spec page ("change one, change the other") instead of silently dropping one copy.
|
||||
|
||||
## Log compaction (log.md)
|
||||
- Entry law: dated entry = what changed + durable lesson + concept-page pointer; narratives recoverable via `git log --follow`. Record the law at the top of the log. Tag historical-era entries.
|
||||
- One `## YYYY-MM-DD` per day, `###` per campaign. Put the fan-out and ledger-verification lesson in the entry — that is what stops the next campaign from trusting a scout.
|
||||
|
||||
## Gates + landing
|
||||
- Gate every commit: `bash scripts/kb-lint.sh omp-docs && python3 scripts/kb-coverage.py omp-docs && bats tests/`. Pre-commit hook prints `error: command not found: caddy` on Caddyfile-touching commits — warning only, commit still lands.
|
||||
- Gate every commit: `bash scripts/kb-lint.sh omp-docs && python3 scripts/kb-coverage.py omp-docs && bats tests/`. Pre-commit hook prints `error: command not found: caddy` on Caddyfile-touching commits — warning only, commit still lands. On franksign: kill the :3000 demo stack first (a live API hangs the gate ~10 min; law lives in `franksign-docs/architecture/testing.md`), then `bun tests/lib/okf-lint.mjs && tests/run.sh docs`.
|
||||
- Before opening the PR: `git diff --stat origin/main..HEAD | grep -i 'tmp\|scratch\|\.bak'` — scaffolding merged under a `.tmp-` name in the 2026-10-19 campaign and needed a follow-up PR to promote.
|
||||
- PR via API (skill `gitea-api-pr-operations` for token-extraction + merge payload); `xdg-open` the PR the moment it opens; comment the campaign ledger on the issue.
|
||||
- After Frank's merge OK: propagate deployment `git -C ~/.omp fetch && git merge --ff-only origin/main`; reap campaign worktree (`git worktree remove --force`) + delete branch local/remote; close issue with landing comment.
|
||||
- PR via API (skills `gitea-api-pr-operations`, `gitea-merge-form-encoded`): token through a 600-perm `-K` curl config, never `-H` (`/proc/<pid>/cmdline` is world-readable); create/PATCH = JSON WITH Content-Type, merge = form body with NO Content-Type, `do=merge&merge_style=no-ff`, success = HTTP 200 empty body verified by landed state, not by re-reading the PR. `shred -u` the cfg and payload afterwards. `xdg-open` the PR the moment it opens; comment the campaign ledger on the issue — ownership-move table, re-read facts with the command that read them, and rejected findings with their evidence.
|
||||
- After Frank's merge OK: propagate the deployment with `git -C ~/.omp fetch && git merge --ff-only origin/main`, but first prove the FF cannot touch live machine-tier state: `comm -12 <(git -C ~/.omp status --porcelain | grep -v '^??' | awk '{print $2}' | sort) <(git diff --name-only HEAD..origin/main | sort)` must print nothing, and re-check the dirty count after the merge.
|
||||
- Worktree reaping is the live-sync heartbeat's job (`wt/*` older than 7 days) unless Frank says otherwise — no `rm`, no deleting a session branch you did not create.
|
||||
|
||||
## Line-edit op syntax trap (bit three times)
|
||||
## Line-edit op syntax trap (bit five times)
|
||||
- Valid ops: `PUT N.=M:` (body `+text` lines), `CUT N.=M` / `CUT N.=M @reg`, `PUT <N @reg`, `PUT >N:` / `PUT <N:`. There is NO `PUT N+=:` form — it lands in the file as a literal garbage line and shifts the insert to the wrong position. For gap-insert use a lone `PUT >N:` or `PUT <N:` per call; pre-compute line numbers against the file map (ops reference original lines within one call, but a failed op poisons positions).
|
||||
- **PUT with an empty body DELETES the range** (franksign 2026-10-21: a typo-fix `PUT 7-7:` with empty body deleted the whole entry). Bodyless deletes: use `CUT N.=M`.
|
||||
- A `CUT`/`PUT` range outside the file errors cleanly (`out of range (file has N lines)`) — that is the cheapest way to learn a body was shorter than assumed; re-read the tail rather than guessing offsets.
|
||||
- Bare body rows get auto-prefixed with `+` and a warning — harmless, but never rely on it: prefix every body row yourself.
|
||||
- Editing a PUT body that ends adjacent to the next original line can silently consume the closing frontmatter `---` — after any frontmatter-range PUT, re-read lines 1–10 of the result.
|
||||
- Recovery recipe when a literal op line lands: read exact bounds, `CUT N.=N` (junk), `CUT A.=B @block` (misplaced section), then `PUT <M @block` at the true anchor, re-read tail, add spacing blank if a `---` now abuts a list line.
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
---
|
||||
name: dotfiles-adoption-audit
|
||||
description: "Use when deciding which userspace configs on a Dank box belong in the command-center dotfiles tier — separates tracked intent from generated expansion, app state, secrets, and config owned by another repo. Not for deploying one known file."
|
||||
---
|
||||
|
||||
# Dotfiles adoption audit
|
||||
|
||||
Decides, with evidence, which files under `~/.config` (and top-level dotfiles) belong in
|
||||
`dank/command-center`'s `dotfiles/` tier. The repo law being applied: **config in, state
|
||||
out**, and **one fact, one owner**.
|
||||
|
||||
## The four gates (all four must pass)
|
||||
|
||||
1. **Installed?** `command -v <prog>` — config for software absent from the box is a prune
|
||||
candidate, never an adoption target. This eliminates most of the clutter.
|
||||
2. **Authored, or the expansion of something else?** Generated files must stay out even
|
||||
though they look like config.
|
||||
3. **Does another repo or installer already write it?** If a script or package owns the
|
||||
file, tracking it here creates a second owner that loses every argument.
|
||||
4. **Could a secret ever live in it?** If yes: template + quarantine, or leave it out.
|
||||
|
||||
## Gate 2 is the one people get wrong — prove generation from mtime
|
||||
|
||||
A generator walking the app list writes dozens of files in the same sub-second window.
|
||||
Compare timestamps across unrelated programs:
|
||||
|
||||
```bash
|
||||
cd ~/.config && stat -c '%y %n' */themes/noctalia* */noctalia.* */colors/noctalia* 2>/dev/null | sort
|
||||
```
|
||||
|
||||
Identical seconds + microsecond progression across ten different programs = generated.
|
||||
Re-check one file days later: a new mtime with no human edit proves regeneration, so the
|
||||
tracked file must be the *source* (palette/config the generator reads), never its output.
|
||||
The correct pattern is a tracked config that *imports* the generated path.
|
||||
|
||||
## Classify the whole tree per file, not per directory
|
||||
|
||||
The tracked unit is the **file symlink**, so a directory can be "real" while containing only
|
||||
tracked symlinks. Never classify by directory:
|
||||
|
||||
```bash
|
||||
cd ~/.config && for d in */; do d=${d%/}; tr=0; rl=0; rlfiles=""
|
||||
while IFS= read -r f; do
|
||||
case "$(readlink -f "$f" 2>/dev/null)" in
|
||||
*/.omp/dotfiles/*) tr=$((tr+1));;
|
||||
*) rl=$((rl+1)); rlfiles="$rlfiles ${f#./}";;
|
||||
esac
|
||||
done < <(find "$d" -maxdepth 2 \( -type f -o -type l \) 2>/dev/null \
|
||||
| grep -vE 'Cache|cache|Cookies|storage|sessions|Crashpad|Default|thumbnails|extensions')
|
||||
[ $((tr+rl)) -eq 0 ] && continue
|
||||
printf '%-22s tracked=%-3s box-only=%-3s %s\n' "$d" "$tr" "$rl" "$(echo $rlfiles | cut -c1-90)"
|
||||
done
|
||||
```
|
||||
|
||||
Then read the headers of suspicious files — installers often confess ("Installed by
|
||||
`infra/setup-dev.sh` into `~/.config/containers/systemd/`") and unit files name another
|
||||
project's `WorkingDirectory`/`ExecStart`, which means that project owns them.
|
||||
|
||||
## Ownership before adoption
|
||||
|
||||
```bash
|
||||
git -C ~/.omp ls-files dotfiles # packages deploy.sh sees
|
||||
grep -rl "<unit-or-file-name>" ~/Projects/<proj> # does a project repo own it?
|
||||
systemctl --user is-active <unit> # dead config is not load-bearing
|
||||
```
|
||||
|
||||
## Secret-gap check (do this before adopting anything env-shaped)
|
||||
|
||||
The ignore list may cover only `.env`, so `frank-job-hunt.env` / `postgres.env` fall
|
||||
through, and a token-shape secret gate (`sk-`, `ghp_`, PEM, JWT) does not match a
|
||||
low-entropy `POSTGRES_PASSWORD=…`. Test both explicitly:
|
||||
|
||||
```bash
|
||||
cd ~/.omp && git check-ignore -v --no-index 'dotfiles/x/postgres.env' || echo "NOT IGNORED"
|
||||
sed -n 's/^patterns=//p' .githooks/pre-commit
|
||||
```
|
||||
|
||||
Fix = add `*.env` to the quarantine block **and** a password-assignment pattern to the gate,
|
||||
with a red test that stages a fake `POSTGRES_PASSWORD=` file.
|
||||
|
||||
## Land it
|
||||
|
||||
Packages are home-relative trees under `dotfiles/<pkg>/`; per-host overrides go in
|
||||
`zz-$(hostname)` (stowed last). `dconf` is loaded via `dconf load /`, `systemd-user` gets
|
||||
`daemon-reload` + conditional `enable --now`. Sequence: `scripts/deploy.sh --check` (dry run,
|
||||
reports divergents) → adopt → `deploy.sh` → confirm live paths are symlinks into the repo →
|
||||
`bats tests/` (includes the real-bundle `kb-lint`/coverage gate) → branch off `origin/main`,
|
||||
PR to `main`, never merge without Frank's OK.
|
||||
@@ -0,0 +1,76 @@
|
||||
---
|
||||
name: dotfiles-deployment-truth-audit
|
||||
description: "Answer \"is my X config actually tracked and deployed?\" per package: classify every file by readlink, expose tracked-then-ignored debt, and re-derive census numbers from the command printed on the page."
|
||||
---
|
||||
|
||||
Answer to "does this repo still track my <pkg> settings?" must be a measurement, never a remembered count. The tracked unit is the **file**, not the directory: a live directory can be a real directory and still be fully tracked, because every file inside it is a symlink into the repo.
|
||||
|
||||
Run in the deployment checkout (the checkout IS the deployment).
|
||||
|
||||
## 1. Tracked set, per package, both tiers
|
||||
|
||||
```bash
|
||||
cd ~/.omp # or the repo that owns dotfiles/
|
||||
git ls-files dotfiles/<pkg> # base tier
|
||||
git ls-files dotfiles/zz-$(hostname) | grep <pkg> # host override tier — counted separately
|
||||
```
|
||||
|
||||
The `zz-*` tier overrides base files by the same relative path, so a per-package count
|
||||
that ignores it under-reports what a machine actually runs.
|
||||
|
||||
## 2. Live wiring — classify every tracked file (never by directory)
|
||||
|
||||
```bash
|
||||
for f in $(git ls-files dotfiles/<pkg>); do
|
||||
rel=${f#dotfiles/<pkg>/}; live="$HOME/$rel"
|
||||
if [ -L "$live" ]; then printf 'link %-50s -> %s\n' "$rel" "$(readlink "$live")"
|
||||
elif [ -e "$live" ]; then printf 'REAL %-50s (NOT a link)\n' "$rel"
|
||||
else printf 'absent %s\n' "$rel"; fi
|
||||
done
|
||||
```
|
||||
|
||||
- `REAL` = the repo copy is a fiction: live state diverged, deployment never reached it.
|
||||
- `absent` = tracked relic, nothing consumes it (candidate for `git rm --cached`).
|
||||
- Spaces in filenames break the `for` loop (`palettes/Gruber Red.json` splits into two
|
||||
rows). Use `git ls-files -z | tr '\0' '\n'` + `while read -r` when names can hold spaces.
|
||||
- Reverse direction finds untracked-but-load-bearing files:
|
||||
`find ~/.config/<pkg> -maxdepth 2 \( -type f -o -type l \) | while read -r p; do case "$(realpath "$p")" in $PWD/dotfiles/*) ;; *) echo "not from repo: $p";; esac; done`
|
||||
|
||||
## 3. Live state that must stay untracked
|
||||
|
||||
Generated palette output and app state are expansion, not authorship. Distinguish by
|
||||
mtime cluster, not by name: `find ~/.local/state/<app> -type f -printf '%TY-%Tm-%Td %p\n'` —
|
||||
a same-second mass write is a generator; single later re-stamps are re-runs. Pointers a
|
||||
switcher script writes (e.g. `cfg/display.kdl -> desk-1-home.kdl`) are machine state too.
|
||||
|
||||
Then check the hazard this creates: grep the entrypoint for includes of those generated
|
||||
files. A **non-optional** include of a correctly-untracked generated file bricks a fresh
|
||||
box (`include "noctalia.kdl"` with no `optional=true` = niri cannot start until the theme
|
||||
pipeline has run once). Report it as a fresh-box failure, not a style note.
|
||||
|
||||
## 4. Tracked-then-ignored debt (invisible to every normal check)
|
||||
|
||||
```bash
|
||||
git ls-files dotfiles | grep AGENTTRASH # trash committed inside the tree
|
||||
git check-ignore --no-index -v <that path> # does an ignore rule match it?
|
||||
```
|
||||
|
||||
`.gitignore` patterns without an interior slash (`.AGENTTRASH/`) match at **any depth**, so
|
||||
the rule was fine all along — the file simply entered history *before* the rule existed, and
|
||||
an ignore rule never reaches an already-tracked path. Only repair: `git rm --cached <path>`
|
||||
plus moving the working copy to the root `.AGENTTRASH/` (never `rm`). `git check-ignore`
|
||||
without `--no-index` prints nothing for tracked paths and reads as "not ignored" — that
|
||||
false negative is the trap.
|
||||
|
||||
## 5. A census number is only valid with its command
|
||||
|
||||
If a page prints a classifier command, every number on it must be reproducible by running
|
||||
that command. A row that the command does not reproduce was written from memory — wrong, not
|
||||
merely stale. Fix the row, and state the correction in the commit message rather than
|
||||
silently editing the number.
|
||||
|
||||
## Verification before landing
|
||||
|
||||
`bats tests/`, `scripts/kb-lint.sh omp-docs`, `scripts/kb-coverage.py omp-docs`, and
|
||||
`scripts/deploy.sh --check` **from the worktree** — `--check` is read-only; never run the
|
||||
real deploy from a worktree, it repoints live symlinks into the worktree the heartbeat reaps.
|
||||
@@ -0,0 +1,80 @@
|
||||
---
|
||||
name: dotfiles-tier-live-audit
|
||||
description: "Audit the command-center dotfiles tier before running deploy.sh: prove per-file tracked/deployed state, refuse generated theme output, catch stale zz overrides, wrong-box dconf keys, and unresolvable systemd ExecStart specifiers."
|
||||
---
|
||||
|
||||
# Deploy-tier audit (`~/.omp` command center)
|
||||
|
||||
Run **before** any real `scripts/deploy.sh` on a box with a live session, and whenever a
|
||||
symlink, palette, wallpaper or unit behaves oddly. Facts and laws live in
|
||||
`omp-docs/tools/dotfiles-deploy.md` + `omp-docs/inventory/home-config-census.md` — this page
|
||||
is only the probe order. Do not restate the laws in new files; the vault owns them.
|
||||
|
||||
## 0. Classify per file, never per directory
|
||||
|
||||
```bash
|
||||
cd ~/.omp
|
||||
for f in $(git ls-files dotfiles/<pkg>); do t=$(readlink -f "$HOME/${f#dotfiles/<pkg>/}")
|
||||
printf '%-50s %s\n' "${f#dotfiles/<pkg>/}" "$([ -L "$HOME/${f#dotfiles/<pkg>/}" ] && echo LINK || echo real)"
|
||||
done
|
||||
```
|
||||
A live directory can be real and still fully tracked (every file a symlink). `niri` 8/2,
|
||||
`gtk-3.0` 3/0. `zz-$HOST` files count in the `zz` tier's row, not the base package's.
|
||||
|
||||
## 1. The plan, and how to read it
|
||||
|
||||
```bash
|
||||
./scripts/deploy.sh --check # writes nothing; read the last line, not the RELINK lines
|
||||
```
|
||||
`--check` prints `RELINK` for **every** path the `zz` tier carries even when the live link
|
||||
already points at the override (base is walked first, intermediate owner change reported).
|
||||
Only `actions=0` is a real summary; confirm a pending change with `readlink`.
|
||||
|
||||
## 2. Generated theme output must never be tracked or deployed
|
||||
|
||||
```bash
|
||||
git ls-files dotfiles | grep -Ei 'noctalia\.(kdl|toml|json|conf|css)|themes/noctalia'
|
||||
```
|
||||
Hit = generated palette output (same-millisecond mtime cluster proves it). Untrack
|
||||
(`git rm --cached` + `.gitignore` patterns); `deploy.sh` refuses the class (`SKIP … generated
|
||||
theme output`). Tracking one means the next sync quarantines the user's live palette and
|
||||
restores the committed colours.
|
||||
|
||||
## 3. Stale `zz` overrides
|
||||
|
||||
For each path present in both base and `zz-$HOST`: `diff` them, then compare each file's
|
||||
`git log -1 --date=short`. A `zz` copy older than the base change is a time bomb — sync
|
||||
relinks cross-package owners **silently** (`nothing to quarantine`). Delete the `zz` copy in
|
||||
the same commit that base absorbed.
|
||||
|
||||
## 4. dconf: compare key by key, never with `diff`
|
||||
|
||||
`dconf load /` writes only keys present, so a text diff misleads both ways. Parse both sides
|
||||
into `[section]/key → value` and report: value conflicts, keys the dump would newly write,
|
||||
and any value containing `/home/` (foreign-box state). File-chooser `last-folder-path`,
|
||||
`window-state`, `picture-uri*` are state, not preferences — delete them from the dump and
|
||||
clear the live keys if they were already deployed here.
|
||||
|
||||
## 5. Units: state, then resolvability
|
||||
|
||||
```bash
|
||||
for u in <units in dotfiles/systemd-user>; do printf '%s %s %s\n' "$u" \
|
||||
"$(systemctl --user is-enabled $u)" "$(systemctl --user is-active $u)"; done
|
||||
```
|
||||
`is-enabled` exits **1** for `linked` exactly as for `disabled` — read the printed word.
|
||||
Never start a unit that is already installed: `swayidle-lock` sits `enabled`/`inactive` and
|
||||
`--now` arms a screen lock mid-session. Specifiers are systemd's, not the shell's: probe
|
||||
`command -v "${bin//%h/$HOME}"` before believing `ExecStart … absent on this host` — the
|
||||
literal `%h/…` string made this repo's own heartbeat look foreign on every box.
|
||||
|
||||
## 6. Landing it
|
||||
|
||||
`bats tests/` (deploy contract runs `scripts/deploy.sh` in a scratch repo with `DEPLOY_REPO`,
|
||||
`DEPLOY_HOST`, stub `HOME`, stub `systemctl` on `PATH`) + `scripts/kb-lint.sh omp-docs` +
|
||||
`scripts/kb-coverage.py omp-docs`. Branch off `origin/main`, PR, never merge without Frank's
|
||||
OK (`gitea-merge-form-encoded`: form-encoded `do=merge`, `merge_style=no-ff`).
|
||||
|
||||
Then deploy from the **deployment root** `~/.omp` only — never from a worktree (live symlinks
|
||||
would point into a checkout the heartbeat reaps in 7 days). Acceptance: `actions=N
|
||||
quarantined=0`, two consecutive `--check` runs both `actions=0`, no `ENABLE … started` for a
|
||||
unit that was already installed.
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
name: e2e-port-squat-triage
|
||||
description: "When an e2e seam on a fixed port fails weirdly (404s on routes that exist, stale behavior, health probe passes but app seems ancient), audit for leftover dev/debug servers squatting the port before believing any red or green."
|
||||
---
|
||||
|
||||
# E2E fixed-port squat triage
|
||||
|
||||
## Symptom classes (all seen with a stale server squatting the seam's fixed port)
|
||||
- Routes that exist in current `dist`/source return 404 ("Cannot POST /api/...") while unit tests through the same app factory pass.
|
||||
- A seam that is *green* but exercises old code: its setup curls `http://host:PORT/health` and gets a response from **whoever owns the port** — a leftover server, not the one it just spawned (the spawned one dies or idles with EADDRINUSE, sometimes still showing in `pgrep`).
|
||||
- Behavior matches an older build (missing mounts/guards) despite a fresh build step.
|
||||
|
||||
## Triage recipe (before touching code)
|
||||
1. Who owns the port:
|
||||
```bash
|
||||
ss -tlnp 2>/dev/null | grep :PORT
|
||||
```
|
||||
2. List every server process with **start times**:
|
||||
```bash
|
||||
for p in $(pgrep -f "path/to/server-entry"); do ps -o pid,lstart,cmd -p $p --no-headers; done
|
||||
```
|
||||
3. Compare start times against the build artifact timestamps (`ls -la dist/...`). A server older than the dist you just built is a liar — its in-memory modules predate the code under test.
|
||||
4. Kill all leftovers, then re-check: `pgrep -f "server-entry" || echo all-dead`.
|
||||
5. Rerun the seam. Only now is red/red and green/green meaningful.
|
||||
|
||||
## Why it happens
|
||||
- `kill %1` in **non-interactive** bash fails silently (no job control) — use explicit PIDs or `pkill -f <pattern>`.
|
||||
- Seam setups that "wait for health" cannot distinguish their own server from a squatter on the same port; a pre-flight "port must be free" assertion is the permanent fix (assert *before* spawn).
|
||||
|
||||
## Hardening (if you own the seam)
|
||||
Add to setup, before spawning: fail fast if the port is already listening — turns the silent liar into a loud error:
|
||||
```bash
|
||||
ss -tln "sport = :PORT" | grep -q LISTEN && { echo "port PORT busy — kill leftovers first" >&2; return 1; }
|
||||
```
|
||||
@@ -0,0 +1,35 @@
|
||||
---
|
||||
name: fixture-delay-window-race-test
|
||||
description: Pin a race/abort bug in a fixture-server test by delaying only the Nth request so the interruption lands inside the in-flight window; assert via a driver-installed unhandledRejection reporter.
|
||||
---
|
||||
|
||||
# Reproducing a timing-window bug against a fixture server
|
||||
|
||||
Use when a defect only appears in a window — a promise created before it is awaited, a kill/abort landing mid-flight, a prefetch interrupted — and a fixture HTTP server answers instantly, so the window never opens. Earned in `dankvoice` issue #7 (`[Unhandled Rejection] AbortError` from `/say stop` during a pipelined synthesis).
|
||||
|
||||
## Why naive attempts pass vacuously
|
||||
|
||||
1. **Uniform delay on the fixture route** (`DV_LATENCY_MS` for every request) makes *all* requests resolve at the same time, so by the time the interruption fires nothing is in flight. The test goes green against the broken code — worse than no test.
|
||||
2. Asserting on a log line count that also counts interruption noise: `dankvoice`'s driver prints `PLAYS <n>` as the number of **player log lines**, and the stub appends `KILLED` on SIGTERM — so a single playback with one interrupt reports `PLAYS 2`. Count the payload lines instead (`grep -c 'PLAYER .*\.wav$'`).
|
||||
|
||||
## The shape that works
|
||||
|
||||
- **Delay only requests after the first.** The first response gets the pipeline into playback; the second stays in flight, which is the window:
|
||||
```ts
|
||||
const spoken = received.filter((c) => c.url?.endsWith("/audio/speech")).length; // already pushed
|
||||
const delay = spoken > 1 ? Number(process.env.DV_SYNTH_MS ?? 0) : 0;
|
||||
if (delay > 0) setTimeout(answer, delay); else answer();
|
||||
```
|
||||
- **Make the interruption land inside it.** Player holds long (`DV_HOLD=2`), interruption at a fixed offset (`DV_STOP_AT=400`) with the delayed response far later (`DV_SYNTH_MS=1200`): margins wide enough that the abort is neither before the request nor after the response.
|
||||
- **Turn unhandled rejections into observable output** in the test driver instead of letting the runtime crash on them — that is also exactly what the real harness does to the user:
|
||||
```ts
|
||||
process.on("unhandledRejection", (reason) => {
|
||||
console.log(`UNHANDLED ${reason instanceof Error ? `${reason.name}: ${reason.message}` : String(reason)}`);
|
||||
});
|
||||
```
|
||||
Then assert `[[ "$output" != *"UNHANDLED"* ]]` plus the concrete behavior (segments actually played, player killed).
|
||||
- **Leak checks measure a delta, not an absolute:** count `dankvoice-*` temp dirs before and after the run and compare; an absolute `== 0` fails on another test's leftovers.
|
||||
|
||||
## Order
|
||||
|
||||
Write the scenario first and watch it fail **on the rejection assertion specifically** — if it fails on some other assertion (a count, a kill line), the window is wrong, not the bug. Then fix, then confirm red→green on that one assertion before running the full suite.
|
||||
@@ -0,0 +1,39 @@
|
||||
---
|
||||
name: fleet-host-process-service-wiring
|
||||
description: "Add a host-process (non-quadlet) systemd USER service/exporter to serverAppConfigs — every file that must change together, with the signal-relay precedent and the Prometheus gateway/LAN reach rules."
|
||||
---
|
||||
|
||||
A host-process service (runs `python3`/a binary on the host, NOT a container) is the right shape when the thing must read host-only state (DRM/cgroup sysfs, `shells to nvidia-smi`, signal-cli) that a container cannot reach without passthrough. Precedent to mirror: `scripts/signal-relay.{py,service}` + `install_signal_relay` in `deploy-remote.sh`. A new one landed 2026-10-20: `scripts/gpu-exporter.{py,service}` + `deploy_gpu_exporter`.
|
||||
|
||||
## The unit is a USER unit, not a quadlet
|
||||
- Hand-written `scripts/<svc>.service`: `[Service]` `ExecStart=/usr/bin/python3 /home/dank/serverAppConfigs/scripts/<svc>.py` (runs straight from the git checkout — a pull updates it, no copy step), `Restart=always`, `[Install] WantedBy=default.target`.
|
||||
- It CAN be `systemctl --user enable`-ed (generated quadlet units cannot on this host's podman generator). With `Linger=yes` for `dank`, `[Install]`+`enable` is the whole boot story.
|
||||
|
||||
## Every file that MUST change together (a lone change on one side fails silently)
|
||||
1. `scripts/<svc>.service` — the unit. Base binds **loopback only**; the LAN address is added by a deploy **drop-in**, so the checked-in unit is never a wildcard.
|
||||
2. `scripts/deploy-remote.sh`:
|
||||
- a `deploy_<svc>()` function: `cp` the unit to `~/.config/systemd/user/`, write a `<svc>.service.d/bind.conf` drop-in with the per-host `Environment=..._BINDS=127.0.0.1,<lan>` (map LAN from `$TARGET`: dankclaw=192.168.0.143, dankserver=192.168.0.235 — do NOT hardcode one host if it runs on both), `daemon-reload`, `systemctl --user enable`+`restart`, then a **loop health gate** (`for i in $(seq 1 15); curl -fsS -m5 http://127.0.0.1:<port>/metrics && return; sleep 1`) — a one-shot curl races the bind and once failed a healthy deploy.
|
||||
- a dispatcher branch `<svc>) deploy_<svc> ;;` in the central `case "$service"` (near the observability branch).
|
||||
- an **allowlist entry on EVERY target** it runs on: `dankclaw:<svc>` and/or `dankserver:<svc>` (the allowlist `case` around the top of the file).
|
||||
3. `scripts/deploy-plan.py`: add `"<svc>"` to `SERVICE_ORDER`, to `DANKCLAW_SERVICES`/`DANKSERVER_SERVICES` (whichever it runs on), and a `scripts/<svc>.*` path-map in the `parts[0] == "scripts"` block: `if name in {"<svc>.py","<svc>.service"}: _add(services, "<svc>")`. `target_services` then filters per target.
|
||||
4. `scripts/test-changed.sh`: a `case` routing `scripts/<svc>.py|scripts/<svc>.service` → its seam + `deployment_scoping.bats` (place before the generic fallbacks). `tests/run.sh` runs all `*.bats` regardless, so the mandatory gate is covered even if this route is missing — but add it for focused runs.
|
||||
5. `shared/prometheus/prometheus.yml`: one scrape job per host, using the **reach rules** (below).
|
||||
6. Dashboard JSON + the owning page under `serverAppConfigs-docs/Infrastructure/Observability.md` (one owner per fact).
|
||||
|
||||
## Prometheus reach rules (the whole cross-file contract)
|
||||
Prometheus is a **bridge container**. On this netavark host a bridge container CANNOT dial the host's own LAN IP at a host-bind (only at a *published* port). Therefore:
|
||||
- **Same host (Dankclaw):** scrape `host.containers.internal:<port>` (netavark delivers it to the host LAN bind). Scraping `192.168.0.143:<port>` reads DOWN forever even though the endpoint is healthy.
|
||||
- **Different host (Dankserver):** plain LAN `192.168.0.235:<port>` reaches it.
|
||||
- Never target a `*.franksdank.cloud` vhost — Prometheus cannot solve an Anubis challenge; it is red by construction.
|
||||
- The exporter must bind BOTH loopback and the host LAN (like `node-exporter`/`cadvisor`), so `host.containers.internal` and the LAN both land.
|
||||
|
||||
## Firewall
|
||||
Leave it LAN-open like node-exporter/cadvisor (unauthenticated metrics on the LAN is the repo's exporter posture). Only signal-relay gets an iptables allowlist (loopback+host+`10.89.0.0/16`+DROP) because it is an unauthenticated message-SENDING webhook — do not copy that lock onto a read-only exporter.
|
||||
|
||||
## Seam test (tests/<svc>.bats) pattern
|
||||
- Black-box the binary with injectable inputs (fixture dirs / stub commands), assert only consumer-visible output; assert **absent means absent** (anchor guards to a sample line `^metric{`, not the always-emitted `# TYPE` header — grepping the bare family name matches the header and false-fails).
|
||||
- Cross-file contracts (grep the other files): the unit's `ExecStart`/bind posture, the deploy function + allowlist + dispatch + health gate, the planner selection (run `deploy-plan.py --target T --paths scripts/<svc>.py`, assert output == `<svc>` per target), the prometheus targets, the dashboard `json.load` + that its `targets[].expr` reference the emitted families.
|
||||
|
||||
## Gotchas hit while building gpu-exporter
|
||||
- Values are float-normalised (`397.40`→`397.4`); a test pinning the literal fails — assert the normalised value. Derive expected byte counts from real units (MiB×1048576, sysfs µW/1e6, temp1_input/1000), don't guess the multiply.
|
||||
- `xdg-open <PR-URL>` after opening the PR (repo law) — don't make Frank hunt in Gitea.
|
||||
@@ -1,62 +1,29 @@
|
||||
---
|
||||
name: fleet-service-runtime-census
|
||||
description: "Answer \"what is service running right now?\" for a dank-fleet service: live endpoint probe plus host units, image ids, live-vs-repo env, and drop-ins."
|
||||
description: "Answer \"what is service running right now?\" for a dank-fleet service: unit state, container image/digest, endpoint contract, loaded env — machine over repo, in that order."
|
||||
---
|
||||
|
||||
# Fleet service runtime census
|
||||
Answer "what is service running right now?" for a dank-fleet service without trusting the repo. Repo state is intent; the machine is truth, and they differ by however long since the last deploy.
|
||||
|
||||
Answer the question "what is `<service>` running currently?" with evidence, not from docs. Docs say what *should* run; the host says what *does*. Run all five probes, then report one table.
|
||||
## Order of operations (cheap → expensive)
|
||||
|
||||
## 1. Probe the published port (works from any LAN box, no ssh)
|
||||
1. **Unit state on the host** (rootless quadlet ⇒ `--user`; some units are root):
|
||||
`ssh -o BatchMode=yes host 'systemctl --user show S -p ActiveState -p SubState -p ExecMainStartTimestamp -p NRestarts'`
|
||||
Boot-loop tell: `NRestarts` climbing with a short `ExecMainStartTimestamp`.
|
||||
2. **The container's own view**: `podman ps --filter name=S --format '{{.Image}} {{.Status}} {{.Names}}'`, then `podman inspect` for `Image` (tag **and** `RepoDigests`) and `Env`.
|
||||
3. **The endpoint contract, from the service itself** — this is the only answer that matters to a consumer:
|
||||
`curl -s host:port/v1/models` (served model id), `curl -s host:port/props` (`n_ctx`, `n_ctx_train`, caps), `curl -s -o /dev/null -w '%{http_code}' host:port/health`.
|
||||
4. **Effective flags**: the live process is the truth. `systemctl --user cat S` shows `EnvironmentFile=`; read that file. A repo `scripts/S.env` can be ahead of, or behind, what the running unit loaded — and a unit restart is what re-reads it.
|
||||
5. **Logs last**: `journalctl --user -u S -n 80 --no-pager` for the boot ledger / refusal messages, after you know whether it is even up.
|
||||
|
||||
```bash
|
||||
curl -s -m 10 http://<host-ip>:<port>/health; echo
|
||||
curl -s -m 10 http://<host-ip>:<port>/v1/audio/voices # per-endpoint extras: /v1/models, /metrics
|
||||
```
|
||||
## Facts that bite
|
||||
|
||||
The extra route matters more than `/health` for identity claims:
|
||||
- llama.cpp and radiance **echo the requested model name** — a 200 proves nothing; read `/v1/models`.
|
||||
- halogen 404s `/completion`, `/ping`, `/props`, `/tokenize`.
|
||||
- The frank-tts façade answers `/health` **without dialing the engine**, so "façade healthy, engine down" is a real state; the voice list in that same response is the honest census.
|
||||
- **A service is up ≠ configured how you think.** llama.cpp's `/v1/models` reports `max_model_len`, not the engine's per-slot ceiling; halogen reports `n_ctx` per slot. Read `/props` before quoting any window number.
|
||||
- **`systemctl is-enabled` answers enablement, not activity.** A timerless service reports `linked`/`static`; `deploy.sh` that prints `ENABLE` for it is lying. Read `systemctl is-enabled --quiet` exit status, not the text.
|
||||
- **`NRestarts: 0` with `ActiveState=activating` is a crash-loop only if the timestamps move**; check `ExecMainStartTimestamp` twice, 5 s apart.
|
||||
- **Read-only is always allowed** from the dev box (`ssh -o BatchMode=yes`); state changes on remote boxes are not, without a per-run OK. Production services (`frank-archon`, `frank-dragoon`, `frank-zealot`) are never stopped/restarted to "see what happens".
|
||||
- **Digest-pinned images**: "when was this image updated?" is answered by `podman inspect` `RepoDigests` + the registry API, not by the unit's start time (a restart re-uses the local digest).
|
||||
|
||||
## 2. Units + containers on the host (ssh as the deploy user, no sudo)
|
||||
## Report
|
||||
|
||||
```bash
|
||||
ssh <host> 'systemctl --user status <svc>.service <engine>.service --no-pager \
|
||||
| grep -E "^(●| Active| Loaded)"; \
|
||||
podman images --format "{{.Repository}}:{{.Tag}} {{.ID}} {{.Created}}"; \
|
||||
podman ps -a --format "{{.Names}}\t{{.Status}}\t{{.Image}}"'
|
||||
```
|
||||
|
||||
- `active (running) since …` + `Up N hours (healthy)` is the answer; `activating (auto-restart)` with a ~50 ms exit is a **preflight refusal**, not an engine fault.
|
||||
- **NEVER prefix these podman calls with `sudo`.** Containers live in the deploy user's rootless storage; `sudo podman inspect frank-tts-engine` returns `no such object` while the plain form succeeds. Treating that as "container missing" is a false finding.
|
||||
|
||||
## 3. Drop-ins — the override that no health gate sees
|
||||
|
||||
```bash
|
||||
ssh <host> 'systemctl --user show <svc>.service -p DropInPaths; \
|
||||
systemctl --user show <engine>.service -p DropInPaths'
|
||||
```
|
||||
|
||||
`DropInPaths=` must be empty. A leftover bench drop-in (`Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env`) makes the next deploy boot a different model at a different shape while `/health` still answers 200 and the e2e gate passes.
|
||||
|
||||
## 4. Live env vs repo env — the only truth test
|
||||
|
||||
```bash
|
||||
# repo side (local checkout)
|
||||
grep -vE '^\s*#|^\s*$' scripts/<svc>.env
|
||||
# host side
|
||||
ssh <host> 'sudo cat /home/dank/serverAppConfigs/scripts/<svc>.env' | grep -vE '^\s*#|^\s*$'
|
||||
```
|
||||
|
||||
Diff them. Per repo law the env file beside the serve script **is** the authority; a doc page that disagrees with it is the bug, and host-side drift means an uncommitted edit. Also check `git log --oneline -3 -- scripts/<svc>.env` to confirm the host isn't ahead of git.
|
||||
|
||||
## 5. Report
|
||||
|
||||
Table of: unit state + since, image repo:tag@short-ID + age, backend/alias actually served (from step 1, not from the name), drop-ins, and whether live env == repo env. State explicitly when a fact came from docs rather than from a probe.
|
||||
|
||||
## Traps that produce false findings
|
||||
|
||||
- Digest-pinned images: the local image ID's *creation date* answers "is this stale" better than the tag; `latest` + `Pull=never` is a locally built image, not a registry pull.
|
||||
- `ssh` user context changes what podman sees (root vs rootless) — see step 2.
|
||||
- Two ports on one host are two providers in `models.yml`; a port mismatch is a rename or drift, not a config typo.
|
||||
One line per layer with the evidence: unit (`active (running)`, since, restarts), container (image + digest), endpoint (served id, `n_ctx`, health code), loaded env (file path + the knobs that matter), and explicitly which of those came from the machine versus the repo — that gap is the answer to most "why is it behaving like that" questions.
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
---
|
||||
name: fleet-status-daily-report
|
||||
description: "Answer \"status update / what happened yesterday\" for the dank fleet: git-window record on origin/main plus a live per-host census (endpoints, units, RAM, DNS), with the reboot-attribution check that stops false incident causes."
|
||||
---
|
||||
|
||||
# Fleet status report (Dankclaw + Dankserver)
|
||||
|
||||
Use for "what's the state of the fleet", "what happened yesterday", or a post-incident sweep. Read-only.
|
||||
For one service's live state, prefer `fleet-service-runtime-census`; this is the fleet-wide sweep.
|
||||
|
||||
## 1. The day's record (repo is the ledger)
|
||||
|
||||
```bash
|
||||
git fetch origin --quiet
|
||||
git log --no-merges --since="YYYY-MM-DD 00:00" --until="YYYY-MM-DD+1 00:00" \
|
||||
--date=format:'%H:%M' --pretty='%ad %h %s' origin/main
|
||||
git log --merges --since=... --pretty='%ad %h %s' --date=format:'%H:%M' origin/main # PR numbers
|
||||
```
|
||||
Merge lines carry the PR number — that is the review trail; the non-merge log is what actually changed.
|
||||
Group commits into streams (endpoint names lead the subject) rather than listing 20 subjects.
|
||||
|
||||
## 2. Live census (parallel: one call per host + one endpoint sweep)
|
||||
|
||||
Endpoint sweep from any LAN box — `/health` is honest on all four (archon binds only after full load):
|
||||
|
||||
```bash
|
||||
for u in 192.168.0.143:7770 192.168.0.143:7771 192.168.0.235:7770 192.168.0.143:7772; do
|
||||
printf "%s %s\n" "$u" "$(curl -s -m 12 -w ' [%{http_code}]' http://$u/health | head -c 160)"; done
|
||||
curl -s http://192.168.0.143:7770/v1/models # dragoon lists EVERY engine it serves, incl. NPU-lane models
|
||||
```
|
||||
|
||||
Per host: `uptime -s; uptime -p`; `systemctl show <llm units> -p ActiveEnterTimestamp -p NRestarts -p DropInPaths`
|
||||
(`sudo` for the root LLM units, `--user` for the quadlets); `list-units --failed`; `free -h`;
|
||||
`journalctl -k --since "<evening>" -p err`; the host's own engine log tail.
|
||||
|
||||
## 3. Attribution rules (where reports go wrong)
|
||||
|
||||
- **Never call a host reboot from a container's start timestamp.** Check `uptime -s` first; if uptime spans the
|
||||
event, the trigger was a **container restart** — find it with
|
||||
`journalctl --user -u <name>.service --since … --until …` + `podman inspect <c> --format '{{.State.StartedAt}}'`.
|
||||
(This exact mistake mislabelled a 2026-10-19 Pi-hole incident as boot-caused; the law's remedy belongs after
|
||||
*any* restart, which is a different action item.)
|
||||
- `podman stats` is a 28-container wall — sort/limit it; RSS of the LLM tenants is what matters.
|
||||
- This podman's inspect template has **no `.Config.PodName`, no `.Config.GroupAdd`, and `.HostConfig.Devices`
|
||||
can read `[]` even when a quadlet `AddDevice=` worked**. Prove GPU use from the engine's own log
|
||||
(`ggml_vulkan: Found 1 Vulkan devices: … Radeon 8060S (RADV GFX1151)`), not from inspect.
|
||||
- Kernel lines: **count** occurrences (`journalctl -k --since '7 days' | grep -c "NETDEV WATCHDOG"`) before
|
||||
calling a one-off a trend.
|
||||
- Failed units: `systemctl show <u> -p ExecMainExitTimestamp -p ExecMainStatus --value` separates
|
||||
"broken since boot" from "broke yesterday" — and a nightly failure (e.g. logrotate) is a live risk, not noise.
|
||||
|
||||
## 4. Report shape
|
||||
|
||||
Stream table (what landed) → live-state table (what is true now) → **corrections to my own earlier record** →
|
||||
open items that need Frank's decision, each with the evidence line and the one action. Flag anything that would
|
||||
touch host config outside the repo as a decision, never a silent fix.
|
||||
@@ -1,35 +1,37 @@
|
||||
---
|
||||
name: frank-job-hunt-maintenance
|
||||
description: "Maintain frank-job-hunt agents: live-census gate hardening loop, registry locator verification, and the workstation/Dankserver/Dankclaw deploy-sync ritual."
|
||||
description: "Maintaining frank-job-hunt: the census gate's fail-closed rules, ATS locator token derivation, and the two-host deploy-sync verification."
|
||||
---
|
||||
|
||||
# frank-job-hunt maintenance
|
||||
Use when maintaining `frank-job-hunt`: the live-census gate, the registry locator, or Dankserver/Dankclaw deploy-sync. The repo is built and deployed; agents live on Dankserver, ntfy on Dankclaw.
|
||||
|
||||
Repo `~/Projects/frank-job-hunt` (Gitea `dank/frank-job-hunt` on Dankclaw 192.168.0.143:3000). Production: Dankserver user timers `frank-job-sweep.timer` 07:30, `frank-job-digest.timer` 08:15 (`Linger=yes`). ntfy relay: Dankclaw quadlet `ntfy.service`, route `ntfy.franksdank.cloud` (no anubis), LAN record in `hosts/custom.list` (Pi-hole v6 file at `/home/dank/Containers/pihole/etc-pihole/hosts/custom.list`; v6 API paths are hostile — edit the file, then `podman exec pihole pihole restartdns`). Secrets: `~/.config/frank-job-hunt.env` (600, workstation + Dankserver).
|
||||
## The two rules that caused every bug here
|
||||
|
||||
## Law (validated 2026-10-11)
|
||||
- The fleet judge (`frank-zealot`, dankclaw:7771) is a **ranker, never a gate** — it inflates anything reaching it (non-technical roles scored 88-91). All discrimination lives in deterministic gates (`agents/lib/gates.ts`); zealot only ranks survivors.
|
||||
- Thinking-model fault: verdict lands in `reasoning_content` with `content:""` → judge sends `chat_template_kwargs:{enable_thinking:false}` (0.7s clean JSON).
|
||||
- **Family beats topic**: role family (recruiter/sales/finance/PM/designer/people-ops/revops/…) gates `gate:scope` unconditionally — no infra-token override ("DevOps Recruiter" is recruiting). Compound technical titles keep the `INFRA_TITLE` override via `NEG_TITLE` branch.
|
||||
- **gate:region**: remote is US-only (visa reality). "Remote - Bangalore" = onsite abroad → gate; `US` regex pass wins over `NONUS` match ("U.S. & Canada" passes).
|
||||
1. **A URL is data.** The census gate and the registry locator treat portal URLs as opaque strings; nothing may template a URL from a company name. `jobs.dankclaw.example` is a fixture host (RFC 2606) — never derive a real domain from a company name, and never click a link the pipeline constructed.
|
||||
2. **A stale row is worse than a missing row.** Every gate must fail closed: an unverifiable posting is reported as unverifiable, not kept as live. Deleting is allowed; asserting is not.
|
||||
|
||||
## Gate hardening loop (live-census, TDD)
|
||||
1. `bun agents/sweep.ts --dry` (full lane ≈ 30–90 s; 852 postings ≈ 62 pass / 790 filed).
|
||||
2. Classify leaks in the top-10: family gap / region gap / scoring tail. Accept long tail (Director/Founder edge cases) — digest + human curation is the design.
|
||||
3. Token fix in `gates.ts` regexes (`NEG_FAMILY` = no override; `METRO`/`US`/`NONUS` for arrangement). NEVER dedupe-check by memory — read file first; the file has bitten line-offset edits repeatedly.
|
||||
4. Pin every leak with a named regression test in `agents/test/gates.test.ts` before/with the fix.
|
||||
5. `bun test agents/test/ && bun x tsc --noEmit` — hooks re-run these at commit.
|
||||
6. Ship: `git commit` (pre-commit = tests + tsc + docs-link + secret scan) → `git push` → `ssh dankserver 'cd ~/Projects/frank-job-hunt && git pull'` (timers pull from master; Dankserver drift = tomorrow's run is stale).
|
||||
## Census gate
|
||||
|
||||
## Registry verification (monthly, `agents/verify-registry.ts`)
|
||||
Scouts' `ats`/`ats_identifier` are guesses (14/84 were wrong). Ritual:
|
||||
1. `bun agents/verify-registry.ts` (dry, ~70 s serial) → REPAIR census.
|
||||
2. `--write` applies; repairs with live jobs preferred over empty boards (0-job boards 404 later).
|
||||
3. Unresolved = honest manual lane (vanity Workdays, self-built ATS incl. majors, neogov) — NOT a code bug; per-entry error census in sweep output is the same signal.
|
||||
4. Update `targets/README.md` census lines + `docs/log.md`; commit/push/pull (same ship step).
|
||||
`scripts/census-gate.sh` decides keep / stale / unverifiable per row.
|
||||
|
||||
## Pitfalls
|
||||
- `--ats=<x>` smoke filters the census line — "N API-sweepable" reflects the filter, not the registry; don't diagnose a phantom bug.
|
||||
- Workday vanity careers URLs (`careers.uw.edu`) are NOT CXS-API-resolvable; tenant-only identifiers are (site path required).
|
||||
- Root `README.md` vs `targets/README.md` name collision in edit anchors: use full path headers.
|
||||
- Zealot endpoint probe: `curl http://192.168.0.143:7771/v1/models`.
|
||||
- Live proof = HTTP GET on the posting URL against the ATS API or the page itself; `404`/`410` ⇒ stale, non-200 with no body ⇒ unverifiable. Never infer from age alone.
|
||||
- The gate runs against the *live* registry, so a row that passes here and then fails in production means the registry drifted — re-run before blaming the gate.
|
||||
- Test seams: fixture ATS server + `…_BASE_URL` env. Tests assert the row's fate (kept / dropped / flagged), never internal state.
|
||||
|
||||
## Registry locator
|
||||
|
||||
Resolves which ATS a company uses (Greenhouse `boards-api.greenhouse.io/v1/boards/<token>/jobs`, Lever `api.lever.co/v0/postings/<token>`, Ashby `api.ashbyhq.com/posting-api/job-board/<token>`, Workable, SmartRecruiters).
|
||||
|
||||
- Token ≠ company name: it comes from the discovered posting URL, so a locator change must keep the derivation (URL → token), not substitute a slug.
|
||||
- Unknown ATS ⇒ unverifiable, not "no jobs". "Zero jobs" is a strong claim; only a successful API answer with an empty list may say it.
|
||||
- Rate limits: 429 ⇒ back off and mark unverifiable for this pass, never stale.
|
||||
|
||||
## Deploy-sync (Dankserver agents, Dankclaw ntfy)
|
||||
|
||||
- Two hosts, two roles: inference/agents on Dankserver, notification sink on Dankclaw. A deploy that only touches one leaves the pair inconsistent — the agent can post to a sink that no longer accepts its token, silently.
|
||||
- Deploy is `scripts/deploy.sh` from the correct checkout (dev side, clean tree); it pushes the branch first, so a stale local `main` makes the push fail — `git merge --ff-only origin/main` before deploying after any externally merged PR.
|
||||
- Verify after deploy on the *served* surface: the agent answers a probe, and a test ntfy message arrives on the phone. "The unit is active" is not the proof.
|
||||
|
||||
## Tests
|
||||
|
||||
Black-box, fixtures over network: fake ATS servers on localhost, fake ntfy sink, temp registry file. Assert output rows and exit status. Never assert that an env var is set or that a helper was called.
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
---
|
||||
name: frank-tts-repoint-voice-consumers
|
||||
description: "Rotate/rename a frank-tts voice across all consumers (repo DEFAULT_VOICE, Open Notebook speaker profiles over REST, docs) after packs change."
|
||||
---
|
||||
|
||||
## When
|
||||
frank-tts voice packs on `/home/dank/voices` (Dankclaw) changed — packs added, removed, or renamed — and old voice names (e.g. `narrator1/2/3`) no longer resolve. Consumers still sending a dead name now 400 with `unknown voice 'x'; available voices: ...`. This is a maintenance sweep, NOT a bug; confirm with the operator whether the change is intentional before "fixing".
|
||||
|
||||
## Facts that make this non-obvious
|
||||
- frank-tts (:7772) resolves a voice as a **directory**: `<name>/reference.wav` must exist. `model` field is ignored; the voice comes solely from `voice`/`voice_id`.
|
||||
- Live census: `curl -s http://127.0.0.1:7772/v1/audio/voices` (and `/health` returns the same list). This is the truth; the vault docs may be stale.
|
||||
- Two consumers carry voice names, in **different stores**:
|
||||
- **repo**: `scripts/frank-tts.env` `DEFAULT_VOICE=` — answers a voice-less request (the omp harness's normal shape). Deploy via git.
|
||||
- **app-side**: Open Notebook speaker profiles (SurrealDB) and OpenWebUI `audio.tts` voice — NOT repo files.
|
||||
|
||||
## Step 1 — repo: DEFAULT_VOICE
|
||||
Edit `scripts/frank-tts.env` `DEFAULT_VOICE=<new real voice>`. Trunk-based: this worktree can't `git checkout -b` (shared-workspace policy); commit on HEAD and `git push origin HEAD:refs/heads/<branch>`, open PR vs `main`. No bats test pins the DEFAULT_VOICE value (the tts bats use `narrator1` as a self-contained fixture that builds its own `reference.wav` — mechanism test, leave it).
|
||||
|
||||
## Step 2 — Open Notebook speaker profiles (REST, host-side, takes effect immediately)
|
||||
Voice ID is **free text passed to the TTS request untouched** (no enum/whitelist). The active fleet profile is `dank-duo` (episode profile `dank-daily` → speaker_config). PW is the host `OPEN_NOTEBOOK_PASSWORD`.
|
||||
|
||||
```bash
|
||||
ssh dankclaw bash -s <<'REMOTE'
|
||||
PW=$(grep -m1 '^OPEN_NOTEBOOK_PASSWORD=' /home/dank/Containers/open-notebook/env | cut -d= -f2- | tr -d '"')
|
||||
cfg=$(mktemp); chmod 600 "$cfg"; printf 'header = "Authorization: Bearer %s"\n' "$PW" > "$cfg"
|
||||
base=http://127.0.0.1:5056/api/speaker-profiles
|
||||
# census: name -> speaker/voice_id
|
||||
curl -s -K "$cfg" "$base" | jq -c '.[]|{name,speakers:[.speakers[]|{name,voice_id}]}'
|
||||
# repoint by NAME (GET-by-name returns full record incl id; PUT whole transformed object to /{id})
|
||||
curl -s -K "$cfg" "$base/dank-duo" > /tmp/s.json
|
||||
id=$(jq -r .id /tmp/s.json)
|
||||
jq '(.speakers[].voice_id) |= ({"OLD1":"NEW1","OLD2":"NEW2"}[.] // .)' /tmp/s.json > /tmp/s2.json
|
||||
curl -s -K "$cfg" -X PUT -H 'Content-Type: application/json' -d @/tmp/s2.json "$base/$id" -w ' HTTP %{http_code}\n'
|
||||
curl -s -K "$cfg" "$base/dank-duo" | jq -c '[.speakers[]|{name,voice_id}]' # verify
|
||||
rm -f "$cfg" /tmp/s.json /tmp/s2.json
|
||||
REMOTE
|
||||
```
|
||||
- A duo has only 2 speakers — an extra voice is available-but-unassigned, not missing.
|
||||
- Upstream's seeded profiles (`business_panel`, `solo_expert`, `tech_experts`) use OpenAI names + `voice_model:null`; they aren't wired to frank-tts — leave them.
|
||||
|
||||
## Step 3 — OpenWebUI (check before touching)
|
||||
`curl -s http://127.0.0.1:<webui-port>/api/config | jq '.audio.tts'` (port from `podman ps`). If `engine`/`openai.*` are null, the active path is `browser-kokoro` (synthesizes in-browser) and the stored voice is **dormant, not broken** — flag, don't edit a live UI unasked.
|
||||
|
||||
## Step 4 — docs (point of use, one owner)
|
||||
Update the census + speaker-mapping lines in `serverAppConfigs-docs/Dankclaw/Frank TTS.md` and `Open Notebook.md` to the live packs. Leave DATED measurement logs (e.g. a `2026-10-19` `/health` probe) as history. Do NOT chase `scripts/voice-packs.env` / `Voice Packs.md` narrator builder defs — ask the operator whether those packs are being rebuilt or retired.
|
||||
|
||||
## Verify (behavior, not config text)
|
||||
Synthesize each new voice and expect 200 + distinct bytes:
|
||||
```bash
|
||||
curl -s -o /dev/null -w '%{http_code} %{size_download}B\n' -X POST http://127.0.0.1:7772/v1/audio/speech \
|
||||
-H 'Content-Type: application/json' -d '{"model":"voxcpm2","input":"test","voice":"<name>"}'
|
||||
```
|
||||
`tests/run.sh` EXIT=0 before commit. `DEFAULT_VOICE` only lands after merge + `./scripts/deploy.sh dankclaw --service frank-tts`; run that from the dev `main` checkout (`/home/blank/Projects/serverAppConfig`), NOT the agent worktree (deploy.sh refuses a non-`main` branch and pushes `main` forward — it must run on the dev box, not the target).
|
||||
@@ -0,0 +1,25 @@
|
||||
---
|
||||
name: frank-tts-voice-locate
|
||||
description: "Answer fleet-voice questions (where voices are stored, add a voice) by jumping straight to the serverAppConfig vault pages and the live 7772 roster probe, instead of re-searching Gitea for a frank-tts repo that does not exist."
|
||||
---
|
||||
|
||||
# Frank-TTS voice facts: skip the search, go to the owners
|
||||
|
||||
Questions like "how do I add a voice?", "where are voices stored?", "why is narratorN missing?" are already answered elsewhere. The frank-tts code does NOT live in a Gitea repo named frank-tts (repo search finds nothing; it is `dank/serverAppConfigs`), and the engine repo is upstream qwentts.cpp pinned at `5916bf7c`.
|
||||
|
||||
## Resolution order (one grep, no repo search)
|
||||
|
||||
1. **Vault pages (canonical, all of it lives here)** in the local checkout `~/Projects/serverAppConfig`:
|
||||
- `serverAppConfigs-docs/Dankclaw/Voice Packs.md` — pack anatomy, the builder, the 10-second source invariant, "Adding a fourth voice" steps, engine-bump arithmetic.
|
||||
- `serverAppConfigs-docs/Dankclaw/Frank TTS Engine.md` — registry law (memory-only; warm hook), routes, Vulkan device law, troubleshooting table.
|
||||
- `serverAppConfigs-docs/Dankclaw/Frank TTS.md` — the 7772 facade contract.
|
||||
- Code/seam: `dankclaw/voiceservice/`, `scripts/build-voice-pack.sh`, `scripts/voice-packs.env`, `tests/voice_pack_unit.bats`.
|
||||
2. **Live roster** (authoritative over any doc census): `curl -s http://192.168.0.143:7772/v1/audio/voices` (bare JSON array; `/health` also echoes `voices`). A missing name = its dir lacks `reference.wav` or the name is not lowercase — check `/home/dank/voices/<name>/` via `ssh dankclaw`.
|
||||
|
||||
## Facts that cost turns to re-derive
|
||||
|
||||
- Voice storage: `/home/dank/voices/<name>/` on Dankclaw (host state, never git; RO mount at `/voices`). Files: `reference.wav` (24 kHz mono, 18–25 s, existence gate), `reference.spk` (8192 B), `reference.rvq` (5456–6864 B), `manifest.json`.
|
||||
- Roster = facade directory scan; engine registry is in-memory, filled by `podman exec frank-tts-engine /app/warm-voices.sh` (boot `ExecStartPost`; re-register replaces, restart costs 6–8 s/voice).
|
||||
- Clients (omp-dankvoice, OpenWebUI) store NO voices — they GET `/v1/audio/voices`; only the chosen name persists client-side (`~/.omp/cache/dankvoice.json` `voice`).
|
||||
- Crop law: past first 10 s (hard invariant), 20–25 s preferred, floor 19.84 s for `.rvq`; sources pinned by sha256 in `scripts/voice-packs.env`; sample media live on workstations (`/home/blank/audiosample*`) and must be copied to `/home/dank/voice-sources` before a build.
|
||||
- Engine `GET /v1/audio/voices` returns `{}` at the pinned commit — always ask the 7772 facade, never 8880, for the list.
|
||||
@@ -0,0 +1,47 @@
|
||||
---
|
||||
name: frank-tts-voice-pack-audit
|
||||
description: Census live frank-tts voice packs against scripts/voice-packs.env to catch name-drift (renamed/retired packs) and untracked CLI-built packs; use after any voice add/rename when reconciling the builder recipe with what is actually on disk.
|
||||
---
|
||||
|
||||
# frank-tts voice-pack audit (builder ↔ live census)
|
||||
|
||||
Read-only census answering: *does `scripts/voice-packs.env` still describe the packs that actually exist, and is every live pack reproducible from the repo?* Run it after anyone adds or renames a voice. The runtime being healthy (all voices synthesize) does NOT mean the builder is honest — the gap is reproducibility, not serving.
|
||||
|
||||
## Why this exists
|
||||
Packs are host state under `/home/dank/voices` (never in git). A voice can be:
|
||||
- **built by the env** (a `VOICE_PACK_<NAME>_*` block exists), or
|
||||
- **built by CLI override** (`build-voice-pack.sh <name> <dir> --source --offset --duration --sha256` with NO env block) — then its only provenance is `/home/dank/voices/<name>/manifest.json`, and `--all` silently skips it.
|
||||
|
||||
Renaming a pack directory (e.g. `narrator1` → `wilwheaton`) preserves the crop bytes and its source sha256, so the env recipe stays *correct but keyed under a dead name*. `build-voice-pack.sh --all` then rebuilds the dead names and skips the live ones.
|
||||
|
||||
## Procedure (on Dankclaw)
|
||||
|
||||
1. **Live set + which builder made each** — the manifest `kind` field tells them apart:
|
||||
- `kind: "voice-pack"` → built by `scripts/build-voice-pack.sh` (env or CLI override).
|
||||
- `kind: "voice-reference-pack"` → the older trial `pipeline.sh`.
|
||||
```
|
||||
for d in /home/dank/voices/*/; do n=$(basename "$d"); [ -f "$d/reference.wav" ] \
|
||||
&& printf '%-16s %8sB %s\n' "$n" "$(stat -c%s "$d/reference.wav")" "$(stat -c %y "$d/reference.wav" | cut -d. -f1)"; done
|
||||
```
|
||||
2. **Per-voice provenance** — `manifest.json` is the sole record for CLI-built packs. Read `name/voice`, `kind`, `source.path`, `source.sha256`, `crop.offset_seconds`, `crop.duration_seconds`. Use `python3 -c 'import json,...'`, NOT a `jq '{a,b,c}'` object — the brace form gets brace-expanded by the remote zsh and errors.
|
||||
3. **Builder tracked list**:
|
||||
```
|
||||
grep -E '^VOICE_PACKS=|^VOICE_PACK_[A-Z0-9_]+_(SOURCE|OFFSET|DURATION|SHA256)=' /home/dank/serverAppConfigs/scripts/voice-packs.env
|
||||
```
|
||||
Diff the names in `VOICE_PACKS` against the live dirs. Three findings to report:
|
||||
- env keys that have **no live dir** (renamed/retired),
|
||||
- live dirs with **no env block** (CLI-built, unreproducible from repo),
|
||||
- matching source **sha256** between a dead env key and a live dir = that live pack is the renamed crop (safe to re-key the env, don't recompute).
|
||||
4. **Distinguish CLI-built from uncommitted env drift** — check the deployed env matches the repo:
|
||||
```
|
||||
cd /home/dank/serverAppConfigs && git diff --stat HEAD -- scripts/voice-packs.env # empty = no drift
|
||||
```
|
||||
Empty diff + a `kind: "voice-pack"` manifest with no env block ⇒ built via CLI overrides (correct, just untracked), NOT a rogue remote edit.
|
||||
|
||||
## Repoint the consumers (separate concern)
|
||||
If voices were renamed/retired, the runtime consumers still point at dead names — see the `frank-tts-repoint-voice-consumers` skill. Consumers that name a voice: `scripts/frank-tts.env` `DEFAULT_VOICE`, Open Notebook `dank-duo` speaker profile (`voice_id`, app-side REST), OpenWebUI `audio.tts.voice` (config DB). A voice-less request to frank-tts 400s naming `DEFAULT_VOICE`, so a dead default is a live break, not just cosmetic.
|
||||
|
||||
## Gotchas
|
||||
- **A published port is reachable at `host.containers.internal` too.** OpenWebUI's docs say use the LAN IP because that name "doesn't resolve on the bridge" — true only for *non-published* ports. frank-tts 7772 IS published, so `host.containers.internal:7772` answers 200 from the open-webui container. Verify before calling a URL the cause; the real break is usually the dead voice name.
|
||||
- **Source custody is split**: `audiosample*.mp3/m4b` live on the workstation (`/home/blank`), newer sources on Dankclaw (`/home/dank/voice-sources`). `--all` is only reproducible once each env SOURCE path points at where the file actually is.
|
||||
- Do NOT edit `voice-packs.env` to add CLI-built voices without confirming each source path + offset + sha against its manifest, then run `./tests/run.sh` (the pack-list invariant in `voice_pack_unit.bats` walks `VOICE_PACKS`).
|
||||
@@ -0,0 +1,50 @@
|
||||
---
|
||||
name: franksign-demo-inspection-stack
|
||||
description: "Boot franksign's prod-shaped stack, seed all assignment lifecycle states through the real API, and open a headed inspection tour (platform + storybook) without squatting the e2e port."
|
||||
---
|
||||
|
||||
Visual-inspection recipe for the franksign repo (Dank boxes). Proven 2026-10-20; use when asked to "show/open the platform" or tour a new slice.
|
||||
|
||||
## Boot (do NOT squat :3100 — the e2e seam curls whoever owns that port)
|
||||
```bash
|
||||
node --env-file=.env apps/api/dist/cli.js # binds :3000 by default; use named service
|
||||
cd apps/web/storybook-static && python3 -m http.server 6007 # gate seam 12 already built it
|
||||
curl -s http://127.0.0.1:3000/health # {"status":"ok","db":"up"}
|
||||
```
|
||||
|
||||
## Seed lifecycle states through the real API (cookie jar, NOT bearer)
|
||||
better-auth sign-up returns a bearer `token`, but the org routes resolve sessions
|
||||
from **cookies only** (`Authorization: Bearer …` → get-session null → 401). Use
|
||||
`-c /tmp/jar` at sign-up and `-b /tmp/jar` after; state-changing calls need an
|
||||
`origin:` header. Keep auth headers out of argv via `curl -K <file>`.
|
||||
|
||||
1. `POST /api/auth/sign-up/email` `{"email","password"(>=8 chars),"name"}` → save cookies (`-c`)
|
||||
2. `POST /api/orgs` `{"name"}` → `org.id`
|
||||
3. `POST /api/orgs/:org/templates` `{"name","baseTemplateId":"sample"}` → `template.id`
|
||||
4. `POST /api/orgs/:org/templates/:tpl/assignments` `{"email"}` → `{token, signUrl}` (repeat per demo signer)
|
||||
5. signed state: `POST /api/releases/:token` `{"values":{"name":…,"date":…,"agreement":true},"signatureName":…}` — no auth needed (token seam)
|
||||
6. voided state: `POST /api/orgs/:org/assignments/:assignmentId/void` (cookie session)
|
||||
Remaining pending assignment → three inspectable pages: `/app/sign/:token` renders form / already-signed / cancelled.
|
||||
|
||||
## Byte-identity spot check (reconstruction contract, issue #18)
|
||||
Fetch `GET /api/releases/:token/pdf` twice with `sleep 1.5` **between** fetches
|
||||
and sha256-compare, plus the org door `GET /api/orgs/:org/assignments/:aid/pdf`
|
||||
(cookie jar). Same-second-only comparisons are blind to clock leaks
|
||||
(pdf-lib's `PDFDocument.load` restamps `/ModDate` unless `updateMetadata:false`).
|
||||
To see printed text: inflate the content streams and unescape `\(`, `\)` —
|
||||
pdf-lib writes hex-string `Tj` operands.
|
||||
|
||||
## Inspection browser (managed Chromium via browser eval)
|
||||
- `viewport` must be `{width, height}` — string fails
|
||||
- `tab.waitFor(ms)` does NOT exist (ms is parsed as selector) → `await Bun.sleep(900)`
|
||||
- open with `headed: true, persist: true`; window lands on the desktop — confirm with `niri msg windows | grep -i chrome`
|
||||
|
||||
## Teardown (hard gate hazard)
|
||||
A live demo API on :3000 **hangs the full `tests/run.sh` gate for ~10 minutes**
|
||||
(observed 2026-10-21: >600s kill with the service up; 14/14 in ~15s without
|
||||
it). Always `proc://franksign-demo-api/kill` BEFORE running the gate; verify
|
||||
`pgrep -f apps/api/dist/cli.js` is empty so the next `tests/run.sh` e2e seam
|
||||
owns :3100 honestly.
|
||||
|
||||
## Adjacent Gitea quirk (same fleet)
|
||||
Closing an issue/PR: `PATCH {"state":"closed"}` → 201; `POST` → 405. Merging needs form-encoded lowercase `do=merge` (see gitea-merge-form-encoded). Label assignment: `PATCH /issues/N` with `labels:[id]` silently clears labels — use `POST /issues/N/labels` `{"labels":[id]}`.
|
||||
@@ -0,0 +1,55 @@
|
||||
---
|
||||
name: franksign-worktree-db-cred-trap
|
||||
description: "Diagnose franksign e2e seam 5 / health 503 db:down in a fresh worktree: setup-dev.sh minted a password that mismatches the shared franksign-db container."
|
||||
---
|
||||
|
||||
# franksign fresh-worktree DB credential trap
|
||||
|
||||
## Symptom
|
||||
`tests/run.sh` in a fresh franksign worktree fails seam 5 ("e2e server did not
|
||||
become healthy") while the server log says `franksign-api listening on :3100`.
|
||||
Manual probe shows the server IS up:
|
||||
|
||||
```bash
|
||||
PORT=3100 node --env-file=.env apps/api/dist/cli.js >/tmp/manual-e2e.log 2>&1 &
|
||||
sleep 2; curl -s http://127.0.0.1:3100/health
|
||||
# -> 503 {"status":"degraded","db":"down",...}
|
||||
```
|
||||
|
||||
Not a regression, not a :3100 port collision (check `ss -tlnp | grep 3100` to rule that out).
|
||||
|
||||
## Root cause
|
||||
`infra/setup-dev.sh` mints a fresh `POSTGRES_PASSWORD` per checkout
|
||||
(`if [ ! -f infra/postgres.env ]`), but `franksign-db` + `franksign-pgdata`
|
||||
are shared box state owned by the main checkout
|
||||
(`~/Projects/franksign`). The new worktree's `.env` DATABASE_URL then fails
|
||||
TCP auth against the initialized volume.
|
||||
|
||||
The setup script masks it: its readiness gate is
|
||||
`podman exec franksign-db pg_isready`, which uses the in-container local
|
||||
socket and passes regardless of password — so setup prints "postgres ready"
|
||||
while the checkout is dead on arrival.
|
||||
|
||||
## Fix (git-ignored files, no commit needed)
|
||||
Copy the shared credentials from the owning checkout:
|
||||
|
||||
```bash
|
||||
grep DATABASE_URL /home/blank/Projects/franksign/.env # source of truth
|
||||
# PUT that exact DATABASE_URL line into this worktree's .env
|
||||
# PUT the matching POSTGRES_PASSWORD into this worktree's infra/postgres.env
|
||||
```
|
||||
|
||||
Gotcha: the shared password may contain literal `$$`. Keep it byte-for-byte —
|
||||
systemd `EnvironmentFile` and `node --env-file` both read values literally,
|
||||
so `$$` in the file IS `$$` in the password. No escaping, no de-escaping.
|
||||
|
||||
## Verify
|
||||
```bash
|
||||
PORT=3100 node --env-file=.env apps/api/dist/cli.js & # then
|
||||
curl -fsS http://127.0.0.1:3100/health # -> {"status":"ok","db":"up"}
|
||||
tests/run.sh # 14/14
|
||||
```
|
||||
|
||||
## Permanent fix (proposed, not yet landed)
|
||||
`setup-dev.sh` should reuse `~/.config/containers/systemd/postgres.env`
|
||||
(symlink to the owning checkout) when present instead of minting new creds.
|
||||
@@ -0,0 +1,93 @@
|
||||
---
|
||||
name: gguf-lora-adapter-fit-census
|
||||
description: "Read-only census before applying a GGUF LoRA/adapter to a llama.cpp (or fork) endpoint: Hub metadata, declared base provenance, runtime flag support, and the behavioural probe that proves the effect."
|
||||
---
|
||||
|
||||
# GGUF LoRA / adapter fit census (read-only, before you promise a file to a payload)
|
||||
|
||||
A 3 MB adapter is a payload change: it edits the compute graph at load. Do the census
|
||||
before editing an endpoint's trio, and prove the EFFECT behaviourally — no metadata,
|
||||
card table, or health gate can see it.
|
||||
|
||||
## 1. Census the Hub repo first (no auth for public repos)
|
||||
|
||||
```bash
|
||||
curl -sS "https://huggingface.co/api/models/<org>/<repo>?blobs=true" | python3 -c "
|
||||
import json,sys; d=json.load(sys.stdin)
|
||||
print(d.get('pipeline_tag'), d.get('library_name'))
|
||||
print([t for t in d.get('tags',[])][:20]) # base_model / base_model:adapter rows matter
|
||||
for s in d['siblings']: print(s['rfilename'], s.get('size'), (s.get('lfs') or {}).get('oid'))"
|
||||
curl -sS "https://huggingface.co/<org>/<repo>/raw/main/README.md" > /tmp/card.md # grep for --lora flags
|
||||
```
|
||||
|
||||
- `base_model` / `base_model_relation: adapter` name the pack the adapter was built for.
|
||||
- **A file with `lfs.oid = None` is a plain git object**: pin the sha256 of the Hub's bytes,
|
||||
not an LFS oid. Fetcher law still applies: `curl -fsSL --fail-with-body`, byte count,
|
||||
sha256, `--verify-only`, no resume.
|
||||
|
||||
## 2. Read the adapter's own GGUF KV section (settles provenance in one read)
|
||||
|
||||
Small python read of the KV pairs — no torch, no download beyond the adapter itself:
|
||||
magic `GGUF`, `u32 ver`, `u64 n_tensors`, `u64 n_kv`; key = `u64 len` + bytes; type `u32`
|
||||
(0 u8,1 i8,2 u16,3 i16,4 u32,5 i32,6 f32,7 bool,8 str,9 array,10 u64,11 i64,12 f64);
|
||||
array = `u32 elem_type` + `u64 count`.
|
||||
|
||||
Keys worth printing: `general.type` (`adapter`), `adapter.type` (`lora`),
|
||||
`adapter.lora_alpha`, and whatever provenance the publisher wrote — e.g. PrismML/abliteration
|
||||
packs carry `prism.abliterate.base_gguf`, `base_sha256`, `direction_row`, `target_layers`,
|
||||
`target_embedding`, `sites`, `basis`.
|
||||
|
||||
**Provenance is metadata, not a gate.** The loader matches **tensor names, not hashes**, so an
|
||||
adapter fit against a sibling quantization of the same model still applies — which is a
|
||||
decision to surface to the operator, not a failure to hide. Compare `base_sha256` against the
|
||||
sha the endpoint's own fetcher pins; if they differ, say so in the PR body, the env header and
|
||||
the endpoint page, and prove the effect behaviourally instead.
|
||||
|
||||
Two publisher facts that decide whether a fork can carry the file at all: the build it was
|
||||
verified on, and what it deliberately does NOT touch (an embedding rank is the usual crash —
|
||||
`token_embd` left alone is a graph constraint, not an omission).
|
||||
|
||||
## 3. Ask the runtime binary, not your memory
|
||||
|
||||
```bash
|
||||
<llama-server> --help | grep -iE "lora|adapter"
|
||||
```
|
||||
Expect `--lora FNAME`, `--lora-scaled FNAME:SCALE,...`, `--lora-init-without-apply`
|
||||
(+ `POST /lora-adapters` for runtime scale). Prefer `--lora-scaled <file>:<scale>` as ONE
|
||||
argv token so a scale can never reach the engine without its file, and follow the endpoint's
|
||||
existing opt-out convention (the `MMPROJ_FILE=` shape: `:`-less default so an explicitly empty
|
||||
value means "off, on purpose"). Gate the deployer on the file's existence with the adapter's
|
||||
OWN fetcher named in the error text.
|
||||
|
||||
## 4. Prove the effect: refusal/behaviour probe, before AND after
|
||||
|
||||
The probe is not a bench, so the no-bench law does not block it — and the **before** half must
|
||||
be captured while the old shape is still live.
|
||||
|
||||
- Same prompts, `temperature` low, both before and after; classify with a refusal-marker regex
|
||||
and print only the opening characters, never the generated body.
|
||||
- **Budget ≥ 2 048 tokens on a thinking model.** A 600-token request can be swallowed whole by
|
||||
the reasoning channel: `content` comes back empty with `finish_reason: length` and
|
||||
`reasoning_content` full — which reads exactly like a refusal that never happened.
|
||||
- A leading `⚠️ Disclaimer`/hedge after the adapter is style, not the circuit: judge the marker
|
||||
sentence, not the tone.
|
||||
|
||||
## 5. Post-deploy reads (the ledger is the instrument)
|
||||
|
||||
- Boot log shows `llama_adapter_lora_init_impl: loading lora adapter from '<file>'` and then
|
||||
**dumps the adapter's KV block** — the provenance mismatch is printed at every start, so the
|
||||
runbook should point at the journal instead of restating it.
|
||||
- Confirm the flag on argv: `tr '\0' ' ' < /proc/$(systemctl show <svc> -p MainPID --value)/cmdline`;
|
||||
confirm `systemctl show <svc> -p DropInPaths --value` is empty (a leftover bench drop-in outranks
|
||||
everything you just deployed).
|
||||
- Shape must be untouched by an adapter: `n_slots` / `n_ctx_slot` / KV lines unchanged, and
|
||||
`/metrics` still answering (a llama.cpp endpoint is invisible to Prometheus without `--metrics`).
|
||||
- Root units log to the journal; rootless podman storage is read WITHOUT sudo (`sudo podman
|
||||
inspect` sees a different storage namespace and reports "no such object").
|
||||
|
||||
## Yields
|
||||
|
||||
One sentence per question: does the file exist and at what size/sha; which pack it declares as
|
||||
its base and whether that matches the served payload; which flag the runtime actually offers;
|
||||
what the before/after probe classified. Anything else is a decision for the operator, and the
|
||||
rollback should be one env line on a file the endpoint already loads.
|
||||
@@ -19,6 +19,8 @@ for p in (home/".config/secrets").iterdir():
|
||||
if m: vals.add(m.group(1))
|
||||
known[p.name] = {v for v in vals if len(v) >= 12}
|
||||
```
|
||||
Also add every secret the repo itself embeds (e.g. credentials living in a *test helper*)
|
||||
by concatenating its quoted fragments, then search for the joined value.
|
||||
Report ONLY `value[:4] + len + sha256[:10]`. Never print a value, and never quote a
|
||||
secret-shaped literal in a doc/commit message — the pre-commit gate will (correctly) block it.
|
||||
|
||||
@@ -41,6 +43,19 @@ git status --porcelain -uall --ignored=matching # !! lines: on-disk ignored fi
|
||||
git config --list --show-origin # local/global/system credentials
|
||||
git ls-files | grep -E '(auth\.json|trust\.json|\.env|\.pem|\.key|identity)' # quarantine
|
||||
```
|
||||
**Invocation forms that actually work** (each of these cost a wasted round-trip):
|
||||
- `git grep --all -F '<fragment>'` — the options MUST precede the pattern; `git grep -l -F '<frag>' --all`
|
||||
treats `--all` as a pathspec and silently returns 0 lines. Grepping a *fragment* (not the full
|
||||
value) is what catches split-concatenation disguises like `"Assf.""1234"`.
|
||||
- `git ls-tree -r --format=... <rev>` works for ONE rev; pass several revs and it emits nothing,
|
||||
no error. For sha→path across all refs use `git rev-list --objects --all`, which yields
|
||||
`"<sha> <path>"` lines directly.
|
||||
- `git cat-file --batch` fed with `rev-list --objects` lines echoes `sha path` headers, so naive
|
||||
header parsing misaligns and content is lost. Feed bare SHAs in chunks, or use
|
||||
`git cat-file --batch-all-objects --batch` and parse `"<sha> blob <size>\n<content>\n"` records.
|
||||
- Check every worktree/checkout: `git -C <other-checkout> rev-parse HEAD` often differs from the
|
||||
audited worktree, so the leak may exist on `main` while the audited tree looks clean.
|
||||
|
||||
LFS: scan `.git/lfs/objects/**` as BYTES (`decode("latin-1")`) — a screenshot is a classic
|
||||
secret carrier and the tracked tree holds only pointers.
|
||||
|
||||
@@ -53,6 +68,9 @@ Flag every one whose value is in `known` as a LIVE leak; list the rest for triag
|
||||
- prose: `Set password: user.set_password(...)`, `authenticate(user, password)`
|
||||
- 40-hex hits that are git object ids / packfile names (`pack-<40hex>.pack`)
|
||||
- memory/KB records that *describe* a secret operation
|
||||
- length-threshold gates that miss short passwords: a check like
|
||||
`len(val) >= 24 and re.fullmatch(...)` clears a real 12-char password, so a passing
|
||||
secret-scan test proves nothing about short credentials
|
||||
A hit is a leak only when the value matches a live secret, decodes to a real credential, or
|
||||
sits in a field a program reads.
|
||||
|
||||
@@ -61,8 +79,12 @@ Check file modes of the secret dir (`0600`, dir `0700`) — world-readable secre
|
||||
most common real finding. Then grep the live values across shell rc files, `~/.gitconfig`,
|
||||
`~/.npmrc`, `~/.netrc`, `~/.curlrc`, `~/.docker/config.json`, `~/.cache/huggingface/token`,
|
||||
`~/.aws/credentials`, `~/.ssh/config` and report filenames only.
|
||||
Also check files the deployer *generates* on hosts: a script that writes a credential file
|
||||
without `chmod 600` (while sibling writes in the same script do chmod) is the same class.
|
||||
|
||||
## 6. Report
|
||||
Per layer: scope numbers + verdict. Every hit: path:line, redacted value, verdict
|
||||
(leak / benign + why). Name the gate's coverage gap if the project's own hook misses shapes
|
||||
you matched. Fix modes/chmod on the spot; changes to a security hook are the operator's call.
|
||||
A secret in git history is only removable by rotation or history rewrite — say so, don't
|
||||
imply an edit fixes it.
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
---
|
||||
name: git-secret-gate-hook
|
||||
description: "Write a fail-closed pre-commit secret-gate hook (staged-content scan) with the grep/ERE traps that kill naive versions, plus its bats fixture test."
|
||||
---
|
||||
|
||||
# Git secret-gate pre-commit hook
|
||||
|
||||
Use when a repo promises a "pre-commit secret gate" but none exists, or when porting the gate to another dank repo. Ships with three traps already paid for.
|
||||
|
||||
## Shape
|
||||
|
||||
- Track the hook at `hooks/pre-commit` (chmod +x). Git never self-installs hooks — wire per clone:
|
||||
`git config core.hooksPath hooks` — run it in the MAIN checkout once; `core.hooksPath` lives in the
|
||||
**common** git dir config, so every linked worktree inherits it.
|
||||
- Do NOT put `hooks` in `package.json` `files` — an installed plugin under node_modules never runs git.
|
||||
- Scan **staged** content (`git show ":$f"`), not the working tree: that is what lands in history.
|
||||
- Denylist by path (README, docs/, tests/, `.env.example`) — files whose job is naming secrets or holding
|
||||
fixture dummies produce theatre, not safety.
|
||||
- Never echo the matched secret; print file + line number only.
|
||||
- Red-first test: fixture repo + `cp` the real hook in + `git config core.hooksPath hooks`, then drive
|
||||
REAL `git commit` attempts: credential-shaped file must exit non-zero AND `git log` must show no new
|
||||
commit (block, not warn); clean commit passes; hook must survive committing itself.
|
||||
|
||||
## The three traps (each cost a red test)
|
||||
|
||||
1. **`--` is a ONE-SHOT end-of-options marker.** `grep -nE -- 'p1' -- 'p2'` treats everything after the
|
||||
first `--` as FILE NAMES: `grep: --: No such file or directory`, rc=2. Multi-pattern = repeat `-e`:
|
||||
`grep -nE -e 'p1' -e 'p2'`.
|
||||
2. **`if hits=$(grep ...)` is fail-OPEN on rc>=2** (rc=2 is truthy, hits empty). Capture `rc=$?`
|
||||
explicitly: `rc>=2` must BLOCK with "pattern error, refusing to pass unchecked" — a gate whose regex
|
||||
silently dies is worse than none.
|
||||
3. **POSIX ERE (`grep -E`) gotchas:** no `(?:...)` groups (use plain `(...)`); a `[:-]` inside a bracket
|
||||
starts a collating element `[:` and dies with "repetition-operator operand invalid" — put `-` FIRST
|
||||
in every class (`[-A-Za-z0-9/+=_.:+]`). Also `-----BEGIN` literals: prefer printf-safe fixtures in tests.
|
||||
|
||||
## Pattern set (shapes, not entropy — say so in the header comment)
|
||||
|
||||
```
|
||||
-----BEGIN[ A-Z0-9]*(OPENSSH|RSA|DSA|EC|PGP)? ?PRIVATE KEY-----
|
||||
AKIA[0-9A-Z]{16}
|
||||
gh[pousr]_[A-Za-z0-9]{36,}|github_pat_[A-Za-z0-9_]{50,}
|
||||
xox[baprs]-[A-Za-z0-9-]{10,}
|
||||
https://hooks\.slack\.com/services/[A-Za-z0-9/]+
|
||||
(aws_secret_access_key|secret(_access)?_key|(api|access|auth)[_-]?key|[a-z_]*token|passwd|password|credential)[[:space:]"']?[[:space:]]*[:=][[:space:]]*["']?[-A-Za-z0-9/+=_.:+]{16,}
|
||||
(Bearer|token|authorization)[[:space:]]*[:=][[:space:]]*["']?[-A-Za-z0-9._+/]{20,}
|
||||
```
|
||||
|
||||
## Test fixture hygiene
|
||||
|
||||
Fake secrets must be inert (RFC 2606 domains, all-zero ids, `AKIA0000000000000EXAMPLE`) AND must not
|
||||
match the hook's own self-scan. Single quotes in ERE patterns: write `'"'"'`.
|
||||
|
||||
## Wiring gotcha (dank worktree policy)
|
||||
|
||||
Set `core.hooksPath` AFTER the merge that lands `hooks/` into main — with the config set but the path
|
||||
missing, git fails every commit in worktrees lacking `hooks/` ("cannot run hooks/pre-commit: No such
|
||||
file or directory"). Order: merge -> land -> `git config core.hooksPath hooks` from main checkout.
|
||||
|
||||
Reference implementation: `dank/dankvoice` `hooks/pre-commit` (PR #19), pinned by `tests/gate.bats`.
|
||||
@@ -1,21 +1,27 @@
|
||||
---
|
||||
name: gitea-api-pr-operations
|
||||
description: "\"Open, list, merge, and diagnose issues/PRs on the Dankclaw Gitea API (192.168.0.143:3000, repo dank/command-center) with the token stored in ~/.config/secrets/gitea — incl. the required merge payload, where review comments actually live on 1.27.3, and how to recover the session id for the mandatory body footer.\""
|
||||
description: "\"Open, list, merge, and diagnose issues/PRs and manage labels on the Dankclaw Gitea API (192.168.0.143:3000) with the token stored in ~/.config/secrets/gitea — incl. the token-only write auth (Basic reads but 422s on POST/PATCH), the required merge payload, where review comments actually live on 1.27.3, and how to recover the session id for the mandatory body footer.\""
|
||||
---
|
||||
|
||||
# Gitea API PR operations (Dankclaw `dank/command-center`)
|
||||
# Gitea API operations (Dankclaw `dank/command-center`, `dank/serverAppConfigs`)
|
||||
|
||||
## Where
|
||||
- Token: `~/.config/secrets/gitea` (`http://user:TOKEN@192.168.0.143` single line).
|
||||
- Token: `~/.config/secrets/gitea` (`http://user:TOKEN@192.168.0.143` single line — git-credential URL form, NOT `username=`/`password=` lines).
|
||||
Extract WITHOUT echoing:
|
||||
`tok=$(sed -nE 's#^http://[^:/@]+:([^@]+)@192\.168\.0\.143.*#\1#p' ~/.config/secrets/gitea | head -1)`
|
||||
- Build a 600-perm temp curl config and use `curl -K`:
|
||||
`cfg=$(mktemp) && chmod 600 "$cfg" && printf 'header = "Authorization: token %s"\n' "$tok" > "$cfg"`
|
||||
Delete `$cfg` after use. NEVER pass `-H "Authorization: token $tok"` (the literal
|
||||
header lands in transcript + logs). NEVER echo the token.
|
||||
- Base URL: `http://192.168.0.143:3000/api/v1/repos/dank/command-center`.
|
||||
- Base URL: `http://192.168.0.143:3000/api/v1/repos/dank/<repo>`.
|
||||
|
||||
## Recipes (all verified live 2026-10-18)
|
||||
## Auth: reads tolerate Basic, writes DO NOT (measured 2026-10-20)
|
||||
GETs succeed with an HTTP Basic header built from the same credential, but every
|
||||
POST/PATCH/DELETE with Basic returns `407/422 {"message":"token is required"}`.
|
||||
Always use the `Authorization: token <tok>` header for everything — never debug
|
||||
"permissions" when the header shape is the fault.
|
||||
|
||||
## Recipes (all verified live)
|
||||
- **Open PR** (token only needed for private repos; this one reads publicly but
|
||||
POST needs auth): `POST /pulls` JSON `{title, head, base:"main", body}`.
|
||||
- **PATCH PR title/body**: `PATCH /pulls/<n>` with JSON body `{title}` and/or
|
||||
@@ -36,6 +42,12 @@ description: "\"Open, list, merge, and diagnose issues/PRs on the Dankclaw Gitea
|
||||
merge --ff-only origin/main` propagates the live deployment.
|
||||
- **Issues**: same shape at `/issues` (`POST {title, body}`), `/issues/<n>/comments`
|
||||
(`POST {body}`, `PATCH .../comments/<id>`) — both work with the temp-cfg auth.
|
||||
- **Labels**: `GET /labels` (list incl. numeric `id`); `POST /labels`
|
||||
`{"name":"<name>","color":"#rrggbb"}` to create; attach by ID with
|
||||
`POST /issues/<n>/labels {"labels":[<id>,...]}` (names are rejected);
|
||||
detach with `DELETE /issues/<n>/labels/<id>`. serverAppConfigs gained
|
||||
needs-triage / needs-info / ready-for-agent / ready-for-human / wontfix
|
||||
(2026-10-20) alongside the older `human-only`.
|
||||
|
||||
## Review comments: the API lies on 1.27.3 (learned 2026-10-18)
|
||||
- `GET /pulls/<n>/comments` (inline review comments) → **404 `not found` on every
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
---
|
||||
name: gitea-credential-store-url-format
|
||||
description: "Auth to the Dankclaw Gitea API without the empty-header trap: ~/.config/secrets/gitea is a git-credential-store URL file, so awk '/^password=/' returns empty; resolve the token via git credential fill, and Gitea writes need Authorization: token (reads pass anonymously on a public repo)."
|
||||
---
|
||||
|
||||
# Gitea auth on Dankclaw: the URL-format credential store
|
||||
|
||||
`~/.config/secrets/gitea` is `git config credential.helper = store --file ~/.config/secrets/gitea`, i.e. a **git-credential-store URL file**:
|
||||
|
||||
```
|
||||
http://dank:<40hex-token>@192.168.0.143:3000
|
||||
```
|
||||
|
||||
It does NOT contain `username=`/`password=` lines. So:
|
||||
|
||||
```bash
|
||||
# WRONG — silently returns empty string
|
||||
p=$(awk -F= '/^password=/{print substr($0,10)}' ~/.config/secrets/gitea)
|
||||
```
|
||||
|
||||
An empty `$p` makes every curl send `Authorization: token ` (blank) → Gitea replies `{"message":"token is required"}` on **writes**, while **reads still return 200** (the repo is public-readable, so a green `GET /labels` or `/issues` proves nothing about auth). Symptom: PR-create returns `token is required` even though a `GET` just "worked".
|
||||
|
||||
## Resolve the real token via git
|
||||
|
||||
```bash
|
||||
p=$(printf 'protocol=http\nhost=192.168.0.143:3000\n\n' \
|
||||
| git credential fill 2>/dev/null \
|
||||
| awk -F= '/^password=/{print substr($0,10)}') # 40-hex token
|
||||
```
|
||||
|
||||
## Write auth = token scheme, via a 600-perm -K file
|
||||
|
||||
```bash
|
||||
cfg=$(mktemp); chmod 600 "$cfg"
|
||||
printf 'header = "Authorization: token %s"\n' "$p" > "$cfg"
|
||||
curl -s -K "$cfg" http://192.168.0.143:3000/api/v1/user # confirms identity
|
||||
curl -s -K "$cfg" -X POST -H 'Content-Type: application/json' --data @body.json "$API/pulls"
|
||||
```
|
||||
|
||||
- Never pass the token with `-H` — it lands in `/proc/<pid>/cmdline`, world-readable.
|
||||
- Confirm identity with `GET /user` before trusting a write path; a `token is required` there means the token never authenticated.
|
||||
- Merges are a separate shape (form-encoded `do=merge`, auth-only cfg, NO `Content-Type` header) — see the gitea-merge-form-encoded skill.
|
||||
@@ -0,0 +1,53 @@
|
||||
---
|
||||
name: gitea-issue-labels-and-pr-curl-traps
|
||||
description: "Dankclaw Gitea API curl traps: PR create needs Content-Type application/json (fake \"[Head]: Required\"), issue labels only via POST /issues/N/labels (PATCH labels silently dropped), token from URL-format secret file."
|
||||
---
|
||||
|
||||
Operate Dankclaw Gitea (`http://192.168.0.143:3000`, repo `dank/franksign` and siblings) with curl without the three traps that produce misleading errors.
|
||||
|
||||
## Token extraction (URL-format store, NOT key=value)
|
||||
|
||||
`~/.config/secrets/gitea` is one line: `http://user:TOKEN@192.168.0.143:3000`. `cat`-ing it yields the whole URL, not the token.
|
||||
|
||||
```bash
|
||||
TOK=$(sed -n 's|.*://[^:]*:\([^@]*\)@.*|\1|p' ~/.config/secrets/gitea | head -1)
|
||||
```
|
||||
|
||||
Never pass the token as a `-H` argument (world-readable in `/proc/<pid>/cmdline`). Write a 600-perm curl config:
|
||||
|
||||
```bash
|
||||
printf 'header = "Authorization: token %s"\n' "$TOK" > /tmp/g.cfg
|
||||
```
|
||||
|
||||
## Trap 1 — PR create answers `"[Head]: Required"` when the body is the problem
|
||||
|
||||
Two independent causes, both look like a missing field:
|
||||
|
||||
1. No `Content-Type: application/json` header → Gitea parses the raw JSON as form data, sees no `head`, returns 422 `[Head]: Required`.
|
||||
2. Putting `data = "@/tmp/body.json"` inside a `-K` curl config does **not** expand the `@`.
|
||||
|
||||
Fix: add `header = "Content-Type: application/json"` to the config and pass `--data @/tmp/body.json` on the command line:
|
||||
|
||||
```bash
|
||||
curl -s -K /tmp/g.cfg -X POST --data @/tmp/pr.json \
|
||||
"http://192.168.0.143:3000/api/v1/repos/dank/franksign/pulls"
|
||||
```
|
||||
|
||||
## Trap 2 — issue label assignment via PATCH /issues/N is silently dropped
|
||||
|
||||
`PATCH /api/v1/repos/{o}/{r}/issues/{n}` with `{"labels":[id]}` returns **200/201 with the issue JSON**, yet a follow-up GET shows `labels: []` (it can even clear existing labels). Labels are a subresource:
|
||||
|
||||
```bash
|
||||
curl -s -K /tmp/g.cfg -X POST --data '{"labels":[4]}' \
|
||||
"http://192.168.0.143:3000/api/v1/repos/dank/franksign/issues/18/labels"
|
||||
```
|
||||
|
||||
200 returns the assigned label array. Resolve label names→ids via `GET /repos/{o}/{r}/labels` (franksign canonical five: ids 2–6; `ready-for-agent` = 4).
|
||||
|
||||
## Verification law
|
||||
|
||||
Both traps fail *successfully*. After any label/PR write, GET the resource and check the field changed; never trust the status code alone.
|
||||
|
||||
## Cleanup
|
||||
|
||||
`rm -f` the temp config + body files (config holds the token).
|
||||
@@ -1,27 +1,26 @@
|
||||
---
|
||||
name: gitea-merge-form-encoded
|
||||
description: "Use when a Gitea API merge on Dankclaw (192.168.0.143:3000) returns HTTP 422 \"[Do]: Required\" — the merge endpoint needs form-encoded fields, not the JSON body that PR creation accepts."
|
||||
description: "Merge a Dankclaw Gitea PR with form-encoded do=merge — and the two body failures: 422 [Do]: Required (JSON body) and jsontext invalid character 'd' (form body with Content-Type: application/json header)."
|
||||
---
|
||||
|
||||
Gitea on Dankclaw (192.168.0.143:3000, `dank/serverAppConfigs`) — merge quirk, measured 2026-10-19.
|
||||
# Gitea merge on Dankclaw: form-encoded, and nothing else
|
||||
|
||||
Creating a PR accepts a JSON body. **Merging does not**: `POST /api/v1/repos/{owner}/{repo}/pulls/{n}/merge` with a JSON body (`{"merge_style":"no-ff", "Do-Not-Reply-From":…, "Reply-To":…}`) or with those values as request headers returns `422 {"message":"[Do]: Required"}` — four variants of that all fail identically, so do not "fix" it by changing the style name or moving the headers around.
|
||||
`POST /api/v1/repos/<owner>/<repo>/pulls/<n>/merge` with `-d 'do=merge&merge_style=no-ff'`.
|
||||
|
||||
Working call (form-encoded):
|
||||
## The two body failures
|
||||
|
||||
```bash
|
||||
tok=$(sed -n 's|http://[^:]*:\([^@]*\)@.*|\1|p' ~/.config/secrets/gitea | head -1) # git credential.helper store file
|
||||
curl -sS -X POST -K <(printf 'header = "Authorization: token %s"\n' "$tok") \
|
||||
--data-urlencode 'do=merge' \
|
||||
--data-urlencode 'merge_style=no-ff' \
|
||||
--data-urlencode 'Do-Not-Reply-From=chief@franksdank.cloud' \
|
||||
--data-urlencode 'Reply-To=chief@franksdank.cloud' \
|
||||
http://192.168.0.143:3000/api/v1/repos/dank/serverAppConfigs/pulls/<N>/merge
|
||||
# 200 with an empty body == merged
|
||||
1. **JSON body** → `422 {"message":"[Do]: Required"}`. The endpoint reads FORM fields, not JSON.
|
||||
2. **Form body but a `Content-Type: application/json` header** (e.g. left over in the curl `-K` config file from an earlier PR-body PATCH on the same request setup) → `422 {"message":"[]: jsontext: invalid character 'd' at start of value"}` — Gitea tries to parse `do=merge...` as JSON.
|
||||
|
||||
## Working pattern
|
||||
|
||||
Keep TWO curl config files when a session both patches PR bodies (JSON) and merges (form):
|
||||
```
|
||||
# merge.cfg — auth ONLY, no Content-Type
|
||||
header = "Authorization: token $TOKEN"
|
||||
```
|
||||
`curl -s -K merge.cfg -X POST -d 'do=merge&merge_style=no-ff' http://192.168.0.143:3000/api/v1/repos/<o>/<r>/pulls/<n>/merge`
|
||||
|
||||
- `do=merge` is the field the 422 was demanding (the web-UI form field name, not an API doc field).
|
||||
- `merge_style` values are the dashed names: `no-ff`, `merge`, `rebase`, `squash`.
|
||||
- Keep the token out of argv: `-K <(printf …)` feeds the `Authorization` header through stdin (a `-H "Authorization: token …"` lands in `/proc/<pid>/cmdline`, world-readable — the 2026-09-16 HF-token leak class).
|
||||
- Verify by fast-forwarding a checkout (`git fetch && git merge --ff-only origin/main`) rather than re-reading the PR.
|
||||
- Fleet law still applies: never merge without an explicit current OK from Frank, and open the PR URL in his browser (`xdg-open`) the moment it exists.
|
||||
- Success = **HTTP 200 with an empty body**; verify by landing a checkout (`git ls-remote origin main`), never by re-reading the PR.
|
||||
- PATCH `/pulls/<n>` (title/body) is the opposite: JSON body WITH the Content-Type header.
|
||||
- Token from `~/.config/secrets/gitea` via a 600-perm `-K` file, never `-H` (`/proc/<pid>/cmdline` is world-readable).
|
||||
|
||||
@@ -1,29 +1,86 @@
|
||||
---
|
||||
name: gitea-pr-api-cycle
|
||||
description: "Open/update/merge PRs and issues on the Dankclaw Gitea API (192.168.0.143:3000, dank/command-center) with the token-safe curl+jq sequence, including the {\"Do\":\"merge\"} merge payload and post-merge live-checkout fast-forward."
|
||||
description: "\"Open/update/comment/merge PRs and issues on the Dankclaw Gitea API (192.168.0.143:3000) with the token-safe sequence (POST /pulls rejects Basic with 'token is required'), numeric label ids, the form-encoded merge field, the async 405 'try again later' merge trap, the empty-token-returns-404 trap, and post-merge live-checkout fast-forward plus the plugin-package-name upgrade gotcha.\""
|
||||
---
|
||||
|
||||
# Gitea PR/issue cycle — dank/command-center
|
||||
# Gitea PR/issue cycle — Dankclaw API
|
||||
|
||||
Friction-earned 2026-10-18 across PRs #15–#20 and issue #18: three separate fumbles (jq trailer flag, 422 merge payload, silent -o/-w swallowing) codified once.
|
||||
Friction-earned 2026-10-18 (PRs #15–#20, issue #18), re-earned 2026-10-19 on `dank/dankvoice` PRs #4–#6 (the merge payload claim was wrong; an empty token masqueraded as a missing repo), and re-earned 2026-10-20 on `dank/command-center` PRs #55…#57 (the 405 recalculation trap). Re-earned again 2026-10-20 on `dank/dankvoice` #25 (merge gate loop worked; suite verdict delivered as an issue comment). Re-earned 2026-10-21 on `dank/command-center` issue #58 / PR #59 (issue labels are numeric ids; the `<type>/<issue#>-<slug>` branch name has to be right *before* the first push). Same day on `dank/serverAppConfigs` #64: the Basic-vs-token split below cost two wasted POSTs.
|
||||
|
||||
## Where
|
||||
- API base: `http://192.168.0.143:3000/api/v1/repos/dank/command-center/{pulls,issues}`
|
||||
- Token: never inline, never echoed. Extract from the git credential file into a 600-perm temp curl config:
|
||||
`tok=$(sed -nE 's#^http://[^:/@]+:([^@]+)@192\.168\.0\.143.*#\1#p' ~/.config/secrets/gitea | head -1)`
|
||||
`cfg=$(mktemp); chmod 600 "$cfg"; printf 'header = "Authorization: token %s"\n' "$tok" > "$cfg"` → `curl -K "$cfg"`; `rm -f "$cfg"` when done.
|
||||
- Unauthenticated GET works for state reads (`GET /pulls/N` → `jq -c '{state,merged,merge_commit_sha}'`).
|
||||
- API base: `http://192.168.0.143:3000/api/v1/repos/dank/<repo>/{pulls,issues}` — repo varies (`command-center`, `dankvoice`, `serverAppConfigs`); numbering is per-repo, so a footer citing `#36` means nothing without the repo.
|
||||
- Web UI pages under `/dank/<repo>/pulls/N` are session-auth private: a plain URL fetch answers **404** even when the API and `xdg-open` work. Merge state questions go through the API (`GET /pulls/N` → `state`, `merged`, `merge_commit_sha`) — never conclude from a failed page fetch.
|
||||
- **Basic vs token split**: the `username=`/`password=` fields of `~/.config/secrets/gitea` as `Authorization: Basic …` work for **reads and issue/PR comment writes**, but `POST /pulls` refuses them with `403 token is required`. Pull-request lifecycle writes need the **token** header. Use the token for everything PR-shaped; never conclude "API is broken" from a Basic 403 on `/pulls`.
|
||||
- Token: never inline, never in argv (`-H "Authorization: token …"` lands in `/proc/<pid>/cmdline`, world-readable). Extract into a 600-perm curl config and hand it to curl with `-K`:
|
||||
```bash
|
||||
tok=$(sed -nE 's#^https?://[^:/@]+:([^@]+)@192\.168\.0\.143.*#\1#p' ~/.config/secrets/gitea | head -1)
|
||||
echo "tok len ${#tok}" # 40 — assert before use
|
||||
cfg=$(mktemp); chmod 600 "$cfg"; printf 'header = "Authorization: token %s"\n' "$tok" > "$cfg"
|
||||
curl -sS -K "$cfg" http://192.168.0.143:3000/api/v1/repos/dank/<repo>/labels
|
||||
rm -f "$cfg"
|
||||
```
|
||||
- **Empty token answers `404 {"message":"not found"}`, NOT 401** — Gitea hides the private repo. A 404 on a path you have used before means auth, not a typo'd repo. The trap that produces the empty token: `awk -F'/' … '{print $NF}'` on the store line (`scheme://user:token@host/`) — the host is in `$3`, the match never fires, you get a silent empty variable. Always assert the length before POSTing.
|
||||
- Multi-line bodies: build JSON in python (`json.dumps({"title":…, "body":open(f).read(),…})`) or `jq -n --arg b "$(cat b.txt)"`. NEVER pass `---` as a bare jq argument (parsed as a flag). Escaped quotes inside `python3 -c '…'` break on `\"` — write the script to a file or use a heredoc.
|
||||
- Always capture status: `-o r.json -w '%{http_code}'`; a bare `-o` hides failures. A `201` response piped straight into `python3 -c` that prints `None None` means you parsed an **error** body (missing `number`) — print `message` too.
|
||||
|
||||
## Body construction (the two traps)
|
||||
- Build JSON with `jq -n --arg t <title> --arg b "$(cat b.txt)" '{title:$t, head:<branch>, base:"main", body:$b}'`; write the body to a temp file first (`printf '%s' … > b.txt`). NEVER pass `---` as a bare jq argument — jq parses it as a flag (`unknown flag: ---`). The AGENTS.md attribution trailer (`---` + `cwd: … · session: …`) goes INSIDE the body string.
|
||||
- Always capture status explicitly: `-o r.json -w '%{http_code}'` and print it; a bare `-o` hides failures (a 422 looked like success-with-no-output once).
|
||||
## Create
|
||||
`POST /pulls` with `{title, body, base:"main", head:<branch>}` — JSON is accepted here (unlike merge). Plain `<branch>` works; the `owner:branch` fork form is unnecessary on same-repo branches.
|
||||
- **409 `pull request already exists for these targets [issue_id: N, …]`** — a PR for that head/base exists already. Common cause: the push message printed `/pulls/new/<branch>` and someone (Frank, or an `xdg-open` you ran) clicked Create. `GET /pulls?state=all`, find it, then `PATCH /pulls/N` with your title+body instead of re-posting.
|
||||
- `GET /pulls?state=all` on this Gitea omits `head`/`base` per entry — read `/pulls/N` for those, or key off `number`/`user`.
|
||||
- Issues: same shape, `POST /issues` `{title, body}`; `PATCH /issues/N` fixes a body retroactively (works for PRs too, since a PR is an issue) — used to correct a false verification claim in a PR that had already been merged. Body update returns **201**, not 200; confirm by reading `body` back, not by status code.
|
||||
- **Issue labels are numeric ids, not names.** `POST /issues` with `"labels":["ready-for-human"]` answers `422 {"message":"[]: json: cannot unmarshal JSON string into Go int64 within \"/labels/0\""}`. Resolve first: `GET /labels` → pick `id` by `name` → post `"labels":[20]`. (PRs take no `labels` field on create; label afterwards via `/issues/N/labels` with `{"labels":[id]}`.)
|
||||
- Comments (the channel for late-arriving evidence, e.g. a test-suite verdict posted after PR creation): `POST /issues/N/comments` with JSON `{body}` — works for PRs too, returns **201**.
|
||||
- **Open the issue before the first commit**, so `Refs #N` fits the message, and get the branch name right before the first push. `git branch -m <new>` + `git push -u origin <new>` + `git push origin --delete <old>` is a clean rename with no PR yet; do it rather than ship a branch that violates `<type>/<issue#>-<slug>` (a `docs/…` branch carrying a `fix:` commit is the common slip, and `git grep` on the old name still shows it).
|
||||
|
||||
## Merge (the 422 trap)
|
||||
`POST /pulls/N/merge` requires body `{"Do":"merge"}` — `Do` is the Required field on this Gitea. `{}`, `{"DoNotMerge":false}`, `{"merge_style":…}` all return 422 (`"[Do]: Required"`). Success may return an EMPTY body — verify by GET afterwards (`state=closed, merged=true`), not by the POST response.
|
||||
- Merge only with Frank's explicit per-PR OK (repo law).
|
||||
- Post-merge the live deployment must follow: `git -C ~/.omp fetch -q origin && git -C ~/.omp merge --ff-only origin/main` (the checkout IS the deployment; branch is `main`).
|
||||
## Merge (the 422 trap) — corrected
|
||||
`POST /pulls/N/merge` needs **form-encoded** fields; the merge form field name is lowercase **`do`**:
|
||||
|
||||
## PR etiquette per AGENTS.md
|
||||
- Immediately after opening: `xdg-open <PR-URL>` lands it in Frank's Firefox.
|
||||
- Redesign mid-review: amend + `git push -f` the branch, then PATCH the PR body (`/pulls/N`, `{"body":…}`) so title/body match the net diff — verify `{number,state,mergeable}` in the response.
|
||||
- Issue creation identical shape: `POST /issues` with `{title, body}`, expect 201; PATCH same endpoint `/issues/N` to fix bodies retroactively, then GET-readback verify.
|
||||
```bash
|
||||
curl -sS -X POST -K "$cfg" \
|
||||
--data-urlencode 'do=merge' --data-urlencode 'merge_style=no-ff' \
|
||||
http://192.168.0.143:3000/api/v1/repos/dank/command-center/pulls/N/merge
|
||||
```
|
||||
|
||||
JSON `{"Do":"merge"}` (what this skill used to claim) and header variants all return `422 {"message":"[Do]: Required"}`. `merge_style` ∈ `no-ff|merge|rebase|squash`. Success returns **200 with an empty body**; verify by landing a checkout rather than re-reading the PR. Merge only with Frank's explicit per-PR OK.
|
||||
|
||||
### The async 405 when several PRs merge in a row
|
||||
Merge #55, then #56 and #57 back-to-back, and #56/#57 answered:
|
||||
|
||||
```
|
||||
405 {"message":"Please try again later","url":"http://dankclaw:3000/api/swagger"}
|
||||
```
|
||||
|
||||
That is not a failure and not rate limiting: merging one PR moves `main`, which invalidates every other PR's cached mergeability, and Gitea recalculates it asynchronously. A merge attempt during recalculation is refused. Remedy — bounded poll on the PR, merge when the flag is true:
|
||||
|
||||
```bash
|
||||
for attempt in 1 2 3 4 5 6 7 8 9 10; do
|
||||
m=$(curl -sS -K "$cfg" "$api/pulls/$n" | python3 -c \
|
||||
'import json,sys;d=json.load(sys.stdin);print(d.get("mergeable"),d.get("merged"))')
|
||||
case "$m" in *True*) <merge POST>; break ;; *) sleep 4 ;; esac
|
||||
done
|
||||
```
|
||||
|
||||
All three landed within ~5 s of retrying. Treat `mergeable` as a readiness gate, not a report. Match `*True*` on the pair defensively: `False True` means already merged — stop, don't merge again.
|
||||
|
||||
### Verifying a merge without touching the deployment
|
||||
`GET /pulls/N` → `state: closed`, `merged: true`, `merge_commit_sha`. Then prove content from the bare repo, not the working tree: `git fetch -q origin && git show origin/main:<path> | grep -n <key line>` and `git log --oneline --graph -4 origin/main` (two parents = the `--no-ff` Frank asks for). A number you deliberately destroyed can legitimately reappear once — quoted in `omp-docs/log.md` as the wrong value; `git grep -c <badvalue> origin/main` naming only the log page is the pass condition, not a leftover. `Closes #N` in the PR body auto-closes the issue; confirm with `GET /issues/N` → `state`.
|
||||
|
||||
## Stacked PRs (no rebase)
|
||||
When two open branches edit the same section of the same file, merge the sibling branch into the later one (`git merge docs/other-branch`) instead of rebasing — Frank rejects the history rewrite. Consequences to state in the PR body: merging the lower PR first reduces the upper diff to just its own work; merging the upper first makes the lower an already-contained no-op. Either order is clean; `git branch -d` on the lower branch refuses until its commits reach `main`, because `-d` checks against the current HEAD.
|
||||
|
||||
## Post-merge landing
|
||||
- Command-center: the checkout IS the deployment — `git -C ~/.omp fetch -q origin && git -C ~/.omp merge --ff-only origin/main`. This refuses if **any** incoming path is dirty in the worktree — see `land-merged-pr-with-blocked-tree-ops` for the shared-workspace case where `checkout` and `stash` are both policy-blocked.
|
||||
- serverAppConfigs: `deploy.sh` pushes the dev checkout's branch first — if the checkout's `main` is behind a merge you just made (via the API), the deploy dies with `error: failed to push some refs`. `git merge --ff-only origin/main` in the deploy checkout BEFORE running `deploy.sh` after any externally-merged PR.
|
||||
- **A PR that changes a path you edited live adds that path to the discard list.** Tonight: the live dotfile fix was dirty *and* incoming, so landing needed `git -C ~/.omp checkout -- <that file>` as well as the usual file. Discard is content-free only when you proved byte identity first (`cmp` the branch file against the live one before merging) — say so in the PR body, and note that if `live-sync.sh` commits the dirty file first the merge stays clean because both sides hold identical bytes.
|
||||
- Package repos (`~/Projects/<pkg>`): same `--ff-only`, then reinstall (`omp plugin upgrade <name>`) and `cmp` the installed `dist/index.js` against the repo `dist/index.js` to prove the live artifact is the merged one. Check `ls -ld node_modules` after landing anything that touched it.
|
||||
- **`omp plugin upgrade` takes the npm package name, not the short name.** `omp plugin upgrade dankvoice` fails with `Failed to upgrade dankvoice: dankvoice is not installed` while the plugin is plainly installed; the package is `omp-dankvoice`. Find the truth with `ls ~/.omp/plugins/node_modules` or `find ~/.omp -path '*<name>/dist/index.js' -not -path '*/Projects/*'` — the installed tree is unscoped (`~/.omp/plugins/node_modules/omp-dankvoice/dist/index.js`), not `@dank/dankvoice`.
|
||||
- **`cmp` exiting 1 BEFORE the upgrade is the expected state** (installed bundle is the pre-merge build, diff reported at some char/line). Do not treat it as a failed deploy and do not "fix" the build; run the upgrade, then `cmp` again for byte identity, and grep the new bundle for a string only the merged code contains (e.g. `grep -c "Heckle instead of summarizing"`) as a second, semantic confirmation.
|
||||
- Prune: `git worktree remove --force <path>` (force: untracked deps link), `git branch -D`, `git push origin --delete <branch>` (allowed: local Gitea). Before removing a worktree, check no live process has a cwd inside it (`ls -l /proc/*/cwd | grep -c "wt/<name>"`) — same guard `live-sync.sh` uses — and copy any file you moved into a gitignored `.AGENTTRASH/` there to its permanent home first, or the removal destroys the only copy.
|
||||
|
||||
## Attribution footer (AGENTS.md)
|
||||
Every issue/PR body ends with `---`, then `cwd: <session cwd> · session: <session id>`. Read the real values, never invent an id.
|
||||
- **The session id is the UUID suffix of the LIVE transcript `.jsonl`, not the session directory name.** Transcript dirs are keyed by cwd (e.g. `-.omp-wt-wt-20261019-020309-6d1fc1f`) and ONE cwd can hold several sessions (resume / continuation forks), so `ls -t ~/.omp/agent/sessions/` cannot tell you which is yours and `ls -t <dir>` is equally blind. Pick the file being appended to right now: `ls -l --time-style=+%H:%M:%S <dir>/*.jsonl` beside `date +%H:%M:%S` — the mtime that equals the current second is the live session. The id is the part after `_` in `<timestamp>_<uuid>.jsonl`.
|
||||
- A `history.db` query for `session_id` filtered by `cwd` usually agrees, but it is not authoritative for WHICH of several sessions in this cwd is live — use the jsonl mtime test.
|
||||
- `local/` does NOT disambiguate: a forked transcript dir carries copies of the same `local://` artifacts, so matching artifact names proves nothing about which session is live.
|
||||
- `cwd` is that session's own working directory (the path the harness states), **not** the build worktree you committed in — when they differ, name the build worktree inside the body and keep `cwd` the session's.
|
||||
- Resolve `local://<name>.md` with `read local://<name>.md` (its header prints the absolute path) instead of guessing the directory.
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
name: gitea-silent-ignore-patch-trap
|
||||
description: "Gitea API answers success codes while silently ignoring mutation bodies: labels in PATCH /issues/N, and any JSON body sent without Content-Type: application/json. Verify mutations with a follow-up GET, never the status code."
|
||||
---
|
||||
|
||||
# Gitea silent-ignore family (issue mutations)
|
||||
|
||||
Two verified failure modes on the Dankclaw Gitea API (1.27.3, verified live 2026-10-21 on `dank/franksign`); both **return a success code and change nothing**:
|
||||
|
||||
1. **`PATCH /repos/{o}/{r}/issues/{n}` with `{"labels":[...]}`** — 200, labels untouched, even *with* a proper JSON Content-Type. Labels move only via the dedicated endpoints:
|
||||
- add: `POST /issues/{n}/labels` with `{"labels":[<id>]}`
|
||||
- replace: `PUT /issues/{n}/labels` with `{"labels":[ids]}` (works without Content-Type)
|
||||
- remove: `DELETE /issues/{n}/labels/{id}`
|
||||
2. **Any JSON body without `header = "Content-Type: application/json"` in the curl config** — e.g. closing with `PATCH {"state":"closed"}` returns **201 and no-ops**; identical request with the header closes. curl's default `-d` type (`application/x-www-form-urlencoded`) makes this silent.
|
||||
|
||||
## Rules
|
||||
- Mutating JSON calls: always send `Content-Type: application/json` alongside `Authorization: token` in the same `-K` config file.
|
||||
- **Never trust the status code for mutations — follow up with `GET` and read the field** (`labels`, `state`). Both traps look like success otherwise.
|
||||
- Token stays out of `-H` args (world-readable `/proc/<pid>/cmdline`): extract from `~/.config/secrets/gitea` into a 600-perm curl config, use `-K`.
|
||||
|
||||
## Quick verification idiom
|
||||
```bash
|
||||
curl -sK "$cfg" "http://192.168.0.143:3000/api/v1/repos/dank/franksign/issues/22" \
|
||||
| jq '{state, labels: [.labels[].name]}'
|
||||
```
|
||||
|
||||
Franksign's vault page `franksign-docs/agents/issue-tracker.md` owns the repo-local copy of this law; this skill exists so other repos (serverAppConfigs, command-center) don't re-burn the cycle.
|
||||
@@ -0,0 +1,47 @@
|
||||
---
|
||||
name: gitea-token-from-http-credential-store
|
||||
description: "Extract the Dankclaw Gitea token from ~/.config/secrets/gitea (it is the URL password, not key=value lines) for API calls, and diagnose the silent empty-token trap that makes public-repo reads pass while writes fail."
|
||||
---
|
||||
|
||||
The Gitea credential store line on Frank's boxes is **http** (not https) with a percent-encoded port, and the **token is the URL password** — there are NO `username=`/`password=` lines:
|
||||
|
||||
```
|
||||
http://dank:<40-char-token>@192.168.0.143%3a3000
|
||||
```
|
||||
|
||||
## Extraction that actually works
|
||||
|
||||
```bash
|
||||
tok=$(sed -nE 's#^[a-z]+://[^:]+:([^@]+)@.*#\1#p' ~/.config/secrets/gitea | head -1)
|
||||
echo "tok len ${#tok}" # must be 40 — assert before any call
|
||||
cfg=$(mktemp); chmod 600 "$cfg"; printf 'header = "Authorization: token %s"\n' "$tok" > "$cfg"
|
||||
curl -s -K "$cfg" http://192.168.0.143:3000/api/v1/<path>
|
||||
rm -f "$cfg" # in the SAME session: the file holds a live token
|
||||
```
|
||||
|
||||
## `git credential-store --file … fill` answers EMPTY — do not debug it
|
||||
|
||||
Tried 2026-10-21 while opening `dank/command-center` PR #61: the obvious extraction
|
||||
|
||||
```bash
|
||||
printf 'protocol=http\nhost=192.168.0.143\nport=3000\n\n' | git credential-store --file ~/.config/secrets/gitea fill
|
||||
```
|
||||
|
||||
prints **nothing**, with no error. Not permissions, not a wrong path: the stored line encodes the port inside the host field (`@192.168.0.143%3a3000`), so a `host=`+`port=` query can never match it. Add `path=` or drop `port=` and it still answers empty. Read the file with `sed` (recipe above) and move on — the chain `TOKEN=$(…) && [ -n "$TOKEN" ] && …` fails silently at the test, which reads like a curl failure because the `&&` hides which link broke.
|
||||
|
||||
## The silent-empty-token trap (bit 2026-10-21, cost a debug detour)
|
||||
|
||||
An `awk -F= '/^username=/{...}'` parse of this file yields `tok=""` **silently** — and reads still appear to work when the repo is **public**: `dank/serverAppConfigs` and `dank/command-center` return real issue/label data unauthenticated, so every census "passes" until the first **write** (issue comment, label, PR create/merge) fails with `{"message":"token is always required"}`-style errors or 404s (private repos 404 on an empty token — that is an auth problem, NOT a missing repo).
|
||||
|
||||
- `echo "tok len ${#tok}"` (must be 40) before believing ANY API result — reads included.
|
||||
- Prove auth once per session: `GET /api/v1/user` → `"login":"dank"`.
|
||||
- `curl -s -K <600-perm file>` carrying `header = "Authorization: token $tok"` — never `-H` on argv (`/proc/<pid>/cmdline` is world-readable; this leaked an HF token once).
|
||||
- `jq -n --arg t … --arg h … --arg b "$(/bin/cat /tmp/pr.md)" '{…}'` — `$(cat file)` gets optimised away in this harness, so use `--rawfile b /tmp/pr.md` instead.
|
||||
|
||||
## Other notes
|
||||
|
||||
- `git credential fill` fails headless here (`could not read Username … terminal prompts disabled`); read the store file directly instead.
|
||||
- Never print the token: mask when inspecting the file —
|
||||
`sed -E 's#(://[^:]+:)[^@]+@#\1REDACTED@#' ~/.config/secrets/gitea`
|
||||
- Merge writes need form-encoded fields, not JSON — see `gitea-pr-api-cycle`.
|
||||
- Issue/PR numbers are **per repo**: `dank/dankvoice` #8–#11 (refactor, hardening, AbortError fix, menu defects) are unrelated to `dank/command-center` #8–#11 (SDLC, trunk-law). Read the body from the repo the work targets before referencing an issue in a commit message.
|
||||
@@ -0,0 +1,48 @@
|
||||
---
|
||||
name: gitignore-trailing-slash-symlink-contamination
|
||||
description: "Trailing-slash gitignore rules don't match symlinks, so worktree dependency symlinks get tracked and pushed as 1-line blobs; same-name rules silently match worktree dirs, and .deploy-stale pollutes. Three fixes with tests."
|
||||
---
|
||||
|
||||
Gitignore semantics trap with three real-world consequences (serverAppConfigs, 2026-10-19):
|
||||
|
||||
## The rule
|
||||
|
||||
`.gitignore` patterns ending in `/` match directories only. Worktree-relative: `**/node_modules/` is `**/node_modules/` → a symlink named `node_modules` does NOT match, so it is "untracked" no more — it gets added, committed, and pushed as a 1-line blob containing its target path.
|
||||
|
||||
## Consequence 1: worktree symlink contamination
|
||||
|
||||
- `git add -A` then commit then push in a worktree whose root holds `node_modules -> ...` (the worktree convention symlink) commits it.
|
||||
- The repo then holds `package/frontend/node_modules` as a text file containing the absolute path on the author's box.
|
||||
- The receiving worktree gets that 1-line file instead of the symlink → "module not found" and "why is this a 30-byte file?".
|
||||
- Detection: `git ls-tree -r origin/main | awk '$1=="120000"'` (mode 120000 = symlink) or `git ls-files -s | grep 120000`.
|
||||
- Fix: ignore rules WITHOUT trailing slash (`**/node_modules`, `**/.venv`) so symlinks match too.
|
||||
|
||||
## Consequence 2: same-name worktree rules silently matching worktree dirs
|
||||
|
||||
A root rule like `wt-*` (meant to ignore local worktree checkouts) matches every path whose last segment starts with `wt-` — so legitimate content under such a directory goes untracked. `git status --ignored` is the only visible trace; detection: `git check-ignore -v <path>` prints the owning rule + line number. (Repo-specific: `dank/command-center`'s `.gitignore` carries a "per-worktree memory buckets" block ignoring `agent/memories/sharpshooter/wt-*/` and `agent/memories/summaries/wt-*/` — intentional there.)
|
||||
|
||||
## Consequence 3: `.deploy-stale`
|
||||
|
||||
`deploy.sh` renames a stale tree to `<path>.deploy-stale` before recreating it. Without an ignore rule those directories land in `git status` as untracked and get swept into `git add -A` commits; they accumulate on every server.
|
||||
|
||||
## Fixes with tests
|
||||
|
||||
```gitignore
|
||||
**/node_modules
|
||||
**/.venv
|
||||
.deploy-stale/
|
||||
```
|
||||
|
||||
Black-box test against the real rules, not a fixture: `git check-ignore -v <symlink-target>` must name the rule. `git check-ignore` takes hypothetical paths — nothing needs to exist on disk:
|
||||
|
||||
```bash
|
||||
git check-ignore -v natives/x.wasm dotfiles/x/.config/noctalia/Noctalia.json
|
||||
```
|
||||
|
||||
Two ignore surfaces must be kept in sync (deploy exclusions vs ignore rules): if a deployer excludes `*/Noctalia*` but the ignore list has only the lowercase class, generated files become tracked-but-never-deployed — test both halves with `git check-ignore`.
|
||||
|
||||
## Recovery (if contamination already pushed)
|
||||
|
||||
1. `git rm --cached <path>` on the symlink blob, commit, push.
|
||||
2. Fix the ignore rule first (else the next `git add -A` re-adds it).
|
||||
3. Untracked ≠ deleted from history: the blob stays in every old commit. Only a rewrite removes it; that is the operator's call, not a cleanup run alone.
|
||||
@@ -0,0 +1,47 @@
|
||||
---
|
||||
name: grafana-provisioned-dashboard-verify
|
||||
description: "Author and verify repo-provisioned dashboards on Dankclaw live: Grafana 13 has no unauthenticated API and no local admin, so prove through the e2e SSO browser path, and close the copier gap that hides never-provisioned boards."
|
||||
---
|
||||
|
||||
## When to use
|
||||
You changed `shared/grafana/provisioning/dashboards/*.json` (or panel PromQL) and must prove the LIVE boards actually receive data — or a panel/whole board reads empty and you must classify: dead expression vs never-provisioned vs legitimately-empty.
|
||||
|
||||
## The two gaps this skill exists for
|
||||
1. **The copier gap.** A repo dashboard JSON is NOT a live dashboard until the deployer carries it. Until 2026-10-21 `copy_grafana_provisioning` in `scripts/deploy-remote.sh` named `llm-fleet.json` alone: the `containers` board was merged, seam-tested, and documented as "working" yet had NEVER been provisioned; the other boards existed on disk (`/opt/grafana/provisioning/dashboards/`) only via one-time hand-copies, and one had silently gone stale. Fixed: copier globs `dashboards/*.json`; `GRAFANA_PROV_ROOT` is the test seam; a bats case executes the copier for real into a sandbox.
|
||||
2. **The API-proof illusion.** The vault used to say "verify through the APIs". On Grafana 13.2.2 that is dead: `/api/health`, `/api/frontend`, `/api/org`, `/login` all 404 **unauthenticated**, and there is NO local admin password (root `.env` sets no `GF_SECURITY_ADMIN_*`; the only account is the Authentik RP login). Do not hunt for credentials; use the browser path below.
|
||||
|
||||
## Procedure
|
||||
### 1. Mechanical expression census (over SSH — Prometheus binds loopback)
|
||||
Extract every panel `expr` from the repo JSONs with python, substitute template vars (`$host`,`$container`,`$job` → `.*`, `$__auto`/`$interval` → `5m`), and run each as an instant query against `http://127.0.0.1:9090/api/v1/query` via `ssh dankclaw`. `curl` to `127.0.0.1:9090` from the dev box is blocked; keep queries on-host. Watch zsh globbing on `?`/`[` — quote the remote command.
|
||||
Empty result ≠ defect automatically:
|
||||
- `ALERTS` panel empty = **healthy** (ALERTS carries only firing/pending series).
|
||||
- Zero by design: no cgroup limits exist on the fleet → a memory-vs-limit panel is dead by premise (rewrite as % of host total, don't delete).
|
||||
- Real join defects: cadvisor `host` label vs node-exporter `job` label sets are incompatible — fix with `group_left label_replace(node_memory_MemTotal_bytes{job=~"node-.*"}, "host", "$1", "job", "node-(.*)")`, and `count by (job) (expr) / on(job) count by (job) (...)` needs the `on(job)` matcher when label sets differ. Parens around `on(host)` after `group_left` are **illegal**; `group_left label_replace(...)` without them parses fine.
|
||||
|
||||
### 2. Prove the provisioned files (filesystem = installation proof only)
|
||||
```
|
||||
ssh dankclaw 'for f in /opt/grafana/provisioning/dashboards/*.json; do \
|
||||
cmp -s "$f" "/home/dank/serverAppConfigs/shared/grafana/provisioning/dashboards/$(basename "$f")" \
|
||||
&& echo "$(basename $f) same" || echo "$(basename $f) DIFFERS"; done'
|
||||
```
|
||||
Missing file or DIFFERS ⇒ the copier (or an unmerged branch) is the bug, not the JSON.
|
||||
|
||||
### 3. Browser proof (the real surface)
|
||||
Throwaway `.mjs` placed **inside `tests/e2e/`** (`@playwright/test` resolves from `node_modules` there; a script in `/tmp` fails with `ERR_MODULE_NOT_FOUND`). Node on this box is Zig-rewritten v26: `readFileSync(p,"text")` throws `ERR_INVALID_ARG_VALUE` — use `readFileSync(p).toString()`.
|
||||
Sequence, exactly as `specs/20-grafana-sso.spec.ts` drives it:
|
||||
1. `newContext({storageState: ".auth/qa-e2e-admin.json", userAgent: REAL_UA})` — state is bound to the homepage origin; on the grafana vhost it is only the Authentik session cookie, so you MUST:
|
||||
2. `goto https://grafana.franksdank.cloud/` and poll until body lacks `"Making sure you're not a bot"` (Anubis PoW warm-up).
|
||||
3. `goto .../login/generic_oauth` and poll until `url` host is the vhost and path ≠ `/login` (RP round-trip). Landing on `/login` = auth flow broken, not the dashboard.
|
||||
4. `goto /d/<uid>?orgId=1&from=now-1h&to=now`, wait ~10 s, then read panels: titles via `.panel-title,[class*='panel-title']`; a panel is `NO-DATA` iff its container text contains `"No data"`, `RENDERED` iff a descendant matches `svg:not(:empty),canvas,img`. Panels lazy-render; scroll (`mouse.wheel`) before judging.
|
||||
Classify: `PANEL-NOT-FOUND` for a whole board ⇒ board not provisioned (step 2). `NO-DATA` with a proven-live expression ⇒ variable/label mismatch — re-run the expression with template vars expanded.
|
||||
Throwaway scripts → `~/.AGENTTRASH/<date>-topic/` after use; never `rm`.
|
||||
|
||||
## Sandboxed deployment-function tests (bats)
|
||||
To behaviorally test a deploy-remote.sh copier: extract with `sed -n '/^func()/,/^}/p'`, `eval` it, stub `run_privileged() { "$@"; }`, point the function's `*_ROOT` seam env var at a sandbox, execute, assert files. Traps:
|
||||
- `VAR=x func` prefix assignments do NOT bind for shell **functions** — `export VAR=...` first.
|
||||
- `mktemp -d` + `trap 'rm -rf' RETURN` races bats' trap machinery: the sandbox vanishes mid-function and bats blames the `trap` line. Use `$BATS_TEST_TMPDIR` (auto-cleaned).
|
||||
- Prove red on the OLD body by patching a `mktemp -d` copy of `tests/ shared/ scripts/` and running bats there — no checkout mutation.
|
||||
- Full suite must run from the repo root (`./tests/run.sh`), not from a subdirectory.
|
||||
|
||||
## Land flow (repo law)
|
||||
Branch off fresh `origin/main`; merge needs Frank's explicit current OK; Gitea merge endpoint wants **JSON** `{"Do":"merge"}` (form-encoded `Do=merge` 422s with `[Do]: Required`); then `deploy.sh dankclaw --service grafana` and repeat steps 2–3 as the post-deploy gate.
|
||||
@@ -0,0 +1,51 @@
|
||||
---
|
||||
name: grep-secret-gate-hook-traps
|
||||
description: "Write a zero-dependency grep-based pre-commit secret gate that fails closed: the GNU grep -- one-shot trap, POSIX ERE limits, bracket collating elements, and rc=2-as-success in command substitution."
|
||||
---
|
||||
|
||||
# Fail-closed grep secret gate (pre-commit hook)
|
||||
|
||||
Use when writing/auditing a bash pre-commit hook (or any CI tripwire) that scans
|
||||
staged content with `grep -E` for credential shapes — no gitleaks available.
|
||||
|
||||
## The four traps (all personally earned; each silently PASSES bad content)
|
||||
|
||||
1. **`--` is a ONE-SHOT end-of-options marker.** `grep -nE -- 'p1' -- 'p2' 'p3'`
|
||||
takes the later `--` and patterns as FILE names → rc=2, errors on stderr.
|
||||
Correct multi-pattern form: repeat `-e` for every pattern; never `--`.
|
||||
2. **`if hits=$(... | grep ...)` treats rc=2 as success.** Any non-zero-but-not-1
|
||||
(broken pattern, unreadable file) enters the `then` branch with EMPTY output.
|
||||
Never branch on the substitution's truthiness. Capture rc explicitly:
|
||||
`hits=$(...); rc=$?` then: `rc>=2` → BLOCK (fail closed — a dead regex is
|
||||
worse than none), `rc==1` → clean, `rc==0` → hits.
|
||||
3. **POSIX ERE (`grep -E`) has no `(?:...)` groups.** Use plain `(...)`.
|
||||
4. **`[:-]` inside a bracket expression starts a collating element** →
|
||||
`grep: repetition-operator operand invalid` (rc=2, see trap 2). Put `-` FIRST
|
||||
in every class: `[-A-Za-z0-9/+=_.:+]`.
|
||||
|
||||
## Gate structure that works
|
||||
|
||||
- Scan **staged content**, not the working tree: `git show ":$f"` over
|
||||
`git diff --cached --name-only --diff-filter=ACM`.
|
||||
- Denylist doc/fixture paths (`README`, `docs?/`, `tests?/`, `*.example`) — else
|
||||
the hook blocks its own patterns and every test fixture.
|
||||
- Report `BLOCKED <file>:<line>` only — never echo the matched secret.
|
||||
- Wire with tracked `hooks/pre-commit` + `git config core.hooksPath hooks`
|
||||
(lives in the COMMON git dir config → one set in the main checkout covers
|
||||
every worktree). Fresh clones are unprotected until it's set; document it.
|
||||
- Worktrees lacking `hooks/` while `core.hooksPath` is set: git dies on commit
|
||||
(`cannot run ... No such file or directory`) — merge main in, don't unset.
|
||||
- Do NOT ship the hook in an npm plugin artifact (`files`): node_modules never
|
||||
runs git; pin its ABSENCE in the pack test instead.
|
||||
|
||||
## Test it red-first through the real seam (bats)
|
||||
|
||||
`git init` a fixture repo, `cp` the real hook in, `git config core.hooksPath hooks`,
|
||||
then drive REAL `git commit` and assert:
|
||||
- credential-shaped `.env` (RFC-2606 domains / all-zero ids as inert fixtures) →
|
||||
non-zero AND `git log` count unchanged (blocks, not warns);
|
||||
- PEM header line → blocked;
|
||||
- clean prose → passes;
|
||||
- the hook committing ITSELF → passes (trap 1/3/4 usually die here);
|
||||
- exit-code masking: `bats ... | tail` returns tail's rc — never `&& commit`
|
||||
behind a piped suite, or you commit on red.
|
||||
@@ -1,48 +1,53 @@
|
||||
---
|
||||
name: halogen-npu-feasibility-census
|
||||
description: "Census whether halogen-flash-server's NPU small-model path can run on a Strix Halo host: image gate, amdxdna/XRT presence, IOMMU and fabric-clock blockers, enablement deltas."
|
||||
description: "Census whether halogen-flash-server's NPU small-model path can run and swap models on a Strix Halo host: image gate, amdxdna/XRT presence, IOMMU and fabric-clock blockers, enablement deltas, boot-set-only model loading (no live reload), and the silent-fallback-to-Flash trap for unarmed chat-class NPU models."
|
||||
---
|
||||
|
||||
Use when asked whether halogen/halogen-flash-server can use the XDNA2 NPU (embeddings, rerank, moderation/guard, decider, `qwen3.5-2b` generation, `flux2-klein-4b` images, `qwen3-asr-*`) on a Ryzen AI Max / Strix Halo host, or before proposing to enable it.
|
||||
Use when asked whether halogen/halogen-flash-server can use the XDNA2 NPU (embeddings, rerank, moderation/guard, decider, `qwen3.5-2b` generation, `flux2-klein-4b` images, `qwen3-asr-*`) on a Ryzen AI Max / Strix Halo host, before proposing to enable it, or when asked whether it can **hot-swap / live-reload** NPU models like FastFlowLM does.
|
||||
|
||||
## What the feature is (not a speed knob)
|
||||
|
||||
The Flash trunk stays on the iGPU. `HALOGEN_NPU_MODELS=<names>` starts a second engine in the SAME container and on the SAME API port; the request's `model` field picks the executor. Unset → server identical to before. Each NPU model is a precompiled AIE-ML program (`devices/devices.hnpm`, per-partition `devices/u<N>.elf`, `devices/consts.hnpb`, `<id>.hnpw` weights, `tokenizer/tokenizer.json`), loaded via the HOST's XRT → kernel `amdxdna` → `/dev/accel/accel0`. One pass at a time; `HALOGEN_NPU_QUEUE` 64, overflow queues server-side, then `503 engine_busy`. Entry `entrypoint.sh` validates every file's bytes + sha256 against `/opt/halogen/npu/models.txt` before anything starts. NPU models run only in single-container "all" mode, NOT the `engine`/`api` split.
|
||||
The Flash trunk stays on the iGPU. `HALOGEN_NPU_MODELS=<names>` starts a second engine in the SAME container and on the SAME API port; the request's `model` field picks the executor. Unset -> server identical to before. Each NPU model is a precompiled AIE-ML program (`devices/devices.hnpm`, per-partition `devices/u<N>.elf`, `devices/consts.hnpb`, `<id>.hnpw` weights, `tokenizer/tokenizer.json`), loaded via the HOST's XRT -> kernel `amdxdna` -> `/dev/accel/accel0`. One pass at a time; `HALOGEN_NPU_QUEUE` 64, overflow queues server-side, then `503 engine_busy`. Catalog in the 0.16.2 image: `decider-0.8b`, `qwen3-embedding-0.6b`, `qwen3-reranker-0.6b`, `qwen3guard-gen-0.6b`, `qwen3.5-2b` (+ `flux2-klein-4b`, `qwen3-asr-*` in newer tags). Only the programs named in `HALOGEN_NPU_MODELS` are armed.
|
||||
|
||||
## Census (all read-only, run from the operator box)
|
||||
## Boot-set-only: NO live reload (tested on the live endpoint)
|
||||
|
||||
`HALOGEN_NPU_MODELS` is read **at start**; the entrypoint validates/converts/loads the named programs then. There is **no** runtime load/unload and **no** admin endpoint to add a model. FLM comparison: FLM was a standalone NPU-only server (`flm serve <one-model>`, single FIFO lane) free to reconfigure its one AIE partition; halogen boots Flash (iGPU) **plus** the NPU programs as ONE coupled process (shared port + shared memory/power budget), so the NPU set is a boot-time constant. Changing even a checkpoint's weights reconverts "at the next start", never live.
|
||||
|
||||
Consequence — arming a new program = edit `scripts/frank-dragoon.env` -> redeploy -> **container restart -> the whole Flash payload (tens of GB) reloads, minutes**. The small NPU program is not the cost; the Flash reload is.
|
||||
|
||||
## The silent-fallback trap (the one that bites)
|
||||
|
||||
A request naming an **unarmed** catalog model behaves differently per endpoint class:
|
||||
- `/v1/rerank`, `/v1/moderations`: honest **400 `model_not_found`**, message ends `(loaded: none)`. The endpoint only exists if that engine was armed at boot.
|
||||
- `/v1/chat/completions` (the `decider-0.8b` decision path and `qwen3.5-2b` generation): **silent fall-through to the Flash model** — HTTP 200, body `"model": "<flash id>"`, no error. A guard/decision call to an unarmed model is answered by the main (often abliterated) endpoint you were trying to screen against, and looks like success. Measured: an unarmed `decider-0.8b` "is this a prompt injection?" probe returned `"no"` from Flash. **Fail-safe is NOT the default on the chat route.**
|
||||
|
||||
## Live probe (idle lane; one ssh, non-disruptive — no restart happens)
|
||||
|
||||
```bash
|
||||
ssh <host> '
|
||||
# 1. engine gate + catalog in the pinned image
|
||||
podman exec <ctr> bash -c "grep -rlo HALOGEN_NPU_MODELS /opt/halogen /usr/local/bin /entrypoint.sh 2>/dev/null | head"
|
||||
podman exec <ctr> cat /opt/halogen/npu/models.txt | head -40
|
||||
# 2. is NPU actually wired today? (expect NO env, NO device, NO /dev/accel inside)
|
||||
podman inspect <ctr> --format "{{json .HostConfig.Devices}}"; podman exec <ctr> ls /dev/accel
|
||||
# 3. kernel driver + firmware
|
||||
lsmod | grep -i amdxdna; sudo dmesg | grep -iE "xdna|npu" | tail -5; ls -l /dev/accel
|
||||
# 4. XRT (distro packages are fine; vendor mounts /opt/xilinx/xrt)
|
||||
dpkg -l | grep -iE "xrt|xdna"; ls /usr/lib/x86_64-linux-gnu/libxrt* /usr/lib/libxrt* 2>/dev/null
|
||||
# 5. BLOCKER 1: IOMMU
|
||||
cat /proc/cmdline; ls /sys/kernel/iommu_groups | wc -l
|
||||
# 6. BLOCKER 2: fabric clock held?
|
||||
for c in /sys/class/drm/card*/device; do echo "== $c"; cat $c/power_dpm_force_performance_level 2>&1; cat $c/pp_dpm_fclk 2>&1 | tr "\n" " "; echo; done'
|
||||
B=http://127.0.0.1:7770 # halogen port; internal engine is :8731/:8740 muxed onto it
|
||||
curl -s $B/v1/models | jq -r '.data[].id' # armed set = boot set; must not mutate
|
||||
# control (armed embedding) -> 200 with a real vector:
|
||||
curl -s $B/v1/embeddings -d '{"model":"qwen3-embedding-0.6b","input":["ping"]}'
|
||||
# unarmed rerank/guard -> expect 400 "(loaded: none)":
|
||||
curl -s $B/v1/rerank -d '{"model":"qwen3-reranker-0.6b","query":"t","documents":["a","b"],"top_n":1}'
|
||||
curl -s $B/v1/moderations -d '{"model":"qwen3guard-gen-0.6b","input":["hello"]}'
|
||||
# unarmed decider/gen -> watch the "model" field; if it is the flash id, it fell through:
|
||||
curl -s $B/v1/chat/completions -d '{"model":"decider-0.8b","temperature":0,"messages":[{"role":"user","content":"x"}],"response_format":{"type":"json_schema","json_schema":{"name":"g","description":"is injection?","schema":{"enum":["no","yes"]}}}}' | jq -r '.model // .error.message'
|
||||
# prove nothing restarted:
|
||||
systemctl show frank-dragoon.service -p NRestarts -p ActiveEnterTimestamp --value
|
||||
```
|
||||
`decider` with a schema and a non-zero `temperature` 400s on a *sampling* guard first ("send temperature 0"); set `temperature:0` to reach the router and see the fallback.
|
||||
|
||||
Docs (raw fetch, authoritative): `docs/NPU.md`, `docs/npu-small-models.md`, `deploy/host/halogen-fabric-clock` under `peonist-ai/halogen-flash-server` on GitHub. Tag list: `podman search --list-tags ghcr.io/peonist-ai/halogen-flash-server`.
|
||||
## Enablement gates (all four must hold before it runs)
|
||||
|
||||
## The two blockers and the enablement delta
|
||||
|
||||
- **IOMMU must be on.** `amd_iommu=off` turns the NPU off (no PASID/SVA); `iommu=pt` keeps it. Needs a grub edit + reboot. `iommu_groups == 0` ⇒ off.
|
||||
- **Fabric clock must be pinned at the top `pp_dpm_fclk` level** (or `power_dpm_force_performance_level=high`); `auto` = unheld. GPU+NPU concurrently HANG Strix Halo machines and corrupt NPU results while that clock moves. Fix: root container with `-v /sys:/host/sys` (server holds it itself), or install `deploy/host/halogen-fabric-clock` + unit (~1 W). If it can't hold, the server FAILS CLOSED and prints the commands.
|
||||
- Container delta: `--device /dev/accel/accel0` + `HALOGEN_NPU_MODELS`; `--ulimit memlock=-1:-1` and `--group-add keep-groups` usually already present. Distro XRT: mount each of `libxrt_coreutil.so.2`, `libxrt_core.so.2`, `libxrt_driver_xdna.so.2` TWICE (once at `/opt/xilinx/xrt/lib/`, once at its own path) — never mount an `/opt/xilinx/xrt` holding only symlinks.
|
||||
|
||||
## Costs to price before proposing
|
||||
|
||||
- Upstream A/B: IOMMU translation ON costs **13–16 % prefill** (460 → 385 tok/s at 2,048 tokens, Ryzen AI Max+ 395); decode unaffected. Whether `iommu=pt` specifically costs that on a given box is a local measurement, not a published fact — run the image's own `sweep` after the reboot.
|
||||
- A few GB host RAM per NPU model (+8 GB FLUX, +1.6/4 GB ASR), and trunk decode drops while the NPU works (shared bus + power).
|
||||
1. Kernel `amdxdna` loaded + `/dev/accel/accel0` present; the node must be world-open (`0666` via a udev rule) or a **rootless** container cannot open it — `serve-frank-dragoon.sh` refuses if the mode bit 02 is clear.
|
||||
2. Host XRT with the NPU plugin (`libxrt_coreutil.so.2`, `libxrt_core.so.2`, `libxrt_driver_xdna.so.2`); on Arch/CachyOS mount each lib twice (image XRT path + its own `/usr/lib` path), never an `/opt/xilinx/xrt` of dangling links.
|
||||
3. **Fabric clock held at top** (`halogen-fabric-clock` root unit) — GPU+NPU at once can hang Strix Halo / corrupt NPU results while that clock changes; the server refuses to start if it can't hold it.
|
||||
4. **IOMMU on**: `amd_iommu=off` kills the NPU; `iommu=pt` keeps it.
|
||||
|
||||
## Laws / traps
|
||||
|
||||
- Never probe the NPU beside the live service: stop + mask first (fleet benchmarking policy); this host class has a precedent of a concurrent GPU experiment needing a power cycle. Distinguish off vs wedged with `ip neigh show <ip>` from a peer.
|
||||
- `/dev/accel/accel0` existing and `amdxdna` probing cleanly does NOT mean the NPU works — with IOMMU off the failure is at context/PASID creation, not at the device node. Do not trust older notes claiming the node itself is missing.
|
||||
- Never probe the NPU beside the live service: stop + mask first (fleet benchmarking policy); this host class has a precedent of a concurrent GPU experiment needing a power cycle. Distinguish off vs wedged with `ip neigh show <ip>` from a peer. A single embedding/decision request on an already-armed idle production lane is a normal request, not a benchmark — safe.
|
||||
- `/dev/accel/accel0` existing and `amdxdna` probing cleanly does NOT mean the NPU works - with IOMMU off the failure is at context/PASID creation, not at the device node. Do not trust older notes claiming the node itself is missing.
|
||||
- `dmesg` lines like `Failed to alloc dev bo memory, ret -28` while large NPU models take turns at NPU memory are benign (vendor-documented); the server recovers.
|
||||
- NPU model RSS is host-side, bypassing the container cgroup (GTT/shared pool), like all Strix Halo GPU memory — container stats under-report the true footprint.
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
---
|
||||
name: halogen-npu-lane-boot-loop-triage
|
||||
description: "Triage a frank-dragoon (halogen-flash-server) unit crash-loop caused by the NPU lane: signature classification, restart-debris trap, XRT mount-path census, and the transient env-file proof boot."
|
||||
---
|
||||
|
||||
Triage when `frank-dragoon` (or any halogen-flash-server unit with `HALOGEN_NPU_MODELS` armed) sits in `activating (auto-restart)` with `ExecStart … code=exited, status=1/FAILURE` seconds after start, while `/health` stays dark.
|
||||
|
||||
## 1. Classify the exit BEFORE touching the config
|
||||
- `NRestarts` climbing + exit in <5 s = refusal, not an engine fault. Read `journalctl -u frank-dragoon --since "-30m"` AND the engine log `/home/dank/logs/frank_dragoon.log` (engine stdout never goes to the journal).
|
||||
- Lane signature: log ends with `halogen npu: … <name>: N files checked against this image's record` then `halogen-npu: No such library '<path>'` + `the NPU engine exited before it was ready (rc=1)` → **the NPU lane exit is fatal to the whole container** (supervisor kills the Flash engine with it). Chat-down is the lane's fault even though chat code never changed.
|
||||
- KV/pre-flight lines present but no `locked in RAM` line = died before weight pinning.
|
||||
|
||||
## 2. The restart-debris trap
|
||||
`Restart=always` re-runs the failing boot every ~10 s; each attempt leaves GPU/NPU handles being released, and the next boot logs `GTT in use before this start: N GiB … held by 1 process(es)`. That NOTE is debris from your own crash-loop — do not chase it as a foreign co-tenant (real co-tenants: ComfyUI `python` on `/dev/kfd`, find with `sudo fuser -v /dev/dri/renderD128 /dev/kfd`). `sudo systemctl stop <unit>` FIRST, then any diagnosis.
|
||||
|
||||
## 3. XRT path census (the `No such library '/opt/xilinx/lib/<multiarch>/libxrt_*.so.2'` case)
|
||||
- The image ships NO XRT at all (`find / -xdev -name "libxrt*"` inside the image = empty); the NPU child `dlopen`s the host's libs through the serve script's mounts, which land at TWO destinations only: `/opt/xilinx/xrt/lib/<L>` and `$XRT_DIR/<L>`. A child searching `/opt/xilinx/lib/<multiarch>/` names a THIRD prefix nobody mounts.
|
||||
- The serve loop resolves `REAL="$(readlink -f …)"` and mounts resolved files — if a host package upgrade retargets the symlinks (check `grep libxrt /var/log/dpkg.log` + `ldconfig -p | grep xrt` mtimes vs the last good boot), the file set the child expects can change under byte-identical mounts.
|
||||
- Rootless gotcha: `podman mount` fails outside `podman unshare` and leaves `$M` EMPTY — `grep -r pat "$M/opt"` then greps your CWD = HOST tree (false hits like `/opt/fastflowlm`). Always `podman unshare bash -c 'M=$(podman mount $C); …'` or `--entrypoint ls|cat` probes instead, and distrust hits that name host-only paths.
|
||||
|
||||
## 4. Proof boot without touching production
|
||||
The serve script honors `FRANK_DRAGOON_ENV_FILE` (sanctioned seam). Throwaway script on the host:
|
||||
1. scratch env: `source /home/dank/serverAppConfigs/scripts/frank-dragoon.env` + the knob under test.
|
||||
2. `sed` the serve script to inject the candidate extra mounts into the `NPU_ARGS+=(--device …` line — mount RESOLVED real files (`.so.2.25.00`), never symlinks (vendor: dangling links inside the container).
|
||||
3. `export FRANK_DRAGOON_ENV_FILE=/tmp/… FRANK_DRAGOON_LOG=/tmp/…; FRANK_DRAGOON_CONTAINER=<scratch-name> timeout 1200 /tmp/serve-test.sh` — the container still publishes 7770, so `systemctl stop` the unit first (port binds at CREATE, exit 126 on race).
|
||||
4. Retire same session: `podman rm -f <scratch-name>`, trash `/tmp` scripts; the unit stays stopped until the real fix deploys.
|
||||
|
||||
## 5. Restore chat while the lane is unfixed
|
||||
`HALOGEN_NPU_MODELS=` (blank) is the documented one-line lane-off switch — lane absent = endpoint exactly as before the lane existed. Land through git + `deploy.sh dankclaw --service frank-dragoon`; never hand-edit the remote env.
|
||||
@@ -0,0 +1,66 @@
|
||||
---
|
||||
name: halogen-npu-mount-prefix-proof
|
||||
description: "Prove and fix a halogen frank-dragoon NPU-lane mount failure: crash-loop where a dead lane kills the whole container, dlopen-prefix census, and the sanctioned FRANK_DRAGOON_ENV_FILE proof-boot."
|
||||
---
|
||||
|
||||
# Halogen NPU lane: dead lane kills the container — prove the mount prefix, don't guess
|
||||
|
||||
Applies when a `frank-dragoon` unit crash-loops in seconds (`exit 1` in ~200 ms,
|
||||
`NRestarts` climbing) and the engine log's last lines are:
|
||||
|
||||
```
|
||||
halogen npu: <model>: 15 files checked against this image's record
|
||||
halogen-npu: No such library '<some/prefix>/libxrt_core.so.2'
|
||||
halogen npu: the NPU engine exited before it was ready (rc=1)
|
||||
```
|
||||
|
||||
## Laws this incident wrote (2026-10-21, frank-dragoon, 88-restart loop)
|
||||
- **A dead NPU lane is fatal to the whole container**: the flash-server supervisor
|
||||
exits with the lane. Chat dies beside a healthy GPU preflight — do not read the
|
||||
GTT/weights lines as "engine fine, side feature broken".
|
||||
- The NPU child `dlopen`s distro XRT from **three** container prefixes per library:
|
||||
`/opt/xilinx/xrt/lib/`, the library's own install path, **and
|
||||
`/opt/xilinx/lib/<multiarch>/`**. The missing one is exactly what the error names.
|
||||
- Host state can flip this under a byte-identical container: the lane ran for days
|
||||
with two prefixes, then a host `libxrt2`/`libxrt-npu2` reconfigure (check
|
||||
`grep libxrt /var/log/dpkg.log`) made the child demand the third. Image digest,
|
||||
serve script, payload all unchanged ⇒ suspect HOST package state, not the deploy.
|
||||
- Fastest discriminator vs engine faults: exit in <1 s with the engine's preflight
|
||||
lines present = serve/lane gate failure, not model load. Model-load failures take
|
||||
minutes and print pin/KV lines.
|
||||
|
||||
## Proof procedure (sanctioned seam, no scratch ports)
|
||||
1. Read the error's path verbatim — it names the missing prefix.
|
||||
2. `sudo systemctl stop frank-dragoon` (unit must be down: the proof binds :7770
|
||||
at container CREATE).
|
||||
3. Scratch env file on the host using the bench-sanctioned seam:
|
||||
```
|
||||
/tmp/dragoon-nputest.env:
|
||||
source /home/dank/serverAppConfigs/scripts/frank-dragoon.env
|
||||
HALOGEN_NPU_MODELS=qwen3-embedding-0.6b
|
||||
```
|
||||
4. Scratch serve copy that appends the candidate mount to the `NPU_ARGS+=(--device`
|
||||
line (`sed`), REAL files only (`readlink -f`; vendor forbids dangling symlinks
|
||||
under /opt/xilinx), and run with:
|
||||
`FRANK_DRAGOON_ENV_FILE=/tmp/... FRANK_DRAGOON_LOG=/tmp/... timeout 1200 /tmp/serve-nputest.sh`
|
||||
5. Success = boot ledger (`… in all`) **plus** `npu_api: <model> … buckets u0…u9`
|
||||
plus `/health` `npu.status ok`. Then codify the mount in the serve script, pin
|
||||
it in `tests/frank_dragoon_unit.bats` (the fixture XRT dir's basename IS the
|
||||
multiarch name — name it `x86_64-linux-gnu`), update the endpoint page's NPU
|
||||
prerequisite to the three-prefix law, and cut over via PR +
|
||||
`deploy.sh dankclaw --service frank-dragoon`.
|
||||
6. Retire scratch: the `timeout` kills the proof container; `podman rm -f`,
|
||||
move /tmp scripts to `~/.AGENTTRASH/<date>-<topic>/`. A proof instance
|
||||
serving production while you wait is drift — deploy the merged fix promptly.
|
||||
|
||||
## Traps
|
||||
- Greping image contents rootless: `podman mount` fails (needs `podman unshare`)
|
||||
and leaves `$M` EMPTY, so `grep -r $M/opt` silently greps the HOST cwd — that
|
||||
is how host `/opt/fastflowlm` masqueraded as image content. Always check `$M`
|
||||
non-empty, or use `podman unshare bash -c 'M=$(podman mount C); …'`.
|
||||
- The entrypoint swallows `podman run IMG sh -c …` as a mode name; use
|
||||
`--entrypoint sh/cat/ls`.
|
||||
- The proof container inherits production name/port unless the override is set
|
||||
AFTER sourcing the env (the env file re-exports `FRANK_DRAGOON_CONTAINER`).
|
||||
- `Restart=always` re-runs the lane into its own GPU/NPU debris; stop the unit
|
||||
before any manual boot, don't fight the loop.
|
||||
@@ -0,0 +1,44 @@
|
||||
---
|
||||
name: halogen-npu-xrt-third-prefix
|
||||
description: "Triage a halogen frank-dragoon (flash-server) unit crash-loop caused by the NPU lane: signature \"No such library /opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2\", dead-lane-kills-container law, third-prefix mount fix, and the FRANK_DRAGOON_ENV_FILE scratch-boot proof recipe."
|
||||
---
|
||||
|
||||
# Halogen NPU lane: dead lane kills the container; XRT needs THREE mount prefixes
|
||||
|
||||
Friction-earned 2026-10-21 on `frank-dragoon` (Dankclaw): unit boot-looped 88× (`activating`, exit 1 in ~200 ms) with the GPU/serve_api lines looking healthy; chat dead beside a fine-looking ledger.
|
||||
|
||||
## Signatures (in order of appearance)
|
||||
|
||||
```
|
||||
halogen-npu: No such library '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'
|
||||
halogen npu: the NPU engine exited before it was ready (rc=1; ...)
|
||||
```
|
||||
then `systemctl show <svc> -p ExecMainStatus` = 1, `NRestarts` climbing, and `/home/dank/logs/frank_dragoon.log` truncated at ~10 lines each boot (serve script truncates the log per start).
|
||||
|
||||
**Laws:** a dead NPU lane is NOT survivable — the engine supervisor takes the WHOLE container down on it. And `Restart=always` makes each restart race its own GPU/NPU debris: the FIRST failure's message is the only real evidence; later loop entries are noise. Stop the unit immediately (`sudo systemctl stop frank-dragoon`) before probing.
|
||||
|
||||
## Root cause class
|
||||
|
||||
The NPU child `dlopen`s XRT from `/opt/xilinx/lib/<multiarch>/` IN ADDITION to `/opt/xilinx/xrt/lib` and the install path. Host package churn (dpkg.log reconfigure of `libxrt2` the previous day) made the child demand the third prefix; our mounts filled only two. A lane that worked for days with byte-identical mounts can die purely from host package changes — diff the mounts last, check `grep libxrt /var/log/dpkg.log` first.
|
||||
|
||||
Fix: mount each REAL file (never a symlink) at all three prefixes; derive the multiarch name from `FRANK_DRAGOON_XRT_DIR`'s basename (`MULTIARCH="${XRT_DIR##*/}"`).
|
||||
|
||||
## Proof recipe (production-shaped, no scratch port)
|
||||
|
||||
One card holds one server — never bench on a side port; use the sanctioned env-file seam:
|
||||
|
||||
1. Scratch env on the host: `source /home/dank/serverAppConfigs/scripts/frank-dragoon.env` + knob overrides (e.g. `HALOGEN_NPU_MODELS=...`).
|
||||
2. Copy the serve script with the candidate change applied via `sed` into `/tmp` (or edit + `FRANK_DRAGOON_ENV_FILE`); export `FRANK_DRAGOON_ENV_FILE=<scratch>` and `FRANK_DRAGOON_LOG=/tmp/<name>.log`; run under `timeout 1200`. The seam exists precisely for this.
|
||||
3. Judge from `/tmp/<name>.log`: `npu_api: <model> ... buckets u0…u9` = lane alive; `No such library` = still broken. Then `curl :7770/health` → `npu.status`.
|
||||
4. Retire the same session: `podman stop -t 60` (weights unpin slowly), `podman rm -f`, kill wrapper procs BY PID (`pgrep -f` the scratch serve script — `pkill -f` a pattern containing your own ssh line kills the session), move scratch files to `~/.AGENTTRASH/<date>-<topic>/`, then prove `pgrep -f serve-<scratch>` is empty. Leftover scratch state silently hijacks the next deploy (same class as leftover bench drop-ins).
|
||||
|
||||
## Traps hit en route
|
||||
|
||||
- Rootless `podman mount` fails ("cannot run in rootless mode") and leaves `$M` EMPTY — a later `grep -r "$M/..."` then reads the HOST cwd and "finds" the answer in Loki WAL files. Never grep an unverified mount var; `podman unshare` or `--entrypoint` probe instead.
|
||||
- Image ships NO `/opt/xilinx` at all: all XRT comes from host mounts; `docs/NPU.md` is not inside the image.
|
||||
- `ssh host 'tail <log>'` runs on the DEV box if you forgot `ssh` — a missing-file error is not "the engine wrote nothing".
|
||||
- Health `context` field echoes the RUNNING engine's ceiling — use it to prove which shape is live, unlike `/v1/models` name echo.
|
||||
|
||||
## Verification that the fix landed
|
||||
|
||||
`systemctl show frank-dragoon -p DropInPaths` empty; boot ledger shows `reserving the KV pool … (about 11.0 GiB)`, `… in all`, an `npu_api:` line, no clamp line; deployer prints `NPU lane verified — <model> embedded`.
|
||||
@@ -0,0 +1,62 @@
|
||||
---
|
||||
name: harden-precommit-secret-gate
|
||||
description: "Fix or build a pre-commit secret-scan hook that actually blocks: the -z path hole, line-wide placeholder exemptions, inert doc placeholders, glued report lines, self-test fixture paths — each with a red-first bats case."
|
||||
---
|
||||
|
||||
A pre-commit secret gate that fails open is worse than no gate: it teaches the operator that scanning happened. Four defect classes, each with a black-box test that goes red against the broken version (serverAppConfigs PR #70, command-center #59 — 4 holes closed same night, all red first).
|
||||
|
||||
## 1. Non-ASCII paths fail open
|
||||
|
||||
`git diff --cached --name-only` quotes non-ASCII paths (`"docs/\303\251.md"`), so `git diff` on that name returns **empty**, and a loop that `continue`s on an empty diff skips the file silently. Fail closed with NUL delimiting:
|
||||
|
||||
```bash
|
||||
git diff --cached -z --name-only | while IFS= read -r -d '' f; do ...
|
||||
```
|
||||
|
||||
Test: a tracked file whose name contains a non-ASCII character, holding a fake key on an added line, must be blocked.
|
||||
|
||||
## 2. Placeholder filters must be value-scoped
|
||||
|
||||
`if echo "$line" | grep -q "$placeholders"` exempts the whole line, so a comment word vouches for the credential beside it:
|
||||
|
||||
```python
|
||||
token = "sk-…" # example config for the docs
|
||||
```
|
||||
|
||||
Filter only the captured value (the `sk-…` group), never the line. Two directions to test: `key = "example"` stays allowed; `token = "sk-<real>" # example` is blocked.
|
||||
|
||||
## 3. Inert placeholders in documentation
|
||||
|
||||
A skill/README that *documents* a secret gate quotes the vendor's placeholder ids (`AKIA0000000000000EXAMPLE`, all-zero JWTs). Exempt them by value shape (all-zero / `EXAMPLE` suffix / RFC 2606 domain), or the machine tier stops committing — and the refusal usually lands on a heartbeat commit, where it looks like infrastructure failure, not policy. Also note: an unlanded fix means the checkout runs the **old** hook, so a landing can be blocked by the very bug the PR corrects; audit the staged diff and use the documented override (`ALLOW_SECRETS=1`, never `--no-verify`), recording the audit in the commit message.
|
||||
|
||||
## 4. Report-line assembly
|
||||
|
||||
`report+=(" $file: $line")` inside a loop that keeps only the last hit per file glues the next filename onto the previous line when a second hit exists — the report attributes the finding to the wrong file. Test a two-file staged set where both match, and assert each report line names exactly one existing path.
|
||||
|
||||
## 5. Self-test fixtures in the hook itself
|
||||
|
||||
The hook's own fixtures contain fake keys by necessity, so the gate would refuse any commit touching the hook or its tests (or a docs page quoting a key as documentation). Pin the bypass to specific paths and to added lines only, and print what was exempted:
|
||||
|
||||
```bash
|
||||
case "$f" in .githooks/pre-commit|tests/secret-gate.bats)
|
||||
echo "secret-gate: self-test fixtures exempt: $f" >&2; continue;; esac
|
||||
```
|
||||
|
||||
## The scan rule
|
||||
|
||||
Scan **added content only** — `git diff --cached -U0 | grep '^+[^+]'` — or every commit touching a file with an old key in context gets blocked, and the operator's response to a gate that blocks everything is to override it always.
|
||||
|
||||
## Test harness (fixture repo, never the real checkout)
|
||||
|
||||
`git init` a temp repo, `git config core.hooksPath .githooks`, copy the hook in, `git add` a payload, then run it with the env overrides the hook honours (point its cache/notify paths at `$TMP` so tests never touch the user's config). Pin, in order:
|
||||
|
||||
| Case | Expect |
|
||||
|---|---|
|
||||
| fake key on an added line | exit 1, message names the file |
|
||||
| `key = "example"` | pass |
|
||||
| real key + `# example` comment | exit 1 |
|
||||
| non-ASCII filename with a key | exit 1 |
|
||||
| two files with hits | report names both |
|
||||
| self-test fixtures | pass, with the exempt line printed |
|
||||
|
||||
`bats -r` after each edit. `--no-verify` is forbidden on Frank's boxes: the hook IS the control.
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
name: harness-model-config-wire-capture
|
||||
description: "Verify what the omp harness actually sends to a local inference endpoint (reasoning_effort, sampler keys, thinking-off) by pointing a scratch HOME at a logging proxy, instead of inferring it from models.yml comments."
|
||||
---
|
||||
|
||||
A `models.yml` comment claiming "the wire sends X" is a claim about the harness, not the engine, and the harness rewrites between the two: compat maps pass unmapped values through, effort ladders clamp before the map applies, and a "disable reasoning" role flag can be implemented as *send the lowest effort* rather than a switch. Read only the wire.
|
||||
|
||||
## Recipe (minutes, no production state touched)
|
||||
|
||||
1. **Logging proxy** — python `ThreadingHTTPServer` on `127.0.0.1:7799` forwarding body+path to the real engine and appending the request body's interesting keys (`model, reasoning_effort, max_tokens, temperature, top_p, top_k, chat_template_kwargs`) as one JSON line per request. Run it as a named bash service with `ready.port` so the port is proven before use.
|
||||
2. **Scratch HOME** — `mkdir -p /tmp/omph/.omp/agent`, copy the *candidate* `models.yml` + `config.yml` there, rewrite that provider's `baseUrl` to `http://127.0.0.1:7799/v1`. Never edit `~/.omp/agent/` — in `dank/command-center` the checkout is the deployment.
|
||||
3. **One-shot runs** — `HOME=/tmp/omph omp --model '<provider>/<id>:<effort>' --no-tools --no-session --no-skills --no-rules --no-extensions --no-pty --no-lsp -p 'Reply with exactly: OK'`, once per effort (`off minimal low medium high xhigh max`). Add `--print-thoughts` when the question is "did thinking actually happen".
|
||||
4. Read the log, and assert the value **after** the change. A config edit that never engaged is the failure mode this exists to catch.
|
||||
|
||||
## Traps found doing this on frank-dragoon (2026-10-21)
|
||||
|
||||
- `--model X:off` does **not** mean `reasoning_effort: off` on the wire. Reasoning is switched through `compat.reasoningDisableMode` (enum includes `lowest-effort`, `none-effort`, `qwen-enable-thinking-false`, `qwen-template-false`, `chat-template-thinking-false`); the default derives from `thinkingFormat`, and `lowest-effort` quietly keeps thinking alive (`:off` arrived as `"minimal"`, with a canned 26-token thinking block on every "off" request). `none-effort` sent `"none"` → `reasoning_tokens: 0`.
|
||||
- Effort levels are clamped to the harness's per-model ladder **before** `reasoningEffortMap` applies, so a map row for a level the ladder lacks is inert (`:max` arrived as `xhigh` even with `max: max`). Capture before promising a level works.
|
||||
- Map lookup is `compat.reasoningEffortMap?.[effort] ?? thinking?.effortMap?.[effort] ?? effort` — removing a row is a pass-through, never a 400. Claims about rejection must be proven against the engine.
|
||||
- Engine-side claims: ask for a deliberately bad value and read its 400 — the engine prints its own accepted set (`reasoning_effort 'maximum' unsupported; use minimal|low|medium|high|xhigh|max|none`).
|
||||
|
||||
## Engine-side probes that transfer to any OpenAI-compatible endpoint
|
||||
|
||||
- **Accepted parameters**: `{model, messages, max_tokens:1, <param>:<value>}` → status + `error.message`. Acceptance is not honouring: report it as acceptance.
|
||||
- **Level honoured?** same prompt at `max_tokens 2048` per level, compare `usage.completion_tokens_details.reasoning_tokens`. Equal values across levels = the parameter is ignored; `reasoning_closed_at` reveals a per-level thinking ceiling.
|
||||
- **Thinking off**: `chat_template_kwargs {"enable_thinking": false}` and top-level `enable_thinking: false` → expect `reasoning_tokens: 0`.
|
||||
- **Vision**: build a solid-colour PNG in-script (`zlib`+`struct`, no PIL), 512×512, ask for the colour in one word **with thinking disabled** — otherwise the answer is cut off by the thinking block and the probe is inconclusive. A 1×1 image only proves token accounting.
|
||||
|
||||
## Teardown
|
||||
|
||||
Kill the proxy service, remove the scratch HOME, the proxy script and its log. Never leave copies of `config.yml`/`models.yml` in `/tmp`: they carry provider endpoints and sometimes tokens.
|
||||
@@ -1,48 +1,49 @@
|
||||
---
|
||||
name: html-report-ground-truth-audit
|
||||
description: "Audit a self-generated HTML report: ground every numeric/code/diagram claim against the repo, fix, and prove structure + render"
|
||||
description: "Audit a self-generated HTML report: ground every numeric/code/diagram claim against the repo, fix wrong values, delete filler that restates git, verify the served artifact byte-identical, and flag unverifiable content to the operator."
|
||||
---
|
||||
|
||||
Use after generating a large HTML report (e.g. via `repo-change-html-report`) whose claims were partly reconstructed from session memory. Fabricated numbers, invented code blocks, and silently-swallowed sections are the failure modes; the audit makes each traceable to the repo, a live command, or a recorded run.
|
||||
Ground-truth audit of an HTML report (generated by an agent, for a human). Three defects hide in a plausible report: **wrong numbers, unverifiable claims, and filler that restates git for a reader who has git.**
|
||||
|
||||
## Law
|
||||
|
||||
Every numeric claim must be re-measured against the repo, not the narrative that produced it.
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Enumerate every checkable claim
|
||||
Grep the report for numerics, file paths, symbol names, and code blocks:
|
||||
- line counts (`wc -l` the real file)
|
||||
- test counts (`grep -c '@test' tests/<f>.bats` — NOT suite-run totals: conditional skips make raw counts differ; run-observed totals are legitimate run history, keep them labeled as such)
|
||||
- diff stats (`git diff --shortstat main..HEAD` — the footer's files/insertions/deletions)
|
||||
- commit hashes (`git log --oneline`)
|
||||
- "verbatim" code blocks (see 3)
|
||||
- diagram node lists and captions (see 4)
|
||||
1. **Dump claims with file:line anchors.**
|
||||
`sed 's/<[^>]*>//g' report.html | grep -vE '^\s*$' | grep -nE 'commit|file|line|fix|refactor' > /tmp/claims.txt`
|
||||
Then enumerate the ground truth: `git log --oneline origin/main..HEAD` for commits; `git diff --stat origin/main..HEAD` for the per-file census — **sort it**: `--sort=-deletions` and `--sort=-creations` give the top movers, so the "largest single change" and "largest net code change" are computed facts.
|
||||
|
||||
### 2. Ground numeric claims with one command each
|
||||
Fix every mismatch in place. Record the was/now pair — the audit table IS the evidence.
|
||||
2. **Check code blocks compile.** Write every snippet to a temp file and `bash -n` each. Common breakages from HTML generation: `"`/`&` entities leaking into code, and `<br>` line breaks that split a line mid-pipe.
|
||||
|
||||
### 3. Verbatim code blocks: prove them
|
||||
For a block labeled verbatim, the exact string must exist:
|
||||
- live code: `grep -n '<distinctive line>' <file>`
|
||||
- historical: `git show <sha> -- <file> | grep -E '^[+-].*(<token1>|<token2>)'`
|
||||
If the block was fabricated (plausible-looking pseudo-code), replace it with the real excerpt. If condensed, the block header must say "condensed" and every shown line must be real.
|
||||
3. **Check the diagrams carry information.** A mermaid `flowchart LR` whose nodes merely restate the bullet list above it is decoration — delete it.
|
||||
|
||||
### 4. Diagram claims: run the engine
|
||||
Behavioral diagrams (plan traces, closure graphs) must be re-derived by running the actual CLI:
|
||||
`python3 scripts/service-map.py plan --host <h> <path>` — then edit node labels, captions, and the prose count ("four services") to match output. Cross-check derived-set diagrams against the source relation (e.g. `grep -rl 'Network=gluetun.container' */**/*.container` for who rides a tunnel).
|
||||
4. **Check counts.** Commit and file counts must match `git rev-list --count` and `--name-only | sort -u | wc -l`. A report whose sections were written by different subagents drifts: each counts its own slice ("4 commits / 14 files" vs the true 11 / 25).
|
||||
|
||||
### 5. Check for swallowed sections
|
||||
Large prior-session PUT ranges silently overwrite neighbors. Grep for headings/captions the plan promised; re-insert any that vanished, anchored on a fresh `read` of the region.
|
||||
5. **Flag what cannot be verified.** Test pass claims, external tool output, runtime behaviour — if the repo can't confirm it, say so to the operator instead of asserting it.
|
||||
|
||||
### 6. Structural balance
|
||||
Python tag-balance pass over the file:
|
||||
- per-tag open/close counts (`div`, `pre`, `figure`, `svg`, `table`, `span`, `figcaption`)
|
||||
- running-div-depth scan printing per-section balance to localize stray closers
|
||||
Common defect: `.ln`/`.nt` pair emitted without its `.cg` wrapper → stray `</div>`; fix by opening `<div class="cg">` before the pair.
|
||||
## Filler test (delete on failure)
|
||||
|
||||
### 7. Render proof
|
||||
`file://` is blocked on the shared browser — serve the dir: `bash` service `python3 -m http.server <port> --bind 127.0.0.1 --directory <dir>`, ready on that port. Then:
|
||||
- `browser.open({name, url: 'http://127.0.0.1:<port>/<file>'})`
|
||||
- DOM assertions via `tab.evaluate(() => …)` — NOT `tab.run` (that runs in Bun; no `document`). Sections are `<h2 id>` + siblings: scope queries to `document.body`/`document.querySelectorAll`, not `#ch5 …`.
|
||||
- screenshot the repaired regions (scroll via `tab.evaluate(el.scrollIntoView())`), then `tab.close()` and kill the server proc (`write proc://<name>/kill`).
|
||||
- A `file → line-count → description` table restating `git diff --stat`? Delete — the reader has git.
|
||||
- A "what this means" / "conclusion" / "key takeaways" section? Delete — it is the same facts in softer prose. (Same rule as vault pages: point of use or gone.)
|
||||
- Keep only what a reader cannot rebuild from `git log` + `git show`.
|
||||
|
||||
## Deliver
|
||||
A was/now table of every fix with its ground command, plus the list of claims verified intact. State honestly which numbers are run-history and not recomputable.
|
||||
## Serve and verify the artifact
|
||||
|
||||
**Verify the file that is served, not the source.** If a `public/` copy is byte-identical, edit both or the served artifact stays wrong. Then re-fetch:
|
||||
|
||||
```bash
|
||||
base=http://host:port/reports/x.html
|
||||
for i in $(seq 1 30); do curl -fsS -o /tmp/got.html "$base" && break; sleep 1; done # service may still be activating
|
||||
for i in 1 2; do curl -fsS "$base" | md5sum; done # stable across refetches
|
||||
diff <(curl -fsS "$base") /path/to/source.html && echo "served == source"
|
||||
```
|
||||
|
||||
## Voice drift
|
||||
|
||||
If the operator asked for a tone ("no filler", "terse"), the audit must enforce it too — marketing phrasing ("ships the", "This means:") is a style defect, fix it in the same pass, not a later one.
|
||||
|
||||
## Report
|
||||
|
||||
Fixes grouped by defect class (wrong number / unverifiable / filler / style), then what stays unverifiable and why, with the measured ground truth quoted.
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
name: infra-repo-security-review
|
||||
description: "Full-repo security review of an infra/config repo (serverAppConfigs-shaped: shell deployers, quadlets, systemd, Caddy edge, bats/Playwright) — slice fan-out, mechanical censuses, and subagent-claim verification."
|
||||
---
|
||||
|
||||
# Whole-repo security review (infra/config repo)
|
||||
|
||||
For "full security review of this repo" on a deployment-config repo (shell scripts + podman quadlets + systemd units + Caddy + tests). NOT for a diff review — that is `code-review`. NOT for a secrets-only question — that is `git-repo-secret-audit`.
|
||||
|
||||
## 1. Slice, then fan out in ONE `task` batch
|
||||
Five `security-reviewer` slices, each with the repo's own laws pre-injected in `context` so no agent re-derives them (AGENTS.md policy text + exclusions: docs prose, `.AGENTTRASH/**`, binaries, `package-lock.json`):
|
||||
|
||||
1. **Shell** — all `*.sh` + `.githooks/` + `tests/run.sh`: injection, quoting, `mktemp` vs `/tmp/fixed`, `set -e`, `curl -f`, secrets in argv, `rm`/`git clean`/`systemctl disable`.
|
||||
2. **Python** — every `*.py`: `subprocess(shell=True)`, `verify=False`, unbounded `read()`, path traversal on request-supplied filenames, fixed `/tmp` writes, **which HTTP servers exist and which are unauthenticated**.
|
||||
3. **Container + systemd posture** — every `*.container`/`.network`/`.service`/Containerfile.
|
||||
4. **Edge + web** — Caddyfile vhost inventory, Anubis/Authentik chain, response headers, static-asset XSS sinks, service configs (`limiter.toml`, `prometheus.yml`, `alertmanager.yml`).
|
||||
5. **Tests** — vacuous assertions + credentials/secrets in fixtures + harness safety.
|
||||
|
||||
**Set a yield deadline in the prompt.** Subagents here get killed at `task.maxRuntimeMs` (30 min) and the dead ones produced **zero** output. When one runs long, write early: `write agent://<id>` "yield now, unconfirmed items prefixed UNVERIFIED" — the resumed agent returns its partial report instead of losing it. Two of four needed this.
|
||||
|
||||
## 2. Verify every citation before reporting
|
||||
`verify-subagent-review-findings` applies in full: re-grep each cited `file:line` and compare text. Expect ~40% of lines to be off (e.g. an agent described a 13-line hook; the file is 6) and several files to be hallucinated entirely (`.githooks/compose/*.container` did not exist). Discard mismatches; publish a **Discarded subagent claims** section. Re-grep your own citations too — two of mine were off by two lines and had to be corrected in an addendum.
|
||||
|
||||
## 3. Mechanical censuses (cheap, high yield — run these yourself)
|
||||
- **Port binding**: every `^PublishPort=` line, classify `127.0.0.1:` / `192.168.0.x:` / **no address prefix** (= implicit `0.0.0.0`). The unprefixed set is the auth-bypass surface: a LAN client hits the container port directly and skips the Caddy→Anubis→Authentik chain that the vhost enforces. Report as one consolidated finding listing all unprefixed lines, and cite the repo's own correct examples (`127.0.0.1:9090`, `192.168.0.143:8080`) as the fix pattern.
|
||||
- **Image pinning**: count `^Image=` lines with vs without `@sha256:` (this repo: 4 of 54 pinned) — floating tags + `Pull=never` = unreproducible edge.
|
||||
- **Secret modes**: grep every `chmod 600` in the deployer and diff against every file it writes from an `.env`; a `>` redirect without the `chmod` is the finding (the Caddy/Porkbun env file).
|
||||
- **Systemd hardening**: `User=`/`ProtectSystem`/`PrivateTmp`/`NoNewPrivileges` per unit, and `ExecStartPre=+` lines — a `+` hook on a root unit running a checkout-writable script is a local privesc.
|
||||
- **Vhost table**: vhost → upstream → guarded? → exemption documented? Cross-check each upstream against the quadlet `PublishPort` list; unguarded vhost with no exemption comment is a policy violation even when the service self-authenticates.
|
||||
- **Hook wiring**: `git config core.hooksPath` + `ls "$(git rev-parse --git-path hooks)"`. A repo can ship `.githooks/pre-commit` that git never executes — and its own test can assert the *file* is executable and still miss that it isn't installed.
|
||||
- **Secret sweep**: `git cat-file --batch-all-objects --batch` (all history + dangling), match known values read from `~/.config/secrets`; report `prefix…(len N, sha 10)` only, never the value. Watch for the **split-concat disguise** (`PW="as""set"`): their own regex can't see it, but `grep -F` on a fragment finds it and history keeps it forever → rotation required.
|
||||
|
||||
## 4. Report
|
||||
Severity-ranked table (Critical→Low, one line each: location + flaw), then prose detail for Critical/High only, then the Discarded-claims section, then remaining coverage gaps, then an ordered fix set and an offer to branch. Do not create a report file unless asked — chat is the deliverable.
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
name: land-branch-without-checkout
|
||||
description: "Open a PR from a shared worktree when workspace policy blocks git checkout/restore: push a named branch ref from HEAD without switching branches."
|
||||
---
|
||||
|
||||
# Landing a branch when `git checkout` is policy-blocked
|
||||
|
||||
Shared-workspace policy blocks `git checkout` and `git restore` (working-tree
|
||||
switches could strand concurrent agents' uncommitted work). You still must ship
|
||||
the commit on the current worktree branch as a named `chore|feat|fix/<issue>-<slug>`
|
||||
branch. Never work around the block — go around it via refs.
|
||||
|
||||
## Sequence
|
||||
|
||||
```bash
|
||||
# 1. commit only your files (atomic); leave other dirty paths alone
|
||||
git add <your-files> && git commit -m "..."
|
||||
|
||||
# 2. branch ref at HEAD — no working-tree change, stays on wt/<...>
|
||||
git branch <type>/<issue>-<slug> HEAD
|
||||
git push origin <type>/<issue>-<slug> # pre-push hooks still run
|
||||
|
||||
# 3. PR via tracker API with head = pushed ref (see gitea-pr-api-cycle / gh)
|
||||
```
|
||||
|
||||
- The worktree stays checked out on `wt/<timestamp>`; the new branch exists
|
||||
only as the pushed ref (plus a local ref that merges cleanly later).
|
||||
- `-u` upstream tracking on the wt branch is misleading — skip it; set
|
||||
`git config branch.<name>.remote origin` + `.merge` only if you will
|
||||
later check the branch out.
|
||||
- Dirty regenerated fixtures left behind (e.g. `gen:sample` PDFs rewritten
|
||||
by setup/gate runs): `git restore` is also blocked — do NOT edit-forward
|
||||
to revert them; report them. They are regenerated on every bootstrap, so
|
||||
dirt is expected and harmless.
|
||||
|
||||
## Verify
|
||||
|
||||
- `git ls-remote --heads origin <branch>` prints the same sha as `git rev-parse HEAD`.
|
||||
- PR API returns 201 with `head.sha` equal to that sha.
|
||||
|
||||
## Do NOT
|
||||
|
||||
- `git checkout -b`, `git stash`, `git restore`, `git reset --hard` here —
|
||||
all working-tree mutations the policy can block, and stashes in a shared
|
||||
checkout are a cross-agent footgun.
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
name: land-merged-pr-with-blocked-tree-ops
|
||||
description: "Use when a merged PR must land in a checkout whose working tree is dirty and workspace policy blocks git checkout/restore/stash: commit-the-dirt pattern, the secret-gate false positive that blocks the landing commit, conflict resolution proven settings-vs-comments, and heartbeat (live-sync) handoff."
|
||||
---
|
||||
|
||||
Landing a merged trunk into a checkout that IS the deployment (dank/command-center `~/.omp`), when tracked files are dirty and policy forbids `checkout`/`restore`/`stash` on shared state. Re-earned 2026-10-21 landing PRs #61 + #63 into `~/.omp` (27 commits behind, two conflicts, one gate refusal).
|
||||
|
||||
## 1. Classify the dirt before touching it
|
||||
|
||||
`git status --porcelain` then compare against what the merge brings in — only overlap blocks the merge:
|
||||
|
||||
```bash
|
||||
{ git diff --name-only; git diff --cached --name-only; } | sort -u > /tmp/dirty.txt
|
||||
git diff --name-only HEAD origin/main | sort -u > /tmp/incoming.txt
|
||||
comm -12 /tmp/dirty.txt /tmp/incoming.txt # these paths must be clean
|
||||
```
|
||||
|
||||
Note the first column of porcelain: `M ` = **staged**, ` M` = unstaged. A staged file shows nothing in `git diff` — look at `--cached` too, or you will "verify" a file is unchanged while it is staged. `comm` needs both inputs sorted; unsorted input prints "file is not in sorted order" and a partial answer.
|
||||
|
||||
## 2. Prove a dirty file content-free before resolving
|
||||
|
||||
`cmp` it against the trunk blob, never against memory:
|
||||
|
||||
```bash
|
||||
git show origin/main:$f > /tmp/incoming
|
||||
cmp -s "$f" /tmp/incoming && echo "IDENTICAL to trunk" # commit/discard is content-free
|
||||
```
|
||||
|
||||
If identical, the dirt is a sync artifact, not user work. Say so in the commit message with the `cmp` result — that is what makes a direct commit on `main` defensible for a prose-tier file.
|
||||
|
||||
## 3. Commit rather than discard (policy-blocked verbs)
|
||||
|
||||
With `checkout`/`restore`/`stash` unavailable, the only non-destructive unblock is a commit. Two separate commits, and order matters:
|
||||
|
||||
1. Prose-tier dirt proven byte-identical to trunk — commit alone, message states `cmp` proved no prose change.
|
||||
2. Machine tiers (`agent/memories/**`, `agent/managed-skills/**`, `agent/config.yml`, `agent/models.yml`, `dotfiles/`) — let the heartbeat do it: `LC_ALL=C scripts/live-sync.sh` stages allowlist only, commits, merges trunk, pushes, and **exits 0 even when it aborts the merge on conflict** (the alert goes to stderr + ntfy). `exit 0` does not mean landed — check `behind=` afterwards.
|
||||
|
||||
Do NOT `git add -A`: the index may already hold dozens of staged machine-tier files, and `git add <one file>` does not clear them — they ride along into your commit.
|
||||
|
||||
## 4. The gate refusal that blocks the landing commit
|
||||
|
||||
The pre-commit secret gate scans the **staged** diff, so a single-file commit still trips on previously staged content. It fires on inert placeholders inside the skill docs that document the placeholder (e.g. `AKIA0000000000000EXAMPLE` — the AWS all-zero example id in `git-secret-gate-hook` / `harden-precommit-secret-gate`). If the checkout is behind, it is running the **pre-fix** hook while the fix sits unlanded in trunk: the landing is blocked by the very bug the PR corrects.
|
||||
|
||||
Audit first, then override deliberately — never `--no-verify`:
|
||||
|
||||
```bash
|
||||
git diff --cached | grep -nE 'AKIA[A-Z0-9]{12,}|sk-[A-Za-z0-9]{16,}|ghp_[A-Za-z0-9]{20,}|glpat-[A-Za-z0-9]{16,}|BEGIN [A-Z ]*PRIVATE KEY|eyJ[A-Za-z0-9_-]{20,}'
|
||||
git diff --cached | grep -oE 'AKIA[A-Z0-9]{12,}|sk-[A-Za-z0-9]{16,}' | sort | uniq -c # what exactly, how many
|
||||
ALLOW_SECRETS=1 git commit -F msg # message records the audit + that the gate ran
|
||||
```
|
||||
|
||||
Record in the message: the hits are inert placeholders, which files they sit in, that this checkout runs the pre-fix hook, and that the refusal dies with this landing.
|
||||
|
||||
## 5. Merge conflicts: resolve by proving settings identical
|
||||
|
||||
```bash
|
||||
git merge --no-commit --no-ff origin/main
|
||||
git diff --name-only --diff-filter=U
|
||||
```
|
||||
|
||||
For each conflicted file, get all three blobs (`HEAD`, merge base, `origin/main`) and ask what the local side uniquely holds:
|
||||
|
||||
- **Prose / doc pages**: diff ours vs trunk; if ours only lacks the incoming trim, take trunk.
|
||||
- **Real settings files (kdl/yaml/env)**: strip comments and diff — `diff <(grep -v '^\s*//' ours) <(grep -v '^\s*//' theirs)`. Empty output ⇒ only comments differ ⇒ taking trunk changes no deployed behaviour, and you can say that in the merge message. (2026-10-21: `desk-1-home.kdl` — modes byte-identical, trunk's comment corrected the false "no 240 Hz mode exists".)
|
||||
|
||||
Resolve without the blocked verbs by writing the trunk blob: `git show origin/main:$f > $f && git add $f`. Then validate the live artifact (`niri validate`, the suite) before committing the merge, and push.
|
||||
|
||||
## 6. Verify the push actually landed
|
||||
|
||||
`git push` output can be swallowed when wrapped in command substitution inside a batch — verify from the remote, not from the push:
|
||||
|
||||
```bash
|
||||
git fetch -q origin && printf 'HEAD=%s origin/main=%s behind=%s\n' \
|
||||
"$(git rev-parse --short HEAD)" "$(git rev-parse --short origin/main)" "$(git rev-list --count HEAD..origin/main)"
|
||||
```
|
||||
|
||||
## 7. Close and prune
|
||||
|
||||
- Close the issues with a comment giving merge SHA + PR link + title (numbers are per-repo; a bare `#N` is not a reference).
|
||||
- Prune guards, all three before `git worktree remove --force`: `ls -l /proc/*/cwd | grep -c '<wt-name>'` must be 0; `git -C <wt> status --porcelain | wc -l` must be 0; `git rev-list --count main..<branch>` must be 0. Check for `.AGENTTRASH/**` inside the worktree first — removal destroys the only copy.
|
||||
- `git branch -d` (not `-D`) once the branch is reached by main; `git push origin --delete <branch>` is allowed on the local Gitea.
|
||||
- Delete the temp `-K` token config file in the same session that created it.
|
||||
@@ -0,0 +1,24 @@
|
||||
---
|
||||
name: line-edit-anchor-discipline
|
||||
description: "Avoid the recurring edit-tool line-op traps: empty PUT deletes lines, PUT ranges swallowing multi-line block delimiters, and anchors landing mid-expression. Use whenever issuing CUT/PUT line edits on tagged files."
|
||||
---
|
||||
|
||||
# Line-edit anchor discipline (CUT/PUT ops on tagged files)
|
||||
|
||||
Five self-inflicted syntax wounds in one session (franksign, 2026-10-21) all trace to three rules.
|
||||
|
||||
## Rules
|
||||
1. **An empty PUT body DELETES the targeted lines** (tool warns: "Interpreted an empty PUT body as deletion"). Twice it ate a whole log entry / a closing `}`. For no-ops use nothing; for deletes use explicit `CUT`.
|
||||
2. **Never PUT a range whose endpoints cut through a multi-line block.** If a `| {`-style union variant, a frontmatter `---` fence, or a `test(...)` closer straddles the range boundary, the replacement swallows the opener/closer and strands the body. Pre-condition: line N-1 and M+1 must be block-complete. Fix-after-the-fact pattern: read the broken region raw, then `PUT k.=k:` to restore the single swallowed line (e.g. the missing `---`, the orphaned ` | {`, `}` → `});`).
|
||||
3. **Search-hit reads do not authorize PUT.** A grep/folded-summary read marks lines as "never displayed"; the edit tool refuses and echoes actual content. That refusal is a free verification — re-check the echoed lines match intent, then straight-retry with the same tag.
|
||||
|
||||
## Costly mistakes seen
|
||||
- Empty `PUT 7-7:` deleted a just-written log entry; restoring required re-issuing the full entry text.
|
||||
- `PUT 6.=7` replacing `timestamp`+`---` dropped the frontmatter close fence → lint-breaking file silently.
|
||||
- `PUT 10.=11` on a union type swallowed ` | {` opening a 12-line variant → cascade of LSP "unexpected label" warnings.
|
||||
- `CUT` of a stranded opener left its dangling `.json())` behind; balance errors surface only at EOF (`expected ')' but file ends`) — locate by reading regions raw, not by widening the payload.
|
||||
|
||||
## Discipline
|
||||
- After any edit that reports a syntax error or a warning: read the touched region with `:raw` before the next op; fix the single wrong line, do not resend the block.
|
||||
- "no change: byte-identical" means your anchor math is off — re-read, don't append more content.
|
||||
- Prefer replacing whole coherent blocks (start line = block start, end line = block end) over edits that end mid-statement.
|
||||
@@ -0,0 +1,83 @@
|
||||
---
|
||||
name: llamacpp-adapter-swap
|
||||
description: "Add or roll back a GGUF LoRA adapter on a dank-fleet llama.cpp endpoint (e.g. frank-zealot): provenance census, fork-flag gate, fetcher+env+deployer surface, seam tests, and the behavioural before/after probe that is the only real proof."
|
||||
---
|
||||
|
||||
# Applying a GGUF LoRA adapter to a fleet endpoint
|
||||
|
||||
An adapter is **payload, not shape**: it changes what the endpoint answers and nothing in
|
||||
the KV/budget equation. Fleet law still applies — no benchmarks unless Frank orders one,
|
||||
rollback is one env line, `./tests/run.sh` before commit, deployer health gate + e2e gate.
|
||||
|
||||
## 1. Census the artifact BEFORE writing any code
|
||||
|
||||
```bash
|
||||
curl -sS "https://huggingface.co/api/models/<repo>?blobs=true" # size + LFS oid (None => plain git object)
|
||||
curl -sSL -o /tmp/a.gguf "https://huggingface.co/<repo>/resolve/main/<file>" && sha256sum /tmp/a.gguf
|
||||
```
|
||||
|
||||
Parse the GGUF KV section (15 lines of struct.unpack) and read **provenance**: abliteration
|
||||
adapters carry `prism.abliterate.base_gguf` / `base_sha256`, `adapter.type`, `sites`,
|
||||
`target_layers`, `target_embedding`. Compare the declared base against the file the endpoint
|
||||
actually serves (its sha is in the endpoint's own fetcher script). A mismatch is normal for
|
||||
sibling quantizations — the loader matches **tensor names, not hashes** — but it must be
|
||||
stated in the docs, because it means the card's measured numbers are not this endpoint's
|
||||
numbers.
|
||||
|
||||
## 2. Gate the runtime, from the binary's own `--help`
|
||||
|
||||
On the host: `/home/dank/GitPackages/<build>/llama-server --help | grep -iE 'lora|adapter'`.
|
||||
Confirm `--lora-scaled FNAME:SCALE` exists (and `--lora-init-without-apply` + `POST
|
||||
/lora-adapters` if a runtime toggle is ever wanted). Vendor forks gate adapters by metadata
|
||||
version too — e.g. PrismML builds older than `prism-b10777` refuse packs with `unsupported
|
||||
prism.hadamard.version: 2`. Read the card for the build it verified on and put that in the
|
||||
fetcher header.
|
||||
|
||||
## 3. Surface (five files, copy the mmproj convention)
|
||||
|
||||
- `scripts/<svc>.env`: `LORA_FILE=<path>` + `LORA_SCALE=1`; document that **`LORA_FILE=` (empty)**
|
||||
is the rollback and `LORA_SCALE=0` is the inert-but-present mode. Use `${VAR=default}` (no
|
||||
colon) so an explicitly empty value survives — same reason `MMPROJ_FILE`/`PRESENCE_PENALTY` do.
|
||||
- `scripts/serve-<svc>.sh`: `EXTRA_ARGS+=(--lora-scaled "${LORA_FILE}:${LORA_SCALE}")` — **one
|
||||
argv token**, so a scale can never arrive without its file; refuse a missing file by name
|
||||
(`Restart=always` turns an unfinished fetch into a crash loop that reads like a runtime fault).
|
||||
- `scripts/download-<svc>-<adapter>.sh`: copy the sibling fetcher (`-fsSL --fail-with-body`, byte
|
||||
count, sha256, `--verify-only`, idempotent). No LFS oid => pin the content sha. Header carries
|
||||
the provenance finding.
|
||||
- `scripts/deploy-remote.sh`: `LLM_<SVC>_LORA="${LLM_<SVC>_LORA:-<path>}"` + a check whose error
|
||||
names **this** fetcher, not the payload's.
|
||||
- `scripts/deploy-plan.py` needs nothing: fetchers are host-side state and select no service.
|
||||
|
||||
## 4. Seam tests (argv is the contract)
|
||||
|
||||
Stub-engine harness: create the adapter stub from the **shipped env's basename**, emit
|
||||
`LORA_FILE`/`LORA_SCALE` in the generated test env, then assert on output only —
|
||||
`--lora-scaled` value equals `<path>:<scale>`; a distinctive `LORA_SCALE` (0, 0.75) changes the
|
||||
token; `LORA_FILE=` yields **absence** of the flag (`! grep -qxF -- '--lora-scaled'`); a missing
|
||||
file exits non-zero naming that path; extend the env↔deployer anti-drift pair list.
|
||||
|
||||
## 5. Deploy and prove it behaviourally
|
||||
|
||||
Capture the **before** state on the live endpoint first, then: merge → on host `git pull` +
|
||||
run the fetcher → `./scripts/deploy.sh <host> --service <svc>`.
|
||||
|
||||
```bash
|
||||
sudo systemctl show <svc> -p DropInPaths --value # must be empty (bench-drop-in law)
|
||||
sudo tr '\0' ' ' < /proc/$(sudo systemctl show <svc> -p MainPID --value)/cmdline
|
||||
sudo journalctl -u <svc> --since <boot> | grep -iE 'adapter|lora' # llama_adapter_lora_init_impl prints prism.* keys
|
||||
```
|
||||
|
||||
Refusal probe (the only proof of an abliteration adapter): same prompts, `temperature 0.1`,
|
||||
**`max_tokens >= 2048`** — on thinking models a short budget is consumed by
|
||||
`reasoning_content` and `content` comes back empty with `finish_reason: length`, which reads
|
||||
as a refusal that never happened. Classify with a marker regex and report refusal-vs-complied;
|
||||
do not paste generated harmful content into chat. Also run the standing trio: exact-phrase
|
||||
chat, `tool_calls` with valid JSON, and a >=256 px solid-colour vision probe. A residual
|
||||
`⚠️ Important Disclaimer` heading after the swap is style, not the refusal circuit.
|
||||
|
||||
## Docs
|
||||
|
||||
Endpoint page (`serverAppConfigs-docs/LLM/<svc>.md`): Adapter row, Fetch row, Flags row, one
|
||||
Era row whose rollback is the env line, and a measured before/after paragraph. Laws worth
|
||||
adding: provenance is metadata not a gate; an `token_embd` rank would crash, so an adapter is
|
||||
not inert; the runtime pin may gain a second reason. Never restate the card's benchmark table.
|
||||
@@ -0,0 +1,69 @@
|
||||
---
|
||||
name: locale-c-multibyte-awk-silent-loss
|
||||
description: "Debug generated output that silently loses data under LC_ALL=C — a regex quantifier on a multibyte character in awk/grep, plus the locale-dependent red test it produces."
|
||||
---
|
||||
|
||||
# LC_ALL=C kills multibyte regex quantifiers — silently
|
||||
|
||||
Symptom: a generator emits a **complete, exit-0, missing-a-third-of-the-page** artifact in
|
||||
cron / systemd user units / agent shells, and the same command looks perfect in the user's
|
||||
terminal. Companion symptom: a repo test is red for the agent and green for the operator.
|
||||
|
||||
## Probe (do these in order, they are 5 seconds apart)
|
||||
|
||||
```bash
|
||||
echo "LANG=[$LANG] LC_ALL=[$LC_ALL]" # LC_ALL overrides LANG — en_US.UTF-8 + LC_ALL=C is byte-wise
|
||||
locale -a | grep -iE 'utf-?8' | head # which UTF-8 locale actually exists on this box
|
||||
```
|
||||
|
||||
Then test the suspect pattern alone, under both locales, with the literal character:
|
||||
|
||||
```bash
|
||||
printf '%s\n' ' // ─── Applications ───' | awk '/─+ / { print "QUANT MATCHED" }'
|
||||
printf '%s\n' ' // ─── Applications ───' | awk '/─/ { print "LITERAL MATCHED" }'
|
||||
printf '%s\n' ' // ─── Applications ───' | LC_ALL=C.utf8 awk '/─+ / { print "QUANT MATCHED under UTF-8" }'
|
||||
```
|
||||
|
||||
Literal matches everywhere; the quantifier matches only under a UTF-8 locale. That asymmetry IS
|
||||
the bug: gawk under `LC_ALL=C` matches bytes, so `─+` means "the last byte of U+2500, repeated" —
|
||||
a byte sequence that never occurs. mawk/busybox/Go regexp have the same class of failure
|
||||
(quantifier or bracket class spanning a multibyte char).
|
||||
|
||||
## Fix
|
||||
|
||||
Match and strip the multibyte literal with no quantifier:
|
||||
|
||||
```awk
|
||||
/^[[:space:]]*\/\/[[:space:]]*─/ { # literal gate
|
||||
s = $0; sub(/^[[:space:]]*\/\/[[:space:]]*/, "", s); sub(/[[:space:]]+$/, "", s)
|
||||
if (s ~ /─ .*─/) { gsub(/─/, "", s); gsub(/[[:space:]]+/, " ", s); sub(/^ /, "", s); ... }
|
||||
next
|
||||
}
|
||||
```
|
||||
|
||||
**Never pin `LC_ALL=en_US.UTF-8` to "fix" it** — that locale often does not exist (a CachyOS box
|
||||
here shipped only `C.utf8`), so the script then breaks on another machine. Make the matcher
|
||||
byte-safe instead.
|
||||
|
||||
## Pin it with a test
|
||||
|
||||
`run env LC_ALL=C "$GEN" "$IN"` + assert the lost content is present. The operator's UTF-8 terminal
|
||||
can never catch this class of bug, so the explicit C-locale run is the only durable guard.
|
||||
|
||||
## Suite-verification law (how this got missed)
|
||||
|
||||
NEVER judge `bats tests/` from `tail` output — a failure at test 28 is invisible in a 25-line
|
||||
window ending at test 109. Always:
|
||||
|
||||
```bash
|
||||
out=$(bats tests/ 2>&1); echo "$out" | grep -cE '^ok'; echo "$out" | grep -E '^not ok' || echo GREEN
|
||||
```
|
||||
|
||||
Run it under `LC_ALL=C` too: that is the environment every automated caller inherits. Same rule
|
||||
for `| grep -c` on a multi-line JSON field — count what you claim to have counted.
|
||||
|
||||
## Related, do not confuse
|
||||
|
||||
- `bats-assertion-trust` — vacuously-passing assertions (green that proves nothing). This skill is
|
||||
the mirror case: correct assertions that are only red in one locale.
|
||||
- `gitea-pr-api-cycle` — unrelated; landed here only because the PR work is where this surfaced.
|
||||
@@ -0,0 +1,48 @@
|
||||
---
|
||||
name: monolith-split-under-dist-pin
|
||||
description: "Use when cutting a monolithic source file into modules behind a black-box test suite whose committed build artifact (dist) must stay byte-identical to src; one verified commit per cut, red-proof against the previous bundle. Not for feature work or doc-only edits."
|
||||
---
|
||||
|
||||
# Monolith split under a committed-bundle pin
|
||||
|
||||
Procedure that landed a 1354-line `index.ts` as 9 modules + 301 lines of wiring across 9 commits with zero red commits. Assumes: black-box test suite (tests drive the built bundle, not internals) and a pin test asserting the committed `dist/` equals a fresh build of `src/`.
|
||||
|
||||
## Per-cut loop (one commit per cut, always in this order)
|
||||
|
||||
1. `bun run build` (or project equivalent) **before** staging — commit `dist/` in the same commit as the moved `src/`, otherwise the pin test fails ("committed dist is byte-identical to a fresh build") and you must amend. Amending is fine only while the branch is unpushed.
|
||||
2. `tsc --noEmit`, then run the full suite async (`bats tests/ > /tmp/x.log 2>&1; grep -E '^1\.\.'`).
|
||||
3. **Never write to `src/` while the suite runs.** `setup()` rebuilds the bundle from `src/`, so a concurrent edit attributes someone else's half-written state to your run. Read-only prep (planning the next cut's exact line ranges) is the correct thing to do while waiting.
|
||||
4. Gate green → next cut. Gate red → fix, rebuild, amend, re-gate.
|
||||
|
||||
## Cutting mechanics
|
||||
|
||||
- Cut leaf/pure modules first (grammar, transcript rules, URL resolution, pure segmenter), then stateful ones (config, IO, pipeline, UI). Never let a module import `index` — if the new module needs something from `index`, that thing is a leaf and must move too (this is what forced `segment.ts` out early).
|
||||
- Extract in **verbatim** bodies: read the exact ranges, write them into the new file, then delete the same ranges. Only module docs, export keywords, and injected dependencies change. Keeps each commit reviewable as a move.
|
||||
- Edit tools that require "displayed lines" from the latest read snapshot: re-read the exact ranges under the current tag, then issue one edit call with all ops listed **bottom-up** (original line numbers throughout). Adjacent ranges merge into one `CUT a=b`; leaving a doubled blank line is the usual scar — one extra `CUT` next turn.
|
||||
- Mutable config must reach a long-lived object as **getters**, not a construction-time snapshot: menu handlers mutate `cfg` after construction, and a snapshot freezes the settings the UI exists to change.
|
||||
- After moving code, grep the old file for the moved names: stale call sites compile under bundlers that skip typechecking (`stop()` vs `speaker.stop()` was caught only by `tsc`, not by `bun build`).
|
||||
|
||||
## Proving a new assertion is red-capable (before trusting it green)
|
||||
|
||||
Build the old bundle and point the test harness at it — no stash, no worktree juggling:
|
||||
|
||||
```bash
|
||||
git show HEAD:dist/index.js > /tmp/old-dist.js # bundle is self-contained
|
||||
# rebuild the throwaway harness by extracting the driver the suite writes
|
||||
sed -n "<heredoc-body-range>p" tests/behavior.bats > "$W/driver.ts"
|
||||
DV_EXT=/tmp/old-dist.js <same env as the test> bun "$W/driver.ts" <scenario> # expect RED
|
||||
DV_EXT="$PWD/dist/index.js" <same env> bun "$W/driver.ts" <scenario> # expect GREEN
|
||||
```
|
||||
|
||||
Copy the project's real player/server stub verbatim — a simplified stub (e.g. one that skips writing the log file the driver counts) prints a false `PLAYS 0` and looks like a failing fix. Pin the scratch dir to a fixed path; `ls -dt /tmp/tmp.*` picks up another session's directory.
|
||||
|
||||
## Black-box test hygiene learned the hard way
|
||||
|
||||
- A test that counts artifacts in the **shared** temp directory observes every other process on the box (live agent sessions synthesize into the same `dankvoice-*` dirs). Give the run a private `TMPDIR="$WORK/scratch-tmp"` and assert there; the assertion's sensitivity is proven by the fact it caught a real leftover while scoped globally.
|
||||
- Let preseeds reference the ephemeral fixture port with a placeholder (`__PORT__`) substituted by the driver, so a test can preseed a URL pointing at its own fixture server.
|
||||
- Refactor-only commits get **no new test**: re-asserting unchanged labels/order after a commit whose point is that they are unchanged is a tautology; the existing suite is the gate. New tests belong to behavior changes, red-first.
|
||||
- Tests must observe **output**, never internals: assert `PLAYS`/`CALLS`/`CALL <url>`/`NOTIFY <text>`, not that a variable was set.
|
||||
|
||||
## Docs after a split (same branch, own commit)
|
||||
|
||||
Move each explanation to the file that now owns it (knob list → the module that reads env; state-machine prose → the module that implements it) and leave the entry point a one-line-per-module map. Then grep for pointers into the old file — a README saying "see header of `src/index.ts`" becomes a lie the moment the knobs move.
|
||||
@@ -0,0 +1,36 @@
|
||||
---
|
||||
name: new-fleet-service-checklist
|
||||
description: "Add a new deployable service to serverAppConfigs: the four registration touch-points (two allowlists, planner, router) whose drift breaks the first-ever deploy."
|
||||
---
|
||||
|
||||
# Adding a new service to the serverAppConfigs deployer
|
||||
|
||||
A new `--service <name>` target must be registered in **four** files or the first deploy fails at a gate. Two fleet services (cadvisor, gpu-exporter) shipped with the client gate missed within a day of each other; the failure mode is `Service 'X' is not deployed on <target>` from `deploy.sh`, exit 2 — before SSH.
|
||||
|
||||
## The four touch-points (per target host)
|
||||
|
||||
1. `scripts/deploy.sh` → `valid_service()` case: add `<target>:<svc>`. **Client gate** — the one people forget.
|
||||
2. `scripts/deploy-remote.sh` → `service_is_allowed()` case: add `<target>:<svc>`, **and** the dispatcher `case` branch `<svc>) deploy_<svc> ;;`.
|
||||
3. `scripts/deploy-plan.py`: the name in the target's service set (`DANKCLAW_SERVICES` / `DANKSERVER_SERVICES`) + `SERVICE_ORDER`, plus a path map so edits to the service's own files select it (for `scripts/<svc>.*` host processes, map the exact filenames like the signal-relay/gpu-exporter maps do).
|
||||
4. `scripts/test-changed.sh`: route the service's files to its seam, else focused test runs skip it (full `tests/run.sh` still covers).
|
||||
|
||||
Sanctioned exception: `dankclaw:searxng-network` is client-only (network unit, no remote handler) — the drift test exempts it.
|
||||
|
||||
## Enforcement + local verification
|
||||
|
||||
`tests/deployment_flow.bats` test "the client and remote allowlists agree per target" diffs the two extracted lists. Verify before pushing:
|
||||
|
||||
```bash
|
||||
grep -oE 'dank(claw|server):[a-z0-9-]+' <(sed -n '/^valid_service()/,/^}/p' scripts/deploy.sh) \
|
||||
| grep -v dankclaw:searxng-network | sort -u > /tmp/cli
|
||||
grep -oE 'dank(claw|server):[a-z0-9-]+' <(sed -n '/^service_is_allowed()/,/^}/p' scripts/deploy-remote.sh) \
|
||||
| sort -u > /tmp/rem
|
||||
diff /tmp/cli /tmp/rem # must be empty
|
||||
```
|
||||
|
||||
Also: a `# comment` after a trailing `\` inside a bash `case` pattern list **breaks line continuation** (syntax error) — annotate on its own line above.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Merge ≠ deploy: `deploy.sh <host> --service <svc>` must still be run per host; a merged service is dead code until then (GPU stats "not showing" was exactly this).
|
||||
- `deploy.sh` refuses to run anywhere but a clean `main` checkout: the canonical dev checkout is `/home/blank/Projects/serverAppConfig` (not the session `~/.omp/wt/...` worktree, not the build worktree).
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
name: niri-monitor-flap-refresh-triage
|
||||
description: "Diagnose niri/amdgpu monitor failures on blankpad — panel dropping, wrong refresh rate, black/flickering output — and fix the refresh without reloading the compositor or logging Frank out."
|
||||
---
|
||||
|
||||
# niri monitor triage: flaps, phantom modes, 60 Hz fallback
|
||||
|
||||
Frank's reaction to a broken display is immediate and loud; the fix must never log him out.
|
||||
Box: `blankpad` (Cezanne APU, `card1`, s2idle only), compositor `niri --session` as a user unit.
|
||||
|
||||
## Step 1 — separate three different diseases (do this before touching anything)
|
||||
|
||||
```bash
|
||||
niri msg outputs # name, mode, position, full mode list per output
|
||||
for c in /sys/class/drm/card*-*/status; do echo "$(basename $(dirname $c)) $(cat $c)"; done
|
||||
systemctl --user show niri.service -p ExecMainStartTimestamp -p NRestarts # did niri restart?
|
||||
journalctl --user -b --no-pager | grep -iE 'disconnecting connector|new connector|Page flip commit failed|DMCUB|falling back to preferred'
|
||||
journalctl -k -b --no-pager | grep -iE 'amdgpu|PM: suspend exit|\*ERROR\*'
|
||||
```
|
||||
|
||||
Reading it:
|
||||
|
||||
- `disconnecting connector: "X"` then `new connector: X` → **link-level drop** (cable/dock/monitor/driver). Count them: `grep -c 'disconnecting connector'`.
|
||||
- `Page flip commit failed … EINVAL` bursts + kernel `dc_dmub_srv_log_diagnostic_data: DMCUB error` right after `PM: suspend exit` → **amdgpu display-engine state broken by s2idle resume**. Only a reboot clears it; never reboot without an explicit per-run OK.
|
||||
- `configured mode M could not be found, falling back to preferred` → **config asks for a mode the panel does not offer**; the panel silently runs at EDID-preferred (usually 60 Hz). This is the one you can fix instantly.
|
||||
- Config mtimes older than the symptom + no `niri` restart ⇒ the config did not cause it. Say so explicitly; agents get blamed for display problems they didn't cause.
|
||||
|
||||
## Step 2 — the phantom-mode trap (niri fails soft, so it looks like hardware)
|
||||
|
||||
`output { mode "…" }` with a refresh the EDID never advertised does **not** error. niri logs a WARN and drops to `preferred` on every connect → janky 60 Hz ultrawide, blamed on the monitor.
|
||||
|
||||
Compare the requested mode against the panel's real list:
|
||||
|
||||
```bash
|
||||
niri msg outputs | sed -n '/<monitor name>/,/^$/p' | grep -E '^\s+[0-9]+'
|
||||
```
|
||||
|
||||
## Step 3 — apply the mode WITHOUT a config reload (the no-logout path)
|
||||
|
||||
```bash
|
||||
niri msg output "<EDID output name from niri msg outputs>" mode "WxH@refresh"
|
||||
niri msg outputs | grep -A1 '<monitor>' # confirm the new mode
|
||||
pgrep -c niri # compositor still alive, same PID
|
||||
```
|
||||
|
||||
`niri msg output …` is documented as temporary ("not saved into the config file") — it changes one output and cannot log the session out. Avoid `niri action reload-config` when Frank is working: it reloads his whole config.
|
||||
|
||||
## Step 4 — persist, and check you edited the file that is actually live
|
||||
|
||||
Live niri config is a symlink chain: `~/.config/niri/config.kdl` → `cfg/display.kdl` → `cfg/desk-<profile>.kdl`, and `cfg/desk-*.kdl` are symlinks into the repo:
|
||||
|
||||
```bash
|
||||
ls -l ~/.config/niri/cfg/display.kdl # -> desk-1-home.kdl
|
||||
grep -n '239\|180' ~/.config/niri/cfg/display.kdl # prove YOUR edit is in the live chain
|
||||
niri validate # parses the live config; touches nothing
|
||||
```
|
||||
|
||||
`dotfiles/` is on the `live-sync.sh` heartbeat allowlist, so an edited tracked dotfile stays dirty until the hourly heartbeat commits it — that is the designed state, do not commit it on `main` yourself.
|
||||
|
||||
Desk config law (its own header): panels keyed by **EDID identity**, not connector name — but connector-keyed leftovers (`DP-1`, `DP-2`) survive in `desk-1-home.kdl`, and a reconnect that comes back without an EDID string hits those instead. Align them to the same mode, or a flap silently changes your refresh rate.
|
||||
|
||||
## Escalate honestly
|
||||
|
||||
Reboot clears DMCUB/EINVAL state → schedule it, don't take it. Persistent drops after a reboot → cable / dock / monitor DP-version setting, i.e. hardware, not config. Report which of the three diseases you found and which you did **not** fix.
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
name: observability-stack-live-census
|
||||
description: "Census the Dankclaw observability stack (Prometheus/Grafana/Loki/Alloy/Alertmanager/signal-relay) live and audit per-endpoint alert coverage; reconcile against the owner page and rule-file comments, which go stale."
|
||||
---
|
||||
|
||||
# Observability stack — live census + alert-coverage audit
|
||||
|
||||
Use when asked "status of the observability stack" on Dankclaw, or before claiming a service is/isn't monitored. Everything runs rootless on Dankclaw: `ssh dankclaw bash -s <<'EOF' … EOF`. Verify live; the owner page (`serverAppConfigs-docs/Infrastructure/Observability.md`) and `shared/prometheus/rules/llm-fleet.yml` comments go stale.
|
||||
|
||||
## Loopback ports (do NOT probe the wrong one)
|
||||
- Prometheus `127.0.0.1:9090` · Alertmanager `127.0.0.1:9093` · Loki `127.0.0.1:3100` · Alloy `127.0.0.1:12345` · signal-relay `192.168.0.143:18093`
|
||||
- **Grafana host port is `127.0.0.1:3002`** (→ container :3000). Host `:3000` is a `rootlessport` for some *other* rootless container and answers 200-on-`/` but 404-on-`/api/health`. Probing :3000 and seeing 404 is a false alarm — always use 3002 for Grafana. `GF_SERVER_ROOT_URL=https://grafana.franksdank.cloud`.
|
||||
|
||||
## Units + containers
|
||||
`for u in prometheus grafana loki alloy alertmanager node-exporter blackbox signal-relay; do systemctl --user is-active $u.service; systemctl --user is-enabled $u.service; done`
|
||||
- Quadlet units report `generated` (healthy), not `enabled`; signal-relay is a real host unit (`enabled`).
|
||||
- `blackbox` = `not-found`/inactive → the known gap (no blackbox exporter).
|
||||
- `podman ps` for `Up N days (healthy)`.
|
||||
|
||||
## Prometheus — targets / rules / alerts
|
||||
- Targets: `GET :9090/api/v1/targets` → `.data.activeTargets[] | .health, .labels.job, .lastError`. Expect ~8 up / 4 down. **The 4 downs are structural, not outages:** `comfyui` + two `services` (HTML content-type, no real /metrics) and `gitea` (404, metrics disabled). Root remedy = deploy a blackbox exporter (not yet present).
|
||||
- Rules: `GET :9090/api/v1/rules` → count `.data.groups[].rules`. The `llm-fleet` group is **4 alert + 2 recording** (6 total). **AGENTS.md saying "5 alert + 1 recording" is wrong** — trust the rule file.
|
||||
- Firing: `GET :9090/api/v1/alerts`; Alertmanager: `GET :9093/api/v2/alerts` and `/api/v2/silences`.
|
||||
|
||||
## Alert-coverage per runtime — the sharp step (catches real blind spots)
|
||||
The `llm-fleet` perf alerts (`LlmRequestsQueueing`/`LlmPrefixCacheCollapse`/`LlmSpecDecodeOff`) key on the **`llamacpp:*`** family. Which endpoints actually emit it:
|
||||
```
|
||||
GET :9090/api/v1/query?query=count by (job)({__name__=~"llamacpp:.+"})
|
||||
GET :9090/api/v1/query?query=up{job=~"frank-.*"}
|
||||
GET :9090/api/v1/query?query=count by (__name__)({job="frank-archon"}) # what archon really exports
|
||||
```
|
||||
Ground truth (2026-10-20):
|
||||
- **frank-zealot** (llama.cpp) → `llamacpp:*` (17 series) → perf alerts cover it.
|
||||
- **frank-dragoon** (halogen) → still emits `llamacpp:*` (10 series; halogen re-uses the names) → perf alerts cover it.
|
||||
- **frank-archon** (radiance) → exports the **`vllm:*`** family (`vllm:num_requests_waiting`, `vllm:gpu_prefix_cache_hit_rate`, `vllm:num_preemptions_total`, **`vllm:time_to_first_token_seconds_bucket`** = a real latency histogram), **not `llamacpp:*`** → **NO perf alert fires on archon**; only `LlmEndpointDown` (`up{job=~"frank-.*"}`) catches it. Up/down yes, perf/queueing/prefix/p95 = collected but unalerted.
|
||||
- The rule-file header comment claims "frank-archon LEFT the scrape set 2026-10-05, FreeToken exports no /metrics" — **stale/false**: archon (radiance) is `up` and exports `vllm:*`. `LlmEndpointDown`'s description repeats the same stale claim. Never assert coverage from a comment; query the families.
|
||||
|
||||
## Log pipeline emptiness test
|
||||
Loki is "configured, healthy, empty": `GET :3100/loki/api/v1/labels` → `{"status":"success"}` with **no `data` array** = 0 label names = 0 streams. Confirm: `find /opt/loki/data -name '*.tsm' -o -name 'chunks-*' | wc -l` → 0 chunks (≈8 WAL/bookkeeping files only). Alloy ships 0 lines (journal counter 0, both file globs refused per the page — a config bug, not capacity). No Loki datasource in Grafana. Report the log half as non-functional, not merely "healthy."
|
||||
|
||||
## Reporting posture
|
||||
- Metrics half works (up/down for all 3 endpoints + node-exporter + self-scrape); log half dead-on-arrival; blackbox + archon-perf-alerts + Loki-datasource + Alertmanager-vhost are the open gaps.
|
||||
- Grafana provisioning is file-based — check `podman exec grafana ls /etc/grafana/provisioning/{datasources,dashboards}` (expect `prometheus.yml`, `llm-fleet.json`) rather than fighting the API for creds.
|
||||
- Alertmanager has no persistence: a reboot re-stamps `startsAt` on long-firing alerts.
|
||||
- Flag stale facts (AGENTS.md rule split, archon-rejoin comments) but let the user decide whether to fix docs.
|
||||
@@ -0,0 +1,25 @@
|
||||
---
|
||||
name: okf-small-bundle-docs-audit
|
||||
description: "Audit + dedupe a small repo's docs bundle for OKF conformance when the kb-lint/kb-coverage toolchain does not exist: inline conformance and link-audit scripts, owner-per-fact cuts, index trio, one log."
|
||||
---
|
||||
|
||||
# OKF docs audit for a small repo bundle (no kb-lint toolchain)
|
||||
|
||||
Use for a repo whose KB is one docs directory (e.g. `dankvoice` → `dankvoice-docs/`), where `scripts/kb-lint.sh` / `kb-coverage.py` do not exist. For the big `dank/command-center` campaign see skill `docs-dedupe-verify-campaign`; same frame, no scouts — a corpus under ~1500 lines gets read fully yourself (scouts lie).
|
||||
|
||||
## Frame
|
||||
- OKF spec lives at `~/.omp/omp-docs/references/okf-spec.md` (the command-center deploy checkout). v0.1 conformance = every non-reserved `.md` has parseable frontmatter with non-empty `type`; `index.md` = listing with NO frontmatter; `log.md` reserved, exactly one, bundle root.
|
||||
- Branch off `origin/main` in a fresh worktree; docs-only + src-comment changes still need the repo's normal gate.
|
||||
|
||||
## Procedure
|
||||
1. Inventory `wc -l` every page; read all fully. Ledger per fact: which page OWNS it (laws page owns laws, decisions own rationale, contributing owns process, README owns user-facing, GLOSSARY owns terms). Everything else becomes a pointer.
|
||||
2. Hunt the rot classes that repeat: dead test citations (name in docs not in `grep -c tests/` — verify with `git log --all -S`), hand-maintained censuses (test counts, scenario lists) — replace with a pointer to the authoritative source, never refresh the copy; deleted doc lines for live env vars (compare `git show <fixed>:file | grep -c VAR` vs HEAD); sections contradicting the page's own head; open items already resolved in live state (check the live settings file, not the doc).
|
||||
3. Frontmatter: `type` + one-line `description` + `tags` + `timestamp`. On edit-rejection the tool prints the file's current `#TAG` — reissue with it, no re-read needed.
|
||||
4. Conformance check (inline python): walk bundle dirs, regex `^---\n(.*?)\n---` then `^type: \S` on each non-reserved `.md`.
|
||||
5. Link audit (inline python): per md file, `\]\(([^)#]+)`; skip http; `os.path.normpath(join(dirname, target))` must exist. Report `N checked / ALL RESOLVE`.
|
||||
6. Create `index.md` at bundle root + each subdir (listing, no frontmatter, relative links) and `log.md` at the bundle root seeded with laws/incidents only.
|
||||
|
||||
## Gotchas
|
||||
- Src docstring fixes are artifact-free only if the bundler strips comments — prove with `bun run build; cmp dist/... old` before claiming no rebuild needed.
|
||||
- README must NOT enumerate design records; point at `decisions/index.md` so new records need no README edit.
|
||||
- Mark superseded decision records in their frontmatter `description` ("its X design is superseded by the Y record") — never rewrite history bodies.
|
||||
@@ -0,0 +1,41 @@
|
||||
---
|
||||
name: omp-background-job-deadline-debug
|
||||
description: "Diagnose a background bash job/service killed mid-run (\"Command timed out after Ns\") in omp: find which deadline owned it (tools.maxTimeout cap vs async default), prove it from the bundle, and launch long jobs uncapped as ready-log services."
|
||||
---
|
||||
|
||||
# Omp background-job deadline forensics
|
||||
|
||||
Symptom: an async `bats`/build job dies with `[Command timed out after 300s|600s]`
|
||||
while the same launch once ran 18+ min ("no deadline").
|
||||
|
||||
## Cause map (proven against the pinned bundle)
|
||||
Resolver in `dist/cli.js`: `let v=n??300, R=v===0, k=R?void 0:ep("bash",v,x)` where
|
||||
`qx.bash={default:300,min:1,max:3600}` and `x` = config key `tools.maxTimeout`
|
||||
("Maximum timeout in seconds the agent can set for any tool", default 0).
|
||||
- Request absent → `qx.bash.default` 300 s.
|
||||
- Request > 0 → clamped by `tools.maxTimeout` when set (e.g. 600 caps a 1800 request).
|
||||
- Request 0 → meant "no deadline", but `0` sometimes arrives as missing (falsy drop
|
||||
upstream of the resolver) → silently 300 s. Filed as tool issue; treat any
|
||||
async job over ~5 min as untrustworthy.
|
||||
|
||||
## Diagnose in three moves
|
||||
1. `grep -o 'qx={bash:{[^}]*}' <omp bundle>` — policy table (`command -v omp` →
|
||||
`readlink -f` → `dist/cli.js`).
|
||||
2. `read cfg://tools` — live `maxTimeout` (value ≠ 0 explains any hard cap).
|
||||
3. Compare kill notices: the number in "killed once it has run Ns" is the resolved
|
||||
deadline; 300 = default reached (request dropped), other values = your request
|
||||
or the config cap.
|
||||
|
||||
## Launching jobs longer than the cap
|
||||
- Named **services are not deadline-capped**: `bash(name:"x", pty:false, command:"…; echo DONE >> log", ready:{log:"DONE", timeout:900})`.
|
||||
Readiness fires on the log pattern; result does NOT auto-deliver — `wait` on it or read `proc://x` after the supervisor notice.
|
||||
- NEVER end the service command with `echo "EXIT $?" >> log` and trust the
|
||||
supervisor's exit code — the trailing echo masks it (supervisor reports 0 while
|
||||
the log says EXIT 1). The log line is the truth; grep the log for the recorded
|
||||
code and `^not ok` before believing any "exit code 0".
|
||||
- Do not raise `tools.maxTimeout` unasked: session-scoped `write cfg://tools/maxTimeout`
|
||||
needs user approval and may be declined — respect it, use the service route.
|
||||
|
||||
## Multi-suite hazard
|
||||
Suites that rebuild shared artifacts per test (e.g. dankvoice bats rebuilds `dist/`)
|
||||
must not run concurrently — one suite at a time, or per-worktree copies only.
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
name: omp-extension-event-probe
|
||||
description: "Prove which lifecycle events the omp harness actually dispatches to extensions, with a throwaway timestamping probe plugin, instead of inferring delivery from side effects."
|
||||
---
|
||||
|
||||
# Prove harness→extension event delivery with a probe plugin
|
||||
|
||||
Use when an extension handler seems never to fire, or when you must know *when* an
|
||||
event arrives relative to another (mid-run vs settle). Never infer dispatch from
|
||||
the feature's side effects: async work in the handler (LLM rewrites, synthesis,
|
||||
network) lags seconds and destroys the timing signal.
|
||||
|
||||
## Shape (no build step needed)
|
||||
|
||||
`/tmp/probe/package.json` — the installer validates the declared entry on disk:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "event-probe",
|
||||
"version": "0.0.0",
|
||||
"type": "module",
|
||||
"exports": "./index.js",
|
||||
"pi": { "extensions": ["./index.js"] }
|
||||
}
|
||||
```
|
||||
|
||||
`/tmp/probe/index.js` — append arrival times, not counts:
|
||||
|
||||
```js
|
||||
import { appendFileSync } from "node:fs";
|
||||
const LOG = "/tmp/probe.log";
|
||||
const stamp = (line) => appendFileSync(LOG, `${new Date().toISOString().slice(11, 23)} ${line}\n`);
|
||||
export default function probe(pi) {
|
||||
pi.on("agent_start", () => stamp("agent_start"));
|
||||
pi.on("turn_end", (e) => stamp(`turn_end turn=${e.turnIndex} toolResults=${(e.toolResults ?? []).length}`));
|
||||
pi.on("agent_end", (e) => stamp(`agent_end messages=${(e.messages ?? []).length} willContinue=${String(e.willContinue)}`));
|
||||
}
|
||||
```
|
||||
|
||||
## Run
|
||||
|
||||
1. `cd ~/.omp/plugins && omp plugin link /tmp/probe` (link, never edit
|
||||
`node_modules`), then `omp plugin doctor` → expect a `✔ plugin:event-probe` line.
|
||||
2. Launch a **fresh** TUI session in a pty (`omp --cwd /tmp/probe-run`, `pty: true`).
|
||||
Extensions load at startup only — an already-running session never sees the probe.
|
||||
3. PTY gotcha: input sent before the splash clears is swallowed. Read the screen
|
||||
first, then submit text and finish with a separate write containing a bare
|
||||
`\r` (LF alone left the text sitting in the composer).
|
||||
4. Drive a task with ≥2 tool rounds so turns and settles separate in time
|
||||
(sequential reads work and need no approval; ask for steps "strictly one
|
||||
after the other" or the model parallelises them into one turn).
|
||||
5. Wait for a terminal line with an async bounded loop (`grep -q agent_end`,
|
||||
`sleep 1` inside one backgrounded bash call) — never poll across turns.
|
||||
|
||||
## Read
|
||||
|
||||
`turn_end` timestamps strictly earlier than `agent_end` ⇒ per-turn delivery
|
||||
works. `willContinue=true` on `agent_end` marks a non-terminal settle (harness
|
||||
maps `isTerminal === false` to it) — handlers that `return` on it never speak.
|
||||
|
||||
## Teardown (mandatory)
|
||||
|
||||
`omp plugin uninstall event-probe`; `omp plugin doctor`; delete `/tmp/probe*`.
|
||||
The link writes into `~/.omp/plugins/package.json` (tracked, live tier) —
|
||||
uninstall restores it; confirm with `grep` for the probe name returning nothing.
|
||||
|
||||
## Ground truth read for free
|
||||
|
||||
Event union + payloads: `~/.omp/plugins/node_modules/@oh-my-pi/pi-coding-agent/dist/types/extensibility/shared-events.d.ts`;
|
||||
the session→extension mapping lives in `src/extensibility/extensions/lifecycle-mirror.ts`
|
||||
(`extensionEventFromSessionEvent`). Extension-level `turn_end` carries
|
||||
`turnIndex`/`message`/`toolResults` but **no `willContinue`** — that field exists
|
||||
only on `agent_end`, so "is this the last turn" cannot be decided from `turn_end`.
|
||||
@@ -1,29 +1,48 @@
|
||||
---
|
||||
name: omp-extension-stale-module-audit
|
||||
description: "Diagnose recurring omp extension handler timeouts/silence when the on-disk fix is already deployed: harness has no extension hot-reload, so running sessions freeze the module at session start (zombie modules)."
|
||||
description: "Diagnose omp extension misbehavior (handler timeouts/silence, cross-instance arbitration failures like simultaneous speech) when the on-disk fix is already deployed: running sessions hold pre-upgrade in-memory bundles; compare ps lstart against artifact mtimes and merge dates."
|
||||
---
|
||||
|
||||
Use when a fix for an omp extension is merged/deployed yet the SAME errors persist ("handler timed out after 30000ms", silence, wrong behavior) in running sessions.
|
||||
# Stale in-memory omp extension bundles
|
||||
|
||||
**Core fact:** the harness loads extension `.ts` modules at session start and never reloads them (no `/reload` for extensions; bundle `reload` hits are LSP/MCP/browser only). A session older than the fix keeps executing stale code regardless of the file on disk.
|
||||
Symptom class: an extension behaves as if unfixed even though the disk artifact
|
||||
contains the fix — recurring handler timeouts, silence, or (multi-instance
|
||||
coordination features) arbitration failures that look like code bugs. Root
|
||||
cause: **omp loads the bundle at session start; `omp plugin upgrade` on disk
|
||||
does not touch the code already running in live sessions.**
|
||||
|
||||
**Procedure:**
|
||||
1. Census the errors from logs, not guesses — timeout lines carry the culprit pids:
|
||||
`grep -h "handler timed out" ~/.omp/logs/omp.<today>*.log | python3 -c 'import sys,json; rows=[json.loads(l) for l in sys.stdin]; rows.sort(key=lambda r:r["timestamp"]); [print(r["timestamp"], r["event"], r["pid"]) for r in rows]'`
|
||||
(`~/.omp/logs/omp.<date>.<pid>.log` — the pid in the filename is the session.)
|
||||
2. Session start times vs deployed file versions:
|
||||
`ps -eo pid,lstart,args | grep '[o]mp'`
|
||||
`git log --format='%h %ad %s' --date=format:'%m-%d %H:%M' -- <extension path>`
|
||||
`git reflog --format='%gd %ad %gs' --date=format:'%m-%d %H:%M' | grep merge` (live-sync pull times = when the file content changed).
|
||||
Any session whose start time precedes the fix's arrival has the stale module.
|
||||
3. Confirm the stale revision actually contains the bad pattern — read what that session loaded, not HEAD:
|
||||
`git show <old-rev>:<path> | grep -n 'await speak\|async (event'`
|
||||
(e.g. an `async agent_end` awaiting a >30 s chain is exactly what the harness's per-handler `Kzc` 30 s race logs; fire-and-forget `void` is fine — detached promises are NOT tracked by the dispatch window.)
|
||||
4. Optional empirical proof when static analysis says "impossible": probe extensions through the real harness:
|
||||
`omp -p --no-extensions -e /tmp/probe.ts "Reply OK"` with a probe that logs handler start/return and `void`s a 40 s detached promise + spawns a long child. Fast exit + no timeout log ⇒ detachment is untracked ⇒ blame stale modules, not the harness.
|
||||
5. Remediation: restart the affected sessions (kill the listed pids or their terminals — the operator session itself may be one of the zombies; say so explicitly). There is no other fix.
|
||||
## Recipe 1: date the sessions against the artifact
|
||||
|
||||
**Gotchas:**
|
||||
- Per-handler attribution in the log is accurate per dispatch; don't blame other extensions before checking the pid's start time.
|
||||
- Endpoints healthy + silence persisting usually means TWO stacked bugs (e.g. stale awaiting module in old sessions AND a mute bug in current code) — diagnose each pid cohort separately.
|
||||
- Extension chain code must use self-owned `AbortSignal.timeout(...)`, never a ctx-scoped signal: harness dispatch signals die when the dispatch window closes.
|
||||
```bash
|
||||
date
|
||||
stat -c '%y %n' ~/.omp/plugins/node_modules/<pkg>/dist/index.js # last upgrade
|
||||
ps -eo pid,lstart,etime,args | grep 'omp' | grep -v grep # each session's boot
|
||||
```
|
||||
|
||||
Any session whose START predates the artifact mtime holds an older bundle —
|
||||
for it, the "fix" never shipped. When the feature landed in git, compare
|
||||
against the merge date, not the last upgrade (`git log --format='%h %cI %s'
|
||||
main | grep -i <feature>`): a session can predate the feature *existing*.
|
||||
|
||||
Fix is always: restart (`/quit`, then `omp --resume <id>`) — resume is
|
||||
lossless. It is the user's live session; never kill it yourself.
|
||||
|
||||
## Recipe 2: cross-process arbitration — a bypasser explains "simultaneous X"
|
||||
|
||||
When a feature serializes across sessions via files (e.g. dankvoice stage:
|
||||
`~/.cache/dankvoice/stage/` with `lease.json` = `{holder,pid,expires}`, one
|
||||
`<ISO>-<session>.json` entry per queued item, FIFO = filename order, lease TTL
|
||||
6 s), overlap between two actors proves **at least one actor never consults
|
||||
the directory** — the mechanism can't serialize against a bypasser. Census the
|
||||
queue live (`watch -g -d 'ls <dir>; cat <dir>/lease.json'`): a live, renewing
|
||||
lease proves the new sessions coordinate; then Recipe 1 finds the bypasser.
|
||||
|
||||
Known residual hazard to note, not silently fix: a lease heartbeat is a timer
|
||||
on the holder's own event loop; a stall longer than the TTL lets a peer steal
|
||||
the lease mid-action. Mitigation = pid-liveness check in the steal path.
|
||||
|
||||
## Verification
|
||||
|
||||
After restarts, the symptom must vanish while the queue shows traffic. If it
|
||||
persists, look for actors on OTHER hosts (file arbitration is per-box), or
|
||||
the TTL-steal hazard above.
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
name: omp-lifecycle-event-selection
|
||||
description: "Pick the right omp harness lifecycle event for narration/telemetry: agent_end is a settle, turn_end waits for tool results, message_end is earliest with content, after_provider_response carries no content."
|
||||
---
|
||||
|
||||
# Choosing an omp lifecycle event (narration, telemetry, hooks)
|
||||
|
||||
Re-derived twice from the installed typings/sources (`@oh-my-pi/pi-coding-agent`, `@oh-my-pi/pi-agent-core`) while fixing dankvoice. Read this before wiring a handler; verify against the installed version's `src/extensibility/extensions/types.ts` and `pi-agent-core/src/agent-loop.ts` when precision matters.
|
||||
|
||||
| event | payload you can use | when it fires | traps |
|
||||
|---|---|---|---|
|
||||
| `after_provider_response` | `status`, `headers`, `requestId`, `metadata` **only** | response headers arrive, **before the stream body is consumed** | No message content at all: useless for anything that speaks or summarizes text. Right for latency telemetry, non-200/abort detection, `requestId` correlation. |
|
||||
| `message_end` | full `message` (assistant, user, tool-result, gate-stop, error) | the moment that message is built (`agent-loop.ts`, assistant case) | Earliest event carrying assistant text. Fires for non-assistant messages too → gate `message.role === "assistant"`. Payload is cloned for extensions. |
|
||||
| `turn_end` | `turnIndex`, `message`, `toolResults` | after that turn's **tool results** come back | Stale by the tool duration: a 14 s tool delays a `turn_end` trigger 14 s. Extension event has **no `willContinue`** (the `turnIndex` is numbered by the lifecycle mirror), so you cannot tell the terminal turn from here. |
|
||||
| `agent_end` | `messages`, `willContinue`, `awaitingAsyncWork` | once per user prompt, when the loop settles | `willContinue: true` = the harness scheduled a continuation (auto-retry / empty-stop), not a user-visible settle — narrating it speaks an answer the model is about to redo. Silence-prone on long-running sessions. |
|
||||
| `agent_start` | — | run begin | Use to reset per-run dedupe state, so a later answer with identical text is not swallowed. |
|
||||
|
||||
## Narration cadence pattern (what dankvoice ships)
|
||||
|
||||
- `turn_end` (or `message_end` if the tool-time lag matters) for mid-run commentary; `agent_end` only for the settled answer.
|
||||
- Drop a mid-run trigger while audio/IO is busy — never queue, never interrupt.
|
||||
- Dedupe the terminal text per run (`agent_start` resets), because the last turn and the settle carry the same string.
|
||||
- Skip `ctx.mode === "print" | "json"` so benches stay silent.
|
||||
|
||||
## Prove delivery, don't infer it
|
||||
|
||||
Audio/UI timing cannot show which events arrive. Link a throwaway extension that appends timestamps per event and run one real session: `omp plugin link /tmp/probe` (package.json with `"pi": { "extensions": ["./index.js"] }`), prompt a task with a slow tool, read the log, then `omp plugin uninstall probe`. Confirmed live 2026-10-19: extensions receive `agent_start`, `turn_end`, `agent_end`; `turn_end` landed seconds before `agent_end`.
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
name: omp-local-artifact-path
|
||||
description: Resolve a local:// artifact to a real filesystem path (for scripts/curl) without guessing the session-slug directory.
|
||||
---
|
||||
|
||||
`local://<name>.md` artifacts live in a per-cwd, per-session directory whose name you cannot guess. Three invented paths failed before this worked (2026-10-19).
|
||||
|
||||
Actual layout:
|
||||
|
||||
```
|
||||
~/.omp/agent/sessions/<cwd-slug>/<timestamp>Z_<session-id>/local/<name>.md
|
||||
```
|
||||
|
||||
`<cwd-slug>` is the working directory with `/` replaced by `-`, **keeping the leading dash** — `/home/blank/.omp/wt/wt-20261019-125509-7c8a99d` under `/home/blank` becomes `-.omp-wt-wt-20261019-125509-7c8a99d`. Dropping the leading `-.` is the mistake that produces `FileNotFoundError`.
|
||||
|
||||
## Get the real filesystem path
|
||||
|
||||
Ask the tool, don't reconstruct it — a ranged read of the artifact reports the resolved path in its header:
|
||||
|
||||
```
|
||||
read local://pr-body-msgend.md:1-1 -> [~/.omp/agent/sessions/-.../local/pr-body-msgend.md#TAG]
|
||||
```
|
||||
|
||||
Or locate it by basename (needs enough depth; `maxdepth 5` misses it — the file sits 6–7 levels down):
|
||||
|
||||
```bash
|
||||
f=$(find ~/.omp/agent/sessions -maxdepth 7 -name 'pr-body-msgend.md' | head -1)
|
||||
cp "$f" /tmp/prbody.md # stable path for the python/curl step
|
||||
```
|
||||
|
||||
Copy to `/tmp` before handing the content to a script: one cheap command, and every later tool call stops depending on session-scoped paths.
|
||||
|
||||
## Related lookups in the same directory tree
|
||||
|
||||
- Session id for the current cwd — `~/.omp/agent/history.db` (prompts table has `session_id`, `cwd`; the column is `prompt`, **not** `text`, and raw `?q=` queries cannot be combined with a table selector or pagination):
|
||||
```
|
||||
read history.db?q=SELECT session_id, cwd, substr(prompt,1,50) AS p FROM history WHERE cwd LIKE '%wt-20261019%' ORDER BY rowid DESC LIMIT 3
|
||||
```
|
||||
Use the real id in issue/PR attribution footers; never reuse an id remembered from a compacted transcript.
|
||||
- Transcript dir for the same session sits beside `local/`.
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
name: omp-search-provider-fallback-debug
|
||||
description: "Diagnose why the omp harness web_search silently uses a fallback provider (e.g. Kagi) instead of the configured primary (SearXNG): the fallbackChains masking pattern, the searxng.endpoint base-URL scheme/guard-chain trap, and the cli.js + content-type probes that prove it."
|
||||
---
|
||||
|
||||
# omp web_search silently using the wrong provider
|
||||
|
||||
Symptom: agent searches return results from a fallback provider (e.g. **Kagi**) even though
|
||||
`modelRoles.web` is set to the intended primary (e.g. `web/searxng`). The primary errors
|
||||
every time and the failure is invisible because a fallback answers successfully.
|
||||
|
||||
## The masking mechanism (always check this first)
|
||||
|
||||
`~/.omp/agent/config.yml`:
|
||||
- `modelRoles.web:` is the primary (e.g. `web/searxng`).
|
||||
- `retry.fallbackChains.web:` is an ordered list tried when the primary throws. The FIRST
|
||||
entry silently serves whenever the primary errors. In this fleet it is `web/kagi`, so any
|
||||
SearXNG error == Kagi answers and you never see an error.
|
||||
|
||||
So "searches go through Kagi" almost always means "SearXNG is failing", not a routing mistake.
|
||||
|
||||
## The SearXNG endpoint trap (the usual root cause here)
|
||||
|
||||
`searxng.endpoint` in config.yml must be a **full base URL pointing at the direct JSON port**,
|
||||
used **verbatim** by omp (no scheme auto-added, no normalize — confirmed in `cli.js`). Two ways
|
||||
it breaks:
|
||||
|
||||
1. **Bare host** (e.g. `search.franksdank.cloud`, no scheme): omp does `new URL(\`${endpoint}/search\`)`
|
||||
→ **throws** immediately (URL needs a scheme). SearXNG never contacted.
|
||||
2. **The guarded vhost** (`https://search.franksdank.cloud`): returns Anubis' **"Making sure
|
||||
you're not a bot!"** challenge page at HTTP **200 `text/html`**. omp calls `o.json()`
|
||||
unconditionally regardless of content-type → throws on the leading `<`. There is
|
||||
deliberately NO API path through the guard chain (JS challenge + UA denylist).
|
||||
|
||||
**Correct value** (direct published port, http scheme preserved — reachable from the LAN only):
|
||||
|
||||
```yaml
|
||||
searxng:
|
||||
endpoint: http://192.168.0.143:8081 # Dankclaw; NOT the vhost, NOT schemeless
|
||||
```
|
||||
|
||||
The published port source (192.168.x/loopback/LAN) is in SearXNG's `pass_ip`, so it bypasses
|
||||
the JSON rate limiter (hardcoded `API_MAX=4`/IP/hr, not settable except via `pass_ip`).
|
||||
|
||||
## Prove it (don't guess)
|
||||
|
||||
1. Content-type probe, both paths — the decisive check:
|
||||
```
|
||||
curl -sk -o /tmp/vh -w "vhost ct=%{content_type}\n" "https://search.franksdank.cloud/search?q=test&format=json" # expect text/html (Anubis)
|
||||
curl -s -o /tmp/dp -w "direct ct=%{content_type}\n" "http://192.168.0.143:8081/search?q=test&format=json" # expect application/json
|
||||
```
|
||||
vhost = `text/html` + direct = `application/json` confirms the trap.
|
||||
2. Code path, in the installed omp dist (NOT a worktree):
|
||||
`/home/blank/.bun/install/global/node_modules/@oh-my-pi/pi-coding-agent/dist/cli.js`
|
||||
(find install: `dirname $(readlink -f $(which omp))`). Grep the SearXNG provider:
|
||||
- `searxng.endpoint` config def → description "Base URL of a self-hosted SearXNG instance".
|
||||
- the request builder `...=e.replace(/\/+$/,""),new URL(\`${r}/search\`)...` → endpoint is
|
||||
verbatim, scheme NOT added.
|
||||
- `X2c`/`J2c` → `if(!o.ok)throw` then `return await o.json()` → HTML 200 passes `.ok` and
|
||||
blows up in `.json()`.
|
||||
|
||||
## Apply + verify
|
||||
|
||||
- Fix is a harness config edit (`cfg://searxng/endpoint` prompts the user, or edit config.yml).
|
||||
Confirm with the user first — it is their live harness config.
|
||||
- Verification after applying: fire a real `web_search` and confirm it returns
|
||||
`provider:"searxng"` (not `"searxng"`-then-fallback / not `"kagi"`).
|
||||
|
||||
## Related gotcha (same masking, different cause)
|
||||
|
||||
If SearXNG answers JSON but returns empty `results` with `unresponsive_engines` (VPN-exit-IP
|
||||
CAPTCHA/429 on google/brave), omp's `vcs` throws a 503 → Kagi again. That's an engine-health
|
||||
problem, fix in `[[SearXNG]]`/`[[SearXNG Providers]]`, not the endpoint.
|
||||
|
||||
## Hardening option
|
||||
|
||||
Drop or reorder `web/kagi` / `web/perplexity` in `retry.fallbackChains.web` so primary failures
|
||||
surface instead of silently spending a paid Kagi credit. Flag to user before editing.
|
||||
@@ -1,49 +1,22 @@
|
||||
---
|
||||
name: omp-worktree-path-gotchas
|
||||
description: "Use in ~/.omp (or any multi-worktree repo) when an edit lands in the wrong checkout, git checkout is policy-refused, or an omp session id is needed — relative-path tool resolution, shared-worktree branch law, session-id lookup, edit auto-prefix trap."
|
||||
description: "Multi-worktree edits/tests land in the wrong checkout: bash persistent-cwd drift, phantom file reverts, which tree a tool really touched"
|
||||
---
|
||||
|
||||
# Multi-worktree gotchas in ~/.omp-style repos
|
||||
## Purpose
|
||||
Answer "why did my file/tests revert or behave differently than what I just wrote?" in repos with several worktrees (e.g. `~/.omp`, `~/Projects/wt-*`, `~/.omp/wt/*`).
|
||||
|
||||
Measured 2026-10-18 building PR #12/#14 in `dank/command-center` worktrees.
|
||||
## Failure modes and fixes
|
||||
|
||||
## 1. Relative paths resolve to the SESSION worktree — always edit by absolute path
|
||||
`edit`/`read`/`write` with a relative `path` resolve against the session CWD
|
||||
(the harness worktree you were spawned in), NEVER the worktree you `cd`'d into
|
||||
via bash. Identical file content in sibling worktrees yields identical
|
||||
`[path#tag]` hashes, so an edit can "succeed" against the WRONG checkout with
|
||||
no warning; only a rejected stale-hash edit reveals it.
|
||||
1. **Bash tool persistent-cwd drift** (the expensive one): the persistent shell keeps its cwd across calls; a call *without* an explicit `cwd` runs in whatever the *previous* call left, which after interleaved work may be a DIFFERENT worktree. Result: reads/greps/appends hit tree A while `git`/`bats` hit tree B — symptoms are "file keeps reverting to HEAD", duplicate test names at shifting line numbers, scenario `CALLS 0` for scenarios that exist, mismatched `grep -n` vs tool-reported line numbers.
|
||||
- **Diagnose:** `pwd` in the same call as the failing command; `wc -l <file>` in BOTH candidate worktrees; compare md5s. The file is not flapping — two files exist and commands alternate between them.
|
||||
- **Fix/prevent:** pass an explicit `cwd` on EVERY bash call once two or more checkouts are in play; never trust a read taken without one as evidence about "the" file.
|
||||
|
||||
- When working in a second worktree, ALL file-tool paths: absolute.
|
||||
- Symptom seen: edit rejected "hash #X is not from this session" while the
|
||||
bash-side `git status` showed the file clean — two different files, two
|
||||
worktrees. Recovery: revert the stray edit (CUT the inserted lines) in the
|
||||
wrong tree, re-read the right tree, redo with absolute header.
|
||||
2. **Symlinked or stale `git` state**: `git status` in a worktree shows a clean tree while the disk file differs from what you intended — the corruption is already committed. Audit committed blobs (`git log --oneline -- <file>`; per-commit `git show <c>:<path> | grep -c <sentinel>`) instead of re-reading the disk copy repeatedly.
|
||||
|
||||
## 2. Branching: shared worktrees refuse git checkout
|
||||
Workspace policy blocks `git checkout` (branch switches AND `--` discards) in
|
||||
harness-managed worktrees — concurrent agents depend on them. To start a
|
||||
branch off fresh trunk:
|
||||
3. **Same-name files across worktrees** (`tests/behavior.bats` exists in each): a `find`/`grep` rooted above the worktrees mixes copies; anchor the search path to one worktree root.
|
||||
|
||||
```bash
|
||||
git -C ~/.omp worktree add -q -b <type>/<issue#>-<slug> ~/.omp/wt/<name> origin/main
|
||||
```
|
||||
|
||||
Never force past the refusal; the new-worktree route is strictly safer.
|
||||
Custom worktrees not matching `wt/wt-*` are invisible to the live-sync reaper —
|
||||
prune them by hand after merge (`git worktree remove`, `git branch -d`).
|
||||
|
||||
## 3. omp session id for attribution footers
|
||||
Fleet law requires `session:` in issue/PR bodies. Authoritative source:
|
||||
`~/.omp/agent/sessions/<cwd-slug>/<timestamp>_<UUID>.jsonl` — the `<UUID>` is
|
||||
the session id. Pick the newest file whose first `{"type":"title"}` line names
|
||||
this conversation to prove the id is yours. The `wt/<ts>-<hash>` dir suffix is
|
||||
NOT the session id (it persists across sessions per cwd).
|
||||
|
||||
## 4. edit-tool auto-prefix trap (recurring)
|
||||
A body row written bare (intending it as the "original line" anchor or as
|
||||
context) is AUTO-PREFIXED with `+` and INSERTED — leaving the old line plus a
|
||||
duplicate. `PUT n.=n:` therefore behaves like insert, not replace, whenever
|
||||
the anchor row lacks an explicit marker. After any `PUT` where the response
|
||||
warns "Auto-prefixed bare body row(s)", immediately `CUT` the stale line number
|
||||
shown in the response (descending order for multiple CUTs).
|
||||
## Repair recipe for a corrupted-but-committed test file
|
||||
- Inventory both versions: `grep -o '@test "[^"]*"' | sort` and `grep -o 'case "[a-z]*":' | sort -u`; `comm` the sets to find lost scenarios and duplicate tests.
|
||||
- Rebuild from the committed copy (`git show HEAD:<path> > <path>`, no checkout), then re-apply only the true deltas: restore missing scenario blocks from the last clean base, dedupe imports, rename scratch-dir mkdtemp prefixes that collide with a test's own orphan-census glob (e.g. `dankvoice-stage-` matching `dankvoice-*`).
|
||||
- Re-run the targeted tests, then the full suite single-process.
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
---
|
||||
name: open-notebook-provision-model-rest
|
||||
description: "Provision or repair an open-notebook model (LLM/TTS/embedding) over its REST API on Dankclaw: credential→model→defaults order, the defaults-store-the-ID trap, the /api/embed 500-that-succeeds, and the two-note ranking proof for embeddings."
|
||||
---
|
||||
|
||||
# open-notebook model provisioning over REST
|
||||
|
||||
Use when adding/repairing a model row (embedding, LLM, TTS) in open-notebook on Dankclaw, when semantic
|
||||
search is dead, or when an embed error appears in the worker log. Config lives in the **database**
|
||||
(encrypted with `OPEN_NOTEBOOK_ENCRYPTION_KEY`) — no repo file mirrors it, REST or the UI are the only doors.
|
||||
|
||||
## Access (never put the bearer in argv)
|
||||
|
||||
The API bearer IS `OPEN_NOTEBOOK_PASSWORD`; the API publishes on loopback `127.0.0.1:5056` only, so run
|
||||
from Dankclaw over ssh. `ssh dankclaw 'curl -H "Authorization: Bearer …"'` leaks the secret into
|
||||
`/proc/<pid>/cmdline` — use a mode-600 curl config:
|
||||
|
||||
```bash
|
||||
PW=$(grep -m1 '^OPEN_NOTEBOOK_PASSWORD=' /home/dank/Containers/open-notebook/env | cut -d= -f2- | tr -d '"')
|
||||
cfg=$(mktemp); chmod 600 "$cfg"; printf 'header = "Authorization: Bearer %s"\n' "$PW" > "$cfg"
|
||||
B=http://127.0.0.1:5056 # not 5055: seerr holds the host's 5055
|
||||
```
|
||||
|
||||
## Order, and the address rule
|
||||
|
||||
`POST /api/credentials` → `POST /api/models` (`credential: <credential id>`) → `PUT /api/models/defaults`
|
||||
→ any profile that names a model (speaker `voice_model`, episode `speaker_config`/`outline_llm`).
|
||||
|
||||
- Credential provider for any in-house OpenAI-compatible server: **`openai_compatible`** (not `openai`),
|
||||
with a dummy `api_key` when the upstream does not authenticate.
|
||||
- The app's URL guard refuses base URLs resolving in `169.254.0.0/16`, so `host.containers.internal`
|
||||
cannot be stored; a **container name on `dankclaw_default`** (e.g. `http://frank-dragoon-halogen:8731/v1`)
|
||||
or another machine's RFC1918 IP both pass.
|
||||
|
||||
## The two traps that eat the most time
|
||||
|
||||
1. **`PUT /api/models/defaults` takes the model ID, never the name.** `"default_embedding_model":
|
||||
"qwen3-embedding-0.6b"` is accepted and echoed back, then every job dies in the worker with
|
||||
`Model with ID qwen3-embedding-0.6b not found`. Use the row's `model:…` id (GET defaults first, change
|
||||
one key, PUT the whole object back — the other keys in that response are ids and must be preserved).
|
||||
2. **`POST /api/embed` answers 500 *after* the work succeeds.** Upstream `EmbedResponse.command_id`
|
||||
declares `str` and the worker returns a `RecordID`, so FastAPI raises in serialization only. The truth is
|
||||
`podman logs open-notebook | grep "Successfully embedded"`, never the HTTP code. Note saves
|
||||
(`POST /api/notes`) submit `embed_note` themselves; `/api/embed` is only for backfills.
|
||||
|
||||
## Proof it actually works
|
||||
|
||||
- Per row: `POST /api/models/<id>/test` → `Embedding dimensions: 1024` (embeddings) or a real completion.
|
||||
- Embeddings, discriminating test: create **two** unrelated notes, then
|
||||
`POST /api/search` `{"query": "...", "type": "vector", "search_notes": true, "limit": 5}` — the enum is
|
||||
`text|vector` (not `semantic`) and the score field is `similarity`. Each query must rank its own note
|
||||
high and the other low (measured 0.659/0.292 vs 0.906/0.318). One note proves nothing: `total_count` is 1
|
||||
either way. Delete the probe notes afterwards (`DELETE /api/notes/<id>`).
|
||||
- Confirm which engine served it: `podman logs <engine container> | grep "POST /v1/embeddings"`.
|
||||
|
||||
Owner pages: `serverAppConfigs-docs/Dankclaw/Open Notebook.md` (app-side shapes), and the endpoint's own
|
||||
page for whatever serves the model.
|
||||
@@ -0,0 +1,41 @@
|
||||
---
|
||||
name: parallel-suite-shared-db-flake
|
||||
description: "Diagnose test suites that pass isolated but fail only under concurrent/uncached runs against a shared database: cross-file teardown wipes, FK-cascaded sessions, wildcard LIKE deletes."
|
||||
---
|
||||
|
||||
# Flaky only under concurrency on a shared DB
|
||||
|
||||
Symptom: a suite passes `vitest run <file>` alone, fails under `turbo run test --force`
|
||||
(or any concurrent uncached runner) with impossible errors — 401 where 404 was
|
||||
expected, rows vanishing mid-test, sessions dying between two awaited requests.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. **Reproduce the concurrency, not the file.** Isolated runs prove nothing.
|
||||
`bun x turbo run test --force --output-logs=errors-only` (or run the two suspect
|
||||
files simultaneously). Never conclude "flake" from one green isolated run.
|
||||
2. **Map the impossible state to a vanished row.** 401/auth failures where the
|
||||
session was created milliseconds earlier = the session/user row was DELETED
|
||||
under the test, not expired. Look for writers, not readers.
|
||||
3. **Grep every suite's teardown for shared-namespace deletes:**
|
||||
`grep -rn "TRUNCATE\|DELETE FROM\|LIKE" <pkg>/src/*.test.*`
|
||||
The classic killer: two test files sign users up under the same domain
|
||||
(`%@studio.test`) and each `afterAll` runs a wildcard `DELETE ... LIKE '%@...'`.
|
||||
With parallel files on one DB, whichever file finishes first cascades away the
|
||||
other's live rows (user FK cascades kill sessions mid-flight).
|
||||
4. **Fix = scope, never coordinate.** Each suite deletes only rows it created:
|
||||
per-file email/id prefixes (`auth*` vs `rel*`), or track created keys and
|
||||
`WHERE email = ANY(...)`. Timing tricks (sleeps, retry) hide the collision.
|
||||
5. **Prove stability:** 3 consecutive `--force` uncached runs green, then the
|
||||
full gate. Cache makes everything look green; only uncached concurrent runs
|
||||
re-exercise the collision window.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Error signatures lie: `expected 401 to be 404` reads as an auth bug but is a
|
||||
deleted-row cascade.
|
||||
- Supertest `cookieOf`-style helpers that THROW on missing cookies prove signup
|
||||
succeeded — so the session died after creation; keep hunting for the deleter.
|
||||
- Shared dev DBs (one Postgres for all workspaces/worktrees) make every wildcard
|
||||
teardown a cross-suite weapon. Suite teardowns on a shared box MUST scope to
|
||||
rows the suite created.
|
||||
@@ -1,68 +1,34 @@
|
||||
---
|
||||
name: parallel-worktree-path-hazard
|
||||
description: "\"Use when several agents edit the same repo in different git worktrees at once, or when an edit/read may have landed in the wrong checkout: which path forms bind where, how to detect with git -C status, and the repair protocol (cp + git show HEAD blob write-back) because git checkout/restore is policy-blocked in a shared workspace.\""
|
||||
description: "Use when several agents edit the same repo in different git worktrees at once, or when an edit lands in the wrong checkout: relative tool paths resolve to the session cwd, not the shell cwd."
|
||||
---
|
||||
|
||||
# Concurrent-worktree path hazard (shared repo, many agents)
|
||||
Use when several agents (or you across two checkouts) edit the same repo in different git worktrees at once, or when an edit/query lands in the wrong checkout.
|
||||
|
||||
Measured 2026-10-18 on `serverAppConfigs` with two agents live at once. The
|
||||
failure is **silent**: the tool reports success, the diff looks right, and the
|
||||
change lives in a *different* worktree.
|
||||
## The silent failure: relative tool paths resolve to the SESSION cwd
|
||||
|
||||
## Which path form binds where
|
||||
- **Parent/top-level session**: a **relative** `path` resolves against the
|
||||
session CWD — its own worktree — never against the worktree named in the
|
||||
prompt. This is how a change meant for `wt-tts-trial-docs` landed in
|
||||
`wt-20261017-…`.
|
||||
- **Subagent**: an **absolute** path to its own assigned worktree can resolve
|
||||
into the parent's worktree when the same *relative* path exists there
|
||||
(`edit` twice onto `dankclaw/.env.example`). Its `write` with the same
|
||||
absolute path was fine — the bug is `edit`/`read` path resolution, not `write`.
|
||||
`read` / `edit` / `write` / `grep` resolve a **relative** `path` against the **session working directory** — never against the `cd` inside a neighboring `bash` call, and not against a `../other-worktree/...` prefix either (that gets normalized back into the session tree). So after `cd ../wt-B && grep …` you can `read path="docs/page.md"` and get **wt-A's** copy.
|
||||
|
||||
Rules that hold:
|
||||
1. A subagent passes paths as `../<its-worktree>/<rel>` (relative to the parent
|
||||
worktree) for `read` **and** `edit`; the parent passes **absolute** paths to
|
||||
any worktree that is not its own CWD.
|
||||
2. After every edit batch, prove the landing site, do not trust the tool's echo:
|
||||
`git -C <intended-worktree> status --short` must list the file. A file you
|
||||
edited that is missing from that list went somewhere else.
|
||||
3. Keep each work item on its own branch/worktree, and keep ONE integration
|
||||
owner (the parent) for shared files — `scripts/deploy-remote.sh`,
|
||||
`scripts/deploy-plan.py`, `tests/`, `serverAppConfigs-docs/index.md`,
|
||||
`AGENTS.md` are the collision hot spots in this repo. Never rebase a
|
||||
sibling's branch.
|
||||
Why it survives "I just read it to check": the returned path is printed workspace-relative and the file name is identical in every checkout, so the output looks like a correct read of the tree you meant. Only content that differs between the checkouts betrays it.
|
||||
|
||||
## Repair protocol (no `git checkout`, no `git restore`)
|
||||
`git checkout -- <path>` and `git restore <path>` are **policy-blocked** in a
|
||||
shared workspace ("overwriting working-tree files … discards changes other
|
||||
concurrent agents rely on"). Unstaging (`git reset [<path>]`,
|
||||
`git restore --staged <path>`) is allowed but does not undo content. So:
|
||||
Rules:
|
||||
- **Absolute paths for every tool call that targets a worktree other than the session's.** No exceptions for quick reads.
|
||||
- Confirm the checkout from content you know differs (a heading, a status line), not from the echoed path.
|
||||
- A `read` error like "Did you mean .AGENTTRASH/<name>" is a signal the relative path resolved into the session tree — re-issue absolute.
|
||||
- Same trap for `bash`: it starts each call in the session cwd, so `cd` twice per call or use absolute paths there too.
|
||||
|
||||
```bash
|
||||
F="path/with spaces if any"
|
||||
# 1. move the change to the worktree that should own it (both at the same base commit)
|
||||
cp "<wrong-wt>/$F" "<right-wt>/$F"
|
||||
# 2. undo in the wrong worktree by writing the committed blob back
|
||||
git -C "<wrong-wt>" show "HEAD:$F" | tee "<wrong-wt>/$F" > /dev/null
|
||||
# 3. prove both sides
|
||||
md5sum <(git -C "<wrong-wt>" show "HEAD:$F") "<wrong-wt>/$F" "<right-wt>/$F"
|
||||
git -C "<wrong-wt>" status --short # must be empty for that file
|
||||
git -C "<right-wt>" diff --stat # must show the intended delta
|
||||
```
|
||||
## Landing a change when the session worktree is stale
|
||||
|
||||
Never fix it blind: if the wrong worktree had its own uncommitted edits to that
|
||||
file, `cp`/blob-write-back would destroy them — check
|
||||
`git -C <wrong-wt> status --short` FIRST and reconcile content, don't overwrite.
|
||||
`git diff --name-only HEAD origin/main` first. If main moved a file you are editing, your edit's base is wrong even though the file "looks fine" — do not commit on the stale base.
|
||||
|
||||
## Tidy after a cancelled sibling
|
||||
A killed subagent can leave a worktree plus an empty branch. Removing them is
|
||||
safe only when verified empty: `git -C <main> log --oneline -3 <branch>` (no
|
||||
unique commits) and `git -C <wt> status --short` (clean), then
|
||||
`git worktree remove <wt>` + `git branch -D <branch>`. An orphan worktree at a
|
||||
stale commit is the same class of trap as a leftover bench drop-in: the next
|
||||
agent reads it as unfinished work.
|
||||
1. `git worktree add ../wt-clean -b <type>/<slug> origin/main`; copy the new files in.
|
||||
2. Re-apply the edit against the clean tree's version **with absolute paths**, then read it back absolutely to prove which tree changed.
|
||||
3. Run the suite inside the clean worktree; commit there.
|
||||
4. Remove your own duplicate edit from the session worktree by editing it away (`CUT`/`PUT`). `git checkout -- <path>` and branch switches are policy-blocked in shared workspaces and should stay blocked — they discard edits other agents rely on.
|
||||
5. Never carry the same uncommitted edit in two worktrees at once; if you must keep both trees, keep a task that reconciles them.
|
||||
|
||||
## Report it
|
||||
File the misresolution to `xd://report_issue` (false positives are fine) and
|
||||
record the incident in the repo worklog — a repair nobody wrote down looks like
|
||||
a hijack later.
|
||||
## Multi-agent hazards in one shared worktree
|
||||
|
||||
- Another agent can stage, commit, or check out files under you; re-read a file immediately before editing it if any tool result hinted at intervening change.
|
||||
- Untracked scratch you drop in a shared tree (reports, notes, `.AGENTTRASH/` entries) is visible to everyone and can be committed by them. Put throwaway artifacts outside the checkout.
|
||||
- Never `git clean -f`, `rm -rf`, or branch switching to "get a clean state" in a tree another agent is using.
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
---
|
||||
name: pdf-lib-determinism-moddate-leak
|
||||
description: "Diagnose pdf-lib output that isn't byte-identical across fetches: PDFDocument.load restamps /ModDate; fake-clock determinism test; inflate object streams to diff."
|
||||
---
|
||||
|
||||
# pdf-lib byte-identical output: the /ModDate clock leak
|
||||
|
||||
## Symptom
|
||||
A "reconstruct-on-read, byte-identical forever" PDF endpoint passes its back-to-back refetch test but returns **different bytes when fetches cross a wall-clock second**. Same-second fetches match; a fetch 1s later differs (often a different stream `/Length`, e.g. object stream 411→420 bytes).
|
||||
|
||||
## Root cause
|
||||
`PDFDocument.load()` has its own `updateMetadata` option (**defaults true**) and stamps `/Info /ModDate` from the wall clock **at load time**, not just at save. `doc.save({updateMetadata:false})` on its own is not enough (and in pdf-lib 1.17 `updateMetadata` isn't even in the `SaveOptions` type — only in `PDFOptions` on `load`); passing it to `save` also fails typecheck.
|
||||
|
||||
## Fix
|
||||
```ts
|
||||
const doc = await PDFDocument.load(templateBytes, { updateMetadata: false });
|
||||
// ... fill, stamp print layer ...
|
||||
return doc.save(); // save never stamps when load didn't
|
||||
```
|
||||
Output becomes a pure function of (template bytes, values).
|
||||
|
||||
## Determinism test law
|
||||
A determinism test MUST straddle a time boundary. Back-to-back fetches/fills are structurally blind to per-second stamping.
|
||||
- Unit: fake the clock (`@sinonjs/fake-timers` or `vi.useFakeTimers`) → fill → jump +2 months → re-fill → `expect(Buffer.from(second)).toEqual(first)`. No sleeps, no flake window. Faking `Date` outside a test framework requires patching `globalThis.Date` BEFORE the module import that uses it.
|
||||
- Prove it red first against the old code (expect the byte/length drift).
|
||||
|
||||
## Byte-diff forensics (when lengths differ)
|
||||
1. Dump the differing region raw: `Buffer.subarray(idx-60, idx+60)` rendered latin1 with non-printables stripped — locate the exact field (`D:YYYYMMDDHHmmss` = clock leak).
|
||||
2. If the diff is inside a compressed object stream, inflate and compare streams:
|
||||
```ts
|
||||
import { inflateSync } from "node:zlib";
|
||||
// find /Type /ObjStm ... stream chunks, inflateSync each, string-compare
|
||||
```
|
||||
3. pdf-lib escapes text-show operands — inflate + unescape `\\(\\[()\\\\]\\)` before substring-asserting printed values.
|
||||
|
||||
## Live proof recipe
|
||||
Seed a submission, then fetch both retrieval doors at t, t+1.5s, t+3s; `sha256sum` all three → one hash. Same-second comparisons prove nothing. And kill any long-running dev service holding the app port first — a live :3000-style stack can hang or false-fail an e2e gate.
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
name: pdf-widget-layer-census
|
||||
description: "Census a real-world \"flat\" PDF before assuming it needs field injection: Quartz/Preview exports often already carry an AcroForm widget layer under underscore rules; also audits pdf-lib flatten residue."
|
||||
---
|
||||
|
||||
# Real-PDF census (widget-first, then judge)
|
||||
|
||||
A PDF that *looks* flat (`pdftotext` shows `____` fill rules) may already contain a
|
||||
full AcroForm widget layer — macOS Preview/Quartz exports write widgets with
|
||||
transparent rects under drawn underscores. Injecting fields into such a file
|
||||
duplicates/shadows widgets. Always census before touching the pipeline.
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Surface facts** (host):
|
||||
```bash
|
||||
pdfinfo file.pdf # "Form: AcroForm" hints at a widget layer
|
||||
pdftotext -layout file.pdf - # label lines + underscore rules = intended fields
|
||||
```
|
||||
2. **Widget census** — authoritative check. With the franksign sidecar image,
|
||||
the image ENTRYPOINT *is* `inject_fields.py`; override it:
|
||||
```bash
|
||||
podman run --rm --entrypoint python -v ~/Downloads:/in:z \
|
||||
localhost/franksign-pdf-field-sidecar:local -c "
|
||||
import pymupdf
|
||||
d = pymupdf.open('/in/file.pdf')
|
||||
for p in d.pages():
|
||||
for w in p.widgets():
|
||||
print(w.field_name, w.field_type_string, [round(v) for v in w.rect])
|
||||
print('widgets:', sum(1 for p in d.pages() for _ in p.widgets()),
|
||||
'annots:', sum(1 for p in d.pages() for _ in p.annots()))
|
||||
"
|
||||
```
|
||||
3. **Fill test**: if widgets exist, `fillTemplate()` (pdf-lib) consumes the file
|
||||
AS-IS — spec.json + sidecar are unnecessary for it. Flatten-verify below.
|
||||
|
||||
## Flatten audit gotchas
|
||||
|
||||
- `pdfinfo` prints `Form: AcroForm` even on correctly flattened pdf-lib output —
|
||||
pdf-lib `flatten()` removes every *widget/annotation* but leaves the field
|
||||
*dictionary*. Truth = MuPDF widget enumeration (0), not the pdfinfo line.
|
||||
- `describeForm(bytes, specJson)` in `packages/pdf` parses the SPEC, not the
|
||||
PDF, for the field list — pass a spec or skip it; calling it spec-less throws
|
||||
`JSON Parse error`.
|
||||
|
||||
## Decision rule
|
||||
|
||||
- widgets > 0 → the widget layer IS the spec (names/types/rects, flip y to
|
||||
bottom-left origin); NEVER run `inject_fields.py` over it.
|
||||
- widgets = 0 → genuine flat form: sidecar injection with a spec (position
|
||||
detection = issue #14 territory: underscore-run rects, label-derived names).
|
||||
- Missing signature widget is normal ("please sign" is prose); franksign
|
||||
decision 19: the typed full name is the signature.
|
||||
|
||||
## Privacy
|
||||
|
||||
Real documents are personal: scratch copies in /tmp, placeholder fill values,
|
||||
destroy scratch (`rm /tmp/*` is fine OUTSIDE the repo; repo deletions still go
|
||||
to `.AGENTTRASH/`). Never commit the document, a spike script touching it, or
|
||||
its extracted text into the vault.
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: plugin-git-install-committed-artifact
|
||||
description: "Use when shipping an omp/pi plugin for omp plugin install git: — the installer clone fails validation (missing build artifact) or silently reuses a stale bun git-mirror cache; recipes for committing dist + drift pins + cache purge."
|
||||
description: "Use when shipping or landing an omp/pi plugin (omp plugin install/upgrade git): install validation fails on a missing build artifact, a stale bun git-mirror cache is reused, the live bundle needs proof it is the merged build, or upgrade says \"not installed\"."
|
||||
---
|
||||
|
||||
# Plugin install from git: committed artifact + fresh mirror
|
||||
@@ -42,6 +42,20 @@ GIT_TERMINAL_PROMPT=0 git clone -q <url> /tmp/x && git -C /tmp/x log --oneline -
|
||||
2. Plain unattended clone contains the entry? If yes, the installer's view is cached → purge mirror.
|
||||
3. Only then suspect the installer itself (validation code lives in the minified bundle; `grep -oE '.{600}<error string>' "$bunfile"` on `readlink -f $(which omp)` shows the exact check).
|
||||
|
||||
## Landing a merged PR on the live box
|
||||
|
||||
After the merge and the `--ff-only` on the dev checkout (`ls -ld node_modules` after landing anything that could have touched it):
|
||||
|
||||
- **`omp plugin upgrade <name>` takes the PACKAGE name, not the install alias.** `omp plugin upgrade dankvoice` fails `dankvoice is not installed` while the live copy sits in `~/.omp/plugins/node_modules/omp-dankvoice/`. Read the directory name (or `package.json` `"name"`) and upgrade that.
|
||||
- Then close the chain of custody with one command — the deployed bytes are the merged bytes:
|
||||
```bash
|
||||
cmp ~/Projects/<repo>/dist/index.js ~/.omp/plugins/node_modules/<pkg>/dist/index.js
|
||||
```
|
||||
If only docs/tests changed, `dist/` is untouched and this should already match; if `src/` changed, the plugin must be upgraded first or this fails with `char <n>` — which is a real finding (stale live bundle), not a test bug.
|
||||
- The session that did the merge keeps the OLD bundle in memory: it is still running pre-merge code. Never conclude "the change does not work" from the merging session.
|
||||
|
||||
## After install — proof without a TUI
|
||||
- `omp plugin doctor` → per-plugin ✔ line.
|
||||
- Audible/behavioral chain when TUI input is unusable (update-banner modal freezes pty submits): run `omp --mode=rpc --no-session` with the feature env armed (dankvoice: `DANKVOICE_AUTO=1`, gate is print/json-only) and watch for objective side effects, e.g. PipeWire: `pw-dump | grep -q "say-<ms>.wav"` proves synth+playback end-to-end.
|
||||
- When pty submits DO work (no modal), the strongest instrument is a stub player plus a clip count, not ears or pixels: `DANKVOICE_PLAYER=/tmp/stub.sh` where the stub appends `$1` to a log and exits 0. Launch the DEPLOYED bundle by path (`omp --no-extensions --no-session -e ~/.omp/plugins/node_modules/<pkg>/dist/index.js`) in a neutral cwd with `DANKVOICE_CONFIG` pointed at a scratch file so the operator's settings are untouched.
|
||||
- **Count clips per narration EVENT, not per turn.** One turn can fire several narrations (dankvoice: `message_end` carries prose+thinking, `agent_end` the settled answer; the dedupe only skips byte-identical text, so thinking-on means both speak). A "too many clips" result is usually correct behaviour — check the harness transcript for the turn count before believing a bug. The player deletes each temp wav, so the log line count is the only surviving evidence: park it in `~/.AGENTTRASH/<date>-<topic>/` if it becomes evidence for an issue.
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
---
|
||||
name: radiance-decode-prefill-interference-census
|
||||
description: "Diagnose decode collapsing while another session prefills on a radiance endpoint: phase-ratio step counters, lag test, metric-prefix and scrape traps, and which knob acts on the measured mechanism."
|
||||
---
|
||||
|
||||
Read `core/config.cpp` (help table + validator) in the PINNED image source before writing any radiance flag, enum member, or metric name into a runbook: `config_parse` refuses unknown enum values at boot, so a remembered name can make the recorded arm undeployable. Enumerate members from the validator's error string, never from memory.
|
||||
|
||||
**Run the census on the pinned build's own metric set.** An older pin ranks the wrong mechanism: on radiance 1.0.13 the bucket table put a "collapse" on a shape whose counters did not exist yet, because every counter resets at a restart and the metric set changes with the build. Prove the window is one shape (`up`, or the first sample of a counter only the current build exports) before ranking buckets.
|
||||
|
||||
## Step ledger first, knobs second
|
||||
|
||||
Three counters decide which mechanism you are looking at, and none of them is a rate:
|
||||
|
||||
- `radiance:engine_step_seconds_decode_total / engine_steps_decode_total` = mean **decode-only** step.
|
||||
- `(engine_step_seconds_total - engine_step_seconds_decode_total) / (engine_steps_total - engine_steps_decode_total)` = mean **chunk step** (a chunk alone, or a chunk with decode rows beside it).
|
||||
- `engine_steps_held_total` — if it is zero, nothing is protecting decoders.
|
||||
|
||||
If decode-only steps stay ~15 ms while the rate collapses, the mechanism is **step-time sharing** (a decode row in a chunk step waits the chunk's whole time), and any eviction/residency story is worth ~10 % of one step. The residency half is not censusable: the arena / expert slab / heat engine / prefill stager have **no metric at all** — they are read from the boot ledger, `/stats`, and `/diagnostics`.
|
||||
|
||||
## Bucket, then lag-test, before believing a cause
|
||||
|
||||
Bucket `radiance:decode_tokens_per_second` by `radiance:prefill_tokens_per_second` from the *same* scrape, step ≤ 30 s (a 15 s scrape against a 120 s step samples one in eight). Per-stream rate = decode gauge ÷ `vllm:num_requests_running`; the tps gauges are ~1 s decayed batch-pooled windows.
|
||||
|
||||
Then shift the prefill series by lags and re-bucket:
|
||||
|
||||
- lag-0-only ⇒ step-time sharing — the chunk is holding the device.
|
||||
- a lagged tail (2–14 min) ⇒ something evicted (arena loan, expert slab, KV).
|
||||
- bursts in `vllm:num_preemptions_total` ⇒ a preempted sequence stopped and is re-prefilling from token 0: a stall, not sludge.
|
||||
|
||||
Also bucket TTFT (`histogram_quantile` over `vllm:time_to_first_token_seconds_bucket`) by the same series: the decode gauge only sees sequences already running, so an arriving request pays the same steps and shows in no decode number.
|
||||
|
||||
## Metric traps
|
||||
|
||||
- **Latency is `vllm:`, cadence is `radiance:`.** TTFT / inter-token / per-output-token / e2e / decode-time histograms and the running/waiting gauges exist only under the vLLM compatibility prefix. Preemption is `vllm:num_preemptions_total` (counter); `radiance:num_requests_preempted` is the *instant* depth. A `radiance:`-only query finds no latency and no preemption and reads as "the endpoint never stalls anyone".
|
||||
- **A 2-minute mean step time dilutes the chunk** — if ~5 % of steps carry a third of the step time, `step_seconds/steps` cannot move. Use the phase ratio.
|
||||
- **`prefix_hit_cut_tokens_total` is prefill load you created yourself**: on a hybrid (linear + paged) model a hit reuses only to the last linear-state checkpoint, so cached tokens replay as prefill. A high prefill rate is not always a client sending new tokens.
|
||||
- Radiance's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`, `token_queue_wait`) may be exported but unscraped; per-request forensics then mean `curl localhost:7770/stats` on the endpoint's host, and `/diagnostics` answers "is the *box* the limit" in one call (configuration drift, VRAM split, per-card link chain and clocks, findings with fixes).
|
||||
- Rootless container logs: read as the deploy user (`podman logs <svc>`), never `sudo podman logs`; check where the serve script redirects before trusting a log file.
|
||||
- In a temporary/remote shell, zsh aborts on a non-matching glob — quote or guard `/sys/...` wildcards.
|
||||
|
||||
## Knob semantics (radiance 1.8.x, GUIDE §4.5; pinned source is referee)
|
||||
|
||||
- `--prefill-interleave N` — the **decoder dial**: N decode-only steps follow any chunk step taken beside decoders; held steps admit nothing. Costs the prefilling prompt's TTFT and a queued request its first step. Counts as `engine_steps_held_total`.
|
||||
- `--busy-prefill-chunk` — a **prefill-speed dial**, not decoder protection: `auto` reserves 1+spec rows per decoding sequence and 1/8 of the budget per waiting request, and the source refuses to halve for decode rows alone. Never below the prefill stager's smallest pass (named on the boot line) — below it every routed layer reads its pooled experts synchronously.
|
||||
- `--prefill-span N` — needs `--max-num-batched-tokens` = N intervals; re-funds a *lone* prefill only, so it cannot buy back held steps while decoders run, and its extra intervals land on the arena that the expert slab loses.
|
||||
- `--checkpoint-policy` — retention only, three names (`geometric-backoff` | `tip-only` | `keep-all`); snapshot writes happen at every interval whatever the policy says (only `--prefill-span` skips them), so `keep-all` buys a bounded replay for pinned host RAM and no extra PCIe. Shortening `--checkpoint-interval` does NOT fix deep divergence: the geometric ladder's coarsest scale is the largest `interval × 2^k` under the depth, so the deepest gap stays ≈ half the prompt while every cut multiplies the writes.
|
||||
- `--checkpoint-slots` is a **total**, `auto` = policy `held()` + 1, per `--max-num-seqs`, within an eighth of host RAM; `--max-num-batched-tokens` is the blunt rung — it raises the arena the slab loses and its resonance on an expert-tiered payload can be violent.
|
||||
|
||||
## Verify an arm, don't assume it
|
||||
|
||||
Flags may be passed conditionally by the serve script, so prove the flag reached argv (`tr '\0' ' ' < /proc/$(pidof <engine>)/cmdline`), then the counter the knob moves (`engine_steps_held_total` must leave zero), then the bucket pair. A deploy is not evidence a knob engaged. Rollback for each of these is its env line, not a rebuild.
|
||||
@@ -0,0 +1,52 @@
|
||||
---
|
||||
name: radiance-flag-truth-from-pinned-source
|
||||
description: "Before writing a radiance flag, value, or metric name into a runbook, prove it against the pinned image's source and the engine's own boot ledger — catches invented option names and stale defaults."
|
||||
---
|
||||
|
||||
# Radiance flag truth from the pinned source
|
||||
|
||||
Use when a radiance (`codeberg.org/StillDeadcode/radiance`) flag, its admitted values, a resolved
|
||||
default, or a `/metrics` name is about to go into a vault page, env comment, or arm proposal — or when
|
||||
a claim about one is being repeated from memory. Docs and old vault pages go stale; the pinned tree and
|
||||
the boot ledger do not.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. **Get the tree at the running version, not at master.** Read the pin:
|
||||
`grep -E 'FRANK_ARCHON_IMAGE' scripts/frank-archon.env` (digest), then
|
||||
`ssh dank@192.168.0.235 'podman images --no-trunc | grep radiance'`.
|
||||
`git clone --branch <ver>` (or checkout the tag) into `/tmp/rad-<ver>`. A `--depth 1` clone makes
|
||||
every `git log -S` report its own tip — never date a flag's introduction from a shallow clone.
|
||||
2. **Flag table is the admitted set.** `core/config.cpp` holds the help strings (each option's legal
|
||||
names/units live in the value column) and the validator branch refuses anything else at boot, e.g.
|
||||
`--checkpoint-policy` → `geometric-backoff | tip-only | keep-all`. `core/rad_core.h` carries the
|
||||
struct with the compiled-in defaults. `tests/config_test.cpp` asserts the admitted set — it is the
|
||||
fastest oracle for "is `lru` a policy?".
|
||||
3. **Live argv beats the repo env.** `sudo tr '\0' '\n' < /proc/$(pidof radiance)/cmdline` — proves the
|
||||
knob actually reached the engine (an env var typo is invisible from the repo).
|
||||
4. **Boot ledger is the referee for every resolved number.** The engine prints its own arithmetic:
|
||||
`ssh dank@192.168.0.235 'podman logs frank-archon 2>&1 | grep -iE "checkpoint|chunk geometry|pools|arena|stager"'`
|
||||
(root unit, but podman storage is dank's — **no sudo**, sudo reads empty root storage). Lines like
|
||||
*"checkpoint slots: auto: 20 … geometric-backoff keeps 9 for a 200000-token session at a 2048-token
|
||||
interval"* are exact; quote them.
|
||||
5. **Reproduce the engine's arithmetic to check you understood it.** Reimplement the policy/geometry
|
||||
function in a throwaway Python eval (`select()` / `held()` in `core/mem/prefix.cpp`, `plan_chunk` in
|
||||
`core/sched/scheduler.cpp`) and confirm the numbers match the ledger line from step 4. Agreement
|
||||
means the rule is understood; disagreement means the read is wrong. Delete the scratch after.
|
||||
6. **Metric names: query the live target, not the guide.** Latency histograms and preemption live under
|
||||
the **`vllm:`** compatibility prefix, cadence under **`radiance:`**; preemption is the counter
|
||||
`vllm:num_preemptions_total` while `radiance:num_requests_preempted` is an instant gauge. Enumerate
|
||||
with `curl -s localhost:7770/metrics | grep '^# HELP'` on Dankserver, or
|
||||
Prometheus `/api/v1/label/__name__/values?match[]={job="frank-archon"}`.
|
||||
|
||||
## Traps that produced real errors here
|
||||
|
||||
- A policy name invented from prose (`--checkpoint-policy floor`) would be a boot refusal; a stale
|
||||
resolved default (`auto` = 512) had been copied between pages. Both survived only in the vault.
|
||||
- `--checkpoint-policy` governs **retention only**; snapshots are written at every interval regardless
|
||||
(only `--prefill-span` skips writes). Read the call site (`retain()` in `core/mem/prefix.cpp`), not
|
||||
the flag help.
|
||||
- Guide/`GUIDE.md` inside an image can be newer or older than the binary; `/diagnostics`
|
||||
`configuration` and the ledger print the running truth.
|
||||
- Facts belong to the vault page that owns the endpoint (`serverAppConfigs-docs/LLM/frank-archon.md`) —
|
||||
this skill owns the *procedure*, not the numbers.
|
||||
@@ -0,0 +1,59 @@
|
||||
---
|
||||
name: radiance-knob-arm-live-verify
|
||||
description: "Arm a radiance scheduler/cadence knob on a live endpoint and prove it engaged: per-knob /metrics meter, argv-vs-default truth, the three behaviour probes, no bench."
|
||||
---
|
||||
|
||||
# Arming a radiance knob on a live endpoint (no bench)
|
||||
|
||||
Use for a one-flag change on a radiance endpoint (e.g. `PREFILL_INTERLEAVE=0→1`,
|
||||
`BUSY_PREFILL_CHUNK`, `CHECKPOINT_POLICY`) under the "swap, don't instrument" law:
|
||||
edit the trio → deploy the one service → prove the flag reached the engine, the
|
||||
engine's own counter moved, and behaviour survived. Never a bench unless Frank orders one.
|
||||
|
||||
## Before you set a value
|
||||
- **Unset means default, and the engine prints what it resolved.** Grep
|
||||
`/proc/$(pidof radiance)/cmdline` (`sudo tr '\0' ' ' < …`) *and* the container ledger
|
||||
(`podman logs <svc>` as the deploy user — rootless storage, `sudo podman` finds nothing).
|
||||
The ledger line names the resolved value and whether a budget clamped it
|
||||
(`--checkpoint-slots 20 [auto] … of a budget of 31.5 GiB, bound by None`).
|
||||
- **Prove the flag exists in the pinned build** before promising it: `--help` from the
|
||||
running container image, not from a vendored tree. Inventing a name (`floor`) cost a
|
||||
doc rewrite; `config_parse` refuses unknown enums at boot.
|
||||
- Some flags are conditional in the serve script (e.g. `--prefill-interleave` is passed
|
||||
only when the env value is `> 0`), so a zero→nonzero arm adds argv; re-check argv after deploy.
|
||||
|
||||
## Meter map (`/metrics`, names carry the `radiance:` prefix)
|
||||
| change | meter that must move |
|
||||
|---|---|
|
||||
| `--prefill-interleave` | `engine_steps_held_total` (read 0 for a 34.7 h boot while the knob sat at 0; 126 within 4 min of arming) |
|
||||
| `--busy-prefill-chunk` | `engine_prefill_chunks_capped_total` |
|
||||
| step-time contention generally | decode-only `engine_step_seconds_decode_total/engine_steps_decode_total` vs `(step_seconds_total − step_seconds_decode_total)/(steps_total − steps_decode_total)` |
|
||||
| prefix retention | `prefix_hit_cut_tokens_total` (labels: `no_snapshot`), `vllm:num_preemptions_total` |
|
||||
|
||||
Trap: **latency lives under `vllm:`, cadence under `radiance:`.** A `radiance:`-only query
|
||||
finds no TTFT/ITP and no preemption and reads as "nothing stalls". The arena / expert slab /
|
||||
heat engine / prefill stager have **no metric at all** — that half is boot ledger + `/stats`.
|
||||
|
||||
## Post-deploy proof, in order
|
||||
1. argv contains the flag (or the ledger shows the resolved value).
|
||||
2. `/health` 200 (`in_flight`, `queue_cap` come back in the body).
|
||||
3. Three probes on the endpoint itself: exact-reply chat (`temperature 0`, expect the
|
||||
literal word — reasoning tokens are normal in `reasoning_content`); a `tools` call
|
||||
expecting a real `tool_calls` object; a **solid-colour PNG via `image_url`** (never the
|
||||
legacy top-level `images` field) with `max_tokens ≥ 512` or thinking eats the budget and
|
||||
a thinking model reads as blind.
|
||||
4. The knob's meter left zero-state. If it didn't move, the arm did nothing — say so.
|
||||
5. Effect is judged later from real traffic buckets, not from a synthetic load.
|
||||
|
||||
## Deploy cycle gotchas that cost the most time
|
||||
- `deploy.sh` must run from the dev checkout literally on `main`, tree clean including
|
||||
untracked files; it pushes its own `main`. Never from an agent worktree or the target.
|
||||
- Name the service (`--service <svc>`); a bare `deploy.sh <host>` is a fleet-wide drain.
|
||||
- Health binds only after the full load (~100 s here, gate budget much larger) — 503 during
|
||||
load is not a fault.
|
||||
- Post-deploy for a recently-benched LLM service: `systemctl show <svc> -p DropInPaths`
|
||||
must be empty before believing anything.
|
||||
|
||||
## Rollback
|
||||
One env line back to the previous value + redeploy the same service. Write the meter and
|
||||
the rollback into the endpoint's vault page (that page is the owner), never into a new page.
|
||||
@@ -1,111 +1,39 @@
|
||||
---
|
||||
name: radiance-kv-cache-resize
|
||||
description: "Resize the KV cache / VRAM pair on a radiance (StillDeadcode) endpoint: engine-budget-table ceiling, pool contract arithmetic, slab-slope law, ledger-first verification, free token-count probes"
|
||||
description: "Resize a radiance endpoint's KV cache/VRAM: the resolver's funding formula, which knob actually moves the carve, the pinned-RAM ledger to recompute, and the boot-ledger lines that referee the result."
|
||||
---
|
||||
|
||||
# Radiance KV / VRAM pair resize
|
||||
|
||||
Use when changing `--vram-kv-mib`, `--vram-weights-mib`, `--max-num-seqs`,
|
||||
`--max-model-len` or `--kv-cache-dtype` on a radiance endpoint
|
||||
(frank-archon pattern), or when asked "what can we do with the free VRAM".
|
||||
|
||||
## 1. The ceiling is the engine's, never a hand-picked reserve
|
||||
|
||||
`rocm-smi` free space is NOT spare capacity. The boot prints a `VRAM budget,
|
||||
per card` table; read it:
|
||||
## Method
|
||||
- The endpoint is sized by ONE env file (`scripts/<svc>.env`) plus the serve script; the boot ledger prints the split before allocating anything: `VRAM budget, per card` → `claimable` → `elastic` → `experts / kv cache`, then `kv: the cache can grow to X a card -- N tokens of context, M sessions at W`.
|
||||
- **`--vram-weights-mib` / `--vram-kv-mib` no longer exist (1.8.1).** The engine REFUSES to start on them (`core/config.cpp`), so this skill has no direct KV knob; the budget resolver derives everything from the cache bound (`--max-num-seqs` x `--max-model-len`), `--gpu-headroom-mib` and `--host-pool-mib`.
|
||||
|
||||
## The funding formula (radiance 1.8.1, source-cited — re-read the pinned tree before trusting it against another build)
|
||||
```
|
||||
card total 31.86 GiB
|
||||
vram_weights 25.00 GiB stated by --vram-weights-mib
|
||||
vram_kv 5.00 GiB stated by --vram-kv-mib
|
||||
unclaimed 477.30 MiB the headroom, the unusable above, rounding
|
||||
W VRAM budgets total 30.00 GiB of the card's 31.86 GiB, so 1.86 GiB is neither
|
||||
budgeted nor accounted. The driver's context, the activation arena and
|
||||
--gpu-headroom-mib come out of it; anything past those is throughput on the floor
|
||||
W card 0 settled at 2.01 GiB free against 209.19 MiB (the 96.00 MiB
|
||||
--gpu-headroom-mib and 113.19 MiB kept for the static region)
|
||||
capacity = card_total - already_held - drift - headroom - arena - step_staging - hw_queues - prefetch
|
||||
elastic = capacity - static_weights
|
||||
expert_floor = min(plane+spare, max(plane + 2*unit_max - host_room, 0)) # planner.cpp expert_vram_floor
|
||||
host_room = host_pool_mib * MiB (less the file tier's staging when disk tiering is on)
|
||||
kv_cap = max_num_seqs * max_model_len # DEMAND, in tokens
|
||||
kv = min(kv_cap, elastic - expert_floor) # what it actually gets
|
||||
tokens = min over paged groups of n_blocks*block_size # the printed number
|
||||
```
|
||||
Read consequence: **`--max-num-seqs` raises demand, `--host-pool-mib` raises supply, and both are the same lever** — every MiB added to the host pool lowers the expert floor one-for-one and the cache takes it (up to `kv_cap`); the leftover returns to the experts. `--max-num-seqs` alone does nothing to capacity: it raises demand while the floor is unchanged, so the printed tokens stay put. The page's "funds ~91 % of demand" table rows are that cap, not a policy percentage.
|
||||
- Derive the plane from a live ledger instead of guessing: `plane + 2*unit ≈ printed_experts + host_pool_bytes`.
|
||||
- `--gpu-headroom-mib` lowers go to the cache but the engine's settle floor is built from it — 48 MiB measured decode-neutral, 24 untested. 48 is the first rung.
|
||||
|
||||
`ceiling = (settled_free_MiB - engine_floor_MiB) + current_pair_sum`.
|
||||
Worked (dankserver, 2026-10-13): 29,120 + (2,058 − 209) = **30,969 MiB/card**.
|
||||
A pair at the ceiling settles ~330 MiB free at worst case. If a bound cannot be
|
||||
traced to a ledger line, it is a personal reserve — label it and put it in the
|
||||
bats gate as such.
|
||||
## Procedure
|
||||
1. Read `scripts/<svc>.env` for the current pool: `GPU_HEADROOM_MIB` (96), `HOST_POOL_MIB` (44,928 per rank), `MAX_SEQS` x `MAX_MODEL_LEN` (demand), `HOST_POOL_MIB`, `PLACEMENT`.
|
||||
2. Compute the target as tokens: tokens x bytes/token-per-card, where bytes/token = (cache_given_bytes / tokens) from the latest ledger. If the target exceeds `elastic - expert_floor`, ALSO lower `expert_floor` via `--host-pool-mib` (up = more room = fewer resident experts) — a bigger host pool buys cache only while experts remain on the card; once the host absorbs the whole plane, more host RAM buys nothing (measured: +640 MiB moved `vram_kv` byte-identical).
|
||||
3. Edit ONE knob (`GPU_HEADROOM_MIB`), deploy service-scoped: `./scripts/deploy.sh <host> --service <svc>`. The engine reloads everything (~2-4 min); the deployer's `LLM_WAIT` covers it.
|
||||
4. Re-read the ledger: `vram_kv` must be the new number, byte-exact or the knob lost to another claim; `mover: N slots can be lent` tells what the expert slab re-borrowed from the enlarged pool — that steal is the decode cost of the resize.
|
||||
5. Re-run the probe (`scripts/probe-frank-archon-kv.py`): depth 1.75x tokens, then the 2x refusal. Update `<svc>.md` Laws with the measured pair (tokens, given, backing %).
|
||||
6. Under-funded? Raise `HOST_POOL_MIB` in 1,024-MiB steps and redeploy; never touch demand to fake the number.
|
||||
|
||||
## 2. Pool contract (the engine enforces it at REQUEST time, not boot)
|
||||
## Pinned-RAM ledger to recompute BEFORE booting (this is where a resize kills the host)
|
||||
`total_pin = tp * HOST_POOL_MIB + PLE/n-gram table + PREFIX_CACHE_MIB + (statics + arena ~2,048) + checkpoint_store`, with `checkpoint_store = slots * 111.98 MiB` and `--checkpoint-slots auto` = `max_num_seqs * (1 + what the retention policy keeps)` — **it grows with the slot count**, so a slot bump raises the pin ledger twice (state + store). The repo's bats guard recomputes this and asserts `TTM_MIN_MIB >= ledger`; when you change pool or slots, update `TTM_MIN_MIB` (and that guard) in the same commit, or the seam fails. The serve script's own gate refuses to start below `TTM_MIN_MIB`; `ttm-preflight.sh` raises the knob to `TTM_TARGET_MIB`, so the target must clear the new floor with slack.
|
||||
|
||||
`pool_mib >= seqs*max_model_len*rate + 110*seqs + 448`
|
||||
|
||||
- rate/token/rank: **7,488 B fp8**, **14,144 B bf16** (1.89×, not 2× — the QSA
|
||||
block key is 3.25 KiB at BOTH dtypes; it is an index, not attention state).
|
||||
- +110 MiB a sequence of recurrent state (`kv_gdn_state` 54 MiB × 2 copies +
|
||||
conv/tail/ple) and 448 MiB of prefix checkpoints, dtype-independent.
|
||||
|
||||
A pool that fails this **boots, serves short chats, then 400s every long
|
||||
request** as `does not fit the 200000 token context, which leaves nothing to
|
||||
generate into` — radiance refuses, never truncates. Encode it as a bats gate.
|
||||
|
||||
`--kv-cache-dtype` takes ONLY `bf16|fp8`, one dtype for the whole cache (no
|
||||
llama.cpp-style exact-K/quantized-V pair).
|
||||
|
||||
## 3. Slab and KV are one budget; the slab is a SLOPE
|
||||
|
||||
- Idle KV VRAM is the expert cache's spillover: ledger
|
||||
`mover: N slots can be lent out of the KV cache (X GiB)`. A bigger pool steals
|
||||
residency twice over.
|
||||
- Slab MiB ↔ resident experts is linear (~1,300 slots per 1,792 MiB measured);
|
||||
the pinned host pool hands back the same slot count as the slab grows
|
||||
(16,235→17,548 slab with 8,893→7,580 pool). The pinned RAM path is the slow
|
||||
path, so slab MiB = throughput.
|
||||
- Measured law (fp8, 43.5k single-stream, Qwen3.8-Flash-Next w4):
|
||||
22,208 → 3,397, 24,000 → 3,290, 25,600 → **3,584**, 26,071 → **4,379 t/s**.
|
||||
Two points inside the ±3 % noise band are NOT a plateau — never infer a
|
||||
threshold from two arms.
|
||||
- **Decode is a separate axis.** It follows KV dtype/bandwidth and draft
|
||||
acceptance, not the slab: fp8/slab 26,071 decoded 124.1 t/s cold where
|
||||
bf16/slab 22,208 decoded 94.1 (512 tokens, 51.5k). Slot count moves the
|
||||
batch-shared per-stream floor (3 concurrent ≈ 6 t/s; solo ≈ 90–120).
|
||||
|
||||
## 4. Verify: ledger first, then live probes (no scratch server)
|
||||
|
||||
1. Deploy the service; read `podman logs <svc>`: `backs N sequences all at
|
||||
M tokens at once` is the capacity referee; check slab slots, `unclaimed`,
|
||||
`sampler: ... R rows` (= seqs × (1+spec), must stay inside the 25-row wht6
|
||||
exactness bound), `settled at … free against … floor`.
|
||||
2. Boundary probe: N concurrent prompts sized to `max_model_len - gen`, each
|
||||
with a unique marker at the head; expect N/N marker retrieval.
|
||||
3. Prefill/decode probe: repo `scripts/probe-frank-archon-kv.py` (reads the
|
||||
shape from the deployed env, so it self-resizes).
|
||||
|
||||
### Free token-count calibration (no prefill burned)
|
||||
|
||||
`/v1/completions` refuses when **prompt tokens alone** exceed the window —
|
||||
`prompt: 202,777 prompt tokens does not fit the 200000 token context` — and the
|
||||
refusal names the count BEFORE any prefill. It does NOT fire on
|
||||
prompt + max_tokens. Corpus ratio measured: **2.7617 chars/token** (2.5714 for
|
||||
the fox filler), so a 200,000-token window ≈ 514,286–552,286 chars. Calibrate
|
||||
with a probe ABOVE the window; below it the request is accepted and pays real
|
||||
prefill (~22 s).
|
||||
|
||||
### Probe traps that produced false results
|
||||
|
||||
- Generate **≥128 tokens** (512 preferred) for any decode number; at 8–9 tokens
|
||||
`inter_token` is first-token noise — rates like 92.7 vs 85.8 are noise.
|
||||
- Radiance `timings` can carry `null` rates — `(t.get(k) or 0)`, and
|
||||
`j.get("timings") or {}`.
|
||||
- The disk prefix cache (64 GiB) survives restarts; put a random nonce at the
|
||||
HEAD of a "cold" prompt. A warm 197k prefill returns in ~10 s and looks like
|
||||
a 60,000 t/s miracle.
|
||||
- With thinking enabled, a small `max_tokens` yields empty `content` +
|
||||
`finish_reason: stop` on a healthy server; set
|
||||
`chat_template_kwargs.enable_thinking=false` in probes.
|
||||
- `echo ===` in an ssh'd zsh dies on `=`-expansion; it silently truncates the
|
||||
command chain.
|
||||
|
||||
## 5. Ship
|
||||
|
||||
`scripts/<svc>.env` pair + header, serve-script header, bats gates (pair ≤
|
||||
ceiling; pool ≥ contract), docs (runbook, `AGENTS.md` fleet row, LLM index,
|
||||
log.md), `./tests/run.sh`, `deploy.sh <host> --service <svc>`, then the probes.
|
||||
Record every prediction as falsifiable in the env comment and write the verdict
|
||||
there when the probe lands — including when it kills the prediction.
|
||||
## Traps
|
||||
- The cache is ELASTIC: the ledger's `kv cache` line is the ceiling, and `mover: ... slots can be lent out of the activation arena` (987 MiB) means the slab borrows arena first, then cache blocks. A bigger ceiling only helps if the heat engine stops re-borrowing it — check `flex_bytes`/`flex_capacity` movement after load, not just the budget line.
|
||||
- A resize that shrinks the pool mid-session preempts live sequences (`vllm:num_preemptions_total` jumps); a resize that grows it cannot retroactively un-preempt. Deploy at low traffic.
|
||||
- Do not resize the KV pool AND the expert placement in one deploy: the `experts` and `kv cache` lines move together and the attribution dies.
|
||||
- The `claimable` arithmetic (`capacity`) is computed ONCE before allocation; `card total` differs per card (26.0 vs 26.68 GiB on dankserver), so the SLOWER card's number is the budget — read the `[0]` and `[1]` `pools:` lines separately, not just the shared budget table.
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
name: react-prod-state-mismatch-bisection
|
||||
description: Diagnose React SPA prod-build state mismatches (state-derived props update but sibling output stays stale) by driving a real browser and bisecting per-render state via throwaway source instrumentation.
|
||||
---
|
||||
|
||||
# Prod React state-mismatch bisection
|
||||
|
||||
Use when a built SPA shows an internally inconsistent DOM — e.g. `aria-current` flips
|
||||
while a sibling conditional slot stays stale, a click handler runs with zero network
|
||||
activity, jsdom tests pass but the prod journey fails. Reasoning from source fails
|
||||
here; only per-render state truth resolves it.
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Drive the real browser, not tests.** Throwaway `@playwright/test` chromium script
|
||||
beside the package that installs it (e.g. `tests/`), seeded via direct API calls +
|
||||
cookie injection:
|
||||
```js
|
||||
import { chromium } from "@playwright/test"; // resolve from tests/ node_modules
|
||||
const ctx = await browser.newContext();
|
||||
await ctx.addCookies([{ name, value, domain: "127.0.0.1", path: "/" }]);
|
||||
page.on("console", ...); page.on("pageerror", ...); page.on("request", ...);
|
||||
```
|
||||
Run it from that directory; a script at repo root or /tmp hits
|
||||
`ERR_MODULE_NOT_FOUND` for workspace deps.
|
||||
|
||||
2. **Fiber prop probe** — call the handler outside React events to split
|
||||
"delegation dead" from "handler wrong", and read the props the fiber holds:
|
||||
```js
|
||||
await page.evaluate(() => {
|
||||
const el = document.querySelector('[data-testid^="thing-"]');
|
||||
const key = Object.keys(el).find((k) => k.startsWith("__reactProps$"));
|
||||
el[key].onClick(); // or .onChange({target:{value:"x"}})
|
||||
});
|
||||
```
|
||||
|
||||
3. **If props updated but output is inconsistent, suspect falsy-equality, not React.**
|
||||
`undefined === undefined` yields `"true"` for `el.id === selectedId` highlights —
|
||||
a fake "state updated" signal. Dump the slot's raw `innerHTML`, don't trust the
|
||||
derived attribute.
|
||||
|
||||
4. **Instruments beats deduction:** insert temporary `console.log`s in the component
|
||||
source (one in each state-mutating callback, one at render logging the state),
|
||||
rebuild, restart the server, re-run the script. Compare *per-render* values — the
|
||||
render where the value is `undefined` instead of the id names the root cause
|
||||
(usually a server payload keyed differently than the client type, e.g. DB row key
|
||||
`orgId` crossing HTTP where the contract says `id`).
|
||||
|
||||
5. **Also grep every suite teardown for broad `DELETE ... LIKE`/`TRUNCATE` on the
|
||||
shared DB when a test flakes only under uncached concurrent runs
|
||||
(`turbo run test --force`) and passes isolated:** parallel vitest files on one DB —
|
||||
the suite that finishes first FK-cascades away the other's live sessions; the
|
||||
symptom is a spurious 401 (or missing row) in the still-running file. Fix by
|
||||
namespacing emails/ids per suite and scoping teardowns to rows the suite created.
|
||||
|
||||
## Cleanup
|
||||
|
||||
Remove instrumentation, move the throwaway script to `.AGENTTRASH/`, never commit the
|
||||
debug script. Verified on franksign: `GET /api/orgs` leaked `orgId` where the client
|
||||
contract was `id` (fake aria-current true, silent auto-select failure).
|
||||
@@ -0,0 +1,35 @@
|
||||
---
|
||||
name: real-pdf-acroform-census
|
||||
description: "Census a real-world PDF before running it through a flat→AcroForm pipeline: check for hidden widget layers, prove fill/flatten, avoid the pdfinfo ghost."
|
||||
---
|
||||
|
||||
Census recipe for a real-world PDF arriving at a flat-form pipeline (e.g. franksign's sidecar that injects AcroForm widgets). Run BEFORE assuming the document is flat.
|
||||
|
||||
## Sequence
|
||||
|
||||
1. **Metadata**: `pdfinfo file.pdf` — note `Form:` line, producer (`macOS ... Quartz PDFContext` = Preview/Preview-era export), page size.
|
||||
2. **Text layer**: `pdftotext -layout file.pdf -` — underscore rules (`____`) look "flat".
|
||||
3. **Widget census (the decisive probe)** — never trust the looks:
|
||||
```bash
|
||||
podman run --rm --entrypoint python -v ~/Downloads:/in:z <sidecar-image>:local -c "
|
||||
import pymupdf
|
||||
d = pymupdf.open('/in/file.pdf')
|
||||
for p in d.pages():
|
||||
for w in p.widgets():
|
||||
print(w.field_name, w.field_type_string, [round(v) for v in w.rect])
|
||||
print('annots:', sum(1 for p in d.pages() for _ in p.annots()))
|
||||
```
|
||||
Traps: a custom ENTRYPOINT script swallows `python` as argv — override with `--entrypoint python`. Mount a directory, not a single file (MuPDF atomic-save EBUSY).
|
||||
4. **Fill live-fire**: pass the file through the fill step as-is with placeholder values; verify with `pdftotext -layout` that values land on the rules.
|
||||
5. **Flatten audit**: re-run the widget census on the output (expect 0) — do NOT read `pdfinfo`.
|
||||
|
||||
## Laws learned
|
||||
|
||||
- macOS Preview/Quartz exports (and most print-to-PDF paths) **already embed AcroForm widgets** under transparent rects over their drawn underscore rules. Field names are human-meaningful (`date_today`, ...). Injecting widgets over them duplicates/shadows fields — the design-time injection stage must be skipped when a widget layer exists (widget-first detection).
|
||||
- pdf-lib `flatten()` removes every **widget** but leaves the AcroForm **field dictionary**: `pdfinfo` keeps printing `Form: AcroForm` on a fully flattened file. Widget enumeration (MuPDF) is the only truth; pdfinfo is the ghost.
|
||||
- Underscore rules in the text layer are drawings, not fields; the text layer alone cannot tell flat from widget-bearing.
|
||||
|
||||
## Verdicts
|
||||
|
||||
- Widgets present → fill as-is; detection spec = widget layer (names, types, rects; flip top-left→bottom-left origin if the pipeline spec uses bottom-left).
|
||||
- Zero widgets → true flat form; needs position detection (underscore runs + nearest preceding label) before injection.
|
||||
@@ -0,0 +1,49 @@
|
||||
---
|
||||
name: recover-externally-reverted-file
|
||||
description: "Rebuild a worktree file that an external process (stale editor buffer, watcher) keeps reverting mid-session: idempotent script rebuild, verify, immediate commit, cat-file restore instead of checkout."
|
||||
---
|
||||
|
||||
# Recovering a File an External Process Keeps Reverting
|
||||
|
||||
Symptom: edits to one file flip-flop under you — grep/wc/read return two different
|
||||
versions minutes apart; tool edit warnings say "file modified externally" repeatedly;
|
||||
committed content disagrees with reads of the same path. Typical cause on Frank's
|
||||
boxes: an IDE window with a stale buffer auto-saving the old version, or another
|
||||
agent's git operation in a sibling worktree.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. **Stop editing surgically.** Line-anchored `edit` calls are unsafe when the disk
|
||||
identity of the file is unstable. Take one snapshot:
|
||||
`cp <file> /tmp/obs1.bats` (twice, a few seconds apart; differing hashes confirm the flapping).
|
||||
2. **Write an idempotent rebuild script** (`/tmp/rebuild-*.py`) that transforms
|
||||
whichever known version is on disk into the target:
|
||||
- every patch is `sub_if_present(pattern, exact_replacement)` /
|
||||
`insert_before(anchor, block)` / `insert_after(anchor, block)`;
|
||||
- each helper first checks whether the block is already present and skips;
|
||||
- anchors must be taken from the committed/original file text, re-grepped,
|
||||
never from memory of an edit that may have been reverted;
|
||||
- print `ok/skip/FAIL` per patch so a mismatch is visible.
|
||||
3. **Apply + verify + commit in ONE bash invocation**:
|
||||
`python3 /tmp/rebuild.py && <syntax check e.g. bats -c / tsc> && git add <file> && git commit ...`
|
||||
The commit is the stable artifact; the disk is not.
|
||||
4. **Restore from git without checkout** (workspace policy blocks `git checkout/reset`
|
||||
in shared checkouts): `git cat-file blob HEAD:<path> > <path>`.
|
||||
Never `git checkout -- <path>` blindly — it silently nukes uncommitted work.
|
||||
5. **Run consumers immediately after the restore/commit** (tests etc. in the same
|
||||
invocation where practical); re-check content with one hash compare
|
||||
(`md5sum <path>; git cat-file blob HEAD:<path> | md5sum`) when results look impossible.
|
||||
6. If duplicates crept in from double-application (flapping + reruns), dedupe by
|
||||
occurrence span (`src.find(marker)` twice, cut the second span) — full-block dedupe
|
||||
fails when the two copies differ in a byte of comment text.
|
||||
|
||||
## Distinguishing real failure from flapping evidence
|
||||
|
||||
If a test fails, re-run the exact command once before believing it: the failing read
|
||||
may have run against the reverted file, not the committed one. Verify what the runner
|
||||
saw (`git hash-object <path>` vs `git rev-parse HEAD:<path>`).
|
||||
|
||||
## Afterward
|
||||
|
||||
Flag the reverter to the user (open editor buffer / watcher / sibling agent); do not
|
||||
fight it — commits and cat-file restores make it harmless.
|
||||
@@ -1,28 +1,27 @@
|
||||
---
|
||||
name: recover-subagent-yield-payload
|
||||
description: "Recover a subagent's full structured report when it exited without yielding (SYSTEM WARNING) — extract aborted yield payloads from the session JSONL, including harness double-encode and split-JSON quirks."
|
||||
description: "Recover or salvage subagent deliverables: missing yields, payloads misrouted to xd://report_issue, and degraded TAKEN-list echoes."
|
||||
---
|
||||
|
||||
Use when a subagent ends with "exited without yielding" yet the transcript shows it repeatedly calling `yield` (each reply "…[+N]" — payload too big for display). The research is usually COMPLETE; only the yield serialization failed. Known harness quirk: the yield `arguments` get wrapped as `{"data": "<json string>"}`, so an object payload is validated as a string → "expected object, received string" retry loop until the agent dies.
|
||||
# Recovering subagent deliverables
|
||||
|
||||
## 1. Locate the session JSONL
|
||||
Transcript header prints the path: `~/.omp/agent/sessions/<cwd-slug>/<session-id>/<AgentName>.jsonl` (worktrees: cwd slug like `-.omp-wt-wt-...`).
|
||||
Three failure modes seen with scout bulk-content tasks, in order of frequency.
|
||||
|
||||
## 2. Extract every aborted yield payload (Python via eval — shell quoting will mangle these)
|
||||
- Parse each line with `json.loads` guarded by try/except (lines may be non-JSON).
|
||||
- Keep `message.role == "assistant"` messages; in `content`, keep entries with `type == "toolCall"` and `name == "yield"`; collect their `arguments`.
|
||||
- Unwrap: `arguments` may be a dict containing `data` (a JSON string) or a plain string; `json.loads` it.
|
||||
## 1. Misrouted payload: scout wrote the deliverable to `xd://report_issue`
|
||||
Symptom: yield summary says "delivered to `xd://report_issue`" instead of containing the content. That device is write-only (a bug-report channel); the response "Noted, thanks!" fooled the scout into thinking delivery worked.
|
||||
|
||||
## 3. Tolerate broken JSON
|
||||
Naive `json.loads` on `data` can fail "Extra data: line 1 column ~4600" — the payload is a complete object followed by `, "architecture": ..., "report": ...}` (agent emitted a structure whose early brace-closure ended the first object). Recover with `json.JSONDecoder().raw_decode`:
|
||||
1. `part1, end = dec.raw_decode(data.strip())`
|
||||
2. `rest = data[end:].strip()` — if it starts with `,`, parse `part2 = dec.raw_decode('{' + rest.lstrip(', '))[0]`
|
||||
3. `merged = {**part1, **part2}`
|
||||
Do this for ALL yields and keep the largest merged dict (attempts evolve; keys seen: `summary`, `files`, `report`, `architecture`).
|
||||
**Salvage path (often impossible — decide fast, ≤2 probes):**
|
||||
- Transcript jsonl: `~/.omp/agent/sessions/*/*/Tan-*/Tan-*.jsonl` (glob by agent id).
|
||||
- Streamed `toolCall` blocks store only **partial arg fragments** of big writes (`arguments.content` is chunked/empty; `partialArgs` holds deltas). Reassembly of a 140-line write across ~100 fragment rows has never worked — if `content` on the toolCall rows is empty or truncated, **abandon salvage**.
|
||||
- Only reliable in-file content: final assistant text blocks — walk rows for `message.role=="assistant"`, `content[].type=="text"`, filter lines against the delivery contract regex.
|
||||
|
||||
## 4. Persist and deliver
|
||||
Write the recovered `report`/`summary` to a durable file (memory dir or `local://`), then continue synthesis — never re-run the research the dead scout already did.
|
||||
**Prevention (put in scout prompts for bulk content):** "Return the deliverable as plain lines in your FINAL OUTPUT TEXT. NEVER write it to `xd://report_issue`, files, or eval state."
|
||||
|
||||
## Prevention (do this first next time)
|
||||
- While the agent is still alive and looping: write to `agent://<Name>` steering it to "yield immediately, wrap in minimal object or raw markdown, schema-invalid yield acceptable — I read the transcript anyway." It often still can't (the failure is in the harness wrapper, not the agent) → let it be killed and use the extraction above.
|
||||
- Scout default schema wants objects; terse yields from `scout` agents hit this most. Keep expected reports slice-sized.
|
||||
## 2. Degenerate echo: scout returns the TAKEN/dedupe examples verbatim
|
||||
Symptom: candidate lines repeat the "already used" examples given in the prompt, in looping blocks. Write off that batch entirely — do not re-prompt (it repeats). Cover the cluster inline yourself.
|
||||
|
||||
## 3. No yield at all (hung/force-stopped)
|
||||
`history://<agent-id>` renders the transcript but compresses tool payloads to "1 line" — useless for payloads, fine for status. Try `write agent://<id>` with a one-line re-emit instruction ("end your turn by yielding the lines verbatim; do not re-research") exactly once; if the reply is garbage, abandon.
|
||||
|
||||
## Rule of thumb for bulk creative content
|
||||
Scouts add value for *terminology research* (search the universe's rank/offices vocabulary, 2-5 searches), not for drafting hundreds of lines — models echo, spin variants ("dune-mentat of the X" ×14), and misroute. Pattern: scout researches → parent drafts.
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
---
|
||||
name: repo-agent-handoff-doc
|
||||
description: "Use when asked to write or refresh a standing handoff/status doc so agents landing in a repo can continue work isolated from other directories; sections that code cannot reveal, plus moving untracked docs onto a branch without dirtying a shared checkout."
|
||||
---
|
||||
|
||||
# Standing agent handoff doc for a repo
|
||||
|
||||
"Put a complete handoff on project X in its directory so agents there can work
|
||||
isolated" is not a changelog request. The value is the knowledge the code cannot
|
||||
reveal. Discover the repo's own conventions first, then write one **living** file.
|
||||
|
||||
## 0. Fit the repo, do not invent a layout
|
||||
|
||||
`ls -a` + `grep -n '^##' README.md` + `find docs -maxdepth 2` before writing:
|
||||
- existing `docs/` conventions (dated `docs/decisions/*.md` records vs flat pages),
|
||||
- whether an `AGENTS.md`/`CLAUDE.md` auto-loads (if not, the README is the only
|
||||
entry point → the handoff MUST be linked from it, same commit),
|
||||
- the exact numbers you will quote: per-file `grep -c '^@test'`, `package.json`
|
||||
`scripts`, module list from `ls src/`. Never write a test count or command from
|
||||
memory.
|
||||
|
||||
## 1. Section checklist (point of use — each answers a question the next agent hits)
|
||||
|
||||
1. **State line**: the commit it describes (`main` at `<sha>`, "after PRs #N") +
|
||||
the rule that whoever leaves work updates it in the same commit. A stale
|
||||
handoff is worse than none: it gets believed.
|
||||
2. **Commands** — build/typecheck/test with real runtime, plus concurrency
|
||||
hazards (e.g. a suite that rebuilds `dist/` in `setup()` → two `bats` runs
|
||||
corrupt each other's byte-identical pin).
|
||||
3. **Test seam and how to add a test** — where the harness mock lives, scenario
|
||||
names, how scripted UI answers are consumed, env dials that are *ignored*
|
||||
(a `setup()` that clobbers your override), and the red-first rule with the
|
||||
one exception (a characterization test that says so in a comment).
|
||||
4. **The laws, each with the test that holds it** — table form. Distinguish
|
||||
load-bearing invariants from incidental behaviour; this is the section that
|
||||
stops a confident agent "fixing" a chosen design.
|
||||
5. **Worktree/dev-dir recipe** — symlinked deps, gitignore rule that keeps the
|
||||
symlink untracked, why the shared checkout stays clean.
|
||||
6. **Ship path** — branch → tests → rebuild committed artifact → PR → merge on the
|
||||
owner's OK → land with `--ff-only` → deploy (the gotcha that the upgrade target
|
||||
is the PACKAGE name, not the directory alias) → `cmp` the live artifact against
|
||||
the merged build → prune. Include the auth traps of the forge (token via a
|
||||
600 curl `-K` file, empty token answering 404 not 401, form-encoded merge).
|
||||
7. **Live verification** — the instrument that produces evidence (stub player,
|
||||
log line counts, pty submit semantics), and explicitly what is NOT a check
|
||||
(print-mode invocation, screen reads of transient toasts, single-take audio
|
||||
comparisons).
|
||||
8. **Environment reality not visible from this code** — the live endpoints,
|
||||
measured limits, refusals; crosslink the README if it owns the table.
|
||||
9. **Open items** — numbered, each actionable alone, each saying who decides if a
|
||||
human must; name the repo a fix belongs in when it is NOT this one.
|
||||
10. **History** — one line per issue/PR. No tables of metrics.
|
||||
|
||||
One owner per fact: link the README/decision records instead of restating knob
|
||||
tables or rationale.
|
||||
|
||||
## 2. Tree hygiene (the costly step)
|
||||
|
||||
Writing straight into a shared checkout leaves an untracked file on `main` — a
|
||||
half-deployed state where the checkout is the deployment, and a dirty tree for
|
||||
everyone else. Move it onto a branch without switching anyone's HEAD:
|
||||
|
||||
```bash
|
||||
mv <repo>/docs/handoff.md /tmp/handoff.md
|
||||
git -C <repo> fetch -q origin
|
||||
git -C <repo> worktree add -q -b docs/<slug> ~/Projects/wt-<topic> origin/main
|
||||
ln -s <repo>/node_modules ~/Projects/wt-<topic>/node_modules # if deps are needed
|
||||
mv /tmp/handoff.md ~/Projects/wt-<topic>/docs/handoff.md
|
||||
git -C <repo> status --porcelain # shared checkout clean again
|
||||
```
|
||||
|
||||
README pointer edit happens **in the worktree** (an edit there is a separate
|
||||
checkout; editing the shared one dirties a tracked file). Then commit, run the
|
||||
suite, push, open the PR, and merge only on the owner's explicit OK.
|
||||
|
||||
## 3. Voice rules
|
||||
|
||||
Prose in the repo's own style (match an existing decision record). State reasons
|
||||
and consequences, not bookkeeping. No measured numbers for things that drift;
|
||||
keep the law, drop the figure.
|
||||
@@ -0,0 +1,44 @@
|
||||
---
|
||||
name: rescue-loose-home-scripts
|
||||
description: "Promote a durable loose $HOME script into dank/command-center: archive-first, black-box bats red-first with env seams, docs page + index + log entry, one atomic commit per instrument, PR to main."
|
||||
---
|
||||
|
||||
Rescue a throwaway-looking `$HOME` script into the command-center repo as a maintained instrument. Used for `remote-job.sh` + `scripts/probes/*.hip` (PR #54, Refs #53); the archive still holds ~120 candidates.
|
||||
|
||||
## 1. Archive first, never delete
|
||||
|
||||
Loose scripts at `$HOME` root move to `~/.AGENTTRASH/home-scripts-<date>/` (`rm` is forbidden; deletion is a pruning session with Frank). Consolidate bulk media separately (`~/audiosamples`). Then check the archive for **instruments**, not convenience: something that answers a question you will ask again, or that carries a trap you paid for.
|
||||
|
||||
## 2. Tests first, and black-box
|
||||
|
||||
Write `tests/<name>.bats` before the script; confirm red (`bats tests/<name>.bats` → all `not ok` because the script is absent).
|
||||
|
||||
- Assert **output, exit codes, and on-disk ordering** only — never internals, never "variable X equals Y".
|
||||
- A test that asserts a race outcome (file exists immediately after a detached `start`) is wrong: assert the *contract* instead (`status` reports RUN/PENDING and exits 1). Rewrite the test, not the code, when the failure is your own assumption.
|
||||
- Design env seams **so the untestable path becomes testable off the target hardware**: `*_SYSFS`, `*_KFD`, `*_HIPCC`, `*_SRC`, `*_BUILD`. A refusal that can only be exercised on a card box is an untested refusal. Stub the toolchain in `$PATH`/via env to drive build-failure and busy-device paths.
|
||||
|
||||
## 3. Safety law in the tool, not the docs
|
||||
|
||||
Anything that measures a GPU must **refuse before it builds or runs** when the device is busy (`gpu_busy_percent` over threshold, or any process holding `/dev/kfd`) and must **stop nothing**: unloading a production endpoint is a human decision (`--force`). Pin the refusal-before-build ordering in a test — otherwise a reordered guard silently reintroduces contention beside a live service.
|
||||
|
||||
Exit-code taxonomy worth reusing: `0` did the work · `2` usage/missing inputs · `3` refused (would have contended) · `4` toolchain/build failure. A client box without the SDK must exit 4 loudly, never report an empty success.
|
||||
|
||||
## 4. An instrument that crashes is not an instrument
|
||||
|
||||
`hipGetDeviceProperties` **segfaults** on the fleet's ROCm build while `hipGetDeviceCount` works fine. The shipped probes called it only to print device name + PCI bus id — replaced by the sysfs topology block the driver prints (`current_link_width`/`current_link_speed` vs `max_*`). Record the trap as a comment in the sources so it is not reintroduced.
|
||||
|
||||
## 5. Docs, three places, in the same commit as the code
|
||||
|
||||
- `omp-docs/tools/<name>.md` — frontmatter per OKF (`type: reference`, title, description, tags, `verified`, `timestamp`); sections: contract (command table + exit status), the law + why, how to read the output, failure-modes table, test seams. Point-of-use only: measured numbers do **not** live here (they belong with the endpoint's runbook vault).
|
||||
- `omp-docs/tools/index.md` — one alphabetical bullet per page.
|
||||
- `omp-docs/log.md` — dated entry above the newest one: what changed + the durable laws, a few lines.
|
||||
|
||||
Gates before commit: `bash scripts/kb-lint.sh` (0 violations) and full `bats tests/` green.
|
||||
|
||||
## 6. Land it
|
||||
|
||||
One commit per instrument (stage that page's index line only — re-add the next page's line for its own commit), message = imperative what + why + `Refs #N`, hooks armed (never `--no-verify`). Push to the local Gitea, open the PR with `gitea-pr-api-cycle` (token from `~/.config/secrets/gitea` via curl `-K`; body ends with `---` then `cwd: <session cwd> · session: <live transcript UUID>`), then `xdg-open <PR-URL>`. **Never merge without Frank's explicit OK.**
|
||||
|
||||
## 7. Report the gap you could not close
|
||||
|
||||
Verification on a client box proves the guard/refusal/usage paths only. If compile-or-run needs the hardware box, say so in the PR under "Not done here" and offer the exact low-risk command rather than implying the artifact is proven.
|
||||
@@ -0,0 +1,39 @@
|
||||
---
|
||||
name: rng-pool-sweep-blackbox-test
|
||||
description: "Prove membership/absence in a randomly sampled pool (epithets-style) from black-box tests: prime-cycled Math.random sweeps every index; pool-first probe for setting precedence."
|
||||
---
|
||||
|
||||
# Deterministic pool sweeps for random-sampling black-box tests
|
||||
|
||||
When a built bundle samples uniformly at random from a curated pool (epithet-style) and tests must prove a setting admits/excludes whole classes of pool entries, do NOT write probabilistic sampling tests. Two proven designs:
|
||||
|
||||
## 1. Full-pool sweep (class membership, distinct counts)
|
||||
|
||||
In the test driver (the process that imports the built bundle), before firing utterances:
|
||||
|
||||
```ts
|
||||
let seed = 0;
|
||||
Math.random = () => (seed++ % PRIME) / PRIME; // PRIME > pool size, e.g. 1009
|
||||
```
|
||||
|
||||
The extension calls `Math.random()` at speak time, so the patch applies regardless of import order. Picks are `floor(random * poolSize)`; with N=PRIME samples the step is `< 1`, so **every index of every pool ≤ PRIME is visited exactly once**. Observable per-utterance text (e.g. the fixture server logs the synthesized `input=`) then reveals the entire sampling pool:
|
||||
|
||||
- distinct picks == pool size exactly (pin the total only if it is itself law);
|
||||
- a sentinel name from class X appears / must never appear per mode — deterministic, zero flake;
|
||||
- absence proofs are exhaustive, not lucky.
|
||||
|
||||
Drive one narration per iteration and gate the loop on the fixture's call counter advancing (`while (speechCount() < before + expected) await nap(2)`), otherwise typed/coalescing paths collapse iterations and counts drift.
|
||||
|
||||
**Wall-clock budget:** one sweep ≈ 20–40 s per pool; bats serializes and jobs cap at 600 s — run at most two sweeps per backgrounded bats `-f` selection, or launch full suites with `timeout: 0`.
|
||||
|
||||
## 2. Pool-first probe (setting precedence, milliseconds)
|
||||
|
||||
With the same patched RNG, the FIRST pick is `pool[0]`. Run each config mode with exactly one utterance and compare the first spoken name:
|
||||
|
||||
- mode-A setting vs env-mode-A → equal (settings key honored);
|
||||
- env-X overriding file-Y → equals env-X's pick (precedence law);
|
||||
- pick strings are pool literals that live in exactly one class (e.g. `comrade operator`).
|
||||
|
||||
Avoid: regex scans over spoken text (possessives/compounds false-fire), probabilistic "absent in 300 samples" tests (flake floor unless the block is contiguous), and reading source internals (white-box).
|
||||
|
||||
Red-first check: the sweep test for a not-yet-filtered pool must fail by SPEAKING the forbidden sentinel — if a "new mode" test passes pre-implementation, the probe accidentally tautologized (e.g. old pool[0] == new pool[0]); pick probe literals whose values differ between old and new behavior. Beware bats `setup()` rebuilding the bundle per test file: a long first sweep lets mid-session src edits leak into later tests' "red" proof — prove red from a pristine tree before implementing.
|
||||
@@ -0,0 +1,65 @@
|
||||
---
|
||||
name: rootless-to-root-quadlet-journal-read
|
||||
description: "Convert a rootless Podman Quadlet (e.g. Alloy) to a ROOT system service so it can read the host journal — the machine-id mount, Network=host for loopback-published Loki, root image mirror, and the user-unit cutover. Use when an sd_journal/loki.source.journal reader ships zero lines with a green health check."
|
||||
---
|
||||
|
||||
# Rootless→root Quadlet so a container can read the host journal
|
||||
|
||||
Symptom: a rootless Quadlet that scrapes the systemd journal (`loki.source.journal`,
|
||||
any `sd_journal` reader) ships **zero** lines — silently, `health` green,
|
||||
`loki_source_journal_*_lines_total 0`, no error in the log.
|
||||
|
||||
## Why (proven)
|
||||
- Journal files are `root:systemd-journal 0640` (`group:adm`, `other::---`). A rootless
|
||||
container spawns with no supplementary groups → `sd_journal` opens an empty view.
|
||||
Proof a rootless read is denied: `podman run --rm -v /var/log/journal:...:ro busybox
|
||||
cat /var/log/journal/*/*.journal` → `DENIED`.
|
||||
- Rootless Quadlet **container** logs also live in `user-<uid>.journal` (journald log
|
||||
driver), not `ctr.log`. A system-journal-only read misses them.
|
||||
|
||||
## The shape that works (root, not rootless)
|
||||
Run it as a **root system Quadlet** (`/etc/containers/systemd/<svc>.container`,
|
||||
`WantedBy=multi-user.target`). Root's DAC override opens every journal file, so one
|
||||
`loki.source.journal` reads `system.journal` **and** `user-*.journal` together.
|
||||
|
||||
Four load-bearing bits (all validated on Dankclaw Alloy 2026-10-20):
|
||||
1. **`Volume=/etc/machine-id:/etc/machine-id:ro`** — the single biggest gotcha.
|
||||
`sd_journal` locates the journal dir by the *local* machine id; without it the
|
||||
container opens **nothing** and reads exactly 0 lines, health green. A root probe
|
||||
without it = 0; with it = 31,825 lines in 20s.
|
||||
2. **`Network=host`** — if the sink (Loki) publishes `127.0.0.1:3100` (loopback), no
|
||||
bridge container (root or rootless) can reach the host loopback; `host.containers.internal`
|
||||
and the LAN IP are both refused. Host network + push to `http://127.0.0.1:3100`.
|
||||
Side effect: bind the app UI to `127.0.0.1` (no `PublishPort`) so it stays loopback-only.
|
||||
3. **`/var/log/journal` + `/run/log/journal`** mounted `:ro`.
|
||||
4. **Image must be in the ROOT store.** A root `podman build` runs in the build
|
||||
container's own resolver and fails DNS here. Build rootless, then mirror:
|
||||
`podman save -o t.tar <ref> && sudo podman load -i t.tar`. Compare image ids so a
|
||||
rebuild actually re-lands (stale root tag is a silent footgun).
|
||||
|
||||
## Deployer cutover (retire the rootless unit FIRST)
|
||||
Both the user and system managers generate `<svc>.service`; two containers named
|
||||
`<svc>` both binding the same loopback port cannot coexist. Before starting the system
|
||||
unit: `systemctl --user stop <svc>.service`; `mv` the user quadlet out of
|
||||
`~/.config/containers/systemd/` (it regenerates from there) to `~/.AGENTTRASH/<date>-<svc>-root-cutover/`;
|
||||
`systemctl --user daemon-reload`. Then install to `/etc/containers/systemd/`,
|
||||
`sudo systemctl daemon-reload`, `sudo systemctl restart <svc>.service`. Generated
|
||||
(system) units cannot be `enable`d here — boot persistence is the `[Install]` section.
|
||||
|
||||
## Probe the exact command BEFORE writing the unit
|
||||
`sudo podman run --rm --net=host -v /etc/machine-id:...:ro -v journal mounts
|
||||
<image> run --server.http.listen-addr=127.0.0.1:<free-port> <config>` — pick a FREE UI
|
||||
port (the deployed instance already owns the real one; a bind conflict reads as
|
||||
"reads 0 lines", not "port in use"). A squashed one-line River/HCL block is invalid —
|
||||
attributes need newline terminators, or the probe dies before reading anything.
|
||||
|
||||
## Verify (query gotchas that caused false "empty" reads)
|
||||
- `loki_source_journal_target_lines_total` must climb.
|
||||
- Loki `/loki/api/v1/query_range` returns `values` as `["<ts_ns>","<line>"]` **arrays** —
|
||||
grep `"ts"` finds nothing; the data is there. `/series` defaults to a short index window
|
||||
and can read empty while recent data is queryable. Use a wide `start`/`end`, no `direction`.
|
||||
- A fresh journal reader replays the whole persistent journal from the oldest; Loki
|
||||
`reject_old_samples` + the per-tenant rate limit discard old/throttled entries
|
||||
(`loki_discarded_samples_total{reason="rate_limited"}`) without counting as writer
|
||||
drops. It self-clears once it reaches the tail; raise the ingestion rate to speed it or
|
||||
start at the head to skip.
|
||||
@@ -0,0 +1,41 @@
|
||||
---
|
||||
name: rotate-fleet-service-host-secret
|
||||
description: "Rotate a Dank-fleet rootless-quadlet service's host-side secret (e.g. open-notebook OPEN_NOTEBOOK_PASSWORD) safely: confirm the deployer's secret seeder is idempotent first so a future deploy won't clobber the new value, change ONLY the target line while holding encryption keys byte-identical, then restart the user unit and prove the rotation (new token 200, old token 401, no token 401). Not for git/Gitea tokens or LLM endpoint secrets."
|
||||
---
|
||||
|
||||
# Rotate a fleet service's host-side secret safely
|
||||
|
||||
Host secrets live OUTSIDE the repo, in `/home/dank/Containers/<svc>/env` (mode 600). The repo only holds the knobs (`scripts/<svc>.env`, secrets declared EMPTY). Changing a secret = edit the host file, never the repo.
|
||||
|
||||
## 1. First: prove the deployer won't clobber it (MANDATORY)
|
||||
`deploy-remote.sh` seeds some secrets from `dankclaw/.env`. Confirm the seed fn is **idempotent** before editing, or a later `deploy.sh` overwrites your value. In `scripts/deploy-remote.sh`, `seed_<svc>_env` must look like open-notebook's:
|
||||
|
||||
```bash
|
||||
for var in A B C D; do grep -q "^${var}=." "$dest" 2>/dev/null || missing=1; done
|
||||
[[ "$missing" == 0 ]] && return 0 # all present → leaves the file untouched
|
||||
if [[ ! -e "$dest" ]]; then ...seed from dankclaw/.env...; fi
|
||||
```
|
||||
|
||||
If it returns-early on all-present, editing `$dest` directly is deploy-safe. If it seeds unconditionally, edit BOTH the host file and `dankclaw/.env` instead.
|
||||
|
||||
## 2. Read, then rewrite only the target line
|
||||
`read ssh://<host>/home/dank/Containers/<svc>/env` (never print secrets to chat — they leak into the transcript). Then `write ssh://...` overwriting with **all** lines, changing only the one secret. **Hold encryption keys byte-identical** (e.g. `OPEN_NOTEBOOK_ENCRYPTION_KEY`): a DB-encrypted credential store is unreadable if that key changes. Then `chmod 600`.
|
||||
- ssh:// `read` emits NO edit tag, so the `edit` tool can't target it — use `write` (full overwrite) with the exact values from the read.
|
||||
|
||||
## 3. Restart the right unit
|
||||
Rootless quadlets are USER units: `ssh <host> systemctl --user restart <svc>.service`. (systemctl --user works over plain ssh here — linger + bus are set up.) Restart only the app unit; DB password unchanged means the DB unit needs no restart.
|
||||
|
||||
## 4. Prove the rotation behaviorally (this is the smoke test)
|
||||
Hit a protected API route three ways, expecting **new=200 / old=401 / none=401**. Pass the bearer via a mode-600 curl config + `-K` (never `-H "Bearer …"` — argv leak):
|
||||
|
||||
```bash
|
||||
ssh <host> 'cfg=$(mktemp); chmod 600 "$cfg"
|
||||
printf "header = \"Authorization: Bearer %s\"\n" "<new-or-old>" > "$cfg"
|
||||
curl -s -o /dev/null -w "%{http_code}\n" -K "$cfg" http://127.0.0.1:5056/api/<protected>'
|
||||
```
|
||||
|
||||
## Gotchas that bit
|
||||
- **Health is on the API loopback port, not the web port.** open-notebook: `/health` answers on **127.0.0.1:5056** (the FastAPI), NOT `:8502` (Next.js, which 404s `/health`). Probing the wrong port reads as "service failed" when it's up and `healthy`. Excluded/auth-free paths (`/`, `/health`, `/docs`, `/openapi.json`, `/redoc`) — everything else needs the bearer.
|
||||
- **`OPEN_NOTEBOOK_PASSWORD` unset = auth OFF** for anything that can reach the port (other containers dial it by name on `dankclaw_default`), so it's mandatory, not cosmetic; deployer refuses to start on blank.
|
||||
- Browser/clients keep the old secret in localStorage → they 401 on next call and re-prompt. Expected.
|
||||
- **Harness command-parser gotcha:** adjacent/interleaved single+double quotes in one bash line (`'...'"$x"'...'`) crash `pi-natives:command: unterminated quote`. Build a curl-config `header=` line with the proven single-quoted printf (`printf 'header = "Authorization: token %s"\n' "$tok"`) and put long bodies in a file via the `write` tool + `jq --rawfile`, not a giant inline `$(printf …)`.
|
||||
@@ -0,0 +1,61 @@
|
||||
---
|
||||
name: run-e2e-specs-from-isolated-worktree
|
||||
description: "Run serverAppConfigs Playwright e2e specs against the live fleet from a feature branch when git checkout is workspace-policy-blocked — isolated worktree + symlinked deps, with the variadic --project and gitignored-auth traps."
|
||||
---
|
||||
|
||||
## When
|
||||
|
||||
You are authoring/verifying a `tests/e2e/*.spec.ts` change for `dank/serverAppConfigs` and the shared workspace blocks `git checkout` / branch switch (concurrent-agent safety). You must run the new specs against the LIVE fleet (they are guard-chain tests: Anubis PoW + authentik outpost + app) but a fresh worktree has no `node_modules`, no `.auth`, and no `.env.local`.
|
||||
|
||||
This is NOT `deploy.sh` (which runs the full e2e gate itself from the main-tracking checkout — see `serverappconfigs-deploy-execution-context`). Use this to iterate/verify individual specs on a branch before opening a PR.
|
||||
|
||||
## Setup the worktree
|
||||
|
||||
The main deploy checkout (has the deps + auth state) is `/home/blank/Projects/serverAppConfig`. Branch off `origin/main` into a throwaway worktree:
|
||||
|
||||
```bash
|
||||
git -C <repo> worktree add -b feat/<topic> /tmp/wt-<topic> origin/main
|
||||
ME=/home/blank/Projects/serverAppConfig/tests/e2e
|
||||
ln -sfn "$ME/node_modules" /tmp/wt-<topic>/tests/e2e/node_modules
|
||||
cp -r "$ME/.auth" /tmp/wt-<topic>/tests/e2e/.auth
|
||||
cp "$ME/.env.local" /tmp/wt-<topic>/tests/e2e/.env.local # gitignored
|
||||
```
|
||||
|
||||
- **`.env.local` is mandatory** — without it the config throws `missing E2E_STANDARD_USER — set it in tests/e2e/.env.local or the environment` at load, even for `--list`. It is gitignored (a bats seam forbids tracking it), so a fresh worktree never has it.
|
||||
- `.auth` holds `qa-e2e.json` / `qa-e2e-admin.json` (the `standard`/`admin`/`guest` storage states); copying it avoids re-login, though the `setup` project refreshes it anyway.
|
||||
- The node_modules symlink resolves fine for playwright.
|
||||
|
||||
## Validate wiring before running
|
||||
|
||||
```bash
|
||||
cd /tmp/wt-<topic>/tests/e2e && npx playwright test --list
|
||||
```
|
||||
|
||||
Confirms your new spec's NUMBER PREFIX is matched into a project. **A spec whose prefix is not in any project's `testMatch` silently never runs** — a green suite with a test that does not exist. The gate is the bats seam `tests/e2e_scaffold.bats` → "every e2e spec is matched by a playwright project". So a new spec `21-foo.spec.ts` MUST also edit `playwright.config.ts`'s `standard` (or other) `testMatch: /(10|12|...|21)-.*\.spec\.ts/`.
|
||||
|
||||
## Running — the `--project` trap
|
||||
|
||||
`--project` is **variadic**: the space form swallows the positional filters as extra project names.
|
||||
|
||||
```bash
|
||||
# WRONG: "21-notebook" parsed as a second project → Project(s) not found
|
||||
npx playwright test --project standard 21-notebook 22-foo
|
||||
|
||||
# RIGHT: equals form, then positional title/file filters
|
||||
npx playwright test --project=standard 21-notebook 22-foo
|
||||
npx playwright test --project=guest 14-guard -g "notebook.franksdank.cloud is gated"
|
||||
```
|
||||
|
||||
`standard`/`admin`/`guest`/`frankbook` all `dependencies: ["setup"]`, so `00-auth.setup.ts` runs first and logs in the qa-e2e accounts (~24 s each). Expect ~50 s wall for a couple of specs.
|
||||
|
||||
## Verify the spec is meaningful, not vacuous
|
||||
|
||||
These specs prove the guard chain handed off to the **app**, so assert a positive app marker (title / app content string / the app's own API returning `status:"success"`), never just `status < 500` or "not the IdP shell" — the latter passes on a 404 or a parked login wall (the 13-sso / 17-hermes class of silent-green bug).
|
||||
|
||||
## Cleanup
|
||||
|
||||
```bash
|
||||
git -C <repo> worktree remove --force /tmp/wt-<topic> # symlink + copied auth are throwaway; branch already pushed
|
||||
```
|
||||
|
||||
Full `./tests/run.sh` (bats, incl. `e2e_scaffold`) must pass before commit; the deploy's own Playwright gate is the canonical live run after merge.
|
||||
@@ -28,14 +28,14 @@ site-packages/flexml/flexml_extras/lib : site-packages/onnxruntime/capi : site-p
|
||||
1. `ldd <deployment>/lib/libonnxruntime_providers_ryzenai.so | grep -E "libxrt_coreutil|libryzen_mm"` — the resolved path settles it alone: `/opt/xilinx/xrt/lib` = good, `.../voe/lib` = this bug.
|
||||
2. `XILINX_XRT=/opt/xilinx/xrt /opt/xilinx/xrt/bin/xrt-smi examine` → `RyzenAI-npu5 | aie2p | 6x8` + `NPU Firmware Version`. Needs `libboost_filesystem1.83.0`, `libboost_program_options1.83.0`, `libuuid1` (26.04 names; the 24.04 docs' 1.74.0 does not exist). A "No such file or directory" for a file that exists = missing interpreter/dep, not a missing binary.
|
||||
3. `strace -f -e trace=openat` — `/dev/accel/accel0` never opened ⇒ failure is pre-device ⇒ loader-level.
|
||||
4. `gdb -q -batch -x <(echo 'catch throw'; echo run; echo 'bt 10')` style catch-throw loop → names `shim_loader` / `Allocator::Alloc`.
|
||||
4. `gdb -q -batch` with `catch throw` + `bt 10` loop → names `shim_loader` / `Allocator::Alloc`.
|
||||
|
||||
Known dead ends (each costs one run to rule out, then stop): `--ulimit memlock`, XRT INI verbosity, `DD_LOG_LEVEL` / `RYZENAI_TRACE` / `RYZENAI_EP_TRACING` (print nothing before the throw), byte-scanning XRT `.so` for PCI IDs (ambiguous), libboost presence for the OGA path (only `xrt-smi` needs it), flag-by-flag container matrices (`--memory`, `--security-opt`, `--name`, extra mounts — all innocent).
|
||||
Known dead ends (one run each, then stop): `--ulimit memlock`; XRT INI verbosity; `DD_LOG_LEVEL` / `RYZENAI_TRACE` / `RYZENAI_EP_TRACING` (print nothing before the throw); byte-scanning XRT `.so` for PCI IDs (ambiguous); libboost presence for the OGA path (only `xrt-smi` needs it); flag-by-flag container matrices (`--memory`, `--security-opt`, `--name`, extra mounts — all innocent).
|
||||
|
||||
## Verified facts on this stack
|
||||
|
||||
- Vendor XRT 2.25.37 userspace (`dpkg-deb -x`, never dpkg-installed) works against Ubuntu's **in-tree `amdxdna` 0.7.0`; no DKMS module needed. `1022:17f0` rev 11 = XDNA2, handled as `aie2p 6x8`.
|
||||
- Vendor XRT 2.25.37 userspace (`dpkg-deb -x`, never dpkg-installed) works against Ubuntu's in-tree `amdxdna` 0.7.0; no DKMS module needed. PCI `1022:17f0` rev 11 = XDNA2, reported as `aie2p 6x8`.
|
||||
- AMD's `quicktest` passing does **not** prove NPU use (it can silently run on CPU); trust `xrt-smi` + logits, not "it didn't crash".
|
||||
- `/dev/accel/accel0` is `root:render 0660`; rootless podman cannot read it (userns drops the gid; `keep-groups`, `keep-id:gid=990`, `--group-add render` all fail) → rootful container, or a udev rule like the GPU nodes'.
|
||||
- OGA 0.14 (RAI 1.8) contract: `GeneratorParams` has only `set_search_options` (no `input_ids`, no `model.generate`); feed with `Generator.append_tokens(ids)`, loop `generate_next_token()`, read `get_sequence(0)`; `max_length` counts **prompt + generated**. Full Fusion (`*_npu_4K`) caps input+output at 4096.
|
||||
- Validate the *content*, not just the exit code: `np.argsort(logits[0])[-8:]`. `append_tokens` returning a real logits tensor proves the graph ran; if argmax is the sequence terminator (`
|
||||
- Validate content, not just exit code: `np.argsort(logits[0])[-8:]`. A real logits tensor after `append_tokens` proves the graph executed (prefill is folded into step 1, so a sub-millisecond first step + a 1-token sequence means it did not); if argmax is the sequence terminator (`
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: ryzenai-npu-oga-linux-spike
|
||||
description: "Run a vendor-compiled AMD Ryzen AI (XDNA2) ONNX-GenAI model on a Linux Strix Halo host from public artifacts: staging, rootless NPU device-access wall, container recipe, and the NuTTS-air speech-token contract."
|
||||
description: "Run an AMD-precompiled AMD Ryzen AI (XDNA2) ONNX-GenAI model on a Linux Strix Halo host from public artifacts: staging, rootless NPU device-access wall, container recipe, and the NuTTS-air speech-token contract (65,536-code vocab, 50 codes/s real-time line)."
|
||||
---
|
||||
|
||||
Run an AMD-precompiled XDNA2 model (OGA / onnxruntime-genai) on a Linux Strix Halo host (Dankclaw-class) without installing the vendor stack on the host.
|
||||
@@ -19,10 +19,12 @@ Tarball has 30 top-level entries: cp312/linux_x86_64 wheels (`onnxruntime_genai_
|
||||
- **Rootless containers cannot read `/dev/accel/accel0`** (`root:render 0660`): the host gid never enters the userns gid map, so `--device` yields `nobody:nogroup`. Verified failing 2026-10-19: `--group-add keep-groups`, `--userns=keep-id`, `--userns=keep-id:gid=990` (process gets gid 990, node still unreadable), `--group-add render`, `--group-add gid:990` (both error). Options: run rootful (`sudo podman`, separate storage → rebuild the image rootful, ~1 min), or add a world-writable udev rule for `accel0` (the fleet already does exactly this for `kfd`/`renderD*` in `99-comfyui-gpu-access.rules`, whose header explains why: systemd user-unit spawns carry no supplementary groups).
|
||||
- **Rootless image build trap:** the vendor installer's `tar -xf` preserves recorded uids (95777 etc.) and aborts under `set -e` with `Cannot change ownership ... Invalid argument`, silently skipping the `deployment`/`onnxruntime` extraction. Patch it in the Containerfile: `sed -i 's/tar -xf/tar --no-same-owner -xf/g' install_ryzen_ai.sh`.
|
||||
- Installer usage: `./install_ryzen_ai.sh -a yes -p /opt/rai -c /opt/rai` (python **3.12** required; base `python:3.12-slim`; needs `linux-libc-dev zip pciutils patch libgomp1`). It builds a venv from the local wheels, reinstalls `onnxruntime-vitisai` then `--no-deps onnxruntime-genai-ryzenai`, installs `requirements_extras.txt` (transformers 4.57.6), patches `bin/activate` (sets `RYZEN_AI_INSTALLATION_PATH=$VIRTUAL_ENV` + `LD_LIBRARY_PATH` into site-packages), and untars `LLM/deployment/onnxruntime/voe/quicktest` into the cpp path.
|
||||
- `xrt-smi` in a container needs `libboost-filesystem1.83.0` + `libboost-program-options1.83.0` (**hyphenated** Ubuntu names) and `apt update` first; boost is **only** needed by `xrt-smi`, never by the OGA/EP path.
|
||||
|
||||
## HF artifact staging
|
||||
|
||||
- Gated (401 on API, 403 on git) for a plain account: `neuphonic/neucodec` (the **only** NeuCodec encoder), `neuphonic/neucodec-onnx-decoder`, `neuphonic/neucodec-onnx-decoder-int8`, `neuphonic/neutts-air`. `amd/neutts-air_rai_1.8.0_npu_4K` is public. Public mirrors of the codec exist (`eugenehp/neucodec` has the full 1.16 GB `pytorch_model.bin`) — treat a mirror of a gated release as a licensing decision for the operator, not a workaround to take silently.
|
||||
- Gated-repo loader trap: `HF_HUB_OFFLINE=1` + `HF_HOME` pointed at a cache laid out as `hub/models--<org>--<repo>/snapshots/<rev>/…` works, but `NeuCodec._from_pretrained` **asserts the repo id** is in `["neuphonic/neucodec", "neuphonic/distill-neucodec"]` — a local directory path is refused outright, so pass the repo id and satisfy it from the cache.
|
||||
- `github.com/neuphonic/neutts` is public and ships `samples/{dave,emily,greta,jo}.{pt,txt,wav}` = **pre-encoded reference codes + transcripts** → a spike needs zero gated bytes.
|
||||
- Auth without leaking the token into `ps`: build a 600-perm credential file with a shell builtin and point git at it —
|
||||
`umask 077; printf 'https://oauth2:%s@huggingface.co\n' "$(cat ~/.cache/huggingface/token)" > ~/.git-credentials-hf`
|
||||
@@ -30,14 +32,30 @@ Tarball has 30 top-level entries: cp312/linux_x86_64 wheels (`onnxruntime_genai_
|
||||
|
||||
## NuTTS-air on XDNA2 (the model-side contract)
|
||||
|
||||
- The package is an OGA model dir: `genai_config.json` (`type: qwen2`, 24 layers, hidden 896, 14 heads / 2 KV, ctx 32768, `provider_options: [{"RyzenAI": {"hybrid_opt_token_backend": "npu"}}]`), plus `dd_metastate_*` AIE artifacts and `model.onnx` + `model.onnx.data`.
|
||||
- **Speech tokens live in the text vocab**: `<|speech_0|>` = id 151671 (10 001 codes), with `<|TEXT_PROMPT_START|>` 151666, `<|TEXT_PROMPT_END|>` 151667, `<|TEXT_REPLACE|>` 151665, `<|SPEECH_REPLACE|>` 151668, `<|SPEECH_GENERATION_START|>` 151669, `<|SPEECH_GENERATION_END|>` 151670. Generation is a plain `og.Generator` loop; the graph's own `eos_token_id` is `[151645, 151643]`, so pass 151670 as an extra eos or stop on it manually.
|
||||
- **Real-time line = 50 speech tokens/s** (`sample_rate=24000`, `hop_length=480`). Judge any NPU/iGPU TTS number against 50.
|
||||
- The package is an OGA model dir: `genai_config.json` (`type: qwen2`, 24 layers, hidden 896, 14 heads / 2 KV, ctx 32768, `vocab_size` **217,652**, `provider_options: [{"RyzenAI": {"hybrid_opt_token_backend": "npu"}}]`), plus `dd_metastate_*` AIE artifacts and `model.onnx` + `model.onnx.data`.
|
||||
- **Speech tokens live in the text vocab, and the codebook is 65,536 — not 10,001.** `tokenizer.json` `added_tokens` holds exactly 65,536 `<|speech_N|>` entries, ids **151,671–217,206** contiguous, over a base BPE vocab of 151,643; specials are 151,644–151,670 (`<|SPEECH_GENERATION_END|>` = 151,670). The vendor template maps a codec index straight to its id: `"".join(f"<|speech_{i}|>" for i in ref_codes)`. An encoded reference really does contain codes in the 60k range, so a 10k assumption silently truncates the space.
|
||||
- The graph's own `eos_token_id` is `[151645, 151643]` (the **chat turn** stop) and `search.eos_token_ids` is unset — pass 151,670 as an extra eos or speech ends after one token.
|
||||
- **Real-time line = 50 speech codes/s** — NeuCodec `sample_rate 24000 / hop_length 480`, measured exactly (49.96–50.00 across three 20–25 s references). Judge any NPU/iGPU TTS number against 50. (Qwen3-TTS, the fleet's deployed arm, is 12.5 codes/s — do not conflate the two bars.)
|
||||
- Reference encode is cheap and cacheable: `librosa.load(sr=16000, mono=True)` → `[1,1,T]` float tensor → `codec.encode_code(audio_or_path=…)` (mirrors `NeuTTS.encode_reference`); CPU encoder RTF 0.28–0.38 on 8 threads. `neucodec` needs `torchtune` (`RotaryPositionalEmbeddings`) — install both `--no-deps --target <dir>` so a second torch never shadows the venv's.
|
||||
- Build the prompt by *importing* the vendor function, never by copying the template:
|
||||
`sys.path.insert(0, <neutts repo>); from neutts.neutts import NeuTTS`, then
|
||||
`NeuTTS._apply_chat_template(SimpleNamespace(tokenizer=hf_tok, input_format="text"), ref_codes, ref_text, text, None)` — works on a duck-typed `self`, so no backbone/codec load. Template: `user\n<|TEXT_REPLACE|>\nassistant\n<|SPEECH_REPLACE|>`; splice `TEXT_PROMPT_START + encode(f"{ref_text} {text}") + TEXT_PROMPT_END` at TEXT_REPLACE, then truncate at SPEECH_REPLACE and append `[SPEECH_GENERATION_START] + encode("".join(f"<|speech_{i}|>"))`.
|
||||
- Voice identity = reference codes + transcript (request-time conditioning). No trained artifact exists to preserve; packs holding only `reference.wav` require the gated encoder once, offline.
|
||||
- Voice identity = reference codes + transcript (request-time conditioning). No trained artifact exists to preserve; packs holding only `reference.wav` require the gated encoder once, offline — **and** a transcript, which a voice pack does not carry (`manifest.json` holds provenance/crop geometry only) and a host without ASR cannot produce.
|
||||
- Full-Fusion input ceiling: 4096 tokens **total**, input + output.
|
||||
|
||||
## Status caveat
|
||||
## Status caveat (2026-10-19)
|
||||
|
||||
Measured on 2026-10-19 up to and including: staged artifacts, image build, device-access matrix, prompt contract. The token-rate measurement (does the vendor userspace drive the in-tree 0.7.0 driver, and does it clear 50 tok/s) is **not** captured here — re-measure, never trust a remembered rate.
|
||||
Staged artifacts, image build, device-access matrix, prompt contract, reference pre-encode:
|
||||
**done and reproducible.** Token generation on the NPU: **NO-GO, and the loader is the
|
||||
wall.** The device is healthy (`xrt-smi validate` PASSED incl. `throughput`,
|
||||
`RyzenAI-npu5 / aie2p / 6×8`, `pyxrt.device(0)` constructs), and `strace` shows the
|
||||
failing load reading `genai_config.json`, opening `model.onnx`, loading
|
||||
`libxrt_driver_xdna.so.2` and opening `/dev/accel/accel0` with successful GEM ioctls —
|
||||
then the RyzenAI EP **declines the graph before touching any `dd_metastate_*`**, silently:
|
||||
`GENAI_ORT_VERBOSE_LOGGING`, `GENAI_LOG_ORT_LIB`, `set_log_options`,
|
||||
`set_log_callback(callable)`, `RYZENAI_EP_TRACING`, `RYZENAI_LLM_PROFILE_INIT` all emit
|
||||
zero lines. Hardlinked mirror dirs of the model tree were ruled out as shadowing. One run
|
||||
loaded and produced finite logits; byte-identical runs never reproduced it. **No token rate
|
||||
was ever measured, so quote none**, and remember `quicktest` success ≠ NPU residency (that
|
||||
build falls back to CPU silently — prove residency via
|
||||
`/sys/class/accel/accel0/device/mem_usage`).
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
---
|
||||
name: ryzenai-oga-model-dir-and-eos-override
|
||||
description: "Debug OGA on XDNA2 when the RyzenAI EP never claims the If-rooted graph (shadow model dirs, bind-mounted configs, missing device), or a TTS/LLM graph stops on the wrong eos token."
|
||||
---
|
||||
|
||||
# OGA on XDNA2: graph not claimed, or generation stops at once
|
||||
|
||||
Two independent failure families in the AMD `onnxruntime-genai` (OGA) + RyzenAI EP stack on
|
||||
Linux XDNA2 (Strix Halo / Ryzen AI Max). Diagnose them in this order — they are separate
|
||||
gates and each has its own instrument.
|
||||
|
||||
## Gate 0 — is the NPU path alive at all?
|
||||
|
||||
Run the vendor's own instrument before touching anything else:
|
||||
|
||||
```bash
|
||||
cd /opt/rai/quicktest && RYZEN_AI_INSTALLATION_PATH=/opt/rai python quicktest.py
|
||||
# want: "Setting environment for STX/KRK" ... "Session successfully initialized" ... "Test Finished"
|
||||
```
|
||||
|
||||
It finds the device by shelling out to `lspci -nn` and grepping `1022:17f0`, so it also
|
||||
proves `lspci` is present in the container. Pass ⇒ the in-tree `amdxdna` driver and the
|
||||
extracted userspace pair correctly; stop blaming the host.
|
||||
|
||||
Separate **XRT enumeration** from **OGA's device probe** — they are not the same check:
|
||||
|
||||
```python
|
||||
import sys; sys.path.insert(0, "/opt/xilinx/xrt/python")
|
||||
import pyxrt; dev = pyxrt.device(0) # constructs => XRT sees the device
|
||||
```
|
||||
|
||||
`pyxrt.device(0)` succeeding while OGA says `No RyzenAI devices detected` means the
|
||||
refusal is OGA's own predicate, inside `libonnxruntime-genai.so` (the string lives there,
|
||||
next to `RyzenAI`, `max_length_for_kv_cache`, `library_path`). EP-side env levers worth
|
||||
pulling: `RYZENAI_EP_TRACING`, `RYZENAI_EP_PERFORMANCE_COUNTERS`, `RYZENAI_LLM_ASYNC_INIT`,
|
||||
`RYZENAI_LLM_PROFILE_INIT`.
|
||||
|
||||
## Gate 1 — `invalid model … Node (If) cond_full`
|
||||
|
||||
This is **ONNX Runtime verifying the raw graph**, which happens when no provider claimed
|
||||
it: the EP registered nothing and the CPU provider was left holding an `If` whose branches
|
||||
mismatch. It is never a corrupted model.
|
||||
|
||||
```bash
|
||||
LD_DEBUG=files LD_DEBUG_OUTPUT=/tmp/ld python loadprobe.py
|
||||
grep -a "calling init:.*ryzenai\|undefined symbol" /tmp/ld.*
|
||||
# observed: libonnxruntime_providers_ryzenai.so loaded, then
|
||||
# symbol lookup error: undefined symbol: GetProvider (fatal)
|
||||
```
|
||||
|
||||
`GetProvider` is not a dynamic symbol in the ORT cores, the shared-provider stub, or the
|
||||
EP (`nm -D` shows neither defines nor wants it), so the failure is registration, not an
|
||||
ELF mismatch. Confirm the cores are identical before chasing versions: both shipped
|
||||
`libonnxruntime.so.1.27.0` copies hash equal.
|
||||
|
||||
Separate the two gates with an explicit global-scope load before importing OGA:
|
||||
|
||||
```python
|
||||
import ctypes
|
||||
ctypes.CDLL("/opt/rai/deployment/lib/libonnxruntime_providers_ryzenai.so",
|
||||
mode=ctypes.RTLD_GLOBAL)
|
||||
import onnxruntime_genai as og
|
||||
og.Model(MODEL) # registration fixed => error MOVES to "No RyzenAI devices detected"
|
||||
```
|
||||
|
||||
An error that **moves** proves you cleared one gate and exposes the next. Silence in the
|
||||
first form means the provider list was empty.
|
||||
|
||||
### Dead ends (measured, do not retry)
|
||||
|
||||
- **Shadow model dirs never work.** Hardlink mirror, symlink mirror, and even a
|
||||
read-only single-file bind mount of a patched `genai_config.json` over the export's own
|
||||
`genai_config.json` all produce the same `If` rejection. The graph is only claimable
|
||||
when loaded from the export's own directory, and there is no config-path override in
|
||||
`0.14.0-dev`: the only env names in the shipped binaries are `GENAI_LOG_ORT_LIB`,
|
||||
`GENAI_ORT_VERBOSE_LOGGING`, `OG_ORT_LIB`. Carrying a config change therefore means
|
||||
either editing the download in place or a **single-file bind mount over the real dir**
|
||||
(`-v file.json:/spike/<model>/genai_config.json:ro`) — and if the EP still refuses, the
|
||||
gate is not the config.
|
||||
- **Bisect the image before believing "environment drift".** Run the same load probe
|
||||
against every local tag; three tags failing identically rules the image out.
|
||||
- **Rule out host state with instruments, not memory:** `fuser -v /dev/accel/accel0`,
|
||||
`dmesg | grep -i xdna`, `stat -c "%i %y"` on the model files, `df -h`, `free -g`.
|
||||
- `sysfs` class is `accel`, not `misc`: glob `/sys/class/accel*/*`. An empty
|
||||
`/sys/class/misc/accel*` is your glob, not a missing device.
|
||||
- `GENAI_ORT_VERBOSE_LOGGING=1` and `og.set_log_options(...)` produce nothing useful in
|
||||
this build; `set_log_options` is a pybind builtin (no `inspect.signature`) and rejects
|
||||
`severity` as an int. Don't burn turns trying to make it talk — use `LD_DEBUG`.
|
||||
- `strace` is absent from the RAI image and `apt` may have no mirror: `LD_DEBUG=files` is
|
||||
the loader tracer that needs nothing installed.
|
||||
|
||||
## Gate 2 — generation stops after one token
|
||||
|
||||
OGA reads its stop list from the **model** section: `model.eos_token_id`. Chat exports
|
||||
carry the chat-turn-close id (e.g. 151645) there, so the model emits it first and the
|
||||
speech/tool budget is never spent — while `search` has **no** eos key: writing
|
||||
`search.eos_token_id` is a parse refusal (`Unknown value "eos_token_id"`), not a no-op.
|
||||
|
||||
- Fix: rewrite `model.eos_token_id` to only the task's own end token (TTS: the
|
||||
speech-generation-end id), or pass it through `set_search_options` if the build accepts
|
||||
that key name.
|
||||
- A sub-millisecond `generate_next_token()` **with finite, high-confidence logits** means
|
||||
the graph ran and chose a stop token — it is not evidence of a CPU fallback.
|
||||
- Verify a token-range census against the export's `added_tokens.json`, not against a
|
||||
guessed vocabulary size: NeuCodec codes reach 65,536 entries (ids 151671…217206), and a
|
||||
census bounded at 10k silently miscounts every code as a text token.
|
||||
- A hardlinked run dir is a loaded gun: `open(dst, "w")` on a hardlink truncates the
|
||||
**original** through the shared inode. `os.unlink` the link before writing a patched
|
||||
file. Also skip `.git/` and mirror directories when walking an HF checkout.
|
||||
|
||||
## Rate measurement
|
||||
|
||||
Once a load succeeds, measure at ≥300 steps with a host-side `%CPU` sampler beside the
|
||||
container (`ps -C python -o pid=,pcpu=,rss=` every 2 s): a device-bound decode waits on
|
||||
`accel0` and stays under one core, a silent CPU fallback pegs at least one. Never report
|
||||
request #1 after a boot — graph capture makes it a warmup artifact.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user