dank/command-center
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(log): compact changelog to entry law (406 -> 284 lines)

Browse Source 
Changelog entries were essay-length incident writeups duplicating the
concept pages they created. New entry law recorded at top of log.md:
one dated entry = what changed + durable lesson + concept-page link;
full narratives recoverable via git log --follow. Every date, headline,
incident, and durable lesson preserved; narrative prose that already
lives on concept pages (caddy handle-ordering, gluetun/limiter field
notes, quadlet gotchas, faillock law, credential-store trap) replaced
by pointers. Historical pi-era entries tagged '(pi era)'.

Also: self-hosted-web-agents.md '44.5% peer-reviewed CUA' figure could
not be re-attributed to any published benchmark via web search today
— dropped with an explicit note; the valid core claim (87.4% = managed
Cloud, not the self-hostable library) kept.

Refs #29
This commit is contained in:
Frank
2026-10-19 04:23:58 -07:00
parent a9b3c495da
commit 986081bf1d
2 changed files with 195 additions and 310 deletions
Download Patch File Download Diff File Expand all files Collapse all files
omp-docs/agentic-web-research/self-hosted-web-agents.md
+9 -2
View File
@@ -29,10 +29,17 @@ Key ideas:
- Ten watchdogs handle CAPTCHAs, crashes, popups, downloads, and security.
- Real browser profiles carry saved logins.
**On the "87.4 percent" number:** that result refers to Browser Use's managed Cloud agent (the Hybrid type that blends structured browser tools with screenshots). Peer-reviewed results for vision-only (CUA) models are much lower, about 44.5 percent on the same task type. Treat any single accuracy number as one benchmark, not a universal score.
**On the "87.4 percent" number:** that result refers to Browser Use's managed
Cloud agent (the Hybrid type that blends structured browser tools with
screenshots), not the open-source library you self-host. Academic
computer-use results on comparable web tasks run far lower, and an earlier
"44.5 percent peer-reviewed CUA" figure carried in this page could not be
re-attributed to a specific published benchmark (checked 2026-10-19) and was
dropped. Treat any single accuracy number as one benchmark, not a universal
score.
```python
# Local model via llama.cpp (OpenAI-compatible endpoint)
# Local model via any OpenAI-compatible endpoint
from browser_use import Agent
from browser_use.llm.openai import ChatOpenAI
omp-docs/log.md
+186 -308
View File
@@ -6,401 +6,279 @@ tags:
- okf
- maintenance
- changelog
timestamp: 2026-08-28
timestamp: 2026-10-19
---
# Bundle Change Log
Entry law (since 2026-10-19): one dated entry = what changed + the durable
lesson, a few lines. Full incident narratives are recoverable from git
(`git log --follow omp-docs/log.md`); knowledge that outlives an entry lives
on its concept page, linked here. Never re-expand an entry in place.
## 2026-10-19 — dankvoice: humanized narration and thinking aloud (Refs #27)
dankvoice answered in verbatim markdown-cleaned text while the built-in
vocalizer speaks humanized prose — the bundle's `Qyr` enhancer (`speech.enhanced`,
system prompt `T6o`) now lives in the extension: every spoken unit is rewritten
by the @tiny role (resolved from `modelRoles.tiny` in config.yml + the provider
baseUrl in models.yml → frank-zealot :7771) under the same budgets (1536 tokens,
6 s, 4000-char window), with mechanical cleanup as the fallback and an empty
rewrite skipping the unit. Auto-speak also narrates the last assistant message's
thinking blocks before the answer (`narration: true` blocks count as answer
text); both are menu-toggleable and persisted (`thinking` in the settings file),
env levers `DANKVOICE_REWRITE_URL/MODEL/REWRITE=0/THINKING/OMPCONFIG`. The
fixture grows a chat route with echo/prose modes; 7 new scenarios, 22 green.
Live smoke: a markdown answer became 3 sequential prose segments through real
zealot + frank-tts, thinking played before its answer.
Spoken units are rewritten by the `@tiny` role (frank-zealot) under the
bundle's speech budgets before synthesis, with mechanical cleanup as fallback;
auto-speak narrates the last message's thinking blocks first; all toggles
persist. 7 new scenarios, 22 green; live smoke through real zealot + frank-tts.
## 2026-10-19 — dankvoice: segmented sequential speech, copied from the bundle's vocalizer
## 2026-10-19 — dankvoice: segmented sequential speech
Silent-answer incident follow-up: `agent_end` awaiting one whole-answer
synthesis hit the harness's 30 s extension-handler timeout
(`handler timed out after 30000ms`) on long answers, and the single frank-tts
request for the entire answer was the stall. Instead of inventing a heuristic,
ported the utterance-splitting state machine out of the built-in speak-aloud
vocalizer (minified class `fHe` in `dist/cli.js`): strong boundaries are
`[.!?…]+` runs followed by whitespace — `!!!!!`/`...` stay one cut — with
abbreviation and inline-code guards; `3.5`/`1,5` survive because a boundary
needs trailing whitespace; fenced code and tables swallow whole; headings and
bullets vanish, ordered markers speak their number ("1, "); links collapse to
labels, URLs to hosts, paths to basenames; run-ons break at commas or word
wraps before 280 chars. `speak()` now synthesizes segment N+1 only after
segment N's player exited — strictly sequential, audio never stacks — and
`agent_end`/`/say <text>` fire-and-forget a chain retired through a run token
by `/say stop`, disarming, or a newer utterance. `DANKVOICE_MAX` became a
per-answer budget of whole leading segments; the hand-rolled `stripForSpeech`
truncator is gone. No LLM "humanization" rewrite happens (the bundle's
`speech.enhanced` rewrites with the tiny/smol model; dankvoice speaks
mechanically cleaned text through the cloned voices). Tests: two new driver
scenarios plus two rewrites, 17/17 in `tests/dankvoice.bats`.
`agent_end` awaiting one whole-answer synthesis hit the harness's 30 s
extension-handler timeout — lesson: extensions must never await work that can
exceed 30 s. Ported the built-in vocalizer's utterance-splitting state machine
(boundary = `[.!?…]+` + whitespace, abbreviation/code guards, per-answer
segment budget); segments synthesize strictly sequentially. `DANKVOICE_MAX` is
now a segment budget. 17/17 in `tests/dankvoice.bats`.
## 2026-10-18 — dankvoice: `/say` options menu with output-device routing (Refs #23)
Added `/say menu` (bare `/say` now opens it; `help` keeps the text listing), the
pi-peon-style settings surface: auto-speak toggle, voice picked from the frank-tts
roster, **output device** (PipeWire sinks via `wpctl` → `pw-play --target` /
`paplay --device=`), player binary, spoken-char cap, and a "speak a test phrase"
utterance that proves the audio path audibly. Choices persist to
`~/.omp/cache/dankvoice.json` (`DANKVOICE_CONFIG`); env now wins over that file,
which wins over defaults — answering the silent-session incident where
PipeWire's remembered Bluetooth route played synthesized answers into nowhere.
`detectPlayer` becomes `availablePlayers` + `playerFor`. Tests gain a fixture
`wpctl`, a scripted `select` in the driver, and three menu scenarios: 15 green
in `tests/dankvoice.bats`, 51 green suite-wide.
`/say menu` (pi-peon-style): auto-speak, voice, PipeWire output device
(`wpctl` → `pw-play --target`), player, char cap, test phrase. Precedence
env > `~/.omp/cache/dankvoice.json` > defaults — born from the incident where
PipeWire's remembered Bluetooth route played answers into nowhere.
## 2026-10-18 — dankvoice: the cloned voices finally speak in the TUI (Refs #21)
Nothing in the harness could pronounce them: the TUI vocalizer and `omp say` gate `--voice` against the bundled Kokoro enum (`omp say --voice narrator1 "test"` → `error: Expected --voice to be one of: af_heart, …`), so frank-tts was reachable only as a file. Added `agent/extensions/dankvoice.ts` — `/say <text>`, `/say on|off` (narrate the final answer at `agent_end`), `/say voice|voices|stop|status` — synthesizing over the OpenAI speech route and playing through `pw-play`/`paplay`/`ffplay`, with the endpoint read from the `api: openai-speech` row of `agent/models.yml` (no host, no secret in the file). Verified live in a real TUI session on the workstation: roster `GET /v1/audio/voices` → `narrator1, narrator2, narrator3`; `/say` produced 99,884 B / 2.08 s and 69,164 B / 1.44 s WAVs handed to the player, temp dirs cleaned afterwards; `/say on` armed the footer (`🔊 narrator1`) and each following finished turn spoke its answer (1.20 s WAV), an unknown voice name answered with the roster instead of speaking, and an `agent_end` carrying `willContinue` (the harness scheduled an auto-retry) is deliberately never narrated. Extension discovery itself proved with a probe file in `~/.omp/agent/extensions/` loaded by a fresh `omp -p` run. Reference page [dankvoice](core/dankvoice.md), registered in the core index; black-box seam `tests/dankvoice.bats` (12 tests).
The harness gates `--voice` against the bundled Kokoro enum, so frank-tts's
cloned voices were unreachable in-TUI. Added `agent/extensions/dankvoice.ts`
(`/say …`) synthesizing over the OpenAI speech route, endpoint read from the
`api: openai-speech` row of `agent/models.yml` (no host or secret in code).
## 2026-10-18 — Noctalia bar layout clobbered by trunk-sync stash, recovered from dangling objects
## 2026-10-18 — Noctalia bar layout clobbered by trunk-sync stash, recovered
Frank's months of GUI-built bar customizations (cat/clock/clock-utc/weather start, taskbar center, capsule groups g1–g4, control-cheat-sheet, thickness/font-scale/opacity styling) vanished from the live bar at 16:46. Root cause: the layout existed **only in the working tree** — the repo copy of `settings.toml` carried a stale plain layout because GUI edits hadn't been committed inside the hourly heartbeat window — and a trunk-sync session clearing the dirty tree for `origin/main` stashed it; the stash entry was subsequently destroyed (three `WIP on main` stashes within 60 s, all later dropped). Recovery: `git fsck --lost-found` surfaced the destroyed stash as dangling commit `7fe3ed8` (16:46:09); candidate validated in a sandboxed `XDG_*_HOME` before restore, clobbered file preserved to `.AGENTTRASH/`, layout restored through the symlink target and proven by screenshot, committed as `c87b378` — the durable copy the layout never had. Lessons codified into managed skills: trunk-sync must preservation-**commit** every dirty deployment path (never stash, never `stash clear`), and GUI layout edits get committed the same session they're made.
GUI-built bar customizations existed only in the working tree; a trunk-sync
stash-then-drop destroyed them. Recovery: `git fsck --lost-found` surfaced the
dropped stash as a dangling commit; validated in a sandboxed `XDG_*_HOME`
before restore. Lesson: GUI-written config is only durable once committed —
the motivation for the live-sync heartbeat below.
## 2026-10-18 — Fleet law: issue/PR attribution footers
Every agent-opened issue and PR body now ends with a `cwd:` + `session:` footer (session id from `/status`), in AGENTS.md of both `dank/command-center` and `dank/serverAppConfigs` (mirrored PRs). Born from the live-sync review: once the machine starts filing and committing on its own, its artifacts must trace back to the box and session that produced them.
Every agent-opened issue/PR body ends with `cwd:` + `session:` footer, in both
`dank/command-center` and `dank/serverAppConfigs` AGENTS.md: machine-made
artifacts must trace to the box and session that produced them.
## 2026-10-18 — live-sync heartbeat for machine-written paths (Refs #11)
Root-caused the month of silent divergence in the live checkout (~61 auto-learned skills on one disk only, a stranded `unexpectedStopDetection` fix, merge-refusals from dirty trees): the checkout IS the deployment, symlinks make machine writes dirty tracked files by design, and no commit law covered that churn. Added `scripts/live-sync.sh` + systemd user timer — allowlist tiers (memories, managed-skills, live config, dotfiles) auto-commit with hooks armed, merge `origin/main`, push, and reap stale `wt/*` worktrees; everything outside the allowlist is never staged, conflicts abort-and-alert. Black-box suite `tests/live-sync.bats` (9 tests). Tier law in AGENTS.md § Live-sync tiers; reference page `tools/live-sync.md` registered in the tools index.
A month of silent divergence in the live checkout proved the checkout-is-
deployment model needs a committer: `scripts/live-sync.sh` + systemd user
timer auto-commits allowlisted tiers (memories, managed-skills, live config,
dotfiles) with hooks armed, merges `origin/main`, pushes, reaps stale `wt/*`.
Non-allowlist dirt never staged; conflicts abort-and-alert. See
[tools/live-sync.md](tools/live-sync.md); tier law in AGENTS.md.
## 2026-10-18 — Fleet law: filed PRs open in Frank's browser immediately
Both repos' git rules now require it: the moment a PR is opened, `xdg-open <PR-URL>` drops a Firefox tab on the workstation (AGENTS.md § Trunk-based SDLC here; § Git Workflow step 6 in `dank/serverAppConfigs`, riding their PR #6 branch).
`xdg-open <PR-URL>` the moment a PR opens — codified in both repos' git rules.
## 2026-10-18 — Trunk law: extension route closed unmerged; direct repo alignment instead
## 2026-10-18 — Trunk law: extension route closed unmerged; direct repo alignment
Built and live-verified a cwd-scoped `before_agent_start` extension (quote probes: fires under `~/Projects`, silent elsewhere; kept as `.AGENTTRASH/trunk-law.js`). **Operator closed PR #9 unmerged** in favor of editing each repo's own AGENTS.md. Durable finding from the exploration: every context-file provider (AGENTS.md walk-up, `.omp/` dirs, CLAUDE.md) resolves at or below the repo root — a `~/Projects/AGENTS.md` for ancestor scope can never load; user-level `APPEND_SYSTEM.md` broadcasts to *all* sessions. Second finding: **serverAppConfigs never had git-flow** — its AGENTS.md mandated no branch regime at all (direct commit+push); a trunk + PR law is being added there by PR in that repo, not corrected.
Operator closed PR #9 (cwd-scoped `before_agent_start` extension) in favor of
editing each repo's AGENTS.md. Durable findings: every context-file provider
(AGENTS.md walk-up, `.omp/` dirs, CLAUDE.md) resolves at or below the repo
root — ancestor-scope files can never load; user-level `APPEND_SYSTEM.md`
broadcasts to all sessions; serverAppConfigs never had git-flow.
## 2026-10-18 — SDLC switched to trunk-based development
`develop` retired by operator decision (single-dev repo; the branch bought nothing and cost a back-merge every release). New law in AGENTS.md § *Trunk-based SDLC*: branch off the freshest `origin/main` before every work item, PRs target `main` directly, `--no-ff` merges, tags `vX.Y.Z` straight on main. No back-merges, no release branches. The last back-merge (`527596a`) lands with this switch; the managed skill `gitflow-back-merge-plumbing` is now historical.
`develop` retired (single-dev repo). Law in AGENTS.md § Trunk-based SDLC:
branch off freshest `origin/main`, PRs target `main`, `--no-ff` merges, tags
`vX.Y.Z` on main. Managed skill `gitflow-back-merge-plumbing` is historical.
## 2026-10-18 — Sibling-vault overlap census; system docs realigned to live ground truth
Census of `omp-docs/` against `~/Projects/serverAppConfig/serverAppConfigs-docs/` (subagent + live-ssh verification) found four contradicting pairs; all OURS-side stale pages corrected, THEIRS side flagged only (their repo, their PR flow). Ground truth from read-only probes on both boxes 2026-10-18.
Census of `omp-docs/` vs `serverAppConfigs-docs` (subagents + live-ssh probes)
found four contradicting pairs; ours fixed, theirs flagged for their PR flow.
Recorded then: RTX 3090 moved Dankserver→Dankclaw 2026-09; dankserver = 2×
Radeon AI PRO R9700 (the "board swap" story was false); served models per
live units. Current state always: [/system/index.md](/system/index.md)
(27B-era zealot figures here are historical; live census 2026-10-19 below).
* **Hardware drift corrected** (`system/dankserver.md`, `system/dankclaw.md` rewritten, `AGENTS.md` machine table): RTX 3090 moved Dankserver→Dankclaw 2026-09; dankserver now 2× Radeon AI PRO R9700 (7700X stays — the "board swap" story is false), 246 GiB RAM, no swap; dankclaw 121 GiB RAM + 64 GiB swap. Served models per live units: frank-archon (:7770 dankserver, radiance TP=2 iq4r), frank-dragoon (:7770 dankclaw, halogen NATIVE v2, iGPU), frank-zealot (:7771 dankclaw, llama.cpp PrismML, Ternary-Bonsai-2-27B PQ2_0, 3090). Both kernels 7.0.0-38-generic; dankclaw :7770 needs no auth.
* **Gitea pages corrected** (`tools/gitea/` index + 3 subpages): `gitea-caddy` quadlet retired — fleet Caddy fronts the domain; data tree is `/home/dank/Containers/gitea/{data,postgres,env}` (2026-09-01 standard), `/opt/gitea/` is a do-not-use leftover; LAN IP, live domain, no WAN address.
* **Pi-era pages demarcated as archive** (`pi/index.md`, `core/index.md` banners; `core/index.md` said settings.md was "Current"): `frank-default-small/large` endpoints, role tables, and 3090/4070 hardware rows now read as pi-era records. `tools/llama-cpp/index.md` fleet note modernized to the one remaining llama.cpp endpoint (zealot :7771).
* **Their-side staleness flagged, NOT edited** (for operator clarification): `Agent/OMP Harness.md` still says thresholdPercent 70 — live config is **95** (`agent/config.yml:84`); `index.md:18` says Dankserver Ubuntu 24.04 vs their own specs page 26.04; `Dankclaw/Stack.md` "current mapping (2026-09-03)" predates the ACTIVE dragoon/zealot runbooks it links beside; `index.md:548` links the superseded pi-era KB at `~/.pi/agent/docs/`.
* Consistent duplicates (no action): niri/noctalia desktop pages, Caddy reference. Divergence clarified: their "composer" = OpenWebUI chat box; ours = omp composer shape — unrelated subjects.
## 2026-10-18 — Status line rebuilt on custom segments; full-source audit
## 2026-10-18 — Status line docs truth audit (full-source re-verification)
New page [core/statusline.md](core/statusline.md) with an audit log — four
first-pass claims were false and corrected against the minified bundle + a
live screenshot matrix (context meter is `box`/`band` only; `composer.shape`
routes segment arrays; `mode` states = Plan/Prewalk/Goal/Vibe/Loop). Bar
rebuilt on `band` + custom segments; frank-archon display name slimmed in
models.yml.
Exhaustive claim-by-claim re-audit of [OMP Status Line Customization](/core/statusline.md) against the minified `dist/cli.js` bundle plus a live screenshot matrix (band × {off, percentage, embedded}, claude × off). Page carries an **Audit log** section; four first-pass claims were false and are corrected (see there): claude's rule is *not* the context meter (meter is `box`/`band` only; the "red rule" was the session-accent filler), the "Tern-only" composer-bar claim was bogus (the readout is `composer.tokenRate`, default off), `segmentOptions` are **not** gated to `preset: custom` (only the segment arrays are), and the `mode` segment's states are Plan/Prewalk/Goal/Vibe/Loop (no yolo). Page expanded with: the 7-preset defaults table, per-shape `statusAttachment`/`bottomBar` model (8 built-ins + plugin shape registry), `contextLine` per-value behavior, the 27-segment catalog with triggers/options, `segmentOptions` schema, and the full `statusLine.*`/`composer.*` knob registry. The bundle's own "(box composer only)" knob description for `contextLine` is upstream-wrong — band honors it.
## 2026-10-09 — Blankpad DE redesign: desk profiles, keybinds, Handy wiring
ServerAppConfigs-docs overlap census: flagged separately for the operator (morning report); no doc merged/changed on either side.
Per-desk niri profiles keyed by EDID long name (connector names reshuffle
between docks; identity travels) + `scripts/desk.sh`; niri 26.04 has **no
hotplug event** (`OutputConfigChanged` is only an ACK per niri-ipc) — the
autoswitch daemon diffs output sets on `WorkspacesChanged`. Keybind cheat
sheet now generated (`scripts/gen-keybinds.sh`); noctalia docs corrected to
the v5 `noctalia msg` API. Handy: niri owns autostart; open breaker —
frank-secondary answers in `reasoning_content` with empty `content`, and
Handy cannot send `chat_template_kwargs.enable_thinking=false`. 27/27 green.
## 2026-10-18 — Status line rebuilt on custom segments; frank-archon display name slimmed
Interactive tune-up of the omp status bar (grilling session, live loop: config edit → watcher repaint → niri screenshot verify). New knowledge page: [OMP Status Line Customization](/core/statusline.md).
* **Mechanics discovered:** `leftSegments`/`rightSegments` are inert unless `preset: custom` (fleet was on `nerd` with dead arrays; `segmentOptions` *do* merge under named presets); `composer.shape` routes them — `band` merges both arrays into one row above the editor (title rides the working row), `claude` splits them into top chip + bottom strip. The context meter exists **only in `box`/`band`** — `contextLine`: `off` plain rule, `embedded` hoists the context segment into the labeled inline meter (`60% ——— × 200K`), `percentage`/`annotated` paint an unlabeled proportional rule meter (all four live-verified); every other shape's rule is a constant session-accent filler. Working row (session title + tok/s readout gated by `composer.tokenRate`, default off) is composer chrome — model/thinking can never join it. Eight composer shapes ship in the bundle; plugins can register more.
* **Final bar (`band`, after a `claude` sojourn):** left = `session_name · vim · mode · hostname · path · git(no branch text) · pr`; right cluster in array order = `model · cost · time_spent` (`context_pct` hoists out of the run to paint the labeled inline meter `60% ——— × 200K` as a flexible spacer between the groups). Cut: tok/s duplicate, clock, cache trio, `context_total`, `subagents`, `session` hash, and `token_in`/`token_out` (cumulative totals live in `/session`). `path`: logical `project/worktree` label (`stripWorkPrefix: true`, kept after a live `false` trial of the full cwd); branch text dropped (worktree branches are autogenerated), dirty badges kept.
* **models.yml:** frank-archon display `name` slimmed to `"frank-archon"` (endpoint suffix dropped; `id` untouched).
* **Out of scope noted:** worktree name generation (`wt-<date>-<hash>`) is omp-internal; possible upstream feature request.
## 2026-10-09 — Blankpad DE redesign: desk profiles, keybind rework, Handy fleet wiring
Branch `blankpad-de-redesign` (worktree `wt/de-redesign-20261009`). Full sweep + redesign of the niri/noctalia/Handy stack on blankpad.
* **Desk profiles.** `cfg/display.kdl` retired to a symlink; per-desk profiles `desk-1-home.kdl` / `desk-2-portable.kdl` declare monitors by **EDID long name** (`// requires:` lines) — connector names reshuffle between docks, identity travels. `scripts/desk.sh` = list/current/switch/cycle/match/capture; apply = relink → `niri validate` (rollback on failure) → `load-config-file` → noctalia notification. `scripts/desk-autoswitch.sh` + `blankpad-desk-autoswitch.service` watch `niri msg --json event-stream`: niri 26.04 has **no hotplug event** (`OutputConfigChanged` is only an ACK that an IPC/config output change was Applied, per niri-ipc); the daemon diffs the sorted output set carried by `WorkspacesChanged` and hot-applies the matching profile (debounced 1.5 s). Proven live with an eDP-1 off/on cycle: `[DP-2 eDP-1] → [DP-2] → [DP-2 eDP-1]`, apply + reload each time. PM1601 EDID still uncaptured (panel disconnected) — connector-keyed placeholder, `desk.sh capture` pending.
* **Keybinds.** `swap-window-left/right` on Mod+CTRL+H/L; move-column demoted to Mod+CTRL+Alt+H/L; 7 column-width presets 12.5–87.5 %; Print trio (full / gui / screen) via noctalia screenshot IPC; launcher Mod+D; Alt+Tab → noctalia window-switcher; Mod+V → clipboard panel (id verified live); Mod+Shift+D → desk cycle; Mod+Shift+Space → `handy --toggle-post-process`; lid-close → `noctalia msg session lock-and-suspend` (pairs with logind `HandleLidSwitch=ignore`). Cheat sheet now GENERATED: `scripts/gen-keybinds.sh` → `omp-docs/tools/niri/keybindings.md` (119 bindings).
* **Noctalia.** `config.toml` + `settings.toml` (lives in `~/.local/state/`, not `~/.config/`) tracked; `taskbar` widget added to bar center, applied live via `noctalia msg config-reload`. Docs corrected to the v5 `noctalia msg` API.
* **Handy.** Self-autostart killed (`~/.config/autostart/Handy.desktop` → `.AGENTTRASH`; `autostart_enabled=false`); niri owns autostart (`spawn-at-startup "handy" "--start-hidden"`). Fleet post-processing wired into the live store: provider `custom` → `http://192.168.0.143:7771/v1`, model `frank-secondary`, prompt `default_improve_transcriptions`. **Open breaker:** frank-secondary answers in `reasoning_content` and leaves `content` empty; per-request cure `chat_template_kwargs.enable_thinking=false` verified, but Handy cannot send it and the server is shared with fleet compaction — dedicated no-think port proposed. 0.9.6→0.9.8 upgrade is operator-owned (`yay -Syu handy-bin`).
* **Scripts + tests.** Two subagents died mid-run (one mid-test-write; one crashed **looping `sudo yay` without a TTY** — see next entry). Their scripts/tests were rebuilt from live-format ground truth: `tests/desk.bats` (16), `tests/gen-keybinds.bats` (6), `tests/niri-config.bats` (5, incl. real `niri validate` on the assembled base+zz tree with each desk profile active). Suite: 27/27 green.
## 2026-10-09 — Incident: agent sudo-loop tripped pam_faillock and locked out the operator's sudo
While attempting the Handy 0.9.8 install, a subagent ran `sudo yay -Syu handy-bin` in a retry loop for ~15 minutes. Harness sudo has no TTY, so every attempt logged 3 `pam_unix` auth failures; `pam_faillock` (deny=3) locked user `blank` out — the operator's own correct-password `sudo`/`pacman -Syu` began failing with "temporarily locked out", which looked exactly like a changed password.
* **No password change:** `/etc/shadow` mtime 2026-09-10 23:31, untouched; no `passwd`/`chpasswd`/`usermod` in the journal.
* **Recovery:** loop process died with the agent; `faillock --user blank --reset` (unprivileged for own user, dir `/run/faillock/blank` is 0700) cleared the counter; verified empty.
* **New law:** agents NEVER run `sudo`/`yay`/`pacman`/`pkexec` on Frank's workstations — root-requiring steps are handed to the operator as exact copy-paste commands. Codified as managed skill `faillock-sudo-lockout-recovery`.
## 2026-10-09 — Incident: agent sudo-loop tripped pam_faillock, locked out the operator
A subagent retried `sudo yay` (no TTY) for ~15 min; `pam_faillock` deny=3
locked user `blank` — correct-password sudo failed, mimicking a password
change. Recovery: `faillock --user blank --reset`. Law: agents never run
`sudo`/`yay`/`pacman`/`pkexec`; root steps go to the operator as exact
commands. Skill `faillock-sudo-lockout-recovery`.
## 2026-10-09 — Full secret audit of the command-center repo: clean
Five-layer audit of `dank/command-center` at `79e780f`, driven by the four live values in `~/.config/secrets/` (gitea, hf_token, hf_token_write, kagi_api_key) plus 21 provider shapes (sk-, hf_, gh[pousr]_, glpat-, AKIA/ASIA, xox, npm_, pypi-, AIza, JWT, PEM/OpenSSH keys, `scheme://user:pass@host` basic-auth URLs, `password|secret|api_key = <literal>` assignments). Findings printed redacted (prefix + length + sha256 prefix).
* **Live-value scan — clean.** 77 revs (5 branches, 2 tags), 765 reachable blobs, 3.4 MiB of blob text: no known live value appears in any tracked blob at any ref.
* **Whole object store, reachable *and* dangling — clean.** `git cat-file --batch-all-objects` finds 767 blobs; 2 are unreachable, never committed on any ref (`git log --all --find-object` empty). One is a 123 KB superseded memory page; the other is a 35-byte test fixture holding a synthetic `sk-` key whose body is the alphabet followed by `1234` — pattern-minted, not issued. No PEM key, no JWT, no basic-auth URL anywhere. (Quoting the literal here tripped the secret gate, which is the gate working as designed.)
* **Shape hits, all triaged benign**: `api_key="sk-no-key-required"` (llama.cpp doc placeholder), `Set password:` / `authenticate(user, password)` prose in the Authentik memory skills, and a 40-hex string in the home-config census that is a git **packfile filename** (`pack-ff7cc541….pack`), not a token.
* **Entropy layer — clean.** Zero tokens ≥ 32 chars with Shannon entropy ≥ 4.3 in HEAD blobs.
* **Payloads, not just pointers**: all 23 git-lfs wallpaper objects (81 MiB) scanned as bytes — no secret shape in any payload.
* **Quarantine list holds**: no tracked file matches `auth.json`/`trust.json`/`.env`/`*.pem`/`*.key`/`identity*`; the 42 ignored paths on disk carry no secret shape; no credential in any git config scope (local/global/system).
* **Fixed on the spot**: `~/.config/secrets/hf_token_write` was mode 0644 (world-readable) → 0600. `~/.cache/huggingface/token` (the standard HF location, holds the same value) verified 0600.
* **Known gap, not yet closed**: `.githooks/pre-commit` covers GitHub/GitLab/Slack/AWS/PEM/JWT/OpenAI-shaped `sk-`, but **not** `hf_…`, `npm_…`, `pypi-…`, `AIza…`, Stripe, Telegram, or a `scheme://user:token@host` basic-auth URL — which is exactly the shape that sat in `remote.origin.url` until today. Widening the gate is an open decision.
Five layers (live-value scan across 77 revs/765 blobs, whole object store
incl. dangling, shape triage, entropy ≥4.3, LFS payloads) — no secret in any
reachable or dangling object; quarantine list holds. Fixed `hf_token_write`
0644→0600. Open gap: the pre-commit gate lacks `hf_`/`npm_`/`pypi-`/`AIza`/
Stripe/Telegram shapes and basic-auth URLs — widening it is an open decision.
## 2026-10-09 — Gitea token moved out of every remote URL
Both workstation clones (`~/.omp` → `dank/command-center`, `~/Projects/serverAppConfig` → `dank/serverAppConfigs`) carried the Gitea HTTP token inside `remote.origin.url`, so it sat in `.git/config`, was printed by `git remote -v`, and was echoed verbatim by the git-lfs lock warning on every push.
Remotes bare; credential in `~/.config/secrets/gitea` (0600) behind a global
`credential.helper store`. Trap (git 2.56.0, measured): `git
credential-store` rejects the `protocol=`/`host=` block shape the man page
shows — the store file must be one URL per line, port included. Verified with
`GIT_TERMINAL_PROMPT=0`. Details: [tools/gitea/client-access.md](tools/gitea/client-access.md).
* **Cutover**: remotes are now bare (`http://192.168.0.143:3000/dank/<repo>.git`); the credential lives in `~/.config/secrets/gitea` (0600) behind a **global** helper — `git config --global credential.helper 'store --file ~/.config/secrets/gitea'` — so every repo on the host authenticates from one file. `lfs.…info/lfs.locksverify true` set on `~/.omp` to silence the warning that leaked the URL.
* **The trap (git 2.56.0, measured)**: `git credential-store` will not parse the `protocol=`/`host=`/`password=` block shape that `git credential fill` accepts on stdin and that the man page shows for `~/.git-credentials`. It returns an empty `get`, and git falls through to askpass: `could not read Username for …`. The store file must be one URL per line, port included: `http://dank:<token>@192.168.0.143:3000`.
* **Verified** with prompts disabled (`GIT_TERMINAL_PROMPT=0`): `git credential-store get` returns the credential, `ls-remote` reads, `push --dry-run` says `Everything up-to-date` on both repos, `git remote get-url origin` contains no `@`, and a grep for the token across `~/.gitconfig`, both `.git/config` files, and the shell rc files returns nothing.
* **Doc**: new [[client-access|client-access]] (`tools/gitea/client-access.md`) — credential file layout, the store-format trap, the verification loop, LFS lock warning, rotation procedure. Cross-linked from the gitea index and management pages.
## 2026-10-09 — Working tree reconciled: callsign renames, compaction move, gitignore quarantine
## 2026-10-09 — Working tree reconciled: callsign renames, Dankclaw compaction, gitignore quarantine
A 200-entry dirty tree (116 deleted, 73 untracked, 11 modified) sat uncommitted across several sessions. Committed as seven atomic units; nothing discarded.
* **Model ids renamed to the unit-name scheme** (`agent/models.yml`): `frank-medium → frank-archon` (dankserver :7770, 200,000 ctx / 16,384 max), `frank-large → frank-dragoon` (Dankclaw :7770, 217,088 / 45,056, `input: [text, image]`), `frank-small → frank-zealot` (Dankclaw-3090 :7771, 200,704 / 16,384). The **services keep the old names on the wire** — `/v1/models` still serves `frank-primary`, `frank-large`, `frank-secondary`; the engines accept an arbitrary request `model` id, so the rename is harness-side only. Probed live: all three chat completions return 200 with `reasoning_content`.
* **Server-side compaction moved** off `192.168.0.235:7770` to `192.168.0.143:7771` (frank-zealot's 3090), threshold 70 % → 85 %, remote streaming v2 + async on; every model's `compactionModel` points at `frank-zealot` so the wire id stays valid at the compaction endpoint. Reference integrity checked programmatically: 22 model references across `modelRoles`, `task.agentModelOverrides`, `compactionModel` — zero unresolved.
* **`.gitignore` quarantine** for runtime state that was tracked-adjacent noise: `autoresearch/`, `agent/predict/`, `local/`, `wt/`, and `agent/memories/--home-blank-.omp-wt-*` (a session inside a git worktree mints a memory bucket keyed on the ephemeral worktree path). All had zero tracked files, so the rules are purely additive — 73 untracked → 68, and the remaining 68 are real memory records that got committed.
* **Skill libraries were renamed, not lost.** `agent/managed-skills` consolidated 94 → 25 (git renders directory renames as mass delete + mass add; pre-cutover content is intact at `12c5e88`). Workstation memory skills re-bucketed by topic: `dotfile-git-workflow` → `ghostty-text-aa` + `niri-display-rotation` + `noctalia-theme`, `handy-service-mgmt` → `handy-flm-inference`, `inference-runtime` → `llama-cpp-parallelism`.
* **Display layout**: blankpad DP-2 back to landscape `3440x1440@239.999` (`transform "normal"`), HDMI-A-1 portrait bezel flipped `270 → 90`, eDP-1 re-pinned rightmost. Supersedes the 2026-09-15 portrait pin (`2baf0c0`).
* **Secrets**: `.zshrc` now reads `KAGI_API_KEY` / `HF_TOKEN` from `~/.config/secrets/*` (600-perm files) instead of carrying literals into a repo published to Gitea. The pre-commit secret gate passed on all seven commits.
200-entry dirty tree committed as seven atomic units. Harness model ids
renamed `frank-{medium,large,small}` → `frank-{archon,dragoon,zealot}` —
**services keep old names on the wire**; the engines accept any request model
id, so renames are harness-side only. Server-side compaction moved to
zealot's 3090 (threshold 70→85 %). `.gitignore` quarantine for worktree-keyed
memory buckets and runtime dirs. Skill renames are not losses: git renders
directory renames as mass delete+add; pre-cutover content intact at `12c5e88`.
## 2026-09-28 — Retired workstation purged from the knowledge base
Frank sold the client workstation. Every identifier for it is gone from this bundle: hostname, both of its LAN addresses, and its Wi-Fi SSID. The knowledge outlives the box; the machine does not.
Sold box: hostname, both LAN addresses, Wi-Fi SSID gone bundle-wide; the
knowledge distilled into [system/network.md](system/network.md) (multi-link
resolver law: a static resolver must be pinned on each *connection*) and
[tools/niri/displays.md](tools/niri/displays.md) (portrait transforms, the
three ways a multi-monitor rig breaks). Hostname stays only where it is a
lookup key (install.sh `--host`, `packages/<hostname>.pkgs`).
* **Distilled, not deleted.** The four per-host system pages carried two pieces of knowledge worth keeping and one that was not:
* `/system/network.md` (new) — LAN topology, plus the multi-link resolver law: a client with two `+DefaultRoute` links flaps its DNS, and the fix is a static resolver pinned on each *connection* (a `resolved.conf` `DNS=` line is only a per-link fallback). Keeps the flush-caches verification loop, because one successful `dig` proves nothing on a flapping resolver.
* `/tools/niri/displays.md` (new) — the multi-monitor rig: portrait transforms, absolute positions, the auto-placed third panel, the iGPU render-node pin, and the three ways it breaks (duplicate output blocks, positions that do not reflow, a mode the panel cannot drive).
* Dropped: the machine's own spec sheet and quick-facts table. The OS and userspace facts they repeated already live in `tools/cachyos.md`.
* **Mechanical scrub** across the rest of the bundle: indexes, tool and research pages, the inventory census, the changelog, one generated HTML report, and four managed skills. Facts kept, names dropped — `install.sh`'s `--host` case list and the `packages/<hostname>.pkgs` filename stay, because the hostname is the lookup key there.
* `AGENTS.md` and `README.md` keep a nameless "workstation" row so the fleet tables still describe the class of box without advertising a machine that is no longer Frank's.
* Out of scope by decision: `agent/memories/**` (past-session records the learn pipeline rewrites) and the retired pages' history — they are recoverable from `.AGENTTRASH/20260928-retired-workstation/` and from git.
## 2026-09-01 — Retry backoff and thinking default
* **Retry backoff**: configured Pi's automatic agent-level retries in `settings.json` with a 10-second base delay, producing 10s, 20s, and 40s delays across three retries.
* **Thinking default**: set `settings.json` `defaultThinkingLevel` to `medium`. `models.json` has no supported per-model default-thinking field; Pi applies this global default to the configured models.
## 2026-09-01 — Retry backoff and thinking default (pi era)
Pi `settings.json`: 10 s base retry backoff (10/20/40 s), `defaultThinkingLevel:
medium` (no per-model thinking field exists in `models.json`).
## 2026-08-28 — KB rebuilt to OKF v0.1; live-verified fleet facts
* **Creation**: the whole `omp-docs/` bundle was copied from the legacy `~/.pi/agent/docs` tree and rebuilt as a git-tracked, OKF v0.1-conformant knowledge base inside the new `command-center` repo (repo root = live agent dir; the checkout *is* the deployment). Every checkable claim was re-verified against the live fleet (read-only probes); unverifiable claims are tagged `unverified` in frontmatter.
* **Update — hardware facts corrected**: Dankserver is a **Ryzen 7 7700X + RTX 3090** (not Strix Halo); Strix Halo (gfx1151) is **Dankclaw**. RAM, storage, kernel, and network facts were probed live and corrected across `system/`.
* **Update — conformance pass**: concept-in-disguise indexes promoted to `<dirname>.md` concept files (identity = path, per spec); index.md files stripped of frontmatter (indexes carry none); all wikilinks and relative links resolved or repointed; `scripts/kb-lint.sh` (OKF conformance + link-integrity linter) exits 0 on the bundle. Vendored `references/okf-spec.md` is exempt from link-lint — its example paths are illustrations, not links.
* **Update — inventory**: home-config census (7 orphaned configs, 10 prune-candidates) recorded under `inventory/` for the pruning session; zero deletions performed. Legacy `~/.pi` tree left untouched and frozen.
* **Deferred to operator** (decisions, not facts): chezmoi source layout (`research/chezmoi-evaluation.md` recommends a separate chezmoi repo) and the permanent secrets tool (`research/secrets-for-agents.md`; interim = pre-commit secret gate + quarantine list; top candidate sops+age). Gitea Projects unit is disabled instance-wide, so the kanban board is a one-time web-UI setup for the operator.
`omp-docs/` copied from legacy `~/.pi/agent/docs`, rebuilt as git-tracked OKF
v0.1 (checkout *is* the deployment); unverifiable claims tagged `unverified`.
Conformance pass promoted concept-in-disguise indexes, stripped index
frontmatter, resolved all links; `scripts/kb-lint.sh` exits 0. Hardware facts
were corrected *then* (3090-on-dankserver since moved — see 2026-10-18).
Inventory census recorded under `inventory/` for the pruning session.
## 2026-08-23 — Homepage deployed to production; Caddy KB doc
* **Homepage is live on Dankclaw.** `./scripts/deploy.sh dankclaw` deploys the static files to `/srv/homepage`, installs the status checker (systemd timer, every 3 min, writes `status.json`), writes `deployed.js` (the last master deployment datetime), renders the Caddyfile (both basicauth hashes from the host `.env`), reloads Caddy, and checks health. Verified end-to-end: `/admin/` (operator) + all shared files serve 200 under the operator credential; `/` and `/admin/` 401 without/wrong credentials; the LLM is healthy.
* **Caddyfile auth design (the hard-won fix).** Caddy *sorts* top-level directives, so a `rewrite` runs before a `basicauth`; the rewrite changed `/admin/` to `/admin.html` and the path-based auth matchers then saw the rewritten path, applying the wrong (family) auth block — the operator password 401'd. Fix: serve each page from its own `handle` block (which preserves file order, so the auth is checked on the original path): `handle /admin*` (operator only), `handle @data` (shared data + assets, family OR operator, so the admin page's fetches reuse the operator credential), and a catch-all `handle` (family). Also: a `handle` inline matcher is a single token (multi-path needs a named matcher); a leading `@name <directive>` is parsed as a matcher *definition* (re-defines `@name`).
* **Caddy mount fix.** The caddy container mounted the repo's `shared/homepage` (static source) to `/srv/homepage`; the generated files (`status.json`, `deployed.js`) are written to the host's `/srv/homepage`. Pointed the mount at the host's `/srv/homepage` so the generated files serve. A compose change needs a container *recreate* (not a reload) — `podman compose up -d caddy` (the deploy skips the compose restart, so this is manual).
* **Caddy KB doc** (operator request: read the docs before guessing at syntax): new `caddy/` section in this bundle — [[caddyfile|caddyfile]] (`caddy/caddyfile.md`) + `caddy/index.md`, cross-linked from the root `index.md`. Grounded in caddyserver.com (fetched 2026-08-23) + a researcher subagent brief.
* **Tests:** 116 green (was 112; +4 for the Caddyfile handle design + the `status.json` 0644 perms). The local `caddy validate` test is skipped (no local caddy binary); the rendered config is validated on the target via the porkbun image before each deploy.
Homepage live on Dankclaw via `scripts/deploy.sh` (status checker timer,
deployed.js, rendered basicauth hashes, health check). Hard-won Caddy lesson:
top-level directives are *sorted*, so `rewrite` runs before `basicauth` and
auth matchers saw rewritten paths — serve each page from its own `handle`
block to preserve order; a `handle` inline matcher is a single token; a
leading `@name <directive>` parses as a matcher definition. Now documented on
[caddy/caddyfile.md](caddy/caddyfile.md). A compose mount change needs a
container *recreate*, not a reload. 116 tests green.
## 2026-08-23 — Homepage design settled (grilling complete)
* **All 38 open questions (Q6–Q43) answered.** The homepage design is now fully settled. Full record: `serverAppConfigs/docs/Infrastructure/Homepage Design Grilling.md` (final decisions + the deltas) and `docs/Infrastructure/Homepage.md`.
* **Key deltas from the initial recommendations:** the probe method is **per service** (`tcp` or `http`); the probe timeout is **5 s**; the layout is **TUI-style panels** (a grid), not a flat list; styling is **toggleable** (full theme default, a toggle to bare unstyled HTML); **do not alter the user's cursor**; **no client-side auto-refresh** (read `status.json` once on load); the footer shows the **last master deployment datetime** (not last-check); the deploy is **git-driven** (master = deployed state); the build is **full TDD**.
* **Per-service docs workstream:** each service gets a `/docs/<svc>/` page (not `/usage/`), written as a **how-to for a regular user** (no maintenance details), password-protected by the same auth as its homepage section, built as each service goes active. The existing reference docs in `docs/` are NOT linked from the homepage (they may contain secrets).
* **Build complete (all 6 slices, full TDD, 112 tests green):** 1 the checker core (tcp/wip/atomic write/generated_at), 2 the http probe, 3 the real `services.json` + schema, 4 the pages + theme + toggle, 5 the Caddyfile auth split (root vs `/admin/`), 6 the git-driven deploy. All committed + pushed. The render logic is verified with a Node DOM shim (9 user panels, correct summary, correct footer datetime).
* **To go live:** the operator adds `CADDY_ADMIN_HASH` (the operator-only password, `caddy hash-password`) to the Dankclaw `.env`, then runs `./scripts/deploy.sh dankclaw`. Until then the admin hash stays a placeholder. The per-service `/docs/` pages, the *arr stack, and the optional `/how/` page remain separate threads.
* **Correction (media stack):** the Dankserver media stack (RomM, Copyparty, Jellyfin, Plex, the *arr) is **UP** — it runs as **bare processes/services, not yet migrated to podman containers**. The 2026-08-22 "down / missing `.env`" finding was wrong: I was probing for the compose stack, which is not how it runs yet. Do not treat it as an outage and do not probe it as down. Its podman migration is a separate, future effort.
All 38 open questions answered; full record lives in serverAppConfigs
(`docs/Infrastructure/Homepage Design Grilling.md`). Key deltas: per-service
tcp/http probes, 5 s timeout, TUI-style panel grid, no client auto-refresh,
git-driven deploy, full TDD. Correction recorded: the Dankserver media stack
runs as bare processes, not containers — never probe it as a compose stack.
## 2026-08-22 — vllmScripts merged into serverAppConfigs
* **Repo merge**: the `vllmScripts` repo (LLM inference fleet: serve scripts, systemd units, env tunables, deploy runbook, research docs) is merged into `serverAppConfigs` with its full git history (two-parent import commit, then reorganization: `docs/` → `docs/LLM/`, `scripts/` merged, `old/` for retired PP/NPU-era scripts). One repo, one checkout per host (`/home/dank/serverAppConfigs`), one `scripts/deploy.sh` covering both workloads per target (podman stack + LLM service relink/restart + both health checks).
* **Units**: frank-default-{small,large,npu}.service now point at `/home/dank/serverAppConfigs/scripts/...`. The retired target checkouts (`/home/dank/vLLM` on Dankserver, `/home/dank/vllmScripts` on Dankclaw) are to be archived on the hosts during the production deploy.
* **Secrets**: the live sudo password that the old vllmScripts AGENTS.md documented in plaintext is NOT carried into the merged repo (a bats leak-check guards it). It remains in the git history of both repos on the LAN Gitea — rotation recommended.
* **KB links**: `system/index.md`, `system/dankserver.md`, `system/dankclaw.md`, and the root `index.md` updated — the "sibling knowledge bases" section is now a single sibling (serverAppConfigs, which contains the LLM fleet KB at `docs/LLM/index.md`).
* **Port move**: `frank-default-large` (dankclaw) moved from `:52625` to `:7770` (operator request: the primary inference server sits on the standard llama.cpp port). Both LLM services now listen on `:7770`, distinguished by host. `models.json` (`llamacpp (dankclaw)`) and the OpenWebUI quadlet (`OPENAI_API_BASE_URL`) re-pointed to `:7770`. `pi/models.md` updated.
* **Homepage design (grilling)**: the `serverAppConfigs` homepage (`docs/Infrastructure/Homepage.md`) is WIP — the Caddy site is live (basicauth 401, `:80` 302) but `/srv/homepage/` is not deployed. Settled design: a single site on Dankclaw (root = user services, `/admin/` = admin services behind a second operator-only password, optional `/how/` tourist page). Status is checked **server-side** — a systemd-timer checker (every 3 min) probes each backend `host:port` and writes a same-origin `status.json`; the page reads only that file (the client-side-ping design is blocked by cross-origin + mixed-content + auth). Full decision log + 38 open questions (Q6–Q43) in `docs/Infrastructure/Homepage Design Grilling.md`. A terminal-TUI-styled HTML progress report was generated for the operator to review and answer the open questions. Full-stack baseline verified: Dankclaw fully up; the Dankserver media stack is down (only Calibre-Web + the LLM are up; no `.env` on the target).
Full-history two-parent import; one repo, one checkout per host, one
`deploy.sh`. The plaintext sudo password from old docs was NOT carried over
(a bats leak-check guards it); it remains in both repos' history on the LAN
Gitea — rotation recommended.
## 2026-08-21 — Fleet ops: passwordless sudo, Hermes SSH keys, sibling KB links
## 2026-08-21 — Fleet ops: passwordless sudo, Hermes SSH keys
* **Passwordless sudo**: added `/etc/sudoers.d/dank-nopasswd` (`dank ALL=(ALL) NOPASSWD: ALL`) on both Dankserver and Dankclaw. Verified `sudo -n true` on both. The pre-existing drop-ins (`frank-small` on Dankserver; `frank-large` + `frank-ling` on Dankclaw) grant per-command NOPASSWD for the frank restart scripts and remain in place.
* **Hermes SSH keys**: the Hermes Agent (podman container `hermes`, project `dankclaw` in `serverAppConfigs`, gateway drops to container uid 10000 = host uid 109999, HOME `/opt/data` on a podman volume) now has a dedicated ed25519 key at `/opt/data/.ssh/id_ed25519` (`hermes-agent@dankclaw`) plus an SSH config for `dankclaw` (192.168.0.143), `dankserver` (192.168.0.235), and the workstation. The public key is in `dank`'s `authorized_keys` on all three machines. Verified from inside the container as the hermes user: all three `ssh <host> hostname` calls succeed.
* **Sibling KB links**: `system/index.md`, `system/dankserver.md`, `system/dankclaw.md`, and the root `index.md` now link the two sibling knowledge bases on the workstation: `~/Projects/vllmScripts/` (inference fleet scripts + runbook, index `docs/index.md`) and `~/Projects/serverAppConfigs/` (Podman deploy configs for both servers, index `docs/index.md`).
`/etc/sudoers.d/dank-nopasswd` on both servers (per-command drop-ins remain);
dedicated ed25519 key for the Hermes container, verified into all three
machines.
## 2026-08-19 — pi-web-access: SearXNG endpoint re-pointed to dankclaw
## 2026-08-19 — pi-web-access: SearXNG endpoint re-pointed to dankclaw (pi era)
* **Config**: `~/.pi/web-search.json` — `searxngBaseUrl` moved from the dead `192.168.0.235:8081` (dankserver, connection refused) to `192.168.0.143:8081` (dankclaw; verified live: HTTP 200, 25 JSON results, 0.8 s). Deleted the redundant `provider: "all"` (the `searchProvider: "searxng"` pin already wins in v0.22). Added `maxInlineContentChars: 50000`. Kept `workflow: "none"` and `ssrf.allowRanges: ["192.168.0.0/24"]`.
* **Docs**: `pi/packages/pi-web-access.md` bumped 0.14.0 → 0.22.0 (18-provider chain, config block, active-setup note, GitHub-clone section, yt-dlp present). `agentic-web-research/` SearXNG endpoint references repointed `.235` → `.143` (index, searxng-tuning, vpn-routing).
* **Note**: pi-web-access caches `web-search.json` in memory per process — the new endpoint takes effect on the next Pi session, not the running one.
* **Packages**: `yt-dlp` installed on the workstation (2026.07.04) — YouTube frame extraction now works; `ffmpeg` (n9.0.1) was already present.
`searxngBaseUrl` moved off dead `.235:8081` to `.143:8081`; pi caches
`web-search.json` per process — new endpoint needs a new session. `yt-dlp`
installed (2026.07.04).
## 2026-08-19 — SearXNG + Gluetun bring-up on dankclaw
Field notes from standing up the search stack (gluetun + searxng + valkey) and confirming agentic JSON search. All four are operational; the SearXNG trio runs via podman-compose (not yet boot-persistent quadlet).
Durable field notes (now on concept pages — see
[agentic-web-research/vpn-routing.md](agentic-web-research/vpn-routing.md),
[searxng-tuning.md](agentic-web-research/searxng-tuning.md),
[engineering/infra/containers.md](engineering/infra/containers.md)):
manual-WireGuard Proton needs `VPN_SERVICE_PROVIDER=custom` + the single-peer
`WIREGUARD_*` env names (not `protonvpn`, not `WIREGUARD_PEERS_*`); gluetun
ships no curl and `:8000` is the control server, not health; the limiter's
JSON cap (`API_MAX=4`/hour) is hardcoded Python — `pass_ip` is the only
bypass; rootless podman strips client IPs on published ports, so SearXNG must
sit behind Caddy for XFF; quadlet generator rejects `CapAdd=`/`Device=` and
generated units enable via `[Install] WantedBy=default.target`. ComfyUI on
gfx1151: build on AMD's ROCm base and filter the torch family out of
ComfyUI's requirements or you clobber the gfx1151 kernels.
* **Gluetun manual WireGuard**: a specific-server ProtonVPN config (private key + peer + endpoint) uses `VPN_SERVICE_PROVIDER=custom`, NOT `protonvpn`. The `protonvpn` provider is account/token-based and rejects a manually-set endpoint with `endpoint port is set`. Current Gluetun uses the single-peer model: `WIREGUARD_PUBLIC_KEY`, `WIREGUARD_ENDPOINT_IP`, `WIREGUARD_ENDPOINT_PORT`, `WIREGUARD_ADDRESSES` (not the old `WIREGUARD_PEERS_*` names). DNS is Gluetun's own `DNS_*` vars, not the WG `DNS` line.
* **Gluetun healthcheck**: the image has no `curl`, and its `:8000` status server 404s on every path. Use `test: ["CMD","sh","-c","test -e /sys/class/net/tun0"]` — tun0 exists only while the tunnel is connected.
* **SearXNG trusted group**: lives in `limiter.toml` (not `settings.yml`). A direct hit on the published port has no `X-Forwarded-For` header, so the bot-detection cannot resolve a client and returns 429 regardless of `pass_ip`. SearXNG must be reached through Caddy (which sets XFF); `trusted_proxies` then needs the podman bridge subnet (dynamic in rootless — pin the network CIDR) so Caddy's source is trusted for XFF extraction.
* **Rootless podman gotchas**: short image names do not resolve (no unqualified-search registries) — qualify with `docker.io/`. Binding privileged ports (Caddy 80/443) needs the one-time sysctl `net.ipv4.ip_unprivileged_port_start=80`.
* **Repo**: `serverAppConfigs` — added `shared/searxng/limiter.toml` + test seam, wired it into `deploy.sh`, fixed the Gluetun compose (peer env vars, qualified images, tun0 healthcheck), documented the rootless-port + Caddy/XFF gotchas, and skipped the ComfyUI GPU tests (migration pending).
* **Boot-persistence (quadlet)**: converted the trio from podman-compose to quadlet so it survives reboot. searxng sets `Network=gluetun.container` (reuses gluetun's netns → exits the tunnel); gluetun + valkey join a pinned `searxng.network` (10.89.2.0/24). Deployed via `scripts/deploy-searxng-quadlet.sh`. Field gotchas (also in `engineering/infra/containers.md`): the generator rejects `CapAdd=`/`Device=` (use `AddCapability=`/`AddDevice=`) and silently skips the unit; generated units cannot be `systemctl enable`d — `[Install] WantedBy=default.target` is the enablement (`is-enabled` = "generated"), and linger makes them boot-start. Verified: all three `is-enabled` = generated, tunnel up (US-WA), agentic JSON search returns 27 on-topic results.
* **open-webui moved to dankclaw (quadlet)**: the old open-webui was a bare container on Dankserver (:3000) pointing at a dead `:9090` backend. It now runs on Dankclaw as a boot-persistent quadlet (`scripts/deploy-open-webui-quadlet.sh`): CPU image `:main` (Dankclaw is CPU-only), isolated on `Network=slirp4netns`, published `:3005` (:3000 is Gitea's HTTP port), data at `/home/dank/Containers/open-webui/data`. Default LLM backend is the local vLLM (`host.containers.internal:52625`). The Dankserver `searxng` + `open-webui` bare containers were retired (data dirs preserved). `podman inspect <name>` can match an IMAGE, not just a container — a removed container still "exists" to inspect if its image remains; use `podman ps -a --filter name=` to check containers.
* **Model/port mapping corrected (live, 2026-08-19)**: the fleet endpoints are the REVERSE of the 2026-08-17 note. Live truth (probed `/v1/models`): `frank-default-large` is on **Dankclaw :52625** (vLLM, completion) and `frank-default-small` is on **Dankserver :7770** (llama.cpp FastMTP, completion+multimodal). The `serverAppConfigs` docs (Stack, OpenWebUI) were corrected; the sibling `vllmScripts` KB still carries the old mapping and should be updated there.
* **SearXNG 429 root cause (the "diva")**: the limiter's JSON cap is `API_MAX = 4` requests/IP/hour and that constant is **hardcoded in Python** (`searx/botdetection/ip_limit.py`), NOT settable in `limiter.toml`. The only bypass is `pass_ip` (SearXNG's `filter_request` returns before `ip_limit` runs when the IP is in `pass_ip`). Second gotcha: **rootless podman strips the client IP on published-port hits** — a request to the published port reaches the container with a slirp/bridge source (`10.88.x`/`10.89.x`), never the real client; only Caddy's `X-Forwarded-For` restores it. So even a LAN-IP curl 429'd. Fix: expand `pass_ip` to loopback (`127.0.0.1/32`, `::1`) + the host's private podman/slirp ranges (`10.88.0.0/16`, `10.89.0.0/16`) + the LAN. Verified: 20 concurrent JSON searches → all 200 (real results), while a spoofed public IP (8.8.8.8) still 429s (limiter still armed). The config file is root-owned (uid of the rootless podman namespace) — update with a one-off `sudo cp` + `systemctl --user restart searxng.service`.
* **ComfyUI GPU (gfx1151) — DONE (Option 1/B)**: the deferred GPU ComfyUI is now built and running. It was previously a bare CPU venv (`/home/dank/ComfyUI`, py3.14, `torch 2.13.0+cu130` with no NVIDIA → CPU fallback). Now it's a **container** `comfyui-rocm:gfx1151` built from `dankclaw/Containerfile.comfyui` on AMD's `rocm/vllm:rocm7.12.0_gfx1151_..._py3.12_pytorch_2.9.1` base (the only torch build where `torch.cuda.is_available()=True` on the Strix Halo APU). Key gotchas: (1) the base image already ships the ROCm torch/torchvision/torchaudio, so the Containerfile must **skip the torch family** from ComfyUI's `requirements.txt` (an unpinned `torch` would clobber the gfx1151 kernels) — filter with `grep -viE '^(torch|torchvision|torchaudio)([^a-z]|$)'`. (2) **rootless podman cannot `--group-add render`** (gid 990 is outside the subgid range 100000+); GPU access comes from the `/dev/kfd`+`/dev/dri` passthrough checked host-side (the run user is already in `render`), so add only the `video` group. Verified: `/system_stats` → `cuda:0 Radeon 8060S`, logs show `AMD arch: gfx1151` + active HIP backend. Boot-persistent via a user systemd unit running `podman compose up -d comfyui` (old venv unit saved as `comfyui.service.venv.bak`). The 4 `comfyui_rocm.bats` tests are un-skipped (the group test now asserts video present + render ABSENT).
## 2026-08-17 — Fleet redo + vLLM tool calling (pi era)
## 2026-08-17 — Fleet redo + vLLM tool calling
Fleet moved off the pipeline-parallel + NPU topology; Quark W8A8 quant chosen
over the Avesed INT8 for clean Qwen3 XML tool calls.
* **Update**: `pi/models.md` — providers rewritten for the fleet redo: `llamacpp (dankserver)` at :7770 and `vllm (dankclaw)` at :52625; model selection strategy and live defaults brought in line with the new topology and the Quark W8A8 model.
* **Context**: the Frank fleet moved off the pipeline-parallel + NPU topology. `frank-default-large` (llama.cpp FastMTP, IQ4_XS) is on Dankserver; `frank-default-small` (vLLM ROCm, nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8) is on Dankclaw with ngram spec and tool calling. The Avesed INT8 quant's tool-call XML fidelity was poor; the Quark quant follows the Qwen3 XML format cleanly.
## 2026-08-16 — Gitea boot-persistence fix (dankclaw)
`unattended-upgrades` reboot exposed that the compose-era Gitea had no
systemd units (~1 h down). Stack rewritten as quadlet (names, `systemd-gitea`
network, linger + `WantedBy=default.target`, app.ini env-override gotcha, UID
100999 ownership) across `tools/gitea/*`; `podman generate systemd` marked
deprecated in [engineering/infra/containers.md](engineering/infra/containers.md).
## 2026-08-16
## 2026-08-13 — Pi subagent limits recorded (pi era)
### Gitea Boot-Persistence Fix (dankclaw)
Effective caps: globalConcurrencyLimit 2, parallel.maxTasks 8 / concurrency 4,
maxSubagentDepth 2, turnBudget 30; no per-model limit. See
[pi/packages/pi-subagents.md](pi/packages/pi-subagents.md).
* **Incident**: `unattended-upgrades` rebooted dankclaw at 15:01. The legacy podman-compose Gitea stack had no systemd units, so rootless podman did not auto-start the containers. Gitea was down ~1 hour.
* **Update**: `tools/gitea/index.md`, `architecture.md`, `management.md`, `data-persistence.md` — rewritten for the quadlet-managed stack: container/unit names, the `systemd-gitea` network (DNS-enabled), boot persistence (linger + `WantedBy=default.target`), the app.ini env-override gotcha, UID 100999 ownership, incident log.
* **Update**: `engineering/infra/containers.md` — marked `podman generate systemd` deprecated (podman 5.7), documented Quadlet with a field-tested gotcha list (default network DNS off, `[Service] EnvironmentFile=` for `${VAR}` expansion, unit naming, bind-mount UID mapping).
## 2026-08-11 — Engineering research area; skills leaderboard verified
## 2026-08-13
Created `engineering/` (SDLC, agent instruction files, stacks, infra,
plugins, skills ecosystem, Signal remote) via parallel research fleet.
Skills leaderboard replaced with the ranked list from the skills.sh API
(zero-based pagination, 200-skill pages).
### Pi Subagents Limits
## 2026-08-11 — Agentic web research section; exhaustive bundle audit
* Added a "Parallel Execution and Limits" section to `pi/packages/pi-subagents.md`.
* Documented the effective values: `globalConcurrencyLimit` 2 (from `extensions/subagent/config.json`), `parallel.maxTasks` 8, `parallel.concurrency` 4, `maxSubagentSpawnsPerSession` unlimited, `maxSubagentDepth` 2, and `turnBudget.maxTurns` 30.
* Noted that no per-model limit exists; all caps are global to a run or a session.
Created `agentic-web-research/` (8 pages) and rewrote them on 2025/2026
evidence (SearXNG valkey deployment, real engine defaults, limiter budgets,
Gluetun pattern, open-model claim triage, MCP transports). Audit: 154+
wikilinks resolved, dead external links fixed, `MemorySleepMode=deep`
verified valid (systemd 256+), ASD-STE100 style pass. `AGENTS.md` created as
the system behavior contract.
## 2026-08-11 (continued)
## 2026-08-06 — Link fixes; ComfyUI docs; major documentation overhaul
### Engineering Research — New Area
Fixed 15 broken wikilinks (missing frontmatter) + title mismatches → zero
broken. Added ComfyUI pages, per-host system docs, pi config tree, tools
entries; orphaned root duplicates moved to `.AGENTTRASH/`.
* Created the `engineering/` research area: SDLC and tooling, agent instruction files, language and framework stacks, infrastructure and cloud, pi plugins, skills ecosystem, and Signal remote control.
* Ran a parallel research fleet (scout + 6 researchers) grounded in current 2025/2026 web sources.
## 2026-07-18 — Maintenance tooling
### Skills Leaderboard — Live Verification
Expanded `references/okf-maintenance.md` (cross-links, health metrics,
troubleshooting); created `core/log.md` and `.okf/config.yaml`.
* Queried the skills.sh API (`GET /api/skills/all-time/{page}`) for the authoritative top 50 skills by all-time installs.
* Replaced the inferred category table in `engineering/skills-ecosystem.md` with the verified ranked list.
* Noted the API’s zero-based pagination and 200-skill page size for future queries.
## 2026-01-15 — OKF conversion
## 2026-08-11 (continued)
### Agentic Web Research — Grounded Revision
* Ran a parallel research fleet (7 researchers + 2 scouts) for current 2025-2026 evidence on SearXNG tuning, VPN split-routing, agentic search, open models, research orchestration, and MCP.
* Rewrote all 8 `agentic-web-research/` pages to match verified facts and the local topology: SearXNG at `192.168.0.235:8081`, llama.cpp inference on `7770` (dankserver/dankclaw) and `8080` (local-ornith), main model `openrouter/deepseek-v4-flash-0731`.
* SearXNG: corrected to Valkey deployment, real engine defaults (DuckDuckGo/Wikipedia/Google News active, Google inactive, Bing disabled), limiter budgets, current `outgoing` defaults.
* VPN: added Gluetun as the recommended pattern, corrected OpenVPN container joins, added `source_ips`/`network: ipv4`.
* Open models: clarified Llama 4 claims (10M ctx = Scout only, single-H100 needs INT4, "beats GPT-4o" is Meta-only), added Qwen2.5-VL-7B as best self-hostable vision model.
* Web agents: corrected the Browser Use "87.4%" claim (managed Cloud variant, not open-source; peer-reviewed 44.5% for CUA), added Skyvern.
* MCP: updated transports (stdio + Streamable HTTP, legacy SSE), current reference servers, added SearXNG MCP, tightened security rules.
### Exhaustive Bundle Audit
* Link audit: resolved all 154+ wikilinks by frontmatter title — **0 broken wikilinks**. Fixed relative-link base errors in `core/index-doc.md`, `packages/index.md`, `pi-web-access.md`, `llama-cpp/setup.md`, `npx-skills/skill-commands.md`, `packages/pi-browser-use/index.md`.
* External URL audit: fixed dead links (SearXNG limiter path, earendil-works/pi blob URLs, AYayOS/qs -> outfoxxed/quickshell, badlogic/pi-mono -> earendil-works/pi).
* Factual fixes: `core/settings.md` stale JSON (defaultProvider/model/thinkingLevel/changelog-version), `pi/models.md` dankclaw auth, `Sleep-Fix-Guide.md` kernel 7.1.3 -> 7.1.4.
* Verified (did not change) `MemorySleepMode=deep`: it is a valid systemd 256+ option.
* ASD-STE100 style pass: removed contractions, banned `may`, and prose semicolons across research/tools/pi docs.
* Added OKF index frontmatter to `core/index.md` and `skills/index.md`.
* Added a "Pi-peon is installed" note to the npx skills ecosystem table and a port-7770 system note to llama.cpp docs.
### Agentic Web Research Section
* Created `agentic-web-research/` — seven new concept pages plus index covering self-hosted web agents, SearXNG tuning, VPN routing, agentic search practices, open models, research effectiveness, and model context protocol.
* Expanded via parallel research: concrete BM25/Pruning/LLM content filters, credibility heuristics, failure-mode guardrails, MCP server and transport guidance.
* Added the section to `index.md` under a new `## [Agentic Web Research]` heading.
* Linked `pi/packages/pi-web-access.md` to the new section.
### System Instruction File
* Created `AGENTS.md` — system behavior contract with a dedicated layer on parallel subagent execution (one `subagent` call with `tasks` + `concurrency`).
### Bundle Audit
* Verified all 154 wikilinks resolve (zero broken).
* Removed stale empty `Alienware M16a2 Sleep/Wake Issues.md` duplicate into `.AGENTTRASH`.
* ran ASD-STE100 style pass: converted `should` to imperative or `must`, and `e.g.` to `for example` across 10 files.
* Removed stale `frank-default-omni` model references and corrected dankclaw port `13305` to `8080` per commit `15d4af4`.
* **Resolved**: the live config is authoritative. Updated both llama.cpp servers to port `7770` and set the default provider to `openrouter` (deepseek) per `settings.json`. Dankserver/dankclaw now 7770; main session default is openrouter.
## 2026-08-06 (continued)
### Link Fixes
* Fixed 15 broken wikilinks — added missing YAML frontmatter to `ghostty/index.md`, `tmux/index.md`, `linux-power-management/index.md`, `gitea/index.md`, `gitea/architecture.md`, `gitea/data-persistence.md`, `gitea/management.md`
* Fixed title mismatches: `OKF v0.1 Specification` → `Open Knowledge Format (OKF) v0.1 Specification`, `Pi Settings` → `Pi Settings System (settings.json)`, `Alienware M16a2 Sleep/Wake Issues` → `Alienware M16a2 Linux Sleep/Wake Issues`, `Architecture Details` → `Gitea Architecture`, `Data Persistence` → `Gitea Data Persistence`, `Management Guide` → `Gitea Management`
* Zero broken links remaining (47 links verified)
## 2026-08-06
### ComfyUI Documentation
* `tools/comfyui/index.md` — ComfyUI setup on dankclaw, architecture, service management
* `tools/comfyui/troubleshooting.md` — Common errors, diagnostics, recovery steps
* `tools/index.md` — Added ComfyUI entry
* `system/dankclaw.md` — Added ComfyUI service, hardware specs, services section
### Major Documentation Overhaul
**Created — System Documentation**
* `system/index.md` — System documentation index with infrastructure diagram
* Per-host system pages (overview, hardware specifications, display configuration, network interfaces) for the workstation — retired later; see `/system/network.md` and `/tools/niri/displays.md`
* `system/dankserver.md` — Primary llama.cpp server documentation
* `system/dankclaw.md` — Secondary llama.cpp server documentation
**Created — Pi Configuration**
* `pi/index.md` — Pi configuration overview with architecture diagram
* `pi/settings.md` — Main settings.json reference (moved from core/)
* `pi/models.md` — models.json provider and model definitions
* `pi/packages/index.md` — Installed packages index
* `pi/packages/pi-peon.md` — CESP/OpenPeon sound-pack player
* `pi/packages/pi-web-access.md` — Web search, fetching, video understanding
* `pi/packages/pi-subagents.md` — Core subagent delegation
* `pi/skills/index.md` — Installed agent skills inventory
**Created — Tools**
* `tools/cachyos/index.md` — CachyOS documentation (kernel, repos, gaming stack)
* `tools/noctalia-shell/index.md` — Noctalia Shell with Niri integration
* `tools/applications/index.md` — Default applications reference
**Updated**
* `core/settings.md` — Updated with current config, subagent overrides, cross-links
* `tools/niri/index.md` — Rewritten with actual system configuration
* `tools/index.md` — Added CachyOS, Noctalia, Applications entries
* `index.md` — Complete rewrite with new sections and cross-links
**Moved / Cleaned**
* `tmux.md` → `tools/tmux/index.md`
* Orphaned root duplicates → `.AGENTTRASH/` (npx-skills.md, okf-maintenance.md, okf-spec.md, vercel-labs-skills.md, vercel-labs-skills-find-skills.md, ornith-llama-cpp-setup.md)
---
## 2026-07-18
* **Enhancement**: Expanded `okf-maintenance.md` — added cross-link management, index management, health metrics, troubleshooting, and quick reference card
* **Creation**: `core/log.md` — change log for the core documentation directory
* **Creation**: `.okf/config.yaml` — bundle tooling configuration
* **Update**: `core/index.md` — added log.md entry and cross-references
## 2026-01-15
* **Initialization**: Converted docs bundle to OKF v0.1 format
* **Migration**: Migrated files to `core/`, `tools/`, `packages/`, `references/` directories
* **Applied**: OKF frontmatter to all files
Docs bundle converted to OKF v0.1; files migrated into `core/`, `tools/`,
`packages/`, `references/` with frontmatter on all files.
## 2025-08-05 — Gitea documentation
Added ](tools/gitea/index.md) documentation to tools. Covers:
* [[architecture|architecture]] — Container topology, Caddy reverse proxy, network diagram.
* [[management|management]] — Podman commands, upgrades, troubleshooting.
* [[data-persistence|data-persistence]] — Volume mounts, backup strategy, restore procedures.
Updated tools index.
Added [tools/gitea/index.md](tools/gitea/index.md) (topology, management,
data persistence) to the tools index.