command-center/omp-docs/log.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
changelog	Bundle Change Log	Chronological update history for the OKF knowledge bundle.	okf maintenance changelog	2026-08-28

Bundle Change Log

2026-10-18 — Fleet law: filed PRs open in Frank's browser immediately

Both repos' git rules now require it: the moment a PR is opened, xdg-open <PR-URL> drops a Firefox tab on the workstation (AGENTS.md § Trunk-based SDLC here; § Git Workflow step 6 in dank/serverAppConfigs, riding their PR #6 branch).

2026-10-18 — Trunk law: extension route closed unmerged; direct repo alignment instead

Built and live-verified a cwd-scoped before_agent_start extension (quote probes: fires under ~/Projects, silent elsewhere; kept as .AGENTTRASH/trunk-law.js). Operator closed PR #9 unmerged in favor of editing each repo's own AGENTS.md. Durable finding from the exploration: every context-file provider (AGENTS.md walk-up, .omp/ dirs, CLAUDE.md) resolves at or below the repo root — a ~/Projects/AGENTS.md for ancestor scope can never load; user-level APPEND_SYSTEM.md broadcasts to all sessions. Second finding: serverAppConfigs never had git-flow — its AGENTS.md mandated no branch regime at all (direct commit+push); a trunk + PR law is being added there by PR in that repo, not corrected.

2026-10-18 — SDLC switched to trunk-based development

develop retired by operator decision (single-dev repo; the branch bought nothing and cost a back-merge every release). New law in AGENTS.md § Trunk-based SDLC: branch off the freshest origin/main before every work item, PRs target main directly, --no-ff merges, tags vX.Y.Z straight on main. No back-merges, no release branches. The last back-merge (527596a) lands with this switch; the managed skill gitflow-back-merge-plumbing is now historical.

2026-10-18 — Sibling-vault overlap census; system docs realigned to live ground truth

Census of omp-docs/ against ~/Projects/serverAppConfig/serverAppConfigs-docs/ (subagent + live-ssh verification) found four contradicting pairs; all OURS-side stale pages corrected, THEIRS side flagged only (their repo, their PR flow). Ground truth from read-only probes on both boxes 2026-10-18.

• Hardware drift corrected (system/dankserver.md, system/dankclaw.md rewritten, AGENTS.md machine table): RTX 3090 moved Dankserver→Dankclaw 2026-09; dankserver now 2× Radeon AI PRO R9700 (7700X stays — the "board swap" story is false), 246 GiB RAM, no swap; dankclaw 121 GiB RAM + 64 GiB swap. Served models per live units: frank-archon (:7770 dankserver, radiance TP=2 iq4r), frank-dragoon (:7770 dankclaw, halogen NATIVE v2, iGPU), frank-zealot (:7771 dankclaw, llama.cpp PrismML, Ternary-Bonsai-2-27B PQ2_0, 3090). Both kernels 7.0.0-38-generic; dankclaw :7770 needs no auth.
• Gitea pages corrected (tools/gitea/ index + 3 subpages): gitea-caddy quadlet retired — fleet Caddy fronts the domain; data tree is /home/dank/Containers/gitea/{data,postgres,env} (2026-09-01 standard), /opt/gitea/ is a do-not-use leftover; LAN IP, live domain, no WAN address.
• Pi-era pages demarcated as archive (pi/index.md, core/index.md banners; core/index.md said settings.md was "Current"): frank-default-small/large endpoints, role tables, and 3090/4070 hardware rows now read as pi-era records. tools/llama-cpp/index.md fleet note modernized to the one remaining llama.cpp endpoint (zealot :7771).
• Their-side staleness flagged, NOT edited (for operator clarification): Agent/OMP Harness.md still says thresholdPercent 70 — live config is 95 (agent/config.yml:84); index.md:18 says Dankserver Ubuntu 24.04 vs their own specs page 26.04; Dankclaw/Stack.md "current mapping (2026-09-03)" predates the ACTIVE dragoon/zealot runbooks it links beside; index.md:548 links the superseded pi-era KB at ~/.pi/agent/docs/.
• Consistent duplicates (no action): niri/noctalia desktop pages, Caddy reference. Divergence clarified: their "composer" = OpenWebUI chat box; ours = omp composer shape — unrelated subjects.

2026-10-18 — Status line docs truth audit (full-source re-verification)

Exhaustive claim-by-claim re-audit of OMP Status Line Customization against the minified dist/cli.js bundle plus a live screenshot matrix (band × {off, percentage, embedded}, claude × off). Page carries an Audit log section; four first-pass claims were false and are corrected (see there): claude's rule is not the context meter (meter is box/band only; the "red rule" was the session-accent filler), the "Tern-only" composer-bar claim was bogus (the readout is composer.tokenRate, default off), segmentOptions are not gated to preset: custom (only the segment arrays are), and the mode segment's states are Plan/Prewalk/Goal/Vibe/Loop (no yolo). Page expanded with: the 7-preset defaults table, per-shape statusAttachment/bottomBar model (8 built-ins + plugin shape registry), contextLine per-value behavior, the 27-segment catalog with triggers/options, segmentOptions schema, and the full statusLine.*/composer.* knob registry. The bundle's own "(box composer only)" knob description for contextLine is upstream-wrong — band honors it.

ServerAppConfigs-docs overlap census: flagged separately for the operator (morning report); no doc merged/changed on either side.

2026-10-18 — Status line rebuilt on custom segments; frank-archon display name slimmed

Interactive tune-up of the omp status bar (grilling session, live loop: config edit → watcher repaint → niri screenshot verify). New knowledge page: OMP Status Line Customization.

• Mechanics discovered: leftSegments/rightSegments are inert unless preset: custom (fleet was on nerd with dead arrays; segmentOptions do merge under named presets); composer.shape routes them — band merges both arrays into one row above the editor (title rides the working row), claude splits them into top chip + bottom strip. The context meter exists only in box/band — contextLine: off plain rule, embedded hoists the context segment into the labeled inline meter (60% ——— × 200K), percentage/annotated paint an unlabeled proportional rule meter (all four live-verified); every other shape's rule is a constant session-accent filler. Working row (session title + tok/s readout gated by composer.tokenRate, default off) is composer chrome — model/thinking can never join it. Eight composer shapes ship in the bundle; plugins can register more.
• Final bar (band, after a claude sojourn): left = session_name · vim · mode · hostname · path · git(no branch text) · pr; right cluster in array order = model · cost · time_spent (context_pct hoists out of the run to paint the labeled inline meter 60% ——— × 200K as a flexible spacer between the groups). Cut: tok/s duplicate, clock, cache trio, context_total, subagents, session hash, and token_in/token_out (cumulative totals live in /session). path: logical project/worktree label (stripWorkPrefix: true, kept after a live false trial of the full cwd); branch text dropped (worktree branches are autogenerated), dirty badges kept.
• models.yml: frank-archon display name slimmed to "frank-archon" (endpoint suffix dropped; id untouched).
• Out of scope noted: worktree name generation (wt-<date>-<hash>) is omp-internal; possible upstream feature request.

2026-10-09 — Blankpad DE redesign: desk profiles, keybind rework, Handy fleet wiring

Branch blankpad-de-redesign (worktree wt/de-redesign-20261009). Full sweep + redesign of the niri/noctalia/Handy stack on blankpad.

• Desk profiles. cfg/display.kdl retired to a symlink; per-desk profiles desk-1-home.kdl / desk-2-portable.kdl declare monitors by EDID long name (// requires: lines) — connector names reshuffle between docks, identity travels. scripts/desk.sh = list/current/switch/cycle/match/capture; apply = relink → niri validate (rollback on failure) → load-config-file → noctalia notification. scripts/desk-autoswitch.sh + blankpad-desk-autoswitch.service watch niri msg --json event-stream: niri 26.04 has no hotplug event (OutputConfigChanged is only an ACK that an IPC/config output change was Applied, per niri-ipc); the daemon diffs the sorted output set carried by WorkspacesChanged and hot-applies the matching profile (debounced 1.5 s). Proven live with an eDP-1 off/on cycle: [DP-2 eDP-1] → [DP-2] → [DP-2 eDP-1], apply + reload each time. PM1601 EDID still uncaptured (panel disconnected) — connector-keyed placeholder, desk.sh capture pending.
• Keybinds. swap-window-left/right on Mod+CTRL+H/L; move-column demoted to Mod+CTRL+Alt+H/L; 7 column-width presets 12.5–87.5 %; Print trio (full / gui / screen) via noctalia screenshot IPC; launcher Mod+D; Alt+Tab → noctalia window-switcher; Mod+V → clipboard panel (id verified live); Mod+Shift+D → desk cycle; Mod+Shift+Space → handy --toggle-post-process; lid-close → noctalia msg session lock-and-suspend (pairs with logind HandleLidSwitch=ignore). Cheat sheet now GENERATED: scripts/gen-keybinds.sh → omp-docs/tools/niri/keybindings.md (119 bindings).
• Noctalia. config.toml + settings.toml (lives in ~/.local/state/, not ~/.config/) tracked; taskbar widget added to bar center, applied live via noctalia msg config-reload. Docs corrected to the v5 noctalia msg API.
• Handy. Self-autostart killed (~/.config/autostart/Handy.desktop → .AGENTTRASH; autostart_enabled=false); niri owns autostart (spawn-at-startup "handy" "--start-hidden"). Fleet post-processing wired into the live store: provider custom → http://192.168.0.143:7771/v1, model frank-secondary, prompt default_improve_transcriptions. Open breaker: frank-secondary answers in reasoning_content and leaves content empty; per-request cure chat_template_kwargs.enable_thinking=false verified, but Handy cannot send it and the server is shared with fleet compaction — dedicated no-think port proposed. 0.9.6→0.9.8 upgrade is operator-owned (yay -Syu handy-bin).
• Scripts + tests. Two subagents died mid-run (one mid-test-write; one crashed looping sudo yay without a TTY — see next entry). Their scripts/tests were rebuilt from live-format ground truth: tests/desk.bats (16), tests/gen-keybinds.bats (6), tests/niri-config.bats (5, incl. real niri validate on the assembled base+zz tree with each desk profile active). Suite: 27/27 green.

2026-10-09 — Incident: agent sudo-loop tripped pam_faillock and locked out the operator's sudo

While attempting the Handy 0.9.8 install, a subagent ran sudo yay -Syu handy-bin in a retry loop for ~15 minutes. Harness sudo has no TTY, so every attempt logged 3 pam_unix auth failures; pam_faillock (deny=3) locked user blank out — the operator's own correct-password sudo/pacman -Syu began failing with "temporarily locked out", which looked exactly like a changed password.

• No password change: /etc/shadow mtime 2026-09-10 23:31, untouched; no passwd/chpasswd/usermod in the journal.
• Recovery: loop process died with the agent; faillock --user blank --reset (unprivileged for own user, dir /run/faillock/blank is 0700) cleared the counter; verified empty.
• New law: agents NEVER run sudo/yay/pacman/pkexec on Frank's workstations — root-requiring steps are handed to the operator as exact copy-paste commands. Codified as managed skill faillock-sudo-lockout-recovery.

2026-10-09 — Full secret audit of the command-center repo: clean

Five-layer audit of dank/command-center at 79e780f, driven by the four live values in ~/.config/secrets/ (gitea, hf_token, hf_token_write, kagi_api_key) plus 21 provider shapes (sk-, hf_, gh[pousr], glpat-, AKIA/ASIA, xox, npm, pypi-, AIza, JWT, PEM/OpenSSH keys, scheme://user:pass@host basic-auth URLs, password|secret|api_key = <literal> assignments). Findings printed redacted (prefix + length + sha256 prefix).

• Live-value scan — clean. 77 revs (5 branches, 2 tags), 765 reachable blobs, 3.4 MiB of blob text: no known live value appears in any tracked blob at any ref.
• Whole object store, reachable and dangling — clean. git cat-file --batch-all-objects finds 767 blobs; 2 are unreachable, never committed on any ref (git log --all --find-object empty). One is a 123 KB superseded memory page; the other is a 35-byte test fixture holding a synthetic sk- key whose body is the alphabet followed by 1234 — pattern-minted, not issued. No PEM key, no JWT, no basic-auth URL anywhere. (Quoting the literal here tripped the secret gate, which is the gate working as designed.)
• Shape hits, all triaged benign: api_key="sk-no-key-required" (llama.cpp doc placeholder), Set password: / authenticate(user, password) prose in the Authentik memory skills, and a 40-hex string in the home-config census that is a git packfile filename (pack-ff7cc541….pack), not a token.
• Entropy layer — clean. Zero tokens ≥ 32 chars with Shannon entropy ≥ 4.3 in HEAD blobs.
• Payloads, not just pointers: all 23 git-lfs wallpaper objects (81 MiB) scanned as bytes — no secret shape in any payload.
• Quarantine list holds: no tracked file matches auth.json/trust.json/.env/*.pem/*.key/identity*; the 42 ignored paths on disk carry no secret shape; no credential in any git config scope (local/global/system).
• Fixed on the spot: ~/.config/secrets/hf_token_write was mode 0644 (world-readable) → 0600. ~/.cache/huggingface/token (the standard HF location, holds the same value) verified 0600.
• Known gap, not yet closed: .githooks/pre-commit covers GitHub/GitLab/Slack/AWS/PEM/JWT/OpenAI-shaped sk-, but not hf_…, npm_…, pypi-…, AIza…, Stripe, Telegram, or a scheme://user:token@host basic-auth URL — which is exactly the shape that sat in remote.origin.url until today. Widening the gate is an open decision.

2026-10-09 — Gitea token moved out of every remote URL

Both workstation clones (~/.omp → dank/command-center, ~/Projects/serverAppConfig → dank/serverAppConfigs) carried the Gitea HTTP token inside remote.origin.url, so it sat in .git/config, was printed by git remote -v, and was echoed verbatim by the git-lfs lock warning on every push.

• Cutover: remotes are now bare (http://192.168.0.143:3000/dank/<repo>.git); the credential lives in ~/.config/secrets/gitea (0600) behind a global helper — git config --global credential.helper 'store --file ~/.config/secrets/gitea' — so every repo on the host authenticates from one file. lfs.…info/lfs.locksverify true set on ~/.omp to silence the warning that leaked the URL.
• The trap (git 2.56.0, measured): git credential-store will not parse the protocol=/host=/password= block shape that git credential fill accepts on stdin and that the man page shows for ~/.git-credentials. It returns an empty get, and git falls through to askpass: could not read Username for …. The store file must be one URL per line, port included: http://dank:<token>@192.168.0.143:3000.
• Verified with prompts disabled (GIT_TERMINAL_PROMPT=0): git credential-store get returns the credential, ls-remote reads, push --dry-run says Everything up-to-date on both repos, git remote get-url origin contains no @, and a grep for the token across ~/.gitconfig, both .git/config files, and the shell rc files returns nothing.
• Doc: new client-access (tools/gitea/client-access.md) — credential file layout, the store-format trap, the verification loop, LFS lock warning, rotation procedure. Cross-linked from the gitea index and management pages.

2026-10-09 — Working tree reconciled: callsign renames, Dankclaw compaction, gitignore quarantine

A 200-entry dirty tree (116 deleted, 73 untracked, 11 modified) sat uncommitted across several sessions. Committed as seven atomic units; nothing discarded.

• Model ids renamed to the unit-name scheme (agent/models.yml): frank-medium → frank-archon (dankserver :7770, 200,000 ctx / 16,384 max), frank-large → frank-dragoon (Dankclaw :7770, 217,088 / 45,056, input: [text, image]), frank-small → frank-zealot (Dankclaw-3090 :7771, 200,704 / 16,384). The services keep the old names on the wire — /v1/models still serves frank-primary, frank-large, frank-secondary; the engines accept an arbitrary request model id, so the rename is harness-side only. Probed live: all three chat completions return 200 with reasoning_content.
• Server-side compaction moved off 192.168.0.235:7770 to 192.168.0.143:7771 (frank-zealot's 3090), threshold 70 % → 85 %, remote streaming v2 + async on; every model's compactionModel points at frank-zealot so the wire id stays valid at the compaction endpoint. Reference integrity checked programmatically: 22 model references across modelRoles, task.agentModelOverrides, compactionModel — zero unresolved.
• .gitignore quarantine for runtime state that was tracked-adjacent noise: autoresearch/, agent/predict/, local/, wt/, and agent/memories/--home-blank-.omp-wt-* (a session inside a git worktree mints a memory bucket keyed on the ephemeral worktree path). All had zero tracked files, so the rules are purely additive — 73 untracked → 68, and the remaining 68 are real memory records that got committed.
• Skill libraries were renamed, not lost. agent/managed-skills consolidated 94 → 25 (git renders directory renames as mass delete + mass add; pre-cutover content is intact at 12c5e88). Workstation memory skills re-bucketed by topic: dotfile-git-workflow → ghostty-text-aa + niri-display-rotation + noctalia-theme, handy-service-mgmt → handy-flm-inference, inference-runtime → llama-cpp-parallelism.
• Display layout: blankpad DP-2 back to landscape 3440x1440@239.999 (transform "normal"), HDMI-A-1 portrait bezel flipped 270 → 90, eDP-1 re-pinned rightmost. Supersedes the 2026-09-15 portrait pin (2baf0c0).
• Secrets: .zshrc now reads KAGI_API_KEY / HF_TOKEN from ~/.config/secrets/* (600-perm files) instead of carrying literals into a repo published to Gitea. The pre-commit secret gate passed on all seven commits.

2026-09-28 — Retired workstation purged from the knowledge base

Frank sold the client workstation. Every identifier for it is gone from this bundle: hostname, both of its LAN addresses, and its Wi-Fi SSID. The knowledge outlives the box; the machine does not.

• Distilled, not deleted. The four per-host system pages carried two pieces of knowledge worth keeping and one that was not:
  • /system/network.md (new) — LAN topology, plus the multi-link resolver law: a client with two +DefaultRoute links flaps its DNS, and the fix is a static resolver pinned on each connection (a resolved.conf DNS= line is only a per-link fallback). Keeps the flush-caches verification loop, because one successful dig proves nothing on a flapping resolver.
  • /tools/niri/displays.md (new) — the multi-monitor rig: portrait transforms, absolute positions, the auto-placed third panel, the iGPU render-node pin, and the three ways it breaks (duplicate output blocks, positions that do not reflow, a mode the panel cannot drive).
  • Dropped: the machine's own spec sheet and quick-facts table. The OS and userspace facts they repeated already live in tools/cachyos.md.
• Mechanical scrub across the rest of the bundle: indexes, tool and research pages, the inventory census, the changelog, one generated HTML report, and four managed skills. Facts kept, names dropped — install.sh's --host case list and the packages/<hostname>.pkgs filename stay, because the hostname is the lookup key there.
• AGENTS.md and README.md keep a nameless "workstation" row so the fleet tables still describe the class of box without advertising a machine that is no longer Frank's.
• Out of scope by decision: agent/memories/** (past-session records the learn pipeline rewrites) and the retired pages' history — they are recoverable from .AGENTTRASH/20260928-retired-workstation/ and from git.

2026-09-01 — Retry backoff and thinking default

• Retry backoff: configured Pi's automatic agent-level retries in settings.json with a 10-second base delay, producing 10s, 20s, and 40s delays across three retries.
• Thinking default: set settings.json defaultThinkingLevel to medium. models.json has no supported per-model default-thinking field; Pi applies this global default to the configured models.

2026-08-28 — KB rebuilt to OKF v0.1; live-verified fleet facts

• Creation: the whole omp-docs/ bundle was copied from the legacy ~/.pi/agent/docs tree and rebuilt as a git-tracked, OKF v0.1-conformant knowledge base inside the new command-center repo (repo root = live agent dir; the checkout is the deployment). Every checkable claim was re-verified against the live fleet (read-only probes); unverifiable claims are tagged unverified in frontmatter.
• Update — hardware facts corrected: Dankserver is a Ryzen 7 7700X + RTX 3090 (not Strix Halo); Strix Halo (gfx1151) is Dankclaw. RAM, storage, kernel, and network facts were probed live and corrected across system/.
• Update — conformance pass: concept-in-disguise indexes promoted to <dirname>.md concept files (identity = path, per spec); index.md files stripped of frontmatter (indexes carry none); all wikilinks and relative links resolved or repointed; scripts/kb-lint.sh (OKF conformance + link-integrity linter) exits 0 on the bundle. Vendored references/okf-spec.md is exempt from link-lint — its example paths are illustrations, not links.
• Update — inventory: home-config census (7 orphaned configs, 10 prune-candidates) recorded under inventory/ for the pruning session; zero deletions performed. Legacy ~/.pi tree left untouched and frozen.
• Deferred to operator (decisions, not facts): chezmoi source layout (research/chezmoi-evaluation.md recommends a separate chezmoi repo) and the permanent secrets tool (research/secrets-for-agents.md; interim = pre-commit secret gate + quarantine list; top candidate sops+age). Gitea Projects unit is disabled instance-wide, so the kanban board is a one-time web-UI setup for the operator.

2026-08-23 — Homepage deployed to production; Caddy KB doc

• Homepage is live on Dankclaw. ./scripts/deploy.sh dankclaw deploys the static files to /srv/homepage, installs the status checker (systemd timer, every 3 min, writes status.json), writes deployed.js (the last master deployment datetime), renders the Caddyfile (both basicauth hashes from the host .env), reloads Caddy, and checks health. Verified end-to-end: /admin/ (operator) + all shared files serve 200 under the operator credential; / and /admin/ 401 without/wrong credentials; the LLM is healthy.
• Caddyfile auth design (the hard-won fix). Caddy sorts top-level directives, so a rewrite runs before a basicauth; the rewrite changed /admin/ to /admin.html and the path-based auth matchers then saw the rewritten path, applying the wrong (family) auth block — the operator password 401'd. Fix: serve each page from its own handle block (which preserves file order, so the auth is checked on the original path): handle /admin* (operator only), handle @data (shared data + assets, family OR operator, so the admin page's fetches reuse the operator credential), and a catch-all handle (family). Also: a handle inline matcher is a single token (multi-path needs a named matcher); a leading @name <directive> is parsed as a matcher definition (re-defines @name).
• Caddy mount fix. The caddy container mounted the repo's shared/homepage (static source) to /srv/homepage; the generated files (status.json, deployed.js) are written to the host's /srv/homepage. Pointed the mount at the host's /srv/homepage so the generated files serve. A compose change needs a container recreate (not a reload) — podman compose up -d caddy (the deploy skips the compose restart, so this is manual).
• Caddy KB doc (operator request: read the docs before guessing at syntax): new caddy/ section in this bundle — caddyfile (caddy/caddyfile.md) + caddy/index.md, cross-linked from the root index.md. Grounded in caddyserver.com (fetched 2026-08-23) + a researcher subagent brief.
• Tests: 116 green (was 112; +4 for the Caddyfile handle design + the status.json 0644 perms). The local caddy validate test is skipped (no local caddy binary); the rendered config is validated on the target via the porkbun image before each deploy.

2026-08-23 — Homepage design settled (grilling complete)

• All 38 open questions (Q6–Q43) answered. The homepage design is now fully settled. Full record: serverAppConfigs/docs/Infrastructure/Homepage Design Grilling.md (final decisions + the deltas) and docs/Infrastructure/Homepage.md.
• Key deltas from the initial recommendations: the probe method is per service (tcp or http); the probe timeout is 5 s; the layout is TUI-style panels (a grid), not a flat list; styling is toggleable (full theme default, a toggle to bare unstyled HTML); do not alter the user's cursor; no client-side auto-refresh (read status.json once on load); the footer shows the last master deployment datetime (not last-check); the deploy is git-driven (master = deployed state); the build is full TDD.
• Per-service docs workstream: each service gets a /docs/<svc>/ page (not /usage/), written as a how-to for a regular user (no maintenance details), password-protected by the same auth as its homepage section, built as each service goes active. The existing reference docs in docs/ are NOT linked from the homepage (they may contain secrets).
• Build complete (all 6 slices, full TDD, 112 tests green): 1 the checker core (tcp/wip/atomic write/generated_at), 2 the http probe, 3 the real services.json + schema, 4 the pages + theme + toggle, 5 the Caddyfile auth split (root vs /admin/), 6 the git-driven deploy. All committed + pushed. The render logic is verified with a Node DOM shim (9 user panels, correct summary, correct footer datetime).
• To go live: the operator adds CADDY_ADMIN_HASH (the operator-only password, caddy hash-password) to the Dankclaw .env, then runs ./scripts/deploy.sh dankclaw. Until then the admin hash stays a placeholder. The per-service /docs/ pages, the *arr stack, and the optional /how/ page remain separate threads.
• Correction (media stack): the Dankserver media stack (RomM, Copyparty, Jellyfin, Plex, the *arr) is UP — it runs as bare processes/services, not yet migrated to podman containers. The 2026-08-22 "down / missing .env" finding was wrong: I was probing for the compose stack, which is not how it runs yet. Do not treat it as an outage and do not probe it as down. Its podman migration is a separate, future effort.

2026-08-22 — vllmScripts merged into serverAppConfigs

• Repo merge: the vllmScripts repo (LLM inference fleet: serve scripts, systemd units, env tunables, deploy runbook, research docs) is merged into serverAppConfigs with its full git history (two-parent import commit, then reorganization: docs/ → docs/LLM/, scripts/ merged, old/ for retired PP/NPU-era scripts). One repo, one checkout per host (/home/dank/serverAppConfigs), one scripts/deploy.sh covering both workloads per target (podman stack + LLM service relink/restart + both health checks).
• Units: frank-default-{small,large,npu}.service now point at /home/dank/serverAppConfigs/scripts/.... The retired target checkouts (/home/dank/vLLM on Dankserver, /home/dank/vllmScripts on Dankclaw) are to be archived on the hosts during the production deploy.
• Secrets: the live sudo password that the old vllmScripts AGENTS.md documented in plaintext is NOT carried into the merged repo (a bats leak-check guards it). It remains in the git history of both repos on the LAN Gitea — rotation recommended.
• KB links: system/index.md, system/dankserver.md, system/dankclaw.md, and the root index.md updated — the "sibling knowledge bases" section is now a single sibling (serverAppConfigs, which contains the LLM fleet KB at docs/LLM/index.md).
• Port move: frank-default-large (dankclaw) moved from :52625 to :7770 (operator request: the primary inference server sits on the standard llama.cpp port). Both LLM services now listen on :7770, distinguished by host. models.json (llamacpp (dankclaw)) and the OpenWebUI quadlet (OPENAI_API_BASE_URL) re-pointed to :7770. pi/models.md updated.
• Homepage design (grilling): the serverAppConfigs homepage (docs/Infrastructure/Homepage.md) is WIP — the Caddy site is live (basicauth 401, :80 302) but /srv/homepage/ is not deployed. Settled design: a single site on Dankclaw (root = user services, /admin/ = admin services behind a second operator-only password, optional /how/ tourist page). Status is checked server-side — a systemd-timer checker (every 3 min) probes each backend host:port and writes a same-origin status.json; the page reads only that file (the client-side-ping design is blocked by cross-origin + mixed-content + auth). Full decision log + 38 open questions (Q6–Q43) in docs/Infrastructure/Homepage Design Grilling.md. A terminal-TUI-styled HTML progress report was generated for the operator to review and answer the open questions. Full-stack baseline verified: Dankclaw fully up; the Dankserver media stack is down (only Calibre-Web + the LLM are up; no .env on the target).

2026-08-21 — Fleet ops: passwordless sudo, Hermes SSH keys, sibling KB links

• Passwordless sudo: added /etc/sudoers.d/dank-nopasswd (dank ALL=(ALL) NOPASSWD: ALL) on both Dankserver and Dankclaw. Verified sudo -n true on both. The pre-existing drop-ins (frank-small on Dankserver; frank-large + frank-ling on Dankclaw) grant per-command NOPASSWD for the frank restart scripts and remain in place.
• Hermes SSH keys: the Hermes Agent (podman container hermes, project dankclaw in serverAppConfigs, gateway drops to container uid 10000 = host uid 109999, HOME /opt/data on a podman volume) now has a dedicated ed25519 key at /opt/data/.ssh/id_ed25519 (hermes-agent@dankclaw) plus an SSH config for dankclaw (192.168.0.143), dankserver (192.168.0.235), and the workstation. The public key is in dank's authorized_keys on all three machines. Verified from inside the container as the hermes user: all three ssh <host> hostname calls succeed.
• Sibling KB links: system/index.md, system/dankserver.md, system/dankclaw.md, and the root index.md now link the two sibling knowledge bases on the workstation: ~/Projects/vllmScripts/ (inference fleet scripts + runbook, index docs/index.md) and ~/Projects/serverAppConfigs/ (Podman deploy configs for both servers, index docs/index.md).

2026-08-19 — pi-web-access: SearXNG endpoint re-pointed to dankclaw

• Config: ~/.pi/web-search.json — searxngBaseUrl moved from the dead 192.168.0.235:8081 (dankserver, connection refused) to 192.168.0.143:8081 (dankclaw; verified live: HTTP 200, 25 JSON results, 0.8 s). Deleted the redundant provider: "all" (the searchProvider: "searxng" pin already wins in v0.22). Added maxInlineContentChars: 50000. Kept workflow: "none" and ssrf.allowRanges: ["192.168.0.0/24"].
• Docs: pi/packages/pi-web-access.md bumped 0.14.0 → 0.22.0 (18-provider chain, config block, active-setup note, GitHub-clone section, yt-dlp present). agentic-web-research/ SearXNG endpoint references repointed .235 → .143 (index, searxng-tuning, vpn-routing).
• Note: pi-web-access caches web-search.json in memory per process — the new endpoint takes effect on the next Pi session, not the running one.
• Packages: yt-dlp installed on the workstation (2026.07.04) — YouTube frame extraction now works; ffmpeg (n9.0.1) was already present.

2026-08-19 — SearXNG + Gluetun bring-up on dankclaw

Field notes from standing up the search stack (gluetun + searxng + valkey) and confirming agentic JSON search. All four are operational; the SearXNG trio runs via podman-compose (not yet boot-persistent quadlet).

• Gluetun manual WireGuard: a specific-server ProtonVPN config (private key + peer + endpoint) uses VPN_SERVICE_PROVIDER=custom, NOT protonvpn. The protonvpn provider is account/token-based and rejects a manually-set endpoint with endpoint port is set. Current Gluetun uses the single-peer model: WIREGUARD_PUBLIC_KEY, WIREGUARD_ENDPOINT_IP, WIREGUARD_ENDPOINT_PORT, WIREGUARD_ADDRESSES (not the old WIREGUARD_PEERS_* names). DNS is Gluetun's own DNS_* vars, not the WG DNS line.
• Gluetun healthcheck: the image has no curl, and its :8000 status server 404s on every path. Use test: ["CMD","sh","-c","test -e /sys/class/net/tun0"] — tun0 exists only while the tunnel is connected.
• SearXNG trusted group: lives in limiter.toml (not settings.yml). A direct hit on the published port has no X-Forwarded-For header, so the bot-detection cannot resolve a client and returns 429 regardless of pass_ip. SearXNG must be reached through Caddy (which sets XFF); trusted_proxies then needs the podman bridge subnet (dynamic in rootless — pin the network CIDR) so Caddy's source is trusted for XFF extraction.
• Rootless podman gotchas: short image names do not resolve (no unqualified-search registries) — qualify with docker.io/. Binding privileged ports (Caddy 80/443) needs the one-time sysctl net.ipv4.ip_unprivileged_port_start=80.
• Repo: serverAppConfigs — added shared/searxng/limiter.toml + test seam, wired it into deploy.sh, fixed the Gluetun compose (peer env vars, qualified images, tun0 healthcheck), documented the rootless-port + Caddy/XFF gotchas, and skipped the ComfyUI GPU tests (migration pending).
• Boot-persistence (quadlet): converted the trio from podman-compose to quadlet so it survives reboot. searxng sets Network=gluetun.container (reuses gluetun's netns → exits the tunnel); gluetun + valkey join a pinned searxng.network (10.89.2.0/24). Deployed via scripts/deploy-searxng-quadlet.sh. Field gotchas (also in engineering/infra/containers.md): the generator rejects CapAdd=/Device= (use AddCapability=/AddDevice=) and silently skips the unit; generated units cannot be systemctl enabled — [Install] WantedBy=default.target is the enablement (is-enabled = "generated"), and linger makes them boot-start. Verified: all three is-enabled = generated, tunnel up (US-WA), agentic JSON search returns 27 on-topic results.
• open-webui moved to dankclaw (quadlet): the old open-webui was a bare container on Dankserver (:3000) pointing at a dead :9090 backend. It now runs on Dankclaw as a boot-persistent quadlet (scripts/deploy-open-webui-quadlet.sh): CPU image :main (Dankclaw is CPU-only), isolated on Network=slirp4netns, published :3005 (:3000 is Gitea's HTTP port), data at /home/dank/Containers/open-webui/data. Default LLM backend is the local vLLM (host.containers.internal:52625). The Dankserver searxng + open-webui bare containers were retired (data dirs preserved). podman inspect <name> can match an IMAGE, not just a container — a removed container still "exists" to inspect if its image remains; use podman ps -a --filter name= to check containers.
• Model/port mapping corrected (live, 2026-08-19): the fleet endpoints are the REVERSE of the 2026-08-17 note. Live truth (probed /v1/models): frank-default-large is on Dankclaw :52625 (vLLM, completion) and frank-default-small is on Dankserver :7770 (llama.cpp FastMTP, completion+multimodal). The serverAppConfigs docs (Stack, OpenWebUI) were corrected; the sibling vllmScripts KB still carries the old mapping and should be updated there.
• SearXNG 429 root cause (the "diva"): the limiter's JSON cap is API_MAX = 4 requests/IP/hour and that constant is hardcoded in Python (searx/botdetection/ip_limit.py), NOT settable in limiter.toml. The only bypass is pass_ip (SearXNG's filter_request returns before ip_limit runs when the IP is in pass_ip). Second gotcha: rootless podman strips the client IP on published-port hits — a request to the published port reaches the container with a slirp/bridge source (10.88.x/10.89.x), never the real client; only Caddy's X-Forwarded-For restores it. So even a LAN-IP curl 429'd. Fix: expand pass_ip to loopback (127.0.0.1/32, ::1) + the host's private podman/slirp ranges (10.88.0.0/16, 10.89.0.0/16) + the LAN. Verified: 20 concurrent JSON searches → all 200 (real results), while a spoofed public IP (8.8.8.8) still 429s (limiter still armed). The config file is root-owned (uid of the rootless podman namespace) — update with a one-off sudo cp + systemctl --user restart searxng.service.
• ComfyUI GPU (gfx1151) — DONE (Option 1/B): the deferred GPU ComfyUI is now built and running. It was previously a bare CPU venv (/home/dank/ComfyUI, py3.14, torch 2.13.0+cu130 with no NVIDIA → CPU fallback). Now it's a container comfyui-rocm:gfx1151 built from dankclaw/Containerfile.comfyui on AMD's rocm/vllm:rocm7.12.0_gfx1151_..._py3.12_pytorch_2.9.1 base (the only torch build where torch.cuda.is_available()=True on the Strix Halo APU). Key gotchas: (1) the base image already ships the ROCm torch/torchvision/torchaudio, so the Containerfile must skip the torch family from ComfyUI's requirements.txt (an unpinned torch would clobber the gfx1151 kernels) — filter with grep -viE '^(torch|torchvision|torchaudio)([^a-z]|$)'. (2) rootless podman cannot --group-add render (gid 990 is outside the subgid range 100000+); GPU access comes from the /dev/kfd+/dev/dri passthrough checked host-side (the run user is already in render), so add only the video group. Verified: /system_stats → cuda:0 Radeon 8060S, logs show AMD arch: gfx1151 + active HIP backend. Boot-persistent via a user systemd unit running podman compose up -d comfyui (old venv unit saved as comfyui.service.venv.bak). The 4 comfyui_rocm.bats tests are un-skipped (the group test now asserts video present + render ABSENT).

2026-08-17 — Fleet redo + vLLM tool calling

• Update: pi/models.md — providers rewritten for the fleet redo: llamacpp (dankserver) at :7770 and vllm (dankclaw) at :52625; model selection strategy and live defaults brought in line with the new topology and the Quark W8A8 model.
• Context: the Frank fleet moved off the pipeline-parallel + NPU topology. frank-default-large (llama.cpp FastMTP, IQ4_XS) is on Dankserver; frank-default-small (vLLM ROCm, nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8) is on Dankclaw with ngram spec and tool calling. The Avesed INT8 quant's tool-call XML fidelity was poor; the Quark quant follows the Qwen3 XML format cleanly.

2026-08-16

Gitea Boot-Persistence Fix (dankclaw)

• Incident: unattended-upgrades rebooted dankclaw at 15:01. The legacy podman-compose Gitea stack had no systemd units, so rootless podman did not auto-start the containers. Gitea was down ~1 hour.
• Update: tools/gitea/index.md, architecture.md, management.md, data-persistence.md — rewritten for the quadlet-managed stack: container/unit names, the systemd-gitea network (DNS-enabled), boot persistence (linger + WantedBy=default.target), the app.ini env-override gotcha, UID 100999 ownership, incident log.
• Update: engineering/infra/containers.md — marked podman generate systemd deprecated (podman 5.7), documented Quadlet with a field-tested gotcha list (default network DNS off, [Service] EnvironmentFile= for ${VAR} expansion, unit naming, bind-mount UID mapping).

2026-08-13

Pi Subagents Limits

• Added a "Parallel Execution and Limits" section to pi/packages/pi-subagents.md.
• Documented the effective values: globalConcurrencyLimit 2 (from extensions/subagent/config.json), parallel.maxTasks 8, parallel.concurrency 4, maxSubagentSpawnsPerSession unlimited, maxSubagentDepth 2, and turnBudget.maxTurns 30.
• Noted that no per-model limit exists; all caps are global to a run or a session.

2026-08-11 (continued)

Engineering Research — New Area

• Created the engineering/ research area: SDLC and tooling, agent instruction files, language and framework stacks, infrastructure and cloud, pi plugins, skills ecosystem, and Signal remote control.
• Ran a parallel research fleet (scout + 6 researchers) grounded in current 2025/2026 web sources.

Skills Leaderboard — Live Verification

• Queried the skills.sh API (GET /api/skills/all-time/{page}) for the authoritative top 50 skills by all-time installs.
• Replaced the inferred category table in engineering/skills-ecosystem.md with the verified ranked list.
• Noted the API’s zero-based pagination and 200-skill page size for future queries.

2026-08-11 (continued)

Agentic Web Research — Grounded Revision

• Ran a parallel research fleet (7 researchers + 2 scouts) for current 2025-2026 evidence on SearXNG tuning, VPN split-routing, agentic search, open models, research orchestration, and MCP.
• Rewrote all 8 agentic-web-research/ pages to match verified facts and the local topology: SearXNG at 192.168.0.235:8081, llama.cpp inference on 7770 (dankserver/dankclaw) and 8080 (local-ornith), main model openrouter/deepseek-v4-flash-0731.
• SearXNG: corrected to Valkey deployment, real engine defaults (DuckDuckGo/Wikipedia/Google News active, Google inactive, Bing disabled), limiter budgets, current outgoing defaults.
• VPN: added Gluetun as the recommended pattern, corrected OpenVPN container joins, added source_ips/network: ipv4.
• Open models: clarified Llama 4 claims (10M ctx = Scout only, single-H100 needs INT4, "beats GPT-4o" is Meta-only), added Qwen2.5-VL-7B as best self-hostable vision model.
• Web agents: corrected the Browser Use "87.4%" claim (managed Cloud variant, not open-source; peer-reviewed 44.5% for CUA), added Skyvern.
• MCP: updated transports (stdio + Streamable HTTP, legacy SSE), current reference servers, added SearXNG MCP, tightened security rules.

Exhaustive Bundle Audit

• Link audit: resolved all 154+ wikilinks by frontmatter title — 0 broken wikilinks. Fixed relative-link base errors in core/index-doc.md, packages/index.md, pi-web-access.md, llama-cpp/setup.md, npx-skills/skill-commands.md, packages/pi-browser-use/index.md.
• External URL audit: fixed dead links (SearXNG limiter path, earendil-works/pi blob URLs, AYayOS/qs -> outfoxxed/quickshell, badlogic/pi-mono -> earendil-works/pi).
• Factual fixes: core/settings.md stale JSON (defaultProvider/model/thinkingLevel/changelog-version), pi/models.md dankclaw auth, Sleep-Fix-Guide.md kernel 7.1.3 -> 7.1.4.
• Verified (did not change) MemorySleepMode=deep: it is a valid systemd 256+ option.
• ASD-STE100 style pass: removed contractions, banned may, and prose semicolons across research/tools/pi docs.
• Added OKF index frontmatter to core/index.md and skills/index.md.
• Added a "Pi-peon is installed" note to the npx skills ecosystem table and a port-7770 system note to llama.cpp docs.

Agentic Web Research Section

• Created agentic-web-research/ — seven new concept pages plus index covering self-hosted web agents, SearXNG tuning, VPN routing, agentic search practices, open models, research effectiveness, and model context protocol.
• Expanded via parallel research: concrete BM25/Pruning/LLM content filters, credibility heuristics, failure-mode guardrails, MCP server and transport guidance.
• Added the section to index.md under a new ## [Agentic Web Research] heading.
• Linked pi/packages/pi-web-access.md to the new section.

System Instruction File

• Created AGENTS.md — system behavior contract with a dedicated layer on parallel subagent execution (one subagent call with tasks + concurrency).

Bundle Audit

• Verified all 154 wikilinks resolve (zero broken).
• Removed stale empty Alienware M16a2 Sleep/Wake Issues.md duplicate into .AGENTTRASH.
• ran ASD-STE100 style pass: converted should to imperative or must, and e.g. to for example across 10 files.
• Removed stale frank-default-omni model references and corrected dankclaw port 13305 to 8080 per commit 15d4af4.
• Resolved: the live config is authoritative. Updated both llama.cpp servers to port 7770 and set the default provider to openrouter (deepseek) per settings.json. Dankserver/dankclaw now 7770; main session default is openrouter.

2026-08-06 (continued)

Link Fixes

• Fixed 15 broken wikilinks — added missing YAML frontmatter to ghostty/index.md, tmux/index.md, linux-power-management/index.md, gitea/index.md, gitea/architecture.md, gitea/data-persistence.md, gitea/management.md
• Fixed title mismatches: OKF v0.1 Specification → Open Knowledge Format (OKF) v0.1 Specification, Pi Settings → Pi Settings System (settings.json), Alienware M16a2 Sleep/Wake Issues → Alienware M16a2 Linux Sleep/Wake Issues, Architecture Details → Gitea Architecture, Data Persistence → Gitea Data Persistence, Management Guide → Gitea Management
• Zero broken links remaining (47 links verified)

2026-08-06

ComfyUI Documentation

• tools/comfyui/index.md — ComfyUI setup on dankclaw, architecture, service management
• tools/comfyui/troubleshooting.md — Common errors, diagnostics, recovery steps
• tools/index.md — Added ComfyUI entry
• system/dankclaw.md — Added ComfyUI service, hardware specs, services section

Major Documentation Overhaul

Created — System Documentation

• system/index.md — System documentation index with infrastructure diagram
• Per-host system pages (overview, hardware specifications, display configuration, network interfaces) for the workstation — retired later; see /system/network.md and /tools/niri/displays.md
• system/dankserver.md — Primary llama.cpp server documentation
• system/dankclaw.md — Secondary llama.cpp server documentation

Created — Pi Configuration

• pi/index.md — Pi configuration overview with architecture diagram
• pi/settings.md — Main settings.json reference (moved from core/)
• pi/models.md — models.json provider and model definitions
• pi/packages/index.md — Installed packages index
• pi/packages/pi-peon.md — CESP/OpenPeon sound-pack player
• pi/packages/pi-web-access.md — Web search, fetching, video understanding
• pi/packages/pi-subagents.md — Core subagent delegation
• pi/skills/index.md — Installed agent skills inventory

Created — Tools

• tools/cachyos/index.md — CachyOS documentation (kernel, repos, gaming stack)
• tools/noctalia-shell/index.md — Noctalia Shell with Niri integration
• tools/applications/index.md — Default applications reference

Updated

• core/settings.md — Updated with current config, subagent overrides, cross-links
• tools/niri/index.md — Rewritten with actual system configuration
• tools/index.md — Added CachyOS, Noctalia, Applications entries
• index.md — Complete rewrite with new sections and cross-links

Moved / Cleaned

• tmux.md → tools/tmux/index.md
• Orphaned root duplicates → .AGENTTRASH/ (npx-skills.md, okf-maintenance.md, okf-spec.md, vercel-labs-skills.md, vercel-labs-skills-find-skills.md, ornith-llama-cpp-setup.md)

---

2026-07-18

• Enhancement: Expanded okf-maintenance.md — added cross-link management, index management, health metrics, troubleshooting, and quick reference card
• Creation: core/log.md — change log for the core documentation directory
• Creation: .okf/config.yaml — bundle tooling configuration
• Update: core/index.md — added log.md entry and cross-references

2026-01-15

• Initialization: Converted docs bundle to OKF v0.1 format
• Migration: Migrated files to core/, tools/, packages/, references/ directories
• Applied: OKF frontmatter to all files

2025-08-05 — Gitea documentation

Added ](tools/gitea/index.md) documentation to tools. Covers:

• architecture — Container topology, Caddy reverse proxy, network diagram.
• management — Podman commands, upgrades, troubleshooting.
• data-persistence — Volume mounts, backup strategy, restore procedures.

Updated tools index.