command-center/omp-docs/log.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
changelog	Bundle Change Log	Chronological update history for the OKF knowledge bundle.	okf maintenance changelog	2026-10-21

Bundle Change Log

Entry law (since 2026-10-19): one dated entry = what changed + the durable lesson, a few lines. Full incident narratives are recoverable from git (git log --follow omp-docs/log.md); knowledge that outlives an entry lives on its concept page, linked here. Never re-expand an entry in place.

This page is the bundle's only work log: one file, at the root. Never start a second one in a subdirectory or under another name — scripts/kb-lint.sh reports any log.md that is not this one.

2026-10-21 — reviewing a day of work, and finding checks that lie

Two-axis review (Standards / Spec) of everything landed 2026-10-20 → 2026-10-21: 51 commits, 30 findings, worked on fix/60-review-findings (issue #60). Three findings were the same shape — a check that reports confidence it did not earn:

• A guard that cannot see must refuse. pcie-probe.sh dropped the /dev/kfd half of its busy check when fuser was absent, and gpu_busy_percent reads 0 for a resident-but-idle service: the tool measured beside a live endpoint while its own LAW said not to. Missing visibility is now a refusal (exit 3), not a narrower check.
• A swallowed status is a false report. deploy.sh printed ENABLE for units systemd refuses to enable (live-sync.service has no [Install] section; its timer arms it), and the loop walked *.service only, so a rebuilt box deployed the heartbeat and never armed its timer. Every outcome now reports what actually happened.
• A gate that reads a quoted path reads nothing. git quotes non-ASCII paths in --name-only output; the quoted name came back as an empty diff and the secret gate continued past it. Fail closed means -z, and it means the placeholder filter reads the assigned value — a comment saying # example cannot vouch for the token beside it.

The law that cost the most, from the probe: a test that observes the tool's environment instead of what the tool passed on cannot see the bug. The stub probe echoed $HIP_VISIBLE_DEVICES rather than its argument, so the suite asserted the mask was set and per-card bandwidth was unobtainable on the 2×R9700 box. Model the platform rule in the stub and the existing test goes red against the old code — that is the red-proof worth having.

Incident, found while verifying the gate: the live-sync heartbeat had failed three hourly ticks, because a managed skill that documents how to write a secret gate quotes the vendor's all-zeros placeholder id and the gate's shape rule fired on the documentation. Prose about credentials is what the machine tier is made of; a gate that blocks it is worked around, and the workaround is the hole. Placeholder judgement reads the matched token — plus what follows it, because a fixed quantifier truncates EXAMPLE to EXA — never the whole line. ALLOW_SECRETS stays the only override, and the heartbeat's own payload was replayed through the fixed hook to prove it lands.

2026-10-21 — a documented mode the panel never had

The home ultrawide had been running at 60 Hz for weeks while desk-1-home.kdl asked for 3440x1440@239.999. The page that owns that file documented the same phantom mode, so every reader — including tonight's inspection — trusted it. The panel does offer a 240-class mode: niri msg outputs lists 3440x1440@239.998. The defect was one character — niri matches mode strings verbatim, and 239.999 is not 239.998.

Laws, now on tools/niri/displays:

• A mode string must match the advertised string verbatim. niri does not round: 3440x1440@239.999 does not select 3440x1440@239.998. It logs configured mode … could not be found, falling back to preferred and drives the EDID preferred mode — 60 Hz here. A one-character typo therefore looks like a dying monitor, and every connector drop re-enacts it.
• Three layers, three signatures: config fallback, DP link drops, and amdgpu DMCUB errors with EINVAL page flips after s2idle. Only the first yields to a file edit; only the last yields to a reboot. Triage table and probes are on the page.
• A number in a page is a claim, not a measurement. Mode strings, refresh rates and tool availability were wrong on several pages at once — wlr-randr was listed as installed while pacman -Qs wlr matches nothing. Read the hardware before writing a mode into a runbook, and destroy the stale copies. A correction is itself a measurement: the first pass at this entry published a second wrong hardware fact, with a verified: date stamped on it.

Fix: desk-1-home.kdl 239.999 → 180.000 on the identity block and both legacy connector blocks, applied live with niri msg output <panel> mode <mode> — the temporary per-output path, so no config reload and no logged-out session.

2026-10-20 — one work log per bundle

The bundle had two: this page and core/log.md, created "for tracking core directory modifications" — two entries of pure migration bookkeeping (file moved, directory created, log created), and a date for the OKF conversion that neither this page nor git agrees with: 2026-07-15 there, 2026-01-15 here, while git log dates the OKF rebuild 2026-08-28. core/index.md was still pointing readers at it, which is how a stale page stays alive. Trashed to omp-docs/core/.AGENTTRASH/log.md — its content is recoverable from git, which is the entire argument against a second log.

Laws, now written down in references/okf-spec.md and references/okf-maintenance.md, and gated: a bundle has ONE work log, log.md, at its root, and scripts/kb-lint.sh reports any log.md that is not the root's — so the sprawl that made logging useless once cannot restart quietly. The agent-side duty is in agent/APPEND_SYSTEM.md § 1: log here, read here, never start another.

2026-10-20 — two instruments rescued from $HOME script sediment

$HOME had accumulated 123 loose scripts at its root (archived to .AGENTTRASH/home-scripts-2026-10-20, ~1.8 GiB of stray audio consolidated into ~/audiosamples). Two were durable instruments and now live in the repo with black-box tests and pages: tools/remote-job and tools/pcie-probe. Refs #53. Laws learned:

• A readiness pattern in a log is not evidence of completion. ssh host 'cmd | tail -f log | sed -n /ready/q' fails three independent ways — the broker aborts the long connection, the pattern can match a header printed before the work starts, and sed -n /x/q exits the moment it matches, tearing down the job it was watching. The only honest signal is a sentinel the job writes last, read over a short-lived connection.
• Never benchmark beside a live service, and never let a tool decide to stop one. The guard refuses before anything is built; unloading a production endpoint is a human's call. A number taken as noise beside serving traffic is both disruptive and wrong.
• An instrument that segfaults is not an instrument. Two of the three shipped .hip probes called hipGetDeviceProperties, which crashes on this ROCm build while hipGetDeviceCount works fine; device identity now comes from sysfs, and the trap is recorded in the sources so it is not reintroduced.

2026-10-20 — the secret gate was reading the wrong files, and one credential had already escaped

Found while deciding what else belongs in dotfiles/. (1) The gate whitelisted extensions (*.md *.yml *.json *.toml *.sh *.ts *.js), so a .env, .container, .ini or extension-less dotfile was never read — exactly the shapes the dotfiles tier is made of. It now scans the whole staged diff for token shapes and, in non-prose files, for credential assignments, and it fails closed when git diff itself errors. That last clause earned its place: the rewrite's first attempt passed -I to git diff, which is not valid there, got an empty file list, and reported success on every commit — that is what a blind gate looks like, and it looks identical to a clean tree. (2) A live credential had already escaped into tracked content: a Caddy basicauth bcrypt hash, its plaintext password, and the Pi-hole internal password, in an auto-learned memory file. Redacted — but both credentials need rotation, because redaction does not un-leak history that every machine has already pulled. (3) home/dank/.omp/dotfiles/** and .pre-pull-backup/** were tracked fiction invisible to deploy.sh, and 23 wallpaper JPEGs (80 MiB) rode along in every clone. Laws: a gate must fail closed, since a silently-passing gate is worse than none — it buys the belief of safety; auto-learned memory is written by a process that has read your terminal, so it is a leak surface and gets gated like any config; systemctl --user enable snapshots an absolute path, so enabling from a worktree means the seven-day reap silently kills the unit while is-enabled goes on saying enabled (blankpad-desk-autoswitch.service was dangling that way). Details: dotfiles-deploy, home-config-census.

2026-10-20 — engineering skills configured against this KB

setup-matt-pocock-skills ran, and its output moved out of the skill's default docs/agents/ into the knowledge base: issue tracker (Gitea on Dankclaw is the tracker — no gh, no tea, token via a 600 curl config), triage labels (the five canonical roles, created on the repo, which carried zero labels before), domain docs (single-context; glossary withheld until a term has no other owner), plus an empty decisions with its format rule. AGENTS.md gained an ## Agent skills pointer block. Law: this repo has one knowledge base, so per-tool configuration is a concept page in omp-docs/ — a second docs/ tree means a second set of index rules, which is where documentation starts disagreeing with itself. Known cost: re-running that setup skill will not recognise omp-docs/skills/* and will offer to recreate docs/agents/; the AGENTS.md pointer is the thing that stops it.

2026-10-19 — dankvoice auto-speak armed at startup

Sessions spoke only on explicit /say: the settings file carried no auto key, and arming is session-scoped by design (/say on never persists). Set "auto": true in ~/.omp/cache/dankvoice.json. Same report, second half: auto-speak listened only to agent_end — a settle that fires once per prompt and is skipped outright when the harness schedules a continuation — so nothing was narrated while an agent worked; fixed in dank/dankvoice#3, which narrates turn_end as well. Law: silence at session start is a config state — read /say status for auto-speak on, then watch pgrep -a pw-play for one player per segment, before blaming the endpoint. Law: whether the harness delivers an event to extensions is settled in minutes by a throwaway probe extension that timestamps turn_end/agent_end — never inferred from audio timing, which the humanizer lags by seconds. Details: dankvoice.

2026-10-19 — LazyVim installed as an NVIM_APPNAME profile

lazyvim had never been installed here — shell history shows the command typed twice and dropped; ~/.config/nvim held only an orphaned matugen.lua palette that nothing loaded. Now: LazyVim starter in dotfiles/lazyvim/ symlinked to ~/.config/lazyvim, a lazyvim zsh function, and the palette applied through RRethy/base16-nvim on ColorScheme. Law: the distro stays out of ~/.config/nvim so bare nvim keeps working as a vanilla quick-edit editor. Details: neovim.

2026-10-19 — dankvoice cut over to the dank/dankvoice package (Refs #34)

Drop-in agent/extensions/dankvoice.ts and its 26-test suite retired; the package carries the behaviour (41/41 black-box suite) and is installed with pi install git:. Law: install and prove /say status speaks on a box before pulling the cutover commit — the shared settings file keeps menu state across both transports, so the swap is stateless once installed.

2026-10-19 — dankvoice: silence and timeouts, root-caused (Refs #32)

The humanizer's raw fetch to frank-zealot omitted chat_template_kwargs. enable_thinking: false, so the Qwen35 hybrid thought instead of answering: 6.9 s of reasoning_content, content: "" — timing out the rewrite twice (12 s dead air) and, when it did answer inside the window, speaking silence. Root-caused by replaying the exact wire body against the live endpoint. Fix: pin the kwarg in the request body; treat empty-content-with-reasoning as failure (mechanical voice), not "nothing worth saying"; the menu's test phrase fires without blocking its handler. Lesson: the harness injects thinking-off for managed roles, but raw HTTP callers must set it themselves — 26/26 green with three red-captured regressions.

2026-10-19 — dankvoice: humanized narration and thinking aloud (Refs #27)

Spoken units are rewritten by the @tiny role (frank-zealot) under the bundle's speech budgets before synthesis, with mechanical cleanup as fallback; auto-speak narrates the last message's thinking blocks first; all toggles persist. 7 new scenarios, 22 green; live smoke through real zealot + frank-tts.

2026-10-19 — dankvoice: segmented sequential speech

agent_end awaiting one whole-answer synthesis hit the harness's 30 s extension-handler timeout — lesson: extensions must never await work that can exceed 30 s. Ported the built-in vocalizer's utterance-splitting state machine (boundary = [.!?…]+ + whitespace, abbreviation/code guards, per-answer segment budget); segments synthesize strictly sequentially. DANKVOICE_MAX is now a segment budget. 17/17 in tests/dankvoice.bats.

2026-10-18 — dankvoice: /say options menu with output-device routing (Refs #23)

/say menu (pi-peon-style): auto-speak, voice, PipeWire output device (wpctl → pw-play --target), player, char cap, test phrase. Precedence env > ~/.omp/cache/dankvoice.json > defaults — born from the incident where PipeWire's remembered Bluetooth route played answers into nowhere.

2026-10-18 — dankvoice: the cloned voices finally speak in the TUI (Refs #21)

The harness gates --voice against the bundled Kokoro enum, so frank-tts's cloned voices were unreachable in-TUI. Added agent/extensions/dankvoice.ts (/say …) synthesizing over the OpenAI speech route, endpoint read from the api: openai-speech row of agent/models.yml (no host or secret in code).

2026-10-18 — Noctalia bar layout clobbered by trunk-sync stash, recovered

GUI-built bar customizations existed only in the working tree; a trunk-sync stash-then-drop destroyed them. Recovery: git fsck --lost-found surfaced the dropped stash as a dangling commit; validated in a sandboxed XDG_*_HOME before restore. Lesson: GUI-written config is only durable once committed — the motivation for the live-sync heartbeat below.

2026-10-18 — Fleet law: issue/PR attribution footers

Every agent-opened issue/PR body ends with cwd: + session: footer, in both dank/command-center and dank/serverAppConfigs AGENTS.md: machine-made artifacts must trace to the box and session that produced them.

2026-10-18 — live-sync heartbeat for machine-written paths (Refs #11)

A month of silent divergence in the live checkout proved the checkout-is- deployment model needs a committer: scripts/live-sync.sh + systemd user timer auto-commits allowlisted tiers (memories, managed-skills, live config, dotfiles) with hooks armed, merges origin/main, pushes, reaps stale wt/*. Non-allowlist dirt never staged; conflicts abort-and-alert. See tools/live-sync.md; tier law in AGENTS.md.

2026-10-18 — Fleet law: filed PRs open in Frank's browser immediately

xdg-open <PR-URL> the moment a PR opens — codified in both repos' git rules.

2026-10-18 — Trunk law: extension route closed unmerged; direct repo alignment

Operator closed PR #9 (cwd-scoped before_agent_start extension) in favor of editing each repo's AGENTS.md. Durable findings: every context-file provider (AGENTS.md walk-up, .omp/ dirs, CLAUDE.md) resolves at or below the repo root — ancestor-scope files can never load; user-level APPEND_SYSTEM.md broadcasts to all sessions; serverAppConfigs never had git-flow.

2026-10-18 — SDLC switched to trunk-based development

develop retired (single-dev repo). Law in AGENTS.md § Trunk-based SDLC: branch off freshest origin/main, PRs target main, --no-ff merges, tags vX.Y.Z on main. Managed skill gitflow-back-merge-plumbing is historical.

2026-10-18 — Sibling-vault overlap census; system docs realigned to live ground truth

Census of omp-docs/ vs serverAppConfigs-docs (subagents + live-ssh probes) found four contradicting pairs; ours fixed, theirs flagged for their PR flow. Recorded then: RTX 3090 moved Dankserver→Dankclaw 2026-09; dankserver = 2× Radeon AI PRO R9700 (the "board swap" story was false); served models per live units. Current state always: /system/index.md (27B-era zealot figures here are historical; live census 2026-10-19 below).

2026-10-18 — Status line rebuilt on custom segments; full-source audit

New page core/statusline.md with an audit log — four first-pass claims were false and corrected against the minified bundle + a live screenshot matrix (context meter is box/band only; composer.shape routes segment arrays; mode states = Plan/Prewalk/Goal/Vibe/Loop). Bar rebuilt on band + custom segments; frank-archon display name slimmed in models.yml.

2026-10-09 — Blankpad DE redesign: desk profiles, keybinds, Handy wiring

Per-desk niri profiles keyed by EDID long name (connector names reshuffle between docks; identity travels) + scripts/desk.sh; niri 26.04 has no hotplug event (OutputConfigChanged is only an ACK per niri-ipc) — the autoswitch daemon diffs output sets on WorkspacesChanged. Keybind cheat sheet now generated (scripts/gen-keybinds.sh); noctalia docs corrected to the v5 noctalia msg API. Handy: niri owns autostart; open breaker — frank-secondary answers in reasoning_content with empty content, and Handy cannot send chat_template_kwargs.enable_thinking=false. 27/27 green.

2026-10-09 — Incident: agent sudo-loop tripped pam_faillock, locked out the operator

A subagent retried sudo yay (no TTY) for ~15 min; pam_faillock deny=3 locked user blank — correct-password sudo failed, mimicking a password change. Recovery: faillock --user blank --reset. Law: agents never run sudo/yay/pacman/pkexec; root steps go to the operator as exact commands. Skill faillock-sudo-lockout-recovery.

2026-10-09 — Full secret audit of the command-center repo: clean

Five layers (live-value scan across 77 revs/765 blobs, whole object store incl. dangling, shape triage, entropy ≥4.3, LFS payloads) — no secret in any reachable or dangling object; quarantine list holds. Fixed hf_token_write 0644→0600. Open gap: the pre-commit gate lacks hf_/npm_/pypi-/AIza/ Stripe/Telegram shapes and basic-auth URLs — widening it is an open decision.

2026-10-09 — Gitea token moved out of every remote URL

Remotes bare; credential in ~/.config/secrets/gitea (0600) behind a global credential.helper store. Trap (git 2.56.0, measured): git credential-store rejects the protocol=/host= block shape the man page shows — the store file must be one URL per line, port included. Verified with GIT_TERMINAL_PROMPT=0. Details: tools/gitea/client-access.md.

2026-10-09 — Working tree reconciled: callsign renames, compaction move, gitignore quarantine

200-entry dirty tree committed as seven atomic units. Harness model ids renamed frank-{medium,large,small} → frank-{archon,dragoon,zealot} — services keep old names on the wire; the engines accept any request model id, so renames are harness-side only. Server-side compaction moved to zealot's 3090 (threshold 70→85 %). .gitignore quarantine for worktree-keyed memory buckets and runtime dirs. Skill renames are not losses: git renders directory renames as mass delete+add; pre-cutover content intact at 12c5e88.

2026-09-28 — Retired workstation purged from the knowledge base

Sold box: hostname, both LAN addresses, Wi-Fi SSID gone bundle-wide; the knowledge distilled into system/network.md (multi-link resolver law: a static resolver must be pinned on each connection) and tools/niri/displays.md (portrait transforms, the three ways a multi-monitor rig breaks). Hostname stays only where it is a lookup key (install.sh --host, packages/<hostname>.pkgs).

2026-09-01 — Retry backoff and thinking default (pi era)

Pi settings.json: 10 s base retry backoff (10/20/40 s), defaultThinkingLevel: medium (no per-model thinking field exists in models.json).

2026-08-28 — KB rebuilt to OKF v0.1; live-verified fleet facts

omp-docs/ copied from legacy ~/.pi/agent/docs, rebuilt as git-tracked OKF v0.1 (checkout is the deployment); unverifiable claims tagged unverified. Conformance pass promoted concept-in-disguise indexes, stripped index frontmatter, resolved all links; scripts/kb-lint.sh exits 0. Hardware facts were corrected then (3090-on-dankserver since moved — see 2026-10-18). Inventory census recorded under inventory/ for the pruning session.

2026-08-23 — Homepage deployed to production; Caddy KB doc

Homepage live on Dankclaw via scripts/deploy.sh (status checker timer, deployed.js, rendered basicauth hashes, health check). Hard-won Caddy lesson: top-level directives are sorted, so rewrite runs before basicauth and auth matchers saw rewritten paths — serve each page from its own handle block to preserve order; a handle inline matcher is a single token; a leading @name <directive> parses as a matcher definition. Now documented on caddy/caddyfile.md. A compose mount change needs a container recreate, not a reload. 116 tests green.

2026-08-23 — Homepage design settled (grilling complete)

All 38 open questions answered; full record lives in serverAppConfigs (docs/Infrastructure/Homepage Design Grilling.md). Key deltas: per-service tcp/http probes, 5 s timeout, TUI-style panel grid, no client auto-refresh, git-driven deploy, full TDD. Correction recorded: the Dankserver media stack runs as bare processes, not containers — never probe it as a compose stack.

2026-08-22 — vllmScripts merged into serverAppConfigs

Full-history two-parent import; one repo, one checkout per host, one deploy.sh. The plaintext sudo password from old docs was NOT carried over (a bats leak-check guards it); it remains in both repos' history on the LAN Gitea — rotation recommended.

2026-08-21 — Fleet ops: passwordless sudo, Hermes SSH keys

/etc/sudoers.d/dank-nopasswd on both servers (per-command drop-ins remain); dedicated ed25519 key for the Hermes container, verified into all three machines.

2026-08-19 — pi-web-access: SearXNG endpoint re-pointed to dankclaw (pi era)

searxngBaseUrl moved off dead .235:8081 to .143:8081; pi caches web-search.json per process — new endpoint needs a new session. yt-dlp installed (2026.07.04).

2026-08-19 — SearXNG + Gluetun bring-up on dankclaw

Durable field notes (now on concept pages — see agentic-web-research/vpn-routing.md, searxng-tuning.md, engineering/infra/containers.md): manual-WireGuard Proton needs VPN_SERVICE_PROVIDER=custom + the single-peer WIREGUARD_* env names (not protonvpn, not WIREGUARD_PEERS_*); gluetun ships no curl and :8000 is the control server, not health; the limiter's JSON cap (API_MAX=4/hour) is hardcoded Python — pass_ip is the only bypass; rootless podman strips client IPs on published ports, so SearXNG must sit behind Caddy for XFF; quadlet generator rejects CapAdd=/Device= and generated units enable via [Install] WantedBy=default.target. ComfyUI on gfx1151: build on AMD's ROCm base and filter the torch family out of ComfyUI's requirements or you clobber the gfx1151 kernels.

2026-08-17 — Fleet redo + vLLM tool calling (pi era)

Fleet moved off the pipeline-parallel + NPU topology; Quark W8A8 quant chosen over the Avesed INT8 for clean Qwen3 XML tool calls.

2026-08-16 — Gitea boot-persistence fix (dankclaw)

unattended-upgrades reboot exposed that the compose-era Gitea had no systemd units (~1 h down). Stack rewritten as quadlet (names, systemd-gitea network, linger + WantedBy=default.target, app.ini env-override gotcha, UID 100999 ownership) across tools/gitea/*; podman generate systemd marked deprecated in engineering/infra/containers.md.

2026-08-13 — Pi subagent limits recorded (pi era)

Effective caps: globalConcurrencyLimit 2, parallel.maxTasks 8 / concurrency 4, maxSubagentDepth 2, turnBudget 30; no per-model limit. See pi/packages/pi-subagents.md.

2026-08-11 — Engineering research area; skills leaderboard verified

Created engineering/ (SDLC, agent instruction files, stacks, infra, plugins, skills ecosystem, Signal remote) via parallel research fleet. Skills leaderboard replaced with the ranked list from the skills.sh API (zero-based pagination, 200-skill pages).

2026-08-11 — Agentic web research section; exhaustive bundle audit

Created agentic-web-research/ (8 pages) and rewrote them on 2025/2026 evidence (SearXNG valkey deployment, real engine defaults, limiter budgets, Gluetun pattern, open-model claim triage, MCP transports). Audit: 154+ wikilinks resolved, dead external links fixed, MemorySleepMode=deep verified valid (systemd 256+), ASD-STE100 style pass. AGENTS.md created as the system behavior contract.

2026-08-06 — Link fixes; ComfyUI docs; major documentation overhaul

Fixed 15 broken wikilinks (missing frontmatter) + title mismatches → zero broken. Added ComfyUI pages, per-host system docs, pi config tree, tools entries; orphaned root duplicates moved to .AGENTTRASH/.

2026-07-18 — Maintenance tooling

Expanded references/okf-maintenance.md (cross-links, health metrics, troubleshooting); created core/log.md and .okf/config.yaml.

2026-01-15 — OKF conversion

Docs bundle converted to OKF v0.1; files migrated into core/, tools/, packages/, references/ with frontmatter on all files.

2025-08-05 — Gitea documentation

Added tools/gitea/index.md (topology, management, data persistence) to the tools index.